OneNote API calls fail with a large number of items in a SharePoint document library


Hello world! Paresh Moradiya and Sri Srinivasan, members of the OneNote API team here. We’ve heard from some of you saying that you’re running into issues when accessing SharePoint-hosted notebooks in very large document libraries. We have come up with some instructions on how you can eliminate these OneNote API failures.

The problem

Applications call the OneNote REST API to create or access OneNote content in Office 365. Sometimes the OneNote API calls can fail when they are executed against SharePoint document libraries that contain a large number of items (items can be folders or files, some of which could be OneNote entities such as notebooks, sections or section groups).

When working with SharePoint document libraries that contain a large number of items, the OneNote API service is subject to throttling measures implemented by SharePoint for the purposes of regulating resource usage. Under these circumstances, the user of the application may see an application-specific error. The application calling the OneNote API will itself receive one of the errors below:

{
“error”: {
“code”: “10008”,
“message”: “One or more of the document libraries on the user or group’s OneDrive contains more than 5,000 OneNote items (notebooks, sections, section groups) and cannot be queried using the API. Please make sure that none of the user or group’s document libraries contains more than 5,000 OneNote items. Please follow the link below for instructions on how to remedy this situation.”,
“@api.url”: “http://aka.ms/onenote-errors#C10008
}

{
“error”: {
“code”: “10013”,
“message”: “One or more of the document libraries on the user or group’s OneDrive contains more than 20,000 items and cannot be indexed for querying using the API. Please follow the link below for instructions on how to remedy this situation.”,
“@api.url”: “http://aka.ms/onenote-errors#C10013
}

The solution

To mitigate the OneNote API failures described above, you must first assess the current state of the SharePoint document libraries on the SharePoint site and then take a few remedial steps based on the current state.

Assess the current state

Download the OneNote API Diagnostics tool available here and follow the steps below.

  1. Enter the following parameters into the tool
    Parameter Description
    SharePoint URL The SharePoint site’s URL. For example, a user’s personal site might use the format https://<<Your Tenant>>-my.SharePoint.com/personal/<<Your email address - Replace all "." with "_">> (contact your administrator for the format of your personal site’s URL)

     

    You can also enter the URL for a non-personal SharePoint site

    User Your Office 365 user name (email address)
    Password Your office 365 password
  2. Click the “Get Info” button. This will fetch the diagnostic information from the specified SharePoint site. The operation may take a few seconds to process.

NOTE: The tool processes only the default document library on the SharePoint site by default (for a user’s personal site, this is the user’s OneDrive for Business). If the API issues above are not resolved by just targeting the default document library, you can choose the tool’s option to target all document libraries on the SharePoint site.

Take appropriate remedial steps

Based on the output of the diagnostics tool, you can take the remedial steps suggested below. These steps are applicable at a document library level. If you chose to target all document libraries on the SharePoint site, you will need to execute the remedial steps below on all the document libraries.

Case 1: Items Count is greater than 20,000

If Items Count is greater than 20,000 and File Type = false or HTML File Type = false, then you need to bring the item count in the document library to under 20,000. You can do this by moving items out of the document library to a different storage location (such as another user or group’s OneDrive). You can also delete the items that are not required anymore.

After you have reduced the item count to under 20,000, rerun the tool to make sure the items count is less than 20,000. Click the “Build Index” button so the required indexes are built for the OneNote API to work properly.

Case 2: Sections Count is greater than 5,000

If Sections Count is greater than 5,000, then you need to bring the section count in the document library to under 5,000. You can do this by moving some of your notebooks out of the document library to a different storage location (such as another user or group’s OneDrive). You can also delete the notebooks that are not required anymore.

After you have reduced the section count to under 5,000, rerun the tool to make sure the sections count is less than 5,000. Ideally you should keep this number below 5,000 so that you can have enough room to create new sections. If the sections count in the document library crosses 5,000 again, you will start seeing the OneNote API failures and you will need to come back here and take the same remedial steps.

Case 3: Folders Count is greater 5,000

If Folders Count is greater than 5,000, then you need to bring the folder count in the document library to under 5,000. You can do this by reorganizing the content in the document library and eliminating folders. You can also delete the folders that are not required anymore.

After you have reduced the folder count to under 5,000, rerun the tool to make sure the folder count is less than 5,000. Ideally you should keep this number below 5,000 so that you can have enough room to create new folders. If the folder count in the document library crosses 5,000 again, you will start seeing the OneNote API failures and you will need to come back here and take the same remedial steps.

Case 4: Notebooks Count is greater than 5,000

If Notebooks Count is greater than 5,000, then you need to bring the notebook count in the document library to under 5,000. You can do this by moving some of your notebooks out of the document library to a different storage location (such as another user or group’s OneDrive). You can also delete the notebooks that are not required anymore.

After you have reduced the notebook count to under 5,000, rerun the tool to make sure the notebook count is less than 5,000. Ideally you should keep this number below 5,000 so that you can have enough room to create new notebooks. If the notebook count in the document library crosses 5,000 again, you will start seeing the OneNote API failures and you will need to come back here and take the same remedial steps.

Comments (12)

  1. James Katt says:

    Please Satya Nadella,
    Get Microsoft’s act together.
    It is silly to have item limitations and limitations on the path length on OneDrive.
    Other Cloud Drives don’t have these limitations.

  2. IoDavide says:

    The 20,000 items limit was removed on OneDrive 2 years ago (https://blogs.office.com/2015/12/16/onedrive-for-business-update-on-storage-plans-and-next-generation-sync-client/), it is your API that is not working properly.

    So what?

    1. IoDavide says:

      In add, the tool doesn’t show anything. It work a few seconds, than show again the regular windows without any data.

  3. Dan Bryan says:

    Still unable to load notebooks in OneNote Web Clipper and getting error message about a list limit being exceeded in Sharepoint. Running diagnostic tool on all lists shows:

    List: Form Templates
    Items Count: 0
    Notebooks Count:0
    Folders Count:0
    Sections Count:0
    Content Type ID:False
    File Type:False
    HTML File Type:False
    Content Type ID:False
    ————————————
    List: Incoming Faxes
    Items Count: 0
    Notebooks Count:0
    Folders Count:0
    Sections Count:0
    Content Type ID:False
    File Type:False
    HTML File Type:False
    ————————————
    List: Legal Reference
    Items Count: 3
    Notebooks Count:0
    Folders Count:0
    Sections Count:0
    Content Type ID:False
    File Type:False
    HTML File Type:False
    ————————————
    List: Master Forms
    Items Count: 1224
    Notebooks Count:0
    Folders Count:101
    Sections Count:0
    Content Type ID:False
    File Type:False
    HTML File Type:False
    ————————————
    List: OneNote Notebooks
    Items Count: 203
    Notebooks Count:4
    Folders Count:33
    Sections Count:117
    Content Type ID:False
    File Type:False
    HTML File Type:False
    ————————————
    List: Permission Test Library
    Items Count: 2
    Notebooks Count:0
    Folders Count:0
    Sections Count:0
    Content Type ID:False
    File Type:False
    HTML File Type:False
    ————————————
    List: Shared Documents
    Items Count: 2
    Notebooks Count:0
    Folders Count:0
    Sections Count:0
    Content Type ID:False
    File Type:False
    HTML File Type:False
    ————————————
    List: Site Assets
    Items Count: 3
    Notebooks Count:0
    Folders Count:2
    Sections Count:0
    Content Type ID:False
    File Type:False
    HTML File Type:False
    ————————————
    List: Site Collection Documents
    Items Count: 0
    Notebooks Count:0
    Folders Count:0
    Sections Count:0
    Content Type ID:False
    File Type:False
    HTML File Type:False
    ————————————
    List: Style Library
    Items Count: 496
    Notebooks Count:0
    Folders Count:163
    Sections Count:0
    Content Type ID:False
    File Type:False
    HTML File Type:False

    I do not seem to be close to any list limits but tool is still not working. Please advise.

  4. Debby Sadler says:

    Please lift the limit on the number of notebooks/pages and folders in regards to being able to use the one note clipper and also saving emails from outlook to one note. I house several notebooks and documents on my business sharepoint site that I must keep active and intact. Moving the items to another person’s notebook or location is not an option and I am sure this is the case for many others that utilize these Microsoft products in the workplace.

  5. Bernhard König says:

    I came here from a link in an error message in OneNote Web Clipper. I understand the issue, but find the proposed solutions to be unacceptable. OneDrive for Business offers large storage capacities to my users and of course they use them. Limiting them to 20,000 files or 5,000 folders makes the whole service more or less useless in it’s current form.

    So my question is: are you actively working on a solution to fix this on your side? Actually, what a tool like Web Clipper needs, is a list of all available notebooks in a OneDrive for Business library. As I understand it, there is no way to filter for those file types over the API (or at least not in an efficient way that would not cause too high API load). But IMHO, it can’t be that hard to offer a call over the API that can do this – and in an efficient way.

    I understand you can’t allow queries over the API that return 20,000 or more files, but in most cases, there will be just a low two-digit number of notebooks in any library – it should be possible to do a fast query for them that does not put excessive load on your API if only the contents are properly indexed.

    I’m fine on the other hand with a notebook count lower than 5,000 … that should be plenty. But 20,000 files or 5,000 folders is a limitation that does not fit to any real world use case of OneDrive. This is something you have to solve, if on the API team or in the OneDrive for Business core team.

  6. Joey says:

    None of these cases apply to me yet I am still having the issue where it will not load the list of notebooks. Here is the output from the tool. Any ideas?

    List: Documents
    Items Count: 11082
    Notebooks Count:6
    Folders Count:2683
    Sections Count:63
    Content Type ID:True
    File Type:True
    HTML File Type:True
    ————————————
    List: Form Templates
    Items Count: 0
    Notebooks Count:0
    Folders Count:0
    Sections Count:0
    Content Type ID:False
    File Type:False
    HTML File Type:False
    Content Type ID:False
    ————————————
    List: Style Library
    Items Count: 5
    Notebooks Count:0
    Folders Count:1
    Sections Count:0
    Content Type ID:False
    File Type:False
    HTML File Type:False

  7. JC says:

    What are you going to do about your own applications using this API, such as “Save emails to OneNote” (aka me@onenote.com)? Because of the 20,000 items limitation I can’t even set up this first-party service.

    1. Hi JC

      This problem is not specific to service but to the storage. SharePoint team has published good articles on how to manage large list/library. I have pulled information from that which is specific to OneNote service. User falls into above categories may have similar issues with our feature like “Save Emails to OneNote”, Web Clippers etc. User should be running above diagnostics against personal site to solve the issue.

  8. Steve Colley says:

    Hi Paresh
    Thanks for providing the diagnostic tool and pointing out the issue. As my app has to deal with thousands of notebooks (each with ~10 sections) I’ve been acutely aware of this issue and already move notebooks that are not actively being worked on out of the Documents folder. Is there any scope for providing a way of just flagging a notebook that should be removed from the “active list” to reduce the resource usage without having to go to all the trouble of moving documents back and forth? Even if it was a file naming convention such a certain character sequence that allows SharePoint to skip a certain notebook folder?
    By the way loving the Onenote api, any chance it might make it out of beta any time soon?
    Cheers Steve

    1. HI Steve

      I am glad that you are enjoying using OneNote API. This is great suggestion but it would not work for this scenario. Inactive notebook would work as secondary filter. Once we mark a notebook inactive, we need to ask SharePoint “Give me all notebooks but not inactive”. SharePoint still need to filter notebooks as primary filter and when its scan goes beyond 5000 it will throw throttling exception to OneNote API layer. We have couple of solutions but that would make API slow and hard to use. Considering this fact, the work around is simple. I am also updating the tool to allow users to move notebooks across document library.

Skip to main content