I need to load a SharePoint list into an Azure Logic App. The standard Get items action can paginate, but retrieving the data takes nearly two minutes. Using an HTTP request and selecting only the fields I need is much faster, but the response stops at 5,000 items, and the HTTP connector does not appear to offer usable pagination. I cannot modify the SharePoint list. Is there a way to retrieve all the items directly in the Logic App, perhaps by following the next link returned in each HTTP response?
3 Answers
If the list is especially large, the `RenderListDataAsStream` SharePoint endpoint can be worth considering. It is designed for large-list views, supports a row limit, and returns a next-page value. It takes more setup than the regular items endpoint, but it can be more tolerant of threshold-related queries.
Another option is to page by the built-in ID rather than depending on an opaque skip token. Query a range such as `ID gt 0 and ID le 5000`, then continue with the next ID range until a page is empty. ID is indexed by SharePoint, so this approach avoids threshold problems and can be easier to control. If you need to filter by a business column but cannot add an index, retrieve the records by ID ranges and apply the remaining filter inside the Logic App. For very large lists, avoid accumulating every page in one array because action outputs and variables have size limits; process or store each page as it arrives instead.
You can handle this with an Until loop instead of relying on the HTTP action's built-in pagination. SharePoint's REST response commonly returns the next page as `odata.nextLink`, which the connector may not recognize automatically. Initialize a string variable such as `nextUrl` with the first request URL and an array variable for results. In each loop, call the URL in `nextUrl`, append the response's `value` array, then set `nextUrl` to `body('HTTP')?['odata.nextLink']` or an empty string when there is no next page. Use bracket notation because the property name contains a dot. Request only the needed fields and use `odata=nometadata` to reduce response size. Also configure the Until loop's count and timeout explicitly.
The HTTP connector's pagination setting may not work here because it expects a different next-link property format. Following the SharePoint link explicitly in the loop is more reliable.
The 5,000-item threshold is generally about how many rows SharePoint must scan for an unindexed filter or sort, not a hard limit on the total number of items you can retrieve. Filtering and ordering by ID is usually the safest workaround.