API Pagination Methods for Large Data Retrieval

Pagination returns the pages of response.

There are 6 types of Pagination in API Data Source of eZintegrations as Out of the box pagination settings. These are industry standard API architecture based pagination

• Next URL Pagination
• Offset Pagination
• Total Page Count
• Pagination with Body
• Cusrsor Pagination
• Encoded Next Token
• Custom Pagination

Next URL Pagination

This pagination will apply when we have URL of next page in our API response. For example we have next link in this key @odata.nextLink

Select Next URL pagination from the dropdown and fill these values there.

• In the above example @odata.nextLink key helps to identify upto which page the pagination will be completed , so it will come in User Key
• Data Collection Key holds the data from the API and we will get the response in that key for example here it is ['value']
• Another Response Parameter key will hold the response of the next url page, This value needs to same as that of Data Collection Key.

Typically being used in Microsoft Graph API, Dropbox, Box Cloud

Offest Pagination

This pagination we will apply when we have "hasMore":true in our API response.

Select offset pagination from the dropdown and fill these values there.

• In the above example hasMore key helps to identify upto which page the pagination will be completed , so it will come in User Key
• As we are using Offset pagination so Key Name to Update will be offset
• Data Collection Key holds the data from the API and we will get the response in that key for example here it is ['items'].

Typically being used in Oracle Netsuite, Oracle Fusion Cloud Applications etc.

Total Page Count

This pagination will apply when we have total_pages key in our API response. For example we have total page Count in this key total_pages

Example 1: When we have the user key in root keys

Select Total Page Count from the dropdown and fill these values there.

• In the above example total_pages key helps to identify upto which page the pagination will be completed , so it will come in User Key
• As we will be updating page value for going to next page data so Key Name to Update will be page
• Data Collection Key holds the data from the API and we will get the response in that key for example here it is ['data'].

Example 2: When we have the user key in nested form.

Select Total Page Count from the dropdown and fill these values there.

• In the above example total_pages key helps to identify upto which page the pagination will be completed , so it will come in User Key, as we can see in the example data total_page key is deep nested so we will pass the User Key as above.
• As we will be updating page value for going to next page data so Key Name to Update will be page
• Data Collection Key holds the data from the API and we will get the response in that key for example here it is ['data'].

Typically being used in all eCommerce Applications

Pagination with Body

This pagination will apply when we have nextPageToken key in our API response. For example we have next page token in this key `nextPageToken

Select Pagination with Body from the dropdown and fill these values there.

• In the above example nextPageToken key helps to identify upto which page the pagination will be completed , so it will come in User Key
• As we are getting a token from this pagination so Key Name to Update will be pageToken
• Data Collection Key holds the data from the API and we will get the response in that key for example here it is ['reports'][0]['data']['rows'].

Typically being used in all Google Products like Google Analytics, Google Ads etc.

Cursor Pagination

This type of pagination ususally sends a cursor in response header when a API is requested. This is typically used in User Interface based APIs. It provides a link for the next page but this comes in header response and not in the dataset

Typically being used in eCommerce Apps like Shopify, WooCommerce etc.

Encoded Next Token

This type of pagination is similar to Next URL Pagination but here the NextToken key is encoded

Very rarely used in API but one such example is Amazon Seller Central APIs

Custom Pagination

In this type of pagination, users can define their own logic to fetch the next set of records when the API does not follow any of the standard pagination styles. A custom Python script can be written to handle special cases, such as encoding/decoding a token or applying transformations before making the next request.

Example 1: Amazon SP API

API Response :

Custom Pagination Script:

import base64

response_parameters = "['items']"

nextToken = api_response['pagination']['nextToken']

result1 = nextToken.rstrip("=")

skip_token = result1

Params:

pageToken : {%skip_token%}

Pre Request Script:

skip_token = ' '

Explanation:

• api_response → refers to the entire JSON response returned by the API.

  • items → contains the actual product records.

  • pagination.nextToken → is the token required for the next page of results.

• The script strips trailing = from the nextToken before assigning it to skip_token.

• The param pageToken dynamically takes {%skip_token%}.

• The pre-request script initializes skip_token as an empty string for the first call.

Example 2: SAP Shipment API

API Response :

Custom Pagination Script:

response_parameters = "['value']"

next_link = api_response.get('@odata.nextLink')

skip_token = next_link.split("$skiptoken=")[-1]

Params:

$skiptoken : {%skip_token%}

Pre Request Script:

skip_token = 0

Explanation:

• api_response → refers to the entire JSON response returned by the API.

  • value → contains the actual inventory records.

  • @odata.nextLink → provides the URL for the next request, including the $skiptoken.

• The script extracts the skip token value by splitting the @odata.nextLink at "$skiptoken=".

• The param $skiptoken dynamically takes {%skip_token%}.

• The pre-request script initializes skip_token to 0 for the first call.