2723468 - How to avoid missing/duplicate data by enabling Server-side Pagination?

You are calling some SuccessFactors API endpoint but missing or duplicate data is showing up in the API response.

"Image/data in this KBA is from SAP internal systems, sample data, or demo systems. Any resemblance to real data is purely coincidental"

SAP SuccessFactors HCM Suite

A single GET/query request can return at most 1000 records and when the number of records is higher than that, the remaining records are returned in different API responses ("pages") until all records are retrieved. This logic is called Pagination.

There are two main types of pagination:

• Client-side Pagination: Client-side pagination uses query options on the client side to create an offset that restricts the amount of data returned from the server;
• Server-side Pagination: Unlike client-side pagination, where the pagination is controlled by client-specified parameters, server-side pagination is controlled on the server side;

During the Client-side Pagination with OData API, if a massive change or a change in some involved record happens, the engine can miss or duplicate some records.

The same can happens when using the deprecated SFAPI protocol (simple entities), which uses an outdated Server-side Pagination logic.

This issue can be avoided by using Server-side Pagination with OData API or using CompoundEmployee API, which has this pagination type as default.

In addition, using Server-side Pagination provides performance boost when querying large data set.

For OData API protocol, there are two types of Server-side pagination, Cursor-based (Cursor-based Pagination) and Snapshot-Based (Snapshot-Based Pagination) and this KBA demonstrates how to use Snapshot-based Pagination in CPI, Boomi, Integration Center (IC) and direct OData API calls.

OData API calls

Include the parameter paging=snapshot in your OData API call as the example below:

https://api12preview.sapsf.eu/odata/v2/EmpJob?$select=userId&paging=snapshot

Boomi

In your SuccessFactors Partner connector settings, enable the "Enable Snapshot Pagination For OData Queries" checkbox.

Note: The customPageSize parameter is set to 200 only to show the example and usage of the parameter, this needs to be adopted by the implementation consultant accordingly based on the response time and number of navigation entities being queried.

CPI

Check out the KBA 2830254 - How to enable Server Snapshot Based in SuccessFactors OData API using SAP Cloud Platform Integration CPI.

Integration Center

In your IC jobs settings, under "Options" OR "Destination Settings" tabs, select the "Enable Server Side Pagination" checkbox and save.

Pagination chapter from SAP SuccessFactors API Reference Guide (OData V2);

miss, missing, missed, duplicated, incorrect, inacurate, wrong, different, discrepant, data, value, values, record, records, return, returns, returned, query, get, api, call, slowness, performance, how to, enable, set, use, snapshot, cursor, server, based , KBA , LOD-SF-INT-ODATA , OData API Framework , LOD-SF-INT-INC , Integration Center , LOD-SF-INT-API , API & Adhoc API Framework , LOD-SF-INT-BOM , Standard SF to 3rd Party Boomi Content , LOD-SF-INT-CPI , Standard SF to 3rd Party CPI (HCI) Content , Problem