2767376 - Employee Records missing in the API response while your scheduled replications are running using last_modified_on filter - Successfactors HXM Suite

You are noticing that few records are getting missed in the response payload while making  CompoundEmployee API calls or OData calls running your processes using Last Modified On as filter i.e. pick only those records which has been changed since last successful run.

One of the potential root cause would be data was not saved completely to database (DB) while API request was triggered i.e. around the time that you see in your API request for LastModifieOn filter parameter.

This usually occurs when:

• Your HR do mass imports using CSV files (Admin center > Import Employee Data or Import and Export Data)
• Your team uses EC Business rules with PostSave

The standard replications or custom replications triggers API call at the very same time when your team is doing changes via CSV mass import and/or EC Business rules with PostSave which can delay the processing of the records in SF database, the filter of LastModifiedOn of the previous run and the subsequent run would not capture the changed employee record.

We have seen this behavior of records missed in the standard replications and no errors can be seen in the Data Replication Monitor UI, middlewares Boomi/CPI or ERP side (SLG1/SRTUTIL). The employee records are updated successfully in the EC system but this information is not received in the ECP/ERP application.

This also occurs more often if your team decided to run integrations in a shorter period (like 1, 2 or 5 minutes between the runs).

SAP SuccessFactors HXM Suite

• CompoundEmployee API
• OData API
• SFAPI 

There is one time gap between the last modified and DB storage in the scenarios described above for replications with APIs running the same time of the data being changed in the SF database level.

If there are any news about further improvements for this topic, they will be communicated via this KBA.

• For situations where you are using restricted access permission for the API user, where possible use admin access instead of restricted access (or ensure the relevant employee population is included, cf. Granting Permissions for Restricted Access to the CompoundEmployee API | SAP Help Portal).
• For situations where you are importing HRIS-based employee data, reduce the import batch size to a lower value. This is done via “Company System and Logo Settings”, setting “Set a batch size for employee and foundation data imports.”
• For situations where you are importing MDF-based employee data, reduce the import batch size to a lower value for this entity. This is done via “Import and Export Data”, Select the action to perform “Import Config”, Select Generic Object.
• For situations where you use EC Business rules with PostSave, one possible solution is changing for OnSave instead, this will avoid this database delay of the field "LastModifieOn".

• For situations where you do not have the EC Business rules and you still have missing records while performing CSV mass imports, as a workaround to avoid any data loss during API call based on middleware and/or standard content used, the recommendation is:

• scheduling additional nightly job would make sure all those records which went missing get replicated successfully.

Also, there would not be any impact on employees whose data has already been replicated successfully during standard scheduled executions.

For that, we are explaining below how to do this process for standard Integration scenarios available in EC Payroll point to point Integration, CPI or Boomi middleware. You can adapt these scenarios for your other possible integrations too (custom application or other scenarios relying with SF APIs OData and SFAPI).

Employee Central Payroll (Point-to-point Integration)

• Copy the affected PTP configuration ID
• via following IMG Activity: Personnel Management -> Integration Settings for SuccessFactors Employee Central Payroll -> Configuration of Point-to-Point Replication -> Configure Compound Employee API Query
• Or via SM34 and view cluster T77SFEC_PTP_COCL 

• Adjust the last modified date for copied PTP configuration ID.
• Go to transaction SM30
• Maintain view HRSFEC_PTP_LMOD
• Add a new entry with the copied PTP configuration ID and adjust the last modified timestamp correctly.

• Schedule a new nightly job for the copied PTP configuration ID.

DELL Boomi 

• For standard content deployed on boomi atom, you can setup another atom where you need to deploy the same version of the standard iFlow with the same configuration.
• You should schedule the deployed iFlow to execute outside business hours to pick all the changes that has been done in a day.
• Last Modified Date in the configuration should be selected accordingly.

Example:

• Go to the atom where your standard iFlow has been deployed and execution currently. Go to the deployed process and go to the "Edit Process Properties".
• Here, you would see highlighted parameter value which will be used in your next iFlow execution and this would be set as value for filter parameter "LastModifiedOn"

• For new atom where you are going to deploy same iflow again, make sure all configuration under "Environment Extension" is same as one used in standard schedule.
• Only, property that you need to set is under "Edit Process Properties" as highlighted above. So, add the property "ECERP_LAST_MODIFIED_DATE" manually and set the value as per your requirement.
• Note that format shout be in UTC only as shown above. So, before you set the value, make sure you convert the timestamp in UTC and maintain here.

SAP Cloud Platform Integration

• CPI scheduled:
• Copy the process second time from the process library (can be deployed on the same tenant) Complete the configuration same as the existing iflow (except the scheduling part). Schedule outside business hours to catch-up all the missed changes. as shown below
• Make sure new iFLow has all configuration similar to the standard schedule one apart from one parameter "USER_SET_LAST_MODIFIED_DATE_TIME" as shown below:

• CPI scheduled via ERP (lastest version BIB):
• Create a second scheduling job in ERP. You should schedule an additional job with a new variant, where a dynamic change date is used (with sy-datum – 1) to have a new replication with the changes included. 

We have this behavior documented in the SFAPI CE handbook (direct link).

We also have this behavior documented in the KBA 2092850 - How to make changes to Employee Data on mass via Import - Employee Central

Records Missing, Employee Master Data Replication, Organization Assignment, CSV mass massive import, Admin center > Import Employee Data or Import and Export Data, API, SFAPI, CompoundEmployee calls, OData, SuccessFactors HCM Suite, data extraction, skipped employees record, time gap, LastModifiedOn, filter, persons not extracted , KBA , LOD-SF-INT-CE , Compound Employee API , LOD-SF-INT , Integrations , LOD-SF-INT-ODATA , OData API Framework , LOD-SF-INT-API , API & Adhoc API Framework , LOD-EC-INT-EE , Employee Integration EC to ERP On Premise , LOD-EC-GCP-PY , Payroll Integration EC to Employee Central Payroll , Problem