3660696 - When Extracting SAP ERP tables the logs show the WHERE clause filtering for some columns that are not mentioned in the SQL filter

Extraction strategy used is Automatic Pagination
Example:
Table is BKPF

SQL Filter is:

KOART IN ('K', 'D')
AND GJAHR >= 2023 
– added 0919, later fixed
AND ( 
    BELNR IN (
        SELECT BELNR
        FROM BKPF
        WHERE (
            (CPUDT >= '2025-01-01' AND CPUDT <= '2025-01-07')
            OR (AEDAT >= '2025-01-01' AND AEDAT <= '2025-01-07')
            OR (UPDDT >= '2025-01-01' AND UPDDT <= '2025-01-07')
        )
    )
)

The extraction logs show the SQL query with the WHERE clause filtering for other fields not in the above clause. Example:

...
FROM "BKPF"
WHERE (AEDAT >= '2025-07-28' OR UPDDT >= '2025-07-28' OR CPUDT >= '2025-07-28') AND ((bukrs > 'ABCD') OR (bukrs = 'ABCD' AND belnr > '1234567890') OR (bukrs = 'ABCD' AND belnr = '1234567890' AND gjahr > '2025')) LIMIT 25000  

When using Automatic Pagination extraction strategy, for every extraction of a page, the values of the key fields last extracted are saved internally and added to the WHERE clause for the next page extraction.

For example:

The table BSEG has a compound key containing 4 fields:
BELNR, BUKRS, GJAHR and MANDT. You should be able to see this in the Tables Columns tab.

When the first page (by default this is 25000 rows) is extracted, internally the last key is saved (the values of BELNR, BUKRS and GJAHR. MANDT stores the Client number which is specified in the Connection itself so it is not necessary to be used here). For the next page extraction (of 25000 rows) the query WHERE clause is build to include these fields so that the new records retrieved are greater than the saved last extracted values.

So the WHERE clause is made up of two parts
1. from the SQL Filter. For example:
WHERE (AEDAT >= '2025-07-28' OR UPDDT >= '2025-07-28' OR CPUDT >= '2025-07-28')

And
2. from the internally saved key values after previous 25000 rows extracted:
AND ((bukrs > 'ABCD') OR (bukrs = 'ABCD' AND belnr > '1234567890') OR (bukrs = 'ABCD' AND belnr = '1234567890' AND gjahr > '2025')) LIMIT 25000

After the second page is extracted, the new latest values of the key fields are saved for the following extraction of 25000 rows and so on.

soe, Process, Intelligence, Automatic, Pagination, Extraction, Key, Fields, logs, log  , KBA , BPI-SIG-PI-DS , Data Source (CSV, Connector) , Bug Filed

SAP Signavio Process Intelligence all versions