3467066 - Data Flow Script Operator Restriction: Data Processed in Batches, Not as Full Table - SAP Datasphere

When creating a data flow and using a script operator in SAP Datasphere, the incoming table is not provided as a complete dataset. Instead, the data is split into multiple batches of rows (depending on the size of the table).

• The transform function is therefore invoked multiple times, once for each batch.
• In each call, the data parameter only contains the rows of the current batch.

As a result, operations that require access to the entire table within the script operator are not possible. For example, removing duplicates.

SAP Datasphere

This behavior is by design. The script operator is optimized for batch-wise processing of incoming data and does not receive the complete table in a single function call. As a result, any operation that relies on global visibility of all rows (e.g., duplicate elimination, sorting, global aggregation) cannot be achieved within the script operator. 

This is a documented restriction of the script operator. See the official help documentation: Create a Script in a Data Flow | SAP Help Portal.

Script operators should only be used for row-level or batch-level transformations that do not depend on having visibility of the entire dataset.

Script Operator Python Reference

script, DataFrame, datasphere, script operator, NumPy, data flow, script operator, transform function, batches, duplicates, restriction, complete table , KBA , DS-DI-DF , Data Flow Runtime , Problem