3467066 - Data Flow Script Operator Restriction: Data Processed in Batches, Not as Full Table - SAP Datasphere

When creating a data flow and using a script operator in SAP Datasphere, the incoming table is not provided as a complete dataset. Instead, the data is split into multiple batches of rows (depending on the size of the table).

• The transform function is therefore invoked multiple times, once for each batch.
• In each call, the data parameter only contains the rows of the current batch.

As a result, operations that require access to the entire table within the script operator are not possible. For example, removing duplicates.

SAP Datasphere

script, DataFrame, datasphere, script operator, NumPy, data flow, script operator, transform function, batches, duplicates, restriction, complete table , KBA , DS-DI-DF , Data Flows , Problem