3803168 - Transformation Flow from file space to storage space is slow - SAP Datasphere

Transformation Flow from File space to Storage Space has performance issue.

• Product: SAP Datasphere
• Transformation Flows

1. Run a Transformation Flow that reads from the object store/file space and writes to a storage space table.
2. Apply a filter on a date column and configure batching (for example, very small batch sizes on a date column that create many batches).
3. Execute the flow and observe long per-batch runtimes and high total duration, especially when many small batches are generated.

• The object store table was optimized with Z-order on high-cardinality key columns not aligned with the Transformation Flow’s filter (date), preventing effective file skipping and causing large file scans from the data lake.
• Very small batch sizes led to many batches and high orchestration overhead per batch, increasing total runtime.

Please consider the following solutions:

1. Enable the Performance-Optimized setting for the Transformation Flow.
See SAP Help:  Change Transformation Flow Settings | SAP Help Portal.

2. Use a Graphical View as the source instead of a SQL View for the Transformation Flow according to KBA 
3732192 Transformation Flow loading from SQL View uses excessive memory compared to Graphical View - SAP Datasphere

3. Increase batch size for simple load-focused flows to reduce the number of batches and orchestration overhead; avoid very small batch sizes that produce many batches. Balance throughput against memory/time limits.

4. Align Z-order with the query filter: re-OPTIMIZE the data in the object store with Z-order on the date column used in filters (for example, posting date) and optionally a secondary selective column (for example, company code). Ensure the Transformation Flow filter predicates target the same columns used for Z-order so that file skipping can occur.
See SAP Help: Define Z-Order Columns While Running an Optimize Task.

If performance remains insufficient after applying the above, refer to the documents below and contact support or consulting for further tuning guidance.

• Refer to: Note 3732192 - Transformation Flow loading from SQL View uses excessive memory compared to Graphical View
• Refer to: Note 3538038 - Important considerations for SAP Datasphere object store
• Refer to: Note 2626344 - Who is the SAP Account Manager/Executive for my company?
• Refer to: Change Transformation Flow Settings
• Refer to: Optimize with Z-order in SAP Datasphere

Datasphere, Transformation Flow, object store, file space, storage space, performance, slow load, batch size, packaging, Z-order, optimize, posting date filter, data lake, file scan, Graphical View, Performance-Optimized , KBA , DS-DI-TRF , Transformation Flows , How To