3775934 - oomkilled error during data flow execution - SAP datasphere

OOMKilled issue in dataflow; error: "OOMKilled Quick fix: increase graph/group memory limits. Current memory limits of killed groups: [{"default":"5Gi"}]"

• Product: SAP Datasphere
• Data Flow Runtime

1. Create or open a data flow that reads from a persisted remote table and writes to a local table.
2. Start the run with default settings (for example, fetch size and batch size not explicitly set).
3. Observe that the data flow stops with the error: "OOMKilled Quick fix: increase graph/group memory limits. Current memory limits of killed groups: [{"default":"5Gi"}]".

• The data flow graph process is limited to approximately 5Gi memory; this limit cannot be increased for data flows.
• Processing very large datasets with default or high fetch/batch sizes leads to memory exhaustion, resulting in the platform stopping the graph with OOMKilled.

1. Explicitly set smaller fetch and batch sizes in the data flow (for example, reduce from 1000 to 500, then to 200, and lower if required) and run again.
2. If the error persists even with very small fetch/batch sizes, limit the data volume per run (for example, split the workload into smaller chunks) and run incrementally.
3. Prefer using Replication Flow for large-scale table replication, as it provides enhanced memory and performance handling compared to Data Flow for such scenarios.
4. If the design requires only straightforward projection/mapping (for example, adding a timestamp column), implement the same logic in Replication Flow mappings and validate.

• SAP Note 3265875 - Previously working graphs fail with OOMKilled
• SAP Note 3345856 - Datasphere and Data Intelligence graphs leading to OOM errors
• refer to: Use Replication Flows and Transformation Flows to Load and Write Data
• refer to: Creating a Replication Flow
• refer to: Define Mappings in a Replication Flow

oomkilled, data flow, out of memory, memory limit, 5gi, fetch size, batch size, data integration, replication flow, large dataset, graph memory, runtime error, data loading, throttling, mapping , KBA , DS-DI-DF , Data Flow Runtime , Problem