3712099 - Does adding data in the same table in data source impact the performance in our Data Pipeline?

• The duration of the pipeline's T&L refresh is increasing over time.
• Concerns regarding performance impact when adding data to the same table in the data source.

• Data Source (CSV, Connector)

1. Store historical BSEG data in the data source table.
2. Continuously add new data to the same table over time.
3. Observe the increasing duration of the pipeline's T&L refresh.

• The Transform step processes extracted data into a single event log file.
• The Load step always overwrites the entire Process data with that new event log.
• Because the Process data (event log) is replaced every run, the Transform must produce whatever the Process requires each time.

• Transform work grows with the volume of input rows.
• As more data is extracted, the Transform must process more rows, so run time increases.
• Because Load overwrites Process data every T&L run must produce a complete (or at least required) event log.

• Add WHERE clauses or other filters in Transform scripts to limit processed rows (examples: date ranges).
• Benefit: less data processed → shorter Transform and Load times.

Refer to: SAP Signavio Process Intelligence Documentation  

Refer to: Data Extraction Options User Guide

data source, process intelligence, historical data, pipeline refresh, T&L refresh duration, performance impact, BSEG data, data table, data management, data pipeline , KBA , BPI-SIG-PI-DS , Data Source (CSV, Connector) , Problem