2785814 - HANA high CPU utilization caused by large backup catalog

• High cpu utilization by HANA hdbindexserver process
• Runtime dumps collected during high cpu are showing e.g:
  
  

  671347[thr=97002]: SqlExecutor, TID: 106, UTID: -1, CID: 202084, LCID: 202084, parent: ---, SQLUserName: "SYSTEM", AppUserName: "joe-adm", AppName: "HDBStudio", ConnCtx: 202084 {LDBID: 2, LCID: 202084}, StmtCtx: (1) 867947425905061 {Parent: 202084, MemoryLimit: 0, ObjHdl: 139944950352896, User: "SYSTEM", Schema: "SYSTEM, SesCtx: {ObjHdl: 139007399698944, CID: 202084, LCID: 202084, User: "SYSTEM", Schema: "SYSTEM, Ver: 7, CtxID: 329726}}, type: "SqlExecutor", method: "", detail: "", command: "
  {
  SQL: SELECT TOP 1 t1.BACKUP_ID, ( SELECT sum(t2.BACKUP_SIZE) FROM public.M_BACKUP_CATALOG_FILES t2 WHERE t2.BACKUP_ID = t1.BACKUP_ID GROUP BY t2.BACKUP_ID ) as BBSIZE, t1.UTC_START_TIME, t1.UTC_END_TIME, t1.ENTRY_TYPE_NAME FROM public.M_BACKUP_CATALOG AS t1 WHERE t1.STATE_NAME = 'successful' AND ( t1.ENTRY_TYPE_NAME = 'complete data backup' OR t1.ENTRY_TYPE_NAME = 'data snapshot' ) ORDER BY t1.UTC_START_TIME DESC

  QUERY PLAN:
  |OPERATOR_NAME|OPERATOR_DETAILS|OPERATOR_PROPERTIES|EXECUTION_ENGINE|DATABASE_NAME|SCHEMA_NAME|TABLE_NAME|TABLE_TYPE|TABLE_SIZE|OUTPUT_SIZE|SUBTREE_SIZE|OPERATOR_ID|PARENT_OPERATOR_ID|LEVEL|POSITION|HOST|PORT|BUFFER_ID|
  |ROW SEARCH|M_BACKUP_CATALOG.BACKUP_ID, SUM(SUM(M_BACKUP_CATALOG_FILES.BACKUP_SIZE)), M_BACKUP_CATALOG.UTC_START_TIME, M_BACKUP_CATALOG.UTC_END_TIME, SUBSTR(T0.ENTRY_TYPE_NAME, 0, 64)|
  ....

  
  RELATED OBJECTS:
  -TABLE:SYS.P_PRIVILEGES_(aaa-bbb-hdb1:32003),SYS.DUMMY(aaa-bbb-hdb1:32003)
  -VIEW:SYS.HAS_NEEDED_SYSTEM_PRIV_INCL_SYS_STAT,SYS.M_BACKUP_CATALOG,SYS.M_BACKUP_CATALOG_FILES
  -SYNONYM:M_BACKUP_CATALOG,M_BACKUP_CATALOG_FILES
  -USER:SYSTEM

  ...
  }..

  " at 0x00007f5cae0ca7f9 in syscall+0x19 (libc.so.6)

• Large number of entries in backup catalog (HANA_BACKUP_Catalog_Size.txt shows CATALOG_SIZE_MB over 1000 (1Gb)) 
• Very regular pattern showing cpu spikes e.g. every hour
• OOM errors showing large allocation e.g : 
  

  Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by exclusive_size_in_use.
  1: Monitoring & Statistical Data: Pool/RowEngine/MonitorView/StatisticsMonitors/M_DEV_BACKUP_CATALOG_LOG 51.41gb (55202974208b)
  2: Monitoring & Statistical Data: Pool/RowEngine/MonitorView/StatisticsMonitors/M_DEV_BACKUP_CATALOG_LOG_FILES 29.66gb (31851651328b)

  
  
  

• SAP HANA PLATFORM Edition 1.0
• SAP HANA PLATFORM Edition 2.0

9aaf8105bdad5308ab706eea28078d28, nameserver, oom, backup catalog, housekeeping, high CPU, performance , KBA , HAN-DB-BAC , SAP HANA Backup & Recovery , HAN-DB-PERF , SAP HANA Database Performance , Problem