3582397 - HANA Tenant suffers PROCESS_ALLOCATION_LIMIT dumps in Multi Tenant MDC Environment

• A HANA Tenant encounters a "PROCESS_ALLOCATION_LIMIT" OOM on the indexserver service. Although the total "used" memory of the instance has not reached anywhere near the defined GLOBAL_ALLOCATION_LIMIT (GAL) : 
  
  

  [752840]{-1}[-1/-1] 2025-02-11 07:03:49.104914 e Memory           mmReportMemoryProblems.cpp(02082) : OUT OF MEMORY occurred.
  MemoryType: NearDRAM
  Host: <>
  Executable: hdbindexserver
  PID: 745217
  Failed to allocate 256kb (262144b).
  Allocation failure type: PROCESS_ALLOCATION_LIMIT
  Current callstack:
  
  Memory consumption information of last failing ProvideMemory, PM-INX=177221:
  IPMM NearDRAM short info:
  GLOBAL_ALLOCATION_LIMIT (GAL) = 728.35gb (782065299456b), SHARED_MEMORY = 11.71gb (12575715328b), CODE_SIZE = 3.07gb (3300929536b), NVM_ALLOCATED_SIZE = 0b, OOM_RESERVATION_ALLOCATOR_SIZE = 96.14mb (100810752b)
  PID=745217 (hdbindexserver), PAL=199.80gb (214534455296b), AB=195.88gb (210333065216b), UA=0, U=194.55gb (208906964893b), FRO=0b, FSL=0b, FMC=0b
  [Some processes may have been skipped]
  Total allocated memory= 627.86gb (674166763520b)
  Total used memory     = 439.53gb (471946724706b)
  Sum AB                = 658290118656
  Sum Used              = 456070079842
  Heap memory fragmentation: 29% (this value may be high if defragmentation does not help solving the current memory request)
  Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by exclusive_size_in_use.
   1: Column Store Tables:                        Pool/ColumnStore/Main/Dictionary/RoDict                                  146.20gb (156989510500b)
   2: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultPageProvider:DataPage     10.44gb (11218614504b) (128mb (134217728b) precharged, 16.25mb (17039360b) in preparation)
   3: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultLPA/ShadowPage            8.67gb (9311371424b) (64mb (67108864b) precharged, 16mb (16777216b) in preparation)
   4: Column Store Tables:                        Pool/ColumnStore/Main/Uncompressed                                       5.36gb (5758254408b)

   
• Furthermore, the 'PROCESS_ALLOCATION_LIMIT' violation reveals the table(s) failing to allocate enough memory for delta merge operations :
  

  [1099870]{-1}[-1/9223372036854775806] 2025-02-19 23:16:47.032069 e delta_merge      SaveMergedAttributeJob.cpp(00102) : Out of memory in save merged column job: table = SID::SCHEMA:BALDAT (t 2853), column id = 208, column name = CLUSTD, rc = 2489, exception = exception  1: no.1000002  (ltt/impl/memory.cpp:189) TID: 1099870
  Allocation failed ; $failure_type$=PROCESS_ALLOCATION_LIMIT; $failure_flag$=; $size$=16777216; $name$=DefaultPageProvider:DataPage; $type$=pool; $inuse_count$=20440; $allocated_size$=12176546824
  
  
   . . . . . 
  
  [2447448]{-1}[-1/9223372036854775806] 2025-02-20 02:26:37.550988 e delta_merge      CsTableMerge.cpp(08231) : do rollbackDeltaMerge merge for idx SID::SCHEMA:BALDAT (t 2853)
  [2447448]{-1}[-1/-1] 2025-02-20 02:26:37.810962 e delta_merge      TRexApiAdminAccessorImpl.cpp(01253) : Merge during recovery failed: table = SCHEMA:BALDAT, container id = 0xfe00018ad2, reason = Out of memory during delta merge; table = SID::SCHEMA:BALDAT (t 2853), exception = exception 1000002: Allocation failed ; 
  $failure_type$=PROCESS_ALLOCATION_LIMIT; $failure_flag$=; $size$=16777216; $name$=DefaultPageProvider:DataPage; $type$=pool; $inuse_count$=20559; $allocated_size$=12068901008(2489)

  
  
  
• The "\SYS\global\hdb\custom\config\DB_" configuration directory reveals the Tenant has a specific allocation limit imposed on top of the already defined GLOBAL_ALLOCATION_LIMIT (GAL).
  The problem PID=745217 isn't consuming anywhere near the GLOBAL_ALLOCATION_LIMIT of 728 GB, but it has exhausted the allocated and used memory of it's own 'allocationlimit' of 204 GB (204596 mbs) :
   
  

  \SYS\global\hdb\custom\config\DB_XXX
  global.ini
  [memorymanager]
  allocationlimit = 204596

• The Nameserver trace may also encounter an OOM at the same time,  with additional details on the memory allocation and consumption of all other services and existing Tenant's under the instance SYSTEMDB, including PID 745217 :
  
  

  GLOBAL_ALLOCATION_LIMIT (GAL) = 728.35gb (782065299456b), SHARED_MEMORY = 11.71gb (12575715328b), CODE_SIZE = 3.07gb (3300929536b), NVM_ALLOCATED_SIZE = 0b, OOM_RESERVATION_ALLOCATOR_SIZE = 96.14mb (100810752b)
  
  PID=744831 (hdbnameserver), PAL=728.47gb (782199517184b), AB=18.06gb (19398242304b), UA=0, U=6.45gb (6931606297b), FRO=0b, FSL=0b, FMC=0b, Blocked=25gb (26843545600b)
  PID=745146 (hdbcompileserver), PAL=728.47gb (782199517184b), AB=9.08gb (9761185792b), UA=0, U=147.26mb (154418481b), FRO=0b, FSL=0b, FMC=0b, Blocked=777.78mb (815570944b)
  PID=745149 (hdbpreprocessor), PAL=728.47gb (782199517184b), AB=9.10gb (9778393088b), UA=0, U=160.09mb (167872659b), FRO=0b, FSL=0b, FMC=0b
  PID=745222 (hdbindexserver), PAL=149.85gb (160900841472b), AB=67.11gb (72064978944b), UA=0, U=43gb (46171162259b), FRO=0b, FSL=1.12gb (1207959552b), FMC=0b
  PID=745235 (hdbindexserver), PAL=399.60gb (429067862016b), AB=249.28gb (267671859200b), UA=0, U=166.32gb (178592715150b), FRO=0b, FSL=2gb (2147483648b), FMC=0b
  PID=745230 (hdbindexserver), PAL=99.90gb (107267227648b), AB=17.93gb (19260018688b), UA=0, U=13gb (13961819629b), FRO=0b, FSL=0b, FMC=0b
  PID=745217 (hdbindexserver), PAL=199.80gb (214534455296b), AB=195.88gb (210333065216b), UA=0, U=194.58gb (208931664642b), FRO=0b, FSL=0b, FMC=0b
  PID=750406 (hdbwebdispatcher), PAL=728.47gb (782199517184b), AB=9.23gb (9913462784b), UA=0, U=404.42mb (424071427b), FRO=0b, FSL=0b, FMC=0b
  PID=750403 (hdbdiserver), PAL=149.85gb (160900841472b), AB=3.55gb (3816882176b), UA=0, U=132.09mb (138516682b), FRO=0b, FSL=0b, FMC=0b
  PID=750400 (hdbdiserver), PAL=199.80gb (214534455296b), AB=5.26gb (5654491136b), UA=0, U=148.53mb (155753122b), FRO=0b, FSL=0b, FMC=0b

SAP HANA Platform Edition 2.0

KBA , HAN-DB , SAP HANA Database , Problem