3363347 - PROCESS_ALLOCATION_LIMIT oom violations on HANA

• The <indexserver_host.port.000.trc> trace files contains out of memory events of type "PROCESS_ALLOCATION_LIMIT" and also high fragmentation where the used memory is still well below the allocated memory limits :

-------------------------------------
[42018]\{336388\}[44/-1] 2023-08-03 07:03:40.028290 e Memory           mmReportMemoryProblems.cpp(01848) : OUT OF MEMORY occurred.
Host: hostname
Executable: hdbindexserver
PID: 14937
Failed to allocate 36.75gb (39461586112b).
Allocation failure type: PROCESS_ALLOCATION_LIMIT
Current callstack:
 1: 0x00007f43256e2e05 in MemoryManager::PoolAllocator::notifyOOMImpl(unsigned long, unsigned long, bool, ltt::allocation_failure_type, bool)+0x1b1 at mmPoolAllocator.cpp:2338 (libhdbbasis.so)
 2: 0x00007f432558c4ed in ltt::allocator::notifyOOM(unsigned long, unsigned long, bool, ltt::allocation_failure_type, bool) [clone .constprop.0]+0x59 at memory.cpp:127 (libhdbbasis.so)
 3: 0x00007f43256e9b10 in MemoryManager::PoolAllocator::oomHandling(unsigned long, unsigned long, bool)+0x60 at mmPoolAllocator.cpp:1248 (libhdbbasis.so)
 4: 0x00007f43256e3273 in MemoryManager::PoolAllocator::allocateAlignedNoThrowImpl(unsigned long, unsigned long, void const*)+0xa0 at mmPoolAllocator.cpp:1871 (libhdbbasis.so)
 5: 0x00007f4340d34e99 in ltt::allocator::allocateAligned(unsigned long, unsigned long)+0x25 at memory.cpp:298 (libhdbcs.so)
 6: 0x00007f4340fadeb1 in ltt::impl::VectorBase<TrexTypes::StringAttributeValue>::init_(unsigned long) [clone .part.0] [clone .lto_priv.0] [clone .lto_priv.0]+0x50 at memory.hpp:1212 (libhdbcs.so)
 7: 0x00007f43413f448d in AttributeEngine::ValueArray::init(unsigned long, bool)+0x1019 at string.hpp:484 (libhdbcs.so)
 8: 0x00007f43374caa85 in JoinEvaluator::TableAccess::jeGetValues(TRexConfig::IndexHandle&, TrexBase::IndexName const&, TRexConfig::Table const&, ltt_adp::vector<unsigned int, ltt::integral_constant<bool, true> > const&, ltt_adp::vector<JoinEvaluator::ValueMode, ltt::integral_constant<bool, true> > const*, ltt_adp::vector<TRexCommonObjects::QueryEntry, ltt::integral_constant<bool, true> > const*, unsigned int, unsigned int, unsigned int, ltt::map<unsigned int, unsigned int, ltt::less<unsigned int>, ltt::rb_tree_balancier> const*, TRexCommonObjects::LeanMainDelta<int> const&, TRexUtils::IndexVectorAligned const*, TRexUtils::BitVector const*, TRexCommonObjects::LeanMainDelta<TrexStore::UdivList*> const*, ltt_adp::vector<TRexCommonObjects::ColumnBase*, ltt::integral_constant<bool, true> >&, ltt_adp::vector<AttributeEngine::ValueArray*, ltt::integral_constant<bool, true> >&, int&, TRexConfig::IndexSyncPoint const*, ltt::allocator&)+0x251 at TableAccess.cpp:1546 (libhdbcsapi.so)
 9: 0x00007f433768ccf4 in JoinEvaluator::JERequestedAttributes::executePopConst(ltt_adp::vector<Executor::PlanData*, ltt::integral_constant<bool, true> > const&, ltt_adp::vector<Executor::PlanData*, ltt::integral_constant<bool, true> > const&, Executor::ExecutionInfo const&) const+0x1f30 at JERequestedAttributes.cpp:977 (libhdbcsapi.so)
10: 0x00007f43371f5fc1 in JoinEvaluator::JEPlanOperation::executePop(ltt_adp::vector<Executor::PlanData*, ltt::integral_constant<bool, true> > const&, ltt_adp::vector<Executor::PlanData*, ltt::integral_constant<bool, true> > const&, TRexCommonObjects::TRexApiError&, Executor::ExecutionInfo const&)+0x1d0 at JEPlanOperation.cpp:304 (libhdbcsapi.so)
11: 0x00007f434ffc4df3 in Executor::X2::runPopTask(Executor::X2::PopTaskInfo&, int&, ltt::allocator&, ltt::allocator&)+0x26f0 at X2.cpp:2528 (libhdbexecutor.so)
12: 0x00007f434ffc5384 in Executor::X2::runPopJob(Executor::X2Job*)+0x60 at X2.cpp:2230 (libhdbexecutor.so)
13: 0x00007f434ffb090f in Executor::X2Job::run(Execution::Context&, Execution::JobObject&)+0x38b at X2.cpp:5287 (libhdbexecutor.so)
14: 0x00007f432596a16f in Execution::JobObjectImpl::run(Execution::JobWorker*)+0xfab at JobExecutorImpl.cpp:1184 (libhdbbasis.so)
15: 0x00007f432595657a in Execution::JobWorker::runJob(ltt::smartptr_handle<Execution::JobObjectForHandle>&)+0x336 at JobExecutorThreads.cpp:375 (libhdbbasis.so)
16: 0x00007f43259496b8 in Execution::JobWorker::run(void*&)+0x764 at JobExecutorThreads.cpp:1280 (libhdbbasis.so)
17: 0x00007f4325906995 in Execution::Thread::staticMainImp(Execution::Thread*)+0x531 at Thread.cpp:523 (libhdbbasis.so)
18: 0x00007f43258f6ce8 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x1e4 at ThreadInterposition.cpp:650 (libhdbbasis.so)
19: 0x00007f4324f0271a in start_thread+0xc6 (libpthread.so.0)
20: 0x00007f43235af17d in __clone+0x69 (libc.so.6)
Memory consumption information of last failing ProvideMemory, PM-INX=71047:
IPMM short info:
GLOBAL_ALLOCATION_LIMIT (GAL) = 1.70tb (1878913974272b), SHARED_MEMORY = 3.55gb (3817435136b), CODE_SIZE = 2.90gb (3115945984b), OOM_RESERVATION_ALLOCATOR_SIZE = 96.14mb (100810752b)
PID=14937 (hdbindexserver), PAL=972.88gb (1044632203264b), AB=969.60gb (1041101901824b), UA=0, U=827.64gb (888679643458b), FRO=0b, FSL=0b, FMC=0b
[Some processes may have been skipped]
Total allocated memory= 1.25tb (1376919588864b)
Total used memory     = 846.30gb (908713346025b)
Sum AB                = 1369986207744
Sum Used              = 901779964905
Heap memory fragmentation: 34% (this value may be high if defragmentation does not help solving the current memory request)

Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by inclusive_size_in_use.
 1: System:                                     /                                                                    827.64gb (888679643458b)
 2: System:                                     Pool                                                                 808.26gb (867866329886b)
 3: Statement Execution & Intermediate Results: Pool/JoinEvaluator                                                   612.46gb (657628821476b)
 4: Statement Execution & Intermediate Results: Pool/JoinEvaluator/JERequestedAttributes                             605.39gb (650037182236b)
 5: Statement Execution & Intermediate Results: Pool/JoinEvaluator/JERequestedAttributes/Results                     605.39gb (650037148036b)
 6: System:                                     Pool/ResourceContainer                                               91.14gb (97869403720b)
 7: System:                                     Pool/ColumnStore                                                     60.29gb (64739596181b)
 8: Column Store Tables:                        Pool/ColumnStore/Main                                                59.90gb (64326913897b)
 9: System:                                     Pool/ResourceContainer/ResourceHeader                                48.49gb (52066081288b)
10: Column Store Tables:                        Pool/ColumnStore/Main/Dictionary                                     37.30gb (40058895989b)
11: Column Store Tables:                        Pool/ColumnStore/Main/Dictionary/RoDict                              24.91gb (26755927749b)
12: System:                                     AllocateOnlyAllocator-unlimited                                      18.71gb (20098296840b)
13: System:                                     AllocateOnlyAllocator-unlimited/FLA-UL<24592,1>/MemoryMapLevel3Nodes 16.48gb (17704592336b)
14: System:                                     AllocateOnlyAllocator-unlimited/FLA-UL<24592,1>                      16.48gb (17704592336b)
15: System:                                     Pool/PersistenceManager                                              15.78gb (16947194592b)
16: Column Store Tables:                        Pool/ColumnStore/Main/Uncompressed                                   13.04gb (14006159176b)
17: Column Store Tables:                        Pool/ColumnStore/Main/Dictionary/ValueDict                           12.38gb (13302967288b)
18: System:                                     Pool/PersistenceManager/PersistentSpace                              10.92gb (11735192920b)
19: Monitoring & Statistical Data:              Pool/Statistics                                                      9.80gb (10523578344b)
20: Column Store Tables:                        Pool/ColumnStore/Main/Rowid                                          8.26gb (8879943616b)

• 'HANA_Memory_Overview_.txt' (Note 1969700) reveals inconsistent configured memory limits on both a license and/or GAL level when compared with the total size of the server's physical memory on the operating system :

-------------------------------------

4 ;License memory limit            ;    1024;                      ;                                                                             

6 ;Physical memory (available)     ;    1008;  1008 (host);                                                                                                                                    

8 ;Physical memory (used)          ;     408;   408 (host);                                                                                                                                           

10;Global allocation limit         ;    1750;  1750 (host);                                                                             

-------------------------------------

SAP HANA, Platform Edition 2.0

SAP HANA, platform Edition 1.0

KBA , HAN-DB , SAP HANA Database , Problem