3352604 - e OSMemory         mmMallocProxy.cpp(xxxxx) : Failed: munmap MM

• HANA Database is unresponsive
• Indexserver traces show numerous memory map errors, ResourceContainerShrink warnings and massive fragmentation of available memory :

[26452]{-1}[-1/-1] 2023-07-05 20:24:02.920349 e OSMemory         mmMallocProxy.cpp(02207) : Failed: munmap MM [0x00007ee252b5a000, 0x00007ee252b6a000[ (65536b | 0x0000000000000000), error 12
[26452]{-1}[-1/-1] 2023-07-05 20:24:02.920593 e OSMemory         mmMallocProxy.cpp(02207) : Failed: munmap MM [0x00007ee252b5a000, 0x00007ee252b6a000[ (65536b | 0x0000000000000000), error 12
[26538]{-1}[-1/-1] 2023-07-05 20:24:04.001173 e OSMemory         mmMallocProxy.cpp(02207) : Failed: munmap MM [0x00007ee252b5a000, 0x00007ee252b6a000[ (65536b | 0x0000000000000000), error 12
[26538]{-1}[-1/-1] 2023-07-05 20:24:04.001464 e OSMemory         mmMallocProxy.cpp(02207) : Failed: munmap MM [0x00007ee252b5a000, 0x00007ee252b6a000[ (65536b | 0x0000000000000000), error 1 

......

[26452]{-1}[-1/-1] 2023-07-05 20:23:21.322703 w ResMan           ResourceContainerShrink.cpp(02478) : Information about shrink at 2023-07-05 20:23:21 317 Local:
Reason for shrink: IPMM (other process). Request ID= 64225
ShrinkCaller
 1: 0x00007f253a7a3a86 in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0xb2 at ResourceContainerShrink.cpp:725 (libhdbbasis.so)
 2: 0x00007f253a79d164 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x640 at ResourceContainerShrink.cpp:2477 (libhdbbasis.so)
 3: 0x00007f253a8aff26 in MemoryManager::CompactionThread::compactCompactors(unsigned long, bool&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x582 at mmCompactionTrigger.cpp:955 (libhdbbasis.so)
 4: 0x00007f253a8ab7a6 in MemoryManager::CompactionThread::doCompactionIntern(MemoryManager::Precharge&, unsigned long, unsigned long, unsigned long, unsigned long&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType, bool)+0xfc2 at mmCompactionTrigger.cpp:1116 (libhdbbasis.so)
 5: 0x00007f253a8ac7b5 in MemoryManager::CompactionThread::doIpmmCompactionIntern(unsigned long, int, unsigned long, unsigned long, unsigned long, MemoryManager::ProcessSlot::RequestedTask)+0x1b1 at mmCompactionTrigger.cpp:1704 (libhdbbasis.so)
 6: 0x00007f253a8ad1ec in MemoryManager::CompactionThread::run(void*&)+0x188 at mmCompactionTrigger.cpp:1659 (libhdbbasis.so)
 7: 0x00007f253aa0d3fd in Execution::Thread::staticMainImp(Execution::Thread*)+0x529 at Thread.cpp:522 (libhdbbasis.so)
 8: 0x00007f253a9feb39 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x165 at ThreadInterposition.cpp:639 (libhdbbasis.so)
 9: 0x00007f253a07e4f9 in start_thread+0xd5 (libpthread.so.0)
10: 0x00007f2538758ecf in __GI___clone+0x3b (libc.so.6)
IPMM short info:
GLOBAL_ALLOCATION_LIMIT (GAL) = 281.87gb (302660976640b), SHARED_MEMORY = 3.71gb (3993792512b), CODE_SIZE = 3.29gb (3539582976b), OOM_RESERVATION_ALLOCATOR_SIZE = 96.14mb (100810752b)
PID=26383 (hdbindexserver), PAL=300.42gb (322578493440b), AB=265.42gb (285003878400b), UA=0, U=52.64gb (56527229591b), FSL=0b, FMC=0b
[Some processes may have been skipped]
Total allocated memory= 281.99gb (302786670592b)
Total used memory     = 67.33gb (72297484846b)
Sum AB                = 295253295104
Sum Used              = 64764109358
Heap memory fragmentation: 76% (this value may be high if defragmentation does not help solving the current memory request)
Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by inclusive_size_in_use.
 1: System:                        /                                                                    52.64gb (56527229591b)
 2: System:                        Pool                                                                 49.63gb (53292362661b)
 3: System:                        Pool/PersistenceManager                                              20.12gb (21609728720b) 

• 'oom_memory_release.trc' files are observed 'indexserver_host.30003.rtedump.20230705-162124.0026383.oom_memory_release.trc'

[MEMORY_OOM]  Information about current out of memory situation: (2023-07-05 16:21:54 432 Local)
OUT OF MEMORY occurred.
Host: <>
Executable: hdbindexserver
PID: 26383

Current callstack:
 1: 0x00007f253a8ed185 in MemoryManager::AllocatorConfigurationSPI::releaseMemoryMostProbablyDueToAWrongConfiguration()+0x21 at mmAllocatorConfigurationImpl.cpp:1675 (libhdbbasis.so)
 2: 0x00007f253a890944 in MemoryManager::HugeAlignmentPool::allocateHugeAlignedNoThrowImpl(MemoryManager::MemoryPool**, ltt::allocator_statistics&, unsigned long, unsigned long, void const*)+0xae0 at mmHugeAlignmentAllocator.cpp:1327 (libhdbbasis.so)
 3: 0x00007f253a85412b in MemoryManager::MemoryPool::allocateAligned(unsigned long, unsigned long, unsigned short, ltt::allocator_statistics&, bool&, bool, void const*)+0x1c7 at mmMemoryPool.cpp:1722 (libhdbbasis.so)
 4: 0x00007f253a8262f6 in MemoryManager::PoolAllocator::allocateAlignedNoThrowImpl(unsigned long, unsigned long, void const*)+0x52 at mmPoolAllocator.cpp:1811 (libhdbbasis.so)
 5: 0x00007f2547cfd89d in RowEngine::Version::PagedAllocatorPartition::allocateAlignedNoThrow(unsigned long, unsigned long, bool)+0x1e9 at memory.cpp:336 (libhdbrskernel.so)
 6: 0x00007f2547cff727 in RowEngine::Version::PartitionedMVCCAllocator::allocateAlignedNoThrowImpl(unsigned long, unsigned long, void const*)+0x63 at MVCCAllocatorImpl.cc:158 (libhdbrskernel.so)
 7: 0x00007f254713e5d8 in ltt::allocator::allocate(unsigned long, void const*)+0x24 at memory.cpp:236 (libhdbrskernel.so)
 8: 0x00007f254713fe77 in ltt::handle_mem_ref::create(unsigned long, void const*, ltt::allocator&) const+0x43 at handle_ref.cpp:18 (libhdbrskernel.so)
 9: 0x00007f2547d07237 in RowEngine::Version::BasicMVCCManager::acquireVersionContext(ptime::Transaction*)+0x63 at MVCCManagerImpl.cc:410 (libhdbrskernel.so)
10: 0x00007f2547e42142 in RowEngine::RSContext::getVersionContextHandle()+0x60 at RSTransactionDataContainer.cc:472 (libhdbrskernel.so)
11: 0x00007f25480fe4b3 in ptime::VersionMgr::createAndRegisterVersionContext(ptime::Transaction*)+0x150 at VersionMgr.cc:1493 (libhdbrskernel.so)
12: 0x00007f2548007025 in ptime::Transaction::start_logging(long)+0xa61 at transmgmt.cc:1446 (libhdbrskernel.so)
13: 0x00007f2548007578 in ptime::Transaction::set_write_trans(bool, long)+0xa4 at transmgmt.cc:1099 (libhdbrskernel.so)
14: 0x00007f25480095d0 in ptime::Transaction::start(ptime::Transaction::ReadUpdateTrans, unsigned long, bool, bool, unsigned long)+0x840 at transmgmt.cc:1020 (libhdbrskernel.so)
15: 0x00007f2552294a79 in TableConsistencyCheck::ColumnTablePartJobNode::runCheck(Execution::Context&, Execution::JobObject&)+0xe5 at Request.cc:502 (libhdbtableconsistencycheck.so)
16: 0x00007f255229687b in TableConsistencyCheck::LocalTableJobNode::run(Execution::Context&, Execution::JobObject&)+0x47 at Request.cc:192 (libhdbtableconsistencycheck.so)
17: 0x00007f253aa6a74c in Execution::JobObjectImpl::run(Execution::JobWorker*)+0x2028 at JobExecutorImpl.cpp:1195 (libhdbbasis.so)
18: 0x00007f253aa54122 in Execution::JobWorker::runJob(ltt::smartptr_handle<Execution::JobObjectForHandle>&)+0x320 at JobExecutorThreads.cpp:383 (libhdbbasis.so)
19: 0x00007f253aa4e5c8 in Execution::JobWorker::run(void*&)+0x764 at JobExecutorThreads.cpp:1283 (libhdbbasis.so)
20: 0x00007f253aa0d3fd in Execution::Thread::staticMainImp(Execution::Thread*)+0x529 at Thread.cpp:522 (libhdbbasis.so)
21: 0x00007f253a9feb39 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x165 at ThreadInterposition.cpp:639 (libhdbbasis.so)
22: 0x00007f253a07e4f9 in start_thread+0xd5 (libpthread.so.0)
23: 0x00007f2538758ecf in __GI___clone+0x3b (libc.so.6)

Memory consumption information:
[No information available.]


Out of memory reasons:
!:! <1> 2x BigBlockAllocator failure, --- precharge ok ---.
    /data/jenkins_prod/workspace/awduyghiom/s/Basis/MemoryManager/impl/mmMemoryPool.cpp (2276)


Could not return 112038625280b to operating system.
This is a configuration problem of your operating system:
Please increase /proc/sys/vm/max_map_count
Memory Configuration:
global_allocation_limit: 288768 (-> 282gb (302795194368b))
allocationlimit:  (-> 300.42gb (322578493440b))

• 'OPERATING_SYSTEM_LIMIT' OOM files are present 'indexserver_hostname.30003.rtedump.20230705-192033.0026383.oom.trc'

[MEMORY_OOM]  Information about current out of memory situation: (2023-07-05 19:20:34 613 Local)
OUT OF MEMORY occurred.
Host: <>
Executable: hdbindexserver
PID: 26383
Failed to allocate 1.12kb (1152b).

Allocation failure type: OPERATING_SYSTEM_LIMIT

Current callstack:
 1: 0x00007f253a824d35 in MemoryManager::PoolAllocator::notifyOOMImpl(unsigned long, unsigned long, bool, ltt::allocation_failure_type, bool)+0x1b1 at mmPoolAllocator.cpp:2284 (libhdbbasis.so)
 2: 0x00007f253a6e17ad in ltt::allocator::notifyOOM(unsigned long, unsigned long, bool, ltt::allocation_failure_type, bool) [clone .constprop.0]+0x59 at memory.cpp:130 (libhdbbasis.so)
 3: 0x00007f253a825bd0 in MemoryManager::PoolAllocator::oomHandling(unsigned long, unsigned long, bool)+0x60 at mmPoolAllocator.cpp:1222 (libhdbbasis.so)
 4: 0x00007f253a826420 in MemoryManager::PoolAllocator::allocateNoThrowImpl(unsigned long, void const*)+0xd0 at mmPoolAllocator.cpp:1733 (libhdbbasis.so)
 5: 0x00007f253a716c28 in ltt::allocator::allocate(unsigned long, void const*)+0x24 at memory.cpp:236 (libhdbbasis.so)
 6: 0x00007f253a718957 in ltt::handle_mem_ref::create(unsigned long, void const*, ltt::allocator&) const+0x43 at handle_ref.cpp:18 (libhdbbasis.so)
 7: 0x00007f253a9fb497 in Execution::TimerThread::TimerCallback::processTime(Synchronization::LockHandle<Synchronization::Mutex, true>&, bool)+0xb3 at JobExecutor.hpp:1226 (libhdbbasis.so)
 8: 0x00007f253a9fbddd in Execution::TimerThreadSingleton::run(void*&)+0x129 at TimerThread.cpp:444 (libhdbbasis.so)
 9: 0x00007f253aa0d3fd in Execution::Thread::staticMainImp(Execution::Thread*)+0x529 at Thread.cpp:522 (libhdbbasis.so)
10: 0x00007f253a9feb39 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x165 at ThreadInterposition.cpp:639 (libhdbbasis.so)
11: 0x00007f253a07e4f9 in start_thread+0xd5 (libpthread.so.0)
12: 0x00007f2538758ecf in __GI___clone+0x3b (libc.so.6)

• The '/proc/self/maps' section reveals that the operating system layer configurations for Linux THP is enabled and that the virtual map memory limit is incredibly low  :

/sys/kernel/mm/transparent_hugepage/enabled:
[always] madvise never

/proc/sys/vm/max_map_count:
65530

SAP HANA Platform Edition 2.0

KBA , HAN-DB , SAP HANA Database , BC-OP-LNX , Linux , Problem