3011480 - How-To : Troubleshooting SAP HANA Memory Consumption

You are interested in analyzing and finding the root cause for current & historic memory consumption on a SAP HANA System or Tenant DB. Symptoms below may also be present on a trace level :

• SAP HANA trace and SAP application client log files contain entries like those described in section '1. Which indications exist for SAP HANA memory problems?' from KBA 1999997 - FAQ: SAP HANA Memory

• HANA service trace files report resource container shrinks 'indexserver_hostname.30003.014.trc' 

[39876]{-1}[-1/-1] 2022-06-25 15:15:01.760988 w ResMan           ResourceContainerShrink.cpp(02478) : Information about shrink at 2022-06-25 15:15:01 756 Local:
Reason for shrink: IPMM (other process). Request ID= 7459557
ShrinkCaller
 1: 0x00007fc57192b196 in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0xb2 at ResourceContainerShrink.cpp:725 (libhdbbasis.so)
 2: 0x00007fc571924804 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x640 at ResourceContainerShrink.cpp:2477 (libhdbbasis.so)
 3: 0x00007fc571a3ee16 in MemoryManager::CompactionThread::compactCompactors(unsigned long, bool&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x582 at mmCompactionTrigger.cpp:955 (libhdbbasis.so)
 4: 0x00007fc571a3d9f6 in MemoryManager::CompactionThread::doCompactionIntern(MemoryManager::Precharge&, unsigned long, unsigned long, unsigned long, unsigned long&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType, bool)+0xfc2 at mmCompactionTrigger.cpp:1116 (libhdbbasis.so)
 5: 0x00007fc571a371c5 in MemoryManager::CompactionThread::doIpmmCompactionIntern(unsigned long, int, unsigned long, unsigned long, unsigned long, MemoryManager::ProcessSlot::RequestedTask)+0x1b1 at mmCompactionTrigger.cpp:1704 (libhdbbasis.so)
 6: 0x00007fc571a3762c in MemoryManager::CompactionThread::run(void*&)+0x188 at mmCompactionTrigger.cpp:1659 (libhdbbasis.so)
 7: 0x00007fc571ba6f85 in Execution::Thread::staticMainImp(Execution::Thread*)+0x531 at Thread.cpp:522 (libhdbbasis.so)
 8: 0x00007fc571b969e8 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x1e4 at ThreadInterposition.cpp:639 (libhdbbasis.so)
 9: 0x00007fc5711f96da in start_thread+0xc6 (libpthread.so.0)
10: 0x00007fc56f9052dd in __clone+0x69 (libc.so.6)
IPMM short info:
GLOBAL_ALLOCATION_LIMIT (GAL) = 3.40tb (3748391813120b), SHARED_MEMORY = 52.74gb (56631324672b), CODE_SIZE = 3.03gb (3255672832b), OOM_RESERVATION_ALLOCATOR_SIZE = 96.14mb (100810752b)
PID=39810 (hdbindexserver), PAL=5.72tb (6292106772480b), AB=3.33tb (3672815403008b), UA=0, U=3.17tb (3494556729214b), FSL=0b, FMC=0b
[Some processes may have been skipped]
Total allocated memory= 3.40tb (3748410744832b)
Total used memory     = 3.24tb (3566143311680b)
Sum AB                = 3688523747328
Sum Used              = 3506256314176
Heap memory fragmentation: 4
Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by inclusive_size_in_use.
 1: System:              /                                                                    3.17tb (3494556620414b)
 2: System:              Pool                                                                 3.11tb (3429630169017b)
 3: System:              Pool/ColumnStore                                                     2.71tb (2990869825029b)
 4: Column Store Tables: Pool/ColumnStore/Main                                                2.69tb (2968513658285b)
 5: Column Store Tables: Pool/ColumnStore/Main/Dictionary                                     1.16tb (1284930983089b)
 6: Column Store Tables: Pool/ColumnStore/Main/Dictionary/RoDict                              1.13tb (1248465481529b)
 7: Column Store Tables: Pool/ColumnStore/Main/Compressed                                     544.96gb (585154575168b)
 8: Column Store Tables: Pool/ColumnStore/Main/Index                                          529.71gb (568775967044b)
 9: Column Store Tables: Pool/ColumnStore/Main/Compressed/Indirect                            395.91gb (425114807104b)
10: Column Store Tables: Pool/ColumnStore/Main/Index/PageableSingle                           278.25gb (298771486148b)
11: Column Store Tables: Pool/ColumnStore/Main/Uncompressed                                   272.28gb (292364867344b)
12: Column Store Tables: Pool/ColumnStore/Main/Index/Single                                   244.06gb (262064112824b)
13: System:              Pool/PersistenceManager                                              225.02gb (241622485937b)
14: Column Store Tables: Pool/ColumnStore/Main/Rowid                                          220.98gb (237282275312b)
15: System:              Pool/PersistenceManager/PersistentSpace                              152.87gb (164143502128b)
16: System:              Pool/PersistenceManager/PersistentSpace/DefaultLPA                   144.03gb (154660909248b)
17: Column Store Tables: Pool/ColumnStore/Main/Compressed/Sparse                              92.91gb (99772760952b)
18: System:              Pool/PersistenceManager/PersistentSpace/DefaultLPA/LOBPage           88.93gb (95491743744b)
19: System:              AllocateOnlyAllocator-unlimited                                      59.21gb (63581997256b)
20: System:              AllocateOnlyAllocator-unlimited/FLA-UL<24592,1>/MemoryMapLevel3Nodes 54.24gb (58241061456b)
Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by exclusive_size_in_use.
 1: Column Store Tables:                        Pool/ColumnStore/Main/Dictionary/RoDict                                1.13tb (1248599699257b) (128mb (134217728b) precharged, 1024kb (1048576b) in preparation)
 2: Column Store Tables:                        Pool/ColumnStore/Main/Compressed/Indirect                              395.91gb (425114807104b)
 3: Column Store Tables:                        Pool/ColumnStore/Main/Index/PageableSingle                             278.25gb (298771486148b)
 4: Column Store Tables:                        Pool/ColumnStore/Main/Uncompressed                                     272.28gb (292364867344b)
 5: Column Store Tables:                        Pool/ColumnStore/Main/Index/Single                                     244.06gb (262064112824b)
 6: Column Store Tables:                        Pool/ColumnStore/Main/Rowid                                            220.98gb (237282275312b)
 7: Column Store Tables:                        Pool/ColumnStore/Main/Compressed/Sparse                                92.91gb (99772760952b)
 8: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultLPA/LOBPage             88.93gb (95491743744b)
 9: System:                                     AllocateOnlyAllocator-unlimited/FLA-UL<24592,1>/MemoryMapLevel3Nodes   54.24gb (58241061456b)
10: Monitoring & Statistical Data:              Pool/Statistics                                                        48.99gb (52611291536b)
11: Column Store Tables:                        Pool/ColumnStore/Main/Compressed/Rle                                   48.55gb (52136899536b)
12: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultLPA/DataPage            36.04gb (38705065984b)
13: Column Store Tables:                        Pool/ColumnStore/Main/Dictionary/ValueDict                             33.96gb (36465500728b)
14: Column Store Tables:                        Pool/PersistenceManager/UnifiedTableContainer                          26.90gb (28887251792b)
15: Statement Execution & Intermediate Results: Pool/JoinEvaluator/TranslationTable                                    22.51gb (24181167860b)
16: System:                                     Pool/RowEngine/QueryExecution/SearchAlloc                              22.45gb (24117130320b)
17: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultLPA/LOBPageControlblock 18.35gb (19704599040b)
18: Column Store Tables:                        Pool/ColumnStore/Delta/Btreeindex                                      13.41gb (14399747128b)
19: System:                                     Pool/RowEngine/Session                                                 12.59gb (13519431847b) (3.37gb (3623878656b) precharged, 25.29mb (26521600b) in preparation)
20: System:                                     Pool/ResourceContainer                                                 10.65gb (11443567032b)

• Dumps may be observed in the client's ABAP system in ST22 similar to the following (3151474 - DBIF_REPO_SQL_ERROR)

     Category Resource bottleneck
     Runtime Errors DBIF_REPO_SQL_ERROR
     Date and Time 27.07.2020 13:25:56 
     
     Short Text
     SQL error 4 while accessing program "LSTXBF99" part "SRC".
     
     What happened?
     Database error text: "cannot allocate enough memory: Out of memory on query fetch"

     . . . . .

• SAP HANA Platform Edition 1.0
• SAP HANA Platform Edition 2.0

high memory consumption, High memory, High Memory Usage, running out of memory, AttributeEngine: not enough memory OOM, out of memory, memory usage, out of memory, 2048, DBSQL_SQL_ERROR, memory allocation failed, 1000002: Allocation failed, Allocation failed, 2048: column store error: search table error, Cannot allocate memory, 2048 column store error, compositelimit_oom, HANA Memory full, HANA Memory Growth, HANA memory increase, HANA memory increased, hana memory issue, HANA Memory Issues, HANA memory license, HANA memory usage, hana memory usage very high, memory 100%, Memory allocation dumps, Memory Allocation Error, Memory allocation failed, Memory allocation failed;exception 1000002, Memory allocation failure, Memory allocation issue, Memory consumption is high, Memory consumption spike, Memory dumps, Memory issue, Memory issues, Memory optimization, Memory overflow, Memory problems, Memory shortage, Memory size in total, memory unloads, Memory usage, Memory utilization, OOM dump, OOM Dumps, oom for hana, OOM HANA issue, OOM Issue, OOM situation, OOM Trace, Out of memory, Out Of Memory Dump, Out of memory Dumps, Out of Memory error, Out Of Memory Issue, Out of memory issues, resident memory, High DB consumption, High Memory allocation, High memory utilization, High Peak memory, statement_memory_limit , KBA , HAN-DB-ENG , SAP HANA DB Engines , Problem