SAP Knowledge Base Article - Preview

3338534 - NVM_ALLOCATED_SIZE persistent memory consuming the GLOBAL_ALLOCATION_LIMIT on HANA

Symptom

  • Resource Container Shrinks and memory pressure are occurring, where persistent memory under the 'NVM_ALLOCATED_SIZE' space is the highest consumer :
[22209]\{358011\}[38/3761858043] 2023-05-22 05:38:32.140031 i TraceContext     TraceContext.cpp(01347) : UserName=SAPABAP1, ApplicationUserName=SAPSUPPORT1, ApplicationName=ABAP:SID, ApplicationSource=CL_FINS_SDM_ACDOCA_NETDT_AMDP=CP:39, Client=100, StatementHash=4c79d84fa817dd6feb18ae674e97517c, EppRootContextId=C15DB5FF84811EDDBE8FDC07A7646F09, EppTransactionId=3DF5405E6F9E0080E00646AA2D555375, EppConnectionId=646A4C25A2B656F2E10000000AA70612, EppConnectionCounter=1, EppComponentName=SID/host_SID_00, EppAction=R_SDM_ONLINE_EXEC, StatementExecutionID=844438590200945
[22209]\{358011\}[38/3761858043] 2023-05-22 05:38:32.140031 w ResMan           ResourceContainerShrink.cpp(03167) : Information about shrink at 2023-05-22 05:38:32 134 Local:
Reason for shrink: Reserve for HBA, user size:  893344248
ShrinkCaller
......

IPMM short info:
GLOBAL_ALLOCATION_LIMIT (GAL) = 972.74gb (1044479148032b), SHARED_MEMORY = 20.74gb (22276599808b), CODE_SIZE = 3.56gb (3830210560b), NVM_ALLOCATED_SIZE = 774.61gb (831733710848b), OOM_RESERVATION_ALLOCATOR_SIZE = 96.14mb (100810752b)
PID=12674 (hdbindexserver), PAL=972.87gb (1044613365760b), AB=162.73gb (174735241216b), UA=0, U=150.95gb (162085162541b), FSL=0b, FMC=0b
[Some processes may have been skipped]
Total allocated memory= 971.06gb (1042668449792b)
Total used memory     = 956.38gb (1026912509163b)
Sum AB                = 184827928576
Sum Used              = 169071987947
Heap memory fragmentation: 1
Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by inclusive_size_in_use.
 1: System:                                     /                                                                    153.80gb (165147820401b)
 2: System:                                     Pool                                                                 149.66gb (160705447952b)
 3: System:                                     Pool/PersistenceManager                                              70.84gb (76072626640b)
 4: System:                                     Pool/PersistenceManager/PersistentSpace                              46.82gb (50278516832b)
 5: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultPageProvider:DataPage 32gb (34367688704b)
 6: Statement Execution & Intermediate Results: Pool/JoinEvaluator                                                   29.01gb (31150471716b)
 7: System:                                     Pool/ColumnStore                                                     27.82gb (29872568965b)
 8: Statement Execution & Intermediate Results: Pool/JoinEvaluator/JEStep2                                           19.21gb (20635558840b)
 9: Column Store Tables:                        Pool/PersistenceManager/UnifiedTableContainer                        17.67gb (18981965152b)
10: Column Store Tables:                        Pool/ColumnStore/Main                                                13.12gb (14088848129b)
11: System:                                     Pool/PersistenceManager/PersistentSpace/LOBPage                      12.02gb (12910800896b)
12: Column Store Tables:                        Pool/ColumnStore/Main/Compressed                                     10.02gb (10760845944b)
13: Column Store Tables:                        Pool/ColumnStore/Main/Compressed/Indirect                            10.02gb (10760831544b)
14: Statement Execution & Intermediate Results: Pool/JoinEvaluator/TranslationTable                                  9.54gb (10254337208b)
15: Statement Execution & Intermediate Results: Pool/ColumnStore/Transient                                           7.28gb (7824327804b)
16: System:                                     Pool/RowEngine                                                       6.09gb (6545781754b)
17: Other:                                      Pool/ColumnStore/TableOptimization                                   4.10gb (4404060656b)
18: Column Store Tables:                        Pool/PersistenceManager/UnifiedTableContainer/MVCC                   3.60gb (3868247752b)
19: Monitoring & Statistical Data:              Pool/Statistics                                                      3.33gb (3576032528b)
20: System:                                     Pool/PersistenceManager/VarSizeEntryFreeSpaceInformation             3.06gb (3289791512b)
Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by exclusive_size_in_use.
 1: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultPageProvider:DataPage     32gb (34367688704b)
 2: Statement Execution & Intermediate Results: Pool/JoinEvaluator/JEStep2                                               20.04gb (21528904632b) (851.96mb (893345792b) precharged, 851.96mb (893345792b) in preparation)
 3: Column Store Tables:                        Pool/PersistenceManager/UnifiedTableContainer                            14.07gb (15113717400b)
 4: System:                                     Pool/PersistenceManager/PersistentSpace/LOBPage                          12.02gb (12910800896b)
 5: Column Store Tables:                        Pool/ColumnStore/Main/Compressed/Indirect                                10.02gb (10761523224b)
 6: Statement Execution & Intermediate Results: Pool/JoinEvaluator/TranslationTable                                      9.54gb (10254337208b)
 7: Statement Execution & Intermediate Results: Pool/ColumnStore/Transient                                               7.28gb (7824327804b)
 8: Other:                                      Pool/ColumnStore/TableOptimization                                       4.10gb (4404060656b)
 9: Column Store Tables:                        Pool/PersistenceManager/UnifiedTableContainer/MVCC                       3.60gb (3868247752b)
10: System:                                     Pool/PersistenceManager/VarSizeEntryFreeSpaceInformation                 3.06gb (3289791512b)
11: Row Store Tables:                           Pool/RowStoreTables/CpbTree                                              2.73gb (2932837120b)
12: Column Store Tables:                        Pool/ColumnStore/Main/Rowid                                              2.68gb (2879124952b)
13: System:                                     Pool/RowEngine/QueryExecution/SearchAlloc                                2.65gb (2846447064b)
14: Caches:                                     Pool/RowEngine/SQLPlan                                                   2.33gb (2502457714b)
15: Statement Execution & Intermediate Results: StackAllocator                                                           1.90gb (2043691008b)
16: System:                                     AllocateOnlyAllocator-unlimited/FLA-UL<24592,1>/MemoryMapLevel3Nodes     1.70gb (1827947952b)
17: Column Store Tables:                        Pool/ColumnStore/Delta                                                   1.29gb (1395456960b)
18: System:                                     Pool/ResourceContainer                                                   1.28gb (1377069192b)
19: Monitoring & Statistical Data:              Pool/Statistics/Semaphores/CondVars                                      1.21gb (1308691200b)
20: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultPageProvider:Controlblock 1.17gb (1262125824b)
Process: AB=162.73gb (174735241216b), U=153.80gb (165147820401b), fragmentation=5%.


.....

 

  • Delta Merge statistics show out of memory failures during critical and manual hard merges for large PMEM tables, in this example ACDOCA. SQL : SELECT * FROM M_DELTA_MERGE_STATISTICS WHERE SUCCESS = 'FALSE'
HOST    PORT    TYPE    SCHEMA_NAME    TABLE_NAME    START_TIME    EXECUTION_TIME    MOTIVATION    SUCCESS    CS_ERROR    ERROR_DESCRIPTION                                                            
vhmglsh1db01    30,440    MERGE    SAPABAP1    ACDOCA    2023-05-22 05:38:11.9500000    435,017    CRITICAL    FALSE    2,489    column store error: [2489] Out of memory during delta merge; table = SID::SAPABAP1:_SYS_SPLIT_ACDOCA~4 (t 111888), exception = exception 1000002: Allocation failed ; $size$=42207880; $name$=Prefix; $type$=pool; $inuse_count$=158; $allocated_size$=84544528; $failure_type$=GLOBAL_ALLOCATION_LIMIT; $failure_flag$= (early exit)                                                          
vhmglsh1db01    30,440    MERGE    SAPABAP1    ACDOCA    2023-05-22 06:31:50.9300000    446,707    HARD    FALSE    2,489    column store error: [2489] Out of memory during delta merge; table = SID::SAPABAP1:_SYS_SPLIT_ACDOCA~14 (t 111888), exception = exception 1000002: Allocation failed ; $size$=536870912; $name$=PageableSingle; $type$=pool; $inuse_count$=1828; $allocated_size$=1148950304; $failure_type$=GLOBAL_ALLOCATION_LIMIT; $failure_flag$=                                                          
vhmglsh1db01    30,440    MERGE    SAPABAP1    ACDOCA    2023-05-22 06:23:29.4420000    1,024,161    HARD    FALSE    2,489    column store error: [2489] Out of memory during delta merge; table = SID::SAPABAP1:_SYS_SPLIT_ACDOCA~8 (t 111888), exception = exception 1000002: Allocation failed ; $size$=5759008; $name$=Rle; $type$=pool; $inuse_count$=37; $allocated_size$=3212901968; $failure_type$=GLOBAL_ALLOCATION_LIMIT; $failure_flag$=                                                          
vhmglsh1db01    30,440    MERGE    SAPABAP1    ACDOCA    2023-05-22 06:23:20.4990000    1,146,441    HARD    FALSE    2,489    column store error: [2489] Out of memory during delta merge; table = SID::SAPABAP1:_SYS_SPLIT_ACDOCA~4 (t 111888), exception = exception 1000002: Allocation failed ; $size$=976660480; $name$=Indirect; $type$=pool; $inuse_count$=2953; $allocated_size$=8773779104; $failure_type$=GLOBAL_ALLOCATION_LIMIT; $failure_flag$= (early exit)                                                          
vhmglsh1db01    30,440    MERGE    SAPABAP1    ACDOCA    2023-05-22 06:23:20.4990000    1,209,660    HARD    FALSE    2,489    column store error: [2489] Out of memory during delta merge; table = SID::SAPABAP1:_SYS_SPLIT_ACDOCA~2 (t 111888), exception = exception 1000002: Allocation failed ; $size$=1381054504; $name$=Rle; $type$=pool; $inuse_count$=6; $allocated_size$=6144; $failure_type$=GLOBAL_ALLOCATION_LIMIT; $failure_flag$=                                                          
vhmglsh1db01    30,440    SPARSE    SAPABAP1    ACDOCA    2023-05-22 06:30:45.3180000    945,196    CRITICAL    FALSE    2,009    "column store error: [2009] Memory allocation failed;prepareOptimizations,object=SID::SAPABAP1:_SYS_SPLIT_ACDOCA~5en,exception 70002009: Memory allocation failed;PrepareOptimizationsJob,object=SID::SAPABAP1:_SYS_SPLIT_ACDOCA~5en,exception 1000002: Allocation failed ; $size$=139831048; $name$=Single; $type$=pool; $inuse_count$=93894; $allocated_size$=1259046168; $failure_type$=GLOBAL_ALLOCATION_LIMIT; $failure_flag$= (early exit)

,out of memory during optimize compression"                                                            
vhmglsh1db01    30,440    SPARSE    SAPABAP1    ACDOCA    2023-05-22 06:30:45.2960000    987,327    CRITICAL    FALSE    2,009    column store error: [2009] Memory allocation failed;doOptimizeCompression,object=SID::SAPABAP1:_SYS_SPLIT_ACDOCA~3en,exception 1000002: Allocation failed ; $size$=38731164; $name$=SAPABAP1:_SYS_SPLIT_ACDOCA~3; $type$=pool; $inuse_count$=286; $allocated_size$=5620619568; $failure_type$=GLOBAL_ALLOCATION_LIMIT; $failure_flag$= (early exit)                                                          
vhmglsh1db01    30,440    SPARSE    SAPABAP1    ACDOCA    2023-05-22 06:48:27.3170000    114,701    CRITICAL    FALSE    2,009    "column store error: [2009] Memory allocation failed;prepareOptimizations,object=SID::SAPABAP1:_SYS_SPLIT_ACDOCA~3en,exception 70002009: Memory allocation failed;PrepareOptimizationsJob,object=SID::SAPABAP1:_SYS_SPLIT_ACDOCA~3en,exception 1000002: Allocation failed ; $size$=993871304; $name$=Single; $type$=pool; $inuse_count$=93834; $allocated_size$=113002904; $failure_type$=GLOBAL_ALLOCATION_LIMIT; $failure_flag$=

,out of memory during optimize compression"                            

 

  • A quick check of the affected table (SELECT * FROM M_CS_TABLES WHERE TABLE_NAME = 'ACDOCA' ) , shows it's PMEM size is 152 GB alone under PERSISTENT_MEMORY_SIZE_IN_TOTAL.
152,716,037,048
  • Total size of the overall columnstore tables in PMEM is 600 GB (SELECT SUM (PERSISTENT_MEMORY_SIZE_IN_TOTAL) FROM M_CS_TABLES), leaving < 400 GB for heap memory and delta merge operations.
601,297,655,942


Read more...

Environment

SAP HANA , Platform Edition 2.0       

Product

SAP HANA, platform edition 2.0

Keywords

NVM_ALLOCATED_SIZE,  ResourceContainerShrink.cpp, TraceContext.cpp, GLOBAL_ALLOCATION_LIMIT, Total used memory,  , KBA , HAN-DB , SAP HANA Database , HAN-DB-ENG , SAP HANA DB Engines , Problem

About this page

This is a preview of a SAP Knowledge Base Article. Click more to access the full version on SAP for Me (Login required).

Search for additional results

Visit SAP Support Portal's SAP Notes and KBA Search.