3226899 - 'Error in local merge: not enough merge tokens for delta merge or optimize compression' observed on a HANA secondary site

• Alerts for a HANA secondary site are observed on the currently running primary site : 
  "Delta Merge was not executed successfully on host:port for table /<>/table_name in schema SAPABAP1 for 1 occurrences in the last 24 hours." - (2445867 - How-To: Interpreting and Resolving SAP HANA Alerts)  

       ('Host:port' have been obfuscated in the image for confidentiality reasons)

Image /data in this KBA is from SAP internal systems, sample data, or demo systems. Any resemblance to real data is purely coincidental. 

• Traces from the indexserver on the Secondary site shows memory exhaustion in the form of Resource Container shrinks and a lack of delta merge tokens :
   
   indexserver_host.port.529.trc
    ++++++++
  

    [114737]{-1}[-1/9223372036854775806] 2022-07-19 12:38:20.897169 e delta_merge      CsTableMerge.cpp(02329) : Error in local merge: not enough merge tokens for delta merge or optimize compression : table = SID::SAPABAP1:<table_name> (t 215523), merge motivation = 
    [96656]{-1}[-1/-1] 2022-07-19 12:39:19.192782 w ResMan           ResourceContainerShrink.cpp(03167) : Information about shrink at 2022-07-19 12:39:19 188 Local:
    Reason for shrink: IPMM (other process). Request ID= 24068893
    ShrinkCaller
       0: 0x00007f429c7e39fb in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0x97 at Basis/ResourceManager/impl/ResourceContainerShrink.cpp:1042 (libhdbbasis.so)
       1: 0x00007f429c7f1641 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x29a0 at Basis/ResourceManager/impl/ResourceContainerShrink.cpp:3166 (libhdbbasis.so)
       2: 0x00007f429c6c98ec in MemoryManager::CompactionThread::compactCompactors(unsigned long, bool&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x518 at Basis/MemoryManager/impl/mmCompactionThread.cpp:269 (libhdbbasis.so)
       3: 0x00007f429c6cad0b in MemoryManager::CompactionThread::doCompactionIntern(MemoryManager::Precharge&, unsigned long, unsigned long, unsigned long, unsigned long&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType, bool)+0x8b7 at Basis/MemoryManager/impl/mmCompactionThread.cpp:433 (libhdbbasis.so)
       4: 0x00007f429c6cc691 in MemoryManager::CompactionThread::doIpmmCompactionIntern(unsigned long, int, unsigned long, unsigned long, unsigned long, MemoryManager::ProcessSlot::RequestedTask)+0x320 at Basis/MemoryManager/impl/mmCompactionTrigger.cpp:458 (libhdbbasis.so)
       5: 0x00007f429c6cd7ce in MemoryManager::CompactionThread::run(Execution::ThreadRC&)+0x22a at Basis/MemoryManager/impl/mmCompactionTrigger.cpp:413 (libhdbbasis.so)
       6: 0x00007f429c52c6c6 in Execution::Thread::staticMainImp(Execution::Thread*)+0x542 at Basis/Execution/impl/Thread.cpp:574 (libhdbbasis.so)
       7: 0x00007f429c533ca5 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x1c1 at Basis/Execution/impl/ThreadInterposition.cpp:693 (libhdbbasis.so)
       8: 0x0000000000000000 in <no symbol>+0x0 (<unknown>)
    IPMM short info:
    GLOBAL_ALLOCATION_LIMIT (GAL) = 699.87gb (751485059072b), SHARED_MEMORY = 22.92gb (24621215744b), CODE_SIZE = 3.21gb (3456172032b), NVM_ALLOCATED_SIZE = 0b, OOM_RESERVATION_ALLOCATOR_SIZE = 96.14mb (100810752b)
    PID=96540 (hdbindexserver), PAL=1.90tb (2094153310208b), AB=661.68gb (710475743232b), UA=0, U=645.16gb (692738926253b), FSL=0b, FMC=0b
    [Some processes may have been skipped]
    Total allocated memory= 699.76gb (751366168576b)
    Total used memory     = 681.54gb (731801395748b)
    Sum AB                = 723288780800
    Sum Used              = 703724007972
    Heap memory fragmentation: 2
  
     . . . . . 
  
    Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by exclusive_size_in_use.
     1: Column Store Tables: Pool/ColumnStore/Main/Dictionary/RoDict                              211.93gb (227569076706b)
     2: Column Store Tables: Pool/ColumnStore/Main/Compressed/Indirect                            86.95gb (93373054952b)
     3: Column Store Tables: Pool/ColumnStore/Main/Index/Single                                   80.83gb (86798670156b)
     4: Column Store Tables: Pool/ColumnStore/Main/Uncompressed                                   62.86gb (67498445656b)
     5: Column Store Tables: Pool/ColumnStore/Main/Rowid                                          61.11gb (65627156544b)
     6: System:              Pool/PersistenceManager/PersistentSpace/DefaultPageProvider:DataPage 23.35gb (25080799232b)
     7: Column Store Tables: Pool/ColumnStore/Main/Dictionary/ValueDict                           19.30gb (20734209968b)
     8: Column Store Tables: Pool/ColumnStore/Main/Compressed/Sparse                              19.07gb (20477639832b)
     9: Column Store Tables: Pool/ColumnStore/Main/Compressed/Rle                                 9.26gb (9953156512b)
    10: Column Store Tables: Pool/PersistenceManager/UnifiedTableContainer                        8.60gb (9235985183b)
    11: System:              AllocateOnlyAllocator-unlimited/FLA-UL<24592,1>/MemoryMapLevel3Nodes 8.02gb (8617184352b)
    12: System:              Pool/PersistenceManager/DisasterRecoverySecondary/ReplayLogCache     5.27gb (5661156736b)
    13: Column Store Tables: Pool/PersistenceManager/UnifiedTableContainer/MVCC                   4.16gb (4475320936b)
    14: Row Store Tables:    Pool/RowStoreTables/CpbTree                                          4.06gb (4361942160b)
    15: Column Store Tables: Pool/ColumnStore/Main/Compressed/Cluster                             3.29gb (3534329312b)
    16: Column Store Tables: Pool/ColumnStore/Main/Compressed/Prefix                              3.16gb (3395067408b)
    17: System:              Pool/PersistenceManager/MidSizeLOBContainerFileIDMapping/Map         2.54gb (2732705008b)
    18: System:              Pool/PersistenceManager/ContainerFileIDMapping/FileIDBackMapper      2.50gb (2684830208b)
    19: System:              Pool/PersistenceManager/ContainerFileIDMapping/FileIDMapper          2gb (2147942912b)
    20: System:              Pool/ColumnStore/System                                              1.80gb (1937359024b)
    Process: AB=661.68gb (710475743232b), U=645.16gb (692738926253b), fragmentation=2%.

    

SAP HANA, platform edition 2.0

system replication, logreplay, logreplay_readaccess, oom , KBA , HAN-DB , SAP HANA Database , HAN-DB-HA , SAP HANA High Availability (System Replication, DR, etc.) , How To