3291783 - WARNING: RA: HANA_CALL timed out error causes cluster failover due to Memory pressure on HANA

• An unexpected failover between a primary and secondary site occurs after a hard shutdown on the primary site

system_availability_hostname.trc
------------------------------------

0;2023-01-13 11:06:48.076106;fed379e4-a859-8547-9531-1fe11627089e;1;ping;;;yes;ok;yes;ok;;;hostname;;;;;worker;worker;1;1;;;;;;;
0;2023-01-13 11:09:36.522921;85c5db0b-a5d7-ef4d-bc4e-8c086ade6bad;1;host_stopping;mode: hard ;;unknown;unknown;stopping;unknown;;;hostname;;;;;unknown;unknown;0;0;;;;;;;
0;2023-01-13 11:09:37.094891;81a8a180-7910-5c4a-840b-c4ae39beb289;1;service_stopping;;;stopping;warning;stopping;warning;stopping;;hostname;30006;webdispatcher;;;worker;worker;1;1;;;;;;;

daemon_hostname.30000.002.trc
----------------------------------
[24780]{-1}[-1/-1] 2023-01-13 11:09:36.520185 i Daemon           SignalsUNIX.cpp(00583) : signo 2=SIGINT from user. errno 0 code 0. Requested 'QUIT'. Sender pid 24772, real user 'sidadm'=1001, executable 'sapstart'

• Investigation shows the failover occurred automatically after a cluster/pacemaker tool detected an unresponsive nameserver :

var/log/messages - 


2023-01-13T11:04:03.788922+09:00 hostname SAPHana(rsc_SAPHana_SID_HDB00)[44016]: WARNING: RA: HANA_CALL timed out after 10 seconds running command 'HDB version'
2023-01-13T11:04:20.388990+09:00 hostname SAPHanaTopology(rsc_SAPHanaTopology_SID_HDB00)[43281]: WARNING: RA: HANA_CALL timed out after 60 seconds running command 'hdbnsutil -sr_stateConfiguration --sapcontrol=1'
2023-01-13T11:05:14.715528+09:00 hostname SAPHana(rsc_SAPHana_SID_HDB00)[44016]: WARNING: RA: HANA_CALL timed out after 60 seconds running command 'hdbnsutil -sr_state'
2023-01-13T11:05:24.440091+09:00 hostname SAPHanaTopology(rsc_SAPHanaTopology_A1P_HDB00)[43281]: WARNING: RA: HANA_CALL timed out after 60 seconds running command 'hdbnsutil -sr_stateConfiguration --sapcontrol=1'
2023-01-13T11:06:26.132815+09:00 hostname SAPHana(rsc_SAPHana_SID_HDB00)[44016]: WARNING: RA: HANA_CALL timed out after 60 seconds running command 'hdbnsutil -sr_state'
2023-01-13T11:06:28.500448+09:00 hostname SAPHanaTopology(rsc_SAPHanaTopology_SID_HDB00)[43281]: WARNING: RA: HANA_CALL timed out after 60 seconds running command 'hdbnsutil -sr_stateConfiguration --sapcontrol=1'
2023-01-13T11:07:32.547155+09:00 hostname SAPHanaTopology(rsc_SAPHanaTopology_SID_HDB00)[43281]: WARNING: RA: HANA_CALL timed out after 60 seconds running command 'HDBSettings.sh getParameter.py --key=global.ini/system_replication/actual_mode --key=global.ini/system_replication/mode --key=global.ini/system_replication/site_name --key=global.ini/system_replication/site_id --sapcontrol=1'
2023-01-13T11:07:39.636239+09:00 hostname SAPHana(rsc_SAPHana_SID_HDB00)[44016]: WARNING: RA: HANA_CALL timed out after 60 seconds running command 'hdbnsutil -sr_state'
2023-01-13T11:08:36.814532+09:00 hostname SAPHanaTopology(rsc_SAPHanaTopology_SID_HDB00)[43281]: WARNING: RA: HANA_CALL timed out after 60 seconds running command 'hdbnsutil -sr_stateHostMapping --sapcontrol=1'
2023-01-13T11:08:56.120394+09:00 hostname SAPHana(rsc_SAPHana_SID_HDB00)[44016]: WARNING: RA: HANA_CALL timed out after 60 seconds running command 'HDBSettings.sh getParameter.py --key=global.ini/system_replication/actual_mode --key=global.ini/system_replication/mode --sapcontrol=1'




2023-01-13T11:14:47.864841+09:00 hostname SAPHanaTopology(rsc_SAPHanaTopology_SID_HDB00)[58136]: WARNING: RA: HANA_CALL stderr from command 'su - sidadm -c' is '', stderr from command 'landscapeHostConfiguration.py --sapcontrol=1' is 'nameserver hostname:30001 not responding.#012nameserver hostname:30001 not responding.'
2023-01-13T11:15:03.639315+09:00 hostname SAPHanaTopology(rsc_SAPHanaTopology_SID_HDB00)[58911]: WARNING: RA: HANA_CALL stderr from command 'su - sidadm -c' is '', stderr from command 'landscapeHostConfiguration.py --sapcontrol=1' is 'nameserver hostname:30001 not responding.#012nameserver hostname:30001 not responding.'

• Memory and Resource Container shrinks are observed on the nameserver and indexserver before the hard shutdown is issued :

nameserver_hostname.30001.097.trc
//////////////////


 [24992]{-1}[-1/-1] 2023-01-13 11:04:20.495089 w ResMan           ResourceContainerShrink.cpp(02492) : Information about shrink at 2023-01-13 11:04:20 492 Local:
  1: 0x00007fb8513bd036 in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0xb2 at ResourceContainerShrink.cpp:725 (libhdbbasis.so)
  2: 0x00007fb8513b57f8 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x5c4 at ResourceContainerShrink.cpp:2491 (libhdbbasis.so)
 
 [24992]{-1}[-1/-1] 2023-01-13 11:07:51.742912 w ResMan           ResourceContainerShrink.cpp(02492) : Information about shrink at 2023-01-13 11:07:51 740 Local:
  1: 0x00007fb8513bd036 in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0xb2 at ResourceContainerShrink.cpp:725 (libhdbbasis.so)
  2: 0x00007fb8513b57f8 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x5c4 at ResourceContainerShrink.cpp:2491 (libhdbbasis.so)
 
 [24992]{-1}[-1/-1] 2023-01-13 11:08:50.231375 w ResMan           ResourceContainerShrink.cpp(02492) : Information about shrink at 2023-01-13 11:08:50 228 Local:
  1: 0x00007fb8513bd036 in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0xb2 at ResourceContainerShrink.cpp:725 (libhdbbasis.so)
  2: 0x00007fb8513b57f8 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x5c4 at ResourceContainerShrink.cpp:2491 (libhdbbasis.so)
    

[24992]{-1}[-1/-1] 2023-01-13 11:08:50.231375 w ResMan           ResourceContainerShrink.cpp(02492) : Information about shrink at 2023-01-13 11:08:50 228 Local:
Reason for shrink: IPMM (other process). Request ID= 296635
ShrinkCaller
 1: 0x00007fb8513bd036 in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0xb2 at ResourceContainerShrink.cpp:725 (libhdbbasis.so)
 2: 0x00007fb8513b57f8 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x5c4 at ResourceContainerShrink.cpp:2491 (libhdbbasis.so)
 3: 0x00007fb8514d34f2 in MemoryManager::CompactionThread::compactCompactors(unsigned long, bool&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x530 at mmCompactionTrigger.cpp:960 (libhdbbasis.so)
 4: 0x00007fb8514cedbe in MemoryManager::CompactionThread::doCompactionIntern(MemoryManager::Precharge&, unsigned long, unsigned long, unsigned long, unsigned long&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType, bool)+0xfba at mmCompactionTrigger.cpp:1122 (libhdbbasis.so)
 5: 0x00007fb8514cfdc5 in MemoryManager::CompactionThread::doIpmmCompactionIntern(unsigned long, int, unsigned long, unsigned long, unsigned long, MemoryManager::ProcessSlot::RequestedTask)+0x1b1 at mmCompactionTrigger.cpp:1712 (libhdbbasis.so)
 6: 0x00007fb8514d0804 in MemoryManager::CompactionThread::run(void*&)+0x190 at mmCompactionTrigger.cpp:1667 (libhdbbasis.so)
 7: 0x00007fb851647415 in Execution::Thread::staticMainImp(Execution::Thread*)+0x531 at Thread.cpp:522 (libhdbbasis.so)
 8: 0x00007fb851637218 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x1e4 at ThreadInterposition.cpp:639 (libhdbbasis.so)
 9: 0x00007fb850c7f539 in start_thread+0xd5 (libpthread.so.0)
10: 0x00007fb84f2ffcff in __GI___clone+0x3b (libc.so.6)
IPMM short info:
GLOBAL_ALLOCATION_LIMIT (GAL) = 484.03gb (519731306496b), SHARED_MEMORY = 9.05gb (9722228736b), CODE_SIZE = 3.26gb (3511488512b), OOM_RESERVATION_ALLOCATOR_SIZE = 96.14mb (100810752b)
PID=24919 (hdbnameserver), PAL=484.16gb (519865524224b), AB=4.30gb (4627865600b), UA=0, U=3.27gb (3516109919b), FSL=0b, FMC=0b
PID=25617 (hdbcompileserver), PAL=484.16gb (519865524224b), AB=295.13mb (309473280b), UA=0, U=149.47mb (156735336b), FSL=0b, FMC=0b, Blocked=483.10mb (506576896b)
PID=25620 (hdbpreprocessor), PAL=484.16gb (519865524224b), AB=467mb (489689088b), UA=0, U=304.22mb (319001879b), FSL=0b, FMC=0b
PID=25676 (hdbxsengine), PAL=484.16gb (519865524224b), AB=1.36gb (1463627776b), UA=0, U=1.10gb (1191859092b), FSL=0b, FMC=0b
PID=25673 (hdbindexserver), PAL=484.16gb (519865524224b), AB=464.65gb (498922434560b), UA=0, U=413.96gb (444495321766b), FSL=0b, FMC=0b
PID=31590 (hdbwebdispatcher), PAL=484.16gb (519865524224b), AB=474.02mb (497053696b), UA=0, U=365.50mb (383256175b), FSL=0b, FMC=0b
Total allocated memory= 483.86gb (519543861248b)
Total used memory     = 431.47gb (463296001415b)
Sum AB                = 506310144000
Sum Used              = 450062284167
Heap memory fragmentation: 10
Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by inclusive_size_in_use.
 1: System:                                     /                                                                    3.27gb (3516109919b)
 2: System:                                     Pool                                                                 2.77gb (2984460245b)





indexserver_hostname.30003.698.trc
///////////////


[4013]\{300355\}[214/-1] 2023-01-13 11:04:24.808851 w ResMan           ResourceContainerShrink.cpp(02492) : Information about shrink at 2023-01-13 11:04:24 804 Local:
 1: 0x00007f14c49e6036 in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0xb2 at ResourceContainerShrink.cpp:725 (libhdbbasis.so)
 2: 0x00007f14c49de7f8 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x5c4 at ResourceContainerShrink.cpp:2491 (libhdbbasis.so)
[4013]\{300355\}[214/-1] 2023-01-13 11:04:35.197906 w ResMan           ResourceContainerShrink.cpp(02492) : Information about shrink at 2023-01-13 11:04:35 192 Local:
 1: 0x00007f14c49e6036 in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0xb2 at ResourceContainerShrink.cpp:725 (libhdbbasis.so)
 2: 0x00007f14c49de7f8 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x5c4 at ResourceContainerShrink.cpp:2491 (libhdbbasis.so)

.....


[25841]{-1}[-1/-1] 2023-01-13 11:08:47.876260 w ResMan           ResourceContainerShrink.cpp(02492) : Information about shrink at 2023-01-13 11:08:47 871 Local:
 1: 0x00007f14c49e6036 in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0xb2 at ResourceContainerShrink.cpp:725 (libhdbbasis.so)
 2: 0x00007f14c49de7f8 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x5c4 at ResourceContainerShrink.cpp:2491 (libhdbbasis.so)
[6030]\{300329\}[-1/-1] 2023-01-13 11:08:50.313978 w ResMan           ResourceContainerShrink.cpp(02492) : Information about shrink at 2023-01-13 11:08:50 309 Local:
 1: 0x00007f14c49e6036 in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0xb2 at ResourceContainerShrink.cpp:725 (libhdbbasis.so)
 2: 0x00007f14c49de7f8 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x5c4 at ResourceContainerShrink.cpp:2491 (libhdbbasis.so)
    
    

[6030]\{300329\}[-1/-1] 2023-01-13 11:08:50.313988 i TraceContext     TraceContext.cpp(01269) : UserName=SAPABAP1, ApplicationUserName=23079118, ApplicationName=ABAP:SID, ApplicationSource=YSDRKAO085:451, Client=100, EppRootContextId=00224868F01F1EEDA4DB6E914EAAD228, EppTransactionId=BB22316DD8F00200E00639E6CF81ABFE, EppConnectionId=DEDB92ED7F6CF184B22800224868F01F, EppConnectionCounter=1, EppComponentName=SID/hostname_SID_00, EppAction=RVV50R10C, StatementExecutionID=844438178925662
[6030]\{300329\}[-1/-1] 2023-01-13 11:08:50.313978 w ResMan           ResourceContainerShrink.cpp(02492) : Information about shrink at 2023-01-13 11:08:50 309 Local:
Reason for shrink: Reserve for BBA, user size:  48500
ShrinkCaller
 1: 0x00007f14c49e6036 in ResourceManager::ShrinkInformation::updateShrinkInformation(char const*, unsigned long, bool)+0xb2 at ResourceContainerShrink.cpp:725 (libhdbbasis.so)
 2: 0x00007f14c49de7f8 in ResourceManager::ResourceContainerImpl::genericShrink(unsigned long, unsigned long, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x5c4 at ResourceContainerShrink.cpp:2491 (libhdbbasis.so)
 3: 0x00007f14c4afc4f2 in MemoryManager::CompactionThread::compactCompactors(unsigned long, bool&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x530 at mmCompactionTrigger.cpp:960 (libhdbbasis.so)
 4: 0x00007f14c4af7dbe in MemoryManager::CompactionThread::doCompactionIntern(MemoryManager::Precharge&, unsigned long, unsigned long, unsigned long, unsigned long&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType, bool)+0xfba at mmCompactionTrigger.cpp:1122 (libhdbbasis.so)
 5: 0x00007f14c4af90b8 in MemoryManager::CompactionThread::doCompaction(MemoryManager::Precharge&, unsigned long, unsigned long, unsigned long, unsigned long&, char const*, unsigned long, MemoryManager::ReclaimMemoryTaskType)+0x14 at mmCompactionTrigger.cpp:1008 (libhdbbasis.so)
 6: 0x00007f14c4acf1a5 in MemoryManager::IPMMReclaimMemory::pmInterProcessProvideMemory(MemoryManager::PMInfo&, MemoryManager::RequestorInfo const&, MemoryManager::ReclaimMemoryTimer const&)+0x1bf1 at mmIPMMReclaimMemory.cpp:2847 (libhdbbasis.so)
 7: 0x00007f14c4ad04d4 in MemoryManager::IPMMReclaimMemory::provideMemoryAndReturnSize(MemoryManager::Precharge&, MemoryManager::ProvideMemoryPurpose, unsigned long, MemoryManager::AllocatorConfigurationSPI&, MemoryManager::AllocationStatus*, char const*, unsigned long, MemoryManager::TPALPrecharge&) [clone .part.0]+0xf10 at mmIPMMReclaimMemory.cpp:1085 (libhdbbasis.so)
 8: 0x00007f14c4ad34f2 in MemoryManager::IPMMReclaimMemory::provideMemory(MemoryManager::Precharge&, MemoryManager::ProvideMemoryPurpose, unsigned long, MemoryManager::AllocatorConfigurationSPI&, MemoryManager::AllocationStatus*, char const*, unsigned long)+0x80 at mmIPMMReclaimMemory.cpp:811 (libhdbbasis.so)
 9: 0x00007f14c4b37660 in MemoryManager::AllocatorConfigurationSPI::provideMemory(MemoryManager::Precharge&, unsigned long, bool, char const*, unsigned long)+0x30 at mmAllocatorConfigurationImpl.cpp:1633 (libhdbbasis.so)
10: 0x00007f14c4a981ee in MemoryManager::MemoryPool::reserveMemoryForPoolAllocatorAllocation(MemoryManager::Precharge&, unsigned long, char const*, unsigned long)+0xea at mmMemoryPool.cpp:2512 (libhdbbasis.so)
11: 0x00007f14c4944da6 in MemoryManager::MemoryPool::reserveMemoryAndAllocateBigOrHugeBlock(unsigned long, bool, unsigned short, ltt::allocator_statistics::SubStats&, MemoryManager::ImpliInfo, void const*, Synchronization::LockHandle<Synchronization::Mutex, false>&) [clone .constprop.0]+0xf2 at mmMemoryPool.cpp:2450 (libhdbbasis.so)
12: 0x00007f14c4a98d00 in MemoryManager::MemoryPool::allocateBigOrHugeBlock(unsigned long, unsigned short, ltt::allocator_statistics&, MemoryManager::ImpliInfo, void const*, bool, Synchronization::LockHandle<Synchronization::Mutex, false>&, bool)+0x270 at mmMemoryPool.cpp:2396 (libhdbbasis.so)
13: 0x00007f14c4a9cde5 in MemoryManager::MemoryPool::allocate(unsigned long, unsigned short, ltt::allocator_statistics&, bool&, bool, void const*)+0x521 at mmMemoryPool.cpp:1445 (libhdbbasis.so)
14: 0x00007f14c4a69444 in MemoryManager::PoolAllocator::allocateNoThrowImpl(unsigned long, void const*)+0x60 at mmPoolAllocator.cpp:1767 (libhdbbasis.so)
15: 0x00007f14d1659e05 in ltt::allocator::allocate(unsigned long)+0x21 at memory.cpp:197 (libhdbrskernel.so)
16: 0x00007f14d392f3f4 in ptime::ShmInfo::setOrAppendInsertData(unsigned char*, FdaChunkHeader_v0_0*, unsigned char*&, unsigned long&)+0x1e0 at sm_shm.cc:209 (libhdbrskernel.so)
17: 0x00007f14dc4fc81c in ptime::ActionExecutor::prepareExecQid(Execution::Context&, ptime::Action&)+0x8a8 at sm_action_executor.cc:342 (libhdbsqlsession.so)
18: 0x00007f14dc4fb2f9 in ptime::ActionExecutor::prepareAction(Execution::Context&, ptime::SessionEvent*, ptime::Action&)+0x35 at sm_action_executor.cc:131 (libhdbsqlsession.so)
19: 0x00007f14dc47805a in ptime::SessionHandler::prepareAction(Execution::Context&, SessionLayer::SessionEvent*, ptime::Action&)+0xc6 at sm_handler.cc:2447 (libhdbsqlsession.so)
20: 0x00007f14dc47a690 in .LTHUNK390.lto_priv.1+0xd0 at sm_handler.cc:2137 (libhdbsqlsession.so)
21: 0x00007f14dc482e5b in ptime::SessionHandler::receiveMessage_(Execution::Context&, ptime::ThreadStatusCallbackBase&, SessionLayer::CommEvent*, bool)+0xb27 at sm_handler.cc:1977 (libhdbsqlsession.so)
22: 0x00007f14dc48426f in ptime::SessionHandler::receiveMessage(Execution::Context&, SessionLayer::CommEvent*, bool)+0x2b at sm_handler.cc:1805 (libhdbsqlsession.so)
23: 0x00007f14dc448f06 in ptime::TcpReceiver::doWork(Execution::Context&, SessionLayer::CommMgr*)+0xae2 at tcp_receiver.cc:388 (libhdbsqlsession.so)
24: 0x00007f14dc444828 in ptime::TcpReceiver::run(void*)+0xd4 at tcp_receiver.cc:489 (libhdbsqlsession.so)
25: 0x00007f14ca074378 in TrexThreads::PoolThread::run()+0x984 at PoolThread.cpp:462 (libhdbbasement.so)
26: 0x00007f14ca0761d0 in TrexThreads::PoolThread::run(void*&)+0x10 at PoolThread.cpp:142 (libhdbbasement.so)
27: 0x00007f14c4c70415 in Execution::Thread::staticMainImp(Execution::Thread*)+0x531 at Thread.cpp:522 (libhdbbasis.so)
28: 0x00007f14c4c60218 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x1e4 at ThreadInterposition.cpp:639 (libhdbbasis.so)
29: 0x00007f14c42a8539 in start_thread+0xd5 (libpthread.so.0)
30: 0x00007f14c2928cff in __GI___clone+0x3b (libc.so.6)
IPMM short info:
GLOBAL_ALLOCATION_LIMIT (GAL) = 484.03gb (519731306496b), SHARED_MEMORY = 9.05gb (9722228736b), CODE_SIZE = 3.26gb (3511488512b), OOM_RESERVATION_ALLOCATOR_SIZE = 96.14mb (100810752b)
PID=25673 (hdbindexserver), PAL=484.16gb (519865524224b), AB=465.62gb (499965513728b), UA=0, U=413.96gb (444495321766b), FSL=0b, FMC=0b
[Some processes may have been skipped]
Total allocated memory= 484.82gb (520577216512b)
Total used memory     = 431.47gb (463295951463b)
Sum AB                = 507343499264
Sum Used              = 450062234215
Heap memory fragmentation: 11

Top "M_HEAP_MEMORY" allocators (component, name, size). Ordered descending by exclusive_size_in_use.
 1: Column Store Tables:                        Pool/ColumnStore/Main/Dictionary/RoDict                              81.62gb (87650140170b)
 2: Statement Execution & Intermediate Results: Pool/JoinEvaluator/JERequestedAttributes/Results                     75.01gb (80550983480b)
 3: System:                                     Pool/parallel/aggregates                                             24.96gb (26801899008b)
 4: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultLPA/LOBPage           22.10gb (23740051456b)
 5: System:                                     Pool/PersistenceManager/PersistentSpace/DefaultLPA/DataPage          16.25gb (17456615424b)
 6: Column Store Tables:                        Pool/ColumnStore/Main/Compressed/Indirect                            12.46gb (13383075336b)
 7: Statement Execution & Intermediate Results: Pool/JoinEvaluator/JEAggregate/Results                               12.04gb (12931685336b)
 8: Column Store Tables:                        Pool/ColumnStore/Main/Index/Single                                   11.09gb (11917791244b)
 9: System:                                     Pool/ResourceContainer                                               10.65gb (11443567032b)
10: Column Store Tables:                        Pool/ColumnStore/Main/Uncompressed                                   10.04gb (10783029608b)
11: Monitoring & Statistical Data:              Pool/Statistics                                                      10gb (10741428272b)
12: Column Store Tables:                        Pool/ColumnStore/Main/Index/PageableSingle                           8.05gb (8649543268b)
13: System:                                     Pool/PersistenceManager/MidSizeLOBContainerFileIDMapping/Map         7.45gb (8002411008b)
14: System:                                     Pool/parallel/compactcol                                             6.62gb (7116068864b)
15: Column Store Tables:                        Pool/ColumnStore/Main/Rowid                                          5.86gb (6300884440b)
16: Statement Execution & Intermediate Results: Pool/L/llang/Debuggee                                                5.24gb (5633387048b)
17: System:                                     AllocateOnlyAllocator-unlimited/FLA-UL<24592,1>/MemoryMapLevel3Nodes 5.06gb (5439135600b)
18: Column Store Tables:                        Pool/PersistenceManager/UnifiedTableContainer                        4.98gb (5355979848b)
19: System:                                     Pool/ResourceContainer/ResourceHeader                                4.02gb (4317408032b)
20: System:                                     Pool/PersistenceManager/MidSizeLOBContainerFileIDMapping/BackMap     3.82gb (4105498128b)
Process: AB=465.62gb (499965513728b), U=414.21gb (444761661454b), fragmentation=11%.

.... 

[25673]{-1}[-1/-1] 2023-01-13 11:09:36.917848 i Service_Shutdown TrexService.cpp(00533) : Preparing for shutting service down
[28340]{-1}[-1/-1] 2023-01-13 11:09:36.917956 i Service_Shutdown TREXIndexServer.cpp(02602) : Stopping data backup
[28340]{-1}[-1/-1] 2023-01-13 11:09:36.918488 i Service_Shutdown TREXIndexServer.cpp(02607) : preparing auditing shutdown
[28340]{-1}[-1/-1] 2023-01-13 11:09:36.919276 i assign           TREXIndexServer.cpp(02614) : unassign from volume 3
[28340]{-1}[-1/-1] 2023-01-13 11:09:36.919437 i Service_Shutdown TREXIndexServer.cpp(02616) : Preparing to shutdown

SAP HANA, platform edition 2.0

SAP HANA, platform edition 1.0

host_stopping, unknown;stopping, SignalsUNIX.cpp, signo 2=SIGINT, HANA_CALL timed out after, ResourceContainerShrink.cpp, Pool/ColumnStore/Main/Dictionary/RoDict, Pool/JoinEvaluator/JERequestedAttributes/Results. , KBA , HAN-DB , SAP HANA Database , HAN-DB-HA , SAP HANA High Availability (System Replication, DR, etc.) , Problem