3484923 - SAP HANA Database Primary and HA sites went down without failover

It is observed that Primary site in replication site went down because of high memory usage.

On Primary Site:

indexserver_xxxxxx.3420.30203.008.trc

[71644]\{304991\}[646/12974590001] 2024-06-03 12:48:25.895676 w Memory           mmPoolAllocator.cpp(01229) : Out of memory for Pool/JoinEvaluator/JEStep2, size 1193472660B, alignment=1B, flags 0x0, reason STATEMENT_MEMORY_LIMIT
[368293]\{304991\}[646/12974590001] 2024-06-03 12:48:26.372335 e Memory           mmReportMemoryProblems.cpp(01834) : Composite limit violation (OUT OF MEMORY) occurred.
Composite limit=500gb (536870912000b)
Root allocator name=Connection/304991/Statement/1309928063353083
Host: <hostname>
Executable: hdbindexserver
PID: 58377
Failed to allocate 1.11gb (1192394384b).

daemon_xxxxxxx.30200.028.trc

[58068]{-1}[-1/-1] 2024-06-03 17:46:05.905559 i Daemon           SignalsUNIX.cpp(00583) : signo 2=SIGINT from user. errno 0 code 0. Requested 'QUIT'. Sender pid 58060, real user 'sidadm'=1001, executable 'sapstart'

[58068]{-1}[-1/-1] 2024-06-03 17:46:05.906072 i Daemon           DaemonHandle.cpp(00094) : Got shutdown event (quit). Stop children processes with timeout 270000 ms

OS error log (/var/log messages)

This memory exhaustion  brings down the Primary system and in an ideal situation a failover should be performed to the HA site but this does not happen. 

SAPHana[323394]: WARNING: RA: HANA_CALL timed out after 60 seconds running command 'landscapeHostConfiguration.py'
SAPHana[323394]: WARNING: RA: HANA_CALL timed out after 60 seconds running command 'landscapeHostConfiguration.py'

HA site traces:

indexserver_xxxxxx.30203.016.trc (HA_site)

[95424]{-1}[-1/-1] 2024-06-03 17:29:52.336069 i PersistenceManag DisasterRecoveryProtocol.cpp(07628) : Asynchronous replication buffer full, accumulated count = 1269, trace cooldown = 300 s

[193471]{-1}[-1/-1] 2024-06-03 17:29:52.336709 i EventHandler     EventManagerImpl.cpp(00951) : Event 'SystemReplicationEvent: site=3, Site 3: exception 3000321: Asynchronous Replication Buffer is Overloaded
' set to state 'handled'
[193471]{-1}[-1/-1] 2024-06-03 17:29:52.336715 i EventHandler     EventManagerImpl.cpp(00951) : Event 'SystemReplicationEvent: site=3, Site 3: exception 3000321: Asynchronous Replication Buffer is Overloaded
' set to state 'handled'
[193471]{-1}[-1/-1] 2024-06-03 17:29:52.336717 i EventHandler     EventManagerImpl.cpp(00951) : Event 'SystemReplicationEvent: site=3, Site 3: exception 3000321: Asynchronous Replication Buffer is Overloaded
' set to state 'handled'
[193471]{-1}[-1/-1] 2024-06-03 17:29:52.336718 i EventHandler     EventManagerImpl.cpp(00951) : Event 'SystemReplicationEvent: site=3, Site 3: exception 3000321: Asynchronous Replication Buffer is Overloaded
' set to state 'handled'

SAP HANA, platform edition

Replication, HA, Failover , KBA , HAN-DB-HA , SAP HANA High Availability (System Replication, DR, etc.) , Problem