3701546 - HANA System Takeover to Secondary Node Caused by Pacemaker Resource Monitoring Timeout

system_availability_xxxx-l01.trc

0;2025-12-09 21:31:06.203819;7be26d01-8949-9f4d-b60e-f846d3c08c34;1;host_stopping;mode: hard user. errno 0 code 0. Requested 'QUIT'. Sender pid 3409364, real user 'admxxxx'=79, executable 'sapstart';;unknown;unknown;stopping;unknown;;;xxxx-l01;;;;;unknown;unknown;0;0;;;;;;;
0;2025-12-09 21:31:06.279602;add134b7-318f-c244-a136-242318e3aa97;1;service_stopping;;;stopping;warning;stopping;warning;stopping;;xxxx-l01;30006;webdispatcher;;;worker;worker;1;1;;;;;;;
0;2025-12-09 21:31:06.280728;05b7e4bb-3fc3-774c-b1be-ba2910591527;1;service_stopping;;;stopping;warning;stopping;warning;stopping;;xxxx-l01;30010;compileserver;;;worker;worker;1;1;;;;;;;
0;2025-12-09 21:31:06.282665;bb2f6797-8231-1448-90a4-0c1afb7b490a;1;service_stopped;;;stopping;warning;stopping;warning;no;;xxxx-l01;30010;compileserver;;;worker;worker;1;1;;;;;;;
0;2025-12-09 21:31:06.386648;83ab5a6e-6268-1e43-86e6-6d59ed436091;1;service_stopped;;;stopping;warning;stopping;warning;no;;xxxx-l01;30006;webdispatcher;;;worker;worker;1;1;;;;;;;

daemon_xxxx-l01.30000.081.trc

[3409372]{-1}[-1/-1] 2025-12-09 21:31:06.203462 i Daemon            SignalsUNIX.cpp(00601) : signo 2=SIGINT from user. errno 0 code 0. Requested 'QUIT'. Sender pid 3409364, real user 'admxxxx'=79, executable 'sapstart'
[3409372]{-1}[-1/-1] 2025-12-09 21:31:06.203664 i Daemon            DaemonHandle.cpp(00094) : Got shutdown event (quit). Stop children processes with timeout 270000 ms
[3409372]{-1}[-1/-1] 2025-12-09 21:31:06.203667 i Daemon            RunningInstance.cpp(00293) : Stop process 'hdbcompileserver', pid 3409742, instance 0
[3409372]{-1}[-1/-1] 2025-12-09 21:31:06.203701 i Daemon            Utilities.cpp(00605) : Stopped child, pid 3409742 with SIGTERM
[3409372]{-1}[-1/-1] 2025-12-09 21:31:06.203702 i Daemon            Events.cpp(00133) : "KillInstanceEvent" is due in 270000 ms. Program 'hdbcompileserver', pid 3409742
[3409372]{-1}[-1/-1] 2025-12-09 21:31:06.203704 i Daemon            RunningInstance.cpp(00293) : Stop process 'hdbindexserver', pid 3409790, instance 43
[3409372]{-1}[-1/-1] 2025-12-09 21:31:06.203718 i Daemon            Utilities.cpp(00605) : Stopped child, pid 3409790 with SIGTERM
[3409372]{-1}[-1/-1] 2025-12-09 21:31:06.203719 i Daemon            Events.cpp(00133) : "KillInstanceEvent" is due in 270000 ms. Program 'hdbindexserver', pid 3409790
[3409372]{-1}[-1/-1] 2025-12-09 21:31:06.203720 i Daemon            RunningInstance.cpp(00293) : Stop process 'hdbindexserver', pid 3409793, instance 40
[3409372]{-1}[-1/-1] 2025-12-09 21:31:06.203732 i Daemon            Utilities.cpp(00605) : Stopped child, pid 3409793 with SIGTERM
[3409372]{-1}[-1/-1] 2025-12-09 21:31:06.203733 i Daemon            Events.cpp(00133) : "KillInstanceEvent" is due in 270000 ms. Program 'hdbindexserver', pid 3409793

nameserver_xxxx-l01.30001.004.trc

[3409426]{-1}[-1/-1] 2025-12-09 21:31:06.274053 i Service_Shutdown TrexService.cpp(00621) : Preparing for shutting service down
[4136048]{-1}[-1/-1] 2025-12-09 21:31:06.274102 i Service_Shutdown TREXIndexServer.cpp(02432) : Preparing to shutdown
[4136048]{-1}[-1/-1] 2025-12-09 21:31:06.274192 i Service_Shutdown TREXIndexServer.cpp(02495) : Stopping data backup

indexserver_xxxx-l01.30043.000.trc

[3409988]{-1}[-1/-1] 2025-12-09 21:31:06.245266 i Service_Shutdown TrexService.cpp(00621) : Preparing for shutting service down
[4135238]{-1}[-1/-1] 2025-12-09 21:31:06.245314 i Service_Shutdown TREXIndexServer.cpp(02432) : Preparing to shutdown
[4135238]{-1}[-1/-1] 2025-12-09 21:31:06.245396 i Service_Shutdown TREXIndexServer.cpp(02495) : Stopping data backup
[4135238]{-1}[-1/-1] 2025-12-09 21:31:06.246484 i assign           TREXIndexServer.cpp(02502) : unassign from volume 2
[4135238]{-1}[-1/-1] 2025-12-09 21:31:06.246486 i Service_Shutdown TREXIndexServer.cpp(02506) : prepare shutdown persistence manager ...

/var/log/messages

Dec  9 21:31:01 xxxx-l01 systemd[1]: Created slice User Slice of UID 0.
Dec  9 21:31:02 xxxx-l01 systemd[1]: Starting User Runtime Directory /run/user/0...
Dec  9 21:31:02 xxxx-l01 pacemaker-controld[13759]: error: Result of monitor operation for SAPHana_xxxx_00 on xxxx-l01: Timed Out after 59s (Resource agent did not complete within 59s)

pacemaker_logs

Dec 09 21:31:02.494 xxxx-l01 pacemaker-execd     [13756] (async_action_complete) info: Resource agent SAPHana_xxxx_00_monitor_61000[1800924] timed out after 59s
Dec 09 21:31:02.495 xxxx-l01 pacemaker-controld  [13759] (log_executor_event)     error: Result of monitor operation for SAPHana_xxxx_00 on xxxx-l01: Timed Out after 59s (Resource agent did not complete within 59s) | graph action unconfirmed; call=146 key=SAPHana_xxxx_00_monitor_61000
Dec 09 21:31:02.495 xxxx-l01 pacemaker-based     [13754] (forward_request)         info: Forwarding cib_modify operation for section status to all (origin=xxxx-l01/crmd/165)
Dec 24 08:40:32.886 xxxx-l01 pacemaker-execd     [13756] (async_action_complete)  info: Resource agent SAPHana_xxxx_00_monitor_61000[3283199] timed out after 59s
Dec 24 08:40:32.886 xxxx-l01 pacemaker-controld  [13759] (log_executor_event)     error: Result of monitor operation for SAPHana_xxxx_00 on xxxx-l01: Timed Out after 59s (Resource agent did not complete within 59s) | graph action unconfirmed; call=206 key=SAPHana_xxxx_00_monitor_61000

Sap Hana Platform Edition 2.0

HANA database failover, secondary node timeout, monitor operation timed out, Daemon server shutdown, Red Hat Linux, system replication, high availability, SIGINT, resource agent timeout, cluster monitor operation. , KBA , HAN-DB-HA , SAP HANA High Availability (System Replication, DR, etc.) , BC-OP-LNX-RH , Red Hat Linux , Problem