3230326 - ABAP Central Services and Enqueue replication server failing to connect with HANA after failover in the Linux cluster group

• SAP ABAP Central Services and Enqueue replication servers are not starting up, after switching ASCS and ERS in the local linux cluster group. (ABAP Central Services (ASCS) & ERS 'Enqueue replication server')  
    

    hostname:sidadm 77> tail sapstartsrv.log
    SAP HA Trace: Fire system command /usr/bin/sap_cluster_connector lsr ...
    SAP HA Trace: searchClusterFile: SID:00 found
    SAP HA Trace: Sun Jul 17 09:13:55 2022
    SAP HA Trace: --- SAP_HA_FindSAPInstance Exit-Code: SAP_HA_DISABLED ---
    [Thr 47435229021952] Sun Jul 17 09:14:33 2022
    Instance status:
    msg_server, MessageServer, GRAY, Stopped, , , 7729
    enserver, EnqueueServer, GRAY, Stopped, , , 7742
    sapwebdisp, Web Dispatcher, GRAY, Stopped, , , 7751
    gwrd, Gateway, GRAY, Stopped, , , 7752

• 'R3Trans -d' is still failing to connect via multiple application instances with the HANA DB as output from the trans.log file : 
    
  

    hostname:sidadm 53> cat trans.log
     4 ETW000 R3trans version 6.26 (release 753 - 19.07.21 - 16:58:42).
     4 ETW000 unicode enabled version
     4 ETW000 ===============================================
     4 ETW000
     4 ETW000 date&time   : 17.07.2022 - 09:53:59
     4 ETW000 control file: <no ctrlfile>
     4 ETW000 R3trans was called as follows: R3trans -d
     4 ETW000  trace at level 1 opened for a given file pointer
     4 ETW000  [     dev trc,00000]  Sun Jul 17 09:53:59 2022
     4 ETW000  [     dev trc,00000]  Loading DB library '/usr/sap/SID/SYS/exe/run/dbhdbslib.so' ...
     4 ETW000  [     dev trc,00000]  Sun Jul 17 09:54:00 2022
     4 ETW000  [     dev trc,00000]  Library '/usr/sap/SID/SYS/exe/run/dbhdbslib.so' loaded
     4 ETW000  [     dev trc,00000]  Version of '/usr/sap/SID/SYS/exe/run/dbhdbslib.so' is "753.02", patchlevel (0.500)
     4 ETW000  [     dev trc,00000]  switch DBSL TRACE LEVEL from 3 to 1
     4 ETW000  [     dev trc,00000]  DBHDBSLIB : version 753.02, patch 0.500 (Make PL 0.500)
  
     . . . . . . . . . 
  
     4 ETW000  [     dev trc,00000]  Loading SQLDBC client runtime (pid=14394)...
     4 ETW000  [     dev trc,00000]  SQLDBC Module  : /usr/sap/SID/SYS/exe/uc/linuxx86_64/hdbclient/libSQLDBCHDB.so
     4 ETW000  [     dev trc,00000]  SQLDBC Runtime : libSQLDBCHDB 2.07.023.1612823292
     4 ETW000  [     dev trc,00000]  SQLDBC client runtime is 2.07.023.1612823292
     4 ETW000  [     dev trc,00000]  Try to connect via secure store (DEFAULT) on connection 0 ...
     4 ETW000  [     dev trc,00000]  connect property [APPLICATION = ABAP:SID]
     4 ETW000  [     dev trc,00000]  connect property [APPLICATIONVERSION = 753 PL 500]
     4 ETW000  [     dev trc,00000]  connect property [KEY = DEFAULT]
     4 ETW000  [     dev trc,00000]  connect property [SQLMODE = SAPR3]
     4 ETW000  [     dev trc,00000]  connect property [TIMEOUT = 0]
     4 ETW000  [     dev trc,00000]  connect property [UNICODE = 1]
     4 ETW000  [     dev trc,00000]  connect property [PACKETSIZE = 1048576]
     4 ETW000  [     dev trc,00000]  connect property [COMPLETEARRAYEXECUTION = 1]
     4 ETW000  [     dev trc,00000]  connect property [VARCHARMODE = ABAP]
     4 ETW000  [     dev trc,00000]  connect property [CACHEDVIEWTS = 1]
     4 ETW000  [dbhdbsql.cpp,00000]  *** ERROR => Connect to database failed, rc=1, rcSQL=-10709
     4 ETW000  [     dev trc,00000]  SQLCODE    : -10709
     4 ETW000  [     dev trc,00000]  SQLERRTEXT : Connect failed (connect timeout expired)
     4 ETW000  [    dblink  ,00000]  ***LOG BY2=>sql error -10709 performing CON
     4 ETW000  [    dblink  ,00000]  ***LOG BY0=>Connect failed (connect timeout expired)
     2EETW169 no connect possible: "DBMS = HDB                              ---  SERVER = '' PORT = ''"

• The HANA system is up and running fine with all services in a 'GREEN' status when checking the process list via: '/usr/sap/hostctrl/exe/sapcontrol -nr <instance number> -function GetProcessList'. The SYSTEMDB and Tenant System IDs are open and available via HANA Studio/Cockpit overview.  No problems are observed with the availability of the HANA DB itself.

SAP HANA, platform edition 2.0

SAP HANA, platform edition 1.0

KBA , HAN-DB , SAP HANA Database , BC-CST-STS , Startup Service , Problem