3490822 - HADR Message: 4002 Login failed on DR_maint for remote connections (with R1 or R2) - SRS

1. After failover from site1 to site2, running "sap_host_available site1" failed with following error on RMA log,

xxxx-06-30 03:07:39 [ERROR] [Thread - 4:1159][RMA] - [ResumeDSI                ] - Task Resume DSI Connection - Failed with error message - The Replication Server connection to '<SID>_site1_R2.master' failed to become active.
xxxx-06-30 03:07:39 [ERROR] [Thread - 4:1159][RMA] - [ResumeDSI                ] - Task Resume DSI Connection - Failed performing command - admin who, no_trunc
xxxx-06-30 03:07:39 [INFO ] [Thread - 4:1159][RMA] - [ResumeDSI                ] - Task Resume DSI Connection - Provided corrective suggestion - Use ISQL to connect to the Replication Server and issue the command 'resume connection to <SID>_site1_R2.master'. Make sure that the connection is active then resubmit the failing command.

2. <SID>_site1_R2.master is a remote connection between site1 ASE and site2 SRS. Check site2 SRS log, there is following error,

I. xxxx/06/30 03:07:28. The DSI thread for database '<SID>_site1_R2.master' is started.
I. xxxx/06/30 03:07:28. The DSI thread for database '<SID>_site1_R2.PL1' is started. 
E. xxxx/06/30 03:07:28. ERROR #1028 DSI(106 <SID>_site1_R2.master) - /generic/useful/cm.c(5369)
    Message from server: Message: 4002, State 1, Severity 14 -- 'Login failed.
'.
E. xxxx/06/30 03:07:28. ERROR #1027 DSI(106 <SID>_site1_R2.master) - /generic/useful/cm.c(5369)
    Open Client Client-Library error: Error: 67175468, Severity 4 -- 'ct_connect(): protocol specific layer: external error: The attempt to connect to the server failed.'.
E. xxxx/06/30 03:07:28. ERROR #13045 DSI(106 <SID>_site1_R2.master) - /generic/useful/cm.c(5373)
    Failed to connect to server '<SID>_site1_R2' as user 'DR_maint'. See CT-Lib and/or server error messages for more information.
E. xxxx/06/30 03:07:28. ERROR #1028 DSI(110 <SID>_site1_R2.<SID>) - /generic/useful/cm.c(5369)
    Message from server: Message: 4002, State 1, Severity 14 -- 'Login failed.
'.
I. xxxx/06/30 03:07:28. The DSI thread for database '<SID>_site1_R2.master' is shutdown. 
E. xxxx/06/30 03:07:28. ERROR #1027 DSI(110 <SID>_site1_R2.<SID>) - /generic/useful/cm.c(5369)
    Open Client Client-Library error: Error: 67175468, Severity 4 -- 'ct_connect(): protocol specific layer: external error: The attempt to connect to the server failed.'.
E. xxxx/06/30 03:07:28. ERROR #13045 DSI(110 <SID>_site1_R2.PL1) - /generic/useful/cm.c(5373)
    Failed to connect to server '<SID>_site1_R2' as user 'DR_maint'. See CT-Lib and/or server error messages for more information.
I. xxxx/06/30 03:07:28. The DSI thread for database '<SID>_site1_R2.<SID>' is shutdown. 

3. This kind of error could also happens when running 'sap_enable_replication', as these operations will bring up remote connections up temporarily to perform some specific task, then bring it down afterwards. 

• Adaptive Server Enterprise Always-on (HADR) 16.0 

ERROR #1028, Message: 4002, DR_maint, remote connection, sap_host_availabe, sap_enable_replication , KBA , BC-SYB-REP , Sybase Replication Server (standalone) , BC-SYB-REP-SAP , Replication with SAP Suite / SAP BW , Problem