3613351 - Failover happens from one site to another in HA environment with unknown cause

Site 1 and site 2 both are encountering network issues and not anything suspicious on indexserver traces.

nameserver_xxx01.trc (site1)

[49067]{-1}[-1/-1] 2025-04-05 11:44:30.894714 e commlib          commlibImpl.cpp(00960) : ERROR: comm::connect to Host: 127.0.0.1, port: 30203, Error: exception  1: no.2110017  (Basis/IO/Stream/impl/NetworkChannel.cpp:2483) TID: 49067
    System error: SO_ERROR has pending error for socket. rc=111: Connection refused. channel={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139639200618520, fd=40, refCnt=1, local=127.0.0.1/46274_tcp, remote=127.0.0.1/30203_tcp, state=ConnectWait, pending=[----]}}}; $Context$=[7f2c58734a1d0012,127.0.0.1:46274,127.0.0.1:30203,TRN,0]; $Context$=[7f2c58734a1d0012,127.0.0.1:46274,127.0.0.1:30203,TRN,0]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139639200618520, fd=INVALID, refCnt=1, local=127.0.0.1/46274_tcp, remote=127.0.0.1/30203_tcp, state=Closed, pending=[----]}}}
exception throw location:
 1: 0x00007f01ad7e792d in .LTHUNK33.lto_priv.1+0x4e9 at NetworkChannel.cpp:2483 (libhdbbasis.so)
[49068]{-1}[-1/-1] 2025-04-05 11:44:30.895425 e commlib          commlibImpl.cpp(00960) : ERROR: comm::connect to Host: 127.0.0.1, port: 30207, Error: exception  1: no.2110017  (Basis/IO/Stream/impl/NetworkChannel.cpp:2483) TID: 49068
    System error: SO_ERROR has pending error for socket. rc=111: Connection refused. channel={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139639063779352, fd=41, refCnt=1, local=127.0.0.1/57242_tcp, remote=127.0.0.1/30207_tcp, state=ConnectWait, pending=[----]}}}; $Context$=[7f2c58734a1d0015,127.0.0.1:57242,127.0.0.1:30207,TRN,0]; $Context$=[7f2c58734a1d0015,127.0.0.1:57242,127.0.0.1:30207,TRN,0]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139639063779352, fd=INVALID, refCnt=1, local=127.0.0.1/57242_tcp, remote=127.0.0.1/30207_tcp, state=Closed, pending=[----]}}}
exception throw location:
 1: 0x00007f01ad7e792d in .LTHUNK33.lto_priv.1+0x4e9 at NetworkChannel.cpp:2483 (libhdbbasis.so)
[49067]{-1}[-1/-1] 2025-04-05 11:44:30.894714 e commlib          commlibImpl.cpp(00960) :  2: 0x00007f01ad7df6d7 in Stream::NetworkChannelSSLFilter::initiateConnection()+0x23 at NetworkChannelSSLFilter.cpp:196 (libhdbbasis.so)
[49069]{-1}[-1/-1] 2025-04-05 11:44:30.896250 e commlib          commlibImpl.cpp(00960) : ERROR: comm::connect to Host: 127.0.0.1, port: 30211, Error: exception  1: no.2110017  (Basis/IO/Stream/impl/NetworkChannel.cpp:2483) TID: 49069
    System error: SO_ERROR has pending error for socket. rc=111: Connection refused. channel={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139638657194008, fd=41, refCnt=1, local=127.0.0.1/42334_tcp, remote=127.0.0.1/30211_tcp, state=ConnectWait, pending=[----]}}}; $Context$=[7f2c58734a1d0014,127.0.0.1:42334,127.0.0.1:30211,TRN,0]; $Context$=[7f2c58734a1d0014,127.0.0.1:42334,127.0.0.1:30211,TRN,0]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139638657194008, fd=INVALID, refCnt=1, local=127.0.0.1/42334_tcp, remote=127.0.0.1/30211_tcp, state=Closed, pending=[----]}}}
exception throw location:
 1: 0x00007f01ad7e792d in .LTHUNK33.lto_priv.1+0x4e9 at NetworkChannel.cpp:2483 (libhdbbasis.so)
[49068]{-1}[-1/-1] 2025-04-05 11:44:30.895425 e commlib          commlibImpl.cpp(00960) :  2: 0x00007f01ad7df6d7 in Stream::NetworkChannelSSLFilter::initiateConnection()+0x23 at NetworkChannelSSLFilter.cpp:196 (libhdbbasis.so)
[49069]{-1}[-1/-1] 2025-04-05 11:44:30.896250 e commlib          commlibImpl.cpp(00960) :  2: 0x00007f01ad7df6d7 in Stream::NetworkChannelSSLFilter::initiateConnection()+0x23 at NetworkChannelSSLFilter.cpp:196 (libhdbbasis.so)
[49067]{-1}[-1/-1] 2025-04-05 11:44:30.894714 e commlib          commlibImpl.cpp(00960) :  3: 0x00007f01ad7df78e in Stream::NetworkChannelSSLFilter::initClientChannel()+0xa at NetworkChannelSSLFilter.cpp:72 (libhdbbasis.so)
[49069]{-1}[-1/-1] 2025-04-05 11:44:30.896250 e commlib          commlibImpl.cpp(00960) :  3: 0x00007f01ad7df78e in Stream::NetworkChannelSSLFilter::initClientChannel()+0xa at NetworkChannelSSLFilter.cpp:72 (libhdbbasis.so)

nameserver_xx02.trc (site2)

[63621]{-1}[-1/-1] 2025-04-05 20:47:13.023530 e commlib          commlibImpl.cpp(00960) : ERROR: comm::connect to Host: 127.0.0.1, port: 30203, Error: exception  1: no.2110017  (Basis/IO/Stream/impl/NetworkChannel.cpp:2483) TID: 63621
    System error: SO_ERROR has pending error for socket. rc=111: Connection refused. channel={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139951548475416, fd=39, refCnt=1, local=127.0.0.1/40404_tcp, remote=127.0.0.1/30203_tcp, state=ConnectWait, pending=[----]}}}; $Context$=[47d896825a1d0012,127.0.0.1:40404,127.0.0.1:30203,TRN,0]; $Context$=[47d896825a1d0012,127.0.0.1:40404,127.0.0.1:30203,TRN,0]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139951548475416, fd=INVALID, refCnt=1, local=127.0.0.1/40404_tcp, remote=127.0.0.1/30203_tcp, state=Closed, pending=[----]}}}
exception throw location:
 1: 0x00007f4a331fe92d in .LTHUNK33.lto_priv.1+0x4e9 at NetworkChannel.cpp:2483 (libhdbbasis.so)
[63622]{-1}[-1/-1] 2025-04-05 20:47:13.024175 e commlib          commlibImpl.cpp(00960) : ERROR: comm::connect to Host: 127.0.0.1, port: 30207, Error: exception  1: no.2110017  (Basis/IO/Stream/impl/NetworkChannel.cpp:2483) TID: 63622
    System error: SO_ERROR has pending error for socket. rc=111: Connection refused. channel={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139953461274648, fd=41, refCnt=1, local=127.0.0.1/51512_tcp, remote=127.0.0.1/30207_tcp, state=ConnectWait, pending=[----]}}}; $Context$=[47d896825a1d0015,127.0.0.1:51512,127.0.0.1:30207,TRN,0]; $Context$=[47d896825a1d0015,127.0.0.1:51512,127.0.0.1:30207,TRN,0]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139953461274648, fd=INVALID, refCnt=1, local=127.0.0.1/51512_tcp, remote=127.0.0.1/30207_tcp, state=Closed, pending=[----]}}}
exception throw location:
 1: 0x00007f4a331fe92d in .LTHUNK33.lto_priv.1+0x4e9 at NetworkChannel.cpp:2483 (libhdbbasis.so)
[63621]{-1}[-1/-1] 2025-04-05 20:47:13.023530 e commlib          commlibImpl.cpp(00960) :  2: 0x00007f4a331f66d7 in Stream::NetworkChannelSSLFilter::initiateConnection()+0x23 at NetworkChannelSSLFilter.cpp:196 (libhdbbasis.so)
[63623]{-1}[-1/-1] 2025-04-05 20:47:13.024976 e commlib          commlibImpl.cpp(00960) : ERROR: comm::connect to Host: 127.0.0.1, port: 30211, Error: exception  1: no.2110017  (Basis/IO/Stream/impl/NetworkChannel.cpp:2483) TID: 63623
    System error: SO_ERROR has pending error for socket. rc=111: Connection refused. channel={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139950944716824, fd=41, refCnt=1, local=127.0.0.1/40404_tcp, remote=127.0.0.1/30211_tcp, state=ConnectWait, pending=[----]}}}; $Context$=[47d896825a1d0014,127.0.0.1:40404,127.0.0.1:30211,TRN,0]; $Context$=[47d896825a1d0014,127.0.0.1:40404,127.0.0.1:30211,TRN,0]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139950944716824, fd=INVALID, refCnt=1, local=127.0.0.1/40404_tcp, remote=127.0.0.1/30211_tcp, state=Closed, pending=[----]}}}
exception throw location:
 1: 0x00007f4a331fe92d in .LTHUNK33.lto_priv.1+0x4e9 at NetworkChannel.cpp:2483 (libhdbbasis.so)
 2: 0x00007f4a331f66d7 in Stream::NetworkChannelSSLFilter::initiateConnection()+0x23 at NetworkChannelSSLFilter.cpp:196 (libhdbbasis.so)
[63622]{-1}[-1/-1] 2025-04-05 20:47:13.024175 e commlib          commlibImpl.cpp(00960) :  2: 0x00007f4a331f66d7 in Stream::NetworkChannelSSLFilter::initiateConnection()+0x23 at NetworkChannelSSLFilter.cpp:196 (libhdbbasis.so)
[63624]{-1}[-1/-1] 2025-04-05 20:47:13.025721 e commlib          commlibImpl.cpp(00960) : ERROR: comm::connect to Host: 127.0.0.1, port: 30240, Error: exception  1: no.2110017  (Basis/IO/Stream/impl/NetworkChannel.cpp:2483) TID: 63624
    System error: SO_ERROR has pending error for socket. rc=111: Connection refused. channel={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139953056065560, fd=41, refCnt=1, local=127.0.0.1/59422_tcp, remote=127.0.0.1/30240_tcp, state=ConnectWait, pending=[----]}}}; $Context$=[47d896825a1d0017,127.0.0.1:59422,127.0.0.1:30240,TRN,0]; $Context$=[47d896825a1d0017,127.0.0.1:59422,127.0.0.1:30240,TRN,0]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139953056065560, fd=INVALID, refCnt=1, local=127.0.0.1/59422_tcp, remote=127.0.0.1/30240_tcp, state=Closed, pending=[----]}}}
exception throw location:
 1: 0x00007f4a331fe92d in .LTHUNK33.lto_priv.1+0x4e9 at NetworkChannel.cpp:2483 (libhdbbasis.so)
 2: 0x00007f4a331f66d7 in Stream::NetworkChannelSSLFilter::initiateConnection()+0x23 at NetworkChannelSSLFilter.cpp:196 (libhdbbasis.so)
[63621]{-1}[-1/-1] 2025-04-05 20:47:13.023530 e commlib          commlibImpl.cpp(00960) :  3: 0x00007f4a331f678e in Stream::NetworkChannelSSLFilter::initClientChannel()+0xa at NetworkChannelSSLFilter.cpp:72 (libhdbbasis.so)
[63623]{-1}[-1/-1] 2025-04-05 20:47:13.024976 e commlib          commlibImpl.cpp(00960) :  3: 0x00007f4a331f678e in Stream::NetworkChannelSSLFilter::initClientChannel()+0xa at NetworkChannelSSLFilter.cpp:72 (libhdbbasis.so)
[63622]{-1}[-1/-1] 2025-04-05 20:47:13.024175 e commlib          commlibImpl.cpp(00960) :  3: 0x00007f4a331f678e in Stream::NetworkChannelSSLFilter::initClientChannel()+0xa at NetworkChannelSSLFilter.cpp:72 (libhdbbasis.so)
[63624]{-1}[-1/-1] 2025-04-05 20:47:13.025721 e commlib          commlibImpl.cpp(00960) :  3: 0x00007f4a331f678e in Stream::NetworkChannelSSLFilter::initClientChannel()+0xa at NetworkChannelSSLFilter.cpp:72 (libhdbbasis.so)

SAP HANA PLATFORM EDITION 2.0

Network, Failover, Replication, local host, commlib           , KBA , HAN-DB-HA , SAP HANA High Availability (System Replication, DR, etc.) , HAN-DB , SAP HANA Database , Problem