3274590 - Frequent 'rc=111: Connection refused' errors occur between nameserver and indexserver

• Hanging network channel communication is occurring between the SYSTEMDB's nameserver using remote virtual ip address and port 31001 "xxx.xx.2.84/31001" and the tenant database indexserver on the local physical host address "xxx.xx.2.83/57848_tcp"
  
• Nameserver trace shows no issues and is responsive at the time of the connection refusal errors on the indexserver.
• Tenant indexserver traces (indexserver_saphana.31040.032.trc) show repeated failures to ping and receive acknowledgement from the nameserver :
  
  

'local=xxx.xx.2.83/57848_tcp, remote=xxx.xx.2.84/31001_tcp, state=ConnectWait, pending=[----]'

[20832]\{200065\}[15/1325431511] 2022-11-04 13:53:46.280552 e commlib          commlibImpl.cpp(00960) :  - 0: public ltt::exception@0x7fff76a8f140
[20832]\{200065\}[15/1325431511] 2022-11-04 13:53:46.299761 e commlib          commlibError.cpp(00267) : Internal Error Details. Basis/IO/Stream/impl/NetworkChannel.cpp:2483: System error: SO_ERROR has pending error for socket. rc=111: Connection refused. channel={<NetworkChannel>={<NetworkChannelBase>={this=562948320404376, fd=45, refCnt=1, local=xxx.xx.2.83/57848_tcp, remote=xxx.xx.2.84/31001_tcp, state=ConnectWait, pending=[----]}}}; $Context$=[7c9ef9ee02023a0c,xxx.xx.2.83:57848,xxx.xx.2.84:31001,TRN,0]
[20832]\{200065\}[15/1325431511] 2022-11-04 13:53:46.299796 e TrexNet          EndPoint.cpp(00255) : ERROR: failed to open channel xxx.xx.2.84:31001! reason: (internal error)
details: 
[20832]\{200065\}[15/1325431511] 2022-11-04 13:53:46.302064 e commlib          commlibImpl.cpp(00960) : ERROR: comm::connect to Host: xxx.xx.2.84, port: 31001, Error: exception  1: no.2110017  (Basis/IO/Stream/impl/NetworkChannel.cpp:2483) TID: 20832
    System error: SO_ERROR has pending error for socket. rc=111: Connection refused. channel={<NetworkChannel>={<NetworkChannelBase>={this=562948320415128, fd=45, refCnt=1, local=xxx.xx.2.83/57868_tcp, remote=xxx.xx.2.84/31001_tcp, state=ConnectWait, pending=[----]}}}; $Context$=[7c9ef9ee02023a0e,xxx.xx.2.83:57868,xxx.xx.2.84:31001,TRN,0]
exception throw location:
 1: 0x00007fff771f0b88 in .LTHUNK33.lto_priv.1+0x5d8 at NetworkChannel.cpp:2483 (libhdbbasis.so)
 2: 0x00007fff771c547c in Stream::NetworkChannelManager::connect(NetworkAccess::NetworkAddress const*, NetworkAccess::NetworkAddress const&, Stream::NetworkChannelParameters const&, bool, ltt::smartptr_handle<Stream::ChannelCallback>*, int, Stream::NetworkChannelComponent)+0x40c at NetworkChannelManager.cpp:190 (libhdbbasis.so)
 3: 0x00007fff771c5a34 in Stream::NetworkChannelManager::connect(NetworkAccess::NetworkAddress const*, NetworkAccess::NetworkAddress const&, Stream::NetworkChannelParameters const&, ltt::smartptr_handle<Stream::ChannelCallback>*, int, Stream::NetworkChannelComponent)+0x34 at NetworkChannelManager.cpp:128 (libhdbbasis.so)
[5229]\{200067\}[14/1325431515] 2022-11-04 13:53:46.298036 e commlib          commlibImpl.cpp(00960) : 12: 0x00007fff87be8ca4 in ptime::QueryExecutor::alter_system(ptime::qp_alter_system const*, char const*)+0x874 at qx_sys.cc:428 (libhdbrskernel.so)
13: 0x00007fff87e7e3a8 in ptime::QueryExecutor::execute(ptime::Transaction&, char const*, ptime::qp_parse_tree const*, ptime::Query::Plan*, ptime::Query::result_t*, ptime::QueryExecutor*)+0x718 at qx.cc:1597 (libhdbrskernel.so)
14: 0x00007fff86d917d4 in ptime::Query::execute(ptime::Transaction&, char const*, ptime::Query::param_t*, ptime::Query::Plan*, ptime::Query::result_t*, ptime::Statement*, bool, bool, char const*, ptime::QueryExecutor*)+0x924 at query.cc:4731 (libhdbrskernel.so)
15: 0x00007fff869c7a58 in ptime::Statement::execute_(Execution::Context&, EAPI::CursorHoldability const&, bool, bool, bool)+0x698 at Statement.cc:2733 (libhdbrskernel.so)
16: 0x00007fff86af8008 in ptime::CallableStatement::execute(Execution::Context&, EAPI::CursorHoldability const&, bool, bool, bool, ptime::Statement::BatchProcessingState, bool, bool, bool)+0x6a8 at CallableStatement.cc:504 (libhdbrskernel.so)
17: 0x00007fff84e89c1c in .LTHUNK350.lto_priv.1+0x5c at CallableStatement.cc:120 (libhdbrskernel.so)
[20832]\{200065\}[15/1325431511] 2022-11-04 13:53:46.302064 e commlib          commlibImpl.cpp(00960) :  4: 0x00007fff7c41085c in comm::connect(void*, char const*, unsigned short, int, Crypto::Configuration*, Stream::NetworkChannelComponent)+0x2cc at commlibImpl.cpp:948 (libhdbbasement.so)
 5: 0x00007fff7c2c89d0 in TrexNet::Channel::open(char const*, char const*, ltt::smartptr_handle<Crypto::Configuration>&)+0x2a0 at Channel.cpp:355 (libhdbbasement.so)
 6: 0x00007fff7c2b88d8 in TrexNet::ServerRep::openNewChannel(char const*, char const*, ltt::smartptr_handle<Crypto::Configuration>&)+0xc8 at EndPoint.cpp:238 (libhdbbasement.so)
 7: 0x00007fff7c2a15c4 in TrexNet::Requestor::getChannel(char const*, char const*, unsigned short, char, ltt::smartptr_handle<Crypto::Configuration>&)+0x124 at Requestor.cpp:168 (libhdbbasement.so)
 8: 0x00007fff7c2a5340 in TrexNet::Request::Request(char const*, TRexUtils::HostAndPort const&, char, ltt::smartptr_handle<Crypto::Configuration>)+0x2d0 at Request.cpp:556 (libhdbbasement.so)
 9: 0x00007fff7c2a5e08 in TrexNet::Request::Request(char const*, char const*, unsigned short, char)+0x108 at Request.cpp:508 (libhdbbasement.so)
10: 0x00007fff7c243204 in TrexService::RemoteService::RequestGuard::RequestGuard(TrexService::RemoteService*, char const*, int)+0xb4 at RemoteService.cpp:230 (libhdbbasement.so)
11: 0x00007fff7c225250 in TrexService::RemoteService::getconfighost(ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&)+0x80 at RemoteService.cpp:1942 (libhdbbasement.so)
12: 0x00007fff88036b1c in DDLHandler.cc.db14790c+0x104 at IniFileContentMonitor.cc:506 (libhdbrskernel.so)
13: 0x00007fff880385c4 in ptime::IniFileContentMonitorHandle::create_objects(ptime::mo_Env*, ptime::Monitor::ListEntry*, ptime::fastvector<bool, 16ul> const&, ptime::Monitor::LimitInfo const*)+0x24 at IniFileContentMonitor.h:106 (libhdbrskernel.so)
14: 0x00007fff876bf06c in ptime::Monitor_scan::do_open(ptime::OperatorEnv&, ptime::QEParams, int) const+0x4dc at qe_monitor_scan.cc:85 (libhdbrskernel.so)
15: 0x00007fff8777da50 in ptime::Table::open(ptime::Env&, ptime::QEParams, int) const+0x230 at qe_table.cc:234 (libhdbrskernel.so)
16: 0x00007fff876e9530 in ptime::Join_cpbtree_index::do_open(ptime::OperatorEnv&, ptime::QEParams, int) const+0xb0 at qe_base.h:639 (libhdbrskernel.so)
[5229]\{200067\}[14/1325431515] 2022-11-04 13:53:46.298036 e commlib          commlibImpl.cpp(00960) : 18: 0x00007fff84e89b70 in .LTHUNK352.lto_priv.1+0x30 at CallableStatement.cc:125 (libhdbrskernel.so)
[20832]\{200065\}[15/1325431511] 2022-11-04 13:53:46.302064 e commlib          commlibImpl.cpp(00960) : 17: 0x00007fff8777da50 in ptime::Table::open(ptime::Env&, ptime::QEParams, int) const+0x230 at qe_table.cc:234 (libhdbrskernel.so)
18: 0x00007fff877312d8 in ptime::Table::open_child(ptime::Env&, ptime::Table const*, ptime::QEParams&, int, bool) const+0x68 at qe_base.h:639 (libhdbrskernel.so)
[5229]\{200067\}[14/1325431515] 2022-11-04 13:53:46.298036 e commlib          commlibImpl.cpp(00960) : 19: 0x00007fff77cab310 in EAPI::CallableStatementHandle::executeUpdate(Execution::Context&, EAPI::StatementProperty const&)+0x90 at CallableStatementHandle.cpp:131 (libhdbeapi.so)
20: 0x00007fff84cf41d8 in sqlscript::ExecuteFunction<int, EAPI::CallableStatementHandle>::executePreparedStatement(EAPI::CallableStatementHandle const&, EAPI::StatementProperty&, int (EAPI::CallableStatementHandle::*)(Execution::Context&, EAPI::StatementProperty const&)) [clone .constprop.0]+0x138 at se_eapi_proxy.cc:602 (libhdbrskernel.so)
21: 0x00007fff86e53250 in sqlscript::SeCallableStatementProxy::executeUpdate(EAPI::CallableStatementHandle const&, EAPI::StatementProperty&)+0x1a0 at se_eapi_proxy.cc:682 (libhdbrskernel.so)
22: 0x00007fff7da65b1c in sqlscript::SeFacade::execExecuteUpdate(EAPI::CallableStatementHandle const&, EAPI::StatementProperty&)+0x3c at SeFacade.cpp:108 (libhdbsqlscriptutil.so)
23: 0x00007fff899df4a8 in sqlscript::exe::Statement::executeUpdate(EAPI::StatementProperty&)+0x48 at EAPI.cpp:144 (libhdbsqlscriptexecutor.so)
24: 0x00007fff89a1d518 in sqlscript::(anonymous namespace)::operate_(sqlscript::SeOpSQLDynamic const&, ltt::vector<ltt::shared_ptr<sqlscript::SeArg, ltt::default_deleter, ltt::integral_constant<bool, false>, ltt::integral_constant<bool, false> > > const&, ltt::shared_ptr<sqlscript::Context, ltt::default_deleter, ltt::integral_constant<bool, false>, ltt::integral_constant<bool, false> > const&, sqlscript::SeOpRuntime*) [clone .lto_priv.0]+0x16d8 at SeOperatorImpl.cpp:1387 (libhdbsqlscriptexecutor.so)
25: 0x00007fff89a0d1fc in sqlscript::SeOperatorImpl::operate(sqlscript::SeOp const&, ltt::vector<ltt::shared_ptr<sqlscript::SeArg, ltt::default_deleter, ltt::integral_constant<bool, false>, ltt::integral_constant<bool, false> > > const&, ltt::shared_ptr<sqlscript::Context, ltt::default_deleter, ltt::integral_constant<bool, false>, ltt::integral_constant<bool, false> > const&, sqlscript::SeOpRuntime*)+0x1c8c at SeOperatorImpl.cpp:3317 (libhdbsqlscriptexecutor.so)
26: 0x00007fff89b139d0 in sqlscript::SeEvaluatorImpl::eval(ltt::smart_ptr<sqlscript::SeOp, ltt::integral_constant<bool, false>, ltt::integral_constant<bool, false> > const&, ltt::vector<ltt::shared_ptr<sqlscript::SeArg, ltt::default_deleter, ltt::integral_constant<bool, false>, ltt::integral_constant<bool, false> > > const&, sqlscript::exe::Env&, sqlscript::SeOpRuntime&)+0xd0 at SeEvaluatorImpl.cpp:606 (libhdbsqlscriptexecutor.so)
exception type information:
 - 0: public ltt::exception@0x7fff76a8f140
[5229]\{200067\}[14/1325431515] 2022-11-04 13:53:46.311506 e commlib          commlibError.cpp(00267) : Internal Error Details. Basis/IO/Stream/impl/NetworkChannel.cpp:2483: System error: SO_ERROR has pending error for socket. rc=111: Connection refused. channel={<NetworkChannel>={<NetworkChannelBase>={this=562938957562392, fd=45, refCnt=1, local=xxx.xx.2.83/57858_tcp, remote=xxx.xx.2.84/31001_tcp, state=ConnectWait, pending=[----]}}}; $Context$=[7c9ef9ee02023a0d,xxx.xx.2.83:57858,xxx.xx.2.84:31001,TRN,0]
[5229]\{200067\}[14/1325431515] 2022-11-04 13:53:46.311540 e TrexNet          EndPoint.cpp(00255) : ERROR: failed to open channel xxx.xx.2.84:31001! reason: (internal error)
details: 

• telnet tests between the addresses and ports reveal the following ... (In this example, the physical_hostname is the IP with 'xxx.xx.2.83' and hostname 'saphana' is the assigned service name (IP with 'xxx.xx.2.83') of the same host.)

sidadm@physical_hostname:/usr/sap/SID/HDB00> telnet xxx.xx.2.83 31001
Trying xxx.xx.2.83...
telnet: connect to address xxx.xx.2.83: Connection refused
sidadm@physical_hostname:/usr/sap/SID/HDB00> telnet xxx.xx.2.84 31001
Trying xxx.xx.2.84...
telnet: connect to address xxx.xx.2.84: Connection refused
sidadm@physical_hostname:/usr/sap/SID/HDB00> telnet xxx.xx.2.84 31040
Trying xxx.xx.2.84...
Connected to xxx.xx.2.84.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
sidadm@physical_hostname:/usr/sap/SID/HDB00> telnet xxx.xx.2.84 31013
Trying xxx.xx.2.84...
Connected to xxx.xx.2.84.

SAP HANA , platform edition 2.0

SAP HANA , platform edition 1.0

KBA , HAN-DB , SAP HANA Database , Problem