2185955 - SAP HANA startup hangs due to orphan nameserver process

The startup of SAP HANA hangs.

One name server trace file contains errors like:

Responder.cpp(00568) : can't listen on port <ip_address>:3<inst_id>01: host unknown
Responder.cpp(00568) : can't listen on port <ip_address>:3<inst_id>02: host unknown

In scale-out landscapes other name server trace files contain errors like the following when trying to contact the above nameserver:

Channel.cpp(00362): active channel <id> from <id> to <ip_address>:3<inst_id>01: reading failed with timeout error; timeout=60000ms elapsed
BufferedIO.cpp(01151) : erroneous channel <id> from <id> to <id_address>:30101: read from channel failed; resetting buffer
30524[thr=137308]: TopologyReplicator at
 1: 0x00007fbf57f787d4 in Execution::ContextFunctions::dumpInfo(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&, bool, bool, bool, bool, bool)+0x2c0 at ContextFunctions.cpp:629 (libhdbbasis.so)
 2: 0x00007fbf598d0db4 in TrexNet::BufferedIO::fillBuffer(bool)+0x450 at BufferedIO.cpp:1152 (libhdbbasement.so)
 3: 0x00007fbf598d4baa in TrexNet::BufferedIO::peekBuffer()+0x46 at BufferedIO.cpp:1326 (libhdbbasement.so)
 4: 0x00007fbf598d4d52 in TrexNet::BufferedNetInput::peek()+0x30 at BufferedIO.cpp:4331 (libhdbbasement.so)
 5: 0x000000000041e5fa in TrexNet::Request::in()+0x36 at Request.h:392 (hdbnameserver)
 6: 0x00007fbf5937f430 in NameServer::TNSInfo::sendRequestTo(NameServer::Request const&, NameServer::Response&, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&, unsigned short, NameServer::TrexNetRequestHolder*)+0x110 at TNSClient.cpp:782 (libhdbcsbase.so)
 7: 0x00007fbf5937ff5a in NameServer::TNSClient::sendRequestTo(NameServer::Request const&, NameServer::Response&, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&, unsigned short, NameServer::TrexNetRequestHolder*)+0x26 at TNSClient.cpp:1021 (libhdbcsbase.so)
 8: 0x00007fbf5af98c18 in NameServer::TopologyReplicator::processRequests()+0x6d4 at Replicator.cpp:477 (libhdbns.so)
 9: 0x00007fbf5af99795 in NameServer::TopologyReplicator::run(void*)+0x61 at Replicator.cpp:410 (libhdbns.so)
10: 0x00007fbf599261f2 in TrexThreads::PoolThread::run()+0x850 at PoolThread.cpp:265 (libhdbbasement.so)
11: 0x00007fbf59927d58 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:124 (libhdbbasement.so)
12: 0x00007fbf57fc4ccf in Execution::Thread::staticMainImp(void**)+0x98b at Thread.cpp:475 (libhdbbasis.so)
13: 0x00007fbf57fc521d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:545 (libhdbbasis.so)
[137308]{-1}[-1/-1] 2015-06-27 04:28:12.312881 e TNS              TNSClient.cpp(00800) : sendRequest storetrees to <host_name>:30101 failed with NetException. data=
TopologyReplicator(<host>:30<inst_id>1) req <id> failed with error 5521, discarding request: method setstarting
TopologyReplicator(<host>:30<inst_id>1) req <id> failed with error 5521, discarding request: method storetrees

A thread of type "Assign" may be visible waiting for 'open data+log volume <id> / local tables'.

In the call stacks one thread may be visible waiting for a request sent to the nameserver:

8977[thr=137398]: Assign at
 1: 0x00007fa116dbebc6 in __GI___poll+0x62 (libc.so.6)
 2: 0x00007fa13555836d in SiPoll+0x19 at siux.cpp:3554 (libhdbbasement.so)
 3: 0x00007fa1355592ce in SiPeek+0x7a at siux.cpp:3657 (libhdbbasement.so)
 4: 0x00007fa135520cb0 in NiIPeek(NITAB*, unsigned char, int)+0x100 at nixxi.cpp:5619 (libhdbbasement.so)
 5: 0x00007fa135521481 in NiIRead(NITAB*, unsigned char*, unsigned int, int, unsigned char, unsigned int*)+0xf0 at nixxi.cpp:4819 (libhdbbasement.so)
 6: 0x00007fa1350cb5b8 in comm::read(void*, void*, int, int)+0x1d4 at commlib.cpp:1754 (libhdbbasement.so)
 7: 0x00007fa135167167 in TrexNet::Channel::receive(void*, int, int&)+0x2d3 at Channel.cpp:349 (libhdbbasement.so)
 8: 0x00007fa1351509ea in TrexNet::BufferedIO::fillBuffer(bool)+0x86 at BufferedIO.cpp:1144 (libhdbbasement.so)
 9: 0x00007fa135154baa in TrexNet::BufferedIO::peekBuffer()+0x46 at BufferedIO.cpp:1326 (libhdbbasement.so)
10: 0x00007fa135154d52 in TrexNet::BufferedNetInput::peek()+0x30 at BufferedIO.cpp:4331 (libhdbbasement.so)
11: 0x000000000046876a in TrexNet::Request::in()+0x36 at Request.h:392 (hdbindexserver)
12: 0x00007fa134bff430 in NameServer::TNSInfo::sendRequestTo(NameServer::Request const&, NameServer::Response&, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&, unsigned short, NameServer::TrexNetRequestHolder*)+0x110 at TNSClient.cpp:782 (libhdbcsbase.so)
13: 0x00007fa134c0149f in NameServer::TNSInfo::sendRequest(NameServer::Request const&, NameServer::Response&)+0x10b at TNSClient.cpp:711 (libhdbcsbase.so)
14: 0x00007fa134c01c4d in NameServer::TNSInfo::processRequest(NameServer::Request const&, NameServer::Response&)+0xe9 at TNSClient.cpp:673 (libhdbcsbase.so)
15: 0x00007fa134c02131 in NameServer::TNSClient::processRequest(NameServer::Request const&, NameServer::Response&)+0x40 at TNSClient.cpp:1045 (libhdbcsbase.so)
16: 0x00007fa134c2e5ab in NameServer::TNSClient::deleteIndex(TrexBase::IndexName const&)+0x167 at TNSClient.cpp:1730 (libhdbcsbase.so)
17: 0x00007fa13741fc20 in TRexAPI::DDL::DropTable::handle()+0x100 at DDL_DropTable.cpp:109 (libhdbcsapi.so)
18: 0x00007fa13741aaba in TRexAPI::DDLRequest::execute()+0x276 at DDLRequest.cpp:110 (libhdbcsapi.so)
19: 0x00007fa13741ffe0 in TRexAPI::DDL::DropTable::execute()+0x100 at DDL_DropTable.cpp:64 (libhdbcsapi.so)
20: 0x00007fa136f3884c in TRexAPI::TRexApiAdmin::deleteIndex(bool, bool)+0x1f8 at TRexApiAdmin.cpp:607 (libhdbcsapi.so)
21: 0x00007fa11a43d957 in ptime::TrexMD::deleteIndex(bool, bool) const+0x33 at trex_metadata.cc:287 (libhdbcswrapper.so)
22: 0x00007fa11c09b82f in ptime::QueryExecutor::drop_trex_table(ptime::ClassPtr<ptime::TableInfo>, bool, bool)+0xa2b at qx_trex.cc:6573 (libhdbrskernel.so)
23: 0x00007fa11bfb8949 in ptime::QueryExecutor::drop_table(ptime::qp_drop_table const*, ptime::ClassPtr<ptime::TableInfo>, ptime::ClassPtr<ptime::TableInfo>)+0xee5 at qx_table.cc:3193 (libhdbrskernel.so)
24: 0x00007fa11bfb9129 in ptime::QueryExecutor::drop_table(ptime::qp_drop_table const*, ptime::ClassPtr<ptime::TableInfo>)+0xd5 at qx_table.cc:3039 (libhdbrskernel.so)
25: 0x00007fa11bf9571e in ptime::QueryExecutor::execute(ptime::Transaction&, char const*, ptime::qp_parse_tree const*, ptime::Query::Plan*, ptime::Query::result_t*)+0x119a at qx.cc:234 (libhdbrskernel.so)
26: 0x00007fa11c9a72d0 in ptime::Query::execute(ptime::Transaction&, char const*, ptime::Query::param_t*, ptime::Query::Plan*, ptime::Query::result_t*, ptime::Statement*, bool)+0x290 at query.cc:522 (libhdbrskernel.so)
27: 0x000000000044e572 in TRexAPI::deleteTable(Newdb::Transaction&, TrexTrace::Tracer&, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&)+0xa0 at TREXIndexServer.cpp:1155 (hdbindexserver)
28: 0x00000000004513e2 in TRexAPI::TREXIndexServer::createTableBoePortalSearch(NameServer::TNSClient&)+0x240 at TREXIndexServer.cpp:1274 (hdbindexserver)
29: 0x000000000045956d in TRexAPI::TREXIndexServer::assign(NameServer::ServiceStartInfo&, bool, TREX_ERROR::TRexError&)+0x489 at TREXIndexServer.cpp:895 (hdbindexserver)
30: 0x000000000048b938 in TRexAPI::AssignThread::run(void*)+0x34 at TREXIndexServer.cpp:408 (hdbindexserver)
31: 0x00007fa1351a61f2 in TrexThreads::PoolThread::run()+0x850 at PoolThread.cpp:265 (libhdbbasement.so)
32: 0x00007fa1351a7d58 in TrexThreads::PoolThread::run(void*&)+0x14 at PoolThread.cpp:124 (libhdbbasement.so)
33: 0x00007fa12a6d0ccf in Execution::Thread::staticMainImp(void**)+0x98b at Thread.cpp:475 (libhdbbasis.so)
34: 0x00007fa12a6d121d in Execution::Thread::staticMain(void*)+0x39 at Thread.cpp:545 (libhdbbasis.so)

SAP HANA

old nameserver process running; , KBA , HAN-DB , SAP HANA Database , Problem