2951467 - Indexserver is shutdown because of a manual start of another Indexserver process

• You observe that the indexserver of a tenant database was shut down without any request;
• SAP HANA is installed as Multiple-Host System, with at least 2 worker nodes and 1 standby node;
• In the daemon trace file daemon_<hostname>.<port>.<context>.trc, you can only find similar entries as follows:

  [000000]{-1}[-1/-1] 2020-07-23 21:13:11.340616 i Daemon           DaemonHandle.cpp(00336) : Process 'hdbindexserver', pid 8747 exited normally with status 0

• In the nameserver trace file nameserver_<hostname>.<port>.<context>.trc, you can find similiar entries as follows:
  

  [00000]{-1}[-1/-1] 2020-07-23 21:04:00.931289 i Service_Shutdown TREXNameServer.cpp(05421) : setStopping(indexserver@<hostname>:<port>)
  [00000]{-1}[-1/-1] 2020-07-23 21:04:00.937705 i service_active_s TREXNameServer.cpp(04162) : Service active states (setStopping - <hostname>:<port>, watchdog=0):

• Within the SYSTEMDB trace directory at /usr/sap/<sid>/HDB<instane_nr>/<hostname>/trace you can find an indexserver trace file. In this trace file you can find entries similar to the following:
  

  [37592]{-1}[-1/-1] 2020-07-23 21:00:03.242077 e self_watchdog SelfWatchdog.cpp(00201) : Process 8747 started at 2020-06-05 17:25:44.091000 is in state S, waiting for its removal
  [37704]{-1}[-1/-1] 2020-07-21 21:00:04.128035 i Basis MachineTopology.cpp(01242) : Read initial topology, active logical cores are 0-159 and numa nodes with active cores are 0-3 and numa nodes with allowed memory are 0-3 and memory limit is 4127463112704
  [37592]{-1}[-1/-1] 2020-07-21 21:00:04.243816 e self_watchdog SelfWatchdog.cpp(00201) : Process 8747 started at 2020-07-05 17:25:44.083000 is in state S, waiting for its removal
  [37592]{-1}[-1/-1] 2020-07-21 21:00:05.245849 e self_watchdog SelfWatchdog.cpp(00201) : Process 8747 started at 2020-07-05 17:25:44.085000 is in state S, waiting for its removal
  [37592]{-1}[-1/-1] 2020-07-21 21:00:06.247855 e self_watchdog SelfWatchdog.cpp(00201) : Process 8747 started at 2020-07-05 17:25:44.087000 is in state S, waiting for its removal
  [37592]{-1}[-1/-1] 2020-07-21 21:00:07.249911 e self_watchdog SelfWatchdog.cpp(00201) : Process 8747 started at 2020-07-05 17:25:44.089000 is in state S, waiting for its removal
  [37592]{-1}[-1/-1] 2020-07-21 21:00:08.433742 e Stream NetworkChannel.cpp(01064) : [673443162ad0014,127.0.0.1:<indexserver port>,-,UNK,0] Cannot bind to a local address: tcp port <indexserver port> on 127.0.0.1 => socket[107285318], user ""<sid>adm:<UID>"
  [37592]{-1}[-1/-1] 2020-07-21 21:00:08.434219 e commlib commlibImpl.cpp(01008) : ERROR/Exception: comm::listen to Host: 127.0.0.1, port: <indexserver port>, Error: Error address in use: $msg$, rc=$sysrc$: $sysmsg$
  [37592]{-1}[-1/-1] 2020-07-21 21:00:08.434323 e TrexNet Responder.cpp(00349) : can't listen on port 127.0.0.1:<indexserver port>: (host unknown)
  details: Internal Error Details. Basis/IO/Stream/impl/NetworkChannel.cpp:1067: Error address in use: $msg$, rc=98: Address already in use; $[1]$=NetworkChannelBase::bindLocal. bind failed; $Context$=[673443162ad0014,127.0.0.1:<indexserver port>,-,UNK,0]; $channel$={<NetworkChannelBase>={this=140377406608776, fd=31, refCnt=1, local=127.0.0.1/<indexserver port>_tcp, remote=(invalid), state=New, pending=[----]}}
  $[2]$=
  tcp port <indexserver port> on 127.0.0.1 => socket[107285318], user ""<sid>adm:<UID>"

• You want to know what caused the unexpected shutdown.

SAP HANA, platform edition

stopping, stopped, defunct, ghost, missing, process disappeared, SAPHANA, saphana, hdbindexserver, indexserver hdb, nameserver, hdbnameserver, daemon, hdbdaemon, sap.hdb, exited, high-availability, ha, standby, failover, fail over, stand-by, stand , KBA , HAN-DB-HA , SAP HANA High Availability (System Replication, DR, etc.) , HAN-DB , SAP HANA Database , Problem