3808709 - SAP HANA System Replication Status UNKNOWN Due to Docstore Port Conflict

The replication between two environments is stuck because the secondary site is unable to initialize some processes.

GetProcessList
OK
name, description, dispstatus, textstatus
hdbdaemon, HDB Daemon, YELLOW, Initializing
hdbcompileserver, HDB Compileserver, GREEN, Running
hdbdiserver, HDB Deployment Infrastructure Server, YELLOW, Scheduled
hdbdocstore, HDB DocStore, YELLOW, Initializing
hdbdpserver, HDB DPserver, GREEN, Running
hdbindexserver, HDB Indexserver, GREEN, Running
hdbindexserver, HDB Indexserver-HPA, GREEN, Running
hdbindexserver, HDB Indexserver-HPU, GREEN, Running
hdbnameserver, HDB Nameserver, GREEN, Running
hdbpreprocessor, HDB Preprocessor, GREEN, Running
hdbwebdispatcher, HDB Web Dispatcher, YELLOW, Scheduled
hdbxsengine, HDB XSEngine, GREEN, Running

By reviewing the docstore traces on the secondary site, it can be observed that the connection from the secondary site to the primary site through the system replication port is being refused. 

[XXXXXX]{-1}[-1/-1] XXXX-XX-XX XX:XX:XX.xxxxxx e Stream           NetworkChannelCompletion.cpp(00505) : [xxx] NetworkChannelCompletionThread #8 {<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=xxx, fd=xx, refCnt=1, idx=8, local=<local_ip>/<local_port>_tcp, remote=<remote_ip>/<remote_port>_tcp, state=ConnectWait, pending=[---c]}}}: Error in asynchronous stream event: exception  1: no.2110001  (Basis/IO/Stream/impl/NetworkChannelCompletion.cpp:431) TID: XXXXXX
    Generic stream error: getsockopt, Event=EPOLLERR - , rc=111: Connection refused; $Context$=[xxx]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=<memory_address>, fd=xx, refCnt=1, idx=8, local=<local_ip>/<local_port>_tcp, remote=<remote_ip>/<remote_port>_tcp, state=ConnectWait, pending=[---c]}}}
exception throw location:

Reviewing the docstore traces on the primary site shows that the system replication port used by the docstore service is already bound to another process.

[XXXXXX]{-1}[-1/-1] XXXX-XX-XX XX:XX:XX.XXXXXX i sr_dataaccess    DisasterRecoveryPrimaryImpl.cpp(01569) : Start listen to specific interface 127.0.0.1 port:4<ins_id>52
[XXXXXX]{-1}[-1/-1] XXXX-XX-XX XX:XX:XX.XXXXXX e Stream           NetworkAddress.cpp(01176) : socket info for 127.0.0.1/4<inst_id>52_tcp: [<socket_raw_data>]
[XXXXXX]{-1}[-1/-1] XXXX-XX-XX XX:XX:XX.XXXXXX e Stream           NetworkChannel.cpp(00609) : [<session_id>,127.0.0.1:4<inst_id>52,-,REP,0] Cannot bind to a local address: tcp port 4<ins_id>52 on 127.0.0.1 => socket[<socket_inode>], user "<database_username>:<uid>", port blocked due to state TCP_ESTABLISHED
[XXXXXX]{-1}[-1/-1] XXXX-XX-XX XX:XX:XX.XXXXXX e sr_dataaccess    DisasterRecoveryPrimaryImpl.cpp(01603) : Listener cannot be started, because port 4<ins_id>52 is already in use!
[XXXXXX]{-1}[-1/-1] XXXX-XX-XX XX:XX:XX.XXXXXX e sr_dataaccess    DisasterRecoveryPrimaryImpl.cpp(01604) : A system replication primary uses replication ports in the range of instance number(s) from <ins_id> to <ins_id>
[XXXXXX]{-1}[-1/-1] XXXX-XX-XX XX:XX:XX.XXXXXX e sr_dataaccess    DisasterRecoveryPrimaryImpl.cpp(01606) : Please check, that there is no other system on this machine using instancenr <ins_id>! This is just a hint and possibly not the root cause ..
[XXXXXX]{-1}[-1/-1] XXXX-XX-XX XX:XX:XX.XXXXXX e sr_dataaccess    DisasterRecoveryPrimaryImpl.cpp(01608) : In general the port range 4<ins_id>00-4<ins_id>99 must not be used by any other process when system replication is turned on!
[XXXXXX]{-1}[-1/-1] XXXX-XX-XX XX:XX:XX.XXXXXX e sr_dataaccess    DisasterRecoveryPrimaryImpl.cpp(01611) : You may need to set ip_local_port_range as Multitenant Database, please check "System Replication with Tenant Databases" section in admin guide and SAP note 2382421, 401162
[XXXXXX]{-1}[-1/-1] XXXX-XX-XX XX:XX:XX.XXXXXX e sr_dataaccess    DisasterRecoveryPrimaryImpl.cpp(01504) : checkAndStartListener(): listener start failed: exception  1: no.2110008  (Basis/IO/Stream/impl/NetworkChannel.cpp:1153) TID: XXXXXX
    Error address in use: $msg$, rc=98: Address already in use; $[1]$=NetworkChannelBase::bindLocal. bind failed; $Context$=[<session_id>,127.0.0.1:4<instance_id>52,-,REP,0]; $channel$={<NetworkChannelBase>={this=<memory_address>, fd=<fd>, refCnt=1, local=127.0.0.1/4<ins_id>52_tcp, remote=(invalid), state=New, pending=[----]}}; $[2]$=tcp port 4<ins_id>52 on 127.0.0.1 => socket[<socket_inode>], user "<username>:<uid>", port blocked due to state TCP_ESTABLISHED

Using netstat, it can be observed that the docstore port is assigned to the hdbwebdispatcher service.

As a result, the docstore service cannot be started on the secondary site, causing the overall system replication status to be displayed as UNKNOWN.

"Image/data in this KBA is from SAP internal systems, sample data, or demo systems. Any resemblance to real data is purely coincidental." 

SAP HANA database

system replication, unknown status, hdbdocstore, docstore, hdbwebdispatcher, address already in use, tcp_established, connection refused, listener cannot be started, ip_local_reserved_ports, ephemeral ports, port conflict, bind failure, failover , KBA , HAN-DB-HA , SAP HANA High Availability (System Replication, DR, etc.) , Problem