3477753 - Hana Update failed

During the upgrade process special in this case from version 2.00.053.00.1605092543 to 2.00.077.00.1713529394, the database does not return after the planned restart from the update. Sapcontrol also fails with an error stating 'connection refused'. This issue started with the SPS02 version and persisted with the upgrade to SPS07. A backup was used to restore the failed upgrade back to SPS02 and then successfully upgraded from SPS02 to SPS05. However, SPS07 is still not working.

[113266]{-1}[-1/-1] 2024-04-30 15:02:54.921156 w HanaNetRPC       HanaNetExt.cpp(00065) : initializing network for client-only operation
[113266]{-1}[-1/-1] 2024-04-30 15:02:54.956080 i FileIO           LockFile.cpp(00330) : Cannot open pid file </usr/sap/ABC/HDB02/abcdesap11//lock/hdbdaemon@30200.pid>: 2 (No such file or directory)
[113266]{-1}[-1/-1] 2024-04-30 15:02:54.956086 w DaemonClient     Client.cpp(01003) : Cannot read daemon pid from its lock file "/usr/sap/ABC/HDB02/abcdesap11//lock/hdbdaemon@30200.pid"
[113266]{-1}[-1/-1] 2024-04-30 15:02:54.956136 e OSMemory         mmSharedMemoryProxy.cpp(00576) : shmget(10201, 0, 384) failed2 (No such file or directory)
[113266]{-1}[-1/-1] 2024-04-30 15:02:54.956986 w commlib          commlibImpl.cpp(00924) : Connection to '127.0.0.1:30201' failed: exception  1: no.2110017  (Basis/IO/Stream/impl/NetworkChannel.cpp:2855) TID: 113266
    System error: Connect on non-blocking socket failed. rc=111: Connection refused. channel={<NetworkChannel>={<NetworkChannelBase>={this=140097048639512, fd=10, refCnt=1, local=127.0.0.1/59882_tcp, remote=127.0.0.1/30201_tcp, state=ConnectWait, pending=[----]}}}; $Context$=[26479134c0040019,127.0.0.1:59882,127.0.0.1:30201,TRN,0]
exception throw location:
   0: 0x00007f6b0170f9a9 in .LTHUNK36.lto_priv.0+0x8e5 at Basis/IO/Stream/impl/NetworkChannel.cpp:2855 (libhdbbasis.so)
   1: 0x00007f6b01724c2f in Stream::NetworkChannelManager::connect(NetworkAccess::NetworkAddress const*, NetworkAccess::NetworkAddress const&, Stream::NetworkChannelParameters const&, bool, ltt::smartptr_handle<Stream::ChannelCallback>*, Stream::NetworkChannelComponent, int, ltt::allocator&)+0x57b at Basis/IO/Stream/impl/NetworkChannelManager.cpp:203 (libhdbbasis.so)
   2: 0x00007f6b023e7b26 in comm::connect(void*, char const*, unsigned short, int, Crypto::Configuration*, Stream::NetworkChannelComponent)+0x182 at commlib/Impl/commlibImpl.cpp:912 (libhdbnetrpc.so)
   3: 0x00007f6b02478490 in TrexNet::Channel::open(char const*, TrexNet::RequestParameters const&)+0x350 at TrexNet/Channel.cpp:326 (libhdbnetrpc.so)
   4: 0x00007f6b02489385 in TrexNet::ServerRep::openNewChannel(char const*, TrexNet::RequestParameters const&)+0x151 at TrexNet/EndPoint.cpp:249 (libhdbnetrpc.so)
   5: 0x00007f6b024a55e0 in TrexNet::Requestor::getChannel(char const*, unsigned short, TrexNet::RequestParameters const&)+0x3d0 at TrexNet/Requestor.cpp:174 (libhdbnetrpc.so)
   6: 0x00007f6b0250343e in TrexNet::Request::Request<TrexNet::Request::Request(ltt::basic_string_view<char, ltt::char_traits<char> >, Basis::HostAndPort const&, TrexNet::RequestParameters)::{lambda()#1}, void>(ltt::basic_string_view<char, ltt::char_traits<char> >, TrexNet::Request::Request(ltt::basic_string_view<char, ltt::char_traits<char> >, Basis::HostAndPort const&, TrexNet::RequestParameters)::{lambda()#1}&&, TrexNet::RequestParameters, unsigned long) [clone .constprop.0]+0x4ea at TrexNet/Request.cpp:416 (libhdbnetrpc.so)
   7: 0x00007f6b02496935 in TrexNet::Request::Request(ltt::basic_string_view<char, ltt::char_traits<char> >, Basis::HostAndPort const&, TrexNet::RequestParameters)+0xe1 at TrexNet/Request.cpp:418 (libhdbnetrpc.so)
   8: 0x00007f6b066193d6 in NameServer::TNSInfo::sendRequestTo(NameServer::Request const&, NameServer::Response&, Basis::HostAndPort const&, NameServer::TrexNetRequestHolder*, bool, char)+0x292 at TREXNameServer/TNSClient/TNSInfo.cpp:569 (libhdbbasement.so)
   9: 0x00007f6b066183da in NameServer::TNSInfo::processRequest(NameServer::Request const&, NameServer::Response&, int)+0x246 at TREXNameServer/TNSClient/TNSInfo.cpp:432 (libhdbbasement.so)
  10: 0x00007f6b065eac76 in NameServer::TNSClientBase<NameServer::TNSClientErrorGuard<false> >::processRequest(NameServer::Request const&, NameServer::Response&)+0x32 at TREXNameServer/TNSClient.cpp:538 (libhdbbasement.so)
  11: 0x00007f6b065a9782 in NameServer::TNSClientBase<NameServer::TNSClientErrorGuard<false> >::storeTrees(ltt::vector<NameServer::TNode> const&)+0x120 at TREXNameServer/TNSClient.cpp:651 (libhdbbasement.so)
  12: 0x0000561fd31e769e in NameServerCmd::CmdActionPingNameserver::performCommand()+0xaa at TREXNameServer/hdbnsutil/Actions/pingNameserver.cpp:35 (hdbnsutil)
  13: 0x0000561fd31b27f6 in main+0xa32 at TREXNameServer/hdbnsutil/TopologyCmdAction.cpp:29 (hdbnsutil)
  14: 0x00007f6b01970ecc in System::mainWrapper(int, char**, char**)+0xa8 at Basis/System/impl/IsInMain.cpp:525 (libhdbbasis.so)
  15: 0x00007f6aff63fd85 in __libc_start_main+0xe1 (libc.so.6)
  16: 0x0000561fd31c5269 in _start+0x25 at ../sysdeps/x86_64/start.S:108 (hdbnsutil)
exception type information:
 - 0: public ltt::exception@0x00007f6b00a9b290
[113266]{-1}[-1/-1] 2024-04-30 15:02:54.963258 w TrexNet          EndPoint.cpp(00265) : failed to open channel to '127.0.0.1:30201&0': (connection refused)
details: 
[113266]{-1}[-1/-1] 2024-04-30 15:02:54.965669 i Basis            TopologyUtil.cpp(01228) : failed.
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.592513 i Basis            TraceStream.cpp(00526) : ==== Starting hdbnsutil, version 2.00.077.00.1713529394 (fa/hana2sp07), build linuxx86_64 8dc29b34937cdba2d1ace1e9fc6e2ccfe650e6aa 2024-04-19 14:42:30 ld4550 gcc ('SAP release 11.3.0-sap7') 11.3.0
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.592540 i Memory           mmEnv.cpp(00203) : Per core memory pools
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.592542 i Basis            LttImportProvider.cpp(00508) : Timer implementation will use system timer.
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.592542 i Basis            ProcessorInfo.cpp(00468) : Using GDT segment limit to determine current CPU ID
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.592543 i Memory           mmProtectionKeyManager.cpp(00054) : ProtectionKeys disabled
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.592544 i Memory           mmGlobalMemoryData.cpp(01669) : Number of memory pools used: 8 + 1
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.592544 i Memory           mmGlobalMemoryData.cpp(01754) : PoolAllocators used
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.592547 i EventHandler     EventManagerImpl.cpp(01019) : AutoEventHandlerCallback::initial(period= off)
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.592548 i EventHandler     EventManagerImpl.cpp(01015) : AutoEventHandlerCallback::reconfigure(period= off --> 60)
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.596541 i Basis            CommandUtil.cpp(00065) : command: /hana/shared/ABC/HDB02/exe/hdbnsutil -sr_state
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.597010 i Basis            CommandUtil.cpp(00098) : called by user 'ABCadm' with UID: 1003 (parent process command line '*************************************************************************hdbupdrep*************************************************' with PID: 113314)
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.597023 w HanaNetRPC       HanaNetExt.cpp(00065) : initializing network for client-only operation
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.630501 i FileIO           LockFile.cpp(00330) : Cannot open pid file </usr/sap/ABC/HDB02/abcdesap11//lock/hdbdaemon@30200.pid>: 2 (No such file or directory)
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.630506 w DaemonClient     Client.cpp(01003) : Cannot read daemon pid from its lock file "/usr/sap/ABC/HDB02/abcdesap11//lock/hdbdaemon@30200.pid"
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.630555 e OSMemory         mmSharedMemoryProxy.cpp(00576) : shmget(10201, 0, 384) failed2 (No such file or directory)
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.631730 w commlib          commlibImpl.cpp(00924) : Connection to '127.0.0.1:30201' failed: exception  1: no.2110017  (Basis/IO/Stream/impl/NetworkChannel.cpp:2855) TID: 113350
    System error: Connect on non-blocking socket failed. rc=111: Connection refused. channel={<NetworkChannel>={<NetworkChannelBase>={this=140260817598488, fd=4, refCnt=1, local=127.0.0.1/59894_tcp, remote=127.0.0.1/30201_tcp, state=ConnectWait, pending=[----]}}}; $Context$=[3184f37c60040019,127.0.0.1:59894,127.0.0.1:30201,TRN,0]
exception throw location:
   0: 0x00007f9122eaf9a9 in .LTHUNK36.lto_priv.0+0x8e5 at Basis/IO/Stream/impl/NetworkChannel.cpp:2855 (libhdbbasis.so)
   1: 0x00007f9122ec4c2f in Stream::NetworkChannelManager::connect(NetworkAccess::NetworkAddress const*, NetworkAccess::NetworkAddress const&, Stream::NetworkChannelParameters const&, bool, ltt::smartptr_handle<Stream::ChannelCallback>*, Stream::NetworkChannelComponent, int, ltt::allocator&)+0x57b at Basis/IO/Stream/impl/NetworkChannelManager.cpp:203 (libhdbbasis.so)
   2: 0x00007f9123b87b26 in comm::connect(void*, char const*, unsigned short, int, Crypto::Configuration*, Stream::NetworkChannelComponent)+0x182 at commlib/Impl/commlibImpl.cpp:912 (libhdbnetrpc.so)
   3: 0x00007f9123c18490 in TrexNet::Channel::open(char const*, TrexNet::RequestParameters const&)+0x350 at TrexNet/Channel.cpp:326 (libhdbnetrpc.so)
   4: 0x00007f9123c29385 in TrexNet::ServerRep::openNewChannel(char const*, TrexNet::RequestParameters const&)+0x151 at TrexNet/EndPoint.cpp:249 (libhdbnetrpc.so)
   5: 0x00007f9123c455e0 in TrexNet::Requestor::getChannel(char const*, unsigned short, TrexNet::RequestParameters const&)+0x3d0 at TrexNet/Requestor.cpp:174 (libhdbnetrpc.so)
   6: 0x00007f9123ca343e in TrexNet::Request::Request<TrexNet::Request::Request(ltt::basic_string_view<char, ltt::char_traits<char> >, Basis::HostAndPort const&, TrexNet::RequestParameters)::{lambda()#1}, void>(ltt::basic_string_view<char, ltt::char_traits<char> >, TrexNet::Request::Request(ltt::basic_string_view<char, ltt::char_traits<char> >, Basis::HostAndPort const&, TrexNet::RequestParameters)::{lambda()#1}&&, TrexNet::RequestParameters, unsigned long) [clone .constprop.0]+0x4ea at TrexNet/Request.cpp:416 (libhdbnetrpc.so)
   7: 0x00007f9123c36935 in TrexNet::Request::Request(ltt::basic_string_view<char, ltt::char_traits<char> >, Basis::HostAndPort const&, TrexNet::RequestParameters)+0xe1 at TrexNet/Request.cpp:418 (libhdbnetrpc.so)
   8: 0x00007f9127db93d6 in NameServer::TNSInfo::sendRequestTo(NameServer::Request const&, NameServer::Response&, Basis::HostAndPort const&, NameServer::TrexNetRequestHolder*, bool, char)+0x292 at TREXNameServer/TNSClient/TNSInfo.cpp:569 (libhdbbasement.so)
   9: 0x00007f9127db83da in NameServer::TNSInfo::processRequest(NameServer::Request const&, NameServer::Response&, int)+0x246 at TREXNameServer/TNSClient/TNSInfo.cpp:432 (libhdbbasement.so)
  10: 0x00007f9127d8ac76 in NameServer::TNSClientBase<NameServer::TNSClientErrorGuard<false> >::processRequest(NameServer::Request const&, NameServer::Response&)+0x32 at TREXNameServer/TNSClient.cpp:538 (libhdbbasement.so)
  11: 0x00007f9127d49782 in NameServer::TNSClientBase<NameServer::TNSClientErrorGuard<false> >::storeTrees(ltt::vector<NameServer::TNode> const&)+0x120 at TREXNameServer/TNSClient.cpp:651 (libhdbbasement.so)
  12: 0x000055dd04484a63 in NameServerCmd::TopologyCmdAction::isNsActive(ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> >&, bool)+0xb0 at TREXNameServer/hdbnsutil/TopologyCmdAction.cpp:86 (hdbnsutil)
  13: 0x000055dd044bcb90 in NameServerCmd::CmdActionSrState::performCommand()+0xa0 at TREXNameServer/hdbnsutil/Actions/srState.cpp:32 (hdbnsutil)
  14: 0x000055dd044637f6 in main+0xa32 at TREXNameServer/hdbnsutil/TopologyCmdAction.cpp:29 (hdbnsutil)
  15: 0x00007f9123110ecc in System::mainWrapper(int, char**, char**)+0xa8 at Basis/System/impl/IsInMain.cpp:525 (libhdbbasis.so)
  16: 0x00007f9120ddfd85 in __libc_start_main+0xe1 (libc.so.6)
  17: 0x000055dd04476269 in _start+0x25 at ../sysdeps/x86_64/start.S:108 (hdbnsutil)
exception type information:
 - 0: public ltt::exception@0x00007f912223d290
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.638262 w TrexNet          EndPoint.cpp(00265) : failed to open channel to '127.0.0.1:30201&0': (connection refused)
details: 
[113350]{-1}[-1/-1] 2024-04-30 15:03:14.638731 i Basis            TopologyUtil.cpp(01225) : done.

SAP HANA DB Version 2.0

SAP HANA, Upgrade, Failed, SELinux, OS Level, SAP Applications, Error Connection Refused, SPS02, SPS07, Backup, Restore, Red Hat Enterprise Linux , KBA , HAN-LM-UPG-DB , Upgrade of HANA Database , Problem