3316858 - Daemon service fails to start with "Cannot create SSL engine: $ErrorText$; $Context$" errors

• A HANA system is failing to startup.
• Output from sapcontrol /usr/sap/hostctrl/exe/sapcontrol -nr <instance number> -function GetProcessList , shows all services in a YELLOW status

GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, YELLOW, Initializing, 2023 03 23 09:58:53, 2:24:43, 64750
hdbcompileserver, HDB Compileserver, YELLOW, Scheduled, , , -1
hdbdiserver, HDB Deployment Infrastructure Server, YELLOW, Scheduled, , , -1
hdbdiserver, HDB Deployment Infrastructure Server-SID, YELLOW, Scheduled, , , -1
hdbdiserver, HDB Deployment Infrastructure Server-SID, YELLOW, Scheduled, , , -1
hdbdocstore, HDB DocStore-SID, YELLOW, Scheduled, , , -1
hdbdpserver, HDB DPserver-SID, YELLOW, Scheduled, , , -1
hdbindexserver, HDB Indexserver-SID, YELLOW, Scheduled, , , -1
hdbindexserver, HDB Indexserver-SID, YELLOW, Scheduled, , , -1
hdbnameserver, HDB Nameserver, YELLOW, Initializing, 2023 03 23 12:22:33, 0:01:03, 148035
hdbpreprocessor, HDB Preprocessor, YELLOW, Scheduled, , , -1
hdbwebdispatcher, HDB Web Dispatcher, YELLOW, Scheduled, , , -1
hdbxscontroller, HDB XS Controller, YELLOW, Scheduled, , , -1
hdbxsengine, HDB XSEngine-SID, YELLOW, Scheduled, , , -1
hdbxsengine, HDB XSEngine-SID, YELLOW, Scheduled, , , -1
hdbxsexecagent, HDB XS Execution Agent, YELLOW, Scheduled, , , -1
hdbxsuaaserver, HDB XS UAA Server, YELLOW, Scheduled, , , -

• The daemon service trace shows the following error pattern repeating over and over again since most recent startup attempts :

[72413]{-1}[-1/-1] 2023-03-23 01:19:09.927498 i Daemon           Daemon.cpp(00155) : Shared memory updated, no daemon state change
[72413]{-1}[-1/-1] 2023-03-23 01:19:10.178854 e Stream           NetworkChannel.cpp(03412) : [d86b9fde900c0019,127.0.0.1:30000,-,DAE,0] Exception occurred in accept: exception  1: no.300012  (Crypto/Shared/SSL/CommonCrypto/Context.cpp:346) TID: 72413
    Cannot create SSL engine: $ErrorText$; $Context$=[d86b9fde900c0373,127.0.0.1:30000,127.0.0.1:58882,DAE,0]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=94908956732840, fd=12, refCnt=1, local=127.0.0.1/30000_tcp, remote=127.0.0.1/58882_tcp, state=Connected, pending=[----]}}}
exception throw location:
   0: 0x00007f1abcd004c8 in Crypto::SSL::CommonCrypto::Context::createEngine(Crypto::SSL::Engine::Type, ltt::allocator&)+0x394 at Crypto/Shared/SSL/CommonCrypto/Context.cpp:346 (libcrypto_base.so)
   1: 0x00007f1abccef52c in Crypto::SSL::Filter::Acceptor::accept(void const*, unsigned long, Crypto::SSL::FilterWithStepByStepHandshake::ServerHandshakePhase)+0xad8 at Crypto/Shared/SSL/Filter.cpp:186 (libcrypto_base.so)
   2: 0x00007f1ac0cc7649 in Stream::NetworkChannelSSLFilter::doSSLHandShake(void const*, unsigned long)+0x1c5 at Basis/IO/Stream/impl/NetworkChannelSSLFilter.cpp:344 (libhdbbasis.so)
   3: 0x00007f1ac0cc80d9 in Stream::NetworkChannelSSLFilter::initServerChannel()+0x55 at Basis/IO/Stream/impl/NetworkChannelSSLFilter.cpp:225 (libhdbbasis.so)
   4: 0x00007f1ac0ccf0ea in Stream::NetworkListener::connectionEvent(Stream::CompletionSet::EventType, int, ltt::refcounted_handle<Stream::NetworkChannel>&, unsigned long)+0x336 at Basis/IO/Stream/impl/NetworkChannel.cpp:3463 (libhdbbasis.so)
   5: 0x00007f1ac0ccf7bd in Stream::NetworkListener::acceptEvent(Stream::CompletionSet::EventType, int)+0x69 at Basis/IO/Stream/impl/NetworkChannel.cpp:3408 (libhdbbasis.so)
   6: 0x00007f1ac1983c1f in TrexDaemon::NetworkListenerChannel::handleEvent(Stream::CompletionSet::EventType, int)+0x2b at TrexDaemon/impl/NetworkListener.cpp:994 (libhdbservice.so)
   7: 0x00007f1ac19a31c3 in TrexDaemon::waitForChild(ltt::vector<ltt::pair<ltt::smart_ptr<TrexDaemon::RunningInstance, ltt::integral_constant<bool, false>, ltt::integral_constant<bool, false> >, ltt::smart_ptr<TrexDaemon::Program, ltt::integral_constant<bool, false>, ltt::integral_constant<bool, false> > > > const&, ltt::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<int, 64> >&, long)+0x660 at TrexDaemon/impl/SignalsUNIX.cpp:831 (libhdbservice.so)
   8: 0x00007f1ac196a60f in TrexDaemon::Daemon::loop()+0x44b at TrexDaemon/impl/DaemonLoop.cpp:237 (libhdbservice.so)
   9: 0x00007f1ac197aefa in daemonMainImpl(int, char**)+0x1ef6 at TrexDaemon/impl/MainImpl.cpp:927 (libhdbservice.so)
  10: 0x00007f1ac0f21d81 in System::mainWrapper(int, char**, char**)+0x80 at Basis/System/impl/IsInMain.cpp:399 (libhdbbasis.so)
  11: 0x00007f1abec1a493 in .annobin_libc_start.c+0xf0 (libc.so.6)
  12: 0x00005651b41468f3 in _sub_I_65535_0.0+0xa0 at ../sysdeps/x86_64/start.S:112 (hdbdaemon)
exception type information:
 - 0: public ltt::exception@0x00007f1ac009d8e8
[72413]{-1}[-1/-1] 2023-03-23 01:19:10.179009 e Daemon           NetworkListener.cpp(01008) : Daemon on 127.0.0.1:30000, timeout 21188 cannot accept new connection
[72413]{-1}[-1/-1] 2023-03-23 01:20:13.970691 i Daemon           DaemonHandle.cpp(00323) : Instance count 1. Searching for terminated processes
[72413]{-1}[-1/-1] 2023-03-23 01:20:13.970777 i Daemon           DaemonHandle.cpp(00334) : Process 'hdbnameserver', pid 398487 exited normally with status 0
[72413]{-1}[-1/-1] 2023-03-23 01:20:13.970856 i Daemon           Events.cpp(00031) : "StartInstanceEvent" is due in 10000 ms. Program 'hdbnameserver', pid 0
[72413]{-1}[-1/-1] 2023-03-23 01:20:13.970876 i Daemon           Daemon.cpp(01142) : Daemon state "starting", runlevels [current:reached:target]=[1:1:5]
[72413]{-1}[-1/-1] 2023-03-23 01:20:13.971009 i Daemon           Daemon.cpp(00155) : Shared memory updated, no daemon state change
[72413]{-1}[-1/-1] 2023-03-23 01:20:13.974057 w Daemon           Daemon.cpp(00757) : Found terminated and planned instance 0 of program 'nameserver'
[72413]{-1}[-1/-1] 2023-03-23 01:20:23.980225 i Daemon           Program.cpp(00038) : Starting instance right now
[72413]{-1}[-1/-1] 2023-03-23 01:20:23.980460 i Daemon           DaemonChild.cpp(00481) : Starting program 'hdbnameserver' with args ''
[72413]{-1}[-1/-1] 2023-03-23 01:20:23.986753 i Environment      Environment.cpp(00236) : Changing environment put PYTHONDONTWRITEBYTECODE=1
[72413]{-1}[-1/-1] 2023-03-23 01:20:23.990021 i Daemon           DaemonChild.cpp(00708) : Child program 'hdbnameserver', pid 399539 started in its own process group
[72413]{-1}[-1/-1] 2023-03-23 01:20:23.990128 i Daemon           RunningInstance.cpp(00259) : Start 'hdbnameserver' as process 399539, process group 399539, instance 0 of <HDB Nameserver>
[72413]{-1}[-1/-1] 2023-03-23 01:20:23.990152 i Daemon           Daemon.cpp(01142) : Daemon state "starting", runlevels [current:reached:target]=[1:1:5]
[72413]{-1}[-1/-1] 2023-03-23 01:20:23.990283 i Daemon           Daemon.cpp(00155) : Shared memory updated, no daemon state change
[72413]{-1}[-1/-1] 2023-03-23 01:20:28.960965 e Stream           NetworkChannel.cpp(03412) : [d86b9fde900c0019,127.0.0.1:30000,-,DAE,0] Exception occurred in accept: exception  1: no.300012  (Crypto/Shared/SSL/CommonCrypto/Context.cpp:346) TID: 72413
    Cannot create SSL engine: $ErrorText$; $Context$=[d86b9fde900c0374,127.0.0.1:30000,127.0.0.1:45924,DAE,0]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=94908956732840, fd=12, refCnt=1, local=127.0.0.1/30000_tcp, remote=127.0.0.1/45924_tcp, state=Connected, pending=[----]}}}
exception throw location:
   0: 0x00007f1abcd004c8 in Crypto::SSL::CommonCrypto::Context::createEngine(Crypto::SSL::Engine::Type, ltt::allocator&)+0x394 at Crypto/Shared/SSL/CommonCrypto/Context.cpp:346 (libcrypto_base.so)
   1: 0x00007f1abccef52c in Crypto::SSL::Filter::Acceptor::accept(void const*, unsigned long, Crypto::SSL::FilterWithStepByStepHandshake::ServerHandshakePhase)+0xad8 at Crypto/Shared/SSL/Filter.cpp:186 (libcrypto_base.so)
   2: 0x00007f1ac0cc7649 in Stream::NetworkChannelSSLFilter::doSSLHandShake(void const*, unsigned long)+0x1c5 at Basis/IO/Stream/impl/NetworkChannelSSLFilter.cpp:344 (libhdbbasis.so)
   3: 0x00007f1ac0cc80d9 in Stream::NetworkChannelSSLFilter::initServerChannel()+0x55 at Basis/IO/Stream/impl/NetworkChannelSSLFilter.cpp:225 (libhdbbasis.so)
   4: 0x00007f1ac0ccf0ea in Stream::NetworkListener::connectionEvent(Stream::CompletionSet::EventType, int, ltt::refcounted_handle<Stream::NetworkChannel>&, unsigned long)+0x336 at Basis/IO/Stream/impl/NetworkChannel.cpp:3463 (libhdbbasis.so)
   5: 0x00007f1ac0ccf7bd in Stream::NetworkListener::acceptEvent(Stream::CompletionSet::EventType, int)+0x69 at Basis/IO/Stream/impl/NetworkChannel.cpp:3408 (libhdbbasis.so)
   6: 0x00007f1ac1983c1f in TrexDaemon::NetworkListenerChannel::handleEvent(Stream::CompletionSet::EventType, int)+0x2b at TrexDaemon/impl/NetworkListener.cpp:994 (libhdbservice.so)
   7: 0x00007f1ac19a31c3 in TrexDaemon::waitForChild(ltt::vector<ltt::pair<ltt::smart_ptr<TrexDaemon::RunningInstance, ltt::integral_constant<bool, false>, ltt::integral_constant<bool, false> >, ltt::smart_ptr<TrexDaemon::Program, ltt::integral_constant<bool, false>, ltt::integral_constant<bool, false> > > > const&, ltt::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<int, 64> >&, long)+0x660 at TrexDaemon/impl/SignalsUNIX.cpp:831 (libhdbservice.so)
   8: 0x00007f1ac196a60f in TrexDaemon::Daemon::loop()+0x44b at TrexDaemon/impl/DaemonLoop.cpp:237 (libhdbservice.so)
   9: 0x00007f1ac197aefa in daemonMainImpl(int, char**)+0x1ef6 at TrexDaemon/impl/MainImpl.cpp:927 (libhdbservice.so)
  10: 0x00007f1ac0f21d81 in System::mainWrapper(int, char**, char**)+0x80 at Basis/System/impl/IsInMain.cpp:399 (libhdbbasis.so)
  11: 0x00007f1abec1a493 in .annobin_libc_start.c+0xf0 (libc.so.6)
  12: 0x00005651b41468f3 in _sub_I_65535_0.0+0xa0 at ../sysdeps/x86_64/start.S:112 (hdbdaemon)
exception type information:
 - 0: public ltt::exception@0x00007f1ac009d8e8
[72413]{-1}[-1/-1] 2023-03-23 01:20:28.961265 e Daemon           NetworkListener.cpp(01008) : Daemon on 127.0.0.1:30000, timeout 7156 cannot accept new connection
[72413]{-1}[-1/-1] 2023-03-23 01:20:30.799021 i Daemon           SignalsUNIX.cpp(00601) : signo 10=SIGUSR1 from user. errno 0 code 0. Requested 'RECONFIGURE'. Sender pid 399539, real user 'sidadm'=248592, executable 'hdbnameserver'
[72413]{-1}[-1/-1] 2023-03-23 01:20:30.799377 i Daemon           DaemonHandle.cpp(00144) : Reconfiguring all services

SAP HANA, Platform Edition 2.0

Daemon.cpp, Shared memory updated, no daemon state change, NetworkChannel.cpp, Stream, SSL, CommonCrypto, createEngine , KBA , HAN-DB , SAP HANA Database , HAN-DB-SEC , SAP HANA Security & User Management , Problem