3561558 - Application Server Cannot Connect to HANA With Error "SSL handshake failed: error code: 336151608, file:ssl/record/rec_lay" in TRANS.LOG

• SAP Application server cannot connect to HANA database and the trans.log generated by 'r3trans -d'  command contains the error :
  
  4 ETW000  [dbhdbsql.cpp,00000]  *** ERROR => Connect to database failed, rc=1, rcSQL=-10709
  4 ETW000  [     dev trc,00000]  SQLCODE    : -107094 ETW000  [     dev trc,00000]  SQLERRTEXT : Connection failed (RTE:[300013] SSL handshake failed: error code: 336151608, file:ssl/record/rec_lay\
  4 ETW000  [     dev trc,00000]               er_s3.c, line:1544 - SSL alert number 80
  4 ETW000                         (<hostname>:<port>))2EETW169 no connect possible: "DBMS = HDB                              ---  SERVER = '' PORT = ''" 
  
  
• In the indexserver trace, the following errors are found :
  
  E Crypto           SecurityHelper.cpp(00149) : [correlation id: 0x896f52857cf3a1e5] exception  1: no.300013  (Crypto/Shared/SSL/CommonCrypto/Engine.cpp:547) TID: 192873   
  SSL handshake failed: SSL error [13]: Unknown error, General error: 0x0000000d | SAPCRYPTOLIB | SSL_accept
  SAPCRYPTO API error
  Cannot allocate memory
  0xa010000d | CRYPT_ | ssl3_accept
  Cannot allocate memory
  0xa010000d | CRYPT_ | ssl3_send_server_key_exchange
  Cannot allocate memory
  0xa010000d | CRYPT_ | ssl3_encode_server_key_exchange
  Cannot allocate memory
  0xa010000d | CRYPT_ | ssl3_sign_server_key_exchange
  Cannot allocate memory
  0xa010000d | CRYPT_ | af_sign_all_no_hash
  Cannot allocate memory (ErrCode: 13)  
  
  e OSMemory         mmMallocProxy.cpp(02370) : Failed: munmap MM [0x00007fa49bc28000, 0x00007fa49bc38000[ (65536b | 0x0000000000000000), error 12 (Cannot allocate memory)
  
  Image/data in this KBA is from SAP internal systems, sample data, or demo systems. Any resemblance to real data is purely coincidental.

• SAP HANA Platform Edition 1.0
• SAP HANA Platform Edition 2.0

error code: 336151608, file:ssl/record/rec_lay\, SSL alert number 80 , KBA , HAN-DB , SAP HANA Database , Problem