3223217 - SAP HANA Concurrency highly consumed

• A HANA system or tenant DB is in a general hanging or unresponsive state.

• The service traces in '/usr/sap/<sid>/HDB<inst>/<host>/trace' covering the timeframe of the event, may include manual or dynamically generated runtime dumps like '<service>_<host>.<port>.rtedump.<timestamp>.trc'  or particular performance related flavors which indicate long running transactions : 'indexserver_hostname.3<inst-id>03.rtedump.20220509-094313.xxxx.longrunning_transaction.trc' or lock waits like 'service>_<host>.<port>.rtedump.<timestamp>.indexhandle_blocked.trc' : (2400007 - FAQ: SAP HANA Runtime Dumps)

• The runtime dumps are fed through and analyzed by the SAP HANA dump analyzer tool (2498739 - How-To: Analyzing Runtime Dumps with SAP HANA Dump Analyzer) and reveal symptoms similar to the following :

• Network communication issues and client session dumps may be observed in the middle of this resource exhaustion in the indexserver 
      
  '4. What are typical problems in the network area?' - 2222200 - FAQ: SAP HANA Network
   
  Be aware that network issues are often not caused by issues in the network infrastructure. Instead these kind of communication issues typically have reasons like:
  • Overload and bad performance of SAP HANA services (SAP Note 2000000)
     .........
  

  [24845]\{338021\}[-1/-1] 2022-05-09 09:35:24.112194 i TraceContext     TraceContext.cpp(01269) : UserName=<>, ApplicationUserName=<>, ApplicationName=XimoStudio, ApplicationSource=Main, StatementExecutionID=844437884692105
  [24845]\{338021\}[-1/-1] 2022-05-09 09:35:24.112135 e SQLSession       sm_handler.cc(02824) : (sockfd:508, channel:0x00007a20efb7b200, event:0x00007a0345c27f40, part:177) communication error: SSL exception  1: no.2110005  (Basis/IO/Stream/impl/NetworkChannel.cpp:1401) TID: 24845
  
  [24845]\{338021\}[-1/-1] 2022-05-09 09:35:24.140789 e SQLSession       sm_handler.cc(03209) : ===== START OF SESSION DUMP ======================
  [24845]\{338021\}[-1/-1] 2022-05-09 09:35:24.140806 e SQLSession       sm_handler.cc(03210) : CHANNEL      : (sockfd:508, part:177) open_time=22064239 ms, exp_size(header)=240, act_size(buffer)=0
  [24845]\{338021\}[-1/-1] 2022-05-09 09:35:24.140822 e SQLSession       sm_handler.cc(03211) : ACTION       : name=PrepareStatement, message_type=PREPARE, stocked_error_code=0, scrollable=0, oabap_stream=0
  [24845]\{338021\}[-1/-1] 2022-05-09 09:35:24.140842 e SQLSession       sm_handler.cc(03212) : CLIENT       : dewdfm1116.win.hec.sap.biz[xxx.xxxx.xxx.xx]:10425(pid=40280)
  [24845]\{338021\}[-1/-1] 2022-05-09 09:35:24.140855 e SQLSession       sm_handler.cc(03213) : PROTOCOL     : 4.1, internal_session_id=1587729897674998, packet_count(message_id)=266, version_exchanged=1, authenticated=1
  [24845]\{338021\}[-1/-1] 2022-05-09 09:35:24.140867 e SQLSession       sm_handler.cc(03214) : DISTRIBUTION : mode=0, version=4
  [24845]\{338021\}[-1/-1] 2022-05-09 09:35:24.140879 e SQLSession       sm_handler.cc(03215) : ROUTING      : anchor=0, primary=0, own=338021, update_tid=0
  [24845]\{338021\}[-1/-1] 2022-05-09 09:35:24.140893 e SQLSession       sm_handler.cc(03217) : ===== END OF SESSION DUMP ========================
    
  
  [93011]{-1}[-1/-1] 2022-05-09 09:53:39.820443 e SQLSession       sm_handler.cc(01286) : (sockfd:825, part:208, channel:0x00006e08fc64ee00, event:0x000071e66bf0ae00) receive duration time=5 ms, channel error: exception  1: no.2110006  (Basis/IO/Stream/impl/NetworkChannel.cpp:1944) TID: 93011
      Error during stream receive: NetworkChannel::receiveRawImp. recv failed, rc=110: Connection timed out; $Context$=[f2dcaa42528b19de,xx.xx.xx.xx:30215,xx.xx.xx.xx:63369,UNK,0]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139927849200664, fd=825, refCnt=1, local=xx.xx.xx.xx/30215_tcp, remote=xx.xx.xx.xx/63369_tcp, state=Connected, pending=[----]}}}
  exception throw location:
  
  
  [26537]{-1}[-1/-1] 2022-05-09 09:37:39.105220 e SQLSession       sm_handler.cc(02824) : (sockfd:1004, channel:0x00006dfd52565280, event:0x0000763409699f40, part:-1) communication error: SSL exception  1: no.2110005  (Basis/IO/Stream/impl/NetworkChannel.cpp:1401) TID: 26537
      Error during stream send: NetworkChannel::sendRawImp. send failed, rc=32: Broken pipe; $Context$=[f2dcaa42528b1bff,xx.xx.xx.xx:30215,xx.xx.xx.xx:27660,UNK,0]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=133112876575768, fd=INVALID, refCnt=1, local=xx.xx.xx.xx/30215_tcp, remote=xx.xx.xx.xx/27660_tcp, state=Closed, pending=[----]}}}
  exception throw location:

SAP HANA, platform edition 2.0

SAP HANA, platform edition 1.0

longrunning_transaction, DB freeze, unresponsive, standstill, high CPU, HANA freeze, Performance, Jobworkers, jobworker, Host CPU Usage, run time, rtedump, workload, default_statement_concurrency_limit, max_concurrency, max_concurrency_hint, HANA Stuck,  , KBA , HAN-DB-PERF , SAP HANA Database Performance , HAN-DB , SAP HANA Database , SV-PERF , Performance Problems , Problem