2024082 - Very high latency after upgrade to 15.7.1 on AIX - SRS

• Very high latency in the inbound queue for non-warm standby connections.
   
• The 'procstack' output for the SRS process shows a lot of threads in the _global_lock_common() and _global_unlock_common() calls as memory is allocated or freed via malloc/free calls, for example:
  
  

  ....
  ---------- tid# 85983339 (pthread ID:  26475) ----------
  0x09000000006e47b4  _global_lock_common(??, ??, ??) + 0x4d4
  0x0900000000099b70  malloc_y(0x200, 0x0, 0x9001000a006ebb0, 0x11005f048, 0x1, 0x116829710, 0x1007f4b58, 0x116829710) + 0x7f0
  0x0900000000010748  malloc_common@AF102_86(??) + 0x28
  0x0000000100214550  mem_memAlloc(??, ??, ??) + 0x130
  0x0000000100214d34  memSegGetSeg(??, ??, ??, ??, ??, ??) + 0x74
  0x0000000100214a74  mem__Allocate(??, ??, ??, ??, ??, ??, ??, ??) + 0xd4
  0x00000001005ecb78  md_deliver_message(??, ??, ??, ??, ??, ??, ??, ??) + 0x178
  0x00000001006ee580  IPRA.$td_deliver_command(??, ??, ??, ??, ??, ??, ??) + 0xc40
  0x00000001006ef2bc  td_deliver_command(??, ??, ??, ??, ??, ??, ??) + 0x5c
  0x00000001006e74c0  _dist_applied_cmd(0x6d0000006d, 0x116829710, 0xa1400000a14, 0x132175600, 0x13239b2c0, 0x132394020, 0x13250b520, 0x1325922c0) + 0x2ba0
  0x00000001006dc448  _dist_service(??, ??) + 0x2a08
  0x00000001006d8c04  dist_wrapper(??) + 0xfc4
  0x0000000100009aac  srv__start_function(??) + 0x2cc
  0x09000000006e5d50  _pthread_body(??) + 0xf0
  ---------- tid# 77660339 (pthread ID:  33644) ----------
  0x09000000006e47b4  _global_lock_common(??, ??, ??) + 0x4d4
  0x0900000000097a5c  free_y(??, ??) + 0x11c
  0x090000000000fd18  free_common(??) + 0xb8
  0x000000010021688c  memFree@AF30_6(??) + 0x12c
  0x0000000100216afc  mem_free@AF31_11(??, ??) + 0x21c
  0x000000010026cb80  _sqt_del_from_cache(??, ??, ??, ??) + 0x840
  0x000000010026b790  _sqt_service(??, ??) + 0x470
  0x00000001002697e8  _sqt_wrap(??) + 0x2e8
  0x0000000100009aac  srv__start_function(??) + 0x2cc
  0x09000000006e5d50  _pthread_body(??) + 0xf0
  ---------- tid# 65798391 (pthread ID:  30165) ----------
  0x09000000006e33f0  _global_unlock_common(??, ??) + 0x10
  0x0900000000099b70  malloc_y(0x800, 0x0, 0x9001000a006ebb0, 0x11005f048, 0x1, 0x11a89d890, 0x1007f24a4, 0x11a89d890) + 0x7f0
  0x0900000000010748  (??) + 0x28
  0x0000000100214550  mem_memAlloc(??, ??, ??) + 0x130
  0x0000000100214d34  memSegGetSeg(??, ??, ??, ??, ??, ??) + 0x74
  0x0000000100214bf4  mem__Allocate(??, ??, ??, ??, ??, ??, ??, ??) + 0x254
  0x00000001005c0f5c  yyparse(??) + 0xfdc
  0x00000001005bfe48  prs_parse_sql(??, ??) + 0x28
  0x00000001005bed94  pak_unpack_command(??, ??, ??, ??, ??, ??) + 0xa74
  0x000000010055fefc  _dsi_parse_cmd(??, ??, ??, ??, ??, ??, ??) + 0x7c
  0x00000001006866d4  dsie<...>(...)(??, ??, ??, ??) + 0x374
  0x0000000100685a58  dsie__ProcessGroup(??, ??, ??) + 0x458
  0x0000000100684b1c  dsie__main(??, ??) + 0x27c
  0x00000001006842e4  dsie__wrap(??) + 0x864
  0x0000000100009aac  srv__start_function(??) + 0x2cc
  0x09000000006e5d50  _pthread_body(??) + 0xf0
  ...

• This started after SRS was upgraded from 15.6 to 15.7.1.

• SAP Replication Server (SRS) 15.7.1
• AIX 6.1 and 7.1

malloc, free, simultaneous, AIXTHREAD_MUTEX_FAST, MALLOCOPTIONS, YIELDLOOPTIME, SPINLOOPTIME, multiheap , KBA , BC-SYB-REP , Sybase Replication Server (standalone) , Problem