3135121 - Dumping shared memory hang with the deadlock in IBM OS function_global_lock_common

• In errorlog, we can see some messages like below: 00:0006:00000:00304:2021/11/30 02:32:35.55 kernel Current process (0x6e450638) infected with signal 11 (SIGSEGV)

           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel Current Process is running on Engine 0
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel Address 0x0900000000046568 (), siginfo (code, address) = (50, 0x4070352c2040704e)
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel **** Saved signal context (0x000000019e6a86e0): **** 
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel __sc_onstack: 0x0, __sc_uerror: 55
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel uc_sigmask: 0xffffffff1ffef877 0xffffffffffffffff 0xffffffffffffffff 0x7fffffffffffffff
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel Machine Save State:
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel PC (iar): 0900000000046568 ()
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel Link Register (lr): 09000000000478ac ()
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel Stack Pointer (stkp): 000000019e6a9bd0
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel msr : a00000000200d032 ctr : 09000000005cc580 cr : 42222428
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel xer : 20000010 00:0006:00000:00304:2021/11/30 02:32:35.55 kernel r0 :
           0000000000000120 r1 : 000000019e6a9bd0 r2 : 09001000a008f8b0
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel r3 : 0000000000000001 r4 : 09001000a009e0e0 r5 : 4070352c20407036
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel r6 : 2c204070372c2040 r7 : 00000001113fc6c0 r8 : 00000001113fc6c0
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel r9 : 000000011136f1c0 r10 : 09001000a009f508 r11 : 09001000a009f508
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel r12 : 000000011136f1c0 r13 : 0000000110d6e800 r14 : 000000027ac5ac08
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel r15 : 0000000000000000 r16 : 0000000241056000 r17 : 0000000000020000
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel r18 : 0000000110000f20 r19 : 0000000000000001 r20 : 0000000000000053
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel r21 : 0000000000000053 r22 : 09001000a009f518 r23 : 0000000000000000
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel r24 : 09001000a0006e60 r25 : 000000011136f180 r26 : 0000000000000040
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel r27 : 000000011136f1c0 r28 : 0000000000000160 r29 : 09001000a009e0e0
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel r30 : 000000011136f180 r31 : 000000011136f190
           00:0006:00000:00304:2021/11/30 02:32:35.55 kernel **** end of signal context ****
           00:0006:00000:00304:2021/11/30 02:32:35.56 server Initiating shared memory dump for signal 11

• Since the dump got hang, we didn't get the full stack. Customer tried to run "procstack" after the server went into hang

	---------- tid# 54460771 (pthread ID:   1543) ----------
	0x090000000059c98c  _global_lock_common(??, ??, ??) + 0x42c
	0x090000000004a2b8  malloc_y(0x0, 0x0, 0x9001000a007e1d8, 0x9001000a0003e88, 0x0, 0x0, 0xf1000b0186527800, 0x8000000000001032) + 0x198
	0x090000000002e308  malloc_common@AF104_87(??) + 0x48
	0x090000000002e270  malloc@AF103_98(??) + 0xd0
	0x09000000000646c8  qsort(??, ??, ??, ??) + 0x88
	0x0000000100b3d83c  shm_build_memmap(??, ??, ??) + 0x5a8
	0x0000000100b3d264  shm__alloc_regions(??, ??) + 0x8
	0x0000000100b3b4c8  shm__memdump(??, ??, ??) + 0x314
	0x0000000100b3a06c  shm_memdump_server(??, ??, ??, ??) + 0x534
	0x0000000100b322b4  csmd_dump_condition(??, ??, ??, ??, ??, ??) + 0x1a8
	0x00000001018c59f8  kisignal(0x1eed31930, 0x0, 0x0) + 0x7a0
	0x0000000000004b34  sig_epilog64() + 0xc
	0x090000000059c988  _global_lock_common(??, ??, ??) + 0x428
	...omitted...
	0x090000000059c988  _global_lock_common(??, ??, ??) + 0x428
	...omitted...
	0x00000001636c95dc  ...  + 0x36c95dc

• SAP Adaptive Enterprise 16.x

• SAP Adaptive Enterprise 15.7

• IBM AIX 7.2

shared memory hang, IBM OS deadlock, signal 11 error, SAP ASE

1. x, SAP ASE
2. 7, IBM AIX
3. 2, shared memory dump, deadlock resolution, memory dump compression, number of dump threads, CR 825938, pthread deadlock, malloc deadlock, qsort removal, parallel dumping, dump compression