3618814 - Work processes are long running with VMC related operations

• Work processes have long runtime on VMC related operations, e.g.<VMC EVENTS COMPILE> and other application programs indicating communications with VMC.
• There are lots of RFC_VMC_COMMUNICATION_ERROR dumps in ST22. 
• In the VM overview of SM52, lots of VMs stuck at starting up status. 

This scenario is supported by kernel snapshot analyzer (SAP KBA 3110588). If you analyze the kernel snapshots or the work folders with kernel snapshot analyzer (SAP KBA 3122914), the root cause, solution, and the corresponding evidence will be documented on the "Operating System Memory Analysis" tab of the kernel snapshot analyzer analysis report automatically. Please check the following KBAs for more information about kernel snapshot analyzer.

• SAP KBA 3110588 - Kernel Snapshot Analyzer: Central KBA
• SAP KBA 3122914 - The step-by-step guide to collect a server snapshot and analyze it with kernel snapshot analyzer.

Please also check the following KBA for the AI features in kernel snapshot analyzer.

• SAP KBA 3622007 - AI-assisted analysis in kernel snapshot analyzer.

More technical details can be checked by the call stacks of work processes via kernel snapshot analyzer analysis report (SAP KBA 2640476), several work processes are running with system call mmap() and some are spinning on getting EM2(for VMC memory management) mutex. 

Similar logs can be found in work process traces that spin on getting EM2 related mutex via Em2MtxLock because being blocked by some work processes sticking at operating system call mmap().

• Work processes waiting for mutex "EM2_CTX_":

…

I  *** WARNING =>  MtxLock 7f4ec5116b30 EM2_CTX_ owner=ABAP_WP_59 deadlock ? [mtSID.c      715]

M  ------------------ C-STACK ----------------------

[…]

dw.sapSID_D00[S](Em2MtxLock+0x1d)[0x5562f5e2b79f,0x1e7b79f]

dw.sapSID_D00[S](Em2CtxGetFreeSlot+0x5c6)[0x5562f5e2ec10,0x1e7ec10]

dw.sapSID_D00[S](Em2CtxTemplateRestore+0x22a)[0x5562f5e2ef66,0x1e7ef66]

dw.sapSID_D00[S](ThJCloneVm(DP_VMC_TYPE)+0xb69)[0x5562f46193b0,0x6693b0]

dw.sapSID_D00[S](ThJCreateContext(DP_VMC_TYPE, DP_VMC_SUB_TYPE, THJ_CREATE_FOR, TH_VMC_DEBUGGING, char16_t*, unsigned char, DP_VMC_INPUT_INFO*)+0x7b7)[0x5562f461b4a6,0x66b4a6]

dw.sapSID_D00[S](ThJAttachVm2Context(DP_VMC_TYPE, DP_VMC_SUB_TYPE, DP_VMC_INPUT_INFO*, unsigned char, TH_VMC_DEBUGGING, char16_t*, unsigned char*)+0x7b2)[0x5562f461cb21,0x66cb21]

• Work processes sticking at mmap() call and holding mutex "EM2_CTX_":

…

M  *** ERROR => ThVmcICallServer: ThVmcAttachServer failed (-1005) [thxxvmc.c    3811]

M  ------------------ C-STACK ----------------------

[...]

libc.so.6[S](__GI___mmap64+0x45)[0x7f4ebfec6a43,0x111a43]

dw.sapSID_D00[S](Es2MapBlocks+0xa9)[0x5562f5e62825,0x1eb2825]

dw.sapSID_D00[S](Es2CtxAttach+0x925)[0x5562f5e5ee84,0x1eaee84]

dw.sapSID_D00[S](Em2HeapCtxAttach_ES2+0xa2)[0x5562f5e2a89b,0x1e7a89b]

dw.sapSID_D00[S](Em2CtxAttach+0x1f4)[0x5562f5e2e06b,0x1e7e06b]

dw.sapSID_D00[S](ThVmcAttachMem(unsigned char, unsigned char)+0x3af)[0x5562f46690f9,0x6b90f9]

[…]

M  DpWpMutexLocked: W59 inside critical section, csTrack=2

M    mtx info at level 2 = "ES2_CTX_"

M    mtx info at level 1 = "EM2_CTX_"

M  ThInsideCriticalSection: mutex locked 

SAP NetWeaver ABAP system 

VMC, Virtual Machine Container, SM52, Es2MapBlocks, mmap, Es2CtxAttach, Em2MtxLock, EM2(Extended Memory 2nd Generation) , KBA , BC-VMC , Virtual Machine Container , Problem