3281130 - LNX: Server hangs at function call mutex_lock_slowpath

===== INTERNAL USE ONLY =====

You notice the server gets hang on operating system level.
The issue gets resolved after rebooting the server.

Following call trace is found in the OS log (/var/log/messages):

Example 1

kernel: INFO: task ABC_00_BTC_W171:26757 blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: ABC_00_BTC_W171 D ffff8a2a76f05280     0 26757  25047 0x00000080
kernel: Call Trace:
kernel: [<ffffffff9f18d2f9>] schedule_preempt_disabled+0x29/0x70
kernel: [<ffffffff9f18b447>] __mutex_lock_slowpath+0xc7/0x1d0
kernel: [<ffffffff9f18a81f>] mutex_lock+0x1f/0x2f
kernel: [<ffffffff9ec5e0f8>] do_last+0x298/0x1340
kernel: [<ffffffff9ec5f26d>] path_openat+0xcd/0x5a0
kernel: [<ffffffff9ec614bd>] do_filp_open+0x4d/0xb0
kernel: [<ffffffff9ec6f607>] ? __alloc_fd+0x47/0x170
kernel: [<ffffffff9ec4d394>] do_sys_open+0x124/0x220
kernel: [<ffffffff9ec4d4ae>] SyS_open+0x1e/0x20
kernel: [<ffffffffc0788cf5>] tmhook_invoke.isra.8+0x45/0x90 [tmhook]
kernel: [<ffffffffc0788ee8>] tmhook_handler+0x1a8/0x2c0 [tmhook]
kernel: [<ffffffff9f199f92>] system_call_fastpath+0x25/0x2a

Example 2

kernel: INFO: task ABC_00_BTC_W193:26776 blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: ABC_00_BTC_W193 D ffff8a2b1e389080     0 26776  25047 0x00000080
kernel: Call Trace:
kernel: [<ffffffff9f18d2f9>] schedule_preempt_disabled+0x29/0x70
kernel: [<ffffffff9f18b447>] __mutex_lock_slowpath+0xc7/0x1d0
kernel: [<ffffffff9f18a81f>] mutex_lock+0x1f/0x2f
kernel: [<ffffffff9f183146>] lookup_slow+0x33/0xa7
kernel: [<ffffffff9ec5d61e>] path_lookupat+0x89e/0x8d0
kernel: [<ffffffff9ec58f97>] ? path_get+0x27/0x30
kernel: [<ffffffff9ec5d67b>] filename_lookup+0x2b/0xc0
kernel: [<ffffffff9ec61387>] user_path_at_empty+0x67/0xc0
kernel: [<ffffffff9ec613f1>] user_path_at+0x11/0x20
kernel: [<ffffffffc0cb8fda>] bmhook_get_file_info_from_upath+0x5a/0xf0 [bmhook]
kernel: [<ffffffffc0cb3b3e>] tmbpf_get_file_info_from_upath+0xe/0x20 [bmhook]
kernel: [<ffffffff9f039396>] ? sock_sendmsg+0xb6/0xf0
kernel: [<ffffffff9ebee7aa>] ? __do_fault.isra.61+0x8a/0x100
kernel: [<ffffffffc0cc465c>] ? bmhook_prehook_handler+0x14c/0x3d0 [bmhook]
kernel: [<ffffffffc0788c4c>] ? tmhook_invoke_prehooks+0x4c/0xb0 [tmhook]
kernel: [<ffffffffc0788ec9>] ? tmhook_handler+0x189/0x2c0 [tmhook]
kernel: [<ffffffff9f199f92>] ? system_call_fastpath+0x25/0x2a 

Example 3

kernel: INFO: task ABC_00_DIA_W27:869789 blocked for more than 122 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: task:ABC_00_DIA_W28  state:D stack:    0 pid:869789 ppid:869738 flags:0x00000000
kernel: Call Trace:
kernel: <TASK>
kernel: __schedule+0x224/0x588
kernel: schedule+0x5f/0xdf
kernel: rwsem_down_write_slowpath+0x1aa/0x3f1
kernel: cifs_strict_writev+0xcf/0x2f0 [cifs]
kernel: new_sync_write+0x146/0x1e6
kernel: vfs_write+0x202/0x291
kernel: ksys_write+0x67/0xe8
kernel: tmhook_invoke+0x48/0xd0 [tmhook]
kernel: tmhook_handler+0x17f/0x224 [tmhook]
kernel: ? do_syscall_64+0x3b/0x8d
kernel: ? tmhook_invoke+0xd0/0xd0 [tmhook]
kernel: do_syscall_64+0x3b/0x8d
kernel: entry_SYSCALL_64_after_hwframe+0x69/0x0

Example 4

kernel: INFO: task EP2_00_BTC_W115:870157 blocked for more than 123 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: task:ABC_00_BTC_W118 state:D stack:    0 pid:870157 ppid:869738 flags:0x00000000
kernel: Call Trace:
kernel: <TASK>
kernel: __schedule+0x224/0x588
kernel: schedule+0x5f/0xdf
kernel: io_schedule+0x42/0x71
kernel: wait_on_page_bit_common+0x11b/0x4b6
kernel: ? __page_cache_alloc+0x90/0x85
kernel: pagecache_get_page+0x5d1/0x783
kernel: grab_cache_page_write_begin+0x1d/0x3f
kernel: cifs_write_begin+0x7b/0x220 [cifs]
kernel: generic_perform_write+0xd1/0x224
kernel: __generic_file_write_iter+0x113/0x1e2
kernel: cifs_strict_writev+0x227/0x2f0 [cifs]
kernel: new_sync_write+0x146/0x1e6
kernel: vfs_write+0x202/0x291
kernel: ksys_write+0x67/0xe8
kernel: tmhook_invoke+0x48/0xd0 [tmhook]
kernel: tmhook_handler+0x17f/0x224 [tmhook]
kernel: ? do_syscall_64+0x3b/0x8d
kernel: ? tmhook_invoke+0xd0/0xd0 [tmhook]
kernel: do_syscall_64+0x3b/0x8d
kernel: entry_SYSCALL_64_after_hwframe+0x69/0x0

 
* The stack trace entries "tmhook" and "bmhook" are from the anti-virus software "Trend micro". 

SAP system on Linux server

schedule_preempt_disabled, mutex_lock_slowpath, path_openat, lookup_slow, tmhook_invoke, tmhook_handler, bmhook_get_file_info_from_upath, tmbpf_get_file_info_from_upath, bmhook_prehook_handler, tmhook_invoke_prehooks, rwsem_down_write_slowpath , KBA , BC-OP-LNX , Linux , Problem