3700585 - Server hung and not accepting connections - SAP IQ

• The SAP IQ server version 16.1 SP05.PL12 experienced a hang and was not accepting connections for a while.
• The server recovered without requiring a reboot.
• During the hang, a series of gstacks were collected.
• Below some possible symptoms:

.iqmsg shows:
 
I. 12/15 10:49:56. 0000123134 140683893389056 s_buf::WaitForSignal_DontCareIfBusy- Timed out on _condvar, timeout value= 10 secondssbuf= 0x7ff1d43c1500 btype= 12 Line 2259533: 
I. 12/15 10:55:34. 0000123520 140683986728704 s_buf::WaitForSignal_DontCareIfBusy- Timed out on _condvar, timeout value= 10 secondssbuf= 0x7ff19dffc1b0 btype= 13 Line 2260881: 
I. 12/15 10:58:33. 0000000000 140679799039744 s_buf::WaitForSignal_DontCareIfBusy- Timed out on _condvar, timeout value= 10 secondssbuf= 0x7ff152f06b00 btype= 7
….
 
Srvlog:
 
I. 12/15 10:49:52.924192 Disconnected TCPIP client's AppInfo: HOST=blkcqar011;EXE=USSS
I. 12/15 10:49:56.018133 Connection terminated abnormally
I. 12/15 10:49:56.018277 Disconnected TCPIP client's AppInfo: HOST=blkcqar011;EXE=USSS
I. 12/15 10:50:05.342522 Connection terminated abnormally
I. 12/15 10:50:05.342670 Disconnected TCPIP client's AppInfo: HOST=blkcetl001;PID=403109;EXE=PowerMart
I. 12/15 10:50:25.260203 Connection terminated abnormally
I. 12/15 10:50:25.260283 Disconnected TCPIP client's AppInfo: HOST=blkcetl001;PID=373673;EXE=PowerMart
I. 12/15 10:50:32.654373 Connection terminated abnormally
I. 12/15 10:50:32.654521 Disconnected TCPIP client's AppInfo: HOST=blkcetl001;PID=403109;EXE=PowerMart
……..
……..
……..
I. 12/15 11:20:50.629862 Connection terminated abnormally
I. 12/15 11:20:50.629898 Disconnected TCPIP client's AppInfo: IP=45.32.160.175;HOST=blkcqap007;OSUSER=zzfiag;OS='Linux 3.10.0-1160.139.1.el7.x86_64 #1 SMP Mon Sep 8 06:49:08 EDT 2025 x';EXE=/GAAS/prod/FIAG/cage/apf/deployment_root/envs/sfi_2024.0/bin/python;PID=0xa069;THREAD=0x7f78f7597740;VERSION=16.0.0.3406;API=CAPI_PYTHON;TIMEZONEADJUSTMENT=-480
W. 12/15 11:21:41.265082 Task 0x7222c0(Request task 718) is trying get forbid mutex held by task 0x719cb0(Request task 685) for more than 60000 ms
W. 12/15 11:21:41.265242 Task 0x719cb0(Request task 685) is trying get forbid mutex held by task 0x71e5d0(Request task 703) for more than 60000 ms
W. 12/15 11:21:41.265393 Task 0x719cb0(Request task 685) state: OS thread id=0x7ff38894f700
W. 12/15 11:21:41.265501 Task 0x71e5d0(Request task 703) state: OS thread id=0x7ff385ccd700
W. 12/15 11:21:41.265574 Task 0x71e5d0(Request task 703) state: worker=0x7ff4c4001000 wakend=0 waiting=0
W. 12/15 11:21:41.265607 Worker 0x7ff4c4001000 state: wakend=0 waiting=0 in_atomic=0 is_dedicated=0 in_critical_section=0 allow_forbid=0
W. 12/15 11:21:41.265646 Task 0x719cb0(Request task 685) state: worker=0x7ff470001000 wakend=0 waiting=0
W. 12/15 11:21:41.265723 Worker 0x7ff470001000 state: wakend=0 waiting=0 in_atomic=0 is_dedicated=0 in_critical_section=0 allow_forbid=0
I. 12/15 11:22:01.077345 Connection terminated abnormally
I. 12/15 11:22:01.077454 Disconnected TCPIP client's AppInfo: HOST=blkcdbs004;PID=67306;EXE=PDBA SQL Capture
……..
……..
……..
 
Gstack:
 
Thread 3626 (Thread 0x7ff38894f700 (LWP 52534)):
#0  0x00007ffff5f0fa35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ffff6da3250 in UnixTask::wait(unsigned short) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#2  0x00007ffff6da07f6 in CondVar::wait(Mutex&) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#3  0x00007ffff6b07567 in WorkerPool::join(Worker*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#4  0x00007ffff6b0a41b in Worker::access_db(Database*, a_db_state) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#5  0x00007ffff6d674b3 in SetDBForConnection(a_strm_conn*, unsigned int, Worker*, Database**, a_string_parms*, unsigned int*, unsigned int*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#6  0x00007ffff6d6c0bc in db__connect(an_sqlpres_receive*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#7  0x00007ffff6d60143 in RequestProcedure::call() () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#8  0x00007ffff6b03a41 in Worker::spawn(Procedure*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#9  0x00007ffff6d63d56 in EngStream::handle_ind(unsigned char, unsigned int) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#10 0x00007ffff6d6588f in EngStream::execute() () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#11 0x00007ffff6d65a0d in RQConnItem::do_request() () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#12 0x00007ffff6d5d7ae in RQBaseItem::do_work(Worker*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#13 0x00007ffff6da14b7 in RequestQueue::worker_body() () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#14 0x00007ffff6d5eb69 in request_task(void*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#15 0x00007ffff71e982e in run_task_body () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#16 0x00007ffff6da3118 in UnixTask::pre_body(void*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#17 0x00007ffff5f0bea5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007ffff5426b0d in clone () from /lib64/libc.so.6
------

This is similar to CR 830456.

In gstack output, we can also see this stack:

=====

Thread 3672 (Thread 0x7ff38f61b700 (LWP 44512)):
#0  0x00007ffff5f1254d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ffff5f0de9b in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007ffff5f0dd68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007ff37b84f3e1 in db_Catalog::FindTableVersion(st_connection*, UITable*, unsigned int, unsigned int, unsigned int) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#4  0x00007ff37b852be9 in db_Catalog::FindTableByITable(UITable*, unsigned int, unsigned int, unsigned int) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#5  0x00007ff37a931a42 in opt_TableReference::InitTableColumns(df_Heap*, opt_Select*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#6  0x00007ff37a916679 in opt_Select::InitializeTables() () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#7  0x00007ff37a916333 in opt_Select::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#8  0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#9  0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#10 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#11 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#12 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#13 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#14 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#15 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#16 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#17 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#18 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#19 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#20 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#21 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#22 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#23 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#24 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#25 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#26 0x00007ff37a935d59 in opt_Union::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#27 0x00007ff37a916374 in opt_Select::InitializeNestedQueries(opt_QueryExp*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#28 0x00007ff37a8768a5 in opt_Driver::GenerateTree(QueryNode*, st_cursor*, a_statement const*, UITable*, UIIndex*, s_bm*, unsigned int*, db_RebuildIndexInfo**, int, a_table*, hos_vector_noerror*, hs_store*, int, db_IQParallelIndexInfo*, rv_MergeCmdState*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#29 0x00007ff37b786bcf in st_cursorIQtoSA::Parse() () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#30 0x00007ff37b78424f in st_cursorIQtoSA::Describe() () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#31 0x00007ff37a522b90 in df_DescribeCursor::Parse() () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#32 0x00007ff37b7830ff in st_command::DoCmdThroughResourceGate() () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#33 0x00007ff37b7cd763 in st_iqdml::DescribeCursor(a_statement*, UIConnection*, UICursor*, a_cursor_spec*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#34 0x00007ff37a4cc884 in UIQDML_DescribeCursor () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#35 0x00007ff37b7d4feb in st_SAIQInterfaceInfo::callFunction() () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#36 0x00007ff37b83b61e in st_SAIQInterface::RunIQFunc(st_SAIQInterfaceInfo*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#37 0x00007ff37b7d5493 in st_SAIQInterface::Execute(int, void*, unsigned int (*)(void*, void*), void*, unsigned int) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#38 0x00007ff37abffb5f in saint_iqthresholddml::DescribeCursor(a_statement*, IConnection*, ICursor*, a_cursor_spec*) () from /usr/local/sybase16/IQ-16_1/lib64/libiq16.so
#39 0x00007ffff6bd2d18 in MakeExternalTabdefs(a_db_cursor*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#40 0x00007ffff6bd3cbf in MakeExternalCursor(a_db_cursor*, an_isolation_level, a_cursor_spec*, a_statement*, unsigned int, unsigned int, unsigned int) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#41 0x00007ffff70d22b9 in MakeVolcanoOptimizerCursor(IDBHeap*, a_statement*, an_isolation_level, a_cursor_spec*, unsigned int, unsigned int, PlanInfo*, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, df_InternalCursor*, dfv_HostVars**, a_set_of_row_descriptors**, a_query_diagnostics*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#42 0x00007ffff6bd3729 in dbi_partialopen(IDBHeap*, a_statement*, unsigned int, a_cursor_spec*, PlanInfo*, unsigned int, unsigned int, df_InternalCursor*, a_query_diagnostics*, unsigned int) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#43 0x00007ffff6d4fdbf in a_plan_cache::OpenPartial(a_stmt*, a_statement*, unsigned int, a_cursor_spec*, PlanInfo*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#44 0x00007ffff6d3d21e in dbi_describe_statement(a_stmt*, unsigned short, unsigned short, a_cursor_spec*, unsigned int*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#45 0x00007ffff6d3d9cd in db__describe_output(Connection*, an_sqlpres_receive*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#46 0x00007ffff6d5fe76 in RequestProcedure::call() () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#47 0x00007ffff6b03a41 in Worker::spawn(Procedure*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#48 0x00007ffff6d63d56 in EngStream::handle_ind(unsigned char, unsigned int) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#49 0x00007ffff6d6588f in EngStream::execute() () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#50 0x00007ffff6d65a0d in RQConnItem::do_request() () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#51 0x00007ffff6d5d7ae in RQBaseItem::do_work(Worker*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#52 0x00007ffff6da14b7 in RequestQueue::worker_body() () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#53 0x00007ffff6d5eb69 in request_task(void*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#54 0x00007ffff71e982e in run_task_body () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#55 0x00007ffff6da3118 in UnixTask::pre_body(void*) () from /usr/local/sybase16/IQ-16_1/lib64/libiqserv16_r.so
#56 0x00007ffff5f0bea5 in start_thread () from /lib64/libpthread.so.0
#57 0x00007ffff5426b0d in clone () from /lib64/libc.so.6

======

Not sure but it seems RLV object.

SAP IQ 16.1 SP05 PL12.

server hang, hung, busy, Connection terminated abnormally, is trying get forbid mutex held by task. , KBA , BC-SYB-IQ , Sybase IQ , Problem