2247070 - Dropping table after upgrade from 15.4 may hang IQ server

• Dropping table after upgrade from 15.4 to 16 SP10 may hang IQ server. Problem table gets dropped after N attempts. All N-1 attempts result in hang and end-user kills IQ spid to get out of this situation.
  After upgrading 'alter database upgrade' was run. All the indexes on problem table were rebuilt using sp_iqrebuildindex.

• Threads of interest in 2 pstacks collected couple of minutes apart show following threads of interest contributing to hang

----- Thread 1035 does not move between 2 pstacks. Some table is getting dropped. 

Thread 1035 (Thread 0x7f5befc52700 (LWP 33967)):
#0  0x000000342ec0f0e3 in pread64 () from /lib64/libpthread.so.0
#1  0x00007f5beb1137dd in hos_pi_pread(hos_fd, void*, unsigned int, long) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#2  0x00007f5beb123e7c in hos_svbio::Read2(unsigned long long, void*, int, int&, int&, unsigned long long*, s_bufman_errorInfo*) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#3  0x00007f5beb33588c in hos_mvbio::Read2(unsigned long long, void*, int, int&, int&, unsigned long long*, s_bufman_errorInfo*) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#4  0x00007f5bebdf8f97 in s_buf::ReadBufAndVerify(s_bufman_errorInfo*, s_diskblockheader*, unsigned long long, unsigned int, unsigned int, int, short, unsigned long long, unsigned int, int, s_bufmanCallerStats*) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#5  0x00007f5beb4d8a51 in s_buf::Read(s_bufman_errorInfo*, unsigned long long, unsigned int, int, short, unsigned long long, unsigned int, unsigned int, s_blockmap const*, s_bufmanCallerStats*) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#6  0x00007f5beb4fb473 in s_bufpartition::Find(s_bufman_errorInfo*, s_connectionCacheStats*, hos_bio*, unsigned long long, unsigned long long, unsigned long long, unsigned int, unsigned int, int, unsigned int, unsigned int, short, s_blockmap*, s_bufman::s_paging, s_bufman::s_dirtyMode, unsigned int, s_bufmanCallerStats*) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#7  0x00007f5beb43b576 in s_blockmap::Find(unsigned long long, short, int, s_bufmanCallerStats*) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#8  0x00007f5bebc3a2ce in s_pageListCursor::PositionPage(unsigned long long) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#9  0x00007f5bebc3c000 in s_pageListCursor::PositionPrefetchedPage(s_pageListCursor::s_paPosition) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#10 0x00007f5beba47b11 in s_garrayReadCursorV2::ResettoNextPage() () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#11 0x00007f5bebacfa19 in hs_ohcidentity::TryUpgradeIdentity(hs_ohcIdentity_15*, hs_ohc*, hs_context*, s_openmode) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#12 0x00007f5bebad1dd6 in hs_ohc::hs_ohc(db_IndexDef*, s_openmode, hs_dp const*) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#13 0x00007f5bebf3fb98 in db_IndexDef::OpenSIndexWithLock(unsigned int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#14 0x00007f5bec0b460a in db_IndexDef::OpenSIndex(unsigned int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#15 0x00007f5bebf3a9ec in db_IndexDef::DestroyIndex(st_txncb*, int, int, int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#16 0x00007f5bebf3517b in db_destroyTableIter::ExecWork(int&, unsigned int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#17 0x00007f5beb36cc6a in workAllocator::DoWork(unsigned int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#18 0x00007f5beb391c04 in hos_team::StartWork(void (*)(void*, void*), void*) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#19 0x00007f5beb367e21 in hos_team::Exec(hos_workIter&) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#20 0x00007f5bebf27937 in db_TableDef::Destroy(st_txncb*) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#21 0x00007f5bebf16568 in db_Catalog::DropTable(UITable*, unsigned int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#22 0x00007f5bebfbe5b9 in db_droptable::Execute() () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#23 0x00007f5bebe3a8f4 in st_command::DoCmdThroughResourceGate() () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#24 0x00007f5bebfc088a in db_iqddl::DropTable(a_statement const*, UIConnection*, UITable*) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#25 0x00007f5beac41c9d in UIQDDL_DropTable () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
....

----- Some threads are trying to destroy index and have some locks. 2 such groups with locking and waiting.

Threads (13): 581 569 565 550 529 522 517 497 489 487 474 466 438
#3  0x00007f5bebf3bf76 in db_IndexDef::CloseSIndex(s_index*, int)
#4  0x00007f5bebf1912a in db_CatalogSIndexPtr::Close(int)
#5  0x00007f5bebf3aa16 in db_IndexDef::DestroyIndex(st_txncb*, int, int, int)
#6  0x00007f5bebf3517b in db_destroyTableIter::ExecWork(int&, unsigned int)
#7  0x00007f5beb36cc6a in workAllocator::DoWork(unsigned int)
#8  0x00007f5beb393594 in hos_thread::Main()
#9  0x00007f5beb2fc9c8 in hos_lwtask::Start(hos_lwtask*)
#10 0x00007f5e480cb1fd in IQWorkerStarter(void*)
#11 0x00007f5e487d0c2d in run_task_body
#12 0x00007f5e48303967 in UnixTask::pre_body(void*)
#14 0x000000342e8e896d in clone

Threads (2): 564 494
#3  0x00007f5beafcd430 in hos_mutex::Lock() const
#4  0x00007f5beafcd48f in hos_mutex_lock::Lock()
#5  0x00007f5beafcd510 in hos_mutex_lock::hos_mutex_lock(hos_mutex const*, hos_mutex_lock::LockStatus)
#6  0x00007f5bec0b435b in db_IndexDef::OpenSIndex(unsigned int)
#7  0x00007f5bebf3a9ec in db_IndexDef::DestroyIndex(st_txncb*, int, int, int)
#8  0x00007f5bebf3517b in db_destroyTableIter::ExecWork(int&, unsigned int)
#9  0x00007f5beb36cc6a in workAllocator::DoWork(unsigned int)
#10 0x00007f5beb393594 in hos_thread::Main()
#11 0x00007f5beb2fc9c8 in hos_lwtask::Start(hos_lwtask*)
#12 0x00007f5e480cb1fd in IQWorkerStarter(void*)
#13 0x00007f5e487d0c2d in run_task_body
#14 0x00007f5e48303967 in UnixTask::pre_body(void*)
#16 0x000000342e8e896d in clone

---- Sample thread 581 with CloseSIndex lock/wait

Thread 581 (Thread 0x7f5bdc2b9700 (LWP 34422)):
#0  0x000000342ec0e2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x000000342ec09588 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x000000342ec09457 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f5bebf3bf76 in db_IndexDef::CloseSIndex(s_index*, int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#4  0x00007f5bebf1912a in db_CatalogSIndexPtr::Close(int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#5  0x00007f5bebf3aa16 in db_IndexDef::DestroyIndex(st_txncb*, int, int, int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#6  0x00007f5bebf3517b in db_destroyTableIter::ExecWork(int&, unsigned int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#7  0x00007f5beb36cc6a in workAllocator::DoWork(unsigned int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
...

---- Sample thread 564 with OpenSIndex lock/wait 

Thread 564 (Thread 0x7f5bdba28700 (LWP 34439)):
#0  0x000000342ec0e2e4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x000000342ec09588 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x000000342ec09457 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f5beafcd430 in hos_mutex::Lock() const () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#4  0x00007f5beafcd48f in hos_mutex_lock::Lock() () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#5  0x00007f5beafcd510 in hos_mutex_lock::hos_mutex_lock(hos_mutex const*, hos_mutex_lock::LockStatus) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#6  0x00007f5bec0b435b in db_IndexDef::OpenSIndex(unsigned int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#7  0x00007f5bebf3a9ec in db_IndexDef::DestroyIndex(st_txncb*, int, int, int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#8  0x00007f5bebf3517b in db_destroyTableIter::ExecWork(int&, unsigned int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
#9  0x00007f5beb36cc6a in workAllocator::DoWork(unsigned int) () from /sybase/IQ-16_0/bin64/../lib64/libiq16.so
...

SAP IQ 16 SP10+

Hang ; pread64 ;  hos_pi_pread ; unresponsive ; hs_ohcidentity::TryUpgradeIdentity ; s_buf::ReadBufAndVerify ; drop table , KBA , BC-SYB-IQ , Sybase IQ , Bug Filed