2884635 - IPHR in-memory tracing in systems under stress can result in a range of performance issues, hangs and associated problems - SAP IQ

• To qualify and a match for this issue the IQ system must be running with the IPHR in-memory trace flags -iqiphrtracing   and   -iq_thread_msg_buf_size   set. If this is not the case then this KBA is not applicable.
  
  
• Thus far this  issue has only been reported on multiplex systems (MPX)  running with a  heavy transaction load and under stress.  It is thought to be predominantly a MPX associated issue.   However it is anticipated that simplex servers may also suffer some of the non MPX specific symptoms listed below .

• In multiplex systems the internal event driven  stored procedure  ev_iqmpxstatus running in the Coordinator (CN)  becomes “stuck” and appears to hang.  This can be seen in the iqmsg file for the CN,  usually the event can run as frequently as once a minute and under normal conditions you see a message pairing like the following …

  I. 01/06 05:54:03. 0000000005 ev_iqmpxstatus started
  I. 01/06 05:54:03. 0000000005 [ev_iqmpxstatus] 0.017986 pass: 0 versionsApplied: 10 LWT: 0.003637 oldTCL: 8 nowTCL: 8

  However when it is stuck only the  “ev_iqmpxstatus started” message is seen then nothing further until the SAP IQ server is restarted.   The “stuck” ev_iqmpxstatus procedure can also be evident when examining the output from stored procedures like sp_iqtransaction.

• Excessive transaction Version build up that can lead to IQ_SYSTEM_MAIN filling up and associated errors and failures this may bring.  Errors associated with this condition include but may not be limited to
• Coordinator is offline
• Failed to resume INC
• Dbspace IQ_SYSTEM_MAIN is OUT OF SPACE
• Server emergency shutdown: tlvlog RW error
• ErrID: 12291 (inc_CommandHandlerException); SQLCode: -1004008, SQLState: 'QIA08', Severity: 14
• GLOBAL ROLLBACK FAILURE
  
  

• A general performance degrade is seen on a range of select or update SQL queries.
  
  
  
• Performance of the DBCC sp_iqcheckdb consistency checker may also degrade significantly.
  
  
  
• The SAP IQ server process hangs during normal shutdown.  The “Database server shutdown requested by DBSTOP” , “Starting shutdown of database” and final checkpoint messages are seen but the server process does not end and has to be manually killed.
  
  
  
• The IQ server restart after hitting this problem may also hang during recovery restart processing.  For MPX systems a forced restart in simplex mode using the -iqmpx_sn 1   startup switch may be  required.
  
  
  
• Thread pstack traces taken of the SAP IQ server process while the ev_iqmpxstatus procedure is “stuck” will show  at least one thread processing calls at  and around the st_txncb::Combine() method , for example …

v2s_bmcontext::InitializeNextSliceReadIterator
v2s_bmcontext::bmlffndhole
v2s_bmcontext::bmfndhole
s_bm::FindHole
s_bm::PrintRange
st_txncb::Combine
st_txnMgr::CombineTransactions
st_txnMgr::DropVersions
st_txnMgr::ProcessQueryActiveInternal
st_txnMgr::ProcessQueryActive()
db_iqutilityCatRepl::Execute()

• There have been some reports of increased likely hood of the IPHR tracing issue also resulting in "Bitmap identity damaged or version" errors as reported in CR 820764

Applies SAP IQ versions that include the in-memory IPHR tracing feature

• SAP IQ 16.0 SP11 PL22
  
  
• SAP IQ 16.1 SP02 PL14
  
  
• SAP IQ 16.1 SP03 PL04
  
  
• SAP IQ 16.1 SP04 PL01

sybase, hung, inmemory, tracing, CR820967, CR#820967, 820967, stacktrace, stktrc, stack trace, slow, s_inccommunicationfailure, '-1004008', inc_PoolHeartbeat, inc_CommandHandler, abort, inconsistent,s_blockmap, exhaustion , CR820764, CR#820764, 820764, , KBA , BC-SYB-IQ , Sybase IQ , Problem