2583839 - Bufman: Incorrect page header read s_corruptdiskblkhdrexception errors that spread - SAP IQ

• Affects only SAP IQ multiplex systems where update type DML operations are done in a secondary writer node.
  
  
• The problem is first seen as a single query failure reporting a main Bufman incorrect page header read  (IPHR) error and exception at s_corruptdiskblkhdrexception.
  

  main Bufman: Incorrect page header read; from slib/s_trycatch.cxx:2928

  There was an s_buf::Read() error during s_bufman::Find().

  Exception Thrown from slib/s_blockmap.cxx:5782

  O/S Err#: 0, ErrID: 2100 (s_corruptdiskblkhdrexception); SQLCode: -1009039, SQLState: 'QSA39', Severity: 14

  main Bufman: An error was detected on a database page. You may have a damaged index.
  
  
  

• The problem will appear to spread to other tables and SQL transactions.  Multiple differing IPHR errors start to be seen and various stack traces start occuring which may then lead to server crashes. Three basic types of IPHR may be seen in the iqmsg file...
  
  
• Tried to read a versioned object but read all 0s, as show by the "head:"" details are all zeros instead of matching the "expected:"
  

  main Bufman: Incorrect page header read; from slib/s_trycatch.cxx:2928

  expected: btype=14,pbn=17135834,txn=721497352,sp=14,nlb=16,npb=16,iscmp=0,uid=130767293,blk=17135834,mn=2242,upmn=2242,flgs=0x80000008

  head: btype=0,pbn=0,txn=0,sp=0,nlb=0,npb=0,iscmp=0,ctr=0,cta=0,ver=0,pb=0,pus=0

• Tried to read a versioned object but read complete garbage, as shown seeing huge txn and pbn numbers in the "head:" information.
  

  expected: btype=14,pbn=544405863,txn=721498621,sp=2,nlb=16,npb=16,iscmp=0,uid=129875876,blk=544405863,mn=21252,upmn=21252,flgs=0x80008008

  head: btype=3856,pbn=1085384071842893584,txn=1085385102366609168,sp=252710672,nlb=3856,npb=3856,iscmp=0,ctr=16,cta=15,ver=16,pb=15,pus=3856

• Tried to read a versioned object but got another object written by a later transaction. The "expected:" object has a correct or reasonable non-zero txn number however the read txn number in the "head:" is a larger number than expected.
  

  expected: btype=14,pbn=506687971,txn=721497984,sp=4,nlb=16,npb=16,iscmp=0,uid=130535688,blk=506687971,mn=81756,upmn=81756,flgs=0x80000008

  head: btype=5,pbn=506687971,txn=728399990,sp=1,nlb=16,npb=3,iscmp=1,ctr=136,cta=134,ver=1,pb=11,pus=3
  
  
  

• No noticeable initial root cause, or hardware faults, system aborts, or problems with softlinks,  or dbfile disk header information errors can be found.
  
  
• The problem is associated with MPX recovery events that can be seen when the corodinator node (CN)  has crashed or shutdown, so in the lead up to the issue these types of messages should be seen in the secondary nodes iqmsg file...
  

  st_database::CompleteMpxRecovery() - change RecoveryState from RECOVERED to RECOVERING

  st_database::CompleteMpxRecovery() - reset RecoveryState to RECOVERED

•  SAP IQ 16.x  multiplex systems on all platforms

sybase, bufferman, corruption, abort, fail, crash, upgrade, CR811872, CR#811872, 811872, co-ordinator, stktrc, 'stack trace', checkdb, dbcc, iq16, disc, querry, blockmap, '16.1', worse , KBA , BC-SYB-IQ , Sybase IQ , Bug Filed