3235170 - After upgraded from IQ16.1 SP02.11 to IQ16.1 SP04.10, appeared Dynamic collision on a Writer and corruptions on objcets -- SAP IQ

After upgrading from IQ16.1 SP02.11 to  IQ16.1 SP04.10, a Writer node hit Dynamic collision and Emergency shutdown. After the error on the Writer,  CN encountered corruptions with "main Bufman: Incorrect page header read" appear and went to crash.

Before the upgrade, customer checked the database and there were no errors. After the upgrade, the database worked fine for a day, then the node crashed with an error: MPX dynamic collision. As a result, found corruptions on objects. Customer had to restore DB from a backup.

Following is sequences of events by chronological order;

1- Writer hits a dynamic collision and an Emergency Shutdown was required:

...
I. 07/17 11:17:49. 0000125922 -- Too many retries on dropping user connections. Server is shutting down.
I. 07/17 11:17:49. 0000125922 ***Emergency Shutdown***
I. 07/17 11:17:49. 0000125922  I. 07/17 11:17:49. Exception Thrown from dblib/db_iqutility.cxx:5351, Err# 46, tid 34 origtid 34
I. 07/17 11:17:49. 0000125922    O/S Err#: 0, ErrID: 1025 (db_catalogException); SQLCode: -1009413, SQLState: 'QCB11', Severity: 23
I. 07/17 11:17:49. 0000125922 [21051]: Secondary server cannot maintain catalog sync. 
-- (dblib/db_iqutility.cxx 5351) 

2- CN hits corruption

...
I. 07/17 11:50:58. 0000101953 [QMap] s_dbmvbio::DeAllocate(0x7fe1285df618): Freelist blocks are already deallocated.  beforeCount =856193904 afterCount =856193904
...
I. 07/17 11:51:02. 0000102190 \{140609131992832\} main Bufman: Incorrect page header read; from slib/s_trycatch.cxx:3175 into buffer 0x7fe0b1641280
I. 07/17 11:51:02. 0000102190 \{140609131992832\} expected: btype=14,pbn=758364152,txn=3316404817,sp=14,nlb=16,npb=16,...

3- Writer hits corruptions

...
I. 07/17 11:54:51. 0000000945 Txn 3316421769 0 3316410374
I. 07/17 11:54:51. 0000000444 \{139924300134144\} main Bufman: Incorrect page header read; from slib/s_trycatch.cxx:3175 into buffer 0x7f49e0236b00
I. 07/17 11:54:51. 0000000444 \{139924300134144\} expected: btype=14,pbn=377144580,txn=3316410479,sp=7,nlb=16,npb=16,...

4- CN hits corruption again and crash (down)

...
I. 07/17 12:22:09. 0000104520 s_blockmap: in use on destruction
I. 07/17 12:22:09. 0000104520 Blockmap Member Variables for 0x7f42a0fc88b8
I. 07/17 12:22:09. 0000104520 From slib/s_ohcidx.cxx:7178
...
I. 07/17 12:22:09. 0000104520 **************************************************
I. 07/17 12:22:09. 0000104520 ***   SAP IQ Abort:
I. 07/17 12:22:09. 0000104520 ***      From:  slib/s_blockmap.cxx:1399
...
I. 07/17 12:22:09. 0000104520 **************************************************

5- As CN was down, writer lost communication with CN

...
I. 07/17 12:24:09. 0000005036 GThrow sqlcode: -1004011 from dblib/db_catalog.cxx:845 Args:  "Database server not found"
I. 07/17 12:24:09. 0000005036  I. 07/17 12:24:09. Exception Thrown from dblib/db_catalog.cxx:845, Err# 5, tid 5 origtid 5
I. 07/17 12:24:09. 0000005036    O/S Err#: 0, ErrID: 12291 (inc_CommandHandlerException); SQLCode: -1004012, SQLState: 'QIA12', Severity: 14
I. 07/17 12:24:09. 0000005036 [21223]: INC communication between coordinator and secondary failed due to : Database server not found 
-- (dblib/db_catalog.cxx 845) 

6- Writer needs to shutdown itself to avoid inconsistency with the CN

...
I. 07/17 12:32:37. 0000004959  I. 07/17 12:32:37. Exception Thrown from stcxtlib/st_txnMgrn.cxx:603, Err# 5, tid 2 origtid 2
I. 07/17 12:32:37. 0000004959    O/S Err#: 0, ErrID: 12291 (inc_CommandHandlerException); SQLCode: -1006274, SQLState: 'QBC74', Severity: 14
I. 07/17 12:32:37. 0000004959 [22036]: Transaction cannot continue due to unrecoverable INC failure. Roll back current transaction. 
-- (stcxtlib/st_txnMgrn.cxx 603) 
I. 07/17 12:32:37. 0000004959 GLOBAL ROLLBACK FAILURE
...
I. 07/17 12:32:37. 0000005432 IQ Checkpoint position: 0
I. 07/17 12:32:37. 0000005432 Chk ShutDQPCxt
I. 07/17 12:32:37. 0000005432 Chk
I. 07/17 12:32:37. 0000005432 ChkDone [NumTxnCP: 0]
I. 07/17 12:32:37. 0000005432 IQ Checkpoint block: 1042082
I. 07/17 12:32:37. 0000005432 PostChk
I. 07/17 12:32:37. 0000000000 CloseDatabase

SAP IQ16.1 SP04.X

"maintain catalog sync", s_trycatch, s_blockmap, "INC communication", "INC failure" , , KBA , BC-SYB-IQ , Sybase IQ , Problem