2879089 - Db2 11.1 crashes with FODC_Panic

Your Db2 instance crashes with an FODC Panic dump,  with the db2diag.log showing symptoms of memory corruption in the communication areas between the db2fmp process and the engine.

The db2diag.log in this example may show something like this:

2019-11-03-01.00.00.686236-300 I14315456E8074 LEVEL: Error
PID : 6180 TID : 140709503952640 PROC : db2sysc 6
INSTANCE: db2pw1 NODE : 006 DB : PW1
APPHDL : 0-45613 APPID: *N0.db2pw1.191103060005
AUTHID : DB2PW1 HOSTNAME: fmsbcclprdb05
EDUID : 28616 EDUNAME: db2agntp (PW1) 6
FUNCTION: DB2 UDB, routine_infrastructure, sqlerInvokeFencedRoutine, probe:1200
DATA #1 : String, 37 bytes
Routine failed with termination error
DATA #2 : Hexdump, 1480 bytes
0x00007FFF8E408368 : 5546 4F42 0000 0000 606C 0000 0000 0000 UFOB....`l......
0x00007FFF8E408378 : 0000 0000 0000 0000 50C2 9303 0200 0000 ........P.......
0x00007FFF8E408388 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x00007FFF8E408548 : 0A00 1000 5359 5350 524F 432E 534E 4150 ....SYSPROC.SNAP
0x00007FFF8E408558 : 5F47 4554 5F44 424D 0000 0000 0000 0000 _GET_DBM........
0x00007FFF8E408568 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x00007FFF8E4086B8 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x00007FFF8E4086C8 : 0000 0000 0000 0053 4E41 505F 4745 545F .......SNAP_GET_
0x00007FFF8E4086D8 : 4442 4D5F 4150 0000 0000 0000 0000 0000 DBM_AP..........
0x00007FFF8E4086E8 : 0000 0000 0000 0000 0000 0000 0000 0000 ................

2019-11-03-01.00.00.705507-300 I14323531E441 LEVEL: Severe
PID : 14841 TID : 140737261233952 PROC : db2fmp (
INSTANCE: db2pw1 NODE : 006
HOSTNAME: fmsbcclprdb05
FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadListener, probe:400
RETCODE : ZRC=0xFFFFFB95=-1131
SQL1131N A stored procedure process has been terminated abnormally.
Routine name: "". Specific name: "".

2019-11-03-01.00.00.709449-300 I14323973E442 LEVEL: Warning
PID : 14841 TID : 140737261233952 PROC : db2fmp (
INSTANCE: db2pw1 NODE : 006
HOSTNAME: fmsbcclprdb05
FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadListener, probe:410
RETCODE : ZRC=0xFFFFFB95=-1131
SQL1131N A stored procedure process has been terminated abnormally.
Routine name: "". Specific name: "".

2019-11-03-01.15.02.856122-300 E15788459E2610 LEVEL: Severe
PID : 5824 TID : 140708698646272 PROC : db2sysc 1
INSTANCE: db2pw1 NODE : 001 DB : PW1
APPHDL : 0-46560 APPID: *N0.db2pw1.191103061911
AUTHID : DB2PW1 HOSTNAME: fmsbcclprdb05
EDUID : 28337 EDUNAME: db2agntp (PW1) 1
FUNCTION: DB2 UDB, SQO Memory Management, sqloDiagnoseFreeBlockFailure, probe:999
MESSAGE : Memory validation failure, diagnostic file dumped.
DATA #1 : String, 28 bytes
Corrupt pool free tree node.
DATA #2 : File name, 40 bytes
5824.140708698646272.mem_diagnostics.txt
CALLSTCK: (Static functions may not be resolved correctly, as they are resolved to the nearest symbol)
[0] 0x00007FFFF074F585 _ZN13SQLO_MEM_POOL32diagnoseMemoryCorruptionAndCrashEmPKcb + 0x285
[1] 0x00007FFFF074A698 _ZN13SQLO_MEM_POOL10MemTreeGetEmmPP17SqloChunkSubgroupPj + 0x188
[2] 0x00007FFFF074F73C _ZN13SQLO_MEM_POOL19allocateMemoryBlockEmmjmPP17SqloChunkSubgroupPjP12SMemLogEvent + 0x6C
[3] 0x00007FFFF072C9E0 sqloGetMemoryBlockExtended + 0xFF0
[4] 0x00007FFFEECB97BE _Z14sqlerGetFMPIPCP18SQLCC_INITSTRUCT_Tj + 0x18E
[5] 0x00007FFFEB8676AE _Z12sqlccGetIPCsP18SQLCC_INITSTRUCT_TP26SQLCC_IPC_CON_REQST_DATA_Tb + 0xB7E
[6] 0x00007FFFEECF1AE8 _Z16sqlerAllocFmpIPCPP18SQLCC_INITSTRUCT_TP8sqeAgenti + 0x188
[7] 0x00007FFFEECE6175 _Z22sqlerGetFmpThreadEntryP11sqlerFmpRowP14sqlerFmpHandleP17sqlerFmpParmsBaseb + 0xB75
[8] 0x00007FFFEECE410E _Z19sqlerGetFmpFromPoolP14sqlerFmpHandleP17sqlerFmpParmsBase + 0x11CE
[9] 0x00007FFFEECE148E _Z24sqlerInvokeFencedRoutineP13sqlerFmpParms + 0x5FE
[10] 0x00007FFFF151B949 _Z18sqlriInvokeInvokerP10sqlri_ufobb + 0x4A9
[11] 0x00007FFFF1522F0C _Z8sqlriutfP8sqlrr_cb + 0x79C
[12] 0x00007FFFF1559E2F _Z11sqlri_tfopnP8sqlrr_cbP9sqlri_tao + 0x10F
[13] 0x00007FFFF131142C _Z8sqlriopnP8sqlrr_cbP9sqlri_taoPi + 0x4AC
[14] 0x00007FFFF1485539 _Z7sqlritaP8sqlrr_cb + 0xAC9
[15] 0x00007FFFF13A2447 _Z15sqlriSectInvokeP8sqlrr_cbP12sqlri_opparm + 0x4B7
[16] 0x00007FFFF0DE2D06 _Z16sqlrr_dss_routerP8sqlrr_cb + 0x806
[17] 0x00007FFFF0DDF6C6 _Z21sqlrr_subagent_routerP8sqeAgentP12SQLE_DB2RA_T + 0x1066
[18] 0x00007FFFEE9731AF /db2/db2pw1/sqllib/lib64/libdb2e.so.1 + 0x50211AF
[19] 0x00007FFFEE971A80 _Z21sqleProcessSubRequestP8sqeAgent + 0x3E0
[20] 0x00007FFFEE9A6007 _ZN8sqeAgent6RunEDUEv + 0x5D7
[21] 0x00007FFFF21C0EE6 _ZN9sqzEDUObj9EDUDriverEv + 0x116
[22] 0x00007FFFF0778878 sqloEDUEntry + 0x578
[23] 0x0000003DBA207AA1 /lib64/libpthread.so.0 + 0x7AA1
[24] 0x0000003DB9EE8C4D clone + 0x6D

Db2 11.1 thru 11.1 MP2 FP2 iFix002SAP

snap_get_dbm snap_get_ap memory corruption DSM AtsTask SQL0438N -438 SQL0440N -440 SQL0727N -727 , KBA , BC-DB-DB6 , DB2 Universal Database for Unix / NT , Problem