2587881 - Database hangs {Stack sqlriPea }

Database gets down and generated FODC folder.

• Fodc stack can also be seen in trap
  =======================
  _intel_fast_memcpy
  sqlri_hsjnUnFormatPayload
  sqlriPeaPassAggValues
  sqlriPeaPassNonEmptyRec
  sqlriPeaEOF
  sqlripea
  sqlriNljnNonPiped
  sqlri_hsjnRecP9sqlri_agotb
• Db2diag.log shows the below error
  ========================
  
  <DATE Timestamp> LEVEL: Error
  PID : 15204516 TID : 87899 PROC : db2sysc 0
  INSTANCE: db2<SID> NODE : 000 DB : ***
  APPHDL : 0-***** APPID: *LOCAL.*****************
  AUTHID : ******* HOSTNAME: *********
  EDUID : 87899 EDUNAME: db2agent (<SID>) 0
  FUNCTION: DB2 UDB, oper system services, sqloEDUCodeTrapHandler, probe:30
  MESSAGE : ADM14005E The following error occurred: "AppErr". First Occurrence
  Data Capture (FODC) has been invoked in the following mode:
  "Automatic". Diagnostic information has been recorded in the
  directory named
  "/db2/<SID>/db2dump/FODC_AppErr_2<DATE Timestamp>/".
  
  OR
  
  

• 2018-05-17-16.30.00.842108+000 E127918563E522 LEVEL: Warning
  PID : ******** TID : ************* PROC : db2sysc 0
  INSTANCE: db2*** NODE : 000 DB : ***
  APPHDL : ************* APPID: ************************
  AUTHID : DB2*** HOSTNAME: sap-***01
  EDUID : 189 EDUNAME: db2agent (***) 0
  FUNCTION: DB2 UDB, runtime interpreter, sqlriPEXInitHashTable, probe:3848
  MESSAGE : Partial Early Aggregation/Distinct running with reduced memory

  2018-05-17-16.30.11.715886+000 I127919086E562 LEVEL: Severe
  PID : ******** TID : ************* PROC : db2sysc 0
  INSTANCE: db2*** NODE : 000 DB : ***
  APPHDL : ************* APPID: ************************
  AUTHID : DB2*** HOSTNAME: sap-***01
  EDUID : 189 EDUNAME: db2agent (***) 0
  FUNCTION: DB2 UDB, runtime interpreter, sqlriSectRelink, probe:577
  DATA #1 : <preformatted>
  Section relink failure: Relink ID 00000000 @ 00007F0882FFA000 to ID A32B5CBF @ 00007F0849D03B40

• FODC trap shows below stacktrace
  =======================
  

  <StackTrace>
  -------Frame------ ------Function + Offset------
  0x000000000000F5A8 ?unknown + 0x0
  0x0900000014B8B2C0 sqlri_hsjnUnFormatPayload__FlP13sqlri_hsjnRecP16sqlri_sqzo_value + 0xA8
  0x09000000120D81A8 sqlriPeaEOF__FP8sqlrr_cbP9sqlri_agob + 0x640
  0x09000000120D7A54 sqlripea1__FP8sqlrr_cb + 0x320
  0x0900000014920FC0 sqlrievl__FP8sqlrr_cb + 0x18
  0x0900000014920FC0 sqlrievl__FP8sqlrr_cb + 0x18
  0x09000000147829BC sqlriSectInvoke__FP8sqlrr_cbP12sqlri_opparm + 0x8C
  0x09000000144216F0 sqlrr_execute__FP14db2UCinterfaceP9UCstpInfo + 0x15D8
  0x0900000014420B68 sqlrr_execute__FP14db2UCinterfaceP9UCstpInfo + 0xA50
  0x0900000014919C78 executeSection__10pvmPackageFP5sqlcaUib + 0x368
  0x090000001491A770 executeQuery__3PVMFUib + 0xFC
  0x090000001491AA38 run__3PVMFv + 0x11C
  0x0900000014326928 sqloInvokeFnArgs + 0x34C
  0x090000001432628C sqlriInvokeInvoker__FP10sqlri_ufobb + 0x1868
  0x090000001432628C sqlriInvokeInvoker__FP10sqlri_ufobb + 0x1868
  0x0900000014324620 sqlriInvokeInvoker__FP10sqlri_ufobb + 0x584
  0x0900000014324058 sqlricall__FP8sqlrr_cb + 0x204
  0x09000000147829BC sqlriSectInvoke__FP8sqlrr_cbP12sqlri_opparm + 0x8C
  0x090000001441DA9C sqlrr_process_execute_request__FP8sqlrr_cbib + 0xC88
  0x0900000014420730 sqlrr_execute__FP14db2UCinterfaceP9UCstpInfo + 0x618
  0x090000001491343C executeSection__10pvmPackageFP5sqlcaUib + 0x16C
  0x090000001491A770 executeQuery__3PVMFUib + 0xFC
  0x090000001491AA38 run__3PVMFv + 0x11C
  0x0900000014326928 sqloInvokeFnArgs + 0x34C
  0x090000001432628C sqlriInvokeInvoker__FP10sqlri_ufobb + 0x1868
  0x090000001432628C sqlriInvokeInvoker__FP10sqlri_ufobb + 0x1868
  0x0900000014324620 sqlriInvokeInvoker__FP10sqlri_ufobb + 0x584
  0x0900000014324058 sqlricall__FP8sqlrr_cb + 0x204
  0x09000000147829BC sqlriSectInvoke__FP8sqlrr_cbP12sqlri_opparm + 0x8C
  0x090000001441DA9C sqlrr_process_execute_request__FP8sqlrr_cbib + 0xC88
  0x0900000014420730 sqlrr_execute__FP14db2UCinterfaceP9UCstpInfo + 0x618
  0x09000000143372F0 sqljsParseRdbAccessed__FP13sqljsDrdaAsCbP13sqljDDMObjectP14db2UCinterface + 0x974
  0x09000000143336CC @72@sqljsSqlam__FP14db2UCinterfaceP8sqeAgentb + 0x153C
  0x09000000143336CC @72@sqljsSqlam__FP14db2UCinterfaceP8sqeAgentb + 0x153C
  0x09000000143332C0 @72@sqljsSqlam__FP14db2UCinterfaceP8sqeAgentb + 0x1130
  0x090000001582C5F0 @72@sqljsDriveRequests__FP8sqeAgentP14db2UCconHandle + 0xA8
  0x090000001582D100 @72@sqljsDrdaAsInnerDriver__FP18SQLCC_INITSTRUCT_Tb + 0x5F8
  0x0900000013AB7B6C RunEDU__8sqeAgentFv + 0x4953C
  0x0900000013B0BF48 RunEDU__8sqeAgentFv + 0x120
  0x0900000014DBB8F0 EDUDriver__9sqzEDUObjFv + 0x134
  0x0900000013C58D38 sqloEDUEntry + 0x390
  </StackTrace>

• CALLSTCK: (Static functions may not be resolved correctly, as they are resolved to the nearest symbol)
  [0] 0x090000001A024BF0 AgentBreathingPoint__8sqeAgentFi + 0x3E4
  [1] 0x090000001AED7CB0 sqldFetchDirect__FP13SQLD_DFM_WORK + 0x274
  [2] 0x090000001AED487C sqldDataFetch__FP8SQLD_CCBP8SQLZ_RIDUlP7SQLI_CB + 0x9E0
  [3] 0x090000001B378734 @161@sqliScanLeaf2NoLocking__FP7SQLI_CBP11SQLI_SFGLOB + 0xB0
  [4] 0x090000001AECE650 sqldRowFetch__FP8sqeAgentP8SQLD_CCBUlT3PP10SQLD_VALUEP8SQLZ_RIDT3P12SQLD_ID_LISTP9SQLP_LSN8 + 0x1518
  [5] 0x090000001AED2B5C sqldRowFetch__FP8sqeAgentP8SQLD_CCBUlT3PP10SQLD_VALUEP8SQLZ_RIDT3P12SQLD_ID_LISTP9SQLP_LSN8 + 0x3F44
  [6] 0x090000001AEDD3FC sqlriNljnNonPiped__FP8sqlrr_cb + 0xA74
  [7] 0x090000001AEDD3FC sqlriNljnNonPiped__FP8sqlrr_cb + 0xA74
  [8] 0x090000001AEDD3FC sqlriNljnNonPiped__FP8sqlrr_cb + 0xA74
  [9] 0x090000001AEDCC94 sqlriNljnNonPiped__FP8sqlrr_cb + 0x30C
  [10] 0x090000001AEDC35C sqlriSectInvoke__FP8sqlrr_cbP12sqlri_opparm + 0xA8
  [11] 0x090000001B19A708 sqlrr_process_fetch_request__FP14db2UCinterface + 0x19C
  [12] 0x090000001B1B1D0C sqlrr_open__FP14db2UCinterfaceP15db2UCCursorInfo + 0x270
  [13] 0x090000001AC9AE90 sqljsParseRdbAccessed__FP13sqljsDrdaAsCbP13sqljDDMObjectP14db2UCinterface + 0x16D8
  [14] 0x090000001ACA4F54 @72@sqljsSqlam__FP14db2UCinterfaceP8sqeAgentb + 0x10F0
  [15] 0x090000001ACA4F54 @72@sqljsSqlam__FP14db2UCinterfaceP8sqeAgentb + 0x10F0
  [16] 0x090000001ACA4B48 @72@sqljsSqlam__FP14db2UCinterfaceP8sqeAgentb + 0xCE4
  [17] 0x090000001C3B9560 @72@sqljsDriveRequests__FP8sqeAgentP14db2UCconHandle + 0xA8
  [18] 0x090000001C3BA074 @72@sqljsDrdaAsInnerDriver__FP18SQLCC_INITSTRUCT_Tb + 0x600
  [19] 0x090000001B9B6890 RunEDU__8sqeAgentFv + 0x4CB0C
  [20] 0x090000001B9EA15C RunEDU__8sqeAgentFv + 0x124
  [21] 0x090000001A87A328 EDUDriver__9sqzEDUObjFv + 0x134
  [22] 0x090000001A536FB8 sqloEDUEntry + 0x394
  [23] 0x0900000000547E10 _pthread_body + 0xF0
  [24] 0xFFFFFFFFFFFFFFFC ?unknown + 0xFFFFFFFF
•  Another Stack
  ==============
  0x00007F08BE5344A5 _Z25ossDumpStackTraceInternalmR11OSSTrapFileiP7siginfoPvmm + 0x0385
  (/db2/db2<sid>/sqllib/lib64/libdb2osse.so.1)
  0x00007F08BE5340AC ossDumpStackTraceV98 + 0x002c
  (/db2/db2<sid>/sqllib/lib64/libdb2osse.so.1)
  0x00007F08BE53017D _ZN11OSSTrapFile6dumpExEmiP7siginfoPvm + 0x00fd
  (/db2/db2<sid>/sqllib/lib64/libdb2osse.so.1)
  0x00007F08C4A60FF4 sqlo_trce + 0x0404
  (/db2/db2<sid>/sqllib/lib64/libdb2e.so.1)
  0x00007F08C4AEA2AC sqloEDUCodeTrapHandler + 0x02ac
  (/db2/db2<sid>/sqllib/lib64/libdb2e.so.1)
  0x00000039D700F7E0 address: 0x00000039D700F7E0 ; dladdress: 0x00000039D7000000 ; offset in lib: 0x000000000000F7E0 ;
  (/lib64/libpthread.so.0)
  0x0000000000426EA1 __intel_ssse3_rep_memcpy + 0x1951
  (db2sysc 0)
  0x00007F08C5AE8C19 _Z25sqlri_hsjnUnFormatPayloadlP13sqlri_hsjnRecP16sqlri_sqzo_value + 0x0079
  (/db2/db2<sid>/sqllib/lib64/libdb2e.so.1)
  0x00007F08C2DF2199 _Z21sqlriPeaPassAggValuesP8sqlrr_cbP13sqlri_hsjnRecP9sqlri_agotb + 0x0069
  (/db2/db2<sid>/sqllib/lib64/libdb2e.so.1)
  0x00007F08C4F0FE3B _Z11sqlriPeaEOFP8sqlrr_cbP9sqlri_agob + 0x037b

•  Db2 for LUW 10.5 FP7 and above.
•  SAP ERP 6.0 / 740/ 731

Db2dump , Database hangs, shut down, MEMORY CORRUPTION, DB2_REDUCED_OPTIMIZATION , KBA , BC-DB-DB6 , DB2 Universal Database for Unix / NT , Problem