2636316 - Signal 11 in mem__pageallocate_this_engine() when using the hashbytes() built-in function with the MD5 algorithm - SAP ASE

• Memory corruption in the procedure cache memory pool or engine local cache.
   
• A connection unexpectedly terminates and the errorlog contains a signal 11 error in the routine mem__pageallocate_this_engine() together with a stacktrace:

Current process (0x290015) infected with signal 11 (SIGSEGV)
Address 0x0x0000000001d2c2d2 (mem__pageallocate_this_engine+0x342), siginfo (code, address) = (1, 0x0x0000004887140008)

• The stacktrace includes functions: 

mem__pageallocate_this_engine
mem_pageallocate)                 
mem_getpages
memalloc
hashbifs_md5_lerun

  Note: A complete stack trace is available under Attachments.

• The errorlog may also include messages which start with ELC_VALIDATION_ERROR.
   
• The ASE may also shutdown after a message saying it was holding Spinlock eresourcep->erprocelc_spin, for example:

Spinlocks held by  kpid 2686997
Spinlock eresourcep->erprocelc_spin at address 0x00000001451b7880 owned by 2686997

• Built-in function hashbytes() with algorithm MD5 is used.  Some examples:
• use  the replication utility rs_subcmd with FACTCMP='Y'
• select hashbytes('md5', ......)
   
• ASE version is between 16.0 SP03 and 16.0 SP03 PL04.

Read more...

• SAP Adaptive Server Enterprise (ASE) 15.7
• SAP Adaptive Server Enterprise (ASE) Cluster Edition (CE) 15.7
• SAP Adaptive Server Enterprise (ASE) Cluster Edition (CE) 16.0
• SAP Adaptive Server Enterprise (ASE) 16.0 SP03 - 16.0 SP03 PL04

SAP Adaptive Server Enterprise 15.7 ; SAP Adaptive Server Enterprise 15.7 Cluster Edition ; SAP Adaptive Server Enterprise 16.0 ; SAP Adaptive Server Enterprise 16.0 Cluster Edition ; Sybase Adaptive Server Enterprise Cluster Edition 15.7

CR814837, CR#814198, 814198, SEGV, segmentation, storage, access, violation, fault, segfault, kisignal, stacktrace,  stack, registers , KBA , BC-SYB-ASE , Sybase ASE Database Platform (non Business Suite) , BC-SYB-ASE-CE , ASE Cluster Edition (Standalone) , Bug Filed