2132635 - ASE crashes with stack trace in kbfalloc() when network password encryption is enabled - SAP ASE

• Signal 11 stack trace in kbfalloc() in ASE errorlog:

Current process (0x12c706a7) infected with signal 11 (SIGSEGV)
Address 0x00000001004e6bcc (kbfalloc+0x11c), siginfo (code, address) = (50, 0x17293c50657b92c2)

• Process holding spinlock on Network Memory Pool:

Spinlock Network memory pool spinlock at address 0000000167a27260 owned by 315033255

• Stack trace shows RSA decryption modules, among others:

kbfalloc
ks_falloc
csi_malloc
local_malloc
default_malloc_ex
CRYPTO_malloc
BN_mod_exp_mont_consttime
BN_mod_exp_mont__fdpr_3
RSA_eay_mod_exp
RSA_eay_private_decrypt
RSA_private_decrypt

• Configuration parameter 'net password encryption reqd' turned on.
• OpenSSL may be in use.
• Timeslice errors may also be reported.
• ASE receiving heavy load of incoming connection attempts.
• ASE crashes.

• SAP Adaptive Server Enterprise (ASE) 15.7
• SAP Adaptive Server Enterprise (ASE) 16.0
• SAP/Sybase Adaptive Server Enterprise (ASE) 15.0
• SAP/Sybase Adaptive Server Enterprise (ASE) 15.5

encrypt, decrypt, sha256, ssl, SEGV, segmentation, storage, access, violation, fault, segfault, kisignal, stacktrace, registers , KBA , server panic , memory corruption , BC-SYB-ASE , Sybase ASE Database Platform (non Business Suite) , BC-SYB-ASE-CE , ASE Cluster Edition (Standalone) , Bug Filed