2758257 - Signal 11 raised by iocRunController thread and ASE server hangs - SAP ASE 16.0

• Port scanning on a server with SSL enabled may cause an "infected with signal 11" error with a stacktrace in ASE. The stack shows a null function name at point of failure:

Current process (0x0) infected with signal 11 (SIGSEGV)
Current Process is running on a Non-Engine Thread
Address 0x0x00002ae2f651d4ba ((null)+0x2ae2f651d4ba), siginfo (code, address) = (1, 0x0x00002ae3a3ffffef)

• ASE gets hung when the SSL port is scanned.  
   
• This part of the stack trace shows an internal thread id of 10 for the iocRunController thread. As shown below the stacktrace does not end normally and stops in ubcheck():

00:0010:00000:00000:<date time info> kernel  Run to completion task iocRunController(46833470) exception 11
00:0010:00000:00000:<date time info> kernel  pc: 0x00000000014a2dc1 nthrd_backtrace+0x3b()
00:0010:00000:00000:<date time info> kernel  pc: 0x00000000014c779e krtctskException+0xc4()
00:0010:00000:00000:<date time info> kernel  pc: 0x00000000014cf90f kisignal+0xa19()
00:0010:00000:00000:<date time info> kernel  ubcheck: addr = 0x7ffffffe is not on the stack starting with address 0x2ae2fa266000 with size 0x200000.

• Other messages in the ASE errorlogs at the time of the failure:

ksmask__rpacket: Invalid tdslength value XXXX, kpid: XXX
originating node of previous invalid TDS packet: 10.XX.XX.XX port : 5000
ssl_server_handshake: vsn 43 kpid XXXX FAILED sslerr

SAP Adaptive Server Enterprise (ASE) 16.0 SP03 and higher

CR818525, CR#818525, 818525, SEGV, segmentation, storage, access, violation, fault, segfault, kisignal, registers , KBA , BC-SYB-ASE , Sybase ASE Database Platform (non Business Suite) , Bug Filed