2883016 - 'CtlibController' exiting following Signal 6 in gsignal(), signal 11, timeslice errors - SAP ASE

• SAP systems on SAP ASE 16.0 SP03 PL07 and above, with HADR replication (Always On setup), stop accepting new connections.
• The problem mainly affects secondary DR server but maybe seen on the primary server on occasion.
• ASE errorlog reports the following error with a stack trace: 

Current process (0x24850205) infected with signal 6 (SIGABRT/SIGIOT)
Address 0x0x00007f5ffef27f67 (gsignal+0x37), siginfo (code, address) = (-6, 0x0x05f5f7ed000074ca)

  The accompanying stacktrace includes the functions:

kisignal
gsignal
(null)     

  Note: A complete stacktrace is available under Attachments.

• The stacktrace error is followed by the message:

Run to completion task 'CtlibController' exiting.

• ASE errorlog may report  'Current process infected with signal 11 (SIGSEGV)' together with a stack trace which includes the functions:

cfree
ct__mp_free
comn_loc_drop
nthrd_backtrace 
krtctskException 
kisignal 
pthread_cond_destroy 
sybnet_dealloc_ep 
sybnet__async_poll

• The above stack trace errors are followed by multiple timeslice errors.

  The  stack trace errors accompanying the timeslice failures, may reference any of the following functions:

(null)
ct_con_alloc
ct__api_con_uninit
ct_con_drop
usctcondrop
pthread_create 

• The following message is also seen amidst the above errors:

hadr_quitfn: kpid 1354956888 releasing HADR sema redirect count 0 flags 0:

• ASE database is inaccessible and unresponsive after the timeslice errors arise.

• ASE may crash while holding following taskblock spinlocks after the above errors.

Spinlocks held by taskblock 0x7ef997caadf8 
Spinlock User Log Cache Spinlocks at address 0x00007ef997260f00 owned by 0x7ef997caadf8

Spinlocks held by taskblock 0x147809288
Spinlock fglockspins at address 0x0000000146c1c980 owned by 0x147809288

Spinlocks held by taskblock 0x7ef997caadf8 
Spinlock Default Data Cache Spinlocks at address 0x00007ef997260f00 owned by 0x7ef997caadf8

• On systems running SAP applications, the affected queries are HADR-monitoring queries (note the 'SQL query' in the stacktrace for the failing process), for example:

INSERT INTO #sevDateMessage (severity, date, message) select Severity, Date, Message
FROM master..hadrGetLog WHERE _logicalHost = 'Sit' AND _serverType = 'RS' AND _startDate = '2020-04-12 01:05:36'

select LogicalHost, ASELogBacklog, RSQueueBacklog, DBName, (case when isnumeric(RSDeviceSize) > 0
then cast(RSDeviceSize as integer) else null end) - (case when isnumeric(RSDeviceUsed) > 0
then cast(RSDeviceUsed as integer) else null end) from master..hadrStatusResource
/* R3:CL_SQL_STATEMENT==============CP:500 T:hadrStatusResource M:001 */ 

• SAP Adaptive Server Enterprise (ASE) 16.0 SP03 PL07 and higher for Business Suite
• SAP NetWeaver (NW)  
• High Availability Disaster Recovery (HADR)
• SAP Adaptive Server Enterprise (ASE) 16.0 SP03 PL07 and higher
• SAP Replication Server (SRS) 16.0 SP03 PL07 and higher

HADR, CtlibController, gsignal, 821748, CR821748, CR#821748, CR#820713, CR820713, 820713, CR#821170, CR821170, 821170, 820370, CR820370, CR#820370, kisignal, stacktrace, CR 820370, 822182, CR822182, CR#822182, SRS, HADR, CR819445, CR#819445, 819445, CR#820657, CR820657, 820657, SEGV, segmentation, storage, access, violation, fault, segfault , KBA , BC-DB-SYB , Business Suite on Adaptive Server Enterprise , BC-SYB-REP , Sybase Replication Server (standalone) , BC-SYB-REP-SAP , Replication with SAP Suite / SAP BW , Bug Filed