2952218 - Primary ASE unavailable after encountering hardware failure on companion node - SRS

• Hardware failure on companion node
• Both the companion Replication Server (SRS) and Adaptive Server (ASE) were not available
• Rep Agent down in primary ASE
• ASE log reports:
  
  
• Error: 9414, Severity: 20, State:
  RepAgent(1): Streaming replication stream CI Library error -1 with severity 2 and message 'Producer receiver 'master' failed to receive data, shutting down transport to 'ocs:hostname:port ssl="CN=SID"'.'. Rep Agent detected error 3 with message 'Unknown' at line 3655 in file ra_ci_scanner.c.
• Rep Agent on database 'master' switched from mode 'sync' to mode 'async' because scanner reopened the stream after a retryable error.
• (CI-Info) Message: 76050, Severity: 0 Producer of Stream 'master' is exiting.
  
  
• stacktrace reported in ASE log:
  
  kernel Current process (0x1dfb02ac) infected with signal 11 (SIGSEGV)
  kernel Current Process is running on Engine 11
  kernel server is using elf symbols for stack decoding (125189 symbols found)
  kernel Address 0x0x00007f3d4e1e2903 ((null)+0x4e1e2903), siginfo (code, address) = (1, 0x0x000000004d000127)
  kernel **** Saved signal context (0x0x00007f28103a1c40): ****
  kernel uc_flags: 0x1, uc_link: 0x(nil)
  kernel uc_sigmask: 0x7bfbf037 0xb 0x1 0x4d000127
  kernel uc_stack: ss_sp: 0x(nil), ss_size: 0x0, ss_flags: 0x2
  kernel General Registers (uc_mcontext.gregs):
  kernel PC : 0x00007f3d4e1e2903 ((null)+0x4e1e2903)
  kernel RAX : 0x00007f27a8000578 RBX : 0x00007f27a817c990
  kernel RCX : 0x0000000000000051 RDX : 0x0000000000000841
  kernel RBP : 0x00007f27a8000020 RSP : 0x00007f28103a2930
  kernel R8 : 0x0000000000000030 R9 : 0x00007f27a8000078
  kernel R10 : 0x00007f27a8000070 R11 : 0x0000000000000050
  kernel R12 : 0x000000004d00010f R13 : 0x00000000000007fa
  kernel R14 : 0x0000000000000840 R15 : 0x0000000000000810
  kernel RDI : 0x0000000000000002 RSI : 0x0000000000028026
  kernel RIP : 0x00007f3d4e1e2903 CSGSFS : 0x0000000000000033
  kernel TRAPNO : 0x000000000000000e ERR : 0x0000000000000004
  kernel EFL : 0x0000000000010206
  kernel **** end of signal context ****
  kernel ************************************
  kernel SQL causing error : print 'ping %1! %2!', @@spid, @@transtate
  kernel ************************************
  server SQL Text: print 'ping %1! %2!', @@spid, @@transtate
  kernel curdb = 4 tempdb = 6 pstat = 0x10000 p2stat = 0x40101000
  kernel p3stat = 0x800 p4stat = 0x0 p5stat = 0x8 p6stat = 0x0 p7stat = 0x10000
  kernel lasterror = 0 preverror = 0 transtate = 1
  kernel curcmd = 332 program = <program>
  kernel extended error information: hostname: <hostname> login: SAPSR3
  kernel pc: 0x0000000001534f00 pcstkwalk+0x482()
  kernel pc: 0x00000000015348bf ucstkgentrace+0x20f()
  kernel pc: 0x00000000015310f2 ucbacktrace+0x54()
  kernel pc: 0x00000000017fc087 terminate_process+0xb17()
  kernel pc: 0x0000000001561594 kisignal+0x868()
  kernel end of stack trace, spid 1491, kpid 502989484, suid 6
  
  
• time slice seen in ASE error log:
  

  kernel timeslice -1001, current process infected at 0x7f3d4e26038b ((null)+0x4e26038b)
  kernel **** Saved signal context (0x0x00007f281148d500): ****
  kernel uc_flags: 0x1, uc_link: 0x(nil)
  kernel uc_sigmask: 0x7bfbf037 0xa 0xfffffffa 0x3424
  kernel uc_stack: ss_sp: 0x(nil), ss_size: 0x0, ss_flags: 0x2
  kernel General Registers (uc_mcontext.gregs):
  kernel PC : 0x00007f3d4e26038b ((null)+0x4e26038b)
  kernel RAX : 0xfffffffffffffffc RBX : 0x00007f27a8000020
  kernel RCX : 0x00007f3d4e26038b RDX : 0x0000000000000002
  kernel RBP : 0x00007f281148eaa0 RSP : 0x00007f281148e1d8
  kernel R8 : 0x00007f3d4ffe0830 R9 : 0x00000000fb67a983
  kernel R10 : (nil) R11 : 0x0000000000000202
  kernel R12 : 0x0000000000000100 R13 : 0x00007f27b02c9ed3
  kernel R14 : 0x00007f27b042ee70 R15 : 0x0000000000000102
  kernel RDI : 0x00007f27a8000020 RSI : 0x0000000000000080
  kernel RIP : 0x00007f3d4e26038b CSGSFS : 0x0000000000000033
  kernel TRAPNO : 0x000000000000000e ERR : 0x0000000000000004
  kernel EFL : 0x0000000000000202
  kernel **** end of signal context ****
  kernel timeslice error: spid 1450 exhausted its 'time slice' of 100 milliseconds and additional 'cpu grace time' of 1000 ticks (100000 milliseconds). It has been marked for termination.
  kernel This Adaptive Server process has had 5410 major and 74639411 minor page faults since boot.

• While the timeslice occurred, the primary ASE was unavailable

• SAP Replication Server (SRS) 16.0 SP03 PL06
• SAP Adaptive Server Enterprise (ASE) 16.0 SP03 PL06 for Business Suite 
• High Availability Disaster Recovery (HADR)   

RepServer, RS, repagent, Rep Agent, time slice, HADR, failover , KBA , BC-SYB-REP-SAP , Replication with SAP Suite / SAP BW , Problem