2730077 - SAP HANA service crashes or gives error during asynchronous file transfer

• You observe SAP HANA servive (eg hdbindexserver) crashes with SIGNAL 6 (SIGABRT);
• For exaomple; the file indexserver_<hostname>.<port>.crashdump.YYYYMMDD-HHMMSS.ffffff.trc contains in the CRASH_STACK session an exception similar to the following:
• Crash Stack 1:
  

  [CRASH_STACK] Stacktrace of crash: (2020-07-19 23:59:59 728 Local)
  ----> Pending exceptions (possible root cause) <----
  --- Pending assertion:
  exception 1: no.2000008 (Basis/IO/FileAccess/impl/LocalFileCompletionThread.cpp:320) TID: 000000
   Error during asynchronous file transfer (io_getevents), rc=5: Input/output error; $fileCallback$=[W] , buffer= 0x00007ff1b8aa1000, offset= 4266209280, size= 0/4194304, file= "<root>/logsegment_000_00000002.dat" ((open, mode= RW, access= rw-rw-r--, flags= ASYNC|DIRECT|LOCK|LAZY_OPEN), factory= (root= "/hana/log/<SID>/mnt00005/hdb00008.00003/" (access= rw-rw-r--, flags= AUTOCREATE_DIRECTORY, usage= LOG, fs= xfs, config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=64,min_submit_batch_size=16,max_submit_batch_size=64))) {shortRetries= 0, fullRetries= 10 (10/10)}
   1: 0x00007ff34876c1f9 in FileAccess::LocalFileIOControlBlock::reportError(char const*, unsigned long, char const*, int)+0x35 at LocalFileCallback.cpp:347 (libhdbbasis.so)
   2: 0x00007ff34876ef99 in FileAccess::LocalFileCompletionThread::run(void*&)+0xe05 at LocalFileCompletionThread.cpp:320 (libhdbbasis.so)
   3: 0x00007ff3487b4b48 in Execution::Thread::staticMainImp(Execution::Thread*)+0x534 at Thread.cpp:529 (libhdbbasis.so)
   4: 0x00007ff3487a479d in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x169 at ThreadInterposition.cpp:636 (libhdbbasis.so)
   5: 0x00007ff347f30dd5 in start_thread+0xc1 (libpthread.so.0)
   6: 0x00007ff3467a502d in __clone+0x69 (libc.so.6)

• Crash Stack 2:
  

  [CRASH_STACK] Stacktrace of crash: (2020-07-04 03:25:34 823 Local)
  ----> Pending exceptions (possible root cause) <----
  --- Pending assertion:
  exception 1: no.2000008 (Basis/IO/FileAccess/impl/LocalFileCompletionThread.cpp:290) TID: 00000
  Error during asynchronous file transfer (io_getevents), rc=5: Input/output error; $fileCallback$=[W] , buffer= 0x00002abf71cfc000, offset= 651657216, size= 0/36864, file= "<root>/logsegment_000_00001104.dat" ((open, mode= RW, access= rw-rw-r--, flags= DIRECT|MUST_EXIST|LOCK|LAZY_OPEN), factory= (root="/sapmnt/log/PB0/mnt00002/hdb00005.00003/" (access= rw-rw-r--, flags= AUTOCREATE_DIRECTORY, usage= LOG, fs= nfs4, config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=64,min_submit_batch_size=16,max_submit_batch_size=64))) {shortRetries= 0, fullRetries= 10 (10/10)}
  exception throw location:
  1: 0x00002abb3acc1efe in FileAccess::LocalFileIOControlBlock::reportError(char const*, unsigned long, char const*, int)+0x4a at LocalFileCallback.cpp:335 (libhdbbasis.so)
  2: 0x00002abb3acc21dc in FileAccess::LocalFileIOControlBlock::handleReturnCode(char const*, unsigned long, char const*, int)+0x168 at LocalFileCallback.cpp:130 (libhdbbasis.so)
  3: 0x00002abb3acc33dc in FileAccess::LocalFileCompletionThread::requestSuccessful(io_event const&, FileAccess::LocalFileIOControlBlock*)+0x68 at LocalFileCompletionThread.cpp:290 (libhdbbasis.so)
  4: 0x00002abb3acc3719 in FileAccess::LocalFileCompletionThread::getNextFinishedRequests()+0x165 at LocalFileCompletionThread.cpp:264 (libhdbbasis.so)
  5: 0x00002abb3acc396d in FileAccess::LocalFileCompletionThread::getNextFinishedRequest()+0x49 at LocalFileCompletionThread.cpp:198 (libhdbbasis.so)
  6: 0x00002abb3acc4428 in FileAccess::LocalFileCompletionThread::run(void*&)+0x34 at LocalFileCompletionThread.cpp:96 (libhdbbasis.so)
  7: 0x00002abb3ac77f26 in Execution::Thread::staticMainImp(void**)+0x412 at Thread.cpp:537 (libhdbbasis.so)
  8: 0x00002abb3ac79ad8 in Execution::Thread::staticMain(void*)+0x34 at ThreadMain.cpp:26 (libhdbbasis.so)

• Crash Stack 3:
  

  [CRASH_STACK] Stacktrace of crash: (2020-07-04 08:25:34 870 Local)
  ----> Pending exceptions (possible root cause) <----
  --- Pending assertion:
  exception 1: no.2000008 (Basis/IO/FileAccess/impl/LocalFileCompletionThread.cpp:320) TID: 94301
  Error during asynchronous file transfer (io_getevents), rc=52: Invalid exchange; $fileCallback$=[W] , buffer= 0x00007f299c194000, offset= 62750720, size= 0/4096, file= "<root>/datavolume_0000.dat" ((open, mode= RW, access= rw-rw-r--, flags= ASYNC|DIRECT|MUST_EXIST|LOCK), factory= (root= "/hana/data/<SID>>/mnt00001/hdb00003.00003/" (access= rw-rw-r--, flags= AUTOCREATE_PATH, usage= DATA, fs= xfs, config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=128,min_submit_batch_size=16,max_submit_batch_size=64))) {shortRetries= 0, fullRetries= 10 (10/10)}
  $res$=
  LogicalPage@lpcb=0x00007f27a8c14600 [0x640ab99L@DefaultPageAccess, FlushPPageNo=0x600000003bd8P [0x3bd8000,4k,0], ConvPPageNo=0x600000003bd8P [0x3bd8000,4k,0]][Manage=1/ExternalCached=0/Disp=Shortterm/InternalShortInRecovery=0/Mod=0/Del=0/isToBeDel=0/Loading=0/needsBarrierForFlush=0/useBarrierForFlush=0/isInFlushQueue=0/isInIO=1/Ref=1/IterRef=0/Check=2], Page= dictionary={block={logical={page={savepoint=1493317,version=1,page_type=UnifiedTableDictionary,checksum_algo=CRC32,delete_mark=bytes(1)[
  0x00000000: 23 #
  ],converter_type=Default,size=4k,checksum=3108899865,flush_counter=0,reserved=0},owner=0xfe02da26be,pageno=0x640ab99L,nextpageno=0x640ab98L},capacity=4096,used=4096,index=5,reserved=0},blocks={blocks={dummy=0}}}
  exception throw location:
  1: 0x00007fe1046e8dd9 in FileAccess::LocalFileIOControlBlock::reportError(char const*, unsigned long, char const*, int)+0x35 at LocalFileCallback.cpp:347 (libhdbbasis.so)
  2: 0x00007fe1046edfd9 in FileAccess::LocalFileCompletionThread::run(void*&)+0xe05 at LocalFileCompletionThread.cpp:320 (libhdbbasis.so)
  3: 0x00007fe1047334e8 in Execution::Thread::staticMainImp(Execution::Thread*)+0x534 at Thread.cpp:529 (libhdbbasis.so)
  4: 0x00007fe104722798 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x164 at ThreadInterposition.cpp:633 (libhdbbasis.so)
  5: 0x00007fe103eb42de in start_thread+0xfa (libpthread.so.0)
  6: 0x00007fe1026b1a63 in __GI___clone+0x40 (libc.so.6)

SAP HANA, platform edition

crash_stack, crashed, short_stack, short, crashing, restarted, down, shutdown, exited, failed, hardware, log io, I/O, reservation conflict, FAILED, failover, kernel, xfs, nfs4, fs, dm, dm, rc=5, rc=52, LocalFileCompletionThread, SIGABRT, SIGNAL6, signals, error detected , KBA , HAN-DB-PER , SAP HANA Database Persistence , HAN-DB-ENG , SAP HANA DB Engines , HAN-DB-HA , SAP HANA High Availability (System Replication, DR, etc.) , HAN-DB-BAC , SAP HANA Backup & Recovery , Problem