2980781 - Error: hdbsrvutil crashes with Signal 6

You noticed srvutil crashed with signal 6. In srvutil.crashdump.xxx.trc log file you can see the following information:

[CRASH_SHORTINFO]  Exception short info: (xxx)
SIGNAL 6 (SIGABRT) caught, sender PID:  43930, PID: 43930, thread: 214[thr=0]: <SYSTEM>, value int: 0, ptr: 0x0000000000000000,

-------------------------

[CRASH_STACK]  Stacktrace of crash: (xxx Local)
----> Pending exceptions (possible root cause) <----
--- Pending assertion:
exception  1: no.1000000  (Basis/Diagnose/impl/TraceConfiguration.cpp:1388) TID: 43930
    Got exception while initial trace configuration: exception  1: no.1000002  (ltt/impl/memory.cpp:291) TID: 43930
    Allocation failed ; $size$=xxx; $name$=TraceAllocator; $type$=MallocAllocatorImpl; $inuse_count$=0; $allocated_size$=0; $alignment$=512; $failure_type$=OPERATING_SYSTEM_LIMIT; $failure_flag$=
exception throw location:
 1: 0x00007f123d069fa5 in Diagnose::RotateKeepIdFileTraceOutputHandler

-------------------------

[CRASH_INMEMORY_TRACE]  InMemoryTrace: (xxx)
[EventHandler:Info]: AutoEventHandlerCallback::reconfigure(period= off --> 60)
[Memory:Error]: OUT OF MEMORY occurred.
[Memory:Error]: Host: xxx
[Memory:Error]: Executable: /usr/sap/PEI/xxx/exe/hdbsrvutil
[Memory:Error]: PID: 43930
[Memory:Error]: Failed to allocate xxxmb (xxxb).
[Memory:Error]: Allocation failure type: OPERATING_SYSTEM_LIMIT
[Memory:Error]: Current callstack:
[Memory:Error]:  1: 0x00007f123ccdb54d in ltt::allocator::notifyOOM(unsigned long, unsigned long, bool, ltt::allocation_failure_type, bool) [clone .constprop.1565]+0x59 at memory.cpp:130 (libhdbbasis.so)

                                           ---

[Memory:Error]: Out of memory occurred. More information in oom runtime dump!
[Memory:Error]: Finished writing oom runtime dump!
[Basis:Error]: Crash at Basis/Diagnose/impl/TraceConfiguration.cpp:1388
[Basis:Error]: Reason:
[Basis:Error]: exception  1: no.1000000  (Basis/Diagnose/impl/TraceConfiguration.cpp:1388) TID: 43930
    Got exception while initial trace configuration: exception  1: no.1000002  (ltt/impl/memory.cpp:291) TID: 43930
    Allocation failed ; $size$=xxx; $name$=TraceAllocator; $type$=MallocAllocatorImpl; $inuse_count$=0; $allocated_size$=0; $alignment$=512; $failure_type$=OPERATING_SYSTEM_LIMIT;

-------------------------

exception type information:
 - 0: public ltt::bad_alloc@xxx VMI
   - 0: public ltt::exception@xxx
   - 64: public std::bad_alloc@xxx SI
     - 64: public std::exception@xxx
----> Symbolic stack backtrace <----

-------------------------

M_OUT_OF_MEMORY_EVENTS[0] - (id= 40, type= public:remote/multi/remote_datacenter, flags= Crashdump RTEdump, priv= Default)
"TIME";"CONNECTION_ID";"STATEMENT_ID";"STATEMENT_HASH";"HEAP_MEMORY_CATEGORY";"MEMORY_REQUEST_SIZE [byte]";"MEMORY_USED_SIZE [byte]";"EVENT_REASON";"MEMORY_LIMIT_SIZE [byte]";"TRACEFILE_NAME"
"1602395242000000 [local]";0;"0";"";"<malloc allocator>/TraceAllocator";104857600;0;"OPERATING_SYSTEM_LIMIT";0;"/usr/sap/PEI/HDB02/xxx/trace/srvutil.rtedump.<timestamp>.oom.trc"

In nameserve trace file (nameserver_xxx.trc) you can see the following entries:

[81540]{-1}[-1/-1] 2020-10-11 07:46:57.427308 i Savepoint        SavepointImpl.cpp(03025) : Savepoint current savepoint version: 8568, restart redo log position: 0x5a52a82, next savepoint version: 8569, last snapshot SP version: 8566
[81619]{-1}[-1/-1] 2020-10-11 07:47:03.895041 e Memory           mmMallocProxy.cpp(01964) : mmap(0x0000000000000000, 4096, 3, 34, -1, 0) failed.12 (Cannot allocate memory)
[81619]{-1}[-1/-1] 2020-10-11 07:47:03.933041 e OSMemory         mmMallocProxy.cpp(02059) : Failed: mmap

An OOM trace has been written right before the crash dump:

srvutil.rtedump.<timestamp>.oom.trc

SAP HANA, platform edition

sig 6, nsutil, GAL, Global Allocation Limit, , KBA , HAN-DB-ENG , SAP HANA DB Engines , Problem