3147405 - XSEngine fails to start after VM Snapshot restore

You are doing VM snapshot restore and experiences the errors below:

• You start SAP HANA and you get the following (or similar) error message:
  
  
  • FAIL: process hdbxsengine HDB XSEngine-HDP not running after VM snapshot restore
    
    
• You can find similar entries in SAP HANA service trace files as follows:
  

  [8515]{-1}[-1/-1] 2022-01-29 05:47:53.279496 i Service_Shutdown TrexService.cpp(00582) : Preparing for shutting service down
  [...]
  
  [8553]{-1}[-1/-1] 2022-01-29 05:47:56.210573 e Stream           NetworkChannelCompletion.cpp(00729) : [e193860d90239ae4,<ip address>:40007,<ip address>:54238,UNK,0] NetworkChannelCompletionThread #0: 
  Error in asynchronous stream event: exception  1: no.2110006  (Basis/IO/Stream/impl/NetworkChannelSSLFilter.cpp:560) TID: 8553
      Error during stream receive: NetworkChannelSSLFilter::receiveRaw: Socket state not valid. , rc=$sysrc$: $sysmsg$; $Context$=[e193860d90239ae4,<ip address>:40007,<ip address>:54238,UNK,0];
  $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139883593219096, fd=389, refCnt=1, idx=0, local=<ip address>/40007_tcp, remote=<ip address>/54238_tcp, state=PreClose, pending=[r---]}}}
  exception throw location:
  [...]
  
  [8553]{-1}[-1/-1] 2022-01-29 05:47:56.214310 e sr_dataaccess    DisasterRecoveryProtocol.cpp(01305) : V 2: HT_Pri (CT_Data[0/0]): virtual void DataAccess::ReplicationProtocolHandler::setError(const ltt::exception&),
  pHandler=0x00007f34654f4000, error=exception 2110006: Error during stream receive: NetworkChannelSSLFilter::receiveRaw: Socket state not valid. , rc=$sysrc$: $sysmsg$; $Context$=[e193860d90239ae4,
  <ip address>:40007,<ip address>:54238,UNK,0]; $channel$={<NetworkChannelSSLFilter>={<NetworkChannelBase>={this=139883593219096, fd=389, refCnt=1, idx=0, local=<ip address>/40007_tcp,
  remote=<ip address>/54238_tcp, state=PreClose, pending=[r---]}}}
   (Basis/IO/Stream/impl/NetworkChannelSSLFilter.cpp:560; 2110006)
  [...]
  
  [23498]{-1}[-1/-1] 2022-01-29 05:53:02.076486 i Basis            TraceStream.cpp(00719) : ==== Starting hdbxsengine, version 2.00.046.00.1581325702 (fa/hana2sp04), build linuxx86_64
  b41c6996c56a493172bd01ce449c0cfab86b847b 2020-02-10 10:35:59 ld5145 gcc (SAP release 20181205, based on SUSE gcc7-7.3.1+r258812-2.15) 7.3.1 20180323 [gcc-7-branch revision 258812]
  [...]
  
  [10559]{-1}[-1/-1] 2022-02-02 08:42:29.207273 e LogReplay        RowStoreTransactionCallback.cc(00532) : exception  1: no.3000221  (DataAccess/impl/PersistenceManagerImpl.cpp:4898) TID: 10559
      Slave volume 2 is inconsistent with master (last commit position on slave: 0x111b0185(0x111b01c4), last known position on master: 0x111b0248)
  exception throw location:
  [...]
  
  [10559]{-1}[-1/-1] 2022-02-02 08:42:29.215729 e Basis            Crash.cpp(00757) : Crash at /data/jenkins_prod/workspace/efgdet97rp/s/ptime/storage/recovery/RowStoreTransactionCallback.cc:543
  Reason:
  exception  1: no.2100002  (ptime/storage/recovery/RowStoreTransactionCallback.cc:543) TID: 10559
      log position inconsistency detected - plz contact to HANA dev support before clearing or throwing away your redo log files

• You can find similar entries in SAP HANA crash dump files as follows:
  

  [CRASH_STACK]  Stacktrace of crash: (2022-02-02 13:51:24 505 Local)
  ----> Pending exceptions (possible root cause) <----
  --- Pending assertion:
  exception  1: no.2100002  (ptime/storage/recovery/RowStoreTransactionCallback.cc:543) TID: 32734
      log position inconsistency detected - plz contact to HANA dev support before clearing or throwing away your redo log files
  exception throw location:
   1: 0x00007fad2903acef in ptime::RowStoreTransactionCallback::replayQueryOpenTransactionState(ltt::vector<DataAccess::PersistenceSession*> const&, ltt::vector<ltt::pair<DataAccess::TransactionCallback::TerminateAction, long> >&)+0xcdb at RowStoreTransactionCallback.cc:543 (libhdbrskernel.so)
   2: 0x00007fad1fd73465 in DataRecovery::RecoveryHandlerImpl::finishRecovery(bool)+0x461 at RecoveryHandlerImpl.cpp:2022 (libhdbdataaccess.so)
   3: 0x00007fad1fd7e2ca in DataRecovery::RecoveryHandlerImpl::finish()+0x76 at RecoveryHandlerImpl.cpp:1841 (libhdbdataaccess.so)
   4: 0x00007fad20057221 in DataAccess::PersistenceManagerImpl::replayOrClearLog(bool, bool, bool, unsigned long&, unsigned long&)+0x1d0 at PersistenceManagerImpl.cpp:3185 (libhdbdataaccess.so)
   5: 0x00007fad2004fc7c in DataAccess::PersistenceManagerImpl::replayImpl(bool)+0x178 at PersistenceManagerImpl.cpp:4981 (libhdbdataaccess.so)
   6: 0x00007fad2243965e in PersistenceLayer::PersistenceSystem::start(NameServer::ServiceStartInfo const&, bool, bool)+0xa8a at PersistenceSystem.cpp:566 (libhdbpersistence.so)
   7: 0x00007fad223f238e in PersistenceLayer::PersistenceFactory::startPersistence(NameServer::ServiceStartInfo&, bool, bool, DataAccess::TablePreloadWriteCallback*, ltt::smartptr_handle<DataAccess::DisasterRecoveryPrimaryCallback>, DataAccess::DisasterRecoveryESCallback*)+0xba at PersistenceFactory.cpp:546 (libhdbpersistence.so)
   8: 0x00007fad3f8c36f9 in PersistenceController::startup(PersistenceLayer::PERSISTENCE_MODE, NameServer::ServiceStartInfo*, bool, DataAccess::TablePreloadWriteCallback*, DataAccess::TablePreloadReadCallback*, Backup::RecoverCbc_Federation*)+0x5d5 at PersistenceController.cpp:721 (libhdblogger.so)
   9: 0x00005601ab5611f5 in TRexAPI::TREXIndexServer::assign(NameServer::ServiceStartInfo&, bool, TREX_ERROR::TRexError&)+0x7d1 at TREXIndexServer.cpp:1236 (hdbxsengine)
  10: 0x00005601ab56a789 in TRexAPI::AssignThread::run(void*)+0x35 at TREXIndexServer.cpp:533 (hdbxsengine)
  11: 0x00007fad21bc8d1b in TrexThreads::PoolThread::run()+0x787 at PoolThread.cpp:462 (libhdbbasement.so)
  12: 0x00007fad21bcb7e0 in TrexThreads::PoolThread::run(void*&)+0x10 at PoolThread.cpp:143 (libhdbbasement.so)
  13: 0x00007fad1cc37b48 in Execution::Thread::staticMainImp(Execution::Thread*)+0x534 at Thread.cpp:529 (libhdbbasis.so)
  14: 0x00007fad1cc2779d in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x169 at ThreadInterposition.cpp:636 (libhdbbasis.so)
  15: 0x00007fad1c3b04f9 in start_thread+0xd5 (libpthread.so.0)
  16: 0x00007fad1aba0fbf in __GI___clone+0x3b (libc.so.6)

SAP HANA, platform edition 2.0

inconsistent, inconsistency, log position, log segment, HANA, saphana, database, DB , KBA , HAN-DB , SAP HANA Database , Problem