3276725 - Savepoint blocked during table reorganization and relocation

• During a table reorganization execution and/or relocation of a database table to another HANA worker node, a long running , blocked savepoint is observed.
• Jobworker threads from the runtime dumps reveal the ALTER DDL transaction's child thread responsible for blocking the savepoint and withholding the CCH lock under the 'SAVEPOINT_SHAREDLOCK_OWNERS' and 'Running Threads' section :
  
  indexserver_hostname.30003.rtedump.20221124-113801.0109857.savepoint_blocked.trc
  
  

[SAVEPOINT_SHAREDLOCK_OWNERS]  Owners of shared SavepointLocks: (2022-11-24 11:38:02 049 Local)
485468[thr=123790]: JobWrk12093, TID: 23, UTID: 37910406751, CID: 316959, LCID: 316959, SEID: 844437815194935, parent: 178501, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: 316959 {LDBID: 3, LCID: 316959}, StmtCtx: (1) 1361329969500519 {Parent: 316959, NAME=_SYS_DEFAULT, P=5, TIMESec=0, PARENT=_SYS_DEFAULT, STL=20, SML=400, SesCtx: {ObjHdl: 138523183039488, CID: 316959, LCID: 316959, User: "<ID>", Schema: "<ID>", Ver: 4, CtxID: 17795}}, (2) 0 {Parent: 0, NAME=_SYS_DEFAULT, P=-1, TIMESec=-1, PARENT=_SYS_DEFAULT, STL=-1, SML=-1, ObjHdl: 140601478385408, User: "SYSTEM", Schema: "SYSTEM", }, WorkloadCtx: ---, type: "JobWorker", method: "execute", detail: "peer: conn=hostname:55922 service=30003", command: "
{
SQL:
ALTER TABLE "SAPABAP"."PAYRQT" MOVE PART 8 TO LOCATION 'hostname:30003' PHYSICAL

485468[thr=123790]: JobWrk12093, TID: 23, UTID: 37910406751, CID: 316959, LCID: 316959, SEID: 844437815194935, parent: 178501, SQLUserName: "", AppUserName: "", AppName: "", ConnCtx: 316959 {LDBID: 3, LCID: 316959}, StmtCtx: (1) 1361329969500519 {Parent: 316959, NAME=_SYS_DEFAULT, P=5, TIMESec=0, PARENT=_SYS_DEFAULT, STL=20, SML=400, SesCtx: {ObjHdl: 138523183039488, CID: 316959, LCID: 316959, User: "<id>", Schema: "<id>", Ver: 4, CtxID: 17795}}, (2) 0 {Parent: 0, NAME=_SYS_DEFAULT, P=-1, TIMESec=-1, PARENT=_SYS_DEFAULT, STL=-1, SML=-1, ObjHdl: 140601478385408, User: "SYSTEM", Schema: "SYSTEM", }, WorkloadCtx: ---, type: "JobWorker", method: "execute", detail: "step 1 of 1", command: "
{
SQL:
ALTER TABLE "SAPABAP"."PAYRQT" MOVE PART 8 TO LOCATION 'hostname:30003' PHYSICAL
}
" at 0x00007fed577bb0b2 in DataAccess::setMemory(void*, char, unsigned long)+0x42 at DataAccess/impl/SetMemory.cpp:27 (libhdbdataaccess.so)


485468[thr=123790]: JobWrk12093 at
   0: 0x00005645fe838dac in memcpy_impl+0x808 (hdbindexserver)
   1: 0x00007fed593ef651 in ltt::basic_streambuf<char, ltt::char_traits<char> >::xsgetn(char*, long)+0x90 at ltt/impl/char_traits.hpp:300 (libhdbbasement.so)
   2: 0x00007fed5abf99b3 in ltt::basic_istream<char, ltt::char_traits<char> >& ltt::impl::istreamRead<char, ltt::char_traits<char> >(ltt::basic_istream<char, ltt::char_traits<char> >&, char*, long) [clone .isra.0]+0x150 at ltt/streambuf.hpp:167 (libhdbunifiedtable.so)
   3: 0x00007fed5aa7db5b in UnifiedTable::PersPageChainDescriptor::importData(DataContainer::PersistentObjectContainer&, ltt::basic_istream<char, ltt::char_traits<char> >&, ltt::unique_ptr<DataAccess::ConsistentChange, ltt::without_alloc_t, ltt::releasable_deleter<DataAccess::ConsistentChange>, ltt::integral_constant<bool, false> >&, DataContainer::VarSizeEntryAddress&, void*)+0x687 at ltt/istream.hpp:284 (libhdbunifiedtable.so)
   4: 0x00007fed5791bfa2 in DataContainer::PersistentObjectContainerImpl::importDataImpl(ltt::basic_istream<char, ltt::char_traits<char> >&, ltt::unique_ptr<DataAccess::ConsistentChange, ltt::without_alloc_t, ltt::releasable_deleter<DataAccess::ConsistentChange>, ltt::integral_constant<bool, false> >&, ltt::impl::HashMap<DataContainer::PersistentObjectAnyPointer, DataContainer::PersistentObjectAnyPointer, ltt::hashtable<DataContainer::PersistentObjectAnyPointer, ltt::pair<DataContainer::PersistentObjectAnyPointer const, DataContainer::PersistentObjectAnyPointer>, ltt::hash<DataContainer::PersistentObjectAnyPointer>, ltt::select1st<ltt::pair<DataContainer::PersistentObjectAnyPointer const, DataContainer::PersistentObjectAnyPointer> >, ltt::equal_to<DataContainer::PersistentObjectAnyPointer>, ltt::hash_size_prim> >&, DataContainer::PersistentObjectAnyPointer, ltt::impl::HashMap<DataContainer::PersistentObjectAnyPointer, bool, ltt::hashtable<DataContainer::PersistentObjectAnyPointer, ltt::pair<DataContainer::PersistentObjectAnyPointer const, bool>, ltt::hash<DataContainer::PersistentObjectAnyPointer>, ltt::select1st<ltt::pair<DataContainer::PersistentObjectAnyPointer const, bool> >, ltt::equal_to<DataContainer::PersistentObjectAnyPointer>, ltt::hash_size_prim> > const&, void*)+0x530 at DataAccess/DataContainer/impl/PersistentObjectContainerImpl.cpp:1629 (libhdbdataaccess.so)
   5: 0x00007fed5791d9f1 in DataContainer::PersistentObjectContainerImpl::importData(ltt::basic_istream<char, ltt::char_traits<char> >&, ltt::unique_ptr<DataAccess::ConsistentChange, ltt::without_alloc_t, ltt::releasable_deleter<DataAccess::ConsistentChange>, ltt::integral_constant<bool, false> >&, DataContainer::PersistentObjectAnyPointer&, ltt::impl::HashMap<DataContainer::PersistentObjectAnyPointer, bool, ltt::hashtable<DataContainer::PersistentObjectAnyPointer, ltt::pair<DataContainer::PersistentObjectAnyPointer const, bool>, ltt::hash<DataContainer::PersistentObjectAnyPointer>, ltt::select1st<ltt::pair<DataContainer::PersistentObjectAnyPointer const, bool> >, ltt::equal_to<DataContainer::PersistentObjectAnyPointer>, ltt::hash_size_prim> > const&, void*)+0x1d0 at DataAccess/DataContainer/impl/PersistentObjectContainerImpl.cpp:801 (libhdbdataaccess.so)
   6: 0x00007fed5a95f1b4 in UnifiedTable::TableContainer::importData(DataAccess::PersistenceSession&, UnifiedTable::ImportSourceProvider&, UnifiedTable::ImportExportType const&, long, long, bool)+0x530 at UnifiedTable/impl/TableContainerImpl.cpp:2493 (libhdbunifiedtable.so)
   7: 0x00007fed785956a3 in CS::DDL::GetVirtualFilesContentResponse::deserialize(TRexUtils::Deserializer&)+0xb50 at CS/DDL/impl/GetVirtualFilesContentRequest.cpp:400 (libhdbddlrequests.so)
   8: 0x00007fed56172231 in TrexNet::NetInput::operator>>(TRexUtils::Serializable&)+0x10 at TrexNet/BufferedIO.cpp:3292 (libhdbnetrpc.so)
   9: 0x00007fed59478b45 in TrexNet::AutonomousRequest::handle()+0x241 at TrexNet/Ext/AsyncRequest.cpp:252 (libhdbbasement.so)
  10: 0x00007fed7859302d in CS::DDL::GetVirtualFilesContent::getPersistentObjectContainerFilesFromServer(ltt::vector<PersistenceLayer::ObjectInfo> const&, PersistenceLayer::IndexStorageLocation const&, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&, TrexBase::IndexName const&) const+0x709 at CS/DDL/impl/GetVirtualFilesContentRequest.cpp:676 (libhdbddlrequests.so)
  11: 0x00007fed785452f4 in CS::DDL::MoveTableDataTargetNodeSharedPhase::moveFilesFromSource(TrexBase::IndexName const&, ltt::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<int, 64> > const&, TrexBase::IndexName const&, ltt::vector<PersistenceLayer::ObjectInfo> const&, UnifiedTable::TableContainerCreationParameterSet::RowOrderType, long) const+0x150 at CS/DDL/impl/MoveTableData.cpp:897 (libhdbddlrequests.so)
  12: 0x00007fed78548312 in CS::DDL::MoveTableDataTargetNodeSharedPhase::handle(CS::DDL::MoveTableDataPartInput const&, TRexCommonObjects::TRexApiError&) const+0xe0 at CS/DDL/impl/MoveTableData.cpp:764 (libhdbddlrequests.so)
  13: 0x00007fed7853988b in CS::DDL::DataRequestBase<CS::DDL::MoveTableDataPartInput, TRexCommonObjects::TRexApiError>::handleAndCheck(CS::DDL::MoveTableDataPartInput const&, TRexCommonObjects::TRexApiError&) const+0x37 at CS/DDL/impl/DataRequestBaseImpl.cpp:487 (libhdbddlrequests.so)
  14: 0x00007fed7853a177 in CS::DDL::DataRequestBase<CS::DDL::MoveTableDataPartInput, TRexCommonObjects::TRexApiError>::handle()+0x743 at CS/DDL/impl/DataRequestBaseImpl.cpp:204 (libhdbddlrequests.so)
  15: 0x00007fed7853ce8b in CS::DDL::DataRequestBase<CS::DDL::MoveTableDataPartInput, TRexCommonObjects::TRexApiError>::handleSplitTable()+0x227 at CS/DDL/impl/DataRequestBaseImpl.cpp:538 (libhdbddlrequests.so)
  16: 0x00007fed78530d6e in CS::DDL::MoveTable::executeTargetNodeSharedPhase()+0xaa at CS/DDL/impl/MoveTable.cpp:514 (libhdbddlrequests.so)
  17: 0x00007fed78534cee in CS::DDL::MoveTable::execute()+0x27a at CS/DDL/impl/MoveTable.cpp:375 (libhdbddlrequests.so)
  18: 0x00007fed3b2a0422 in SQLDDL::CSDDLWrapperImpl::createMoveTableRequest(Metadata::TableInfo*, unsigned int, ltt_adp::basic_string<char, ltt::char_traits<char>, ltt::integral_constant<bool, true> > const&, int, bool, ltt::shared_ptr<partitioning::TreeSpecNodeBase, ltt::default_deleter, ltt::integral_constant<bool, false>, ltt::integral_constant<bool, false> >, TRexCommonObjects::TRexApiError&)+0x780 at SQLDDL/Wrapper/WrapperImpl.cpp:598 (libhdbcsddlwrapperimpl.so)
  19: 0x00007fed79b1b095 in ptime::QueryExecutor::move_cs_table(ptime::ClassPtr<Metadata::TableInfo>, unsigned int, ltt::shared_ptr<partitioning::TreeSpecNodeBase, ltt::default_deleter, ltt::integral_constant<bool, false>, ltt::integral_constant<bool, false> >, char const*, bool)+0x6e1 at ptime/query/plan_executor/ddl/qx_trex.cc:8358 (libhdbrskernel.so)
  20: 0x00007fed79b1edd4 in ptime::QueryExecutor::alter_move_to_location(ptime::ClassPtr<Metadata::TableInfo>, ptime::qp_alter_table const*)+0x220 at ptime/query/plan_executor/ddl/qx_trex.cc:8092 (libhdbrskernel.so)
  21: 0x00007fed79b1f907 in ptime::QueryExecutor::trex_alter_table(ptime::qp_alter_table const*, ptime::Query::result_t*)+0x963 at ptime/query/plan_executor/ddl/qx_trex.cc:2018 (libhdbrskernel.so)
  22: 0x00007fed79a7c944 in ptime::QueryExecutor::alter_table_common(ptime::qp_alter_table const*, ptime::ClassPtr<Metadata::TableInfo>, ptime::Query::result_t*)+0x2d0 at ptime/query/plan_executor/ddl/qx_table.cc:6756 (libhdbrskernel.so)
  23: 0x00007fed79a80709 in ptime::QueryExecutor::alter_table(ptime::qp_alter_table const*, ptime::ClassPtr<Metadata::TableInfo>, ptime::Query::result_t*)+0x1e5 at ptime/query/plan_executor/ddl/qx_table.cc:6642 (libhdbrskernel.so)
  24: 0x00007fed79a80c0d in ptime::QueryExecutor::alter_table(ptime::qp_alter_table const*, ptime::Query::result_t*)+0x1a9 at ptime/query/plan_executor/ddl/qx_table.cc:6558 (libhdbrskernel.so)
  25: 0x00007fed79860bb9 in ptime::QueryExecutor::execute(ptime::Transaction&, char const*, ptime::qp_parse_tree const*, ptime::Query::Plan*, ptime::Query::result_t*, ptime::QueryExecutor*)+0xaa5 at ptime/query/plan_executor/ddl/qx.cc:595 (libhdbrskernel.so)
  26: 0x00007fed7a547e16 in ptime::Query::execute(ptime::Transaction&, char const*, ptime::Query::param_t*, ptime::Query::Plan*, ptime::Query::result_t*, ptime::Statement*, bool, bool, char const*, ptime::QueryExecutor*)+0x4d2 at ptime/query/query.cc:4880 (libhdbrskernel.so)
  27: 0x00007fed7a7f10d2 in ptime::Statement::execute_(Execution::Context&, EAPI::CursorHoldability const&, bool, bool, bool)+0xa40 at ptime/session/eapi/jdbc/Statement.cc:2799 (libhdbrskernel.so)
  28: 0x00007fed7a7f1b9b in ptime::Statement::execute(Execution::Context&, char const*, char const*, char const*, EAPI::CursorHoldability const&, bool, bool, bool)+0x1c7 at ptime/session/eapi/jdbc/Statement.cc:2106 (libhdbrskernel.so)
  29: 0x00007fed7a79bdd7 in ptime::Statement::execute(Execution::Context&, EAPI::StatementProperty const&, char const*)+0x93 at ptime/session/eapi/jdbc/Statement.cc:2072 (libhdbrskernel.so)
  30: 0x00007fed5f113aa2 in EAPI::StatementHandle::execute(Execution::Context&, char const*)+0x30 at EAPI/impl/StatementHandle.cpp:148 (libhdbeapi.so)
  31: 0x00007fed7e69332e in TableRedist::ReorgPlanExecuterV1::executeGroup(Execution::Context&)+0x109a at TableDistribution/TableRedistribution/Execute/TableRedistPlanExecuteBase.cpp:250 (libhdbreorg.so)
  32: 0x00007fed7e684706 in TableRedist::ReorgPlanExecuterBase::run(Execution::Context&, Execution::MiniJobObject&)+0x12 at TableDistribution/TableRedistribution/Execute/TableRedistPlanExecuteBase.cpp:87 (libhdbreorg.so)
  33: 0x00007fed552de96f in Execution::MiniJobsWorkerJob::run(Execution::Context&, Execution::JobObject&)+0x13b at Basis/Execution/impl/JobContext.cpp:1696 (libhdbbasis.so)
  34: 0x00007fed552fa120 in Execution::JobObjectImpl::run(Execution::JobWorker*)+0x15b0 at Basis/Execution/impl/JobExecutionLog.hpp:155 (libhdbbasis.so)
  35: 0x00007fed55307683 in Execution::JobWorker::runJob(ltt::smartptr_handle<Execution::JobObjectForHandle>&)+0x860 at Basis/Execution/impl/JobExecutorThreads.cpp:366 (libhdbbasis.so)
  36: 0x00007fed5530922b in Execution::JobWorker::run(Execution::ThreadRC&)+0x887 at Basis/Execution/impl/JobExecutorThreads.cpp:1354 (libhdbbasis.so)
  37: 0x00007fed5535a956 in Execution::Thread::staticMainImp(Execution::Thread*)+0x542 at Basis/Execution/impl/Thread.cpp:574 (libhdbbasis.so)
  38: 0x00007fed5535b295 in Execution::pthreadFunctionWrapper(Execution::Thread*)+0x1c1 at Basis/Execution/impl/ThreadInterposition.cpp:693 (libhdbbasis.so)
  39: 0x0000000000000000 in <no symbol>+0x0 (<unknown>)

SAP HANA, platform edition 2.0

SAVEPOINT_SHAREDLOCK_OWNERS, savepoint_blocked.trc, MOVE PART 8 TO LOCATION , JobWrk, ConsistentChangeLock . , KBA , HAN-DB , SAP HANA Database , HAN-DB-PERF , SAP HANA Database Performance , Problem