2815245 - Backup of a tenant database after recovering SYSTEMDB failed.

After recovering SYSTEMDB, backup for a tenant database failed due to 'Inconsistent SSFS!' error.

Data volume has not been encrypted.

[ backup.log ]
============================================================
2019-07-01T23:28:17+08:00 P029088 16bae269d6a INFO BACKUP state of service: xsengine, erpdev:31007, volume: 2, BackupAbortSavepointFinished
2019-07-01T23:28:17+08:00 P029088 16bae269d6a INFO BACKUP state of service: dpserver, erpdev:31011, volume: 4, BackupAbortSavepointFinished
2019-07-01T23:28:17+08:00 P029088 16bae269d6a INFO BACKUP state of service: indexserver, erpdev:31003, volume: 3, BackupAbortSavepointFinished
2019-07-01T23:28:17+08:00 P029088 16bae269d6a ERROR BACKUP SAVE DATA finished with error: [447] backup could not be completed, [301154] Failed to check consistency during readContent: Inconsistent SSFS!
===========================================================

[ indexserver trace ]
===========================================================
[29686]{-1}[-1/-1] 2019-07-01 23:15:22.291324 e Backup SaveDataManager.cpp(00620) : :::: BACKUP finished with error: [447] backup could not be completed, [301154] Failed to check consistency during readContent: Inconsistent SSFS! ::::
[29684]{-1}[-1/-1] 2019-07-01 23:28:16.887084 w Crypto RootKeyStoreConsistencyChecker.cpp(00152) : SSFS key of type/version Persistence/0 is inconsistent (key has changed)
[29684]{-1}[-1/-1] 2019-07-01 23:28:16.888483 w Crypto RootKeyStoreConsistencyChecker.cpp(00152) : SSFS key of type/version Backup/0 is inconsistent (key has changed)
[29684]{-1}[-1/-1] 2019-07-01 23:28:16.889028 w Crypto RootKeyStoreConsistencyChecker.cpp(00152) : SSFS key of type/version Redo log/0 is inconsistent (key has changed)
[29684]{-1}[-1/-1] 2019-07-01 23:28:16.889653 e Crypto RootKeyStoreConsistencyChecker.cpp(00082) : SSFS inconsistent: PERSISTENCE BACKUP LOG
[29684]{-1}[-1/-1] 2019-07-01 23:28:16.889659 e Crypto RootKeyManagerSsfs.cpp(00122) : Failed to check consistency during readContent: Inconsistent SSFS!
[29684]{-1}[-1/-1] 2019-07-01 23:28:16.889753 e Backup BackupExe_RequestHandler.cpp(00370) : (S)Exception=<?xml version="1.0" encoding="UTF-8"?><error>
<exception>
<node file="Crypto/RootKeyManager/RootKeyManagerSsfs.cpp" line= "123" number="301154">RmFpbGVkIHRvIGNoZWNrIGNvbnNpc3RlbmN5IGR1cmluZyByZWFkQ29udGVudDogSW5jb25zaXN0ZW50IFNTRlMh</node>
</exception>
</error>
|(I)ReturnCode=1|
[30061]{-1}[-1/-1] 2019-07-01 23:28:16.889971 e Backup BackupMgr_ExecutorJob.cpp(00138) : Sending BackupSaveMetadata erpdev:31003: exception 1: no.301154 (unknown file:0) TID: 30061
Failed to check consistency during readContent: Inconsistent SSFS!
===========================================================

Also, resetting the consistency of the SSFS on the failing tenant Database was not helpful.
===========================================================
hdbcons -p <pid of indexserver of inconsistent tenant> "crypto ssfs resetCache"
hdbcons -p <pid of indexserver of inconsistent tenant> "crypto ssfs resetConsistency"
===========================================================

HANA 2.0

tenant, backup, crypto ssfs resetConsistency, Inconsistent, SSFS, readContent , KBA , HAN-DB-SEC , SAP HANA Security & User Management , Problem