3626928 - Log Backup Stopped Working Suddenly

The log volume (/hana/log/<SID>) became full. Upon checking the state of the log segments, most of them were in state "Truncated".
Further investigation indicates that regular and automatic log backups have stopped. 

• The backup.log shows log backup has started, but there is no log backup finish messages.
  --------------------------------------
  2025-06-23T13:05:27+08:00  P0050312      1979b2d500f INFO    LOGBCKUP state of service: indexserver, hostname:30091, volume: 5, BackupLogStarted
  2025-06-23T13:05:27+08:00  P0050312      1979b2d500f INFO    LOGBCKUP to backint: /usr/sap/SID/SYS/global/hdb/backint/DB_DB0/log_backup_5_0_4330915262976_4330915270656
  2025-06-23T13:08:37+08:00  P0050312      1979b30368f INFO    LOGBCKUP state of service: xsengine, hostname:30007, volume: 2, BackupLogStarted
  2025-06-23T13:08:37+08:00  P0050312      1979b30368f INFO    LOGBCKUP to backint: /usr/sap/SID/SYS/global/hdb/backint/DB_DB0/log_backup_2_0_1484172480_1484178688
  2025-06-23T21:02:45+08:00  P0050312      1979ce24ba0 INFO    BACKUP   SAVE DATA started [1750683765664]
  -------------------------------------- 
  
  
• The backint.log does not contain any logs around the issue time period.
  
  
• The indexserver trace does not contain any log backup related errors.
  
  
• HANA thread samples show that there are log backup threads in the "Running" state for a long time. For example: 
  ---------------------------------
  58267;2025/06/23 13:10:19.613;      1; 20.00;hostname;30085; 1;  0;   0;212201.2;     -1;    53896;           ; ;LogBackupThread                            ;LogBackupThread                                              ;Running                         ; ;?                                                                 ;no SQL (LogBackupThread)                                              ;LogBackupThread                                                                                                                                                                                                                                                                               ;?                  ;LogBackupThread                                     ;?              ;LogBackupThread                                                                                                                                                                                                                                                 ;                   ;                     ;?            ;        -1;                -1;-1        ;                             ;                          ;                                                                                                                                                                                                                                                                ;?               ;no SQL (LogBackupThread)                                              ;      -1;        ;2025;06   ;23 ;13  ;        58;        3536;  212201241
  ---------------------------------
  
  
• HANA call stack view does not contain ay information about the log backup threads.
  
  
• The Linux OS system log (/var/log/messages) contains the following messages.
  ---------------------------------
  Jun 23 13:05:51 hostname kernel: INFO: task LogBackupThread:53896 blocked for more than 120 seconds.
  Jun 23 13:05:51 hostname kernel: INFO: task LogBackupThread:53896 blocked for more than 120 seconds.
  Jun 23 13:05:51 hostname kernel: INFO: task LogBackupThread:53896 blocked for more than 120 seconds.
  Jun 23 13:05:51 hostname kernel:      Tainted: G           OE    --------- -  - 4.18.0-193.40.1.el8_2.x86_64 #1
  Jun 23 13:05:51 hostname kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Jun 23 13:05:51 hostname kernel: LogBackupThread D    0 53896  47961 0x00004080
  Jun 23 13:05:51 hostname kernel: Call Trace:
  Jun 23 13:05:51 hostname kernel:      Tainted: G           OE    --------- -  - 4.18.0-193.40.1.el8_2.x86_64 #1
  Jun 23 13:05:51 hostname kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Jun 23 13:05:51 hostname kernel: Call Trace:
  Jun 23 13:05:51 hostname kernel:      Tainted: G           OE    --------- -  - 4.18.0-193.40.1.el8_2.x86_64 #1
  Jun 23 13:05:51 hostname kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Jun 23 13:05:51 hostname kernel: LogBackupThread D    0 53896  47961 0x00004080
  Jun 23 13:05:51 hostname kernel: Call Trace:
  Jun 23 13:05:51 hostname kernel: ? __schedule+0x24f/0x650
  Jun 23 13:05:51 hostname kernel: ? __schedule+0x24f/0x650
  Jun 23 13:05:51 hostname kernel: ? __schedule+0x24f/0x650
  Jun 23 13:05:51 hostname kernel: ? nfs_check_cache_invalid+0x33/0x90 [nfs]
  Jun 23 13:05:51 hostname kernel: ? nfs_check_cache_invalid+0x33/0x90 [nfs]
  Jun 23 13:05:51 hostname kernel: ? nfs_check_cache_invalid+0x33/0x90 [nfs]
  Jun 23 13:05:51 hostname kernel: schedule+0x2f/0xa0
  Jun 23 13:05:51 hostname kernel: schedule+0x2f/0xa0
  Jun 23 13:05:51 hostname kernel: schedule+0x2f/0xa0
  Jun 23 13:05:51 hostname kernel: rwsem_down_read_slowpath+0x360/0x3d0
  Jun 23 13:05:51 hostname kernel: rwsem_down_read_slowpath+0x360/0x3d0
  Jun 23 13:05:51 hostname kernel: rwsem_down_read_slowpath+0x360/0x3d0
  Jun 23 13:05:51 hostname kernel: lookup_slow+0x27/0x50
  Jun 23 13:05:51 hostname kernel: lookup_slow+0x27/0x50
  Jun 23 13:05:51 hostname kernel: lookup_slow+0x27/0x50
  Jun 23 13:05:51 hostname kernel: walk_component+0x1bf/0x330
  Jun 23 13:05:51 hostname kernel: walk_component+0x1bf/0x330
  Jun 23 13:05:51 hostname kernel: walk_component+0x1bf/0x330
  Jun 23 13:05:51 hostname kernel: ? xfs_iomap_write_unwritten+0x2d0/0x2d0 [xfs]
  Jun 23 13:05:51 hostname kernel: ? xfs_iomap_write_unwritten+0x2d0/0x2d0 [xfs]
  Jun 23 13:05:51 hostname kernel: ? xfs_iomap_write_unwritten+0x2d0/0x2d0 [xfs]
  Jun 23 13:05:51 hostname kernel: path_lookupat.isra.49+0x75/0x200
  Jun 23 13:05:51 hostname kernel: path_lookupat.isra.49+0x75/0x200
  Jun 23 13:05:51 hostname kernel: path_lookupat.isra.49+0x75/0x200
  Jun 23 13:05:51 hostname kernel: filename_lookup.part.63+0xa0/0x170
  Jun 23 13:05:51 hostname kernel: filename_lookup.part.63+0xa0/0x170
  Jun 23 13:05:51 hostname kernel: filename_lookup.part.63+0xa0/0x170
  Jun 23 13:05:51 hostname kernel: ? __audit_getname+0x96/0xb0
  Jun 23 13:05:51 hostname kernel: ? __audit_getname+0x96/0xb0
  Jun 23 13:05:51 hostname kernel: ? __audit_getname+0x96/0xb0
  Jun 23 13:05:51 hostname kernel: vfs_statx+0x73/0xe0
  Jun 23 13:05:51 hostname kernel: vfs_statx+0x73/0xe0
  Jun 23 13:05:51 hostname kernel: vfs_statx+0x73/0xe0
  Jun 23 13:05:51 hostname kernel: __do_sys_newstat+0x39/0x70
  Jun 23 13:05:51 hostname kernel: __do_sys_newstat+0x39/0x70
  Jun 23 13:05:51 hostname kernel: __do_sys_newstat+0x39/0x70
  Jun 23 13:05:51 hostname kernel: ? syscall_trace_enter+0x1d3/0x2c0
  Jun 23 13:05:51 hostname kernel: ? syscall_trace_enter+0x1d3/0x2c0
  Jun 23 13:05:51 hostname kernel: ? syscall_trace_enter+0x1d3/0x2c0
  Jun 23 13:05:51 hostname kernel: ? __audit_syscall_exit+0x249/0x2a0
  Jun 23 13:05:51 hostname kernel: ? __audit_syscall_exit+0x249/0x2a0
  Jun 23 13:05:51 hostname kernel: ? __audit_syscall_exit+0x249/0x2a0
  Jun 23 13:05:51 hostname kernel: do_syscall_64+0x5b/0x1a0
  Jun 23 13:05:51 hostname kernel: do_syscall_64+0x5b/0x1a0
  Jun 23 13:05:51 hostname kernel: do_syscall_64+0x5b/0x1a0
  Jun 23 13:05:51 hostname kernel: entry_SYSCALL_64_after_hwframe+0x65/0xca
  Jun 23 13:05:51 hostname kernel: entry_SYSCALL_64_after_hwframe+0x65/0xca
  Jun 23 13:05:51 hostname kernel: entry_SYSCALL_64_after_hwframe+0x65/0xca
  Jun 23 13:05:51 hostname kernel: RIP: 0033:0x7fdeeadadf59
  Jun 23 13:05:51 hostname kernel: RIP: 0033:0x7fdeeadadf59
  Jun 23 13:05:51 hostname kernel: RIP: 0033:0x7fdeeadadf59
  Jun 23 13:05:51 hostname kernel: Code: Bad RIP value.
  Jun 23 13:05:51 hostname kernel: Code: Bad RIP value.
  Jun 23 13:05:51 hostname kernel: Code: Bad RIP value.
  ---------------------------------

 "Image/data in this KBA is from SAP internal systems, sample data, or demo systems. Any resemblance to real data is purely coincidental."

SAP HANA, platform edition 2.0

HANA, log volume. /hana/log, full, log backup, stop, hang, Running, LogBackupThread, blocked for more than 120 seconds, hung_task_timeout_secs, Linux, kernel, Tainted, nfs_check_cache_invalid, nfs , KBA , HAN-DB-BAC , SAP HANA Backup & Recovery , Problem