3278379 - Shadow pages consuming the data volume on HANA

• A growing number of shadow pages on the HANA database persistence data file(s) '/hana/data/DB_SID/mnt00002/hdb00004.00003/datavolume_0000.dat' are observed filling the capacity of the persistence space or leading to disk full events.

• A review of the consumers on the data persistence files reveals massive Shadow pages' consumption under 'Hana_Disk_Data_Overview' & 'HANA_Disks_Data_Pages' (Note 1969700) :

hana_disk_data_overview_

 ;HOST      ;PORT ;ALLOC_GB  ;USED_GB   ;COLTAB_GB;ROWTAB_GB;FLOB_GB  ;PLOB_GB  ;HISTORY_GB;SHADOW_GB ;LIVECACHE_GB;DOCSTORE_GB;OTHER_GB  
1;hostname;30003;   2032.28;   2053.40;  1367.63;     0.06;     0.00;     0.02;      0.00;    687.46;        0.00;       0.00;     -1.77
2;hostname;30003;   1442.06;   1229.99;  1229.89;     0.06;     0.00;     0.00;      0.00;      0.00;        0.00;       0.00;      0.03
3;hostname;30003;    715.45;    584.09;   563.64;     3.49;    14.55;     1.93;      0.01;      0.03;        0.00;       0.00;      0.41


HANA_Disks_Data_Pages_

   ;SAMPLE_TIME        ;HOST      ;PORT ;VOLUME_NAME;PAGE_CLASS;PAGE_SIZE;USED_PAGES;SHADOW_PAGES;USED_SIZE_GB;SHADOW_SIZE_GB;SHADOW_PCT
1  ;2022/11/17 02:56:45;hostname;  %03;any        ;any       ;      any;    598956;      357024;     1365.94;        687.46;     50.32
2  ;2022/11/17 02:51:43;hostname;  %03;any        ;any       ;      any;    598956;      357024;     1365.94;        687.46;     50.32

• Mini check reports may show alerts like these too. 

125;M0355;Time since last savepoint (s)                    ;hostname;75                                                           ;<= 900              ; ; 2100009
126;     ;                                                 ;hostname;171787                                                       ;<= 900              ;X; 2100009


150;M0383;Max. size of shadow pages (GB, short-term)       ;hostname;3.75                                                         ;<= 200.00           ; ; 2100009
151;     ;                                                 ;hostname;687.46                                                       ;<= 200.00           ;X; 2100009

• Disk full events can occur in this instance too. The last successful savepoint was here @ 2022-11-15 03:13:31 just before the disk got full on that data file. :

indexserver_hostname.30003.266.trc

[43074]{-1}[-1/-1] 2022-11-15 02:53:24.834386 i Savepoint        SavepointImpl.cpp(03049) : Savepoint current savepoint version: 81487, restart redo log position: 0x1e722f6dc3, next savepoint version: 81488, last snapshot SP version: 0
[43074]{-1}[-1/-1] 2022-11-15 02:58:24.869093 i Savepoint        SavepointImpl.cpp(03049) : Savepoint current savepoint version: 81488, restart redo log position: 0x1e72492689, next savepoint version: 81489, last snapshot SP version: 0
[43074]{-1}[-1/-1] 2022-11-15 03:03:25.120917 i Savepoint        SavepointImpl.cpp(03049) : Savepoint current savepoint version: 81489, restart redo log position: 0x1e74734a9a, next savepoint version: 81490, last snapshot SP version: 0
[43074]{-1}[-1/-1] 2022-11-15 03:08:28.853888 i Savepoint        SavepointImpl.cpp(03049) : Savepoint current savepoint version: 81490, restart redo log position: 0x1e7712169a, next savepoint version: 81491, last snapshot SP version: 0
[43074]{-1}[-1/-1] 2022-11-15 03:13:31.898069 i Savepoint        SavepointImpl.cpp(03049) : Savepoint current savepoint version: 81491, restart redo log position: 0x1e77ee8503, next savepoint version: 81492, last snapshot SP version: 0
[42950]{-1}[-1/-1] 2022-11-15 03:20:27.858165 i EventHandler     LocalFileCallback.cpp(00491) : [DISKFULL] (1st request) [W] , buffer= 0x00007e22d9aa6000, offset= 1706662297600, size= 0/16777216, file= "<root>/datavolume_0000.dat" ((open, mode= RW, access= rw-rw-r--, flags= ASYNC|DIRECT|MUST_EXIST|LOCK), factory= (root= "/hana/data/DB_SID/mnt00002/hdb00004.00003/" (access= rw-rw-r--, flags= AUTOCREATE_PATH, usage= DATA, fs= xfs, config= (async_write_submit_active=on,async_write_submit_blocks=all,async_read_submit=on,num_submit_queues=1,num_completion_queues=1,size_kernel_io_queue=512,max_parallel_io_requests=64,min_submit_batch_size=16,max_submit_batch_size=64))) {shortRetries= 0, fullRetries= 0 (0/10)}
[42950]{-1}[-1/-1] 2022-11-15 03:20:27.858237 i EventHandler     EventManagerImpl.cpp(00675) : New event reported: 'DiskFullEvent[id= 4, path= /hana/data/DB_SID/mnt00002/hdb00004.00003/, state= NEW]'
[42950]{-1}[-1/-1] 2022-11-15 03:20:27.858248 i Logger           LoggerImpl.cpp(00104) : Logger notified of new DiskFull: DiskFullEvent[id= 4, path= /hana/data/DB_SID/mnt00002/hdb00004.00003/, state= NEW]
[72148]{-1}[-1/-1] 2022-11-15 03:21:11.280101 i TraceContext     TraceContext.cpp(01269) : UserName=
[72148]{-1}[-1/-1] 2022-11-15 03:21:11.279959 i EventHandler     LocalFileCallback.cpp(00517) : [DISKFULL] restarting queue with 717 requests
[72148]{-1}[-1/-1] 2022-11-15 03:21:11.280128 i EventHandler     EventManagerImpl.cpp(00951) : Event 'DiskFullEvent[id= 4, path= /hana/data/DB_SID/mnt00002/hdb00004.00003/, state= HANDLING]' set to state 'handled'
[42950]{-1}[-1/-1] 2022-11-15 03:21:11.280246 i Logger           LoggerImpl.cpp(00117) : Logger notified of not yet handled DiskFull: DiskFullEvent[id= 4, path= /hana/data/DB_SID/mnt00002/hdb00004.00003/, state= NEW]

......

          System has been online since then with no indication of any successful savepoints since :

[42950]{-1}[-1/-1] 2022-11-17 04:27:40.123532 i Logger           LoggerImpl.cpp(00117) : Logger notified of not yet handled DiskFull: DiskFullEvent[id= 4, path= /hana/data/DB_SID/mnt00002/hdb00004.00003/, state= NEW]

SAP HANA, platform edition 2.0

SAP HANA, platform edition 1.0

data volume full, disk full, hana/data, hana/data full, data volume growing , KBA , HAN-DB-PER , SAP HANA Database Persistence , Problem