diff --git a/documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt b/documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt index 87b82776..ee61e51f 100644 --- a/documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt +++ b/documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt @@ -24,3 +24,72 @@ 2026-10-20T16:29:44 health=200 0.010823 diskread=7GB 2026-10-20T16:30:45 health=200 0.009510 diskread=9GB 2026-10-20T16:31:47 health=200 0.259351 diskread=10GB +2026-10-20T16:32:49 health=200 0.137947 diskread=13GB +2026-10-20T16:33:51 health=200 0.048527 diskread=14GB +2026-10-20T16:34:53 health=200 0.236325 diskread=15GB +2026-10-20T16:35:55 health=200 0.729571 diskread=15GB +2026-10-20T16:36:57 health=200 0.306518 diskread=15GB +2026-10-20T16:37:59 health=200 0.328963 diskread=15GB +2026-10-20T16:39:01 health=200 0.135941 diskread=15GB +2026-10-20T16:40:03 health=200 0.342418 diskread=15GB +2026-10-20T16:41:04 health=200 0.057255 diskread=15GB +2026-10-20T16:42:06 health=200 0.055219 diskread=15GB +2026-10-20T16:43:08 health=200 0.047625 diskread=15GB +2026-10-20T16:44:09 health=200 0.048725 diskread=15GB +2026-10-20T16:45:11 health=200 0.047987 diskread=15GB +2026-10-20T16:46:13 health=200 0.046522 diskread=15GB +2026-10-20T16:47:14 health=200 0.045488 diskread=15GB +2026-10-20T16:48:16 health=200 0.060400 diskread=15GB +2026-10-20T16:49:18 health=200 0.053359 diskread=16GB +2026-10-20T16:50:19 health=200 0.045452 diskread=16GB +2026-10-20T16:51:21 health=200 0.054048 diskread=16GB +2026-10-20T16:52:23 health=200 0.046350 diskread=16GB +2026-10-20T16:53:25 health=200 0.059690 diskread=16GB +2026-10-20T16:54:27 health=200 0.289137 diskread=16GB +2026-10-20T16:55:29 health=200 0.274621 diskread=16GB +2026-10-20T16:56:31 health=200 0.150415 diskread=16GB +2026-10-20T16:57:33 health=200 0.407832 diskread=16GB +2026-10-20T16:58:34 health=200 0.109780 diskread=16GB +2026-10-20T16:59:36 health=200 0.166653 diskread=16GB +2026-10-20T17:00:38 health=200 0.157504 diskread=16GB +2026-10-20T17:01:39 health=200 0.010882 diskread=17GB +2026-10-20T17:02:41 health=200 0.010171 diskread=19GB +2026-10-20T17:03:42 health=200 0.010227 diskread=19GB +2026-10-20T17:04:44 health=200 0.010828 diskread=21GB +2026-10-20T17:05:46 health=200 0.010940 diskread=23GB +2026-10-20T17:06:48 health=200 0.014798 diskread=25GB +2026-10-20T17:07:49 health=200 0.030366 diskread=27GB +2026-10-20T17:08:51 health=200 0.012455 diskread=27GB +2026-10-20T17:09:52 health=200 0.012290 diskread=27GB +2026-10-20T17:10:54 health=200 0.010052 diskread=27GB +2026-10-20T17:11:56 health=200 0.011465 diskread=27GB +2026-10-20T17:12:57 health=200 0.015121 diskread=27GB +2026-10-20T17:13:59 health=200 0.011157 diskread=27GB +2026-10-20T17:15:01 health=200 0.012024 diskread=27GB +2026-10-20T17:16:02 health=200 0.010357 diskread=28GB +2026-10-20T17:17:04 health=200 0.017752 diskread=33GB +2026-10-20T17:18:05 health=200 0.009962 diskread=33GB +2026-10-20T17:19:07 health=200 0.011363 diskread=33GB +2026-10-20T17:20:09 health=200 0.011524 diskread=34GB +2026-10-20T17:21:10 health=200 0.009969 diskread=34GB +2026-10-20T17:22:12 health=200 0.841755 diskread=35GB +2026-10-20T17:23:14 health=200 0.010010 diskread=38GB +2026-10-20T17:24:16 health=200 0.012834 diskread=42GB +2026-10-20T17:25:18 health=200 0.012292 diskread=42GB +2026-10-20T17:26:20 health=200 0.014307 diskread=42GB +2026-10-20T17:27:21 health=200 0.016217 diskread=42GB +2026-10-20T17:28:23 health=200 0.010172 diskread=42GB +2026-10-20T17:29:25 health=000 0.331161 diskread=0GB +2026-10-20T17:30:29 health=000 3.054892 diskread=0GB +2026-10-20T17:31:31 health=200 0.011323 diskread=3GB +2026-10-20T17:32:33 health=200 0.009743 diskread=4GB +2026-10-20T17:33:35 health=200 0.012219 diskread=4GB +2026-10-20T17:34:39 health=000 3.077297 diskread=0GB +2026-10-20T17:35:44 health=000 3.069825 diskread=0GB +2026-10-20T17:36:46 health=200 0.011239 diskread=3GB +2026-10-20T17:37:47 health=200 0.010636 diskread=4GB +2026-10-20T17:38:49 health=200 0.009873 diskread=4GB +2026-10-20T17:39:50 health=200 0.011005 diskread=4GB +2026-10-20T17:40:52 health=200 0.011215 diskread=5GB +2026-10-20T17:41:53 health=200 0.011258 diskread=5GB +2026-10-20T17:42:54 health=200 0.012194 diskread=5GB diff --git a/documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt b/documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt index bceede3c..120feabd 100644 --- a/documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt +++ b/documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt @@ -29,4 +29,143 @@ 16:30:19Z used=3288MB avail=808MB load1=3.46 cpu=46% paperless-webserver:533/1280MB 0% | immich-server:502/3072MB 12% | immich-machine-learning:431/2048MB 1% | immich-postgres:132/768MB 3% | bookstack:131/512MB 0% | felhom-controller:116/4096MB 46% | mealie:103/1000MB 87% | bookstack-db:97/256MB 0% 16:30:50Z used=3638MB avail=458MB load1=2.35 cpu=23% paperless-webserver:533/1280MB 0% | immich-server:502/3072MB 12% | immich-machine-learning:431/2048MB 1% | immich-postgres:132/768MB 3% | bookstack:131/512MB 0% | felhom-controller:116/4096MB 46% | mealie:103/1000MB 87% | bookstack-db:97/256MB 0% 16:31:20Z used=3638MB avail=458MB load1=4.47 cpu=95% immich-server:908/3072MB 0% | mealie:244/1000MB 0% | immich-machine-learning:168/2048MB 0% | felhom-controller:166/4096MB 126% | nextcloud:162/1024MB 0% | nextcloud-db:131/512MB 0% | paperless-webserver:114/1280MB 66% | actualbudget:85/256MB 0% -16:31:50Z used=3438MB avail=658MB load1=3.84 cpu=44% immich-server:908/3072MB 0% | mealie:244/1000MB 0% | immich-machine-learning:168/2048MB 0% | felhom-controller:166/4096MB 126% | nextcloud:162/1024MB 0% | nextcloud-db:131/512MB 0% | paperless-webs \ No newline at end of file +16:31:50Z used=3438MB avail=658MB load1=3.84 cpu=44% immich-server:908/3072MB 0% | mealie:244/1000MB 0% | immich-machine-learning:168/2048MB 0% | felhom-controller:166/4096MB 126% | nextcloud:162/1024MB 0% | nextcloud-db:131/512MB 0% | paperless-webserver:114/1280MB 66% | actualbudget:85/256MB 0% +16:32:21Z used=3638MB avail=458MB load1=3.85 cpu=61% paperless-webserver:526/1280MB 0% | mealie:250/1000MB 0% | nextcloud-db:123/512MB 0% | bookstack-db:116/256MB 0% | felhom-controller:114/4096MB 84% | traefik:84/4096MB 2% | actualbudget:62/256MB 0% | nextcloud:50/1024MB 0% +16:32:51Z used=3288MB avail=808MB load1=5.41 cpu=48% paperless-webserver:526/1280MB 0% | mealie:250/1000MB 0% | nextcloud-db:123/512MB 0% | bookstack-db:116/256MB 0% | felhom-controller:114/4096MB 84% | traefik:84/4096MB 2% | actualbudget:62/256MB 0% | nextcloud:50/1024MB 0% +16:33:23Z used=3038MB avail=1058MB load1=4.16 cpu=48% immich-server:842/3072MB 5% | paperless-webserver:422/1280MB 0% | mealie:208/1000MB 0% | felhom-controller:96/4096MB 0% | immich-machine-learning:89/2048MB 0% | immich-postgres:79/768MB 0% | bookstack-db:74/256MB 0% | traefik:56/4096MB 26% +16:33:53Z used=3638MB avail=458MB load1=3.01 cpu=11% immich-server:842/3072MB 5% | paperless-webserver:422/1280MB 0% | mealie:208/1000MB 0% | felhom-controller:96/4096MB 0% | immich-machine-learning:89/2048MB 0% | immich-postgres:79/768MB 0% | bookstack-db:74/256MB 0% | traefik:56/4096MB 26% +16:34:23Z used=3638MB avail=458MB load1=2.18 cpu=33% immich-server:882/3072MB 4% | felhom-controller:227/4096MB 38% | mealie:208/1000MB 0% | nextcloud:170/1024MB 0% | nextcloud-db:134/512MB 0% | immich-machine-learning:97/2048MB 0% | paperless-webserver:91/1280MB 0% | actualbudget:87/256MB 0% +16:34:54Z used=3638MB avail=458MB load1=2.12 cpu=8% immich-server:882/3072MB 4% | felhom-controller:227/4096MB 38% | mealie:208/1000MB 0% | nextcloud:170/1024MB 0% | nextcloud-db:134/512MB 0% | immich-machine-learning:97/2048MB 0% | paperless-webserver:91/1280MB 0% | actualbudget:87/256MB 0% +16:35:25Z used=3638MB avail=458MB load1=1.42 cpu=7% immich-server:840/3072MB 7% | paperless-webserver:714/1280MB 0% | felhom-controller:273/4096MB 17% | mealie:217/1000MB 0% | nextcloud:177/1024MB 0% | nextcloud-db:123/512MB 7% | immich-machine-learning:102/2048MB 0% | actualbudget:96/256MB 0% +16:35:55Z used=3638MB avail=458MB load1=1 cpu=11% immich-server:840/3072MB 7% | paperless-webserver:714/1280MB 0% | felhom-controller:273/4096MB 17% | mealie:217/1000MB 0% | nextcloud:177/1024MB 0% | nextcloud-db:123/512MB 7% | immich-machine-learning:102/2048MB 0% | actualbudget:96/256MB 0% +16:36:26Z used=3638MB avail=458MB load1=1.33 cpu=7% immich-server:735/3072MB 6% | paperless-webserver:561/1280MB 0% | felhom-controller:281/4096MB 168% | mealie:218/1000MB 0% | nextcloud-db:110/512MB 6% | immich-machine-learning:103/2048MB 0% | actualbudget:96/256MB 0% | immich-postgres:94/768MB 0% +16:36:56Z used=3638MB avail=458MB load1=0.8 cpu=8% immich-server:735/3072MB 6% | paperless-webserver:561/1280MB 0% | felhom-controller:281/4096MB 168% | mealie:218/1000MB 0% | nextcloud-db:110/512MB 6% | immich-machine-learning:103/2048MB 0% | actualbudget:96/256MB 0% | immich-postgres:94/768MB 0% +16:37:26Z used=3638MB avail=458MB load1=0.55 cpu=8% immich-server:736/3072MB 5% | paperless-webserver:561/1280MB 0% | felhom-controller:250/4096MB 26% | mealie:218/1000MB 0% | nextcloud-db:110/512MB 8% | immich-machine-learning:103/2048MB 0% | actualbudget:96/256MB 0% | immich-postgres:94/768MB 0% +16:37:58Z used=3638MB avail=458MB load1=0.62 cpu=7% immich-server:739/3072MB 0% | paperless-webserver:562/1280MB 0% | felhom-controller:251/4096MB 36% | mealie:237/1000MB 0% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 0% +16:38:28Z used=3638MB avail=458MB load1=0.57 cpu=3% immich-server:739/3072MB 0% | paperless-webserver:562/1280MB 0% | felhom-controller:251/4096MB 36% | mealie:237/1000MB 0% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 0% +16:38:59Z used=3638MB avail=458MB load1=0.35 cpu=5% immich-server:739/3072MB 0% | paperless-webserver:561/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 18% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 31% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 0% +16:39:29Z used=3638MB avail=458MB load1=0.36 cpu=3% immich-server:739/3072MB 0% | paperless-webserver:561/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 18% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 31% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 0% +16:40:00Z used=3638MB avail=458MB load1=0.38 cpu=4% immich-server:739/3072MB 0% | paperless-webserver:562/1280MB 0% | mealie:237/1000MB 26% | felhom-controller:225/4096MB 38% | immich-machine-learning:118/2048MB 27% | nextcloud-db:110/512MB 0% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 0% +16:40:30Z used=3638MB avail=458MB load1=0.76 cpu=10% immich-server:739/3072MB 0% | paperless-webserver:562/1280MB 0% | mealie:237/1000MB 26% | felhom-controller:225/4096MB 38% | immich-machine-learning:118/2048MB 27% | nextcloud-db:110/512MB 0% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 0% +16:41:02Z used=3638MB avail=458MB load1=1.27 cpu=19% immich-server:741/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 25% | felhom-controller:224/4096MB 137% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 10% +16:41:32Z used=3638MB avail=458MB load1=1.57 cpu=10% immich-server:741/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 25% | felhom-controller:224/4096MB 137% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 10% +16:42:03Z used=3638MB avail=458MB load1=1.02 cpu=15% immich-server:741/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 37% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:95/768MB 9% +16:42:34Z used=3638MB avail=458MB load1=0.8 cpu=9% immich-server:741/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 37% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:95/768MB 9% +16:43:04Z used=3638MB avail=458MB load1=0.64 cpu=14% immich-server:739/3072MB 3% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 21% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:94/768MB 0% +16:43:34Z used=3638MB avail=458MB load1=0.45 cpu=10% immich-server:739/3072MB 3% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 21% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:94/768MB 0% +16:44:04Z used=3638MB avail=458MB load1=0.35 cpu=5% immich-server:740/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 1% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:94/768MB 0% +16:44:35Z used=3638MB avail=458MB load1=0.4 cpu=5% immich-server:740/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 1% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:94/768MB 0% +16:45:06Z used=3638MB avail=458MB load1=0.24 cpu=5% immich-server:750/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:236/1000MB 0% | felhom-controller:224/4096MB 21% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:97/768MB 0% +16:45:36Z used=3638MB avail=458MB load1=0.27 cpu=5% immich-server:750/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:236/1000MB 0% | felhom-controller:224/4096MB 21% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:97/768MB 0% +16:46:06Z used=3638MB avail=458MB load1=1.22 cpu=14% immich-server:747/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:234/1000MB 0% | felhom-controller:233/4096MB 154% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:100/256MB 0% | immich-postgres:98/768MB 0% +16:46:36Z used=3638MB avail=458MB load1=1.26 cpu=5% immich-server:747/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:234/1000MB 0% | felhom-controller:233/4096MB 154% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:100/256MB 0% | immich-postgres:98/768MB 0% +16:47:06Z used=3638MB avail=458MB load1=0.94 cpu=5% immich-server:747/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:234/1000MB 0% | felhom-controller:234/4096MB 21% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:100/256MB 0% | immich-postgres:98/768MB 0% +16:47:37Z used=3638MB avail=458MB load1=0.84 cpu=6% immich-server:747/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:234/1000MB 0% | felhom-controller:234/4096MB 21% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:100/256MB 0% | immich-postgres:98/768MB 0% +16:48:09Z used=3638MB avail=458MB load1=0.64 cpu=8% immich-server:748/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:234/1000MB 0% | felhom-controller:175/4096MB 36% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:100/256MB 0% | immich-postgres:98/768MB 0% +16:48:39Z used=3638MB avail=458MB load1=1.1 cpu=14% immich-server:748/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:234/1000MB 0% | felhom-controller:175/4096MB 36% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:100/256MB 0% | immich-postgres:98/768MB 0% +16:49:10Z used=3638MB avail=458MB load1=0.78 cpu=6% immich-server:675/3072MB 0% | paperless-webserver:529/1280MB 0% | felhom-controller:270/4096MB 19% | mealie:220/1000MB 0% | actualbudget:119/256MB 0% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:49:40Z used=3638MB avail=458MB load1=0.65 cpu=4% immich-server:675/3072MB 0% | paperless-webserver:529/1280MB 0% | felhom-controller:270/4096MB 19% | mealie:220/1000MB 0% | actualbudget:119/256MB 0% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:50:10Z used=3638MB avail=458MB load1=0.66 cpu=4% immich-server:694/3072MB 0% | paperless-webserver:538/1280MB 0% | felhom-controller:268/4096MB 51% | mealie:219/1000MB 0% | actualbudget:119/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:50:40Z used=3638MB avail=458MB load1=0.53 cpu=4% immich-server:694/3072MB 0% | paperless-webserver:538/1280MB 0% | felhom-controller:268/4096MB 51% | mealie:219/1000MB 0% | actualbudget:119/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:51:10Z used=3638MB avail=458MB load1=0.78 cpu=4% immich-server:700/3072MB 0% | paperless-webserver:537/1280MB 1% | felhom-controller:241/4096MB 160% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:51:41Z used=3638MB avail=458MB load1=0.77 cpu=5% immich-server:700/3072MB 0% | paperless-webserver:537/1280MB 1% | felhom-controller:241/4096MB 160% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:52:11Z used=3638MB avail=458MB load1=0.63 cpu=6% immich-server:701/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:242/4096MB 34% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:52:42Z used=3638MB avail=458MB load1=0.41 cpu=5% immich-server:701/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:242/4096MB 34% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:53:13Z used=3638MB avail=458MB load1=0.44 cpu=4% immich-server:703/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:235/4096MB 20% | mealie:218/1000MB 0% | actualbudget:119/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:53:43Z used=3638MB avail=458MB load1=0.47 cpu=6% immich-server:703/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:235/4096MB 20% | mealie:218/1000MB 0% | actualbudget:119/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:54:13Z used=3638MB avail=458MB load1=0.34 cpu=5% immich-server:704/3072MB 2% | paperless-webserver:537/1280MB 0% | felhom-controller:236/4096MB 35% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:54:44Z used=3638MB avail=458MB load1=0.55 cpu=5% immich-server:704/3072MB 2% | paperless-webserver:537/1280MB 0% | felhom-controller:236/4096MB 35% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:55:15Z used=3638MB avail=458MB load1=0.46 cpu=4% immich-server:705/3072MB 1% | paperless-webserver:537/1280MB 0% | felhom-controller:234/4096MB 19% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:55:45Z used=3638MB avail=458MB load1=0.55 cpu=11% immich-server:705/3072MB 1% | paperless-webserver:537/1280MB 0% | felhom-controller:234/4096MB 19% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:56:16Z used=3638MB avail=458MB load1=0.7 cpu=13% immich-server:710/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:232/4096MB 164% | mealie:217/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:56:47Z used=3638MB avail=458MB load1=0.42 cpu=7% immich-server:710/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:232/4096MB 164% | mealie:217/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:57:17Z used=3638MB avail=458MB load1=0.59 cpu=12% immich-server:710/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:231/4096MB 17% | mealie:217/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:57:47Z used=3638MB avail=458MB load1=0.48 cpu=6% immich-server:710/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:231/4096MB 17% | mealie:217/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +16:58:18Z used=3638MB avail=458MB load1=0.42 cpu=12% immich-server:712/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:232/4096MB 38% | mealie:217/1000MB 0% | actualbudget:118/256MB 1% | nextcloud-db:112/512MB 6% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 9% +16:58:50Z used=3638MB avail=458MB load1=0.72 cpu=7% immich-server:712/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:232/4096MB 38% | mealie:217/1000MB 0% | actualbudget:118/256MB 1% | nextcloud-db:112/512MB 6% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 9% +16:59:20Z used=3638MB avail=458MB load1=0.63 cpu=45% immich-server:713/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:233/4096MB 24% | mealie:217/1000MB 0% | actualbudget:118/256MB 12% | nextcloud-db:113/512MB 5% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 8% +16:59:50Z used=3638MB avail=458MB load1=0.56 cpu=10% immich-server:713/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:233/4096MB 24% | mealie:217/1000MB 0% | actualbudget:118/256MB 12% | nextcloud-db:113/512MB 5% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 8% +17:00:21Z used=3638MB avail=458MB load1=0.61 cpu=30% immich-server:713/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:232/4096MB 37% | mealie:217/1000MB 0% | actualbudget:118/256MB 11% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +17:00:51Z used=3638MB avail=458MB load1=0.72 cpu=17% immich-server:713/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:232/4096MB 37% | mealie:217/1000MB 0% | actualbudget:118/256MB 11% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +17:01:22Z used=3638MB avail=458MB load1=1.81 cpu=52% immich-server:720/3072MB 0% | paperless-webserver:538/1280MB 0% | felhom-controller:238/4096MB 138% | mealie:216/1000MB 0% | actualbudget:117/256MB 10% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +17:01:52Z used=3638MB avail=458MB load1=1.42 cpu=25% immich-server:720/3072MB 0% | paperless-webserver:538/1280MB 0% | felhom-controller:238/4096MB 138% | mealie:216/1000MB 0% | actualbudget:117/256MB 10% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0% +17:02:22Z used=3638MB avail=458MB load1=0.93 cpu=29% immich-server:548/3072MB 0% | paperless-webserver:496/1280MB 0% | mealie:174/1000MB 0% | felhom-controller:170/4096MB 0% | nextcloud-db:99/512MB 0% | immich-machine-learning:88/2048MB 0% | nextcloud:70/1024MB 0% | immich-postgres:67/768MB 0% +17:02:52Z used=3638MB avail=458MB load1=0.56 cpu=10% immich-server:548/3072MB 0% | paperless-webserver:496/1280MB 0% | mealie:174/1000MB 0% | felhom-controller:170/4096MB 0% | nextcloud-db:99/512MB 0% | immich-machine-learning:88/2048MB 0% | nextcloud:70/1024MB 0% | immich-postgres:67/768MB 0% +17:03:22Z used=3638MB avail=458MB load1=0.41 cpu=27% immich-server:555/3072MB 0% | paperless-webserver:496/1280MB 0% | mealie:177/1000MB 0% | felhom-controller:169/4096MB 17% | nextcloud-db:99/512MB 0% | immich-machine-learning:88/2048MB 0% | nextcloud:71/1024MB 0% | immich-postgres:67/768MB 0% +17:03:52Z used=3638MB avail=458MB load1=0.42 cpu=10% immich-server:555/3072MB 0% | paperless-webserver:496/1280MB 0% | mealie:177/1000MB 0% | felhom-controller:169/4096MB 17% | nextcloud-db:99/512MB 0% | immich-machine-learning:88/2048MB 0% | nextcloud:71/1024MB 0% | immich-postgres:67/768MB 0% +17:04:22Z used=3638MB avail=458MB load1=0.8 cpu=65% immich-server:556/3072MB 1% | paperless-webserver:496/1280MB 0% | mealie:177/1000MB 0% | felhom-controller:169/4096MB 48% | nextcloud-db:99/512MB 0% | immich-machine-learning:88/2048MB 0% | nextcloud:71/1024MB 0% | immich-postgres:67/768MB 0% +17:04:53Z used=1590MB avail=2506MB load1=3.55 cpu=20% immich-server:556/3072MB 1% | paperless-webserver:496/1280MB 0% | mealie:177/1000MB 0% | felhom-controller:169/4096MB 48% | nextcloud-db:99/512MB 0% | immich-machine-learning:88/2048MB 0% | nextcloud:71/1024MB 0% | immich-postgres:67/768MB 0% +17:05:23Z used=3288MB avail=808MB load1=4.55 cpu=76% paperless-webserver:498/1280MB 0% | immich-machine-learning:431/2048MB 0% | felhom-controller:190/4096MB 23% | mealie:167/1000MB 0% | bookstack:132/512MB 6% | immich-postgres:129/768MB 0% | traefik:127/4096MB 2% | bookstack-db:118/256MB 0% +17:05:53Z used=3038MB avail=1058MB load1=3.51 cpu=28% paperless-webserver:498/1280MB 0% | immich-machine-learning:431/2048MB 0% | felhom-controller:190/4096MB 23% | mealie:167/1000MB 0% | bookstack:132/512MB 6% | immich-postgres:129/768MB 0% | traefik:127/4096MB 2% | bookstack-db:118/256MB 0% +17:06:23Z used=3638MB avail=458MB load1=4.16 cpu=76% immich-server:891/3072MB 0% | mealie:237/1000MB 0% | nextcloud:161/1024MB 4% | nextcloud-db:125/512MB 0% | immich-machine-learning:118/2048MB 23% | actualbudget:112/256MB 0% | felhom-controller:92/4096MB 147% | immich-postgres:81/768MB 8% +17:06:53Z used=3638MB avail=458MB load1=4.32 cpu=50% immich-server:891/3072MB 0% | mealie:237/1000MB 0% | nextcloud:161/1024MB 4% | nextcloud-db:125/512MB 0% | immich-machine-learning:118/2048MB 23% | actualbudget:112/256MB 0% | felhom-controller:92/4096MB 147% | immich-postgres:81/768MB 8% +17:07:23Z used=3638MB avail=458MB load1=3.32 cpu=43% immich-server:867/3072MB 0% | paperless-webserver:540/1280MB 1% | mealie:232/1000MB 0% | felhom-controller:185/4096MB 193% | nextcloud-db:112/512MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | bookstack-db:66/256MB 0% +17:07:53Z used=3638MB avail=458MB load1=2.39 cpu=16% immich-server:867/3072MB 0% | paperless-webserver:540/1280MB 1% | mealie:232/1000MB 0% | felhom-controller:185/4096MB 193% | nextcloud-db:112/512MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | bookstack-db:66/256MB 0% +17:08:24Z used=3638MB avail=458MB load1=1.63 cpu=27% immich-server:682/3072MB 2% | paperless-webserver:497/1280MB 0% | mealie:212/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:84/256MB 0% | felhom-controller:79/4096MB 38% +17:08:54Z used=3638MB avail=458MB load1=0.99 cpu=11% immich-server:682/3072MB 2% | paperless-webserver:497/1280MB 0% | mealie:212/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:84/256MB 0% | felhom-controller:79/4096MB 38% +17:09:24Z used=3638MB avail=458MB load1=0.67 cpu=28% immich-server:663/3072MB 2% | paperless-webserver:497/1280MB 0% | mealie:212/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:85/256MB 0% | felhom-controller:78/4096MB 16% +17:09:54Z used=3638MB avail=458MB load1=0.56 cpu=9% immich-server:663/3072MB 2% | paperless-webserver:497/1280MB 0% | mealie:212/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:85/256MB 0% | felhom-controller:78/4096MB 16% +17:10:24Z used=3638MB avail=458MB load1=0.67 cpu=28% immich-server:663/3072MB 0% | paperless-webserver:496/1280MB 0% | mealie:212/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:85/256MB 0% | felhom-controller:79/4096MB 32% +17:10:54Z used=3638MB avail=458MB load1=0.48 cpu=16% immich-server:663/3072MB 0% | paperless-webserver:496/1280MB 0% | mealie:212/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:85/256MB 0% | felhom-controller:79/4096MB 32% +17:11:24Z used=3638MB avail=458MB load1=0.55 cpu=28% immich-server:663/3072MB 0% | paperless-webserver:494/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 0% | felhom-controller:77/4096MB 138% +17:11:54Z used=3638MB avail=458MB load1=0.63 cpu=9% immich-server:663/3072MB 0% | paperless-webserver:494/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 0% | felhom-controller:77/4096MB 138% +17:12:25Z used=3638MB avail=458MB load1=0.66 cpu=27% immich-server:663/3072MB 0% | paperless-webserver:495/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 7% | felhom-controller:77/4096MB 32% +17:12:55Z used=3638MB avail=458MB load1=0.4 cpu=8% immich-server:663/3072MB 0% | paperless-webserver:495/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 7% | felhom-controller:77/4096MB 32% +17:13:25Z used=3638MB avail=458MB load1=0.38 cpu=27% immich-server:664/3072MB 0% | paperless-webserver:494/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 6% | felhom-controller:77/4096MB 16% +17:13:55Z used=3638MB avail=458MB load1=0.84 cpu=9% immich-server:664/3072MB 0% | paperless-webserver:494/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 6% | felhom-controller:77/4096MB 16% +17:14:25Z used=3638MB avail=458MB load1=0.88 cpu=47% immich-server:665/3072MB 0% | paperless-webserver:494/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 7% | felhom-controller:77/4096MB 40% +17:14:55Z used=3638MB avail=458MB load1=0.68 cpu=13% immich-server:665/3072MB 0% | paperless-webserver:494/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 7% | felhom-controller:77/4096MB 40% +17:15:25Z used=3638MB avail=458MB load1=0.6 cpu=4% immich-server:660/3072MB 0% | paperless-webserver:493/1280MB 0% | mealie:110/1000MB 94% | immich-machine-learning:109/2048MB 0% | immich-postgres:100/768MB 0% | nextcloud-db:99/512MB 6% | bookstack-db:85/256MB 0% | felhom-controller:78/4096MB 40% +17:15:55Z used=3638MB avail=458MB load1=0.68 cpu=12% immich-server:660/3072MB 0% | paperless-webserver:493/1280MB 0% | mealie:110/1000MB 94% | immich-machine-learning:109/2048MB 0% | immich-postgres:100/768MB 0% | nextcloud-db:99/512MB 6% | bookstack-db:85/256MB 0% | felhom-controller:78/4096MB 40% +17:16:25Z used=3638MB avail=458MB load1=2.61 cpu=26% immich-server:658/3072MB 0% | paperless-webserver:492/1280MB 0% | mealie:293/1000MB 1% | immich-machine-learning:109/2048MB 0% | immich-postgres:100/768MB 0% | nextcloud-db:98/512MB 7% | felhom-controller:88/4096MB 158% | bookstack-db:84/256MB 0% +17:16:56Z used=3638MB avail=458MB load1=3.83 cpu=14% immich-server:658/3072MB 0% | paperless-webserver:492/1280MB 0% | mealie:293/1000MB 1% | immich-machine-learning:109/2048MB 0% | immich-postgres:100/768MB 0% | nextcloud-db:98/512MB 7% | felhom-controller:88/4096MB 158% | bookstack-db:84/256MB 0% +17:17:27Z used=3638MB avail=458MB load1=2.51 cpu=6% felhom-controller:952/4096MB 22% | immich-server:550/3072MB 0% | paperless-webserver:436/1280MB 0% | mealie:214/1000MB 0% | immich-machine-learning:108/2048MB 0% | nextcloud-db:78/512MB 0% | immich-postgres:62/768MB 0% | nextcloud:61/1024MB 0% +17:17:57Z used=3638MB avail=458MB load1=1.67 cpu=8% felhom-controller:951/4096MB 39% | immich-server:596/3072MB 1% | paperless-webserver:436/1280MB 0% | mealie:215/1000MB 0% | immich-machine-learning:117/2048MB 0% | traefik:100/4096MB 0% | nextcloud-db:77/512MB 0% | immich-postgres:73/768MB 0% +17:18:27Z used=3638MB avail=458MB load1=1.21 cpu=6% felhom-controller:951/4096MB 39% | immich-server:596/3072MB 1% | paperless-webserver:436/1280MB 0% | mealie:215/1000MB 0% | immich-machine-learning:117/2048MB 0% | traefik:100/4096MB 0% | nextcloud-db:77/512MB 0% | immich-postgres:73/768MB 0% +17:18:57Z used=3638MB avail=458MB load1=0.86 cpu=9% felhom-controller:950/4096MB 17% | immich-server:597/3072MB 1% | paperless-webserver:437/1280MB 0% | mealie:215/1000MB 0% | immich-machine-learning:117/2048MB 0% | traefik:100/4096MB 0% | nextcloud-db:77/512MB 0% | immich-postgres:73/768MB 0% +17:19:27Z used=3638MB avail=458MB load1=0.72 cpu=6% felhom-controller:950/4096MB 17% | immich-server:597/3072MB 1% | paperless-webserver:437/1280MB 0% | mealie:215/1000MB 0% | immich-machine-learning:117/2048MB 0% | traefik:100/4096MB 0% | nextcloud-db:77/512MB 0% | immich-postgres:73/768MB 0% +17:19:58Z used=3638MB avail=458MB load1=0.79 cpu=29% felhom-controller:967/4096MB 41% | immich-server:600/3072MB 0% | paperless-webserver:438/1280MB 0% | mealie:215/1000MB 0% | traefik:142/4096MB 0% | nextcloud-db:123/512MB 6% | immich-machine-learning:117/2048MB 0% | immich-postgres:73/768MB 0% +17:20:28Z used=3638MB avail=458MB load1=0.74 cpu=17% felhom-controller:967/4096MB 41% | immich-server:600/3072MB 0% | paperless-webserver:438/1280MB 0% | mealie:215/1000MB 0% | traefik:142/4096MB 0% | nextcloud-db:123/512MB 6% | immich-machine-learning:117/2048MB 0% | immich-postgres:73/768MB 0% +17:20:58Z used=3638MB avail=458MB load1=0.83 cpu=10% felhom-controller:946/4096MB 146% | immich-server:607/3072MB 1% | paperless-webserver:459/1280MB 0% | mealie:247/1000MB 0% | nextcloud:152/1024MB 0% | nextcloud-db:121/512MB 0% | immich-machine-learning:120/2048MB 0% | immich-postgres:71/768MB 9% +17:21:28Z used=3638MB avail=458MB load1=0.64 cpu=4% felhom-controller:946/4096MB 146% | immich-server:607/3072MB 1% | paperless-webserver:459/1280MB 0% | mealie:247/1000MB 0% | nextcloud:152/1024MB 0% | nextcloud-db:121/512MB 0% | immich-machine-learning:120/2048MB 0% | immich-postgres:71/768MB 9% +17:21:58Z used=3438MB avail=658MB load1=1.36 cpu=31% felhom-controller:946/4096MB 146% | immich-server:607/3072MB 1% | paperless-webserver:459/1280MB 0% | mealie:247/1000MB 0% | nextcloud:152/1024MB 0% | nextcloud-db:121/512MB 0% | immich-machine-learning:120/2048MB 0% | immich-postgres:71/768MB 9% +17:22:28Z used=3638MB avail=458MB load1=1.95 cpu=20% immich-server:532/3072MB 0% | paperless-webserver:449/1280MB 0% | mealie:219/1000MB 0% | nextcloud-db:132/512MB 4% | immich-postgres:129/768MB 10% | bookstack-db:113/256MB 101% | felhom-controller:105/4096MB 57% | immich-machine-learning:91/2048MB 0% +17:22:58Z used=3288MB avail=808MB load1=2.14 cpu=25% immich-server:903/3072MB 0% | paperless-webserver:317/1280MB 0% | mealie:232/1000MB 22% | immich-machine-learning:168/2048MB 0% | felhom-controller:99/4096MB 17% | immich-postgres:81/768MB 0% | actualbudget:79/256MB 0% | traefik:70/4096MB 0% +17:23:28Z used=3638MB avail=458MB load1=1.82 cpu=6% immich-server:903/3072MB 0% | paperless-webserver:317/1280MB 0% | mealie:232/1000MB 22% | immich-machine-learning:168/2048MB 0% | felhom-controller:99/4096MB 17% | immich-postgres:81/768MB 0% | actualbudget:79/256MB 0% | traefik:70/4096MB 0% +17:23:59Z used=3638MB avail=458MB load1=2.37 cpu=46% immich-server:979/3072MB 0% | paperless-webserver:249/1280MB 180% | mealie:249/1000MB 12% | immich-machine-learning:168/2048MB 30% | felhom-controller:126/4096MB 63% | nextcloud-db:118/512MB 0% | actualbudget:115/256MB 0% | immich-postgres:103/768MB 0% +17:24:29Z used=3638MB avail=458MB load1=2.35 cpu=9% immich-server:979/3072MB 0% | paperless-webserver:249/1280MB 180% | mealie:249/1000MB 12% | immich-machine-learning:168/2048MB 30% | felhom-controller:126/4096MB 63% | nextcloud-db:118/512MB 0% | actualbudget:115/256MB 0% | immich-postgres:103/768MB 0% +17:24:59Z used=3638MB avail=458MB load1=3.64 cpu=25% immich-server:806/3072MB 0% | paperless-webserver:590/1280MB 0% | mealie:181/1000MB 0% | felhom-controller:173/4096MB 158% | immich-machine-learning:156/2048MB 21% | nextcloud-db:95/512MB 0% | immich-postgres:70/768MB 0% | bookstack:68/512MB 0% +17:25:29Z used=3638MB avail=458MB load1=2.34 cpu=3% immich-server:806/3072MB 0% | paperless-webserver:590/1280MB 0% | mealie:181/1000MB 0% | felhom-controller:173/4096MB 158% | immich-machine-learning:156/2048MB 21% | nextcloud-db:95/512MB 0% | immich-postgres:70/768MB 0% | bookstack:68/512MB 0% +17:25:59Z used=3638MB avail=458MB load1=1.94 cpu=11% immich-server:808/3072MB 1% | paperless-webserver:590/1280MB 0% | mealie:187/1000MB 0% | immich-machine-learning:142/2048MB 0% | felhom-controller:135/4096MB 0% | nextcloud-db:95/512MB 0% | immich-postgres:79/768MB 0% | nextcloud:71/1024MB 0% +17:26:29Z used=3638MB avail=458MB load1=1.44 cpu=11% immich-server:808/3072MB 1% | paperless-webserver:590/1280MB 0% | mealie:187/1000MB 0% | immich-machine-learning:142/2048MB 0% | felhom-controller:135/4096MB 0% | nextcloud-db:95/512MB 0% | immich-postgres:79/768MB 0% | nextcloud:71/1024MB 0% +17:26:59Z used=3638MB avail=458MB load1=1.75 cpu=5% immich-server:732/3072MB 0% | paperless-webserver:590/1280MB 0% | felhom-controller:216/4096MB 17% | mealie:188/1000MB 0% | immich-machine-learning:142/2048MB 0% | nextcloud-db:95/512MB 0% | immich-postgres:79/768MB 0% | nextcloud:72/1024MB 0% +17:27:30Z used=3638MB avail=458MB load1=1.28 cpu=2% immich-server:732/3072MB 0% | paperless-webserver:590/1280MB 0% | felhom-controller:216/4096MB 17% | mealie:188/1000MB 0% | immich-machine-learning:142/2048MB 0% | nextcloud-db:95/512MB 0% | immich-postgres:79/768MB 0% | nextcloud:72/1024MB 0% +17:28:00Z used=3638MB avail=458MB load1=2.24 cpu=17% immich-server:731/3072MB 1% | paperless-webserver:590/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:173/4096MB 76% | immich-machine-learning:142/2048MB 0% | nextcloud-db:95/512MB 0% | bookstack-db:81/256MB 0% | bookstack:81/512MB 0% +17:28:30Z NO ANSWER TimeoutExpired +17:29:25Z used=NoneMB avail=NoneMB load1=None cpu=0% +17:30:00Z used=NoneMB avail=NoneMB load1=None cpu=0% +17:30:36Z used=NoneMB avail=NoneMB load1=None cpu=0% +17:31:13Z used=1590MB avail=2506MB load1=4.44 cpu=0% mealie:222/1000MB 57% | traefik:186/4096MB 23% | nextcloud:170/1024MB 0% | felhom-controller:167/4096MB 48% | nextcloud-db:166/512MB 0% | actualbudget:159/256MB 0% | paperless-webserver:156/1280MB 63% | filebrowser:145/256MB 0% +17:31:43Z used=3638MB avail=458MB load1=5.72 cpu=76% mealie:222/1000MB 57% | traefik:186/4096MB 23% | nextcloud:170/1024MB 0% | felhom-controller:167/4096MB 48% | nextcloud-db:166/512MB 0% | actualbudget:159/256MB 0% | paperless-webserver:156/1280MB 63% | filebrowser:145/256MB 0% +17:32:13Z used=3638MB avail=458MB load1=3.96 cpu=46% immich-server:896/3072MB 1% | mealie:216/1000MB 0% | immich-machine-learning:163/2048MB 0% | paperless-webserver:154/1280MB 195% | nextcloud-db:121/512MB 0% | felhom-controller:92/4096MB 0% | bookstack-db:91/256MB 0% | immich-postgres:79/768MB 0% +17:32:43Z used=3638MB avail=458MB load1=3.33 cpu=62% immich-server:896/3072MB 1% | mealie:216/1000MB 0% | immich-machine-learning:163/2048MB 0% | paperless-webserver:154/1280MB 195% | nextcloud-db:121/512MB 0% | felhom-controller:92/4096MB 0% | bookstack-db:91/256MB 0% | immich-postgres:79/768MB 0% +17:33:13Z used=3638MB avail=458MB load1=2.15 cpu=8% immich-server:887/3072MB 1% | paperless-webserver:628/1280MB 5% | mealie:216/1000MB 0% | immich-machine-learning:158/2048MB 0% | nextcloud-db:113/512MB 0% | felhom-controller:95/4096MB 1% | bookstack-db:88/256MB 0% | immich-postgres:79/768MB 0% +17:33:43Z NO ANSWER TimeoutExpired +17:34:38Z used=NoneMB avail=NoneMB load1=None cpu=0% +17:35:15Z used=NoneMB avail=NoneMB load1=None cpu=0% +17:35:50Z used=NoneMB avail=NoneMB load1=None cpu=0% +17:36:24Z used=3638MB avail=458MB load1=3.94 cpu=78% traefik:187/4096MB 23% | nextcloud:170/1024MB 0% | actualbudget:160/256MB 15% | mealie:159/1000MB 53% | nextcloud-db:153/512MB 0% | filebrowser:145/256MB 0% | immich-postgres:128/768MB 1% | paperless-webserver:122/1280MB 53% +17:36:54Z used=3638MB avail=458MB load1=3.48 cpu=7% traefik:187/4096MB 23% | nextcloud:170/1024MB 0% | actualbudget:160/256MB 15% | mealie:159/1000MB 53% | nextcloud-db:153/512MB 0% | filebrowser:145/256MB 0% | immich-postgres:128/768MB 1% | paperless-webserver:122/1280MB 53% +17:37:25Z used=3638MB avail=458MB load1=3.33 cpu=60% immich-server:1048/3072MB 0% | mealie:332/1000MB 0% | immich-machine-learning:167/2048MB 0% | paperless-webserver:150/1280MB 100% | nextcloud-db:137/512MB 0% | bookstack:133/512MB 0% | felhom-controller:128/4096MB 5% | nextcloud:116/1024MB 0% +17:37:55Z used=3638MB avail=458MB load1=2.46 cpu=6% immich-server:1048/3072MB 0% | mealie:332/1000MB 0% | immich-machine-learning:167/2048MB 0% | paperless-webserver:150/1280MB 100% | nextcloud-db:137/512MB 0% | bookstack:133/512MB 0% | felhom-controller:128/4096MB 5% | nextcloud:116/1024MB 0% +17:38:25Z used=3638MB avail=458MB load1=2.9 cpu=14% immich-server:914/3072MB 3% | paperless-webserver:541/1280MB 0% | mealie:216/1000MB 0% | immich-machine-learning:154/2048MB 0% | nextcloud-db:97/512MB 0% | bookstack-db:96/256MB 0% | felhom-controller:82/4096MB 24% | immich-postgres:65/768MB 9% +17:38:55Z used=3638MB avail=458MB load1=2.13 cpu=11% immich-server:914/3072MB 3% | paperless-webserver:541/1280MB 0% | mealie:216/1000MB 0% | immich-machine-learning:154/2048MB 0% | nextcloud-db:97/512MB 0% | bookstack-db:96/256MB 0% | felhom-controller:82/4096MB 24% | immich-postgres:65/768MB 9% +17:39:25Z used=3638MB avail=458MB load1=1.37 cpu=30% immich-server:717/3072MB 1% | paperless-webserver:518/1280MB 0% | mealie:216/1000MB 0% | immich-machine-learning:152/2048MB 0% | felhom-controller:135/4096MB 6% | bookstack-db:112/256MB 0% | nextcloud-db:94/512MB 0% | bookstack:91/512MB 0% +17:39:56Z used=3638MB avail=458MB load1=1.41 cpu=24% immich-server:717/3072MB 1% | paperless-webserver:518/1280MB 0% | mealie:216/1000MB 0% | immich-machine-learning:152/2048MB 0% | felhom-controller:135/4096MB 6% | bookstack-db:112/256MB 0% | nextcloud-db:94/512MB 0% | bookstack:91/512MB 0% +17:40:26Z used=3638MB avail=458MB load1=1.82 cpu=33% immich-server:584/3072MB 1% | paperless-webserver:426/1280MB 0% | mealie:172/1000MB 0% | felhom-controller:149/4096MB 5% | immich-machine-learning:128/2048MB 0% | nextcloud:104/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:87/512MB 0% +17:40:56Z used=3638MB avail=458MB load1=1.37 cpu=22% immich-server:584/3072MB 1% | paperless-webserver:426/1280MB 0% | mealie:172/1000MB 0% | felhom-controller:149/4096MB 5% | immich-machine-learning:128/2048MB 0% | nextcloud:104/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:87/512MB 0% +17:41:26Z used=3638MB avail=458MB load1=1.09 cpu=32% immich-server:584/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:250/1000MB 0% | felhom-controller:152/4096MB 17% | immich-machine-learning:129/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% +17:41:56Z used=3638MB avail=458MB load1=0.8 cpu=29% immich-server:584/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:250/1000MB 0% | felhom-controller:152/4096MB 17% | immich-machine-learning:129/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% +17:42:26Z used=3638MB avail=458MB load1=0.84 cpu=29% immich-server:585/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:206/1000MB 0% | felhom-controller:153/4096MB 20% | immich-machine-learning:129/2048MB 0% | nextcloud:119/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% +17:42:56Z used=3638MB avail=458MB load1=0.58 cpu=9% immich-server:585/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:206/1000MB 0% | felhom-controller:153/4096MB 20% | immich-machine-learning:129/2048MB 0% | nextcloud:119/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0% +17:43:27Z used=3638MB avail=458MB load1=0.43 cpu=30% immich-server:585/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:154/4096MB 1% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | book \ No newline at end of file diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/00-expected.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/00-expected.txt new file mode 100644 index 00000000..65728240 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/00-expected.txt @@ -0,0 +1 @@ +Expected (written 17:29 before the act): after the power returns BookStack is either finished, put back exactly (D20), or held with a clear message; never running on a half-restored mix without saying so. diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/01-restore-and-cut.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/01-restore-and-cut.txt new file mode 100644 index 00000000..14335be8 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/01-restore-and-cut.txt @@ -0,0 +1,11 @@ +17:28:15Z reconstitute ('302', {'_raw': ''}) +0.1 {"running": true, "op": "offbox-reconstitute", "stack": "bookstack", "started_at": "2026-10-20T17:28:15.759334434Z", "last": {"op": "offbox-restore", "stack": "bookstack", "ok": true, "message": "A(z) bookstack teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/belso/backups/offsite- +1.1 {"running": true, "op": "offbox-reconstitute", "stack": "bookstack", "started_at": "2026-10-20T17:28:15.759334434Z", "last": {"op": "offbox-restore", "stack": "bookstack", "ok": true, "message": "A(z) bookstack teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/belso/backups/offsite- +2.1 {"running": true, "op": "offbox-reconstitute", "stack": "bookstack", "started_at": "2026-10-20T17:28:15.759334434Z", "last": {"op": "offbox-restore", "stack": "bookstack", "ok": true, "message": "A(z) bookstack teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/belso/backups/offsite- +3.1 {"running": true, "op": "offbox-reconstitute", "stack": "bookstack", "started_at": "2026-10-20T17:28:15.759334434Z", "last": {"op": "offbox-restore", "stack": "bookstack", "ok": true, "message": "A(z) bookstack teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/belso/backups/offsite- +4.2 {"running": true, "op": "offbox-reconstitute", "stack": "bookstack", "started_at": "2026-10-20T17:28:15.759334434Z", "last": {"op": "offbox-restore", "stack": "bookstack", "ok": true, "message": "A(z) bookstack teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/belso/backups/offsite- +5.2 {"running": true, "op": "offbox-reconstitute", "stack": "bookstack", "started_at": "2026-10-20T17:28:15.759334434Z", "last": {"op": "offbox-restore", "stack": "bookstack", "ok": true, "message": "A(z) bookstack teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/belso/backups/offsite- +17:28:21Z PLUG PULLED +17:28:23Z VM off +power on 17:28:57 +health 200 17:31:16 diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/02-after-power-on.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/02-after-power-on.txt new file mode 100644 index 00000000..58ea1817 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/02-after-power-on.txt @@ -0,0 +1,13 @@ +bookstack state: running hold: None [('bookstack', 'Up About a minute (health'), ('bookstack-db', 'Up About a minute (health')] +restore-status: {"running": false, "started_at": "0001-01-01T00:00:00Z", "last": {"op": "offbox-reconstitute", "stack": "bookstack", "ok": false, "message": "A visszaállítás megszakadt (a doboz újraindult) — indítsd el újra.", "finished_at": "2026-10-20T17:31:09.628677552Z", "interrupted": true}, "last_recent": true} +app page: BookStack — Felhom.eu document.documentElement.className=document.documentElement.className.replace('no-js','js'); function csrfHeaders(){var el=document.querySelector('meta[name="csrf-token"]');return el?{'X-CSRF-Token':el.content}:{};} var fhEN=document.documentElement.lang==='en'; function fhNum(v,prec){var s=Number(v).toFixed(prec);return fhEN?s:s.replace('.',',');} function fhIsoDate(iso){return fhEN?iso:iso.replace(/^(\d{4})-(\d{2})-(\d{2})/,'$1. $2. $3.');} function fhDateTime(d){return f +17:31:01 WARN [appstop] crash recovery: an off-site restore (op "offbox-reconstitute:bookstack") was interrupted and left 1 app(s) stopped — restarting them: [bookstack] +17:31:01 INFO [stacks] Starting stack: bookstack +17:31:09 INFO [stacks] Stack bookstack started successfully (took 7.4s) +17:31:09 INFO [appstop] crash recovery: restarted bookstack after the interrupted an off-site restore +17:31:09 WARN [backup] restore of bookstack (offbox-reconstitute, started 2026-10-20T17:28:15Z) was INTERRUPTED by a controller stop — recorded as failed; the household is told to run it again +17:31:10 INFO [report] hub wait channel active (hold ≤240s) +17:31:10 INFO Event pushed: restore_interrupted (warning) [hu-only] — A(z) bookstack visszaállítása megszakadt, mert a doboz újraindult — indítsd el újra. +17:31:12 INFO [stacks] Stack bookstack post-start status: +17:31:12 INFO [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 running Up 3 seconds (health: starting) +17:31:12 INFO [stacks] bookstack-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 16 seconds (healthy) diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/03-bookstack-data-after-cut.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/03-bookstack-data-after-cut.txt new file mode 100644 index 00000000..353bd77c --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/03-bookstack-data-after-cut.txt @@ -0,0 +1,2 @@ +pages: 200 ['Családi wiki 0. oldal', 'Családi wiki 1. oldal', 'Családi wiki 2. oldal', 'Családi wiki 3. oldal', 'Családi wiki 4. oldal', 'Elo oldal a mentes utan'] +images: 200 5 diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/04-log-before-cut.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/04-log-before-cut.txt new file mode 100644 index 00000000..99e01264 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/04-log-before-cut.txt @@ -0,0 +1 @@ +ring from 2026-10-20T17:31:09Z diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/05-second-cut.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/05-second-cut.txt new file mode 100644 index 00000000..fe53ed83 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/05-second-cut.txt @@ -0,0 +1,6 @@ +scratch again ('302', {'_raw': ''}) +scratch: A(z) bookstack teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/belso/backups/offsite-restore/bookstack — a saját fájljaiddal együtt +17:33:24Z reconstitute ('302', {'_raw': ''}) +app state during: [(0.1, 'running'), (1.1, 'running'), (2.1, 'running'), (3.2, 'running'), (4.2, 'running'), (5.2, 'running'), (6.2, 'running'), (7.3, 'running'), (8.3, 'stopped'), (9.3, 'stopped'), (10.3, 'stopped'), (11.4, 'starting'), (12.4, 'starting'), (13.4, 'starting')] +17:33:38Z PLUG PULLED +health 200 17:36:13 diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/06-after-second-cut.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/06-after-second-cut.txt new file mode 100644 index 00000000..2e2c6cbb --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/06-after-second-cut.txt @@ -0,0 +1,99 @@ +bookstack state: running hold: None +restore-status: {"op": "offbox-reconstitute", "stack": "bookstack", "ok": false, "message": "A visszaállítás megszakadt (a doboz újraindult) — indítsd el újra.", "finished_at": "2026-10-20T17:36:10.806850319Z", "interrupted": true} +14:49:19 INFO [web] setup gate actualbudget: the household passed (a dashboard session vouched for this browser) — visitor 192.168.0.158 +15:12:06 INFO [samba] deploy skipped — household SMB password not set yet +15:13:28 INFO [samba] deploy skipped — household SMB password not set yet +15:18:28 INFO [samba] deploy skipped — household SMB password not set yet +15:23:28 INFO [samba] deploy skipped — household SMB password not set yet +15:24:23 INFO [samba] household SMB password applied for user=felhom +16:15:54 INFO [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false) +16:15:56 INFO [report] hub wait channel active (hold ≤240s) +16:28:59 INFO [backup] Discovered 4 database(s): paperless-postgres(postgres), nextcloud-db(mariadb), immich-postgres(postgres), bookstack-db(mariadb) +16:29:03 INFO [backup] DB dump: bookstack-db → bookstack-mariadb.sql (71.9 KB, 393ms, 41 tables) +16:29:07 INFO [backup] Stopping bookstack for safe volume dump +16:29:07 INFO [stacks] Stopping stack: bookstack +16:29:11 INFO [stacks] Stack bookstack stopped successfully (took 4.4s) +16:29:12 INFO [backup] Volume dump: bookstack/bookstack_bookstack_config → 610.5 KB +16:29:13 INFO [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.4 MB +16:29:13 INFO [backup] Restarting bookstack after volume dump +16:29:13 INFO [stacks] Starting stack: bookstack +16:29:19 INFO [stacks] Stack bookstack started successfully (took 6.3s) +16:29:22 INFO [stacks] Stack bookstack post-start status: +16:29:22 INFO [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 running Up 3 seconds (health: starting) +16:29:22 INFO [stacks] bookstack-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy) +16:30:58 INFO [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +16:30:58 INFO [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +16:31:39 INFO [backup] Discovered 4 database(s): paperless-postgres(postgres), nextcloud-db(mariadb), immich-postgres(postgres), bookstack-db(mariadb) +16:31:43 INFO [backup] DB dump: bookstack-db → bookstack-mariadb.sql (71.9 KB, 427ms, 41 tables) +16:31:44 INFO [backup] Stopping bookstack for safe volume dump +16:31:44 INFO [stacks] Stopping stack: bookstack +16:31:49 INFO [stacks] Stack bookstack stopped successfully (took 4.7s) +16:31:49 INFO [backup] Volume dump: bookstack/bookstack_bookstack_config → 110.0 KB +16:31:50 INFO [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.4 MB +16:31:50 INFO [backup] Restarting bookstack after volume dump +16:31:50 INFO [stacks] Starting stack: bookstack +16:31:56 INFO [stacks] Stack bookstack started successfully (took 6.3s) +16:32:00 INFO [stacks] Stack bookstack post-start status: +16:32:00 INFO [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 running Up 3 seconds (health: starting) +16:32:00 INFO [stacks] bookstack-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy) +16:33:32 INFO [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +16:34:21 INFO [offbox] backed up bookstack (/mnt/sys_drive/felhom-data/backups/primary/bookstack, 0 mandatory path(s)) +17:04:13 INFO [backup] Discovered 4 database(s): paperless-postgres(postgres), nextcloud-db(mariadb), immich-postgres(postgres), bookstack-db(mariadb) +17:04:17 INFO [backup] DB dump: bookstack-db → bookstack-mariadb.sql (71.9 KB, 462ms, 41 tables) +17:04:19 INFO [backup] Stopping bookstack for safe volume dump +17:04:19 INFO [stacks] Stopping stack: bookstack +17:04:23 INFO [stacks] Stack bookstack stopped successfully (took 4.3s) +17:04:23 INFO [backup] Volume dump: bookstack/bookstack_bookstack_config → 181.0 KB +17:04:24 INFO [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.4 MB +17:04:24 INFO [backup] Restarting bookstack after volume dump +17:04:24 INFO [stacks] Starting stack: bookstack +17:04:31 INFO [stacks] Stack bookstack started successfully (took 6.2s) +17:04:38 INFO [stacks] Stack bookstack post-start status: +17:04:38 INFO [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 running Up 7 seconds (health: starting) +17:04:38 INFO [stacks] bookstack-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 13 seconds (healthy) +17:05:57 INFO [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +17:06:24 INFO [offbox] backed up bookstack (/mnt/sys_drive/felhom-data/backups/primary/bookstack, 0 mandatory path(s)) +17:16:02 INFO [backup] Tier 2 copied bookstack → /mnt/felhom-drives/hdd_1/backups/secondary/bookstack (153.7 MB, 0 leg(s), 0s) +17:16:02 INFO Event pushed: crossdrive_completed (info) [hu-only] — Másodlagos mentés elkészült: bookstack +17:21:44 INFO [backup] Discovered 4 database(s): nextcloud-db(mariadb), paperless-postgres(postgres), immich-postgres(postgres), bookstack-db(mariadb) +17:21:48 INFO [backup] DB dump: bookstack-db → bookstack-mariadb.sql (71.9 KB, 424ms, 41 tables) +17:21:50 INFO [backup] Stopping bookstack for safe volume dump +17:21:50 INFO [stacks] Stopping stack: bookstack +17:21:54 INFO [stacks] Stack bookstack stopped successfully (took 4.3s) +17:21:54 INFO [backup] Volume dump: bookstack/bookstack_bookstack_config → 152.0 KB +17:21:56 INFO [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.4 MB +17:21:56 INFO [backup] Restarting bookstack after volume dump +17:21:56 INFO [stacks] Starting stack: bookstack +17:22:03 INFO [stacks] Stack bookstack started successfully (took 6.3s) +17:22:09 INFO [stacks] Stack bookstack post-start status: +17:22:09 INFO [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 running Up 6 seconds (health: starting) +17:22:09 INFO [stacks] bookstack-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 11 seconds (healthy) +17:23:32 INFO [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +17:23:45 INFO [offbox] backed up bookstack (/mnt/sys_drive/felhom-data/backups/primary/bookstack, 0 mandatory path(s)) +17:25:52 INFO [shares] household credential restored into the sharing service +17:26:32 INFO [offbox] radicale: scratch holds a UNIT-ONLY restore (snapshot d5bc5a88) — not a full copy, so place-to-live stays closed +17:27:51 INFO [web] off-box full-restore prepared for bookstack (size 153.7 MB) — awaiting the customer's confirm; no restore has started +17:31:01 WARN [appstop] crash recovery: an off-site restore (op "offbox-reconstitute:bookstack") was interrupted and left 1 app(s) stopped — restarting them: [bookstack] +17:31:01 INFO [stacks] Starting stack: bookstack +17:31:09 INFO [stacks] Stack bookstack started successfully (took 7.4s) +17:31:09 INFO [appstop] crash recovery: restarted bookstack after the interrupted an off-site restore +17:31:09 WARN [backup] restore of bookstack (offbox-reconstitute, started 2026-10-20T17:28:15Z) was INTERRUPTED by a controller stop — recorded as failed; the household is told to run it again +17:31:09 INFO [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: true) +17:31:10 INFO [report] hub wait channel active (hold ≤240s) +17:31:10 INFO Event pushed: restore_interrupted (warning) [hu-only] — A(z) bookstack visszaállítása megszakadt, mert a doboz újraindult — indítsd el újra. +17:31:12 INFO [stacks] Stack bookstack post-start status: +17:31:12 INFO [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 running Up 3 seconds (health: starting) +17:31:12 INFO [stacks] bookstack-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 16 seconds (healthy) +17:36:03 WARN [appstop] crash recovery: an off-site restore (op "offbox-reconstitute:bookstack") was interrupted and left 1 app(s) stopped — restarting them: [bookstack] +17:36:03 INFO [stacks] Starting stack: bookstack +17:36:10 INFO [stacks] Stack bookstack started successfully (took 6.9s) +17:36:10 INFO [appstop] crash recovery: restarted bookstack after the interrupted an off-site restore +17:36:10 WARN [backup] restore of bookstack (offbox-reconstitute, started 2026-10-20T17:33:24Z) was INTERRUPTED by a controller stop — recorded as failed; the household is told to run it again +17:36:10 INFO [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: true) +17:36:11 INFO Event pushed: restore_interrupted (warning) [hu-only] — A(z) bookstack visszaállítása megszakadt, mert a doboz újraindult — indítsd el újra. +17:36:11 INFO [report] hub wait channel active (hold ≤240s) +17:36:13 INFO [stacks] Stack bookstack post-start status: +17:36:13 INFO [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 running Up 4 seconds (health: starting) +17:36:13 INFO [stacks] bookstack-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 16 seconds (healthy) +pages: 200 ['Családi wiki 0. oldal', 'Családi wiki 1. oldal', 'Családi wiki 2. oldal', 'Családi wiki 3. oldal', 'Családi wiki 4. oldal'] +images: 200 5 diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/README.md b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/README.md new file mode 100644 index 00000000..b2b92277 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/README.md @@ -0,0 +1,12 @@ +# B6 #2 — power cut in the middle of an off-site restore (2026-10-20) + +Expected (`00-expected.txt`, written first): after the power returns the app is finished, put back exactly (D20), or held +with a clear message — never silently running on a half-restored state. + +| cut | when | after power-on (2 min 20 s to the dashboard) | data | verdict | +|---|---|---|---|---| +| 1 | 6 s into „Teljes visszaállítás" of BookStack (before any data moved) | crash recovery restarted BookStack; restore recorded „A visszaállítás megszakadt (a doboz újraindult) — indítsd el újra." + event `restore_interrupted` | the LIVE data, untouched (6 pages incl. the newer one) | **as expected** | +| 2 | 14 s in: the app had been stopped at 8 s, the database was „starting" for the replay | the SAME recovery: BookStack restarted, the same „megszakadt — indítsd el újra" | **the BACKUP's data** (5 pages): the household's newer page is gone, and nothing says so; no hold; the undo copy written before the change is not used | **NOT as expected** — R-1188 (a) proven live → R-1195 | + +Alarm: `restore_interrupted` pushed (true); the message tells the household to restart, not that its data already +changed. Healed by: running the restore again (17:38, OK in 30 s). diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/README.md b/documentation/audits/rehearsal-2026-10-20/B8-restores/README.md new file mode 100644 index 00000000..5067125e --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/README.md @@ -0,0 +1,23 @@ +# B8 — every customer restore path, once each, on the live rehearsal box (2026-10-20, controller 0.336.0) + +Endpoint-level (the POSTs the backups pages send). Data checked through each app's OWN front door (WebDAV sha256 against +the A5 manifest, the app's API counts). Inventory: `architecture/_recovery-inventory-2026-07-28.md` Part A. + +| path | app | what the household did | page said (D7) | data | app opens | time | +|---|---|---|---|---|---|---| +| A.1.1 the app's own unit | Mealie | a recipe added after the backup, then „Visszaállítás" | „A(z) mealie: 1 adatkötet visszaállítva — az alkalmazás újraindult." — does NOT say which point in time | the 2 backed-up recipes; the newer one gone (rewind, as designed) | yes | 17 s | +| A.1.2 second-drive files | Nextcloud | 3 files deleted by WebDAV, „Fájlok visszaállítása" from the second drive | „nextcloud: 3 fájl visszaállítva a másodlagos másolatból. Az alkalmazás adatbázisa és belső kötetei nem tartoznak ebbe a visszaállításba." | **3/3 sha256 match**, visible in Nextcloud | yes | 17 s | +| A.1.3 off-site → verification copy | Radicale | „ellenőrzés külön mappába" | „…beállításai és adatbázisa visszaállítva ellenőrző mappába: /mnt/felhom-drives/belso/backups/offsite-restore/radicale … A meglévő adatok változatlanok." (a raw path) | not checkable by the household (a folder) | live app untouched | 5 s | +| A.1.4 off-site missing-only into live | Nextcloud | 3 files deleted, full scratch, „hiányzó fájlok" | „A(z) nextcloud hiányzó fájljai helyreállítva az élő adatok közé." — does NOT say how many | **3/3 sha256 match** | yes | 36 s + 10 s | +| A.1.4 + NFC/NFD | share „Filmek" | both forms of „Ödön születésnapja – fénykép lista.txt" (NFC 46 B, NFD 52 B) on the share, captured off-site, both deleted, share restore + place | „Megosztások visszaállítva: 2 fájl … Nem állítható vissza (a mappa nincs élő adatmeghajtón): beolvasas." | **BOTH names come back** (byte-for-byte compare, as the code reads). Nextcloud itself folds NFD into NFC on upload (a PUT of the NFD name answered 204 and overwrote the NFC file) | — | 25 s + 1 s | +| A.1.5 full off-site (files + database) | BookStack | a page added after the backup; full scratch; „Teljes visszaállítás" | „A(z) bookstack: 0 fájl és 2 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20. 19:21) — az alkalmazás újraindult." | 5 pages + 5 images = the backup; the newer page gone | yes | 7 s + 30 s | +| A.1.6 shares | „Filmek" | see the NFC/NFD row | see above; the „beolvasas" share is one the household never made | — | — | — | +| A.1.7 `.fab` export → import | Mealie | export (83.9 KB) → a recipe added → import | the import's six steps all „done" | recipes = the export's | yes | ~1 min | + +**Inventory drift (since July):** three more customer entries exist (the second drive's unit restore and whole restore, the +kept-data Load — C1 reviewer 1); the off-site restore is now one wizard with three choices (verify copy / missing-only / +full) instead of separate paths. + +**Power cut in the middle of A.1.5 (B6 #2):** see `../B6-failures/s2-power-cut-mid-restore/` — the second cut (14 s, after +the old data was replaced) left BookStack RUNNING on the backup's data while the page said „megszakadt — indítsd el +újra"; the newer page was gone; no hold; the undo copy not used. diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a11-mealie-after.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a11-mealie-after.txt new file mode 100644 index 00000000..13e986b2 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a11-mealie-after.txt @@ -0,0 +1 @@ +recipes: ['Csirkepörkölt', 'Gulyásleves'] diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a11-mealie-before.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a11-mealie-before.txt new file mode 100644 index 00000000..3b70a1c2 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a11-mealie-before.txt @@ -0,0 +1,2 @@ +add 201 +recipes: ['Csirkepörkölt', 'Gulyásleves', 'Rakott krumpli (��l��, a ment��s ut��n)'] diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a11-mealie-restore.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a11-mealie-restore.txt new file mode 100644 index 00000000..209c9c59 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a11-mealie-restore.txt @@ -0,0 +1,5 @@ +17:14:49Z points offered: [{"time": "2026-10-20T17:05:05Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}] +17:14:49Z POST /backup/restore helyi -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started'] +17:14:49Z status: {"running": true, "op": "restore", "stack": "mealie", "started_at": "2026-10-20T17:14:49.497712392Z"} +17:15:09Z status: {"running": false, "op": "restore", "stack": "mealie", "started_at": "2026-10-20T17:14:49.497712392Z", "last": {"op": "restore", "stack": "mealie", "ok": true, "message": "A(z) mealie: 1 adatkötet visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T17:15:06.635820001Z"}, "last_recent": true} +17:15:09Z app state: running diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a12-nc-deleted.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a12-nc-deleted.txt new file mode 100644 index 00000000..b7557a9c --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a12-nc-deleted.txt @@ -0,0 +1,3 @@ +GET 404 sha-differs Csalad/Archívum_0/szint00/jegyzet_0_0_0.txt +GET 404 sha-differs Csalad/Archívum_3/szint00/szint01/szint02/szint03/szint04/jegyzet_3_4_7.txt +GET 404 sha-differs Csalad/Kulonleges_nevek/regi_1980.txt diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a12-restore.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a12-restore.txt new file mode 100644 index 00000000..84faf63f --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a12-restore.txt @@ -0,0 +1,6 @@ +17:19:48Z POST /backup/tier2/restore -> HTTP/2 302 ['location: /backups/apps?flash=flash.restore.file_started'] +17:19:48Z {"running": true, "op": "tier2-restore", "stack": "nextcloud", "started_at": "2026-10-20T17:19:48.974904767Z", "last": {"op": "restore", "stack": "mealie", "ok": true, "message": "A(z) mealie: 1 adatkötet visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T17:15:06.635820001Z"}, "last_recent": true} +17:20:09Z {"running": false, "op": "tier2-restore", "stack": "nextcloud", "started_at": "2026-10-20T17:19:48.974904767Z", "last": {"op": "tier2-restore", "stack": "nextcloud", "ok": true, "message": "nextcloud: 3 fájl visszaállítva a másodlagos másolatból. Az alkalmazás adatbázisa és belső kötetei nem tartoznak ebbe a visszaállításba.", "finished_at": "2026-10-20T17:20:05.776010314Z"}, "last_recent": true} +GET 200 SHA-MATCH Csalad/Archívum_0/szint00/jegyzet_0_0_0.txt +GET 200 SHA-MATCH Csalad/Archívum_3/szint00/szint01/szint02/szint03/szint04/jegyzet_3_4_7.txt +GET 200 SHA-MATCH Csalad/Kulonleges_nevek/regi_1980.txt diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a12-tier2-run.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a12-tier2-run.txt new file mode 100644 index 00000000..30572ebe --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a12-tier2-run.txt @@ -0,0 +1,2 @@ +17:15:59Z tier2 run ('200', {'ok': True, 'message': '2. mentés elindítva'}) +20 s {"db_dump": {"count": 4, "duration": "1m56.172129998s", "last_run": "2026-10-20T17:06:09.353210042Z", "success": true}, "enabled": true, "running": false} diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a12-tier2-status.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a12-tier2-status.txt new file mode 100644 index 00000000..86619576 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a12-tier2-status.txt @@ -0,0 +1,8 @@ +/backups/apps | 2. mentés rsync → belso Naponta Utolsó sikeres: 2 perce Sikeres 1,0 GB A Felhom nem tudja ellenőrizni, hogy ez a meghajtó valóban másik lemez-e (virtuális gépben és egye +/backups/apps | 2. mentés rsync → belso Naponta Utolsó sikeres: 2 perce Sikeres 1,2 GB A Felhom nem tudja ellenőrizni, hogy ez a meghajtó valóban másik lemez-e (virtuális gépben és egye +/backups/apps | 2. mentés rsync → belso Naponta Utolsó sikeres: 2 perce Sikeres 72,3 MB A Felhom nem tudja ellenőrizni, hogy ez a meghajtó valóban másik lemez-e (virtuális gépben és egy +/backups/apps | 2. mentés rsync → hdd_1 Naponta Utolsó sikeres: 2 perce Sikeres 1,5 MB A Felhom nem tudja ellenőrizni, hogy ez a meghajtó valóban másik lemez-e (virtuális gépben és egye +/backups/apps | 2. mentés rsync → hdd_1 Naponta Utolsó sikeres: 2 perce Sikeres 34,0 KB A Felhom nem tudja ellenőrizni, hogy ez a meghajtó valóban másik lemez-e (virtuális gépben és egy +/backups/apps | 2. mentés rsync → hdd_1 Naponta Utolsó sikeres: 2 perce Sikeres 391,3 KB A Felhom nem tudja ellenőrizni, hogy ez a meghajtó valóban másik lemez-e (virtuális gépben és eg +/backups/apps | 2. mentés rsync → hdd_1 Naponta Utolsó sikeres: 3 perce Sikeres 153,7 MB A Felhom nem tudja ellenőrizni, hogy ez a meghajtó valóban másik lemez-e (virtuális gépben és eg +/backups/apps | 2. mentés rsync → hdd_1 Naponta Utolsó sikeres: 3 perce Sikeres 524,9 KB A Felhom nem tudja ellenőrizni, hogy ez a meghajtó valóban másik lemez-e (virtuális gépben és eg diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a13-radicale-verify-copy.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a13-radicale-verify-copy.txt new file mode 100644 index 00000000..eedd0225 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a13-radicale-verify-copy.txt @@ -0,0 +1,3 @@ +17:26:21Z offsite unit restore ('302', {'_raw': ''}) +17:26:31Z {"running": false, "op": "offbox-restore", "stack": "radicale", "started_at": "2026-10-20T17:26:21.701643535Z", "last": {"op": "offbox-restore", "stack": "radicale", "ok": true, "message": "A(z) radicale beállításai és adatbázisa visszaállítva ellenőrző mappába: /mnt/felhom-drives/belso/backups/offsite-restore/radicale. A saját fájljaid (dokumentumok, képek, feltöltések) NEM kerültek vissza — ez az ellenőrző visszaállítás csak az alkalmazás beállításait és adatbázisát hozza vissza. Ha a fájljaidra van szükséged, indítsd el a „Teljes visszaállítás előkészítése” lépést ezen az oldalon. A meglévő adatok változatlanok.", "finished_at": "2026-10-20T17:26:26.412981068Z"}, "last_recent": true} +page: Radicale Vaultwarden Pillanatkép: — Válassz alkalmazást — Még nincs mentés felhasználói adattal. A visszaállítás felülírja az alkalmazás jelenlegi adatait a kiválasztott mentés állapotával. Az alkalmazás a folyamat során automatikusan leáll és újraindul. Megértettem, visszaállítás indítása. Visszaállítás indítása Importálás mentett csomagból (.fab) Visszaállítás a távoli tárolóból Válaszd ki az alkalmazást, és a következő oldalon döntsd el, mit szeretnél: ellenőrzést külön mappába, csak a hiányzó fájlok visszahozását, vagy teljes visszaállítást. Egyik sem indul el kérdés nélkül. ActualBudget Visszaállítás… BookStack Visszaállítás… Immich Visszaállítás… Mealie Visszaállítás… Nextcloud Visszaállítás… Paperless-ngx Visszaállítás… Radicale Visszaállítás… Vaultwarden Visszaállítás… Megosztások Megosztások visszaállítása Meglévő ellenőrző másolatok Ezek a visszaállított másolatok helyet foglal diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a14-nc-deleted.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a14-nc-deleted.txt new file mode 100644 index 00000000..475be2c3 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a14-nc-deleted.txt @@ -0,0 +1,3 @@ +DELETE 204 Csalad/Archívum_0/szint00/jegyzet_0_0_0.txt +DELETE 204 Csalad/Archívum_3/szint00/szint01/szint02/szint03/szint04/jegyzet_3_4_7.txt +DELETE 204 Csalad/Kulonleges_nevek/regi_1980.txt diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a14-nextcloud-place.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a14-nextcloud-place.txt new file mode 100644 index 00000000..71cf72a0 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a14-nextcloud-place.txt @@ -0,0 +1,7 @@ +17:39:31Z full scratch 302 +17:40:07Z scratch: A(z) nextcloud teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/hdd_1/backups/offsite-restore/nextcloud — a saját fájljaiddal együtt. A meglévő adatok változatlanok. +17:40:07Z place 302 +10 s {"op": "offbox-place", "stack": "nextcloud", "ok": true, "message": "A(z) nextcloud hiányzó fájljai helyreállítva az élő adatok közé.", "finished_at": "2026-10-20T17:40:09.26526517Z"} +GET 200 SHA-MATCH Csalad/Archívum_0/szint00/jegyzet_0_0_0.txt +GET 200 SHA-MATCH Csalad/Archívum_3/szint00/szint01/szint02/szint03/szint04/jegyzet_3_4_7.txt +GET 200 SHA-MATCH Csalad/Kulonleges_nevek/regi_1980.txt diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a14-nfd-put.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a14-nfd-put.txt new file mode 100644 index 00000000..15c2b7b0 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a14-nfd-put.txt @@ -0,0 +1,2 @@ +PUT NFD 204 + NFC 46 bytes: Ödön születésnapja – fénykép lista.txt diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a14-share-nfd-put.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a14-share-nfd-put.txt new file mode 100644 index 00000000..841e0d00 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a14-share-nfd-put.txt @@ -0,0 +1,6 @@ +NT_STATUS_OBJECT_NAME_COLLISION making remote directory \Nevek +NT_STATUS_OBJECT_NAME_COLLISION making remote directory \Nevek + NFC 1 bytes: . + NFC 2 bytes: .. + NFC 46 bytes: Ödön születésnapja – fénykép lista.txt + NFD 52 bytes: Ödön születésnapja – fénykép lista.txt diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a15-bookstack-before.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a15-bookstack-before.txt new file mode 100644 index 00000000..ba35d3c2 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a15-bookstack-before.txt @@ -0,0 +1,3 @@ +add 200 +pages: 200 ['Családi wiki 0. oldal', 'Családi wiki 1. oldal', 'Családi wiki 2. oldal', 'Családi wiki 3. oldal', 'Családi wiki 4. oldal', 'Elo oldal a mentes utan'] +images: 200 5 diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a15-bookstack-full-download.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a15-bookstack-full-download.txt new file mode 100644 index 00000000..24cddb43 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a15-bookstack-full-download.txt @@ -0,0 +1,3 @@ +17:27:47Z prepare full ('302', {'_raw': ''}) +17:27:51Z confirm full download ('302', {'_raw': ''}) +17:28:01Z {"running": false, "op": "offbox-restore", "stack": "bookstack", "started_at": "2026-10-20T17:27:51.155760714Z", "last": {"op": "offbox-restore", "stack": "bookstack", "ok": true, "message": "A(z) bookstack teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/belso/backups/offsite-restore/bookstack — a saját fájljaiddal együtt. A meglévő adatok változatlanok.", "finished_at": "2026-10-20T17:27:58.223086567Z"}, "last_recent": true} diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a15-bookstack-full-restore.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a15-bookstack-full-restore.txt new file mode 100644 index 00000000..abb60614 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a15-bookstack-full-restore.txt @@ -0,0 +1,4 @@ +17:38:21Z reconstitute 302 +30 s {"op": "offbox-reconstitute", "stack": "bookstack", "ok": true, "message": "A(z) bookstack: 0 fájl és 2 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20. 19:21) — az alkalmazás újraindult.", "finished_at": "2026-10-20T17:38:51.543211508Z"} +pages: 200 ['Családi wiki 0. oldal', 'Családi wiki 1. oldal', 'Családi wiki 2. oldal', 'Családi wiki 3. oldal', 'Családi wiki 4. oldal'] +images: 200 5 diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a16-share-deleted.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a16-share-deleted.txt new file mode 100644 index 00000000..beb9ea8d --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a16-share-deleted.txt @@ -0,0 +1,3 @@ +deleted + NFC 1 bytes: . + NFC 2 bytes: .. diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a16-shares-place.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a16-shares-place.txt new file mode 100644 index 00000000..b07416ec --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a16-shares-place.txt @@ -0,0 +1,10 @@ +17:25:51Z shares place ('302', {'_raw': ''}) +17:26:01Z {"running": false, "op": "shares-place", "stack": "Megosztások", "started_at": "2026-10-20T17:25:51.916294258Z", "last": {"op": "shares-place", "stack": "Megosztások", "ok": true, "message": "Megosztások visszaállítva: 2 fájl, 0 megosztás-beállítás. A(z) Filmek megosztás beállítása már létezik — a meglévő maradt. Nem állítható vissza (a mappa nincs élő adatmeghajtón): beolvasas.", "finished_at": "2026-10-20T17:25:52.098357207Z"}, "last_recent": true} + NFC 1 bytes: . + NFC 2 bytes: .. + NFC 46 bytes: Ödön születésnapja – fénykép lista.txt + NFD 52 bytes: Ödön születésnapja – fénykép lista.txt + Ödön születésnapja – fénykép lista.txt N 56 Tue Oct 20 17:21:31 2026 + Ödön születésnapja – fénykép lista.txt N 56 Tue Oct 20 17:21:31 2026 + Csaladi_film_2 (2026).mkv A 949649288 Tue Oct 20 15:26:40 2026 + Csaladi_film_1 (2026).mkv A 949649129 Tue Oct 20 15:26:29 2026 diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a16-shares-restore.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a16-shares-restore.txt new file mode 100644 index 00000000..7cc0a1bb --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a16-shares-restore.txt @@ -0,0 +1,2 @@ +17:24:28Z shares restore ('302', {'_raw': ''}) +17:25:41Z {"running": false, "op": "shares-restore", "stack": "Megosztások", "started_at": "2026-10-20T17:24:28.837367141Z", "last": {"op": "shares-restore", "stack": "Megosztások", "ok": true, "message": "A megosztások visszaállítása elkészült — most helyreállíthatod az élő adatok közé.", "finished_at": "2026-10-20T17:24:53.261682125Z"}, "last_recent": true} diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a17-after-change.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a17-after-change.txt new file mode 100644 index 00000000..e67c2da4 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a17-after-change.txt @@ -0,0 +1,2 @@ +add 201 +recipes: ['Csirkepörkölt', 'Gulyásleves', 'Lecso'] diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a17-export.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a17-export.txt new file mode 100644 index 00000000..610e4641 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a17-export.txt @@ -0,0 +1,2 @@ +export 200 {'ok': True} +export status: {"ok": true, "error": "", "output_path": "/mnt/felhom-drives/hdd_1/exports/mealie_20261020-174037.fab", "output_size": "83.9 KB"} diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/a17-import.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/a17-import.txt new file mode 100644 index 00000000..efa9b6e3 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/a17-import.txt @@ -0,0 +1,3 @@ +import 200 {'ok': True} +import status: {"ok": true, "error": "", "steps": [{"label": "Csomag megnyitása", "status": "done"}, {"label": "Alkalmazás előkészítése", "status": "done"}, {"label": "Konfiguráció visszaállítása", "status": "done"}, {"label": "Felhasználói adatok visszaállítása", "status": "done"}, {"label": "Adatbázis visszaállítása", "status": "done"}, {"label": "Alkalmazás indítása", "status": "done"}]} +recipes: ['Csirkepörkölt', 'Gulyásleves'] diff --git a/documentation/audits/rehearsal-2026-10-20/B8-restores/offsite-run-3.txt b/documentation/audits/rehearsal-2026-10-20/B8-restores/offsite-run-3.txt new file mode 100644 index 00000000..05ae31c9 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B8-restores/offsite-run-3.txt @@ -0,0 +1,2 @@ +17:21:44Z press 302 +17:24:14Z ok snaps= 26 size= 1.1 GB dur= 2m23s err= diff --git a/documentation/audits/rehearsal-2026-10-20/journal.md b/documentation/audits/rehearsal-2026-10-20/journal.md index eb5060bc..c4e6d7cd 100644 --- a/documentation/audits/rehearsal-2026-10-20/journal.md +++ b/documentation/audits/rehearsal-2026-10-20/journal.md @@ -59,3 +59,7 @@ | 16:28:59 | B4 | „Mentés most" (guide §9): 2 min (the guide says ~20 s for two apps), 4 database dumps, success | | 16:31:28 | B6 #5 | `tc netem` on the VM's own tap (demo-hp): upload 1 Mbit/s + 5 % loss both ways — harness note: this slows the LAN answers too, which a real slow internet line would not | | 16:31:37 | B6 #5 | „Távoli mentés most" pressed (all 8 apps are ON for off-site by default; the share „Filmek" too) → status running, phase dump. The page also shows the household an SSH-key / known_hosts form („no blind TOFU", „ssh-keyscan") | +| 17:01–17:08 | B6 #5 | see `B6-failures/s5-slow-net/README.md`: run failed (Immich) under 1 Mbit/s + 5 % loss; the second press at full speed ok in 3 min 38 s | +| 17:01 | mail | the operator mailbox (second channel) shows host_stale 13:33, app_oom 13:34 (immich-machine-learning), host_crash_restart ×2, node_stale/recovered, backup_failed 17:01; the household got „A biztonsági mentés sikertelen!" — R-1193 corrected; oob_degraded at 12:03 on the fresh box → R-1194 | +| 17:14–17:42 | B8 | all seven restore paths, see `B8-restores/README.md`; NFC/NFD: both names come back on the share; Nextcloud folds NFD into NFC itself | +| 17:28–17:36 | B6 #2 | power cut mid full off-site restore, twice: at 6 s safe; at 14 s the app runs on the backup's data while the page says „megszakadt — indítsd el újra" → R-1195 | diff --git a/documentation/audits/rehearsal-2026-10-20/tools/b8.py b/documentation/audits/rehearsal-2026-10-20/tools/b8.py new file mode 100644 index 00000000..c296e1b7 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/tools/b8.py @@ -0,0 +1,30 @@ +#!/usr/bin/env python3 +"""b8.py — the household's restore buttons, endpoint-level (the same POSTs the backups pages send), B8. + b8.py points STACK — the restore points the page offers + b8.py unit STACK [SNAPSHOT_ID] — „Visszaállítás" (POST /backup/restore), then the status until done +Env: CTL_BASE, CTL_DOMAIN, CTL_PW_ENV.""" +import json, os, subprocess, sys, time +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))); import ctl +def say(*a): print(time.strftime("%H:%M:%SZ", time.gmtime()), *a, flush=True) +def points(stack): + c, d = ctl.call("GET", f"/api/backup/snapshots?stack={stack}", timeout=90) + data = d.get("data") if isinstance(d, dict) else None + return data if isinstance(data, list) else next((data[k] for k in ("snapshots", "items", "restore_points") if isinstance(data, dict) and isinstance(data.get(k), list)), []) +def unit(stack, sid=None): + pts = points(stack); say("points offered:", json.dumps(pts, ensure_ascii=False)[:500]) + sid = sid or (pts[0].get("short_id") or pts[0].get("id")) + if ctl._SESS is None: ctl.login() + out = subprocess.run(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", ctl.HOST, "-H", "Cookie: " + ctl._SESS, "-X", "POST", + "--data-urlencode", "_csrf=" + ctl._CSRF, "--data-urlencode", "stack_name=" + stack, + "--data-urlencode", "snapshot_id=" + sid, ctl.BASE + "/backup/restore"], capture_output=True, text=True).stdout + say("POST /backup/restore", sid, "->", out.splitlines()[0] if out else "?", [l for l in out.splitlines() if l.lower().startswith("location")][:1]) + t0, last = time.time(), None + while time.time() - t0 < 1800: + c, d = ctl.call("GET", "/api/backup/restore-status", timeout=90); dd = (d.get("data") or {}) if isinstance(d, dict) else {} + s = json.dumps(dd, ensure_ascii=False)[:700] + if s != last: say("status:", s); last = s + if not dd.get("running") and time.time() - t0 > 8: break + time.sleep(5) + c, d = ctl.call("GET", f"/api/stacks/{stack}"); say("app state:", (d.get("data") or {}).get("state")) +if __name__ == "__main__": + {"points": lambda *a: print(json.dumps(points(*a), ensure_ascii=False, indent=1)), "unit": unit}[sys.argv[1]](*sys.argv[2:]) diff --git a/documentation/audits/rehearsal-2026-10-20/tools/bs_cnt.py b/documentation/audits/rehearsal-2026-10-20/tools/bs_cnt.py new file mode 100644 index 00000000..fb5a8c90 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/tools/bs_cnt.py @@ -0,0 +1,11 @@ +#!/usr/bin/python3 +"""bs_cnt.py [add NAME] — BookStack through its API: optionally add a page, then list pages and images (B8 A.1.5).""" +import json, os, sys +sys.path.insert(0, "/root/work"); from apps import http +b = "https://wiki.tester4.test"; tid, tsec = os.environ["TOK"].split(":", 1); H = {"Authorization": f"Token {tid}:{tsec}"} +if len(sys.argv) > 2 and sys.argv[1] == "add": + print("add", http("POST", b + "/api/pages", {"book_id": 1, "name": sys.argv[2], "markdown": "élő oldal a mentés után"}, H)[0]) +st, out, _ = http("GET", b + "/api/pages?count=100", None, H) +print("pages:", st, sorted(x["name"] for x in json.loads(out).get("data", [])) if st == 200 else out[:200]) +st, out, _ = http("GET", b + "/api/image-gallery?count=100", None, H) +print("images:", st, len(json.loads(out).get("data", [])) if st == 200 else out[:200]) diff --git a/documentation/audits/rehearsal-2026-10-20/tools/mealie_cnt.py b/documentation/audits/rehearsal-2026-10-20/tools/mealie_cnt.py new file mode 100644 index 00000000..6a5457cc --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/tools/mealie_cnt.py @@ -0,0 +1,11 @@ +#!/usr/bin/python3 +"""mealie_cnt.py [add NAME] — log in to Mealie, optionally add a recipe, print every recipe's name (B8 A.1.1).""" +import json, os, sys, urllib.parse +sys.path.insert(0, "/root/work"); from apps import http +b = "https://recipes.tester4.test" +st, out, _ = http("POST", b + "/api/auth/token", urllib.parse.urlencode({"username": "changeme@example.com", "password": os.environ["PW"]}).encode(), {"Content-Type": "application/x-www-form-urlencoded"}) +H = {"Authorization": "Bearer " + json.loads(out)["access_token"]} +if len(sys.argv) > 2 and sys.argv[1] == "add": + print("add", http("POST", b + "/api/recipes", {"name": sys.argv[2]}, H)[0]) +st, out, _ = http("GET", b + "/api/recipes?perPage=100", None, H) +print("recipes:", sorted(x["name"] for x in json.loads(out)["items"])) diff --git a/documentation/audits/rehearsal-2026-10-20/tools/nc_files.sh b/documentation/audits/rehearsal-2026-10-20/tools/nc_files.sh new file mode 100755 index 00000000..73ad629c --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/tools/nc_files.sh @@ -0,0 +1,11 @@ +#!/bin/bash +# nc_files.sh del|check — delete (or check by sha256 through WebDAV) three of Ödön's Nextcloud files (B8 A.1.2/A.1.4) +B=https://cloud.tester4.test/remote.php/dav/files/odon +F=("Csalad/Archívum_0/szint00/jegyzet_0_0_0.txt" "Csalad/Archívum_3/szint00/szint01/szint02/szint03/szint04/jegyzet_3_4_7.txt" "Csalad/Kulonleges_nevek/regi_1980.txt") +enc() { python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$1"; } +for f in "${F[@]}"; do + u="$B/$(enc "$f")" + if [ "$1" = del ]; then echo "DELETE $(curl -sk -u "odon:$PW" -X DELETE -o /dev/null -w %{http_code} "$u") $f" + else want=$(grep -F "nextcloud/$f " /root/data/MANIFEST.tsv | cut -f3); got=$(curl -sk -u "odon:$PW" "$u" -o - -w "" | sha256sum | cut -d' ' -f1) + code=$(curl -sk -u "odon:$PW" -o /dev/null -w %{http_code} "$u"); echo "GET $code $( [ "$want" = "$got" ] && echo SHA-MATCH || echo sha-differs ) $f"; fi +done diff --git a/documentation/audits/rehearsal-2026-10-20/tools/nc_nfd.py b/documentation/audits/rehearsal-2026-10-20/tools/nc_nfd.py new file mode 100644 index 00000000..f2cf0af1 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/tools/nc_nfd.py @@ -0,0 +1,19 @@ +#!/usr/bin/python3 +"""nc_nfd.py put|list — put the NFD-named twin of „Ödön születésnapja – fénykép lista.txt" into Nextcloud by WebDAV +(as a Mac client would send it), then list the folder by PROPFIND and show each name's form (NFC/NFD) and bytes.""" +import base64, os, sys, unicodedata, urllib.parse +sys.path.insert(0, "/root/work"); from apps import http +U = "odon"; H = {"Authorization": "Basic " + base64.b64encode(f"{U}:{os.environ['PW']}".encode()).decode()} +d = "/root/data/nextcloud/Csalad/Kulonleges_nevek" +B = f"https://cloud.tester4.test/remote.php/dav/files/{U}/Csalad/Kulonleges_nevek/" +nfd = unicodedata.normalize("NFD", "Ödön születésnapja – fénykép lista.txt") +if sys.argv[1] == "put": + body = open(os.path.join(d, nfd), "rb").read() + print("PUT NFD", http("PUT", B + urllib.parse.quote(nfd), body, H)[0]) +st, out, _ = http("PROPFIND", B, None, dict(H, Depth="1")) +import re +for href in re.findall(rb"([^<]+)", out): + name = urllib.parse.unquote(href.decode().rstrip("/").split("/")[-1]) + if "rk" in name or "dön" in unicodedata.normalize("NFC", name): + form = "NFC" if name == unicodedata.normalize("NFC", name) else "NFD" if name == unicodedata.normalize("NFD", name) else "mixed" + print(f" {form} {len(name.encode())} bytes: {name}") diff --git a/documentation/audits/rehearsal-2026-10-20/tools/share_nfd.sh b/documentation/audits/rehearsal-2026-10-20/tools/share_nfd.sh new file mode 100755 index 00000000..70af591e --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/tools/share_nfd.sh @@ -0,0 +1,15 @@ +#!/bin/bash +# share_nfd.sh put|list|del — the NFC and the NFD form of one name on the „Filmek" share (SMB), as a Windows PC and a Mac would send them +cd /root/data/nextcloud/Csalad/Kulonleges_nevek || exit 1 +smb() { smbclient //felhom.tester4.test/Filmek -U felhom --password="$PW" -m SMB3 -c "$1" 2>&1 | grep -v "^Try\|^$"; } +case $1 in +put) for f in *lista.txt; do smb "mkdir Nevek; cd Nevek; put \"$f\" \"$f\"" | tail -1; done ;; +del) for f in *lista.txt; do smb "cd Nevek; del \"$f\""; done; echo deleted ;; +esac +smb "cd Nevek; ls" | grep -v "blocks of size" | python3 -c " +import sys,unicodedata +for l in sys.stdin: + n=l.strip().rsplit(' ',1)[0].strip() + if n in ('.','..') or not n: continue + nm=n.split(' ')[0].strip(); f='NFC' if nm==unicodedata.normalize('NFC',nm) else 'NFD' + print(' ',f,len(nm.encode()),'bytes:',nm)" diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 4efa8417..79fc9231 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -151,7 +151,7 @@ stopping line that lies. | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC | | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC | -## Backup & restore — 41 rows (P1 2, P2 9, P3 18, P4 12) +## Backup & restore — 42 rows (P1 2, P2 10, P3 18, P4 12) | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| @@ -166,6 +166,7 @@ stopping line that lies. | **R-1180** | Backup & restore | P2 | **The nightly second-drive mirror (`rsync --delete`) can run DURING a restore from that same mirror and delete files not yet copied back.** `RunAllTier2` (`tier2.go:479-520`) takes no backup flag and does not ask about a running restore; the whole restore holds only the backup flag (`tier2_whole.go:301`), the night leg only `LegLock` (`cmd/controller/main.go:1348-1352`). A long restore of a photo library at 03:30, or 15 min after a missed night, overlaps. PROVEN against production code with the real `rsync -a --delete` mirror (`audits/dataloss-hunt-2026-10-20/C3-T1-mirror-during-restore/`): with the backup flag held by a restore, `RunAllTier2` still mirrors, and the mirror goes from [a,b,c,d] to [a,b] — the two files the restore was bringing back are gone from the only copy. Data-loss hunt 2026-10-20 (C1 reviewer 4, T1). | OPEN | — | Fix: the mirror leg skips (and says so) while any restore runs; test with the existing tier-2 seams | CC | | **R-1181** | Backup & restore | P2 | **A missing or emptied app folder deletes its second-drive copy within one night; for OPTIONAL folders (audiobooks, comics, ROMs, Immich's external library) that copy was the only one.** A missing declared leg is skipped (`tier2_capture.go:55-61`, silent for optional legs) and then reconciled away (`tier2.go:454`); an emptied one is mirrored with `rsync -a --delete` (`:744-751`). PROVEN for a missing folder against production code (`audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/`): the mirror's `photos` deleted, run „ok", warning only „nem kerültek a másodlagos mentésbe". 07 §8.2 chose „data legs are not guarded" for a leg that LEAVES the class set; a declared leg that is MISSING is not that case. Optional legs are in no off-site tier (07 §6.2). | WAITING-ON-OPERATOR | decision: keep a missing leg's mirror (and say so) until the household releases it? And: does an accidental folder deletion need a second-drive history for optional legs? | Operator: answer in STATUS | operator | | **R-1182** | Backup & restore | P2 | **Abandoning ONE set-aside off-site copy deletes EVERY retained key package of the customer, including ones that open other set-aside copies still stored.** hub `api/handler.go:626-627` → `PurgeSupersededEscrowForCustomer` (`store.go:4185-4190`) deletes customer-wide; the comment right above (`:4176-4179`) states the invariant this breaks. CONFIRMED in code (the SQL deletes every retained row of the customer, live and deleted hosts alike; nothing ties a row to the abandoned path). The R-1176 reinstall makes a second set-aside copy likely. Data-loss hunt 2026-10-20 (C1 reviewer 3, K2). | OPEN | — | Fix: purge only the package whose key opens the abandoned path (record the path or repo id with each retained row); hub store test as red-proof | CC | +| **R-1195** | Backup & restore | P2 | **A power cut in the middle of a full off-site restore can leave the app running on the BACKUP's data while the page says only „the restore was interrupted, start it again" — the household's newer data is gone and nothing says so.** PROVEN LIVE in the rehearsal 2026-10-20 (`audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/`): BookStack, a page added after the backup; „Teljes visszaállítás"; power cut at 14 s (the app stopped at 8 s, the database starting for the replay) → after power-on the crash recovery restarted the app (`appstop_marker.go`), recorded „A visszaállítás megszakadt (a doboz újraindult) — indítsd el újra." and pushed `restore_interrupted`; BookStack ran on the backup's 5 pages, the newer page gone; no hold; the undo copy written before the change (`offbox_restore_undo.go`) was not used. A cut at 6 s (before data moved) behaved correctly. This is R-1188 (a) proven; R-1188 keeps its other items. | OPEN | — | On start after an interrupted restore: put the undo copy back (D20's „exactly as it was") or hold the app and say „a visszaállítás félbeszakadt — az adatok a mentés állapotában lehetnek"; a test that kills the controller between the volume step and the replay | CC | | **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** **READY 2026-10-20 — measured and planned, nothing changed; the decision is the operator's (STATUS D21).** Re-measured on 9202 (Immich v3.2.4): off-site ~1.33 GB = model cache 824 MB (62 %), raw database folder 283 MB (21 %), Immich's own backups 115 MB (9 %), logical dump 54 MB, originals 49 MB, thumbnails 2 MB — 102 MB must survive. A: leave out all three re-fetchable parts (−92 %; needs a never-run restore from the dump alone on the vectorchord image). B: only the model cache (−62 %; needs one restore with an empty cache). Pick: B now, A's other parts after their proof (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`). | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC | | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC | | **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC | @@ -193,7 +194,7 @@ stopping line that lies. | **R-1185** | Backup & restore | P3 | **A database dump that fails its own content check has already replaced last night's good dump, and the second drive copies it an hour later.** `appbackup/dbdump.go:388-422`: size check, rename over the good file (`:399`), then `ValidateDump` (`:419`). Volume tars are promoted with no content check either (`backup.go:902`). Tier 1 is one point per app by design (07 §6.1); the ORDER is the defect. PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/`, a docker stub prints a header-only dump): `ValidateDump` says invalid („dump file too small"), `DumpOneTo` returns no error, and last night's good dump (with its `CREATE TABLE`) is gone from disk. Data-loss hunt 2026-10-20 (D5). | OPEN | — | Fix: validate the temp file before the rename; keep the last good dump on a failed check | CC | | **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC | | **R-1187** | Backup & restore | P3 | **`.fab` export/import candidates from the data-loss hunt, not yet proven:** (a) an export silently drops a drive folder that holds a symlink (`appexport/fabplan.go:292`, `export.go:714`); (b) the import never sets file owners and drops symlinks (`restore.go:1157-1176`); (c) import into a STOPPED app merges into the existing volumes, and the default export (stop ticked) carries no SQL dump (`restore.go:361-374`, `export.go:262,366`, `appbackup/dbdump.go:142`); (d) no concurrency guard, no undo, no record (`restore.go:181-186`, `web/handler_export.go:312-353`); (e) PROVEN and split out as R-1191 (the import drops the family gate); (f) it trusts the bundle's HDD_PATH (`restore.go:589-631`); (g) the DB import runs psql without ON_ERROR_STOP and misses postgis (`restore.go:1052-1053, 928-937`). Read by reviewer 1; main session spot-checked (a); (e) proven live → R-1191. Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Prove (c) first on 9202 with a throwaway app; split into rows only what reproduces | CC | -| **R-1188** | Backup & restore | P3 | **Restore-path candidates from the data-loss hunt, not yet proven (one confirmed in code):** (a) a crash or power cut mid-restore restarts the app on a mixed state and never reads the undo copy (`appstop_marker.go:307`, `offbox_restore_undo.go:137`, `bootrecon/bootrecon.go:210-213`); (b) a failed `volume rm` is ignored before the untar (`restore.go:218`); (c) PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-R12-leftover-temp/`): an interrupted second-drive whole restore leaves `.felhom-restoring`, and EVERY retry fails „fájlmásolás sikertelen: … file exists" with the file never back (`tier2_whole.go:86-87`, O_EXCL); (d) the full off-site download has a 30-min cap per press and the household is not told to retry (`web/offbox_handlers.go:455`); (e) place and full restore re-read the LATEST snapshot instead of the downloaded one (`offbox_restore.go:675-679`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Fix (c) now (remove a stale temp of the same name before the copy); prove (a) with the B6 power-cut-mid-restore scenario | CC | +| **R-1188** | Backup & restore | P3 | **Restore-path candidates from the data-loss hunt, not yet proven (one confirmed in code):** (a) PROVEN LIVE → R-1195 (a power cut mid-restore restarts the app on the backup's data; the undo copy is not used); (b) a failed `volume rm` is ignored before the untar (`restore.go:218`); (c) PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-R12-leftover-temp/`): an interrupted second-drive whole restore leaves `.felhom-restoring`, and EVERY retry fails „fájlmásolás sikertelen: … file exists" with the file never back (`tier2_whole.go:86-87`, O_EXCL); (d) the full off-site download has a 30-min cap per press and the household is not told to retry (`web/offbox_handlers.go:455`); (e) place and full restore re-read the LATEST snapshot instead of the downloaded one (`offbox_restore.go:675-679`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Fix (c) now (remove a stale temp of the same name before the copy); prove (a) with the B6 power-cut-mid-restore scenario | CC | | **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt, not yet proven:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it, and the hub reads the same clock (agent `localapi/server.go:1178-1192`, `backup/runner.go:607-629`, hub `monitor/deadline.go:243`; R-973 cap counts future runs `backup_cap.go:61`); (b) the whole-guest start creeps ~5 min/night until a night is skipped silently (`server.go:1184`, `quiesce.go:150, 807-818`); (c) the whole-guest quiesce stops apps mid-restore or mid-update (`quiesce.go:213-275, 565`); (d) CONFIRMED in code: the off-site proof and integrity check are plain daily jobs outside the catch-up (`nightchain.go` header: the catch-up runs only the three backup legs), so they never run on a box that is off at 05:30–06:00, while the comment at `cmd/controller/main.go:1575-1577` says the opposite; (e) CONFIRMED in code: the hub's 05:00 check wants a dump „since Budapest midnight" (`hub/internal/monitor/deadline.go:416-419, 556-565`), so an evening backup time (which the banner suggests) raises a false `expected_dbdump_missed` error every morning — and on 2026-10-25 the check also runs at 04:00 (R-1183); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — design question; (g) spring forward 2027-03-28 fires dump and second copy together; (h) self-update busy check before the pull only (`updater.go:532-545, 764`); (i) a double-clicked restore shows failed while it runs (`web/handlers.go:1853-1874`). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`. | VERIFY | — | (a) is B6 #8's live test; (b) read the agent journal on demo-hp; the rest by unit tests | CC | | **R-1190** | Backup & restore | P3 | **Key and retention candidates from the data-loss hunt, not yet proven:** (a) a recovery ceremony while off-site is not configured seals a package with no repo password and supersedes the good one (`escrow_handlers.go:217`, agent `main.go:2975-2981`, hub `store.go:3392`); (b) the global „create your recovery code" bar links to the ceremony the backups page hides during a recovery offer (`escrow_banner.go:23-33` vs `backups_remote.html:144-160`); (c) after a reuse-path reinstall the new PBS key K is never escrowed and nothing compares fingerprints (`felhom-pbs-apply:99`, `escrow_confirm.go:182-205`); (d) whole-guest off-site copies run before any escrow exists (design question vs 07 §6.1's Tier-3 gate); (e) the escrow PUT is two statements; (f) the operator's off-site re-issue resets the provider password before saving it (hub `offsite.go:217-245`); (g) „Megőrzött adatok" can list a running app's live folder when its app.yaml is unreadable (`kept.go:220-235, 330-402`); (h) a unit restore can point an app back at an old drive and F5 then prunes the current drive's unit (`backup.go:1517-1574`). Evidence `audits/dataloss-hunt-2026-10-20/C1-keys-candidates.md`, `C1-delete-retention-candidates.md`. | VERIFY | — | Unit tests first for (a), (e), (g) | CC |