diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/01-run.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/01-run.txt new file mode 100644 index 00000000..60d05a40 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/01-run.txt @@ -0,0 +1,18 @@ +ntp off 07:57:38 +plus3d=2026-10-24T07:57:38 (box off: clean stop now, start with the clock 3 days on) +down 07:59:33 +update VM 9320: -startdate 2026-10-24T07:57:38 +started with RTC 2026-10-24T07:57:38 at real 07:59:36 +real 2026-10-21T08:00:42Z box Sat, 24 Oct 2026 07:58:45 GMT +correction 08:00:54: the box's newest backup records are dated 2026-10-25 01:31Z (left by #7), so 2026-10-24 is not '3 days off'. Restarting at 2026-10-28T06:00:00Z = ~3 days after those records. +down 08:02:49 +update VM 9320: -startdate 2026-10-28T06:00:00 +started with RTC 2026-10-28T06:00:00 at real 08:02:53 +real 2026-10-21T08:04:00Z box Wed, 28 Oct 2026 06:01:07 GMT +real 2026-10-21T08:04:20Z box Wed, 28 Oct 2026 06:01:28 GMT +real 2026-10-21T08:18:44Z box Wed, 28 Oct 2026 06:15:51 GMT +#15 restart mid-catch-up at real 08:18:44, box Wed, 28 Oct 2026 06:15:51 GMT → RTC 2026-10-28T06:16:51 +down 08:20:39 +update VM 9320: -startdate 2026-10-28T06:16:51 +started with RTC 2026-10-28T06:16:51 at real 08:20:42 +real 2026-10-21T08:21:25Z box Wed, 28 Oct 2026 06:17:34 GMT diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/02-catchup-start.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/02-catchup-start.txt new file mode 100644 index 00000000..b07243e2 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/02-catchup-start.txt @@ -0,0 +1,2 @@ +2026-10-28T06:00:37Z [quiesce] loop started (poll 5m0s, max-quiesce 30m0s) +2026-10-28T06:00:37Z [catch-up] controller start: the box missed [db-dump tier2 offsite] (last scheduled 2026-10-28 04:15) — ONE catch-up in 15m0s (backup legs only; app updates wait for a real night) diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/03-household-view.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/03-household-view.txt new file mode 100644 index 00000000..71f26208 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/03-household-view.txt @@ -0,0 +1,2 @@ +1081 + Indítópult — Felhom.eu Indítópult Vezérlőpult Alkalmazások Tárhely Meghajtók Hálózati tárhely Biztonsági mentés Áttekintés Távoli mentés Alkalmazások Visszaállítás Megosztás Hálózati megosztás Rendszermonitor Hibakeresés Beállítások Rendszer Értesítések Biztonság és hozzáférés 0.336.0 Magyar English Kijelentkezés ↗ A legutóbbi mentés 3 nappal ezelőtt készült. A doboz ki volt kapcsolva a mentés idején (02:30). Válassz egy olyan időpontot, amikor a doboz általában be van kapcsolva. Mentési idő módosítása Bezárás Egy védett rendszerszolgáltatás nem fut: felhom-samba Rendszermonitor → A processzor terhelése magas: 97% Rendszermonitor → Indítópult tester4.test Indítópult megosztása A ActualBudget B BookStack F Filebrowser I Immich M Mealie N Nextcloud P Paperless-ngx R Radicale V Vaultwarden Indítópult megosztása Egy megosztható link, amely csak megmutatja az alkalmazásokat és új lapon megny diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/04-catchup-running.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/04-catchup-running.txt new file mode 100644 index 00000000..dd7012a1 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/04-catchup-running.txt @@ -0,0 +1,7 @@ +2026-10-28T06:15:37Z [catch-up] running the missed db-dump leg +2026-10-28T06:15:37Z [catch-up] running the missed db-dump leg +2026-10-28T06:15:41Z [backup] Stopping actualbudget for safe volume dump +2026-10-28T06:15:41Z [stacks] Stopping stack: actualbudget +2026-10-28T06:15:42Z [backup] Volume dump: actualbudget/actualbudget_actualbudget_data → 508.5 KB +2026-10-28T06:15:43Z [backup] Stopping bookstack for safe volume dump +2026-10-28T06:15:43Z [stacks] Stopping stack: bookstack diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/05-after-restart.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/05-after-restart.txt new file mode 100644 index 00000000..e544684b --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/05-after-restart.txt @@ -0,0 +1,13 @@ +2026-10-28T06:17:27Z [appstop] crash recovery: an app-data backup (volume dump) (op "volume-dump:paperless-ngx") was interrupted and left 1 app(s) stopped — restarting them: [paperless-ngx] +2026-10-28T06:17:27Z [appstop] refusing to restart "paperless-ngx" after an interrupted operation: drive /mnt/felhom-drives/hdd_1 is not a live mountpoint +2026-10-28T06:17:27Z [appstop] crash recovery: NOT restarting paperless-ngx — start refused by a deliberate holder: drive /mnt/felhom-drives/hdd_1 is not a live mountpoint; the marker is KEPT and the holder owns the resta +2026-10-28T06:17:27Z [appstop] crash recovery: 1 app(s) were deliberately NOT restarted (drive absent) — KEEPING the marker; this is the gate working, not a fault: [paperless-ngx] +2026-10-28T06:17:27Z [backup] restore record wired: /opt/docker/felhom-controller/data/restore-status.json (interrupted at startup: false) +2026-10-28T06:17:27Z [appstop] an app-data backup (volume dump) was interrupted by a controller restart — 1 app(s) are left stopped and HELD: their data drive is not available, so the drive gate restarts them when it retu +2026-10-28T06:17:27Z [catch-up] controller start: the box missed [db-dump tier2 offsite] (last scheduled 2026-10-28 04:15) — ONE catch-up in 15m0s (backup legs only; app updates wait for a real night) +{'actualbudget': 'starting', 'bookstack': 'running', 'immich': 'stopped', 'mealie': 'running', 'nextcloud': 'degraded', 'paperless-ngx': 'degraded', 'radicale': 'running', 'vaultwarden': 'running'} +08:23:33Z {'actualbudget': 'running', 'bookstack': 'running', 'immich': 'running', 'mealie': 'running', 'nextcloud': 'running', 'paperless-ngx': 'running', 'radicale': 'running', 'vaultwarden': 'running'} +2026-10-28T06:17:52Z [bootrecon] "paperless-ngx" is a boot orphan by intent but is HELD (an app-data operation is holding it — the app-stop guard restarts it when the operation ends) — NOT starting it; +2026-10-28T06:17:52Z [stacks] Starting stack: immich +2026-10-28T06:18:06Z [stacks] Starting stack: immich +2026-10-28T06:18:37Z [stacks] Starting stack: paperless-ngx diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/06-catchup-done.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/06-catchup-done.txt new file mode 100644 index 00000000..f4bdb9a3 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/06-catchup-done.txt @@ -0,0 +1,12 @@ +2026-10-28T06:37:18Z [backup] Tier 2 run complete: 8 app(s) processed (incl. volume-only — F6) +2026-10-28T06:37:19Z [catch-up] running the missed offsite leg +2026-10-28T06:37:28Z [stacks] execCommand failed: exit status 1 +TIMEOUT +2026-10-28T06:39:30Z [offbox] backed up paperless-ngx (/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx, 1 mandatory path(s)) +2026-10-28T06:39:32Z [offbox] backed up radicale (/mnt/sys_drive/felhom-data/backups/primary/radicale, 0 mandatory path(s)) +2026-10-28T06:39:34Z [offbox] backed up vaultwarden (/mnt/sys_drive/felhom-data/backups/primary/vaultwarden, 0 mandatory path(s)) +2026-10-28T06:39:36Z [offbox] backed up actualbudget (/mnt/sys_drive/felhom-data/backups/primary/actualbudget, 0 mandatory path(s)) +2026-10-28T06:39:39Z [offbox] backed up bookstack (/mnt/sys_drive/felhom-data/backups/primary/bookstack, 0 mandatory path(s)) +2026-10-28T06:39:46Z [offbox] backed up immich (/mnt/felhom-drives/hdd_1/backups/primary/immich, 1 mandatory path(s)) +2026-10-28T06:39:48Z [offbox] backed up mealie (/mnt/sys_drive/felhom-data/backups/primary/mealie, 0 mandatory path(s)) +2026-10-28T06:39:52Z [offbox] retention skipped (after-run): no clean-up window now (not due (last window 2026-10-20T17:07:48Z)) — nothing deleted (decision 68) diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/07-box-log.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/07-box-log.txt new file mode 100644 index 00000000..99ea3a3e --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/07-box-log.txt @@ -0,0 +1,200 @@ +2026-10-28T06:37:19Z INFO [web] Login from 192.168.0.180 +2026-10-28T06:37:19Z INFO [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (493.1 KB, 531ms, 72 tables) +2026-10-28T06:37:19Z INFO [settings] Settings saved +2026-10-28T06:37:20Z INFO [backup] DB dump: nextcloud-db → nextcloud-mariadb.sql (3.4 MB, 769ms, 131 tables) +2026-10-28T06:37:20Z INFO [settings] Settings saved +2026-10-28T06:37:22Z INFO [backup] DB dump: immich-postgres → immich-postgres.sql (51.1 MB, 1.445s, 71 tables) +2026-10-28T06:37:22Z INFO [settings] Settings saved +2026-10-28T06:37:22Z INFO [backup] DB dump: bookstack-db → bookstack-mariadb.sql (71.9 KB, 393ms, 41 tables) +2026-10-28T06:37:22Z INFO [settings] Settings saved +2026-10-28T06:37:22Z INFO [backup] Stopping actualbudget for safe volume dump +2026-10-28T06:37:22Z INFO [stacks] Stopping stack: actualbudget +2026-10-28T06:37:23Z INFO [stacks] Stack actualbudget stopped successfully (took 0.3s) +2026-10-28T06:37:23Z INFO [stacks] Status refresh: 18 containers across 68 stacks +2026-10-28T06:37:23Z INFO [backup] Volume dump: actualbudget/actualbudget_actualbudget_data → 508.5 KB +2026-10-28T06:37:23Z INFO [backup] Restarting actualbudget after volume dump +2026-10-28T06:37:23Z INFO [stacks] Starting stack: actualbudget +2026-10-28T06:37:24Z INFO [stacks] Stack actualbudget started successfully (took 0.3s) +2026-10-28T06:37:24Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:37:24Z INFO [backup] Stopping bookstack for safe volume dump +2026-10-28T06:37:24Z INFO [stacks] Stopping stack: bookstack +2026-10-28T06:37:27Z INFO [stacks] Stack actualbudget post-start status: +2026-10-28T06:37:27Z INFO [stacks] actualbudget actualbudget/actual-server:26.9.0@sha256:552beab3dec8c93d46b8b9245612d63c3f123b8a45063a474f53e229b17621d3 running Up 3 seconds (health: starting) +2026-10-28T06:37:27Z INFO [scheduler] Running job: backup-cache +2026-10-28T06:37:27Z INFO [scheduler] Running job: docker-socket-users +2026-10-28T06:37:27Z INFO [scheduler] Running job: offsite-credential-retry +2026-10-28T06:37:27Z INFO [scheduler] Job offsite-credential-retry completed (took 0s) +2026-10-28T06:37:27Z INFO [scheduler] Running job: fill-watch-interval +2026-10-28T06:37:27Z INFO [scheduler] Running job: stack-scan +2026-10-28T06:37:27Z INFO [scheduler] Running job: system-health +2026-10-28T06:37:27Z INFO [fillwatch] checked 4 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026-10-28T06:37:27Z INFO [scheduler] Job fill-watch-interval completed (took 1ms) +2026-10-28T06:37:27Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:37:28Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available) +2026-10-28T06:37:28Z INFO [scheduler] Job docker-socket-users completed (took 303ms) +2026-10-28T06:37:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:37:28Z INFO [scheduler] Job stack-scan completed (took 338ms) +2026-10-28T06:37:28Z INFO [backup] Found 7 DB dump files across drives +2026-10-28T06:37:28Z INFO [monitor] Health check: status=ok +2026-10-28T06:37:28Z INFO [scheduler] Job system-health completed (took 367ms) +2026-10-28T06:37:28Z INFO [scheduler] Running job: agent-channel-health +2026-10-28T06:37:28Z INFO [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node) +2026-10-28T06:37:28Z INFO [backup] Discovered 4 databases +2026-10-28T06:37:28Z ERROR [stacks] execCommand failed: exit status 1 +2026-10-28T06:37:28Z INFO [backup] Recovery unit captured for actualbudget → /mnt/sys_drive/felhom-data/backups/primary/actualbudget (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026-10-28T06:37:28Z INFO [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026-10-28T06:37:28Z INFO [backup] Recovery unit captured for immich → /mnt/felhom-drives/hdd_1/backups/primary/immich (images=4, secrets-referenced=1, data_keys=0, portable-carried=1/1, withheld=0) +2026-10-28T06:37:28Z INFO [backup] Recovery unit captured for mealie → /mnt/sys_drive/felhom-data/backups/primary/mealie (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026-10-28T06:37:28Z INFO [backup] Recovery unit captured for nextcloud → /mnt/felhom-drives/hdd_1/backups/primary/nextcloud (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026-10-28T06:37:28Z INFO [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026-10-28T06:37:28Z INFO [backup] Recovery unit captured for radicale → /mnt/sys_drive/felhom-data/backups/primary/radicale (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026-10-28T06:37:28Z INFO [backup] Recovery unit captured for vaultwarden → /mnt/sys_drive/felhom-data/backups/primary/vaultwarden (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026-10-28T06:37:28Z INFO [backup] Backup status cache refreshed +2026-10-28T06:37:28Z INFO [scheduler] Job backup-cache completed (took 821ms) +2026-10-28T06:37:28Z INFO [stacks] Stack bookstack stopped successfully (took 4.4s) +2026-10-28T06:37:28Z INFO [stacks] Status refresh: 17 containers across 68 stacks +2026-10-28T06:37:29Z INFO [backup] Volume dump: bookstack/bookstack_bookstack_config → 265.5 KB +2026-10-28T06:37:31Z INFO [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.5 MB +2026-10-28T06:37:31Z INFO [backup] Restarting bookstack after volume dump +2026-10-28T06:37:31Z INFO [stacks] Starting stack: bookstack +2026-10-28T06:37:35Z INFO [scheduler] Job agent-channel-health completed (took 7.308s) +2026-10-28T06:37:37Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:37:38Z INFO [stacks] Stack bookstack started successfully (took 6.2s) +2026-10-28T06:37:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:37:38Z INFO [backup] Stopping immich for safe volume dump +2026-10-28T06:37:38Z INFO [stacks] Stopping stack: immich +2026-10-28T06:37:39Z INFO [stacks] Stack immich stopped successfully (took 0.8s) +2026-10-28T06:37:39Z INFO [stacks] Status refresh: 15 containers across 68 stacks +2026-10-28T06:37:41Z INFO [stacks] Stack bookstack post-start status: +2026-10-28T06:37:41Z INFO [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 running Up 3 seconds (health: starting) +2026-10-28T06:37:41Z INFO [stacks] bookstack-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy) +2026-10-28T06:37:42Z ERROR [quiesce] cycle error: check due: agentapi: GET /backup/tiers: Get "https://169.254.253.1:8443/backup/tiers": context deadline exceeded (Client.Timeout exceeded while awaiting headers) +2026-10-28T06:37:42Z INFO [web] Login from 192.168.0.180 +2026-10-28T06:37:48Z INFO [stacks] Status refresh: 15 containers across 68 stacks +2026-10-28T06:37:49Z INFO Health probes: 1 ok (of 1 probed) +2026-10-28T06:37:49Z INFO [backup] Volume dump: immich/immich_immich_ml_cache → 785.6 MB +2026-10-28T06:37:55Z INFO [backup] Volume dump: immich/immich_immich_postgres_data → 328.3 MB +2026-10-28T06:37:55Z INFO [backup] Restarting immich after volume dump +2026-10-28T06:37:55Z INFO [stacks] Starting stack: immich +2026-10-28T06:37:57Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:37:57Z INFO Health probes: 1 ok (of 1 probed) +2026-10-28T06:38:02Z INFO [web] Login from 192.168.0.180 +2026-10-28T06:38:07Z INFO [stacks] Stack immich started successfully (took 11.7s) +2026-10-28T06:38:07Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:38:07Z INFO [backup] Stopping mealie for safe volume dump +2026-10-28T06:38:07Z INFO [stacks] Stopping stack: mealie +2026-10-28T06:38:07Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:38:10Z INFO [stacks] Stack immich post-start status: +2026-10-28T06:38:10Z INFO [stacks] immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4@sha256:e16c2f166a8174901959fdf85e2e4c7bd1ebc4b37e0b6655de97c41408a260c4 running Up 14 seconds (healthy) +2026-10-28T06:38:10Z INFO [stacks] immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0@sha256:1a078b237c1d9b420b0ee59147386b4aa60d3a07a8e6a402fc84a57e41b043a4 running Up 14 seconds (healthy) +2026-10-28T06:38:10Z INFO [stacks] immich-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy) +2026-10-28T06:38:10Z INFO [stacks] immich-server ghcr.io/immich-app/immich-server:v3.2.4@sha256:d317916b28090c33eb36b308464ea391f8b7df1d850fcfea227a39ec879718c2 running Up 3 seconds (health: starting) +2026-10-28T06:38:11Z INFO [stacks] Stack mealie stopped successfully (took 3.9s) +2026-10-28T06:38:11Z INFO [stacks] Status refresh: 18 containers across 68 stacks +2026-10-28T06:38:12Z INFO [backup] Volume dump: mealie/mealie_mealie_data → 1.7 MB +2026-10-28T06:38:12Z INFO [backup] Restarting mealie after volume dump +2026-10-28T06:38:12Z INFO [stacks] Starting stack: mealie +2026-10-28T06:38:12Z INFO [stacks] Stack mealie started successfully (took 0.5s) +2026-10-28T06:38:12Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:38:12Z INFO [backup] Stopping nextcloud for safe volume dump +2026-10-28T06:38:12Z INFO [stacks] Stopping stack: nextcloud +2026-10-28T06:38:14Z INFO [stacks] Stack nextcloud stopped successfully (took 1.9s) +2026-10-28T06:38:14Z INFO [stacks] Status refresh: 16 containers across 68 stacks +2026-10-28T06:38:15Z INFO [stacks] Stack mealie post-start status: +2026-10-28T06:38:15Z INFO [stacks] mealie ghcr.io/mealie-recipes/mealie:v3.28.0@sha256:8b02290f4d1806f02acac6f25f6d48a3c965612fda1f8e914d5af533276f8688 running Up 3 seconds (health: starting) +2026-10-28T06:38:17Z INFO [stacks] Status refresh: 16 containers across 68 stacks +2026-10-28T06:38:23Z INFO [web] Login from 192.168.0.180 +2026-10-28T06:38:27Z INFO [stacks] Status refresh: 16 containers across 68 stacks +2026-10-28T06:38:27Z INFO [deadapp] check alive: 40 scans since boot, 8 deployed app(s) evaluated, 0 currently down +2026-10-28T06:38:28Z INFO [scheduler] Running job: agent-channel-health +2026-10-28T06:38:36Z INFO [scheduler] Job agent-channel-health completed (took 7.844s) +2026-10-28T06:38:37Z INFO [stacks] Status refresh: 16 containers across 68 stacks +2026-10-28T06:38:40Z INFO [backup] Volume dump: nextcloud/nextcloud_nextcloud_html → 761.6 MB +2026-10-28T06:38:42Z INFO [backup] Volume dump: nextcloud/nextcloud_nextcloud_db_data → 203.4 MB +2026-10-28T06:38:42Z INFO [backup] Restarting nextcloud after volume dump +2026-10-28T06:38:42Z INFO [stacks] Starting stack: nextcloud +2026-10-28T06:38:43Z INFO [web] Login from 192.168.0.180 +2026-10-28T06:38:47Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:38:47Z INFO Health probes: 2 ok (of 2 probed) +2026-10-28T06:38:48Z INFO [stacks] Stack nextcloud started successfully (took 6.4s) +2026-10-28T06:38:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:38:48Z INFO [backup] Stopping paperless-ngx for safe volume dump +2026-10-28T06:38:48Z INFO [stacks] Stopping stack: paperless-ngx +2026-10-28T06:38:51Z INFO [stacks] Stack nextcloud post-start status: +2026-10-28T06:38:51Z INFO [stacks] nextcloud nextcloud:34.0.4-apache@sha256:37b109885aa3cba3e056362a899556a625c986f2c17cd2c0cc157d21c789f53b running Up 3 seconds (health: starting) +2026-10-28T06:38:51Z INFO [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy) +2026-10-28T06:38:51Z INFO [stacks] nextcloud-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 9 seconds (healthy) +2026-10-28T06:38:57Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:38:59Z INFO [stacks] Stack paperless-ngx stopped successfully (took 10.7s) +2026-10-28T06:38:59Z INFO [stacks] Status refresh: 16 containers across 68 stacks +2026-10-28T06:39:00Z INFO [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 1.3 MB +2026-10-28T06:39:01Z INFO [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 70.4 MB +2026-10-28T06:39:01Z INFO [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_redis_data → 3.0 KB +2026-10-28T06:39:01Z INFO [backup] Restarting paperless-ngx after volume dump +2026-10-28T06:39:01Z INFO [stacks] Starting stack: paperless-ngx +2026-10-28T06:39:03Z INFO [web] Login from 192.168.0.180 +2026-10-28T06:39:07Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:07Z INFO Health probes: 1 ok (of 1 probed) +2026-10-28T06:39:13Z INFO [stacks] Stack paperless-ngx started successfully (took 11.4s) +2026-10-28T06:39:13Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:13Z INFO [backup] Stopping radicale for safe volume dump +2026-10-28T06:39:13Z INFO [stacks] Stopping stack: radicale +2026-10-28T06:39:14Z INFO [stacks] Stack radicale stopped successfully (took 0.9s) +2026-10-28T06:39:14Z INFO [stacks] Status refresh: 18 containers across 68 stacks +2026-10-28T06:39:14Z INFO [backup] Volume dump: radicale/radicale_radicale_data → 15.5 KB +2026-10-28T06:39:14Z INFO [backup] Restarting radicale after volume dump +2026-10-28T06:39:14Z INFO [stacks] Starting stack: radicale +2026-10-28T06:39:15Z INFO [stacks] Stack radicale started successfully (took 0.5s) +2026-10-28T06:39:15Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:15Z INFO [backup] Stopping vaultwarden for safe volume dump +2026-10-28T06:39:15Z INFO [stacks] Stopping stack: vaultwarden +2026-10-28T06:39:16Z INFO [stacks] Stack vaultwarden stopped successfully (took 0.5s) +2026-10-28T06:39:16Z INFO [stacks] Status refresh: 18 containers across 68 stacks +2026-10-28T06:39:16Z INFO [stacks] Stack paperless-ngx post-start status: +2026-10-28T06:39:16Z INFO [stacks] paperless-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 14 seconds (healthy) +2026-10-28T06:39:16Z INFO [stacks] paperless-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy) +2026-10-28T06:39:16Z INFO [stacks] paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15@sha256:6c86cad803970ea782683a8e80e7403444c5bf3cf70de63b4d3c8e87500db92f running Up 3 seconds (health: starting) +2026-10-28T06:39:16Z INFO [backup] Volume dump: vaultwarden/vaultwarden_vaultwarden_data → 371.5 KB +2026-10-28T06:39:16Z INFO [backup] Restarting vaultwarden after volume dump +2026-10-28T06:39:16Z INFO [stacks] Starting stack: vaultwarden +2026-10-28T06:39:17Z INFO [stacks] Stack vaultwarden started successfully (took 0.7s) +2026-10-28T06:39:17Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:18Z INFO [backup] App-data backup completed: 4 databases (55.1 MB total), 8 volume dump(s) (1m58.881s) +2026-10-28T06:39:18Z INFO [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026-10-28T06:39:18Z INFO [backup] Recovery unit captured for immich → /mnt/felhom-drives/hdd_1/backups/primary/immich (images=4, secrets-referenced=1, data_keys=0, portable-carried=1/1, withheld=0) +2026-10-28T06:39:18Z INFO [backup] Recovery unit captured for mealie → /mnt/sys_drive/felhom-data/backups/primary/mealie (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026-10-28T06:39:18Z INFO [backup] Recovery unit captured for nextcloud → /mnt/felhom-drives/hdd_1/backups/primary/nextcloud (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026-10-28T06:39:18Z INFO [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026-10-28T06:39:18Z INFO [backup] Recovery unit captured for radicale → /mnt/sys_drive/felhom-data/backups/primary/radicale (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026-10-28T06:39:18Z INFO [backup] Recovery unit captured for vaultwarden → /mnt/sys_drive/felhom-data/backups/primary/vaultwarden (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026-10-28T06:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:18Z INFO [offbox] pre-push dump leg completed in 1m59.396s — snapshot pair is coherent +2026-10-28T06:39:18Z INFO [stacks] Stack radicale post-start status: +2026-10-28T06:39:18Z INFO [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting) +2026-10-28T06:39:21Z INFO [stacks] Stack vaultwarden post-start status: +2026-10-28T06:39:21Z INFO [stacks] vaultwarden vaultwarden/server:1.37.4-alpine@sha256:19ea2b669945d842dadc3d500f2d6a795866a003330f59a47b9634658bae98bb running Up 3 seconds (health: starting) +2026-10-28T06:39:27Z INFO [settings] Settings saved +2026-10-28T06:39:27Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:27Z INFO [offbox] backed up nextcloud (/mnt/felhom-drives/hdd_1/backups/primary/nextcloud, 1 mandatory path(s)) +2026-10-28T06:39:27Z INFO [scheduler] Running job: stack-scan +2026-10-28T06:39:27Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:27Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available) +2026-10-28T06:39:27Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:27Z INFO [scheduler] Job stack-scan completed (took 257ms) +2026-10-28T06:39:28Z INFO Health probes: 2 ok (of 2 probed) +2026-10-28T06:39:28Z INFO [scheduler] Running job: agent-channel-health +2026-10-28T06:39:30Z INFO [settings] Settings saved +2026-10-28T06:39:30Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:30Z INFO [offbox] backed up paperless-ngx (/mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx, 1 mandatory path(s)) +2026-10-28T06:39:32Z INFO [settings] Settings saved +2026-10-28T06:39:32Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-28T06:39:32Z INFO [offbox] backed up radicale (/mnt/sys_drive/felhom-data/backups/primary/radicale, 0 mandatory path(s)) +2026-10-28T06:39:33Z INFO [web] Login from 192.168.0.180 \ No newline at end of file diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/README.md b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/README.md new file mode 100644 index 00000000..39498dd5 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/README.md @@ -0,0 +1,21 @@ +# B6 ★10 — the box off for 3 days, + #15 a restart in the middle of the catch-up (2026-10-21 07:57–08:45 real UTC) — PASS + +Method (`00-expected.txt`, `01-run.txt`): NTP blocked; clean stop; start with the clock 3 days on. **Correction, recorded +at 08:00:54 before any result:** the first start (2026-10-24) was wrong — #7 had left the box's backup records dated +2026-10-25, so 10-24 was not „3 days off". Restarted at 2026-10-28 06:00Z, three days after those records. + +| box time | what happened | +|---|---| +| 06:00:37Z | „[catch-up] controller start: the box missed [db-dump tier2 offsite] (last scheduled 2026-10-28 04:15) — **ONE catch-up in 15m0s** (backup legs only; app updates wait for a real night)" | +| ~06:05Z | the household's start page: „A legutóbbi mentés 3 nappal ezelőtt készült. A doboz ki volt kapcsolva a mentés idején (02:30). Válassz egy olyan időpontot, amikor a doboz általában be van kapcsolva." + „Mentési idő módosítása" (`03-household-view.txt`) | +| 06:15:37Z | the catch-up starts: database leg, apps stopped one by one for their volume dumps | +| 06:15:51Z | **#15: clean restart** while the dumps ran | +| 06:17:27Z | after the restart: „an app-data backup (volume dump) (op volume-dump:paperless-ngx) was interrupted … left 1 app(s) stopped" — Paperless kept stopped with its marker because the data drive was not yet mounted; „ONE catch-up in 15m0s" scheduled again | +| 06:18:37Z | Paperless started by the drive gate; all 8 apps running (08:23 real) | +| 06:32–06:39:52Z | the catch-up: database copy, second drive (8 apps), off-site (8 apps); retention skipped, nothing deleted | + +**Verdict: PASS.** One catch-up for three missed nights, a clear note to the household, an interrupted catch-up +recovered with no app left down and no copy broken. One `execCommand failed: exit status 1` at 06:37:28Z (inside the +off-site leg's start) had no visible effect; not chased (time). +**Left behind by the method:** off-site snapshots dated 2026-10-28 on the Storage Box (with the 2026-10-25 ones from #7). +B7 restores from this repository, so the newest snapshot it sees is a „future" one — that is part of what B7 reads. diff --git a/documentation/audits/rehearsal-2026-10-20/journal.md b/documentation/audits/rehearsal-2026-10-20/journal.md index 8afef49a..6dfd64b7 100644 --- a/documentation/audits/rehearsal-2026-10-20/journal.md +++ b/documentation/audits/rehearsal-2026-10-20/journal.md @@ -85,3 +85,4 @@ | 05:40 | mail | the nightly DooPlex secret sweep named three rehearsal secrets in session records: OLD_BIND_TOKEN, BIND_TOKEN, SETUP_CODE — my mail reads (full text of the bind and setup mails) carried them in; all three are one-time and already spent. From now on mail is read names-only | | 05:58–07:13 | B6 ★7 | DST live: the first 02:30 skipped, the second ran once (01:30:00Z box), PASS; side effects of the 4-day jump: a local whole-box archive and off-site snapshots dated 2026-10-25; W30 | | 07:15–07:58 | B6 ★8 | −2 y (OS raised it to 2026-04-13): „Hub nem elérhető" only, Paperless stopped silently → R-1202; +2 y: whole-box archive dated 2028; back to real time: ep0 WireGuard tunnel dead (handshake replay) → R-1201 (P2); R-1189 (a) half-live | +| 07:57–08:45 | B6 ★10 + #15 | 3 days off (box clock 2026-10-28): ONE catch-up, clear household note; a restart mid-catch-up recovered (marker, drive gate, one new catch-up); 8 apps off-site; PASS |