From a3be7525cddd5d06191ae18e1292d73a7cd2d128 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 21 Oct 2026 09:13:35 +0200 Subject: [PATCH] =?UTF-8?q?rehearsal=202026-10-20:=20B6=20#7=20DST=20live?= =?UTF-8?q?=20=E2=80=94=20PASS=20(one=20database=20copy=20at=20the=20secon?= =?UTF-8?q?d=2002:30);=20#8/#10=20expectations=20written=20before?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- REPORT-rehearsal-2026-10-20.md | 15 +- .../s10-three-days-off/00-expected.txt | 6 + .../B6-failures/s7-dst/00-expected.txt | 10 + .../B6-failures/s7-dst/01-run.txt | 9 + .../s7-dst/02-box-log-0020-0135Z.txt | 200 ++++++++++++++++++ .../B6-failures/s7-dst/README.md | 23 ++ .../s8-wrong-clock/00-expected.txt | 9 + .../rehearsal-2026-10-20/B9-words/words.md | 1 + .../audits/rehearsal-2026-10-20/journal.md | 1 + 9 files changed, 271 insertions(+), 3 deletions(-) create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/00-expected.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/00-expected.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/01-run.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/02-box-log-0020-0135Z.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/README.md create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s8-wrong-clock/00-expected.txt diff --git a/REPORT-rehearsal-2026-10-20.md b/REPORT-rehearsal-2026-10-20.md index 0e4d28a7..8c68dfee 100644 --- a/REPORT-rehearsal-2026-10-20.md +++ b/REPORT-rehearsal-2026-10-20.md @@ -76,9 +76,18 @@ names come back on the share; Nextcloud itself folds NFD into NFC); A.1.5 BookSt ## Night chain and DST -PENDING (morning). Code replay: the controller runs each daily job once on 2026-10-25 (a 02:xx job at the second, -CET occurrence); the hub runs every daily job twice (R-1183); spring 2027 puts the 02:30 dump on the 03:30 -second-drive copy (`B6-failures/dst-replay/`). +**The first night (B5), no hands:** database copy 00:30Z PASS (3 m 27 s); second-drive copy 01:30Z PASS (8 apps + the +share, 58 s); off-site 02:15Z PASS (44 snapshots, 3 m 6 s, nothing deleted; the hub's own listing agrees: 44 files, +newest 02:18Z); whole-box backup NOT RUN — not due under the 24 h rule, because the install's own backup at 12:09Z +started the clock (R-1200, P3: the family's first day reaches a whole-box copy only on the second night). At 02:46Z +demo-hp's own kernel update rebooted the test host and shut the rehearsal VM down (harness, not a Tester4 defect; a BYO +host skips that step by design). Health 200 all night before that; memory steady. **The morning data snapshot is +identical to the evening one, byte for byte** (`B5-night/README.md`). + +**DST (#7):** PENDING — running on the box now. So far: at the first 02:30 (summer time) the database copy did NOT run, +as the code replay said. Code replay: the controller runs each daily job once on 2026-10-25 (a 02:xx job at the second, +CET occurrence); the hub runs every daily job twice (R-1183); spring 2027 puts the 02:30 dump on the 03:30 second-drive +copy (`B6-failures/dst-replay/`). ## Interventions (operator-only acts) diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/00-expected.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/00-expected.txt new file mode 100644 index 00000000..e3bed154 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s10-three-days-off/00-expected.txt @@ -0,0 +1,6 @@ +Written 2026-10-21 ~06:20 UTC, before #10 ran. +Method: clean stop; start with the RTC 3 days after the real time (NTP blocked), as if the box had been off for 3 +days; during the catch-up, one restart (#15); then the real time, NTP open. +Expected: one catch-up for the missed nights (database copy, second drive, off-site), not three; the household sees a +„missed backups" note and then a catch-up line; the restart in the middle leaves nothing half-done — the catch-up +starts again or is recorded as interrupted, and no copy is left broken. diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/00-expected.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/00-expected.txt new file mode 100644 index 00000000..2d3fe31e --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/00-expected.txt @@ -0,0 +1,10 @@ +Written 2026-10-21 ~05:52 UTC, before #7 ran. +Method (plan amendment 20:27): NTP (UDP 123) and NTS-KE (TCP 4460) dropped at the VM's tap; the VM restarted cleanly +with its RTC at 2026-10-25T00:20:00Z (= 02:20 CEST); it runs ~80 real minutes, through 03:00 CEST → 02:00 CET, to +~02:40 CET (01:40Z). Positive control for the clock: the box's own HTTP Date header (a different channel than the RTC +setting). demo-hp's clock is not touched. +Expected from the code replay (B6-failures/dst-replay): the box's 02:30 database copy runs ONCE, at the SECOND +(CET) 02:30 = 01:30Z box time, not at 02:30 CEST (00:30Z). Nothing runs twice. The dashboard shows local times in CET +after the change. +Side effect of the method, not a DST result: the box wakes „4 days later" than its last backup; a missed-night +catch-up may start at once. diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/01-run.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/01-run.txt new file mode 100644 index 00000000..a2589a3e --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/01-run.txt @@ -0,0 +1,9 @@ +} +ntp off 05:58:38 +down 06:00:32 +update VM 9320: -startdate 2026-10-25T00:20:00 +started with RTC 2026-10-25T00:20:00 at real 06:00:35 +real 2026-10-21T06:01:14Z box Sun, 25 Oct 2026 00:20:39 GMT +real 2026-10-21T07:12:43Z box Sun, 25 Oct 2026 01:32:08 GMT + iifname "tap9320i0" udp dport 123 counter packets 142 bytes 10792 drop + iifname "tap9320i0" tcp dport 4460 counter packets 0 bytes 0 drop diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/02-box-log-0020-0135Z.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/02-box-log-0020-0135Z.txt new file mode 100644 index 00000000..3e104876 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/02-box-log-0020-0135Z.txt @@ -0,0 +1,200 @@ +2026-10-25T01:28:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:28:38Z INFO [scheduler] Running job: stack-scan +2026-10-25T01:28:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:28:38Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available) +2026-10-25T01:28:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:28:38Z INFO [scheduler] Job stack-scan completed (took 159ms) +2026-10-25T01:28:39Z INFO [scheduler] Running job: agent-channel-health +2026-10-25T01:28:39Z INFO [scheduler] Job agent-channel-health completed (took 532ms) +2026-10-25T01:28:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:28:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:29:08Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:29:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:29:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:29:28Z INFO Health probes: 1 ok (of 1 probed) +2026-10-25T01:29:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:29:39Z INFO [scheduler] Running job: agent-channel-health +2026-10-25T01:29:39Z INFO [scheduler] Job agent-channel-health completed (took 466ms) +2026-10-25T01:29:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:29:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:29:58Z INFO Health probes: 1 ok (of 1 probed) +2026-10-25T01:30:00Z INFO [scheduler] Running job: db-dump +2026-10-25T01:30:00Z INFO [backup] Starting database dump run +2026-10-25T01:30:00Z INFO [backup] Discovered 4 databases +2026-10-25T01:30:00Z INFO [backup] Discovered 4 database(s): paperless-postgres(postgres), nextcloud-db(mariadb), immich-postgres(postgres), bookstack-db(mariadb) +2026-10-25T01:30:00Z INFO [backup] DB dump: paperless-postgres → paperless-ngx-postgres.sql (490.0 KB, 449ms, 72 tables) +2026-10-25T01:30:00Z INFO [settings] Settings saved +2026-10-25T01:30:01Z INFO [backup] DB dump: nextcloud-db → nextcloud-mariadb.sql (3.4 MB, 762ms, 131 tables) +2026-10-25T01:30:01Z INFO [settings] Settings saved +2026-10-25T01:30:03Z INFO [backup] DB dump: immich-postgres → immich-postgres.sql (51.1 MB, 1.61s, 71 tables) +2026-10-25T01:30:03Z INFO [settings] Settings saved +2026-10-25T01:30:03Z INFO [backup] DB dump: bookstack-db → bookstack-mariadb.sql (71.9 KB, 363ms, 41 tables) +2026-10-25T01:30:03Z INFO [settings] Settings saved +2026-10-25T01:30:03Z INFO [backup] Stopping actualbudget for safe volume dump +2026-10-25T01:30:03Z INFO [stacks] Stopping stack: actualbudget +2026-10-25T01:30:04Z INFO [stacks] Stack actualbudget stopped successfully (took 0.3s) +2026-10-25T01:30:04Z INFO [stacks] Status refresh: 18 containers across 68 stacks +2026-10-25T01:30:04Z INFO [backup] Volume dump: actualbudget/actualbudget_actualbudget_data → 508.5 KB +2026-10-25T01:30:04Z INFO [backup] Restarting actualbudget after volume dump +2026-10-25T01:30:04Z INFO [stacks] Starting stack: actualbudget +2026-10-25T01:30:04Z INFO [stacks] Stack actualbudget started successfully (took 0.4s) +2026-10-25T01:30:05Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:30:05Z INFO [backup] Stopping bookstack for safe volume dump +2026-10-25T01:30:05Z INFO [stacks] Stopping stack: bookstack +2026-10-25T01:30:08Z INFO [stacks] Stack actualbudget post-start status: +2026-10-25T01:30:08Z INFO [stacks] actualbudget actualbudget/actual-server:26.9.0@sha256:552beab3dec8c93d46b8b9245612d63c3f123b8a45063a474f53e229b17621d3 running Up 3 seconds (health: starting) +2026-10-25T01:30:08Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:30:09Z INFO [stacks] Stack bookstack stopped successfully (took 4.5s) +2026-10-25T01:30:09Z INFO [stacks] Status refresh: 17 containers across 68 stacks +2026-10-25T01:30:10Z INFO [backup] Volume dump: bookstack/bookstack_bookstack_config → 353.0 KB +2026-10-25T01:30:11Z INFO [backup] Volume dump: bookstack/bookstack_bookstack_db_data → 153.5 MB +2026-10-25T01:30:11Z INFO [backup] Restarting bookstack after volume dump +2026-10-25T01:30:11Z INFO [stacks] Starting stack: bookstack +2026-10-25T01:30:18Z INFO [stacks] Stack bookstack started successfully (took 6.2s) +2026-10-25T01:30:18Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:30:18Z INFO [backup] Stopping immich for safe volume dump +2026-10-25T01:30:18Z INFO [stacks] Stopping stack: immich +2026-10-25T01:30:18Z INFO [stacks] Status refresh: 17 containers across 68 stacks +2026-10-25T01:30:18Z INFO Health probes: 1 ok (of 1 probed) +2026-10-25T01:30:19Z INFO [stacks] Stack immich stopped successfully (took 0.9s) +2026-10-25T01:30:19Z INFO [stacks] Status refresh: 15 containers across 68 stacks +2026-10-25T01:30:21Z INFO [stacks] Stack bookstack post-start status: +2026-10-25T01:30:21Z INFO [stacks] bookstack lscr.io/linuxserver/bookstack:26.09.1@sha256:99cd1f5707c1911afad213adec5c9739763b76f843d1477142231834ecdcb6f7 running Up 3 seconds (health: starting) +2026-10-25T01:30:21Z INFO [stacks] bookstack-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy) +2026-10-25T01:30:29Z INFO [stacks] Status refresh: 15 containers across 68 stacks +2026-10-25T01:30:30Z INFO [backup] Volume dump: immich/immich_immich_ml_cache → 785.6 MB +2026-10-25T01:30:34Z INFO [backup] Volume dump: immich/immich_immich_postgres_data → 328.3 MB +2026-10-25T01:30:34Z INFO [backup] Restarting immich after volume dump +2026-10-25T01:30:34Z INFO [stacks] Starting stack: immich +2026-10-25T01:30:38Z INFO [scheduler] Running job: fill-watch-interval +2026-10-25T01:30:38Z INFO [fillwatch] checked 4 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok +2026-10-25T01:30:38Z INFO [scheduler] Job fill-watch-interval completed (took 1ms) +2026-10-25T01:30:38Z INFO [scheduler] Running job: stack-scan +2026-10-25T01:30:38Z INFO [scheduler] Running job: backup-cache +2026-10-25T01:30:38Z INFO [scheduler] Running job: system-health +2026-10-25T01:30:38Z INFO [scheduler] Running job: docker-socket-users +2026-10-25T01:30:38Z INFO [scheduler] Running job: offsite-credential-retry +2026-10-25T01:30:38Z INFO [scheduler] Job offsite-credential-retry completed (took 0s) +2026-10-25T01:30:38Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:30:39Z INFO [stacks] ScanStacks complete: 68 stacks found (8 deployed, 60 available) +2026-10-25T01:30:39Z INFO [scheduler] Job docker-socket-users completed (took 274ms) +2026-10-25T01:30:39Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:30:39Z INFO [scheduler] Job stack-scan completed (took 302ms) +2026-10-25T01:30:39Z INFO [backup] Found 7 DB dump files across drives +2026-10-25T01:30:39Z INFO [monitor] Health check: status=ok +2026-10-25T01:30:39Z INFO [scheduler] Job system-health completed (took 334ms) +2026-10-25T01:30:39Z INFO [scheduler] Running job: agent-channel-health +2026-10-25T01:30:39Z INFO [backup] Discovered 4 databases +2026-10-25T01:30:39Z INFO [infra] cloudflared skipped — no cf_tunnel_token configured (LAN-only node) +2026-10-25T01:30:39Z INFO [backup] Recovery unit captured for actualbudget → /mnt/sys_drive/felhom-data/backups/primary/actualbudget (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0) +2026-10-25T01:30:39Z INFO [backup] Recovery unit captured for bookstack → /mnt/sys_drive/felhom-data/backups/primary/bookstack (images=2, secrets-referenced=3, data_keys=1, portable-carried=2/2, withheld=1) +2026-10-25T01:30:39Z INFO [backup] Recovery unit captured for immich → /mnt/felhom-drives/hdd_1/backups/primary/immich (images=4, secrets-referenced=1, data_keys=0, portable-carried=1/1, withheld=0) +2026-10-25T01:30:39Z INFO [backup] Recovery unit captured for nextcloud → /mnt/felhom-drives/hdd_1/backups/primary/nextcloud (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026-10-25T01:30:39Z INFO [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026-10-25T01:30:39Z INFO [backup] Backup status cache refreshed +2026-10-25T01:30:39Z INFO [scheduler] Job backup-cache completed (took 653ms) +2026-10-25T01:30:39Z INFO [scheduler] Job agent-channel-health completed (took 562ms) +2026-10-25T01:30:46Z INFO [stacks] Stack immich started successfully (took 11.2s) +2026-10-25T01:30:46Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:30:46Z INFO [backup] Stopping mealie for safe volume dump +2026-10-25T01:30:46Z INFO [stacks] Stopping stack: mealie +2026-10-25T01:30:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:30:48Z INFO Health probes: 1 ok (of 1 probed) +2026-10-25T01:30:49Z INFO [stacks] Stack immich post-start status: +2026-10-25T01:30:49Z INFO [stacks] immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4@sha256:e16c2f166a8174901959fdf85e2e4c7bd1ebc4b37e0b6655de97c41408a260c4 running Up 13 seconds (healthy) +2026-10-25T01:30:49Z INFO [stacks] immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0@sha256:1a078b237c1d9b420b0ee59147386b4aa60d3a07a8e6a402fc84a57e41b043a4 running Up 13 seconds (healthy) +2026-10-25T01:30:49Z INFO [stacks] immich-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy) +2026-10-25T01:30:49Z INFO [stacks] immich-server ghcr.io/immich-app/immich-server:v3.2.4@sha256:d317916b28090c33eb36b308464ea391f8b7df1d850fcfea227a39ec879718c2 running Up 3 seconds (health: starting) +2026-10-25T01:30:49Z INFO [stacks] Stack mealie stopped successfully (took 3.2s) +2026-10-25T01:30:49Z INFO [stacks] Status refresh: 18 containers across 68 stacks +2026-10-25T01:30:50Z INFO [backup] Volume dump: mealie/mealie_mealie_data → 1.6 MB +2026-10-25T01:30:50Z INFO [backup] Restarting mealie after volume dump +2026-10-25T01:30:50Z INFO [stacks] Starting stack: mealie +2026-10-25T01:30:50Z INFO [stacks] Stack mealie started successfully (took 0.4s) +2026-10-25T01:30:50Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:30:50Z INFO [backup] Stopping nextcloud for safe volume dump +2026-10-25T01:30:50Z INFO [stacks] Stopping stack: nextcloud +2026-10-25T01:30:52Z INFO [stacks] Stack nextcloud stopped successfully (took 1.8s) +2026-10-25T01:30:52Z INFO [stacks] Status refresh: 16 containers across 68 stacks +2026-10-25T01:30:53Z INFO [stacks] Stack mealie post-start status: +2026-10-25T01:30:53Z INFO [stacks] mealie ghcr.io/mealie-recipes/mealie:v3.28.0@sha256:8b02290f4d1806f02acac6f25f6d48a3c965612fda1f8e914d5af533276f8688 running Up 3 seconds (health: starting) +2026-10-25T01:30:54Z INFO [backup] Volume dump: nextcloud/nextcloud_nextcloud_db_data → 203.4 MB +2026-10-25T01:30:58Z INFO [stacks] Status refresh: 16 containers across 68 stacks +2026-10-25T01:31:08Z INFO [stacks] Status refresh: 16 containers across 68 stacks +2026-10-25T01:31:18Z INFO [stacks] Status refresh: 16 containers across 68 stacks +2026-10-25T01:31:18Z INFO Health probes: 2 ok (of 2 probed) +2026-10-25T01:31:21Z INFO [backup] Volume dump: nextcloud/nextcloud_nextcloud_html → 761.6 MB +2026-10-25T01:31:21Z INFO [backup] Restarting nextcloud after volume dump +2026-10-25T01:31:21Z INFO [stacks] Starting stack: nextcloud +2026-10-25T01:31:27Z INFO [stacks] Stack nextcloud started successfully (took 6.5s) +2026-10-25T01:31:27Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:27Z INFO [backup] Stopping paperless-ngx for safe volume dump +2026-10-25T01:31:27Z INFO [stacks] Stopping stack: paperless-ngx +2026-10-25T01:31:28Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:30Z INFO [stacks] Stack nextcloud post-start status: +2026-10-25T01:31:30Z INFO [stacks] nextcloud nextcloud:34.0.4-apache@sha256:37b109885aa3cba3e056362a899556a625c986f2c17cd2c0cc157d21c789f53b running Up 3 seconds (health: starting) +2026-10-25T01:31:30Z INFO [stacks] nextcloud-db mariadb:12.3@sha256:805c8e104bd563d5bfa24fadd3f31cd419ea859cb5277f32b5dbf2db714f9ed1 running Up 9 seconds (healthy) +2026-10-25T01:31:30Z INFO [stacks] nextcloud-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 9 seconds (healthy) +2026-10-25T01:31:35Z INFO [web] Login from 192.168.0.180 +2026-10-25T01:31:37Z INFO [stacks] Stack paperless-ngx stopped successfully (took 9.5s) +2026-10-25T01:31:37Z INFO [stacks] Status refresh: 16 containers across 68 stacks +2026-10-25T01:31:37Z INFO [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_data → 1.1 MB +2026-10-25T01:31:38Z INFO [stacks] Status refresh: 16 containers across 68 stacks +2026-10-25T01:31:38Z INFO Health probes: 3 ok (of 3 probed) +2026-10-25T01:31:38Z INFO [deadapp] check alive: 140 scans since boot, 8 deployed app(s) evaluated, 0 currently down +2026-10-25T01:31:38Z INFO [deadapp] 1 stopped app(s) have NO recorded customer intent, so their dead-app alarm is suppressed by the unknown-intent fallback (R-386): paperless-ngx. This closes itself as each app is started or stopped through the interface. +2026-10-25T01:31:39Z INFO [scheduler] Running job: agent-channel-health +2026-10-25T01:31:39Z INFO [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_postgres_data → 70.4 MB +2026-10-25T01:31:39Z INFO [scheduler] Job agent-channel-health completed (took 753ms) +2026-10-25T01:31:40Z INFO [backup] Volume dump: paperless-ngx/paperless-ngx_paperless_redis_data → 2.5 KB +2026-10-25T01:31:40Z INFO [backup] Restarting paperless-ngx after volume dump +2026-10-25T01:31:40Z INFO [stacks] Starting stack: paperless-ngx +2026-10-25T01:31:47Z INFO [web] Login from 192.168.0.180 +2026-10-25T01:31:48Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:50Z INFO [stacks] Stack paperless-ngx started successfully (took 10.9s) +2026-10-25T01:31:51Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:51Z INFO [backup] Stopping radicale for safe volume dump +2026-10-25T01:31:51Z INFO [stacks] Stopping stack: radicale +2026-10-25T01:31:51Z INFO [stacks] Stack radicale stopped successfully (took 0.7s) +2026-10-25T01:31:51Z INFO [stacks] Status refresh: 18 containers across 68 stacks +2026-10-25T01:31:52Z INFO [backup] Volume dump: radicale/radicale_radicale_data → 15.5 KB +2026-10-25T01:31:52Z INFO [backup] Restarting radicale after volume dump +2026-10-25T01:31:52Z INFO [stacks] Starting stack: radicale +2026-10-25T01:31:52Z INFO [stacks] Stack radicale started successfully (took 0.5s) +2026-10-25T01:31:53Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:53Z INFO [backup] Stopping vaultwarden for safe volume dump +2026-10-25T01:31:53Z INFO [stacks] Stopping stack: vaultwarden +2026-10-25T01:31:53Z INFO [stacks] Stack vaultwarden stopped successfully (took 0.4s) +2026-10-25T01:31:53Z INFO [stacks] Status refresh: 18 containers across 68 stacks +2026-10-25T01:31:54Z INFO [backup] Volume dump: vaultwarden/vaultwarden_vaultwarden_data → 371.5 KB +2026-10-25T01:31:54Z INFO [backup] Restarting vaultwarden after volume dump +2026-10-25T01:31:54Z INFO [stacks] Starting stack: vaultwarden +2026-10-25T01:31:54Z INFO [stacks] Stack paperless-ngx post-start status: +2026-10-25T01:31:54Z INFO [stacks] paperless-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 13 seconds (healthy) +2026-10-25T01:31:54Z INFO [stacks] paperless-redis redis:7-alpine@sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499 running Up 14 seconds (healthy) +2026-10-25T01:31:54Z INFO [stacks] paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15@sha256:6c86cad803970ea782683a8e80e7403444c5bf3cf70de63b4d3c8e87500db92f running Up 3 seconds (health: starting) +2026-10-25T01:31:54Z INFO [stacks] Stack vaultwarden started successfully (took 0.7s) +2026-10-25T01:31:55Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:55Z INFO [backup] App-data backup completed: 4 databases (55.1 MB total), 8 volume dump(s) (1m55.223s) +2026-10-25T01:31:55Z INFO [backup] Recovery unit captured for mealie → /mnt/sys_drive/felhom-data/backups/primary/mealie (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026-10-25T01:31:55Z INFO [backup] Recovery unit captured for nextcloud → /mnt/felhom-drives/hdd_1/backups/primary/nextcloud (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026-10-25T01:31:55Z INFO [backup] Recovery unit captured for paperless-ngx → /mnt/felhom-drives/hdd_1/backups/primary/paperless-ngx (images=3, secrets-referenced=3, data_keys=0, portable-carried=2/2, withheld=1) +2026-10-25T01:31:55Z INFO [backup] Recovery unit captured for radicale → /mnt/sys_drive/felhom-data/backups/primary/radicale (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026-10-25T01:31:55Z INFO [backup] Recovery unit captured for vaultwarden → /mnt/sys_drive/felhom-data/backups/primary/vaultwarden (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1) +2026-10-25T01:31:55Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:55Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:55Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:55Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:55Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:55Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:55Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:55Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:31:55Z INFO [scheduler] Job db-dump completed (took 1m55.675s) +2026-10-25T01:31:55Z INFO Event pushed: db_dump_completed (info) [hu-only] — Adatbázis mentés elkészült +2026-10-25T01:31:56Z INFO [stacks] Stack radicale post-start status: +2026-10-25T01:31:56Z INFO [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting) +2026-10-25T01:31:58Z INFO [stacks] Stack vaultwarden post-start status: +2026-10-25T01:31:58Z INFO [stacks] vaultwarden vaultwarden/server:1.37.4-alpine@sha256:19ea2b669945d842dadc3d500f2d6a795866a003330f59a47b9634658bae98bb running Up 3 seconds (health: starting) +2026-10-25T01:31:58Z INFO [stacks] Status refresh: 19 containers across 68 stacks +2026-10-25T01:32:08Z INFO [web] Login from 192.168.0.180 +2026-10-25T01:32:08Z INFO [web] Login from 192.168.0.180 diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/README.md b/documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/README.md new file mode 100644 index 00000000..930d9044 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s7-dst/README.md @@ -0,0 +1,23 @@ +# B6 ★7 — the clocks go back (2026-10-25 03:00 CEST → 02:00 CET), live on the rehearsal box — AS EXPECTED + +Run 2026-10-21 05:58–07:13 real UTC. Method (`00-expected.txt`): NTP dropped at the VM's tap (142 packets dropped — +positive control), the VM restarted cleanly with its RTC at 2026-10-25 00:20:00Z (02:20 CEST), then left alone. +Clock control: the box's own HTTP Date header, read 5 times (`01-run.txt`). demo-hp's clock untouched. + +| box time (UTC / local) | what happened | expected? | +|---|---|---| +| 00:20Z / 02:20 CEST | boot; all apps up in ~1 min | — | +| 00:25:42Z / 02:25 CEST | the box found the whole-box tier „due" (its clock says 4 days passed) → 8 apps paused 28 s, local whole-box backup written as `vzdump-lxc-9201-2026_10_25-02_26_01` (4.3 GB); a restore test at 02:51 CEST passed. The weekly PBS tier was not due — nothing went to ep0 | side effect of the method (written before) | +| 00:30Z / **first 02:30 (CEST)** | **no database copy** — only the routine jobs | YES (the replay) | +| 00:37–00:41Z | a missed-night catch-up: second drive (8 apps + share) and off-site (dump leg 2 m, 8 apps; retention skipped). The off-site snapshots are **dated 2026-10-25** — 4 days ahead of the real date | side effect of the method | +| 01:00Z | 03:00 CEST → 02:00 CET; the dashboard's ages stay right („36 perce") | YES | +| 01:30:00Z / **second 02:30 (CET)** | **the database copy ran, once**: 4 databases + 8 volume dumps, 1 m 55 s, event `db_dump_completed` | YES (the replay) | + +**Verdict: PASS for the box.** Each daily job runs once on the DST night, at the second 02:30. The hub's double run +(R-1183) is not covered by this test (it is on the real hub, on the real Sunday). + +**Text gap (W30):** inside the repeated hour the backups page shows times without a summer/winter mark: at 02:02 CET it +says „Utolsó teljes mentés 2026. 10. 25. 02:26 (36 perce)" — a time that looks later than the clock. + +**What the method left behind (for #8):** a local whole-box archive and off-site snapshots dated 4 days in the future. +Evidence: `02-box-log-0020-0135Z.txt` (the box's own log ring), the night copier `../../B5-night/logs/`. diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s8-wrong-clock/00-expected.txt b/documentation/audits/rehearsal-2026-10-20/B6-failures/s8-wrong-clock/00-expected.txt new file mode 100644 index 00000000..4f9afee0 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s8-wrong-clock/00-expected.txt @@ -0,0 +1,9 @@ +Written 2026-10-21 ~06:20 UTC, before #8 ran. +Method: NTP stays blocked; clean restart with the RTC at (real − 2 years) for ~12 min; then at (real + 2 years) for +~12 min; then the real time with NTP open again. Clock control: the box's HTTP Date header. +Expected: at −2 y nothing deletes or replaces a newer copy; nothing is „due" that is not; the box and the hub should +say the clock is wrong (if they can see it). At +2 y: every backup looks 2 years old → catch-up runs; off-site +retention must NOT delete the real snapshots (the guard refuses future/too-new plans); a local whole-box archive +dated +2 y is written. After the return: the D3/T11 question — does the whole-box tier go silent because its newest +archive is dated in the future, and does anything (box page, hub) say so? +Note: #7 already left a whole-box archive dated 2026-10-25 (4 days ahead) — the same question in small. diff --git a/documentation/audits/rehearsal-2026-10-20/B9-words/words.md b/documentation/audits/rehearsal-2026-10-20/B9-words/words.md index 4325747d..82eaa3a3 100644 --- a/documentation/audits/rehearsal-2026-10-20/B9-words/words.md +++ b/documentation/audits/rehearsal-2026-10-20/B9-words/words.md @@ -31,3 +31,4 @@ | W27 | an English BROWSER | the whole dashboard stays Hungarian with `en-US` — it follows the household's setting, not the browser | by design (a visitor switch exists for signed-out pages) | — (NOT-A-FINDING: the household's setting wins, `10-localisation.md`) | | W28 | share restore | „Nem állítható vissza (a mappa nincs élő adatmeghajtón): beolvasas." | names a share („beolvasas") the household never created | say what it is (Paperless's scan folder?) or skip it silently | | W29 | storage card after a plain decommission | „Kiváltva — Adatok átköltöztetve ide: . () Időpont: …" | the migration sentence shown with empty blanks when nothing was moved | „Leszerelve, adat nem volt rajta" | +| W30 | backups page inside the repeated hour (B6 #7, 2026-10-25 02:02 CET) | „Utolsó teljes mentés 2026. 10. 25. 02:26 (36 perce)" | the time looks later than the clock: no summer/winter mark during the one repeated hour a year | show „(nyári idő)" / „(téli idő)" only when the time falls in the repeated hour; low priority | diff --git a/documentation/audits/rehearsal-2026-10-20/journal.md b/documentation/audits/rehearsal-2026-10-20/journal.md index 37fa031e..0b0a8b5f 100644 --- a/documentation/audits/rehearsal-2026-10-20/journal.md +++ b/documentation/audits/rehearsal-2026-10-20/journal.md @@ -83,3 +83,4 @@ | 05:36 | B5 | morning snapshot IDENTICAL to the evening one (`B5-night/03-…`) | | 05:27–05:45 | due checks | R-1019 closed (demo-felhom read-only, two channels), R-895 first verdict recorded, R-435 moved to 10-27 (logs gone) | | 05:40 | mail | the nightly DooPlex secret sweep named three rehearsal secrets in session records: OLD_BIND_TOKEN, BIND_TOKEN, SETUP_CODE — my mail reads (full text of the bind and setup mails) carried them in; all three are one-time and already spent. From now on mail is read names-only | +| 05:58–07:13 | B6 ★7 | DST live: the first 02:30 skipped, the second ran once (01:30:00Z box), PASS; side effects of the 4-day jump: a local whole-box archive and off-site snapshots dated 2026-10-25; W30 |