From ce8cf2c4779ed476b9e335fb55b6273781bbc104 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 20 Oct 2026 22:18:53 +0200 Subject: [PATCH] rehearsal 2026-10-20: B6 #1 README, journal to 20:14, B5 night expectations written before the night --- .../B4-use/health-log.txt | 5 +++++ .../rehearsal-2026-10-20/B4-use/memwatch.txt | 12 ++++++++++- .../B5-night/00-expected.txt | 21 +++++++++++++++++++ .../s1-power-cut-mid-install/README.md | 16 ++++++++++++++ .../audits/rehearsal-2026-10-20/journal.md | 9 ++++++++ 5 files changed, 62 insertions(+), 1 deletion(-) create mode 100644 documentation/audits/rehearsal-2026-10-20/B5-night/00-expected.txt create mode 100644 documentation/audits/rehearsal-2026-10-20/B6-failures/s1-power-cut-mid-install/README.md diff --git a/documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt b/documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt index d1e441be..b3e3c8b1 100644 --- a/documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt +++ b/documentation/audits/rehearsal-2026-10-20/B4-use/health-log.txt @@ -239,3 +239,8 @@ 2026-10-20T20:11:06 health=200 0.014048 diskread=4GB 2026-10-20T20:12:07 health=200 0.011040 diskread=4GB 2026-10-20T20:13:08 health=200 0.010723 diskread=5GB +2026-10-20T20:14:10 health=200 0.011760 diskread=5GB +2026-10-20T20:15:11 health=200 0.009534 diskread=5GB +2026-10-20T20:16:13 health=200 0.009574 diskread=5GB +2026-10-20T20:17:15 health=200 0.016393 diskread=5GB +2026-10-20T20:18:16 health=200 0.009594 diskread=5GB diff --git a/documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt b/documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt index 573e6b63..e738a7ab 100644 --- a/documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt +++ b/documentation/audits/rehearsal-2026-10-20/B4-use/memwatch.txt @@ -464,4 +464,14 @@ 20:11:56Z used=3638MB avail=458MB load1=2.5 cpu=36% immich-server:766/3072MB 0% | mealie:251/1000MB 0% | immich-machine-learning:134/2048MB 0% | felhom-controller:124/4096MB 22% | nextcloud-db:104/512MB 0% | immich-postgres:89/768MB 0% | bookstack-db:82/256MB 0% | traefik:65/4096MB 0% 20:12:26Z used=3638MB avail=458MB load1=2.34 cpu=5% immich-server:668/3072MB 0% | mealie:214/1000MB 0% | felhom-controller:124/4096MB 9% | immich-postgres:99/768MB 0% | immich-machine-learning:96/2048MB 0% | nextcloud-db:85/512MB 0% | paperless-webserver:83/1280MB 204% | filebrowser:69/256MB 0% 20:12:56Z used=3638MB avail=458MB load1=2.77 cpu=76% immich-server:668/3072MB 0% | mealie:214/1000MB 0% | felhom-controller:124/4096MB 9% | immich-postgres:99/768MB 0% | immich-machine-learning:96/2048MB 0% | nextcloud-db:85/512MB 0% | paperless-webserver:83/1280MB 204% | filebrowser:69/256MB 0% -20:13:27Z used=3638MB avail=458MB load1=2.77 cpu=32% immich-server:662/3072MB 0% | paperless-webserver:529/1280MB 86% | mealie:192/1000MB 0% | traefik:113/4096MB 4% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 0% | immich-machine- \ No newline at end of file +20:13:27Z used=3638MB avail=458MB load1=2.77 cpu=32% immich-server:662/3072MB 0% | paperless-webserver:529/1280MB 86% | mealie:192/1000MB 0% | traefik:113/4096MB 4% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 0% | immich-machine-learning:72/2048MB 0% | filebrowser:65/256MB 0% +20:13:57Z used=3638MB avail=458MB load1=1.82 cpu=3% immich-server:662/3072MB 0% | paperless-webserver:529/1280MB 86% | mealie:192/1000MB 0% | traefik:113/4096MB 4% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 0% | immich-machine-learning:72/2048MB 0% | filebrowser:65/256MB 0% +20:14:27Z used=3638MB avail=458MB load1=1.29 cpu=5% immich-server:663/3072MB 0% | paperless-webserver:530/1280MB 0% | mealie:192/1000MB 0% | traefik:113/4096MB 0% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 0% | immich-machine-learning:72/2048MB 0% | nextcloud:71/1024MB 1% +20:14:57Z used=3638MB avail=458MB load1=1.45 cpu=4% immich-server:663/3072MB 0% | paperless-webserver:530/1280MB 0% | mealie:192/1000MB 0% | traefik:113/4096MB 0% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 0% | immich-machine-learning:72/2048MB 0% | nextcloud:71/1024MB 1% +20:15:27Z used=3638MB avail=458MB load1=0.88 cpu=6% immich-server:666/3072MB 0% | paperless-webserver:531/1280MB 0% | mealie:192/1000MB 0% | traefik:110/4096MB 0% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 5% | immich-machine-learning:72/2048MB 0% | nextcloud:71/1024MB 0% +20:15:57Z used=3638MB avail=458MB load1=0.72 cpu=2% immich-server:666/3072MB 0% | paperless-webserver:531/1280MB 0% | mealie:192/1000MB 0% | traefik:110/4096MB 0% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 5% | immich-machine-learning:72/2048MB 0% | nextcloud:71/1024MB 0% +20:16:27Z used=3638MB avail=458MB load1=0.44 cpu=7% immich-server:666/3072MB 0% | paperless-webserver:531/1280MB 0% | mealie:192/1000MB 0% | traefik:111/4096MB 0% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 5% | immich-machine-learning:72/2048MB 0% | nextcloud:71/1024MB 0% +20:16:57Z used=3638MB avail=458MB load1=0.72 cpu=8% immich-server:666/3072MB 0% | paperless-webserver:531/1280MB 0% | mealie:192/1000MB 0% | traefik:111/4096MB 0% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 5% | immich-machine-learning:72/2048MB 0% | nextcloud:71/1024MB 0% +20:17:27Z used=3638MB avail=458MB load1=0.49 cpu=5% immich-server:588/3072MB 1% | paperless-webserver:536/1280MB 0% | mealie:189/1000MB 0% | nextcloud:74/1024MB 0% | immich-machine-learning:68/2048MB 0% | immich-postgres:64/768MB 9% | traefik:59/4096MB 0% | actualbudget:59/256MB 0% +20:17:58Z used=3638MB avail=458MB load1=0.54 cpu=4% immich-server:588/3072MB 1% | paperless-webserver:536/1280MB 0% | mealie:189/1000MB 0% | nextcloud:74/1024MB 0% | immich-machine-learning:68/2048MB 0% | immich-postgres:64/768MB 9% | traefik:59/4096MB 0% | actualbudget:59/256MB 0% +20:18:28Z used=3638MB avail=458MB load1=0.54 cpu=8% immich-server:667/3072MB 46% | paperless-webserver:536/1280MB 0% | mealie:189/1000MB 0% | felhom-controller:97/4096MB 138% | nextcloud:74/1024MB 0% | immich-machine-learning:68/2048MB 0% | imm \ No newline at end of file diff --git a/documentation/audits/rehearsal-2026-10-20/B5-night/00-expected.txt b/documentation/audits/rehearsal-2026-10-20/B5-night/00-expected.txt new file mode 100644 index 00000000..0ff7b6c1 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B5-night/00-expected.txt @@ -0,0 +1,21 @@ +B5 — the first night, no hands. Written 2026-10-20 20:25 UTC, BEFORE the night. + +Schedule, read from the box's own backups page (local CEST = UTC+2): + database copy 02:30 (00:30Z) · local copy (second drive) 03:30 (01:30Z) · off-site 04:15 (02:15Z) · + whole-box backup between 04:30 and 08:30 (02:30Z–06:30Z), to the local store and the PBS server. + +No hands: from 21:00Z until the morning nobody acts on the box as the household or as the operator. Two passive +readers keep running from DooPlex (they change nothing): B4-use/health-log.txt (health every 60 s) and +B4-use/memwatch.txt (the box's own memory/CPU numbers every 30 s). Nothing else touches Tester4. + +Expected, each with a POSITIVE observable to look for in the morning: +1. Database copy at ~00:30Z for all 8 apps — the backups page „Utolsó adatbázis-mentés" dated tonight; the box log + shows the dump lines with tonight's time. +2. Local copy at ~01:30Z — the second-drive card dated tonight (hdd_2 was removed again in #13; if no second drive + is assigned, the page must SAY there is none, not show a success). +3. Off-site at ~02:15Z — „Távoli mentés" ok with tonight's date; the snapshot list holds one new snapshot per class. + Morning: compare per class with the A5 manifest (counts and sha256 of a sample), and check the films share. +4. Whole-box backup 02:30Z–06:30Z — „Utolsó teljes mentés" dated tonight on local and PBS. +5. The box does not freeze (R-1192): memwatch keeps answering; health 200 every minute. +6. Mail: the household gets nothing on a good night; the operator gets nothing about Tester4 on a good night. +Anything missing from 1–6 is a finding, written with its evidence before any fix or retry. diff --git a/documentation/audits/rehearsal-2026-10-20/B6-failures/s1-power-cut-mid-install/README.md b/documentation/audits/rehearsal-2026-10-20/B6-failures/s1-power-cut-mid-install/README.md new file mode 100644 index 00000000..51763f14 --- /dev/null +++ b/documentation/audits/rehearsal-2026-10-20/B6-failures/s1-power-cut-mid-install/README.md @@ -0,0 +1,16 @@ +# B6 ★1 — power cut in the middle of an app install (PrivateBin, a throwaway app) — 2026-10-20 20:00–20:05 UTC — AS EXPECTED + +Method: endpoint-level install (`POST /api/stacks/privatebin/deploy`, the same request the install page sends), then +`qm stop 9320` on demo-hp 4 s later (the image pull was running). Power back 20 s later. + +| UTC | observed | +|---|---| +| 20:00:53 | install accepted (202) with the memory warning; the card shows „not_deployed" while the pull runs | +| 20:00:57 | plug pulled | +| 20:01:17 → 20:03:30 | power on → dashboard health 200 (2 min 13 s) | +| 20:03:25 | the controller: `install privatebin was INTERRUPTED by a controller restart — removing what it started (volumes kept) and reporting it (R-681)` | +| 20:03:27 | event `app_deploy_failed` to the hub: „Alkalmazás telepítése nem sikerült: PrivateBin — interrupted by a controller restart before it finished — install it again" — the reason part is English (W25 class) | +| after | PrivateBin not installed, no pins, no half-state; the app page offers a normal install | +| 20:04–20:05 | installed again: running in 15 s; then stopped and removed (data + backups) through the product — `verified: true` | + +**Verdict: PASS.** Clean rollback, an honest event, a normal reinstall. One text gap: the reason in the message is English. diff --git a/documentation/audits/rehearsal-2026-10-20/journal.md b/documentation/audits/rehearsal-2026-10-20/journal.md index c4e6d7cd..974b60cd 100644 --- a/documentation/audits/rehearsal-2026-10-20/journal.md +++ b/documentation/audits/rehearsal-2026-10-20/journal.md @@ -63,3 +63,12 @@ | 17:01 | mail | the operator mailbox (second channel) shows host_stale 13:33, app_oom 13:34 (immich-machine-learning), host_crash_restart ×2, node_stale/recovered, backup_failed 17:01; the household got „A biztonsági mentés sikertelen!" — R-1193 corrected; oob_degraded at 12:03 on the fresh box → R-1194 | | 17:14–17:42 | B8 | all seven restore paths, see `B8-restores/README.md`; NFC/NFD: both names come back on the share; Nextcloud folds NFD into NFC itself | | 17:28–17:36 | B6 #2 | power cut mid full off-site restore, twice: at 6 s safe; at 14 s the app runs on the backup's data while the page says „megszakadt — indítsd el újra" → R-1195 | +| 17:51 | B6 #9 | two actions at once: remove during a restore → 409 with a clear sentence; double „Visszaállítás" → the second refused; status stayed true — `B6-failures/s9-two-at-once` | +| 17:52–17:56 | B6 #13 | a second 50 GB drive hot-plugged and initialised as `hdd_2`; **not offered as the backup target** (only the data disk is offered) → R-1198; safe removal worked, the decommission card has blank texts (W29) | +| ~17:55 | B6 #16 | the dashboard in English: follows the household's setting, not the browser; 15 pages read; a few Hungarian strings left (W26) | +| 17:56–18:05 | B6 ★4 | data drive pulled during uploads, twice: apps stopped in 42/34 s with a clear card; back by themselves in 62/54 s; the last uploads came back as 0 B or missing → R-1196 | +| 18:11–18:43 | B6 ★3 | data disk filled to 100 % over SMB: warning at 89 %; „Mentés most" refused before writing; old copies kept; healed after the filler was deleted. The system disk at 97 % not run (reason in README) | +| 18:43–19:43 | B6 #12 | DNS dead for one hour: dashboard OK; operator got host/node_stale after ~37 min; the household was told nothing; ONLINE 6 min after DNS came back | +| 19:54–20:00 | B6 ★6 | off-site blocked (shortened): the run failed in 2.5 min with raw English; unblocked → catch-up OK in 2 min 37 s | +| 20:00–20:05 | B6 ★1 | power cut mid PrivateBin install: clean rollback + an event; reinstall in 15 s; removed again | +| 20:10–20:14 | B6 #14 | Paperless removed keeping its data, installed again with „use my kept data": 47 of 47 documents back, 47 of 47 originals download |