rehearsal 2026-10-20: B6 #1 README, journal to 20:14, B5 night expectations written before the night

2026-10-20 22:18:53 +02:00
parent 4a557f54a5
commit ce8cf2c477
5 changed files with 62 additions and 1 deletions
@@ -239,3 +239,8 @@
 2026-10-20T20:11:06 health=200 0.014048 diskread=4GB
 2026-10-20T20:12:07 health=200 0.011040 diskread=4GB
 2026-10-20T20:13:08 health=200 0.010723 diskread=5GB
+2026-10-20T20:14:10 health=200 0.011760 diskread=5GB
+2026-10-20T20:15:11 health=200 0.009534 diskread=5GB
+2026-10-20T20:16:13 health=200 0.009574 diskread=5GB
+2026-10-20T20:17:15 health=200 0.016393 diskread=5GB
+2026-10-20T20:18:16 health=200 0.009594 diskread=5GB
@@ -464,4 +464,14 @@
 20:11:56Z used=3638MB avail=458MB load1=2.5 cpu=36% immich-server:766/3072MB 0% | mealie:251/1000MB 0% | immich-machine-learning:134/2048MB 0% | felhom-controller:124/4096MB 22% | nextcloud-db:104/512MB 0% | immich-postgres:89/768MB 0% | bookstack-db:82/256MB 0% | traefik:65/4096MB 0%
 20:12:26Z used=3638MB avail=458MB load1=2.34 cpu=5% immich-server:668/3072MB 0% | mealie:214/1000MB 0% | felhom-controller:124/4096MB 9% | immich-postgres:99/768MB 0% | immich-machine-learning:96/2048MB 0% | nextcloud-db:85/512MB 0% | paperless-webserver:83/1280MB 204% | filebrowser:69/256MB 0%
 20:12:56Z used=3638MB avail=458MB load1=2.77 cpu=76% immich-server:668/3072MB 0% | mealie:214/1000MB 0% | felhom-controller:124/4096MB 9% | immich-postgres:99/768MB 0% | immich-machine-learning:96/2048MB 0% | nextcloud-db:85/512MB 0% | paperless-webserver:83/1280MB 204% | filebrowser:69/256MB 0%
-20:13:27Z used=3638MB avail=458MB load1=2.77 cpu=32% immich-server:662/3072MB 0% | paperless-webserver:529/1280MB 86% | mealie:192/1000MB 0% | traefik:113/4096MB 4% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 0% | immich-machine-
+20:13:27Z used=3638MB avail=458MB load1=2.77 cpu=32% immich-server:662/3072MB 0% | paperless-webserver:529/1280MB 86% | mealie:192/1000MB 0% | traefik:113/4096MB 4% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 0% | immich-machine-learning:72/2048MB 0% | filebrowser:65/256MB 0%
+20:13:57Z used=3638MB avail=458MB load1=1.82 cpu=3% immich-server:662/3072MB 0% | paperless-webserver:529/1280MB 86% | mealie:192/1000MB 0% | traefik:113/4096MB 4% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 0% | immich-machine-learning:72/2048MB 0% | filebrowser:65/256MB 0%
+20:14:27Z used=3638MB avail=458MB load1=1.29 cpu=5% immich-server:663/3072MB 0% | paperless-webserver:530/1280MB 0% | mealie:192/1000MB 0% | traefik:113/4096MB 0% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 0% | immich-machine-learning:72/2048MB 0% | nextcloud:71/1024MB 1%
+20:14:57Z used=3638MB avail=458MB load1=1.45 cpu=4% immich-server:663/3072MB 0% | paperless-webserver:530/1280MB 0% | mealie:192/1000MB 0% | traefik:113/4096MB 0% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 0% | immich-machine-learning:72/2048MB 0% | nextcloud:71/1024MB 1%
+20:15:27Z used=3638MB avail=458MB load1=0.88 cpu=6% immich-server:666/3072MB 0% | paperless-webserver:531/1280MB 0% | mealie:192/1000MB 0% | traefik:110/4096MB 0% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 5% | immich-machine-learning:72/2048MB 0% | nextcloud:71/1024MB 0%
+20:15:57Z used=3638MB avail=458MB load1=0.72 cpu=2% immich-server:666/3072MB 0% | paperless-webserver:531/1280MB 0% | mealie:192/1000MB 0% | traefik:110/4096MB 0% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 5% | immich-machine-learning:72/2048MB 0% | nextcloud:71/1024MB 0%
+20:16:27Z used=3638MB avail=458MB load1=0.44 cpu=7% immich-server:666/3072MB 0% | paperless-webserver:531/1280MB 0% | mealie:192/1000MB 0% | traefik:111/4096MB 0% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 5% | immich-machine-learning:72/2048MB 0% | nextcloud:71/1024MB 0%
+20:16:57Z used=3638MB avail=458MB load1=0.72 cpu=8% immich-server:666/3072MB 0% | paperless-webserver:531/1280MB 0% | mealie:192/1000MB 0% | traefik:111/4096MB 0% | immich-postgres:98/768MB 0% | nextcloud-db:76/512MB 5% | immich-machine-learning:72/2048MB 0% | nextcloud:71/1024MB 0%
+20:17:27Z used=3638MB avail=458MB load1=0.49 cpu=5% immich-server:588/3072MB 1% | paperless-webserver:536/1280MB 0% | mealie:189/1000MB 0% | nextcloud:74/1024MB 0% | immich-machine-learning:68/2048MB 0% | immich-postgres:64/768MB 9% | traefik:59/4096MB 0% | actualbudget:59/256MB 0%
+20:17:58Z used=3638MB avail=458MB load1=0.54 cpu=4% immich-server:588/3072MB 1% | paperless-webserver:536/1280MB 0% | mealie:189/1000MB 0% | nextcloud:74/1024MB 0% | immich-machine-learning:68/2048MB 0% | immich-postgres:64/768MB 9% | traefik:59/4096MB 0% | actualbudget:59/256MB 0%
+20:18:28Z used=3638MB avail=458MB load1=0.54 cpu=8% immich-server:667/3072MB 46% | paperless-webserver:536/1280MB 0% | mealie:189/1000MB 0% | felhom-controller:97/4096MB 138% | nextcloud:74/1024MB 0% | immich-machine-learning:68/2048MB 0% | imm
@@ -0,0 +1,21 @@
+B5 — the first night, no hands. Written 2026-10-20 20:25 UTC, BEFORE the night.
+
+Schedule, read from the box's own backups page (local CEST = UTC+2):
+  database copy 02:30 (00:30Z) · local copy (second drive) 03:30 (01:30Z) · off-site 04:15 (02:15Z) ·
+  whole-box backup between 04:30 and 08:30 (02:30Z–06:30Z), to the local store and the PBS server.
+
+No hands: from 21:00Z until the morning nobody acts on the box as the household or as the operator. Two passive
+readers keep running from DooPlex (they change nothing): B4-use/health-log.txt (health every 60 s) and
+B4-use/memwatch.txt (the box's own memory/CPU numbers every 30 s). Nothing else touches Tester4.
+
+Expected, each with a POSITIVE observable to look for in the morning:
+1. Database copy at ~00:30Z for all 8 apps — the backups page „Utolsó adatbázis-mentés" dated tonight; the box log
+   shows the dump lines with tonight's time.
+2. Local copy at ~01:30Z — the second-drive card dated tonight (hdd_2 was removed again in #13; if no second drive
+   is assigned, the page must SAY there is none, not show a success).
+3. Off-site at ~02:15Z — „Távoli mentés" ok with tonight's date; the snapshot list holds one new snapshot per class.
+   Morning: compare per class with the A5 manifest (counts and sha256 of a sample), and check the films share.
+4. Whole-box backup 02:30Z–06:30Z — „Utolsó teljes mentés" dated tonight on local and PBS.
+5. The box does not freeze (R-1192): memwatch keeps answering; health 200 every minute.
+6. Mail: the household gets nothing on a good night; the operator gets nothing about Tester4 on a good night.
+Anything missing from 1–6 is a finding, written with its evidence before any fix or retry.
@@ -0,0 +1,16 @@
+# B6 ★1 — power cut in the middle of an app install (PrivateBin, a throwaway app) — 2026-10-20 20:00–20:05 UTC — AS EXPECTED
+
+Method: endpoint-level install (`POST /api/stacks/privatebin/deploy`, the same request the install page sends), then
+`qm stop 9320` on demo-hp 4 s later (the image pull was running). Power back 20 s later.
+
+| UTC | observed |
+|---|---|
+| 20:00:53 | install accepted (202) with the memory warning; the card shows „not_deployed" while the pull runs |
+| 20:00:57 | plug pulled |
+| 20:01:17 → 20:03:30 | power on → dashboard health 200 (2 min 13 s) |
+| 20:03:25 | the controller: `install privatebin was INTERRUPTED by a controller restart — removing what it started (volumes kept) and reporting it (R-681)` |
+| 20:03:27 | event `app_deploy_failed` to the hub: „Alkalmazás telepítése nem sikerült: PrivateBin — interrupted by a controller restart before it finished — install it again" — the reason part is English (W25 class) |
+| after | PrivateBin not installed, no pins, no half-state; the app page offers a normal install |
+| 20:04–20:05 | installed again: running in 15 s; then stopped and removed (data + backups) through the product — `verified: true` |
+
+**Verdict: PASS.** Clean rollback, an honest event, a normal reinstall. One text gap: the reason in the message is English.
@@ -63,3 +63,12 @@
 | 17:01 | mail | the operator mailbox (second channel) shows host_stale 13:33, app_oom 13:34 (immich-machine-learning), host_crash_restart ×2, node_stale/recovered, backup_failed 17:01; the household got „A biztonsági mentés sikertelen!" — R-1193 corrected; oob_degraded at 12:03 on the fresh box → R-1194 |
 | 17:14–17:42 | B8 | all seven restore paths, see `B8-restores/README.md`; NFC/NFD: both names come back on the share; Nextcloud folds NFD into NFC itself |
 | 17:28–17:36 | B6 #2 | power cut mid full off-site restore, twice: at 6 s safe; at 14 s the app runs on the backup's data while the page says „megszakadt — indítsd el újra" → R-1195 |
+| 17:51 | B6 #9 | two actions at once: remove during a restore → 409 with a clear sentence; double „Visszaállítás" → the second refused; status stayed true — `B6-failures/s9-two-at-once` |
+| 17:52–17:56 | B6 #13 | a second 50 GB drive hot-plugged and initialised as `hdd_2`; **not offered as the backup target** (only the data disk is offered) → R-1198; safe removal worked, the decommission card has blank texts (W29) |
+| ~17:55 | B6 #16 | the dashboard in English: follows the household's setting, not the browser; 15 pages read; a few Hungarian strings left (W26) |
+| 17:56–18:05 | B6 ★4 | data drive pulled during uploads, twice: apps stopped in 42/34 s with a clear card; back by themselves in 62/54 s; the last uploads came back as 0 B or missing → R-1196 |
+| 18:11–18:43 | B6 ★3 | data disk filled to 100 % over SMB: warning at 89 %; „Mentés most" refused before writing; old copies kept; healed after the filler was deleted. The system disk at 97 % not run (reason in README) |
+| 18:43–19:43 | B6 #12 | DNS dead for one hour: dashboard OK; operator got host/node_stale after ~37 min; the household was told nothing; ONLINE 6 min after DNS came back |
+| 19:54–20:00 | B6 ★6 | off-site blocked (shortened): the run failed in 2.5 min with raw English; unblocked → catch-up OK in 2 min 37 s |
+| 20:00–20:05 | B6 ★1 | power cut mid PrivateBin install: clean rollback + an event; reinstall in 15 s; removed again |
+| 20:10–20:14 | B6 #14 | Paperless removed keeping its data, installed again with „use my kept data": 47 of 47 documents back, 47 of 47 originals download |