95 lines
19 KiB
Markdown
95 lines
19 KiB
Markdown
# Rehearsal 2026-10-20 — journal (UTC unless marked; box local = CEST)
|
||
|
||
| UTC | phase | observation |
|
||
|---|---|---|
|
||
| 09:31 | A0 | baselines read; controller moved to v0.336.0 (floor) at 08:54 UTC before the run began — the frozen build is 0.336.0, not the brief's 0.335.0 |
|
||
| 09:33 | A2 | old host Tester4-31fa0e is GONE from /hosts (precondition met) |
|
||
| 09:35 | A2 | Tester4 has NO off-site tier (hub /offsite lists only customers with an enabled descriptor; 4 others listed). Asked the operator to switch it on + to put the owner passphrase into sec |
|
||
| 09:37 | A3 | demo-hp: 15 GiB RAM available, nvme-scratch 697 GB free, local 11 GB free |
|
||
| 09:39 | C1 | four read-only reviewers started on lean clones (reports/register removed from the clones) |
|
||
| 09:40 | B1 | ISO downloaded from the link on felhom.eu/letoltes in 62 s; sha256 dceacae5… = the page = the .sha256 file |
|
||
| 09:42 | B1 | VM 9320 rehearsal1020-box1 created (4 vCPU, 8 GiB, 256 GB + 200 GB raw on nvme-scratch) and started |
|
||
| 09:42 | B1/B9 | GRUB auto-boots „Felhom telepítés / Install Felhom"; text „válassza ki a lemezt" is FORMAL (Ön) while the guide and dashboard say „te" |
|
||
| 09:46:50 | B1 | Install pressed (graphical installer, Hungarian keyboard kept, driven by `tools/vmctl.sh`) |
|
||
| ≤09:51 | B1 | Box rebooted by itself (auto-reboot ticked, the VM boots disk first = „USB pulled"); console shows the pairing code — 4 min after Install. Code stored in sec, blacked out on `screens/install/11-console.png` |
|
||
| 09:51 | A4 | Browser LXC 9330 works: Chromium 155.0.8059.79 (Debian 13 package) + Playwright 1.52.0; phone 390×844 and computer 1366×768, hu-HU and en-US |
|
||
| 09:52 | B1 | FINDING: on felhom.eu/letoltes the ISO link is the white bold FILE NAME — no colour, no underline, no button, on phone and computer (`screens/b1-site/`). No public page links to /letoltes (by design: the guide carries the URL) |
|
||
| 09:55 | B2 | Mailbox: the newest bind link is from 2026-10-18 19:41 UTC (sent by the old host's delete, valid to 10-25); its text says „Elkészült a Felhom dobozod, és készen áll az összekötésre" while no box existed |
|
||
| 09:56 | B2 | the 10-18 17:42 link (used then) → „Ez a hivatkozás érvénytelen vagy lejárt" + an „Új linket kérek" button (not pressed) |
|
||
| 09:56:58 | B2 | bind attempt 1: right pairing code + passphrase with one letter changed → „A megadott adatok nem megfelelőek…" — generic, correct; the page does not say how many of the 5 tries are left |
|
||
| 09:57:09 | B2 | bind attempt 2: the stored value TESTER4_PASSWORD (from 2026-10-17, before Tester4 was re-created 10-18) → refused: it is stale. 2/5 used. **BLOCKED on the operator: the owner passphrase** |
|
||
| 12:00:57 | B2 | the operator's passphrase arrives in sec (intervention 1: an operator act the guide requires — the passphrase is handed over out of band by design) |
|
||
| 12:01:3x | B2 | bind attempt 3 (right code + passphrase) → „Sikeres összekötés" (`screens/b2-bind/`) |
|
||
| 12:02:25 | B2 | mail „[Felhom] Új beállító kód — újratelepült a szervered" (the guide's reinstall subject); code stored in sec |
|
||
| 12:02:43 | B2 | hub: host Tester4-816e21 created (NO REPORT yet); console turns to „a doboz össze van kötve" — the guide's „known bug: the code stays on screen" is FIXED (guide note stale); a stray „V" glyph follows the sentence (R-725 shape) |
|
||
| ≤12:13 | B2 | hub: Tester4-816e21 ONLINE, agent 0.161.0, controller 0.336.0 (the golden 0.331.0 → floor by itself); dashboard guest at 192.168.0.133 (DHCP) |
|
||
| 12:13 | B2 | harness: the household's address `https://felhom.tester4.test` has no DNS/tunnel (tester4.test is reserved); the browser LXC maps the name to the guest's LAN address in /etc/hosts, TLS not verified |
|
||
| 12:14:26 | B2 | claim with a typo in the setup code → „Hibás vagy lejárt kód" |
|
||
| 12:14:4x | B2 | claim with the right code + a 12-character password (exactly the minimum) → 302 → /launcher. **Install pressed → claimed: 27 min 50 s, of which 70 min of waiting for the passphrase are excluded (bind attempt 3 at 12:01)** |
|
||
| 12:15 | B2 | the yellow bar „A távoli mentés szünetel, amíg nem hozod létre a helyreállítási kódot" shows at once; the hub's Offsite page now lists Tester4 (0 B, no quota shown) and a WireGuard peer 10.77.0.6 → the operator switched off-site ON between 12:01 and 12:15 (intervention 2, an operator prerequisite the guide states) |
|
||
| 12:16:22 | B2 | recovery code ceremony through the bar's link: dashboard password → code ready in ~5 s → shown once (10 words) → the page asks words 2 and 7 → accepted → „Befejezés" → /backups/remote „A helyreállítási kód letétbe helyezve" — 16 s in all. The code went page → 0600 file → sec, never printed; every screenshot has the code blacked out (`screens/b2-escrow/`) |
|
||
| 12:16 | B2/B9 | the ceremony page shows technical checks to the household („Mentési tároló (PBS)", „Helyreállítási (DR) szint", „/usr/bin/age", „Rendszerjogosultság"); its step numbers jump 1 → 5 → 6; the remote page shows „restic + SFTP" and the Storage Box login `u629488-sub5@…your-storagebox.de /home/felhom-repo` |
|
||
| 12:17:13 | B2 | six wrong dashboard passwords from a fresh browser: five „Hibás jelszó.", the sixth „Túl sok hibás próbálkozás erről a címről. Próbáld újra egy perc múlva." (1 minute, per address). The guide's „15 percre zárol" is the SETUP page's rule; the login lock is not in the guide |
|
||
| — | B2 | the old (used) bind link: tested 09:56 → „érvénytelen vagy lejárt" + „Új linket kérek" (not pressed) |
|
||
| 12:18:09 | B3 | family card: „Ödön" and „Írisz" refused — „A név 1–32 karakter: kisbetű, szám, pont, kötőjel vagy aláhúzás." → added as odon, irisz, kata; each password answered once (→ sec) |
|
||
| 12:19 | B3 | the 200 GB second disk is NOT used by itself: the box made a 54.8 GB „Belső tárhely" on the system disk; the storage page offers „Új meghajtó inicializálása". The household pressed it: the first press did nothing (the required „Csatlakoztatási név" — „/mnt/<név>" — was empty; only a browser tooltip); with `hdd_1` → „A meghajtó sikeresen inicializálva és regisztrálva", default for new installs (12:19:48) |
|
||
| 12:22–12:55 | B3 | through each app's own install page (real Chromium): **Immich** OK 117 s (2048 MB); **Nextcloud** admin `odon` + a password with spaces and ő/ű: „Időtúllépés — az alkalmazás még indulhat" at ~2 min, running a few minutes later; **Paperless-ngx** admin `irisz` + a 64-character password OK; **Vaultwarden** sign-up switched ON OK; **Mealie** OK; **Actual Budget** OK; **Radicale** OK; **BookStack** OK; **Jellyfin REFUSED**: „Nincs elég memória … telepítés után 4150 MB kellene, a gép 3712 MB-ot tud adni. Távolíts el egy alkalmazást…" (button disabled; no link to a page). 8 of 9 installed; memory meter 3638 / 3712 MB (98 %) |
|
||
| 12:22 | B3 | harness slip: my screenshot step blanked every field before the first Nextcloud press, so the form was sent empty and refused (15 min lost); fixed (values restored after each photo) and re-run |
|
||
| 12:24–12:55 | B9 | „Első lépések" texts: Immich mixes English („Administration → External Libraries → add /external/photos"); Paperless says „felhasználó \"admin\"" although the household set `irisz`; Vaultwarden says sign-up is closed although the household opened it; **Mealie names `mealie.tester4.test` while the app is at `recipes.tester4.test`** |
|
||
| 12:56:33 | B4 | Paperless: 53 PDFs posted through `/api/documents/post_document/` → 53 accepted (processing is the app's background work) |
|
||
| 12:57 | B4 | Radicale: calendar „Család" (MKCALENDAR 201) + 2 events (201, one crossing the DST change); Mealie: Gulyásleves + Csirkepörkölt with ingredients, steps and photos (201/200/200) |
|
||
| 12:57:15 | B4 | Nextcloud: the 3 393-file tree over WebDAV (rclone, 4 transfers) started; rclone keeps only ONE of the NFC/NFD name pair („Duplicate object found in source") and skips the symlink — tool limits, not the box's; the 1.5 GiB file FAILED at 12:57:27 with an HTML error page |
|
||
| 12:58–12:59 | B4 | Immich: the setup gate shows a stranger „Ez az alkalmazás még beállításra vár"; Ödön registered as admin through Immich's own form; the gate opened by itself within 20 s |
|
||
| 12:59:52 | B4 | Immich upload through `/api/assets`: the FIRST photo hangs 300 s and times out |
|
||
| ~13:05 → | B4 | **THE BOX STOPS ANSWERING.** The dashboard's `/api/health` gives no answer in 10–15 s; the VM reads ~4 GB/min from its disk (158 GB → 209 GB in 15 min) while network traffic stands still; ping fine. 13:12 the upload stopped (the family gives up) — no recovery 10 min later. **The hub reads the box ONLINE / OK all along** (host memory 39 %, controller OK, 13:22). Likely: the apps' own background work after the uploads (Paperless OCR of 53 PDFs, Immich's analysis, Nextcloud) on a guest booked to 98 % of its memory — NOT measured (no shell in the box) |
|
||
| 13:24 | B4 | no inside view of the box — tried: root SSH with the install-time password → „Permission denied" (replaced at step 4b, by design); DooPlex's key → denied; the hub-vaulted recovery password → operator-only reveal (the token is refused); the agent's local API token → only on the box. So the stall is described from outside only |
|
||
| 14:00:01 | B4 | the box answers again by itself (~55 min after the freeze began) — and freezes again within minutes (14:05–14:10, 4 GB/min of disk reads) |
|
||
| 14:11:06 | B4 | **the family pulls the plug** (VM power off 15 s, on) → `/api/health` 200 at 14:13:43 (2 min 22 s); the box's own history: load 139–152 at 13:33/13:55, no samples 12:57–13:33; the guest has 4096 MB in all, 3638 MB booked by the 8 apps |
|
||
| 14:23 | B4 | what survived: Paperless 47 documents (8 failed: the 100-page, the password-protected, the scanned image-only PDF, and 5 ordinary invoices); Immich 149 photos; Nextcloud 251 of 3 393 files |
|
||
| 14:24–15:20 | B4 | gentle resume (one transfer at a time): Nextcloud tree complete except the 1.5 GiB file; health stayed 200 for 60 min |
|
||
| 14:42–14:58 | B4 | Vaultwarden: Ödön's account through the web vault (the page then shows an English extension ad); 50 logins in 4 folders + 2 attachments through the official `bw` CLI (2024.9 was refused 422 — the CLI must match the server; 2026.7.0 worked). Folder SHARED between two members: NOT done (needs an organisation). BookStack: API token made on the admin's own page; a book + 5 pages, each with an image. Actual Budget (English-only UI, v26.9.0): server password, „Start budgeting", account „Családi számla", 200 transactions through its CSV dialog → −17 326 000 Ft = the CSV's sum |
|
||
| 15:00 | B4 | phone + computer walk: launcher, dashboard, app pages and „Útmutató" of Immich, Paperless, Vaultwarden, backups page — no sideways scroll; the backups page says honestly that it cannot prove the second drive is another disk (in a VM) |
|
||
| 15:11–15:24 | B4 | network share: the first „switch on" was a harness slip; then „Beállítás mentve. A megosztási szolgáltatás előkészítése folyamatban…" — but the service waits for the share PASSWORD (log: „deploy skipped — household SMB password not set yet") and the page never says so; password set → „Állapot: fut". The new share „Filmek" on the data disk has **Felhőmentés ON by default** |
|
||
| 15:26 | B4 | the two films (1.9 GB) copied onto the share over SMB in 35 s |
|
||
| 15:27 | B4 | Nextcloud 1.5 GiB file: a single WebDAV PUT is refused 413 „Request Entity Too Large" after 2 MB (a Windows/Mac WebDAV drive mount would fail on any file over ~1 GB); chunked upload started |
|
||
| 15:27:55 | B4 | **SECOND FREEZE** — one Immich upload + one chunked Nextcloud upload running; the Immich upload times out; disk reads since the reboot 5 GB → 91 GB. Uploads stopped by the family |
|
||
| 15:28–16:02 | B4 | the second freeze lasted ~35 min; back 16:02:49 — and froze again within a minute (the family's next look: an Immich login + a Nextcloud file count) |
|
||
| 16:13:36 | B4 | **plug pulled a second time** → health 200 at 16:16:06 (2 min 16 s). From now on `B4-use/memwatch.txt` records the box's own per-app memory/CPU every 30 s and `B4-use/health-log.txt` the health every minute |
|
||
| 16:16–16:28 | B4 | after the reboot: Immich server jumps to ~1.1 GB, Paperless to ~550 MB (re-reading its PDFs), load ~4.6 → calm (< 1) by 16:20; the real per-app total ~2.5 GB of the guest's 4 GB |
|
||
| 16:28:59 | B4 | „Mentés most" (guide §9): 2 min (the guide says ~20 s for two apps), 4 database dumps, success |
|
||
| 16:31:28 | B6 #5 | `tc netem` on the VM's own tap (demo-hp): upload 1 Mbit/s + 5 % loss both ways — harness note: this slows the LAN answers too, which a real slow internet line would not |
|
||
| 16:31:37 | B6 #5 | „Távoli mentés most" pressed (all 8 apps are ON for off-site by default; the share „Filmek" too) → status running, phase dump. The page also shows the household an SSH-key / known_hosts form („no blind TOFU", „ssh-keyscan") |
|
||
| 17:01–17:08 | B6 #5 | see `B6-failures/s5-slow-net/README.md`: run failed (Immich) under 1 Mbit/s + 5 % loss; the second press at full speed ok in 3 min 38 s |
|
||
| 17:01 | mail | the operator mailbox (second channel) shows host_stale 13:33, app_oom 13:34 (immich-machine-learning), host_crash_restart ×2, node_stale/recovered, backup_failed 17:01; the household got „A biztonsági mentés sikertelen!" — R-1193 corrected; oob_degraded at 12:03 on the fresh box → R-1194 |
|
||
| 17:14–17:42 | B8 | all seven restore paths, see `B8-restores/README.md`; NFC/NFD: both names come back on the share; Nextcloud folds NFD into NFC itself |
|
||
| 17:28–17:36 | B6 #2 | power cut mid full off-site restore, twice: at 6 s safe; at 14 s the app runs on the backup's data while the page says „megszakadt — indítsd el újra" → R-1195 |
|
||
| 17:51 | B6 #9 | two actions at once: remove during a restore → 409 with a clear sentence; double „Visszaállítás" → the second refused; status stayed true — `B6-failures/s9-two-at-once` |
|
||
| 17:52–17:56 | B6 #13 | a second 50 GB drive hot-plugged and initialised as `hdd_2`; **not offered as the backup target** (only the data disk is offered) → R-1198; safe removal worked, the decommission card has blank texts (W29) |
|
||
| ~17:55 | B6 #16 | the dashboard in English: follows the household's setting, not the browser; 15 pages read; a few Hungarian strings left (W26) |
|
||
| 17:56–18:05 | B6 ★4 | data drive pulled during uploads, twice: apps stopped in 42/34 s with a clear card; back by themselves in 62/54 s; the last uploads came back as 0 B or missing → R-1196 |
|
||
| 18:11–18:43 | B6 ★3 | data disk filled to 100 % over SMB: warning at 89 %; „Mentés most" refused before writing; old copies kept; healed after the filler was deleted. The system disk at 97 % not run (reason in README) |
|
||
| 18:43–19:43 | B6 #12 | DNS dead for one hour: dashboard OK; operator got host/node_stale after ~37 min; the household was told nothing; ONLINE 6 min after DNS came back |
|
||
| 19:54–20:00 | B6 ★6 | off-site blocked (shortened): the run failed in 2.5 min with raw English; unblocked → catch-up OK in 2 min 37 s |
|
||
| 20:00–20:05 | B6 ★1 | power cut mid PrivateBin install: clean rollback + an event; reinstall in 15 s; removed again |
|
||
| 20:10–20:14 | B6 #14 | Paperless removed keeping its data, installed again with „use my kept data": 47 of 47 documents back, 47 of 47 originals download |
|
||
| 00:30–00:33:27 | B5 | night database copy: 8 apps, 4 databases (55 MB) + 8 volume dumps, recovery units captured, 3 m 27 s, event `db_dump_completed` — from the copied log ring (`B5-night/logs/`) |
|
||
| 00:42 | harness | my slip: from 20:24 the two watcher files in the repo were frozen — `git pull --autostash` replaced them while the watchers kept writing into the old (deleted) files. Recovered in full from the watchers' open files (`/proc/<pid>/fd/1`); the watchers restarted writing OUTSIDE the repo and are copied in at each commit. Nothing lost. Health 200 all evening except the deliberate power cuts (17:30–17:35, 20:01–20:02) |
|
||
| 00:45 | harness | push refused by the due-checks gate: R-895, R-435, R-1019 came due on 2026-10-21 (other work; their events are at 01:30 and 05:10 UTC today, so not measurable at 00:45). Commits stay local on DooPlex until the morning; then measured read-only or moved with a reason. No `--no-verify` |
|
||
| 01:30–01:30:58 | B5 | second-drive copy: 8 apps + the share (1.8 GB), PASS |
|
||
| 02:15–02:18:11 | B5 | off-site: 8 apps + share, 44 snapshots, 3 m 6 s, nothing deleted, PASS (the hub's own 05:10 listing: 44 files, newest 02:18Z) |
|
||
| 02:30 | B5 | whole-box window: not due (24 h rule from the install's 12:09Z backup) → R-1200 |
|
||
| 02:45:59 | harness | demo-hp's own kernel step rebooted the HOST; it shut the rehearsal VM down cleanly (no auto-start) — `B5-night/02-host-reboot.txt`; the operator got Tester4 host/node stale + down mails (expected, caused by the harness) |
|
||
| 05:24–05:26 | B5 | VM started by me; dashboard 200 in 39 s; all 8 apps running at 05:26 |
|
||
| 05:36 | B5 | morning snapshot IDENTICAL to the evening one (`B5-night/03-…`) |
|
||
| 05:27–05:45 | due checks | R-1019 closed (demo-felhom read-only, two channels), R-895 first verdict recorded, R-435 moved to 10-27 (logs gone) |
|
||
| 05:40 | mail | the nightly DooPlex secret sweep named three rehearsal secrets in session records: OLD_BIND_TOKEN, BIND_TOKEN, SETUP_CODE — my mail reads (full text of the bind and setup mails) carried them in; all three are one-time and already spent. From now on mail is read names-only |
|
||
| 05:58–07:13 | B6 ★7 | DST live: the first 02:30 skipped, the second ran once (01:30:00Z box), PASS; side effects of the 4-day jump: a local whole-box archive and off-site snapshots dated 2026-10-25; W30 |
|
||
| 07:15–07:58 | B6 ★8 | −2 y (OS raised it to 2026-04-13): „Hub nem elérhető" only, Paperless stopped silently → R-1202; +2 y: whole-box archive dated 2028; back to real time: ep0 WireGuard tunnel dead (handshake replay) → R-1204 (P2); R-1189 (a) half-live |
|
||
| 07:57–08:45 | B6 ★10 + #15 | 3 days off (box clock 2026-10-28): ONE catch-up, clear household note; a restart mid-catch-up recovered (marker, drive gate, one new catch-up); 8 apps off-site; PASS |
|
||
| 08:44:37 | B7 | the box dies: hard stop, `qm destroy 9320 --purge` (own VM, own disks) |
|
||
| 08:44:55–08:53 | B7 | reinstall from the same public ISO, the guide as written; pairing screen after ~5 min. The pairing code was OCR'd into sec (two reads agreed) and blacked out in the screenshot |
|
||
| ~08:54 | B7 | the forgotten press: the spent link → „Új linket kérek" → „Kész … néhány percen belül új e-mailt kapsz"; NO mail in 4 min (control: older mails found) → R-1203 (P2) |
|
||
| 08:59 | B7 | the operator asked (in the terminal) to delete host Tester4-816e21 |
|
||
| ~09:11 | B7 | the operator: „Host deleted" (intervention 3; asked 08:59). The hub mailed the household a new bind link at 09:11:21 by itself |
|
||
| 09:12–09:40 | B7 | bind 09:12, setup mail 09:13, guest 09:33, claim 09:35; the rebuilt dashboard offers only to CREATE a recovery code and cannot read the off-site store → the household cannot restore (R-1176 proven live); the new code NOT created |
|