diff --git a/documentation/audits/evidence-bignight-2026-09-14/journal.md b/documentation/audits/evidence-bignight-2026-09-14/journal.md new file mode 100644 index 00000000..52a50db4 --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/journal.md @@ -0,0 +1,125 @@ +# Journal — BIGNIGHT 2026-09-14 (a household's first month in one night) + +Times UTC unless marked. Hub logs print CEST (UTC+2). Brief: `drills/BIGNIGHT-2026-09-14.md`. + +## Phase 1 — baselines (read 17:29–17:40Z from live source) + +| repo | main == origin/main | version | +|---|---|---| +| felhom-controller | `406755fa8fba` | v0.242.0 (CHANGELOG top) | +| felhom-agent | `4586f0f7f6d1` | v0.130.0 (tree: one untracked `scripts/__pycache__/`, no tracked change) | +| felhom.eu | `a4d684412b49` | hub v0.113.0 (CHANGELOG top; live image `felhom-hub:0.113.0`) | +| app-catalog-felhom.eu | `6d6eec307939` | — | + +Register: highest id R-508; 221 table rows (grep `^| R-n`). + +ISO 1.27.1: `/mnt/5_hdd/felhom.eu/felhom-iso/out/felhom-installer-1.27.1-pve9.2-1.iso`, 1 705 322 496 B, +sha256 `25637007d5a7120ff9faa6b5b7ead3e33c0a361ac2d67e9fd4e0ee77c034c053` = `.sha256` = manifest +`output-sha256`; manifest `repo-commit 27e8ec86…`, mode release, answer-file NONE. **Found, not rebuilt.** + +DooPlex headroom: `/mnt/5_hdd` 37 %, `/` 52 %. + +demo-hp (17:29Z): `qm list` empty; 9201 + 9202 running (each `memory: 25898`, `cores: 7`); +host RAM 29 994 MB total, 24 652 MB available; 8 threads (Ryzen V1756B); storages `local` 56.96 %, +`local-lvm` 44.17 %, `nvme-scratch` (dir `/mnt/hdd_1`, is_mountpoint yes) 1.03 %. + +Gmail connector: authenticates as **`felhom.eu@gmail.com`** — the catch-all for `@felhom.eu` +(threads to `admin@felhom.eu` and `drill0242@felhom.eu` land there). Read only; nothing sent. + +## Phase 1 — the Tester 1 record (hub `GET /customers/tester-1`, Basic auth, 17:30Z) + +Customer ID `tester-1` · Name „Tester 1" · Domain `enkicsifelhom.hu` · **Email `tester1@felhom.eu`** · +Config MANAGED. Status tile still reads `down`, „Last report 1h ago · Controller 0.242.0" — the stale +report of the destroyed VM 331 (host record deleted 16:46:52Z). Claim: „Claimed 1h ago · generation 1". +Edit tab: `dr_tier` **ON** (all four DR steps „waiting — no host enrolled yet"); **`offsite_enabled` +OFF** („Enable offsite (provisions a Hetzner Storage Box on save)"). Controller floor v0.242.0. +Mailbox: `to:tester1@felhom.eu` → 0 threads before tonight. + +**Brief vs record — off-site.** The brief says "Off-site (Tier 3) is ON … its own namespace on ep0". +On the record, the ep0 namespace is the **DR tier (PBS whole-guest)**; the **Tier-3 restic off-site +is OFF**, and turning it on provisions a Hetzner Storage Box (money, and a new external resource). +**Not followed:** I do not tick it — money is fenced by the rules file §1. "Off-site" tonight = the DR +tier to ep0 as configured. Consequence, stated up front: the Phase 4 off-site integrity check and the +Phase 6 app restore "from off-site" have no restic tier to act on; each is recorded as such when reached. + +**Harness slip:** the first read of the customer page printed the page's text into this session's +tool output, which includes the customer's hub **API key** (not the retrieval passphrase — that is +masked). It stayed on DooPlex; not written to any file. + +## Tunnel before any box (17:31:19Z, from DooPlex) — `tunnel-before-no-box.txt` + +DNS resolves to Cloudflare (both resolvers); `https://felhom.enkicsifelhom.hu` → **530, `error code: 1033`** +(no connector). Expected with no box. + +## Venue — VM 333 `bignight-household` on demo-hp (created 17:31:52Z) + +q35 / OVMF (`pre-enrolled-keys=0`), **4 cores**, `cpu=host`, **16 384 MB**, virtio NIC +`BC:24:11:F2:E2:95` on vmbr0, `scsi0` **200 G** qcow2 on `nvme-scratch` (dir at `/mnt/hdd_1`, its root), +ISO 1.27.1 on `ide2` (sha verified on the HP = `25637007…`). **One disk at install**; the data disk is +added after. + +**Why these numbers.** System disk = doorstep VM 331's 200 G. Memory: the HP has 29 994 MB; 9201 and +9202 together used ≈ 5.3 GB at 17:29Z with 24 652 MB available. Both LXCs carry a 25 898 MB *limit*, +so no VM size leaves both limits whole; 16 GB leaves ≈ 8 GB free plus 8 GB swap for 9201's real load. +Cores: the 0242 drill's 4 of 8 threads. + +Power-on **17:32:13Z**. GRUB (`s01`, `s02`): the two Felhom entries; **text mode** chosen (H4). + +## Phase 2.1 — install from 1.27.1 (text mode), one disk + +| UTC | screen | what it said / what was done | +|---|---|---| +| 17:34:5x | `s03` | English Proxmox EULA → „I agree" | +| 17:35:11 | `s04` | „Target harddisk: /dev/sda (QEMU HARDDISK) (200.00 GiB)" — **one disk, no choice to make** → Next | +| 17:35:24 | `s05` | Country Hungary · Timezone Europe/Budapest · Keyboard layout **Hungarian** (H2: changed to U.S. English, `s06`–`s09`; one dropped keystroke landed on „Turkish" first, corrected) | +| 17:37:00 | `s11` | „Root password [at least 8 characters]" · Confirm · „Administrator email" **prefilled `mail@example.invalid`** → 20-char generated password (0600 file on DooPlex), `tester1@felhom.eu` (the guide: „a saját e-mail címedet") | +| 17:39:01 | `s13` | nic0 `bc:24:11:f2:e2:95, virtio_net` · Hostname **`pve.example.invalid`** · IP `192.168.0.136`/24 (the DHCP lease, frozen static) · GW `192.168.0.1` · DNS `192.168.0.250` · „[X] Pin network interface names" → hostname set `felhom.enkicsifelhom.hu` per the guide (`s15`) | +| 17:40:36 | `s16` | Summary: ext4 · /dev/sda · Europe/Budapest · U.S. English · tester1@felhom.eu · nic0 · felhom.enkicsifelhom.hu · 192.168.0.136/24 · 192.168.0.1 · 192.168.0.250 · „[X] Automatically reboot after successful installation" (H3: unticked, `s17`) | +| 17:41:02 | — | Install pressed | +| ≤17:43:47 | `s18` | „Success — Installation finished - reboot now?" (**≤ 2 m 45 s** copying; qcow2 4 058 MB, settled) | +| 17:44:xx | first boot | H3: `qm stop`, `--delete ide2`, `--boot order=scsi0`, `qm start` | + +Seeds prepared on DooPlex (scratchpad, tmpfs): 200 JPEG 1600×1200 noise + caption (264 MB), 20 three-page +PDFs (2.3 MB), a 50 MB random file, a 150 s 1280×720 H.264 video (98.9 MB). + +## Phase 2.2 — first screen, and waiting for the mail + +**17:44:53Z — hub lists Unclaimed appliance 28** (38 s after power-on): smbios uuid `1ecd1c40…`, pairing +code `***-***` (redacted), MAC `bc:24:11:f2:e2:95`, „Standard PC (Q35 + ICH9, 2009)", 15.6 GB, three SSH +host keys. Bind list offers „Tester 1 (0 hosts)". + +**The console, 17:45:31Z (`s19`, code redacted), verbatim:** + +``` + Felhom otthoni szerver + + Ezen a gépen most nincs dolgod, és bejelentkezni sem kell. + A beállításhoz kövesd a Felhomtól kapott útmutatót. + +felhom login: +============================================== + Felhom — a doboz készen áll, és a párosításra vár. + + Párosító kód: ***-*** + + Nyisd meg az e-mailben kapott linket, és add meg + ezt a kódot és a Tulajdonosi jelmondatodat + (az 5 szót a Felhom üzemeltetőjétől kaptad). + + Ez a képernyő magától frissül — nincs teendő a + doboznál, és nyugodtan itt hagyhatod bekapcsolva. +============================================== +``` + +No Proxmox `:8006` line on the first boot (the 1.27.1 fix holds on a fresh install). Text says +„otthoni szerver"; the guide §3 quotes „Ezen a képernyőn nincs teendőd" — **the guide's quote does not +match the screen word for word** (meaning is the same). + +**Mailbox, 17:46Z:** `to:tester1@felhom.eu` → 0 threads. The console asks for „az e-mailben kapott linket". +Hub source: the self-bind link is auto-sent only at **customer creation** (`configs.go:725`) and at +**RESET completion** (`customer_reset.go:162`); `tester-1` was created 32 days ago with no e-mail, so no +link was ever sent to `tester1@felhom.eu`. Waiting the brief's 10 minutes (to 17:55Z) before acting. + +**17:55:11Z — ten minutes, 0 messages to `tester1@felhom.eu`.** Filed **R-509 (P1)** at 17:56Z, before acting. +**Intervention I1:** the operator's „Send self-bind link" button on the customer's Setup tab is pressed +(a volunteer cannot press it). diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s01-grub.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s01-grub.png new file mode 100644 index 00000000..c51e16bd Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s01-grub.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s02-grub-tui-selected.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s02-grub-tui-selected.png new file mode 100644 index 00000000..0f0644e4 Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s02-grub-tui-selected.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s03-after-boot.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s03-after-boot.png new file mode 100644 index 00000000..543e5fc1 Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s03-after-boot.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s04-disk.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s04-disk.png new file mode 100644 index 00000000..f04f8638 Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s04-disk.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s05-locale.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s05-locale.png new file mode 100644 index 00000000..4e4c4e75 Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s05-locale.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s06-kb-list.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s06-kb-list.png new file mode 100644 index 00000000..73b02500 Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s06-kb-list.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s07-kb-us.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s07-kb-us.png new file mode 100644 index 00000000..73d77508 Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s07-kb-us.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s08-kb-us2.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s08-kb-us2.png new file mode 100644 index 00000000..73d77508 Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s08-kb-us2.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s09-kb-us3.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s09-kb-us3.png new file mode 100644 index 00000000..6bf46c84 Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s09-kb-us3.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s10-next-focus.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s10-next-focus.png new file mode 100644 index 00000000..eeec68c3 Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s10-next-focus.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s11-password.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s11-password.png new file mode 100644 index 00000000..3937f79a Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s11-password.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s12-password-filled.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s12-password-filled.png new file mode 100644 index 00000000..4fc31cba Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s12-password-filled.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s13-network.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s13-network.png new file mode 100644 index 00000000..bd52897b Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s13-network.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s14-focus-probe.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s14-focus-probe.png new file mode 100644 index 00000000..50a91fee Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s14-focus-probe.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s15-hostname-typed.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s15-hostname-typed.png new file mode 100644 index 00000000..02ac1379 Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s15-hostname-typed.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s16-summary.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s16-summary.png new file mode 100644 index 00000000..4f5af52a Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s16-summary.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s17-install-focus.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s17-install-focus.png new file mode 100644 index 00000000..6401db56 Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s17-install-focus.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s18-install-done.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s18-install-done.png new file mode 100644 index 00000000..14f3b2e7 Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s18-install-done.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s19-firstboot-console.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s19-firstboot-console.png new file mode 100644 index 00000000..0b7af5ee Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s19-firstboot-console.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/screens/s20-console-after-disk-attach.png b/documentation/audits/evidence-bignight-2026-09-14/screens/s20-console-after-disk-attach.png new file mode 100644 index 00000000..0b7af5ee Binary files /dev/null and b/documentation/audits/evidence-bignight-2026-09-14/screens/s20-console-after-disk-attach.png differ diff --git a/documentation/audits/evidence-bignight-2026-09-14/tunnel-before-no-box.txt b/documentation/audits/evidence-bignight-2026-09-14/tunnel-before-no-box.txt new file mode 100644 index 00000000..37602a90 --- /dev/null +++ b/documentation/audits/evidence-bignight-2026-09-14/tunnel-before-no-box.txt @@ -0,0 +1,5 @@ +=== tunnel BEFORE (no box) 2026-09-14T17:31:19Z +dig @1.1.1.1 felhom.enkicsifelhom.hu: 172.67.205.21 104.21.22.132 +dig @192.168.0.1 felhom.enkicsifelhom.hu: 104.21.22.132 172.67.205.21 +https://felhom.enkicsifelhom.hu -> 530 0.199202s +error code: 1033 diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index dce6c19c..aef5a016 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -712,6 +712,7 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-506** | **[P3-LOW] `day0-install.md` A.1 says "the controller manages per-app hostnames itself via the tunnel" — it does not.** MEASURED 2026-09-14: no code under `felhom-controller/controller/internal` creates tunnel ingress, DNS records or tunnel configurations (`grep -i 'ingress\|cfd_tunnel\|/configurations\|dns_records'` → only comments saying cloudflared is deployed when a token exists; positive control: the geo-restriction CF API use IS found in `cmd/controller/main.go`). The controller's only Cloudflare act is geo-restriction; the tunnel runs `tunnel run` with the token, so its routes come from Cloudflare's remote config set by the operator. A reader following A.1 skips the one step that makes the dashboard reachable (R-505). **Fix shape:** A.1 names the public-hostname step and its service settings, copied from a working tunnel. | **READY — rank P3-LOW; owner: CC (doc), operator (the settings to copy)** | | **R-507** | **[P3-LOW] The proof-install harness cannot drive the graphical installer, so a release's graphical entry is proven only up to its password screen.** MEASURED 2026-09-14 on VM 332 (ISO 1.27.0): `qm sendkey 332 tab` did not move focus (both password copies landed in one field), `mouse_move 1237 772` + `mouse_button 1` did not move the cursor or press Next, while `alt-n` did advance a page. The TUI entry is fully drivable. The gate's "proof install on BOTH menu entries" was met for 1.26.1 (by a person) and not for 1.27.x. **Fix shape:** measure QEMU `input-send-event` with absolute coordinates, or a VNC client on DooPlex; until then a release's graphical proof is an operator click-through. | **READY — rank P3-LOW; owner: CC** | | **R-508** | **[P2-MEDIUM] Customer `tester-1` has no registered e-mail, so neither the self-bind link nor the setup code can reach a volunteer.** MEASURED 2026-09-14: the edit form's `email` value is empty; on bind the hub logged `[ERROR] [claim] claim code generated (gen 1) but customer tester-1 has NO registered email — deliver via resend after setting one`. A volunteer onboarded on this record would sit at „A szerver beállítása" with no code. **What it needs:** the operator sets the volunteer's address on the record before sending the guide (day-0 A.2). The hub's customer page could warn when a record with an unclaimed box has no e-mail — the log line exists, the page says nothing. | **WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: operator (record), CC (page warning)** | +| **R-509** | **[P1-HIGH] A box installed for an EXISTING customer never gets the self-bind e-mail the console tells the volunteer to open.** MEASURED 2026-09-14 (BIGNIGHT, VM 333, ISO 1.27.1): customer `tester-1` now has `tester1@felhom.eu` registered; the box registered as appliance 28 at 17:44:53Z and its console says „Nyisd meg az e-mailben kapott linket"; **ten minutes later the mailbox (read through the Gmail connector) held 0 messages to that address.** Cause, from source: the hub auto-sends the link only at customer creation (`hub/internal/web/configs.go:725`) and at RESET completion (`customer_reset.go:162`); a customer whose e-mail was added later, or whose previous box was destroyed, never receives one unless the operator presses „Send self-bind link". The volunteer guide's operator prerequisites do not list that press. Intervention **I1** of the big night (the operator's button pressed). **Fix shape (for the operator to choose):** send the link when an unclaimed appliance registers and a customer with no host is waiting, or add the press to the guide's operator prerequisites (day-0 A.2). | **READY — rank P1-HIGH; owner: CC (hub fix) · operator (which fix shape)** |