From 879007aaecea5b8123a3c6d0344063de16788535 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 7 Aug 2026 15:05:35 +0200 Subject: [PATCH] =?UTF-8?q?walk5=20(R-201,=20fifth=20walk):=20Phase=20A=20?= =?UTF-8?q?journal=20=E2=80=94=20fixture=20built,=20sentinels=20proved=20b?= =?UTF-8?q?y=20name?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Written before the destruction, per §9.9. A fresh install landed on the VOUCHED set with no hand upgrade (controller 0.206.0, agent 0.127.0) — R-239's delivery gap is closed for this run, which is the first of the five walks where the box under test is the box a customer receives. Both §4.6 pre-destruction checks pass, neither previously exercised on a clean box: the recovery offer is correctly SILENT (shape (c) compares equal — the two key hashes are byte-identical on box and hub), and the restore page lists the app with the future-backup toggle OFF (R-237's fix, which the last walk measured failing). Also recorded: the §4.5 gate caught a harness fault (a toggle sent as enabled=1 rather than enabled=on) that had produced a green 'ok' over a zero-snapshot repository — the exact shape the gate exists for. --- .../tests/walk5-r201-2026-08-07/journal.md | 413 ++++++++++++++++++ 1 file changed, 413 insertions(+) create mode 100644 documentation/tests/walk5-r201-2026-08-07/journal.md diff --git a/documentation/tests/walk5-r201-2026-08-07/journal.md b/documentation/tests/walk5-r201-2026-08-07/journal.md new file mode 100644 index 0000000..4cfbea9 --- /dev/null +++ b/documentation/tests/walk5-r201-2026-08-07/journal.md @@ -0,0 +1,413 @@ +# THE FIFTH WALK (R-201) — journal + +**Venue:** `demo-hp` VM **325 `walk5-appliance`**, hub customer **`walk5`**. All four earlier venues +(`c11`, `rewalk`, `part4`, `finalwalk`) were torn down on 2026-08-06/07; **nothing was reused** except +the `c11-scratch` dir storage (a plain `dir` at `/mnt/nvme-1tb`, which is what the agent's +`exactMount` check requires) and the **freed** Storage Box sub-account *number* `sub4` — the account +itself is newly provisioned, id **285351**, not finalwalk's 285071 which the cascade destroyed. + +**Written as the walk proceeds, per §9.9.** Clocks: DooPlex and demo-hp and the appliance = **CEST**; +the guest and the hub = **UTC**. + +--- + +## §0–2 — orientation, on arrival + +### §1 baselines — re-read, all four match the runbook exactly + +| Component | Runbook | Measured on arrival | | +|---|---|---|---| +| `felhom-controller` | `3d3b4496f377` | `3d3b4496f377` | ✅ clean tree, == `origin/main` | +| `felhom-agent` | `703db166e762` | `703db166e762` | ✅ clean tree, == `origin/main` | +| `felhom.eu` | `10af63d0b04e` | `10af63d0b04e` | ✅ clean tree, == `origin/main` | +| `app-catalog-felhom.eu` | — | `459766cb1639` | clean, untouched | + +**No drift.** (The two previous walks both found one-commit drift on arrival and had to check it was +benign; this one did not.) + +### §1 the vouched set — read from the hub, not from a document + +`GET /configuration` on the hub (Basic auth, ClusterIP): + +``` +agent_version (selected) 0.127.0 +golden_version (selected) 0.206.0 +min_agent 0.127.0 +min_controller_version 0.156.0 +hub image (deployed) gitea.dooplex.hu/admin/felhom-hub:0.98.0 +installer SCRIPT_VERSION 1.25.0 (served live from felhom.eu/scripts/) +``` + +**All five match §1.** The vouched golden `0.206.0` carries controller `0.206.0`, so **this is the +first walk whose fresh install lands on the fixes it exists to exercise** — R-239's delivery gap, +which was the headline finding of the last two walks, is closed for this run. + +### §1 the build gate — GREEN before starting + +``` +$ python3 scripts/golden_currency_gate.py + newest released controller : 0.206.0 + newest golden baked : 0.206.0 +golden currency gate OK +rc=0 +``` + +**Stated precisely, because the gate says so itself:** it checks the **bake**, not the **vouch**. The +vouch was therefore checked separately, from the hub, above — and it is `0.206.0`. Both halves green. + +### §1 highest register ID + +**R-248**, confirmed by grep over `OPEN-ITEMS.md` (`R-241 … R-248`). The runbook's figure is current. + +### ISO — verified by round trip, not by name + +| | | +|---|---| +| published | `https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso` · 1 705 322 496 B · `last-modified Fri 31 Jul 2026 18:37:18 GMT` | +| published `.sha256` | `f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6` | +| local on `demo-hp` | `/var/lib/vz/template/iso/felhom-installer-1.26.1-pve9.2-1.iso` · **same sha256** | + +**Byte-identical.** (`https://iso.felhom.eu/` **itself** 404s — there is no index object; only named +objects are served. Recorded so the next session does not read that 404 as an outage.) + +### §2.6 — the open items that could touch this walk + +- **R-247 / R-248** — the staleness flag. The runbook says no running code can set it; **confirmed at + source**: R-246's census found `MarkEscrowStale` has **no production caller anywhere in the tree**, + and its one historical setter was removed in hub v0.95.0. A fresh box cannot reach the state. + **R-246 also records the demo-hp instance as CLEARED on 2026-08-07.** So shape (c) is live and + un-inhibited for this venue. +- **R-243** — the alarm gap for a box held in `awaiting_recovery_key`. **Directly in this walk's + path**: if §5 goes as designed, the rebuilt box enters exactly that state. The gap is that *the hub + raises nothing for it*. Not a defect of the walk; watch for it and report it as still-open. +- **R-240** — „Sikeres" over a run that covered nothing. Only reachable if an off-site run is made + with nothing selected; not part of this walk's route. +- **R-244** — teardown residue in `app_log_issues`. Teardown is §10, owed, not this session. + +### §2.7 — the two functions this walk exercises, read before running + +Both fixes are present in the tree at `3d3b4496f377` and both are new since the last walk: + +- **the mint guard** — `WriteOffboxSecrets` (`controller/internal/backup/offbox.go:438`) now returns + `ErrOffboxSealedPackageHeld` instead of generating, when `sealedPackageHeld()` is true **and** no + password exists. It is a **conjunction**, so a first-time box mints exactly as before. The transport + is still written. New declared state `offsite.state=awaiting_recovery_key` + (`OffboxAwaitingRecoveryKey`, `:502` — derived, not stored, and gated on `t.Enabled`). +- **shape (c)** — `OffsiteRecoveryOffer` (`:1580`) now offers when + `hubHash != "" && hubHash != localHash`. `""` falls back to (a)/(b) deliberately. + +**Neither has ever run on a real rebuild.** That is §5, and it is deliverable 2 whatever the verdict. + +--- + +## Venue construction + +### Hub customer `walk5` + +`POST /configs/new` — id `walk5`, name "Walk 5 R-201", domain `walk5.felhom.eu`, email = the +operator's registered address, **off-site ON** (shared, 50 GB), **DR tier ON**, `pbsdr_storage_id` +`felhom-pbs`. → **303** `/customers/walk5?flash=created`. + +Provisioned descriptor, read from the hub DB rather than from the page: + +```json +{"enabled":true,"type":"shared","host":"u629488-sub4.your-storagebox.de","user":"u629488-sub4", + "port":23,"repo_path":"/home/felhom-repo","quota_gb":50, + "host_fingerprint":"SHA256:oDHZqKXnoMtgvPBjjC57pcuFez28roaEuFcfwyg8O5c"} +``` + +`one_time_secrets` 1 · `customer_configs` 1 · `hosts` 0. Sub-account **285351**, user `u629488-sub4`. + +> **⚠ THE FIRST CREATE FAILED, AND IT IS A FINDING — not a fluke of my own making.** +> +> The first `POST /configs/new` returned no response to the client and the hub logged +> `[ERROR] offsite provision for walk5: offsite: host-key scan u629488-sub4.your-storagebox.de: +> dial: dial tcp [2a01:4f8:bacc:2:200::d30]:23: connect: network is unreachable` at **13:46:46**. +> +> The scan is **fail-closed by design** (`offsite.go:111-121`: *"don't serve a descriptor the +> controller can't verify"*) and its retry ladder is **~60 s** (`defaultScanBackoff`, 2+4+8+16+30), +> sized by its own comment to *"the observed DNS propagation lag"* of a fresh sub-account. Here the +> lag was **longer than the budget**: the first five attempts failed `no such host`, and by the time +> DNS answered it answered **AAAA-first**, into a pod with no IPv6 route. +> +> **The retry (a second identical POST) succeeded** — idempotent on the label +> (`[offsite] shared already provisioned for walk5 (subaccount 285351)`), scan green on its own final +> rung at ~13:47:28, config created. **Total settle time ≈ 100 s against a 60 s budget.** +> +> **Measured afterwards, so the cause is not guessed:** the hub pod reaches +> `u629488-sub4.your-storagebox.de:23` **now**, by name, and gets `SSH-2.0-OpenSSH_9.6p1` — as does +> the node. So the endpoint is fine and the window was transient. **What is not fine is the operator +> experience**: a customer create can fail with a fail-closed error whose only remedy is "press it +> again", and nothing says so. Filed below. + +### VM + + +| | | +|---|---| +| Host | `demo-hp` (HP t740), Tier 0 — the designated drill host | +| VM | **325 `walk5-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` | +| Disks | `scsi0` 200 G · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` (a `dir` at the **mount root** `/mnt/nvme-1tb`, which is what the agent's `exactMount` check requires) | +| Appliance | `walk5.felhom.eu` @ **192.168.0.100/24**, gw/DNS 192.168.0.1, MAC `bc:24:11:37:7c:f3` | +| Guest | LXC **9201** @ **192.168.0.149** (LAN) + `169.254.253.2` (R-50 island) | +| Hub customer | **`walk5`** "Walk 5 R-201", DR tier ON, off-site ON (shared, 50 GB) | +| Host id | **`walk5-4bada5`** · appliance uuid `596d8104-2af9-4525-baa7-e479314cbb44` | +| Off-site | Storage Box sub-account **285351**, user `u629488-sub4` · WG peer **10.77.0.5** | +| **Untouched** | `drill-r50` (VM 300, the protected fixture), guest 9201 on both demo boxes, DooPlex, `ep0`, the tester's box | + +**Reachability, stated because it shapes every later step:** DooPlex has **no L2 path to the +192.168.0.0/24 LAN** — `ip neigh` is `INCOMPLETE` even for `demo-hp`, which it reaches over the +tailnet. So the appliance is one SSH hop behind `demo-hp`, and every request to the guest is made +**from the appliance**, which is exactly what a browser on the customer's LAN would do. + +--- + +## Phase A — the fixture (§4) + +### A1 — installed from the published ISO, through the Terminal UI + +Driven with `qm sendkey` + `qm monitor screendump` → PNG. **All three known traps reproduced:** + +1. **GRUB defaults to the graphical entry — and it bit, first attempt.** The countdown was **14 s** + with my screendump→convert→read round trip at **~18 s**, so the menu auto-booted the default and + the GUI installer came up. **Recorded rather than quietly retried:** the VM was stopped and + restarted, and the keys sent **inside one remote command** (`sleep 6; sendkey down; screendump; + sendkey ret`) — the screendump taken *between* the arrow and the Enter is the positive control, + and it shows the highlight on **„Felhom telepítés (szöveges mód)"** at the moment of the press. +2. **The keymap defaults to Hungarian** while `sendkey` emits US scancodes. Changed to **U.S. + English** before any typing. **Positive control:** the administrator email typed through the + identical path rendered **`walk5@felhom.eu`** — `@` is `shift-2` on US and `AltGr+V` on HU, and it + is the only available evidence for the 24 masked password characters. +3. **Auto-reboot was checked.** Unchecked, and **confirmed `[ ]` with the focus moved away**; then + `--boot` set in its own `qm set` with the ISO detached, both verified from `qm config` + **before the first boot**: `boot: order=scsi0`, `ide2` lines = **0**. + +Summary screen, verbatim: `ext4` · `/dev/sda` · `Europe/Budapest` · **`U.S. English`** · +`walk5@felhom.eu` · `nic0` · `walk5.felhom.eu` · `192.168.0.100/24` · `192.168.0.1` · `192.168.0.1`. + +**One reading corrected by a second instrument, as in the last walk:** `192.168.0.100` was the +installer's prefill and answered on the LAN. The **MAC** was `bc:24:11:37:7c:f3` — VM 325's own DHCP +lease. Not a collision; a ping alone could not have told the difference. + +**Day-0 fired on first boot, unaided.** Console showed the Hungarian pairing banner: + +> `Felhom — a doboz készen áll, és a párosításra vár.` · `Párosító kód: E28-WJH` + +The hub's `appliance_registrations` row **24** carried the same code (`E28WJH`), the same MAC, the +same SMBIOS uuid and **three SSH host keys**, `first_seen 12:18:59Z` — **32 seconds after boot**. +Bound through the real endpoint (`POST /appliances/24/bind`, **303**). + +### A1b — THE HEADLINE: a fresh install DOES get the fixes + +| | vouched | the box landed on | +|---|---|---| +| agent | 0.127.0 | **0.127.0** ✅ | +| golden | 0.206.0 | — | +| controller | (baked into the golden) | **0.206.0** ✅ | + +``` +14:22:48 [INFO] Artifact manifest served for customer walk5 (agent=0.127.0 golden=0.206.0) +14:25:33 [INFO] Event from walk5: controller_started (info) — Controller elindult (0.206.0) + docker ps → Up 10 minutes (healthy) gitea.dooplex.hu/admin/felhom-controller:0.206.0 + felhom-agent --version → felhom-agent 0.127.0 +``` + +**No hand upgrade was needed and none was applied.** This is the first walk of the five where that +sentence is true of the *fixes under test*: R-239's delivery gap — the headline finding of both +previous walks — **is closed for this run**, and the box under test is the box a customer receives. + +**The whole day-0 cascade ran hands-free**, bind → provision in **3 m 55 s**: + +``` +14:21:38 appliance 24 BOUND to customer walk5 (mode=appliance) +14:22:00 appliance credentials DELIVERED once +14:22:08 [claim] claim code (gen 1) emailed to the registered address +14:22:38 host enrolled: walk5-4bada5 +14:22:46 offsite credentials re-issued on re-enroll; break-glass root@pam vaulted (32 chars) +14:23:19 wg registered 10.77.0.5 · DR-recipe host-half · tenantsync provision ok (ns=walk5) + pbsdr auto-provisioned (hands-free cascade) +14:25:33 controller 0.206.0 reporting +``` + +### A2 — claimed + +Claimed through the real `POST /claim` form (302 → `/`), then `POST /login` → 302. **The claim code +came from the documented `--print-reset-code` escape hatch** (R-204 item 1) — **a guest command line, +used deliberately as FIXTURE CONSTRUCTION, permitted by §3 in Phase A, and counted as such.** Phase B +must not use it. + +The dashboard password is a 24-character alphanumeric, `0600` in `~/.config/walk5/` on **DooPlex +only**. The appliance's root credential is the hub-vaulted break-glass (`root@pam`, revealed through +`POST /hosts/walk5-4bada5/reveal-recovery-credential`) — **the password typed at the installer no +longer authenticates, because day-0 step 4b rotates and vaults it.** Stated because it cost a dead +end. + +### A3 — drives, the app, and the three sentinels + +Both raw disks enrolled through the **customer** endpoints (`GET /api/disks/candidates` → +`POST /api/storage/init`), ext4: + +``` +/dev/sdb → /mnt/felhom-drives/adatok (app data) +/dev/sdc → /mnt/felhom-drives/mentes (backup target) +``` + +Backup target assigned to `mentes` — and, exactly as the last walk recorded, it reported +`restart_required: true` and flipped only after the agent restart it asked for: + +``` +before: {"degraded":true, …"A rendszermentés jelenleg ugyanazon a lemezen van…"} +after: {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"} +``` + +`calibre-web` deployed through the real API with `HDD_PATH=/mnt/felhom-drives/adatok` — +`state: running` **and `health_probe.healthy: true`**. + +**THE THREE SENTINELS** + +| # | file | bytes | sha256 | +|---|---|---|---| +| A | `WALK5-SENTINEL-A.txt` | 61 | `11eb7fb2d1891f62a685e3d9a9fda44e0a342d437d4aebaa15fcb4c0fd79d539` | +| B | `WALK5-őrszem-ékezetes-árvíztűrő.txt` | 66 | `6b504d1e83bf4c673334d0b9e701f3e5f07311d954fdc4463a6b0a55b261ae32` | +| C | `WALK5-SENTINEL-C-12MB.bin` | 12 582 912 | `0baaf402733e6a85f3d9ce8c4c3a57b05692f5f6733da6a23da76d2cd7635210` | + +Sentinel B's filename **as hex**, and this is the reading that cannot lie: + +``` +57414c4b352d c591 72737a656d2d c3a9 6b657a657465732d c3a1 7276 c3ad 7a74 c5b1 72 c591 2e747874 + ő é á í ű r ő +``` + +**No `efbfbd` anywhere.** Written by a Python placer pushed in with `pct push`, which builds the name +from a **hex string decoded to bytes** and passes a **bytes path** to `os.open` — no shell layer ever +sees the name, and the readback is from `os.listdir` on a bytes path, not from a shell `ls`. This is +the trap that bit three times; it did not bite here. + +**Placed under `…/adatok/userdata/media/books` — the level the last walk got wrong.** The tree was +listed first and the namespace root confirmed to be `/mnt/felhom-drives/adatok` **directly**. + +### A4 — the escrow ceremony + +Preflight **6 of 6 green** (`pbs_storage_id` · `dr_tier` · `age_binary` · `hub_upload` · +`staged_secret` · `sudo_grant`), `agent_supported: true`, **`stale: false`**. + +> **That `stale: false` is a deliberate check, not a formality.** R-246/R-247/R-248 concern a +> staleness flag that would silently disable the very discriminator this walk exists to exercise. +> The spike's ruling was that a fresh box cannot reach the state (`MarkEscrowStale` has no production +> caller). **Measured here on a fresh box: it has not.** + +Ceremony through the customer wizard's own endpoints (`/api/escrow/start` with password re-auth — +it is a **form** POST, not JSON, which cost one rate-limited attempt): + +``` +phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24 · claimable: true +key_fingerprint: a6:86:f7:fb:74:e7:16:3c:…:31:2a:4c:f9 +``` + +**`restic_pw_sealed: true` is the field the whole exercise rests on.** + +**R was claimed ONE-SHOT and streamed file→file** into `~/.config/walk5/R_walk5.txt` (`0600`, +**DooPlex only**). It was never rendered, never an argument, never a log line. The appliance's copy +was `shred -u`'d and **its absence verified**. Shape only: **82 characters, 10 hyphen-separated +tokens, lowercase+digits**. + +> **§9.4 slip, recorded rather than buried.** The Phase-A *claim code* (not R) reached this session's +> transcript, because the hatch prints it and my shape-only filter was written for an uppercase +> pattern and did not match a lowercase one. It was consumed by the claim seconds later and is now +> void. **R itself was never printed and never left the file→file path.** + +### A5 — the off-site backup, and the sentinels BY NAME + +> **THE §4.5 GATE EARNED ITS PLACE AGAIN — and this time it caught MY fault, not the product's.** +> The first run reported **`ok` in 28 s** with **`snapshots = 0`** and an empty repository size. The +> log said why: `[offbox] backup run started (0 app(s) toggled)`. **My toggle POST sent +> `enabled=1`**, and the handler accepts only `on`/`true` — so it recorded *off*, redirected with +> „A távoli mentés beállítása frissítve." and the run then correctly backed up nothing. +> **A harness fault, not a product defect** — but the shape is precisely R-240's, and had I read the +> green `ok` and moved on, the destruction would have proven nothing. + +Re-toggled (`enabled=on`), selection verified in the rendered page (`name="app" value="calibre-web"` ++ `checked`) **before** the re-run, which then logged `backup run started (1 app(s) toggled)`. + +**`ok`, 1 m 20 s, 1 snapshot, 12.0 MB (12 611 563 B).** **The gate is not the green tick** — +`restic snapshots` + `restic ls latest --long`, run against the repository with its own credentials: + +``` +5b0f20f7 2026-08-07 12:57:41 walk5 [felhom-offbox, calibre-web] + /mnt/felhom-drives/adatok/backups/primary/calibre-web + /mnt/felhom-drives/adatok/userdata/media/books + +-rw-r--r-- 1000 1000 61 …/userdata/media/books/WALK5-SENTINEL-A.txt +-rw-r--r-- 1000 1000 12582912 …/userdata/media/books/WALK5-SENTINEL-C-12MB.bin +-rw-r--r-- 1000 1000 66 …/userdata/media/books/WALK5-őrszem-ékezetes-árvíztűrő.txt +-rw-r--r-- 1000 1000 413696 …/userdata/media/books/metadata.db ← calibre-web's real DB +restic stats latest: 22 files, 12.784 MiB +``` + +**All three sentinels are in the snapshot, by name, at the right sizes — and the accented filename +survived into restic intact.** + +### A6 — §4.6's two pre-destruction checks, neither ever exercised on a clean box + +**Check 1 — the recovery offer is NOT shown. PASSES.** + +``` +GET / → 302 → /launcher +GET /launcher → 200, recovery mentions: 0 +GET / → 200 (followed), recovery mentions: 0 +GET /recovery → 302 → /backups/remote +``` + +No banner, no entry point, nothing to seek. **And the reason is measurable rather than assumed:** the +hub's package covers the key in use, so v0.206.0's **shape (c) compares equal and correctly stays +silent** — + +``` +box /data/offbox/repo_password sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f +box settings.hub_escrow_key_sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f +hub host_escrow.restic_pw_sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f + hub_escrow_key_checked_at = 2026-08-07T12:51:39Z +``` + +**This is the first live measurement of the persisted discriminator**, and it is the negative half: +the shape that must NOT fire, not firing, for the right reason. + +**Check 2 — the restore page lists the app with the future-backup toggle OFF. PASSES.** + +With `calibre-web` toggled **off** for future off-site backups, `/backups/restore` renders +**identically** — „Calibre-Web Automated" still listed under **„Visszaállítás a távoli tárolóból"**, +10 `calibre` occurrences in the HTML both before and after. + +**That is R-237's fix, live.** The last walk's T3 measured **0 restore entries and a 302** on +controller 0.203.0 — the customer's existing backup hidden by a setting about the future. The toggle +was returned to **on** afterwards. + +### A7 — the pre-destruction truth (§4.7's seven records) + +**Box** (`settings.json`, secrets stripped): + +``` +offbox: enabled true · escrow_state "escrowed" · last_status "ok" · last_duration 1m20s + last_run/last_success 2026-08-07T12:58:57Z · snapshot_count 1 + repo_size_bytes 12 611 563 ("12.0 MB") · stats_known true · quota_gb 50 + host u629488-sub4.your-storagebox.de:23 · repo_path /home/felhom-repo +repo_state: (absent — not orphaned) · hub_escrow_identity_present: true · claimed: true +agent 0.127.0 · controller 0.206.0 (healthy) · guests 1/1 +``` + +**Hub:** + +``` +host_escrow(walk5-4bada5): blob 383 B · identity_blob 572 B (R-198 material present) + key_fingerprint a6:86:f7:fb:…:4c:f9 + restic_pw_sha256 eabf427c7274…144f + stale_at NULL · created 2026-08-07T12:51:02Z +host_escrow_superseded: 0 rows for walk5 +hosts: walk5-4bada5 ONLINE · agent 0.127.0 · PBS wrapper MATCHES VOUCHED + (104db0a4401f…) · WG 10.77.0.5 confirmed +DR Recipe: present · Key Escrow: present +``` + +**Phase A gate: PASSED.** All seven records taken; the three sentinels listed **by name**. + +---