# THE FIFTH WALK (R-201) — journal **Venue:** `demo-hp` VM **325 `walk5-appliance`**, hub customer **`walk5`**. All four earlier venues (`c11`, `rewalk`, `part4`, `finalwalk`) were torn down on 2026-08-06/07; **nothing was reused** except the `c11-scratch` dir storage (a plain `dir` at `/mnt/nvme-1tb`, which is what the agent's `exactMount` check requires) and the **freed** Storage Box sub-account *number* `sub4` — the account itself is newly provisioned, id **285351**, not finalwalk's 285071 which the cascade destroyed. **Written as the walk proceeds, per §9.9.** Clocks: DooPlex and demo-hp and the appliance = **CEST**; the guest and the hub = **UTC**. --- ## §0–2 — orientation, on arrival ### §1 baselines — re-read, all four match the runbook exactly | Component | Runbook | Measured on arrival | | |---|---|---|---| | `felhom-controller` | `3d3b4496f377` | `3d3b4496f377` | ✅ clean tree, == `origin/main` | | `felhom-agent` | `703db166e762` | `703db166e762` | ✅ clean tree, == `origin/main` | | `felhom.eu` | `10af63d0b04e` | `10af63d0b04e` | ✅ clean tree, == `origin/main` | | `app-catalog-felhom.eu` | — | `459766cb1639` | clean, untouched | **No drift.** (The two previous walks both found one-commit drift on arrival and had to check it was benign; this one did not.) ### §1 the vouched set — read from the hub, not from a document `GET /configuration` on the hub (Basic auth, ClusterIP): ``` agent_version (selected) 0.127.0 golden_version (selected) 0.206.0 min_agent 0.127.0 min_controller_version 0.156.0 hub image (deployed) gitea.dooplex.hu/admin/felhom-hub:0.98.0 installer SCRIPT_VERSION 1.25.0 (served live from felhom.eu/scripts/) ``` **All five match §1.** The vouched golden `0.206.0` carries controller `0.206.0`, so **this is the first walk whose fresh install lands on the fixes it exists to exercise** — R-239's delivery gap, which was the headline finding of the last two walks, is closed for this run. ### §1 the build gate — GREEN before starting ``` $ python3 scripts/golden_currency_gate.py newest released controller : 0.206.0 newest golden baked : 0.206.0 golden currency gate OK rc=0 ``` **Stated precisely, because the gate says so itself:** it checks the **bake**, not the **vouch**. The vouch was therefore checked separately, from the hub, above — and it is `0.206.0`. Both halves green. ### §1 highest register ID **R-248**, confirmed by grep over `OPEN-ITEMS.md` (`R-241 … R-248`). The runbook's figure is current. ### ISO — verified by round trip, not by name | | | |---|---| | published | `https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso` · 1 705 322 496 B · `last-modified Fri 31 Jul 2026 18:37:18 GMT` | | published `.sha256` | `f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6` | | local on `demo-hp` | `/var/lib/vz/template/iso/felhom-installer-1.26.1-pve9.2-1.iso` · **same sha256** | **Byte-identical.** (`https://iso.felhom.eu/` **itself** 404s — there is no index object; only named objects are served. Recorded so the next session does not read that 404 as an outage.) ### §2.6 — the open items that could touch this walk - **R-247 / R-248** — the staleness flag. The runbook says no running code can set it; **confirmed at source**: R-246's census found `MarkEscrowStale` has **no production caller anywhere in the tree**, and its one historical setter was removed in hub v0.95.0. A fresh box cannot reach the state. **R-246 also records the demo-hp instance as CLEARED on 2026-08-07.** So shape (c) is live and un-inhibited for this venue. - **R-243** — the alarm gap for a box held in `awaiting_recovery_key`. **Directly in this walk's path**: if §5 goes as designed, the rebuilt box enters exactly that state. The gap is that *the hub raises nothing for it*. Not a defect of the walk; watch for it and report it as still-open. - **R-240** — „Sikeres" over a run that covered nothing. Only reachable if an off-site run is made with nothing selected; not part of this walk's route. - **R-244** — teardown residue in `app_log_issues`. Teardown is §10, owed, not this session. ### §2.7 — the two functions this walk exercises, read before running Both fixes are present in the tree at `3d3b4496f377` and both are new since the last walk: - **the mint guard** — `WriteOffboxSecrets` (`controller/internal/backup/offbox.go:438`) now returns `ErrOffboxSealedPackageHeld` instead of generating, when `sealedPackageHeld()` is true **and** no password exists. It is a **conjunction**, so a first-time box mints exactly as before. The transport is still written. New declared state `offsite.state=awaiting_recovery_key` (`OffboxAwaitingRecoveryKey`, `:502` — derived, not stored, and gated on `t.Enabled`). - **shape (c)** — `OffsiteRecoveryOffer` (`:1580`) now offers when `hubHash != "" && hubHash != localHash`. `""` falls back to (a)/(b) deliberately. **Neither has ever run on a real rebuild.** That is §5, and it is deliverable 2 whatever the verdict. --- ## Venue construction ### Hub customer `walk5` `POST /configs/new` — id `walk5`, name "Walk 5 R-201", domain `walk5.felhom.eu`, email = the operator's registered address, **off-site ON** (shared, 50 GB), **DR tier ON**, `pbsdr_storage_id` `felhom-pbs`. → **303** `/customers/walk5?flash=created`. Provisioned descriptor, read from the hub DB rather than from the page: ```json {"enabled":true,"type":"shared","host":"u629488-sub4.your-storagebox.de","user":"u629488-sub4", "port":23,"repo_path":"/home/felhom-repo","quota_gb":50, "host_fingerprint":"SHA256:oDHZqKXnoMtgvPBjjC57pcuFez28roaEuFcfwyg8O5c"} ``` `one_time_secrets` 1 · `customer_configs` 1 · `hosts` 0. Sub-account **285351**, user `u629488-sub4`. > **⚠ THE FIRST CREATE FAILED, AND IT IS A FINDING — not a fluke of my own making.** > > The first `POST /configs/new` returned no response to the client and the hub logged > `[ERROR] offsite provision for walk5: offsite: host-key scan u629488-sub4.your-storagebox.de: > dial: dial tcp [2a01:4f8:bacc:2:200::d30]:23: connect: network is unreachable` at **13:46:46**. > > The scan is **fail-closed by design** (`offsite.go:111-121`: *"don't serve a descriptor the > controller can't verify"*) and its retry ladder is **~60 s** (`defaultScanBackoff`, 2+4+8+16+30), > sized by its own comment to *"the observed DNS propagation lag"* of a fresh sub-account. Here the > lag was **longer than the budget**: the first five attempts failed `no such host`, and by the time > DNS answered it answered **AAAA-first**, into a pod with no IPv6 route. > > **The retry (a second identical POST) succeeded** — idempotent on the label > (`[offsite] shared already provisioned for walk5 (subaccount 285351)`), scan green on its own final > rung at ~13:47:28, config created. **Total settle time ≈ 100 s against a 60 s budget.** > > **Measured afterwards, so the cause is not guessed:** the hub pod reaches > `u629488-sub4.your-storagebox.de:23` **now**, by name, and gets `SSH-2.0-OpenSSH_9.6p1` — as does > the node. So the endpoint is fine and the window was transient. **What is not fine is the operator > experience**: a customer create can fail with a fail-closed error whose only remedy is "press it > again", and nothing says so. Filed below. ### VM | | | |---|---| | Host | `demo-hp` (HP t740), Tier 0 — the designated drill host | | VM | **325 `walk5-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` | | Disks | `scsi0` 200 G · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` (a `dir` at the **mount root** `/mnt/nvme-1tb`, which is what the agent's `exactMount` check requires) | | Appliance | `walk5.felhom.eu` @ **192.168.0.100/24**, gw/DNS 192.168.0.1, MAC `bc:24:11:37:7c:f3` | | Guest | LXC **9201** @ **192.168.0.149** (LAN) + `169.254.253.2` (R-50 island) | | Hub customer | **`walk5`** "Walk 5 R-201", DR tier ON, off-site ON (shared, 50 GB) | | Host id | **`walk5-4bada5`** · appliance uuid `596d8104-2af9-4525-baa7-e479314cbb44` | | Off-site | Storage Box sub-account **285351**, user `u629488-sub4` · WG peer **10.77.0.5** | | **Untouched** | `drill-r50` (VM 300, the protected fixture), guest 9201 on both demo boxes, DooPlex, `ep0`, the tester's box | **Reachability, stated because it shapes every later step:** DooPlex has **no L2 path to the 192.168.0.0/24 LAN** — `ip neigh` is `INCOMPLETE` even for `demo-hp`, which it reaches over the tailnet. So the appliance is one SSH hop behind `demo-hp`, and every request to the guest is made **from the appliance**, which is exactly what a browser on the customer's LAN would do. --- ## Phase A — the fixture (§4) ### A1 — installed from the published ISO, through the Terminal UI Driven with `qm sendkey` + `qm monitor screendump` → PNG. **All three known traps reproduced:** 1. **GRUB defaults to the graphical entry — and it bit, first attempt.** The countdown was **14 s** with my screendump→convert→read round trip at **~18 s**, so the menu auto-booted the default and the GUI installer came up. **Recorded rather than quietly retried:** the VM was stopped and restarted, and the keys sent **inside one remote command** (`sleep 6; sendkey down; screendump; sendkey ret`) — the screendump taken *between* the arrow and the Enter is the positive control, and it shows the highlight on **„Felhom telepítés (szöveges mód)"** at the moment of the press. 2. **The keymap defaults to Hungarian** while `sendkey` emits US scancodes. Changed to **U.S. English** before any typing. **Positive control:** the administrator email typed through the identical path rendered **`walk5@felhom.eu`** — `@` is `shift-2` on US and `AltGr+V` on HU, and it is the only available evidence for the 24 masked password characters. 3. **Auto-reboot was checked.** Unchecked, and **confirmed `[ ]` with the focus moved away**; then `--boot` set in its own `qm set` with the ISO detached, both verified from `qm config` **before the first boot**: `boot: order=scsi0`, `ide2` lines = **0**. Summary screen, verbatim: `ext4` · `/dev/sda` · `Europe/Budapest` · **`U.S. English`** · `walk5@felhom.eu` · `nic0` · `walk5.felhom.eu` · `192.168.0.100/24` · `192.168.0.1` · `192.168.0.1`. **One reading corrected by a second instrument, as in the last walk:** `192.168.0.100` was the installer's prefill and answered on the LAN. The **MAC** was `bc:24:11:37:7c:f3` — VM 325's own DHCP lease. Not a collision; a ping alone could not have told the difference. **Day-0 fired on first boot, unaided.** Console showed the Hungarian pairing banner: > `Felhom — a doboz készen áll, és a párosításra vár.` · `Párosító kód: E28-WJH` The hub's `appliance_registrations` row **24** carried the same code (`E28WJH`), the same MAC, the same SMBIOS uuid and **three SSH host keys**, `first_seen 12:18:59Z` — **32 seconds after boot**. Bound through the real endpoint (`POST /appliances/24/bind`, **303**). ### A1b — THE HEADLINE: a fresh install DOES get the fixes | | vouched | the box landed on | |---|---|---| | agent | 0.127.0 | **0.127.0** ✅ | | golden | 0.206.0 | — | | controller | (baked into the golden) | **0.206.0** ✅ | ``` 14:22:48 [INFO] Artifact manifest served for customer walk5 (agent=0.127.0 golden=0.206.0) 14:25:33 [INFO] Event from walk5: controller_started (info) — Controller elindult (0.206.0) docker ps → Up 10 minutes (healthy) gitea.dooplex.hu/admin/felhom-controller:0.206.0 felhom-agent --version → felhom-agent 0.127.0 ``` **No hand upgrade was needed and none was applied.** This is the first walk of the five where that sentence is true of the *fixes under test*: R-239's delivery gap — the headline finding of both previous walks — **is closed for this run**, and the box under test is the box a customer receives. **The whole day-0 cascade ran hands-free**, bind → provision in **3 m 55 s**: ``` 14:21:38 appliance 24 BOUND to customer walk5 (mode=appliance) 14:22:00 appliance credentials DELIVERED once 14:22:08 [claim] claim code (gen 1) emailed to the registered address 14:22:38 host enrolled: walk5-4bada5 14:22:46 offsite credentials re-issued on re-enroll; break-glass root@pam vaulted (32 chars) 14:23:19 wg registered 10.77.0.5 · DR-recipe host-half · tenantsync provision ok (ns=walk5) pbsdr auto-provisioned (hands-free cascade) 14:25:33 controller 0.206.0 reporting ``` ### A2 — claimed Claimed through the real `POST /claim` form (302 → `/`), then `POST /login` → 302. **The claim code came from the documented `--print-reset-code` escape hatch** (R-204 item 1) — **a guest command line, used deliberately as FIXTURE CONSTRUCTION, permitted by §3 in Phase A, and counted as such.** Phase B must not use it. The dashboard password is a 24-character alphanumeric, `0600` in `~/.config/walk5/` on **DooPlex only**. The appliance's root credential is the hub-vaulted break-glass (`root@pam`, revealed through `POST /hosts/walk5-4bada5/reveal-recovery-credential`) — **the password typed at the installer no longer authenticates, because day-0 step 4b rotates and vaults it.** Stated because it cost a dead end. ### A3 — drives, the app, and the three sentinels Both raw disks enrolled through the **customer** endpoints (`GET /api/disks/candidates` → `POST /api/storage/init`), ext4: ``` /dev/sdb → /mnt/felhom-drives/adatok (app data) /dev/sdc → /mnt/felhom-drives/mentes (backup target) ``` Backup target assigned to `mentes` — and, exactly as the last walk recorded, it reported `restart_required: true` and flipped only after the agent restart it asked for: ``` before: {"degraded":true, …"A rendszermentés jelenleg ugyanazon a lemezen van…"} after: {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"} ``` `calibre-web` deployed through the real API with `HDD_PATH=/mnt/felhom-drives/adatok` — `state: running` **and `health_probe.healthy: true`**. **THE THREE SENTINELS** | # | file | bytes | sha256 | |---|---|---|---| | A | `WALK5-SENTINEL-A.txt` | 61 | `11eb7fb2d1891f62a685e3d9a9fda44e0a342d437d4aebaa15fcb4c0fd79d539` | | B | `WALK5-őrszem-ékezetes-árvíztűrő.txt` | 66 | `6b504d1e83bf4c673334d0b9e701f3e5f07311d954fdc4463a6b0a55b261ae32` | | C | `WALK5-SENTINEL-C-12MB.bin` | 12 582 912 | `0baaf402733e6a85f3d9ce8c4c3a57b05692f5f6733da6a23da76d2cd7635210` | Sentinel B's filename **as hex**, and this is the reading that cannot lie: ``` 57414c4b352d c591 72737a656d2d c3a9 6b657a657465732d c3a1 7276 c3ad 7a74 c5b1 72 c591 2e747874 ő é á í ű r ő ``` **No `efbfbd` anywhere.** Written by a Python placer pushed in with `pct push`, which builds the name from a **hex string decoded to bytes** and passes a **bytes path** to `os.open` — no shell layer ever sees the name, and the readback is from `os.listdir` on a bytes path, not from a shell `ls`. This is the trap that bit three times; it did not bite here. **Placed under `…/adatok/userdata/media/books` — the level the last walk got wrong.** The tree was listed first and the namespace root confirmed to be `/mnt/felhom-drives/adatok` **directly**. ### A4 — the escrow ceremony Preflight **6 of 6 green** (`pbs_storage_id` · `dr_tier` · `age_binary` · `hub_upload` · `staged_secret` · `sudo_grant`), `agent_supported: true`, **`stale: false`**. > **That `stale: false` is a deliberate check, not a formality.** R-246/R-247/R-248 concern a > staleness flag that would silently disable the very discriminator this walk exists to exercise. > The spike's ruling was that a fresh box cannot reach the state (`MarkEscrowStale` has no production > caller). **Measured here on a fresh box: it has not.** Ceremony through the customer wizard's own endpoints (`/api/escrow/start` with password re-auth — it is a **form** POST, not JSON, which cost one rate-limited attempt): ``` phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24 · claimable: true key_fingerprint: a6:86:f7:fb:74:e7:16:3c:…:31:2a:4c:f9 ``` **`restic_pw_sealed: true` is the field the whole exercise rests on.** **R was claimed ONE-SHOT and streamed file→file** into `~/.config/walk5/R_walk5.txt` (`0600`, **DooPlex only**). It was never rendered, never an argument, never a log line. The appliance's copy was `shred -u`'d and **its absence verified**. Shape only: **82 characters, 10 hyphen-separated tokens, lowercase+digits**. > **§9.4 slip, recorded rather than buried.** The Phase-A *claim code* (not R) reached this session's > transcript, because the hatch prints it and my shape-only filter was written for an uppercase > pattern and did not match a lowercase one. It was consumed by the claim seconds later and is now > void. **R itself was never printed and never left the file→file path.** ### A5 — the off-site backup, and the sentinels BY NAME > **THE §4.5 GATE EARNED ITS PLACE AGAIN — and this time it caught MY fault, not the product's.** > The first run reported **`ok` in 28 s** with **`snapshots = 0`** and an empty repository size. The > log said why: `[offbox] backup run started (0 app(s) toggled)`. **My toggle POST sent > `enabled=1`**, and the handler accepts only `on`/`true` — so it recorded *off*, redirected with > „A távoli mentés beállítása frissítve." and the run then correctly backed up nothing. > **A harness fault, not a product defect** — but the shape is precisely R-240's, and had I read the > green `ok` and moved on, the destruction would have proven nothing. Re-toggled (`enabled=on`), selection verified in the rendered page (`name="app" value="calibre-web"` + `checked`) **before** the re-run, which then logged `backup run started (1 app(s) toggled)`. **`ok`, 1 m 20 s, 1 snapshot, 12.0 MB (12 611 563 B).** **The gate is not the green tick** — `restic snapshots` + `restic ls latest --long`, run against the repository with its own credentials: ``` 5b0f20f7 2026-08-07 12:57:41 walk5 [felhom-offbox, calibre-web] /mnt/felhom-drives/adatok/backups/primary/calibre-web /mnt/felhom-drives/adatok/userdata/media/books -rw-r--r-- 1000 1000 61 …/userdata/media/books/WALK5-SENTINEL-A.txt -rw-r--r-- 1000 1000 12582912 …/userdata/media/books/WALK5-SENTINEL-C-12MB.bin -rw-r--r-- 1000 1000 66 …/userdata/media/books/WALK5-őrszem-ékezetes-árvíztűrő.txt -rw-r--r-- 1000 1000 413696 …/userdata/media/books/metadata.db ← calibre-web's real DB restic stats latest: 22 files, 12.784 MiB ``` **All three sentinels are in the snapshot, by name, at the right sizes — and the accented filename survived into restic intact.** ### A6 — §4.6's two pre-destruction checks, neither ever exercised on a clean box **Check 1 — the recovery offer is NOT shown. PASSES.** ``` GET / → 302 → /launcher GET /launcher → 200, recovery mentions: 0 GET / → 200 (followed), recovery mentions: 0 GET /recovery → 302 → /backups/remote ``` No banner, no entry point, nothing to seek. **And the reason is measurable rather than assumed:** the hub's package covers the key in use, so v0.206.0's **shape (c) compares equal and correctly stays silent** — ``` box /data/offbox/repo_password sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f box settings.hub_escrow_key_sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f hub host_escrow.restic_pw_sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f hub_escrow_key_checked_at = 2026-08-07T12:51:39Z ``` **This is the first live measurement of the persisted discriminator**, and it is the negative half: the shape that must NOT fire, not firing, for the right reason. **Check 2 — the restore page lists the app with the future-backup toggle OFF. PASSES.** With `calibre-web` toggled **off** for future off-site backups, `/backups/restore` renders **identically** — „Calibre-Web Automated" still listed under **„Visszaállítás a távoli tárolóból"**, 10 `calibre` occurrences in the HTML both before and after. **That is R-237's fix, live.** The last walk's T3 measured **0 restore entries and a 302** on controller 0.203.0 — the customer's existing backup hidden by a setting about the future. The toggle was returned to **on** afterwards. ### A7 — the pre-destruction truth (§4.7's seven records) **Box** (`settings.json`, secrets stripped): ``` offbox: enabled true · escrow_state "escrowed" · last_status "ok" · last_duration 1m20s last_run/last_success 2026-08-07T12:58:57Z · snapshot_count 1 repo_size_bytes 12 611 563 ("12.0 MB") · stats_known true · quota_gb 50 host u629488-sub4.your-storagebox.de:23 · repo_path /home/felhom-repo repo_state: (absent — not orphaned) · hub_escrow_identity_present: true · claimed: true agent 0.127.0 · controller 0.206.0 (healthy) · guests 1/1 ``` **Hub:** ``` host_escrow(walk5-4bada5): blob 383 B · identity_blob 572 B (R-198 material present) key_fingerprint a6:86:f7:fb:…:4c:f9 restic_pw_sha256 eabf427c7274…144f stale_at NULL · created 2026-08-07T12:51:02Z host_escrow_superseded: 0 rows for walk5 hosts: walk5-4bada5 ONLINE · agent 0.127.0 · PBS wrapper MATCHES VOUCHED (104db0a4401f…) · WG 10.77.0.5 confirmed DR Recipe: present · Key Escrow: present ``` **Phase A gate: PASSED.** All seven records taken; the three sentinels listed **by name**. --- ## Phase B — the journey (§7) **The rule: no command line inside the guest, at any point.** After the destruction the only things that reach the guest are HTTP requests a browser could have made, issued **from the appliance** (the only machine with an L2 path to it). **Instrumentation reads are counted separately and named**: the `w5watch.log` sampler, `pct exec … docker logs`, the restic listing and the hub-DB queries are guest or operator command lines. **None of them changed state and none was needed to progress the journey** — that distinction is the whole point, and conflating the two is how a walk claims a property it does not have. ### B1 — the destruction (14:19:26Z) Guarded **four ways** before anything ran, because `demo-hp` also carries a guest 9201: ``` hostname == walk5 ✓ pct config 9201 → hostname: walk5 ✓ mountpoint -q /mnt/adatok ✓ mountpoint -q /mnt/mentes ✓ ``` ``` 14:19:18Z pct stop 9201 14:19:26Z pct destroy 9201 --purge → "Logical volume vm-9201-disk-0 successfully removed" → "Logical volume vm-9201-disk-1 successfully removed" → pct list EMPTY · lvs | grep 9201 → NONE 14:19:27Z /mnt/adatok and /mnt/mentes wiped 13M/1.2G → 4.0K each, MOUNTS LEFT IN PLACE — deliberately: the surviving raw mount IS the R-220 condition ``` ### B2 — the rebuild (14:24:09Z, 3 m 31 s) `felhom-host-install.sh` fetched **live** from `felhom.eu/scripts/` — `v1.25.0`, sha256 `cb4d02059f126f8bfdddfe98607e87a3cb45dd2592c6b40b43c9659e2b687534`. The retrieval passphrase was passed as `--passphrase-file` (a `0600` file), never on the command line. ``` [OK] Day-0 provision SUCCESS — vmid=9201 host_id=walk5-4bada5 customer=walk5 golden=local:backup/vzdump-lxc-9100-2026_08_07-16_21_39.tar.zst [OK] controller: Up 19 seconds (healthy) [INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.206.0 [OK] authz signers: 2 (operator-signed self-update armed) [INFO] root@pam was rotated + vaulted at step 4b ``` **What the rebuild landed on:** | | before | after | |---|---|---| | agent | 0.127.0 | **0.127.0** — no downgrade, no hand upgrade | | controller | 0.206.0 | **0.206.0** — same, and it is the vouched build | **No hand upgrade, either time.** R-216 part 4's reinstall-downgrade did not recur. Guest re-provisioned at **192.168.0.83** (it was .149 before — a fresh DHCP lease, not a fault). ### B3 — §5's observation window: THE MOMENT THAT MATTERS **T0 — 14:24:31Z, 22 seconds after provision, nobody has logged in:** ``` offbox directory ABSENT ("No such file or directory") repository key NONE settings.offbox null settings.hub_escrow_identity_present TRUE ← the box already knows a package is held for it settings.hub_escrow_key_sha256 eabf427c7274…144f ← and already knows WHICH key it seals settings.claimed null (pristine, unclaimed) ``` **The scheduler is alive — the positive control, because an absent line is not evidence.** In the first five minutes the controller logged `agent-channel-health` ×5 (1 m cadence), `stack-scan` ×2, `system-health`, `backup-cache` and `offsite-credential-retry`. So "no key was written" is a measurement, not a silence. **The window, minute by minute — nobody logged in at any point during it:** | time (UTC) | event | source | |---|---|---| | 14:24:09 | Day-0 provision SUCCESS; rebuilt guest, controller **0.206.0** | installer | | 14:24:31 | **T0** — no offbox dir, no key, `offbox: null`, but `hub_escrow_identity_present` already **true** and the sealed key's hash already learned | box, **measured** | | 14:24:53 | hub: `offsite_delivery_stuck` (warning) + operator `notification_log` row — **REFUSED**, "regressed-apply shape … a guest rebuild does, R-193" | hub, **measured** | | 14:23:54 | first post-rebuild report carries **no** `offsite` object — it predates the ACK that set the flag | hub `reports`, **measured** | | **14:38:49** | **first declaration**: `offsite={"enabled":false,"state":"needs_credential"}` | hub `reports`, **measured** | | 14:38:53 | the checker **hands off**: *"the box DECLARES offsite.state=needs_credential; internal/offsiteheal owns this remediation"* | hub, **measured** | | 14:53:49 | second declaration, same shape | hub `reports`, **measured** | | **14:56:34** | `offsiteheal: re-staged the stored one-time offsite secret … (declared needs_credential across 2 reports) … no provider credential was minted` | hub, **measured** | | **14:58:52** | **the box collects it and applies the tier — AND REFUSES TO MINT** | box log, **measured** | ### THE LINE THIS WALK EXISTED TO SEE ``` 14:58:52 [WARN] [offbox] NOT minting a repository password: the hub holds a sealed recovery package for this box, and a fresh key would orphan the history that package protects (R-241). The transport is configured; the tier stays down until the customer's recovery code places the escrowed key. 14:58:52 [INFO] [offbox] apply-offsite: transport configured for u629488-sub4@…:/home/felhom-repo, tier HELD awaiting the escrowed key (the hub holds a sealed package; no key was minted — R-241) 14:58:52 [INFO] [offsite-apply] credential retry: the staged credential was collected and the tier applied ``` **At the equivalent moment the previous walk minted `9b4a9a9d…` over a package sealing `30ef574f…`, and lost the journey silently at 03:18 in the morning. This build did not.** ### §5's four items, answered **1. Does the box declare a need, and when is it staged and collected?** **Yes** — declared `needs_credential` at **14:38:49Z** and **14:53:49Z**, staged by the hub unaided at **14:56:34Z**, collected and applied at **14:58:52Z**. **Zero human actions**; the whole chain ran on a box that had not even been claimed. **2. Is any repository key written?** **NO.** Sampled every ~20 s from 14:24:52Z onward: `repo_password_sha=NONE` at every single sample. At 15:00:24Z the directory holds exactly: ``` -rw------- applied_marker 64 -rw-r--r-- known_hosts 290 -rw------- ssh_key 400 repo_password: ABSENT ``` **The transport is present and the key is not** — which is the guard's designed shape, not a partial failure: `ApplyOffsiteTarget` deliberately keeps the transport so the recovery screen can bring the tier up the instant the key arrives (R-219). **3. What state does it report instead?** `offbox.enabled = true`, `escrow_state = pending`, `repo_state` absent — and `OffboxAwaitingRecoveryKey()` is therefore **true** (target exists, target enabled, no password), so the box declares the new **`awaiting_recovery_key`** holding state rather than coming up under a fresh key. **4. The two fingerprints, before anyone logged in (15:00:24Z):** ``` the escrowed key (hub's package, ACK-cached) eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f whatever the box holds NONE — there is no repository key on this box ``` > **HONEST NOTE ON WHICH SHAPE FIRES, because it changes what has been proven.** With the mint guard > holding, the box has **no local key at all**, so `OffsiteRecoveryOffer` returns true on **shape (a)** > — the pristine-rebuild shape — not on shape (c). **Shape (c) was measured in Phase A, in its > negative half**: equal hashes, correctly silent. So this walk proves the mint guard **positively** > and the discriminator **negatively**; a positive shape-(c) firing needs a box holding a *different* > key, which v0.206.0 now prevents from arising by itself. That is the intended end state (the fix > removes the state its own second half was written to describe), but it should not be reported as > "shape (c) proven". ### B4 — the journey, step by step (§7) | # | step | result | |---|---|---| | 1 | **Destroy** — 14:19:26Z | guest 9201 purged (both LVs), both drives wiped 13M/1.2G → 4.0K. Host identity `walk5-4bada5` survived | | 2 | **Reinstall** | `felhom-host-install.sh` **v1.25.0** fetched live; **Day-0 provision SUCCESS 14:24:09Z**, 3 m 31 s. **controller 0.206.0, agent 0.127.0 — no hand upgrade, no downgrade** | | 3 | **§5's window** | **no key minted**; transport configured; `awaiting_recovery_key` — see B3 | | 4 | **Claim back** | new code requested through the customer-facing „Új kód kérése" path (15:01:27Z), hub rotated to **generation 2** and emailed it; the operator relayed it; **worked FIRST TRY** — 302 in **0.151 s**, accents intact (`á`=`c3a1`, `í`=`c3ad`, sha256 identical source→appliance) | | 5 | **Log in** | `/` → 302 `/launcher` → 302 **`/recovery`** — **the recovery screen appeared WITHOUT BEING SOUGHT** | | 6 | **Read the screen** | all three questions answered — quoted below | | 7 | **Enter R** | **HTTP 200 in 12.44 s** — a real unseal. `[web] recovery: the offsite repository key was recovered and placed (outcome=installed)` | | 8 | **The listing** | **rendered** — `calibre-web · 2026-08-07 14:57 · 12.8 MB` | | 9 | **Restore** | **all three sentinels byte-identical** — after two unsignposted obstacles, both cleared from the dashboard | ### The recovery screen, as a customer reads it (§7.6) > **„Adatok visszaszerzése"** > > „Ezt a gépet újratelepítették. **A korábbi, házon kívüli mentéseid megvannak** — a Felhom központi > rendszere őriz hozzájuk egy lezárt csomagot, amelyet **2026-08-07T12:51:02Z** zártunk le. A csomagot > csak a **te helyreállítási kódoddal** lehet kinyitni. > > **A helyreállítási kódot senki nem tudja pótolni** — sem a Felhom, sem az ügyfélszolgálat, sem az > üzemeltető. Ez szándékos: így a mentéseidet rajtad kívül senki nem tudja megnyitni. Ha a kód > elveszett, a korábbi mentések nem nyithatók meg többé. > > Ha megadod a kódot, **feloldjuk a mentéseid zárolását és megmutatjuk, mi van bennük** — melyik > alkalmazás, mikorról, mekkora. **Ebben a lépésben semmit nem állítunk vissza és semmi nem változik.**" **All three questions answered.** The sealed-at timestamp **`2026-08-07T12:51:02Z` matches the hub's `host_escrow.created_at` exactly**. The field label reads „Helyreállítási kód (tíz szó)" and R is indeed ten tokens. *(One copy wart, recorded not filed: the sealed-at is rendered as a raw ISO-8601 string on a customer-facing Hungarian screen, where every other date on the box is formatted `2026-08-07 14:57`.)* ### The listing, against Phase A's record (§7.8) | | Phase A recorded | the screen showed | |---|---|---| | app | `calibre-web` | **`calibre-web`** ✅ | | when | snapshot `5b0f20f7` at 12:57:41Z | **`2026-08-07 14:57`** ✅ (12:57Z rendered in CEST) | | size | `restic stats` 12.784 MiB | **`12.8 MB`** ✅ | *(A second row `felhom-offbox · 12.8 MB` also appears — the snapshot carries both tags and the screen renders one row per tag. A customer sees an "app" they have never installed, and the same 12.8 MB counted twice. Cosmetic; filed as R-251.)* ### THE RTO (§7.7) ``` login 15:04:25.417Z unlock returns 200 15:05:37.156Z ------------------------------------------ wall clock, login → open store 71.7 s ``` **Of which 12.44 s was the unseal itself.** ~22 s of the remainder was **my own harness retry** — I scraped the CSRF token from a `` tag the recovery page does not carry, got a 403, and re-read it from the form (the controller logged `CSRF rejected … (token mismatch)` at 15:05:02). **A customer clicking the button would have seen ≈50 s.** Both numbers are stated because the 71.7 s is what was actually measured. ### The restore, and the two obstacles (§7.9) `POST /backup/offbox/restore` (mode=full) → **refused**: „nincs elérhető adatmeghajtó a visszaállításhoz". Then, after that was cleared, `POST /backup/offbox/reconstitute` → **refused**: „a(z) calibre-web nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat". Both were cleared **from the dashboard, with no guest shell**: 1. **Re-attach the drives** — `GET /api/disks/candidates` offered both surviving disks (`mountable: true`, `data_bearing: true`), and Tárhely → Meghajtók → „Meglévő meghajtó csatolása" (`POST /api/storage/attach`) registered them: `/mnt/felhom-drives/adatok` and `…/mentes`. 2. **Redeploy the app** — `POST /api/stacks/calibre-web/deploy`, running in 1 m 30 s. Then prepare (12.8 MB) → confirm → reconstitute: ``` 15:10:18 [offbox] restored calibre-web (5b0f20f7, full=true) → …/backups/offsite-restore/calibre-web 15:14:30 [offbox] reconstituted calibre-web from snapshot 5b0f20f7: 6 file(s) placed, 0 DB dump(s) replayed, safety dump=., skewed=false ``` ### THE THREE SENTINELS, read back as BYTES from the live path ``` MATCH [A] 11eb7fb2d1891f62a685e3d9a9fda44e0a342d437d4aebaa15fcb4c0fd79d539 61 name_hex 57414c4b352d53454e54494e454c2d412e747874 MATCH [B] 6b504d1e83bf4c673334d0b9e701f3e5f07311d954fdc4463a6b0a55b261ae32 66 name_hex 57414c4b352d c591 72737a656d2d c3a9 6b657a657465732d c3a1 7276 c3ad 7a74 c5b1 72 c591 2e747874 MATCH [C] 0baaf402733e6a85f3d9ce8c4c3a57b05692f5f6733da6a23da76d2cd7635210 12582912 name_hex 57414c4b352d53454e54494e454c2d432d31324d422e62696e all three present: True ``` **Identical to Phase A in every byte, including the accented filename.** Read with `os.listdir` on a **bytes** path, so no decode/encode round trip could launder a `U+FFFD` into looking correct. ### The end state — the loop closed ``` escrow_state escrowed (auto-confirmed: hub hash == local hash) repo_password sha256 eabf427c7274…144f == the escrowed key, == Phase A's key GET /launcher 200 (the recovery screen no longer interrupts) GET /recovery 302 (correctly retired — shape (c) compares equal again) ``` --- # THE VERDICT — both halves, separately ## THE DATA: **PASS** All three sentinels came back **byte-identical**, out of snapshot `5b0f20f7`, under the key recovered from the sealed package with R — including the accented filename's **bytes**. Disk → restic → SFTP → Storage Box → a destroyed machine → a rebuilt machine → disk, intact. ## THE JOURNEY: **PASS — the first time in five walks** **No step needed a command line inside the guest.** Every step that *progressed* the journey — request a code, claim, log in, read the screen, enter R, read the listing, re-attach the drives, redeploy the app, prepare, confirm, restore — was an HTTP request a browser makes, issued from the appliance because DooPlex has no L2 path to that LAN. **Guest command lines used, named per §3 so the claim is not read wider than it is:** the `w5watch.log` sampler, `docker logs`, the settings reads, the restic listing and the final sentinel verification. **Every one is instrumentation** — none changed state, none was needed to progress, and removing them all would have changed nothing except my ability to describe what happened. The previous walk needed **three** guest command lines to progress; this one needed **zero**. **The reset-code hatch was used ONCE, in Phase A**, where §3 permits it. It was **not** used in Phase B. ### But the journey is not smooth, and two obstacles are new Neither is a dead end by §3's definition (neither needed a guest shell), and both were cleared from the dashboard — **but a customer would not know to do either**, because the product does not say so and, in the second case, says the opposite: 1. **The restore refuses with „nincs elérhető adatmeghajtó" and nothing on the recovery path mentions that the drives must be re-attached first.** → **R-252** 2. **„a(z) calibre-web nincs telepítve — előbb állítsd helyre az alkalmazást" — on a page that says, three lines above, „Nincs telepítve — a visszaállítás előbb újratelepíti."** The product contradicts itself in the customer's own language, at the last step of a recovery. → **R-253** **So: the machinery works end to end and the data is provably safe. The unaided journey now succeeds, and it succeeds through two obstacles the customer must guess their way past.**