Files
felhom.eu/documentation/tests/walk5-r201-2026-08-07/journal.md
T
admin 879007aaec
gates / gates (push) Successful in 21s
walk5 (R-201, fifth walk): Phase A journal — fixture built, sentinels proved by name
Written before the destruction, per §9.9. A fresh install landed on the VOUCHED
set with no hand upgrade (controller 0.206.0, agent 0.127.0) — R-239's delivery
gap is closed for this run, which is the first of the five walks where the box
under test is the box a customer receives.

Both §4.6 pre-destruction checks pass, neither previously exercised on a clean
box: the recovery offer is correctly SILENT (shape (c) compares equal — the two
key hashes are byte-identical on box and hub), and the restore page lists the app
with the future-backup toggle OFF (R-237's fix, which the last walk measured
failing).

Also recorded: the §4.5 gate caught a harness fault (a toggle sent as enabled=1
rather than enabled=on) that had produced a green 'ok' over a zero-snapshot
repository — the exact shape the gate exists for.
2026-08-07 15:05:35 +02:00

414 lines
21 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# THE FIFTH WALK (R-201) — journal
**Venue:** `demo-hp` VM **325 `walk5-appliance`**, hub customer **`walk5`**. All four earlier venues
(`c11`, `rewalk`, `part4`, `finalwalk`) were torn down on 2026-08-06/07; **nothing was reused** except
the `c11-scratch` dir storage (a plain `dir` at `/mnt/nvme-1tb`, which is what the agent's
`exactMount` check requires) and the **freed** Storage Box sub-account *number* `sub4` — the account
itself is newly provisioned, id **285351**, not finalwalk's 285071 which the cascade destroyed.
**Written as the walk proceeds, per §9.9.** Clocks: DooPlex and demo-hp and the appliance = **CEST**;
the guest and the hub = **UTC**.
---
## §02 — orientation, on arrival
### §1 baselines — re-read, all four match the runbook exactly
| Component | Runbook | Measured on arrival | |
|---|---|---|---|
| `felhom-controller` | `3d3b4496f377` | `3d3b4496f377` | ✅ clean tree, == `origin/main` |
| `felhom-agent` | `703db166e762` | `703db166e762` | ✅ clean tree, == `origin/main` |
| `felhom.eu` | `10af63d0b04e` | `10af63d0b04e` | ✅ clean tree, == `origin/main` |
| `app-catalog-felhom.eu` | — | `459766cb1639` | clean, untouched |
**No drift.** (The two previous walks both found one-commit drift on arrival and had to check it was
benign; this one did not.)
### §1 the vouched set — read from the hub, not from a document
`GET /configuration` on the hub (Basic auth, ClusterIP):
```
agent_version (selected) 0.127.0
golden_version (selected) 0.206.0
min_agent 0.127.0
min_controller_version 0.156.0
hub image (deployed) gitea.dooplex.hu/admin/felhom-hub:0.98.0
installer SCRIPT_VERSION 1.25.0 (served live from felhom.eu/scripts/)
```
**All five match §1.** The vouched golden `0.206.0` carries controller `0.206.0`, so **this is the
first walk whose fresh install lands on the fixes it exists to exercise** — R-239's delivery gap,
which was the headline finding of the last two walks, is closed for this run.
### §1 the build gate — GREEN before starting
```
$ python3 scripts/golden_currency_gate.py
newest released controller : 0.206.0
newest golden baked : 0.206.0
golden currency gate OK
rc=0
```
**Stated precisely, because the gate says so itself:** it checks the **bake**, not the **vouch**. The
vouch was therefore checked separately, from the hub, above — and it is `0.206.0`. Both halves green.
### §1 highest register ID
**R-248**, confirmed by grep over `OPEN-ITEMS.md` (`R-241 … R-248`). The runbook's figure is current.
### ISO — verified by round trip, not by name
| | |
|---|---|
| published | `https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso` · 1 705 322 496 B · `last-modified Fri 31 Jul 2026 18:37:18 GMT` |
| published `.sha256` | `f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6` |
| local on `demo-hp` | `/var/lib/vz/template/iso/felhom-installer-1.26.1-pve9.2-1.iso` · **same sha256** |
**Byte-identical.** (`https://iso.felhom.eu/` **itself** 404s — there is no index object; only named
objects are served. Recorded so the next session does not read that 404 as an outage.)
### §2.6 — the open items that could touch this walk
- **R-247 / R-248** — the staleness flag. The runbook says no running code can set it; **confirmed at
source**: R-246's census found `MarkEscrowStale` has **no production caller anywhere in the tree**,
and its one historical setter was removed in hub v0.95.0. A fresh box cannot reach the state.
**R-246 also records the demo-hp instance as CLEARED on 2026-08-07.** So shape (c) is live and
un-inhibited for this venue.
- **R-243** — the alarm gap for a box held in `awaiting_recovery_key`. **Directly in this walk's
path**: if §5 goes as designed, the rebuilt box enters exactly that state. The gap is that *the hub
raises nothing for it*. Not a defect of the walk; watch for it and report it as still-open.
- **R-240** — „Sikeres" over a run that covered nothing. Only reachable if an off-site run is made
with nothing selected; not part of this walk's route.
- **R-244** — teardown residue in `app_log_issues`. Teardown is §10, owed, not this session.
### §2.7 — the two functions this walk exercises, read before running
Both fixes are present in the tree at `3d3b4496f377` and both are new since the last walk:
- **the mint guard** — `WriteOffboxSecrets` (`controller/internal/backup/offbox.go:438`) now returns
`ErrOffboxSealedPackageHeld` instead of generating, when `sealedPackageHeld()` is true **and** no
password exists. It is a **conjunction**, so a first-time box mints exactly as before. The transport
is still written. New declared state `offsite.state=awaiting_recovery_key`
(`OffboxAwaitingRecoveryKey`, `:502` — derived, not stored, and gated on `t.Enabled`).
- **shape (c)** — `OffsiteRecoveryOffer` (`:1580`) now offers when
`hubHash != "" && hubHash != localHash`. `""` falls back to (a)/(b) deliberately.
**Neither has ever run on a real rebuild.** That is §5, and it is deliverable 2 whatever the verdict.
---
## Venue construction
### Hub customer `walk5`
`POST /configs/new` — id `walk5`, name "Walk 5 R-201", domain `walk5.felhom.eu`, email = the
operator's registered address, **off-site ON** (shared, 50 GB), **DR tier ON**, `pbsdr_storage_id`
`felhom-pbs`. → **303** `/customers/walk5?flash=created`.
Provisioned descriptor, read from the hub DB rather than from the page:
```json
{"enabled":true,"type":"shared","host":"u629488-sub4.your-storagebox.de","user":"u629488-sub4",
"port":23,"repo_path":"/home/felhom-repo","quota_gb":50,
"host_fingerprint":"SHA256:oDHZqKXnoMtgvPBjjC57pcuFez28roaEuFcfwyg8O5c"}
```
`one_time_secrets` 1 · `customer_configs` 1 · `hosts` 0. Sub-account **285351**, user `u629488-sub4`.
> **⚠ THE FIRST CREATE FAILED, AND IT IS A FINDING — not a fluke of my own making.**
>
> The first `POST /configs/new` returned no response to the client and the hub logged
> `[ERROR] offsite provision for walk5: offsite: host-key scan u629488-sub4.your-storagebox.de:
> dial: dial tcp [2a01:4f8:bacc:2:200::d30]:23: connect: network is unreachable` at **13:46:46**.
>
> The scan is **fail-closed by design** (`offsite.go:111-121`: *"don't serve a descriptor the
> controller can't verify"*) and its retry ladder is **~60 s** (`defaultScanBackoff`, 2+4+8+16+30),
> sized by its own comment to *"the observed DNS propagation lag"* of a fresh sub-account. Here the
> lag was **longer than the budget**: the first five attempts failed `no such host`, and by the time
> DNS answered it answered **AAAA-first**, into a pod with no IPv6 route.
>
> **The retry (a second identical POST) succeeded** — idempotent on the label
> (`[offsite] shared already provisioned for walk5 (subaccount 285351)`), scan green on its own final
> rung at ~13:47:28, config created. **Total settle time ≈ 100 s against a 60 s budget.**
>
> **Measured afterwards, so the cause is not guessed:** the hub pod reaches
> `u629488-sub4.your-storagebox.de:23` **now**, by name, and gets `SSH-2.0-OpenSSH_9.6p1` — as does
> the node. So the endpoint is fine and the window was transient. **What is not fine is the operator
> experience**: a customer create can fail with a fail-closed error whose only remedy is "press it
> again", and nothing says so. Filed below.
### VM
| | |
|---|---|
| Host | `demo-hp` (HP t740), Tier 0 — the designated drill host |
| VM | **325 `walk5-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` |
| Disks | `scsi0` 200 G · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c11-scratch` (a `dir` at the **mount root** `/mnt/nvme-1tb`, which is what the agent's `exactMount` check requires) |
| Appliance | `walk5.felhom.eu` @ **192.168.0.100/24**, gw/DNS 192.168.0.1, MAC `bc:24:11:37:7c:f3` |
| Guest | LXC **9201** @ **192.168.0.149** (LAN) + `169.254.253.2` (R-50 island) |
| Hub customer | **`walk5`** "Walk 5 R-201", DR tier ON, off-site ON (shared, 50 GB) |
| Host id | **`walk5-4bada5`** · appliance uuid `596d8104-2af9-4525-baa7-e479314cbb44` |
| Off-site | Storage Box sub-account **285351**, user `u629488-sub4` · WG peer **10.77.0.5** |
| **Untouched** | `drill-r50` (VM 300, the protected fixture), guest 9201 on both demo boxes, DooPlex, `ep0`, the tester's box |
**Reachability, stated because it shapes every later step:** DooPlex has **no L2 path to the
192.168.0.0/24 LAN** — `ip neigh` is `INCOMPLETE` even for `demo-hp`, which it reaches over the
tailnet. So the appliance is one SSH hop behind `demo-hp`, and every request to the guest is made
**from the appliance**, which is exactly what a browser on the customer's LAN would do.
---
## Phase A — the fixture (§4)
### A1 — installed from the published ISO, through the Terminal UI
Driven with `qm sendkey` + `qm monitor screendump` → PNG. **All three known traps reproduced:**
1. **GRUB defaults to the graphical entry — and it bit, first attempt.** The countdown was **14 s**
with my screendump→convert→read round trip at **~18 s**, so the menu auto-booted the default and
the GUI installer came up. **Recorded rather than quietly retried:** the VM was stopped and
restarted, and the keys sent **inside one remote command** (`sleep 6; sendkey down; screendump;
sendkey ret`) — the screendump taken *between* the arrow and the Enter is the positive control,
and it shows the highlight on **„Felhom telepítés (szöveges mód)"** at the moment of the press.
2. **The keymap defaults to Hungarian** while `sendkey` emits US scancodes. Changed to **U.S.
English** before any typing. **Positive control:** the administrator email typed through the
identical path rendered **`walk5@felhom.eu`** — `@` is `shift-2` on US and `AltGr+V` on HU, and it
is the only available evidence for the 24 masked password characters.
3. **Auto-reboot was checked.** Unchecked, and **confirmed `[ ]` with the focus moved away**; then
`--boot` set in its own `qm set` with the ISO detached, both verified from `qm config`
**before the first boot**: `boot: order=scsi0`, `ide2` lines = **0**.
Summary screen, verbatim: `ext4` · `/dev/sda` · `Europe/Budapest` · **`U.S. English`** ·
`walk5@felhom.eu` · `nic0` · `walk5.felhom.eu` · `192.168.0.100/24` · `192.168.0.1` · `192.168.0.1`.
**One reading corrected by a second instrument, as in the last walk:** `192.168.0.100` was the
installer's prefill and answered on the LAN. The **MAC** was `bc:24:11:37:7c:f3` — VM 325's own DHCP
lease. Not a collision; a ping alone could not have told the difference.
**Day-0 fired on first boot, unaided.** Console showed the Hungarian pairing banner:
> `Felhom — a doboz készen áll, és a párosításra vár.` · `Párosító kód: E28-WJH`
The hub's `appliance_registrations` row **24** carried the same code (`E28WJH`), the same MAC, the
same SMBIOS uuid and **three SSH host keys**, `first_seen 12:18:59Z`**32 seconds after boot**.
Bound through the real endpoint (`POST /appliances/24/bind`, **303**).
### A1b — THE HEADLINE: a fresh install DOES get the fixes
| | vouched | the box landed on |
|---|---|---|
| agent | 0.127.0 | **0.127.0** ✅ |
| golden | 0.206.0 | — |
| controller | (baked into the golden) | **0.206.0** ✅ |
```
14:22:48 [INFO] Artifact manifest served for customer walk5 (agent=0.127.0 golden=0.206.0)
14:25:33 [INFO] Event from walk5: controller_started (info) — Controller elindult (0.206.0)
docker ps → Up 10 minutes (healthy) gitea.dooplex.hu/admin/felhom-controller:0.206.0
felhom-agent --version → felhom-agent 0.127.0
```
**No hand upgrade was needed and none was applied.** This is the first walk of the five where that
sentence is true of the *fixes under test*: R-239's delivery gap — the headline finding of both
previous walks — **is closed for this run**, and the box under test is the box a customer receives.
**The whole day-0 cascade ran hands-free**, bind → provision in **3 m 55 s**:
```
14:21:38 appliance 24 BOUND to customer walk5 (mode=appliance)
14:22:00 appliance credentials DELIVERED once
14:22:08 [claim] claim code (gen 1) emailed to the registered address
14:22:38 host enrolled: walk5-4bada5
14:22:46 offsite credentials re-issued on re-enroll; break-glass root@pam vaulted (32 chars)
14:23:19 wg registered 10.77.0.5 · DR-recipe host-half · tenantsync provision ok (ns=walk5)
pbsdr auto-provisioned (hands-free cascade)
14:25:33 controller 0.206.0 reporting
```
### A2 — claimed
Claimed through the real `POST /claim` form (302 → `/`), then `POST /login` → 302. **The claim code
came from the documented `--print-reset-code` escape hatch** (R-204 item 1) — **a guest command line,
used deliberately as FIXTURE CONSTRUCTION, permitted by §3 in Phase A, and counted as such.** Phase B
must not use it.
The dashboard password is a 24-character alphanumeric, `0600` in `~/.config/walk5/` on **DooPlex
only**. The appliance's root credential is the hub-vaulted break-glass (`root@pam`, revealed through
`POST /hosts/walk5-4bada5/reveal-recovery-credential`) — **the password typed at the installer no
longer authenticates, because day-0 step 4b rotates and vaults it.** Stated because it cost a dead
end.
### A3 — drives, the app, and the three sentinels
Both raw disks enrolled through the **customer** endpoints (`GET /api/disks/candidates`
`POST /api/storage/init`), ext4:
```
/dev/sdb → /mnt/felhom-drives/adatok (app data)
/dev/sdc → /mnt/felhom-drives/mentes (backup target)
```
Backup target assigned to `mentes` — and, exactly as the last walk recorded, it reported
`restart_required: true` and flipped only after the agent restart it asked for:
```
before: {"degraded":true, …"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after: {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}
```
`calibre-web` deployed through the real API with `HDD_PATH=/mnt/felhom-drives/adatok`
`state: running` **and `health_probe.healthy: true`**.
**THE THREE SENTINELS**
| # | file | bytes | sha256 |
|---|---|---|---|
| A | `WALK5-SENTINEL-A.txt` | 61 | `11eb7fb2d1891f62a685e3d9a9fda44e0a342d437d4aebaa15fcb4c0fd79d539` |
| B | `WALK5-őrszem-ékezetes-árvíztűrő.txt` | 66 | `6b504d1e83bf4c673334d0b9e701f3e5f07311d954fdc4463a6b0a55b261ae32` |
| C | `WALK5-SENTINEL-C-12MB.bin` | 12 582 912 | `0baaf402733e6a85f3d9ce8c4c3a57b05692f5f6733da6a23da76d2cd7635210` |
Sentinel B's filename **as hex**, and this is the reading that cannot lie:
```
57414c4b352d c591 72737a656d2d c3a9 6b657a657465732d c3a1 7276 c3ad 7a74 c5b1 72 c591 2e747874
ő é á í ű r ő
```
**No `efbfbd` anywhere.** Written by a Python placer pushed in with `pct push`, which builds the name
from a **hex string decoded to bytes** and passes a **bytes path** to `os.open` — no shell layer ever
sees the name, and the readback is from `os.listdir` on a bytes path, not from a shell `ls`. This is
the trap that bit three times; it did not bite here.
**Placed under `…/adatok/userdata/media/books` — the level the last walk got wrong.** The tree was
listed first and the namespace root confirmed to be `/mnt/felhom-drives/adatok` **directly**.
### A4 — the escrow ceremony
Preflight **6 of 6 green** (`pbs_storage_id` · `dr_tier` · `age_binary` · `hub_upload` ·
`staged_secret` · `sudo_grant`), `agent_supported: true`, **`stale: false`**.
> **That `stale: false` is a deliberate check, not a formality.** R-246/R-247/R-248 concern a
> staleness flag that would silently disable the very discriminator this walk exists to exercise.
> The spike's ruling was that a fresh box cannot reach the state (`MarkEscrowStale` has no production
> caller). **Measured here on a fresh box: it has not.**
Ceremony through the customer wizard's own endpoints (`/api/escrow/start` with password re-auth —
it is a **form** POST, not JSON, which cost one rate-limited attempt):
```
phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24 · claimable: true
key_fingerprint: a6:86:f7:fb:74:e7:16:3c:…:31:2a:4c:f9
```
**`restic_pw_sealed: true` is the field the whole exercise rests on.**
**R was claimed ONE-SHOT and streamed file→file** into `~/.config/walk5/R_walk5.txt` (`0600`,
**DooPlex only**). It was never rendered, never an argument, never a log line. The appliance's copy
was `shred -u`'d and **its absence verified**. Shape only: **82 characters, 10 hyphen-separated
tokens, lowercase+digits**.
> **§9.4 slip, recorded rather than buried.** The Phase-A *claim code* (not R) reached this session's
> transcript, because the hatch prints it and my shape-only filter was written for an uppercase
> pattern and did not match a lowercase one. It was consumed by the claim seconds later and is now
> void. **R itself was never printed and never left the file→file path.**
### A5 — the off-site backup, and the sentinels BY NAME
> **THE §4.5 GATE EARNED ITS PLACE AGAIN — and this time it caught MY fault, not the product's.**
> The first run reported **`ok` in 28 s** with **`snapshots = 0`** and an empty repository size. The
> log said why: `[offbox] backup run started (0 app(s) toggled)`. **My toggle POST sent
> `enabled=1`**, and the handler accepts only `on`/`true` — so it recorded *off*, redirected with
> „A távoli mentés beállítása frissítve." and the run then correctly backed up nothing.
> **A harness fault, not a product defect** — but the shape is precisely R-240's, and had I read the
> green `ok` and moved on, the destruction would have proven nothing.
Re-toggled (`enabled=on`), selection verified in the rendered page (`name="app" value="calibre-web"`
+ `checked`) **before** the re-run, which then logged `backup run started (1 app(s) toggled)`.
**`ok`, 1 m 20 s, 1 snapshot, 12.0 MB (12 611 563 B).** **The gate is not the green tick**
`restic snapshots` + `restic ls latest --long`, run against the repository with its own credentials:
```
5b0f20f7 2026-08-07 12:57:41 walk5 [felhom-offbox, calibre-web]
/mnt/felhom-drives/adatok/backups/primary/calibre-web
/mnt/felhom-drives/adatok/userdata/media/books
-rw-r--r-- 1000 1000 61 …/userdata/media/books/WALK5-SENTINEL-A.txt
-rw-r--r-- 1000 1000 12582912 …/userdata/media/books/WALK5-SENTINEL-C-12MB.bin
-rw-r--r-- 1000 1000 66 …/userdata/media/books/WALK5-őrszem-ékezetes-árvíztűrő.txt
-rw-r--r-- 1000 1000 413696 …/userdata/media/books/metadata.db ← calibre-web's real DB
restic stats latest: 22 files, 12.784 MiB
```
**All three sentinels are in the snapshot, by name, at the right sizes — and the accented filename
survived into restic intact.**
### A6 — §4.6's two pre-destruction checks, neither ever exercised on a clean box
**Check 1 — the recovery offer is NOT shown. PASSES.**
```
GET / → 302 → /launcher
GET /launcher → 200, recovery mentions: 0
GET / → 200 (followed), recovery mentions: 0
GET /recovery → 302 → /backups/remote
```
No banner, no entry point, nothing to seek. **And the reason is measurable rather than assumed:** the
hub's package covers the key in use, so v0.206.0's **shape (c) compares equal and correctly stays
silent** —
```
box /data/offbox/repo_password sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
box settings.hub_escrow_key_sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
hub host_escrow.restic_pw_sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
hub_escrow_key_checked_at = 2026-08-07T12:51:39Z
```
**This is the first live measurement of the persisted discriminator**, and it is the negative half:
the shape that must NOT fire, not firing, for the right reason.
**Check 2 — the restore page lists the app with the future-backup toggle OFF. PASSES.**
With `calibre-web` toggled **off** for future off-site backups, `/backups/restore` renders
**identically** — „Calibre-Web Automated" still listed under **„Visszaállítás a távoli tárolóból"**,
10 `calibre` occurrences in the HTML both before and after.
**That is R-237's fix, live.** The last walk's T3 measured **0 restore entries and a 302** on
controller 0.203.0 — the customer's existing backup hidden by a setting about the future. The toggle
was returned to **on** afterwards.
### A7 — the pre-destruction truth (§4.7's seven records)
**Box** (`settings.json`, secrets stripped):
```
offbox: enabled true · escrow_state "escrowed" · last_status "ok" · last_duration 1m20s
last_run/last_success 2026-08-07T12:58:57Z · snapshot_count 1
repo_size_bytes 12 611 563 ("12.0 MB") · stats_known true · quota_gb 50
host u629488-sub4.your-storagebox.de:23 · repo_path /home/felhom-repo
repo_state: (absent — not orphaned) · hub_escrow_identity_present: true · claimed: true
agent 0.127.0 · controller 0.206.0 (healthy) · guests 1/1
```
**Hub:**
```
host_escrow(walk5-4bada5): blob 383 B · identity_blob 572 B (R-198 material present)
key_fingerprint a6:86:f7:fb:…:4c:f9
restic_pw_sha256 eabf427c7274…144f
stale_at NULL · created 2026-08-07T12:51:02Z
host_escrow_superseded: 0 rows for walk5
hosts: walk5-4bada5 ONLINE · agent 0.127.0 · PBS wrapper MATCHES VOUCHED
(104db0a4401f…) · WG 10.77.0.5 confirmed
DR Recipe: present · Key Escrow: present
```
**Phase A gate: PASSED.** All seven records taken; the three sentinels listed **by name**.
---