felhom.eu/documentation/tests/walk5-r201-2026-08-07/journal.md

THE FIFTH WALK (R-201) — journal

Venue: demo-hp VM 325 walk5-appliance, hub customer walk5. All four earlier venues (c11, rewalk, part4, finalwalk) were torn down on 2026-08-06/07; nothing was reused except the c11-scratch dir storage (a plain dir at /mnt/nvme-1tb, which is what the agent's exactMount check requires) and the freed Storage Box sub-account number sub4 — the account itself is newly provisioned, id 285351, not finalwalk's 285071 which the cascade destroyed.

Written as the walk proceeds, per §9.9. Clocks: DooPlex and demo-hp and the appliance = CEST; the guest and the hub = UTC.

---

§0–2 — orientation, on arrival

§1 baselines — re-read, all four match the runbook exactly

Component	Runbook	Measured on arrival
felhom-controller	3d3b4496f377	3d3b4496f377	✅ clean tree, == origin/main
felhom-agent	703db166e762	703db166e762	✅ clean tree, == origin/main
felhom.eu	10af63d0b04e	10af63d0b04e	✅ clean tree, == origin/main
app-catalog-felhom.eu	—	459766cb1639	clean, untouched

No drift. (The two previous walks both found one-commit drift on arrival and had to check it was benign; this one did not.)

§1 the vouched set — read from the hub, not from a document

GET /configuration on the hub (Basic auth, ClusterIP):

agent_version   (selected)  0.127.0
golden_version  (selected)  0.206.0
min_agent                   0.127.0
min_controller_version      0.156.0
hub image (deployed)        gitea.dooplex.hu/admin/felhom-hub:0.98.0
installer SCRIPT_VERSION    1.25.0   (served live from felhom.eu/scripts/)

All five match §1. The vouched golden 0.206.0 carries controller 0.206.0, so this is the first walk whose fresh install lands on the fixes it exists to exercise — R-239's delivery gap, which was the headline finding of the last two walks, is closed for this run.

§1 the build gate — GREEN before starting

$ python3 scripts/golden_currency_gate.py
  newest released controller : 0.206.0
  newest golden baked        : 0.206.0
golden currency gate OK
rc=0

Stated precisely, because the gate says so itself: it checks the bake, not the vouch. The vouch was therefore checked separately, from the hub, above — and it is 0.206.0. Both halves green.

§1 highest register ID

R-248, confirmed by grep over OPEN-ITEMS.md (R-241 … R-248). The runbook's figure is current.

ISO — verified by round trip, not by name

published	https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso · 1 705 322 496 B · last-modified Fri 31 Jul 2026 18:37:18 GMT
published .sha256	f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6
local on demo-hp	/var/lib/vz/template/iso/felhom-installer-1.26.1-pve9.2-1.iso · same sha256

Byte-identical. (https://iso.felhom.eu/ itself 404s — there is no index object; only named objects are served. Recorded so the next session does not read that 404 as an outage.)

§2.6 — the open items that could touch this walk

• R-247 / R-248 — the staleness flag. The runbook says no running code can set it; confirmed at source: R-246's census found MarkEscrowStale has no production caller anywhere in the tree, and its one historical setter was removed in hub v0.95.0. A fresh box cannot reach the state. R-246 also records the demo-hp instance as CLEARED on 2026-08-07. So shape (c) is live and un-inhibited for this venue.
• R-243 — the alarm gap for a box held in awaiting_recovery_key. Directly in this walk's path: if §5 goes as designed, the rebuilt box enters exactly that state. The gap is that the hub raises nothing for it. Not a defect of the walk; watch for it and report it as still-open.
• R-240 — „Sikeres" over a run that covered nothing. Only reachable if an off-site run is made with nothing selected; not part of this walk's route.
• R-244 — teardown residue in app_log_issues. Teardown is §10, owed, not this session.

§2.7 — the two functions this walk exercises, read before running

Both fixes are present in the tree at 3d3b4496f377 and both are new since the last walk:

• the mint guard — WriteOffboxSecrets (controller/internal/backup/offbox.go:438) now returns ErrOffboxSealedPackageHeld instead of generating, when sealedPackageHeld() is true and no password exists. It is a conjunction, so a first-time box mints exactly as before. The transport is still written. New declared state offsite.state=awaiting_recovery_key (OffboxAwaitingRecoveryKey, :502 — derived, not stored, and gated on t.Enabled).
• shape (c) — OffsiteRecoveryOffer (:1580) now offers when hubHash != "" && hubHash != localHash. "" falls back to (a)/(b) deliberately.

Neither has ever run on a real rebuild. That is §5, and it is deliverable 2 whatever the verdict.

---

Venue construction

Hub customer walk5

POST /configs/new — id walk5, name "Walk 5 R-201", domain walk5.felhom.eu, email = the operator's registered address, off-site ON (shared, 50 GB), DR tier ON, pbsdr_storage_id felhom-pbs. → 303 /customers/walk5?flash=created.

Provisioned descriptor, read from the hub DB rather than from the page:

{"enabled":true,"type":"shared","host":"u629488-sub4.your-storagebox.de","user":"u629488-sub4",
 "port":23,"repo_path":"/home/felhom-repo","quota_gb":50,
 "host_fingerprint":"SHA256:oDHZqKXnoMtgvPBjjC57pcuFez28roaEuFcfwyg8O5c"}

one_time_secrets 1 · customer_configs 1 · hosts 0. Sub-account 285351, user u629488-sub4.

⚠ THE FIRST CREATE FAILED, AND IT IS A FINDING — not a fluke of my own making.

The first POST /configs/new returned no response to the client and the hub logged [ERROR] offsite provision for walk5: offsite: host-key scan u629488-sub4.your-storagebox.de: dial: dial tcp [2a01:4f8:bacc:2:200::d30]:23: connect: network is unreachable at 13:46:46.

The scan is fail-closed by design (offsite.go:111-121: "don't serve a descriptor the controller can't verify") and its retry ladder is ~60 s (defaultScanBackoff, 2+4+8+16+30), sized by its own comment to "the observed DNS propagation lag" of a fresh sub-account. Here the lag was longer than the budget: the first five attempts failed no such host, and by the time DNS answered it answered AAAA-first, into a pod with no IPv6 route.

The retry (a second identical POST) succeeded — idempotent on the label ([offsite] shared already provisioned for walk5 (subaccount 285351)), scan green on its own final rung at ~13:47:28, config created. Total settle time ≈ 100 s against a 60 s budget.

Measured afterwards, so the cause is not guessed: the hub pod reaches u629488-sub4.your-storagebox.de:23 now, by name, and gets SSH-2.0-OpenSSH_9.6p1 — as does the node. So the endpoint is fine and the window was transient. What is not fine is the operator experience: a customer create can fail with a fail-closed error whose only remedy is "press it again", and nothing says so. Filed below.

VM

Host	demo-hp (HP t740), Tier 0 — the designated drill host
VM	325 walk5-appliance — q35/OVMF (pre-enrolled-keys=0), 4 cores, 8 GB, cpu=host
Disks	scsi0 200 G · scsi1 50 G · scsi2 50 G, qcow2 on c11-scratch (a dir at the mount root /mnt/nvme-1tb, which is what the agent's exactMount check requires)
Appliance	walk5.felhom.eu @ 192.168.0.100/24, gw/DNS 192.168.0.1, MAC bc:24:11:37:7c:f3
Guest	LXC 9201 @ 192.168.0.149 (LAN) + 169.254.253.2 (R-50 island)
Hub customer	walk5 "Walk 5 R-201", DR tier ON, off-site ON (shared, 50 GB)
Host id	walk5-4bada5 · appliance uuid 596d8104-2af9-4525-baa7-e479314cbb44
Off-site	Storage Box sub-account 285351, user u629488-sub4 · WG peer 10.77.0.5
Untouched	drill-r50 (VM 300, the protected fixture), guest 9201 on both demo boxes, DooPlex, ep0, the tester's box

Reachability, stated because it shapes every later step: DooPlex has no L2 path to the 192.168.0.0/24 LAN — ip neigh is INCOMPLETE even for demo-hp, which it reaches over the tailnet. So the appliance is one SSH hop behind demo-hp, and every request to the guest is made from the appliance, which is exactly what a browser on the customer's LAN would do.

---

Phase A — the fixture (§4)

A1 — installed from the published ISO, through the Terminal UI

Driven with qm sendkey + qm monitor screendump → PNG. All three known traps reproduced:

1. GRUB defaults to the graphical entry — and it bit, first attempt. The countdown was 14 s with my screendump→convert→read round trip at ~18 s, so the menu auto-booted the default and the GUI installer came up. Recorded rather than quietly retried: the VM was stopped and restarted, and the keys sent inside one remote command (sleep 6; sendkey down; screendump; sendkey ret) — the screendump taken between the arrow and the Enter is the positive control, and it shows the highlight on „Felhom telepítés (szöveges mód)" at the moment of the press.
2. The keymap defaults to Hungarian while sendkey emits US scancodes. Changed to U.S. English before any typing. Positive control: the administrator email typed through the identical path rendered walk5@felhom.eu — @ is shift-2 on US and AltGr+V on HU, and it is the only available evidence for the 24 masked password characters.
3. Auto-reboot was checked. Unchecked, and confirmed [ ] with the focus moved away; then --boot set in its own qm set with the ISO detached, both verified from qm config before the first boot: boot: order=scsi0, ide2 lines = 0.

Summary screen, verbatim: ext4 · /dev/sda · Europe/Budapest · U.S. English · walk5@felhom.eu · nic0 · walk5.felhom.eu · 192.168.0.100/24 · 192.168.0.1 · 192.168.0.1.

One reading corrected by a second instrument, as in the last walk: 192.168.0.100 was the installer's prefill and answered on the LAN. The MAC was bc:24:11:37:7c:f3 — VM 325's own DHCP lease. Not a collision; a ping alone could not have told the difference.

Day-0 fired on first boot, unaided. Console showed the Hungarian pairing banner:

Felhom — a doboz készen áll, és a párosításra vár. · Párosító kód: E28-WJH

The hub's appliance_registrations row 24 carried the same code (E28WJH), the same MAC, the same SMBIOS uuid and three SSH host keys, first_seen 12:18:59Z — 32 seconds after boot. Bound through the real endpoint (POST /appliances/24/bind, 303).

A1b — THE HEADLINE: a fresh install DOES get the fixes

	vouched	the box landed on
agent	0.127.0	0.127.0 ✅
golden	0.206.0	—
controller	(baked into the golden)	0.206.0 ✅

14:22:48 [INFO] Artifact manifest served for customer walk5 (agent=0.127.0 golden=0.206.0)
14:25:33 [INFO] Event from walk5: controller_started (info) — Controller elindult (0.206.0)
   docker ps → Up 10 minutes (healthy) gitea.dooplex.hu/admin/felhom-controller:0.206.0
   felhom-agent --version → felhom-agent 0.127.0

No hand upgrade was needed and none was applied. This is the first walk of the five where that sentence is true of the fixes under test: R-239's delivery gap — the headline finding of both previous walks — is closed for this run, and the box under test is the box a customer receives.

The whole day-0 cascade ran hands-free, bind → provision in 3 m 55 s:

14:21:38 appliance 24 BOUND to customer walk5 (mode=appliance)
14:22:00 appliance credentials DELIVERED once
14:22:08 [claim] claim code (gen 1) emailed to the registered address
14:22:38 host enrolled: walk5-4bada5
14:22:46 offsite credentials re-issued on re-enroll; break-glass root@pam vaulted (32 chars)
14:23:19 wg registered 10.77.0.5 · DR-recipe host-half · tenantsync provision ok (ns=walk5)
         pbsdr auto-provisioned (hands-free cascade)
14:25:33 controller 0.206.0 reporting

A2 — claimed

Claimed through the real POST /claim form (302 → /), then POST /login → 302. The claim code came from the documented --print-reset-code escape hatch (R-204 item 1) — a guest command line, used deliberately as FIXTURE CONSTRUCTION, permitted by §3 in Phase A, and counted as such. Phase B must not use it.

The dashboard password is a 24-character alphanumeric, 0600 in ~/.config/walk5/ on DooPlex only. The appliance's root credential is the hub-vaulted break-glass (root@pam, revealed through POST /hosts/walk5-4bada5/reveal-recovery-credential) — the password typed at the installer no longer authenticates, because day-0 step 4b rotates and vaults it. Stated because it cost a dead end.

A3 — drives, the app, and the three sentinels

Both raw disks enrolled through the customer endpoints (GET /api/disks/candidates → POST /api/storage/init), ext4:

/dev/sdb → /mnt/felhom-drives/adatok     (app data)
/dev/sdc → /mnt/felhom-drives/mentes     (backup target)

Backup target assigned to mentes — and, exactly as the last walk recorded, it reported restart_required: true and flipped only after the agent restart it asked for:

before: {"degraded":true, …"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after:  {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}

calibre-web deployed through the real API with HDD_PATH=/mnt/felhom-drives/adatok — state: running and health_probe.healthy: true.

THE THREE SENTINELS

#	file	bytes	sha256
A	WALK5-SENTINEL-A.txt	61	11eb7fb2d1891f62a685e3d9a9fda44e0a342d437d4aebaa15fcb4c0fd79d539
B	WALK5-őrszem-ékezetes-árvíztűrő.txt	66	6b504d1e83bf4c673334d0b9e701f3e5f07311d954fdc4463a6b0a55b261ae32
C	WALK5-SENTINEL-C-12MB.bin	12 582 912	0baaf402733e6a85f3d9ce8c4c3a57b05692f5f6733da6a23da76d2cd7635210

Sentinel B's filename as hex, and this is the reading that cannot lie:

57414c4b352d c591 72737a656d2d c3a9 6b657a657465732d c3a1 7276 c3ad 7a74 c5b1 72 c591 2e747874
             ő                 é                    á         í       ű     r  ő

No efbfbd anywhere. Written by a Python placer pushed in with pct push, which builds the name from a hex string decoded to bytes and passes a bytes path to os.open — no shell layer ever sees the name, and the readback is from os.listdir on a bytes path, not from a shell ls. This is the trap that bit three times; it did not bite here.

Placed under …/adatok/userdata/media/books — the level the last walk got wrong. The tree was listed first and the namespace root confirmed to be /mnt/felhom-drives/adatok directly.

A4 — the escrow ceremony

Preflight 6 of 6 green (pbs_storage_id · dr_tier · age_binary · hub_upload · staged_secret · sudo_grant), agent_supported: true, stale: false.

That stale: false is a deliberate check, not a formality. R-246/R-247/R-248 concern a staleness flag that would silently disable the very discriminator this walk exists to exercise. The spike's ruling was that a fresh box cannot reach the state (MarkEscrowStale has no production caller). Measured here on a fresh box: it has not.

Ceremony through the customer wizard's own endpoints (/api/escrow/start with password re-auth — it is a form POST, not JSON, which cost one rate-limited attempt):

phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24 · claimable: true
key_fingerprint: a6:86:f7:fb:74:e7:16:3c:…:31:2a:4c:f9

restic_pw_sealed: true is the field the whole exercise rests on.

R was claimed ONE-SHOT and streamed file→file into ~/.config/walk5/R_walk5.txt (0600, DooPlex only). It was never rendered, never an argument, never a log line. The appliance's copy was shred -u'd and its absence verified. Shape only: 82 characters, 10 hyphen-separated tokens, lowercase+digits.

§9.4 slip, recorded rather than buried. The Phase-A claim code (not R) reached this session's transcript, because the hatch prints it and my shape-only filter was written for an uppercase pattern and did not match a lowercase one. It was consumed by the claim seconds later and is now void. R itself was never printed and never left the file→file path.

A5 — the off-site backup, and the sentinels BY NAME

THE §4.5 GATE EARNED ITS PLACE AGAIN — and this time it caught MY fault, not the product's. The first run reported ok in 28 s with snapshots = 0 and an empty repository size. The log said why: [offbox] backup run started (0 app(s) toggled). My toggle POST sent enabled=1, and the handler accepts only on/true — so it recorded off, redirected with „A távoli mentés beállítása frissítve." and the run then correctly backed up nothing. A harness fault, not a product defect — but the shape is precisely R-240's, and had I read the green ok and moved on, the destruction would have proven nothing.

Re-toggled (enabled=on), selection verified in the rendered page (name="app" value="calibre-web"

• checked) before the re-run, which then logged backup run started (1 app(s) toggled).

ok, 1 m 20 s, 1 snapshot, 12.0 MB (12 611 563 B). The gate is not the green tick — restic snapshots + restic ls latest --long, run against the repository with its own credentials:

5b0f20f7  2026-08-07 12:57:41  walk5  [felhom-offbox, calibre-web]
    /mnt/felhom-drives/adatok/backups/primary/calibre-web
    /mnt/felhom-drives/adatok/userdata/media/books

-rw-r--r-- 1000 1000       61  …/userdata/media/books/WALK5-SENTINEL-A.txt
-rw-r--r-- 1000 1000 12582912  …/userdata/media/books/WALK5-SENTINEL-C-12MB.bin
-rw-r--r-- 1000 1000       66  …/userdata/media/books/WALK5-őrszem-ékezetes-árvíztűrő.txt
-rw-r--r-- 1000 1000   413696  …/userdata/media/books/metadata.db        ← calibre-web's real DB
restic stats latest: 22 files, 12.784 MiB

All three sentinels are in the snapshot, by name, at the right sizes — and the accented filename survived into restic intact.

A6 — §4.6's two pre-destruction checks, neither ever exercised on a clean box

Check 1 — the recovery offer is NOT shown. PASSES.

GET /          → 302 → /launcher
GET /launcher  → 200,  recovery mentions: 0
GET /          → 200 (followed), recovery mentions: 0
GET /recovery  → 302 → /backups/remote

No banner, no entry point, nothing to seek. And the reason is measurable rather than assumed: the hub's package covers the key in use, so v0.206.0's shape (c) compares equal and correctly stays silent —

box   /data/offbox/repo_password  sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
box   settings.hub_escrow_key_sha256    = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
hub   host_escrow.restic_pw_sha256      = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
      hub_escrow_key_checked_at         = 2026-08-07T12:51:39Z

This is the first live measurement of the persisted discriminator, and it is the negative half: the shape that must NOT fire, not firing, for the right reason.

Check 2 — the restore page lists the app with the future-backup toggle OFF. PASSES.

With calibre-web toggled off for future off-site backups, /backups/restore renders identically — „Calibre-Web Automated" still listed under „Visszaállítás a távoli tárolóból", 10 calibre occurrences in the HTML both before and after.

That is R-237's fix, live. The last walk's T3 measured 0 restore entries and a 302 on controller 0.203.0 — the customer's existing backup hidden by a setting about the future. The toggle was returned to on afterwards.

A7 — the pre-destruction truth (§4.7's seven records)

Box (settings.json, secrets stripped):

offbox: enabled true · escrow_state "escrowed" · last_status "ok" · last_duration 1m20s
        last_run/last_success 2026-08-07T12:58:57Z · snapshot_count 1
        repo_size_bytes 12 611 563 ("12.0 MB") · stats_known true · quota_gb 50
        host u629488-sub4.your-storagebox.de:23 · repo_path /home/felhom-repo
repo_state: (absent — not orphaned) · hub_escrow_identity_present: true · claimed: true
agent 0.127.0 · controller 0.206.0 (healthy) · guests 1/1

Hub:

host_escrow(walk5-4bada5):  blob 383 B · identity_blob 572 B (R-198 material present)
                            key_fingerprint a6:86:f7:fb:…:4c:f9
                            restic_pw_sha256 eabf427c7274…144f
                            stale_at NULL · created 2026-08-07T12:51:02Z
host_escrow_superseded:     0 rows for walk5
hosts:                      walk5-4bada5 ONLINE · agent 0.127.0 · PBS wrapper MATCHES VOUCHED
                            (104db0a4401f…) · WG 10.77.0.5 confirmed
DR Recipe: present · Key Escrow: present

Phase A gate: PASSED. All seven records taken; the three sentinels listed by name.

---