THE DATA: PASS. All three sentinels byte-identical out of snapshot 5b0f20f7, including the accented filename's bytes, read back as bytes from the live path. THE JOURNEY: PASS — the first time in five walks. Zero guest command lines were needed to progress; the previous walk needed three. The reset-code hatch was used once, in Phase A only. §5's observation, which stands on its own whatever the verdict: at 14:58:52Z the rebuilt box collected its re-staged credential, configured the transport, and REFUSED TO MINT a repository password over the sealed package the hub holds. At the equivalent moment the previous walk minted a fresh key and lost the journey silently at 03:18. Sampled every 20s from T0: no key at any moment. Honest about which shape fired: with the mint guard holding there is no local key, so the offer comes from shape (a), not shape (c). Shape (c) was measured in Phase A in its NEGATIVE half (equal hashes, correctly silent). This proves the mint guard positively and the discriminator negatively. RTO 71.7s login to open store, of which 12.44s was the unseal and ~22s my own CSRF harness retry. Two new customer-facing obstacles, neither needing a shell but neither signposted: the restore refuses on unattached drives, and refuses because the app is not installed on a page that says the restore reinstalls it.
37 KiB
THE FIFTH WALK (R-201) — journal
Venue: demo-hp VM 325 walk5-appliance, hub customer walk5. All four earlier venues
(c11, rewalk, part4, finalwalk) were torn down on 2026-08-06/07; nothing was reused except
the c11-scratch dir storage (a plain dir at /mnt/nvme-1tb, which is what the agent's
exactMount check requires) and the freed Storage Box sub-account number sub4 — the account
itself is newly provisioned, id 285351, not finalwalk's 285071 which the cascade destroyed.
Written as the walk proceeds, per §9.9. Clocks: DooPlex and demo-hp and the appliance = CEST; the guest and the hub = UTC.
§0–2 — orientation, on arrival
§1 baselines — re-read, all four match the runbook exactly
| Component | Runbook | Measured on arrival | |
|---|---|---|---|
felhom-controller |
3d3b4496f377 |
3d3b4496f377 |
✅ clean tree, == origin/main |
felhom-agent |
703db166e762 |
703db166e762 |
✅ clean tree, == origin/main |
felhom.eu |
10af63d0b04e |
10af63d0b04e |
✅ clean tree, == origin/main |
app-catalog-felhom.eu |
— | 459766cb1639 |
clean, untouched |
No drift. (The two previous walks both found one-commit drift on arrival and had to check it was benign; this one did not.)
§1 the vouched set — read from the hub, not from a document
GET /configuration on the hub (Basic auth, ClusterIP):
agent_version (selected) 0.127.0
golden_version (selected) 0.206.0
min_agent 0.127.0
min_controller_version 0.156.0
hub image (deployed) gitea.dooplex.hu/admin/felhom-hub:0.98.0
installer SCRIPT_VERSION 1.25.0 (served live from felhom.eu/scripts/)
All five match §1. The vouched golden 0.206.0 carries controller 0.206.0, so this is the
first walk whose fresh install lands on the fixes it exists to exercise — R-239's delivery gap,
which was the headline finding of the last two walks, is closed for this run.
§1 the build gate — GREEN before starting
$ python3 scripts/golden_currency_gate.py
newest released controller : 0.206.0
newest golden baked : 0.206.0
golden currency gate OK
rc=0
Stated precisely, because the gate says so itself: it checks the bake, not the vouch. The
vouch was therefore checked separately, from the hub, above — and it is 0.206.0. Both halves green.
§1 highest register ID
R-248, confirmed by grep over OPEN-ITEMS.md (R-241 … R-248). The runbook's figure is current.
ISO — verified by round trip, not by name
| published | https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso · 1 705 322 496 B · last-modified Fri 31 Jul 2026 18:37:18 GMT |
published .sha256 |
f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6 |
local on demo-hp |
/var/lib/vz/template/iso/felhom-installer-1.26.1-pve9.2-1.iso · same sha256 |
Byte-identical. (https://iso.felhom.eu/ itself 404s — there is no index object; only named
objects are served. Recorded so the next session does not read that 404 as an outage.)
§2.6 — the open items that could touch this walk
- R-247 / R-248 — the staleness flag. The runbook says no running code can set it; confirmed at
source: R-246's census found
MarkEscrowStalehas no production caller anywhere in the tree, and its one historical setter was removed in hub v0.95.0. A fresh box cannot reach the state. R-246 also records the demo-hp instance as CLEARED on 2026-08-07. So shape (c) is live and un-inhibited for this venue. - R-243 — the alarm gap for a box held in
awaiting_recovery_key. Directly in this walk's path: if §5 goes as designed, the rebuilt box enters exactly that state. The gap is that the hub raises nothing for it. Not a defect of the walk; watch for it and report it as still-open. - R-240 — „Sikeres" over a run that covered nothing. Only reachable if an off-site run is made with nothing selected; not part of this walk's route.
- R-244 — teardown residue in
app_log_issues. Teardown is §10, owed, not this session.
§2.7 — the two functions this walk exercises, read before running
Both fixes are present in the tree at 3d3b4496f377 and both are new since the last walk:
- the mint guard —
WriteOffboxSecrets(controller/internal/backup/offbox.go:438) now returnsErrOffboxSealedPackageHeldinstead of generating, whensealedPackageHeld()is true and no password exists. It is a conjunction, so a first-time box mints exactly as before. The transport is still written. New declared stateoffsite.state=awaiting_recovery_key(OffboxAwaitingRecoveryKey,:502— derived, not stored, and gated ont.Enabled). - shape (c) —
OffsiteRecoveryOffer(:1580) now offers whenhubHash != "" && hubHash != localHash.""falls back to (a)/(b) deliberately.
Neither has ever run on a real rebuild. That is §5, and it is deliverable 2 whatever the verdict.
Venue construction
Hub customer walk5
POST /configs/new — id walk5, name "Walk 5 R-201", domain walk5.felhom.eu, email = the
operator's registered address, off-site ON (shared, 50 GB), DR tier ON, pbsdr_storage_id
felhom-pbs. → 303 /customers/walk5?flash=created.
Provisioned descriptor, read from the hub DB rather than from the page:
{"enabled":true,"type":"shared","host":"u629488-sub4.your-storagebox.de","user":"u629488-sub4",
"port":23,"repo_path":"/home/felhom-repo","quota_gb":50,
"host_fingerprint":"SHA256:oDHZqKXnoMtgvPBjjC57pcuFez28roaEuFcfwyg8O5c"}
one_time_secrets 1 · customer_configs 1 · hosts 0. Sub-account 285351, user u629488-sub4.
⚠ THE FIRST CREATE FAILED, AND IT IS A FINDING — not a fluke of my own making.
The first
POST /configs/newreturned no response to the client and the hub logged[ERROR] offsite provision for walk5: offsite: host-key scan u629488-sub4.your-storagebox.de: dial: dial tcp [2a01:4f8:bacc:2:200::d30]:23: connect: network is unreachableat 13:46:46.The scan is fail-closed by design (
offsite.go:111-121: "don't serve a descriptor the controller can't verify") and its retry ladder is ~60 s (defaultScanBackoff, 2+4+8+16+30), sized by its own comment to "the observed DNS propagation lag" of a fresh sub-account. Here the lag was longer than the budget: the first five attempts failedno such host, and by the time DNS answered it answered AAAA-first, into a pod with no IPv6 route.The retry (a second identical POST) succeeded — idempotent on the label (
[offsite] shared already provisioned for walk5 (subaccount 285351)), scan green on its own final rung at ~13:47:28, config created. Total settle time ≈ 100 s against a 60 s budget.Measured afterwards, so the cause is not guessed: the hub pod reaches
u629488-sub4.your-storagebox.de:23now, by name, and getsSSH-2.0-OpenSSH_9.6p1— as does the node. So the endpoint is fine and the window was transient. What is not fine is the operator experience: a customer create can fail with a fail-closed error whose only remedy is "press it again", and nothing says so. Filed below.
VM
| Host | demo-hp (HP t740), Tier 0 — the designated drill host |
| VM | 325 walk5-appliance — q35/OVMF (pre-enrolled-keys=0), 4 cores, 8 GB, cpu=host |
| Disks | scsi0 200 G · scsi1 50 G · scsi2 50 G, qcow2 on c11-scratch (a dir at the mount root /mnt/nvme-1tb, which is what the agent's exactMount check requires) |
| Appliance | walk5.felhom.eu @ 192.168.0.100/24, gw/DNS 192.168.0.1, MAC bc:24:11:37:7c:f3 |
| Guest | LXC 9201 @ 192.168.0.149 (LAN) + 169.254.253.2 (R-50 island) |
| Hub customer | walk5 "Walk 5 R-201", DR tier ON, off-site ON (shared, 50 GB) |
| Host id | walk5-4bada5 · appliance uuid 596d8104-2af9-4525-baa7-e479314cbb44 |
| Off-site | Storage Box sub-account 285351, user u629488-sub4 · WG peer 10.77.0.5 |
| Untouched | drill-r50 (VM 300, the protected fixture), guest 9201 on both demo boxes, DooPlex, ep0, the tester's box |
Reachability, stated because it shapes every later step: DooPlex has no L2 path to the
192.168.0.0/24 LAN — ip neigh is INCOMPLETE even for demo-hp, which it reaches over the
tailnet. So the appliance is one SSH hop behind demo-hp, and every request to the guest is made
from the appliance, which is exactly what a browser on the customer's LAN would do.
Phase A — the fixture (§4)
A1 — installed from the published ISO, through the Terminal UI
Driven with qm sendkey + qm monitor screendump → PNG. All three known traps reproduced:
- GRUB defaults to the graphical entry — and it bit, first attempt. The countdown was 14 s
with my screendump→convert→read round trip at ~18 s, so the menu auto-booted the default and
the GUI installer came up. Recorded rather than quietly retried: the VM was stopped and
restarted, and the keys sent inside one remote command (
sleep 6; sendkey down; screendump; sendkey ret) — the screendump taken between the arrow and the Enter is the positive control, and it shows the highlight on „Felhom telepítés (szöveges mód)" at the moment of the press. - The keymap defaults to Hungarian while
sendkeyemits US scancodes. Changed to U.S. English before any typing. Positive control: the administrator email typed through the identical path renderedwalk5@felhom.eu—@isshift-2on US andAltGr+Von HU, and it is the only available evidence for the 24 masked password characters. - Auto-reboot was checked. Unchecked, and confirmed
[ ]with the focus moved away; then--bootset in its ownqm setwith the ISO detached, both verified fromqm configbefore the first boot:boot: order=scsi0,ide2lines = 0.
Summary screen, verbatim: ext4 · /dev/sda · Europe/Budapest · U.S. English ·
walk5@felhom.eu · nic0 · walk5.felhom.eu · 192.168.0.100/24 · 192.168.0.1 · 192.168.0.1.
One reading corrected by a second instrument, as in the last walk: 192.168.0.100 was the
installer's prefill and answered on the LAN. The MAC was bc:24:11:37:7c:f3 — VM 325's own DHCP
lease. Not a collision; a ping alone could not have told the difference.
Day-0 fired on first boot, unaided. Console showed the Hungarian pairing banner:
Felhom — a doboz készen áll, és a párosításra vár.·Párosító kód: E28-WJH
The hub's appliance_registrations row 24 carried the same code (E28WJH), the same MAC, the
same SMBIOS uuid and three SSH host keys, first_seen 12:18:59Z — 32 seconds after boot.
Bound through the real endpoint (POST /appliances/24/bind, 303).
A1b — THE HEADLINE: a fresh install DOES get the fixes
| vouched | the box landed on | |
|---|---|---|
| agent | 0.127.0 | 0.127.0 ✅ |
| golden | 0.206.0 | — |
| controller | (baked into the golden) | 0.206.0 ✅ |
14:22:48 [INFO] Artifact manifest served for customer walk5 (agent=0.127.0 golden=0.206.0)
14:25:33 [INFO] Event from walk5: controller_started (info) — Controller elindult (0.206.0)
docker ps → Up 10 minutes (healthy) gitea.dooplex.hu/admin/felhom-controller:0.206.0
felhom-agent --version → felhom-agent 0.127.0
No hand upgrade was needed and none was applied. This is the first walk of the five where that sentence is true of the fixes under test: R-239's delivery gap — the headline finding of both previous walks — is closed for this run, and the box under test is the box a customer receives.
The whole day-0 cascade ran hands-free, bind → provision in 3 m 55 s:
14:21:38 appliance 24 BOUND to customer walk5 (mode=appliance)
14:22:00 appliance credentials DELIVERED once
14:22:08 [claim] claim code (gen 1) emailed to the registered address
14:22:38 host enrolled: walk5-4bada5
14:22:46 offsite credentials re-issued on re-enroll; break-glass root@pam vaulted (32 chars)
14:23:19 wg registered 10.77.0.5 · DR-recipe host-half · tenantsync provision ok (ns=walk5)
pbsdr auto-provisioned (hands-free cascade)
14:25:33 controller 0.206.0 reporting
A2 — claimed
Claimed through the real POST /claim form (302 → /), then POST /login → 302. The claim code
came from the documented --print-reset-code escape hatch (R-204 item 1) — a guest command line,
used deliberately as FIXTURE CONSTRUCTION, permitted by §3 in Phase A, and counted as such. Phase B
must not use it.
The dashboard password is a 24-character alphanumeric, 0600 in ~/.config/walk5/ on DooPlex
only. The appliance's root credential is the hub-vaulted break-glass (root@pam, revealed through
POST /hosts/walk5-4bada5/reveal-recovery-credential) — the password typed at the installer no
longer authenticates, because day-0 step 4b rotates and vaults it. Stated because it cost a dead
end.
A3 — drives, the app, and the three sentinels
Both raw disks enrolled through the customer endpoints (GET /api/disks/candidates →
POST /api/storage/init), ext4:
/dev/sdb → /mnt/felhom-drives/adatok (app data)
/dev/sdc → /mnt/felhom-drives/mentes (backup target)
Backup target assigned to mentes — and, exactly as the last walk recorded, it reported
restart_required: true and flipped only after the agent restart it asked for:
before: {"degraded":true, …"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after: {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}
calibre-web deployed through the real API with HDD_PATH=/mnt/felhom-drives/adatok —
state: running and health_probe.healthy: true.
THE THREE SENTINELS
| # | file | bytes | sha256 |
|---|---|---|---|
| A | WALK5-SENTINEL-A.txt |
61 | 11eb7fb2d1891f62a685e3d9a9fda44e0a342d437d4aebaa15fcb4c0fd79d539 |
| B | WALK5-őrszem-ékezetes-árvíztűrő.txt |
66 | 6b504d1e83bf4c673334d0b9e701f3e5f07311d954fdc4463a6b0a55b261ae32 |
| C | WALK5-SENTINEL-C-12MB.bin |
12 582 912 | 0baaf402733e6a85f3d9ce8c4c3a57b05692f5f6733da6a23da76d2cd7635210 |
Sentinel B's filename as hex, and this is the reading that cannot lie:
57414c4b352d c591 72737a656d2d c3a9 6b657a657465732d c3a1 7276 c3ad 7a74 c5b1 72 c591 2e747874
ő é á í ű r ő
No efbfbd anywhere. Written by a Python placer pushed in with pct push, which builds the name
from a hex string decoded to bytes and passes a bytes path to os.open — no shell layer ever
sees the name, and the readback is from os.listdir on a bytes path, not from a shell ls. This is
the trap that bit three times; it did not bite here.
Placed under …/adatok/userdata/media/books — the level the last walk got wrong. The tree was
listed first and the namespace root confirmed to be /mnt/felhom-drives/adatok directly.
A4 — the escrow ceremony
Preflight 6 of 6 green (pbs_storage_id · dr_tier · age_binary · hub_upload ·
staged_secret · sudo_grant), agent_supported: true, stale: false.
That
stale: falseis a deliberate check, not a formality. R-246/R-247/R-248 concern a staleness flag that would silently disable the very discriminator this walk exists to exercise. The spike's ruling was that a fresh box cannot reach the state (MarkEscrowStalehas no production caller). Measured here on a fresh box: it has not.
Ceremony through the customer wizard's own endpoints (/api/escrow/start with password re-auth —
it is a form POST, not JSON, which cost one rate-limited attempt):
phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24 · claimable: true
key_fingerprint: a6:86:f7:fb:74:e7:16:3c:…:31:2a:4c:f9
restic_pw_sealed: true is the field the whole exercise rests on.
R was claimed ONE-SHOT and streamed file→file into ~/.config/walk5/R_walk5.txt (0600,
DooPlex only). It was never rendered, never an argument, never a log line. The appliance's copy
was shred -u'd and its absence verified. Shape only: 82 characters, 10 hyphen-separated
tokens, lowercase+digits.
§9.4 slip, recorded rather than buried. The Phase-A claim code (not R) reached this session's transcript, because the hatch prints it and my shape-only filter was written for an uppercase pattern and did not match a lowercase one. It was consumed by the claim seconds later and is now void. R itself was never printed and never left the file→file path.
A5 — the off-site backup, and the sentinels BY NAME
THE §4.5 GATE EARNED ITS PLACE AGAIN — and this time it caught MY fault, not the product's. The first run reported
okin 28 s withsnapshots = 0and an empty repository size. The log said why:[offbox] backup run started (0 app(s) toggled). My toggle POST sentenabled=1, and the handler accepts onlyon/true— so it recorded off, redirected with „A távoli mentés beállítása frissítve." and the run then correctly backed up nothing. A harness fault, not a product defect — but the shape is precisely R-240's, and had I read the greenokand moved on, the destruction would have proven nothing.
Re-toggled (enabled=on), selection verified in the rendered page (name="app" value="calibre-web"
checked) before the re-run, which then loggedbackup run started (1 app(s) toggled).
ok, 1 m 20 s, 1 snapshot, 12.0 MB (12 611 563 B). The gate is not the green tick —
restic snapshots + restic ls latest --long, run against the repository with its own credentials:
5b0f20f7 2026-08-07 12:57:41 walk5 [felhom-offbox, calibre-web]
/mnt/felhom-drives/adatok/backups/primary/calibre-web
/mnt/felhom-drives/adatok/userdata/media/books
-rw-r--r-- 1000 1000 61 …/userdata/media/books/WALK5-SENTINEL-A.txt
-rw-r--r-- 1000 1000 12582912 …/userdata/media/books/WALK5-SENTINEL-C-12MB.bin
-rw-r--r-- 1000 1000 66 …/userdata/media/books/WALK5-őrszem-ékezetes-árvíztűrő.txt
-rw-r--r-- 1000 1000 413696 …/userdata/media/books/metadata.db ← calibre-web's real DB
restic stats latest: 22 files, 12.784 MiB
All three sentinels are in the snapshot, by name, at the right sizes — and the accented filename survived into restic intact.
A6 — §4.6's two pre-destruction checks, neither ever exercised on a clean box
Check 1 — the recovery offer is NOT shown. PASSES.
GET / → 302 → /launcher
GET /launcher → 200, recovery mentions: 0
GET / → 200 (followed), recovery mentions: 0
GET /recovery → 302 → /backups/remote
No banner, no entry point, nothing to seek. And the reason is measurable rather than assumed: the hub's package covers the key in use, so v0.206.0's shape (c) compares equal and correctly stays silent —
box /data/offbox/repo_password sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
box settings.hub_escrow_key_sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
hub host_escrow.restic_pw_sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
hub_escrow_key_checked_at = 2026-08-07T12:51:39Z
This is the first live measurement of the persisted discriminator, and it is the negative half: the shape that must NOT fire, not firing, for the right reason.
Check 2 — the restore page lists the app with the future-backup toggle OFF. PASSES.
With calibre-web toggled off for future off-site backups, /backups/restore renders
identically — „Calibre-Web Automated" still listed under „Visszaállítás a távoli tárolóból",
10 calibre occurrences in the HTML both before and after.
That is R-237's fix, live. The last walk's T3 measured 0 restore entries and a 302 on controller 0.203.0 — the customer's existing backup hidden by a setting about the future. The toggle was returned to on afterwards.
A7 — the pre-destruction truth (§4.7's seven records)
Box (settings.json, secrets stripped):
offbox: enabled true · escrow_state "escrowed" · last_status "ok" · last_duration 1m20s
last_run/last_success 2026-08-07T12:58:57Z · snapshot_count 1
repo_size_bytes 12 611 563 ("12.0 MB") · stats_known true · quota_gb 50
host u629488-sub4.your-storagebox.de:23 · repo_path /home/felhom-repo
repo_state: (absent — not orphaned) · hub_escrow_identity_present: true · claimed: true
agent 0.127.0 · controller 0.206.0 (healthy) · guests 1/1
Hub:
host_escrow(walk5-4bada5): blob 383 B · identity_blob 572 B (R-198 material present)
key_fingerprint a6:86:f7:fb:…:4c:f9
restic_pw_sha256 eabf427c7274…144f
stale_at NULL · created 2026-08-07T12:51:02Z
host_escrow_superseded: 0 rows for walk5
hosts: walk5-4bada5 ONLINE · agent 0.127.0 · PBS wrapper MATCHES VOUCHED
(104db0a4401f…) · WG 10.77.0.5 confirmed
DR Recipe: present · Key Escrow: present
Phase A gate: PASSED. All seven records taken; the three sentinels listed by name.
Phase B — the journey (§7)
The rule: no command line inside the guest, at any point. After the destruction the only things
that reach the guest are HTTP requests a browser could have made, issued from the appliance (the
only machine with an L2 path to it). Instrumentation reads are counted separately and named: the
w5watch.log sampler, pct exec … docker logs, the restic listing and the hub-DB queries are guest
or operator command lines. None of them changed state and none was needed to progress the journey —
that distinction is the whole point, and conflating the two is how a walk claims a property it does
not have.
B1 — the destruction (14:19:26Z)
Guarded four ways before anything ran, because demo-hp also carries a guest 9201:
hostname == walk5 ✓
pct config 9201 → hostname: walk5 ✓
mountpoint -q /mnt/adatok ✓
mountpoint -q /mnt/mentes ✓
14:19:18Z pct stop 9201
14:19:26Z pct destroy 9201 --purge
→ "Logical volume vm-9201-disk-0 successfully removed"
→ "Logical volume vm-9201-disk-1 successfully removed"
→ pct list EMPTY · lvs | grep 9201 → NONE
14:19:27Z /mnt/adatok and /mnt/mentes wiped 13M/1.2G → 4.0K each,
MOUNTS LEFT IN PLACE — deliberately: the surviving raw mount IS the R-220 condition
B2 — the rebuild (14:24:09Z, 3 m 31 s)
felhom-host-install.sh fetched live from felhom.eu/scripts/ — v1.25.0,
sha256 cb4d02059f126f8bfdddfe98607e87a3cb45dd2592c6b40b43c9659e2b687534. The retrieval passphrase
was passed as --passphrase-file (a 0600 file), never on the command line.
[OK] Day-0 provision SUCCESS — vmid=9201 host_id=walk5-4bada5 customer=walk5
golden=local:backup/vzdump-lxc-9100-2026_08_07-16_21_39.tar.zst
[OK] controller: Up 19 seconds (healthy)
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.206.0
[OK] authz signers: 2 (operator-signed self-update armed)
[INFO] root@pam was rotated + vaulted at step 4b
What the rebuild landed on:
| before | after | |
|---|---|---|
| agent | 0.127.0 | 0.127.0 — no downgrade, no hand upgrade |
| controller | 0.206.0 | 0.206.0 — same, and it is the vouched build |
No hand upgrade, either time. R-216 part 4's reinstall-downgrade did not recur.
Guest re-provisioned at 192.168.0.83 (it was .149 before — a fresh DHCP lease, not a fault).
B3 — §5's observation window: THE MOMENT THAT MATTERS
T0 — 14:24:31Z, 22 seconds after provision, nobody has logged in:
offbox directory ABSENT ("No such file or directory")
repository key NONE
settings.offbox null
settings.hub_escrow_identity_present TRUE ← the box already knows a package is held for it
settings.hub_escrow_key_sha256 eabf427c7274…144f ← and already knows WHICH key it seals
settings.claimed null (pristine, unclaimed)
The scheduler is alive — the positive control, because an absent line is not evidence. In the
first five minutes the controller logged agent-channel-health ×5 (1 m cadence), stack-scan ×2,
system-health, backup-cache and offsite-credential-retry. So "no key was written" is a
measurement, not a silence.
The window, minute by minute — nobody logged in at any point during it:
| time (UTC) | event | source |
|---|---|---|
| 14:24:09 | Day-0 provision SUCCESS; rebuilt guest, controller 0.206.0 | installer |
| 14:24:31 | T0 — no offbox dir, no key, offbox: null, but hub_escrow_identity_present already true and the sealed key's hash already learned |
box, measured |
| 14:24:53 | hub: offsite_delivery_stuck (warning) + operator notification_log row — REFUSED, "regressed-apply shape … a guest rebuild does, R-193" |
hub, measured |
| 14:23:54 | first post-rebuild report carries no offsite object — it predates the ACK that set the flag |
hub reports, measured |
| 14:38:49 | first declaration: offsite={"enabled":false,"state":"needs_credential"} |
hub reports, measured |
| 14:38:53 | the checker hands off: "the box DECLARES offsite.state=needs_credential; internal/offsiteheal owns this remediation" | hub, measured |
| 14:53:49 | second declaration, same shape | hub reports, measured |
| 14:56:34 | offsiteheal: re-staged the stored one-time offsite secret … (declared needs_credential across 2 reports) … no provider credential was minted |
hub, measured |
| 14:58:52 | the box collects it and applies the tier — AND REFUSES TO MINT | box log, measured |
THE LINE THIS WALK EXISTED TO SEE
14:58:52 [WARN] [offbox] NOT minting a repository password: the hub holds a sealed recovery package
for this box, and a fresh key would orphan the history that package protects (R-241).
The transport is configured; the tier stays down until the customer's recovery code places
the escrowed key.
14:58:52 [INFO] [offbox] apply-offsite: transport configured for u629488-sub4@…:/home/felhom-repo,
tier HELD awaiting the escrowed key (the hub holds a sealed package; no key was minted — R-241)
14:58:52 [INFO] [offsite-apply] credential retry: the staged credential was collected and the tier applied
At the equivalent moment the previous walk minted 9b4a9a9d… over a package sealing 30ef574f…,
and lost the journey silently at 03:18 in the morning. This build did not.
§5's four items, answered
1. Does the box declare a need, and when is it staged and collected? Yes — declared
needs_credential at 14:38:49Z and 14:53:49Z, staged by the hub unaided at 14:56:34Z,
collected and applied at 14:58:52Z. Zero human actions; the whole chain ran on a box that had
not even been claimed.
2. Is any repository key written? NO. Sampled every ~20 s from 14:24:52Z onward:
repo_password_sha=NONE at every single sample. At 15:00:24Z the directory holds exactly:
-rw------- applied_marker 64
-rw-r--r-- known_hosts 290
-rw------- ssh_key 400
repo_password: ABSENT
The transport is present and the key is not — which is the guard's designed shape, not a partial
failure: ApplyOffsiteTarget deliberately keeps the transport so the recovery screen can bring the
tier up the instant the key arrives (R-219).
3. What state does it report instead? offbox.enabled = true, escrow_state = pending,
repo_state absent — and OffboxAwaitingRecoveryKey() is therefore true (target exists, target
enabled, no password), so the box declares the new awaiting_recovery_key holding state rather
than coming up under a fresh key.
4. The two fingerprints, before anyone logged in (15:00:24Z):
the escrowed key (hub's package, ACK-cached) eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
whatever the box holds NONE — there is no repository key on this box
HONEST NOTE ON WHICH SHAPE FIRES, because it changes what has been proven. With the mint guard holding, the box has no local key at all, so
OffsiteRecoveryOfferreturns true on shape (a) — the pristine-rebuild shape — not on shape (c). Shape (c) was measured in Phase A, in its negative half: equal hashes, correctly silent. So this walk proves the mint guard positively and the discriminator negatively; a positive shape-(c) firing needs a box holding a different key, which v0.206.0 now prevents from arising by itself. That is the intended end state (the fix removes the state its own second half was written to describe), but it should not be reported as "shape (c) proven".
B4 — the journey, step by step (§7)
| # | step | result |
|---|---|---|
| 1 | Destroy — 14:19:26Z | guest 9201 purged (both LVs), both drives wiped 13M/1.2G → 4.0K. Host identity walk5-4bada5 survived |
| 2 | Reinstall | felhom-host-install.sh v1.25.0 fetched live; Day-0 provision SUCCESS 14:24:09Z, 3 m 31 s. controller 0.206.0, agent 0.127.0 — no hand upgrade, no downgrade |
| 3 | §5's window | no key minted; transport configured; awaiting_recovery_key — see B3 |
| 4 | Claim back | new code requested through the customer-facing „Új kód kérése" path (15:01:27Z), hub rotated to generation 2 and emailed it; the operator relayed it; worked FIRST TRY — 302 in 0.151 s, accents intact (á=c3a1, í=c3ad, sha256 identical source→appliance) |
| 5 | Log in | / → 302 /launcher → 302 /recovery — the recovery screen appeared WITHOUT BEING SOUGHT |
| 6 | Read the screen | all three questions answered — quoted below |
| 7 | Enter R | HTTP 200 in 12.44 s — a real unseal. [web] recovery: the offsite repository key was recovered and placed (outcome=installed) |
| 8 | The listing | rendered — calibre-web · 2026-08-07 14:57 · 12.8 MB |
| 9 | Restore | all three sentinels byte-identical — after two unsignposted obstacles, both cleared from the dashboard |
The recovery screen, as a customer reads it (§7.6)
„Adatok visszaszerzése"
„Ezt a gépet újratelepítették. A korábbi, házon kívüli mentéseid megvannak — a Felhom központi rendszere őriz hozzájuk egy lezárt csomagot, amelyet 2026-08-07T12:51:02Z zártunk le. A csomagot csak a te helyreállítási kódoddal lehet kinyitni.
A helyreállítási kódot senki nem tudja pótolni — sem a Felhom, sem az ügyfélszolgálat, sem az üzemeltető. Ez szándékos: így a mentéseidet rajtad kívül senki nem tudja megnyitni. Ha a kód elveszett, a korábbi mentések nem nyithatók meg többé.
Ha megadod a kódot, feloldjuk a mentéseid zárolását és megmutatjuk, mi van bennük — melyik alkalmazás, mikorról, mekkora. Ebben a lépésben semmit nem állítunk vissza és semmi nem változik."
All three questions answered. The sealed-at timestamp 2026-08-07T12:51:02Z matches the hub's
host_escrow.created_at exactly. The field label reads „Helyreállítási kód (tíz szó)" and R is
indeed ten tokens.
(One copy wart, recorded not filed: the sealed-at is rendered as a raw ISO-8601 string on a
customer-facing Hungarian screen, where every other date on the box is formatted 2026-08-07 14:57.)
The listing, against Phase A's record (§7.8)
| Phase A recorded | the screen showed | |
|---|---|---|
| app | calibre-web |
calibre-web ✅ |
| when | snapshot 5b0f20f7 at 12:57:41Z |
2026-08-07 14:57 ✅ (12:57Z rendered in CEST) |
| size | restic stats 12.784 MiB |
12.8 MB ✅ |
(A second row felhom-offbox · 12.8 MB also appears — the snapshot carries both tags and the screen
renders one row per tag. A customer sees an "app" they have never installed, and the same 12.8 MB
counted twice. Cosmetic; filed as R-251.)
THE RTO (§7.7)
login 15:04:25.417Z
unlock returns 200 15:05:37.156Z
------------------------------------------
wall clock, login → open store 71.7 s
Of which 12.44 s was the unseal itself. ~22 s of the remainder was my own harness retry — I
scraped the CSRF token from a <meta> tag the recovery page does not carry, got a 403, and re-read it
from the form (the controller logged CSRF rejected … (token mismatch) at 15:05:02). A customer
clicking the button would have seen ≈50 s. Both numbers are stated because the 71.7 s is what was
actually measured.
The restore, and the two obstacles (§7.9)
POST /backup/offbox/restore (mode=full) → refused: „nincs elérhető adatmeghajtó a
visszaállításhoz". Then, after that was cleared, POST /backup/offbox/reconstitute → refused:
„a(z) calibre-web nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat".
Both were cleared from the dashboard, with no guest shell:
- Re-attach the drives —
GET /api/disks/candidatesoffered both surviving disks (mountable: true,data_bearing: true), and Tárhely → Meghajtók → „Meglévő meghajtó csatolása" (POST /api/storage/attach) registered them:/mnt/felhom-drives/adatokand…/mentes. - Redeploy the app —
POST /api/stacks/calibre-web/deploy, running in 1 m 30 s.
Then prepare (12.8 MB) → confirm → reconstitute:
15:10:18 [offbox] restored calibre-web (5b0f20f7, full=true) → …/backups/offsite-restore/calibre-web
15:14:30 [offbox] reconstituted calibre-web from snapshot 5b0f20f7: 6 file(s) placed,
0 DB dump(s) replayed, safety dump=., skewed=false
THE THREE SENTINELS, read back as BYTES from the live path
MATCH [A] 11eb7fb2d1891f62a685e3d9a9fda44e0a342d437d4aebaa15fcb4c0fd79d539 61
name_hex 57414c4b352d53454e54494e454c2d412e747874
MATCH [B] 6b504d1e83bf4c673334d0b9e701f3e5f07311d954fdc4463a6b0a55b261ae32 66
name_hex 57414c4b352d c591 72737a656d2d c3a9 6b657a657465732d c3a1 7276 c3ad 7a74 c5b1 72 c591 2e747874
MATCH [C] 0baaf402733e6a85f3d9ce8c4c3a57b05692f5f6733da6a23da76d2cd7635210 12582912
name_hex 57414c4b352d53454e54494e454c2d432d31324d422e62696e
all three present: True
Identical to Phase A in every byte, including the accented filename. Read with os.listdir on a
bytes path, so no decode/encode round trip could launder a U+FFFD into looking correct.
The end state — the loop closed
escrow_state escrowed (auto-confirmed: hub hash == local hash)
repo_password sha256 eabf427c7274…144f == the escrowed key, == Phase A's key
GET /launcher 200 (the recovery screen no longer interrupts)
GET /recovery 302 (correctly retired — shape (c) compares equal again)
THE VERDICT — both halves, separately
THE DATA: PASS
All three sentinels came back byte-identical, out of snapshot 5b0f20f7, under the key recovered
from the sealed package with R — including the accented filename's bytes. Disk → restic → SFTP →
Storage Box → a destroyed machine → a rebuilt machine → disk, intact.
THE JOURNEY: PASS — the first time in five walks
No step needed a command line inside the guest. Every step that progressed the journey — request a code, claim, log in, read the screen, enter R, read the listing, re-attach the drives, redeploy the app, prepare, confirm, restore — was an HTTP request a browser makes, issued from the appliance because DooPlex has no L2 path to that LAN.
Guest command lines used, named per §3 so the claim is not read wider than it is: the w5watch.log
sampler, docker logs, the settings reads, the restic listing and the final sentinel verification.
Every one is instrumentation — none changed state, none was needed to progress, and removing them
all would have changed nothing except my ability to describe what happened. The previous walk needed
three guest command lines to progress; this one needed zero.
The reset-code hatch was used ONCE, in Phase A, where §3 permits it. It was not used in Phase B.
But the journey is not smooth, and two obstacles are new
Neither is a dead end by §3's definition (neither needed a guest shell), and both were cleared from the dashboard — but a customer would not know to do either, because the product does not say so and, in the second case, says the opposite:
- The restore refuses with „nincs elérhető adatmeghajtó" and nothing on the recovery path mentions that the drives must be re-attached first. → R-252
- „a(z) calibre-web nincs telepítve — előbb állítsd helyre az alkalmazást" — on a page that says, three lines above, „Nincs telepítve — a visszaállítás előbb újratelepíti." The product contradicts itself in the customer's own language, at the last step of a recovery. → R-253
So: the machinery works end to end and the data is provably safe. The unaided journey now succeeds, and it succeeds through two obstacles the customer must guess their way past.