Files
felhom.eu/documentation/tests/walk5-r201-2026-08-07/journal.md
T
admin 0691bc59a5 walk5 (R-201): Phase B + the verdict — BOTH HALVES PASS
THE DATA: PASS. All three sentinels byte-identical out of snapshot 5b0f20f7,
including the accented filename's bytes, read back as bytes from the live path.

THE JOURNEY: PASS — the first time in five walks. Zero guest command lines were
needed to progress; the previous walk needed three. The reset-code hatch was used
once, in Phase A only.

§5's observation, which stands on its own whatever the verdict: at 14:58:52Z the
rebuilt box collected its re-staged credential, configured the transport, and
REFUSED TO MINT a repository password over the sealed package the hub holds. At
the equivalent moment the previous walk minted a fresh key and lost the journey
silently at 03:18. Sampled every 20s from T0: no key at any moment.

Honest about which shape fired: with the mint guard holding there is no local key,
so the offer comes from shape (a), not shape (c). Shape (c) was measured in Phase A
in its NEGATIVE half (equal hashes, correctly silent). This proves the mint guard
positively and the discriminator negatively.

RTO 71.7s login to open store, of which 12.44s was the unseal and ~22s my own
CSRF harness retry.

Two new customer-facing obstacles, neither needing a shell but neither signposted:
the restore refuses on unattached drives, and refuses because the app is not
installed on a page that says the restore reinstalls it.
2026-08-07 17:18:37 +02:00

37 KiB
Raw Blame History

THE FIFTH WALK (R-201) — journal

Venue: demo-hp VM 325 walk5-appliance, hub customer walk5. All four earlier venues (c11, rewalk, part4, finalwalk) were torn down on 2026-08-06/07; nothing was reused except the c11-scratch dir storage (a plain dir at /mnt/nvme-1tb, which is what the agent's exactMount check requires) and the freed Storage Box sub-account number sub4 — the account itself is newly provisioned, id 285351, not finalwalk's 285071 which the cascade destroyed.

Written as the walk proceeds, per §9.9. Clocks: DooPlex and demo-hp and the appliance = CEST; the guest and the hub = UTC.


§02 — orientation, on arrival

§1 baselines — re-read, all four match the runbook exactly

Component Runbook Measured on arrival
felhom-controller 3d3b4496f377 3d3b4496f377 clean tree, == origin/main
felhom-agent 703db166e762 703db166e762 clean tree, == origin/main
felhom.eu 10af63d0b04e 10af63d0b04e clean tree, == origin/main
app-catalog-felhom.eu 459766cb1639 clean, untouched

No drift. (The two previous walks both found one-commit drift on arrival and had to check it was benign; this one did not.)

§1 the vouched set — read from the hub, not from a document

GET /configuration on the hub (Basic auth, ClusterIP):

agent_version   (selected)  0.127.0
golden_version  (selected)  0.206.0
min_agent                   0.127.0
min_controller_version      0.156.0
hub image (deployed)        gitea.dooplex.hu/admin/felhom-hub:0.98.0
installer SCRIPT_VERSION    1.25.0   (served live from felhom.eu/scripts/)

All five match §1. The vouched golden 0.206.0 carries controller 0.206.0, so this is the first walk whose fresh install lands on the fixes it exists to exercise — R-239's delivery gap, which was the headline finding of the last two walks, is closed for this run.

§1 the build gate — GREEN before starting

$ python3 scripts/golden_currency_gate.py
  newest released controller : 0.206.0
  newest golden baked        : 0.206.0
golden currency gate OK
rc=0

Stated precisely, because the gate says so itself: it checks the bake, not the vouch. The vouch was therefore checked separately, from the hub, above — and it is 0.206.0. Both halves green.

§1 highest register ID

R-248, confirmed by grep over OPEN-ITEMS.md (R-241 … R-248). The runbook's figure is current.

ISO — verified by round trip, not by name

published https://iso.felhom.eu/felhom-installer-1.26.1-pve9.2-1.iso · 1 705 322 496 B · last-modified Fri 31 Jul 2026 18:37:18 GMT
published .sha256 f3cc86d5f0ec68bba4155c994b4fa84e208d50209bb6e815636c99e5441059a6
local on demo-hp /var/lib/vz/template/iso/felhom-installer-1.26.1-pve9.2-1.iso · same sha256

Byte-identical. (https://iso.felhom.eu/ itself 404s — there is no index object; only named objects are served. Recorded so the next session does not read that 404 as an outage.)

§2.6 — the open items that could touch this walk

  • R-247 / R-248 — the staleness flag. The runbook says no running code can set it; confirmed at source: R-246's census found MarkEscrowStale has no production caller anywhere in the tree, and its one historical setter was removed in hub v0.95.0. A fresh box cannot reach the state. R-246 also records the demo-hp instance as CLEARED on 2026-08-07. So shape (c) is live and un-inhibited for this venue.
  • R-243 — the alarm gap for a box held in awaiting_recovery_key. Directly in this walk's path: if §5 goes as designed, the rebuilt box enters exactly that state. The gap is that the hub raises nothing for it. Not a defect of the walk; watch for it and report it as still-open.
  • R-240 — „Sikeres" over a run that covered nothing. Only reachable if an off-site run is made with nothing selected; not part of this walk's route.
  • R-244 — teardown residue in app_log_issues. Teardown is §10, owed, not this session.

§2.7 — the two functions this walk exercises, read before running

Both fixes are present in the tree at 3d3b4496f377 and both are new since the last walk:

  • the mint guardWriteOffboxSecrets (controller/internal/backup/offbox.go:438) now returns ErrOffboxSealedPackageHeld instead of generating, when sealedPackageHeld() is true and no password exists. It is a conjunction, so a first-time box mints exactly as before. The transport is still written. New declared state offsite.state=awaiting_recovery_key (OffboxAwaitingRecoveryKey, :502 — derived, not stored, and gated on t.Enabled).
  • shape (c)OffsiteRecoveryOffer (:1580) now offers when hubHash != "" && hubHash != localHash. "" falls back to (a)/(b) deliberately.

Neither has ever run on a real rebuild. That is §5, and it is deliverable 2 whatever the verdict.


Venue construction

Hub customer walk5

POST /configs/new — id walk5, name "Walk 5 R-201", domain walk5.felhom.eu, email = the operator's registered address, off-site ON (shared, 50 GB), DR tier ON, pbsdr_storage_id felhom-pbs. → 303 /customers/walk5?flash=created.

Provisioned descriptor, read from the hub DB rather than from the page:

{"enabled":true,"type":"shared","host":"u629488-sub4.your-storagebox.de","user":"u629488-sub4",
 "port":23,"repo_path":"/home/felhom-repo","quota_gb":50,
 "host_fingerprint":"SHA256:oDHZqKXnoMtgvPBjjC57pcuFez28roaEuFcfwyg8O5c"}

one_time_secrets 1 · customer_configs 1 · hosts 0. Sub-account 285351, user u629488-sub4.

⚠ THE FIRST CREATE FAILED, AND IT IS A FINDING — not a fluke of my own making.

The first POST /configs/new returned no response to the client and the hub logged [ERROR] offsite provision for walk5: offsite: host-key scan u629488-sub4.your-storagebox.de: dial: dial tcp [2a01:4f8:bacc:2:200::d30]:23: connect: network is unreachable at 13:46:46.

The scan is fail-closed by design (offsite.go:111-121: "don't serve a descriptor the controller can't verify") and its retry ladder is ~60 s (defaultScanBackoff, 2+4+8+16+30), sized by its own comment to "the observed DNS propagation lag" of a fresh sub-account. Here the lag was longer than the budget: the first five attempts failed no such host, and by the time DNS answered it answered AAAA-first, into a pod with no IPv6 route.

The retry (a second identical POST) succeeded — idempotent on the label ([offsite] shared already provisioned for walk5 (subaccount 285351)), scan green on its own final rung at ~13:47:28, config created. Total settle time ≈ 100 s against a 60 s budget.

Measured afterwards, so the cause is not guessed: the hub pod reaches u629488-sub4.your-storagebox.de:23 now, by name, and gets SSH-2.0-OpenSSH_9.6p1 — as does the node. So the endpoint is fine and the window was transient. What is not fine is the operator experience: a customer create can fail with a fail-closed error whose only remedy is "press it again", and nothing says so. Filed below.

VM

Host demo-hp (HP t740), Tier 0 — the designated drill host
VM 325 walk5-appliance — q35/OVMF (pre-enrolled-keys=0), 4 cores, 8 GB, cpu=host
Disks scsi0 200 G · scsi1 50 G · scsi2 50 G, qcow2 on c11-scratch (a dir at the mount root /mnt/nvme-1tb, which is what the agent's exactMount check requires)
Appliance walk5.felhom.eu @ 192.168.0.100/24, gw/DNS 192.168.0.1, MAC bc:24:11:37:7c:f3
Guest LXC 9201 @ 192.168.0.149 (LAN) + 169.254.253.2 (R-50 island)
Hub customer walk5 "Walk 5 R-201", DR tier ON, off-site ON (shared, 50 GB)
Host id walk5-4bada5 · appliance uuid 596d8104-2af9-4525-baa7-e479314cbb44
Off-site Storage Box sub-account 285351, user u629488-sub4 · WG peer 10.77.0.5
Untouched drill-r50 (VM 300, the protected fixture), guest 9201 on both demo boxes, DooPlex, ep0, the tester's box

Reachability, stated because it shapes every later step: DooPlex has no L2 path to the 192.168.0.0/24 LANip neigh is INCOMPLETE even for demo-hp, which it reaches over the tailnet. So the appliance is one SSH hop behind demo-hp, and every request to the guest is made from the appliance, which is exactly what a browser on the customer's LAN would do.


Phase A — the fixture (§4)

A1 — installed from the published ISO, through the Terminal UI

Driven with qm sendkey + qm monitor screendump → PNG. All three known traps reproduced:

  1. GRUB defaults to the graphical entry — and it bit, first attempt. The countdown was 14 s with my screendump→convert→read round trip at ~18 s, so the menu auto-booted the default and the GUI installer came up. Recorded rather than quietly retried: the VM was stopped and restarted, and the keys sent inside one remote command (sleep 6; sendkey down; screendump; sendkey ret) — the screendump taken between the arrow and the Enter is the positive control, and it shows the highlight on „Felhom telepítés (szöveges mód)" at the moment of the press.
  2. The keymap defaults to Hungarian while sendkey emits US scancodes. Changed to U.S. English before any typing. Positive control: the administrator email typed through the identical path rendered walk5@felhom.eu@ is shift-2 on US and AltGr+V on HU, and it is the only available evidence for the 24 masked password characters.
  3. Auto-reboot was checked. Unchecked, and confirmed [ ] with the focus moved away; then --boot set in its own qm set with the ISO detached, both verified from qm config before the first boot: boot: order=scsi0, ide2 lines = 0.

Summary screen, verbatim: ext4 · /dev/sda · Europe/Budapest · U.S. English · walk5@felhom.eu · nic0 · walk5.felhom.eu · 192.168.0.100/24 · 192.168.0.1 · 192.168.0.1.

One reading corrected by a second instrument, as in the last walk: 192.168.0.100 was the installer's prefill and answered on the LAN. The MAC was bc:24:11:37:7c:f3 — VM 325's own DHCP lease. Not a collision; a ping alone could not have told the difference.

Day-0 fired on first boot, unaided. Console showed the Hungarian pairing banner:

Felhom — a doboz készen áll, és a párosításra vár. · Párosító kód: E28-WJH

The hub's appliance_registrations row 24 carried the same code (E28WJH), the same MAC, the same SMBIOS uuid and three SSH host keys, first_seen 12:18:59Z32 seconds after boot. Bound through the real endpoint (POST /appliances/24/bind, 303).

A1b — THE HEADLINE: a fresh install DOES get the fixes

vouched the box landed on
agent 0.127.0 0.127.0
golden 0.206.0
controller (baked into the golden) 0.206.0
14:22:48 [INFO] Artifact manifest served for customer walk5 (agent=0.127.0 golden=0.206.0)
14:25:33 [INFO] Event from walk5: controller_started (info) — Controller elindult (0.206.0)
   docker ps → Up 10 minutes (healthy) gitea.dooplex.hu/admin/felhom-controller:0.206.0
   felhom-agent --version → felhom-agent 0.127.0

No hand upgrade was needed and none was applied. This is the first walk of the five where that sentence is true of the fixes under test: R-239's delivery gap — the headline finding of both previous walks — is closed for this run, and the box under test is the box a customer receives.

The whole day-0 cascade ran hands-free, bind → provision in 3 m 55 s:

14:21:38 appliance 24 BOUND to customer walk5 (mode=appliance)
14:22:00 appliance credentials DELIVERED once
14:22:08 [claim] claim code (gen 1) emailed to the registered address
14:22:38 host enrolled: walk5-4bada5
14:22:46 offsite credentials re-issued on re-enroll; break-glass root@pam vaulted (32 chars)
14:23:19 wg registered 10.77.0.5 · DR-recipe host-half · tenantsync provision ok (ns=walk5)
         pbsdr auto-provisioned (hands-free cascade)
14:25:33 controller 0.206.0 reporting

A2 — claimed

Claimed through the real POST /claim form (302 → /), then POST /login → 302. The claim code came from the documented --print-reset-code escape hatch (R-204 item 1) — a guest command line, used deliberately as FIXTURE CONSTRUCTION, permitted by §3 in Phase A, and counted as such. Phase B must not use it.

The dashboard password is a 24-character alphanumeric, 0600 in ~/.config/walk5/ on DooPlex only. The appliance's root credential is the hub-vaulted break-glass (root@pam, revealed through POST /hosts/walk5-4bada5/reveal-recovery-credential) — the password typed at the installer no longer authenticates, because day-0 step 4b rotates and vaults it. Stated because it cost a dead end.

A3 — drives, the app, and the three sentinels

Both raw disks enrolled through the customer endpoints (GET /api/disks/candidatesPOST /api/storage/init), ext4:

/dev/sdb → /mnt/felhom-drives/adatok     (app data)
/dev/sdc → /mnt/felhom-drives/mentes     (backup target)

Backup target assigned to mentes — and, exactly as the last walk recorded, it reported restart_required: true and flipped only after the agent restart it asked for:

before: {"degraded":true, …"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after:  {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}

calibre-web deployed through the real API with HDD_PATH=/mnt/felhom-drives/adatokstate: running and health_probe.healthy: true.

THE THREE SENTINELS

# file bytes sha256
A WALK5-SENTINEL-A.txt 61 11eb7fb2d1891f62a685e3d9a9fda44e0a342d437d4aebaa15fcb4c0fd79d539
B WALK5-őrszem-ékezetes-árvíztűrő.txt 66 6b504d1e83bf4c673334d0b9e701f3e5f07311d954fdc4463a6b0a55b261ae32
C WALK5-SENTINEL-C-12MB.bin 12 582 912 0baaf402733e6a85f3d9ce8c4c3a57b05692f5f6733da6a23da76d2cd7635210

Sentinel B's filename as hex, and this is the reading that cannot lie:

57414c4b352d c591 72737a656d2d c3a9 6b657a657465732d c3a1 7276 c3ad 7a74 c5b1 72 c591 2e747874
             ő                 é                    á         í       ű     r  ő

No efbfbd anywhere. Written by a Python placer pushed in with pct push, which builds the name from a hex string decoded to bytes and passes a bytes path to os.open — no shell layer ever sees the name, and the readback is from os.listdir on a bytes path, not from a shell ls. This is the trap that bit three times; it did not bite here.

Placed under …/adatok/userdata/media/books — the level the last walk got wrong. The tree was listed first and the namespace root confirmed to be /mnt/felhom-drives/adatok directly.

A4 — the escrow ceremony

Preflight 6 of 6 green (pbs_storage_id · dr_tier · age_binary · hub_upload · staged_secret · sudo_grant), agent_supported: true, stale: false.

That stale: false is a deliberate check, not a formality. R-246/R-247/R-248 concern a staleness flag that would silently disable the very discriminator this walk exists to exercise. The spike's ruling was that a fresh box cannot reach the state (MarkEscrowStale has no production caller). Measured here on a fresh box: it has not.

Ceremony through the customer wizard's own endpoints (/api/escrow/start with password re-auth — it is a form POST, not JSON, which cost one rate-limited attempt):

phase: done · restic_pw_sealed: TRUE · uploaded: true · entropy_bits: 129.24 · claimable: true
key_fingerprint: a6:86:f7:fb:74:e7:16:3c:…:31:2a:4c:f9

restic_pw_sealed: true is the field the whole exercise rests on.

R was claimed ONE-SHOT and streamed file→file into ~/.config/walk5/R_walk5.txt (0600, DooPlex only). It was never rendered, never an argument, never a log line. The appliance's copy was shred -u'd and its absence verified. Shape only: 82 characters, 10 hyphen-separated tokens, lowercase+digits.

§9.4 slip, recorded rather than buried. The Phase-A claim code (not R) reached this session's transcript, because the hatch prints it and my shape-only filter was written for an uppercase pattern and did not match a lowercase one. It was consumed by the claim seconds later and is now void. R itself was never printed and never left the file→file path.

A5 — the off-site backup, and the sentinels BY NAME

THE §4.5 GATE EARNED ITS PLACE AGAIN — and this time it caught MY fault, not the product's. The first run reported ok in 28 s with snapshots = 0 and an empty repository size. The log said why: [offbox] backup run started (0 app(s) toggled). My toggle POST sent enabled=1, and the handler accepts only on/true — so it recorded off, redirected with „A távoli mentés beállítása frissítve." and the run then correctly backed up nothing. A harness fault, not a product defect — but the shape is precisely R-240's, and had I read the green ok and moved on, the destruction would have proven nothing.

Re-toggled (enabled=on), selection verified in the rendered page (name="app" value="calibre-web"

  • checked) before the re-run, which then logged backup run started (1 app(s) toggled).

ok, 1 m 20 s, 1 snapshot, 12.0 MB (12 611 563 B). The gate is not the green tickrestic snapshots + restic ls latest --long, run against the repository with its own credentials:

5b0f20f7  2026-08-07 12:57:41  walk5  [felhom-offbox, calibre-web]
    /mnt/felhom-drives/adatok/backups/primary/calibre-web
    /mnt/felhom-drives/adatok/userdata/media/books

-rw-r--r-- 1000 1000       61  …/userdata/media/books/WALK5-SENTINEL-A.txt
-rw-r--r-- 1000 1000 12582912  …/userdata/media/books/WALK5-SENTINEL-C-12MB.bin
-rw-r--r-- 1000 1000       66  …/userdata/media/books/WALK5-őrszem-ékezetes-árvíztűrő.txt
-rw-r--r-- 1000 1000   413696  …/userdata/media/books/metadata.db        ← calibre-web's real DB
restic stats latest: 22 files, 12.784 MiB

All three sentinels are in the snapshot, by name, at the right sizes — and the accented filename survived into restic intact.

A6 — §4.6's two pre-destruction checks, neither ever exercised on a clean box

Check 1 — the recovery offer is NOT shown. PASSES.

GET /          → 302 → /launcher
GET /launcher  → 200,  recovery mentions: 0
GET /          → 200 (followed), recovery mentions: 0
GET /recovery  → 302 → /backups/remote

No banner, no entry point, nothing to seek. And the reason is measurable rather than assumed: the hub's package covers the key in use, so v0.206.0's shape (c) compares equal and correctly stays silent

box   /data/offbox/repo_password  sha256 = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
box   settings.hub_escrow_key_sha256    = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
hub   host_escrow.restic_pw_sha256      = eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
      hub_escrow_key_checked_at         = 2026-08-07T12:51:39Z

This is the first live measurement of the persisted discriminator, and it is the negative half: the shape that must NOT fire, not firing, for the right reason.

Check 2 — the restore page lists the app with the future-backup toggle OFF. PASSES.

With calibre-web toggled off for future off-site backups, /backups/restore renders identically — „Calibre-Web Automated" still listed under „Visszaállítás a távoli tárolóból", 10 calibre occurrences in the HTML both before and after.

That is R-237's fix, live. The last walk's T3 measured 0 restore entries and a 302 on controller 0.203.0 — the customer's existing backup hidden by a setting about the future. The toggle was returned to on afterwards.

A7 — the pre-destruction truth (§4.7's seven records)

Box (settings.json, secrets stripped):

offbox: enabled true · escrow_state "escrowed" · last_status "ok" · last_duration 1m20s
        last_run/last_success 2026-08-07T12:58:57Z · snapshot_count 1
        repo_size_bytes 12 611 563 ("12.0 MB") · stats_known true · quota_gb 50
        host u629488-sub4.your-storagebox.de:23 · repo_path /home/felhom-repo
repo_state: (absent — not orphaned) · hub_escrow_identity_present: true · claimed: true
agent 0.127.0 · controller 0.206.0 (healthy) · guests 1/1

Hub:

host_escrow(walk5-4bada5):  blob 383 B · identity_blob 572 B (R-198 material present)
                            key_fingerprint a6:86:f7:fb:…:4c:f9
                            restic_pw_sha256 eabf427c7274…144f
                            stale_at NULL · created 2026-08-07T12:51:02Z
host_escrow_superseded:     0 rows for walk5
hosts:                      walk5-4bada5 ONLINE · agent 0.127.0 · PBS wrapper MATCHES VOUCHED
                            (104db0a4401f…) · WG 10.77.0.5 confirmed
DR Recipe: present · Key Escrow: present

Phase A gate: PASSED. All seven records taken; the three sentinels listed by name.


Phase B — the journey (§7)

The rule: no command line inside the guest, at any point. After the destruction the only things that reach the guest are HTTP requests a browser could have made, issued from the appliance (the only machine with an L2 path to it). Instrumentation reads are counted separately and named: the w5watch.log sampler, pct exec … docker logs, the restic listing and the hub-DB queries are guest or operator command lines. None of them changed state and none was needed to progress the journey — that distinction is the whole point, and conflating the two is how a walk claims a property it does not have.

B1 — the destruction (14:19:26Z)

Guarded four ways before anything ran, because demo-hp also carries a guest 9201:

hostname == walk5                      ✓
pct config 9201 → hostname: walk5      ✓
mountpoint -q /mnt/adatok              ✓
mountpoint -q /mnt/mentes              ✓
14:19:18Z  pct stop 9201
14:19:26Z  pct destroy 9201 --purge
           → "Logical volume vm-9201-disk-0 successfully removed"
           → "Logical volume vm-9201-disk-1 successfully removed"
           → pct list EMPTY · lvs | grep 9201 → NONE
14:19:27Z  /mnt/adatok and /mnt/mentes wiped 13M/1.2G → 4.0K each,
           MOUNTS LEFT IN PLACE — deliberately: the surviving raw mount IS the R-220 condition

B2 — the rebuild (14:24:09Z, 3 m 31 s)

felhom-host-install.sh fetched live from felhom.eu/scripts/v1.25.0, sha256 cb4d02059f126f8bfdddfe98607e87a3cb45dd2592c6b40b43c9659e2b687534. The retrieval passphrase was passed as --passphrase-file (a 0600 file), never on the command line.

[OK] Day-0 provision SUCCESS — vmid=9201 host_id=walk5-4bada5 customer=walk5
     golden=local:backup/vzdump-lxc-9100-2026_08_07-16_21_39.tar.zst
[OK]   controller: Up 19 seconds (healthy)
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.206.0
[OK]   authz signers: 2 (operator-signed self-update armed)
[INFO] root@pam was rotated + vaulted at step 4b

What the rebuild landed on:

before after
agent 0.127.0 0.127.0 — no downgrade, no hand upgrade
controller 0.206.0 0.206.0 — same, and it is the vouched build

No hand upgrade, either time. R-216 part 4's reinstall-downgrade did not recur.

Guest re-provisioned at 192.168.0.83 (it was .149 before — a fresh DHCP lease, not a fault).

B3 — §5's observation window: THE MOMENT THAT MATTERS

T0 — 14:24:31Z, 22 seconds after provision, nobody has logged in:

offbox directory                  ABSENT  ("No such file or directory")
repository key                    NONE
settings.offbox                   null
settings.hub_escrow_identity_present   TRUE   ← the box already knows a package is held for it
settings.hub_escrow_key_sha256    eabf427c7274…144f  ← and already knows WHICH key it seals
settings.claimed                  null   (pristine, unclaimed)

The scheduler is alive — the positive control, because an absent line is not evidence. In the first five minutes the controller logged agent-channel-health ×5 (1 m cadence), stack-scan ×2, system-health, backup-cache and offsite-credential-retry. So "no key was written" is a measurement, not a silence.

The window, minute by minute — nobody logged in at any point during it:

time (UTC) event source
14:24:09 Day-0 provision SUCCESS; rebuilt guest, controller 0.206.0 installer
14:24:31 T0 — no offbox dir, no key, offbox: null, but hub_escrow_identity_present already true and the sealed key's hash already learned box, measured
14:24:53 hub: offsite_delivery_stuck (warning) + operator notification_log row — REFUSED, "regressed-apply shape … a guest rebuild does, R-193" hub, measured
14:23:54 first post-rebuild report carries no offsite object — it predates the ACK that set the flag hub reports, measured
14:38:49 first declaration: offsite={"enabled":false,"state":"needs_credential"} hub reports, measured
14:38:53 the checker hands off: "the box DECLARES offsite.state=needs_credential; internal/offsiteheal owns this remediation" hub, measured
14:53:49 second declaration, same shape hub reports, measured
14:56:34 offsiteheal: re-staged the stored one-time offsite secret … (declared needs_credential across 2 reports) … no provider credential was minted hub, measured
14:58:52 the box collects it and applies the tier — AND REFUSES TO MINT box log, measured

THE LINE THIS WALK EXISTED TO SEE

14:58:52 [WARN] [offbox] NOT minting a repository password: the hub holds a sealed recovery package
         for this box, and a fresh key would orphan the history that package protects (R-241).
         The transport is configured; the tier stays down until the customer's recovery code places
         the escrowed key.
14:58:52 [INFO] [offbox] apply-offsite: transport configured for u629488-sub4@…:/home/felhom-repo,
         tier HELD awaiting the escrowed key (the hub holds a sealed package; no key was minted — R-241)
14:58:52 [INFO] [offsite-apply] credential retry: the staged credential was collected and the tier applied

At the equivalent moment the previous walk minted 9b4a9a9d… over a package sealing 30ef574f…, and lost the journey silently at 03:18 in the morning. This build did not.

§5's four items, answered

1. Does the box declare a need, and when is it staged and collected? Yes — declared needs_credential at 14:38:49Z and 14:53:49Z, staged by the hub unaided at 14:56:34Z, collected and applied at 14:58:52Z. Zero human actions; the whole chain ran on a box that had not even been claimed.

2. Is any repository key written? NO. Sampled every ~20 s from 14:24:52Z onward: repo_password_sha=NONE at every single sample. At 15:00:24Z the directory holds exactly:

-rw------- applied_marker   64
-rw-r--r-- known_hosts     290
-rw------- ssh_key         400
            repo_password: ABSENT

The transport is present and the key is not — which is the guard's designed shape, not a partial failure: ApplyOffsiteTarget deliberately keeps the transport so the recovery screen can bring the tier up the instant the key arrives (R-219).

3. What state does it report instead? offbox.enabled = true, escrow_state = pending, repo_state absent — and OffboxAwaitingRecoveryKey() is therefore true (target exists, target enabled, no password), so the box declares the new awaiting_recovery_key holding state rather than coming up under a fresh key.

4. The two fingerprints, before anyone logged in (15:00:24Z):

the escrowed key (hub's package, ACK-cached)  eabf427c72749fdad355d8cbcadedd8430c8398999b6385cfb1960899167144f
whatever the box holds                        NONE — there is no repository key on this box

HONEST NOTE ON WHICH SHAPE FIRES, because it changes what has been proven. With the mint guard holding, the box has no local key at all, so OffsiteRecoveryOffer returns true on shape (a) — the pristine-rebuild shape — not on shape (c). Shape (c) was measured in Phase A, in its negative half: equal hashes, correctly silent. So this walk proves the mint guard positively and the discriminator negatively; a positive shape-(c) firing needs a box holding a different key, which v0.206.0 now prevents from arising by itself. That is the intended end state (the fix removes the state its own second half was written to describe), but it should not be reported as "shape (c) proven".

B4 — the journey, step by step (§7)

# step result
1 Destroy — 14:19:26Z guest 9201 purged (both LVs), both drives wiped 13M/1.2G → 4.0K. Host identity walk5-4bada5 survived
2 Reinstall felhom-host-install.sh v1.25.0 fetched live; Day-0 provision SUCCESS 14:24:09Z, 3 m 31 s. controller 0.206.0, agent 0.127.0 — no hand upgrade, no downgrade
3 §5's window no key minted; transport configured; awaiting_recovery_key — see B3
4 Claim back new code requested through the customer-facing „Új kód kérése" path (15:01:27Z), hub rotated to generation 2 and emailed it; the operator relayed it; worked FIRST TRY — 302 in 0.151 s, accents intact (á=c3a1, í=c3ad, sha256 identical source→appliance)
5 Log in / → 302 /launcher → 302 /recoverythe recovery screen appeared WITHOUT BEING SOUGHT
6 Read the screen all three questions answered — quoted below
7 Enter R HTTP 200 in 12.44 s — a real unseal. [web] recovery: the offsite repository key was recovered and placed (outcome=installed)
8 The listing renderedcalibre-web · 2026-08-07 14:57 · 12.8 MB
9 Restore all three sentinels byte-identical — after two unsignposted obstacles, both cleared from the dashboard

The recovery screen, as a customer reads it (§7.6)

„Adatok visszaszerzése"

„Ezt a gépet újratelepítették. A korábbi, házon kívüli mentéseid megvannak — a Felhom központi rendszere őriz hozzájuk egy lezárt csomagot, amelyet 2026-08-07T12:51:02Z zártunk le. A csomagot csak a te helyreállítási kódoddal lehet kinyitni.

A helyreállítási kódot senki nem tudja pótolni — sem a Felhom, sem az ügyfélszolgálat, sem az üzemeltető. Ez szándékos: így a mentéseidet rajtad kívül senki nem tudja megnyitni. Ha a kód elveszett, a korábbi mentések nem nyithatók meg többé.

Ha megadod a kódot, feloldjuk a mentéseid zárolását és megmutatjuk, mi van bennük — melyik alkalmazás, mikorról, mekkora. Ebben a lépésben semmit nem állítunk vissza és semmi nem változik."

All three questions answered. The sealed-at timestamp 2026-08-07T12:51:02Z matches the hub's host_escrow.created_at exactly. The field label reads „Helyreállítási kód (tíz szó)" and R is indeed ten tokens.

(One copy wart, recorded not filed: the sealed-at is rendered as a raw ISO-8601 string on a customer-facing Hungarian screen, where every other date on the box is formatted 2026-08-07 14:57.)

The listing, against Phase A's record (§7.8)

Phase A recorded the screen showed
app calibre-web calibre-web
when snapshot 5b0f20f7 at 12:57:41Z 2026-08-07 14:57 (12:57Z rendered in CEST)
size restic stats 12.784 MiB 12.8 MB

(A second row felhom-offbox · 12.8 MB also appears — the snapshot carries both tags and the screen renders one row per tag. A customer sees an "app" they have never installed, and the same 12.8 MB counted twice. Cosmetic; filed as R-251.)

THE RTO (§7.7)

login                      15:04:25.417Z
unlock returns 200         15:05:37.156Z
------------------------------------------
wall clock, login → open store      71.7 s

Of which 12.44 s was the unseal itself. ~22 s of the remainder was my own harness retry — I scraped the CSRF token from a <meta> tag the recovery page does not carry, got a 403, and re-read it from the form (the controller logged CSRF rejected … (token mismatch) at 15:05:02). A customer clicking the button would have seen ≈50 s. Both numbers are stated because the 71.7 s is what was actually measured.

The restore, and the two obstacles (§7.9)

POST /backup/offbox/restore (mode=full) → refused: „nincs elérhető adatmeghajtó a visszaállításhoz". Then, after that was cleared, POST /backup/offbox/reconstituterefused: „a(z) calibre-web nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat".

Both were cleared from the dashboard, with no guest shell:

  1. Re-attach the drivesGET /api/disks/candidates offered both surviving disks (mountable: true, data_bearing: true), and Tárhely → Meghajtók → „Meglévő meghajtó csatolása" (POST /api/storage/attach) registered them: /mnt/felhom-drives/adatok and …/mentes.
  2. Redeploy the appPOST /api/stacks/calibre-web/deploy, running in 1 m 30 s.

Then prepare (12.8 MB) → confirm → reconstitute:

15:10:18  [offbox] restored calibre-web (5b0f20f7, full=true) → …/backups/offsite-restore/calibre-web
15:14:30  [offbox] reconstituted calibre-web from snapshot 5b0f20f7: 6 file(s) placed,
          0 DB dump(s) replayed, safety dump=., skewed=false

THE THREE SENTINELS, read back as BYTES from the live path

MATCH [A] 11eb7fb2d1891f62a685e3d9a9fda44e0a342d437d4aebaa15fcb4c0fd79d539        61
          name_hex 57414c4b352d53454e54494e454c2d412e747874
MATCH [B] 6b504d1e83bf4c673334d0b9e701f3e5f07311d954fdc4463a6b0a55b261ae32        66
          name_hex 57414c4b352d c591 72737a656d2d c3a9 6b657a657465732d c3a1 7276 c3ad 7a74 c5b1 72 c591 2e747874
MATCH [C] 0baaf402733e6a85f3d9ce8c4c3a57b05692f5f6733da6a23da76d2cd7635210  12582912
          name_hex 57414c4b352d53454e54494e454c2d432d31324d422e62696e
all three present: True

Identical to Phase A in every byte, including the accented filename. Read with os.listdir on a bytes path, so no decode/encode round trip could launder a U+FFFD into looking correct.

The end state — the loop closed

escrow_state              escrowed        (auto-confirmed: hub hash == local hash)
repo_password sha256      eabf427c7274…144f   == the escrowed key, == Phase A's key
GET /launcher             200             (the recovery screen no longer interrupts)
GET /recovery             302             (correctly retired — shape (c) compares equal again)

THE VERDICT — both halves, separately

THE DATA: PASS

All three sentinels came back byte-identical, out of snapshot 5b0f20f7, under the key recovered from the sealed package with R — including the accented filename's bytes. Disk → restic → SFTP → Storage Box → a destroyed machine → a rebuilt machine → disk, intact.

THE JOURNEY: PASS — the first time in five walks

No step needed a command line inside the guest. Every step that progressed the journey — request a code, claim, log in, read the screen, enter R, read the listing, re-attach the drives, redeploy the app, prepare, confirm, restore — was an HTTP request a browser makes, issued from the appliance because DooPlex has no L2 path to that LAN.

Guest command lines used, named per §3 so the claim is not read wider than it is: the w5watch.log sampler, docker logs, the settings reads, the restic listing and the final sentinel verification. Every one is instrumentation — none changed state, none was needed to progress, and removing them all would have changed nothing except my ability to describe what happened. The previous walk needed three guest command lines to progress; this one needed zero.

The reset-code hatch was used ONCE, in Phase A, where §3 permits it. It was not used in Phase B.

But the journey is not smooth, and two obstacles are new

Neither is a dead end by §3's definition (neither needed a guest shell), and both were cleared from the dashboard — but a customer would not know to do either, because the product does not say so and, in the second case, says the opposite:

  1. The restore refuses with „nincs elérhető adatmeghajtó" and nothing on the recovery path mentions that the drives must be re-attached first.R-252
  2. „a(z) calibre-web nincs telepítve — előbb állítsd helyre az alkalmazást" — on a page that says, three lines above, „Nincs telepítve — a visszaállítás előbb újratelepíti." The product contradicts itself in the customer's own language, at the last step of a recovery. → R-253

So: the machinery works end to end and the data is provably safe. The unaided journey now succeeds, and it succeeds through two obstacles the customer must guess their way past.