felhom.eu/documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md

REHEARSAL — the BYO reinstall walk (2026-08-09)

Status: COMPLETE. All phases walked; the integrity verdict is BYTE-IDENTICAL. §1–§8 were written before the destructive phase deliberately — a finding that exists only in a session that later crashes is a finding nobody has — and are left as written, including one claim later refuted by test (§8.3 → §9a) and one by measurement (§7.2 F-9).

Venue: demo-hp (HP t740, felhom-host, guest 9201, customer demo-hp). Operator-approved at STOP 1. Driven from DooPlex. All times UTC unless marked; the host runs CEST (UTC+2).

---

1. Baselines — re-confirmed live on arrival, not taken from the spec

	spec said	live reading	source
felhom-agent	v0.128.0 @ 28ba8593b8	0.128.0 on demo-felhom, 0.127.0 on demo-hp; HEAD == origin/main == 28ba8593b8	felhom-agent --version on both nodes; git rev-parse
felhom-controller	v0.210.0 @ c732fe1283	0.210.0 demo-felhom, 0.208.0 demo-hp; HEAD == origin/main == c732fe1283	hub /configs; git rev-parse
hub	v0.101.0 @ 56f8aa611c	0.101.0 (deployed image tag matches)	kubectl get deploy hub; page footer
Day-0 manifest	golden 0.210.0 / agent 0.128.0 / min 0.127.0	all three already saved — golden 0.210.0 (b9f701fa…), agent 0.128.0 (c6eba73b…), min agent 0.127.0, wrapper 104db0a4…	hub /configuration, selected <option> values

All three repos were clean and at origin/main before anything ran.

The fleet is not uniform, and the venue is the box that is behind — demo-hp runs agent 0.127.0 and controller 0.208.0 against a manifest vouching 0.128.0/0.210.0. That is an argument for the venue: the reinstall genuinely exercises the manifest rather than re-installing what is already there.

---

2. Venue comparison — the fidelity table that decided it

The runbook's criterion is "the box whose current state most resembles what the tester will have".

criterion	demo-felhom (N100)	demo-hp (t740)
customer apps deployed	1 — opengist	3 — privatebin, opengist, calibre-web
enrolled user-data drive	none registered. /mnt/felhom-drives empty; controller logs [WARN] Storage paths: no storage paths registered	/mnt/felhom-drives/Felhom-Share + NVMe enrolled 2026-07-22; one registered storage path /mnt/sys_drive
off-site tier has actually run	NO — offsite.state=needs_credential, no run has ever succeeded	YES — 18 restic snapshots, daily, unbroken
escrow / recovery ceremony	present, generation 2	present, generation 2
install shape	mkimage loader, Secure Boot OFF (firmware workaround)	shim loader, Secure Boot ENABLED — the customer shape
collateral on the box	none	drill-r50 (VM 300); leftover c11-scratch storage

Chosen: demo-hp. It wins every fidelity criterion that distinguishes them.

drill-r50 is not at risk, and this was checked rather than assumed: VM 300 is not in the felhom pool (pvesh get /pools/felhom → members [9201]), its disks are on local-lvm, and --uninstall removes no storage definitions and no non-pool guest — its own end-of-teardown statement says the enrolled drives are "unmounted only, NEVER wiped" and that it never deletes PBS backups, hub records or escrow blobs.

Correction recorded, because it was part of the comparison put in front of the operator: the enrolled-drive row was first reported as "real files on an enrolled drive". Felhom-Share is empty — the share exists and is enrolled, but holds no files. demo-hp still wins the row (demo-hp has a registered storage path; demo-felhom has none), on a narrower margin than first stated.

---

3. Pre-phase — R-268, the leaked per-guest local-API token

Done first and independently, on demo-felhom guest 9201, as the runbook required. No token value reached stdout at any point: old value captured file→file from bootstrap.json, new value minted into a file, both shredded at the end.

3.1 The rotation, and its proof

step	old token	new token
before rotation	HTTP 200	—
bogus token (negative control)	HTTP 401	—
after the on-disk rotation, before any reload	HTTP 200 ← see 3.2	200
after that lookup forced a reload	HTTP 401	—
after systemctl restart felhom-agent	HTTP 401	HTTP 200

Positive observable for the controller half: local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible, enumerating mp8/mp9/mp0 — data only the agent can supply. An absent error would not have been evidence.

R-268 is CLOSED. The old token is refused; the new one works; the controller's channel is up.

3.2 FINDING — an out-of-process token rotation does not revoke the old token

localapi.TokenStore.Mint's doc comment states "last-write wins — any previous token for this guest is revoked". Across processes that is false until something else forces a reload. The long-lived agent serves Lookup from an in-memory index and re-reads the store only on a miss (the B3 reload-on-miss optimisation). A superseded token is still a direct map hit, so it authorises.

Red-proved twice. A temporary unit probe (Scenario C with the lookup order reversed — present the rotated-out token first) fails on the claim; the shipped TestTokenStore_ReloadOnMiss_RemintCoherence passes only because it looks up the NEW token first, which is what triggers the reload that then evicts the old hash. Then proved on real hardware: the row marked ← above is a live 200 from a token that the store on disk had already superseded.

This is the CLAUDE.md case exactly — a comment asserting an invariant with a test that appears to pin it and does not. The honest fix is either a test that pins the reversed order, or an eviction that does not depend on an unrelated lookup missing first.

3.3 FINDING — R-268's own stated rotation recipe is incomplete

The register row says the new plaintext "must also be written into the guest's /etc/felhom-bootstrap/bootstrap.json or the in-guest controller loses its agent access". Writing bootstrap.json is not sufficient. bootstrap.ensureLocalAPI returns early when cfg.LocalAPI.Endpoint != "" — it fills an absent block and never refreshes a present one — so after a rotation the controller keeps serving the old token from its own controller.yaml across a restart. Proved live: two controller restarts, still HTTP 401. The rotation only completed once local_api.token was written into controller.yaml itself.

The neighbouring DetectEndpointDrift compares the endpoint and deliberately does not compare the token ("a token mismatch is a different failure"). So this shape is known to be unmodelled, and R-78 ("which file is authoritative") is its parent.

3.4 FINDING — the agent-channel alarm never closes, and its own prescribed remedy is why

channelhealth alerted correctly and promptly: agent_channel_unauthorized, no debounce, pushed to the hub — "agent rejected the controller token (HTTP 401) — token stale/rotated (re-bootstrap)".

The channel then recovered, and no recovery event was ever sent. In Checker.Check's UP branch the notify is guarded by prev != "" && prev != "up", and a controller restart resets state to "". The alert's own instruction is to re-bootstrap — i.e. restart the controller — so following the alarm's advice structurally guarantees the recovery is silent. The down side is deliberately asymmetric (F2: a born-down channel alerts on cycle 1); the up side is not.

Operator-visible result on the hub: two agent_channel_unauthorized errors, one sent and one suppressed by cooldown, and nothing afterwards. The customer dashboard is fine — SetDashboard reflects current state every cycle. It is the operator's trail that ends on "down".

---

4. Gate 0

4.1 The recovery code — CONFIRMED IN THE OPERATOR'S HANDS (STOP 1)

Not printed, not logged, not requested. It will be supplied at the screen at STOP 3.

4.2 The dataset — planted and fingerprinted

Planted at /var/lib/felhom/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/ — inside the Calibre library, which is real app-owned customer data and is demonstrably carried off-site (the path appears in every felhom-offbox,calibre-web snapshot). Chosen after checking coverage rather than assuming it: the per-app tier-1 unit carries only the named docker volume, so a file dropped in an arbitrary place would not have travelled.

sha256 (16)	bytes	name	name bytes (hex)
54b773c46bbfd994…	3 145 728	binary-3mb.bin	62696e6172792d336d622e62696e
52a5c5ebfcac247f…	59	árvíztűrő-tükörfúrógép.txt	c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702e747874
15d2024dfc224162…	25	nested/őszibarack.md	6e65737465642fc591737a6962617261636b2e6d64
924497918e55fe6d…	21	plain.txt	706c61696e2e747874

Manifest: documentation/audits/evidence/ → recorded in the session scratchpad as GATE0-before-manifest.json. The manifest keys on raw name bytes, not rendered text.

4.3 The positive control — the comparator was watched failing, three ways

control	expected	result
identical copy	PASS	PASS, exit 0
one bit flipped at byte 1 500 000 of the 3 MB binary	FAIL	FAIL — named the file and both sha256s
NFC→NFD rename of the accented name (renders identically)	FAIL	FAIL — caught as MISSING+EXTRA on the name bytes
one file deleted	FAIL	FAIL — named the missing file

The NFD case is the one that matters: a comparison of rendered filenames would have passed it. Every altered copy was discarded after the control ran.

4.4 The off-site tier — has run since the dataset was planted

Triggered through the product's own dashboard button (POST /backup/offbox/run → flash „A távoli mentés elindult"), not by invoking restic by hand.

[offbox] backup run started (3 app(s) toggled)
[offbox] pre-push dump leg completed in 7.329s — snapshot pair is coherent
[offbox] backed up calibre-web (…/backups/primary/calibre-web, 1 mandatory path(s))
[offbox] backup OK: 3 app(s) backed up, 18 snapshot(s), 1m22s

The restore point, by identity — never by size:

snapshot	time (UTC)	tag	paths
41c830db	2026-08-09 08:30:38	felhom-offbox,calibre-web	…/backups/primary/calibre-web + …/userdata/media/books
9e38b84c	2026-08-09 08:30:49	felhom-offbox,opengist	…/backups/primary/opengist
78b93f04	2026-08-09 08:30:53	felhom-offbox,privatebin	…/backups/primary/privatebin

restic ls 41c830db confirms all four planted files are present, both accented filenames included.

4.5 The box, captured

GATE0-demo-hp-before.txt (282 lines): agent version, pveversion, pct/qm lists, pool membership, pvesm status, storage.cfg, df, lsblk with serials, the agent unit + service user, sudoers, the pveum users/tokens/roles/ACL, the guest hook, the WireGuard tunnel, the OOB belt, the agent state dir, the install state file, PBS snapshots, local vzdump archives, and the enrolled-drive tree.

---

5. Findings so far, ranked by what they cost the person in front of you

Nothing yet is rank 1 or 2 — the walk has not started.

Rank 3 — misleads

• F-1 · A rotated-out local-API token still authorises (§3.2). A customer would do nothing — they cannot see it. An operator rotating a leaked token would believe it revoked when it is not. The shipped test passes for the wrong reason.
• F-2 · R-268's rotation recipe is incomplete (§3.3). Whoever follows it gets a box whose agent channel is dead across restarts, with a correct-sounding alarm and no hint that controller.yaml is the file that matters.
• F-3 · The agent-channel alarm never closes (§3.4). The operator fixes it as instructed and is never told it worked; the hub's last word stays "down".
• F-4 · Three hub surfaces jointly misrepresent a healthy off-site tier as absent. The customer page reads Snapshots 0 / Repo Size 0 MB / Integrity Unknown (that panel renders the local tier, while the healthy offsite object — snapshot_count: 18, last_status: ok — sits unrendered in the same report); the Offsite page shows 0.0 GB (true, but a 162 KB repo rounds to nothing); and a stale offsite_delivery_stuck event from 2026-08-07 reads as current. Three independent surfaces agreeing on a wrong picture is how a working backup gets "fixed". This one caught me — it produced a wrong statement to the operator at STOP 1, corrected before it changed anything.
• F-5 · There is no operator-triggerable off-site backup. The only route is the customer's dashboard button; signed_jobs needs a signing key the hub does not hold. Sibling of R-177.

Rank 4 — cosmetic / hygiene

• F-6 · Leave-behinds from earlier sessions, found while capturing state:
  • demo-hp host: storage c11-scratch (Campaign 11), and felhom-backup:…vzdump-lxc-9100-2026_08_03… for a guest that no longer exists;
  • demo-hp guest 9201 /root: .dpw (13 bytes), .h, .sec.html, all 2026-08-07 16:08;
  • demo-felhom guest 9201: /root/r204-backup/offbox.orig.
• F-7 · The managed-update floor is 0.200.0, not 0.156.0 — the runbook's §8.3 premise is stale. Confirmed twice: hub /configuration reads "Effective floor v0.200.0 — source: DB (hub_settings); env fallback would be v0.120.0", and both boxes log settle-gate: GO — at/above floor 0.200.0. The ruling's substance still needs recording; its number must not be written down as 0.156.0.

Not a finding, checked and cleared

• The two userdata trees under sys_drive are the same directory surfaced twice, not a backed-up copy beside an unprotected one. Checked against calibre-web's actual bind mounts.

---

6. Steps taken off-path, and what they cost

Rule 2 (no shell after the install command returns) does not bind yet — the walk has not begun. These are Gate 0 preparation acts by the operator, recorded because they are mutations:

1. R-268 rotation on demo-felhom — token store appended, bootstrap.json and controller.yaml rewritten, agent and controller restarted. Required by the runbook's pre-phase. Not the venue.
2. demo-hp's dashboard password was re-set to the value in the operator's credentials file. The customer-owned password was not known to this session, and no operator-side route to the off-site button exists (F-5). The prior password_hash is preserved at /root/.orighash and /root/settings.json.rehearsal-orig in guest 9201; claimed was left true. Done on operator instruction. Cost to fidelity: none for the walk — P1 destroys this guest, and P4 re-claims with a password the operator chooses.
3. The off-site run was started by a script pressing the dashboard's own endpoint, with a real session and a real CSRF token, rather than by a person clicking. The server path is identical; only the click is synthetic.

---

7. P1 — removing Felhom from the host

--uninstall, the product's own path, dry-run first. 60 seconds (08:37:23 → 08:38:23 UTC), exit 0. The typed-vmid confirmation was satisfied, not bypassed.

7.1 The reversal, proved rather than trusted

component	before	after
agent binary	/usr/local/bin/felhom-agent 0.127.0	ABSENT
felhom-agent service user	present	ABSENT
felhom-op (OOB) user	present	ABSENT
/etc/felhom-sshd (OOB belt)	present	ABSENT
systemd units	agent, mgmt-watchdog, felhom-sshd, oob-nft, shared-parent	all removed
pveum roles	FelhomAgentBase/Guest/Store	none
pveum user + token	felhom-agent@pve!agent	gone; user list is root@pam only
pveum ACL	pool + 9 scratch-vmid + 3 storage grants	empty
felhom pool	members [9201]	[] — pool removed
guest 9201	running	destroyed
guest-hook snippet	present	removed
dnsmasq snippets	present	removed
/var/lib/felhom-agent, /var/lib/felhom-install	present	removed
KEPT as declared		PBS snapshots (2), local vzdump (4), /mnt/felhom-drives/Felhom-Share, storage defs

Space returned: local-lvm 20 904 790 → 12 355 143 KiB (≈8.5 GiB); local ≈64 MiB; the NVMe unchanged (backups deliberately kept). drill-r50 (VM 300) untouched, as predicted.

7.2 What it left behind that it did not declare

F-8 · Five orphaned agent.json.* backups survive, each carrying a hub API key and a Proxmox token. /etc/felhom-agent/ remains with agent.json.campaign8-before, .campaign9-before, .campaign9-prev, .pre-e-target-move, .pre-prunegate.bak — all 0600, three owned by the now-deleted service user's uid 999, each holding a 64-char hub.api_key and a 59-char proxmox.token.

The uninstall's own WIPED statement claims it removes "config (+ its .bak backups)", and scripts/CHANGELOG records F1 as "uninstall now purges the agent config's .bak* siblings (one held a live hub api_key)". That fix does not match the filenames in use — and note it misses agent.json.pre-prunegate.bak, a file that literally ends in .bak.

Assessed rather than assumed: these particular secrets are SUPERSEDED, not live. The orphaned hub.api_key hashes to a5d2222a…; the hub's current key for demo-hp is 8c59d1b6…. The Proxmox token was deleted by this same uninstall. The exposure is timing-dependent, not absent — a backup taken shortly before an uninstall would hold a current key, which is exactly what R-179's precedent records. The broken thing is the stated guarantee.

And the reinstall hands them to the new service account. The rebuilt box recreates felhom-agent at uid 999 — the same uid the deleted one had — so the three backups owned by uid 999 become the new account's files. Verified: su -s /bin/sh felhom-agent -c 'test -r …' succeeds on agent.json.campaign8-before and agent.json.pre-prunegate.bak. A fresh install's service account inherits read access to the previous install's credentials. Superseded here; live if the backups are recent.

F-9 · A stale sudoers file survives — and it is INERT. Hypothesis raised, then refuted by test. /etc/sudoers.d/felhom-agent.bak-pre-e2a (21 833 bytes, 2026-07-28, 0440 root:root) survives the uninstall and grants felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, …, and visudo -c -f on it says parsed OK.

The obvious worry — that the reinstall recreates the felhom-agent user and the box silently regains a pre-E-2a privilege set — was tested against the rebuilt box and is FALSE. With the user recreated (uid 999) and this as the only felhom file in /etc/sudoers.d, sudo -l -U felhom-agent returns "User felhom-agent is not allowed to run sudo on felhom-host." sudo ignores files in sudoers.d whose names contain a dot, so …​.bak-pre-e2a is never loaded. That is a robust reason, not luck — visudo -c -f <file> checks a file explicitly and therefore does not tell you whether sudo loads it.

What is left is rank 4: 21 KB of the privilege grammar lingering on a host Felhom was told to leave, inside a teardown that claims to have removed the sudoers file.

F-10 · The WireGuard tunnel to Felhom's off-site endpoint stays up. After an uninstall, wg-quick@wg-felhom is enabled and active, /etc/wireguard/wg-felhom.conf present, with a live handshake to 167.233.158.164:443 52 seconds old and 5.86 GiB received / 2.48 GiB sent on the counter. It appears in neither the WIPED nor the KEPT list. A box told to leave Felhom keeps a live inbound route into Felhom's infrastructure, and the operator is not told.

F-11 · Smaller undeclared residue: /etc/felhom/ (.bootstrap-done, appliance-pairing-code — the console pairing code); /etc/systemd/system/felhom-bootstrap.service + /usr/local/sbin/felhom-bootstrap.sh; the vmbr9 island-bridge stanza (169.254.253.1/30) still in /etc/network/interfaces.

F-12 · A teardown-ordering warning: user config - ignore invalid acl role 'FelhomAgentStore' — the role is deleted before an ACL that still references it. Harmless here; noise that would mask a real one.

---

8. P2 — the preflight, as the owner runs it

8.1 The refusals, judged on their messages

First run, no caps: byo mode requires explicit --cores and --memory (noisy-neighbor caps on a host you do not own). Good — names the missing thing and the reason.

Second run: it refused again —

[ERROR] a resolver is already bound to :53 on this host — Felhom needs the guest reachable by name on your LAN.
  Stop or reconfigure that resolver, OR point your LAN DNS at the guest's address, then re-run.
  (Felhom does NOT touch DNS services on a host it does not own — this is a refusal, not a change.)

As a message this is excellent: the finding, the evidence, two routes out, and an explicit promise about what it will not do.

8.2 F-13 (RANK 1 — STOPS THE VISIT) · Felhom's uninstall leaves the exact condition that makes Felhom's reinstall refuse

The resolver it is refusing over is Felhom's own, and the chain is fully evidenced:

fact	evidence
Felhom installed dnsmasq	/var/lib/dpkg/info/dnsmasq.list dated 2026-07-21 18:24 CEST — demo-hp's Felhom day-0
it was constrained by a Felhom snippet	/etc/dnsmasq.d/ held Felhom snippets before; only README after
the uninstall restarted it unconstrained	the running process started 2026-08-09 10:37:39 CEST — inside the uninstall window (10:37:23–10:38:23)
unconstrained, it grabs the wildcard	0.0.0.0:53 and [::]:53, --local-service, empty dnsmasq.conf
it is not PVE's	/etc/pve/sdn/ empty; unit is stock /usr/lib/systemd/system/dnsmasq.service, enabled

The uninstall mentions it only as "the 'sudo' and 'dnsmasq' packages were left installed (system packages)" — but dnsmasq is not a system package here; Felhom installed it, and it left it enabled and running as a wildcard resolver.

What a customer would do next: read a message that says a resolver is in the way, conclude their own network setup is at fault, and start debugging DNS they never configured. On a BYO box with no shell guidance, this is where the visit stops.

Confirmed by counterfactual: systemctl stop dnsmasq && systemctl disable dnsmasq → host DNS (:53): free → PRE-FLIGHT PASS. Nothing else changed. (Recorded as an intervention — it is the action the message tells the owner to take.)

8.3 F-14 (RANK 2) · A local golden is adopted with NO version and NO checksum check

Preflight reported golden (local): local:backup/vzdump-lxc-9100-2026_08_03-07_33_00.tar.zst. Step 7 is:

if [[ -n "$GOLDEN_VOLID" ]] && ! $FORCE_GITEA_GOLDEN; then
    log_skip "  using local golden: $GOLDEN_VOLID"
    _state_mark golden; return 0
fi

The hub manifest's golden.sha256 — the whole point of the hub vouching a different trust root than Gitea — is consulted only on the fetch path. A locally-present archive bypasses the vouch entirely: no version comparison, no digest, no warning.

What that archive actually contains, read out of the tar: gitea.dooplex.hu/admin/felhom-controller:0.192.0, against a vouched golden of 0.210.0.

Two consequences, and the second is the serious one:

1. The rebuilt box comes up 18 controller releases stale, reporting success.
2. 0.192.0 is below 0.200.0 — the release where R-193's off-site recovery SCREEN shipped. So a customer reinstalled today returns on a controller that cannot run the recovery ceremony their data depends on. It is also born below the managed-update floor (0.200.0), and the updater's auto-target is the floor, never the newest — so it converges to 0.200.0, still not the vouched build.

This is the R-111/R-115/R-120 drift family, one layer down: the R-120 gate guards what the operator may vouch; nothing guards what an install actually takes. It also compounds with P1 — the uninstall deliberately keeps the old golden ("golden vzdump left in place"), and the reinstall then adopts it.

The walk continues unflagged (--force-gitea-golden NOT passed), because measuring the path a real reinstall takes is the point. §9 records which controller version actually came up.

8.4 The BYO mutation disclosure — read as a customer

Comprehensive and honest: every pveum artifact, the sudoers file, the state and config paths, the packages it will install, the guest's capped size, the self-update authority, and — creditably — the outbound WireGuard tunnel, in plain words, with the note that the operator can disable it per customer. It closes with NOT touched in byo mode: root@pam (no break-glass), host DNS (:53), OOB sshd.

One material omission, given §8.3: it does not say that a pre-existing local golden may be used instead of the vouched one. The disclosure lists everything the install will create and nothing about what it may reuse.

---

9. P3 — the install. IT FAILED, AND THE WALK STOPS HERE

Started 08:51:33 UTC, failed 08:52:17 UTC (44 s), exit 1, at step 5/8. Run unflagged, as a real owner would. The typed hostname acknowledgement was given, not bypassed.

[INFO]   manifest: agent v0.128.0 (sha c6eba73bf9b9ad69…), golden v0.210.0
[OK]     verified sha256 c6eba73bf9b9ad69… matches the hub manifest
[OK]     installed /usr/local/bin/felhom-agent (felhom-agent 0.128.0)
[OK]     created service user felhom-agent
curl: (22) The requested URL returned error: 404
[ERROR] raw fetch failed: configs/felhom-mkfs-guarded.sh (agent tag v0.128.0 — is that version tagged in felhom-agent?)

9.1 F-15 (RANK 1 — STOPS EVERY INSTALL, FLEET-WIDE) · The vouched agent version was never tagged

v0.128.0 does not exist as a git tag — not locally, not on Gitea. The newest tag is v0.127.0. The binary was published to the Gitea package registry (which is why it is selectable in the hub's artifact dropdown and why its sha256 verified), but the release's tag step never happened.

ref	HTTP
raw/branch/main/configs/felhom-mkfs-guarded.sh	200
raw/tag/v0.127.0/configs/felhom-mkfs-guarded.sh	200
raw/tag/v0.128.0/configs/felhom-mkfs-guarded.sh	404

The commit that is 0.128.0 exists and is on main: 28ba8593b8 — v0.128.0 — R-221: the escrow seed is asserted every tick, not remembered once. Only the tag is missing.

This is R-184 happening. R-184 reads: "Nothing prevents the hub from vouching an agent version that was never released. The R-115 gate proves every RELEASED version is installable, but it works from git tags." It was filed as a gap. It is now a live outage.

It is the collision of a good fix with an open gap. R-183 correctly hardened the installer to pin config fetches to the vouched tag (hostinstall_gates.py even asserts the pin exists — ok: fetch_raw pins the agent configs to the vouched agent version). Nothing asserts that the pinned ref resolves. So the first vouch naming an untagged version fails every install closed.

The release script already knows. felhom-agent/scripts/release-agent.sh:23 says in as many words: "raw/tag/v<version>, a released version without a git tag 404s a box mid-install, as root." The hazard was understood, documented, and the tag step was still missed.

Blast radius — stated precisely, not inflated:

• Every fresh install and every reinstall, on every host, fails at step 5/8 for as long as the Day-0 manifest vouches agent 0.128.0. That manifest was approved on 2026-08-09; the approval is what armed it.
• Existing boxes are unaffected. demo-felhom runs 0.128.0 happily — the self-update path takes the binary from the package registry, not from a tag.
• So the damage is confined to the install path. That is also the only path a new customer, or a reinstalling tester, ever walks.

What a customer would do next: nothing. curl: (22) … 404 and a question about a git tag is not a message a household can act on. The visit ends here and a developer is required.

9.2 The box's state, left as found for inspection

--uninstall is not re-run and no manual repair is attempted (rule 1). demo-hp is mid-install:

state.json  completed: [preflight, token, grows, enroll]   mode: byo
agent       0.128.0 installed, unit NOT created (not-found), inactive
user        felhom-agent uid=999 created
pveum       2 users, 3 roles, 63 ACL lines, pool 'felhom' recreated (empty)
guest       none

The documented --resume path exists and was deliberately not used: it cannot help. Resume replays from enroll, hits the same 404, and fails identically — the missing tag is not a transient.

9.3 The one-line unblock, for the operator to decide

cd /mnt/5_hdd/felhom.eu/git/felhom-agent
git tag v0.128.0 28ba8593b8 && git push origin v0.128.0

This publishes the tag for a commit that is already on main and already released as a package — it completes a half-finished release rather than changing any code. The alternative, rolling the manifest back to agent 0.127.0, costs R-221 — the fix STATUS.md calls "the last item blocking a customer from something we promise them" — so it is the worse option.

Not done in this session. Publishing a release tag is an outward-facing act and the runbook is explicit: "If a defect is so severe that the run cannot continue, stop and report; do not fix it here."

9.4 What F-14 still owes

Step 7 never ran, so whether the install would actually have adopted the stale 0.192.0 golden is not yet observed — F-14 rests on the code path plus the preflight's golden (local): …9100…2026_08_03… line and the archive's own felhom-controller:0.192.0 marker. That is strong, but it is not the observation. When the tag is published and the walk resumes, the first thing to record at step 7/8 is which golden it takes.

---

9a. P3 resumed — and the install succeeded

Unblocked on operator instruction ("proceed"). v0.128.0 was annotated at 28ba8593b8 and pushed after an independent download confirmed the published package's sha256 equals the hub's vouched c6eba73b…. Both config URLs then served 200. --resume completed in 3 m 49 s (09:30:30 → 09:34:19 UTC): Day-0 provision SUCCESS, controller 0.210.0, agent 0.128.0, guest 9201 onboot, pool member, all 16 ACL assertions green, controller healthy in ~18 s.

R-274 partly refuted, and recorded as such. Step 7 fetched the vouched 0.210.0 — because --resume skips preflight, and preflight is where local auto-discovery sets GOLDEN_VOLID. So the fresh and resume paths disagree, and the resume path is the safe one. What survives: discovery is sort | tail -1 (newest local) with no manifest comparison, so a box whose newest local golden predates the vouched one still installs stale. Full text in the register.

10. P4 — first contact, as the customer

A clean pass, and worth saying so plainly. Within four minutes of the install finishing, the customer's own URL — fetched from outside the box, over the public internet — served:

A szerver beállítása · Demo HP Add meg az e-mailben kapott beállító kódot, majd válassz saját jelszót a vezérlőpult védelméhez. Beállító kód · Új jelszó (min. 12 karakter) · Új jelszó megerősítése Nem kaptad meg a kódot? Új kód kérése

Unprompted, in Hungarian, naming the customer, with a self-service route if the code never arrived, and nothing anywhere asking for a command line. The hub showed the host ONLINE at agent 0.128.0.

Two findings here, neither fatal: the hub still read "Claimed 18d ago" while the box was serving its first-run page (R-283), and the code that arrives is named three different things across the three surfaces, with the mail pointing at an „Elfelejtett jelszó" page the box does not show (R-282). The reset code was nonetheless accepted on the setup page — 302 and a session — so this is naming, not function.

11. P5 — getting the machine back to work

11.1 The recovery screen, unsought — the headline pass

The first thing on the dashboard after claiming, with nothing sought:

Adatok visszaszerzése — Ezt a gépet újratelepítették. A korábbi, házon kívüli mentéseid megvannak — a sealed package held centrally, sealed at 2026-08-04T11:11:37Z, openable only with the customer's code; nobody can replace it — not Felhom, not support, not the operator; and „Ebben a lépésben semmit nem állítunk vissza és semmi nem változik." Field: Helyreállítási kód (tíz szó).

R-193's screen meeting reality on a genuinely rebuilt box. It answered all three of its questions and its seal date matches host_escrow.created_at exactly.

(Nit: the seal date is rendered raw as 2026-08-04T11:11:37Z to a Hungarian household — ISO-8601 in UTC where a localised date belongs.)

11.2 STOP 3 — the unlock

Entered at the box's own screen, from a file, never on a command line. 21 seconds, and it listed what it found without restoring anything:

app	legutóbbi mentés	méret
calibre-web	2026-08-09 10:30 (CEST)	3.8 MB
felhom-offbox	″	6.8 KB
opengist	″	182.3 KB
privatebin	″	6.8 KB

That is exactly the Gate 0 snapshot set, seen from the customer's side.

11.3 The wall — R-280, and it is rank 1

The restore page diagnoses the situation perfectly and then sends the customer to an empty page:

„Előbb csatold vissza az adatmeghajtót. … Ez két kattintás: Tárhely → Meghajtók, »Meglévő meghajtó csatolása«. Utána gyere vissza ide."

It is not two clicks; it is zero possible clicks. GET /api/disks/candidates → {"initialize":[],"attach":[]}. The agent is fine — GET /api/disks returns the NVMe in full — but handleDiskCandidates builds both lists from the unclaimed-disk scan, and demo-hp's NVMe is deliberately both the user-data drive and the felhom-backup target, so it is claimed and never offered. Correct for initialize; over-broad for attach, which is non-destructive by definition.

It cascades: no store → Calibre-Web's install page degrades to „Nincs regisztrált adattároló — adja meg kézzel az útvonalat"; no app → every restore row reads „Nincs telepítve".

Escape hatch used, and recorded as off-path: POST /settings/storage/add with storage_path=/mnt/sys_drive succeeded first try — an internal path, the very one registered before the wipe, that no household customer could produce. Everything unblocked immediately afterwards and the deploy form became a proper picker („Tárhely (sys_drive) — 64.2 GB szabad").

11.4 App redeploy

calibre-web deployed from the catalogue through the dashboard's own API in 1 m 36 s, running and healthy. The catalogue itself was complete and in Hungarian, every app „Nincs telepítve".

(Also observed: the deploy page warned „A kiválasztott tárhely majdnem megtelt." over an option carrying data-free-percent="93" and „64.2 GB szabad" — an apparent inverted threshold, R-284.)

12. P6 — getting the data back

Three options, each explained, none starting without a question. R-204 item 3 is visibly fixed: option 1 states in as many words that it does not bring the customer's own files back.

step	wall clock
full-restore prepare (3.8 MB)	8 s
full-restore execute	13.2 s (10:27:32.08 → 10:27:45.29 UTC)

„A(z) calibre-web teljes mentése visszaállítva ellenőrző mappába: /mnt/sys_drive/felhom-data/backups/offsite-restore/calibre-web — a saját fájljaiddal együtt. A meglévő adatok változatlanok."

Restored to a verification folder, not into place — honest about it, and R-213's territory.

THE INTEGRITY VERDICT — BYTE-IDENTICAL

expected 4 file(s); found 4
VERDICT: BYTE-IDENTICAL

Compared against GATE0-before-manifest.json, keyed on raw name bytes:

sha256	bytes	name
54b773c46bbfd994…	3 145 728	binary-3mb.bin
52a5c5ebfcac247f…	59	árvíztűrő-tükörfúrógép.txt — name bytes c3a1… identical, NFC preserved
15d2024dfc224162…	25	nested/őszibarack.md — name bytes identical
924497918e55fe6d…	21	plain.txt

Four expected, four restored, zero differences. The accented filenames survived as bytes, not merely as rendered text — the discriminator the Gate 0 positive control was built to enforce.

13. P7 — what the hub thought was happening

It thought nothing. Across the whole reinstall the hub recorded zero events and zero notifications for demo-hp. Positive control (standing rule 3): the hub logged 2 events all day across all customers, newest db_dump_completed at 00:30:07 — the store is reachable and the silence is real.

• The good half: no false alarm fired during a legitimate reinstall, which is what P7 watches for.
• The owed half (R-281): escrow_blob_served exists as the tripwire for exactly this moment — "If no recovery is in progress on that box, investigate" — and has fired for demo-hp before (last 2026-08-04 20:12:54). Today's real unseal fired it not at all. A reinstall and a stolen machine look identical to the operator.

14. Teardown — all four layers

1. The machine — my instruments (fp.py, restored.json, the installer, the passphrase file) removed. Guest 9201 running and healthy. drill-r50 (VM 300) untouched throughout, as predicted.
2. The host — local-lvm 36.97 % → 21.85 % (bare) → 32.35 % (rebuilt). Pre-existing leftovers found and deliberately not removed, recorded instead: c11-scratch, the orphaned vzdump-lxc-9100 archives (now three).
3. The hub — no customer or appliance record created; demo-hp retained deliberately per the runbook. Nothing to delete.
4. The off-site side — 18 snapshots, newest still 9e38b84c / 78b93f04 at 08:30 UTC, i.e. unchanged since Gate 0. The restore was a pure read. No prune, no forget, no delete by me; the only retention that ran was inside the product's own backup call at Gate 0, which reported 18 snapshot(s) itself.

15. The answer to §2's question

Yes — the data comes back, byte for byte. No — not in one sitting, and not without a shell.

The walk completed: P1 ✓ P2 ✓ P3 ✓ P4 ✓ P5 ✓ P6 ✓ P7 ✓, and the verdict is BYTE-IDENTICAL. But it completed only because two hard stops were cleared by someone who could open a terminal and read source code:

1. R-273 — the install died at 5/8 on a git tag that was never pushed. Cleared by a release action.
2. R-280 — the data drive could not be re-attached through any dashboard route, while the page promised „két kattintás". Cleared by POSTing an internal path a customer could not know.

Neither is a data-integrity problem. Both are journey problems, and both stop a household dead. This is the same shape the R-201 walks kept finding: the data half passes, the journey half fails.

What the product did beautifully, and should not be lost in the finding count: the box came up on its own at the vouched version; the setup page appeared unprompted, in Hungarian, naming the customer; the recovery screen appeared without being sought and stated plainly that nothing would be changed by unlocking; the unlock took 21 s; the restore took 13.2 s and said honestly that it had put the files in a verification folder rather than back in place. Sixteen ACL assertions verified themselves. No false alarm fired.

Wall clocks

segment
P1 uninstall	60 s
P2 preflight (2 refusals, then pass)	~6 min
P3 install — first attempt, FAILED	44 s
P3 install — resumed, SUCCESS	3 m 49 s
P4 first contact (box live on its own URL)	within ~4 min of install
STOP 3 unlock	21 s
app redeploy (calibre-web)	1 m 36 s
restore prepare + execute	8 s + 13.2 s
bare machine → verified files	1 h 49 m 22 s (08:38:23 → 10:27:45 UTC)
— of which the product's own work	≈ 7 m 47 s

The 1 h 49 m must not be quoted as the customer number — it is dominated by the R-273 diagnosis and release fix (~38 min) and by two waits on a human. The ≈ 7 m 47 s must not be quoted either: it is what the product costs when someone already knows every answer. The honest figure for an unaided household is undefined, because an unaided household does not finish.