felhom.eu/documentation/audits/REHEARSAL-byo-reinstall-2026-08-09.md

REHEARSAL — the BYO reinstall walk (2026-08-09)

Status: IN PROGRESS. Pre-phase and Gate 0 are complete; STOP 1 passed. The walk (P1–P7) has not started. This file is written before the destructive phase deliberately — a finding that exists only in a session that later crashes is a finding nobody has.

Venue: demo-hp (HP t740, felhom-host, guest 9201, customer demo-hp). Operator-approved at STOP 1. Driven from DooPlex. All times UTC unless marked; the host runs CEST (UTC+2).

---

1. Baselines — re-confirmed live on arrival, not taken from the spec

	spec said	live reading	source
felhom-agent	v0.128.0 @ 28ba8593b8	0.128.0 on demo-felhom, 0.127.0 on demo-hp; HEAD == origin/main == 28ba8593b8	felhom-agent --version on both nodes; git rev-parse
felhom-controller	v0.210.0 @ c732fe1283	0.210.0 demo-felhom, 0.208.0 demo-hp; HEAD == origin/main == c732fe1283	hub /configs; git rev-parse
hub	v0.101.0 @ 56f8aa611c	0.101.0 (deployed image tag matches)	kubectl get deploy hub; page footer
Day-0 manifest	golden 0.210.0 / agent 0.128.0 / min 0.127.0	all three already saved — golden 0.210.0 (b9f701fa…), agent 0.128.0 (c6eba73b…), min agent 0.127.0, wrapper 104db0a4…	hub /configuration, selected <option> values

All three repos were clean and at origin/main before anything ran.

The fleet is not uniform, and the venue is the box that is behind — demo-hp runs agent 0.127.0 and controller 0.208.0 against a manifest vouching 0.128.0/0.210.0. That is an argument for the venue: the reinstall genuinely exercises the manifest rather than re-installing what is already there.

---

2. Venue comparison — the fidelity table that decided it

The runbook's criterion is "the box whose current state most resembles what the tester will have".

criterion	demo-felhom (N100)	demo-hp (t740)
customer apps deployed	1 — opengist	3 — privatebin, opengist, calibre-web
enrolled user-data drive	none registered. /mnt/felhom-drives empty; controller logs [WARN] Storage paths: no storage paths registered	/mnt/felhom-drives/Felhom-Share + NVMe enrolled 2026-07-22; one registered storage path /mnt/sys_drive
off-site tier has actually run	NO — offsite.state=needs_credential, no run has ever succeeded	YES — 18 restic snapshots, daily, unbroken
escrow / recovery ceremony	present, generation 2	present, generation 2
install shape	mkimage loader, Secure Boot OFF (firmware workaround)	shim loader, Secure Boot ENABLED — the customer shape
collateral on the box	none	drill-r50 (VM 300); leftover c11-scratch storage

Chosen: demo-hp. It wins every fidelity criterion that distinguishes them.

drill-r50 is not at risk, and this was checked rather than assumed: VM 300 is not in the felhom pool (pvesh get /pools/felhom → members [9201]), its disks are on local-lvm, and --uninstall removes no storage definitions and no non-pool guest — its own end-of-teardown statement says the enrolled drives are "unmounted only, NEVER wiped" and that it never deletes PBS backups, hub records or escrow blobs.

Correction recorded, because it was part of the comparison put in front of the operator: the enrolled-drive row was first reported as "real files on an enrolled drive". Felhom-Share is empty — the share exists and is enrolled, but holds no files. demo-hp still wins the row (demo-hp has a registered storage path; demo-felhom has none), on a narrower margin than first stated.

---

3. Pre-phase — R-268, the leaked per-guest local-API token

Done first and independently, on demo-felhom guest 9201, as the runbook required. No token value reached stdout at any point: old value captured file→file from bootstrap.json, new value minted into a file, both shredded at the end.

3.1 The rotation, and its proof

step	old token	new token
before rotation	HTTP 200	—
bogus token (negative control)	HTTP 401	—
after the on-disk rotation, before any reload	HTTP 200 ← see 3.2	200
after that lookup forced a reload	HTTP 401	—
after systemctl restart felhom-agent	HTTP 401	HTTP 200

Positive observable for the controller half: local-api: channel up (agent 169.254.253.1:8443) — guest 9201, 3 mount(s) visible, enumerating mp8/mp9/mp0 — data only the agent can supply. An absent error would not have been evidence.

R-268 is CLOSED. The old token is refused; the new one works; the controller's channel is up.

3.2 FINDING — an out-of-process token rotation does not revoke the old token

localapi.TokenStore.Mint's doc comment states "last-write wins — any previous token for this guest is revoked". Across processes that is false until something else forces a reload. The long-lived agent serves Lookup from an in-memory index and re-reads the store only on a miss (the B3 reload-on-miss optimisation). A superseded token is still a direct map hit, so it authorises.

Red-proved twice. A temporary unit probe (Scenario C with the lookup order reversed — present the rotated-out token first) fails on the claim; the shipped TestTokenStore_ReloadOnMiss_RemintCoherence passes only because it looks up the NEW token first, which is what triggers the reload that then evicts the old hash. Then proved on real hardware: the row marked ← above is a live 200 from a token that the store on disk had already superseded.

This is the CLAUDE.md case exactly — a comment asserting an invariant with a test that appears to pin it and does not. The honest fix is either a test that pins the reversed order, or an eviction that does not depend on an unrelated lookup missing first.

3.3 FINDING — R-268's own stated rotation recipe is incomplete

The register row says the new plaintext "must also be written into the guest's /etc/felhom-bootstrap/bootstrap.json or the in-guest controller loses its agent access". Writing bootstrap.json is not sufficient. bootstrap.ensureLocalAPI returns early when cfg.LocalAPI.Endpoint != "" — it fills an absent block and never refreshes a present one — so after a rotation the controller keeps serving the old token from its own controller.yaml across a restart. Proved live: two controller restarts, still HTTP 401. The rotation only completed once local_api.token was written into controller.yaml itself.

The neighbouring DetectEndpointDrift compares the endpoint and deliberately does not compare the token ("a token mismatch is a different failure"). So this shape is known to be unmodelled, and R-78 ("which file is authoritative") is its parent.

3.4 FINDING — the agent-channel alarm never closes, and its own prescribed remedy is why

channelhealth alerted correctly and promptly: agent_channel_unauthorized, no debounce, pushed to the hub — "agent rejected the controller token (HTTP 401) — token stale/rotated (re-bootstrap)".

The channel then recovered, and no recovery event was ever sent. In Checker.Check's UP branch the notify is guarded by prev != "" && prev != "up", and a controller restart resets state to "". The alert's own instruction is to re-bootstrap — i.e. restart the controller — so following the alarm's advice structurally guarantees the recovery is silent. The down side is deliberately asymmetric (F2: a born-down channel alerts on cycle 1); the up side is not.

Operator-visible result on the hub: two agent_channel_unauthorized errors, one sent and one suppressed by cooldown, and nothing afterwards. The customer dashboard is fine — SetDashboard reflects current state every cycle. It is the operator's trail that ends on "down".

---

4. Gate 0

4.1 The recovery code — CONFIRMED IN THE OPERATOR'S HANDS (STOP 1)

Not printed, not logged, not requested. It will be supplied at the screen at STOP 3.

4.2 The dataset — planted and fingerprinted

Planted at /var/lib/felhom/sys_drive/felhom-data/userdata/media/books/rehearsal-2026-08-09/ — inside the Calibre library, which is real app-owned customer data and is demonstrably carried off-site (the path appears in every felhom-offbox,calibre-web snapshot). Chosen after checking coverage rather than assuming it: the per-app tier-1 unit carries only the named docker volume, so a file dropped in an arbitrary place would not have travelled.

sha256 (16)	bytes	name	name bytes (hex)
54b773c46bbfd994…	3 145 728	binary-3mb.bin	62696e6172792d336d622e62696e
52a5c5ebfcac247f…	59	árvíztűrő-tükörfúrógép.txt	c3a17276c3ad7a74c5b172c5912d74c3bc6bc3b67266c3ba72c3b367c3a9702e747874
15d2024dfc224162…	25	nested/őszibarack.md	6e65737465642fc591737a6962617261636b2e6d64
924497918e55fe6d…	21	plain.txt	706c61696e2e747874

Manifest: documentation/audits/evidence/ → recorded in the session scratchpad as GATE0-before-manifest.json. The manifest keys on raw name bytes, not rendered text.

4.3 The positive control — the comparator was watched failing, three ways

control	expected	result
identical copy	PASS	PASS, exit 0
one bit flipped at byte 1 500 000 of the 3 MB binary	FAIL	FAIL — named the file and both sha256s
NFC→NFD rename of the accented name (renders identically)	FAIL	FAIL — caught as MISSING+EXTRA on the name bytes
one file deleted	FAIL	FAIL — named the missing file

The NFD case is the one that matters: a comparison of rendered filenames would have passed it. Every altered copy was discarded after the control ran.

4.4 The off-site tier — has run since the dataset was planted

Triggered through the product's own dashboard button (POST /backup/offbox/run → flash „A távoli mentés elindult"), not by invoking restic by hand.

[offbox] backup run started (3 app(s) toggled)
[offbox] pre-push dump leg completed in 7.329s — snapshot pair is coherent
[offbox] backed up calibre-web (…/backups/primary/calibre-web, 1 mandatory path(s))
[offbox] backup OK: 3 app(s) backed up, 18 snapshot(s), 1m22s

The restore point, by identity — never by size:

snapshot	time (UTC)	tag	paths
41c830db	2026-08-09 08:30:38	felhom-offbox,calibre-web	…/backups/primary/calibre-web + …/userdata/media/books
9e38b84c	2026-08-09 08:30:49	felhom-offbox,opengist	…/backups/primary/opengist
78b93f04	2026-08-09 08:30:53	felhom-offbox,privatebin	…/backups/primary/privatebin

restic ls 41c830db confirms all four planted files are present, both accented filenames included.

4.5 The box, captured

GATE0-demo-hp-before.txt (282 lines): agent version, pveversion, pct/qm lists, pool membership, pvesm status, storage.cfg, df, lsblk with serials, the agent unit + service user, sudoers, the pveum users/tokens/roles/ACL, the guest hook, the WireGuard tunnel, the OOB belt, the agent state dir, the install state file, PBS snapshots, local vzdump archives, and the enrolled-drive tree.

---

5. Findings so far, ranked by what they cost the person in front of you

Nothing yet is rank 1 or 2 — the walk has not started.

Rank 3 — misleads

• F-1 · A rotated-out local-API token still authorises (§3.2). A customer would do nothing — they cannot see it. An operator rotating a leaked token would believe it revoked when it is not. The shipped test passes for the wrong reason.
• F-2 · R-268's rotation recipe is incomplete (§3.3). Whoever follows it gets a box whose agent channel is dead across restarts, with a correct-sounding alarm and no hint that controller.yaml is the file that matters.
• F-3 · The agent-channel alarm never closes (§3.4). The operator fixes it as instructed and is never told it worked; the hub's last word stays "down".
• F-4 · Three hub surfaces jointly misrepresent a healthy off-site tier as absent. The customer page reads Snapshots 0 / Repo Size 0 MB / Integrity Unknown (that panel renders the local tier, while the healthy offsite object — snapshot_count: 18, last_status: ok — sits unrendered in the same report); the Offsite page shows 0.0 GB (true, but a 162 KB repo rounds to nothing); and a stale offsite_delivery_stuck event from 2026-08-07 reads as current. Three independent surfaces agreeing on a wrong picture is how a working backup gets "fixed". This one caught me — it produced a wrong statement to the operator at STOP 1, corrected before it changed anything.
• F-5 · There is no operator-triggerable off-site backup. The only route is the customer's dashboard button; signed_jobs needs a signing key the hub does not hold. Sibling of R-177.

Rank 4 — cosmetic / hygiene

• F-6 · Leave-behinds from earlier sessions, found while capturing state:
  • demo-hp host: storage c11-scratch (Campaign 11), and felhom-backup:…vzdump-lxc-9100-2026_08_03… for a guest that no longer exists;
  • demo-hp guest 9201 /root: .dpw (13 bytes), .h, .sec.html, all 2026-08-07 16:08;
  • demo-felhom guest 9201: /root/r204-backup/offbox.orig.
• F-7 · The managed-update floor is 0.200.0, not 0.156.0 — the runbook's §8.3 premise is stale. Confirmed twice: hub /configuration reads "Effective floor v0.200.0 — source: DB (hub_settings); env fallback would be v0.120.0", and both boxes log settle-gate: GO — at/above floor 0.200.0. The ruling's substance still needs recording; its number must not be written down as 0.156.0.

Not a finding, checked and cleared

• The two userdata trees under sys_drive are the same directory surfaced twice, not a backed-up copy beside an unprotected one. Checked against calibre-web's actual bind mounts.

---

6. Steps taken off-path, and what they cost

Rule 2 (no shell after the install command returns) does not bind yet — the walk has not begun. These are Gate 0 preparation acts by the operator, recorded because they are mutations:

1. R-268 rotation on demo-felhom — token store appended, bootstrap.json and controller.yaml rewritten, agent and controller restarted. Required by the runbook's pre-phase. Not the venue.
2. demo-hp's dashboard password was re-set to the value in the operator's credentials file. The customer-owned password was not known to this session, and no operator-side route to the off-site button exists (F-5). The prior password_hash is preserved at /root/.orighash and /root/settings.json.rehearsal-orig in guest 9201; claimed was left true. Done on operator instruction. Cost to fidelity: none for the walk — P1 destroys this guest, and P4 re-claims with a password the operator chooses.
3. The off-site run was started by a script pressing the dashboard's own endpoint, with a real session and a real CSRF token, rather than by a person clicking. The server path is identical; only the click is synthetic.

---

7. What happens next

P1 — remove Felhom from demo-hp with --uninstall, then measure what it leaves behind. Nothing in §1–§4 is repeated after that point; the "before" is fixed as of this file.