felhom.eu/REPORT.md

REPORT — R-201: THE DRILL PASSED. A customer's file survived a machine rebuild

2026-08-04, 21:30–23:15 · demo-hp deliberately rebuilt · no code, no version bump · demo-felhom untouched. Record: documentation/audits/DRILL-r201-night-run-2026-08-04.md.

---

1. THE VERDICT — PASS

pre-wipe   sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
restored   sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c

Byte-identical. The controller's data volume was destroyed and the file deleted from disk; the customer's recovery code, the customer's restore flow, and the file came back. The Felhom backup story is proved end to end for the first time.

2. Snapshot count at step 9 — 3, not 1

repo_state: null (the repository opened, not orphaned), snapshot_count: 3, repo_size_bytes: 42 026 — the pre-wipe size exactly. The pre-wipe snapshot e6132ae5 was still present with the file inside it. No fresh history was started over the old one — the failure this drill was designed to catch did not occur.

3. §5's five conditions, recorded before the wipe

1. sentinel listed by name in snapshot e6132ae5 (19:36:26), 181 B.
2. rollback archive vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst (1 606 765 083 B) verified by a full zstd stream read (4 867 573 760 B) with the sentinel confirmed inside.
3. §3's option — §4 below.
4. demo-felhom health=ok, escrow_state=escrowed, untouched throughout.
5. nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 %.

One precondition had drifted and was repaired, not worked around: the staged snapshot had lost the sentinel because the afternoon's experiment produced a later same-day snapshot and forget --keep-daily 7 --group-by host,tags pruned the good one. One backup re-established it, re-verified by listing. A good snapshot is not durable against a later bad run on the same day.

4. §3 — the recovery code

Option B as already in place, improved: no new copy was made, so nothing needed shredding. The operator's own ~/.config/credentials (0600) was the source; the code was piped to stdin for each invocation — never an argument, never exported, never a second file, never logged. Destroying the operator's own store would have destroyed their record; because no extra copy existed, there is nothing left to prove gone.

Verified anyway, with the planted-copy positive control: 0 hits in the agent journal, 0 in the controller log, 0 files under /tmp, /var/tmp, /var/lib/felhom-agent, /root, 0 leftover felhom-idesc-* dirs — and the same sweep found a planted copy (1), then 0 after shredding it.

5. Every step's observable

step	observable
6	fresh data dir 20:00:2x; new encryption.key; claimed = None; offbox = null
7	identity_blob 572 B, restic_pw_sha256 8a9e33aa4da6…, updated_at still 11:11:37 — UNCHANGED. Nothing re-escrowed itself
8	recovered 8a9e33aa4da6… — matches the pre-wipe on-disk key and the hub's record
9a	[INSTALLED] … reads back identical
9b	after Re-issue + apply, the on-disk key is still the recovered one
9c	repository OPENED — repo_state: null, 3 snapshots, 42 026 B
10	restored sha256 byte-identical
11	this record

Superseded escrow rows remained 2 throughout — no ceremony was run at any point.

6. Part 2 — not run

Gate: "the drill PASSED; its evidence is written down; and there is time." The first two are met; the third is not — 23:15, after a full destructive cycle, and Part 2 is a second wipe. Deliberately left for a fresh session with drill 1's result already recorded, so a second wipe cannot overwrite it. R-198's retention remains unit-proven only — nothing has yet superseded a key in production, and it is now the last unproven link in this chain.

7. Teardown — three layers

layer	state
the guest	healthy and re-armed: controller v0.197.0 claimed, six app containers serving, all three apps re-toggled for off-site, the sentinel restored to its live location and verified. The verification-copy tree is kept as evidence. Scratch band empty
the host	vzdump snapshot LVs released cleanly; one new 1.6 GB archive on nvme-1tb (883 GB free); nothing deleted
the hub	no new customer records — demo-hp is the same row throughout (a controller-data rebuild, not a re-enrolment). One Re-issue staged a one-time secret (consumed 20:16) and set stale_at. The two superseded escrow rows are unchanged

Nothing deleted on the storage endpoint, including the ~1.2 GB of orphaned ciphertext — ruled, still owed, deliberately not ridden along with a drill.

8. The capability-map row

Now reads "A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end", PROVEN-LIVE. What it deliberately does not claim: the journey is manual and undocumented (R-204); the scope is demo-hp and a controller-data rebuild, not a total host loss and not a guest reprovision; and R-198's retention is still unit-proven.

9. New findings — R-204, expanded into the gap list

The four steps between a recovered key and a restored file, all measured while walking them:

1. R-193 — a rebuilt controller cannot configure its off-site tier (no unconsumed offsite password; the ledger shows its predecessor consumed it at 07:12:06). Needs an operator Re-issue.
2. The claim gate — a rebuilt box is unclaimed, so every controller endpoint is intercepted. And the local escape hatch does not work unaided: --print-reset-code writes the new hash to settings.json while the running controller keeps its old copy in memory, so effectiveClaimCode() never sees it. Restart the controller between minting and claiming — two claim attempts failed before this was diagnosed.
3. R-196 — the Re-issue sets stale_at (20:15:49) while restic_pw_sha256 is unchanged, which gates every off-site run. Cleared with the manual confirm, never a ceremony.
4. mode=unit is the restore default and returns the recovery unit, not the userdata leg. A customer told to "restore from off-site" gets their app definition and not their documents, and nothing in the outcome says so. The worst of the four, because it fails silently at the last step.

10. CI

Docs push only; run number and task id in the session summary. --no-verify not used.

11. Observations — noticed, NOT acted on

1. A same-day re-run replaces the day's snapshot (forget --keep-daily 7 --group-by host,tags). Any drill depending on a specific snapshot surviving must account for it.
2. The hub's ClusterIP is not reachable from DooPlex's host network — the operator UI needs a kubectl port-forward; the memory's curl -u :$HUB_PW recipe omits that.
3. --print-reset-code mints a new generation on every invocation — three wasted generations here (4, 5, 6) while diagnosing the in-memory staleness.
4. The restore wrote into backups/offsite-restore/<app>/ mirroring the full absolute path — deep but unambiguous; worth knowing before writing customer-facing copy about where files land.