felhom.eu/REPORT.md

REPORT — R-201 night run: the key came back; the verdict did not

2026-08-04, 21:30–22:40, unattended. demo-hp was deliberately rebuilt. No code, no version bump. demo-felhom untouched. Full record: documentation/audits/DRILL-r201-night-run-2026-08-04.md.

---

1. THE VERDICT — not reached

sentinel sha256, pre-wipe	643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
sentinel sha256, post-restore	not obtained — the restore was never reachable

Not a FAIL. Nothing came back wrong and no fresh history was started; the box never got as far as running a backup. What it is instead: the first proof that the key survives and returns, plus the measured reason a customer still cannot use it.

After a real rebuild — controller data volume destroyed, sentinel deleted from disk — the customer's recovery code produced 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a, byte-identical to the pre-wipe on-disk key and to the hub's independent record. That has never been shown before.

2. Snapshot count at step 9 — not obtained

The off-site run was never permitted to start (§4). And a count would have been a poor discriminator anyway: restic's same-day forget --keep-daily 7 --group-by host,tags keeps one snapshot per tag per day, so a successful reopen would have shown 3, not 4. The real observable is whether the pre-wipe snapshot e6132ae5 survives with the sentinel in it — recorded as the resume step.

3. §5's five conditions, recorded before the wipe

1. sentinel listed by name — snapshot e6132ae5 (19:36:26), -rw-r--r-- 1000 1000 181 …/DRILL-SENTINEL.txt.
2. rollback archive verified — vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst, 1 606 765 083 B, full zstd stream read OK (4 867 573 760 B), sentinel confirmed inside it.
3. §3's option — §5 below.
4. demo-felhom health=ok, escrow_state=escrowed, untouched throughout.
5. space — nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 %.

One precondition had drifted and was repaired, not worked around. The staged snapshot no longer held the sentinel: the afternoon's mandatory data path missing experiment produced a later same-day calibre-web snapshot and forget had pruned the good one. The fixture on disk was correct, so one backup re-established it and it was re-verified by listing. Lesson: a good snapshot is not durable against a later bad run on the same day.

4. Every step's observable

step	observable
6	fresh data dir stamped 20:00:2x; new encryption.key (32 B); claimed = None; offbox = null
7	identity_blob 572 B, restic_pw_sha256 8a9e33aa4da6…, updated_at still 11:11:37 — UNCHANGED across the wipe. Nothing re-escrowed itself
8	recovered 8a9e33aa4da6… — matches pre-wipe on-disk and the hub's record. Exit 1 is correct and designed: nothing local to compare against, the rebuilt-box shape
9a	[INSTALLED] … reads back identical — the "installed" branch's first real run
9b	after the Re-issue and apply, the on-disk key is still 8a9e33aa4da6… — WriteOffboxSecrets kept it
9c	blocked — see below
10–11	not run

The wipe was faithful to the incident, deliberately. The 2026-08-03 rebuild R-193 is filed against was not a guest reprovision — the journal shows guest 9201 running continuously through that window with no pct destroy, no pct restore and no --selftest=provision. What changed was the controller and its data volume. A guest destroy plus an unrehearsed provisioning chain, improvised unattended, is what §8.10 exists to prevent.

5. §3 — the recovery code

Option B as already in place, improved: no new copy was created, so nothing needed shredding. The operator had placed R_DEMO-HP in their own ~/.config/credentials (0600) two sessions ago for this purpose. It was read from there and piped to stdin for the two invocations that needed it — never an argument, never exported, never written to a second file, never logged. Destroying the operator's own store would have destroyed their record; because no additional copy existed, there is nothing left to prove gone.

Verified anyway, with the planted-copy positive control: 0 hits in the agent journal, 0 in the controller log, 0 files under /tmp, /var/tmp, /var/lib/felhom-agent, /root, 0 leftover felhom-idesc-* staging dirs — and the same sweep found a planted copy (1), then 0 after shredding it. The instrument is shown sensitive rather than assumed to be.

6. Part 2 — did not run

Its gate is "the drill PASSED". It did not. A second wipe would have destroyed the state that makes the first drill finishable in five minutes. R-198's retention therefore remains unit-proven only — nothing has yet superseded a key in production.

7. Teardown — three layers

layer	state
the guest	nothing torn down — the wipe is the evidence; all six app containers serving; the recovered key on disk; scratch band empty
the host	vzdump snapshot LVs released cleanly; one new 1.6 GB archive on nvme-1tb (883 GB free); nothing deleted
the hub	no new customer records — demo-hp is the same row throughout, because the rebuild was a controller-data wipe and not a re-enrolment. One Re-issue staged a one-time secret (consumed 20:16) and set stale_at. The two superseded escrow rows are unchanged — no ceremony ran

Nothing deleted on the storage endpoint, including the ~1.2 GB of orphaned ciphertext — ruled, still owed, deliberately not ridden along with a drill.

8. The capability-map rows

A new row records the key as PROVEN-LIVE after a real rebuild, and states plainly what it does not claim: no file has been restored, and the four links of R-204 stand between the recovered key and a usable repository. The R-199 back-pointer added earlier today stands.

9. New findings

• R-204 (NEW) — the recovered key cannot be used: a rebuilt box cannot configure its tier (R-193); the Re-issue that fixes that marks the escrow stale though the key never changed (R-196, measured at 20:15:49 with restic_pw_sha256 unchanged); a stale escrow gates every run; and the only way to clear it is a ceremony that destroys the recovered key. Plus a fourth link in no design document: a rebuilt box is unclaimed, so the claim gate intercepts every controller endpoint.
• R-196 re-scoped — no longer a documentation nit; it is on the critical path for recovery.
• R-201 — the wipe happened, the key came back, ~5 minutes from a verdict with a person present.
• R-198 — still unit-proven only; Part 2 gated out.
• R-202 open; the ciphertext deletion still owed.

10. CI

Docs push only. Run number and task id in the session summary. --no-verify not used.

11. Observations — noticed, NOT acted on

1. The claim gate is an undocumented first step of every recovery. Before a customer can do anything on a rebuilt box — including recovering their backups — they must re-claim it.
2. --print-reset-code output needs parsing care: the captured value was 73 characters, i.e. more than the code itself. Not chased; the claim was abandoned when the session stopped.
3. A same-day re-run replaces the day's snapshot. Worth knowing before designing any drill that depends on a specific snapshot surviving.
4. The hub's ClusterIP is not reachable from DooPlex's host network — the operator UI needs a kubectl port-forward. The curl -u :$HUB_PW recipe in memory omits that.