# REPORT — R-201: **THE DRILL PASSED.** A customer's file survived a machine rebuild

**2026-08-04, 21:30–23:15** · `demo-hp` deliberately rebuilt · no code, no version bump ·
`demo-felhom` untouched. Record: `documentation/audits/DRILL-r201-night-run-2026-08-04.md`.

---

## 1. THE VERDICT — PASS

```
pre-wipe   sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
restored   sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
```

**Byte-identical.** The controller's data volume was destroyed and the file deleted from disk; the
customer's recovery code, the customer's restore flow, and the file came back. **The Felhom backup
story is proved end to end for the first time.**

## 2. Snapshot count at step 9 — **3, not 1**

`repo_state: null` (the repository **opened**, not orphaned), `snapshot_count: 3`, `repo_size_bytes:
42 026` — the pre-wipe size exactly. The pre-wipe snapshot `e6132ae5` was still present with the file
inside it. **No fresh history was started over the old one** — the failure this drill was designed to
catch did not occur.

## 3. §5's five conditions, recorded before the wipe

1. sentinel **listed by name** in snapshot `e6132ae5` (19:36:26), 181 B.
2. rollback archive `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst` (1 606 765 083 B) **verified by a
   full zstd stream read** (4 867 573 760 B) with the sentinel confirmed inside.
3. §3's option — §4 below.
4. `demo-felhom` `health=ok`, `escrow_state=escrowed`, untouched throughout.
5. nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 %.

**One precondition had drifted and was repaired, not worked around:** the staged snapshot had lost the
sentinel because the afternoon's experiment produced a *later* same-day snapshot and `forget
--keep-daily 7 --group-by host,tags` pruned the good one. One backup re-established it, re-verified by
listing. **A good snapshot is not durable against a later bad run on the same day.**

## 4. §3 — the recovery code

**Option B as already in place, improved: no new copy was made, so nothing needed shredding.** The
operator's own `~/.config/credentials` (`0600`) was the source; the code was **piped to stdin** for
each invocation — never an argument, never exported, never a second file, never logged. Destroying the
operator's own store would have destroyed their record; because no extra copy existed, there is nothing
left to prove gone.

**Verified anyway, with the planted-copy positive control:** 0 hits in the agent journal, 0 in the
controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover
`felhom-idesc-*` dirs — and the same sweep found a planted copy (**1**), then **0** after shredding it.

## 5. Every step's observable

| step | observable |
|---|---|
| 6 | fresh data dir `20:00:2x`; new `encryption.key`; `claimed = None`; `offbox = null` |
| **7** | **`identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`, `updated_at` still `11:11:37` — UNCHANGED.** Nothing re-escrowed itself |
| **8** | **recovered `8a9e33aa4da6…`** — matches the pre-wipe on-disk key and the hub's record |
| 9a | `[INSTALLED] … reads back identical` |
| 9b | after Re-issue + apply, the on-disk key is **still** the recovered one |
| 9c | **repository OPENED** — `repo_state: null`, 3 snapshots, 42 026 B |
| **10** | **restored sha256 byte-identical** |
| 11 | this record |

Superseded escrow rows remained **2** throughout — **no ceremony was run at any point.**

## 6. Part 2 — not run

Gate: *"the drill PASSED; its evidence is written down; and there is time."* The first two are met; the
third is not — 23:15, after a full destructive cycle, and Part 2 is a second wipe. Deliberately left
for a fresh session with drill 1's result already recorded, so a second wipe cannot overwrite it.
**R-198's retention remains unit-proven only** — nothing has yet superseded a key in production, and it
is now the last unproven link in this chain.

## 7. Teardown — three layers

| layer | state |
|---|---|
| the guest | **healthy and re-armed**: controller v0.197.0 claimed, six app containers serving, all three apps re-toggled for off-site, the sentinel restored to its live location and verified. The verification-copy tree is **kept as evidence**. Scratch band empty |
| the host | vzdump snapshot LVs released cleanly; one new 1.6 GB archive on nvme-1tb (883 GB free); nothing deleted |
| the hub | **no new customer records** — `demo-hp` is the same row throughout (a controller-data rebuild, not a re-enrolment). One Re-issue staged a one-time secret (consumed 20:16) and set `stale_at`. **The two superseded escrow rows are unchanged** |

**Nothing deleted on the storage endpoint**, including the ~1.2 GB of orphaned ciphertext — ruled,
still owed, deliberately not ridden along with a drill.

## 8. The capability-map row

Now reads *"A customer's file survives a machine rebuild and comes back — the whole off-site story, end
to end"*, **PROVEN-LIVE**. What it deliberately does **not** claim: the journey is manual and
undocumented (R-204); the scope is `demo-hp` and a **controller-data** rebuild, **not** a total host
loss and **not** a guest reprovision; and R-198's retention is still unit-proven.

## 9. New findings — R-204, expanded into the gap list

The four steps between a recovered key and a restored file, all measured while walking them:

1. **R-193** — a rebuilt controller cannot configure its off-site tier (`no unconsumed offsite
   password`; the ledger shows its predecessor consumed it at 07:12:06). Needs an operator Re-issue.
2. **The claim gate** — a rebuilt box is unclaimed, so *every* controller endpoint is intercepted. And
   **the local escape hatch does not work unaided**: `--print-reset-code` writes the new hash to
   `settings.json` while the running controller keeps its old copy in memory, so `effectiveClaimCode()`
   never sees it. **Restart the controller between minting and claiming** — two claim attempts failed
   before this was diagnosed.
3. **R-196** — the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged, which
   gates every off-site run. Cleared with the **manual** confirm, never a ceremony.
4. **`mode=unit` is the restore default** and returns the recovery unit, **not** the userdata leg. A
   customer told to "restore from off-site" gets their app definition and not their documents, and
   nothing in the outcome says so. **The worst of the four**, because it fails silently at the last
   step.

## 10. CI

Docs push only; run number and task id in the session summary. **`--no-verify` not used.**

## 11. Observations — noticed, NOT acted on

1. **A same-day re-run replaces the day's snapshot** (`forget --keep-daily 7 --group-by host,tags`).
   Any drill depending on a specific snapshot surviving must account for it.
2. **The hub's ClusterIP is not reachable from DooPlex's host network** — the operator UI needs a
   `kubectl port-forward`; the memory's `curl -u :$HUB_PW` recipe omits that.
3. **`--print-reset-code` mints a new generation on every invocation** — three wasted generations here
   (4, 5, 6) while diagnosing the in-memory staleness.
4. **The restore wrote into `backups/offsite-restore/<app>/` mirroring the full absolute path** — deep
   but unambiguous; worth knowing before writing customer-facing copy about where files land.