felhom.eu/REPORT.md

# REPORT — R-201 night run: the key came back; the verdict did not

**2026-08-04, 21:30–22:40, unattended.** `demo-hp` was deliberately rebuilt. No code, no version bump.
`demo-felhom` untouched. Full record: `documentation/audits/DRILL-r201-night-run-2026-08-04.md`.

---

## 1. THE VERDICT — not reached

| | |
|---|---|
| sentinel sha256, pre-wipe | `643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` |
| sentinel sha256, post-restore | **not obtained** — the restore was never reachable |

**Not a FAIL.** Nothing came back wrong and no fresh history was started; the box never got as far as
running a backup. **What it is instead:** the first proof that the key survives and returns, plus the
measured reason a customer still cannot use it.

> **After a real rebuild — controller data volume destroyed, sentinel deleted from disk — the
> customer's recovery code produced `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`,
> byte-identical to the pre-wipe on-disk key and to the hub's independent record.** That has never been
> shown before.

## 2. Snapshot count at step 9 — not obtained

The off-site run was never permitted to start (§4). **And a count would have been a poor
discriminator anyway:** restic's same-day `forget --keep-daily 7 --group-by host,tags` keeps one
snapshot per tag per day, so a successful reopen would have shown 3, not 4. The real observable is
whether the pre-wipe snapshot `e6132ae5` survives with the sentinel in it — recorded as the resume
step.

## 3. §5's five conditions, recorded before the wipe

1. **sentinel listed by name** — snapshot `e6132ae5` (19:36:26), `-rw-r--r-- 1000 1000 181 …/DRILL-SENTINEL.txt`.
2. **rollback archive verified** — `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst`, 1 606 765 083 B,
   **full zstd stream read OK** (4 867 573 760 B), sentinel confirmed inside it.
3. §3's option — §5 below.
4. **demo-felhom** `health=ok`, `escrow_state=escrowed`, untouched throughout.
5. **space** — nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 %.

**One precondition had drifted and was repaired, not worked around.** The staged snapshot no longer
held the sentinel: the afternoon's `mandatory data path missing` experiment produced a *later* same-day
calibre-web snapshot and `forget` had pruned the good one. The fixture on disk was correct, so one
backup re-established it and it was re-verified by listing. **Lesson: a good snapshot is not durable
against a later bad run on the same day.**

## 4. Every step's observable

| step | observable |
|---|---|
| 6 | fresh data dir stamped `20:00:2x`; **new `encryption.key`** (32 B); `claimed = None`; `offbox = null` |
| **7** | **`identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`, `updated_at` still `11:11:37` — UNCHANGED across the wipe.** Nothing re-escrowed itself |
| **8** | **recovered `8a9e33aa4da6…`** — matches pre-wipe on-disk and the hub's record. Exit 1 is correct and designed: nothing local to compare against, the rebuilt-box shape |
| 9a | `[INSTALLED] … reads back identical` — the "installed" branch's first real run |
| 9b | after the Re-issue and apply, the on-disk key is **still** `8a9e33aa4da6…` — `WriteOffboxSecrets` kept it |
| 9c | **blocked** — see below |
| 10–11 | **not run** |

**The wipe was faithful to the incident, deliberately.** The 2026-08-03 rebuild R-193 is filed against
was **not** a guest reprovision — the journal shows guest 9201 running continuously through that window
with no `pct destroy`, no `pct restore` and no `--selftest=provision`. What changed was the controller
and its data volume. A guest destroy plus an unrehearsed provisioning chain, improvised unattended, is
what §8.10 exists to prevent.

## 5. §3 — the recovery code

**Option B as already in place, improved: no new copy was created, so nothing needed shredding.** The
operator had placed `R_DEMO-HP` in their own `~/.config/credentials` (`0600`) two sessions ago for this
purpose. It was read from there and **piped to stdin** for the two invocations that needed it — never
an argument, never exported, never written to a second file, never logged. Destroying the operator's
own store would have destroyed their record; because no additional copy existed, there is nothing left
to prove gone.

**Verified anyway, with the planted-copy positive control:** 0 hits in the agent journal, 0 in the
controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover
`felhom-idesc-*` staging dirs — and the same sweep found a planted copy (**1**), then **0** after
shredding it. The instrument is shown sensitive rather than assumed to be.

## 6. Part 2 — did not run

Its gate is "the drill PASSED". It did not. A second wipe would have destroyed the state that makes the
first drill finishable in five minutes. **R-198's retention therefore remains unit-proven only** —
nothing has yet superseded a key in production.

## 7. Teardown — three layers

| layer | state |
|---|---|
| the guest | **nothing torn down** — the wipe is the evidence; all six app containers serving; the recovered key on disk; scratch band empty |
| the host | vzdump snapshot LVs released cleanly; one new 1.6 GB archive on nvme-1tb (883 GB free); nothing deleted |
| the hub | **no new customer records** — `demo-hp` is the same row throughout, because the rebuild was a controller-data wipe and not a re-enrolment. One Re-issue staged a one-time secret (consumed 20:16) and set `stale_at`. The two superseded escrow rows are **unchanged** — no ceremony ran |

**Nothing deleted on the storage endpoint**, including the ~1.2 GB of orphaned ciphertext — ruled,
still owed, deliberately not ridden along with a drill.

## 8. The capability-map rows

A new row records the key as **PROVEN-LIVE after a real rebuild**, and states plainly what it does not
claim: **no file has been restored**, and the four links of R-204 stand between the recovered key and a
usable repository. The R-199 back-pointer added earlier today stands.

## 9. New findings

- **R-204 (NEW)** — the recovered key cannot be used: a rebuilt box cannot configure its tier (R-193);
  the Re-issue that fixes that marks the escrow stale though the key never changed (R-196, **measured
  at `20:15:49` with `restic_pw_sha256` unchanged**); a stale escrow gates every run; and the only way
  to clear it is a ceremony that **destroys the recovered key**. Plus a fourth link in no design
  document: a rebuilt box is **unclaimed**, so the claim gate intercepts every controller endpoint.
- **R-196 re-scoped** — no longer a documentation nit; it is on the critical path for recovery.
- **R-201** — the wipe happened, the key came back, ~5 minutes from a verdict with a person present.
- **R-198** — still unit-proven only; Part 2 gated out.
- **R-202 open; the ciphertext deletion still owed.**

## 10. CI

Docs push only. Run number and task id in the session summary. **`--no-verify` not used.**

## 11. Observations — noticed, NOT acted on

1. **The claim gate is an undocumented first step of every recovery.** Before a customer can do
   anything on a rebuilt box — including recovering their backups — they must re-claim it.
2. **`--print-reset-code` output needs parsing care**: the captured value was 73 characters, i.e. more
   than the code itself. Not chased; the claim was abandoned when the session stopped.
3. **A same-day re-run replaces the day's snapshot.** Worth knowing before designing any drill that
   depends on a specific snapshot surviving.
4. **The hub's ClusterIP is not reachable from DooPlex's host network** — the operator UI needs a
   `kubectl port-forward`. The `curl -u :$HUB_PW` recipe in memory omits that.