# REPORT — DRILL: the retained key, and the two fixes nobody had watched work (2026-08-12) **Class:** drill (unattended, destructive on Tier 0) + spike for Phase C's first step **Venues:** `drill-r50` (nested PVE on DooPlex), `demo-felhom` (guest 9201) — both Tier 0. **`demo-hp` was never touched. `peti-felhom` was never contacted. No abandon countdown was started, shortened or triggered.** **Full record:** `documentation/audits/DRILL-retained-key-2026-08-12.md` --- ## The answer to the question this drill existed to answer **(a) Is the old key kept? YES** — proven for the first time in the fleet's history. **(b) Does the kept key open the old backups? YES** — three planted files, including a Hungarian accented filename verified as raw bytes, restored **byte-identical** from a store the machine itself could no longer open. **(c) Can the customer get there through the product? NO — and they are told their correct code is wrong.** The brief said to be ready for the answer to be no, and our own records predicted retention would be *"a box we fill and cannot open"*. **That was half right, and the wrong half was the one nobody had checked.** The box opens. What does not exist is the door: `ListSupersededEscrow` (`hub/internal/store/store.go:2841`) is the only reader of a retained key and has **zero production callers**; the recovery path selects `FROM host_escrow` — the current row only. Asked with the very code that had just opened the retained row by hand, the product answered *"the recovery code did not open the sealed bundle — nothing was written"*. → **R-304, rank 1** Consequences: the census answer **stands**; the countdown banner's promise is **true in substance, false in practice**; the capability map's recovery claim **has been moved** with today's evidence. ## What shipped **`installer-v1.27.0` published** — tag cut and **both** `--ref`s in `manifests/webpage.yaml` bumped (sidecar line 327, init container line 372). Publication was earned: both faults were watched happening first, from a machine reset to factory state. - **R-300 CLOSED** — pre-fix uninstall left dnsmasq `enabled`/`active` on `0.0.0.0:53`; the next byo install refused, exit 1. Fixed path: recorded `not present before Felhom` → `stopping + disabling it` → `:53 FREE` → preflight PASS. The owner's side proven too (record `yes` → left running). - **R-297 CLOSED** — a stale `golden-0.98.3.tar.zst` planted as newest-by-filename; v1.25.0 took it with no comparison and **the box came up on controller 0.98.3** against a vouched 0.213.0 — below the floor and below v0.206.0 where the off-site recovery screen exists. Fixed path re-fetched and sha-verified the vouched golden (landed 0.213.0); an operator-named stale archive was **refused**. ## Findings opened — ceiling R-303 → R-310 | # | Rank | What | |---|---|---| | **R-304** | **1** | Retained key has no product route; the correct old code is reported as wrong | | **R-305** | 2 | The R-300 cleanup fires **once per machine** — the leftover returns on the second reinstall (proven, cycles 2/3) | | **R-308** | 2 | Stored controller `PASSWORD` no longer opens demo-felhom (`Hibás jelszó`) — not the quoting trap | | **R-306** | 3 | `--preflight-only` says *"no state written"* and writes `state.json` — with an ownership answer that can be wrong | | **R-309** | 3 | The day-0 runbook says pushing publishes the installer; false since R-110 (measured: public URL served 1.25.0 while `main` had 1.27.0) | | **R-310** | 4 | Duplicated sentence in the golden refusal; `--uninstall` needs a pty and `--force` does not bypass it | | **R-307** | — | **Operator decision, deadline 2026-08-24** — see below | ## What needs you **`demo-felhom` carries a live abandon countdown** — started 2026-08-10, **firing 2026-08-24**, for `/home/felhom-repo.orphaned-20260810`. This drill did **not** start it and deliberately did **not** cancel it. The brief's end state asked for no countdown anywhere; satisfying that means choosing: **cancel it** (copy kept indefinitely, storage cost, no data risk) or **let it run** (copy deleted, irreversibly). **Doing nothing selects deletion.** → **R-307** ## End state - **`demo-felhom`** — up, reporting, healthy, on the vouched pair; `repo_password` restored to the original (`sha c60c8bc737a6b7c6…`), escrow re-sealed and uploaded, off-site repo reachable (`restic snapshots` exit 0). Its recovery code was rotated by the final ceremony and `R_DEMO-FELHOM` updated in place (prior file backed up alongside). Planted data removed; eight secret-bearing files **shredded**. - **`demo-hp`** — untouched, reporting. - **`drill-r50`** — **reverted to snapshot `virgin`, powered off.** - **Hub** — two new retained rows (the P1 and P2 blobs), deliberately kept as the fixture proving the retention works. `drill-r50-0a4f9a` re-used, not duplicated: no new scratch customer. - **Off-site** — only demo-felhom's own repository path touched, `backup` the only mutating verb used. **No prune, no forget, no delete, no rename anywhere.** One snapshot added and deliberately left: `6ea85413`, 66 KiB, tagged `drill-retained-key-20260812` — removable by ID if you want it gone. ## Honest gaps - **The Phase A logs did not survive** the intermediate revert to `virgin`. Every quotation in the audit is verbatim from the live run, but the raw files are gone. Procedural lesson, recorded. - The planted data reached the store via `restic` directly, not the dashboard button, because of R-308 — so the app-backup→unit→offsite chain went unexercised. Not what this drill measured. - Wall clock **≈ 1 h 13 min** against a 4–5 h envelope. Nothing was dropped; Phase C ran concurrently with Phase B on a different machine.