5.7 KiB
REPORT — DRILL: the retained key, and the two fixes nobody had watched work (2026-08-12)
Class: drill (unattended, destructive on Tier 0) + spike for Phase C's first step
Venues: drill-r50 (nested PVE on DooPlex), demo-felhom (guest 9201) — both Tier 0.
demo-hp was never touched. peti-felhom was never contacted. No abandon countdown was started,
shortened or triggered.
Full record: documentation/audits/DRILL-retained-key-2026-08-12.md
The answer to the question this drill existed to answer
(a) Is the old key kept? YES — proven for the first time in the fleet's history. (b) Does the kept key open the old backups? YES — three planted files, including a Hungarian accented filename verified as raw bytes, restored byte-identical from a store the machine itself could no longer open. (c) Can the customer get there through the product? NO — and they are told their correct code is wrong.
The brief said to be ready for the answer to be no, and our own records predicted retention would be
"a box we fill and cannot open". That was half right, and the wrong half was the one nobody had
checked. The box opens. What does not exist is the door: ListSupersededEscrow
(hub/internal/store/store.go:2841) is the only reader of a retained key and has zero production
callers; the recovery path selects FROM host_escrow — the current row only. Asked with the very
code that had just opened the retained row by hand, the product answered "the recovery code did not
open the sealed bundle — nothing was written". → R-304, rank 1
Consequences: the census answer stands; the countdown banner's promise is true in substance, false in practice; the capability map's recovery claim has been moved with today's evidence.
What shipped
installer-v1.27.0 published — tag cut and both --refs in manifests/webpage.yaml bumped
(sidecar line 327, init container line 372). Publication was earned: both faults were watched
happening first, from a machine reset to factory state.
- R-300 CLOSED — pre-fix uninstall left dnsmasq
enabled/activeon0.0.0.0:53; the next byo install refused, exit 1. Fixed path: recordednot present before Felhom→stopping + disabling it→:53 FREE→ preflight PASS. The owner's side proven too (recordyes→ left running). - R-297 CLOSED — a stale
golden-0.98.3.tar.zstplanted as newest-by-filename; v1.25.0 took it with no comparison and the box came up on controller 0.98.3 against a vouched 0.213.0 — below the floor and below v0.206.0 where the off-site recovery screen exists. Fixed path re-fetched and sha-verified the vouched golden (landed 0.213.0); an operator-named stale archive was refused.
Findings opened — ceiling R-303 → R-310
| # | Rank | What |
|---|---|---|
| R-304 | 1 | Retained key has no product route; the correct old code is reported as wrong |
| R-305 | 2 | The R-300 cleanup fires once per machine — the leftover returns on the second reinstall (proven, cycles 2/3) |
| R-308 | 2 | Stored controller PASSWORD no longer opens demo-felhom (Hibás jelszó) — not the quoting trap |
| R-306 | 3 | --preflight-only says "no state written" and writes state.json — with an ownership answer that can be wrong |
| R-309 | 3 | The day-0 runbook says pushing publishes the installer; false since R-110 (measured: public URL served 1.25.0 while main had 1.27.0) |
| R-310 | 4 | Duplicated sentence in the golden refusal; --uninstall needs a pty and --force does not bypass it |
| R-307 | — | Operator decision, deadline 2026-08-24 — see below |
What needs you
demo-felhom carries a live abandon countdown — started 2026-08-10, firing 2026-08-24, for
/home/felhom-repo.orphaned-20260810. This drill did not start it and deliberately did not
cancel it. The brief's end state asked for no countdown anywhere; satisfying that means choosing:
cancel it (copy kept indefinitely, storage cost, no data risk) or let it run (copy deleted,
irreversibly). Doing nothing selects deletion. → R-307
End state
demo-felhom— up, reporting, healthy, on the vouched pair;repo_passwordrestored to the original (sha c60c8bc737a6b7c6…), escrow re-sealed and uploaded, off-site repo reachable (restic snapshotsexit 0). Its recovery code was rotated by the final ceremony andR_DEMO-FELHOMupdated in place (prior file backed up alongside). Planted data removed; eight secret-bearing files shredded.demo-hp— untouched, reporting.drill-r50— reverted to snapshotvirgin, powered off.- Hub — two new retained rows (the P1 and P2 blobs), deliberately kept as the fixture proving the
retention works.
drill-r50-0a4f9are-used, not duplicated: no new scratch customer. - Off-site — only demo-felhom's own repository path touched,
backupthe only mutating verb used. No prune, no forget, no delete, no rename anywhere. One snapshot added and deliberately left:6ea85413, 66 KiB, taggeddrill-retained-key-20260812— removable by ID if you want it gone.
Honest gaps
- The Phase A logs did not survive the intermediate revert to
virgin. Every quotation in the audit is verbatim from the live run, but the raw files are gone. Procedural lesson, recorded. - The planted data reached the store via
resticdirectly, not the dashboard button, because of R-308 — so the app-backup→unit→offsite chain went unexercised. Not what this drill measured. - Wall clock ≈ 1 h 03 min against a 4–5 h envelope. Nothing was dropped; Phase C ran concurrently with Phase B on a different machine.