88 lines
5.7 KiB
Markdown
88 lines
5.7 KiB
Markdown
# REPORT — DRILL: the retained key, and the two fixes nobody had watched work (2026-08-12)
|
||
|
||
**Class:** drill (unattended, destructive on Tier 0) + spike for Phase C's first step
|
||
**Venues:** `drill-r50` (nested PVE on DooPlex), `demo-felhom` (guest 9201) — both Tier 0.
|
||
**`demo-hp` was never touched. `peti-felhom` was never contacted. No abandon countdown was started,
|
||
shortened or triggered.**
|
||
**Full record:** `documentation/audits/DRILL-retained-key-2026-08-12.md`
|
||
|
||
---
|
||
|
||
## The answer to the question this drill existed to answer
|
||
|
||
**(a) Is the old key kept? YES** — proven for the first time in the fleet's history.
|
||
**(b) Does the kept key open the old backups? YES** — three planted files, including a Hungarian
|
||
accented filename verified as raw bytes, restored **byte-identical** from a store the machine itself
|
||
could no longer open.
|
||
**(c) Can the customer get there through the product? NO — and they are told their correct code is
|
||
wrong.**
|
||
|
||
The brief said to be ready for the answer to be no, and our own records predicted retention would be
|
||
*"a box we fill and cannot open"*. **That was half right, and the wrong half was the one nobody had
|
||
checked.** The box opens. What does not exist is the door: `ListSupersededEscrow`
|
||
(`hub/internal/store/store.go:2841`) is the only reader of a retained key and has **zero production
|
||
callers**; the recovery path selects `FROM host_escrow` — the current row only. Asked with the very
|
||
code that had just opened the retained row by hand, the product answered *"the recovery code did not
|
||
open the sealed bundle — nothing was written"*. → **R-304, rank 1**
|
||
|
||
Consequences: the census answer **stands**; the countdown banner's promise is **true in substance,
|
||
false in practice**; the capability map's recovery claim **has been moved** with today's evidence.
|
||
|
||
## What shipped
|
||
|
||
**`installer-v1.27.0` published** — tag cut and **both** `--ref`s in `manifests/webpage.yaml` bumped
|
||
(sidecar line 327, init container line 372). Publication was earned: both faults were watched
|
||
happening first, from a machine reset to factory state.
|
||
|
||
- **R-300 CLOSED** — pre-fix uninstall left dnsmasq `enabled`/`active` on `0.0.0.0:53`; the next byo
|
||
install refused, exit 1. Fixed path: recorded `not present before Felhom` → `stopping + disabling
|
||
it` → `:53 FREE` → preflight PASS. The owner's side proven too (record `yes` → left running).
|
||
- **R-297 CLOSED** — a stale `golden-0.98.3.tar.zst` planted as newest-by-filename; v1.25.0 took it
|
||
with no comparison and **the box came up on controller 0.98.3** against a vouched 0.213.0 — below
|
||
the floor and below v0.206.0 where the off-site recovery screen exists. Fixed path re-fetched and
|
||
sha-verified the vouched golden (landed 0.213.0); an operator-named stale archive was **refused**.
|
||
|
||
## Findings opened — ceiling R-303 → R-310
|
||
|
||
| # | Rank | What |
|
||
|---|---|---|
|
||
| **R-304** | **1** | Retained key has no product route; the correct old code is reported as wrong |
|
||
| **R-305** | 2 | The R-300 cleanup fires **once per machine** — the leftover returns on the second reinstall (proven, cycles 2/3) |
|
||
| **R-308** | 2 | Stored controller `PASSWORD` no longer opens demo-felhom (`Hibás jelszó`) — not the quoting trap |
|
||
| **R-306** | 3 | `--preflight-only` says *"no state written"* and writes `state.json` — with an ownership answer that can be wrong |
|
||
| **R-309** | 3 | The day-0 runbook says pushing publishes the installer; false since R-110 (measured: public URL served 1.25.0 while `main` had 1.27.0) |
|
||
| **R-310** | 4 | Duplicated sentence in the golden refusal; `--uninstall` needs a pty and `--force` does not bypass it |
|
||
| **R-307** | — | **Operator decision, deadline 2026-08-24** — see below |
|
||
|
||
## What needs you
|
||
|
||
**`demo-felhom` carries a live abandon countdown** — started 2026-08-10, **firing 2026-08-24**, for
|
||
`/home/felhom-repo.orphaned-20260810`. This drill did **not** start it and deliberately did **not**
|
||
cancel it. The brief's end state asked for no countdown anywhere; satisfying that means choosing:
|
||
**cancel it** (copy kept indefinitely, storage cost, no data risk) or **let it run** (copy deleted,
|
||
irreversibly). **Doing nothing selects deletion.** → **R-307**
|
||
|
||
## End state
|
||
|
||
- **`demo-felhom`** — up, reporting, healthy, on the vouched pair; `repo_password` restored to the
|
||
original (`sha c60c8bc737a6b7c6…`), escrow re-sealed and uploaded, off-site repo reachable
|
||
(`restic snapshots` exit 0). Its recovery code was rotated by the final ceremony and
|
||
`R_DEMO-FELHOM` updated in place (prior file backed up alongside). Planted data removed; eight
|
||
secret-bearing files **shredded**.
|
||
- **`demo-hp`** — untouched, reporting.
|
||
- **`drill-r50`** — **reverted to snapshot `virgin`, powered off.**
|
||
- **Hub** — two new retained rows (the P1 and P2 blobs), deliberately kept as the fixture proving the
|
||
retention works. `drill-r50-0a4f9a` re-used, not duplicated: no new scratch customer.
|
||
- **Off-site** — only demo-felhom's own repository path touched, `backup` the only mutating verb used.
|
||
**No prune, no forget, no delete, no rename anywhere.** One snapshot added and deliberately left:
|
||
`6ea85413`, 66 KiB, tagged `drill-retained-key-20260812` — removable by ID if you want it gone.
|
||
|
||
## Honest gaps
|
||
|
||
- **The Phase A logs did not survive** the intermediate revert to `virgin`. Every quotation in the
|
||
audit is verbatim from the live run, but the raw files are gone. Procedural lesson, recorded.
|
||
- The planted data reached the store via `restic` directly, not the dashboard button, because of
|
||
R-308 — so the app-backup→unit→offsite chain went unexercised. Not what this drill measured.
|
||
- Wall clock **≈ 1 h 03 min** against a 4–5 h envelope. Nothing was dropped; Phase C ran concurrently
|
||
with Phase B on a different machine.
|