DRILL: the retained key works, and the customer cannot reach it
gates / gates (push) Successful in 23s

Three verdicts, kept separate because collapsing them is how this assumption
survived a week.

(a) The material IS retained. host_escrow_superseded id 11 is the first retained
row in fleet history to carry identity_blob (572 B), byte-identical to the
pre-supersession row (sha256 a10032341c8584ed...).

(b) The retained material DOES open the old store. Unsealed with the old recovery
code it yielded a password byte-identical to the pre-change one, and restored
three planted files byte-identical from a store the box itself could no longer
open - including a Hungarian accented filename verified as raw bytes. Negative
control ran first and failed closed.

(c) The customer has NO route, and is misinformed. ListSupersededEscrow has zero
production callers; the recovery path selects FROM host_escrow. Asked with the
code that had just worked by hand, the product answered "the recovery code did
not open the sealed bundle". A valid code for retained history is reported as a
bad code - the R-224 class again. R-304, rank 1.

Both installer faults were watched happening first, so installer-v1.27.0 is now
published (tag + both webpage.yaml refs). Pre-fix: the box came up on controller
0.98.3 against a vouched 0.213.0, below the floor and below the version carrying
the recovery screen; and our own uninstall left dnsmasq on 0.0.0.0:53 so our own
next install refused. R-297 and R-300 CLOSED.

Also filed R-305 (the dnsmasq fix fires once per machine - the leftover returns
on the second reinstall, proven), R-306 (--preflight-only writes state it says it
does not), R-307 (a live abandon countdown on demo-felhom, firing 2026-08-24 -
operator decision), R-308 (stored controller password stale), R-309 (the day-0
runbook's publication claim has been false since R-110), R-310 (two edges).

Ceiling R-303 -> R-310. Capability map moved: the retention claim is now marked
operator-only. Phase A logs did not survive the intermediate revert; recorded.
This commit is contained in:
2026-08-12 17:41:56 +02:00
parent fbe1155fbb
commit 1d5f2b8bb6
12 changed files with 2445 additions and 90 deletions
+70 -69
View File
@@ -1,86 +1,87 @@
# REPORT — the record, the second promise, the removal chain (2026-08-12)
# REPORT — DRILL: the retained key, and the two fixes nobody had watched work (2026-08-12)
Commits `f76cbf0` (Part 0), `2389544` (spec + bake evidence), `125aec1` (R-300), `890a474` (STATUS).
Controller: `68f3e12` (R-299). All gates green in both repos; **`--no-verify` used nowhere** — the
currency gate refused a push mid-session and stayed refused until the bake made it true.
**Class:** drill (unattended, destructive on Tier 0) + spike for Phase C's first step
**Venues:** `drill-r50` (nested PVE on DooPlex), `demo-felhom` (guest 9201) — both Tier 0.
**`demo-hp` was never touched. `peti-felhom` was never contacted. No abandon countdown was started,
shortened or triggered.**
**Full record:** `documentation/audits/DRILL-retained-key-2026-08-12.md`
## Part 0 — the record now says what is true
---
Re-read from the hub's own store, with a control (escrow query returns 1+1 for each demo box, 0+0 for
`drill-r50`). Confirmed: **3 hosts total, no peti row under any name**; 4 escrow rows hub-wide, all
demo; `host_deletions` id=1 `peti-felhom-86d37d` **2026-07-15 08:56:22**, `escrow_acked = 0`; last
report **2026-07-15 08:39:00 UTC**, controller 0.115.0, `offsite: {escrow_state: "pending",
snapshot_count: 0}`; local app-data repo empty; dashboard never claimed. The `PETI` row and `STATUS.md`
now state that the mitigation they named **does not exist** — and leave the parked/not-parked ruling
open, because that is the operator's and the fact does not need his opinion to be true.
## The answer to the question this drill existed to answer
**Contact since the deletion — answered without touching the machine.** No inbound row of any kind
after 2026-07-15 08:39; the only later rows are the hub's OWN alarms (`source = hub`: `node_stale`
09:09:32, `node_down` 09:39:32). No contact attempt, accepted or rejected, in the current hub pod's
logs (since 2026-08-09 17:26Z) — grep proven by **851 `demo-hp` hits against 0 for peti, 0
unauthorized**. **The window 2026-07-15 → 2026-08-09 cannot be answered from records**: a report from a
deleted host 401s and is not persisted, and those logs are gone.
**(a) Is the old key kept? YES** — proven for the first time in the fleet's history.
**(b) Does the kept key open the old backups? YES** — three planted files, including a Hungarian
accented filename verified as raw bytes, restored **byte-identical** from a store the machine itself
could no longer open.
**(c) Can the customer get there through the product? NO — and they are told their correct code is
wrong.**
## Part 1/2 — the second promise (controller v0.212.0)
The brief said to be ready for the answer to be no, and our own records predicted retention would be
*"a box we fill and cannot open"*. **That was half right, and the wrong half was the one nobody had
checked.** The box opens. What does not exist is the door: `ListSupersededEscrow`
(`hub/internal/store/store.go:2841`) is the only reader of a retained key and has **zero production
callers**; the recovery path selects `FROM host_escrow` — the current row only. Asked with the very
code that had just opened the retained row by hand, the product answered *"the recovery code did not
open the sealed bundle — nothing was written"*. → **R-304, rank 1**
`backups_remote.html` line 98 — the **always-visible** half — still ended *„…visszaállíthatók
lehetnek"*. Replaced; the two accurate halves kept.
Consequences: the census answer **stands**; the countdown banner's promise is **true in substance,
false in practice**; the capability map's recovery claim **has been moved** with today's evidence.
**Why it survived, which is the useful part:** the spec called that line *"Accurate; keep"*, **and the
guard matched one INFLECTION** (`visszaállítható lehet`) that the plural does not contain. Guard
broadened to the stem `visszaállíthat`. Spec corrected in both places.
## What shipped
**Plant → convict → remove → pass:** planted the exact shipped plural → the stem guard **FAILED** and
quoted it back; the old singular guard **does not match that sentence at all** (`False`, shown as a
pure string fact, not a contaminated source grep); removed → 5/5 orphan-card tests pass.
**`installer-v1.27.0` published** — tag cut and **both** `--ref`s in `manifests/webpage.yaml` bumped
(sidecar line 327, init container line 372). Publication was earned: both faults were watched
happening first, from a machine reset to factory state.
Two instrument defects fixed on the way: the guard's failure message sliced rendered HTML at a **byte**
offset and cut Hungarian mid-character (now rune-safe); and a first pass at the bake's acceptance
markers returned a false `0` through shell quoting — re-run with `grep -F`, because a zero from a
broken instrument is not a measurement.
- **R-300 CLOSED** — pre-fix uninstall left dnsmasq `enabled`/`active` on `0.0.0.0:53`; the next byo
install refused, exit 1. Fixed path: recorded `not present before Felhom``stopping + disabling
it``:53 FREE` → preflight PASS. The owner's side proven too (record `yes` → left running).
- **R-297 CLOSED** — a stale `golden-0.98.3.tar.zst` planted as newest-by-filename; v1.25.0 took it
with no comparison and **the box came up on controller 0.98.3** against a vouched 0.213.0 — below
the floor and below v0.206.0 where the off-site recovery screen exists. Fixed path re-fetched and
sha-verified the vouched golden (landed 0.213.0); an operator-named stale archive was **refused**.
## Part 3 — the removal chain (R-300), CODE ONLY
## Findings opened — ceiling R-303 → R-310
Confirmed at source: uninstall removes the snippet and **restarts** (`:1075-1082`), leaving the unit
enabled; the byo preflight then refuses on `:53`.
| # | Rank | What |
|---|---|---|
| **R-304** | **1** | Retained key has no product route; the correct old code is reported as wrong |
| **R-305** | 2 | The R-300 cleanup fires **once per machine** — the leftover returns on the second reinstall (proven, cycles 2/3) |
| **R-308** | 2 | Stored controller `PASSWORD` no longer opens demo-felhom (`Hibás jelszó`) — not the quoting trap |
| **R-306** | 3 | `--preflight-only` says *"no state written"* and writes `state.json` — with an ownership answer that can be wrong |
| **R-309** | 3 | The day-0 runbook says pushing publishes the installer; false since R-110 (measured: public URL served 1.25.0 while `main` had 1.27.0) |
| **R-310** | 4 | Duplicated sentence in the golden refusal; `--uninstall` needs a pty and `--force` does not bypass it |
| **R-307** | — | **Operator decision, deadline 2026-08-24** — see below |
**The prompt's framing needed one correction:** the installer does **not** install dnsmasq — the
**agent** does (`lanresolver.go:107`), conditionally, and `felhom-agent` is fenced this session. So
ownership is recorded at **preflight**, before anything is installed, which is the only moment it is a
fact — not a package mtime. At removal: Felhom's → stop+disable; the owner's → restart only; **no
record (every box in the field) → restart only, fail-safe, with the reason and the command logged.**
The refusal keeps its two routes and its promise, and gains the missing line naming our own leftover.
## What needs you
**NOT OBSERVED LIVE.** The `drill-r50` install→uninstall→install cycle was not run, so the wrong
outcome was never quoted and **no `installer-v1.27.0` tag is cut.**
**`demo-felhom` carries a live abandon countdown** — started 2026-08-10, **firing 2026-08-24**, for
`/home/felhom-repo.orphaned-20260810`. This drill did **not** start it and deliberately did **not**
cancel it. The brief's end state asked for no countdown anywhere; satisfying that means choosing:
**cancel it** (copy kept indefinitely, storage cost, no data risk) or **let it run** (copy deleted,
irreversibly). **Doing nothing selects deletion.****R-307**
## Golden 0.212.0 — baked, published, round-trip verified
## End state
`4b0a7dacc503c38732ed0a44949398639248c7fbd90758a1e4a047c21a7a15d8`, 656 611 277 B, served bytes
re-downloaded and hashed identical. All six markers counted with `grep -F`. Token never on a command
line; leak grep 0, believable because a planted-token control grepped 1. Drill VM reverted to `virgin`.
Evidence: `documentation/tests/golden-0.212.0-2026-08-12/`.
- **`demo-felhom`** — up, reporting, healthy, on the vouched pair; `repo_password` restored to the
original (`sha c60c8bc737a6b7c6…`), escrow re-sealed and uploaded, off-site repo reachable
(`restic snapshots` exit 0). Its recovery code was rotated by the final ceremony and
`R_DEMO-FELHOM` updated in place (prior file backed up alongside). Planted data removed; eight
secret-bearing files **shredded**.
- **`demo-hp`** — untouched, reporting.
- **`drill-r50`** — **reverted to snapshot `virgin`, powered off.**
- **Hub** — two new retained rows (the P1 and P2 blobs), deliberately kept as the fixture proving the
retention works. `drill-r50-0a4f9a` re-used, not duplicated: no new scratch customer.
- **Off-site** — only demo-felhom's own repository path touched, `backup` the only mutating verb used.
**No prune, no forget, no delete, no rename anywhere.** One snapshot added and deliberately left:
`6ea85413`, 66 KiB, tagged `drill-retained-key-20260812` — removable by ID if you want it gone.
## DROPPED — named plainly
## Honest gaps
- **The hub half of the naming (R-295)** — dropped first, exactly as the drop order allows. The four
hub surfaces were NOT enumerated at `file:line`; that enumeration is still owed.
- **The stale-golden observation (R-297)** — dropped second. Nothing published, which is the safe state.
- **And one that was NOT droppable: R-300's live cycle.** The code shipped; the demonstration did not.
Named here rather than shortened silently.
## Observations, not acted on
- **R-301** — the abandon countdown banner (`layout.html:143`) states the retired promise a third time
and un-hedged. It is probably TRUE where it renders, and it renders on every page; a rebuilt box can
have an active countdown while its store is orphaned. Not established: whether the two sentences name
the same bytes. Left alone deliberately — this session was fenced to the orphan card.
- The orphan card now says "we cannot determine / it depends on the key / write to us" **twice** once a
customer clicks through — reinforcement at the decision point rather than a contradiction, but worth
an eye if the card is revisited.
## Deliberately out of scope
The CI runs that fail with no log; the twenty facts the machines report that nothing reads; the nine
grey claims; the storage page's separate empty-list cause (R-298); and proving a *retained* key can
actually open an old store — the one thing the retention fix has never been shown to do.
- **The Phase A logs did not survive** the intermediate revert to `virgin`. Every quotation in the
audit is verbatim from the live run, but the raw files are gone. Procedural lesson, recorded.
- The planted data reached the store via `restic` directly, not the dashboard button, because of
R-308 — so the app-backup→unit→offsite chain went unexercised. Not what this drill measured.
- Wall clock **≈ 1 h 13 min** against a 45 h envelope. Nothing was dropped; Phase C ran concurrently
with Phase B on a different machine.