DRILL: the retained key works, and the customer cannot reach it
gates / gates (push) Successful in 23s
gates / gates (push) Successful in 23s
Three verdicts, kept separate because collapsing them is how this assumption survived a week. (a) The material IS retained. host_escrow_superseded id 11 is the first retained row in fleet history to carry identity_blob (572 B), byte-identical to the pre-supersession row (sha256 a10032341c8584ed...). (b) The retained material DOES open the old store. Unsealed with the old recovery code it yielded a password byte-identical to the pre-change one, and restored three planted files byte-identical from a store the box itself could no longer open - including a Hungarian accented filename verified as raw bytes. Negative control ran first and failed closed. (c) The customer has NO route, and is misinformed. ListSupersededEscrow has zero production callers; the recovery path selects FROM host_escrow. Asked with the code that had just worked by hand, the product answered "the recovery code did not open the sealed bundle". A valid code for retained history is reported as a bad code - the R-224 class again. R-304, rank 1. Both installer faults were watched happening first, so installer-v1.27.0 is now published (tag + both webpage.yaml refs). Pre-fix: the box came up on controller 0.98.3 against a vouched 0.213.0, below the floor and below the version carrying the recovery screen; and our own uninstall left dnsmasq on 0.0.0.0:53 so our own next install refused. R-297 and R-300 CLOSED. Also filed R-305 (the dnsmasq fix fires once per machine - the leftover returns on the second reinstall, proven), R-306 (--preflight-only writes state it says it does not), R-307 (a live abandon countdown on demo-felhom, firing 2026-08-24 - operator decision), R-308 (stored controller password stale), R-309 (the day-0 runbook's publication claim has been false since R-110), R-310 (two edges). Ceiling R-303 -> R-310. Capability map moved: the retention claim is now marked operator-only. Phase A logs did not survive the intermediate revert; recorded.
This commit is contained in:
@@ -1,86 +1,87 @@
|
||||
# REPORT — the record, the second promise, the removal chain (2026-08-12)
|
||||
# REPORT — DRILL: the retained key, and the two fixes nobody had watched work (2026-08-12)
|
||||
|
||||
Commits `f76cbf0` (Part 0), `2389544` (spec + bake evidence), `125aec1` (R-300), `890a474` (STATUS).
|
||||
Controller: `68f3e12` (R-299). All gates green in both repos; **`--no-verify` used nowhere** — the
|
||||
currency gate refused a push mid-session and stayed refused until the bake made it true.
|
||||
**Class:** drill (unattended, destructive on Tier 0) + spike for Phase C's first step
|
||||
**Venues:** `drill-r50` (nested PVE on DooPlex), `demo-felhom` (guest 9201) — both Tier 0.
|
||||
**`demo-hp` was never touched. `peti-felhom` was never contacted. No abandon countdown was started,
|
||||
shortened or triggered.**
|
||||
**Full record:** `documentation/audits/DRILL-retained-key-2026-08-12.md`
|
||||
|
||||
## Part 0 — the record now says what is true
|
||||
---
|
||||
|
||||
Re-read from the hub's own store, with a control (escrow query returns 1+1 for each demo box, 0+0 for
|
||||
`drill-r50`). Confirmed: **3 hosts total, no peti row under any name**; 4 escrow rows hub-wide, all
|
||||
demo; `host_deletions` id=1 `peti-felhom-86d37d` **2026-07-15 08:56:22**, `escrow_acked = 0`; last
|
||||
report **2026-07-15 08:39:00 UTC**, controller 0.115.0, `offsite: {escrow_state: "pending",
|
||||
snapshot_count: 0}`; local app-data repo empty; dashboard never claimed. The `PETI` row and `STATUS.md`
|
||||
now state that the mitigation they named **does not exist** — and leave the parked/not-parked ruling
|
||||
open, because that is the operator's and the fact does not need his opinion to be true.
|
||||
## The answer to the question this drill existed to answer
|
||||
|
||||
**Contact since the deletion — answered without touching the machine.** No inbound row of any kind
|
||||
after 2026-07-15 08:39; the only later rows are the hub's OWN alarms (`source = hub`: `node_stale`
|
||||
09:09:32, `node_down` 09:39:32). No contact attempt, accepted or rejected, in the current hub pod's
|
||||
logs (since 2026-08-09 17:26Z) — grep proven by **851 `demo-hp` hits against 0 for peti, 0
|
||||
unauthorized**. **The window 2026-07-15 → 2026-08-09 cannot be answered from records**: a report from a
|
||||
deleted host 401s and is not persisted, and those logs are gone.
|
||||
**(a) Is the old key kept? YES** — proven for the first time in the fleet's history.
|
||||
**(b) Does the kept key open the old backups? YES** — three planted files, including a Hungarian
|
||||
accented filename verified as raw bytes, restored **byte-identical** from a store the machine itself
|
||||
could no longer open.
|
||||
**(c) Can the customer get there through the product? NO — and they are told their correct code is
|
||||
wrong.**
|
||||
|
||||
## Part 1/2 — the second promise (controller v0.212.0)
|
||||
The brief said to be ready for the answer to be no, and our own records predicted retention would be
|
||||
*"a box we fill and cannot open"*. **That was half right, and the wrong half was the one nobody had
|
||||
checked.** The box opens. What does not exist is the door: `ListSupersededEscrow`
|
||||
(`hub/internal/store/store.go:2841`) is the only reader of a retained key and has **zero production
|
||||
callers**; the recovery path selects `FROM host_escrow` — the current row only. Asked with the very
|
||||
code that had just opened the retained row by hand, the product answered *"the recovery code did not
|
||||
open the sealed bundle — nothing was written"*. → **R-304, rank 1**
|
||||
|
||||
`backups_remote.html` line 98 — the **always-visible** half — still ended *„…visszaállíthatók
|
||||
lehetnek"*. Replaced; the two accurate halves kept.
|
||||
Consequences: the census answer **stands**; the countdown banner's promise is **true in substance,
|
||||
false in practice**; the capability map's recovery claim **has been moved** with today's evidence.
|
||||
|
||||
**Why it survived, which is the useful part:** the spec called that line *"Accurate; keep"*, **and the
|
||||
guard matched one INFLECTION** (`visszaállítható lehet`) that the plural does not contain. Guard
|
||||
broadened to the stem `visszaállíthat`. Spec corrected in both places.
|
||||
## What shipped
|
||||
|
||||
**Plant → convict → remove → pass:** planted the exact shipped plural → the stem guard **FAILED** and
|
||||
quoted it back; the old singular guard **does not match that sentence at all** (`False`, shown as a
|
||||
pure string fact, not a contaminated source grep); removed → 5/5 orphan-card tests pass.
|
||||
**`installer-v1.27.0` published** — tag cut and **both** `--ref`s in `manifests/webpage.yaml` bumped
|
||||
(sidecar line 327, init container line 372). Publication was earned: both faults were watched
|
||||
happening first, from a machine reset to factory state.
|
||||
|
||||
Two instrument defects fixed on the way: the guard's failure message sliced rendered HTML at a **byte**
|
||||
offset and cut Hungarian mid-character (now rune-safe); and a first pass at the bake's acceptance
|
||||
markers returned a false `0` through shell quoting — re-run with `grep -F`, because a zero from a
|
||||
broken instrument is not a measurement.
|
||||
- **R-300 CLOSED** — pre-fix uninstall left dnsmasq `enabled`/`active` on `0.0.0.0:53`; the next byo
|
||||
install refused, exit 1. Fixed path: recorded `not present before Felhom` → `stopping + disabling
|
||||
it` → `:53 FREE` → preflight PASS. The owner's side proven too (record `yes` → left running).
|
||||
- **R-297 CLOSED** — a stale `golden-0.98.3.tar.zst` planted as newest-by-filename; v1.25.0 took it
|
||||
with no comparison and **the box came up on controller 0.98.3** against a vouched 0.213.0 — below
|
||||
the floor and below v0.206.0 where the off-site recovery screen exists. Fixed path re-fetched and
|
||||
sha-verified the vouched golden (landed 0.213.0); an operator-named stale archive was **refused**.
|
||||
|
||||
## Part 3 — the removal chain (R-300), CODE ONLY
|
||||
## Findings opened — ceiling R-303 → R-310
|
||||
|
||||
Confirmed at source: uninstall removes the snippet and **restarts** (`:1075-1082`), leaving the unit
|
||||
enabled; the byo preflight then refuses on `:53`.
|
||||
| # | Rank | What |
|
||||
|---|---|---|
|
||||
| **R-304** | **1** | Retained key has no product route; the correct old code is reported as wrong |
|
||||
| **R-305** | 2 | The R-300 cleanup fires **once per machine** — the leftover returns on the second reinstall (proven, cycles 2/3) |
|
||||
| **R-308** | 2 | Stored controller `PASSWORD` no longer opens demo-felhom (`Hibás jelszó`) — not the quoting trap |
|
||||
| **R-306** | 3 | `--preflight-only` says *"no state written"* and writes `state.json` — with an ownership answer that can be wrong |
|
||||
| **R-309** | 3 | The day-0 runbook says pushing publishes the installer; false since R-110 (measured: public URL served 1.25.0 while `main` had 1.27.0) |
|
||||
| **R-310** | 4 | Duplicated sentence in the golden refusal; `--uninstall` needs a pty and `--force` does not bypass it |
|
||||
| **R-307** | — | **Operator decision, deadline 2026-08-24** — see below |
|
||||
|
||||
**The prompt's framing needed one correction:** the installer does **not** install dnsmasq — the
|
||||
**agent** does (`lanresolver.go:107`), conditionally, and `felhom-agent` is fenced this session. So
|
||||
ownership is recorded at **preflight**, before anything is installed, which is the only moment it is a
|
||||
fact — not a package mtime. At removal: Felhom's → stop+disable; the owner's → restart only; **no
|
||||
record (every box in the field) → restart only, fail-safe, with the reason and the command logged.**
|
||||
The refusal keeps its two routes and its promise, and gains the missing line naming our own leftover.
|
||||
## What needs you
|
||||
|
||||
**NOT OBSERVED LIVE.** The `drill-r50` install→uninstall→install cycle was not run, so the wrong
|
||||
outcome was never quoted and **no `installer-v1.27.0` tag is cut.**
|
||||
**`demo-felhom` carries a live abandon countdown** — started 2026-08-10, **firing 2026-08-24**, for
|
||||
`/home/felhom-repo.orphaned-20260810`. This drill did **not** start it and deliberately did **not**
|
||||
cancel it. The brief's end state asked for no countdown anywhere; satisfying that means choosing:
|
||||
**cancel it** (copy kept indefinitely, storage cost, no data risk) or **let it run** (copy deleted,
|
||||
irreversibly). **Doing nothing selects deletion.** → **R-307**
|
||||
|
||||
## Golden 0.212.0 — baked, published, round-trip verified
|
||||
## End state
|
||||
|
||||
`4b0a7dacc503c38732ed0a44949398639248c7fbd90758a1e4a047c21a7a15d8`, 656 611 277 B, served bytes
|
||||
re-downloaded and hashed identical. All six markers counted with `grep -F`. Token never on a command
|
||||
line; leak grep 0, believable because a planted-token control grepped 1. Drill VM reverted to `virgin`.
|
||||
Evidence: `documentation/tests/golden-0.212.0-2026-08-12/`.
|
||||
- **`demo-felhom`** — up, reporting, healthy, on the vouched pair; `repo_password` restored to the
|
||||
original (`sha c60c8bc737a6b7c6…`), escrow re-sealed and uploaded, off-site repo reachable
|
||||
(`restic snapshots` exit 0). Its recovery code was rotated by the final ceremony and
|
||||
`R_DEMO-FELHOM` updated in place (prior file backed up alongside). Planted data removed; eight
|
||||
secret-bearing files **shredded**.
|
||||
- **`demo-hp`** — untouched, reporting.
|
||||
- **`drill-r50`** — **reverted to snapshot `virgin`, powered off.**
|
||||
- **Hub** — two new retained rows (the P1 and P2 blobs), deliberately kept as the fixture proving the
|
||||
retention works. `drill-r50-0a4f9a` re-used, not duplicated: no new scratch customer.
|
||||
- **Off-site** — only demo-felhom's own repository path touched, `backup` the only mutating verb used.
|
||||
**No prune, no forget, no delete, no rename anywhere.** One snapshot added and deliberately left:
|
||||
`6ea85413`, 66 KiB, tagged `drill-retained-key-20260812` — removable by ID if you want it gone.
|
||||
|
||||
## DROPPED — named plainly
|
||||
## Honest gaps
|
||||
|
||||
- **The hub half of the naming (R-295)** — dropped first, exactly as the drop order allows. The four
|
||||
hub surfaces were NOT enumerated at `file:line`; that enumeration is still owed.
|
||||
- **The stale-golden observation (R-297)** — dropped second. Nothing published, which is the safe state.
|
||||
- **And one that was NOT droppable: R-300's live cycle.** The code shipped; the demonstration did not.
|
||||
Named here rather than shortened silently.
|
||||
|
||||
## Observations, not acted on
|
||||
|
||||
- **R-301** — the abandon countdown banner (`layout.html:143`) states the retired promise a third time
|
||||
and un-hedged. It is probably TRUE where it renders, and it renders on every page; a rebuilt box can
|
||||
have an active countdown while its store is orphaned. Not established: whether the two sentences name
|
||||
the same bytes. Left alone deliberately — this session was fenced to the orphan card.
|
||||
- The orphan card now says "we cannot determine / it depends on the key / write to us" **twice** once a
|
||||
customer clicks through — reinforcement at the decision point rather than a contradiction, but worth
|
||||
an eye if the card is revisited.
|
||||
|
||||
## Deliberately out of scope
|
||||
|
||||
The CI runs that fail with no log; the twenty facts the machines report that nothing reads; the nine
|
||||
grey claims; the storage page's separate empty-list cause (R-298); and proving a *retained* key can
|
||||
actually open an old store — the one thing the retention fix has never been shown to do.
|
||||
- **The Phase A logs did not survive** the intermediate revert to `virgin`. Every quotation in the
|
||||
audit is verbatim from the live run, but the raw files are gone. Procedural lesson, recorded.
|
||||
- The planted data reached the store via `restic` directly, not the dashboard button, because of
|
||||
R-308 — so the app-backup→unit→offsite chain went unexercised. Not what this drill measured.
|
||||
- Wall clock **≈ 1 h 13 min** against a 4–5 h envelope. Nothing was dropped; Phase C ran concurrently
|
||||
with Phase B on a different machine.
|
||||
|
||||
Reference in New Issue
Block a user