hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196)
gates / gates (push) Successful in 7s

R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and
identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY
password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE
ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site
data key, which is the secret the retention was built to preserve. And because the copy happens
as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing
a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups
stay recoverable. Both demo boxes crossed that line on 2026-08-04.

  - identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy
    routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion.
  - ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob.
  - CountCurrentEscrowWithIdentity is the census of who the fix protects.
  - Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone.
  - Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why
    the pre-existing retention test stayed green for two months asserting the mechanism.

R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises
offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and
differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity
blob retained, a changed key is "this history now depends on an older recovery code", not a loss.

R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and
regressed, each stating its own measurement; the regressed text withdraws the Re-issue
recommendation. Every self-heal refusal leaves a notification_log row with its reason. The
guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in
the alert text so the limitation travels with the number. offsite_delivery_stuck and
offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither
has a customerMessages entry, which is not a block.

R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the
restic repo password. It resets the PROVIDER password and cannot touch the repo password, which
is generated on the box. All five corrected; the staleness mark documented as precautionary. The
BEHAVIOUR stays open.

Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is
still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).
This commit is contained in:
2026-08-04 12:56:58 +02:00
parent 3f2b7bc023
commit 91cabdde1b
28 changed files with 1122 additions and 140 deletions
+104
View File
@@ -1,3 +1,107 @@
## v0.93.0 — the retention keeps the key it was built to keep, and three things stop lying (2026-08-04, R-198/R-197/R-196/R-192)
### R-198 — the superseded-escrow retention was preserving the wrong key, and the ceremony was destroying the right one
`host_escrow_superseded` shipped in v0.60.0 with a `blob` column — the **K-escrow**, i.e. the PBS
datastore encryption key. `identity_blob` was added to `host_escrow` **later**, by the slice-10D
`ALTER TABLE`, and **never to the retained table**. The offsite **restic repository password** — the
data key for every off-site app-data snapshot — lives inside `identity_blob`. So
`demoteCurrentEscrowTx`, whose own comment calls it *"THE ONE escrow row-copy routine"*, retained the
whole-guest key and silently dropped the off-site data key: **precisely the secret the reinstall-orphan
incident the feature was built for was about.**
**What makes it the most serious defect this project has recorded is not the missing column — it is
which act destroys the key.** The copy happens as the new blob overwrites the old, so the destroying
act is the **escrow ceremony**: the exact thing a rebuilt box tells its customer to run, on a card
that promises in Hungarian that the old backups stay recoverable with their recovery code. Both demo
boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08), and their previous repository passwords
are now unrecoverable whether or not anyone kept the recovery codes.
- `host_escrow_superseded` gains `identity_blob BLOB` (in the CREATE for fresh databases **and** as an
additive `ALTER TABLE`, following the existing idiom; tolerated on re-run, changes no existing row).
- `demoteCurrentEscrowTx` carries it — so **both** callers are fixed by one change: the re-escrow path
(`SaveHostEscrow`) and the host-delete demotion (`DeleteHost`).
- `ListSupersededEscrow` reads it back and `store.HostEscrow` gains `IdentityBlob`, so a retained blob
is reachable from Go at all. `CountCurrentEscrowWithIdentity` is the census of who the fix protects.
- The comment above the table now records that the ruling stated there was not met, and what it cost.
- **Nothing is backfillable and the code says so:** rows superseded before this were written without
the identity blob and their source rows are already overwritten.
**Tests assert the CONSEQUENCE, not the mechanism** — the pre-existing
`TestSaveHostEscrow_RetainsSuperseded` was green throughout, because it asserted that a retained row
exists with the old K-blob. `TestSaveHostEscrow_RetainsIdentityBlob` asserts the retained row can still
yield a repository password, and pins the load-bearing ordering (the identity blob is written **after**
`SaveHostEscrow`, so the demote sees the previous generation). `TestDeleteHost_DemotesIdentityBlob`
proves the shared routine through its **other** caller — a shared routine tested through one caller is
how a fix gets believed on a path nobody exercised. Red-proofs: dropping the column from the copy fails
both; fixing only the re-escrow caller fails the delete one while the re-escrow one passes.
### R-197 — the hub held both halves of the evidence and compared them nowhere
`restic_pw_sha256` is stored on the current escrow and carried to the retained row on every
supersession. Comparing the two is what answered the 2026-08-04 recon's hardest question in one query —
and nothing in the hub did it. demo-felhom's repository password changed on 2026-08-03, orphaning 36
snapshots / 1.14 GB, and **no event, e-mail, card or log line said so for thirteen hours.**
`SaveHostEscrow` now returns the hash it replaced, and `handleHostEscrowPut` raises
**`offsite_repo_key_changed`** (warning, operator-only, edge-triggered once per supersession) when both
hashes are known and differ. The in-between shapes are logged rather than dropped, so *"we chose not to
alarm"* and *"the check did not run"* never look identical. **No hash value travels** — not in the
message, not in the details; the event names the host, the customer and the retained count.
**Severity chosen for the world this release creates, not the one it inherits:** before R-198 a changed
key meant the previous history was unopenable by anyone, ever, which would have argued for `error`.
From v0.93.0 the superseding ceremony retains the old identity blob, so the situation is *"this
customer's off-site history now depends on an older recovery code"* — operator-actionable, not a loss.
`warning` also routes: the dispatcher treats `info` as an intentional non-notify.
### R-192 (partially) — the alert describes what was measured, and a refusal leaves a trace
demo-hp e-mailed this daily from 2026-08-03: *"one-time password consumed 284h19m0s ago and 500
report(s) since carry no offbox target — the credential is likely burned … Re-issue delivers a fresh
one."* `OffsiteReportsSinceConsume` for that same call was **500**: every one of those reports carried a
target. The message stated the precise negation of its own measurement and prescribed a remedy for a
failure mode that had not occurred — while the self-heal correctly declined, **in complete silence**.
- The one stuck state is now reported as the two situations it actually covers, **burned** and
**regressed**, each stating its own measurement and carrying its own recommendation (the regressed
text explicitly withdraws Re-issue and points at what removes an offbox target — a guest rebuild,
R-193). `offsite_reports_since_consume` is in the details for the first time.
- **The counting window is named in the text.** `CountReportsOffsiteSince` reads `ORDER BY id LIMIT
500` — the **oldest** 500 reports after the consume — so on a long-lived box the counts describe the
start of the window, not the present. **That guard defect stays OPEN (R-192)** because its correct
shape depends on the recovery chain that is not yet assembled (R-199/R-200/R-201); naming the window
is how it stays visible instead of being laundered into a confident sentence.
- **Every self-heal refusal now leaves a record** — a `notification_log` row on the operator channel
with status `refused` and its reason, the R-182 suppressed-e-mail precedent. It rides the stuck
event's 24 h cadence, so it sits beside the e-mail it explains rather than accumulating per tick.
**The guard's logic is unchanged**: the set of situations in which the heal fires is what it was.
- `offsite_delivery_stuck` and `offsite_credential_restaged` are added to `operatorOnlyEvents`. Neither
was ever registered, and neither has a `customerMessages` entry — which is **not** a block, so a
customer with a configured recipient was in line for an English e-mail about one-time passwords being
*"likely burned"*. Narrowing only; the operator channel is untouched.
### R-196 — five comments corrected (the spec expected three)
`ReissueCredentials` resets the **provider account** password. It does not and cannot touch the restic
repository password, which is generated on the box and never leaves it except sealed under R. Five
places in the hub said otherwise: `offsite/offsite.go` (the `MarkEscrowStale` justification),
`api/handler.go` (the F3 re-enroll comment), and three in `store/store.go` (the `stale_at` ALTER, the
`MarkEscrowStale` doc comment, and `EscrowStatus.Stale`). **That false belief sent two separate
investigations the wrong way in one day.** All five now say what the code does, name the correction and
cite the recon; the staleness mark is documented as **precautionary**, not evidential. **The behaviour
is unchanged and R-196 stays OPEN** — the false-staleness alarm it causes is a behaviour change and
must not ride a comment-correction release.
### Not in this release
**R-199, R-200 and R-201 remain open** — the hub's blob-serving endpoints still have no client, the
password-injection seam still has no form, and nothing in the recovery path has ever been exercised.
**Part 5 hit its gate and the orphan card is untouched** — see R-202: a single ACK boolean can say
whether a retained recoverable blob exists, but not whether one covers *the key this orphaned
repository was written under*, and a conditional promise that is still capable of being false is worse
on that surface than a hedged sentence.
## v0.92.0 — a customer with no machine ever bound does not alarm (2026-08-04, R-195)
`david` is a real prospective customer whose record was created 2026-08-01 with **no host ever bound**