hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and
identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY
password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE
ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site
data key, which is the secret the retention was built to preserve. And because the copy happens
as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing
a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups
stay recoverable. Both demo boxes crossed that line on 2026-08-04.
- identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy
routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion.
- ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob.
- CountCurrentEscrowWithIdentity is the census of who the fix protects.
- Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone.
- Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why
the pre-existing retention test stayed green for two months asserting the mechanism.
R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises
offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and
differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity
blob retained, a changed key is "this history now depends on an older recovery code", not a loss.
R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and
regressed, each stating its own measurement; the regressed text withdraws the Re-issue
recommendation. Every self-heal refusal leaves a notification_log row with its reason. The
guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in
the alert text so the limitation travels with the number. offsite_delivery_stuck and
offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither
has a customerMessages entry, which is not a block.
R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the
restic repo password. It resets the PROVIDER password and cannot touch the repo password, which
is generated on the box. All five corrected; the staleness mark documented as precautionary. The
BEHAVIOUR stays open.
Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is
still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).
This commit is contained in:
@@ -1,3 +1,107 @@
|
||||
## v0.93.0 — the retention keeps the key it was built to keep, and three things stop lying (2026-08-04, R-198/R-197/R-196/R-192)
|
||||
|
||||
### R-198 — the superseded-escrow retention was preserving the wrong key, and the ceremony was destroying the right one
|
||||
|
||||
`host_escrow_superseded` shipped in v0.60.0 with a `blob` column — the **K-escrow**, i.e. the PBS
|
||||
datastore encryption key. `identity_blob` was added to `host_escrow` **later**, by the slice-10D
|
||||
`ALTER TABLE`, and **never to the retained table**. The offsite **restic repository password** — the
|
||||
data key for every off-site app-data snapshot — lives inside `identity_blob`. So
|
||||
`demoteCurrentEscrowTx`, whose own comment calls it *"THE ONE escrow row-copy routine"*, retained the
|
||||
whole-guest key and silently dropped the off-site data key: **precisely the secret the reinstall-orphan
|
||||
incident the feature was built for was about.**
|
||||
|
||||
**What makes it the most serious defect this project has recorded is not the missing column — it is
|
||||
which act destroys the key.** The copy happens as the new blob overwrites the old, so the destroying
|
||||
act is the **escrow ceremony**: the exact thing a rebuilt box tells its customer to run, on a card
|
||||
that promises in Hungarian that the old backups stay recoverable with their recovery code. Both demo
|
||||
boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08), and their previous repository passwords
|
||||
are now unrecoverable whether or not anyone kept the recovery codes.
|
||||
|
||||
- `host_escrow_superseded` gains `identity_blob BLOB` (in the CREATE for fresh databases **and** as an
|
||||
additive `ALTER TABLE`, following the existing idiom; tolerated on re-run, changes no existing row).
|
||||
- `demoteCurrentEscrowTx` carries it — so **both** callers are fixed by one change: the re-escrow path
|
||||
(`SaveHostEscrow`) and the host-delete demotion (`DeleteHost`).
|
||||
- `ListSupersededEscrow` reads it back and `store.HostEscrow` gains `IdentityBlob`, so a retained blob
|
||||
is reachable from Go at all. `CountCurrentEscrowWithIdentity` is the census of who the fix protects.
|
||||
- The comment above the table now records that the ruling stated there was not met, and what it cost.
|
||||
- **Nothing is backfillable and the code says so:** rows superseded before this were written without
|
||||
the identity blob and their source rows are already overwritten.
|
||||
|
||||
**Tests assert the CONSEQUENCE, not the mechanism** — the pre-existing
|
||||
`TestSaveHostEscrow_RetainsSuperseded` was green throughout, because it asserted that a retained row
|
||||
exists with the old K-blob. `TestSaveHostEscrow_RetainsIdentityBlob` asserts the retained row can still
|
||||
yield a repository password, and pins the load-bearing ordering (the identity blob is written **after**
|
||||
`SaveHostEscrow`, so the demote sees the previous generation). `TestDeleteHost_DemotesIdentityBlob`
|
||||
proves the shared routine through its **other** caller — a shared routine tested through one caller is
|
||||
how a fix gets believed on a path nobody exercised. Red-proofs: dropping the column from the copy fails
|
||||
both; fixing only the re-escrow caller fails the delete one while the re-escrow one passes.
|
||||
|
||||
### R-197 — the hub held both halves of the evidence and compared them nowhere
|
||||
|
||||
`restic_pw_sha256` is stored on the current escrow and carried to the retained row on every
|
||||
supersession. Comparing the two is what answered the 2026-08-04 recon's hardest question in one query —
|
||||
and nothing in the hub did it. demo-felhom's repository password changed on 2026-08-03, orphaning 36
|
||||
snapshots / 1.14 GB, and **no event, e-mail, card or log line said so for thirteen hours.**
|
||||
|
||||
`SaveHostEscrow` now returns the hash it replaced, and `handleHostEscrowPut` raises
|
||||
**`offsite_repo_key_changed`** (warning, operator-only, edge-triggered once per supersession) when both
|
||||
hashes are known and differ. The in-between shapes are logged rather than dropped, so *"we chose not to
|
||||
alarm"* and *"the check did not run"* never look identical. **No hash value travels** — not in the
|
||||
message, not in the details; the event names the host, the customer and the retained count.
|
||||
|
||||
**Severity chosen for the world this release creates, not the one it inherits:** before R-198 a changed
|
||||
key meant the previous history was unopenable by anyone, ever, which would have argued for `error`.
|
||||
From v0.93.0 the superseding ceremony retains the old identity blob, so the situation is *"this
|
||||
customer's off-site history now depends on an older recovery code"* — operator-actionable, not a loss.
|
||||
`warning` also routes: the dispatcher treats `info` as an intentional non-notify.
|
||||
|
||||
### R-192 (partially) — the alert describes what was measured, and a refusal leaves a trace
|
||||
|
||||
demo-hp e-mailed this daily from 2026-08-03: *"one-time password consumed 284h19m0s ago and 500
|
||||
report(s) since carry no offbox target — the credential is likely burned … Re-issue delivers a fresh
|
||||
one."* `OffsiteReportsSinceConsume` for that same call was **500**: every one of those reports carried a
|
||||
target. The message stated the precise negation of its own measurement and prescribed a remedy for a
|
||||
failure mode that had not occurred — while the self-heal correctly declined, **in complete silence**.
|
||||
|
||||
- The one stuck state is now reported as the two situations it actually covers, **burned** and
|
||||
**regressed**, each stating its own measurement and carrying its own recommendation (the regressed
|
||||
text explicitly withdraws Re-issue and points at what removes an offbox target — a guest rebuild,
|
||||
R-193). `offsite_reports_since_consume` is in the details for the first time.
|
||||
- **The counting window is named in the text.** `CountReportsOffsiteSince` reads `ORDER BY id LIMIT
|
||||
500` — the **oldest** 500 reports after the consume — so on a long-lived box the counts describe the
|
||||
start of the window, not the present. **That guard defect stays OPEN (R-192)** because its correct
|
||||
shape depends on the recovery chain that is not yet assembled (R-199/R-200/R-201); naming the window
|
||||
is how it stays visible instead of being laundered into a confident sentence.
|
||||
- **Every self-heal refusal now leaves a record** — a `notification_log` row on the operator channel
|
||||
with status `refused` and its reason, the R-182 suppressed-e-mail precedent. It rides the stuck
|
||||
event's 24 h cadence, so it sits beside the e-mail it explains rather than accumulating per tick.
|
||||
**The guard's logic is unchanged**: the set of situations in which the heal fires is what it was.
|
||||
- `offsite_delivery_stuck` and `offsite_credential_restaged` are added to `operatorOnlyEvents`. Neither
|
||||
was ever registered, and neither has a `customerMessages` entry — which is **not** a block, so a
|
||||
customer with a configured recipient was in line for an English e-mail about one-time passwords being
|
||||
*"likely burned"*. Narrowing only; the operator channel is untouched.
|
||||
|
||||
### R-196 — five comments corrected (the spec expected three)
|
||||
|
||||
`ReissueCredentials` resets the **provider account** password. It does not and cannot touch the restic
|
||||
repository password, which is generated on the box and never leaves it except sealed under R. Five
|
||||
places in the hub said otherwise: `offsite/offsite.go` (the `MarkEscrowStale` justification),
|
||||
`api/handler.go` (the F3 re-enroll comment), and three in `store/store.go` (the `stale_at` ALTER, the
|
||||
`MarkEscrowStale` doc comment, and `EscrowStatus.Stale`). **That false belief sent two separate
|
||||
investigations the wrong way in one day.** All five now say what the code does, name the correction and
|
||||
cite the recon; the staleness mark is documented as **precautionary**, not evidential. **The behaviour
|
||||
is unchanged and R-196 stays OPEN** — the false-staleness alarm it causes is a behaviour change and
|
||||
must not ride a comment-correction release.
|
||||
|
||||
### Not in this release
|
||||
|
||||
**R-199, R-200 and R-201 remain open** — the hub's blob-serving endpoints still have no client, the
|
||||
password-injection seam still has no form, and nothing in the recovery path has ever been exercised.
|
||||
**Part 5 hit its gate and the orphan card is untouched** — see R-202: a single ACK boolean can say
|
||||
whether a retained recoverable blob exists, but not whether one covers *the key this orphaned
|
||||
repository was written under*, and a conditional promise that is still capable of being false is worse
|
||||
on that surface than a hedged sentence.
|
||||
|
||||
## v0.92.0 — a customer with no machine ever bound does not alarm (2026-08-04, R-195)
|
||||
|
||||
`david` is a real prospective customer whose record was created 2026-08-01 with **no host ever bound**
|
||||
|
||||
Reference in New Issue
Block a user