From 91cabdde1b5f553fd09207bad1e38b4ed04a758b Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 4 Aug 2026 12:56:58 +0200 Subject: [PATCH] hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site data key, which is the secret the retention was built to preserve. And because the copy happens as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups stay recoverable. Both demo boxes crossed that line on 2026-08-04. - identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion. - ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob. - CountCurrentEscrowWithIdentity is the census of who the fix protects. - Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone. - Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why the pre-existing retention test stayed green for two months asserting the mechanism. R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity blob retained, a changed key is "this history now depends on an older recovery code", not a loss. R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and regressed, each stating its own measurement; the regressed text withdraws the Re-issue recommendation. Every self-heal refusal leaves a notification_log row with its reason. The guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in the alert text so the limitation travels with the number. offsite_delivery_stuck and offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither has a customerMessages entry, which is not a block. R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the restic repo password. It resets the PROVIDER password and cannot touch the repo password, which is generated on the box. All five corrected; the staleness mark documented as precautionary. The BEHAVIOUR stays open. Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202). --- CONTEXT.md | 32 +++ REUSE.md | 1 + STATUS.md | 112 ++++----- .../architecture/00-capability-map.md | 2 +- documentation/architecture/03-host-agent.md | 36 +++ documentation/backlog/OPEN-ITEMS.md | 33 ++- documentation/backlog/ROADMAP.md | 4 + hub/CHANGELOG.md | 104 ++++++++ .../api/escrow_repo_key_changed_test.go | 146 ++++++++++++ hub/internal/api/escrow_test.go | 2 +- hub/internal/api/handler.go | 79 ++++++- hub/internal/monitor/offsite_delivery.go | 120 ++++++++-- .../monitor/offsite_delivery_honesty_test.go | 223 ++++++++++++++++++ hub/internal/monitor/offsite_neverran_test.go | 2 +- hub/internal/notify/dispatcher.go | 17 ++ hub/internal/offsite/offsite.go | 31 ++- hub/internal/offsite/offsite_test.go | 4 +- hub/internal/store/customer_reset_test.go | 4 +- .../store/escrow_identity_retention_test.go | 168 +++++++++++++ hub/internal/store/escrow_superseded_test.go | 6 +- hub/internal/store/host_delete_demote_test.go | 4 +- hub/internal/store/host_delete_test.go | 2 +- hub/internal/store/store.go | 112 +++++++-- hub/internal/web/customer_delete_test.go | 4 +- hub/internal/web/customer_reset_test.go | 4 +- hub/internal/web/hosts_delete_test.go | 6 +- hub/internal/web/hosts_test.go | 2 +- hub/internal/web/pbsdr_test.go | 2 +- 28 files changed, 1122 insertions(+), 140 deletions(-) create mode 100644 hub/internal/api/escrow_repo_key_changed_test.go create mode 100644 hub/internal/monitor/offsite_delivery_honesty_test.go create mode 100644 hub/internal/store/escrow_identity_retention_test.go diff --git a/CONTEXT.md b/CONTEXT.md index ed6653f..040a9f5 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -17,6 +17,38 @@ ## Standing rulings +**S-28 — the escrow retention now covers the OFF-SITE data key, and customer-present recovery is the +accepted design, which makes that retention load-bearing (2026-08-04, R-198/R-197; hub v0.93.0).** + +**The contract changed, and this is the sentence to carry:** when an escrow blob supersedes another, +`demoteCurrentEscrowTx` retains **both** sealed keys — the K-escrow `blob` (PBS datastore key) **and** +`identity_blob` (the age-wrapped bundle carrying the restic **repository** password). Until v0.93.0 it +retained only the first, so the retention preserved the whole-guest key and destroyed the off-site data +key, and **the destroying act was the escrow ceremony a rebuilt box asks its customer to run.** + +*Why this is load-bearing rather than tidy:* the operator ruled on 2026-08-04 that **candidate (c) is +refused — no repository password is retained on the Proxmox host.** With no host-retained copy, the +**customer-present recovery path is the only way back from a rebuild**, and it runs entirely through +the retained identity blob. A session that touches escrow custody is touching the single copy. + +*Facts a future session needs before touching this area:* +- **The ordering is load-bearing and invisible from the routine.** `SaveHostDRBundle` writes + `identity_blob` **after** `SaveHostEscrow` returns (the escrow PUT, `api/handler.go`), so at demote + time `host_escrow` still holds the OLD identity blob. Invert that and the retained bytes become the + NEW blob filed under the OLD hash — recoverable-looking and wrong. Pinned by + `TestSaveHostEscrow_RetainsIdentityBlob`. +- **`demoteCurrentEscrowTx` is shared by re-escrow and host-delete.** Both have a test; fixing one + caller instead of the routine is a demonstrated red-proof, not a hypothetical. +- **Nothing was backfilled and nothing could be** — pre-v0.93.0 retained rows have no identity blob and + their source rows are already overwritten. Both demo boxes' pre-2026-08-04 repository passwords are + gone permanently, recovery code or not. +- **A retained key is not a recovery.** The hub's blob-serving endpoints have no client (R-199), the + injection seam has no form (R-200), and the path has never been exercised (R-201). v0.93.0 makes the + key survive; it does not hand it back. +- **`offsite_repo_key_changed`** (R-197, warning, operator-only, edge-triggered on a supersession where + both hashes are known and differ) is the evidential signal that a box's off-site data key moved. It + carries **no hash value**. `MarkEscrowStale` is **precautionary**, not evidential — see S-26(a). + **S-27 — a customer with NO machine ever bound is UNKNOWN, silently; one that was bound and went quiet still alarms (2026-08-04, R-195; hub v0.92.0).** Operator ruling, implemented as `store.HasEverBoundHost` (live `hosts` row OR `host_deletions` tombstone) consulted once at the top of diff --git a/REUSE.md b/REUSE.md index d37ae96..da5f4e9 100644 --- a/REUSE.md +++ b/REUSE.md @@ -78,6 +78,7 @@ | `(*Store).HasEverBoundHost` (v0.92.0, R-195) | hub/internal/store/store.go | `(customerID) (bool, error)` | Any verdict that must not fire for a customer with **no machine ever bound** — "was anything ever expected of this customer" | `hosts` row **OR** `host_deletions` tombstone. **NOT a liveness check and never a substitute for one:** a box that was bound and went silent returns `true` and must keep alarming — that is the case any change here breaks first (pinned by `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`). Callers **fail OPEN** on its error: an unreadable binding must never SUPPRESS an alarm. Do **not** re-derive this from report presence — `store.GetCustomers()` (and therefore the staleness checker's `down` state) is a query over `reports`, so a never-reported customer has no state at all, which is exactly how the daily false alarm reached `david`. | | `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). | | `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" }}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `