hub v0.93.0: the retention keeps the key it was built to keep (R-198) + three honesty fixes (R-197, R-192, R-196)
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
R-198 — host_escrow_superseded shipped with `blob` (the K-escrow / PBS datastore key) and
identity_blob was added to host_escrow LATER, never here. The offsite restic REPOSITORY
password lives in identity_blob. So demoteCurrentEscrowTx -- whose own comment calls it "THE
ONE escrow row-copy routine" -- retained the whole-guest key and silently dropped the off-site
data key, which is the secret the retention was built to preserve. And because the copy happens
as the new blob overwrites the old, the destroying act was the ESCROW CEREMONY: the exact thing
a rebuilt box tells its customer to run, on a card promising in Hungarian that the old backups
stay recoverable. Both demo boxes crossed that line on 2026-08-04.
- identity_blob added to the table (CREATE + additive ALTER) and carried in the shared copy
routine, so BOTH callers are fixed at once: re-escrow and host-delete demotion.
- ListSupersededEscrow reads it back; store.HostEscrow gains IdentityBlob.
- CountCurrentEscrowWithIdentity is the census of who the fix protects.
- Nothing is backfillable: pre-v0.93.0 retained rows have no blob and their sources are gone.
- Tests assert the CONSEQUENCE (a retained row can still yield a repo password), which is why
the pre-existing retention test stayed green for two months asserting the mechanism.
R-197 — SaveHostEscrow returns the hash it replaced; the escrow PUT raises
offsite_repo_key_changed (warning, operator-only, edge-triggered) when both hashes are known and
differ. No hash value travels. Severity chosen for the world v0.93.0 creates: with the identity
blob retained, a changed key is "this history now depends on an older recovery code", not a loss.
R-192 (half) — the stuck alert now reports the two shapes it actually covers, burned and
regressed, each stating its own measurement; the regressed text withdraws the Re-issue
recommendation. Every self-heal refusal leaves a notification_log row with its reason. The
guard's logic is unchanged; its 500-oldest-reports scoping stays OPEN and the window is named in
the alert text so the limitation travels with the number. offsite_delivery_stuck and
offsite_credential_restaged are added to operatorOnlyEvents -- neither was registered and neither
has a customerMessages entry, which is not a block.
R-196 — five comments (not the three the spec expected) claimed ReissueCredentials rotates the
restic repo password. It resets the PROVIDER password and cannot touch the repo password, which
is generated on the box. All five corrected; the staleness mark documented as precautionary. The
BEHAVIOUR stays open.
Not in this release: R-199, R-200, R-201 remain open -- the chain that hands the key back is
still unassembled. Part 5 hit its gate; the orphan card is untouched (R-202).
This commit is contained in:
+32
@@ -17,6 +17,38 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-28 — the escrow retention now covers the OFF-SITE data key, and customer-present recovery is the
|
||||
accepted design, which makes that retention load-bearing (2026-08-04, R-198/R-197; hub v0.93.0).**
|
||||
|
||||
**The contract changed, and this is the sentence to carry:** when an escrow blob supersedes another,
|
||||
`demoteCurrentEscrowTx` retains **both** sealed keys — the K-escrow `blob` (PBS datastore key) **and**
|
||||
`identity_blob` (the age-wrapped bundle carrying the restic **repository** password). Until v0.93.0 it
|
||||
retained only the first, so the retention preserved the whole-guest key and destroyed the off-site data
|
||||
key, and **the destroying act was the escrow ceremony a rebuilt box asks its customer to run.**
|
||||
|
||||
*Why this is load-bearing rather than tidy:* the operator ruled on 2026-08-04 that **candidate (c) is
|
||||
refused — no repository password is retained on the Proxmox host.** With no host-retained copy, the
|
||||
**customer-present recovery path is the only way back from a rebuild**, and it runs entirely through
|
||||
the retained identity blob. A session that touches escrow custody is touching the single copy.
|
||||
|
||||
*Facts a future session needs before touching this area:*
|
||||
- **The ordering is load-bearing and invisible from the routine.** `SaveHostDRBundle` writes
|
||||
`identity_blob` **after** `SaveHostEscrow` returns (the escrow PUT, `api/handler.go`), so at demote
|
||||
time `host_escrow` still holds the OLD identity blob. Invert that and the retained bytes become the
|
||||
NEW blob filed under the OLD hash — recoverable-looking and wrong. Pinned by
|
||||
`TestSaveHostEscrow_RetainsIdentityBlob`.
|
||||
- **`demoteCurrentEscrowTx` is shared by re-escrow and host-delete.** Both have a test; fixing one
|
||||
caller instead of the routine is a demonstrated red-proof, not a hypothetical.
|
||||
- **Nothing was backfilled and nothing could be** — pre-v0.93.0 retained rows have no identity blob and
|
||||
their source rows are already overwritten. Both demo boxes' pre-2026-08-04 repository passwords are
|
||||
gone permanently, recovery code or not.
|
||||
- **A retained key is not a recovery.** The hub's blob-serving endpoints have no client (R-199), the
|
||||
injection seam has no form (R-200), and the path has never been exercised (R-201). v0.93.0 makes the
|
||||
key survive; it does not hand it back.
|
||||
- **`offsite_repo_key_changed`** (R-197, warning, operator-only, edge-triggered on a supersession where
|
||||
both hashes are known and differ) is the evidential signal that a box's off-site data key moved. It
|
||||
carries **no hash value**. `MarkEscrowStale` is **precautionary**, not evidential — see S-26(a).
|
||||
|
||||
**S-27 — a customer with NO machine ever bound is UNKNOWN, silently; one that was bound and went quiet
|
||||
still alarms (2026-08-04, R-195; hub v0.92.0).** Operator ruling, implemented as
|
||||
`store.HasEverBoundHost` (live `hosts` row OR `host_deletions` tombstone) consulted once at the top of
|
||||
|
||||
@@ -78,6 +78,7 @@
|
||||
| `(*Store).HasEverBoundHost` (v0.92.0, R-195) | hub/internal/store/store.go | `(customerID) (bool, error)` | Any verdict that must not fire for a customer with **no machine ever bound** — "was anything ever expected of this customer" | `hosts` row **OR** `host_deletions` tombstone. **NOT a liveness check and never a substitute for one:** a box that was bound and went silent returns `true` and must keep alarming — that is the case any change here breaks first (pinned by `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`). Callers **fail OPEN** on its error: an unreadable binding must never SUPPRESS an alarm. Do **not** re-derive this from report presence — `store.GetCustomers()` (and therefore the staleness checker's `down` state) is a query over `reports`, so a never-reported customer has no state at all, which is exactly how the daily false alarm reached `david`. |
|
||||
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
|
||||
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
|
||||
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts** — `blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** `SaveHostDRBundle` writes `identity_blob` AFTER `SaveHostEscrow` returns, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
|
||||
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
|
||||
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
|
||||
| `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |
|
||||
|
||||
@@ -20,40 +20,33 @@ Proven end to end on real hardware.
|
||||
|
||||
## What's broken
|
||||
|
||||
- **Rebuilding a machine throws away its off-site backup HISTORY — and that is worse than we thought
|
||||
yesterday.** We spent today establishing what a rebuild actually destroys, and the answer changes the
|
||||
picture. There are three secrets involved. The one we have been talking about — the password to the
|
||||
storage provider — is the one you can re-issue any time, and we did. **The one that matters is the
|
||||
key that encrypts the backups themselves. The machine invents that key, and a rebuilt machine
|
||||
invents a brand-new one**, so the old backups can no longer be opened by the machine that wrote
|
||||
them.
|
||||
**Both demo machines did exactly this, and one of them did it silently.** demo-hp lost access to
|
||||
15 backups (41 MB). demo-felhom — the one we called lucky, because its off-site copy came back on
|
||||
its own in 76 seconds — **lost access to 36 backups (1.14 GB), and nothing said a word for
|
||||
thirteen hours.** The luck restored the delivery, not the history.
|
||||
Nothing has been deleted, and this was measured without touching either machine or any password.
|
||||
**The next scheduled off-site run is tomorrow at 04:15, and it will refuse rather than quietly start
|
||||
a new history** — the machine already knows how to recognise this and asks before resetting.
|
||||
**There is a decision here for you — see "Waiting on you".** *(R-193, R-192, and two new items)*
|
||||
- **And the safety net we believed was under all of this is not there.** We traced the whole path today —
|
||||
from a dead machine to reopened backups — and it does not join up. The important part is not that
|
||||
several steps are manual; it is this: **the central hub keeps the old sealed key when a machine
|
||||
re-seals, and we found it keeps the wrong one.** It keeps the key for the whole-machine backups and
|
||||
**not** the key for the off-site file backups — the one this entire problem is about. So the moment
|
||||
a rebuilt machine is asked to make a new recovery code, which is exactly what it asks the customer to
|
||||
do, **the last copy of the old key is gone.** Both demo machines crossed that line yesterday morning.
|
||||
**This means the 51 orphaned backups were beyond reach even if you had kept the recovery codes** —
|
||||
which is a relief in one narrow sense and much worse in every other. It also means the customer is
|
||||
currently told, in Hungarian on their own screen, that their old backups "may later be restorable
|
||||
with the matching recovery code". That sentence is not true today. It is a small fix — one missing
|
||||
column — and it must land before anything else here. *(R-198)*
|
||||
- **Nothing in the recovery path has ever been performed.** Making a recovery code is proven, live,
|
||||
with a real customer. **Using one is not.** No sealed package has ever been handed back to a machine
|
||||
(the hub has the buttons for it; nothing anywhere presses them), no recovered key has ever been put
|
||||
back, no old backup store has ever been reopened, and no file has ever been restored from a recovered
|
||||
key. There is a form missing, a step missing, and a connection missing between two parts of the
|
||||
system. We designed the exercise that would prove it end to end — see "Waiting on you".
|
||||
*(R-199, R-200, R-201)*
|
||||
- **Rebuilding a machine still throws away its off-site backup HISTORY.** The machine invents the key
|
||||
that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one, so the old
|
||||
backups can no longer be opened by the machine that wrote them. Both demo machines did this on 3–4
|
||||
August — demo-hp lost access to 15 backups (41 MB), demo-felhom to 36 (1.14 GB), and **the second one
|
||||
happened in silence for thirteen hours**. From today that silence is fixed (below) and the old key
|
||||
survives the ceremony (below), but **the rebuild itself still starts a fresh history**, and getting
|
||||
the old one back is a path nobody has walked. *(R-193)*
|
||||
- *(fixed 4 Aug)* ~~The safety net we believed was under all of this is not there.~~ **The central hub
|
||||
was keeping the wrong key, and the fix shipped the same day.** When a machine re-seals its backups
|
||||
under a new recovery code, the hub keeps the old sealed package — but it was keeping the one for the
|
||||
whole-machine backups and **not** the one for the off-site file backups, which is the whole point. And
|
||||
because the swap happens in one motion, **the act that destroyed the old key was the recovery-code
|
||||
ceremony itself** — the exact thing a rebuilt machine asks its customer to do, on a screen promising
|
||||
the old backups stay recoverable. Both keys are now kept. **What this cannot undo:** anything
|
||||
superseded before today is gone for good, which includes both demo machines' pre-4-August keys — so
|
||||
those 51 orphaned backups were beyond reach even if the recovery codes had been kept. *(R-198)*
|
||||
- **Nothing in the recovery path has ever been performed, and that has not changed today.** Making a
|
||||
recovery code is proven, live, with a real customer. **Using one is not.** No sealed package has ever
|
||||
been handed back to a machine, no recovered key has ever been put back, no old backup store has ever
|
||||
been reopened. Today's work makes the key **survive**; it does not build the path that hands it back.
|
||||
That is what the proof exercise is for. *(R-199, R-200, R-201)*
|
||||
- **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
|
||||
says the old backups may later be restorable with the matching recovery code. From today that is true
|
||||
for machines that re-seal from now on and **false for anything already orphaned** — and the machine
|
||||
cannot tell which case it is in without the recovery path above. We deliberately did **not** patch the
|
||||
sentence: a conditional promise that can still be wrong is worse on that screen than a vague one.
|
||||
*(R-202)*
|
||||
- *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine
|
||||
and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer
|
||||
asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing
|
||||
@@ -96,9 +89,10 @@ Proven end to end on real hardware.
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** waiting on your answer about the backup key (above). Two small honesty fixes are ready to
|
||||
go the moment you say so — making a changed backup key raise an alarm, and correcting an email that
|
||||
currently states the opposite of what it measured. Neither needs a decision.
|
||||
- **Now:** the proof exercise — wipe a demo machine and recover it with a saved recovery code. It is
|
||||
designed and waiting on your go-ahead. Both honesty fixes shipped today: a changed backup key now
|
||||
raises an alarm on the day, and the email that stated the opposite of what it measured now describes
|
||||
what it actually saw.
|
||||
- **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
|
||||
refuse and ask, rather than start fresh — but that is a prediction from the code, not something
|
||||
anyone has watched happen.
|
||||
@@ -109,24 +103,11 @@ Proven end to end on real hardware.
|
||||
|
||||
## Waiting on you
|
||||
|
||||
- **A real decision, and it is the important one on this page.** Every machine is going to be
|
||||
reinstalled at some point, and each reinstall throws away the key that opens its off-site backups.
|
||||
There are only two honest answers, and picking either is fine — picking neither is what leaves it
|
||||
where it is.
|
||||
**(a) Keep a copy of that key on the Proxmox host**, which survives a rebuild. The machinery for
|
||||
this already exists and is barely any work. The price: the key would then sit on a machine you have
|
||||
root on, so "only the customer's recovery code can open these backups" stops being true in the
|
||||
strict sense. It was never quite true anyway — you hold root everywhere — but this makes a real
|
||||
second copy, and copies are exactly what we are already uneasy about elsewhere.
|
||||
**(b) Leave the key sealed under the customer's recovery code**, accept that every rebuild starts a
|
||||
fresh off-site history, and make sure the system says so loudly on the day it happens instead of
|
||||
never. Old backups are kept, not deleted, but they keep consuming the customer's off-site space.
|
||||
**We recommend doing the "says so loudly" part immediately regardless** — it is cheap, it costs
|
||||
nothing else, and it is the piece whose absence let a gigabyte of history go quiet for thirteen
|
||||
hours. What we are asking you to choose is (a). *(R-193; the full reasoning is in the findings
|
||||
document)*
|
||||
**Correction to (b), from today:** "old backups are kept" is true, but "openable later with the
|
||||
recovery code" is **not** — see R-198 above. Option (b) is only honest once that is fixed.
|
||||
- *(decided 4 Aug — recorded here until the work it unblocks is done)* You chose **not** to keep a
|
||||
copy of the backup key on the Proxmox host. That is a coherent choice and it has a consequence worth
|
||||
restating: **the customer's own recovery code is now the only route back from a rebuild**, so the
|
||||
sealed package the hub keeps is the single copy. Today's fix is what makes that true rather than
|
||||
wishful — until this morning the ceremony destroyed it. *(R-193)*
|
||||
- **Your ruling about the recovery screen has been priced, and it can be built.** You said: a freshly
|
||||
installed machine that finds a sealed package waiting should say so loudly, offer the customer a box
|
||||
to type their recovery code into, and show what would come back before doing anything. **All of it is
|
||||
@@ -146,11 +127,12 @@ Proven end to end on real hardware.
|
||||
marked file comes back byte for byte. About half a day, of which under an hour needs you. The failure
|
||||
we are watching for is precise: **if the recovered machine reports one backup instead of the ones we
|
||||
put there, it started a fresh history and the proof failed** — "the store opened" is not good enough.
|
||||
It costs that machine's contents if it goes wrong, which is acceptable there. **Do R-198 first**, or
|
||||
the exercise walks a path that is missing a piece. *(R-201)*
|
||||
- **The orphaned backups still on the storage box.** About 1.2 GB across the two demo machines, sitting
|
||||
in set-aside stores nobody can open and nothing prunes, using the 50 GB allowance. We did not touch
|
||||
them. **Delete, or leave?** *(R-193)*
|
||||
It costs that machine's contents if it goes wrong, which is acceptable there. **You said yes, after
|
||||
R-198 — and R-198 shipped this morning, so this is ready to run on your word.** *(R-201)*
|
||||
- **The orphaned backups on the storage box — you said delete, and it is still owed.** About 1.2 GB
|
||||
across the two demo machines, in set-aside stores nobody can open and nothing prunes. Deliberately not
|
||||
done today: erasing data on the storage endpoint is not something that should ride along with a
|
||||
database change, so it wants its own session. *(R-193)*
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
||||
@@ -160,11 +142,11 @@ Proven end to end on real hardware.
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-04 (latest)** — Traced the whole recovery path end to end and it does not join up. The hub
|
||||
keeps the wrong sealed key when a machine re-seals, so a rebuilt machine's old off-site key is lost the
|
||||
moment the customer is asked for a new recovery code. Nothing in the recovery path has ever been
|
||||
performed. Your ruling about the recovery screen is priced and buildable. The proof exercise is
|
||||
designed and awaiting your go-ahead. *(R-198, R-199, R-200, R-201, R-193)*
|
||||
- **2026-08-04 (latest)** — **Fixed the worst of it.** The hub now keeps the off-site backup key when a
|
||||
machine re-seals, instead of only the whole-machine one — four lines of database change that stop the
|
||||
recovery ceremony from destroying the thing it exists to protect. A changed backup key now raises an
|
||||
alarm the same day. The daily email that described the opposite of what it measured now states what it
|
||||
saw, and a refusal to auto-repair leaves a record instead of silence. *(R-198, R-197, R-192, R-196)*
|
||||
- **2026-08-04 (later)** — Established what a machine rebuild actually destroys: not the storage
|
||||
password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost
|
||||
access to their off-site history, one of them silently. A decision is now waiting on you. The daily
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -378,6 +378,42 @@ nicety.
|
||||
**loses Felhom's backup guarantees** — no PBS verify / monitoring on storage we can't reach. An
|
||||
honest signup-time tradeoff, not a hidden one.
|
||||
|
||||
#### Custody across generations — what the hub retains when a blob is superseded (hub v0.60.0; **contract corrected hub v0.93.0, R-198**)
|
||||
|
||||
A ceremony that seals a *different* passphrase does not destroy the blob it replaces: the hub copies
|
||||
the current `host_escrow` row into `host_escrow_superseded` **before** overwriting it, so a previous
|
||||
generation stays openable with the recovery code that sealed it. The same routine
|
||||
(`demoteCurrentEscrowTx`) is used when a host is deleted, which **demotes** custody rather than
|
||||
destroying it. The hub never decrypts either table.
|
||||
|
||||
**The contract as originally shipped was incomplete, and the gap is worth recording because it was
|
||||
invisible for two months.** An escrow generation is **two** sealed artifacts — the K-escrow `blob`
|
||||
(the PBS datastore key `K`) and `identity_blob` (the age-wrapped `IdentityBundle`, which carries the
|
||||
offsite **restic repository password** as well as the tunnel/PBS tokens and the WG key). Retention
|
||||
covered only the first. So a supersession preserved the whole-guest key and destroyed the off-site
|
||||
data key — and because the copy happens as the new blob lands, **the destroying act was the ceremony
|
||||
itself**, i.e. the step a rebuilt box asks its customer to perform. Corrected in hub v0.93.0: the
|
||||
retained row carries both.
|
||||
|
||||
**Two consequences that belong in this doc rather than a changelog:**
|
||||
|
||||
- **Generational custody is now the recovery precondition for the offsite tier.** The operator ruled
|
||||
on 2026-08-04 that no repository password is retained on the Proxmox host (R-193 candidate (c),
|
||||
refused). With no host-side copy, a rebuilt box's only route back to its previous off-site history
|
||||
is *the customer's older recovery code opening the retained identity blob* — which exists only if
|
||||
the supersession that replaced it happened on hub v0.93.0 or later.
|
||||
- **Retention is custody, not recovery, and this doc must not be read as claiming otherwise.** The
|
||||
hub's restore-mode serving of these blobs (§ slice 10D) has **no client** in the agent, the
|
||||
controller's recovered-password injection seam has **no user interface**, and no part of the chain
|
||||
has ever been exercised outside a unit test — the one live identity round-trip (2026-06-10)
|
||||
predates the `ResticRepoPassword` field. Tracked as R-199/R-200/R-201; evidence
|
||||
`documentation/audits/RECON-offsite-dr-chain-2026-08-04.md`.
|
||||
|
||||
**The honesty property above is unchanged and now has a second edge:** losing `R` and the box makes
|
||||
the offsite backups unrecoverable by anyone — *and* an older generation's data is recoverable only
|
||||
with **that generation's** recovery code, which the customer must still hold. A current recovery code
|
||||
does not open a previous generation's blob.
|
||||
|
||||
## 9. Provisioning & DR flows
|
||||
|
||||
**Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -57,6 +57,10 @@
|
||||
| R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. **Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17):** the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above |
|
||||
| R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/<id>` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED + PROVEN-LIVE agent v0.89.0** (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → `converged state=adopted` in ~3 s, ACLs restored, offsite active). Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. |
|
||||
| R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** |
|
||||
| R-198 | **The superseded-escrow retention kept the K-escrow and dropped the identity blob** — so it preserved the PBS key and destroyed the offsite restic repository password, with the escrow CEREMONY as the destroying act | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | Corrects the escrow-ceremony capability-map row's v0.60.0 retention claim (was struck through 2026-08-04, now restated as what ships). One column + one SELECT list, in the shared `demoteCurrentEscrowTx`, so both callers are fixed at once. Nothing backfillable. **Load-bearing, not tidy:** with R-193(c) refused, customer-present recovery is the only route back from a rebuild and it runs entirely through the retained identity blob |
|
||||
| R-197 | **The hub stored both halves of "did this box's offsite data key change" and compared them nowhere** | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | New `offsite_repo_key_changed` (warning, operator-only, edge-triggered, no hash values). Flips nothing in the capability map — it adds a signal, not a capability |
|
||||
| R-196 | **Five comments claimed `ReissueCredentials` rotates the restic repo password** — it resets the PROVIDER password and cannot touch the repo password | S | **comments corrected (hub v0.93.0); the BEHAVIOUR stays open** | The false-staleness alarm is a behaviour change and must not ride a comment-correction release — and it matters MORE now, because under R-198 an unnecessary ceremony supersedes a blob rather than being harmless bookkeeping |
|
||||
| R-202 | **The orphan card promises recoverability unconditionally**, which after R-198 is true going forward and false for anything already orphaned | S | **gate hit 2026-08-04 — card untouched, sentence still live** | Blocked on knowing which escrow generation an orphaned repo belongs to (R-199/R-201). A single ACK boolean can say a retained recoverable blob EXISTS but not that one COVERS this repo; a conditional promise that can still be false is worse on that surface than a hedged one |
|
||||
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
|
||||
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
|
||||
| R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |
|
||||
|
||||
@@ -1,3 +1,107 @@
|
||||
## v0.93.0 — the retention keeps the key it was built to keep, and three things stop lying (2026-08-04, R-198/R-197/R-196/R-192)
|
||||
|
||||
### R-198 — the superseded-escrow retention was preserving the wrong key, and the ceremony was destroying the right one
|
||||
|
||||
`host_escrow_superseded` shipped in v0.60.0 with a `blob` column — the **K-escrow**, i.e. the PBS
|
||||
datastore encryption key. `identity_blob` was added to `host_escrow` **later**, by the slice-10D
|
||||
`ALTER TABLE`, and **never to the retained table**. The offsite **restic repository password** — the
|
||||
data key for every off-site app-data snapshot — lives inside `identity_blob`. So
|
||||
`demoteCurrentEscrowTx`, whose own comment calls it *"THE ONE escrow row-copy routine"*, retained the
|
||||
whole-guest key and silently dropped the off-site data key: **precisely the secret the reinstall-orphan
|
||||
incident the feature was built for was about.**
|
||||
|
||||
**What makes it the most serious defect this project has recorded is not the missing column — it is
|
||||
which act destroys the key.** The copy happens as the new blob overwrites the old, so the destroying
|
||||
act is the **escrow ceremony**: the exact thing a rebuilt box tells its customer to run, on a card
|
||||
that promises in Hungarian that the old backups stay recoverable with their recovery code. Both demo
|
||||
boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08), and their previous repository passwords
|
||||
are now unrecoverable whether or not anyone kept the recovery codes.
|
||||
|
||||
- `host_escrow_superseded` gains `identity_blob BLOB` (in the CREATE for fresh databases **and** as an
|
||||
additive `ALTER TABLE`, following the existing idiom; tolerated on re-run, changes no existing row).
|
||||
- `demoteCurrentEscrowTx` carries it — so **both** callers are fixed by one change: the re-escrow path
|
||||
(`SaveHostEscrow`) and the host-delete demotion (`DeleteHost`).
|
||||
- `ListSupersededEscrow` reads it back and `store.HostEscrow` gains `IdentityBlob`, so a retained blob
|
||||
is reachable from Go at all. `CountCurrentEscrowWithIdentity` is the census of who the fix protects.
|
||||
- The comment above the table now records that the ruling stated there was not met, and what it cost.
|
||||
- **Nothing is backfillable and the code says so:** rows superseded before this were written without
|
||||
the identity blob and their source rows are already overwritten.
|
||||
|
||||
**Tests assert the CONSEQUENCE, not the mechanism** — the pre-existing
|
||||
`TestSaveHostEscrow_RetainsSuperseded` was green throughout, because it asserted that a retained row
|
||||
exists with the old K-blob. `TestSaveHostEscrow_RetainsIdentityBlob` asserts the retained row can still
|
||||
yield a repository password, and pins the load-bearing ordering (the identity blob is written **after**
|
||||
`SaveHostEscrow`, so the demote sees the previous generation). `TestDeleteHost_DemotesIdentityBlob`
|
||||
proves the shared routine through its **other** caller — a shared routine tested through one caller is
|
||||
how a fix gets believed on a path nobody exercised. Red-proofs: dropping the column from the copy fails
|
||||
both; fixing only the re-escrow caller fails the delete one while the re-escrow one passes.
|
||||
|
||||
### R-197 — the hub held both halves of the evidence and compared them nowhere
|
||||
|
||||
`restic_pw_sha256` is stored on the current escrow and carried to the retained row on every
|
||||
supersession. Comparing the two is what answered the 2026-08-04 recon's hardest question in one query —
|
||||
and nothing in the hub did it. demo-felhom's repository password changed on 2026-08-03, orphaning 36
|
||||
snapshots / 1.14 GB, and **no event, e-mail, card or log line said so for thirteen hours.**
|
||||
|
||||
`SaveHostEscrow` now returns the hash it replaced, and `handleHostEscrowPut` raises
|
||||
**`offsite_repo_key_changed`** (warning, operator-only, edge-triggered once per supersession) when both
|
||||
hashes are known and differ. The in-between shapes are logged rather than dropped, so *"we chose not to
|
||||
alarm"* and *"the check did not run"* never look identical. **No hash value travels** — not in the
|
||||
message, not in the details; the event names the host, the customer and the retained count.
|
||||
|
||||
**Severity chosen for the world this release creates, not the one it inherits:** before R-198 a changed
|
||||
key meant the previous history was unopenable by anyone, ever, which would have argued for `error`.
|
||||
From v0.93.0 the superseding ceremony retains the old identity blob, so the situation is *"this
|
||||
customer's off-site history now depends on an older recovery code"* — operator-actionable, not a loss.
|
||||
`warning` also routes: the dispatcher treats `info` as an intentional non-notify.
|
||||
|
||||
### R-192 (partially) — the alert describes what was measured, and a refusal leaves a trace
|
||||
|
||||
demo-hp e-mailed this daily from 2026-08-03: *"one-time password consumed 284h19m0s ago and 500
|
||||
report(s) since carry no offbox target — the credential is likely burned … Re-issue delivers a fresh
|
||||
one."* `OffsiteReportsSinceConsume` for that same call was **500**: every one of those reports carried a
|
||||
target. The message stated the precise negation of its own measurement and prescribed a remedy for a
|
||||
failure mode that had not occurred — while the self-heal correctly declined, **in complete silence**.
|
||||
|
||||
- The one stuck state is now reported as the two situations it actually covers, **burned** and
|
||||
**regressed**, each stating its own measurement and carrying its own recommendation (the regressed
|
||||
text explicitly withdraws Re-issue and points at what removes an offbox target — a guest rebuild,
|
||||
R-193). `offsite_reports_since_consume` is in the details for the first time.
|
||||
- **The counting window is named in the text.** `CountReportsOffsiteSince` reads `ORDER BY id LIMIT
|
||||
500` — the **oldest** 500 reports after the consume — so on a long-lived box the counts describe the
|
||||
start of the window, not the present. **That guard defect stays OPEN (R-192)** because its correct
|
||||
shape depends on the recovery chain that is not yet assembled (R-199/R-200/R-201); naming the window
|
||||
is how it stays visible instead of being laundered into a confident sentence.
|
||||
- **Every self-heal refusal now leaves a record** — a `notification_log` row on the operator channel
|
||||
with status `refused` and its reason, the R-182 suppressed-e-mail precedent. It rides the stuck
|
||||
event's 24 h cadence, so it sits beside the e-mail it explains rather than accumulating per tick.
|
||||
**The guard's logic is unchanged**: the set of situations in which the heal fires is what it was.
|
||||
- `offsite_delivery_stuck` and `offsite_credential_restaged` are added to `operatorOnlyEvents`. Neither
|
||||
was ever registered, and neither has a `customerMessages` entry — which is **not** a block, so a
|
||||
customer with a configured recipient was in line for an English e-mail about one-time passwords being
|
||||
*"likely burned"*. Narrowing only; the operator channel is untouched.
|
||||
|
||||
### R-196 — five comments corrected (the spec expected three)
|
||||
|
||||
`ReissueCredentials` resets the **provider account** password. It does not and cannot touch the restic
|
||||
repository password, which is generated on the box and never leaves it except sealed under R. Five
|
||||
places in the hub said otherwise: `offsite/offsite.go` (the `MarkEscrowStale` justification),
|
||||
`api/handler.go` (the F3 re-enroll comment), and three in `store/store.go` (the `stale_at` ALTER, the
|
||||
`MarkEscrowStale` doc comment, and `EscrowStatus.Stale`). **That false belief sent two separate
|
||||
investigations the wrong way in one day.** All five now say what the code does, name the correction and
|
||||
cite the recon; the staleness mark is documented as **precautionary**, not evidential. **The behaviour
|
||||
is unchanged and R-196 stays OPEN** — the false-staleness alarm it causes is a behaviour change and
|
||||
must not ride a comment-correction release.
|
||||
|
||||
### Not in this release
|
||||
|
||||
**R-199, R-200 and R-201 remain open** — the hub's blob-serving endpoints still have no client, the
|
||||
password-injection seam still has no form, and nothing in the recovery path has ever been exercised.
|
||||
**Part 5 hit its gate and the orphan card is untouched** — see R-202: a single ACK boolean can say
|
||||
whether a retained recoverable blob exists, but not whether one covers *the key this orphaned
|
||||
repository was written under*, and a conditional promise that is still capable of being false is worse
|
||||
on that surface than a hedged sentence.
|
||||
|
||||
## v0.92.0 — a customer with no machine ever bound does not alarm (2026-08-04, R-195)
|
||||
|
||||
`david` is a real prospective customer whose record was created 2026-08-01 with **no host ever bound**
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-197 (v0.93.0) — a changed offsite repository key must be REPORTED. Both halves of the comparison
|
||||
// (host_escrow.restic_pw_sha256 and the row it replaces) have been in the database since SLICE 3 and
|
||||
// nothing read them: demo-felhom's repository password changed on 2026-08-03, orphaning 36 snapshots /
|
||||
// 1.14 GB, and no event, e-mail, card or log line said so for thirteen hours.
|
||||
//
|
||||
// Driven through the real endpoint (PUT /hosts/<id>/escrow), not by calling the emitter, so the wiring
|
||||
// is part of what is proven.
|
||||
|
||||
// escrowBodyWithHash is escrowBody plus the sealed-password hash and an identity blob — the fork-4
|
||||
// upload shape. Hash VALUES here are test fixtures, never real secrets.
|
||||
func escrowBodyWithHash(blob []byte, resticPwSHA, identity string) string {
|
||||
m := map[string]string{
|
||||
"blob_b64": base64.StdEncoding.EncodeToString(blob),
|
||||
"key_fingerprint": "ab:cd:ef",
|
||||
"posture": "zero_knowledge",
|
||||
"created_at": "2026-08-04T05:00:00Z",
|
||||
"restic_pw_sha256": resticPwSHA,
|
||||
}
|
||||
if identity != "" {
|
||||
m["identity_blob_b64"] = base64.StdEncoding.EncodeToString([]byte(identity))
|
||||
}
|
||||
b, _ := json.Marshal(m)
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// Scenario D — a new escrow sealing a DIFFERENT repo password raises exactly one operator signal,
|
||||
// naming the host, carrying NO hash value.
|
||||
// RED-PROOF: remove the maybeEmitRepoKeyChanged call from handleHostEscrowPut (or make its predicate
|
||||
// always fall through) → no event → this FAILS.
|
||||
func TestEscrowPut_ChangedRepoKey_RaisesSignal(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
|
||||
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_GEN1", "age-gen1")); rr.Code != http.StatusOK {
|
||||
t.Fatalf("first PUT = %d (%s)", rr.Code, rr.Body.String())
|
||||
}
|
||||
// Generation 1 is onboarding, not a change — nothing may fire yet.
|
||||
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
|
||||
t.Fatalf("a FIRST escrow must not raise %s: %+v", eventRepoKeyChanged, ev)
|
||||
}
|
||||
|
||||
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "SHA_GEN2", "age-gen2")); rr.Code != http.StatusOK {
|
||||
t.Fatalf("second PUT = %d (%s)", rr.Code, rr.Body.String())
|
||||
}
|
||||
|
||||
ev, err := st.GetLatestEventByType("c1", eventRepoKeyChanged)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ev == nil {
|
||||
t.Fatal("R-197: the repository key demonstrably changed and NO signal was raised — this is the " +
|
||||
"thirteen-hour silence that let demo-felhom lose 1.14 GB unremarked")
|
||||
}
|
||||
if ev.Severity != "warning" {
|
||||
t.Errorf("severity = %q, want warning (info is an intentional non-notify — the operator would not be told)", ev.Severity)
|
||||
}
|
||||
if !strings.Contains(ev.Message, "h1") {
|
||||
t.Errorf("message must name the host, got %q", ev.Message)
|
||||
}
|
||||
// NO HASH VALUE may travel — not in the message, not in the details.
|
||||
for _, forbidden := range []string{"SHA_GEN1", "SHA_GEN2"} {
|
||||
if strings.Contains(ev.Message, forbidden) || strings.Contains(ev.DetailsJSON, forbidden) {
|
||||
t.Errorf("a repo-password hash value leaked into the event (%s): msg=%q details=%q", forbidden, ev.Message, ev.DetailsJSON)
|
||||
}
|
||||
}
|
||||
// Exactly one, not one per anything.
|
||||
evs, _ := st.GetRecentEvents("c1", 50)
|
||||
n := 0
|
||||
for _, e := range evs {
|
||||
if e.EventType == eventRepoKeyChanged {
|
||||
n++
|
||||
}
|
||||
}
|
||||
if n != 1 {
|
||||
t.Fatalf("%s fired %d times for one supersession, want 1 (edge-triggered)", eventRepoKeyChanged, n)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario E — an UNCHANGED key says nothing. A customer re-running a ceremony for good reasons must
|
||||
// not be punished for it with an alarm.
|
||||
func TestEscrowPut_UnchangedRepoKey_Silent(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
|
||||
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_SAME", "age-1"))
|
||||
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "SHA_SAME", "age-2"))
|
||||
|
||||
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
|
||||
t.Fatalf("a same-hash re-ceremony must be SILENT, got: %+v", ev)
|
||||
}
|
||||
// And it must not have created a superseded row either (pre-existing contract, re-asserted here
|
||||
// because Scenario E's silence would also be produced by the supersede branch never running).
|
||||
if n, _ := st.CountSupersededEscrow("h1"); n != 0 {
|
||||
t.Fatalf("same-hash re-upload created %d superseded row(s), want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
// The in-between shapes must not be silent-and-indistinguishable either: a hash-less NEW blob cannot
|
||||
// show whether the key changed, and saying nothing at all would look identical to "unchanged".
|
||||
// Asserted at the level the code offers — no event (correct: nothing was measured) and no crash.
|
||||
func TestEscrowPut_HashlessSupersession_NoSignal(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||
|
||||
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_GEN1", "age-1"))
|
||||
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "", "age-2")); rr.Code != http.StatusOK {
|
||||
t.Fatalf("hash-less supersession PUT = %d", rr.Code)
|
||||
}
|
||||
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
|
||||
t.Fatalf("a hash-less new blob cannot EVIDENCE a change; it must not claim one: %+v", ev)
|
||||
}
|
||||
// The retention still ran (the hashes differ, so this IS a supersession) — the key is kept even
|
||||
// though the change cannot be evidenced.
|
||||
if n, _ := st.CountSupersededEscrow("h1"); n != 1 {
|
||||
t.Fatalf("superseded rows = %d, want 1", n)
|
||||
}
|
||||
}
|
||||
|
||||
// R-97a's lesson, pinned in the SAME commit that mints the type: an operator-tier event that is not in
|
||||
// operatorOnlyEvents reaches customers as raw English, because a missing customerMessages entry is NOT
|
||||
// a block. Checked here rather than in notify so the type's two registers are asserted together.
|
||||
func TestRepoKeyChanged_IsOperatorOnly(t *testing.T) {
|
||||
if !notify.IsOperatorOnly(eventRepoKeyChanged) {
|
||||
t.Fatalf("%s is not registered operator-only — a customer would receive operator-grade English "+
|
||||
"about escrow custody", eventRepoKeyChanged)
|
||||
}
|
||||
// The two R-192 types have the same property and the same reason (see the register's comment).
|
||||
for _, et := range []string{"offsite_delivery_stuck", "offsite_credential_restaged"} {
|
||||
if !notify.IsOperatorOnly(et) {
|
||||
t.Errorf("%s is not registered operator-only", et)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -155,7 +155,7 @@ func TestReportACK_EscrowStatus(t *testing.T) {
|
||||
}
|
||||
|
||||
// escrow row with identity blob + hash → the ACK carries all three fields
|
||||
if _, err := st.SaveHostEscrow("hv1", []byte("k-blob"), "fp", "zero_knowledge", "2026-07-09T20:00:00Z", "abc123"); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow("hv1", []byte("k-blob"), "fp", "zero_knowledge", "2026-07-09T20:00:00Z", "abc123"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveHostDRBundle("hv1", []byte("identity-blob"), "{}"); err != nil {
|
||||
|
||||
@@ -1065,9 +1065,13 @@ func (h *Handler) reissueOnReenroll(cc *store.CustomerConfig) {
|
||||
}
|
||||
}
|
||||
// F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
|
||||
// one-time password only ever reached the OLD controller). The re-issuer resets the restic repo
|
||||
// password, which makes the OLD escrow blob stale — the offsite provisioner invalidates the
|
||||
// escrow (2.3) and emits both events. Skips silently when offsite isn't provisioned/enabled.
|
||||
// one-time password only ever reached the OLD controller). ⚠ CORRECTED 2026-08-04 (R-196): this
|
||||
// used to claim "the re-issuer resets the restic repo password, which makes the OLD escrow blob
|
||||
// stale". It does not and cannot — the re-issuer resets the PROVIDER account password; the
|
||||
// repository password is generated on the box and never leaves it except sealed under R. The
|
||||
// provisioner does mark the escrow stale, but PRECAUTIONARILY (see the reasoning at
|
||||
// offsite.ReissueCredentials), not because this call rotated anything.
|
||||
// Skips silently when offsite isn't provisioned/enabled.
|
||||
if h.offsiteReissuer != nil {
|
||||
if err := h.offsiteReissuer(context.Background(), cc.CustomerID); err != nil {
|
||||
h.logger.Printf("[WARN] offsite re-issue on re-enroll for %s failed: %v", cc.CustomerID, err)
|
||||
@@ -1141,7 +1145,7 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
|
||||
// Store the OPAQUE bytes. No decrypt path exists — the hub cannot open this. Part B (v0.60.0):
|
||||
// when this upload supersedes a DIFFERENT-passphrase old blob, the old one is RETAINED (not
|
||||
// overwritten) so its recovery-code-recoverable history survives (Viktor's data-first ruling).
|
||||
superseded, serr := h.store.SaveHostEscrow(pathHostID, blob, req.KeyFingerprint, req.Posture, createdAt, req.ResticPwSHA256)
|
||||
superseded, prevPwSHA, serr := h.store.SaveHostEscrow(pathHostID, blob, req.KeyFingerprint, req.Posture, createdAt, req.ResticPwSHA256)
|
||||
if serr != nil {
|
||||
h.logger.Printf("[ERROR] Failed to store escrow for host %s: %v", pathHostID, serr)
|
||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||
@@ -1157,6 +1161,10 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
|
||||
"A korábbi helyreállítási csomag megőrizve (új kulcs érkezett).", string(details), "hub"); eerr != nil {
|
||||
h.logger.Printf("[WARN] escrow_superseded event save failed for %s: %v", pathHostID, eerr)
|
||||
}
|
||||
// R-197: the box's offsite DATA key demonstrably changed. Both halves of that comparison
|
||||
// have been stored since SLICE 3 and nothing read them — demo-felhom's key changed on
|
||||
// 2026-08-03 and nothing said so for thirteen hours.
|
||||
h.maybeEmitRepoKeyChanged(host.CustomerID, pathHostID, prevPwSHA, req.ResticPwSHA256, n)
|
||||
}
|
||||
}
|
||||
// Slice 10D.1: optionally store the IDENTITY escrow blob + the non-secret DR directive alongside
|
||||
@@ -1185,6 +1193,69 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
|
||||
w.Write([]byte(`{"status":"ok"}`))
|
||||
}
|
||||
|
||||
// eventRepoKeyChanged (R-197) — the box's offsite restic REPOSITORY password changed, proven by the
|
||||
// hub's own stored hashes. Hub-internal (not in allowedEventTypes, like escrow_superseded) and
|
||||
// registered operator-only in notify.operatorOnlyEvents.
|
||||
const eventRepoKeyChanged = "offsite_repo_key_changed"
|
||||
|
||||
// maybeEmitRepoKeyChanged raises ONE operator signal per supersession when the sealed offsite repo
|
||||
// password demonstrably changed. Both hashes have been stored since SLICE 3 (host_escrow and, since
|
||||
// v0.60.0, host_escrow_superseded) and NOTHING compared them: demo-felhom's repository password
|
||||
// changed on 2026-08-03, orphaning 36 snapshots / 1.14 GB, and no event, e-mail, card or log line
|
||||
// said so for thirteen hours — the comparison that eventually found it is this one
|
||||
// (audits/RECON-offsite-dr-chain-2026-08-04.md, R-197).
|
||||
//
|
||||
// THE PREDICATE IS DELIBERATELY NARROW: both hashes known AND different. A first-ever hash (prev "")
|
||||
// is onboarding, not a change; a hash-less supersession (now "") cannot show a change happened; an
|
||||
// identical hash is a re-ceremony of the SAME password, which is a normal healthy act and must stay
|
||||
// silent or a customer is punished for re-running a ceremony. The in-between shapes are LOGGED rather
|
||||
// than dropped, so "we chose not to alarm" and "the check did not run" never look identical.
|
||||
//
|
||||
// SEVERITY = warning, chosen for the world v0.93.0 creates rather than the one it inherits. Before
|
||||
// R-198 a changed key meant the previous history was unopenable by anyone, ever — that would have
|
||||
// argued for error. From v0.93.0 the superseding ceremony RETAINS the old identity blob, so the
|
||||
// previous history stays recoverable with the recovery code that sealed it: the situation is "this
|
||||
// customer's off-site history now depends on an older recovery code", which is operator-actionable
|
||||
// (check the orphan card, expect a fresh repository) and is not a loss. warning also routes: the
|
||||
// dispatcher notifies on warning/error/critical and treats info as an intentional non-notify, and
|
||||
// the whole point of this row is that the operator learns on the day.
|
||||
//
|
||||
// EDGE-TRIGGERED: called only from the superseded branch of the escrow PUT, i.e. once per
|
||||
// supersession, never per report. No timer lives here — the dispatcher owns cooldown.
|
||||
//
|
||||
// NO HASH VALUE TRAVELS. The message and the details name the host, the customer and the retained
|
||||
// count only. The hashes are non-reversible, but a hash is still a fingerprint of a live secret and
|
||||
// this project's rule is that values do not leave the store.
|
||||
func (h *Handler) maybeEmitRepoKeyChanged(customerID, hostID, prevSHA, newSHA string, retained int) {
|
||||
switch {
|
||||
case prevSHA == "":
|
||||
h.logger.Printf("[INFO] escrow for host %s: no previous repo-password hash recorded (first hash, or a legacy hash-less blob) — repo-key-change check not applicable", hostID)
|
||||
return
|
||||
case newSHA == "":
|
||||
h.logger.Printf("[WARN] escrow for host %s: the NEW blob carries no repo-password hash (hash-less supersession) — whether the repository key changed CANNOT be determined from the hub's data", hostID)
|
||||
return
|
||||
case prevSHA == newSHA:
|
||||
return // same password re-sealed: a healthy re-ceremony (Scenario E — silence is correct)
|
||||
}
|
||||
msg := fmt.Sprintf("Offsite repository key CHANGED for host %s: the new escrow seals a different repository password than the one it replaced. "+
|
||||
"The previous off-site history is no longer opened by this box's current key. The superseding blob was retained (%d held), so that history stays recoverable with the recovery code that sealed it — verify the box's off-site tier reports a repository rather than an orphan card, and expect the next backup to start a fresh history.",
|
||||
hostID, retained)
|
||||
details, _ := json.Marshal(map[string]any{
|
||||
"host_id": hostID,
|
||||
"retained_count": retained,
|
||||
"repo_key": "changed", // never the hash values
|
||||
})
|
||||
if _, err := h.store.SaveEvent(customerID, eventRepoKeyChanged, "warning", msg, string(details), "hub"); err != nil {
|
||||
h.logger.Printf("[WARN] %s event save failed for %s: %v", eventRepoKeyChanged, hostID, err)
|
||||
return // audit row first: an e-mail without its event row lies (the OffsiteChecker convention)
|
||||
}
|
||||
h.logger.Printf("[WARN] offsite repository key CHANGED for host %s (customer %s) — previous history now depends on the superseded recovery code; %d retained blob(s)",
|
||||
hostID, customerID, retained)
|
||||
if h.dispatcher != nil {
|
||||
go h.dispatcher.ProcessEvent(customerID, eventRepoKeyChanged, "warning", msg, string(details), "hub")
|
||||
}
|
||||
}
|
||||
|
||||
// handleHostRecoveryCredentialPut vaults a host's break-glass root@pam console credential (TASK G1).
|
||||
// SELF-SCOPED (a host key writes only its own; global may write any) — day-0 posts it with the
|
||||
// host api_key. The secret is stored at rest and NEVER logged (only the username + a length are
|
||||
|
||||
@@ -94,40 +94,112 @@ func (c *OffsiteDeliveryChecker) Check() {
|
||||
if age < stuckAfter {
|
||||
continue // normal convergence window
|
||||
}
|
||||
c.maybeEmitStuck(cfg.CustomerID, status, age)
|
||||
c.maybeHeal(cfg.CustomerID, status)
|
||||
emitted := c.maybeEmitStuck(cfg.CustomerID, status, age)
|
||||
c.maybeHeal(cfg.CustomerID, status, emitted)
|
||||
}
|
||||
}
|
||||
|
||||
// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer.
|
||||
func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) {
|
||||
// deliveryShape names the two situations the ONE stuck state actually covers. They need different
|
||||
// text and different advice, and conflating them is R-192's defect (a).
|
||||
type deliveryShape string
|
||||
|
||||
const (
|
||||
// shapeBurned — NO report since the consume carried an offbox target: the apply never persisted.
|
||||
// Re-issue is the indicated action.
|
||||
shapeBurned deliveryShape = "burned"
|
||||
// shapeRegressed — reports since the consume DID carry an offbox target and the latest does not:
|
||||
// the credential worked and the target was later lost (a guest rebuild does exactly this, R-193).
|
||||
// Re-issue is NOT indicated; it treats a symptom whose cause is elsewhere.
|
||||
shapeRegressed deliveryShape = "regressed"
|
||||
)
|
||||
|
||||
func shapeOf(status offsite.DeliveryStatus) deliveryShape {
|
||||
if status.OffsiteReportsSinceConsume == 0 {
|
||||
return shapeBurned
|
||||
}
|
||||
return shapeRegressed
|
||||
}
|
||||
|
||||
// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer. Returns
|
||||
// whether it emitted, so the heal's refusal record rides the same cadence rather than inventing one.
|
||||
//
|
||||
// R-192 defect (a), fixed here: the message used to interpolate ReportsSinceConsume (the TOTAL) into
|
||||
// a hardcoded phrase "report(s) since carry no offbox target", and never consulted
|
||||
// OffsiteReportsSinceConsume — the field that says the opposite. On demo-hp it stated, daily, that
|
||||
// 500 reports carried no offbox target when all 500 of them did, and prescribed Re-issue for a
|
||||
// failure mode that had not occurred. The message now STATES WHAT WAS MEASURED and lets the operator
|
||||
// read it; the recommendation follows the shape rather than being hardcoded.
|
||||
//
|
||||
// THE WINDOW IS NAMED ON PURPOSE. CountReportsOffsiteSince reads `ORDER BY id LIMIT 500` — the OLDEST
|
||||
// 500 reports after the consume, not the newest — so on a long-lived customer these counts describe
|
||||
// the beginning of the window and not the present. That is a real scoping defect (R-192's other half)
|
||||
// and it stays OPEN because its correct shape depends on the recovery chain that is not yet
|
||||
// assembled (R-199/R-200/R-201). Naming the window in the text is how it stays visible instead of
|
||||
// being laundered into a confident sentence — an instrument that can silently mis-scope its results
|
||||
// must say so where it reports them.
|
||||
func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) bool {
|
||||
last, err := c.store.LastEventAt(customerID, eventDeliveryStuck)
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] offsite-delivery: %s: cooldown read: %v", customerID, err)
|
||||
return
|
||||
return false
|
||||
}
|
||||
if !last.IsZero() && c.now().Sub(last) < stuckCooldown {
|
||||
return
|
||||
return false
|
||||
}
|
||||
msg := fmt.Sprintf("Offsite delivery stuck: one-time password consumed %s ago and %d report(s) since carry no offbox target — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one.",
|
||||
shape := shapeOf(status)
|
||||
var msg string
|
||||
switch shape {
|
||||
case shapeBurned:
|
||||
msg = fmt.Sprintf("Offsite delivery stuck (BURNED-credential shape): the one-time password was consumed %s ago; of the first %d report(s) after that consume, NONE carried an offbox target, and the latest report carries none either. The credential never reached a persisted apply. Re-issue delivers a fresh one.",
|
||||
age.Round(time.Minute), status.ReportsSinceConsume)
|
||||
default:
|
||||
msg = fmt.Sprintf("Offsite delivery stuck (REGRESSED-apply shape): the one-time password was consumed %s ago; of the first %d report(s) after that consume, %d DID carry an offbox target — and the latest report carries none. The credential was applied and worked; the target was lost afterwards. Re-issue is NOT the indicated action: find what removed the offbox target (a guest rebuild does, R-193). Automatic restage is deliberately withheld for this shape. NOTE: the counts cover at most the first 500 reports after the consume, so on a long-lived box they describe the start of the window, not now (R-192, open).",
|
||||
age.Round(time.Minute), status.ReportsSinceConsume, status.OffsiteReportsSinceConsume)
|
||||
}
|
||||
details, _ := json.Marshal(map[string]any{
|
||||
"state": string(status.State),
|
||||
"shape": string(shape),
|
||||
"consumed_at": status.Since.UTC().Format(time.RFC3339),
|
||||
"reports_since_consume": status.ReportsSinceConsume,
|
||||
"offsite_reports_since_consume": status.OffsiteReportsSinceConsume,
|
||||
"count_window": "oldest 500 reports after consumed_at (R-192, open)",
|
||||
})
|
||||
c.emit(customerID, eventDeliveryStuck, "warning", msg, string(details))
|
||||
return true
|
||||
}
|
||||
|
||||
// maybeHeal fires the R-71c self-heal when the burned-credential shape is unambiguous:
|
||||
// consumed ≥ stuckAfter ago, ≥ healMinReports consecutive reports since with ZERO offbox evidence,
|
||||
// one heal per healCooldown — and the R-39(a) guard holds at act time.
|
||||
func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus) {
|
||||
//
|
||||
// R-192 defect (b), fixed here: every refusal above the "not configured" line now leaves a RECORD.
|
||||
// The regressed-shape branch used to be a bare `return`, so the operator received a daily e-mail with
|
||||
// the wrong story, no heal, and nothing anywhere saying why the heal declined — "we chose not to act"
|
||||
// and "the heal never ran" looked identical. `offsite_credential_restaged` has never fired for any
|
||||
// customer, and until now that fact was indistinguishable from the checker being dead.
|
||||
//
|
||||
// The record is a notification_log row (the dispatcher's suppressed-operator-e-mail precedent, R-182:
|
||||
// a decision not to act is written down on the channel it would have used). It rides `recordRefusal`
|
||||
// — true only when the stuck event was emitted this pass — so it appears once per stuckCooldown
|
||||
// beside the e-mail it explains, rather than once per monitor tick. The GUARD ITSELF IS UNCHANGED:
|
||||
// the set of situations in which the heal fires is byte-for-byte what it was; only the silence is
|
||||
// gone. The two conditions are split into separate branches solely so each refusal can name its own
|
||||
// reason.
|
||||
func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus, recordRefusal bool) {
|
||||
if c.reissuer == nil {
|
||||
return // no provisioner configured: the heal does not exist on this hub, so there is nothing to explain
|
||||
}
|
||||
if status.OffsiteReportsSinceConsume != 0 {
|
||||
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
|
||||
"regressed-apply shape: %d of the first %d report(s) after the consume DID carry an offbox target, so a burned credential is ruled out — a restage would treat a symptom whose cause is elsewhere. Operator's call (R-193).",
|
||||
status.OffsiteReportsSinceConsume, status.ReportsSinceConsume))
|
||||
return
|
||||
}
|
||||
if status.ReportsSinceConsume < healMinReports || status.OffsiteReportsSinceConsume != 0 {
|
||||
return // box not reporting enough, or offbox evidence exists (regressed-apply shape) → operator's call
|
||||
if status.ReportsSinceConsume < healMinReports {
|
||||
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
|
||||
"only %d report(s) since the consume (need %d): the box has not reported enough for the burned shape to be unambiguous.",
|
||||
status.ReportsSinceConsume, healMinReports))
|
||||
return
|
||||
}
|
||||
last, err := c.store.LastEventAt(customerID, eventCredentialRestaged)
|
||||
if err != nil {
|
||||
@@ -146,8 +218,8 @@ func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.Del
|
||||
return
|
||||
}
|
||||
if info == nil || info.ConsumedAt.IsZero() {
|
||||
c.logger.Printf("[INFO] offsite-delivery: %s: heal refused — secret row is now %s (R-39(a) guard: never restage over an unconsumed secret)",
|
||||
customerID, secretShape(info))
|
||||
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
|
||||
"R-39(a) guard: the secret row is now %s — restaging over an unconsumed secret would clobber a password the box may be about to consume.", secretShape(info)))
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
|
||||
@@ -167,6 +239,22 @@ func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.Del
|
||||
string(details))
|
||||
}
|
||||
|
||||
// recordHealRefusal makes a decision NOT to self-heal visible. Always logs; additionally writes a
|
||||
// notification_log row on the "operator" channel with status "refused" when `record` is set (the
|
||||
// stuck event was emitted this pass), so the refusal sits next to the e-mail that prompted the
|
||||
// question. A LogNotification failure is logged, never swallowed, and never blocks the refusal — the
|
||||
// refusal is the primary effect.
|
||||
func (c *OffsiteDeliveryChecker) recordHealRefusal(customerID string, record bool, reason string) {
|
||||
c.logger.Printf("[INFO] offsite-delivery: %s: self-heal REFUSED — %s", customerID, reason)
|
||||
if !record {
|
||||
return
|
||||
}
|
||||
if err := c.store.LogNotification(customerID, eventCredentialRestaged, "warning",
|
||||
"Automatic offsite credential restage was NOT performed.", "refused", reason, "operator"); err != nil {
|
||||
c.logger.Printf("[WARN] offsite-delivery: %s: could not record the heal refusal: %v", customerID, err)
|
||||
}
|
||||
}
|
||||
|
||||
func secretShape(info *store.OneTimeSecretInfo) string {
|
||||
if info == nil {
|
||||
return "absent"
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-192 (v0.93.0) — the alert must describe what was MEASURED, and a refusal to self-heal must leave a
|
||||
// trace.
|
||||
//
|
||||
// What was wrong, quoted from the live e-mail demo-hp sent daily from 2026-08-03: "one-time password
|
||||
// consumed 284h19m0s ago and 500 report(s) since carry no offbox target — the credential is likely
|
||||
// burned … Re-issue delivers a fresh one." OffsiteReportsSinceConsume for that same call was 500:
|
||||
// every one of those reports carried a target. The message stated the precise negation of its own
|
||||
// measurement and prescribed a remedy for a failure mode that had not occurred — while the self-heal
|
||||
// correctly declined, in complete silence.
|
||||
|
||||
// regressedFixture puts c1 into demo-hp's real shape: consumed >1h ago, reports WITH an offbox target
|
||||
// after the consume, then reports WITHOUT — the latest carrying none.
|
||||
//
|
||||
// The report timestamps get jitter deliberately: a perfectly regular series lands exactly on
|
||||
// boundaries and survives mutations that a realistic one catches.
|
||||
func (h dtHarness) regressedFixture(t *testing.T, withOffsite, without int) {
|
||||
t.Helper()
|
||||
if err := h.st.SaveOneTimeSecret("c1", "x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
consumed := time.Now().UTC().Add(-30 * time.Hour).Format("2006-01-02 15:04:05")
|
||||
staged := time.Now().UTC().Add(-31 * time.Hour).Format("2006-01-02 15:04:05")
|
||||
if err := h.st.SetOneTimeSecretTimesForTest("c1", staged, consumed); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
jitter := []int{0, 3, 1, 7, 2, 5, 11, 4}
|
||||
for i := 0; i < withOffsite; i++ {
|
||||
if err := h.st.SaveReport("c1", []byte(dtReportWithOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
time.Sleep(time.Duration(jitter[i%len(jitter)]) * time.Millisecond)
|
||||
}
|
||||
for i := 0; i < without; i++ {
|
||||
if err := h.st.SaveReport("c1", []byte(dtReportNoOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
time.Sleep(time.Duration(jitter[(i+3)%len(jitter)]) * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func (h dtHarness) refusalRows(t *testing.T) []store.NotificationLogEntry {
|
||||
t.Helper()
|
||||
all, err := h.st.GetRecentNotifications("c1", 50)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out []store.NotificationLogEntry
|
||||
for _, n := range all {
|
||||
if n.Status == "refused" {
|
||||
out = append(out, n)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Scenario F — the message states the measured state, not a denied cause.
|
||||
// RED-PROOF: restore the old single hardcoded sentence (interpolate ReportsSinceConsume into
|
||||
// "report(s) since carry no offbox target") → the regressed assertions FAIL.
|
||||
func TestDeliveryChecker_RegressedShape_MessageMatchesMeasurement(t *testing.T) {
|
||||
h := newDTHarness(t, true)
|
||||
h.regressedFixture(t, 6, 2)
|
||||
|
||||
h.checker.Check()
|
||||
|
||||
saved := h.savedEvents(t, "offsite_delivery_stuck")
|
||||
if len(saved) != 1 {
|
||||
t.Fatalf("stuck events = %d, want 1", len(saved))
|
||||
}
|
||||
msg := saved[0].Message
|
||||
|
||||
// It must NOT assert the thing the data contradicts.
|
||||
if strings.Contains(msg, "carry no offbox target") || strings.Contains(msg, "likely burned") {
|
||||
t.Fatalf("R-192(a): the message still asserts a burned credential on a REGRESSED shape — this is "+
|
||||
"the sentence that stated the negation of its own measurement:\n%s", msg)
|
||||
}
|
||||
// It must name the shape it detected and the measurement behind it.
|
||||
if !strings.Contains(msg, "REGRESSED") {
|
||||
t.Errorf("the message must name the detected shape, got:\n%s", msg)
|
||||
}
|
||||
if !strings.Contains(msg, "6 DID carry an offbox target") {
|
||||
t.Errorf("the message must state the measured offbox-bearing count (6), got:\n%s", msg)
|
||||
}
|
||||
// It must not prescribe the action for a failure mode that did not occur.
|
||||
if !strings.Contains(msg, "Re-issue is NOT the indicated action") {
|
||||
t.Errorf("the message must withdraw the Re-issue recommendation for this shape, got:\n%s", msg)
|
||||
}
|
||||
// The counting window is named, because CountReportsOffsiteSince reads the OLDEST 500 — an
|
||||
// instrument that can silently mis-scope must say so where it reports (R-192's other half, open).
|
||||
if !strings.Contains(msg, "500") {
|
||||
t.Errorf("the message must name its counting window, got:\n%s", msg)
|
||||
}
|
||||
|
||||
var details map[string]any
|
||||
if err := json.Unmarshal([]byte(saved[0].DetailsJSON), &details); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if details["shape"] != "regressed" {
|
||||
t.Errorf("details.shape = %v, want regressed", details["shape"])
|
||||
}
|
||||
// The field whose absence was defect (a) is now carried.
|
||||
if got, ok := details["offsite_reports_since_consume"].(float64); !ok || int(got) != 6 {
|
||||
t.Errorf("details must carry offsite_reports_since_consume=6, got %v", details["offsite_reports_since_consume"])
|
||||
}
|
||||
}
|
||||
|
||||
// The burned shape keeps its own (correct) text and its recommendation — the fix must not blunt the
|
||||
// case that was always right.
|
||||
func TestDeliveryChecker_BurnedShape_KeepsItsRecommendation(t *testing.T) {
|
||||
h := newDTHarness(t, false)
|
||||
h.burnedFixture(t, 5)
|
||||
|
||||
h.checker.Check()
|
||||
|
||||
saved := h.savedEvents(t, "offsite_delivery_stuck")
|
||||
if len(saved) != 1 {
|
||||
t.Fatalf("stuck events = %d, want 1", len(saved))
|
||||
}
|
||||
msg := saved[0].Message
|
||||
if !strings.Contains(msg, "BURNED") || !strings.Contains(msg, "NONE carried an offbox target") {
|
||||
t.Errorf("the burned shape must state its own measurement, got:\n%s", msg)
|
||||
}
|
||||
if !strings.Contains(msg, "Re-issue delivers a fresh one") {
|
||||
t.Errorf("the burned shape must keep its recommendation, got:\n%s", msg)
|
||||
}
|
||||
var details map[string]any
|
||||
_ = json.Unmarshal([]byte(saved[0].DetailsJSON), &details)
|
||||
if details["shape"] != "burned" {
|
||||
t.Errorf("details.shape = %v, want burned", details["shape"])
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario G — a refusal to self-heal leaves a durable record naming its reason.
|
||||
// RED-PROOF: replace the recordHealRefusal call in the regressed branch with a bare `return`
|
||||
// (production behaviour up to v0.92.0) → no row → this FAILS.
|
||||
func TestDeliveryChecker_HealRefusal_LeavesARecord(t *testing.T) {
|
||||
h := newDTHarness(t, true)
|
||||
h.regressedFixture(t, 6, 2)
|
||||
|
||||
h.checker.Check()
|
||||
|
||||
if h.reissuer.count() != 0 {
|
||||
t.Fatal("the heal must NOT fire on a regressed shape — the guard is unchanged by this fix")
|
||||
}
|
||||
rows := h.refusalRows(t)
|
||||
if len(rows) != 1 {
|
||||
t.Fatalf("R-192(b): the heal declined and left %d record(s), want 1 — 'we chose not to act' and "+
|
||||
"'the heal never ran' must not look identical", len(rows))
|
||||
}
|
||||
r := rows[0]
|
||||
if r.Channel != "operator" {
|
||||
t.Errorf("refusal channel = %q, want operator (the channel the heal's own event would use)", r.Channel)
|
||||
}
|
||||
if !strings.Contains(r.ErrorMessage, "regressed-apply shape") {
|
||||
t.Errorf("the record must name WHY, got %q", r.ErrorMessage)
|
||||
}
|
||||
if !strings.Contains(r.ErrorMessage, "6 of the first 8") {
|
||||
t.Errorf("the record must carry the measurement it refused on, got %q", r.ErrorMessage)
|
||||
}
|
||||
}
|
||||
|
||||
// The R-39(a) guard's refusal is recorded too — it was the one refusal that already logged, and it
|
||||
// still must not be the only one that does.
|
||||
//
|
||||
// Reaching it needs the real TOCTOU, not a pre-staged row: staging BEFORE Check() makes the derive
|
||||
// itself return staged_awaiting_consume and the pass never reaches the heal at all. Same technique as
|
||||
// TestDeliveryChecker_R39aGuard_NeverRestagesOverUnconsumed — the operator stages mid-tick, on the
|
||||
// stuck event's dispatch, which runs after the derive and before maybeHeal.
|
||||
func TestDeliveryChecker_R39aRefusal_LeavesARecord(t *testing.T) {
|
||||
h := newDTHarness(t, true)
|
||||
h.burnedFixture(t, 5)
|
||||
|
||||
base := h.checker.onEvent
|
||||
h.checker.onEvent = func(cid, et, sev, msg, det, src string) {
|
||||
if et == "offsite_delivery_stuck" {
|
||||
if err := h.st.SaveOneTimeSecret("c1", "operator-fresh"); err != nil {
|
||||
t.Errorf("mid-tick stage: %v", err)
|
||||
}
|
||||
if err := h.st.SetOneTimeSecretTimesForTest("c1", "2026-07-23 12:00:00", ""); err != nil {
|
||||
t.Errorf("mid-tick stamp: %v", err)
|
||||
}
|
||||
}
|
||||
base(cid, et, sev, msg, det, src)
|
||||
}
|
||||
|
||||
h.checker.Check()
|
||||
|
||||
if h.reissuer.count() != 0 {
|
||||
t.Fatal("R-39(a): the heal must never restage over an unconsumed secret")
|
||||
}
|
||||
rows := h.refusalRows(t)
|
||||
if len(rows) != 1 {
|
||||
t.Fatalf("the R-39(a) refusal left %d record(s), want 1", len(rows))
|
||||
}
|
||||
if !strings.Contains(rows[0].ErrorMessage, "R-39(a)") {
|
||||
t.Errorf("the record must name the guard, got %q", rows[0].ErrorMessage)
|
||||
}
|
||||
}
|
||||
|
||||
// The refusal record rides the stuck event's 24h cadence: it explains the e-mail the operator just
|
||||
// received, and must not accumulate one row per monitor tick.
|
||||
func TestDeliveryChecker_HealRefusal_RidesTheStuckCadence(t *testing.T) {
|
||||
h := newDTHarness(t, true)
|
||||
h.regressedFixture(t, 6, 2)
|
||||
|
||||
h.checker.Check()
|
||||
h.checker.Check() // the stuck event is swallowed by its cooldown; so is the refusal record
|
||||
h.checker.Check()
|
||||
|
||||
if n := len(h.refusalRows(t)); n != 1 {
|
||||
t.Fatalf("refusal records = %d after three passes, want 1 (one per stuck e-mail, not per tick)", n)
|
||||
}
|
||||
}
|
||||
@@ -102,7 +102,7 @@ func TestOffsiteStale_NeverRanEscrowAnchorWins(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// escrow blob stored now (SaveHostEscrow stamps updated_at with datetime('now'))
|
||||
if _, err := st.SaveHostEscrow("h1", []byte("blob"), "fp", "zero_knowledge", time.Now().UTC().Format(time.RFC3339), "sha"); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow("h1", []byte("blob"), "fp", "zero_knowledge", time.Now().UTC().Format(time.RFC3339), "sha"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
saveOffsiteReport(t, st, "c1", offsiteJSON(true, "escrowed", "", "", 0, 50))
|
||||
|
||||
@@ -421,6 +421,23 @@ var operatorOnlyEvents = map[string]bool{
|
||||
// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
|
||||
// once (v0.78.0) and the comment above records it.
|
||||
"backup_run_failures": true,
|
||||
// R-197 (v0.93.0). "The sealed offsite repository key changed" is a custody fact about escrow
|
||||
// blobs. A customer can take no action on it — the remedy is the operator's inspection of the
|
||||
// off-site tier — and the text is operator-grade English naming host ids and retained-blob
|
||||
// counts. Listed here in the SAME commit that mints the type: an operator-tier type that is not
|
||||
// registered here reaches customers as raw English, because a missing customerMessages entry is
|
||||
// NOT a block (the v0.78.0 defect recorded above).
|
||||
"offsite_repo_key_changed": true,
|
||||
// R-192 (v0.93.0). These two predate the register and were never added to it — a real gap, not a
|
||||
// tidy-up. `offsite_delivery_stuck` is severity warning, has no customerMessages entry, and
|
||||
// therefore fell through to FormatCustomerEmail's raw-English fallback: a customer whose box hit
|
||||
// the stuck shape was in line for an English e-mail about one-time passwords being "likely
|
||||
// burned". Measured on the live hub: notification_log holds operator rows for demo-hp and no
|
||||
// customer rows — which is NOT evidence the leg is blocked (it is equally consistent with no
|
||||
// configured recipient), so the register makes it structural instead of incidental. Narrowing
|
||||
// only: the operator channel is untouched.
|
||||
"offsite_delivery_stuck": true,
|
||||
"offsite_credential_restaged": true,
|
||||
}
|
||||
|
||||
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
|
||||
|
||||
@@ -195,13 +195,30 @@ func (p *Provisioner) ReissueCredentials(ctx context.Context, customerID, typ st
|
||||
return fmt.Errorf("offsite: store re-issued one-time password: %w", err)
|
||||
}
|
||||
|
||||
// v0.57.0 (2.3, the escrow-honesty fix): the restic repo password just changed, so any existing
|
||||
// key-escrow blob — which sealed the OLD password — is now STALE. A recovery code minted against
|
||||
// it would decrypt a password that no longer opens the repo. Mark the escrow stale so the hub
|
||||
// stops advertising "ceremony done" and the customer's escrow wizard is offered again; a fresh
|
||||
// ceremony seals the new password and clears the flag. Every credential change also emits a
|
||||
// visible customer event (offsite_reissued always; escrow_stale only when a blob was invalidated).
|
||||
// Best-effort: the password reset already succeeded — a bookkeeping failure here must not fail it.
|
||||
// v0.57.0 (2.3, the escrow-honesty fix). ⚠ ITS STATED REASON WAS FALSE AND IS CORRECTED HERE
|
||||
// (R-196, 2026-08-04). The comment used to read "the restic repo password just changed" — this
|
||||
// function does NOT change it and CANNOT: the repository password is generated on the box
|
||||
// (controller `WriteOffboxSecrets`) and never leaves it except sealed inside the R-wrapped escrow.
|
||||
// What this function changes is the PROVIDER account password (①), which is a different secret at
|
||||
// a different layer. The false premise survived because the EFFECT (a stale escrow) is real, so
|
||||
// nobody checked the CAUSE — and it sent two separate investigations the wrong way in one day
|
||||
// (audits/SPIKE-offsite-credential-recovery-2026-08-04.md Q4).
|
||||
//
|
||||
// WHY THE MARK IS STILL MADE, on honest grounds: a re-issue is the operator's response to a
|
||||
// broken delivery, and the box's re-apply may mint a fresh repository password (it does exactly
|
||||
// that whenever `<DataDir>/offbox/repo_password` is absent — the guest-rebuild shape). The mark is
|
||||
// therefore PRECAUTIONARY, not evidential. The evidential signal is R-197's
|
||||
// `offsite_repo_key_changed`, which fires on a measured hash difference at the next ceremony.
|
||||
// KNOWN CONSEQUENCE, left in place deliberately: on the ordinary re-issue shape — a box that still
|
||||
// holds its repo_password file — the password does NOT change, so this marks a healthy escrow
|
||||
// stale and asks the customer for an unnecessary ceremony. R-196 stays OPEN for that; it is a
|
||||
// behaviour change, not a comment fix, and it must not ride a comment-correction commit.
|
||||
//
|
||||
// Mark the escrow stale so the hub stops advertising "ceremony done" and the customer's escrow
|
||||
// wizard is offered again; a fresh ceremony seals the current password and clears the flag. Every
|
||||
// credential change also emits a visible customer event (offsite_reissued always; escrow_stale
|
||||
// only when a blob was invalidated). Best-effort: the password reset already succeeded — a
|
||||
// bookkeeping failure here must not fail it.
|
||||
escrowStaled := false
|
||||
if host, herr := p.Store.GetHostByCustomer(customerID); herr == nil && host != nil {
|
||||
if esc, eerr := p.Store.GetHostEscrow(host.HostID); eerr == nil && esc != nil {
|
||||
|
||||
@@ -226,7 +226,7 @@ func TestReissue_InvalidatesEscrow(t *testing.T) {
|
||||
if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Before re-issue: current escrow — the hub serves the sealed hash and is NOT stale.
|
||||
@@ -256,7 +256,7 @@ func TestReissue_InvalidatesEscrow(t *testing.T) {
|
||||
}
|
||||
|
||||
// A fresh ceremony (new blob sealing the new password) clears stale + serves the new hash.
|
||||
if _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob-2"), "SHA256:fp", "zero_knowledge", "2026-07-16T01:00:00Z", "NEWHASH"); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob-2"), "SHA256:fp", "zero_knowledge", "2026-07-16T01:00:00Z", "NEWHASH"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
es, _ = st.GetEscrowStatusForCustomer(cust)
|
||||
|
||||
@@ -25,10 +25,10 @@ func seedRetainedBlob(t *testing.T, st *Store, customerID string) {
|
||||
if err := st.UpsertHost(&Host{HostID: hostID, CustomerID: customerID, APIKey: "h"}); err != nil {
|
||||
t.Fatalf("upsert host: %v", err)
|
||||
}
|
||||
if _, err := st.SaveHostEscrow(hostID, []byte("A"), "fpA", "p", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow(hostID, []byte("A"), "fpA", "p", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||
t.Fatalf("escrow A: %v", err)
|
||||
}
|
||||
if _, err := st.SaveHostEscrow(hostID, []byte("B"), "fpB", "p", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow(hostID, []byte("B"), "fpB", "p", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||
t.Fatalf("escrow B: %v", err)
|
||||
}
|
||||
if err := st.DeleteHost(hostID, true); err != nil {
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-198 (v0.93.0) — the retention must keep the OFF-SITE data key, not only the PBS key.
|
||||
//
|
||||
// WHAT WAS BROKEN, and why these tests are the ones that would have caught it: host_escrow_superseded
|
||||
// shipped with `blob` (the K-escrow / PBS datastore key) and `identity_blob` was added to host_escrow
|
||||
// by a later ALTER and never to the retained table. The restic REPOSITORY password lives inside
|
||||
// identity_blob. So every supersession retained the whole-guest key and destroyed the off-site data
|
||||
// key — the exact secret the retention exists to preserve — and the destroying act is the escrow
|
||||
// ceremony a rebuilt box asks its customer to run.
|
||||
//
|
||||
// The pre-existing TestSaveHostEscrow_RetainsSuperseded was GREEN throughout, because it asserts the
|
||||
// MECHANISM (a retained row exists, with the old K-blob) and not the CONSEQUENCE (the retained row can
|
||||
// still yield a repository password). These assert the consequence.
|
||||
|
||||
// Scenario A — a re-escrow retains BOTH sealed keys.
|
||||
// RED-PROOF: drop `identity_blob` from demoteCurrentEscrowTx's INSERT/SELECT (production behaviour up
|
||||
// to v0.92.0) → the retained row's identity blob is nil → this FAILS.
|
||||
func TestSaveHostEscrow_RetainsIdentityBlob(t *testing.T) {
|
||||
st := newTestStore(t)
|
||||
const h = "hid-1"
|
||||
oldIdentity := []byte("age-blob-sealing-REPO-PASSWORD-OLD")
|
||||
newIdentity := []byte("age-blob-sealing-REPO-PASSWORD-NEW")
|
||||
|
||||
// Generation 1: the K-escrow, then the identity blob — the real order the escrow PUT uses
|
||||
// (SaveHostEscrow, then SaveHostDRBundle).
|
||||
if _, _, err := st.SaveHostEscrow(h, []byte("k-blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveHostDRBundle(h, oldIdentity, `{"gen":1}`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Generation 2 with a DIFFERENT sealed repo password → supersede.
|
||||
sup, prev, err := st.SaveHostEscrow(h, []byte("k-blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !sup {
|
||||
t.Fatal("a different-passphrase upload must supersede")
|
||||
}
|
||||
if prev != "SHA_OLD" {
|
||||
t.Fatalf("prevResticPwSHA256 = %q, want SHA_OLD (R-197 needs the replaced hash)", prev)
|
||||
}
|
||||
if err := st.SaveHostDRBundle(h, newIdentity, `{"gen":2}`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
retained, err := st.ListSupersededEscrow(h)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(retained) != 1 {
|
||||
t.Fatalf("retained rows = %d, want 1", len(retained))
|
||||
}
|
||||
// THE ASSERTION THIS WHOLE ITEM IS ABOUT: the off-site data key survived the supersession.
|
||||
if retained[0].IdentityBlob == nil {
|
||||
t.Fatal("R-198: the retained row carries NO identity blob — the off-site repository password " +
|
||||
"was destroyed by the ceremony that was supposed to preserve it")
|
||||
}
|
||||
// And it is the PREVIOUS generation's blob, not the one that replaced it. This pins the ordering
|
||||
// dependency named on demoteCurrentEscrowTx: the identity blob is written AFTER SaveHostEscrow, so
|
||||
// the demote sees the old one. If that order ever inverts, the retained bytes would silently be
|
||||
// the new blob filed under the old hash — recoverable-looking and wrong.
|
||||
if !bytes.Equal(retained[0].IdentityBlob, oldIdentity) {
|
||||
t.Fatalf("retained identity blob is not the PREVIOUS generation (got %q) — the demote ran after the overwrite",
|
||||
retained[0].IdentityBlob)
|
||||
}
|
||||
if retained[0].ResticPwSHA256 != "SHA_OLD" || string(retained[0].Blob) != "k-blob-old" {
|
||||
t.Fatalf("retained row is not the old generation: %+v", retained[0])
|
||||
}
|
||||
// Current row unchanged in behaviour: the NEW generation, both blobs.
|
||||
if bundle, berr := st.GetHostDRBundle(h); berr != nil || bundle == nil || !bytes.Equal(bundle.IdentityBlob, newIdentity) {
|
||||
t.Fatalf("current identity blob is not the new one: %+v (%v)", bundle, berr)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario B — deleting a host demotes BOTH blobs too. demoteCurrentEscrowTx is shared by the
|
||||
// re-escrow path and the host-delete path; a shared routine proven through one caller is how a fix
|
||||
// gets believed on a path nobody exercised.
|
||||
// RED-PROOF: fix only the re-escrow caller (e.g. carry the column in SaveHostEscrow's own SQL instead
|
||||
// of in the shared routine) → this FAILS while Scenario A passes.
|
||||
func TestDeleteHost_DemotesIdentityBlob(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
const hostID, cust = "hid-del", "cust-del"
|
||||
identity := []byte("age-blob-sealing-REPO-PASSWORD")
|
||||
if err := s.UpsertHost(&Host{HostID: hostID, CustomerID: cust, APIKey: "k"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := s.SaveHostEscrow(hostID, []byte("k-blob"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.SaveHostDRBundle(hostID, identity, `{}`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if err := s.DeleteHost(hostID, true); err != nil {
|
||||
t.Fatalf("DeleteHost: %v", err)
|
||||
}
|
||||
retained, err := s.ListSupersededEscrow(hostID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(retained) != 1 {
|
||||
t.Fatalf("demoted rows = %d, want 1", len(retained))
|
||||
}
|
||||
if !bytes.Equal(retained[0].IdentityBlob, identity) {
|
||||
t.Fatalf("R-198: host delete demoted custody WITHOUT the identity blob (got %q) — the off-site "+
|
||||
"repository password was destroyed by a host delete", retained[0].IdentityBlob)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario C — a legacy host whose current escrow has NO identity blob supersedes cleanly. The column
|
||||
// is nullable on purpose: a NOT NULL constraint here would make the fix block a ceremony, which is a
|
||||
// worse failure than the one it repairs.
|
||||
func TestSaveHostEscrow_SupersedesWithoutIdentityBlob(t *testing.T) {
|
||||
st := newTestStore(t)
|
||||
const h = "hid-legacy"
|
||||
if _, _, err := st.SaveHostEscrow(h, []byte("k-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// No SaveHostDRBundle — a slice-7-era upload.
|
||||
sup, prev, err := st.SaveHostEscrow(h, []byte("k-new"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
|
||||
if err != nil {
|
||||
t.Fatalf("a supersession of an identity-less escrow must not fail: %v", err)
|
||||
}
|
||||
if !sup || prev != "SHA_OLD" {
|
||||
t.Fatalf("superseded=%v prev=%q, want true/SHA_OLD", sup, prev)
|
||||
}
|
||||
retained, err := st.ListSupersededEscrow(h)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(retained) != 1 {
|
||||
t.Fatalf("retained rows = %d, want 1", len(retained))
|
||||
}
|
||||
if retained[0].IdentityBlob != nil {
|
||||
t.Fatalf("a legacy row must retain a NULL identity blob, got %q", retained[0].IdentityBlob)
|
||||
}
|
||||
}
|
||||
|
||||
// CountCurrentEscrowWithIdentity is the census §8.1 asks for: which hosts hold an identity blob today
|
||||
// and are therefore protected from the next ceremony by this fix. Asserted rather than eyeballed,
|
||||
// because the report quotes its numbers.
|
||||
func TestCountCurrentEscrowWithIdentity(t *testing.T) {
|
||||
st := newTestStore(t)
|
||||
if _, _, err := st.SaveHostEscrow("with-id", []byte("k"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA1"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveHostDRBundle("with-id", []byte("age-blob"), `{}`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := st.SaveHostEscrow("without-id", []byte("k"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA2"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
total, withIdentity, err := st.CountCurrentEscrowWithIdentity()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if total != 2 || withIdentity != 1 {
|
||||
t.Fatalf("census = %d/%d, want 2 total / 1 with identity", withIdentity, total)
|
||||
}
|
||||
}
|
||||
@@ -11,7 +11,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
|
||||
const h = "h1"
|
||||
|
||||
// 1st upload (P_old) — nothing to supersede.
|
||||
sup, err := st.SaveHostEscrow(h, []byte("blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD")
|
||||
sup, _, err := st.SaveHostEscrow(h, []byte("blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -20,7 +20,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
|
||||
}
|
||||
|
||||
// 2nd upload (P_new, DIFFERENT sha) — must supersede + retain the old.
|
||||
sup, err = st.SaveHostEscrow(h, []byte("blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
|
||||
sup, _, err = st.SaveHostEscrow(h, []byte("blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -45,7 +45,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
|
||||
}
|
||||
|
||||
// 3rd upload, SAME sha as current — idempotent (re-ceremony of the same password): NO supersede row.
|
||||
sup, err = st.SaveHostEscrow(h, []byte("blob-new-2"), "fp-new", "zk", "2026-07-16T01:00:00Z", "SHA_NEW")
|
||||
sup, _, err = st.SaveHostEscrow(h, []byte("blob-new-2"), "fp-new", "zk", "2026-07-16T01:00:00Z", "SHA_NEW")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
@@ -15,10 +15,10 @@ func TestDeleteHost_DemotesEscrowNeverDestroys(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// current escrow = SHA_A, one superseded = SHA_OLD (two uploads with different passphrases).
|
||||
if _, err := s.SaveHostEscrow(hostID, []byte("blob-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
|
||||
if _, _, err := s.SaveHostEscrow(hostID, []byte("blob-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.SaveHostEscrow(hostID, []byte("blob-A"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
|
||||
if _, _, err := s.SaveHostEscrow(hostID, []byte("blob-A"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n, _ := s.CountSupersededEscrow(hostID); n != 1 {
|
||||
|
||||
@@ -57,7 +57,7 @@ func seedHostWithArtifacts(t *testing.T, s *Store, hostID, customerID string) {
|
||||
"PK-"+hostID, "ip-"+hostID, hostID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.SaveHostEscrow(hostID, []byte("opaque-escrow"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||
if _, _, err := s.SaveHostEscrow(hostID, []byte("opaque-escrow"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
+91
-21
@@ -374,6 +374,22 @@ func (s *Store) migrate() error {
|
||||
-- "history destroyed" into "history recoverable with the recovery code"). Append-only; the hub
|
||||
-- never decrypts; NO pruning (the blobs are tiny + R-encrypted; custody unchanged). The ACK and
|
||||
-- restore-serving read host_escrow (the CURRENT row) — never this table.
|
||||
--
|
||||
-- THE RULING ABOVE WAS NOT MET FOR TWO MONTHS, AND THIS IS THE RECORD OF IT (R-198, fixed
|
||||
-- v0.93.0). This table shipped with the blob column — the K-escrow, i.e. the PBS datastore key
|
||||
-- — and identity_blob was added to host_escrow LATER (the slice-10D ALTER below), never here.
|
||||
-- The offsite restic REPOSITORY password lives in identity_blob, not in blob. So the retention
|
||||
-- preserved the whole-guest key and silently dropped the off-site data key: precisely the
|
||||
-- secret the reinstall-orphan incident was about. Worse, the copy happens as the new blob
|
||||
-- overwrites the old, so the destroying act was the ESCROW CEREMONY — the exact thing a
|
||||
-- rebuilt box asks its customer to run, on a card promising the old backups stay recoverable.
|
||||
-- Both demo boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08) and their previous
|
||||
-- repository passwords are unrecoverable, recovery code or not.
|
||||
-- identity_blob is now carried (see demoteCurrentEscrowTx, which is still THE ONE row-copy
|
||||
-- routine). Pinned by TestSaveHostEscrow_RetainsIdentityBlob and
|
||||
-- TestDeleteHost_DemotesIdentityBlob — the routine is proven through BOTH of its callers,
|
||||
-- because a shared routine tested through one caller is how a fix gets believed on a path
|
||||
-- nobody exercised. Evidence: audits/RECON-offsite-dr-chain-2026-08-04.md §7.
|
||||
CREATE TABLE IF NOT EXISTS host_escrow_superseded (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
host_id TEXT NOT NULL,
|
||||
@@ -382,7 +398,8 @@ func (s *Store) migrate() error {
|
||||
posture TEXT NOT NULL DEFAULT '',
|
||||
created_at DATETIME NOT NULL,
|
||||
restic_pw_sha256 TEXT NOT NULL DEFAULT '',
|
||||
superseded_at DATETIME NOT NULL DEFAULT (datetime('now'))
|
||||
superseded_at DATETIME NOT NULL DEFAULT (datetime('now')),
|
||||
identity_blob BLOB
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_host_escrow_superseded_host ON host_escrow_superseded(host_id);
|
||||
|
||||
@@ -414,16 +431,28 @@ func (s *Store) migrate() error {
|
||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN identity_blob BLOB`)
|
||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN directive_json TEXT NOT NULL DEFAULT '{}'`)
|
||||
|
||||
// R-198 (v0.93.0) — the SAME column on the RETAINED table. It was added above and not here, and
|
||||
// that omission is what made the retention keep the wrong key for two months (see the comment on
|
||||
// host_escrow_superseded). Additive and tolerated on re-run, exactly like the lines above; it
|
||||
// changes no existing row. Rows superseded BEFORE this ships were written without the identity
|
||||
// blob and their source rows are already overwritten — there is nothing to backfill, and the
|
||||
// v0.93.0 report records that as a looked-at fact rather than a deduction.
|
||||
s.db.Exec(`ALTER TABLE host_escrow_superseded ADD COLUMN identity_blob BLOB`)
|
||||
|
||||
// SLICE 3 (escrow auto-confirm) — sha256 hex of the offsite restic repo password sealed in the
|
||||
// identity blob. The hash of a 256-bit random secret is non-reversible/non-brute-forceable — safe to
|
||||
// store and serve; it lets the controller VERIFY "the escrow covers the CURRENT repo password"
|
||||
// instead of trusting blob-presence. NULL/'' = a legacy or password-less blob (never auto-confirms).
|
||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN restic_pw_sha256 TEXT`)
|
||||
|
||||
// v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at is set when the offsite repo
|
||||
// password is re-issued: the blob then seals a password that no longer opens the repo, so the
|
||||
// hub must stop advertising "ceremony done" and withhold the (now non-matching) restic_pw_sha256
|
||||
// from the auto-confirm ACK. NULL = current; a fresh ceremony (SaveHostEscrow) clears it.
|
||||
// v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at is set when the offsite PROVIDER
|
||||
// credentials are re-issued. ⚠ CORRECTED 2026-08-04 (R-196): it used to say "when the offsite repo
|
||||
// password is re-issued", which nothing in the hub does — the repository password is generated on
|
||||
// the box and never leaves it except sealed under R. The flag is PRECAUTIONARY (the box's re-apply
|
||||
// MAY mint a fresh repository password), not evidence that it changed; the evidential signal is
|
||||
// R-197's offsite_repo_key_changed. While set, the hub stops advertising "ceremony done" and
|
||||
// withholds the possibly-non-matching restic_pw_sha256 from the auto-confirm ACK. NULL = current;
|
||||
// a fresh ceremony (SaveHostEscrow) clears it.
|
||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN stale_at DATETIME`)
|
||||
|
||||
// dr_recipe (SPIKE-dr-recipe-2026-06-16): the secret-free DR reconstruction recipe, stored
|
||||
@@ -2529,6 +2558,11 @@ type HostEscrow struct {
|
||||
// StaleAt (v0.57.0, 2.3) — non-empty when the offsite password was re-issued after this blob was
|
||||
// sealed: the blob is stale (seals a password that no longer opens the repo). Cleared by a fresh ceremony.
|
||||
StaleAt string
|
||||
// IdentityBlob (R-198, v0.93.0) — the age-wrapped identity bundle, which is where the offsite
|
||||
// restic REPOSITORY password lives. Populated by ListSupersededEscrow so a retained blob is
|
||||
// reachable from Go at all; nil for pre-v0.93.0 retained rows and for hosts that never uploaded
|
||||
// one. Opaque: useless without the customer's recovery code, which the hub never holds.
|
||||
IdentityBlob []byte
|
||||
}
|
||||
|
||||
// SaveHostEscrow stores (last-write-wins) the OPAQUE escrow blob for a host. The hub keeps the
|
||||
@@ -2538,16 +2572,35 @@ type HostEscrow struct {
|
||||
// host_escrow_superseded before overwriting the current row (Part B, v0.60.0). A same-sha re-upload
|
||||
// (idempotent re-ceremony of the same password) refreshes the current row and does NOT create a
|
||||
// superseded row.
|
||||
//
|
||||
// R-197 (v0.93.0): it also returns prevResticPwSHA256 — the hash the row being replaced sealed ("" when
|
||||
// no row existed, or when it was a legacy hash-less blob). Both halves of "did the box's offsite DATA
|
||||
// key change?" have been in this database since SLICE 3 and NOTHING compared them; demo-felhom's key
|
||||
// changed on 2026-08-03 and no signal of any kind fired for thirteen hours. The comparison is the
|
||||
// caller's (api.handleHostEscrowPut) because the event needs the customer id; returning the value is
|
||||
// this function's part. The VALUE is a non-reversible hash of a 256-bit random secret and is never
|
||||
// logged, mailed or written to a report — see the caller.
|
||||
// demoteCurrentEscrowTx copies the host's CURRENT host_escrow row (if any) into
|
||||
// host_escrow_superseded as a retained blob, inside the given tx. This is THE ONE escrow row-copy
|
||||
// routine (v0.60.0): SaveHostEscrow uses it to retain a superseded different-passphrase blob before
|
||||
// overwriting, and DeleteHost (v0.60.1) uses it to DEMOTE the current blob to retained custody
|
||||
// instead of destroying it. Returns the number of rows copied (0 when the host has no current row).
|
||||
// The hub never decrypts; custody is unchanged.
|
||||
//
|
||||
// R-198 (v0.93.0): `identity_blob` is copied too. It was omitted from this SELECT for two months, so
|
||||
// every supersession retained the PBS datastore key and destroyed the offsite restic repository
|
||||
// password — the one secret the retention exists to preserve. Copying more opaque bytes gains the hub
|
||||
// NO knowledge: it still has no recovery code and no decrypt path.
|
||||
//
|
||||
// ORDERING THIS DEPENDS ON, stated because it is load-bearing and invisible from here: the identity
|
||||
// blob is written by SaveHostDRBundle AFTER SaveHostEscrow returns (api/handler.go, the escrow PUT),
|
||||
// so at demote time host_escrow still holds the OLD identity blob. If that order ever changes, this
|
||||
// routine silently retains the NEW blob under the OLD blob's hash — pinned by
|
||||
// TestSaveHostEscrow_RetainsIdentityBlob, which asserts the retained bytes are the previous ones.
|
||||
func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
|
||||
res, err := tx.Exec(`
|
||||
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
|
||||
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
|
||||
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at, identity_blob)
|
||||
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now'), identity_blob
|
||||
FROM host_escrow WHERE host_id = ?`, hostID)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
@@ -2555,10 +2608,10 @@ func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
|
||||
return res.RowsAffected()
|
||||
}
|
||||
|
||||
func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, err error) {
|
||||
func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, prevResticPwSHA256 string, err error) {
|
||||
tx, err := s.db.Begin()
|
||||
if err != nil {
|
||||
return false, err
|
||||
return false, "", err
|
||||
}
|
||||
defer func() {
|
||||
if err != nil {
|
||||
@@ -2577,11 +2630,14 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
|
||||
exists = false
|
||||
default:
|
||||
err = scanErr
|
||||
return false, err
|
||||
return false, "", err
|
||||
}
|
||||
if exists {
|
||||
prevResticPwSHA256 = curSHA // R-197: the caller compares; "" = no row or a legacy hash-less blob
|
||||
}
|
||||
if exists && curSHA != resticPwSHA256 {
|
||||
if _, err = demoteCurrentEscrowTx(tx, hostID); err != nil {
|
||||
return false, err
|
||||
return false, prevResticPwSHA256, err
|
||||
}
|
||||
superseded = true
|
||||
}
|
||||
@@ -2598,10 +2654,10 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
|
||||
stale_at = NULL,
|
||||
updated_at = datetime('now')`,
|
||||
hostID, blob, keyFingerprint, posture, createdAt, resticPwSHA256); err != nil {
|
||||
return false, err
|
||||
return false, prevResticPwSHA256, err
|
||||
}
|
||||
err = tx.Commit()
|
||||
return superseded, err
|
||||
return superseded, prevResticPwSHA256, err
|
||||
}
|
||||
|
||||
// CountSupersededEscrow returns how many retained (superseded) escrow blobs the hub holds for a host
|
||||
@@ -2616,7 +2672,7 @@ func (s *Store) CountSupersededEscrow(hostID string) (int, error) {
|
||||
// first. Opaque bytes — the hub never decrypts. Seeds the future guided-recovery flow (R-26).
|
||||
func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
|
||||
rows, err := s.db.Query(`
|
||||
SELECT host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at
|
||||
SELECT host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at, identity_blob
|
||||
FROM host_escrow_superseded WHERE host_id = ? ORDER BY id DESC`, hostID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -2625,7 +2681,7 @@ func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
|
||||
var out []HostEscrow
|
||||
for rows.Next() {
|
||||
var e HostEscrow
|
||||
if err := rows.Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.ResticPwSHA256, &e.UpdatedAt); err != nil {
|
||||
if err := rows.Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.ResticPwSHA256, &e.UpdatedAt, &e.IdentityBlob); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, e)
|
||||
@@ -2633,10 +2689,22 @@ func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3) — called when the offsite repo
|
||||
// password is re-issued, because the blob then seals a password that no longer opens the repo. No-op
|
||||
// when no escrow row exists; idempotent (only stamps the first re-issue since the last ceremony; a
|
||||
// fresh ceremony clears stale_at via SaveHostEscrow's ON CONFLICT).
|
||||
// CountCurrentEscrowWithIdentity returns how many hosts hold a CURRENT escrow row, and how many of
|
||||
// those carry an identity blob — i.e. the population whose offsite repository password R-198's fix
|
||||
// now protects from the next ceremony, and the remainder for whom there is nothing to protect
|
||||
// because no identity blob was ever uploaded. Read-only; no blob or hash leaves this call.
|
||||
func (s *Store) CountCurrentEscrowWithIdentity() (total, withIdentity int, err error) {
|
||||
err = s.db.QueryRow(`
|
||||
SELECT COUNT(*), COALESCE(SUM(identity_blob IS NOT NULL), 0) FROM host_escrow`).Scan(&total, &withIdentity)
|
||||
return total, withIdentity, err
|
||||
}
|
||||
|
||||
// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3) — called when the offsite
|
||||
// PROVIDER credentials are re-issued. ⚠ CORRECTED 2026-08-04 (R-196): it used to say "when the offsite
|
||||
// repo password is re-issued", which no hub path does; see the reasoning at offsite.ReissueCredentials.
|
||||
// The flag is precautionary — the box's re-apply may mint a fresh repository password — not a
|
||||
// measurement that one did. No-op when no escrow row exists; idempotent (only stamps the first
|
||||
// re-issue since the last ceremony; a fresh ceremony clears stale_at via SaveHostEscrow's ON CONFLICT).
|
||||
func (s *Store) MarkEscrowStale(hostID string) error {
|
||||
_, err := s.db.Exec(`UPDATE host_escrow SET stale_at = datetime('now') WHERE host_id = ? AND stale_at IS NULL`, hostID)
|
||||
return err
|
||||
@@ -2665,9 +2733,11 @@ type EscrowStatus struct {
|
||||
IdentityBlobPresent bool `json:"identity_blob_present"`
|
||||
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
|
||||
CreatedAt string `json:"created_at,omitempty"`
|
||||
// Stale (v0.57.0, 2.3) — true when the offsite password was re-issued after the blob was sealed.
|
||||
// Stale (v0.57.0, 2.3) — true when the offsite PROVIDER credentials were re-issued after the blob
|
||||
// was sealed. ⚠ CORRECTED 2026-08-04 (R-196): it used to say "the offsite password was re-issued",
|
||||
// which reads as the repository password and is not what happens; see MarkEscrowStale.
|
||||
// When stale the ResticPwSHA256 is WITHHELD (emptied) so the controller cannot auto-confirm against
|
||||
// a hash that no longer matches the live repo password — the ceremony must run again.
|
||||
// a hash that may no longer match the live repo password — the ceremony must run again.
|
||||
Stale bool `json:"escrow_stale,omitempty"`
|
||||
}
|
||||
|
||||
|
||||
@@ -74,11 +74,11 @@ func seedDeletable(t *testing.T, st *store.Store, customerID string) string {
|
||||
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi-" + hostID, LastReportAt: &long}); err != nil {
|
||||
t.Fatalf("seed host: %v", err)
|
||||
}
|
||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||
t.Fatalf("seed escrow A: %v", err)
|
||||
}
|
||||
// A second save supersedes A → one RETAINED blob + one CURRENT blob before the cascade runs.
|
||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||
t.Fatalf("seed escrow B: %v", err)
|
||||
}
|
||||
if err := st.SaveOneTimeSecret(customerID, "one-time-pw"); err != nil {
|
||||
|
||||
@@ -37,10 +37,10 @@ func seedResettable(t *testing.T, st *store.Store, customerID string) {
|
||||
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi"}); err != nil {
|
||||
t.Fatalf("seed host: %v", err)
|
||||
}
|
||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||
t.Fatalf("seed escrow A: %v", err)
|
||||
}
|
||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||
t.Fatalf("seed escrow B: %v", err)
|
||||
}
|
||||
if err := st.DeleteHost(hostID, true); err != nil { // demotes current → retained; records host_deletions
|
||||
|
||||
@@ -64,7 +64,7 @@ func TestHostDelete_EscrowAckRequired(t *testing.T) {
|
||||
if err := st.UpsertHost(&store.Host{HostID: "esc-host", CustomerID: "c2", APIKey: "k"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := st.SaveHostEscrow("esc-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow("esc-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
@@ -110,7 +110,7 @@ func TestHostDelete_ImpactJSON(t *testing.T) {
|
||||
if err := st.UpsertHost(&store.Host{HostID: "imp-host", CustomerID: "c4", APIKey: "SECRET-KEY"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := st.SaveHostEscrow("imp-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow("imp-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.UpsertGuestFromReport(&store.Guest{GuestID: store.GuestID("imp-host", 100),
|
||||
@@ -159,7 +159,7 @@ func TestHostDelete_HappyPath(t *testing.T) {
|
||||
if err := st.UpsertHost(&store.Host{HostID: "dr-drill", CustomerID: "c5", APIKey: "k"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := st.SaveHostEscrow("dr-drill", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow("dr-drill", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rr := postHostDelete(t, s, "dr-drill", url.Values{
|
||||
|
||||
@@ -130,7 +130,7 @@ func TestHandleHostDetail(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// DR + escrow present (escrow row must exist before the DR bundle UPDATE).
|
||||
if _, err := st.SaveHostEscrow("demo-felhom-01", []byte("opaque-escrow"), "fp", "posture", "2026-06-01T00:00:00Z", ""); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow("demo-felhom-01", []byte("opaque-escrow"), "fp", "posture", "2026-06-01T00:00:00Z", ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveHostDRBundle("demo-felhom-01", []byte("opaque-identity"), `{"v":1}`); err != nil {
|
||||
|
||||
@@ -404,7 +404,7 @@ func TestPBSDR_F14AutoReissueOnAckedDeletion(t *testing.T) {
|
||||
if err := st.UpsertHost(&store.Host{HostID: "peti-00-dead", CustomerID: "peti", APIKey: "oldkey"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := st.SaveHostEscrow("peti-00-dead", []byte("opaque"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||
if _, _, err := st.SaveHostEscrow("peti-00-dead", []byte("opaque"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.DeleteHost("peti-00-dead", true); err != nil {
|
||||
|
||||
Reference in New Issue
Block a user