Compare commits
36 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| c917251eeb | |||
| ee9d9bf203 | |||
| 0dbd954fec | |||
| 975a690fbe | |||
| 5c7d67102d | |||
| d1a8edb332 | |||
| 2a7ac03c47 | |||
| b228fd102d | |||
| 24acc8d816 | |||
| 73fb595e38 | |||
| a0c4b607a6 | |||
| 846253baa8 | |||
| dd089265e8 | |||
| 435f4a5229 | |||
| 9f31956201 | |||
| 40687b0921 | |||
| 91cabdde1b | |||
| 3f2b7bc023 | |||
| d26f49ad68 | |||
| b7c5cfc061 | |||
| f581ac1349 | |||
| 7fff45d688 | |||
| f456835bbc | |||
| e06920d930 | |||
| 813239d8d3 | |||
| a684a81796 | |||
| 71662336aa | |||
| 51871a7ea6 | |||
| 33f0ab2f33 | |||
| 2584dfb938 | |||
| 185661062a | |||
| 967ddf2f44 | |||
| d692c46db0 | |||
| b93ee06abc | |||
| e3187c86d5 | |||
| 311dc06c13 |
+309
@@ -17,6 +17,315 @@
|
|||||||
|
|
||||||
## Standing rulings
|
## Standing rulings
|
||||||
|
|
||||||
|
**S-32 — THREE OF S-31's FOUR MANUAL STEPS ARE CLOSED (2026-08-05, R-204 items 1–3 / R-196).
|
||||||
|
controller v0.198.0 + hub v0.95.0. Read this BEFORE S-31 — it supersedes S-31's steps 2–5.**
|
||||||
|
|
||||||
|
- **Step 1 (the reset code) — CLOSED.** `web.effectiveClaimCode` now calls `settings.ReloadClaimCode()`
|
||||||
|
before applying the settings-vs-config precedence, so a code minted by the SEPARATE
|
||||||
|
`--print-reset-code` process is visible without a restart. **The precedence rule is unchanged** —
|
||||||
|
the defect was the freshness of the settings value, not which source wins. It is a READ-THROUGH on
|
||||||
|
the claim path only, deliberately **not** a watcher, a signal handler or a TTL: a TTL would leave a
|
||||||
|
window in which a superseded code still works, which is worse than the bug. `effectiveClaimCode`
|
||||||
|
now returns an error and **every caller fails closed** (`claimGateActive` keeps the gate UP; an
|
||||||
|
absent settings file is NOT an error — a pre-first-save box legitimately falls back to the
|
||||||
|
controller.yaml bake). **S-31's "restart the controller between minting and claiming" is obsolete.**
|
||||||
|
- **Steps 2–4 (the false staleness) — CLOSED.** `offsite.ReissueCredentials` no longer calls
|
||||||
|
`MarkEscrowStale` and no longer emits `escrow_stale`; `offsite_reissued` is untouched. **The
|
||||||
|
mechanism that made this expensive, recorded because it is not obvious:** `stale_at` makes
|
||||||
|
`GetEscrowStatusForCustomer` WITHHOLD `restic_pw_sha256` — which is the exact value the controller's
|
||||||
|
Scenario-F re-check (`report.EscrowAutoConfirmer.reconcileEscrowed`) compares against its live repo
|
||||||
|
password on every ACK. **The precautionary mark was blinding the measurement that would have told
|
||||||
|
the truth**, leaving the box able to report only the hash-LESS reason, which is false. Coverage
|
||||||
|
after removal: that per-ACK comparison (continuous, box-side) plus R-197's `offsite_repo_key_changed`
|
||||||
|
(edge-triggered, hub-side, at a supersession). **`MarkEscrowStale` is KEPT with no caller** — the
|
||||||
|
flag is still read by the ACK, the operator card and the PBS-DR view, and the right way to set it is
|
||||||
|
a future caller that has MEASURED a change; `TestEscrowStaleMechanism_StillWithholdsAndClears` stops
|
||||||
|
it decaying meanwhile. **The manual `/backup/offbox/confirm-escrow` workaround is no longer needed
|
||||||
|
after a Re-issue.**
|
||||||
|
- **Step 5 (the restore default) — CLOSED.** `restoreScratchOutcomeMsg` states, for `mode=unit`, what
|
||||||
|
came back, that the customer's own files did NOT, and the step that gets them; the wizard's intent
|
||||||
|
card 1 states its scope before the choice. **The `mode=full` size gate is untouched** and pinned as
|
||||||
|
unchanged. **The default stays `unit`:** all three wizard forms set `mode` explicitly, so changing
|
||||||
|
it would alter nothing the customer sees while silently changing a hand-crafted mode-less POST.
|
||||||
|
- **WHAT REMAINS is S-31's step 1-before-the-list: a rebuilt box still cannot obtain an off-site
|
||||||
|
credential unaided** (the one-time password was consumed by its predecessor), so an operator
|
||||||
|
Re-issue is still required. That is **R-193**, it needs an operator ruling, and it is why the
|
||||||
|
capability-map recovery row keeps its *"with a person present"* qualifier.
|
||||||
|
- **Live-validated on demo boxes, endpoint-level** (no browser on DooPlex): reset codes on
|
||||||
|
demo-felhom 9201 with `restarts=0` across the mints; the unit-restore outcome and the full-restore
|
||||||
|
size-gate reveal on demo-hp 9201, using `privatebin` so the drill's `calibre-web` scratch was not
|
||||||
|
touched. **The Part 2 change was NOT fired live on demo-hp** — a Re-issue there was out of scope.
|
||||||
|
|
||||||
|
**S-31 — THE DRILL PASSED: a customer's file survives a machine rebuild and comes back byte-identical.
|
||||||
|
The capability is proven; the customer JOURNEY is four undocumented manual steps (2026-08-04, R-201/R-204).**
|
||||||
|
|
||||||
|
**Proven on hardware, end to end:** demo-hp's controller data volume was destroyed and the sentinel
|
||||||
|
deleted from disk. The recovery code produced `8a9e33aa4da6…` (byte-identical to the pre-wipe on-disk
|
||||||
|
key AND the hub's independent record); it installed on the bare box; the **existing repository opened**
|
||||||
|
(`repo_state: null`, 3 snapshots, 42 026 B = the pre-wipe size — not a fresh history); and the customer
|
||||||
|
restore flow returned the sentinel at `643166269103a25c…`, **byte-identical**. `identity_blob` was
|
||||||
|
unchanged throughout and **no ceremony was run at any point** (superseded rows still 2).
|
||||||
|
|
||||||
|
**But it took four manual steps that are in no design document (R-204), all measured:**
|
||||||
|
1. a rebuilt controller cannot configure its off-site tier — the one-time password was consumed by its
|
||||||
|
predecessor (`no unconsumed offsite password`, R-193);
|
||||||
|
2. the Re-issue that fixes that sets `stale_at` **while `restic_pw_sha256` is unchanged** (R-196);
|
||||||
|
3. a stale escrow makes the hub withhold the hash from the ACK → `EscrowAutoConfirmer` can never flip
|
||||||
|
`pending → escrowed` → `OffboxRunnable` refuses every run;
|
||||||
|
4. it was cleared with the **manual** confirm (`/backup/offbox/confirm-escrow`), NOT a ceremony — a
|
||||||
|
ceremony would have superseded the identity blob and destroyed the recovered key;
|
||||||
|
5. and the restore's **default `mode=unit` returns the recovery unit, not the customer's files** —
|
||||||
|
`mode=full` is what returns documents. A customer following the obvious path gets the wrong thing
|
||||||
|
and is told nothing.
|
||||||
|
|
||||||
|
**Before any of it, a rebuilt box is unclaimed** and the claim gate intercepts every controller
|
||||||
|
endpoint. **The local escape hatch does not work unaided:** `--print-reset-code` writes the new hash to
|
||||||
|
`settings.json` while the running controller keeps its old copy in memory, so the code never matches —
|
||||||
|
**restart the controller between minting and claiming.**
|
||||||
|
|
||||||
|
*Facts a future session needs:*
|
||||||
|
- **A guest rebuild in this fleet is a controller-DATA-VOLUME loss, not a guest reprovision.** The
|
||||||
|
2026-08-03 incident R-193 is filed against ran with guest 9201 up throughout — no `pct destroy`, no
|
||||||
|
`pct restore`, no `--selftest=provision`. Reproduce it that way.
|
||||||
|
- **A good snapshot is not durable against a later bad run on the same day.** `forget --keep-daily 7
|
||||||
|
--group-by host,tags` keeps one per tag per day; a later, worse snapshot evicts a good one.
|
||||||
|
- **Never run a ceremony while a recovery is in flight** — it supersedes the identity blob. Under
|
||||||
|
v0.93.0 the old blob is retained, but nothing serves a superseded blob back (R-199).
|
||||||
|
|
||||||
|
**S-30 — the R-201 drill was PREPARED and HALTED BEFORE THE WIPE (2026-08-04). Nothing was wiped.**
|
||||||
|
|
||||||
|
It stopped at step 4 because the sentinel file was **not in the off-site snapshot** while the run
|
||||||
|
reported `ok` (**R-203**): the deploy-time `${USERDATA_PATH}` root and the backup-time `NamespaceRoot`
|
||||||
|
disagree for an app whose `HDD_PATH` equals `system_data_path` — and the same compose used both roots.
|
||||||
|
Wiping would have destroyed the only copy of the sentinel and proven nothing.
|
||||||
|
|
||||||
|
*Established live on the way, all firsts, and all worth carrying:*
|
||||||
|
- **A rebuilt box's off-site run REFUSES** — `repo_state: orphaned`, the orphan card, and
|
||||||
|
`offbox_repo_orphaned` at the hub. It does **not** silently start a fresh history. Closes R-193's Q3.
|
||||||
|
- **The orphan reset works**: move-aside to `/home/felhom-repo.orphaned-20260804`, never delete.
|
||||||
|
- **demo-hp's pre-rebuild off-site history is permanently unrecoverable** — its key is in superseded
|
||||||
|
row id 3 with `identity_blob` NULL, superseded **four hours before** v0.93.0 fixed the retention.
|
||||||
|
- **Neither pre-existing off-site-toggled app has a restorable file leg** — both are named-volume-only,
|
||||||
|
which the tier tars but the customer restore never unpacks. Any drill needing a recoverable file
|
||||||
|
needs an app with a **mandatory userdata path**; `calibre-web` is now deployed on demo-hp as that
|
||||||
|
fixture, with the sentinel in place.
|
||||||
|
|
||||||
|
*Still not established, unchanged:* **no file has ever been restored from an off-site backup after a
|
||||||
|
wipe**, and Part 0's install path (controller v0.196.0) has never run against a live recovery.
|
||||||
|
|
||||||
|
**S-29 — a box may fetch its OWN sealed recovery blob with its OWN credential; the operator-driven DR
|
||||||
|
path is a separate thing and stays gated (2026-08-04, R-199; hub v0.94.0 + agent v0.125.0 + controller
|
||||||
|
v0.195.0).**
|
||||||
|
|
||||||
|
**The decision, so it is never re-litigated from scratch.** Self-service recovery needs the box to
|
||||||
|
obtain its own sealed package. The only retrieval that existed was operator-gated behind recovery mode
|
||||||
|
and the global key (`dr.go`), which cannot serve a customer-present flow. `GET /hosts/{id}/escrow` now
|
||||||
|
serves that ONE object to its authenticated owner.
|
||||||
|
|
||||||
|
- **Why it is safe:** the blob is opaque; the hub has never held R and has no decrypt path; a wrong
|
||||||
|
code fails closed at age's scrypt KDF (measured live — `exit status 1`, no plaintext, nothing
|
||||||
|
written); the caller already authenticates as this host for its report, desired state, WG
|
||||||
|
registration and PBS token, so this adds no new identity, only a new object.
|
||||||
|
- **What it changes, which is the whole of the trade:** obtaining the blob used to require the
|
||||||
|
OPERATOR to arm recovery mode. It now needs only that box's credential. **Fewer parties are
|
||||||
|
required.** The mitigation is that the capability is AUDITED, not silent: every retrieval raises
|
||||||
|
`escrow_blob_served` (warning, operator-only), written before the bytes leave.
|
||||||
|
- **The override is one constant.** `api.escrowSelfServiceRetrieval` — false re-imposes the
|
||||||
|
recovery-mode requirement on this endpoint and changes nothing else. Do not scatter the condition.
|
||||||
|
- **`handleReEnroll` / `handleGetRestoreDirective` are UNTOUCHED** and must stay so: they rotate the
|
||||||
|
host API key and serve the K-escrow and the directive as well. Different object, different gate.
|
||||||
|
Pinned by `TestEscrowGet_OperatorDRPathUnchanged`.
|
||||||
|
|
||||||
|
**Facts a future session needs before touching this area:**
|
||||||
|
- **The unseal lives in the AGENT and must stay there.** `age` is an agent runtime dependency
|
||||||
|
(`/usr/bin/age`, present on both demo hosts) and is deliberately absent from the controller image;
|
||||||
|
the blob is host-scoped and its only writer is the agent under the per-host key.
|
||||||
|
- **The agent returns ONE field.** Not the tunnel token, not the PBS token, not the WG key — the
|
||||||
|
controller is a trust tier down. Narrowing cost nothing and is not recoverable later.
|
||||||
|
- **R's rules:** in memory for one call, cleared on every path, never on disk, never in argv, never
|
||||||
|
logged, never echoed. The controller CLI reads it from STDIN for exactly that reason. Verified live:
|
||||||
|
0 log lines, 0 files, 0 leftover staging dirs, with a positive control proving the search worked.
|
||||||
|
- **The chain today: links 1–8 walked, 9–11 not.** The KEY comes back. Nothing installs it, reopens a
|
||||||
|
repository with it, or restores a file — R-200's remaining half and R-201.
|
||||||
|
|
||||||
|
**S-28 — the escrow retention now covers the OFF-SITE data key, and customer-present recovery is the
|
||||||
|
accepted design, which makes that retention load-bearing (2026-08-04, R-198/R-197; hub v0.93.0).**
|
||||||
|
|
||||||
|
**The contract changed, and this is the sentence to carry:** when an escrow blob supersedes another,
|
||||||
|
`demoteCurrentEscrowTx` retains **both** sealed keys — the K-escrow `blob` (PBS datastore key) **and**
|
||||||
|
`identity_blob` (the age-wrapped bundle carrying the restic **repository** password). Until v0.93.0 it
|
||||||
|
retained only the first, so the retention preserved the whole-guest key and destroyed the off-site data
|
||||||
|
key, and **the destroying act was the escrow ceremony a rebuilt box asks its customer to run.**
|
||||||
|
|
||||||
|
*Why this is load-bearing rather than tidy:* the operator ruled on 2026-08-04 that **candidate (c) is
|
||||||
|
refused — no repository password is retained on the Proxmox host.** With no host-retained copy, the
|
||||||
|
**customer-present recovery path is the only way back from a rebuild**, and it runs entirely through
|
||||||
|
the retained identity blob. A session that touches escrow custody is touching the single copy.
|
||||||
|
|
||||||
|
*Facts a future session needs before touching this area:*
|
||||||
|
- **The ordering is load-bearing and invisible from the routine.** `SaveHostDRBundle` writes
|
||||||
|
`identity_blob` **after** `SaveHostEscrow` returns (the escrow PUT, `api/handler.go`), so at demote
|
||||||
|
time `host_escrow` still holds the OLD identity blob. Invert that and the retained bytes become the
|
||||||
|
NEW blob filed under the OLD hash — recoverable-looking and wrong. Pinned by
|
||||||
|
`TestSaveHostEscrow_RetainsIdentityBlob`.
|
||||||
|
- **`demoteCurrentEscrowTx` is shared by re-escrow and host-delete.** Both have a test; fixing one
|
||||||
|
caller instead of the routine is a demonstrated red-proof, not a hypothetical.
|
||||||
|
- **Nothing was backfilled and nothing could be** — pre-v0.93.0 retained rows have no identity blob and
|
||||||
|
their source rows are already overwritten. Both demo boxes' pre-2026-08-04 repository passwords are
|
||||||
|
gone permanently, recovery code or not.
|
||||||
|
- **A retained key is not a recovery.** The hub's blob-serving endpoints have no client (R-199), the
|
||||||
|
injection seam has no form (R-200), and the path has never been exercised (R-201). v0.93.0 makes the
|
||||||
|
key survive; it does not hand it back.
|
||||||
|
- **`offsite_repo_key_changed`** (R-197, warning, operator-only, edge-triggered on a supersession where
|
||||||
|
both hashes are known and differ) is the evidential signal that a box's off-site data key moved. It
|
||||||
|
carries **no hash value**. `MarkEscrowStale` is **precautionary**, not evidential — see S-26(a).
|
||||||
|
|
||||||
|
**S-27 — a customer with NO machine ever bound is UNKNOWN, silently; one that was bound and went quiet
|
||||||
|
still alarms (2026-08-04, R-195; hub v0.92.0).** Operator ruling, implemented as
|
||||||
|
`store.HasEverBoundHost` (live `hosts` row OR `host_deletions` tombstone) consulted once at the top of
|
||||||
|
`CheckBackupDeadlines`' loop. **The discriminator is "was a host EVER bound", never "has a report
|
||||||
|
arrived"** — a box that was installed and never phoned home is a real fault. Fail-**open** on a read
|
||||||
|
error; the deferral is logged with its own counter; the R-81 anchored-verdict structure is untouched.
|
||||||
|
|
||||||
|
*The mechanism is worth carrying, because the guard that should have covered this was keyed off the
|
||||||
|
wrong thing:* the existing down-skip reads `StalenessChecker.GetState()`, whose map is seeded from
|
||||||
|
`store.GetCustomers()` — **a query over the `reports` table**. A customer with zero reports is in no
|
||||||
|
row, gets no state, and `GetState()` returns `""` not `"down"`, so **the skip misses exactly the
|
||||||
|
customer it would most obviously cover.** `david` (created 2026-08-01, no machine) e-mailed an
|
||||||
|
`expected_dbdump_missed` ERROR three mornings running; `peti-felhom` — active, host deleted
|
||||||
|
2026-07-15 — does not, because its 482 old reports make it `down`. Generalise it: **a "skip the dead"
|
||||||
|
guard built on evidence of life cannot see something that was never alive.**
|
||||||
|
|
||||||
|
**S-26 — the one-shot secret is the recoverable one; the irreplaceable one is minted fresh on every
|
||||||
|
guest rebuild (2026-08-04, R-193 spike — `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`).
|
||||||
|
No code shipped for it; the decision is the operator's.**
|
||||||
|
|
||||||
|
Two sentences, because they are what every future session needs before touching this area:
|
||||||
|
|
||||||
|
- **Q1 — what is one-shot.** Of the three secrets in the offsite path, the hub's **one-time
|
||||||
|
provider password** is one-shot but freely re-issuable, the box's **SFTP key** is regenerated on
|
||||||
|
every apply by design, and the **restic repository password** — the DATA key, which the agent's own
|
||||||
|
source calls *"irreplaceable"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the only one
|
||||||
|
nothing can restage.
|
||||||
|
- **Q2 — what a rebuild costs.** A rebuilt controller **mints a brand-new repository password**
|
||||||
|
(`WriteOffboxSecrets`, `offbox.go:392`, generates whenever `<DataDir>/offbox/repo_password` is
|
||||||
|
absent) and **no automatic path ever consults the escrowed one** — `InjectOffboxPassword` has exactly
|
||||||
|
one caller in the whole repo, a web form a human pastes into — so **every guest rebuild orphans the
|
||||||
|
previous off-site history.**
|
||||||
|
|
||||||
|
*Measured without touching a box, and the method is reusable:* `host_escrow.restic_pw_sha256` vs
|
||||||
|
`host_escrow_superseded.restic_pw_sha256`. demo-hp `8e03eddf…`→`8a9e33aa…` (15 snapshots / 40.9 MB
|
||||||
|
orphaned); demo-felhom `48741892…`→`c60c8bc7…` (**36 snapshots / 1.14 GB**). **demo-felhom is the half
|
||||||
|
that matters:** R-193 recorded it as having survived the rebuild by luck, and it did — its *delivery*
|
||||||
|
recovered in 76 s off a stale staged secret. **Its repository did not**, and nothing said so for 13 h.
|
||||||
|
*Luck restored the plumbing, not the data.*
|
||||||
|
|
||||||
|
*Three corrections that must not be re-inherited.* **(a)** `ReissueCredentials` does **not** rotate the
|
||||||
|
restic password — R-39's record, `offsite.go:198-201` and `api/handler.go:1067-1069` all say it does,
|
||||||
|
and all three are wrong (→ **R-196**, the eighth entry in `CLAUDE.md`'s table). **(b)** Candidate (b),
|
||||||
|
*"recoverable from escrow at re-bootstrap"*, is **not implementable** — the escrow is R-wrapped and
|
||||||
|
zero-knowledge (D6), so only a customer-present ceremony can open it, which is the manual form that
|
||||||
|
already exists. **(c)** Candidate (a), *"the hub restages automatically"*, **already exists** as
|
||||||
|
`reissueOnReenroll`'s F3 leg and is wired to the wrong event — it sits behind `handleHostEnroll`'s
|
||||||
|
mint-once-reuse short-circuit, and a **guest** rebuild leaves the `hosts` row intact. **Shipping it
|
||||||
|
first would have made both boxes look healthy on 2026-08-04 while their snapshots were orphaned** —
|
||||||
|
strictly worse than the current loud failure.
|
||||||
|
|
||||||
|
*The candidate nobody had named,* and the only one aimed at the actual harm: **the agent survives a
|
||||||
|
guest rebuild**, already receives the repo password over the pinned local API
|
||||||
|
(`POST /escrow/stage-secret`) and already writes it to a fixed 0600 path — it merely **wipes** it after
|
||||||
|
the ceremony. Retaining and serving it back needs no new seam. Its price is one real trade: a copy of
|
||||||
|
the data key at rest on the Proxmox host. **That trade is the operator's to make and the spike does not
|
||||||
|
make it.**
|
||||||
|
|
||||||
|
**S-24 — offsite retention is ep0's, and the box asks for none (2026-08-04, R-191; installer 1.25.0).**
|
||||||
|
R-89 moved offsite pruning server-side and box tokens stay write-only. The 2026-07-26 "two weeks"
|
||||||
|
ruling was not reversed — **where it is ENFORCED moved, and the installer's `keep_last: 2` did not
|
||||||
|
follow**, so every weekly run uploaded fine and then failed the job on a refused prune. The offsite
|
||||||
|
tier now writes `keep_last: 0`, which the agent's existing guard already reads as *never prune from
|
||||||
|
the box*; the local tier is untouched. A gate asserts it. **Before switching it off, ep0 was read:**
|
||||||
|
both namespaces have a prune job at 03:30 keep-last 2 that has run daily since 2026-07-27 (18 tasks,
|
||||||
|
all OK). **If that ever stops, `keep_last: 0` is unbounded growth** — check ep0's prune jobs before
|
||||||
|
assuming the offsite tier is retained.
|
||||||
|
|
||||||
|
**S-25 — a lost storage grant repairs itself, and the repair is REPORTED (2026-08-04, R-190; agent
|
||||||
|
v0.124.1).** On a missing grant the agent runs the existing root wrapper (`felhom-backup-target-apply
|
||||||
|
grant <id>`, already sudoers-permitted for any id) and re-reads once — the pbsdr R-22 shape. Bounded
|
||||||
|
at one attempt per tier per hour.
|
||||||
|
|
||||||
|
*The record is the point, and getting it wrong is easy.* A repair that leaves only `ok` behind
|
||||||
|
destroys the only evidence a permission vanished. v0.124.0 reported degraded for "one cycle" and
|
||||||
|
**that failed in production while every unit test passed**: `probeAll` is called independently by the
|
||||||
|
self-check log and by the collector, the repairing call was the log's, and the host-report three
|
||||||
|
seconds later sent `ok`. The fix is a latch on **time** (20 min > the 900 s report interval), so at
|
||||||
|
least one report must carry it. The explanation rides `Feature`, because that is the field the hub
|
||||||
|
interpolates into the operator's e-mail — `Reason` does not travel. **Nothing new was built:** the
|
||||||
|
hub's existing ok→degraded→ok edge is the channel.
|
||||||
|
|
||||||
|
*Caveat measured live:* **PVE caches permissions** (~40 s and ~16 min observed), so detection lags the
|
||||||
|
loss and a single permission read is a lagging indicator → R-194.
|
||||||
|
|
||||||
|
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
|
||||||
|
(2026-08-04). Scope: those two boxes, not the fleet.**
|
||||||
|
|
||||||
|
Four SCHEDULED runs overnight, none triggered by hand: demo-felhom host **83.8 s** / offsite
|
||||||
|
**540.4 s**; demo-hp host **109.3 s** / offsite **300.1 s**. Every one restored into a scratch guest,
|
||||||
|
booted, verified and destroyed itself.
|
||||||
|
|
||||||
|
*What this closes.* Until yesterday every live restore-proof this project held was on the OFFSITE
|
||||||
|
tier. The on-box tier — the one an ordinary recovery uses — had never been proven on either box, and
|
||||||
|
not because it failed: the agent could not read the storage it lives on (R-185), so it never saw an
|
||||||
|
archive there to test.
|
||||||
|
|
||||||
|
*What was observed for the first time.* Both boxes had BOTH tiers due simultaneously. Never-proven
|
||||||
|
sorts first, so each took its host tier, deferred the offsite one, and picked that up on the next
|
||||||
|
evaluation six hours later — R-86's ordering and the one-heavy-operation gate, working together,
|
||||||
|
unsupervised. The host-tier proof then reached the hub through R-189's merge, which is that path
|
||||||
|
carrying a host-tier entry for the first time.
|
||||||
|
|
||||||
|
*The asymmetry worth remembering:* a host-tier restore is **83–109 s**; an offsite one is
|
||||||
|
**300–540 s**. The tier that matters for an ordinary recovery is also the cheapest to prove.
|
||||||
|
|
||||||
|
**S-21 — an empty listing cannot distinguish FORBIDDEN from NEWBORN, so the box asks the permission
|
||||||
|
question directly (2026-08-03, R-185; agent v0.123.0 + installer 1.24.0).**
|
||||||
|
|
||||||
|
*The defect.* On both demo boxes the agent's token had `FelhomAgentStore` on `local`, `local-lvm` and
|
||||||
|
`felhom-pbs` and **not** on `felhom-backup` — the storage the same installer configured as
|
||||||
|
`local_backup_target`. The content API answered `{"data":[]}` through the token while root listed
|
||||||
|
three archives. `pickForThisRun` skipped the tier as *"no settled archive yet"*, which is exactly
|
||||||
|
what a brand-new tier reports, so the host tier was never restore-testable and nothing said so.
|
||||||
|
|
||||||
|
*The rule.* The permission question has a definite answer where the listing does not. `Permissions`
|
||||||
|
reads `/access/permissions?path=/storage/<target>` **as the agent's own token** — asking as root
|
||||||
|
answers a different question and always says yes — and one `capability.Status` per configured tier
|
||||||
|
reports it. The probed set comes from `BackupTiers()`, never a fixed list: a hardcoded probe list is
|
||||||
|
the defect reproduced inside the fix.
|
||||||
|
|
||||||
|
*The measured trap, because the obvious reading is wrong.* An ungranted path answers **neither empty
|
||||||
|
nor 403**: it carries the privileges inherited from the box-wide `/` grant
|
||||||
|
(`Sys.Audit, SDN.Use, Datastore.Audit`). Testing for path-presence, or for `Datastore.Audit`, reports
|
||||||
|
a blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**, and re-measuring is required
|
||||||
|
before that constant is ever changed.
|
||||||
|
|
||||||
|
*Criticality, weighed once.* Critical, because the hub alerts only on critical and a non-critical
|
||||||
|
entry would ride the report and alert nobody. **Except** the `local` fallback target, which
|
||||||
|
host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it
|
||||||
|
does not page — turning an ordinary documented setup into an alert is how a signal becomes something
|
||||||
|
an operator archives unread. It never consults content, so it cannot alarm on a newborn tier by
|
||||||
|
construction, and it never reports ok when it could not ask.
|
||||||
|
|
||||||
|
**S-22 — the installer's Scenario-F arm must finish the job, not just leave the definition alone
|
||||||
|
(2026-08-03, R-185).** `configure_backup_target` has two arms. Case A creates the storage and grants
|
||||||
|
in the same breath. The reuse arm — *"the target already exists"* — returned **without granting**, and
|
||||||
|
that, not `PVE_STORAGES`, is where the drift came from: a box whose target pre-dated the install
|
||||||
|
(the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token
|
||||||
|
could not read. The reuse arm now ensures the ACL through the same guarded wrapper. **Scenario F is
|
||||||
|
unviolated** — the storage DEFINITION is untouched, and granting the role the agent is supposed to
|
||||||
|
have on the target this script is about to write into `agent.json` is finishing the job, not
|
||||||
|
retargeting the box. `$BACKUP_TARGET_ID` stays OUT of `PVE_STORAGES`: that list is granted a step
|
||||||
|
before the target is resolved, and `--acl-storages` entries are preflight-checked for existence.
|
||||||
|
A gate asserts every arm that resolves the target also grants on it.
|
||||||
|
|
||||||
**S-19 — a restore-test PROOF is durable and reportable; a FAILURE is neither, and that asymmetry is
|
**S-19 — a restore-test PROOF is durable and reportable; a FAILURE is neither, and that asymmetry is
|
||||||
the design (2026-08-03, R-189; agent v0.122.0).**
|
the design (2026-08-03, R-189; agent v0.122.0).**
|
||||||
|
|
||||||
|
|||||||
@@ -1,137 +1,160 @@
|
|||||||
# REPORT — R-86 Part 2: the staleness window learns each tier's rhythm (2026-08-03)
|
# REPORT — R-196 / R-204 item 2 (hub v0.95.0), 2026-08-05
|
||||||
|
|
||||||
> **A LATER SESSION THE SAME DAY (R-189 · R-188 · R-186) touched this repo for REGISTERS AND DOCS
|
**A re-issue no longer marks a healthy escrow stale.** One behaviour change, one register closed, and
|
||||||
> ONLY** — `OPEN-ITEMS.md`, `STATUS.md`, `CONTEXT.md` (S-19, S-20) and the capability map. **No hub
|
the coverage claim proved rather than assumed. The controller's half of R-204 (items 1 and 3) is
|
||||||
> code changed and the hub was not bumped**: the defect was the agent no longer sending
|
`felhom-controller` v0.198.0.
|
||||||
> `restore_tests[]`, not the hub failing to read them. That session's report is
|
|
||||||
> `felhom-agent/REPORT.md`. What follows is the last hub IMPLEMENTATION, v0.91.1, which is what this
|
|
||||||
> file is for.
|
|
||||||
|
|
||||||
**Date:** 2026-08-03 · **Repo:** `felhom.eu` hub **v0.90.1 → v0.91.0 → v0.91.1**
|
|
||||||
(`323f45a`, `046df30`), manifests `687fedd`, `ff2655c`. Ships **with** `felhom-agent` v0.121.0, not
|
|
||||||
after it. Ruling recorded: **ep0 is Tier 2, protected**.
|
|
||||||
|
|
||||||
## 1. Baselines, re-read on arrival
|
## 1. Baselines, re-read on arrival
|
||||||
|
|
||||||
| Repo | `main` @ commit | Version | Matched §1? |
|
| Repo | Expected | Found |
|
||||||
|---|---|---|---|
|
|---|---|---|
|
||||||
| `felhom.eu` | `e34b614e5b65` | hub CHANGELOG top entry `v0.90.0`; `manifests/hub.yaml` ran **`0.90.1`** | **Yes — and the discrepancy was real** |
|
| `felhom.eu` | `2a7ac03c4726` / hub v0.94.0 (deployed `felhom-hub:0.94.0`) | **exact match**, tree clean, `HEAD == origin/main` |
|
||||||
|
|
||||||
The missing `v0.90.1` CHANGELOG entry is backfilled. It was built, deployed and recorded in
|
**§3.2's landmark had DRIFTED, and the drift changed the work.** The task described
|
||||||
`REPORT.md` + the R-182 row on 2026-08-03 (`f21e7ca`) but never given a CHANGELOG entry — and
|
`offsite.go:222-231` under a known-consequence comment saying the mark was made on a false premise.
|
||||||
`REPORT.md` is overwritten every session, so the per-repo history under-reported what was running.
|
That comment had already been rewritten by the R-196 comment-correction commit, and the version on
|
||||||
|
`main` gave a **non-false** ground for the mark: *"the box's re-apply may mint a fresh repository
|
||||||
|
password (it does exactly that whenever `<DataDir>/offbox/repo_password` is absent — the
|
||||||
|
guest-rebuild shape)"*. So the question was no longer "delete a comment's lie" but "is the shape it
|
||||||
|
guards actually covered elsewhere?" — which is Scenario D, and §8.2 says to stop and report if it is
|
||||||
|
not. It is; §3 below is the evidence.
|
||||||
|
|
||||||
## 2. Why Part 2 could not ship later
|
## 2. What changed
|
||||||
|
|
||||||
The agent now proves a tier **once per archive generation**, so a tier backed up weekly is proved
|
`offsite.ReissueCredentials` no longer calls `MarkEscrowStale` and no longer emits the `escrow_stale`
|
||||||
weekly — correctly, in perfect health. `restoreProvenStaleAfter` was a flat 7 days and its own comment
|
event. **`offsite_reissued` is untouched** and still fires on every re-issue. The known-consequence
|
||||||
derived that number *from the cadence R-86 removes*.
|
comment is rewritten to record what was done, when, and why — with the disagreement below stated in
|
||||||
|
it rather than absorbed.
|
||||||
|
|
||||||
The brief said a weekly tier would "sit exactly on that line". That is literally true, and the
|
**What the mark actually cost, established mechanically rather than asserted** (this is why it was a
|
||||||
arithmetic is worth stating: proofs land at `archive + settle`, so just before the next proof the
|
blocker and not a nit):
|
||||||
current one is **exactly one interval old** — 168 h against a 168 h window, and `age > window` is false
|
|
||||||
by a hair. It did not sit near the line, it sat **on** it. Any ordinary delay — a backup landing late,
|
|
||||||
one evaluation deferred behind a running backup — tips a healthy tier into a nightly alarm.
|
|
||||||
|
|
||||||
## 3. The window as derived
|
1. `stale_at` set → `GetEscrowStatusForCustomer` **withholds** `restic_pw_sha256` from the report ACK.
|
||||||
|
2. With no hash, the controller's SLICE-3 auto-confirm returns early and cannot flip
|
||||||
|
`pending → escrowed`.
|
||||||
|
3. `OffboxRunnable() = OffboxConfigured() && EscrowState == "escrowed"` → **every off-site backup
|
||||||
|
refused**, indefinitely, on a box whose repository key was never in doubt.
|
||||||
|
4. The customer is told to re-run the recovery ceremony — which mints a new recovery code and
|
||||||
|
supersedes the sealed blob. **During a recovery that is the one act that would have destroyed the
|
||||||
|
key just recovered.**
|
||||||
|
|
||||||
```
|
## 3. Scenario D — the evidence that the removed marking is covered
|
||||||
interval = declaredArchiveInterval(tier) // host 26h, offsite 8d
|
|
||||||
if observedOK && observed > interval { interval = observed } // observation may only WIDEN
|
|
||||||
window = clamp(4 * interval, floor 7d, cap 12d)
|
|
||||||
```
|
|
||||||
|
|
||||||
- **4 generations** — the settle generation plus ~3 missed opportunities: deliberately the same
|
The mark was precautionary and aimed at ONE shape: a re-issue followed by a box that mints a fresh
|
||||||
tolerance the flat constant expressed. The change is to the *rhythm*, not to the patience.
|
repository password (the guest-rebuild shape, where `offbox/repo_password` is absent). That shape is
|
||||||
- **The declared rhythm is `backupStaleAfter` (26 h) / `offsiteBackupStaleAfter` (8 d)** — the very
|
measured in two independent places, and **the mark was blinding one of them**:
|
||||||
thresholds the backup-freshness checker already judges those tiers against. Not a second opinion:
|
|
||||||
if the two checkers disagreed about how often a tier receives an archive, one would be alarming on
|
|
||||||
the other's model.
|
|
||||||
- **The observation** comes from reports the hub already holds: `pbs_snapshots[]` plus successful
|
|
||||||
`backups[]` attributed by **target type** (slice-A.4 — a PBS-targeted vzdump appears in *both*
|
|
||||||
arrays, and classifying by array membership would attribute an offsite archive to the host tier).
|
|
||||||
Mean gap over distinct archive times; fewer than two generations ⇒ unobservable.
|
|
||||||
- **Floor 7 d** — the old constant kept as a floor, so no tier is judged more tightly than before.
|
|
||||||
- **Cap 12 d** — strictly inside the 2-week offsite retention with two days to spare, so a tier is
|
|
||||||
never called stale against an archive PBS has already pruned.
|
|
||||||
|
|
||||||
| tier | interval used | window | proof age in health | verdict |
|
- **Continuous, box-side — the real coverage.** `report.EscrowAutoConfirmer.reconcileEscrowed`
|
||||||
|---|---|---|---|---|
|
(controller) compares the ACK's sealed `restic_pw_sha256` against the box's CURRENT local repo
|
||||||
| daily host | 26 h declared | 4×26 h → **floor 168 h** | ≤ ~54 h | OK — numerically unchanged from before |
|
password on **every report ACK**, raising the stale flag, the customer card and the
|
||||||
| weekly offsite | 8 d declared (7 d observed loses to it) | 4×8 d → **cap 288 h** | ≤ ~198 h | OK, 3.75 d of margin |
|
„create a new recovery code" CTA on a mismatch. That is a **measurement**, not a guess, and it is
|
||||||
| newborn offsite | 8 d declared | **288 h** | — | UNKNOWN until the anchor passes |
|
continuous rather than edge-triggered. Pinned by the controller's
|
||||||
|
`TestEscrowStale_MismatchWarnsOnceAndFlags` — re-run green this session.
|
||||||
|
**And step 1 above was blinding it:** a stale flag empties the very hash that comparison needs, so
|
||||||
|
the box could only reach the hash-LESS branch and report *"the hub's current blob carries NO
|
||||||
|
password hash"* — which is false. Removing the mark restores the true signal.
|
||||||
|
- **Edge-triggered, hub-side.** R-197's `offsite_repo_key_changed` fires from
|
||||||
|
`api.maybeEmitRepoKeyChanged` on a proven hash difference across a supersession and pages the
|
||||||
|
operator. **Red-proved:** removing the `maybeEmitRepoKeyChanged` call from `handleHostEscrowPut`
|
||||||
|
made `TestEscrowPut_ChangedRepoKey_RaisesSignal` fail with *"the repository key demonstrably changed
|
||||||
|
and NO signal was raised"*, while the two silence tests stayed green.
|
||||||
|
|
||||||
## 4. v0.91.1 — a flaw v0.91.0 shipped with, found by checking the live box
|
**Disagreement recorded, per the R-96 standing rule.** Scenario D as written asks that a real key
|
||||||
|
change also *"mark the escrow stale"*. **It must not, and nothing was changed to make it.** The hub
|
||||||
|
learns of a real change at the instant a supersession **seals the new password** — i.e. when the
|
||||||
|
escrow is at its freshest. Marking it stale there would ask the customer for a ceremony to fix the
|
||||||
|
ceremony that just ran. The correct consequence at that instant is the operator alarm, which is
|
||||||
|
exactly what R-197 already does. This is recorded in the code comment, the CHANGELOG and OPEN-ITEMS,
|
||||||
|
not only here.
|
||||||
|
|
||||||
demo-felhom's offsite tier holds two retained snapshots, `2026-07-27T19:55:41Z` and
|
## 4. `MarkEscrowStale` is kept with no caller — deliberately
|
||||||
`2026-07-28T04:49:43Z` — **8 h 54 m apart**, because one is a healing artefact and the other a real
|
|
||||||
weekly run. A mean-gap estimate therefore reads a **weekly** tier as nine-hourly: ×4 = 36 h, the floor
|
|
||||||
lifts it to 168 h, and a weekly tier proved weekly reaches ~8.25 days of proof age. **The false alarm
|
|
||||||
this whole task exists to prevent would have returned within a week, on the box it had just shipped
|
|
||||||
to.**
|
|
||||||
|
|
||||||
Fixed by `max(observed, declared)`. A gap *shorter* than the declared rhythm is routine and means
|
Per task §5 it was not to be modified, and it is not deleted either. The `stale_at` flag remains live
|
||||||
nothing (a retry, a manual run, a heal, a catch-up after an outage); a gap *longer* than it is real
|
and correct — read by the ACK, the operator config card and the PBS-DR view — and the right way to
|
||||||
information. **Cost, stated rather than hidden:** a tier that truly runs faster than its declared
|
set it is a **future EVIDENTIAL caller** that has measured a key change rather than guessed at one.
|
||||||
rhythm gets a wider window than it needs, i.e. a slower `restore_test_stale`. Right direction for a
|
Its doc comment now says so plainly instead of naming a caller that no longer exists, and
|
||||||
signal meaning *unverified* — *broken now* is `restore_test_failed`, immediate and untouched.
|
`TestEscrowStaleMechanism_StillWithholdsAndClears` keeps the mechanism from decaying to inert while
|
||||||
|
nothing writes it (the seam-built-but-never-wired shape, in reverse).
|
||||||
|
|
||||||
## 5. Kept, because it was earned
|
The schema comment and `EscrowStatus.Stale`'s comment were corrected the same way — each of the three
|
||||||
|
previously asserted a writer that is now gone.
|
||||||
|
|
||||||
- **Absence is UNKNOWN** until an anchored window has passed (R-81's structure, unchanged).
|
## 5. Files modified
|
||||||
- **The stale signal stays edge-triggered** (`staleStates` untouched).
|
|
||||||
- **`restore_test_failed` and `restore_test_stale` stay DISTINCT** — one says your recovery is broken,
|
| File | Change |
|
||||||
the other that it is unverified, and the second is the one that quietly becomes the first.
|
|---|---|
|
||||||
- **Every reason string now states the window it was judged against** — R-100's corollary: when a
|
| `hub/internal/offsite/offsite.go` | the pessimistic `MarkEscrowStale` + `escrow_stale` event removed; comment rewritten to record the change, the coverage and the disagreement |
|
||||||
verdict changes what it counts from, the alarm text must change with it, or an operator reads
|
| `hub/internal/offsite/offsite_test.go` | `TestReissue_InvalidatesEscrow` **replaced in place by its exact inverse** `TestReissue_DoesNotMarkAHealthyEscrowStale`; new `TestEscrowStaleMechanism_StillWithholdsAndClears` |
|
||||||
"limit 168h" under a tier judged at 288 h and dismisses a true alarm.
|
| `hub/internal/store/store.go` | three comments corrected (`MarkEscrowStale`, the `stale_at` schema note, `EscrowStatus.Stale`) — each had named a writer that no longer exists |
|
||||||
- **The window READ is unchanged in cost** (14 days) — enough to find proof inside the widest window
|
| `manifests/hub.yaml` | image tag `0.94.0` → `0.95.0` |
|
||||||
and to see two generations of a weekly tier.
|
| `hub/CHANGELOG.md`, `CONTEXT.md`, `STATUS.md`, `documentation/…` | v0.95.0 entry; ruling **S-32**; the register and architecture updates below |
|
||||||
|
|
||||||
|
**Commits on `main`:** `d1a8edb` (behaviour + tests + comments) · `5c7d671` (CHANGELOG) ·
|
||||||
|
`975a690` (manifest bump).
|
||||||
|
|
||||||
|
**Deploy:** built + pushed `felhom-hub:0.95.0`, bumped `manifests/hub.yaml`, pushed, then a
|
||||||
|
**deliberate ArgoCD hard-refresh + sync** (auto-sync stays off; no `kubectl set image` anywhere).
|
||||||
|
Result: app `felhom` **Synced / Healthy**, `deploy/hub` rolled out, running image
|
||||||
|
`gitea.dooplex.hu/admin/felhom-hub:0.95.0`, startup log clean (offsite provisioning, pool-box checker,
|
||||||
|
PBS-DR reconciler and all six host checkers initialised; `Listening on :8080`).
|
||||||
|
|
||||||
## 6. Tests and red-proofs
|
## 6. Tests and red-proofs
|
||||||
|
|
||||||
| Test | Asserts | Mutation | Observed |
|
Green gate: `cd hub && go build ./... && go vet ./... && go test ./...` — **rc=0**.
|
||||||
|
`python3 scripts/repo_gates.py --fast` — **all five gates OK**.
|
||||||
|
|
||||||
|
| Test | Result | Red-proof — what was mutated | Outcome |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `TestRestoreTest_HealthyWeeklyTierIsNeverStale` | a healthy weekly tier **with jitter** is never stale across 6 weeks | window pinned flat at `restoreProvenWindowFloor` | **FAIL** — `week 0: … proof age 172h0m0s, window 168h0m0s … "limit 168h0m0s"` |
|
| `TestReissue_DoesNotMarkAHealthyEscrowStale` (C) | PASS | restored the pessimistic `MarkEscrowStale` block in `ReissueCredentials`, exactly as it was | **FAILED** — *"a re-issue marked a HEALTHY escrow stale…"* |
|
||||||
| `TestRestoreProvenWindow_Contract` | floor, cap, declared fallback, **observation may only widen** | `observed > interval` → `observed > 0` | **FAIL** — `window(pbs, observed=8h54m0s ok=true) = 168h0m0s, want 288h0m0s` |
|
| `TestEscrowStaleMechanism_StillWithholdsAndClears` | PASS | same mutation | **stayed GREEN** — correctly: the mutation restores a *caller*, not a break in the mechanism. That split is the evidence Scenario C's assertion is about the caller and not the flag. |
|
||||||
| `TestObservedArchiveIntervals_FromReports` | rhythm observed per tier; a PBS vzdump is **not** counted into the host tier | — | pass |
|
| `TestEscrowPut_ChangedRepoKey_RaisesSignal` (D) | PASS | removed the `maybeEmitRepoKeyChanged` call from `handleHostEscrowPut` | **FAILED** — *"the repository key demonstrably changed and NO signal was raised"* |
|
||||||
| `TestRestoreTest_WeeklyTierThatStopsBeingProvedStillAlarms` | a window that never fires is a deletion, not a fix | — | pass |
|
| `TestEscrowPut_UnchangedRepoKey_Silent`, `TestEscrowPut_HashlessSupersession_NoSignal` | PASS | same | stayed green — the detector's silence branches are independent |
|
||||||
| existing R-85/R-81 suite | anchored UNKNOWN, edge-trigger, distinct events, newborn silence | — | pass, unchanged |
|
| controller `TestEscrowStale_MismatchWarnsOnceAndFlags` | PASS | — (cited as the continuous-coverage pin) | — |
|
||||||
|
|
||||||
**A hollow test caught by its own red-proof.** The first Scenario-G fixture used a perfectly regular
|
Scenario C asserts the **consequence** (the ACK still carries the hash, so auto-confirm can proceed)
|
||||||
weekly tier and **PASSED under the flat-window mutation**, because a regular tier's proof age lands on
|
rather than the mechanism (that a function was not called), because the hash is what the drill's
|
||||||
exactly 168 h and `>` lets it through. It would have shipped Part 1 and its false alarm together while
|
blockage actually turned on. It also asserts that `offsite_reissued` still fires — removing a false
|
||||||
looking like proof. The jitter is what makes it a test — and it is also the truth about the old
|
alarm must not remove the true notice.
|
||||||
constant.
|
|
||||||
|
|
||||||
## 7. Deployment
|
## 7. Live validation
|
||||||
|
|
||||||
| Step | Evidence |
|
**Per task §12 point 5, a live re-issue was NOT run, and must not have been on demo-hp** — it would
|
||||||
|---|---|
|
have been a credential rotation on the box holding the drill's evidence. Part 2 is proved by test and
|
||||||
| Images built + pushed | `felhom-hub:0.91.0`, then `0.91.1` |
|
by the deployment being live and healthy. The controller-side halves of R-204 were validated live and
|
||||||
| Manifest bumped in git | `687fedd`, `ff2655c` — never `kubectl set image` |
|
are reported in `felhom-controller/REPORT.md`.
|
||||||
| ArgoCD deliberate sync | `sync=Synced health=Healthy` |
|
|
||||||
| Running image | `gitea.dooplex.hu/admin/felhom-hub:0.91.1` |
|
|
||||||
| Startup log | checkers initialized; `Listening on :8080` |
|
|
||||||
|
|
||||||
## 8. Part 3 — the ep0 ruling, recorded
|
## 8. Register and documentation
|
||||||
|
|
||||||
`runbooks/target-selection.md` had carried *"D-d did not name ep0 either way. Confirm it explicitly"*
|
- **`OPEN-ITEMS.md`** — **R-196 → CLOSED (hub v0.95.0)**; **R-204 → items 1–3 CLOSED, item 4 OPEN
|
||||||
for two days. **Operator ruling, 2026-08-03: ep0 is protected.** Recorded three ways — the tier table
|
(→ R-193)** with its dependency named. The header block is updated and states explicitly that
|
||||||
names it, the standing question is gone, and its per-machine section is retitled **Tier 2, PROTECTED**.
|
**R-202**, **the ~1.2 GB orphaned-ciphertext deletion** and **R-198's retention (still UNIT-PROVEN
|
||||||
`CONTEXT.md` S-18 records that this **extends D-d's protected list to three machines** (DooPlex,
|
ONLY — the second deliberate wipe is the next item)** all **remain open**, so nothing is presumed
|
||||||
Peti's cluster, ep0) and that it is a *classification*, not new prohibitions: destroying datastores,
|
closed by association. R-201 is recorded as PASSED. **R-204 is still the highest ID; nothing new
|
||||||
prune jobs, tunnel config or nftables rules was already forbidden by what it would destroy, and **the
|
was minted.**
|
||||||
ordinary off-site read a restore-test performs remains permitted** — which this session then exercised.
|
- **`architecture/00-capability-map.md`** — the recovery row now says three of the four crutches are
|
||||||
|
gone, names the fixes and their evidence, and states that **item 4 (R-193) is the one that remains**
|
||||||
|
and is why the row **keeps its "with a person present" qualifier**. The **R-199 back-pointer was
|
||||||
|
already present** on the adjacent key-recovery row (added when that row was last corrected), so it
|
||||||
|
needed no further action — verified, not assumed.
|
||||||
|
- **`architecture/07-backup-architecture.md`** — **new §7.0, "What a customer can and cannot do
|
||||||
|
ALONE"**: the four steps in a table with what each cost and its status, plus the honest current
|
||||||
|
answer. This is the section a future reader will use to answer the question.
|
||||||
|
- **`documentation/backlog/ROADMAP.md`** — R-196 and R-204 collapsed per the coupling rule.
|
||||||
|
- **`CONTEXT.md`** — new standing ruling **S-32**, which supersedes S-31's steps 2–5 and carries the
|
||||||
|
blinding mechanism, the fail-closed rule and the "no TTL" reasoning forward.
|
||||||
|
- **`STATUS.md`** — rewritten to **one screen** (191 → ~90 lines) per its own header. It also had a
|
||||||
|
corrupted, half-overwritten "What we're working on" section left from the drill session, which is
|
||||||
|
now gone. Next item stated as the retention drill.
|
||||||
|
|
||||||
## 9. Registers and docs
|
**CI:** felhom.eu runs **154** (`5c7d671`, code) and **155** (`975a690`, manifest) — both success.
|
||||||
|
`--no-verify` was **not** used; the pre-push gate ran and passed on every push.
|
||||||
|
|
||||||
- **R-86 → CLOSED**, shipped + proven live, with the trap in its own wording recorded on the row.
|
## 9. Observations — noticed, NOT acted on
|
||||||
- **R-87 → re-ranked UP**: R-86 built most of what it waited for. What remains is restic-specific.
|
|
||||||
- **R-185 / R-186 / R-187 → filed** (grep established 185–187 free; R-184 was the highest in use).
|
- **`allowedEventTypes` still lists `escrow_stale`**, which after this change has **no producer** in
|
||||||
- `ROADMAP.md` R-86 collapsed, keeping the reasoning and **correcting the shape the row itself
|
either repo. It is inert rather than harmful; removing an allowlist entry is a behaviour change and
|
||||||
proposed** — which was the never-fires version.
|
is out of this session's scope.
|
||||||
- `STATUS.md` rewritten for the operator, trimmed back to one screen.
|
- `MarkEscrowStale` is now dead code by call-graph. Kept on purpose (§4 above) — but if a future
|
||||||
- `07-backup-architecture.md` §3 carries the new contract (S-1); `00-capability-map.md`'s restore-test
|
session's linter or cleanup pass proposes deleting it, the reason it exists is in its doc comment
|
||||||
row upgraded with its live citation; `CONTEXT.md` gains S-17 (the rule + the trap + the config key)
|
and in the test that exercises it.
|
||||||
and S-18 (ep0).
|
- `/` on DooPlex is at **86%** used — under the 90% abort line, but worth watching before large builds.
|
||||||
|
|||||||
@@ -75,8 +75,11 @@
|
|||||||
| `(*Store).GetHostRecoveryMeta` + `(*Server).handleHostRevealRecoveryCredential` | hub/internal/store/host_recovery.go · hub/internal/web/hosts.go | `(hostID) (*HostRecoveryMeta, error)` · `POST /hosts/{id}/reveal-recovery-credential` | The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) | **Use `GetHostRecoveryMeta` on any page-render path** — its struct and its `SELECT` both omit the `secret` column, so it cannot leak one; `GetHostRecoveryCredential` (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE `recovery_credential_revealed` event via `SaveEvent` and calls NO dispatcher (the `handleRequestLogTail` shape). `api/handler.go handleAdminGetRecoveryCredential` (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133. |
|
| `(*Store).GetHostRecoveryMeta` + `(*Server).handleHostRevealRecoveryCredential` | hub/internal/store/host_recovery.go · hub/internal/web/hosts.go | `(hostID) (*HostRecoveryMeta, error)` · `POST /hosts/{id}/reveal-recovery-credential` | The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) | **Use `GetHostRecoveryMeta` on any page-render path** — its struct and its `SELECT` both omit the `secret` column, so it cannot leak one; `GetHostRecoveryCredential` (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE `recovery_credential_revealed` event via `SaveEvent` and calls NO dispatcher (the `handleRequestLogTail` shape). `api/handler.go handleAdminGetRecoveryCredential` (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133. |
|
||||||
| `host_detail_body` sub-template | hub/internal/web/templates/host_detail_body.html | `{{template "host_detail_body" .}}` | Rendering a host's detail sections on ANY surface | One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with `{{.HostID}}` (the customer page renders N instances). |
|
| `host_detail_body` sub-template | hub/internal/web/templates/host_detail_body.html | `{{template "host_detail_body" .}}` | Rendering a host's detail sections on ANY surface | One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with `{{.HostID}}` (the customer page renders N instances). |
|
||||||
| `(*Store).ListHostsByCustomer` | hub/internal/store/store.go (~L1620) | `(customerID) ([]Host, error)` | A customer's hosts, host_id order | A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer. |
|
| `(*Store).ListHostsByCustomer` | hub/internal/store/store.go (~L1620) | `(customerID) ([]Host, error)` | A customer's hosts, host_id order | A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer. |
|
||||||
|
| `(*Store).HasEverBoundHost` (v0.92.0, R-195) | hub/internal/store/store.go | `(customerID) (bool, error)` | Any verdict that must not fire for a customer with **no machine ever bound** — "was anything ever expected of this customer" | `hosts` row **OR** `host_deletions` tombstone. **NOT a liveness check and never a substitute for one:** a box that was bound and went silent returns `true` and must keep alarming — that is the case any change here breaks first (pinned by `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`). Callers **fail OPEN** on its error: an unreadable binding must never SUPPRESS an alarm. Do **not** re-derive this from report presence — `store.GetCustomers()` (and therefore the staleness checker's `down` state) is a query over `reports`, so a never-reported customer has no state at all, which is exactly how the daily false alarm reached `david`. |
|
||||||
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
|
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
|
||||||
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
|
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
|
||||||
|
| `handleHostEscrowGet` + `escrowSelfServiceRetrieval` | hub/internal/api/handler.go | `GET /hosts/{id}/escrow` | The box-authenticated MIRROR of the escrow PUT — self-scoped by the per-host key. **`escrowSelfServiceRetrieval` is THE single decision point** for whether a box may read its own blob without operator-armed recovery mode (§8.2 vs §8.3, R-199): flip it, do not scatter the condition. **Never merge this with `dr.go`'s `handleReEnroll`/`handleGetRestoreDirective`** — those rotate the host API key and serve the K-escrow and directive too, and keep their recovery-mode gate (pinned by `TestEscrowGet_OperatorDRPathUnchanged`). Every successful retrieval MUST raise `escrow_blob_served` before the bytes leave; that audit row is the mitigation the trade rests on. |
|
||||||
|
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts** — `blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** `SaveHostDRBundle` writes `identity_blob` AFTER `SaveHostEscrow` returns, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
|
||||||
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
|
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
|
||||||
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
|
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
|
||||||
| `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |
|
| `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# STATUS — what works, what's broken, what's next
|
# STATUS — what works, what's broken, what's next
|
||||||
|
|
||||||
**Updated 2026-08-03.**
|
**Updated 2026-08-05.**
|
||||||
|
|
||||||
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
||||||
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
||||||
@@ -13,71 +13,107 @@
|
|||||||
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer,
|
A blank machine boots the Felhom disc, installs itself unattended, and is claimed by the customer,
|
||||||
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
who sets their own password. They install apps from a catalogue of fifty-three, share files over the
|
||||||
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
home network, and open apps from a launcher or a shared link. Backups run on their own to three
|
||||||
places — the machine's drive, a second drive, and an encrypted off-site copy — and a customer can
|
places — the machine's drive, a second drive, and an encrypted off-site copy.
|
||||||
restore files and app data from the drive alone. Apps come back after a power cut: hard-reset the demo
|
|
||||||
box six times, everything returned every time, and an app switched off deliberately stayed off.
|
**And the whole backup promise is now proved.** On 4 August we destroyed a machine on purpose and
|
||||||
Proven end to end on real hardware.
|
deleted a marked file from its disk. Using the recovery code you saved: the backup key came back
|
||||||
|
**identical, character for character**; the existing off-site store **opened** rather than starting
|
||||||
|
over; and the file was restored **byte for byte identical**. *(R-201)*
|
||||||
|
|
||||||
## What's broken
|
## What's broken
|
||||||
|
|
||||||
|
- **A customer still cannot do that recovery alone — but only one step is left.** Getting from "the
|
||||||
|
key is recoverable" to "the file is back" took four steps that appeared in no instructions. **Three
|
||||||
|
are fixed today** *(R-204)*: the local reset-code tool works on the first try instead of needing the
|
||||||
|
controller restarted; re-issuing the storage credential no longer falsely marks the recovery key
|
||||||
|
"stale" (which used to stop every off-site backup and invite the one act that would have destroyed
|
||||||
|
the recovered key); and the everyday restore now **says in plain Hungarian that it returned the
|
||||||
|
app's settings and database and not your documents**, and names the button that does. **The step
|
||||||
|
that remains is the first one:** a rebuilt machine cannot get a storage credential by itself,
|
||||||
|
because the one-time password was used up by its predecessor — so you still have to press
|
||||||
|
Re-issue. That is a design decision waiting on you, below. *(R-193)*
|
||||||
|
- **Rebuilding a machine still throws away its off-site backup HISTORY.** The machine invents the key
|
||||||
|
that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one. Both demo
|
||||||
|
machines did this on 3–4 August — 51 backups (~1.2 GB) between them. The old key now survives the
|
||||||
|
recovery ceremony, and a changed key now raises an alarm the same day, but **the rebuild itself
|
||||||
|
still starts a fresh history**. *(R-193)*
|
||||||
|
- **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
|
||||||
|
says the old backups may later be restorable with the matching recovery code. That is true for
|
||||||
|
machines that re-seal from now on and **false for anything already orphaned** — and the machine
|
||||||
|
cannot tell which case it is in. We deliberately did **not** patch the sentence: a conditional
|
||||||
|
promise that can still be wrong is worse there than a vague one. *(R-202)*
|
||||||
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
||||||
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
|
also delete it. A daily snapshot is armed as a stopgap. *(R-95, R-87)*
|
||||||
*(R-95, R-87)*
|
|
||||||
- **Nothing else open that affects a customer.** The three faults that were on this list yesterday —
|
|
||||||
the reserve watching the wrong step, the last app whose data was never saved, and the alert that
|
|
||||||
told you about one app and swallowed the rest — are all fixed and proven on a real machine.
|
|
||||||
|
|
||||||
## What shipped recently
|
## What shipped recently
|
||||||
|
|
||||||
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
- **2026-08-05** — **The DooPlex server's disk is out of danger: 86% full → 54%, and the storage layer
|
||||||
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
is unstuck.** The cause was leftover working data from building our own software — 157 GB of it,
|
||||||
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
growing about 5 GB a day, which nothing was allowed to delete. **148 GB came back in 86 seconds.**
|
||||||
day, because the machine correctly refuses to re-prove an archive it has already proven. It is now
|
The storage layer had already stopped accepting new copies of any volume onto that disk; that is
|
||||||
written to disk with the result and survives. This was caught happening, not predicted: a real
|
fixed the same day. A **30 GB ceiling** is now in place and was **proved to work by deliberately
|
||||||
14.5 GB off-site restore passed and left no record at all. **(2)** Every release had about a
|
overfilling it and watching it evict** — not by assuming the setting took. Two things that failed
|
||||||
fifty-fifty chance of emailing you a failure for a release that worked; the version tag is now
|
quietly around it: the warning meant to catch exactly this **could never fire** (fixed and proved),
|
||||||
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
|
and the weekly cleanup is still forbidden from touching the thing that grows (next session).
|
||||||
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
|
*(R-205 … R-211)*
|
||||||
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
|
- **2026-08-05** — **The build data now lives on the second SSD, moved with nothing lost.** All 345
|
||||||
- **Each backup is now proved, instead of the clock being obeyed.** The machine used to re-test a
|
images, every saved volume and both development databases came through identical — checked before
|
||||||
backup every twenty-four hours on a timer that restarted whenever the software was updated, so the
|
the original was touched and again afterwards, and confirmed by running a real build on the moved
|
||||||
test happened at an arbitrary time and a fresh backup could sit unproven while an old one was
|
copy. **The server's own services never went down:** Gitea, the registry, the hub and the backup
|
||||||
re-checked. Now a copy is tested once, about a day after it is made, and not again until there is a
|
system run on a separate system and stayed up throughout. The second SSD now also **reserves 80 GB**
|
||||||
newer one — so the daily copy is proved daily and the weekly off-site copy weekly, each on its own.
|
for this, so the storage layer can no longer quietly claim the space and repeat what happened to the
|
||||||
The alert that says "this copy has not been proved lately" learned each copy's own rhythm in the
|
first disk. The old copy is kept as the way back until the machine next restarts. *(R-209)*
|
||||||
same change; without that it would have started complaining every night about a system that is
|
- **2026-08-05** — **Three of the four recovery crutches removed.** The reset code works first time;
|
||||||
working. *(R-86)*
|
a credential re-issue no longer blocks off-site backups on a healthy machine; and the default
|
||||||
- **A failed backup now tells you about every app, once**, with every failure written down whether or
|
restore no longer quietly returns the wrong thing. *(R-204 items 1–3, R-196)*
|
||||||
not it is emailed, and a message the machine decides not to send now leaves a record saying so.
|
- **2026-08-04 (night)** — **The drill PASSED**, end to end, on real hardware. *(R-201)*
|
||||||
*(R-182)*
|
- **2026-08-04** — The folder-left-out-of-the-backup problem fixed both halves: the app and its backup
|
||||||
|
look in the same directory, and a backup that misses a folder marked essential reports *incomplete*
|
||||||
|
instead of success. *(R-203)*
|
||||||
|
- **2026-08-04** — The hub now keeps the off-site backup key when a machine re-seals, instead of only
|
||||||
|
the whole-machine one, and a machine can fetch its own sealed package back. *(R-198, R-199)*
|
||||||
|
- **2026-08-04** — The daily false alarm about David is gone; a vanished permission now repairs itself
|
||||||
|
**and says it had to**; the weekly off-site backup stopped reporting failure after a successful
|
||||||
|
upload. *(R-195, R-190, R-191)*
|
||||||
|
|
||||||
## What we're working on
|
## What we're working on
|
||||||
|
|
||||||
- **Now:** nothing outstanding.
|
- **Next: the retention proof.** The hub keeping the old sealed key when a machine re-seals is the one
|
||||||
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
remaining link that has **never run outside a test**. Proving it needs a second deliberate wipe on
|
||||||
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
the spare demo machine, and it is its own procedure. *(R-198)*
|
||||||
*(R-87)*
|
- **Then:** the orphaned-backup deletion you asked for (below), and the off-site copy the machine can
|
||||||
- **After:** the off-site copy that the machine making it can still erase. *(R-95)*
|
still erase. *(R-193, R-95)*
|
||||||
|
|
||||||
## Waiting on you
|
## Waiting on you
|
||||||
|
|
||||||
|
- **One thing to read after the machine next restarts — and nothing to do until then.** You told me
|
||||||
|
not to restart DooPlex, so I did not, and the move to the second SSD has therefore never been
|
||||||
|
through a restart. It works right now and nothing was lost, but a restart is the one test that
|
||||||
|
matters for this kind of change, and it has not happened. **I made it check itself:** whenever the
|
||||||
|
machine next starts, for any reason, it writes a plain PASS or FAIL line to
|
||||||
|
`/var/log/felhom-store-postboot-check.log`. **If it says PASS, the old copy can be deleted and 34 GB
|
||||||
|
comes back.** Until then I have deliberately kept that old copy, which is the only quick way back —
|
||||||
|
it is why the disk sits at 54% rather than lower. *(R-209a)*
|
||||||
|
- **One list to rule on: 193 old images that exist only on this machine.** 131 controller versions and
|
||||||
|
62 hub versions are not in the registry, so they cannot be re-downloaded — all of them old
|
||||||
|
(controller up to 0.135.0, hub up to 0.57.0; everything newer is safely in the registry). **Nothing
|
||||||
|
was deleted.** Worth knowing before you spend time on it: they only account for about 27 GB against
|
||||||
|
199 GB now free, so this is about clutter, not space. *(R-210)*
|
||||||
|
- **The one-shot credential decision — this is now the last thing between a customer and an unaided
|
||||||
|
recovery.** A rebuilt machine has no storage credential of its own, so an operator must press
|
||||||
|
Re-issue. Everything after that point is now self-service. Deciding how a rebuilt machine should get
|
||||||
|
a credential is the remaining design question. *(R-193, R-204 item 4)*
|
||||||
|
- **The recovery screen you described has been priced, and it can be built.** A freshly installed
|
||||||
|
machine that finds a sealed package waiting should say so, offer a box for the recovery code, and
|
||||||
|
show what would come back before doing anything. One thing to weigh, deliberately not decided: that
|
||||||
|
screen is reachable by anyone with the household's dashboard password, and the preview reveals
|
||||||
|
backup dates and app names. *(R-193)*
|
||||||
|
- **The orphaned backups on the storage box — you said delete, and it is still owed.** About 1.2 GB
|
||||||
|
across the two demo machines, in set-aside stores nobody can open and nothing prunes. It wants its
|
||||||
|
own session rather than riding along with other work. *(R-193)*
|
||||||
|
- *(decided 4 Aug)* You chose **not** to keep a copy of the backup key on the Proxmox host, which
|
||||||
|
makes the customer's own recovery code the only route back from a rebuild. *(R-193)*
|
||||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||||
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
- **One small question, not urgent.** The automatic version check cannot see which version you have
|
||||||
machines to install, only which ones exist. Closing that needs either a password given to the build
|
told machines to install, only which ones exist. *(R-184)*
|
||||||
server or a check inside the hub itself. *(R-184)*
|
|
||||||
- **Nothing else.** The question about whether the off-site endpoint counts as protected is
|
|
||||||
**settled — it does**, and the machine list now says so instead of asking.
|
|
||||||
|
|
||||||
## Changed since last update
|
|
||||||
|
|
||||||
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
|
|
||||||
that could vanish on a restart (seen happening), a release that emailed a failure for a release
|
|
||||||
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
|
|
||||||
- **2026-08-03** — Backups are now proved one at a time, each about a day after it is made, instead of
|
|
||||||
on a timer; the "not proved lately" alert learned each copy's own rhythm so it does not complain
|
|
||||||
about a healthy weekly copy. You settled that the off-site endpoint is protected, and the machine
|
|
||||||
list records it. One thing found while testing: on the small demo machine the agent cannot see its
|
|
||||||
own local backups at all — a permission that was never granted — so that copy has never been
|
|
||||||
test-restored there. Written down, not yet fixed. *(R-86, R-185)*
|
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@@ -378,6 +378,60 @@ nicety.
|
|||||||
**loses Felhom's backup guarantees** — no PBS verify / monitoring on storage we can't reach. An
|
**loses Felhom's backup guarantees** — no PBS verify / monitoring on storage we can't reach. An
|
||||||
honest signup-time tradeoff, not a hidden one.
|
honest signup-time tradeoff, not a hidden one.
|
||||||
|
|
||||||
|
#### Custody across generations — what the hub retains when a blob is superseded (hub v0.60.0; **contract corrected hub v0.93.0, R-198**)
|
||||||
|
|
||||||
|
A ceremony that seals a *different* passphrase does not destroy the blob it replaces: the hub copies
|
||||||
|
the current `host_escrow` row into `host_escrow_superseded` **before** overwriting it, so a previous
|
||||||
|
generation stays openable with the recovery code that sealed it. The same routine
|
||||||
|
(`demoteCurrentEscrowTx`) is used when a host is deleted, which **demotes** custody rather than
|
||||||
|
destroying it. The hub never decrypts either table.
|
||||||
|
|
||||||
|
**The contract as originally shipped was incomplete, and the gap is worth recording because it was
|
||||||
|
invisible for two months.** An escrow generation is **two** sealed artifacts — the K-escrow `blob`
|
||||||
|
(the PBS datastore key `K`) and `identity_blob` (the age-wrapped `IdentityBundle`, which carries the
|
||||||
|
offsite **restic repository password** as well as the tunnel/PBS tokens and the WG key). Retention
|
||||||
|
covered only the first. So a supersession preserved the whole-guest key and destroyed the off-site
|
||||||
|
data key — and because the copy happens as the new blob lands, **the destroying act was the ceremony
|
||||||
|
itself**, i.e. the step a rebuilt box asks its customer to perform. Corrected in hub v0.93.0: the
|
||||||
|
retained row carries both.
|
||||||
|
|
||||||
|
**Two consequences that belong in this doc rather than a changelog:**
|
||||||
|
|
||||||
|
- **Generational custody is now the recovery precondition for the offsite tier.** The operator ruled
|
||||||
|
on 2026-08-04 that no repository password is retained on the Proxmox host (R-193 candidate (c),
|
||||||
|
refused). With no host-side copy, a rebuilt box's only route back to its previous off-site history
|
||||||
|
is *the customer's older recovery code opening the retained identity blob* — which exists only if
|
||||||
|
the supersession that replaced it happened on hub v0.93.0 or later.
|
||||||
|
- **Retention is custody, not recovery, and this doc must not be read as claiming otherwise.** The
|
||||||
|
hub's restore-mode serving of these blobs (§ slice 10D) has **no client** in the agent, the
|
||||||
|
controller's recovered-password injection seam has **no user interface**, and no part of the chain
|
||||||
|
has ever been exercised outside a unit test — the one live identity round-trip (2026-06-10)
|
||||||
|
predates the `ResticRepoPassword` field. Tracked as R-199/R-200/R-201; evidence
|
||||||
|
`documentation/audits/RECON-offsite-dr-chain-2026-08-04.md`.
|
||||||
|
|
||||||
|
#### Serving a blob back to its own box (hub v0.94.0 + agent v0.125.0, R-199)
|
||||||
|
|
||||||
|
Retention without retrieval is custody, not recovery. Since 2026-08-04 a host reads its own sealed
|
||||||
|
identity blob from `GET /api/v1/hosts/{host_id}/escrow` — self-scoped by the per-host key, the mirror
|
||||||
|
of the PUT that stored it — and opens it locally with the customer's recovery code via the agent's
|
||||||
|
`POST /escrow/recover-offsite-password`, which returns **only** the offsite repository password.
|
||||||
|
|
||||||
|
**This is a deliberate widening of who can obtain the ciphertext**, recorded in `CONTEXT.md` S-29 and
|
||||||
|
in the handler itself: it previously took an operator arming recovery mode, and now takes only that
|
||||||
|
box's own credential. The blob remains unopenable without R, and every retrieval raises an
|
||||||
|
operator-visible `escrow_blob_served` event. The operator-driven re-enroll/restore-directive path is
|
||||||
|
unchanged and keeps its recovery-mode gate.
|
||||||
|
|
||||||
|
**Proven end to end on demo-felhom, 2026-08-04:** the recovered repository password's sha256 was
|
||||||
|
byte-identical to the one on the box, and to the hash the hub had independently recorded. **The chain
|
||||||
|
stops there** — nothing installs a recovered password, reopens a repository with it, or restores a
|
||||||
|
file.
|
||||||
|
|
||||||
|
**The honesty property above is unchanged and now has a second edge:** losing `R` and the box makes
|
||||||
|
the offsite backups unrecoverable by anyone — *and* an older generation's data is recoverable only
|
||||||
|
with **that generation's** recovery code, which the customer must still hold. A current recovery code
|
||||||
|
does not open a previous generation's blob.
|
||||||
|
|
||||||
## 9. Provisioning & DR flows
|
## 9. Provisioning & DR flows
|
||||||
|
|
||||||
**Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs
|
**Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs
|
||||||
|
|||||||
@@ -278,7 +278,10 @@ The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what
|
|||||||
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
|
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
|
||||||
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
|
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
|
||||||
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** |
|
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** |
|
||||||
| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` | **yes** (per-customer key) |
|
| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` — **and the box asks for none** (R-191) | **yes** (per-customer key) |
|
||||||
|
|
||||||
|
> **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does.
|
||||||
|
|
||||||
|
|
||||||
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
|
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
|
||||||
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
|
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
|
||||||
@@ -348,6 +351,26 @@ about whether the bytes can be turned back into a working system, and a system c
|
|||||||
completely while having no executable recovery route for a given failure. That is not a hypothetical
|
completely while having no executable recovery route for a given failure. That is not a hypothetical
|
||||||
here — §8 has rows where it is the actual state.
|
here — §8 has rows where it is the actual state.
|
||||||
|
|
||||||
|
### 7.0 What a customer can and cannot do ALONE — the four steps the drill found
|
||||||
|
|
||||||
|
> **[FACT] Added 2026-08-05.** The 2026-08-04 R-201 drill (`audits/DRILL-r201-night-run-2026-08-04.md`)
|
||||||
|
> is the first end-to-end proof that a customer's file survives a machine rebuild and comes back
|
||||||
|
> byte-identical. **It passed with an operator present, and four manual interventions stood between
|
||||||
|
> "the key is recoverable" and "the file is back" — none of which was in any design document.** They
|
||||||
|
> are recorded here because this is the section a future reader will use to answer *"can the customer
|
||||||
|
> do this alone?"*, and until 2026-08-05 the answer was no for reasons nothing wrote down.
|
||||||
|
|
||||||
|
| # | The step | Why it stopped a customer | Status |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | **The reset code was refused on the first try** | `--print-reset-code` runs as a SEPARATE process (`docker exec`); it persists a new code while the running controller keeps the old one in its cache, so the code the customer is told to type never matches. **The controller had to be restarted in between**, which nothing said. Two attempts failed during the drill before that was worked out. | **CLOSED — controller v0.198.0** (R-204 item 1). `effectiveClaimCode` reads through to the persisted state. Read-through, not a TTL: a TTL would leave a window in which a superseded code still works. Fails closed on an unreadable state. Proven live on demo-felhom 9201 with nothing restarted. |
|
||||||
|
| 2 | **Re-issuing the off-site credential marked the recovery escrow "stale"** | A stale escrow withholds `restic_pw_sha256` from the report ACK → the controller's auto-confirm cannot flip `pending→escrowed` → `OffboxRunnable()` is false → **every off-site backup refused**. The customer is then told to re-run the recovery ceremony, **which is the one act that would have destroyed the key just recovered.** The repository password had not changed at all. | **CLOSED — hub v0.95.0** (R-204 item 2 / R-196). The precautionary mark is gone. The real case is measured by the controller's Scenario-F hash re-check each ACK — which the mark was BLINDING by emptying that very hash — and by R-197's `offsite_repo_key_changed` at a supersession. |
|
||||||
|
| 3 | **The restore's default returned the wrong thing, silently** | `mode=unit` restores the recovery unit — the app's definition, configuration and database dumps — and **not the customer's files**; the userdata in the same snapshot is excluded by `--include`. The outcome message was one sentence for both modes and named neither scope. On the last step of a disaster recovery, the default quietly did not do what the person asked. | **CLOSED — controller v0.198.0** (R-204 item 3). The unit outcome names what came back, what did not, and the step that gets it; the wizard's intent card states its scope before the choice. **The size gate on `mode=full` is untouched**, and the default stays `unit`. |
|
||||||
|
| 4 | **A rebuilt box cannot obtain an off-site credential unaided** | The one-time provider password was spent by its predecessor, so the rebuilt guest has nothing to authenticate with and an **operator Re-issue** is required. | **OPEN → R-193.** This is a design decision (the one-shot credential model), not a defect, and it needs an operator ruling. **It is the reason the capability-map row keeps its "with a person present" qualifier.** |
|
||||||
|
|
||||||
|
> **[FACT] The honest current answer:** three of the four are gone. **A customer alone still cannot
|
||||||
|
> complete a rebuild recovery**, because step 4 requires the operator. Everything after step 4 —
|
||||||
|
> claim, restore, verify — is now self-service.
|
||||||
|
|
||||||
### 7.1 The dependency graph
|
### 7.1 The dependency graph
|
||||||
|
|
||||||
> **[FACT] SUPERSEDED IN PART BY D5 — SHIPPED 2026-07-30 (controller v0.188.0).** Leg 1 below
|
> **[FACT] SUPERSEDED IN PART BY D5 — SHIPPED 2026-07-30 (controller v0.188.0).** Leg 1 below
|
||||||
|
|||||||
@@ -0,0 +1,265 @@
|
|||||||
|
# DRILL — R-201, the night run: **PASS. The customer's file came back, byte-identical.**
|
||||||
|
|
||||||
|
**Date:** 2026-08-04, 21:30–23:15 · **Box:** `demo-hp` (Tier 0) · unattended, then **finished with the
|
||||||
|
operator present**. The box is up, healthy, and fully re-armed.
|
||||||
|
|
||||||
|
> **THE VERDICT: PASS.**
|
||||||
|
>
|
||||||
|
> ```
|
||||||
|
> pre-wipe sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||||
|
> restored sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> A machine was destroyed on purpose — controller data volume gone, the customer's file deleted from
|
||||||
|
> disk — and **the file came back byte-identical**, through the customer's own recovery code and the
|
||||||
|
> customer's own restore flow. **This is the first time the Felhom backup story has been proved end to
|
||||||
|
> end.**
|
||||||
|
>
|
||||||
|
> The key recovered byte-identical (`8a9e33aa4da6…`), the existing repository **opened rather than
|
||||||
|
> being re-initialised** (`repo_state: null`, 3 snapshots, 42 026 B — the pre-wipe size exactly), and
|
||||||
|
> the pre-wipe snapshot `e6132ae5` was still there with the file inside it.
|
||||||
|
>
|
||||||
|
> **It took four undocumented steps to get there**, each measured, and they are the other half of the
|
||||||
|
> result — see §3. A customer hitting this alone today would not get their data back.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. The verdict — **PASS**
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| sentinel sha256, pre-wipe (step 3) | `643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` |
|
||||||
|
| sentinel sha256, restored (step 10) | **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`** |
|
||||||
|
| **match** | **YES — byte-identical** |
|
||||||
|
| snapshot count at step 9 | **3** — the pre-wipe count. **NOT 1.** No fresh history was started |
|
||||||
|
| repo size at step 9 | **42 026 B** — the pre-wipe size exactly |
|
||||||
|
| `repo_state` | **`null`** — the repository OPENED. Not orphaned |
|
||||||
|
|
||||||
|
The restore ran through the customer flow (`POST /backup/offbox/restore`, `mode=full`, two-step
|
||||||
|
confirm), out of snapshot `e6132ae5` — the one taken **before** the wipe.
|
||||||
|
|
||||||
|
**A detail worth keeping:** the first restore attempt used the default `mode=unit`, which recovers the
|
||||||
|
recovery unit only and **not** the userdata leg. `mode=full` is what returns the customer's files. A
|
||||||
|
customer told to "restore from off-site" would, on the default, get their app definition back and not
|
||||||
|
their documents — and nothing in that outcome says so.
|
||||||
|
|
||||||
|
## 2. What was proven, in order, on hardware
|
||||||
|
|
||||||
|
### Step 6 — the wipe
|
||||||
|
|
||||||
|
The controller's data volume was destroyed and the bootstrap re-ran. Fresh data directory, everything
|
||||||
|
stamped `2026-08-04 20:00:2x`:
|
||||||
|
|
||||||
|
```
|
||||||
|
encryption.key 32 B 2026-08-04 20:00:21 ← brand new: every pre-wipe app secret is undecryptable
|
||||||
|
claimed = None ← the fresh-install signal
|
||||||
|
offbox = null ← no off-site target
|
||||||
|
```
|
||||||
|
|
||||||
|
**Faithful to the incident, and deliberately so.** The 2026-08-03 rebuild that R-193 is filed against
|
||||||
|
was **not** a guest reprovision — the journal shows guest 9201 running continuously through that
|
||||||
|
window, with no `pct destroy`, no `pct restore` and no `--selftest=provision`. What changed was the
|
||||||
|
controller and its data volume. Reproducing *that* is what the drill needs; a guest destroy plus an
|
||||||
|
unrehearsed provisioning chain, improvised unattended, is what §8.10 exists to prevent.
|
||||||
|
|
||||||
|
### Step 7 — the assertion that keeps recovery possible: **PASSED**
|
||||||
|
|
||||||
|
```
|
||||||
|
host_escrow (demo-hp-bb76ea), AFTER the wipe:
|
||||||
|
identity_blob = 572 bytes ← unchanged
|
||||||
|
restic_pw_sha256 = 8a9e33aa4da6… ← unchanged
|
||||||
|
updated_at = 2026-08-04 11:11:37 ← unchanged; nothing re-escrowed
|
||||||
|
stale_at = NULL
|
||||||
|
```
|
||||||
|
|
||||||
|
**No ceremony was run and nothing re-escrowed itself.** The sealed key survived the rebuild untouched.
|
||||||
|
|
||||||
|
### Step 8 — **THE KEY CAME BACK**
|
||||||
|
|
||||||
|
```
|
||||||
|
=== offsite key recovery check (R-200) — compares, never installs ===
|
||||||
|
recovered sha256: 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a
|
||||||
|
[FAIL] there is no repository password on this box to compare against
|
||||||
|
(the recovery itself SUCCEEDED — this box simply has no local key. That is the
|
||||||
|
rebuilt-box shape, where the next step is to INSTALL rather than compare.)
|
||||||
|
```
|
||||||
|
|
||||||
|
Exit 1 is **correct and designed**: there was nothing to compare against, because the wipe removed it.
|
||||||
|
The recovered hash matches the pre-wipe on-disk key and the hub's own record — three independent
|
||||||
|
sources agreeing, one of them recovered through the full chain (hub → agent → unseal → extract) on a
|
||||||
|
box that had just lost everything it knew.
|
||||||
|
|
||||||
|
### Step 9a — installed cleanly
|
||||||
|
|
||||||
|
```
|
||||||
|
=== offsite key recovery INSTALL (R-200) ===
|
||||||
|
on-disk sha256: (none — this box has no repository password)
|
||||||
|
recovered sha256: 8a9e33aa4da6…
|
||||||
|
[INSTALLED] the recovered repository password is in place and reads back identical.
|
||||||
|
```
|
||||||
|
|
||||||
|
The "installed" branch — the rebuilt-box shape v0.196.0 was written for — took its first real run.
|
||||||
|
|
||||||
|
### Step 9b — the apply kept it
|
||||||
|
|
||||||
|
After the target was reconfigured, the on-disk key was still `8a9e33aa4da6…`. `WriteOffboxSecrets`
|
||||||
|
found the file present and kept it, exactly as documented.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. The wall — three blockers, each measured
|
||||||
|
|
||||||
|
### (a) R-193, reconfirmed live: a rebuilt controller cannot configure its off-site tier
|
||||||
|
|
||||||
|
```
|
||||||
|
[INFO] [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.197.0)
|
||||||
|
[WARN] [offsite-apply] reconcile: offsite-apply: consume one-time password:
|
||||||
|
no unconsumed offsite password (already consumed or none provisioned)
|
||||||
|
(retries on next config refresh/restart)
|
||||||
|
```
|
||||||
|
|
||||||
|
The ledger, measured: `demo-hp` one-time secret created `07:11:51`, consumed `07:12:06` — **by the
|
||||||
|
previous controller**. The rebuilt one has nothing to consume and retries forever. Exactly the state
|
||||||
|
demo-hp sat in for 25 hours on 2026-08-03.
|
||||||
|
|
||||||
|
**Remedy:** an operator Re-issue. Performed here through the designed endpoint
|
||||||
|
(`POST /configs/demo-hp/offsite-reissue`, HTTP 303), after which the target configured normally.
|
||||||
|
|
||||||
|
### (b) R-196, measured live — and it lands squarely on the recovery path
|
||||||
|
|
||||||
|
```
|
||||||
|
host_escrow (demo-hp-bb76ea), after the Re-issue:
|
||||||
|
stale_at = 2026-08-04 20:15:49 ← set
|
||||||
|
restic_pw_sha256 = 8a9e33aa4da6… ← UNCHANGED. The key did not move.
|
||||||
|
```
|
||||||
|
|
||||||
|
**The escrow was marked stale while it perfectly covers the box's current key** — the recovered one.
|
||||||
|
That is R-196's false staleness, and here it is not a cosmetic lie: a stale escrow makes the hub
|
||||||
|
**withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending →
|
||||||
|
escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses to run any off-site backup**.
|
||||||
|
|
||||||
|
> **The remedy for (a) disables the recovery it was needed for.** The only documented way to clear a
|
||||||
|
> stale escrow is a fresh ceremony — which supersedes the identity blob, i.e. **destroys the very key
|
||||||
|
> being recovered**. Under hub v0.93.0 the old blob is now *retained*, but nothing can serve a
|
||||||
|
> superseded blob back (R-199's inventory: the endpoint serves the CURRENT row only).
|
||||||
|
|
||||||
|
**Superseded rows still number 2** — no ceremony was run tonight. The key is intact.
|
||||||
|
|
||||||
|
### (c) The claim gate — undocumented as a recovery step
|
||||||
|
|
||||||
|
```
|
||||||
|
[DEBUG] [web] claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)
|
||||||
|
```
|
||||||
|
|
||||||
|
A rebuilt box is **unclaimed** (`claimed = None`, fresh `settings.json`), and the claim gate correctly
|
||||||
|
intercepts every non-claim route. **So no controller endpoint can be driven at all** — not the manual
|
||||||
|
escrow confirm, not the backup trigger — until the customer re-claims the box. Both POSTs returned 302
|
||||||
|
to the claim page and neither reached its handler; `last_run` stayed null, which is why the earlier
|
||||||
|
reading of "the run was refused" needed this second look to be accurate.
|
||||||
|
|
||||||
|
This is correct behaviour and it is **not wrong** — but it is a step in the customer's recovery journey
|
||||||
|
that appears in no design document, and it comes *before* anything else can happen.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. How it was finished, and the fourth undocumented step
|
||||||
|
|
||||||
|
The unattended run stopped at the wall in §3 rather than improvising further at 22:35 (§8.10). With the
|
||||||
|
operator present the remaining path was walked, and it took **four steps a customer would have to
|
||||||
|
discover**:
|
||||||
|
|
||||||
|
1. **Re-issue** the offsite credential (hub, operator-only) — otherwise the box can never configure its
|
||||||
|
tier. §3(a).
|
||||||
|
2. **Re-claim the box.** The local escape hatch (`--print-reset-code`) mints a code — **and does not
|
||||||
|
work on its own**: it writes the new hash to `settings.json` while the RUNNING controller keeps its
|
||||||
|
old copy in memory, so `effectiveClaimCode()` never sees it and every attempt returns
|
||||||
|
*"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming.** Two
|
||||||
|
claim attempts failed on this before it was diagnosed; the generation counter reached 6.
|
||||||
|
3. **Confirm the escrow** manually (`/backup/offbox/confirm-escrow`) → `escrowed`, so `OffboxRunnable`
|
||||||
|
permits a run. **Not a ceremony** — the identity blob was never superseded (still 2 retained rows).
|
||||||
|
4. **Restore with `mode=full`.** The default `mode=unit` returns the recovery unit and **not** the
|
||||||
|
customer's files.
|
||||||
|
|
||||||
|
**None of these four is in any design document, and the drill needed all four.**
|
||||||
|
|
||||||
|
## 5. §5's five conditions, as recorded before the wipe
|
||||||
|
|
||||||
|
| # | Condition | Evidence |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | §4 confirmed, sentinel **listed by name** | snapshot `e6132ae5` (19:36:26) — `-rw-r--r-- 1000 1000 181 … /DRILL-SENTINEL.txt` |
|
||||||
|
| 2 | deliberate rollback archive, **verified** | `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst`, 1 606 765 083 B; **full zstd stream read OK** (4 867 573 760 B uncompressed); the sentinel confirmed **inside** it |
|
||||||
|
| 3 | §3's option | see §6 |
|
||||||
|
| 4 | demo-felhom untouched | `health=ok`, `escrow_state=escrowed`, not touched at any point |
|
||||||
|
| 5 | free space | nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 % |
|
||||||
|
|
||||||
|
**A precondition had drifted and was repaired before the wipe, not worked around.** The staged
|
||||||
|
snapshot no longer contained the sentinel: the afternoon's `[WARN] mandatory data path missing`
|
||||||
|
experiment produced a *later* same-day calibre-web snapshot, and restic's `forget --keep-daily 7
|
||||||
|
--group-by host,tags` had pruned the good one in its favour. The fixture on disk was correct, so one
|
||||||
|
backup re-established it and it was re-verified by listing. **Lesson worth carrying: a good snapshot is
|
||||||
|
not durable against a later bad run on the same day.**
|
||||||
|
|
||||||
|
## 6. §3 — the recovery code
|
||||||
|
|
||||||
|
**Option B as already in place, with a strict improvement: no new copy was created, so nothing needed
|
||||||
|
shredding.**
|
||||||
|
|
||||||
|
The operator placed `R_DEMO-HP` in their own `~/.config/credentials` on DooPlex (mode `0600`) two
|
||||||
|
sessions ago, deliberately, for this purpose. This session read it from there and **piped it to stdin**
|
||||||
|
for each of the two invocations that needed it. It was never an argument, never exported, never written
|
||||||
|
to a second file, and never logged.
|
||||||
|
|
||||||
|
**Nothing was shredded, and that is the point:** the operator's own permanent store is theirs, not a
|
||||||
|
drill artefact, and destroying it would have destroyed their record. Because no additional copy was
|
||||||
|
made, there is nothing left behind to prove gone — a stronger position than option B's
|
||||||
|
create-then-shred.
|
||||||
|
|
||||||
|
**Verified afterwards** with the planted-copy positive control, exactly as on 2026-08-04 afternoon —
|
||||||
|
see §8.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. The state the box is left in — healthy and re-armed
|
||||||
|
|
||||||
|
```
|
||||||
|
controller felhom-controller:0.197.0, healthy, CLAIMED
|
||||||
|
apps privatebin opengist calibre-web filebrowser cloudflared traefik — all serving
|
||||||
|
escrow_state escrowed (no ceremony was run; identity_blob untouched)
|
||||||
|
repo key 8a9e33aa4da6… THE RECOVERED KEY — and the repository opens with it
|
||||||
|
off-site 3 snapshots, 42 026 B, repo_state null; all three apps re-toggled for off-site
|
||||||
|
sentinel restored to its live location, sha256 643166269103a25c… — verified
|
||||||
|
```
|
||||||
|
|
||||||
|
The verification copy under `backups/offsite-restore/calibre-web/` is **left in place as evidence**.
|
||||||
|
|
||||||
|
## 8. R persisted nowhere — searched, with a positive control
|
||||||
|
|
||||||
|
Swept the agent journal, the controller container log, and `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`,
|
||||||
|
`/root` on demo-hp for the recovery code: **0 hits**, and **0** leftover `felhom-idesc-*` unseal
|
||||||
|
staging directories. A planted copy was found by the same sweep (**1**) and not found after shredding
|
||||||
|
(**0**), so the instrument is shown sensitive rather than assumed to be.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Teardown — three layers
|
||||||
|
|
||||||
|
| layer | state |
|
||||||
|
|---|---|
|
||||||
|
| the guest | **Nothing torn down.** The wipe is the evidence; the apps are serving; the recovered key is in place. The scratch band is empty — no restore-test guest was created tonight. |
|
||||||
|
| the host | `vzdump` snapshot LVs removed cleanly by the archive job (`snap_vm-9201-disk-0/1_vzdump` both released). One new archive, 1.6 GB, on `nvme-1tb` (883 GB free). Nothing deleted. |
|
||||||
|
| the hub | **No new customer records.** `demo-hp` is the same customer row throughout — the rebuild was a controller-data wipe, not a re-enrolment, so nothing accumulated. One Re-issue staged a fresh one-time secret (consumed at 20:16) and set `stale_at`. The two superseded escrow rows are **unchanged** — no ceremony ran. |
|
||||||
|
|
||||||
|
**Nothing was deleted on the storage endpoint**, including the ~1.2 GB of previously-orphaned
|
||||||
|
ciphertext. Ruled, still owed, and deliberately not ridden along with a drill.
|
||||||
|
|
||||||
|
## 10. Part 2 — not run
|
||||||
|
|
||||||
|
Its gate is *"the drill PASSED; its evidence is written down; and there is time."* The first two are
|
||||||
|
now met. **The third was not** — it was 23:15 after a full destructive cycle, and Part 2 is a second
|
||||||
|
wipe. It is deliberately left for a session that starts fresh, with drill 1's result already recorded
|
||||||
|
so a second wipe cannot overwrite it.
|
||||||
|
|
||||||
|
**R-198's retention therefore remains unit-proven only.** Nothing has yet superseded a key in
|
||||||
|
production, so the retention's live behaviour is still unobserved — and it is now the last unproven
|
||||||
|
link in the chain this drill exercised. The check is cheap: wipe again, do NOT recover, let a ceremony
|
||||||
|
seal a different password, then read the superseded row's `identity_blob`.
|
||||||
@@ -0,0 +1,311 @@
|
|||||||
|
# DRILL RECORD — R-201, the wipe-and-recover proof: **PREPARED, HALTED BEFORE THE WIPE**
|
||||||
|
|
||||||
|
**Date:** 2026-08-04 · **Box:** `demo-hp` (Tier 0, the designated drill host) · **Nothing was wiped.**
|
||||||
|
**Outcome:** the drill did not reach its verdict. It was halted at step 4 by a defect that makes the
|
||||||
|
verdict unobtainable — and that defect is worth more than the drill.
|
||||||
|
|
||||||
|
> **THE HEADLINE.** A customer-declared **mandatory** data directory was **silently absent from the
|
||||||
|
> off-site snapshot**, while the backup reported `ok` with three snapshots. The only trace is one
|
||||||
|
> `[WARN]` line inside the controller container. The hub, the card and the counters all say the backup
|
||||||
|
> succeeded. → **R-203**
|
||||||
|
>
|
||||||
|
> **Nothing irreversible happened.** The wipe never ran. `demo-hp` is left in a BETTER state than it
|
||||||
|
> started: it now has a working off-site repository (it had an orphaned one), three snapshots, and a
|
||||||
|
> sentinel file on disk.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. The verdict — not reached, and why that is the correct outcome
|
||||||
|
|
||||||
|
The drill's pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established that
|
||||||
|
the sentinel is not in the off-site snapshot at all.** Wiping the box would therefore have:
|
||||||
|
|
||||||
|
- destroyed the sentinel, which exists only on that box;
|
||||||
|
- proven nothing about recovery, because there would be nothing to recover;
|
||||||
|
- and done so *after* the point of no return.
|
||||||
|
|
||||||
|
The runbook's own rule applies: *"If this session finds itself writing code beyond Part 0, stop. That
|
||||||
|
means a precondition was wrong, and the finding outranks the drill."* A precondition was wrong. It was
|
||||||
|
one the runbook's P1–P6 table did not contain, because nobody knew to look for it.
|
||||||
|
|
||||||
|
**Sentinel sha256 (step 3), recorded and still on the box:**
|
||||||
|
`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`
|
||||||
|
at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt` (181 B).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. R-203 — the defect that halted the drill
|
||||||
|
|
||||||
|
**Measured, twice, on the live box.**
|
||||||
|
|
||||||
|
| what | path | exists? |
|
||||||
|
|---|---|---|
|
||||||
|
| the app's live bind, where the customer's files actually land | `/mnt/sys_drive/userdata/media/books` | **YES** (the sentinel is here) |
|
||||||
|
| the path the off-site capture set treats as the mandatory directory | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** |
|
||||||
|
|
||||||
|
The controller's own log, verbatim:
|
||||||
|
|
||||||
|
```
|
||||||
|
[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite:
|
||||||
|
/mnt/sys_drive/felhom-data/userdata/media/books
|
||||||
|
[INFO] [offbox] backed up calibre-web (…/backups/primary/calibre-web, 0 mandatory path(s))
|
||||||
|
[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s
|
||||||
|
```
|
||||||
|
|
||||||
|
**The run reported `ok`.** `last_status: ok`, `last_success` stamped, `snapshot_count: 3`. Nothing
|
||||||
|
customer-visible, nothing hub-visible and nothing in any counter says the mandatory directory was
|
||||||
|
dropped. This is `CLAUDE.md`'s recurring shape — *a path the customer thinks is protected is not in the
|
||||||
|
snapshot* — and the code even has the right words for it in a WARN nobody reads.
|
||||||
|
|
||||||
|
**The mechanism, from source.** `NamespaceRoot(drivePath, inGuestDrive)`
|
||||||
|
(`appbackup/paths.go:28-33`) appends `felhom-data` **when the drive IS the system data path**
|
||||||
|
(`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`,
|
||||||
|
`backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is documented and computed
|
||||||
|
as **`<HDD_PATH>/userdata`** (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive`
|
||||||
|
and an app deployed at `HDD_PATH=/mnt/sys_drive`, those two produce different directories.
|
||||||
|
|
||||||
|
**And the same compose file used BOTH roots.** From `docker inspect calibre-web`:
|
||||||
|
|
||||||
|
```
|
||||||
|
bind /mnt/sys_drive/felhom-data/userdata/import/calibre -> /cwa-book-ingest ← felhom-data root
|
||||||
|
bind /mnt/sys_drive/userdata/media/books -> /calibre-library ← NO felhom-data root
|
||||||
|
```
|
||||||
|
|
||||||
|
`${IMPORT_PATH}` resolved *with* the segment; `${USERDATA_PATH}` resolved *without* it. One deploy, one
|
||||||
|
template, two roots.
|
||||||
|
|
||||||
|
### What is measured and what is not — stated because the scope changes the fix
|
||||||
|
|
||||||
|
- **MEASURED:** the two paths disagree; the mandatory directory is absent from the snapshot; the run
|
||||||
|
reports `ok`; the only signal is a container-log WARN.
|
||||||
|
- **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a *supported* choice. It was chosen because
|
||||||
|
demo-hp's only registered drive, `Felhom-Share`, is a NAS and was **correctly refused** as an app
|
||||||
|
namespace (R-108's `RefuseAsAppNamespace`, working as designed). `/mnt/sys_drive` was **accepted**
|
||||||
|
(HTTP 202) rather than refused.
|
||||||
|
|
||||||
|
**Either branch is a defect, which is why this is filed regardless:**
|
||||||
|
|
||||||
|
- if the system drive **is** a supported app namespace → the userdata path resolution is wrong for
|
||||||
|
every app deployed on it, and their mandatory directories are silently unprotected;
|
||||||
|
- if it is **not** supported → the deploy accepted a namespace it should have refused, exactly as it
|
||||||
|
refused the NAS one call earlier, and the refusal that exists is not reaching this case.
|
||||||
|
|
||||||
|
**What must NOT be concluded from this drill:** that off-site backups are broken generally. The two
|
||||||
|
pre-existing apps (`opengist`, `privatebin`) declare **no** mandatory userdata paths — everything they
|
||||||
|
own is in named volumes — so they are unaffected, and their snapshots are real.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Preconditions, each measured
|
||||||
|
|
||||||
|
| # | Precondition | Result |
|
||||||
|
|---|---|---|
|
||||||
|
| **P1** | operator holds the recovery code | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file (`~/.config/credentials`). *The code itself appears nowhere in this record.* |
|
||||||
|
| **P2** | demo-hp on agent v0.125.0 + controller v0.196.0 | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed the published v0.125.0 (sha `f7d8339b53d9…`, verified against the release) and controller v0.196.0. `age` present at `/usr/bin/age`. |
|
||||||
|
| **P3** | the current `identity_blob` seals the repository under test | **PASS with a caveat that reshaped the drill** — `identity_blob` = 572 B, `restic_pw_sha256` = `8a9e33aa4da6…`. But **no repository existed under that key** (see §4). |
|
||||||
|
| **P4** | a deliberate, verified whole-guest archive as rollback | **NOT TAKEN** — deliberately. It is only needed for the wipe, and the wipe did not happen. |
|
||||||
|
| **P5** | demo-felhom untouched and healthy | **PASS** — not touched at any point. |
|
||||||
|
| **P6** | space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 % used, guest 64 GB free. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Step-by-step, with every observable
|
||||||
|
|
||||||
|
### Step 1 — starting state (hub, read-only)
|
||||||
|
|
||||||
|
`demo-hp-bb76ea`: `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`, created `2026-08-04T11:11:37Z`, `escrow_state: escrowed`.
|
||||||
|
Report: `snapshot_count: 0`, `repo_size_bytes: 0`, **no `last_run`/`last_status` at all** — the shape of
|
||||||
|
a controller that has never run an off-site backup in this lifetime.
|
||||||
|
|
||||||
|
### Step 1b — the repository was ORPHANED, and this is the first live proof of a prediction
|
||||||
|
|
||||||
|
The 2026-08-04 spike predicted from source — and could not measure — that a rebuilt box's next
|
||||||
|
off-site run would hit a **third** outcome: neither reattaching the old snapshots nor silently starting
|
||||||
|
a fresh history, but **refusing**. Measured here, twice over.
|
||||||
|
|
||||||
|
**Read-only probe first** (`restic cat config` with the current key, writes nothing):
|
||||||
|
|
||||||
|
```
|
||||||
|
Fatal: wrong password or no key found
|
||||||
|
```
|
||||||
|
|
||||||
|
— the exact string `classifyResticProbe` maps to `"orphaned"`.
|
||||||
|
|
||||||
|
**Then the real run, through the customer UI endpoint** (`POST /backup/offbox/run`):
|
||||||
|
|
||||||
|
```
|
||||||
|
[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous,
|
||||||
|
no-longer-available key; runs will skip until reset
|
||||||
|
[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset)
|
||||||
|
[INFO] Event pushed: offbox_repo_orphaned (warning) — A távoli mentési tároló elárvult …
|
||||||
|
```
|
||||||
|
|
||||||
|
`repo_state: "orphaned"`, `orphaned_at: 2026-08-04T12:37:30Z`, `last_status: "error"`, the orphan card
|
||||||
|
rendered on `/backups/remote`, and the event reached the hub (HTTP 200).
|
||||||
|
|
||||||
|
> **The system stopped and said so. It did not quietly start a new history over the old one.**
|
||||||
|
> That closes R-193's open Q3 — and it is the good half of this month's story.
|
||||||
|
|
||||||
|
**Why the repository was orphaned:** the 15 snapshots / 40.9 MB at `/home/felhom-repo` were written
|
||||||
|
under key `8e03eddf9ff7…` before the 2026-08-03 rebuild. That key survives only in superseded escrow
|
||||||
|
row **id 3**, which carries **`identity_blob` = NULL** — it was superseded at `2026-08-04 07:15:36`,
|
||||||
|
**four hours before** hub v0.93.0 fixed the retention. Unrecoverable, permanently, with or without a
|
||||||
|
recovery code.
|
||||||
|
|
||||||
|
### Step 1c — the reset (operator-authorised)
|
||||||
|
|
||||||
|
The orphan card's own reset, confirmed by the operator during the session:
|
||||||
|
|
||||||
|
```
|
||||||
|
[WARN] [offbox] resetting orphaned repo (operator-confirmed (claimed)):
|
||||||
|
move-aside /home/felhom-repo -> /home/felhom-repo.orphaned-20260804, then re-init
|
||||||
|
[INFO] [offbox] orphaned repo reset complete — old history set aside at
|
||||||
|
/home/felhom-repo.orphaned-20260804 (move-aside, not deleted); fresh repo initialized
|
||||||
|
[INFO] Event pushed: offbox_repo_reset (info)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Nothing was deleted.** The reset path had never run in anger before; it works.
|
||||||
|
|
||||||
|
### Steps 2–3 — the recovery code and the sentinel
|
||||||
|
|
||||||
|
The operator ran the ceremonies for **both** boxes earlier the same day and saved the codes. Measured
|
||||||
|
hub-side: the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z`
|
||||||
|
(demo-felhom), and `restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated
|
||||||
|
them as same-password re-ceremonies: **no superseded row, no `offsite_repo_key_changed`**. That is
|
||||||
|
v0.93.0's Scenario E, live.
|
||||||
|
|
||||||
|
**A file-leg app had to be deployed**, and this is a precondition the runbook did not anticipate:
|
||||||
|
neither off-site-toggled app on the box (`opengist`, `privatebin`) has a restorable file leg — both
|
||||||
|
keep everything in **named volumes**, which the off-site tier backs up as tars but the customer restore
|
||||||
|
flow **never unpacks** (`offbox_reconstitute.go`). A sentinel in either would have been unrecoverable
|
||||||
|
by design.
|
||||||
|
|
||||||
|
`calibre-web` was chosen: it declares `userdata: media/books class: mandatory`, and it is a
|
||||||
|
single-container app. Deployed through the real API (`POST /api/stacks/calibre-web/deploy`, HTTP 202),
|
||||||
|
toggled for off-site, and a Tier-1 recovery unit captured
|
||||||
|
(`Recovery unit captured for calibre-web → …/backups/primary/calibre-web`).
|
||||||
|
|
||||||
|
Sentinel written, `sync`ed, hashed: **`643166269103a25c…`**, 181 B.
|
||||||
|
|
||||||
|
### Step 4 — the pre-wipe off-site backup: **`ok`, and wrong**
|
||||||
|
|
||||||
|
```
|
||||||
|
last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped
|
||||||
|
snapshot_count 3 · repo_size_bytes 30 636 · repo_state null
|
||||||
|
```
|
||||||
|
|
||||||
|
Three snapshots, three apps, 37 s — and **the sentinel is in none of them**, per §2.
|
||||||
|
|
||||||
|
### Steps 5–11 — NOT RUN
|
||||||
|
|
||||||
|
Step 5 (verify the pre-wipe snapshot contents) would have confirmed the absence a second way; it is
|
||||||
|
moot given §2. Steps 6–11 (archive, wipe, reinstall, recover, install, restore, compare) were **not
|
||||||
|
started**. The §7 STOP was never reached, because the drill failed its own precondition first.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Part 0 — shipped, tested, deployed (R-200's plumbing half)
|
||||||
|
|
||||||
|
`--recover-offsite-install` (controller **v0.196.0**, commit `1b1366b`): same fetch → unseal → extract
|
||||||
|
path as `--recover-offsite-check`, same STDIN discipline for R, but it **places** the recovered
|
||||||
|
password via `InjectOffboxPassword`.
|
||||||
|
|
||||||
|
- **Confirmation is a second invocation.** Without `--confirm-install` it prints both hashes and writes
|
||||||
|
nothing. A single interactive prompt would have had to share stdin with R.
|
||||||
|
- **Three outcomes, named distinctly:** *installed* (no local password — the rebuilt-box shape),
|
||||||
|
*unchanged* (identical key already present, nothing written), *refused* (a DIFFERENT key present;
|
||||||
|
installing would clobber the key the current repository is encrypted under — exit 2, no force offered).
|
||||||
|
- It re-reads the file after writing rather than trusting the call's return.
|
||||||
|
|
||||||
|
**Tests + red-proof.** `go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK.
|
||||||
|
Removing the confirmation gate makes the dry run write the password and fails
|
||||||
|
`TestRecoverAndInstall_InstallsOnABareBox` with *"the DRY RUN wrote the password"*. The R-persistence
|
||||||
|
test carries a **positive control** — a planted copy of the code is found by the sweep, then removed
|
||||||
|
and not found — because an absence check is worth only what its sensitivity is.
|
||||||
|
|
||||||
|
**It was deployed to demo-hp and never exercised against a live recovery**, because the drill halted
|
||||||
|
before step 9. Its unit proof stands; its live proof does not exist.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. What this drill did and did not establish
|
||||||
|
|
||||||
|
**Established, live, for the first time:**
|
||||||
|
|
||||||
|
1. A rebuilt box's off-site repository is **orphaned and the run refuses** — the spike's predicted
|
||||||
|
third outcome, measured. It does not silently start a fresh history.
|
||||||
|
2. The **orphan reset works**: move-aside, never delete, fresh repo initialised, event pushed.
|
||||||
|
3. **A customer-declared mandatory data directory can be silently absent from the off-site snapshot
|
||||||
|
while the run reports `ok`** (R-203).
|
||||||
|
4. demo-hp's pre-rebuild off-site history is **permanently unrecoverable** — its key was destroyed four
|
||||||
|
hours before the fix that would have kept it.
|
||||||
|
|
||||||
|
**NOT established — and unchanged from before this session:**
|
||||||
|
|
||||||
|
- **No file has ever been restored from an off-site backup after a wipe.** R-201's question is still
|
||||||
|
open, and its pass condition is unchanged.
|
||||||
|
- Part 0's install path has never run against a live recovery.
|
||||||
|
- The v0.93.0 `identity_blob` retention is **still unit-proven only** — nothing in this session
|
||||||
|
superseded a key, so nothing exercised it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. State left behind, and teardown
|
||||||
|
|
||||||
|
**Deliberately not torn down** — this is evidence, and the box is better off than it was:
|
||||||
|
|
||||||
|
| layer | state |
|
||||||
|
|---|---|
|
||||||
|
| the guest | `calibre-web` deployed and running, off-site-toggled, sentinel file in place. **Kept** as the drill fixture for the resumed run — it is the only app on either demo box with a restorable file leg. |
|
||||||
|
| the off-site repository | fresh, working, 3 snapshots, 30 636 B. The old 15-snapshot history is **set aside**, not deleted, at `/home/felhom-repo.orphaned-20260804`. |
|
||||||
|
| the host | agent v0.125.0, controller v0.196.0, `pvesm` unchanged apart from normal usage. |
|
||||||
|
| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` events recorded for `demo-hp`. No scratch customer records were created — **nothing was reinstalled**. |
|
||||||
|
|
||||||
|
**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — that act is ruled and still owed,
|
||||||
|
and §8.3 of the runbook forbids riding it along with a drill. The reset added
|
||||||
|
`/home/felhom-repo.orphaned-20260804` (≈41 MB) to what is set aside.
|
||||||
|
|
||||||
|
**No `R` persisted anywhere** — the recovery code was never used in this session. It was read only to
|
||||||
|
confirm the key exists in the credentials file; no unseal was performed on demo-hp.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7b. UPDATE, same day — the blocker is FIXED and the drill is ready to resume
|
||||||
|
|
||||||
|
Controller **v0.197.0** shipped both halves of R-203:
|
||||||
|
|
||||||
|
- **the paths agree.** On demo-hp the live bind moved from `/mnt/sys_drive/userdata/media/books` to
|
||||||
|
`/mnt/sys_drive/felhom-data/userdata/media/books` — the directory the capture set looks in. The
|
||||||
|
capture log went from `0 mandatory path(s)` to **`1 mandatory path(s)`**.
|
||||||
|
- **`ok` means it.** A run that cannot capture a MANDATORY directory now reports **`incomplete`**,
|
||||||
|
names the app and the folders, and raises the operator digest — instead of `ok` with a warning
|
||||||
|
beside it.
|
||||||
|
|
||||||
|
**And the sentinel is in the snapshot, listed by name:**
|
||||||
|
|
||||||
|
```
|
||||||
|
$ restic ls -l latest --tag calibre-web
|
||||||
|
-rw-r--r-- 1000 1000 181 2026-08-04 12:53:06 /mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
sha256 **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`** — byte-identical to §1,
|
||||||
|
verified after the fix's migration moved the file to the corrected directory.
|
||||||
|
|
||||||
|
**One live check could NOT be reproduced, and is recorded rather than claimed.** Hiding the books
|
||||||
|
directory to watch the `incomplete` verdict fire on hardware did not work: the running container's
|
||||||
|
bind mount **recreated** the directory, so `os.Stat` succeeded and there was no gap. That is itself
|
||||||
|
worth knowing — a bind-mounted directory cannot easily be "missing" while its app runs, so the
|
||||||
|
mandatory-gap condition arises in practice when the path resolves somewhere the app never binds (the
|
||||||
|
R-203 case), not when a live app's own directory vanishes. The verdict is proven by the run-level
|
||||||
|
test, which drives the real `RunOffboxBackup` and asserts both `incomplete` and the operator signal.
|
||||||
|
The fixture was restored and the sentinel re-verified at the same hash.
|
||||||
|
|
||||||
|
## 8. To resume the drill
|
||||||
|
|
||||||
|
1. **Fix or scope R-203.** Until the mandatory userdata path lands in the snapshot, no sentinel can
|
||||||
|
survive a wipe and the drill cannot reach its verdict.
|
||||||
|
2. Re-run steps 4–5 and confirm the sentinel IS in the snapshot — by listing it, not by a green status.
|
||||||
|
3. Then P4 (the deliberate archive), the §7 STOP, and steps 6–11 as written.
|
||||||
|
|
||||||
|
Everything else is already in place: the code, the versions, the recovery code, the working repository,
|
||||||
|
the file-leg app and the sentinel.
|
||||||
@@ -0,0 +1,531 @@
|
|||||||
|
# RECON — can a customer actually get their backups back?
|
||||||
|
|
||||||
|
**Date:** 2026-08-04 · **Items:** R-193, R-192 (+ new: R-198…R-201) · **Class:** recon — **no production code**
|
||||||
|
**Session scope:** read-only everywhere. Nothing built, deployed, wiped, rotated, re-issued or deleted.
|
||||||
|
|
||||||
|
> **Headline, before the detail.** The chain is **not** assembled, and the reason is worse than "two
|
||||||
|
> links are manual". **The hub's superseded-escrow retention — the feature built for exactly this
|
||||||
|
> incident — does not retain the offsite repository password.** `host_escrow_superseded` has no
|
||||||
|
> `identity_blob` column, and `demoteCurrentEscrowTx` copies only the K-escrow blob. So the last copy
|
||||||
|
> of a repo password is destroyed by the **next escrow ceremony** — the very act the system tells the
|
||||||
|
> customer to perform when their escrow goes `pending` after a rebuild. The recovery window closes
|
||||||
|
> because the remedy closes it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Confirmed baselines (re-read on arrival)
|
||||||
|
|
||||||
|
| Repo | `main` @ commit | tree | Version |
|
||||||
|
|------|-----------------|------|---------|
|
||||||
|
| `felhom-agent` | `856a127cd60b0be797867c162f0ae49c7b64993e` | clean | v0.124.1 |
|
||||||
|
| `felhom-controller` | `0887fd676d608e7c103fa13b3e87536ad2481811` | clean | v0.194.0 |
|
||||||
|
| `felhom.eu` | `d26f49ad680e2be2ce2888fd58dff7dee886e4ad` | clean | hub v0.92.0 |
|
||||||
|
|
||||||
|
All three match the prompt's cited commits exactly. `git status --porcelain` empty in each.
|
||||||
|
|
||||||
|
**Highest register ID: R-197** (`OPEN-ITEMS.md` R-196/R-197 were minted by yesterday's spike;
|
||||||
|
R-194/R-195 by the R-190 arc). Grepped across `OPEN-ITEMS.md`, `ROADMAP.md`, `STATUS.md`,
|
||||||
|
`CONTEXT.md`. New items in this session therefore start at **R-198**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Instruments — what each can and cannot answer
|
||||||
|
|
||||||
|
| # | Instrument | Exact read | Known limit |
|
||||||
|
|---|---|---|---|
|
||||||
|
| I1 | source | the three repos at the commits above | the authority; every claim below cites a file:line |
|
||||||
|
| I2 | hub SQLite snapshot | `kubectl -n felhom-system exec pod/hub-6775889d95-gc8sk -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | the `-wal` is load-bearing (copied; `PRAGMA integrity_check` → `ok`) |
|
||||||
|
| I2-fresh | freshness proof | `MAX(host_reports.received_at)` = `2026-08-04 10:00:40`, `datetime('now')` = `2026-08-04 10:03:48` | **positive observable**: newest row 3 m 08 s old, not "the query did not error" |
|
||||||
|
| I3 | live filesystem, demo-felhom guest 9201 | `ssh felhom-pve "pct exec 9201 -- ls -la --time-style=full-iso …"` | read-only `ls`; mtimes only, no file contents read |
|
||||||
|
| I4 | caller census | `grep -rn <symbol> --include=*.go` across all four repos | proves absence of a caller in Go; templates checked separately |
|
||||||
|
|
||||||
|
**Not used, deliberately:** no ceremony, no re-issue, no rotation, no `selftest=identity-consume`, no
|
||||||
|
write of any kind, nothing touched on the storage endpoint.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Q1 — the chain, link by link
|
||||||
|
|
||||||
|
From a dead node to an open repository. **Status vocabulary as specified.**
|
||||||
|
|
||||||
|
| # | Link | Component / function | Who invokes it | Status |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| 1 | repo password is minted | controller `Manager.WriteOffboxSecrets` → mint at `internal/backup/offbox.go:392-401`; generator `:405` | `ApplyOffsiteTarget` (`offbox.go:474`) ← `ConfigureOffbox` ← the apply-bridge (`offsiteapply.go:221,256`) | **PROVEN-LIVE** — fires on every fresh data dir; measured twice this week (I2 hash change, I3 mtime) |
|
||||||
|
| 2 | password staged to the agent | controller `PushOffboxPasswordForEscrow` (`offbox.go:530`) → agent `POST /escrow/stage-secret` (`localapi/server.go:518`) → `escrow.StagedResticPasswordPath()` = `/var/lib/felhom-agent/escrow-stage/restic_repo_password`, 0600 (`identity.go:44-47`) | the controller enable/ceremony path | **PROVEN-LIVE** |
|
||||||
|
| 3 | sealed under R at the ceremony | `AttachResticPassword` (`identity.go:73`) → `WrapIdentityBundle` → `WrapIdentity` (`identity.go:114`, `age -p -a`, scrypt + ChaCha20-Poly1305) | agent `--selftest=escrow-create`, driven by the controller wizard (`/escrow/ceremony`, `localapi/server.go:525-527`) | **PROVEN-LIVE** — customer-facing wizard first live firing 2026-07-18 (`00-capability-map.md`, escrow row) |
|
||||||
|
| 4 | stored by the hub | `PUT /api/v1/hosts/<id>/escrow` (`api/handler.go:239`, `handleHostEscrowPut :1104`) → `host_escrow.identity_blob` | agent `main.go:2937` | **PROVEN-LIVE** — measured: `identity_blob` present, 572 B, for **both** hosts (I2) |
|
||||||
|
| 5 | **old blob retained on supersession** | `demoteCurrentEscrowTx` (`store/store.go:2547-2556`) → `host_escrow_superseded` | `SaveHostEscrow`, `DeleteHost` | **BROKEN FOR THIS PURPOSE.** The INSERT copies `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256` — **not `identity_blob`**, and the table has **no such column** (`PRAGMA table_info`, I2). The K-escrow survives; **the restic repo password does not** |
|
||||||
|
| 6 | served back to a re-enrolling box | hub `handleReEnroll` (`api/dr.go:101`) and `handleGetRestoreDirective` (`:155`), returning `identity_escrow_b64`; recovery-mode gated, armed with the **global operator key** (`:34`) | **NOBODY.** I4: zero callers in `felhom-agent` (no `ReEnroll` symbol, no `/re-enroll` or `/restore-directive` string anywhere in the agent's hub client, `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`); zero in the hub UI (no template mentions them); zero in `scripts/` or any runbook | **implemented-never-exercised, and CLIENTLESS.** The only documented retrieval is by hand: `sqlite3 "SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …"` on a `kubectl cp`-ed `hub.db` (project memory, 2026-07-04 S5 prep) |
|
||||||
|
| 7 | unsealed with R | `UnwrapIdentityBundle` (`identity.go:172`) → `UnwrapIdentity` (`:139`) | **`runSelftestIdentityConsume` only** (`cmd/felhom-agent/main.go:2845-2900`), reading R from `FELHOM_RECOVERY_CODE`. I4: no other production caller | **manual-only — the only caller is a `--selftest` mode** |
|
||||||
|
| 8 | the repo password extracted | none. The selftest writes the **whole bundle JSON** to `-keydest` 0600 (`main.go:2872`) and prints *"identity recovered (tunnel_token + pbs_token)"* — it does not mention, extract or route `restic_repo_password` at all | a human reads the JSON | **missing** |
|
||||||
|
| 9 | placed on the rebuilt controller | `InjectOffboxPassword` (`offbox.go:541`) | one caller: `offboxInjectPasswordHandler` (`web/offbox_handlers.go:189`) at `POST /backup/offbox/inject-password` (`web/server.go:510`). **I4: no template in the repo contains that path or any form posting to it** | **manual-only, and UI-unreachable** — the handler exists, the form does not. A person must POST it by hand with a valid session cookie + CSRF token |
|
||||||
|
| 10 | the existing repo opens | `ensureOffboxRepo` (`offbox.go:653`): `restic cat config` → success clears `RepoState` | the scheduled/manual offbox run | **implemented-never-exercised** on a *recovered* password (exercised constantly on a live one) |
|
||||||
|
| 11 | a backup or restore proves it | — | — | **never** |
|
||||||
|
|
||||||
|
### The chain in one sentence
|
||||||
|
|
||||||
|
Links 1–4 are real and proven. **Link 5 silently drops the payload.** Links 6–9 are each individually
|
||||||
|
manual, and link 6 has no client at all. Link 11 has never happened.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Q2 — where the chain stops today
|
||||||
|
|
||||||
|
**The first link that is not automatic is #5, and it is not merely manual — it is lossy.**
|
||||||
|
|
||||||
|
Two distinct stopping points, and the order matters:
|
||||||
|
|
||||||
|
**(a) If the box has NOT re-escrowed since the rebuild** — the current `host_escrow.identity_blob`
|
||||||
|
still seals the *old* password. Recovery is possible in principle, and a person must:
|
||||||
|
|
||||||
|
1. Arm recovery mode on the hub (`PUT /api/v1/admin/hosts/<id>/recovery-mode`, global operator key) —
|
||||||
|
or skip it entirely, because nothing consumes the endpoint it gates;
|
||||||
|
2. `kubectl cp` / `exec cat` the hub DB and `sqlite3 … writefile('/root/idblob', identity_blob)`;
|
||||||
|
3. copy the blob to a machine with the agent binary and `age`;
|
||||||
|
4. `FELHOM_RECOVERY_CODE=… felhom-agent --selftest=identity-consume -blob /root/idblob -keydest /root/bundle.json`;
|
||||||
|
5. open `bundle.json` by hand and read `restic_repo_password` out of it;
|
||||||
|
6. log in to the customer's dashboard and `POST /backup/offbox/inject-password` **by hand** (curl with
|
||||||
|
session + CSRF) — there is no form;
|
||||||
|
7. re-run the apply so `WriteOffboxSecrets` finds the file present and keeps it;
|
||||||
|
8. trigger a run and hope `restic cat config` succeeds.
|
||||||
|
|
||||||
|
Eight manual steps, five of them requiring operator-tier access to the hub, one requiring the customer
|
||||||
|
to be present with R, and one (step 6) with no user interface at all.
|
||||||
|
|
||||||
|
**(b) If the box HAS re-escrowed** — which is what the system pushes the customer to do, because a
|
||||||
|
rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`,
|
||||||
|
`offbox.go:570`), and the card says *„Helyreállítási kód szükséges"* — then the old `identity_blob`
|
||||||
|
has been **overwritten** by `SaveHostEscrow` and **not carried** into the superseded row. At that point
|
||||||
|
**no procedure exists**, with or without R, at any level of effort. The password is gone.
|
||||||
|
|
||||||
|
Both demo boxes are in state (b) as of 2026-08-04 07:15:36 (demo-hp) and 07:20:08 (demo-felhom) — I2,
|
||||||
|
`host_escrow.updated_at`. This is an **independent, and much stronger, reason** the 51 orphaned
|
||||||
|
snapshots are unrecoverable than "nobody kept the recovery codes". Keeping R would not have helped.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Q3 — can a rebuilt controller know it should recover?
|
||||||
|
|
||||||
|
### RULING: **YES — the hub already tells it, on every single report, and the controller throws the answer away.**
|
||||||
|
|
||||||
|
**What the box knows at the mint decision.** `WriteOffboxSecrets` is called from
|
||||||
|
`ApplyOffsiteTarget`, driven by the apply-bridge with the hub's offsite **descriptor**, whose fields
|
||||||
|
are `host, user, port, repo_path, host_fingerprint, quota_gb` (`offsiteapply.go:198-204`). **Nothing
|
||||||
|
about escrow is in scope at that point.** Locally, the only signal is the one the mint branch already
|
||||||
|
reads: `os.Stat(m.offboxPwPath())` returning `IsNotExist` — which *is* the fresh-data-dir signal, at
|
||||||
|
exactly the right line, but cannot distinguish "never configured" from "rebuilt".
|
||||||
|
|
||||||
|
**What the box is nonetheless already told.** Every report ACK carries an `escrow` object
|
||||||
|
(`api/handler.go:504-510`), consumed as `report.EscrowStatus`
|
||||||
|
(`internal/report/escrow_confirm.go:19-23`):
|
||||||
|
|
||||||
|
```go
|
||||||
|
type EscrowStatus struct {
|
||||||
|
IdentityBlobPresent bool `json:"identity_blob_present"`
|
||||||
|
ResticPwSHA256 string `json:"restic_pw_sha256"`
|
||||||
|
CreatedAt string `json:"created_at"`
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
That is precisely *"the hub has a blob, and here is the hash of the password it covers"* — arriving
|
||||||
|
every report cycle, on a channel that already exists, with no new API and no new secret.
|
||||||
|
|
||||||
|
**And it is discarded.** `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:75-84`) returns
|
||||||
|
immediately unless `Pending()` or `Escrowed()` — both of which require `OffboxConfigured()` **and** a
|
||||||
|
non-nil target (`cmd/controller/main.go:580-590`). On a freshly rebuilt box, before the offsite apply,
|
||||||
|
there is no target, so the status is dropped on the floor. It is never persisted anywhere: I4 finds
|
||||||
|
exactly **one** reference to the ACK field in the whole controller
|
||||||
|
(`cmd/controller/main.go:641: escrowConfirmer.Reconcile(resp.Escrow)`).
|
||||||
|
|
||||||
|
**Therefore a fail-closed refusal IS implementable**, with no new hub endpoint:
|
||||||
|
|
||||||
|
- persist the last-seen ACK escrow status alongside settings (the precedent exists — `ClaimSync`
|
||||||
|
caches the hub's claim-code state into `settings.json` idempotently by generation,
|
||||||
|
`internal/report/claim_sync.go:39-53`, and it does so *set-only, never cleared on hub silence*,
|
||||||
|
which is the correct shape here too);
|
||||||
|
- at the mint branch, refuse when the password file is absent **and** the cached status says
|
||||||
|
`identity_blob_present && restic_pw_sha256 != ""`.
|
||||||
|
|
||||||
|
**One ordering caveat, stated rather than assumed.** On a truly fresh box the apply can race the first
|
||||||
|
ACK. Measured on demo-hp: first post-rebuild report `2026-08-03 06:12:19`, offsite apply
|
||||||
|
`2026-08-04 07:12` — the ACK preceded the apply by 25 hours, but that gap was an incident, not a
|
||||||
|
design. A fail-closed rule handles the race correctly by construction: refuse to mint until the escrow
|
||||||
|
status is known, and retry next cycle (the apply-bridge is already built to retry and to persist
|
||||||
|
nothing on a partial failure, `offsiteapply.go:1-6`).
|
||||||
|
|
||||||
|
**Corroborating measurement that the data dir really is fresh on a rebuild** (I3, demo-felhom guest
|
||||||
|
9201, `/var/lib/docker/volumes/felhom-controller-data/_data/data/`):
|
||||||
|
|
||||||
|
| file | mtime |
|
||||||
|
|---|---|
|
||||||
|
| the data directory's parent | `2026-08-03 07:17:25` |
|
||||||
|
| `encryption.key` (32 B) | `2026-08-03 07:17:26` |
|
||||||
|
| `offbox/` | `2026-08-03 07:18:02` |
|
||||||
|
| `offbox/repo_password` (64 B) | `2026-08-03 07:18:02` |
|
||||||
|
|
||||||
|
The volume, the app-secret encryption key and the repo password are all newborn, 36 seconds apart.
|
||||||
|
Corroborated hub-side (I2): `json_extract(report_json,'$.claimed')` reads **1 → 0 → 1** across the
|
||||||
|
rebuild on both boxes — demo-hp `0` in 5 reports `2026-08-03 06:12:19`–`06:21:03`; demo-felhom `0` in 4
|
||||||
|
reports `07:17:54`–`07:19:46`. `Claimed` is set only by `SetClaimed()`, whose sole caller is the claim
|
||||||
|
submit handler (`web/claim.go:349`), so the return to `1` was a **human re-claim**, not automation.
|
||||||
|
|
||||||
|
> **Note, not acted on:** a fresh `encryption.key` means every pre-rebuild `app.yaml` `ENC:` secret is
|
||||||
|
> undecryptable — the C-1 circular dependency in `_recovery-inventory-2026-07-28.md` fired on both
|
||||||
|
> demo boxes this week. That is outside R-193's scope and is recorded as an observation (§13).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Q4 — what the stale-escrow detector actually covers
|
||||||
|
|
||||||
|
### RULING: **the prompt's premise conflates two different mechanisms. `escrow_confirm.go` DOES fire on a mint. But it fires into a log file, and the remedy it prescribes destroys the last copy of the old key.**
|
||||||
|
|
||||||
|
**The conflation, stated because it matters.** Yesterday's spike's sentence *"wired to the one path
|
||||||
|
that doesn't change the key and absent from the one that does"* is about the **hub's `escrow_stale`
|
||||||
|
event**, emitted by `ReissueCredentials` (`hub/internal/offsite/offsite.go:198-201`) — that is R-196,
|
||||||
|
and it stands. It is **not** about `controller/internal/report/escrow_confirm.go`. Two different
|
||||||
|
detectors; only one of them was ever claimed to be misplaced.
|
||||||
|
|
||||||
|
**What `escrow_confirm.go` does on a mint, from source.** After a rebuild the apply creates a target
|
||||||
|
with `EscrowState: "pending"` (the `cur == nil` fallthrough, `offbox.go:482-494`). So `Pending()` is
|
||||||
|
true, the pending branch runs (`escrow_confirm.go:86-107`), the hub's hash (sealing the OLD password)
|
||||||
|
is compared against `LocalHash()` (the NEW minted one), they differ, and it logs:
|
||||||
|
|
||||||
|
```
|
||||||
|
[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password
|
||||||
|
(hub hash %.12s… != local %.12s…) — run the escrow ceremony … staying pending
|
||||||
|
```
|
||||||
|
|
||||||
|
**So: confirmed, it fires.** The detector is not absent from the mint path.
|
||||||
|
|
||||||
|
**Three limits, each of which matters more than the fact that it fires.**
|
||||||
|
|
||||||
|
1. **It is a controller-local log line.** No event, no e-mail, no hub-side signal. `StaleBlob()` — the
|
||||||
|
flag the web card renders — is set **only** by the escrowed branch (`reconcileEscrowed`,
|
||||||
|
`:130-165`); the pending branch sets no display flag. The operator learns nothing.
|
||||||
|
2. **Its prescription is the destructive act.** *"run the escrow ceremony"* → a new ceremony →
|
||||||
|
`SaveHostEscrow` overwrites `host_escrow.identity_blob` → `demoteCurrentEscrowTx` does not carry it
|
||||||
|
→ **the old repo password ceases to exist**. The detector correctly identifies the situation and
|
||||||
|
then tells the customer to close the door.
|
||||||
|
3. **It cannot distinguish the two situations that need different answers** — "you re-keyed
|
||||||
|
deliberately" and "your machine was rebuilt and your history is about to be orphaned" produce the
|
||||||
|
identical warning.
|
||||||
|
|
||||||
|
So: **less needs building than the prompt feared for the *detection*, and more needs building for the
|
||||||
|
*consequence*.** The comparison exists; what does not exist is a path from that comparison to either a
|
||||||
|
signal or a recovery.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Q5 — is the superseded-escrow retention real, and what does it retain?
|
||||||
|
|
||||||
|
### RULING: **the retention is real, it retains the wrong blob, and nothing can read it back.**
|
||||||
|
|
||||||
|
**What it retains — source and live, agreeing.** `demoteCurrentEscrowTx` (`store/store.go:2547-2556`):
|
||||||
|
|
||||||
|
```sql
|
||||||
|
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
|
||||||
|
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
|
||||||
|
FROM host_escrow WHERE host_id = ?
|
||||||
|
```
|
||||||
|
|
||||||
|
`identity_blob` is not in the column list. Live schema (I2, `PRAGMA table_info`):
|
||||||
|
|
||||||
|
| table | columns |
|
||||||
|
|---|---|
|
||||||
|
| `host_escrow` | `host_id, blob, key_fingerprint, posture, created_at, updated_at, **identity_blob**, directive_json, restic_pw_sha256, stale_at` |
|
||||||
|
| `host_escrow_superseded` | `id, host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at` |
|
||||||
|
|
||||||
|
**`blob` is the K-escrow — the PBS datastore encryption key**, wrapped by the PBS-native scrypt path.
|
||||||
|
**`identity_blob` is the age-wrapped `IdentityBundle`** — the one carrying `ResticRepoPassword`
|
||||||
|
(`felhom-agent/internal/escrow/identity.go:34-39`), plus `TunnelToken`, `PBSToken` and
|
||||||
|
`WGPrivateKey`. Live sizes confirm they are different objects: both hosts' current rows hold
|
||||||
|
`blob` = 383 B **and** `identity_blob` = 572 B; both superseded rows hold `blob` = 383 B and nothing
|
||||||
|
else.
|
||||||
|
|
||||||
|
**So, exactly what a customer with the matching OLD recovery code could recover from a retained blob:**
|
||||||
|
|
||||||
|
- ✅ the PBS datastore encryption key **K** — i.e. the Plane-2 whole-guest offsite ciphertext;
|
||||||
|
- ❌ **not** the restic repository password — the Tier-3 offsite app-data history;
|
||||||
|
- ❌ not the tunnel token, not the PBS access token, not the WireGuard private key.
|
||||||
|
|
||||||
|
**And there is no read path.** I4: `ListSupersededEscrow` (`store/store.go:2617`) has **zero
|
||||||
|
production callers** — only two tests. The only production surface is `CountSupersededEscrow`, used
|
||||||
|
for a number on the host page (`web/hosts.go:573`) and a log line (`api/handler.go:1151`). The schema
|
||||||
|
comment says so itself: *"The ACK and restore-serving read `host_escrow` (the CURRENT row) — never
|
||||||
|
this table."* Retrieval is `sqlite3` on the hub pod, by hand, by the operator.
|
||||||
|
|
||||||
|
**The claim this contradicts.** `00-capability-map.md` (escrow-ceremony row) records hub v0.60.0 as:
|
||||||
|
|
||||||
|
> *"a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS
|
||||||
|
> it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code
|
||||||
|
> (turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable')."*
|
||||||
|
|
||||||
|
The same claim appears in the schema comment (`store.go:370-375`) and, in Hungarian, on the customer's
|
||||||
|
own orphan card (`controller/internal/web/templates/backups_remote.html:66,69`):
|
||||||
|
|
||||||
|
> *„A meglévő mentések nem sérültek — a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók
|
||||||
|
> lehetnek."* · *„A régi előzmény félretéve marad (nem törlődik), és a hozzá tartozó helyreállítási
|
||||||
|
> kóddal később visszaállítható lehet."*
|
||||||
|
|
||||||
|
**For the offsite restic repository — the incident it names — that is false.** The retained blob does
|
||||||
|
not contain the key that opens it. This is the **ninth** entry in `CLAUDE.md`'s table of comments
|
||||||
|
asserting an invariant the code does not provide, and the first that is also **customer-facing copy**.
|
||||||
|
→ **R-198**, and the capability-map correction in §9.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Q6 — has any part of this ever been exercised?
|
||||||
|
|
||||||
|
**Method:** read every audit, validation doc, findings file and REPORT that mentions escrow consume,
|
||||||
|
identity-consume, re-enroll or inject-password; then check what the cited evidence actually covers.
|
||||||
|
|
||||||
|
### Proven
|
||||||
|
|
||||||
|
| what | evidence | scope of the proof |
|
||||||
|
|---|---|---|
|
||||||
|
| ceremony (create + seal + upload) | `00-capability-map.md` escrow row; `tests/VALIDATION-n100-rehearsal-2026-07-18.md` S6 — customer drove the wizard, R claimed one-shot 16:56:39, auto-confirmed 16:56:41 | the **create** side only |
|
||||||
|
| identity-bundle round-trip with a real R on a secret-less box | `tests/slice10d-identity-restore-spike-findings.md` §1, 2026-06-10 — wrap→unwrap byte-identical (`sha256` match); wrong-R **fails closed**, no plaintext emitted | **PRE-FORK-4.** The bundle then was `{tunnel_token, pbs_token}`. `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** (`felhom-agent/CHANGELOG.md:2344`) — a month later. **The proof does not cover the field this whole question is about** |
|
||||||
|
| `ResticRepoPassword` survives the wrap/unwrap | agent v0.77.0 CHANGELOG: *"`IdentityBundle` round-trip carries `ResticRepoPassword` byte-exact + not-in-blob + wrong-R fails closed"* | **unit test only** |
|
||||||
|
| `InjectOffboxPassword` places and refuses correctly | `controller/internal/backup/offbox_test.go:952-990`, incl. the companion showing a fresh mint without inject | **unit test only** |
|
||||||
|
|
||||||
|
### Never exercised — named in those words
|
||||||
|
|
||||||
|
- **A blob has never been served to a box.** Link 6 has no client (§3).
|
||||||
|
- **A fork-4 bundle has never been unsealed with a real R outside a unit test.** The only production
|
||||||
|
caller is a `--selftest` mode; the one live consume ever prepared (S5 Part 4-B, 2026-07-04) was
|
||||||
|
explicitly **deferred and never run** — its own resume note still reads *"DEFERRED 2026-07-04, not
|
||||||
|
run"* (project memory; `06-offsite-connectivity.md:327` marks the destructive drill operator-gated).
|
||||||
|
- **A recovered repo password has never been injected into a controller.**
|
||||||
|
- **An existing offsite repository has never been reopened with a recovered password.**
|
||||||
|
- **No restore of any kind has ever been performed from a recovered secret.**
|
||||||
|
|
||||||
|
`_recovery-inventory-2026-07-28.md` already said most of this and said it accurately — A.2.7 records
|
||||||
|
*"the **consume** side is proven at spike level … the destructive in-place drill is explicitly
|
||||||
|
operator-gated and unrun"*, and C.1 row 1 says *"escrow **creation** proven live; **consume never
|
||||||
|
exercised destructively**"*. **That document is right and this session confirms it.** What it did not
|
||||||
|
catch — because it was written from `host_escrow`, not from the supersession path — is Q5.
|
||||||
|
|
||||||
|
**Capability-map correction owed:** not for a "recovery proven" claim (no row makes one), but for the
|
||||||
|
**retention** claim quoted in §7. See §9.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Q7 — the operator's ruled shape, assessed row by row
|
||||||
|
|
||||||
|
> **Operator ruling, 2026-08-04, verbatim:** *If a node is a fresh install AND the hub has a recovery
|
||||||
|
> blob, then the controller should yell that recovery is available, and provide a form for the customer
|
||||||
|
> to enter the recovery key. After unlocking the blob, the controller should show what will be
|
||||||
|
> recovered before proceeding.*
|
||||||
|
|
||||||
|
| The ruling requires | Exists? | What it needs, and what it costs |
|
||||||
|
|---|---|---|
|
||||||
|
| the controller knows it is a **fresh install** | **YES, at the right line** | `os.Stat(m.offboxPwPath())` → `IsNotExist` is already the mint branch (`offbox.go:392`). It is trustworthy as *"this controller has no repo password"*, which is the only fact the decision needs; it does **not** by itself distinguish first-ever install from rebuild — that is what the next row supplies. Corroborators exist but are weaker: `Claimed` is set-only and was measured re-set by a human within minutes (§5); a fresh `encryption.key` is a stronger signal but is not currently read by anything. **Cost: zero — the branch already exists.** |
|
||||||
|
| the controller knows **the hub has a blob** | **YES on the wire, NO in the code** | `EscrowStatus{IdentityBlobPresent, ResticPwSHA256, CreatedAt}` arrives on every report ACK and is discarded when no offbox target exists (§5). **Needs:** persist it (the `ClaimSync` set-only-by-generation pattern is the precedent, `claim_sync.go:39-53`). **Cost: small — controller only, one settings field, one reconciler, no hub change.** |
|
||||||
|
| it can **yell** rather than silently mint | **NO** | The refusal belongs **inside `WriteOffboxSecrets`, at the mint branch** — the one place that decides. Returning a new sentinel (`ErrOffboxRecoveryAvailable`, sibling of `ErrOffboxOrphaned`) propagates through `ApplyOffsiteTarget` → `ConfigureOffbox` → the apply-bridge, which already treats a failed configure as "persist nothing, retry next cycle" (`offsiteapply.go:5-6, 256`) — so a refusal is **fail-safe by construction**, not a new failure mode. Meanwhile the customer sees a card on `/backups/remote` (the orphan card at `backups_remote.html:60-70` is the exact precedent) and offsite runs stay blocked, which they already are (`OffboxRunnable`). **Cost: small-medium — controller only.** **Design hazard to name:** a first-ever install of a *re-used customer record* would also trip this; the refusal must be dismissible ("start fresh anyway"), and dismissal must be the act that mints. |
|
||||||
|
| a **form takes R** | **NO — and R is not a password** | The controller UI **emits** R (`backups_escrow.html:235-236`, `/backup/escrow/…/claim` returns it once) and has **never accepted** one. The only thing resembling an entry point is `POST /backup/offbox/inject-password`, which takes a **64-hex repo password** (`offboxRepoPwPattern`, `offbox.go:543`) — a different secret at a different layer — and **has no form in any template** (§3 link 9). **Cost: small for the form; the hard part is the next row.** |
|
||||||
|
| the blob is **unsealed** with R | **NO — and there is a real boundary** | Unsealing is `age -d`. `ageBinary = "/usr/bin/age"` (`identity.go:24`) and the code live in the **agent**. The **controller image does not ship `age`** (`controller/Dockerfile:56-70` — restic, openssh-client, sshpass, sqlite3, docker-cli; no age) and must not: the agent is the privileged tier by design. **The crossing already has a transport and a precedent:** the pinned local API carries `POST /escrow/ceremony` + `GET /escrow/ceremony/status` + `POST /escrow/ceremony/claim` (`localapi/server.go:524-527`) — i.e. the agent **already** handles R server-side for the create direction and hands the result back exactly once. **Needs: one new agent local-API endpoint** (`POST /escrow/unseal`: blob + R in, the single `restic_repo_password` field out, R never logged, R never persisted) — the exact mirror of the claim endpoint. **Plus** a way for the blob to reach the agent: the agent's hub client has no escrow **GET** (only the `PUT` at `main.go:2937`), and `dr.go`'s re-enroll is the wrong tool (it rotates the host API key and needs operator-armed recovery mode). **Needs: a second small hub endpoint** — `GET /hosts/<id>/escrow`, per-host-key authed, serving opaque bytes to the box that owns them. Zero-knowledge is untouched: the hub still never decrypts, and the bytes are useless without R. **Cost: medium — three repos, two new endpoints, both narrow.** |
|
||||||
|
| **"show what will be recovered" before proceeding** | **the primitive EXISTS; the flow does not** | `restic snapshots --json` + `restic stats --mode raw-data --json` are already how the box counts snapshots and sizes the repo (`offboxRecordStats`, `offbox.go:1234-1265`). Both are **read-only against the repository** — nothing is written, no lock is taken, no `init`. So a preview can show: **snapshot count, per-snapshot time, host, tags (= app names), paths, and the deduplicated repo size** — all before committing. **Committing requires only that the recovered password be written to `offboxPwPath()`**, and even that is reversible in principle. **The clean shape:** unseal → hold the recovered password **in memory** → run the two read-only probes against the existing repo → render → and only on the customer's confirmation call `InjectOffboxPassword`. **Cost: small — one handler, one template, no new restic capability.** **Caveat, measured not assumed:** `offboxRecordStats` today also persists `RepoSizeBytes` into settings as a side effect; a preview path must not reuse it verbatim. |
|
||||||
|
| **proceeding** actually recovers | **NO end-to-end** | `InjectOffboxPassword` → next `WriteOffboxSecrets` keeps the file → `ensureOffboxRepo` probes and clears `RepoState`. Every piece is implemented; **none has ever run against a recovered password** (§8). **Cost: zero new code, but it is precisely what the drill in §10 has to prove.** |
|
||||||
|
|
||||||
|
### Things the ruling implies that do not exist, priced
|
||||||
|
|
||||||
|
| missing piece | cost | note |
|
||||||
|
|---|---|---|
|
||||||
|
| persisted ACK escrow status + fail-closed mint refusal | **S** | controller only; the `ClaimSync` pattern applies directly |
|
||||||
|
| the "recovery is available" card + dismissal | **S** | one template, one flag; the orphan card is the template precedent |
|
||||||
|
| a customer-facing **recovery-code** form (distinct from the password form) | **S** | the password form's handler exists but has no UI; both need building |
|
||||||
|
| agent local-API `POST /escrow/unseal` | **M** | new privileged surface; mirror of `/escrow/ceremony/claim` |
|
||||||
|
| hub `GET /hosts/<id>/escrow` (opaque, per-host key) | **S** | new read surface on the hub; zero-knowledge preserved |
|
||||||
|
| preview handler + template driven by read-only restic probes | **S** | primitive already exists |
|
||||||
|
| **carrying `identity_blob` into `host_escrow_superseded`** | **S** | one column, one `SELECT` list — **and without it the whole feature protects only boxes that have not yet re-escrowed** (→ R-198). This is a prerequisite, not a nice-to-have |
|
||||||
|
|
||||||
|
### The security question the ruling raises — stated, not answered
|
||||||
|
|
||||||
|
**What gates the surface today.** The controller dashboard is behind `RequireAuth`
|
||||||
|
(`internal/web/auth.go:52`): a bcrypt password (`effectivePasswordHash()` — customer-set in
|
||||||
|
`settings.json`, else operator-provisioned in `controller.yaml`), a 7-day session cookie, 5 failed
|
||||||
|
logins per minute then lockout, and `CsrfProtect` on every mutating route. The claim gate serves only
|
||||||
|
the claim page on an unclaimed box. So a recovery form would sit **behind the customer's dashboard
|
||||||
|
password**, on a controller reachable at whatever address that box's dashboard is reachable at.
|
||||||
|
|
||||||
|
**What a preview would expose to someone who guesses or steals R.** Not file contents — but the
|
||||||
|
snapshot list is not nothing: **backup dates and cadence, the app names (restic tags), the paths, and
|
||||||
|
the repository size.** That is a map of the household's digital life, and it is precisely the metadata
|
||||||
|
the zero-knowledge posture otherwise keeps out of everyone's hands including the operator's.
|
||||||
|
|
||||||
|
**Three specifics the operator should weigh, without this session choosing:**
|
||||||
|
|
||||||
|
1. **Guessing R is not the threat; holding the dashboard password is.** R is ten EFF words
|
||||||
|
(~129 bits, `00-capability-map.md` agent v0.93.0 note) and `age`'s scrypt is deliberately slow —
|
||||||
|
online guessing is not a route. But **anyone with the dashboard password can already restore live
|
||||||
|
offsite data** (`_recovery-inventory-2026-07-28.md` A.1.3–A.1.6). The marginal new exposure is
|
||||||
|
confined to the *orphaned/previous* repository, which today nobody can read at all.
|
||||||
|
2. **The form is an oracle.** A correct/incorrect R answer distinguishable in the UI turns the
|
||||||
|
dashboard into a verification service for a stolen recovery code. `UnwrapIdentity` already
|
||||||
|
fails closed with a single generic error (`identity.go:157`) — the UI must not do better than that,
|
||||||
|
and should rate-limit like the login path does.
|
||||||
|
3. **R will pass through the agent.** Whatever the form does, R crosses the local API to a
|
||||||
|
root-adjacent daemon. The ceremony already does this in the other direction, so the precedent is
|
||||||
|
set — but it means R is momentarily present in agent process memory on the Proxmox host, which is
|
||||||
|
the same host D6 says the operator has root on. **This is the same trade as spike option (c), in a
|
||||||
|
smaller and time-bounded form**, and it should be decided alongside it rather than separately.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Q8 — the drill design
|
||||||
|
|
||||||
|
**Why a deliberate cycle.** No recovery code exists for either demo box, and — per §7 — even one would
|
||||||
|
no longer open the old repositories. The proof therefore requires creating R, keeping it, and walking
|
||||||
|
the full circle. **This section is the design; the drill is not run in this session.**
|
||||||
|
|
||||||
|
### Target and preconditions
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| **Box** | **`demo-hp`** (HP t740, node `felhom-host`, host `demo-hp-bb76ea`). Tier 0 — disposable, and the designated drill host by the 2026-07-25 operator ruling. **Not demo-felhom** — it is the N100 that also carries the ISO/rehearsal lineage; leave one demo box in a known-good state throughout |
|
||||||
|
| **Guest** | the customer guest on that host (rebuild target) |
|
||||||
|
| **Data** | one deliberately identifiable app + a sentinel file whose content is a known string and whose sha256 is recorded before the wipe |
|
||||||
|
| **Prereq** | the box is healthy, offsite target configured and `escrow_state: escrowed`, and its **current** `host_escrow.identity_blob` seals the password of the repo that will be tested |
|
||||||
|
| **Duration** | ~3–4 h wall clock, of which ~45 min is operator-attended (steps 2, 5, 8, 9) |
|
||||||
|
| **Blocking dependency** | **steps 8–9 cannot be executed as designed today** — links 6, 8 and 9 of the chain do not exist as automation. The drill as written uses the manual substitutes (§4a) and **that is the point**: it measures the *existing* chain honestly before anything is built |
|
||||||
|
|
||||||
|
### The cycle
|
||||||
|
|
||||||
|
| # | Step | Who | Observable that must be recorded |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | Record the starting state: `host_escrow.restic_pw_sha256`, `identity_blob` length, `$.offsite.snapshot_count`, `repo_size_bytes` | CC | a hash and two counters, from the hub DB |
|
||||||
|
| 2 | **Run the escrow ceremony through the customer wizard and KEEP R** | **HUMAN** | R written to the operator's password manager under a named entry; **its storage location recorded in the drill report — the code itself never** |
|
||||||
|
| 3 | Deploy an app, write the sentinel file, record its sha256 | CC | sha256 + path |
|
||||||
|
| 4 | Run an off-site backup through the real UI endpoint; wait for completion | CC | `$.offsite.last_status = ok`, `snapshot_count` increased by ≥1, `last_success` stamped |
|
||||||
|
| 5 | **Verify the snapshot from outside**: `restic snapshots --json` against the repo | CC | snapshot id + tag + timestamp — the pre-wipe truth to compare against |
|
||||||
|
| 6 | **Wipe**: destroy the guest and reinstall it (the same procedure that produced the 2026-08-03 rebuild) | **HUMAN** decision, CC executes | new `controller_started`; `claimed` drops to 0; a new `encryption.key` mtime |
|
||||||
|
| 7 | Let the box come back and reach the mint. **Do NOT run a new ceremony** | CC | the controller log carries the `escrow-confirm` mismatch WARN (§6); `escrow_state: pending`; `host_escrow.identity_blob` **unchanged** (this is the assertion that keeps recovery possible) |
|
||||||
|
| 8 | **Recover**: export `identity_blob` from the hub, unseal with the kept R, extract `restic_repo_password` | **HUMAN** (R is theirs) | `identity-consume` exits 0; the bundle JSON contains a non-empty `restic_repo_password`; **the value is never printed or logged** |
|
||||||
|
| 9 | **Inject** the recovered password (`POST /backup/offbox/inject-password`, by hand — there is no form), re-apply, run an off-site backup | **HUMAN**-assisted | `restic cat config` succeeds; **`snapshot_count` ≥ the step-5 count + 1**, not 1 |
|
||||||
|
| 10 | **Restore the sentinel** through the customer restore flow and compare sha256 | CC | **byte-identical sha256 to step 3** |
|
||||||
|
| 11 | Record everything, tear down nothing that would hide the evidence | CC | drill report |
|
||||||
|
|
||||||
|
### The single observation that makes this a FAILURE rather than a pass
|
||||||
|
|
||||||
|
> **Step 10's sha256 does not match step 3's — or step 9 reports a snapshot count of 1.**
|
||||||
|
|
||||||
|
A snapshot count of 1 means a **new, empty history** was started over the old one: the repository did
|
||||||
|
not reopen, and every earlier snapshot is orphaned. **Explicitly: "the repository opened" is not the
|
||||||
|
pass condition.** `restic cat config` succeeding proves the password is right; it does not prove the
|
||||||
|
customer's data came back. Only the sha256 comparison does.
|
||||||
|
|
||||||
|
### Secondary failures worth recording separately (each is a finding, not a pass/fail)
|
||||||
|
|
||||||
|
- Step 7 finds `identity_blob` **changed** — something re-escrowed automatically → the recovery window
|
||||||
|
closes without a human act (would be new and serious).
|
||||||
|
- Step 8's bundle has an **empty** `restic_repo_password` — a pre-fork-4 or hash-less blob shape.
|
||||||
|
- Step 9's `ensureOffboxRepo` classifies `orphaned` despite the correct password → the classifier or
|
||||||
|
the transport, not the key.
|
||||||
|
|
||||||
|
### What could make it fail for uninteresting reasons
|
||||||
|
|
||||||
|
The `age` binary absent on the machine running step 8; the hub DB export producing a truncated blob
|
||||||
|
(use `writefile`, verify the byte length against `length(identity_blob)` = 572); the CSRF token in
|
||||||
|
step 9 (`felhom_session` is dropped by curl's cookie jar — project memory
|
||||||
|
`controller-api-headless-access`); and the R-71a settle gate deferring the apply in step 9 (wait for
|
||||||
|
its GO line rather than retrying).
|
||||||
|
|
||||||
|
### Rollback, and whether the loss is acceptable
|
||||||
|
|
||||||
|
**If the recovery does not work, what has been lost is:** demo-hp's guest, its app data since the last
|
||||||
|
whole-guest archive, and its off-site history. **On that box that is acceptable** — it is Tier 0,
|
||||||
|
explicitly disposable, and it is the box the target-selection runbook names for drills. **The
|
||||||
|
rollback** is a whole-guest restore from the local/PBS archive taken immediately before step 6, which
|
||||||
|
is a proven path on this fleet (restore-test passed on demo-hp's host tier in 109.3 s on 2026-08-04).
|
||||||
|
**Take that archive deliberately as part of step 6 and verify it exists before destroying anything** —
|
||||||
|
do not rely on the nightly.
|
||||||
|
|
||||||
|
**Not acceptable, and therefore not proposed:** running this on demo-felhom (leaves no healthy demo
|
||||||
|
box), on DooPlex (Tier 2, it *is* the recovery chain), or on any real customer's box.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Every query used
|
||||||
|
|
||||||
|
| # | Source | Query / read | Value read |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | git | `git -C <repo> rev-parse HEAD`; `git status --porcelain` | the three commits in §1; all empty |
|
||||||
|
| 2 | hub DB | `PRAGMA integrity_check` | `ok` |
|
||||||
|
| 3 | hub DB | `SELECT MAX(received_at) FROM host_reports; SELECT datetime('now')` | `2026-08-04 10:00:40` / `2026-08-04 10:03:48` |
|
||||||
|
| 4 | hub DB | `SELECT host_id, length(blob), length(identity_blob), substr(restic_pw_sha256,1,12), created_at, updated_at FROM host_escrow` | `demo-felhom-8363b5, 383, 572, c60c8bc737a6, 2026-08-04T07:20:07Z, 2026-08-04 07:20:08` · `demo-hp-bb76ea, 383, 572, 8a9e33aa4da6, 2026-08-04T07:15:36Z, 2026-08-04 07:15:36` |
|
||||||
|
| 5 | hub DB | `SELECT id, host_id, length(blob), substr(restic_pw_sha256,1,12), created_at, superseded_at FROM host_escrow_superseded` | `3, demo-hp-bb76ea, 383, 8e03eddf9ff7, 2026-07-23T10:01:17Z, 2026-08-04 07:15:36` · `4, demo-felhom-8363b5, 383, 48741892f0ef, 2026-07-21T08:38:33Z, 2026-08-04 07:20:08` |
|
||||||
|
| 6 | hub DB | `PRAGMA table_info(host_escrow)` / `(host_escrow_superseded)` | the two column lists in §7 — `identity_blob` present in the first, **absent from the second** |
|
||||||
|
| 7 | hub DB | latest report per customer: `json_extract(report_json,'$.claimed' / '$.offsite.escrow_state' / '.snapshot_count' / '.repo_state' / '.last_status' / '.last_run')` | demo-felhom `2026-08-04 09:49:42`: `1, escrowed, 0, NULL, NULL, NULL` · demo-hp `2026-08-04 09:56:57`: `1, escrowed, 0, NULL, NULL, NULL` |
|
||||||
|
| 8 | hub DB | `claimed` grouped over reports since `2026-08-03 05:00` | demo-hp: `1` ×124, **`0` ×5 (06:12:19→06:21:03)** · demo-felhom: `1` ×120, **`0` ×4 (07:17:54→07:19:46)** |
|
||||||
|
| 9 | hub DB | `SELECT customer_id, event_type, severity, created_at FROM events WHERE created_at > '2026-08-03 00:00'` | no `escrow_stale`, no `offsite_reissued`, no `offbox_repo_orphaned`, no claim event in the window; `controller_started` ×4 demo-hp / ×4 demo-felhom around the rebuilds |
|
||||||
|
| 10 | guest 9201 (I3) | `ls -la --time-style=full-iso …/felhom-controller-data/_data/data{,/offbox}` | the mtime table in §5 |
|
||||||
|
| 11 | source (I4) | `grep -rn "InjectOffboxPassword" --include=*.go .` | 1 definition, 1 handler caller, 5 test uses, 1 comment — **no production caller besides the handler** |
|
||||||
|
| 12 | source (I4) | `grep -rn "inject" internal/web/templates/` | one unrelated hit (an XSS comment) — **no form** |
|
||||||
|
| 13 | source (I4) | `grep -rni "reenroll\|re_enroll" --include=*.go .` in `felhom-agent` | **zero hits** |
|
||||||
|
| 14 | source (I4) | `grep -rn "re-enroll\|recovery-mode\|restore-directive" --include=*.sh --include=*.md --include=*.py` workspace-wide | only design docs + project memory; no script, no runbook step |
|
||||||
|
| 15 | source (I4) | `grep -rn "ListSupersededEscrow"` in `felhom.eu` | 2 test callers, 0 production |
|
||||||
|
| 16 | source (I4) | `grep -rn "\.Escrow\b" --include=*.go .` in `felhom-controller` | one hit: `cmd/controller/main.go:641` |
|
||||||
|
| 17 | source | `felhom-agent/CHANGELOG.md` | `v0.77.0 — fork-4: escrow the offsite restic repo password under R (2026-07-09)` |
|
||||||
|
| 18 | source | `controller/Dockerfile:36-70` | runtime package list — **no `age`** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. What could not be established
|
||||||
|
|
||||||
|
Named, not guessed.
|
||||||
|
|
||||||
|
1. **The 2026-08-05 ~02:15 UTC scheduled run's outcome.** Still unmeasured — both boxes became
|
||||||
|
runnable only on 2026-08-04 morning and neither has run since (query 7: `last_run` absent on both).
|
||||||
|
Yesterday's spike predicts `ErrOffboxOrphaned`; that prediction is **unchanged and still unproven**.
|
||||||
|
2. **Whether demo-hp's data dir was fresh.** Established for demo-felhom by direct mtime measurement
|
||||||
|
(I3). demo-hp was not measured — it has no baked SSH key and reaching it needs the break-glass
|
||||||
|
credential, which this read-only session did not use. The hub-side `claimed 1→0→1` signature is
|
||||||
|
identical on both boxes, which is suggestive, not decisive.
|
||||||
|
3. **Whether the orphaned ciphertext still exists at the provider.** Unchanged from yesterday: reading
|
||||||
|
the endpoint needs the SFTP credential, which was not touched.
|
||||||
|
4. **Whether any pre-2026-07-21 escrow generation ever existed for demo-felhom.** The superseded table
|
||||||
|
holds two rows in the entire database; earlier generations are not retained (and, per §7, would not
|
||||||
|
have carried the identity blob anyway).
|
||||||
|
5. **Whether `age` is present on any machine that would run step 8 of the drill.** Not checked — the
|
||||||
|
agent declares `/usr/bin/age` as a runtime dependency but this session did not verify it on either
|
||||||
|
demo host.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. Observations — noticed, not acted on
|
||||||
|
|
||||||
|
1. **A fresh `encryption.key` on every rebuild** (I3: 32 B, mtime `2026-08-03 07:17:26`) means every
|
||||||
|
pre-rebuild `app.yaml` `ENC:` secret on that box is undecryptable. This is exactly circular
|
||||||
|
dependency **C-1** in `_recovery-inventory-2026-07-28.md`, and it fired live on both demo boxes this
|
||||||
|
week without anyone naming it. It is a strictly larger blast radius than the offsite repo and it is
|
||||||
|
**not** covered by R-193. Worth its own item if the operator agrees it is not already implied by C-1.
|
||||||
|
2. **`Claimed` returning to `1` within minutes** of a rebuild, with no event of any kind recording it,
|
||||||
|
means "the customer claimed this box" has no audit trail. There is no `customer_claimed` event type.
|
||||||
|
3. **`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting** — harmless today,
|
||||||
|
a trap for anyone reusing it as a read-only preview primitive (§9).
|
||||||
|
4. **`handleReEnroll` returns the new API key in the response body** (`dr.go:148`) while also logging
|
||||||
|
the rotation. Not a defect (the box needs it), but it is a secret in an HTTP response on an endpoint
|
||||||
|
nothing currently calls — worth a second look before anything starts calling it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14. The operator's decisions — stated, and NOT answered here
|
||||||
|
|
||||||
|
1. **Run the drill (§10) or not?** It is the only thing that converts *"the pieces look right"* into
|
||||||
|
*"a customer got their data back"*. It costs demo-hp for an afternoon and, if it fails, that box's
|
||||||
|
history. **Recommendation: yes — but fix `identity_blob` retention (R-198) first**, because a drill
|
||||||
|
that walks the current chain will pass or fail on a chain that is missing a link everyone believed
|
||||||
|
was there.
|
||||||
|
2. **The orphaned snapshots at the endpoint** (40.9 MB + 1.14 GB, presumed present, §12.3). They are
|
||||||
|
**unrecoverable by two independent mechanisms** now: no recovery code was kept, *and* the sealed
|
||||||
|
passwords were destroyed by the 2026-08-04 re-ceremonies. They consume the 50 GB soft quota
|
||||||
|
indefinitely; nothing prunes a moved-aside restic repo. **Delete, or leave?** This session did not
|
||||||
|
act on it and states it as the operator's.
|
||||||
|
3. **The unchanged R-193 question** — agent-retained repo password (spike option c) versus accepting
|
||||||
|
that every rebuild starts a new repository. **This session adds one input to that decision:** the
|
||||||
|
"accept it, provided the system says so loudly" branch currently rests on a retention feature that
|
||||||
|
does not retain the relevant secret. Under option (d)-alone as it stands today, *"the old history
|
||||||
|
stays recoverable with your recovery code"* is not true. Either R-198 is fixed, or that branch has
|
||||||
|
to be re-described honestly to the customer.
|
||||||
|
</content>
|
||||||
|
</invoke>
|
||||||
@@ -0,0 +1,826 @@
|
|||||||
|
# SPIKE — DooPlex build-cache containment (cap at 30 GB, relocate to SSD2)
|
||||||
|
|
||||||
|
**Date:** 2026-08-05 · **Host:** DooPlex (192.168.0.180) — **Tier 2, protected**, not a drill target
|
||||||
|
**Class:** Spike. §1, §2, §3, §4, §13-verify, §15 apply; §5–§12 skipped (no production Go code written).
|
||||||
|
**Anchor:** `dooplex_report.md` (2026-08-05 read-only disk audit). Every number in it was treated as a
|
||||||
|
claim to re-measure, not a fact to carry forward.
|
||||||
|
|
||||||
|
> Unit convention: **GB = decimal (10⁹)**, because `df -B1`, `du -B1` and Prometheus emit raw bytes.
|
||||||
|
> `docker buildx inspect` renders **GiB** (2³⁰) — where that matters it is labelled.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verdict
|
||||||
|
|
||||||
|
**The mechanism is confirmed, with one correction and one refutation, and the follow-up is safe to spec.**
|
||||||
|
|
||||||
|
- **Confirmed:** a `builder.gc` policy in `/etc/docker/daemon.json` **is** honoured under the containerd
|
||||||
|
worker, and it **does** evict. Proven by naming a 440 MB cache record present at one build and absent
|
||||||
|
two builds later — not by the absence of an error.
|
||||||
|
- **Corrected:** it is honoured **only in the `policy` array form**. The flat form
|
||||||
|
(`{"gc": {"reservedSpace": …}}`) is **silently ignored** — the daemon starts, logs nothing, and keeps
|
||||||
|
its built-in defaults. `dockerd --validate` is worthless here: it returned `configuration OK` for a
|
||||||
|
bogus key **and** for a config that then crashed the daemon.
|
||||||
|
- **Refuted:** the audit's proposal to change Docker's `data-root` would **not** move the build cache. It
|
||||||
|
moves 0.62 GB. The 181.4 GB belongs to the **system containerd**, relocated via `root` in
|
||||||
|
`/etc/containerd/config.toml`.
|
||||||
|
- **The prune is done and it was the whole rescue.** `/` went **86% → 53% used**, and Longhorn's default
|
||||||
|
disk went **`Schedulable=False (DiskPressure)` → `Schedulable=True`**. The live consequence is cleared.
|
||||||
|
- **The largest single finding is not the cap.** 90.5 GB of the 157 GB was one layer re-downloading on
|
||||||
|
every build, and the root cause is a **one-line Dockerfile ordering defect** — `ARG VERSION` declared
|
||||||
|
above the module-download step. Fixing it removes the source; the cap only bounds the symptom.
|
||||||
|
- **P6 (the SSD2 move) was NOT executed** — it stops at the operator, as specified. The pre-analysis
|
||||||
|
says the move is safe as measured but carries a real Longhorn risk, and the prune removed its urgency.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Confirmed baselines
|
||||||
|
|
||||||
|
| Repo | `main` @ commit at spike start | Drift vs the task's stated baseline |
|
||||||
|
|---|---|---|
|
||||||
|
| `homelab-manifests` | `5188058d0e43` | none — matched |
|
||||||
|
| `felhom.eu` | `0dbd954fec90` | none — matched |
|
||||||
|
|
||||||
|
Both trees clean (`git status --porcelain` empty) and `HEAD == origin/main` before any commit.
|
||||||
|
|
||||||
|
**Register:** highest allocated at baseline was **R-204**, re-confirmed in `OPEN-ITEMS.md`. **R-205**
|
||||||
|
onward were free and are allocated here (R-205 … R-211).
|
||||||
|
|
||||||
|
**One path correction:** `homelab-manifests` is **not** in `/mnt/5_hdd/felhom.eu/git/` with the other
|
||||||
|
repos — it lives at **`/home/kisfenyo/git/homelab-manifests`**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. P0 — arrival state, expected vs measured
|
||||||
|
|
||||||
|
Measured live before anything was changed. **Values that matched are recorded, not omitted.**
|
||||||
|
|
||||||
|
| Fact | Expected (audit) | **Measured** | |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Root fs size | 477,393,182,720 B | **477,393,182,720 B** | match |
|
||||||
|
| Root fs used | 387.3 GB | **387,332,755,456 B** | match (+10.8 MB in 2 h) |
|
||||||
|
| Root fs avail | 65.7 GB | **65,734,934,528 B** | match |
|
||||||
|
| Root fs percent | 86% | **86%** | match |
|
||||||
|
| Docker server version | 29.6.2 | **29.6.2** (build `3d80467`) | match |
|
||||||
|
| Storage driver | overlayfs, containerd snapshotter | **`overlayfs`, `driver-type: io.containerd.snapshotter.v1`** | match |
|
||||||
|
| Docker Root Dir | `/var/lib/docker`, ~0.6 GB | **`/var/lib/docker`, 618,541,056 B** | match |
|
||||||
|
| containerd root | `/var/lib/containerd`, ~181.4 GB | **181,394,223,104 B** | match |
|
||||||
|
| Build cache total / reclaimable | 157 GB / 147.8 GB | **157 GB / 147.8 GB**, 1446 records | match |
|
||||||
|
| Images / containers | 345 images, 2 running | **345 unique** (350 tag lines), **2 running** | match |
|
||||||
|
| `/etc/docker/daemon.json` | does not exist | **does not exist** (`/etc/docker/` exists, empty) | match |
|
||||||
|
| `/etc/buildkit/` | does not exist | **does not exist** | match |
|
||||||
|
| SSD2 device / mount / fs / free | ~240 GB, ~203 GB free | **`/dev/sde2`, `/mnt/ssd_2`, ext4, 235,148,750,848 B fs, 203,034,353,664 B free** | match |
|
||||||
|
| SSD1 Longhorn | `Schedulable=False (DiskPressure)` | **`Schedulable=False`, `DiskPressure`, 18.85% avail** | match |
|
||||||
|
| HDD2 Longhorn | also False | **`Schedulable=False`, `DiskPressure`, 24.91% avail** | match |
|
||||||
|
| Housekeeping timer | active, last Sun 2026-08-02 | **active**, last **2026-08-02 05:35:10 CEST**, next 08-09, `ExecMainStatus=0` | match |
|
||||||
|
|
||||||
|
**Prohibition honoured:** `node-housekeeping.sh` was **not run**, in any mode. Its state was read from
|
||||||
|
systemd, from the textfile on disk, and from Prometheus. The textfile's mtime is still
|
||||||
|
`2026-08-02 05:35:11` — the 8-week reclaim history is intact.
|
||||||
|
|
||||||
|
**What the audit did not measure — the SSD2 Longhorn record** (`Crucial-SSD-240G`,
|
||||||
|
`/mnt/ssd_2/data/longhorn`):
|
||||||
|
|
||||||
|
| field | value |
|
||||||
|
|---|---:|
|
||||||
|
| `storageMaximum` | 235,148,750,848 |
|
||||||
|
| `storageAvailable` | 214,958,080,000 (**91.41%**) |
|
||||||
|
| `storageScheduled` | 139,586,437,120 |
|
||||||
|
| `storageReserved` | **0** |
|
||||||
|
| condition | **`Schedulable=True`** |
|
||||||
|
| replicas | **12** |
|
||||||
|
|
||||||
|
**Two things stand out and both matter for P6.** SSD2 is the **only** Longhorn disk reserving **zero**
|
||||||
|
(HDD2 and HDD4 each reserve 500 GB). And `storageScheduled` (139.6 GB) exceeds actual usage
|
||||||
|
(`df`: 20,094,939,136 B) by **6.9×** — Longhorn is heavily overcommitted on this disk.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. P1 — which daemon owns the bytes
|
||||||
|
|
||||||
|
> **The single configuration change that relocates the build cache is `root = "<path>"` in
|
||||||
|
> `/etc/containerd/config.toml`, owned by `containerd.service` — the SYSTEM containerd
|
||||||
|
> (PID 2698677, socket `/run/containerd/containerd.sock`).**
|
||||||
|
|
||||||
|
**The audit's `data-root` proposal is refuted, and the split is measured:**
|
||||||
|
|
||||||
|
| tree | size | moved by `data-root`? |
|
||||||
|
|---|---:|---|
|
||||||
|
| `/var/lib/docker` | **618,541,056 B** (volumes 321 MB, **buildkit metadata 282 MB**, containers 14 MB) | yes |
|
||||||
|
| `/var/lib/containerd` | **181,394,223,104 B** (overlayfs snapshots **170,357,870,592 B**, content 10.98 GB) | **no** |
|
||||||
|
|
||||||
|
`dockerd` runs as `/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock` with **no
|
||||||
|
systemd drop-ins**. BuildKit keeps its *record database* under Docker Root Dir (0.28 GB) but its *layer
|
||||||
|
bytes* in containerd's snapshotter — which is why `data-root` looks plausible and is wrong.
|
||||||
|
|
||||||
|
In `/etc/containerd/config.toml` the `root` key is **present but commented out**, so it defaults to
|
||||||
|
`/var/lib/containerd`. Uncommenting it is the whole change.
|
||||||
|
|
||||||
|
**k3s is unaffected — confirmed, not assumed.** k3s runs a **separate** containerd instance with its own
|
||||||
|
socket (`/run/k3s/containerd/containerd.sock`) and its own root
|
||||||
|
(`/var/lib/rancher/k3s/agent/containerd`, **79,669,473,280 B**). Nothing in this spike touches it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. P2 — which GC key schema this release accepts
|
||||||
|
|
||||||
|
**`dockerd --validate` exists on 29.6.2 and is not an oracle for this subtree.** Measured:
|
||||||
|
|
||||||
|
| candidate | `--validate` | reality |
|
||||||
|
|---|---|---|
|
||||||
|
| `defaultKeepStorage` (old flat form) | `configuration OK` | — |
|
||||||
|
| `policy` array | `configuration OK` | **honoured** |
|
||||||
|
| `reservedSpace` / `maxUsedSpace` / `minFreeSpace` flat | `configuration OK` | **silently ignored** |
|
||||||
|
| **`totallyBogusKey`** | **`configuration OK`** | proves the validator does no field checking |
|
||||||
|
| `policy` with `"filter": [three values]` | **`configuration OK`** | **crashed the daemon on restart** |
|
||||||
|
|
||||||
|
> **The real oracle is `docker buildx inspect`, which renders the *effective* policy.** That is an
|
||||||
|
> assertion of effect; `--validate` and "no error in the log" are both assertions of absence, and both
|
||||||
|
> were measured to lie here.
|
||||||
|
|
||||||
|
**The flat form is silently ignored.** With `{"gc":{"enabled":true,"reservedSpace":"1GB","maxUsedSpace":"2GB","minFreeSpace":"10GB"}}`
|
||||||
|
applied and the daemon restarted, `buildx inspect` still reported the built-in defaults, and
|
||||||
|
`journalctl -u docker.service` carried **no warning, no deprecation notice and no error**.
|
||||||
|
|
||||||
|
**The `filter` key takes ONE value per policy entry, not an array.** An array is fatal at startup:
|
||||||
|
|
||||||
|
```
|
||||||
|
error initializing buildkit: error creating buildkit instance: filters expect only one value
|
||||||
|
```
|
||||||
|
|
||||||
|
The daemon then entered a systemd restart loop. **This is also how the two `unless-stopped` dev
|
||||||
|
containers ended up stopped** — and they did **not** come back when the daemon later started cleanly;
|
||||||
|
they needed a manual `docker start`. A follow-up that templates this file must **restart and verify**,
|
||||||
|
never validate-and-assume.
|
||||||
|
|
||||||
|
**Documented statement on containerd-worker applicability:** none found in `dockerd --help` or the
|
||||||
|
installed docs. Recorded as **unknown from documentation**, and settled empirically by P5.
|
||||||
|
|
||||||
|
**The default policy is not "no policy".** Before any change, `buildx inspect` reported built-in rules
|
||||||
|
with **Reserved 41.91 GiB / Max Used 331.6 GiB / Min Free 82.89 GiB**. The audit's "there is no BuildKit
|
||||||
|
GC policy on this host" is *almost* right — there was one, sized so large it could never bind: the cache
|
||||||
|
at 157 GB was less than half the 331.6 GiB ceiling.
|
||||||
|
|
||||||
|
### The validated `daemon.json`, verbatim (final state, left in place deliberately)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"builder": {
|
||||||
|
"gc": {
|
||||||
|
"enabled": true,
|
||||||
|
"policy": [
|
||||||
|
{ "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" },
|
||||||
|
{ "all": true, "reservedSpace": "10GB", "maxUsedSpace": "30GB", "minFreeSpace": "20GB" }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Effective policy confirmed live (note the **GiB** rendering — `30GB` in JSON becomes a 30 GiB rule):
|
||||||
|
|
||||||
|
```
|
||||||
|
GC Policy rule#0: All: false Reserved Space: 10GiB Max Used Space: 30GiB Min Free Space: 20GiB
|
||||||
|
GC Policy rule#1: All: true Reserved Space: 10GiB Max Used Space: 30GiB Min Free Space: 20GiB
|
||||||
|
```
|
||||||
|
|
||||||
|
**No deprecation warning was emitted** for any accepted form.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. P3 — the prune: claimed, actual, settle
|
||||||
|
|
||||||
|
Pre-checked: no build in progress, P0 and P7 captured first.
|
||||||
|
|
||||||
|
| | value |
|
||||||
|
|---|---:|
|
||||||
|
| **Claimed by `docker builder prune -af`** | **156.9 GB** (1444 records deleted, 86 s wall-clock) |
|
||||||
|
| **Actual filesystem reclaim** | **150,353,829,888 B (150.35 GB)** |
|
||||||
|
| **Time to settle** | **synchronous — full reclaim visible at t+0** |
|
||||||
|
|
||||||
|
Settle poll (t+0 already showed +150,354,477,056 B; two consecutive readings <50 MB apart at t+60s;
|
||||||
|
the residual −0.4 MB and −0.3 MB steps are ordinary background writes, not lagging reclaim):
|
||||||
|
|
||||||
|
| t | avail | step delta |
|
||||||
|
|---|---:|---:|
|
||||||
|
| 0 s | 215,947,522,048 | — |
|
||||||
|
| 30 s | 215,947,141,120 | −380,928 |
|
||||||
|
| 60 s | 215,946,874,880 | −266,240 |
|
||||||
|
|
||||||
|
**The 6.5 GB gap between claimed and actual is not an error** — it is the layers still shared with
|
||||||
|
retained images (`Shared: 8.6 GB` before the prune). **A weekly job must measure the filesystem, not
|
||||||
|
trust the command's `Total:`.**
|
||||||
|
|
||||||
|
**Unlike containerd's image GC, this returns bytes synchronously**, so the follow-up needs no
|
||||||
|
`settle_imagefs` equivalent — but the settle poll is what *established* that, and it cost 60 seconds.
|
||||||
|
|
||||||
|
| | before | after |
|
||||||
|
|---|---:|---:|
|
||||||
|
| `/` used | 387,474,644,992 B (**86%**) | 240,301,133,824 B (**53%**) |
|
||||||
|
| `/` avail | 65,593,044,992 B | 215,946,874,880 B |
|
||||||
|
| `/var/lib/containerd` | 181,537,714,176 B | 31,160,053,760 B |
|
||||||
|
| build cache | 156.9 GB / 1443 records | **0 B / 0 records** |
|
||||||
|
| images | 345 | **345** (untouched) |
|
||||||
|
| dev containers | 2 healthy | **2 healthy** |
|
||||||
|
|
||||||
|
### The live consequence, cleared
|
||||||
|
|
||||||
|
Longhorn re-evaluated within 60 s:
|
||||||
|
|
||||||
|
| disk | before | after |
|
||||||
|
|---|---|---|
|
||||||
|
| `default-disk-…` (SSD1, `/var/lib/longhorn/`) | `Schedulable=False (DiskPressure)`, 89,967,820,800 B, **18.85%** | **`Schedulable=True`**, 240,228,761,600 B, **50.32%** |
|
||||||
|
| `HDD2` | `Schedulable=False (DiskPressure)`, 24.91% | **unchanged — still False, 24.91%** |
|
||||||
|
|
||||||
|
**HDD2 is untouched by this work and remains unschedulable.** It is a separate item, stated here so it
|
||||||
|
is not read as fixed by association.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. P5 — proving the ceiling is enforced
|
||||||
|
|
||||||
|
**Rollback stated before the first mutation** and never needed beyond the recovery below:
|
||||||
|
`sudo rm /etc/docker/daemon.json && sudo systemctl restart docker` — the file did not exist, so removal
|
||||||
|
is a complete revert.
|
||||||
|
|
||||||
|
Test ceiling **2 GB** (below one build's working set, deliberately, so eviction is reachable).
|
||||||
|
|
||||||
|
### Build-by-build totals
|
||||||
|
|
||||||
|
| step | build cache total | reclaimable | records |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| baseline (post-prune) | 0 B | 0 B | 0 |
|
||||||
|
| controller 1 | 2.583 GB | 2.583 GB | 22 |
|
||||||
|
| hub 1 | **3.703 GB** ← peak | 3.703 GB | 37 |
|
||||||
|
| controller 2 | 3.525 GB | 3.525 GB | 37 |
|
||||||
|
| hub 2 | 3.051 GB | 3.051 GB | 38 |
|
||||||
|
| controller 3 | 2.817 GB | 2.817 GB | 36 |
|
||||||
|
| hub 3 | **2.782 GB** | 2.782 GB | 38 |
|
||||||
|
|
||||||
|
**The total stops rising and trends down** — the cap binds. Records plateau at 36–38.
|
||||||
|
|
||||||
|
### The named eviction — the assertion that matters
|
||||||
|
|
||||||
|
Set-differencing the record IDs, **4 records present at controller-build-1 were absent by
|
||||||
|
controller-build-3**, including the exact class that accumulated 208 copies:
|
||||||
|
|
||||||
|
| ID | size | created | description |
|
||||||
|
|---|---:|---|---|
|
||||||
|
| **`9ak8egnxrg6rkibic2zo44u1`** | **440 MB** | 07:28:34 | **`mount / from exec /bin/sh -c go mod download \|\| true`** |
|
||||||
|
| `kkic530irq4em04kywzwrsjp` | 214.6 MB | 07:28:37 | `… go build -ldflags …` |
|
||||||
|
| `yvityhta4wppwa8cxs525c3g` | 58.83 MB | 07:28:36 | `… go mod tidy` |
|
||||||
|
| `mvttjfrimnt1ttt0ogry75ue` | 49.8 MB | 07:28:35 | `[builder 5/7] COPY . .` |
|
||||||
|
|
||||||
|
Across the whole sequence: `h1→h3` evicted **17** records, `c2→c3` evicted **10**.
|
||||||
|
|
||||||
|
### Tolerance, stated honestly
|
||||||
|
|
||||||
|
Steady state settled at **2.78–2.82 GB against a 2 GiB (2.147 GB) rule — about 30% over**, with a peak
|
||||||
|
of **3.70 GB (~72% over)** mid-sequence.
|
||||||
|
|
||||||
|
**This overshoot is an artefact of the test, not a property of the cap, and should not be extrapolated to
|
||||||
|
30 GB.** A single controller build produces ~2.6 GB of cache in one shot, so a 2 GB ceiling was set
|
||||||
|
*below one build's working set* — GC cannot get under it without deleting the cache it just created. At
|
||||||
|
30 GB the same working set is ~9% of the ceiling and GC has ample room.
|
||||||
|
|
||||||
|
### GC is build-triggered, not timer-driven
|
||||||
|
|
||||||
|
Six minutes idle, polled each minute: **total unchanged at 2.782 GB, records unchanged at 38.** The
|
||||||
|
cache does not shrink on its own. A burst of builds can overshoot and *stay* overshot until the next
|
||||||
|
build — which is the argument for keeping a weekly `docker builder prune` as a backstop (R-206), rather
|
||||||
|
than relying on the cap alone.
|
||||||
|
|
||||||
|
### Then the ruled 30 GB
|
||||||
|
|
||||||
|
Applied, restarted, and confirmed by the effective policy (§4). Both dev containers came back healthy
|
||||||
|
and **their data is intact — asserted positively, not by absence of complaint**:
|
||||||
|
|
||||||
|
| observable | before | after |
|
||||||
|
|---|---:|---:|
|
||||||
|
| postgres `count(*) from pg_database` | 4 | **4** |
|
||||||
|
| postgres `pg_database_size('jarr_dev')` | 175,135,767 | **175,135,767** |
|
||||||
|
| postgres public tables | 31 | **31** |
|
||||||
|
| redis `DBSIZE` | 2437 | **2437** |
|
||||||
|
| docker volumes | 8 | **8, identical names** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. P7 — why the module-download layer re-runs
|
||||||
|
|
||||||
|
**Root cause found and proven: `ARG VERSION` / `ARG GIT_COMMIT` are declared in the builder stage
|
||||||
|
*above* the module-download step, so a new version value invalidates every `RUN` beneath them.**
|
||||||
|
|
||||||
|
### The evidence chain
|
||||||
|
|
||||||
|
**1. Every record was used exactly once.** All **208** `go mod download` records (206 at audit time, +2
|
||||||
|
from this spike's builds) carried **`Usage count: 1`**, spanning 2026-07-06 → 2026-08-05, totalling
|
||||||
|
91.36 GB at 438–440 MB each. Not one was ever reused.
|
||||||
|
|
||||||
|
**2. Two builds through `build.sh`, byte-identical tree, both missed.**
|
||||||
|
|
||||||
|
| step | build 1 | build 2 |
|
||||||
|
|---|---|---|
|
||||||
|
| `#8 WORKDIR /build` | CACHED | CACHED |
|
||||||
|
| `#9 COPY go.mod ./` | **CACHED** | **CACHED** |
|
||||||
|
| **`#10 RUN go mod download \|\| true`** | **executed** | **executed** |
|
||||||
|
| `#11 COPY . .` | executed | executed |
|
||||||
|
| `#12 RUN go mod tidy` | executed | executed |
|
||||||
|
| `#13 RUN … go build …` | executed | executed |
|
||||||
|
| `#14/#15/#16` (stage-1 `RUN`s) | CACHED | CACHED |
|
||||||
|
|
||||||
|
`go.mod` was **byte-identical** (`d8ded3f0798c…`) in the repo and in the workspace, mtime unchanged
|
||||||
|
(`2026-07-24 11:54:08`).
|
||||||
|
|
||||||
|
**3. `build.sh`'s churn was ruled out by fingerprinting.** Content-hashing all 737 files across
|
||||||
|
`rm -rf` + `cp -a` + host-side `go mod tidy`: **`tidy` changes nothing**, and the reproduced tree is
|
||||||
|
**content-identical** to the live workspace. `go.mod`'s inode number was even identical across the churn.
|
||||||
|
|
||||||
|
**4. A direct `docker build` on the untouched workspace cached *everything*** — including
|
||||||
|
`RUN go mod download` — proving BuildKit's cache is healthy and the Dockerfile is cacheable.
|
||||||
|
|
||||||
|
**5. The decisive isolation.** Same workspace, same command, only the build-arg value varied:
|
||||||
|
|
||||||
|
| run | `--build-arg VERSION` | `COPY go.mod ./` | **`RUN go mod download`** |
|
||||||
|
|---|---|---|---|
|
||||||
|
| E1 | **new** (`spike-p7e`) | CACHED | **executed** |
|
||||||
|
| E2 | **same** (`spike-p7e`) | CACHED | **CACHED** |
|
||||||
|
|
||||||
|
**The mechanism, and it explains the `#9 CACHED / #10 executed` signature exactly:** a `COPY`'s cache key
|
||||||
|
is content-based and carries no environment, so it hits; a `RUN`'s key includes the stage **environment**,
|
||||||
|
which contains every `ARG` declared above it. Since every real build passes a fresh `VERSION` (and often
|
||||||
|
a fresh `GIT_COMMIT`), the module-download layer is invalidated **on every build, without exception** —
|
||||||
|
206 builds, 206 retained 440 MB snapshots, 90.5 GB.
|
||||||
|
|
||||||
|
**Both Dockerfiles have it.** `felhom.eu/hub/Dockerfile` declares `ARG VERSION` / `ARG BUILD_TIME` above
|
||||||
|
`COPY go.mod go.sum*` → `RUN go mod download`. **And because both produce byte-identical `buildx du`
|
||||||
|
description strings, the 208 records are a COMBINED count** and must not be attributed to one project.
|
||||||
|
|
||||||
|
**Fix shape — not applied here, filed as R-208:** move the `ARG` declarations down to just above the
|
||||||
|
final `go build`. One line per Dockerfile. **This is worth more than the cap and the move combined:** the
|
||||||
|
cap bounds the symptom at 30 GB; this removes the source.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. P4 — the image store, enumerated (nothing deleted)
|
||||||
|
|
||||||
|
345 unique images across 350 tag lines (5 double-tagged), **0 dangling**. Only two are held by running
|
||||||
|
containers: `redis:7-alpine` and `postgres:16-alpine`. `docker image prune -a` would remove **343**.
|
||||||
|
|
||||||
|
**Size the prize before spending a decision on it.** Per-tag sizes sum to 139.29 GB, but that
|
||||||
|
double-counts shared layers. `docker system df` puts the **real dedup'd footprint at 31.02 GB, of which
|
||||||
|
27.02 GB is reclaimable** — roughly a fifth of what the build-cache prune already returned.
|
||||||
|
|
||||||
|
| GB (per-tag sum) | tags | repository |
|
||||||
|
|---:|---:|---|
|
||||||
|
| 118.44 | **207** | `gitea.dooplex.hu/admin/felhom-controller` |
|
||||||
|
| 7.25 | **107** | `gitea.dooplex.hu/admin/felhom-hub` |
|
||||||
|
| 4.34 | 5 | `gitea.dooplex.hu/admin/jarr` |
|
||||||
|
| 1.69 | 1 | `ghcr.io/seanmorley15/adventurelog-backend` |
|
||||||
|
| 0.94 | 1 | `ghcr.io/papra-hq/papra` |
|
||||||
|
| 0.72 | 1 | `codewithcj/sparkyfitness_server` |
|
||||||
|
| 0.72 | 1 | `postgis/postgis` |
|
||||||
|
| 0.59 | 3 | `gitea.dooplex.hu/admin/sparkyfitness-export` |
|
||||||
|
| (23 more, each ≤0.53 GB) | | |
|
||||||
|
|
||||||
|
### Registry presence — which are recoverable
|
||||||
|
|
||||||
|
Queried against the live Gitea registry v2 API:
|
||||||
|
|
||||||
|
| repo | tags in registry | tags local | **local-only (NOT re-pullable)** |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| `felhom-controller` | 76 | 207 | **131** |
|
||||||
|
| `felhom-hub` | 45 | 107 | **62** |
|
||||||
|
|
||||||
|
**The local-only tags are all old.** Controller `0.39.0`–`0.135.0` plus `v0.35.0`–`v0.39.0`; hub
|
||||||
|
`0.9.0`–`0.57.0` plus `v0.7.2`–`v0.13.0`. **Everything from controller `0.136.0` and hub `0.58.0`
|
||||||
|
upward is in the registry** and therefore recoverable by `docker pull`.
|
||||||
|
|
||||||
|
**This is the list the operator still owes a ruling on (R-210).** CC's view: not worth doing for the
|
||||||
|
space — ~27 GB against 199 GB now free. The only real benefit is dropping unrecoverable clutter.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. P8 — what else writes to Docker
|
||||||
|
|
||||||
|
**The bounded list of build triggers on the host Docker daemon is: interactive/manual runs only.**
|
||||||
|
|
||||||
|
- **18 systemd timers** enumerated — none invokes docker. (`node-housekeeping`, `dooplex-backup`,
|
||||||
|
`apt-daily*`, `sysstat*`, `logrotate`, `fstrim`, `e2scrub_all`, `smart-fancontrol`, …)
|
||||||
|
- **Root crontab: empty** (comments only). **User crontab: one entry**, `sync-hetzner-backups.sh`,
|
||||||
|
writes under `/mnt/5_hdd`.
|
||||||
|
- **`/etc/cron.d`: 4 real files** — anacron, e2scrub_all, sysstat, zfsutils. None touches docker.
|
||||||
|
|
||||||
|
**The hole the audit left open is now closed with positive evidence.** The audit concluded "no CI runner
|
||||||
|
exists on this host" from systemd/process/crontab alone — but a Gitea Actions runner **does** exist as a
|
||||||
|
**cluster pod** (R-168, 2026-08-02). It cannot reach the host daemon:
|
||||||
|
|
||||||
|
- **162 pods swept across all namespaces: `0` mount any `docker`/`containerd` hostPath.**
|
||||||
|
- `act-runner` (`gitea-system`, image `felhom-act-runner:0.1.0`) has one volume — a PVC at `/data`.
|
||||||
|
`privileged: false`, `allowPrivilegeEscalation: false`, **all capabilities dropped**, no docker socket.
|
||||||
|
- Its Dockerfile states the absence is deliberate: *"DELIBERATELY NOT INSTALLED: a Go toolchain, Node, or
|
||||||
|
Docker … There is no container runtime in here on purpose."*
|
||||||
|
- `gates.yml` runs exactly one command — `python3 scripts/<repo>_gates.py --fast` — plus a git fetch and
|
||||||
|
a failure-only Resend alarm. **No `docker build` anywhere.**
|
||||||
|
|
||||||
|
**So the audit's ~7 builds/day attribution to interactive use is confirmed**, and now on a complete
|
||||||
|
enumeration rather than a partial one.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. P9 — the monitoring rule fix (shipped)
|
||||||
|
|
||||||
|
**File:** `homelab-manifests/mon-system/prometheus-rules.yaml`, alert
|
||||||
|
`RootFsPressureDespiteHousekeeping`. **Commit `6808a4b`.**
|
||||||
|
|
||||||
|
**Change:** `and` → **`and on(instance)`**. A bare `and` requires identical label sets; the left vector
|
||||||
|
carries `{device,fstype,instance,job,mountpoint}` and the right — a node_exporter *textfile* metric —
|
||||||
|
carries only `{instance,job}`. Nothing ever matched.
|
||||||
|
|
||||||
|
### Red-proof (required, and both numbers recorded)
|
||||||
|
|
||||||
|
**Threshold substituted `0.25` → `0.5`, and the reason is stated rather than hidden:** P3's prune had
|
||||||
|
already taken `/` to 44.6% available, so the shipped threshold's left side is now legitimately false. The
|
||||||
|
defect is in **label matching**, not in the threshold, so the proof is run at a threshold where both
|
||||||
|
sides are individually true — which is exactly the condition the original rule was meant to catch.
|
||||||
|
|
||||||
|
| expression | `result_count` |
|
||||||
|
|---|---:|
|
||||||
|
| left side alone (`avail/size < 0.5`) | **1** |
|
||||||
|
| right side alone (`time() − last_success < 1209600`) | **1** |
|
||||||
|
| **as shipped — bare `and`** | **0** |
|
||||||
|
| **fixed — `and on(instance)`** | **1** |
|
||||||
|
|
||||||
|
For completeness, at the shipped 0.25 threshold **both** forms return 0 today — because the left side is
|
||||||
|
false now, not because the fix failed.
|
||||||
|
|
||||||
|
**Annotation rewritten.** It pointed at Longhorn replicas/snapshots (~10% of the growth). It now names
|
||||||
|
`docker buildx du` first, records the 30 GB cap in `/etc/docker/daemon.json`, and notes that a reverted
|
||||||
|
or unapplied cap looks exactly like this alert firing — so the alert text carries its own diagnosis.
|
||||||
|
|
||||||
|
### Deployment — GitOps, no `kubectl set image`, no live ConfigMap edit
|
||||||
|
|
||||||
|
ArgoCD app `monitoring` (auto-sync **off**): hard-refresh → `OutOfSync` at `6808a4be54a7` → deliberate
|
||||||
|
sync → **`Synced` / `Healthy`**, operation phase **`Succeeded`**.
|
||||||
|
|
||||||
|
**Verified in Prometheus, not assumed from ArgoCD:**
|
||||||
|
|
||||||
|
```
|
||||||
|
query : … < 0.25 and on (instance) time() - node_housekeeping_last_success_timestamp_seconds < 1209600
|
||||||
|
state : inactive health : ok lastError : (none)
|
||||||
|
```
|
||||||
|
|
||||||
|
`inactive` is now the **correct** answer — `/` really is fine at 44.6% available.
|
||||||
|
|
||||||
|
> **A near-miss worth recording: I invented a full commit SHA from ArgoCD's truncated display and the
|
||||||
|
> sync failed with `upload-pack: not our ref`.** It failed loudly and cost one retry. It would not have
|
||||||
|
> failed loudly had the fabricated SHA happened to exist.
|
||||||
|
|
||||||
|
### The finding P9 turned up on the way — R-211
|
||||||
|
|
||||||
|
**There is no config-reloader.** The `mon-system/prometheus` Deployment runs **one** container
|
||||||
|
(`prom/prometheus:v3.12.0`) with no reloader sidecar. After the sync the updated rules file was present
|
||||||
|
**inside the pod** (`grep -c "and on(instance)"` → 3 on the mounted symlink) while the Prometheus rules
|
||||||
|
API **still served the old expression for 4+ minutes**, with no error anywhere. It took effect only after
|
||||||
|
an explicit `POST /-/reload` (HTTP 200; `--web.enable-lifecycle` was already set).
|
||||||
|
|
||||||
|
**The consequence is general:** every rule edit in this repo has silently not applied until something
|
||||||
|
happened to restart the pod. `Synced/Healthy` was true and beside the point. Filed as **R-211**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. P6 — the SSD2 move: **EXECUTED 2026-08-05 on operator authorisation**
|
||||||
|
|
||||||
|
> **Executed after the operator ruled "proceed", having read the pre-analysis below. The
|
||||||
|
> `storageReserved` condition CC named was applied with it. The operator subsequently ruled
|
||||||
|
> explicitly: DO NOT REBOOT DooPlex — so the reboot validation is DEFERRED, and what that leaves
|
||||||
|
> unproven is stated in §11.9 rather than glossed.**
|
||||||
|
|
||||||
|
### 11.0 Execution record
|
||||||
|
|
||||||
|
**Blast radius, established before anything was stopped:** k3s runs a **separate** containerd
|
||||||
|
(`/run/k3s/containerd/containerd.sock`), so **Gitea, the registry, the hub, PBS, Longhorn and all
|
||||||
|
~160 pods stayed up throughout**. The only workloads affected were the two `jarr-*` dev containers.
|
||||||
|
`ctr -n moby containers list` confirmed exactly those two on the system containerd.
|
||||||
|
|
||||||
|
| # | step | result |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | stop `docker.service`, `docker.socket`, `containerd.service` | all `inactive` |
|
||||||
|
| 2 | `mkdir /mnt/ssd_2/containerd` (mode 711, root:root) | ok |
|
||||||
|
| 3 | `rsync -aHAX --numeric-ids` | **550,891 files, 33,276,370,041 B, 4 m 58 s** |
|
||||||
|
| 4 | `mv /var/lib/containerd → /var/lib/containerd.pre-move-2026-08-05` | **moved aside, never deleted** |
|
||||||
|
| 5 | `root = '/mnt/ssd_2/containerd'` in `/etc/containerd/config.toml` (backup kept) | one-line diff |
|
||||||
|
| 6 | `RequiresMountsFor=/mnt/ssd_2` drop-ins on **both** `containerd.service` and `docker.service` | resolved into both units |
|
||||||
|
| 7 | start + verify | see below |
|
||||||
|
| 8 | `storageReserved = 80 GB` on SSD2 | applied |
|
||||||
|
|
||||||
|
**`-X` is load-bearing** — overlayfs snapshots carry `trusted.overlay.*` xattrs, and losing them
|
||||||
|
would silently corrupt layer stacking.
|
||||||
|
|
||||||
|
### 11.1 Copy verified BEFORE the original was touched
|
||||||
|
|
||||||
|
Four independent observables, all matching:
|
||||||
|
|
||||||
|
| check | source | destination |
|
||||||
|
|---|---:|---:|
|
||||||
|
| filesystem entries | 550,891 | **550,891** |
|
||||||
|
| `trusted.overlay` xattrs | 448 | **448** |
|
||||||
|
| files with link count >1 (hardlinks) | 37,243 | **37,243** |
|
||||||
|
| `io.containerd.metadata.v1.bolt/meta.db` sha256 | `c60ecfb4ee134255…` | **`c60ecfb4ee134255…`** |
|
||||||
|
|
||||||
|
(Byte totals differ by +14 MB — ext4 block allocation, not content.)
|
||||||
|
|
||||||
|
### 11.2 Nothing was lost
|
||||||
|
|
||||||
|
| | pre-move | post-move |
|
||||||
|
|---|---:|---:|
|
||||||
|
| unique images | 345 | **345 — identical ID set** |
|
||||||
|
| tag lines | 350 | **350 — identical set** |
|
||||||
|
| docker volumes | 8 | **8 — identical set** |
|
||||||
|
| build cache | 2.782 GB / 38 records | **2.782 GB / 38 records** |
|
||||||
|
| GC ceiling | 30 GiB | **30 GiB, still in force** |
|
||||||
|
| postgres databases | 4 | **4** |
|
||||||
|
| postgres `jarr_dev` bytes | 175,135,767 | **175,135,767** |
|
||||||
|
| postgres public tables | 31 | **31** |
|
||||||
|
| redis `DBSIZE` | 2437 | **2437** |
|
||||||
|
|
||||||
|
**End-to-end proof, not just inventory:** a real `./build.sh` run against the relocated store
|
||||||
|
completed `rc=0`, produced an image, and grew the cache 2.782 → 4.012 GB. The scratch tag was removed
|
||||||
|
and `:latest` restored to the anchor digest afterwards.
|
||||||
|
|
||||||
|
### 11.3 The store is demonstrably on SSD2
|
||||||
|
|
||||||
|
```
|
||||||
|
containerd config dump → root = '/mnt/ssd_2/containerd'
|
||||||
|
du /mnt/ssd_2/containerd → 35,500,183,552 B
|
||||||
|
/var/lib/containerd → does not exist
|
||||||
|
```
|
||||||
|
|
||||||
|
### 11.4 The guard — and a trap found while proving it
|
||||||
|
|
||||||
|
**First attempt did NOT refuse, and that is a finding worth more than the test.** With
|
||||||
|
`RequiresMountsFor=/mnt/absent-guard-test-2026-08-05` — a path with **no mount unit at all** —
|
||||||
|
containerd **started normally**. `RequiresMountsFor` on a path systemd has no mount unit for is a
|
||||||
|
**silent no-op**. So a typo in the guard path buys nothing and says nothing: the same
|
||||||
|
*built-but-never-wired* shape this project has shipped repeatedly.
|
||||||
|
|
||||||
|
**So the guard was verified positively, at the unit level:**
|
||||||
|
|
||||||
|
```
|
||||||
|
containerd.service Requires=mnt-ssd_2.mount After=mnt-ssd_2.mount
|
||||||
|
docker.service Requires=mnt-ssd_2.mount
|
||||||
|
```
|
||||||
|
|
||||||
|
**Then the refusal itself was proven with a genuinely absent DEVICE.** `/mnt/ssd_2` hosts **12 live
|
||||||
|
Longhorn replicas** and must never be unmounted, and editing `fstab` on a production host risks
|
||||||
|
emergency mode at boot — so a **temporary synthetic `.mount` unit** pointing at a nonexistent UUID
|
||||||
|
was used instead, with a temporary `RequiresMountsFor` on the real `containerd.service`:
|
||||||
|
|
||||||
|
```
|
||||||
|
A dependency job for containerd.service failed.
|
||||||
|
systemd: Dependency failed for containerd.service - containerd container runtime.
|
||||||
|
systemd: containerd.service: Job containerd.service/start failed with result 'dependency'.
|
||||||
|
containerd is-active: inactive
|
||||||
|
```
|
||||||
|
|
||||||
|
**containerd REFUSED to start rather than creating an empty store on the root disk** — exactly the
|
||||||
|
required behaviour, and exactly the signature the ep0 runbook describes. All test artifacts were
|
||||||
|
removed; only `10-ssd2-mount.conf` remains on each unit and the guard resolves to `/mnt/ssd_2` alone.
|
||||||
|
|
||||||
|
### 11.5 Longhorn — the condition CC made the move conditional on
|
||||||
|
|
||||||
|
`storageReserved` on SSD2 raised **0 → 80,000,000,000 B**, covering the containerd tree so Longhorn's
|
||||||
|
scheduler can no longer oversubscribe into the space the store needs.
|
||||||
|
|
||||||
|
| disk | available | reserved | `Schedulable` |
|
||||||
|
|---|---:|---:|---|
|
||||||
|
| **Crucial-SSD-240G (SSD2)** | 179,516,211,200 (**76.34%**) | **80,000,000,000** | **True** |
|
||||||
|
| default-disk (SSD1) | 236,978,176,000 (49.64%) | 143,217,954,816 | **True** |
|
||||||
|
| HDD4 | 1,793,798,963,200 (60.82%) | 536,870,912,000 | True |
|
||||||
|
| HDD2 | 2,470,864,486,400 (24.91%) | 536,870,912,000 | **False** — pre-existing, untouched |
|
||||||
|
|
||||||
|
SSD2 remains schedulable with the reserve applied. **Rollback: patch `storageReserved` back to 0.**
|
||||||
|
|
||||||
|
### 11.6 A standing post-boot assertion was installed
|
||||||
|
|
||||||
|
`felhom-store-postboot-check.service` (oneshot, `After=docker.service`, enabled) runs
|
||||||
|
`/usr/local/sbin/felhom-store-postboot-check.sh` at **every** boot and logs to
|
||||||
|
`/var/log/felhom-store-postboot-check.log`. It asserts, with **positive** observables:
|
||||||
|
|
||||||
|
- `/mnt/ssd_2` is mounted;
|
||||||
|
- containerd's effective `root` is `/mnt/ssd_2/containerd`;
|
||||||
|
- **`/var/lib/containerd` does NOT exist** — i.e. no empty store was recreated on the root disk, which
|
||||||
|
is the precise trap the guard exists to prevent;
|
||||||
|
- docker sees ≥100 images;
|
||||||
|
- both dev containers are running.
|
||||||
|
|
||||||
|
Dry-run at install time: **RESULT: PASS** on all six lines. It was proven working *before* being
|
||||||
|
relied upon.
|
||||||
|
|
||||||
|
### 11.7 Disk position after the move
|
||||||
|
|
||||||
|
| | value |
|
||||||
|
|---|---:|
|
||||||
|
| `/` | 224 GB used, **199 GB free, 54%** |
|
||||||
|
| `/mnt/ssd_2` | 52 GB used, **157 GB free, 25%** |
|
||||||
|
| store on SSD2 | 35,500,183,552 B |
|
||||||
|
| **rollback copy retained on `/`** | **34,299,498,496 B** |
|
||||||
|
|
||||||
|
**`/` did not drop below 54% because the rollback copy is deliberately still there.** Reclaiming it
|
||||||
|
returns ~34 GB — see §11.9.
|
||||||
|
|
||||||
|
### 11.8 Rollback, complete and one sequence
|
||||||
|
|
||||||
|
1. `sudo systemctl stop docker docker.socket containerd`
|
||||||
|
2. `sudo mv /var/lib/containerd.pre-move-2026-08-05 /var/lib/containerd`
|
||||||
|
3. `sudo cp /etc/containerd/config.toml.pre-move-2026-08-05 /etc/containerd/config.toml`
|
||||||
|
4. `sudo rm -rf /etc/systemd/system/{containerd,docker}.service.d/10-ssd2-mount.conf`
|
||||||
|
5. `sudo systemctl disable --now felhom-store-postboot-check.service && sudo rm /etc/systemd/system/felhom-store-postboot-check.service /usr/local/sbin/felhom-store-postboot-check.sh`
|
||||||
|
6. `sudo kubectl patch nodes.longhorn.io -n longhorn-system dooplex --type merge -p '{"spec":{"disks":{"Crucial-SSD-240G":{"storageReserved":0}}}}'`
|
||||||
|
7. `sudo systemctl daemon-reload && sudo systemctl start containerd docker`
|
||||||
|
8. `sudo rm -rf /mnt/ssd_2/containerd`
|
||||||
|
|
||||||
|
### 11.9 **NOT PROVEN: the reboot. Stated, not glossed.**
|
||||||
|
|
||||||
|
**The operator ruled explicitly that DooPlex must not be rebooted, so the reboot validation did not
|
||||||
|
happen.** Uptime is unbroken (7 weeks 6 days, since 2026-06-10) — verified, not assumed.
|
||||||
|
|
||||||
|
**By this task's own standard — *"a move that has not survived a reboot has not been validated"* — the
|
||||||
|
move is NOT fully validated,** and it would be dishonest to close it as if it were. Precisely:
|
||||||
|
|
||||||
|
- **PROVEN (mechanism):** the guard is wired into both units (`Requires=` + `After=mnt-ssd_2.mount`),
|
||||||
|
and containerd **refuses to start** when a required mount's device is genuinely absent.
|
||||||
|
- **NOT PROVEN (consequence):** that a real boot sequence mounts `/mnt/ssd_2` before containerd starts
|
||||||
|
*on this host, in the real ordering*. Mount-ordering reasoning is exactly the class this project has
|
||||||
|
been burned by, and the mechanism proof does not substitute for it. `CLAUDE.md` prefers the
|
||||||
|
consequence over the mechanism; here only the mechanism is available.
|
||||||
|
|
||||||
|
**Two consequences follow, and both are deliberate:**
|
||||||
|
|
||||||
|
1. **The rollback copy `/var/lib/containerd.pre-move-2026-08-05` (34.3 GB) STAYS** until a reboot has
|
||||||
|
validated the move. Deleting it now would trade a cheap 34 GB for the only cheap way back.
|
||||||
|
2. **The next reboot — whenever it happens, planned or not — validates this automatically.** The
|
||||||
|
post-boot service writes `RESULT: PASS` or `RESULT: FAIL` to
|
||||||
|
`/var/log/felhom-store-postboot-check.log`. **Read that file after the next boot**; on `PASS`, the
|
||||||
|
rollback copy can be removed and ~34 GB returned to `/`.
|
||||||
|
|
||||||
|
Tracked as **R-209a**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11-bis. P6 — the pre-analysis the ruling was made on
|
||||||
|
|
||||||
|
*(Retained as written, before execution, so the ruling can be audited against what it was given.)*
|
||||||
|
|
||||||
|
**First, the framing changed underneath this probe.** P3's prune took `/` to 53% used and un-stuck
|
||||||
|
SSD1's Longhorn disk. **The move is no longer a rescue; it is a deliberate choice.** Per standing rule 4,
|
||||||
|
CC's disagreement is stated rather than absorbed: **the cap alone resolved the problem the move was meant
|
||||||
|
to solve, and the move carries a Longhorn risk the cap does not.**
|
||||||
|
|
||||||
|
### Mechanism — three candidates
|
||||||
|
|
||||||
|
| | mechanism | survives package upgrade | expressible in Ansible | visible reading the host cold |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| **(a)** | **`root = "/mnt/ssd_2/…"` in `/etc/containerd/config.toml`** | **yes** — dpkg conffile, local edits preserved | **yes** — a template, the file is already managed-shaped | **yes** — one line in the daemon's own config |
|
||||||
|
| (b) | systemd mount unit / fstab bind onto `/var/lib/containerd` | yes | yes | **weak** — invisible unless you check `findmnt` |
|
||||||
|
| (c) | symlink | fragile | yes | **weak**, and overlayfs mount semantics through a symlink are unproven here |
|
||||||
|
|
||||||
|
**Recommendation: (a).** It is the only one that a person reading the host cold can *see*, and the key
|
||||||
|
already exists in the file (commented out).
|
||||||
|
|
||||||
|
### The boot-order trap, and its guard
|
||||||
|
|
||||||
|
If containerd or Docker starts before `/mnt/ssd_2` is mounted, the daemon **creates an empty store on the
|
||||||
|
root disk** and 345 images appear to have vanished.
|
||||||
|
|
||||||
|
**Guard:** `RequiresMountsFor=/mnt/ssd_2` on **`containerd.service`** and **`docker.service`**. Neither
|
||||||
|
has it today — `containerd.service` carries only `After=network.target dbus.service`.
|
||||||
|
|
||||||
|
> **`RequiresMountsFor` RE-MOUNTS; it does not refuse.** systemd pulls the mount unit in and *starts* it,
|
||||||
|
> so an unmounted-but-available device is silently re-mounted and everything looks fine. It refuses only
|
||||||
|
> when the device is genuinely **unavailable**. **Testing it by unmounting proves nothing** — it must be
|
||||||
|
> tested by pointing at an absent device. This is the ep0 datastore lesson
|
||||||
|
> (`RUNBOOK-ep0-datastore-volume-2026-07-27.md`), earned on this project.
|
||||||
|
|
||||||
|
**And a move that has not survived a reboot has not been validated.** Mount-ordering reasoning is exactly
|
||||||
|
the class this project has been burned by.
|
||||||
|
|
||||||
|
### The Longhorn consequence — the part that decides it
|
||||||
|
|
||||||
|
`Crucial-SSD-240G`, `storageMaximum` 235,148,750,848, **25% floor = 58,787,187,712**:
|
||||||
|
|
||||||
|
| scenario | resulting `storageAvailable` | as % | vs the 25% floor |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| today, untouched | 214,958,080,000 | 91.41% | +66.4 pp |
|
||||||
|
| move cache only (30 GB) | 184,958,080,000 | 78.66% | **+53.7 pp** |
|
||||||
|
| move whole containerd tree today (34.3 GB) | 180,658,569,216 | 76.83% | +51.8 pp |
|
||||||
|
| **move at steady state (~31.5 GB images + 30 GB cache ≈ 65 GB)** | 149,958,080,000 | **63.77%** | **+38.8 pp** |
|
||||||
|
|
||||||
|
**As measured, the move is safe by a wide margin — nowhere near the 10-pp caution band.**
|
||||||
|
|
||||||
|
**But the worst case is not comfortable, and it is not hypothetical enough to omit.** `storageScheduled`
|
||||||
|
on SSD2 is **139,586,437,120** while actual usage is **20,094,939,136** — a 6.9× overcommit. If those 12
|
||||||
|
volumes ever inflate to their scheduled size:
|
||||||
|
|
||||||
|
| scenario | available | as % | vs floor |
|
||||||
|
|---|---:|---:|---|
|
||||||
|
| + cache only (30 GB) | 65,562,313,728 | 27.88% | +2.9 pp — inside the caution band |
|
||||||
|
| **+ steady state (65 GB)** | 30,562,313,728 | **13.00%** | **−12.0 pp → `Schedulable=False`** |
|
||||||
|
|
||||||
|
That is precisely the failure that just took SSD1 out of service.
|
||||||
|
|
||||||
|
> **Recommendation: proceed only together with setting Longhorn's `storageReserved` on SSD2 to cover the
|
||||||
|
> containerd tree (~80 GB).** SSD2 reserving **zero** while HDD2 and HDD4 each reserve 500 GB is an
|
||||||
|
> anomaly in its own right, and it is what lets Longhorn oversubscribe into the space the cache needs.
|
||||||
|
> Without that, CC recommends **not moving** — the cap has already done the job.
|
||||||
|
|
||||||
|
### The sequence, prepared and not executed
|
||||||
|
|
||||||
|
Each step with its rollback. **Do not run without the operator.**
|
||||||
|
|
||||||
|
| # | step | rollback |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | `systemctl stop docker docker.socket containerd` | `systemctl start containerd docker` |
|
||||||
|
| 2 | `mkdir -p /mnt/ssd_2/containerd` | `rmdir` |
|
||||||
|
| 3 | `rsync -aHAX --numeric-ids /var/lib/containerd/ /mnt/ssd_2/containerd/` (small now — ~34 GB, and ~2.8 GB of it is cache) | delete the copy; the original is untouched |
|
||||||
|
| 4 | `mv /var/lib/containerd /var/lib/containerd.pre-move` (**move aside, never delete**) | `mv` back |
|
||||||
|
| 5 | set `root = '/mnt/ssd_2/containerd'` in `/etc/containerd/config.toml` | restore the commented line |
|
||||||
|
| 6 | drop-in `RequiresMountsFor=/mnt/ssd_2` on `containerd.service` **and** `docker.service`; `systemctl daemon-reload` | delete the drop-ins |
|
||||||
|
| 7 | `systemctl start containerd docker`; **verify `docker images` = 345, both dev containers up, `du` confirms the store's real path** | steps 5→4→1 in reverse |
|
||||||
|
| 8 | set SSD2 `storageReserved` ≈ 80 GB in the Longhorn node CR | set back to 0 |
|
||||||
|
| 9 | **`reboot`, then re-verify everything in step 7** | as above |
|
||||||
|
| 10 | test the guard by pointing at an **absent** device — an unmount alone proves nothing | restore fstab |
|
||||||
|
| 11 | only after a clean reboot + a full build: `rm -rf /var/lib/containerd.pre-move` | — none; do this last |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Teardown
|
||||||
|
|
||||||
|
**Nothing was provisioned.** No machine, no VM, no LXC guest, no hub record, no customer, no Longhorn
|
||||||
|
volume, no PVC. There is nothing to decommission.
|
||||||
|
|
||||||
|
**Cleaned up:**
|
||||||
|
|
||||||
|
- The six candidate `daemon.json` files written for P2 (in the session scratch dir, never in
|
||||||
|
`/etc/docker/`) — **removed**.
|
||||||
|
- The scratch workspace fingerprint tree (`ws_sim`) — **removed**.
|
||||||
|
- **Nine scratch image tags** created by the probe builds — `felhom-spike:{p7c,e1,e2}`,
|
||||||
|
`felhom-controller:{spike-p7a,spike-p7b,spike-p7d,evict-c1,evict-c2,evict-c3}`,
|
||||||
|
`felhom-hub:{evict-h1,evict-h2,evict-h3}` — **all removed**.
|
||||||
|
- `felhom-controller:latest` **restored** to `sha256:bdcd4a4ae8e006f1…` (0.198.0), byte-matching the
|
||||||
|
anchor captured before the first build. Image count back to **345**, the P0 baseline.
|
||||||
|
|
||||||
|
**Intentional state left in place, recorded here so it does not read as a leftover:**
|
||||||
|
**`/etc/docker/daemon.json` exists**, with the content in §4. It did not exist before this session.
|
||||||
|
Removing it plus a daemon restart is a complete revert.
|
||||||
|
|
||||||
|
**Not cleaned up, deliberately:** the ~2.78 GB of build cache from the eviction sequence. It is the
|
||||||
|
working cache under a live 30 GB policy, not scratch.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. Final state
|
||||||
|
|
||||||
|
| | at arrival | **now** |
|
||||||
|
|---|---:|---:|
|
||||||
|
| `/` used | 387,332,755,456 B (**86%**) | **240,301,133,824 B (54%)** |
|
||||||
|
| `/` available | 65,734,934,528 B | **212,766,556,160 B** |
|
||||||
|
| `/var/lib/containerd` | 181,394,223,104 B | **34,299,510,784 B** |
|
||||||
|
| build cache | 157 GB / 1446 records | **2.782 GB / 38 records, capped at 30 GiB** |
|
||||||
|
| images | 345 | **345** |
|
||||||
|
| dev containers | 2 healthy | **2 healthy, data verified intact** |
|
||||||
|
| SSD1 Longhorn | **`Schedulable=False (DiskPressure)`** | **`Schedulable=True`, 50.32%** |
|
||||||
|
| `RootFsPressureDespiteHousekeeping` | **dead — could never fire** | **live, `health=ok`, red-proven** |
|
||||||
|
| growth driver | unbounded, ~5.25 GB/day | **capped; source diagnosed (R-208)** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14. Observations — noticed, deliberately not acted on
|
||||||
|
|
||||||
|
1. **`build.sh` exists in two copies** — `felhom-controller/controller/build.sh` (in git) and
|
||||||
|
`/mnt/5_hdd/felhom.eu/build/felhom-controller/build.sh` (the operative one). They differ by **one
|
||||||
|
comment line**. The hub solved this with a symlink into the repo; the controller has not. Not fixed —
|
||||||
|
out of scope, but it is a drift channel.
|
||||||
|
2. **`build.sh` tags `:latest` on every build**, including throwaway version tags. A spike build silently
|
||||||
|
moved `latest` off 0.198.0; it was restored from an anchor captured beforehand. A `--no-latest` flag,
|
||||||
|
or tagging `latest` only on `--push`, would remove the trap.
|
||||||
|
3. **The default BuildKit GC policy is derived from disk size**, so a bigger root disk would have raised
|
||||||
|
the 331.6 GiB ceiling too. The cap needs to be explicit no matter how much disk is added.
|
||||||
|
4. **`node-housekeeping.sh`'s own `avail_bytes()` uses `df … | tail -1`** — the truncating-instrument
|
||||||
|
shape this project has been bitten by twice. It is *correct* here (`df` on one path emits exactly one
|
||||||
|
data row), so it was left alone, but it is worth knowing it is there.
|
||||||
|
5. **HDD2 remains `Schedulable=False (DiskPressure)`** at 24.91%, a hair under the 25% floor with 2.47 TB
|
||||||
|
free and its 500 GB reserve satisfied. Untouched by this work, and not fixed by it.
|
||||||
|
6. **The `mon-system` Prometheus has `--web.enable-admin-api`** as well as `--web.enable-lifecycle`. The
|
||||||
|
admin API permits series deletion. Noted, not acted on.
|
||||||
|
7. **Longhorn on SSD2 is overcommitted 6.9×** (139.6 GB scheduled vs 20.1 GB used). That is normal for
|
||||||
|
thin provisioning, but combined with `storageReserved = 0` it is what makes the P6 worst case reach
|
||||||
|
the floor.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 15. Register
|
||||||
|
|
||||||
|
Allocated this session: **R-205** (CLOSED — the monitoring rule, shipped), **R-206** (Ansible: cap +
|
||||||
|
prune + narrowed Docker ban), **R-207** (the `DRY_RUN` guard), **R-208** (the `ARG`-ordering cache miss —
|
||||||
|
the largest remaining win), **R-209** (WAITING-ON-OPERATOR — whether to move to SSD2 at all),
|
||||||
|
**R-210** (WAITING-ON-OPERATOR — the image list), **R-211** (no Prometheus config-reloader).
|
||||||
@@ -0,0 +1,540 @@
|
|||||||
|
# SPIKE — what is actually one-shot, and what a guest rebuild really costs
|
||||||
|
|
||||||
|
**Date:** 2026-08-04 · **Items:** R-193 (rebuild drops offsite), R-192 (the alert says the opposite of
|
||||||
|
what it measured) · **Class:** spike — **no production code shipped for R-193 or R-192**
|
||||||
|
**Baselines read:** `felhom.eu` @ `f456835bbcc4` (hub v0.91.1) · `felhom-controller` @ `0887fd67` ·
|
||||||
|
`felhom-agent` @ `856a127`
|
||||||
|
|
||||||
|
> **This document ends in ranked options and a STOP.** It establishes facts; it does not choose.
|
||||||
|
> The operator's question is stated, unanswered, at the end.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
> ### ANNOTATION 2026-08-04 (later the same day) — read this before acting on §Q8
|
||||||
|
>
|
||||||
|
> A follow-up recon traced the escrow→recovery chain link by link
|
||||||
|
> (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **Two corrections to this document, and they pull
|
||||||
|
> in opposite directions.**
|
||||||
|
>
|
||||||
|
> **(1) Q8 candidate (b) — "not implementable as specified" — is TOO STRONG and is overturned in
|
||||||
|
> part.** Zero-knowledge means the *hub* cannot open the blob; the *customer* can, with R, which is
|
||||||
|
> what the ceremony exists for. The mechanism to hand a rebuilt box its old repo password back —
|
||||||
|
> customer-present, not unattended — is a real design, and the operator has since ruled on its shape.
|
||||||
|
> What (b) got right is narrower than what it said: **an UNATTENDED rebuild cannot recover from a
|
||||||
|
> zero-knowledge escrow.** A customer-present one can. The recon prices that shape row by row.
|
||||||
|
>
|
||||||
|
> **(2) A LOAD-BEARING FACT THIS SPIKE DID NOT CHECK, and it makes (d)-alone weaker than ranked
|
||||||
|
> here.** `host_escrow_superseded` **has no `identity_blob` column**, and `demoteCurrentEscrowTx`
|
||||||
|
> (`hub/internal/store/store.go:2547-2556`) copies only the K-escrow `blob`. So the retained
|
||||||
|
> "superseded" blob holds the **PBS datastore key**, not the **restic repository password**. The
|
||||||
|
> retention feature does not retain the secret it was built to retain, and the **next escrow ceremony
|
||||||
|
> — which the system tells the customer to run whenever a rebuild leaves the escrow `pending` — is
|
||||||
|
> what destroys the last copy.** Both demo boxes crossed that line on 2026-08-04 at 07:15:36 and
|
||||||
|
> 07:20:08. → **R-198**.
|
||||||
|
>
|
||||||
|
> **Consequence for this document's recommendation.** "Ship (d), then decide (c)" still holds, but
|
||||||
|
> (d)'s implicit promise — *the old history stays recoverable with the recovery code* — is **false as
|
||||||
|
> the code stands**, and is repeated verbatim to the customer in Hungarian on the orphan card. (d) is
|
||||||
|
> not complete without R-198.
|
||||||
|
>
|
||||||
|
> **Unchanged and still correct:** Q1, Q2, Q4 (R-196), Q5, Q6, Q7 (R-192) and the Q8 (a)/(c)/(d)
|
||||||
|
> analyses. **Q3 remains UNMEASURED** — neither box has run since; the 2026-08-05 ~02:15 UTC run is
|
||||||
|
> still the decisive one.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. Why this exists
|
||||||
|
|
||||||
|
Two candidate fixes were named when R-193 was filed, and neither could be weighed, because two prior
|
||||||
|
session reports described the same mechanism in contradictory terms:
|
||||||
|
|
||||||
|
- R-39's record: Re-issue *"rotates the restic password and makes the escrow stale"*.
|
||||||
|
- R-193's own resolution note: the one-time password is *"only the transport credential used once to
|
||||||
|
install the box's own SSH key"*.
|
||||||
|
|
||||||
|
Those two worlds imply very different fates for the fifteen existing snapshots. This spike settles it
|
||||||
|
**from source first, live state second**, so a measurement can contradict the code rather than be
|
||||||
|
interpreted by it.
|
||||||
|
|
||||||
|
**The headline, before the detail:** *both prior claims are wrong in the way that matters.* Re-issue
|
||||||
|
does **not** touch the restic repository password — but a **guest rebuild does**, silently, on every
|
||||||
|
box, with nothing marking the escrow stale and nothing telling anyone. Both demo boxes lost repository
|
||||||
|
continuity on 2026-08-03/04. The one that "recovered by luck" lost it too.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. The instruments, and what each one can and cannot answer
|
||||||
|
|
||||||
|
Stated up front so a wrong instrument is visible here rather than inside a conclusion (§8 rule 2).
|
||||||
|
|
||||||
|
| # | Instrument | Exact query / read | Known limit |
|
||||||
|
|---|---|---|---|
|
||||||
|
| I1 | hub SQLite snapshot | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale at copy time (mtime 08:34 vs wal 10:45 CEST). Copying only the main file would have produced confident answers from a stale file — the exact R-3 shape. |
|
||||||
|
| I1-check | freshness proof | `PRAGMA integrity_check` → `ok`; `MAX(host_reports.received_at)` = `2026-08-04 08:45:38`, `datetime('now')` = `08:49:46` | **Positive observable**: newest row 4 min old. Not "the query returned no error". |
|
||||||
|
| I2 | `host_escrow.restic_pw_sha256` + `host_escrow_superseded.restic_pw_sha256` | see §2 | sha256 of a 256-bit random secret — non-reversible, safe to compare. Answers *"did the repo password change"* **without touching a box or a credential**. |
|
||||||
|
| I3 | `reports.report_json → $.offsite` | grouped distinct-object query with first/last seen | The report echoes the box's own target state; absence of the key = no target configured. |
|
||||||
|
| I4 | `one_time_secrets` | `SELECT customer_id, created_at, consumed_at` | **`customer_id` is the PRIMARY KEY** — one row per customer, last-write-wins. It **cannot** answer "how many credentials were delivered"; a Re-issue destroys the prior row. demo-hp's 2026-07-23 consume is no longer in this table. |
|
||||||
|
| I5 | source | `hub/internal/offsite/*`, `hub/internal/api/*`, `controller/internal/{backup,offsiteapply}`, `agent/internal/escrow` | — |
|
||||||
|
|
||||||
|
**Not used, deliberately:** no `ReissueCredentials`, no ceremony, no config change, no deletion, no
|
||||||
|
write to any box. Nothing on either box or on the storage endpoint was altered by this session.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q2 — Does a rebuilt controller reuse the escrowed repository password, or mint a new one?
|
||||||
|
|
||||||
|
### **RULING: it MINTS A NEW ONE. The escrowed password is never consulted on any automatic path.**
|
||||||
|
|
||||||
|
**Method — source.** The apply-bridge's terminal step is
|
||||||
|
`Enabler.ConfigureOffbox(...)` (`controller/internal/offsiteapply/offsiteapply.go:221,256`), which
|
||||||
|
reaches `Manager.ApplyOffsiteTarget` → `WriteOffboxSecrets`
|
||||||
|
(`controller/internal/backup/offbox.go:370`). Quoted verbatim, `offbox.go:392-401`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Auto-generate the repo password once (0600), never log it.
|
||||||
|
if _, err := os.Stat(m.offboxPwPath()); os.IsNotExist(err) {
|
||||||
|
pw, gerr := generateOffboxPassword()
|
||||||
|
...
|
||||||
|
if werr := os.WriteFile(m.offboxPwPath(), []byte(pw), 0o600); werr != nil {
|
||||||
|
```
|
||||||
|
|
||||||
|
`offboxPwPath()` is `<DataDir>/offbox/repo_password` (`offbox.go:363`). A rebuilt guest has a fresh
|
||||||
|
data dir, so the file is absent, so **a fresh 256-bit password is minted**.
|
||||||
|
|
||||||
|
**The only path that recovers the old one is `InjectOffboxPassword`** (`offbox.go:541`), whose sole
|
||||||
|
caller in the entire repo is `offboxInjectPasswordHandler`
|
||||||
|
(`controller/internal/web/offbox_handlers.go:189`) — a **web form where a human pastes the recovered
|
||||||
|
password**. Verified by census: `grep -rn "InjectOffboxPassword" --include=*.go .` → three hits, one
|
||||||
|
definition, one caller, one comment. **The apply-bridge never calls it.**
|
||||||
|
|
||||||
|
**Method — live, and it is decisive without touching a box (I2).** The hub already stores the sha256
|
||||||
|
of the escrowed repo password, so the question is answerable as a hash comparison:
|
||||||
|
|
||||||
|
| host | superseded (pre-rebuild) | current (post-rebuild) | verdict |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `demo-hp-bb76ea` | `8e03eddf9ff7…` created `2026-07-23T10:01:17Z` | `8a9e33aa4da6…` created `2026-08-04T07:15:36Z` | **CHANGED** |
|
||||||
|
| `demo-felhom-8363b5` | `48741892f0ef…` created `2026-07-21T08:38:33Z` | `c60c8bc737a6…` created `2026-08-04T07:20:07Z` | **CHANGED** |
|
||||||
|
|
||||||
|
**Both boxes minted a new repository password.** demo-hp's pre-rebuild escrow (`8e03…`, sealed eight
|
||||||
|
minutes after its 2026-07-23 09:53:41 apply) is the key to its 15 snapshots. That key now exists
|
||||||
|
**only inside a superseded, R-wrapped escrow blob** that the hub cannot open.
|
||||||
|
|
||||||
|
### **This is bigger than R-193 as filed, and it hits the box R-193 called lucky.**
|
||||||
|
|
||||||
|
R-193 records demo-felhom as having *"survived the SAME rebuild by luck"* — 76 seconds of downtime.
|
||||||
|
Measured (I3), demo-felhom's pre-rebuild offsite object was:
|
||||||
|
|
||||||
|
```
|
||||||
|
last_run 2026-08-03T02:17:44Z · last_status ok · snapshot_count 36 · repo_size_bytes 1 136 685 919
|
||||||
|
```
|
||||||
|
|
||||||
|
**36 snapshots, 1.14 GB.** Since 2026-08-03 07:19:10 UTC it has reported `snapshot_count: 0,
|
||||||
|
repo_size_bytes: 0` in every one of 109 reports, and its repo password hash changed.
|
||||||
|
|
||||||
|
> **The luck recovered DELIVERY. It did not recover the REPOSITORY.** The staged secret restored the
|
||||||
|
> transport in 76 seconds and the box then minted a brand-new repo password anyway. The contrast
|
||||||
|
> R-193 draws between the two boxes is real for the *credential*, and **false for the data**: both
|
||||||
|
> boxes lost repository continuity, one loudly and one silently. The silent one is worse.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q3 — What did the 04:15 run report?
|
||||||
|
|
||||||
|
### **RULING: UNMEASURED — the decisive run has not happened yet. And the binary the question offers is the wrong one; source says the answer will be NEITHER 15 nor 1.**
|
||||||
|
|
||||||
|
**Why it has not happened.** The scheduled off-box run fires at ~02:15 UTC (04:15 CEST). Measured
|
||||||
|
from I3 — `last_run` values `2026-08-02T02:15:52Z`, `2026-08-03T02:16:39Z` on demo-hp;
|
||||||
|
`2026-08-01T02:17:54Z`, `2026-08-02T02:17:01Z`, `2026-08-03T02:17:44Z` on demo-felhom. **Daily, both
|
||||||
|
boxes.** Then:
|
||||||
|
|
||||||
|
| box | why 2026-08-04 02:15 UTC produced nothing |
|
||||||
|
|---|---|
|
||||||
|
| demo-hp | no offsite target existed at all — the `offsite` key was **ABSENT** from every report between `2026-08-03 06:12:19` and `2026-08-04 07:12:01` UTC (111 reports). The re-issue landed at 07:15:47, five hours after the window. |
|
||||||
|
| demo-felhom | the target existed but `escrow_state: pending` from `2026-08-03 07:19:10` to `2026-08-04 07:20:13`. `OffboxRunnable` requires configured **AND** escrowed (`offbox.go:564-569`), so the fork-4 gate blocked the run. The ceremony landed at 07:20:28. |
|
||||||
|
|
||||||
|
Both boxes became runnable only this morning. **The decisive run is 2026-08-05 ~02:15 UTC on both.**
|
||||||
|
Reported as unmeasured rather than inferred.
|
||||||
|
|
||||||
|
### What source says will happen — and it is a third outcome
|
||||||
|
|
||||||
|
Same sub-account, same repo path, new password. Measured coordinates (I3, `$.dr_recipe`):
|
||||||
|
`u629488-sub3.your-storagebox.de:/home/felhom-repo` — **unchanged** across the whole incident, and
|
||||||
|
`repoPath` is a compile-time constant (`hub/internal/offsite/offsite.go:85`). So the next run opens an
|
||||||
|
**existing repo with the wrong key**, which the codebase already has a name for
|
||||||
|
(`controller/internal/backup/offbox.go:67-93`):
|
||||||
|
|
||||||
|
```go
|
||||||
|
// ErrOffboxOrphaned is the sentinel returned when the offsite repo exists but is keyed under a
|
||||||
|
// passphrase this controller no longer has (the reinstall shape) ...
|
||||||
|
case strings.Contains(s, "wrong password or no key found"):
|
||||||
|
return "orphaned"
|
||||||
|
```
|
||||||
|
|
||||||
|
and `ensureOffboxRepo` (`offbox.go:663-679`) branches on claim state:
|
||||||
|
|
||||||
|
```go
|
||||||
|
if !m.settings.GetClaimed() { // UNCLAIMED → auto move-aside + re-init
|
||||||
|
...
|
||||||
|
m.markOrphaned()
|
||||||
|
return ErrOffboxOrphaned // CLAIMED → skip the run, show the orphan card
|
||||||
|
```
|
||||||
|
|
||||||
|
Both boxes report `claimed: 1` (I3, `$.claimed`, latest report each). **So the predicted outcome is:
|
||||||
|
the run REFUSES with `ErrOffboxOrphaned`, produces no snapshot at all, and waits for the customer to
|
||||||
|
confirm a reset.** Not 15 (reattached) and not 1 (silently fresh) — a **third** outcome the question's
|
||||||
|
framing did not contain. That is the good news half: the system is not going to quietly start a new
|
||||||
|
history over the old one. It will stop and say so.
|
||||||
|
|
||||||
|
**Consequence if the customer confirms the reset:** `resetOrphanedRepo` (`offbox.go:270-314`) moves the
|
||||||
|
old repo aside and re-inits — *"move-aside, not deleted"*. The 40.9 MB + 1.14 GB of old ciphertext
|
||||||
|
survives, unreadable without the superseded escrow, **and keeps consuming the 50 GB soft quota
|
||||||
|
indefinitely.** Nothing prunes a moved-aside restic repo (S-24 covers ep0's PBS namespaces, a
|
||||||
|
different tier).
|
||||||
|
|
||||||
|
**To measure it tomorrow:** re-read `$.offsite.last_status` / `last_error` on both boxes after
|
||||||
|
02:20 UTC, and `$.offsite.repo_state` for the orphan flag. **Predicted `ErrOffboxOrphaned`, not a
|
||||||
|
snapshot count.** Whoever picks this up should record which of the three actually occurred — a
|
||||||
|
prediction from source is not a measurement.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q1 — What is one-shot, and what is not?
|
||||||
|
|
||||||
|
Three secrets, and they have almost nothing in common.
|
||||||
|
|
||||||
|
| | **① storage-provider password** | **② the box's SFTP key** | **③ restic repository password** |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **What it is** | the Hetzner sub-account / box password | ed25519 keypair for `sftp` transport | the **data** key — restic's repo encryption |
|
||||||
|
| **Generated by** | **hub** — `genPassword()` (`offsite.go:481`) | **controller** — `KeyGen.Generate()` (`offsiteapply.go:234`) | **controller** — `generateOffboxPassword()`, 32 random bytes hex (`offbox.go:406`) |
|
||||||
|
| **Stored where** | `one_time_secrets` (hub, plaintext, **one row per customer**) | `<DataDir>/offbox/ssh_key` 0600 (box only) | `<DataDir>/offbox/repo_password` 0600 (box) + inside the R-wrapped escrow blob (hub, opaque) |
|
||||||
|
| **Delivered how** | `GET …/offsite/consume-password`, **served exactly once** then marked consumed (`api/offsite.go:9-31`) | never delivered — installed onto the provider by ssh-copy-id using ① | never delivered anywhere; it never leaves the box except into the escrow ceremony |
|
||||||
|
| **Re-issuable?** | **YES** — `ReissueCredentials` resets it at the provider, any time, operator-initiated | **YES** — regenerated on every full apply; `dr_recipe.go:45` states it plainly: *"the SFTP access key is regenerated at DR"* | **NO automatic path.** Recoverable only by unsealing the escrow with the customer's recovery code and pasting it into `offboxInjectPasswordHandler` |
|
||||||
|
| **What a guest rebuild does to it** | nothing (it lives on the hub / at the provider) | destroys it; a fresh one is minted and installed — **harmless** | **destroys it; a fresh one is minted — CATASTROPHIC for the existing repo** |
|
||||||
|
| **Escrowed?** | no | no | **yes** — `IdentityBundle.ResticRepoPassword` |
|
||||||
|
|
||||||
|
The agent's own source names the asymmetry, and it is the sentence the whole spike turns on
|
||||||
|
(`felhom-agent/internal/escrow/identity.go:35-39`):
|
||||||
|
|
||||||
|
> `ResticRepoPassword` … **It is the DATA key for the offsite tier — irreplaceable (unlike the SFTP
|
||||||
|
> access key, which is regenerable at DR).**
|
||||||
|
|
||||||
|
**The one-shot thing (①) is the recoverable one. The irreplaceable thing (③) is the one nothing
|
||||||
|
re-stages.** R-193 was filed against ①. The damage is in ③.
|
||||||
|
|
||||||
|
**Instrument caveat, worth carrying (I4):** `one_time_secrets` has `customer_id` as PRIMARY KEY, and
|
||||||
|
`SaveOneTimeSecret` is last-write-wins **by design** (the R-39(a) guard comment in
|
||||||
|
`monitor/offsite_delivery.go:26-30` depends on it). So the delivery ledger holds only the newest
|
||||||
|
credential per customer: demo-hp's 2026-07-23 consume, on which R-192's whole diagnosis rests, was
|
||||||
|
**overwritten** by the 2026-08-04 Re-issue and is no longer readable from that table.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q4 — Which claim about Re-issue is true?
|
||||||
|
|
||||||
|
### **RULING: `ReissueCredentials` touches ONLY the provider credential (①). It does NOT touch the restic repository password. R-39's record is wrong — and the hub's own comments repeat the same wrong claim in three places.**
|
||||||
|
|
||||||
|
**Method — source, quoted rather than summarised.** `ReissueCredentials`
|
||||||
|
(`hub/internal/offsite/offsite.go:150-228`) does exactly four things:
|
||||||
|
|
||||||
|
1. `genPassword()`;
|
||||||
|
2. `ResetSubaccountPassword` / `ResetBoxPassword` + `WaitAction` — **the provider account password**;
|
||||||
|
3. `Store.SaveOneTimeSecret(customerID, pw)` — stage ① for delivery;
|
||||||
|
4. bookkeeping: `MarkEscrowStale` + `offsite_reissued` + `escrow_stale` events.
|
||||||
|
|
||||||
|
There is **no reference to a restic password anywhere in the function**, and none is possible — the
|
||||||
|
repo password is generated on the box and never leaves it except into the escrow blob. The hub does
|
||||||
|
not hold it in any openable form (`api/handler.go`: *"The hub stores the bytes and NEVER decrypts them
|
||||||
|
(it has no recovery code)"*).
|
||||||
|
|
||||||
|
**But step 4 is justified by a claim that is false**, `offsite.go:198-201`:
|
||||||
|
|
||||||
|
> `// v0.57.0 (2.3, the escrow-honesty fix): the restic repo password just changed, so any existing`
|
||||||
|
> `// key-escrow blob — which sealed the OLD password — is now STALE.`
|
||||||
|
|
||||||
|
The escrow *is* marked stale; the stated reason for marking it is not something this function does.
|
||||||
|
The same false premise appears twice more: `api/handler.go:1067-1069` (*"The re-issuer resets the
|
||||||
|
restic repo password, which makes the OLD escrow blob stale"*) and `OPEN-ITEMS.md`'s R-193 row, which
|
||||||
|
inherited it from R-39.
|
||||||
|
|
||||||
|
> **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does
|
||||||
|
> not provide** — and the first where the comment is not merely unenforced but factually describes a
|
||||||
|
> different function. It survived because it reads as settled and because its *effect* (a stale escrow)
|
||||||
|
> is real, so nobody checked the *cause*.
|
||||||
|
|
||||||
|
**Consequence, and it is a live defect, not a documentation nit.** On the ordinary Re-issue shape —
|
||||||
|
a consumed-but-failed install on a box that still has its `repo_password` file — the box re-applies,
|
||||||
|
`WriteOffboxSecrets` finds the file present and **keeps it**, and the repo password is unchanged. The
|
||||||
|
hub has nonetheless told the customer, in Hungarian, that their recovery escrow is stale and asked
|
||||||
|
them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** → **R-196**.
|
||||||
|
|
||||||
|
**Corroboration against what actually happened on demo-hp yesterday.** The escrow went `pending` after
|
||||||
|
the Re-issue and the task asks: if the repo password was untouched, what explains that? **It is fully
|
||||||
|
explained, and not by the Re-issue.** Two independent mechanisms, both hub-side-invisible:
|
||||||
|
|
||||||
|
1. `ApplyOffsiteTarget` (`offbox.go:482-494`) carries `EscrowState` over from the *existing* target —
|
||||||
|
but the rebuilt guest had **no existing target**, so `cur` was nil, so the fallthrough
|
||||||
|
`if tgt.EscrowState != "escrowed" { tgt.EscrowState = "pending" }` fired.
|
||||||
|
2. The box had minted a **new** repo password moments earlier (Q2), so `pending` was *correct* — the
|
||||||
|
new key genuinely was not escrowed.
|
||||||
|
|
||||||
|
**The Re-issue's `escrow_stale` event fired at 07:11:51 for the wrong reason and happened to be true
|
||||||
|
for a different one.** Coincidence, not correctness. The proof that these are independent is
|
||||||
|
**demo-felhom**: it received **no Re-issue at all** (measured — its only `escrow_stale` /
|
||||||
|
`offsite_reissued` events in the entire history are dated `2026-07-21 08:29:29`), its repo password
|
||||||
|
changed anyway, and **nothing marked its escrow stale for thirteen hours.**
|
||||||
|
|
||||||
|
> **The mechanism is exactly inverted.** `escrow_stale` is wired to the one path that does **not**
|
||||||
|
> change the repo password, and absent from the path that **does**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q5 — Why did one box recover itself and the other not?
|
||||||
|
|
||||||
|
### **RULING: confirmed from the ledger — an unconsumed staged secret, thirteen days old. And the recovery was partial in a way the narrative missed.**
|
||||||
|
|
||||||
|
**Method — I4, the `one_time_secrets` ledger, no values read:**
|
||||||
|
|
||||||
|
| customer | `created_at` | `consumed_at` |
|
||||||
|
|---|---|---|
|
||||||
|
| `demo-felhom` | `2026-07-21 08:29:29` | `2026-08-03 07:17:58` |
|
||||||
|
| `demo-hp` | `2026-08-04 07:11:51` | `2026-08-04 07:12:06` |
|
||||||
|
|
||||||
|
demo-felhom's row was **created 2026-07-21 and sat UNCONSUMED for 13 days** — traceable to a Re-issue
|
||||||
|
at that exact timestamp (its `offsite_reissued` + `escrow_stale` event pair, `2026-07-21 08:29:29`).
|
||||||
|
When the rebuild dropped its target on 2026-08-03, the fresh controller consumed the spare at 07:17:58
|
||||||
|
and was reporting `offsite` again by 07:19:10. **76 seconds, on a credential nobody had planned to
|
||||||
|
leave there.** Confirmed from the ledger, not the narrative.
|
||||||
|
|
||||||
|
**What would have happened without one:** exactly demo-hp — `consume` → **404** (`api/offsite.go:20-22`,
|
||||||
|
`no unconsumed offsite password`), the apply-bridge returns
|
||||||
|
`offsite-apply: consume one-time password: …` and retries forever, and the tier stays absent until an
|
||||||
|
operator acts. demo-hp sat there **25 hours** (`2026-08-03 06:12:19` → `2026-08-04 07:12:01`).
|
||||||
|
|
||||||
|
**The correction to the contrast (see Q2):** demo-felhom's spare restored *delivery* only. Its 36
|
||||||
|
snapshots / 1.14 GB were orphaned by the same rebuild. **Neither box's data survived; one box's
|
||||||
|
plumbing did.**
|
||||||
|
|
||||||
|
*Not established:* demo-felhom's escrow generation before 2026-07-21. `host_escrow_superseded` holds
|
||||||
|
only two rows in the whole database (ids 3 and 4, one per box, both superseded 2026-08-04), so whether
|
||||||
|
the 2026-07-21 Re-issue's ceremony re-sealed an *unchanged* password — the predicted false-staleness
|
||||||
|
shape — **cannot be shown from this data**. Stated as unknown rather than asserted.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q6 — Why is delivery one-shot at all?
|
||||||
|
|
||||||
|
### **RULING: the design's reason is that ① is a LIVE PROVIDER PASSWORD, not a bootstrap token; and the honest answer to the security question is that an automatic restage weakens EXPOSURE WINDOW, not authentication.**
|
||||||
|
|
||||||
|
**The design's own reasons, from source:**
|
||||||
|
|
||||||
|
- `api/offsite.go:9-12` — *"serves the one-time transient offsite password to the controller **EXACTLY
|
||||||
|
ONCE** … The value is returned once then marked consumed — a second call 404s. NEVER logged."*
|
||||||
|
- `offsite.go:145-149` — *"the **EXPLICIT** operator recovery for a consumed-password dead-end … It is
|
||||||
|
**NOT** implicit rotation: ProvisionOffsite never calls this."*
|
||||||
|
- `offsiteapply.go:1-6` — the apply-bridge is *"idempotent (a descriptor hash marker prevents
|
||||||
|
**re-consuming a spent password**)"*, i.e. the design treats a wasted consume as a real hazard, which
|
||||||
|
R-71a's whole settle-gate exists to prevent.
|
||||||
|
|
||||||
|
**The security half, stated rather than skipped.** The question is: *could a machine that is not the
|
||||||
|
real one obtain a credential via an automatic restage?*
|
||||||
|
|
||||||
|
**What stands there today is authentication, and a restage does not remove it.** The consume endpoint
|
||||||
|
is gated by `checkAuthCustomer` (`api/offsite.go:14-18`) — **the customer's API key, the same
|
||||||
|
credential as the config pull**. Anything that could consume a restaged password can already pull the
|
||||||
|
customer's whole config. So an automatic restage **grants no new access to an unauthenticated party**.
|
||||||
|
|
||||||
|
**What it genuinely weakens is the exposure window.** Today a live Hetzner sub-account password is
|
||||||
|
fetchable only in the seconds after an operator deliberately staged one. Under an automatic restage it
|
||||||
|
becomes fetchable **whenever the box reports no offsite target** — a condition an attacker holding the
|
||||||
|
API key can arguably induce. The credential's blast radius at the provider is the customer's own
|
||||||
|
sub-account (label-scoped; `ReissueCredentials` refuses unless the label lookup finds **exactly one**,
|
||||||
|
`offsite.go:164,180`), so this is a widening of *when*, not of *what*.
|
||||||
|
|
||||||
|
**And one further guard must survive any change** — R-39(a), `monitor/offsite_delivery.go:26-30`:
|
||||||
|
`SaveOneTimeSecret` is last-write-wins, so restaging on top of an **unconsumed** secret clobbers a
|
||||||
|
password a box may be about to consume. Any automatic restage must re-read the row immediately before
|
||||||
|
acting and refuse unless it is still a **consumed** row. That guard is correct and is not what R-192 is
|
||||||
|
about.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q7 — R-192's guard: exactly what it reads, and why it read wrong
|
||||||
|
|
||||||
|
### **RULING: both halves confirmed, with numbers. The guard reads the OLDEST 500 reports since the consume, and for demo-hp all 500 predated the rebuild by six days.**
|
||||||
|
|
||||||
|
**The query, quoted** (`hub/internal/store/store.go:987`):
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT report_json FROM reports WHERE customer_id = ? AND received_at > ? ORDER BY id LIMIT 500
|
||||||
|
```
|
||||||
|
|
||||||
|
`ORDER BY id` ascending with `LIMIT 500` = **the oldest 500**, not the newest.
|
||||||
|
|
||||||
|
**Reproduced against the live data (I1)**, with demo-hp's real consume anchor `2026-07-23 09:53:41`
|
||||||
|
(recovered from R-192's record — it is no longer in `one_time_secrets`, see I4):
|
||||||
|
|
||||||
|
| measurement | value |
|
||||||
|
|---|---|
|
||||||
|
| `total` (what the guard sees) | **500** |
|
||||||
|
| `withOffsite` (what the guard sees) | **500** |
|
||||||
|
| oldest report in that window | `2026-07-23 09:53:47` |
|
||||||
|
| newest report in that window | **`2026-07-28 11:17:40`** |
|
||||||
|
| true `total` over the same window, no LIMIT | 1174 |
|
||||||
|
| true `withOffsite` over the same window, no LIMIT | 1063 (⇒ 111 without — matching the 111 ABSENT reports in I3) |
|
||||||
|
|
||||||
|
**The whole 500-report evidence set ends 2026-07-28 — six days before the 2026-08-03 rebuild.**
|
||||||
|
|
||||||
|
**Defect (a) — the message is exactly inverted.** `maybeEmitStuck`
|
||||||
|
(`monitor/offsite_delivery.go:~110`) interpolates `status.ReportsSinceConsume` into a string that
|
||||||
|
hardcodes the phrase *"carry no offbox target"*, and never reads `OffsiteReportsSinceConsume`. The
|
||||||
|
live e-mail, quoted in full from the events table:
|
||||||
|
|
||||||
|
> *"Offsite delivery stuck: one-time password consumed 284h19m0s ago and **500 report(s) since carry
|
||||||
|
> no offbox target** — the credential is likely burned (apply died between consume and persist).
|
||||||
|
> Re-issue delivers a fresh one."*
|
||||||
|
|
||||||
|
`OffsiteReportsSinceConsume` for that same call was **500**. Every one of them carried a target. The
|
||||||
|
message states the precise negation of its own measurement, and prescribes a remedy for a failure mode
|
||||||
|
that did not occur.
|
||||||
|
|
||||||
|
**Defect (b) — the heal refuses silently, on stale evidence.** `maybeHeal` returns bare on
|
||||||
|
`status.OffsiteReportsSinceConsume != 0` (it was 500), with **no log line**. `offsite_credential_restaged`
|
||||||
|
has never fired for any customer — confirmed: zero rows of that event type in the database.
|
||||||
|
|
||||||
|
**What the guard SHOULD discriminate on — the shape, not the code.** The guard's intent is right:
|
||||||
|
*"there is offbox evidence, so this is a regressed apply, not a burned credential — the operator's
|
||||||
|
call."* Its error is that **it asks the question of the wrong time period**. Offbox evidence from
|
||||||
|
before a rebuild is not evidence that the credential still works. The discriminator must be
|
||||||
|
**recency-bounded and rebuild-aware**: judge on evidence *after the newest `controller_started` /
|
||||||
|
config-hash change*, or on the latest N reports — never on "everything since the consume". And a
|
||||||
|
refusal must **name its reason in the log**, because "we chose not to heal" and "the heal never ran"
|
||||||
|
must not look identical (the S-16 rule, one layer down).
|
||||||
|
|
||||||
|
**Do not fix it in isolation.** Under Q2's finding, a successful auto-restage on demo-hp would have
|
||||||
|
restored the transport and the box would *still* have minted a new repo password. **The heal cannot
|
||||||
|
protect the data; it can only protect the plumbing.** Whatever shape it takes must say so.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q8 — What each candidate would cost
|
||||||
|
|
||||||
|
**A fourth option exists and is the important one**, because (a) and (b) both address ① and Q2 shows
|
||||||
|
the damage is in ③.
|
||||||
|
|
||||||
|
### (a) The hub restages automatically when a re-enrolled box reports no offsite
|
||||||
|
|
||||||
|
**A version of this ALREADY EXISTS and is wired to the wrong event.** `reissueOnReenroll`
|
||||||
|
(`hub/internal/api/handler.go:1051-1084`), leg **F3**:
|
||||||
|
|
||||||
|
> *"F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
|
||||||
|
> one-time password only ever reached the OLD controller)."*
|
||||||
|
|
||||||
|
It is called from `handleHostEnroll` — but **after** a mint-once-reuse short-circuit
|
||||||
|
(`handler.go:1004-1016`): `if existing != nil { …return… }`. **A guest rebuild leaves the `hosts` row
|
||||||
|
intact** (measured: `demo-hp-bb76ea` is the same host_id throughout the incident; only the agent leaf
|
||||||
|
re-keyed, `host_leaf_changed` 2026-08-03 06:09:40), so re-enroll returns the existing credential and
|
||||||
|
**F3 is never reached.**
|
||||||
|
|
||||||
|
- **Changes:** hub only. Either lower the trigger from host-enrollment to a controller-level rebuild
|
||||||
|
signal (the `config_hash` change + `controller_started` pair the hub already receives and already
|
||||||
|
logs), or fix R-192's guard so the existing R-71c self-heal can act.
|
||||||
|
- **Weakens:** the exposure window in Q6. Needs the R-39(a) unconsumed-secret guard kept intact.
|
||||||
|
- **Does NOT solve:** ③. The box still mints a new repo password and still orphans the repo. **This
|
||||||
|
option restores the plumbing to a box whose data key is already gone.**
|
||||||
|
- **Cost:** small. **Value against the actual harm: near zero.**
|
||||||
|
|
||||||
|
### (b) The credential becomes recoverable from escrow at re-bootstrap
|
||||||
|
|
||||||
|
> **ANNOTATED 2026-08-04 (recon) — this ruling is overturned in part. See the annotation at the top.**
|
||||||
|
> The constraint is real but narrower than written: an **unattended** rebuild cannot recover from a
|
||||||
|
> zero-knowledge escrow; a **customer-present** one can, and the operator has ruled on that shape.
|
||||||
|
> The recon also found that the chain this option would use is clientless at three links and lossy at
|
||||||
|
> a fourth (`RECON-offsite-dr-chain-2026-08-04.md` §3–§4).
|
||||||
|
|
||||||
|
- **BLOCKED AS STATED, and this is a hard constraint, not an estimate.** The escrow blob is
|
||||||
|
**R-wrapped and zero-knowledge** — the hub holds opaque bytes and has no recovery code
|
||||||
|
(`api/handler.go`, and D6 in `CONTEXT.md` S-3). **A rebuilt box cannot unseal it without the
|
||||||
|
customer's recovery code.** Any design that says "recover it at re-bootstrap" is describing a
|
||||||
|
customer-present ceremony, i.e. what the existing manual `offboxInjectPasswordHandler` already is.
|
||||||
|
- **Does NOT solve:** an unattended rebuild, which is the whole scenario.
|
||||||
|
- **Cost: not implementable as specified.**
|
||||||
|
|
||||||
|
### (c) — NOT PREVIOUSLY NAMED — the agent retains the repo password across a guest rebuild
|
||||||
|
|
||||||
|
The agent lives **on the Proxmox host** and survives a guest rebuild. It already receives the repo
|
||||||
|
password today: the controller pushes it over the pinned local API
|
||||||
|
(`POST /escrow/stage-secret` → `agent/internal/localapi/escrow_stage.go`), and the agent writes it to
|
||||||
|
the fixed 0600 path `/var/lib/felhom-agent/escrow-stage/restic_repo_password`
|
||||||
|
(`agent/internal/escrow/identity.go:44-46`) **transiently — wiped by the ceremony**
|
||||||
|
(`WipeStagedResticPassword`). The seam, the transport, the pinning and the file are all already built.
|
||||||
|
|
||||||
|
- **Changes:** agent (retain rather than wipe; serve back over the same pinned local API) + controller
|
||||||
|
(on a fresh data dir, ask the agent before minting — the `InjectOffboxPassword` seam already exists
|
||||||
|
and is exactly the right shape).
|
||||||
|
- **Weakens:** the escrow's *zero-knowledge* posture in one specific way — the irreplaceable data key
|
||||||
|
would now sit at rest on the Proxmox host, which the operator has root on. **Measured against D6,
|
||||||
|
that is not a new exposure in principle** (D6 already states plainly that *"the operator cannot read
|
||||||
|
customer data"* was never the security property, and the operator holds root on every box) — but it
|
||||||
|
IS a new copy, and copies are the thing R-133 is already open about for the hub DB. **This is a real
|
||||||
|
trade and it is the operator's to make.**
|
||||||
|
- **Solves:** exactly the failure that occurred, unattended, on every rebuild, without any hub
|
||||||
|
involvement and without any credential leaving the premises.
|
||||||
|
- **Cost:** medium. Two repos, one existing seam each.
|
||||||
|
|
||||||
|
### (d) Do neither — keep the manual Re-issue, and fix what LIES
|
||||||
|
|
||||||
|
- **Changes:** R-192's message + refusal log; correct the three false comments about what Re-issue
|
||||||
|
rotates; and — the part that actually matters — **make a repo-password change VISIBLE**. The hub
|
||||||
|
already receives `restic_pw_sha256` on every escrow upload and already stores it. Comparing the new
|
||||||
|
hash against the superseded one is a two-line verdict, and it is the signal that was missing on
|
||||||
|
demo-felhom for thirteen hours → **R-197**. (Note the shape: **this spike answered its own hardest
|
||||||
|
question with a comparison the hub could be making automatically and is not.** Both values were
|
||||||
|
already in the database; nothing read them.)
|
||||||
|
- **Weakens:** nothing.
|
||||||
|
- **Does NOT solve:** the data loss. Every rebuild still orphans the repo; the operator simply finds
|
||||||
|
out the same day instead of never.
|
||||||
|
- **Cost:** small.
|
||||||
|
|
||||||
|
### Ranked, with a recommendation
|
||||||
|
|
||||||
|
| rank | option | why |
|
||||||
|
|---|---|---|
|
||||||
|
| **1** | **(d) now — the honesty pass** | It is cheap, it weakens nothing, and **it is the only option that helps the incident already in flight.** The alarm that lies is worse than no alarm; the repo-password-changed detector is the one signal whose absence let demo-felhom lose 1.14 GB of history silently. Ship this regardless of what is decided about the rest. |
|
||||||
|
| **2** | **(c) — agent-retained repo password** | The only candidate that addresses ③, which is where the harm is. Every seam it needs exists. It costs one deliberate trade-off the operator must actually make. |
|
||||||
|
| **3** | **(a) — auto-restage** | Worth doing **after** (c), never instead of it. On its own it restores the transport to a box whose data key is already gone, and would have made both boxes look healthy on 2026-08-04 while the snapshots were orphaned — a *strictly worse* outcome than the current loud failure. |
|
||||||
|
| **4** | **(b) — recover from escrow at re-bootstrap** | Not implementable as stated; the escrow is zero-knowledge by design. |
|
||||||
|
|
||||||
|
**Recommendation: ship (d), then decide (c).** And do not ship (a) first — it would have hidden this.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What could not be established
|
||||||
|
|
||||||
|
Named rather than guessed (§8 rule 4).
|
||||||
|
|
||||||
|
1. **The 2026-08-05 02:15 UTC run's actual outcome** (Q3). Predicted `ErrOffboxOrphaned` from source +
|
||||||
|
claim state; **not measured**.
|
||||||
|
2. **Whether the orphaned ciphertext still exists at the provider.** 40.9 MB (demo-hp) + 1.14 GB
|
||||||
|
(demo-felhom) are *presumed* present at `/home/felhom-repo` — reading the endpoint needs the SFTP
|
||||||
|
credential, which this session did not touch. The Hetzner API has no directory-listing surface.
|
||||||
|
3. **Whether demo-felhom's 2026-07-21 Re-issue re-sealed an unchanged password** (the predicted
|
||||||
|
false-staleness shape, Q4/Q5). Only two `host_escrow_superseded` rows exist in the whole database;
|
||||||
|
the generation before 2026-07-21 is not retained.
|
||||||
|
4. **What removed demo-hp's offsite target at the rebuild** is fully explained (fresh data dir), but
|
||||||
|
**why the guest was rebuilt on both boxes on consecutive mornings** is outside this spike.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The operator's question — stated, and NOT answered here
|
||||||
|
|
||||||
|
> **The irreplaceable secret in the offsite tier is the restic repository password. It is generated on
|
||||||
|
> the box, it is destroyed by a guest rebuild, and the only copy that survives is sealed under the
|
||||||
|
> customer's recovery code — which nothing but a human can open. Every machine is going to be
|
||||||
|
> reinstalled.**
|
||||||
|
>
|
||||||
|
> **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host, where
|
||||||
|
> you already have root — or do you want it to stay sealed under the customer's recovery code and
|
||||||
|
> accept that every rebuild starts a new repository and orphans the old one, provided the system says
|
||||||
|
> so loudly on the day it happens?**
|
||||||
|
|
||||||
|
Option (c) is the first. Option (d) alone is the second. **This spike does not choose.**
|
||||||
File diff suppressed because one or more lines are too long
@@ -26,7 +26,7 @@
|
|||||||
| R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 |
|
| R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 |
|
||||||
| R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 |
|
| R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 |
|
||||||
| R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/<customer>`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 |
|
| R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/<customer>`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 |
|
||||||
| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path |
|
| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path. **SCOPE-CORRECTED 2026-08-04 (R-193 spike, `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`): "auto-fire on re-enrollment" means HOST re-enrollment, and only when the host record was DELETED first.** `reissueOnReenroll` sits behind `handleHostEnroll`'s mint-once-reuse short-circuit (`if existing != nil { return }`, `api/handler.go:1004-1016`), so a **guest** rebuild — which leaves the `hosts` row intact — never reaches F2 or F3. Measured live: demo-hp's guest was rebuilt 2026-08-03, `demo-hp-bb76ea` was unchanged throughout, F3 did not fire, and the box had **no off-site copy for 25 hours** until an operator ran Re-issue by hand. Peti's clean-slate (a real host delete + re-enroll) still walks the supported path; **the arc does not cover a guest rebuild, and that is the gap R-193 owns.** Note also that F3 alone would NOT have saved the data — the rebuilt controller mints a fresh restic repo password regardless (spike Q2) |
|
||||||
| R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
|
| R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
|
||||||
| R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |
|
| R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |
|
||||||
| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC `p=none` → `p=quarantine` | S | **DONE 2026-07-21 (all three halves)** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu half CLOSED — operator-verified 2026-07-21**: a `[Felhom] Teszt értesítés` was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by `p=quarantine`, **R-4 is COMPLETE** |
|
| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC `p=none` → `p=quarantine` | S | **DONE 2026-07-21 (all three halves)** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu half CLOSED — operator-verified 2026-07-21**: a `[Felhom] Teszt értesítés` was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by `p=quarantine`, **R-4 is COMPLETE** |
|
||||||
@@ -57,6 +57,17 @@
|
|||||||
| R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. **Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17):** the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above |
|
| R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. **Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17):** the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above |
|
||||||
| R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/<id>` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED + PROVEN-LIVE agent v0.89.0** (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → `converged state=adopted` in ~3 s, ACLs restored, offsite active). Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. |
|
| R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/<id>` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED + PROVEN-LIVE agent v0.89.0** (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → `converged state=adopted` in ~3 s, ACLs restored, offsite active). Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. |
|
||||||
| R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** |
|
| R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** |
|
||||||
|
| R-198 | **The superseded-escrow retention kept the K-escrow and dropped the identity blob** — so it preserved the PBS key and destroyed the offsite restic repository password, with the escrow CEREMONY as the destroying act | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | Corrects the escrow-ceremony capability-map row's v0.60.0 retention claim (was struck through 2026-08-04, now restated as what ships). One column + one SELECT list, in the shared `demoteCurrentEscrowTx`, so both callers are fixed at once. Nothing backfillable. **Load-bearing, not tidy:** with R-193(c) refused, customer-present recovery is the only route back from a rebuild and it runs entirely through the retained identity blob |
|
||||||
|
| R-197 | **The hub stored both halves of "did this box's offsite data key change" and compared them nowhere** | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | New `offsite_repo_key_changed` (warning, operator-only, edge-triggered, no hash values). Flips nothing in the capability map — it adds a signal, not a capability |
|
||||||
|
| R-196 | **Five comments claimed `ReissueCredentials` rotates the restic repo password** — it resets the PROVIDER password and cannot touch the repo password | S | **CLOSED — behaviour shipped hub v0.95.0 (2026-08-05)** | The false-staleness alarm is a behaviour change and must not ride a comment-correction release — and it matters MORE now, because under R-198 an unnecessary ceremony supersedes a blob rather than being harmless bookkeeping **Collapsed 2026-08-05.** The comment correction landed in v0.93.0; the BEHAVIOUR landed in v0.95.0 with R-204 item 2 — the precautionary `MarkEscrowStale` and its `escrow_stale` event are gone from `ReissueCredentials`. Removing a signal was safe because the real case is measured twice over, and the mark was **blinding** one of them: a stale flag empties `restic_pw_sha256` from the ACK, which is what the controller's per-ACK re-check compares against. |
|
||||||
|
| R-202 | **The orphan card promises recoverability unconditionally**, which after R-198 is true going forward and false for anything already orphaned | S | **gate hit 2026-08-04 — card untouched, sentence still live** | Blocked on knowing which escrow generation an orphaned repo belongs to (R-199/R-201). A single ACK boolean can say a retained recoverable blob EXISTS but not that one COVERS this repo; a conditional promise that can still be false is worse on that surface than a hedged one |
|
||||||
|
| R-199 | **The hub served recovery blobs on endpoints with no client anywhere** — the DR capstone's retrieval had never been called by the agent, the hub UI, a script or a runbook | M | **SHIPPED + PROVEN-LIVE (hub v0.94.0 + agent v0.125.0, 2026-08-04)** | Adds the capability-map row *"the offsite repository password can be RECOVERED from the sealed escrow"* as PROVEN-LIVE — for the KEY only, explicitly not for a restore. Chain links 6–8 walked on demo-felhom: recovered sha256 == on-disk sha256 == the hub's stored hash. Carries a deliberate security trade (a box's own credential now suffices where an operator-armed recovery mode was required), recorded in the handler and overridable by one constant |
|
||||||
|
| R-200 | **The password-injection seam had a handler and no form** | M | **plumbing SHIPPED (controller v0.195.0); the form is NOT built** | `--recover-offsite-check` compares by hash and refuses to install. The customer-facing shape (yell → recovery-code form → preview → proceed) is priced against a chain that now exists rather than one that was assumed |
|
||||||
|
| R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN |
|
||||||
|
| R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 |
|
||||||
|
| R-203 | **The app and its backup looked in different directories, and a run that skipped a mandatory folder still said `ok`** | M | **SHIPPED + PROVEN-LIVE (controller v0.197.0, 2026-08-04)** | Adds the capability-map row *"off-site app-data capture covers MANDATORY paths on both drive layouts"* as PROVEN-LIVE, and corrects that row's predecessor, which was optimistic. Unblocks R-201 |
|
||||||
|
| R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **ITEMS 1–3 SHIPPED (controller v0.198.0 + hub v0.95.0, 2026-08-05); ITEM 4 OPEN → R-193** | Collapsed per the coupling rule. The fix taken was R-196's — remove the guess, keep the two MEASURED signals (the controller's per-ACK hash re-check, which the stale flag was blinding, and R-197's supersession detector). Plus the claim-code read-through and the restore-scope wording. **Flips the capability-map recovery row from four crutches to one**, and that one is R-193's operator Re-issue, which keeps the row's *with a person present* qualifier. Does NOT touch R-198's retention, which is still unit-proven and needs the second drill |
|
||||||
|
| R-201 | **The wipe-and-recover drill** | L | **PASSED + PROVEN-LIVE (2026-08-04 night)** | Flips the off-site app-data row to *"a customer's file survives a machine rebuild and comes back"* — PROVEN-LIVE, scoped to demo-hp and to a controller-data rebuild. The journey took four undocumented manual steps (R-204); the capability is real, the customer experience is not yet |
|
||||||
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
|
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
|
||||||
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
|
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
|
||||||
| R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |
|
| R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |
|
||||||
|
|||||||
@@ -431,7 +431,7 @@ label. Filed under E-2.
|
|||||||
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
|
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
|
||||||
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
|
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
|
||||||
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
|
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
|
||||||
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. |
|
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. **CORRECTION 2026-08-03, same day, measured on the box: it DID surface as a 403, exactly as this item predicted — the earlier annotation here said otherwise and was wrong.** demo-felhom's local-api backup jobs 403'd **six times** between 09:24 and 17:34 CEST: `POST /nodes/demo-felhom/vzdump -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.Allocate)`. The hub raised `whole_guest_backup_failed` at the first one (*"retrying with backoff"*) and edge-triggering correctly suppressed the rest, so the operator was told once. It ALSO surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). |
|
||||||
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
|
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
|
||||||
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
|
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
|
||||||
| 8 | The honest **single-drive label**. |
|
| 8 | The honest **single-drive label**. |
|
||||||
|
|||||||
@@ -1,3 +1,237 @@
|
|||||||
|
## v0.95.0 — a re-issue no longer marks a healthy escrow stale (2026-08-05, R-196 / R-204 item 2)
|
||||||
|
|
||||||
|
`ReissueCredentials` called `MarkEscrowStale` on every re-issue that found an escrow row. The mark was
|
||||||
|
**precautionary, not evidential**: it guessed that the box's re-apply MIGHT mint a fresh repository
|
||||||
|
password. On the ordinary re-issue shape — a box that still holds its `<DataDir>/offbox/repo_password`
|
||||||
|
— the password does not change, so it marked a HEALTHY escrow stale. The 2026-08-04 recovery drill
|
||||||
|
(R-201) is what promoted this from a nit to a blocker.
|
||||||
|
|
||||||
|
**What the mark actually did**, because "asks for an unnecessary ceremony" understates it:
|
||||||
|
|
||||||
|
1. `stale_at` set → `GetEscrowStatusForCustomer` WITHHOLDS `restic_pw_sha256` from the report ACK.
|
||||||
|
2. With no hash, the controller's SLICE-3 auto-confirm cannot flip pending→escrowed.
|
||||||
|
3. `OffboxRunnable() = OffboxConfigured() && EscrowState=="escrowed"` → **every off-site backup
|
||||||
|
refused, indefinitely**, on a box whose key was never in doubt.
|
||||||
|
4. The customer is told to re-run the recovery ceremony — which mints a new recovery code and
|
||||||
|
supersedes the sealed blob. During a recovery that is the one act that would have destroyed the key
|
||||||
|
just recovered.
|
||||||
|
|
||||||
|
A precautionary flag that stops the data protection it is guarding is not conservative.
|
||||||
|
|
||||||
|
**Removing it leaves no gap — the case it guessed at is measured elsewhere, and better.**
|
||||||
|
|
||||||
|
- **Continuous, box-side:** the controller compares the ACK's sealed hash against its CURRENT local repo
|
||||||
|
password on every report ACK (`report.EscrowAutoConfirmer.reconcileEscrowed`, the Scenario-F
|
||||||
|
re-check). In the guest-rebuild shape — the only shape where a re-issue is followed by a fresh
|
||||||
|
repository password — that mismatches within one report cycle and raises the stale card plus the
|
||||||
|
„create a new recovery code" CTA. Pinned by the controller's `TestEscrowStale_MismatchWarnsOnceAndFlags`.
|
||||||
|
**And the mark was BLINDING it:** by emptying the hash (step 1) it removed the very value that
|
||||||
|
comparison needs, leaving the box able to report only the hash-LESS reason, which is false.
|
||||||
|
- **Edge-triggered, hub-side:** R-197's `offsite_repo_key_changed` fires on a proven hash difference
|
||||||
|
across a supersession and pages the operator. Pinned by `TestEscrowPut_ChangedRepoKey_RaisesSignal`;
|
||||||
|
red-proved by removing the `maybeEmitRepoKeyChanged` call.
|
||||||
|
|
||||||
|
`offsite_reissued` is unchanged and still always fires — the customer must still learn that the
|
||||||
|
credential moved. Only the stale marking and its `escrow_stale` event are gone.
|
||||||
|
|
||||||
|
**`MarkEscrowStale` is kept, not deleted, and now has no caller** — deliberately. The `stale_at` flag
|
||||||
|
is still live and correct (read by the ACK, the operator card and the PBS-DR view); the right way to
|
||||||
|
set it is a future caller that has MEASURED a key change rather than guessed at one. Its doc now says
|
||||||
|
so, and `TestEscrowStaleMechanism_StillWithholdsAndClears` keeps the mechanism from decaying while
|
||||||
|
nothing writes it.
|
||||||
|
|
||||||
|
**Disagreement recorded** (the R-96 standing rule): the task's Scenario D asked that a real key change
|
||||||
|
also "mark the escrow stale". It must not, and nothing was changed to make it — the hub learns of a
|
||||||
|
real change at the moment a supersession SEALS THE NEW PASSWORD, i.e. when the escrow is freshest.
|
||||||
|
Marking it stale there would ask for a ceremony to fix the ceremony that just ran. The correct
|
||||||
|
consequence at that instant is the operator alarm, which is what R-197 already does.
|
||||||
|
|
||||||
|
`TestReissue_InvalidatesEscrow` is replaced in place by `TestReissue_DoesNotMarkAHealthyEscrowStale`,
|
||||||
|
its exact inverse, so the reversal is visible in `git log -L`. The old test was not wrong about the
|
||||||
|
effect it measured; it was wrong about the cause, and it pinned the defect.
|
||||||
|
|
||||||
|
## v0.94.0 — a box can fetch its own sealed recovery package (2026-08-04, R-199 link 6)
|
||||||
|
|
||||||
|
**Chain link 6 had no client.** The hub has served the identity blob since slice 10D, from
|
||||||
|
`handleReEnroll` / `handleGetRestoreDirective` — gated on operator-armed recovery mode and the global
|
||||||
|
key. Nothing in the agent, the hub UI, any script or any runbook ever called either. The only
|
||||||
|
documented retrieval was `sqlite3 … writefile(…)` by hand on a `kubectl cp`-ed database.
|
||||||
|
|
||||||
|
**`GET /api/v1/hosts/{host_id}/escrow`** — the box-authenticated MIRROR of the PUT that put the blob
|
||||||
|
there. Self-scoped: a per-host key reads only its own; the global key may read any, the same asymmetry
|
||||||
|
the PUT has. A host with no sealed bundle gets `200 {present:false}` — a clean answer, because a 404
|
||||||
|
is indistinguishable from an unknown host and a bare empty 200 from a zero-length blob, and neither
|
||||||
|
of those is what "no ceremony has run yet" means.
|
||||||
|
|
||||||
|
**THE TRADE, RECORDED IN THE HANDLER RATHER THAN INFERRED.** Before this, obtaining the blob required
|
||||||
|
the OPERATOR to arm recovery mode. Now whoever controls a rebuilt box can obtain it with that box's
|
||||||
|
own credential. That is a real reduction in the number of parties required, and it is the whole of the
|
||||||
|
trade — they still cannot open it, because the hub has never held R and a wrong code fails closed at
|
||||||
|
age's scrypt KDF. **The mitigation is that the capability is audited rather than silent:** every
|
||||||
|
successful retrieval raises `escrow_blob_served` (warning, operator-only), recorded before the bytes
|
||||||
|
leave. A silent capability on this object is the shape the last two weeks were spent removing.
|
||||||
|
|
||||||
|
`escrowSelfServiceRetrieval` is a single named constant — the §8.2/§8.3 decision point. Flipping it to
|
||||||
|
false re-imposes the recovery-mode requirement on this endpoint and changes nothing else, so the
|
||||||
|
operator can overrule the trade at the cost of a boolean rather than a redesign.
|
||||||
|
|
||||||
|
**The operator-driven DR path is untouched** — same gate, same behaviour, pinned by a test that
|
||||||
|
exercises re-enroll and restore-directive with recovery mode off and on. Red-proofs observed: removing
|
||||||
|
the ownership check makes a cross-host read succeed (host A served host B's blob); removing the audit
|
||||||
|
record makes the retrieval silent.
|
||||||
|
|
||||||
|
**Not in this release:** the customer-facing flow. No card, no form, no preview — those are designed on
|
||||||
|
ground that has been walked, and R-200/R-201 are that walk.
|
||||||
|
|
||||||
|
## v0.93.0 — the retention keeps the key it was built to keep, and three things stop lying (2026-08-04, R-198/R-197/R-196/R-192)
|
||||||
|
|
||||||
|
### R-198 — the superseded-escrow retention was preserving the wrong key, and the ceremony was destroying the right one
|
||||||
|
|
||||||
|
`host_escrow_superseded` shipped in v0.60.0 with a `blob` column — the **K-escrow**, i.e. the PBS
|
||||||
|
datastore encryption key. `identity_blob` was added to `host_escrow` **later**, by the slice-10D
|
||||||
|
`ALTER TABLE`, and **never to the retained table**. The offsite **restic repository password** — the
|
||||||
|
data key for every off-site app-data snapshot — lives inside `identity_blob`. So
|
||||||
|
`demoteCurrentEscrowTx`, whose own comment calls it *"THE ONE escrow row-copy routine"*, retained the
|
||||||
|
whole-guest key and silently dropped the off-site data key: **precisely the secret the reinstall-orphan
|
||||||
|
incident the feature was built for was about.**
|
||||||
|
|
||||||
|
**What makes it the most serious defect this project has recorded is not the missing column — it is
|
||||||
|
which act destroys the key.** The copy happens as the new blob overwrites the old, so the destroying
|
||||||
|
act is the **escrow ceremony**: the exact thing a rebuilt box tells its customer to run, on a card
|
||||||
|
that promises in Hungarian that the old backups stay recoverable with their recovery code. Both demo
|
||||||
|
boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08), and their previous repository passwords
|
||||||
|
are now unrecoverable whether or not anyone kept the recovery codes.
|
||||||
|
|
||||||
|
- `host_escrow_superseded` gains `identity_blob BLOB` (in the CREATE for fresh databases **and** as an
|
||||||
|
additive `ALTER TABLE`, following the existing idiom; tolerated on re-run, changes no existing row).
|
||||||
|
- `demoteCurrentEscrowTx` carries it — so **both** callers are fixed by one change: the re-escrow path
|
||||||
|
(`SaveHostEscrow`) and the host-delete demotion (`DeleteHost`).
|
||||||
|
- `ListSupersededEscrow` reads it back and `store.HostEscrow` gains `IdentityBlob`, so a retained blob
|
||||||
|
is reachable from Go at all. `CountCurrentEscrowWithIdentity` is the census of who the fix protects.
|
||||||
|
- The comment above the table now records that the ruling stated there was not met, and what it cost.
|
||||||
|
- **Nothing is backfillable and the code says so:** rows superseded before this were written without
|
||||||
|
the identity blob and their source rows are already overwritten.
|
||||||
|
|
||||||
|
**Tests assert the CONSEQUENCE, not the mechanism** — the pre-existing
|
||||||
|
`TestSaveHostEscrow_RetainsSuperseded` was green throughout, because it asserted that a retained row
|
||||||
|
exists with the old K-blob. `TestSaveHostEscrow_RetainsIdentityBlob` asserts the retained row can still
|
||||||
|
yield a repository password, and pins the load-bearing ordering (the identity blob is written **after**
|
||||||
|
`SaveHostEscrow`, so the demote sees the previous generation). `TestDeleteHost_DemotesIdentityBlob`
|
||||||
|
proves the shared routine through its **other** caller — a shared routine tested through one caller is
|
||||||
|
how a fix gets believed on a path nobody exercised. Red-proofs: dropping the column from the copy fails
|
||||||
|
both; fixing only the re-escrow caller fails the delete one while the re-escrow one passes.
|
||||||
|
|
||||||
|
### R-197 — the hub held both halves of the evidence and compared them nowhere
|
||||||
|
|
||||||
|
`restic_pw_sha256` is stored on the current escrow and carried to the retained row on every
|
||||||
|
supersession. Comparing the two is what answered the 2026-08-04 recon's hardest question in one query —
|
||||||
|
and nothing in the hub did it. demo-felhom's repository password changed on 2026-08-03, orphaning 36
|
||||||
|
snapshots / 1.14 GB, and **no event, e-mail, card or log line said so for thirteen hours.**
|
||||||
|
|
||||||
|
`SaveHostEscrow` now returns the hash it replaced, and `handleHostEscrowPut` raises
|
||||||
|
**`offsite_repo_key_changed`** (warning, operator-only, edge-triggered once per supersession) when both
|
||||||
|
hashes are known and differ. The in-between shapes are logged rather than dropped, so *"we chose not to
|
||||||
|
alarm"* and *"the check did not run"* never look identical. **No hash value travels** — not in the
|
||||||
|
message, not in the details; the event names the host, the customer and the retained count.
|
||||||
|
|
||||||
|
**Severity chosen for the world this release creates, not the one it inherits:** before R-198 a changed
|
||||||
|
key meant the previous history was unopenable by anyone, ever, which would have argued for `error`.
|
||||||
|
From v0.93.0 the superseding ceremony retains the old identity blob, so the situation is *"this
|
||||||
|
customer's off-site history now depends on an older recovery code"* — operator-actionable, not a loss.
|
||||||
|
`warning` also routes: the dispatcher treats `info` as an intentional non-notify.
|
||||||
|
|
||||||
|
### R-192 (partially) — the alert describes what was measured, and a refusal leaves a trace
|
||||||
|
|
||||||
|
demo-hp e-mailed this daily from 2026-08-03: *"one-time password consumed 284h19m0s ago and 500
|
||||||
|
report(s) since carry no offbox target — the credential is likely burned … Re-issue delivers a fresh
|
||||||
|
one."* `OffsiteReportsSinceConsume` for that same call was **500**: every one of those reports carried a
|
||||||
|
target. The message stated the precise negation of its own measurement and prescribed a remedy for a
|
||||||
|
failure mode that had not occurred — while the self-heal correctly declined, **in complete silence**.
|
||||||
|
|
||||||
|
- The one stuck state is now reported as the two situations it actually covers, **burned** and
|
||||||
|
**regressed**, each stating its own measurement and carrying its own recommendation (the regressed
|
||||||
|
text explicitly withdraws Re-issue and points at what removes an offbox target — a guest rebuild,
|
||||||
|
R-193). `offsite_reports_since_consume` is in the details for the first time.
|
||||||
|
- **The counting window is named in the text.** `CountReportsOffsiteSince` reads `ORDER BY id LIMIT
|
||||||
|
500` — the **oldest** 500 reports after the consume — so on a long-lived box the counts describe the
|
||||||
|
start of the window, not the present. **That guard defect stays OPEN (R-192)** because its correct
|
||||||
|
shape depends on the recovery chain that is not yet assembled (R-199/R-200/R-201); naming the window
|
||||||
|
is how it stays visible instead of being laundered into a confident sentence.
|
||||||
|
- **Every self-heal refusal now leaves a record** — a `notification_log` row on the operator channel
|
||||||
|
with status `refused` and its reason, the R-182 suppressed-e-mail precedent. It rides the stuck
|
||||||
|
event's 24 h cadence, so it sits beside the e-mail it explains rather than accumulating per tick.
|
||||||
|
**The guard's logic is unchanged**: the set of situations in which the heal fires is what it was.
|
||||||
|
- `offsite_delivery_stuck` and `offsite_credential_restaged` are added to `operatorOnlyEvents`. Neither
|
||||||
|
was ever registered, and neither has a `customerMessages` entry — which is **not** a block, so a
|
||||||
|
customer with a configured recipient was in line for an English e-mail about one-time passwords being
|
||||||
|
*"likely burned"*. Narrowing only; the operator channel is untouched.
|
||||||
|
|
||||||
|
### R-196 — five comments corrected (the spec expected three)
|
||||||
|
|
||||||
|
`ReissueCredentials` resets the **provider account** password. It does not and cannot touch the restic
|
||||||
|
repository password, which is generated on the box and never leaves it except sealed under R. Five
|
||||||
|
places in the hub said otherwise: `offsite/offsite.go` (the `MarkEscrowStale` justification),
|
||||||
|
`api/handler.go` (the F3 re-enroll comment), and three in `store/store.go` (the `stale_at` ALTER, the
|
||||||
|
`MarkEscrowStale` doc comment, and `EscrowStatus.Stale`). **That false belief sent two separate
|
||||||
|
investigations the wrong way in one day.** All five now say what the code does, name the correction and
|
||||||
|
cite the recon; the staleness mark is documented as **precautionary**, not evidential. **The behaviour
|
||||||
|
is unchanged and R-196 stays OPEN** — the false-staleness alarm it causes is a behaviour change and
|
||||||
|
must not ride a comment-correction release.
|
||||||
|
|
||||||
|
### Not in this release
|
||||||
|
|
||||||
|
**R-199, R-200 and R-201 remain open** — the hub's blob-serving endpoints still have no client, the
|
||||||
|
password-injection seam still has no form, and nothing in the recovery path has ever been exercised.
|
||||||
|
**Part 5 hit its gate and the orphan card is untouched** — see R-202: a single ACK boolean can say
|
||||||
|
whether a retained recoverable blob exists, but not whether one covers *the key this orphaned
|
||||||
|
repository was written under*, and a conditional promise that is still capable of being false is worse
|
||||||
|
on that surface than a hedged sentence.
|
||||||
|
|
||||||
|
## v0.92.0 — a customer with no machine ever bound does not alarm (2026-08-04, R-195)
|
||||||
|
|
||||||
|
`david` is a real prospective customer whose record was created 2026-08-01 with **no host ever bound**
|
||||||
|
— `hosts=0`, `host_deletions=0`, `host_reports=0`, `reports=0` — and it e-mailed an
|
||||||
|
`expected_dbdump_missed` **error** at 03:00 UTC on 2026-08-02, 08-03 and 08-04. Nothing has ever been
|
||||||
|
expected of that customer, so the honest verdict is UNKNOWN, not missed.
|
||||||
|
|
||||||
|
**The mechanism is the interesting half, because the guard that should have covered this is keyed off
|
||||||
|
the wrong thing.** `CheckBackupDeadlines` already skips nodes that are `down`, and that skip reads
|
||||||
|
`StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — **a query over the
|
||||||
|
`reports` table**. A customer with zero reports is in no row, gets no state at all, and `GetState()`
|
||||||
|
returns `""` rather than `"down"`. **The skip misses exactly the customer it would most obviously
|
||||||
|
cover.** Corroborated on live data: `peti-felhom` is active with a host deleted 2026-07-15 and does
|
||||||
|
*not* alarm, because it has 482 old reports and is therefore `down`. The backup half was already safe
|
||||||
|
(`reportJSON == ""` → skip); the DB-dump half had no guard at all.
|
||||||
|
|
||||||
|
- **`store.HasEverBoundHost(customerID)`** — a live `hosts` row **OR** a `host_deletions` tombstone.
|
||||||
|
Consulted once per customer at the top of the deadline loop; a false verdict skips the customer with
|
||||||
|
an INFO line and its own counter in the summary.
|
||||||
|
- **The discriminator is "was a host EVER bound", NOT "has a report arrived", and that is the whole
|
||||||
|
design.** A box that was installed, bound and then went silent is a real fault and must keep
|
||||||
|
alarming — it has a `hosts` row, so it is judged. Only a customer that never had a machine is
|
||||||
|
UNKNOWN. The tombstone is included for the same reason: a customer whose host was deleted *had* one,
|
||||||
|
and that judgement belongs to the staleness checker, not to this predicate.
|
||||||
|
- **Fail-OPEN on a read error.** An unreadable binding must never SUPPRESS a real alarm; it logs and
|
||||||
|
judges anyway.
|
||||||
|
- **The anchored-verdict structure is untouched** (R-81/v0.73.0). This is the same invariant — absence
|
||||||
|
is UNKNOWN until an anchor elapses — applied one level up, at the question of whether there is a
|
||||||
|
subject at all.
|
||||||
|
|
||||||
|
**Tests, including the one that matters.** `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`
|
||||||
|
is a real shape — a machine installed and never phoned home — and it is indistinguishable from `david`
|
||||||
|
on every signal except the one the guard reads. If the guard is ever "simplified" to key off report
|
||||||
|
presence, customer age or a name pattern, it goes red. Plus a bound-then-went-quiet case, a
|
||||||
|
deleted-host case, and the predicate's own three inputs. **Companion red-proof observed:** removing the
|
||||||
|
guard fails `TestCheckBackupDeadlines_NeverBoundHost_Silent` with `got [expected_dbdump_missed]` —
|
||||||
|
verbatim the event `david` sent three mornings running. Restored after.
|
||||||
|
|
||||||
|
**`david`'s record was not modified.** The record is correct; the alarm was what was wrong.
|
||||||
|
|
||||||
|
**Shipped alongside a spike that deliberately shipped NO code** —
|
||||||
|
`documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md` (R-193/R-192): what is one-shot
|
||||||
|
in the offsite path, and what a guest rebuild really costs. It ends in ranked options and a question
|
||||||
|
for the operator; R-196 and R-197 were minted from it and are not implemented here.
|
||||||
|
|
||||||
## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
|
## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
|
||||||
|
|
||||||
**Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
|
**Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
|
||||||
|
|||||||
@@ -0,0 +1,203 @@
|
|||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-199 (hub v0.94.0) — the box-authenticated retrieval of a host's OWN sealed identity blob.
|
||||||
|
// This is a new surface on the most sensitive object in the system; these tests exist to pin the
|
||||||
|
// three properties that make it defensible: it is self-scoped, it is honest when there is nothing to
|
||||||
|
// serve, and it is never silent.
|
||||||
|
|
||||||
|
func seedEscrowedHost(t *testing.T, st *store.Store, hostID, customerID, apiKey string, identity []byte) {
|
||||||
|
t.Helper()
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: apiKey}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("k-escrow"), "fp", "zero_knowledge", "2026-08-04T11:00:00Z", "SHA"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(identity) > 0 {
|
||||||
|
if err := st.SaveHostDRBundle(hostID, identity, `{}`); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The happy path: a host reads back exactly the bytes it uploaded, verbatim.
|
||||||
|
func TestEscrowGet_ServesOwnBlobVerbatim(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
identity := []byte("\x00\x01age-wrapped-identity-bundle\xff")
|
||||||
|
seedEscrowedHost(t, st, "h1", "c1", "HKEY", identity)
|
||||||
|
|
||||||
|
rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", "")
|
||||||
|
if rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("GET escrow = %d, want 200 (%s)", rr.Code, rr.Body.String())
|
||||||
|
}
|
||||||
|
var out struct {
|
||||||
|
HostID string `json:"host_id"`
|
||||||
|
Present bool `json:"present"`
|
||||||
|
B64 string `json:"identity_escrow_b64"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(rr.Body.Bytes(), &out); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !out.Present || out.HostID != "h1" {
|
||||||
|
t.Fatalf("unexpected envelope: %+v", out)
|
||||||
|
}
|
||||||
|
got, err := base64.StdEncoding.DecodeString(out.B64)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if string(got) != string(identity) {
|
||||||
|
t.Fatal("the served blob is not the stored blob — the hub must return ciphertext verbatim")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario C — a box can fetch ONLY its own blob.
|
||||||
|
// RED-PROOF: delete the `!isGlobal && authHostID != pathHostID` check in handleHostEscrowGet →
|
||||||
|
// cross-host retrieval succeeds → this FAILS. Without that line, any host key is a fleet-wide reader
|
||||||
|
// of every customer's sealed bundle.
|
||||||
|
func TestEscrowGet_CrossHostRefused(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
seedEscrowedHost(t, st, "hostA", "custA", "KEY-A", []byte("A-identity"))
|
||||||
|
seedEscrowedHost(t, st, "hostB", "custB", "KEY-B", []byte("B-identity-SECRET"))
|
||||||
|
|
||||||
|
rr := do(h, http.MethodGet, "/hosts/hostB/escrow", "KEY-A", "")
|
||||||
|
if rr.Code != http.StatusForbidden {
|
||||||
|
t.Fatalf("host A reading host B's blob = %d, want 403 — a cross-host read of a sealed bundle "+
|
||||||
|
"must be impossible on every code path (body: %s)", rr.Code, rr.Body.String())
|
||||||
|
}
|
||||||
|
if b := rr.Body.String(); len(b) > 0 && (contains(b, "B-identity") || contains(b, base64.StdEncoding.EncodeToString([]byte("B-identity-SECRET")))) {
|
||||||
|
t.Fatal("the refusal body leaked the other host's blob")
|
||||||
|
}
|
||||||
|
// Unauthenticated is refused too, and does not leak which hosts exist.
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/hostB/escrow", "", ""); rr.Code != http.StatusUnauthorized {
|
||||||
|
t.Fatalf("unauthenticated GET = %d, want 401", rr.Code)
|
||||||
|
}
|
||||||
|
// The global operator key MAY read any — the same asymmetry the PUT has.
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/hostB/escrow", globalKey, ""); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("global key GET = %d, want 200", rr.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func contains(hay, needle string) bool {
|
||||||
|
return len(needle) > 0 && len(hay) >= len(needle) && (func() bool {
|
||||||
|
for i := 0; i+len(needle) <= len(hay); i++ {
|
||||||
|
if hay[i:i+len(needle)] == needle {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
})()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario D — a host with no sealed bundle gets a CLEAN answer: 200 with present=false. Not a 404
|
||||||
|
// (indistinguishable from an unknown host), not an empty 200 without the flag (indistinguishable
|
||||||
|
// from a zero-length blob). Both of those read as a fault to a caller whose situation is simply
|
||||||
|
// "no ceremony has run yet".
|
||||||
|
func TestEscrowGet_NoBlobIsCleanNone(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
seedEscrowedHost(t, st, "h1", "c1", "HKEY", nil) // K-escrow only, no identity blob
|
||||||
|
|
||||||
|
rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", "")
|
||||||
|
if rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("no-blob GET = %d, want 200 with present=false", rr.Code)
|
||||||
|
}
|
||||||
|
var out struct {
|
||||||
|
Present bool `json:"present"`
|
||||||
|
B64 string `json:"identity_escrow_b64"`
|
||||||
|
}
|
||||||
|
json.Unmarshal(rr.Body.Bytes(), &out)
|
||||||
|
if out.Present || out.B64 != "" {
|
||||||
|
t.Fatalf("a host with no bundle must report present=false and no bytes, got %+v", out)
|
||||||
|
}
|
||||||
|
// An unknown host is a DIFFERENT answer — the two must not collapse into one.
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/nope/escrow", globalKey, ""); rr.Code != http.StatusNotFound {
|
||||||
|
t.Fatalf("unknown host = %d, want 404 (distinct from a known host with no bundle)", rr.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario E — every successful retrieval leaves a durable record naming the host.
|
||||||
|
// RED-PROOF: remove the SaveEvent call in handleHostEscrowGet → no event → this FAILS. A silent
|
||||||
|
// capability on this object is the whole reason §8.2's trade is acceptable at all.
|
||||||
|
func TestEscrowGet_LeavesARecord(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
seedEscrowedHost(t, st, "h1", "c1", "HKEY", []byte("age-wrapped"))
|
||||||
|
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", ""); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("GET = %d", rr.Code)
|
||||||
|
}
|
||||||
|
ev, err := st.GetLatestEventByType("c1", eventEscrowBlobServed)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if ev == nil {
|
||||||
|
t.Fatal("R-199: a sealed recovery blob was served and NOTHING recorded it — the audit row is " +
|
||||||
|
"the mitigation that makes this endpoint's trade acceptable")
|
||||||
|
}
|
||||||
|
if ev.Severity != "warning" {
|
||||||
|
t.Errorf("severity = %q, want warning: info is an intentional non-notify, so the operator would "+
|
||||||
|
"never hear that the recovery path was used", ev.Severity)
|
||||||
|
}
|
||||||
|
if !contains(ev.Message, "h1") {
|
||||||
|
t.Errorf("the record must name the host, got %q", ev.Message)
|
||||||
|
}
|
||||||
|
// The record must not carry the blob itself — it is an audit row, not a second copy.
|
||||||
|
if contains(ev.Message, "age-wrapped") || contains(ev.DetailsJSON, "age-wrapped") ||
|
||||||
|
contains(ev.DetailsJSON, base64.StdEncoding.EncodeToString([]byte("age-wrapped"))) {
|
||||||
|
t.Errorf("the audit record embedded the blob: msg=%q details=%q", ev.Message, ev.DetailsJSON)
|
||||||
|
}
|
||||||
|
// A "no blob" answer is NOT a retrieval and must not raise one.
|
||||||
|
seedEscrowedHost(t, st, "h2", "c2", "KEY2", nil)
|
||||||
|
do(h, http.MethodGet, "/hosts/h2/escrow", "KEY2", "")
|
||||||
|
if ev, _ := st.GetLatestEventByType("c2", eventEscrowBlobServed); ev != nil {
|
||||||
|
t.Fatal("a present=false answer served no blob and must raise no retrieval record")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Operator-tier by construction, registered in the same commit that mints the type.
|
||||||
|
func TestEscrowBlobServed_IsOperatorOnly(t *testing.T) {
|
||||||
|
if !notify.IsOperatorOnly(eventEscrowBlobServed) {
|
||||||
|
t.Fatalf("%s is not registered operator-only — a customer would receive operator-grade English "+
|
||||||
|
"about their sealed recovery bundle being handed out", eventEscrowBlobServed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario G — the operator-driven DR path is UNTOUCHED: same gate, same behaviour. The new sibling
|
||||||
|
// must not loosen it, and must not share it.
|
||||||
|
func TestEscrowGet_OperatorDRPathUnchanged(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
seedEscrowedHost(t, st, "h1", "c1", "HKEY", []byte("age-wrapped"))
|
||||||
|
|
||||||
|
// re-enroll without recovery mode: still refused.
|
||||||
|
if rr := do(h, http.MethodPost, "/hosts/h1/re-enroll", "HKEY", `{"new_api_key":"x"}`); rr.Code != http.StatusForbidden {
|
||||||
|
t.Fatalf("re-enroll without recovery mode = %d, want 403 — the new endpoint must not loosen it", rr.Code)
|
||||||
|
}
|
||||||
|
// restore-directive without recovery mode: still refused.
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/h1/restore-directive", "HKEY", ""); rr.Code != http.StatusForbidden {
|
||||||
|
t.Fatalf("restore-directive without recovery mode = %d, want 403", rr.Code)
|
||||||
|
}
|
||||||
|
// …and arming it still works, i.e. the old path is functional, not merely closed.
|
||||||
|
if err := st.SetRecoveryMode("h1", time.Now().UTC().Add(10*time.Minute)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/h1/restore-directive", "HKEY", ""); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("restore-directive WITH recovery mode = %d, want 200", rr.Code)
|
||||||
|
}
|
||||||
|
// The new endpoint does NOT depend on recovery mode (that is the §8.2 trade, made explicit here so
|
||||||
|
// a future change to escrowSelfServiceRetrieval is visible as a test change).
|
||||||
|
if err := st.ClearRecoveryMode("h1"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", ""); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("self-service escrow GET with recovery mode OFF = %d, want 200 (escrowSelfServiceRetrieval=true)", rr.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,146 @@
|
|||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-197 (v0.93.0) — a changed offsite repository key must be REPORTED. Both halves of the comparison
|
||||||
|
// (host_escrow.restic_pw_sha256 and the row it replaces) have been in the database since SLICE 3 and
|
||||||
|
// nothing read them: demo-felhom's repository password changed on 2026-08-03, orphaning 36 snapshots /
|
||||||
|
// 1.14 GB, and no event, e-mail, card or log line said so for thirteen hours.
|
||||||
|
//
|
||||||
|
// Driven through the real endpoint (PUT /hosts/<id>/escrow), not by calling the emitter, so the wiring
|
||||||
|
// is part of what is proven.
|
||||||
|
|
||||||
|
// escrowBodyWithHash is escrowBody plus the sealed-password hash and an identity blob — the fork-4
|
||||||
|
// upload shape. Hash VALUES here are test fixtures, never real secrets.
|
||||||
|
func escrowBodyWithHash(blob []byte, resticPwSHA, identity string) string {
|
||||||
|
m := map[string]string{
|
||||||
|
"blob_b64": base64.StdEncoding.EncodeToString(blob),
|
||||||
|
"key_fingerprint": "ab:cd:ef",
|
||||||
|
"posture": "zero_knowledge",
|
||||||
|
"created_at": "2026-08-04T05:00:00Z",
|
||||||
|
"restic_pw_sha256": resticPwSHA,
|
||||||
|
}
|
||||||
|
if identity != "" {
|
||||||
|
m["identity_blob_b64"] = base64.StdEncoding.EncodeToString([]byte(identity))
|
||||||
|
}
|
||||||
|
b, _ := json.Marshal(m)
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario D — a new escrow sealing a DIFFERENT repo password raises exactly one operator signal,
|
||||||
|
// naming the host, carrying NO hash value.
|
||||||
|
// RED-PROOF: remove the maybeEmitRepoKeyChanged call from handleHostEscrowPut (or make its predicate
|
||||||
|
// always fall through) → no event → this FAILS.
|
||||||
|
func TestEscrowPut_ChangedRepoKey_RaisesSignal(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||||
|
|
||||||
|
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_GEN1", "age-gen1")); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("first PUT = %d (%s)", rr.Code, rr.Body.String())
|
||||||
|
}
|
||||||
|
// Generation 1 is onboarding, not a change — nothing may fire yet.
|
||||||
|
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
|
||||||
|
t.Fatalf("a FIRST escrow must not raise %s: %+v", eventRepoKeyChanged, ev)
|
||||||
|
}
|
||||||
|
|
||||||
|
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "SHA_GEN2", "age-gen2")); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("second PUT = %d (%s)", rr.Code, rr.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
ev, err := st.GetLatestEventByType("c1", eventRepoKeyChanged)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if ev == nil {
|
||||||
|
t.Fatal("R-197: the repository key demonstrably changed and NO signal was raised — this is the " +
|
||||||
|
"thirteen-hour silence that let demo-felhom lose 1.14 GB unremarked")
|
||||||
|
}
|
||||||
|
if ev.Severity != "warning" {
|
||||||
|
t.Errorf("severity = %q, want warning (info is an intentional non-notify — the operator would not be told)", ev.Severity)
|
||||||
|
}
|
||||||
|
if !strings.Contains(ev.Message, "h1") {
|
||||||
|
t.Errorf("message must name the host, got %q", ev.Message)
|
||||||
|
}
|
||||||
|
// NO HASH VALUE may travel — not in the message, not in the details.
|
||||||
|
for _, forbidden := range []string{"SHA_GEN1", "SHA_GEN2"} {
|
||||||
|
if strings.Contains(ev.Message, forbidden) || strings.Contains(ev.DetailsJSON, forbidden) {
|
||||||
|
t.Errorf("a repo-password hash value leaked into the event (%s): msg=%q details=%q", forbidden, ev.Message, ev.DetailsJSON)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Exactly one, not one per anything.
|
||||||
|
evs, _ := st.GetRecentEvents("c1", 50)
|
||||||
|
n := 0
|
||||||
|
for _, e := range evs {
|
||||||
|
if e.EventType == eventRepoKeyChanged {
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if n != 1 {
|
||||||
|
t.Fatalf("%s fired %d times for one supersession, want 1 (edge-triggered)", eventRepoKeyChanged, n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario E — an UNCHANGED key says nothing. A customer re-running a ceremony for good reasons must
|
||||||
|
// not be punished for it with an alarm.
|
||||||
|
func TestEscrowPut_UnchangedRepoKey_Silent(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||||
|
|
||||||
|
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_SAME", "age-1"))
|
||||||
|
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "SHA_SAME", "age-2"))
|
||||||
|
|
||||||
|
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
|
||||||
|
t.Fatalf("a same-hash re-ceremony must be SILENT, got: %+v", ev)
|
||||||
|
}
|
||||||
|
// And it must not have created a superseded row either (pre-existing contract, re-asserted here
|
||||||
|
// because Scenario E's silence would also be produced by the supersede branch never running).
|
||||||
|
if n, _ := st.CountSupersededEscrow("h1"); n != 0 {
|
||||||
|
t.Fatalf("same-hash re-upload created %d superseded row(s), want 0", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The in-between shapes must not be silent-and-indistinguishable either: a hash-less NEW blob cannot
|
||||||
|
// show whether the key changed, and saying nothing at all would look identical to "unchanged".
|
||||||
|
// Asserted at the level the code offers — no event (correct: nothing was measured) and no crash.
|
||||||
|
func TestEscrowPut_HashlessSupersession_NoSignal(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||||
|
|
||||||
|
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_GEN1", "age-1"))
|
||||||
|
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "", "age-2")); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("hash-less supersession PUT = %d", rr.Code)
|
||||||
|
}
|
||||||
|
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
|
||||||
|
t.Fatalf("a hash-less new blob cannot EVIDENCE a change; it must not claim one: %+v", ev)
|
||||||
|
}
|
||||||
|
// The retention still ran (the hashes differ, so this IS a supersession) — the key is kept even
|
||||||
|
// though the change cannot be evidenced.
|
||||||
|
if n, _ := st.CountSupersededEscrow("h1"); n != 1 {
|
||||||
|
t.Fatalf("superseded rows = %d, want 1", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// R-97a's lesson, pinned in the SAME commit that mints the type: an operator-tier event that is not in
|
||||||
|
// operatorOnlyEvents reaches customers as raw English, because a missing customerMessages entry is NOT
|
||||||
|
// a block. Checked here rather than in notify so the type's two registers are asserted together.
|
||||||
|
func TestRepoKeyChanged_IsOperatorOnly(t *testing.T) {
|
||||||
|
if !notify.IsOperatorOnly(eventRepoKeyChanged) {
|
||||||
|
t.Fatalf("%s is not registered operator-only — a customer would receive operator-grade English "+
|
||||||
|
"about escrow custody", eventRepoKeyChanged)
|
||||||
|
}
|
||||||
|
// The two R-192 types have the same property and the same reason (see the register's comment).
|
||||||
|
for _, et := range []string{"offsite_delivery_stuck", "offsite_credential_restaged"} {
|
||||||
|
if !notify.IsOperatorOnly(et) {
|
||||||
|
t.Errorf("%s is not registered operator-only", et)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -155,7 +155,7 @@ func TestReportACK_EscrowStatus(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// escrow row with identity blob + hash → the ACK carries all three fields
|
// escrow row with identity blob + hash → the ACK carries all three fields
|
||||||
if _, err := st.SaveHostEscrow("hv1", []byte("k-blob"), "fp", "zero_knowledge", "2026-07-09T20:00:00Z", "abc123"); err != nil {
|
if _, _, err := st.SaveHostEscrow("hv1", []byte("k-blob"), "fp", "zero_knowledge", "2026-07-09T20:00:00Z", "abc123"); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := st.SaveHostDRBundle("hv1", []byte("identity-blob"), "{}"); err != nil {
|
if err := st.SaveHostDRBundle("hv1", []byte("identity-blob"), "{}"); err != nil {
|
||||||
|
|||||||
+200
-4
@@ -239,6 +239,12 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
|
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
|
||||||
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
|
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
|
||||||
h.handleHostEscrowPut(w, r, hostID)
|
h.handleHostEscrowPut(w, r, hostID)
|
||||||
|
// R-199 (v0.94.0): the box-authenticated MIRROR of the PUT above — a host reads back its own
|
||||||
|
// opaque identity blob so it can be unsealed with the customer's recovery code. Distinct from the
|
||||||
|
// operator-driven DR path in dr.go, which stays exactly as it is (see handleHostEscrowGet).
|
||||||
|
case r.Method == http.MethodGet && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
|
||||||
|
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
|
||||||
|
h.handleHostEscrowGet(w, r, hostID)
|
||||||
// G1 break-glass: day-0 vaults the root@pam console credential (self-scoped host key); the
|
// G1 break-glass: day-0 vaults the root@pam console credential (self-scoped host key); the
|
||||||
// operator retrieves it via the /admin/ path (global key only).
|
// operator retrieves it via the /admin/ path (global key only).
|
||||||
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/recovery-credential"):
|
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/recovery-credential"):
|
||||||
@@ -1065,9 +1071,13 @@ func (h *Handler) reissueOnReenroll(cc *store.CustomerConfig) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
// F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
|
// F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
|
||||||
// one-time password only ever reached the OLD controller). The re-issuer resets the restic repo
|
// one-time password only ever reached the OLD controller). ⚠ CORRECTED 2026-08-04 (R-196): this
|
||||||
// password, which makes the OLD escrow blob stale — the offsite provisioner invalidates the
|
// used to claim "the re-issuer resets the restic repo password, which makes the OLD escrow blob
|
||||||
// escrow (2.3) and emits both events. Skips silently when offsite isn't provisioned/enabled.
|
// stale". It does not and cannot — the re-issuer resets the PROVIDER account password; the
|
||||||
|
// repository password is generated on the box and never leaves it except sealed under R. The
|
||||||
|
// provisioner does mark the escrow stale, but PRECAUTIONARILY (see the reasoning at
|
||||||
|
// offsite.ReissueCredentials), not because this call rotated anything.
|
||||||
|
// Skips silently when offsite isn't provisioned/enabled.
|
||||||
if h.offsiteReissuer != nil {
|
if h.offsiteReissuer != nil {
|
||||||
if err := h.offsiteReissuer(context.Background(), cc.CustomerID); err != nil {
|
if err := h.offsiteReissuer(context.Background(), cc.CustomerID); err != nil {
|
||||||
h.logger.Printf("[WARN] offsite re-issue on re-enroll for %s failed: %v", cc.CustomerID, err)
|
h.logger.Printf("[WARN] offsite re-issue on re-enroll for %s failed: %v", cc.CustomerID, err)
|
||||||
@@ -1141,7 +1151,7 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
|
|||||||
// Store the OPAQUE bytes. No decrypt path exists — the hub cannot open this. Part B (v0.60.0):
|
// Store the OPAQUE bytes. No decrypt path exists — the hub cannot open this. Part B (v0.60.0):
|
||||||
// when this upload supersedes a DIFFERENT-passphrase old blob, the old one is RETAINED (not
|
// when this upload supersedes a DIFFERENT-passphrase old blob, the old one is RETAINED (not
|
||||||
// overwritten) so its recovery-code-recoverable history survives (Viktor's data-first ruling).
|
// overwritten) so its recovery-code-recoverable history survives (Viktor's data-first ruling).
|
||||||
superseded, serr := h.store.SaveHostEscrow(pathHostID, blob, req.KeyFingerprint, req.Posture, createdAt, req.ResticPwSHA256)
|
superseded, prevPwSHA, serr := h.store.SaveHostEscrow(pathHostID, blob, req.KeyFingerprint, req.Posture, createdAt, req.ResticPwSHA256)
|
||||||
if serr != nil {
|
if serr != nil {
|
||||||
h.logger.Printf("[ERROR] Failed to store escrow for host %s: %v", pathHostID, serr)
|
h.logger.Printf("[ERROR] Failed to store escrow for host %s: %v", pathHostID, serr)
|
||||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||||
@@ -1157,6 +1167,10 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
|
|||||||
"A korábbi helyreállítási csomag megőrizve (új kulcs érkezett).", string(details), "hub"); eerr != nil {
|
"A korábbi helyreállítási csomag megőrizve (új kulcs érkezett).", string(details), "hub"); eerr != nil {
|
||||||
h.logger.Printf("[WARN] escrow_superseded event save failed for %s: %v", pathHostID, eerr)
|
h.logger.Printf("[WARN] escrow_superseded event save failed for %s: %v", pathHostID, eerr)
|
||||||
}
|
}
|
||||||
|
// R-197: the box's offsite DATA key demonstrably changed. Both halves of that comparison
|
||||||
|
// have been stored since SLICE 3 and nothing read them — demo-felhom's key changed on
|
||||||
|
// 2026-08-03 and nothing said so for thirteen hours.
|
||||||
|
h.maybeEmitRepoKeyChanged(host.CustomerID, pathHostID, prevPwSHA, req.ResticPwSHA256, n)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Slice 10D.1: optionally store the IDENTITY escrow blob + the non-secret DR directive alongside
|
// Slice 10D.1: optionally store the IDENTITY escrow blob + the non-secret DR directive alongside
|
||||||
@@ -1185,6 +1199,188 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
|
|||||||
w.Write([]byte(`{"status":"ok"}`))
|
w.Write([]byte(`{"status":"ok"}`))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// eventRepoKeyChanged (R-197) — the box's offsite restic REPOSITORY password changed, proven by the
|
||||||
|
// hub's own stored hashes. Hub-internal (not in allowedEventTypes, like escrow_superseded) and
|
||||||
|
// registered operator-only in notify.operatorOnlyEvents.
|
||||||
|
const eventRepoKeyChanged = "offsite_repo_key_changed"
|
||||||
|
|
||||||
|
// eventEscrowBlobServed (R-199) — a host retrieved its own sealed identity blob. Hub-internal,
|
||||||
|
// operator-only. See handleHostEscrowGet for why every retrieval is loud.
|
||||||
|
const eventEscrowBlobServed = "escrow_blob_served"
|
||||||
|
|
||||||
|
// escrowSelfServiceRetrieval is THE SINGLE DECISION POINT for the §8.2/§8.3 trade (R-199).
|
||||||
|
//
|
||||||
|
// true (§8.2, shipped v0.94.0) — a host may read its own blob whenever it authenticates as itself.
|
||||||
|
// false (§8.3, the fallback) — the same read additionally requires operator-armed recovery mode.
|
||||||
|
//
|
||||||
|
// It is one condition on purpose: the operator may overrule the trade below, and switching must cost a
|
||||||
|
// boolean rather than a redesign. Everything else in the recovery chain is identical either way.
|
||||||
|
const escrowSelfServiceRetrieval = true
|
||||||
|
|
||||||
|
// handleHostEscrowGet serves a host its OWN opaque identity-escrow blob (R-199, v0.94.0).
|
||||||
|
//
|
||||||
|
// WHAT THIS GIVES OUT, WHY IT IS SAFE, AND WHAT IT CHANGES ABOUT WHO IS REQUIRED — recorded here so the
|
||||||
|
// next reader finds the trade rather than inferring it (the dr.go header convention).
|
||||||
|
//
|
||||||
|
// WHAT: the age-wrapped `IdentityBundle` — opaque ciphertext. It carries the offsite restic repository
|
||||||
|
// password, the tunnel token, the PBS token and the WG key. The hub stores these bytes and has no
|
||||||
|
// decrypt path; the recovery code R that opens them exists only in the customer's hands.
|
||||||
|
//
|
||||||
|
// WHY IT IS SAFE TO GIVE OUT: the blob is useless without R (age scrypt + ChaCha20-Poly1305; a wrong R
|
||||||
|
// fails closed at the KDF, never to a plausible-but-wrong bundle), and a 10-word EFF code carries ~129
|
||||||
|
// bits. The caller already authenticates as this host for its report, its desired state, its WG
|
||||||
|
// registration and its PBS token — this adds no new identity, only a new object, and it is the exact
|
||||||
|
// MIRROR of the PUT above, which is how the blob got here in the first place.
|
||||||
|
//
|
||||||
|
// WHAT IT CHANGES, STATED PLAINLY BECAUSE IT IS THE WHOLE OF THE TRADE: before this, obtaining the blob
|
||||||
|
// required the OPERATOR to arm recovery mode with the global key (dr.go). Now whoever controls a
|
||||||
|
// rebuilt box can obtain it with that box's own credential. That is a real reduction in the number of
|
||||||
|
// parties required. They still cannot open it. The mitigation is that the capability is AUDITED rather
|
||||||
|
// than silent: every successful retrieval raises an operator event (below), because a silent capability
|
||||||
|
// is the shape this project has spent two weeks removing.
|
||||||
|
//
|
||||||
|
// THE OPERATOR-DRIVEN DR PATH IS UNTOUCHED. `handleReEnroll` / `handleGetRestoreDirective` keep their
|
||||||
|
// recovery-mode gate and their global-key arming, and they serve the K-escrow and the directive as
|
||||||
|
// well. This endpoint serves ONE object to ONE authenticated owner. Do not merge them.
|
||||||
|
func (h *Handler) handleHostEscrowGet(w http.ResponseWriter, r *http.Request, pathHostID string) {
|
||||||
|
authHostID, _, isGlobal, ok := h.checkAuthHost(r)
|
||||||
|
if !ok {
|
||||||
|
http.Error(w, "Unauthorized", http.StatusUnauthorized)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if pathHostID == "" {
|
||||||
|
http.Error(w, "Missing host_id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// SELF-SCOPED: a per-host key reads only its OWN escrow. The global operator key may read any —
|
||||||
|
// the same asymmetry the PUT has. Without this line any host key is a fleet-wide blob reader.
|
||||||
|
if !isGlobal && authHostID != pathHostID {
|
||||||
|
h.logger.Printf("[WARN] escrow GET REFUSED: host %s asked for %s's blob (self-scope)", authHostID, pathHostID)
|
||||||
|
http.Error(w, "Forbidden: host_id mismatch", http.StatusForbidden)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
host, err := h.store.GetHost(pathHostID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if host == nil {
|
||||||
|
http.Error(w, "Unknown host_id", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// §8.3 fallback lives here and nowhere else.
|
||||||
|
if !escrowSelfServiceRetrieval && !host.InRecoveryMode(time.Now().UTC()) {
|
||||||
|
h.logger.Printf("[WARN] escrow GET REFUSED for %s — self-service retrieval is disabled and recovery mode is not armed", pathHostID)
|
||||||
|
http.Error(w, "Forbidden: host not in recovery mode (operator must arm it)", http.StatusForbidden)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
bundle, berr := h.store.GetHostDRBundle(pathHostID)
|
||||||
|
if berr != nil {
|
||||||
|
h.logger.Printf("[ERROR] escrow GET for %s: %v", pathHostID, berr)
|
||||||
|
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// A host with no sealed bundle gets a CLEAN ANSWER, not a fault: 200 with present=false. A 404
|
||||||
|
// here would be indistinguishable from an unknown host, and an empty 200 without the flag would be
|
||||||
|
// indistinguishable from a zero-length blob — both read as "something is broken" to a caller whose
|
||||||
|
// situation is simply "no ceremony has run yet".
|
||||||
|
if bundle == nil || len(bundle.IdentityBlob) == 0 {
|
||||||
|
h.logger.Printf("[INFO] escrow GET for %s: no identity blob stored (no ceremony has run)", pathHostID)
|
||||||
|
writeJSON(w, http.StatusOK, map[string]any{"host_id": pathHostID, "present": false, "identity_escrow_b64": ""})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// THE MITIGATION (§8.2). Recorded BEFORE the bytes leave, so a retrieval cannot be served without
|
||||||
|
// its audit row; a save failure is logged and does NOT block the response (the blob is opaque and
|
||||||
|
// refusing it would break a recovery over an audit hiccup — but the log line always exists).
|
||||||
|
//
|
||||||
|
// SEVERITY = warning, i.e. it reaches the operator by e-mail. Retrieval is not routine today: it
|
||||||
|
// happens during a recovery and nowhere else. IF a customer-facing self-service flow ever makes it
|
||||||
|
// routine, revisit this — but revisit it deliberately, do not let it decay to info because the
|
||||||
|
// mail became annoying.
|
||||||
|
if host.CustomerID != "" {
|
||||||
|
msg := fmt.Sprintf("Recovery blob served: host %s retrieved its own sealed identity escrow (%d opaque bytes). "+
|
||||||
|
"This is the recovery path in use — the blob cannot be opened without the customer's recovery code, which the hub never holds. "+
|
||||||
|
"If no recovery is in progress on that box, investigate.", pathHostID, len(bundle.IdentityBlob))
|
||||||
|
details, _ := json.Marshal(map[string]any{
|
||||||
|
"host_id": pathHostID,
|
||||||
|
"blob_bytes": len(bundle.IdentityBlob),
|
||||||
|
"self_scope": !isGlobal,
|
||||||
|
})
|
||||||
|
if _, eerr := h.store.SaveEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub"); eerr != nil {
|
||||||
|
h.logger.Printf("[WARN] %s event save FAILED for %s (serving anyway): %v", eventEscrowBlobServed, pathHostID, eerr)
|
||||||
|
} else if h.dispatcher != nil {
|
||||||
|
go h.dispatcher.ProcessEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
h.logger.Printf("[WARN] escrow blob SERVED to host %s (%d opaque bytes, self_scope=%v) — recovery path in use",
|
||||||
|
pathHostID, len(bundle.IdentityBlob), !isGlobal)
|
||||||
|
writeJSON(w, http.StatusOK, map[string]any{
|
||||||
|
"host_id": pathHostID,
|
||||||
|
"present": true,
|
||||||
|
"identity_escrow_b64": base64.StdEncoding.EncodeToString(bundle.IdentityBlob),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// maybeEmitRepoKeyChanged raises ONE operator signal per supersession when the sealed offsite repo
|
||||||
|
// password demonstrably changed. Both hashes have been stored since SLICE 3 (host_escrow and, since
|
||||||
|
// v0.60.0, host_escrow_superseded) and NOTHING compared them: demo-felhom's repository password
|
||||||
|
// changed on 2026-08-03, orphaning 36 snapshots / 1.14 GB, and no event, e-mail, card or log line
|
||||||
|
// said so for thirteen hours — the comparison that eventually found it is this one
|
||||||
|
// (audits/RECON-offsite-dr-chain-2026-08-04.md, R-197).
|
||||||
|
//
|
||||||
|
// THE PREDICATE IS DELIBERATELY NARROW: both hashes known AND different. A first-ever hash (prev "")
|
||||||
|
// is onboarding, not a change; a hash-less supersession (now "") cannot show a change happened; an
|
||||||
|
// identical hash is a re-ceremony of the SAME password, which is a normal healthy act and must stay
|
||||||
|
// silent or a customer is punished for re-running a ceremony. The in-between shapes are LOGGED rather
|
||||||
|
// than dropped, so "we chose not to alarm" and "the check did not run" never look identical.
|
||||||
|
//
|
||||||
|
// SEVERITY = warning, chosen for the world v0.93.0 creates rather than the one it inherits. Before
|
||||||
|
// R-198 a changed key meant the previous history was unopenable by anyone, ever — that would have
|
||||||
|
// argued for error. From v0.93.0 the superseding ceremony RETAINS the old identity blob, so the
|
||||||
|
// previous history stays recoverable with the recovery code that sealed it: the situation is "this
|
||||||
|
// customer's off-site history now depends on an older recovery code", which is operator-actionable
|
||||||
|
// (check the orphan card, expect a fresh repository) and is not a loss. warning also routes: the
|
||||||
|
// dispatcher notifies on warning/error/critical and treats info as an intentional non-notify, and
|
||||||
|
// the whole point of this row is that the operator learns on the day.
|
||||||
|
//
|
||||||
|
// EDGE-TRIGGERED: called only from the superseded branch of the escrow PUT, i.e. once per
|
||||||
|
// supersession, never per report. No timer lives here — the dispatcher owns cooldown.
|
||||||
|
//
|
||||||
|
// NO HASH VALUE TRAVELS. The message and the details name the host, the customer and the retained
|
||||||
|
// count only. The hashes are non-reversible, but a hash is still a fingerprint of a live secret and
|
||||||
|
// this project's rule is that values do not leave the store.
|
||||||
|
func (h *Handler) maybeEmitRepoKeyChanged(customerID, hostID, prevSHA, newSHA string, retained int) {
|
||||||
|
switch {
|
||||||
|
case prevSHA == "":
|
||||||
|
h.logger.Printf("[INFO] escrow for host %s: no previous repo-password hash recorded (first hash, or a legacy hash-less blob) — repo-key-change check not applicable", hostID)
|
||||||
|
return
|
||||||
|
case newSHA == "":
|
||||||
|
h.logger.Printf("[WARN] escrow for host %s: the NEW blob carries no repo-password hash (hash-less supersession) — whether the repository key changed CANNOT be determined from the hub's data", hostID)
|
||||||
|
return
|
||||||
|
case prevSHA == newSHA:
|
||||||
|
return // same password re-sealed: a healthy re-ceremony (Scenario E — silence is correct)
|
||||||
|
}
|
||||||
|
msg := fmt.Sprintf("Offsite repository key CHANGED for host %s: the new escrow seals a different repository password than the one it replaced. "+
|
||||||
|
"The previous off-site history is no longer opened by this box's current key. The superseding blob was retained (%d held), so that history stays recoverable with the recovery code that sealed it — verify the box's off-site tier reports a repository rather than an orphan card, and expect the next backup to start a fresh history.",
|
||||||
|
hostID, retained)
|
||||||
|
details, _ := json.Marshal(map[string]any{
|
||||||
|
"host_id": hostID,
|
||||||
|
"retained_count": retained,
|
||||||
|
"repo_key": "changed", // never the hash values
|
||||||
|
})
|
||||||
|
if _, err := h.store.SaveEvent(customerID, eventRepoKeyChanged, "warning", msg, string(details), "hub"); err != nil {
|
||||||
|
h.logger.Printf("[WARN] %s event save failed for %s: %v", eventRepoKeyChanged, hostID, err)
|
||||||
|
return // audit row first: an e-mail without its event row lies (the OffsiteChecker convention)
|
||||||
|
}
|
||||||
|
h.logger.Printf("[WARN] offsite repository key CHANGED for host %s (customer %s) — previous history now depends on the superseded recovery code; %d retained blob(s)",
|
||||||
|
hostID, customerID, retained)
|
||||||
|
if h.dispatcher != nil {
|
||||||
|
go h.dispatcher.ProcessEvent(customerID, eventRepoKeyChanged, "warning", msg, string(details), "hub")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// handleHostRecoveryCredentialPut vaults a host's break-glass root@pam console credential (TASK G1).
|
// handleHostRecoveryCredentialPut vaults a host's break-glass root@pam console credential (TASK G1).
|
||||||
// SELF-SCOPED (a host key writes only its own; global may write any) — day-0 posts it with the
|
// SELF-SCOPED (a host key writes only its own; global may write any) — day-0 posts it with the
|
||||||
// host api_key. The secret is stored at rest and NEVER logged (only the username + a length are
|
// host api_key. The secret is stored at rest and NEVER logged (only the username + a length are
|
||||||
|
|||||||
@@ -336,7 +336,7 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
|||||||
midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest)
|
midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest)
|
||||||
sinceUTC := midnightBudapest.UTC()
|
sinceUTC := midnightBudapest.UTC()
|
||||||
|
|
||||||
var backupMissed, dbdumpMissed, skipped, deferred int
|
var backupMissed, dbdumpMissed, skipped, deferred, unbound int
|
||||||
|
|
||||||
for _, id := range customerIDs {
|
for _, id := range customerIDs {
|
||||||
// Skip nodes that are down — they already have staleness events
|
// Skip nodes that are down — they already have staleness events
|
||||||
@@ -350,6 +350,37 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── R-195: a customer with NO machine EVER bound is UNKNOWN, not missed ────────────────
|
||||||
|
//
|
||||||
|
// Both verdicts below ask "did the thing we expect every day happen?". For a customer
|
||||||
|
// that has never had a machine bound, nothing has ever been expected, so the honest
|
||||||
|
// answer is UNKNOWN — the same invariant assessBackupFreshness states above, applied one
|
||||||
|
// level up, at the question of whether there is a subject at all.
|
||||||
|
//
|
||||||
|
// The discriminator is "was a host EVER bound", NOT "has a report arrived". That is the
|
||||||
|
// case this check must not break: a box that was installed, bound, and then went silent
|
||||||
|
// has a real fault and must keep alarming. It is bound, so it is judged.
|
||||||
|
//
|
||||||
|
// WHY THIS WAS REACHABLE AT ALL, measured 2026-08-04: the down-skip above is what
|
||||||
|
// protects every other silent customer, and it reads the staleness checker's state — which
|
||||||
|
// is seeded from the `reports` table (store.GetCustomers). A customer that has NEVER
|
||||||
|
// reported appears in no report row, so it gets no staleness state at all and GetState()
|
||||||
|
// returns "" rather than "down". The skip misses exactly the customer it would most
|
||||||
|
// obviously cover, and the DB-dump half below then fires every night: `david`, a
|
||||||
|
// prospective customer whose record was created 2026-08-01 with no machine ever bound,
|
||||||
|
// e-mailed an expected_dbdump_missed ERROR at 03:00 UTC on three consecutive days.
|
||||||
|
//
|
||||||
|
// Fail-open on a read error: an unreadable binding must never SUPPRESS a real alarm.
|
||||||
|
if bound, berr := s.HasEverBoundHost(id); berr != nil {
|
||||||
|
logger.Printf("[WARN] Deadline check: failed to read host binding for %s (judging anyway): %v", id, berr)
|
||||||
|
} else if !bound {
|
||||||
|
// Visible, per the v0.73.0 Part-7 precedent below: a quiet check must never be
|
||||||
|
// indistinguishable from a check that did not run. Once daily, one line per customer.
|
||||||
|
logger.Printf("[INFO] Deadline check: %s has no host EVER bound — all deadline verdicts UNKNOWN (no alarm)", id)
|
||||||
|
unbound++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
// Backup freshness from the agent's host-report (PBS snapshots + vzdump),
|
// Backup freshness from the agent's host-report (PBS snapshots + vzdump),
|
||||||
// the authoritative offsite-backup signal post-slice-8C.
|
// the authoritative offsite-backup signal post-slice-8C.
|
||||||
reportJSON, rerr := s.GetLatestHostReportJSON(id)
|
reportJSON, rerr := s.GetLatestHostReportJSON(id)
|
||||||
@@ -423,6 +454,6 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down)",
|
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down), %d unknown (no host ever bound)",
|
||||||
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped)
|
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped, unbound)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,148 @@
|
|||||||
|
package monitor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-195 — a customer with NO machine EVER bound must not alarm; a customer WITH one must.
|
||||||
|
//
|
||||||
|
// Origin, measured on the live hub 2026-08-04: `david` is a prospective customer whose record was
|
||||||
|
// created 2026-08-01 16:51:49 with no host ever bound (hosts=0, host_deletions=0, host_reports=0,
|
||||||
|
// reports=0). It e-mailed an `expected_dbdump_missed` ERROR at 03:00 UTC on 08-02, 08-03 and 08-04.
|
||||||
|
//
|
||||||
|
// The mechanism, established at source: the down-skip in CheckBackupDeadlines reads
|
||||||
|
// StalenessChecker.GetState(), whose map is seeded from store.GetCustomers() — a query over the
|
||||||
|
// `reports` table. A customer with zero reports is in no row, so it has no state, and GetState()
|
||||||
|
// returns "" rather than "down". The skip that protects every other silent customer misses the one
|
||||||
|
// that never reported at all.
|
||||||
|
//
|
||||||
|
// These tests pin BOTH halves. A suite that only proved the silence would pass against an
|
||||||
|
// implementation that never alarms, which is strictly worse than the defect it replaces.
|
||||||
|
|
||||||
|
// newUnboundStore creates a store holding ONE active customer and NO host row at all.
|
||||||
|
func newUnboundStore(t *testing.T) *store.Store {
|
||||||
|
t.Helper()
|
||||||
|
st, err := store.New(filepath.Join(t.TempDir(), "test.db"), log.New(io.Discard, "", 0))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("store.New: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { st.Close() })
|
||||||
|
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ck", RetrievalPassword: "p"}); err != nil {
|
||||||
|
t.Fatalf("SaveCustomerConfig: %v", err)
|
||||||
|
}
|
||||||
|
return st
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Half 1: the silence ────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_NeverBoundHost_Silent is the david case.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed): deleting the HasEverBoundHost guard from CheckBackupDeadlines
|
||||||
|
// makes this test fail with
|
||||||
|
//
|
||||||
|
// deadline_unbound_test.go: a customer with NO host ever bound must raise NOTHING;
|
||||||
|
// got [expected_dbdump_missed]
|
||||||
|
//
|
||||||
|
// which is verbatim the event `david` e-mailed three mornings running. Restored after.
|
||||||
|
func TestCheckBackupDeadlines_NeverBoundHost_Silent(t *testing.T) {
|
||||||
|
st := newUnboundStore(t)
|
||||||
|
// No UpsertHost, no host-report, no db_dump_completed event — nothing has ever been expected.
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if len(got) != 0 {
|
||||||
|
t.Fatalf("a customer with NO host ever bound must raise NOTHING; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHasEverBoundHost pins the predicate itself across its three inputs, because the whole
|
||||||
|
// behaviour above turns on it and a predicate that answered `false` for everything would make the
|
||||||
|
// test above pass while silencing the entire fleet.
|
||||||
|
func TestHasEverBoundHost(t *testing.T) {
|
||||||
|
st := newUnboundStore(t)
|
||||||
|
|
||||||
|
if bound, err := st.HasEverBoundHost("c1"); err != nil || bound {
|
||||||
|
t.Fatalf("no host rows → want (false,nil); got (%v,%v)", bound, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||||
|
t.Fatalf("UpsertHost: %v", err)
|
||||||
|
}
|
||||||
|
if bound, err := st.HasEverBoundHost("c1"); err != nil || !bound {
|
||||||
|
t.Fatalf("live host row → want (true,nil); got (%v,%v)", bound, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// An unknown customer is never bound — the predicate must not answer from another customer's rows.
|
||||||
|
if bound, err := st.HasEverBoundHost("nobody"); err != nil || bound {
|
||||||
|
t.Fatalf("unknown customer → want (false,nil); got (%v,%v)", bound, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Half 2: THE RED-PROOF THAT MATTERS — a bound machine must still alarm ───────────────────────
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms is the case the change could break,
|
||||||
|
// and it is a real shape: a machine that was installed and bound and never phoned home. It has a
|
||||||
|
// `hosts` row and zero reports — indistinguishable from `david` on every signal EXCEPT the one the
|
||||||
|
// guard discriminates on. If the guard is ever "simplified" to key off report presence, customer
|
||||||
|
// age or a name pattern, this test goes red.
|
||||||
|
func TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms(t *testing.T) {
|
||||||
|
st := newUnboundStore(t)
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||||
|
t.Fatalf("UpsertHost: %v", err)
|
||||||
|
}
|
||||||
|
// No host-report and no db_dump_completed — the box was bound and never said anything.
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if !has(got, "expected_dbdump_missed") {
|
||||||
|
t.Fatalf("a BOUND machine that never reported is a real fault and must still alarm; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_BoundThenWentQuiet_StillAlarms: the machine reported once, days ago,
|
||||||
|
// and stopped. Nothing about the guard may suppress that.
|
||||||
|
//
|
||||||
|
// The staleness checker's down-skip is nil here (runDeadline passes nil), which is deliberate: it
|
||||||
|
// isolates THIS guard. In production a genuinely down node is skipped by staleness and gets its own
|
||||||
|
// node_down event — that path is unchanged and is not what this test is about.
|
||||||
|
func TestCheckBackupDeadlines_BoundThenWentQuiet_StillAlarms(t *testing.T) {
|
||||||
|
st := newUnboundStore(t)
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||||
|
t.Fatalf("UpsertHost: %v", err)
|
||||||
|
}
|
||||||
|
// One host-report whose newest backup evidence is 9 days old, and a db dump that last
|
||||||
|
// completed 5 days ago (i.e. not since midnight).
|
||||||
|
report := hostReportJSON(t, [][2]string{{rfc(-9 * 24 * time.Hour), "ok"}}, nil)
|
||||||
|
if err := st.SaveHostReport("h1", "c1", []byte(report), store.HostReportDenorm{}); err != nil {
|
||||||
|
t.Fatalf("SaveHostReport: %v", err)
|
||||||
|
}
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if !has(got, "expected_backup_missed") {
|
||||||
|
t.Fatalf("a bound machine that went quiet with stale backups must still raise expected_backup_missed; got %v", got)
|
||||||
|
}
|
||||||
|
if !has(got, "expected_dbdump_missed") {
|
||||||
|
t.Fatalf("a bound machine that went quiet must still raise expected_dbdump_missed; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_DeletedHost_StillJudged: the host row is gone but a tombstone remains
|
||||||
|
// (peti-felhom's live shape). The customer HAD a machine, so this check must not take over the
|
||||||
|
// judgement — it hands off to the staleness down-skip exactly as before the change.
|
||||||
|
func TestCheckBackupDeadlines_DeletedHost_StillJudged(t *testing.T) {
|
||||||
|
st := newUnboundStore(t)
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||||
|
t.Fatalf("UpsertHost: %v", err)
|
||||||
|
}
|
||||||
|
if err := st.DeleteHost("h1", false); err != nil {
|
||||||
|
t.Fatalf("DeleteHost: %v", err)
|
||||||
|
}
|
||||||
|
if bound, err := st.HasEverBoundHost("c1"); err != nil || !bound {
|
||||||
|
t.Fatalf("a DELETED host is still a machine that was once bound → want (true,nil); got (%v,%v)", bound, err)
|
||||||
|
}
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if !has(got, "expected_dbdump_missed") {
|
||||||
|
t.Fatalf("a customer whose host was deleted was still bound and stays judged here; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -94,40 +94,112 @@ func (c *OffsiteDeliveryChecker) Check() {
|
|||||||
if age < stuckAfter {
|
if age < stuckAfter {
|
||||||
continue // normal convergence window
|
continue // normal convergence window
|
||||||
}
|
}
|
||||||
c.maybeEmitStuck(cfg.CustomerID, status, age)
|
emitted := c.maybeEmitStuck(cfg.CustomerID, status, age)
|
||||||
c.maybeHeal(cfg.CustomerID, status)
|
c.maybeHeal(cfg.CustomerID, status, emitted)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer.
|
// deliveryShape names the two situations the ONE stuck state actually covers. They need different
|
||||||
func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) {
|
// text and different advice, and conflating them is R-192's defect (a).
|
||||||
|
type deliveryShape string
|
||||||
|
|
||||||
|
const (
|
||||||
|
// shapeBurned — NO report since the consume carried an offbox target: the apply never persisted.
|
||||||
|
// Re-issue is the indicated action.
|
||||||
|
shapeBurned deliveryShape = "burned"
|
||||||
|
// shapeRegressed — reports since the consume DID carry an offbox target and the latest does not:
|
||||||
|
// the credential worked and the target was later lost (a guest rebuild does exactly this, R-193).
|
||||||
|
// Re-issue is NOT indicated; it treats a symptom whose cause is elsewhere.
|
||||||
|
shapeRegressed deliveryShape = "regressed"
|
||||||
|
)
|
||||||
|
|
||||||
|
func shapeOf(status offsite.DeliveryStatus) deliveryShape {
|
||||||
|
if status.OffsiteReportsSinceConsume == 0 {
|
||||||
|
return shapeBurned
|
||||||
|
}
|
||||||
|
return shapeRegressed
|
||||||
|
}
|
||||||
|
|
||||||
|
// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer. Returns
|
||||||
|
// whether it emitted, so the heal's refusal record rides the same cadence rather than inventing one.
|
||||||
|
//
|
||||||
|
// R-192 defect (a), fixed here: the message used to interpolate ReportsSinceConsume (the TOTAL) into
|
||||||
|
// a hardcoded phrase "report(s) since carry no offbox target", and never consulted
|
||||||
|
// OffsiteReportsSinceConsume — the field that says the opposite. On demo-hp it stated, daily, that
|
||||||
|
// 500 reports carried no offbox target when all 500 of them did, and prescribed Re-issue for a
|
||||||
|
// failure mode that had not occurred. The message now STATES WHAT WAS MEASURED and lets the operator
|
||||||
|
// read it; the recommendation follows the shape rather than being hardcoded.
|
||||||
|
//
|
||||||
|
// THE WINDOW IS NAMED ON PURPOSE. CountReportsOffsiteSince reads `ORDER BY id LIMIT 500` — the OLDEST
|
||||||
|
// 500 reports after the consume, not the newest — so on a long-lived customer these counts describe
|
||||||
|
// the beginning of the window and not the present. That is a real scoping defect (R-192's other half)
|
||||||
|
// and it stays OPEN because its correct shape depends on the recovery chain that is not yet
|
||||||
|
// assembled (R-199/R-200/R-201). Naming the window in the text is how it stays visible instead of
|
||||||
|
// being laundered into a confident sentence — an instrument that can silently mis-scope its results
|
||||||
|
// must say so where it reports them.
|
||||||
|
func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) bool {
|
||||||
last, err := c.store.LastEventAt(customerID, eventDeliveryStuck)
|
last, err := c.store.LastEventAt(customerID, eventDeliveryStuck)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
c.logger.Printf("[WARN] offsite-delivery: %s: cooldown read: %v", customerID, err)
|
c.logger.Printf("[WARN] offsite-delivery: %s: cooldown read: %v", customerID, err)
|
||||||
return
|
return false
|
||||||
}
|
}
|
||||||
if !last.IsZero() && c.now().Sub(last) < stuckCooldown {
|
if !last.IsZero() && c.now().Sub(last) < stuckCooldown {
|
||||||
return
|
return false
|
||||||
|
}
|
||||||
|
shape := shapeOf(status)
|
||||||
|
var msg string
|
||||||
|
switch shape {
|
||||||
|
case shapeBurned:
|
||||||
|
msg = fmt.Sprintf("Offsite delivery stuck (BURNED-credential shape): the one-time password was consumed %s ago; of the first %d report(s) after that consume, NONE carried an offbox target, and the latest report carries none either. The credential never reached a persisted apply. Re-issue delivers a fresh one.",
|
||||||
|
age.Round(time.Minute), status.ReportsSinceConsume)
|
||||||
|
default:
|
||||||
|
msg = fmt.Sprintf("Offsite delivery stuck (REGRESSED-apply shape): the one-time password was consumed %s ago; of the first %d report(s) after that consume, %d DID carry an offbox target — and the latest report carries none. The credential was applied and worked; the target was lost afterwards. Re-issue is NOT the indicated action: find what removed the offbox target (a guest rebuild does, R-193). Automatic restage is deliberately withheld for this shape. NOTE: the counts cover at most the first 500 reports after the consume, so on a long-lived box they describe the start of the window, not now (R-192, open).",
|
||||||
|
age.Round(time.Minute), status.ReportsSinceConsume, status.OffsiteReportsSinceConsume)
|
||||||
}
|
}
|
||||||
msg := fmt.Sprintf("Offsite delivery stuck: one-time password consumed %s ago and %d report(s) since carry no offbox target — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one.",
|
|
||||||
age.Round(time.Minute), status.ReportsSinceConsume)
|
|
||||||
details, _ := json.Marshal(map[string]any{
|
details, _ := json.Marshal(map[string]any{
|
||||||
"state": string(status.State),
|
"state": string(status.State),
|
||||||
"consumed_at": status.Since.UTC().Format(time.RFC3339),
|
"shape": string(shape),
|
||||||
"reports_since_consume": status.ReportsSinceConsume,
|
"consumed_at": status.Since.UTC().Format(time.RFC3339),
|
||||||
|
"reports_since_consume": status.ReportsSinceConsume,
|
||||||
|
"offsite_reports_since_consume": status.OffsiteReportsSinceConsume,
|
||||||
|
"count_window": "oldest 500 reports after consumed_at (R-192, open)",
|
||||||
})
|
})
|
||||||
c.emit(customerID, eventDeliveryStuck, "warning", msg, string(details))
|
c.emit(customerID, eventDeliveryStuck, "warning", msg, string(details))
|
||||||
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
// maybeHeal fires the R-71c self-heal when the burned-credential shape is unambiguous:
|
// maybeHeal fires the R-71c self-heal when the burned-credential shape is unambiguous:
|
||||||
// consumed ≥ stuckAfter ago, ≥ healMinReports consecutive reports since with ZERO offbox evidence,
|
// consumed ≥ stuckAfter ago, ≥ healMinReports consecutive reports since with ZERO offbox evidence,
|
||||||
// one heal per healCooldown — and the R-39(a) guard holds at act time.
|
// one heal per healCooldown — and the R-39(a) guard holds at act time.
|
||||||
func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus) {
|
//
|
||||||
|
// R-192 defect (b), fixed here: every refusal above the "not configured" line now leaves a RECORD.
|
||||||
|
// The regressed-shape branch used to be a bare `return`, so the operator received a daily e-mail with
|
||||||
|
// the wrong story, no heal, and nothing anywhere saying why the heal declined — "we chose not to act"
|
||||||
|
// and "the heal never ran" looked identical. `offsite_credential_restaged` has never fired for any
|
||||||
|
// customer, and until now that fact was indistinguishable from the checker being dead.
|
||||||
|
//
|
||||||
|
// The record is a notification_log row (the dispatcher's suppressed-operator-e-mail precedent, R-182:
|
||||||
|
// a decision not to act is written down on the channel it would have used). It rides `recordRefusal`
|
||||||
|
// — true only when the stuck event was emitted this pass — so it appears once per stuckCooldown
|
||||||
|
// beside the e-mail it explains, rather than once per monitor tick. The GUARD ITSELF IS UNCHANGED:
|
||||||
|
// the set of situations in which the heal fires is byte-for-byte what it was; only the silence is
|
||||||
|
// gone. The two conditions are split into separate branches solely so each refusal can name its own
|
||||||
|
// reason.
|
||||||
|
func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus, recordRefusal bool) {
|
||||||
if c.reissuer == nil {
|
if c.reissuer == nil {
|
||||||
|
return // no provisioner configured: the heal does not exist on this hub, so there is nothing to explain
|
||||||
|
}
|
||||||
|
if status.OffsiteReportsSinceConsume != 0 {
|
||||||
|
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
|
||||||
|
"regressed-apply shape: %d of the first %d report(s) after the consume DID carry an offbox target, so a burned credential is ruled out — a restage would treat a symptom whose cause is elsewhere. Operator's call (R-193).",
|
||||||
|
status.OffsiteReportsSinceConsume, status.ReportsSinceConsume))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if status.ReportsSinceConsume < healMinReports || status.OffsiteReportsSinceConsume != 0 {
|
if status.ReportsSinceConsume < healMinReports {
|
||||||
return // box not reporting enough, or offbox evidence exists (regressed-apply shape) → operator's call
|
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
|
||||||
|
"only %d report(s) since the consume (need %d): the box has not reported enough for the burned shape to be unambiguous.",
|
||||||
|
status.ReportsSinceConsume, healMinReports))
|
||||||
|
return
|
||||||
}
|
}
|
||||||
last, err := c.store.LastEventAt(customerID, eventCredentialRestaged)
|
last, err := c.store.LastEventAt(customerID, eventCredentialRestaged)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -146,8 +218,8 @@ func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.Del
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
if info == nil || info.ConsumedAt.IsZero() {
|
if info == nil || info.ConsumedAt.IsZero() {
|
||||||
c.logger.Printf("[INFO] offsite-delivery: %s: heal refused — secret row is now %s (R-39(a) guard: never restage over an unconsumed secret)",
|
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
|
||||||
customerID, secretShape(info))
|
"R-39(a) guard: the secret row is now %s — restaging over an unconsumed secret would clobber a password the box may be about to consume.", secretShape(info)))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
|
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
|
||||||
@@ -167,6 +239,22 @@ func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.Del
|
|||||||
string(details))
|
string(details))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// recordHealRefusal makes a decision NOT to self-heal visible. Always logs; additionally writes a
|
||||||
|
// notification_log row on the "operator" channel with status "refused" when `record` is set (the
|
||||||
|
// stuck event was emitted this pass), so the refusal sits next to the e-mail that prompted the
|
||||||
|
// question. A LogNotification failure is logged, never swallowed, and never blocks the refusal — the
|
||||||
|
// refusal is the primary effect.
|
||||||
|
func (c *OffsiteDeliveryChecker) recordHealRefusal(customerID string, record bool, reason string) {
|
||||||
|
c.logger.Printf("[INFO] offsite-delivery: %s: self-heal REFUSED — %s", customerID, reason)
|
||||||
|
if !record {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := c.store.LogNotification(customerID, eventCredentialRestaged, "warning",
|
||||||
|
"Automatic offsite credential restage was NOT performed.", "refused", reason, "operator"); err != nil {
|
||||||
|
c.logger.Printf("[WARN] offsite-delivery: %s: could not record the heal refusal: %v", customerID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func secretShape(info *store.OneTimeSecretInfo) string {
|
func secretShape(info *store.OneTimeSecretInfo) string {
|
||||||
if info == nil {
|
if info == nil {
|
||||||
return "absent"
|
return "absent"
|
||||||
|
|||||||
@@ -0,0 +1,223 @@
|
|||||||
|
package monitor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-192 (v0.93.0) — the alert must describe what was MEASURED, and a refusal to self-heal must leave a
|
||||||
|
// trace.
|
||||||
|
//
|
||||||
|
// What was wrong, quoted from the live e-mail demo-hp sent daily from 2026-08-03: "one-time password
|
||||||
|
// consumed 284h19m0s ago and 500 report(s) since carry no offbox target — the credential is likely
|
||||||
|
// burned … Re-issue delivers a fresh one." OffsiteReportsSinceConsume for that same call was 500:
|
||||||
|
// every one of those reports carried a target. The message stated the precise negation of its own
|
||||||
|
// measurement and prescribed a remedy for a failure mode that had not occurred — while the self-heal
|
||||||
|
// correctly declined, in complete silence.
|
||||||
|
|
||||||
|
// regressedFixture puts c1 into demo-hp's real shape: consumed >1h ago, reports WITH an offbox target
|
||||||
|
// after the consume, then reports WITHOUT — the latest carrying none.
|
||||||
|
//
|
||||||
|
// The report timestamps get jitter deliberately: a perfectly regular series lands exactly on
|
||||||
|
// boundaries and survives mutations that a realistic one catches.
|
||||||
|
func (h dtHarness) regressedFixture(t *testing.T, withOffsite, without int) {
|
||||||
|
t.Helper()
|
||||||
|
if err := h.st.SaveOneTimeSecret("c1", "x"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
consumed := time.Now().UTC().Add(-30 * time.Hour).Format("2006-01-02 15:04:05")
|
||||||
|
staged := time.Now().UTC().Add(-31 * time.Hour).Format("2006-01-02 15:04:05")
|
||||||
|
if err := h.st.SetOneTimeSecretTimesForTest("c1", staged, consumed); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
jitter := []int{0, 3, 1, 7, 2, 5, 11, 4}
|
||||||
|
for i := 0; i < withOffsite; i++ {
|
||||||
|
if err := h.st.SaveReport("c1", []byte(dtReportWithOffsite)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
time.Sleep(time.Duration(jitter[i%len(jitter)]) * time.Millisecond)
|
||||||
|
}
|
||||||
|
for i := 0; i < without; i++ {
|
||||||
|
if err := h.st.SaveReport("c1", []byte(dtReportNoOffsite)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
time.Sleep(time.Duration(jitter[(i+3)%len(jitter)]) * time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h dtHarness) refusalRows(t *testing.T) []store.NotificationLogEntry {
|
||||||
|
t.Helper()
|
||||||
|
all, err := h.st.GetRecentNotifications("c1", 50)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var out []store.NotificationLogEntry
|
||||||
|
for _, n := range all {
|
||||||
|
if n.Status == "refused" {
|
||||||
|
out = append(out, n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario F — the message states the measured state, not a denied cause.
|
||||||
|
// RED-PROOF: restore the old single hardcoded sentence (interpolate ReportsSinceConsume into
|
||||||
|
// "report(s) since carry no offbox target") → the regressed assertions FAIL.
|
||||||
|
func TestDeliveryChecker_RegressedShape_MessageMatchesMeasurement(t *testing.T) {
|
||||||
|
h := newDTHarness(t, true)
|
||||||
|
h.regressedFixture(t, 6, 2)
|
||||||
|
|
||||||
|
h.checker.Check()
|
||||||
|
|
||||||
|
saved := h.savedEvents(t, "offsite_delivery_stuck")
|
||||||
|
if len(saved) != 1 {
|
||||||
|
t.Fatalf("stuck events = %d, want 1", len(saved))
|
||||||
|
}
|
||||||
|
msg := saved[0].Message
|
||||||
|
|
||||||
|
// It must NOT assert the thing the data contradicts.
|
||||||
|
if strings.Contains(msg, "carry no offbox target") || strings.Contains(msg, "likely burned") {
|
||||||
|
t.Fatalf("R-192(a): the message still asserts a burned credential on a REGRESSED shape — this is "+
|
||||||
|
"the sentence that stated the negation of its own measurement:\n%s", msg)
|
||||||
|
}
|
||||||
|
// It must name the shape it detected and the measurement behind it.
|
||||||
|
if !strings.Contains(msg, "REGRESSED") {
|
||||||
|
t.Errorf("the message must name the detected shape, got:\n%s", msg)
|
||||||
|
}
|
||||||
|
if !strings.Contains(msg, "6 DID carry an offbox target") {
|
||||||
|
t.Errorf("the message must state the measured offbox-bearing count (6), got:\n%s", msg)
|
||||||
|
}
|
||||||
|
// It must not prescribe the action for a failure mode that did not occur.
|
||||||
|
if !strings.Contains(msg, "Re-issue is NOT the indicated action") {
|
||||||
|
t.Errorf("the message must withdraw the Re-issue recommendation for this shape, got:\n%s", msg)
|
||||||
|
}
|
||||||
|
// The counting window is named, because CountReportsOffsiteSince reads the OLDEST 500 — an
|
||||||
|
// instrument that can silently mis-scope must say so where it reports (R-192's other half, open).
|
||||||
|
if !strings.Contains(msg, "500") {
|
||||||
|
t.Errorf("the message must name its counting window, got:\n%s", msg)
|
||||||
|
}
|
||||||
|
|
||||||
|
var details map[string]any
|
||||||
|
if err := json.Unmarshal([]byte(saved[0].DetailsJSON), &details); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if details["shape"] != "regressed" {
|
||||||
|
t.Errorf("details.shape = %v, want regressed", details["shape"])
|
||||||
|
}
|
||||||
|
// The field whose absence was defect (a) is now carried.
|
||||||
|
if got, ok := details["offsite_reports_since_consume"].(float64); !ok || int(got) != 6 {
|
||||||
|
t.Errorf("details must carry offsite_reports_since_consume=6, got %v", details["offsite_reports_since_consume"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The burned shape keeps its own (correct) text and its recommendation — the fix must not blunt the
|
||||||
|
// case that was always right.
|
||||||
|
func TestDeliveryChecker_BurnedShape_KeepsItsRecommendation(t *testing.T) {
|
||||||
|
h := newDTHarness(t, false)
|
||||||
|
h.burnedFixture(t, 5)
|
||||||
|
|
||||||
|
h.checker.Check()
|
||||||
|
|
||||||
|
saved := h.savedEvents(t, "offsite_delivery_stuck")
|
||||||
|
if len(saved) != 1 {
|
||||||
|
t.Fatalf("stuck events = %d, want 1", len(saved))
|
||||||
|
}
|
||||||
|
msg := saved[0].Message
|
||||||
|
if !strings.Contains(msg, "BURNED") || !strings.Contains(msg, "NONE carried an offbox target") {
|
||||||
|
t.Errorf("the burned shape must state its own measurement, got:\n%s", msg)
|
||||||
|
}
|
||||||
|
if !strings.Contains(msg, "Re-issue delivers a fresh one") {
|
||||||
|
t.Errorf("the burned shape must keep its recommendation, got:\n%s", msg)
|
||||||
|
}
|
||||||
|
var details map[string]any
|
||||||
|
_ = json.Unmarshal([]byte(saved[0].DetailsJSON), &details)
|
||||||
|
if details["shape"] != "burned" {
|
||||||
|
t.Errorf("details.shape = %v, want burned", details["shape"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario G — a refusal to self-heal leaves a durable record naming its reason.
|
||||||
|
// RED-PROOF: replace the recordHealRefusal call in the regressed branch with a bare `return`
|
||||||
|
// (production behaviour up to v0.92.0) → no row → this FAILS.
|
||||||
|
func TestDeliveryChecker_HealRefusal_LeavesARecord(t *testing.T) {
|
||||||
|
h := newDTHarness(t, true)
|
||||||
|
h.regressedFixture(t, 6, 2)
|
||||||
|
|
||||||
|
h.checker.Check()
|
||||||
|
|
||||||
|
if h.reissuer.count() != 0 {
|
||||||
|
t.Fatal("the heal must NOT fire on a regressed shape — the guard is unchanged by this fix")
|
||||||
|
}
|
||||||
|
rows := h.refusalRows(t)
|
||||||
|
if len(rows) != 1 {
|
||||||
|
t.Fatalf("R-192(b): the heal declined and left %d record(s), want 1 — 'we chose not to act' and "+
|
||||||
|
"'the heal never ran' must not look identical", len(rows))
|
||||||
|
}
|
||||||
|
r := rows[0]
|
||||||
|
if r.Channel != "operator" {
|
||||||
|
t.Errorf("refusal channel = %q, want operator (the channel the heal's own event would use)", r.Channel)
|
||||||
|
}
|
||||||
|
if !strings.Contains(r.ErrorMessage, "regressed-apply shape") {
|
||||||
|
t.Errorf("the record must name WHY, got %q", r.ErrorMessage)
|
||||||
|
}
|
||||||
|
if !strings.Contains(r.ErrorMessage, "6 of the first 8") {
|
||||||
|
t.Errorf("the record must carry the measurement it refused on, got %q", r.ErrorMessage)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The R-39(a) guard's refusal is recorded too — it was the one refusal that already logged, and it
|
||||||
|
// still must not be the only one that does.
|
||||||
|
//
|
||||||
|
// Reaching it needs the real TOCTOU, not a pre-staged row: staging BEFORE Check() makes the derive
|
||||||
|
// itself return staged_awaiting_consume and the pass never reaches the heal at all. Same technique as
|
||||||
|
// TestDeliveryChecker_R39aGuard_NeverRestagesOverUnconsumed — the operator stages mid-tick, on the
|
||||||
|
// stuck event's dispatch, which runs after the derive and before maybeHeal.
|
||||||
|
func TestDeliveryChecker_R39aRefusal_LeavesARecord(t *testing.T) {
|
||||||
|
h := newDTHarness(t, true)
|
||||||
|
h.burnedFixture(t, 5)
|
||||||
|
|
||||||
|
base := h.checker.onEvent
|
||||||
|
h.checker.onEvent = func(cid, et, sev, msg, det, src string) {
|
||||||
|
if et == "offsite_delivery_stuck" {
|
||||||
|
if err := h.st.SaveOneTimeSecret("c1", "operator-fresh"); err != nil {
|
||||||
|
t.Errorf("mid-tick stage: %v", err)
|
||||||
|
}
|
||||||
|
if err := h.st.SetOneTimeSecretTimesForTest("c1", "2026-07-23 12:00:00", ""); err != nil {
|
||||||
|
t.Errorf("mid-tick stamp: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
base(cid, et, sev, msg, det, src)
|
||||||
|
}
|
||||||
|
|
||||||
|
h.checker.Check()
|
||||||
|
|
||||||
|
if h.reissuer.count() != 0 {
|
||||||
|
t.Fatal("R-39(a): the heal must never restage over an unconsumed secret")
|
||||||
|
}
|
||||||
|
rows := h.refusalRows(t)
|
||||||
|
if len(rows) != 1 {
|
||||||
|
t.Fatalf("the R-39(a) refusal left %d record(s), want 1", len(rows))
|
||||||
|
}
|
||||||
|
if !strings.Contains(rows[0].ErrorMessage, "R-39(a)") {
|
||||||
|
t.Errorf("the record must name the guard, got %q", rows[0].ErrorMessage)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The refusal record rides the stuck event's 24h cadence: it explains the e-mail the operator just
|
||||||
|
// received, and must not accumulate one row per monitor tick.
|
||||||
|
func TestDeliveryChecker_HealRefusal_RidesTheStuckCadence(t *testing.T) {
|
||||||
|
h := newDTHarness(t, true)
|
||||||
|
h.regressedFixture(t, 6, 2)
|
||||||
|
|
||||||
|
h.checker.Check()
|
||||||
|
h.checker.Check() // the stuck event is swallowed by its cooldown; so is the refusal record
|
||||||
|
h.checker.Check()
|
||||||
|
|
||||||
|
if n := len(h.refusalRows(t)); n != 1 {
|
||||||
|
t.Fatalf("refusal records = %d after three passes, want 1 (one per stuck e-mail, not per tick)", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -102,7 +102,7 @@ func TestOffsiteStale_NeverRanEscrowAnchorWins(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
// escrow blob stored now (SaveHostEscrow stamps updated_at with datetime('now'))
|
// escrow blob stored now (SaveHostEscrow stamps updated_at with datetime('now'))
|
||||||
if _, err := st.SaveHostEscrow("h1", []byte("blob"), "fp", "zero_knowledge", time.Now().UTC().Format(time.RFC3339), "sha"); err != nil {
|
if _, _, err := st.SaveHostEscrow("h1", []byte("blob"), "fp", "zero_knowledge", time.Now().UTC().Format(time.RFC3339), "sha"); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
saveOffsiteReport(t, st, "c1", offsiteJSON(true, "escrowed", "", "", 0, 50))
|
saveOffsiteReport(t, st, "c1", offsiteJSON(true, "escrowed", "", "", 0, 50))
|
||||||
|
|||||||
@@ -421,6 +421,29 @@ var operatorOnlyEvents = map[string]bool{
|
|||||||
// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
|
// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
|
||||||
// once (v0.78.0) and the comment above records it.
|
// once (v0.78.0) and the comment above records it.
|
||||||
"backup_run_failures": true,
|
"backup_run_failures": true,
|
||||||
|
// R-197 (v0.93.0). "The sealed offsite repository key changed" is a custody fact about escrow
|
||||||
|
// blobs. A customer can take no action on it — the remedy is the operator's inspection of the
|
||||||
|
// off-site tier — and the text is operator-grade English naming host ids and retained-blob
|
||||||
|
// counts. Listed here in the SAME commit that mints the type: an operator-tier type that is not
|
||||||
|
// registered here reaches customers as raw English, because a missing customerMessages entry is
|
||||||
|
// NOT a block (the v0.78.0 defect recorded above).
|
||||||
|
"offsite_repo_key_changed": true,
|
||||||
|
// R-192 (v0.93.0). These two predate the register and were never added to it — a real gap, not a
|
||||||
|
// tidy-up. `offsite_delivery_stuck` is severity warning, has no customerMessages entry, and
|
||||||
|
// therefore fell through to FormatCustomerEmail's raw-English fallback: a customer whose box hit
|
||||||
|
// the stuck shape was in line for an English e-mail about one-time passwords being "likely
|
||||||
|
// burned". Measured on the live hub: notification_log holds operator rows for demo-hp and no
|
||||||
|
// customer rows — which is NOT evidence the leg is blocked (it is equally consistent with no
|
||||||
|
// configured recipient), so the register makes it structural instead of incidental. Narrowing
|
||||||
|
// only: the operator channel is untouched.
|
||||||
|
"offsite_delivery_stuck": true,
|
||||||
|
"offsite_credential_restaged": true,
|
||||||
|
// R-199 (v0.94.0). A host retrieved its own sealed recovery blob. Operator-tier by construction:
|
||||||
|
// it names host ids and opaque byte counts, the customer can take no action on it, and its whole
|
||||||
|
// purpose is that the operator sees a capability being used. Registered in the same commit that
|
||||||
|
// mints the type — an operator-tier type absent from this register reaches customers as raw
|
||||||
|
// English (the v0.78.0 defect recorded above).
|
||||||
|
"escrow_blob_served": true,
|
||||||
}
|
}
|
||||||
|
|
||||||
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
|
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
|
||||||
|
|||||||
@@ -195,35 +195,60 @@ func (p *Provisioner) ReissueCredentials(ctx context.Context, customerID, typ st
|
|||||||
return fmt.Errorf("offsite: store re-issued one-time password: %w", err)
|
return fmt.Errorf("offsite: store re-issued one-time password: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// v0.57.0 (2.3, the escrow-honesty fix): the restic repo password just changed, so any existing
|
// v0.57.0 (2.3, the escrow-honesty fix). ⚠ ITS STATED REASON WAS FALSE AND IS CORRECTED HERE
|
||||||
// key-escrow blob — which sealed the OLD password — is now STALE. A recovery code minted against
|
// (R-196, 2026-08-04). The comment used to read "the restic repo password just changed" — this
|
||||||
// it would decrypt a password that no longer opens the repo. Mark the escrow stale so the hub
|
// function does NOT change it and CANNOT: the repository password is generated on the box
|
||||||
// stops advertising "ceremony done" and the customer's escrow wizard is offered again; a fresh
|
// (controller `WriteOffboxSecrets`) and never leaves it except sealed inside the R-wrapped escrow.
|
||||||
// ceremony seals the new password and clears the flag. Every credential change also emits a
|
// What this function changes is the PROVIDER account password (①), which is a different secret at
|
||||||
// visible customer event (offsite_reissued always; escrow_stale only when a blob was invalidated).
|
// a different layer. The false premise survived because the EFFECT (a stale escrow) is real, so
|
||||||
// Best-effort: the password reset already succeeded — a bookkeeping failure here must not fail it.
|
// nobody checked the CAUSE — and it sent two separate investigations the wrong way in one day
|
||||||
escrowStaled := false
|
// (audits/SPIKE-offsite-credential-recovery-2026-08-04.md Q4).
|
||||||
if host, herr := p.Store.GetHostByCustomer(customerID); herr == nil && host != nil {
|
//
|
||||||
if esc, eerr := p.Store.GetHostEscrow(host.HostID); eerr == nil && esc != nil {
|
// ⚠ THE MARK IS GONE (R-196 / R-204 item 2, hub v0.95.0, 2026-08-05). What used to stand here —
|
||||||
if serr := p.Store.MarkEscrowStale(host.HostID); serr != nil {
|
// a `MarkEscrowStale` on every re-issue that found an escrow row, plus an `escrow_stale` customer
|
||||||
p.logf("[offsite] WARN mark-escrow-stale for %s: %v", customerID, serr)
|
// event — was PRECAUTIONARY, not evidential: it guessed that the box's re-apply MIGHT mint a fresh
|
||||||
} else {
|
// repository password. On the ordinary re-issue shape (a box that still holds its
|
||||||
escrowStaled = true
|
// `<DataDir>/offbox/repo_password`) the password does not change, so it marked a HEALTHY escrow
|
||||||
}
|
// stale. The 2026-08-04 recovery drill (R-201) is what promoted this from a nit to a blocker.
|
||||||
}
|
//
|
||||||
}
|
// WHAT THE MARK ACTUALLY DID, mechanically, because "it asked for an unnecessary ceremony"
|
||||||
|
// understates it by a lot:
|
||||||
|
// 1. `stale_at` set → `GetEscrowStatusForCustomer` WITHHOLDS `restic_pw_sha256` from the report
|
||||||
|
// ACK (store.go, the v0.57.0 rule).
|
||||||
|
// 2. With no hash, the controller's SLICE-3 auto-confirm cannot flip pending→escrowed
|
||||||
|
// (report.EscrowAutoConfirmer.Reconcile returns early on an empty hash).
|
||||||
|
// 3. `OffboxRunnable() = OffboxConfigured() && EscrowState=="escrowed"` → EVERY off-site backup
|
||||||
|
// is refused, indefinitely, on a box whose key was never in doubt.
|
||||||
|
// 4. The customer is told to re-run the recovery ceremony — which mints a NEW recovery code and
|
||||||
|
// supersedes the sealed blob. During a recovery that is the one act that would have destroyed
|
||||||
|
// the key just recovered.
|
||||||
|
// A precautionary flag that stops the data-protection it is guarding is not conservative.
|
||||||
|
//
|
||||||
|
// WHY REMOVING IT LEAVES NO GAP — the case it guessed at is MEASURED elsewhere, and better:
|
||||||
|
// • Continuous, box-side: the controller compares the ACK's sealed hash against its CURRENT
|
||||||
|
// local repo password on EVERY report ACK (`reconcileEscrowed`, the Scenario-F re-check). In
|
||||||
|
// the guest-rebuild shape — the only shape where a re-issue is followed by a fresh repository
|
||||||
|
// password — that comparison mismatches within one report cycle and raises the stale card plus
|
||||||
|
// the „create a new recovery code" CTA. It is a measurement, not a guess.
|
||||||
|
// AND THE MARK WAS BLINDING IT: by emptying the hash (step 1 above) it removed the very value
|
||||||
|
// that comparison needs, so the box could only report the hash-LESS reason, which is false.
|
||||||
|
// • Edge-triggered, hub-side: R-197's `offsite_repo_key_changed` fires on a proven hash
|
||||||
|
// difference across a supersession (api.maybeEmitRepoKeyChanged) and pages the operator.
|
||||||
|
//
|
||||||
|
// DISAGREEMENT RECORDED, per the R-96 standing rule: the task's Scenario D asks that a real key
|
||||||
|
// change "marks the escrow stale". It must NOT, and nothing here was changed to make it: the hub
|
||||||
|
// learns of a real change at the moment a supersession SEALS THE NEW PASSWORD, i.e. when the escrow
|
||||||
|
// is freshest. Marking it stale there would ask for a ceremony to fix the ceremony that just ran.
|
||||||
|
// The correct consequence at that instant is the operator alarm, which is what R-197 does.
|
||||||
|
//
|
||||||
|
// `offsite_reissued` is UNCHANGED and still always fires — the customer must still learn that the
|
||||||
|
// credential moved. Best-effort: the password reset already succeeded, so a bookkeeping failure
|
||||||
|
// here must not fail it.
|
||||||
if _, serr := p.Store.SaveEvent(customerID, "offsite_reissued", "info",
|
if _, serr := p.Store.SaveEvent(customerID, "offsite_reissued", "info",
|
||||||
"Az offsite (házon kívüli) mentési hozzáférést újra kiadtuk — az új egyszeri jelszót a vezérlő a következő frissítéskor átveszi.",
|
"Az offsite (házon kívüli) mentési hozzáférést újra kiadtuk — az új egyszeri jelszót a vezérlő a következő frissítéskor átveszi.",
|
||||||
"", "hub"); serr != nil {
|
"", "hub"); serr != nil {
|
||||||
p.logf("[offsite] WARN save offsite_reissued event for %s: %v", customerID, serr)
|
p.logf("[offsite] WARN save offsite_reissued event for %s: %v", customerID, serr)
|
||||||
}
|
}
|
||||||
if escrowStaled {
|
|
||||||
if _, serr := p.Store.SaveEvent(customerID, "escrow_stale", "warning",
|
|
||||||
"A helyreállítási kulcs-letét elavult az offsite jelszó cseréje miatt — futtasd le újra a helyreállítási szertartást (Biztonsági mentés → Helyreállítás).",
|
|
||||||
"", "hub"); serr != nil {
|
|
||||||
p.logf("[offsite] WARN save escrow_stale event for %s: %v", customerID, serr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -211,12 +211,27 @@ func TestReissue_RefusesAmbiguousLookup(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// v0.57.0 (2.3, escrow honesty) — re-issuing offsite credentials INVALIDATES the key-escrow blob:
|
// SCENARIO C (R-196 / R-204 item 2, hub v0.95.0) — re-issuing the offsite PROVIDER credential must
|
||||||
// the blob sealed the OLD repo password, so a recovery code minted against it would decrypt a
|
// LEAVE A HEALTHY ESCROW ALONE.
|
||||||
// password that no longer opens the repo. RED-PROOF (Scenario C): on pre-fix code (no MarkEscrowStale
|
//
|
||||||
// in ReissueCredentials + no stale plumbing) the hub keeps advertising the escrow as current after a
|
// THIS TEST REPLACES `TestReissue_InvalidatesEscrow`, which asserted the opposite and was wrong on a
|
||||||
// re-issue and keeps serving its restic-hash for auto-confirm — this test asserts it does NEITHER.
|
// false premise: a re-issue resets the storage-provider account password and cannot touch the restic
|
||||||
func TestReissue_InvalidatesEscrow(t *testing.T) {
|
// REPOSITORY password, which is generated on the box and never leaves it except sealed under R. The
|
||||||
|
// old assertion looked right because the EFFECT it checked (a stale escrow) was real — nobody checked
|
||||||
|
// the CAUSE. Kept as one test rather than deleted so the inversion is visible in `git log -L`.
|
||||||
|
//
|
||||||
|
// WHAT IT ASSERTS IS THE CONSEQUENCE, not the mechanism: not "MarkEscrowStale was not called" but
|
||||||
|
// "the ACK still carries the hash", because the hash is what the controller's auto-confirm needs and
|
||||||
|
// its absence is what blocked every off-site backup during the 2026-08-04 drill.
|
||||||
|
//
|
||||||
|
// RED-PROOF: restore the marking — put back
|
||||||
|
//
|
||||||
|
// if host, _ := p.Store.GetHostByCustomer(customerID); host != nil {
|
||||||
|
// if esc, _ := p.Store.GetHostEscrow(host.HostID); esc != nil { p.Store.MarkEscrowStale(host.HostID) }
|
||||||
|
// }
|
||||||
|
//
|
||||||
|
// in ReissueCredentials → es.Stale becomes true, the hash is withheld, and this test fails on both.
|
||||||
|
func TestReissue_DoesNotMarkAHealthyEscrowStale(t *testing.T) {
|
||||||
p, _, st := newTestProvisioner(t)
|
p, _, st := newTestProvisioner(t)
|
||||||
const cust = "cust-esc"
|
const cust = "cust-esc"
|
||||||
if _, err := p.ProvisionOffsite(context.Background(), cust, Input{Enabled: true, Type: "shared", QuotaGB: 10}); err != nil {
|
if _, err := p.ProvisionOffsite(context.Background(), cust, Input{Enabled: true, Type: "shared", QuotaGB: 10}); err != nil {
|
||||||
@@ -226,10 +241,9 @@ func TestReissue_InvalidatesEscrow(t *testing.T) {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
|
if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
// Before re-issue: current escrow — the hub serves the sealed hash and is NOT stale.
|
|
||||||
es, err := st.GetEscrowStatusForCustomer(cust)
|
es, err := st.GetEscrowStatusForCustomer(cust)
|
||||||
if err != nil || es == nil {
|
if err != nil || es == nil {
|
||||||
t.Fatalf("escrow status (before): %v", err)
|
t.Fatalf("escrow status (before): %v", err)
|
||||||
@@ -238,25 +252,56 @@ func TestReissue_InvalidatesEscrow(t *testing.T) {
|
|||||||
t.Fatalf("pre-reissue escrow must be current: stale=%v hash=%q", es.Stale, es.ResticPwSHA256)
|
t.Fatalf("pre-reissue escrow must be current: stale=%v hash=%q", es.Stale, es.ResticPwSHA256)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Re-issue the offsite credential — the repo password just changed under the sealed blob.
|
// Re-issue the offsite PROVIDER credential. The box still holds its repo_password file, so the
|
||||||
|
// repository password does NOT change — this is the ordinary re-issue shape.
|
||||||
if err := p.ReissueCredentials(context.Background(), cust, "shared"); err != nil {
|
if err := p.ReissueCredentials(context.Background(), cust, "shared"); err != nil {
|
||||||
t.Fatalf("reissue: %v", err)
|
t.Fatalf("reissue: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// After: the escrow is STALE and the restic-hash is WITHHELD (no auto-confirm against a dead key).
|
|
||||||
es, err = st.GetEscrowStatusForCustomer(cust)
|
es, err = st.GetEscrowStatusForCustomer(cust)
|
||||||
if err != nil || es == nil {
|
if err != nil || es == nil {
|
||||||
t.Fatalf("escrow status (after): %v", err)
|
t.Fatalf("escrow status (after): %v", err)
|
||||||
}
|
}
|
||||||
if !es.Stale {
|
if es.Stale {
|
||||||
t.Fatal("RED-PROOF: escrow must be STALE after an offsite re-issue (the hub was advertising ceremony-done against a key the repo no longer accepts)")
|
t.Fatal("a re-issue marked a HEALTHY escrow stale — this blocks every off-site backup and asks the customer for a ceremony that would supersede a good key (R-196)")
|
||||||
}
|
}
|
||||||
if es.ResticPwSHA256 != "" {
|
if es.ResticPwSHA256 != "OLDHASH" {
|
||||||
t.Fatalf("a stale escrow must WITHHOLD the restic hash to inhibit auto-confirm, got %q", es.ResticPwSHA256)
|
t.Fatalf("the sealed hash must keep flowing to the ACK so the controller can auto-confirm; got %q", es.ResticPwSHA256)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The customer is still TOLD the credential moved — removing the false alarm must not remove the
|
||||||
|
// true notice. `offsite_reissued` is untouched by this change.
|
||||||
|
ev, eerr := st.GetLatestEventByType(cust, "offsite_reissued")
|
||||||
|
if eerr != nil || ev == nil {
|
||||||
|
t.Fatalf("offsite_reissued must still fire on every re-issue (err=%v ev=%+v)", eerr, ev)
|
||||||
|
}
|
||||||
|
// …and the false alarm is gone.
|
||||||
|
if bad, _ := st.GetLatestEventByType(cust, "escrow_stale"); bad != nil {
|
||||||
|
t.Fatalf("a re-issue must not raise escrow_stale on a healthy escrow: %+v", bad)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The other half of Scenario C: the stale MECHANISM is intact and still does its job when something
|
||||||
|
// legitimately sets it. Without this, "nothing marks stale any more" could decay into "stale no longer
|
||||||
|
// works", and the next evidential caller would ship inert (the seam-built-but-never-wired shape).
|
||||||
|
func TestEscrowStaleMechanism_StillWithholdsAndClears(t *testing.T) {
|
||||||
|
_, _, st := newTestProvisioner(t)
|
||||||
|
const cust = "cust-stale-mech"
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := st.MarkEscrowStale(cust + "-01"); err != nil {
|
||||||
|
t.Fatalf("MarkEscrowStale: %v", err)
|
||||||
|
}
|
||||||
|
es, _ := st.GetEscrowStatusForCustomer(cust)
|
||||||
|
if es == nil || !es.Stale || es.ResticPwSHA256 != "" {
|
||||||
|
t.Fatalf("a marked escrow must read stale AND withhold the hash: %+v", es)
|
||||||
|
}
|
||||||
// A fresh ceremony (new blob sealing the new password) clears stale + serves the new hash.
|
// A fresh ceremony (new blob sealing the new password) clears stale + serves the new hash.
|
||||||
if _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob-2"), "SHA256:fp", "zero_knowledge", "2026-07-16T01:00:00Z", "NEWHASH"); err != nil {
|
if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("blob-2"), "SHA256:fp", "zero_knowledge", "2026-07-16T01:00:00Z", "NEWHASH"); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
es, _ = st.GetEscrowStatusForCustomer(cust)
|
es, _ = st.GetEscrowStatusForCustomer(cust)
|
||||||
|
|||||||
@@ -25,10 +25,10 @@ func seedRetainedBlob(t *testing.T, st *Store, customerID string) {
|
|||||||
if err := st.UpsertHost(&Host{HostID: hostID, CustomerID: customerID, APIKey: "h"}); err != nil {
|
if err := st.UpsertHost(&Host{HostID: hostID, CustomerID: customerID, APIKey: "h"}); err != nil {
|
||||||
t.Fatalf("upsert host: %v", err)
|
t.Fatalf("upsert host: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow(hostID, []byte("A"), "fpA", "p", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("A"), "fpA", "p", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||||
t.Fatalf("escrow A: %v", err)
|
t.Fatalf("escrow A: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow(hostID, []byte("B"), "fpB", "p", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("B"), "fpB", "p", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||||
t.Fatalf("escrow B: %v", err)
|
t.Fatalf("escrow B: %v", err)
|
||||||
}
|
}
|
||||||
if err := st.DeleteHost(hostID, true); err != nil {
|
if err := st.DeleteHost(hostID, true); err != nil {
|
||||||
|
|||||||
@@ -0,0 +1,168 @@
|
|||||||
|
package store
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-198 (v0.93.0) — the retention must keep the OFF-SITE data key, not only the PBS key.
|
||||||
|
//
|
||||||
|
// WHAT WAS BROKEN, and why these tests are the ones that would have caught it: host_escrow_superseded
|
||||||
|
// shipped with `blob` (the K-escrow / PBS datastore key) and `identity_blob` was added to host_escrow
|
||||||
|
// by a later ALTER and never to the retained table. The restic REPOSITORY password lives inside
|
||||||
|
// identity_blob. So every supersession retained the whole-guest key and destroyed the off-site data
|
||||||
|
// key — the exact secret the retention exists to preserve — and the destroying act is the escrow
|
||||||
|
// ceremony a rebuilt box asks its customer to run.
|
||||||
|
//
|
||||||
|
// The pre-existing TestSaveHostEscrow_RetainsSuperseded was GREEN throughout, because it asserts the
|
||||||
|
// MECHANISM (a retained row exists, with the old K-blob) and not the CONSEQUENCE (the retained row can
|
||||||
|
// still yield a repository password). These assert the consequence.
|
||||||
|
|
||||||
|
// Scenario A — a re-escrow retains BOTH sealed keys.
|
||||||
|
// RED-PROOF: drop `identity_blob` from demoteCurrentEscrowTx's INSERT/SELECT (production behaviour up
|
||||||
|
// to v0.92.0) → the retained row's identity blob is nil → this FAILS.
|
||||||
|
func TestSaveHostEscrow_RetainsIdentityBlob(t *testing.T) {
|
||||||
|
st := newTestStore(t)
|
||||||
|
const h = "hid-1"
|
||||||
|
oldIdentity := []byte("age-blob-sealing-REPO-PASSWORD-OLD")
|
||||||
|
newIdentity := []byte("age-blob-sealing-REPO-PASSWORD-NEW")
|
||||||
|
|
||||||
|
// Generation 1: the K-escrow, then the identity blob — the real order the escrow PUT uses
|
||||||
|
// (SaveHostEscrow, then SaveHostDRBundle).
|
||||||
|
if _, _, err := st.SaveHostEscrow(h, []byte("k-blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := st.SaveHostDRBundle(h, oldIdentity, `{"gen":1}`); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Generation 2 with a DIFFERENT sealed repo password → supersede.
|
||||||
|
sup, prev, err := st.SaveHostEscrow(h, []byte("k-blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !sup {
|
||||||
|
t.Fatal("a different-passphrase upload must supersede")
|
||||||
|
}
|
||||||
|
if prev != "SHA_OLD" {
|
||||||
|
t.Fatalf("prevResticPwSHA256 = %q, want SHA_OLD (R-197 needs the replaced hash)", prev)
|
||||||
|
}
|
||||||
|
if err := st.SaveHostDRBundle(h, newIdentity, `{"gen":2}`); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
retained, err := st.ListSupersededEscrow(h)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(retained) != 1 {
|
||||||
|
t.Fatalf("retained rows = %d, want 1", len(retained))
|
||||||
|
}
|
||||||
|
// THE ASSERTION THIS WHOLE ITEM IS ABOUT: the off-site data key survived the supersession.
|
||||||
|
if retained[0].IdentityBlob == nil {
|
||||||
|
t.Fatal("R-198: the retained row carries NO identity blob — the off-site repository password " +
|
||||||
|
"was destroyed by the ceremony that was supposed to preserve it")
|
||||||
|
}
|
||||||
|
// And it is the PREVIOUS generation's blob, not the one that replaced it. This pins the ordering
|
||||||
|
// dependency named on demoteCurrentEscrowTx: the identity blob is written AFTER SaveHostEscrow, so
|
||||||
|
// the demote sees the old one. If that order ever inverts, the retained bytes would silently be
|
||||||
|
// the new blob filed under the old hash — recoverable-looking and wrong.
|
||||||
|
if !bytes.Equal(retained[0].IdentityBlob, oldIdentity) {
|
||||||
|
t.Fatalf("retained identity blob is not the PREVIOUS generation (got %q) — the demote ran after the overwrite",
|
||||||
|
retained[0].IdentityBlob)
|
||||||
|
}
|
||||||
|
if retained[0].ResticPwSHA256 != "SHA_OLD" || string(retained[0].Blob) != "k-blob-old" {
|
||||||
|
t.Fatalf("retained row is not the old generation: %+v", retained[0])
|
||||||
|
}
|
||||||
|
// Current row unchanged in behaviour: the NEW generation, both blobs.
|
||||||
|
if bundle, berr := st.GetHostDRBundle(h); berr != nil || bundle == nil || !bytes.Equal(bundle.IdentityBlob, newIdentity) {
|
||||||
|
t.Fatalf("current identity blob is not the new one: %+v (%v)", bundle, berr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario B — deleting a host demotes BOTH blobs too. demoteCurrentEscrowTx is shared by the
|
||||||
|
// re-escrow path and the host-delete path; a shared routine proven through one caller is how a fix
|
||||||
|
// gets believed on a path nobody exercised.
|
||||||
|
// RED-PROOF: fix only the re-escrow caller (e.g. carry the column in SaveHostEscrow's own SQL instead
|
||||||
|
// of in the shared routine) → this FAILS while Scenario A passes.
|
||||||
|
func TestDeleteHost_DemotesIdentityBlob(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
const hostID, cust = "hid-del", "cust-del"
|
||||||
|
identity := []byte("age-blob-sealing-REPO-PASSWORD")
|
||||||
|
if err := s.UpsertHost(&Host{HostID: hostID, CustomerID: cust, APIKey: "k"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, _, err := s.SaveHostEscrow(hostID, []byte("k-blob"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := s.SaveHostDRBundle(hostID, identity, `{}`); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.DeleteHost(hostID, true); err != nil {
|
||||||
|
t.Fatalf("DeleteHost: %v", err)
|
||||||
|
}
|
||||||
|
retained, err := s.ListSupersededEscrow(hostID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(retained) != 1 {
|
||||||
|
t.Fatalf("demoted rows = %d, want 1", len(retained))
|
||||||
|
}
|
||||||
|
if !bytes.Equal(retained[0].IdentityBlob, identity) {
|
||||||
|
t.Fatalf("R-198: host delete demoted custody WITHOUT the identity blob (got %q) — the off-site "+
|
||||||
|
"repository password was destroyed by a host delete", retained[0].IdentityBlob)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario C — a legacy host whose current escrow has NO identity blob supersedes cleanly. The column
|
||||||
|
// is nullable on purpose: a NOT NULL constraint here would make the fix block a ceremony, which is a
|
||||||
|
// worse failure than the one it repairs.
|
||||||
|
func TestSaveHostEscrow_SupersedesWithoutIdentityBlob(t *testing.T) {
|
||||||
|
st := newTestStore(t)
|
||||||
|
const h = "hid-legacy"
|
||||||
|
if _, _, err := st.SaveHostEscrow(h, []byte("k-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// No SaveHostDRBundle — a slice-7-era upload.
|
||||||
|
sup, prev, err := st.SaveHostEscrow(h, []byte("k-new"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("a supersession of an identity-less escrow must not fail: %v", err)
|
||||||
|
}
|
||||||
|
if !sup || prev != "SHA_OLD" {
|
||||||
|
t.Fatalf("superseded=%v prev=%q, want true/SHA_OLD", sup, prev)
|
||||||
|
}
|
||||||
|
retained, err := st.ListSupersededEscrow(h)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(retained) != 1 {
|
||||||
|
t.Fatalf("retained rows = %d, want 1", len(retained))
|
||||||
|
}
|
||||||
|
if retained[0].IdentityBlob != nil {
|
||||||
|
t.Fatalf("a legacy row must retain a NULL identity blob, got %q", retained[0].IdentityBlob)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// CountCurrentEscrowWithIdentity is the census §8.1 asks for: which hosts hold an identity blob today
|
||||||
|
// and are therefore protected from the next ceremony by this fix. Asserted rather than eyeballed,
|
||||||
|
// because the report quotes its numbers.
|
||||||
|
func TestCountCurrentEscrowWithIdentity(t *testing.T) {
|
||||||
|
st := newTestStore(t)
|
||||||
|
if _, _, err := st.SaveHostEscrow("with-id", []byte("k"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA1"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := st.SaveHostDRBundle("with-id", []byte("age-blob"), `{}`); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, _, err := st.SaveHostEscrow("without-id", []byte("k"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA2"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
total, withIdentity, err := st.CountCurrentEscrowWithIdentity()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if total != 2 || withIdentity != 1 {
|
||||||
|
t.Fatalf("census = %d/%d, want 2 total / 1 with identity", withIdentity, total)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -11,7 +11,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
|
|||||||
const h = "h1"
|
const h = "h1"
|
||||||
|
|
||||||
// 1st upload (P_old) — nothing to supersede.
|
// 1st upload (P_old) — nothing to supersede.
|
||||||
sup, err := st.SaveHostEscrow(h, []byte("blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD")
|
sup, _, err := st.SaveHostEscrow(h, []byte("blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -20,7 +20,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// 2nd upload (P_new, DIFFERENT sha) — must supersede + retain the old.
|
// 2nd upload (P_new, DIFFERENT sha) — must supersede + retain the old.
|
||||||
sup, err = st.SaveHostEscrow(h, []byte("blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
|
sup, _, err = st.SaveHostEscrow(h, []byte("blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -45,7 +45,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// 3rd upload, SAME sha as current — idempotent (re-ceremony of the same password): NO supersede row.
|
// 3rd upload, SAME sha as current — idempotent (re-ceremony of the same password): NO supersede row.
|
||||||
sup, err = st.SaveHostEscrow(h, []byte("blob-new-2"), "fp-new", "zk", "2026-07-16T01:00:00Z", "SHA_NEW")
|
sup, _, err = st.SaveHostEscrow(h, []byte("blob-new-2"), "fp-new", "zk", "2026-07-16T01:00:00Z", "SHA_NEW")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -15,10 +15,10 @@ func TestDeleteHost_DemotesEscrowNeverDestroys(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
// current escrow = SHA_A, one superseded = SHA_OLD (two uploads with different passphrases).
|
// current escrow = SHA_A, one superseded = SHA_OLD (two uploads with different passphrases).
|
||||||
if _, err := s.SaveHostEscrow(hostID, []byte("blob-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
|
if _, _, err := s.SaveHostEscrow(hostID, []byte("blob-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := s.SaveHostEscrow(hostID, []byte("blob-A"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
|
if _, _, err := s.SaveHostEscrow(hostID, []byte("blob-A"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if n, _ := s.CountSupersededEscrow(hostID); n != 1 {
|
if n, _ := s.CountSupersededEscrow(hostID); n != 1 {
|
||||||
|
|||||||
@@ -57,7 +57,7 @@ func seedHostWithArtifacts(t *testing.T, s *Store, hostID, customerID string) {
|
|||||||
"PK-"+hostID, "ip-"+hostID, hostID); err != nil {
|
"PK-"+hostID, "ip-"+hostID, hostID); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := s.SaveHostEscrow(hostID, []byte("opaque-escrow"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
|
if _, _, err := s.SaveHostEscrow(hostID, []byte("opaque-escrow"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+126
-20
@@ -374,6 +374,22 @@ func (s *Store) migrate() error {
|
|||||||
-- "history destroyed" into "history recoverable with the recovery code"). Append-only; the hub
|
-- "history destroyed" into "history recoverable with the recovery code"). Append-only; the hub
|
||||||
-- never decrypts; NO pruning (the blobs are tiny + R-encrypted; custody unchanged). The ACK and
|
-- never decrypts; NO pruning (the blobs are tiny + R-encrypted; custody unchanged). The ACK and
|
||||||
-- restore-serving read host_escrow (the CURRENT row) — never this table.
|
-- restore-serving read host_escrow (the CURRENT row) — never this table.
|
||||||
|
--
|
||||||
|
-- THE RULING ABOVE WAS NOT MET FOR TWO MONTHS, AND THIS IS THE RECORD OF IT (R-198, fixed
|
||||||
|
-- v0.93.0). This table shipped with the blob column — the K-escrow, i.e. the PBS datastore key
|
||||||
|
-- — and identity_blob was added to host_escrow LATER (the slice-10D ALTER below), never here.
|
||||||
|
-- The offsite restic REPOSITORY password lives in identity_blob, not in blob. So the retention
|
||||||
|
-- preserved the whole-guest key and silently dropped the off-site data key: precisely the
|
||||||
|
-- secret the reinstall-orphan incident was about. Worse, the copy happens as the new blob
|
||||||
|
-- overwrites the old, so the destroying act was the ESCROW CEREMONY — the exact thing a
|
||||||
|
-- rebuilt box asks its customer to run, on a card promising the old backups stay recoverable.
|
||||||
|
-- Both demo boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08) and their previous
|
||||||
|
-- repository passwords are unrecoverable, recovery code or not.
|
||||||
|
-- identity_blob is now carried (see demoteCurrentEscrowTx, which is still THE ONE row-copy
|
||||||
|
-- routine). Pinned by TestSaveHostEscrow_RetainsIdentityBlob and
|
||||||
|
-- TestDeleteHost_DemotesIdentityBlob — the routine is proven through BOTH of its callers,
|
||||||
|
-- because a shared routine tested through one caller is how a fix gets believed on a path
|
||||||
|
-- nobody exercised. Evidence: audits/RECON-offsite-dr-chain-2026-08-04.md §7.
|
||||||
CREATE TABLE IF NOT EXISTS host_escrow_superseded (
|
CREATE TABLE IF NOT EXISTS host_escrow_superseded (
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
host_id TEXT NOT NULL,
|
host_id TEXT NOT NULL,
|
||||||
@@ -382,7 +398,8 @@ func (s *Store) migrate() error {
|
|||||||
posture TEXT NOT NULL DEFAULT '',
|
posture TEXT NOT NULL DEFAULT '',
|
||||||
created_at DATETIME NOT NULL,
|
created_at DATETIME NOT NULL,
|
||||||
restic_pw_sha256 TEXT NOT NULL DEFAULT '',
|
restic_pw_sha256 TEXT NOT NULL DEFAULT '',
|
||||||
superseded_at DATETIME NOT NULL DEFAULT (datetime('now'))
|
superseded_at DATETIME NOT NULL DEFAULT (datetime('now')),
|
||||||
|
identity_blob BLOB
|
||||||
);
|
);
|
||||||
CREATE INDEX IF NOT EXISTS idx_host_escrow_superseded_host ON host_escrow_superseded(host_id);
|
CREATE INDEX IF NOT EXISTS idx_host_escrow_superseded_host ON host_escrow_superseded(host_id);
|
||||||
|
|
||||||
@@ -414,16 +431,28 @@ func (s *Store) migrate() error {
|
|||||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN identity_blob BLOB`)
|
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN identity_blob BLOB`)
|
||||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN directive_json TEXT NOT NULL DEFAULT '{}'`)
|
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN directive_json TEXT NOT NULL DEFAULT '{}'`)
|
||||||
|
|
||||||
|
// R-198 (v0.93.0) — the SAME column on the RETAINED table. It was added above and not here, and
|
||||||
|
// that omission is what made the retention keep the wrong key for two months (see the comment on
|
||||||
|
// host_escrow_superseded). Additive and tolerated on re-run, exactly like the lines above; it
|
||||||
|
// changes no existing row. Rows superseded BEFORE this ships were written without the identity
|
||||||
|
// blob and their source rows are already overwritten — there is nothing to backfill, and the
|
||||||
|
// v0.93.0 report records that as a looked-at fact rather than a deduction.
|
||||||
|
s.db.Exec(`ALTER TABLE host_escrow_superseded ADD COLUMN identity_blob BLOB`)
|
||||||
|
|
||||||
// SLICE 3 (escrow auto-confirm) — sha256 hex of the offsite restic repo password sealed in the
|
// SLICE 3 (escrow auto-confirm) — sha256 hex of the offsite restic repo password sealed in the
|
||||||
// identity blob. The hash of a 256-bit random secret is non-reversible/non-brute-forceable — safe to
|
// identity blob. The hash of a 256-bit random secret is non-reversible/non-brute-forceable — safe to
|
||||||
// store and serve; it lets the controller VERIFY "the escrow covers the CURRENT repo password"
|
// store and serve; it lets the controller VERIFY "the escrow covers the CURRENT repo password"
|
||||||
// instead of trusting blob-presence. NULL/'' = a legacy or password-less blob (never auto-confirms).
|
// instead of trusting blob-presence. NULL/'' = a legacy or password-less blob (never auto-confirms).
|
||||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN restic_pw_sha256 TEXT`)
|
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN restic_pw_sha256 TEXT`)
|
||||||
|
|
||||||
// v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at is set when the offsite repo
|
// v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at marks an escrow blob that may no
|
||||||
// password is re-issued: the blob then seals a password that no longer opens the repo, so the
|
// longer cover the box's live repository password. While set, the hub stops advertising "ceremony
|
||||||
// hub must stop advertising "ceremony done" and withhold the (now non-matching) restic_pw_sha256
|
// done" and withholds restic_pw_sha256 from the auto-confirm ACK. NULL = current; a fresh ceremony
|
||||||
// from the auto-confirm ACK. NULL = current; a fresh ceremony (SaveHostEscrow) clears it.
|
// (SaveHostEscrow) clears it.
|
||||||
|
// ⚠ NOTHING SETS IT as of v0.95.0 (R-196 / R-204 item 2). The only writer was the PRECAUTIONARY
|
||||||
|
// mark on offsite re-issue, which guessed rather than measured and so blocked off-site backups on
|
||||||
|
// boxes whose key had not changed. The column and its readers stay; see MarkEscrowStale for what a
|
||||||
|
// legitimate future writer would have to prove first.
|
||||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN stale_at DATETIME`)
|
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN stale_at DATETIME`)
|
||||||
|
|
||||||
// dr_recipe (SPIKE-dr-recipe-2026-06-16): the secret-free DR reconstruction recipe, stored
|
// dr_recipe (SPIKE-dr-recipe-2026-06-16): the secret-free DR reconstruction recipe, stored
|
||||||
@@ -2262,6 +2291,28 @@ func (s *Store) GetHostByCustomer(customerID string) (*Host, error) {
|
|||||||
return h, err
|
return h, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// HasEverBoundHost reports whether a machine was EVER bound to this customer — a live row in
|
||||||
|
// `hosts` OR a tombstone in `host_deletions`. It answers "was anything ever expected of this
|
||||||
|
// customer", which is the question the deadline verdicts actually need (R-195).
|
||||||
|
//
|
||||||
|
// It is deliberately NOT "has a report arrived", and the distinction is the whole point: a box
|
||||||
|
// that was installed, bound, and then went silent IS bound, and its silence is a real fault that
|
||||||
|
// must keep alarming. Only a customer that never had a machine at all is UNKNOWN.
|
||||||
|
//
|
||||||
|
// `host_deletions` is included because a customer whose host was removed HAD one — the deadline
|
||||||
|
// caller reaches its down-skip for that shape, and this predicate must not quietly take over a
|
||||||
|
// judgement the staleness checker owns.
|
||||||
|
func (s *Store) HasEverBoundHost(customerID string) (bool, error) {
|
||||||
|
var n int
|
||||||
|
if err := s.db.QueryRow(
|
||||||
|
`SELECT EXISTS(SELECT 1 FROM hosts WHERE customer_id = ?)
|
||||||
|
OR EXISTS(SELECT 1 FROM host_deletions WHERE customer_id = ?)`,
|
||||||
|
customerID, customerID).Scan(&n); err != nil {
|
||||||
|
return false, err
|
||||||
|
}
|
||||||
|
return n != 0, nil
|
||||||
|
}
|
||||||
|
|
||||||
// ListHostsByCustomer returns the customer's hosts ordered by host_id (v0.47.0 — the
|
// ListHostsByCustomer returns the customer's hosts ordered by host_id (v0.47.0 — the
|
||||||
// customer page's Host tab is a LIST by design: 1 host today, N for a later HA cluster).
|
// customer page's Host tab is a LIST by design: 1 host today, N for a later HA cluster).
|
||||||
// Uses the idx_hosts_customer index.
|
// Uses the idx_hosts_customer index.
|
||||||
@@ -2507,6 +2558,11 @@ type HostEscrow struct {
|
|||||||
// StaleAt (v0.57.0, 2.3) — non-empty when the offsite password was re-issued after this blob was
|
// StaleAt (v0.57.0, 2.3) — non-empty when the offsite password was re-issued after this blob was
|
||||||
// sealed: the blob is stale (seals a password that no longer opens the repo). Cleared by a fresh ceremony.
|
// sealed: the blob is stale (seals a password that no longer opens the repo). Cleared by a fresh ceremony.
|
||||||
StaleAt string
|
StaleAt string
|
||||||
|
// IdentityBlob (R-198, v0.93.0) — the age-wrapped identity bundle, which is where the offsite
|
||||||
|
// restic REPOSITORY password lives. Populated by ListSupersededEscrow so a retained blob is
|
||||||
|
// reachable from Go at all; nil for pre-v0.93.0 retained rows and for hosts that never uploaded
|
||||||
|
// one. Opaque: useless without the customer's recovery code, which the hub never holds.
|
||||||
|
IdentityBlob []byte
|
||||||
}
|
}
|
||||||
|
|
||||||
// SaveHostEscrow stores (last-write-wins) the OPAQUE escrow blob for a host. The hub keeps the
|
// SaveHostEscrow stores (last-write-wins) the OPAQUE escrow blob for a host. The hub keeps the
|
||||||
@@ -2516,16 +2572,35 @@ type HostEscrow struct {
|
|||||||
// host_escrow_superseded before overwriting the current row (Part B, v0.60.0). A same-sha re-upload
|
// host_escrow_superseded before overwriting the current row (Part B, v0.60.0). A same-sha re-upload
|
||||||
// (idempotent re-ceremony of the same password) refreshes the current row and does NOT create a
|
// (idempotent re-ceremony of the same password) refreshes the current row and does NOT create a
|
||||||
// superseded row.
|
// superseded row.
|
||||||
|
//
|
||||||
|
// R-197 (v0.93.0): it also returns prevResticPwSHA256 — the hash the row being replaced sealed ("" when
|
||||||
|
// no row existed, or when it was a legacy hash-less blob). Both halves of "did the box's offsite DATA
|
||||||
|
// key change?" have been in this database since SLICE 3 and NOTHING compared them; demo-felhom's key
|
||||||
|
// changed on 2026-08-03 and no signal of any kind fired for thirteen hours. The comparison is the
|
||||||
|
// caller's (api.handleHostEscrowPut) because the event needs the customer id; returning the value is
|
||||||
|
// this function's part. The VALUE is a non-reversible hash of a 256-bit random secret and is never
|
||||||
|
// logged, mailed or written to a report — see the caller.
|
||||||
// demoteCurrentEscrowTx copies the host's CURRENT host_escrow row (if any) into
|
// demoteCurrentEscrowTx copies the host's CURRENT host_escrow row (if any) into
|
||||||
// host_escrow_superseded as a retained blob, inside the given tx. This is THE ONE escrow row-copy
|
// host_escrow_superseded as a retained blob, inside the given tx. This is THE ONE escrow row-copy
|
||||||
// routine (v0.60.0): SaveHostEscrow uses it to retain a superseded different-passphrase blob before
|
// routine (v0.60.0): SaveHostEscrow uses it to retain a superseded different-passphrase blob before
|
||||||
// overwriting, and DeleteHost (v0.60.1) uses it to DEMOTE the current blob to retained custody
|
// overwriting, and DeleteHost (v0.60.1) uses it to DEMOTE the current blob to retained custody
|
||||||
// instead of destroying it. Returns the number of rows copied (0 when the host has no current row).
|
// instead of destroying it. Returns the number of rows copied (0 when the host has no current row).
|
||||||
// The hub never decrypts; custody is unchanged.
|
// The hub never decrypts; custody is unchanged.
|
||||||
|
//
|
||||||
|
// R-198 (v0.93.0): `identity_blob` is copied too. It was omitted from this SELECT for two months, so
|
||||||
|
// every supersession retained the PBS datastore key and destroyed the offsite restic repository
|
||||||
|
// password — the one secret the retention exists to preserve. Copying more opaque bytes gains the hub
|
||||||
|
// NO knowledge: it still has no recovery code and no decrypt path.
|
||||||
|
//
|
||||||
|
// ORDERING THIS DEPENDS ON, stated because it is load-bearing and invisible from here: the identity
|
||||||
|
// blob is written by SaveHostDRBundle AFTER SaveHostEscrow returns (api/handler.go, the escrow PUT),
|
||||||
|
// so at demote time host_escrow still holds the OLD identity blob. If that order ever changes, this
|
||||||
|
// routine silently retains the NEW blob under the OLD blob's hash — pinned by
|
||||||
|
// TestSaveHostEscrow_RetainsIdentityBlob, which asserts the retained bytes are the previous ones.
|
||||||
func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
|
func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
|
||||||
res, err := tx.Exec(`
|
res, err := tx.Exec(`
|
||||||
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
|
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at, identity_blob)
|
||||||
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
|
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now'), identity_blob
|
||||||
FROM host_escrow WHERE host_id = ?`, hostID)
|
FROM host_escrow WHERE host_id = ?`, hostID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return 0, err
|
||||||
@@ -2533,10 +2608,10 @@ func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
|
|||||||
return res.RowsAffected()
|
return res.RowsAffected()
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, err error) {
|
func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, prevResticPwSHA256 string, err error) {
|
||||||
tx, err := s.db.Begin()
|
tx, err := s.db.Begin()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false, err
|
return false, "", err
|
||||||
}
|
}
|
||||||
defer func() {
|
defer func() {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -2555,11 +2630,14 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
|
|||||||
exists = false
|
exists = false
|
||||||
default:
|
default:
|
||||||
err = scanErr
|
err = scanErr
|
||||||
return false, err
|
return false, "", err
|
||||||
|
}
|
||||||
|
if exists {
|
||||||
|
prevResticPwSHA256 = curSHA // R-197: the caller compares; "" = no row or a legacy hash-less blob
|
||||||
}
|
}
|
||||||
if exists && curSHA != resticPwSHA256 {
|
if exists && curSHA != resticPwSHA256 {
|
||||||
if _, err = demoteCurrentEscrowTx(tx, hostID); err != nil {
|
if _, err = demoteCurrentEscrowTx(tx, hostID); err != nil {
|
||||||
return false, err
|
return false, prevResticPwSHA256, err
|
||||||
}
|
}
|
||||||
superseded = true
|
superseded = true
|
||||||
}
|
}
|
||||||
@@ -2576,10 +2654,10 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
|
|||||||
stale_at = NULL,
|
stale_at = NULL,
|
||||||
updated_at = datetime('now')`,
|
updated_at = datetime('now')`,
|
||||||
hostID, blob, keyFingerprint, posture, createdAt, resticPwSHA256); err != nil {
|
hostID, blob, keyFingerprint, posture, createdAt, resticPwSHA256); err != nil {
|
||||||
return false, err
|
return false, prevResticPwSHA256, err
|
||||||
}
|
}
|
||||||
err = tx.Commit()
|
err = tx.Commit()
|
||||||
return superseded, err
|
return superseded, prevResticPwSHA256, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// CountSupersededEscrow returns how many retained (superseded) escrow blobs the hub holds for a host
|
// CountSupersededEscrow returns how many retained (superseded) escrow blobs the hub holds for a host
|
||||||
@@ -2594,7 +2672,7 @@ func (s *Store) CountSupersededEscrow(hostID string) (int, error) {
|
|||||||
// first. Opaque bytes — the hub never decrypts. Seeds the future guided-recovery flow (R-26).
|
// first. Opaque bytes — the hub never decrypts. Seeds the future guided-recovery flow (R-26).
|
||||||
func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
|
func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
|
||||||
rows, err := s.db.Query(`
|
rows, err := s.db.Query(`
|
||||||
SELECT host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at
|
SELECT host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at, identity_blob
|
||||||
FROM host_escrow_superseded WHERE host_id = ? ORDER BY id DESC`, hostID)
|
FROM host_escrow_superseded WHERE host_id = ? ORDER BY id DESC`, hostID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -2603,7 +2681,7 @@ func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
|
|||||||
var out []HostEscrow
|
var out []HostEscrow
|
||||||
for rows.Next() {
|
for rows.Next() {
|
||||||
var e HostEscrow
|
var e HostEscrow
|
||||||
if err := rows.Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.ResticPwSHA256, &e.UpdatedAt); err != nil {
|
if err := rows.Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.ResticPwSHA256, &e.UpdatedAt, &e.IdentityBlob); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
out = append(out, e)
|
out = append(out, e)
|
||||||
@@ -2611,9 +2689,33 @@ func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
|
|||||||
return out, rows.Err()
|
return out, rows.Err()
|
||||||
}
|
}
|
||||||
|
|
||||||
// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3) — called when the offsite repo
|
// CountCurrentEscrowWithIdentity returns how many hosts hold a CURRENT escrow row, and how many of
|
||||||
// password is re-issued, because the blob then seals a password that no longer opens the repo. No-op
|
// those carry an identity blob — i.e. the population whose offsite repository password R-198's fix
|
||||||
// when no escrow row exists; idempotent (only stamps the first re-issue since the last ceremony; a
|
// now protects from the next ceremony, and the remainder for whom there is nothing to protect
|
||||||
|
// because no identity blob was ever uploaded. Read-only; no blob or hash leaves this call.
|
||||||
|
func (s *Store) CountCurrentEscrowWithIdentity() (total, withIdentity int, err error) {
|
||||||
|
err = s.db.QueryRow(`
|
||||||
|
SELECT COUNT(*), COALESCE(SUM(identity_blob IS NOT NULL), 0) FROM host_escrow`).Scan(&total, &withIdentity)
|
||||||
|
return total, withIdentity, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3).
|
||||||
|
//
|
||||||
|
// ⚠ IT HAS NO CALLER as of hub v0.95.0 (R-196 / R-204 item 2), and that is deliberate, not an
|
||||||
|
// oversight. Its ONE caller was `offsite.ReissueCredentials`, which called it on every re-issue on
|
||||||
|
// the PRECAUTIONARY grounds that the box's re-apply might mint a fresh repository password. It
|
||||||
|
// usually does not, so the call marked healthy escrows stale — and because a stale flag WITHHOLDS
|
||||||
|
// restic_pw_sha256 from the ACK (GetEscrowStatusForCustomer, below), it blocked every off-site backup
|
||||||
|
// on those boxes and asked the customer for a ceremony that would supersede a perfectly good key. The
|
||||||
|
// full reasoning, and the two measured signals that cover the real case, are at
|
||||||
|
// offsite.ReissueCredentials.
|
||||||
|
//
|
||||||
|
// KEPT, not deleted, because the FLAG is still live and correct — `stale_at` is read by the ACK, the
|
||||||
|
// operator card and the PBS-DR view, and a future EVIDENTIAL caller (one that has measured a key
|
||||||
|
// change rather than guessed at one) is the right way to set it. Pinned by
|
||||||
|
// TestReissue_DoesNotMarkAHealthyEscrowStale: if a caller reappears without that evidence, it fails.
|
||||||
|
//
|
||||||
|
// No-op when no escrow row exists; idempotent (only stamps the first mark since the last ceremony; a
|
||||||
// fresh ceremony clears stale_at via SaveHostEscrow's ON CONFLICT).
|
// fresh ceremony clears stale_at via SaveHostEscrow's ON CONFLICT).
|
||||||
func (s *Store) MarkEscrowStale(hostID string) error {
|
func (s *Store) MarkEscrowStale(hostID string) error {
|
||||||
_, err := s.db.Exec(`UPDATE host_escrow SET stale_at = datetime('now') WHERE host_id = ? AND stale_at IS NULL`, hostID)
|
_, err := s.db.Exec(`UPDATE host_escrow SET stale_at = datetime('now') WHERE host_id = ? AND stale_at IS NULL`, hostID)
|
||||||
@@ -2643,9 +2745,13 @@ type EscrowStatus struct {
|
|||||||
IdentityBlobPresent bool `json:"identity_blob_present"`
|
IdentityBlobPresent bool `json:"identity_blob_present"`
|
||||||
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
|
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
|
||||||
CreatedAt string `json:"created_at,omitempty"`
|
CreatedAt string `json:"created_at,omitempty"`
|
||||||
// Stale (v0.57.0, 2.3) — true when the offsite password was re-issued after the blob was sealed.
|
// Stale (v0.57.0, 2.3) — true when `stale_at` is stamped on the host's escrow row. ⚠ SINCE hub
|
||||||
|
// v0.95.0 (R-196 / R-204 item 2) NOTHING STAMPS IT: the one caller was the precautionary
|
||||||
|
// re-issue mark, and it was removed for marking healthy escrows stale. See MarkEscrowStale.
|
||||||
// When stale the ResticPwSHA256 is WITHHELD (emptied) so the controller cannot auto-confirm against
|
// When stale the ResticPwSHA256 is WITHHELD (emptied) so the controller cannot auto-confirm against
|
||||||
// a hash that no longer matches the live repo password — the ceremony must run again.
|
// a hash that may no longer match the live repo password — the ceremony must run again. That
|
||||||
|
// withholding is exactly why the precautionary caller had to go: it BLINDED the controller's own
|
||||||
|
// hash comparison, which is the measurement that actually detects a changed repository password.
|
||||||
Stale bool `json:"escrow_stale,omitempty"`
|
Stale bool `json:"escrow_stale,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -74,11 +74,11 @@ func seedDeletable(t *testing.T, st *store.Store, customerID string) string {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi-" + hostID, LastReportAt: &long}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi-" + hostID, LastReportAt: &long}); err != nil {
|
||||||
t.Fatalf("seed host: %v", err)
|
t.Fatalf("seed host: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||||
t.Fatalf("seed escrow A: %v", err)
|
t.Fatalf("seed escrow A: %v", err)
|
||||||
}
|
}
|
||||||
// A second save supersedes A → one RETAINED blob + one CURRENT blob before the cascade runs.
|
// A second save supersedes A → one RETAINED blob + one CURRENT blob before the cascade runs.
|
||||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||||
t.Fatalf("seed escrow B: %v", err)
|
t.Fatalf("seed escrow B: %v", err)
|
||||||
}
|
}
|
||||||
if err := st.SaveOneTimeSecret(customerID, "one-time-pw"); err != nil {
|
if err := st.SaveOneTimeSecret(customerID, "one-time-pw"); err != nil {
|
||||||
|
|||||||
@@ -37,10 +37,10 @@ func seedResettable(t *testing.T, st *store.Store, customerID string) {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi"}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi"}); err != nil {
|
||||||
t.Fatalf("seed host: %v", err)
|
t.Fatalf("seed host: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||||
t.Fatalf("seed escrow A: %v", err)
|
t.Fatalf("seed escrow A: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||||
t.Fatalf("seed escrow B: %v", err)
|
t.Fatalf("seed escrow B: %v", err)
|
||||||
}
|
}
|
||||||
if err := st.DeleteHost(hostID, true); err != nil { // demotes current → retained; records host_deletions
|
if err := st.DeleteHost(hostID, true); err != nil { // demotes current → retained; records host_deletions
|
||||||
|
|||||||
@@ -64,7 +64,7 @@ func TestHostDelete_EscrowAckRequired(t *testing.T) {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: "esc-host", CustomerID: "c2", APIKey: "k"}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: "esc-host", CustomerID: "c2", APIKey: "k"}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow("esc-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
if _, _, err := st.SaveHostEscrow("esc-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -110,7 +110,7 @@ func TestHostDelete_ImpactJSON(t *testing.T) {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: "imp-host", CustomerID: "c4", APIKey: "SECRET-KEY"}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: "imp-host", CustomerID: "c4", APIKey: "SECRET-KEY"}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow("imp-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
if _, _, err := st.SaveHostEscrow("imp-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := st.UpsertGuestFromReport(&store.Guest{GuestID: store.GuestID("imp-host", 100),
|
if err := st.UpsertGuestFromReport(&store.Guest{GuestID: store.GuestID("imp-host", 100),
|
||||||
@@ -159,7 +159,7 @@ func TestHostDelete_HappyPath(t *testing.T) {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: "dr-drill", CustomerID: "c5", APIKey: "k"}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: "dr-drill", CustomerID: "c5", APIKey: "k"}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow("dr-drill", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
if _, _, err := st.SaveHostEscrow("dr-drill", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
rr := postHostDelete(t, s, "dr-drill", url.Values{
|
rr := postHostDelete(t, s, "dr-drill", url.Values{
|
||||||
|
|||||||
@@ -130,7 +130,7 @@ func TestHandleHostDetail(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
// DR + escrow present (escrow row must exist before the DR bundle UPDATE).
|
// DR + escrow present (escrow row must exist before the DR bundle UPDATE).
|
||||||
if _, err := st.SaveHostEscrow("demo-felhom-01", []byte("opaque-escrow"), "fp", "posture", "2026-06-01T00:00:00Z", ""); err != nil {
|
if _, _, err := st.SaveHostEscrow("demo-felhom-01", []byte("opaque-escrow"), "fp", "posture", "2026-06-01T00:00:00Z", ""); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := st.SaveHostDRBundle("demo-felhom-01", []byte("opaque-identity"), `{"v":1}`); err != nil {
|
if err := st.SaveHostDRBundle("demo-felhom-01", []byte("opaque-identity"), `{"v":1}`); err != nil {
|
||||||
|
|||||||
@@ -404,7 +404,7 @@ func TestPBSDR_F14AutoReissueOnAckedDeletion(t *testing.T) {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: "peti-00-dead", CustomerID: "peti", APIKey: "oldkey"}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: "peti-00-dead", CustomerID: "peti", APIKey: "oldkey"}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow("peti-00-dead", []byte("opaque"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
|
if _, _, err := st.SaveHostEscrow("peti-00-dead", []byte("opaque"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := st.DeleteHost("peti-00-dead", true); err != nil {
|
if err := st.DeleteHost("peti-00-dead", true); err != nil {
|
||||||
|
|||||||
+1
-1
@@ -125,7 +125,7 @@ spec:
|
|||||||
spec:
|
spec:
|
||||||
containers:
|
containers:
|
||||||
- name: hub
|
- name: hub
|
||||||
image: gitea.dooplex.hu/admin/felhom-hub:0.91.1
|
image: gitea.dooplex.hu/admin/felhom-hub:0.95.0
|
||||||
ports:
|
ports:
|
||||||
- containerPort: 8080
|
- containerPort: 8080
|
||||||
name: http
|
name: http
|
||||||
|
|||||||
@@ -324,7 +324,7 @@ spec:
|
|||||||
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
||||||
args:
|
args:
|
||||||
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
||||||
- --ref=installer-v1.23.0
|
- --ref=installer-v1.25.0
|
||||||
- --root=/git-scripts
|
- --root=/git-scripts
|
||||||
- --link=current
|
- --link=current
|
||||||
- --period=30s
|
- --period=30s
|
||||||
@@ -369,7 +369,7 @@ spec:
|
|||||||
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
||||||
args:
|
args:
|
||||||
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
||||||
- --ref=installer-v1.23.0
|
- --ref=installer-v1.25.0
|
||||||
- --root=/git-scripts
|
- --root=/git-scripts
|
||||||
- --link=current
|
- --link=current
|
||||||
- --one-time
|
- --one-time
|
||||||
|
|||||||
@@ -1,3 +1,26 @@
|
|||||||
|
## 1.25.0 — the off-site tier stops asking to prune (2026-08-04, R-191)
|
||||||
|
|
||||||
|
**A backup that worked must not report failure.** The off-site tier was written with `keep_last: 2`,
|
||||||
|
so every weekly run uploaded its snapshot successfully and then failed the whole job on a prune the
|
||||||
|
box's token is deliberately refused: `prune 'ct/9201': permission check failed - missing
|
||||||
|
Datastore.Modify|Datastore.Prune` → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the
|
||||||
|
operator's inbox. Every week, on both boxes, about a backup that had already succeeded.
|
||||||
|
|
||||||
|
**R-89 moved off-site pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay
|
||||||
|
write-only, so a box can never delete its own off-site history. The 2026-07-26 "two weeks" ruling was
|
||||||
|
not reversed; where it is ENFORCED moved, and this value did not follow. The tier now writes
|
||||||
|
`keep_last: 0`, which the agent's existing guard (`allowPBSPrune = !primary && keep_last > 0`) already
|
||||||
|
reads as "never prune from the box" — no agent change needed.
|
||||||
|
|
||||||
|
**VERIFIED BEFORE CHANGING IT** (read-only on ep0, 2026-08-04): prune jobs `prune-demo-felhom` and
|
||||||
|
`prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
|
||||||
|
and have run **every day since 2026-07-27 — 18 tasks, all `status=OK`**, the newest showing
|
||||||
|
`retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Disabling
|
||||||
|
the client-side prune without that check would have traded a weekly false alarm for unbounded growth.
|
||||||
|
|
||||||
|
A gate now asserts the off-site tier carries no client-side prune, so the value cannot drift back
|
||||||
|
quietly. The local tier's retention is untouched.
|
||||||
|
|
||||||
## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185)
|
## 1.24.0 — a pre-existing backup target is granted too (2026-08-03, R-185)
|
||||||
|
|
||||||
**`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the
|
**`configure_backup_target` has two arms and only one of them granted.** The Case A arm creates the
|
||||||
|
|||||||
@@ -184,7 +184,7 @@
|
|||||||
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
SCRIPT_VERSION="1.24.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
SCRIPT_VERSION="1.25.0" # the SINGLE version source (F-1): -h and the run banners follow it.
|
||||||
# The hub used to carry a copy for its Setup tab; R-94 DELETED it
|
# The hub used to carry a copy for its Setup tab; R-94 DELETED it
|
||||||
# (2026-08-02) because the hub cannot know which version a box runs —
|
# (2026-08-02) because the hub cannot know which version a box runs —
|
||||||
# the Setup command fetches this script at run time. scripts/
|
# the Setup command fetches this script at run time. scripts/
|
||||||
@@ -2479,8 +2479,30 @@ for _k,_v in {"unit_dir":"/etc/systemd/system","stage_dir":"/var/lib/felhom-agen
|
|||||||
base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20})
|
base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20})
|
||||||
# R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions —
|
# R-82: local DAILY + offsite WEEKLY. The two tiers carry SEPARATE cadences and retentions —
|
||||||
# keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob
|
# keep_last=3 is three DAYS on the daily tier and three WEEKS on a weekly one, so one shared knob
|
||||||
# would guarantee that one of them is wrong. keep_last=2 on the offsite tier = two weeks (operator
|
# would guarantee that one of them is wrong.
|
||||||
# ruling 2026-07-26).
|
#
|
||||||
|
# THE OFFSITE TIER CARRIES NO CLIENT-SIDE RETENTION, AND THAT IS THE POINT (R-191, 2026-08-04).
|
||||||
|
# It used to be written `keep_last: 2` ("two weeks", operator ruling 2026-07-26). **R-89 then moved
|
||||||
|
# offsite pruning SERVER-SIDE** — ep0 runs a per-namespace prune job and box tokens stay write-only,
|
||||||
|
# deliberately, so that a box can never delete its own offsite history. The 2026-07-26 ruling was not
|
||||||
|
# reversed; where it is ENFORCED moved, and this value did not follow.
|
||||||
|
#
|
||||||
|
# The consequence was weekly and wrong in the worst direction: vzdump UPLOADED the snapshot fine and
|
||||||
|
# then failed the whole job on the prune the token is refused —
|
||||||
|
# `prune 'ct/9201': permission check failed - missing Datastore.Modify|Datastore.Prune`
|
||||||
|
# → `TASK ERROR: job errors` → `whole_guest_backup_failed` in the operator's inbox. Every week, on
|
||||||
|
# both boxes, about a backup that had already succeeded. A tier that cries wolf weekly is a tier
|
||||||
|
# whose real failure nobody will see.
|
||||||
|
#
|
||||||
|
# VERIFIED BEFORE CHANGING IT (2026-08-04, read-only on ep0): prune jobs `prune-demo-felhom` and
|
||||||
|
# `prune-demo-hp` exist on datastore `felhom-offsite`, one per namespace, schedule 03:30, keep-last 2,
|
||||||
|
# and have run EVERY DAY since 2026-07-27 — 18 tasks, all `status=OK`, the newest showing
|
||||||
|
# `retention options: --ns demo-felhom --max-depth 0 --keep-last 2` and keeping exactly two. Retention
|
||||||
|
# happens; it happens THERE. **If that ever stops being true, this zero is unbounded growth** — check
|
||||||
|
# ep0's prune jobs before assuming the offsite tier is retained.
|
||||||
|
#
|
||||||
|
# `keep_last: 0` means "never prune from the box" and is the value the agent's own guard reads
|
||||||
|
# (allowPBSPrune = !primary && keep_last > 0), so no agent change is needed to honour it.
|
||||||
#
|
#
|
||||||
# The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only
|
# The offsite tier is written even though `felhom-pbs` does not exist yet: that storage appears only
|
||||||
# when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent
|
# when the hub provisions the DR tier. The agent DEFERS a tier whose target storage is absent
|
||||||
@@ -2491,7 +2513,7 @@ base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_sec
|
|||||||
# setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here —
|
# setdefault: an EXISTING box's backup block is preserved WHOLE. Upgrades never gain the tier here —
|
||||||
# they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start
|
# they are migrated explicitly (R-82 Slice D.2), so an in-place upgrade can never silently start
|
||||||
# writing to an offsite datastore.
|
# writing to an offsite datastore.
|
||||||
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]})
|
base.setdefault('backup', {"local_backup_target":os.environ.get('BACKUP_TARGET_RESOLVED','local'),"local_backup_retention":3,"restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0,"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":0}]})
|
||||||
base.setdefault('local_api', {})
|
base.setdefault('local_api', {})
|
||||||
base['local_api'].setdefault('enable', True)
|
base['local_api'].setdefault('enable', True)
|
||||||
base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR']
|
base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR']
|
||||||
|
|||||||
@@ -233,6 +233,32 @@ else:
|
|||||||
"invisible to the agent, it is never restore-tested, and an empty listing looks exactly "
|
"invisible to the agent, it is never restore-tested, and an empty listing looks exactly "
|
||||||
"like a brand-new tier." % (resolutions, grants))
|
"like a brand-new tier." % (resolutions, grants))
|
||||||
|
|
||||||
|
# ── R-191: the OFFSITE tier must not arm a client-side prune ─────────────────────────────────
|
||||||
|
#
|
||||||
|
# R-89 moved offsite pruning SERVER-SIDE — ep0 runs a per-namespace prune job and box tokens stay
|
||||||
|
# write-only, so the box is REFUSED if it asks. When this default was `keep_last: 2` the effect was a
|
||||||
|
# weekly lie: vzdump uploaded the snapshot, then failed the whole job on the prune, and the operator
|
||||||
|
# was told the offsite backup had failed when it had succeeded.
|
||||||
|
#
|
||||||
|
# The assertion is on the OFFSITE entry only. The local tier's `local_backup_retention` is untouched
|
||||||
|
# and must stay untouched — it prunes correctly and is allowed to.
|
||||||
|
m = re.search(r'"backup_targets":\s*\[(.*?)\]', src, re.S)
|
||||||
|
if not m:
|
||||||
|
fail("cannot find backup_targets in the rendered agent.json defaults — the offsite-retention "
|
||||||
|
"assertion cannot run, and a check that cannot run must never report OK (R-191)")
|
||||||
|
else:
|
||||||
|
targets = m.group(1)
|
||||||
|
kl = re.search(r'"keep_last"\s*:\s*(\d+)', targets)
|
||||||
|
if not kl:
|
||||||
|
fail("the offsite backup_target carries no keep_last at all — expected an explicit 0 "
|
||||||
|
"(R-191: 0 means 'never prune from the box'; absent is not the same statement)")
|
||||||
|
elif kl.group(1) != "0":
|
||||||
|
fail("the offsite backup_target arms a CLIENT-SIDE prune (keep_last=%s). R-89 moved offsite "
|
||||||
|
"pruning server-side to ep0 and box tokens are write-only, so every weekly run will "
|
||||||
|
"upload successfully and then FAIL the job on a refused prune (R-191)." % kl.group(1))
|
||||||
|
else:
|
||||||
|
ok("the offsite tier arms no client-side prune (keep_last=0; retention is ep0's prune jobs)")
|
||||||
|
|
||||||
print()
|
print()
|
||||||
if fails:
|
if fails:
|
||||||
print("hostinstall gates: %d FAILURE(S)" % len(fails))
|
print("hostinstall gates: %d FAILURE(S)" % len(fails))
|
||||||
|
|||||||
Reference in New Issue
Block a user