Compare commits
23 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 5c7d67102d | |||
| d1a8edb332 | |||
| 2a7ac03c47 | |||
| b228fd102d | |||
| 24acc8d816 | |||
| 73fb595e38 | |||
| a0c4b607a6 | |||
| 846253baa8 | |||
| dd089265e8 | |||
| 435f4a5229 | |||
| 9f31956201 | |||
| 40687b0921 | |||
| 91cabdde1b | |||
| 3f2b7bc023 | |||
| d26f49ad68 | |||
| b7c5cfc061 | |||
| f581ac1349 | |||
| 7fff45d688 | |||
| f456835bbc | |||
| e06920d930 | |||
| 813239d8d3 | |||
| a684a81796 | |||
| 71662336aa |
+209
@@ -17,6 +17,215 @@
|
|||||||
|
|
||||||
## Standing rulings
|
## Standing rulings
|
||||||
|
|
||||||
|
**S-31 — THE DRILL PASSED: a customer's file survives a machine rebuild and comes back byte-identical.
|
||||||
|
The capability is proven; the customer JOURNEY is four undocumented manual steps (2026-08-04, R-201/R-204).**
|
||||||
|
|
||||||
|
**Proven on hardware, end to end:** demo-hp's controller data volume was destroyed and the sentinel
|
||||||
|
deleted from disk. The recovery code produced `8a9e33aa4da6…` (byte-identical to the pre-wipe on-disk
|
||||||
|
key AND the hub's independent record); it installed on the bare box; the **existing repository opened**
|
||||||
|
(`repo_state: null`, 3 snapshots, 42 026 B = the pre-wipe size — not a fresh history); and the customer
|
||||||
|
restore flow returned the sentinel at `643166269103a25c…`, **byte-identical**. `identity_blob` was
|
||||||
|
unchanged throughout and **no ceremony was run at any point** (superseded rows still 2).
|
||||||
|
|
||||||
|
**But it took four manual steps that are in no design document (R-204), all measured:**
|
||||||
|
1. a rebuilt controller cannot configure its off-site tier — the one-time password was consumed by its
|
||||||
|
predecessor (`no unconsumed offsite password`, R-193);
|
||||||
|
2. the Re-issue that fixes that sets `stale_at` **while `restic_pw_sha256` is unchanged** (R-196);
|
||||||
|
3. a stale escrow makes the hub withhold the hash from the ACK → `EscrowAutoConfirmer` can never flip
|
||||||
|
`pending → escrowed` → `OffboxRunnable` refuses every run;
|
||||||
|
4. it was cleared with the **manual** confirm (`/backup/offbox/confirm-escrow`), NOT a ceremony — a
|
||||||
|
ceremony would have superseded the identity blob and destroyed the recovered key;
|
||||||
|
5. and the restore's **default `mode=unit` returns the recovery unit, not the customer's files** —
|
||||||
|
`mode=full` is what returns documents. A customer following the obvious path gets the wrong thing
|
||||||
|
and is told nothing.
|
||||||
|
|
||||||
|
**Before any of it, a rebuilt box is unclaimed** and the claim gate intercepts every controller
|
||||||
|
endpoint. **The local escape hatch does not work unaided:** `--print-reset-code` writes the new hash to
|
||||||
|
`settings.json` while the running controller keeps its old copy in memory, so the code never matches —
|
||||||
|
**restart the controller between minting and claiming.**
|
||||||
|
|
||||||
|
*Facts a future session needs:*
|
||||||
|
- **A guest rebuild in this fleet is a controller-DATA-VOLUME loss, not a guest reprovision.** The
|
||||||
|
2026-08-03 incident R-193 is filed against ran with guest 9201 up throughout — no `pct destroy`, no
|
||||||
|
`pct restore`, no `--selftest=provision`. Reproduce it that way.
|
||||||
|
- **A good snapshot is not durable against a later bad run on the same day.** `forget --keep-daily 7
|
||||||
|
--group-by host,tags` keeps one per tag per day; a later, worse snapshot evicts a good one.
|
||||||
|
- **Never run a ceremony while a recovery is in flight** — it supersedes the identity blob. Under
|
||||||
|
v0.93.0 the old blob is retained, but nothing serves a superseded blob back (R-199).
|
||||||
|
|
||||||
|
**S-30 — the R-201 drill was PREPARED and HALTED BEFORE THE WIPE (2026-08-04). Nothing was wiped.**
|
||||||
|
|
||||||
|
It stopped at step 4 because the sentinel file was **not in the off-site snapshot** while the run
|
||||||
|
reported `ok` (**R-203**): the deploy-time `${USERDATA_PATH}` root and the backup-time `NamespaceRoot`
|
||||||
|
disagree for an app whose `HDD_PATH` equals `system_data_path` — and the same compose used both roots.
|
||||||
|
Wiping would have destroyed the only copy of the sentinel and proven nothing.
|
||||||
|
|
||||||
|
*Established live on the way, all firsts, and all worth carrying:*
|
||||||
|
- **A rebuilt box's off-site run REFUSES** — `repo_state: orphaned`, the orphan card, and
|
||||||
|
`offbox_repo_orphaned` at the hub. It does **not** silently start a fresh history. Closes R-193's Q3.
|
||||||
|
- **The orphan reset works**: move-aside to `/home/felhom-repo.orphaned-20260804`, never delete.
|
||||||
|
- **demo-hp's pre-rebuild off-site history is permanently unrecoverable** — its key is in superseded
|
||||||
|
row id 3 with `identity_blob` NULL, superseded **four hours before** v0.93.0 fixed the retention.
|
||||||
|
- **Neither pre-existing off-site-toggled app has a restorable file leg** — both are named-volume-only,
|
||||||
|
which the tier tars but the customer restore never unpacks. Any drill needing a recoverable file
|
||||||
|
needs an app with a **mandatory userdata path**; `calibre-web` is now deployed on demo-hp as that
|
||||||
|
fixture, with the sentinel in place.
|
||||||
|
|
||||||
|
*Still not established, unchanged:* **no file has ever been restored from an off-site backup after a
|
||||||
|
wipe**, and Part 0's install path (controller v0.196.0) has never run against a live recovery.
|
||||||
|
|
||||||
|
**S-29 — a box may fetch its OWN sealed recovery blob with its OWN credential; the operator-driven DR
|
||||||
|
path is a separate thing and stays gated (2026-08-04, R-199; hub v0.94.0 + agent v0.125.0 + controller
|
||||||
|
v0.195.0).**
|
||||||
|
|
||||||
|
**The decision, so it is never re-litigated from scratch.** Self-service recovery needs the box to
|
||||||
|
obtain its own sealed package. The only retrieval that existed was operator-gated behind recovery mode
|
||||||
|
and the global key (`dr.go`), which cannot serve a customer-present flow. `GET /hosts/{id}/escrow` now
|
||||||
|
serves that ONE object to its authenticated owner.
|
||||||
|
|
||||||
|
- **Why it is safe:** the blob is opaque; the hub has never held R and has no decrypt path; a wrong
|
||||||
|
code fails closed at age's scrypt KDF (measured live — `exit status 1`, no plaintext, nothing
|
||||||
|
written); the caller already authenticates as this host for its report, desired state, WG
|
||||||
|
registration and PBS token, so this adds no new identity, only a new object.
|
||||||
|
- **What it changes, which is the whole of the trade:** obtaining the blob used to require the
|
||||||
|
OPERATOR to arm recovery mode. It now needs only that box's credential. **Fewer parties are
|
||||||
|
required.** The mitigation is that the capability is AUDITED, not silent: every retrieval raises
|
||||||
|
`escrow_blob_served` (warning, operator-only), written before the bytes leave.
|
||||||
|
- **The override is one constant.** `api.escrowSelfServiceRetrieval` — false re-imposes the
|
||||||
|
recovery-mode requirement on this endpoint and changes nothing else. Do not scatter the condition.
|
||||||
|
- **`handleReEnroll` / `handleGetRestoreDirective` are UNTOUCHED** and must stay so: they rotate the
|
||||||
|
host API key and serve the K-escrow and the directive as well. Different object, different gate.
|
||||||
|
Pinned by `TestEscrowGet_OperatorDRPathUnchanged`.
|
||||||
|
|
||||||
|
**Facts a future session needs before touching this area:**
|
||||||
|
- **The unseal lives in the AGENT and must stay there.** `age` is an agent runtime dependency
|
||||||
|
(`/usr/bin/age`, present on both demo hosts) and is deliberately absent from the controller image;
|
||||||
|
the blob is host-scoped and its only writer is the agent under the per-host key.
|
||||||
|
- **The agent returns ONE field.** Not the tunnel token, not the PBS token, not the WG key — the
|
||||||
|
controller is a trust tier down. Narrowing cost nothing and is not recoverable later.
|
||||||
|
- **R's rules:** in memory for one call, cleared on every path, never on disk, never in argv, never
|
||||||
|
logged, never echoed. The controller CLI reads it from STDIN for exactly that reason. Verified live:
|
||||||
|
0 log lines, 0 files, 0 leftover staging dirs, with a positive control proving the search worked.
|
||||||
|
- **The chain today: links 1–8 walked, 9–11 not.** The KEY comes back. Nothing installs it, reopens a
|
||||||
|
repository with it, or restores a file — R-200's remaining half and R-201.
|
||||||
|
|
||||||
|
**S-28 — the escrow retention now covers the OFF-SITE data key, and customer-present recovery is the
|
||||||
|
accepted design, which makes that retention load-bearing (2026-08-04, R-198/R-197; hub v0.93.0).**
|
||||||
|
|
||||||
|
**The contract changed, and this is the sentence to carry:** when an escrow blob supersedes another,
|
||||||
|
`demoteCurrentEscrowTx` retains **both** sealed keys — the K-escrow `blob` (PBS datastore key) **and**
|
||||||
|
`identity_blob` (the age-wrapped bundle carrying the restic **repository** password). Until v0.93.0 it
|
||||||
|
retained only the first, so the retention preserved the whole-guest key and destroyed the off-site data
|
||||||
|
key, and **the destroying act was the escrow ceremony a rebuilt box asks its customer to run.**
|
||||||
|
|
||||||
|
*Why this is load-bearing rather than tidy:* the operator ruled on 2026-08-04 that **candidate (c) is
|
||||||
|
refused — no repository password is retained on the Proxmox host.** With no host-retained copy, the
|
||||||
|
**customer-present recovery path is the only way back from a rebuild**, and it runs entirely through
|
||||||
|
the retained identity blob. A session that touches escrow custody is touching the single copy.
|
||||||
|
|
||||||
|
*Facts a future session needs before touching this area:*
|
||||||
|
- **The ordering is load-bearing and invisible from the routine.** `SaveHostDRBundle` writes
|
||||||
|
`identity_blob` **after** `SaveHostEscrow` returns (the escrow PUT, `api/handler.go`), so at demote
|
||||||
|
time `host_escrow` still holds the OLD identity blob. Invert that and the retained bytes become the
|
||||||
|
NEW blob filed under the OLD hash — recoverable-looking and wrong. Pinned by
|
||||||
|
`TestSaveHostEscrow_RetainsIdentityBlob`.
|
||||||
|
- **`demoteCurrentEscrowTx` is shared by re-escrow and host-delete.** Both have a test; fixing one
|
||||||
|
caller instead of the routine is a demonstrated red-proof, not a hypothetical.
|
||||||
|
- **Nothing was backfilled and nothing could be** — pre-v0.93.0 retained rows have no identity blob and
|
||||||
|
their source rows are already overwritten. Both demo boxes' pre-2026-08-04 repository passwords are
|
||||||
|
gone permanently, recovery code or not.
|
||||||
|
- **A retained key is not a recovery.** The hub's blob-serving endpoints have no client (R-199), the
|
||||||
|
injection seam has no form (R-200), and the path has never been exercised (R-201). v0.93.0 makes the
|
||||||
|
key survive; it does not hand it back.
|
||||||
|
- **`offsite_repo_key_changed`** (R-197, warning, operator-only, edge-triggered on a supersession where
|
||||||
|
both hashes are known and differ) is the evidential signal that a box's off-site data key moved. It
|
||||||
|
carries **no hash value**. `MarkEscrowStale` is **precautionary**, not evidential — see S-26(a).
|
||||||
|
|
||||||
|
**S-27 — a customer with NO machine ever bound is UNKNOWN, silently; one that was bound and went quiet
|
||||||
|
still alarms (2026-08-04, R-195; hub v0.92.0).** Operator ruling, implemented as
|
||||||
|
`store.HasEverBoundHost` (live `hosts` row OR `host_deletions` tombstone) consulted once at the top of
|
||||||
|
`CheckBackupDeadlines`' loop. **The discriminator is "was a host EVER bound", never "has a report
|
||||||
|
arrived"** — a box that was installed and never phoned home is a real fault. Fail-**open** on a read
|
||||||
|
error; the deferral is logged with its own counter; the R-81 anchored-verdict structure is untouched.
|
||||||
|
|
||||||
|
*The mechanism is worth carrying, because the guard that should have covered this was keyed off the
|
||||||
|
wrong thing:* the existing down-skip reads `StalenessChecker.GetState()`, whose map is seeded from
|
||||||
|
`store.GetCustomers()` — **a query over the `reports` table**. A customer with zero reports is in no
|
||||||
|
row, gets no state, and `GetState()` returns `""` not `"down"`, so **the skip misses exactly the
|
||||||
|
customer it would most obviously cover.** `david` (created 2026-08-01, no machine) e-mailed an
|
||||||
|
`expected_dbdump_missed` ERROR three mornings running; `peti-felhom` — active, host deleted
|
||||||
|
2026-07-15 — does not, because its 482 old reports make it `down`. Generalise it: **a "skip the dead"
|
||||||
|
guard built on evidence of life cannot see something that was never alive.**
|
||||||
|
|
||||||
|
**S-26 — the one-shot secret is the recoverable one; the irreplaceable one is minted fresh on every
|
||||||
|
guest rebuild (2026-08-04, R-193 spike — `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`).
|
||||||
|
No code shipped for it; the decision is the operator's.**
|
||||||
|
|
||||||
|
Two sentences, because they are what every future session needs before touching this area:
|
||||||
|
|
||||||
|
- **Q1 — what is one-shot.** Of the three secrets in the offsite path, the hub's **one-time
|
||||||
|
provider password** is one-shot but freely re-issuable, the box's **SFTP key** is regenerated on
|
||||||
|
every apply by design, and the **restic repository password** — the DATA key, which the agent's own
|
||||||
|
source calls *"irreplaceable"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the only one
|
||||||
|
nothing can restage.
|
||||||
|
- **Q2 — what a rebuild costs.** A rebuilt controller **mints a brand-new repository password**
|
||||||
|
(`WriteOffboxSecrets`, `offbox.go:392`, generates whenever `<DataDir>/offbox/repo_password` is
|
||||||
|
absent) and **no automatic path ever consults the escrowed one** — `InjectOffboxPassword` has exactly
|
||||||
|
one caller in the whole repo, a web form a human pastes into — so **every guest rebuild orphans the
|
||||||
|
previous off-site history.**
|
||||||
|
|
||||||
|
*Measured without touching a box, and the method is reusable:* `host_escrow.restic_pw_sha256` vs
|
||||||
|
`host_escrow_superseded.restic_pw_sha256`. demo-hp `8e03eddf…`→`8a9e33aa…` (15 snapshots / 40.9 MB
|
||||||
|
orphaned); demo-felhom `48741892…`→`c60c8bc7…` (**36 snapshots / 1.14 GB**). **demo-felhom is the half
|
||||||
|
that matters:** R-193 recorded it as having survived the rebuild by luck, and it did — its *delivery*
|
||||||
|
recovered in 76 s off a stale staged secret. **Its repository did not**, and nothing said so for 13 h.
|
||||||
|
*Luck restored the plumbing, not the data.*
|
||||||
|
|
||||||
|
*Three corrections that must not be re-inherited.* **(a)** `ReissueCredentials` does **not** rotate the
|
||||||
|
restic password — R-39's record, `offsite.go:198-201` and `api/handler.go:1067-1069` all say it does,
|
||||||
|
and all three are wrong (→ **R-196**, the eighth entry in `CLAUDE.md`'s table). **(b)** Candidate (b),
|
||||||
|
*"recoverable from escrow at re-bootstrap"*, is **not implementable** — the escrow is R-wrapped and
|
||||||
|
zero-knowledge (D6), so only a customer-present ceremony can open it, which is the manual form that
|
||||||
|
already exists. **(c)** Candidate (a), *"the hub restages automatically"*, **already exists** as
|
||||||
|
`reissueOnReenroll`'s F3 leg and is wired to the wrong event — it sits behind `handleHostEnroll`'s
|
||||||
|
mint-once-reuse short-circuit, and a **guest** rebuild leaves the `hosts` row intact. **Shipping it
|
||||||
|
first would have made both boxes look healthy on 2026-08-04 while their snapshots were orphaned** —
|
||||||
|
strictly worse than the current loud failure.
|
||||||
|
|
||||||
|
*The candidate nobody had named,* and the only one aimed at the actual harm: **the agent survives a
|
||||||
|
guest rebuild**, already receives the repo password over the pinned local API
|
||||||
|
(`POST /escrow/stage-secret`) and already writes it to a fixed 0600 path — it merely **wipes** it after
|
||||||
|
the ceremony. Retaining and serving it back needs no new seam. Its price is one real trade: a copy of
|
||||||
|
the data key at rest on the Proxmox host. **That trade is the operator's to make and the spike does not
|
||||||
|
make it.**
|
||||||
|
|
||||||
|
**S-24 — offsite retention is ep0's, and the box asks for none (2026-08-04, R-191; installer 1.25.0).**
|
||||||
|
R-89 moved offsite pruning server-side and box tokens stay write-only. The 2026-07-26 "two weeks"
|
||||||
|
ruling was not reversed — **where it is ENFORCED moved, and the installer's `keep_last: 2` did not
|
||||||
|
follow**, so every weekly run uploaded fine and then failed the job on a refused prune. The offsite
|
||||||
|
tier now writes `keep_last: 0`, which the agent's existing guard already reads as *never prune from
|
||||||
|
the box*; the local tier is untouched. A gate asserts it. **Before switching it off, ep0 was read:**
|
||||||
|
both namespaces have a prune job at 03:30 keep-last 2 that has run daily since 2026-07-27 (18 tasks,
|
||||||
|
all OK). **If that ever stops, `keep_last: 0` is unbounded growth** — check ep0's prune jobs before
|
||||||
|
assuming the offsite tier is retained.
|
||||||
|
|
||||||
|
**S-25 — a lost storage grant repairs itself, and the repair is REPORTED (2026-08-04, R-190; agent
|
||||||
|
v0.124.1).** On a missing grant the agent runs the existing root wrapper (`felhom-backup-target-apply
|
||||||
|
grant <id>`, already sudoers-permitted for any id) and re-reads once — the pbsdr R-22 shape. Bounded
|
||||||
|
at one attempt per tier per hour.
|
||||||
|
|
||||||
|
*The record is the point, and getting it wrong is easy.* A repair that leaves only `ok` behind
|
||||||
|
destroys the only evidence a permission vanished. v0.124.0 reported degraded for "one cycle" and
|
||||||
|
**that failed in production while every unit test passed**: `probeAll` is called independently by the
|
||||||
|
self-check log and by the collector, the repairing call was the log's, and the host-report three
|
||||||
|
seconds later sent `ok`. The fix is a latch on **time** (20 min > the 900 s report interval), so at
|
||||||
|
least one report must carry it. The explanation rides `Feature`, because that is the field the hub
|
||||||
|
interpolates into the operator's e-mail — `Reason` does not travel. **Nothing new was built:** the
|
||||||
|
hub's existing ok→degraded→ok edge is the channel.
|
||||||
|
|
||||||
|
*Caveat measured live:* **PVE caches permissions** (~40 s and ~16 min observed), so detection lags the
|
||||||
|
loss and a single permission read is a lagging indicator → R-194.
|
||||||
|
|
||||||
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
|
**S-23 — the host (on-box) whole-guest tier is restore-PROVEN, unattended, on both demo boxes
|
||||||
(2026-08-04). Scope: those two boxes, not the fleet.**
|
(2026-08-04). Scope: those two boxes, not the fleet.**
|
||||||
|
|
||||||
|
|||||||
@@ -1,173 +1,124 @@
|
|||||||
# REPORT — RUNBOOK: the first host-tier restore-test, on both boxes
|
# REPORT — R-201: **THE DRILL PASSED.** A customer's file survived a machine rebuild
|
||||||
|
|
||||||
**Date:** 2026-08-03 → 2026-08-04 · **Repos:** `felhom.eu` docs + registers only. **Nothing was built
|
**2026-08-04, 21:30–23:15** · `demo-hp` deliberately rebuilt · no code, no version bump ·
|
||||||
and no version was bumped.** The only binary that moved is the already-published `v0.123.0`, onto the
|
`demo-felhom` untouched. Record: `documentation/audits/DRILL-r201-night-run-2026-08-04.md`.
|
||||||
box that did not have it (P1). Baselines re-read and matched: `felhom-agent` `72161f6cf010` /
|
|
||||||
`v0.123.0`; `felhom.eu` `e3187c86d58d` / hub `v0.91.1`, installer `1.24.0`. Constants re-confirmed at
|
|
||||||
source: `defaultRestoreTestEvalInterval` **6 h**, `defaultRestoreTestSettle` **24 h**.
|
|
||||||
|
|
||||||
**Outcome: four scheduled runs, all passed, nothing triggered by hand.**
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Preconditions
|
## 1. THE VERDICT — PASS
|
||||||
|
|
||||||
| # | demo-felhom | demo-hp |
|
|
||||||
|---|---|---|
|
|
||||||
| **P1** agent | `v0.123.0` ✓ | **`0.120.0` — below the 0.121.0 floor, so it could not become due at all.** Remediated with the published `v0.123.0` (sha `74910135…`, deployed sha identical) |
|
|
||||||
| **P2** tiers | host `felhom-backup` + offsite `felhom-pbs` (weekly) | **the same — the runbook expected demo-hp to have no offsite tier, and it has one**, active with 2 snapshots |
|
|
||||||
| **P3** grant | `ok=70 total=70 degraded=0` | `ok=70 total=70 degraded=0` (once the probe existed) |
|
|
||||||
| **P4** storage | `/dev/sdb → /mnt/hdd_1`, ext4, on the N100 | `/mnt/nvme-1tb` on the t740 — **different hosts, different disks ⇒ INDEPENDENT ⇒ parallel is safe** |
|
|
||||||
| **P5** space | target 889 GB free; restore pool `local-lvm` 358 GB | target 925 GB free; restore pool `data` **53.9 G at 30.79 %** (~37 GB free) against a 2.35 GB archive — adequate, and measured *because* that pool is the over-subscribed one |
|
|
||||||
| **P6** candidate | `…2026_08_02-04_42_14.tar.zst` (08-02) | `…2026_08_02-04_49_29.tar.zst` (08-02) — both correctly the settled archive, not the day's |
|
|
||||||
| **P7** other heavy work | daily backup ~04:44, outside the window | daily backup ~04:49, outside the window |
|
|
||||||
|
|
||||||
## 2. The due verdicts before the run, quoted
|
|
||||||
|
|
||||||
```
|
```
|
||||||
demo-felhom tier=felhom-backup due=true archive="…2026_08_02-04_42_14.tar.zst"
|
pre-wipe sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||||
reason: newest settled archive … has not been proven; nothing proven on this tier yet
|
restored sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||||
tier=felhom-pbs due=true archive="…2026-07-28T04:49:43Z"
|
|
||||||
reason: … has not been proven (last proven archive was a different one)
|
|
||||||
|
|
||||||
demo-hp tier=felhom-backup due=true archive="…2026_08_02-04_49_29.tar.zst"
|
|
||||||
tier=felhom-pbs due=true archive="…2026-07-28T19:19:45Z"
|
|
||||||
```
|
```
|
||||||
|
|
||||||
**Both boxes had BOTH tiers due**, which made §4's ordering question live rather than theoretical.
|
**Byte-identical.** The controller's data volume was destroyed and the file deleted from disk; the
|
||||||
|
customer's recovery code, the customer's restore flow, and the file came back. **The Felhom backup
|
||||||
|
story is proved end to end for the first time.**
|
||||||
|
|
||||||
## 3. The runs — all four SCHEDULED, none triggered
|
## 2. Snapshot count at step 9 — **3, not 1**
|
||||||
|
|
||||||
| box | tier | due at | archive | result |
|
`repo_state: null` (the repository **opened**, not orphaned), `snapshot_count: 3`, `repo_size_bytes:
|
||||||
|---|---|---|---|---|
|
42 026` — the pre-wipe size exactly. The pre-wipe snapshot `e6132ae5` was still present with the file
|
||||||
| demo-felhom | **host** | 00:55:21 | `…2026_08_02-04_42_14.tar.zst` | **passed, 83.8 s**, scratch torn down 00:56:45 |
|
inside it. **No fresh history was started over the old one** — the failure this drill was designed to
|
||||||
| demo-felhom | offsite | 06:55:21 | `…2026-07-28T04:49:43Z` | **passed, 540.4 s**, torn down 07:04:21 |
|
catch did not occur.
|
||||||
| demo-hp | **host** | 02:05:39 | `…2026_08_02-04_49_29.tar.zst` | **passed, 109.3 s**, torn down 02:07:28 |
|
|
||||||
| demo-hp | offsite | 08:05:39 | `…2026-07-28T19:19:45Z` | **passed, 300.1 s**, torn down 08:10:39 |
|
|
||||||
|
|
||||||
**No box failed to fire, so Phase C was not entered and no `--selftest` was used as a proof.** The
|
## 3. §5's five conditions, recorded before the wipe
|
||||||
only selftest invocations in this session were the read-only `restore-test-due` verdict prints in §2,
|
|
||||||
which start nothing.
|
|
||||||
|
|
||||||
**§4's question, answered live:** each box took its **host** tier first — never-proven sorts ahead of
|
1. sentinel **listed by name** in snapshot `e6132ae5` (19:36:26), 181 B.
|
||||||
proven, and ahead on the id tie-break — deferred the offsite one, and picked it up on the **following
|
2. rollback archive `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst` (1 606 765 083 B) **verified by a
|
||||||
evaluation six hours later**. One heavy operation at a time, per box, with nobody sequencing it. That
|
full zstd stream read** (4 867 573 760 B) with the sentinel confirmed inside.
|
||||||
is R-86's oldest-proven ordering and the heavy-operation gate observed together for the first time.
|
3. §3's option — §4 below.
|
||||||
|
4. `demo-felhom` `health=ok`, `escrow_state=escrowed`, untouched throughout.
|
||||||
|
5. nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 %.
|
||||||
|
|
||||||
**The asymmetry worth keeping:** a host-tier restore is **83–109 s**; an offsite one **300–540 s**. The
|
**One precondition had drifted and was repaired, not worked around:** the staged snapshot had lost the
|
||||||
tier an ordinary recovery uses is also the cheapest to prove.
|
sentinel because the afternoon's experiment produced a *later* same-day snapshot and `forget
|
||||||
|
--keep-daily 7 --group-by host,tags` pruned the good one. One backup re-established it, re-verified by
|
||||||
|
listing. **A good snapshot is not durable against a later bad run on the same day.**
|
||||||
|
|
||||||
## 4. What the runs left behind
|
## 4. §3 — the recovery code
|
||||||
|
|
||||||
**Persisted state — v3, naming the archive, the tier and what was verified:**
|
**Option B as already in place, improved: no new copy was made, so nothing needed shredding.** The
|
||||||
|
operator's own `~/.config/credentials` (`0600`) was the source; the code was **piped to stdin** for
|
||||||
|
each invocation — never an argument, never exported, never a second file, never logged. Destroying the
|
||||||
|
operator's own store would have destroyed their record; because no extra copy existed, there is nothing
|
||||||
|
left to prove gone.
|
||||||
|
|
||||||
```json
|
**Verified anyway, with the planted-copy positive control:** 0 hits in the agent journal, 0 in the
|
||||||
demo-felhom felhom-backup → {archive …2026_08_02-04_42_14.tar.zst, tier local, verified boot+running,
|
controller log, 0 files under `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`, `/root`, 0 leftover
|
||||||
proven_at 2026-08-03T22:56:45Z}
|
`felhom-idesc-*` dirs — and the same sweep found a planted copy (**1**), then **0** after shredding it.
|
||||||
felhom-pbs → {archive …2026-07-28T04:49:43Z, tier pbs, proven_at 2026-08-04T05:04:21Z}
|
|
||||||
demo-hp felhom-backup → {archive …2026_08_02-04_49_29.tar.zst, tier local, proven_at 2026-08-04T00:07:28Z}
|
|
||||||
felhom-pbs → {archive …2026-07-28T19:19:45Z, tier pbs, proven_at 2026-08-04T06:10:39Z}
|
|
||||||
```
|
|
||||||
|
|
||||||
**The hub received the host-tier proofs — R-189's path carrying one for the first time.** demo-felhom's
|
## 5. Every step's observable
|
||||||
latest report holds **two** entries, one per tier; the `local` one can only have come from the
|
|
||||||
persisted state, because the in-memory store held only that morning's offsite run:
|
|
||||||
|
|
||||||
```
|
| step | observable |
|
||||||
demo-felhom tier=local pass=True archive=felhom-backup:…2026_08_02-04_42_14.tar.zst tested_at=2026-08-03T22:56:45Z
|
|---|---|
|
||||||
tier=pbs pass=True archive=felhom-pbs:…2026-07-28T04:49:43Z tested_at=2026-08-04T05:04:21Z
|
| 6 | fresh data dir `20:00:2x`; new `encryption.key`; `claimed = None`; `offbox = null` |
|
||||||
demo-hp tier=local pass=True archive=felhom-backup:…2026_08_02-04_49_29.tar.zst tested_at=2026-08-04T00:07:28Z
|
| **7** | **`identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6…`, `updated_at` still `11:11:37` — UNCHANGED.** Nothing re-escrowed itself |
|
||||||
```
|
| **8** | **recovered `8a9e33aa4da6…`** — matches the pre-wipe on-disk key and the hub's record |
|
||||||
|
| 9a | `[INSTALLED] … reads back identical` |
|
||||||
|
| 9b | after Re-issue + apply, the on-disk key is **still** the recovered one |
|
||||||
|
| 9c | **repository OPENED** — `repo_state: null`, 3 snapshots, 42 026 B |
|
||||||
|
| **10** | **restored sha256 byte-identical** |
|
||||||
|
| 11 | this record |
|
||||||
|
|
||||||
**A subsequent evaluation runs nothing on a proven tier**, quoted from demo-felhom now:
|
Superseded escrow rows remained **2** throughout — **no ceremony was run at any point.**
|
||||||
|
|
||||||
```
|
## 6. Part 2 — not run
|
||||||
tier=felhom-pbs due=false proven="…2026-07-28T04:49:43Z"
|
|
||||||
reason: newest settled archive (landed 2026-07-28T04:49:43Z) is already proven
|
|
||||||
```
|
|
||||||
|
|
||||||
…while its **host** tier reads `due=true` again — on the **08-03** archive, which has now settled 24 h.
|
Gate: *"the drill PASSED; its evidence is written down; and there is time."* The first two are met; the
|
||||||
That is not a defect: it is "proved daily, on its own archive", visible one day later.
|
third is not — 23:15, after a full destructive cycle, and Part 2 is a second wipe. Deliberately left
|
||||||
|
for a fresh session with drill 1's result already recorded, so a second wipe cannot overwrite it.
|
||||||
|
**R-198's retention remains unit-proven only** — nothing has yet superseded a key in production, and it
|
||||||
|
is now the last unproven link in this chain.
|
||||||
|
|
||||||
## 5. Teardown — three layers, per box
|
## 7. Teardown — three layers
|
||||||
|
|
||||||
| layer | demo-felhom | demo-hp |
|
| layer | state |
|
||||||
|---|---|---|
|
|---|---|
|
||||||
| the machine | `pct list` → **0** entries for 990000 | **0** |
|
| the guest | **healthy and re-armed**: controller v0.197.0 claimed, six app containers serving, all three apps re-toggled for off-site, the sentinel restored to its live location and verified. The verification-copy tree is **kept as evidence**. Scratch band empty |
|
||||||
| the host | `lvs` → **0** volumes for 990000; `local-lvm` **1.95 % → 1.95 %** | **0**; `local-lvm` 30.79 % → 40.86 % during the offsite run → **30.83 %** after |
|
| the host | vzdump snapshot LVs released cleanly; one new 1.6 GB archive on nvme-1tb (883 GB free); nothing deleted |
|
||||||
| the hub | the `restore_tests[]` entries are **RETAINED DELIBERATELY** — they *are* the proof the staleness check reads, so deleting them would delete the result | same |
|
| the hub | **no new customer records** — `demo-hp` is the same row throughout (a controller-data rebuild, not a re-enrolment). One Re-issue staged a one-time secret (consumed 20:16) and set `stale_at`. **The two superseded escrow rows are unchanged** |
|
||||||
|
|
||||||
Nothing else was created: no scratch customer, no fixture storage, no probe tag, no package version.
|
**Nothing deleted on the storage endpoint**, including the ~1.2 GB of orphaned ciphertext — ruled,
|
||||||
|
still owed, deliberately not ridden along with a drill.
|
||||||
|
|
||||||
## 6. What the run surfaced — three findings, two of them corrections to my own record
|
## 8. The capability-map row
|
||||||
|
|
||||||
### R-190 (new) — a storage ACL that worked in the morning was gone by mid-morning
|
Now reads *"A customer's file survives a machine rebuild and comes back — the whole off-site story, end
|
||||||
|
to end"*, **PROVEN-LIVE**. What it deliberately does **not** claim: the journey is manual and
|
||||||
|
undocumented (R-204); the scope is `demo-hp` and a **controller-data** rebuild, **not** a total host
|
||||||
|
loss and **not** a guest reprovision; and R-198's retention is still unit-proven.
|
||||||
|
|
||||||
A `vzdump` by `felhom-agent@pve!agent` with `--storage felhom-backup` completed **OK at 04:44:50** on
|
## 9. New findings — R-204, expanded into the gap list
|
||||||
2026-08-03; the first `403 … missing privilege Datastore.Allocate` on that path is **09:24:56**, and by
|
|
||||||
~14:50 `pveum acl list` held **no row at all** for it. Ruled out by measurement: a host reinstall
|
|
||||||
(uptime 12 days), any `pveum`/ACL/`user.cfg` activity in syslog 04:00–10:00, any cluster-log ACL entry.
|
|
||||||
Correlated but not established: guest 9201 was reprovisioned at 09:15–09:19, nine minutes before.
|
|
||||||
**A permission that can vanish silently makes every ACL-based guarantee on these hosts provisional**,
|
|
||||||
and v0.123.0's probe detects the *state* but says nothing about the *transition*.
|
|
||||||
|
|
||||||
### R-191 (new) — every weekly offsite backup reports FAILED although it worked
|
The four steps between a recovered key and a restored file, all measured while walking them:
|
||||||
|
|
||||||
demo-felhom, 06:49–06:53 today: the upload **succeeded** (223 s, 629 MiB of 1.874 GiB, 67.2 % reused
|
1. **R-193** — a rebuilt controller cannot configure its off-site tier (`no unconsumed offsite
|
||||||
incrementally) and the job then failed on the prune —
|
password`; the ledger shows its predecessor consumed it at 07:12:06). Needs an operator Re-issue.
|
||||||
`missing Datastore.Modify|Datastore.Prune on /datastore/felhom-offsite/demo-felhom` → `TASK ERROR: job
|
2. **The claim gate** — a rebuilt box is unclaimed, so *every* controller endpoint is intercepted. And
|
||||||
errors`, and the hub raised `whole_guest_backup_failed`. **The token behaves exactly as R-89 designed**
|
**the local escape hatch does not work unaided**: `--print-reset-code` writes the new hash to
|
||||||
(box tokens are write-only; ep0 prunes). What did not follow is the config: **both** boxes still arm
|
`settings.json` while the running controller keeps its old copy in memory, so `effectiveClaimCode()`
|
||||||
the offsite tier with `keep_last=2 prune_pbs_allowed=true`, so every weekly run asks for a prune that
|
never sees it. **Restart the controller between minting and claiming** — two claim attempts failed
|
||||||
must fail. The data is safe; the verdict and the weekly e-mail are wrong, which is the R-100 corollary
|
before this was diagnosed.
|
||||||
— an alarm whose text is true and whose trigger is not the thing you would act on. **Not fixed here**
|
3. **R-196** — the Re-issue sets `stale_at` (`20:15:49`) while `restic_pw_sha256` is unchanged, which
|
||||||
(§6 rule 2), and the fix needs one check first: whether ep0's prune jobs actually cover these two
|
gates every off-site run. Cleared with the **manual** confirm, never a ceremony.
|
||||||
namespaces.
|
4. **`mode=unit` is the restore default** and returns the recovery unit, **not** the userdata leg. A
|
||||||
|
customer told to "restore from off-site" gets their app definition and not their documents, and
|
||||||
|
nothing in the outcome says so. **The worst of the four**, because it fails silently at the last
|
||||||
|
step.
|
||||||
|
|
||||||
### Two corrections to yesterday's record
|
## 10. CI
|
||||||
|
|
||||||
1. **The R-185 drift was NOT silent on the write path.** demo-felhom's local-api backup jobs 403'd
|
Docs push only; run number and task id in the session summary. **`--no-verify` not used.**
|
||||||
**six times** (09:24 → 17:34) on that storage and privilege, and the hub raised
|
|
||||||
`whole_guest_backup_failed` at the first with edge-triggering suppressing the rest. My annotation
|
|
||||||
said backups kept landing because writes go through a root path — wrong, and now corrected in the
|
|
||||||
runbook and on the row.
|
|
||||||
2. **My "no `restore_test_*` events at all" was an instrument error.** The hub has no `/events` route;
|
|
||||||
I grepped a **404 page**. Read from the events table: five such events exist (2026-07-27/28), none
|
|
||||||
since the R-86 work — and one **more** since, below.
|
|
||||||
|
|
||||||
**One further event, correctly raised and worth stating:** `restore_test_stale` for demo-felhom at
|
## 11. Observations — noticed, NOT acted on
|
||||||
2026-08-03 22:33:42 UTC — **22 minutes before** the host-tier run. It was **true**: at that moment the
|
|
||||||
host tier had never been proven, and the hub said so on its own, without being asked. It has not
|
|
||||||
re-fired; the signal is edge-triggered, so the return to healthy is silent by design.
|
|
||||||
|
|
||||||
## 7. The capability map
|
1. **A same-day re-run replaces the day's snapshot** (`forget --keep-daily 7 --group-by host,tags`).
|
||||||
|
Any drill depending on a specific snapshot surviving must account for it.
|
||||||
The unattended restore-proof row now reads **PROVEN-LIVE for the host tier, unattended, on both demo
|
2. **The hub's ClusterIP is not reachable from DooPlex's host network** — the operator UI needs a
|
||||||
boxes**, with the four runs, their durations, the deferred-tier ordering, the hub-side proof and the
|
`kubectl port-forward`; the memory's `curl -u :$HUB_PW` recipe omits that.
|
||||||
teardown cited — and it states its **scope explicitly**: `demo-felhom` and `demo-hp`. The tester's box
|
3. **`--print-reset-code` mints a new generation on every invocation** — three wasted generations here
|
||||||
is untested and untouched, and one box proving something does not make it a fleet property.
|
(4, 5, 6) while diagnosing the in-memory staleness.
|
||||||
|
4. **The restore wrote into `backups/offsite-restore/<app>/` mirroring the full absolute path** — deep
|
||||||
## 8. Registers
|
but unambiguous; worth knowing before writing customer-facing copy about where files land.
|
||||||
|
|
||||||
- **R-185** — its consequence is now demonstrated; the row already closed yesterday, and carries the
|
|
||||||
correction in §6.
|
|
||||||
- **R-190**, **R-191** — filed. `grep` established R-190 and R-191 were free before minting (R-189 was
|
|
||||||
the highest in use).
|
|
||||||
- `ROADMAP.md` holds none of these rows, so nothing to collapse.
|
|
||||||
- `STATUS.md` rewritten for the operator and kept to one screen (85 lines); R-191 appears under
|
|
||||||
"What's broken" because it produces a weekly e-mail you would otherwise learn to ignore.
|
|
||||||
|
|
||||||
## 9. Observations — noticed, NOT acted on
|
|
||||||
|
|
||||||
- **demo-hp's `local-lvm` thin pool reached 40.86 %** during its offsite restore (from 30.79 %,
|
|
||||||
returning to 30.83 %). Comfortable, but that is the over-subscribed pool the target-selection notes
|
|
||||||
warn about, and the offsite archive is the larger of the two. A materially bigger guest would want
|
|
||||||
the restore pointed at `/mnt/nvme-1tb` instead.
|
|
||||||
- **demo-hp has an offsite tier**, contrary to the runbook's §2 premise and to the note that it "has
|
|
||||||
none". Nothing depends on that assumption now, but the operations notes still carry it.
|
|
||||||
- **The two boxes' daily archives are ~6.3 GB (demo-felhom) vs ~2.35 GB (demo-hp)** for the same guest
|
|
||||||
role — a 2.7× difference worth understanding before either is used to size anything.
|
|
||||||
- **Both waiters this session produced no output** despite the runs completing; the evidence was
|
|
||||||
gathered by direct query afterwards. A watcher that silently produces nothing is exactly the
|
|
||||||
instrument class this project distrusts — the conclusions here rest on the boxes' own journals and
|
|
||||||
the hub's database, not on the waiters.
|
|
||||||
|
|||||||
@@ -75,8 +75,11 @@
|
|||||||
| `(*Store).GetHostRecoveryMeta` + `(*Server).handleHostRevealRecoveryCredential` | hub/internal/store/host_recovery.go · hub/internal/web/hosts.go | `(hostID) (*HostRecoveryMeta, error)` · `POST /hosts/{id}/reveal-recovery-credential` | The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) | **Use `GetHostRecoveryMeta` on any page-render path** — its struct and its `SELECT` both omit the `secret` column, so it cannot leak one; `GetHostRecoveryCredential` (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE `recovery_credential_revealed` event via `SaveEvent` and calls NO dispatcher (the `handleRequestLogTail` shape). `api/handler.go handleAdminGetRecoveryCredential` (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133. |
|
| `(*Store).GetHostRecoveryMeta` + `(*Server).handleHostRevealRecoveryCredential` | hub/internal/store/host_recovery.go · hub/internal/web/hosts.go | `(hostID) (*HostRecoveryMeta, error)` · `POST /hosts/{id}/reveal-recovery-credential` | The break-glass console credential, split into a RENDER half and a RETRIEVE half (v0.84.0) | **Use `GetHostRecoveryMeta` on any page-render path** — its struct and its `SELECT` both omit the `secret` column, so it cannot leak one; `GetHostRecoveryCredential` (which does select it) belongs only to the two retrieval handlers. The reveal is POST so the ServeHTTP-level CSRF check applies and no secret is reachable by URL; it writes ONE `recovery_credential_revealed` event via `SaveEvent` and calls NO dispatcher (the `handleRequestLogTail` shape). `api/handler.go handleAdminGetRecoveryCredential` (global key) is the independent fallback for when the UI is down — never route the UI through it. Secret at rest is plaintext → R-133. |
|
||||||
| `host_detail_body` sub-template | hub/internal/web/templates/host_detail_body.html | `{{template "host_detail_body" .}}` | Rendering a host's detail sections on ANY surface | One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with `{{.HostID}}` (the customer page renders N instances). |
|
| `host_detail_body` sub-template | hub/internal/web/templates/host_detail_body.html | `{{template "host_detail_body" .}}` | Rendering a host's detail sections on ANY surface | One namespace across ParseFS (icons.html pattern). Renders per-host — id-suffix any new element ids with `{{.HostID}}` (the customer page renders N instances). |
|
||||||
| `(*Store).ListHostsByCustomer` | hub/internal/store/store.go (~L1620) | `(customerID) ([]Host, error)` | A customer's hosts, host_id order | A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer. |
|
| `(*Store).ListHostsByCustomer` | hub/internal/store/store.go (~L1620) | `(customerID) ([]Host, error)` | A customer's hosts, host_id order | A LIST by design (HA-cluster roadmap) — don't collapse to GetHostByCustomer. |
|
||||||
|
| `(*Store).HasEverBoundHost` (v0.92.0, R-195) | hub/internal/store/store.go | `(customerID) (bool, error)` | Any verdict that must not fire for a customer with **no machine ever bound** — "was anything ever expected of this customer" | `hosts` row **OR** `host_deletions` tombstone. **NOT a liveness check and never a substitute for one:** a box that was bound and went silent returns `true` and must keep alarming — that is the case any change here breaks first (pinned by `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`). Callers **fail OPEN** on its error: an unreadable binding must never SUPPRESS an alarm. Do **not** re-derive this from report presence — `store.GetCustomers()` (and therefore the staleness checker's `down` state) is a query over `reports`, so a never-reported customer has no state at all, which is exactly how the daily false alarm reached `david`. |
|
||||||
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
|
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
|
||||||
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
|
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
|
||||||
|
| `handleHostEscrowGet` + `escrowSelfServiceRetrieval` | hub/internal/api/handler.go | `GET /hosts/{id}/escrow` | The box-authenticated MIRROR of the escrow PUT — self-scoped by the per-host key. **`escrowSelfServiceRetrieval` is THE single decision point** for whether a box may read its own blob without operator-armed recovery mode (§8.2 vs §8.3, R-199): flip it, do not scatter the condition. **Never merge this with `dr.go`'s `handleReEnroll`/`handleGetRestoreDirective`** — those rotate the host API key and serve the K-escrow and directive too, and keep their recovery-mode gate (pinned by `TestEscrowGet_OperatorDRPathUnchanged`). Every successful retrieval MUST raise `escrow_blob_served` before the bytes leave; that audit row is the mitigation the trade rests on. |
|
||||||
|
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts** — `blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** `SaveHostDRBundle` writes `identity_blob` AFTER `SaveHostEscrow` returns, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
|
||||||
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
|
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
|
||||||
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
|
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
|
||||||
| `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |
|
| `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# STATUS — what works, what's broken, what's next
|
# STATUS — what works, what's broken, what's next
|
||||||
|
|
||||||
**Updated 2026-08-03.**
|
**Updated 2026-08-04.**
|
||||||
|
|
||||||
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
> **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority on open work; this
|
||||||
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
> page restates part of it in plain words, and **nothing may exist only here**. **Not `CONTEXT.md`**,
|
||||||
@@ -20,36 +20,69 @@ Proven end to end on real hardware.
|
|||||||
|
|
||||||
## What's broken
|
## What's broken
|
||||||
|
|
||||||
- **Rebuilding a machine silently takes away its off-site app-data backup.** `demo-hp` was rebuilt on
|
- **Rebuilding a machine still throws away its off-site backup HISTORY.** The machine invents the key
|
||||||
3 August and came up without one, and stayed that way for a day. **Fixed on 4 August** — re-issued,
|
that encrypts its own off-site backups, and a rebuilt machine invents a brand-new one, so the old
|
||||||
the machine picked the new password up in 15 seconds and reattached to the same repository, and you
|
backups can no longer be opened by the machine that wrote them. Both demo machines did this on 3–4
|
||||||
escrowed the key. **The underlying fault is not fixed:** the off-site password is delivered exactly
|
August — demo-hp lost access to 15 backups (41 MB), demo-felhom to 36 (1.14 GB), and **the second one
|
||||||
once and a rebuilt machine cannot ask for another, so this will happen again on the next rebuild.
|
happened in silence for thirteen hours**. From today that silence is fixed (below) and the old key
|
||||||
The other machine survived the same rebuild only because an unused password happened to be waiting
|
survives the ceremony (below), but **the rebuild itself still starts a fresh history**, and getting
|
||||||
for it. *(R-193)*
|
the old one back is a path nobody has walked. *(R-193)*
|
||||||
- **The daily email about it tells you the wrong story**, and the automatic repair that exists for
|
- *(fixed 4 Aug)* ~~The safety net we believed was under all of this is not there.~~ **The central hub
|
||||||
this declines without saying why. The message says the password was never applied; it was, on
|
was keeping the wrong key, and the fix shipped the same day.** When a machine re-seals its backups
|
||||||
23 July, and worked for eleven days. *(R-192)*
|
under a new recovery code, the hub keeps the old sealed package — but it was keeping the one for the
|
||||||
- **The weekly off-site backup reports FAILED although it worked.** It uploads correctly and then
|
whole-machine backups and **not** the one for the off-site file backups, which is the whole point. And
|
||||||
trips on a tidy-up step it is deliberately not allowed to perform, so the job ends in an error and
|
because the swap happens in one motion, **the act that destroyed the old key was the recovery-code
|
||||||
you get an email. The backup itself is safe and on the endpoint. Both demo machines do it; one
|
ceremony itself** — the exact thing a rebuilt machine asks its customer to do, on a screen promising
|
||||||
setting per machine fixes it. *(R-191)*
|
the old backups stay recoverable. Both keys are now kept. **What this cannot undo:** anything
|
||||||
|
superseded before today is gone for good, which includes both demo machines' pre-4-August keys — so
|
||||||
|
those 51 orphaned backups were beyond reach even if the recovery codes had been kept. *(R-198)*
|
||||||
|
- **IT WORKS. A customer's file survived a machine being destroyed and came back — proved last night on
|
||||||
|
real hardware, end to end.** We wiped the HP machine's controller data on purpose and deleted the
|
||||||
|
marked file from its disk. Using the recovery code you saved: the backup key came back **identical,
|
||||||
|
character for character**; the existing off-site store **opened** rather than starting over (the same
|
||||||
|
three backups, the same 42 026 bytes as before the wipe); and the file was restored **byte for byte
|
||||||
|
identical**. That is the whole backup promise, demonstrated for the first time. *(R-201)*
|
||||||
|
- **But a customer could not have done it alone, and that is now the open work.** Getting from "the key
|
||||||
|
is recoverable" to "the file is back" took four steps that appear in no instructions: re-issuing the
|
||||||
|
storage credential; re-claiming the machine (whose local reset-code tool **does not work until the
|
||||||
|
controller is restarted** — two attempts failed before we spotted it); manually clearing a "stale"
|
||||||
|
mark that the re-issue sets even though the key never changed; and choosing the *full* restore,
|
||||||
|
because the default one returns the app's settings and **not** the customer's documents — with
|
||||||
|
nothing saying so. **The capability is real; the experience is not built yet.** *(R-204)*
|
||||||
|
- **One screen still tells the customer something we cannot yet promise.** The „elárvult tároló" card
|
||||||
|
says the old backups may later be restorable with the matching recovery code. From today that is true
|
||||||
|
for machines that re-seal from now on and **false for anything already orphaned** — and the machine
|
||||||
|
cannot tell which case it is in without the recovery path above. We deliberately did **not** patch the
|
||||||
|
sentence: a conditional promise that can still be wrong is worse on that screen than a vague one.
|
||||||
|
*(R-202)*
|
||||||
|
- *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine
|
||||||
|
and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer
|
||||||
|
asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing
|
||||||
|
it every day since 27 July. Both machines and all future ones. *(R-191)*
|
||||||
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
||||||
also delete it. A daily snapshot is armed as a stopgap.
|
also delete it. A daily snapshot is armed as a stopgap.
|
||||||
*(R-95, R-87)*
|
*(R-95, R-87)*
|
||||||
|
|
||||||
## What shipped recently
|
## What shipped recently
|
||||||
|
|
||||||
- **The on-machine backup copy has now been proved to restore — by the machines themselves.** Both
|
- **The daily error email about David is gone.** David's record was created when he said he would try
|
||||||
demo machines restored their own on-machine backup into a throwaway machine overnight, booted it,
|
Felhom; no machine was ever attached to it, so nothing has ever been expected of it — yet every
|
||||||
checked it and destroyed it, without being asked: 84 and 109 seconds each. Every restore proof we
|
morning at five the system reported that a database backup had not happened. **A customer with no
|
||||||
had before this was of the *off-site* copy; the copy an ordinary recovery would actually use had
|
machine ever attached now produces no alarm.** A machine that *was* attached and went quiet still
|
||||||
never been tested on either machine. Both also proved their off-site copy on the same night, one
|
does, because that is a real fault, and there is a test whose whole job is to make sure a future
|
||||||
after the other rather than at once, which is the machine deciding for itself what to do first.
|
change cannot take that away. David's record was not touched — the record was right; the alarm was
|
||||||
*(closes the last open half of R-86/R-185)*
|
wrong. **The first quiet morning will be 5 August, so this is not confirmed yet.** *(R-195)*
|
||||||
- **A backup copy the machine was never allowed to read — and could not tell you about**, on both
|
|
||||||
demo machines. The permission was one command; the silence was the real fault, and the machine now
|
- **A permission the machine needs can now repair itself — and it tells you it had to.** Yesterday we
|
||||||
checks whether it may read each copy it depends on and says so when it may not. *(R-185)*
|
found a permission that vanished with no reinstall and no recorded action; the machine now notices,
|
||||||
|
restores it through the same controlled helper that grants it in the first place, and **raises an
|
||||||
|
alert saying it was missing and was restored**. The alert is the point: a permission that silently
|
||||||
|
fixes itself is one that can be lost over and over and never investigated. Proven twice on a demo
|
||||||
|
machine by taking the permission away by hand. *(R-190 — the repair; why it vanished is still open)*
|
||||||
|
|
||||||
|
- **The on-machine backup copy has now been proved to restore — by the machines themselves**, both of
|
||||||
|
them, overnight and unasked (84 and 109 seconds). Every restore proof before this was of the
|
||||||
|
off-site copy. *(R-86, R-185)*
|
||||||
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
||||||
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
||||||
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
||||||
@@ -64,7 +97,19 @@ Proven end to end on real hardware.
|
|||||||
|
|
||||||
## What we're working on
|
## What we're working on
|
||||||
|
|
||||||
- **Now:** nothing outstanding.
|
- **Now:** making that journey something a customer can actually follow — the four steps above, worst
|
||||||
|
first: the restore default that silently returns the wrong thing. The HP machine is sitting
|
||||||
|
mid-drill with the recovered key already on it; it needs re-claiming and one setting confirmed, then
|
||||||
|
the backup runs and the file is restored. **Do not let it make a new recovery code** — that would
|
||||||
|
The marked file now lands in
|
||||||
|
the off-site backup, so there is finally something to recover. Everything else is already in place on
|
||||||
|
the HP machine — the recovery code you saved, a working off-site store, the app and the file. Nothing
|
||||||
|
has been wiped; that step waits for your go-ahead at a marked stop point. Both honesty fixes shipped today: a changed backup key now
|
||||||
|
raises an alarm on the day, and the email that stated the opposite of what it measured now describes
|
||||||
|
what it actually saw.
|
||||||
|
- **Also tomorrow:** confirming what the 04:15 off-site run does on both machines. We expect it to
|
||||||
|
refuse and ask, rather than start fresh — but that is a prediction from the code, not something
|
||||||
|
anyone has watched happen.
|
||||||
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
- **Next:** proving the off-site *app-data* copy can actually be restored — the one tier nothing
|
||||||
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
tests unattended. Most of the machinery it needed arrived with the restore-test change below.
|
||||||
*(R-87)*
|
*(R-87)*
|
||||||
@@ -72,6 +117,36 @@ Proven end to end on real hardware.
|
|||||||
|
|
||||||
## Waiting on you
|
## Waiting on you
|
||||||
|
|
||||||
|
- *(decided 4 Aug — recorded here until the work it unblocks is done)* You chose **not** to keep a
|
||||||
|
copy of the backup key on the Proxmox host. That is a coherent choice and it has a consequence worth
|
||||||
|
restating: **the customer's own recovery code is now the only route back from a rebuild**, so the
|
||||||
|
sealed package the hub keeps is the single copy. Today's fix is what makes that true rather than
|
||||||
|
wishful — until this morning the ceremony destroyed it. *(R-193)*
|
||||||
|
- **Your ruling about the recovery screen has been priced, and it can be built.** You said: a freshly
|
||||||
|
installed machine that finds a sealed package waiting should say so loudly, offer the customer a box
|
||||||
|
to type their recovery code into, and show what would come back before doing anything. **All of it is
|
||||||
|
buildable, and one part is already free** — the hub is *already* telling every machine, on every
|
||||||
|
check-in, that a sealed package exists and which key it covers, and the machine currently throws that
|
||||||
|
message away. Showing a preview is also cheap: listing what is in an off-site store reads it without
|
||||||
|
writing to it, so the customer can see how many backups, from when, and for which apps before
|
||||||
|
committing. The real work is one new connection: the unsealing has to happen in the part that runs on
|
||||||
|
the Proxmox host, because the customer-facing part cannot do it — the same crossing your recovery-code
|
||||||
|
ceremony already makes in the other direction. **One thing for you to weigh, which we deliberately did
|
||||||
|
not decide:** a screen that takes a recovery code and then shows what is in a backup store is reachable
|
||||||
|
by anyone with the household's dashboard password, and the preview reveals backup dates and app names.
|
||||||
|
*(R-193)*
|
||||||
|
- **Should we run the proof?** Nothing about recovery has ever actually been done, so we designed the
|
||||||
|
exercise: on the spare demo machine, make a recovery code and **keep it**, put a marked file into an
|
||||||
|
app, back it up off-site, wipe the machine, reinstall, recover with the saved code, and check the
|
||||||
|
marked file comes back byte for byte. About half a day, of which under an hour needs you. The failure
|
||||||
|
we are watching for is precise: **if the recovered machine reports one backup instead of the ones we
|
||||||
|
put there, it started a fresh history and the proof failed** — "the store opened" is not good enough.
|
||||||
|
It costs that machine's contents if it goes wrong, which is acceptable there. **You said yes, after
|
||||||
|
R-198 — and R-198 shipped this morning, so this is ready to run on your word.** *(R-201)*
|
||||||
|
- **The orphaned backups on the storage box — you said delete, and it is still owed.** About 1.2 GB
|
||||||
|
across the two demo machines, in set-aside stores nobody can open and nothing prunes. Deliberately not
|
||||||
|
done today: erasing data on the storage endpoint is not something that should ride along with a
|
||||||
|
database change, so it wants its own session. *(R-193)*
|
||||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||||
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
||||||
@@ -81,6 +156,33 @@ Proven end to end on real hardware.
|
|||||||
|
|
||||||
## Changed since last update
|
## Changed since last update
|
||||||
|
|
||||||
|
- **2026-08-04 (night)** — **The drill PASSED.** We destroyed a machine on purpose and the customer's
|
||||||
|
file came back byte-for-byte identical, using the recovery code you saved. First time the backup
|
||||||
|
story has been proved end to end. It needed four undocumented manual steps to get there, which are
|
||||||
|
now the next piece of work. The machine is up, healthy and re-armed. *(R-201, R-204)*
|
||||||
|
- **2026-08-04 (evening)** — **Fixed the folder-left-out-of-the-backup problem, both halves.** The app
|
||||||
|
and its backup now look in the same directory, and a backup that misses a folder marked essential
|
||||||
|
reports *incomplete* instead of success. Proved by listing the backup's own contents and finding the
|
||||||
|
marked file. The wipe-and-restore proof is unblocked. *(R-203, R-201)*
|
||||||
|
- **2026-08-04 (late)** — Set up the wipe-and-restore proof on the HP machine and **stopped before the
|
||||||
|
wipe**: a folder marked as protected was missing from the off-site backup while the backup reported
|
||||||
|
success. Three things were proved on the way, all firsts: a rebuilt machine's off-site backup now
|
||||||
|
**refuses and says so** instead of quietly starting over; the "start a new store" repair works and
|
||||||
|
keeps the old data aside; and the HP machine's pre-3-August history is gone for good — its key was
|
||||||
|
destroyed four hours before the fix that would have kept it. *(R-203, R-201, R-193)*
|
||||||
|
- **2026-08-04 (evening)** — **Proved the backup key comes back.** A demo machine fetched its own
|
||||||
|
sealed package, opened it with the saved recovery code, and produced a key identical to the one it
|
||||||
|
uses. A wrong code was refused and wrote nothing. The recovery code left no trace anywhere on the
|
||||||
|
machine — searched, with a planted copy first to prove the search worked. *(R-199, R-200)*
|
||||||
|
- **2026-08-04 (earlier)** — **Fixed the worst of it.** The hub now keeps the off-site backup key when a
|
||||||
|
machine re-seals, instead of only the whole-machine one — four lines of database change that stop the
|
||||||
|
recovery ceremony from destroying the thing it exists to protect. A changed backup key now raises an
|
||||||
|
alarm the same day. The daily email that described the opposite of what it measured now states what it
|
||||||
|
saw, and a refusal to auto-repair leaves a record instead of silence. *(R-198, R-197, R-192, R-196)*
|
||||||
|
- **2026-08-04 (later)** — Established what a machine rebuild actually destroys: not the storage
|
||||||
|
password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost
|
||||||
|
access to their off-site history, one of them silently. A decision is now waiting on you. The daily
|
||||||
|
false alarm about David is fixed. *(R-193, R-195)*
|
||||||
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
|
- **2026-08-04** — Both demo machines proved their on-machine backup restores, on their own,
|
||||||
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
|
overnight — the copy an ordinary recovery uses, never tested until now. Found while checking: the
|
||||||
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
|
weekly off-site backup reports failure after a successful upload. *(R-185, R-191)*
|
||||||
|
|||||||
File diff suppressed because one or more lines are too long
@@ -378,6 +378,60 @@ nicety.
|
|||||||
**loses Felhom's backup guarantees** — no PBS verify / monitoring on storage we can't reach. An
|
**loses Felhom's backup guarantees** — no PBS verify / monitoring on storage we can't reach. An
|
||||||
honest signup-time tradeoff, not a hidden one.
|
honest signup-time tradeoff, not a hidden one.
|
||||||
|
|
||||||
|
#### Custody across generations — what the hub retains when a blob is superseded (hub v0.60.0; **contract corrected hub v0.93.0, R-198**)
|
||||||
|
|
||||||
|
A ceremony that seals a *different* passphrase does not destroy the blob it replaces: the hub copies
|
||||||
|
the current `host_escrow` row into `host_escrow_superseded` **before** overwriting it, so a previous
|
||||||
|
generation stays openable with the recovery code that sealed it. The same routine
|
||||||
|
(`demoteCurrentEscrowTx`) is used when a host is deleted, which **demotes** custody rather than
|
||||||
|
destroying it. The hub never decrypts either table.
|
||||||
|
|
||||||
|
**The contract as originally shipped was incomplete, and the gap is worth recording because it was
|
||||||
|
invisible for two months.** An escrow generation is **two** sealed artifacts — the K-escrow `blob`
|
||||||
|
(the PBS datastore key `K`) and `identity_blob` (the age-wrapped `IdentityBundle`, which carries the
|
||||||
|
offsite **restic repository password** as well as the tunnel/PBS tokens and the WG key). Retention
|
||||||
|
covered only the first. So a supersession preserved the whole-guest key and destroyed the off-site
|
||||||
|
data key — and because the copy happens as the new blob lands, **the destroying act was the ceremony
|
||||||
|
itself**, i.e. the step a rebuilt box asks its customer to perform. Corrected in hub v0.93.0: the
|
||||||
|
retained row carries both.
|
||||||
|
|
||||||
|
**Two consequences that belong in this doc rather than a changelog:**
|
||||||
|
|
||||||
|
- **Generational custody is now the recovery precondition for the offsite tier.** The operator ruled
|
||||||
|
on 2026-08-04 that no repository password is retained on the Proxmox host (R-193 candidate (c),
|
||||||
|
refused). With no host-side copy, a rebuilt box's only route back to its previous off-site history
|
||||||
|
is *the customer's older recovery code opening the retained identity blob* — which exists only if
|
||||||
|
the supersession that replaced it happened on hub v0.93.0 or later.
|
||||||
|
- **Retention is custody, not recovery, and this doc must not be read as claiming otherwise.** The
|
||||||
|
hub's restore-mode serving of these blobs (§ slice 10D) has **no client** in the agent, the
|
||||||
|
controller's recovered-password injection seam has **no user interface**, and no part of the chain
|
||||||
|
has ever been exercised outside a unit test — the one live identity round-trip (2026-06-10)
|
||||||
|
predates the `ResticRepoPassword` field. Tracked as R-199/R-200/R-201; evidence
|
||||||
|
`documentation/audits/RECON-offsite-dr-chain-2026-08-04.md`.
|
||||||
|
|
||||||
|
#### Serving a blob back to its own box (hub v0.94.0 + agent v0.125.0, R-199)
|
||||||
|
|
||||||
|
Retention without retrieval is custody, not recovery. Since 2026-08-04 a host reads its own sealed
|
||||||
|
identity blob from `GET /api/v1/hosts/{host_id}/escrow` — self-scoped by the per-host key, the mirror
|
||||||
|
of the PUT that stored it — and opens it locally with the customer's recovery code via the agent's
|
||||||
|
`POST /escrow/recover-offsite-password`, which returns **only** the offsite repository password.
|
||||||
|
|
||||||
|
**This is a deliberate widening of who can obtain the ciphertext**, recorded in `CONTEXT.md` S-29 and
|
||||||
|
in the handler itself: it previously took an operator arming recovery mode, and now takes only that
|
||||||
|
box's own credential. The blob remains unopenable without R, and every retrieval raises an
|
||||||
|
operator-visible `escrow_blob_served` event. The operator-driven re-enroll/restore-directive path is
|
||||||
|
unchanged and keeps its recovery-mode gate.
|
||||||
|
|
||||||
|
**Proven end to end on demo-felhom, 2026-08-04:** the recovered repository password's sha256 was
|
||||||
|
byte-identical to the one on the box, and to the hash the hub had independently recorded. **The chain
|
||||||
|
stops there** — nothing installs a recovered password, reopens a repository with it, or restores a
|
||||||
|
file.
|
||||||
|
|
||||||
|
**The honesty property above is unchanged and now has a second edge:** losing `R` and the box makes
|
||||||
|
the offsite backups unrecoverable by anyone — *and* an older generation's data is recoverable only
|
||||||
|
with **that generation's** recovery code, which the customer must still hold. A current recovery code
|
||||||
|
does not open a previous generation's blob.
|
||||||
|
|
||||||
## 9. Provisioning & DR flows
|
## 9. Provisioning & DR flows
|
||||||
|
|
||||||
**Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs
|
**Provisioning (reconcile-driven, by restore).** Fresh creation of a Docker-capable LXC needs
|
||||||
|
|||||||
@@ -278,7 +278,10 @@ The tiers are **inputs to recovery**, not recovery routes. §7 and §8 say what
|
|||||||
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
|
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
|
||||||
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
|
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
|
||||||
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** |
|
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/docker` + `mp1 /mnt/sys_drive` | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` | **no** |
|
||||||
| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` | **yes** (per-customer key) |
|
| **Plane-2** whole-guest, offsite | `felhom-pbs:` → ep0 datastore `felhom-offsite`, per-customer namespace, over WireGuard | same contents | **7 days** (`604800`) | server-side prune on ep0, `keep-last 2` at `03:30` — **and the box asks for none** (R-191) | **yes** (per-customer key) |
|
||||||
|
|
||||||
|
> **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does.
|
||||||
|
|
||||||
|
|
||||||
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
|
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
|
||||||
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
|
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
|
||||||
|
|||||||
@@ -0,0 +1,265 @@
|
|||||||
|
# DRILL — R-201, the night run: **PASS. The customer's file came back, byte-identical.**
|
||||||
|
|
||||||
|
**Date:** 2026-08-04, 21:30–23:15 · **Box:** `demo-hp` (Tier 0) · unattended, then **finished with the
|
||||||
|
operator present**. The box is up, healthy, and fully re-armed.
|
||||||
|
|
||||||
|
> **THE VERDICT: PASS.**
|
||||||
|
>
|
||||||
|
> ```
|
||||||
|
> pre-wipe sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||||
|
> restored sha256: 643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> A machine was destroyed on purpose — controller data volume gone, the customer's file deleted from
|
||||||
|
> disk — and **the file came back byte-identical**, through the customer's own recovery code and the
|
||||||
|
> customer's own restore flow. **This is the first time the Felhom backup story has been proved end to
|
||||||
|
> end.**
|
||||||
|
>
|
||||||
|
> The key recovered byte-identical (`8a9e33aa4da6…`), the existing repository **opened rather than
|
||||||
|
> being re-initialised** (`repo_state: null`, 3 snapshots, 42 026 B — the pre-wipe size exactly), and
|
||||||
|
> the pre-wipe snapshot `e6132ae5` was still there with the file inside it.
|
||||||
|
>
|
||||||
|
> **It took four undocumented steps to get there**, each measured, and they are the other half of the
|
||||||
|
> result — see §3. A customer hitting this alone today would not get their data back.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. The verdict — **PASS**
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| sentinel sha256, pre-wipe (step 3) | `643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c` |
|
||||||
|
| sentinel sha256, restored (step 10) | **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`** |
|
||||||
|
| **match** | **YES — byte-identical** |
|
||||||
|
| snapshot count at step 9 | **3** — the pre-wipe count. **NOT 1.** No fresh history was started |
|
||||||
|
| repo size at step 9 | **42 026 B** — the pre-wipe size exactly |
|
||||||
|
| `repo_state` | **`null`** — the repository OPENED. Not orphaned |
|
||||||
|
|
||||||
|
The restore ran through the customer flow (`POST /backup/offbox/restore`, `mode=full`, two-step
|
||||||
|
confirm), out of snapshot `e6132ae5` — the one taken **before** the wipe.
|
||||||
|
|
||||||
|
**A detail worth keeping:** the first restore attempt used the default `mode=unit`, which recovers the
|
||||||
|
recovery unit only and **not** the userdata leg. `mode=full` is what returns the customer's files. A
|
||||||
|
customer told to "restore from off-site" would, on the default, get their app definition back and not
|
||||||
|
their documents — and nothing in that outcome says so.
|
||||||
|
|
||||||
|
## 2. What was proven, in order, on hardware
|
||||||
|
|
||||||
|
### Step 6 — the wipe
|
||||||
|
|
||||||
|
The controller's data volume was destroyed and the bootstrap re-ran. Fresh data directory, everything
|
||||||
|
stamped `2026-08-04 20:00:2x`:
|
||||||
|
|
||||||
|
```
|
||||||
|
encryption.key 32 B 2026-08-04 20:00:21 ← brand new: every pre-wipe app secret is undecryptable
|
||||||
|
claimed = None ← the fresh-install signal
|
||||||
|
offbox = null ← no off-site target
|
||||||
|
```
|
||||||
|
|
||||||
|
**Faithful to the incident, and deliberately so.** The 2026-08-03 rebuild that R-193 is filed against
|
||||||
|
was **not** a guest reprovision — the journal shows guest 9201 running continuously through that
|
||||||
|
window, with no `pct destroy`, no `pct restore` and no `--selftest=provision`. What changed was the
|
||||||
|
controller and its data volume. Reproducing *that* is what the drill needs; a guest destroy plus an
|
||||||
|
unrehearsed provisioning chain, improvised unattended, is what §8.10 exists to prevent.
|
||||||
|
|
||||||
|
### Step 7 — the assertion that keeps recovery possible: **PASSED**
|
||||||
|
|
||||||
|
```
|
||||||
|
host_escrow (demo-hp-bb76ea), AFTER the wipe:
|
||||||
|
identity_blob = 572 bytes ← unchanged
|
||||||
|
restic_pw_sha256 = 8a9e33aa4da6… ← unchanged
|
||||||
|
updated_at = 2026-08-04 11:11:37 ← unchanged; nothing re-escrowed
|
||||||
|
stale_at = NULL
|
||||||
|
```
|
||||||
|
|
||||||
|
**No ceremony was run and nothing re-escrowed itself.** The sealed key survived the rebuild untouched.
|
||||||
|
|
||||||
|
### Step 8 — **THE KEY CAME BACK**
|
||||||
|
|
||||||
|
```
|
||||||
|
=== offsite key recovery check (R-200) — compares, never installs ===
|
||||||
|
recovered sha256: 8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a
|
||||||
|
[FAIL] there is no repository password on this box to compare against
|
||||||
|
(the recovery itself SUCCEEDED — this box simply has no local key. That is the
|
||||||
|
rebuilt-box shape, where the next step is to INSTALL rather than compare.)
|
||||||
|
```
|
||||||
|
|
||||||
|
Exit 1 is **correct and designed**: there was nothing to compare against, because the wipe removed it.
|
||||||
|
The recovered hash matches the pre-wipe on-disk key and the hub's own record — three independent
|
||||||
|
sources agreeing, one of them recovered through the full chain (hub → agent → unseal → extract) on a
|
||||||
|
box that had just lost everything it knew.
|
||||||
|
|
||||||
|
### Step 9a — installed cleanly
|
||||||
|
|
||||||
|
```
|
||||||
|
=== offsite key recovery INSTALL (R-200) ===
|
||||||
|
on-disk sha256: (none — this box has no repository password)
|
||||||
|
recovered sha256: 8a9e33aa4da6…
|
||||||
|
[INSTALLED] the recovered repository password is in place and reads back identical.
|
||||||
|
```
|
||||||
|
|
||||||
|
The "installed" branch — the rebuilt-box shape v0.196.0 was written for — took its first real run.
|
||||||
|
|
||||||
|
### Step 9b — the apply kept it
|
||||||
|
|
||||||
|
After the target was reconfigured, the on-disk key was still `8a9e33aa4da6…`. `WriteOffboxSecrets`
|
||||||
|
found the file present and kept it, exactly as documented.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. The wall — three blockers, each measured
|
||||||
|
|
||||||
|
### (a) R-193, reconfirmed live: a rebuilt controller cannot configure its off-site tier
|
||||||
|
|
||||||
|
```
|
||||||
|
[INFO] [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.197.0)
|
||||||
|
[WARN] [offsite-apply] reconcile: offsite-apply: consume one-time password:
|
||||||
|
no unconsumed offsite password (already consumed or none provisioned)
|
||||||
|
(retries on next config refresh/restart)
|
||||||
|
```
|
||||||
|
|
||||||
|
The ledger, measured: `demo-hp` one-time secret created `07:11:51`, consumed `07:12:06` — **by the
|
||||||
|
previous controller**. The rebuilt one has nothing to consume and retries forever. Exactly the state
|
||||||
|
demo-hp sat in for 25 hours on 2026-08-03.
|
||||||
|
|
||||||
|
**Remedy:** an operator Re-issue. Performed here through the designed endpoint
|
||||||
|
(`POST /configs/demo-hp/offsite-reissue`, HTTP 303), after which the target configured normally.
|
||||||
|
|
||||||
|
### (b) R-196, measured live — and it lands squarely on the recovery path
|
||||||
|
|
||||||
|
```
|
||||||
|
host_escrow (demo-hp-bb76ea), after the Re-issue:
|
||||||
|
stale_at = 2026-08-04 20:15:49 ← set
|
||||||
|
restic_pw_sha256 = 8a9e33aa4da6… ← UNCHANGED. The key did not move.
|
||||||
|
```
|
||||||
|
|
||||||
|
**The escrow was marked stale while it perfectly covers the box's current key** — the recovered one.
|
||||||
|
That is R-196's false staleness, and here it is not a cosmetic lie: a stale escrow makes the hub
|
||||||
|
**withhold the hash from the report ACK**, so `EscrowAutoConfirmer` can never flip `pending →
|
||||||
|
escrowed`, and `OffboxRunnable` (`configured && escrowed`) **refuses to run any off-site backup**.
|
||||||
|
|
||||||
|
> **The remedy for (a) disables the recovery it was needed for.** The only documented way to clear a
|
||||||
|
> stale escrow is a fresh ceremony — which supersedes the identity blob, i.e. **destroys the very key
|
||||||
|
> being recovered**. Under hub v0.93.0 the old blob is now *retained*, but nothing can serve a
|
||||||
|
> superseded blob back (R-199's inventory: the endpoint serves the CURRENT row only).
|
||||||
|
|
||||||
|
**Superseded rows still number 2** — no ceremony was run tonight. The key is intact.
|
||||||
|
|
||||||
|
### (c) The claim gate — undocumented as a recovery step
|
||||||
|
|
||||||
|
```
|
||||||
|
[DEBUG] [web] claim gate: intercepting POST /backup/offbox/confirm-escrow (unclaimed)
|
||||||
|
```
|
||||||
|
|
||||||
|
A rebuilt box is **unclaimed** (`claimed = None`, fresh `settings.json`), and the claim gate correctly
|
||||||
|
intercepts every non-claim route. **So no controller endpoint can be driven at all** — not the manual
|
||||||
|
escrow confirm, not the backup trigger — until the customer re-claims the box. Both POSTs returned 302
|
||||||
|
to the claim page and neither reached its handler; `last_run` stayed null, which is why the earlier
|
||||||
|
reading of "the run was refused" needed this second look to be accurate.
|
||||||
|
|
||||||
|
This is correct behaviour and it is **not wrong** — but it is a step in the customer's recovery journey
|
||||||
|
that appears in no design document, and it comes *before* anything else can happen.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. How it was finished, and the fourth undocumented step
|
||||||
|
|
||||||
|
The unattended run stopped at the wall in §3 rather than improvising further at 22:35 (§8.10). With the
|
||||||
|
operator present the remaining path was walked, and it took **four steps a customer would have to
|
||||||
|
discover**:
|
||||||
|
|
||||||
|
1. **Re-issue** the offsite credential (hub, operator-only) — otherwise the box can never configure its
|
||||||
|
tier. §3(a).
|
||||||
|
2. **Re-claim the box.** The local escape hatch (`--print-reset-code`) mints a code — **and does not
|
||||||
|
work on its own**: it writes the new hash to `settings.json` while the RUNNING controller keeps its
|
||||||
|
old copy in memory, so `effectiveClaimCode()` never sees it and every attempt returns
|
||||||
|
*"Hibás vagy lejárt kód"*. **The controller must be restarted between minting and claiming.** Two
|
||||||
|
claim attempts failed on this before it was diagnosed; the generation counter reached 6.
|
||||||
|
3. **Confirm the escrow** manually (`/backup/offbox/confirm-escrow`) → `escrowed`, so `OffboxRunnable`
|
||||||
|
permits a run. **Not a ceremony** — the identity blob was never superseded (still 2 retained rows).
|
||||||
|
4. **Restore with `mode=full`.** The default `mode=unit` returns the recovery unit and **not** the
|
||||||
|
customer's files.
|
||||||
|
|
||||||
|
**None of these four is in any design document, and the drill needed all four.**
|
||||||
|
|
||||||
|
## 5. §5's five conditions, as recorded before the wipe
|
||||||
|
|
||||||
|
| # | Condition | Evidence |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | §4 confirmed, sentinel **listed by name** | snapshot `e6132ae5` (19:36:26) — `-rw-r--r-- 1000 1000 181 … /DRILL-SENTINEL.txt` |
|
||||||
|
| 2 | deliberate rollback archive, **verified** | `vzdump-lxc-9201-2026_08_04-21_44_24.tar.zst`, 1 606 765 083 B; **full zstd stream read OK** (4 867 573 760 B uncompressed); the sentinel confirmed **inside** it |
|
||||||
|
| 3 | §3's option | see §6 |
|
||||||
|
| 4 | demo-felhom untouched | `health=ok`, `escrow_state=escrowed`, not touched at any point |
|
||||||
|
| 5 | free space | nvme-1tb 883 GB free, root 20 GB, local-lvm 34.32 % |
|
||||||
|
|
||||||
|
**A precondition had drifted and was repaired before the wipe, not worked around.** The staged
|
||||||
|
snapshot no longer contained the sentinel: the afternoon's `[WARN] mandatory data path missing`
|
||||||
|
experiment produced a *later* same-day calibre-web snapshot, and restic's `forget --keep-daily 7
|
||||||
|
--group-by host,tags` had pruned the good one in its favour. The fixture on disk was correct, so one
|
||||||
|
backup re-established it and it was re-verified by listing. **Lesson worth carrying: a good snapshot is
|
||||||
|
not durable against a later bad run on the same day.**
|
||||||
|
|
||||||
|
## 6. §3 — the recovery code
|
||||||
|
|
||||||
|
**Option B as already in place, with a strict improvement: no new copy was created, so nothing needed
|
||||||
|
shredding.**
|
||||||
|
|
||||||
|
The operator placed `R_DEMO-HP` in their own `~/.config/credentials` on DooPlex (mode `0600`) two
|
||||||
|
sessions ago, deliberately, for this purpose. This session read it from there and **piped it to stdin**
|
||||||
|
for each of the two invocations that needed it. It was never an argument, never exported, never written
|
||||||
|
to a second file, and never logged.
|
||||||
|
|
||||||
|
**Nothing was shredded, and that is the point:** the operator's own permanent store is theirs, not a
|
||||||
|
drill artefact, and destroying it would have destroyed their record. Because no additional copy was
|
||||||
|
made, there is nothing left behind to prove gone — a stronger position than option B's
|
||||||
|
create-then-shred.
|
||||||
|
|
||||||
|
**Verified afterwards** with the planted-copy positive control, exactly as on 2026-08-04 afternoon —
|
||||||
|
see §8.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. The state the box is left in — healthy and re-armed
|
||||||
|
|
||||||
|
```
|
||||||
|
controller felhom-controller:0.197.0, healthy, CLAIMED
|
||||||
|
apps privatebin opengist calibre-web filebrowser cloudflared traefik — all serving
|
||||||
|
escrow_state escrowed (no ceremony was run; identity_blob untouched)
|
||||||
|
repo key 8a9e33aa4da6… THE RECOVERED KEY — and the repository opens with it
|
||||||
|
off-site 3 snapshots, 42 026 B, repo_state null; all three apps re-toggled for off-site
|
||||||
|
sentinel restored to its live location, sha256 643166269103a25c… — verified
|
||||||
|
```
|
||||||
|
|
||||||
|
The verification copy under `backups/offsite-restore/calibre-web/` is **left in place as evidence**.
|
||||||
|
|
||||||
|
## 8. R persisted nowhere — searched, with a positive control
|
||||||
|
|
||||||
|
Swept the agent journal, the controller container log, and `/tmp`, `/var/tmp`, `/var/lib/felhom-agent`,
|
||||||
|
`/root` on demo-hp for the recovery code: **0 hits**, and **0** leftover `felhom-idesc-*` unseal
|
||||||
|
staging directories. A planted copy was found by the same sweep (**1**) and not found after shredding
|
||||||
|
(**0**), so the instrument is shown sensitive rather than assumed to be.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Teardown — three layers
|
||||||
|
|
||||||
|
| layer | state |
|
||||||
|
|---|---|
|
||||||
|
| the guest | **Nothing torn down.** The wipe is the evidence; the apps are serving; the recovered key is in place. The scratch band is empty — no restore-test guest was created tonight. |
|
||||||
|
| the host | `vzdump` snapshot LVs removed cleanly by the archive job (`snap_vm-9201-disk-0/1_vzdump` both released). One new archive, 1.6 GB, on `nvme-1tb` (883 GB free). Nothing deleted. |
|
||||||
|
| the hub | **No new customer records.** `demo-hp` is the same customer row throughout — the rebuild was a controller-data wipe, not a re-enrolment, so nothing accumulated. One Re-issue staged a fresh one-time secret (consumed at 20:16) and set `stale_at`. The two superseded escrow rows are **unchanged** — no ceremony ran. |
|
||||||
|
|
||||||
|
**Nothing was deleted on the storage endpoint**, including the ~1.2 GB of previously-orphaned
|
||||||
|
ciphertext. Ruled, still owed, and deliberately not ridden along with a drill.
|
||||||
|
|
||||||
|
## 10. Part 2 — not run
|
||||||
|
|
||||||
|
Its gate is *"the drill PASSED; its evidence is written down; and there is time."* The first two are
|
||||||
|
now met. **The third was not** — it was 23:15 after a full destructive cycle, and Part 2 is a second
|
||||||
|
wipe. It is deliberately left for a session that starts fresh, with drill 1's result already recorded
|
||||||
|
so a second wipe cannot overwrite it.
|
||||||
|
|
||||||
|
**R-198's retention therefore remains unit-proven only.** Nothing has yet superseded a key in
|
||||||
|
production, so the retention's live behaviour is still unobserved — and it is now the last unproven
|
||||||
|
link in the chain this drill exercised. The check is cheap: wipe again, do NOT recover, let a ceremony
|
||||||
|
seal a different password, then read the superseded row's `identity_blob`.
|
||||||
@@ -0,0 +1,311 @@
|
|||||||
|
# DRILL RECORD — R-201, the wipe-and-recover proof: **PREPARED, HALTED BEFORE THE WIPE**
|
||||||
|
|
||||||
|
**Date:** 2026-08-04 · **Box:** `demo-hp` (Tier 0, the designated drill host) · **Nothing was wiped.**
|
||||||
|
**Outcome:** the drill did not reach its verdict. It was halted at step 4 by a defect that makes the
|
||||||
|
verdict unobtainable — and that defect is worth more than the drill.
|
||||||
|
|
||||||
|
> **THE HEADLINE.** A customer-declared **mandatory** data directory was **silently absent from the
|
||||||
|
> off-site snapshot**, while the backup reported `ok` with three snapshots. The only trace is one
|
||||||
|
> `[WARN]` line inside the controller container. The hub, the card and the counters all say the backup
|
||||||
|
> succeeded. → **R-203**
|
||||||
|
>
|
||||||
|
> **Nothing irreversible happened.** The wipe never ran. `demo-hp` is left in a BETTER state than it
|
||||||
|
> started: it now has a working off-site repository (it had an orphaned one), three snapshots, and a
|
||||||
|
> sentinel file on disk.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. The verdict — not reached, and why that is the correct outcome
|
||||||
|
|
||||||
|
The drill's pass condition is a byte-identical sentinel sha256 after a wipe. **Step 4 established that
|
||||||
|
the sentinel is not in the off-site snapshot at all.** Wiping the box would therefore have:
|
||||||
|
|
||||||
|
- destroyed the sentinel, which exists only on that box;
|
||||||
|
- proven nothing about recovery, because there would be nothing to recover;
|
||||||
|
- and done so *after* the point of no return.
|
||||||
|
|
||||||
|
The runbook's own rule applies: *"If this session finds itself writing code beyond Part 0, stop. That
|
||||||
|
means a precondition was wrong, and the finding outranks the drill."* A precondition was wrong. It was
|
||||||
|
one the runbook's P1–P6 table did not contain, because nobody knew to look for it.
|
||||||
|
|
||||||
|
**Sentinel sha256 (step 3), recorded and still on the box:**
|
||||||
|
`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`
|
||||||
|
at `/mnt/sys_drive/userdata/media/books/DRILL-SENTINEL.txt` (181 B).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. R-203 — the defect that halted the drill
|
||||||
|
|
||||||
|
**Measured, twice, on the live box.**
|
||||||
|
|
||||||
|
| what | path | exists? |
|
||||||
|
|---|---|---|
|
||||||
|
| the app's live bind, where the customer's files actually land | `/mnt/sys_drive/userdata/media/books` | **YES** (the sentinel is here) |
|
||||||
|
| the path the off-site capture set treats as the mandatory directory | `/mnt/sys_drive/felhom-data/userdata/media/books` | **NO** |
|
||||||
|
|
||||||
|
The controller's own log, verbatim:
|
||||||
|
|
||||||
|
```
|
||||||
|
[WARN] [offbox] calibre-web: mandatory data path missing on disk, skipped from offsite:
|
||||||
|
/mnt/sys_drive/felhom-data/userdata/media/books
|
||||||
|
[INFO] [offbox] backed up calibre-web (…/backups/primary/calibre-web, 0 mandatory path(s))
|
||||||
|
[INFO] [offbox] backup OK: 3 app(s) backed up, 3 snapshot(s), 37s
|
||||||
|
```
|
||||||
|
|
||||||
|
**The run reported `ok`.** `last_status: ok`, `last_success` stamped, `snapshot_count: 3`. Nothing
|
||||||
|
customer-visible, nothing hub-visible and nothing in any counter says the mandatory directory was
|
||||||
|
dropped. This is `CLAUDE.md`'s recurring shape — *a path the customer thinks is protected is not in the
|
||||||
|
snapshot* — and the code even has the right words for it in a WARN nobody reads.
|
||||||
|
|
||||||
|
**The mechanism, from source.** `NamespaceRoot(drivePath, inGuestDrive)`
|
||||||
|
(`appbackup/paths.go:28-33`) appends `felhom-data` **when the drive IS the system data path**
|
||||||
|
(`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath != m.systemDataPath)`,
|
||||||
|
`backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is documented and computed
|
||||||
|
as **`<HDD_PATH>/userdata`** (`stacks/classify_binds.go:14`). With `system_data_path: /mnt/sys_drive`
|
||||||
|
and an app deployed at `HDD_PATH=/mnt/sys_drive`, those two produce different directories.
|
||||||
|
|
||||||
|
**And the same compose file used BOTH roots.** From `docker inspect calibre-web`:
|
||||||
|
|
||||||
|
```
|
||||||
|
bind /mnt/sys_drive/felhom-data/userdata/import/calibre -> /cwa-book-ingest ← felhom-data root
|
||||||
|
bind /mnt/sys_drive/userdata/media/books -> /calibre-library ← NO felhom-data root
|
||||||
|
```
|
||||||
|
|
||||||
|
`${IMPORT_PATH}` resolved *with* the segment; `${USERDATA_PATH}` resolved *without* it. One deploy, one
|
||||||
|
template, two roots.
|
||||||
|
|
||||||
|
### What is measured and what is not — stated because the scope changes the fix
|
||||||
|
|
||||||
|
- **MEASURED:** the two paths disagree; the mandatory directory is absent from the snapshot; the run
|
||||||
|
reports `ok`; the only signal is a container-log WARN.
|
||||||
|
- **NOT ESTABLISHED:** whether `HDD_PATH=/mnt/sys_drive` is a *supported* choice. It was chosen because
|
||||||
|
demo-hp's only registered drive, `Felhom-Share`, is a NAS and was **correctly refused** as an app
|
||||||
|
namespace (R-108's `RefuseAsAppNamespace`, working as designed). `/mnt/sys_drive` was **accepted**
|
||||||
|
(HTTP 202) rather than refused.
|
||||||
|
|
||||||
|
**Either branch is a defect, which is why this is filed regardless:**
|
||||||
|
|
||||||
|
- if the system drive **is** a supported app namespace → the userdata path resolution is wrong for
|
||||||
|
every app deployed on it, and their mandatory directories are silently unprotected;
|
||||||
|
- if it is **not** supported → the deploy accepted a namespace it should have refused, exactly as it
|
||||||
|
refused the NAS one call earlier, and the refusal that exists is not reaching this case.
|
||||||
|
|
||||||
|
**What must NOT be concluded from this drill:** that off-site backups are broken generally. The two
|
||||||
|
pre-existing apps (`opengist`, `privatebin`) declare **no** mandatory userdata paths — everything they
|
||||||
|
own is in named volumes — so they are unaffected, and their snapshots are real.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Preconditions, each measured
|
||||||
|
|
||||||
|
| # | Precondition | Result |
|
||||||
|
|---|---|---|
|
||||||
|
| **P1** | operator holds the recovery code | **PASS** — `R_DEMO-HP`, recorded by the operator in the DooPlex credentials file (`~/.config/credentials`). *The code itself appears nowhere in this record.* |
|
||||||
|
| **P2** | demo-hp on agent v0.125.0 + controller v0.196.0 | **FAILED ON ARRIVAL, FIXED** — the box was on agent **v0.124.1**. Deployed the published v0.125.0 (sha `f7d8339b53d9…`, verified against the release) and controller v0.196.0. `age` present at `/usr/bin/age`. |
|
||||||
|
| **P3** | the current `identity_blob` seals the repository under test | **PASS with a caveat that reshaped the drill** — `identity_blob` = 572 B, `restic_pw_sha256` = `8a9e33aa4da6…`. But **no repository existed under that key** (see §4). |
|
||||||
|
| **P4** | a deliberate, verified whole-guest archive as rollback | **NOT TAKEN** — deliberately. It is only needed for the wipe, and the wipe did not happen. |
|
||||||
|
| **P5** | demo-felhom untouched and healthy | **PASS** — not touched at any point. |
|
||||||
|
| **P6** | space | **PASS** — `felhom-backup` 927 GB free, `local-lvm` 30.9 % used, guest 64 GB free. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Step-by-step, with every observable
|
||||||
|
|
||||||
|
### Step 1 — starting state (hub, read-only)
|
||||||
|
|
||||||
|
`demo-hp-bb76ea`: `identity_blob` 572 B, `restic_pw_sha256` `8a9e33aa4da6769c5aea1831f87759e10930e2ec1dea0062576484e0598d080a`, created `2026-08-04T11:11:37Z`, `escrow_state: escrowed`.
|
||||||
|
Report: `snapshot_count: 0`, `repo_size_bytes: 0`, **no `last_run`/`last_status` at all** — the shape of
|
||||||
|
a controller that has never run an off-site backup in this lifetime.
|
||||||
|
|
||||||
|
### Step 1b — the repository was ORPHANED, and this is the first live proof of a prediction
|
||||||
|
|
||||||
|
The 2026-08-04 spike predicted from source — and could not measure — that a rebuilt box's next
|
||||||
|
off-site run would hit a **third** outcome: neither reattaching the old snapshots nor silently starting
|
||||||
|
a fresh history, but **refusing**. Measured here, twice over.
|
||||||
|
|
||||||
|
**Read-only probe first** (`restic cat config` with the current key, writes nothing):
|
||||||
|
|
||||||
|
```
|
||||||
|
Fatal: wrong password or no key found
|
||||||
|
```
|
||||||
|
|
||||||
|
— the exact string `classifyResticProbe` maps to `"orphaned"`.
|
||||||
|
|
||||||
|
**Then the real run, through the customer UI endpoint** (`POST /backup/offbox/run`):
|
||||||
|
|
||||||
|
```
|
||||||
|
[WARN] [offbox] offsite repo ORPHANED — remote holds backups written under a previous,
|
||||||
|
no-longer-available key; runs will skip until reset
|
||||||
|
[WARN] [offbox] run skipped — offsite repo orphaned (card shown; awaiting reset)
|
||||||
|
[INFO] Event pushed: offbox_repo_orphaned (warning) — A távoli mentési tároló elárvult …
|
||||||
|
```
|
||||||
|
|
||||||
|
`repo_state: "orphaned"`, `orphaned_at: 2026-08-04T12:37:30Z`, `last_status: "error"`, the orphan card
|
||||||
|
rendered on `/backups/remote`, and the event reached the hub (HTTP 200).
|
||||||
|
|
||||||
|
> **The system stopped and said so. It did not quietly start a new history over the old one.**
|
||||||
|
> That closes R-193's open Q3 — and it is the good half of this month's story.
|
||||||
|
|
||||||
|
**Why the repository was orphaned:** the 15 snapshots / 40.9 MB at `/home/felhom-repo` were written
|
||||||
|
under key `8e03eddf9ff7…` before the 2026-08-03 rebuild. That key survives only in superseded escrow
|
||||||
|
row **id 3**, which carries **`identity_blob` = NULL** — it was superseded at `2026-08-04 07:15:36`,
|
||||||
|
**four hours before** hub v0.93.0 fixed the retention. Unrecoverable, permanently, with or without a
|
||||||
|
recovery code.
|
||||||
|
|
||||||
|
### Step 1c — the reset (operator-authorised)
|
||||||
|
|
||||||
|
The orphan card's own reset, confirmed by the operator during the session:
|
||||||
|
|
||||||
|
```
|
||||||
|
[WARN] [offbox] resetting orphaned repo (operator-confirmed (claimed)):
|
||||||
|
move-aside /home/felhom-repo -> /home/felhom-repo.orphaned-20260804, then re-init
|
||||||
|
[INFO] [offbox] orphaned repo reset complete — old history set aside at
|
||||||
|
/home/felhom-repo.orphaned-20260804 (move-aside, not deleted); fresh repo initialized
|
||||||
|
[INFO] Event pushed: offbox_repo_reset (info)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Nothing was deleted.** The reset path had never run in anger before; it works.
|
||||||
|
|
||||||
|
### Steps 2–3 — the recovery code and the sentinel
|
||||||
|
|
||||||
|
The operator ran the ceremonies for **both** boxes earlier the same day and saved the codes. Measured
|
||||||
|
hub-side: the new escrow rows are stamped `2026-08-04T11:11:37Z` (demo-hp) and `11:13:06Z`
|
||||||
|
(demo-felhom), and `restic_pw_sha256` is **unchanged** on both — so `SaveHostEscrow` correctly treated
|
||||||
|
them as same-password re-ceremonies: **no superseded row, no `offsite_repo_key_changed`**. That is
|
||||||
|
v0.93.0's Scenario E, live.
|
||||||
|
|
||||||
|
**A file-leg app had to be deployed**, and this is a precondition the runbook did not anticipate:
|
||||||
|
neither off-site-toggled app on the box (`opengist`, `privatebin`) has a restorable file leg — both
|
||||||
|
keep everything in **named volumes**, which the off-site tier backs up as tars but the customer restore
|
||||||
|
flow **never unpacks** (`offbox_reconstitute.go`). A sentinel in either would have been unrecoverable
|
||||||
|
by design.
|
||||||
|
|
||||||
|
`calibre-web` was chosen: it declares `userdata: media/books class: mandatory`, and it is a
|
||||||
|
single-container app. Deployed through the real API (`POST /api/stacks/calibre-web/deploy`, HTTP 202),
|
||||||
|
toggled for off-site, and a Tier-1 recovery unit captured
|
||||||
|
(`Recovery unit captured for calibre-web → …/backups/primary/calibre-web`).
|
||||||
|
|
||||||
|
Sentinel written, `sync`ed, hashed: **`643166269103a25c…`**, 181 B.
|
||||||
|
|
||||||
|
### Step 4 — the pre-wipe off-site backup: **`ok`, and wrong**
|
||||||
|
|
||||||
|
```
|
||||||
|
last_run 2026-08-04T12:54:50Z · last_status ok · last_success stamped
|
||||||
|
snapshot_count 3 · repo_size_bytes 30 636 · repo_state null
|
||||||
|
```
|
||||||
|
|
||||||
|
Three snapshots, three apps, 37 s — and **the sentinel is in none of them**, per §2.
|
||||||
|
|
||||||
|
### Steps 5–11 — NOT RUN
|
||||||
|
|
||||||
|
Step 5 (verify the pre-wipe snapshot contents) would have confirmed the absence a second way; it is
|
||||||
|
moot given §2. Steps 6–11 (archive, wipe, reinstall, recover, install, restore, compare) were **not
|
||||||
|
started**. The §7 STOP was never reached, because the drill failed its own precondition first.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Part 0 — shipped, tested, deployed (R-200's plumbing half)
|
||||||
|
|
||||||
|
`--recover-offsite-install` (controller **v0.196.0**, commit `1b1366b`): same fetch → unseal → extract
|
||||||
|
path as `--recover-offsite-check`, same STDIN discipline for R, but it **places** the recovered
|
||||||
|
password via `InjectOffboxPassword`.
|
||||||
|
|
||||||
|
- **Confirmation is a second invocation.** Without `--confirm-install` it prints both hashes and writes
|
||||||
|
nothing. A single interactive prompt would have had to share stdin with R.
|
||||||
|
- **Three outcomes, named distinctly:** *installed* (no local password — the rebuilt-box shape),
|
||||||
|
*unchanged* (identical key already present, nothing written), *refused* (a DIFFERENT key present;
|
||||||
|
installing would clobber the key the current repository is encrypted under — exit 2, no force offered).
|
||||||
|
- It re-reads the file after writing rather than trusting the call's return.
|
||||||
|
|
||||||
|
**Tests + red-proof.** `go build && go vet && go test ./...` rc=0; `controller_gates.py --fast` OK.
|
||||||
|
Removing the confirmation gate makes the dry run write the password and fails
|
||||||
|
`TestRecoverAndInstall_InstallsOnABareBox` with *"the DRY RUN wrote the password"*. The R-persistence
|
||||||
|
test carries a **positive control** — a planted copy of the code is found by the sweep, then removed
|
||||||
|
and not found — because an absence check is worth only what its sensitivity is.
|
||||||
|
|
||||||
|
**It was deployed to demo-hp and never exercised against a live recovery**, because the drill halted
|
||||||
|
before step 9. Its unit proof stands; its live proof does not exist.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. What this drill did and did not establish
|
||||||
|
|
||||||
|
**Established, live, for the first time:**
|
||||||
|
|
||||||
|
1. A rebuilt box's off-site repository is **orphaned and the run refuses** — the spike's predicted
|
||||||
|
third outcome, measured. It does not silently start a fresh history.
|
||||||
|
2. The **orphan reset works**: move-aside, never delete, fresh repo initialised, event pushed.
|
||||||
|
3. **A customer-declared mandatory data directory can be silently absent from the off-site snapshot
|
||||||
|
while the run reports `ok`** (R-203).
|
||||||
|
4. demo-hp's pre-rebuild off-site history is **permanently unrecoverable** — its key was destroyed four
|
||||||
|
hours before the fix that would have kept it.
|
||||||
|
|
||||||
|
**NOT established — and unchanged from before this session:**
|
||||||
|
|
||||||
|
- **No file has ever been restored from an off-site backup after a wipe.** R-201's question is still
|
||||||
|
open, and its pass condition is unchanged.
|
||||||
|
- Part 0's install path has never run against a live recovery.
|
||||||
|
- The v0.93.0 `identity_blob` retention is **still unit-proven only** — nothing in this session
|
||||||
|
superseded a key, so nothing exercised it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. State left behind, and teardown
|
||||||
|
|
||||||
|
**Deliberately not torn down** — this is evidence, and the box is better off than it was:
|
||||||
|
|
||||||
|
| layer | state |
|
||||||
|
|---|---|
|
||||||
|
| the guest | `calibre-web` deployed and running, off-site-toggled, sentinel file in place. **Kept** as the drill fixture for the resumed run — it is the only app on either demo box with a restorable file leg. |
|
||||||
|
| the off-site repository | fresh, working, 3 snapshots, 30 636 B. The old 15-snapshot history is **set aside**, not deleted, at `/home/felhom-repo.orphaned-20260804`. |
|
||||||
|
| the host | agent v0.125.0, controller v0.196.0, `pvesm` unchanged apart from normal usage. |
|
||||||
|
| the hub | `offbox_repo_orphaned` + `offbox_repo_reset` events recorded for `demo-hp`. No scratch customer records were created — **nothing was reinstalled**. |
|
||||||
|
|
||||||
|
**The ~1.2 GB of previously-orphaned ciphertext was NOT deleted** — that act is ruled and still owed,
|
||||||
|
and §8.3 of the runbook forbids riding it along with a drill. The reset added
|
||||||
|
`/home/felhom-repo.orphaned-20260804` (≈41 MB) to what is set aside.
|
||||||
|
|
||||||
|
**No `R` persisted anywhere** — the recovery code was never used in this session. It was read only to
|
||||||
|
confirm the key exists in the credentials file; no unseal was performed on demo-hp.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7b. UPDATE, same day — the blocker is FIXED and the drill is ready to resume
|
||||||
|
|
||||||
|
Controller **v0.197.0** shipped both halves of R-203:
|
||||||
|
|
||||||
|
- **the paths agree.** On demo-hp the live bind moved from `/mnt/sys_drive/userdata/media/books` to
|
||||||
|
`/mnt/sys_drive/felhom-data/userdata/media/books` — the directory the capture set looks in. The
|
||||||
|
capture log went from `0 mandatory path(s)` to **`1 mandatory path(s)`**.
|
||||||
|
- **`ok` means it.** A run that cannot capture a MANDATORY directory now reports **`incomplete`**,
|
||||||
|
names the app and the folders, and raises the operator digest — instead of `ok` with a warning
|
||||||
|
beside it.
|
||||||
|
|
||||||
|
**And the sentinel is in the snapshot, listed by name:**
|
||||||
|
|
||||||
|
```
|
||||||
|
$ restic ls -l latest --tag calibre-web
|
||||||
|
-rw-r--r-- 1000 1000 181 2026-08-04 12:53:06 /mnt/sys_drive/felhom-data/userdata/media/books/DRILL-SENTINEL.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
sha256 **`643166269103a25cf41d34a26b75fd6ebba0a837bbcd7e0d2b5aba7106bcbe7c`** — byte-identical to §1,
|
||||||
|
verified after the fix's migration moved the file to the corrected directory.
|
||||||
|
|
||||||
|
**One live check could NOT be reproduced, and is recorded rather than claimed.** Hiding the books
|
||||||
|
directory to watch the `incomplete` verdict fire on hardware did not work: the running container's
|
||||||
|
bind mount **recreated** the directory, so `os.Stat` succeeded and there was no gap. That is itself
|
||||||
|
worth knowing — a bind-mounted directory cannot easily be "missing" while its app runs, so the
|
||||||
|
mandatory-gap condition arises in practice when the path resolves somewhere the app never binds (the
|
||||||
|
R-203 case), not when a live app's own directory vanishes. The verdict is proven by the run-level
|
||||||
|
test, which drives the real `RunOffboxBackup` and asserts both `incomplete` and the operator signal.
|
||||||
|
The fixture was restored and the sentinel re-verified at the same hash.
|
||||||
|
|
||||||
|
## 8. To resume the drill
|
||||||
|
|
||||||
|
1. **Fix or scope R-203.** Until the mandatory userdata path lands in the snapshot, no sentinel can
|
||||||
|
survive a wipe and the drill cannot reach its verdict.
|
||||||
|
2. Re-run steps 4–5 and confirm the sentinel IS in the snapshot — by listing it, not by a green status.
|
||||||
|
3. Then P4 (the deliberate archive), the §7 STOP, and steps 6–11 as written.
|
||||||
|
|
||||||
|
Everything else is already in place: the code, the versions, the recovery code, the working repository,
|
||||||
|
the file-leg app and the sentinel.
|
||||||
@@ -0,0 +1,531 @@
|
|||||||
|
# RECON — can a customer actually get their backups back?
|
||||||
|
|
||||||
|
**Date:** 2026-08-04 · **Items:** R-193, R-192 (+ new: R-198…R-201) · **Class:** recon — **no production code**
|
||||||
|
**Session scope:** read-only everywhere. Nothing built, deployed, wiped, rotated, re-issued or deleted.
|
||||||
|
|
||||||
|
> **Headline, before the detail.** The chain is **not** assembled, and the reason is worse than "two
|
||||||
|
> links are manual". **The hub's superseded-escrow retention — the feature built for exactly this
|
||||||
|
> incident — does not retain the offsite repository password.** `host_escrow_superseded` has no
|
||||||
|
> `identity_blob` column, and `demoteCurrentEscrowTx` copies only the K-escrow blob. So the last copy
|
||||||
|
> of a repo password is destroyed by the **next escrow ceremony** — the very act the system tells the
|
||||||
|
> customer to perform when their escrow goes `pending` after a rebuild. The recovery window closes
|
||||||
|
> because the remedy closes it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Confirmed baselines (re-read on arrival)
|
||||||
|
|
||||||
|
| Repo | `main` @ commit | tree | Version |
|
||||||
|
|------|-----------------|------|---------|
|
||||||
|
| `felhom-agent` | `856a127cd60b0be797867c162f0ae49c7b64993e` | clean | v0.124.1 |
|
||||||
|
| `felhom-controller` | `0887fd676d608e7c103fa13b3e87536ad2481811` | clean | v0.194.0 |
|
||||||
|
| `felhom.eu` | `d26f49ad680e2be2ce2888fd58dff7dee886e4ad` | clean | hub v0.92.0 |
|
||||||
|
|
||||||
|
All three match the prompt's cited commits exactly. `git status --porcelain` empty in each.
|
||||||
|
|
||||||
|
**Highest register ID: R-197** (`OPEN-ITEMS.md` R-196/R-197 were minted by yesterday's spike;
|
||||||
|
R-194/R-195 by the R-190 arc). Grepped across `OPEN-ITEMS.md`, `ROADMAP.md`, `STATUS.md`,
|
||||||
|
`CONTEXT.md`. New items in this session therefore start at **R-198**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Instruments — what each can and cannot answer
|
||||||
|
|
||||||
|
| # | Instrument | Exact read | Known limit |
|
||||||
|
|---|---|---|---|
|
||||||
|
| I1 | source | the three repos at the commits above | the authority; every claim below cites a file:line |
|
||||||
|
| I2 | hub SQLite snapshot | `kubectl -n felhom-system exec pod/hub-6775889d95-gc8sk -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | the `-wal` is load-bearing (copied; `PRAGMA integrity_check` → `ok`) |
|
||||||
|
| I2-fresh | freshness proof | `MAX(host_reports.received_at)` = `2026-08-04 10:00:40`, `datetime('now')` = `2026-08-04 10:03:48` | **positive observable**: newest row 3 m 08 s old, not "the query did not error" |
|
||||||
|
| I3 | live filesystem, demo-felhom guest 9201 | `ssh felhom-pve "pct exec 9201 -- ls -la --time-style=full-iso …"` | read-only `ls`; mtimes only, no file contents read |
|
||||||
|
| I4 | caller census | `grep -rn <symbol> --include=*.go` across all four repos | proves absence of a caller in Go; templates checked separately |
|
||||||
|
|
||||||
|
**Not used, deliberately:** no ceremony, no re-issue, no rotation, no `selftest=identity-consume`, no
|
||||||
|
write of any kind, nothing touched on the storage endpoint.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Q1 — the chain, link by link
|
||||||
|
|
||||||
|
From a dead node to an open repository. **Status vocabulary as specified.**
|
||||||
|
|
||||||
|
| # | Link | Component / function | Who invokes it | Status |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| 1 | repo password is minted | controller `Manager.WriteOffboxSecrets` → mint at `internal/backup/offbox.go:392-401`; generator `:405` | `ApplyOffsiteTarget` (`offbox.go:474`) ← `ConfigureOffbox` ← the apply-bridge (`offsiteapply.go:221,256`) | **PROVEN-LIVE** — fires on every fresh data dir; measured twice this week (I2 hash change, I3 mtime) |
|
||||||
|
| 2 | password staged to the agent | controller `PushOffboxPasswordForEscrow` (`offbox.go:530`) → agent `POST /escrow/stage-secret` (`localapi/server.go:518`) → `escrow.StagedResticPasswordPath()` = `/var/lib/felhom-agent/escrow-stage/restic_repo_password`, 0600 (`identity.go:44-47`) | the controller enable/ceremony path | **PROVEN-LIVE** |
|
||||||
|
| 3 | sealed under R at the ceremony | `AttachResticPassword` (`identity.go:73`) → `WrapIdentityBundle` → `WrapIdentity` (`identity.go:114`, `age -p -a`, scrypt + ChaCha20-Poly1305) | agent `--selftest=escrow-create`, driven by the controller wizard (`/escrow/ceremony`, `localapi/server.go:525-527`) | **PROVEN-LIVE** — customer-facing wizard first live firing 2026-07-18 (`00-capability-map.md`, escrow row) |
|
||||||
|
| 4 | stored by the hub | `PUT /api/v1/hosts/<id>/escrow` (`api/handler.go:239`, `handleHostEscrowPut :1104`) → `host_escrow.identity_blob` | agent `main.go:2937` | **PROVEN-LIVE** — measured: `identity_blob` present, 572 B, for **both** hosts (I2) |
|
||||||
|
| 5 | **old blob retained on supersession** | `demoteCurrentEscrowTx` (`store/store.go:2547-2556`) → `host_escrow_superseded` | `SaveHostEscrow`, `DeleteHost` | **BROKEN FOR THIS PURPOSE.** The INSERT copies `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256` — **not `identity_blob`**, and the table has **no such column** (`PRAGMA table_info`, I2). The K-escrow survives; **the restic repo password does not** |
|
||||||
|
| 6 | served back to a re-enrolling box | hub `handleReEnroll` (`api/dr.go:101`) and `handleGetRestoreDirective` (`:155`), returning `identity_escrow_b64`; recovery-mode gated, armed with the **global operator key** (`:34`) | **NOBODY.** I4: zero callers in `felhom-agent` (no `ReEnroll` symbol, no `/re-enroll` or `/restore-directive` string anywhere in the agent's hub client, `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`); zero in the hub UI (no template mentions them); zero in `scripts/` or any runbook | **implemented-never-exercised, and CLIENTLESS.** The only documented retrieval is by hand: `sqlite3 "SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …"` on a `kubectl cp`-ed `hub.db` (project memory, 2026-07-04 S5 prep) |
|
||||||
|
| 7 | unsealed with R | `UnwrapIdentityBundle` (`identity.go:172`) → `UnwrapIdentity` (`:139`) | **`runSelftestIdentityConsume` only** (`cmd/felhom-agent/main.go:2845-2900`), reading R from `FELHOM_RECOVERY_CODE`. I4: no other production caller | **manual-only — the only caller is a `--selftest` mode** |
|
||||||
|
| 8 | the repo password extracted | none. The selftest writes the **whole bundle JSON** to `-keydest` 0600 (`main.go:2872`) and prints *"identity recovered (tunnel_token + pbs_token)"* — it does not mention, extract or route `restic_repo_password` at all | a human reads the JSON | **missing** |
|
||||||
|
| 9 | placed on the rebuilt controller | `InjectOffboxPassword` (`offbox.go:541`) | one caller: `offboxInjectPasswordHandler` (`web/offbox_handlers.go:189`) at `POST /backup/offbox/inject-password` (`web/server.go:510`). **I4: no template in the repo contains that path or any form posting to it** | **manual-only, and UI-unreachable** — the handler exists, the form does not. A person must POST it by hand with a valid session cookie + CSRF token |
|
||||||
|
| 10 | the existing repo opens | `ensureOffboxRepo` (`offbox.go:653`): `restic cat config` → success clears `RepoState` | the scheduled/manual offbox run | **implemented-never-exercised** on a *recovered* password (exercised constantly on a live one) |
|
||||||
|
| 11 | a backup or restore proves it | — | — | **never** |
|
||||||
|
|
||||||
|
### The chain in one sentence
|
||||||
|
|
||||||
|
Links 1–4 are real and proven. **Link 5 silently drops the payload.** Links 6–9 are each individually
|
||||||
|
manual, and link 6 has no client at all. Link 11 has never happened.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Q2 — where the chain stops today
|
||||||
|
|
||||||
|
**The first link that is not automatic is #5, and it is not merely manual — it is lossy.**
|
||||||
|
|
||||||
|
Two distinct stopping points, and the order matters:
|
||||||
|
|
||||||
|
**(a) If the box has NOT re-escrowed since the rebuild** — the current `host_escrow.identity_blob`
|
||||||
|
still seals the *old* password. Recovery is possible in principle, and a person must:
|
||||||
|
|
||||||
|
1. Arm recovery mode on the hub (`PUT /api/v1/admin/hosts/<id>/recovery-mode`, global operator key) —
|
||||||
|
or skip it entirely, because nothing consumes the endpoint it gates;
|
||||||
|
2. `kubectl cp` / `exec cat` the hub DB and `sqlite3 … writefile('/root/idblob', identity_blob)`;
|
||||||
|
3. copy the blob to a machine with the agent binary and `age`;
|
||||||
|
4. `FELHOM_RECOVERY_CODE=… felhom-agent --selftest=identity-consume -blob /root/idblob -keydest /root/bundle.json`;
|
||||||
|
5. open `bundle.json` by hand and read `restic_repo_password` out of it;
|
||||||
|
6. log in to the customer's dashboard and `POST /backup/offbox/inject-password` **by hand** (curl with
|
||||||
|
session + CSRF) — there is no form;
|
||||||
|
7. re-run the apply so `WriteOffboxSecrets` finds the file present and keeps it;
|
||||||
|
8. trigger a run and hope `restic cat config` succeeds.
|
||||||
|
|
||||||
|
Eight manual steps, five of them requiring operator-tier access to the hub, one requiring the customer
|
||||||
|
to be present with R, and one (step 6) with no user interface at all.
|
||||||
|
|
||||||
|
**(b) If the box HAS re-escrowed** — which is what the system pushes the customer to do, because a
|
||||||
|
rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`,
|
||||||
|
`offbox.go:570`), and the card says *„Helyreállítási kód szükséges"* — then the old `identity_blob`
|
||||||
|
has been **overwritten** by `SaveHostEscrow` and **not carried** into the superseded row. At that point
|
||||||
|
**no procedure exists**, with or without R, at any level of effort. The password is gone.
|
||||||
|
|
||||||
|
Both demo boxes are in state (b) as of 2026-08-04 07:15:36 (demo-hp) and 07:20:08 (demo-felhom) — I2,
|
||||||
|
`host_escrow.updated_at`. This is an **independent, and much stronger, reason** the 51 orphaned
|
||||||
|
snapshots are unrecoverable than "nobody kept the recovery codes". Keeping R would not have helped.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Q3 — can a rebuilt controller know it should recover?
|
||||||
|
|
||||||
|
### RULING: **YES — the hub already tells it, on every single report, and the controller throws the answer away.**
|
||||||
|
|
||||||
|
**What the box knows at the mint decision.** `WriteOffboxSecrets` is called from
|
||||||
|
`ApplyOffsiteTarget`, driven by the apply-bridge with the hub's offsite **descriptor**, whose fields
|
||||||
|
are `host, user, port, repo_path, host_fingerprint, quota_gb` (`offsiteapply.go:198-204`). **Nothing
|
||||||
|
about escrow is in scope at that point.** Locally, the only signal is the one the mint branch already
|
||||||
|
reads: `os.Stat(m.offboxPwPath())` returning `IsNotExist` — which *is* the fresh-data-dir signal, at
|
||||||
|
exactly the right line, but cannot distinguish "never configured" from "rebuilt".
|
||||||
|
|
||||||
|
**What the box is nonetheless already told.** Every report ACK carries an `escrow` object
|
||||||
|
(`api/handler.go:504-510`), consumed as `report.EscrowStatus`
|
||||||
|
(`internal/report/escrow_confirm.go:19-23`):
|
||||||
|
|
||||||
|
```go
|
||||||
|
type EscrowStatus struct {
|
||||||
|
IdentityBlobPresent bool `json:"identity_blob_present"`
|
||||||
|
ResticPwSHA256 string `json:"restic_pw_sha256"`
|
||||||
|
CreatedAt string `json:"created_at"`
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
That is precisely *"the hub has a blob, and here is the hash of the password it covers"* — arriving
|
||||||
|
every report cycle, on a channel that already exists, with no new API and no new secret.
|
||||||
|
|
||||||
|
**And it is discarded.** `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:75-84`) returns
|
||||||
|
immediately unless `Pending()` or `Escrowed()` — both of which require `OffboxConfigured()` **and** a
|
||||||
|
non-nil target (`cmd/controller/main.go:580-590`). On a freshly rebuilt box, before the offsite apply,
|
||||||
|
there is no target, so the status is dropped on the floor. It is never persisted anywhere: I4 finds
|
||||||
|
exactly **one** reference to the ACK field in the whole controller
|
||||||
|
(`cmd/controller/main.go:641: escrowConfirmer.Reconcile(resp.Escrow)`).
|
||||||
|
|
||||||
|
**Therefore a fail-closed refusal IS implementable**, with no new hub endpoint:
|
||||||
|
|
||||||
|
- persist the last-seen ACK escrow status alongside settings (the precedent exists — `ClaimSync`
|
||||||
|
caches the hub's claim-code state into `settings.json` idempotently by generation,
|
||||||
|
`internal/report/claim_sync.go:39-53`, and it does so *set-only, never cleared on hub silence*,
|
||||||
|
which is the correct shape here too);
|
||||||
|
- at the mint branch, refuse when the password file is absent **and** the cached status says
|
||||||
|
`identity_blob_present && restic_pw_sha256 != ""`.
|
||||||
|
|
||||||
|
**One ordering caveat, stated rather than assumed.** On a truly fresh box the apply can race the first
|
||||||
|
ACK. Measured on demo-hp: first post-rebuild report `2026-08-03 06:12:19`, offsite apply
|
||||||
|
`2026-08-04 07:12` — the ACK preceded the apply by 25 hours, but that gap was an incident, not a
|
||||||
|
design. A fail-closed rule handles the race correctly by construction: refuse to mint until the escrow
|
||||||
|
status is known, and retry next cycle (the apply-bridge is already built to retry and to persist
|
||||||
|
nothing on a partial failure, `offsiteapply.go:1-6`).
|
||||||
|
|
||||||
|
**Corroborating measurement that the data dir really is fresh on a rebuild** (I3, demo-felhom guest
|
||||||
|
9201, `/var/lib/docker/volumes/felhom-controller-data/_data/data/`):
|
||||||
|
|
||||||
|
| file | mtime |
|
||||||
|
|---|---|
|
||||||
|
| the data directory's parent | `2026-08-03 07:17:25` |
|
||||||
|
| `encryption.key` (32 B) | `2026-08-03 07:17:26` |
|
||||||
|
| `offbox/` | `2026-08-03 07:18:02` |
|
||||||
|
| `offbox/repo_password` (64 B) | `2026-08-03 07:18:02` |
|
||||||
|
|
||||||
|
The volume, the app-secret encryption key and the repo password are all newborn, 36 seconds apart.
|
||||||
|
Corroborated hub-side (I2): `json_extract(report_json,'$.claimed')` reads **1 → 0 → 1** across the
|
||||||
|
rebuild on both boxes — demo-hp `0` in 5 reports `2026-08-03 06:12:19`–`06:21:03`; demo-felhom `0` in 4
|
||||||
|
reports `07:17:54`–`07:19:46`. `Claimed` is set only by `SetClaimed()`, whose sole caller is the claim
|
||||||
|
submit handler (`web/claim.go:349`), so the return to `1` was a **human re-claim**, not automation.
|
||||||
|
|
||||||
|
> **Note, not acted on:** a fresh `encryption.key` means every pre-rebuild `app.yaml` `ENC:` secret is
|
||||||
|
> undecryptable — the C-1 circular dependency in `_recovery-inventory-2026-07-28.md` fired on both
|
||||||
|
> demo boxes this week. That is outside R-193's scope and is recorded as an observation (§13).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Q4 — what the stale-escrow detector actually covers
|
||||||
|
|
||||||
|
### RULING: **the prompt's premise conflates two different mechanisms. `escrow_confirm.go` DOES fire on a mint. But it fires into a log file, and the remedy it prescribes destroys the last copy of the old key.**
|
||||||
|
|
||||||
|
**The conflation, stated because it matters.** Yesterday's spike's sentence *"wired to the one path
|
||||||
|
that doesn't change the key and absent from the one that does"* is about the **hub's `escrow_stale`
|
||||||
|
event**, emitted by `ReissueCredentials` (`hub/internal/offsite/offsite.go:198-201`) — that is R-196,
|
||||||
|
and it stands. It is **not** about `controller/internal/report/escrow_confirm.go`. Two different
|
||||||
|
detectors; only one of them was ever claimed to be misplaced.
|
||||||
|
|
||||||
|
**What `escrow_confirm.go` does on a mint, from source.** After a rebuild the apply creates a target
|
||||||
|
with `EscrowState: "pending"` (the `cur == nil` fallthrough, `offbox.go:482-494`). So `Pending()` is
|
||||||
|
true, the pending branch runs (`escrow_confirm.go:86-107`), the hub's hash (sealing the OLD password)
|
||||||
|
is compared against `LocalHash()` (the NEW minted one), they differ, and it logs:
|
||||||
|
|
||||||
|
```
|
||||||
|
[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password
|
||||||
|
(hub hash %.12s… != local %.12s…) — run the escrow ceremony … staying pending
|
||||||
|
```
|
||||||
|
|
||||||
|
**So: confirmed, it fires.** The detector is not absent from the mint path.
|
||||||
|
|
||||||
|
**Three limits, each of which matters more than the fact that it fires.**
|
||||||
|
|
||||||
|
1. **It is a controller-local log line.** No event, no e-mail, no hub-side signal. `StaleBlob()` — the
|
||||||
|
flag the web card renders — is set **only** by the escrowed branch (`reconcileEscrowed`,
|
||||||
|
`:130-165`); the pending branch sets no display flag. The operator learns nothing.
|
||||||
|
2. **Its prescription is the destructive act.** *"run the escrow ceremony"* → a new ceremony →
|
||||||
|
`SaveHostEscrow` overwrites `host_escrow.identity_blob` → `demoteCurrentEscrowTx` does not carry it
|
||||||
|
→ **the old repo password ceases to exist**. The detector correctly identifies the situation and
|
||||||
|
then tells the customer to close the door.
|
||||||
|
3. **It cannot distinguish the two situations that need different answers** — "you re-keyed
|
||||||
|
deliberately" and "your machine was rebuilt and your history is about to be orphaned" produce the
|
||||||
|
identical warning.
|
||||||
|
|
||||||
|
So: **less needs building than the prompt feared for the *detection*, and more needs building for the
|
||||||
|
*consequence*.** The comparison exists; what does not exist is a path from that comparison to either a
|
||||||
|
signal or a recovery.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Q5 — is the superseded-escrow retention real, and what does it retain?
|
||||||
|
|
||||||
|
### RULING: **the retention is real, it retains the wrong blob, and nothing can read it back.**
|
||||||
|
|
||||||
|
**What it retains — source and live, agreeing.** `demoteCurrentEscrowTx` (`store/store.go:2547-2556`):
|
||||||
|
|
||||||
|
```sql
|
||||||
|
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
|
||||||
|
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
|
||||||
|
FROM host_escrow WHERE host_id = ?
|
||||||
|
```
|
||||||
|
|
||||||
|
`identity_blob` is not in the column list. Live schema (I2, `PRAGMA table_info`):
|
||||||
|
|
||||||
|
| table | columns |
|
||||||
|
|---|---|
|
||||||
|
| `host_escrow` | `host_id, blob, key_fingerprint, posture, created_at, updated_at, **identity_blob**, directive_json, restic_pw_sha256, stale_at` |
|
||||||
|
| `host_escrow_superseded` | `id, host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at` |
|
||||||
|
|
||||||
|
**`blob` is the K-escrow — the PBS datastore encryption key**, wrapped by the PBS-native scrypt path.
|
||||||
|
**`identity_blob` is the age-wrapped `IdentityBundle`** — the one carrying `ResticRepoPassword`
|
||||||
|
(`felhom-agent/internal/escrow/identity.go:34-39`), plus `TunnelToken`, `PBSToken` and
|
||||||
|
`WGPrivateKey`. Live sizes confirm they are different objects: both hosts' current rows hold
|
||||||
|
`blob` = 383 B **and** `identity_blob` = 572 B; both superseded rows hold `blob` = 383 B and nothing
|
||||||
|
else.
|
||||||
|
|
||||||
|
**So, exactly what a customer with the matching OLD recovery code could recover from a retained blob:**
|
||||||
|
|
||||||
|
- ✅ the PBS datastore encryption key **K** — i.e. the Plane-2 whole-guest offsite ciphertext;
|
||||||
|
- ❌ **not** the restic repository password — the Tier-3 offsite app-data history;
|
||||||
|
- ❌ not the tunnel token, not the PBS access token, not the WireGuard private key.
|
||||||
|
|
||||||
|
**And there is no read path.** I4: `ListSupersededEscrow` (`store/store.go:2617`) has **zero
|
||||||
|
production callers** — only two tests. The only production surface is `CountSupersededEscrow`, used
|
||||||
|
for a number on the host page (`web/hosts.go:573`) and a log line (`api/handler.go:1151`). The schema
|
||||||
|
comment says so itself: *"The ACK and restore-serving read `host_escrow` (the CURRENT row) — never
|
||||||
|
this table."* Retrieval is `sqlite3` on the hub pod, by hand, by the operator.
|
||||||
|
|
||||||
|
**The claim this contradicts.** `00-capability-map.md` (escrow-ceremony row) records hub v0.60.0 as:
|
||||||
|
|
||||||
|
> *"a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS
|
||||||
|
> it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code
|
||||||
|
> (turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable')."*
|
||||||
|
|
||||||
|
The same claim appears in the schema comment (`store.go:370-375`) and, in Hungarian, on the customer's
|
||||||
|
own orphan card (`controller/internal/web/templates/backups_remote.html:66,69`):
|
||||||
|
|
||||||
|
> *„A meglévő mentések nem sérültek — a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók
|
||||||
|
> lehetnek."* · *„A régi előzmény félretéve marad (nem törlődik), és a hozzá tartozó helyreállítási
|
||||||
|
> kóddal később visszaállítható lehet."*
|
||||||
|
|
||||||
|
**For the offsite restic repository — the incident it names — that is false.** The retained blob does
|
||||||
|
not contain the key that opens it. This is the **ninth** entry in `CLAUDE.md`'s table of comments
|
||||||
|
asserting an invariant the code does not provide, and the first that is also **customer-facing copy**.
|
||||||
|
→ **R-198**, and the capability-map correction in §9.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Q6 — has any part of this ever been exercised?
|
||||||
|
|
||||||
|
**Method:** read every audit, validation doc, findings file and REPORT that mentions escrow consume,
|
||||||
|
identity-consume, re-enroll or inject-password; then check what the cited evidence actually covers.
|
||||||
|
|
||||||
|
### Proven
|
||||||
|
|
||||||
|
| what | evidence | scope of the proof |
|
||||||
|
|---|---|---|
|
||||||
|
| ceremony (create + seal + upload) | `00-capability-map.md` escrow row; `tests/VALIDATION-n100-rehearsal-2026-07-18.md` S6 — customer drove the wizard, R claimed one-shot 16:56:39, auto-confirmed 16:56:41 | the **create** side only |
|
||||||
|
| identity-bundle round-trip with a real R on a secret-less box | `tests/slice10d-identity-restore-spike-findings.md` §1, 2026-06-10 — wrap→unwrap byte-identical (`sha256` match); wrong-R **fails closed**, no plaintext emitted | **PRE-FORK-4.** The bundle then was `{tunnel_token, pbs_token}`. `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** (`felhom-agent/CHANGELOG.md:2344`) — a month later. **The proof does not cover the field this whole question is about** |
|
||||||
|
| `ResticRepoPassword` survives the wrap/unwrap | agent v0.77.0 CHANGELOG: *"`IdentityBundle` round-trip carries `ResticRepoPassword` byte-exact + not-in-blob + wrong-R fails closed"* | **unit test only** |
|
||||||
|
| `InjectOffboxPassword` places and refuses correctly | `controller/internal/backup/offbox_test.go:952-990`, incl. the companion showing a fresh mint without inject | **unit test only** |
|
||||||
|
|
||||||
|
### Never exercised — named in those words
|
||||||
|
|
||||||
|
- **A blob has never been served to a box.** Link 6 has no client (§3).
|
||||||
|
- **A fork-4 bundle has never been unsealed with a real R outside a unit test.** The only production
|
||||||
|
caller is a `--selftest` mode; the one live consume ever prepared (S5 Part 4-B, 2026-07-04) was
|
||||||
|
explicitly **deferred and never run** — its own resume note still reads *"DEFERRED 2026-07-04, not
|
||||||
|
run"* (project memory; `06-offsite-connectivity.md:327` marks the destructive drill operator-gated).
|
||||||
|
- **A recovered repo password has never been injected into a controller.**
|
||||||
|
- **An existing offsite repository has never been reopened with a recovered password.**
|
||||||
|
- **No restore of any kind has ever been performed from a recovered secret.**
|
||||||
|
|
||||||
|
`_recovery-inventory-2026-07-28.md` already said most of this and said it accurately — A.2.7 records
|
||||||
|
*"the **consume** side is proven at spike level … the destructive in-place drill is explicitly
|
||||||
|
operator-gated and unrun"*, and C.1 row 1 says *"escrow **creation** proven live; **consume never
|
||||||
|
exercised destructively**"*. **That document is right and this session confirms it.** What it did not
|
||||||
|
catch — because it was written from `host_escrow`, not from the supersession path — is Q5.
|
||||||
|
|
||||||
|
**Capability-map correction owed:** not for a "recovery proven" claim (no row makes one), but for the
|
||||||
|
**retention** claim quoted in §7. See §9.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Q7 — the operator's ruled shape, assessed row by row
|
||||||
|
|
||||||
|
> **Operator ruling, 2026-08-04, verbatim:** *If a node is a fresh install AND the hub has a recovery
|
||||||
|
> blob, then the controller should yell that recovery is available, and provide a form for the customer
|
||||||
|
> to enter the recovery key. After unlocking the blob, the controller should show what will be
|
||||||
|
> recovered before proceeding.*
|
||||||
|
|
||||||
|
| The ruling requires | Exists? | What it needs, and what it costs |
|
||||||
|
|---|---|---|
|
||||||
|
| the controller knows it is a **fresh install** | **YES, at the right line** | `os.Stat(m.offboxPwPath())` → `IsNotExist` is already the mint branch (`offbox.go:392`). It is trustworthy as *"this controller has no repo password"*, which is the only fact the decision needs; it does **not** by itself distinguish first-ever install from rebuild — that is what the next row supplies. Corroborators exist but are weaker: `Claimed` is set-only and was measured re-set by a human within minutes (§5); a fresh `encryption.key` is a stronger signal but is not currently read by anything. **Cost: zero — the branch already exists.** |
|
||||||
|
| the controller knows **the hub has a blob** | **YES on the wire, NO in the code** | `EscrowStatus{IdentityBlobPresent, ResticPwSHA256, CreatedAt}` arrives on every report ACK and is discarded when no offbox target exists (§5). **Needs:** persist it (the `ClaimSync` set-only-by-generation pattern is the precedent, `claim_sync.go:39-53`). **Cost: small — controller only, one settings field, one reconciler, no hub change.** |
|
||||||
|
| it can **yell** rather than silently mint | **NO** | The refusal belongs **inside `WriteOffboxSecrets`, at the mint branch** — the one place that decides. Returning a new sentinel (`ErrOffboxRecoveryAvailable`, sibling of `ErrOffboxOrphaned`) propagates through `ApplyOffsiteTarget` → `ConfigureOffbox` → the apply-bridge, which already treats a failed configure as "persist nothing, retry next cycle" (`offsiteapply.go:5-6, 256`) — so a refusal is **fail-safe by construction**, not a new failure mode. Meanwhile the customer sees a card on `/backups/remote` (the orphan card at `backups_remote.html:60-70` is the exact precedent) and offsite runs stay blocked, which they already are (`OffboxRunnable`). **Cost: small-medium — controller only.** **Design hazard to name:** a first-ever install of a *re-used customer record* would also trip this; the refusal must be dismissible ("start fresh anyway"), and dismissal must be the act that mints. |
|
||||||
|
| a **form takes R** | **NO — and R is not a password** | The controller UI **emits** R (`backups_escrow.html:235-236`, `/backup/escrow/…/claim` returns it once) and has **never accepted** one. The only thing resembling an entry point is `POST /backup/offbox/inject-password`, which takes a **64-hex repo password** (`offboxRepoPwPattern`, `offbox.go:543`) — a different secret at a different layer — and **has no form in any template** (§3 link 9). **Cost: small for the form; the hard part is the next row.** |
|
||||||
|
| the blob is **unsealed** with R | **NO — and there is a real boundary** | Unsealing is `age -d`. `ageBinary = "/usr/bin/age"` (`identity.go:24`) and the code live in the **agent**. The **controller image does not ship `age`** (`controller/Dockerfile:56-70` — restic, openssh-client, sshpass, sqlite3, docker-cli; no age) and must not: the agent is the privileged tier by design. **The crossing already has a transport and a precedent:** the pinned local API carries `POST /escrow/ceremony` + `GET /escrow/ceremony/status` + `POST /escrow/ceremony/claim` (`localapi/server.go:524-527`) — i.e. the agent **already** handles R server-side for the create direction and hands the result back exactly once. **Needs: one new agent local-API endpoint** (`POST /escrow/unseal`: blob + R in, the single `restic_repo_password` field out, R never logged, R never persisted) — the exact mirror of the claim endpoint. **Plus** a way for the blob to reach the agent: the agent's hub client has no escrow **GET** (only the `PUT` at `main.go:2937`), and `dr.go`'s re-enroll is the wrong tool (it rotates the host API key and needs operator-armed recovery mode). **Needs: a second small hub endpoint** — `GET /hosts/<id>/escrow`, per-host-key authed, serving opaque bytes to the box that owns them. Zero-knowledge is untouched: the hub still never decrypts, and the bytes are useless without R. **Cost: medium — three repos, two new endpoints, both narrow.** |
|
||||||
|
| **"show what will be recovered" before proceeding** | **the primitive EXISTS; the flow does not** | `restic snapshots --json` + `restic stats --mode raw-data --json` are already how the box counts snapshots and sizes the repo (`offboxRecordStats`, `offbox.go:1234-1265`). Both are **read-only against the repository** — nothing is written, no lock is taken, no `init`. So a preview can show: **snapshot count, per-snapshot time, host, tags (= app names), paths, and the deduplicated repo size** — all before committing. **Committing requires only that the recovered password be written to `offboxPwPath()`**, and even that is reversible in principle. **The clean shape:** unseal → hold the recovered password **in memory** → run the two read-only probes against the existing repo → render → and only on the customer's confirmation call `InjectOffboxPassword`. **Cost: small — one handler, one template, no new restic capability.** **Caveat, measured not assumed:** `offboxRecordStats` today also persists `RepoSizeBytes` into settings as a side effect; a preview path must not reuse it verbatim. |
|
||||||
|
| **proceeding** actually recovers | **NO end-to-end** | `InjectOffboxPassword` → next `WriteOffboxSecrets` keeps the file → `ensureOffboxRepo` probes and clears `RepoState`. Every piece is implemented; **none has ever run against a recovered password** (§8). **Cost: zero new code, but it is precisely what the drill in §10 has to prove.** |
|
||||||
|
|
||||||
|
### Things the ruling implies that do not exist, priced
|
||||||
|
|
||||||
|
| missing piece | cost | note |
|
||||||
|
|---|---|---|
|
||||||
|
| persisted ACK escrow status + fail-closed mint refusal | **S** | controller only; the `ClaimSync` pattern applies directly |
|
||||||
|
| the "recovery is available" card + dismissal | **S** | one template, one flag; the orphan card is the template precedent |
|
||||||
|
| a customer-facing **recovery-code** form (distinct from the password form) | **S** | the password form's handler exists but has no UI; both need building |
|
||||||
|
| agent local-API `POST /escrow/unseal` | **M** | new privileged surface; mirror of `/escrow/ceremony/claim` |
|
||||||
|
| hub `GET /hosts/<id>/escrow` (opaque, per-host key) | **S** | new read surface on the hub; zero-knowledge preserved |
|
||||||
|
| preview handler + template driven by read-only restic probes | **S** | primitive already exists |
|
||||||
|
| **carrying `identity_blob` into `host_escrow_superseded`** | **S** | one column, one `SELECT` list — **and without it the whole feature protects only boxes that have not yet re-escrowed** (→ R-198). This is a prerequisite, not a nice-to-have |
|
||||||
|
|
||||||
|
### The security question the ruling raises — stated, not answered
|
||||||
|
|
||||||
|
**What gates the surface today.** The controller dashboard is behind `RequireAuth`
|
||||||
|
(`internal/web/auth.go:52`): a bcrypt password (`effectivePasswordHash()` — customer-set in
|
||||||
|
`settings.json`, else operator-provisioned in `controller.yaml`), a 7-day session cookie, 5 failed
|
||||||
|
logins per minute then lockout, and `CsrfProtect` on every mutating route. The claim gate serves only
|
||||||
|
the claim page on an unclaimed box. So a recovery form would sit **behind the customer's dashboard
|
||||||
|
password**, on a controller reachable at whatever address that box's dashboard is reachable at.
|
||||||
|
|
||||||
|
**What a preview would expose to someone who guesses or steals R.** Not file contents — but the
|
||||||
|
snapshot list is not nothing: **backup dates and cadence, the app names (restic tags), the paths, and
|
||||||
|
the repository size.** That is a map of the household's digital life, and it is precisely the metadata
|
||||||
|
the zero-knowledge posture otherwise keeps out of everyone's hands including the operator's.
|
||||||
|
|
||||||
|
**Three specifics the operator should weigh, without this session choosing:**
|
||||||
|
|
||||||
|
1. **Guessing R is not the threat; holding the dashboard password is.** R is ten EFF words
|
||||||
|
(~129 bits, `00-capability-map.md` agent v0.93.0 note) and `age`'s scrypt is deliberately slow —
|
||||||
|
online guessing is not a route. But **anyone with the dashboard password can already restore live
|
||||||
|
offsite data** (`_recovery-inventory-2026-07-28.md` A.1.3–A.1.6). The marginal new exposure is
|
||||||
|
confined to the *orphaned/previous* repository, which today nobody can read at all.
|
||||||
|
2. **The form is an oracle.** A correct/incorrect R answer distinguishable in the UI turns the
|
||||||
|
dashboard into a verification service for a stolen recovery code. `UnwrapIdentity` already
|
||||||
|
fails closed with a single generic error (`identity.go:157`) — the UI must not do better than that,
|
||||||
|
and should rate-limit like the login path does.
|
||||||
|
3. **R will pass through the agent.** Whatever the form does, R crosses the local API to a
|
||||||
|
root-adjacent daemon. The ceremony already does this in the other direction, so the precedent is
|
||||||
|
set — but it means R is momentarily present in agent process memory on the Proxmox host, which is
|
||||||
|
the same host D6 says the operator has root on. **This is the same trade as spike option (c), in a
|
||||||
|
smaller and time-bounded form**, and it should be decided alongside it rather than separately.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Q8 — the drill design
|
||||||
|
|
||||||
|
**Why a deliberate cycle.** No recovery code exists for either demo box, and — per §7 — even one would
|
||||||
|
no longer open the old repositories. The proof therefore requires creating R, keeping it, and walking
|
||||||
|
the full circle. **This section is the design; the drill is not run in this session.**
|
||||||
|
|
||||||
|
### Target and preconditions
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| **Box** | **`demo-hp`** (HP t740, node `felhom-host`, host `demo-hp-bb76ea`). Tier 0 — disposable, and the designated drill host by the 2026-07-25 operator ruling. **Not demo-felhom** — it is the N100 that also carries the ISO/rehearsal lineage; leave one demo box in a known-good state throughout |
|
||||||
|
| **Guest** | the customer guest on that host (rebuild target) |
|
||||||
|
| **Data** | one deliberately identifiable app + a sentinel file whose content is a known string and whose sha256 is recorded before the wipe |
|
||||||
|
| **Prereq** | the box is healthy, offsite target configured and `escrow_state: escrowed`, and its **current** `host_escrow.identity_blob` seals the password of the repo that will be tested |
|
||||||
|
| **Duration** | ~3–4 h wall clock, of which ~45 min is operator-attended (steps 2, 5, 8, 9) |
|
||||||
|
| **Blocking dependency** | **steps 8–9 cannot be executed as designed today** — links 6, 8 and 9 of the chain do not exist as automation. The drill as written uses the manual substitutes (§4a) and **that is the point**: it measures the *existing* chain honestly before anything is built |
|
||||||
|
|
||||||
|
### The cycle
|
||||||
|
|
||||||
|
| # | Step | Who | Observable that must be recorded |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | Record the starting state: `host_escrow.restic_pw_sha256`, `identity_blob` length, `$.offsite.snapshot_count`, `repo_size_bytes` | CC | a hash and two counters, from the hub DB |
|
||||||
|
| 2 | **Run the escrow ceremony through the customer wizard and KEEP R** | **HUMAN** | R written to the operator's password manager under a named entry; **its storage location recorded in the drill report — the code itself never** |
|
||||||
|
| 3 | Deploy an app, write the sentinel file, record its sha256 | CC | sha256 + path |
|
||||||
|
| 4 | Run an off-site backup through the real UI endpoint; wait for completion | CC | `$.offsite.last_status = ok`, `snapshot_count` increased by ≥1, `last_success` stamped |
|
||||||
|
| 5 | **Verify the snapshot from outside**: `restic snapshots --json` against the repo | CC | snapshot id + tag + timestamp — the pre-wipe truth to compare against |
|
||||||
|
| 6 | **Wipe**: destroy the guest and reinstall it (the same procedure that produced the 2026-08-03 rebuild) | **HUMAN** decision, CC executes | new `controller_started`; `claimed` drops to 0; a new `encryption.key` mtime |
|
||||||
|
| 7 | Let the box come back and reach the mint. **Do NOT run a new ceremony** | CC | the controller log carries the `escrow-confirm` mismatch WARN (§6); `escrow_state: pending`; `host_escrow.identity_blob` **unchanged** (this is the assertion that keeps recovery possible) |
|
||||||
|
| 8 | **Recover**: export `identity_blob` from the hub, unseal with the kept R, extract `restic_repo_password` | **HUMAN** (R is theirs) | `identity-consume` exits 0; the bundle JSON contains a non-empty `restic_repo_password`; **the value is never printed or logged** |
|
||||||
|
| 9 | **Inject** the recovered password (`POST /backup/offbox/inject-password`, by hand — there is no form), re-apply, run an off-site backup | **HUMAN**-assisted | `restic cat config` succeeds; **`snapshot_count` ≥ the step-5 count + 1**, not 1 |
|
||||||
|
| 10 | **Restore the sentinel** through the customer restore flow and compare sha256 | CC | **byte-identical sha256 to step 3** |
|
||||||
|
| 11 | Record everything, tear down nothing that would hide the evidence | CC | drill report |
|
||||||
|
|
||||||
|
### The single observation that makes this a FAILURE rather than a pass
|
||||||
|
|
||||||
|
> **Step 10's sha256 does not match step 3's — or step 9 reports a snapshot count of 1.**
|
||||||
|
|
||||||
|
A snapshot count of 1 means a **new, empty history** was started over the old one: the repository did
|
||||||
|
not reopen, and every earlier snapshot is orphaned. **Explicitly: "the repository opened" is not the
|
||||||
|
pass condition.** `restic cat config` succeeding proves the password is right; it does not prove the
|
||||||
|
customer's data came back. Only the sha256 comparison does.
|
||||||
|
|
||||||
|
### Secondary failures worth recording separately (each is a finding, not a pass/fail)
|
||||||
|
|
||||||
|
- Step 7 finds `identity_blob` **changed** — something re-escrowed automatically → the recovery window
|
||||||
|
closes without a human act (would be new and serious).
|
||||||
|
- Step 8's bundle has an **empty** `restic_repo_password` — a pre-fork-4 or hash-less blob shape.
|
||||||
|
- Step 9's `ensureOffboxRepo` classifies `orphaned` despite the correct password → the classifier or
|
||||||
|
the transport, not the key.
|
||||||
|
|
||||||
|
### What could make it fail for uninteresting reasons
|
||||||
|
|
||||||
|
The `age` binary absent on the machine running step 8; the hub DB export producing a truncated blob
|
||||||
|
(use `writefile`, verify the byte length against `length(identity_blob)` = 572); the CSRF token in
|
||||||
|
step 9 (`felhom_session` is dropped by curl's cookie jar — project memory
|
||||||
|
`controller-api-headless-access`); and the R-71a settle gate deferring the apply in step 9 (wait for
|
||||||
|
its GO line rather than retrying).
|
||||||
|
|
||||||
|
### Rollback, and whether the loss is acceptable
|
||||||
|
|
||||||
|
**If the recovery does not work, what has been lost is:** demo-hp's guest, its app data since the last
|
||||||
|
whole-guest archive, and its off-site history. **On that box that is acceptable** — it is Tier 0,
|
||||||
|
explicitly disposable, and it is the box the target-selection runbook names for drills. **The
|
||||||
|
rollback** is a whole-guest restore from the local/PBS archive taken immediately before step 6, which
|
||||||
|
is a proven path on this fleet (restore-test passed on demo-hp's host tier in 109.3 s on 2026-08-04).
|
||||||
|
**Take that archive deliberately as part of step 6 and verify it exists before destroying anything** —
|
||||||
|
do not rely on the nightly.
|
||||||
|
|
||||||
|
**Not acceptable, and therefore not proposed:** running this on demo-felhom (leaves no healthy demo
|
||||||
|
box), on DooPlex (Tier 2, it *is* the recovery chain), or on any real customer's box.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Every query used
|
||||||
|
|
||||||
|
| # | Source | Query / read | Value read |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | git | `git -C <repo> rev-parse HEAD`; `git status --porcelain` | the three commits in §1; all empty |
|
||||||
|
| 2 | hub DB | `PRAGMA integrity_check` | `ok` |
|
||||||
|
| 3 | hub DB | `SELECT MAX(received_at) FROM host_reports; SELECT datetime('now')` | `2026-08-04 10:00:40` / `2026-08-04 10:03:48` |
|
||||||
|
| 4 | hub DB | `SELECT host_id, length(blob), length(identity_blob), substr(restic_pw_sha256,1,12), created_at, updated_at FROM host_escrow` | `demo-felhom-8363b5, 383, 572, c60c8bc737a6, 2026-08-04T07:20:07Z, 2026-08-04 07:20:08` · `demo-hp-bb76ea, 383, 572, 8a9e33aa4da6, 2026-08-04T07:15:36Z, 2026-08-04 07:15:36` |
|
||||||
|
| 5 | hub DB | `SELECT id, host_id, length(blob), substr(restic_pw_sha256,1,12), created_at, superseded_at FROM host_escrow_superseded` | `3, demo-hp-bb76ea, 383, 8e03eddf9ff7, 2026-07-23T10:01:17Z, 2026-08-04 07:15:36` · `4, demo-felhom-8363b5, 383, 48741892f0ef, 2026-07-21T08:38:33Z, 2026-08-04 07:20:08` |
|
||||||
|
| 6 | hub DB | `PRAGMA table_info(host_escrow)` / `(host_escrow_superseded)` | the two column lists in §7 — `identity_blob` present in the first, **absent from the second** |
|
||||||
|
| 7 | hub DB | latest report per customer: `json_extract(report_json,'$.claimed' / '$.offsite.escrow_state' / '.snapshot_count' / '.repo_state' / '.last_status' / '.last_run')` | demo-felhom `2026-08-04 09:49:42`: `1, escrowed, 0, NULL, NULL, NULL` · demo-hp `2026-08-04 09:56:57`: `1, escrowed, 0, NULL, NULL, NULL` |
|
||||||
|
| 8 | hub DB | `claimed` grouped over reports since `2026-08-03 05:00` | demo-hp: `1` ×124, **`0` ×5 (06:12:19→06:21:03)** · demo-felhom: `1` ×120, **`0` ×4 (07:17:54→07:19:46)** |
|
||||||
|
| 9 | hub DB | `SELECT customer_id, event_type, severity, created_at FROM events WHERE created_at > '2026-08-03 00:00'` | no `escrow_stale`, no `offsite_reissued`, no `offbox_repo_orphaned`, no claim event in the window; `controller_started` ×4 demo-hp / ×4 demo-felhom around the rebuilds |
|
||||||
|
| 10 | guest 9201 (I3) | `ls -la --time-style=full-iso …/felhom-controller-data/_data/data{,/offbox}` | the mtime table in §5 |
|
||||||
|
| 11 | source (I4) | `grep -rn "InjectOffboxPassword" --include=*.go .` | 1 definition, 1 handler caller, 5 test uses, 1 comment — **no production caller besides the handler** |
|
||||||
|
| 12 | source (I4) | `grep -rn "inject" internal/web/templates/` | one unrelated hit (an XSS comment) — **no form** |
|
||||||
|
| 13 | source (I4) | `grep -rni "reenroll\|re_enroll" --include=*.go .` in `felhom-agent` | **zero hits** |
|
||||||
|
| 14 | source (I4) | `grep -rn "re-enroll\|recovery-mode\|restore-directive" --include=*.sh --include=*.md --include=*.py` workspace-wide | only design docs + project memory; no script, no runbook step |
|
||||||
|
| 15 | source (I4) | `grep -rn "ListSupersededEscrow"` in `felhom.eu` | 2 test callers, 0 production |
|
||||||
|
| 16 | source (I4) | `grep -rn "\.Escrow\b" --include=*.go .` in `felhom-controller` | one hit: `cmd/controller/main.go:641` |
|
||||||
|
| 17 | source | `felhom-agent/CHANGELOG.md` | `v0.77.0 — fork-4: escrow the offsite restic repo password under R (2026-07-09)` |
|
||||||
|
| 18 | source | `controller/Dockerfile:36-70` | runtime package list — **no `age`** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. What could not be established
|
||||||
|
|
||||||
|
Named, not guessed.
|
||||||
|
|
||||||
|
1. **The 2026-08-05 ~02:15 UTC scheduled run's outcome.** Still unmeasured — both boxes became
|
||||||
|
runnable only on 2026-08-04 morning and neither has run since (query 7: `last_run` absent on both).
|
||||||
|
Yesterday's spike predicts `ErrOffboxOrphaned`; that prediction is **unchanged and still unproven**.
|
||||||
|
2. **Whether demo-hp's data dir was fresh.** Established for demo-felhom by direct mtime measurement
|
||||||
|
(I3). demo-hp was not measured — it has no baked SSH key and reaching it needs the break-glass
|
||||||
|
credential, which this read-only session did not use. The hub-side `claimed 1→0→1` signature is
|
||||||
|
identical on both boxes, which is suggestive, not decisive.
|
||||||
|
3. **Whether the orphaned ciphertext still exists at the provider.** Unchanged from yesterday: reading
|
||||||
|
the endpoint needs the SFTP credential, which was not touched.
|
||||||
|
4. **Whether any pre-2026-07-21 escrow generation ever existed for demo-felhom.** The superseded table
|
||||||
|
holds two rows in the entire database; earlier generations are not retained (and, per §7, would not
|
||||||
|
have carried the identity blob anyway).
|
||||||
|
5. **Whether `age` is present on any machine that would run step 8 of the drill.** Not checked — the
|
||||||
|
agent declares `/usr/bin/age` as a runtime dependency but this session did not verify it on either
|
||||||
|
demo host.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. Observations — noticed, not acted on
|
||||||
|
|
||||||
|
1. **A fresh `encryption.key` on every rebuild** (I3: 32 B, mtime `2026-08-03 07:17:26`) means every
|
||||||
|
pre-rebuild `app.yaml` `ENC:` secret on that box is undecryptable. This is exactly circular
|
||||||
|
dependency **C-1** in `_recovery-inventory-2026-07-28.md`, and it fired live on both demo boxes this
|
||||||
|
week without anyone naming it. It is a strictly larger blast radius than the offsite repo and it is
|
||||||
|
**not** covered by R-193. Worth its own item if the operator agrees it is not already implied by C-1.
|
||||||
|
2. **`Claimed` returning to `1` within minutes** of a rebuild, with no event of any kind recording it,
|
||||||
|
means "the customer claimed this box" has no audit trail. There is no `customer_claimed` event type.
|
||||||
|
3. **`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting** — harmless today,
|
||||||
|
a trap for anyone reusing it as a read-only preview primitive (§9).
|
||||||
|
4. **`handleReEnroll` returns the new API key in the response body** (`dr.go:148`) while also logging
|
||||||
|
the rotation. Not a defect (the box needs it), but it is a secret in an HTTP response on an endpoint
|
||||||
|
nothing currently calls — worth a second look before anything starts calling it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14. The operator's decisions — stated, and NOT answered here
|
||||||
|
|
||||||
|
1. **Run the drill (§10) or not?** It is the only thing that converts *"the pieces look right"* into
|
||||||
|
*"a customer got their data back"*. It costs demo-hp for an afternoon and, if it fails, that box's
|
||||||
|
history. **Recommendation: yes — but fix `identity_blob` retention (R-198) first**, because a drill
|
||||||
|
that walks the current chain will pass or fail on a chain that is missing a link everyone believed
|
||||||
|
was there.
|
||||||
|
2. **The orphaned snapshots at the endpoint** (40.9 MB + 1.14 GB, presumed present, §12.3). They are
|
||||||
|
**unrecoverable by two independent mechanisms** now: no recovery code was kept, *and* the sealed
|
||||||
|
passwords were destroyed by the 2026-08-04 re-ceremonies. They consume the 50 GB soft quota
|
||||||
|
indefinitely; nothing prunes a moved-aside restic repo. **Delete, or leave?** This session did not
|
||||||
|
act on it and states it as the operator's.
|
||||||
|
3. **The unchanged R-193 question** — agent-retained repo password (spike option c) versus accepting
|
||||||
|
that every rebuild starts a new repository. **This session adds one input to that decision:** the
|
||||||
|
"accept it, provided the system says so loudly" branch currently rests on a retention feature that
|
||||||
|
does not retain the relevant secret. Under option (d)-alone as it stands today, *"the old history
|
||||||
|
stays recoverable with your recovery code"* is not true. Either R-198 is fixed, or that branch has
|
||||||
|
to be re-described honestly to the customer.
|
||||||
|
</content>
|
||||||
|
</invoke>
|
||||||
@@ -0,0 +1,540 @@
|
|||||||
|
# SPIKE — what is actually one-shot, and what a guest rebuild really costs
|
||||||
|
|
||||||
|
**Date:** 2026-08-04 · **Items:** R-193 (rebuild drops offsite), R-192 (the alert says the opposite of
|
||||||
|
what it measured) · **Class:** spike — **no production code shipped for R-193 or R-192**
|
||||||
|
**Baselines read:** `felhom.eu` @ `f456835bbcc4` (hub v0.91.1) · `felhom-controller` @ `0887fd67` ·
|
||||||
|
`felhom-agent` @ `856a127`
|
||||||
|
|
||||||
|
> **This document ends in ranked options and a STOP.** It establishes facts; it does not choose.
|
||||||
|
> The operator's question is stated, unanswered, at the end.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
> ### ANNOTATION 2026-08-04 (later the same day) — read this before acting on §Q8
|
||||||
|
>
|
||||||
|
> A follow-up recon traced the escrow→recovery chain link by link
|
||||||
|
> (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **Two corrections to this document, and they pull
|
||||||
|
> in opposite directions.**
|
||||||
|
>
|
||||||
|
> **(1) Q8 candidate (b) — "not implementable as specified" — is TOO STRONG and is overturned in
|
||||||
|
> part.** Zero-knowledge means the *hub* cannot open the blob; the *customer* can, with R, which is
|
||||||
|
> what the ceremony exists for. The mechanism to hand a rebuilt box its old repo password back —
|
||||||
|
> customer-present, not unattended — is a real design, and the operator has since ruled on its shape.
|
||||||
|
> What (b) got right is narrower than what it said: **an UNATTENDED rebuild cannot recover from a
|
||||||
|
> zero-knowledge escrow.** A customer-present one can. The recon prices that shape row by row.
|
||||||
|
>
|
||||||
|
> **(2) A LOAD-BEARING FACT THIS SPIKE DID NOT CHECK, and it makes (d)-alone weaker than ranked
|
||||||
|
> here.** `host_escrow_superseded` **has no `identity_blob` column**, and `demoteCurrentEscrowTx`
|
||||||
|
> (`hub/internal/store/store.go:2547-2556`) copies only the K-escrow `blob`. So the retained
|
||||||
|
> "superseded" blob holds the **PBS datastore key**, not the **restic repository password**. The
|
||||||
|
> retention feature does not retain the secret it was built to retain, and the **next escrow ceremony
|
||||||
|
> — which the system tells the customer to run whenever a rebuild leaves the escrow `pending` — is
|
||||||
|
> what destroys the last copy.** Both demo boxes crossed that line on 2026-08-04 at 07:15:36 and
|
||||||
|
> 07:20:08. → **R-198**.
|
||||||
|
>
|
||||||
|
> **Consequence for this document's recommendation.** "Ship (d), then decide (c)" still holds, but
|
||||||
|
> (d)'s implicit promise — *the old history stays recoverable with the recovery code* — is **false as
|
||||||
|
> the code stands**, and is repeated verbatim to the customer in Hungarian on the orphan card. (d) is
|
||||||
|
> not complete without R-198.
|
||||||
|
>
|
||||||
|
> **Unchanged and still correct:** Q1, Q2, Q4 (R-196), Q5, Q6, Q7 (R-192) and the Q8 (a)/(c)/(d)
|
||||||
|
> analyses. **Q3 remains UNMEASURED** — neither box has run since; the 2026-08-05 ~02:15 UTC run is
|
||||||
|
> still the decisive one.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. Why this exists
|
||||||
|
|
||||||
|
Two candidate fixes were named when R-193 was filed, and neither could be weighed, because two prior
|
||||||
|
session reports described the same mechanism in contradictory terms:
|
||||||
|
|
||||||
|
- R-39's record: Re-issue *"rotates the restic password and makes the escrow stale"*.
|
||||||
|
- R-193's own resolution note: the one-time password is *"only the transport credential used once to
|
||||||
|
install the box's own SSH key"*.
|
||||||
|
|
||||||
|
Those two worlds imply very different fates for the fifteen existing snapshots. This spike settles it
|
||||||
|
**from source first, live state second**, so a measurement can contradict the code rather than be
|
||||||
|
interpreted by it.
|
||||||
|
|
||||||
|
**The headline, before the detail:** *both prior claims are wrong in the way that matters.* Re-issue
|
||||||
|
does **not** touch the restic repository password — but a **guest rebuild does**, silently, on every
|
||||||
|
box, with nothing marking the escrow stale and nothing telling anyone. Both demo boxes lost repository
|
||||||
|
continuity on 2026-08-03/04. The one that "recovered by luck" lost it too.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. The instruments, and what each one can and cannot answer
|
||||||
|
|
||||||
|
Stated up front so a wrong instrument is visible here rather than inside a conclusion (§8 rule 2).
|
||||||
|
|
||||||
|
| # | Instrument | Exact query / read | Known limit |
|
||||||
|
|---|---|---|---|
|
||||||
|
| I1 | hub SQLite snapshot | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale at copy time (mtime 08:34 vs wal 10:45 CEST). Copying only the main file would have produced confident answers from a stale file — the exact R-3 shape. |
|
||||||
|
| I1-check | freshness proof | `PRAGMA integrity_check` → `ok`; `MAX(host_reports.received_at)` = `2026-08-04 08:45:38`, `datetime('now')` = `08:49:46` | **Positive observable**: newest row 4 min old. Not "the query returned no error". |
|
||||||
|
| I2 | `host_escrow.restic_pw_sha256` + `host_escrow_superseded.restic_pw_sha256` | see §2 | sha256 of a 256-bit random secret — non-reversible, safe to compare. Answers *"did the repo password change"* **without touching a box or a credential**. |
|
||||||
|
| I3 | `reports.report_json → $.offsite` | grouped distinct-object query with first/last seen | The report echoes the box's own target state; absence of the key = no target configured. |
|
||||||
|
| I4 | `one_time_secrets` | `SELECT customer_id, created_at, consumed_at` | **`customer_id` is the PRIMARY KEY** — one row per customer, last-write-wins. It **cannot** answer "how many credentials were delivered"; a Re-issue destroys the prior row. demo-hp's 2026-07-23 consume is no longer in this table. |
|
||||||
|
| I5 | source | `hub/internal/offsite/*`, `hub/internal/api/*`, `controller/internal/{backup,offsiteapply}`, `agent/internal/escrow` | — |
|
||||||
|
|
||||||
|
**Not used, deliberately:** no `ReissueCredentials`, no ceremony, no config change, no deletion, no
|
||||||
|
write to any box. Nothing on either box or on the storage endpoint was altered by this session.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q2 — Does a rebuilt controller reuse the escrowed repository password, or mint a new one?
|
||||||
|
|
||||||
|
### **RULING: it MINTS A NEW ONE. The escrowed password is never consulted on any automatic path.**
|
||||||
|
|
||||||
|
**Method — source.** The apply-bridge's terminal step is
|
||||||
|
`Enabler.ConfigureOffbox(...)` (`controller/internal/offsiteapply/offsiteapply.go:221,256`), which
|
||||||
|
reaches `Manager.ApplyOffsiteTarget` → `WriteOffboxSecrets`
|
||||||
|
(`controller/internal/backup/offbox.go:370`). Quoted verbatim, `offbox.go:392-401`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Auto-generate the repo password once (0600), never log it.
|
||||||
|
if _, err := os.Stat(m.offboxPwPath()); os.IsNotExist(err) {
|
||||||
|
pw, gerr := generateOffboxPassword()
|
||||||
|
...
|
||||||
|
if werr := os.WriteFile(m.offboxPwPath(), []byte(pw), 0o600); werr != nil {
|
||||||
|
```
|
||||||
|
|
||||||
|
`offboxPwPath()` is `<DataDir>/offbox/repo_password` (`offbox.go:363`). A rebuilt guest has a fresh
|
||||||
|
data dir, so the file is absent, so **a fresh 256-bit password is minted**.
|
||||||
|
|
||||||
|
**The only path that recovers the old one is `InjectOffboxPassword`** (`offbox.go:541`), whose sole
|
||||||
|
caller in the entire repo is `offboxInjectPasswordHandler`
|
||||||
|
(`controller/internal/web/offbox_handlers.go:189`) — a **web form where a human pastes the recovered
|
||||||
|
password**. Verified by census: `grep -rn "InjectOffboxPassword" --include=*.go .` → three hits, one
|
||||||
|
definition, one caller, one comment. **The apply-bridge never calls it.**
|
||||||
|
|
||||||
|
**Method — live, and it is decisive without touching a box (I2).** The hub already stores the sha256
|
||||||
|
of the escrowed repo password, so the question is answerable as a hash comparison:
|
||||||
|
|
||||||
|
| host | superseded (pre-rebuild) | current (post-rebuild) | verdict |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `demo-hp-bb76ea` | `8e03eddf9ff7…` created `2026-07-23T10:01:17Z` | `8a9e33aa4da6…` created `2026-08-04T07:15:36Z` | **CHANGED** |
|
||||||
|
| `demo-felhom-8363b5` | `48741892f0ef…` created `2026-07-21T08:38:33Z` | `c60c8bc737a6…` created `2026-08-04T07:20:07Z` | **CHANGED** |
|
||||||
|
|
||||||
|
**Both boxes minted a new repository password.** demo-hp's pre-rebuild escrow (`8e03…`, sealed eight
|
||||||
|
minutes after its 2026-07-23 09:53:41 apply) is the key to its 15 snapshots. That key now exists
|
||||||
|
**only inside a superseded, R-wrapped escrow blob** that the hub cannot open.
|
||||||
|
|
||||||
|
### **This is bigger than R-193 as filed, and it hits the box R-193 called lucky.**
|
||||||
|
|
||||||
|
R-193 records demo-felhom as having *"survived the SAME rebuild by luck"* — 76 seconds of downtime.
|
||||||
|
Measured (I3), demo-felhom's pre-rebuild offsite object was:
|
||||||
|
|
||||||
|
```
|
||||||
|
last_run 2026-08-03T02:17:44Z · last_status ok · snapshot_count 36 · repo_size_bytes 1 136 685 919
|
||||||
|
```
|
||||||
|
|
||||||
|
**36 snapshots, 1.14 GB.** Since 2026-08-03 07:19:10 UTC it has reported `snapshot_count: 0,
|
||||||
|
repo_size_bytes: 0` in every one of 109 reports, and its repo password hash changed.
|
||||||
|
|
||||||
|
> **The luck recovered DELIVERY. It did not recover the REPOSITORY.** The staged secret restored the
|
||||||
|
> transport in 76 seconds and the box then minted a brand-new repo password anyway. The contrast
|
||||||
|
> R-193 draws between the two boxes is real for the *credential*, and **false for the data**: both
|
||||||
|
> boxes lost repository continuity, one loudly and one silently. The silent one is worse.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q3 — What did the 04:15 run report?
|
||||||
|
|
||||||
|
### **RULING: UNMEASURED — the decisive run has not happened yet. And the binary the question offers is the wrong one; source says the answer will be NEITHER 15 nor 1.**
|
||||||
|
|
||||||
|
**Why it has not happened.** The scheduled off-box run fires at ~02:15 UTC (04:15 CEST). Measured
|
||||||
|
from I3 — `last_run` values `2026-08-02T02:15:52Z`, `2026-08-03T02:16:39Z` on demo-hp;
|
||||||
|
`2026-08-01T02:17:54Z`, `2026-08-02T02:17:01Z`, `2026-08-03T02:17:44Z` on demo-felhom. **Daily, both
|
||||||
|
boxes.** Then:
|
||||||
|
|
||||||
|
| box | why 2026-08-04 02:15 UTC produced nothing |
|
||||||
|
|---|---|
|
||||||
|
| demo-hp | no offsite target existed at all — the `offsite` key was **ABSENT** from every report between `2026-08-03 06:12:19` and `2026-08-04 07:12:01` UTC (111 reports). The re-issue landed at 07:15:47, five hours after the window. |
|
||||||
|
| demo-felhom | the target existed but `escrow_state: pending` from `2026-08-03 07:19:10` to `2026-08-04 07:20:13`. `OffboxRunnable` requires configured **AND** escrowed (`offbox.go:564-569`), so the fork-4 gate blocked the run. The ceremony landed at 07:20:28. |
|
||||||
|
|
||||||
|
Both boxes became runnable only this morning. **The decisive run is 2026-08-05 ~02:15 UTC on both.**
|
||||||
|
Reported as unmeasured rather than inferred.
|
||||||
|
|
||||||
|
### What source says will happen — and it is a third outcome
|
||||||
|
|
||||||
|
Same sub-account, same repo path, new password. Measured coordinates (I3, `$.dr_recipe`):
|
||||||
|
`u629488-sub3.your-storagebox.de:/home/felhom-repo` — **unchanged** across the whole incident, and
|
||||||
|
`repoPath` is a compile-time constant (`hub/internal/offsite/offsite.go:85`). So the next run opens an
|
||||||
|
**existing repo with the wrong key**, which the codebase already has a name for
|
||||||
|
(`controller/internal/backup/offbox.go:67-93`):
|
||||||
|
|
||||||
|
```go
|
||||||
|
// ErrOffboxOrphaned is the sentinel returned when the offsite repo exists but is keyed under a
|
||||||
|
// passphrase this controller no longer has (the reinstall shape) ...
|
||||||
|
case strings.Contains(s, "wrong password or no key found"):
|
||||||
|
return "orphaned"
|
||||||
|
```
|
||||||
|
|
||||||
|
and `ensureOffboxRepo` (`offbox.go:663-679`) branches on claim state:
|
||||||
|
|
||||||
|
```go
|
||||||
|
if !m.settings.GetClaimed() { // UNCLAIMED → auto move-aside + re-init
|
||||||
|
...
|
||||||
|
m.markOrphaned()
|
||||||
|
return ErrOffboxOrphaned // CLAIMED → skip the run, show the orphan card
|
||||||
|
```
|
||||||
|
|
||||||
|
Both boxes report `claimed: 1` (I3, `$.claimed`, latest report each). **So the predicted outcome is:
|
||||||
|
the run REFUSES with `ErrOffboxOrphaned`, produces no snapshot at all, and waits for the customer to
|
||||||
|
confirm a reset.** Not 15 (reattached) and not 1 (silently fresh) — a **third** outcome the question's
|
||||||
|
framing did not contain. That is the good news half: the system is not going to quietly start a new
|
||||||
|
history over the old one. It will stop and say so.
|
||||||
|
|
||||||
|
**Consequence if the customer confirms the reset:** `resetOrphanedRepo` (`offbox.go:270-314`) moves the
|
||||||
|
old repo aside and re-inits — *"move-aside, not deleted"*. The 40.9 MB + 1.14 GB of old ciphertext
|
||||||
|
survives, unreadable without the superseded escrow, **and keeps consuming the 50 GB soft quota
|
||||||
|
indefinitely.** Nothing prunes a moved-aside restic repo (S-24 covers ep0's PBS namespaces, a
|
||||||
|
different tier).
|
||||||
|
|
||||||
|
**To measure it tomorrow:** re-read `$.offsite.last_status` / `last_error` on both boxes after
|
||||||
|
02:20 UTC, and `$.offsite.repo_state` for the orphan flag. **Predicted `ErrOffboxOrphaned`, not a
|
||||||
|
snapshot count.** Whoever picks this up should record which of the three actually occurred — a
|
||||||
|
prediction from source is not a measurement.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q1 — What is one-shot, and what is not?
|
||||||
|
|
||||||
|
Three secrets, and they have almost nothing in common.
|
||||||
|
|
||||||
|
| | **① storage-provider password** | **② the box's SFTP key** | **③ restic repository password** |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **What it is** | the Hetzner sub-account / box password | ed25519 keypair for `sftp` transport | the **data** key — restic's repo encryption |
|
||||||
|
| **Generated by** | **hub** — `genPassword()` (`offsite.go:481`) | **controller** — `KeyGen.Generate()` (`offsiteapply.go:234`) | **controller** — `generateOffboxPassword()`, 32 random bytes hex (`offbox.go:406`) |
|
||||||
|
| **Stored where** | `one_time_secrets` (hub, plaintext, **one row per customer**) | `<DataDir>/offbox/ssh_key` 0600 (box only) | `<DataDir>/offbox/repo_password` 0600 (box) + inside the R-wrapped escrow blob (hub, opaque) |
|
||||||
|
| **Delivered how** | `GET …/offsite/consume-password`, **served exactly once** then marked consumed (`api/offsite.go:9-31`) | never delivered — installed onto the provider by ssh-copy-id using ① | never delivered anywhere; it never leaves the box except into the escrow ceremony |
|
||||||
|
| **Re-issuable?** | **YES** — `ReissueCredentials` resets it at the provider, any time, operator-initiated | **YES** — regenerated on every full apply; `dr_recipe.go:45` states it plainly: *"the SFTP access key is regenerated at DR"* | **NO automatic path.** Recoverable only by unsealing the escrow with the customer's recovery code and pasting it into `offboxInjectPasswordHandler` |
|
||||||
|
| **What a guest rebuild does to it** | nothing (it lives on the hub / at the provider) | destroys it; a fresh one is minted and installed — **harmless** | **destroys it; a fresh one is minted — CATASTROPHIC for the existing repo** |
|
||||||
|
| **Escrowed?** | no | no | **yes** — `IdentityBundle.ResticRepoPassword` |
|
||||||
|
|
||||||
|
The agent's own source names the asymmetry, and it is the sentence the whole spike turns on
|
||||||
|
(`felhom-agent/internal/escrow/identity.go:35-39`):
|
||||||
|
|
||||||
|
> `ResticRepoPassword` … **It is the DATA key for the offsite tier — irreplaceable (unlike the SFTP
|
||||||
|
> access key, which is regenerable at DR).**
|
||||||
|
|
||||||
|
**The one-shot thing (①) is the recoverable one. The irreplaceable thing (③) is the one nothing
|
||||||
|
re-stages.** R-193 was filed against ①. The damage is in ③.
|
||||||
|
|
||||||
|
**Instrument caveat, worth carrying (I4):** `one_time_secrets` has `customer_id` as PRIMARY KEY, and
|
||||||
|
`SaveOneTimeSecret` is last-write-wins **by design** (the R-39(a) guard comment in
|
||||||
|
`monitor/offsite_delivery.go:26-30` depends on it). So the delivery ledger holds only the newest
|
||||||
|
credential per customer: demo-hp's 2026-07-23 consume, on which R-192's whole diagnosis rests, was
|
||||||
|
**overwritten** by the 2026-08-04 Re-issue and is no longer readable from that table.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q4 — Which claim about Re-issue is true?
|
||||||
|
|
||||||
|
### **RULING: `ReissueCredentials` touches ONLY the provider credential (①). It does NOT touch the restic repository password. R-39's record is wrong — and the hub's own comments repeat the same wrong claim in three places.**
|
||||||
|
|
||||||
|
**Method — source, quoted rather than summarised.** `ReissueCredentials`
|
||||||
|
(`hub/internal/offsite/offsite.go:150-228`) does exactly four things:
|
||||||
|
|
||||||
|
1. `genPassword()`;
|
||||||
|
2. `ResetSubaccountPassword` / `ResetBoxPassword` + `WaitAction` — **the provider account password**;
|
||||||
|
3. `Store.SaveOneTimeSecret(customerID, pw)` — stage ① for delivery;
|
||||||
|
4. bookkeeping: `MarkEscrowStale` + `offsite_reissued` + `escrow_stale` events.
|
||||||
|
|
||||||
|
There is **no reference to a restic password anywhere in the function**, and none is possible — the
|
||||||
|
repo password is generated on the box and never leaves it except into the escrow blob. The hub does
|
||||||
|
not hold it in any openable form (`api/handler.go`: *"The hub stores the bytes and NEVER decrypts them
|
||||||
|
(it has no recovery code)"*).
|
||||||
|
|
||||||
|
**But step 4 is justified by a claim that is false**, `offsite.go:198-201`:
|
||||||
|
|
||||||
|
> `// v0.57.0 (2.3, the escrow-honesty fix): the restic repo password just changed, so any existing`
|
||||||
|
> `// key-escrow blob — which sealed the OLD password — is now STALE.`
|
||||||
|
|
||||||
|
The escrow *is* marked stale; the stated reason for marking it is not something this function does.
|
||||||
|
The same false premise appears twice more: `api/handler.go:1067-1069` (*"The re-issuer resets the
|
||||||
|
restic repo password, which makes the OLD escrow blob stale"*) and `OPEN-ITEMS.md`'s R-193 row, which
|
||||||
|
inherited it from R-39.
|
||||||
|
|
||||||
|
> **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does
|
||||||
|
> not provide** — and the first where the comment is not merely unenforced but factually describes a
|
||||||
|
> different function. It survived because it reads as settled and because its *effect* (a stale escrow)
|
||||||
|
> is real, so nobody checked the *cause*.
|
||||||
|
|
||||||
|
**Consequence, and it is a live defect, not a documentation nit.** On the ordinary Re-issue shape —
|
||||||
|
a consumed-but-failed install on a box that still has its `repo_password` file — the box re-applies,
|
||||||
|
`WriteOffboxSecrets` finds the file present and **keeps it**, and the repo password is unchanged. The
|
||||||
|
hub has nonetheless told the customer, in Hungarian, that their recovery escrow is stale and asked
|
||||||
|
them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** → **R-196**.
|
||||||
|
|
||||||
|
**Corroboration against what actually happened on demo-hp yesterday.** The escrow went `pending` after
|
||||||
|
the Re-issue and the task asks: if the repo password was untouched, what explains that? **It is fully
|
||||||
|
explained, and not by the Re-issue.** Two independent mechanisms, both hub-side-invisible:
|
||||||
|
|
||||||
|
1. `ApplyOffsiteTarget` (`offbox.go:482-494`) carries `EscrowState` over from the *existing* target —
|
||||||
|
but the rebuilt guest had **no existing target**, so `cur` was nil, so the fallthrough
|
||||||
|
`if tgt.EscrowState != "escrowed" { tgt.EscrowState = "pending" }` fired.
|
||||||
|
2. The box had minted a **new** repo password moments earlier (Q2), so `pending` was *correct* — the
|
||||||
|
new key genuinely was not escrowed.
|
||||||
|
|
||||||
|
**The Re-issue's `escrow_stale` event fired at 07:11:51 for the wrong reason and happened to be true
|
||||||
|
for a different one.** Coincidence, not correctness. The proof that these are independent is
|
||||||
|
**demo-felhom**: it received **no Re-issue at all** (measured — its only `escrow_stale` /
|
||||||
|
`offsite_reissued` events in the entire history are dated `2026-07-21 08:29:29`), its repo password
|
||||||
|
changed anyway, and **nothing marked its escrow stale for thirteen hours.**
|
||||||
|
|
||||||
|
> **The mechanism is exactly inverted.** `escrow_stale` is wired to the one path that does **not**
|
||||||
|
> change the repo password, and absent from the path that **does**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q5 — Why did one box recover itself and the other not?
|
||||||
|
|
||||||
|
### **RULING: confirmed from the ledger — an unconsumed staged secret, thirteen days old. And the recovery was partial in a way the narrative missed.**
|
||||||
|
|
||||||
|
**Method — I4, the `one_time_secrets` ledger, no values read:**
|
||||||
|
|
||||||
|
| customer | `created_at` | `consumed_at` |
|
||||||
|
|---|---|---|
|
||||||
|
| `demo-felhom` | `2026-07-21 08:29:29` | `2026-08-03 07:17:58` |
|
||||||
|
| `demo-hp` | `2026-08-04 07:11:51` | `2026-08-04 07:12:06` |
|
||||||
|
|
||||||
|
demo-felhom's row was **created 2026-07-21 and sat UNCONSUMED for 13 days** — traceable to a Re-issue
|
||||||
|
at that exact timestamp (its `offsite_reissued` + `escrow_stale` event pair, `2026-07-21 08:29:29`).
|
||||||
|
When the rebuild dropped its target on 2026-08-03, the fresh controller consumed the spare at 07:17:58
|
||||||
|
and was reporting `offsite` again by 07:19:10. **76 seconds, on a credential nobody had planned to
|
||||||
|
leave there.** Confirmed from the ledger, not the narrative.
|
||||||
|
|
||||||
|
**What would have happened without one:** exactly demo-hp — `consume` → **404** (`api/offsite.go:20-22`,
|
||||||
|
`no unconsumed offsite password`), the apply-bridge returns
|
||||||
|
`offsite-apply: consume one-time password: …` and retries forever, and the tier stays absent until an
|
||||||
|
operator acts. demo-hp sat there **25 hours** (`2026-08-03 06:12:19` → `2026-08-04 07:12:01`).
|
||||||
|
|
||||||
|
**The correction to the contrast (see Q2):** demo-felhom's spare restored *delivery* only. Its 36
|
||||||
|
snapshots / 1.14 GB were orphaned by the same rebuild. **Neither box's data survived; one box's
|
||||||
|
plumbing did.**
|
||||||
|
|
||||||
|
*Not established:* demo-felhom's escrow generation before 2026-07-21. `host_escrow_superseded` holds
|
||||||
|
only two rows in the whole database (ids 3 and 4, one per box, both superseded 2026-08-04), so whether
|
||||||
|
the 2026-07-21 Re-issue's ceremony re-sealed an *unchanged* password — the predicted false-staleness
|
||||||
|
shape — **cannot be shown from this data**. Stated as unknown rather than asserted.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q6 — Why is delivery one-shot at all?
|
||||||
|
|
||||||
|
### **RULING: the design's reason is that ① is a LIVE PROVIDER PASSWORD, not a bootstrap token; and the honest answer to the security question is that an automatic restage weakens EXPOSURE WINDOW, not authentication.**
|
||||||
|
|
||||||
|
**The design's own reasons, from source:**
|
||||||
|
|
||||||
|
- `api/offsite.go:9-12` — *"serves the one-time transient offsite password to the controller **EXACTLY
|
||||||
|
ONCE** … The value is returned once then marked consumed — a second call 404s. NEVER logged."*
|
||||||
|
- `offsite.go:145-149` — *"the **EXPLICIT** operator recovery for a consumed-password dead-end … It is
|
||||||
|
**NOT** implicit rotation: ProvisionOffsite never calls this."*
|
||||||
|
- `offsiteapply.go:1-6` — the apply-bridge is *"idempotent (a descriptor hash marker prevents
|
||||||
|
**re-consuming a spent password**)"*, i.e. the design treats a wasted consume as a real hazard, which
|
||||||
|
R-71a's whole settle-gate exists to prevent.
|
||||||
|
|
||||||
|
**The security half, stated rather than skipped.** The question is: *could a machine that is not the
|
||||||
|
real one obtain a credential via an automatic restage?*
|
||||||
|
|
||||||
|
**What stands there today is authentication, and a restage does not remove it.** The consume endpoint
|
||||||
|
is gated by `checkAuthCustomer` (`api/offsite.go:14-18`) — **the customer's API key, the same
|
||||||
|
credential as the config pull**. Anything that could consume a restaged password can already pull the
|
||||||
|
customer's whole config. So an automatic restage **grants no new access to an unauthenticated party**.
|
||||||
|
|
||||||
|
**What it genuinely weakens is the exposure window.** Today a live Hetzner sub-account password is
|
||||||
|
fetchable only in the seconds after an operator deliberately staged one. Under an automatic restage it
|
||||||
|
becomes fetchable **whenever the box reports no offsite target** — a condition an attacker holding the
|
||||||
|
API key can arguably induce. The credential's blast radius at the provider is the customer's own
|
||||||
|
sub-account (label-scoped; `ReissueCredentials` refuses unless the label lookup finds **exactly one**,
|
||||||
|
`offsite.go:164,180`), so this is a widening of *when*, not of *what*.
|
||||||
|
|
||||||
|
**And one further guard must survive any change** — R-39(a), `monitor/offsite_delivery.go:26-30`:
|
||||||
|
`SaveOneTimeSecret` is last-write-wins, so restaging on top of an **unconsumed** secret clobbers a
|
||||||
|
password a box may be about to consume. Any automatic restage must re-read the row immediately before
|
||||||
|
acting and refuse unless it is still a **consumed** row. That guard is correct and is not what R-192 is
|
||||||
|
about.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q7 — R-192's guard: exactly what it reads, and why it read wrong
|
||||||
|
|
||||||
|
### **RULING: both halves confirmed, with numbers. The guard reads the OLDEST 500 reports since the consume, and for demo-hp all 500 predated the rebuild by six days.**
|
||||||
|
|
||||||
|
**The query, quoted** (`hub/internal/store/store.go:987`):
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT report_json FROM reports WHERE customer_id = ? AND received_at > ? ORDER BY id LIMIT 500
|
||||||
|
```
|
||||||
|
|
||||||
|
`ORDER BY id` ascending with `LIMIT 500` = **the oldest 500**, not the newest.
|
||||||
|
|
||||||
|
**Reproduced against the live data (I1)**, with demo-hp's real consume anchor `2026-07-23 09:53:41`
|
||||||
|
(recovered from R-192's record — it is no longer in `one_time_secrets`, see I4):
|
||||||
|
|
||||||
|
| measurement | value |
|
||||||
|
|---|---|
|
||||||
|
| `total` (what the guard sees) | **500** |
|
||||||
|
| `withOffsite` (what the guard sees) | **500** |
|
||||||
|
| oldest report in that window | `2026-07-23 09:53:47` |
|
||||||
|
| newest report in that window | **`2026-07-28 11:17:40`** |
|
||||||
|
| true `total` over the same window, no LIMIT | 1174 |
|
||||||
|
| true `withOffsite` over the same window, no LIMIT | 1063 (⇒ 111 without — matching the 111 ABSENT reports in I3) |
|
||||||
|
|
||||||
|
**The whole 500-report evidence set ends 2026-07-28 — six days before the 2026-08-03 rebuild.**
|
||||||
|
|
||||||
|
**Defect (a) — the message is exactly inverted.** `maybeEmitStuck`
|
||||||
|
(`monitor/offsite_delivery.go:~110`) interpolates `status.ReportsSinceConsume` into a string that
|
||||||
|
hardcodes the phrase *"carry no offbox target"*, and never reads `OffsiteReportsSinceConsume`. The
|
||||||
|
live e-mail, quoted in full from the events table:
|
||||||
|
|
||||||
|
> *"Offsite delivery stuck: one-time password consumed 284h19m0s ago and **500 report(s) since carry
|
||||||
|
> no offbox target** — the credential is likely burned (apply died between consume and persist).
|
||||||
|
> Re-issue delivers a fresh one."*
|
||||||
|
|
||||||
|
`OffsiteReportsSinceConsume` for that same call was **500**. Every one of them carried a target. The
|
||||||
|
message states the precise negation of its own measurement, and prescribes a remedy for a failure mode
|
||||||
|
that did not occur.
|
||||||
|
|
||||||
|
**Defect (b) — the heal refuses silently, on stale evidence.** `maybeHeal` returns bare on
|
||||||
|
`status.OffsiteReportsSinceConsume != 0` (it was 500), with **no log line**. `offsite_credential_restaged`
|
||||||
|
has never fired for any customer — confirmed: zero rows of that event type in the database.
|
||||||
|
|
||||||
|
**What the guard SHOULD discriminate on — the shape, not the code.** The guard's intent is right:
|
||||||
|
*"there is offbox evidence, so this is a regressed apply, not a burned credential — the operator's
|
||||||
|
call."* Its error is that **it asks the question of the wrong time period**. Offbox evidence from
|
||||||
|
before a rebuild is not evidence that the credential still works. The discriminator must be
|
||||||
|
**recency-bounded and rebuild-aware**: judge on evidence *after the newest `controller_started` /
|
||||||
|
config-hash change*, or on the latest N reports — never on "everything since the consume". And a
|
||||||
|
refusal must **name its reason in the log**, because "we chose not to heal" and "the heal never ran"
|
||||||
|
must not look identical (the S-16 rule, one layer down).
|
||||||
|
|
||||||
|
**Do not fix it in isolation.** Under Q2's finding, a successful auto-restage on demo-hp would have
|
||||||
|
restored the transport and the box would *still* have minted a new repo password. **The heal cannot
|
||||||
|
protect the data; it can only protect the plumbing.** Whatever shape it takes must say so.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Q8 — What each candidate would cost
|
||||||
|
|
||||||
|
**A fourth option exists and is the important one**, because (a) and (b) both address ① and Q2 shows
|
||||||
|
the damage is in ③.
|
||||||
|
|
||||||
|
### (a) The hub restages automatically when a re-enrolled box reports no offsite
|
||||||
|
|
||||||
|
**A version of this ALREADY EXISTS and is wired to the wrong event.** `reissueOnReenroll`
|
||||||
|
(`hub/internal/api/handler.go:1051-1084`), leg **F3**:
|
||||||
|
|
||||||
|
> *"F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
|
||||||
|
> one-time password only ever reached the OLD controller)."*
|
||||||
|
|
||||||
|
It is called from `handleHostEnroll` — but **after** a mint-once-reuse short-circuit
|
||||||
|
(`handler.go:1004-1016`): `if existing != nil { …return… }`. **A guest rebuild leaves the `hosts` row
|
||||||
|
intact** (measured: `demo-hp-bb76ea` is the same host_id throughout the incident; only the agent leaf
|
||||||
|
re-keyed, `host_leaf_changed` 2026-08-03 06:09:40), so re-enroll returns the existing credential and
|
||||||
|
**F3 is never reached.**
|
||||||
|
|
||||||
|
- **Changes:** hub only. Either lower the trigger from host-enrollment to a controller-level rebuild
|
||||||
|
signal (the `config_hash` change + `controller_started` pair the hub already receives and already
|
||||||
|
logs), or fix R-192's guard so the existing R-71c self-heal can act.
|
||||||
|
- **Weakens:** the exposure window in Q6. Needs the R-39(a) unconsumed-secret guard kept intact.
|
||||||
|
- **Does NOT solve:** ③. The box still mints a new repo password and still orphans the repo. **This
|
||||||
|
option restores the plumbing to a box whose data key is already gone.**
|
||||||
|
- **Cost:** small. **Value against the actual harm: near zero.**
|
||||||
|
|
||||||
|
### (b) The credential becomes recoverable from escrow at re-bootstrap
|
||||||
|
|
||||||
|
> **ANNOTATED 2026-08-04 (recon) — this ruling is overturned in part. See the annotation at the top.**
|
||||||
|
> The constraint is real but narrower than written: an **unattended** rebuild cannot recover from a
|
||||||
|
> zero-knowledge escrow; a **customer-present** one can, and the operator has ruled on that shape.
|
||||||
|
> The recon also found that the chain this option would use is clientless at three links and lossy at
|
||||||
|
> a fourth (`RECON-offsite-dr-chain-2026-08-04.md` §3–§4).
|
||||||
|
|
||||||
|
- **BLOCKED AS STATED, and this is a hard constraint, not an estimate.** The escrow blob is
|
||||||
|
**R-wrapped and zero-knowledge** — the hub holds opaque bytes and has no recovery code
|
||||||
|
(`api/handler.go`, and D6 in `CONTEXT.md` S-3). **A rebuilt box cannot unseal it without the
|
||||||
|
customer's recovery code.** Any design that says "recover it at re-bootstrap" is describing a
|
||||||
|
customer-present ceremony, i.e. what the existing manual `offboxInjectPasswordHandler` already is.
|
||||||
|
- **Does NOT solve:** an unattended rebuild, which is the whole scenario.
|
||||||
|
- **Cost: not implementable as specified.**
|
||||||
|
|
||||||
|
### (c) — NOT PREVIOUSLY NAMED — the agent retains the repo password across a guest rebuild
|
||||||
|
|
||||||
|
The agent lives **on the Proxmox host** and survives a guest rebuild. It already receives the repo
|
||||||
|
password today: the controller pushes it over the pinned local API
|
||||||
|
(`POST /escrow/stage-secret` → `agent/internal/localapi/escrow_stage.go`), and the agent writes it to
|
||||||
|
the fixed 0600 path `/var/lib/felhom-agent/escrow-stage/restic_repo_password`
|
||||||
|
(`agent/internal/escrow/identity.go:44-46`) **transiently — wiped by the ceremony**
|
||||||
|
(`WipeStagedResticPassword`). The seam, the transport, the pinning and the file are all already built.
|
||||||
|
|
||||||
|
- **Changes:** agent (retain rather than wipe; serve back over the same pinned local API) + controller
|
||||||
|
(on a fresh data dir, ask the agent before minting — the `InjectOffboxPassword` seam already exists
|
||||||
|
and is exactly the right shape).
|
||||||
|
- **Weakens:** the escrow's *zero-knowledge* posture in one specific way — the irreplaceable data key
|
||||||
|
would now sit at rest on the Proxmox host, which the operator has root on. **Measured against D6,
|
||||||
|
that is not a new exposure in principle** (D6 already states plainly that *"the operator cannot read
|
||||||
|
customer data"* was never the security property, and the operator holds root on every box) — but it
|
||||||
|
IS a new copy, and copies are the thing R-133 is already open about for the hub DB. **This is a real
|
||||||
|
trade and it is the operator's to make.**
|
||||||
|
- **Solves:** exactly the failure that occurred, unattended, on every rebuild, without any hub
|
||||||
|
involvement and without any credential leaving the premises.
|
||||||
|
- **Cost:** medium. Two repos, one existing seam each.
|
||||||
|
|
||||||
|
### (d) Do neither — keep the manual Re-issue, and fix what LIES
|
||||||
|
|
||||||
|
- **Changes:** R-192's message + refusal log; correct the three false comments about what Re-issue
|
||||||
|
rotates; and — the part that actually matters — **make a repo-password change VISIBLE**. The hub
|
||||||
|
already receives `restic_pw_sha256` on every escrow upload and already stores it. Comparing the new
|
||||||
|
hash against the superseded one is a two-line verdict, and it is the signal that was missing on
|
||||||
|
demo-felhom for thirteen hours → **R-197**. (Note the shape: **this spike answered its own hardest
|
||||||
|
question with a comparison the hub could be making automatically and is not.** Both values were
|
||||||
|
already in the database; nothing read them.)
|
||||||
|
- **Weakens:** nothing.
|
||||||
|
- **Does NOT solve:** the data loss. Every rebuild still orphans the repo; the operator simply finds
|
||||||
|
out the same day instead of never.
|
||||||
|
- **Cost:** small.
|
||||||
|
|
||||||
|
### Ranked, with a recommendation
|
||||||
|
|
||||||
|
| rank | option | why |
|
||||||
|
|---|---|---|
|
||||||
|
| **1** | **(d) now — the honesty pass** | It is cheap, it weakens nothing, and **it is the only option that helps the incident already in flight.** The alarm that lies is worse than no alarm; the repo-password-changed detector is the one signal whose absence let demo-felhom lose 1.14 GB of history silently. Ship this regardless of what is decided about the rest. |
|
||||||
|
| **2** | **(c) — agent-retained repo password** | The only candidate that addresses ③, which is where the harm is. Every seam it needs exists. It costs one deliberate trade-off the operator must actually make. |
|
||||||
|
| **3** | **(a) — auto-restage** | Worth doing **after** (c), never instead of it. On its own it restores the transport to a box whose data key is already gone, and would have made both boxes look healthy on 2026-08-04 while the snapshots were orphaned — a *strictly worse* outcome than the current loud failure. |
|
||||||
|
| **4** | **(b) — recover from escrow at re-bootstrap** | Not implementable as stated; the escrow is zero-knowledge by design. |
|
||||||
|
|
||||||
|
**Recommendation: ship (d), then decide (c).** And do not ship (a) first — it would have hidden this.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What could not be established
|
||||||
|
|
||||||
|
Named rather than guessed (§8 rule 4).
|
||||||
|
|
||||||
|
1. **The 2026-08-05 02:15 UTC run's actual outcome** (Q3). Predicted `ErrOffboxOrphaned` from source +
|
||||||
|
claim state; **not measured**.
|
||||||
|
2. **Whether the orphaned ciphertext still exists at the provider.** 40.9 MB (demo-hp) + 1.14 GB
|
||||||
|
(demo-felhom) are *presumed* present at `/home/felhom-repo` — reading the endpoint needs the SFTP
|
||||||
|
credential, which this session did not touch. The Hetzner API has no directory-listing surface.
|
||||||
|
3. **Whether demo-felhom's 2026-07-21 Re-issue re-sealed an unchanged password** (the predicted
|
||||||
|
false-staleness shape, Q4/Q5). Only two `host_escrow_superseded` rows exist in the whole database;
|
||||||
|
the generation before 2026-07-21 is not retained.
|
||||||
|
4. **What removed demo-hp's offsite target at the rebuild** is fully explained (fresh data dir), but
|
||||||
|
**why the guest was rebuilt on both boxes on consecutive mornings** is outside this spike.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The operator's question — stated, and NOT answered here
|
||||||
|
|
||||||
|
> **The irreplaceable secret in the offsite tier is the restic repository password. It is generated on
|
||||||
|
> the box, it is destroyed by a guest rebuild, and the only copy that survives is sealed under the
|
||||||
|
> customer's recovery code — which nothing but a human can open. Every machine is going to be
|
||||||
|
> reinstalled.**
|
||||||
|
>
|
||||||
|
> **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host, where
|
||||||
|
> you already have root — or do you want it to stay sealed under the customer's recovery code and
|
||||||
|
> accept that every rebuild starts a new repository and orphans the old one, provided the system says
|
||||||
|
> so loudly on the day it happens?**
|
||||||
|
|
||||||
|
Option (c) is the first. Option (d) alone is the second. **This spike does not choose.**
|
||||||
File diff suppressed because one or more lines are too long
@@ -26,7 +26,7 @@
|
|||||||
| R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 |
|
| R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 |
|
||||||
| R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 |
|
| R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 |
|
||||||
| R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/<customer>`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 |
|
| R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/<customer>`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 |
|
||||||
| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path |
|
| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path. **SCOPE-CORRECTED 2026-08-04 (R-193 spike, `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`): "auto-fire on re-enrollment" means HOST re-enrollment, and only when the host record was DELETED first.** `reissueOnReenroll` sits behind `handleHostEnroll`'s mint-once-reuse short-circuit (`if existing != nil { return }`, `api/handler.go:1004-1016`), so a **guest** rebuild — which leaves the `hosts` row intact — never reaches F2 or F3. Measured live: demo-hp's guest was rebuilt 2026-08-03, `demo-hp-bb76ea` was unchanged throughout, F3 did not fire, and the box had **no off-site copy for 25 hours** until an operator ran Re-issue by hand. Peti's clean-slate (a real host delete + re-enroll) still walks the supported path; **the arc does not cover a guest rebuild, and that is the gap R-193 owns.** Note also that F3 alone would NOT have saved the data — the rebuilt controller mints a fresh restic repo password regardless (spike Q2) |
|
||||||
| R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
|
| R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
|
||||||
| R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |
|
| R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |
|
||||||
| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC `p=none` → `p=quarantine` | S | **DONE 2026-07-21 (all three halves)** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu half CLOSED — operator-verified 2026-07-21**: a `[Felhom] Teszt értesítés` was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by `p=quarantine`, **R-4 is COMPLETE** |
|
| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu; tighten DMARC `p=none` → `p=quarantine` | S | **DONE 2026-07-21 (all three halves)** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu half CLOSED — operator-verified 2026-07-21**: a `[Felhom] Teszt értesítés` was received at a freemail.hu address. With the gmail half closed by the 2026-07-18 rehearsal and the DNS half by `p=quarantine`, **R-4 is COMPLETE** |
|
||||||
@@ -57,6 +57,17 @@
|
|||||||
| R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. **Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17):** the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above |
|
| R-16 | Operator hygiene: campaign6 autofs orphan (clears on host reboot) + tied-CreatedAt flash duplicates (audiobookshelf/komga/romm) | XS | open (doc-drift bit CLOSED) | Viktor's own action items from 6D/6E. **Doc-drift leftover CLOSED (host-install v1.17.0, 2026-07-17):** the R-20-noted stale "EMPTY by default" operator-key comment corrected (keys are PINNED). Remaining = the two operator items above |
|
||||||
| R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/<id>` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED + PROVEN-LIVE agent v0.89.0** (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → `converged state=adopted` in ~3 s, ACLs restored, offsite active). Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. |
|
| R-22 | **PBS-DR pre-check self-grant (F4).** On a non-default storage id the token-auth `GET /storage/<id>` pre-check 403s (no ACL yet) and used to abort before the root-run `grant` that creates it. | S | **SHIPPED + PROVEN-LIVE agent v0.89.0** (2026-07-17) | On a 403 the reconcile self-grants via the root wrapper + re-reads, then converges. Red-proof `TestSelfGrant_PreCheck403DoesNotAbortBeforeGrant`; live-reproduced on the demo (marker aside + ACLs revoked → self-grant → `converged state=adopted` in ~3 s, ACLs restored, offsite active). Origin `tests/VALIDATION-n100-baremetal-2026-07-16.md` F4. |
|
||||||
| R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** |
|
| R-17 | Old-box archive (u629193-sub1) retirement — 9/9 byte-identical restores verified | XS | **CLOSED 2026-07-22 — archive deleted (operator console)** | **operator ruling 2026-07-21**: delete the archive now; the 9/9 byte-identical restores are the evidence it is no longer needed. **CC spike-lite (2026-07-21) — it CANNOT be deleted through existing tooling, and CC did not improvise credentialed API calls:** (a) `felhom-tenantsync.sh` does not touch Hetzner at all — it manages PBS namespaces/tokens only; (b) the hub CAN delete sub-accounts (`hetznerapi.DeleteSubaccount`), but its ONLY caller is `offsite.Deprovision`, which selects by the `felhom-customer=<id>` label and runs as part of a customer RESET — a legacy archive sub-account from the 2026-07-09/10 spikes carries no such label, so no tooling verb targets it; (c) the only in-repo references are TEST FIXTURES (`hub/internal/hetznerapi/fake.go` generates the `u629193-sub%d` pattern; `felhom-controller/.../dr_recipe_test.go` uses it as a literal) — nothing live points at it. **Operator console steps:** Hetzner Storage Box console → box `u629193` → Sub-accounts → `u629193-sub1` → confirm the directory holds only the old-box archive → Delete. Deleting the sub-account destroys its data; the 9/9 verification is the sign-off. **CLOSED 2026-07-22: the operator executed the console action — `u629193-sub1` deleted, confirmed no sub-accounts remain on PBS-storage-1** |
|
||||||
|
| R-198 | **The superseded-escrow retention kept the K-escrow and dropped the identity blob** — so it preserved the PBS key and destroyed the offsite restic repository password, with the escrow CEREMONY as the destroying act | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | Corrects the escrow-ceremony capability-map row's v0.60.0 retention claim (was struck through 2026-08-04, now restated as what ships). One column + one SELECT list, in the shared `demoteCurrentEscrowTx`, so both callers are fixed at once. Nothing backfillable. **Load-bearing, not tidy:** with R-193(c) refused, customer-present recovery is the only route back from a rebuild and it runs entirely through the retained identity blob |
|
||||||
|
| R-197 | **The hub stored both halves of "did this box's offsite data key change" and compared them nowhere** | S | **SHIPPED (hub v0.93.0, 2026-08-04)** | New `offsite_repo_key_changed` (warning, operator-only, edge-triggered, no hash values). Flips nothing in the capability map — it adds a signal, not a capability |
|
||||||
|
| R-196 | **Five comments claimed `ReissueCredentials` rotates the restic repo password** — it resets the PROVIDER password and cannot touch the repo password | S | **comments corrected (hub v0.93.0); the BEHAVIOUR stays open** | The false-staleness alarm is a behaviour change and must not ride a comment-correction release — and it matters MORE now, because under R-198 an unnecessary ceremony supersedes a blob rather than being harmless bookkeeping |
|
||||||
|
| R-202 | **The orphan card promises recoverability unconditionally**, which after R-198 is true going forward and false for anything already orphaned | S | **gate hit 2026-08-04 — card untouched, sentence still live** | Blocked on knowing which escrow generation an orphaned repo belongs to (R-199/R-201). A single ACK boolean can say a retained recoverable blob EXISTS but not that one COVERS this repo; a conditional promise that can still be false is worse on that surface than a hedged one |
|
||||||
|
| R-199 | **The hub served recovery blobs on endpoints with no client anywhere** — the DR capstone's retrieval had never been called by the agent, the hub UI, a script or a runbook | M | **SHIPPED + PROVEN-LIVE (hub v0.94.0 + agent v0.125.0, 2026-08-04)** | Adds the capability-map row *"the offsite repository password can be RECOVERED from the sealed escrow"* as PROVEN-LIVE — for the KEY only, explicitly not for a restore. Chain links 6–8 walked on demo-felhom: recovered sha256 == on-disk sha256 == the hub's stored hash. Carries a deliberate security trade (a box's own credential now suffices where an operator-armed recovery mode was required), recorded in the handler and overridable by one constant |
|
||||||
|
| R-200 | **The password-injection seam had a handler and no form** | M | **plumbing SHIPPED (controller v0.195.0); the form is NOT built** | `--recover-offsite-check` compares by hash and refuses to install. The customer-facing shape (yell → recovery-code form → preview → proceed) is priced against a chain that now exists rather than one that was assumed |
|
||||||
|
| R-203 | **A mandatory customer data directory was silently absent from the off-site snapshot while the run reported `ok`** — the deploy-time `${USERDATA_PATH}` root and the backup-time namespace root disagree for an app on the system drive | M | **OPEN — halted the R-201 drill 2026-08-04** | Flips nothing yet. Blocks the off-site app-data row from ever earning a customer-file-restored badge. Fix shape: one root function, and a MANDATORY skip must be customer/hub-visible rather than a container-log WARN |
|
||||||
|
| R-201 | **The wipe-and-recover drill** | L | **PREPARED, HALTED BEFORE THE WIPE (2026-08-04)** | Nothing irreversible done. Established live for the first time: a rebuilt box's off-site run REFUSES (the orphan card, not a silent fresh history — closes R-193 Q3), the orphan reset works move-aside-never-delete, and demo-hp's pre-rebuild key is permanently gone (superseded four hours before v0.93.0). Resume needs R-203 |
|
||||||
|
| R-203 | **The app and its backup looked in different directories, and a run that skipped a mandatory folder still said `ok`** | M | **SHIPPED + PROVEN-LIVE (controller v0.197.0, 2026-08-04)** | Adds the capability-map row *"off-site app-data capture covers MANDATORY paths on both drive layouts"* as PROVEN-LIVE, and corrects that row's predecessor, which was optimistic. Unblocks R-201 |
|
||||||
|
| R-204 | **The recovered key cannot be used: the Re-issue that reconfigures a rebuilt box's off-site tier marks the escrow stale, which gates every run, and the only way to clear it destroys the key** | M | **OPEN — the wall the 2026-08-04 night drill hit** | Blocks R-201's verdict. Fix is most likely R-196's (mark stale on evidence, not on a Re-issue) or letting a verified recovered key confirm the escrow without a ceremony |
|
||||||
|
| R-201 | **The wipe-and-recover drill** | L | **PASSED + PROVEN-LIVE (2026-08-04 night)** | Flips the off-site app-data row to *"a customer's file survives a machine rebuild and comes back"* — PROVEN-LIVE, scoped to demo-hp and to a controller-data rebuild. The journey took four undocumented manual steps (R-204); the capability is real, the customer experience is not yet |
|
||||||
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
|
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
|
||||||
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
|
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
|
||||||
| R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |
|
| R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, 2026-07-16): **(a) — BANKED 2026-07-21 (both legs).** The operator-UI save->apply round trip is PROVEN: the STOP-2 global-floor save (hub `18:56:27 CEST`) released the controller's held wait in the **same second** (`16:56:27Z wait woke: generation=1 - firing out-of-cycle report`), with the report built 2 s later; the ring also shows `wait baseline generation=0` at startup (baseline recorded WITHOUT firing, as designed) then `generation=1`, so the generation advanced past 0. **RESTART LEG BANKED 2026-07-21 — the floor was moved to a version the box did NOT run, and the swap fired EXACTLY ONCE.** Operator saved global floor 0.153.0 → **v0.154.0** (a real version boundary, unlike the 2026-07-20 attempt which targeted an already-running version and therefore proved nothing). Timeline (guest UTC): `06:57:13` UpdateState `pending` written `initiated_by=auto-floor` → `06:57:17` agent `controller swap requested 0.153.0 -> 0.154.0` → `06:57:19` `image file written, restarting bootstrap` → `06:57:21` container StartedAt + UpdateState `completed_at` → `06:57:29` agent `new controller healthy`. **16 s end to end.** Assertions over the whole window (06:50 → 07:29, 39 min): `controller swap requested` = **1**, agent-driven bootstrap restarts = **1**, `new controller healthy` = **1**, rollback/swap-failed/unhealthy = **0**, container `RestartCount` = **0**. `VerifyStartup` banked it on the next boot (`Post-update startup: update successful (0.153.0 → 0.154.0)`) and the `06:57:52` periodic check logged `Current version 0.154.0 is up to date` — the at/above-floor branch correctly doing nothing. **No storm, no rollback, no second attempt.** *Caveat, disclosed: a hand-deploy of v0.155.0 at `07:17:10` sits inside the observation window and is what StartedAt shows after that point; it never touches `SwapController`, so the swap-count assertions above are uncontaminated across the full window. A second, unplanned confirmation of the at/above-floor branch came with it — after the hand-deploy the box ran 0.155.0 against a 0.154.0 floor and the updater logged `Current version 0.155.0 is up to date` and did nothing.* Evidence: `felhom-controller/REPORT.md` §6 (2026-07-21). *(Superseded note:)* **the self-restart single-fire leg was** - the floor was set to a version the box ALREADY ran, so there was no work and no restart. Finish by bumping the floor to a version the box does NOT run, debug ring open, asserting EXACTLY ONE restart. **Trap found while banking this: the wake is `logx.Debugf`, so it is INVISIBLE in `docker logs` at INFO** and lives only in the debug ring (`GET /api/debug/logs?level=DEBUG`) - a hunter looking at stdout wrongly concludes the box never woke; arguably (b) generalised. (b) cosmetic: the Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect | S | **(a) BANKED in full; only (b) cosmetic remains** | Map row "config/state change round-trips in seconds" flipped PARTIAL->PROVEN-LIVE 2026-07-21 on this evidence. Evidence: `felhom-controller/REPORT.md` 4f |
|
||||||
|
|||||||
@@ -1,3 +1,237 @@
|
|||||||
|
## v0.95.0 — a re-issue no longer marks a healthy escrow stale (2026-08-05, R-196 / R-204 item 2)
|
||||||
|
|
||||||
|
`ReissueCredentials` called `MarkEscrowStale` on every re-issue that found an escrow row. The mark was
|
||||||
|
**precautionary, not evidential**: it guessed that the box's re-apply MIGHT mint a fresh repository
|
||||||
|
password. On the ordinary re-issue shape — a box that still holds its `<DataDir>/offbox/repo_password`
|
||||||
|
— the password does not change, so it marked a HEALTHY escrow stale. The 2026-08-04 recovery drill
|
||||||
|
(R-201) is what promoted this from a nit to a blocker.
|
||||||
|
|
||||||
|
**What the mark actually did**, because "asks for an unnecessary ceremony" understates it:
|
||||||
|
|
||||||
|
1. `stale_at` set → `GetEscrowStatusForCustomer` WITHHOLDS `restic_pw_sha256` from the report ACK.
|
||||||
|
2. With no hash, the controller's SLICE-3 auto-confirm cannot flip pending→escrowed.
|
||||||
|
3. `OffboxRunnable() = OffboxConfigured() && EscrowState=="escrowed"` → **every off-site backup
|
||||||
|
refused, indefinitely**, on a box whose key was never in doubt.
|
||||||
|
4. The customer is told to re-run the recovery ceremony — which mints a new recovery code and
|
||||||
|
supersedes the sealed blob. During a recovery that is the one act that would have destroyed the key
|
||||||
|
just recovered.
|
||||||
|
|
||||||
|
A precautionary flag that stops the data protection it is guarding is not conservative.
|
||||||
|
|
||||||
|
**Removing it leaves no gap — the case it guessed at is measured elsewhere, and better.**
|
||||||
|
|
||||||
|
- **Continuous, box-side:** the controller compares the ACK's sealed hash against its CURRENT local repo
|
||||||
|
password on every report ACK (`report.EscrowAutoConfirmer.reconcileEscrowed`, the Scenario-F
|
||||||
|
re-check). In the guest-rebuild shape — the only shape where a re-issue is followed by a fresh
|
||||||
|
repository password — that mismatches within one report cycle and raises the stale card plus the
|
||||||
|
„create a new recovery code" CTA. Pinned by the controller's `TestEscrowStale_MismatchWarnsOnceAndFlags`.
|
||||||
|
**And the mark was BLINDING it:** by emptying the hash (step 1) it removed the very value that
|
||||||
|
comparison needs, leaving the box able to report only the hash-LESS reason, which is false.
|
||||||
|
- **Edge-triggered, hub-side:** R-197's `offsite_repo_key_changed` fires on a proven hash difference
|
||||||
|
across a supersession and pages the operator. Pinned by `TestEscrowPut_ChangedRepoKey_RaisesSignal`;
|
||||||
|
red-proved by removing the `maybeEmitRepoKeyChanged` call.
|
||||||
|
|
||||||
|
`offsite_reissued` is unchanged and still always fires — the customer must still learn that the
|
||||||
|
credential moved. Only the stale marking and its `escrow_stale` event are gone.
|
||||||
|
|
||||||
|
**`MarkEscrowStale` is kept, not deleted, and now has no caller** — deliberately. The `stale_at` flag
|
||||||
|
is still live and correct (read by the ACK, the operator card and the PBS-DR view); the right way to
|
||||||
|
set it is a future caller that has MEASURED a key change rather than guessed at one. Its doc now says
|
||||||
|
so, and `TestEscrowStaleMechanism_StillWithholdsAndClears` keeps the mechanism from decaying while
|
||||||
|
nothing writes it.
|
||||||
|
|
||||||
|
**Disagreement recorded** (the R-96 standing rule): the task's Scenario D asked that a real key change
|
||||||
|
also "mark the escrow stale". It must not, and nothing was changed to make it — the hub learns of a
|
||||||
|
real change at the moment a supersession SEALS THE NEW PASSWORD, i.e. when the escrow is freshest.
|
||||||
|
Marking it stale there would ask for a ceremony to fix the ceremony that just ran. The correct
|
||||||
|
consequence at that instant is the operator alarm, which is what R-197 already does.
|
||||||
|
|
||||||
|
`TestReissue_InvalidatesEscrow` is replaced in place by `TestReissue_DoesNotMarkAHealthyEscrowStale`,
|
||||||
|
its exact inverse, so the reversal is visible in `git log -L`. The old test was not wrong about the
|
||||||
|
effect it measured; it was wrong about the cause, and it pinned the defect.
|
||||||
|
|
||||||
|
## v0.94.0 — a box can fetch its own sealed recovery package (2026-08-04, R-199 link 6)
|
||||||
|
|
||||||
|
**Chain link 6 had no client.** The hub has served the identity blob since slice 10D, from
|
||||||
|
`handleReEnroll` / `handleGetRestoreDirective` — gated on operator-armed recovery mode and the global
|
||||||
|
key. Nothing in the agent, the hub UI, any script or any runbook ever called either. The only
|
||||||
|
documented retrieval was `sqlite3 … writefile(…)` by hand on a `kubectl cp`-ed database.
|
||||||
|
|
||||||
|
**`GET /api/v1/hosts/{host_id}/escrow`** — the box-authenticated MIRROR of the PUT that put the blob
|
||||||
|
there. Self-scoped: a per-host key reads only its own; the global key may read any, the same asymmetry
|
||||||
|
the PUT has. A host with no sealed bundle gets `200 {present:false}` — a clean answer, because a 404
|
||||||
|
is indistinguishable from an unknown host and a bare empty 200 from a zero-length blob, and neither
|
||||||
|
of those is what "no ceremony has run yet" means.
|
||||||
|
|
||||||
|
**THE TRADE, RECORDED IN THE HANDLER RATHER THAN INFERRED.** Before this, obtaining the blob required
|
||||||
|
the OPERATOR to arm recovery mode. Now whoever controls a rebuilt box can obtain it with that box's
|
||||||
|
own credential. That is a real reduction in the number of parties required, and it is the whole of the
|
||||||
|
trade — they still cannot open it, because the hub has never held R and a wrong code fails closed at
|
||||||
|
age's scrypt KDF. **The mitigation is that the capability is audited rather than silent:** every
|
||||||
|
successful retrieval raises `escrow_blob_served` (warning, operator-only), recorded before the bytes
|
||||||
|
leave. A silent capability on this object is the shape the last two weeks were spent removing.
|
||||||
|
|
||||||
|
`escrowSelfServiceRetrieval` is a single named constant — the §8.2/§8.3 decision point. Flipping it to
|
||||||
|
false re-imposes the recovery-mode requirement on this endpoint and changes nothing else, so the
|
||||||
|
operator can overrule the trade at the cost of a boolean rather than a redesign.
|
||||||
|
|
||||||
|
**The operator-driven DR path is untouched** — same gate, same behaviour, pinned by a test that
|
||||||
|
exercises re-enroll and restore-directive with recovery mode off and on. Red-proofs observed: removing
|
||||||
|
the ownership check makes a cross-host read succeed (host A served host B's blob); removing the audit
|
||||||
|
record makes the retrieval silent.
|
||||||
|
|
||||||
|
**Not in this release:** the customer-facing flow. No card, no form, no preview — those are designed on
|
||||||
|
ground that has been walked, and R-200/R-201 are that walk.
|
||||||
|
|
||||||
|
## v0.93.0 — the retention keeps the key it was built to keep, and three things stop lying (2026-08-04, R-198/R-197/R-196/R-192)
|
||||||
|
|
||||||
|
### R-198 — the superseded-escrow retention was preserving the wrong key, and the ceremony was destroying the right one
|
||||||
|
|
||||||
|
`host_escrow_superseded` shipped in v0.60.0 with a `blob` column — the **K-escrow**, i.e. the PBS
|
||||||
|
datastore encryption key. `identity_blob` was added to `host_escrow` **later**, by the slice-10D
|
||||||
|
`ALTER TABLE`, and **never to the retained table**. The offsite **restic repository password** — the
|
||||||
|
data key for every off-site app-data snapshot — lives inside `identity_blob`. So
|
||||||
|
`demoteCurrentEscrowTx`, whose own comment calls it *"THE ONE escrow row-copy routine"*, retained the
|
||||||
|
whole-guest key and silently dropped the off-site data key: **precisely the secret the reinstall-orphan
|
||||||
|
incident the feature was built for was about.**
|
||||||
|
|
||||||
|
**What makes it the most serious defect this project has recorded is not the missing column — it is
|
||||||
|
which act destroys the key.** The copy happens as the new blob overwrites the old, so the destroying
|
||||||
|
act is the **escrow ceremony**: the exact thing a rebuilt box tells its customer to run, on a card
|
||||||
|
that promises in Hungarian that the old backups stay recoverable with their recovery code. Both demo
|
||||||
|
boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08), and their previous repository passwords
|
||||||
|
are now unrecoverable whether or not anyone kept the recovery codes.
|
||||||
|
|
||||||
|
- `host_escrow_superseded` gains `identity_blob BLOB` (in the CREATE for fresh databases **and** as an
|
||||||
|
additive `ALTER TABLE`, following the existing idiom; tolerated on re-run, changes no existing row).
|
||||||
|
- `demoteCurrentEscrowTx` carries it — so **both** callers are fixed by one change: the re-escrow path
|
||||||
|
(`SaveHostEscrow`) and the host-delete demotion (`DeleteHost`).
|
||||||
|
- `ListSupersededEscrow` reads it back and `store.HostEscrow` gains `IdentityBlob`, so a retained blob
|
||||||
|
is reachable from Go at all. `CountCurrentEscrowWithIdentity` is the census of who the fix protects.
|
||||||
|
- The comment above the table now records that the ruling stated there was not met, and what it cost.
|
||||||
|
- **Nothing is backfillable and the code says so:** rows superseded before this were written without
|
||||||
|
the identity blob and their source rows are already overwritten.
|
||||||
|
|
||||||
|
**Tests assert the CONSEQUENCE, not the mechanism** — the pre-existing
|
||||||
|
`TestSaveHostEscrow_RetainsSuperseded` was green throughout, because it asserted that a retained row
|
||||||
|
exists with the old K-blob. `TestSaveHostEscrow_RetainsIdentityBlob` asserts the retained row can still
|
||||||
|
yield a repository password, and pins the load-bearing ordering (the identity blob is written **after**
|
||||||
|
`SaveHostEscrow`, so the demote sees the previous generation). `TestDeleteHost_DemotesIdentityBlob`
|
||||||
|
proves the shared routine through its **other** caller — a shared routine tested through one caller is
|
||||||
|
how a fix gets believed on a path nobody exercised. Red-proofs: dropping the column from the copy fails
|
||||||
|
both; fixing only the re-escrow caller fails the delete one while the re-escrow one passes.
|
||||||
|
|
||||||
|
### R-197 — the hub held both halves of the evidence and compared them nowhere
|
||||||
|
|
||||||
|
`restic_pw_sha256` is stored on the current escrow and carried to the retained row on every
|
||||||
|
supersession. Comparing the two is what answered the 2026-08-04 recon's hardest question in one query —
|
||||||
|
and nothing in the hub did it. demo-felhom's repository password changed on 2026-08-03, orphaning 36
|
||||||
|
snapshots / 1.14 GB, and **no event, e-mail, card or log line said so for thirteen hours.**
|
||||||
|
|
||||||
|
`SaveHostEscrow` now returns the hash it replaced, and `handleHostEscrowPut` raises
|
||||||
|
**`offsite_repo_key_changed`** (warning, operator-only, edge-triggered once per supersession) when both
|
||||||
|
hashes are known and differ. The in-between shapes are logged rather than dropped, so *"we chose not to
|
||||||
|
alarm"* and *"the check did not run"* never look identical. **No hash value travels** — not in the
|
||||||
|
message, not in the details; the event names the host, the customer and the retained count.
|
||||||
|
|
||||||
|
**Severity chosen for the world this release creates, not the one it inherits:** before R-198 a changed
|
||||||
|
key meant the previous history was unopenable by anyone, ever, which would have argued for `error`.
|
||||||
|
From v0.93.0 the superseding ceremony retains the old identity blob, so the situation is *"this
|
||||||
|
customer's off-site history now depends on an older recovery code"* — operator-actionable, not a loss.
|
||||||
|
`warning` also routes: the dispatcher treats `info` as an intentional non-notify.
|
||||||
|
|
||||||
|
### R-192 (partially) — the alert describes what was measured, and a refusal leaves a trace
|
||||||
|
|
||||||
|
demo-hp e-mailed this daily from 2026-08-03: *"one-time password consumed 284h19m0s ago and 500
|
||||||
|
report(s) since carry no offbox target — the credential is likely burned … Re-issue delivers a fresh
|
||||||
|
one."* `OffsiteReportsSinceConsume` for that same call was **500**: every one of those reports carried a
|
||||||
|
target. The message stated the precise negation of its own measurement and prescribed a remedy for a
|
||||||
|
failure mode that had not occurred — while the self-heal correctly declined, **in complete silence**.
|
||||||
|
|
||||||
|
- The one stuck state is now reported as the two situations it actually covers, **burned** and
|
||||||
|
**regressed**, each stating its own measurement and carrying its own recommendation (the regressed
|
||||||
|
text explicitly withdraws Re-issue and points at what removes an offbox target — a guest rebuild,
|
||||||
|
R-193). `offsite_reports_since_consume` is in the details for the first time.
|
||||||
|
- **The counting window is named in the text.** `CountReportsOffsiteSince` reads `ORDER BY id LIMIT
|
||||||
|
500` — the **oldest** 500 reports after the consume — so on a long-lived box the counts describe the
|
||||||
|
start of the window, not the present. **That guard defect stays OPEN (R-192)** because its correct
|
||||||
|
shape depends on the recovery chain that is not yet assembled (R-199/R-200/R-201); naming the window
|
||||||
|
is how it stays visible instead of being laundered into a confident sentence.
|
||||||
|
- **Every self-heal refusal now leaves a record** — a `notification_log` row on the operator channel
|
||||||
|
with status `refused` and its reason, the R-182 suppressed-e-mail precedent. It rides the stuck
|
||||||
|
event's 24 h cadence, so it sits beside the e-mail it explains rather than accumulating per tick.
|
||||||
|
**The guard's logic is unchanged**: the set of situations in which the heal fires is what it was.
|
||||||
|
- `offsite_delivery_stuck` and `offsite_credential_restaged` are added to `operatorOnlyEvents`. Neither
|
||||||
|
was ever registered, and neither has a `customerMessages` entry — which is **not** a block, so a
|
||||||
|
customer with a configured recipient was in line for an English e-mail about one-time passwords being
|
||||||
|
*"likely burned"*. Narrowing only; the operator channel is untouched.
|
||||||
|
|
||||||
|
### R-196 — five comments corrected (the spec expected three)
|
||||||
|
|
||||||
|
`ReissueCredentials` resets the **provider account** password. It does not and cannot touch the restic
|
||||||
|
repository password, which is generated on the box and never leaves it except sealed under R. Five
|
||||||
|
places in the hub said otherwise: `offsite/offsite.go` (the `MarkEscrowStale` justification),
|
||||||
|
`api/handler.go` (the F3 re-enroll comment), and three in `store/store.go` (the `stale_at` ALTER, the
|
||||||
|
`MarkEscrowStale` doc comment, and `EscrowStatus.Stale`). **That false belief sent two separate
|
||||||
|
investigations the wrong way in one day.** All five now say what the code does, name the correction and
|
||||||
|
cite the recon; the staleness mark is documented as **precautionary**, not evidential. **The behaviour
|
||||||
|
is unchanged and R-196 stays OPEN** — the false-staleness alarm it causes is a behaviour change and
|
||||||
|
must not ride a comment-correction release.
|
||||||
|
|
||||||
|
### Not in this release
|
||||||
|
|
||||||
|
**R-199, R-200 and R-201 remain open** — the hub's blob-serving endpoints still have no client, the
|
||||||
|
password-injection seam still has no form, and nothing in the recovery path has ever been exercised.
|
||||||
|
**Part 5 hit its gate and the orphan card is untouched** — see R-202: a single ACK boolean can say
|
||||||
|
whether a retained recoverable blob exists, but not whether one covers *the key this orphaned
|
||||||
|
repository was written under*, and a conditional promise that is still capable of being false is worse
|
||||||
|
on that surface than a hedged sentence.
|
||||||
|
|
||||||
|
## v0.92.0 — a customer with no machine ever bound does not alarm (2026-08-04, R-195)
|
||||||
|
|
||||||
|
`david` is a real prospective customer whose record was created 2026-08-01 with **no host ever bound**
|
||||||
|
— `hosts=0`, `host_deletions=0`, `host_reports=0`, `reports=0` — and it e-mailed an
|
||||||
|
`expected_dbdump_missed` **error** at 03:00 UTC on 2026-08-02, 08-03 and 08-04. Nothing has ever been
|
||||||
|
expected of that customer, so the honest verdict is UNKNOWN, not missed.
|
||||||
|
|
||||||
|
**The mechanism is the interesting half, because the guard that should have covered this is keyed off
|
||||||
|
the wrong thing.** `CheckBackupDeadlines` already skips nodes that are `down`, and that skip reads
|
||||||
|
`StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — **a query over the
|
||||||
|
`reports` table**. A customer with zero reports is in no row, gets no state at all, and `GetState()`
|
||||||
|
returns `""` rather than `"down"`. **The skip misses exactly the customer it would most obviously
|
||||||
|
cover.** Corroborated on live data: `peti-felhom` is active with a host deleted 2026-07-15 and does
|
||||||
|
*not* alarm, because it has 482 old reports and is therefore `down`. The backup half was already safe
|
||||||
|
(`reportJSON == ""` → skip); the DB-dump half had no guard at all.
|
||||||
|
|
||||||
|
- **`store.HasEverBoundHost(customerID)`** — a live `hosts` row **OR** a `host_deletions` tombstone.
|
||||||
|
Consulted once per customer at the top of the deadline loop; a false verdict skips the customer with
|
||||||
|
an INFO line and its own counter in the summary.
|
||||||
|
- **The discriminator is "was a host EVER bound", NOT "has a report arrived", and that is the whole
|
||||||
|
design.** A box that was installed, bound and then went silent is a real fault and must keep
|
||||||
|
alarming — it has a `hosts` row, so it is judged. Only a customer that never had a machine is
|
||||||
|
UNKNOWN. The tombstone is included for the same reason: a customer whose host was deleted *had* one,
|
||||||
|
and that judgement belongs to the staleness checker, not to this predicate.
|
||||||
|
- **Fail-OPEN on a read error.** An unreadable binding must never SUPPRESS a real alarm; it logs and
|
||||||
|
judges anyway.
|
||||||
|
- **The anchored-verdict structure is untouched** (R-81/v0.73.0). This is the same invariant — absence
|
||||||
|
is UNKNOWN until an anchor elapses — applied one level up, at the question of whether there is a
|
||||||
|
subject at all.
|
||||||
|
|
||||||
|
**Tests, including the one that matters.** `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms`
|
||||||
|
is a real shape — a machine installed and never phoned home — and it is indistinguishable from `david`
|
||||||
|
on every signal except the one the guard reads. If the guard is ever "simplified" to key off report
|
||||||
|
presence, customer age or a name pattern, it goes red. Plus a bound-then-went-quiet case, a
|
||||||
|
deleted-host case, and the predicate's own three inputs. **Companion red-proof observed:** removing the
|
||||||
|
guard fails `TestCheckBackupDeadlines_NeverBoundHost_Silent` with `got [expected_dbdump_missed]` —
|
||||||
|
verbatim the event `david` sent three mornings running. Restored after.
|
||||||
|
|
||||||
|
**`david`'s record was not modified.** The record is correct; the alarm was what was wrong.
|
||||||
|
|
||||||
|
**Shipped alongside a spike that deliberately shipped NO code** —
|
||||||
|
`documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md` (R-193/R-192): what is one-shot
|
||||||
|
in the offsite path, and what a guest rebuild really costs. It ends in ranked options and a question
|
||||||
|
for the operator; R-196 and R-197 were minted from it and are not implemented here.
|
||||||
|
|
||||||
## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
|
## v0.91.1 — observation may only WIDEN a tier's window, never tighten it (2026-08-03, R-86 Part 2)
|
||||||
|
|
||||||
**Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
|
**Found by checking v0.91.0 against the live box before trusting it, not by review.** demo-felhom's
|
||||||
|
|||||||
@@ -0,0 +1,203 @@
|
|||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-199 (hub v0.94.0) — the box-authenticated retrieval of a host's OWN sealed identity blob.
|
||||||
|
// This is a new surface on the most sensitive object in the system; these tests exist to pin the
|
||||||
|
// three properties that make it defensible: it is self-scoped, it is honest when there is nothing to
|
||||||
|
// serve, and it is never silent.
|
||||||
|
|
||||||
|
func seedEscrowedHost(t *testing.T, st *store.Store, hostID, customerID, apiKey string, identity []byte) {
|
||||||
|
t.Helper()
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: apiKey}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("k-escrow"), "fp", "zero_knowledge", "2026-08-04T11:00:00Z", "SHA"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(identity) > 0 {
|
||||||
|
if err := st.SaveHostDRBundle(hostID, identity, `{}`); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The happy path: a host reads back exactly the bytes it uploaded, verbatim.
|
||||||
|
func TestEscrowGet_ServesOwnBlobVerbatim(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
identity := []byte("\x00\x01age-wrapped-identity-bundle\xff")
|
||||||
|
seedEscrowedHost(t, st, "h1", "c1", "HKEY", identity)
|
||||||
|
|
||||||
|
rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", "")
|
||||||
|
if rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("GET escrow = %d, want 200 (%s)", rr.Code, rr.Body.String())
|
||||||
|
}
|
||||||
|
var out struct {
|
||||||
|
HostID string `json:"host_id"`
|
||||||
|
Present bool `json:"present"`
|
||||||
|
B64 string `json:"identity_escrow_b64"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(rr.Body.Bytes(), &out); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !out.Present || out.HostID != "h1" {
|
||||||
|
t.Fatalf("unexpected envelope: %+v", out)
|
||||||
|
}
|
||||||
|
got, err := base64.StdEncoding.DecodeString(out.B64)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if string(got) != string(identity) {
|
||||||
|
t.Fatal("the served blob is not the stored blob — the hub must return ciphertext verbatim")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario C — a box can fetch ONLY its own blob.
|
||||||
|
// RED-PROOF: delete the `!isGlobal && authHostID != pathHostID` check in handleHostEscrowGet →
|
||||||
|
// cross-host retrieval succeeds → this FAILS. Without that line, any host key is a fleet-wide reader
|
||||||
|
// of every customer's sealed bundle.
|
||||||
|
func TestEscrowGet_CrossHostRefused(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
seedEscrowedHost(t, st, "hostA", "custA", "KEY-A", []byte("A-identity"))
|
||||||
|
seedEscrowedHost(t, st, "hostB", "custB", "KEY-B", []byte("B-identity-SECRET"))
|
||||||
|
|
||||||
|
rr := do(h, http.MethodGet, "/hosts/hostB/escrow", "KEY-A", "")
|
||||||
|
if rr.Code != http.StatusForbidden {
|
||||||
|
t.Fatalf("host A reading host B's blob = %d, want 403 — a cross-host read of a sealed bundle "+
|
||||||
|
"must be impossible on every code path (body: %s)", rr.Code, rr.Body.String())
|
||||||
|
}
|
||||||
|
if b := rr.Body.String(); len(b) > 0 && (contains(b, "B-identity") || contains(b, base64.StdEncoding.EncodeToString([]byte("B-identity-SECRET")))) {
|
||||||
|
t.Fatal("the refusal body leaked the other host's blob")
|
||||||
|
}
|
||||||
|
// Unauthenticated is refused too, and does not leak which hosts exist.
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/hostB/escrow", "", ""); rr.Code != http.StatusUnauthorized {
|
||||||
|
t.Fatalf("unauthenticated GET = %d, want 401", rr.Code)
|
||||||
|
}
|
||||||
|
// The global operator key MAY read any — the same asymmetry the PUT has.
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/hostB/escrow", globalKey, ""); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("global key GET = %d, want 200", rr.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func contains(hay, needle string) bool {
|
||||||
|
return len(needle) > 0 && len(hay) >= len(needle) && (func() bool {
|
||||||
|
for i := 0; i+len(needle) <= len(hay); i++ {
|
||||||
|
if hay[i:i+len(needle)] == needle {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
})()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario D — a host with no sealed bundle gets a CLEAN answer: 200 with present=false. Not a 404
|
||||||
|
// (indistinguishable from an unknown host), not an empty 200 without the flag (indistinguishable
|
||||||
|
// from a zero-length blob). Both of those read as a fault to a caller whose situation is simply
|
||||||
|
// "no ceremony has run yet".
|
||||||
|
func TestEscrowGet_NoBlobIsCleanNone(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
seedEscrowedHost(t, st, "h1", "c1", "HKEY", nil) // K-escrow only, no identity blob
|
||||||
|
|
||||||
|
rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", "")
|
||||||
|
if rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("no-blob GET = %d, want 200 with present=false", rr.Code)
|
||||||
|
}
|
||||||
|
var out struct {
|
||||||
|
Present bool `json:"present"`
|
||||||
|
B64 string `json:"identity_escrow_b64"`
|
||||||
|
}
|
||||||
|
json.Unmarshal(rr.Body.Bytes(), &out)
|
||||||
|
if out.Present || out.B64 != "" {
|
||||||
|
t.Fatalf("a host with no bundle must report present=false and no bytes, got %+v", out)
|
||||||
|
}
|
||||||
|
// An unknown host is a DIFFERENT answer — the two must not collapse into one.
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/nope/escrow", globalKey, ""); rr.Code != http.StatusNotFound {
|
||||||
|
t.Fatalf("unknown host = %d, want 404 (distinct from a known host with no bundle)", rr.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario E — every successful retrieval leaves a durable record naming the host.
|
||||||
|
// RED-PROOF: remove the SaveEvent call in handleHostEscrowGet → no event → this FAILS. A silent
|
||||||
|
// capability on this object is the whole reason §8.2's trade is acceptable at all.
|
||||||
|
func TestEscrowGet_LeavesARecord(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
seedEscrowedHost(t, st, "h1", "c1", "HKEY", []byte("age-wrapped"))
|
||||||
|
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", ""); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("GET = %d", rr.Code)
|
||||||
|
}
|
||||||
|
ev, err := st.GetLatestEventByType("c1", eventEscrowBlobServed)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if ev == nil {
|
||||||
|
t.Fatal("R-199: a sealed recovery blob was served and NOTHING recorded it — the audit row is " +
|
||||||
|
"the mitigation that makes this endpoint's trade acceptable")
|
||||||
|
}
|
||||||
|
if ev.Severity != "warning" {
|
||||||
|
t.Errorf("severity = %q, want warning: info is an intentional non-notify, so the operator would "+
|
||||||
|
"never hear that the recovery path was used", ev.Severity)
|
||||||
|
}
|
||||||
|
if !contains(ev.Message, "h1") {
|
||||||
|
t.Errorf("the record must name the host, got %q", ev.Message)
|
||||||
|
}
|
||||||
|
// The record must not carry the blob itself — it is an audit row, not a second copy.
|
||||||
|
if contains(ev.Message, "age-wrapped") || contains(ev.DetailsJSON, "age-wrapped") ||
|
||||||
|
contains(ev.DetailsJSON, base64.StdEncoding.EncodeToString([]byte("age-wrapped"))) {
|
||||||
|
t.Errorf("the audit record embedded the blob: msg=%q details=%q", ev.Message, ev.DetailsJSON)
|
||||||
|
}
|
||||||
|
// A "no blob" answer is NOT a retrieval and must not raise one.
|
||||||
|
seedEscrowedHost(t, st, "h2", "c2", "KEY2", nil)
|
||||||
|
do(h, http.MethodGet, "/hosts/h2/escrow", "KEY2", "")
|
||||||
|
if ev, _ := st.GetLatestEventByType("c2", eventEscrowBlobServed); ev != nil {
|
||||||
|
t.Fatal("a present=false answer served no blob and must raise no retrieval record")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Operator-tier by construction, registered in the same commit that mints the type.
|
||||||
|
func TestEscrowBlobServed_IsOperatorOnly(t *testing.T) {
|
||||||
|
if !notify.IsOperatorOnly(eventEscrowBlobServed) {
|
||||||
|
t.Fatalf("%s is not registered operator-only — a customer would receive operator-grade English "+
|
||||||
|
"about their sealed recovery bundle being handed out", eventEscrowBlobServed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario G — the operator-driven DR path is UNTOUCHED: same gate, same behaviour. The new sibling
|
||||||
|
// must not loosen it, and must not share it.
|
||||||
|
func TestEscrowGet_OperatorDRPathUnchanged(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
seedEscrowedHost(t, st, "h1", "c1", "HKEY", []byte("age-wrapped"))
|
||||||
|
|
||||||
|
// re-enroll without recovery mode: still refused.
|
||||||
|
if rr := do(h, http.MethodPost, "/hosts/h1/re-enroll", "HKEY", `{"new_api_key":"x"}`); rr.Code != http.StatusForbidden {
|
||||||
|
t.Fatalf("re-enroll without recovery mode = %d, want 403 — the new endpoint must not loosen it", rr.Code)
|
||||||
|
}
|
||||||
|
// restore-directive without recovery mode: still refused.
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/h1/restore-directive", "HKEY", ""); rr.Code != http.StatusForbidden {
|
||||||
|
t.Fatalf("restore-directive without recovery mode = %d, want 403", rr.Code)
|
||||||
|
}
|
||||||
|
// …and arming it still works, i.e. the old path is functional, not merely closed.
|
||||||
|
if err := st.SetRecoveryMode("h1", time.Now().UTC().Add(10*time.Minute)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/h1/restore-directive", "HKEY", ""); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("restore-directive WITH recovery mode = %d, want 200", rr.Code)
|
||||||
|
}
|
||||||
|
// The new endpoint does NOT depend on recovery mode (that is the §8.2 trade, made explicit here so
|
||||||
|
// a future change to escrowSelfServiceRetrieval is visible as a test change).
|
||||||
|
if err := st.ClearRecoveryMode("h1"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if rr := do(h, http.MethodGet, "/hosts/h1/escrow", "HKEY", ""); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("self-service escrow GET with recovery mode OFF = %d, want 200 (escrowSelfServiceRetrieval=true)", rr.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,146 @@
|
|||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-197 (v0.93.0) — a changed offsite repository key must be REPORTED. Both halves of the comparison
|
||||||
|
// (host_escrow.restic_pw_sha256 and the row it replaces) have been in the database since SLICE 3 and
|
||||||
|
// nothing read them: demo-felhom's repository password changed on 2026-08-03, orphaning 36 snapshots /
|
||||||
|
// 1.14 GB, and no event, e-mail, card or log line said so for thirteen hours.
|
||||||
|
//
|
||||||
|
// Driven through the real endpoint (PUT /hosts/<id>/escrow), not by calling the emitter, so the wiring
|
||||||
|
// is part of what is proven.
|
||||||
|
|
||||||
|
// escrowBodyWithHash is escrowBody plus the sealed-password hash and an identity blob — the fork-4
|
||||||
|
// upload shape. Hash VALUES here are test fixtures, never real secrets.
|
||||||
|
func escrowBodyWithHash(blob []byte, resticPwSHA, identity string) string {
|
||||||
|
m := map[string]string{
|
||||||
|
"blob_b64": base64.StdEncoding.EncodeToString(blob),
|
||||||
|
"key_fingerprint": "ab:cd:ef",
|
||||||
|
"posture": "zero_knowledge",
|
||||||
|
"created_at": "2026-08-04T05:00:00Z",
|
||||||
|
"restic_pw_sha256": resticPwSHA,
|
||||||
|
}
|
||||||
|
if identity != "" {
|
||||||
|
m["identity_blob_b64"] = base64.StdEncoding.EncodeToString([]byte(identity))
|
||||||
|
}
|
||||||
|
b, _ := json.Marshal(m)
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario D — a new escrow sealing a DIFFERENT repo password raises exactly one operator signal,
|
||||||
|
// naming the host, carrying NO hash value.
|
||||||
|
// RED-PROOF: remove the maybeEmitRepoKeyChanged call from handleHostEscrowPut (or make its predicate
|
||||||
|
// always fall through) → no event → this FAILS.
|
||||||
|
func TestEscrowPut_ChangedRepoKey_RaisesSignal(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||||
|
|
||||||
|
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_GEN1", "age-gen1")); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("first PUT = %d (%s)", rr.Code, rr.Body.String())
|
||||||
|
}
|
||||||
|
// Generation 1 is onboarding, not a change — nothing may fire yet.
|
||||||
|
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
|
||||||
|
t.Fatalf("a FIRST escrow must not raise %s: %+v", eventRepoKeyChanged, ev)
|
||||||
|
}
|
||||||
|
|
||||||
|
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "SHA_GEN2", "age-gen2")); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("second PUT = %d (%s)", rr.Code, rr.Body.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
ev, err := st.GetLatestEventByType("c1", eventRepoKeyChanged)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if ev == nil {
|
||||||
|
t.Fatal("R-197: the repository key demonstrably changed and NO signal was raised — this is the " +
|
||||||
|
"thirteen-hour silence that let demo-felhom lose 1.14 GB unremarked")
|
||||||
|
}
|
||||||
|
if ev.Severity != "warning" {
|
||||||
|
t.Errorf("severity = %q, want warning (info is an intentional non-notify — the operator would not be told)", ev.Severity)
|
||||||
|
}
|
||||||
|
if !strings.Contains(ev.Message, "h1") {
|
||||||
|
t.Errorf("message must name the host, got %q", ev.Message)
|
||||||
|
}
|
||||||
|
// NO HASH VALUE may travel — not in the message, not in the details.
|
||||||
|
for _, forbidden := range []string{"SHA_GEN1", "SHA_GEN2"} {
|
||||||
|
if strings.Contains(ev.Message, forbidden) || strings.Contains(ev.DetailsJSON, forbidden) {
|
||||||
|
t.Errorf("a repo-password hash value leaked into the event (%s): msg=%q details=%q", forbidden, ev.Message, ev.DetailsJSON)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Exactly one, not one per anything.
|
||||||
|
evs, _ := st.GetRecentEvents("c1", 50)
|
||||||
|
n := 0
|
||||||
|
for _, e := range evs {
|
||||||
|
if e.EventType == eventRepoKeyChanged {
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if n != 1 {
|
||||||
|
t.Fatalf("%s fired %d times for one supersession, want 1 (edge-triggered)", eventRepoKeyChanged, n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario E — an UNCHANGED key says nothing. A customer re-running a ceremony for good reasons must
|
||||||
|
// not be punished for it with an alarm.
|
||||||
|
func TestEscrowPut_UnchangedRepoKey_Silent(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||||
|
|
||||||
|
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_SAME", "age-1"))
|
||||||
|
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "SHA_SAME", "age-2"))
|
||||||
|
|
||||||
|
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
|
||||||
|
t.Fatalf("a same-hash re-ceremony must be SILENT, got: %+v", ev)
|
||||||
|
}
|
||||||
|
// And it must not have created a superseded row either (pre-existing contract, re-asserted here
|
||||||
|
// because Scenario E's silence would also be produced by the supersede branch never running).
|
||||||
|
if n, _ := st.CountSupersededEscrow("h1"); n != 0 {
|
||||||
|
t.Fatalf("same-hash re-upload created %d superseded row(s), want 0", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The in-between shapes must not be silent-and-indistinguishable either: a hash-less NEW blob cannot
|
||||||
|
// show whether the key changed, and saying nothing at all would look identical to "unchanged".
|
||||||
|
// Asserted at the level the code offers — no event (correct: nothing was measured) and no crash.
|
||||||
|
func TestEscrowPut_HashlessSupersession_NoSignal(t *testing.T) {
|
||||||
|
h, st, _ := newTestHandler(t)
|
||||||
|
st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "HKEY"})
|
||||||
|
|
||||||
|
do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k1"), "SHA_GEN1", "age-1"))
|
||||||
|
if rr := do(h, http.MethodPut, "/hosts/h1/escrow", "HKEY", escrowBodyWithHash([]byte("k2"), "", "age-2")); rr.Code != http.StatusOK {
|
||||||
|
t.Fatalf("hash-less supersession PUT = %d", rr.Code)
|
||||||
|
}
|
||||||
|
if ev, _ := st.GetLatestEventByType("c1", eventRepoKeyChanged); ev != nil {
|
||||||
|
t.Fatalf("a hash-less new blob cannot EVIDENCE a change; it must not claim one: %+v", ev)
|
||||||
|
}
|
||||||
|
// The retention still ran (the hashes differ, so this IS a supersession) — the key is kept even
|
||||||
|
// though the change cannot be evidenced.
|
||||||
|
if n, _ := st.CountSupersededEscrow("h1"); n != 1 {
|
||||||
|
t.Fatalf("superseded rows = %d, want 1", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// R-97a's lesson, pinned in the SAME commit that mints the type: an operator-tier event that is not in
|
||||||
|
// operatorOnlyEvents reaches customers as raw English, because a missing customerMessages entry is NOT
|
||||||
|
// a block. Checked here rather than in notify so the type's two registers are asserted together.
|
||||||
|
func TestRepoKeyChanged_IsOperatorOnly(t *testing.T) {
|
||||||
|
if !notify.IsOperatorOnly(eventRepoKeyChanged) {
|
||||||
|
t.Fatalf("%s is not registered operator-only — a customer would receive operator-grade English "+
|
||||||
|
"about escrow custody", eventRepoKeyChanged)
|
||||||
|
}
|
||||||
|
// The two R-192 types have the same property and the same reason (see the register's comment).
|
||||||
|
for _, et := range []string{"offsite_delivery_stuck", "offsite_credential_restaged"} {
|
||||||
|
if !notify.IsOperatorOnly(et) {
|
||||||
|
t.Errorf("%s is not registered operator-only", et)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -155,7 +155,7 @@ func TestReportACK_EscrowStatus(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// escrow row with identity blob + hash → the ACK carries all three fields
|
// escrow row with identity blob + hash → the ACK carries all three fields
|
||||||
if _, err := st.SaveHostEscrow("hv1", []byte("k-blob"), "fp", "zero_knowledge", "2026-07-09T20:00:00Z", "abc123"); err != nil {
|
if _, _, err := st.SaveHostEscrow("hv1", []byte("k-blob"), "fp", "zero_knowledge", "2026-07-09T20:00:00Z", "abc123"); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := st.SaveHostDRBundle("hv1", []byte("identity-blob"), "{}"); err != nil {
|
if err := st.SaveHostDRBundle("hv1", []byte("identity-blob"), "{}"); err != nil {
|
||||||
|
|||||||
+200
-4
@@ -239,6 +239,12 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
|||||||
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
|
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
|
||||||
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
|
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
|
||||||
h.handleHostEscrowPut(w, r, hostID)
|
h.handleHostEscrowPut(w, r, hostID)
|
||||||
|
// R-199 (v0.94.0): the box-authenticated MIRROR of the PUT above — a host reads back its own
|
||||||
|
// opaque identity blob so it can be unsealed with the customer's recovery code. Distinct from the
|
||||||
|
// operator-driven DR path in dr.go, which stays exactly as it is (see handleHostEscrowGet).
|
||||||
|
case r.Method == http.MethodGet && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/escrow"):
|
||||||
|
hostID := strings.TrimSuffix(strings.TrimPrefix(path, "/hosts/"), "/escrow")
|
||||||
|
h.handleHostEscrowGet(w, r, hostID)
|
||||||
// G1 break-glass: day-0 vaults the root@pam console credential (self-scoped host key); the
|
// G1 break-glass: day-0 vaults the root@pam console credential (self-scoped host key); the
|
||||||
// operator retrieves it via the /admin/ path (global key only).
|
// operator retrieves it via the /admin/ path (global key only).
|
||||||
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/recovery-credential"):
|
case r.Method == http.MethodPut && strings.HasPrefix(path, "/hosts/") && strings.HasSuffix(path, "/recovery-credential"):
|
||||||
@@ -1065,9 +1071,13 @@ func (h *Handler) reissueOnReenroll(cc *store.CustomerConfig) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
// F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
|
// F3 — offsite continuity: re-stage the one-time offsite password to the fresh controller (the
|
||||||
// one-time password only ever reached the OLD controller). The re-issuer resets the restic repo
|
// one-time password only ever reached the OLD controller). ⚠ CORRECTED 2026-08-04 (R-196): this
|
||||||
// password, which makes the OLD escrow blob stale — the offsite provisioner invalidates the
|
// used to claim "the re-issuer resets the restic repo password, which makes the OLD escrow blob
|
||||||
// escrow (2.3) and emits both events. Skips silently when offsite isn't provisioned/enabled.
|
// stale". It does not and cannot — the re-issuer resets the PROVIDER account password; the
|
||||||
|
// repository password is generated on the box and never leaves it except sealed under R. The
|
||||||
|
// provisioner does mark the escrow stale, but PRECAUTIONARILY (see the reasoning at
|
||||||
|
// offsite.ReissueCredentials), not because this call rotated anything.
|
||||||
|
// Skips silently when offsite isn't provisioned/enabled.
|
||||||
if h.offsiteReissuer != nil {
|
if h.offsiteReissuer != nil {
|
||||||
if err := h.offsiteReissuer(context.Background(), cc.CustomerID); err != nil {
|
if err := h.offsiteReissuer(context.Background(), cc.CustomerID); err != nil {
|
||||||
h.logger.Printf("[WARN] offsite re-issue on re-enroll for %s failed: %v", cc.CustomerID, err)
|
h.logger.Printf("[WARN] offsite re-issue on re-enroll for %s failed: %v", cc.CustomerID, err)
|
||||||
@@ -1141,7 +1151,7 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
|
|||||||
// Store the OPAQUE bytes. No decrypt path exists — the hub cannot open this. Part B (v0.60.0):
|
// Store the OPAQUE bytes. No decrypt path exists — the hub cannot open this. Part B (v0.60.0):
|
||||||
// when this upload supersedes a DIFFERENT-passphrase old blob, the old one is RETAINED (not
|
// when this upload supersedes a DIFFERENT-passphrase old blob, the old one is RETAINED (not
|
||||||
// overwritten) so its recovery-code-recoverable history survives (Viktor's data-first ruling).
|
// overwritten) so its recovery-code-recoverable history survives (Viktor's data-first ruling).
|
||||||
superseded, serr := h.store.SaveHostEscrow(pathHostID, blob, req.KeyFingerprint, req.Posture, createdAt, req.ResticPwSHA256)
|
superseded, prevPwSHA, serr := h.store.SaveHostEscrow(pathHostID, blob, req.KeyFingerprint, req.Posture, createdAt, req.ResticPwSHA256)
|
||||||
if serr != nil {
|
if serr != nil {
|
||||||
h.logger.Printf("[ERROR] Failed to store escrow for host %s: %v", pathHostID, serr)
|
h.logger.Printf("[ERROR] Failed to store escrow for host %s: %v", pathHostID, serr)
|
||||||
http.Error(w, "Internal error", http.StatusInternalServerError)
|
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||||
@@ -1157,6 +1167,10 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
|
|||||||
"A korábbi helyreállítási csomag megőrizve (új kulcs érkezett).", string(details), "hub"); eerr != nil {
|
"A korábbi helyreállítási csomag megőrizve (új kulcs érkezett).", string(details), "hub"); eerr != nil {
|
||||||
h.logger.Printf("[WARN] escrow_superseded event save failed for %s: %v", pathHostID, eerr)
|
h.logger.Printf("[WARN] escrow_superseded event save failed for %s: %v", pathHostID, eerr)
|
||||||
}
|
}
|
||||||
|
// R-197: the box's offsite DATA key demonstrably changed. Both halves of that comparison
|
||||||
|
// have been stored since SLICE 3 and nothing read them — demo-felhom's key changed on
|
||||||
|
// 2026-08-03 and nothing said so for thirteen hours.
|
||||||
|
h.maybeEmitRepoKeyChanged(host.CustomerID, pathHostID, prevPwSHA, req.ResticPwSHA256, n)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Slice 10D.1: optionally store the IDENTITY escrow blob + the non-secret DR directive alongside
|
// Slice 10D.1: optionally store the IDENTITY escrow blob + the non-secret DR directive alongside
|
||||||
@@ -1185,6 +1199,188 @@ func (h *Handler) handleHostEscrowPut(w http.ResponseWriter, r *http.Request, pa
|
|||||||
w.Write([]byte(`{"status":"ok"}`))
|
w.Write([]byte(`{"status":"ok"}`))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// eventRepoKeyChanged (R-197) — the box's offsite restic REPOSITORY password changed, proven by the
|
||||||
|
// hub's own stored hashes. Hub-internal (not in allowedEventTypes, like escrow_superseded) and
|
||||||
|
// registered operator-only in notify.operatorOnlyEvents.
|
||||||
|
const eventRepoKeyChanged = "offsite_repo_key_changed"
|
||||||
|
|
||||||
|
// eventEscrowBlobServed (R-199) — a host retrieved its own sealed identity blob. Hub-internal,
|
||||||
|
// operator-only. See handleHostEscrowGet for why every retrieval is loud.
|
||||||
|
const eventEscrowBlobServed = "escrow_blob_served"
|
||||||
|
|
||||||
|
// escrowSelfServiceRetrieval is THE SINGLE DECISION POINT for the §8.2/§8.3 trade (R-199).
|
||||||
|
//
|
||||||
|
// true (§8.2, shipped v0.94.0) — a host may read its own blob whenever it authenticates as itself.
|
||||||
|
// false (§8.3, the fallback) — the same read additionally requires operator-armed recovery mode.
|
||||||
|
//
|
||||||
|
// It is one condition on purpose: the operator may overrule the trade below, and switching must cost a
|
||||||
|
// boolean rather than a redesign. Everything else in the recovery chain is identical either way.
|
||||||
|
const escrowSelfServiceRetrieval = true
|
||||||
|
|
||||||
|
// handleHostEscrowGet serves a host its OWN opaque identity-escrow blob (R-199, v0.94.0).
|
||||||
|
//
|
||||||
|
// WHAT THIS GIVES OUT, WHY IT IS SAFE, AND WHAT IT CHANGES ABOUT WHO IS REQUIRED — recorded here so the
|
||||||
|
// next reader finds the trade rather than inferring it (the dr.go header convention).
|
||||||
|
//
|
||||||
|
// WHAT: the age-wrapped `IdentityBundle` — opaque ciphertext. It carries the offsite restic repository
|
||||||
|
// password, the tunnel token, the PBS token and the WG key. The hub stores these bytes and has no
|
||||||
|
// decrypt path; the recovery code R that opens them exists only in the customer's hands.
|
||||||
|
//
|
||||||
|
// WHY IT IS SAFE TO GIVE OUT: the blob is useless without R (age scrypt + ChaCha20-Poly1305; a wrong R
|
||||||
|
// fails closed at the KDF, never to a plausible-but-wrong bundle), and a 10-word EFF code carries ~129
|
||||||
|
// bits. The caller already authenticates as this host for its report, its desired state, its WG
|
||||||
|
// registration and its PBS token — this adds no new identity, only a new object, and it is the exact
|
||||||
|
// MIRROR of the PUT above, which is how the blob got here in the first place.
|
||||||
|
//
|
||||||
|
// WHAT IT CHANGES, STATED PLAINLY BECAUSE IT IS THE WHOLE OF THE TRADE: before this, obtaining the blob
|
||||||
|
// required the OPERATOR to arm recovery mode with the global key (dr.go). Now whoever controls a
|
||||||
|
// rebuilt box can obtain it with that box's own credential. That is a real reduction in the number of
|
||||||
|
// parties required. They still cannot open it. The mitigation is that the capability is AUDITED rather
|
||||||
|
// than silent: every successful retrieval raises an operator event (below), because a silent capability
|
||||||
|
// is the shape this project has spent two weeks removing.
|
||||||
|
//
|
||||||
|
// THE OPERATOR-DRIVEN DR PATH IS UNTOUCHED. `handleReEnroll` / `handleGetRestoreDirective` keep their
|
||||||
|
// recovery-mode gate and their global-key arming, and they serve the K-escrow and the directive as
|
||||||
|
// well. This endpoint serves ONE object to ONE authenticated owner. Do not merge them.
|
||||||
|
func (h *Handler) handleHostEscrowGet(w http.ResponseWriter, r *http.Request, pathHostID string) {
|
||||||
|
authHostID, _, isGlobal, ok := h.checkAuthHost(r)
|
||||||
|
if !ok {
|
||||||
|
http.Error(w, "Unauthorized", http.StatusUnauthorized)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if pathHostID == "" {
|
||||||
|
http.Error(w, "Missing host_id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// SELF-SCOPED: a per-host key reads only its OWN escrow. The global operator key may read any —
|
||||||
|
// the same asymmetry the PUT has. Without this line any host key is a fleet-wide blob reader.
|
||||||
|
if !isGlobal && authHostID != pathHostID {
|
||||||
|
h.logger.Printf("[WARN] escrow GET REFUSED: host %s asked for %s's blob (self-scope)", authHostID, pathHostID)
|
||||||
|
http.Error(w, "Forbidden: host_id mismatch", http.StatusForbidden)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
host, err := h.store.GetHost(pathHostID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if host == nil {
|
||||||
|
http.Error(w, "Unknown host_id", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// §8.3 fallback lives here and nowhere else.
|
||||||
|
if !escrowSelfServiceRetrieval && !host.InRecoveryMode(time.Now().UTC()) {
|
||||||
|
h.logger.Printf("[WARN] escrow GET REFUSED for %s — self-service retrieval is disabled and recovery mode is not armed", pathHostID)
|
||||||
|
http.Error(w, "Forbidden: host not in recovery mode (operator must arm it)", http.StatusForbidden)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
bundle, berr := h.store.GetHostDRBundle(pathHostID)
|
||||||
|
if berr != nil {
|
||||||
|
h.logger.Printf("[ERROR] escrow GET for %s: %v", pathHostID, berr)
|
||||||
|
http.Error(w, "Internal error", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// A host with no sealed bundle gets a CLEAN ANSWER, not a fault: 200 with present=false. A 404
|
||||||
|
// here would be indistinguishable from an unknown host, and an empty 200 without the flag would be
|
||||||
|
// indistinguishable from a zero-length blob — both read as "something is broken" to a caller whose
|
||||||
|
// situation is simply "no ceremony has run yet".
|
||||||
|
if bundle == nil || len(bundle.IdentityBlob) == 0 {
|
||||||
|
h.logger.Printf("[INFO] escrow GET for %s: no identity blob stored (no ceremony has run)", pathHostID)
|
||||||
|
writeJSON(w, http.StatusOK, map[string]any{"host_id": pathHostID, "present": false, "identity_escrow_b64": ""})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// THE MITIGATION (§8.2). Recorded BEFORE the bytes leave, so a retrieval cannot be served without
|
||||||
|
// its audit row; a save failure is logged and does NOT block the response (the blob is opaque and
|
||||||
|
// refusing it would break a recovery over an audit hiccup — but the log line always exists).
|
||||||
|
//
|
||||||
|
// SEVERITY = warning, i.e. it reaches the operator by e-mail. Retrieval is not routine today: it
|
||||||
|
// happens during a recovery and nowhere else. IF a customer-facing self-service flow ever makes it
|
||||||
|
// routine, revisit this — but revisit it deliberately, do not let it decay to info because the
|
||||||
|
// mail became annoying.
|
||||||
|
if host.CustomerID != "" {
|
||||||
|
msg := fmt.Sprintf("Recovery blob served: host %s retrieved its own sealed identity escrow (%d opaque bytes). "+
|
||||||
|
"This is the recovery path in use — the blob cannot be opened without the customer's recovery code, which the hub never holds. "+
|
||||||
|
"If no recovery is in progress on that box, investigate.", pathHostID, len(bundle.IdentityBlob))
|
||||||
|
details, _ := json.Marshal(map[string]any{
|
||||||
|
"host_id": pathHostID,
|
||||||
|
"blob_bytes": len(bundle.IdentityBlob),
|
||||||
|
"self_scope": !isGlobal,
|
||||||
|
})
|
||||||
|
if _, eerr := h.store.SaveEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub"); eerr != nil {
|
||||||
|
h.logger.Printf("[WARN] %s event save FAILED for %s (serving anyway): %v", eventEscrowBlobServed, pathHostID, eerr)
|
||||||
|
} else if h.dispatcher != nil {
|
||||||
|
go h.dispatcher.ProcessEvent(host.CustomerID, eventEscrowBlobServed, "warning", msg, string(details), "hub")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
h.logger.Printf("[WARN] escrow blob SERVED to host %s (%d opaque bytes, self_scope=%v) — recovery path in use",
|
||||||
|
pathHostID, len(bundle.IdentityBlob), !isGlobal)
|
||||||
|
writeJSON(w, http.StatusOK, map[string]any{
|
||||||
|
"host_id": pathHostID,
|
||||||
|
"present": true,
|
||||||
|
"identity_escrow_b64": base64.StdEncoding.EncodeToString(bundle.IdentityBlob),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// maybeEmitRepoKeyChanged raises ONE operator signal per supersession when the sealed offsite repo
|
||||||
|
// password demonstrably changed. Both hashes have been stored since SLICE 3 (host_escrow and, since
|
||||||
|
// v0.60.0, host_escrow_superseded) and NOTHING compared them: demo-felhom's repository password
|
||||||
|
// changed on 2026-08-03, orphaning 36 snapshots / 1.14 GB, and no event, e-mail, card or log line
|
||||||
|
// said so for thirteen hours — the comparison that eventually found it is this one
|
||||||
|
// (audits/RECON-offsite-dr-chain-2026-08-04.md, R-197).
|
||||||
|
//
|
||||||
|
// THE PREDICATE IS DELIBERATELY NARROW: both hashes known AND different. A first-ever hash (prev "")
|
||||||
|
// is onboarding, not a change; a hash-less supersession (now "") cannot show a change happened; an
|
||||||
|
// identical hash is a re-ceremony of the SAME password, which is a normal healthy act and must stay
|
||||||
|
// silent or a customer is punished for re-running a ceremony. The in-between shapes are LOGGED rather
|
||||||
|
// than dropped, so "we chose not to alarm" and "the check did not run" never look identical.
|
||||||
|
//
|
||||||
|
// SEVERITY = warning, chosen for the world v0.93.0 creates rather than the one it inherits. Before
|
||||||
|
// R-198 a changed key meant the previous history was unopenable by anyone, ever — that would have
|
||||||
|
// argued for error. From v0.93.0 the superseding ceremony RETAINS the old identity blob, so the
|
||||||
|
// previous history stays recoverable with the recovery code that sealed it: the situation is "this
|
||||||
|
// customer's off-site history now depends on an older recovery code", which is operator-actionable
|
||||||
|
// (check the orphan card, expect a fresh repository) and is not a loss. warning also routes: the
|
||||||
|
// dispatcher notifies on warning/error/critical and treats info as an intentional non-notify, and
|
||||||
|
// the whole point of this row is that the operator learns on the day.
|
||||||
|
//
|
||||||
|
// EDGE-TRIGGERED: called only from the superseded branch of the escrow PUT, i.e. once per
|
||||||
|
// supersession, never per report. No timer lives here — the dispatcher owns cooldown.
|
||||||
|
//
|
||||||
|
// NO HASH VALUE TRAVELS. The message and the details name the host, the customer and the retained
|
||||||
|
// count only. The hashes are non-reversible, but a hash is still a fingerprint of a live secret and
|
||||||
|
// this project's rule is that values do not leave the store.
|
||||||
|
func (h *Handler) maybeEmitRepoKeyChanged(customerID, hostID, prevSHA, newSHA string, retained int) {
|
||||||
|
switch {
|
||||||
|
case prevSHA == "":
|
||||||
|
h.logger.Printf("[INFO] escrow for host %s: no previous repo-password hash recorded (first hash, or a legacy hash-less blob) — repo-key-change check not applicable", hostID)
|
||||||
|
return
|
||||||
|
case newSHA == "":
|
||||||
|
h.logger.Printf("[WARN] escrow for host %s: the NEW blob carries no repo-password hash (hash-less supersession) — whether the repository key changed CANNOT be determined from the hub's data", hostID)
|
||||||
|
return
|
||||||
|
case prevSHA == newSHA:
|
||||||
|
return // same password re-sealed: a healthy re-ceremony (Scenario E — silence is correct)
|
||||||
|
}
|
||||||
|
msg := fmt.Sprintf("Offsite repository key CHANGED for host %s: the new escrow seals a different repository password than the one it replaced. "+
|
||||||
|
"The previous off-site history is no longer opened by this box's current key. The superseding blob was retained (%d held), so that history stays recoverable with the recovery code that sealed it — verify the box's off-site tier reports a repository rather than an orphan card, and expect the next backup to start a fresh history.",
|
||||||
|
hostID, retained)
|
||||||
|
details, _ := json.Marshal(map[string]any{
|
||||||
|
"host_id": hostID,
|
||||||
|
"retained_count": retained,
|
||||||
|
"repo_key": "changed", // never the hash values
|
||||||
|
})
|
||||||
|
if _, err := h.store.SaveEvent(customerID, eventRepoKeyChanged, "warning", msg, string(details), "hub"); err != nil {
|
||||||
|
h.logger.Printf("[WARN] %s event save failed for %s: %v", eventRepoKeyChanged, hostID, err)
|
||||||
|
return // audit row first: an e-mail without its event row lies (the OffsiteChecker convention)
|
||||||
|
}
|
||||||
|
h.logger.Printf("[WARN] offsite repository key CHANGED for host %s (customer %s) — previous history now depends on the superseded recovery code; %d retained blob(s)",
|
||||||
|
hostID, customerID, retained)
|
||||||
|
if h.dispatcher != nil {
|
||||||
|
go h.dispatcher.ProcessEvent(customerID, eventRepoKeyChanged, "warning", msg, string(details), "hub")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// handleHostRecoveryCredentialPut vaults a host's break-glass root@pam console credential (TASK G1).
|
// handleHostRecoveryCredentialPut vaults a host's break-glass root@pam console credential (TASK G1).
|
||||||
// SELF-SCOPED (a host key writes only its own; global may write any) — day-0 posts it with the
|
// SELF-SCOPED (a host key writes only its own; global may write any) — day-0 posts it with the
|
||||||
// host api_key. The secret is stored at rest and NEVER logged (only the username + a length are
|
// host api_key. The secret is stored at rest and NEVER logged (only the username + a length are
|
||||||
|
|||||||
@@ -336,7 +336,7 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
|||||||
midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest)
|
midnightBudapest := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, budapest)
|
||||||
sinceUTC := midnightBudapest.UTC()
|
sinceUTC := midnightBudapest.UTC()
|
||||||
|
|
||||||
var backupMissed, dbdumpMissed, skipped, deferred int
|
var backupMissed, dbdumpMissed, skipped, deferred, unbound int
|
||||||
|
|
||||||
for _, id := range customerIDs {
|
for _, id := range customerIDs {
|
||||||
// Skip nodes that are down — they already have staleness events
|
// Skip nodes that are down — they already have staleness events
|
||||||
@@ -350,6 +350,37 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── R-195: a customer with NO machine EVER bound is UNKNOWN, not missed ────────────────
|
||||||
|
//
|
||||||
|
// Both verdicts below ask "did the thing we expect every day happen?". For a customer
|
||||||
|
// that has never had a machine bound, nothing has ever been expected, so the honest
|
||||||
|
// answer is UNKNOWN — the same invariant assessBackupFreshness states above, applied one
|
||||||
|
// level up, at the question of whether there is a subject at all.
|
||||||
|
//
|
||||||
|
// The discriminator is "was a host EVER bound", NOT "has a report arrived". That is the
|
||||||
|
// case this check must not break: a box that was installed, bound, and then went silent
|
||||||
|
// has a real fault and must keep alarming. It is bound, so it is judged.
|
||||||
|
//
|
||||||
|
// WHY THIS WAS REACHABLE AT ALL, measured 2026-08-04: the down-skip above is what
|
||||||
|
// protects every other silent customer, and it reads the staleness checker's state — which
|
||||||
|
// is seeded from the `reports` table (store.GetCustomers). A customer that has NEVER
|
||||||
|
// reported appears in no report row, so it gets no staleness state at all and GetState()
|
||||||
|
// returns "" rather than "down". The skip misses exactly the customer it would most
|
||||||
|
// obviously cover, and the DB-dump half below then fires every night: `david`, a
|
||||||
|
// prospective customer whose record was created 2026-08-01 with no machine ever bound,
|
||||||
|
// e-mailed an expected_dbdump_missed ERROR at 03:00 UTC on three consecutive days.
|
||||||
|
//
|
||||||
|
// Fail-open on a read error: an unreadable binding must never SUPPRESS a real alarm.
|
||||||
|
if bound, berr := s.HasEverBoundHost(id); berr != nil {
|
||||||
|
logger.Printf("[WARN] Deadline check: failed to read host binding for %s (judging anyway): %v", id, berr)
|
||||||
|
} else if !bound {
|
||||||
|
// Visible, per the v0.73.0 Part-7 precedent below: a quiet check must never be
|
||||||
|
// indistinguishable from a check that did not run. Once daily, one line per customer.
|
||||||
|
logger.Printf("[INFO] Deadline check: %s has no host EVER bound — all deadline verdicts UNKNOWN (no alarm)", id)
|
||||||
|
unbound++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
// Backup freshness from the agent's host-report (PBS snapshots + vzdump),
|
// Backup freshness from the agent's host-report (PBS snapshots + vzdump),
|
||||||
// the authoritative offsite-backup signal post-slice-8C.
|
// the authoritative offsite-backup signal post-slice-8C.
|
||||||
reportJSON, rerr := s.GetLatestHostReportJSON(id)
|
reportJSON, rerr := s.GetLatestHostReportJSON(id)
|
||||||
@@ -423,6 +454,6 @@ func CheckBackupDeadlines(s *store.Store, staleness *StalenessChecker, onEvent E
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down)",
|
logger.Printf("[INFO] Deadline check: %d customers, %d backup missed, %d backup unknown (deferred), %d dbdump missed, %d skipped (down), %d unknown (no host ever bound)",
|
||||||
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped)
|
len(customerIDs), backupMissed, deferred, dbdumpMissed, skipped, unbound)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,148 @@
|
|||||||
|
package monitor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-195 — a customer with NO machine EVER bound must not alarm; a customer WITH one must.
|
||||||
|
//
|
||||||
|
// Origin, measured on the live hub 2026-08-04: `david` is a prospective customer whose record was
|
||||||
|
// created 2026-08-01 16:51:49 with no host ever bound (hosts=0, host_deletions=0, host_reports=0,
|
||||||
|
// reports=0). It e-mailed an `expected_dbdump_missed` ERROR at 03:00 UTC on 08-02, 08-03 and 08-04.
|
||||||
|
//
|
||||||
|
// The mechanism, established at source: the down-skip in CheckBackupDeadlines reads
|
||||||
|
// StalenessChecker.GetState(), whose map is seeded from store.GetCustomers() — a query over the
|
||||||
|
// `reports` table. A customer with zero reports is in no row, so it has no state, and GetState()
|
||||||
|
// returns "" rather than "down". The skip that protects every other silent customer misses the one
|
||||||
|
// that never reported at all.
|
||||||
|
//
|
||||||
|
// These tests pin BOTH halves. A suite that only proved the silence would pass against an
|
||||||
|
// implementation that never alarms, which is strictly worse than the defect it replaces.
|
||||||
|
|
||||||
|
// newUnboundStore creates a store holding ONE active customer and NO host row at all.
|
||||||
|
func newUnboundStore(t *testing.T) *store.Store {
|
||||||
|
t.Helper()
|
||||||
|
st, err := store.New(filepath.Join(t.TempDir(), "test.db"), log.New(io.Discard, "", 0))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("store.New: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { st.Close() })
|
||||||
|
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ck", RetrievalPassword: "p"}); err != nil {
|
||||||
|
t.Fatalf("SaveCustomerConfig: %v", err)
|
||||||
|
}
|
||||||
|
return st
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Half 1: the silence ────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_NeverBoundHost_Silent is the david case.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (observed): deleting the HasEverBoundHost guard from CheckBackupDeadlines
|
||||||
|
// makes this test fail with
|
||||||
|
//
|
||||||
|
// deadline_unbound_test.go: a customer with NO host ever bound must raise NOTHING;
|
||||||
|
// got [expected_dbdump_missed]
|
||||||
|
//
|
||||||
|
// which is verbatim the event `david` e-mailed three mornings running. Restored after.
|
||||||
|
func TestCheckBackupDeadlines_NeverBoundHost_Silent(t *testing.T) {
|
||||||
|
st := newUnboundStore(t)
|
||||||
|
// No UpsertHost, no host-report, no db_dump_completed event — nothing has ever been expected.
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if len(got) != 0 {
|
||||||
|
t.Fatalf("a customer with NO host ever bound must raise NOTHING; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHasEverBoundHost pins the predicate itself across its three inputs, because the whole
|
||||||
|
// behaviour above turns on it and a predicate that answered `false` for everything would make the
|
||||||
|
// test above pass while silencing the entire fleet.
|
||||||
|
func TestHasEverBoundHost(t *testing.T) {
|
||||||
|
st := newUnboundStore(t)
|
||||||
|
|
||||||
|
if bound, err := st.HasEverBoundHost("c1"); err != nil || bound {
|
||||||
|
t.Fatalf("no host rows → want (false,nil); got (%v,%v)", bound, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||||
|
t.Fatalf("UpsertHost: %v", err)
|
||||||
|
}
|
||||||
|
if bound, err := st.HasEverBoundHost("c1"); err != nil || !bound {
|
||||||
|
t.Fatalf("live host row → want (true,nil); got (%v,%v)", bound, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// An unknown customer is never bound — the predicate must not answer from another customer's rows.
|
||||||
|
if bound, err := st.HasEverBoundHost("nobody"); err != nil || bound {
|
||||||
|
t.Fatalf("unknown customer → want (false,nil); got (%v,%v)", bound, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Half 2: THE RED-PROOF THAT MATTERS — a bound machine must still alarm ───────────────────────
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms is the case the change could break,
|
||||||
|
// and it is a real shape: a machine that was installed and bound and never phoned home. It has a
|
||||||
|
// `hosts` row and zero reports — indistinguishable from `david` on every signal EXCEPT the one the
|
||||||
|
// guard discriminates on. If the guard is ever "simplified" to key off report presence, customer
|
||||||
|
// age or a name pattern, this test goes red.
|
||||||
|
func TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms(t *testing.T) {
|
||||||
|
st := newUnboundStore(t)
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||||
|
t.Fatalf("UpsertHost: %v", err)
|
||||||
|
}
|
||||||
|
// No host-report and no db_dump_completed — the box was bound and never said anything.
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if !has(got, "expected_dbdump_missed") {
|
||||||
|
t.Fatalf("a BOUND machine that never reported is a real fault and must still alarm; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_BoundThenWentQuiet_StillAlarms: the machine reported once, days ago,
|
||||||
|
// and stopped. Nothing about the guard may suppress that.
|
||||||
|
//
|
||||||
|
// The staleness checker's down-skip is nil here (runDeadline passes nil), which is deliberate: it
|
||||||
|
// isolates THIS guard. In production a genuinely down node is skipped by staleness and gets its own
|
||||||
|
// node_down event — that path is unchanged and is not what this test is about.
|
||||||
|
func TestCheckBackupDeadlines_BoundThenWentQuiet_StillAlarms(t *testing.T) {
|
||||||
|
st := newUnboundStore(t)
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||||
|
t.Fatalf("UpsertHost: %v", err)
|
||||||
|
}
|
||||||
|
// One host-report whose newest backup evidence is 9 days old, and a db dump that last
|
||||||
|
// completed 5 days ago (i.e. not since midnight).
|
||||||
|
report := hostReportJSON(t, [][2]string{{rfc(-9 * 24 * time.Hour), "ok"}}, nil)
|
||||||
|
if err := st.SaveHostReport("h1", "c1", []byte(report), store.HostReportDenorm{}); err != nil {
|
||||||
|
t.Fatalf("SaveHostReport: %v", err)
|
||||||
|
}
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if !has(got, "expected_backup_missed") {
|
||||||
|
t.Fatalf("a bound machine that went quiet with stale backups must still raise expected_backup_missed; got %v", got)
|
||||||
|
}
|
||||||
|
if !has(got, "expected_dbdump_missed") {
|
||||||
|
t.Fatalf("a bound machine that went quiet must still raise expected_dbdump_missed; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCheckBackupDeadlines_DeletedHost_StillJudged: the host row is gone but a tombstone remains
|
||||||
|
// (peti-felhom's live shape). The customer HAD a machine, so this check must not take over the
|
||||||
|
// judgement — it hands off to the staleness down-skip exactly as before the change.
|
||||||
|
func TestCheckBackupDeadlines_DeletedHost_StillJudged(t *testing.T) {
|
||||||
|
st := newUnboundStore(t)
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: "h1", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||||
|
t.Fatalf("UpsertHost: %v", err)
|
||||||
|
}
|
||||||
|
if err := st.DeleteHost("h1", false); err != nil {
|
||||||
|
t.Fatalf("DeleteHost: %v", err)
|
||||||
|
}
|
||||||
|
if bound, err := st.HasEverBoundHost("c1"); err != nil || !bound {
|
||||||
|
t.Fatalf("a DELETED host is still a machine that was once bound → want (true,nil); got (%v,%v)", bound, err)
|
||||||
|
}
|
||||||
|
got := runDeadline(t, st)
|
||||||
|
if !has(got, "expected_dbdump_missed") {
|
||||||
|
t.Fatalf("a customer whose host was deleted was still bound and stays judged here; got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -94,40 +94,112 @@ func (c *OffsiteDeliveryChecker) Check() {
|
|||||||
if age < stuckAfter {
|
if age < stuckAfter {
|
||||||
continue // normal convergence window
|
continue // normal convergence window
|
||||||
}
|
}
|
||||||
c.maybeEmitStuck(cfg.CustomerID, status, age)
|
emitted := c.maybeEmitStuck(cfg.CustomerID, status, age)
|
||||||
c.maybeHeal(cfg.CustomerID, status)
|
c.maybeHeal(cfg.CustomerID, status, emitted)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer.
|
// deliveryShape names the two situations the ONE stuck state actually covers. They need different
|
||||||
func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) {
|
// text and different advice, and conflating them is R-192's defect (a).
|
||||||
|
type deliveryShape string
|
||||||
|
|
||||||
|
const (
|
||||||
|
// shapeBurned — NO report since the consume carried an offbox target: the apply never persisted.
|
||||||
|
// Re-issue is the indicated action.
|
||||||
|
shapeBurned deliveryShape = "burned"
|
||||||
|
// shapeRegressed — reports since the consume DID carry an offbox target and the latest does not:
|
||||||
|
// the credential worked and the target was later lost (a guest rebuild does exactly this, R-193).
|
||||||
|
// Re-issue is NOT indicated; it treats a symptom whose cause is elsewhere.
|
||||||
|
shapeRegressed deliveryShape = "regressed"
|
||||||
|
)
|
||||||
|
|
||||||
|
func shapeOf(status offsite.DeliveryStatus) deliveryShape {
|
||||||
|
if status.OffsiteReportsSinceConsume == 0 {
|
||||||
|
return shapeBurned
|
||||||
|
}
|
||||||
|
return shapeRegressed
|
||||||
|
}
|
||||||
|
|
||||||
|
// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer. Returns
|
||||||
|
// whether it emitted, so the heal's refusal record rides the same cadence rather than inventing one.
|
||||||
|
//
|
||||||
|
// R-192 defect (a), fixed here: the message used to interpolate ReportsSinceConsume (the TOTAL) into
|
||||||
|
// a hardcoded phrase "report(s) since carry no offbox target", and never consulted
|
||||||
|
// OffsiteReportsSinceConsume — the field that says the opposite. On demo-hp it stated, daily, that
|
||||||
|
// 500 reports carried no offbox target when all 500 of them did, and prescribed Re-issue for a
|
||||||
|
// failure mode that had not occurred. The message now STATES WHAT WAS MEASURED and lets the operator
|
||||||
|
// read it; the recommendation follows the shape rather than being hardcoded.
|
||||||
|
//
|
||||||
|
// THE WINDOW IS NAMED ON PURPOSE. CountReportsOffsiteSince reads `ORDER BY id LIMIT 500` — the OLDEST
|
||||||
|
// 500 reports after the consume, not the newest — so on a long-lived customer these counts describe
|
||||||
|
// the beginning of the window and not the present. That is a real scoping defect (R-192's other half)
|
||||||
|
// and it stays OPEN because its correct shape depends on the recovery chain that is not yet
|
||||||
|
// assembled (R-199/R-200/R-201). Naming the window in the text is how it stays visible instead of
|
||||||
|
// being laundered into a confident sentence — an instrument that can silently mis-scope its results
|
||||||
|
// must say so where it reports them.
|
||||||
|
func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) bool {
|
||||||
last, err := c.store.LastEventAt(customerID, eventDeliveryStuck)
|
last, err := c.store.LastEventAt(customerID, eventDeliveryStuck)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
c.logger.Printf("[WARN] offsite-delivery: %s: cooldown read: %v", customerID, err)
|
c.logger.Printf("[WARN] offsite-delivery: %s: cooldown read: %v", customerID, err)
|
||||||
return
|
return false
|
||||||
}
|
}
|
||||||
if !last.IsZero() && c.now().Sub(last) < stuckCooldown {
|
if !last.IsZero() && c.now().Sub(last) < stuckCooldown {
|
||||||
return
|
return false
|
||||||
|
}
|
||||||
|
shape := shapeOf(status)
|
||||||
|
var msg string
|
||||||
|
switch shape {
|
||||||
|
case shapeBurned:
|
||||||
|
msg = fmt.Sprintf("Offsite delivery stuck (BURNED-credential shape): the one-time password was consumed %s ago; of the first %d report(s) after that consume, NONE carried an offbox target, and the latest report carries none either. The credential never reached a persisted apply. Re-issue delivers a fresh one.",
|
||||||
|
age.Round(time.Minute), status.ReportsSinceConsume)
|
||||||
|
default:
|
||||||
|
msg = fmt.Sprintf("Offsite delivery stuck (REGRESSED-apply shape): the one-time password was consumed %s ago; of the first %d report(s) after that consume, %d DID carry an offbox target — and the latest report carries none. The credential was applied and worked; the target was lost afterwards. Re-issue is NOT the indicated action: find what removed the offbox target (a guest rebuild does, R-193). Automatic restage is deliberately withheld for this shape. NOTE: the counts cover at most the first 500 reports after the consume, so on a long-lived box they describe the start of the window, not now (R-192, open).",
|
||||||
|
age.Round(time.Minute), status.ReportsSinceConsume, status.OffsiteReportsSinceConsume)
|
||||||
}
|
}
|
||||||
msg := fmt.Sprintf("Offsite delivery stuck: one-time password consumed %s ago and %d report(s) since carry no offbox target — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one.",
|
|
||||||
age.Round(time.Minute), status.ReportsSinceConsume)
|
|
||||||
details, _ := json.Marshal(map[string]any{
|
details, _ := json.Marshal(map[string]any{
|
||||||
"state": string(status.State),
|
"state": string(status.State),
|
||||||
"consumed_at": status.Since.UTC().Format(time.RFC3339),
|
"shape": string(shape),
|
||||||
"reports_since_consume": status.ReportsSinceConsume,
|
"consumed_at": status.Since.UTC().Format(time.RFC3339),
|
||||||
|
"reports_since_consume": status.ReportsSinceConsume,
|
||||||
|
"offsite_reports_since_consume": status.OffsiteReportsSinceConsume,
|
||||||
|
"count_window": "oldest 500 reports after consumed_at (R-192, open)",
|
||||||
})
|
})
|
||||||
c.emit(customerID, eventDeliveryStuck, "warning", msg, string(details))
|
c.emit(customerID, eventDeliveryStuck, "warning", msg, string(details))
|
||||||
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
// maybeHeal fires the R-71c self-heal when the burned-credential shape is unambiguous:
|
// maybeHeal fires the R-71c self-heal when the burned-credential shape is unambiguous:
|
||||||
// consumed ≥ stuckAfter ago, ≥ healMinReports consecutive reports since with ZERO offbox evidence,
|
// consumed ≥ stuckAfter ago, ≥ healMinReports consecutive reports since with ZERO offbox evidence,
|
||||||
// one heal per healCooldown — and the R-39(a) guard holds at act time.
|
// one heal per healCooldown — and the R-39(a) guard holds at act time.
|
||||||
func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus) {
|
//
|
||||||
|
// R-192 defect (b), fixed here: every refusal above the "not configured" line now leaves a RECORD.
|
||||||
|
// The regressed-shape branch used to be a bare `return`, so the operator received a daily e-mail with
|
||||||
|
// the wrong story, no heal, and nothing anywhere saying why the heal declined — "we chose not to act"
|
||||||
|
// and "the heal never ran" looked identical. `offsite_credential_restaged` has never fired for any
|
||||||
|
// customer, and until now that fact was indistinguishable from the checker being dead.
|
||||||
|
//
|
||||||
|
// The record is a notification_log row (the dispatcher's suppressed-operator-e-mail precedent, R-182:
|
||||||
|
// a decision not to act is written down on the channel it would have used). It rides `recordRefusal`
|
||||||
|
// — true only when the stuck event was emitted this pass — so it appears once per stuckCooldown
|
||||||
|
// beside the e-mail it explains, rather than once per monitor tick. The GUARD ITSELF IS UNCHANGED:
|
||||||
|
// the set of situations in which the heal fires is byte-for-byte what it was; only the silence is
|
||||||
|
// gone. The two conditions are split into separate branches solely so each refusal can name its own
|
||||||
|
// reason.
|
||||||
|
func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus, recordRefusal bool) {
|
||||||
if c.reissuer == nil {
|
if c.reissuer == nil {
|
||||||
|
return // no provisioner configured: the heal does not exist on this hub, so there is nothing to explain
|
||||||
|
}
|
||||||
|
if status.OffsiteReportsSinceConsume != 0 {
|
||||||
|
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
|
||||||
|
"regressed-apply shape: %d of the first %d report(s) after the consume DID carry an offbox target, so a burned credential is ruled out — a restage would treat a symptom whose cause is elsewhere. Operator's call (R-193).",
|
||||||
|
status.OffsiteReportsSinceConsume, status.ReportsSinceConsume))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if status.ReportsSinceConsume < healMinReports || status.OffsiteReportsSinceConsume != 0 {
|
if status.ReportsSinceConsume < healMinReports {
|
||||||
return // box not reporting enough, or offbox evidence exists (regressed-apply shape) → operator's call
|
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
|
||||||
|
"only %d report(s) since the consume (need %d): the box has not reported enough for the burned shape to be unambiguous.",
|
||||||
|
status.ReportsSinceConsume, healMinReports))
|
||||||
|
return
|
||||||
}
|
}
|
||||||
last, err := c.store.LastEventAt(customerID, eventCredentialRestaged)
|
last, err := c.store.LastEventAt(customerID, eventCredentialRestaged)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -146,8 +218,8 @@ func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.Del
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
if info == nil || info.ConsumedAt.IsZero() {
|
if info == nil || info.ConsumedAt.IsZero() {
|
||||||
c.logger.Printf("[INFO] offsite-delivery: %s: heal refused — secret row is now %s (R-39(a) guard: never restage over an unconsumed secret)",
|
c.recordHealRefusal(customerID, recordRefusal, fmt.Sprintf(
|
||||||
customerID, secretShape(info))
|
"R-39(a) guard: the secret row is now %s — restaging over an unconsumed secret would clobber a password the box may be about to consume.", secretShape(info)))
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
|
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
|
||||||
@@ -167,6 +239,22 @@ func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.Del
|
|||||||
string(details))
|
string(details))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// recordHealRefusal makes a decision NOT to self-heal visible. Always logs; additionally writes a
|
||||||
|
// notification_log row on the "operator" channel with status "refused" when `record` is set (the
|
||||||
|
// stuck event was emitted this pass), so the refusal sits next to the e-mail that prompted the
|
||||||
|
// question. A LogNotification failure is logged, never swallowed, and never blocks the refusal — the
|
||||||
|
// refusal is the primary effect.
|
||||||
|
func (c *OffsiteDeliveryChecker) recordHealRefusal(customerID string, record bool, reason string) {
|
||||||
|
c.logger.Printf("[INFO] offsite-delivery: %s: self-heal REFUSED — %s", customerID, reason)
|
||||||
|
if !record {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := c.store.LogNotification(customerID, eventCredentialRestaged, "warning",
|
||||||
|
"Automatic offsite credential restage was NOT performed.", "refused", reason, "operator"); err != nil {
|
||||||
|
c.logger.Printf("[WARN] offsite-delivery: %s: could not record the heal refusal: %v", customerID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func secretShape(info *store.OneTimeSecretInfo) string {
|
func secretShape(info *store.OneTimeSecretInfo) string {
|
||||||
if info == nil {
|
if info == nil {
|
||||||
return "absent"
|
return "absent"
|
||||||
|
|||||||
@@ -0,0 +1,223 @@
|
|||||||
|
package monitor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-192 (v0.93.0) — the alert must describe what was MEASURED, and a refusal to self-heal must leave a
|
||||||
|
// trace.
|
||||||
|
//
|
||||||
|
// What was wrong, quoted from the live e-mail demo-hp sent daily from 2026-08-03: "one-time password
|
||||||
|
// consumed 284h19m0s ago and 500 report(s) since carry no offbox target — the credential is likely
|
||||||
|
// burned … Re-issue delivers a fresh one." OffsiteReportsSinceConsume for that same call was 500:
|
||||||
|
// every one of those reports carried a target. The message stated the precise negation of its own
|
||||||
|
// measurement and prescribed a remedy for a failure mode that had not occurred — while the self-heal
|
||||||
|
// correctly declined, in complete silence.
|
||||||
|
|
||||||
|
// regressedFixture puts c1 into demo-hp's real shape: consumed >1h ago, reports WITH an offbox target
|
||||||
|
// after the consume, then reports WITHOUT — the latest carrying none.
|
||||||
|
//
|
||||||
|
// The report timestamps get jitter deliberately: a perfectly regular series lands exactly on
|
||||||
|
// boundaries and survives mutations that a realistic one catches.
|
||||||
|
func (h dtHarness) regressedFixture(t *testing.T, withOffsite, without int) {
|
||||||
|
t.Helper()
|
||||||
|
if err := h.st.SaveOneTimeSecret("c1", "x"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
consumed := time.Now().UTC().Add(-30 * time.Hour).Format("2006-01-02 15:04:05")
|
||||||
|
staged := time.Now().UTC().Add(-31 * time.Hour).Format("2006-01-02 15:04:05")
|
||||||
|
if err := h.st.SetOneTimeSecretTimesForTest("c1", staged, consumed); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
jitter := []int{0, 3, 1, 7, 2, 5, 11, 4}
|
||||||
|
for i := 0; i < withOffsite; i++ {
|
||||||
|
if err := h.st.SaveReport("c1", []byte(dtReportWithOffsite)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
time.Sleep(time.Duration(jitter[i%len(jitter)]) * time.Millisecond)
|
||||||
|
}
|
||||||
|
for i := 0; i < without; i++ {
|
||||||
|
if err := h.st.SaveReport("c1", []byte(dtReportNoOffsite)); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
time.Sleep(time.Duration(jitter[(i+3)%len(jitter)]) * time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h dtHarness) refusalRows(t *testing.T) []store.NotificationLogEntry {
|
||||||
|
t.Helper()
|
||||||
|
all, err := h.st.GetRecentNotifications("c1", 50)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var out []store.NotificationLogEntry
|
||||||
|
for _, n := range all {
|
||||||
|
if n.Status == "refused" {
|
||||||
|
out = append(out, n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario F — the message states the measured state, not a denied cause.
|
||||||
|
// RED-PROOF: restore the old single hardcoded sentence (interpolate ReportsSinceConsume into
|
||||||
|
// "report(s) since carry no offbox target") → the regressed assertions FAIL.
|
||||||
|
func TestDeliveryChecker_RegressedShape_MessageMatchesMeasurement(t *testing.T) {
|
||||||
|
h := newDTHarness(t, true)
|
||||||
|
h.regressedFixture(t, 6, 2)
|
||||||
|
|
||||||
|
h.checker.Check()
|
||||||
|
|
||||||
|
saved := h.savedEvents(t, "offsite_delivery_stuck")
|
||||||
|
if len(saved) != 1 {
|
||||||
|
t.Fatalf("stuck events = %d, want 1", len(saved))
|
||||||
|
}
|
||||||
|
msg := saved[0].Message
|
||||||
|
|
||||||
|
// It must NOT assert the thing the data contradicts.
|
||||||
|
if strings.Contains(msg, "carry no offbox target") || strings.Contains(msg, "likely burned") {
|
||||||
|
t.Fatalf("R-192(a): the message still asserts a burned credential on a REGRESSED shape — this is "+
|
||||||
|
"the sentence that stated the negation of its own measurement:\n%s", msg)
|
||||||
|
}
|
||||||
|
// It must name the shape it detected and the measurement behind it.
|
||||||
|
if !strings.Contains(msg, "REGRESSED") {
|
||||||
|
t.Errorf("the message must name the detected shape, got:\n%s", msg)
|
||||||
|
}
|
||||||
|
if !strings.Contains(msg, "6 DID carry an offbox target") {
|
||||||
|
t.Errorf("the message must state the measured offbox-bearing count (6), got:\n%s", msg)
|
||||||
|
}
|
||||||
|
// It must not prescribe the action for a failure mode that did not occur.
|
||||||
|
if !strings.Contains(msg, "Re-issue is NOT the indicated action") {
|
||||||
|
t.Errorf("the message must withdraw the Re-issue recommendation for this shape, got:\n%s", msg)
|
||||||
|
}
|
||||||
|
// The counting window is named, because CountReportsOffsiteSince reads the OLDEST 500 — an
|
||||||
|
// instrument that can silently mis-scope must say so where it reports (R-192's other half, open).
|
||||||
|
if !strings.Contains(msg, "500") {
|
||||||
|
t.Errorf("the message must name its counting window, got:\n%s", msg)
|
||||||
|
}
|
||||||
|
|
||||||
|
var details map[string]any
|
||||||
|
if err := json.Unmarshal([]byte(saved[0].DetailsJSON), &details); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if details["shape"] != "regressed" {
|
||||||
|
t.Errorf("details.shape = %v, want regressed", details["shape"])
|
||||||
|
}
|
||||||
|
// The field whose absence was defect (a) is now carried.
|
||||||
|
if got, ok := details["offsite_reports_since_consume"].(float64); !ok || int(got) != 6 {
|
||||||
|
t.Errorf("details must carry offsite_reports_since_consume=6, got %v", details["offsite_reports_since_consume"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The burned shape keeps its own (correct) text and its recommendation — the fix must not blunt the
|
||||||
|
// case that was always right.
|
||||||
|
func TestDeliveryChecker_BurnedShape_KeepsItsRecommendation(t *testing.T) {
|
||||||
|
h := newDTHarness(t, false)
|
||||||
|
h.burnedFixture(t, 5)
|
||||||
|
|
||||||
|
h.checker.Check()
|
||||||
|
|
||||||
|
saved := h.savedEvents(t, "offsite_delivery_stuck")
|
||||||
|
if len(saved) != 1 {
|
||||||
|
t.Fatalf("stuck events = %d, want 1", len(saved))
|
||||||
|
}
|
||||||
|
msg := saved[0].Message
|
||||||
|
if !strings.Contains(msg, "BURNED") || !strings.Contains(msg, "NONE carried an offbox target") {
|
||||||
|
t.Errorf("the burned shape must state its own measurement, got:\n%s", msg)
|
||||||
|
}
|
||||||
|
if !strings.Contains(msg, "Re-issue delivers a fresh one") {
|
||||||
|
t.Errorf("the burned shape must keep its recommendation, got:\n%s", msg)
|
||||||
|
}
|
||||||
|
var details map[string]any
|
||||||
|
_ = json.Unmarshal([]byte(saved[0].DetailsJSON), &details)
|
||||||
|
if details["shape"] != "burned" {
|
||||||
|
t.Errorf("details.shape = %v, want burned", details["shape"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario G — a refusal to self-heal leaves a durable record naming its reason.
|
||||||
|
// RED-PROOF: replace the recordHealRefusal call in the regressed branch with a bare `return`
|
||||||
|
// (production behaviour up to v0.92.0) → no row → this FAILS.
|
||||||
|
func TestDeliveryChecker_HealRefusal_LeavesARecord(t *testing.T) {
|
||||||
|
h := newDTHarness(t, true)
|
||||||
|
h.regressedFixture(t, 6, 2)
|
||||||
|
|
||||||
|
h.checker.Check()
|
||||||
|
|
||||||
|
if h.reissuer.count() != 0 {
|
||||||
|
t.Fatal("the heal must NOT fire on a regressed shape — the guard is unchanged by this fix")
|
||||||
|
}
|
||||||
|
rows := h.refusalRows(t)
|
||||||
|
if len(rows) != 1 {
|
||||||
|
t.Fatalf("R-192(b): the heal declined and left %d record(s), want 1 — 'we chose not to act' and "+
|
||||||
|
"'the heal never ran' must not look identical", len(rows))
|
||||||
|
}
|
||||||
|
r := rows[0]
|
||||||
|
if r.Channel != "operator" {
|
||||||
|
t.Errorf("refusal channel = %q, want operator (the channel the heal's own event would use)", r.Channel)
|
||||||
|
}
|
||||||
|
if !strings.Contains(r.ErrorMessage, "regressed-apply shape") {
|
||||||
|
t.Errorf("the record must name WHY, got %q", r.ErrorMessage)
|
||||||
|
}
|
||||||
|
if !strings.Contains(r.ErrorMessage, "6 of the first 8") {
|
||||||
|
t.Errorf("the record must carry the measurement it refused on, got %q", r.ErrorMessage)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The R-39(a) guard's refusal is recorded too — it was the one refusal that already logged, and it
|
||||||
|
// still must not be the only one that does.
|
||||||
|
//
|
||||||
|
// Reaching it needs the real TOCTOU, not a pre-staged row: staging BEFORE Check() makes the derive
|
||||||
|
// itself return staged_awaiting_consume and the pass never reaches the heal at all. Same technique as
|
||||||
|
// TestDeliveryChecker_R39aGuard_NeverRestagesOverUnconsumed — the operator stages mid-tick, on the
|
||||||
|
// stuck event's dispatch, which runs after the derive and before maybeHeal.
|
||||||
|
func TestDeliveryChecker_R39aRefusal_LeavesARecord(t *testing.T) {
|
||||||
|
h := newDTHarness(t, true)
|
||||||
|
h.burnedFixture(t, 5)
|
||||||
|
|
||||||
|
base := h.checker.onEvent
|
||||||
|
h.checker.onEvent = func(cid, et, sev, msg, det, src string) {
|
||||||
|
if et == "offsite_delivery_stuck" {
|
||||||
|
if err := h.st.SaveOneTimeSecret("c1", "operator-fresh"); err != nil {
|
||||||
|
t.Errorf("mid-tick stage: %v", err)
|
||||||
|
}
|
||||||
|
if err := h.st.SetOneTimeSecretTimesForTest("c1", "2026-07-23 12:00:00", ""); err != nil {
|
||||||
|
t.Errorf("mid-tick stamp: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
base(cid, et, sev, msg, det, src)
|
||||||
|
}
|
||||||
|
|
||||||
|
h.checker.Check()
|
||||||
|
|
||||||
|
if h.reissuer.count() != 0 {
|
||||||
|
t.Fatal("R-39(a): the heal must never restage over an unconsumed secret")
|
||||||
|
}
|
||||||
|
rows := h.refusalRows(t)
|
||||||
|
if len(rows) != 1 {
|
||||||
|
t.Fatalf("the R-39(a) refusal left %d record(s), want 1", len(rows))
|
||||||
|
}
|
||||||
|
if !strings.Contains(rows[0].ErrorMessage, "R-39(a)") {
|
||||||
|
t.Errorf("the record must name the guard, got %q", rows[0].ErrorMessage)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The refusal record rides the stuck event's 24h cadence: it explains the e-mail the operator just
|
||||||
|
// received, and must not accumulate one row per monitor tick.
|
||||||
|
func TestDeliveryChecker_HealRefusal_RidesTheStuckCadence(t *testing.T) {
|
||||||
|
h := newDTHarness(t, true)
|
||||||
|
h.regressedFixture(t, 6, 2)
|
||||||
|
|
||||||
|
h.checker.Check()
|
||||||
|
h.checker.Check() // the stuck event is swallowed by its cooldown; so is the refusal record
|
||||||
|
h.checker.Check()
|
||||||
|
|
||||||
|
if n := len(h.refusalRows(t)); n != 1 {
|
||||||
|
t.Fatalf("refusal records = %d after three passes, want 1 (one per stuck e-mail, not per tick)", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -102,7 +102,7 @@ func TestOffsiteStale_NeverRanEscrowAnchorWins(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
// escrow blob stored now (SaveHostEscrow stamps updated_at with datetime('now'))
|
// escrow blob stored now (SaveHostEscrow stamps updated_at with datetime('now'))
|
||||||
if _, err := st.SaveHostEscrow("h1", []byte("blob"), "fp", "zero_knowledge", time.Now().UTC().Format(time.RFC3339), "sha"); err != nil {
|
if _, _, err := st.SaveHostEscrow("h1", []byte("blob"), "fp", "zero_knowledge", time.Now().UTC().Format(time.RFC3339), "sha"); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
saveOffsiteReport(t, st, "c1", offsiteJSON(true, "escrowed", "", "", 0, 50))
|
saveOffsiteReport(t, st, "c1", offsiteJSON(true, "escrowed", "", "", 0, 50))
|
||||||
|
|||||||
@@ -421,6 +421,29 @@ var operatorOnlyEvents = map[string]bool{
|
|||||||
// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
|
// block — `FormatCustomerEmail` falls back to the raw English message. That mistake shipped
|
||||||
// once (v0.78.0) and the comment above records it.
|
// once (v0.78.0) and the comment above records it.
|
||||||
"backup_run_failures": true,
|
"backup_run_failures": true,
|
||||||
|
// R-197 (v0.93.0). "The sealed offsite repository key changed" is a custody fact about escrow
|
||||||
|
// blobs. A customer can take no action on it — the remedy is the operator's inspection of the
|
||||||
|
// off-site tier — and the text is operator-grade English naming host ids and retained-blob
|
||||||
|
// counts. Listed here in the SAME commit that mints the type: an operator-tier type that is not
|
||||||
|
// registered here reaches customers as raw English, because a missing customerMessages entry is
|
||||||
|
// NOT a block (the v0.78.0 defect recorded above).
|
||||||
|
"offsite_repo_key_changed": true,
|
||||||
|
// R-192 (v0.93.0). These two predate the register and were never added to it — a real gap, not a
|
||||||
|
// tidy-up. `offsite_delivery_stuck` is severity warning, has no customerMessages entry, and
|
||||||
|
// therefore fell through to FormatCustomerEmail's raw-English fallback: a customer whose box hit
|
||||||
|
// the stuck shape was in line for an English e-mail about one-time passwords being "likely
|
||||||
|
// burned". Measured on the live hub: notification_log holds operator rows for demo-hp and no
|
||||||
|
// customer rows — which is NOT evidence the leg is blocked (it is equally consistent with no
|
||||||
|
// configured recipient), so the register makes it structural instead of incidental. Narrowing
|
||||||
|
// only: the operator channel is untouched.
|
||||||
|
"offsite_delivery_stuck": true,
|
||||||
|
"offsite_credential_restaged": true,
|
||||||
|
// R-199 (v0.94.0). A host retrieved its own sealed recovery blob. Operator-tier by construction:
|
||||||
|
// it names host ids and opaque byte counts, the customer can take no action on it, and its whole
|
||||||
|
// purpose is that the operator sees a capability being used. Registered in the same commit that
|
||||||
|
// mints the type — an operator-tier type absent from this register reaches customers as raw
|
||||||
|
// English (the v0.78.0 defect recorded above).
|
||||||
|
"escrow_blob_served": true,
|
||||||
}
|
}
|
||||||
|
|
||||||
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
|
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
|
||||||
|
|||||||
@@ -195,35 +195,60 @@ func (p *Provisioner) ReissueCredentials(ctx context.Context, customerID, typ st
|
|||||||
return fmt.Errorf("offsite: store re-issued one-time password: %w", err)
|
return fmt.Errorf("offsite: store re-issued one-time password: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// v0.57.0 (2.3, the escrow-honesty fix): the restic repo password just changed, so any existing
|
// v0.57.0 (2.3, the escrow-honesty fix). ⚠ ITS STATED REASON WAS FALSE AND IS CORRECTED HERE
|
||||||
// key-escrow blob — which sealed the OLD password — is now STALE. A recovery code minted against
|
// (R-196, 2026-08-04). The comment used to read "the restic repo password just changed" — this
|
||||||
// it would decrypt a password that no longer opens the repo. Mark the escrow stale so the hub
|
// function does NOT change it and CANNOT: the repository password is generated on the box
|
||||||
// stops advertising "ceremony done" and the customer's escrow wizard is offered again; a fresh
|
// (controller `WriteOffboxSecrets`) and never leaves it except sealed inside the R-wrapped escrow.
|
||||||
// ceremony seals the new password and clears the flag. Every credential change also emits a
|
// What this function changes is the PROVIDER account password (①), which is a different secret at
|
||||||
// visible customer event (offsite_reissued always; escrow_stale only when a blob was invalidated).
|
// a different layer. The false premise survived because the EFFECT (a stale escrow) is real, so
|
||||||
// Best-effort: the password reset already succeeded — a bookkeeping failure here must not fail it.
|
// nobody checked the CAUSE — and it sent two separate investigations the wrong way in one day
|
||||||
escrowStaled := false
|
// (audits/SPIKE-offsite-credential-recovery-2026-08-04.md Q4).
|
||||||
if host, herr := p.Store.GetHostByCustomer(customerID); herr == nil && host != nil {
|
//
|
||||||
if esc, eerr := p.Store.GetHostEscrow(host.HostID); eerr == nil && esc != nil {
|
// ⚠ THE MARK IS GONE (R-196 / R-204 item 2, hub v0.95.0, 2026-08-05). What used to stand here —
|
||||||
if serr := p.Store.MarkEscrowStale(host.HostID); serr != nil {
|
// a `MarkEscrowStale` on every re-issue that found an escrow row, plus an `escrow_stale` customer
|
||||||
p.logf("[offsite] WARN mark-escrow-stale for %s: %v", customerID, serr)
|
// event — was PRECAUTIONARY, not evidential: it guessed that the box's re-apply MIGHT mint a fresh
|
||||||
} else {
|
// repository password. On the ordinary re-issue shape (a box that still holds its
|
||||||
escrowStaled = true
|
// `<DataDir>/offbox/repo_password`) the password does not change, so it marked a HEALTHY escrow
|
||||||
}
|
// stale. The 2026-08-04 recovery drill (R-201) is what promoted this from a nit to a blocker.
|
||||||
}
|
//
|
||||||
}
|
// WHAT THE MARK ACTUALLY DID, mechanically, because "it asked for an unnecessary ceremony"
|
||||||
|
// understates it by a lot:
|
||||||
|
// 1. `stale_at` set → `GetEscrowStatusForCustomer` WITHHOLDS `restic_pw_sha256` from the report
|
||||||
|
// ACK (store.go, the v0.57.0 rule).
|
||||||
|
// 2. With no hash, the controller's SLICE-3 auto-confirm cannot flip pending→escrowed
|
||||||
|
// (report.EscrowAutoConfirmer.Reconcile returns early on an empty hash).
|
||||||
|
// 3. `OffboxRunnable() = OffboxConfigured() && EscrowState=="escrowed"` → EVERY off-site backup
|
||||||
|
// is refused, indefinitely, on a box whose key was never in doubt.
|
||||||
|
// 4. The customer is told to re-run the recovery ceremony — which mints a NEW recovery code and
|
||||||
|
// supersedes the sealed blob. During a recovery that is the one act that would have destroyed
|
||||||
|
// the key just recovered.
|
||||||
|
// A precautionary flag that stops the data-protection it is guarding is not conservative.
|
||||||
|
//
|
||||||
|
// WHY REMOVING IT LEAVES NO GAP — the case it guessed at is MEASURED elsewhere, and better:
|
||||||
|
// • Continuous, box-side: the controller compares the ACK's sealed hash against its CURRENT
|
||||||
|
// local repo password on EVERY report ACK (`reconcileEscrowed`, the Scenario-F re-check). In
|
||||||
|
// the guest-rebuild shape — the only shape where a re-issue is followed by a fresh repository
|
||||||
|
// password — that comparison mismatches within one report cycle and raises the stale card plus
|
||||||
|
// the „create a new recovery code" CTA. It is a measurement, not a guess.
|
||||||
|
// AND THE MARK WAS BLINDING IT: by emptying the hash (step 1 above) it removed the very value
|
||||||
|
// that comparison needs, so the box could only report the hash-LESS reason, which is false.
|
||||||
|
// • Edge-triggered, hub-side: R-197's `offsite_repo_key_changed` fires on a proven hash
|
||||||
|
// difference across a supersession (api.maybeEmitRepoKeyChanged) and pages the operator.
|
||||||
|
//
|
||||||
|
// DISAGREEMENT RECORDED, per the R-96 standing rule: the task's Scenario D asks that a real key
|
||||||
|
// change "marks the escrow stale". It must NOT, and nothing here was changed to make it: the hub
|
||||||
|
// learns of a real change at the moment a supersession SEALS THE NEW PASSWORD, i.e. when the escrow
|
||||||
|
// is freshest. Marking it stale there would ask for a ceremony to fix the ceremony that just ran.
|
||||||
|
// The correct consequence at that instant is the operator alarm, which is what R-197 does.
|
||||||
|
//
|
||||||
|
// `offsite_reissued` is UNCHANGED and still always fires — the customer must still learn that the
|
||||||
|
// credential moved. Best-effort: the password reset already succeeded, so a bookkeeping failure
|
||||||
|
// here must not fail it.
|
||||||
if _, serr := p.Store.SaveEvent(customerID, "offsite_reissued", "info",
|
if _, serr := p.Store.SaveEvent(customerID, "offsite_reissued", "info",
|
||||||
"Az offsite (házon kívüli) mentési hozzáférést újra kiadtuk — az új egyszeri jelszót a vezérlő a következő frissítéskor átveszi.",
|
"Az offsite (házon kívüli) mentési hozzáférést újra kiadtuk — az új egyszeri jelszót a vezérlő a következő frissítéskor átveszi.",
|
||||||
"", "hub"); serr != nil {
|
"", "hub"); serr != nil {
|
||||||
p.logf("[offsite] WARN save offsite_reissued event for %s: %v", customerID, serr)
|
p.logf("[offsite] WARN save offsite_reissued event for %s: %v", customerID, serr)
|
||||||
}
|
}
|
||||||
if escrowStaled {
|
|
||||||
if _, serr := p.Store.SaveEvent(customerID, "escrow_stale", "warning",
|
|
||||||
"A helyreállítási kulcs-letét elavult az offsite jelszó cseréje miatt — futtasd le újra a helyreállítási szertartást (Biztonsági mentés → Helyreállítás).",
|
|
||||||
"", "hub"); serr != nil {
|
|
||||||
p.logf("[offsite] WARN save escrow_stale event for %s: %v", customerID, serr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -211,12 +211,27 @@ func TestReissue_RefusesAmbiguousLookup(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// v0.57.0 (2.3, escrow honesty) — re-issuing offsite credentials INVALIDATES the key-escrow blob:
|
// SCENARIO C (R-196 / R-204 item 2, hub v0.95.0) — re-issuing the offsite PROVIDER credential must
|
||||||
// the blob sealed the OLD repo password, so a recovery code minted against it would decrypt a
|
// LEAVE A HEALTHY ESCROW ALONE.
|
||||||
// password that no longer opens the repo. RED-PROOF (Scenario C): on pre-fix code (no MarkEscrowStale
|
//
|
||||||
// in ReissueCredentials + no stale plumbing) the hub keeps advertising the escrow as current after a
|
// THIS TEST REPLACES `TestReissue_InvalidatesEscrow`, which asserted the opposite and was wrong on a
|
||||||
// re-issue and keeps serving its restic-hash for auto-confirm — this test asserts it does NEITHER.
|
// false premise: a re-issue resets the storage-provider account password and cannot touch the restic
|
||||||
func TestReissue_InvalidatesEscrow(t *testing.T) {
|
// REPOSITORY password, which is generated on the box and never leaves it except sealed under R. The
|
||||||
|
// old assertion looked right because the EFFECT it checked (a stale escrow) was real — nobody checked
|
||||||
|
// the CAUSE. Kept as one test rather than deleted so the inversion is visible in `git log -L`.
|
||||||
|
//
|
||||||
|
// WHAT IT ASSERTS IS THE CONSEQUENCE, not the mechanism: not "MarkEscrowStale was not called" but
|
||||||
|
// "the ACK still carries the hash", because the hash is what the controller's auto-confirm needs and
|
||||||
|
// its absence is what blocked every off-site backup during the 2026-08-04 drill.
|
||||||
|
//
|
||||||
|
// RED-PROOF: restore the marking — put back
|
||||||
|
//
|
||||||
|
// if host, _ := p.Store.GetHostByCustomer(customerID); host != nil {
|
||||||
|
// if esc, _ := p.Store.GetHostEscrow(host.HostID); esc != nil { p.Store.MarkEscrowStale(host.HostID) }
|
||||||
|
// }
|
||||||
|
//
|
||||||
|
// in ReissueCredentials → es.Stale becomes true, the hash is withheld, and this test fails on both.
|
||||||
|
func TestReissue_DoesNotMarkAHealthyEscrowStale(t *testing.T) {
|
||||||
p, _, st := newTestProvisioner(t)
|
p, _, st := newTestProvisioner(t)
|
||||||
const cust = "cust-esc"
|
const cust = "cust-esc"
|
||||||
if _, err := p.ProvisionOffsite(context.Background(), cust, Input{Enabled: true, Type: "shared", QuotaGB: 10}); err != nil {
|
if _, err := p.ProvisionOffsite(context.Background(), cust, Input{Enabled: true, Type: "shared", QuotaGB: 10}); err != nil {
|
||||||
@@ -226,10 +241,9 @@ func TestReissue_InvalidatesEscrow(t *testing.T) {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
|
if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
// Before re-issue: current escrow — the hub serves the sealed hash and is NOT stale.
|
|
||||||
es, err := st.GetEscrowStatusForCustomer(cust)
|
es, err := st.GetEscrowStatusForCustomer(cust)
|
||||||
if err != nil || es == nil {
|
if err != nil || es == nil {
|
||||||
t.Fatalf("escrow status (before): %v", err)
|
t.Fatalf("escrow status (before): %v", err)
|
||||||
@@ -238,25 +252,56 @@ func TestReissue_InvalidatesEscrow(t *testing.T) {
|
|||||||
t.Fatalf("pre-reissue escrow must be current: stale=%v hash=%q", es.Stale, es.ResticPwSHA256)
|
t.Fatalf("pre-reissue escrow must be current: stale=%v hash=%q", es.Stale, es.ResticPwSHA256)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Re-issue the offsite credential — the repo password just changed under the sealed blob.
|
// Re-issue the offsite PROVIDER credential. The box still holds its repo_password file, so the
|
||||||
|
// repository password does NOT change — this is the ordinary re-issue shape.
|
||||||
if err := p.ReissueCredentials(context.Background(), cust, "shared"); err != nil {
|
if err := p.ReissueCredentials(context.Background(), cust, "shared"); err != nil {
|
||||||
t.Fatalf("reissue: %v", err)
|
t.Fatalf("reissue: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// After: the escrow is STALE and the restic-hash is WITHHELD (no auto-confirm against a dead key).
|
|
||||||
es, err = st.GetEscrowStatusForCustomer(cust)
|
es, err = st.GetEscrowStatusForCustomer(cust)
|
||||||
if err != nil || es == nil {
|
if err != nil || es == nil {
|
||||||
t.Fatalf("escrow status (after): %v", err)
|
t.Fatalf("escrow status (after): %v", err)
|
||||||
}
|
}
|
||||||
if !es.Stale {
|
if es.Stale {
|
||||||
t.Fatal("RED-PROOF: escrow must be STALE after an offsite re-issue (the hub was advertising ceremony-done against a key the repo no longer accepts)")
|
t.Fatal("a re-issue marked a HEALTHY escrow stale — this blocks every off-site backup and asks the customer for a ceremony that would supersede a good key (R-196)")
|
||||||
}
|
}
|
||||||
if es.ResticPwSHA256 != "" {
|
if es.ResticPwSHA256 != "OLDHASH" {
|
||||||
t.Fatalf("a stale escrow must WITHHOLD the restic hash to inhibit auto-confirm, got %q", es.ResticPwSHA256)
|
t.Fatalf("the sealed hash must keep flowing to the ACK so the controller can auto-confirm; got %q", es.ResticPwSHA256)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The customer is still TOLD the credential moved — removing the false alarm must not remove the
|
||||||
|
// true notice. `offsite_reissued` is untouched by this change.
|
||||||
|
ev, eerr := st.GetLatestEventByType(cust, "offsite_reissued")
|
||||||
|
if eerr != nil || ev == nil {
|
||||||
|
t.Fatalf("offsite_reissued must still fire on every re-issue (err=%v ev=%+v)", eerr, ev)
|
||||||
|
}
|
||||||
|
// …and the false alarm is gone.
|
||||||
|
if bad, _ := st.GetLatestEventByType(cust, "escrow_stale"); bad != nil {
|
||||||
|
t.Fatalf("a re-issue must not raise escrow_stale on a healthy escrow: %+v", bad)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The other half of Scenario C: the stale MECHANISM is intact and still does its job when something
|
||||||
|
// legitimately sets it. Without this, "nothing marks stale any more" could decay into "stale no longer
|
||||||
|
// works", and the next evidential caller would ship inert (the seam-built-but-never-wired shape).
|
||||||
|
func TestEscrowStaleMechanism_StillWithholdsAndClears(t *testing.T) {
|
||||||
|
_, _, st := newTestProvisioner(t)
|
||||||
|
const cust = "cust-stale-mech"
|
||||||
|
if err := st.UpsertHost(&store.Host{HostID: cust + "-01", CustomerID: cust, APIKey: "k"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("blob"), "SHA256:fp", "zero_knowledge", "2026-07-16T00:00:00Z", "OLDHASH"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := st.MarkEscrowStale(cust + "-01"); err != nil {
|
||||||
|
t.Fatalf("MarkEscrowStale: %v", err)
|
||||||
|
}
|
||||||
|
es, _ := st.GetEscrowStatusForCustomer(cust)
|
||||||
|
if es == nil || !es.Stale || es.ResticPwSHA256 != "" {
|
||||||
|
t.Fatalf("a marked escrow must read stale AND withhold the hash: %+v", es)
|
||||||
|
}
|
||||||
// A fresh ceremony (new blob sealing the new password) clears stale + serves the new hash.
|
// A fresh ceremony (new blob sealing the new password) clears stale + serves the new hash.
|
||||||
if _, err := st.SaveHostEscrow(cust+"-01", []byte("opaque-blob-2"), "SHA256:fp", "zero_knowledge", "2026-07-16T01:00:00Z", "NEWHASH"); err != nil {
|
if _, _, err := st.SaveHostEscrow(cust+"-01", []byte("blob-2"), "SHA256:fp", "zero_knowledge", "2026-07-16T01:00:00Z", "NEWHASH"); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
es, _ = st.GetEscrowStatusForCustomer(cust)
|
es, _ = st.GetEscrowStatusForCustomer(cust)
|
||||||
|
|||||||
@@ -25,10 +25,10 @@ func seedRetainedBlob(t *testing.T, st *Store, customerID string) {
|
|||||||
if err := st.UpsertHost(&Host{HostID: hostID, CustomerID: customerID, APIKey: "h"}); err != nil {
|
if err := st.UpsertHost(&Host{HostID: hostID, CustomerID: customerID, APIKey: "h"}); err != nil {
|
||||||
t.Fatalf("upsert host: %v", err)
|
t.Fatalf("upsert host: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow(hostID, []byte("A"), "fpA", "p", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("A"), "fpA", "p", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||||
t.Fatalf("escrow A: %v", err)
|
t.Fatalf("escrow A: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow(hostID, []byte("B"), "fpB", "p", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("B"), "fpB", "p", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||||
t.Fatalf("escrow B: %v", err)
|
t.Fatalf("escrow B: %v", err)
|
||||||
}
|
}
|
||||||
if err := st.DeleteHost(hostID, true); err != nil {
|
if err := st.DeleteHost(hostID, true); err != nil {
|
||||||
|
|||||||
@@ -0,0 +1,168 @@
|
|||||||
|
package store
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-198 (v0.93.0) — the retention must keep the OFF-SITE data key, not only the PBS key.
|
||||||
|
//
|
||||||
|
// WHAT WAS BROKEN, and why these tests are the ones that would have caught it: host_escrow_superseded
|
||||||
|
// shipped with `blob` (the K-escrow / PBS datastore key) and `identity_blob` was added to host_escrow
|
||||||
|
// by a later ALTER and never to the retained table. The restic REPOSITORY password lives inside
|
||||||
|
// identity_blob. So every supersession retained the whole-guest key and destroyed the off-site data
|
||||||
|
// key — the exact secret the retention exists to preserve — and the destroying act is the escrow
|
||||||
|
// ceremony a rebuilt box asks its customer to run.
|
||||||
|
//
|
||||||
|
// The pre-existing TestSaveHostEscrow_RetainsSuperseded was GREEN throughout, because it asserts the
|
||||||
|
// MECHANISM (a retained row exists, with the old K-blob) and not the CONSEQUENCE (the retained row can
|
||||||
|
// still yield a repository password). These assert the consequence.
|
||||||
|
|
||||||
|
// Scenario A — a re-escrow retains BOTH sealed keys.
|
||||||
|
// RED-PROOF: drop `identity_blob` from demoteCurrentEscrowTx's INSERT/SELECT (production behaviour up
|
||||||
|
// to v0.92.0) → the retained row's identity blob is nil → this FAILS.
|
||||||
|
func TestSaveHostEscrow_RetainsIdentityBlob(t *testing.T) {
|
||||||
|
st := newTestStore(t)
|
||||||
|
const h = "hid-1"
|
||||||
|
oldIdentity := []byte("age-blob-sealing-REPO-PASSWORD-OLD")
|
||||||
|
newIdentity := []byte("age-blob-sealing-REPO-PASSWORD-NEW")
|
||||||
|
|
||||||
|
// Generation 1: the K-escrow, then the identity blob — the real order the escrow PUT uses
|
||||||
|
// (SaveHostEscrow, then SaveHostDRBundle).
|
||||||
|
if _, _, err := st.SaveHostEscrow(h, []byte("k-blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := st.SaveHostDRBundle(h, oldIdentity, `{"gen":1}`); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Generation 2 with a DIFFERENT sealed repo password → supersede.
|
||||||
|
sup, prev, err := st.SaveHostEscrow(h, []byte("k-blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !sup {
|
||||||
|
t.Fatal("a different-passphrase upload must supersede")
|
||||||
|
}
|
||||||
|
if prev != "SHA_OLD" {
|
||||||
|
t.Fatalf("prevResticPwSHA256 = %q, want SHA_OLD (R-197 needs the replaced hash)", prev)
|
||||||
|
}
|
||||||
|
if err := st.SaveHostDRBundle(h, newIdentity, `{"gen":2}`); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
retained, err := st.ListSupersededEscrow(h)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(retained) != 1 {
|
||||||
|
t.Fatalf("retained rows = %d, want 1", len(retained))
|
||||||
|
}
|
||||||
|
// THE ASSERTION THIS WHOLE ITEM IS ABOUT: the off-site data key survived the supersession.
|
||||||
|
if retained[0].IdentityBlob == nil {
|
||||||
|
t.Fatal("R-198: the retained row carries NO identity blob — the off-site repository password " +
|
||||||
|
"was destroyed by the ceremony that was supposed to preserve it")
|
||||||
|
}
|
||||||
|
// And it is the PREVIOUS generation's blob, not the one that replaced it. This pins the ordering
|
||||||
|
// dependency named on demoteCurrentEscrowTx: the identity blob is written AFTER SaveHostEscrow, so
|
||||||
|
// the demote sees the old one. If that order ever inverts, the retained bytes would silently be
|
||||||
|
// the new blob filed under the old hash — recoverable-looking and wrong.
|
||||||
|
if !bytes.Equal(retained[0].IdentityBlob, oldIdentity) {
|
||||||
|
t.Fatalf("retained identity blob is not the PREVIOUS generation (got %q) — the demote ran after the overwrite",
|
||||||
|
retained[0].IdentityBlob)
|
||||||
|
}
|
||||||
|
if retained[0].ResticPwSHA256 != "SHA_OLD" || string(retained[0].Blob) != "k-blob-old" {
|
||||||
|
t.Fatalf("retained row is not the old generation: %+v", retained[0])
|
||||||
|
}
|
||||||
|
// Current row unchanged in behaviour: the NEW generation, both blobs.
|
||||||
|
if bundle, berr := st.GetHostDRBundle(h); berr != nil || bundle == nil || !bytes.Equal(bundle.IdentityBlob, newIdentity) {
|
||||||
|
t.Fatalf("current identity blob is not the new one: %+v (%v)", bundle, berr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario B — deleting a host demotes BOTH blobs too. demoteCurrentEscrowTx is shared by the
|
||||||
|
// re-escrow path and the host-delete path; a shared routine proven through one caller is how a fix
|
||||||
|
// gets believed on a path nobody exercised.
|
||||||
|
// RED-PROOF: fix only the re-escrow caller (e.g. carry the column in SaveHostEscrow's own SQL instead
|
||||||
|
// of in the shared routine) → this FAILS while Scenario A passes.
|
||||||
|
func TestDeleteHost_DemotesIdentityBlob(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
const hostID, cust = "hid-del", "cust-del"
|
||||||
|
identity := []byte("age-blob-sealing-REPO-PASSWORD")
|
||||||
|
if err := s.UpsertHost(&Host{HostID: hostID, CustomerID: cust, APIKey: "k"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, _, err := s.SaveHostEscrow(hostID, []byte("k-blob"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := s.SaveHostDRBundle(hostID, identity, `{}`); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.DeleteHost(hostID, true); err != nil {
|
||||||
|
t.Fatalf("DeleteHost: %v", err)
|
||||||
|
}
|
||||||
|
retained, err := s.ListSupersededEscrow(hostID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(retained) != 1 {
|
||||||
|
t.Fatalf("demoted rows = %d, want 1", len(retained))
|
||||||
|
}
|
||||||
|
if !bytes.Equal(retained[0].IdentityBlob, identity) {
|
||||||
|
t.Fatalf("R-198: host delete demoted custody WITHOUT the identity blob (got %q) — the off-site "+
|
||||||
|
"repository password was destroyed by a host delete", retained[0].IdentityBlob)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scenario C — a legacy host whose current escrow has NO identity blob supersedes cleanly. The column
|
||||||
|
// is nullable on purpose: a NOT NULL constraint here would make the fix block a ceremony, which is a
|
||||||
|
// worse failure than the one it repairs.
|
||||||
|
func TestSaveHostEscrow_SupersedesWithoutIdentityBlob(t *testing.T) {
|
||||||
|
st := newTestStore(t)
|
||||||
|
const h = "hid-legacy"
|
||||||
|
if _, _, err := st.SaveHostEscrow(h, []byte("k-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// No SaveHostDRBundle — a slice-7-era upload.
|
||||||
|
sup, prev, err := st.SaveHostEscrow(h, []byte("k-new"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("a supersession of an identity-less escrow must not fail: %v", err)
|
||||||
|
}
|
||||||
|
if !sup || prev != "SHA_OLD" {
|
||||||
|
t.Fatalf("superseded=%v prev=%q, want true/SHA_OLD", sup, prev)
|
||||||
|
}
|
||||||
|
retained, err := st.ListSupersededEscrow(h)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(retained) != 1 {
|
||||||
|
t.Fatalf("retained rows = %d, want 1", len(retained))
|
||||||
|
}
|
||||||
|
if retained[0].IdentityBlob != nil {
|
||||||
|
t.Fatalf("a legacy row must retain a NULL identity blob, got %q", retained[0].IdentityBlob)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// CountCurrentEscrowWithIdentity is the census §8.1 asks for: which hosts hold an identity blob today
|
||||||
|
// and are therefore protected from the next ceremony by this fix. Asserted rather than eyeballed,
|
||||||
|
// because the report quotes its numbers.
|
||||||
|
func TestCountCurrentEscrowWithIdentity(t *testing.T) {
|
||||||
|
st := newTestStore(t)
|
||||||
|
if _, _, err := st.SaveHostEscrow("with-id", []byte("k"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA1"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := st.SaveHostDRBundle("with-id", []byte("age-blob"), `{}`); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, _, err := st.SaveHostEscrow("without-id", []byte("k"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA2"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
total, withIdentity, err := st.CountCurrentEscrowWithIdentity()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if total != 2 || withIdentity != 1 {
|
||||||
|
t.Fatalf("census = %d/%d, want 2 total / 1 with identity", withIdentity, total)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -11,7 +11,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
|
|||||||
const h = "h1"
|
const h = "h1"
|
||||||
|
|
||||||
// 1st upload (P_old) — nothing to supersede.
|
// 1st upload (P_old) — nothing to supersede.
|
||||||
sup, err := st.SaveHostEscrow(h, []byte("blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD")
|
sup, _, err := st.SaveHostEscrow(h, []byte("blob-old"), "fp-old", "zk", "2026-07-09T00:00:00Z", "SHA_OLD")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -20,7 +20,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// 2nd upload (P_new, DIFFERENT sha) — must supersede + retain the old.
|
// 2nd upload (P_new, DIFFERENT sha) — must supersede + retain the old.
|
||||||
sup, err = st.SaveHostEscrow(h, []byte("blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
|
sup, _, err = st.SaveHostEscrow(h, []byte("blob-new"), "fp-new", "zk", "2026-07-16T00:00:00Z", "SHA_NEW")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -45,7 +45,7 @@ func TestSaveHostEscrow_RetainsSuperseded(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// 3rd upload, SAME sha as current — idempotent (re-ceremony of the same password): NO supersede row.
|
// 3rd upload, SAME sha as current — idempotent (re-ceremony of the same password): NO supersede row.
|
||||||
sup, err = st.SaveHostEscrow(h, []byte("blob-new-2"), "fp-new", "zk", "2026-07-16T01:00:00Z", "SHA_NEW")
|
sup, _, err = st.SaveHostEscrow(h, []byte("blob-new-2"), "fp-new", "zk", "2026-07-16T01:00:00Z", "SHA_NEW")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -15,10 +15,10 @@ func TestDeleteHost_DemotesEscrowNeverDestroys(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
// current escrow = SHA_A, one superseded = SHA_OLD (two uploads with different passphrases).
|
// current escrow = SHA_A, one superseded = SHA_OLD (two uploads with different passphrases).
|
||||||
if _, err := s.SaveHostEscrow(hostID, []byte("blob-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
|
if _, _, err := s.SaveHostEscrow(hostID, []byte("blob-old"), "fp", "zk", "2026-07-09T00:00:00Z", "SHA_OLD"); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := s.SaveHostEscrow(hostID, []byte("blob-A"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
|
if _, _, err := s.SaveHostEscrow(hostID, []byte("blob-A"), "fp", "zk", "2026-07-16T00:00:00Z", "SHA_A"); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if n, _ := s.CountSupersededEscrow(hostID); n != 1 {
|
if n, _ := s.CountSupersededEscrow(hostID); n != 1 {
|
||||||
|
|||||||
@@ -57,7 +57,7 @@ func seedHostWithArtifacts(t *testing.T, s *Store, hostID, customerID string) {
|
|||||||
"PK-"+hostID, "ip-"+hostID, hostID); err != nil {
|
"PK-"+hostID, "ip-"+hostID, hostID); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := s.SaveHostEscrow(hostID, []byte("opaque-escrow"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
|
if _, _, err := s.SaveHostEscrow(hostID, []byte("opaque-escrow"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+126
-20
@@ -374,6 +374,22 @@ func (s *Store) migrate() error {
|
|||||||
-- "history destroyed" into "history recoverable with the recovery code"). Append-only; the hub
|
-- "history destroyed" into "history recoverable with the recovery code"). Append-only; the hub
|
||||||
-- never decrypts; NO pruning (the blobs are tiny + R-encrypted; custody unchanged). The ACK and
|
-- never decrypts; NO pruning (the blobs are tiny + R-encrypted; custody unchanged). The ACK and
|
||||||
-- restore-serving read host_escrow (the CURRENT row) — never this table.
|
-- restore-serving read host_escrow (the CURRENT row) — never this table.
|
||||||
|
--
|
||||||
|
-- THE RULING ABOVE WAS NOT MET FOR TWO MONTHS, AND THIS IS THE RECORD OF IT (R-198, fixed
|
||||||
|
-- v0.93.0). This table shipped with the blob column — the K-escrow, i.e. the PBS datastore key
|
||||||
|
-- — and identity_blob was added to host_escrow LATER (the slice-10D ALTER below), never here.
|
||||||
|
-- The offsite restic REPOSITORY password lives in identity_blob, not in blob. So the retention
|
||||||
|
-- preserved the whole-guest key and silently dropped the off-site data key: precisely the
|
||||||
|
-- secret the reinstall-orphan incident was about. Worse, the copy happens as the new blob
|
||||||
|
-- overwrites the old, so the destroying act was the ESCROW CEREMONY — the exact thing a
|
||||||
|
-- rebuilt box asks its customer to run, on a card promising the old backups stay recoverable.
|
||||||
|
-- Both demo boxes crossed that line on 2026-08-04 (07:15:36 and 07:20:08) and their previous
|
||||||
|
-- repository passwords are unrecoverable, recovery code or not.
|
||||||
|
-- identity_blob is now carried (see demoteCurrentEscrowTx, which is still THE ONE row-copy
|
||||||
|
-- routine). Pinned by TestSaveHostEscrow_RetainsIdentityBlob and
|
||||||
|
-- TestDeleteHost_DemotesIdentityBlob — the routine is proven through BOTH of its callers,
|
||||||
|
-- because a shared routine tested through one caller is how a fix gets believed on a path
|
||||||
|
-- nobody exercised. Evidence: audits/RECON-offsite-dr-chain-2026-08-04.md §7.
|
||||||
CREATE TABLE IF NOT EXISTS host_escrow_superseded (
|
CREATE TABLE IF NOT EXISTS host_escrow_superseded (
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
host_id TEXT NOT NULL,
|
host_id TEXT NOT NULL,
|
||||||
@@ -382,7 +398,8 @@ func (s *Store) migrate() error {
|
|||||||
posture TEXT NOT NULL DEFAULT '',
|
posture TEXT NOT NULL DEFAULT '',
|
||||||
created_at DATETIME NOT NULL,
|
created_at DATETIME NOT NULL,
|
||||||
restic_pw_sha256 TEXT NOT NULL DEFAULT '',
|
restic_pw_sha256 TEXT NOT NULL DEFAULT '',
|
||||||
superseded_at DATETIME NOT NULL DEFAULT (datetime('now'))
|
superseded_at DATETIME NOT NULL DEFAULT (datetime('now')),
|
||||||
|
identity_blob BLOB
|
||||||
);
|
);
|
||||||
CREATE INDEX IF NOT EXISTS idx_host_escrow_superseded_host ON host_escrow_superseded(host_id);
|
CREATE INDEX IF NOT EXISTS idx_host_escrow_superseded_host ON host_escrow_superseded(host_id);
|
||||||
|
|
||||||
@@ -414,16 +431,28 @@ func (s *Store) migrate() error {
|
|||||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN identity_blob BLOB`)
|
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN identity_blob BLOB`)
|
||||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN directive_json TEXT NOT NULL DEFAULT '{}'`)
|
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN directive_json TEXT NOT NULL DEFAULT '{}'`)
|
||||||
|
|
||||||
|
// R-198 (v0.93.0) — the SAME column on the RETAINED table. It was added above and not here, and
|
||||||
|
// that omission is what made the retention keep the wrong key for two months (see the comment on
|
||||||
|
// host_escrow_superseded). Additive and tolerated on re-run, exactly like the lines above; it
|
||||||
|
// changes no existing row. Rows superseded BEFORE this ships were written without the identity
|
||||||
|
// blob and their source rows are already overwritten — there is nothing to backfill, and the
|
||||||
|
// v0.93.0 report records that as a looked-at fact rather than a deduction.
|
||||||
|
s.db.Exec(`ALTER TABLE host_escrow_superseded ADD COLUMN identity_blob BLOB`)
|
||||||
|
|
||||||
// SLICE 3 (escrow auto-confirm) — sha256 hex of the offsite restic repo password sealed in the
|
// SLICE 3 (escrow auto-confirm) — sha256 hex of the offsite restic repo password sealed in the
|
||||||
// identity blob. The hash of a 256-bit random secret is non-reversible/non-brute-forceable — safe to
|
// identity blob. The hash of a 256-bit random secret is non-reversible/non-brute-forceable — safe to
|
||||||
// store and serve; it lets the controller VERIFY "the escrow covers the CURRENT repo password"
|
// store and serve; it lets the controller VERIFY "the escrow covers the CURRENT repo password"
|
||||||
// instead of trusting blob-presence. NULL/'' = a legacy or password-less blob (never auto-confirms).
|
// instead of trusting blob-presence. NULL/'' = a legacy or password-less blob (never auto-confirms).
|
||||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN restic_pw_sha256 TEXT`)
|
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN restic_pw_sha256 TEXT`)
|
||||||
|
|
||||||
// v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at is set when the offsite repo
|
// v0.57.0 (2.3, escrow honesty on offsite re-issue) — stale_at marks an escrow blob that may no
|
||||||
// password is re-issued: the blob then seals a password that no longer opens the repo, so the
|
// longer cover the box's live repository password. While set, the hub stops advertising "ceremony
|
||||||
// hub must stop advertising "ceremony done" and withhold the (now non-matching) restic_pw_sha256
|
// done" and withholds restic_pw_sha256 from the auto-confirm ACK. NULL = current; a fresh ceremony
|
||||||
// from the auto-confirm ACK. NULL = current; a fresh ceremony (SaveHostEscrow) clears it.
|
// (SaveHostEscrow) clears it.
|
||||||
|
// ⚠ NOTHING SETS IT as of v0.95.0 (R-196 / R-204 item 2). The only writer was the PRECAUTIONARY
|
||||||
|
// mark on offsite re-issue, which guessed rather than measured and so blocked off-site backups on
|
||||||
|
// boxes whose key had not changed. The column and its readers stay; see MarkEscrowStale for what a
|
||||||
|
// legitimate future writer would have to prove first.
|
||||||
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN stale_at DATETIME`)
|
s.db.Exec(`ALTER TABLE host_escrow ADD COLUMN stale_at DATETIME`)
|
||||||
|
|
||||||
// dr_recipe (SPIKE-dr-recipe-2026-06-16): the secret-free DR reconstruction recipe, stored
|
// dr_recipe (SPIKE-dr-recipe-2026-06-16): the secret-free DR reconstruction recipe, stored
|
||||||
@@ -2262,6 +2291,28 @@ func (s *Store) GetHostByCustomer(customerID string) (*Host, error) {
|
|||||||
return h, err
|
return h, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// HasEverBoundHost reports whether a machine was EVER bound to this customer — a live row in
|
||||||
|
// `hosts` OR a tombstone in `host_deletions`. It answers "was anything ever expected of this
|
||||||
|
// customer", which is the question the deadline verdicts actually need (R-195).
|
||||||
|
//
|
||||||
|
// It is deliberately NOT "has a report arrived", and the distinction is the whole point: a box
|
||||||
|
// that was installed, bound, and then went silent IS bound, and its silence is a real fault that
|
||||||
|
// must keep alarming. Only a customer that never had a machine at all is UNKNOWN.
|
||||||
|
//
|
||||||
|
// `host_deletions` is included because a customer whose host was removed HAD one — the deadline
|
||||||
|
// caller reaches its down-skip for that shape, and this predicate must not quietly take over a
|
||||||
|
// judgement the staleness checker owns.
|
||||||
|
func (s *Store) HasEverBoundHost(customerID string) (bool, error) {
|
||||||
|
var n int
|
||||||
|
if err := s.db.QueryRow(
|
||||||
|
`SELECT EXISTS(SELECT 1 FROM hosts WHERE customer_id = ?)
|
||||||
|
OR EXISTS(SELECT 1 FROM host_deletions WHERE customer_id = ?)`,
|
||||||
|
customerID, customerID).Scan(&n); err != nil {
|
||||||
|
return false, err
|
||||||
|
}
|
||||||
|
return n != 0, nil
|
||||||
|
}
|
||||||
|
|
||||||
// ListHostsByCustomer returns the customer's hosts ordered by host_id (v0.47.0 — the
|
// ListHostsByCustomer returns the customer's hosts ordered by host_id (v0.47.0 — the
|
||||||
// customer page's Host tab is a LIST by design: 1 host today, N for a later HA cluster).
|
// customer page's Host tab is a LIST by design: 1 host today, N for a later HA cluster).
|
||||||
// Uses the idx_hosts_customer index.
|
// Uses the idx_hosts_customer index.
|
||||||
@@ -2507,6 +2558,11 @@ type HostEscrow struct {
|
|||||||
// StaleAt (v0.57.0, 2.3) — non-empty when the offsite password was re-issued after this blob was
|
// StaleAt (v0.57.0, 2.3) — non-empty when the offsite password was re-issued after this blob was
|
||||||
// sealed: the blob is stale (seals a password that no longer opens the repo). Cleared by a fresh ceremony.
|
// sealed: the blob is stale (seals a password that no longer opens the repo). Cleared by a fresh ceremony.
|
||||||
StaleAt string
|
StaleAt string
|
||||||
|
// IdentityBlob (R-198, v0.93.0) — the age-wrapped identity bundle, which is where the offsite
|
||||||
|
// restic REPOSITORY password lives. Populated by ListSupersededEscrow so a retained blob is
|
||||||
|
// reachable from Go at all; nil for pre-v0.93.0 retained rows and for hosts that never uploaded
|
||||||
|
// one. Opaque: useless without the customer's recovery code, which the hub never holds.
|
||||||
|
IdentityBlob []byte
|
||||||
}
|
}
|
||||||
|
|
||||||
// SaveHostEscrow stores (last-write-wins) the OPAQUE escrow blob for a host. The hub keeps the
|
// SaveHostEscrow stores (last-write-wins) the OPAQUE escrow blob for a host. The hub keeps the
|
||||||
@@ -2516,16 +2572,35 @@ type HostEscrow struct {
|
|||||||
// host_escrow_superseded before overwriting the current row (Part B, v0.60.0). A same-sha re-upload
|
// host_escrow_superseded before overwriting the current row (Part B, v0.60.0). A same-sha re-upload
|
||||||
// (idempotent re-ceremony of the same password) refreshes the current row and does NOT create a
|
// (idempotent re-ceremony of the same password) refreshes the current row and does NOT create a
|
||||||
// superseded row.
|
// superseded row.
|
||||||
|
//
|
||||||
|
// R-197 (v0.93.0): it also returns prevResticPwSHA256 — the hash the row being replaced sealed ("" when
|
||||||
|
// no row existed, or when it was a legacy hash-less blob). Both halves of "did the box's offsite DATA
|
||||||
|
// key change?" have been in this database since SLICE 3 and NOTHING compared them; demo-felhom's key
|
||||||
|
// changed on 2026-08-03 and no signal of any kind fired for thirteen hours. The comparison is the
|
||||||
|
// caller's (api.handleHostEscrowPut) because the event needs the customer id; returning the value is
|
||||||
|
// this function's part. The VALUE is a non-reversible hash of a 256-bit random secret and is never
|
||||||
|
// logged, mailed or written to a report — see the caller.
|
||||||
// demoteCurrentEscrowTx copies the host's CURRENT host_escrow row (if any) into
|
// demoteCurrentEscrowTx copies the host's CURRENT host_escrow row (if any) into
|
||||||
// host_escrow_superseded as a retained blob, inside the given tx. This is THE ONE escrow row-copy
|
// host_escrow_superseded as a retained blob, inside the given tx. This is THE ONE escrow row-copy
|
||||||
// routine (v0.60.0): SaveHostEscrow uses it to retain a superseded different-passphrase blob before
|
// routine (v0.60.0): SaveHostEscrow uses it to retain a superseded different-passphrase blob before
|
||||||
// overwriting, and DeleteHost (v0.60.1) uses it to DEMOTE the current blob to retained custody
|
// overwriting, and DeleteHost (v0.60.1) uses it to DEMOTE the current blob to retained custody
|
||||||
// instead of destroying it. Returns the number of rows copied (0 when the host has no current row).
|
// instead of destroying it. Returns the number of rows copied (0 when the host has no current row).
|
||||||
// The hub never decrypts; custody is unchanged.
|
// The hub never decrypts; custody is unchanged.
|
||||||
|
//
|
||||||
|
// R-198 (v0.93.0): `identity_blob` is copied too. It was omitted from this SELECT for two months, so
|
||||||
|
// every supersession retained the PBS datastore key and destroyed the offsite restic repository
|
||||||
|
// password — the one secret the retention exists to preserve. Copying more opaque bytes gains the hub
|
||||||
|
// NO knowledge: it still has no recovery code and no decrypt path.
|
||||||
|
//
|
||||||
|
// ORDERING THIS DEPENDS ON, stated because it is load-bearing and invisible from here: the identity
|
||||||
|
// blob is written by SaveHostDRBundle AFTER SaveHostEscrow returns (api/handler.go, the escrow PUT),
|
||||||
|
// so at demote time host_escrow still holds the OLD identity blob. If that order ever changes, this
|
||||||
|
// routine silently retains the NEW blob under the OLD blob's hash — pinned by
|
||||||
|
// TestSaveHostEscrow_RetainsIdentityBlob, which asserts the retained bytes are the previous ones.
|
||||||
func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
|
func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
|
||||||
res, err := tx.Exec(`
|
res, err := tx.Exec(`
|
||||||
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
|
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at, identity_blob)
|
||||||
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
|
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now'), identity_blob
|
||||||
FROM host_escrow WHERE host_id = ?`, hostID)
|
FROM host_escrow WHERE host_id = ?`, hostID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return 0, err
|
||||||
@@ -2533,10 +2608,10 @@ func demoteCurrentEscrowTx(tx *sql.Tx, hostID string) (int64, error) {
|
|||||||
return res.RowsAffected()
|
return res.RowsAffected()
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, err error) {
|
func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, posture, createdAt, resticPwSHA256 string) (superseded bool, prevResticPwSHA256 string, err error) {
|
||||||
tx, err := s.db.Begin()
|
tx, err := s.db.Begin()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false, err
|
return false, "", err
|
||||||
}
|
}
|
||||||
defer func() {
|
defer func() {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -2555,11 +2630,14 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
|
|||||||
exists = false
|
exists = false
|
||||||
default:
|
default:
|
||||||
err = scanErr
|
err = scanErr
|
||||||
return false, err
|
return false, "", err
|
||||||
|
}
|
||||||
|
if exists {
|
||||||
|
prevResticPwSHA256 = curSHA // R-197: the caller compares; "" = no row or a legacy hash-less blob
|
||||||
}
|
}
|
||||||
if exists && curSHA != resticPwSHA256 {
|
if exists && curSHA != resticPwSHA256 {
|
||||||
if _, err = demoteCurrentEscrowTx(tx, hostID); err != nil {
|
if _, err = demoteCurrentEscrowTx(tx, hostID); err != nil {
|
||||||
return false, err
|
return false, prevResticPwSHA256, err
|
||||||
}
|
}
|
||||||
superseded = true
|
superseded = true
|
||||||
}
|
}
|
||||||
@@ -2576,10 +2654,10 @@ func (s *Store) SaveHostEscrow(hostID string, blob []byte, keyFingerprint, postu
|
|||||||
stale_at = NULL,
|
stale_at = NULL,
|
||||||
updated_at = datetime('now')`,
|
updated_at = datetime('now')`,
|
||||||
hostID, blob, keyFingerprint, posture, createdAt, resticPwSHA256); err != nil {
|
hostID, blob, keyFingerprint, posture, createdAt, resticPwSHA256); err != nil {
|
||||||
return false, err
|
return false, prevResticPwSHA256, err
|
||||||
}
|
}
|
||||||
err = tx.Commit()
|
err = tx.Commit()
|
||||||
return superseded, err
|
return superseded, prevResticPwSHA256, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// CountSupersededEscrow returns how many retained (superseded) escrow blobs the hub holds for a host
|
// CountSupersededEscrow returns how many retained (superseded) escrow blobs the hub holds for a host
|
||||||
@@ -2594,7 +2672,7 @@ func (s *Store) CountSupersededEscrow(hostID string) (int, error) {
|
|||||||
// first. Opaque bytes — the hub never decrypts. Seeds the future guided-recovery flow (R-26).
|
// first. Opaque bytes — the hub never decrypts. Seeds the future guided-recovery flow (R-26).
|
||||||
func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
|
func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
|
||||||
rows, err := s.db.Query(`
|
rows, err := s.db.Query(`
|
||||||
SELECT host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at
|
SELECT host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at, identity_blob
|
||||||
FROM host_escrow_superseded WHERE host_id = ? ORDER BY id DESC`, hostID)
|
FROM host_escrow_superseded WHERE host_id = ? ORDER BY id DESC`, hostID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -2603,7 +2681,7 @@ func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
|
|||||||
var out []HostEscrow
|
var out []HostEscrow
|
||||||
for rows.Next() {
|
for rows.Next() {
|
||||||
var e HostEscrow
|
var e HostEscrow
|
||||||
if err := rows.Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.ResticPwSHA256, &e.UpdatedAt); err != nil {
|
if err := rows.Scan(&e.HostID, &e.Blob, &e.KeyFingerprint, &e.Posture, &e.CreatedAt, &e.ResticPwSHA256, &e.UpdatedAt, &e.IdentityBlob); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
out = append(out, e)
|
out = append(out, e)
|
||||||
@@ -2611,9 +2689,33 @@ func (s *Store) ListSupersededEscrow(hostID string) ([]HostEscrow, error) {
|
|||||||
return out, rows.Err()
|
return out, rows.Err()
|
||||||
}
|
}
|
||||||
|
|
||||||
// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3) — called when the offsite repo
|
// CountCurrentEscrowWithIdentity returns how many hosts hold a CURRENT escrow row, and how many of
|
||||||
// password is re-issued, because the blob then seals a password that no longer opens the repo. No-op
|
// those carry an identity blob — i.e. the population whose offsite repository password R-198's fix
|
||||||
// when no escrow row exists; idempotent (only stamps the first re-issue since the last ceremony; a
|
// now protects from the next ceremony, and the remainder for whom there is nothing to protect
|
||||||
|
// because no identity blob was ever uploaded. Read-only; no blob or hash leaves this call.
|
||||||
|
func (s *Store) CountCurrentEscrowWithIdentity() (total, withIdentity int, err error) {
|
||||||
|
err = s.db.QueryRow(`
|
||||||
|
SELECT COUNT(*), COALESCE(SUM(identity_blob IS NOT NULL), 0) FROM host_escrow`).Scan(&total, &withIdentity)
|
||||||
|
return total, withIdentity, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// MarkEscrowStale flags a host's escrow blob as stale (v0.57.0, 2.3).
|
||||||
|
//
|
||||||
|
// ⚠ IT HAS NO CALLER as of hub v0.95.0 (R-196 / R-204 item 2), and that is deliberate, not an
|
||||||
|
// oversight. Its ONE caller was `offsite.ReissueCredentials`, which called it on every re-issue on
|
||||||
|
// the PRECAUTIONARY grounds that the box's re-apply might mint a fresh repository password. It
|
||||||
|
// usually does not, so the call marked healthy escrows stale — and because a stale flag WITHHOLDS
|
||||||
|
// restic_pw_sha256 from the ACK (GetEscrowStatusForCustomer, below), it blocked every off-site backup
|
||||||
|
// on those boxes and asked the customer for a ceremony that would supersede a perfectly good key. The
|
||||||
|
// full reasoning, and the two measured signals that cover the real case, are at
|
||||||
|
// offsite.ReissueCredentials.
|
||||||
|
//
|
||||||
|
// KEPT, not deleted, because the FLAG is still live and correct — `stale_at` is read by the ACK, the
|
||||||
|
// operator card and the PBS-DR view, and a future EVIDENTIAL caller (one that has measured a key
|
||||||
|
// change rather than guessed at one) is the right way to set it. Pinned by
|
||||||
|
// TestReissue_DoesNotMarkAHealthyEscrowStale: if a caller reappears without that evidence, it fails.
|
||||||
|
//
|
||||||
|
// No-op when no escrow row exists; idempotent (only stamps the first mark since the last ceremony; a
|
||||||
// fresh ceremony clears stale_at via SaveHostEscrow's ON CONFLICT).
|
// fresh ceremony clears stale_at via SaveHostEscrow's ON CONFLICT).
|
||||||
func (s *Store) MarkEscrowStale(hostID string) error {
|
func (s *Store) MarkEscrowStale(hostID string) error {
|
||||||
_, err := s.db.Exec(`UPDATE host_escrow SET stale_at = datetime('now') WHERE host_id = ? AND stale_at IS NULL`, hostID)
|
_, err := s.db.Exec(`UPDATE host_escrow SET stale_at = datetime('now') WHERE host_id = ? AND stale_at IS NULL`, hostID)
|
||||||
@@ -2643,9 +2745,13 @@ type EscrowStatus struct {
|
|||||||
IdentityBlobPresent bool `json:"identity_blob_present"`
|
IdentityBlobPresent bool `json:"identity_blob_present"`
|
||||||
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
|
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
|
||||||
CreatedAt string `json:"created_at,omitempty"`
|
CreatedAt string `json:"created_at,omitempty"`
|
||||||
// Stale (v0.57.0, 2.3) — true when the offsite password was re-issued after the blob was sealed.
|
// Stale (v0.57.0, 2.3) — true when `stale_at` is stamped on the host's escrow row. ⚠ SINCE hub
|
||||||
|
// v0.95.0 (R-196 / R-204 item 2) NOTHING STAMPS IT: the one caller was the precautionary
|
||||||
|
// re-issue mark, and it was removed for marking healthy escrows stale. See MarkEscrowStale.
|
||||||
// When stale the ResticPwSHA256 is WITHHELD (emptied) so the controller cannot auto-confirm against
|
// When stale the ResticPwSHA256 is WITHHELD (emptied) so the controller cannot auto-confirm against
|
||||||
// a hash that no longer matches the live repo password — the ceremony must run again.
|
// a hash that may no longer match the live repo password — the ceremony must run again. That
|
||||||
|
// withholding is exactly why the precautionary caller had to go: it BLINDED the controller's own
|
||||||
|
// hash comparison, which is the measurement that actually detects a changed repository password.
|
||||||
Stale bool `json:"escrow_stale,omitempty"`
|
Stale bool `json:"escrow_stale,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -74,11 +74,11 @@ func seedDeletable(t *testing.T, st *store.Store, customerID string) string {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi-" + hostID, LastReportAt: &long}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi-" + hostID, LastReportAt: &long}); err != nil {
|
||||||
t.Fatalf("seed host: %v", err)
|
t.Fatalf("seed host: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||||
t.Fatalf("seed escrow A: %v", err)
|
t.Fatalf("seed escrow A: %v", err)
|
||||||
}
|
}
|
||||||
// A second save supersedes A → one RETAINED blob + one CURRENT blob before the cascade runs.
|
// A second save supersedes A → one RETAINED blob + one CURRENT blob before the cascade runs.
|
||||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||||
t.Fatalf("seed escrow B: %v", err)
|
t.Fatalf("seed escrow B: %v", err)
|
||||||
}
|
}
|
||||||
if err := st.SaveOneTimeSecret(customerID, "one-time-pw"); err != nil {
|
if err := st.SaveOneTimeSecret(customerID, "one-time-pw"); err != nil {
|
||||||
|
|||||||
@@ -37,10 +37,10 @@ func seedResettable(t *testing.T, st *store.Store, customerID string) {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi"}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: hostID, CustomerID: customerID, APIKey: "hapi"}); err != nil {
|
||||||
t.Fatalf("seed host: %v", err)
|
t.Fatalf("seed host: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("blobA"), "fpA", "posture", "2026-01-01T00:00:00Z", "shaA"); err != nil {
|
||||||
t.Fatalf("seed escrow A: %v", err)
|
t.Fatalf("seed escrow A: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
if _, _, err := st.SaveHostEscrow(hostID, []byte("blobB"), "fpB", "posture", "2026-01-02T00:00:00Z", "shaB"); err != nil {
|
||||||
t.Fatalf("seed escrow B: %v", err)
|
t.Fatalf("seed escrow B: %v", err)
|
||||||
}
|
}
|
||||||
if err := st.DeleteHost(hostID, true); err != nil { // demotes current → retained; records host_deletions
|
if err := st.DeleteHost(hostID, true); err != nil { // demotes current → retained; records host_deletions
|
||||||
|
|||||||
@@ -64,7 +64,7 @@ func TestHostDelete_EscrowAckRequired(t *testing.T) {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: "esc-host", CustomerID: "c2", APIKey: "k"}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: "esc-host", CustomerID: "c2", APIKey: "k"}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow("esc-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
if _, _, err := st.SaveHostEscrow("esc-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -110,7 +110,7 @@ func TestHostDelete_ImpactJSON(t *testing.T) {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: "imp-host", CustomerID: "c4", APIKey: "SECRET-KEY"}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: "imp-host", CustomerID: "c4", APIKey: "SECRET-KEY"}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow("imp-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
if _, _, err := st.SaveHostEscrow("imp-host", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := st.UpsertGuestFromReport(&store.Guest{GuestID: store.GuestID("imp-host", 100),
|
if err := st.UpsertGuestFromReport(&store.Guest{GuestID: store.GuestID("imp-host", 100),
|
||||||
@@ -159,7 +159,7 @@ func TestHostDelete_HappyPath(t *testing.T) {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: "dr-drill", CustomerID: "c5", APIKey: "k"}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: "dr-drill", CustomerID: "c5", APIKey: "k"}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow("dr-drill", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
if _, _, err := st.SaveHostEscrow("dr-drill", []byte("blob"), "fp", "p", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
rr := postHostDelete(t, s, "dr-drill", url.Values{
|
rr := postHostDelete(t, s, "dr-drill", url.Values{
|
||||||
|
|||||||
@@ -130,7 +130,7 @@ func TestHandleHostDetail(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
// DR + escrow present (escrow row must exist before the DR bundle UPDATE).
|
// DR + escrow present (escrow row must exist before the DR bundle UPDATE).
|
||||||
if _, err := st.SaveHostEscrow("demo-felhom-01", []byte("opaque-escrow"), "fp", "posture", "2026-06-01T00:00:00Z", ""); err != nil {
|
if _, _, err := st.SaveHostEscrow("demo-felhom-01", []byte("opaque-escrow"), "fp", "posture", "2026-06-01T00:00:00Z", ""); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := st.SaveHostDRBundle("demo-felhom-01", []byte("opaque-identity"), `{"v":1}`); err != nil {
|
if err := st.SaveHostDRBundle("demo-felhom-01", []byte("opaque-identity"), `{"v":1}`); err != nil {
|
||||||
|
|||||||
@@ -404,7 +404,7 @@ func TestPBSDR_F14AutoReissueOnAckedDeletion(t *testing.T) {
|
|||||||
if err := st.UpsertHost(&store.Host{HostID: "peti-00-dead", CustomerID: "peti", APIKey: "oldkey"}); err != nil {
|
if err := st.UpsertHost(&store.Host{HostID: "peti-00-dead", CustomerID: "peti", APIKey: "oldkey"}); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if _, err := st.SaveHostEscrow("peti-00-dead", []byte("opaque"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
|
if _, _, err := st.SaveHostEscrow("peti-00-dead", []byte("opaque"), "fp", "posture", "2026-07-01T00:00:00Z", ""); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if err := st.DeleteHost("peti-00-dead", true); err != nil {
|
if err := st.DeleteHost("peti-00-dead", true); err != nil {
|
||||||
|
|||||||
+1
-1
@@ -125,7 +125,7 @@ spec:
|
|||||||
spec:
|
spec:
|
||||||
containers:
|
containers:
|
||||||
- name: hub
|
- name: hub
|
||||||
image: gitea.dooplex.hu/admin/felhom-hub:0.91.1
|
image: gitea.dooplex.hu/admin/felhom-hub:0.94.0
|
||||||
ports:
|
ports:
|
||||||
- containerPort: 8080
|
- containerPort: 8080
|
||||||
name: http
|
name: http
|
||||||
|
|||||||
@@ -324,7 +324,7 @@ spec:
|
|||||||
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
||||||
args:
|
args:
|
||||||
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
||||||
- --ref=installer-v1.24.0
|
- --ref=installer-v1.25.0
|
||||||
- --root=/git-scripts
|
- --root=/git-scripts
|
||||||
- --link=current
|
- --link=current
|
||||||
- --period=30s
|
- --period=30s
|
||||||
@@ -369,7 +369,7 @@ spec:
|
|||||||
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
image: registry.k8s.io/git-sync/git-sync:v4.4.0
|
||||||
args:
|
args:
|
||||||
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
- --repo=https://gitea.dooplex.hu/admin/felhom.eu.git
|
||||||
- --ref=installer-v1.24.0
|
- --ref=installer-v1.25.0
|
||||||
- --root=/git-scripts
|
- --root=/git-scripts
|
||||||
- --link=current
|
- --link=current
|
||||||
- --one-time
|
- --one-time
|
||||||
|
|||||||
Reference in New Issue
Block a user