diff --git a/REPORT.md b/REPORT.md index 0fd7919..db8be22 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,329 +1,121 @@ -# REPORT — R-193/R-192 spike (no code) + R-195 phantom-customer alarm (hub v0.92.0) +# REPORT — RECON: can a customer actually get their backups back? (2026-08-04) -**Date:** 2026-08-04 · **Class:** spike (Parts 0–3) + one fenced implementation (Part 4) -**Boundary held:** the spike shipped **no production code for R-193 or R-192**, and Part 4 never -touched the spike's subject matter. +**Class:** recon / spike. **No production code. No build, no deploy, no version bump.** +**Baselines re-read on arrival:** `felhom-agent` `856a127cd60b` (v0.124.1) · `felhom-controller` +`0887fd676d60` (v0.194.0) · `felhom.eu` `d26f49ad680e` (hub v0.92.0). All three trees clean. -> Supersedes the earlier 2026-08-04 report (R-191 + R-190), per the overwrite convention. Nothing -> durable from it lived only here — it is recorded in `hub/CHANGELOG.md`, `CONTEXT.md` S-24/S-25 and -> `OPEN-ITEMS.md` R-190/R-191. +> Supersedes the earlier 2026-08-04 report (R-193/R-192 spike + R-195), per the overwrite convention. +> Nothing durable from it lived only here — it is recorded in `hub/CHANGELOG.md`, `OPEN-ITEMS.md` +> R-192/R-193/R-195/R-196/R-197 and `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`. + +**Deliverable:** `documentation/audits/RECON-offsite-dr-chain-2026-08-04.md` — Q1–Q8 with method, +measurement and ruling per question, the eleven-link chain table, the drill design, every query used, +and the operator's open decisions. --- -## 1. Confirmed baselines, re-read at the cited commit +## 1. What was established -| Repo | `main` @ commit on arrival | Version | End state | -|---|---|---|---| -| `felhom.eu` | `f456835bbcc4` ✔ matches the spec | hub `v0.91.1` (deployed `felhom-hub:0.91.1`) | **`v0.92.0`** built, pushed, manifest bumped, synced, **live** | -| `felhom-controller` | `0887fd676d60` | — | **untouched** (read-only) | -| `felhom-agent` | `856a127` | — | **untouched** (read-only) | +**The chain from a dead node to an open repository is not assembled, and the failure is deeper than +"some links are manual".** -Clean-tree gate passed before the build: `git status --porcelain` empty, `HEAD == origin/main`. -Highest register ID on arrival was **R-194**; grepped before minting. +1. **Links 1–4 are PROVEN-LIVE** — mint → stage to the agent → seal under R → store on the hub. +2. **Link 5 is lossy, and this is the finding of the session.** `host_escrow_superseded` has **no + `identity_blob` column**, and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`) + copies only the K-escrow `blob`. The **restic repository password lives in `identity_blob`** — so + the retention feature built for the reinstall-orphan incident does not retain the secret that + incident is about. The **next escrow ceremony destroys the last copy**, and that ceremony is exactly + what the system tells a rebuilt box's customer to run. Both demo boxes crossed that line on + 2026-08-04 at 07:15:36 and 07:20:08. → **R-198** +3. **Link 6 has no client anywhere.** `handleReEnroll` / `handleGetRestoreDirective` serve the blobs; + zero callers in the agent, the hub UI, any script or any runbook. Retrieval is `sqlite3` by hand on + the hub pod. → **R-199** +4. **Links 7–8 are manual-only.** `UnwrapIdentityBundle`'s only production caller is + `--selftest=identity-consume`, reading R from `FELHOM_RECOVERY_CODE`; nothing extracts + `restic_repo_password` from the recovered bundle. +5. **Link 9's form does not exist.** `POST /backup/offbox/inject-password` is routed, handled and + unit-tested; **no template contains it.** → **R-200** +6. **Link 11 has never happened.** No blob served, no fork-4 bundle unsealed with a real R outside a + unit test, no recovered password injected, no repo reopened, no restore. The one live round-trip + proof (2026-06-10) **predates the `ResticRepoPassword` field by a month**. → **R-201** ---- +**And the most useful thing settled: a fail-closed refusal IS implementable.** The hub already sends +every box `escrow{identity_blob_present, restic_pw_sha256, created_at}` on every report ACK +(`api/handler.go:504-510`), and the controller **discards it** whenever no offbox target exists +(`report/escrow_confirm.go:75-84`). Persisting it needs no new hub API and no new secret; the +`ClaimSync` set-only pattern is the precedent. -## 2. Q1–Q8 — each answered or explicitly marked unmeasured +**The operator's ruled recovery-screen shape is buildable** and is priced row by row in §9 of the +findings doc: fresh-install signal exists (the mint branch's own `os.Stat`); hub-has-a-blob exists on +the wire; the preview is cheap and genuinely read-only (`restic snapshots --json` + +`stats --mode raw-data --json` are already the box's own counters); the one real cost is crossing +agent→controller for the unseal, because the controller image ships no `age` — one new agent local-API +endpoint mirroring `/escrow/ceremony/claim`, plus a narrow hub `GET /hosts//escrow`. -Full method, quotes and query output: **`documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md`**. +## 2. Corrections owed and made -### Q2 and Q3 first — they decide whether the rest matters +- **Yesterday's spike annotated, not rewritten** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`): + a dated block at the top plus an inline note on Q8(b). Its candidate (b) verdict *"not implementable + against a zero-knowledge escrow"* is **too strong** — unattended recovery is impossible; a + customer-present recovery is not. Everything else in that spike stands, and its Q3 remains unmeasured. +- **Capability map corrected** (`architecture/00-capability-map.md`, escrow-ceremony row): the hub + v0.60.0 retention claim *"a previous passphrase stays recoverable with its recovery code"* is struck + through and replaced with what the code does — K-escrow only, no read path, `ListSupersededEscrow` + has zero production callers. +- **The same false promise is made to the customer in Hungarian** on the orphan card + (`controller/internal/web/templates/backups_remote.html:66,69`). Recorded in R-198; **not edited** — + this session ships no controller change. +- **The prompt's Q4 premise is a conflation, and it is corrected in the doc:** `escrow_confirm.go` + **does** fire on a mint (pending branch, hash mismatch, one WARN). The "wired to the wrong path" + finding was about the hub's `escrow_stale` event (R-196), a different mechanism. What is missing is + not the detection but the consequence — the warning reaches a log file and prescribes the act that + closes the recovery window. -**Q2 — does a rebuilt controller reuse the escrowed repository password, or mint a new one?** -**RULING: it MINTS A NEW ONE.** -*Method — source:* `ConfigureOffbox` → `ApplyOffsiteTarget` → `WriteOffboxSecrets`, which generates a -fresh 256-bit password whenever `/offbox/repo_password` is absent -(`felhom-controller/controller/internal/backup/offbox.go:392-401`). A rebuilt guest has a fresh data -dir. The only recovery path, `InjectOffboxPassword`, has **exactly one caller in the entire repo** -(census: `grep -rn "InjectOffboxPassword" --include=*.go .` → 3 hits, 1 definition, 1 caller, 1 -comment) — `web/offbox_handlers.go:189`, a web form a human pastes into. **The apply-bridge never -calls it.** -*Method — live, without touching a box:* `host_escrow.restic_pw_sha256` vs -`host_escrow_superseded.restic_pw_sha256`. **demo-hp `8e03eddf…` → `8a9e33aa…`; demo-felhom -`48741892…` → `c60c8bc7…`. Both CHANGED.** -**This is bigger than R-193 as filed:** demo-felhom, which R-193 records as having survived by luck, -had **36 snapshots / 1,136,685,919 B** before its rebuild and has reported `snapshot_count: 0` in all -109 reports since. Its luck restored **delivery**, not the **repository**. +## 3. Register -**Q3 — what did the 04:15 run report?** **UNMEASURED, and the binary in the question is wrong.** -The run is daily at ~02:15 UTC (measured `last_run` values on both boxes). Neither box could run on -2026-08-04: demo-hp had no target at all until 07:15:47 UTC; demo-felhom's target was -`escrow_state: pending` from 08-03 07:19:10 to 08-04 07:20:13, which `OffboxRunnable` blocks -(configured **AND** escrowed). **The decisive run is 2026-08-05 ~02:15 UTC.** Source predicts a -**third** outcome, neither 15 nor 1: same account + same repo path (`u629488-sub3:/home/felhom-repo`; -`repoPath` is a compile-time constant) + new password + `claimed: 1` ⇒ `ensureOffboxRepo` classifies -**orphaned** → `ErrOffboxOrphaned` → **the run refuses and shows the orphan card.** Recorded as a -prediction, not a measurement. +- **New: R-198, R-199, R-200, R-201** (highest previous ID was R-197; grepped across `OPEN-ITEMS.md`, + `ROADMAP.md`, `STATUS.md`, `CONTEXT.md` before minting). +- **Updated: R-193** (seven-part addendum: (b) overturned in part, the chain census, the retention + finding, fail-closed implementable, the operator ruling priced, the drill, Q3 still unmeasured) and + **R-192** (two inputs, no verdict change). +- **`STATUS.md`** updated in operator language: two new "what's broken" entries, three new + "waiting on you" items, a "changed since" line. -### The rest +## 4. Live measurements taken (read-only) -- **Q1 — what is one-shot.** Full table in the spike. Headline: the hub's one-time provider password - is one-shot **but freely re-issuable**; the box's SFTP key is **regenerated on every apply by - design**; the **restic repository password** is the one nothing can restage — the agent's own source - calls it *"irreplaceable (unlike the SFTP access key…)"* - (`felhom-agent/internal/escrow/identity.go:35-39`). **R-193 was filed against the recoverable - secret; the damage is in the irreplaceable one.** -- **Q4 — see §3 below.** -- **Q5 — why one box recovered and the other not. CONFIRMED FROM THE LEDGER.** demo-felhom's - `one_time_secrets` row was created `2026-07-21 08:29:29` (traceable to a Re-issue at that exact - timestamp) and sat **UNCONSUMED for 13 days** until the rebuild consumed it at - `2026-08-03 07:17:58`. Without one: exactly demo-hp — `consume` → **404** and a retry loop, 25 hours - unprotected. **Correction to the contrast:** the recovery was of the plumbing only (Q2). -- **Q6 — why delivery is one-shot.** The design's own reasons quoted from three sites. **The security - half, stated:** the consume endpoint is gated by `checkAuthCustomer` — **the customer's API key, the - same credential as the config pull** — so an automatic restage grants **no new access to an - unauthenticated party**. What it genuinely widens is the **exposure window**: a live provider - password becomes fetchable whenever the box reports no offsite target, rather than only in the - seconds after an operator acts. Blast radius stays the customer's own label-scoped sub-account. The - R-39(a) unconsumed-secret guard must survive any change. -- **Q7 — R-192's guard. BOTH HALVES CONFIRMED WITH NUMBERS.** See §4. -- **Q8 — what each candidate costs.** See §5. +Hub SQLite snapshot with the `-wal` (`kubectl exec … cat`), `PRAGMA integrity_check` → `ok`, freshness +proven positively (`MAX(host_reports.received_at)` = `2026-08-04 10:00:40` vs `datetime('now')` = +`10:03:48`). Escrow tables, `PRAGMA table_info` on both, latest-report `$.offsite.*` and `$.claimed` +per customer, the `claimed` 1→0→1 transition across both rebuilds, and the events table since +2026-08-03. Plus one `ls -la --time-style=full-iso` inside guest 9201 proving the data dir, the +`encryption.key` and `offbox/repo_password` are all newborn as of `2026-08-03 07:17:25`–`07:18:02`. +Every query with its field names and values is in §11 of the findings doc. ---- +**Nothing was written, run, rotated, re-issued, deleted or reconfigured anywhere.** No ceremony, no +`selftest=identity-consume`, no touch on the storage endpoint. -## 3. Which of the two contradictory claims about Re-issue is true +## 5. What could not be established -**The claim that Re-issue "only resets the transport credential" is TRUE about what -`ReissueCredentials` does. R-39's "rotates the restic password" is FALSE.** +The 2026-08-05 ~02:15 UTC run's outcome (still the decisive measurement; neither box has run — +`last_run` absent on both); whether demo-hp's data dir was fresh (measured only on demo-felhom; +demo-hp needs the break-glass credential this read-only session did not use); whether the orphaned +ciphertext still exists at the provider; whether `age` is present on either demo host. Named in §12 +of the findings doc. -From source (`hub/internal/offsite/offsite.go:150-228`), the function does four things: `genPassword`, -`ResetSubaccountPassword`/`ResetBoxPassword`, `SaveOneTimeSecret`, and bookkeeping. **There is no -reference to a restic password anywhere in it, and none is possible** — that password is generated on -the box and never leaves it except into the R-wrapped escrow, which the hub cannot open. +## 6. Observations recorded, not acted on -**But the bookkeeping is justified by the false premise**, `offsite.go:198-201`: *"the restic repo -password just changed, so any existing key-escrow blob … is now STALE"*. The same wrong claim appears -at `api/handler.go:1067-1069` and in R-193's own row. **Eighth entry in `CLAUDE.md`'s table of -comments asserting a guarantee the code does not provide** — the first where the comment factually -describes a *different function*. → **R-196**. +A fresh `encryption.key` on every rebuild makes every pre-rebuild `app.yaml` `ENC:` secret +undecryptable — circular dependency **C-1** of `_recovery-inventory-2026-07-28.md`, which fired live on +both demo boxes this week and is a strictly larger blast radius than the offsite repo. `Claimed` +returning to 1 with no event means "the customer claimed this box" has no audit trail. +`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting. `handleReEnroll` returns a +freshly minted API key in its response body on an endpoint nothing currently calls. -**What corroborated it, and the corroboration is the strongest part.** The task asked: if the repo -password was untouched, what explains demo-hp's escrow going `pending` after the Re-issue? **Fully -explained, and not by the Re-issue:** `ApplyOffsiteTarget` carries `EscrowState` over from the -*existing* target, the rebuilt guest had none, so the `pending` fallthrough fired — and the box had -just minted a new repo password, so `pending` was *correct for a different reason*. -**The proof they are independent is demo-felhom:** it received **no Re-issue at all** (its only -`escrow_stale`/`offsite_reissued` pair is dated `2026-07-21 08:29:29`), its repo password changed -anyway, and **nothing marked its escrow stale for 13 hours.** The mechanism is exactly inverted: -`escrow_stale` is wired to the one path that does **not** change the repo password and absent from the -path that does. +## 7. Conventions ---- - -## 4. Every query used, with its fields and values - -The DB snapshot was taken from the live pod and its **freshness proven with a positive observable**, -not with "the query returned no error": - -| # | Query / read | Result | -|---|---|---| -| 1 | `kubectl exec deploy/hub -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` | **The `-wal` is load-bearing**: `hub.db` alone was 2 h 15 m stale (mtime 08:34 vs wal 10:45 CEST). Copying only the main file is the R-3 shape. | -| 2 | `PRAGMA integrity_check` | `ok` | -| 3 | `MAX(host_reports.received_at)` vs `datetime('now')` | `2026-08-04 08:45:38` vs `08:49:46` — **newest row 4 min old** | -| 4 | `SELECT customer_id, status, created_at FROM customer_configs` | `demo-felhom` active · `peti-felhom` active · `demo-hp` active · `drill-r50` **blocked** · `david` active (created `2026-08-01 16:51:49`). **`sess-f` is gone — confirmed, not assumed.** `drill-r50` untouched. | -| 5 | `hosts` / `host_deletions` / `host_reports` / `reports` counts for `david` | **0 / 0 / 0 / 0** — never bound | -| 6 | `events WHERE event_type='expected_dbdump_missed' AND created_at > '2026-07-25'` | **only `david`**, 03:00 UTC on 08-02, 08-03, 08-04 | -| 7 | `SELECT customer_id, MAX(received_at), COUNT(*) FROM reports GROUP BY customer_id` | `demo-hp` 1329 · `demo-felhom` 6690 · `drill-r50` 183 (last 07-29) · `peti-felhom` 482 (last **2026-07-15**) — the reason peti-felhom is `down` and does not alarm | -| 8 | distinct `$.offsite` objects per box with first/last seen | demo-hp: `15 snapshots / 40 927 044 B` until `08-03 05:59:21`, **ABSENT** `06:12:19`→`08-04 07:12:01` (111 reports), then `pending` 07:15:47, `escrowed` 07:16:02, `snapshot_count 0`. demo-felhom: `36 snapshots / 1 136 685 919 B` until `08-03 07:01:28`, ABSENT ×1, then `pending` `07:19:10`→`08-04 07:20:13`, `escrowed` 07:20:28, `snapshot_count 0` | -| 9 | `host_escrow` + `host_escrow_superseded` `restic_pw_sha256` | **the decisive measurement** — both changed (§2) | -| 10 | `one_time_secrets` (no values read) | demo-felhom created `2026-07-21 08:29:29`, consumed `2026-08-03 07:17:58`; demo-hp created+consumed `2026-08-04 07:11:51`/`07:12:06` | -| 11 | `$.claimed` on the latest report of each box | **`1` on both** — which selects `ensureOffboxRepo`'s refuse branch | -| 12 | `CountReportsOffsiteSince` reproduced verbatim (`ORDER BY id LIMIT 500`) for demo-hp @ `2026-07-23 09:53:41` | **`total=500, withOffsite=500`**, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** | -| 13 | same window, **no LIMIT** | `total=1174, withOffsite=1063` (⇒ 111 without — matches #8) | -| 14 | full text of the live `offsite_delivery_stuck` message | *"…and **500 report(s) since carry no offbox target**…"* while `OffsiteReportsSinceConsume` was **500** | -| 15 | `COUNT(*)` of `offsite_credential_restaged` | **0, for every customer, ever** | -| 16 | the shipped predicate run verbatim against live data (post-deploy) | `david` → **0**; `demo-felhom`/`demo-hp`/`drill-r50`/`peti-felhom` → **1** | - -**Instrument limits recorded rather than discovered later:** `one_time_secrets` has `customer_id` as -PRIMARY KEY and is last-write-wins **by design**, so it holds only the newest credential per customer — -demo-hp's `2026-07-23` consume, on which R-192's whole diagnosis rests, was **overwritten** by -yesterday's Re-issue and is no longer readable from that table. The `2026-07-23 09:53:41` anchor in -query #12 comes from R-192's record, and that is stated in the spike rather than presented as a live -read. - ---- - -## 5. Ranked options, with a recommendation - -| rank | option | verdict | -|---|---|---| -| **1** | **(d) the honesty pass — now** | Cheap, weakens nothing, and **the only option that helps the incident already in flight**. Two parts: correct R-192's message + log its refusal (**R-196**), and **detect a changed `restic_pw_sha256`** — both hashes are already in the hub's DB and **nothing reads them** (**R-197**). | -| **2** | **(c) agent-retained repo password — NOT PREVIOUSLY NAMED** | The only candidate aimed at the irreplaceable secret. The agent lives on the Proxmox host, **survives a guest rebuild**, already receives the password over the pinned local API and already writes it to a fixed 0600 path — it merely **wipes** it after the ceremony. Every seam exists. Price: a copy of the data key at rest on the host. **The operator's trade to make.** | -| **3** | **(a) hub restages automatically** | **Already exists** as `reissueOnReenroll`'s F3 leg, wired to the wrong event (behind `handleHostEnroll`'s mint-once-reuse short-circuit; a guest rebuild leaves the `hosts` row intact). Worth doing **after** (c), never instead: on its own it restores the transport to a box whose data key is already gone, and **would have made both boxes look healthy on 2026-08-04 while their snapshots were orphaned** — strictly worse than the current loud failure. | -| **4** | **(b) recover from escrow at re-bootstrap** | **Not implementable as stated.** The escrow is R-wrapped and zero-knowledge; the hub has no recovery code. "Recoverable at re-bootstrap" describes a customer-present ceremony — i.e. the manual form that already exists. | - -**Recommendation: ship (d) now; then decide (c). Do not ship (a) first.** - -### The operator's question — stated, and NOT answered - -> The irreplaceable secret is the restic repository password. It is generated on the box, destroyed by -> a guest rebuild, and the only surviving copy is sealed under the customer's recovery code. Every -> machine will be reinstalled. -> **Do you want that key to survive a rebuild unattended — by keeping a copy on the Proxmox host, -> where you already have root — or do you want it to stay sealed and accept that every rebuild starts -> a new repository, provided the system says so loudly on the day it happens?** - -**STOPPED here. Neither candidate was begun.** - ---- - -## 6. Part 4 — the phantom-customer alarm (SHIPPED) - -**Files changed** -- `hub/internal/store/store.go` — **new** `(*Store).HasEverBoundHost(customerID) (bool, error)`: - a live `hosts` row **OR** a `host_deletions` tombstone. -- `hub/internal/monitor/deadline.go` — one guard at the top of `CheckBackupDeadlines`' customer loop, - plus an `unbound` counter in the summary line. -- `hub/internal/monitor/deadline_unbound_test.go` — **new**, 5 tests. - -**Commit:** `7fff45d688fa` (code + tests + spike + docs) · `f581ac134964` (manifest bump). - -**The mechanism, established at source and corroborated live.** The existing down-skip could never -have covered this: it reads `StalenessChecker.GetState()`, whose map is seeded from -`store.GetCustomers()` — **a query over the `reports` table** (`store.go:1055-1067`). A customer with -zero reports is in no row, gets no state, and `GetState()` returns `""` rather than `"down"`. **The -skip misses exactly the customer it would most obviously cover.** Corroborated by query #7 and by the -live post-deploy startup log — `Staleness checker initialized: 2 ok, 0 stale, 1 down` — where the -`1 down` is `peti-felhom` (host deleted 2026-07-15, 482 old reports) and `david` is **absent from the -census entirely**. - -**The discriminator is "was a host EVER bound", not "has a report arrived"**, which is the point: -a box that was installed, bound and went silent has a `hosts` row and keeps alarming. **Fail-open on a -read error** — an unreadable binding must never suppress a real alarm. The deferral is **logged with -its own counter** (the v0.73.0 Part-7 precedent). The anchored-verdict structure is untouched, and -`david`'s record was not modified. - -**Sibling verdicts checked rather than assumed.** The census of every customer-iterating checker: -`deadline.go` backup half (already guarded by `reportJSON == ""`), `restoretest.go` (guarded by -`latest == ""`), `staleness.go` + `offsite.go` (iterate `GetCustomers()`, report-derived — a hostless -customer never appears), `offsite_delivery.go` (a hostless customer derives `DeliveryNoSecret` and is -skipped). **The DB-dump half was the only verdict that could fire**; the guard covers both verdicts in -that function uniformly. - -**Test results — the suite run and the exit code read BEFORE any commit, never in one command.** - -``` -go build ./... → BUILD OK -go vet ./... → VET OK -go test ./... → rc=0, 17 packages ok, no FAIL lines -``` - -The 13 `TestCheckBackupDeadlines*`/`TestHasEverBoundHost` tests were run with `-v` and **the `=== RUN` -lines were checked**, per `CLAUDE.md`'s rule that a `-run` pattern matching nothing prints `ok` and -exits 0. - -**THE RED-PROOF THAT MATTERS.** `TestCheckBackupDeadlines_BoundButNeverReported_StillAlarms` is the -case the change could break — a real box that was installed and never phoned home, indistinguishable -from `david` on every signal *except* the one the guard reads. With the guard removed it **still -passes**, which is the correct result and proves the test is aimed at the guard's *risk*, not its -presence. The companion red-proof of the fix itself, **observed**: - -``` -=== RUN TestCheckBackupDeadlines_NeverBoundHost_Silent - deadline_unbound_test.go:58: a customer with NO host ever bound must raise NOTHING; - got [expected_dbdump_missed] ---- FAIL: TestCheckBackupDeadlines_NeverBoundHost_Silent -``` - -— verbatim the event `david` sent three mornings running. `deadline.go` restored from a backup taken -before the mutation; full suite re-run green afterwards. - -**Deployed version.** `felhom-hub:0.92.0` built + pushed (`sha256:ca71140cf54a…`), -`manifests/hub.yaml` bumped `0.91.1 → 0.92.0` in git, ArgoCD hard-refresh + **deliberate sync** (never -`kubectl set image`). Verified: app **Synced / Healthy**, `rollout status` → `deployment "hub" -successfully rolled out` (rc=0), running image `gitea.dooplex.hu/admin/felhom-hub:0.92.0`, pod -`hub-6775889d95-gc8sk` 1/1 Running, clean startup log. - -**Is the 05:00 alarm confirmed silent? NO — CONFIRMATION IS PENDING, and it is stated rather than -assumed.** The hub's own log says `deadline-check: next run at 2026-08-05 05:00 CEST (in 17h53m57s)`; -there is no manual trigger (`CheckBackupDeadlines` has exactly one caller, the scheduler at -`cmd/hub/main.go:598`). An absent alarm tomorrow would in any case be weak evidence, so a **positive** -observable was taken instead: the shipped predicate was run verbatim against a fresh copy of the live -DB (query #16) — `david` → **0** (will be skipped), every other customer → **1** (behaviour -unchanged). **The 05:00 firing itself is still owed.** - ---- - -## 7. `OPEN-ITEMS.md` rows, and the capability map - -- **R-192** — updated with Q7's confirmed numbers; **NOT closed**, and now carries a *new* reason not - to fix it in isolation (a successful auto-restage would have protected the plumbing while the data - was already gone). -- **R-193** — updated with the whole spike; **NOT closed**, state moved to **WAITING-ON-OPERATOR**, - owner `CC + operator`. The old "15+ vs 1 decides it" line is struck through and replaced with the - measured answer plus the third predicted outcome. -- **R-195** *(new)* — the phantom-customer alarm. **SHIPPED**, hub v0.92.0. -- **R-196** *(new)* — `escrow_stale` wired to the path that does not change the repo password, absent - from the one that does. **OPEN.** -- **R-197** *(new)* — the hub holds both `restic_pw_sha256` values and reads neither. **OPEN.** - -**The capability map DID need touching, and the §N.5 escalation did NOT fire — the row was not -overclaiming.** `00-capability-map.md`'s offsite row already recorded reinstall-continuity: *"a -recreated data volume that orphaned the repo … is now CLASSIFIED … + a move-aside (never-delete) -reset"*, shipped controller v0.142.0 and live-fired 2026-07-18. **That independently corroborates the -Q3 prediction from a different direction.** What it did not record — and now does — is that this fires -on an **ordinary, planned, unattended guest rebuild** on every box, with the measured 2026-08-03/04 -instance, the hash evidence, and the inverted `escrow_stale` wiring. **Status unchanged -(PROVEN-LIVE):** the guard is correct and is not the gap; the cadence is. - -`ROADMAP.md`'s newest row is R-171 and none of R-190…R-197 has one, so **there was nothing to -collapse.** One correction *was* warranted under the coupling rule: R-1's row claimed the v0.57.0 -reinstall arc makes offsite re-issue *"auto-fire on re-enrollment"* — scope-corrected in place, since -that means **host** re-enrollment after a host **delete**, and a guest rebuild never reaches it. - -`CONTEXT.md` gained **S-26** (the spike's Q1/Q2 headlines + the three corrections that must not be -re-inherited) and **S-27** (the R-195 ruling and its mechanism). `REUSE.md` gained the -`HasEverBoundHost` row with its "never a substitute for a liveness check" trap. `STATUS.md` rewritten -for the operator, one screen. - ---- - -## 8. CI, task ids, and `--no-verify` - -| commit | what | CI task id | run # | conclusion | -|---|---|---|---|---| -| `7fff45d688fa` | R-195 code + tests + the spike + all docs | **126** | 72 | **success** | -| `f581ac134964` | `manifests/hub.yaml` 0.91.1 → 0.92.0 | **127** | 73 | **success** | -| `b7c5cfc06188` | this report + `STATUS.md` + the ROADMAP scope correction | **128** | 74 | **success** | - -*(A report cannot contain the run id of the commit that adds it; the docs-only follow-up that appended -run 74 to this table is reported in the session summary.)* - -Matched by `head_sha` via -`curl -s "https://gitea.dooplex.hu/api/v1/repos/admin/felhom.eu/actions/tasks?limit=5"` — a PULL -check, not a reliance on the failure mail. -**`--no-verify` was NOT used.** The pre-push hook ran and reported `gates OK - push proceeding`; -`python3 scripts/repo_gates.py` was additionally run in full (all five gates OK) before committing. - -**Teardown:** the spike provisioned nothing — no guest, no VM, no storage, no credential. The only -artefact is a hub-DB snapshot in the session scratchpad, outside the repo. - -**Read-only discipline held:** no `ReissueCredentials`, no ceremony, no credential rotation, no -change to any box's offsite configuration, no snapshot deleted, no customer record modified. -`drill-r50` untouched; `sess-f`'s removal **confirmed** rather than assumed. - ---- - -## 9. Observations — noticed and NOT acted on - -1. **The orphaned ciphertext is retained and keeps consuming quota, with nothing to prune it.** - `resetOrphanedRepo` is move-aside-never-delete by design (correct), but 40.9 MB + 1.14 GB of - unopenable data will sit against a 50 GB soft quota indefinitely. S-24's ep0 prune jobs cover the - **PBS** namespaces, a different tier. Not filed — it belongs to whichever option is chosen for - R-193. -2. **`host_escrow_superseded` holds only two rows in the entire database**, both from 2026-08-04. That - is why the spike could not establish whether demo-felhom's 2026-07-21 Re-issue re-sealed an - *unchanged* password — the predicted false-staleness shape of R-196. Retention policy for that - table was not investigated. -3. **Both demo guests were rebuilt on consecutive mornings** (demo-hp 08-03 06:12, demo-felhom 08-03 - 07:17, both to controller 0.192.0 with new config hashes). *Why* is outside this spike's scope and - was not chased. -4. **`api/handler.go:1067-1069`'s comment is the third copy of R-196's false premise**, and it sits in - a different package from the other two. Whoever fixes R-196 should grep rather than fix the one - they were shown. -5. **`peti-felhom` is `active` with its host deleted since 2026-07-15**, judged only by the `down` - skip. It is correct today and would start alarming the moment it reported once. Not a defect, but - the shape is worth knowing. -6. **The `-shm`/`-wal` trap generalises.** Any future session reading the hub DB from the pod must - copy all three files. Reading `hub.db` alone today would have produced confident answers from a - file 2 h 15 m stale — the R-3 shape with a different filename. +- **No `CHANGELOG.md` entry.** `felhom.eu` has no root changelog — they are per-area (`hub/`, + `scripts/`, `website/`) and log code changes. This session changed none of those areas. Stated + rather than silently skipped. +- **`REUSE.md`** untouched — no shared helper or pattern moved. +- **Gates + CI run id** are recorded in the session summary. + diff --git a/STATUS.md b/STATUS.md index 11522f3..697c58e 100644 --- a/STATUS.md +++ b/STATUS.md @@ -35,6 +35,25 @@ Proven end to end on real hardware. **The next scheduled off-site run is tomorrow at 04:15, and it will refuse rather than quietly start a new history** — the machine already knows how to recognise this and asks before resetting. **There is a decision here for you — see "Waiting on you".** *(R-193, R-192, and two new items)* +- **And the safety net we believed was under all of this is not there.** We traced the whole path today — + from a dead machine to reopened backups — and it does not join up. The important part is not that + several steps are manual; it is this: **the central hub keeps the old sealed key when a machine + re-seals, and we found it keeps the wrong one.** It keeps the key for the whole-machine backups and + **not** the key for the off-site file backups — the one this entire problem is about. So the moment + a rebuilt machine is asked to make a new recovery code, which is exactly what it asks the customer to + do, **the last copy of the old key is gone.** Both demo machines crossed that line yesterday morning. + **This means the 51 orphaned backups were beyond reach even if you had kept the recovery codes** — + which is a relief in one narrow sense and much worse in every other. It also means the customer is + currently told, in Hungarian on their own screen, that their old backups "may later be restorable + with the matching recovery code". That sentence is not true today. It is a small fix — one missing + column — and it must land before anything else here. *(R-198)* +- **Nothing in the recovery path has ever been performed.** Making a recovery code is proven, live, + with a real customer. **Using one is not.** No sealed package has ever been handed back to a machine + (the hub has the buttons for it; nothing anywhere presses them), no recovered key has ever been put + back, no old backup store has ever been reopened, and no file has ever been restored from a recovered + key. There is a form missing, a step missing, and a connection missing between two parts of the + system. We designed the exercise that would prove it end to end — see "Waiting on you". + *(R-199, R-200, R-201)* - *(fixed 4 Aug)* ~~The weekly off-site backup reports FAILED although it worked.~~ It uploaded fine and then tripped on a tidy-up step it is deliberately not allowed to perform. The machine no longer asks — tidying up is the endpoint's job, and **that was checked first**: the endpoint has been doing @@ -106,6 +125,32 @@ Proven end to end on real hardware. nothing else, and it is the piece whose absence let a gigabyte of history go quiet for thirteen hours. What we are asking you to choose is (a). *(R-193; the full reasoning is in the findings document)* + **Correction to (b), from today:** "old backups are kept" is true, but "openable later with the + recovery code" is **not** — see R-198 above. Option (b) is only honest once that is fixed. +- **Your ruling about the recovery screen has been priced, and it can be built.** You said: a freshly + installed machine that finds a sealed package waiting should say so loudly, offer the customer a box + to type their recovery code into, and show what would come back before doing anything. **All of it is + buildable, and one part is already free** — the hub is *already* telling every machine, on every + check-in, that a sealed package exists and which key it covers, and the machine currently throws that + message away. Showing a preview is also cheap: listing what is in an off-site store reads it without + writing to it, so the customer can see how many backups, from when, and for which apps before + committing. The real work is one new connection: the unsealing has to happen in the part that runs on + the Proxmox host, because the customer-facing part cannot do it — the same crossing your recovery-code + ceremony already makes in the other direction. **One thing for you to weigh, which we deliberately did + not decide:** a screen that takes a recovery code and then shows what is in a backup store is reachable + by anyone with the household's dashboard password, and the preview reveals backup dates and app names. + *(R-193)* +- **Should we run the proof?** Nothing about recovery has ever actually been done, so we designed the + exercise: on the spare demo machine, make a recovery code and **keep it**, put a marked file into an + app, back it up off-site, wipe the machine, reinstall, recover with the saved code, and check the + marked file comes back byte for byte. About half a day, of which under an hour needs you. The failure + we are watching for is precise: **if the recovered machine reports one backup instead of the ones we + put there, it started a fresh history and the proof failed** — "the store opened" is not good enough. + It costs that machine's contents if it goes wrong, which is acceptable there. **Do R-198 first**, or + the exercise walks a path that is missing a piece. *(R-201)* +- **The orphaned backups still on the storage box.** About 1.2 GB across the two demo machines, sitting + in set-aside stores nobody can open and nothing prunes, using the 50 GB allowance. We did not touch + them. **Delete, or leave?** *(R-193)* - **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a session log; nothing suggests anyone else saw it. *(R-132)* - **One small question, not urgent.** The automatic check cannot see which version you have told @@ -115,6 +160,11 @@ Proven end to end on real hardware. ## Changed since last update +- **2026-08-04 (latest)** — Traced the whole recovery path end to end and it does not join up. The hub + keeps the wrong sealed key when a machine re-seals, so a rebuilt machine's old off-site key is lost the + moment the customer is asked for a new recovery code. Nothing in the recovery path has ever been + performed. Your ruling about the recovery screen is priced and buildable. The proof exercise is + designed and awaiting your go-ahead. *(R-198, R-199, R-200, R-201, R-193)* - **2026-08-04 (later)** — Established what a machine rebuild actually destroys: not the storage password we had been re-issuing, but the key that encrypts the backups. Both demo machines lost access to their off-site history, one of them silently. A decision is now waiting on you. The daily diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 109d297..5093a8d 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -36,7 +36,7 @@ | Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`.rootpw.txt`) | scripts v1.24.0 (`scripts/iso/felhom-bootstrap.sh` `network_gate`/`sweep_nics`, `build-felhom-iso.sh` rootpw emission) | **PROVEN-LIVE (nested drill — nested ≠ metal: metal proof rides the next real multi-NIC install)** | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` — dead-NIC install from the virgin v1.24.0 ISO baked the 192.168.100.2 fallback (WITH a dead default gateway), the R-59 screen painted on the console (screendump captured), and after the cable move the box swept to the working NIC, re-leased and **self-registered at the hub unaided in under a minute**; the drill also caught + fixed the stale-fallback-route trap (flush before the bounded dhclient) and verified the emitted rootpw against the installed box's shadow hash | R-59 ships as a first-boot gate, not an install-time abort (recorded deviation — the fallback is the auto-installer's own, initrd hook out of scope); sweep is structurally first-boot-only (`state.json` gate + unit done-flag condition); a box past install-start gets the screen but its interfaces are never touched | | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. **Deliverability (R-4), gmail half DONE 2026-07-18:** the rehearsal's claim email was the first sent under the tightened DMARC `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu remains Viktor's open half.** (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) | | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends | -| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test | +| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention) — ⚠ CORRECTED 2026-08-04, the claim as written is FALSE for the offsite tier:** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`). ~~so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable")~~ **It retains the K-ESCROW BLOB ONLY. `host_escrow_superseded` has no `identity_blob` column and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`) does not copy one, so what stays recoverable with the old recovery code is the PBS datastore key — NOT the restic repository password, which lives in `identity_blob` and is OVERWRITTEN by the next ceremony.** Measured live 2026-08-04: both current rows hold `blob`=383 B **and** `identity_blob`=572 B; both superseded rows hold `blob`=383 B only. The reinstall-orphan incident this sentence names is therefore still "history destroyed" for Tier-3 — and the same false promise is made to the customer in Hungarian on the orphan card. → **R-198**, evidence `audits/RECON-offsite-dr-chain-2026-08-04.md` §7. **There is also no read path:** `ListSupersededEscrow` has zero production callers; retrieval is `sqlite3` on the hub pod, by hand. Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody`. **agent v0.93.0 (2026-07-21) — recovery codes can no longer contain a hyphenated word.** The EFF large list holds exactly four entries containing the hyphen the words are joined with (`drop-down`, `felt-tip`, `t-shirt`, `yo-yo`); drawing one produced a code that reads as 11 words instead of 10 — ambiguous to transcribe in exactly the situation R exists for. They are now excluded **from GENERATION only**: the draw space goes 7776 → 7772 and a 10-word code 129.248 → 129.241 bits, still well clear of the 128-bit floor. **Every code already issued remains valid** — R is verified as a whole passphrase by the PBS scrypt KDF and is never re-split, so no customer needs to re-run a ceremony. This also retired the long-standing ~1/5 `TestGenerateRecoveryCode_EntropyAndFormat` flake, which was this defect and not a flaky test | | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** (2026-07-21) | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant **2026-07-21 — the R-39 fleet fix SHIPPED (hub v0.68.0 + agent v0.91.2), closing the self-heal chain end to end.** The three defects that let a box be `applied` and dead simultaneously are each addressed: the hub stamps a monotonic `secret_generation` into the descriptor so a credential re-key finally MOVES the content hash the agent re-applies on; the wrapper gains a narrow `read` verb so the non-root agent can read the credential it writes (it never could — `/etc/pve/priv` is 0700 root:www-data, which made the verify loop blind by construction); and `pbs.ProbeAuth` turns a 401 into a loud `auth_failed` that the existing `pbsdrheal` damper escalates to a fresh mint. Plus a consumed_at honesty gauge for the disagreement no single tier can see (box says `applied`, hub's staged secret never consumed). Proven live on felhom-pve: the agent read its credential through the wrapper (`rc=0`) and probed successfully (`credential probe OK storage=felhom-pbs`). **STOP-2 RAN 2026-07-21 AND THE CHAIN CLOSED — 13 SECONDS, operator click to converged.** The operator pressed **Re-issue PBS credentials**; the identical click on 2026-07-18 did nothing at all. Full chain (hub UTC / host CEST = UTC+2): `08:39:31Z` hub mints a fresh secret, **generation 0 → 1**, and the descriptor gains `"secret_generation": 1` — with `token_id` and `fingerprint` **byte-identical**, i.e. exactly the re-key shape that used to be invisible → `10:39:34` the agent READS its credential through the wrapper (leg b — the read that was impossible until v0.91.0) → `10:39:38` **`ERROR pbsdr: the DR endpoint REJECTED this box's credential — the tier is applied and DEAD` `previous_state=applied`** (leg c: the exact R-39 failure state, detected out loud for the first time ever) → `10:39:45` **`one-time token secret consumed`** `secret_len=36` (leg a: **NO short-circuit** — this is the line that never appeared on 2026-07-18) → `10:39:45` `felhom-pbs-apply reconcile` (the set-only wrapper, no `--server`) → `10:39:47` **`pbsdr: converged state=applied`**. Corroboration: the agent marker hash moved to `afbb3b41…` (it was byte-identical to the pre-reissue marker in the failure); `consumed_at` stamped `08:39:45Z`; the on-disk secret's mtime moved `2026-07-18 20:28:52` → `2026-07-21 10:39:45`; a live probe with the NEW credential returns **200**; three consecutive hub reports trace the whole state machine `applied → auth_failed → applied`; and **zero** `pbsdr_selfheal` escalations fired — the box healed through the descriptor path before the damper was ever needed, with exactly ONE mint and ONE consume and no `consumed-failed.json`. **Row upgraded to PROVEN-LIVE (2026-07-21).** Evidence: `felhom-agent/REPORT.md` (2026-07-21). | | **Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE** — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE (2026-07-26)** | `audits/SPIKE-r82-phase0-2026-07-26.md`; per-repo CHANGELOGs/REPORTs. **Restore round-trip on demo-hp:** `--selftest=restore-test` against `felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z` → `pass:true`, `verified:"boot+running"`, **`mount_parity:"ok"`** (`mp0=/var/lib/docker 50G`, `mp1=/mnt/sys_drive 20G`, mp8/mp9 throwaway stand-ins for the archived binds), `source_tier:"pbs"`, 4m5s restore+boot+verify+teardown, scratch band clean afterwards and the live guest untouched. | **This row is distinct from the DR-tier row above, which proves ACTIVATION, not ARRIVAL.** That tier was PROVEN-LIVE as `applied` since 2026-07-21 while demo-felhom held ONE snapshot (a healing artifact) and demo-hp held **zero, ever** — "applied and empty", the R-39 shape one level quieter. **What earns PROVEN-LIVE here:** (a) demo-hp's FIRST EVER offsite backup landed (4.25 GB into a verifiably empty namespace); (b) it **restores into a bootable, mount-complete guest** — `mount_parity` is the non-hollow half, since a boot-only verify cannot see a missing data volume; (c) **the multi-tier quiesce ran through the real UI endpoint** (`POST /api/guest-backup/trigger`, authed+CSRF) and produced **exactly ONE stop/start pair with BOTH backups inside it** — `quiescing 1 stack(s)` 17:01:39 → local done 17:02:56 *"next tier may start (app still quiesced)"* → felhom-pbs snapshotted 17:03:06 → `unquiescing` 17:03:06. **App downtime 1m27s for both tiers**, and the app came back healthy. **Known gaps, recorded not hidden:** the SCHEDULED restore-test still only selects the PRIMARY tier, so the offsite tier is never AUTOMATICALLY restore-tested (the manual/selftest path is proven, the unattended one is not); the hub infers "PBS ⇒ weekly" from storage TYPE rather than a reported cadence; the installer-default fleet flip awaits a full weekly cycle. → **R-82** | | **Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard** | agent v0.104.0 → **v0.121.0**, hub v0.77.0 → **v0.91.0** | **PROVEN-LIVE (2026-08-03)** | per-repo CHANGELOGs; `backlog/SPEC-r85-phase4-5-2026-07-26.md`. Unit red-proofs for tier rotation, restart-survival, the one-heavy-op gate, failure-emits-an-event, and newborn silence. | **The row above is earned by a MANUAL `--selftest=restore-test`; this one is about the SCHEDULED path, and the distinction is the whole point.** Before R-85 the scheduler could only ever see `cfg.Backup.BackupTarget()`, so the offsite tier was never a candidate — and a failed restore-test was a `[WARN]` line with no event at all, which was true for the LOCAL tier that WAS being tested. Now: oldest-first rotation across every configured tier (operator ruling 2026-07-26), persisted so it survives a restart; a restore-test joins the host-wide one-heavy-operation gate so it never contends with a backup over the same tunnel; and the hub raises two DISTINCT operator-tier signals — `restore_test_failed` (broken now) and `restore_test_stale` (unverified, not known-broken), anchored on R-81 so a newborn box never alarms. **Why this was NOT PROVEN-LIVE until now:** rotation had not been observed selecting both tiers across consecutive UNATTENDED cadences — at a 24h cadence a multi-day window — and a single passing run proves the code path, not the schedule. → **R-85**. **R-86 (agent v0.121.0 + hub v0.91.0, 2026-08-03) replaced the schedule and the observation became possible in one afternoon**, because what has to be observed is no longer a multi-day rotation but a RULE: a tier is due when its newest archive that has settled ~24 h has not been proven. **UPGRADE EVIDENCE — a real unattended run on demo-felhom (2026-08-03), triggered by DUE-NESS, not by a timer.** The scheduler's own log: `15:14:38 restore-test tier is DUE … target=felhom-pbs archive=felhom-pbs:backup/ct/9201/2026-07-28T04:49:43Z … reason="newest settled archive … has not been proven"` → `proxmox-backup-client restore --crypt-mode=encrypt` under the agent's own token → `15:25:08 gate decision class=guest_destroy guest=990000 allowed=true` → `15:25:14 scratch guest torn down` → **`15:25:14 backup: scheduled restore-test passed archive=felhom-pbs:… duration_s=635.1`**. A **14.5 GB encrypted offsite archive pulled from ep0 over the WAN**, restored, booted, verified and destroyed in 635 s, unattended. **The three things a timer could not show**, all verified after it: the state names THAT archive (`{"felhom-pbs":{"archive":"…2026-07-28T04:49:43Z","proven_at":"2026-08-03T13:25:14Z"}}`); a second evaluation reports `due=false … is already proven` and runs nothing; and **an agent restart runs nothing**, which is the defect a person actually noticed — every deploy used to restart the timer. Teardown verified at all three layers: guest absent from `pct list`, **zero** `990000` volumes in `lvs`, and the hub-side `restore_tests[]` entry deliberately RETAINED (it IS the proof the staleness check reads). **R-189 (agent v0.122.0, 2026-08-03) closed the reporting half of this row, and it was a REAL gap in the evidence path:** the proof above reached the hub only because no restart intervened — `restore_tests[]` came solely from an in-memory store, so the 15:25:14 PASS was in fact LOST when the agent restarted 2 m 43 s later for a deploy (`0 restore-tests` on the next two host-reports). Under per-archive due-ness the box would not have repeated the work for a week. The persisted per-tier proof (with the archive, and now the tier) is merged into the report, so a proof survives a restart — one entry per tier, newest wins, and a record that cannot be described honestly is not emitted. **THE HOST TIER'S HALF OF THIS ROW WAS OPTIMISTIC UNTIL 2026-08-03, and it is worth saying plainly.** Every live restore-test cited here is on the OFFSITE tier. The HOST tier was not merely unproven on demo-felhom — it was **unprovable**, and on demo-hp equally: the agent's token had no ACL on `/storage/felhom-backup`, the storage both boxes configure as `local_backup_target`, so the content API answered `{"data":[]}` through the token while root listed three archives. The scheduler skipped it as *"no settled archive yet"* — which is exactly what a brand-new tier reports — so nothing ever said so (**R-185**). **CLOSED 2026-08-03 — agent v0.123.0 + installer 1.24.0.** The grant is applied on both demo boxes (the token now lists 3 and 4 archives), the installer's reuse arm grants on a pre-existing target, and the agent now **asks whether it may read each tier it depends on** instead of inferring it from an empty list: a critical degraded capability naming the storage and the missing role, which the hub alerted and emailed on **while the box was still blind**. **THE HOST TIER IS NOW PROVEN-LIVE, UNATTENDED, ON BOTH DEMO BOXES (2026-08-04) — this row's optimistic half is cashed.** Four SCHEDULED runs, nothing triggered by hand: **demo-felhom** host tier `…2026_08_02-04_42_14.tar.zst` **passed in 83.8 s** at 00:55, offsite `…2026-07-28T04:49:43Z` **passed in 540.4 s** at 06:55; **demo-hp** host tier `…2026_08_02-04_49_29.tar.zst` **passed in 109.3 s** at 02:05, offsite `…2026-07-28T19:19:45Z` **passed in 300.1 s** at 08:05. Each restored into a scratch guest, booted, verified and destroyed itself; `pct list` and `lvs` show zero `990000` afterwards on both, and both boxes' `local-lvm` returned to their pre-run figures (1.95 % and 30.83 %). **Both boxes had BOTH tiers due at once and the ordering was observed live for the first time under R-86's rule:** never-proven sorted first, so each box took its HOST tier, deferred the offsite one, and picked it up on the following evaluation six hours later — one heavy operation at a time, per box, without anyone sequencing it. **The proofs reached the hub**, which is R-189 carrying a host-tier entry for the first time: demo-felhom's report holds **two** entries, one per tier, and the `local` one can only have come from the persisted state because the in-memory store held only that morning's offsite run. **SCOPE, stated because one box proving something does not make it a fleet property:** this covers `demo-felhom` and `demo-hp`. The tester's box is untested and untouched. Evidence: `felhom-agent/REPORT.md`, `felhom.eu/REPORT.md` | diff --git a/documentation/audits/RECON-offsite-dr-chain-2026-08-04.md b/documentation/audits/RECON-offsite-dr-chain-2026-08-04.md new file mode 100644 index 0000000..fd6fa4a --- /dev/null +++ b/documentation/audits/RECON-offsite-dr-chain-2026-08-04.md @@ -0,0 +1,531 @@ +# RECON — can a customer actually get their backups back? + +**Date:** 2026-08-04 · **Items:** R-193, R-192 (+ new: R-198…R-201) · **Class:** recon — **no production code** +**Session scope:** read-only everywhere. Nothing built, deployed, wiped, rotated, re-issued or deleted. + +> **Headline, before the detail.** The chain is **not** assembled, and the reason is worse than "two +> links are manual". **The hub's superseded-escrow retention — the feature built for exactly this +> incident — does not retain the offsite repository password.** `host_escrow_superseded` has no +> `identity_blob` column, and `demoteCurrentEscrowTx` copies only the K-escrow blob. So the last copy +> of a repo password is destroyed by the **next escrow ceremony** — the very act the system tells the +> customer to perform when their escrow goes `pending` after a rebuild. The recovery window closes +> because the remedy closes it. + +--- + +## 1. Confirmed baselines (re-read on arrival) + +| Repo | `main` @ commit | tree | Version | +|------|-----------------|------|---------| +| `felhom-agent` | `856a127cd60b0be797867c162f0ae49c7b64993e` | clean | v0.124.1 | +| `felhom-controller` | `0887fd676d608e7c103fa13b3e87536ad2481811` | clean | v0.194.0 | +| `felhom.eu` | `d26f49ad680e2be2ce2888fd58dff7dee886e4ad` | clean | hub v0.92.0 | + +All three match the prompt's cited commits exactly. `git status --porcelain` empty in each. + +**Highest register ID: R-197** (`OPEN-ITEMS.md` R-196/R-197 were minted by yesterday's spike; +R-194/R-195 by the R-190 arc). Grepped across `OPEN-ITEMS.md`, `ROADMAP.md`, `STATUS.md`, +`CONTEXT.md`. New items in this session therefore start at **R-198**. + +--- + +## 2. Instruments — what each can and cannot answer + +| # | Instrument | Exact read | Known limit | +|---|---|---|---| +| I1 | source | the three repos at the commits above | the authority; every claim below cites a file:line | +| I2 | hub SQLite snapshot | `kubectl -n felhom-system exec pod/hub-6775889d95-gc8sk -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | the `-wal` is load-bearing (copied; `PRAGMA integrity_check` → `ok`) | +| I2-fresh | freshness proof | `MAX(host_reports.received_at)` = `2026-08-04 10:00:40`, `datetime('now')` = `2026-08-04 10:03:48` | **positive observable**: newest row 3 m 08 s old, not "the query did not error" | +| I3 | live filesystem, demo-felhom guest 9201 | `ssh felhom-pve "pct exec 9201 -- ls -la --time-style=full-iso …"` | read-only `ls`; mtimes only, no file contents read | +| I4 | caller census | `grep -rn --include=*.go` across all four repos | proves absence of a caller in Go; templates checked separately | + +**Not used, deliberately:** no ceremony, no re-issue, no rotation, no `selftest=identity-consume`, no +write of any kind, nothing touched on the storage endpoint. + +--- + +## 3. Q1 — the chain, link by link + +From a dead node to an open repository. **Status vocabulary as specified.** + +| # | Link | Component / function | Who invokes it | Status | +|---|---|---|---|---| +| 1 | repo password is minted | controller `Manager.WriteOffboxSecrets` → mint at `internal/backup/offbox.go:392-401`; generator `:405` | `ApplyOffsiteTarget` (`offbox.go:474`) ← `ConfigureOffbox` ← the apply-bridge (`offsiteapply.go:221,256`) | **PROVEN-LIVE** — fires on every fresh data dir; measured twice this week (I2 hash change, I3 mtime) | +| 2 | password staged to the agent | controller `PushOffboxPasswordForEscrow` (`offbox.go:530`) → agent `POST /escrow/stage-secret` (`localapi/server.go:518`) → `escrow.StagedResticPasswordPath()` = `/var/lib/felhom-agent/escrow-stage/restic_repo_password`, 0600 (`identity.go:44-47`) | the controller enable/ceremony path | **PROVEN-LIVE** | +| 3 | sealed under R at the ceremony | `AttachResticPassword` (`identity.go:73`) → `WrapIdentityBundle` → `WrapIdentity` (`identity.go:114`, `age -p -a`, scrypt + ChaCha20-Poly1305) | agent `--selftest=escrow-create`, driven by the controller wizard (`/escrow/ceremony`, `localapi/server.go:525-527`) | **PROVEN-LIVE** — customer-facing wizard first live firing 2026-07-18 (`00-capability-map.md`, escrow row) | +| 4 | stored by the hub | `PUT /api/v1/hosts//escrow` (`api/handler.go:239`, `handleHostEscrowPut :1104`) → `host_escrow.identity_blob` | agent `main.go:2937` | **PROVEN-LIVE** — measured: `identity_blob` present, 572 B, for **both** hosts (I2) | +| 5 | **old blob retained on supersession** | `demoteCurrentEscrowTx` (`store/store.go:2547-2556`) → `host_escrow_superseded` | `SaveHostEscrow`, `DeleteHost` | **BROKEN FOR THIS PURPOSE.** The INSERT copies `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256` — **not `identity_blob`**, and the table has **no such column** (`PRAGMA table_info`, I2). The K-escrow survives; **the restic repo password does not** | +| 6 | served back to a re-enrolling box | hub `handleReEnroll` (`api/dr.go:101`) and `handleGetRestoreDirective` (`:155`), returning `identity_escrow_b64`; recovery-mode gated, armed with the **global operator key** (`:34`) | **NOBODY.** I4: zero callers in `felhom-agent` (no `ReEnroll` symbol, no `/re-enroll` or `/restore-directive` string anywhere in the agent's hub client, `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`); zero in the hub UI (no template mentions them); zero in `scripts/` or any runbook | **implemented-never-exercised, and CLIENTLESS.** The only documented retrieval is by hand: `sqlite3 "SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …"` on a `kubectl cp`-ed `hub.db` (project memory, 2026-07-04 S5 prep) | +| 7 | unsealed with R | `UnwrapIdentityBundle` (`identity.go:172`) → `UnwrapIdentity` (`:139`) | **`runSelftestIdentityConsume` only** (`cmd/felhom-agent/main.go:2845-2900`), reading R from `FELHOM_RECOVERY_CODE`. I4: no other production caller | **manual-only — the only caller is a `--selftest` mode** | +| 8 | the repo password extracted | none. The selftest writes the **whole bundle JSON** to `-keydest` 0600 (`main.go:2872`) and prints *"identity recovered (tunnel_token + pbs_token)"* — it does not mention, extract or route `restic_repo_password` at all | a human reads the JSON | **missing** | +| 9 | placed on the rebuilt controller | `InjectOffboxPassword` (`offbox.go:541`) | one caller: `offboxInjectPasswordHandler` (`web/offbox_handlers.go:189`) at `POST /backup/offbox/inject-password` (`web/server.go:510`). **I4: no template in the repo contains that path or any form posting to it** | **manual-only, and UI-unreachable** — the handler exists, the form does not. A person must POST it by hand with a valid session cookie + CSRF token | +| 10 | the existing repo opens | `ensureOffboxRepo` (`offbox.go:653`): `restic cat config` → success clears `RepoState` | the scheduled/manual offbox run | **implemented-never-exercised** on a *recovered* password (exercised constantly on a live one) | +| 11 | a backup or restore proves it | — | — | **never** | + +### The chain in one sentence + +Links 1–4 are real and proven. **Link 5 silently drops the payload.** Links 6–9 are each individually +manual, and link 6 has no client at all. Link 11 has never happened. + +--- + +## 4. Q2 — where the chain stops today + +**The first link that is not automatic is #5, and it is not merely manual — it is lossy.** + +Two distinct stopping points, and the order matters: + +**(a) If the box has NOT re-escrowed since the rebuild** — the current `host_escrow.identity_blob` +still seals the *old* password. Recovery is possible in principle, and a person must: + +1. Arm recovery mode on the hub (`PUT /api/v1/admin/hosts//recovery-mode`, global operator key) — + or skip it entirely, because nothing consumes the endpoint it gates; +2. `kubectl cp` / `exec cat` the hub DB and `sqlite3 … writefile('/root/idblob', identity_blob)`; +3. copy the blob to a machine with the agent binary and `age`; +4. `FELHOM_RECOVERY_CODE=… felhom-agent --selftest=identity-consume -blob /root/idblob -keydest /root/bundle.json`; +5. open `bundle.json` by hand and read `restic_repo_password` out of it; +6. log in to the customer's dashboard and `POST /backup/offbox/inject-password` **by hand** (curl with + session + CSRF) — there is no form; +7. re-run the apply so `WriteOffboxSecrets` finds the file present and keeps it; +8. trigger a run and hope `restic cat config` succeeds. + +Eight manual steps, five of them requiring operator-tier access to the hub, one requiring the customer +to be present with R, and one (step 6) with no user interface at all. + +**(b) If the box HAS re-escrowed** — which is what the system pushes the customer to do, because a +rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`, +`offbox.go:570`), and the card says *„Helyreállítási kód szükséges"* — then the old `identity_blob` +has been **overwritten** by `SaveHostEscrow` and **not carried** into the superseded row. At that point +**no procedure exists**, with or without R, at any level of effort. The password is gone. + +Both demo boxes are in state (b) as of 2026-08-04 07:15:36 (demo-hp) and 07:20:08 (demo-felhom) — I2, +`host_escrow.updated_at`. This is an **independent, and much stronger, reason** the 51 orphaned +snapshots are unrecoverable than "nobody kept the recovery codes". Keeping R would not have helped. + +--- + +## 5. Q3 — can a rebuilt controller know it should recover? + +### RULING: **YES — the hub already tells it, on every single report, and the controller throws the answer away.** + +**What the box knows at the mint decision.** `WriteOffboxSecrets` is called from +`ApplyOffsiteTarget`, driven by the apply-bridge with the hub's offsite **descriptor**, whose fields +are `host, user, port, repo_path, host_fingerprint, quota_gb` (`offsiteapply.go:198-204`). **Nothing +about escrow is in scope at that point.** Locally, the only signal is the one the mint branch already +reads: `os.Stat(m.offboxPwPath())` returning `IsNotExist` — which *is* the fresh-data-dir signal, at +exactly the right line, but cannot distinguish "never configured" from "rebuilt". + +**What the box is nonetheless already told.** Every report ACK carries an `escrow` object +(`api/handler.go:504-510`), consumed as `report.EscrowStatus` +(`internal/report/escrow_confirm.go:19-23`): + +```go +type EscrowStatus struct { + IdentityBlobPresent bool `json:"identity_blob_present"` + ResticPwSHA256 string `json:"restic_pw_sha256"` + CreatedAt string `json:"created_at"` +} +``` + +That is precisely *"the hub has a blob, and here is the hash of the password it covers"* — arriving +every report cycle, on a channel that already exists, with no new API and no new secret. + +**And it is discarded.** `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:75-84`) returns +immediately unless `Pending()` or `Escrowed()` — both of which require `OffboxConfigured()` **and** a +non-nil target (`cmd/controller/main.go:580-590`). On a freshly rebuilt box, before the offsite apply, +there is no target, so the status is dropped on the floor. It is never persisted anywhere: I4 finds +exactly **one** reference to the ACK field in the whole controller +(`cmd/controller/main.go:641: escrowConfirmer.Reconcile(resp.Escrow)`). + +**Therefore a fail-closed refusal IS implementable**, with no new hub endpoint: + +- persist the last-seen ACK escrow status alongside settings (the precedent exists — `ClaimSync` + caches the hub's claim-code state into `settings.json` idempotently by generation, + `internal/report/claim_sync.go:39-53`, and it does so *set-only, never cleared on hub silence*, + which is the correct shape here too); +- at the mint branch, refuse when the password file is absent **and** the cached status says + `identity_blob_present && restic_pw_sha256 != ""`. + +**One ordering caveat, stated rather than assumed.** On a truly fresh box the apply can race the first +ACK. Measured on demo-hp: first post-rebuild report `2026-08-03 06:12:19`, offsite apply +`2026-08-04 07:12` — the ACK preceded the apply by 25 hours, but that gap was an incident, not a +design. A fail-closed rule handles the race correctly by construction: refuse to mint until the escrow +status is known, and retry next cycle (the apply-bridge is already built to retry and to persist +nothing on a partial failure, `offsiteapply.go:1-6`). + +**Corroborating measurement that the data dir really is fresh on a rebuild** (I3, demo-felhom guest +9201, `/var/lib/docker/volumes/felhom-controller-data/_data/data/`): + +| file | mtime | +|---|---| +| the data directory's parent | `2026-08-03 07:17:25` | +| `encryption.key` (32 B) | `2026-08-03 07:17:26` | +| `offbox/` | `2026-08-03 07:18:02` | +| `offbox/repo_password` (64 B) | `2026-08-03 07:18:02` | + +The volume, the app-secret encryption key and the repo password are all newborn, 36 seconds apart. +Corroborated hub-side (I2): `json_extract(report_json,'$.claimed')` reads **1 → 0 → 1** across the +rebuild on both boxes — demo-hp `0` in 5 reports `2026-08-03 06:12:19`–`06:21:03`; demo-felhom `0` in 4 +reports `07:17:54`–`07:19:46`. `Claimed` is set only by `SetClaimed()`, whose sole caller is the claim +submit handler (`web/claim.go:349`), so the return to `1` was a **human re-claim**, not automation. + +> **Note, not acted on:** a fresh `encryption.key` means every pre-rebuild `app.yaml` `ENC:` secret is +> undecryptable — the C-1 circular dependency in `_recovery-inventory-2026-07-28.md` fired on both +> demo boxes this week. That is outside R-193's scope and is recorded as an observation (§13). + +--- + +## 6. Q4 — what the stale-escrow detector actually covers + +### RULING: **the prompt's premise conflates two different mechanisms. `escrow_confirm.go` DOES fire on a mint. But it fires into a log file, and the remedy it prescribes destroys the last copy of the old key.** + +**The conflation, stated because it matters.** Yesterday's spike's sentence *"wired to the one path +that doesn't change the key and absent from the one that does"* is about the **hub's `escrow_stale` +event**, emitted by `ReissueCredentials` (`hub/internal/offsite/offsite.go:198-201`) — that is R-196, +and it stands. It is **not** about `controller/internal/report/escrow_confirm.go`. Two different +detectors; only one of them was ever claimed to be misplaced. + +**What `escrow_confirm.go` does on a mint, from source.** After a rebuild the apply creates a target +with `EscrowState: "pending"` (the `cur == nil` fallthrough, `offbox.go:482-494`). So `Pending()` is +true, the pending branch runs (`escrow_confirm.go:86-107`), the hub's hash (sealing the OLD password) +is compared against `LocalHash()` (the NEW minted one), they differ, and it logs: + +``` +[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password +(hub hash %.12s… != local %.12s…) — run the escrow ceremony … staying pending +``` + +**So: confirmed, it fires.** The detector is not absent from the mint path. + +**Three limits, each of which matters more than the fact that it fires.** + +1. **It is a controller-local log line.** No event, no e-mail, no hub-side signal. `StaleBlob()` — the + flag the web card renders — is set **only** by the escrowed branch (`reconcileEscrowed`, + `:130-165`); the pending branch sets no display flag. The operator learns nothing. +2. **Its prescription is the destructive act.** *"run the escrow ceremony"* → a new ceremony → + `SaveHostEscrow` overwrites `host_escrow.identity_blob` → `demoteCurrentEscrowTx` does not carry it + → **the old repo password ceases to exist**. The detector correctly identifies the situation and + then tells the customer to close the door. +3. **It cannot distinguish the two situations that need different answers** — "you re-keyed + deliberately" and "your machine was rebuilt and your history is about to be orphaned" produce the + identical warning. + +So: **less needs building than the prompt feared for the *detection*, and more needs building for the +*consequence*.** The comparison exists; what does not exist is a path from that comparison to either a +signal or a recovery. + +--- + +## 7. Q5 — is the superseded-escrow retention real, and what does it retain? + +### RULING: **the retention is real, it retains the wrong blob, and nothing can read it back.** + +**What it retains — source and live, agreeing.** `demoteCurrentEscrowTx` (`store/store.go:2547-2556`): + +```sql +INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at) +SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now') +FROM host_escrow WHERE host_id = ? +``` + +`identity_blob` is not in the column list. Live schema (I2, `PRAGMA table_info`): + +| table | columns | +|---|---| +| `host_escrow` | `host_id, blob, key_fingerprint, posture, created_at, updated_at, **identity_blob**, directive_json, restic_pw_sha256, stale_at` | +| `host_escrow_superseded` | `id, host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at` | + +**`blob` is the K-escrow — the PBS datastore encryption key**, wrapped by the PBS-native scrypt path. +**`identity_blob` is the age-wrapped `IdentityBundle`** — the one carrying `ResticRepoPassword` +(`felhom-agent/internal/escrow/identity.go:34-39`), plus `TunnelToken`, `PBSToken` and +`WGPrivateKey`. Live sizes confirm they are different objects: both hosts' current rows hold +`blob` = 383 B **and** `identity_blob` = 572 B; both superseded rows hold `blob` = 383 B and nothing +else. + +**So, exactly what a customer with the matching OLD recovery code could recover from a retained blob:** + +- ✅ the PBS datastore encryption key **K** — i.e. the Plane-2 whole-guest offsite ciphertext; +- ❌ **not** the restic repository password — the Tier-3 offsite app-data history; +- ❌ not the tunnel token, not the PBS access token, not the WireGuard private key. + +**And there is no read path.** I4: `ListSupersededEscrow` (`store/store.go:2617`) has **zero +production callers** — only two tests. The only production surface is `CountSupersededEscrow`, used +for a number on the host page (`web/hosts.go:573`) and a log line (`api/handler.go:1151`). The schema +comment says so itself: *"The ACK and restore-serving read `host_escrow` (the CURRENT row) — never +this table."* Retrieval is `sqlite3` on the hub pod, by hand, by the operator. + +**The claim this contradicts.** `00-capability-map.md` (escrow-ceremony row) records hub v0.60.0 as: + +> *"a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS +> it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code +> (turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable')."* + +The same claim appears in the schema comment (`store.go:370-375`) and, in Hungarian, on the customer's +own orphan card (`controller/internal/web/templates/backups_remote.html:66,69`): + +> *„A meglévő mentések nem sérültek — a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók +> lehetnek."* · *„A régi előzmény félretéve marad (nem törlődik), és a hozzá tartozó helyreállítási +> kóddal később visszaállítható lehet."* + +**For the offsite restic repository — the incident it names — that is false.** The retained blob does +not contain the key that opens it. This is the **ninth** entry in `CLAUDE.md`'s table of comments +asserting an invariant the code does not provide, and the first that is also **customer-facing copy**. +→ **R-198**, and the capability-map correction in §9. + +--- + +## 8. Q6 — has any part of this ever been exercised? + +**Method:** read every audit, validation doc, findings file and REPORT that mentions escrow consume, +identity-consume, re-enroll or inject-password; then check what the cited evidence actually covers. + +### Proven + +| what | evidence | scope of the proof | +|---|---|---| +| ceremony (create + seal + upload) | `00-capability-map.md` escrow row; `tests/VALIDATION-n100-rehearsal-2026-07-18.md` S6 — customer drove the wizard, R claimed one-shot 16:56:39, auto-confirmed 16:56:41 | the **create** side only | +| identity-bundle round-trip with a real R on a secret-less box | `tests/slice10d-identity-restore-spike-findings.md` §1, 2026-06-10 — wrap→unwrap byte-identical (`sha256` match); wrong-R **fails closed**, no plaintext emitted | **PRE-FORK-4.** The bundle then was `{tunnel_token, pbs_token}`. `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** (`felhom-agent/CHANGELOG.md:2344`) — a month later. **The proof does not cover the field this whole question is about** | +| `ResticRepoPassword` survives the wrap/unwrap | agent v0.77.0 CHANGELOG: *"`IdentityBundle` round-trip carries `ResticRepoPassword` byte-exact + not-in-blob + wrong-R fails closed"* | **unit test only** | +| `InjectOffboxPassword` places and refuses correctly | `controller/internal/backup/offbox_test.go:952-990`, incl. the companion showing a fresh mint without inject | **unit test only** | + +### Never exercised — named in those words + +- **A blob has never been served to a box.** Link 6 has no client (§3). +- **A fork-4 bundle has never been unsealed with a real R outside a unit test.** The only production + caller is a `--selftest` mode; the one live consume ever prepared (S5 Part 4-B, 2026-07-04) was + explicitly **deferred and never run** — its own resume note still reads *"DEFERRED 2026-07-04, not + run"* (project memory; `06-offsite-connectivity.md:327` marks the destructive drill operator-gated). +- **A recovered repo password has never been injected into a controller.** +- **An existing offsite repository has never been reopened with a recovered password.** +- **No restore of any kind has ever been performed from a recovered secret.** + +`_recovery-inventory-2026-07-28.md` already said most of this and said it accurately — A.2.7 records +*"the **consume** side is proven at spike level … the destructive in-place drill is explicitly +operator-gated and unrun"*, and C.1 row 1 says *"escrow **creation** proven live; **consume never +exercised destructively**"*. **That document is right and this session confirms it.** What it did not +catch — because it was written from `host_escrow`, not from the supersession path — is Q5. + +**Capability-map correction owed:** not for a "recovery proven" claim (no row makes one), but for the +**retention** claim quoted in §7. See §9. + +--- + +## 9. Q7 — the operator's ruled shape, assessed row by row + +> **Operator ruling, 2026-08-04, verbatim:** *If a node is a fresh install AND the hub has a recovery +> blob, then the controller should yell that recovery is available, and provide a form for the customer +> to enter the recovery key. After unlocking the blob, the controller should show what will be +> recovered before proceeding.* + +| The ruling requires | Exists? | What it needs, and what it costs | +|---|---|---| +| the controller knows it is a **fresh install** | **YES, at the right line** | `os.Stat(m.offboxPwPath())` → `IsNotExist` is already the mint branch (`offbox.go:392`). It is trustworthy as *"this controller has no repo password"*, which is the only fact the decision needs; it does **not** by itself distinguish first-ever install from rebuild — that is what the next row supplies. Corroborators exist but are weaker: `Claimed` is set-only and was measured re-set by a human within minutes (§5); a fresh `encryption.key` is a stronger signal but is not currently read by anything. **Cost: zero — the branch already exists.** | +| the controller knows **the hub has a blob** | **YES on the wire, NO in the code** | `EscrowStatus{IdentityBlobPresent, ResticPwSHA256, CreatedAt}` arrives on every report ACK and is discarded when no offbox target exists (§5). **Needs:** persist it (the `ClaimSync` set-only-by-generation pattern is the precedent, `claim_sync.go:39-53`). **Cost: small — controller only, one settings field, one reconciler, no hub change.** | +| it can **yell** rather than silently mint | **NO** | The refusal belongs **inside `WriteOffboxSecrets`, at the mint branch** — the one place that decides. Returning a new sentinel (`ErrOffboxRecoveryAvailable`, sibling of `ErrOffboxOrphaned`) propagates through `ApplyOffsiteTarget` → `ConfigureOffbox` → the apply-bridge, which already treats a failed configure as "persist nothing, retry next cycle" (`offsiteapply.go:5-6, 256`) — so a refusal is **fail-safe by construction**, not a new failure mode. Meanwhile the customer sees a card on `/backups/remote` (the orphan card at `backups_remote.html:60-70` is the exact precedent) and offsite runs stay blocked, which they already are (`OffboxRunnable`). **Cost: small-medium — controller only.** **Design hazard to name:** a first-ever install of a *re-used customer record* would also trip this; the refusal must be dismissible ("start fresh anyway"), and dismissal must be the act that mints. | +| a **form takes R** | **NO — and R is not a password** | The controller UI **emits** R (`backups_escrow.html:235-236`, `/backup/escrow/…/claim` returns it once) and has **never accepted** one. The only thing resembling an entry point is `POST /backup/offbox/inject-password`, which takes a **64-hex repo password** (`offboxRepoPwPattern`, `offbox.go:543`) — a different secret at a different layer — and **has no form in any template** (§3 link 9). **Cost: small for the form; the hard part is the next row.** | +| the blob is **unsealed** with R | **NO — and there is a real boundary** | Unsealing is `age -d`. `ageBinary = "/usr/bin/age"` (`identity.go:24`) and the code live in the **agent**. The **controller image does not ship `age`** (`controller/Dockerfile:56-70` — restic, openssh-client, sshpass, sqlite3, docker-cli; no age) and must not: the agent is the privileged tier by design. **The crossing already has a transport and a precedent:** the pinned local API carries `POST /escrow/ceremony` + `GET /escrow/ceremony/status` + `POST /escrow/ceremony/claim` (`localapi/server.go:524-527`) — i.e. the agent **already** handles R server-side for the create direction and hands the result back exactly once. **Needs: one new agent local-API endpoint** (`POST /escrow/unseal`: blob + R in, the single `restic_repo_password` field out, R never logged, R never persisted) — the exact mirror of the claim endpoint. **Plus** a way for the blob to reach the agent: the agent's hub client has no escrow **GET** (only the `PUT` at `main.go:2937`), and `dr.go`'s re-enroll is the wrong tool (it rotates the host API key and needs operator-armed recovery mode). **Needs: a second small hub endpoint** — `GET /hosts//escrow`, per-host-key authed, serving opaque bytes to the box that owns them. Zero-knowledge is untouched: the hub still never decrypts, and the bytes are useless without R. **Cost: medium — three repos, two new endpoints, both narrow.** | +| **"show what will be recovered" before proceeding** | **the primitive EXISTS; the flow does not** | `restic snapshots --json` + `restic stats --mode raw-data --json` are already how the box counts snapshots and sizes the repo (`offboxRecordStats`, `offbox.go:1234-1265`). Both are **read-only against the repository** — nothing is written, no lock is taken, no `init`. So a preview can show: **snapshot count, per-snapshot time, host, tags (= app names), paths, and the deduplicated repo size** — all before committing. **Committing requires only that the recovered password be written to `offboxPwPath()`**, and even that is reversible in principle. **The clean shape:** unseal → hold the recovered password **in memory** → run the two read-only probes against the existing repo → render → and only on the customer's confirmation call `InjectOffboxPassword`. **Cost: small — one handler, one template, no new restic capability.** **Caveat, measured not assumed:** `offboxRecordStats` today also persists `RepoSizeBytes` into settings as a side effect; a preview path must not reuse it verbatim. | +| **proceeding** actually recovers | **NO end-to-end** | `InjectOffboxPassword` → next `WriteOffboxSecrets` keeps the file → `ensureOffboxRepo` probes and clears `RepoState`. Every piece is implemented; **none has ever run against a recovered password** (§8). **Cost: zero new code, but it is precisely what the drill in §10 has to prove.** | + +### Things the ruling implies that do not exist, priced + +| missing piece | cost | note | +|---|---|---| +| persisted ACK escrow status + fail-closed mint refusal | **S** | controller only; the `ClaimSync` pattern applies directly | +| the "recovery is available" card + dismissal | **S** | one template, one flag; the orphan card is the template precedent | +| a customer-facing **recovery-code** form (distinct from the password form) | **S** | the password form's handler exists but has no UI; both need building | +| agent local-API `POST /escrow/unseal` | **M** | new privileged surface; mirror of `/escrow/ceremony/claim` | +| hub `GET /hosts//escrow` (opaque, per-host key) | **S** | new read surface on the hub; zero-knowledge preserved | +| preview handler + template driven by read-only restic probes | **S** | primitive already exists | +| **carrying `identity_blob` into `host_escrow_superseded`** | **S** | one column, one `SELECT` list — **and without it the whole feature protects only boxes that have not yet re-escrowed** (→ R-198). This is a prerequisite, not a nice-to-have | + +### The security question the ruling raises — stated, not answered + +**What gates the surface today.** The controller dashboard is behind `RequireAuth` +(`internal/web/auth.go:52`): a bcrypt password (`effectivePasswordHash()` — customer-set in +`settings.json`, else operator-provisioned in `controller.yaml`), a 7-day session cookie, 5 failed +logins per minute then lockout, and `CsrfProtect` on every mutating route. The claim gate serves only +the claim page on an unclaimed box. So a recovery form would sit **behind the customer's dashboard +password**, on a controller reachable at whatever address that box's dashboard is reachable at. + +**What a preview would expose to someone who guesses or steals R.** Not file contents — but the +snapshot list is not nothing: **backup dates and cadence, the app names (restic tags), the paths, and +the repository size.** That is a map of the household's digital life, and it is precisely the metadata +the zero-knowledge posture otherwise keeps out of everyone's hands including the operator's. + +**Three specifics the operator should weigh, without this session choosing:** + +1. **Guessing R is not the threat; holding the dashboard password is.** R is ten EFF words + (~129 bits, `00-capability-map.md` agent v0.93.0 note) and `age`'s scrypt is deliberately slow — + online guessing is not a route. But **anyone with the dashboard password can already restore live + offsite data** (`_recovery-inventory-2026-07-28.md` A.1.3–A.1.6). The marginal new exposure is + confined to the *orphaned/previous* repository, which today nobody can read at all. +2. **The form is an oracle.** A correct/incorrect R answer distinguishable in the UI turns the + dashboard into a verification service for a stolen recovery code. `UnwrapIdentity` already + fails closed with a single generic error (`identity.go:157`) — the UI must not do better than that, + and should rate-limit like the login path does. +3. **R will pass through the agent.** Whatever the form does, R crosses the local API to a + root-adjacent daemon. The ceremony already does this in the other direction, so the precedent is + set — but it means R is momentarily present in agent process memory on the Proxmox host, which is + the same host D6 says the operator has root on. **This is the same trade as spike option (c), in a + smaller and time-bounded form**, and it should be decided alongside it rather than separately. + +--- + +## 10. Q8 — the drill design + +**Why a deliberate cycle.** No recovery code exists for either demo box, and — per §7 — even one would +no longer open the old repositories. The proof therefore requires creating R, keeping it, and walking +the full circle. **This section is the design; the drill is not run in this session.** + +### Target and preconditions + +| | | +|---|---| +| **Box** | **`demo-hp`** (HP t740, node `felhom-host`, host `demo-hp-bb76ea`). Tier 0 — disposable, and the designated drill host by the 2026-07-25 operator ruling. **Not demo-felhom** — it is the N100 that also carries the ISO/rehearsal lineage; leave one demo box in a known-good state throughout | +| **Guest** | the customer guest on that host (rebuild target) | +| **Data** | one deliberately identifiable app + a sentinel file whose content is a known string and whose sha256 is recorded before the wipe | +| **Prereq** | the box is healthy, offsite target configured and `escrow_state: escrowed`, and its **current** `host_escrow.identity_blob` seals the password of the repo that will be tested | +| **Duration** | ~3–4 h wall clock, of which ~45 min is operator-attended (steps 2, 5, 8, 9) | +| **Blocking dependency** | **steps 8–9 cannot be executed as designed today** — links 6, 8 and 9 of the chain do not exist as automation. The drill as written uses the manual substitutes (§4a) and **that is the point**: it measures the *existing* chain honestly before anything is built | + +### The cycle + +| # | Step | Who | Observable that must be recorded | +|---|---|---|---| +| 1 | Record the starting state: `host_escrow.restic_pw_sha256`, `identity_blob` length, `$.offsite.snapshot_count`, `repo_size_bytes` | CC | a hash and two counters, from the hub DB | +| 2 | **Run the escrow ceremony through the customer wizard and KEEP R** | **HUMAN** | R written to the operator's password manager under a named entry; **its storage location recorded in the drill report — the code itself never** | +| 3 | Deploy an app, write the sentinel file, record its sha256 | CC | sha256 + path | +| 4 | Run an off-site backup through the real UI endpoint; wait for completion | CC | `$.offsite.last_status = ok`, `snapshot_count` increased by ≥1, `last_success` stamped | +| 5 | **Verify the snapshot from outside**: `restic snapshots --json` against the repo | CC | snapshot id + tag + timestamp — the pre-wipe truth to compare against | +| 6 | **Wipe**: destroy the guest and reinstall it (the same procedure that produced the 2026-08-03 rebuild) | **HUMAN** decision, CC executes | new `controller_started`; `claimed` drops to 0; a new `encryption.key` mtime | +| 7 | Let the box come back and reach the mint. **Do NOT run a new ceremony** | CC | the controller log carries the `escrow-confirm` mismatch WARN (§6); `escrow_state: pending`; `host_escrow.identity_blob` **unchanged** (this is the assertion that keeps recovery possible) | +| 8 | **Recover**: export `identity_blob` from the hub, unseal with the kept R, extract `restic_repo_password` | **HUMAN** (R is theirs) | `identity-consume` exits 0; the bundle JSON contains a non-empty `restic_repo_password`; **the value is never printed or logged** | +| 9 | **Inject** the recovered password (`POST /backup/offbox/inject-password`, by hand — there is no form), re-apply, run an off-site backup | **HUMAN**-assisted | `restic cat config` succeeds; **`snapshot_count` ≥ the step-5 count + 1**, not 1 | +| 10 | **Restore the sentinel** through the customer restore flow and compare sha256 | CC | **byte-identical sha256 to step 3** | +| 11 | Record everything, tear down nothing that would hide the evidence | CC | drill report | + +### The single observation that makes this a FAILURE rather than a pass + +> **Step 10's sha256 does not match step 3's — or step 9 reports a snapshot count of 1.** + +A snapshot count of 1 means a **new, empty history** was started over the old one: the repository did +not reopen, and every earlier snapshot is orphaned. **Explicitly: "the repository opened" is not the +pass condition.** `restic cat config` succeeding proves the password is right; it does not prove the +customer's data came back. Only the sha256 comparison does. + +### Secondary failures worth recording separately (each is a finding, not a pass/fail) + +- Step 7 finds `identity_blob` **changed** — something re-escrowed automatically → the recovery window + closes without a human act (would be new and serious). +- Step 8's bundle has an **empty** `restic_repo_password` — a pre-fork-4 or hash-less blob shape. +- Step 9's `ensureOffboxRepo` classifies `orphaned` despite the correct password → the classifier or + the transport, not the key. + +### What could make it fail for uninteresting reasons + +The `age` binary absent on the machine running step 8; the hub DB export producing a truncated blob +(use `writefile`, verify the byte length against `length(identity_blob)` = 572); the CSRF token in +step 9 (`felhom_session` is dropped by curl's cookie jar — project memory +`controller-api-headless-access`); and the R-71a settle gate deferring the apply in step 9 (wait for +its GO line rather than retrying). + +### Rollback, and whether the loss is acceptable + +**If the recovery does not work, what has been lost is:** demo-hp's guest, its app data since the last +whole-guest archive, and its off-site history. **On that box that is acceptable** — it is Tier 0, +explicitly disposable, and it is the box the target-selection runbook names for drills. **The +rollback** is a whole-guest restore from the local/PBS archive taken immediately before step 6, which +is a proven path on this fleet (restore-test passed on demo-hp's host tier in 109.3 s on 2026-08-04). +**Take that archive deliberately as part of step 6 and verify it exists before destroying anything** — +do not rely on the nightly. + +**Not acceptable, and therefore not proposed:** running this on demo-felhom (leaves no healthy demo +box), on DooPlex (Tier 2, it *is* the recovery chain), or on any real customer's box. + +--- + +## 11. Every query used + +| # | Source | Query / read | Value read | +|---|---|---|---| +| 1 | git | `git -C rev-parse HEAD`; `git status --porcelain` | the three commits in §1; all empty | +| 2 | hub DB | `PRAGMA integrity_check` | `ok` | +| 3 | hub DB | `SELECT MAX(received_at) FROM host_reports; SELECT datetime('now')` | `2026-08-04 10:00:40` / `2026-08-04 10:03:48` | +| 4 | hub DB | `SELECT host_id, length(blob), length(identity_blob), substr(restic_pw_sha256,1,12), created_at, updated_at FROM host_escrow` | `demo-felhom-8363b5, 383, 572, c60c8bc737a6, 2026-08-04T07:20:07Z, 2026-08-04 07:20:08` · `demo-hp-bb76ea, 383, 572, 8a9e33aa4da6, 2026-08-04T07:15:36Z, 2026-08-04 07:15:36` | +| 5 | hub DB | `SELECT id, host_id, length(blob), substr(restic_pw_sha256,1,12), created_at, superseded_at FROM host_escrow_superseded` | `3, demo-hp-bb76ea, 383, 8e03eddf9ff7, 2026-07-23T10:01:17Z, 2026-08-04 07:15:36` · `4, demo-felhom-8363b5, 383, 48741892f0ef, 2026-07-21T08:38:33Z, 2026-08-04 07:20:08` | +| 6 | hub DB | `PRAGMA table_info(host_escrow)` / `(host_escrow_superseded)` | the two column lists in §7 — `identity_blob` present in the first, **absent from the second** | +| 7 | hub DB | latest report per customer: `json_extract(report_json,'$.claimed' / '$.offsite.escrow_state' / '.snapshot_count' / '.repo_state' / '.last_status' / '.last_run')` | demo-felhom `2026-08-04 09:49:42`: `1, escrowed, 0, NULL, NULL, NULL` · demo-hp `2026-08-04 09:56:57`: `1, escrowed, 0, NULL, NULL, NULL` | +| 8 | hub DB | `claimed` grouped over reports since `2026-08-03 05:00` | demo-hp: `1` ×124, **`0` ×5 (06:12:19→06:21:03)** · demo-felhom: `1` ×120, **`0` ×4 (07:17:54→07:19:46)** | +| 9 | hub DB | `SELECT customer_id, event_type, severity, created_at FROM events WHERE created_at > '2026-08-03 00:00'` | no `escrow_stale`, no `offsite_reissued`, no `offbox_repo_orphaned`, no claim event in the window; `controller_started` ×4 demo-hp / ×4 demo-felhom around the rebuilds | +| 10 | guest 9201 (I3) | `ls -la --time-style=full-iso …/felhom-controller-data/_data/data{,/offbox}` | the mtime table in §5 | +| 11 | source (I4) | `grep -rn "InjectOffboxPassword" --include=*.go .` | 1 definition, 1 handler caller, 5 test uses, 1 comment — **no production caller besides the handler** | +| 12 | source (I4) | `grep -rn "inject" internal/web/templates/` | one unrelated hit (an XSS comment) — **no form** | +| 13 | source (I4) | `grep -rni "reenroll\|re_enroll" --include=*.go .` in `felhom-agent` | **zero hits** | +| 14 | source (I4) | `grep -rn "re-enroll\|recovery-mode\|restore-directive" --include=*.sh --include=*.md --include=*.py` workspace-wide | only design docs + project memory; no script, no runbook step | +| 15 | source (I4) | `grep -rn "ListSupersededEscrow"` in `felhom.eu` | 2 test callers, 0 production | +| 16 | source (I4) | `grep -rn "\.Escrow\b" --include=*.go .` in `felhom-controller` | one hit: `cmd/controller/main.go:641` | +| 17 | source | `felhom-agent/CHANGELOG.md` | `v0.77.0 — fork-4: escrow the offsite restic repo password under R (2026-07-09)` | +| 18 | source | `controller/Dockerfile:36-70` | runtime package list — **no `age`** | + +--- + +## 12. What could not be established + +Named, not guessed. + +1. **The 2026-08-05 ~02:15 UTC scheduled run's outcome.** Still unmeasured — both boxes became + runnable only on 2026-08-04 morning and neither has run since (query 7: `last_run` absent on both). + Yesterday's spike predicts `ErrOffboxOrphaned`; that prediction is **unchanged and still unproven**. +2. **Whether demo-hp's data dir was fresh.** Established for demo-felhom by direct mtime measurement + (I3). demo-hp was not measured — it has no baked SSH key and reaching it needs the break-glass + credential, which this read-only session did not use. The hub-side `claimed 1→0→1` signature is + identical on both boxes, which is suggestive, not decisive. +3. **Whether the orphaned ciphertext still exists at the provider.** Unchanged from yesterday: reading + the endpoint needs the SFTP credential, which was not touched. +4. **Whether any pre-2026-07-21 escrow generation ever existed for demo-felhom.** The superseded table + holds two rows in the entire database; earlier generations are not retained (and, per §7, would not + have carried the identity blob anyway). +5. **Whether `age` is present on any machine that would run step 8 of the drill.** Not checked — the + agent declares `/usr/bin/age` as a runtime dependency but this session did not verify it on either + demo host. + +--- + +## 13. Observations — noticed, not acted on + +1. **A fresh `encryption.key` on every rebuild** (I3: 32 B, mtime `2026-08-03 07:17:26`) means every + pre-rebuild `app.yaml` `ENC:` secret on that box is undecryptable. This is exactly circular + dependency **C-1** in `_recovery-inventory-2026-07-28.md`, and it fired live on both demo boxes this + week without anyone naming it. It is a strictly larger blast radius than the offsite repo and it is + **not** covered by R-193. Worth its own item if the operator agrees it is not already implied by C-1. +2. **`Claimed` returning to `1` within minutes** of a rebuild, with no event of any kind recording it, + means "the customer claimed this box" has no audit trail. There is no `customer_claimed` event type. +3. **`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting** — harmless today, + a trap for anyone reusing it as a read-only preview primitive (§9). +4. **`handleReEnroll` returns the new API key in the response body** (`dr.go:148`) while also logging + the rotation. Not a defect (the box needs it), but it is a secret in an HTTP response on an endpoint + nothing currently calls — worth a second look before anything starts calling it. + +--- + +## 14. The operator's decisions — stated, and NOT answered here + +1. **Run the drill (§10) or not?** It is the only thing that converts *"the pieces look right"* into + *"a customer got their data back"*. It costs demo-hp for an afternoon and, if it fails, that box's + history. **Recommendation: yes — but fix `identity_blob` retention (R-198) first**, because a drill + that walks the current chain will pass or fail on a chain that is missing a link everyone believed + was there. +2. **The orphaned snapshots at the endpoint** (40.9 MB + 1.14 GB, presumed present, §12.3). They are + **unrecoverable by two independent mechanisms** now: no recovery code was kept, *and* the sealed + passwords were destroyed by the 2026-08-04 re-ceremonies. They consume the 50 GB soft quota + indefinitely; nothing prunes a moved-aside restic repo. **Delete, or leave?** This session did not + act on it and states it as the operator's. +3. **The unchanged R-193 question** — agent-retained repo password (spike option c) versus accepting + that every rebuild starts a new repository. **This session adds one input to that decision:** the + "accept it, provided the system says so loudly" branch currently rests on a retention feature that + does not retain the relevant secret. Under option (d)-alone as it stands today, *"the old history + stays recoverable with your recovery code"* is not true. Either R-198 is fixed, or that branch has + to be re-described honestly to the customer. + + diff --git a/documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md b/documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md index 8a81fc4..49e016b 100644 --- a/documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md +++ b/documentation/audits/SPIKE-offsite-credential-recovery-2026-08-04.md @@ -10,6 +10,39 @@ what it measured) · **Class:** spike — **no production code shipped for R-193 --- +> ### ANNOTATION 2026-08-04 (later the same day) — read this before acting on §Q8 +> +> A follow-up recon traced the escrow→recovery chain link by link +> (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **Two corrections to this document, and they pull +> in opposite directions.** +> +> **(1) Q8 candidate (b) — "not implementable as specified" — is TOO STRONG and is overturned in +> part.** Zero-knowledge means the *hub* cannot open the blob; the *customer* can, with R, which is +> what the ceremony exists for. The mechanism to hand a rebuilt box its old repo password back — +> customer-present, not unattended — is a real design, and the operator has since ruled on its shape. +> What (b) got right is narrower than what it said: **an UNATTENDED rebuild cannot recover from a +> zero-knowledge escrow.** A customer-present one can. The recon prices that shape row by row. +> +> **(2) A LOAD-BEARING FACT THIS SPIKE DID NOT CHECK, and it makes (d)-alone weaker than ranked +> here.** `host_escrow_superseded` **has no `identity_blob` column**, and `demoteCurrentEscrowTx` +> (`hub/internal/store/store.go:2547-2556`) copies only the K-escrow `blob`. So the retained +> "superseded" blob holds the **PBS datastore key**, not the **restic repository password**. The +> retention feature does not retain the secret it was built to retain, and the **next escrow ceremony +> — which the system tells the customer to run whenever a rebuild leaves the escrow `pending` — is +> what destroys the last copy.** Both demo boxes crossed that line on 2026-08-04 at 07:15:36 and +> 07:20:08. → **R-198**. +> +> **Consequence for this document's recommendation.** "Ship (d), then decide (c)" still holds, but +> (d)'s implicit promise — *the old history stays recoverable with the recovery code* — is **false as +> the code stands**, and is repeated verbatim to the customer in Hungarian on the orphan card. (d) is +> not complete without R-198. +> +> **Unchanged and still correct:** Q1, Q2, Q4 (R-196), Q5, Q6, Q7 (R-192) and the Q8 (a)/(c)/(d) +> analyses. **Q3 remains UNMEASURED** — neither box has run since; the 2026-08-05 ~02:15 UTC run is +> still the decisive one. + +--- + ## 0. Why this exists Two candidate fixes were named when R-193 was filed, and neither could be weighed, because two prior @@ -412,6 +445,12 @@ re-keyed, `host_leaf_changed` 2026-08-03 06:09:40), so re-enroll returns the exi ### (b) The credential becomes recoverable from escrow at re-bootstrap +> **ANNOTATED 2026-08-04 (recon) — this ruling is overturned in part. See the annotation at the top.** +> The constraint is real but narrower than written: an **unattended** rebuild cannot recover from a +> zero-knowledge escrow; a **customer-present** one can, and the operator has ruled on that shape. +> The recon also found that the chain this option would use is clientless at three links and lossy at +> a fourth (`RECON-offsite-dr-chain-2026-08-04.md` §3–§4). + - **BLOCKED AS STATED, and this is a hard constraint, not an estimate.** The escrow blob is **R-wrapped and zero-knowledge** — the hub holds opaque bytes and has no recovery code (`api/handler.go`, and D6 in `CONTEXT.md` S-3). **A rebuilt box cannot unseal it without the diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 572077e..4257221 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -39,8 +39,12 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-195** | ~~**A customer with no machine ever bound e-mailed an `expected_dbdump_missed` ERROR every morning.** `david` — a real prospective customer whose record was created 2026-08-01 16:51:49 with **hosts=0, host_deletions=0, host_reports=0, reports=0** — raised the alarm at 03:00 UTC on 08-02, 08-03 and 08-04~~ | **SHIPPED** (hub **v0.92.0**, 2026-08-04) | — | **The mechanism, and it is the interesting half: the skip that protects every other silent customer is keyed off having reported at least once.** `CheckBackupDeadlines`' down-skip reads `StalenessChecker.GetState()`, whose map is seeded from `store.GetCustomers()` — a query **over the `reports` table**. A customer with zero reports appears in no row, gets no state, and `GetState()` returns `""` rather than `"down"`. The skip therefore misses exactly the customer it would most obviously cover. (Corroborated live: `peti-felhom` is active with a host deleted 2026-07-15 and **does not** alarm — it has 482 old reports, so it is `down` and skipped.) The backup half was already safe (`reportJSON == ""` → skip); the DB-dump half had no guard at all. **Fix:** `store.HasEverBoundHost(customerID)` = a live `hosts` row **OR** a `host_deletions` tombstone, consulted once per customer at the top of the deadline loop. **The discriminator is deliberately "was a host EVER bound", NOT "has a report arrived"** — a box that was installed and went silent is a real fault and must keep alarming; that is the case the change could break and it has its own test. Fail-**open** on a read error (an unreadable binding must never SUPPRESS a real alarm), and the deferral is LOGGED with its own counter (the v0.73.0 Part-7 precedent: a quiet check must not look like a check that did not run). The anchored-verdict structure is untouched. **Red-proof observed:** deleting the guard fails `TestCheckBackupDeadlines_NeverBoundHost_Silent` with `got [expected_dbdump_missed]` — verbatim the event `david` sent three mornings running. `david`'s record was NOT modified; the record was correct and the alarm was what was wrong | — | | **R-196** | **`escrow_stale` is wired to the ONE path that does not change the repo password, and absent from the path that does.** `ReissueCredentials` (`hub/internal/offsite/offsite.go:150-228`) resets **only** the Hetzner sub-account/box password and stages a fresh one-time secret — it contains no reference to a restic password and *cannot*, since that password is generated on the box and never leaves it except into the R-wrapped escrow. Yet it calls `MarkEscrowStale` on the stated grounds that *"the restic repo password just changed"* (`offsite.go:198-201`), and the same false premise is repeated at `api/handler.go:1067-1069` and in R-39's record | **OPEN** | — | **This is the EIGHTH entry in `CLAUDE.md`'s table of comments asserting a guarantee the code does not provide** — and the first where the comment factually describes a *different function*. It survived because its EFFECT (a stale escrow) is real, so nobody checked its CAUSE. **The live defect, not a documentation nit:** on the ordinary Re-issue shape (a consumed-but-failed install on a box that still holds its `repo_password` file) the box re-applies, `WriteOffboxSecrets` finds the file present and **keeps it**, the repo password is unchanged — and the hub has told the customer in Hungarian that their recovery escrow is stale and asked them to re-run the ceremony. **A false staleness alarm and an unnecessary ceremony.** **The inverse is the worse half and is R-193's:** demo-felhom's repo password *did* change on 2026-08-03 with **no Re-issue anywhere in its history** (its only `escrow_stale`/`offsite_reissued` pair is dated 2026-07-21 08:29:29) and therefore **nothing marked its escrow stale for 13 h**. **Fix shape:** mark the escrow stale on the evidence that it IS stale — a changed `restic_pw_sha256` (→ R-197) — not on a Re-issue; and correct all three comments in the same commit. **Not established, so not asserted:** whether the 2026-07-21 Re-issue on demo-felhom re-sealed an unchanged password (the predicted false-staleness shape) — `host_escrow_superseded` holds only two rows in the whole DB, both from 2026-08-04, so the prior generation is not retained. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q4 | CC | | **R-197** | **The hub holds both halves of the evidence that a box's offsite DATA key changed, and reads neither.** `restic_pw_sha256` is stored on `host_escrow` and carried to `host_escrow_superseded` on every re-escrow. Comparing the two is what let this spike answer its hardest question in one query — and **nothing in the hub does it** | **OPEN** | — | **Why this is the cheapest real fix on the table.** A changed repo password means the previous offsite repository is now unopenable by the box, i.e. the customer's off-site history is orphaned. That is the single most consequential state change in the backup system, it is **already fully observable from data the hub owns**, and today it produces **no event, no e-mail, no card and no log line** — demo-felhom's went unremarked for 13 hours and would have gone unremarked indefinitely had this spike not run. **Two-line verdict:** on `SaveHostEscrow`, if the incoming `restic_pw_sha256` differs from the row being superseded, emit a distinct operator event naming the orphaned generation. **Deliberately an EVENT, not a heal** — nothing should act on this automatically until R-193's (c)-vs-accept decision is taken; the point is that the operator learns on the day. **Pair with R-196**, which is the same signal aimed at the right trigger. **Generalises past this row:** *a comparison the system could be making from data it already stores, and is not, is a silence with no cost of entry* — cf. R-190's store-grant probe, where the state was read and the TRANSITION was not. Source: `audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q8 option (d) | CC | -| **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **OPEN** | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the same spike:** under R-193's Q2 finding a successful auto-restage would have restored demo-hp's TRANSPORT while the box minted a new repo password anyway — the heal can protect the plumbing and **cannot** protect the data, and had it fired on 2026-08-04 both boxes would have looked healthy while their snapshots were orphaned. **That is strictly worse than the current loud failure.** Whatever shape the fix takes must say so in its message. | CC | -| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **OPEN** | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTP access key…)"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the one a rebuild destroys, and **nothing automatic ever restages it**: `WriteOffboxSecrets` mints a fresh 256-bit password whenever `/offbox/repo_password` is absent (`offbox.go:392`), and the only recovery path, `InjectOffboxPassword`, has **exactly one caller in the whole repo** — a web form a human pastes into (`web/offbox_handlers.go:189`). **(2) MEASURED, WITHOUT TOUCHING A BOX:** the hub already stores `restic_pw_sha256` on both the live and the superseded escrow, so the question is a hash comparison. demo-hp `8e03eddf…` → `8a9e33aa…`; demo-felhom `48741892…` → `c60c8bc7…`. **Both boxes minted a new repository password.** **(3) THE CONTRAST IN THIS ROW IS FALSE FOR THE DATA.** demo-felhom's 76-second "lucky" recovery restored **delivery only** — its pre-rebuild object carried **36 snapshots / 1.14 GB** (`repo_size_bytes 1136685919`) and it has reported `snapshot_count: 0` in all 109 reports since, with a changed repo password and **nothing marking its escrow stale for 13 h**. Both boxes lost repository continuity; one loudly, one silently, and **the silent one is worse**. **(4) Q3 IS UNMEASURED AND THE BINARY WAS WRONG.** Neither box could run on 2026-08-04 02:15 UTC (demo-hp had no target at all until 07:15; demo-felhom's target was `escrow_state: pending`, which `OffboxRunnable` blocks) — **the decisive run is 2026-08-05 ~02:15 UTC**. Source predicts a **third** outcome, neither 15 nor 1: same account + same repo path (`u629488-sub3:/home/felhom-repo`, and `repoPath` is a compile-time constant) + new password + `claimed: 1` ⇒ `ensureOffboxRepo` classifies **orphaned** and returns `ErrOffboxOrphaned` — **the run refuses and shows the orphan card**, it does not start a silent fresh history. **Record which of the three actually occurs; a prediction from source is not a measurement.** **(5) CANDIDATE (b) IS NOT IMPLEMENTABLE AS STATED** — the escrow is R-wrapped/zero-knowledge and the hub has no recovery code, so "recoverable from escrow at re-bootstrap" describes a customer-present ceremony, i.e. the manual form that already exists. **(6) CANDIDATE (a) ALREADY EXISTS AND IS WIRED TO THE WRONG EVENT:** `reissueOnReenroll`'s **F3 leg** does exactly this (`api/handler.go:1051-1084`) but sits behind `handleHostEnroll`'s mint-once-reuse short-circuit (`if existing != nil { return }`), and a **guest** rebuild leaves the `hosts` row intact — so F3 is never reached. **(7) A NEW CANDIDATE (c), not previously named and recommended second:** the **agent survives a guest rebuild**, already receives the repo password over the pinned local API (`POST /escrow/stage-secret`) and already writes it to a fixed 0600 path — it merely **wipes** it after the ceremony. Retaining it and serving it back is the only candidate that addresses the irreplaceable secret, and every seam it needs exists. Its cost is one deliberate trade the operator must make (a copy of the data key at rest on the Proxmox host — see D6). **SPIKE RECOMMENDATION: ship the honesty pass (R-196 + R-197) now; then decide (c). Do NOT ship (a) first — it would have hidden this.** **NOT CLOSED — WAITING-ON-OPERATOR for the (c)-vs-accept-it decision, stated at the end of the spike.** | CC + operator | +| **R-198** | **The hub's superseded-escrow retention does NOT retain the offsite repository password — and the ceremony the system tells the customer to run is what destroys the last copy.** `host_escrow_superseded` has **no `identity_blob` column**, and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`) copies only `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256`. `blob` is the **K-escrow** (the PBS datastore key, PBS-native scrypt); the **restic repo password lives in `identity_blob`** (`felhom-agent/internal/escrow/identity.go:34-39`, age-wrapped `IdentityBundle`). Measured live: both hosts' current rows hold `blob`=383 B **and** `identity_blob`=572 B; both superseded rows hold `blob`=383 B and nothing else | **OPEN** | — | **This is the NINTH entry in `CLAUDE.md`'s table of comments asserting an invariant the code does not provide, and the first that is ALSO customer-facing copy.** The claim appears three times: the schema comment (`store.go:370-375`, *"so the old passphrase stays customer-R-recoverable … turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable'"*), the capability map's escrow row, and — in Hungarian, to the customer, on the orphan card — `controller/internal/web/templates/backups_remote.html:66,69` (*„a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók lehetnek"*). **For the offsite restic repository, the incident it names, all three are false.** **Why it is worse than a missing column:** a rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`, `offbox.go:570`), the card says „Helyreállítási kód szükséges", and the controller's own detector logs *"run the escrow ceremony"* (`report/escrow_confirm.go:100`) — so **the prescribed remedy is the act that overwrites `host_escrow.identity_blob` and loses the old password forever**. Both demo boxes crossed that line on **2026-08-04 at 07:15:36 (demo-hp) and 07:20:08 (demo-felhom)**. **This is an independent, stronger reason the 51 orphaned demo snapshots are unrecoverable than "nobody kept the recovery codes" — keeping R would not have helped.** **Fix shape (small):** add `identity_blob` to the superseded table and to `demoteCurrentEscrowTx`'s SELECT list; pin it with a test that asserts the CONSEQUENCE (a superseded row can still yield a repo password) rather than the mechanism. **Then correct all three claims in the same commit** — including the Hungarian card, which must not promise what the system cannot do. **Prerequisite for R-193's (d)-alone branch** and for the drill. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §7 | CC | +| **R-199** | **The hub serves recovery blobs on two endpoints that have no client anywhere in the system.** `handleReEnroll` (`hub/internal/api/dr.go:101`) and `handleGetRestoreDirective` (`:155`) return `identity_escrow_b64` + `k_escrow_b64`, gated on operator-armed recovery mode. Census: **zero** callers in `felhom-agent` (no `ReEnroll` symbol at all; `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`), **zero** in the hub UI or any template, **zero** in `scripts/` or any runbook | **OPEN** | — | **The documented retrieval path is a human with `sqlite3`:** `SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …` on a `kubectl cp`-ed `hub.db` — recorded in project memory as the 2026-07-04 S5 prep, where hub-side blob serving was explicitly deemed *"Part 3 NOT needed"*. **This is the built-but-never-wired class at the DR capstone**, and it is why the chain from a dead node to an open repository has no automatable middle. **What a recovery flow actually needs is smaller than what exists:** a narrow `GET /hosts//escrow` authed with the box's own per-host key, serving opaque bytes to the box that owns them — zero-knowledge untouched, and far lighter than `re-enroll`, which **rotates the host API key** and returns the new key in the response body (`dr.go:130,148`). **Decide before building:** whether `re-enroll`/`restore-directive` should get a client, be replaced by the narrow GET, or be retired. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 6 | CC | +| **R-200** | **The DR password-injection seam has a handler, a route and tests — and no form.** `POST /backup/offbox/inject-password` is routed (`controller/internal/web/server.go:510`) to `offboxInjectPasswordHandler` (`offbox_handlers.go:174-196`) → `InjectOffboxPassword` (`backup/offbox.go:541`). **No template in the repository contains that path or any form posting to it** (grep over `internal/web/templates/`: one unrelated hit, an XSS comment) | **OPEN** | — | **The tenth instance of this project's built-but-never-wired class, and the exact shape `CLAUDE.md` and `felhom.eu/CLAUDE.md`'s seam-wiring rule were written for:** handler tests that POST directly (`offbox_escrow_test.go:167,180`) prove nothing about reachability. To use the only implemented recovery seam today, a person must hand-craft an authenticated POST with a session cookie and CSRF token. **Note the layering while fixing it:** this form takes a **64-hex repo password**, not a recovery code (`offboxRepoPwPattern`, `offbox.go:543`) — they are different secrets at different layers, and the operator's 2026-08-04 ruling asks for a form that takes **R**. Build the R form and treat this one as the operator/DR fallback it was written as, but **ship it with a render test per branch of whatever gate it sits behind**. Source: `audits/RECON-offsite-dr-chain-2026-08-04.md` §3 link 9 | CC | +| **R-201** | **Nothing in the offsite DR chain has ever been exercised past the ceremony — and the one live proof that exists predates the field it is cited for.** `slice10d-identity-restore-spike-findings.md` §1 (2026-06-10) proved a real wrap→unwrap of an identity bundle with a real R on a secret-less box, byte-identical, wrong-R failing closed. **That bundle was `{tunnel_token, pbs_token}`.** `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** — a month later — and is **unit-proven only** | **OPEN** | — | **Never exercised, in these words:** a blob has never been served to a box; a fork-4 bundle has never been unsealed with a real R outside a unit test (the only production caller is `--selftest=identity-consume`, `cmd/felhom-agent/main.go:2845`, reading R from `FELHOM_RECOVERY_CODE`); a recovered repo password has never been injected; an existing offsite repository has never been reopened with one; **no restore of any kind has ever been performed from a recovered secret.** The one live consume ever prepared (S5 Part 4-B, 2026-07-04) is still marked *"DEFERRED, not run"*. **`_recovery-inventory-2026-07-28.md` already recorded this accurately** (A.2.7, C.1 row 1, C-2) — this row exists so the gap has an owner and a closing condition, not only a description. **Closing condition = the drill** designed in `audits/RECON-offsite-dr-chain-2026-08-04.md` §10, whose pass condition is a **byte-identical sha256 of a sentinel file restored after a wipe** — explicitly NOT "the repository opened". **Do not run it before R-198 is fixed**: the drill would walk a chain that is missing a link everyone believed was there | CC + operator | +| **R-192** | **`offsite_delivery_stuck` tells the operator the opposite of what the detector measured, and the self-heal silently refuses for exactly the reason the message denies.** demo-hp has been e-mailing this daily since 2026-08-03 06:12 UTC: *"one-time password consumed 284h ago and **500 report(s) since carry no offbox target** — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one."* **Measured against the hub's own data: all 500 of those reports DO carry an offbox target** | **OPEN** | — | **What actually happened on that box:** the credential was consumed 2026-07-23 09:53:41 and **applied successfully** — the controller reported an `offsite` object continuously until **2026-08-03 05:59:21 UTC**, then it **vanished at 06:12:19** and has been absent for **108 consecutive reports** since. So this is a **regressed apply**, not a burned credential. **Two distinct defects, and the second explains the first's invisibility.** **(a)** `maybeEmitStuck` builds its message from `status.ReportsSinceConsume` (the TOTAL) while hardcoding the phrase *"carry no offbox target"*, and never consults `status.OffsiteReportsSinceConsume` — which is the field that says the opposite. The recommended action (*Re-issue*) is aimed at a failure mode that did not occur. This is R-100's corollary again: an alarm whose text stopped matching what its verdict counts. **(b)** `maybeHeal` refuses **silently** (`OffsiteReportsSinceConsume != 0` → *"regressed-apply shape → operator's call"*, a bare `return` with no log line), so the operator gets a daily e-mail with the wrong story, no heal, and nothing anywhere saying why the heal declined. `offsite_credential_restaged` has **never** fired, on any customer. **The underlying condition is REAL and is the part that matters:** demo-hp currently reports no offsite target at all, i.e. that box's customer app-data has **no off-site copy right now** — and it has been that way since 08:12 CEST on 2026-08-03. A spot check inside the controller container found no restic environment, consistent with the report. **What removed it is not established** and is the first thing to find out. **Fix shape:** the message must state which shape was detected (burned vs regressed) and say what to do for each; the heal's refusal must log its reason; and the regressed shape probably deserves its own event type rather than borrowing the burned one. **Do NOT 'fix' it by widening the heal to restage over a regression** — the guard is right, only mute **CAUSE ESTABLISHED 2026-08-04 (operator confirms no hub-side offsite config change).** The regression is a **guest REBUILD**: at 06:09:40 `host_leaf_changed` (agent re-keyed), at 06:12:18 `controller_started (0.192.0)` — the controller went **0.187.0 → 0.192.0** with a **new config hash** (`1f725a2e843c` → `744e83d72c80`) — and the report at 06:12:19 is the first without `offsite`. The pre-rebuild object was fully healthy: `escrow_state: escrowed, last_status: ok, last_success 2026-08-03T02:16:39Z, snapshot_count 15, repo_size 40.9 MB`. → **R-193** owns the rebuild half. **AND THE HEAL'S GUARD IS WRONG FOR EXACTLY THIS CASE, which is why the automation that exists to fix it declined.** `maybeHeal` refuses when `OffsiteReportsSinceConsume != 0`, reading that as *"the apply regressed, so it is the operator's call"*. But `CountReportsOffsiteSince` counts the **OLDEST 500 reports since the consume** (`ORDER BY id LIMIT 500`) — for demo-hp all 500 predate the rebuild. **Offbox evidence from before a rebuild is not evidence that the credential still works**, so the guard reads healthy history as a reason not to heal a box that demonstrably cannot apply. The fix is to judge on RECENT evidence (e.g. the latest N reports, or evidence after the newest `controller_started`), not on everything since the consume. **SPIKE 2026-08-04 — BOTH HALVES CONFIRMED WITH NUMBERS, still OPEN, still not fixed here** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md` Q7). The query is quoted at source (`store.go:987`): `... ORDER BY id LIMIT 500` = **the oldest 500**. Reproduced against the live hub DB with demo-hp's real consume anchor `2026-07-23 09:53:41`: the guard sees `total=500, withOffsite=500`, spanning `2026-07-23 09:53:47` → **`2026-07-28 11:17:40`** — the entire evidence set ends **six days before** the 2026-08-03 rebuild. The true window totals are `1174 / 1063` (⇒ 111 without, matching the 111 offsite-less reports). So the e-mail's *"500 report(s) since carry no offbox target"* interpolates `ReportsSinceConsume` while `OffsiteReportsSinceConsume` was **500** — the message states the precise negation of its own measurement. `offsite_credential_restaged` has **never fired for any customer** (zero rows of that type in the DB — checked, not assumed). **A NEW REASON NOT TO FIX THIS IN ISOLATION, from the same spike:** under R-193's Q2 finding a successful auto-restage would have restored demo-hp's TRANSPORT while the box minted a new repo password anyway — the heal can protect the plumbing and **cannot** protect the data, and had it fired on 2026-08-04 both boxes would have looked healthy while their snapshots were orphaned. **That is strictly worse than the current loud failure.** Whatever shape the fix takes must say so in its message. **RECON 2026-08-04 adds two inputs and changes no verdict** (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **(1) The guard's own conclusion — that a fix here protects the plumbing and not the data — is now stronger, not weaker:** even a perfect restage leaves the rebuilt box minting a fresh repo password, and per **R-198** the old one is destroyed by the re-ceremony the box is pushed into. Whatever shape the message takes must say that in the same breath, or it will read as an all-clear. **(2) The recency-bounded discriminator this row asks for has a ready anchor the hub already receives:** the report ACK's `escrow{identity_blob_present, restic_pw_sha256}` moves when a box re-keys, so "evidence since the newest re-key" is computable from data already stored — the same observation R-197 makes, aimed at this guard's time window. | CC | +| **R-193** | **A guest rebuild silently drops the off-site app-data tier, and nothing restages the credential.** demo-hp was rebuilt on 2026-08-03 (controller 0.187.0 → 0.192.0, new config hash, agent leaf re-keyed at 06:09:40). Before it, the offsite tier was healthy and working — `escrow_state: escrowed`, `last_status: ok`, last success **02:16:39Z that morning**, **15 snapshots, 40.9 MB**. After it: no `offsite` object in any of **108** reports, and **no off-site copy of that customer's app data since 08:12 CEST on 2026-08-03** | **OPEN** | — | **Mechanism, fully evidenced.** The restic credential reaches a box exactly once, as a one-time secret. demo-hp's was consumed **2026-07-23 09:53:41**; the rebuilt controller came up with a fresh data volume, no copy of it, and **no way to ask for another** — the hub is the only side that can stage one, and it will not re-stage a consumed secret on its own (the R-71c self-heal would, but it refuses — see R-192). **demo-felhom survived the SAME rebuild by luck, and the contrast is the proof:** its secret was created 2026-07-21 and still **UNCONSUMED**, so when its config hash changed at 07:17:54 and `offsite` dropped for exactly one report, it consumed the staged secret at **07:17:58** and was reporting `offsite` again by 07:19:10. One box had a spare credential staged and recovered in 76 seconds; the other did not and has been unprotected for a day. **That difference was not a design decision — it was an accident of which box happened to have an unconsumed secret lying around.** **Why this is not just "re-issue it":** the remedy (Re-issue) resets the sub-account password via the Hetzner API and, per R-39's record, **rotates the restic password and makes the escrow STALE** — so it needs the recovery-code ceremony re-run, and the continuity of the 15 existing snapshots under the new credential must be VERIFIED, not assumed (`hub v0.60.0` retains superseded escrow, and the orphan guard is move-aside-never-delete). That is an operator act with a customer-facing consequence, so it is not something to fire automatically without deciding the escrow question first. **What to design:** a rebuild is a normal, expected event on these boxes — the offsite tier must survive one, either by the hub restaging automatically when a re-enrolled box reports no offsite (the R-192 guard fix makes this safe), or by the credential being recoverable from escrow at re-bootstrap rather than delivered once and unrecoverable **RESOLVED ON THE BOX 2026-08-04 (operator-authorised).** Re-issue fired through the designed endpoint (`POST /configs/demo-hp/offsite-reissue`, HTTP 303): hub staged a fresh one-time password at **07:11:51**, the box's config hash moved `744e83d7` → `5eee0e42`, R-71a's settle-gate reported **GO** (*at/above floor 0.156.0, we are 0.194.0*), the password was **consumed 15 s later at 07:12:06**, and the controller logged *offsite configured for u629488-sub3@…:/home/felhom-repo* at 07:12:09 — the **same sub-account (275124) and the same repo path**, since Re-issue resets the sub-account password and the one-time password is only the transport credential used once to install the box's own SSH key. **THE ESCROW DID NOT RECOVER BY ITSELF — a correction to this session's own first reading.** `escrow_state` went `pending` → `escrowed` 15 s after the apply and CC inferred an automatic re-escrow; **the operator had run the ceremony**. It needed a human, on BOTH boxes: demo-hp escrowed 07:16:02, demo-felhom (whose offsite re-applied on its own the previous day but whose escrow had been `pending` ever since) escrowed 07:20:28. **A 15-second state change is not evidence of automation** — that is the same class as reading an absent log line as success. **Snapshot continuity is NOT yet established and must not be assumed from the counters:** both boxes report `snapshot_count: 0, repo_size_bytes: 0`, but the run-history keys (`last_run`, `last_status`, `last_success`) are **absent entirely** rather than zeroed — the shape of a controller that has never run an offbox backup in this lifetime, not of an empty repo. demo-hp's pre-rebuild object carried all three plus 15 snapshots / 40.9 MB. ~~**The next scheduled `offbox-backup` (04:15) decides it:** 15+ snapshots ⇒ the repo reattached; 1 ⇒ it started fresh and the old snapshots are orphaned-but-retained.~~ **SPIKE 2026-08-04 SETTLED IT WITHOUT WAITING, AND THE ITEM IS BIGGER THAN FILED** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`). **(1) THE ONE-SHOT SECRET IS NOT WHERE THE HARM IS.** Three secrets exist; the one-time password is the *recoverable* one (the operator can reset it at the provider any time) and the box's SFTP key is regenerable by design. The **restic repository password** — the DATA key, which the agent's own source calls *"irreplaceable (unlike the SFTP access key…)"* (`felhom-agent/internal/escrow/identity.go:35-39`) — is the one a rebuild destroys, and **nothing automatic ever restages it**: `WriteOffboxSecrets` mints a fresh 256-bit password whenever `/offbox/repo_password` is absent (`offbox.go:392`), and the only recovery path, `InjectOffboxPassword`, has **exactly one caller in the whole repo** — a web form a human pastes into (`web/offbox_handlers.go:189`). **(2) MEASURED, WITHOUT TOUCHING A BOX:** the hub already stores `restic_pw_sha256` on both the live and the superseded escrow, so the question is a hash comparison. demo-hp `8e03eddf…` → `8a9e33aa…`; demo-felhom `48741892…` → `c60c8bc7…`. **Both boxes minted a new repository password.** **(3) THE CONTRAST IN THIS ROW IS FALSE FOR THE DATA.** demo-felhom's 76-second "lucky" recovery restored **delivery only** — its pre-rebuild object carried **36 snapshots / 1.14 GB** (`repo_size_bytes 1136685919`) and it has reported `snapshot_count: 0` in all 109 reports since, with a changed repo password and **nothing marking its escrow stale for 13 h**. Both boxes lost repository continuity; one loudly, one silently, and **the silent one is worse**. **(4) Q3 IS UNMEASURED AND THE BINARY WAS WRONG.** Neither box could run on 2026-08-04 02:15 UTC (demo-hp had no target at all until 07:15; demo-felhom's target was `escrow_state: pending`, which `OffboxRunnable` blocks) — **the decisive run is 2026-08-05 ~02:15 UTC**. Source predicts a **third** outcome, neither 15 nor 1: same account + same repo path (`u629488-sub3:/home/felhom-repo`, and `repoPath` is a compile-time constant) + new password + `claimed: 1` ⇒ `ensureOffboxRepo` classifies **orphaned** and returns `ErrOffboxOrphaned` — **the run refuses and shows the orphan card**, it does not start a silent fresh history. **Record which of the three actually occurs; a prediction from source is not a measurement.** **(5) CANDIDATE (b) IS NOT IMPLEMENTABLE AS STATED** — the escrow is R-wrapped/zero-knowledge and the hub has no recovery code, so "recoverable from escrow at re-bootstrap" describes a customer-present ceremony, i.e. the manual form that already exists. **(6) CANDIDATE (a) ALREADY EXISTS AND IS WIRED TO THE WRONG EVENT:** `reissueOnReenroll`'s **F3 leg** does exactly this (`api/handler.go:1051-1084`) but sits behind `handleHostEnroll`'s mint-once-reuse short-circuit (`if existing != nil { return }`), and a **guest** rebuild leaves the `hosts` row intact — so F3 is never reached. **(7) A NEW CANDIDATE (c), not previously named and recommended second:** the **agent survives a guest rebuild**, already receives the repo password over the pinned local API (`POST /escrow/stage-secret`) and already writes it to a fixed 0600 path — it merely **wipes** it after the ceremony. Retaining it and serving it back is the only candidate that addresses the irreplaceable secret, and every seam it needs exists. Its cost is one deliberate trade the operator must make (a copy of the data key at rest on the Proxmox host — see D6). **SPIKE RECOMMENDATION: ship the honesty pass (R-196 + R-197) now; then decide (c). Do NOT ship (a) first — it would have hidden this.** **NOT CLOSED — WAITING-ON-OPERATOR for the (c)-vs-accept-it decision, stated at the end of the spike.** **RECON 2026-08-04 — the chain was traced link by link, and the picture is worse than the spike's** (`audits/RECON-offsite-dr-chain-2026-08-04.md`). **(A) THE SPIKE'S CANDIDATE (b) IS OVERTURNED IN PART.** "Recoverable from escrow" is not blocked by zero-knowledge — the *hub* cannot open the blob, the *customer* can, with R. What is genuinely impossible is an **unattended** recovery. A **customer-present** one is a real design, and the operator has now ruled on its shape (below). **(B) THE CHAIN IS NOT ASSEMBLED — eleven links, and the automation stops at four.** Mint → stage → seal → store on the hub are PROVEN-LIVE. Then: the hub's blob-serving endpoints have **no client anywhere** (**R-199**); unsealing's only production caller is a `--selftest` mode reading R from an env var; nothing extracts `restic_repo_password` from the recovered bundle; the injection seam **has no form** (**R-200**); and reopening an existing repo with a recovered password has **never happened** (**R-201**). **(C) THE LOAD-BEARING NEW FACT, and it retires this row's own "hub v0.60.0 retains superseded escrow" premise: the retention does NOT retain the restic repo password** — `host_escrow_superseded` has no `identity_blob` column (**R-198**). Both demo boxes' old passwords were destroyed by the 2026-08-04 re-ceremonies, so the orphaned snapshots are unrecoverable for a **second, independent** reason; keeping R would not have helped. **(D) A FAIL-CLOSED REFUSAL IS IMPLEMENTABLE — this is the most useful thing settled.** The hub already tells every box, on every report ACK, `escrow{identity_blob_present, restic_pw_sha256, created_at}` (`hub/internal/api/handler.go:504-510`) — and the controller **discards it** whenever no offbox target exists (`report/escrow_confirm.go:75-84`). Persisting it (the `ClaimSync` set-only pattern, `report/claim_sync.go:39-53`) and refusing to mint when a blob covers a password we do not have needs **no new hub API and no new secret**. **(E) OPERATOR RULING, 2026-08-04, recorded verbatim:** *"If a node is a fresh install AND the hub has a recovery blob, then the controller should yell that recovery is available, and provide a form for the customer to enter the recovery key. After unlocking the blob, the controller should show what will be recovered before proceeding."* Priced row by row in the recon §9: fresh-install signal **exists** (the mint branch's own `os.Stat`); hub-has-a-blob **exists on the wire, S to persist**; the yell **S**; an **R** form **does not exist** (the UI has only ever *emitted* R) **S**; unsealing must cross agent→controller because the controller image ships no `age` — **M**, one new agent local-API endpoint mirroring `/escrow/ceremony/claim`, plus a narrow hub `GET /hosts//escrow`; the **preview is cheap and read-only** — `restic snapshots --json` + `stats --mode raw-data --json` are already how the box counts snapshots (`offbox.go:1234-1265`), so count, dates, sizes, app tags and paths are all knowable before committing **S**. **Security question put to the operator, not answered:** the form sits behind the dashboard password (bcrypt + CSRF + 5/min lockout); the preview exposes backup cadence and app names; the form is an **oracle** for a stolen R and must fail as generically as `UnwrapIdentity` already does; and R transits the agent, which is the same trade as option (c) in a smaller, time-bounded form. **(F) THE DRILL IS DESIGNED AND NOT RUN** (recon §10): demo-hp, ~3–4 h, R kept deliberately, sentinel file sha256 before and after, **pass = byte-identical sha256, NOT "the repository opened"**, fail = a snapshot count of 1. **Run R-198's fix first.** **(G) Q3 STILL UNMEASURED:** neither box has run since (`last_run` absent on both, 2026-08-04 09:49/09:56 reports) — the decisive run remains **2026-08-05 ~02:15 UTC**. | CC + operator | | — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC | | — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator | | **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — |