# REPORT — RECON: can a customer actually get their backups back? (2026-08-04) **Class:** recon / spike. **No production code. No build, no deploy, no version bump.** **Baselines re-read on arrival:** `felhom-agent` `856a127cd60b` (v0.124.1) · `felhom-controller` `0887fd676d60` (v0.194.0) · `felhom.eu` `d26f49ad680e` (hub v0.92.0). All three trees clean. > Supersedes the earlier 2026-08-04 report (R-193/R-192 spike + R-195), per the overwrite convention. > Nothing durable from it lived only here — it is recorded in `hub/CHANGELOG.md`, `OPEN-ITEMS.md` > R-192/R-193/R-195/R-196/R-197 and `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`. **Deliverable:** `documentation/audits/RECON-offsite-dr-chain-2026-08-04.md` — Q1–Q8 with method, measurement and ruling per question, the eleven-link chain table, the drill design, every query used, and the operator's open decisions. --- ## 1. What was established **The chain from a dead node to an open repository is not assembled, and the failure is deeper than "some links are manual".** 1. **Links 1–4 are PROVEN-LIVE** — mint → stage to the agent → seal under R → store on the hub. 2. **Link 5 is lossy, and this is the finding of the session.** `host_escrow_superseded` has **no `identity_blob` column**, and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`) copies only the K-escrow `blob`. The **restic repository password lives in `identity_blob`** — so the retention feature built for the reinstall-orphan incident does not retain the secret that incident is about. The **next escrow ceremony destroys the last copy**, and that ceremony is exactly what the system tells a rebuilt box's customer to run. Both demo boxes crossed that line on 2026-08-04 at 07:15:36 and 07:20:08. → **R-198** 3. **Link 6 has no client anywhere.** `handleReEnroll` / `handleGetRestoreDirective` serve the blobs; zero callers in the agent, the hub UI, any script or any runbook. Retrieval is `sqlite3` by hand on the hub pod. → **R-199** 4. **Links 7–8 are manual-only.** `UnwrapIdentityBundle`'s only production caller is `--selftest=identity-consume`, reading R from `FELHOM_RECOVERY_CODE`; nothing extracts `restic_repo_password` from the recovered bundle. 5. **Link 9's form does not exist.** `POST /backup/offbox/inject-password` is routed, handled and unit-tested; **no template contains it.** → **R-200** 6. **Link 11 has never happened.** No blob served, no fork-4 bundle unsealed with a real R outside a unit test, no recovered password injected, no repo reopened, no restore. The one live round-trip proof (2026-06-10) **predates the `ResticRepoPassword` field by a month**. → **R-201** **And the most useful thing settled: a fail-closed refusal IS implementable.** The hub already sends every box `escrow{identity_blob_present, restic_pw_sha256, created_at}` on every report ACK (`api/handler.go:504-510`), and the controller **discards it** whenever no offbox target exists (`report/escrow_confirm.go:75-84`). Persisting it needs no new hub API and no new secret; the `ClaimSync` set-only pattern is the precedent. **The operator's ruled recovery-screen shape is buildable** and is priced row by row in §9 of the findings doc: fresh-install signal exists (the mint branch's own `os.Stat`); hub-has-a-blob exists on the wire; the preview is cheap and genuinely read-only (`restic snapshots --json` + `stats --mode raw-data --json` are already the box's own counters); the one real cost is crossing agent→controller for the unseal, because the controller image ships no `age` — one new agent local-API endpoint mirroring `/escrow/ceremony/claim`, plus a narrow hub `GET /hosts//escrow`. ## 2. Corrections owed and made - **Yesterday's spike annotated, not rewritten** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`): a dated block at the top plus an inline note on Q8(b). Its candidate (b) verdict *"not implementable against a zero-knowledge escrow"* is **too strong** — unattended recovery is impossible; a customer-present recovery is not. Everything else in that spike stands, and its Q3 remains unmeasured. - **Capability map corrected** (`architecture/00-capability-map.md`, escrow-ceremony row): the hub v0.60.0 retention claim *"a previous passphrase stays recoverable with its recovery code"* is struck through and replaced with what the code does — K-escrow only, no read path, `ListSupersededEscrow` has zero production callers. - **The same false promise is made to the customer in Hungarian** on the orphan card (`controller/internal/web/templates/backups_remote.html:66,69`). Recorded in R-198; **not edited** — this session ships no controller change. - **The prompt's Q4 premise is a conflation, and it is corrected in the doc:** `escrow_confirm.go` **does** fire on a mint (pending branch, hash mismatch, one WARN). The "wired to the wrong path" finding was about the hub's `escrow_stale` event (R-196), a different mechanism. What is missing is not the detection but the consequence — the warning reaches a log file and prescribes the act that closes the recovery window. ## 3. Register - **New: R-198, R-199, R-200, R-201** (highest previous ID was R-197; grepped across `OPEN-ITEMS.md`, `ROADMAP.md`, `STATUS.md`, `CONTEXT.md` before minting). - **Updated: R-193** (seven-part addendum: (b) overturned in part, the chain census, the retention finding, fail-closed implementable, the operator ruling priced, the drill, Q3 still unmeasured) and **R-192** (two inputs, no verdict change). - **`STATUS.md`** updated in operator language: two new "what's broken" entries, three new "waiting on you" items, a "changed since" line. ## 4. Live measurements taken (read-only) Hub SQLite snapshot with the `-wal` (`kubectl exec … cat`), `PRAGMA integrity_check` → `ok`, freshness proven positively (`MAX(host_reports.received_at)` = `2026-08-04 10:00:40` vs `datetime('now')` = `10:03:48`). Escrow tables, `PRAGMA table_info` on both, latest-report `$.offsite.*` and `$.claimed` per customer, the `claimed` 1→0→1 transition across both rebuilds, and the events table since 2026-08-03. Plus one `ls -la --time-style=full-iso` inside guest 9201 proving the data dir, the `encryption.key` and `offbox/repo_password` are all newborn as of `2026-08-03 07:17:25`–`07:18:02`. Every query with its field names and values is in §11 of the findings doc. **Nothing was written, run, rotated, re-issued, deleted or reconfigured anywhere.** No ceremony, no `selftest=identity-consume`, no touch on the storage endpoint. ## 5. What could not be established The 2026-08-05 ~02:15 UTC run's outcome (still the decisive measurement; neither box has run — `last_run` absent on both); whether demo-hp's data dir was fresh (measured only on demo-felhom; demo-hp needs the break-glass credential this read-only session did not use); whether the orphaned ciphertext still exists at the provider; whether `age` is present on either demo host. Named in §12 of the findings doc. ## 6. Observations recorded, not acted on A fresh `encryption.key` on every rebuild makes every pre-rebuild `app.yaml` `ENC:` secret undecryptable — circular dependency **C-1** of `_recovery-inventory-2026-07-28.md`, which fired live on both demo boxes this week and is a strictly larger blast radius than the offsite repo. `Claimed` returning to 1 with no event means "the customer claimed this box" has no audit trail. `offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting. `handleReEnroll` returns a freshly minted API key in its response body on an endpoint nothing currently calls. ## 7. Conventions - **No `CHANGELOG.md` entry.** `felhom.eu` has no root changelog — they are per-area (`hub/`, `scripts/`, `website/`) and log code changes. This session changed none of those areas. Stated rather than silently skipped. - **`REUSE.md`** untouched — no shared helper or pattern moved. - **Gates + CI run id** are recorded in the session summary.