felhom.eu/REPORT.md

REPORT — RECON: can a customer actually get their backups back? (2026-08-04)

Class: recon / spike. No production code. No build, no deploy, no version bump. Baselines re-read on arrival: felhom-agent 856a127cd60b (v0.124.1) · felhom-controller 0887fd676d60 (v0.194.0) · felhom.eu d26f49ad680e (hub v0.92.0). All three trees clean.

Supersedes the earlier 2026-08-04 report (R-193/R-192 spike + R-195), per the overwrite convention. Nothing durable from it lived only here — it is recorded in hub/CHANGELOG.md, OPEN-ITEMS.md R-192/R-193/R-195/R-196/R-197 and audits/SPIKE-offsite-credential-recovery-2026-08-04.md.

Deliverable: documentation/audits/RECON-offsite-dr-chain-2026-08-04.md — Q1–Q8 with method, measurement and ruling per question, the eleven-link chain table, the drill design, every query used, and the operator's open decisions.

---

1. What was established

The chain from a dead node to an open repository is not assembled, and the failure is deeper than "some links are manual".

1. Links 1–4 are PROVEN-LIVE — mint → stage to the agent → seal under R → store on the hub.
2. Link 5 is lossy, and this is the finding of the session. host_escrow_superseded has no identity_blob column, and demoteCurrentEscrowTx (hub/internal/store/store.go:2547-2556) copies only the K-escrow blob. The restic repository password lives in identity_blob — so the retention feature built for the reinstall-orphan incident does not retain the secret that incident is about. The next escrow ceremony destroys the last copy, and that ceremony is exactly what the system tells a rebuilt box's customer to run. Both demo boxes crossed that line on 2026-08-04 at 07:15:36 and 07:20:08. → R-198
3. Link 6 has no client anywhere. handleReEnroll / handleGetRestoreDirective serve the blobs; zero callers in the agent, the hub UI, any script or any runbook. Retrieval is sqlite3 by hand on the hub pod. → R-199
4. Links 7–8 are manual-only. UnwrapIdentityBundle's only production caller is --selftest=identity-consume, reading R from FELHOM_RECOVERY_CODE; nothing extracts restic_repo_password from the recovered bundle.
5. Link 9's form does not exist. POST /backup/offbox/inject-password is routed, handled and unit-tested; no template contains it. → R-200
6. Link 11 has never happened. No blob served, no fork-4 bundle unsealed with a real R outside a unit test, no recovered password injected, no repo reopened, no restore. The one live round-trip proof (2026-06-10) predates the ResticRepoPassword field by a month. → R-201

And the most useful thing settled: a fail-closed refusal IS implementable. The hub already sends every box escrow{identity_blob_present, restic_pw_sha256, created_at} on every report ACK (api/handler.go:504-510), and the controller discards it whenever no offbox target exists (report/escrow_confirm.go:75-84). Persisting it needs no new hub API and no new secret; the ClaimSync set-only pattern is the precedent.

The operator's ruled recovery-screen shape is buildable and is priced row by row in §9 of the findings doc: fresh-install signal exists (the mint branch's own os.Stat); hub-has-a-blob exists on the wire; the preview is cheap and genuinely read-only (restic snapshots --json + stats --mode raw-data --json are already the box's own counters); the one real cost is crossing agent→controller for the unseal, because the controller image ships no age — one new agent local-API endpoint mirroring /escrow/ceremony/claim, plus a narrow hub GET /hosts/<id>/escrow.

2. Corrections owed and made

• Yesterday's spike annotated, not rewritten (audits/SPIKE-offsite-credential-recovery-2026-08-04.md): a dated block at the top plus an inline note on Q8(b). Its candidate (b) verdict "not implementable against a zero-knowledge escrow" is too strong — unattended recovery is impossible; a customer-present recovery is not. Everything else in that spike stands, and its Q3 remains unmeasured.
• Capability map corrected (architecture/00-capability-map.md, escrow-ceremony row): the hub v0.60.0 retention claim "a previous passphrase stays recoverable with its recovery code" is struck through and replaced with what the code does — K-escrow only, no read path, ListSupersededEscrow has zero production callers.
• The same false promise is made to the customer in Hungarian on the orphan card (controller/internal/web/templates/backups_remote.html:66,69). Recorded in R-198; not edited — this session ships no controller change.
• The prompt's Q4 premise is a conflation, and it is corrected in the doc: escrow_confirm.go does fire on a mint (pending branch, hash mismatch, one WARN). The "wired to the wrong path" finding was about the hub's escrow_stale event (R-196), a different mechanism. What is missing is not the detection but the consequence — the warning reaches a log file and prescribes the act that closes the recovery window.

3. Register

• New: R-198, R-199, R-200, R-201 (highest previous ID was R-197; grepped across OPEN-ITEMS.md, ROADMAP.md, STATUS.md, CONTEXT.md before minting).
• Updated: R-193 (seven-part addendum: (b) overturned in part, the chain census, the retention finding, fail-closed implementable, the operator ruling priced, the drill, Q3 still unmeasured) and R-192 (two inputs, no verdict change).
• STATUS.md updated in operator language: two new "what's broken" entries, three new "waiting on you" items, a "changed since" line.

4. Live measurements taken (read-only)

Hub SQLite snapshot with the -wal (kubectl exec … cat), PRAGMA integrity_check → ok, freshness proven positively (MAX(host_reports.received_at) = 2026-08-04 10:00:40 vs datetime('now') = 10:03:48). Escrow tables, PRAGMA table_info on both, latest-report $.offsite.* and $.claimed per customer, the claimed 1→0→1 transition across both rebuilds, and the events table since 2026-08-03. Plus one ls -la --time-style=full-iso inside guest 9201 proving the data dir, the encryption.key and offbox/repo_password are all newborn as of 2026-08-03 07:17:25–07:18:02. Every query with its field names and values is in §11 of the findings doc.

Nothing was written, run, rotated, re-issued, deleted or reconfigured anywhere. No ceremony, no selftest=identity-consume, no touch on the storage endpoint.

5. What could not be established

The 2026-08-05 ~02:15 UTC run's outcome (still the decisive measurement; neither box has run — last_run absent on both); whether demo-hp's data dir was fresh (measured only on demo-felhom; demo-hp needs the break-glass credential this read-only session did not use); whether the orphaned ciphertext still exists at the provider; whether age is present on either demo host. Named in §12 of the findings doc.

6. Observations recorded, not acted on

A fresh encryption.key on every rebuild makes every pre-rebuild app.yaml ENC: secret undecryptable — circular dependency C-1 of _recovery-inventory-2026-07-28.md, which fired live on both demo boxes this week and is a strictly larger blast radius than the offsite repo. Claimed returning to 1 with no event means "the customer claimed this box" has no audit trail. offboxRecordStats persists RepoSizeBytes as a side effect of counting. handleReEnroll returns a freshly minted API key in its response body on an endpoint nothing currently calls.

7. Conventions

• No CHANGELOG.md entry. felhom.eu has no root changelog — they are per-area (hub/, scripts/, website/) and log code changes. This session changed none of those areas. Stated rather than silently skipped.
• REUSE.md untouched — no shared helper or pattern moved.
• Gates + CI run id are recorded in the session summary.