admin
3f2b7bc023
gates / gates (push) Successful in 7s
Read-only recon of the escrow -> recovery chain, from a dead node to an open
repository. No production code, no build, no version bump.
Headline: the hub's superseded-escrow retention does NOT retain the offsite
repository password. host_escrow_superseded has no identity_blob column and
demoteCurrentEscrowTx copies only the K-escrow blob, so what survives a
supersession is the PBS datastore key, not the restic repo password. The next
escrow ceremony -- which the system tells a rebuilt box's customer to run --
destroys the last copy. Both demo boxes crossed that line on 2026-08-04.
Also established:
- the hub's blob-serving endpoints (re-enroll / restore-directive) have zero
callers anywhere: agent, hub UI, scripts, runbooks (R-199)
- POST /backup/offbox/inject-password is routed and handled but no template
contains the form (R-200)
- nothing in the recovery path has ever been exercised; the one live
round-trip proof (2026-06-10) predates the ResticRepoPassword field (R-201)
- a fail-closed mint refusal IS implementable: the report ACK already carries
escrow{identity_blob_present, restic_pw_sha256} and the controller discards
it whenever no offbox target exists
Corrections: yesterday's spike annotated (candidate (b) overturned in part --
unattended recovery is impossible, customer-present is not); capability-map
retention claim struck through and replaced with what the code does.
Deliverable: documentation/audits/RECON-offsite-dr-chain-2026-08-04.md
Register: new R-198..R-201; R-193 and R-192 updated; STATUS.md refreshed.
122 lines
7.9 KiB
Markdown
122 lines
7.9 KiB
Markdown
# REPORT — RECON: can a customer actually get their backups back? (2026-08-04)
|
||
|
||
**Class:** recon / spike. **No production code. No build, no deploy, no version bump.**
|
||
**Baselines re-read on arrival:** `felhom-agent` `856a127cd60b` (v0.124.1) · `felhom-controller`
|
||
`0887fd676d60` (v0.194.0) · `felhom.eu` `d26f49ad680e` (hub v0.92.0). All three trees clean.
|
||
|
||
> Supersedes the earlier 2026-08-04 report (R-193/R-192 spike + R-195), per the overwrite convention.
|
||
> Nothing durable from it lived only here — it is recorded in `hub/CHANGELOG.md`, `OPEN-ITEMS.md`
|
||
> R-192/R-193/R-195/R-196/R-197 and `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`.
|
||
|
||
**Deliverable:** `documentation/audits/RECON-offsite-dr-chain-2026-08-04.md` — Q1–Q8 with method,
|
||
measurement and ruling per question, the eleven-link chain table, the drill design, every query used,
|
||
and the operator's open decisions.
|
||
|
||
---
|
||
|
||
## 1. What was established
|
||
|
||
**The chain from a dead node to an open repository is not assembled, and the failure is deeper than
|
||
"some links are manual".**
|
||
|
||
1. **Links 1–4 are PROVEN-LIVE** — mint → stage to the agent → seal under R → store on the hub.
|
||
2. **Link 5 is lossy, and this is the finding of the session.** `host_escrow_superseded` has **no
|
||
`identity_blob` column**, and `demoteCurrentEscrowTx` (`hub/internal/store/store.go:2547-2556`)
|
||
copies only the K-escrow `blob`. The **restic repository password lives in `identity_blob`** — so
|
||
the retention feature built for the reinstall-orphan incident does not retain the secret that
|
||
incident is about. The **next escrow ceremony destroys the last copy**, and that ceremony is exactly
|
||
what the system tells a rebuilt box's customer to run. Both demo boxes crossed that line on
|
||
2026-08-04 at 07:15:36 and 07:20:08. → **R-198**
|
||
3. **Link 6 has no client anywhere.** `handleReEnroll` / `handleGetRestoreDirective` serve the blobs;
|
||
zero callers in the agent, the hub UI, any script or any runbook. Retrieval is `sqlite3` by hand on
|
||
the hub pod. → **R-199**
|
||
4. **Links 7–8 are manual-only.** `UnwrapIdentityBundle`'s only production caller is
|
||
`--selftest=identity-consume`, reading R from `FELHOM_RECOVERY_CODE`; nothing extracts
|
||
`restic_repo_password` from the recovered bundle.
|
||
5. **Link 9's form does not exist.** `POST /backup/offbox/inject-password` is routed, handled and
|
||
unit-tested; **no template contains it.** → **R-200**
|
||
6. **Link 11 has never happened.** No blob served, no fork-4 bundle unsealed with a real R outside a
|
||
unit test, no recovered password injected, no repo reopened, no restore. The one live round-trip
|
||
proof (2026-06-10) **predates the `ResticRepoPassword` field by a month**. → **R-201**
|
||
|
||
**And the most useful thing settled: a fail-closed refusal IS implementable.** The hub already sends
|
||
every box `escrow{identity_blob_present, restic_pw_sha256, created_at}` on every report ACK
|
||
(`api/handler.go:504-510`), and the controller **discards it** whenever no offbox target exists
|
||
(`report/escrow_confirm.go:75-84`). Persisting it needs no new hub API and no new secret; the
|
||
`ClaimSync` set-only pattern is the precedent.
|
||
|
||
**The operator's ruled recovery-screen shape is buildable** and is priced row by row in §9 of the
|
||
findings doc: fresh-install signal exists (the mint branch's own `os.Stat`); hub-has-a-blob exists on
|
||
the wire; the preview is cheap and genuinely read-only (`restic snapshots --json` +
|
||
`stats --mode raw-data --json` are already the box's own counters); the one real cost is crossing
|
||
agent→controller for the unseal, because the controller image ships no `age` — one new agent local-API
|
||
endpoint mirroring `/escrow/ceremony/claim`, plus a narrow hub `GET /hosts/<id>/escrow`.
|
||
|
||
## 2. Corrections owed and made
|
||
|
||
- **Yesterday's spike annotated, not rewritten** (`audits/SPIKE-offsite-credential-recovery-2026-08-04.md`):
|
||
a dated block at the top plus an inline note on Q8(b). Its candidate (b) verdict *"not implementable
|
||
against a zero-knowledge escrow"* is **too strong** — unattended recovery is impossible; a
|
||
customer-present recovery is not. Everything else in that spike stands, and its Q3 remains unmeasured.
|
||
- **Capability map corrected** (`architecture/00-capability-map.md`, escrow-ceremony row): the hub
|
||
v0.60.0 retention claim *"a previous passphrase stays recoverable with its recovery code"* is struck
|
||
through and replaced with what the code does — K-escrow only, no read path, `ListSupersededEscrow`
|
||
has zero production callers.
|
||
- **The same false promise is made to the customer in Hungarian** on the orphan card
|
||
(`controller/internal/web/templates/backups_remote.html:66,69`). Recorded in R-198; **not edited** —
|
||
this session ships no controller change.
|
||
- **The prompt's Q4 premise is a conflation, and it is corrected in the doc:** `escrow_confirm.go`
|
||
**does** fire on a mint (pending branch, hash mismatch, one WARN). The "wired to the wrong path"
|
||
finding was about the hub's `escrow_stale` event (R-196), a different mechanism. What is missing is
|
||
not the detection but the consequence — the warning reaches a log file and prescribes the act that
|
||
closes the recovery window.
|
||
|
||
## 3. Register
|
||
|
||
- **New: R-198, R-199, R-200, R-201** (highest previous ID was R-197; grepped across `OPEN-ITEMS.md`,
|
||
`ROADMAP.md`, `STATUS.md`, `CONTEXT.md` before minting).
|
||
- **Updated: R-193** (seven-part addendum: (b) overturned in part, the chain census, the retention
|
||
finding, fail-closed implementable, the operator ruling priced, the drill, Q3 still unmeasured) and
|
||
**R-192** (two inputs, no verdict change).
|
||
- **`STATUS.md`** updated in operator language: two new "what's broken" entries, three new
|
||
"waiting on you" items, a "changed since" line.
|
||
|
||
## 4. Live measurements taken (read-only)
|
||
|
||
Hub SQLite snapshot with the `-wal` (`kubectl exec … cat`), `PRAGMA integrity_check` → `ok`, freshness
|
||
proven positively (`MAX(host_reports.received_at)` = `2026-08-04 10:00:40` vs `datetime('now')` =
|
||
`10:03:48`). Escrow tables, `PRAGMA table_info` on both, latest-report `$.offsite.*` and `$.claimed`
|
||
per customer, the `claimed` 1→0→1 transition across both rebuilds, and the events table since
|
||
2026-08-03. Plus one `ls -la --time-style=full-iso` inside guest 9201 proving the data dir, the
|
||
`encryption.key` and `offbox/repo_password` are all newborn as of `2026-08-03 07:17:25`–`07:18:02`.
|
||
Every query with its field names and values is in §11 of the findings doc.
|
||
|
||
**Nothing was written, run, rotated, re-issued, deleted or reconfigured anywhere.** No ceremony, no
|
||
`selftest=identity-consume`, no touch on the storage endpoint.
|
||
|
||
## 5. What could not be established
|
||
|
||
The 2026-08-05 ~02:15 UTC run's outcome (still the decisive measurement; neither box has run —
|
||
`last_run` absent on both); whether demo-hp's data dir was fresh (measured only on demo-felhom;
|
||
demo-hp needs the break-glass credential this read-only session did not use); whether the orphaned
|
||
ciphertext still exists at the provider; whether `age` is present on either demo host. Named in §12
|
||
of the findings doc.
|
||
|
||
## 6. Observations recorded, not acted on
|
||
|
||
A fresh `encryption.key` on every rebuild makes every pre-rebuild `app.yaml` `ENC:` secret
|
||
undecryptable — circular dependency **C-1** of `_recovery-inventory-2026-07-28.md`, which fired live on
|
||
both demo boxes this week and is a strictly larger blast radius than the offsite repo. `Claimed`
|
||
returning to 1 with no event means "the customer claimed this box" has no audit trail.
|
||
`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting. `handleReEnroll` returns a
|
||
freshly minted API key in its response body on an endpoint nothing currently calls.
|
||
|
||
## 7. Conventions
|
||
|
||
- **No `CHANGELOG.md` entry.** `felhom.eu` has no root changelog — they are per-area (`hub/`,
|
||
`scripts/`, `website/`) and log code changes. This session changed none of those areas. Stated
|
||
rather than silently skipped.
|
||
- **`REUSE.md`** untouched — no shared helper or pattern moved.
|
||
- **Gates + CI run id** are recorded in the session summary.
|
||
</content>
|