felhom.eu/documentation/audits/RECON-offsite-dr-chain-2026-08-04.md

# RECON — can a customer actually get their backups back?

**Date:** 2026-08-04 · **Items:** R-193, R-192 (+ new: R-198…R-201) · **Class:** recon — **no production code**
**Session scope:** read-only everywhere. Nothing built, deployed, wiped, rotated, re-issued or deleted.

> **Headline, before the detail.** The chain is **not** assembled, and the reason is worse than "two
> links are manual". **The hub's superseded-escrow retention — the feature built for exactly this
> incident — does not retain the offsite repository password.** `host_escrow_superseded` has no
> `identity_blob` column, and `demoteCurrentEscrowTx` copies only the K-escrow blob. So the last copy
> of a repo password is destroyed by the **next escrow ceremony** — the very act the system tells the
> customer to perform when their escrow goes `pending` after a rebuild. The recovery window closes
> because the remedy closes it.

---

## 1. Confirmed baselines (re-read on arrival)

| Repo | `main` @ commit | tree | Version |
|------|-----------------|------|---------|
| `felhom-agent` | `856a127cd60b0be797867c162f0ae49c7b64993e` | clean | v0.124.1 |
| `felhom-controller` | `0887fd676d608e7c103fa13b3e87536ad2481811` | clean | v0.194.0 |
| `felhom.eu` | `d26f49ad680e2be2ce2888fd58dff7dee886e4ad` | clean | hub v0.92.0 |

All three match the prompt's cited commits exactly. `git status --porcelain` empty in each.

**Highest register ID: R-197** (`OPEN-ITEMS.md` R-196/R-197 were minted by yesterday's spike;
R-194/R-195 by the R-190 arc). Grepped across `OPEN-ITEMS.md`, `ROADMAP.md`, `STATUS.md`,
`CONTEXT.md`. New items in this session therefore start at **R-198**.

---

## 2. Instruments — what each can and cannot answer

| # | Instrument | Exact read | Known limit |
|---|---|---|---|
| I1 | source | the three repos at the commits above | the authority; every claim below cites a file:line |
| I2 | hub SQLite snapshot | `kubectl -n felhom-system exec pod/hub-6775889d95-gc8sk -- cat /data/{hub.db,hub.db-wal,hub.db-shm}` → local `sqlite3` | the `-wal` is load-bearing (copied; `PRAGMA integrity_check` → `ok`) |
| I2-fresh | freshness proof | `MAX(host_reports.received_at)` = `2026-08-04 10:00:40`, `datetime('now')` = `2026-08-04 10:03:48` | **positive observable**: newest row 3 m 08 s old, not "the query did not error" |
| I3 | live filesystem, demo-felhom guest 9201 | `ssh felhom-pve "pct exec 9201 -- ls -la --time-style=full-iso …"` | read-only `ls`; mtimes only, no file contents read |
| I4 | caller census | `grep -rn <symbol> --include=*.go` across all four repos | proves absence of a caller in Go; templates checked separately |

**Not used, deliberately:** no ceremony, no re-issue, no rotation, no `selftest=identity-consume`, no
write of any kind, nothing touched on the storage endpoint.

---

## 3. Q1 — the chain, link by link

From a dead node to an open repository. **Status vocabulary as specified.**

| # | Link | Component / function | Who invokes it | Status |
|---|---|---|---|---|
| 1 | repo password is minted | controller `Manager.WriteOffboxSecrets` → mint at `internal/backup/offbox.go:392-401`; generator `:405` | `ApplyOffsiteTarget` (`offbox.go:474`) ← `ConfigureOffbox` ← the apply-bridge (`offsiteapply.go:221,256`) | **PROVEN-LIVE** — fires on every fresh data dir; measured twice this week (I2 hash change, I3 mtime) |
| 2 | password staged to the agent | controller `PushOffboxPasswordForEscrow` (`offbox.go:530`) → agent `POST /escrow/stage-secret` (`localapi/server.go:518`) → `escrow.StagedResticPasswordPath()` = `/var/lib/felhom-agent/escrow-stage/restic_repo_password`, 0600 (`identity.go:44-47`) | the controller enable/ceremony path | **PROVEN-LIVE** |
| 3 | sealed under R at the ceremony | `AttachResticPassword` (`identity.go:73`) → `WrapIdentityBundle` → `WrapIdentity` (`identity.go:114`, `age -p -a`, scrypt + ChaCha20-Poly1305) | agent `--selftest=escrow-create`, driven by the controller wizard (`/escrow/ceremony`, `localapi/server.go:525-527`) | **PROVEN-LIVE** — customer-facing wizard first live firing 2026-07-18 (`00-capability-map.md`, escrow row) |
| 4 | stored by the hub | `PUT /api/v1/hosts/<id>/escrow` (`api/handler.go:239`, `handleHostEscrowPut :1104`) → `host_escrow.identity_blob` | agent `main.go:2937` | **PROVEN-LIVE** — measured: `identity_blob` present, 572 B, for **both** hosts (I2) |
| 5 | **old blob retained on supersession** | `demoteCurrentEscrowTx` (`store/store.go:2547-2556`) → `host_escrow_superseded` | `SaveHostEscrow`, `DeleteHost` | **BROKEN FOR THIS PURPOSE.** The INSERT copies `host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256` — **not `identity_blob`**, and the table has **no such column** (`PRAGMA table_info`, I2). The K-escrow survives; **the restic repo password does not** |
| 6 | served back to a re-enrolling box | hub `handleReEnroll` (`api/dr.go:101`) and `handleGetRestoreDirective` (`:155`), returning `identity_escrow_b64`; recovery-mode gated, armed with the **global operator key** (`:34`) | **NOBODY.** I4: zero callers in `felhom-agent` (no `ReEnroll` symbol, no `/re-enroll` or `/restore-directive` string anywhere in the agent's hub client, `internal/hub/client.go` reaches only `desired-state`, `wg`, `pbs/consume-token`, `jobs`); zero in the hub UI (no template mentions them); zero in `scripts/` or any runbook | **implemented-never-exercised, and CLIENTLESS.** The only documented retrieval is by hand: `sqlite3 "SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …"` on a `kubectl cp`-ed `hub.db` (project memory, 2026-07-04 S5 prep) |
| 7 | unsealed with R | `UnwrapIdentityBundle` (`identity.go:172`) → `UnwrapIdentity` (`:139`) | **`runSelftestIdentityConsume` only** (`cmd/felhom-agent/main.go:2845-2900`), reading R from `FELHOM_RECOVERY_CODE`. I4: no other production caller | **manual-only — the only caller is a `--selftest` mode** |
| 8 | the repo password extracted | none. The selftest writes the **whole bundle JSON** to `-keydest` 0600 (`main.go:2872`) and prints *"identity recovered (tunnel_token + pbs_token)"* — it does not mention, extract or route `restic_repo_password` at all | a human reads the JSON | **missing** |
| 9 | placed on the rebuilt controller | `InjectOffboxPassword` (`offbox.go:541`) | one caller: `offboxInjectPasswordHandler` (`web/offbox_handlers.go:189`) at `POST /backup/offbox/inject-password` (`web/server.go:510`). **I4: no template in the repo contains that path or any form posting to it** | **manual-only, and UI-unreachable** — the handler exists, the form does not. A person must POST it by hand with a valid session cookie + CSRF token |
| 10 | the existing repo opens | `ensureOffboxRepo` (`offbox.go:653`): `restic cat config` → success clears `RepoState` | the scheduled/manual offbox run | **implemented-never-exercised** on a *recovered* password (exercised constantly on a live one) |
| 11 | a backup or restore proves it | — | — | **never** |

### The chain in one sentence

Links 1–4 are real and proven. **Link 5 silently drops the payload.** Links 6–9 are each individually
manual, and link 6 has no client at all. Link 11 has never happened.

---

## 4. Q2 — where the chain stops today

**The first link that is not automatic is #5, and it is not merely manual — it is lossy.**

Two distinct stopping points, and the order matters:

**(a) If the box has NOT re-escrowed since the rebuild** — the current `host_escrow.identity_blob`
still seals the *old* password. Recovery is possible in principle, and a person must:

1. Arm recovery mode on the hub (`PUT /api/v1/admin/hosts/<id>/recovery-mode`, global operator key) —
   or skip it entirely, because nothing consumes the endpoint it gates;
2. `kubectl cp` / `exec cat` the hub DB and `sqlite3 … writefile('/root/idblob', identity_blob)`;
3. copy the blob to a machine with the agent binary and `age`;
4. `FELHOM_RECOVERY_CODE=… felhom-agent --selftest=identity-consume -blob /root/idblob -keydest /root/bundle.json`;
5. open `bundle.json` by hand and read `restic_repo_password` out of it;
6. log in to the customer's dashboard and `POST /backup/offbox/inject-password` **by hand** (curl with
   session + CSRF) — there is no form;
7. re-run the apply so `WriteOffboxSecrets` finds the file present and keeps it;
8. trigger a run and hope `restic cat config` succeeds.

Eight manual steps, five of them requiring operator-tier access to the hub, one requiring the customer
to be present with R, and one (step 6) with no user interface at all.

**(b) If the box HAS re-escrowed** — which is what the system pushes the customer to do, because a
rebuilt box lands in `EscrowState: pending`, offsite runs are blocked (`OffboxRunnable`,
`offbox.go:570`), and the card says *„Helyreállítási kód szükséges"* — then the old `identity_blob`
has been **overwritten** by `SaveHostEscrow` and **not carried** into the superseded row. At that point
**no procedure exists**, with or without R, at any level of effort. The password is gone.

Both demo boxes are in state (b) as of 2026-08-04 07:15:36 (demo-hp) and 07:20:08 (demo-felhom) — I2,
`host_escrow.updated_at`. This is an **independent, and much stronger, reason** the 51 orphaned
snapshots are unrecoverable than "nobody kept the recovery codes". Keeping R would not have helped.

---

## 5. Q3 — can a rebuilt controller know it should recover?

### RULING: **YES — the hub already tells it, on every single report, and the controller throws the answer away.**

**What the box knows at the mint decision.** `WriteOffboxSecrets` is called from
`ApplyOffsiteTarget`, driven by the apply-bridge with the hub's offsite **descriptor**, whose fields
are `host, user, port, repo_path, host_fingerprint, quota_gb` (`offsiteapply.go:198-204`). **Nothing
about escrow is in scope at that point.** Locally, the only signal is the one the mint branch already
reads: `os.Stat(m.offboxPwPath())` returning `IsNotExist` — which *is* the fresh-data-dir signal, at
exactly the right line, but cannot distinguish "never configured" from "rebuilt".

**What the box is nonetheless already told.** Every report ACK carries an `escrow` object
(`api/handler.go:504-510`), consumed as `report.EscrowStatus`
(`internal/report/escrow_confirm.go:19-23`):

```go
type EscrowStatus struct {
	IdentityBlobPresent bool   `json:"identity_blob_present"`
	ResticPwSHA256      string `json:"restic_pw_sha256"`
	CreatedAt           string `json:"created_at"`
}
```

That is precisely *"the hub has a blob, and here is the hash of the password it covers"* — arriving
every report cycle, on a channel that already exists, with no new API and no new secret.

**And it is discarded.** `EscrowAutoConfirmer.Reconcile` (`escrow_confirm.go:75-84`) returns
immediately unless `Pending()` or `Escrowed()` — both of which require `OffboxConfigured()` **and** a
non-nil target (`cmd/controller/main.go:580-590`). On a freshly rebuilt box, before the offsite apply,
there is no target, so the status is dropped on the floor. It is never persisted anywhere: I4 finds
exactly **one** reference to the ACK field in the whole controller
(`cmd/controller/main.go:641: escrowConfirmer.Reconcile(resp.Escrow)`).

**Therefore a fail-closed refusal IS implementable**, with no new hub endpoint:

- persist the last-seen ACK escrow status alongside settings (the precedent exists — `ClaimSync`
  caches the hub's claim-code state into `settings.json` idempotently by generation,
  `internal/report/claim_sync.go:39-53`, and it does so *set-only, never cleared on hub silence*,
  which is the correct shape here too);
- at the mint branch, refuse when the password file is absent **and** the cached status says
  `identity_blob_present && restic_pw_sha256 != ""`.

**One ordering caveat, stated rather than assumed.** On a truly fresh box the apply can race the first
ACK. Measured on demo-hp: first post-rebuild report `2026-08-03 06:12:19`, offsite apply
`2026-08-04 07:12` — the ACK preceded the apply by 25 hours, but that gap was an incident, not a
design. A fail-closed rule handles the race correctly by construction: refuse to mint until the escrow
status is known, and retry next cycle (the apply-bridge is already built to retry and to persist
nothing on a partial failure, `offsiteapply.go:1-6`).

**Corroborating measurement that the data dir really is fresh on a rebuild** (I3, demo-felhom guest
9201, `/var/lib/docker/volumes/felhom-controller-data/_data/data/`):

| file | mtime |
|---|---|
| the data directory's parent | `2026-08-03 07:17:25` |
| `encryption.key` (32 B) | `2026-08-03 07:17:26` |
| `offbox/` | `2026-08-03 07:18:02` |
| `offbox/repo_password` (64 B) | `2026-08-03 07:18:02` |

The volume, the app-secret encryption key and the repo password are all newborn, 36 seconds apart.
Corroborated hub-side (I2): `json_extract(report_json,'$.claimed')` reads **1 → 0 → 1** across the
rebuild on both boxes — demo-hp `0` in 5 reports `2026-08-03 06:12:19`–`06:21:03`; demo-felhom `0` in 4
reports `07:17:54`–`07:19:46`. `Claimed` is set only by `SetClaimed()`, whose sole caller is the claim
submit handler (`web/claim.go:349`), so the return to `1` was a **human re-claim**, not automation.

> **Note, not acted on:** a fresh `encryption.key` means every pre-rebuild `app.yaml` `ENC:` secret is
> undecryptable — the C-1 circular dependency in `_recovery-inventory-2026-07-28.md` fired on both
> demo boxes this week. That is outside R-193's scope and is recorded as an observation (§13).

---

## 6. Q4 — what the stale-escrow detector actually covers

### RULING: **the prompt's premise conflates two different mechanisms. `escrow_confirm.go` DOES fire on a mint. But it fires into a log file, and the remedy it prescribes destroys the last copy of the old key.**

**The conflation, stated because it matters.** Yesterday's spike's sentence *"wired to the one path
that doesn't change the key and absent from the one that does"* is about the **hub's `escrow_stale`
event**, emitted by `ReissueCredentials` (`hub/internal/offsite/offsite.go:198-201`) — that is R-196,
and it stands. It is **not** about `controller/internal/report/escrow_confirm.go`. Two different
detectors; only one of them was ever claimed to be misplaced.

**What `escrow_confirm.go` does on a mint, from source.** After a rebuild the apply creates a target
with `EscrowState: "pending"` (the `cur == nil` fallthrough, `offbox.go:482-494`). So `Pending()` is
true, the pending branch runs (`escrow_confirm.go:86-107`), the hub's hash (sealing the OLD password)
is compared against `LocalHash()` (the NEW minted one), they differ, and it logs:

```
[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password
(hub hash %.12s… != local %.12s…) — run the escrow ceremony … staying pending
```

**So: confirmed, it fires.** The detector is not absent from the mint path.

**Three limits, each of which matters more than the fact that it fires.**

1. **It is a controller-local log line.** No event, no e-mail, no hub-side signal. `StaleBlob()` — the
   flag the web card renders — is set **only** by the escrowed branch (`reconcileEscrowed`,
   `:130-165`); the pending branch sets no display flag. The operator learns nothing.
2. **Its prescription is the destructive act.** *"run the escrow ceremony"* → a new ceremony →
   `SaveHostEscrow` overwrites `host_escrow.identity_blob` → `demoteCurrentEscrowTx` does not carry it
   → **the old repo password ceases to exist**. The detector correctly identifies the situation and
   then tells the customer to close the door.
3. **It cannot distinguish the two situations that need different answers** — "you re-keyed
   deliberately" and "your machine was rebuilt and your history is about to be orphaned" produce the
   identical warning.

So: **less needs building than the prompt feared for the *detection*, and more needs building for the
*consequence*.** The comparison exists; what does not exist is a path from that comparison to either a
signal or a recovery.

---

## 7. Q5 — is the superseded-escrow retention real, and what does it retain?

### RULING: **the retention is real, it retains the wrong blob, and nothing can read it back.**

**What it retains — source and live, agreeing.** `demoteCurrentEscrowTx` (`store/store.go:2547-2556`):

```sql
INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
FROM host_escrow WHERE host_id = ?
```

`identity_blob` is not in the column list. Live schema (I2, `PRAGMA table_info`):

| table | columns |
|---|---|
| `host_escrow` | `host_id, blob, key_fingerprint, posture, created_at, updated_at, **identity_blob**, directive_json, restic_pw_sha256, stale_at` |
| `host_escrow_superseded` | `id, host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at` |

**`blob` is the K-escrow — the PBS datastore encryption key**, wrapped by the PBS-native scrypt path.
**`identity_blob` is the age-wrapped `IdentityBundle`** — the one carrying `ResticRepoPassword`
(`felhom-agent/internal/escrow/identity.go:34-39`), plus `TunnelToken`, `PBSToken` and
`WGPrivateKey`. Live sizes confirm they are different objects: both hosts' current rows hold
`blob` = 383 B **and** `identity_blob` = 572 B; both superseded rows hold `blob` = 383 B and nothing
else.

**So, exactly what a customer with the matching OLD recovery code could recover from a retained blob:**

- ✅ the PBS datastore encryption key **K** — i.e. the Plane-2 whole-guest offsite ciphertext;
- ❌ **not** the restic repository password — the Tier-3 offsite app-data history;
- ❌ not the tunnel token, not the PBS access token, not the WireGuard private key.

**And there is no read path.** I4: `ListSupersededEscrow` (`store/store.go:2617`) has **zero
production callers** — only two tests. The only production surface is `CountSupersededEscrow`, used
for a number on the host page (`web/hosts.go:573`) and a log line (`api/handler.go:1151`). The schema
comment says so itself: *"The ACK and restore-serving read `host_escrow` (the CURRENT row) — never
this table."* Retrieval is `sqlite3` on the hub pod, by hand, by the operator.

**The claim this contradicts.** `00-capability-map.md` (escrow-ceremony row) records hub v0.60.0 as:

> *"a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS
> it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code
> (turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable')."*

The same claim appears in the schema comment (`store.go:370-375`) and, in Hungarian, on the customer's
own orphan card (`controller/internal/web/templates/backups_remote.html:66,69`):

> *„A meglévő mentések nem sérültek — a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók
> lehetnek."* · *„A régi előzmény félretéve marad (nem törlődik), és a hozzá tartozó helyreállítási
> kóddal később visszaállítható lehet."*

**For the offsite restic repository — the incident it names — that is false.** The retained blob does
not contain the key that opens it. This is the **ninth** entry in `CLAUDE.md`'s table of comments
asserting an invariant the code does not provide, and the first that is also **customer-facing copy**.
→ **R-198**, and the capability-map correction in §9.

---

## 8. Q6 — has any part of this ever been exercised?

**Method:** read every audit, validation doc, findings file and REPORT that mentions escrow consume,
identity-consume, re-enroll or inject-password; then check what the cited evidence actually covers.

### Proven

| what | evidence | scope of the proof |
|---|---|---|
| ceremony (create + seal + upload) | `00-capability-map.md` escrow row; `tests/VALIDATION-n100-rehearsal-2026-07-18.md` S6 — customer drove the wizard, R claimed one-shot 16:56:39, auto-confirmed 16:56:41 | the **create** side only |
| identity-bundle round-trip with a real R on a secret-less box | `tests/slice10d-identity-restore-spike-findings.md` §1, 2026-06-10 — wrap→unwrap byte-identical (`sha256` match); wrong-R **fails closed**, no plaintext emitted | **PRE-FORK-4.** The bundle then was `{tunnel_token, pbs_token}`. `ResticRepoPassword` was added in agent **v0.77.0 on 2026-07-09** (`felhom-agent/CHANGELOG.md:2344`) — a month later. **The proof does not cover the field this whole question is about** |
| `ResticRepoPassword` survives the wrap/unwrap | agent v0.77.0 CHANGELOG: *"`IdentityBundle` round-trip carries `ResticRepoPassword` byte-exact + not-in-blob + wrong-R fails closed"* | **unit test only** |
| `InjectOffboxPassword` places and refuses correctly | `controller/internal/backup/offbox_test.go:952-990`, incl. the companion showing a fresh mint without inject | **unit test only** |

### Never exercised — named in those words

- **A blob has never been served to a box.** Link 6 has no client (§3).
- **A fork-4 bundle has never been unsealed with a real R outside a unit test.** The only production
  caller is a `--selftest` mode; the one live consume ever prepared (S5 Part 4-B, 2026-07-04) was
  explicitly **deferred and never run** — its own resume note still reads *"DEFERRED 2026-07-04, not
  run"* (project memory; `06-offsite-connectivity.md:327` marks the destructive drill operator-gated).
- **A recovered repo password has never been injected into a controller.**
- **An existing offsite repository has never been reopened with a recovered password.**
- **No restore of any kind has ever been performed from a recovered secret.**

`_recovery-inventory-2026-07-28.md` already said most of this and said it accurately — A.2.7 records
*"the **consume** side is proven at spike level … the destructive in-place drill is explicitly
operator-gated and unrun"*, and C.1 row 1 says *"escrow **creation** proven live; **consume never
exercised destructively**"*. **That document is right and this session confirms it.** What it did not
catch — because it was written from `host_escrow`, not from the supersession path — is Q5.

**Capability-map correction owed:** not for a "recovery proven" claim (no row makes one), but for the
**retention** claim quoted in §7. See §9.

---

## 9. Q7 — the operator's ruled shape, assessed row by row

> **Operator ruling, 2026-08-04, verbatim:** *If a node is a fresh install AND the hub has a recovery
> blob, then the controller should yell that recovery is available, and provide a form for the customer
> to enter the recovery key. After unlocking the blob, the controller should show what will be
> recovered before proceeding.*

| The ruling requires | Exists? | What it needs, and what it costs |
|---|---|---|
| the controller knows it is a **fresh install** | **YES, at the right line** | `os.Stat(m.offboxPwPath())` → `IsNotExist` is already the mint branch (`offbox.go:392`). It is trustworthy as *"this controller has no repo password"*, which is the only fact the decision needs; it does **not** by itself distinguish first-ever install from rebuild — that is what the next row supplies. Corroborators exist but are weaker: `Claimed` is set-only and was measured re-set by a human within minutes (§5); a fresh `encryption.key` is a stronger signal but is not currently read by anything. **Cost: zero — the branch already exists.** |
| the controller knows **the hub has a blob** | **YES on the wire, NO in the code** | `EscrowStatus{IdentityBlobPresent, ResticPwSHA256, CreatedAt}` arrives on every report ACK and is discarded when no offbox target exists (§5). **Needs:** persist it (the `ClaimSync` set-only-by-generation pattern is the precedent, `claim_sync.go:39-53`). **Cost: small — controller only, one settings field, one reconciler, no hub change.** |
| it can **yell** rather than silently mint | **NO** | The refusal belongs **inside `WriteOffboxSecrets`, at the mint branch** — the one place that decides. Returning a new sentinel (`ErrOffboxRecoveryAvailable`, sibling of `ErrOffboxOrphaned`) propagates through `ApplyOffsiteTarget` → `ConfigureOffbox` → the apply-bridge, which already treats a failed configure as "persist nothing, retry next cycle" (`offsiteapply.go:5-6, 256`) — so a refusal is **fail-safe by construction**, not a new failure mode. Meanwhile the customer sees a card on `/backups/remote` (the orphan card at `backups_remote.html:60-70` is the exact precedent) and offsite runs stay blocked, which they already are (`OffboxRunnable`). **Cost: small-medium — controller only.** **Design hazard to name:** a first-ever install of a *re-used customer record* would also trip this; the refusal must be dismissible ("start fresh anyway"), and dismissal must be the act that mints. |
| a **form takes R** | **NO — and R is not a password** | The controller UI **emits** R (`backups_escrow.html:235-236`, `/backup/escrow/…/claim` returns it once) and has **never accepted** one. The only thing resembling an entry point is `POST /backup/offbox/inject-password`, which takes a **64-hex repo password** (`offboxRepoPwPattern`, `offbox.go:543`) — a different secret at a different layer — and **has no form in any template** (§3 link 9). **Cost: small for the form; the hard part is the next row.** |
| the blob is **unsealed** with R | **NO — and there is a real boundary** | Unsealing is `age -d`. `ageBinary = "/usr/bin/age"` (`identity.go:24`) and the code live in the **agent**. The **controller image does not ship `age`** (`controller/Dockerfile:56-70` — restic, openssh-client, sshpass, sqlite3, docker-cli; no age) and must not: the agent is the privileged tier by design. **The crossing already has a transport and a precedent:** the pinned local API carries `POST /escrow/ceremony` + `GET /escrow/ceremony/status` + `POST /escrow/ceremony/claim` (`localapi/server.go:524-527`) — i.e. the agent **already** handles R server-side for the create direction and hands the result back exactly once. **Needs: one new agent local-API endpoint** (`POST /escrow/unseal`: blob + R in, the single `restic_repo_password` field out, R never logged, R never persisted) — the exact mirror of the claim endpoint. **Plus** a way for the blob to reach the agent: the agent's hub client has no escrow **GET** (only the `PUT` at `main.go:2937`), and `dr.go`'s re-enroll is the wrong tool (it rotates the host API key and needs operator-armed recovery mode). **Needs: a second small hub endpoint** — `GET /hosts/<id>/escrow`, per-host-key authed, serving opaque bytes to the box that owns them. Zero-knowledge is untouched: the hub still never decrypts, and the bytes are useless without R. **Cost: medium — three repos, two new endpoints, both narrow.** |
| **"show what will be recovered" before proceeding** | **the primitive EXISTS; the flow does not** | `restic snapshots --json` + `restic stats --mode raw-data --json` are already how the box counts snapshots and sizes the repo (`offboxRecordStats`, `offbox.go:1234-1265`). Both are **read-only against the repository** — nothing is written, no lock is taken, no `init`. So a preview can show: **snapshot count, per-snapshot time, host, tags (= app names), paths, and the deduplicated repo size** — all before committing. **Committing requires only that the recovered password be written to `offboxPwPath()`**, and even that is reversible in principle. **The clean shape:** unseal → hold the recovered password **in memory** → run the two read-only probes against the existing repo → render → and only on the customer's confirmation call `InjectOffboxPassword`. **Cost: small — one handler, one template, no new restic capability.** **Caveat, measured not assumed:** `offboxRecordStats` today also persists `RepoSizeBytes` into settings as a side effect; a preview path must not reuse it verbatim. |
| **proceeding** actually recovers | **NO end-to-end** | `InjectOffboxPassword` → next `WriteOffboxSecrets` keeps the file → `ensureOffboxRepo` probes and clears `RepoState`. Every piece is implemented; **none has ever run against a recovered password** (§8). **Cost: zero new code, but it is precisely what the drill in §10 has to prove.** |

### Things the ruling implies that do not exist, priced

| missing piece | cost | note |
|---|---|---|
| persisted ACK escrow status + fail-closed mint refusal | **S** | controller only; the `ClaimSync` pattern applies directly |
| the "recovery is available" card + dismissal | **S** | one template, one flag; the orphan card is the template precedent |
| a customer-facing **recovery-code** form (distinct from the password form) | **S** | the password form's handler exists but has no UI; both need building |
| agent local-API `POST /escrow/unseal` | **M** | new privileged surface; mirror of `/escrow/ceremony/claim` |
| hub `GET /hosts/<id>/escrow` (opaque, per-host key) | **S** | new read surface on the hub; zero-knowledge preserved |
| preview handler + template driven by read-only restic probes | **S** | primitive already exists |
| **carrying `identity_blob` into `host_escrow_superseded`** | **S** | one column, one `SELECT` list — **and without it the whole feature protects only boxes that have not yet re-escrowed** (→ R-198). This is a prerequisite, not a nice-to-have |

### The security question the ruling raises — stated, not answered

**What gates the surface today.** The controller dashboard is behind `RequireAuth`
(`internal/web/auth.go:52`): a bcrypt password (`effectivePasswordHash()` — customer-set in
`settings.json`, else operator-provisioned in `controller.yaml`), a 7-day session cookie, 5 failed
logins per minute then lockout, and `CsrfProtect` on every mutating route. The claim gate serves only
the claim page on an unclaimed box. So a recovery form would sit **behind the customer's dashboard
password**, on a controller reachable at whatever address that box's dashboard is reachable at.

**What a preview would expose to someone who guesses or steals R.** Not file contents — but the
snapshot list is not nothing: **backup dates and cadence, the app names (restic tags), the paths, and
the repository size.** That is a map of the household's digital life, and it is precisely the metadata
the zero-knowledge posture otherwise keeps out of everyone's hands including the operator's.

**Three specifics the operator should weigh, without this session choosing:**

1. **Guessing R is not the threat; holding the dashboard password is.** R is ten EFF words
   (~129 bits, `00-capability-map.md` agent v0.93.0 note) and `age`'s scrypt is deliberately slow —
   online guessing is not a route. But **anyone with the dashboard password can already restore live
   offsite data** (`_recovery-inventory-2026-07-28.md` A.1.3–A.1.6). The marginal new exposure is
   confined to the *orphaned/previous* repository, which today nobody can read at all.
2. **The form is an oracle.** A correct/incorrect R answer distinguishable in the UI turns the
   dashboard into a verification service for a stolen recovery code. `UnwrapIdentity` already
   fails closed with a single generic error (`identity.go:157`) — the UI must not do better than that,
   and should rate-limit like the login path does.
3. **R will pass through the agent.** Whatever the form does, R crosses the local API to a
   root-adjacent daemon. The ceremony already does this in the other direction, so the precedent is
   set — but it means R is momentarily present in agent process memory on the Proxmox host, which is
   the same host D6 says the operator has root on. **This is the same trade as spike option (c), in a
   smaller and time-bounded form**, and it should be decided alongside it rather than separately.

---

## 10. Q8 — the drill design

**Why a deliberate cycle.** No recovery code exists for either demo box, and — per §7 — even one would
no longer open the old repositories. The proof therefore requires creating R, keeping it, and walking
the full circle. **This section is the design; the drill is not run in this session.**

### Target and preconditions

| | |
|---|---|
| **Box** | **`demo-hp`** (HP t740, node `felhom-host`, host `demo-hp-bb76ea`). Tier 0 — disposable, and the designated drill host by the 2026-07-25 operator ruling. **Not demo-felhom** — it is the N100 that also carries the ISO/rehearsal lineage; leave one demo box in a known-good state throughout |
| **Guest** | the customer guest on that host (rebuild target) |
| **Data** | one deliberately identifiable app + a sentinel file whose content is a known string and whose sha256 is recorded before the wipe |
| **Prereq** | the box is healthy, offsite target configured and `escrow_state: escrowed`, and its **current** `host_escrow.identity_blob` seals the password of the repo that will be tested |
| **Duration** | ~3–4 h wall clock, of which ~45 min is operator-attended (steps 2, 5, 8, 9) |
| **Blocking dependency** | **steps 8–9 cannot be executed as designed today** — links 6, 8 and 9 of the chain do not exist as automation. The drill as written uses the manual substitutes (§4a) and **that is the point**: it measures the *existing* chain honestly before anything is built |

### The cycle

| # | Step | Who | Observable that must be recorded |
|---|---|---|---|
| 1 | Record the starting state: `host_escrow.restic_pw_sha256`, `identity_blob` length, `$.offsite.snapshot_count`, `repo_size_bytes` | CC | a hash and two counters, from the hub DB |
| 2 | **Run the escrow ceremony through the customer wizard and KEEP R** | **HUMAN** | R written to the operator's password manager under a named entry; **its storage location recorded in the drill report — the code itself never** |
| 3 | Deploy an app, write the sentinel file, record its sha256 | CC | sha256 + path |
| 4 | Run an off-site backup through the real UI endpoint; wait for completion | CC | `$.offsite.last_status = ok`, `snapshot_count` increased by ≥1, `last_success` stamped |
| 5 | **Verify the snapshot from outside**: `restic snapshots --json` against the repo | CC | snapshot id + tag + timestamp — the pre-wipe truth to compare against |
| 6 | **Wipe**: destroy the guest and reinstall it (the same procedure that produced the 2026-08-03 rebuild) | **HUMAN** decision, CC executes | new `controller_started`; `claimed` drops to 0; a new `encryption.key` mtime |
| 7 | Let the box come back and reach the mint. **Do NOT run a new ceremony** | CC | the controller log carries the `escrow-confirm` mismatch WARN (§6); `escrow_state: pending`; `host_escrow.identity_blob` **unchanged** (this is the assertion that keeps recovery possible) |
| 8 | **Recover**: export `identity_blob` from the hub, unseal with the kept R, extract `restic_repo_password` | **HUMAN** (R is theirs) | `identity-consume` exits 0; the bundle JSON contains a non-empty `restic_repo_password`; **the value is never printed or logged** |
| 9 | **Inject** the recovered password (`POST /backup/offbox/inject-password`, by hand — there is no form), re-apply, run an off-site backup | **HUMAN**-assisted | `restic cat config` succeeds; **`snapshot_count` ≥ the step-5 count + 1**, not 1 |
| 10 | **Restore the sentinel** through the customer restore flow and compare sha256 | CC | **byte-identical sha256 to step 3** |
| 11 | Record everything, tear down nothing that would hide the evidence | CC | drill report |

### The single observation that makes this a FAILURE rather than a pass

> **Step 10's sha256 does not match step 3's — or step 9 reports a snapshot count of 1.**

A snapshot count of 1 means a **new, empty history** was started over the old one: the repository did
not reopen, and every earlier snapshot is orphaned. **Explicitly: "the repository opened" is not the
pass condition.** `restic cat config` succeeding proves the password is right; it does not prove the
customer's data came back. Only the sha256 comparison does.

### Secondary failures worth recording separately (each is a finding, not a pass/fail)

- Step 7 finds `identity_blob` **changed** — something re-escrowed automatically → the recovery window
  closes without a human act (would be new and serious).
- Step 8's bundle has an **empty** `restic_repo_password` — a pre-fork-4 or hash-less blob shape.
- Step 9's `ensureOffboxRepo` classifies `orphaned` despite the correct password → the classifier or
  the transport, not the key.

### What could make it fail for uninteresting reasons

The `age` binary absent on the machine running step 8; the hub DB export producing a truncated blob
(use `writefile`, verify the byte length against `length(identity_blob)` = 572); the CSRF token in
step 9 (`felhom_session` is dropped by curl's cookie jar — project memory
`controller-api-headless-access`); and the R-71a settle gate deferring the apply in step 9 (wait for
its GO line rather than retrying).

### Rollback, and whether the loss is acceptable

**If the recovery does not work, what has been lost is:** demo-hp's guest, its app data since the last
whole-guest archive, and its off-site history. **On that box that is acceptable** — it is Tier 0,
explicitly disposable, and it is the box the target-selection runbook names for drills. **The
rollback** is a whole-guest restore from the local/PBS archive taken immediately before step 6, which
is a proven path on this fleet (restore-test passed on demo-hp's host tier in 109.3 s on 2026-08-04).
**Take that archive deliberately as part of step 6 and verify it exists before destroying anything** —
do not rely on the nightly.

**Not acceptable, and therefore not proposed:** running this on demo-felhom (leaves no healthy demo
box), on DooPlex (Tier 2, it *is* the recovery chain), or on any real customer's box.

---

## 11. Every query used

| # | Source | Query / read | Value read |
|---|---|---|---|
| 1 | git | `git -C <repo> rev-parse HEAD`; `git status --porcelain` | the three commits in §1; all empty |
| 2 | hub DB | `PRAGMA integrity_check` | `ok` |
| 3 | hub DB | `SELECT MAX(received_at) FROM host_reports; SELECT datetime('now')` | `2026-08-04 10:00:40` / `2026-08-04 10:03:48` |
| 4 | hub DB | `SELECT host_id, length(blob), length(identity_blob), substr(restic_pw_sha256,1,12), created_at, updated_at FROM host_escrow` | `demo-felhom-8363b5, 383, 572, c60c8bc737a6, 2026-08-04T07:20:07Z, 2026-08-04 07:20:08` · `demo-hp-bb76ea, 383, 572, 8a9e33aa4da6, 2026-08-04T07:15:36Z, 2026-08-04 07:15:36` |
| 5 | hub DB | `SELECT id, host_id, length(blob), substr(restic_pw_sha256,1,12), created_at, superseded_at FROM host_escrow_superseded` | `3, demo-hp-bb76ea, 383, 8e03eddf9ff7, 2026-07-23T10:01:17Z, 2026-08-04 07:15:36` · `4, demo-felhom-8363b5, 383, 48741892f0ef, 2026-07-21T08:38:33Z, 2026-08-04 07:20:08` |
| 6 | hub DB | `PRAGMA table_info(host_escrow)` / `(host_escrow_superseded)` | the two column lists in §7 — `identity_blob` present in the first, **absent from the second** |
| 7 | hub DB | latest report per customer: `json_extract(report_json,'$.claimed' / '$.offsite.escrow_state' / '.snapshot_count' / '.repo_state' / '.last_status' / '.last_run')` | demo-felhom `2026-08-04 09:49:42`: `1, escrowed, 0, NULL, NULL, NULL` · demo-hp `2026-08-04 09:56:57`: `1, escrowed, 0, NULL, NULL, NULL` |
| 8 | hub DB | `claimed` grouped over reports since `2026-08-03 05:00` | demo-hp: `1` ×124, **`0` ×5 (06:12:19→06:21:03)** · demo-felhom: `1` ×120, **`0` ×4 (07:17:54→07:19:46)** |
| 9 | hub DB | `SELECT customer_id, event_type, severity, created_at FROM events WHERE created_at > '2026-08-03 00:00'` | no `escrow_stale`, no `offsite_reissued`, no `offbox_repo_orphaned`, no claim event in the window; `controller_started` ×4 demo-hp / ×4 demo-felhom around the rebuilds |
| 10 | guest 9201 (I3) | `ls -la --time-style=full-iso …/felhom-controller-data/_data/data{,/offbox}` | the mtime table in §5 |
| 11 | source (I4) | `grep -rn "InjectOffboxPassword" --include=*.go .` | 1 definition, 1 handler caller, 5 test uses, 1 comment — **no production caller besides the handler** |
| 12 | source (I4) | `grep -rn "inject" internal/web/templates/` | one unrelated hit (an XSS comment) — **no form** |
| 13 | source (I4) | `grep -rni "reenroll\|re_enroll" --include=*.go .` in `felhom-agent` | **zero hits** |
| 14 | source (I4) | `grep -rn "re-enroll\|recovery-mode\|restore-directive" --include=*.sh --include=*.md --include=*.py` workspace-wide | only design docs + project memory; no script, no runbook step |
| 15 | source (I4) | `grep -rn "ListSupersededEscrow"` in `felhom.eu` | 2 test callers, 0 production |
| 16 | source (I4) | `grep -rn "\.Escrow\b" --include=*.go .` in `felhom-controller` | one hit: `cmd/controller/main.go:641` |
| 17 | source | `felhom-agent/CHANGELOG.md` | `v0.77.0 — fork-4: escrow the offsite restic repo password under R (2026-07-09)` |
| 18 | source | `controller/Dockerfile:36-70` | runtime package list — **no `age`** |

---

## 12. What could not be established

Named, not guessed.

1. **The 2026-08-05 ~02:15 UTC scheduled run's outcome.** Still unmeasured — both boxes became
   runnable only on 2026-08-04 morning and neither has run since (query 7: `last_run` absent on both).
   Yesterday's spike predicts `ErrOffboxOrphaned`; that prediction is **unchanged and still unproven**.
2. **Whether demo-hp's data dir was fresh.** Established for demo-felhom by direct mtime measurement
   (I3). demo-hp was not measured — it has no baked SSH key and reaching it needs the break-glass
   credential, which this read-only session did not use. The hub-side `claimed 1→0→1` signature is
   identical on both boxes, which is suggestive, not decisive.
3. **Whether the orphaned ciphertext still exists at the provider.** Unchanged from yesterday: reading
   the endpoint needs the SFTP credential, which was not touched.
4. **Whether any pre-2026-07-21 escrow generation ever existed for demo-felhom.** The superseded table
   holds two rows in the entire database; earlier generations are not retained (and, per §7, would not
   have carried the identity blob anyway).
5. **Whether `age` is present on any machine that would run step 8 of the drill.** Not checked — the
   agent declares `/usr/bin/age` as a runtime dependency but this session did not verify it on either
   demo host.

---

## 13. Observations — noticed, not acted on

1. **A fresh `encryption.key` on every rebuild** (I3: 32 B, mtime `2026-08-03 07:17:26`) means every
   pre-rebuild `app.yaml` `ENC:` secret on that box is undecryptable. This is exactly circular
   dependency **C-1** in `_recovery-inventory-2026-07-28.md`, and it fired live on both demo boxes this
   week without anyone naming it. It is a strictly larger blast radius than the offsite repo and it is
   **not** covered by R-193. Worth its own item if the operator agrees it is not already implied by C-1.
2. **`Claimed` returning to `1` within minutes** of a rebuild, with no event of any kind recording it,
   means "the customer claimed this box" has no audit trail. There is no `customer_claimed` event type.
3. **`offboxRecordStats` persists `RepoSizeBytes` as a side effect of counting** — harmless today,
   a trap for anyone reusing it as a read-only preview primitive (§9).
4. **`handleReEnroll` returns the new API key in the response body** (`dr.go:148`) while also logging
   the rotation. Not a defect (the box needs it), but it is a secret in an HTTP response on an endpoint
   nothing currently calls — worth a second look before anything starts calling it.

---

## 14. The operator's decisions — stated, and NOT answered here

1. **Run the drill (§10) or not?** It is the only thing that converts *"the pieces look right"* into
   *"a customer got their data back"*. It costs demo-hp for an afternoon and, if it fails, that box's
   history. **Recommendation: yes — but fix `identity_blob` retention (R-198) first**, because a drill
   that walks the current chain will pass or fail on a chain that is missing a link everyone believed
   was there.
2. **The orphaned snapshots at the endpoint** (40.9 MB + 1.14 GB, presumed present, §12.3). They are
   **unrecoverable by two independent mechanisms** now: no recovery code was kept, *and* the sealed
   passwords were destroyed by the 2026-08-04 re-ceremonies. They consume the 50 GB soft quota
   indefinitely; nothing prunes a moved-aside restic repo. **Delete, or leave?** This session did not
   act on it and states it as the operator's.
3. **The unchanged R-193 question** — agent-retained repo password (spike option c) versus accepting
   that every rebuild starts a new repository. **This session adds one input to that decision:** the
   "accept it, provided the system says so loudly" branch currently rests on a retention feature that
   does not retain the relevant secret. Under option (d)-alone as it stands today, *"the old history
   stays recoverable with your recovery code"* is not true. Either R-198 is fixed, or that branch has
   to be re-described honestly to the customer.
</content>
</invoke>