felhom.eu/documentation/audits/RECON-offsite-dr-chain-2026-08-04.md

RECON — can a customer actually get their backups back?

Date: 2026-08-04 · Items: R-193, R-192 (+ new: R-198…R-201) · Class: recon — no production code Session scope: read-only everywhere. Nothing built, deployed, wiped, rotated, re-issued or deleted.

Headline, before the detail. The chain is not assembled, and the reason is worse than "two links are manual". The hub's superseded-escrow retention — the feature built for exactly this incident — does not retain the offsite repository password. host_escrow_superseded has no identity_blob column, and demoteCurrentEscrowTx copies only the K-escrow blob. So the last copy of a repo password is destroyed by the next escrow ceremony — the very act the system tells the customer to perform when their escrow goes pending after a rebuild. The recovery window closes because the remedy closes it.

---

1. Confirmed baselines (re-read on arrival)

Repo	main @ commit	tree	Version
felhom-agent	856a127cd60b0be797867c162f0ae49c7b64993e	clean	v0.124.1
felhom-controller	0887fd676d608e7c103fa13b3e87536ad2481811	clean	v0.194.0
felhom.eu	d26f49ad680e2be2ce2888fd58dff7dee886e4ad	clean	hub v0.92.0

All three match the prompt's cited commits exactly. git status --porcelain empty in each.

Highest register ID: R-197 (OPEN-ITEMS.md R-196/R-197 were minted by yesterday's spike; R-194/R-195 by the R-190 arc). Grepped across OPEN-ITEMS.md, ROADMAP.md, STATUS.md, CONTEXT.md. New items in this session therefore start at R-198.

---

2. Instruments — what each can and cannot answer

#	Instrument	Exact read	Known limit
I1	source	the three repos at the commits above	the authority; every claim below cites a file:line
I2	hub SQLite snapshot	kubectl -n felhom-system exec pod/hub-6775889d95-gc8sk -- cat /data/{hub.db,hub.db-wal,hub.db-shm} → local sqlite3	the -wal is load-bearing (copied; PRAGMA integrity_check → ok)
I2-fresh	freshness proof	MAX(host_reports.received_at) = 2026-08-04 10:00:40, datetime('now') = 2026-08-04 10:03:48	positive observable: newest row 3 m 08 s old, not "the query did not error"
I3	live filesystem, demo-felhom guest 9201	ssh felhom-pve "pct exec 9201 -- ls -la --time-style=full-iso …"	read-only ls; mtimes only, no file contents read
I4	caller census	grep -rn <symbol> --include=*.go across all four repos	proves absence of a caller in Go; templates checked separately

Not used, deliberately: no ceremony, no re-issue, no rotation, no selftest=identity-consume, no write of any kind, nothing touched on the storage endpoint.

---

3. Q1 — the chain, link by link

From a dead node to an open repository. Status vocabulary as specified.

#	Link	Component / function	Who invokes it	Status
1	repo password is minted	controller Manager.WriteOffboxSecrets → mint at internal/backup/offbox.go:392-401; generator :405	ApplyOffsiteTarget (offbox.go:474) ← ConfigureOffbox ← the apply-bridge (offsiteapply.go:221,256)	PROVEN-LIVE — fires on every fresh data dir; measured twice this week (I2 hash change, I3 mtime)
2	password staged to the agent	controller PushOffboxPasswordForEscrow (offbox.go:530) → agent POST /escrow/stage-secret (localapi/server.go:518) → escrow.StagedResticPasswordPath() = /var/lib/felhom-agent/escrow-stage/restic_repo_password, 0600 (identity.go:44-47)	the controller enable/ceremony path	PROVEN-LIVE
3	sealed under R at the ceremony	AttachResticPassword (identity.go:73) → WrapIdentityBundle → WrapIdentity (identity.go:114, age -p -a, scrypt + ChaCha20-Poly1305)	agent --selftest=escrow-create, driven by the controller wizard (/escrow/ceremony, localapi/server.go:525-527)	PROVEN-LIVE — customer-facing wizard first live firing 2026-07-18 (00-capability-map.md, escrow row)
4	stored by the hub	PUT /api/v1/hosts/<id>/escrow (api/handler.go:239, handleHostEscrowPut :1104) → host_escrow.identity_blob	agent main.go:2937	PROVEN-LIVE — measured: identity_blob present, 572 B, for both hosts (I2)
5	old blob retained on supersession	demoteCurrentEscrowTx (store/store.go:2547-2556) → host_escrow_superseded	SaveHostEscrow, DeleteHost	BROKEN FOR THIS PURPOSE. The INSERT copies host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256 — not identity_blob, and the table has no such column (PRAGMA table_info, I2). The K-escrow survives; the restic repo password does not
6	served back to a re-enrolling box	hub handleReEnroll (api/dr.go:101) and handleGetRestoreDirective (:155), returning identity_escrow_b64; recovery-mode gated, armed with the global operator key (:34)	NOBODY. I4: zero callers in felhom-agent (no ReEnroll symbol, no /re-enroll or /restore-directive string anywhere in the agent's hub client, internal/hub/client.go reaches only desired-state, wg, pbs/consume-token, jobs); zero in the hub UI (no template mentions them); zero in scripts/ or any runbook	implemented-never-exercised, and CLIENTLESS. The only documented retrieval is by hand: sqlite3 "SELECT writefile('/root/idblob', identity_blob) FROM host_escrow …" on a kubectl cp-ed hub.db (project memory, 2026-07-04 S5 prep)
7	unsealed with R	UnwrapIdentityBundle (identity.go:172) → UnwrapIdentity (:139)	runSelftestIdentityConsume only (cmd/felhom-agent/main.go:2845-2900), reading R from FELHOM_RECOVERY_CODE. I4: no other production caller	manual-only — the only caller is a --selftest mode
8	the repo password extracted	none. The selftest writes the whole bundle JSON to -keydest 0600 (main.go:2872) and prints "identity recovered (tunnel_token + pbs_token)" — it does not mention, extract or route restic_repo_password at all	a human reads the JSON	missing
9	placed on the rebuilt controller	InjectOffboxPassword (offbox.go:541)	one caller: offboxInjectPasswordHandler (web/offbox_handlers.go:189) at POST /backup/offbox/inject-password (web/server.go:510). I4: no template in the repo contains that path or any form posting to it	manual-only, and UI-unreachable — the handler exists, the form does not. A person must POST it by hand with a valid session cookie + CSRF token
10	the existing repo opens	ensureOffboxRepo (offbox.go:653): restic cat config → success clears RepoState	the scheduled/manual offbox run	implemented-never-exercised on a recovered password (exercised constantly on a live one)
11	a backup or restore proves it	—	—	never

The chain in one sentence

Links 1–4 are real and proven. Link 5 silently drops the payload. Links 6–9 are each individually manual, and link 6 has no client at all. Link 11 has never happened.

---

4. Q2 — where the chain stops today

The first link that is not automatic is #5, and it is not merely manual — it is lossy.

Two distinct stopping points, and the order matters:

(a) If the box has NOT re-escrowed since the rebuild — the current host_escrow.identity_blob still seals the old password. Recovery is possible in principle, and a person must:

1. Arm recovery mode on the hub (PUT /api/v1/admin/hosts/<id>/recovery-mode, global operator key) — or skip it entirely, because nothing consumes the endpoint it gates;
2. kubectl cp / exec cat the hub DB and sqlite3 … writefile('/root/idblob', identity_blob);
3. copy the blob to a machine with the agent binary and age;
4. FELHOM_RECOVERY_CODE=… felhom-agent --selftest=identity-consume -blob /root/idblob -keydest /root/bundle.json;
5. open bundle.json by hand and read restic_repo_password out of it;
6. log in to the customer's dashboard and POST /backup/offbox/inject-password by hand (curl with session + CSRF) — there is no form;
7. re-run the apply so WriteOffboxSecrets finds the file present and keeps it;
8. trigger a run and hope restic cat config succeeds.

Eight manual steps, five of them requiring operator-tier access to the hub, one requiring the customer to be present with R, and one (step 6) with no user interface at all.

(b) If the box HAS re-escrowed — which is what the system pushes the customer to do, because a rebuilt box lands in EscrowState: pending, offsite runs are blocked (OffboxRunnable, offbox.go:570), and the card says „Helyreállítási kód szükséges" — then the old identity_blob has been overwritten by SaveHostEscrow and not carried into the superseded row. At that point no procedure exists, with or without R, at any level of effort. The password is gone.

Both demo boxes are in state (b) as of 2026-08-04 07:15:36 (demo-hp) and 07:20:08 (demo-felhom) — I2, host_escrow.updated_at. This is an independent, and much stronger, reason the 51 orphaned snapshots are unrecoverable than "nobody kept the recovery codes". Keeping R would not have helped.

---

5. Q3 — can a rebuilt controller know it should recover?

RULING: YES — the hub already tells it, on every single report, and the controller throws the answer away.

What the box knows at the mint decision. WriteOffboxSecrets is called from ApplyOffsiteTarget, driven by the apply-bridge with the hub's offsite descriptor, whose fields are host, user, port, repo_path, host_fingerprint, quota_gb (offsiteapply.go:198-204). Nothing about escrow is in scope at that point. Locally, the only signal is the one the mint branch already reads: os.Stat(m.offboxPwPath()) returning IsNotExist — which is the fresh-data-dir signal, at exactly the right line, but cannot distinguish "never configured" from "rebuilt".

What the box is nonetheless already told. Every report ACK carries an escrow object (api/handler.go:504-510), consumed as report.EscrowStatus (internal/report/escrow_confirm.go:19-23):

type EscrowStatus struct {
	IdentityBlobPresent bool   `json:"identity_blob_present"`
	ResticPwSHA256      string `json:"restic_pw_sha256"`
	CreatedAt           string `json:"created_at"`
}

That is precisely "the hub has a blob, and here is the hash of the password it covers" — arriving every report cycle, on a channel that already exists, with no new API and no new secret.

And it is discarded. EscrowAutoConfirmer.Reconcile (escrow_confirm.go:75-84) returns immediately unless Pending() or Escrowed() — both of which require OffboxConfigured() and a non-nil target (cmd/controller/main.go:580-590). On a freshly rebuilt box, before the offsite apply, there is no target, so the status is dropped on the floor. It is never persisted anywhere: I4 finds exactly one reference to the ACK field in the whole controller (cmd/controller/main.go:641: escrowConfirmer.Reconcile(resp.Escrow)).

Therefore a fail-closed refusal IS implementable, with no new hub endpoint:

• persist the last-seen ACK escrow status alongside settings (the precedent exists — ClaimSync caches the hub's claim-code state into settings.json idempotently by generation, internal/report/claim_sync.go:39-53, and it does so set-only, never cleared on hub silence, which is the correct shape here too);
• at the mint branch, refuse when the password file is absent and the cached status says identity_blob_present && restic_pw_sha256 != "".

One ordering caveat, stated rather than assumed. On a truly fresh box the apply can race the first ACK. Measured on demo-hp: first post-rebuild report 2026-08-03 06:12:19, offsite apply 2026-08-04 07:12 — the ACK preceded the apply by 25 hours, but that gap was an incident, not a design. A fail-closed rule handles the race correctly by construction: refuse to mint until the escrow status is known, and retry next cycle (the apply-bridge is already built to retry and to persist nothing on a partial failure, offsiteapply.go:1-6).

Corroborating measurement that the data dir really is fresh on a rebuild (I3, demo-felhom guest 9201, /var/lib/docker/volumes/felhom-controller-data/_data/data/):

file	mtime
the data directory's parent	2026-08-03 07:17:25
encryption.key (32 B)	2026-08-03 07:17:26
offbox/	2026-08-03 07:18:02
offbox/repo_password (64 B)	2026-08-03 07:18:02

The volume, the app-secret encryption key and the repo password are all newborn, 36 seconds apart. Corroborated hub-side (I2): json_extract(report_json,'$.claimed') reads 1 → 0 → 1 across the rebuild on both boxes — demo-hp 0 in 5 reports 2026-08-03 06:12:19–06:21:03; demo-felhom 0 in 4 reports 07:17:54–07:19:46. Claimed is set only by SetClaimed(), whose sole caller is the claim submit handler (web/claim.go:349), so the return to 1 was a human re-claim, not automation.

Note, not acted on: a fresh encryption.key means every pre-rebuild app.yaml ENC: secret is undecryptable — the C-1 circular dependency in _recovery-inventory-2026-07-28.md fired on both demo boxes this week. That is outside R-193's scope and is recorded as an observation (§13).

---

6. Q4 — what the stale-escrow detector actually covers

RULING: the prompt's premise conflates two different mechanisms. escrow_confirm.go DOES fire on a mint. But it fires into a log file, and the remedy it prescribes destroys the last copy of the old key.

The conflation, stated because it matters. Yesterday's spike's sentence "wired to the one path that doesn't change the key and absent from the one that does" is about the hub's escrow_stale event, emitted by ReissueCredentials (hub/internal/offsite/offsite.go:198-201) — that is R-196, and it stands. It is not about controller/internal/report/escrow_confirm.go. Two different detectors; only one of them was ever claimed to be misplaced.

What escrow_confirm.go does on a mint, from source. After a rebuild the apply creates a target with EscrowState: "pending" (the cur == nil fallthrough, offbox.go:482-494). So Pending() is true, the pending branch runs (escrow_confirm.go:86-107), the hub's hash (sealing the OLD password) is compared against LocalHash() (the NEW minted one), they differ, and it logs:

[WARN] [escrow-confirm] the hub's escrow blob does not cover the CURRENT repo password
(hub hash %.12s… != local %.12s…) — run the escrow ceremony … staying pending

So: confirmed, it fires. The detector is not absent from the mint path.

Three limits, each of which matters more than the fact that it fires.

1. It is a controller-local log line. No event, no e-mail, no hub-side signal. StaleBlob() — the flag the web card renders — is set only by the escrowed branch (reconcileEscrowed, :130-165); the pending branch sets no display flag. The operator learns nothing.
2. Its prescription is the destructive act. "run the escrow ceremony" → a new ceremony → SaveHostEscrow overwrites host_escrow.identity_blob → demoteCurrentEscrowTx does not carry it → the old repo password ceases to exist. The detector correctly identifies the situation and then tells the customer to close the door.
3. It cannot distinguish the two situations that need different answers — "you re-keyed deliberately" and "your machine was rebuilt and your history is about to be orphaned" produce the identical warning.

So: less needs building than the prompt feared for the detection, and more needs building for the consequence. The comparison exists; what does not exist is a path from that comparison to either a signal or a recovery.

---

7. Q5 — is the superseded-escrow retention real, and what does it retain?

RULING: the retention is real, it retains the wrong blob, and nothing can read it back.

What it retains — source and live, agreeing. demoteCurrentEscrowTx (store/store.go:2547-2556):

INSERT INTO host_escrow_superseded (host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at)
SELECT host_id, blob, key_fingerprint, posture, created_at, COALESCE(restic_pw_sha256, ''), datetime('now')
FROM host_escrow WHERE host_id = ?

identity_blob is not in the column list. Live schema (I2, PRAGMA table_info):

table	columns
host_escrow	host_id, blob, key_fingerprint, posture, created_at, updated_at, **identity_blob**, directive_json, restic_pw_sha256, stale_at
host_escrow_superseded	id, host_id, blob, key_fingerprint, posture, created_at, restic_pw_sha256, superseded_at

blob is the K-escrow — the PBS datastore encryption key, wrapped by the PBS-native scrypt path. identity_blob is the age-wrapped IdentityBundle — the one carrying ResticRepoPassword (felhom-agent/internal/escrow/identity.go:34-39), plus TunnelToken, PBSToken and WGPrivateKey. Live sizes confirm they are different objects: both hosts' current rows hold blob = 383 B and identity_blob = 572 B; both superseded rows hold blob = 383 B and nothing else.

So, exactly what a customer with the matching OLD recovery code could recover from a retained blob:

• ✅ the PBS datastore encryption key K — i.e. the Plane-2 whole-guest offsite ciphertext;
• ❌ not the restic repository password — the Tier-3 offsite app-data history;
• ❌ not the tunnel token, not the PBS access token, not the WireGuard private key.

And there is no read path. I4: ListSupersededEscrow (store/store.go:2617) has zero production callers — only two tests. The only production surface is CountSupersededEscrow, used for a number on the host page (web/hosts.go:573) and a log line (api/handler.go:1151). The schema comment says so itself: "The ACK and restore-serving read host_escrow (the CURRENT row) — never this table." Retrieval is sqlite3 on the hub pod, by hand, by the operator.

The claim this contradicts. 00-capability-map.md (escrow-ceremony row) records hub v0.60.0 as:

"a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (host_escrow_superseded), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from 'history destroyed' into 'history recoverable')."

The same claim appears in the schema comment (store.go:370-375) and, in Hungarian, on the customer's own orphan card (controller/internal/web/templates/backups_remote.html:66,69):

„A meglévő mentések nem sérültek — a hozzájuk tartozó helyreállítási kóddal később visszaállíthatók lehetnek." · „A régi előzmény félretéve marad (nem törlődik), és a hozzá tartozó helyreállítási kóddal később visszaállítható lehet."

For the offsite restic repository — the incident it names — that is false. The retained blob does not contain the key that opens it. This is the ninth entry in CLAUDE.md's table of comments asserting an invariant the code does not provide, and the first that is also customer-facing copy. → R-198, and the capability-map correction in §9.

---

8. Q6 — has any part of this ever been exercised?

Method: read every audit, validation doc, findings file and REPORT that mentions escrow consume, identity-consume, re-enroll or inject-password; then check what the cited evidence actually covers.

Proven

what	evidence	scope of the proof
ceremony (create + seal + upload)	00-capability-map.md escrow row; tests/VALIDATION-n100-rehearsal-2026-07-18.md S6 — customer drove the wizard, R claimed one-shot 16:56:39, auto-confirmed 16:56:41	the create side only
identity-bundle round-trip with a real R on a secret-less box	tests/slice10d-identity-restore-spike-findings.md §1, 2026-06-10 — wrap→unwrap byte-identical (sha256 match); wrong-R fails closed, no plaintext emitted	PRE-FORK-4. The bundle then was {tunnel_token, pbs_token}. ResticRepoPassword was added in agent v0.77.0 on 2026-07-09 (felhom-agent/CHANGELOG.md:2344) — a month later. The proof does not cover the field this whole question is about
ResticRepoPassword survives the wrap/unwrap	agent v0.77.0 CHANGELOG: "IdentityBundle round-trip carries ResticRepoPassword byte-exact + not-in-blob + wrong-R fails closed"	unit test only
InjectOffboxPassword places and refuses correctly	controller/internal/backup/offbox_test.go:952-990, incl. the companion showing a fresh mint without inject	unit test only

Never exercised — named in those words

• A blob has never been served to a box. Link 6 has no client (§3).
• A fork-4 bundle has never been unsealed with a real R outside a unit test. The only production caller is a --selftest mode; the one live consume ever prepared (S5 Part 4-B, 2026-07-04) was explicitly deferred and never run — its own resume note still reads "DEFERRED 2026-07-04, not run" (project memory; 06-offsite-connectivity.md:327 marks the destructive drill operator-gated).
• A recovered repo password has never been injected into a controller.
• An existing offsite repository has never been reopened with a recovered password.
• No restore of any kind has ever been performed from a recovered secret.

_recovery-inventory-2026-07-28.md already said most of this and said it accurately — A.2.7 records "the consume side is proven at spike level … the destructive in-place drill is explicitly operator-gated and unrun", and C.1 row 1 says "escrow creation proven live; consume never exercised destructively". That document is right and this session confirms it. What it did not catch — because it was written from host_escrow, not from the supersession path — is Q5.

Capability-map correction owed: not for a "recovery proven" claim (no row makes one), but for the retention claim quoted in §7. See §9.

---

9. Q7 — the operator's ruled shape, assessed row by row

Operator ruling, 2026-08-04, verbatim: If a node is a fresh install AND the hub has a recovery blob, then the controller should yell that recovery is available, and provide a form for the customer to enter the recovery key. After unlocking the blob, the controller should show what will be recovered before proceeding.

The ruling requires	Exists?	What it needs, and what it costs
the controller knows it is a fresh install	YES, at the right line	os.Stat(m.offboxPwPath()) → IsNotExist is already the mint branch (offbox.go:392). It is trustworthy as "this controller has no repo password", which is the only fact the decision needs; it does not by itself distinguish first-ever install from rebuild — that is what the next row supplies. Corroborators exist but are weaker: Claimed is set-only and was measured re-set by a human within minutes (§5); a fresh encryption.key is a stronger signal but is not currently read by anything. Cost: zero — the branch already exists.
the controller knows the hub has a blob	YES on the wire, NO in the code	EscrowStatus{IdentityBlobPresent, ResticPwSHA256, CreatedAt} arrives on every report ACK and is discarded when no offbox target exists (§5). Needs: persist it (the ClaimSync set-only-by-generation pattern is the precedent, claim_sync.go:39-53). Cost: small — controller only, one settings field, one reconciler, no hub change.
it can yell rather than silently mint	NO	The refusal belongs inside WriteOffboxSecrets, at the mint branch — the one place that decides. Returning a new sentinel (ErrOffboxRecoveryAvailable, sibling of ErrOffboxOrphaned) propagates through ApplyOffsiteTarget → ConfigureOffbox → the apply-bridge, which already treats a failed configure as "persist nothing, retry next cycle" (offsiteapply.go:5-6, 256) — so a refusal is fail-safe by construction, not a new failure mode. Meanwhile the customer sees a card on /backups/remote (the orphan card at backups_remote.html:60-70 is the exact precedent) and offsite runs stay blocked, which they already are (OffboxRunnable). Cost: small-medium — controller only. Design hazard to name: a first-ever install of a re-used customer record would also trip this; the refusal must be dismissible ("start fresh anyway"), and dismissal must be the act that mints.
a form takes R	NO — and R is not a password	The controller UI emits R (backups_escrow.html:235-236, /backup/escrow/…/claim returns it once) and has never accepted one. The only thing resembling an entry point is POST /backup/offbox/inject-password, which takes a 64-hex repo password (offboxRepoPwPattern, offbox.go:543) — a different secret at a different layer — and has no form in any template (§3 link 9). Cost: small for the form; the hard part is the next row.
the blob is unsealed with R	NO — and there is a real boundary	Unsealing is age -d. ageBinary = "/usr/bin/age" (identity.go:24) and the code live in the agent. The controller image does not ship age (controller/Dockerfile:56-70 — restic, openssh-client, sshpass, sqlite3, docker-cli; no age) and must not: the agent is the privileged tier by design. The crossing already has a transport and a precedent: the pinned local API carries POST /escrow/ceremony + GET /escrow/ceremony/status + POST /escrow/ceremony/claim (localapi/server.go:524-527) — i.e. the agent already handles R server-side for the create direction and hands the result back exactly once. Needs: one new agent local-API endpoint (POST /escrow/unseal: blob + R in, the single restic_repo_password field out, R never logged, R never persisted) — the exact mirror of the claim endpoint. Plus a way for the blob to reach the agent: the agent's hub client has no escrow GET (only the PUT at main.go:2937), and dr.go's re-enroll is the wrong tool (it rotates the host API key and needs operator-armed recovery mode). Needs: a second small hub endpoint — GET /hosts/<id>/escrow, per-host-key authed, serving opaque bytes to the box that owns them. Zero-knowledge is untouched: the hub still never decrypts, and the bytes are useless without R. Cost: medium — three repos, two new endpoints, both narrow.
"show what will be recovered" before proceeding	the primitive EXISTS; the flow does not	restic snapshots --json + restic stats --mode raw-data --json are already how the box counts snapshots and sizes the repo (offboxRecordStats, offbox.go:1234-1265). Both are read-only against the repository — nothing is written, no lock is taken, no init. So a preview can show: snapshot count, per-snapshot time, host, tags (= app names), paths, and the deduplicated repo size — all before committing. Committing requires only that the recovered password be written to offboxPwPath(), and even that is reversible in principle. The clean shape: unseal → hold the recovered password in memory → run the two read-only probes against the existing repo → render → and only on the customer's confirmation call InjectOffboxPassword. Cost: small — one handler, one template, no new restic capability. Caveat, measured not assumed: offboxRecordStats today also persists RepoSizeBytes into settings as a side effect; a preview path must not reuse it verbatim.
proceeding actually recovers	NO end-to-end	InjectOffboxPassword → next WriteOffboxSecrets keeps the file → ensureOffboxRepo probes and clears RepoState. Every piece is implemented; none has ever run against a recovered password (§8). Cost: zero new code, but it is precisely what the drill in §10 has to prove.

Things the ruling implies that do not exist, priced

missing piece	cost	note
persisted ACK escrow status + fail-closed mint refusal	S	controller only; the ClaimSync pattern applies directly
the "recovery is available" card + dismissal	S	one template, one flag; the orphan card is the template precedent
a customer-facing recovery-code form (distinct from the password form)	S	the password form's handler exists but has no UI; both need building
agent local-API POST /escrow/unseal	M	new privileged surface; mirror of /escrow/ceremony/claim
hub GET /hosts/<id>/escrow (opaque, per-host key)	S	new read surface on the hub; zero-knowledge preserved
preview handler + template driven by read-only restic probes	S	primitive already exists
carrying identity_blob into host_escrow_superseded	S	one column, one SELECT list — and without it the whole feature protects only boxes that have not yet re-escrowed (→ R-198). This is a prerequisite, not a nice-to-have

The security question the ruling raises — stated, not answered

What gates the surface today. The controller dashboard is behind RequireAuth (internal/web/auth.go:52): a bcrypt password (effectivePasswordHash() — customer-set in settings.json, else operator-provisioned in controller.yaml), a 7-day session cookie, 5 failed logins per minute then lockout, and CsrfProtect on every mutating route. The claim gate serves only the claim page on an unclaimed box. So a recovery form would sit behind the customer's dashboard password, on a controller reachable at whatever address that box's dashboard is reachable at.

What a preview would expose to someone who guesses or steals R. Not file contents — but the snapshot list is not nothing: backup dates and cadence, the app names (restic tags), the paths, and the repository size. That is a map of the household's digital life, and it is precisely the metadata the zero-knowledge posture otherwise keeps out of everyone's hands including the operator's.

Three specifics the operator should weigh, without this session choosing:

1. Guessing R is not the threat; holding the dashboard password is. R is ten EFF words (~129 bits, 00-capability-map.md agent v0.93.0 note) and age's scrypt is deliberately slow — online guessing is not a route. But anyone with the dashboard password can already restore live offsite data (_recovery-inventory-2026-07-28.md A.1.3–A.1.6). The marginal new exposure is confined to the orphaned/previous repository, which today nobody can read at all.
2. The form is an oracle. A correct/incorrect R answer distinguishable in the UI turns the dashboard into a verification service for a stolen recovery code. UnwrapIdentity already fails closed with a single generic error (identity.go:157) — the UI must not do better than that, and should rate-limit like the login path does.
3. R will pass through the agent. Whatever the form does, R crosses the local API to a root-adjacent daemon. The ceremony already does this in the other direction, so the precedent is set — but it means R is momentarily present in agent process memory on the Proxmox host, which is the same host D6 says the operator has root on. This is the same trade as spike option (c), in a smaller and time-bounded form, and it should be decided alongside it rather than separately.

---

10. Q8 — the drill design

Why a deliberate cycle. No recovery code exists for either demo box, and — per §7 — even one would no longer open the old repositories. The proof therefore requires creating R, keeping it, and walking the full circle. This section is the design; the drill is not run in this session.

Target and preconditions

Box	demo-hp (HP t740, node felhom-host, host demo-hp-bb76ea). Tier 0 — disposable, and the designated drill host by the 2026-07-25 operator ruling. Not demo-felhom — it is the N100 that also carries the ISO/rehearsal lineage; leave one demo box in a known-good state throughout
Guest	the customer guest on that host (rebuild target)
Data	one deliberately identifiable app + a sentinel file whose content is a known string and whose sha256 is recorded before the wipe
Prereq	the box is healthy, offsite target configured and escrow_state: escrowed, and its current host_escrow.identity_blob seals the password of the repo that will be tested
Duration	~3–4 h wall clock, of which ~45 min is operator-attended (steps 2, 5, 8, 9)
Blocking dependency	steps 8–9 cannot be executed as designed today — links 6, 8 and 9 of the chain do not exist as automation. The drill as written uses the manual substitutes (§4a) and that is the point: it measures the existing chain honestly before anything is built

The cycle

#	Step	Who	Observable that must be recorded
1	Record the starting state: host_escrow.restic_pw_sha256, identity_blob length, $.offsite.snapshot_count, repo_size_bytes	CC	a hash and two counters, from the hub DB
2	Run the escrow ceremony through the customer wizard and KEEP R	HUMAN	R written to the operator's password manager under a named entry; its storage location recorded in the drill report — the code itself never
3	Deploy an app, write the sentinel file, record its sha256	CC	sha256 + path
4	Run an off-site backup through the real UI endpoint; wait for completion	CC	$.offsite.last_status = ok, snapshot_count increased by ≥1, last_success stamped
5	Verify the snapshot from outside: restic snapshots --json against the repo	CC	snapshot id + tag + timestamp — the pre-wipe truth to compare against
6	Wipe: destroy the guest and reinstall it (the same procedure that produced the 2026-08-03 rebuild)	HUMAN decision, CC executes	new controller_started; claimed drops to 0; a new encryption.key mtime
7	Let the box come back and reach the mint. Do NOT run a new ceremony	CC	the controller log carries the escrow-confirm mismatch WARN (§6); escrow_state: pending; host_escrow.identity_blob unchanged (this is the assertion that keeps recovery possible)
8	Recover: export identity_blob from the hub, unseal with the kept R, extract restic_repo_password	HUMAN (R is theirs)	identity-consume exits 0; the bundle JSON contains a non-empty restic_repo_password; the value is never printed or logged
9	Inject the recovered password (POST /backup/offbox/inject-password, by hand — there is no form), re-apply, run an off-site backup	HUMAN-assisted	restic cat config succeeds; snapshot_count ≥ the step-5 count + 1, not 1
10	Restore the sentinel through the customer restore flow and compare sha256	CC	byte-identical sha256 to step 3
11	Record everything, tear down nothing that would hide the evidence	CC	drill report

The single observation that makes this a FAILURE rather than a pass

Step 10's sha256 does not match step 3's — or step 9 reports a snapshot count of 1.

A snapshot count of 1 means a new, empty history was started over the old one: the repository did not reopen, and every earlier snapshot is orphaned. Explicitly: "the repository opened" is not the pass condition. restic cat config succeeding proves the password is right; it does not prove the customer's data came back. Only the sha256 comparison does.

Secondary failures worth recording separately (each is a finding, not a pass/fail)

• Step 7 finds identity_blob changed — something re-escrowed automatically → the recovery window closes without a human act (would be new and serious).
• Step 8's bundle has an empty restic_repo_password — a pre-fork-4 or hash-less blob shape.
• Step 9's ensureOffboxRepo classifies orphaned despite the correct password → the classifier or the transport, not the key.

What could make it fail for uninteresting reasons

The age binary absent on the machine running step 8; the hub DB export producing a truncated blob (use writefile, verify the byte length against length(identity_blob) = 572); the CSRF token in step 9 (felhom_session is dropped by curl's cookie jar — project memory controller-api-headless-access); and the R-71a settle gate deferring the apply in step 9 (wait for its GO line rather than retrying).

Rollback, and whether the loss is acceptable

If the recovery does not work, what has been lost is: demo-hp's guest, its app data since the last whole-guest archive, and its off-site history. On that box that is acceptable — it is Tier 0, explicitly disposable, and it is the box the target-selection runbook names for drills. The rollback is a whole-guest restore from the local/PBS archive taken immediately before step 6, which is a proven path on this fleet (restore-test passed on demo-hp's host tier in 109.3 s on 2026-08-04). Take that archive deliberately as part of step 6 and verify it exists before destroying anything — do not rely on the nightly.

Not acceptable, and therefore not proposed: running this on demo-felhom (leaves no healthy demo box), on DooPlex (Tier 2, it is the recovery chain), or on any real customer's box.

---

11. Every query used

#	Source	Query / read	Value read
1	git	git -C <repo> rev-parse HEAD; git status --porcelain	the three commits in §1; all empty
2	hub DB	PRAGMA integrity_check	ok
3	hub DB	SELECT MAX(received_at) FROM host_reports; SELECT datetime('now')	2026-08-04 10:00:40 / 2026-08-04 10:03:48
4	hub DB	SELECT host_id, length(blob), length(identity_blob), substr(restic_pw_sha256,1,12), created_at, updated_at FROM host_escrow	demo-felhom-8363b5, 383, 572, c60c8bc737a6, 2026-08-04T07:20:07Z, 2026-08-04 07:20:08 · demo-hp-bb76ea, 383, 572, 8a9e33aa4da6, 2026-08-04T07:15:36Z, 2026-08-04 07:15:36
5	hub DB	SELECT id, host_id, length(blob), substr(restic_pw_sha256,1,12), created_at, superseded_at FROM host_escrow_superseded	3, demo-hp-bb76ea, 383, 8e03eddf9ff7, 2026-07-23T10:01:17Z, 2026-08-04 07:15:36 · 4, demo-felhom-8363b5, 383, 48741892f0ef, 2026-07-21T08:38:33Z, 2026-08-04 07:20:08
6	hub DB	PRAGMA table_info(host_escrow) / (host_escrow_superseded)	the two column lists in §7 — identity_blob present in the first, absent from the second
7	hub DB	latest report per customer: json_extract(report_json,'$.claimed' / '$.offsite.escrow_state' / '.snapshot_count' / '.repo_state' / '.last_status' / '.last_run')	demo-felhom 2026-08-04 09:49:42: 1, escrowed, 0, NULL, NULL, NULL · demo-hp 2026-08-04 09:56:57: 1, escrowed, 0, NULL, NULL, NULL
8	hub DB	claimed grouped over reports since 2026-08-03 05:00	demo-hp: 1 ×124, 0 ×5 (06:12:19→06:21:03) · demo-felhom: 1 ×120, 0 ×4 (07:17:54→07:19:46)
9	hub DB	SELECT customer_id, event_type, severity, created_at FROM events WHERE created_at > '2026-08-03 00:00'	no escrow_stale, no offsite_reissued, no offbox_repo_orphaned, no claim event in the window; controller_started ×4 demo-hp / ×4 demo-felhom around the rebuilds
10	guest 9201 (I3)	ls -la --time-style=full-iso …/felhom-controller-data/_data/data{,/offbox}	the mtime table in §5
11	source (I4)	grep -rn "InjectOffboxPassword" --include=*.go .	1 definition, 1 handler caller, 5 test uses, 1 comment — no production caller besides the handler
12	source (I4)	grep -rn "inject" internal/web/templates/	one unrelated hit (an XSS comment) — no form
13	source (I4)	grep -rni "reenroll|re_enroll" --include=*.go . in felhom-agent	zero hits
14	source (I4)	grep -rn "re-enroll|recovery-mode|restore-directive" --include=*.sh --include=*.md --include=*.py workspace-wide	only design docs + project memory; no script, no runbook step
15	source (I4)	grep -rn "ListSupersededEscrow" in felhom.eu	2 test callers, 0 production
16	source (I4)	grep -rn "\.Escrow\b" --include=*.go . in felhom-controller	one hit: cmd/controller/main.go:641
17	source	felhom-agent/CHANGELOG.md	v0.77.0 — fork-4: escrow the offsite restic repo password under R (2026-07-09)
18	source	controller/Dockerfile:36-70	runtime package list — no age

---

12. What could not be established

Named, not guessed.

1. The 2026-08-05 ~02:15 UTC scheduled run's outcome. Still unmeasured — both boxes became runnable only on 2026-08-04 morning and neither has run since (query 7: last_run absent on both). Yesterday's spike predicts ErrOffboxOrphaned; that prediction is unchanged and still unproven.
2. Whether demo-hp's data dir was fresh. Established for demo-felhom by direct mtime measurement (I3). demo-hp was not measured — it has no baked SSH key and reaching it needs the break-glass credential, which this read-only session did not use. The hub-side claimed 1→0→1 signature is identical on both boxes, which is suggestive, not decisive.
3. Whether the orphaned ciphertext still exists at the provider. Unchanged from yesterday: reading the endpoint needs the SFTP credential, which was not touched.
4. Whether any pre-2026-07-21 escrow generation ever existed for demo-felhom. The superseded table holds two rows in the entire database; earlier generations are not retained (and, per §7, would not have carried the identity blob anyway).
5. Whether age is present on any machine that would run step 8 of the drill. Not checked — the agent declares /usr/bin/age as a runtime dependency but this session did not verify it on either demo host.

---

13. Observations — noticed, not acted on

1. A fresh encryption.key on every rebuild (I3: 32 B, mtime 2026-08-03 07:17:26) means every pre-rebuild app.yaml ENC: secret on that box is undecryptable. This is exactly circular dependency C-1 in _recovery-inventory-2026-07-28.md, and it fired live on both demo boxes this week without anyone naming it. It is a strictly larger blast radius than the offsite repo and it is not covered by R-193. Worth its own item if the operator agrees it is not already implied by C-1.
2. Claimed returning to 1 within minutes of a rebuild, with no event of any kind recording it, means "the customer claimed this box" has no audit trail. There is no customer_claimed event type.
3. offboxRecordStats persists RepoSizeBytes as a side effect of counting — harmless today, a trap for anyone reusing it as a read-only preview primitive (§9).
4. handleReEnroll returns the new API key in the response body (dr.go:148) while also logging the rotation. Not a defect (the box needs it), but it is a secret in an HTTP response on an endpoint nothing currently calls — worth a second look before anything starts calling it.

---

14. The operator's decisions — stated, and NOT answered here

1. Run the drill (§10) or not? It is the only thing that converts "the pieces look right" into "a customer got their data back". It costs demo-hp for an afternoon and, if it fails, that box's history. Recommendation: yes — but fix identity_blob retention (R-198) first, because a drill that walks the current chain will pass or fail on a chain that is missing a link everyone believed was there.
2. The orphaned snapshots at the endpoint (40.9 MB + 1.14 GB, presumed present, §12.3). They are unrecoverable by two independent mechanisms now: no recovery code was kept, and the sealed passwords were destroyed by the 2026-08-04 re-ceremonies. They consume the 50 GB soft quota indefinitely; nothing prunes a moved-aside restic repo. Delete, or leave? This session did not act on it and states it as the operator's.
3. The unchanged R-193 question — agent-retained repo password (spike option c) versus accepting that every rebuild starts a new repository. This session adds one input to that decision: the "accept it, provided the system says so loudly" branch currently rests on a retention feature that does not retain the relevant secret. Under option (d)-alone as it stands today, "the old history stays recoverable with your recovery code" is not true. Either R-198 is fixed, or that branch has to be re-described honestly to the customer.