felhom.eu/REPORT.md

REPORT — the offsite backup key comes back (2026-08-04, R-199 closed, R-200 half)

Class: implementation across three repos + one decisive live proof. Nothing wiped, deleted, rotated or reconfigured; no customer-facing change in any language.

---

1. SCENARIO A — THE RESULT, FIRST, BECAUSE IT IS THE SESSION

On demo-felhom, 2026-08-04 13:49 CEST, through the real endpoints end to end:

=== offsite key recovery check (R-200) — compares, never installs ===
  on-disk   sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb
  recovered sha256: c60c8bc737a6b7c6647c7849283f52087f650a885babedb4ef5fdf9a5c9543cb
  [MATCH] the offsite repository password IS recoverable from the sealed escrow.
  Nothing was written: this check compares and never installs.
EXIT=0

The hashes matched. And they match a third, independent source: host_escrow.restic_pw_sha256 on the hub reads c60c8bc737a6… for demo-felhom-8363b5, recorded at the ceremony and never involved in this comparison. The box's file, the sealed bundle and the hub's record all agree.

The offsite repository password is recoverable from the sealed escrow with the customer's recovery code. That had never been demonstrated — the one live round-trip on record (2026-06-10) predates the field by a month.

2. Part 0 — the recovery code

Answered by the operator before the session began: the 2026-08-04 07:16/07:20 codes had not been kept, so the operator ran a fresh ceremony on both nodes and saved the codes (supplied to this session as R_DEMO-FELHOM in the DooPlex credentials file). No ceremony was run by CC.

Measured hub-side, and worth recording because it exercised two of yesterday's behaviours live: the new escrow rows are stamped 2026-08-04T11:11:37Z (demo-hp) and 11:13:06Z (demo-felhom), and restic_pw_sha256 is unchanged on both — so SaveHostEscrow correctly treated them as same-password re-ceremonies: no superseded row was created and no offsite_repo_key_changed fired. That is v0.93.0's Scenario E, live. It also means yesterday's identity_blob retention is still unit-proven and structurally verified, not yet exercised live — no supersession has happened.

Subject box: demo-felhom (Tier 0, reachable with a baked key, current escrow). Not demo-hp (break-glass only), not DooPlex, not ep0.

3. The §8.2 decision as built, and where the gate is

Built as §8.2: GET /api/v1/hosts/{host_id}/escrow, authenticated with the box's own per-host key, self-scoped (global may read any — the same asymmetry the PUT has).

The single gate point is api.escrowSelfServiceRetrieval (hub/internal/api/handler.go), a named constant. false re-imposes the recovery-mode requirement on this endpoint and changes nothing else — the same unseal, the same extraction, the same proof, minus the self-service property. The trade is written above the handler in dr.go's style: what it gives out, why it is safe, and what it changes about who is required — obtaining the blob used to need an operator to arm recovery mode and now needs only that box's credential. They still cannot open it. The mitigation is the audit row.

4. Files, commits, deployed versions

Repo	Commits	Version	Deployed
felhom.eu	435f4a5 (code) · dd08926 (manifest) · docs commit below	hub v0.94.0	ArgoCD Synced/Healthy, felhom-hub:0.94.0
felhom-agent	6d79047	v0.125.0	published sha f7d8339b53d9…, verified by independent download, installed on demo-felhom (felhom-agent --version → 0.125.0, active). NOT vouched
felhom-controller	9640e51	v0.195.0	felhom-controller:0.195.0 on guest 9201, Up (healthy)

Hub: internal/api/handler.go (route + handleHostEscrowGet + escrowSelfServiceRetrieval), internal/notify/dispatcher.go (escrow_blob_served → operatorOnlyEvents), internal/api/escrow_get_test.go (new). Agent: internal/escrow/recover.go (new — OffsiteKeyRecoverer, distinct errors), internal/hub/client.go (FetchIdentityEscrow), internal/localapi/escrow_recover.go (new route), internal/localapi/server.go (seam + registration), cmd/felhom-agent/main.go (production wiring + the §8.6 selftest message fix), plus recover_test.go and escrow_recover_wiring_test.go (new). Controller: internal/backup/offbox_recovery_check.go + offbox_recovery_cli.go (new), internal/agentapi/escrow.go (RecoverOffsiteRepoPassword), cmd/controller/main.go (--recover-offsite-check), plus offbox_recovery_check_test.go (new).

Green gate in each repo after each phase (go build && go vet && go test ./..., rc=0), plus repo_gates.py / agent_gates.py / controller_gates.py — all OK. No test run was combined with a commit.

5. Tests and every red-proof

Scenario	Result	Red-proof — mutation → outcome
A recovered key == on-disk key	PASS (unit, real age) + PROVEN LIVE	return bundle.PBSToken instead of ResticRepoPassword → FAIL: "the recovered key is not the sealed repository password (len 24 vs 64)". This is the mutation that mattered: every field of that bundle is a non-empty string that looks like a secret
B wrong R fails closed, nothing written	PASS + PROVEN LIVE	— (the property is age's KDF, asserted rather than guarded, so there is no guard to remove)
C a box fetches only its own blob	PASS	delete the !isGlobal && authHostID != pathHostID check → FAIL: host A was served host B's blob, body and all
D no blob → a clean "none"	PASS	—
E every retrieval leaves a record	PASS + PROVEN LIVE	if false && on the audit block → FAIL: "a sealed recovery blob was served and NOTHING recorded it"
F R persists nowhere	PASS + PROVEN LIVE	write R to a temp file inside the unseal → FAIL (see the correction below)
G the operator DR path is untouched	PASS	—
H the seam is wired (AST)	PASS	_ = escrowRecoverer with the Options field removed → FAIL: "localapi.Options … carries no EscrowRecovery field"
§8.5 compare-never-install	PASS + PROVEN LIVE	add InjectOffboxPassword to the check → FAIL: "the check INSTALLED the recovered password"

TWO MUTATIONS DID NOT REMOVE EVERY GUARD ON THE FIRST ATTEMPT, AND BOTH ARE RECORDED RATHER THAN QUIETLY REDONE (§9.13 — two of the last four sessions found a red-proof that passed):

1. Scenario F. The planted leak wrote R into the walked directory and the test still passed — because a later failing call OVERWROTE the leak file with the wrong code, and the test only scanned for the first one. Instrumenting the mutation (printing where it wrote) is what found it. The test was wrong, not the mutation. It now asserts emptiness of the TMPDIR tree as its primary check — nothing under it is created by the test, so any survivor is a leak regardless of content — with the content scan kept as defence in depth over all three secrets. Re-run: FAIL, "the unseal left 1 file(s) behind under TMPDIR".
2. Scenario H. Commenting the wiring line out failed to compile (declared and not used), which is not a red-proof. The mutation that compiles keeps the variable alive (_ = escrowRecoverer) and drops the struct field. Re-run: FAIL as quoted above.

Every -run filter was verified to have matched (-v, === RUN lines counted) before any result was read as a proof.

6. The live validation trail, in order

1. Wrong code first, deliberately (13:44:44) — hub: escrow blob SERVED to host demo-felhom-8363b5 (572 opaque bytes, self_scope=true) — recovery path in use; agent: offsite key recovery FAILED … err="escrow: the recovery code did not unwrap the identity escrow (wrong recovery code, or a corrupt blob): exit status 1"; CLI: [FAIL] … nothing was written., exit 1. This is Scenario B live AND it proves links 6 and 7 ran independently of the success — the blob was fetched and the unseal was attempted and refused.
2. The real code (13:49:47) — §1's output, exit 0.
3. Nothing was written — offbox/repo_password mtime on the live box is still 2026-08-03 07:18:02, unchanged by a check that ran at 11:49 UTC. The whole offbox/ directory is byte-for-byte the pre-check listing.

7. The retrieval record as it appears at the hub

customer_id  severity  created_at           details_json
demo-felhom  warning   2026-08-04 11:44:44  {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true}
demo-felhom  warning   2026-08-04 11:49:47  {"blob_bytes":572,"host_id":"demo-felhom-8363b5","self_scope":true}

And the routing, which is the half that shows the register works:

customer_id  channel   status      created_at
demo-felhom  operator  sent        2026-08-04 11:44:44
demo-felhom  customer  skipped     2026-08-04 11:44:44      ← operator_only
demo-felhom  operator  suppressed  2026-08-04 11:49:47      ← 1h operator cooldown, itself recorded (R-182)
demo-felhom  customer  skipped     2026-08-04 11:49:47

Both retrievals raised an event; the operator was mailed for the first; the second's mail was cooldown-suppressed and that suppression is written down. The audit is per-retrieval and complete even where the mail is not.

8. Evidence that R persisted nowhere — the search, not a claim

Searched on the subject host with the pattern read from a file (never in argv), across the agent journal, the controller's container log, and /tmp, /var/tmp, /var/lib/felhom-agent, /root:

agent journal lines containing R: 0
controller log lines containing R: 0
host files containing R: 0
leftover felhom-idesc-* staging dirs: 0
staged escrow secret file: 0 entries

With a positive control, because an absent match is not a measurement unless the instrument is shown to work: a copy of R was planted under /tmp, the same sweep found 1, the copy was shredded, and the sweep returned 0. The instrument detects R when R is there.

R reached the box only as: the credentials-file value on DooPlex → a pipe → the container's stdin → the request body → the agent's memory. It was never an argument, never a file on either machine, and appears in no log line at any level.

9. Register

• R-199 → SHIPPED + PROVEN-LIVE. Links 6, 7 and 8 assembled and walked.
• R-200 → plumbing shipped; the FORM is not built and the row stays open for it and for link 9.
• R-201 → OPEN, and the distinction is deliberate: the KEY is proven recoverable; a recovered password has never been installed, no repository has been reopened under one, and no file has ever been restored. The drill's pass condition is unchanged — a byte-identical sentinel file, not "the store opened".
• R-202 → OPEN and untouched. The orphan card still promises recoverability unconditionally.
• The orphaned-ciphertext deletion (~1.2 GB) is STILL OWED — operator ruling 2 of 2026-08-04, deliberately not ridden along with a code release.
• Capability map, 03-host-agent.md §8a, CONTEXT.md S-29, REUSE.md and ROADMAP.md updated; STATUS.md rewritten for the operator.

10. The capability-map row, and what it deliberately does not claim

Added: "The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code" — PROVEN-LIVE (2026-08-04), with the evidence above.

What the row explicitly refuses to claim, stated in the row itself because the previous over-claim on this subject was struck out four hours earlier: it covers the key, not the data. No recovered password has been installed, no repository reopened, no file restored. And the proof used a box whose local key still exists — the rebuilt-box case, where there is nothing to compare against, is exactly what the drill covers and it has not run.

11. CI

Run numbers, task ids and conclusions for all commits are quoted in the session summary. --no-verify was NOT used — every push ran its repo's pre-push gate and passed.

12. Teardown

Nothing was provisioned. The temporary pattern file used by the R sweep and its positive control were shredded on the host by the same command that created them (verified: 0 files remain). No diagnostic artifact persists on either machine. The agent's previous binary is retained as /usr/local/bin/felhom-agent.bak-0.124.1, per the documented deploy.

13. Observations — noticed, NOT acted on

1. The escrow_blob_served mail is subject to the 1-hour operator cooldown — visible above, where the second retrieval's mail was suppressed. The EVENT rows are per-retrieval so the audit is complete, but if retrieval ever becomes routine, a burst would produce one mail. Worth revisiting when the customer-facing flow lands, not before.
2. ListSupersededEscrow still has no production caller. Recovery reads the CURRENT blob; a customer needing a PREVIOUS generation's key (which is what R-198's retention exists for) has no path at all. That is the next gap in this area after the drill.
3. The controller binary is at /usr/local/bin/felhom-controller, not /app/… — the first diagnostic invocation failed on a guessed path. Worth a line in the runbook when one is written.
4. age is at the hardcoded /usr/bin/age on both demo hosts (1.2.1 on demo-felhom) — the recon listed this as unestablished; it is now established for demo-felhom. ageBinary is a package variable with no config override, so a host that installs age elsewhere would fail the unseal at the last step.
5. Yesterday's identity_blob retention has still not been exercised live — the operator's fresh ceremonies sealed the same password, so no supersession occurred. It remains unit-proven with a verified schema; the first real supersession will be its live proof.