felhom.eu/documentation/audits/VALIDATION-fork4-custody-2026-07-09.md

VALIDATION — fork-4 custody: live escrow ceremony + DR recovery (supervised)

Date: 2026-07-09 · Class: risky/supervised operational validation (RUNBOOK) · Executor: Claude Code live, Viktor on the 🛑 secret steps (R capture, recovery). No code, no version bump. Closes the offsite arc: the full enable→stage→ceremony(real R)→confirm→gated-run cycle + DR recovery of the restic password from the production hub blob, decrypting the real Storage-Box repo. Non-destructive (K read-only/byte-unchanged; live repo_password backed-up-then-restored; demo repo never destructively written).

---

1. VERDICT

fork-4 custody is VALIDATED end-to-end. Anchored on Phase 3 (the atomicity gate opens exactly on confirm — skip-before, run-after) and Phase 4a (a password recovered from the production hub blob, via the real identity-consume with the ceremony's fresh R, decrypted the real demo repo and is byte-identical to the live password). Live inject→restore (4b) reproduced a 45 MB db-dump byte-perfect through the real endpoints. The offsite tier is now proven safe for real data, and the standing demo offsite is un-paused.

---

2. Baselines + the K gate

Thing	Value
agent	v0.77.0 active on felhom-pve, caps 56/56 degraded=0
controller	:0.105.0 Up (healthy) on guest 9201
demo host / offbox	host_id demo-felhom-01; sub-account u629193-sub1 @ u629193-sub1.your-storagebox.de:23, RepoPath /home/felhom-demo-repo; started gated (run refused: "…a kulcs letétbe helyezésére vár.")
K gate	PASS — /etc/pve/priv/storage/felhom-pbs.enc (root 0600) present; ceremony uses --storage felhom-pbs; age + proxmox-backup-client present
repo_password sha (Phase 0)	24c44f49… (recorded as the byte-match target; hash only)
demo repo (Phase 0)	2 snapshots (immich, audiobookshelf)

3. Phase 1 — enable → stage → pending ✅

POST /backup/offbox/config (enabled) → controller EscrowState="pending"; the agent staged the pushed password (0600 felhom-agent-owned file at StagedResticPasswordPath, 64 bytes, sha 24c44f49… = the repo password byte-identical), and the value was NOT in any agent log (0 hits). A POST /backup/offbox/run while pending was refused ("…letétbe helyezésére vár.") — the gate holds pre-escrow.

4. Phase 2 — the escrow ceremony (SUPERVISED, real R) 🛑 ✅

Viktor ran felhom-agent --selftest=escrow-create --storage felhom-pbs --upload (as root — K is root-0600). Result: self-verify OK, fresh R surfaced once (Viktor captured it; never logged/recorded), the opaque blob(s) uploaded to the hub. Verified after: the staged secret was wiped (StagedResticPasswordPath gone); K byte-unchanged (sha 1dfa412b… before == after); no R/password in the agent log (0 hits). The hub stored (host demo-felhom-01, created 2026-07-09T14:16:16Z): identity blob 483 B + K-escrow 383 B.

5. Phase 3 — the atomicity proof ✅

• Before confirm: POST /backup/offbox/run → skipped (flash "vár") — the ceremony alone does not flip state.
• POST /backup/offbox/confirm-escrow → EscrowState="escrowed" (flash "…megerősítve — a NAS-mentés mostantól futhat.").
• After confirm: POST /backup/offbox/run → proceeded → LastStatus="ok", 2 app(s) backed up (immich + audiobookshelf), 2 snapshots, 43s. The gate opens exactly on confirm — the live atomicity cycle.

6. Phase 4 — DR recovery from the PRODUCTION hub blob 🛑 ✅

The identity blob the hub stored in Phase 2 was fetched from the hub store (host_escrow.identity_blob for demo-felhom-01, age-armored, 484 B) and staged on felhom-pve. (Method note: fetched from the hub DB, i.e. the actual production-stored blob; the fuller re-enroll DR endpoint requires operator-armed recovery mode — not exercised here to avoid side effects on the demo host. The blob recovered IS the one the ceremony uploaded.)

• Recover (Viktor, R via FELHOM_RECOVERY_CODE): felhom-agent --selftest=identity-consume --blob … --keydest recovered-bundle.json → selftest=identity-consume OK; the recovered bundle (0600, never printed) carries restic_repo_password = P′.
• 4a — decrypt proof: sha256(P′) == sha256(live repo_password) = 24c44f49… (hashes only), and restic … snapshots with RESTIC_PASSWORD_FILE=P′ against the real repo listed the 2 real snapshots (audiobookshelf 6adb3f96, immich 0d035a8d, from the Phase-3 run). The recovered-from-hub password is a working repo key — the whole production chain (ceremony → hub → recover → decrypt) validated.
• 4b — live inject→restore (rollback-guarded): backed up repo_password, removed it (simulate fresh), POST /backup/offbox/inject-password {password:P′} → placed 0600 (sha 24c44f49…); POST /backup/offbox/restore {app:immich} → the restored immich-postgres.sql matched the USB source byte-perfect (d78466fc…, 45321478 B) after the async restore completed. (First check was a poll race on the 45 MB file — matched on completion.)

7. Phase 5 — end state

• Kept escrowed: the demo offbox is EscrowState="escrowed" — offsite runs resume (the demo offsite is un-paused, a good end state). Viktor holds the demo's R out-of-band (existence recorded, not the value).
• Live repo_password restored from the rollback backup — sha 24c44f49…, unchanged from Phase 0.
• All scratch shredded: felhom-pve DR dir (P′ + bundle JSON shreded, blob removed), container P′ + the offbox-restore verify dir, the 180 hub-DB snapshot + blob. No secret (R, P′, repo password) in any agent/controller log (0 hits).

8. Observations

• identity-consume OK message says "identity recovered (tunnel_token + pbs_token)" — it omits the recovered restic_repo_password (cosmetic; the field IS in the written bundle). A one-line message fix.
• confirm-escrow currently trusts the operator (pilot): it flips to escrowed with no hub-side proof the blob actually landed. The provisioning task should replace it with a hub-verified auto-confirm (close the operator-forgets/operator-lies footgun).
• Fetching the DR blob required a hub-DB read; a first-class operator "download my escrow blob" path (or the recovery-mode re-enroll flow) would make DR self-serve.

9. Arc status / NEXT

The offsite tier (restic-to-Storage-Box, direct, sub-account-per-customer, password escrowed under R, atomic enable, DR-recoverable) is now proven safe for real customer data. NEXT: the hub-provisioning + soft-quota spec (both storage models, hub-verified auto-confirm, soft quota); the per-customer onboarding runbook (sub-account + key + known_hosts + RepoPath + enable→ceremony→confirm); the identity-consume message fix; the Tier-2 DoubleClickRefused flake.