# VALIDATION — fork-4 custody: live escrow ceremony + DR recovery (supervised)

**Date:** 2026-07-09 · **Class:** risky/supervised operational validation (RUNBOOK) · **Executor:** Claude
Code live, Viktor on the 🛑 secret steps (R capture, recovery). **No code, no version bump.** Closes the
offsite arc: the full enable→stage→ceremony(real R)→confirm→gated-run cycle + DR recovery of the restic
password from the **production hub blob**, decrypting the real Storage-Box repo. Non-destructive (K
read-only/byte-unchanged; live `repo_password` backed-up-then-restored; demo repo never destructively written).

---

## 1. VERDICT

**fork-4 custody is VALIDATED end-to-end.** Anchored on **Phase 3** (the atomicity gate opens *exactly* on
confirm — skip-before, run-after) and **Phase 4a** (a password recovered from the production hub blob, via
the real `identity-consume` with the ceremony's fresh R, **decrypted the real demo repo** and is byte-identical
to the live password). Live inject→restore (4b) reproduced a 45 MB db-dump byte-perfect through the real
endpoints. The offsite tier is now proven safe for real data, and the standing demo offsite is un-paused.

---

## 2. Baselines + the K gate

| Thing | Value |
|-------|-------|
| agent | v0.77.0 active on felhom-pve, caps 56/56 degraded=0 |
| controller | `:0.105.0 Up (healthy)` on guest 9201 |
| demo host / offbox | host_id `demo-felhom-01`; sub-account `u629193-sub1` @ `u629193-sub1.your-storagebox.de:23`, RepoPath `/home/felhom-demo-repo`; started **gated** (run refused: "…a kulcs letétbe helyezésére vár.") |
| **K gate** | **PASS** — `/etc/pve/priv/storage/felhom-pbs.enc` (root 0600) present; ceremony uses `--storage felhom-pbs`; `age` + `proxmox-backup-client` present |
| repo_password sha (Phase 0) | `24c44f49…` (recorded as the byte-match target; hash only) |
| demo repo (Phase 0) | 2 snapshots (immich, audiobookshelf) |

## 3. Phase 1 — enable → stage → pending ✅
`POST /backup/offbox/config` (enabled) → controller `EscrowState="pending"`; the agent **staged** the pushed
password (0600 `felhom-agent`-owned file at `StagedResticPasswordPath`, 64 bytes, sha `24c44f49…` = the repo
password byte-identical), and the value was **NOT in any agent log (0 hits)**. A `POST /backup/offbox/run`
while pending was **refused** ("…letétbe helyezésére vár.") — the gate holds pre-escrow.

## 4. Phase 2 — the escrow ceremony (SUPERVISED, real R) 🛑 ✅
Viktor ran `felhom-agent --selftest=escrow-create --storage felhom-pbs --upload` (as root — K is root-0600).
Result: **self-verify OK**, fresh **R** surfaced once (Viktor captured it; never logged/recorded), the opaque
blob(s) **uploaded to the hub**. Verified after: the **staged secret was wiped** (`StagedResticPasswordPath`
gone); **K byte-unchanged** (sha `1dfa412b…` before == after); **no R/password in the agent log** (0 hits).
The hub stored (host `demo-felhom-01`, created `2026-07-09T14:16:16Z`): identity blob 483 B + K-escrow 383 B.

## 5. Phase 3 — the atomicity proof ✅
- **Before confirm:** `POST /backup/offbox/run` → **skipped** (flash "vár") — the ceremony alone does not flip state.
- `POST /backup/offbox/confirm-escrow` → `EscrowState="escrowed"` (flash "…megerősítve — a NAS-mentés mostantól futhat.").
- **After confirm:** `POST /backup/offbox/run` → **proceeded** → `LastStatus="ok"`, **2 app(s) backed up
  (immich + audiobookshelf), 2 snapshots, 43s**. The gate opens exactly on confirm — the live atomicity cycle.

## 6. Phase 4 — DR recovery from the PRODUCTION hub blob 🛑 ✅
The identity blob the hub stored in Phase 2 was fetched from the hub store (`host_escrow.identity_blob` for
`demo-felhom-01`, age-armored, 484 B) and staged on felhom-pve. *(Method note: fetched from the hub DB, i.e.
the actual production-stored blob; the fuller re-enroll DR endpoint requires operator-armed recovery mode —
not exercised here to avoid side effects on the demo host. The blob recovered IS the one the ceremony uploaded.)*
- **Recover (Viktor, R via `FELHOM_RECOVERY_CODE`):** `felhom-agent --selftest=identity-consume --blob …
  --keydest recovered-bundle.json` → `selftest=identity-consume OK`; the recovered bundle (0600, never printed)
  carries `restic_repo_password` = **P′**.
- **4a — decrypt proof:** `sha256(P′) == sha256(live repo_password) = 24c44f49…` (hashes only), and
  `restic … snapshots` with `RESTIC_PASSWORD_FILE=P′` against the real repo **listed the 2 real snapshots**
  (audiobookshelf `6adb3f96`, immich `0d035a8d`, from the Phase-3 run). The recovered-from-hub password is a
  **working repo key** — the whole production chain (ceremony → hub → recover → decrypt) validated.
- **4b — live inject→restore (rollback-guarded):** backed up `repo_password`, removed it (simulate fresh),
  `POST /backup/offbox/inject-password {password:P′}` → placed 0600 (sha `24c44f49…`); `POST
  /backup/offbox/restore {app:immich}` → the restored `immich-postgres.sql` matched the USB source
  **byte-perfect** (`d78466fc…`, 45321478 B) after the async restore completed. (First check was a poll race
  on the 45 MB file — matched on completion.)

## 7. Phase 5 — end state
- **Kept escrowed:** the demo offbox is `EscrowState="escrowed"` — offsite runs resume (the demo offsite is
  un-paused, a good end state). **Viktor holds the demo's R out-of-band** (existence recorded, not the value).
- Live `repo_password` restored from the rollback backup — sha `24c44f49…`, **unchanged** from Phase 0.
- All scratch shredded: felhom-pve DR dir (P′ + bundle JSON `shred`ed, blob removed), container P′ + the
  offbox-restore verify dir, the 180 hub-DB snapshot + blob. **No secret (R, P′, repo password) in any
  agent/controller log (0 hits).**

## 8. Observations
- `identity-consume` OK message says "identity recovered (tunnel_token + pbs_token)" — it **omits the
  recovered `restic_repo_password`** (cosmetic; the field IS in the written bundle). A one-line message fix.
- `confirm-escrow` currently **trusts the operator** (pilot): it flips to escrowed with no hub-side proof the
  blob actually landed. The provisioning task should replace it with a **hub-verified auto-confirm** (close
  the operator-forgets/operator-lies footgun).
- Fetching the DR blob required a hub-DB read; a first-class operator "download my escrow blob" path (or the
  recovery-mode re-enroll flow) would make DR self-serve.

## 9. Arc status / NEXT
The offsite tier (restic-to-Storage-Box, direct, sub-account-per-customer, password escrowed under R, atomic
enable, DR-recoverable) is now **proven safe for real customer data**. **NEXT:** the hub-provisioning +
soft-quota spec (both storage models, hub-verified auto-confirm, soft quota); the per-customer onboarding
runbook (sub-account + key + known_hosts + RepoPath + enable→ceremony→confirm); the `identity-consume`
message fix; the Tier-2 `DoubleClickRefused` flake.