From c1cfbe817b44e7a9eabb12add27b1a97cc028f23 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 9 Jul 2026 16:31:58 +0200 Subject: [PATCH] docs(audit): fork-4 custody VALIDATED end-to-end live (supervised) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full cycle on the demo: enable→stage→real escrow-create (K+restic-pw under a fresh R, uploaded to hub, staged wiped, K byte-unchanged)→atomicity proof (run skipped before confirm, proceeded after — gate opens exactly on confirm). DR money shot: restic password recovered from the PRODUCTION hub blob (via identity-consume with R) is byte-identical to the live pw (sha 24c44f49) and decrypted the real box repo; inject→restore endpoints reproduced a 45MB db-dump byte-perfect. Demo left escrowed (un-paused); R held out-of-band; no secret in logs; repo_password unchanged. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CONTEXT.md | 13 +++ .../VALIDATION-fork4-custody-2026-07-09.md | 91 +++++++++++++++++++ 2 files changed, 104 insertions(+) create mode 100644 documentation/audits/VALIDATION-fork4-custody-2026-07-09.md diff --git a/CONTEXT.md b/CONTEXT.md index 98f850e..e189065 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,19 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-09 — fork-4 custody VALIDATED end-to-end live (supervised).** The full custody cycle proven on + the demo (agent v0.77.0 + controller v0.105.0): enable → controller pushes the repo password to the agent → + **real escrow ceremony** (`--selftest=escrow-create`, K + restic-pw under a fresh R, uploaded to the hub, + staged secret wiped, K byte-unchanged) → **atomicity proof:** a run **skipped** before `confirm-escrow` and + **proceeded** after (2 apps, 2 snapshots) — the gate opens exactly on confirm. **DR money shot:** the restic + password recovered from the **production hub blob** (`host_escrow.identity_blob` for `demo-felhom-01`, via + `identity-consume` with R) is byte-identical to the live password (`sha 24c44f49…`) and **decrypted the real + box repo**; the real `inject-password`→`restore` endpoints reproduced a 45 MB db-dump byte-perfect. Demo + offbox left **escrowed** (un-paused); Viktor holds R out-of-band; no secret in any log; live `repo_password` + unchanged. Record: `documentation/audits/VALIDATION-fork4-custody-2026-07-09.md`. **NEXT:** hub-provisioning + + soft-quota spec (hub-verified auto-confirm replacing operator-trust; both storage models); per-customer + onboarding runbook; the `identity-consume` OK-message fix (omits the recovered restic field). + - **2026-07-09 — offbox v0.104.0 VALIDATED live against the Storage Box (supervised).** Closed the offsite arc: controller `:0.104.0`@`908e4b9` ran the real `/backup/offbox/*` endpoints against a fresh box **sub-account** (`u629193-sub1`, RepoPath `/home/felhom-demo-repo`). The capstone proof — **immich** diff --git a/documentation/audits/VALIDATION-fork4-custody-2026-07-09.md b/documentation/audits/VALIDATION-fork4-custody-2026-07-09.md new file mode 100644 index 0000000..7d1f50c --- /dev/null +++ b/documentation/audits/VALIDATION-fork4-custody-2026-07-09.md @@ -0,0 +1,91 @@ +# VALIDATION — fork-4 custody: live escrow ceremony + DR recovery (supervised) + +**Date:** 2026-07-09 · **Class:** risky/supervised operational validation (RUNBOOK) · **Executor:** Claude +Code live, Viktor on the 🛑 secret steps (R capture, recovery). **No code, no version bump.** Closes the +offsite arc: the full enable→stage→ceremony(real R)→confirm→gated-run cycle + DR recovery of the restic +password from the **production hub blob**, decrypting the real Storage-Box repo. Non-destructive (K +read-only/byte-unchanged; live `repo_password` backed-up-then-restored; demo repo never destructively written). + +--- + +## 1. VERDICT + +**fork-4 custody is VALIDATED end-to-end.** Anchored on **Phase 3** (the atomicity gate opens *exactly* on +confirm — skip-before, run-after) and **Phase 4a** (a password recovered from the production hub blob, via +the real `identity-consume` with the ceremony's fresh R, **decrypted the real demo repo** and is byte-identical +to the live password). Live inject→restore (4b) reproduced a 45 MB db-dump byte-perfect through the real +endpoints. The offsite tier is now proven safe for real data, and the standing demo offsite is un-paused. + +--- + +## 2. Baselines + the K gate + +| Thing | Value | +|-------|-------| +| agent | v0.77.0 active on felhom-pve, caps 56/56 degraded=0 | +| controller | `:0.105.0 Up (healthy)` on guest 9201 | +| demo host / offbox | host_id `demo-felhom-01`; sub-account `u629193-sub1` @ `u629193-sub1.your-storagebox.de:23`, RepoPath `/home/felhom-demo-repo`; started **gated** (run refused: "…a kulcs letétbe helyezésére vár.") | +| **K gate** | **PASS** — `/etc/pve/priv/storage/felhom-pbs.enc` (root 0600) present; ceremony uses `--storage felhom-pbs`; `age` + `proxmox-backup-client` present | +| repo_password sha (Phase 0) | `24c44f49…` (recorded as the byte-match target; hash only) | +| demo repo (Phase 0) | 2 snapshots (immich, audiobookshelf) | + +## 3. Phase 1 — enable → stage → pending ✅ +`POST /backup/offbox/config` (enabled) → controller `EscrowState="pending"`; the agent **staged** the pushed +password (0600 `felhom-agent`-owned file at `StagedResticPasswordPath`, 64 bytes, sha `24c44f49…` = the repo +password byte-identical), and the value was **NOT in any agent log (0 hits)**. A `POST /backup/offbox/run` +while pending was **refused** ("…letétbe helyezésére vár.") — the gate holds pre-escrow. + +## 4. Phase 2 — the escrow ceremony (SUPERVISED, real R) 🛑 ✅ +Viktor ran `felhom-agent --selftest=escrow-create --storage felhom-pbs --upload` (as root — K is root-0600). +Result: **self-verify OK**, fresh **R** surfaced once (Viktor captured it; never logged/recorded), the opaque +blob(s) **uploaded to the hub**. Verified after: the **staged secret was wiped** (`StagedResticPasswordPath` +gone); **K byte-unchanged** (sha `1dfa412b…` before == after); **no R/password in the agent log** (0 hits). +The hub stored (host `demo-felhom-01`, created `2026-07-09T14:16:16Z`): identity blob 483 B + K-escrow 383 B. + +## 5. Phase 3 — the atomicity proof ✅ +- **Before confirm:** `POST /backup/offbox/run` → **skipped** (flash "vár") — the ceremony alone does not flip state. +- `POST /backup/offbox/confirm-escrow` → `EscrowState="escrowed"` (flash "…megerősítve — a NAS-mentés mostantól futhat."). +- **After confirm:** `POST /backup/offbox/run` → **proceeded** → `LastStatus="ok"`, **2 app(s) backed up + (immich + audiobookshelf), 2 snapshots, 43s**. The gate opens exactly on confirm — the live atomicity cycle. + +## 6. Phase 4 — DR recovery from the PRODUCTION hub blob 🛑 ✅ +The identity blob the hub stored in Phase 2 was fetched from the hub store (`host_escrow.identity_blob` for +`demo-felhom-01`, age-armored, 484 B) and staged on felhom-pve. *(Method note: fetched from the hub DB, i.e. +the actual production-stored blob; the fuller re-enroll DR endpoint requires operator-armed recovery mode — +not exercised here to avoid side effects on the demo host. The blob recovered IS the one the ceremony uploaded.)* +- **Recover (Viktor, R via `FELHOM_RECOVERY_CODE`):** `felhom-agent --selftest=identity-consume --blob … + --keydest recovered-bundle.json` → `selftest=identity-consume OK`; the recovered bundle (0600, never printed) + carries `restic_repo_password` = **P′**. +- **4a — decrypt proof:** `sha256(P′) == sha256(live repo_password) = 24c44f49…` (hashes only), and + `restic … snapshots` with `RESTIC_PASSWORD_FILE=P′` against the real repo **listed the 2 real snapshots** + (audiobookshelf `6adb3f96`, immich `0d035a8d`, from the Phase-3 run). The recovered-from-hub password is a + **working repo key** — the whole production chain (ceremony → hub → recover → decrypt) validated. +- **4b — live inject→restore (rollback-guarded):** backed up `repo_password`, removed it (simulate fresh), + `POST /backup/offbox/inject-password {password:P′}` → placed 0600 (sha `24c44f49…`); `POST + /backup/offbox/restore {app:immich}` → the restored `immich-postgres.sql` matched the USB source + **byte-perfect** (`d78466fc…`, 45321478 B) after the async restore completed. (First check was a poll race + on the 45 MB file — matched on completion.) + +## 7. Phase 5 — end state +- **Kept escrowed:** the demo offbox is `EscrowState="escrowed"` — offsite runs resume (the demo offsite is + un-paused, a good end state). **Viktor holds the demo's R out-of-band** (existence recorded, not the value). +- Live `repo_password` restored from the rollback backup — sha `24c44f49…`, **unchanged** from Phase 0. +- All scratch shredded: felhom-pve DR dir (P′ + bundle JSON `shred`ed, blob removed), container P′ + the + offbox-restore verify dir, the 180 hub-DB snapshot + blob. **No secret (R, P′, repo password) in any + agent/controller log (0 hits).** + +## 8. Observations +- `identity-consume` OK message says "identity recovered (tunnel_token + pbs_token)" — it **omits the + recovered `restic_repo_password`** (cosmetic; the field IS in the written bundle). A one-line message fix. +- `confirm-escrow` currently **trusts the operator** (pilot): it flips to escrowed with no hub-side proof the + blob actually landed. The provisioning task should replace it with a **hub-verified auto-confirm** (close + the operator-forgets/operator-lies footgun). +- Fetching the DR blob required a hub-DB read; a first-class operator "download my escrow blob" path (or the + recovery-mode re-enroll flow) would make DR self-serve. + +## 9. Arc status / NEXT +The offsite tier (restic-to-Storage-Box, direct, sub-account-per-customer, password escrowed under R, atomic +enable, DR-recoverable) is now **proven safe for real customer data**. **NEXT:** the hub-provisioning + +soft-quota spec (both storage models, hub-verified auto-confirm, soft quota); the per-customer onboarding +runbook (sub-account + key + known_hosts + RepoPath + enable→ceremony→confirm); the `identity-consume` +message fix; the Tier-2 `DoubleClickRefused` flake.