admin
c1cfbe817b
Full cycle on the demo: enable→stage→real escrow-create (K+restic-pw under a fresh R, uploaded to hub, staged wiped, K byte-unchanged)→atomicity proof (run skipped before confirm, proceeded after — gate opens exactly on confirm). DR money shot: restic password recovered from the PRODUCTION hub blob (via identity-consume with R) is byte-identical to the live pw (sha 24c44f49) and decrypted the real box repo; inject→restore endpoints reproduced a 45MB db-dump byte-perfect. Demo left escrowed (un-paused); R held out-of-band; no secret in logs; repo_password unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
92 lines
6.8 KiB
Markdown
92 lines
6.8 KiB
Markdown
# VALIDATION — fork-4 custody: live escrow ceremony + DR recovery (supervised)
|
||
|
||
**Date:** 2026-07-09 · **Class:** risky/supervised operational validation (RUNBOOK) · **Executor:** Claude
|
||
Code live, Viktor on the 🛑 secret steps (R capture, recovery). **No code, no version bump.** Closes the
|
||
offsite arc: the full enable→stage→ceremony(real R)→confirm→gated-run cycle + DR recovery of the restic
|
||
password from the **production hub blob**, decrypting the real Storage-Box repo. Non-destructive (K
|
||
read-only/byte-unchanged; live `repo_password` backed-up-then-restored; demo repo never destructively written).
|
||
|
||
---
|
||
|
||
## 1. VERDICT
|
||
|
||
**fork-4 custody is VALIDATED end-to-end.** Anchored on **Phase 3** (the atomicity gate opens *exactly* on
|
||
confirm — skip-before, run-after) and **Phase 4a** (a password recovered from the production hub blob, via
|
||
the real `identity-consume` with the ceremony's fresh R, **decrypted the real demo repo** and is byte-identical
|
||
to the live password). Live inject→restore (4b) reproduced a 45 MB db-dump byte-perfect through the real
|
||
endpoints. The offsite tier is now proven safe for real data, and the standing demo offsite is un-paused.
|
||
|
||
---
|
||
|
||
## 2. Baselines + the K gate
|
||
|
||
| Thing | Value |
|
||
|-------|-------|
|
||
| agent | v0.77.0 active on felhom-pve, caps 56/56 degraded=0 |
|
||
| controller | `:0.105.0 Up (healthy)` on guest 9201 |
|
||
| demo host / offbox | host_id `demo-felhom-01`; sub-account `u629193-sub1` @ `u629193-sub1.your-storagebox.de:23`, RepoPath `/home/felhom-demo-repo`; started **gated** (run refused: "…a kulcs letétbe helyezésére vár.") |
|
||
| **K gate** | **PASS** — `/etc/pve/priv/storage/felhom-pbs.enc` (root 0600) present; ceremony uses `--storage felhom-pbs`; `age` + `proxmox-backup-client` present |
|
||
| repo_password sha (Phase 0) | `24c44f49…` (recorded as the byte-match target; hash only) |
|
||
| demo repo (Phase 0) | 2 snapshots (immich, audiobookshelf) |
|
||
|
||
## 3. Phase 1 — enable → stage → pending ✅
|
||
`POST /backup/offbox/config` (enabled) → controller `EscrowState="pending"`; the agent **staged** the pushed
|
||
password (0600 `felhom-agent`-owned file at `StagedResticPasswordPath`, 64 bytes, sha `24c44f49…` = the repo
|
||
password byte-identical), and the value was **NOT in any agent log (0 hits)**. A `POST /backup/offbox/run`
|
||
while pending was **refused** ("…letétbe helyezésére vár.") — the gate holds pre-escrow.
|
||
|
||
## 4. Phase 2 — the escrow ceremony (SUPERVISED, real R) 🛑 ✅
|
||
Viktor ran `felhom-agent --selftest=escrow-create --storage felhom-pbs --upload` (as root — K is root-0600).
|
||
Result: **self-verify OK**, fresh **R** surfaced once (Viktor captured it; never logged/recorded), the opaque
|
||
blob(s) **uploaded to the hub**. Verified after: the **staged secret was wiped** (`StagedResticPasswordPath`
|
||
gone); **K byte-unchanged** (sha `1dfa412b…` before == after); **no R/password in the agent log** (0 hits).
|
||
The hub stored (host `demo-felhom-01`, created `2026-07-09T14:16:16Z`): identity blob 483 B + K-escrow 383 B.
|
||
|
||
## 5. Phase 3 — the atomicity proof ✅
|
||
- **Before confirm:** `POST /backup/offbox/run` → **skipped** (flash "vár") — the ceremony alone does not flip state.
|
||
- `POST /backup/offbox/confirm-escrow` → `EscrowState="escrowed"` (flash "…megerősítve — a NAS-mentés mostantól futhat.").
|
||
- **After confirm:** `POST /backup/offbox/run` → **proceeded** → `LastStatus="ok"`, **2 app(s) backed up
|
||
(immich + audiobookshelf), 2 snapshots, 43s**. The gate opens exactly on confirm — the live atomicity cycle.
|
||
|
||
## 6. Phase 4 — DR recovery from the PRODUCTION hub blob 🛑 ✅
|
||
The identity blob the hub stored in Phase 2 was fetched from the hub store (`host_escrow.identity_blob` for
|
||
`demo-felhom-01`, age-armored, 484 B) and staged on felhom-pve. *(Method note: fetched from the hub DB, i.e.
|
||
the actual production-stored blob; the fuller re-enroll DR endpoint requires operator-armed recovery mode —
|
||
not exercised here to avoid side effects on the demo host. The blob recovered IS the one the ceremony uploaded.)*
|
||
- **Recover (Viktor, R via `FELHOM_RECOVERY_CODE`):** `felhom-agent --selftest=identity-consume --blob …
|
||
--keydest recovered-bundle.json` → `selftest=identity-consume OK`; the recovered bundle (0600, never printed)
|
||
carries `restic_repo_password` = **P′**.
|
||
- **4a — decrypt proof:** `sha256(P′) == sha256(live repo_password) = 24c44f49…` (hashes only), and
|
||
`restic … snapshots` with `RESTIC_PASSWORD_FILE=P′` against the real repo **listed the 2 real snapshots**
|
||
(audiobookshelf `6adb3f96`, immich `0d035a8d`, from the Phase-3 run). The recovered-from-hub password is a
|
||
**working repo key** — the whole production chain (ceremony → hub → recover → decrypt) validated.
|
||
- **4b — live inject→restore (rollback-guarded):** backed up `repo_password`, removed it (simulate fresh),
|
||
`POST /backup/offbox/inject-password {password:P′}` → placed 0600 (sha `24c44f49…`); `POST
|
||
/backup/offbox/restore {app:immich}` → the restored `immich-postgres.sql` matched the USB source
|
||
**byte-perfect** (`d78466fc…`, 45321478 B) after the async restore completed. (First check was a poll race
|
||
on the 45 MB file — matched on completion.)
|
||
|
||
## 7. Phase 5 — end state
|
||
- **Kept escrowed:** the demo offbox is `EscrowState="escrowed"` — offsite runs resume (the demo offsite is
|
||
un-paused, a good end state). **Viktor holds the demo's R out-of-band** (existence recorded, not the value).
|
||
- Live `repo_password` restored from the rollback backup — sha `24c44f49…`, **unchanged** from Phase 0.
|
||
- All scratch shredded: felhom-pve DR dir (P′ + bundle JSON `shred`ed, blob removed), container P′ + the
|
||
offbox-restore verify dir, the 180 hub-DB snapshot + blob. **No secret (R, P′, repo password) in any
|
||
agent/controller log (0 hits).**
|
||
|
||
## 8. Observations
|
||
- `identity-consume` OK message says "identity recovered (tunnel_token + pbs_token)" — it **omits the
|
||
recovered `restic_repo_password`** (cosmetic; the field IS in the written bundle). A one-line message fix.
|
||
- `confirm-escrow` currently **trusts the operator** (pilot): it flips to escrowed with no hub-side proof the
|
||
blob actually landed. The provisioning task should replace it with a **hub-verified auto-confirm** (close
|
||
the operator-forgets/operator-lies footgun).
|
||
- Fetching the DR blob required a hub-DB read; a first-class operator "download my escrow blob" path (or the
|
||
recovery-mode re-enroll flow) would make DR self-serve.
|
||
|
||
## 9. Arc status / NEXT
|
||
The offsite tier (restic-to-Storage-Box, direct, sub-account-per-customer, password escrowed under R, atomic
|
||
enable, DR-recoverable) is now **proven safe for real customer data**. **NEXT:** the hub-provisioning +
|
||
soft-quota spec (both storage models, hub-verified auto-confirm, soft quota); the per-customer onboarding
|
||
runbook (sub-account + key + known_hosts + RepoPath + enable→ceremony→confirm); the `identity-consume`
|
||
message fix; the Tier-2 `DoubleClickRefused` flake.
|