# SPIKE — fork-4 restic-offsite password custody (escrow-under-R + controller→agent hand-off) **Date:** 2026-07-09 · **Class:** spike (crypto/mechanism validation before the fork-4 impl) · **Executor:** Claude Code live, Viktor supervising the 🛑 real-password read. **No production code shipped; all probes were throwaway, discarded.** Validates the custody design for the restic-offsite repo password: it rides the existing customer-recovery-code (R) zero-knowledge escrow (`age`-under-R), the controller pushes it to the agent for escrow-create, and enabling offsite (re-)issues a fresh R. --- ## 1. VERDICT **Custody-under-R for the restic offsite repo password is VALIDATED.** Anchored on **Probe 3 (the money shot):** a password recovered from an `age`-under-R escrow of the *real* repo password **decrypted the real standing demo repo on the Storage Box** — it listed the real immich + audiobookshelf snapshots, proving the recovered value is a *working repo key*, not merely a byte-match. The crypto round-trip is byte-perfect, a wrong R fails closed with zero plaintext, the controller→agent hand-off is additive on an already-proven transport, and the DR pre-place re-injection seam is honored. The fork-4 impl is now mechanical (blueprint §7). --- ## 2. Probe 1 — crypto round-trip + wrong-R fail-closed ✅ Throwaway `go test` in `felhom-agent/internal/escrow` (run on the build host, deleted; git clean), using the **generic** `WrapIdentity`/`UnwrapIdentity` on a throwaway struct that ADDS `restic_repo_password` — so nothing in the production `IdentityBundle` was touched: - **Round-trip:** a synthetic 256-bit-hex restic password wrapped under a generated R and unwrapped came back **byte-identical** (len 64), other bundle fields intact. (~2.5 s — real scrypt.) - **Wrong R:** `UnwrapIdentity(blob, R+"-WRONG")` **errored and emitted ZERO plaintext bytes** (fails closed at the scrypt KDF — never a plausible-but-wrong password); the right R still recovered the same blob. ## 3. Probe 2 — controller→agent hand-off shape (source review) ✅ The controller→agent local API **already** carries authenticated POST-with-JSON-body extensively — `POST /backup`, `/disks/format` (`{device,fstype}`), `/controller/swap` (`{image}`), `/disks/decommission` (`{where}`), `/disks/eject`, `/disks/assign` — all `withGuest` auth+scoped. A password **stage endpoint is additive on this proven channel**, not new transport. **Staging-hygiene shape to implement (the new part):** the agent receives the pushed password, holds it **transient** (in-memory or a 0600 scratch), feeds it to escrow-create **once**, then **wipes** it; the value is **never logged** (log the field NAME only, mirroring `escrow.AttachWGKey`'s discipline). No live throwaway endpoint was needed — the risk is staging hygiene, a design property stated here, not a transport to prove. ## 4. Probe 3 — THE MONEY SHOT: recovered password decrypts the REAL repo ✅ 🛑 Non-destructive; the live password file and repo were only read. Flow (hashes only — no value ever printed): 1. 🛑 Real password `P` piped **file-to-file** from the container's `offbox/repo_password` (64 bytes) to a 0600 scratch on the build host (exercises the guest→host data flow the real hand-off will carry). 2. Throwaway harness (real `escrow.WrapIdentity`): escrow `P` under a fresh R → `blob`; simulate DR → `UnwrapIdentity(blob, R)` = `P'`. **`sha256(P) == sha256(P') = 24c44f49…`** (len 64). 3. `P'` moved to a 0600 container scratch; ran the **exact** offbox `restic … snapshots` (the real `sftp.command`: `ssh u629193-sub1@…-p 23 …-i offbox/ssh_key -s sftp`, repo `sftp:…:/home/felhom-demo-repo`) with `RESTIC_PASSWORD_FILE=P'`: ``` 14a3bbba audiobookshelf /mnt/felhom-drives/felhom-usb/backups/primary/audiobookshelf 7ba512bf immich /mnt/felhom-drives/felhom-usb/backups/primary/immich 2 snapshots ``` **The recovered password opened the real repo.** This is the acceptance criterion. ## 5. Probe 4 — DR re-injection seam ✅ (+ a gap to close in the impl) Throwaway `go test` in `felhom-controller/internal/backup`: pre-place a known "recovered" password at `offbox/repo_password`, then call `WriteOffboxSecrets(...)` → the file was **byte-unchanged** (the `os.Stat … IsNotExist` guard means gen only runs when absent). So a DR flow can **pre-place** the recovered password and the controller uses it, never overwrites. **Gap for the impl:** the config endpoint (`POST /backup/offbox/config`) carries `ssh_key` + `known_hosts` but **NOT** the repo password (auto-gen or pre-place only). So DR-restore needs an **explicit inject step** that writes the 0600 password file *before* offbox config runs (or a small `POST /backup/offbox/inject-password`). Pre-placing is honored; there is just no endpoint for it today. ## 6. Probe 5 — hygiene / entropy ✅ Repo password is **256-bit** (`generateOffboxPassword` = 32 rand bytes → 64 hex; confirmed 64-byte file). `age -p` uses **scrypt** → a wrong R is expensive (the wrong-R probe took ~3.8 s of KDF, not a cheap brute). No password value was logged anywhere in the spike — only sha256 hashes and field names. ## 7. Impl blueprint (for the fork-4 spec) - **Agent** — add `ResticRepoPassword string \`json:"restic_repo_password,omitempty"\`` to `IdentityBundle` (rides the existing `WrapIdentityBundle`/`UnwrapIdentityBundle` age-under-R path — Probe 1 proves it carries). Add a **stage endpoint** on the local API (controller-push; `withGuest` auth; hold transient, feed escrow-create once, wipe, never log the value). Re-add the reserved `restic_repo_coord` as **`DRResticCoord`** in `dr_recipe.go` — Host/User/Port/RepoPath/host-key-fingerprint, **coordinates ONLY** (per the never-a-secret boundary + the `_NoSecrets` test), mirroring `DRPBSCoord`. - **Controller** — at offsite-enable: **push** the repo password to the agent, then trigger **escrow-create (fresh R)**. DR-restore path **pre-places** the recovered password before `WriteOffboxSecrets` (Probe 4), via the explicit inject step (Probe 5 gap). - **Orchestration** — offsite-enable → controller-push → agent escrow-create → **fresh R surfaced to the customer once** (supersedes any prior R) → opaque R-blob to the hub (reuse the existing K-blob storage). **Because R is never retained, the password MUST enter the escrow at this ceremony — no retro-fit** (same constraint as the S3 WG key). - **Hub** — stores only the opaque R-blob (zero-knowledge preserved) + the non-secret `restic_repo_coord`. - **DR recovery flow** — recover R → `UnwrapIdentityBundle` → password; read `restic_repo_coord`; operator re-provisions SFTP access (**fresh sub-account key — access is regenerable, deliberately NOT escrowed**); pre-place password → configure offbox → restore. ## 8. NOT done / boundaries - The **SFTP key + sub-account credential are ACCESS** (regenerable at DR via a fresh console sub-account + key install) → **deliberately NOT escrowed**; only the repo *password* (irreplaceable — it's the data key) and the non-secret coordinates are. - The real PBS key **K** and `proxmox-backup-client` were **not** involved — the restic password rides the generic `age`-under-R path, no K needed. - Multi-customer provisioning + soft-quota, and the hub-driven both-models config, are separate tasks. - This spike does not build any of the blueprint — it validates the mechanisms the fork-4 impl spec will wire. ## 9. Teardown state (clean) All throwaway probes deleted (agent `internal/escrow`, controller `internal/backup` — both git-clean, nothing committed). All scratch secrets shredded (P/P'/blob/R on the build host; P' in the container). **The live `offbox/repo_password` is untouched** (still 64 bytes, original timestamp — only ever read) and the demo repo is intact (2 snapshots, as listed in Probe 3). Only deliverable committed: this findings doc.