diff --git a/documentation/audits/SPIKE-restic-password-custody-2026-07-09.md b/documentation/audits/SPIKE-restic-password-custody-2026-07-09.md new file mode 100644 index 0000000..3dd6e88 --- /dev/null +++ b/documentation/audits/SPIKE-restic-password-custody-2026-07-09.md @@ -0,0 +1,104 @@ +# SPIKE — fork-4 restic-offsite password custody (escrow-under-R + controller→agent hand-off) + +**Date:** 2026-07-09 · **Class:** spike (crypto/mechanism validation before the fork-4 impl) · **Executor:** +Claude Code live, Viktor supervising the 🛑 real-password read. **No production code shipped; all probes were +throwaway, discarded.** Validates the custody design for the restic-offsite repo password: it rides the +existing customer-recovery-code (R) zero-knowledge escrow (`age`-under-R), the controller pushes it to the +agent for escrow-create, and enabling offsite (re-)issues a fresh R. + +--- + +## 1. VERDICT + +**Custody-under-R for the restic offsite repo password is VALIDATED.** Anchored on **Probe 3 (the money +shot):** a password recovered from an `age`-under-R escrow of the *real* repo password **decrypted the real +standing demo repo on the Storage Box** — it listed the real immich + audiobookshelf snapshots, proving the +recovered value is a *working repo key*, not merely a byte-match. The crypto round-trip is byte-perfect, a +wrong R fails closed with zero plaintext, the controller→agent hand-off is additive on an already-proven +transport, and the DR pre-place re-injection seam is honored. The fork-4 impl is now mechanical (blueprint §7). + +--- + +## 2. Probe 1 — crypto round-trip + wrong-R fail-closed ✅ +Throwaway `go test` in `felhom-agent/internal/escrow` (run on the build host, deleted; git clean), using the +**generic** `WrapIdentity`/`UnwrapIdentity` on a throwaway struct that ADDS `restic_repo_password` — so +nothing in the production `IdentityBundle` was touched: +- **Round-trip:** a synthetic 256-bit-hex restic password wrapped under a generated R and unwrapped came back + **byte-identical** (len 64), other bundle fields intact. (~2.5 s — real scrypt.) +- **Wrong R:** `UnwrapIdentity(blob, R+"-WRONG")` **errored and emitted ZERO plaintext bytes** (fails closed + at the scrypt KDF — never a plausible-but-wrong password); the right R still recovered the same blob. + +## 3. Probe 2 — controller→agent hand-off shape (source review) ✅ +The controller→agent local API **already** carries authenticated POST-with-JSON-body extensively — +`POST /backup`, `/disks/format` (`{device,fstype}`), `/controller/swap` (`{image}`), +`/disks/decommission` (`{where}`), `/disks/eject`, `/disks/assign` — all `withGuest` auth+scoped. A password +**stage endpoint is additive on this proven channel**, not new transport. +**Staging-hygiene shape to implement (the new part):** the agent receives the pushed password, holds it +**transient** (in-memory or a 0600 scratch), feeds it to escrow-create **once**, then **wipes** it; the value +is **never logged** (log the field NAME only, mirroring `escrow.AttachWGKey`'s discipline). No live throwaway +endpoint was needed — the risk is staging hygiene, a design property stated here, not a transport to prove. + +## 4. Probe 3 — THE MONEY SHOT: recovered password decrypts the REAL repo ✅ 🛑 +Non-destructive; the live password file and repo were only read. Flow (hashes only — no value ever printed): +1. 🛑 Real password `P` piped **file-to-file** from the container's `offbox/repo_password` (64 bytes) to a + 0600 scratch on the build host (exercises the guest→host data flow the real hand-off will carry). +2. Throwaway harness (real `escrow.WrapIdentity`): escrow `P` under a fresh R → `blob`; simulate DR → + `UnwrapIdentity(blob, R)` = `P'`. **`sha256(P) == sha256(P') = 24c44f49…`** (len 64). +3. `P'` moved to a 0600 container scratch; ran the **exact** offbox `restic … snapshots` (the real + `sftp.command`: `ssh u629193-sub1@…-p 23 …-i offbox/ssh_key -s sftp`, repo + `sftp:…:/home/felhom-demo-repo`) with `RESTIC_PASSWORD_FILE=P'`: + ``` + 14a3bbba audiobookshelf /mnt/felhom-drives/felhom-usb/backups/primary/audiobookshelf + 7ba512bf immich /mnt/felhom-drives/felhom-usb/backups/primary/immich + 2 snapshots + ``` + **The recovered password opened the real repo.** This is the acceptance criterion. + +## 5. Probe 4 — DR re-injection seam ✅ (+ a gap to close in the impl) +Throwaway `go test` in `felhom-controller/internal/backup`: pre-place a known "recovered" password at +`offbox/repo_password`, then call `WriteOffboxSecrets(...)` → the file was **byte-unchanged** (the +`os.Stat … IsNotExist` guard means gen only runs when absent). So a DR flow can **pre-place** the recovered +password and the controller uses it, never overwrites. +**Gap for the impl:** the config endpoint (`POST /backup/offbox/config`) carries `ssh_key` + `known_hosts` +but **NOT** the repo password (auto-gen or pre-place only). So DR-restore needs an **explicit inject step** +that writes the 0600 password file *before* offbox config runs (or a small `POST /backup/offbox/inject-password`). +Pre-placing is honored; there is just no endpoint for it today. + +## 6. Probe 5 — hygiene / entropy ✅ +Repo password is **256-bit** (`generateOffboxPassword` = 32 rand bytes → 64 hex; confirmed 64-byte file). +`age -p` uses **scrypt** → a wrong R is expensive (the wrong-R probe took ~3.8 s of KDF, not a cheap brute). +No password value was logged anywhere in the spike — only sha256 hashes and field names. + +## 7. Impl blueprint (for the fork-4 spec) +- **Agent** — add `ResticRepoPassword string \`json:"restic_repo_password,omitempty"\`` to `IdentityBundle` + (rides the existing `WrapIdentityBundle`/`UnwrapIdentityBundle` age-under-R path — Probe 1 proves it + carries). Add a **stage endpoint** on the local API (controller-push; `withGuest` auth; hold transient, + feed escrow-create once, wipe, never log the value). Re-add the reserved `restic_repo_coord` as + **`DRResticCoord`** in `dr_recipe.go` — Host/User/Port/RepoPath/host-key-fingerprint, **coordinates ONLY** + (per the never-a-secret boundary + the `_NoSecrets` test), mirroring `DRPBSCoord`. +- **Controller** — at offsite-enable: **push** the repo password to the agent, then trigger + **escrow-create (fresh R)**. DR-restore path **pre-places** the recovered password before + `WriteOffboxSecrets` (Probe 4), via the explicit inject step (Probe 5 gap). +- **Orchestration** — offsite-enable → controller-push → agent escrow-create → **fresh R surfaced to the + customer once** (supersedes any prior R) → opaque R-blob to the hub (reuse the existing K-blob storage). + **Because R is never retained, the password MUST enter the escrow at this ceremony — no retro-fit** (same + constraint as the S3 WG key). +- **Hub** — stores only the opaque R-blob (zero-knowledge preserved) + the non-secret `restic_repo_coord`. +- **DR recovery flow** — recover R → `UnwrapIdentityBundle` → password; read `restic_repo_coord`; operator + re-provisions SFTP access (**fresh sub-account key — access is regenerable, deliberately NOT escrowed**); + pre-place password → configure offbox → restore. + +## 8. NOT done / boundaries +- The **SFTP key + sub-account credential are ACCESS** (regenerable at DR via a fresh console sub-account + + key install) → **deliberately NOT escrowed**; only the repo *password* (irreplaceable — it's the data key) + and the non-secret coordinates are. +- The real PBS key **K** and `proxmox-backup-client` were **not** involved — the restic password rides the + generic `age`-under-R path, no K needed. +- Multi-customer provisioning + soft-quota, and the hub-driven both-models config, are separate tasks. +- This spike does not build any of the blueprint — it validates the mechanisms the fork-4 impl spec will wire. + +## 9. Teardown state (clean) +All throwaway probes deleted (agent `internal/escrow`, controller `internal/backup` — both git-clean, nothing +committed). All scratch secrets shredded (P/P'/blob/R on the build host; P' in the container). **The live +`offbox/repo_password` is untouched** (still 64 bytes, original timestamp — only ever read) and the demo repo +is intact (2 snapshots, as listed in Probe 3). Only deliverable committed: this findings doc.