# SPIKE — fork-4 restic-offsite password custody (escrow-under-R + controller→agent hand-off)

**Date:** 2026-07-09 · **Class:** spike (crypto/mechanism validation before the fork-4 impl) · **Executor:**
Claude Code live, Viktor supervising the 🛑 real-password read. **No production code shipped; all probes were
throwaway, discarded.** Validates the custody design for the restic-offsite repo password: it rides the
existing customer-recovery-code (R) zero-knowledge escrow (`age`-under-R), the controller pushes it to the
agent for escrow-create, and enabling offsite (re-)issues a fresh R.

---

## 1. VERDICT

**Custody-under-R for the restic offsite repo password is VALIDATED.** Anchored on **Probe 3 (the money
shot):** a password recovered from an `age`-under-R escrow of the *real* repo password **decrypted the real
standing demo repo on the Storage Box** — it listed the real immich + audiobookshelf snapshots, proving the
recovered value is a *working repo key*, not merely a byte-match. The crypto round-trip is byte-perfect, a
wrong R fails closed with zero plaintext, the controller→agent hand-off is additive on an already-proven
transport, and the DR pre-place re-injection seam is honored. The fork-4 impl is now mechanical (blueprint §7).

---

## 2. Probe 1 — crypto round-trip + wrong-R fail-closed ✅
Throwaway `go test` in `felhom-agent/internal/escrow` (run on the build host, deleted; git clean), using the
**generic** `WrapIdentity`/`UnwrapIdentity` on a throwaway struct that ADDS `restic_repo_password` — so
nothing in the production `IdentityBundle` was touched:
- **Round-trip:** a synthetic 256-bit-hex restic password wrapped under a generated R and unwrapped came back
  **byte-identical** (len 64), other bundle fields intact. (~2.5 s — real scrypt.)
- **Wrong R:** `UnwrapIdentity(blob, R+"-WRONG")` **errored and emitted ZERO plaintext bytes** (fails closed
  at the scrypt KDF — never a plausible-but-wrong password); the right R still recovered the same blob.

## 3. Probe 2 — controller→agent hand-off shape (source review) ✅
The controller→agent local API **already** carries authenticated POST-with-JSON-body extensively —
`POST /backup`, `/disks/format` (`{device,fstype}`), `/controller/swap` (`{image}`),
`/disks/decommission` (`{where}`), `/disks/eject`, `/disks/assign` — all `withGuest` auth+scoped. A password
**stage endpoint is additive on this proven channel**, not new transport.
**Staging-hygiene shape to implement (the new part):** the agent receives the pushed password, holds it
**transient** (in-memory or a 0600 scratch), feeds it to escrow-create **once**, then **wipes** it; the value
is **never logged** (log the field NAME only, mirroring `escrow.AttachWGKey`'s discipline). No live throwaway
endpoint was needed — the risk is staging hygiene, a design property stated here, not a transport to prove.

## 4. Probe 3 — THE MONEY SHOT: recovered password decrypts the REAL repo ✅ 🛑
Non-destructive; the live password file and repo were only read. Flow (hashes only — no value ever printed):
1. 🛑 Real password `P` piped **file-to-file** from the container's `offbox/repo_password` (64 bytes) to a
   0600 scratch on the build host (exercises the guest→host data flow the real hand-off will carry).
2. Throwaway harness (real `escrow.WrapIdentity`): escrow `P` under a fresh R → `blob`; simulate DR →
   `UnwrapIdentity(blob, R)` = `P'`. **`sha256(P) == sha256(P') = 24c44f49…`** (len 64).
3. `P'` moved to a 0600 container scratch; ran the **exact** offbox `restic … snapshots` (the real
   `sftp.command`: `ssh u629193-sub1@…-p 23 …-i offbox/ssh_key -s sftp`, repo
   `sftp:…:/home/felhom-demo-repo`) with `RESTIC_PASSWORD_FILE=P'`:
   ```
   14a3bbba  audiobookshelf  /mnt/felhom-drives/felhom-usb/backups/primary/audiobookshelf
   7ba512bf  immich          /mnt/felhom-drives/felhom-usb/backups/primary/immich
   2 snapshots
   ```
   **The recovered password opened the real repo.** This is the acceptance criterion.

## 5. Probe 4 — DR re-injection seam ✅ (+ a gap to close in the impl)
Throwaway `go test` in `felhom-controller/internal/backup`: pre-place a known "recovered" password at
`offbox/repo_password`, then call `WriteOffboxSecrets(...)` → the file was **byte-unchanged** (the
`os.Stat … IsNotExist` guard means gen only runs when absent). So a DR flow can **pre-place** the recovered
password and the controller uses it, never overwrites.
**Gap for the impl:** the config endpoint (`POST /backup/offbox/config`) carries `ssh_key` + `known_hosts`
but **NOT** the repo password (auto-gen or pre-place only). So DR-restore needs an **explicit inject step**
that writes the 0600 password file *before* offbox config runs (or a small `POST /backup/offbox/inject-password`).
Pre-placing is honored; there is just no endpoint for it today.

## 6. Probe 5 — hygiene / entropy ✅
Repo password is **256-bit** (`generateOffboxPassword` = 32 rand bytes → 64 hex; confirmed 64-byte file).
`age -p` uses **scrypt** → a wrong R is expensive (the wrong-R probe took ~3.8 s of KDF, not a cheap brute).
No password value was logged anywhere in the spike — only sha256 hashes and field names.

## 7. Impl blueprint (for the fork-4 spec)
- **Agent** — add `ResticRepoPassword string \`json:"restic_repo_password,omitempty"\`` to `IdentityBundle`
  (rides the existing `WrapIdentityBundle`/`UnwrapIdentityBundle` age-under-R path — Probe 1 proves it
  carries). Add a **stage endpoint** on the local API (controller-push; `withGuest` auth; hold transient,
  feed escrow-create once, wipe, never log the value). Re-add the reserved `restic_repo_coord` as
  **`DRResticCoord`** in `dr_recipe.go` — Host/User/Port/RepoPath/host-key-fingerprint, **coordinates ONLY**
  (per the never-a-secret boundary + the `_NoSecrets` test), mirroring `DRPBSCoord`.
- **Controller** — at offsite-enable: **push** the repo password to the agent, then trigger
  **escrow-create (fresh R)**. DR-restore path **pre-places** the recovered password before
  `WriteOffboxSecrets` (Probe 4), via the explicit inject step (Probe 5 gap).
- **Orchestration** — offsite-enable → controller-push → agent escrow-create → **fresh R surfaced to the
  customer once** (supersedes any prior R) → opaque R-blob to the hub (reuse the existing K-blob storage).
  **Because R is never retained, the password MUST enter the escrow at this ceremony — no retro-fit** (same
  constraint as the S3 WG key).
- **Hub** — stores only the opaque R-blob (zero-knowledge preserved) + the non-secret `restic_repo_coord`.
- **DR recovery flow** — recover R → `UnwrapIdentityBundle` → password; read `restic_repo_coord`; operator
  re-provisions SFTP access (**fresh sub-account key — access is regenerable, deliberately NOT escrowed**);
  pre-place password → configure offbox → restore.

## 8. NOT done / boundaries
- The **SFTP key + sub-account credential are ACCESS** (regenerable at DR via a fresh console sub-account +
  key install) → **deliberately NOT escrowed**; only the repo *password* (irreplaceable — it's the data key)
  and the non-secret coordinates are.
- The real PBS key **K** and `proxmox-backup-client` were **not** involved — the restic password rides the
  generic `age`-under-R path, no K needed.
- Multi-customer provisioning + soft-quota, and the hub-driven both-models config, are separate tasks.
- This spike does not build any of the blueprint — it validates the mechanisms the fork-4 impl spec will wire.

## 9. Teardown state (clean)
All throwaway probes deleted (agent `internal/escrow`, controller `internal/backup` — both git-clean, nothing
committed). All scratch secrets shredded (P/P'/blob/R on the build host; P' in the container). **The live
`offbox/repo_password` is untouched** (still 64 bytes, original timestamp — only ever read) and the demo repo
is intact (2 snapshots, as listed in Probe 3). Only deliverable committed: this findings doc.
