felhom.eu/documentation/audits/SPIKE-dr-recipe-2026-06-16.md

# SPIKE — Phase 2 DR recipe: characterize existing DR, define the secret-free recipe, lock the boundary (+ storage diagnosis)

**Date:** 2026-06-16
**Type:** Report-only spike. **No code changes, no version bump.** Every claim cited against pushed source at `file:line`. Part 0 is a **live** diagnosis on guest 9201 / `felhom-pve` (one non-destructive fix applied — restarting already-present apps — and noted).
**Grounding:** `SPIKE-infra-backup-2026-06-15.md` line 24 — the revival is "a slim, **secret-free** DR recipe (customer → apps → drive/durable-id intents → guest sizing) that complements PBS's ingredients. Secrets are never in it; they are recovered from the PBS whole-CT snapshot, never regenerated."

---

## One-screen summary

- **PART 0 (storage diagnosis):** the flash-backed apps did **not** crash and felhom-flash did **not** drop. **Root cause: an operator `pct reboot` (PVE `vzreboot`) of guest 9201 at 10:26 UTC + a boot-ordering race.** On guest boot, dockerd auto-starts the `unless-stopped` flash apps **~18 s before** the agent re-binds felhom-flash into the guest; docker fails to create the not-yet-present bind source (`mkdir /mnt/felhom-drives/felhom-flash/userdata: permission denied`), and because that is a **create-time** mount failure (`RestartCount=0`) the restart policy never retries — the apps stay `Exited` forever even after the bind lands. The drive is healthy, mounted, bound (host + guest), data intact; no USB drop, no durable-id failure, the drive-gate was uninvolved. **The v0.70.0 self-restart is RULED OUT** (it restarts the felhom-controller *Docker container* via `os.Exit(0)`, never the LXC guest, and ran 11:04–11:17 UTC — ~38 min *after* the exits). Fix applied: started the 7 apps via the controller (drive present now) — all back `Up`. **Architectural gap flagged:** the intermediary mount is C1-immune (guest boots) but does not prevent this app-start race.
- **PART 1 (existing DR):** **DONE** — PBS whole-CT escrow + consume (`internal/escrow/*`, fingerprint-gated, real-data drilled), identity-escrow (`{tunnel_token, pbs_token}` age-wrapped under the same recovery code R), tunnel re-establishment, and the hub's **zero-knowledge** recovery-mode/re-enroll orchestration (`hub/internal/api/dr.go`). **PENDING** — the agent-side recovery orchestration that *consumes* the directive (the agent carries `restore_directive` forward-compat but explicitly does **not** act on it — `internal/desired/syncer.go:92`), and the secret-free recipe itself.
- **PART 2 (recipe):** the gap is the **non-secret scaffolding PBS does not capture** — guest sizing, drive inventory (durable-id → role → mount → enroll/decommission intent), PVE storage defs, app inventory + per-app storage bindings. The agent **already holds every fact** (`StorageTarget{durable_id, role, mount_path, total/used/avail}` at `report.go:86`; guest specs; PBS coords) — it can emit the recipe. The hub→agent wire **already reserves** `storage_manifest`/`backup_policy`/`pbs_namespace` placeholders (`report.go:271-273`). **Boundary:** every recipe field is an identifier/intent/size — no key, password, token, or hash. Secrets stay in PBS + escrow.
- **PART 3:** fork list + recommendation: emit the recipe **agent-side as an additive host-report section** (it owns the storage truth), store it **plaintext** on the hub next to the host-report, version the wire shape, and **defer** the re-enrollment auth (recovery-mode consumption) to the existing slice-10D track — the recipe complements escrow+PBS, it does not touch them.

---

# PART 0 — Storage diagnosis: felhom-flash plugged in, yet flash-backed apps exited

## What I observed (live, 12:00–12:10 UTC)

Seven flash-backed apps `Exited`: audiobookshelf, calibre-web, immich-server, jellyfin, komga, radarr, romm. Exit codes **143** (audiobookshelf/immich-server/komga) and **128** (calibre-web/jellyfin/radarr/romm). felhom-flash physically connected.

**Timezone note (this was load-bearing):** docker reports UTC (`…Z`); the host journal is `+02:00` (CEST). All times below normalized to **UTC** unless marked "local".

## End-to-end walk (not "is the device present")

1. **The drive is healthy and fully bound — host side.** `lsblk`: felhom-flash = `/dev/sdb1`, ext4, `LABEL=…`, `UUID=81a26531-62d8-408d-812f-a178b1d35310`, mounted at `/mnt/felhom-flash`. The intermediary chain is intact:
   - stable parent: `/dev/mapper/pve-root on /mnt/felhom-drives` (on rootfs, always present),
   - per-drive: `/dev/sdb1 on /mnt/felhom-drives/felhom-flash`.
2. **Bound through to the guest — guest side.** Inside 9201, `/proc/mounts` shows `/dev/sdb1 /mnt/felhom-drives/felhom-flash ext4 rw,relatime`, and `ls /mnt/felhom-drives/felhom-flash` → `appdata backups media userdata`. **The flash data is reachable in the guest right now.**
3. **The agent never lost it.** The reconcile loop logs every ~20 s, continuously: `reconcile: enrolled drive bound under shared parent (live, no reboot) … where=/mnt/felhom-flash guest_path=/mnt/felhom-drives/felhom-flash durable_id=uuid:81a26531-…` — including across the incident window. (`internal/reconcile` + `internal/localapi` guest-bind; durable-id matched.)
4. **No kernel-level USB event.** `dmesg -T` and `journalctl -k --since 10:20 --until 10:35` (UTC window) show **zero** USB disconnect / re-enumeration / `sd`/`ext4 I/O error` lines. **The task's "transiently dropped + re-enumerated under a new node" hypothesis is NOT supported** — the kernel never saw the device leave, and the agent re-bound it by the same `uuid:81a26531` throughout. Durable-id tracking did **not** fail.

## What actually happened (root cause, with evidence)

5. **Guest 9201 was rebooted by the operator at 10:26 UTC.** `uptime -s` = boot 10:26 (12:26 local); `/proc/uptime` ≈ 1.7 h. Host journal at 12:26:08 local: `pvedaemon[…]: <root@pam> end task UPID:demo-felhom:…:vzreboot:9201:root@pam: OK`. This is a **PVE-level `pct reboot`** of the LXC guest, operator-initiated.
6. **The "exits" are the reboot shutdown, not a fault.** romm's own log: `[2026-06-16 12:26:02] Stopping nginx … Stopping gunicorn … Worker … was sent SIGTERM!`; jellyfin: `[12:25:59] Disposing CoreAppHost`. Exit 143 = 128+SIGTERM; 128 = the image's PID1 exit on SIGTERM. The FinishedAt timestamps (10:25:58–10:26:02 UTC) are the guest-shutdown moment.
7. **All seven are `restart: unless-stopped`** — same policy as the DB/redis containers (`romm-db`, `immich-postgres`, …), which **did** come back (`Up ~1.7 h`). So policy is not the differentiator.
8. **THE SMOKING GUN — a create-time bind-mount failure at boot:**
   ```
   romm     State.Error = "error while creating mount source path
     '/mnt/felhom-drives/felhom-flash/userdata/roms':
      mkdir /mnt/felhom-drives/felhom-flash/userdata: permission denied"   RestartCount=0
   jellyfin State.Error = "...mkdir /mnt/felhom-drives/felhom-flash/userdata: permission denied"  RestartCount=0
   ```
   On boot, dockerd tries to restart the flash apps. Their binds point under `/mnt/felhom-drives/felhom-flash/…`, but the **real felhom-flash is not bound into the guest yet** — the agent re-binds it ~18 s after boot (host journal: `guest-attach: drive bound under shared parent (normalized to one bind, live) … where=/mnt/felhom-flash … 12:26:26` local = 10:26:26 UTC, vs boot 10:26:08). At that instant `/mnt/felhom-drives/felhom-flash` is the **empty, `nobody`-owned** stable-parent placeholder (the parent is `/dev/mapper/pve-root`, `drwxr-xr-x nobody nogroup` under the unprivileged-LXC idmap), so the guest-root docker daemon's attempt to `mkdir` the missing bind source is **permission-denied**.
9. **`RestartCount=0` → the restart policy never retries.** A *create-time* mount failure is not an exit-after-run, so `unless-stopped` does not apply — the container is left `Exited` permanently even though the bind lands 18 s later. The DB containers have no such bind (named volumes on rootfs) → they start immediately and stay up. That is the whole asymmetry.

## Ruling out the v0.70.0 self-restart (explicitly, as requested)

- The self-restart shipped this session is `os.Exit(0)` of the **felhom-controller Docker container** (`controller/internal/api/selfrestart.go`); the container is `restart: unless-stopped`, so Docker restarts *that one container*. It **never** issues a guest reboot.
- The incident was a **PVE `vzreboot` of the LXC guest** (host journal, step 5) — a different layer entirely.
- Timing: the exits were 10:25:58–10:26:02 UTC; my self-restart testing ran 11:04–11:17 UTC (controller `StartedAt` 11:17:51 UTC, "Up 46 min" at 12:04). The exits **precede** my testing by ~38 min. (The task's cited "~11:54–11:57" window matches neither the exits nor any controller restart I made; the controller's only live restart is 11:17:51 UTC.) **Not implicated.**

## Fix applied (non-destructive, and it validates the diagnosis)

The drive is present and the data intact, so the only thing wrong is that the apps never started. I started them through the **controller's own pipeline** (`POST /api/stacks/{name}/start`, the same path the UI uses) — romm first as a canary (`Up`, accessing `/romm/library` on flash), then the rest. Result: `audiobookshelf, calibre-web, jellyfin, radarr` healthy; `immich-server, komga` health-starting (normal); romm up. **That they start cleanly now confirms the root cause was the boot-ordering race, not any drive fault.** No remount/rebind was needed (the mount was already correct); no destructive action taken.

## Architectural gap (flag — out of this spike's build scope, worth a follow-up)

The intermediary-mount model (`SPIKE-intermediary-mount-2026-06-15.md`) is **C1-immune** (the guest boots clean because the bind *source* is the always-present stable parent). But it does **not** prevent an **app-start race**: the per-drive bind is (re)asserted by the agent's reconcile tick **seconds after** the guest's dockerd has already tried (and permanently failed, `RestartCount=0`) to auto-start drive-backed apps. Any operator `pct reboot`, host reboot, or guest auto-restart reproduces this. Candidate fixes (for a future slice, not here):
- **(A) order the bind before docker:** have the agent assert guest binds as part of guest-start (or gate the in-guest `docker.service` start until `/mnt/felhom-drives/<drive>` is populated), so the source exists before dockerd's auto-start.
- **(B) controller startup reconcile:** after confirming a drive bind is live, the controller starts any drive-backed app that is `Exited` with a create-time mount error (a targeted, idempotent "drive-present → start gate-eligible apps" pass). *Note: today's drive-gate stops apps when a drive goes missing but does not appear to re-start apps that failed to create at boot — that asymmetry is the bug surface.*
- **(Not C)** putting the drive back as a `pct mpN` would make it present at boot but reintroduces C1 (drive absent at boot → guest bricks) — the very thing the intermediary model removed.

---

# PART 1 — The existing DR mechanism (cited)

## 1A. Escrow + recovery — what is escrowed, how it's wrapped, what it unlocks

**Two secrets, one recovery code `R`.** `R` = 10 EFF-large-wordlist words (`internal/escrow/wordlist.go:47`, `RecoveryCodeWords=10`, ~129.2 bits), hyphen-joined, surfaced to the customer once, never logged or hub-stored.

- **K-escrow (the PBS encryption key).** The live PBS client key `K` (e.g. `/etc/pve/priv/storage/felhom-pbs.enc`) is copied and re-keyed to `kdf=scrypt` under `R` via PBS-native `proxmox-backup-client key change-passphrase` (`internal/escrow/escrow.go` `Wrap()` ~143). Result: an opaque blob PBS itself cannot open without `R`. `CreateResult.Blob` (`escrow.go:49`).
- **Identity-escrow (10D.1).** `IdentityBundle{TunnelToken, PBSToken}` (`internal/escrow/identity.go:22-27`) — "the secrets a re-enrolling box needs to come back 'as host X'" — wrapped under the **same `R`** via `age -p` (scrypt + ChaCha20-Poly1305) → `CreateResult.IdentityBlob` (`escrow.go:56`; `identity.go` `WrapIdentity()` ~29). Additive: the K-escrow path is unchanged when nil (`escrow.go:43`).
- **Self-verify before shipping:** Create unwraps a copy with `R` and checks the recovered fingerprint matches, "Never ship a blob we can't recover" (`escrow.go` ~96).

**The hub is zero-knowledge.** `escrowUploadRequest{BlobB64, KeyFingerprint, Posture, IdentityBlobB64, DirectiveJSON}` (`hub/internal/api/handler.go:619-631`); the handler "Store the OPAQUE bytes. No decrypt path exists — the hub cannot open this" (`handler.go:678` area), persisted via `SaveHostEscrow` / `SaveHostDRBundle` (`hub/internal/store/store.go`, `GetHostDRBundle` returns opaque `KEscrowBlob`/`IdentityBlob`/`DirectiveJSON`). `dr.go:14-18`: "the hub ORCHESTRATES recovery … but holds **no usable secret** … the escrow blobs it serves are opaque (need R, which the hub never has)."

**Consume + fingerprint gate (the unlock path).** `internal/escrow/consume.go` — four inputs: `(blob, R, expectedFingerprint, keyDest)` (`consume.go:13-32`). Flow: (1) `Unwrap` with `R` (`key change-passphrase --kdf none`); a wrong `R` "fails closed at the scrypt KDF: nonzero exit, no key emitted" (`consume.go:63`). (2) **Fingerprint gate** — recompute the recovered key's fingerprint and compare to the hub-served `expectedFingerprint`; mismatch → fail fast, no install (`consume.go:69-79`). (3) Atomic install at `keyDest` 0600 (`consume.go:81`). Tested: right-R+right-FP installs; wrong-R no install; FP-mismatch no install (`internal/escrow/consume_test.go`).

**What `R` ultimately unlocks → how it decrypts the snapshot.** `R` → unwrap K-escrow → plaintext **`K` (PBS client encryption key)** → installed at `keyDest` (`$XDG_CONFIG_HOME/proxmox-backup/encryption-key.json` or `--keyfile`). PBS restore reads `K` from there and decrypts the client-side-encrypted snapshot chunks. `R` is the only out-of-band secret; it also unlocks the identity bundle for tunnel + PBS re-auth. (See 1B for the encryption side.)

## 1B. What the PBS whole-CT snapshot contains vs not

- **CONTAINS:** the guest **rootfs + CT config** via vzdump → PBS. "A Docker NAMED volume lives in the LXC rootfs (`/var/lib/docker/volumes/…/_data`) and is ALWAYS captured by vzdump" (`internal/proxmox/doc.go:52`). So the in-guest `app.yaml` (encrypted), the controller's encryption key, DB dumps, configs, and named-volume app data are all in the snapshot. Backup is `CrashConsistent: true` (`internal/backup/runner.go:83`).
- **DOES NOT contain — the bulk-volume gap:** external user-data drives (`felhom-usb`, `felhom-flash`) are bind-mounted `mpN` with `backup=0`/unset, which vzdump **excludes**. `uncoveredMountpoints` lists them (`internal/backup/runner.go:91-98, 228-252`: "covered ONLY when it carries an explicit `backup=1`"). These are restic cross-drive territory, **not** in the PBS whole-CT snapshot — and notably, **`felhom-flash`'s `userdata/media`, `appdata/romm/resources`, etc. (Part 0) are NOT in PBS.** Surfaced as `Backup.UncoveredVolumes` (`internal/hub/report.go:178-194`).
- **Encrypted?** Yes, client-side, by `K`. `PBSSnapshot.Encrypted` is derived from any data file with `crypt-mode == "encrypt"` (`internal/pbs/report.go:44-53`, `client.go:94`). The PBS server is zero-knowledge: "the PBS server has no client key" (`internal/pbs/doc.go:20`). Verify is key-free / ciphertext-level (`internal/pbs/client.go:60`).
- **Host-report fields:** `Backup{TargetID, VMID, Archive, Mode, CrashConsistent, SizeBytes, Success, Error, StartedAt, DurationSeconds, UncoveredVolumes}` (`report.go:182-194`); `PBSSnapshot{Namespace, BackupType, BackupID, BackupTime, SizeBytes, Owner, Protected, Encrypted, VerifyState, VerifyUPID}` (`report.go:223-234`); `VerifyState` ∈ `ok|failed|none` is "the load-bearing field" (`report.go:219`).

## 1C. The capstone drill + DONE vs PENDING

Drilled in `documentation/tests/slice10-escrow-consumption-spike-findings.md` and `slice10d-identity-restore-spike-findings.md`:
- **S1 identity round-trip (DONE):** `{tunnel_token, pbs_token}` recovered byte-identical on a fresh box from blob + `R`; wrong `R` fails closed.
- **S2 tunnel re-establishment (DONE):** the new connector with the recovered tunnel token re-registers; Cloudflare routes to it (no DNS change). **Caveat flagged in the drill:** the old connector/token stay valid → 10D must rotate them or delete the stale connector, and the hub's CF token is only zone-scoped WAF-Edit, not account-scoped Tunnel:Edit — a production blocker.
- **S3 PBS whole-CT restore (DONE):** real encrypted `spike-lxc` (~2.5 GB, `crypt-mode=encrypt`) restored on a keyless box using only the recovered `K`, fingerprint-gated.

**Manual / operator-in-the-loop steps:** customer provides `R`; operator arms recovery-mode on the hub (the out-of-band "the old box is truly lost" gate); operator performs the external credential rotation (tunnel/PBS); operator signs a destructive restore-overwrite.

**Hub side — DONE.** `hub/internal/api/dr.go:34-183`: `handleSetRecoveryMode`/`handleClearRecoveryMode` (TTL-bounded, auto-expires), `handleReEnroll` (revokes old host key, mints new, serves opaque blobs + directive), `handleGetRestoreDirective` (gated on recovery mode). `store.go`: `InRecoveryMode`, `SetRecoveryMode`/`ClearRecoveryMode`, `RotateHostAPIKey`, `GetHostDRBundle`. Desired-state serving (`SetHostDesired`, `GET /hosts/{id}/desired-state`) is live (slice 10A).

**Agent side — PENDING (the gap).** The agent **carries but does not consume** the recovery path: `internal/desired/syncer.go:92-94` — "`restore_directive` present (consumed in slice 10D — ignored in 10A)"; the wire struct `WireDesiredState.RestoreDirective` is "forward-compat … 10A carries it through to the cache but does NOT consume it" (`internal/hub/report.go:266, 289-296`). There is **no** agent code that detects recovery mode, fetches re-enroll blobs, unwraps with `R`, or executes a restore. `internal/escrow/consume.go:17`: "The DR orchestration around it (re-enroll in restore mode, source the directive …)" is the missing wrapper.

**The current end-to-end DR gap, precisely:** the crypto and real-data recovery are proven and the hub can serve the opaque blobs + directive, but **(a)** the agent has no recovery-orchestration executor, **(b)** the out-of-band `R` handshake (how the fresh box obtains `R`) is unspecified, **(c)** external credential rotation (CF tunnel + PBS token) is manual/blocked on token scope, and **(d)** there is **no secret-free recipe** telling the operator what host scaffolding to rebuild before the PBS bytes can land. This spike specs **(d)**.

---

# PART 2 — The secret-free DR recipe (the gap it fills)

## 2A. THE GAP — what PBS does *not* capture (the scaffolding the operator must rebuild)

To put the PBS bytes back, the operator must first **rebuild the host + guest + storage scaffolding on new hardware**. None of this is in the PBS whole-CT snapshot (which is *inside-the-guest* bytes) or in the escrow (which is *secrets*):

1. **Guest sizing** — cores / memory_bytes / disk_bytes to recreate the LXC at the right size (matches `GuestSpec` already on the wire, `desired-state.golden.json:8`).
2. **Drive inventory** — for each user-data drive: **durable-id** (`uuid:…`, wipe-binding `byid:<wwn>`/`byuuid:`), **role** (primary | bulk-data | vzdump-target | pbs-offsite), **mount path** (`/mnt/<name>` → intermediary `/mnt/felhom-drives/<name>`), capacity, and **enroll vs decommissioned intent**. This is exactly what `StorageTarget` already reports (2B).
3. **PVE storage definitions** — which Proxmox storages exist (id, type ∈ `local-dir|lvmthin|usb|nfs|cifs|pbs|local`, content), so the new host's `/etc/pve/storage.cfg` can be rebuilt (`StorageTarget.{Name,Type,Content}`).
4. **App inventory + per-app storage bindings** — which apps are deployed and which drive/path each binds (e.g. romm.library → `felhom-flash:userdata/roms`) — so apps come up against the right (restic-restored) bulk data. The **controller** owns this (catalog refs + non-secret deploy fields + storage-role intents).
5. **PBS coordinates** — datastore/namespace/snapshot id to restore from (`PBSSnapshot.{Namespace,BackupID}`, the desired-state `pbs_namespace`).

**Recipe contents (proposed, all non-secret):**
```
recipe_version: 1
customer_id, domain, tier
guest:        { vmid, cores, memory_bytes, disk_bytes }
pbs:          { repo_id, namespace, latest_snapshot_id }       # coordinates, NOT the key
drives:       [ { durable_id, wipe_durable_id, role, mount_path, fs_type,
                  total_bytes, intent: enrolled|decommissioned, userdata_layout } ]
pve_storage:  [ { name, type, content } ]                       # to rebuild storage.cfg
apps:         [ { name, catalog_ref, enabled,
                  storage_bindings: [ { container_path, drive_durable_id, subpath } ],
                  non_secret_deploy_fields: { ... } } ]         # NO env secrets
```
The recipe is the **re-provision plan**; PBS is the **ingredients**; escrow holds the **keys**. Three jobs, three stores.

## 2B. SOURCE / FLOW — reuse, don't duplicate

- **The agent already holds the storage/guest facts.** `StorageTarget` (`internal/hub/report.go:86-121`) carries `Name, Type, DurableID, State, Reachable, Total/Used/AvailBytes, Content, MountPath, BackingDevice, ClassHint, Role`. Guest specs and PBS snapshots are already in the host-report (`Backups[]`, `PBSSnapshots[]`, guest list). **The agent can emit the storage+guest+PBS half of the recipe from data it already collects** — additively, as a new host-report section (or a dedicated push), no new privileged reads.
- **The controller owns the app half** — app inventory, catalog refs, and per-app storage bindings (it generates the compose and knows the drive bindings; Part 0 showed romm's binds resolve to `felhom-flash/userdata/roms`). It already pushes a hub report; the app recipe is an additive, secret-free section there.
- **Where the hub stores it (plaintext-safe, non-secret).** Next to the host-report (e.g. a `dr_recipe` column/table keyed by host/customer + `recipe_version`), **not** encrypted (it has no secrets) and **separate** from `host_escrow` (which stays opaque). This is the clean inverse of the retired infra-backup: same "re-provision plan" goal, zero secrets, so plaintext-at-rest is correct rather than a violation.
- **How the operator consumes it during DR.** The hub→agent wire **already reserves the landing slot**: `WireDesiredState.StorageManifest` / `BackupPolicy` / `PBSNamespace` are forward-compat opaque placeholders "kept opaque so the wire is stable as those land" (`internal/hub/report.go:266-273`). DR flow: operator reads the stored recipe → authors the new host's **desired-state** (guest specs + `storage_manifest` from the recipe) → the agent provisions the guest at the right size and re-enrolls drives by durable-id with their roles → **then** escrow recovers `K`/identity and PBS restores the rootfs bytes (and restic restores the bulk drives) into the rebuilt scaffolding. The recipe feeds provisioning; escrow+PBS feed the data. No overlap.

## 2C. BOUNDARY — the Phase-1 lesson, proven field-by-field

**The rule (non-negotiable):** the recipe contains **only identifiers, intents, sizes, and coordinates** — never a key, password, token, hash, or any `ENC:`/secret value. Secrets live in the PBS whole-CT snapshot and the escrow blobs, recovered with `R`, **never regenerated** (consistent with the controller's fail-closed `data_key` gate). This is exactly where the retired infra-backup failed: it shipped `encryption_key_b64`, `restic_password`, and a `controller_config_b64` with `cf_api_token`/`session_secret`/`password_hash` (`SPIKE-infra-backup-2026-06-15.md:100-115`) — a zero-knowledge violation. The recipe must not.

| Recipe field | Sensitive? | Why not |
|---|---|---|
| `customer_id`, `domain`, `tier` | No | Public identifiers |
| `guest.{cores,memory_bytes,disk_bytes,vmid}` | No | Sizing integers (already on the wire, `GuestSpec`) |
| `drives[].durable_id` / `wipe_durable_id` | No | FS-UUID / `byid` WWN — a hardware identifier, not a credential |
| `drives[].{role,mount_path,fs_type,total_bytes,intent,userdata_layout}` | No | Topology + intent; identical class to today's `StorageTarget` (already reported plaintext) |
| `pve_storage[].{name,type,content}` | No | Storage definitions, no auth (the PBS *token* is escrowed, not here) |
| `pbs.{repo_id,namespace,latest_snapshot_id}` | No | Coordinates; the encryption **key** is escrow-only, the access **token** is identity-escrow-only |
| `apps[].{name,catalog_ref,enabled,storage_bindings}` | No | Catalog reference + path bindings; compose comes from the catalog/PBS |
| `apps[].non_secret_deploy_fields` | **Must be filtered** | Only non-secret deploy fields; **every secret-typed field excluded at the source** |

**The one place a leak could enter** is `apps[].non_secret_deploy_fields` (free-form deploy config). The boundary is enforced at the **emitter** (controller), which already distinguishes secret vs non-secret deploy fields (it encrypts `ENC:` secrets in `app.yaml` and the retired infra-backup work confirmed the controller knows which env fields are secret-typed).

**A test that would catch a secret leaking in** (spec for the build slice): a `recipe_no_secrets_test` that (1) builds a recipe from a fixture customer whose apps include secret-typed deploy fields (passwords/tokens/keys) and asserts the serialized recipe JSON contains **none** of those values and **no** key whose name matches `(?i)(password|secret|token|key|hash|passphrase|api[_-]?key|ENC:)`; (2) a golden-shape test pinning the recipe's allowed top-level keys so a new field can't silently land without review; (3) a property check that every `apps[].*` field passing through is on an explicit **allowlist**, not a denylist (allowlist = a new sensitive field is excluded by default). Mirror it on both emitters (controller app-half, agent storage-half) so drift is caught on either side — the same cross-repo golden discipline already used for the host-report contract.

---

# PART 3 — Fork list + recommendation

**Recommendation (spec the recipe; defer the auth; never touch the secrets path):**

1. **Recipe schema + fields** — adopt the 2A shape (`recipe_version` + customer/guest/pbs/drives/pve_storage/apps). **Fork:** exact field set for `apps[].non_secret_deploy_fields` (allowlist contents) — needs the catalog/controller owner to enumerate the non-secret deploy fields per app. *Recommend: start with `{catalog_ref, enabled, storage_bindings}` only; add non-secret deploy fields incrementally behind the allowlist test.*
2. **Agent-emit vs hub-derive** — **emit from the agent** (storage/guest/PBS half) as an **additive host-report section**, and from the **controller** (app half) as an additive report section; the hub **assembles + stores**, it does not derive facts it doesn't own. *Rationale: the agent is the storage source of truth (`StorageTarget`), the controller owns the app set; the hub deriving either would re-introduce drift.* **Fork:** one combined recipe record vs two half-records stitched at read time.
3. **Storage location/format** — **plaintext** JSON on the hub, keyed by host/customer + `recipe_version`, in a **dedicated `dr_recipe` store separate from `host_escrow`** (opaque) and from the retired `infra_backup*` tables (dropped Phase 1). Hub-manageable: download / prune / pin / "drive a DR re-provision". *Recommend: new table, not a reuse of `infra_backup_versions`.*
4. **Tie-in to escrow / PBS / capstone** — the recipe is the **re-provision plan only**: it feeds **provisioning + desired-state** (the reserved `WireDesiredState.StorageManifest` placeholder, `report.go:271`), then the **existing** escrow→`K` and PBS-restore (Part 1) put secrets+bytes into the rebuilt scaffolding, and **restic** restores the bulk drives the recipe enumerated (the `UncoveredVolumes` gap). The recipe **must not** carry keys/tokens (those stay in escrow) — it only names the `pbs.namespace`/`snapshot_id` and `drives[].durable_id` the restore targets.
5. **Re-enrollment auth (recovery-mode toggle + out-of-band `R`)** — **DEFER.** It is the separate PENDING slice-10D track (agent-side consumption of `restore_directive` + recovery mode, `syncer.go:92`). This slice **specs and emits/stores the recipe**; it does not implement the restore orchestration or the `R` handshake. *Rationale: the recipe is valuable on its own (operator can rebuild scaffolding by hand today) and is independently testable; coupling it to the auth slice delays both.*
6. **Recipe wire-shape versioning** — carry an explicit `recipe_version` (start at 1) and keep the hub store and both emitters version-aware, with a cross-repo **golden** byte-pinned like the existing `desired-state.golden.json` / host-report contracts, so the agent and controller halves can evolve without silent drift. **Fork:** strict-reject vs ignore-unknown-fields on version skew. *Recommend: ignore-unknown on read (forward-compat, as the wire already does for `storage_manifest`), pin known keys in the golden.*

**Bottom line:** every fact the recipe needs is already collected (agent `StorageTarget` + guests + PBS; controller app set + bindings), the landing slot already exists (`WireDesiredState.StorageManifest`), and the boundary is enforceable at the emitter with an allowlist + no-secrets test. The recipe complements escrow (keys) and PBS/restic (bytes) without duplicating or weakening either — turning "I have PBS bytes" into "I can re-provision *this* customer."

---

## Appendix — Part 0 evidence index (live, read-only except the app-start fix)

- App states / exit codes / `State.Error` / `RestartCount`: `docker inspect` on guest 9201.
- Guest reboot: host `journalctl` `vzreboot:9201:root@pam OK` @ 12:26:08 local; guest `uptime -s` 10:26 UTC.
- Mount chain: host `lsblk`/`mount`; guest `/proc/mounts` + `ls /mnt/felhom-drives/felhom-flash`.
- No USB event: host `dmesg -T` + `journalctl -k` (10:20–10:35 UTC) empty for usb/sd/ext4.
- Agent rebind: `journalctl -u felhom-agent` `guest-attach: drive bound … 12:26:26` local + continuous `reconcile: enrolled drive bound`.
- Fix: `POST /api/stacks/{name}/start` ×7 → all `Up`; romm canary reads `/romm/library` on flash.

*Report-only. No source modified. The only production action was restarting seven already-deployed apps whose drive + data were present — service restoration, non-destructive, and itself the confirmation of the root cause.*