Off-site safety finished: Parts A-F evidence, decision 74, golden 0.290.0 recorded (vouched, floor 0.290.0), restore walked from the DooPlex copy, register 330 -> 328 (R-823/824/826/827/828/830 closed; R-833, R-834 opened)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

documentation/runbooks/ep0-datastore-copy.md

@@ -14,6 +14,8 @@ household's whole-box backups are encrypted with that household's own `encryptio
 | DooPlex PBS | datastore `ep0-copy` at `/mnt/5_hdd/backup/ep0-copy` | the copy |
 | DooPlex PBS | sync job `ep0-felhom-offsite`, daily 05:00, `remove-vanished false` | the nightly pull (ep0's prune runs 03:30). **It never removes what ep0 removed** — a deletion on ep0 does not reach the copy |
 | DooPlex PBS | verify job `verify-ep0-copy`, Saturdays 06:30 | reads the copy back |
+| DooPlex PBS | prune job `prune-ep0-copy`, daily 07:30, **keep-weekly 8**, all namespaces (decision 71) | keeps the last 8 weekly copies per group; runs after the 05:00 pull, never during it |
+| DooPlex PBS | garbage collection on `ep0-copy`, Sundays 08:30 | frees the chunks the prune released |
 | DooPlex PBS | notification target `felhom-operator` (SMTP via Resend → admin@felhom.eu) + matcher `felhom-operator-errors` (every error) | a failed pull or verify reaches the operator. Proven 2026-10-03 with a test mail |
 
 Secrets, all out of git: the ep0 token secret in `/etc/proxmox-backup/remote.cfg` (root:backup 0640, base64 —
@@ -31,20 +33,46 @@ sudo find /mnt/5_hdd/backup/ep0-copy/ns -mindepth 4 -maxdepth 4 -type d | sort #
 ## If ep0 is lost — restore a household's whole box from the DooPlex copy
 
 The copy is a normal PBS datastore. Two routes, both needing the household's PBS `encryption-key` (escrowed,
-recovered with the household's recovery code — the same as restoring from ep0):
+recovered with the household's recovery code — the same as restoring from ep0).
 
-1. **Point the box's host at DooPlex instead of ep0.** On the household's Proxmox host, add a PBS storage for
-   DooPlex's PBS (`ep0-copy`, namespace = the customer id) with the household's key, then restore the CT from it as
-   from ep0. DooPlex's PBS must be reachable from the host (it is not public today — the operator decides the route
-   at the time: a temporary tunnel, or a new endpoint).
-2. **Rebuild the endpoint.** Provision a new ep0 (06 §5), then pull back: on the new ep0 add DooPlex as a remote
-   and run `proxmox-backup-manager pull <dooplex-remote> ep0-copy felhom-offsite`. Every box then reconnects as before.
+### Route 1 — the host reads DooPlex directly. **WALKED 2026-10-04 on demo-hp** (`audits/offsite-finish-2026-10-04/partD-restore-walk.txt`)
 
-Do **not** prune or garbage-collect the copy tighter than ep0's own retention. The copy has no prune job today and
-grows with every nightly backup (R-828).
+1. **On DooPlex:** a read-only token for the restore (`proxmox-backup-manager user generate-token root@pam <name>`, then
+   `acl update /datastore/ep0-copy DatastoreReader --auth-id 'root@pam!<name>'`). Keep the secret in a file only.
+2. **On the host:** put the token in `/etc/pve/priv/storage/<id>.pw` (0600) and the household's PBS key in
+   `/etc/pve/priv/storage/<id>.enc`, then append the storage entry to `/etc/pve/storage.cfg`:
+   `pbs: <id>` / `datastore ep0-copy` / `server <DooPlex>` / `content backup` / `fingerprint <DooPlex PBS cert>` /
+   `namespace <customer>` / `username root@pam!<name>`.
+   **Do not use `pvesm add pbs` without `--password`:** it validates with the password from its command line, fails 401,
+   and on failure DELETES the `.pw`/`.enc` files you placed (measured). Passing `--password` puts the token on argv.
+3. `pvesm list <id>` → the household's snapshots (measured: 2 s). `pct restore <scratch VMID> <id>:backup/ct/<vmid>/<time>
+   --storage <dir storage> --unique 1` (measured: **186 s for a 15 GB-logical / 14 GB-on-disk backup** over the LAN, key
+   fingerprint printed by the restore).
+4. **⚠ BEFORE ANYTHING ELSE — the restored config is the PRODUCTION one:** `onboot: 1`, `mp8` bound to the host's REAL
+   household drives (`/mnt/felhom-drives`) and `mp9` to the original guest's bootstrap. Starting it, or a host reboot,
+   runs a second controller for the same household against the same drives. On a restore BESIDE the original:
+   `pct set <vmid> --onboot 0 --delete mp8,mp9` immediately (R-834). On a true replacement host, where the original is
+   gone, the binds are what you want.
+5. Read the data without starting it: `pct mount <vmid>` → `/var/lib/lxc/<vmid>/rootfs` (measured: 1 s; rootfs, the
+   controller data volume and `/var/lib/felhom` present, `settings.json` dated 10 min before the backup) → `pct unmount`.
+6. Teardown: `pct destroy <vmid> --purge`; `pvesm remove <id>` (removes the entry and its priv files — never the
+   household's own `felhom-pbs.enc`); on DooPlex delete the token and its ACL.
+
+**Reachability.** DooPlex's PBS (`:8007`) is reachable on DooPlex's LAN only. For a host on another network — a
+household's home — the options, none built (the operator decides at the time, R-832 is the long-term answer):
+(a) a temporary SSH forward from the host to DooPlex, as DooPlex already does to ep0 (needs an SSH key on DooPlex for
+that host); (b) a WireGuard peer on DooPlex's existing tailscale/k3s network for the duration; (c) route 2 below —
+rebuild an endpoint and pull back, so every host reconnects the usual way.
+
+### Route 2 — rebuild the endpoint. Not walked.
+
+Provision a new ep0 (06 §5), then on it add DooPlex as a remote and run
+`proxmox-backup-manager pull <dooplex-remote> ep0-copy felhom-offsite`. Every box then reconnects as before.
+
+Do **not** prune the copy tighter than decision 71 (8 weekly copies); never tighter than ep0's own retention.
 
 ## Remove
 
-`sudo proxmox-backup-manager sync-job remove ep0-felhom-offsite; … verify-job remove verify-ep0-copy;`
+`sudo proxmox-backup-manager sync-job remove ep0-felhom-offsite; … verify-job remove verify-ep0-copy; … prune-job remove prune-ep0-copy;`
 `sudo systemctl disable --now felhom-ep0-pbs-tunnel.service`; on ep0
 `proxmox-backup-manager user delete-token root@pam dooplex-sync`. The datastore's bytes stay until removed by hand.