From 1907dab29667f3074a0f5287c5fec5d6094e51f1 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 4 Jul 2026 17:21:05 +0200 Subject: [PATCH] =?UTF-8?q?docs(offsite):=20S4=20SHIPPED=20=E2=80=94=20per?= =?UTF-8?q?-customer=20PBS=20tenancy=20+=20v4-pin/watchdog=20(live=202026-?= =?UTF-8?q?07-04)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - runbook offsite-endpoint.md: new §4a (endpoint per-customer tenancy) + §4b (box-side storage + agent wiring). Confirmed minimal ACL: DatastoreBackup on /datastore// (NOT /ns/) to BOTH user felhom@pbs AND token felhom@pbs! (PBS privsep = intersection); cross-tenant 403; ns-scoped verify works with DatastoreBackup; DatastoreBackup can't prune (safety). Box: box-born encryption key, argv-safe .pw/.enc + storage.cfg, agent PVE FelhomAgentStore grant, pbs-secrets WARN fix, local_backup_target retarget. - doc-06 §3.4 SHIPPED (ns-aware PBS client v0.67.0 rationale); §4.2 v4-pin + watchdog SHIPPED v0.66.0 (closes the dual-stack trap); S4 roadmap row done + follow-ups (restore-test WAN deadline + scratch VM.Allocate; Tier-1/Tier-2 target split). Live: real vzdump of 9201 → ciphertext in ns over the tunnel; verify=ok under the box's own scoped token; WARN gone; restore decrypts with the box-born key. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- .../architecture/06-offsite-connectivity.md | 48 ++++++---- documentation/runbooks/offsite-endpoint.md | 96 +++++++++++++++++++ 2 files changed, 127 insertions(+), 17 deletions(-) diff --git a/documentation/architecture/06-offsite-connectivity.md b/documentation/architecture/06-offsite-connectivity.md index 049f85f..8300c64 100644 --- a/documentation/architecture/06-offsite-connectivity.md +++ b/documentation/architecture/06-offsite-connectivity.md @@ -111,15 +111,25 @@ before this ships (the drilled `--rescope-acl` posture is unaffected). Steps 3-4 and 5-6 are order-independent (WG tolerates either side appearing first); the reconcile loop retries until both halves exist. -### 3.4 PBS over the tunnel +### 3.4 PBS over the tunnel — **SHIPPED 2026-07-04 (S4, agent v0.65-0.67)** -The PBS client config (`internal/pbs/client.go:24-31`) changes **only its `Server` value**: the -endpoint's tunnel-internal address (`pbs_tunnel_ip`) instead of a LAN IP. The TLS exact-cert pin +The PBS client config changes **only its `Server` value**: the endpoint's tunnel-internal address +(`pbs_tunnel_ip`, i.e. `10.77.0.1`) instead of a LAN IP. The TLS exact-cert pin (`internal/pbs/pin.go:17-35`) sits **above** WG and is unchanged through the tunnel — spike-proven both ways (P2b: pin validated through the tunnel; wrong pin rejected). Per-customer -namespace + per-customer escrowed encryption key exactly as today (D5); the endpoint sees +namespace + per-customer box-born encryption key exactly as today (D5); the endpoint sees ciphertext only (01 §10). +**Live-proven end-to-end 2026-07-04** (runbook §4a/§4b): a real vzdump of guest 9201 lands as +ciphertext (`root.pxar` crypt-mode `encrypt`) in ns `demo-felhom-01` on `felhom-offsite` over the +tunnel; the box lists + verifies **only its own namespace** under a `DatastoreBackup` token +(`felhom@pbs!demo-felhom-01`), cross-tenant access 403s. **One agent addition was required and made +(v0.67.0):** the PBS client became **namespace-aware** — `Snapshots` passes `?ns=`, `Verify` passes +`ns=` — because the ns-unaware datastore-root calls 403 for a per-tenant token; a whole-datastore +verify would have needed `Datastore.Verify` (~admin), breaking isolation. The confirmed minimal +tenant ACL (`DatastoreBackup` on `/datastore//` to BOTH the user and the token — PBS privsep += intersection) and its gotchas live in runbook §4a. + ### 3.5 Key custody + DR re-establishment The WG private key joins the **escrowed IdentityBundle** @@ -149,24 +159,28 @@ production endpoint exists. WG roaming re-handshakes from the new source address automatically. Nothing anywhere references the customer's IP — the hub peer registry stores pubkey+`/32` only (§3.2). No customer-side DNS, ever. -- **4.2 Endpoint referenced by DNS + a re-resolve watchdog.** The endpoint is `dns_name`, not a - bare IP, so it can move. Trap to design around: `wg-quick` resolves `Endpoint=` **once** at - service start — a later endpoint re-IP would strand running tunnels. The agent's reconcile loop - therefore owns re-resolution: on tunnel-unhealthy (no recent handshake), re-resolve and re-apply - `wg set … endpoint=` (slice-3). Endpoint moves are rare, hub-announced (coords in desired-state), - and self-heal even without the watchdog on service restart. +- **4.2 Endpoint referenced by DNS + a re-resolve watchdog — SHIPPED v0.66.0 (v4-pinned).** The + endpoint is `dns_name`, not a bare IP, so it can move. Trap: `wg-quick` resolves `Endpoint=` + **once** at service start — a later endpoint re-IP would strand running tunnels. The agent now + **resolves the A record itself and writes the v4 LITERAL into the conf** (`internal/wgtunnel` + `Resolver` seam, `LookupNetIP "ip4"` → lowest address, deterministic fleet-wide), and a + **watchdog** (`Manager.Watchdog`, loop-driven) re-resolves when the handshake age exceeds + `wg_tunnel.stale_after_seconds` (default 180): IP changed → re-render + restart (re-IP recovery); + IP same → no churn. Steady state does **zero DNS + zero execs** (the resolved IP is cached); a + DNS failure keeps the last conf (never a teardown). This also **closes the dual-stack trap + below** — pinning the A literal means the tunnel can never silently ride un-NATed IPv6. Endpoint + moves are rare, hub-announced, and self-heal on service restart even without the watchdog. **Dual-stack resolution trap (2026-07-04 CGNAT smoke test).** The endpoint is dual-stack (A + AAAA). On a box with working IPv6, `wg-quick` resolves and pins the **AAAA** at bring-up and the tunnel rides **un-NATed IPv6** — functionally fine (often better: no NAT), but it means the v4/NAT path is only exercised when v6 is absent. Two consequences: reasoning about "CGNAT traversal" applies to the v4 fallback only; and the re-resolve watchdog must not silently flip families mid-life in a way that strands a half-open v6 mapping. `AI_ADDRCONFIG` already gives the - safe default — a v4-only box never sees the AAAA — but a v6-capable box will prefer it. The - client MTU floor (§4.3) is deliberately chosen **family-agnostic (1280)** *so that* MTU - correctness does not depend on this resolution outcome. A **deterministic v4-pin** — the agent - resolving ep0's A record and writing the literal IP into the conf (a natural extension of this - watchdog) — remains an **open, separate determinism question**; it is **not required for MTU - correctness** and is tracked on its own, not built by S3.1. + safe default — a v4-only box never sees the AAAA — but a v6-capable box would prefer it. The + client MTU floor (§4.3) is family-agnostic (1280) so MTU correctness never depended on this; + and the **v4-pin above (v0.66.0) now resolves it outright** — the agent writes the A literal, so + the tunnel always takes the v4/NAT path regardless of the box's v6 capability. (Was tracked as an + open determinism question through S3.1; closed by S4's v4-pin.) - **4.3 MTU/MSS — DECIDED: fleet-wide client MTU 1280 (agent v0.65.0, S3.1).** WG costs 60 bytes on v4 / 80 on IPv6. The unit ships explicit **`MTU 1280`** — the **IPv6-minimum link MTU** (RFC 8200 guarantees every path carries ≥ 1280), so outer = 1280+60 = 1340 (v4) / 1280+80 = 1360 (v6), @@ -281,7 +295,7 @@ vantage (Peti VM 110) remains the thorough confirmation but no longer gates anyt | **S1 — endpoint provisioning + peer-sync surface** — **SHIPPED 2026-07-04** (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: **hub pushes over SSH** (`x/crypto/ssh`, `FixedHostKey` pin + pinned-type `HostKeyAlgorithms`) to a forced-command reconcile script; runbook `documentation/runbooks/offsite-endpoint.md`. Done-criterion met live: hub add/remove → `wg show` on the endpoint; tunnel `ep0.felhom.eu:443` → PBS through the wg0-only 8007 rule. | ~~provision the production VM…~~ (see runbook; production endpoint = a later re-run) | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓ | | **S2 — hub schema + desired-state** — **SHIPPED 2026-07-04** (hub v0.33.0). `POST /hosts/{id}/wg` (self-scoped) + merge-at-read `wireguard` block (§3.2 shape = the golden) + one-per-host index + re-key-in-place + `/offsite` UI page. Live-proven against the real felhom-pve record incl. old-agent (v0.63.0) tolerance. | ~~endpoint coords + peer registry…~~ | a registered box's desired-state carries its assignment ✓; hub UI shows the peer registry ✓ | | **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) | -| **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) | +| **S4 — PBS over the tunnel — SHIPPED 2026-07-04** (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep `DatastoreBackup` token, dual-grant user+token on `/datastore//`, cross-tenant 403); box `felhom-offsite` PVE storage `Server=10.77.0.1`, box-born encryption key, token-secret WARN fix, `local_backup_target` retarget; agent **v0.66.0** v4-pin + re-resolve watchdog + FELHOM_WG Critical, **v0.67.0** namespace-aware PBS client. | ~~point the box's PBS storage at `pbs_tunnel_ip:8007`~~ | real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: restore-test scheduler WAN deadline + scratch `VM.Allocate`; Tier-1-local-vs-Tier-2-offsite target split (`local_backup_target` is currently a single target). | | **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* | | **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially | diff --git a/documentation/runbooks/offsite-endpoint.md b/documentation/runbooks/offsite-endpoint.md index 3666fa5..9d6ef17 100644 --- a/documentation/runbooks/offsite-endpoint.md +++ b/documentation/runbooks/offsite-endpoint.md @@ -132,6 +132,102 @@ datastore parent is `/srv` (world-traversable — a datastore under `/root` fail `Permission denied` for the `backup` user). GUI: reachable ONLY via SSH port-forward (`ssh -L 8007:127.0.0.1:8007 root@`) or through the tunnel — by design, no public 8007. +## 4a. Per-customer PBS tenancy (S4 — live-confirmed 2026-07-04 on felhom-hetzner) + +Each customer box gets ITS OWN namespace + a privilege-separated token scoped to that namespace, +on the single shared `felhom-offsite` datastore (doc 06 D5). One shared PBS user `felhom@pbs` +holds a per-customer token each; cross-tenant isolation is enforced by the per-token ACL. + +**On the endpoint, per customer `` (e.g. `demo-felhom-01` = the box's `host_id`):** +```sh +DS=felhom-offsite +# 1. namespace — created via a THROWAWAY root admin token (root@pam has no password on the cloud +# image; proxmox-backup-manager has NO `namespace` subcommand — namespaces are client-side): +proxmox-backup-manager user generate-token root@pam nsadmin > /root/.nsadmin.raw # umask 077 +NSADMIN=$(sed -n 's/.*"value": "\([^"]*\)".*/\1/p' /root/.nsadmin.raw); rm -f /root/.nsadmin.raw +proxmox-backup-manager acl update /datastore/$DS DatastoreAdmin --auth-id 'root@pam!nsadmin' +export PBS_FINGERPRINT="$(proxmox-backup-manager cert info | awk '/Fingerprint/{print $3}')" +PBS_PASSWORD="$NSADMIN" proxmox-backup-client namespace create --repository "root@pam!nsadmin@localhost:$DS" +# 2. the shared user (once) + this customer's privilege-separated token (secret → root-only file): +proxmox-backup-manager user create felhom@pbs --comment 'offsite tenancy' 2>/dev/null || true +proxmox-backup-manager user generate-token felhom@pbs > /root/.tok-.raw +sed -n 's/.*"value": "\([^"]*\)".*/\1/p' /root/.tok-.raw > /root/.tok-.secret # 0600; rm the .raw +# 3. THE ACL — the load-bearing gotcha (see below): DatastoreBackup on the NAMESPACE PATH, to BOTH +# the user AND the token: +proxmox-backup-manager acl update /datastore/$DS/ DatastoreBackup --auth-id 'felhom@pbs' +proxmox-backup-manager acl update /datastore/$DS/ DatastoreBackup --auth-id 'felhom@pbs!' +# 4. clean up the throwaway admin token: +proxmox-backup-manager user delete-token root@pam nsadmin +proxmox-backup-manager acl update /datastore/$DS DatastoreAdmin --auth-id 'root@pam!nsadmin' --delete +``` +Hand `/root/.tok-.secret` + the cert fingerprint to the box out-of-band (they become the PVE +storage `.pw` + `fingerprint` — §4b). + +**The confirmed minimal ACL + the gotchas (all live-proven, don't re-derive):** +- **ACL path is `/datastore//` — NOT `/datastore//ns/`.** The `/ns/` form is the + *client repository* path; the *ACL object* path drops it. Granting `/ns/` silently no-ops + (the permission check runs against `/datastore//`). +- **PBS privsep tokens = intersection(user perms, token perms).** The token's own ACL alone is + NOT enough — the **user** `felhom@pbs` must ALSO be granted on the namespace, or the token has + the empty intersection and every op 403s (`missing Datastore.Backup`). Isolation is preserved + anyway: token B's effective perms = user(has A's + B's ns) ∩ token-B(only B's ns) = B's ns only. + Adversarially confirmed: token A → list/backup ns B ⇒ **403**. +- **`DatastoreBackup` is the whole grant** — it covers backup, list-own, restore-own, AND + **namespace-scoped verify** (`POST /verify` with `ns=` — no `Datastore.Verify`/admin needed). + It deliberately does **NOT** cover prune/forget (`Datastore.Modify|Prune`) — the box **cannot + delete its own backups**; pruning is an operator/endpoint job (matches "the runner never prunes + PBS"). No token ever exceeds `DatastoreBackup`; no admin token lives on the endpoint for a box. + +## 4b. Box-side storage + agent wiring (S4 — on the customer box, as root) + +```sh +FP=; UN='felhom@pbs!'; PWSTORE=/etc/pve/priv/storage +# 1. encryption key — BORN ON THE BOX (never leaves as plaintext; the endpoint only ever sees +# ciphertext). Its fingerprint goes in storage.cfg: +proxmox-backup-client key create $PWSTORE/felhom-offsite.enc --kdf none +ENCFP=$(sed -n 's/.*"fingerprint":"\([^"]*\)".*/\1/p' $PWSTORE/felhom-offsite.enc) +chown root:www-data $PWSTORE/felhom-offsite.enc && chmod 600 $PWSTORE/felhom-offsite.enc +# 2. token secret → PVE .pw store — ARGV-SAFE (pvesm add only takes --password on argv AND it +# validates the connection, so a dummy won't do; write the .pw file + storage.cfg directly): +install -o root -g www-data -m 600 /root/.tok-.secret $PWSTORE/felhom-offsite.pw +cat >> /etc/pve/storage.cfg < + encryption-key $ENCFP + username $UN +CFG +pvesm status --storage felhom-offsite # expect: active (PVE connects over the tunnel) +# 3. agent PVE token needs the new storage (dual-grant, PVE privsep like PBS): +pveum acl modify /storage/felhom-offsite --roles FelhomAgentStore --users 'felhom-agent@pve' +pveum acl modify /storage/felhom-offsite --roles FelhomAgentStore --tokens 'felhom-agent@pve!agent' +# 4. THE WARN FIX — the non-root agent cannot read /etc/pve/priv (root 0700); give it an owned copy: +install -d -o felhom-agent -g felhom-agent -m 0700 /var/lib/felhom-agent/pbs-secrets +for ID in felhom-pbs felhom-offsite; do + install -o felhom-agent -g felhom-agent -m 0600 /etc/pve/priv/storage/$ID.pw /var/lib/felhom-agent/pbs-secrets/$ID.pw +done +# then in agent.json backup{}: "pbs_secret_dir":"/var/lib/felhom-agent/pbs-secrets" +# 5. RETARGET the box's backups offsite — the field is backup.local_backup_target (BackupTarget()): +# agent.json backup{}: "local_backup_target":"felhom-offsite" (defaultBackupTarget stays felhom-pbs in code) +systemctl restart felhom-agent # verify: journal has ZERO "cannot read token secret"; verify loop + # logs `verify cycle complete datastore=felhom-offsite` +``` +Verified live 2026-07-04: a real vzdump of guest 9201 (crash-consistent, ~4.6 GB compressed over +the tunnel, ~20 min) lands as **ciphertext** in ns `demo-felhom-01` (`root.pxar` crypt-mode +`encrypt`); the ns-scoped verify runs under the box's own `felhom@pbs!demo-felhom-01` token and +returns `verify=ok`; the WARN is gone; a restore reads it back decrypting with the on-box key. + +**Provisioning notes / follow-ups (S4):** (a) the `.pw` copy in `pbs-secrets` is a STATIC snapshot +of the PVE token secret — a token rotation must re-copy it (a sync step for the hub-driven-tenancy +slice). (b) The restore-test scheduler's internal restore-task deadline (~10 min) is too short for +a WAN restore of a large volume, and the agent PVE token lacks `VM.Allocate` on the scratch VMID +band for teardown — both are box-side config fixes needed before the offsite restore-test can run +unattended (raise the restore deadline for the `pbs` tier; grant the scratch-band VM.Allocate). + ## 5. Peersync surface ```sh