admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(offsite): S4 SHIPPED — per-customer PBS tenancy + v4-pin/watchdog (live 2026-07-04)

Browse Source 
- runbook offsite-endpoint.md: new §4a (endpoint per-customer tenancy) + §4b
  (box-side storage + agent wiring). Confirmed minimal ACL: DatastoreBackup on
  /datastore/<ds>/<ns> (NOT /ns/<ns>) to BOTH user felhom@pbs AND token
  felhom@pbs!<ns> (PBS privsep = intersection); cross-tenant 403; ns-scoped
  verify works with DatastoreBackup; DatastoreBackup can't prune (safety).
  Box: box-born encryption key, argv-safe .pw/.enc + storage.cfg, agent PVE
  FelhomAgentStore grant, pbs-secrets WARN fix, local_backup_target retarget.
- doc-06 §3.4 SHIPPED (ns-aware PBS client v0.67.0 rationale); §4.2 v4-pin +
  watchdog SHIPPED v0.66.0 (closes the dual-stack trap); S4 roadmap row done +
  follow-ups (restore-test WAN deadline + scratch VM.Allocate; Tier-1/Tier-2
  target split).

Live: real vzdump of 9201 → ciphertext in ns over the tunnel; verify=ok under
the box's own scoped token; WARN gone; restore decrypts with the box-born key.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
authentik Default Admin
2026-07-04 17:21:05 +02:00
parent a7f954dde7
commit 1907dab296
2 changed files with 127 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/architecture/06-offsite-connectivity.md
+31 -17
View File
@@ -111,15 +111,25 @@ before this ships (the drilled `--rescope-acl` posture is unaffected).
Steps 3-4 and 5-6 are order-independent (WG tolerates either side appearing first); the reconcile
loop retries until both halves exist.
### 3.4 PBS over the tunnel
### 3.4 PBS over the tunnel — **SHIPPED 2026-07-04 (S4, agent v0.65-0.67)**
The PBS client config (`internal/pbs/client.go:24-31`) changes **only its `Server` value**: the
endpoint's tunnel-internal address (`pbs_tunnel_ip`) instead of a LAN IP. The TLS exact-cert pin
The PBS client config changes **only its `Server` value**: the endpoint's tunnel-internal address
(`pbs_tunnel_ip`, i.e. `10.77.0.1`) instead of a LAN IP. The TLS exact-cert pin
(`internal/pbs/pin.go:17-35`) sits **above** WG and is unchanged through the tunnel —
spike-proven both ways (P2b: pin validated through the tunnel; wrong pin rejected). Per-customer
namespace + per-customer escrowed encryption key exactly as today (D5); the endpoint sees
namespace + per-customer box-born encryption key exactly as today (D5); the endpoint sees
ciphertext only (01 §10).
**Live-proven end-to-end 2026-07-04** (runbook §4a/§4b): a real vzdump of guest 9201 lands as
ciphertext (`root.pxar` crypt-mode `encrypt`) in ns `demo-felhom-01` on `felhom-offsite` over the
tunnel; the box lists + verifies **only its own namespace** under a `DatastoreBackup` token
(`felhom@pbs!demo-felhom-01`), cross-tenant access 403s. **One agent addition was required and made
(v0.67.0):** the PBS client became **namespace-aware** — `Snapshots` passes `?ns=`, `Verify` passes
`ns=` — because the ns-unaware datastore-root calls 403 for a per-tenant token; a whole-datastore
verify would have needed `Datastore.Verify` (~admin), breaking isolation. The confirmed minimal
tenant ACL (`DatastoreBackup` on `/datastore/<ds>/<ns>` to BOTH the user and the token — PBS privsep
= intersection) and its gotchas live in runbook §4a.
### 3.5 Key custody + DR re-establishment
The WG private key joins the **escrowed IdentityBundle**
@@ -149,24 +159,28 @@ production endpoint exists.
WG roaming re-handshakes from the new source address automatically. Nothing anywhere references
the customer's IP — the hub peer registry stores pubkey+`/32` only (§3.2). No customer-side DNS,
ever.
- **4.2 Endpoint referenced by DNS + a re-resolve watchdog.** The endpoint is `dns_name`, not a
bare IP, so it can move. Trap to design around: `wg-quick` resolves `Endpoint=` **once** at
service start — a later endpoint re-IP would strand running tunnels. The agent's reconcile loop
therefore owns re-resolution: on tunnel-unhealthy (no recent handshake), re-resolve and re-apply
`wg set … endpoint=` (slice-3). Endpoint moves are rare, hub-announced (coords in desired-state),
and self-heal even without the watchdog on service restart.
- **4.2 Endpoint referenced by DNS + a re-resolve watchdog — SHIPPED v0.66.0 (v4-pinned).** The
endpoint is `dns_name`, not a bare IP, so it can move. Trap: `wg-quick` resolves `Endpoint=`
**once** at service start — a later endpoint re-IP would strand running tunnels. The agent now
**resolves the A record itself and writes the v4 LITERAL into the conf** (`internal/wgtunnel`
`Resolver` seam, `LookupNetIP "ip4"` → lowest address, deterministic fleet-wide), and a
**watchdog** (`Manager.Watchdog`, loop-driven) re-resolves when the handshake age exceeds
`wg_tunnel.stale_after_seconds` (default 180): IP changed → re-render + restart (re-IP recovery);
IP same → no churn. Steady state does **zero DNS + zero execs** (the resolved IP is cached); a
DNS failure keeps the last conf (never a teardown). This also **closes the dual-stack trap
below** — pinning the A literal means the tunnel can never silently ride un-NATed IPv6. Endpoint
moves are rare, hub-announced, and self-heal on service restart even without the watchdog.
**Dual-stack resolution trap (2026-07-04 CGNAT smoke test).** The endpoint is dual-stack (A +
AAAA). On a box with working IPv6, `wg-quick` resolves and pins the **AAAA** at bring-up and the
tunnel rides **un-NATed IPv6** — functionally fine (often better: no NAT), but it means the
v4/NAT path is only exercised when v6 is absent. Two consequences: reasoning about "CGNAT
traversal" applies to the v4 fallback only; and the re-resolve watchdog must not silently flip
families mid-life in a way that strands a half-open v6 mapping. `AI_ADDRCONFIG` already gives the
safe default — a v4-only box never sees the AAAA — but a v6-capable box will prefer it. The
client MTU floor (§4.3) is deliberately chosen **family-agnostic (1280)** *so that* MTU
correctness does not depend on this resolution outcome. A **deterministic v4-pin** — the agent
resolving ep0's A record and writing the literal IP into the conf (a natural extension of this
watchdog) — remains an **open, separate determinism question**; it is **not required for MTU
correctness** and is tracked on its own, not built by S3.1.
safe default — a v4-only box never sees the AAAA — but a v6-capable box would prefer it. The
client MTU floor (§4.3) is family-agnostic (1280) so MTU correctness never depended on this;
and the **v4-pin above (v0.66.0) now resolves it outright** — the agent writes the A literal, so
the tunnel always takes the v4/NAT path regardless of the box's v6 capability. (Was tracked as an
open determinism question through S3.1; closed by S4's v4-pin.)
- **4.3 MTU/MSS — DECIDED: fleet-wide client MTU 1280 (agent v0.65.0, S3.1).** WG costs 60 bytes
on v4 / 80 on IPv6. The unit ships explicit **`MTU 1280`** — the **IPv6-minimum link MTU** (RFC
8200 guarantees every path carries ≥ 1280), so outer = 1280+60 = 1340 (v4) / 1280+80 = 1360 (v6),
@@ -281,7 +295,7 @@ vantage (Peti VM 110) remains the thorough confirmation but no longer gates anyt
| **S1 — endpoint provisioning + peer-sync surface** — **SHIPPED 2026-07-04** (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: **hub pushes over SSH** (`x/crypto/ssh`, `FixedHostKey` pin + pinned-type `HostKeyAlgorithms`) to a forced-command reconcile script; runbook `documentation/runbooks/offsite-endpoint.md`. Done-criterion met live: hub add/remove → `wg show` on the endpoint; tunnel `ep0.felhom.eu:443` → PBS through the wg0-only 8007 rule. | ~~provision the production VM…~~ (see runbook; production endpoint = a later re-run) | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓ |
| **S2 — hub schema + desired-state** — **SHIPPED 2026-07-04** (hub v0.33.0). `POST /hosts/{id}/wg` (self-scoped) + merge-at-read `wireguard` block (§3.2 shape = the golden) + one-per-host index + re-key-in-place + `/offsite` UI page. Live-proven against the real felhom-pve record incl. old-agent (v0.63.0) tolerance. | ~~endpoint coords + peer registry…~~ | a registered box's desired-state carries its assignment ✓; hub UI shows the peer registry ✓ |
| **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) |
| **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) |
| **S4 — PBS over the tunnel — SHIPPED 2026-07-04** (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep `DatastoreBackup` token, dual-grant user+token on `/datastore/<ds>/<ns>`, cross-tenant 403); box `felhom-offsite` PVE storage `Server=10.77.0.1`, box-born encryption key, token-secret WARN fix, `local_backup_target` retarget; agent **v0.66.0** v4-pin + re-resolve watchdog + FELHOM_WG Critical, **v0.67.0** namespace-aware PBS client. | ~~point the box's PBS storage at `pbs_tunnel_ip:8007`~~ | real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: restore-test scheduler WAN deadline + scratch `VM.Allocate`; Tier-1-local-vs-Tier-2-offsite target split (`local_backup_target` is currently a single target). |
| **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* |
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
documentation/runbooks/offsite-endpoint.md
+96
View File
@@ -132,6 +132,102 @@ datastore parent is `/srv` (world-traversable — a datastore under `/root` fail
`Permission denied` for the `backup` user). GUI: reachable ONLY via SSH port-forward
(`ssh -L 8007:127.0.0.1:8007 root@<box>`) or through the tunnel — by design, no public 8007.
## 4a. Per-customer PBS tenancy (S4 — live-confirmed 2026-07-04 on felhom-hetzner)
Each customer box gets ITS OWN namespace + a privilege-separated token scoped to that namespace,
on the single shared `felhom-offsite` datastore (doc 06 D5). One shared PBS user `felhom@pbs`
holds a per-customer token each; cross-tenant isolation is enforced by the per-token ACL.
**On the endpoint, per customer `<ns>` (e.g. `demo-felhom-01` = the box's `host_id`):**
```sh
DS=felhom-offsite
# 1. namespace — created via a THROWAWAY root admin token (root@pam has no password on the cloud
# image; proxmox-backup-manager has NO `namespace` subcommand — namespaces are client-side):
proxmox-backup-manager user generate-token root@pam nsadmin > /root/.nsadmin.raw # umask 077
NSADMIN=$(sed -n 's/.*"value": "\([^"]*\)".*/\1/p' /root/.nsadmin.raw); rm -f /root/.nsadmin.raw
proxmox-backup-manager acl update /datastore/$DS DatastoreAdmin --auth-id 'root@pam!nsadmin'
export PBS_FINGERPRINT="$(proxmox-backup-manager cert info | awk '/Fingerprint/{print $3}')"
PBS_PASSWORD="$NSADMIN" proxmox-backup-client namespace create <ns> --repository "root@pam!nsadmin@localhost:$DS"
# 2. the shared user (once) + this customer's privilege-separated token (secret → root-only file):
proxmox-backup-manager user create felhom@pbs --comment 'offsite tenancy' 2>/dev/null || true
proxmox-backup-manager user generate-token felhom@pbs <ns> > /root/.tok-<ns>.raw
sed -n 's/.*"value": "\([^"]*\)".*/\1/p' /root/.tok-<ns>.raw > /root/.tok-<ns>.secret # 0600; rm the .raw
# 3. THE ACL — the load-bearing gotcha (see below): DatastoreBackup on the NAMESPACE PATH, to BOTH
# the user AND the token:
proxmox-backup-manager acl update /datastore/$DS/<ns> DatastoreBackup --auth-id 'felhom@pbs'
proxmox-backup-manager acl update /datastore/$DS/<ns> DatastoreBackup --auth-id 'felhom@pbs!<ns>'
# 4. clean up the throwaway admin token:
proxmox-backup-manager user delete-token root@pam nsadmin
proxmox-backup-manager acl update /datastore/$DS DatastoreAdmin --auth-id 'root@pam!nsadmin' --delete
```
Hand `/root/.tok-<ns>.secret` + the cert fingerprint to the box out-of-band (they become the PVE
storage `.pw` + `fingerprint` — §4b).
**The confirmed minimal ACL + the gotchas (all live-proven, don't re-derive):**
- **ACL path is `/datastore/<ds>/<ns>` — NOT `/datastore/<ds>/ns/<ns>`.** The `/ns/` form is the
*client repository* path; the *ACL object* path drops it. Granting `/ns/<ns>` silently no-ops
(the permission check runs against `/datastore/<ds>/<ns>`).
- **PBS privsep tokens = intersection(user perms, token perms).** The token's own ACL alone is
NOT enough — the **user** `felhom@pbs` must ALSO be granted on the namespace, or the token has
the empty intersection and every op 403s (`missing Datastore.Backup`). Isolation is preserved
anyway: token B's effective perms = user(has A's + B's ns) ∩ token-B(only B's ns) = B's ns only.
Adversarially confirmed: token A → list/backup ns B ⇒ **403**.
- **`DatastoreBackup` is the whole grant** — it covers backup, list-own, restore-own, AND
**namespace-scoped verify** (`POST /verify` with `ns=<ns>` — no `Datastore.Verify`/admin needed).
It deliberately does **NOT** cover prune/forget (`Datastore.Modify|Prune`) — the box **cannot
delete its own backups**; pruning is an operator/endpoint job (matches "the runner never prunes
PBS"). No token ever exceeds `DatastoreBackup`; no admin token lives on the endpoint for a box.
## 4b. Box-side storage + agent wiring (S4 — on the customer box, as root)
```sh
FP=<endpoint cert fingerprint>; UN='felhom@pbs!<ns>'; PWSTORE=/etc/pve/priv/storage
# 1. encryption key — BORN ON THE BOX (never leaves as plaintext; the endpoint only ever sees
# ciphertext). Its fingerprint goes in storage.cfg:
proxmox-backup-client key create $PWSTORE/felhom-offsite.enc --kdf none
ENCFP=$(sed -n 's/.*"fingerprint":"\([^"]*\)".*/\1/p' $PWSTORE/felhom-offsite.enc)
chown root:www-data $PWSTORE/felhom-offsite.enc && chmod 600 $PWSTORE/felhom-offsite.enc
# 2. token secret → PVE .pw store — ARGV-SAFE (pvesm add only takes --password on argv AND it
# validates the connection, so a dummy won't do; write the .pw file + storage.cfg directly):
install -o root -g www-data -m 600 /root/.tok-<ns>.secret $PWSTORE/felhom-offsite.pw
cat >> /etc/pve/storage.cfg <<CFG
pbs: felhom-offsite
datastore felhom-offsite
server 10.77.0.1
content backup
fingerprint $FP
namespace <ns>
encryption-key $ENCFP
username $UN
CFG
pvesm status --storage felhom-offsite # expect: active (PVE connects over the tunnel)
# 3. agent PVE token needs the new storage (dual-grant, PVE privsep like PBS):
pveum acl modify /storage/felhom-offsite --roles FelhomAgentStore --users 'felhom-agent@pve'
pveum acl modify /storage/felhom-offsite --roles FelhomAgentStore --tokens 'felhom-agent@pve!agent'
# 4. THE WARN FIX — the non-root agent cannot read /etc/pve/priv (root 0700); give it an owned copy:
install -d -o felhom-agent -g felhom-agent -m 0700 /var/lib/felhom-agent/pbs-secrets
for ID in felhom-pbs felhom-offsite; do
install -o felhom-agent -g felhom-agent -m 0600 /etc/pve/priv/storage/$ID.pw /var/lib/felhom-agent/pbs-secrets/$ID.pw
done
# then in agent.json backup{}: "pbs_secret_dir":"/var/lib/felhom-agent/pbs-secrets"
# 5. RETARGET the box's backups offsite — the field is backup.local_backup_target (BackupTarget()):
# agent.json backup{}: "local_backup_target":"felhom-offsite" (defaultBackupTarget stays felhom-pbs in code)
systemctl restart felhom-agent # verify: journal has ZERO "cannot read token secret"; verify loop
# logs `verify cycle complete datastore=felhom-offsite`
```
Verified live 2026-07-04: a real vzdump of guest 9201 (crash-consistent, ~4.6 GB compressed over
the tunnel, ~20 min) lands as **ciphertext** in ns `demo-felhom-01` (`root.pxar` crypt-mode
`encrypt`); the ns-scoped verify runs under the box's own `felhom@pbs!demo-felhom-01` token and
returns `verify=ok`; the WARN is gone; a restore reads it back decrypting with the on-box key.
**Provisioning notes / follow-ups (S4):** (a) the `.pw` copy in `pbs-secrets` is a STATIC snapshot
of the PVE token secret — a token rotation must re-copy it (a sync step for the hub-driven-tenancy
slice). (b) The restore-test scheduler's internal restore-task deadline (~10 min) is too short for
a WAN restore of a large volume, and the agent PVE token lacks `VM.Allocate` on the scratch VMID
band for teardown — both are box-side config fixes needed before the offsite restore-test can run
unattended (raise the restore deadline for the `pbs` tier; grant the scratch-band VM.Allocate).
## 5. Peersync surface
```sh