admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(offsite): S4 SHIPPED — per-customer PBS tenancy + v4-pin/watchdog (live 2026-07-04)

Browse Source 
- runbook offsite-endpoint.md: new §4a (endpoint per-customer tenancy) + §4b
  (box-side storage + agent wiring). Confirmed minimal ACL: DatastoreBackup on
  /datastore/<ds>/<ns> (NOT /ns/<ns>) to BOTH user felhom@pbs AND token
  felhom@pbs!<ns> (PBS privsep = intersection); cross-tenant 403; ns-scoped
  verify works with DatastoreBackup; DatastoreBackup can't prune (safety).
  Box: box-born encryption key, argv-safe .pw/.enc + storage.cfg, agent PVE
  FelhomAgentStore grant, pbs-secrets WARN fix, local_backup_target retarget.
- doc-06 §3.4 SHIPPED (ns-aware PBS client v0.67.0 rationale); §4.2 v4-pin +
  watchdog SHIPPED v0.66.0 (closes the dual-stack trap); S4 roadmap row done +
  follow-ups (restore-test WAN deadline + scratch VM.Allocate; Tier-1/Tier-2
  target split).

Live: real vzdump of 9201 → ciphertext in ns over the tunnel; verify=ok under
the box's own scoped token; WARN gone; restore decrypts with the box-born key.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
authentik Default Admin
2026-07-04 17:21:05 +02:00
parent a7f954dde7
commit 1907dab296
2 changed files with 127 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/architecture/06-offsite-connectivity.md
+31 -17
View File
@@ -111,15 +111,25 @@ before this ships (the drilled `--rescope-acl` posture is unaffected).
Steps 3-4 and 5-6 are order-independent (WG tolerates either side appearing first); the reconcile
loop retries until both halves exist.
### 3.4 PBS over the tunnel
### 3.4 PBS over the tunnel — **SHIPPED 2026-07-04 (S4, agent v0.65-0.67)**
The PBS client config (`internal/pbs/client.go:24-31`) changes **only its `Server` value**: the
endpoint's tunnel-internal address (`pbs_tunnel_ip`) instead of a LAN IP. The TLS exact-cert pin
The PBS client config changes **only its `Server` value**: the endpoint's tunnel-internal address
(`pbs_tunnel_ip`, i.e. `10.77.0.1`) instead of a LAN IP. The TLS exact-cert pin
(`internal/pbs/pin.go:17-35`) sits **above** WG and is unchanged through the tunnel —
spike-proven both ways (P2b: pin validated through the tunnel; wrong pin rejected). Per-customer
namespace + per-customer escrowed encryption key exactly as today (D5); the endpoint sees
namespace + per-customer box-born encryption key exactly as today (D5); the endpoint sees
ciphertext only (01 §10).
**Live-proven end-to-end 2026-07-04** (runbook §4a/§4b): a real vzdump of guest 9201 lands as
ciphertext (`root.pxar` crypt-mode `encrypt`) in ns `demo-felhom-01` on `felhom-offsite` over the
tunnel; the box lists + verifies **only its own namespace** under a `DatastoreBackup` token
(`felhom@pbs!demo-felhom-01`), cross-tenant access 403s. **One agent addition was required and made
(v0.67.0):** the PBS client became **namespace-aware** — `Snapshots` passes `?ns=`, `Verify` passes
`ns=` — because the ns-unaware datastore-root calls 403 for a per-tenant token; a whole-datastore
verify would have needed `Datastore.Verify` (~admin), breaking isolation. The confirmed minimal
tenant ACL (`DatastoreBackup` on `/datastore/<ds>/<ns>` to BOTH the user and the token — PBS privsep
= intersection) and its gotchas live in runbook §4a.
### 3.5 Key custody + DR re-establishment
The WG private key joins the **escrowed IdentityBundle**
@@ -149,24 +159,28 @@ production endpoint exists.
WG roaming re-handshakes from the new source address automatically. Nothing anywhere references
the customer's IP — the hub peer registry stores pubkey+`/32` only (§3.2). No customer-side DNS,
ever.
- **4.2 Endpoint referenced by DNS + a re-resolve watchdog.** The endpoint is `dns_name`, not a
bare IP, so it can move. Trap to design around: `wg-quick` resolves `Endpoint=` **once** at
service start — a later endpoint re-IP would strand running tunnels. The agent's reconcile loop
therefore owns re-resolution: on tunnel-unhealthy (no recent handshake), re-resolve and re-apply
`wg set … endpoint=` (slice-3). Endpoint moves are rare, hub-announced (coords in desired-state),
and self-heal even without the watchdog on service restart.
- **4.2 Endpoint referenced by DNS + a re-resolve watchdog — SHIPPED v0.66.0 (v4-pinned).** The
endpoint is `dns_name`, not a bare IP, so it can move. Trap: `wg-quick` resolves `Endpoint=`
**once** at service start — a later endpoint re-IP would strand running tunnels. The agent now
**resolves the A record itself and writes the v4 LITERAL into the conf** (`internal/wgtunnel`
`Resolver` seam, `LookupNetIP "ip4"` → lowest address, deterministic fleet-wide), and a
**watchdog** (`Manager.Watchdog`, loop-driven) re-resolves when the handshake age exceeds
`wg_tunnel.stale_after_seconds` (default 180): IP changed → re-render + restart (re-IP recovery);
IP same → no churn. Steady state does **zero DNS + zero execs** (the resolved IP is cached); a
DNS failure keeps the last conf (never a teardown). This also **closes the dual-stack trap
below** — pinning the A literal means the tunnel can never silently ride un-NATed IPv6. Endpoint
moves are rare, hub-announced, and self-heal on service restart even without the watchdog.
**Dual-stack resolution trap (2026-07-04 CGNAT smoke test).** The endpoint is dual-stack (A +
AAAA). On a box with working IPv6, `wg-quick` resolves and pins the **AAAA** at bring-up and the
tunnel rides **un-NATed IPv6** — functionally fine (often better: no NAT), but it means the
v4/NAT path is only exercised when v6 is absent. Two consequences: reasoning about "CGNAT
traversal" applies to the v4 fallback only; and the re-resolve watchdog must not silently flip
families mid-life in a way that strands a half-open v6 mapping. `AI_ADDRCONFIG` already gives the
safe default — a v4-only box never sees the AAAA — but a v6-capable box will prefer it. The
client MTU floor (§4.3) is deliberately chosen **family-agnostic (1280)** *so that* MTU
correctness does not depend on this resolution outcome. A **deterministic v4-pin** — the agent
resolving ep0's A record and writing the literal IP into the conf (a natural extension of this
watchdog) — remains an **open, separate determinism question**; it is **not required for MTU
correctness** and is tracked on its own, not built by S3.1.
safe default — a v4-only box never sees the AAAA — but a v6-capable box would prefer it. The
client MTU floor (§4.3) is family-agnostic (1280) so MTU correctness never depended on this;
and the **v4-pin above (v0.66.0) now resolves it outright** — the agent writes the A literal, so
the tunnel always takes the v4/NAT path regardless of the box's v6 capability. (Was tracked as an
open determinism question through S3.1; closed by S4's v4-pin.)
- **4.3 MTU/MSS — DECIDED: fleet-wide client MTU 1280 (agent v0.65.0, S3.1).** WG costs 60 bytes
on v4 / 80 on IPv6. The unit ships explicit **`MTU 1280`** — the **IPv6-minimum link MTU** (RFC
8200 guarantees every path carries ≥ 1280), so outer = 1280+60 = 1340 (v4) / 1280+80 = 1360 (v6),
@@ -281,7 +295,7 @@ vantage (Peti VM 110) remains the thorough confirmation but no longer gates anyt
| **S1 — endpoint provisioning + peer-sync surface** — **SHIPPED 2026-07-04** (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: **hub pushes over SSH** (`x/crypto/ssh`, `FixedHostKey` pin + pinned-type `HostKeyAlgorithms`) to a forced-command reconcile script; runbook `documentation/runbooks/offsite-endpoint.md`. Done-criterion met live: hub add/remove → `wg show` on the endpoint; tunnel `ep0.felhom.eu:443` → PBS through the wg0-only 8007 rule. | ~~provision the production VM…~~ (see runbook; production endpoint = a later re-run) | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓ |
| **S2 — hub schema + desired-state** — **SHIPPED 2026-07-04** (hub v0.33.0). `POST /hosts/{id}/wg` (self-scoped) + merge-at-read `wireguard` block (§3.2 shape = the golden) + one-per-host index + re-key-in-place + `/offsite` UI page. Live-proven against the real felhom-pve record incl. old-agent (v0.63.0) tolerance. | ~~endpoint coords + peer registry…~~ | a registered box's desired-state carries its assignment ✓; hub UI shows the peer registry ✓ |
| **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) |
| **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) |
| **S4 — PBS over the tunnel — SHIPPED 2026-07-04** (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep `DatastoreBackup` token, dual-grant user+token on `/datastore/<ds>/<ns>`, cross-tenant 403); box `felhom-offsite` PVE storage `Server=10.77.0.1`, box-born encryption key, token-secret WARN fix, `local_backup_target` retarget; agent **v0.66.0** v4-pin + re-resolve watchdog + FELHOM_WG Critical, **v0.67.0** namespace-aware PBS client. | ~~point the box's PBS storage at `pbs_tunnel_ip:8007`~~ | real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: restore-test scheduler WAN deadline + scratch `VM.Allocate`; Tier-1-local-vs-Tier-2-offsite target split (`local_backup_target` is currently a single target). |
| **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* |
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |