docs: REPORT + CONTEXT for S4 (PBS over the tunnel, live-validated)

Scenario-D green: real vzdump of 9201 → ciphertext in ns over the tunnel;
ns-scoped verify=ok under the box's own DatastoreBackup token; WARN gone;
restore round-tripped (decrypt with box-born key → boot → teardown).

Findings recorded: retarget field is local_backup_target (reverted to local —
single-target offsite is near-continuous given the ~30-min controller cadence;
needs Tier-1/Tier-2 split); restore-test scheduler needs a WAN deadline +
scratch VM.Allocate; escrow left as an operator-present step.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

CONTEXT.md

@@ -5,6 +5,24 @@
 
 ## Current
 
+- **v0.66.0 + v0.67.0** (2026-07-04, live on felhom-pve) — **S4: PBS over the tunnel**. **v0.66.0**:
+  wgtunnel **v4-pin** (renderConf writes the resolved A LITERAL, never DNS/AAAA; `Resolver` seam,
+  lowest addr; cached → steady-state zero-DNS/zero-exec) + **re-resolve watchdog** (`Manager.Watchdog`,
+  loop-only; handshake stale > `stale_after_seconds`=180 → re-resolve → IP-changed re-render+restart)
+  + FELHOM_WG **Critical** flips (conf-install/enable/restart/handshake-read). **v0.67.0**:
+  **namespace-aware PBS client** (Config.Namespace → `Snapshots ?ns=`, `Verify ns=`; root-ns
+  unchanged) — the operator-approved fix after Phase-1 showed the ns-unaware datastore-root 403s a
+  per-tenant token. **Live Scenario-D (all green):** real vzdump of 9201 → **ciphertext** in ns
+  `demo-felhom-01` over the tunnel; ns-scoped verify=ok under the box's own `felhom@pbs!demo-felhom-01`
+  token; WARN gone; restore round-tripped (decrypt with box-born key → boot → teardown).
+  **Confirmed tenant ACL (felhom-hetzner):** `DatastoreBackup` on `/datastore/felhom-offsite/<ns>`
+  (NOT `/ns/<ns>`) to BOTH user `felhom@pbs` AND token (privsep=intersection; cross-ns 403);
+  DatastoreBackup can't prune (safety). **FINDINGS:** retarget field is `local_backup_target` (not
+  `backup_target`); retarget REVERTED to `local` (controller backs up ~every 30 min → single-target
+  offsite = near-continuous 20-min uploads; needs Tier-1/Tier-2 split); restore-test scheduler needs
+  a WAN restore deadline + scratch-band `VM.Allocate` before it runs offsite unattended. **OPEN:**
+  escrow-create (OPERATOR-PRESENT, new R); publish 0.66/0.67 + Day-0 vouch; S5 DR consume. Rollback:
+  `felhom-agent.bak-0.65.0`/`.bak-0.66.0`. Detail: REPORT.md + doc-06 §3.4/§4.2 + runbook §4a/§4b.
 - **v0.65.0** (2026-07-04, live on felhom-pve) — **S3.1 offsite-tunnel client MTU 1420 → 1280**:
   resolves `06 §4.3`'s OPEN DECISION left by the CGNAT smoke test. 1420 **silently black-holed bulk
   TCP** on sub-~1480 paths (mobile ~1400, DS-Lite ~1452) — handshake+ping healthy, PBS TLS page

REPORT.md

@@ -1,76 +1,76 @@
-# REPORT — S3.1: offsite-tunnel client MTU 1420 → 1280 (v0.65.0)
+# REPORT — S4: PBS over the tunnel (agent v0.65→v0.67 + live tenancy on felhom-hetzner)
 
-**Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (the expendable
-demo box; Peti's box untouched; endpoint untouched by construction). **Design of record:**
-`felhom.eu/documentation/architecture/06-offsite-connectivity.md` §4.3 (OPEN DECISION → resolved).
+**Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (box) +
+felhom-hetzner (endpoint). **Design of record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md`
+§3.4 (SHIPPED), §4.2 (v4-pin SHIPPED), S4 roadmap row; runbook `offsite-endpoint.md` §4a/§4b.
 
-## 1. Baselines & commits
+## 1. Baselines & commits (felhom-agent, all pushed to `main`)
 
-Start: felhom-agent @ `03edb40` (v0.64.0); felhom.eu @ `2028785` (docs). Commits pushed to `main`:
+Start: `c618fc6` (v0.65.0, published+vouched).
+- **`734f45c`** v0.66.0 — wgtunnel v4-pin + re-resolve watchdog; FELHOM_WG Critical flips.
+- **`027948b`** v0.67.0 — namespace-aware PBS client (operator-approved, see §3).
+- this commit — CHANGELOG (0.66/0.67 above) + CONTEXT + this REPORT.
+felhom.eu docs: `1907dab` (runbook §4a/§4b + doc-06 §3.4/§4.2/S4 row).
 
-- **`312fd5e`** — `wgtunnel: client MTU 1420 → 1280` (manager.go const + render, manager_test.go
-  golden + red-proof note, report.go stale comment, CHANGELOG v0.65.0).
-- **`<this commit>`** — CONTEXT + this REPORT.
-- felhom.eu docs (§4.3 decision + §4.2/§4.3 wording + S6 row): separate one commit — see felhom.eu REPORT.
+## 2. Agent code (Parts 1+2 + the approved ns-aware addition)
 
-## 2. What changed & why
+- **v4-pin (doc §4.2, `internal/wgtunnel`).** `renderConf` takes the pre-resolved IPv4 literal →
+  `Endpoint = <ip>:<port>` (never the DNS name, never AAAA). `Resolver` seam (`LookupNetIP "ip4"`,
+  lowest address = deterministic fleet-wide); renderConf stays pure; resolved IP cached →
+  steady-state Apply = zero DNS + zero execs. DNS failure keeps last conf (never a teardown).
+- **Re-resolve watchdog (`Manager.Watchdog`, loop-only).** Handshake age > `stale_after_seconds`
+  (default 180) → re-resolve; IP changed → re-render+restart (endpoint re-IP recovery); same → no
+  churn (throttled). Reuses `wg show latest-handshakes` (never `dump`).
+- **Capability Critical flips.** `wg-conf-install`/`wg-enable`/`wg-restart`/`wg-handshake-read` →
+  Critical=true (backups ride the tunnel now); apt-install + disable stay non-critical.
+- **Namespace-aware PBS client (v0.67.0, `internal/pbs`).** Config.Namespace → `Snapshots` passes
+  `?ns=`, `Verify` passes `ns=`. Root-ns clients unchanged. Why: Phase-1 found the ns-unaware
+  datastore-root calls 403 for a per-tenant token, and whole-datastore verify needed
+  `Datastore.Verify` (~admin, breaking isolation). `proxmox.Storage.Namespace` parsed from
+  `/storage`; `pbsTargetsFromPVE` threads it. Green gate + red-proofs (v4-pin golden, watchdog
+  negatives, capability set, ns-scoping) all fire.
 
-The 2026-07-04 CGNAT smoke test found the shipped interface MTU **1420** silently **black-holes
-bulk TCP** on any path below ~1480 B (mobile ~1400, DS-Lite ~1452): the WG handshake and ping stay
-healthy (small packets) while the PBS TLS page — and, at S4, the backup itself — drops. Resolved
-`§4.3`'s OPEN DECISION with a **permanent, fleet-wide, family-agnostic client MTU of 1280** (RFC
-8200 IPv6-minimum link MTU; outer = 1280+60 v4 / +80 v6, fits every realistic path).
+## 3. Phase-1 endpoint tenancy probe (felhom-hetzner) — the empirical spike
 
-- `internal/wgtunnel/manager.go`: new `const clientMTU = 1280` (single home, doc-referenced);
-  `renderConf` emits `MTU = %d` from it.
-- **Client-only by construction:** interface MTU caps box→PBS; advertised MSS (=MTU−40) caps
-  PBS→box. The endpoint's `wg0` is **untouched** → zero live-endpoint risk, one constant changed.
-- Rejected (recorded in §4.3): auto-probe / per-connection-type (fragile moving part for a
-  non-metric); MSS-clamp (no forwarded flows). No wire/JSON-golden change (MTU is client-derived).
+Created ns `demo-felhom-01` + privsep token `felhom@pbs!demo-felhom-01`; adversarial second tenant.
+**Confirmed minimal ACL:** `DatastoreBackup` on `/datastore/felhom-offsite/<ns>` (namespace path,
+**NOT** `/ns/<ns>`) granted to **BOTH** the user `felhom@pbs` **and** the token — PBS privsep =
+intersection(user, token); isolation holds (cross-ns list/backup → **403**, proven). `DatastoreBackup`
+covers backup/list-own/restore-own **and ns-scoped verify** (no `Datastore.Verify`/admin), but
+**not** prune/forget (`Datastore.Modify|Prune`) — the box can't delete its own backups (safety).
+Probe tenants cleaned up; only the real `demo-felhom-01` tenant remains.
 
-## 3. Tests — render golden + red-proof (§10)
+## 4. Box-side + live Scenario-D validation (felhom-pve)
 
-- `go build ./... && go vet ./... && go test ./...` — **all green** (local Go 1.26).
-- `TestRenderConf_Golden` pins the exact bytes incl. `MTU = 1280` (not a "contains MTU" check).
-- **Red-proof:** flipped `const clientMTU` → 1420 → the golden **FAILED** on the MTU line
-  (`got MTU = 1420 / want MTU = 1280`); restored to 1280 → **PASS**. The assertion is non-vacuous.
+- `felhom-offsite` PVE storage (`Server=10.77.0.1` over the tunnel, box-born encryption key,
+  argv-safe `.pw`/`.enc`+storage.cfg direct-write) → `pvesm status` active. Agent PVE token granted
+  `FelhomAgentStore` on the new storage (dual-grant). WARN fix: `pbs-secrets/` (0700 felhom-agent)
+  with both `.pw` copies + `pbs_secret_dir` → **zero WARN**, both datastores in the verify cycle.
+- **Scenario D — ALL GREEN:** a real vzdump of guest 9201 (crash-consistent, ~4.6 GB compressed,
+  ~20 min over the tunnel) landed as **ciphertext** in ns `demo-felhom-01` (`root.pxar` crypt-mode
+  `encrypt`); the agent report **listed** it (ns-aware `Snapshots`); the ns-scoped **verify ran
+  under the box's own `felhom@pbs!demo-felhom-01` token and returned `ok`** (no admin, no 403); the
+  WARN grep = 0; a **restore round-tripped** — restored from offsite, **decrypted with the box-born
+  key** (`b0:fe:2a:4e…`), **booted** (running, systemd, uptime 14 s), torn down clean.
 
-## 4. Deploy + live wired smoke (§13) — felhom-pve, agent v0.65.0
+## 5. NOT yet live-validated / findings / observations
 
-Built `-X main.version=0.65.0` on 180; deployed with `felhom-agent.bak-0.64.0` backup; `--version`
-confirms **0.65.0**. Agent re-rendered + re-applied the conf on restart (hash-gated apply caught
-the MTU change):
-
-1. **Rendered MTU:** `grep MTU /etc/wireguard/wg-felhom.conf` → `MTU = 1280`; live iface
-   `ip link show wg-felhom` → `mtu 1280`. ✅
-2. **Tunnel:** `wg-quick@wg-felhom` active; handshake age 43 s; `ping 10.77.0.1` 3/3, ~33 ms. ✅
-3. **PBS page (no regression on the good path):** `curl -sk https://10.77.0.1:8007/` → full
-   `<!DOCTYPE html>…` login page at MTU 1280. ✅
-4. Journal clean — no capability degradation, no errors.
-
-Scenario B (no regression at 1280 on the working wired path) **proven live**. The constrained-path
-fix itself is established by arithmetic + prior field data (1280 < the 1340 that already worked in
-the smoke test; 1420 was the value that failed) — the optional phone re-tether re-proof was **not
-run** (not required per §13; skippable).
-
-## 5. Not yet live-validated / open
-
-- **True-CGNAT-SIM retest** — low risk, deferred ("when able"); the smoke SIM had a public mobile
-  IPv4, so 100.64/10 double-NAT was not reproduced (unchanged by this task).
-- **PBS backup over the tunnel (S4)** — the next slice; this fix makes its bulk-TCP floor safe.
-- **Optional constrained-path re-proof** (phone tether at MTU 1280) — skipped; available if extra
-  confidence wanted.
-
-## 6. Observations
-
-- The v4-pin (agent resolves ep0's A record + writes the literal, per §4.2's watchdog) is a
-  reasonable **later determinism** improvement but is **not needed for MTU correctness** — 1280 is
-  safe on both families' overhead. Recorded in the doc as a separate future note; green-lightable
-  independently whenever the re-resolve watchdog work happens.
-- ~~OPERATOR follow-up: publish 0.65.0 + Day-0 vouch~~ **DONE same day (operator-assisted browser
-  session):** 0.65.0 published to Gitea (`publish-agent.sh` on felhom-pve with the out-of-band
-  `/root/.golden-secrets.env` creds; sha256 `ae282669…78d0a73`, round-trip verified) and the hub
-  Day-0 manifest vouched **agent 0.65.0 + golden 0.98.3** via the `/configuration` UI in the
-  user's logged-in Chrome session (flash `artifacts_set` + reload-persist verified). `/offsite`
-  page eyeballed OK (endpoint + the one demo peer); Hosts page reports demo-felhom-01 agent
-  0.65.0 ONLINE. Rollback artifact on the box: `felhom-agent.bak-0.64.0`.
+- **The restore-test SCHEDULER couldn't complete the offsite restore-test** (the round-trip was
+  proven MANUALLY as root instead). Two box-side follow-ups: (a) its internal restore-task deadline
+  (~10 min) is too short for a WAN restore of a large volume; (b) the agent PVE token lacks
+  `VM.Allocate` on the scratch VMID band for teardown. Raise the deadline for the `pbs` tier + grant
+  the scratch-band `VM.Allocate` before the offsite restore-test can run unattended.
+- **Retarget reverted.** The controller triggers a full backup ~every 30 min; with a SINGLE backup
+  target (`local_backup_target`) retargeted to `felhom-offsite`, each becomes a ~20-min offsite
+  upload — near-continuous, uplink-saturating. Reverted to `local`. **Offsite-as-default needs the
+  Tier-1(local, frequent)/Tier-2(offsite, periodic) split** — the single-target model can't be the
+  default. The offsite storage + tenancy stay live for scheduled/on-demand offsite backups.
+- **Spec field-name correction:** the retarget field is `backup.local_backup_target` (not
+  `backup_target`); `defaultBackupTarget` (`felhom-pbs`) unchanged in code.
+- Deferred: DR consume of the bundle incl. WG key + offsite restore with only R (S5); hub-driven
+  tenancy automation (hub creates ns+token, serves `pbs_namespace` on desired-state); endpoint
+  datastore sizing for real volumes; true-CGNAT-SIM retest (parked).
+- **OPERATOR-PRESENT (Part 5, NOT run):** escrow-create of the offsite token + tunnel token + WG
+  key mints a NEW recovery code `R` the operator must record — deliberately left for an interactive
+  operator run (never `--upload` a blob whose R nobody recorded). OPERATOR: publish 0.66/0.67 +
+  Day-0 vouch. Rollback binaries on the box: `felhom-agent.bak-0.65.0` / `.bak-0.66.0`.