felhom-agent/REPORT.md

REPORT — S4: PBS over the tunnel (agent v0.65→v0.67 + live tenancy on felhom-hetzner)

Date: 2026-07-04 · Class: implementation + live validation on felhom-pve (box) + felhom-hetzner (endpoint). Design of record: felhom.eu/documentation/architecture/06-offsite-connectivity.md §3.4 (SHIPPED), §4.2 (v4-pin SHIPPED), S4 roadmap row; runbook offsite-endpoint.md §4a/§4b.

1. Baselines & commits (felhom-agent, all pushed to main)

Start: c618fc6 (v0.65.0, published+vouched).

• 734f45c v0.66.0 — wgtunnel v4-pin + re-resolve watchdog; FELHOM_WG Critical flips.
• 027948b v0.67.0 — namespace-aware PBS client (operator-approved, see §3).
• this commit — CHANGELOG (0.66/0.67 above) + CONTEXT + this REPORT. felhom.eu docs: 1907dab (runbook §4a/§4b + doc-06 §3.4/§4.2/S4 row).

2. Agent code (Parts 1+2 + the approved ns-aware addition)

• v4-pin (doc §4.2, internal/wgtunnel). renderConf takes the pre-resolved IPv4 literal → Endpoint = <ip>:<port> (never the DNS name, never AAAA). Resolver seam (LookupNetIP "ip4", lowest address = deterministic fleet-wide); renderConf stays pure; resolved IP cached → steady-state Apply = zero DNS + zero execs. DNS failure keeps last conf (never a teardown).
• Re-resolve watchdog (Manager.Watchdog, loop-only). Handshake age > stale_after_seconds (default 180) → re-resolve; IP changed → re-render+restart (endpoint re-IP recovery); same → no churn (throttled). Reuses wg show latest-handshakes (never dump).
• Capability Critical flips. wg-conf-install/wg-enable/wg-restart/wg-handshake-read → Critical=true (backups ride the tunnel now); apt-install + disable stay non-critical.
• Namespace-aware PBS client (v0.67.0, internal/pbs). Config.Namespace → Snapshots passes ?ns=, Verify passes ns=. Root-ns clients unchanged. Why: Phase-1 found the ns-unaware datastore-root calls 403 for a per-tenant token, and whole-datastore verify needed Datastore.Verify (~admin, breaking isolation). proxmox.Storage.Namespace parsed from /storage; pbsTargetsFromPVE threads it. Green gate + red-proofs (v4-pin golden, watchdog negatives, capability set, ns-scoping) all fire.

3. Phase-1 endpoint tenancy probe (felhom-hetzner) — the empirical spike

Created ns demo-felhom-01 + privsep token felhom@pbs!demo-felhom-01; adversarial second tenant. Confirmed minimal ACL: DatastoreBackup on /datastore/felhom-offsite/<ns> (namespace path, NOT /ns/<ns>) granted to BOTH the user felhom@pbs and the token — PBS privsep = intersection(user, token); isolation holds (cross-ns list/backup → 403, proven). DatastoreBackup covers backup/list-own/restore-own and ns-scoped verify (no Datastore.Verify/admin), but not prune/forget (Datastore.Modify|Prune) — the box can't delete its own backups (safety). Probe tenants cleaned up; only the real demo-felhom-01 tenant remains.

4. Box-side + live Scenario-D validation (felhom-pve)

• felhom-offsite PVE storage (Server=10.77.0.1 over the tunnel, box-born encryption key, argv-safe .pw/.enc+storage.cfg direct-write) → pvesm status active. Agent PVE token granted FelhomAgentStore on the new storage (dual-grant). WARN fix: pbs-secrets/ (0700 felhom-agent) with both .pw copies + pbs_secret_dir → zero WARN, both datastores in the verify cycle.
• Scenario D — ALL GREEN: a real vzdump of guest 9201 (crash-consistent, ~4.6 GB compressed, ~20 min over the tunnel) landed as ciphertext in ns demo-felhom-01 (root.pxar crypt-mode encrypt); the agent report listed it (ns-aware Snapshots); the ns-scoped verify ran under the box's own felhom@pbs!demo-felhom-01 token and returned ok (no admin, no 403); the WARN grep = 0; a restore round-tripped — restored from offsite, decrypted with the box-born key (b0:fe:2a:4e…), booted (running, systemd, uptime 14 s), torn down clean.

5. NOT yet live-validated / findings / observations

• The restore-test SCHEDULER couldn't complete the offsite restore-test (the round-trip was proven MANUALLY as root instead). Two box-side follow-ups: (a) its internal restore-task deadline (~10 min) is too short for a WAN restore of a large volume; (b) the agent PVE token lacks VM.Allocate on the scratch VMID band for teardown. Raise the deadline for the pbs tier + grant the scratch-band VM.Allocate before the offsite restore-test can run unattended.
• Retarget reverted. The controller triggers a full backup ~every 30 min; with a SINGLE backup target (local_backup_target) retargeted to felhom-offsite, each becomes a ~20-min offsite upload — near-continuous, uplink-saturating. Reverted to local. Offsite-as-default needs the Tier-1(local, frequent)/Tier-2(offsite, periodic) split — the single-target model can't be the default. The offsite storage + tenancy stay live for scheduled/on-demand offsite backups.
• Spec field-name correction: the retarget field is backup.local_backup_target (not backup_target); defaultBackupTarget (felhom-pbs) unchanged in code.
• Deferred: DR consume of the bundle incl. WG key + offsite restore with only R (S5); hub-driven tenancy automation (hub creates ns+token, serves pbs_namespace on desired-state); endpoint datastore sizing for real volumes; true-CGNAT-SIM retest (parked).
• OPERATOR-PRESENT (Part 5, NOT run): escrow-create of the offsite token + tunnel token + WG key mints a NEW recovery code R the operator must record — deliberately left for an interactive operator run (never --upload a blob whose R nobody recorded). OPERATOR: publish 0.66/0.67 + Day-0 vouch. Rollback binaries on the box: felhom-agent.bak-0.65.0 / .bak-0.66.0.