felhom-agent/REPORT.md

# REPORT — S4: PBS over the tunnel (agent v0.65→v0.67 + live tenancy on felhom-hetzner)

**Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (box) +
felhom-hetzner (endpoint). **Design of record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md`
§3.4 (SHIPPED), §4.2 (v4-pin SHIPPED), S4 roadmap row; runbook `offsite-endpoint.md` §4a/§4b.

## 1. Baselines & commits (felhom-agent, all pushed to `main`)

Start: `c618fc6` (v0.65.0, published+vouched).
- **`734f45c`** v0.66.0 — wgtunnel v4-pin + re-resolve watchdog; FELHOM_WG Critical flips.
- **`027948b`** v0.67.0 — namespace-aware PBS client (operator-approved, see §3).
- this commit — CHANGELOG (0.66/0.67 above) + CONTEXT + this REPORT.
felhom.eu docs: `1907dab` (runbook §4a/§4b + doc-06 §3.4/§4.2/S4 row).

## 2. Agent code (Parts 1+2 + the approved ns-aware addition)

- **v4-pin (doc §4.2, `internal/wgtunnel`).** `renderConf` takes the pre-resolved IPv4 literal →
  `Endpoint = <ip>:<port>` (never the DNS name, never AAAA). `Resolver` seam (`LookupNetIP "ip4"`,
  lowest address = deterministic fleet-wide); renderConf stays pure; resolved IP cached →
  steady-state Apply = zero DNS + zero execs. DNS failure keeps last conf (never a teardown).
- **Re-resolve watchdog (`Manager.Watchdog`, loop-only).** Handshake age > `stale_after_seconds`
  (default 180) → re-resolve; IP changed → re-render+restart (endpoint re-IP recovery); same → no
  churn (throttled). Reuses `wg show latest-handshakes` (never `dump`).
- **Capability Critical flips.** `wg-conf-install`/`wg-enable`/`wg-restart`/`wg-handshake-read` →
  Critical=true (backups ride the tunnel now); apt-install + disable stay non-critical.
- **Namespace-aware PBS client (v0.67.0, `internal/pbs`).** Config.Namespace → `Snapshots` passes
  `?ns=`, `Verify` passes `ns=`. Root-ns clients unchanged. Why: Phase-1 found the ns-unaware
  datastore-root calls 403 for a per-tenant token, and whole-datastore verify needed
  `Datastore.Verify` (~admin, breaking isolation). `proxmox.Storage.Namespace` parsed from
  `/storage`; `pbsTargetsFromPVE` threads it. Green gate + red-proofs (v4-pin golden, watchdog
  negatives, capability set, ns-scoping) all fire.

## 3. Phase-1 endpoint tenancy probe (felhom-hetzner) — the empirical spike

Created ns `demo-felhom-01` + privsep token `felhom@pbs!demo-felhom-01`; adversarial second tenant.
**Confirmed minimal ACL:** `DatastoreBackup` on `/datastore/felhom-offsite/<ns>` (namespace path,
**NOT** `/ns/<ns>`) granted to **BOTH** the user `felhom@pbs` **and** the token — PBS privsep =
intersection(user, token); isolation holds (cross-ns list/backup → **403**, proven). `DatastoreBackup`
covers backup/list-own/restore-own **and ns-scoped verify** (no `Datastore.Verify`/admin), but
**not** prune/forget (`Datastore.Modify|Prune`) — the box can't delete its own backups (safety).
Probe tenants cleaned up; only the real `demo-felhom-01` tenant remains.

## 4. Box-side + live Scenario-D validation (felhom-pve)

- `felhom-offsite` PVE storage (`Server=10.77.0.1` over the tunnel, box-born encryption key,
  argv-safe `.pw`/`.enc`+storage.cfg direct-write) → `pvesm status` active. Agent PVE token granted
  `FelhomAgentStore` on the new storage (dual-grant). WARN fix: `pbs-secrets/` (0700 felhom-agent)
  with both `.pw` copies + `pbs_secret_dir` → **zero WARN**, both datastores in the verify cycle.
- **Scenario D — ALL GREEN:** a real vzdump of guest 9201 (crash-consistent, ~4.6 GB compressed,
  ~20 min over the tunnel) landed as **ciphertext** in ns `demo-felhom-01` (`root.pxar` crypt-mode
  `encrypt`); the agent report **listed** it (ns-aware `Snapshots`); the ns-scoped **verify ran
  under the box's own `felhom@pbs!demo-felhom-01` token and returned `ok`** (no admin, no 403); the
  WARN grep = 0; a **restore round-tripped** — restored from offsite, **decrypted with the box-born
  key** (`b0:fe:2a:4e…`), **booted** (running, systemd, uptime 14 s), torn down clean.

## 5. NOT yet live-validated / findings / observations

- **The restore-test SCHEDULER couldn't complete the offsite restore-test** (the round-trip was
  proven MANUALLY as root instead). Two box-side follow-ups: (a) its internal restore-task deadline
  (~10 min) is too short for a WAN restore of a large volume; (b) the agent PVE token lacks
  `VM.Allocate` on the scratch VMID band for teardown. Raise the deadline for the `pbs` tier + grant
  the scratch-band `VM.Allocate` before the offsite restore-test can run unattended.
- **Retarget reverted.** The controller triggers a full backup ~every 30 min; with a SINGLE backup
  target (`local_backup_target`) retargeted to `felhom-offsite`, each becomes a ~20-min offsite
  upload — near-continuous, uplink-saturating. Reverted to `local`. **Offsite-as-default needs the
  Tier-1(local, frequent)/Tier-2(offsite, periodic) split** — the single-target model can't be the
  default. The offsite storage + tenancy stay live for scheduled/on-demand offsite backups.
- **Spec field-name correction:** the retarget field is `backup.local_backup_target` (not
  `backup_target`); `defaultBackupTarget` (`felhom-pbs`) unchanged in code.
- Deferred: DR consume of the bundle incl. WG key + offsite restore with only R (S5); hub-driven
  tenancy automation (hub creates ns+token, serves `pbs_namespace` on desired-state); endpoint
  datastore sizing for real volumes; true-CGNAT-SIM retest (parked).
- **OPERATOR-PRESENT (Part 5, NOT run):** escrow-create of the offsite token + tunnel token + WG
  key mints a NEW recovery code `R` the operator must record — deliberately left for an interactive
  operator run (never `--upload` a blob whose R nobody recorded). OPERATOR: publish 0.66/0.67 +
  Day-0 vouch. Rollback binaries on the box: `felhom-agent.bak-0.65.0` / `.bak-0.66.0`.