From fee1fcfccd663839eaa2d3699f4102a68879abf9 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 4 Jul 2026 17:25:49 +0200 Subject: [PATCH] docs: REPORT + CONTEXT for S4 (PBS over the tunnel, live-validated) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scenario-D green: real vzdump of 9201 → ciphertext in ns over the tunnel; ns-scoped verify=ok under the box's own DatastoreBackup token; WARN gone; restore round-tripped (decrypt with box-born key → boot → teardown). Findings recorded: retarget field is local_backup_target (reverted to local — single-target offsite is near-continuous given the ~30-min controller cadence; needs Tier-1/Tier-2 split); restore-test scheduler needs a WAN deadline + scratch VM.Allocate; escrow left as an operator-present step. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CONTEXT.md | 18 ++++++++ REPORT.md | 130 ++++++++++++++++++++++++++--------------------------- 2 files changed, 83 insertions(+), 65 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 1f8a08b..8eb5462 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -5,6 +5,24 @@ ## Current +- **v0.66.0 + v0.67.0** (2026-07-04, live on felhom-pve) — **S4: PBS over the tunnel**. **v0.66.0**: + wgtunnel **v4-pin** (renderConf writes the resolved A LITERAL, never DNS/AAAA; `Resolver` seam, + lowest addr; cached → steady-state zero-DNS/zero-exec) + **re-resolve watchdog** (`Manager.Watchdog`, + loop-only; handshake stale > `stale_after_seconds`=180 → re-resolve → IP-changed re-render+restart) + + FELHOM_WG **Critical** flips (conf-install/enable/restart/handshake-read). **v0.67.0**: + **namespace-aware PBS client** (Config.Namespace → `Snapshots ?ns=`, `Verify ns=`; root-ns + unchanged) — the operator-approved fix after Phase-1 showed the ns-unaware datastore-root 403s a + per-tenant token. **Live Scenario-D (all green):** real vzdump of 9201 → **ciphertext** in ns + `demo-felhom-01` over the tunnel; ns-scoped verify=ok under the box's own `felhom@pbs!demo-felhom-01` + token; WARN gone; restore round-tripped (decrypt with box-born key → boot → teardown). + **Confirmed tenant ACL (felhom-hetzner):** `DatastoreBackup` on `/datastore/felhom-offsite/` + (NOT `/ns/`) to BOTH user `felhom@pbs` AND token (privsep=intersection; cross-ns 403); + DatastoreBackup can't prune (safety). **FINDINGS:** retarget field is `local_backup_target` (not + `backup_target`); retarget REVERTED to `local` (controller backs up ~every 30 min → single-target + offsite = near-continuous 20-min uploads; needs Tier-1/Tier-2 split); restore-test scheduler needs + a WAN restore deadline + scratch-band `VM.Allocate` before it runs offsite unattended. **OPEN:** + escrow-create (OPERATOR-PRESENT, new R); publish 0.66/0.67 + Day-0 vouch; S5 DR consume. Rollback: + `felhom-agent.bak-0.65.0`/`.bak-0.66.0`. Detail: REPORT.md + doc-06 §3.4/§4.2 + runbook §4a/§4b. - **v0.65.0** (2026-07-04, live on felhom-pve) — **S3.1 offsite-tunnel client MTU 1420 → 1280**: resolves `06 §4.3`'s OPEN DECISION left by the CGNAT smoke test. 1420 **silently black-holed bulk TCP** on sub-~1480 paths (mobile ~1400, DS-Lite ~1452) — handshake+ping healthy, PBS TLS page diff --git a/REPORT.md b/REPORT.md index 7770bf7..28a01f5 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,76 +1,76 @@ -# REPORT — S3.1: offsite-tunnel client MTU 1420 → 1280 (v0.65.0) +# REPORT — S4: PBS over the tunnel (agent v0.65→v0.67 + live tenancy on felhom-hetzner) -**Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (the expendable -demo box; Peti's box untouched; endpoint untouched by construction). **Design of record:** -`felhom.eu/documentation/architecture/06-offsite-connectivity.md` §4.3 (OPEN DECISION → resolved). +**Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (box) + +felhom-hetzner (endpoint). **Design of record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md` +§3.4 (SHIPPED), §4.2 (v4-pin SHIPPED), S4 roadmap row; runbook `offsite-endpoint.md` §4a/§4b. -## 1. Baselines & commits +## 1. Baselines & commits (felhom-agent, all pushed to `main`) -Start: felhom-agent @ `03edb40` (v0.64.0); felhom.eu @ `2028785` (docs). Commits pushed to `main`: +Start: `c618fc6` (v0.65.0, published+vouched). +- **`734f45c`** v0.66.0 — wgtunnel v4-pin + re-resolve watchdog; FELHOM_WG Critical flips. +- **`027948b`** v0.67.0 — namespace-aware PBS client (operator-approved, see §3). +- this commit — CHANGELOG (0.66/0.67 above) + CONTEXT + this REPORT. +felhom.eu docs: `1907dab` (runbook §4a/§4b + doc-06 §3.4/§4.2/S4 row). -- **`312fd5e`** — `wgtunnel: client MTU 1420 → 1280` (manager.go const + render, manager_test.go - golden + red-proof note, report.go stale comment, CHANGELOG v0.65.0). -- **``** — CONTEXT + this REPORT. -- felhom.eu docs (§4.3 decision + §4.2/§4.3 wording + S6 row): separate one commit — see felhom.eu REPORT. +## 2. Agent code (Parts 1+2 + the approved ns-aware addition) -## 2. What changed & why +- **v4-pin (doc §4.2, `internal/wgtunnel`).** `renderConf` takes the pre-resolved IPv4 literal → + `Endpoint = :` (never the DNS name, never AAAA). `Resolver` seam (`LookupNetIP "ip4"`, + lowest address = deterministic fleet-wide); renderConf stays pure; resolved IP cached → + steady-state Apply = zero DNS + zero execs. DNS failure keeps last conf (never a teardown). +- **Re-resolve watchdog (`Manager.Watchdog`, loop-only).** Handshake age > `stale_after_seconds` + (default 180) → re-resolve; IP changed → re-render+restart (endpoint re-IP recovery); same → no + churn (throttled). Reuses `wg show latest-handshakes` (never `dump`). +- **Capability Critical flips.** `wg-conf-install`/`wg-enable`/`wg-restart`/`wg-handshake-read` → + Critical=true (backups ride the tunnel now); apt-install + disable stay non-critical. +- **Namespace-aware PBS client (v0.67.0, `internal/pbs`).** Config.Namespace → `Snapshots` passes + `?ns=`, `Verify` passes `ns=`. Root-ns clients unchanged. Why: Phase-1 found the ns-unaware + datastore-root calls 403 for a per-tenant token, and whole-datastore verify needed + `Datastore.Verify` (~admin, breaking isolation). `proxmox.Storage.Namespace` parsed from + `/storage`; `pbsTargetsFromPVE` threads it. Green gate + red-proofs (v4-pin golden, watchdog + negatives, capability set, ns-scoping) all fire. -The 2026-07-04 CGNAT smoke test found the shipped interface MTU **1420** silently **black-holes -bulk TCP** on any path below ~1480 B (mobile ~1400, DS-Lite ~1452): the WG handshake and ping stay -healthy (small packets) while the PBS TLS page — and, at S4, the backup itself — drops. Resolved -`§4.3`'s OPEN DECISION with a **permanent, fleet-wide, family-agnostic client MTU of 1280** (RFC -8200 IPv6-minimum link MTU; outer = 1280+60 v4 / +80 v6, fits every realistic path). +## 3. Phase-1 endpoint tenancy probe (felhom-hetzner) — the empirical spike -- `internal/wgtunnel/manager.go`: new `const clientMTU = 1280` (single home, doc-referenced); - `renderConf` emits `MTU = %d` from it. -- **Client-only by construction:** interface MTU caps box→PBS; advertised MSS (=MTU−40) caps - PBS→box. The endpoint's `wg0` is **untouched** → zero live-endpoint risk, one constant changed. -- Rejected (recorded in §4.3): auto-probe / per-connection-type (fragile moving part for a - non-metric); MSS-clamp (no forwarded flows). No wire/JSON-golden change (MTU is client-derived). +Created ns `demo-felhom-01` + privsep token `felhom@pbs!demo-felhom-01`; adversarial second tenant. +**Confirmed minimal ACL:** `DatastoreBackup` on `/datastore/felhom-offsite/` (namespace path, +**NOT** `/ns/`) granted to **BOTH** the user `felhom@pbs` **and** the token — PBS privsep = +intersection(user, token); isolation holds (cross-ns list/backup → **403**, proven). `DatastoreBackup` +covers backup/list-own/restore-own **and ns-scoped verify** (no `Datastore.Verify`/admin), but +**not** prune/forget (`Datastore.Modify|Prune`) — the box can't delete its own backups (safety). +Probe tenants cleaned up; only the real `demo-felhom-01` tenant remains. -## 3. Tests — render golden + red-proof (§10) +## 4. Box-side + live Scenario-D validation (felhom-pve) -- `go build ./... && go vet ./... && go test ./...` — **all green** (local Go 1.26). -- `TestRenderConf_Golden` pins the exact bytes incl. `MTU = 1280` (not a "contains MTU" check). -- **Red-proof:** flipped `const clientMTU` → 1420 → the golden **FAILED** on the MTU line - (`got MTU = 1420 / want MTU = 1280`); restored to 1280 → **PASS**. The assertion is non-vacuous. +- `felhom-offsite` PVE storage (`Server=10.77.0.1` over the tunnel, box-born encryption key, + argv-safe `.pw`/`.enc`+storage.cfg direct-write) → `pvesm status` active. Agent PVE token granted + `FelhomAgentStore` on the new storage (dual-grant). WARN fix: `pbs-secrets/` (0700 felhom-agent) + with both `.pw` copies + `pbs_secret_dir` → **zero WARN**, both datastores in the verify cycle. +- **Scenario D — ALL GREEN:** a real vzdump of guest 9201 (crash-consistent, ~4.6 GB compressed, + ~20 min over the tunnel) landed as **ciphertext** in ns `demo-felhom-01` (`root.pxar` crypt-mode + `encrypt`); the agent report **listed** it (ns-aware `Snapshots`); the ns-scoped **verify ran + under the box's own `felhom@pbs!demo-felhom-01` token and returned `ok`** (no admin, no 403); the + WARN grep = 0; a **restore round-tripped** — restored from offsite, **decrypted with the box-born + key** (`b0:fe:2a:4e…`), **booted** (running, systemd, uptime 14 s), torn down clean. -## 4. Deploy + live wired smoke (§13) — felhom-pve, agent v0.65.0 +## 5. NOT yet live-validated / findings / observations -Built `-X main.version=0.65.0` on 180; deployed with `felhom-agent.bak-0.64.0` backup; `--version` -confirms **0.65.0**. Agent re-rendered + re-applied the conf on restart (hash-gated apply caught -the MTU change): - -1. **Rendered MTU:** `grep MTU /etc/wireguard/wg-felhom.conf` → `MTU = 1280`; live iface - `ip link show wg-felhom` → `mtu 1280`. ✅ -2. **Tunnel:** `wg-quick@wg-felhom` active; handshake age 43 s; `ping 10.77.0.1` 3/3, ~33 ms. ✅ -3. **PBS page (no regression on the good path):** `curl -sk https://10.77.0.1:8007/` → full - `…` login page at MTU 1280. ✅ -4. Journal clean — no capability degradation, no errors. - -Scenario B (no regression at 1280 on the working wired path) **proven live**. The constrained-path -fix itself is established by arithmetic + prior field data (1280 < the 1340 that already worked in -the smoke test; 1420 was the value that failed) — the optional phone re-tether re-proof was **not -run** (not required per §13; skippable). - -## 5. Not yet live-validated / open - -- **True-CGNAT-SIM retest** — low risk, deferred ("when able"); the smoke SIM had a public mobile - IPv4, so 100.64/10 double-NAT was not reproduced (unchanged by this task). -- **PBS backup over the tunnel (S4)** — the next slice; this fix makes its bulk-TCP floor safe. -- **Optional constrained-path re-proof** (phone tether at MTU 1280) — skipped; available if extra - confidence wanted. - -## 6. Observations - -- The v4-pin (agent resolves ep0's A record + writes the literal, per §4.2's watchdog) is a - reasonable **later determinism** improvement but is **not needed for MTU correctness** — 1280 is - safe on both families' overhead. Recorded in the doc as a separate future note; green-lightable - independently whenever the re-resolve watchdog work happens. -- ~~OPERATOR follow-up: publish 0.65.0 + Day-0 vouch~~ **DONE same day (operator-assisted browser - session):** 0.65.0 published to Gitea (`publish-agent.sh` on felhom-pve with the out-of-band - `/root/.golden-secrets.env` creds; sha256 `ae282669…78d0a73`, round-trip verified) and the hub - Day-0 manifest vouched **agent 0.65.0 + golden 0.98.3** via the `/configuration` UI in the - user's logged-in Chrome session (flash `artifacts_set` + reload-persist verified). `/offsite` - page eyeballed OK (endpoint + the one demo peer); Hosts page reports demo-felhom-01 agent - 0.65.0 ONLINE. Rollback artifact on the box: `felhom-agent.bak-0.64.0`. +- **The restore-test SCHEDULER couldn't complete the offsite restore-test** (the round-trip was + proven MANUALLY as root instead). Two box-side follow-ups: (a) its internal restore-task deadline + (~10 min) is too short for a WAN restore of a large volume; (b) the agent PVE token lacks + `VM.Allocate` on the scratch VMID band for teardown. Raise the deadline for the `pbs` tier + grant + the scratch-band `VM.Allocate` before the offsite restore-test can run unattended. +- **Retarget reverted.** The controller triggers a full backup ~every 30 min; with a SINGLE backup + target (`local_backup_target`) retargeted to `felhom-offsite`, each becomes a ~20-min offsite + upload — near-continuous, uplink-saturating. Reverted to `local`. **Offsite-as-default needs the + Tier-1(local, frequent)/Tier-2(offsite, periodic) split** — the single-target model can't be the + default. The offsite storage + tenancy stay live for scheduled/on-demand offsite backups. +- **Spec field-name correction:** the retarget field is `backup.local_backup_target` (not + `backup_target`); `defaultBackupTarget` (`felhom-pbs`) unchanged in code. +- Deferred: DR consume of the bundle incl. WG key + offsite restore with only R (S5); hub-driven + tenancy automation (hub creates ns+token, serves `pbs_namespace` on desired-state); endpoint + datastore sizing for real volumes; true-CGNAT-SIM retest (parked). +- **OPERATOR-PRESENT (Part 5, NOT run):** escrow-create of the offsite token + tunnel token + WG + key mints a NEW recovery code `R` the operator must record — deliberately left for an interactive + operator run (never `--upload` a blob whose R nobody recorded). OPERATOR: publish 0.66/0.67 + + Day-0 vouch. Rollback binaries on the box: `felhom-agent.bak-0.65.0` / `.bak-0.66.0`.