From 567cf9f4014679cefd18b6db313b968646e56cbe Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 4 Jul 2026 19:54:46 +0200 Subject: [PATCH] docs: REPORT + CONTEXT for S4.1 (tier-aware deadline + teardown phantom confirmed) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live diagnosis on the agent-token path: offsite restore-test pass:true, verified:boot+running, teardown succeeded (no 403), scratch band clean. The VM.Allocate follow-up was a PHANTOM — the earlier 403 was the 10m-timeout firing teardown against a not-yet-pool-associated guest; with the timeout fixed the guest is in /pool/felhom and teardown is authorized. No ACL/host-install change. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CONTEXT.md | 12 +++++ REPORT.md | 126 ++++++++++++++++++++++++++--------------------------- 2 files changed, 73 insertions(+), 65 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 8eb5462..73d46fb 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -5,6 +5,18 @@ ## Current +- **v0.68.0** (2026-07-04, live on felhom-pve) — **S4.1: unattended offsite restore-test**. + **Tier-aware restore-task deadline:** `RestoreTestSpec.RestoreTaskTimeout` (0→10m default) from + `config.RestoreTestPBSRestoreTimeoutSeconds` (accessor default **120m**), set only when + `SourceTier=="pbs"` (`main.restoreTaskTimeout`); local tier UNCHANGED. Fixes the WAN restore being + killed at 10m → mid-restore teardown → leaked scratch. **Teardown "VM.Allocate" follow-up = + PHANTOM (diagnosed, not blind-fixed):** ran the restore-test on the AGENT-TOKEN path sourcing the + offsite (pbs) backup → `pass:true verified:boot+running`, teardown succeeded (`torn down + vmid=990000`, no 403), scratch band clean. The earlier 403 was the 10m-timeout consequence (guest + not yet pool-associated); the scratch is restored INTO `/pool/felhom` (ACL already grants + VM.Allocate) so teardown is authorized once the restore completes. **No ACL/host-install change.** + OPEN: publish 0.68.0 + Day-0 vouch; Tier-1/Tier-2 split for offsite-as-default; S5 DR consume. + Rollback `felhom-agent.bak-0.67.0`. Detail: REPORT.md. - **v0.66.0 + v0.67.0** (2026-07-04, live on felhom-pve) — **S4: PBS over the tunnel**. **v0.66.0**: wgtunnel **v4-pin** (renderConf writes the resolved A LITERAL, never DNS/AAAA; `Resolver` seam, lowest addr; cached → steady-state zero-DNS/zero-exec) + **re-resolve watchdog** (`Manager.Watchdog`, diff --git a/REPORT.md b/REPORT.md index 28a01f5..b164a18 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,76 +1,72 @@ -# REPORT — S4: PBS over the tunnel (agent v0.65→v0.67 + live tenancy on felhom-hetzner) +# REPORT — S4.1: unattended offsite restore-test (tier-aware deadline + teardown diagnosis) -**Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (box) + -felhom-hetzner (endpoint). **Design of record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md` -§3.4 (SHIPPED), §4.2 (v4-pin SHIPPED), S4 roadmap row; runbook `offsite-endpoint.md` §4a/§4b. +**Date:** 2026-07-04 · **Class:** implementation + live diagnosis on felhom-pve (agent token). +**Design of record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md` S4 row; +runbook `offsite-endpoint.md` §4b. -## 1. Baselines & commits (felhom-agent, all pushed to `main`) +## 1. Baselines & commits (felhom-agent → `main`) -Start: `c618fc6` (v0.65.0, published+vouched). -- **`734f45c`** v0.66.0 — wgtunnel v4-pin + re-resolve watchdog; FELHOM_WG Critical flips. -- **`027948b`** v0.67.0 — namespace-aware PBS client (operator-approved, see §3). -- this commit — CHANGELOG (0.66/0.67 above) + CONTEXT + this REPORT. -felhom.eu docs: `1907dab` (runbook §4a/§4b + doc-06 §3.4/§4.2/S4 row). +Start: `fee1fcf` (v0.67.0). +- **`dc70e15`** v0.68.0 — tier-aware restore-task deadline (Part 1). +- this commit — REPORT + CONTEXT (the diagnosis outcome). -## 2. Agent code (Parts 1+2 + the approved ns-aware addition) +## 2. Part 1 — tier-aware restore-task deadline -- **v4-pin (doc §4.2, `internal/wgtunnel`).** `renderConf` takes the pre-resolved IPv4 literal → - `Endpoint = :` (never the DNS name, never AAAA). `Resolver` seam (`LookupNetIP "ip4"`, - lowest address = deterministic fleet-wide); renderConf stays pure; resolved IP cached → - steady-state Apply = zero DNS + zero execs. DNS failure keeps last conf (never a teardown). -- **Re-resolve watchdog (`Manager.Watchdog`, loop-only).** Handshake age > `stale_after_seconds` - (default 180) → re-resolve; IP changed → re-render+restart (endpoint re-IP recovery); same → no - churn (throttled). Reuses `wg show latest-handshakes` (never `dump`). -- **Capability Critical flips.** `wg-conf-install`/`wg-enable`/`wg-restart`/`wg-handshake-read` → - Critical=true (backups ride the tunnel now); apt-install + disable stay non-critical. -- **Namespace-aware PBS client (v0.67.0, `internal/pbs`).** Config.Namespace → `Snapshots` passes - `?ns=`, `Verify` passes `ns=`. Root-ns clients unchanged. Why: Phase-1 found the ns-unaware - datastore-root calls 403 for a per-tenant token, and whole-datastore verify needed - `Datastore.Verify` (~admin, breaking isolation). `proxmox.Storage.Namespace` parsed from - `/storage`; `pbsTargetsFromPVE` threads it. Green gate + red-proofs (v4-pin golden, watchdog - negatives, capability set, ns-scoping) all fire. +The restore-task wait (`restoretest.go` ~L246) used `WaitOptions{}` → the 10m default. A WAN (pbs) +restore of a large guest exceeds it, so the wait expired mid-restore and teardown then fired against +a still-restoring guest. Fix: +- `RestoreTestSpec.RestoreTaskTimeout` (0 → 10m default); the restore `WaitTask` passes it. +- `config.BackupConfig.RestoreTestPBSRestoreTimeoutSeconds` + accessor `RestoreTestPBSRestoreTimeout()` + (positive as-is, else **120m**). +- `main.restoreTaskTimeout(cfg, tier)` — the configured PBS timeout only when `SourceTier=="pbs"`, + else 0. Set at BOTH spec builds (scheduler + selftest). **Local tier unchanged (10m).** +- Tests: `WaitOptions.Timeout` pbs→120m / local→0 (red-proofed against `WaitOptions{}`: pbs + assertion fails); accessor contract (0/neg→120m, positive as-is). Full `go test ./...` green. -## 3. Phase-1 endpoint tenancy probe (felhom-hetzner) — the empirical spike +## 3. Part 2 — teardown diagnosis (the "grant VM.Allocate" follow-up) — PHANTOM CONFIRMED -Created ns `demo-felhom-01` + privsep token `felhom@pbs!demo-felhom-01`; adversarial second tenant. -**Confirmed minimal ACL:** `DatastoreBackup` on `/datastore/felhom-offsite/` (namespace path, -**NOT** `/ns/`) granted to **BOTH** the user `felhom@pbs` **and** the token — PBS privsep = -intersection(user, token); isolation holds (cross-ns list/backup → **403**, proven). `DatastoreBackup` -covers backup/list-own/restore-own **and ns-scoped verify** (no `Datastore.Verify`/admin), but -**not** prune/forget (`Datastore.Modify|Prune`) — the box can't delete its own backups (safety). -Probe tenants cleaned up; only the real `demo-felhom-01` tenant remains. +Ran the restore-test on the **agent-token path** (selftest = the identical `RunRestoreTest` the +scheduler runs; all PVE ops authenticate as `felhom-agent@pve!agent`), sourcing the **offsite +(pbs-tier)** backup of guest 9201, with v0.68.0's 120m timeout. -## 4. Box-side + live Scenario-D validation (felhom-pve) +**Scenario B (done-criterion) — GREEN:** +``` +source_archive: felhom-offsite:backup/ct/9201/2026-07-04T14:55:44Z +source_tier: pbs · pass: true · verified: boot+running · scratch_vmid: 990000 · duration 4m49s +restore → net-link-down → boot (running) → teardown → scratch band clean (no leak) +``` -- `felhom-offsite` PVE storage (`Server=10.77.0.1` over the tunnel, box-born encryption key, - argv-safe `.pw`/`.enc`+storage.cfg direct-write) → `pvesm status` active. Agent PVE token granted - `FelhomAgentStore` on the new storage (dual-grant). WARN fix: `pbs-secrets/` (0700 felhom-agent) - with both `.pw` copies + `pbs_secret_dir` → **zero WARN**, both datastores in the verify cycle. -- **Scenario D — ALL GREEN:** a real vzdump of guest 9201 (crash-consistent, ~4.6 GB compressed, - ~20 min over the tunnel) landed as **ciphertext** in ns `demo-felhom-01` (`root.pxar` crypt-mode - `encrypt`); the agent report **listed** it (ns-aware `Snapshots`); the ns-scoped **verify ran - under the box's own `felhom@pbs!demo-felhom-01` token and returned `ok`** (no admin, no 403); the - WARN grep = 0; a **restore round-tripped** — restored from offsite, **decrypted with the box-born - key** (`b0:fe:2a:4e…`), **booted** (running, systemd, uptime 14 s), torn down clean. +**Scenario C — the teardown succeeded as the agent token; (b) is a PHANTOM:** +``` +gate decision: class=guest_destroy guest=990000 allowed=true reason=benign +restore-test: scratch guest torn down vmid=990000 +pct list | grep 9900 → no scratch guests +``` +No 403, no `VM.Allocate` error. Root cause of the earlier 403 (REPORT of S4): the 10m wait expired +**mid-restore**, so teardown hit a guest that existed but was **not yet pool-associated** (the pool +membership lands with restore completion) — a bare `/vms/990000` the pool ACL didn't cover. With the +timeout fixed → the restore completes → the guest is a member of `/pool/felhom` (restored with +`Pool=DefaultPool`, `restoretest.go` L229) → its destroy is authorized by the existing +`FelhomAgentGuest` grant (`VM.Allocate` + `Pool.Allocate` at `/pool/felhom`). **No ACL change, no +host-install edit** — the pool-scoped containment (`SPIKE-pool-scoped-acl-2026-07-01`) stands. -## 5. NOT yet live-validated / findings / observations +(Note: this run's restore was fast — home *download* ≫ *upload*; the earlier ~10-min stall was link +contention. The 120m ceiling covers the slow case regardless — for an unattended nightly test a +false timeout is the failure mode to prevent.) -- **The restore-test SCHEDULER couldn't complete the offsite restore-test** (the round-trip was - proven MANUALLY as root instead). Two box-side follow-ups: (a) its internal restore-task deadline - (~10 min) is too short for a WAN restore of a large volume; (b) the agent PVE token lacks - `VM.Allocate` on the scratch VMID band for teardown. Raise the deadline for the `pbs` tier + grant - the scratch-band `VM.Allocate` before the offsite restore-test can run unattended. -- **Retarget reverted.** The controller triggers a full backup ~every 30 min; with a SINGLE backup - target (`local_backup_target`) retargeted to `felhom-offsite`, each becomes a ~20-min offsite - upload — near-continuous, uplink-saturating. Reverted to `local`. **Offsite-as-default needs the - Tier-1(local, frequent)/Tier-2(offsite, periodic) split** — the single-target model can't be the - default. The offsite storage + tenancy stay live for scheduled/on-demand offsite backups. -- **Spec field-name correction:** the retarget field is `backup.local_backup_target` (not - `backup_target`); `defaultBackupTarget` (`felhom-pbs`) unchanged in code. -- Deferred: DR consume of the bundle incl. WG key + offsite restore with only R (S5); hub-driven - tenancy automation (hub creates ns+token, serves `pbs_namespace` on desired-state); endpoint - datastore sizing for real volumes; true-CGNAT-SIM retest (parked). -- **OPERATOR-PRESENT (Part 5, NOT run):** escrow-create of the offsite token + tunnel token + WG - key mints a NEW recovery code `R` the operator must record — deliberately left for an interactive - operator run (never `--upload` a blob whose R nobody recorded). OPERATOR: publish 0.66/0.67 + - Day-0 vouch. Rollback binaries on the box: `felhom-agent.bak-0.65.0` / `.bak-0.66.0`. +## 4. Live state after / cleanup + +Agent **v0.68.0** live on felhom-pve; the temporary `local_backup_target=felhom-offsite` (set so the +selftest sourced the pbs-tier backup) **reverted to `local`**; WARN still 0; scratch band clean; +temp logs removed. Rollback binary: `felhom-agent.bak-0.67.0`. + +## 5. NOT yet live-validated / follow-ups / observations + +- **The restore-test is now unattended-capable for the pbs tier** (deadline fixed; teardown proven + as the token). The scheduler cadence stays disabled on the demo (`restore_test_cadence_seconds=0`) + — the proof used the identical selftest path. +- **Tier-1(local, frequent)/Tier-2(offsite, periodic) split** still needed before offsite can be the + DEFAULT backup target (the controller's ~30-min cadence + a single target = near-continuous offsite + uploads). Unchanged from S4. +- Deferred: S5 DR consume (WG key + offsite restore with only R); hub-driven tenancy automation; + true-CGNAT-SIM retest. OPERATOR: publish 0.68.0 + Day-0 vouch.