docs: REPORT + CONTEXT for S4.1 (tier-aware deadline + teardown phantom confirmed)

Live diagnosis on the agent-token path: offsite restore-test pass:true,
verified:boot+running, teardown succeeded (no 403), scratch band clean. The
VM.Allocate follow-up was a PHANTOM — the earlier 403 was the 10m-timeout firing
teardown against a not-yet-pool-associated guest; with the timeout fixed the
guest is in /pool/felhom and teardown is authorized. No ACL/host-install change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

CONTEXT.md

@@ -5,6 +5,18 @@
 
 ## Current
 
+- **v0.68.0** (2026-07-04, live on felhom-pve) — **S4.1: unattended offsite restore-test**.
+  **Tier-aware restore-task deadline:** `RestoreTestSpec.RestoreTaskTimeout` (0→10m default) from
+  `config.RestoreTestPBSRestoreTimeoutSeconds` (accessor default **120m**), set only when
+  `SourceTier=="pbs"` (`main.restoreTaskTimeout`); local tier UNCHANGED. Fixes the WAN restore being
+  killed at 10m → mid-restore teardown → leaked scratch. **Teardown "VM.Allocate" follow-up =
+  PHANTOM (diagnosed, not blind-fixed):** ran the restore-test on the AGENT-TOKEN path sourcing the
+  offsite (pbs) backup → `pass:true verified:boot+running`, teardown succeeded (`torn down
+  vmid=990000`, no 403), scratch band clean. The earlier 403 was the 10m-timeout consequence (guest
+  not yet pool-associated); the scratch is restored INTO `/pool/felhom` (ACL already grants
+  VM.Allocate) so teardown is authorized once the restore completes. **No ACL/host-install change.**
+  OPEN: publish 0.68.0 + Day-0 vouch; Tier-1/Tier-2 split for offsite-as-default; S5 DR consume.
+  Rollback `felhom-agent.bak-0.67.0`. Detail: REPORT.md.
 - **v0.66.0 + v0.67.0** (2026-07-04, live on felhom-pve) — **S4: PBS over the tunnel**. **v0.66.0**:
   wgtunnel **v4-pin** (renderConf writes the resolved A LITERAL, never DNS/AAAA; `Resolver` seam,
   lowest addr; cached → steady-state zero-DNS/zero-exec) + **re-resolve watchdog** (`Manager.Watchdog`,

REPORT.md

@@ -1,76 +1,72 @@
-# REPORT — S4: PBS over the tunnel (agent v0.65→v0.67 + live tenancy on felhom-hetzner)
+# REPORT — S4.1: unattended offsite restore-test (tier-aware deadline + teardown diagnosis)
 
-**Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (box) +
+**Date:** 2026-07-04 · **Class:** implementation + live diagnosis on felhom-pve (agent token).
-felhom-hetzner (endpoint). **Design of record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md`
+**Design of record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md` S4 row;
-§3.4 (SHIPPED), §4.2 (v4-pin SHIPPED), S4 roadmap row; runbook `offsite-endpoint.md` §4a/§4b.
+runbook `offsite-endpoint.md` §4b.
 
-## 1. Baselines & commits (felhom-agent, all pushed to `main`)
+## 1. Baselines & commits (felhom-agent → `main`)
 
-Start: `c618fc6` (v0.65.0, published+vouched).
+Start: `fee1fcf` (v0.67.0).
-- **`734f45c`** v0.66.0 — wgtunnel v4-pin + re-resolve watchdog; FELHOM_WG Critical flips.
+- **`dc70e15`** v0.68.0 — tier-aware restore-task deadline (Part 1).
-- **`027948b`** v0.67.0 — namespace-aware PBS client (operator-approved, see §3).
+- this commit — REPORT + CONTEXT (the diagnosis outcome).
-- this commit — CHANGELOG (0.66/0.67 above) + CONTEXT + this REPORT.
-felhom.eu docs: `1907dab` (runbook §4a/§4b + doc-06 §3.4/§4.2/S4 row).
 
-## 2. Agent code (Parts 1+2 + the approved ns-aware addition)
+## 2. Part 1 — tier-aware restore-task deadline
 
-- **v4-pin (doc §4.2, `internal/wgtunnel`).** `renderConf` takes the pre-resolved IPv4 literal →
+The restore-task wait (`restoretest.go` ~L246) used `WaitOptions{}` → the 10m default. A WAN (pbs)
-  `Endpoint = <ip>:<port>` (never the DNS name, never AAAA). `Resolver` seam (`LookupNetIP "ip4"`,
+restore of a large guest exceeds it, so the wait expired mid-restore and teardown then fired against
-  lowest address = deterministic fleet-wide); renderConf stays pure; resolved IP cached →
+a still-restoring guest. Fix:
-  steady-state Apply = zero DNS + zero execs. DNS failure keeps last conf (never a teardown).
+- `RestoreTestSpec.RestoreTaskTimeout` (0 → 10m default); the restore `WaitTask` passes it.
-- **Re-resolve watchdog (`Manager.Watchdog`, loop-only).** Handshake age > `stale_after_seconds`
+- `config.BackupConfig.RestoreTestPBSRestoreTimeoutSeconds` + accessor `RestoreTestPBSRestoreTimeout()`
-  (default 180) → re-resolve; IP changed → re-render+restart (endpoint re-IP recovery); same → no
+  (positive as-is, else **120m**).
-  churn (throttled). Reuses `wg show latest-handshakes` (never `dump`).
+- `main.restoreTaskTimeout(cfg, tier)` — the configured PBS timeout only when `SourceTier=="pbs"`,
-- **Capability Critical flips.** `wg-conf-install`/`wg-enable`/`wg-restart`/`wg-handshake-read` →
+  else 0. Set at BOTH spec builds (scheduler + selftest). **Local tier unchanged (10m).**
-  Critical=true (backups ride the tunnel now); apt-install + disable stay non-critical.
+- Tests: `WaitOptions.Timeout` pbs→120m / local→0 (red-proofed against `WaitOptions{}`: pbs
-- **Namespace-aware PBS client (v0.67.0, `internal/pbs`).** Config.Namespace → `Snapshots` passes
+  assertion fails); accessor contract (0/neg→120m, positive as-is). Full `go test ./...` green.
-  `?ns=`, `Verify` passes `ns=`. Root-ns clients unchanged. Why: Phase-1 found the ns-unaware
-  datastore-root calls 403 for a per-tenant token, and whole-datastore verify needed
-  `Datastore.Verify` (~admin, breaking isolation). `proxmox.Storage.Namespace` parsed from
-  `/storage`; `pbsTargetsFromPVE` threads it. Green gate + red-proofs (v4-pin golden, watchdog
-  negatives, capability set, ns-scoping) all fire.
 
-## 3. Phase-1 endpoint tenancy probe (felhom-hetzner) — the empirical spike
+## 3. Part 2 — teardown diagnosis (the "grant VM.Allocate" follow-up) — PHANTOM CONFIRMED
 
-Created ns `demo-felhom-01` + privsep token `felhom@pbs!demo-felhom-01`; adversarial second tenant.
+Ran the restore-test on the **agent-token path** (selftest = the identical `RunRestoreTest` the
-**Confirmed minimal ACL:** `DatastoreBackup` on `/datastore/felhom-offsite/<ns>` (namespace path,
+scheduler runs; all PVE ops authenticate as `felhom-agent@pve!agent`), sourcing the **offsite
-**NOT** `/ns/<ns>`) granted to **BOTH** the user `felhom@pbs` **and** the token — PBS privsep =
+(pbs-tier)** backup of guest 9201, with v0.68.0's 120m timeout.
-intersection(user, token); isolation holds (cross-ns list/backup → **403**, proven). `DatastoreBackup`
-covers backup/list-own/restore-own **and ns-scoped verify** (no `Datastore.Verify`/admin), but
-**not** prune/forget (`Datastore.Modify|Prune`) — the box can't delete its own backups (safety).
-Probe tenants cleaned up; only the real `demo-felhom-01` tenant remains.
 
-## 4. Box-side + live Scenario-D validation (felhom-pve)
+**Scenario B (done-criterion) — GREEN:**
+```
+source_archive: felhom-offsite:backup/ct/9201/2026-07-04T14:55:44Z
+source_tier: pbs · pass: true · verified: boot+running · scratch_vmid: 990000 · duration 4m49s
+restore → net-link-down → boot (running) → teardown → scratch band clean (no leak)
+```
 
-- `felhom-offsite` PVE storage (`Server=10.77.0.1` over the tunnel, box-born encryption key,
+**Scenario C — the teardown succeeded as the agent token; (b) is a PHANTOM:**
-  argv-safe `.pw`/`.enc`+storage.cfg direct-write) → `pvesm status` active. Agent PVE token granted
+```
-  `FelhomAgentStore` on the new storage (dual-grant). WARN fix: `pbs-secrets/` (0700 felhom-agent)
+gate decision: class=guest_destroy guest=990000 allowed=true reason=benign
-  with both `.pw` copies + `pbs_secret_dir` → **zero WARN**, both datastores in the verify cycle.
+restore-test: scratch guest torn down vmid=990000
-- **Scenario D — ALL GREEN:** a real vzdump of guest 9201 (crash-consistent, ~4.6 GB compressed,
+pct list | grep 9900 → no scratch guests
-  ~20 min over the tunnel) landed as **ciphertext** in ns `demo-felhom-01` (`root.pxar` crypt-mode
+```
-  `encrypt`); the agent report **listed** it (ns-aware `Snapshots`); the ns-scoped **verify ran
+No 403, no `VM.Allocate` error. Root cause of the earlier 403 (REPORT of S4): the 10m wait expired
-  under the box's own `felhom@pbs!demo-felhom-01` token and returned `ok`** (no admin, no 403); the
+**mid-restore**, so teardown hit a guest that existed but was **not yet pool-associated** (the pool
-  WARN grep = 0; a **restore round-tripped** — restored from offsite, **decrypted with the box-born
+membership lands with restore completion) — a bare `/vms/990000` the pool ACL didn't cover. With the
-  key** (`b0:fe:2a:4e…`), **booted** (running, systemd, uptime 14 s), torn down clean.
+timeout fixed → the restore completes → the guest is a member of `/pool/felhom` (restored with
+`Pool=DefaultPool`, `restoretest.go` L229) → its destroy is authorized by the existing
+`FelhomAgentGuest` grant (`VM.Allocate` + `Pool.Allocate` at `/pool/felhom`). **No ACL change, no
+host-install edit** — the pool-scoped containment (`SPIKE-pool-scoped-acl-2026-07-01`) stands.
 
-## 5. NOT yet live-validated / findings / observations
+(Note: this run's restore was fast — home *download* ≫ *upload*; the earlier ~10-min stall was link
+contention. The 120m ceiling covers the slow case regardless — for an unattended nightly test a
+false timeout is the failure mode to prevent.)
 
-- **The restore-test SCHEDULER couldn't complete the offsite restore-test** (the round-trip was
+## 4. Live state after / cleanup
-  proven MANUALLY as root instead). Two box-side follow-ups: (a) its internal restore-task deadline
+
-  (~10 min) is too short for a WAN restore of a large volume; (b) the agent PVE token lacks
+Agent **v0.68.0** live on felhom-pve; the temporary `local_backup_target=felhom-offsite` (set so the
-  `VM.Allocate` on the scratch VMID band for teardown. Raise the deadline for the `pbs` tier + grant
+selftest sourced the pbs-tier backup) **reverted to `local`**; WARN still 0; scratch band clean;
-  the scratch-band `VM.Allocate` before the offsite restore-test can run unattended.
+temp logs removed. Rollback binary: `felhom-agent.bak-0.67.0`.
-- **Retarget reverted.** The controller triggers a full backup ~every 30 min; with a SINGLE backup
+
-  target (`local_backup_target`) retargeted to `felhom-offsite`, each becomes a ~20-min offsite
+## 5. NOT yet live-validated / follow-ups / observations
-  upload — near-continuous, uplink-saturating. Reverted to `local`. **Offsite-as-default needs the
+
-  Tier-1(local, frequent)/Tier-2(offsite, periodic) split** — the single-target model can't be the
+- **The restore-test is now unattended-capable for the pbs tier** (deadline fixed; teardown proven
-  default. The offsite storage + tenancy stay live for scheduled/on-demand offsite backups.
+  as the token). The scheduler cadence stays disabled on the demo (`restore_test_cadence_seconds=0`)
-- **Spec field-name correction:** the retarget field is `backup.local_backup_target` (not
+  — the proof used the identical selftest path.
-  `backup_target`); `defaultBackupTarget` (`felhom-pbs`) unchanged in code.
+- **Tier-1(local, frequent)/Tier-2(offsite, periodic) split** still needed before offsite can be the
-- Deferred: DR consume of the bundle incl. WG key + offsite restore with only R (S5); hub-driven
+  DEFAULT backup target (the controller's ~30-min cadence + a single target = near-continuous offsite
-  tenancy automation (hub creates ns+token, serves `pbs_namespace` on desired-state); endpoint
+  uploads). Unchanged from S4.
-  datastore sizing for real volumes; true-CGNAT-SIM retest (parked).
+- Deferred: S5 DR consume (WG key + offsite restore with only R); hub-driven tenancy automation;
-- **OPERATOR-PRESENT (Part 5, NOT run):** escrow-create of the offsite token + tunnel token + WG
+  true-CGNAT-SIM retest. OPERATOR: publish 0.68.0 + Day-0 vouch.
-  key mints a NEW recovery code `R` the operator must record — deliberately left for an interactive
-  operator run (never `--upload` a blob whose R nobody recorded). OPERATOR: publish 0.66/0.67 +
-  Day-0 vouch. Rollback binaries on the box: `felhom-agent.bak-0.65.0` / `.bak-0.66.0`.