docs(offsite): S4.1 — restore-test unattended (tier-aware deadline; VM.Allocate phantom)

Offsite restore-test now runs unattended on the agent-token path (agent v0.68.0):
tier-aware restore-task timeout (pbs→120m, local→10m). The scratch-band
VM.Allocate follow-up was a phantom — the scratch restores INTO /pool/felhom
(ACL already grants it); the earlier 403 was the 10m-timeout consequence. No
ACL/host-install change. doc-06 S4 row + runbook §4b updated.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

documentation/architecture/06-offsite-connectivity.md

@@ -295,7 +295,7 @@ vantage (Peti VM 110) remains the thorough confirmation but no longer gates anyt
 | **S1 — endpoint provisioning + peer-sync surface** — **SHIPPED 2026-07-04** (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: **hub pushes over SSH** (`x/crypto/ssh`, `FixedHostKey` pin + pinned-type `HostKeyAlgorithms`) to a forced-command reconcile script; runbook `documentation/runbooks/offsite-endpoint.md`. Done-criterion met live: hub add/remove → `wg show` on the endpoint; tunnel `ep0.felhom.eu:443` → PBS through the wg0-only 8007 rule. | ~~provision the production VM…~~ (see runbook; production endpoint = a later re-run) | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓ |
 | **S2 — hub schema + desired-state** — **SHIPPED 2026-07-04** (hub v0.33.0). `POST /hosts/{id}/wg` (self-scoped) + merge-at-read `wireguard` block (§3.2 shape = the golden) + one-per-host index + re-key-in-place + `/offsite` UI page. Live-proven against the real felhom-pve record incl. old-agent (v0.63.0) tolerance. | ~~endpoint coords + peer registry…~~ | a registered box's desired-state carries its assignment ✓; hub UI shows the peer registry ✓ |
 | **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) |
-| **S4 — PBS over the tunnel — SHIPPED 2026-07-04** (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep `DatastoreBackup` token, dual-grant user+token on `/datastore/<ds>/<ns>`, cross-tenant 403); box `felhom-offsite` PVE storage `Server=10.77.0.1`, box-born encryption key, token-secret WARN fix, `local_backup_target` retarget; agent **v0.66.0** v4-pin + re-resolve watchdog + FELHOM_WG Critical, **v0.67.0** namespace-aware PBS client. | ~~point the box's PBS storage at `pbs_tunnel_ip:8007`~~ | real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: restore-test scheduler WAN deadline + scratch `VM.Allocate`; Tier-1-local-vs-Tier-2-offsite target split (`local_backup_target` is currently a single target). |
+| **S4 — PBS over the tunnel — SHIPPED 2026-07-04** (agent v0.65-0.67; runbook §4a/§4b). Endpoint per-customer tenancy (ns + privsep `DatastoreBackup` token, dual-grant user+token on `/datastore/<ds>/<ns>`, cross-tenant 403); box `felhom-offsite` PVE storage `Server=10.77.0.1`, box-born encryption key, token-secret WARN fix, `local_backup_target` retarget; agent **v0.66.0** v4-pin + re-resolve watchdog + FELHOM_WG Critical, **v0.67.0** namespace-aware PBS client. | ~~point the box's PBS storage at `pbs_tunnel_ip:8007`~~ | real vzdump of 9201 lands as ciphertext in ns over the tunnel ✓; ns-scoped verify=ok under the box's own token ✓; WARN gone ✓; restore decrypts with the box-born key ✓. Follow-ups: ~~restore-test scheduler WAN deadline + scratch `VM.Allocate`~~ **RESOLVED S4.1 (agent v0.68.0)** — tier-aware restore-task timeout (pbs→120m, local→10m); the offsite restore-test now runs unattended on the agent-token path (`pass:true`, teardown clean). The `VM.Allocate` item was a **phantom** — the earlier 403 was the 10m-timeout consequence (guest not yet in `/pool/felhom`); no ACL change. Remaining: Tier-1-local-vs-Tier-2-offsite target split (`local_backup_target` is a single target; the ~30-min controller cadence makes offsite-as-default near-continuous). |
 | **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* |
 | **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
 

documentation/runbooks/offsite-endpoint.md

@@ -223,10 +223,12 @@ returns `verify=ok`; the WARN is gone; a restore reads it back decrypting with t
 
 **Provisioning notes / follow-ups (S4):** (a) the `.pw` copy in `pbs-secrets` is a STATIC snapshot
 of the PVE token secret — a token rotation must re-copy it (a sync step for the hub-driven-tenancy
-slice). (b) The restore-test scheduler's internal restore-task deadline (~10 min) is too short for
-a WAN restore of a large volume, and the agent PVE token lacks `VM.Allocate` on the scratch VMID
-band for teardown — both are box-side config fixes needed before the offsite restore-test can run
-unattended (raise the restore deadline for the `pbs` tier; grant the scratch-band VM.Allocate).
+slice). (b) The offsite restore-test now runs **unattended** (S4.1, agent v0.68.0): the restore-task
+wait is tier-aware (`restore_test_pbs_restore_timeout_seconds`, default 120m for the `pbs` tier;
+local stays 10m). The earlier "grant scratch-band `VM.Allocate`" idea was a **phantom** — the scratch
+is restored INTO `/pool/felhom` (whose ACL already grants `VM.Allocate`), so teardown is authorized
+once the restore completes; the earlier 403 was the 10m-timeout firing teardown against a
+not-yet-pooled guest. Live-proven on the agent-token path: `pass:true`, teardown clean, band empty.
 
 ## 5. Peersync surface