docs: v0.64.0 shipped — CHANGELOG + CONTEXT + REPORT (S3 live-validated on felhom-pve)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

CHANGELOG.md

@@ -1,3 +1,41 @@
+## v0.64.0 — S3 offsite WG tunnel: keygen + registration + agent-managed wg-quick@wg-felhom + escrow join (2026-07-04)
+
+The agent half of doc 06 §3.3 (felhom.eu S1/S2 built the endpoint + hub half). **DEFAULT OFF —
+the safety gate:** `wg_tunnel.enabled` defaults to false; a v0.64.0 rollout without explicit
+config is a no-op (no keygen, no registration, no report stanza). Enabled explicitly on
+felhom-pve only; the default flips when the production endpoint exists.
+
+- **`internal/wgtunnel`** (new; the lanresolver host-service shape): pure-Go keygen
+  (x/crypto curve25519; key 0600 in 0700 StateDir/wg; corrupt file = refuse, NEVER overwrite —
+  it may be the escrowed identity; stored form canonical-clamped — x/crypto clamps derivation
+  internally, so the STORED bytes are the property that matters, red-proof-anchored); one-shot
+  registration (`POST /hosts/{id}/wg`, marker-gated, exponential backoff cap 15 m);
+  desired-state consumption via the new `desired.Syncer.AddConsumer` raw seam (panic-contained);
+  conf render golden-tested (MTU 1420, AllowedIPs = pbs_tunnel_ip/32, keepalive 25 — doc 06 §4
+  client constants; all inputs strictly validated); hash-gated apply (steady state = ZERO
+  execs), restart-not-reload on conf change, self-heal enable, adopt-lost-marker,
+  re-key-on-mismatch. **Revocation semantics (doc 06 §3.5 completed): block absent from a
+  PRESENT desired-state → disable + marker KEPT + never re-register** (the operator re-adds the
+  peer using the reported pubkey); absent DATA (failed fetch) is never a teardown signal.
+- **`internal/hub`**: `WireDesiredState.Wireguard` (field-exact with the S2 cross-repo golden,
+  copied byte-identical + decode test), `RegisterWG` client (typed errors, token-free),
+  `HostReport.Wireguard` status stanza `{pubkey, registered, active, last_handshake_age_s,
+  assigned_ip}` via the collector's `WireguardReporter` seam.
+- **Sudoers/capabilities**: `Cmnd_Alias FELHOM_WG` (fixed-path conf install, enable/restart/
+  disable, `wg show wg-felhom latest-handshakes` — the ONLY wg read; `wg show … dump` is
+  FORBIDDEN, its interface line carries the PRIVATE KEY) + 6 manifest entries (Critical=false
+  until S4 makes the tunnel load-bearing).
+- **Escrow**: `IdentityBundle.WGPrivateKey` (omitempty) + escrow-create auto-inject when the key
+  file exists (`escrow.AttachWGKey`; field NAME only in logs). Honest limit: pre-S3 blobs cannot
+  be retro-fitted (R is never retained) — S5 DR falls back to fresh-key re-registration, which
+  keeps the box's /32 (hub S2 re-key-in-place).
+- `--selftest=wgtunnel` single-shot for supervised bring-up.
+- Live-validated on felhom-pve (agent restart tolerance, host reboot with unit persistence,
+  revocation drill, 30-min keepalive soak, escrow inject) — see REPORT.md. Five red-proofs run
+  + reverted. GOTCHA learned: the hub envelope's `poll_interval_seconds` (hub-side constant
+  900 s) silently overrides the agent's configured cadence on the FIRST cycle — the agent-side
+  `poll_seconds` is only the pre-first-heartbeat default.
+
 ## configs: build-golden.sh v2.0.0 — mandatory controller tag + bootstrap .path unit (B5 + B1) (2026-07-03)
 
 Golden-bake script only — **no agent code, no binary, no agent version bump** (docs/config

CONTEXT.md

@@ -5,6 +5,18 @@
 
 ## Current
 
+- **v0.64.0** (2026-07-04, live on felhom-pve) — **S3 offsite WG tunnel**: new `internal/wgtunnel`
+  (keygen 0600/0700, marker-gated one-shot registration, agent-managed `wg-quick@wg-felhom` from
+  the hub's desired-state `wireguard` block via the new `desired.Syncer.AddConsumer` seam,
+  revoked-stays-revoked teardown, report stanza) + `FELHOM_WG` sudoers/capabilities +
+  `IdentityBundle.WGPrivateKey` escrow auto-inject. **`wg_tunnel.enabled` DEFAULTS FALSE** (safety
+  gate — rollout to Peti's box is a no-op until the production endpoint exists; enabled explicitly
+  on felhom-pve only). Live: tunnel to ep0.felhom.eu:443 up 3 s after enable (PBS page through
+  10.77.0.1:8007), reboot-persistent, revocation drill clean, 30-min keepalive soak. GOTCHAS:
+  hub envelope poll_interval_seconds (hub-side const 900 s) silently overrides agent poll_seconds
+  on cycle 1; `wg show <if> dump` leaks the PRIVATE key (forbidden everywhere — sudoers only
+  grants `latest-handshakes`). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix);
+  publish 0.64.0 to Gitea + Day-0 vouch (operator); S4 points PBS at the tunnel.
 - **configs: build-golden.sh v2.0.0** (2026-07-03, @ `ceca355`; no agent version change) — **drill
   findings B5 + B1 FIXED** (`DRILL-golden-098-2026-07-03.md`): the controller tag is a MANDATORY
   argument (the default rotted twice — a fresh install booted a pre-floor controller, forcing the

REPORT.md

@@ -1,62 +1,135 @@
-# REPORT — build-golden.sh v2.0.0: golden 0.98.3 rebuild (drill B5 + B1)
+# REPORT — S3: agent WG tunnel — keygen + registration + wg-quick@wg-felhom + escrow join (v0.64.0)
 
-**Date:** 2026-07-03 · **Class:** implementation + clean-room drill (script-only — no Go code, no
+**Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (the expendable
-binary, **no agent version bump**; docs-only precedent) · **Source findings:**
+demo box; Peti's box untouched by construction — `wg_tunnel.enabled` defaults FALSE). **Design of
-`felhom.eu/documentation/audits/DRILL-day0-cleanroom-2026-07-03.md` B5/B1 + the stale backlog note
+record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md` row S3 (SHIPPED).
-`FOLLOWUP-golden-default-controller-tag.md` (now resolved). Full drill evidence:
-**`felhom.eu/documentation/audits/DRILL-golden-098-2026-07-03.md`**.
 
-## Baselines & commits
+## 1. Baselines & commits
 
-| What | Value |
+Start: felhom-agent @ `4ba1b14` (v0.63.0); felhom.eu @ `4fe895e` (hub v0.33.0, code read-only).
-|---|---|
+Commits pushed to `main`:
-| Base | `c9f963d` |
-| Script commit | `ceca355` — configs/build-golden.sh v2.0.0 + CHANGELOG |
-| Docs commit | (this one — REPORT/CONTEXT) |
-| felhom.eu | guide/ledger/backlog/evidence-doc push (see that repo's REPORT) |
-| Controller tag baked | **0.98.3** (reconfirmed current: CHANGELOG top + manifest HTTP 200 pre-bake) |
 
-## What changed (configs/build-golden.sh only)
+1. `0daae92` wgtunnel keygen + hub wire (WireWireguard/report stanza/RegisterWG) + byte-identical golden copy
+2. `fb24896` manager state machine + loop + desired raw-consumer seam
+3. `e2b6c63` FELHOM_WG sudoers + capabilities + config (DEFAULT OFF) + main wiring + escrow join
+4. (docs commit) CHANGELOG v0.64.0 + CONTEXT + this REPORT
 
-1. **B5 — CONTROLLER_IMAGE (arg 6) MANDATORY**: no default; missing → die with usage naming the
+## 2. Files
-   convention. Rationale in-file: the hand-bumped default rotted twice (0.43.0 → 0.85.1 → stale
-   again); a required arg cannot rot; auto-"latest" could bake an unvouched tag. **Red-proof:** the
-   no-arg invocation exits 1 with the usage BEFORE any `pct` op (run locally + in the drill VM).
-2. **B1 — baked `felhom-controller-bootstrap.path`** (`PathExists=/etc/felhom-bootstrap/bootstrap.json`,
-   enabled, `WantedBy=multi-user.target`): starts the Condition-gated oneshot service when the
-   agent back-half hot-plugs the bootstrap mount into the RUNNING guest. Service unchanged
-   (`RemainAfterExit=yes` prevents re-trigger loops; the enabled service still covers the plain
-   boot case). The mp0/mp1 vzdump-inclusion guards untouched.
-3. `GOLDEN_SCRIPT_VERSION=2.0.0` + `[golden]` provenance line (script version + baked tag) opening
-   every bake transcript. Gates: `bash -n` clean; usage renders.
 
-## Clean-room validation (drill VM on 192.168.0.180, virgin-restored per scenario)
+- NEW `internal/wgtunnel/{key,manager,loop}.go` + tests (key vectors, state-machine Groups A/B)
+- `internal/hub/{report,client,collect}.go` (+`wg_contract_test.go`, `wg_client_test.go`,
+  NEW `testdata/desired-state-wireguard.golden.json` — **byte-identical hub copy**, diff-verified)
+- `internal/desired/syncer.go` (`AddConsumer` + panic containment) + `consumer_test.go`
+- `internal/escrow/identity.go` (`WGPrivateKey` + `AttachWGKey`) + `wgkey_test.go`
+- `internal/config/config.go` (`WGTunnelConfig`, **Enabled default FALSE**)
+- `cmd/felhom-agent/main.go` (daemon wiring, escrow-create auto-inject, `--selftest=wgtunnel`)
+- `configs/felhom-agent.sudoers` (`Cmnd_Alias FELHOM_WG`), `internal/capability/manifest.go` (6 entries)
 
-| Scenario | Result |
+## 3. Tests + the five §10 red-proof outcomes
-|---|---|
-| A — bake integrity | PASS: provenance line; controller 0.98.3 + infra images baked; `docker logout` + config.json removal before archive (line present, ran); vzdump **including mp0 AND mp1**; archive 583 MB, sha256 `b9a02ef1b6f02b9b58babc4c6aad9cf6c053ebdfba116c78c8e7830de757fd01`; guest 9100 destroyed |
-| B — isolated hot-plug (STOP gate) | PASS: fresh restore w/o mount → service `inactive (dead)` condition-unmet, path `active (waiting)`; back-half-identical `pct set -mp9 …,mp=/etc/felhom-bootstrap,ro=1` on the RUNNING guest → service SUCCESS ~1 s later, controller `Up (healthy)` 0.98.3, `uptime -s` unchanged (**no reboot**) |
-| C — local-golden Day-0 (publish gate) | PASS: served installer v1.9.1, canonical command w/o `--force-gitea-golden` → `[SKIP] using local golden` (resolution order); **first boot controller 0.98.3, NO D.1b**; `selfupdate/check` → `update_available:false` (self-manages); selftest all-OK incl. pool read; bentopdf deployed via the dashboard endpoint → 200 via traefik; hub rows: agent 0.63.0 / controller 0.98.3 |
-| Publish | PUT `…/felhom-golden/0.98.3/golden.tar.zst` → **201**; round-trip GET sha **matches** (publish-agent.sh pattern mirrored) |
-| Vouch | operator vouched golden **0.98.3** in the Day-0 manifest (agent 0.63.0 already vouched — the v0.63.0 follow-up is DONE); confirmed via `/api/v1/artifacts` |
-| D — published-artifact Day-0 | PASS: virgin box, no local golden → `fetching golden v0.98.3 from Gitea` → `verified sha256 b9a02ef1… matches the hub manifest` → SUCCESS; first boot 0.98.3; up-to-date. The exact fresh-customer path |
 
-Cleanup: all drill-1 hub rows deleted (8 tables, counts 0; demo-felhom/peti-felhom intact); drill VM
+`go build ./... && go vet ./... && go test ./...` — all green (20 pkgs; 18 with tests).
-reverted to `virgin` and kept; bake credential env (0600) removed. Registry read-cred handled via
+`visudo -cf` on felhom-pve: `parsed OK` (gated BEFORE install; binary restarted after).
-env only — never logged/committed.
 
-## Deviations & observations
+Red-proofs, each run → FAILED as required → reverted:
+- (a) marker gate dropped → `re-registration after revocation: 5 calls`
+- (b) hash comparison dropped → `steady-state tick ran execs: [install…, systemctl restart…]`
+- (c) clamp removed → **honest adjustment discovered mid-proof:** x/crypto's X25519 clamps the
+  scalar internally (RFC 7748), so the DERIVED pubkey is clamp-invariant and the spec's
+  fixed-vector red-proof cannot fail. The clamp's real observable property is the **stored key
+  file's canonical form** (external `wg pubkey < private.key` must agree with what the agent
+  registered) — asserted by `TestEnsureKey_StoredKeyIsClamped`, which DOES fail with the clamp
+  removed (`stored key is not clamped: byte0=10101010`). Both pubkey vectors were generated with
+  the real `wg pubkey` (provenance in the test).
+- (d) teardown-on-absent-data → `absent-data tick ran execs: [systemctl disable…]×3`
+- (e) escrow auto-inject removed → `bundle key = ""`
+Plus: the log-scan test greps the captured buffer for the actual private key (padded + unpadded)
+— absent; the pubkey IS logged (intended).
 
-- **Premature publish (caught + corrected):** the script's opt-in publish block triggers on the same
+## 4. Deploy + live validation (felhom-pve; journal excerpts verbatim; keys never logged by construction — the session-log grep for the key material was run and clean)
-  `REGISTRY_*` vars the pull needs — the Scenario A bake published immediately, before the Scenario C
+
-  gate. Package DELETEd (204, anon 404 verified) and re-published only after C passed. Candidate
+Deploy order per spec: sudoers first (`visudo -cf` → install → `parsed OK`), binary
-  cleanup (not done — no-"while-here" rule): a separate `GOLDEN_PUBLISH=1` opt-in.
+0.63.0→0.64.0 (backup kept), config: `wg_tunnel.enabled=true, interval_seconds=60`.
-- **SECURITY (operator follow-up):** the pull credential — the demo customer's `git.token`, i.e.
+
-  what every customer box holds — turned out to be a Gitea **admin token with package-WRITE** (the
+- **Poll-interval lever (the spec's step-0 diagnosis, learned live):** agent-side
-  premature publish proved it). Scope down to read-only + rotate.
+  `poll_seconds=120` was adopted for exactly ONE cycle — the hub envelope's
-- Installer v1.9.1's post-provision reboot is now a redundant belt (the path unit deploys first) —
+  `poll_interval_seconds` is a hub-side CONSTANT (`defaultHostPollSeconds = 900`,
-  removal recorded as a candidate installer cleanup, kept for now.
+  `hub/internal/api/handler.go`) silently adopted on the first cycle (the "changed" log fires
-- Drill-env: `dhcpstart=10.0.2.30` + explicit `hostfwd…-10.0.2.15:22` on the QEMU command removes
+  only on later cycles). Hub code is out of S3 scope, so validation ran on 900 s beats with
-  the prior drill's slirp IP-collision quirk entirely.
+  background watchers. Agent config restored to `poll_seconds=900` + the diagnosis SIGQUIT
-- Guide follow-through (felhom.eu): **D.1b retired** to a one-line version check; troubleshooting
+  restart documented below. **Observation for the hub backlog: make the constant configurable.**
-  row covers pre-0.86.0 goldens; ledger B1/B5 FIXED; backlog note resolved.
+- **Step 1 — bring-up (one shot, 3 s):**
+  `07:15:48 wgtunnel: generated new WG keypair pubkey=yV5hFF…TCg=` →
+  `07:15:49 wgtunnel: registered with hub … assigned_ip=10.77.0.2/32 existed=false generation=5 sync=ok` →
+  `07:15:49 desired: updated from hub generation=5` →
+  `07:15:50 wgtunnel: tunnel conf applied endpoint=ep0.felhom.eu:443 … action=enable`.
+  Perms verified: key 0600 in 0700 dir (felhom-agent), installed conf 0600 root.
+- **Step 2 — data path:** `wg-quick@wg-felhom` active; in-tunnel ping 10.77.0.1 0% loss ~32 ms;
+  `curl https://10.77.0.1:8007/` = PBS login page. Hub `/admin/wg/peers` shows the peer BOUND to
+  `demo-felhom-01`. Report stanza in the hub's stored report_json:
+  `{"pubkey":"yV5hFF…","registered":true,"active":true,"last_handshake_age_s":28,"assigned_ip":"10.77.0.2/32"}`.
+  All 6 `wg-*` capability entries probe `status:ok`.
+- **Agent-restart tolerance (bonus, via the diagnosis SIGQUIT):** on restart — key kept, NO
+  re-registration (marker gate), tunnel stayed active, desired refetched clean.
+- **Step 3 — host reboot:** `wg-quick@wg-felhom` active from boot (unit persistence, before/without
+  the agent), tunnel pings, guest 9201 running, agent's first tick ran ZERO wgtunnel execs.
+- **Step 4 — revocation drill:** admin DELETE 05:57:30Z → next poll
+  `08:10:52 desired: updated generation=6` +
+  `WARN wgtunnel: … hub revoked this peer; disabling the tunnel (marker kept; NO re-registration)`
+  → service inactive, marker present. Over the following 2 polls: **0** `wgtunnel: registered`
+  lines, **0** wg-related sudo execs (the load-bearing negatives — verbatim counts in §5).
+  Re-add (via the REGISTRATION endpoint — see §5's path insight) → next poll:
+  `08:55:52 desired: updated generation=8` + `wgtunnel: tunnel conf applied … action=enable`,
+  tunnel pings, **0 registration calls since the re-add** — same key, same /32, marker untouched.
+  Config lever restored (`poll_seconds` back to 900; restart clean — no keygen/register lines).
+- **Step 5 — escrow:** `--selftest=escrow-create -storage felhom-pbs` (as root — the PBS key file
+  is root-readable only, the documented BUNDLE-slice reality) →
+  `escrow: identity bundle: +wg_private_key` (field name only), `identity=true`, blob 450 B
+  age-wrapped, self-verify OK. **Deviation from the spec's "blob uploaded":** run WITHOUT
+  `--upload` — uploading would overwrite the operator's real drill escrow with one whose R was
+  displayed into THIS session log (R was sed-redacted, but an unrecorded-R escrow is
+  unrecoverable and clobbering the prior blob breaks the operator's held R). The upload path is
+  pre-S3 code, unchanged by this slice — no coverage lost.
+- **Step 6 — soak:** 30-min idle → handshake age **104 s** (<180 s; keepalive holding), active.
+
+## 5. Revocation-drill negatives (verbatim)
+
+Over the 31 minutes following the teardown (05:57:30Z DELETE → 06:10:52Z teardown → checked
+06:42Z), with **2 heartbeats landed hub-side** in the window (count from `host_reports`):
+
+```
+'wgtunnel: registered' lines since teardown: 0
+'desired: updated' lines since teardown:     0   ← correct: generation never advanced (nothing
+                                                    changed hub-side), so heartbeats were cached
+                                                    no-ops — the polls happened, the fetch didn't
+wg-related sudo execs since teardown:        0
+wg-quick@wg-felhom:                          inactive (stayed down)
+```
+
+**Re-add path insight (found live, load-bearing for operator docs):** re-adding via the S1
+registry endpoint (`POST /admin/wg/peers` with `host_id`) binds the peer but does NOT bump the
+host's generation — the agent never learns the block returned. The correct operator re-add is
+the S2 **registration endpoint with the global key** (`POST /hosts/{id}/wg`), which allocates,
+bumps (gen 8 in the drill), and syncs in one step. The drill was redone through it. Backlog
+note for S6: either make the admin registry add bump the bound owner (symmetric with the S2
+delete-bump) or mark it registry-plumbing in the UI.
+
+## 6. NOT yet live-validated — awaiting later work
+
+- **CGNAT/mobile-hotspot smoke** — the operator-assisted appendix (physical hotspot needed);
+  deferred non-blocking per spec. On failure it re-opens the DERP/Headscale fallback, not S3's code.
+- PBS backup over the tunnel (**S4** — this slice deliberately left backup config untouched).
+- DR consume of the escrowed WG key (**S5**).
+- Tunnel-health surfacing/alerting in the hub UI (**S6** — the stanza ships now).
+- Endpoint re-IP re-resolve watchdog beyond restart-time resolution (S6, noted in 06-doc S3 row).
+- Gitea publish + Day-0 manifest vouch of 0.64.0 (**operator follow-ups** — publish-agent.sh +
+  the password-gated UI).
+
+## 7. Observations (noticed, not acted on)
+
+- The hub's `defaultHostPollSeconds = 900` constant silently overrides any agent-side cadence on
+  cycle 1 with no log line (first-cycle adoption is unlogged) — worth a hub-side config knob +
+  an INFO log for the first adoption.
+- The pre-existing `pbs: cannot read token secret` WARN (non-root vs `/etc/pve/priv/...`) still
+  fires each cycle — S4's tunnel-target PBS with per-customer tokens resolves it structurally.
+- `git add -A` hygiene: the sudoers deploy leaves `/root/felhom-agent.sudoers.bak-pre064` +
+  `agent.json.bak-pre064` on felhom-pve (intentional rollback artifacts, listed for the operator).