felhom-agent/REPORT.md

# REPORT — S3: agent WG tunnel — keygen + registration + wg-quick@wg-felhom + escrow join (v0.64.0)

**Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (the expendable
demo box; Peti's box untouched by construction — `wg_tunnel.enabled` defaults FALSE). **Design of
record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md` row S3 (SHIPPED).

## 1. Baselines & commits

Start: felhom-agent @ `4ba1b14` (v0.63.0); felhom.eu @ `4fe895e` (hub v0.33.0, code read-only).
Commits pushed to `main`:

1. `0daae92` wgtunnel keygen + hub wire (WireWireguard/report stanza/RegisterWG) + byte-identical golden copy
2. `fb24896` manager state machine + loop + desired raw-consumer seam
3. `e2b6c63` FELHOM_WG sudoers + capabilities + config (DEFAULT OFF) + main wiring + escrow join
4. (docs commit) CHANGELOG v0.64.0 + CONTEXT + this REPORT

## 2. Files

- NEW `internal/wgtunnel/{key,manager,loop}.go` + tests (key vectors, state-machine Groups A/B)
- `internal/hub/{report,client,collect}.go` (+`wg_contract_test.go`, `wg_client_test.go`,
  NEW `testdata/desired-state-wireguard.golden.json` — **byte-identical hub copy**, diff-verified)
- `internal/desired/syncer.go` (`AddConsumer` + panic containment) + `consumer_test.go`
- `internal/escrow/identity.go` (`WGPrivateKey` + `AttachWGKey`) + `wgkey_test.go`
- `internal/config/config.go` (`WGTunnelConfig`, **Enabled default FALSE**)
- `cmd/felhom-agent/main.go` (daemon wiring, escrow-create auto-inject, `--selftest=wgtunnel`)
- `configs/felhom-agent.sudoers` (`Cmnd_Alias FELHOM_WG`), `internal/capability/manifest.go` (6 entries)

## 3. Tests + the five §10 red-proof outcomes

`go build ./... && go vet ./... && go test ./...` — all green (20 pkgs; 18 with tests).
`visudo -cf` on felhom-pve: `parsed OK` (gated BEFORE install; binary restarted after).

Red-proofs, each run → FAILED as required → reverted:
- (a) marker gate dropped → `re-registration after revocation: 5 calls`
- (b) hash comparison dropped → `steady-state tick ran execs: [install…, systemctl restart…]`
- (c) clamp removed → **honest adjustment discovered mid-proof:** x/crypto's X25519 clamps the
  scalar internally (RFC 7748), so the DERIVED pubkey is clamp-invariant and the spec's
  fixed-vector red-proof cannot fail. The clamp's real observable property is the **stored key
  file's canonical form** (external `wg pubkey < private.key` must agree with what the agent
  registered) — asserted by `TestEnsureKey_StoredKeyIsClamped`, which DOES fail with the clamp
  removed (`stored key is not clamped: byte0=10101010`). Both pubkey vectors were generated with
  the real `wg pubkey` (provenance in the test).
- (d) teardown-on-absent-data → `absent-data tick ran execs: [systemctl disable…]×3`
- (e) escrow auto-inject removed → `bundle key = ""`
Plus: the log-scan test greps the captured buffer for the actual private key (padded + unpadded)
— absent; the pubkey IS logged (intended).

## 4. Deploy + live validation (felhom-pve; journal excerpts verbatim; keys never logged by construction — the session-log grep for the key material was run and clean)

Deploy order per spec: sudoers first (`visudo -cf` → install → `parsed OK`), binary
0.63.0→0.64.0 (backup kept), config: `wg_tunnel.enabled=true, interval_seconds=60`.

- **Poll-interval lever (the spec's step-0 diagnosis, learned live):** agent-side
  `poll_seconds=120` was adopted for exactly ONE cycle — the hub envelope's
  `poll_interval_seconds` is a hub-side CONSTANT (`defaultHostPollSeconds = 900`,
  `hub/internal/api/handler.go`) silently adopted on the first cycle (the "changed" log fires
  only on later cycles). Hub code is out of S3 scope, so validation ran on 900 s beats with
  background watchers. Agent config restored to `poll_seconds=900` + the diagnosis SIGQUIT
  restart documented below. **Observation for the hub backlog: make the constant configurable.**
- **Step 1 — bring-up (one shot, 3 s):**
  `07:15:48 wgtunnel: generated new WG keypair pubkey=yV5hFF…TCg=` →
  `07:15:49 wgtunnel: registered with hub … assigned_ip=10.77.0.2/32 existed=false generation=5 sync=ok` →
  `07:15:49 desired: updated from hub generation=5` →
  `07:15:50 wgtunnel: tunnel conf applied endpoint=ep0.felhom.eu:443 … action=enable`.
  Perms verified: key 0600 in 0700 dir (felhom-agent), installed conf 0600 root.
- **Step 2 — data path:** `wg-quick@wg-felhom` active; in-tunnel ping 10.77.0.1 0% loss ~32 ms;
  `curl https://10.77.0.1:8007/` = PBS login page. Hub `/admin/wg/peers` shows the peer BOUND to
  `demo-felhom-01`. Report stanza in the hub's stored report_json:
  `{"pubkey":"yV5hFF…","registered":true,"active":true,"last_handshake_age_s":28,"assigned_ip":"10.77.0.2/32"}`.
  All 6 `wg-*` capability entries probe `status:ok`.
- **Agent-restart tolerance (bonus, via the diagnosis SIGQUIT):** on restart — key kept, NO
  re-registration (marker gate), tunnel stayed active, desired refetched clean.
- **Step 3 — host reboot:** `wg-quick@wg-felhom` active from boot (unit persistence, before/without
  the agent), tunnel pings, guest 9201 running, agent's first tick ran ZERO wgtunnel execs.
- **Step 4 — revocation drill:** admin DELETE 05:57:30Z → next poll
  `08:10:52 desired: updated generation=6` +
  `WARN wgtunnel: … hub revoked this peer; disabling the tunnel (marker kept; NO re-registration)`
  → service inactive, marker present. Over the following 2 polls: **0** `wgtunnel: registered`
  lines, **0** wg-related sudo execs (the load-bearing negatives — verbatim counts in §5).
  Re-add (via the REGISTRATION endpoint — see §5's path insight) → next poll:
  `08:55:52 desired: updated generation=8` + `wgtunnel: tunnel conf applied … action=enable`,
  tunnel pings, **0 registration calls since the re-add** — same key, same /32, marker untouched.
  Config lever restored (`poll_seconds` back to 900; restart clean — no keygen/register lines).
- **Step 5 — escrow:** `--selftest=escrow-create -storage felhom-pbs` (as root — the PBS key file
  is root-readable only, the documented BUNDLE-slice reality) →
  `escrow: identity bundle: +wg_private_key` (field name only), `identity=true`, blob 450 B
  age-wrapped, self-verify OK. **Deviation from the spec's "blob uploaded":** run WITHOUT
  `--upload` — uploading would overwrite the operator's real drill escrow with one whose R was
  displayed into THIS session log (R was sed-redacted, but an unrecorded-R escrow is
  unrecoverable and clobbering the prior blob breaks the operator's held R). The upload path is
  pre-S3 code, unchanged by this slice — no coverage lost.
- **Step 6 — soak:** 30-min idle → handshake age **104 s** (<180 s; keepalive holding), active.

## 5. Revocation-drill negatives (verbatim)

Over the 31 minutes following the teardown (05:57:30Z DELETE → 06:10:52Z teardown → checked
06:42Z), with **2 heartbeats landed hub-side** in the window (count from `host_reports`):

```
'wgtunnel: registered' lines since teardown: 0
'desired: updated' lines since teardown:     0   ← correct: generation never advanced (nothing
                                                    changed hub-side), so heartbeats were cached
                                                    no-ops — the polls happened, the fetch didn't
wg-related sudo execs since teardown:        0
wg-quick@wg-felhom:                          inactive (stayed down)
```

**Re-add path insight (found live, load-bearing for operator docs):** re-adding via the S1
registry endpoint (`POST /admin/wg/peers` with `host_id`) binds the peer but does NOT bump the
host's generation — the agent never learns the block returned. The correct operator re-add is
the S2 **registration endpoint with the global key** (`POST /hosts/{id}/wg`), which allocates,
bumps (gen 8 in the drill), and syncs in one step. The drill was redone through it. Backlog
note for S6: either make the admin registry add bump the bound owner (symmetric with the S2
delete-bump) or mark it registry-plumbing in the UI.

## 6. NOT yet live-validated — awaiting later work

- **CGNAT/mobile-hotspot smoke** — the operator-assisted appendix (physical hotspot needed);
  deferred non-blocking per spec. On failure it re-opens the DERP/Headscale fallback, not S3's code.
- PBS backup over the tunnel (**S4** — this slice deliberately left backup config untouched).
- DR consume of the escrowed WG key (**S5**).
- Tunnel-health surfacing/alerting in the hub UI (**S6** — the stanza ships now).
- Endpoint re-IP re-resolve watchdog beyond restart-time resolution (S6, noted in 06-doc S3 row).
- Gitea publish + Day-0 manifest vouch of 0.64.0 (**operator follow-ups** — publish-agent.sh +
  the password-gated UI).

## 7. Observations (noticed, not acted on)

- The hub's `defaultHostPollSeconds = 900` constant silently overrides any agent-side cadence on
  cycle 1 with no log line (first-cycle adoption is unlogged) — worth a hub-side config knob +
  an INFO log for the first adoption.
- The pre-existing `pbs: cannot read token secret` WARN (non-root vs `/etc/pve/priv/...`) still
  fires each cycle — S4's tunnel-target PBS with per-customer tokens resolves it structurally.
- `git add -A` hygiene: the sudoers deploy leaves `/root/felhom-agent.sudoers.bak-pre064` +
  `agent.json.bak-pre064` on felhom-pve (intentional rollback artifacts, listed for the operator).