felhom-agent/REPORT.md

REPORT — S3: agent WG tunnel — keygen + registration + wg-quick@wg-felhom + escrow join (v0.64.0)

Date: 2026-07-04 · Class: implementation + live validation on felhom-pve (the expendable demo box; Peti's box untouched by construction — wg_tunnel.enabled defaults FALSE). Design of record: felhom.eu/documentation/architecture/06-offsite-connectivity.md row S3 (SHIPPED).

1. Baselines & commits

Start: felhom-agent @ 4ba1b14 (v0.63.0); felhom.eu @ 4fe895e (hub v0.33.0, code read-only). Commits pushed to main:

1. 0daae92 wgtunnel keygen + hub wire (WireWireguard/report stanza/RegisterWG) + byte-identical golden copy
2. fb24896 manager state machine + loop + desired raw-consumer seam
3. e2b6c63 FELHOM_WG sudoers + capabilities + config (DEFAULT OFF) + main wiring + escrow join
4. (docs commit) CHANGELOG v0.64.0 + CONTEXT + this REPORT

2. Files

• NEW internal/wgtunnel/{key,manager,loop}.go + tests (key vectors, state-machine Groups A/B)
• internal/hub/{report,client,collect}.go (+wg_contract_test.go, wg_client_test.go, NEW testdata/desired-state-wireguard.golden.json — byte-identical hub copy, diff-verified)
• internal/desired/syncer.go (AddConsumer + panic containment) + consumer_test.go
• internal/escrow/identity.go (WGPrivateKey + AttachWGKey) + wgkey_test.go
• internal/config/config.go (WGTunnelConfig, Enabled default FALSE)
• cmd/felhom-agent/main.go (daemon wiring, escrow-create auto-inject, --selftest=wgtunnel)
• configs/felhom-agent.sudoers (Cmnd_Alias FELHOM_WG), internal/capability/manifest.go (6 entries)

3. Tests + the five §10 red-proof outcomes

go build ./... && go vet ./... && go test ./... — all green (20 pkgs; 18 with tests). visudo -cf on felhom-pve: parsed OK (gated BEFORE install; binary restarted after).

Red-proofs, each run → FAILED as required → reverted:

• (a) marker gate dropped → re-registration after revocation: 5 calls
• (b) hash comparison dropped → steady-state tick ran execs: [install…, systemctl restart…]
• (c) clamp removed → honest adjustment discovered mid-proof: x/crypto's X25519 clamps the scalar internally (RFC 7748), so the DERIVED pubkey is clamp-invariant and the spec's fixed-vector red-proof cannot fail. The clamp's real observable property is the stored key file's canonical form (external wg pubkey < private.key must agree with what the agent registered) — asserted by TestEnsureKey_StoredKeyIsClamped, which DOES fail with the clamp removed (stored key is not clamped: byte0=10101010). Both pubkey vectors were generated with the real wg pubkey (provenance in the test).
• (d) teardown-on-absent-data → absent-data tick ran execs: [systemctl disable…]×3
• (e) escrow auto-inject removed → bundle key = "" Plus: the log-scan test greps the captured buffer for the actual private key (padded + unpadded) — absent; the pubkey IS logged (intended).

4. Deploy + live validation (felhom-pve; journal excerpts verbatim; keys never logged by construction — the session-log grep for the key material was run and clean)

Deploy order per spec: sudoers first (visudo -cf → install → parsed OK), binary 0.63.0→0.64.0 (backup kept), config: wg_tunnel.enabled=true, interval_seconds=60.

• Poll-interval lever (the spec's step-0 diagnosis, learned live): agent-side poll_seconds=120 was adopted for exactly ONE cycle — the hub envelope's poll_interval_seconds is a hub-side CONSTANT (defaultHostPollSeconds = 900, hub/internal/api/handler.go) silently adopted on the first cycle (the "changed" log fires only on later cycles). Hub code is out of S3 scope, so validation ran on 900 s beats with background watchers. Agent config restored to poll_seconds=900 + the diagnosis SIGQUIT restart documented below. Observation for the hub backlog: make the constant configurable.
• Step 1 — bring-up (one shot, 3 s): 07:15:48 wgtunnel: generated new WG keypair pubkey=yV5hFF…TCg= → 07:15:49 wgtunnel: registered with hub … assigned_ip=10.77.0.2/32 existed=false generation=5 sync=ok → 07:15:49 desired: updated from hub generation=5 → 07:15:50 wgtunnel: tunnel conf applied endpoint=ep0.felhom.eu:443 … action=enable. Perms verified: key 0600 in 0700 dir (felhom-agent), installed conf 0600 root.
• Step 2 — data path: wg-quick@wg-felhom active; in-tunnel ping 10.77.0.1 0% loss ~32 ms; curl https://10.77.0.1:8007/ = PBS login page. Hub /admin/wg/peers shows the peer BOUND to demo-felhom-01. Report stanza in the hub's stored report_json: {"pubkey":"yV5hFF…","registered":true,"active":true,"last_handshake_age_s":28,"assigned_ip":"10.77.0.2/32"}. All 6 wg-* capability entries probe status:ok.
• Agent-restart tolerance (bonus, via the diagnosis SIGQUIT): on restart — key kept, NO re-registration (marker gate), tunnel stayed active, desired refetched clean.
• Step 3 — host reboot: wg-quick@wg-felhom active from boot (unit persistence, before/without the agent), tunnel pings, guest 9201 running, agent's first tick ran ZERO wgtunnel execs.
• Step 4 — revocation drill: admin DELETE 05:57:30Z → next poll 08:10:52 desired: updated generation=6 + WARN wgtunnel: … hub revoked this peer; disabling the tunnel (marker kept; NO re-registration) → service inactive, marker present. Over the following 2 polls: 0 wgtunnel: registered lines, 0 wg-related sudo execs (the load-bearing negatives — verbatim counts in §5). Re-add (via the REGISTRATION endpoint — see §5's path insight) → next poll: 08:55:52 desired: updated generation=8 + wgtunnel: tunnel conf applied … action=enable, tunnel pings, 0 registration calls since the re-add — same key, same /32, marker untouched. Config lever restored (poll_seconds back to 900; restart clean — no keygen/register lines).
• Step 5 — escrow: --selftest=escrow-create -storage felhom-pbs (as root — the PBS key file is root-readable only, the documented BUNDLE-slice reality) → escrow: identity bundle: +wg_private_key (field name only), identity=true, blob 450 B age-wrapped, self-verify OK. Deviation from the spec's "blob uploaded": run WITHOUT --upload — uploading would overwrite the operator's real drill escrow with one whose R was displayed into THIS session log (R was sed-redacted, but an unrecorded-R escrow is unrecoverable and clobbering the prior blob breaks the operator's held R). The upload path is pre-S3 code, unchanged by this slice — no coverage lost.
• Step 6 — soak: 30-min idle → handshake age 104 s (<180 s; keepalive holding), active.

5. Revocation-drill negatives (verbatim)

Over the 31 minutes following the teardown (05:57:30Z DELETE → 06:10:52Z teardown → checked 06:42Z), with 2 heartbeats landed hub-side in the window (count from host_reports):

'wgtunnel: registered' lines since teardown: 0
'desired: updated' lines since teardown:     0   ← correct: generation never advanced (nothing
                                                    changed hub-side), so heartbeats were cached
                                                    no-ops — the polls happened, the fetch didn't
wg-related sudo execs since teardown:        0
wg-quick@wg-felhom:                          inactive (stayed down)

Re-add path insight (found live, load-bearing for operator docs): re-adding via the S1 registry endpoint (POST /admin/wg/peers with host_id) binds the peer but does NOT bump the host's generation — the agent never learns the block returned. The correct operator re-add is the S2 registration endpoint with the global key (POST /hosts/{id}/wg), which allocates, bumps (gen 8 in the drill), and syncs in one step. The drill was redone through it. Backlog note for S6: either make the admin registry add bump the bound owner (symmetric with the S2 delete-bump) or mark it registry-plumbing in the UI.

6. NOT yet live-validated — awaiting later work

• CGNAT/mobile-hotspot smoke — the operator-assisted appendix (physical hotspot needed); deferred non-blocking per spec. On failure it re-opens the DERP/Headscale fallback, not S3's code.
• PBS backup over the tunnel (S4 — this slice deliberately left backup config untouched).
• DR consume of the escrowed WG key (S5).
• Tunnel-health surfacing/alerting in the hub UI (S6 — the stanza ships now).
• Endpoint re-IP re-resolve watchdog beyond restart-time resolution (S6, noted in 06-doc S3 row).
• Gitea publish + Day-0 manifest vouch of 0.64.0 (operator follow-ups — publish-agent.sh + the password-gated UI).

7. Observations (noticed, not acted on)

• The hub's defaultHostPollSeconds = 900 constant silently overrides any agent-side cadence on cycle 1 with no log line (first-cycle adoption is unlogged) — worth a hub-side config knob + an INFO log for the first adoption.
• The pre-existing pbs: cannot read token secret WARN (non-root vs /etc/pve/priv/...) still fires each cycle — S4's tunnel-target PBS with per-customer tokens resolves it structurally.
• git add -A hygiene: the sudoers deploy leaves /root/felhom-agent.sudoers.bak-pre064 + agent.json.bak-pre064 on felhom-pve (intentional rollback artifacts, listed for the operator).