diff --git a/CHANGELOG.md b/CHANGELOG.md index 354084f..8c3ef6d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,41 @@ +## v0.64.0 — S3 offsite WG tunnel: keygen + registration + agent-managed wg-quick@wg-felhom + escrow join (2026-07-04) + +The agent half of doc 06 §3.3 (felhom.eu S1/S2 built the endpoint + hub half). **DEFAULT OFF — +the safety gate:** `wg_tunnel.enabled` defaults to false; a v0.64.0 rollout without explicit +config is a no-op (no keygen, no registration, no report stanza). Enabled explicitly on +felhom-pve only; the default flips when the production endpoint exists. + +- **`internal/wgtunnel`** (new; the lanresolver host-service shape): pure-Go keygen + (x/crypto curve25519; key 0600 in 0700 StateDir/wg; corrupt file = refuse, NEVER overwrite — + it may be the escrowed identity; stored form canonical-clamped — x/crypto clamps derivation + internally, so the STORED bytes are the property that matters, red-proof-anchored); one-shot + registration (`POST /hosts/{id}/wg`, marker-gated, exponential backoff cap 15 m); + desired-state consumption via the new `desired.Syncer.AddConsumer` raw seam (panic-contained); + conf render golden-tested (MTU 1420, AllowedIPs = pbs_tunnel_ip/32, keepalive 25 — doc 06 §4 + client constants; all inputs strictly validated); hash-gated apply (steady state = ZERO + execs), restart-not-reload on conf change, self-heal enable, adopt-lost-marker, + re-key-on-mismatch. **Revocation semantics (doc 06 §3.5 completed): block absent from a + PRESENT desired-state → disable + marker KEPT + never re-register** (the operator re-adds the + peer using the reported pubkey); absent DATA (failed fetch) is never a teardown signal. +- **`internal/hub`**: `WireDesiredState.Wireguard` (field-exact with the S2 cross-repo golden, + copied byte-identical + decode test), `RegisterWG` client (typed errors, token-free), + `HostReport.Wireguard` status stanza `{pubkey, registered, active, last_handshake_age_s, + assigned_ip}` via the collector's `WireguardReporter` seam. +- **Sudoers/capabilities**: `Cmnd_Alias FELHOM_WG` (fixed-path conf install, enable/restart/ + disable, `wg show wg-felhom latest-handshakes` — the ONLY wg read; `wg show … dump` is + FORBIDDEN, its interface line carries the PRIVATE KEY) + 6 manifest entries (Critical=false + until S4 makes the tunnel load-bearing). +- **Escrow**: `IdentityBundle.WGPrivateKey` (omitempty) + escrow-create auto-inject when the key + file exists (`escrow.AttachWGKey`; field NAME only in logs). Honest limit: pre-S3 blobs cannot + be retro-fitted (R is never retained) — S5 DR falls back to fresh-key re-registration, which + keeps the box's /32 (hub S2 re-key-in-place). +- `--selftest=wgtunnel` single-shot for supervised bring-up. +- Live-validated on felhom-pve (agent restart tolerance, host reboot with unit persistence, + revocation drill, 30-min keepalive soak, escrow inject) — see REPORT.md. Five red-proofs run + + reverted. GOTCHA learned: the hub envelope's `poll_interval_seconds` (hub-side constant + 900 s) silently overrides the agent's configured cadence on the FIRST cycle — the agent-side + `poll_seconds` is only the pre-first-heartbeat default. + ## configs: build-golden.sh v2.0.0 — mandatory controller tag + bootstrap .path unit (B5 + B1) (2026-07-03) Golden-bake script only — **no agent code, no binary, no agent version bump** (docs/config diff --git a/CONTEXT.md b/CONTEXT.md index fa48957..042593b 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -5,6 +5,18 @@ ## Current +- **v0.64.0** (2026-07-04, live on felhom-pve) — **S3 offsite WG tunnel**: new `internal/wgtunnel` + (keygen 0600/0700, marker-gated one-shot registration, agent-managed `wg-quick@wg-felhom` from + the hub's desired-state `wireguard` block via the new `desired.Syncer.AddConsumer` seam, + revoked-stays-revoked teardown, report stanza) + `FELHOM_WG` sudoers/capabilities + + `IdentityBundle.WGPrivateKey` escrow auto-inject. **`wg_tunnel.enabled` DEFAULTS FALSE** (safety + gate — rollout to Peti's box is a no-op until the production endpoint exists; enabled explicitly + on felhom-pve only). Live: tunnel to ep0.felhom.eu:443 up 3 s after enable (PBS page through + 10.77.0.1:8007), reboot-persistent, revocation drill clean, 30-min keepalive soak. GOTCHAS: + hub envelope poll_interval_seconds (hub-side const 900 s) silently overrides agent poll_seconds + on cycle 1; `wg show dump` leaks the PRIVATE key (forbidden everywhere — sudoers only + grants `latest-handshakes`). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix); + publish 0.64.0 to Gitea + Day-0 vouch (operator); S4 points PBS at the tunnel. - **configs: build-golden.sh v2.0.0** (2026-07-03, @ `ceca355`; no agent version change) — **drill findings B5 + B1 FIXED** (`DRILL-golden-098-2026-07-03.md`): the controller tag is a MANDATORY argument (the default rotted twice — a fresh install booted a pre-floor controller, forcing the diff --git a/REPORT.md b/REPORT.md index 678dfa7..0954705 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,62 +1,135 @@ -# REPORT — build-golden.sh v2.0.0: golden 0.98.3 rebuild (drill B5 + B1) +# REPORT — S3: agent WG tunnel — keygen + registration + wg-quick@wg-felhom + escrow join (v0.64.0) -**Date:** 2026-07-03 · **Class:** implementation + clean-room drill (script-only — no Go code, no -binary, **no agent version bump**; docs-only precedent) · **Source findings:** -`felhom.eu/documentation/audits/DRILL-day0-cleanroom-2026-07-03.md` B5/B1 + the stale backlog note -`FOLLOWUP-golden-default-controller-tag.md` (now resolved). Full drill evidence: -**`felhom.eu/documentation/audits/DRILL-golden-098-2026-07-03.md`**. +**Date:** 2026-07-04 · **Class:** implementation + live validation on felhom-pve (the expendable +demo box; Peti's box untouched by construction — `wg_tunnel.enabled` defaults FALSE). **Design of +record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md` row S3 (SHIPPED). -## Baselines & commits +## 1. Baselines & commits -| What | Value | -|---|---| -| Base | `c9f963d` | -| Script commit | `ceca355` — configs/build-golden.sh v2.0.0 + CHANGELOG | -| Docs commit | (this one — REPORT/CONTEXT) | -| felhom.eu | guide/ledger/backlog/evidence-doc push (see that repo's REPORT) | -| Controller tag baked | **0.98.3** (reconfirmed current: CHANGELOG top + manifest HTTP 200 pre-bake) | +Start: felhom-agent @ `4ba1b14` (v0.63.0); felhom.eu @ `4fe895e` (hub v0.33.0, code read-only). +Commits pushed to `main`: -## What changed (configs/build-golden.sh only) +1. `0daae92` wgtunnel keygen + hub wire (WireWireguard/report stanza/RegisterWG) + byte-identical golden copy +2. `fb24896` manager state machine + loop + desired raw-consumer seam +3. `e2b6c63` FELHOM_WG sudoers + capabilities + config (DEFAULT OFF) + main wiring + escrow join +4. (docs commit) CHANGELOG v0.64.0 + CONTEXT + this REPORT -1. **B5 — CONTROLLER_IMAGE (arg 6) MANDATORY**: no default; missing → die with usage naming the - convention. Rationale in-file: the hand-bumped default rotted twice (0.43.0 → 0.85.1 → stale - again); a required arg cannot rot; auto-"latest" could bake an unvouched tag. **Red-proof:** the - no-arg invocation exits 1 with the usage BEFORE any `pct` op (run locally + in the drill VM). -2. **B1 — baked `felhom-controller-bootstrap.path`** (`PathExists=/etc/felhom-bootstrap/bootstrap.json`, - enabled, `WantedBy=multi-user.target`): starts the Condition-gated oneshot service when the - agent back-half hot-plugs the bootstrap mount into the RUNNING guest. Service unchanged - (`RemainAfterExit=yes` prevents re-trigger loops; the enabled service still covers the plain - boot case). The mp0/mp1 vzdump-inclusion guards untouched. -3. `GOLDEN_SCRIPT_VERSION=2.0.0` + `[golden]` provenance line (script version + baked tag) opening - every bake transcript. Gates: `bash -n` clean; usage renders. +## 2. Files -## Clean-room validation (drill VM on 192.168.0.180, virgin-restored per scenario) +- NEW `internal/wgtunnel/{key,manager,loop}.go` + tests (key vectors, state-machine Groups A/B) +- `internal/hub/{report,client,collect}.go` (+`wg_contract_test.go`, `wg_client_test.go`, + NEW `testdata/desired-state-wireguard.golden.json` — **byte-identical hub copy**, diff-verified) +- `internal/desired/syncer.go` (`AddConsumer` + panic containment) + `consumer_test.go` +- `internal/escrow/identity.go` (`WGPrivateKey` + `AttachWGKey`) + `wgkey_test.go` +- `internal/config/config.go` (`WGTunnelConfig`, **Enabled default FALSE**) +- `cmd/felhom-agent/main.go` (daemon wiring, escrow-create auto-inject, `--selftest=wgtunnel`) +- `configs/felhom-agent.sudoers` (`Cmnd_Alias FELHOM_WG`), `internal/capability/manifest.go` (6 entries) -| Scenario | Result | -|---|---| -| A — bake integrity | PASS: provenance line; controller 0.98.3 + infra images baked; `docker logout` + config.json removal before archive (line present, ran); vzdump **including mp0 AND mp1**; archive 583 MB, sha256 `b9a02ef1b6f02b9b58babc4c6aad9cf6c053ebdfba116c78c8e7830de757fd01`; guest 9100 destroyed | -| B — isolated hot-plug (STOP gate) | PASS: fresh restore w/o mount → service `inactive (dead)` condition-unmet, path `active (waiting)`; back-half-identical `pct set -mp9 …,mp=/etc/felhom-bootstrap,ro=1` on the RUNNING guest → service SUCCESS ~1 s later, controller `Up (healthy)` 0.98.3, `uptime -s` unchanged (**no reboot**) | -| C — local-golden Day-0 (publish gate) | PASS: served installer v1.9.1, canonical command w/o `--force-gitea-golden` → `[SKIP] using local golden` (resolution order); **first boot controller 0.98.3, NO D.1b**; `selfupdate/check` → `update_available:false` (self-manages); selftest all-OK incl. pool read; bentopdf deployed via the dashboard endpoint → 200 via traefik; hub rows: agent 0.63.0 / controller 0.98.3 | -| Publish | PUT `…/felhom-golden/0.98.3/golden.tar.zst` → **201**; round-trip GET sha **matches** (publish-agent.sh pattern mirrored) | -| Vouch | operator vouched golden **0.98.3** in the Day-0 manifest (agent 0.63.0 already vouched — the v0.63.0 follow-up is DONE); confirmed via `/api/v1/artifacts` | -| D — published-artifact Day-0 | PASS: virgin box, no local golden → `fetching golden v0.98.3 from Gitea` → `verified sha256 b9a02ef1… matches the hub manifest` → SUCCESS; first boot 0.98.3; up-to-date. The exact fresh-customer path | +## 3. Tests + the five §10 red-proof outcomes -Cleanup: all drill-1 hub rows deleted (8 tables, counts 0; demo-felhom/peti-felhom intact); drill VM -reverted to `virgin` and kept; bake credential env (0600) removed. Registry read-cred handled via -env only — never logged/committed. +`go build ./... && go vet ./... && go test ./...` — all green (20 pkgs; 18 with tests). +`visudo -cf` on felhom-pve: `parsed OK` (gated BEFORE install; binary restarted after). -## Deviations & observations +Red-proofs, each run → FAILED as required → reverted: +- (a) marker gate dropped → `re-registration after revocation: 5 calls` +- (b) hash comparison dropped → `steady-state tick ran execs: [install…, systemctl restart…]` +- (c) clamp removed → **honest adjustment discovered mid-proof:** x/crypto's X25519 clamps the + scalar internally (RFC 7748), so the DERIVED pubkey is clamp-invariant and the spec's + fixed-vector red-proof cannot fail. The clamp's real observable property is the **stored key + file's canonical form** (external `wg pubkey < private.key` must agree with what the agent + registered) — asserted by `TestEnsureKey_StoredKeyIsClamped`, which DOES fail with the clamp + removed (`stored key is not clamped: byte0=10101010`). Both pubkey vectors were generated with + the real `wg pubkey` (provenance in the test). +- (d) teardown-on-absent-data → `absent-data tick ran execs: [systemctl disable…]×3` +- (e) escrow auto-inject removed → `bundle key = ""` +Plus: the log-scan test greps the captured buffer for the actual private key (padded + unpadded) +— absent; the pubkey IS logged (intended). -- **Premature publish (caught + corrected):** the script's opt-in publish block triggers on the same - `REGISTRY_*` vars the pull needs — the Scenario A bake published immediately, before the Scenario C - gate. Package DELETEd (204, anon 404 verified) and re-published only after C passed. Candidate - cleanup (not done — no-"while-here" rule): a separate `GOLDEN_PUBLISH=1` opt-in. -- **SECURITY (operator follow-up):** the pull credential — the demo customer's `git.token`, i.e. - what every customer box holds — turned out to be a Gitea **admin token with package-WRITE** (the - premature publish proved it). Scope down to read-only + rotate. -- Installer v1.9.1's post-provision reboot is now a redundant belt (the path unit deploys first) — - removal recorded as a candidate installer cleanup, kept for now. -- Drill-env: `dhcpstart=10.0.2.30` + explicit `hostfwd…-10.0.2.15:22` on the QEMU command removes - the prior drill's slirp IP-collision quirk entirely. -- Guide follow-through (felhom.eu): **D.1b retired** to a one-line version check; troubleshooting - row covers pre-0.86.0 goldens; ledger B1/B5 FIXED; backlog note resolved. +## 4. Deploy + live validation (felhom-pve; journal excerpts verbatim; keys never logged by construction — the session-log grep for the key material was run and clean) + +Deploy order per spec: sudoers first (`visudo -cf` → install → `parsed OK`), binary +0.63.0→0.64.0 (backup kept), config: `wg_tunnel.enabled=true, interval_seconds=60`. + +- **Poll-interval lever (the spec's step-0 diagnosis, learned live):** agent-side + `poll_seconds=120` was adopted for exactly ONE cycle — the hub envelope's + `poll_interval_seconds` is a hub-side CONSTANT (`defaultHostPollSeconds = 900`, + `hub/internal/api/handler.go`) silently adopted on the first cycle (the "changed" log fires + only on later cycles). Hub code is out of S3 scope, so validation ran on 900 s beats with + background watchers. Agent config restored to `poll_seconds=900` + the diagnosis SIGQUIT + restart documented below. **Observation for the hub backlog: make the constant configurable.** +- **Step 1 — bring-up (one shot, 3 s):** + `07:15:48 wgtunnel: generated new WG keypair pubkey=yV5hFF…TCg=` → + `07:15:49 wgtunnel: registered with hub … assigned_ip=10.77.0.2/32 existed=false generation=5 sync=ok` → + `07:15:49 desired: updated from hub generation=5` → + `07:15:50 wgtunnel: tunnel conf applied endpoint=ep0.felhom.eu:443 … action=enable`. + Perms verified: key 0600 in 0700 dir (felhom-agent), installed conf 0600 root. +- **Step 2 — data path:** `wg-quick@wg-felhom` active; in-tunnel ping 10.77.0.1 0% loss ~32 ms; + `curl https://10.77.0.1:8007/` = PBS login page. Hub `/admin/wg/peers` shows the peer BOUND to + `demo-felhom-01`. Report stanza in the hub's stored report_json: + `{"pubkey":"yV5hFF…","registered":true,"active":true,"last_handshake_age_s":28,"assigned_ip":"10.77.0.2/32"}`. + All 6 `wg-*` capability entries probe `status:ok`. +- **Agent-restart tolerance (bonus, via the diagnosis SIGQUIT):** on restart — key kept, NO + re-registration (marker gate), tunnel stayed active, desired refetched clean. +- **Step 3 — host reboot:** `wg-quick@wg-felhom` active from boot (unit persistence, before/without + the agent), tunnel pings, guest 9201 running, agent's first tick ran ZERO wgtunnel execs. +- **Step 4 — revocation drill:** admin DELETE 05:57:30Z → next poll + `08:10:52 desired: updated generation=6` + + `WARN wgtunnel: … hub revoked this peer; disabling the tunnel (marker kept; NO re-registration)` + → service inactive, marker present. Over the following 2 polls: **0** `wgtunnel: registered` + lines, **0** wg-related sudo execs (the load-bearing negatives — verbatim counts in §5). + Re-add (via the REGISTRATION endpoint — see §5's path insight) → next poll: + `08:55:52 desired: updated generation=8` + `wgtunnel: tunnel conf applied … action=enable`, + tunnel pings, **0 registration calls since the re-add** — same key, same /32, marker untouched. + Config lever restored (`poll_seconds` back to 900; restart clean — no keygen/register lines). +- **Step 5 — escrow:** `--selftest=escrow-create -storage felhom-pbs` (as root — the PBS key file + is root-readable only, the documented BUNDLE-slice reality) → + `escrow: identity bundle: +wg_private_key` (field name only), `identity=true`, blob 450 B + age-wrapped, self-verify OK. **Deviation from the spec's "blob uploaded":** run WITHOUT + `--upload` — uploading would overwrite the operator's real drill escrow with one whose R was + displayed into THIS session log (R was sed-redacted, but an unrecorded-R escrow is + unrecoverable and clobbering the prior blob breaks the operator's held R). The upload path is + pre-S3 code, unchanged by this slice — no coverage lost. +- **Step 6 — soak:** 30-min idle → handshake age **104 s** (<180 s; keepalive holding), active. + +## 5. Revocation-drill negatives (verbatim) + +Over the 31 minutes following the teardown (05:57:30Z DELETE → 06:10:52Z teardown → checked +06:42Z), with **2 heartbeats landed hub-side** in the window (count from `host_reports`): + +``` +'wgtunnel: registered' lines since teardown: 0 +'desired: updated' lines since teardown: 0 ← correct: generation never advanced (nothing + changed hub-side), so heartbeats were cached + no-ops — the polls happened, the fetch didn't +wg-related sudo execs since teardown: 0 +wg-quick@wg-felhom: inactive (stayed down) +``` + +**Re-add path insight (found live, load-bearing for operator docs):** re-adding via the S1 +registry endpoint (`POST /admin/wg/peers` with `host_id`) binds the peer but does NOT bump the +host's generation — the agent never learns the block returned. The correct operator re-add is +the S2 **registration endpoint with the global key** (`POST /hosts/{id}/wg`), which allocates, +bumps (gen 8 in the drill), and syncs in one step. The drill was redone through it. Backlog +note for S6: either make the admin registry add bump the bound owner (symmetric with the S2 +delete-bump) or mark it registry-plumbing in the UI. + +## 6. NOT yet live-validated — awaiting later work + +- **CGNAT/mobile-hotspot smoke** — the operator-assisted appendix (physical hotspot needed); + deferred non-blocking per spec. On failure it re-opens the DERP/Headscale fallback, not S3's code. +- PBS backup over the tunnel (**S4** — this slice deliberately left backup config untouched). +- DR consume of the escrowed WG key (**S5**). +- Tunnel-health surfacing/alerting in the hub UI (**S6** — the stanza ships now). +- Endpoint re-IP re-resolve watchdog beyond restart-time resolution (S6, noted in 06-doc S3 row). +- Gitea publish + Day-0 manifest vouch of 0.64.0 (**operator follow-ups** — publish-agent.sh + + the password-gated UI). + +## 7. Observations (noticed, not acted on) + +- The hub's `defaultHostPollSeconds = 900` constant silently overrides any agent-side cadence on + cycle 1 with no log line (first-cycle adoption is unlogged) — worth a hub-side config knob + + an INFO log for the first adoption. +- The pre-existing `pbs: cannot read token secret` WARN (non-root vs `/etc/pve/priv/...`) still + fires each cycle — S4's tunnel-target PBS with per-customer tokens resolves it structurally. +- `git add -A` hygiene: the sudoers deploy leaves `/root/felhom-agent.sudoers.bak-pre064` + + `agent.json.bak-pre064` on felhom-pve (intentional rollback artifacts, listed for the operator).