docs: 06-doc S3 SHIPPED (agent v0.64.0) + agent-side revocation semantics + CONTEXT/REPORT

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-04 08:58:44 +02:00
parent 4fe895eaa4
commit 76ee25e5e8
4 changed files with 48 additions and 100 deletions
@@ -3,6 +3,17 @@
 > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
 > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.

+- **2026-07-04 — S3 SHIPPED: agent WG tunnel (felhom-agent v0.64.0; docs-only here)** — the
+  doc-06 §3.3 handshake is now END-TO-END LIVE on demo: agent keygen → `POST /hosts/{id}/wg` →
+  desired-state `wireguard` block → agent-managed `wg-quick@wg-felhom` → PBS page over
+  10.77.0.1:8007 in 3 s. **`wg_tunnel.enabled` defaults FALSE agent-side** (rollouts to other
+  boxes are no-ops until the production endpoint exists). Revocation semantics completed (§3.5):
+  removed peer → agent disables + keeps its marker + NEVER re-registers; operator re-adds via the
+  pubkey the heartbeat report now carries. Escrow: `IdentityBundle.WGPrivateKey` auto-injected on
+  new escrow-create runs (pre-S3 blobs can't be retrofitted — honest fallback = fresh-key
+  re-register, /32 kept). OPEN: CGNAT/mobile-hotspot smoke (operator-assisted appendix, deferred
+  non-blocking); S4 = PBS over the tunnel; hub-side S6 = tunnel-health alerting on the new report
+  stanza. Detail: felhom-agent CHANGELOG v0.64.0 + REPORT.
 - **2026-07-04 — S2 SHIPPED: box-facing WG registration + wireguard desired-state block + /offsite UI (hub v0.33.0)** —
  doc-06 row S2. DECISIONS: the WG assignment is **hub-owned state, merged into served
  desired-state at READ time** — stored `desired_json` stays a pure operator blob; admin PUT with
@@ -4,105 +4,22 @@

 ---

-# REPORT — S2: box-facing WG registration + `wireguard` desired-state block + /offsite UI (2026-07-04)
+# REPORT — S3 wrap-up (docs-only in this repo; the slice lives in felhom-agent v0.64.0) (2026-07-04)

-Doc-06 roadmap row S2 shipped: hub v0.32.1 → **v0.33.0** live on k3s; peersync script v1.0.0 →
-**v1.0.1** redeployed to the dev endpoint. The agent was NOT touched; the S3 contract travels
-via the new golden only.
+felhom.eu carried **no code** in S3 — hub v0.33.0 (S2) already served the registration endpoint
+ merged `wireguard` block. This commit records the S3 outcome in the design of record:

-## 1. Baselines
+- `documentation/architecture/06-offsite-connectivity.md`: roadmap row **S3 → SHIPPED**
+  (agent v0.64.0; `wg_tunnel.enabled` defaults FALSE until the production endpoint exists);
+  §3.5 gains the **agent-side revocation semantics** (register-once marker; block-absent-from-
+  present-state = disable + marker kept + never self-re-register; re-add via
+  `POST /hosts/{id}/wg` with the global key — the raw registry add does NOT bump the host
+  generation, found live) and the rollout gate.
+- `CONTEXT.md`: S3 entry (end-to-end handshake live on demo: keygen → register → block →
+  wg-quick → PBS page in 3 s; revocation drill + 30-min soak + reboot persistence).

-Start: felhom.eu @ `740cc42` (hub v0.32.1, peersync v1.0.0), felhom-agent @ `4ba1b14` (v0.63.0,
-read-only). Live baseline captured before validation: `demo-felhom-01` at generation **2**,
-desired-state `{"guests":[]}`, WG registry empty, guest 9201 running (controller 0.98.3
-healthy), agent v0.63.0, poll interval 900 s.
-
-## 2. Files created/modified
-
- `hub/internal/store/store.go` (partial unique index `idx_wg_peers_host`; `BumpHostDesired`),
-  `hub/internal/store/wg.go` (+`allocateWGPeerTx` extraction, `RegisterWGPeerForHost`,
-  `GetWGPeerForHost`, `WGPeer.CreatedAt`), `wg_test.go` (Group A)
- `hub/internal/api/wg.go` (`handleRegisterHostWG`, `mergeWireguard`, delete-bumps-owner),
-  `handler.go` (routing case; merge call in `handleGetDesiredState`; `wireguard`-key rejection
-  in `handleAdminSetDesiredState`), `wg_test.go` (Group B), NEW
-  `testdata/desired-state-wireguard.golden.json` (**S3 cross-repo contract**)
- `hub/internal/web/offsite.go` + `templates/offsite.html` + `offsite_test.go` (Group C);
-  `server.go` routing; Offsite nav link in all 9 page templates
- `scripts/felhom-peersync.sh` v1.0.1 (exit-swallow fix)
- `manifests/hub.yaml` (image 0.33.0), `hub/CHANGELOG.md`, `scripts/CHANGELOG.md`,
-  `documentation/architecture/06-offsite-connectivity.md` (§3.2 concrete wire shape + S2 row
-  SHIPPED), `CONTEXT.md`, this REPORT
-
-## 3. Commits (pushed to `main`)
-
-1. `fcf84a0` store: host binding + index + BumpHostDesired
-2. `ba52005` API: registration + merge-at-read + hub-owned-key guard + golden
-3. `13203c2` web: /offsite page + nav
-4. `30d762d` peersync v1.0.1; `99246c3` manifest 0.33.0 bump
-5. `265f46c` docs (+ this hash-fix follow-up)
-
-## 4. Tests + the five §10 red-proofs
-
-`cd hub && go build ./... && go vet ./... && go test ./...` — all green (12 packages).
-New: 6 store tests (Group A), 5 API tests (Group B), 2 web render tests (Group C).
-`TestDesiredStateGolden_RoundTripsThroughHub` passes **UNMODIFIED**; the S1 suites pass
-unmodified through the allocator extraction.
-
-Red-proofs, each run → FAILED as required → reverted:
- (a) self-scope check dropped → `cross-host register = 200, want 403`
- (b) merge made unconditional → caught by the no-peer-serves-no-block assertion
-  (`GET serves a wireguard block that was never legitimately registered`); the golden test
-  itself survived only via the fail-safe unmerged path (endpoint record absent in its setup) —
-  the pass-through property is covered either way
- (c) `wireguard`-key rejection dropped → `PUT with wireguard key = 200, want 400`
- (d) bump-on-idempotent → both negatives fail (`gen=2 want 1`, `push count 2 want 1`)
- (e) script exit-swallow (sandbox, stub `wg-quick` exit-1-after-partial + recording stub `wg`):
-  pre-fix shape invoked `wg` with rc=0 despite the strip failure; fixed shape errors first,
-  `wg` never invoked
-
-## 5. Deploy + live validation (real pipeline, secrets redacted)
-
- hub 0.33.0 rolled out via ArgoCD (`deployment "hub" successfully rolled out`, image
-  `felhom-hub:0.33.0`, `[INFO] WG peer-sync enabled`); peersync v1.0.1 on the endpoint
-  (`head` shows the v1.0.1 header), shellcheck-clean on 180.
- **Registration through the REAL per-host auth path** (curl run ON felhom-pve; its key never
-  left the box; throwaway keypair minted + private key shredded unused):
-  `{"assigned_ip":"10.77.0.2/32","existed":false,"generation":3,"sync":"ok"}` — gen 2→3.
- Served desired-state carried exactly the golden-shaped block (endpoint coords + pubkey +
-  assigned_ip) alongside the untouched `guests` key; endpoint `wg show wg0` listed the pubkey
-  with `10.77.0.2/32`.
- **Old-agent tolerance (the S3 fleet-safety gate), verbatim journal:**
-  `time=2026-07-04T01:04:29.267+02:00 level=INFO msg="desired: updated from hub" generation=3 guests=0`
-  — the v0.63.0 typed parse dropped the unknown key; guest count unchanged; NO error/warn from
-  the desired/reconcile path (the only WARN in the window is the pre-existing, documented
-  `pbs: cannot read token secret` non-root permission issue — unrelated to S2).
-  Guest-state re-capture = byte-identical to baseline (9201 running, controller 0.98.3 healthy).
- **Removal:** DELETE → `{"status":"ok","sync":"ok"}` → generation 4; desired-state back to the
-  pure operator blob (`{"guests":[]}`, no wireguard key); registry `{"peers":[]}`; endpoint
-  `wg show` 0 peers. Gen-4 clean fetch by the agent, verbatim:
-  `time=2026-07-04T01:19:29.228+02:00 level=INFO msg="desired: updated from hub" generation=4 guests=0`
-  — no errors; the box is fully back to baseline except the two expected generation increments.
- `/offsite` route wired + auth-gated live (302 → /login); page rendering proven by Group C
-  unit tests — the live view is operator-password-gated (eyeball on next login).
-
-**Generation accounting for demo-felhom-01:** baseline 2 → 3 (registration) → 4 (removal) —
-exactly the two expected increments; desired_json blob never modified (byte-identity asserted
-in tests; live blob returned to the identical `{"guests":[]}`).
-
-## 6. NOT yet live-validated — awaiting S3+
-
- Agent keygen/registration/`wg-felhom` service + IdentityBundle escrow join + CGNAT/mobile
-  smoke (**S3**, risky/supervised).
- A real PBS backup over the tunnel into a namespace (**S4**).
- DR re-establishment consuming the escrowed WG key (**S5**).
- UI mutations + tunnel-health columns (**S3/S6**).
-
-## 7. Observations (noticed, not acted on)
-
- The agent's 900 s poll makes desired-state validation slow-loop (two ≤16-min waits in this
-  run) — S3 test plans should budget for it or temporarily shorten the interval on the test box.
- `hub-config`'s `report_api_key` doubles as the only global admin key; a scoped operator token
-  would narrow the blast radius of the endpoint-mutation surface (pre-existing, S6-adjacent).
- The pre-existing `pbs: cannot read token secret` WARN on felhom-pve (non-root agent vs
-  `/etc/pve/priv/storage/felhom-pbs.pw`) fires every cycle — known BUNDLE-slice issue, gets
-  resolved naturally when S4 points PBS at the tunnel with per-customer tokens.
+Full evidence: **felhom-agent REPORT.md** (journal excerpts, five red-proofs, the
+hub-poll-constant gotcha, the re-add-path insight). Still open: CGNAT/mobile-hotspot smoke
+(operator-assisted appendix), S4 PBS-over-tunnel, S5 DR consume, S6 tunnel-health alerting +
+the two S6 backlog notes (make `defaultHostPollSeconds` configurable + log first-cycle adoption;
+make the admin registry-add bump the bound owner or mark it registry-plumbing in the UI).
@@ -131,6 +131,16 @@ code, restores the WG key, and re-establishes the tunnel; hub-side the peer entr
 the endpoint: the box is off the offsite network immediately, auditable, no box cooperation
 needed.

+**Revocation semantics, agent side (S3-final):** the agent registers ONCE (key exists + no local
+marker). A `wireguard` block absent from a PRESENT desired-state = revocation → the agent
+disables the service, KEEPS its marker, and never re-registers on its own — revoked stays
+revoked until the operator re-adds the peer (the heartbeat report carries the box's pubkey as
+the recovery handle; **re-add via `POST /hosts/{id}/wg` with the global key** — the raw registry
+add does not bump the host's generation). The only automatic re-registration is the
+pubkey-mismatch (DR/re-key) path, backoff-bounded. Absent DATA (a failed fetch) is never a
+teardown signal. **Rollout gate:** `wg_tunnel.enabled` defaults FALSE agent-side until the
+production endpoint exists.
+
 ---

 ## 4. Robustness (production details beyond the spike)
@@ -225,7 +235,7 @@ block slices 1-2.
 |---|---|---|
 | **S1 — endpoint provisioning + peer-sync surface** — **SHIPPED 2026-07-04** (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: **hub pushes over SSH** (`x/crypto/ssh`, `FixedHostKey` pin + pinned-type `HostKeyAlgorithms`) to a forced-command reconcile script; runbook `documentation/runbooks/offsite-endpoint.md`. Done-criterion met live: hub add/remove → `wg show` on the endpoint; tunnel `ep0.felhom.eu:443` → PBS through the wg0-only 8007 rule. | ~~provision the production VM…~~ (see runbook; production endpoint = a later re-run) | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓ |
 | **S2 — hub schema + desired-state** — **SHIPPED 2026-07-04** (hub v0.33.0). `POST /hosts/{id}/wg` (self-scoped) + merge-at-read `wireguard` block (§3.2 shape = the golden) + one-per-host index + re-key-in-place + `/offsite` UI page. Live-proven against the real felhom-pve record incl. old-agent (v0.63.0) tolerance. | ~~endpoint coords + peer registry…~~ | a registered box's desired-state carries its assignment ✓; hub UI shows the peer registry ✓ |
-| **S3 — agent WG service (the agent-managed-unit pilot)** | keygen + registration; reconcile `wg-felhom` from desired-state (sudoers additions on the `:16-18` pattern); re-resolve watchdog (§4.2); MTU 1420; IdentityBundle `+ WGPrivateKey`; tunnel-health into the report. **Risky/supervised class** (host service + sudoers + escrow change) | fresh box joins by §3.3 end-to-end; tunnel survives agent restart + endpoint re-IP; unhealthy tunnel alerts |
+| **S3 — agent WG service (the agent-managed-unit pilot)** — **SHIPPED 2026-07-04** (agent v0.64.0; `wg_tunnel.enabled` DEFAULT FALSE until the production endpoint exists). `internal/wgtunnel`: keygen → marker-gated registration → `wg-quick@wg-felhom` reconciled from the served `wireguard` block (restart-not-reload; hash-gated zero-exec steady state); **revocation completed §3.5**: block absent from a PRESENT desired-state → disable + marker kept + never re-register (operator re-adds via the reported pubkey); absent DATA never tears down. `FELHOM_WG` sudoers (latest-handshakes is the ONLY wg read — `dump` leaks the private key); `IdentityBundle.WGPrivateKey` (pre-S3 blobs can't be retrofitted — S5 falls back to fresh-key re-register, /32 kept); report stanza = §4.6's tunnel-health input. Endpoint re-resolution: wg-quick resolves at unit (re)start; the manager restarts on conf change and the operator path covers re-IP (full §4.2 watchdog → S6 with tunnel-health alerting). Live on felhom-pve: up in 3 s, reboot-persistent, revocation drill, 30-min soak. | ~~keygen + registration…~~ | fresh box joins by §3.3 end-to-end ✓; tunnel survives agent restart + host reboot ✓; unhealthy-tunnel ALERTING → S6 (stanza shipped) |
 | **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) |
 | **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* |
 | **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit, CGNAT smoke-test closure if not yet done | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
@@ -1,5 +1,15 @@
 # Felhom scripts — Changelog

+## docs — 06-doc S3 row SHIPPED + agent-side revocation semantics (2026-07-04)
+
+Docs-only companion to **felhom-agent v0.64.0** (the S3 slice — keygen, registration,
+agent-managed `wg-quick@wg-felhom`, escrow join; live-validated on felhom-pve incl. revocation
+drill, reboot persistence, 30-min soak). 06-doc §3.5 now records: register-once marker,
+revoked-stays-revoked, re-add via the registration endpoint (the raw registry add doesn't bump
+the host generation — live finding), `wg_tunnel.enabled` default-FALSE rollout gate. S6 backlog
+notes added (hub poll constant configurable + first-adoption log; registry-add bump-or-label).
+CGNAT/mobile-hotspot appendix deferred (operator-assisted; §7's open validation stands).
+
 ## felhom-peersync.sh v1.0.1 — strip out of process substitution (exit-swallow fix) (2026-07-04)

 The S1 REPORT's exit-swallow class, fixed: `wg syncconf wg0 <(wg-quick strip "$tmp")` hid the