From 740cc427344cbd636874840237304e185eee7dc1 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 4 Jul 2026 00:14:35 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20S1=20shipped=20=E2=80=94=20hub=20v0.32.?= =?UTF-8?q?0/0.32.1=20CHANGELOG,=20peersync=20v1.0.0,=2006=20S1=20status,?= =?UTF-8?q?=20CONTEXT,=20REPORT?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CONTEXT.md | 16 +++ REPORT.md | 126 ++++++++++++------ .../architecture/06-offsite-connectivity.md | 2 +- hub/CHANGELOG.md | 39 ++++++ scripts/CHANGELOG.md | 15 +++ 5 files changed, 154 insertions(+), 44 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 5b79387..08d4ad8 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,22 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-04 — S1 SHIPPED: offsite endpoint live + hub peer-sync (hub v0.32.1)** — the doc-06 + roadmap's first slice, done-criterion met live. Peer-sync design point RESOLVED: **hub pushes + over SSH** to a forced-command `felhom-peersync` script (pull rejected: weakens immediate + revocation; HTTPS API rejected: new binary + third public port). Endpoint = the dev + `felhom-hetzner` box, provisioned by the NEW runbook `documentation/runbooks/offsite-endpoint.md` + (WG on **443/udp**, subnet 10.77.0.0/24, PBS datastore `felhom-offsite` at `/srv/pbs-felhom`, + nftables drop-all + wg0-only 8007, `ep0.felhom.eu` grey-cloud). Hub: `wg_endpoints`/`wg_peers` + + /32 allocator + `/admin/wg/*` (global-key, pubkey-in-body) + `internal/wgsync` (FixedHostKey pin + + **pinned-type HostKeyAlgorithms — live-caught bug**: multi-hostkey sshd presented ECDSA vs the + ed25519 pin; fixed v0.32.1 + regression test) + full-list declarative reconciler (5-min drift + repair). Live: add/remove→wg show, malformed→state-untouched, reboot→persist+converge, client + tunnel via ep0:443→PBS login page, public 8007 closed. INCIDENT handled: first WG server key + leaked via `wg show dump` into the session log → rotated immediately (warning now in the + runbook). OPERATOR follow-up: **fix the AAAA record** (currently `…7aa1::`, must be `…7aa1::1`). + Next: **S2** (hub schema host-join + `wireguard` block on WireDesiredState + box-facing + registration + UI). - **2026-07-03 — OFFSITE-CONNECTIVITY DESIGN-OF-RECORD WRITTEN (architecture Part 06)** — `documentation/architecture/06-offsite-connectivity.md`: records the six settled decisions (plain WG / host-side agent-managed `wg-felhom` as the agent-managed-unit PILOT / one shared diff --git a/REPORT.md b/REPORT.md index e6be8ae..eb4be65 100644 --- a/REPORT.md +++ b/REPORT.md @@ -4,59 +4,99 @@ --- -# REPORT — architecture Part 06: offsite-connectivity design-of-record (2026-07-03) +# REPORT — S1: offsite endpoint provisioning + hub→endpoint WG peer-sync (2026-07-04) -Docs-only. Created `documentation/architecture/06-offsite-connectivity.md` (design-of-record for -the offsite-backup transport); resolved the `day0-install.md` backlog line to cite spike + design -doc; CONTEXT + scripts/CHANGELOG updated. No code, no version bump, no deploy, no endpoint -provisioning. +Doc-06 roadmap row S1, shipped and live-validated end-to-end. Hub v0.31.0 → **v0.32.1** live on +k3s; dev endpoint (`felhom-hetzner`, 167.233.158.164) provisioned by the new runbook. The S1 +design point is RESOLVED: **peer-sync = hub pushes over SSH** (pinned host key, forced-command +script) — pull/signed-manifest and an HTTPS push API rejected per the operator-locked decision. -## Baselines (verified fresh-pulled before writing) +## 1. Baselines -- felhom.eu @ `bf099f6` (spike doc commit; this task's commit lands on top) -- felhom-agent @ `4ba1b14` (v0.63.0, read-only grounding) +Start: felhom.eu @ `7fb20d5` (hub v0.31.0), felhom-agent @ `4ba1b14` (untouched, read-only). -## Grounding citations used (for reviewer validation against Gitea) +## 2. Files created/modified -| Claim in the doc | Cited at | -|---|---| -| Backup = outbound push; PBS at LAN address | `01-topology-and-trust.md` §7; `felhom-agent/internal/pbs/client.go:25-26` (`Server` documented as `"192.168.0.180"`, port default 8007) | -| Public edge HTTP-only, no public UDP | `felhom.eu/manifests/hub.yaml:233` (`ingressClassName: nginx-internal`) | -| TLS pin above WG, unchanged through tunnel | `felhom-agent/internal/pbs/pin.go:17-35` (exact-cert `VerifyPeerCertificate`); spike P2b (positive + negative through-tunnel proof) | -| Desired-state channel + generation gate | `felhom-agent/internal/hub/report.go:270` (`DesiredGeneration`), `report.go:289-296` (`WireDesiredState` + forward-compat siblings), `internal/desired/syncer.go:44-61` (fetch only on generation advance) | -| IdentityBundle to extend | `felhom-agent/internal/escrow/identity.go:24-27` (`{TunnelToken, PBSToken}`) | -| Agent already installs+enables host units | `felhom-agent/configs/felhom-agent.sudoers:16-18` (`install … /etc/systemd/system/*.mount`, `systemctl enable --now`) | -| Hub drives external infra via held token | `felhom.eu/hub/internal/cloudflare/unblock.go` (CF token, WAF mutation) | -| cloudflared = later adopter of the S3 pattern | `felhom-agent/internal/hub/cloudflared.go:9-12` (read-only prober, self-described "seam for the tunnel-management slice") | -| All measurements | `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (P0–P7) | +- `hub/internal/store/store.go` (migration: `wg_endpoints` + `wg_peers`), NEW `hub/internal/store/wg.go` + `wg_test.go` +- NEW `hub/internal/wgsync/{client,reconciler}.go` + tests (incl. in-process SSH server) +- NEW `hub/internal/api/wg.go` + `wg_test.go`; `hub/internal/api/handler.go` (5 routing cases + `wgSyncer` field) +- `hub/cmd/hub/main.go` (env wiring, disabled-INFO pattern) +- NEW `scripts/felhom-peersync.sh` (v1.0.0, shellcheck-clean on 180) +- `manifests/hub.yaml` (image 0.32.1, `WG_ENDPOINT_SSH_*` env, optional Secret mount `wg-endpoint-ssh`) +- NEW `documentation/runbooks/offsite-endpoint.md` (executed live; live-run corrections folded in) +- `hub/CHANGELOG.md`, `scripts/CHANGELOG.md`, `documentation/architecture/06-offsite-connectivity.md` (S1 row → SHIPPED), `CONTEXT.md`, this REPORT -## What the doc marks DECIDED (recorded, not re-litigated) +## 3. Commits (all pushed to `main`) -D1 plain WG (no Headscale for traversal — separable future fleet layer); D2 host-side -**agent-managed** `wg-felhom` (pilot for agent-managed host services; install-script placement -rejected as one-shot; in-guest rejected on trust grounds — spike P7 shows it *works*, so the -refutation is architectural); D3 one shared dumb hub-driven endpoint VM (WG + PBS, no agent); -D4 hub = source of truth, `wireguard` block on the existing desired-state channel; D5 one -datastore + per-customer namespaces; D6 PBS ON the VM — relay-through-DooPlex rejected -(non-scaling through the ~40 Mbit home uplink, couples all DR to home uptime; DooPlex PBS = -dev/demo only). +1. `b18f6ae` store: tables + /32 allocator + Group-A tests +2. `fbeeacb` wgsync + /admin/wg API + main wiring + Groups B/C/D tests +3. `6f5fb19` peersync script + manifest surface +4. `0fa7ea1` **live-found fix**: constrain `HostKeyAlgorithms` to the pinned key type + regression test +5. `5e1933b` manifest → 0.32.1 +6. `d2cc023` runbook (executed + corrected) +7. (this commit) CHANGELOGs + CONTEXT + 06 S1 status + REPORT -## What the doc marks DEFERRED / OPEN (honest ledger, §7) +## 4. Tests + red-proofs -- **Open validation (one):** true-CGNAT traversal — unmeasured (operator line = plain-NAT, spike - P0); closure = mobile-hotspot smoke test, thorough = Peti VM 110; does not block S1/S2. -- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — slice-1 design point. -- WG key-rotation mechanics; endpoint scaling/sharding/HU-hosting; endpoint-VM self-DR depth. +`cd hub && go build ./... && go vet ./... && go test ./...` — **all green** (store/api/wgsync + +all pre-existing packages). Test count: 15 new test functions across 4 new test files +(`wg_test.go` ×2, `client_test.go`, `reconciler_test.go`); pre-existing tests untouched. -## Slice roadmap set (each a future task) +All four §10 companion red-proofs run → **failed as required** → reverted → green: +- (a) allocator ignores existing rows → 3 tests fail (duplicate-IP surfaced by the UNIQUE backstop) +- (b) `isGlobal` gate dropped on POST → per-host-key 403 test fails ("sync ran despite 403") +- (c) `FixedHostKey` → `InsecureIgnoreHostKey` → wrong-host-key test fails ("the pin is dead") +- (d) full-list → delta-only push → removed-peer/full-list test fails ("surviving peer missing") +- (bonus) the new multi-hostkey regression test fails with the `HostKeyAlgorithms` fix removed + (that IS the live bug reproduced in-process). -S1 endpoint provisioning + peer-sync surface → S2 hub schema + desired-state → S3 agent WG -service (**risky/supervised**: sudoers + host service + escrow change) → S4 PBS-over-tunnel → -S5 DR re-establishment (merges with DR-completeness) → S6 monitoring/hardening. -**MVP = S1→S2→S3, then S4.** +## 5. Deployed + live validation (verbatim, secrets redacted) -## Author notes +- Rollout: `deployment "hub" successfully rolled out`, image `felhom-hub:0.32.1`, log: + `[INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, user felhom-peersync)`. +- **Live bug found by validation:** first push failed `ssh: host key mismatch` — stock sshd holds + multiple host keys and default negotiation presented ECDSA against our ed25519 pin. Fixed + (v0.32.1) + regression-tested. The pin itself was proven working by the very failure. +- Scenario A live: POST peer → `{"assigned_ip":"10.77.0.2/32","sync":"ok"}` → endpoint + `wg show wg0` lists exactly that pubkey/IP; conf persisted; `[Interface]` section byte-identical. +- Scenario B live: DELETE → `{"status":"ok","sync":"ok"}` → 0 peers in `wg show` AND conf; + repeat DELETE → 404 (no sync); wrong bearer → 403. +- Script negatives on the box: malformed JSON / bad pubkey / own-IP peer → exit 1 each, + `wg show` + conf hash **byte-identical** before/after. +- Reboot: wg0 active on 443 with persisted set, nftables drop policies up, PBS active; next hub + push converges (`sync:"ok"`). +- Tunnel smoke (after operator created DNS): client on demo-felhom via `ep0.felhom.eu:443` → + ping 10.77.0.1 (0% loss, ~38 ms) → `curl https://10.77.0.1:8007/` returns the **PBS login + page** (wg0-only 8007 rule works) → public `https://167.233.158.164:8007` **unreachable** + (P2b-style re-proof). +- **Incident (handled):** the first WG server private key leaked into the session log via + `wg show wg0 dump | cut` (field 1 of the interface line IS the private key). Rotated + immediately (new keypair + conf rebuild + hub re-registration, zero-peer window); warning + added to the runbook. No key material in any committed file. -None — no §2 decision looked wrong while writing; the one place the task's own earlier framing -was contradicted by evidence (guest-WG infeasibility) was already resolved by the spike and is -recorded as an architectural (not technical) refutation in D2. +## 6. demo-felhom cleanup assertion (spike-§6 style) + +`wgs1` interface down+gone ("Device does not exist"); throwaway client key + conf **shredded**; +no `s1-test`/`wgs1` files in /tmp; `/etc/wireguard/` empty, mtime Apr 2025 (untouched); +`wireguard` module unloaded. Hub registry empty (`{"peers":[]}`); endpoint `wg show` 0 peers. +The only persistent state anywhere: the endpoint box itself (intended — it IS the deliverable), +the hub DB endpoint row, and `Secret/wg-endpoint-ssh` (key exists only there; local copy shredded). + +## 7. NOT yet live-validated — awaiting later slices + +- Box-facing registration + `wireguard` block on `WireDesiredState` + generation bumps + hub UI (**S2**). +- Agent-managed `wg-felhom` on a real box + the CGNAT/mobile-hotspot smoke (**S3**). +- A real PBS **backup** over the tunnel into a customer namespace (**S4** — the smoke only proved + API reachability through the tunnel). +- DR re-establishment consuming the escrowed WG key (**S5**). + +## 8. Observations (noticed, not acted on) + +- The endpoint's **AAAA record is wrong**: `2a01:4f8:1c16:7aa1::` (subnet zero address) instead + of `…7aa1::1` — v4 works; a v6-preferring client would fail. **Operator: fix the AAAA.** +- `~/build/felhom-hub/build.sh` does its own source sync (epilogue references + `~/git/felhom.eu`); its epilogue still suggests `kubectl set image` — contradicts the GitOps + rule (cosmetic, build script only). +- Home-resolver DNS propagation lagged public DNS by ~2 min (first `wg-quick up` failed to + resolve; retry succeeded) — noted in the runbook. +- Website renders no hub version — N/A for this release (grep confirmed). diff --git a/documentation/architecture/06-offsite-connectivity.md b/documentation/architecture/06-offsite-connectivity.md index 8c89e91..7daa8cf 100644 --- a/documentation/architecture/06-offsite-connectivity.md +++ b/documentation/architecture/06-offsite-connectivity.md @@ -203,7 +203,7 @@ block slices 1-2. | Slice | Scope | Done = | |---|---|---| -| **S1 — endpoint provisioning + peer-sync surface** | provision the production VM (WG server, PBS, firewall, `/srv` datastore, namespaces); resolve the push-vs-pull peer-sync design point; re-provision runbook | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing | +| **S1 — endpoint provisioning + peer-sync surface** — **SHIPPED 2026-07-04** (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: **hub pushes over SSH** (`x/crypto/ssh`, `FixedHostKey` pin + pinned-type `HostKeyAlgorithms`) to a forced-command reconcile script; runbook `documentation/runbooks/offsite-endpoint.md`. Done-criterion met live: hub add/remove → `wg show` on the endpoint; tunnel `ep0.felhom.eu:443` → PBS through the wg0-only 8007 rule. | ~~provision the production VM…~~ (see runbook; production endpoint = a later re-run) | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓ | | **S2 — hub schema + desired-state** | endpoint coords + peer registry; `wireguard` block on `WireDesiredState`; box-facing pubkey-registration endpoint; generation bump on peer changes | a registered box's desired-state carries its assignment; hub UI shows the peer registry | | **S3 — agent WG service (the agent-managed-unit pilot)** | keygen + registration; reconcile `wg-felhom` from desired-state (sudoers additions on the `:16-18` pattern); re-resolve watchdog (§4.2); MTU 1420; IdentityBundle `+ WGPrivateKey`; tunnel-health into the report. **Risky/supervised class** (host service + sudoers + escrow change) | fresh box joins by §3.3 end-to-end; tunnel survives agent restart + endpoint re-IP; unhealthy tunnel alerts | | **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) | diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index a6ef495..9fe51d6 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,44 @@ # Felhom Hub — Changelog +## v0.32.0 + v0.32.1 — S1 offsite connectivity: WG endpoint record + peer registry + pinned-SSH peer-sync (2026-07-04) + +The hub side of doc 06's roadmap row S1 (`documentation/architecture/06-offsite-connectivity.md`), +resolving the slice-1 design point: **peer-sync = hub pushes over SSH** to a forced-command +reconcile script on the endpoint (pull/signed-manifest rejected — weakens immediate revocation; +HTTPS push API rejected — a new versioned binary + third public port for nothing). + +- **Store** (`internal/store/wg.go` + migration in `store.go`, commit `b18f6ae`): `wg_endpoints` + (single expected row "ep0") + `wg_peers` (presence = desired state; no status column — that's + the S2 host-join). `AddWGPeer` = one tx, idempotent on pubkey, lowest-free-host `/32` allocation + skipping network/`pbs_tunnel_ip`/broadcast, `UNIQUE(assigned_ip)` race backstop + one internal + retry; typed `ErrWGEndpointUnset`/`ErrWGSubnetExhausted`. +- **wgsync** (`internal/wgsync/`, commits `fbeeacb` + `0fa7ea1`): `x/crypto/ssh` push client with + `ssh.FixedHostKey` pin (no insecure fallback, ever) **+ `HostKeyAlgorithms` constrained to the + pinned key's type** — the live validation caught a stock multi-hostkey sshd presenting ECDSA + against the ed25519 pin (legitimate server refused); regression-tested with an in-process + dual-hostkey SSH server. Reconciler pushes the FULL peer list (never deltas — drift repair by + construction) on `Trigger()` or a 5-min tick; payload + `{"version":1,"interface":"wg0","peers":[{pubkey, allowed_ip}]}`, deterministic order. +- **API** (`internal/api/wg.go`): `PUT/GET /admin/wg/endpoint`, `POST/DELETE/GET /admin/wg/peers` + — GLOBAL key only (the `handleAdminSetDesiredState` gate); pubkey validated 44-b64/32-byte; + **DELETE takes the pubkey in the JSON body** (base64 `/`+`+` keep pubkeys out of URL paths); + mutation responses carry `sync: ok | deferred: | disabled` — the DB is the source of + truth, a failed push defers to the reconciler. +- **Wiring** (`cmd/hub/main.go`): `WG_ENDPOINT_SSH_{ADDR,USER,KEY_FILE,HOSTKEY}` env (key from + the mounted `Secret/wg-endpoint-ssh`, host key non-secret plain env); any piece missing → + `[INFO] WG peer-sync disabled` and mutations still work DB-only. +- **Tests:** allocator (exact IPs, freed-IP reuse, /30 exhaustion), API auth/validation with a + fake syncer, SSH client against an in-process server (exact payload bytes, stderr surfacing, + wrong-host-key refusal, multi-hostkey pin), reconciler (full-list, retry-on-tick, no-mutation + drift push, removed-peer-absent negative). Four red-proofs run and reverted (allocator-ignores- + rows, gate removal, InsecureIgnoreHostKey, delta-only push) — each failed its test. +- **Live-validated** end-to-end on the dev endpoint (`felhom-hetzner`, runbook + `documentation/runbooks/offsite-endpoint.md`): add → `wg show` on the box; delete → gone + (+404/403 paths); malformed payloads leave wg state byte-identical; endpoint reboot → persisted + set + hub push converges; client tunnel `ep0.felhom.eu:443` → PBS login page via the wg0-only + 8007 rule; public 8007 unreachable. v0.32.1 = the HostKeyAlgorithms fix (0.32.0 image was + already pulled by the cluster; tag kept immutable). + ## docs — Felhom skills introduced + CLAUDE.md refresh (2026-07-03) Repo-level docs work alongside v0.31.0 (no hub code in this entry): diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 35d2884..cd36dd4 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,5 +1,20 @@ # Felhom scripts — Changelog +## felhom-peersync.sh v1.0.0 — the offsite endpoint's WG reconcile script (2026-07-04) + +S1 (doc 06 §5): the forced-command target the hub's wgsync pushes to (runbook +`offsite-endpoint.md` step 5 installs it as `/usr/local/bin/felhom-peersync`, root:root 0755, +invoked via a one-line sudoers grant from the `felhom-peersync` user's `restrict,command=` +authorized_keys entry). Validate-FIRST design: jq contract check (version 1, interface wg0, +44-b64 pubkeys, `10.77.0.x/32` allowed_ips, never the endpoint's own .1) rejects on stderr with +exit 1 before touching anything; then head-file + generated `[Peer]` blocks into a same-fs tmp, +`wg syncconf <(wg-quick strip …)` from the TMP (exact-match: adds/removes without bouncing the +interface), and only on success the atomic `mv` to `/etc/wireguard/wg0.conf` — runtime and boot +config can never diverge in the failure direction. Zero-peer payload = valid wipe. Never reads +or prints the private key; no `wg-quick save`; no second mode. shellcheck-clean. Live-proven on +felhom-hetzner incl. the negatives (malformed JSON / bad pubkey / own-IP peer → exit 1, wg state +byte-identical) and reboot persistence. + ## docs — architecture Part 06: offsite connectivity design-of-record (2026-07-03) `documentation/architecture/06-offsite-connectivity.md` — the settled offsite-backup-transport