docs: S2 shipped — hub v0.33.0 CHANGELOG, peersync v1.0.1, 06 §3.2 wire shape + S2 status, CONTEXT, REPORT

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
2026-07-04 01:20:10 +02:00
parent 99246c3c7d
commit 265f46c0e7
5 changed files with 164 additions and 82 deletions
+15
View File
@@ -3,6 +3,21 @@
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
- **2026-07-04 — S2 SHIPPED: box-facing WG registration + wireguard desired-state block + /offsite UI (hub v0.33.0)** —
doc-06 row S2. DECISIONS: the WG assignment is **hub-owned state, merged into served
desired-state at READ time** — stored `desired_json` stays a pure operator blob; admin PUT with
a top-level `wireguard` key → 400 (no copy-paste clobber); **re-key-in-place keeps the /32**
(stable tunnel addressing across rotation/DR); one bound peer per host (partial unique index);
generation bumps ONLY on real change (register/re-key/adopt/bound-delete — idempotent paths
move nothing). `POST /hosts/{id}/wg` self-scoped per-host key; NEW golden
`desired-state-wireguard.golden.json` = the S3 cross-repo contract (byte-identical agent copy
rule). peersync **v1.0.1** (strip out of process substitution — the S1 exit-swallow finding;
sandbox red-proofed) redeployed to the dev endpoint. Live-proven on the REAL demo-felhom-01
record via the real per-host key: gen 2→3, served block = golden shape, endpoint `wg show`
carried the peer, v0.63.0 agent tolerated the unknown key (fetch clean, guests untouched),
then removed → gen 4, all baselines restored. `/offsite` page unit-proven (live view is
operator-password-gated — eyeball after next login). Next: **S3** (agent keygen +
registration + `wg-felhom` service + escrow join + CGNAT smoke — risky/supervised).
- **2026-07-04 — S1 SHIPPED: offsite endpoint live + hub peer-sync (hub v0.32.1)** — the doc-06
roadmap's first slice, done-criterion met live. Peer-sync design point RESOLVED: **hub pushes
over SSH** to a forced-command `felhom-peersync` script (pull rejected: weakens immediate
+82 -76
View File
@@ -4,99 +4,105 @@
---
# REPORT — S1: offsite endpoint provisioning + hub→endpoint WG peer-sync (2026-07-04)
# REPORT — S2: box-facing WG registration + `wireguard` desired-state block + /offsite UI (2026-07-04)
Doc-06 roadmap row S1, shipped and live-validated end-to-end. Hub v0.31.0**v0.32.1** live on
k3s; dev endpoint (`felhom-hetzner`, 167.233.158.164) provisioned by the new runbook. The S1
design point is RESOLVED: **peer-sync = hub pushes over SSH** (pinned host key, forced-command
script) — pull/signed-manifest and an HTTPS push API rejected per the operator-locked decision.
Doc-06 roadmap row S2 shipped: hub v0.32.1**v0.33.0** live on k3s; peersync script v1.0.0 →
**v1.0.1** redeployed to the dev endpoint. The agent was NOT touched; the S3 contract travels
via the new golden only.
## 1. Baselines
Start: felhom.eu @ `7fb20d5` (hub v0.31.0), felhom-agent @ `4ba1b14` (untouched, read-only).
Start: felhom.eu @ `740cc42` (hub v0.32.1, peersync v1.0.0), felhom-agent @ `4ba1b14` (v0.63.0,
read-only). Live baseline captured before validation: `demo-felhom-01` at generation **2**,
desired-state `{"guests":[]}`, WG registry empty, guest 9201 running (controller 0.98.3
healthy), agent v0.63.0, poll interval 900 s.
## 2. Files created/modified
- `hub/internal/store/store.go` (migration: `wg_endpoints` + `wg_peers`), NEW `hub/internal/store/wg.go` + `wg_test.go`
- NEW `hub/internal/wgsync/{client,reconciler}.go` + tests (incl. in-process SSH server)
- NEW `hub/internal/api/wg.go` + `wg_test.go`; `hub/internal/api/handler.go` (5 routing cases + `wgSyncer` field)
- `hub/cmd/hub/main.go` (env wiring, disabled-INFO pattern)
- NEW `scripts/felhom-peersync.sh` (v1.0.0, shellcheck-clean on 180)
- `manifests/hub.yaml` (image 0.32.1, `WG_ENDPOINT_SSH_*` env, optional Secret mount `wg-endpoint-ssh`)
- NEW `documentation/runbooks/offsite-endpoint.md` (executed live; live-run corrections folded in)
- `hub/CHANGELOG.md`, `scripts/CHANGELOG.md`, `documentation/architecture/06-offsite-connectivity.md` (S1 row → SHIPPED), `CONTEXT.md`, this REPORT
- `hub/internal/store/store.go` (partial unique index `idx_wg_peers_host`; `BumpHostDesired`),
`hub/internal/store/wg.go` (+`allocateWGPeerTx` extraction, `RegisterWGPeerForHost`,
`GetWGPeerForHost`, `WGPeer.CreatedAt`), `wg_test.go` (Group A)
- `hub/internal/api/wg.go` (`handleRegisterHostWG`, `mergeWireguard`, delete-bumps-owner),
`handler.go` (routing case; merge call in `handleGetDesiredState`; `wireguard`-key rejection
in `handleAdminSetDesiredState`), `wg_test.go` (Group B), NEW
`testdata/desired-state-wireguard.golden.json` (**S3 cross-repo contract**)
- `hub/internal/web/offsite.go` + `templates/offsite.html` + `offsite_test.go` (Group C);
`server.go` routing; Offsite nav link in all 9 page templates
- `scripts/felhom-peersync.sh` v1.0.1 (exit-swallow fix)
- `manifests/hub.yaml` (image 0.33.0), `hub/CHANGELOG.md`, `scripts/CHANGELOG.md`,
`documentation/architecture/06-offsite-connectivity.md` (§3.2 concrete wire shape + S2 row
SHIPPED), `CONTEXT.md`, this REPORT
## 3. Commits (all pushed to `main`)
## 3. Commits (pushed to `main`)
1. `b18f6ae` store: tables + /32 allocator + Group-A tests
2. `fbeeacb` wgsync + /admin/wg API + main wiring + Groups B/C/D tests
3. `6f5fb19` peersync script + manifest surface
4. `0fa7ea1` **live-found fix**: constrain `HostKeyAlgorithms` to the pinned key type + regression test
5. `5e1933b` manifest → 0.32.1
6. `d2cc023` runbook (executed + corrected)
7. (this commit) CHANGELOGs + CONTEXT + 06 S1 status + REPORT
1. `fcf84a0` store: host binding + index + BumpHostDesired
2. `ba52005` API: registration + merge-at-read + hub-owned-key guard + golden
3. `13203c2` web: /offsite page + nav
4. (script commit) peersync v1.0.1 + `5e...` manifest 0.33.0 bump
5. (this commit) docs
## 4. Tests + red-proofs
## 4. Tests + the five §10 red-proofs
`cd hub && go build ./... && go vet ./... && go test ./...`**all green** (store/api/wgsync +
all pre-existing packages). Test count: 15 new test functions across 4 new test files
(`wg_test.go` ×2, `client_test.go`, `reconciler_test.go`); pre-existing tests untouched.
`cd hub && go build ./... && go vet ./... && go test ./...` — all green (12 packages).
New: 6 store tests (Group A), 5 API tests (Group B), 2 web render tests (Group C).
`TestDesiredStateGolden_RoundTripsThroughHub` passes **UNMODIFIED**; the S1 suites pass
unmodified through the allocator extraction.
All four §10 companion red-proofs run → **failed as required** → reverted → green:
- (a) allocator ignores existing rows → 3 tests fail (duplicate-IP surfaced by the UNIQUE backstop)
- (b) `isGlobal` gate dropped on POST → per-host-key 403 test fails ("sync ran despite 403")
- (c) `FixedHostKey``InsecureIgnoreHostKey` → wrong-host-key test fails ("the pin is dead")
- (d) full-list → delta-only push → removed-peer/full-list test fails ("surviving peer missing")
- (bonus) the new multi-hostkey regression test fails with the `HostKeyAlgorithms` fix removed
(that IS the live bug reproduced in-process).
Red-proofs, each run → FAILED as required → reverted:
- (a) self-scope check dropped → `cross-host register = 200, want 403`
- (b) merge made unconditional → caught by the no-peer-serves-no-block assertion
(`GET serves a wireguard block that was never legitimately registered`); the golden test
itself survived only via the fail-safe unmerged path (endpoint record absent in its setup) —
the pass-through property is covered either way
- (c) `wireguard`-key rejection dropped → `PUT with wireguard key = 200, want 400`
- (d) bump-on-idempotent → both negatives fail (`gen=2 want 1`, `push count 2 want 1`)
- (e) script exit-swallow (sandbox, stub `wg-quick` exit-1-after-partial + recording stub `wg`):
pre-fix shape invoked `wg` with rc=0 despite the strip failure; fixed shape errors first,
`wg` never invoked
## 5. Deployed + live validation (verbatim, secrets redacted)
## 5. Deploy + live validation (real pipeline, secrets redacted)
- Rollout: `deployment "hub" successfully rolled out`, image `felhom-hub:0.32.1`, log:
`[INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, user felhom-peersync)`.
- **Live bug found by validation:** first push failed `ssh: host key mismatch` — stock sshd holds
multiple host keys and default negotiation presented ECDSA against our ed25519 pin. Fixed
(v0.32.1) + regression-tested. The pin itself was proven working by the very failure.
- Scenario A live: POST peer → `{"assigned_ip":"10.77.0.2/32","sync":"ok"}` → endpoint
`wg show wg0` lists exactly that pubkey/IP; conf persisted; `[Interface]` section byte-identical.
- Scenario B live: DELETE → `{"status":"ok","sync":"ok"}` → 0 peers in `wg show` AND conf;
repeat DELETE → 404 (no sync); wrong bearer → 403.
- Script negatives on the box: malformed JSON / bad pubkey / own-IP peer → exit 1 each,
`wg show` + conf hash **byte-identical** before/after.
- Reboot: wg0 active on 443 with persisted set, nftables drop policies up, PBS active; next hub
push converges (`sync:"ok"`).
- Tunnel smoke (after operator created DNS): client on demo-felhom via `ep0.felhom.eu:443`
ping 10.77.0.1 (0% loss, ~38 ms) → `curl https://10.77.0.1:8007/` returns the **PBS login
page** (wg0-only 8007 rule works) → public `https://167.233.158.164:8007` **unreachable**
(P2b-style re-proof).
- **Incident (handled):** the first WG server private key leaked into the session log via
`wg show wg0 dump | cut` (field 1 of the interface line IS the private key). Rotated
immediately (new keypair + conf rebuild + hub re-registration, zero-peer window); warning
added to the runbook. No key material in any committed file.
- hub 0.33.0 rolled out via ArgoCD (`deployment "hub" successfully rolled out`, image
`felhom-hub:0.33.0`, `[INFO] WG peer-sync enabled`); peersync v1.0.1 on the endpoint
(`head` shows the v1.0.1 header), shellcheck-clean on 180.
- **Registration through the REAL per-host auth path** (curl run ON felhom-pve; its key never
left the box; throwaway keypair minted + private key shredded unused):
`{"assigned_ip":"10.77.0.2/32","existed":false,"generation":3,"sync":"ok"}` — gen 2→3.
- Served desired-state carried exactly the golden-shaped block (endpoint coords + pubkey +
assigned_ip) alongside the untouched `guests` key; endpoint `wg show wg0` listed the pubkey
with `10.77.0.2/32`.
- **Old-agent tolerance (the S3 fleet-safety gate), verbatim journal:**
`time=2026-07-04T01:04:29.267+02:00 level=INFO msg="desired: updated from hub" generation=3 guests=0`
— the v0.63.0 typed parse dropped the unknown key; guest count unchanged; NO error/warn from
the desired/reconcile path (the only WARN in the window is the pre-existing, documented
`pbs: cannot read token secret` non-root permission issue — unrelated to S2).
Guest-state re-capture = byte-identical to baseline (9201 running, controller 0.98.3 healthy).
- **Removal:** DELETE → `{"status":"ok","sync":"ok"}` → generation 4; desired-state back to the
pure operator blob (`{"guests":[]}`, no wireguard key); registry `{"peers":[]}`; endpoint
`wg show` 0 peers. Gen-4 clean fetch by the agent, verbatim:
`time=2026-07-04T01:19:29.228+02:00 level=INFO msg="desired: updated from hub" generation=4 guests=0`
— no errors; the box is fully back to baseline except the two expected generation increments.
- `/offsite` route wired + auth-gated live (302 → /login); page rendering proven by Group C
unit tests — the live view is operator-password-gated (eyeball on next login).
## 6. demo-felhom cleanup assertion (spike-§6 style)
**Generation accounting for demo-felhom-01:** baseline 2 → 3 (registration) → 4 (removal) —
exactly the two expected increments; desired_json blob never modified (byte-identity asserted
in tests; live blob returned to the identical `{"guests":[]}`).
`wgs1` interface down+gone ("Device does not exist"); throwaway client key + conf **shredded**;
no `s1-test`/`wgs1` files in /tmp; `/etc/wireguard/` empty, mtime Apr 2025 (untouched);
`wireguard` module unloaded. Hub registry empty (`{"peers":[]}`); endpoint `wg show` 0 peers.
The only persistent state anywhere: the endpoint box itself (intended — it IS the deliverable),
the hub DB endpoint row, and `Secret/wg-endpoint-ssh` (key exists only there; local copy shredded).
## 6. NOT yet live-validated — awaiting S3+
## 7. NOT yet live-validated — awaiting later slices
- Box-facing registration + `wireguard` block on `WireDesiredState` + generation bumps + hub UI (**S2**).
- Agent-managed `wg-felhom` on a real box + the CGNAT/mobile-hotspot smoke (**S3**).
- A real PBS **backup** over the tunnel into a customer namespace (**S4** — the smoke only proved
API reachability through the tunnel).
- Agent keygen/registration/`wg-felhom` service + IdentityBundle escrow join + CGNAT/mobile
smoke (**S3**, risky/supervised).
- A real PBS backup over the tunnel into a namespace (**S4**).
- DR re-establishment consuming the escrowed WG key (**S5**).
- UI mutations + tunnel-health columns (**S3/S6**).
## 8. Observations (noticed, not acted on)
## 7. Observations (noticed, not acted on)
- The endpoint's **AAAA record is wrong**: `2a01:4f8:1c16:7aa1::` (subnet zero address) instead
of `…7aa1::1` — v4 works; a v6-preferring client would fail. **Operator: fix the AAAA.**
- `~/build/felhom-hub/build.sh` does its own source sync (epilogue references
`~/git/felhom.eu`); its epilogue still suggests `kubectl set image` — contradicts the GitOps
rule (cosmetic, build script only).
- Home-resolver DNS propagation lagged public DNS by ~2 min (first `wg-quick up` failed to
resolve; retry succeeded) — noted in the runbook.
- Website renders no hub version — N/A for this release (grep confirmed).
- The agent's 900 s poll makes desired-state validation slow-loop (two ≤16-min waits in this
run) — S3 test plans should budget for it or temporarily shorten the interval on the test box.
- `hub-config`'s `report_api_key` doubles as the only global admin key; a scoped operator token
would narrow the blast radius of the endpoint-mutation surface (pre-existing, S6-adjacent).
- The pre-existing `pbs: cannot read token secret` WARN on felhom-pve (non-root agent vs
`/etc/pve/priv/storage/felhom-pbs.pw`) fires every cycle — known BUNDLE-slice issue, gets
resolved naturally when S4 points PBS at the tunnel with per-customer tokens.
@@ -63,11 +63,31 @@ New trust-boundary rows (extends 01 §5):
endpoint-mutation credential (out-of-band secret, never in desired-state).
- **Per-host peer entry** (joins the customer/host record, 01 §6): `wg_pubkey` (box-generated,
box-registered), `assigned_ip` (the box's tunnel `/32`, hub-assigned), `status`.
- **On the wire:** a `wireguard` block on `WireDesiredState` (`report.go:289` — sibling to
`Guests`/`RestoreDirective`, same forward-compat carriage rule as `report.go:284-288`: carried +
cached by old agents, acted on by new ones): endpoint coords + this box's assignment. Delivery
needs **zero new plumbing** — bump `DesiredGeneration`, the agent's existing syncer fetches
(`syncer.go:44-61`).
- **On the wire (S2-final — the shape S3 consumes; golden:
`hub/internal/api/testdata/desired-state-wireguard.golden.json`, agent copy must stay
byte-identical):** a `wireguard` key on the served desired-state, **merged at READ time**
the stored `desired_json` stays a pure operator blob (the hub REJECTS an admin PUT containing
a top-level `wireguard` key), hosts without a peer get byte-identical pass-through, and old
agents drop the unknown key in `encoding/json` (live-proven on v0.63.0):
```json
"wireguard": {
"endpoint": {
"dns_name": "ep0.felhom.eu",
"wg_port": 443,
"server_pubkey": "<44-char base64>",
"pbs_tunnel_ip": "10.77.0.1"
},
"pubkey": "<the box's registered pubkey>",
"assigned_ip": "10.77.0.2/32"
}
```
Client-side `AllowedIPs`, `PersistentKeepalive=25`, and `MTU 1420` are deliberately NOT wire
fields — S3 agent constants derived from `pbs_tunnel_ip` + §4. Delivery needs **zero new
plumbing** — registration/re-key/unbind bump `DesiredGeneration`, the agent's existing syncer
fetches (`syncer.go:44-61`). Re-key replaces the pubkey IN PLACE and keeps the `/32` (stable
addressing); one bound peer per host is index-enforced.
### 3.3 Provisioning handshake (Day-0 join)
@@ -204,7 +224,7 @@ block slices 1-2.
| Slice | Scope | Done = |
|---|---|---|
| **S1 — endpoint provisioning + peer-sync surface** — **SHIPPED 2026-07-04** (dev endpoint; hub v0.32.1). Peer-sync design point RESOLVED: **hub pushes over SSH** (`x/crypto/ssh`, `FixedHostKey` pin + pinned-type `HostKeyAlgorithms`) to a forced-command reconcile script; runbook `documentation/runbooks/offsite-endpoint.md`. Done-criterion met live: hub add/remove → `wg show` on the endpoint; tunnel `ep0.felhom.eu:443` → PBS through the wg0-only 8007 rule. | ~~provision the production VM…~~ (see runbook; production endpoint = a later re-run) | hub can add/remove a peer on the live endpoint via its credential; runbook re-creates the VM from nothing ✓ |
| **S2 — hub schema + desired-state** | endpoint coords + peer registry; `wireguard` block on `WireDesiredState`; box-facing pubkey-registration endpoint; generation bump on peer changes | a registered box's desired-state carries its assignment; hub UI shows the peer registry |
| **S2 — hub schema + desired-state** — **SHIPPED 2026-07-04** (hub v0.33.0). `POST /hosts/{id}/wg` (self-scoped) + merge-at-read `wireguard` block (§3.2 shape = the golden) + one-per-host index + re-key-in-place + `/offsite` UI page. Live-proven against the real felhom-pve record incl. old-agent (v0.63.0) tolerance. | ~~endpoint coords + peer registry…~~ | a registered box's desired-state carries its assignment; hub UI shows the peer registry |
| **S3 — agent WG service (the agent-managed-unit pilot)** | keygen + registration; reconcile `wg-felhom` from desired-state (sudoers additions on the `:16-18` pattern); re-resolve watchdog (§4.2); MTU 1420; IdentityBundle `+ WGPrivateKey`; tunnel-health into the report. **Risky/supervised class** (host service + sudoers + escrow change) | fresh box joins by §3.3 end-to-end; tunnel survives agent restart + endpoint re-IP; unhealthy tunnel alerts |
| **S4 — PBS over the tunnel** | point the box's PBS storage at `pbs_tunnel_ip:8007`; verify pin-through-tunnel | real backup lands in the box's namespace over the tunnel (nearly free — spike-proven path) |
| **S5 — DR re-establishment** | consume `WGPrivateKey` in restore mode; re-peer on re-provision | host-loss drill restores offsite connectivity with only the recovery code. *Merges with the DR-completeness arc.* |
+31
View File
@@ -1,5 +1,36 @@
# Felhom Hub — Changelog
## v0.33.0 — S2 offsite connectivity: box-facing WG registration + wireguard desired-state block + /offsite UI (2026-07-04)
Doc 06 roadmap row S2 (commits `fcf84a0`/`ba52005`/`13203c2`); the S2 architectural decision:
the stored `desired_json` stays a pure OPERATOR blob — the WG assignment is HUB-owned state,
**merged into the served desired-state at READ time**, never written into the store.
- **Store** (`internal/store/wg.go` + `store.go`): `RegisterWGPeerForHost` (idempotent /
**re-key-in-place keeps the /32** — stable tunnel addressing across rotation/DR / adopt-unbound
S1 rows / typed `ErrWGPubkeyBoundElsewhere` — a key is never silently stolen); partial unique
index `idx_wg_peers_host` = one bound peer per host; `BumpHostDesired` bumps ONLY the
generation (the merge changes served state, not the blob); `allocateWGPeerTx` extracted from
the S1 path behavior-neutrally (S1 tests unmodified). `WGPeer` gains `CreatedAt`.
- **API** (`internal/api/wg.go` + `handler.go`): `POST /hosts/{id}/wg` — per-host key
SELF-SCOPED (global = operator/DR path); generation bump + endpoint push ONLY on real change
(idempotent re-register moves nothing — asserted negatives). `mergeWireguard` injects
`{endpoint{dns_name,wg_port,server_pubkey,pbs_tunnel_ip}, pubkey, assigned_ip}` into served
desired-state; no peer → **byte-identical pass-through** (the cross-repo golden test passes
UNMODIFIED); any merge failure → fail-safe unmerged serve (never 500 the control channel).
`handleAdminSetDesiredState` REJECTS a top-level `wireguard` key (400 — an operator
copy-paste-PUT can never clobber the hub-owned block). Admin DELETE of a BOUND peer bumps the
owning host; unbound deletes move no generation. NEW golden
`testdata/desired-state-wireguard.golden.json` = **the S3 cross-repo contract** (agent copy
must stay byte-identical).
- **UI** (`internal/web/offsite.go` + `templates/offsite.html`): read-only `/offsite` page —
endpoint card + peer table (truncated pubkeys, full value in title; bound peers link to
`/hosts/<id>`); Offsite nav link in all 9 page templates. Mutations stay on the admin API
(UI actions arrive with tunnel health, S3/S6).
- **Tests:** Groups A/B/C; five red-proofs run + reverted (self-scope drop, unconditional
merge, rejection drop, bump-on-idempotent, script exit-swallow — see scripts/CHANGELOG
v1.0.1). Old-agent (v0.63.0) tolerance proven live against the real felhom-pve record.
## v0.32.0 + v0.32.1 — S1 offsite connectivity: WG endpoint record + peer registry + pinned-SSH peer-sync (2026-07-04)
The hub side of doc 06's roadmap row S1 (`documentation/architecture/06-offsite-connectivity.md`),
+10
View File
@@ -1,5 +1,15 @@
# Felhom scripts — Changelog
## felhom-peersync.sh v1.0.1 — strip out of process substitution (exit-swallow fix) (2026-07-04)
The S1 REPORT's exit-swallow class, fixed: `wg syncconf wg0 <(wg-quick strip "$tmp")` hid the
strip exit code — a corrupt `wg0.conf.head` could feed syncconf empty/partial input that WIPES
the live peer set while the script exits 0 (and the bad conf is then persisted). v1.0.1 runs
strip as its own step into `$tmpdir/stripped`; a strip failure aborts BEFORE `wg` is invoked.
Sandbox red-proof (stub `wg-quick` exiting 1 after partial output + recording stub `wg`):
pre-fix shape invoked wg and returned rc=0; fixed shape errors first, wg never called.
Redeployed to the dev endpoint (runbook step 5 install); shellcheck clean.
## felhom-peersync.sh v1.0.0 — the offsite endpoint's WG reconcile script (2026-07-04)
S1 (doc 06 §5): the forced-command target the hub's wgsync pushes to (runbook