felhom.eu/REPORT.md

felhom.eu — task reports

Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md; the scripts history lives in scripts/CHANGELOG.md.

---

REPORT — S1: offsite endpoint provisioning + hub→endpoint WG peer-sync (2026-07-04)

Doc-06 roadmap row S1, shipped and live-validated end-to-end. Hub v0.31.0 → v0.32.1 live on k3s; dev endpoint (felhom-hetzner, 167.233.158.164) provisioned by the new runbook. The S1 design point is RESOLVED: peer-sync = hub pushes over SSH (pinned host key, forced-command script) — pull/signed-manifest and an HTTPS push API rejected per the operator-locked decision.

1. Baselines

Start: felhom.eu @ 7fb20d5 (hub v0.31.0), felhom-agent @ 4ba1b14 (untouched, read-only).

2. Files created/modified

• hub/internal/store/store.go (migration: wg_endpoints + wg_peers), NEW hub/internal/store/wg.go + wg_test.go
• NEW hub/internal/wgsync/{client,reconciler}.go + tests (incl. in-process SSH server)
• NEW hub/internal/api/wg.go + wg_test.go; hub/internal/api/handler.go (5 routing cases + wgSyncer field)
• hub/cmd/hub/main.go (env wiring, disabled-INFO pattern)
• NEW scripts/felhom-peersync.sh (v1.0.0, shellcheck-clean on 180)
• manifests/hub.yaml (image 0.32.1, WG_ENDPOINT_SSH_* env, optional Secret mount wg-endpoint-ssh)
• NEW documentation/runbooks/offsite-endpoint.md (executed live; live-run corrections folded in)
• hub/CHANGELOG.md, scripts/CHANGELOG.md, documentation/architecture/06-offsite-connectivity.md (S1 row → SHIPPED), CONTEXT.md, this REPORT

3. Commits (all pushed to main)

1. b18f6ae store: tables + /32 allocator + Group-A tests
2. fbeeacb wgsync + /admin/wg API + main wiring + Groups B/C/D tests
3. 6f5fb19 peersync script + manifest surface
4. 0fa7ea1 live-found fix: constrain HostKeyAlgorithms to the pinned key type + regression test
5. 5e1933b manifest → 0.32.1
6. d2cc023 runbook (executed + corrected)
7. (this commit) CHANGELOGs + CONTEXT + 06 S1 status + REPORT

4. Tests + red-proofs

cd hub && go build ./... && go vet ./... && go test ./... — all green (store/api/wgsync + all pre-existing packages). Test count: 15 new test functions across 4 new test files (wg_test.go ×2, client_test.go, reconciler_test.go); pre-existing tests untouched.

All four §10 companion red-proofs run → failed as required → reverted → green:

• (a) allocator ignores existing rows → 3 tests fail (duplicate-IP surfaced by the UNIQUE backstop)
• (b) isGlobal gate dropped on POST → per-host-key 403 test fails ("sync ran despite 403")
• (c) FixedHostKey → InsecureIgnoreHostKey → wrong-host-key test fails ("the pin is dead")
• (d) full-list → delta-only push → removed-peer/full-list test fails ("surviving peer missing")
• (bonus) the new multi-hostkey regression test fails with the HostKeyAlgorithms fix removed (that IS the live bug reproduced in-process).

5. Deployed + live validation (verbatim, secrets redacted)

• Rollout: deployment "hub" successfully rolled out, image felhom-hub:0.32.1, log: [INFO] WG peer-sync enabled (endpoint 167.233.158.164:22, user felhom-peersync).
• Live bug found by validation: first push failed ssh: host key mismatch — stock sshd holds multiple host keys and default negotiation presented ECDSA against our ed25519 pin. Fixed (v0.32.1) + regression-tested. The pin itself was proven working by the very failure.
• Scenario A live: POST peer → {"assigned_ip":"10.77.0.2/32","sync":"ok"} → endpoint wg show wg0 lists exactly that pubkey/IP; conf persisted; [Interface] section byte-identical.
• Scenario B live: DELETE → {"status":"ok","sync":"ok"} → 0 peers in wg show AND conf; repeat DELETE → 404 (no sync); wrong bearer → 403.
• Script negatives on the box: malformed JSON / bad pubkey / own-IP peer → exit 1 each, wg show + conf hash byte-identical before/after.
• Reboot: wg0 active on 443 with persisted set, nftables drop policies up, PBS active; next hub push converges (sync:"ok").
• Tunnel smoke (after operator created DNS): client on demo-felhom via ep0.felhom.eu:443 → ping 10.77.0.1 (0% loss, ~38 ms) → curl https://10.77.0.1:8007/ returns the PBS login page (wg0-only 8007 rule works) → public https://167.233.158.164:8007 unreachable (P2b-style re-proof).
• Incident (handled): the first WG server private key leaked into the session log via wg show wg0 dump | cut (field 1 of the interface line IS the private key). Rotated immediately (new keypair + conf rebuild + hub re-registration, zero-peer window); warning added to the runbook. No key material in any committed file.

6. demo-felhom cleanup assertion (spike-§6 style)

wgs1 interface down+gone ("Device does not exist"); throwaway client key + conf shredded; no s1-test/wgs1 files in /tmp; /etc/wireguard/ empty, mtime Apr 2025 (untouched); wireguard module unloaded. Hub registry empty ({"peers":[]}); endpoint wg show 0 peers. The only persistent state anywhere: the endpoint box itself (intended — it IS the deliverable), the hub DB endpoint row, and Secret/wg-endpoint-ssh (key exists only there; local copy shredded).

7. NOT yet live-validated — awaiting later slices

• Box-facing registration + wireguard block on WireDesiredState + generation bumps + hub UI (S2).
• Agent-managed wg-felhom on a real box + the CGNAT/mobile-hotspot smoke (S3).
• A real PBS backup over the tunnel into a customer namespace (S4 — the smoke only proved API reachability through the tunnel).
• DR re-establishment consuming the escrowed WG key (S5).

8. Observations (noticed, not acted on)

• The endpoint's AAAA record is wrong: 2a01:4f8:1c16:7aa1:: (subnet zero address) instead of …7aa1::1 — v4 works; a v6-preferring client would fail. Operator: fix the AAAA.
• ~/build/felhom-hub/build.sh does its own source sync (epilogue references ~/git/felhom.eu); its epilogue still suggests kubectl set image — contradicts the GitOps rule (cosmetic, build script only).
• Home-resolver DNS propagation lagged public DNS by ~2 min (first wg-quick up failed to resolve; retry succeeded) — noted in the runbook.
• Website renders no hub version — N/A for this release (grep confirmed).