From 62b73dfa18e588454f2ae79a441fd1642eaf6362 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 16 Jul 2026 21:10:31 +0200 Subject: [PATCH] =?UTF-8?q?docs(hub):=20v0.58.0=20Direction-2=20wait=20cha?= =?UTF-8?q?nnel=20=E2=80=94=20REPORT=20+=20CONTEXT=20(live-validated:=2024?= =?UTF-8?q?0s=20no-annotation=20hold,=200.047s=20wake,=20restart=20no-stor?= =?UTF-8?q?m)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CONTEXT.md | 14 +++++++ REPORT.md | 107 ++++++++++++++++++++++++++++++----------------------- 2 files changed, 74 insertions(+), 47 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index d0f0d37..e08aa86 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,20 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-16 — DIRECTION-2 IMMEDIATE-SYNC SHIPPED: hub v0.58.0.** The hub→box "sync now" wait + channel (option b of `documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`). New + `internal/intent` in-memory per-customer generation notifier (Bump/Wait/Close; coalesce-to-latest; + race-closer; in-memory by design → a restart costs one harmless full-state report, never a storm). + `GET /api/v1/wait` long-poll: per-customer only (global key 400; A can't see B), 240 s hold with a + 25 s heartbeat newline that defeats nginx's 60 s `proxy_read_timeout` **without any ingress + annotation** (WriteTimeout lifted per-connection via `ResponseController`; global timeouts + untouched). Every operator-intent web handler bumps AFTER its store write (config CRUD, claim + resend, offsite re-issue/freeze, password regen, block/unblock, floors [global bumps all managed], + controller log-tail + log-bundle). `intentHub.Close()` before `server.Shutdown`. Pairs with + controller v0.140.0 (the long-poll client). The immediate-sync arc is COMPLETE for the controller + plane (Direction 1 = ctrl v0.139.0, Direction 2 = hub v0.58.0 / ctrl v0.140.0); the agent-plane + ep0-poke stays PARKED in the OOB arc. Detail: `hub/CHANGELOG.md` v0.58.0, `REPORT.md`. + - **2026-07-16 — REINSTALL ARC SHIPPED: hub v0.57.0 (F2/F3/2.3).** The clean-slate reinstall-of-existing-customer path is now first-class (Peti R-1 prerequisite). On host-enroll's mint path (single-bump-per-re-enroll): claim re-issue for a claimed customer (`ReissueForReenroll`, diff --git a/REPORT.md b/REPORT.md index 3c35f26..15c61d8 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,56 +2,69 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## SPIKE — immediate-sync transport (hub→box "sync now") — 2026-07-16 +## Direction-2 immediate-sync — hub→box wait channel (hub v0.58.0) — 2026-07-16 -**Docs-only spike.** No production code, no image, no manifest change. Every live mutation on the -demo arc was inventoried and reverted; the arc was re-verified green. Findings doc: -[`documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`](documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md). -Baselines re-confirmed at session start: felhom.eu `a50822af` v0.57.0 · felhom-agent `c040c180` -v0.88.0 · felhom-controller `8f3564c1` v0.138.0. +Implements option (b) of `documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`. An +operator action on the hub now reaches the box in **seconds**. Pairs with controller **v0.140.0** +(the long-poll client). Baselines: felhom.eu `10e07f57` → hub **v0.58.0** (deployed); controller +`f665bbed` → **v0.140.0**. -### Question -Viktor's ruling: a user-triggered config change must round-trip in **seconds**, not the 15-min -report cycle. Measure the two candidate "sync now" transports empirically before any Task-2 spec. +### What shipped (hub) +- **`internal/intent` — in-memory operator-intent notifier.** Per-customer generation counter + + waiter registry: `Bump` advances + wakes (coalesce to LATEST — counter, not queue); `Wait` + returns on gen-change / ctx / maxHold / Close, with a pre-register gen-check closing the + bump-before-connect race; `Close` wakes all. In-memory BY DESIGN (a restart resets generations → + the box's `!=` compare costs exactly one harmless full-state report, never a storm). No + persistence, no schema. +- **`GET /api/v1/wait` (api/wait.go).** `checkAuthCustomer`; per-customer only (global key → 400; + customer resolved from the key, no `customer_id` param → A can't observe B). 240 s hold with a + **25 s heartbeat newline** — nginx's `proxy_read_timeout` is measured between upstream reads, so + the heartbeat keeps the default 60 s from firing → **no ingress annotation / manifest change**. + Contentless `{"gen":N}` completion; per-connection write-deadline via `http.NewResponseController` + (global `http.Server` timeouts untouched). +- **Intent bumps (web), fire-after-commit.** config create/update/delete, claim resend, offsite + re-issue (UI + re-enroll seam), offsite freeze/unfreeze, retrieval-password regen, block/unblock, + per-customer floor, global floor (bumps every config-managed customer), controller log-tail + request, controller log-bundle request (the AGENT ring rides the heartbeat envelope — deliberately + not bumped). +- **Wiring + shutdown.** One `intent.New()` shared by web + API; `intentHub.Close()` before + `server.Shutdown`. nil-safe (unset hub → wait 503, bumps no-op). -### Verdict — both GO, and complementary (not competing) -- **(b) long-poll — recommended primary for user config.** A hanging GET the hub completes on - change. Reaches the **controller** (the config puller) with a **hub-only** change, **no ep0 - coupling, no WG dependency, no agent train**. **Binding ceiling today = 60 s**, from TWO places - measured/sourced: nginx `proxy_read_timeout` **60 s** (proven from the upstream-timeout log line; - ingress-nginx's default single retry pushes the client-visible 504 to ~120 s) **and** the hub's - in-code `http.Server{WriteTimeout: 60s}` (`hub/cmd/hub/main.go:339-341`). Both are config lifts. -- **(a) ep0-relayed poke — reserved as the agent-plane nudge.** Hub → forced-command on ep0 → ep0 - originates a contentless UDP datagram → box. Reaches the **agent** (WG terminates host-side), - **~0.42 s** per fresh SSH session (~0.40 s of that is SSH setup; a pooled session → ~27 ms + 18 ms). - Needs **zero infra change on ep0 or the box** and is cryptographically confined to registered - peers — but needs an agent train (UDP listener → heartbeat trigger). -- **Interval shortening stays REJECTED** (fleet-wide continuous load, no true immediacy). +### Tests + red-proofs (all run-fail-reverted) +- intent: coalesce-to-latest (return the as-of-register snapshot → `TestWait_CoalescesBurstToLatestGen` + fails) and race-closer (drop the pre-register check → `TestWait_RaceCloser_BumpBeforeWaitNotLost` + fails). Plus timeout/ctx/close/isolation/stress. api/wait: 401/400/503, completes-on-bump, + timeout-with-heartbeats, race-closer, customer isolation. Full hub suite green (`go build/vet/test + ./...`); intent race-tested on the build server. -### Evidence highlights -- **Viktor's Q2:** `hub.felhom.eu` is **DNS-only** (CNAME → `dooplex.hopto.org` → the operator home - line `37.191.56.193`), **no Cloudflare**; split-horizon to the `nginx-internal` LB `192.168.0.192` - on-LAN. Real chain: client → operator home-NAT → nginx-internal → hub pod. -- **P1** ep0→box origination works with **`ip_forward` and the `forward` chain untouched** (endpoint- - originated traffic never transits the forward chain); ~29 ms last hop; negative proven at the - kernel crypto layer (`EKEYREJECTED` to an unregistered `/32`). -- **P2** 5/5 pokes delivered end-to-end; the hub image is Alpine (no ssh/bash/python) → a production - poke must reuse the in-process `x/crypto/ssh` client (the `wgsync` pattern); a lost poke is - harmless by construction. -- **P3** tuned ingress holds cleanly to **600 s on-LAN**, and **the operator home NAT holds an idle - held connection the full 600 s** — so the NAT is not the constraint (≤10 min); only the twin 60 s - config limits are. +### Deploy (GitOps) +Image `felhom-hub:0.58.0` built + pushed on 180; `manifests/hub.yaml` bumped 0.57.0→0.58.0 (commit +`59eb1f8`); ArgoCD hard-refresh + patch-sync → **Synced/Healthy**, rollout complete, live image +`:0.58.0`, listening. Route smoke: `/api/v1/wait` → 401 (no/bad auth), 404 (wrong method). -### Ground-truth drift caught since the 07-05 OOB spike (recorded as findings) -- ep0 `ip_forward` is now **1** (was 0) — the OOB operator-peer feature has shipped; a **second real - box** (`10.77.0.3`) is now on the arc (untouched). -- The operator `/32` (`10.77.0.250`) is now **durably rendered** into the box's `wg-felhom` conf — - the 07-05 finding-1 gap is closed. -- Cleanup bar was set to the **live** baseline (`ip_forward=1`, root ak = 2), not the plan's stale - "must be 0" assumption. +### Live validation (demo arc, customer `demo-felhom`) +- **Completion-on-difference path (the code path a bump triggers):** a box-side curl with a + mismatched `?gen=999999` returned `{"gen":0}` in **0.047 s** through the real ingress — sub-100 ms + wake delivery, DNS-only nginx path. +- **Hold proof / no-annotation claim:** a box-side curl at `?gen=0` (matching the current + generation) held cleanly for **240.04 s** (HTTP 200 → `{"gen":0}`, **10 heartbeat newlines** + observed) — **surviving well past nginx's 60 s default with the 25 s heartbeat, no ingress + annotation applied**. The controller's + own Waiter logged **zero wait-channel errors over the observation window** (it holds + reconnects + silently), a second confirmation. **No-annotation claim: HELD; the ingress fallback was NOT needed.** +- **Scenario C — hub restart resilience (live):** `kubectl rollout restart deploy/hub` mid-hold → + the box's Waiter logged **exactly one** WARN (`wait channel error: unexpected EOF — backing off`), + reconnected cleanly, and fired **0 out-of-cycle reports** (the generation was 0 before and after, + so there was correctly nothing to deliver — no restart storm). The immediate-probe endpoint + returned 200 the moment the pod was ready. (Cosmetic: the "recovered" INFO is deferred until the + reconnected 240 s hold completes, since `pollOnce` blocks — the reconnect itself is immediate.) + The "exactly one fire" restart case needs a prior non-zero generation, i.e. an operator bump. +- **Round-trip save→applied via the operator UI** requires an operator login (CC lacks the operator + password by standing constraint) — the mechanism is proven by the unit tests + the live + completion-path proof, and the delivery chain (report ACK → config_version → `ConfigRefresher` + re-pull + self-restart) is the already-live v0.139.0 / v0.26.0 path (unchanged). -### Cleanup (verified) -ep0 forced-command line removed (root ak 2→2, 0 spike lines), poke script shredded, `ip_forward` -still 1, forward chain empty; k8s sleeper (deploy/svc/ingress/cm) deleted; throwaway keypair -shredded; box nft empty, no listeners, no temp files. Arc green: tunnel live (`felhom-offsite` -active), guest 9201 controller healthy, hub pod 1/1, reconciler still pushing 3 peers. +### Not changed / parked +Report ACK schema, the 15-min cycle, the 900 s agent poll, and the global `http.Server` timeouts are +byte-identical. The agent-plane ep0-relayed poke stays PARKED in the OOB / mutual-repair arc +(Viktor's ruling + spike P4).