docs(spike): immediate-sync transport — poke-relay vs long-poll, both GO/complementary
Empirical spike (demo arc, docs-only, no product code). Measures the two hub->box 'sync now' transports. Verdict: (b) long-poll = primary for user config (hub-only, reaches controller, no ep0/WG/agent coupling; binding ceiling = twin 60s nginx proxy_read_timeout + hub WriteTimeout, both config lifts); (a) ep0-relayed poke = agent-plane nudge (~0.42s, zero ep0/box infra change, EKEYREJECTED-confined). Viktor Q2 answered: hub.felhom.eu is DNS-only, no Cloudflare. All live mutations reverted; arc re-verified green.
This commit is contained in:
@@ -2,63 +2,56 @@
|
||||
|
||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||||
|
||||
## Reinstall-of-existing-customer arc — F2/F3/F4/2.3 (hub v0.57.0) — 2026-07-16
|
||||
## SPIKE — immediate-sync transport (hub→box "sync now") — 2026-07-16
|
||||
|
||||
Closes the N100 physical-run findings (`documentation/tests/VALIDATION-n100-baremetal-2026-07-16.md`).
|
||||
**Only the hub shipped** — controller + scripts unchanged (both spec premises contradicted by source,
|
||||
below). Baseline: felhom.eu @ `dd961a6`, hub v0.56.0 → **v0.57.0**.
|
||||
**Docs-only spike.** No production code, no image, no manifest change. Every live mutation on the
|
||||
demo arc was inventoried and reverted; the arc was re-verified green. Findings doc:
|
||||
[`documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md`](documentation/audits/SPIKE-immediate-sync-transport-2026-07-16.md).
|
||||
Baselines re-confirmed at session start: felhom.eu `a50822af` v0.57.0 · felhom-agent `c040c180`
|
||||
v0.88.0 · felhom-controller `8f3564c1` v0.138.0.
|
||||
|
||||
### Fork verdicts (source-verified)
|
||||
- **2.1 = FORK B.** `hub/internal/claim/engine.go` + `handler.go:413-429` (`MarkClaimed`, set-only): the
|
||||
hub stores the claim code + a **claimed boolean**, never the password hash (controller-owned by the
|
||||
arc's design). Fork A (hub delivers the hash) would violate the design → **B** (bump generation + email).
|
||||
- **3.1 = no controller change (spec premise contradicted).** `controller/internal/agentapi/escrow.go`
|
||||
`EscrowPreflight` fetches the prereq checklist **live from the agent** (`GET /escrow/preflight`); it
|
||||
never sources `pbs_storage_id` from hub config. The agent seeds it on PBS-DR convergence. The fresh
|
||||
box "lacked" it only because F4's 403 blocked convergence — Part 0 + the demo grant restore it. **No
|
||||
controller v0.138.0.**
|
||||
### Question
|
||||
Viktor's ruling: a user-triggered config change must round-trip in **seconds**, not the 15-min
|
||||
report cycle. Measure the two candidate "sync now" transports empirically before any Task-2 spec.
|
||||
|
||||
### Part 0 — live unblock (done, reversible)
|
||||
Granted `FelhomAgentStore` on `/storage/felhom-offsite` (user + token) on felhom-pve. The agent's next
|
||||
tick converged: created the storage, ran `felhom-pbs-apply grant`, **seeded `escrow.pbs_storage_id`**,
|
||||
`pbsdr: converged state=applied`; `pvesm` shows **felhom-offsite pbs active**. The 403s stopped.
|
||||
### Verdict — both GO, and complementary (not competing)
|
||||
- **(b) long-poll — recommended primary for user config.** A hanging GET the hub completes on
|
||||
change. Reaches the **controller** (the config puller) with a **hub-only** change, **no ep0
|
||||
coupling, no WG dependency, no agent train**. **Binding ceiling today = 60 s**, from TWO places
|
||||
measured/sourced: nginx `proxy_read_timeout` **60 s** (proven from the upstream-timeout log line;
|
||||
ingress-nginx's default single retry pushes the client-visible 504 to ~120 s) **and** the hub's
|
||||
in-code `http.Server{WriteTimeout: 60s}` (`hub/cmd/hub/main.go:339-341`). Both are config lifts.
|
||||
- **(a) ep0-relayed poke — reserved as the agent-plane nudge.** Hub → forced-command on ep0 → ep0
|
||||
originates a contentless UDP datagram → box. Reaches the **agent** (WG terminates host-side),
|
||||
**~0.42 s** per fresh SSH session (~0.40 s of that is SSH setup; a pooled session → ~27 ms + 18 ms).
|
||||
Needs **zero infra change on ep0 or the box** and is cryptographically confined to registered
|
||||
peers — but needs an agent train (UDP listener → heartbeat trigger).
|
||||
- **Interval shortening stays REJECTED** (fleet-wide continuous load, no true immediacy).
|
||||
|
||||
### Part 2 — hub v0.57.0 (shipped)
|
||||
- **F2 claim continuity** — `claim.ReissueForReenroll` (claimed → bump generation once + reset email;
|
||||
no-op unclaimed), hooked at the host-enroll **mint path** (single-bump-per-re-enroll). Event
|
||||
`claim_reissued_reenroll`.
|
||||
- **F3 offsite continuity** — the re-enroll path calls `web.ReissueOffsiteForCustomer` (same machinery
|
||||
as the manual button; wired via `SetOffsiteReissuer`). Event `offsite_reissued`.
|
||||
- **2.3 escrow honesty** — `offsite.ReissueCredentials` marks the escrow stale (`store.MarkEscrowStale`;
|
||||
cleared by the next ceremony); the ACK **withholds** the mismatched `restic_pw_sha256`; the DR
|
||||
checklist shows **stale** not "done." Event `escrow_stale`.
|
||||
### Evidence highlights
|
||||
- **Viktor's Q2:** `hub.felhom.eu` is **DNS-only** (CNAME → `dooplex.hopto.org` → the operator home
|
||||
line `37.191.56.193`), **no Cloudflare**; split-horizon to the `nginx-internal` LB `192.168.0.192`
|
||||
on-LAN. Real chain: client → operator home-NAT → nginx-internal → hub pod.
|
||||
- **P1** ep0→box origination works with **`ip_forward` and the `forward` chain untouched** (endpoint-
|
||||
originated traffic never transits the forward chain); ~29 ms last hop; negative proven at the
|
||||
kernel crypto layer (`EKEYREJECTED` to an unregistered `/32`).
|
||||
- **P2** 5/5 pokes delivered end-to-end; the hub image is Alpine (no ssh/bash/python) → a production
|
||||
poke must reuse the in-process `x/crypto/ssh` client (the `wgsync` pattern); a lost poke is
|
||||
harmless by construction.
|
||||
- **P3** tuned ingress holds cleanly to **600 s on-LAN**, and **the operator home NAT holds an idle
|
||||
held connection the full 600 s** — so the NAT is not the constraint (≤10 min); only the twin 60 s
|
||||
config limits are.
|
||||
|
||||
### Scenarios / red-proofs
|
||||
- **A (claim)**: `TestReissueForReenroll` — claimed rotates + reset email; unclaimed no-op. PASS.
|
||||
- **C (escrow, red-proof)**: `TestReissue_InvalidatesEscrow` — after a re-issue the escrow is STALE and
|
||||
the hash is withheld. **Red-proof run-fail-revert:** with `MarkEscrowStale` gutted → FAIL ("escrow
|
||||
must be STALE… advertising ceremony-done against a key the repo no longer accepts"); restored → PASS.
|
||||
- Green gate: `go build ./... && go vet ./... && go test ./...` all clean.
|
||||
- **Deployed:** `felhom-hub:0.57.0` (manifest `dd563a0`, ArgoCD Synced/Healthy, rolled out, clean
|
||||
startup incl. the `stale_at` migration). Controller/scripts/agent unchanged.
|
||||
- **Scenario E (live demo):** the agent's `/escrow/preflight` is **all 6 green** — `pbs_storage_id`
|
||||
(felhom-offsite), `dr_tier` applied, `age_binary`, `hub_upload`, `staged_secret`, `sudo_grant`;
|
||||
overall `ok`. The wizard's start is offered. **CC STOPPED — the ceremony is Viktor's step.**
|
||||
### Ground-truth drift caught since the 07-05 OOB spike (recorded as findings)
|
||||
- ep0 `ip_forward` is now **1** (was 0) — the OOB operator-peer feature has shipped; a **second real
|
||||
box** (`10.77.0.3`) is now on the arc (untouched).
|
||||
- The operator `/32` (`10.77.0.250`) is now **durably rendered** into the box's `wg-felhom` conf —
|
||||
the 07-05 finding-1 gap is closed.
|
||||
- Cleanup bar was set to the **live** baseline (`ip_forward=1`, root ak = 2), not the plan's stale
|
||||
"must be 0" assumption.
|
||||
|
||||
### Observation (agent-side, recorded not acted on)
|
||||
The agent seeds `escrow.pbs_storage_id` into agent.json on PBS-DR convergence, but the **running**
|
||||
agent's escrow-preflight kept reporting "not configured" until a `systemctl restart felhom-agent`
|
||||
(config reload). The seed writes the file but doesn't propagate to the live preflight path — a small
|
||||
agent-reload gap to fold into the R-22 agent train.
|
||||
|
||||
### F4 — NOT an installer fix (reported)
|
||||
The DR storage id lives in the agent-domain **pbs_dr descriptor** (`web/pbsdr.go`), provisioned post-WG
|
||||
— not in the installer-fetched config — so `step_agent_config()` can't derive it. The block is a
|
||||
bootstrap circularity: the agent's token-auth pre-check `GET /storage/<id>` 403s and aborts before its
|
||||
own root-run `felhom-pbs-apply grant`. **Root fix is agent-side → ROADMAP R-22 (agent train)**; the demo
|
||||
was unblocked live (Part 0). Default-storage-id customers (all new/Peti installs) already work — F4 only
|
||||
bites non-default ids. Per Viktor: do NOT flip the demo's descriptor id.
|
||||
|
||||
### NOT done
|
||||
The escrow **ceremony itself** — Viktor's S10 step (CC's boundary is prereqs-green; CC never runs the
|
||||
ceremony, touches R, or mints a recovery code).
|
||||
### Cleanup (verified)
|
||||
ep0 forced-command line removed (root ak 2→2, 0 spike lines), poke script shredded, `ip_forward`
|
||||
still 1, forward chain empty; k8s sleeper (deploy/svc/ingress/cm) deleted; throwaway keypair
|
||||
shredded; box nft empty, no listeners, no temp files. Arc green: tunnel live (`felhom-offsite`
|
||||
active), guest 9201 controller healthy, hub pod 1/1, reconciler still pushing 3 peers.
|
||||
|
||||
Reference in New Issue
Block a user