Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 50c090eeaa | |||
| 30972d8f54 |
+16
@@ -3,6 +3,22 @@
|
||||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||||
|
||||
- **2026-07-17 — SYSTEM-INITIATED IMMEDIACY wired per-plane (hub v0.63.0).** The immediate-sync arc
|
||||
covered only OPERATOR-initiated changes; **system-initiated** desired-state mutations bumped the
|
||||
generation silently, so a freshly onboarded box waited a full agent tick for state the hub had already
|
||||
minted (observed live at slice-C onboarding). Decision: wire the existing, live-proven notifiers into
|
||||
every system site on the CORRECT plane — agent-plane `poke.Notifier` at `PBSDRAutoProvision` (the exact
|
||||
lag), `ReissuePBSDR` (which also lifts the pbsdrheal reconciler's escalation with zero reconciler
|
||||
changes), `handlePBSDRReissue`, and the two admin api desired-state writers (new nil-safe `Poker` seam);
|
||||
controller-plane `intent.Hub.Bump` at `reissueOnReenroll`. All fire AFTER the store write only. The
|
||||
**pbsdrheal Restage path is deliberately poke-free** (no generation bump → the agent's 60 s pbsdr ticker
|
||||
is its pickup path — a poke there is a verified no-op); the WG register/delete bumps stay poke-free
|
||||
(undeliverable pre-tunnel / transport removed — the agent fast-tick SECONDARY owns the register leg).
|
||||
The **first-hold-baseline race** is still OPEN by design (a bump landing during a fresh controller's
|
||||
FIRST wait is recorded as baseline without firing — the ruled fix is carrying `intent_gen` in the report
|
||||
ACK, its own future task). Unit-tested + red-proofed; live legs fire only on real system events (fold
|
||||
into the rehearsal bind sequence). See REPORT.md + hub/CHANGELOG.md v0.63.0.
|
||||
|
||||
- **2026-07-17 — R-21 SLICE C SHIPPED: the universal secret-free ISO (hub v0.62.0 + scripts v1.19.0, LIVE).**
|
||||
The distributed ISO carries NO customer secret. A box booted from `build-felhom-iso.sh --pairing`
|
||||
self-registers as an UNCLAIMED APPLIANCE (`POST /api/v1/appliance/register`, keyed by SMBIOS-uuid +
|
||||
|
||||
@@ -2,86 +2,139 @@
|
||||
|
||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||||
|
||||
## R-21 slice C — the universal secret-free ISO (hub v0.62.0 + scripts v1.19.0) — 2026-07-17
|
||||
## Hub v0.63.0 — system-initiated immediacy (poke + intent bump at every mutation site) — 2026-07-17
|
||||
|
||||
**Baselines:** hub v0.61.0 → **v0.62.0**; scripts v1.18.0 (`3172df1`) → **v1.19.0**. Commits `36c5cd5`
|
||||
(code), `9964ae6` (manifest). Hub **v0.62.0 LIVE** on k3s (Synced/Healthy). R-21 now shipped A+B+C.
|
||||
### 1. Baseline used
|
||||
|
||||
### What shipped
|
||||
- **felhom.eu** @ `4c9b0e87063278cddbe51e3966a8f8592c7c18f2` ("docs: R-21 slice C …", 2026-07-17),
|
||||
clean tree on `main`, local == `origin/main` re-confirmed before starting.
|
||||
- **Hub** v0.62.0 → **v0.63.0**. Deploy manifest `manifests/hub.yaml` image line `felhom-hub:0.62.0` → `:0.63.0`.
|
||||
- Both immediate-sync transports were already PROVEN-LIVE (agent poke: agent v0.89.0 + hub v0.59.0,
|
||||
2026-07-17; intent wait channel: hub v0.58.0 / controller v0.140.0). This task is **call-site wiring
|
||||
only** — no new mechanism, no spike.
|
||||
|
||||
The last R-21 slice: a **generic, secret-free** ISO. Blank box → unattended install → the box
|
||||
registers itself at the hub as an *unclaimed appliance* → the operator binds it to a customer → the hub
|
||||
delivers `customer-id` + retrieval passphrase **once** → day-0 completes via the existing slice-A path.
|
||||
The distributed ISO carries no secret (§4.4). Slice-A direct mode (env-baked) survives unchanged.
|
||||
### 2. Files created / modified
|
||||
|
||||
### Registration payload (as implemented) + token model
|
||||
**Implementation (commit `30972d8`):**
|
||||
- `hub/internal/web/pbsdr.go` — +3 pokes: `PBSDRAutoProvision`, `ReissuePBSDR`, `handlePBSDRReissue` (each after the successful `SetHostDesired`).
|
||||
- `hub/internal/api/handler.go` — new nil-safe `Poker` interface + `poker` field + `SetPoker`; poke in `handleAdminSetDesiredState`; intent `Bump` in `reissueOnReenroll`.
|
||||
- `hub/internal/api/wg.go` — `PokeAllHosts` in `handleAdminSetOperatorPeer` (only on a successful `BumpAllHostGenerations`).
|
||||
- `hub/cmd/hub/main.go` — capture the one `poke.Notifier` and wire it to BOTH planes (`webServer.SetPoke(n)` + `apiHandler.SetPoker(n)`); startup log "web + api admin seams armed".
|
||||
- `hub/internal/web/pbsdr_poke_test.go` (new) — 5 web tests (Groups A, B).
|
||||
- `hub/internal/api/poke_seam_test.go` (new) — 5 api tests (Groups C, D).
|
||||
- `hub/CHANGELOG.md` — v0.63.0 entry (newest on top).
|
||||
- `REUSE.md` — new canonical-pattern row: "Immediate-sync notify (per plane)".
|
||||
|
||||
`POST /api/v1/appliance/register` body: `{uuid, macs[], ssh_host_pubkeys[], hw{product, cpu, mem_kb}}`.
|
||||
Keyed by **(uuid, mac_set)** — the N100 DMI verdict (serials = "Default string") + cheap-board duplicate
|
||||
SMBIOS UUIDs make the **MAC set the tiebreaker** (same uuid + different mac-set = distinct appliance).
|
||||
**Token model:** a random **256-bit** appliance token per registration, returned once, stored only as
|
||||
**sha256** (`token_hash`); it is the box's sole pre-day-0 credential, never logged. Invalidated on
|
||||
discard (`token_hash` blanked + status sticky) and after consume. `GET /api/v1/appliance/poll` (Bearer
|
||||
token): unknown/discarded → **404** (no oracle) · unbound → **204** · bound → **200** + creds (consumed
|
||||
once) · delivered → **410**. The passphrase is read live from `customer_configs` (plaintext, as the
|
||||
day-0 command already requires) and never logged. register is the ONE unauthenticated endpoint, per-IP
|
||||
rate-limited (the ingress already geo-restricts to HU).
|
||||
**Docs (commit `<DOCS_HASH>`):**
|
||||
- `REPORT.md` (this file, overwritten), `CONTEXT.md` (decision recorded), `documentation/architecture/00-capability-map.md` (immediacy row note + version), `documentation/backlog/ROADMAP.md` (new R-28: agent fast-tick SECONDARY).
|
||||
|
||||
### Red-proofs (run-fail-revert, all recorded)
|
||||
**Manifest (commit `<MANIFEST_HASH>`):** `manifests/hub.yaml` image tag → `felhom-hub:0.63.0`.
|
||||
|
||||
- **One-shot delivery (C):** defeat `MarkApplianceDelivered`'s bound→delivered flip → the second poll
|
||||
re-delivers the passphrase (200) instead of 410 → `TestAppliancePoll_OneShotAnd410` FAILS → restored.
|
||||
- **Register idempotency (A):** drop the `(uuid, mac_set)` upsert branch → the re-register hits the
|
||||
UNIQUE backstop (500) → `TestApplianceRegister_Idempotent` FAILS → restored. (The UNIQUE index is the
|
||||
load-bearing backstop; the upsert is the graceful update.)
|
||||
- **Direct-mode regression (D):** `test/bootstrap-modes.sh` — a direct env drives `run_direct` and makes
|
||||
**ZERO** `/appliance/*` calls (fake hub records none); the pairing code path is provably not entered.
|
||||
Plus 404-no-oracle + sticky-discard, bind staging/refusal + no-host-count-gate, and the render test.
|
||||
### 3. Per-commit hashes pushed to `main`
|
||||
|
||||
### Deploy verification
|
||||
| # | Hash | Contents |
|
||||
|---|------|----------|
|
||||
| 1 | `30972d8` | code + tests + CHANGELOG + REUSE |
|
||||
| 2 | `<DOCS_HASH>` | REPORT + CONTEXT + capability-map + ROADMAP |
|
||||
| 3 | `<MANIFEST_HASH>` | manifests/hub.yaml image bump → 0.63.0 |
|
||||
|
||||
Hub 0.62.0 rolled out (Synced/Healthy, image confirmed). **Live smoke through the public ingress:**
|
||||
`POST /appliance/register` → 200 with a 256-bit token; poll (valid, unbound) → **204**; poll (bogus
|
||||
token) → **404** (no oracle). (The `smoke-test-uuid` record is CC test litter — discard on the hub.)
|
||||
### 4. Test results + red-proof outcomes
|
||||
|
||||
### Scenario E (the composition proof) + the secret-free artifact assertion
|
||||
**New tests (10), all green:**
|
||||
|
||||
- **Generic ISO built** `felhom-pve-9.2-1-v1.19.0-generic-generic.iso` (sha
|
||||
`f37080c835300b6cc2831bd6e5a483cad307f4d009469ee4206442fe9bebd348`). **Secret-free asserted on the
|
||||
artifact:** manifest `mode: pairing` / `secret-bearing: no`; the env decoded from the ISO's
|
||||
`/proxmox-first-boot` stub carries **only** `FELHOM_HUB_URL` + `FELHOM_INSTALL_URL` — no
|
||||
`FELHOM_CUSTOMER_ID`, no `FELHOM_RETRIEVAL_PASSPHRASE`.
|
||||
- **Nested boot → register (CONFIRMED live):** the generic ISO booted on scratch VM 311 (SeaBIOS, LAN,
|
||||
uuid `7ed8cb9e-…`, MAC `bc:24:11:c3:b3:d0`) → **zero-touch install** (answer file consumed, disk `sda`
|
||||
auto-selected, country HU) → reboot → installed system → first-boot pairing → **registered at the
|
||||
LIVE hub**: `[INFO] appliance registered: new unclaimed box (uuid=7ed8cb9e-…, macs=1, ssh_keys=3)` —
|
||||
the box gathered its MAC + all 3 SSH host keys exactly as designed. It now polls 204 (unbound),
|
||||
**staged for the operator bind**. The record + VM are left in place as the handoff. *(Boot-order
|
||||
lesson: the scratch VM must boot `scsi0;ide2` — disk-first, CD-fallback — or a CD-first order
|
||||
reinstalls in a loop and never reaches first-boot.)*
|
||||
- **The bind is the coordinated Viktor moment** (operator-password-gated; CC cannot log in). CC has
|
||||
staged everything: the box is registered and visible; a Viktor-created drill customer + the 2-minute
|
||||
bind click complete boot→register→**bind**→delivery→day-0, then the drill customer is torn down via
|
||||
RESET then Delete (re-exercising v0.61.0/v0.60.1). The bind→delivery→day-0 legs are proven by the
|
||||
hub unit tests + the `bootstrap-modes.sh` delivery handoff; the live composition folds into the
|
||||
supervised rehearsal, **which now runs the complete final product flow in one pass**.
|
||||
| Test | Group | Asserts |
|
||||
|------|-------|---------|
|
||||
| `TestPBSDR_AutoProvisionPokes` | A | auto-provision success → exactly one poke to the resolved WG /32 (10.77.0.2) within 2 s |
|
||||
| `TestPBSDR_AutoProvisionBlockedDoesNotPoke` | A (neg) | blocked precondition (no WG peer) → zero pokes |
|
||||
| `TestPBSDR_ReissueCorePokes` | B1 | `ReissuePBSDR` success → one poke to the host's /32 |
|
||||
| `TestPBSDR_ReissueCoreErrorDoesNotPoke` | B (neg) | tenantsync-error reissue → zero pokes (fire-after-commit) |
|
||||
| `TestPBSDR_HandleReissuePokes` | B2 | operator button success → one poke |
|
||||
| `TestAdminSetDesiredState_PokesTargetHostOnly` | C1 | admin-set success → PokeHost==1 (target only), PokeAllHosts==0; invalid-JSON path → 0 |
|
||||
| `TestAdminSetDesiredState_NilPokerNoPanic` | C1 (nil) | nil poker → mutation still 200, no panic |
|
||||
| `TestAdminSetOperatorPeer_PokesFleet` | C2 | operator-peer success → PokeAllHosts==1, PokeHost==0 |
|
||||
| `TestReenroll_BumpsIntent` | D | re-enroll → intent generation advances (synchronous) |
|
||||
| `TestReenroll_NilIntentHubNoPanic` | D (nil) | nil intent hub → enroll still 201, no panic |
|
||||
|
||||
### SSH-host-key pinning disposition
|
||||
Async web assertions synchronize on a buffered channel in the fake sender with a bounded `select`
|
||||
(≤2 s), never sleep-poll. The web fake sender structurally satisfies poke's unexported `pokeSender`
|
||||
seam (value passed, type never named); the notifier is a REAL `poke.Notifier` over the test store —
|
||||
exactly as `main.go` builds it. The api fake `Poker` satisfies the exported `api.Poker` interface and
|
||||
counts calls synchronously.
|
||||
|
||||
The registration payload carries the box's SSH host public keys; they are **stored** with the appliance
|
||||
record and their **fingerprints displayed** in the unclaimed-appliances UI. They are **not** auto-attached
|
||||
to the host on bind — the `hosts` row is minted later by the box's own `/host-enroll` (mint-once-reuse),
|
||||
and no clean "attach host keys to host" hand-off surface exists today. Per the spec's fallback, this is
|
||||
the store+display+**Observation** path, not new plumbing (a future item can wire TOFU-free pinning).
|
||||
**Red-proofs (run-fail-restore), one representative per scenario group:**
|
||||
|
||||
### Observations
|
||||
| Group | Removed | Test | Result |
|
||||
|-------|---------|------|--------|
|
||||
| A | `PBSDRAutoProvision` poke (Part 1.1) | `TestPBSDR_AutoProvisionPokes` | FAILED red ("no poke within 2s… 10.77.0.2") → restored |
|
||||
| B | `ReissuePBSDR` poke (Part 1.2) | `TestPBSDR_ReissueCorePokes` | FAILED red ("no poke within 2s… 10.77.0.2") → restored |
|
||||
| C | `handleAdminSetDesiredState` poke (Part 2.2) | `TestAdminSetDesiredState_PokesTargetHostOnly` | FAILED red ("PokeHost targets = [], want [h1]") → restored |
|
||||
| D | `reissueOnReenroll` bump (Part 3) | `TestReenroll_BumpsIntent` | FAILED red ("intent generation = 0, want > 0") → restored |
|
||||
|
||||
- **Bind never gates on host count** (multi-host customers are real — Peti); the picker shows counts as
|
||||
display only. A post-RESET customer is naturally hostless and binds fine (tested).
|
||||
- **Pre-bind provenance** is the `appliance_registrations` row's own timestamps (register/discard have no
|
||||
customer to scope an `events` row to — `events.customer_id` is NOT NULL); `appliance_bound` /
|
||||
`appliance_credential_delivered` events are recorded once a customer exists.
|
||||
- **Universal disk selection is out of slice-C scope:** `profiles/generic.profile` defaults to the first
|
||||
SATA disk (`sda`); NVMe/other targets need a profile variant (a future concern, noted in the profile).
|
||||
- **Poll cadence = systemd:** the bootstrap does one poll per invocation and exits non-zero when unbound,
|
||||
reusing `Restart=on-failure`/`RestartSec=30` as the 30s poll timer (no long-running-oneshot timeout).
|
||||
These tests fail on the pre-fix tree by construction; a representative removal per group was
|
||||
demonstrated (B2 and C2 share their group's wiring shape with the demonstrated B1/C1 removals).
|
||||
|
||||
### 5. Test count before/after
|
||||
|
||||
- Package `internal/web`: +5 tests (new `pbsdr_poke_test.go`).
|
||||
- Package `internal/api`: +5 tests (new `poke_seam_test.go`).
|
||||
- Full suite after: `go build ./... && go vet ./... && go test ./...` — **all packages green** (api + web
|
||||
are the only two exercising the new code; every other package unchanged).
|
||||
|
||||
### 6. Deployed version + pod/log verification
|
||||
|
||||
<DEPLOY_VERIFICATION>
|
||||
|
||||
### 7. NOT yet live-validated (real system-initiated events only)
|
||||
|
||||
The new pokes/bumps fire only on real system events, none manufacturable non-destructively from this
|
||||
task. Acceptance bar here = **unit tests + deploy + startup-log verification**. Unfired live legs, to be
|
||||
observed at the rehearsal's bind sequence (`RUNBOOK-rehearsal-v3 S5`):
|
||||
|
||||
- **Auto-provision poke** — fires when a DR-ON customer's host registers its WG peer for the first time
|
||||
and the atom provisions (the exact observed slice-C lag).
|
||||
- **Reconciler-escalation poke** — fires when the pbsdrheal reconciler escalates to `ReissuePBSDR` (a box
|
||||
stuck in `waiting_secret`/`consumed_failed` after losing its converged marker).
|
||||
- **Clean-slate re-enroll bump** — fires when a wiped, re-minted box re-enrolls for a claimed/offsite
|
||||
customer (on that path the controller usually does not exist yet; its startup fetch covers that shape).
|
||||
- **Operator-button / admin-api pokes** (`handlePBSDRReissue`, admin desired-state / operator-peer) — the
|
||||
operator-triggered legs; exercisable in a supervised session but not fired here.
|
||||
|
||||
The **first-hold-baseline race** stays OPEN by design: a bump landing during a fresh controller's FIRST
|
||||
wait is recorded as baseline without firing (the ruled fix is carrying `intent_gen` in the report ACK —
|
||||
its own future task, out of scope).
|
||||
|
||||
### 8. Part-4 audit table — EVERY desired-state mutation site
|
||||
|
||||
Re-grepped at implementation time for all non-test callers of `SetHostDesired`, `BumpHostDesired`,
|
||||
`BumpAllHostGenerations`, plus the system-initiated customer-deliverable writers. No forgotten fourth
|
||||
caller — every site below is accounted for.
|
||||
|
||||
| Site (file:landmark) | Initiator | Plane | Notification after this task | Reason if none |
|
||||
|---|---|---|---|---|
|
||||
| `web/pbsdr.go` `applyPBSDR` disable (~L143→148) | operator (form) | agent | poke (pre-existing) | — |
|
||||
| `web/pbsdr.go` `applyPBSDR` re-enable/storage-id (~L168→173) | operator (form) | agent | poke (pre-existing) | — |
|
||||
| `web/pbsdr.go` `applyPBSDR` fresh-provision (~L179→186) | operator (form) | agent | poke (pre-existing) | — |
|
||||
| `web/pbsdr.go` `pbsdrProvisionAtom` `SetHostDesired` (~L273) | — (shared atom) | agent | poke at each caller | callers poke: `applyPBSDR` (L186) + `PBSDRAutoProvision` (NEW) |
|
||||
| `web/pbsdr.go` `PBSDRAutoProvision` (~L312) | **system** (WG-reg hook) | agent | **NEW poke** | — (the observed lag) |
|
||||
| `web/pbsdr.go` `ReissuePBSDR` `SetHostDesired` (~L357) | **system** (reconciler escalation) | agent | **NEW poke** | — (also covers pbsdrheal, zero reconciler change) |
|
||||
| `web/pbsdr.go` `handlePBSDRReissue` `SetHostDesired` (~L409) | operator (button) | agent | **NEW poke** | — |
|
||||
| `api/handler.go` `handleAdminSetDesiredState` `SetHostDesired` (~L1419) | operator/system (admin api) | agent | **NEW poke** (`PokeHost`) | — |
|
||||
| `api/wg.go` `handleAdminSetOperatorPeer` `BumpAllHostGenerations` (~L504) | operator (admin api) | agent | **NEW poke** (`PokeAllHosts`, fire-after-commit) | — |
|
||||
| `api/wg.go` `handleRegisterWGPeer` `BumpHostDesired` (~L291) | system (agent registration) | agent | **none (deliberate)** | box's tunnel does not exist until it fetches the WG block — a poke is undeliverable by construction; the agent fast-tick SECONDARY (R-28) owns this leg |
|
||||
| `api/wg.go` WG peer delete `BumpHostDesired` (~L422) | operator | agent | **none (deliberate)** | the mutation removes the transport |
|
||||
| `pbsdrheal` **Restage** path | system (reconciler) | agent | **none (deliberate)** | no generation bump → the agent's 60 s pbsdr ticker is the pickup path (a poke is a verified no-op); `internal/pbsdrheal/` byte-unchanged |
|
||||
| `api/handler.go` `reissueOnReenroll` (F2 claim + F3 offsite, ~L1071) | **system** (clean-slate re-enroll) | controller | **NEW intent bump** | — |
|
||||
| `web/configs.go` bumpIntent ×11 (L564/633/657/702/742/779/796/842/859/876/1055) | operator | controller | intent bump (pre-existing) | — |
|
||||
| `web/customer_reset.go` bumpIntent (~L226) | operator | controller | intent bump (pre-existing) | — |
|
||||
| `web/logtail.go` bumpIntent (~L35) / `web/logbundle.go` bumpIntent (~L68) | operator | controller | intent bump (pre-existing) | — |
|
||||
|
||||
### 9. Observations (not acted on)
|
||||
|
||||
- **`handlePBSDRReissue` ↔ `ReissuePBSDR` duplication.** The operator button re-implements the reissue
|
||||
atom's tail (reissue → store secret → descriptor bump) instead of calling `ReissuePBSDR`, deliberately,
|
||||
to keep granular per-step HTTP error strings (the "keep in lockstep" comment marks the known
|
||||
duplication). Both now poke after their `SetHostDesired`. Not deduped (per spec forbidden-dedup); if
|
||||
ever unified, the handler's error-string granularity is the constraint to preserve.
|
||||
- **`pbsdrProvisionAtom` is poke-free by design** — its two callers poke on success (form path at L186,
|
||||
hook path at the new L312). This keeps the poke at the decision point (blocked vs minted) rather than
|
||||
inside the shared atom, matching the existing L186 structure.
|
||||
|
||||
@@ -131,6 +131,7 @@
|
||||
| Web POST action | hub/internal/web/configs.go `handleSetGlobalFloor` (~L602) | CSRF enforced centrally in web ServeHTTP; validate via `normalize*` helper; POST-redirect-GET with `?flash=` token; log INFO on success. |
|
||||
| Optional dependency injection | hub/internal/web/server.go `Set*` setters (~L131–148) / api handler `SetDispatcher` etc. | Constructor takes hard deps; optional subsystems wired via `SetX` after construction; nil = graceful degradation (never panic). |
|
||||
| Seam-injected sender for tests | hub/internal/notify/dispatcher.go `sendEmailFn` (~L33) | Function-field defaulting to the real impl in the constructor; tests overwrite it. Same idea: `mailRateLimiter.now`, `mailrelay.Sender` fake. |
|
||||
| Immediate-sync notify (per plane) | agent plane: `poke.Notifier` (`web.Server.poke` / `api.Handler.poker` via `SetPoke`/`SetPoker`) — controller plane: `intent.Hub.Bump` (`web.Server.bumpIntent`, `api.Handler.intentHub`) | EVERY desired-state mutation fires the RIGHT plane's notifier AFTER the successful store write, never on an error path (fire-after-commit). Agent-plane pokes a HOST when its generation moved (`SetHostDesired`/`Bump*HostDesired`); controller-plane bumps a CUSTOMER on a controller-visible change. Both receivers COALESCE bursts — add NO hub-side dedup. Deliberate non-sites need a documented reason (undeliverable pre-tunnel, transport removed, or no generation bump → the 60 s ticker is the pickup path). Both seams nil-safe: unset = the ≤15-min cycle still reconciles. Full site inventory: REPORT.md audit table (v0.63.0). |
|
||||
| Website page | website/index.html | UTF-8 **with BOM**; shared `<nav>`/`<footer>` byte-identical across pages (only `class="active"` differs); two-tone H1 = `<h1>…<span>accent</span></h1>`; all styling in website/assets/site.css tokens (`:root`) — zero embedded `<style>`; `?v=N` cache-bust on site.css/icons.svg; umami snippet; no CDN fonts; no emoji (sprite icons.svg instead). |
|
||||
| Gate script | scripts/site_gates.py | Byte-level mechanical gates (BOM, emoji codepoint ranges, nav/footer diff, analytics, banned tokens, cache-bust); run `python scripts/site_gates.py` after ANY website change; non-zero exit on failure. |
|
||||
| Fetch-validate-install (shell) | scripts/felhom-host-install.sh `step_agent_install` (~L1108) | `fetch_raw` to mktemp → syntax-check (`bash -n`) → `install -m0755 -o root -g root` → only then activate; guarded-mkfs wrapper installed BEFORE the sudoers that references it (ordering is the safety property). All mutations through `run()` (dry-run aware). |
|
||||
|
||||
@@ -108,7 +108,7 @@
|
||||
| Scenario | Components | Status | Evidence | Gap / roadmap |
|
||||
|---|---|---|---|---|
|
||||
| Customer/host management: 8-tab detail, scoped auto-refresh, safe stale-host deletion, capability chips | hub v0.47–0.53 | **PROVEN-LIVE** | hub v0.53.0 dead-host roll-up live on the Peti cluster (proxmox1 down 23h); `CAMPAIGN-4-2026-07-13` (operator UI driven live); `DRILL-day0-take2` F-16 (offsite/freeze buttons live) | 8-tab render + capability chips are **render-test-validated** (hub UI is password-gated; CC cannot log in). (Cited "daily operator use" was a no-doc citation; `AUDIT-hub-gui-2026-06-30` predates these features at hub v0.25) |
|
||||
| Config/state change round-trips in **seconds** (hub↔box immediacy; 15-min cycle stays the backbone): box→hub out-of-cycle report (Dir 1) + hub→box `GET /api/v1/wait` long-poll wake (Dir 2) | controller v0.139/140, hub v0.58 | **PARTIAL** | Transport proven live through the real DNS-only ingress: `SPIKE-immediate-sync-transport-2026-07-16` + hub v0.58.0 / controller v0.140.0 REPORTs — 240 s no-annotation hold (25 s heartbeat defeats nginx's 60 s `proxy_read_timeout`, no ingress change), 0.047 s wake-on-change, hub `rollout restart` = 1 WARN + 0-storm reconnect; Dir-1 2 s box→hub round-trip live in controller v0.139.0 | The operator-UI **save→apply** round-trip is not fired end-to-end live (hub UI password-gated; CC can't log in) → R-23; the wake transport and the ACK→config_version→`ConfigRefresher` delivery chain are each proven, only the UI-triggered bump leg is unexercised. Agent-plane (host-domain desired-state) poke **first slice PROVEN-LIVE** (Direction-2a, agent v0.89.0 + hub v0.59.0, 2026-07-17): contentless ep0-relayed UDP poke → agent immediate desired-state cycle, per `SPIKE-immediate-sync-transport-2026-07-16` P4. Full path live-proven: a real operator manifest save fired `poke: sync-poke delivered to 10.77.0.2`; the box (0.89.0) received it and logged `poke received → triggering an immediate desired-state cycle` → `out-of-band report triggered` — **~31 ms ep0→box, sub-ms to the report cycle** (WG-confined, from 10.77.0.1 to the 10.77.0.2-bound socket); save→tick ≈ ~0.45 s (SSH-dominated), well under ≤2–3 s. R-13 first slice (listener+sender only; the rest of the mutual-repair arc stays open) |
|
||||
| Config/state change round-trips in **seconds** (hub↔box immediacy; 15-min cycle stays the backbone): box→hub out-of-cycle report (Dir 1) + hub→box `GET /api/v1/wait` long-poll wake (Dir 2) | controller v0.139/140, hub v0.58/0.63 | **PARTIAL** | Transport proven live through the real DNS-only ingress: `SPIKE-immediate-sync-transport-2026-07-16` + hub v0.58.0 / controller v0.140.0 REPORTs — 240 s no-annotation hold (25 s heartbeat defeats nginx's 60 s `proxy_read_timeout`, no ingress change), 0.047 s wake-on-change, hub `rollout restart` = 1 WARN + 0-storm reconnect; Dir-1 2 s box→hub round-trip live in controller v0.139.0 | The operator-UI **save→apply** round-trip is not fired end-to-end live (hub UI password-gated; CC can't log in) → R-23; the wake transport and the ACK→config_version→`ConfigRefresher` delivery chain are each proven, only the UI-triggered bump leg is unexercised. Agent-plane (host-domain desired-state) poke **first slice PROVEN-LIVE** (Direction-2a, agent v0.89.0 + hub v0.59.0, 2026-07-17): contentless ep0-relayed UDP poke → agent immediate desired-state cycle, per `SPIKE-immediate-sync-transport-2026-07-16` P4. Full path live-proven: a real operator manifest save fired `poke: sync-poke delivered to 10.77.0.2`; the box (0.89.0) received it and logged `poke received → triggering an immediate desired-state cycle` → `out-of-band report triggered` — **~31 ms ep0→box, sub-ms to the report cycle** (WG-confined, from 10.77.0.1 to the 10.77.0.2-bound socket); save→tick ≈ ~0.45 s (SSH-dominated), well under ≤2–3 s. R-13 first slice (listener+sender only; the rest of the mutual-repair arc stays open). **System-initiated immediacy wired (hub v0.63.0, this REPORT):** the mutation sites that only OPERATOR actions used to notify now fire the correct plane's notifier when the hub itself mints state — agent-plane pokes at `PBSDRAutoProvision` (the observed slice-C lag), `ReissuePBSDR` (also the pbsdrheal escalation), `handlePBSDRReissue`, and the two admin desired-state api writers; controller-plane bump at `reissueOnReenroll`. Unit-tested + red-proofed, not yet fired on a real system event (folds into the rehearsal bind sequence). Still PARTIAL: the R-23 operator-UI save→apply leg and the agent **fast-tick-until-first-convergence** SECONDARY (the WG-registration leg a poke can't reach pre-tunnel) remain unfired live |
|
||||
| Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules | hub v0.45/0.53, agent | **PARTIAL** | `runbooks/publish-train-rules.md`; demo-fleet updates proven | **Never proven on a real remote customer** — parked trains `RUNBOOK-publish-0.79/0.81/0.85-*` await Peti → R-1 |
|
||||
| Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed | agent v0.70+ | **PROVEN-LIVE** (demo) | `SPIKE-agent-selfupdate-2026-07-05` | Remote-customer proof pending → R-1 |
|
||||
| Controller self-update: anonymous registry, no credentials in guest | controller v0.112 | **PROVEN-LIVE** (demo) | 07-10 arc | |
|
||||
|
||||
@@ -54,6 +54,7 @@
|
||||
| R-24 | **Guest resources as hub desired-state (live resize).** F5 (host-install v1.17.0) auto-sizes RAM/cores at INSTALL only. Make guest cores/RAM a per-host `pbs_dr`-sibling descriptor field the agent reconciles (`pct set -memory/-cores`), so the operator can right-size a running box from the hub — and land it in seconds via the agent-plane poke (R-13). | M | idea (F5 follow-on) | Follows F5 (`VALIDATION-n100` — appliance auto-size shipped); the live-resize path reuses the desired-state + poke machinery (agent v0.89 / hub v0.59). Would flip a new map row "operator right-sizes a running guest from the hub" |
|
||||
| R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster |
|
||||
| R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | **first slice PROVEN-LIVE (poke channel)** | **FIRST SLICE PROVEN-LIVE — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-17):** the ep0-relayed contentless poke (hub→ep0 `felhom-poke` forced-cmd→UDP→box WG /32:51822, peer-confined, zero ep0/box infra change) reaches the **agent** and fires an immediate desired-state cycle. Full path live: real operator manifest save → `sync-poke delivered to 10.77.0.2`; box → `poke received → immediate desired-state cycle` (~31 ms ep0→box, save→tick ≈ ~0.45 s). This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per `SPIKE-immediate-sync-transport-2026-07-16` P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately |
|
||||
| R-28 | **Agent fast-tick-until-first-convergence (immediacy SECONDARY).** The hub v0.63.0 system-initiated pokes cannot reach a box on the ONE leg that matters most at onboarding: the WG-**registration** auto-provision (the box's tunnel does not exist until it fetches the WG block, so a poke is undeliverable by construction — the register bump is deliberately poke-free). Close it from the agent side: while ANY desired-state item is still unapplied (e.g. a `pbs_dr` descriptor freshly minted, guests not yet at desired run-state), the agent ticks on a fast **30 s** state-based cadence instead of the 15-min backbone, and **self-disarms** the instant it converges. State-based, not a fixed burst — no timer to leak, no fleet-wide load once converged. | M | idea | Rides the **agent v0.90.0** train (agent-repo task; hub already mints the state). Names the coupling: flips the capability-map immediacy row's remaining "WG-registration leg unfired live" caveat → PROVEN once a real onboarding converges in seconds without a poke. Candidate train-passenger: the **Guests-0/0** onboarding observation (a box reporting 0/0 guests briefly at first boot) — the same fast-tick shortens that window. Pairs with R-13 (poke channel) + R-23 (UI leg) as the third immediacy leg |
|
||||
| R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | |
|
||||
| R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11) |
|
||||
| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED (slices A+B+C, 2026-07-17)** — physical N100 boot + the live boot→bind→day-0 composition fold into the supervised rehearsal (R-1) | **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* |
|
||||
|
||||
@@ -1,5 +1,41 @@
|
||||
# Felhom Hub — Changelog
|
||||
|
||||
## v0.63.0 — system-initiated immediacy: wire the proven poke/bump notifiers into every mutation site that lacked one (2026-07-17)
|
||||
|
||||
The immediate-sync arc (Dir-1 trigger, Dir-2b wait channel, Dir-2a agent poke) covered only
|
||||
**operator-initiated** desired-state changes. **System-initiated** mutations still bumped the
|
||||
generation silently, so a freshly onboarded box waited a full agent tick (≤15 min) for state the hub
|
||||
had already minted — observed live during slice-C onboarding. This wires the existing, live-proven
|
||||
notifiers (`poke.Notifier` for the agent plane, `intent.Hub.Bump` for the controller plane) into
|
||||
every system-initiated site that lacked one. No new mechanism — call-site wiring only.
|
||||
|
||||
- **Agent-plane pokes (`internal/web/pbsdr.go`):** `PBSDRAutoProvision` (the exact observed lag — the
|
||||
WG-registration hands-free provision) now pokes on success; `ReissuePBSDR` (the shared core, which
|
||||
also gives the **pbsdrheal reconciler's escalation** its immediacy with zero reconciler changes) and
|
||||
the operator button `handlePBSDRReissue` poke after their descriptor bump. All fire ONLY after the
|
||||
successful `SetHostDesired`, never on a blocked/error path.
|
||||
- **Agent-plane Poker seam (`internal/api/handler.go`, `internal/api/wg.go`):** a new nil-safe `Poker`
|
||||
interface (`PokeHost`/`PokeAllHosts`, satisfied by `*poke.Notifier`) + `SetPoker`. `handleAdminSetDesiredState`
|
||||
pokes the target host after a successful admin desired-state write; `handleAdminSetOperatorPeer` fires
|
||||
a fleet `PokeAllHosts` — but only when `BumpAllHostGenerations` succeeded (fire-after-commit).
|
||||
- **Controller-plane bump (`internal/api/handler.go`):** `reissueOnReenroll` (the clean-slate F2 claim +
|
||||
F3 offsite re-issue) now `intentHub.Bump`s the customer so a long-polling controller wakes in seconds
|
||||
instead of on the 15-min cycle. Nil-guarded; one unconditional bump (coalesced, over-bump harmless).
|
||||
- **main.go:** one `poke.Notifier` instance now feeds BOTH planes' system sites — `webServer.SetPoke(n)`
|
||||
**and** `apiHandler.SetPoker(n)`; startup log: "web + api admin seams armed".
|
||||
- **Deliberate non-sites (unchanged, documented in REPORT audit table):** WG peer *register* (box tunnel
|
||||
doesn't exist pre-fetch — a poke is undeliverable; the agent fast-tick SECONDARY owns this leg), WG peer
|
||||
*delete* (the mutation removes the transport), and the **pbsdrheal Restage** path (no generation bump →
|
||||
the agent's 60 s pbsdr ticker is the pickup path; a poke there is a verified no-op). `internal/pbsdrheal/`
|
||||
is byte-unchanged.
|
||||
- **Tests + red-proofs (run-fail-revert):** 10 new non-hollow tests. Web (async, channel-synchronized
|
||||
fake sender): auto-provision pokes the resolved WG /32, blocked-precondition pokes nothing, reissue-core
|
||||
+ operator-button poke, a failed reissue pokes nothing. API (synchronous fake Poker): admin-set pokes
|
||||
the target host only (0 on invalid-JSON), operator-peer fires one fleet poke, nil-seam no-panic; re-enroll
|
||||
advances the intent generation, nil-hub no-panic. Red-proofs demonstrated one representative removal per
|
||||
group (A/B web pokes, C admin poke, D re-enroll bump) — each FAILED red, then restored. Green:
|
||||
`go build/vet/test` all pass.
|
||||
|
||||
## v0.62.0 — R-21 slice C: the universal ISO — unclaimed-appliance registration + operator bind + one-shot delivery (2026-07-17)
|
||||
|
||||
The hub half of the universal, **secret-free** bare-metal ISO. A box booted from the generic ISO
|
||||
|
||||
+6
-2
@@ -444,8 +444,12 @@ func main() {
|
||||
}, logger); err != nil {
|
||||
logger.Printf("[ERROR] agent-plane poke disabled: %v", err)
|
||||
} else {
|
||||
webServer.SetPoke(poke.NewNotifier(dataStore, pokeClient, logger))
|
||||
logger.Printf("[INFO] agent-plane poke enabled (endpoint %s, user %s)", wgAddr, wgUser)
|
||||
// One notifier, both planes' system-initiated sites (v0.63.0): the web server's
|
||||
// pbsdr writes AND the api handler's admin desired-state / operator-peer writes.
|
||||
n := poke.NewNotifier(dataStore, pokeClient, logger)
|
||||
webServer.SetPoke(n)
|
||||
apiHandler.SetPoker(n)
|
||||
logger.Printf("[INFO] agent-plane poke enabled (endpoint %s, user %s; web + api admin seams armed)", wgAddr, wgUser)
|
||||
}
|
||||
} else {
|
||||
logger.Printf("[INFO] agent-plane poke disabled (key or endpoint not configured)")
|
||||
|
||||
@@ -35,6 +35,16 @@ type LatestVersionProvider interface {
|
||||
LatestVersion() string
|
||||
}
|
||||
|
||||
// Poker is the agent-plane immediate-sync seam (v0.63.0): satisfied by *poke.Notifier. A system-
|
||||
// initiated desired-state write here (admin-set, operator-peer bump) fires a contentless, fire-and-
|
||||
// forget nudge so the box ticks in seconds. nil = poke disabled — mutations still persist; the box
|
||||
// picks them up on its next ≤15-min cycle. Both methods are safe to call on a nil *poke.Notifier,
|
||||
// but every call site still guards with `if h.poker != nil` (the field itself may be nil).
|
||||
type Poker interface {
|
||||
PokeHost(hostID string)
|
||||
PokeAllHosts()
|
||||
}
|
||||
|
||||
// Handler handles API endpoints for report ingest and customer queries.
|
||||
type Handler struct {
|
||||
store *store.Store
|
||||
@@ -83,6 +93,12 @@ type Handler struct {
|
||||
// notifier that GET /api/v1/wait long-polls against. nil = wait endpoint returns 503 (the box
|
||||
// falls back to the 15-min cycle). Shared with the web server, whose intent handlers Bump it.
|
||||
intentHub *intent.Hub
|
||||
|
||||
// poker (v0.63.0, Direction-2a agent-plane immediate-sync) fires a fire-and-forget nudge after a
|
||||
// system-initiated HOST desired-state write (admin-set desired-state, operator-peer bump) so the
|
||||
// box ticks in seconds instead of ≤15 min. Shared with the web server (same *poke.Notifier). nil
|
||||
// = poke disabled (a no-op; the report cycle still reconciles).
|
||||
poker Poker
|
||||
}
|
||||
|
||||
// SetClaimEngine wires the customer-claim code engine (nil-safe everywhere it is used).
|
||||
@@ -112,6 +128,12 @@ func (h *Handler) SetIntentHub(hub *intent.Hub) {
|
||||
h.intentHub = hub
|
||||
}
|
||||
|
||||
// SetPoker wires the agent-plane immediate-sync notifier (v0.63.0; nil-safe — an unset poker makes
|
||||
// the admin desired-state writes fire no nudge, and the box picks the change up on its next cycle).
|
||||
func (h *Handler) SetPoker(p Poker) {
|
||||
h.poker = p
|
||||
}
|
||||
|
||||
// New creates a new API handler.
|
||||
func New(store *store.Store, apiKey, resendAPIKey, fromEmail string, templateProvider ConfigTemplateProvider, logger *log.Logger) *Handler {
|
||||
return &Handler{
|
||||
@@ -1051,6 +1073,15 @@ func (h *Handler) reissueOnReenroll(cc *store.CustomerConfig) {
|
||||
h.logger.Printf("[WARN] offsite re-issue on re-enroll for %s failed: %v", cc.CustomerID, err)
|
||||
}
|
||||
}
|
||||
// Direction-2 (v0.63.0): wake a long-polling controller so the re-staged claim code / offsite
|
||||
// password ride the next ACK in seconds, not on the 15-min cycle. Both legs above are
|
||||
// best-effort; an over-bump costs one cheap wake. (On the clean-slate path the controller
|
||||
// usually does not exist yet — its startup fetch covers that shape; a bump landing during a
|
||||
// fresh controller's FIRST hold is recorded as baseline without firing — the known open
|
||||
// observation, fixed later by carrying intent_gen in the report ACK. Out of scope here.)
|
||||
if h.intentHub != nil {
|
||||
h.intentHub.Bump(cc.CustomerID)
|
||||
}
|
||||
}
|
||||
|
||||
// escrowUploadRequest is the agent→hub wire shape for the OPAQUE PBS recovery-code escrow blob
|
||||
@@ -1396,6 +1427,9 @@ func (h *Handler) handleAdminSetDesiredState(w http.ResponseWriter, r *http.Requ
|
||||
return
|
||||
}
|
||||
h.logger.Printf("[INFO] admin-set desired-state for host %s (generation now %d, %d bytes)", pathHostID, gen, len(body))
|
||||
if h.poker != nil {
|
||||
h.poker.PokeHost(pathHostID) // agent-plane immediate-sync (Direction-2a): generation bumped → nudge the box now
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
json.NewEncoder(w).Encode(map[string]interface{}{"status": "ok", "generation": gen})
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
package api
|
||||
|
||||
// v0.63.0 system-initiated immediacy — the api-side agent-plane pokes (admin desired-state write,
|
||||
// operator-peer bump) and the controller-plane intent bump on clean-slate re-enroll. The api sites
|
||||
// call the Poker SYNCHRONOUSLY, so a counting fake is deterministic right after the request. The
|
||||
// negatives (error/invalid paths, nil seams) carry explicit zero-count / no-panic assertions.
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/intent"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// fakePoker counts calls per method and records PokeHost targets. Satisfies api.Poker.
|
||||
type fakePoker struct {
|
||||
mu sync.Mutex
|
||||
hostCalls []string
|
||||
allCalls int
|
||||
}
|
||||
|
||||
func (f *fakePoker) PokeHost(hostID string) {
|
||||
f.mu.Lock()
|
||||
f.hostCalls = append(f.hostCalls, hostID)
|
||||
f.mu.Unlock()
|
||||
}
|
||||
func (f *fakePoker) PokeAllHosts() {
|
||||
f.mu.Lock()
|
||||
f.allCalls++
|
||||
f.mu.Unlock()
|
||||
}
|
||||
func (f *fakePoker) hosts() []string {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]string(nil), f.hostCalls...)
|
||||
}
|
||||
func (f *fakePoker) all() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.allCalls
|
||||
}
|
||||
|
||||
// C1 — a successful admin desired-state write pokes exactly the target host (and never the fleet);
|
||||
// the invalid-JSON path pokes nothing (fire-after-commit).
|
||||
func TestAdminSetDesiredState_PokesTargetHostOnly(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
seedHost(t, st, "h1", "c1", "HKEY1")
|
||||
p := &fakePoker{}
|
||||
h.SetPoker(p)
|
||||
|
||||
if rr := do(h, http.MethodPut, "/admin/hosts/h1/desired-state", globalKey, `{"guests":[]}`); rr.Code != http.StatusOK {
|
||||
t.Fatalf("admin-set = %d body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if got := p.hosts(); len(got) != 1 || got[0] != "h1" {
|
||||
t.Fatalf("PokeHost targets = %v, want [h1]", got)
|
||||
}
|
||||
if p.all() != 0 {
|
||||
t.Errorf("PokeAllHosts = %d, want 0 (a single-host write is not a fleet nudge)", p.all())
|
||||
}
|
||||
|
||||
// Invalid JSON is rejected at the door → NO poke.
|
||||
before := len(p.hosts())
|
||||
if rr := do(h, http.MethodPut, "/admin/hosts/h1/desired-state", globalKey, `not json`); rr.Code != http.StatusBadRequest {
|
||||
t.Fatalf("malformed admin-set = %d, want 400", rr.Code)
|
||||
}
|
||||
if len(p.hosts()) != before {
|
||||
t.Errorf("a poke fired on the invalid-JSON path (targets now %v)", p.hosts())
|
||||
}
|
||||
}
|
||||
|
||||
// C1 (nil seam) — an unset poker leaves the mutation intact and never panics.
|
||||
func TestAdminSetDesiredState_NilPokerNoPanic(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
seedHost(t, st, "h1", "c1", "HKEY1")
|
||||
// No SetPoker — h.poker is nil.
|
||||
if rr := do(h, http.MethodPut, "/admin/hosts/h1/desired-state", globalKey, `{"guests":[]}`); rr.Code != http.StatusOK {
|
||||
t.Fatalf("admin-set with nil poker = %d, want 200 (mutation must still succeed)", rr.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// C2 — a successful operator-peer write bumps EVERY host generation → exactly one fleet poke, and
|
||||
// never a single-host poke. (The BumpAllHostGenerations-error negative is assert-by-inspection: the
|
||||
// poke sits in the `else if h.poker != nil` arm of the bump result, so a bump error skips it — there
|
||||
// is no non-destructive seam to force a store bump failure in this fixture.)
|
||||
func TestAdminSetOperatorPeer_PokesFleet(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
seedHost(t, st, "h1", "c1", "HKEY1")
|
||||
putTestEndpoint(t, h)
|
||||
p := &fakePoker{}
|
||||
h.SetPoker(p)
|
||||
|
||||
if rr := do(h, http.MethodPut, "/admin/wg/operator-peer", globalKey,
|
||||
`{"pubkey":"`+opTestPubkey+`","assigned_ip":"10.77.0.250"}`); rr.Code != http.StatusOK {
|
||||
t.Fatalf("set operator peer = %d body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if p.all() != 1 {
|
||||
t.Errorf("PokeAllHosts = %d, want 1", p.all())
|
||||
}
|
||||
if got := p.hosts(); len(got) != 0 {
|
||||
t.Errorf("PokeHost fired on a fleet-wide change: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario D — clean-slate re-enroll bumps the controller-plane intent so a long-polling controller
|
||||
// wakes in seconds. reissueOnReenroll is synchronous, so the generation advances by return time.
|
||||
// Red-proof: remove the Part-3 bump → Generation stays 0.
|
||||
func TestReenroll_BumpsIntent(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ckey", RetrievalPassword: "pass-phrase"})
|
||||
hub := intent.New()
|
||||
h.SetIntentHub(hub)
|
||||
|
||||
gen0 := hub.Generation("c1")
|
||||
if rr := doEnroll(h, "c1", "pass-phrase"); rr.Code != http.StatusCreated {
|
||||
t.Fatalf("enroll = %d body=%s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if gen := hub.Generation("c1"); gen <= gen0 {
|
||||
t.Fatalf("intent generation = %d, want > %d (re-enroll must bump)", gen, gen0)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario D (nil seam) — with no intent hub wired, enroll still succeeds and never panics.
|
||||
func TestReenroll_NilIntentHubNoPanic(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ckey", RetrievalPassword: "pass-phrase"})
|
||||
// No SetIntentHub — h.intentHub is nil.
|
||||
if rr := doEnroll(h, "c1", "pass-phrase"); rr.Code != http.StatusCreated {
|
||||
t.Fatalf("enroll with nil intent hub = %d, want 201", rr.Code)
|
||||
}
|
||||
}
|
||||
@@ -504,6 +504,8 @@ func (h *Handler) handleAdminSetOperatorPeer(w http.ResponseWriter, r *http.Requ
|
||||
bumped, berr := h.store.BumpAllHostGenerations()
|
||||
if berr != nil {
|
||||
h.logger.Printf("[WARN] operator peer set but generation bump failed: %v", berr)
|
||||
} else if h.poker != nil {
|
||||
h.poker.PokeAllHosts() // agent-plane immediate-sync (Direction-2a): every host generation moved → fleet nudge (fire-after-commit)
|
||||
}
|
||||
h.logger.Printf("[INFO] operator OOB peer set: %s -> %s/32 (sync=%s, %d host generations bumped)",
|
||||
req.Pubkey, req.AssignedIP, syncStatus, bumped)
|
||||
|
||||
@@ -310,6 +310,7 @@ func (s *Server) PBSDRAutoProvision(ctx context.Context, customerID string) {
|
||||
return
|
||||
}
|
||||
s.logger.Printf("[INFO] pbsdr auto-provisioned for %s on WG registration (hands-free cascade)", customerID)
|
||||
s.poke.PokeHost(host.HostID) // freshly auto-provisioned + generation bumped → nudge the box now (the observed slice-C lag)
|
||||
}
|
||||
|
||||
// ReissuePBSDR re-keys the customer's ep0 PBS token and re-arms the agent — the non-HTTP core shared
|
||||
@@ -357,6 +358,7 @@ func (s *Server) ReissuePBSDR(ctx context.Context, customerID string) error {
|
||||
return fmt.Errorf("pbsdr reissue for %s: descriptor bump: %w", customerID, err)
|
||||
}
|
||||
s.logger.Printf("[INFO] pbsdr credentials re-issued for %s (host %s; fresh consume-once secret stored, withheld from logs)", customerID, host.HostID)
|
||||
s.poke.PokeHost(host.HostID) // agent-plane immediate-sync (Direction-2a): re-consume signal lands in seconds (also the pbsdrheal escalation path)
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -412,6 +414,7 @@ func (s *Server) handlePBSDRReissue(w http.ResponseWriter, r *http.Request, cust
|
||||
return
|
||||
}
|
||||
s.logger.Printf("[INFO] pbsdr credentials re-issued for %s (host %s; fresh consume-once secret stored)", customerID, host.HostID)
|
||||
s.poke.PokeHost(host.HostID) // agent-plane immediate-sync (Direction-2a): re-consume signal lands in seconds
|
||||
http.Redirect(w, r, "/customers/"+customerID+"?flash=pbsdr_reissued#tab=edit", http.StatusSeeOther)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,197 @@
|
||||
package web
|
||||
|
||||
// v0.63.0 system-initiated immediacy — the agent-plane pokes wired into the pbsdr mutation sites.
|
||||
// These assert the EFFECT (the resolved WG /32 received a nudge), synchronize on a channel with a
|
||||
// bounded timeout (never sleep-poll), and carry explicit zero-count negatives on the error/blocked
|
||||
// paths. The notifier is a REAL poke.Notifier built over the test store + a channel-carrying fake
|
||||
// sender (structural satisfaction of poke's unexported sender seam — the value is passed, the type
|
||||
// is never named), exactly as main.go builds it.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/poke"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/tenantsync"
|
||||
)
|
||||
|
||||
func quietWebLogger() *log.Logger { return log.New(io.Discard, "", 0) }
|
||||
|
||||
// chanSender is a fake poke sender: it records every target and, for async assertions, publishes
|
||||
// each one on a buffered channel so a test can select-with-timeout instead of sleeping.
|
||||
type chanSender struct {
|
||||
mu sync.Mutex
|
||||
targets []string
|
||||
ch chan string
|
||||
err error
|
||||
}
|
||||
|
||||
func newChanSender() *chanSender { return &chanSender{ch: make(chan string, 8)} }
|
||||
|
||||
func (f *chanSender) Poke(_ context.Context, boxWGIP string) error {
|
||||
f.mu.Lock()
|
||||
f.targets = append(f.targets, boxWGIP)
|
||||
f.mu.Unlock()
|
||||
select {
|
||||
case f.ch <- boxWGIP:
|
||||
default:
|
||||
}
|
||||
return f.err
|
||||
}
|
||||
|
||||
func (f *chanSender) count() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return len(f.targets)
|
||||
}
|
||||
|
||||
// wirePoke builds a real notifier over the test store + fake sender and installs it on the server.
|
||||
func wirePoke(s *Server, st *store.Store, snd *chanSender) {
|
||||
s.SetPoke(poke.NewNotifier(st, snd, quietWebLogger()))
|
||||
}
|
||||
|
||||
// expectPoke asserts exactly one poke to wantIP arrives within the bound, and none trails it.
|
||||
func expectPoke(t *testing.T, snd *chanSender, wantIP string) {
|
||||
t.Helper()
|
||||
select {
|
||||
case got := <-snd.ch:
|
||||
if got != wantIP {
|
||||
t.Fatalf("poke target = %q, want %q", got, wantIP)
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatalf("no poke within 2s (want one to %s)", wantIP)
|
||||
}
|
||||
// No second poke should follow (a short settle window; the send is buffered so a stray fires fast).
|
||||
select {
|
||||
case extra := <-snd.ch:
|
||||
t.Fatalf("a second, unexpected poke fired to %q", extra)
|
||||
case <-time.After(150 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
|
||||
// expectNoPoke asserts no poke arrives within a bounded settle window (the negative-path contract).
|
||||
func expectNoPoke(t *testing.T, snd *chanSender) {
|
||||
t.Helper()
|
||||
select {
|
||||
case got := <-snd.ch:
|
||||
t.Fatalf("a poke fired to %q on a path that must not poke", got)
|
||||
case <-time.After(300 * time.Millisecond):
|
||||
}
|
||||
if n := snd.count(); n != 0 {
|
||||
t.Fatalf("sender received %d pokes, want 0", n)
|
||||
}
|
||||
}
|
||||
|
||||
func peerIP(t *testing.T, st *store.Store, hostID string) string {
|
||||
t.Helper()
|
||||
p, err := st.GetWGPeerForHost(hostID)
|
||||
if err != nil || p == nil {
|
||||
t.Fatalf("peer for %s: %v", hostID, err)
|
||||
}
|
||||
return p.AssignedIP
|
||||
}
|
||||
|
||||
// Scenario A — the observed slice-C lag: PBSDRAutoProvision pokes the box after a hands-free
|
||||
// provision. Red-proof: remove the 1.1 insertion → expectPoke times out.
|
||||
func TestPBSDR_AutoProvisionPokes(t *testing.T) {
|
||||
fake := &fakeTenancy{secret: "AUTO-SECRET"}
|
||||
s, st, _ := newPBSDRServer(t, fake)
|
||||
if err := st.RemoveWGPeer("PETIPUBKEY"); err != nil {
|
||||
t.Fatalf("remove seed peer: %v", err)
|
||||
}
|
||||
// Flag ON while the peer is missing → stored intent, no provision (and so no poke path).
|
||||
postUpdate(t, s, url.Values{"dr_tier": {"on"}})
|
||||
|
||||
// The agent registers its WG key; the hook fires the atom → provision succeeds → poke.
|
||||
if _, _, err := st.RegisterWGPeerForHost("peti-01", "PETIPUBKEY"); err != nil {
|
||||
t.Fatalf("register peer: %v", err)
|
||||
}
|
||||
snd := newChanSender()
|
||||
wirePoke(s, st, snd)
|
||||
wantIP := peerIP(t, st, "peti-01")
|
||||
|
||||
s.PBSDRAutoProvision(context.Background(), "peti")
|
||||
expectPoke(t, snd, wantIP)
|
||||
}
|
||||
|
||||
// Scenario A (negative) — a blocked precondition (no WG peer) mints nothing, so NO poke fires.
|
||||
func TestPBSDR_AutoProvisionBlockedDoesNotPoke(t *testing.T) {
|
||||
fake := &fakeTenancy{secret: "S"}
|
||||
s, st, _ := newPBSDRServer(t, fake)
|
||||
if err := st.RemoveWGPeer("PETIPUBKEY"); err != nil {
|
||||
t.Fatalf("remove seed peer: %v", err)
|
||||
}
|
||||
postUpdate(t, s, url.Values{"dr_tier": {"on"}})
|
||||
|
||||
snd := newChanSender()
|
||||
wirePoke(s, st, snd)
|
||||
s.PBSDRAutoProvision(context.Background(), "peti") // peer still absent → blocked, no mint
|
||||
expectNoPoke(t, snd)
|
||||
}
|
||||
|
||||
// Scenario B1 — ReissuePBSDR (the reconciler-escalation core) pokes after the descriptor bump.
|
||||
// Red-proof: remove the 1.2 insertion → expectPoke times out.
|
||||
func TestPBSDR_ReissueCorePokes(t *testing.T) {
|
||||
fake := &fakeTenancy{secret: "OLD"}
|
||||
s, st, _ := newPBSDRServer(t, fake) // poke NOT wired yet → the provision below fires none
|
||||
postUpdate(t, s, url.Values{"dr_tier": {"on"}})
|
||||
st.ConsumeHostPBSSecret("peti-01")
|
||||
|
||||
snd := newChanSender()
|
||||
wirePoke(s, st, snd)
|
||||
wantIP := peerIP(t, st, "peti-01")
|
||||
|
||||
fake.secret = "FRESH"
|
||||
if err := s.ReissuePBSDR(context.Background(), "peti"); err != nil {
|
||||
t.Fatalf("ReissuePBSDR: %v", err)
|
||||
}
|
||||
expectPoke(t, snd, wantIP)
|
||||
}
|
||||
|
||||
// Scenario B (negative) — a reissue that FAILS (tenantsync error) never reaches SetHostDesired,
|
||||
// so NO poke fires (fire-after-commit).
|
||||
func TestPBSDR_ReissueCoreErrorDoesNotPoke(t *testing.T) {
|
||||
fake := &fakeTenancy{secret: "OLD"}
|
||||
s, st, _ := newPBSDRServer(t, fake)
|
||||
postUpdate(t, s, url.Values{"dr_tier": {"on"}})
|
||||
|
||||
snd := newChanSender()
|
||||
wirePoke(s, st, snd)
|
||||
|
||||
fake.err = errors.New("ssh boom")
|
||||
if err := s.ReissuePBSDR(context.Background(), "peti"); err == nil {
|
||||
t.Fatal("ReissuePBSDR returned nil on a tenantsync error")
|
||||
}
|
||||
expectNoPoke(t, snd)
|
||||
}
|
||||
|
||||
// Scenario B2 — the operator button (handlePBSDRReissue) pokes after the descriptor bump.
|
||||
func TestPBSDR_HandleReissuePokes(t *testing.T) {
|
||||
fake := &fakeTenancy{secret: "OLD"}
|
||||
s, st, _ := newPBSDRServer(t, fake)
|
||||
postUpdate(t, s, url.Values{"dr_tier": {"on"}})
|
||||
st.ConsumeHostPBSSecret("peti-01")
|
||||
|
||||
snd := newChanSender()
|
||||
wirePoke(s, st, snd)
|
||||
wantIP := peerIP(t, st, "peti-01")
|
||||
|
||||
fake.secret = "FRESH"
|
||||
req := httptest.NewRequest("POST", "/configs/peti/pbsdr-reissue", nil)
|
||||
rr := httptest.NewRecorder()
|
||||
s.handlePBSDRReissue(rr, req, "peti")
|
||||
if rr.Code != 303 {
|
||||
t.Fatalf("reissue = %d (%s), want 303", rr.Code, rr.Body.String())
|
||||
}
|
||||
expectPoke(t, snd, wantIP)
|
||||
}
|
||||
|
||||
var _ = tenantsync.ErrTokenExists // keep the tenantsync import stable across edits
|
||||
Reference in New Issue
Block a user