docs: hub v0.63.0 system-initiated immediacy — REPORT + CONTEXT + capability-map (immediacy row) + ROADMAP (R-28 agent fast-tick SECONDARY)
This commit is contained in:
@@ -2,86 +2,139 @@
|
||||
|
||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||||
|
||||
## R-21 slice C — the universal secret-free ISO (hub v0.62.0 + scripts v1.19.0) — 2026-07-17
|
||||
## Hub v0.63.0 — system-initiated immediacy (poke + intent bump at every mutation site) — 2026-07-17
|
||||
|
||||
**Baselines:** hub v0.61.0 → **v0.62.0**; scripts v1.18.0 (`3172df1`) → **v1.19.0**. Commits `36c5cd5`
|
||||
(code), `9964ae6` (manifest). Hub **v0.62.0 LIVE** on k3s (Synced/Healthy). R-21 now shipped A+B+C.
|
||||
### 1. Baseline used
|
||||
|
||||
### What shipped
|
||||
- **felhom.eu** @ `4c9b0e87063278cddbe51e3966a8f8592c7c18f2` ("docs: R-21 slice C …", 2026-07-17),
|
||||
clean tree on `main`, local == `origin/main` re-confirmed before starting.
|
||||
- **Hub** v0.62.0 → **v0.63.0**. Deploy manifest `manifests/hub.yaml` image line `felhom-hub:0.62.0` → `:0.63.0`.
|
||||
- Both immediate-sync transports were already PROVEN-LIVE (agent poke: agent v0.89.0 + hub v0.59.0,
|
||||
2026-07-17; intent wait channel: hub v0.58.0 / controller v0.140.0). This task is **call-site wiring
|
||||
only** — no new mechanism, no spike.
|
||||
|
||||
The last R-21 slice: a **generic, secret-free** ISO. Blank box → unattended install → the box
|
||||
registers itself at the hub as an *unclaimed appliance* → the operator binds it to a customer → the hub
|
||||
delivers `customer-id` + retrieval passphrase **once** → day-0 completes via the existing slice-A path.
|
||||
The distributed ISO carries no secret (§4.4). Slice-A direct mode (env-baked) survives unchanged.
|
||||
### 2. Files created / modified
|
||||
|
||||
### Registration payload (as implemented) + token model
|
||||
**Implementation (commit `30972d8`):**
|
||||
- `hub/internal/web/pbsdr.go` — +3 pokes: `PBSDRAutoProvision`, `ReissuePBSDR`, `handlePBSDRReissue` (each after the successful `SetHostDesired`).
|
||||
- `hub/internal/api/handler.go` — new nil-safe `Poker` interface + `poker` field + `SetPoker`; poke in `handleAdminSetDesiredState`; intent `Bump` in `reissueOnReenroll`.
|
||||
- `hub/internal/api/wg.go` — `PokeAllHosts` in `handleAdminSetOperatorPeer` (only on a successful `BumpAllHostGenerations`).
|
||||
- `hub/cmd/hub/main.go` — capture the one `poke.Notifier` and wire it to BOTH planes (`webServer.SetPoke(n)` + `apiHandler.SetPoker(n)`); startup log "web + api admin seams armed".
|
||||
- `hub/internal/web/pbsdr_poke_test.go` (new) — 5 web tests (Groups A, B).
|
||||
- `hub/internal/api/poke_seam_test.go` (new) — 5 api tests (Groups C, D).
|
||||
- `hub/CHANGELOG.md` — v0.63.0 entry (newest on top).
|
||||
- `REUSE.md` — new canonical-pattern row: "Immediate-sync notify (per plane)".
|
||||
|
||||
`POST /api/v1/appliance/register` body: `{uuid, macs[], ssh_host_pubkeys[], hw{product, cpu, mem_kb}}`.
|
||||
Keyed by **(uuid, mac_set)** — the N100 DMI verdict (serials = "Default string") + cheap-board duplicate
|
||||
SMBIOS UUIDs make the **MAC set the tiebreaker** (same uuid + different mac-set = distinct appliance).
|
||||
**Token model:** a random **256-bit** appliance token per registration, returned once, stored only as
|
||||
**sha256** (`token_hash`); it is the box's sole pre-day-0 credential, never logged. Invalidated on
|
||||
discard (`token_hash` blanked + status sticky) and after consume. `GET /api/v1/appliance/poll` (Bearer
|
||||
token): unknown/discarded → **404** (no oracle) · unbound → **204** · bound → **200** + creds (consumed
|
||||
once) · delivered → **410**. The passphrase is read live from `customer_configs` (plaintext, as the
|
||||
day-0 command already requires) and never logged. register is the ONE unauthenticated endpoint, per-IP
|
||||
rate-limited (the ingress already geo-restricts to HU).
|
||||
**Docs (commit `<DOCS_HASH>`):**
|
||||
- `REPORT.md` (this file, overwritten), `CONTEXT.md` (decision recorded), `documentation/architecture/00-capability-map.md` (immediacy row note + version), `documentation/backlog/ROADMAP.md` (new R-28: agent fast-tick SECONDARY).
|
||||
|
||||
### Red-proofs (run-fail-revert, all recorded)
|
||||
**Manifest (commit `<MANIFEST_HASH>`):** `manifests/hub.yaml` image tag → `felhom-hub:0.63.0`.
|
||||
|
||||
- **One-shot delivery (C):** defeat `MarkApplianceDelivered`'s bound→delivered flip → the second poll
|
||||
re-delivers the passphrase (200) instead of 410 → `TestAppliancePoll_OneShotAnd410` FAILS → restored.
|
||||
- **Register idempotency (A):** drop the `(uuid, mac_set)` upsert branch → the re-register hits the
|
||||
UNIQUE backstop (500) → `TestApplianceRegister_Idempotent` FAILS → restored. (The UNIQUE index is the
|
||||
load-bearing backstop; the upsert is the graceful update.)
|
||||
- **Direct-mode regression (D):** `test/bootstrap-modes.sh` — a direct env drives `run_direct` and makes
|
||||
**ZERO** `/appliance/*` calls (fake hub records none); the pairing code path is provably not entered.
|
||||
Plus 404-no-oracle + sticky-discard, bind staging/refusal + no-host-count-gate, and the render test.
|
||||
### 3. Per-commit hashes pushed to `main`
|
||||
|
||||
### Deploy verification
|
||||
| # | Hash | Contents |
|
||||
|---|------|----------|
|
||||
| 1 | `30972d8` | code + tests + CHANGELOG + REUSE |
|
||||
| 2 | `<DOCS_HASH>` | REPORT + CONTEXT + capability-map + ROADMAP |
|
||||
| 3 | `<MANIFEST_HASH>` | manifests/hub.yaml image bump → 0.63.0 |
|
||||
|
||||
Hub 0.62.0 rolled out (Synced/Healthy, image confirmed). **Live smoke through the public ingress:**
|
||||
`POST /appliance/register` → 200 with a 256-bit token; poll (valid, unbound) → **204**; poll (bogus
|
||||
token) → **404** (no oracle). (The `smoke-test-uuid` record is CC test litter — discard on the hub.)
|
||||
### 4. Test results + red-proof outcomes
|
||||
|
||||
### Scenario E (the composition proof) + the secret-free artifact assertion
|
||||
**New tests (10), all green:**
|
||||
|
||||
- **Generic ISO built** `felhom-pve-9.2-1-v1.19.0-generic-generic.iso` (sha
|
||||
`f37080c835300b6cc2831bd6e5a483cad307f4d009469ee4206442fe9bebd348`). **Secret-free asserted on the
|
||||
artifact:** manifest `mode: pairing` / `secret-bearing: no`; the env decoded from the ISO's
|
||||
`/proxmox-first-boot` stub carries **only** `FELHOM_HUB_URL` + `FELHOM_INSTALL_URL` — no
|
||||
`FELHOM_CUSTOMER_ID`, no `FELHOM_RETRIEVAL_PASSPHRASE`.
|
||||
- **Nested boot → register (CONFIRMED live):** the generic ISO booted on scratch VM 311 (SeaBIOS, LAN,
|
||||
uuid `7ed8cb9e-…`, MAC `bc:24:11:c3:b3:d0`) → **zero-touch install** (answer file consumed, disk `sda`
|
||||
auto-selected, country HU) → reboot → installed system → first-boot pairing → **registered at the
|
||||
LIVE hub**: `[INFO] appliance registered: new unclaimed box (uuid=7ed8cb9e-…, macs=1, ssh_keys=3)` —
|
||||
the box gathered its MAC + all 3 SSH host keys exactly as designed. It now polls 204 (unbound),
|
||||
**staged for the operator bind**. The record + VM are left in place as the handoff. *(Boot-order
|
||||
lesson: the scratch VM must boot `scsi0;ide2` — disk-first, CD-fallback — or a CD-first order
|
||||
reinstalls in a loop and never reaches first-boot.)*
|
||||
- **The bind is the coordinated Viktor moment** (operator-password-gated; CC cannot log in). CC has
|
||||
staged everything: the box is registered and visible; a Viktor-created drill customer + the 2-minute
|
||||
bind click complete boot→register→**bind**→delivery→day-0, then the drill customer is torn down via
|
||||
RESET then Delete (re-exercising v0.61.0/v0.60.1). The bind→delivery→day-0 legs are proven by the
|
||||
hub unit tests + the `bootstrap-modes.sh` delivery handoff; the live composition folds into the
|
||||
supervised rehearsal, **which now runs the complete final product flow in one pass**.
|
||||
| Test | Group | Asserts |
|
||||
|------|-------|---------|
|
||||
| `TestPBSDR_AutoProvisionPokes` | A | auto-provision success → exactly one poke to the resolved WG /32 (10.77.0.2) within 2 s |
|
||||
| `TestPBSDR_AutoProvisionBlockedDoesNotPoke` | A (neg) | blocked precondition (no WG peer) → zero pokes |
|
||||
| `TestPBSDR_ReissueCorePokes` | B1 | `ReissuePBSDR` success → one poke to the host's /32 |
|
||||
| `TestPBSDR_ReissueCoreErrorDoesNotPoke` | B (neg) | tenantsync-error reissue → zero pokes (fire-after-commit) |
|
||||
| `TestPBSDR_HandleReissuePokes` | B2 | operator button success → one poke |
|
||||
| `TestAdminSetDesiredState_PokesTargetHostOnly` | C1 | admin-set success → PokeHost==1 (target only), PokeAllHosts==0; invalid-JSON path → 0 |
|
||||
| `TestAdminSetDesiredState_NilPokerNoPanic` | C1 (nil) | nil poker → mutation still 200, no panic |
|
||||
| `TestAdminSetOperatorPeer_PokesFleet` | C2 | operator-peer success → PokeAllHosts==1, PokeHost==0 |
|
||||
| `TestReenroll_BumpsIntent` | D | re-enroll → intent generation advances (synchronous) |
|
||||
| `TestReenroll_NilIntentHubNoPanic` | D (nil) | nil intent hub → enroll still 201, no panic |
|
||||
|
||||
### SSH-host-key pinning disposition
|
||||
Async web assertions synchronize on a buffered channel in the fake sender with a bounded `select`
|
||||
(≤2 s), never sleep-poll. The web fake sender structurally satisfies poke's unexported `pokeSender`
|
||||
seam (value passed, type never named); the notifier is a REAL `poke.Notifier` over the test store —
|
||||
exactly as `main.go` builds it. The api fake `Poker` satisfies the exported `api.Poker` interface and
|
||||
counts calls synchronously.
|
||||
|
||||
The registration payload carries the box's SSH host public keys; they are **stored** with the appliance
|
||||
record and their **fingerprints displayed** in the unclaimed-appliances UI. They are **not** auto-attached
|
||||
to the host on bind — the `hosts` row is minted later by the box's own `/host-enroll` (mint-once-reuse),
|
||||
and no clean "attach host keys to host" hand-off surface exists today. Per the spec's fallback, this is
|
||||
the store+display+**Observation** path, not new plumbing (a future item can wire TOFU-free pinning).
|
||||
**Red-proofs (run-fail-restore), one representative per scenario group:**
|
||||
|
||||
### Observations
|
||||
| Group | Removed | Test | Result |
|
||||
|-------|---------|------|--------|
|
||||
| A | `PBSDRAutoProvision` poke (Part 1.1) | `TestPBSDR_AutoProvisionPokes` | FAILED red ("no poke within 2s… 10.77.0.2") → restored |
|
||||
| B | `ReissuePBSDR` poke (Part 1.2) | `TestPBSDR_ReissueCorePokes` | FAILED red ("no poke within 2s… 10.77.0.2") → restored |
|
||||
| C | `handleAdminSetDesiredState` poke (Part 2.2) | `TestAdminSetDesiredState_PokesTargetHostOnly` | FAILED red ("PokeHost targets = [], want [h1]") → restored |
|
||||
| D | `reissueOnReenroll` bump (Part 3) | `TestReenroll_BumpsIntent` | FAILED red ("intent generation = 0, want > 0") → restored |
|
||||
|
||||
- **Bind never gates on host count** (multi-host customers are real — Peti); the picker shows counts as
|
||||
display only. A post-RESET customer is naturally hostless and binds fine (tested).
|
||||
- **Pre-bind provenance** is the `appliance_registrations` row's own timestamps (register/discard have no
|
||||
customer to scope an `events` row to — `events.customer_id` is NOT NULL); `appliance_bound` /
|
||||
`appliance_credential_delivered` events are recorded once a customer exists.
|
||||
- **Universal disk selection is out of slice-C scope:** `profiles/generic.profile` defaults to the first
|
||||
SATA disk (`sda`); NVMe/other targets need a profile variant (a future concern, noted in the profile).
|
||||
- **Poll cadence = systemd:** the bootstrap does one poll per invocation and exits non-zero when unbound,
|
||||
reusing `Restart=on-failure`/`RestartSec=30` as the 30s poll timer (no long-running-oneshot timeout).
|
||||
These tests fail on the pre-fix tree by construction; a representative removal per group was
|
||||
demonstrated (B2 and C2 share their group's wiring shape with the demonstrated B1/C1 removals).
|
||||
|
||||
### 5. Test count before/after
|
||||
|
||||
- Package `internal/web`: +5 tests (new `pbsdr_poke_test.go`).
|
||||
- Package `internal/api`: +5 tests (new `poke_seam_test.go`).
|
||||
- Full suite after: `go build ./... && go vet ./... && go test ./...` — **all packages green** (api + web
|
||||
are the only two exercising the new code; every other package unchanged).
|
||||
|
||||
### 6. Deployed version + pod/log verification
|
||||
|
||||
<DEPLOY_VERIFICATION>
|
||||
|
||||
### 7. NOT yet live-validated (real system-initiated events only)
|
||||
|
||||
The new pokes/bumps fire only on real system events, none manufacturable non-destructively from this
|
||||
task. Acceptance bar here = **unit tests + deploy + startup-log verification**. Unfired live legs, to be
|
||||
observed at the rehearsal's bind sequence (`RUNBOOK-rehearsal-v3 S5`):
|
||||
|
||||
- **Auto-provision poke** — fires when a DR-ON customer's host registers its WG peer for the first time
|
||||
and the atom provisions (the exact observed slice-C lag).
|
||||
- **Reconciler-escalation poke** — fires when the pbsdrheal reconciler escalates to `ReissuePBSDR` (a box
|
||||
stuck in `waiting_secret`/`consumed_failed` after losing its converged marker).
|
||||
- **Clean-slate re-enroll bump** — fires when a wiped, re-minted box re-enrolls for a claimed/offsite
|
||||
customer (on that path the controller usually does not exist yet; its startup fetch covers that shape).
|
||||
- **Operator-button / admin-api pokes** (`handlePBSDRReissue`, admin desired-state / operator-peer) — the
|
||||
operator-triggered legs; exercisable in a supervised session but not fired here.
|
||||
|
||||
The **first-hold-baseline race** stays OPEN by design: a bump landing during a fresh controller's FIRST
|
||||
wait is recorded as baseline without firing (the ruled fix is carrying `intent_gen` in the report ACK —
|
||||
its own future task, out of scope).
|
||||
|
||||
### 8. Part-4 audit table — EVERY desired-state mutation site
|
||||
|
||||
Re-grepped at implementation time for all non-test callers of `SetHostDesired`, `BumpHostDesired`,
|
||||
`BumpAllHostGenerations`, plus the system-initiated customer-deliverable writers. No forgotten fourth
|
||||
caller — every site below is accounted for.
|
||||
|
||||
| Site (file:landmark) | Initiator | Plane | Notification after this task | Reason if none |
|
||||
|---|---|---|---|---|
|
||||
| `web/pbsdr.go` `applyPBSDR` disable (~L143→148) | operator (form) | agent | poke (pre-existing) | — |
|
||||
| `web/pbsdr.go` `applyPBSDR` re-enable/storage-id (~L168→173) | operator (form) | agent | poke (pre-existing) | — |
|
||||
| `web/pbsdr.go` `applyPBSDR` fresh-provision (~L179→186) | operator (form) | agent | poke (pre-existing) | — |
|
||||
| `web/pbsdr.go` `pbsdrProvisionAtom` `SetHostDesired` (~L273) | — (shared atom) | agent | poke at each caller | callers poke: `applyPBSDR` (L186) + `PBSDRAutoProvision` (NEW) |
|
||||
| `web/pbsdr.go` `PBSDRAutoProvision` (~L312) | **system** (WG-reg hook) | agent | **NEW poke** | — (the observed lag) |
|
||||
| `web/pbsdr.go` `ReissuePBSDR` `SetHostDesired` (~L357) | **system** (reconciler escalation) | agent | **NEW poke** | — (also covers pbsdrheal, zero reconciler change) |
|
||||
| `web/pbsdr.go` `handlePBSDRReissue` `SetHostDesired` (~L409) | operator (button) | agent | **NEW poke** | — |
|
||||
| `api/handler.go` `handleAdminSetDesiredState` `SetHostDesired` (~L1419) | operator/system (admin api) | agent | **NEW poke** (`PokeHost`) | — |
|
||||
| `api/wg.go` `handleAdminSetOperatorPeer` `BumpAllHostGenerations` (~L504) | operator (admin api) | agent | **NEW poke** (`PokeAllHosts`, fire-after-commit) | — |
|
||||
| `api/wg.go` `handleRegisterWGPeer` `BumpHostDesired` (~L291) | system (agent registration) | agent | **none (deliberate)** | box's tunnel does not exist until it fetches the WG block — a poke is undeliverable by construction; the agent fast-tick SECONDARY (R-28) owns this leg |
|
||||
| `api/wg.go` WG peer delete `BumpHostDesired` (~L422) | operator | agent | **none (deliberate)** | the mutation removes the transport |
|
||||
| `pbsdrheal` **Restage** path | system (reconciler) | agent | **none (deliberate)** | no generation bump → the agent's 60 s pbsdr ticker is the pickup path (a poke is a verified no-op); `internal/pbsdrheal/` byte-unchanged |
|
||||
| `api/handler.go` `reissueOnReenroll` (F2 claim + F3 offsite, ~L1071) | **system** (clean-slate re-enroll) | controller | **NEW intent bump** | — |
|
||||
| `web/configs.go` bumpIntent ×11 (L564/633/657/702/742/779/796/842/859/876/1055) | operator | controller | intent bump (pre-existing) | — |
|
||||
| `web/customer_reset.go` bumpIntent (~L226) | operator | controller | intent bump (pre-existing) | — |
|
||||
| `web/logtail.go` bumpIntent (~L35) / `web/logbundle.go` bumpIntent (~L68) | operator | controller | intent bump (pre-existing) | — |
|
||||
|
||||
### 9. Observations (not acted on)
|
||||
|
||||
- **`handlePBSDRReissue` ↔ `ReissuePBSDR` duplication.** The operator button re-implements the reissue
|
||||
atom's tail (reissue → store secret → descriptor bump) instead of calling `ReissuePBSDR`, deliberately,
|
||||
to keep granular per-step HTTP error strings (the "keep in lockstep" comment marks the known
|
||||
duplication). Both now poke after their `SetHostDesired`. Not deduped (per spec forbidden-dedup); if
|
||||
ever unified, the handler's error-string granularity is the constraint to preserve.
|
||||
- **`pbsdrProvisionAtom` is poke-free by design** — its two callers poke on success (form path at L186,
|
||||
hook path at the new L312). This keeps the poke at the decision point (blocked vs minted) rather than
|
||||
inside the shared atom, matching the existing L186 structure.
|
||||
|
||||
Reference in New Issue
Block a user