docs: hub v0.63.0 system-initiated immediacy — REPORT + CONTEXT + capability-map (immediacy row) + ROADMAP (R-28 agent fast-tick SECONDARY)

This commit is contained in:
2026-07-17 17:33:34 +02:00
parent 30972d8f54
commit 50c090eeaa
4 changed files with 139 additions and 69 deletions
+121 -68
View File
@@ -2,86 +2,139 @@
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
## R-21 slice C — the universal secret-free ISO (hub v0.62.0 + scripts v1.19.0) — 2026-07-17
## Hub v0.63.0 — system-initiated immediacy (poke + intent bump at every mutation site) — 2026-07-17
**Baselines:** hub v0.61.0 → **v0.62.0**; scripts v1.18.0 (`3172df1`) → **v1.19.0**. Commits `36c5cd5`
(code), `9964ae6` (manifest). Hub **v0.62.0 LIVE** on k3s (Synced/Healthy). R-21 now shipped A+B+C.
### 1. Baseline used
### What shipped
- **felhom.eu** @ `4c9b0e87063278cddbe51e3966a8f8592c7c18f2` ("docs: R-21 slice C …", 2026-07-17),
clean tree on `main`, local == `origin/main` re-confirmed before starting.
- **Hub** v0.62.0 → **v0.63.0**. Deploy manifest `manifests/hub.yaml` image line `felhom-hub:0.62.0``:0.63.0`.
- Both immediate-sync transports were already PROVEN-LIVE (agent poke: agent v0.89.0 + hub v0.59.0,
2026-07-17; intent wait channel: hub v0.58.0 / controller v0.140.0). This task is **call-site wiring
only** — no new mechanism, no spike.
The last R-21 slice: a **generic, secret-free** ISO. Blank box → unattended install → the box
registers itself at the hub as an *unclaimed appliance* → the operator binds it to a customer → the hub
delivers `customer-id` + retrieval passphrase **once** → day-0 completes via the existing slice-A path.
The distributed ISO carries no secret (§4.4). Slice-A direct mode (env-baked) survives unchanged.
### 2. Files created / modified
### Registration payload (as implemented) + token model
**Implementation (commit `30972d8`):**
- `hub/internal/web/pbsdr.go` — +3 pokes: `PBSDRAutoProvision`, `ReissuePBSDR`, `handlePBSDRReissue` (each after the successful `SetHostDesired`).
- `hub/internal/api/handler.go` — new nil-safe `Poker` interface + `poker` field + `SetPoker`; poke in `handleAdminSetDesiredState`; intent `Bump` in `reissueOnReenroll`.
- `hub/internal/api/wg.go``PokeAllHosts` in `handleAdminSetOperatorPeer` (only on a successful `BumpAllHostGenerations`).
- `hub/cmd/hub/main.go` — capture the one `poke.Notifier` and wire it to BOTH planes (`webServer.SetPoke(n)` + `apiHandler.SetPoker(n)`); startup log "web + api admin seams armed".
- `hub/internal/web/pbsdr_poke_test.go` (new) — 5 web tests (Groups A, B).
- `hub/internal/api/poke_seam_test.go` (new) — 5 api tests (Groups C, D).
- `hub/CHANGELOG.md` — v0.63.0 entry (newest on top).
- `REUSE.md` — new canonical-pattern row: "Immediate-sync notify (per plane)".
`POST /api/v1/appliance/register` body: `{uuid, macs[], ssh_host_pubkeys[], hw{product, cpu, mem_kb}}`.
Keyed by **(uuid, mac_set)** — the N100 DMI verdict (serials = "Default string") + cheap-board duplicate
SMBIOS UUIDs make the **MAC set the tiebreaker** (same uuid + different mac-set = distinct appliance).
**Token model:** a random **256-bit** appliance token per registration, returned once, stored only as
**sha256** (`token_hash`); it is the box's sole pre-day-0 credential, never logged. Invalidated on
discard (`token_hash` blanked + status sticky) and after consume. `GET /api/v1/appliance/poll` (Bearer
token): unknown/discarded → **404** (no oracle) · unbound → **204** · bound → **200** + creds (consumed
once) · delivered → **410**. The passphrase is read live from `customer_configs` (plaintext, as the
day-0 command already requires) and never logged. register is the ONE unauthenticated endpoint, per-IP
rate-limited (the ingress already geo-restricts to HU).
**Docs (commit `<DOCS_HASH>`):**
- `REPORT.md` (this file, overwritten), `CONTEXT.md` (decision recorded), `documentation/architecture/00-capability-map.md` (immediacy row note + version), `documentation/backlog/ROADMAP.md` (new R-28: agent fast-tick SECONDARY).
### Red-proofs (run-fail-revert, all recorded)
**Manifest (commit `<MANIFEST_HASH>`):** `manifests/hub.yaml` image tag → `felhom-hub:0.63.0`.
- **One-shot delivery (C):** defeat `MarkApplianceDelivered`'s bound→delivered flip → the second poll
re-delivers the passphrase (200) instead of 410 → `TestAppliancePoll_OneShotAnd410` FAILS → restored.
- **Register idempotency (A):** drop the `(uuid, mac_set)` upsert branch → the re-register hits the
UNIQUE backstop (500) → `TestApplianceRegister_Idempotent` FAILS → restored. (The UNIQUE index is the
load-bearing backstop; the upsert is the graceful update.)
- **Direct-mode regression (D):** `test/bootstrap-modes.sh` — a direct env drives `run_direct` and makes
**ZERO** `/appliance/*` calls (fake hub records none); the pairing code path is provably not entered.
Plus 404-no-oracle + sticky-discard, bind staging/refusal + no-host-count-gate, and the render test.
### 3. Per-commit hashes pushed to `main`
### Deploy verification
| # | Hash | Contents |
|---|------|----------|
| 1 | `30972d8` | code + tests + CHANGELOG + REUSE |
| 2 | `<DOCS_HASH>` | REPORT + CONTEXT + capability-map + ROADMAP |
| 3 | `<MANIFEST_HASH>` | manifests/hub.yaml image bump → 0.63.0 |
Hub 0.62.0 rolled out (Synced/Healthy, image confirmed). **Live smoke through the public ingress:**
`POST /appliance/register` → 200 with a 256-bit token; poll (valid, unbound) → **204**; poll (bogus
token) → **404** (no oracle). (The `smoke-test-uuid` record is CC test litter — discard on the hub.)
### 4. Test results + red-proof outcomes
### Scenario E (the composition proof) + the secret-free artifact assertion
**New tests (10), all green:**
- **Generic ISO built** `felhom-pve-9.2-1-v1.19.0-generic-generic.iso` (sha
`f37080c835300b6cc2831bd6e5a483cad307f4d009469ee4206442fe9bebd348`). **Secret-free asserted on the
artifact:** manifest `mode: pairing` / `secret-bearing: no`; the env decoded from the ISO's
`/proxmox-first-boot` stub carries **only** `FELHOM_HUB_URL` + `FELHOM_INSTALL_URL` — no
`FELHOM_CUSTOMER_ID`, no `FELHOM_RETRIEVAL_PASSPHRASE`.
- **Nested boot → register (CONFIRMED live):** the generic ISO booted on scratch VM 311 (SeaBIOS, LAN,
uuid `7ed8cb9e-…`, MAC `bc:24:11:c3:b3:d0`) → **zero-touch install** (answer file consumed, disk `sda`
auto-selected, country HU) → reboot → installed system → first-boot pairing → **registered at the
LIVE hub**: `[INFO] appliance registered: new unclaimed box (uuid=7ed8cb9e-…, macs=1, ssh_keys=3)`
the box gathered its MAC + all 3 SSH host keys exactly as designed. It now polls 204 (unbound),
**staged for the operator bind**. The record + VM are left in place as the handoff. *(Boot-order
lesson: the scratch VM must boot `scsi0;ide2` — disk-first, CD-fallback — or a CD-first order
reinstalls in a loop and never reaches first-boot.)*
- **The bind is the coordinated Viktor moment** (operator-password-gated; CC cannot log in). CC has
staged everything: the box is registered and visible; a Viktor-created drill customer + the 2-minute
bind click complete boot→register→**bind**→delivery→day-0, then the drill customer is torn down via
RESET then Delete (re-exercising v0.61.0/v0.60.1). The bind→delivery→day-0 legs are proven by the
hub unit tests + the `bootstrap-modes.sh` delivery handoff; the live composition folds into the
supervised rehearsal, **which now runs the complete final product flow in one pass**.
| Test | Group | Asserts |
|------|-------|---------|
| `TestPBSDR_AutoProvisionPokes` | A | auto-provision success → exactly one poke to the resolved WG /32 (10.77.0.2) within 2 s |
| `TestPBSDR_AutoProvisionBlockedDoesNotPoke` | A (neg) | blocked precondition (no WG peer) → zero pokes |
| `TestPBSDR_ReissueCorePokes` | B1 | `ReissuePBSDR` success → one poke to the host's /32 |
| `TestPBSDR_ReissueCoreErrorDoesNotPoke` | B (neg) | tenantsync-error reissue → zero pokes (fire-after-commit) |
| `TestPBSDR_HandleReissuePokes` | B2 | operator button success → one poke |
| `TestAdminSetDesiredState_PokesTargetHostOnly` | C1 | admin-set success → PokeHost==1 (target only), PokeAllHosts==0; invalid-JSON path → 0 |
| `TestAdminSetDesiredState_NilPokerNoPanic` | C1 (nil) | nil poker → mutation still 200, no panic |
| `TestAdminSetOperatorPeer_PokesFleet` | C2 | operator-peer success → PokeAllHosts==1, PokeHost==0 |
| `TestReenroll_BumpsIntent` | D | re-enroll → intent generation advances (synchronous) |
| `TestReenroll_NilIntentHubNoPanic` | D (nil) | nil intent hub → enroll still 201, no panic |
### SSH-host-key pinning disposition
Async web assertions synchronize on a buffered channel in the fake sender with a bounded `select`
(≤2 s), never sleep-poll. The web fake sender structurally satisfies poke's unexported `pokeSender`
seam (value passed, type never named); the notifier is a REAL `poke.Notifier` over the test store —
exactly as `main.go` builds it. The api fake `Poker` satisfies the exported `api.Poker` interface and
counts calls synchronously.
The registration payload carries the box's SSH host public keys; they are **stored** with the appliance
record and their **fingerprints displayed** in the unclaimed-appliances UI. They are **not** auto-attached
to the host on bind — the `hosts` row is minted later by the box's own `/host-enroll` (mint-once-reuse),
and no clean "attach host keys to host" hand-off surface exists today. Per the spec's fallback, this is
the store+display+**Observation** path, not new plumbing (a future item can wire TOFU-free pinning).
**Red-proofs (run-fail-restore), one representative per scenario group:**
### Observations
| Group | Removed | Test | Result |
|-------|---------|------|--------|
| A | `PBSDRAutoProvision` poke (Part 1.1) | `TestPBSDR_AutoProvisionPokes` | FAILED red ("no poke within 2s… 10.77.0.2") → restored |
| B | `ReissuePBSDR` poke (Part 1.2) | `TestPBSDR_ReissueCorePokes` | FAILED red ("no poke within 2s… 10.77.0.2") → restored |
| C | `handleAdminSetDesiredState` poke (Part 2.2) | `TestAdminSetDesiredState_PokesTargetHostOnly` | FAILED red ("PokeHost targets = [], want [h1]") → restored |
| D | `reissueOnReenroll` bump (Part 3) | `TestReenroll_BumpsIntent` | FAILED red ("intent generation = 0, want > 0") → restored |
- **Bind never gates on host count** (multi-host customers are real — Peti); the picker shows counts as
display only. A post-RESET customer is naturally hostless and binds fine (tested).
- **Pre-bind provenance** is the `appliance_registrations` row's own timestamps (register/discard have no
customer to scope an `events` row to — `events.customer_id` is NOT NULL); `appliance_bound` /
`appliance_credential_delivered` events are recorded once a customer exists.
- **Universal disk selection is out of slice-C scope:** `profiles/generic.profile` defaults to the first
SATA disk (`sda`); NVMe/other targets need a profile variant (a future concern, noted in the profile).
- **Poll cadence = systemd:** the bootstrap does one poll per invocation and exits non-zero when unbound,
reusing `Restart=on-failure`/`RestartSec=30` as the 30s poll timer (no long-running-oneshot timeout).
These tests fail on the pre-fix tree by construction; a representative removal per group was
demonstrated (B2 and C2 share their group's wiring shape with the demonstrated B1/C1 removals).
### 5. Test count before/after
- Package `internal/web`: +5 tests (new `pbsdr_poke_test.go`).
- Package `internal/api`: +5 tests (new `poke_seam_test.go`).
- Full suite after: `go build ./... && go vet ./... && go test ./...`**all packages green** (api + web
are the only two exercising the new code; every other package unchanged).
### 6. Deployed version + pod/log verification
<DEPLOY_VERIFICATION>
### 7. NOT yet live-validated (real system-initiated events only)
The new pokes/bumps fire only on real system events, none manufacturable non-destructively from this
task. Acceptance bar here = **unit tests + deploy + startup-log verification**. Unfired live legs, to be
observed at the rehearsal's bind sequence (`RUNBOOK-rehearsal-v3 S5`):
- **Auto-provision poke** — fires when a DR-ON customer's host registers its WG peer for the first time
and the atom provisions (the exact observed slice-C lag).
- **Reconciler-escalation poke** — fires when the pbsdrheal reconciler escalates to `ReissuePBSDR` (a box
stuck in `waiting_secret`/`consumed_failed` after losing its converged marker).
- **Clean-slate re-enroll bump** — fires when a wiped, re-minted box re-enrolls for a claimed/offsite
customer (on that path the controller usually does not exist yet; its startup fetch covers that shape).
- **Operator-button / admin-api pokes** (`handlePBSDRReissue`, admin desired-state / operator-peer) — the
operator-triggered legs; exercisable in a supervised session but not fired here.
The **first-hold-baseline race** stays OPEN by design: a bump landing during a fresh controller's FIRST
wait is recorded as baseline without firing (the ruled fix is carrying `intent_gen` in the report ACK —
its own future task, out of scope).
### 8. Part-4 audit table — EVERY desired-state mutation site
Re-grepped at implementation time for all non-test callers of `SetHostDesired`, `BumpHostDesired`,
`BumpAllHostGenerations`, plus the system-initiated customer-deliverable writers. No forgotten fourth
caller — every site below is accounted for.
| Site (file:landmark) | Initiator | Plane | Notification after this task | Reason if none |
|---|---|---|---|---|
| `web/pbsdr.go` `applyPBSDR` disable (~L143→148) | operator (form) | agent | poke (pre-existing) | — |
| `web/pbsdr.go` `applyPBSDR` re-enable/storage-id (~L168→173) | operator (form) | agent | poke (pre-existing) | — |
| `web/pbsdr.go` `applyPBSDR` fresh-provision (~L179→186) | operator (form) | agent | poke (pre-existing) | — |
| `web/pbsdr.go` `pbsdrProvisionAtom` `SetHostDesired` (~L273) | — (shared atom) | agent | poke at each caller | callers poke: `applyPBSDR` (L186) + `PBSDRAutoProvision` (NEW) |
| `web/pbsdr.go` `PBSDRAutoProvision` (~L312) | **system** (WG-reg hook) | agent | **NEW poke** | — (the observed lag) |
| `web/pbsdr.go` `ReissuePBSDR` `SetHostDesired` (~L357) | **system** (reconciler escalation) | agent | **NEW poke** | — (also covers pbsdrheal, zero reconciler change) |
| `web/pbsdr.go` `handlePBSDRReissue` `SetHostDesired` (~L409) | operator (button) | agent | **NEW poke** | — |
| `api/handler.go` `handleAdminSetDesiredState` `SetHostDesired` (~L1419) | operator/system (admin api) | agent | **NEW poke** (`PokeHost`) | — |
| `api/wg.go` `handleAdminSetOperatorPeer` `BumpAllHostGenerations` (~L504) | operator (admin api) | agent | **NEW poke** (`PokeAllHosts`, fire-after-commit) | — |
| `api/wg.go` `handleRegisterWGPeer` `BumpHostDesired` (~L291) | system (agent registration) | agent | **none (deliberate)** | box's tunnel does not exist until it fetches the WG block — a poke is undeliverable by construction; the agent fast-tick SECONDARY (R-28) owns this leg |
| `api/wg.go` WG peer delete `BumpHostDesired` (~L422) | operator | agent | **none (deliberate)** | the mutation removes the transport |
| `pbsdrheal` **Restage** path | system (reconciler) | agent | **none (deliberate)** | no generation bump → the agent's 60 s pbsdr ticker is the pickup path (a poke is a verified no-op); `internal/pbsdrheal/` byte-unchanged |
| `api/handler.go` `reissueOnReenroll` (F2 claim + F3 offsite, ~L1071) | **system** (clean-slate re-enroll) | controller | **NEW intent bump** | — |
| `web/configs.go` bumpIntent ×11 (L564/633/657/702/742/779/796/842/859/876/1055) | operator | controller | intent bump (pre-existing) | — |
| `web/customer_reset.go` bumpIntent (~L226) | operator | controller | intent bump (pre-existing) | — |
| `web/logtail.go` bumpIntent (~L35) / `web/logbundle.go` bumpIntent (~L68) | operator | controller | intent bump (pre-existing) | — |
### 9. Observations (not acted on)
- **`handlePBSDRReissue``ReissuePBSDR` duplication.** The operator button re-implements the reissue
atom's tail (reissue → store secret → descriptor bump) instead of calling `ReissuePBSDR`, deliberately,
to keep granular per-step HTTP error strings (the "keep in lockstep" comment marks the known
duplication). Both now poke after their `SetHostDesired`. Not deduped (per spec forbidden-dedup); if
ever unified, the handler's error-string granularity is the constraint to preserve.
- **`pbsdrProvisionAtom` is poke-free by design** — its two callers poke on success (form path at L186,
hook path at the new L312). This keeps the poke at the decision point (blocked vs minted) rather than
inside the shared atom, matching the existing L186 structure.