From 1a68b53b06ef9220717a2c60c57c54a4596ebae1 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 30 Jul 2026 10:42:43 +0200 Subject: [PATCH] hub v0.82.0 (R-120): the vouch path REFUSES a golden the fleet has already outrun The golden's version IS the controller it bakes (build-golden.sh:345 defaults GOLDEN_VERSION to the controller tag), so a golden behind the newest deployed controller means every FRESH install lands on stale application code. On the R-120 occurrence that stale code shipped a customer-facing falsehood: a box from the 0.185.1 golden told a customer whose backup drive had fallen out that the backup was on the same disk as the system -- false, the drive was gone -- and offered a different drive as the remedy. WHY A GATE, NOT A REMINDER. The gap has opened three times: R-111 (golden's agent 17 releases behind), R-115 (agent built and deployed, never published), R-120 (this). The first two were closed by re-baking and remembering; remembering then failed again. R-29 is the standing proof that a check nobody runs is worse than none because it reads as coverage -- hostinstall_gates.py sat RED and uninvoked across three version bumps and hub_confirm_gate.py has never run at all. So the property that matters is not whether a check exists but whether it BLOCKS. - Wired into handleSetArtifacts (internal/web/configs.go), immediately before the only write, on the sole UI path to SetArtifactManifest -- it runs on every vouch without anyone choosing to. A script in scripts/ would have been a fourth orphan. - It REFUSES (operator ruling, 2026-07-30), with a flash naming the remedy. - Signal: store.NewestReportedControllerVersion() over reports.controller_version, SEMVER-compared in Go -- MAX() in SQL ranks 0.99.0 above 0.186.0, a pair this fleet has shipped. No outbound call, no new credential. - Fail-open in exactly two deliberate cases: an empty golden field (clearing the manifest is legitimate) and an unknown fleet version (a new hub must vouch its first golden). NEAR-MISS RECORDED: the first draft read guests.controller_version, a column that exists in the schema and that NOTHING writes -- it would always have seen "" and failed open, i.e. inert, this gate's own failure shape. Caught by grepping for a writer before trusting the column. Blind spot stated rather than papered over: a controller no box has ever run is invisible to this signal. Not the failure that has bitten -- all three instances were deployed-newer-than-baked. 4 tests through the PRODUCTION handler over httptest, never an injected seam. The refusal asserts both the flash and that the manifest was NOT written, because a gate that redirects and saves anyway reads as enforcement while providing none. Red-proof: deleting the block makes the stale golden vouchable and both assertions fail. ROADMAP R-29's audit list now records this as the FIRST enforced gate, so the contrast with its three orphans is kept rather than lost. The orphans are unchanged. Suite rc=0 read separately from this commit. --- documentation/backlog/ROADMAP.md | 2 +- hub/CHANGELOG.md | 48 +++++++++ hub/internal/store/store.go | 42 ++++++++ hub/internal/web/configs.go | 31 ++++++ hub/internal/web/floor_ui_test.go | 98 +++++++++++++++++++ hub/internal/web/templates/configuration.html | 3 + manifests/hub.yaml | 2 +- 7 files changed, 224 insertions(+), 2 deletions(-) diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 61cef50..acc4eac 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -161,7 +161,7 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the | R-96 | **Two standing rules were agreed in chat and never committed** | XS | idea — found 2026-07-27 | Verified by grep across every `CLAUDE.md`, `CONTEXT.md`, `REUSE.md` and `skills/`: **(a) "never combine a test run and a commit in one command"** — zero hits, despite three separate occurrences (the worst pushed a red suite because `packages ok: 28` was read while `rc=1` was not); **(b) "a 'no access' claim must list what was tried"** — zero hits, despite a wrong "no access exists" verdict on ep0 on 2026-07-27 where the working route was documented in memory and only one path had been attempted. **The third rule IS committed** — `CONTEXT.md:8` S-1, N.5's third leg (architecture docs same-session coupled). A rule that lives only in a chat log binds nobody; both belong in `CONTEXT.md` beside S-1/S-2 | | R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog | | R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**) -| R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** **2026-07-29 — audit list extended, and a THIRD independent re-raise absorbed under the same rule (again no new ID):** add `scripts/hostinstall_gates.py`, which **postdates this item** (it comes from drill F-1, 2026-07-12) and is therefore not a design-v2 gate — but it is the identical failure shape and is tracked as **R-94 leg (b)**. It is **RED as of 2026-07-29**: `hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1, with its nine other assertions green. `scripts/hub_confirm_gate.py`, already on the list above, was **verified orphan on the same date**. Both confirmed by repo-wide grep across all file types plus sibling repos, `~/.claude` settings/skills/hooks, `.git/hooks` (no non-sample hooks exist), a Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) and a CI-directory find (**`felhom.eu` has no CI configuration at all**) — all 19 hits are docstrings, code comments or prose; **zero are invocations.** Only `site_gates.py` is mandated (`CLAUDE.md:153`); `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **Now also filed in `OPEN-ITEMS.md`** — this item predates the 2026-07-27 register rebuild and was never carried across, so an open item about work not getting done was itself missing from the page that decides what gets done | +| R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** **2026-07-29 — audit list extended, and a THIRD independent re-raise absorbed under the same rule (again no new ID):** add `scripts/hostinstall_gates.py`, which **postdates this item** (it comes from drill F-1, 2026-07-12) and is therefore not a design-v2 gate — but it is the identical failure shape and is tracked as **R-94 leg (b)**. It is **RED as of 2026-07-29**: `hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1, with its nine other assertions green. `scripts/hub_confirm_gate.py`, already on the list above, was **verified orphan on the same date**. Both confirmed by repo-wide grep across all file types plus sibling repos, `~/.claude` settings/skills/hooks, `.git/hooks` (no non-sample hooks exist), a Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) and a CI-directory find (**`felhom.eu` has no CI configuration at all**) — all 19 hits are docstrings, code comments or prose; **zero are invocations.** Only `site_gates.py` is mandated (`CLAUDE.md:153`); `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **Now also filed in `OPEN-ITEMS.md`** — this item predates the 2026-07-27 register rebuild and was never carried across, so an open item about work not getting done was itself missing from the page that decides what gets done. **2026-07-30 — THE FIRST ENTRY ON THE OTHER SIDE OF THE LEDGER, recorded so the contrast is not lost:** the **R-120 golden-staleness gate** (hub v0.82.0, `hub/internal/web/configs.go` `handleSetArtifacts`) **IS enforced.** It is not a script in `scripts/` that someone must remember; it sits inside the only UI path that writes `SetArtifactManifest`, so it runs on every vouch whether or not anyone chose to run it, and it **refuses** (operator ruling, 2026-07-30) rather than warning — because this row's whole finding is that a non-blocking check reads as coverage it is not providing. It compares the submitted golden against the newest controller any box has reported (`store.NewestReportedControllerVersion`) and is pinned by four tests driven through the production handler over `httptest`, not an injected seam, plus a red-proof: deleting the block makes the stale golden vouchable again. **Note the near-miss worth keeping:** the first draft read `guests.controller_version`, a column that exists in the schema and that **nothing writes** — it would have been an inert gate, i.e. this row's exact failure shape, caught by grepping for a writer before trusting the column. **The three orphans above are unchanged and still orphaned** — this entry proves the pattern is available, not that the backlog moved | | R-37 | **Post-RESET health card shows stale pre-RESET warnings.** After a RESET the card should read **„RESET óta nincs adat"** instead of carrying warnings about a lifecycle that no longer exists. | XS | **SHIPPED (hub v0.67.0, 2026-07-18)** | The customer page raises a banner when a RESET **completed** after the newest report, quoting „RESET óta nincs adat" and the reset timestamp, because until the box reports again every health figure describes a lifecycle that no longer exists. Deliberately narrow: an **in-flight** reset does not trigger it (only a completed one), and it **clears itself** on the first post-RESET report. Ties resolve to STALE — SQLite timestamps are second-resolution and a same-second report almost certainly arrived just before the reset destroyed what it describes; erring the other way would hide the banner exactly when it matters most. Red-proofed (neutering the predicate fails the assertion). — Origin: 2026-07-18 rehearsal. Same family as R-36 — the hub knows the state changed and the UI has not caught up | | R-38 | **Installer GRUB slice.** A single default „Felhom telepítés" entry; the **interactive installers REMOVED** (safety: an interactive entry is how a wrong-disk manual install happens); felhom background. | S | **SHIPPED (scripts v1.22.0, 2026-07-19)** | Origin: 2026-07-18 rehearsal, alongside R-21's physical closure. Every ISO is repacked after `prepare-iso`: exactly **one** entry („Felhom telepítés", default, 5 s), with Graphical / Terminal UI / serial and the whole **Advanced Options** submenu (nomodeset ×2, three debug variants, Rescue Boot, memtest, UEFI settings) **not emitted** — not hidden, not password-gated. Boot behavior unchanged: the `linux`/`initrd` lines are lifted **verbatim at repack time** from the ISO's own automated entry, so a PVE bump tracks automatically; the build fails if they are missing, if the append line has lost `proxmox-start-auto-installer`, or if `auto-installer-mode.toml` is absent. Gated for 1 entry / 0 submenus / no live banned references, then **re-verified by reading the menu back out of the finished ISO**. Background generated at repack time from `website/assets/og-image_2.png` (one source, no second copy to drift), with the card's own grid phase-locked across the letterbox fill. Live-validated on the nested canary (UEFI/OVMF): renders, auto-fires, reaches the same match-nothing abort as v1.21.0. **Squashfs/theme rebranding still DEFERRED** — post-GRUB screens remain Proxmox-branded. GRUB's `e`/`c` keys also remain reachable (suppressing them needs a superuser password — a separate decision); the menu no longer advertises them | diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 416f959..69123b7 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,51 @@ +## v0.82.0 — R-120: the vouch path refuses a golden the fleet has already outrun (2026-07-30) + +**The mechanism half of R-120.** The golden's version *is* the controller it bakes +(`felhom-agent configs/build-golden.sh:345` defaults `GOLDEN_VERSION` to `${CONTROLLER_IMAGE##*:}`), so a +golden left behind the newest deployed controller means every **fresh install** lands on stale +application code. On the R-120 occurrence that stale code shipped a customer-facing **falsehood**: a box +installed from the 0.185.1 golden told a customer whose backup drive had fallen out that *"the backup is +on the same disk as the system"* — false, the drive was gone — and offered a different drive as the +remedy. 0.186.0 is the release that made that message true, and no new box had it. + +**Why a gate and not a reminder.** This gap has opened **three times** — **R-111** (the golden's agent 17 +releases behind), **R-115** (an agent built and deployed but never published), **R-120** (this). The first +two were closed by re-baking and remembering; remembering then failed again. And **R-29** is the standing +proof that a check nobody runs is *worse* than none, because it reads as coverage: +`hostinstall_gates.py` sat RED and invoked by nothing across three version bumps while every report said +green, and `hub_confirm_gate.py` has never run at all. + +So the distinguishing property is not *does a check exist* but **does it block**: + +- It lives in **`handleSetArtifacts`** (`internal/web/configs.go`), immediately before the only write — + the sole UI path to `store.SetArtifactManifest`. It therefore runs on every vouch **without anyone + choosing to run it**. A script in `scripts/` asserting the same fact would have been a fourth orphan. +- It **REFUSES** (operator ruling, 2026-07-30), with an operator-legible flash naming the remedy, rather + than warning. +- Signal: `store.NewestReportedControllerVersion()` — the highest controller version any box has + reported, from `reports.controller_version` (the column `SaveReport` denormalises). **Semver-compared + in Go, not `MAX()` in SQL**, which would rank 0.99.0 above 0.186.0 — a pair this fleet has actually + shipped. No outbound call, no new credential. + +**Fail-open in exactly two cases, both deliberate:** an empty golden field (clearing the manifest is a +legitimate act) and an unknown fleet version (a new hub must be able to vouch its first golden). + +**Known blind spot, stated rather than papered over:** a controller no box has ever run is invisible to +this signal, so a golden baked behind an *unreleased* controller still passes. That is a real limit, and +it is not the failure that has bitten — all three instances were "deployed newer than baked". + +**A near-miss worth recording.** The first draft read `guests.controller_version` — a column that exists +in the schema (`store.go:294`) and that **nothing writes**. That gate would always have seen `""` and +failed open: inert, i.e. precisely the R-29 shape it exists to prevent. Caught by grepping for a writer +before trusting the column. + +**Tests: 4, through the production handler over `httptest`, never an injected seam** — because a gate +that can be inert is the thing this gate exists to prevent, and three shipped defects in this project +were fully green with the seam disconnected. Refusal asserts **both** the flash **and** that the manifest +was not written (a gate that redirects and saves anyway reads as enforcement while providing none); +plus the allow cases, both fail-open cases, and the semver-ordering case. Red-proof: deleting the block +makes the stale golden vouchable and both refusal assertions fail. + ## v0.81.0 — E-2: the absent backup target gets its own signal (2026-07-29) **Hub half of E-2, and it ships FIRST by necessity:** an event type the hub does not allowlist makes diff --git a/hub/internal/store/store.go b/hub/internal/store/store.go index 86830e8..168603c 100644 --- a/hub/internal/store/store.go +++ b/hub/internal/store/store.go @@ -1616,6 +1616,48 @@ func (s *Store) GetGlobalMinControllerVersion() string { return s.defaultMinControllerVersion } +// NewestReportedControllerVersion returns the highest controller version ANY customer has reported, or +// "" when none has. Semver-ordered in Go, not in SQL: `MAX(controller_version)` would compare lexically +// and rank 0.99.0 above 0.186.0 — which is the exact pair this gate has to get right. +// +// It reads `reports.controller_version`, the column SaveReport denormalises out of every report +// (store.go:903). It deliberately does NOT read `guests.controller_version`: that column exists in the +// schema (:294) and **nothing writes it**, so a gate keyed on it would always see "" and fail open — +// an inert gate, which is the exact class R-29 is about. Verified by grep before writing this. +// +// R-120's gate signal. The hub cannot ask "what is the newest controller that exists" — it has no +// registry credential and makes no outbound call at vouch time — but it does know what the FLEET is +// running, and that is the signal that matters: the failure this exists to catch is a golden left +// behind a controller **already deployed**. It has happened three times (R-111, R-115, R-120) and on +// the R-120 occurrence felhom-pve was reporting 0.186.0 while the manifest vouched a 0.185.1 golden — +// exactly the comparison below. +// +// KNOWN BLIND SPOT, stated rather than papered over: a controller no box has ever run is invisible +// here, so a golden baked behind an unreleased controller still passes. That is a real limit and it is +// not the failure mode that has bitten — the three instances were all "deployed newer than baked". +func (s *Store) NewestReportedControllerVersion() string { + rows, err := s.db.Query(`SELECT DISTINCT controller_version FROM reports WHERE controller_version IS NOT NULL AND controller_version != ''`) + if err != nil { + return "" // unreadable → the gate degrades to "cannot compare", never to a false refusal + } + defer rows.Close() + newest := "" + for rows.Next() { + var v string + if rows.Scan(&v) != nil { + continue + } + v = strings.TrimPrefix(strings.TrimSpace(v), "v") + if v == "" { + continue + } + if newest == "" || semver.Compare(v, newest) > 0 { + newest = v + } + } + return newest +} + // GlobalFloorResolution is the full picture of the effective global floor for the operator UI: the // resolved value + WHICH source won + both raw inputs. It makes the "a manifest save silently armed // a live floor" incident (publish-train 0.81/0.113) permanently visible — the operator can see the diff --git a/hub/internal/web/configs.go b/hub/internal/web/configs.go index 2ad4db1..27e7204 100644 --- a/hub/internal/web/configs.go +++ b/hub/internal/web/configs.go @@ -1139,6 +1139,37 @@ func (s *Server) handleSetArtifacts(w http.ResponseWriter, r *http.Request) { http.Redirect(w, r, "/configuration?flash=artifact_sha_invalid", http.StatusSeeOther) return } + // ── R-120 GATE — refuses a golden older than the controller the fleet is already running ───────── + // + // WHY THIS IS A GATE AND NOT A SCRIPT. The golden's version IS the controller it bakes + // (felhom-agent configs/build-golden.sh: GOLDEN_VERSION defaults to ${CONTROLLER_IMAGE##*:}), so a + // golden behind the newest deployed controller means every FRESH install lands on stale + // application code. That has happened three times — R-111 (the golden's agent 17 releases behind), + // R-115 (an agent built and deployed but never published), R-120 (this: the golden a controller + // release behind, shipping a customer-facing FALSEHOOD, since 0.186.0 is what made the + // absent-backup-target message true). The first two were closed by re-baking and remembering, and + // remembering then failed again — which is why this is enforcement, not a reminder. + // + // It lives HERE, immediately before the only write, because handleSetArtifacts is the sole UI path + // to SetArtifactManifest: it therefore runs without anyone choosing to run it. R-29 is the standing + // proof that the alternative does not work — `hostinstall_gates.py` sat RED and invoked by nothing + // across three version bumps while every report said green, and `hub_confirm_gate.py` has never run + // at all. A check in scripts/ asserting this same fact would have been a fourth orphan. + // + // It REFUSES rather than warns (operator ruling, 2026-07-30): a non-blocking check reads as coverage + // it is not providing, which is R-29's whole finding. + // + // FAIL-OPEN, deliberately, in exactly two cases: an empty golden field (clearing the manifest is a + // legitimate operator act) and an unknown fleet version (no guest has reported one — a brand-new hub + // must be able to vouch its first golden). Neither is the drift this catches. + if goldenVer != "" { + if newest := s.store.NewestReportedControllerVersion(); newest != "" && compareVersions(goldenVer, newest) < 0 { + s.logger.Printf("[WARN] artifact vouch REFUSED: golden %s is older than the newest controller the fleet reports (%s) — "+ + "a fresh install would land on stale application code (R-120)", goldenVer, newest) + http.Redirect(w, r, "/configuration?flash=golden_behind_fleet", http.StatusSeeOther) + return + } + } if err := s.store.SetArtifactManifest(store.ArtifactManifest{ AgentVersion: agentVer, AgentSHA256: agentSHA, diff --git a/hub/internal/web/floor_ui_test.go b/hub/internal/web/floor_ui_test.go index b74a094..cb62bc5 100644 --- a/hub/internal/web/floor_ui_test.go +++ b/hub/internal/web/floor_ui_test.go @@ -152,3 +152,101 @@ func excerpt(s string) string { } return s } + +// ── R-120: the vouch path REFUSES a golden older than the controller the fleet already runs ────── +// +// These drive handleSetArtifacts — THE production vouch path, the only UI writer of +// SetArtifactManifest — over httptest. Deliberately NOT through an injected seam: the whole point of +// this gate is that it cannot be inert, and three shipped defects in this project were fully green +// with the seam disconnected. The assertions check BOTH the operator-visible outcome and that the +// manifest was not written, because a gate that redirects but still saves is worse than none. + +// RED-PROOF: delete the `goldenVer != ""` gate block in handleSetArtifacts → this fails, because the +// stale golden is accepted and the manifest is overwritten. +func TestVouchRefusesGoldenBehindFleet(t *testing.T) { + s, st := newTestServer(t) + // The R-120 situation exactly: a box is running 0.186.0 while the operator vouches a 0.185.1 golden. + seedReport(t, st, "demo", "0.186.0") + if err := st.SetArtifactManifest(store.ArtifactManifest{ + AgentVersion: "0.116.0", GoldenVersion: "0.186.0", + AgentSHA256: strings.Repeat("a", 64), GoldenSHA256: strings.Repeat("b", 64), + }); err != nil { + t.Fatal(err) + } + + form := url.Values{"agent_version": {"0.116.0"}, "golden_version": {"0.185.1"}} + form.Set("agent_sha256", strings.Repeat("c", 64)) + form.Set("golden_sha256", strings.Repeat("d", 64)) + r := httptest.NewRequest(http.MethodPost, "/configuration/artifacts", strings.NewReader(form.Encode())) + r.Header.Set("Content-Type", "application/x-www-form-urlencoded") + w := httptest.NewRecorder() + s.handleSetArtifacts(w, r) + + if got := w.Header().Get("Location"); !strings.Contains(got, "golden_behind_fleet") { + t.Errorf("expected the refusal flash, got Location=%q — the operator would not know it was refused", got) + } + // The load-bearing half: the manifest must be UNCHANGED. + if m := st.GetArtifactManifest(); m.GoldenVersion != "0.186.0" { + t.Fatalf("REFUSED but still wrote the manifest: golden=%q, want 0.186.0 untouched. A gate that "+ + "redirects and saves anyway is worse than no gate — it reads as enforcement.", m.GoldenVersion) + } +} + +// The gate must not block legitimate vouches, or it gets disabled and becomes another R-29 orphan. +func TestVouchAllowsGoldenAtOrAheadOfFleet(t *testing.T) { + s, st := newTestServer(t) + seedReport(t, st, "demo", "0.186.0") + for _, golden := range []string{"0.186.0", "0.187.0"} { + form := url.Values{"agent_version": {"0.116.0"}, "golden_version": {golden}} + form.Set("agent_sha256", strings.Repeat("a", 64)) + form.Set("golden_sha256", strings.Repeat("b", 64)) + r := httptest.NewRequest(http.MethodPost, "/configuration/artifacts", strings.NewReader(form.Encode())) + r.Header.Set("Content-Type", "application/x-www-form-urlencoded") + w := httptest.NewRecorder() + s.handleSetArtifacts(w, r) + if m := st.GetArtifactManifest(); m.GoldenVersion != golden { + t.Errorf("golden %s (>= fleet 0.186.0) should be vouchable, manifest holds %q", golden, m.GoldenVersion) + } + } +} + +// Fail-open case 1: a hub whose fleet has reported nothing must be able to vouch its first golden. +func TestVouchAllowedWhenFleetVersionUnknown(t *testing.T) { + s, st := newTestServer(t) + form := url.Values{"agent_version": {"0.116.0"}, "golden_version": {"0.185.1"}} + form.Set("agent_sha256", strings.Repeat("a", 64)) + form.Set("golden_sha256", strings.Repeat("b", 64)) + r := httptest.NewRequest(http.MethodPost, "/configuration/artifacts", strings.NewReader(form.Encode())) + r.Header.Set("Content-Type", "application/x-www-form-urlencoded") + w := httptest.NewRecorder() + s.handleSetArtifacts(w, r) + if m := st.GetArtifactManifest(); m.GoldenVersion != "0.185.1" { + t.Errorf("with no reported fleet version the gate must fail OPEN; manifest holds %q", m.GoldenVersion) + } +} + +// Fail-open case 2: clearing the golden field is a legitimate operator act, not drift. +func TestVouchAllowsClearingGolden(t *testing.T) { + s, st := newTestServer(t) + seedReport(t, st, "demo", "0.186.0") + form := url.Values{"agent_version": {"0.116.0"}, "golden_version": {""}} + form.Set("agent_sha256", strings.Repeat("a", 64)) + r := httptest.NewRequest(http.MethodPost, "/configuration/artifacts", strings.NewReader(form.Encode())) + r.Header.Set("Content-Type", "application/x-www-form-urlencoded") + w := httptest.NewRecorder() + s.handleSetArtifacts(w, r) + if got := w.Header().Get("Location"); strings.Contains(got, "golden_behind_fleet") { + t.Error("clearing the golden must not trip the gate") + } +} + +// The comparison must be SEMVER, not lexical — 0.99.0 vs 0.186.0 is the pair that breaks string order, +// and it is not hypothetical: the fleet has shipped both 0.9x and 0.18x controllers. +func TestNewestReportedControllerIsSemverOrdered(t *testing.T) { + _, st := newTestServer(t) + seedReport(t, st, "a", "0.99.0") + seedReport(t, st, "b", "0.186.0") + if got := st.NewestReportedControllerVersion(); got != "0.186.0" { + t.Errorf("NewestReportedControllerVersion() = %q, want 0.186.0 — a lexical MAX() would return 0.99.0", got) + } +} diff --git a/hub/internal/web/templates/configuration.html b/hub/internal/web/templates/configuration.html index d651d37..9275c05 100644 --- a/hub/internal/web/templates/configuration.html +++ b/hub/internal/web/templates/configuration.html @@ -47,6 +47,9 @@ {{if eq .Flash "artifact_sha_invalid"}}
Couldn't set the checksum — the Gitea sha lookup failed (version missing / Gitea unreachable) or the manually-entered sha is invalid. Manifest unchanged.
{{end}} + {{if eq .Flash "golden_behind_fleet"}} +
Refused: that golden is older than the controller the fleet already runs. A fresh install would land on stale application code — which is R-120, where new boxes shipped a controller that told customers the wrong thing about a missing backup drive. Manifest unchanged. Re-bake the golden on the current controller, publish it, then vouch it here.
+ {{end}} {{if eq .Flash "pw_changed"}}
Login password changed. It is already in effect — use it next time you sign in. Existing sessions stay logged in.
{{end}} diff --git a/manifests/hub.yaml b/manifests/hub.yaml index 92e3617..970eb2e 100644 --- a/manifests/hub.yaml +++ b/manifests/hub.yaml @@ -125,7 +125,7 @@ spec: spec: containers: - name: hub - image: gitea.dooplex.hu/admin/felhom-hub:0.81.0 + image: gitea.dooplex.hu/admin/felhom-hub:0.82.0 ports: - containerPort: 8080 name: http