felhom.eu/documentation/runbooks/publish-train-rules.md

Publish-train rules — coupled controller/agent releases

Standing gates for every artifact publish train (agent binary + controller golden + Day-0 manifest + floor). Each rule cites the incident that created it. The box-level backstop (rule 4) exists because rules 1–3 are operator discipline and discipline fails; the rules stay the primary control.

1. Manifest before floor (the GL-1 rule, restated)

Publish and vouch the artifacts in the Day-0 manifest before any floor movement. A floor that points at an unvouched (or unpublished) version bricks self-updates: boxes are told to move to a version they cannot verify. (GL-1 supply-chain arc; see the go-live package records.)

Since hub v0.112.0 (operator ruling 2026-09-13, R-472): the manifest leads the floor inside the golden; above it, the release's own declared MinAgent does. A controller image is pulled by tag from the registry, so a floor above the golden does not need a golden to be delivered — it needs to know which agent the release requires. The operator declares that with the floor, in the min_agent field next to it, read from the release's controller CHANGELOG.md header (**MinAgent: X.Y.Z**, pinned by controller/scripts/minagent_header_gate.py).

• At or below the vouched golden: the manifest's MinAgent governs, exactly as before. A declared value is recorded and not used.
• Above the golden, declared: the hub compares each box's agent against the declared MinAgent (rule 3's comparison, same hold text). Hub log: managed floor SERVED … from declared. The Hosts page marks the box floor: declared MinAgent.
• Above the golden, undeclared: still HELD beyond the golden, unchanged (R-216). Both floor forms refuse to save such a floor (floor_needs_min_agent), so the hold is now a sentence at save time instead of a silent fleet-wide stop.
• A declaration belongs to the exact floor it was saved with (FLOOR=MINAGENT). Moving the floor without a new declaration never carries the old one forward.

2. The manifest screen carries the LIVE DB floor — save the floor field LAST

The hub's Day-0 manifest UI also persists the global floor as a DB row (hub_settings.min_controller_version — hub/internal/store/store.go, GetGlobalMinControllerVersion). The DB row overrides the env floor and acts immediately on the next report cycle — it is not staged by the manifest vouch.

Incident: publish train 0.81/0.113 (2026-07-11, documentation/pilot/RUNBOOK-publish-0.81-0.113-2026-07-11.md): the operator saved the manifest screen with the floor field filled; the floor acted at once and controller 0.113.0 reached Peti's box ~9 minutes before agent 0.81.0 — the exact forbidden skew, benign only because the box had zero NAS shares.

Rule: fill the floor field last, in a separate save, only after rule 3's fleet check passes.

3. MinAgent fleet gate — HUB-ENFORCED PER-BOX since hub v0.45.0

A controller release that depends on coupled agent behavior declares MinAgent: X.Y.Z in its CHANGELOG entry header line (felhom-controller convention, since v0.114.0; retroactively, 0.113.0's effective MinAgent was 0.81.0 for the NAS add). The floor may not effectively push a box past a controller whose MinAgent that box's agent does not yet meet.

Hub-enforced per-box since hub v0.45.0 (the manual fleet check is retired): the operator sets the golden's MinAgent in the Day-0 artifact manifest (Configuration → Day-0 artifacts → "Min agent"; blank = uncoupled release, no gating). At report-ACK time the hub compares each box's reported hosts.agent_version against that MinAgent (store.ResolveManagedFloor, hub/internal/store/store.go): agent ≥ MinAgent → the controller floor is served; agent below MinAgent or unknown → the floor is HELD (the ACK omits the directive) and the box is flagged on the Hosts dashboard (floor held: agent <v> < MinAgent <w>) — a held box is visible, never silently stale. The operator still SETS MinAgent at manifest time; the hub does the per-box gating. This mechanises the "agent BEFORE controller floor" ordering that rule 2's incident violated by hand.

Effective-floor visibility (same v0.45.0): the floor card shows the resolved value + its source (DB hub_settings vs env DEFAULT_MIN_CONTROLLER_VERSION, both raw values when they differ), and the floor save is behind a type-to-confirm dialog stating the live below-floor blast radius — so rule 2's "the floor acts immediately" is impossible to miss.

4. Box-level backstop: the controller's capability gate

Since controller v0.114.0, a coupled feature entry point probes the agent's capability (controller/internal/agentapi/features.go — route probe: 2xx ⇒ supported, 404 ⇒ older agent, transport/5xx ⇒ indeterminate, never "too old") and refuses up front with an honest Hungarian message instead of failing mid-pipeline. This turns a violated ordering into a graceful refusal — it does not license sloppy trains: rules 1–3 remain the primary control.

Convention for new coupled features: add a row to the featureProbes table AND the featureMinAgent table + a Supports gate call at the feature's entry point, and declare MinAgent per rule 3. Since controller v0.115.0 + agent v0.82.0 the agent reports its version in the X-Felhom-Agent-Version response header, so Supports decides by version comparison when the version is known and only falls back to the route probe for header-less (≤0.81) agents — capability detection is now explicit, not probe-inferred.

5. Every ISO build asserts golden ≥ the managed floor (the R-71 gate)

Incident: DIAG-f10-demo-hp-offsite-2026-07-23.md / R-71. A box installed from an ISO whose golden controller is BELOW the hub's managed floor boots below the floor, so the day-0 managed update fires within minutes of first boot — racing the offsite apply-bridge in exactly the window that burned demo-hp's one-time offsite credential (2 days unprotected). The gap is invisible at build time unless something checks it.

Rule: a build where the managed floor exceeds golden must fail loudly, at build time, not ship. build-felhom-iso.sh carries the assertion itself (assert_golden_ge_floor, gated on FELHOM_ASSERT_GOLDEN = the hub's artifact_golden_version and FELHOM_ASSERT_FLOOR = its min_controller_version); it prints both versions and dies on golden < floor. Resolve the two values operator-side before the build and pass them in — e.g. from the hub DB (hub_settings.artifact_golden_version / min_controller_version) or the operator artifacts UI — so the gate is enforced, not skipped (an unset input warns LOUDLY and does not silently pass). The fix for a tripped gate is never to lower the floor: republish golden ≥ floor and vouch it (rule 1), then rebuild. This gate composes with rule 1 — the manifest still leads the floor; this one stops an ISO from carrying a golden the floor has already outrun.

Dated note, 2026-08-18 — the current pair is golden 0.216.0 / floor 0.216.0. Equal, so the R-71 gate passes. Recorded so the next build has numbers to hand, and deliberately not as a replacement for the instruction above: resolve both LIVE from the hub before a build. A snapshot that looks authoritative is exactly how build-golden.sh's hand-bumped CONTROLLER_IMAGE default rotted twice. Treat these as a sanity check on what you read, never as a substitute for reading it. (Floor raised to 0.216.0 by the operator on 2026-08-18 in a separate save after the vouch, per rule 2 — see R-343, which records that the raise moved demo-felhom nine seconds later.)