admin
bcdd5b2058
R-601 said demo-hp was unreachable. The operator looked at the hub and said it was online. It was, and had been up four and a half weeks, reporting every few minutes. Both of my SSH routes pointed at stale addresses: `demo-hp` at a tailnet peer for a box that has no tailscale installed at all, and `demo-hp-lan` at 192.168.0.87 when the box is statically on .104 since a reprovision. The hub had carried the right address in every host report, and `ip neigh` on felhom-pve had .104 four lines above the .87 I quoted — I searched that output for the address I expected instead of reading it for the address that was there. Both ssh entries repointed and verified; nodes.md corrected, including that the tailnet route for this box does not exist. The hunt then found R-604, which is the real defect: demo-hp carried a per-customer floor override of 0.243.0 left over from the 2026-09-16 drill, so it had silently missed the raises to 0.253.0, 0.254.0, 0.257.0 and 0.259.0. `managed floor SERVED` fires once per change by design, so a box behind a static override is silent for ever and its silence is indistinguishable from a box that already logged. Cleared; demo-hp self-updated to 0.259.0 in under four minutes and its claim page now answers "Wrong or expired code" in English. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
3.5 KiB
Floor raise to 0.259.0 — and the box it nearly missed
2026-09-21, at the operator's request. Global controller floor 0.257.0 → 0.259.0,
min_agent 0.131.0 declared alongside it (R-472: the floor is above the vouched golden
0.258.0, so the declaration is what carries it).
What the hub logged
08:58:36 Global controller-version floor set to "0.259.0" (declared MinAgent "0.131.0")
08:58:38 managed floor SERVED for demo-felhom: floor 0.259.0, agent requirement "0.131.0"
from declared (golden 0.258.0)
And nothing for demo-hp — which reported one second later, at 08:58:39, and stayed on 0.258.0.
Why: a per-customer override nobody could see (R-604)
customer_configs.min_controller_version for demo-hp held 0.243.0 — a per-customer floor that
wins over the global one. It is a leftover from the 2026-09-16 drill, whose golden was 0.243.0.
R-343 measured on 2026-08-18 that all five rows were EMPTY and recorded that as a safety
property: "zero — all five customer_configs rows carry an empty min_controller_version, so
nothing hides behind a lower override." It stopped being true and nothing surfaced the change.
demo-hp had silently missed the raises to 0.253.0, 0.254.0, 0.257.0 and 0.259.0.
The silence is structural, not accidental. managed floor SERVED fires once per change
(h.floorNotes, hub/internal/api/handler.go:600) — deliberately, because a box reports every few
minutes. So a box whose override never changes is silent for ever, and that silence looks exactly
like the silence of a box that already logged its line. A session raises the floor, reads one SERVED
line, and reasonably concludes the fleet took it.
Cleared, and the result
Override cleared (rollback line: POST /customers/demo-hp/floor with
min_controller_version=0.243.0, min_agent=0.131.0):
09:11:02 Customer demo-hp controller-version floor override set to "" (declared MinAgent "")
09:11:05 managed floor SERVED for demo-hp: floor 0.259.0, agent requirement "0.131.0"
from declared (golden 0.258.0)
demo-hp then updated itself, in under four minutes, with nothing deployed by hand:
07:11:18 UTC [selfupdate] SetFloor: floor "" → "0.259.0"
07:11:18 UTC [selfupdate] maybeAutoUpdate: current 0.259.0 >= floor 0.259.0 — no action
07:11:22 UTC [offsite-apply] settle-gate: GO — at/above floor 0.259.0 (we are 0.259.0)
The floor delivered the FIX, not a version string. On demo-hp's claim page, reached at the
controller's own address with its Host header and the felhom_lang=en cookie:
http=200 alert-error">Wrong or expired code
That is the exact screen the 2026-09-20 English drill stopped on, now in English on a box nobody hand-deployed — which is the only thing that shows a floor raise did its job.
Fleet after
| customer | controller | last report | override |
|---|---|---|---|
| demo-felhom | 0.259.0 | 2026-09-21 07:05 | none |
| demo-hp | 0.259.0 | 2026-09-21 07:11 | none (cleared today) |
| tester-1 | 0.245.0 | 2026-09-17 — down | none |
| drill-r50 | 0.213.0 | 2026-08-12 — down | none |
| peti-felhom | 0.115.0 | 2026-07-15 — host row deleted, cannot receive a floor | none |
All five overrides are now empty, which restores the property R-343 recorded. The three boxes that are down take the floor unattended if they ever return — untested on this version.