admin
bcdd5b2058
gates / gates (push) Successful in 28s
R-601 said demo-hp was unreachable. The operator looked at the hub and said it was online. It was, and had been up four and a half weeks, reporting every few minutes. Both of my SSH routes pointed at stale addresses: `demo-hp` at a tailnet peer for a box that has no tailscale installed at all, and `demo-hp-lan` at 192.168.0.87 when the box is statically on .104 since a reprovision. The hub had carried the right address in every host report, and `ip neigh` on felhom-pve had .104 four lines above the .87 I quoted — I searched that output for the address I expected instead of reading it for the address that was there. Both ssh entries repointed and verified; nodes.md corrected, including that the tailnet route for this box does not exist. The hunt then found R-604, which is the real defect: demo-hp carried a per-customer floor override of 0.243.0 left over from the 2026-09-16 drill, so it had silently missed the raises to 0.253.0, 0.254.0, 0.257.0 and 0.259.0. `managed floor SERVED` fires once per change by design, so a box behind a static override is silent for ever and its silence is indistinguishable from a box that already logged. Cleared; demo-hp self-updated to 0.259.0 in under four minutes and its claim page now answers "Wrong or expired code" in English. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
74 lines
3.5 KiB
Markdown
74 lines
3.5 KiB
Markdown
# Floor raise to 0.259.0 — and the box it nearly missed
|
|
|
|
**2026-09-21, at the operator's request.** Global controller floor `0.257.0` → **`0.259.0`**,
|
|
`min_agent` **`0.131.0`** declared alongside it (R-472: the floor is above the vouched golden
|
|
0.258.0, so the declaration is what carries it).
|
|
|
|
## What the hub logged
|
|
|
|
```
|
|
08:58:36 Global controller-version floor set to "0.259.0" (declared MinAgent "0.131.0")
|
|
08:58:38 managed floor SERVED for demo-felhom: floor 0.259.0, agent requirement "0.131.0"
|
|
from declared (golden 0.258.0)
|
|
```
|
|
|
|
**And nothing for demo-hp** — which reported one second later, at 08:58:39, and stayed on 0.258.0.
|
|
|
|
## Why: a per-customer override nobody could see (R-604)
|
|
|
|
`customer_configs.min_controller_version` for demo-hp held **`0.243.0`** — a per-customer floor that
|
|
wins over the global one. It is a leftover from the **2026-09-16 drill**, whose golden was 0.243.0.
|
|
|
|
**R-343 measured on 2026-08-18 that all five rows were EMPTY** and recorded that as a safety
|
|
property: *"zero — all five `customer_configs` rows carry an empty `min_controller_version`, so
|
|
nothing hides behind a lower override."* It stopped being true and nothing surfaced the change.
|
|
demo-hp had silently missed the raises to **0.253.0, 0.254.0, 0.257.0 and 0.259.0**.
|
|
|
|
**The silence is structural, not accidental.** `managed floor SERVED` fires once per *change*
|
|
(`h.floorNotes`, `hub/internal/api/handler.go:600`) — deliberately, because a box reports every few
|
|
minutes. So a box whose override never changes is silent **for ever**, and that silence looks exactly
|
|
like the silence of a box that already logged its line. A session raises the floor, reads one SERVED
|
|
line, and reasonably concludes the fleet took it.
|
|
|
|
## Cleared, and the result
|
|
|
|
Override cleared (**rollback line:** POST `/customers/demo-hp/floor` with
|
|
`min_controller_version=0.243.0`, `min_agent=0.131.0`):
|
|
|
|
```
|
|
09:11:02 Customer demo-hp controller-version floor override set to "" (declared MinAgent "")
|
|
09:11:05 managed floor SERVED for demo-hp: floor 0.259.0, agent requirement "0.131.0"
|
|
from declared (golden 0.258.0)
|
|
```
|
|
|
|
demo-hp then updated **itself**, in under four minutes, with nothing deployed by hand:
|
|
|
|
```
|
|
07:11:18 UTC [selfupdate] SetFloor: floor "" → "0.259.0"
|
|
07:11:18 UTC [selfupdate] maybeAutoUpdate: current 0.259.0 >= floor 0.259.0 — no action
|
|
07:11:22 UTC [offsite-apply] settle-gate: GO — at/above floor 0.259.0 (we are 0.259.0)
|
|
```
|
|
|
|
**The floor delivered the FIX, not a version string.** On demo-hp's claim page, reached at the
|
|
controller's own address with its `Host` header and the `felhom_lang=en` cookie:
|
|
|
|
```
|
|
http=200 alert-error">Wrong or expired code
|
|
```
|
|
|
|
That is the exact screen the 2026-09-20 English drill stopped on, now in English **on a box nobody
|
|
hand-deployed** — which is the only thing that shows a floor raise did its job.
|
|
|
|
## Fleet after
|
|
|
|
| customer | controller | last report | override |
|
|
|---|---|---|---|
|
|
| demo-felhom | **0.259.0** | 2026-09-21 07:05 | none |
|
|
| demo-hp | **0.259.0** | 2026-09-21 07:11 | none (cleared today) |
|
|
| tester-1 | 0.245.0 | 2026-09-17 — **down** | none |
|
|
| drill-r50 | 0.213.0 | 2026-08-12 — **down** | none |
|
|
| peti-felhom | 0.115.0 | 2026-07-15 — host row deleted, cannot receive a floor | none |
|
|
|
|
**All five overrides are now empty**, which restores the property R-343 recorded. The three boxes
|
|
that are down take the floor unattended if they ever return — untested on this version.
|