v0.190.0 — the boot settle window, both gates on intent, and R-171
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
R-171 (a regression v0.189.0 introduced, CONFIRMED on hardware before any fix was written). Replacing isBootOrphan's container-count term with recorded intent made a drive-gate-stopped app read as a boot orphan: the gate stops apps with `compose down` (zero containers) and never touches desired_state, because it is not the customer. Observed on 9201 with the drive held unmounted — the sweep found and started it, burned both attempts, and handed it to the dead-app alarm. The write hazard did not materialise (the unbound mountpoint is host-root-owned and the guest is unprivileged) but that protection is accidental and untested. New consumer-side seam bootrecon.StartGate, fail-safe (cannot determine ⇒ do not start), wired in main.go. The rule is not new: the API's startGatedByMissingDrive already refuses this; the sweep bypassed it. R-157 mechanism A. The sweep looked once at T+5s, deriving candidates from a fleet docker was still restoring — three of six hard resets. Now a settle-then- sweep window: sample every 5s, settled after 3 identical samples, sweep ONCE at the end; ends on settled or a 50s budget, and the log says which. The budget is 50s because settle+budget+one retry must stay under the 90s dead-app grace — a test rejected 60s at 95s. A window that overruns emits a LATE RECOVERY warn rather than the grace being widened to hide it. Widening the window made two more holders reachable, so the one gate covers all three: an absent drive, a quiesce, and an in-flight app-data operation — reusing quiesce.SuppressedStacks() and a new read-only AppStopGuard.HeldStacks(). R-170. shouldRecreateOnBoot now reads desired_state with the identical three-way table; absent keeps the old hasContainers behaviour exactly. Its comment argued for the container count and was rewritten. presentStable is untouched. The two gates' agreement is pinned from both sides against one fixture table. 27/27 packages green; 6 red-proofs observed FAIL then restored.
This commit is contained in:
@@ -1,5 +1,82 @@
|
||||
## Changelog
|
||||
|
||||
### v0.190.0 — the boot-recovery story finished, and a regression v0.189.0 opened (2026-08-02, R-157 A · R-170 · R-171)
|
||||
|
||||
**R-171 — a regression introduced by v0.189.0, found by reading the diff and CONFIRMED on hardware
|
||||
before anything was written.** v0.189.0 correctly replaced `isBootOrphan`'s `len(Containers) > 0`
|
||||
term with the customer's recorded intent. But the **drive-absent gate** stops apps with `compose
|
||||
down` (zero containers) and never touches `desired_state`, because it is not the customer — so a
|
||||
gate-stopped app began reading as a boot orphan. Observed live on guest 9201 with the drive held
|
||||
unmounted:
|
||||
|
||||
```
|
||||
[gate] drive ABSENT /mnt/felhom-drives/hdd_1 — stopped+blocked 2 app(s): [calibre-web immich]
|
||||
[bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [calibre-web] — up to 2 attempt(s)
|
||||
[bootrecon] attempt 1/2: start "calibre-web" failed … attempt 2/2: … gave up
|
||||
[bootrecon] recovered=[] still down=[calibre-web] (the dead-app alarm now owns these)
|
||||
```
|
||||
|
||||
The **write** hazard did not materialise: compose failed with `mkdir /mnt/felhom-drives/hdd_1/
|
||||
userdata: permission denied`, because the unbound mountpoint is host-root-owned and the guest is
|
||||
unprivileged. **That protection is accidental** — no code chose it, no test pinned it, and it is one
|
||||
`chown` (or one privileged guest) away from gone. The harm that DID occur is real on every box: two
|
||||
wasted attempts and a **false dead-app alarm for an app the drive gate is deliberately holding**.
|
||||
|
||||
The fix is not a new rule. The **API's own start path already refuses this** —
|
||||
`startGatedByMissingDrive` returns a Hungarian refusal to the customer — and the sweep bypassed it by
|
||||
calling `Manager.StartStack` directly. New consumer-side seam `bootrecon.StartGate`, wired in
|
||||
`main.go`, gives the sweep the same question to ask. Fail-safe by contract: **cannot determine ⇒ do
|
||||
not start.** New `Manager.DriveLive` reuses the userdata belt's own `isMountPoint` seam so the two
|
||||
cannot drift. Held apps are reported as `HeldByDrive`, deliberately **not** as `StillDown` — that is
|
||||
the alarm's bucket and putting them there is the false alarm being removed.
|
||||
Evidence: `felhom.eu/documentation/audits/DIAG-bootrecon-drive-absent-2026-08-02.md`.
|
||||
|
||||
**R-157 mechanism A — the sweep that looked once.** `runBootReconcile` waited 5 s and swept exactly
|
||||
once, deriving its candidate set from a fleet docker was still restoring; measured failing on **three
|
||||
of six hard resets**. It is now a **settle-then-sweep window**: sample the fleet (name, state,
|
||||
container count) every **5 s**, call it settled after **3 identical samples**, and sweep **once**, at
|
||||
the end, on a settled fleet. The window terminates on whichever comes first — settled, or a **50 s
|
||||
budget** — and the log says which, because "settled and found nothing" and "ran out of time still
|
||||
churning" are different facts about the box.
|
||||
|
||||
**The budget is 50 s and not 60 s because a test said so.** `bootReconcileSettle` (5 s) + budget +
|
||||
one `DefaultRetryDelay` (30 s) must stay under `deadAppBootGrace` (90 s) so a successful recovery is
|
||||
SILENT. 60 s was the first choice; `TestBootWindow_CommonCaseFitsInsideTheDeadAppGrace` rejected it
|
||||
at 95 s. Extending the grace to fit was rejected outright (§8.3) — that hides a late recovery instead
|
||||
of reporting it. A window that genuinely overruns now emits a **`LATE RECOVERY` WARN naming the
|
||||
apps**, so a stale alarm never stands without counter-evidence.
|
||||
|
||||
**Sampling is read-only and there is still exactly ONE sweep.** Sweeping per sample was rejected: the
|
||||
sweep's own `StartStack` changes the fleet, so it would never observe a settled one. The per-app
|
||||
attempt bound is untouched — this widens a bounded window, it does not remove the bound.
|
||||
|
||||
**Widening the window made two more holders reachable (§8.2), so the gate covers all three.** The old
|
||||
T+5 s sweep never overlapped a **quiesce** (starting an app mid-backup defeats the point of
|
||||
quiescing) or a **running app-data operation** (restarting an app under its own tar). Both are now
|
||||
refused through the same seam, reusing `quiesce.SuppressedStacks()` and a new read-only
|
||||
`AppStopGuard.HeldStacks()` rather than second implementations.
|
||||
|
||||
**R-170 — the second boot gate stops guessing.** `shouldRecreateOnBoot` still ended in
|
||||
`&& hasContainers`, so the two boot gates disagreed about the same question. It now reads
|
||||
`desired_state` with the identical three-way table: `stopped` → never; `running` → recreate whatever
|
||||
the container count; **absent → exactly the pre-v0.190.0 `hasContainers` behaviour**. Its comment
|
||||
argued at length *for* the container count and has been rewritten — a correct implementation under a
|
||||
comment arguing the opposite is worse than either alone. **`presentStable` is untouched and still
|
||||
load-bearing**: an app whose drive is absent is never recreated here, which is the very term the boot
|
||||
sweep was missing. The agreement between the gates is pinned from **both sides** against an identical
|
||||
fixture table (`TestBothBootGatesAgreeOnIntent` / `TestShouldRecreateOnBoot_AgreesWithBootrecon`),
|
||||
because the two cannot be called from one package without an import cycle.
|
||||
|
||||
**Tests: +25 across 3 packages (27/27 packages green).** Timing is tested by shrinking the window
|
||||
constants, never by sleeping. Red-proofs, each observed FAIL then restored: A (restore the
|
||||
single-sweep shape), B (remove the budget → the test **hangs**, the unbounded shape), C (drop the
|
||||
`desired_state: stopped` branch → the customer-stopped app is started **twice** by the widened
|
||||
window), D (restore `&& hasContainers`), G (remove the start-gate check → the drive-absent app is
|
||||
started), H (comment out `SetDriveGate` → fails while the string is **still present**, which is what
|
||||
the AST walk is for).
|
||||
|
||||
No hub change, no agent coupling, no user-visible string, no backup/restore/catalogue change.
|
||||
|
||||
### v0.189.0 — the box stops guessing what the customer wanted (2026-08-02, R-166 / decision D-b)
|
||||
|
||||
**The defect.** When an app was not running, the controller had to work out *why*, and it worked it
|
||||
|
||||
Reference in New Issue
Block a user