v0.190.0 — the boot settle window, both gates on intent, and R-171
gates / gates (push) Successful in 8s

R-171 (a regression v0.189.0 introduced, CONFIRMED on hardware before any fix
was written). Replacing isBootOrphan's container-count term with recorded intent
made a drive-gate-stopped app read as a boot orphan: the gate stops apps with
`compose down` (zero containers) and never touches desired_state, because it is
not the customer. Observed on 9201 with the drive held unmounted — the sweep
found and started it, burned both attempts, and handed it to the dead-app alarm.
The write hazard did not materialise (the unbound mountpoint is host-root-owned
and the guest is unprivileged) but that protection is accidental and untested.
New consumer-side seam bootrecon.StartGate, fail-safe (cannot determine ⇒ do not
start), wired in main.go. The rule is not new: the API's startGatedByMissingDrive
already refuses this; the sweep bypassed it.

R-157 mechanism A. The sweep looked once at T+5s, deriving candidates from a
fleet docker was still restoring — three of six hard resets. Now a settle-then-
sweep window: sample every 5s, settled after 3 identical samples, sweep ONCE at
the end; ends on settled or a 50s budget, and the log says which. The budget is
50s because settle+budget+one retry must stay under the 90s dead-app grace — a
test rejected 60s at 95s. A window that overruns emits a LATE RECOVERY warn
rather than the grace being widened to hide it.

Widening the window made two more holders reachable, so the one gate covers all
three: an absent drive, a quiesce, and an in-flight app-data operation — reusing
quiesce.SuppressedStacks() and a new read-only AppStopGuard.HeldStacks().

R-170. shouldRecreateOnBoot now reads desired_state with the identical three-way
table; absent keeps the old hasContainers behaviour exactly. Its comment argued
for the container count and was rewritten. presentStable is untouched. The two
gates' agreement is pinned from both sides against one fixture table.

27/27 packages green; 6 red-proofs observed FAIL then restored.
This commit is contained in:
2026-08-02 19:56:20 +02:00
parent 3446609420
commit 582135f861
13 changed files with 1272 additions and 43 deletions
+77
View File
@@ -1,5 +1,82 @@
## Changelog
### v0.190.0 — the boot-recovery story finished, and a regression v0.189.0 opened (2026-08-02, R-157 A · R-170 · R-171)
**R-171 — a regression introduced by v0.189.0, found by reading the diff and CONFIRMED on hardware
before anything was written.** v0.189.0 correctly replaced `isBootOrphan`'s `len(Containers) > 0`
term with the customer's recorded intent. But the **drive-absent gate** stops apps with `compose
down` (zero containers) and never touches `desired_state`, because it is not the customer — so a
gate-stopped app began reading as a boot orphan. Observed live on guest 9201 with the drive held
unmounted:
```
[gate] drive ABSENT /mnt/felhom-drives/hdd_1 — stopped+blocked 2 app(s): [calibre-web immich]
[bootrecon] Boot reconciliation: 1 boot-orphaned app(s) found: [calibre-web] — up to 2 attempt(s)
[bootrecon] attempt 1/2: start "calibre-web" failed … attempt 2/2: … gave up
[bootrecon] recovered=[] still down=[calibre-web] (the dead-app alarm now owns these)
```
The **write** hazard did not materialise: compose failed with `mkdir /mnt/felhom-drives/hdd_1/
userdata: permission denied`, because the unbound mountpoint is host-root-owned and the guest is
unprivileged. **That protection is accidental** — no code chose it, no test pinned it, and it is one
`chown` (or one privileged guest) away from gone. The harm that DID occur is real on every box: two
wasted attempts and a **false dead-app alarm for an app the drive gate is deliberately holding**.
The fix is not a new rule. The **API's own start path already refuses this**
`startGatedByMissingDrive` returns a Hungarian refusal to the customer — and the sweep bypassed it by
calling `Manager.StartStack` directly. New consumer-side seam `bootrecon.StartGate`, wired in
`main.go`, gives the sweep the same question to ask. Fail-safe by contract: **cannot determine ⇒ do
not start.** New `Manager.DriveLive` reuses the userdata belt's own `isMountPoint` seam so the two
cannot drift. Held apps are reported as `HeldByDrive`, deliberately **not** as `StillDown` — that is
the alarm's bucket and putting them there is the false alarm being removed.
Evidence: `felhom.eu/documentation/audits/DIAG-bootrecon-drive-absent-2026-08-02.md`.
**R-157 mechanism A — the sweep that looked once.** `runBootReconcile` waited 5 s and swept exactly
once, deriving its candidate set from a fleet docker was still restoring; measured failing on **three
of six hard resets**. It is now a **settle-then-sweep window**: sample the fleet (name, state,
container count) every **5 s**, call it settled after **3 identical samples**, and sweep **once**, at
the end, on a settled fleet. The window terminates on whichever comes first — settled, or a **50 s
budget** — and the log says which, because "settled and found nothing" and "ran out of time still
churning" are different facts about the box.
**The budget is 50 s and not 60 s because a test said so.** `bootReconcileSettle` (5 s) + budget +
one `DefaultRetryDelay` (30 s) must stay under `deadAppBootGrace` (90 s) so a successful recovery is
SILENT. 60 s was the first choice; `TestBootWindow_CommonCaseFitsInsideTheDeadAppGrace` rejected it
at 95 s. Extending the grace to fit was rejected outright (§8.3) — that hides a late recovery instead
of reporting it. A window that genuinely overruns now emits a **`LATE RECOVERY` WARN naming the
apps**, so a stale alarm never stands without counter-evidence.
**Sampling is read-only and there is still exactly ONE sweep.** Sweeping per sample was rejected: the
sweep's own `StartStack` changes the fleet, so it would never observe a settled one. The per-app
attempt bound is untouched — this widens a bounded window, it does not remove the bound.
**Widening the window made two more holders reachable (§8.2), so the gate covers all three.** The old
T+5 s sweep never overlapped a **quiesce** (starting an app mid-backup defeats the point of
quiescing) or a **running app-data operation** (restarting an app under its own tar). Both are now
refused through the same seam, reusing `quiesce.SuppressedStacks()` and a new read-only
`AppStopGuard.HeldStacks()` rather than second implementations.
**R-170 — the second boot gate stops guessing.** `shouldRecreateOnBoot` still ended in
`&& hasContainers`, so the two boot gates disagreed about the same question. It now reads
`desired_state` with the identical three-way table: `stopped` → never; `running` → recreate whatever
the container count; **absent → exactly the pre-v0.190.0 `hasContainers` behaviour**. Its comment
argued at length *for* the container count and has been rewritten — a correct implementation under a
comment arguing the opposite is worse than either alone. **`presentStable` is untouched and still
load-bearing**: an app whose drive is absent is never recreated here, which is the very term the boot
sweep was missing. The agreement between the gates is pinned from **both sides** against an identical
fixture table (`TestBothBootGatesAgreeOnIntent` / `TestShouldRecreateOnBoot_AgreesWithBootrecon`),
because the two cannot be called from one package without an import cycle.
**Tests: +25 across 3 packages (27/27 packages green).** Timing is tested by shrinking the window
constants, never by sleeping. Red-proofs, each observed FAIL then restored: A (restore the
single-sweep shape), B (remove the budget → the test **hangs**, the unbounded shape), C (drop the
`desired_state: stopped` branch → the customer-stopped app is started **twice** by the widened
window), D (restore `&& hasContainers`), G (remove the start-gate check → the drive-absent app is
started), H (comment out `SetDriveGate` → fails while the string is **still present**, which is what
the AST walk is for).
No hub change, no agent coupling, no user-visible string, no backup/restore/catalogue change.
### v0.189.0 — the box stops guessing what the customer wanted (2026-08-02, R-166 / decision D-b)
**The defect.** When an app was not running, the controller had to work out *why*, and it worked it