v0.190.0 — the boot settle window, both gates on intent, and R-171
gates / gates (push) Successful in 8s

R-171 (a regression v0.189.0 introduced, CONFIRMED on hardware before any fix
was written). Replacing isBootOrphan's container-count term with recorded intent
made a drive-gate-stopped app read as a boot orphan: the gate stops apps with
`compose down` (zero containers) and never touches desired_state, because it is
not the customer. Observed on 9201 with the drive held unmounted — the sweep
found and started it, burned both attempts, and handed it to the dead-app alarm.
The write hazard did not materialise (the unbound mountpoint is host-root-owned
and the guest is unprivileged) but that protection is accidental and untested.
New consumer-side seam bootrecon.StartGate, fail-safe (cannot determine ⇒ do not
start), wired in main.go. The rule is not new: the API's startGatedByMissingDrive
already refuses this; the sweep bypassed it.

R-157 mechanism A. The sweep looked once at T+5s, deriving candidates from a
fleet docker was still restoring — three of six hard resets. Now a settle-then-
sweep window: sample every 5s, settled after 3 identical samples, sweep ONCE at
the end; ends on settled or a 50s budget, and the log says which. The budget is
50s because settle+budget+one retry must stay under the 90s dead-app grace — a
test rejected 60s at 95s. A window that overruns emits a LATE RECOVERY warn
rather than the grace being widened to hide it.

Widening the window made two more holders reachable, so the one gate covers all
three: an absent drive, a quiesce, and an in-flight app-data operation — reusing
quiesce.SuppressedStacks() and a new read-only AppStopGuard.HeldStacks().

R-170. shouldRecreateOnBoot now reads desired_state with the identical three-way
table; absent keeps the old hasContainers behaviour exactly. Its comment argued
for the container count and was rewritten. presentStable is untouched. The two
gates' agreement is pinned from both sides against one fixture table.

27/27 packages green; 6 red-proofs observed FAIL then restored.
This commit is contained in:
2026-08-02 19:56:20 +02:00
parent 3446609420
commit 582135f861
13 changed files with 1272 additions and 43 deletions
+31
View File
@@ -1902,6 +1902,37 @@ upgrade. Where intent is unknown the sweep falls back to the old inference rathe
answer, and a running-only startup **backfill** converges the unambiguous cases (deployed and
observed up) without waiting for a button press. `stopped` is never backfilled from any signal.
**The sweep no longer looks only once (R-157 mechanism A, v0.190.0).** It used to sample the fleet at
T+5 s and return; at that moment docker is still restoring containers after a hard reset, so an app
that had not yet settled into a down state was never a candidate — **measured failing on three of six
hard resets**. It is now a **settle-then-sweep window**: the fleet (name, state, container count) is
sampled every 5 s, called settled after 3 identical samples, and swept **once**, at the end, on a
settled fleet. The window ends on whichever comes first — settled, or a **50 s budget** — and the log
says which. Sampling is read-only and there is still exactly one sweep with its per-app attempt bound
intact: this widens a bounded window, it does not remove the bound. `settle (5 s) + budget (50 s) +
one 30 s retry = 85 s` stays inside the 90 s `deadAppBootGrace`, which is what keeps a successful
recovery silent; a window that genuinely overruns emits a `LATE RECOVERY` WARN naming the apps rather
than the grace being widened to hide it.
**The sweep asks before it starts (R-171, v0.190.0).** Three things legitimately hold an app down,
and starting it would be the wrong repair for all three: its **data drive is absent** (compose would
create the bind sources on the guest rootfs — the hazard the drive gate exists to prevent), a
**quiesce** is holding it for a whole-guest backup, or an **app-data operation** (volume dump, offsite
restore, `.fab` export) is holding it. All three are refused through one seam, reusing the signals
their owners already publish. **Fail-safe: a drive whose liveness cannot be determined is treated as
absent.** Held apps are reported separately from `StillDown` — they are not a fault the sweep failed
to fix, and reporting them as one is a false alarm. This closed a regression v0.189.0 introduced:
before it, a drive-gate-stopped app read as `running` + zero containers, so the sweep started it,
burned both attempts and handed it to the dead-app alarm.
**Both boot gates read intent (R-170, v0.190.0).** There are two: the R-52 sweep above, and the
drive-backed **boot recreate gate** (`shouldRecreateOnBoot`, `internal/web/intermediary.go`), which
re-creates a drive-backed app onto its re-propagated drive after a guest reboot. Until v0.190.0 the
second still ended in `&& hasContainers`, so the two disagreed about the same question. It now uses
the identical three-way table — `stopped` → never, `running` → recreate whatever the container count,
absent → exactly the old `hasContainers` behaviour. Its drive-presence term is untouched and still
load-bearing: an app whose drive is absent is never recreated there either.
**Desired state — who owns it (R-166, v0.189.0).** `app.yaml` gains `desired_state`, a tri-state
`""` / `running` / `stopped`. It is written by **the customer's own action and nothing else**: the
`/api/stacks/{name}/{action}` switch (`start`/`restart`/`update` → running, `stop` → stopped),