v0.190.0 — the boot settle window, both gates on intent, and R-171
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
R-171 (a regression v0.189.0 introduced, CONFIRMED on hardware before any fix was written). Replacing isBootOrphan's container-count term with recorded intent made a drive-gate-stopped app read as a boot orphan: the gate stops apps with `compose down` (zero containers) and never touches desired_state, because it is not the customer. Observed on 9201 with the drive held unmounted — the sweep found and started it, burned both attempts, and handed it to the dead-app alarm. The write hazard did not materialise (the unbound mountpoint is host-root-owned and the guest is unprivileged) but that protection is accidental and untested. New consumer-side seam bootrecon.StartGate, fail-safe (cannot determine ⇒ do not start), wired in main.go. The rule is not new: the API's startGatedByMissingDrive already refuses this; the sweep bypassed it. R-157 mechanism A. The sweep looked once at T+5s, deriving candidates from a fleet docker was still restoring — three of six hard resets. Now a settle-then- sweep window: sample every 5s, settled after 3 identical samples, sweep ONCE at the end; ends on settled or a 50s budget, and the log says which. The budget is 50s because settle+budget+one retry must stay under the 90s dead-app grace — a test rejected 60s at 95s. A window that overruns emits a LATE RECOVERY warn rather than the grace being widened to hide it. Widening the window made two more holders reachable, so the one gate covers all three: an absent drive, a quiesce, and an in-flight app-data operation — reusing quiesce.SuppressedStacks() and a new read-only AppStopGuard.HeldStacks(). R-170. shouldRecreateOnBoot now reads desired_state with the identical three-way table; absent keeps the old hasContainers behaviour exactly. Its comment argued for the container count and was rewritten. presentStable is untouched. The two gates' agreement is pinned from both sides against one fixture table. 27/27 packages green; 6 red-proofs observed FAIL then restored.
This commit is contained in:
@@ -1902,6 +1902,37 @@ upgrade. Where intent is unknown the sweep falls back to the old inference rathe
|
||||
answer, and a running-only startup **backfill** converges the unambiguous cases (deployed and
|
||||
observed up) without waiting for a button press. `stopped` is never backfilled from any signal.
|
||||
|
||||
**The sweep no longer looks only once (R-157 mechanism A, v0.190.0).** It used to sample the fleet at
|
||||
T+5 s and return; at that moment docker is still restoring containers after a hard reset, so an app
|
||||
that had not yet settled into a down state was never a candidate — **measured failing on three of six
|
||||
hard resets**. It is now a **settle-then-sweep window**: the fleet (name, state, container count) is
|
||||
sampled every 5 s, called settled after 3 identical samples, and swept **once**, at the end, on a
|
||||
settled fleet. The window ends on whichever comes first — settled, or a **50 s budget** — and the log
|
||||
says which. Sampling is read-only and there is still exactly one sweep with its per-app attempt bound
|
||||
intact: this widens a bounded window, it does not remove the bound. `settle (5 s) + budget (50 s) +
|
||||
one 30 s retry = 85 s` stays inside the 90 s `deadAppBootGrace`, which is what keeps a successful
|
||||
recovery silent; a window that genuinely overruns emits a `LATE RECOVERY` WARN naming the apps rather
|
||||
than the grace being widened to hide it.
|
||||
|
||||
**The sweep asks before it starts (R-171, v0.190.0).** Three things legitimately hold an app down,
|
||||
and starting it would be the wrong repair for all three: its **data drive is absent** (compose would
|
||||
create the bind sources on the guest rootfs — the hazard the drive gate exists to prevent), a
|
||||
**quiesce** is holding it for a whole-guest backup, or an **app-data operation** (volume dump, offsite
|
||||
restore, `.fab` export) is holding it. All three are refused through one seam, reusing the signals
|
||||
their owners already publish. **Fail-safe: a drive whose liveness cannot be determined is treated as
|
||||
absent.** Held apps are reported separately from `StillDown` — they are not a fault the sweep failed
|
||||
to fix, and reporting them as one is a false alarm. This closed a regression v0.189.0 introduced:
|
||||
before it, a drive-gate-stopped app read as `running` + zero containers, so the sweep started it,
|
||||
burned both attempts and handed it to the dead-app alarm.
|
||||
|
||||
**Both boot gates read intent (R-170, v0.190.0).** There are two: the R-52 sweep above, and the
|
||||
drive-backed **boot recreate gate** (`shouldRecreateOnBoot`, `internal/web/intermediary.go`), which
|
||||
re-creates a drive-backed app onto its re-propagated drive after a guest reboot. Until v0.190.0 the
|
||||
second still ended in `&& hasContainers`, so the two disagreed about the same question. It now uses
|
||||
the identical three-way table — `stopped` → never, `running` → recreate whatever the container count,
|
||||
absent → exactly the old `hasContainers` behaviour. Its drive-presence term is untouched and still
|
||||
load-bearing: an app whose drive is absent is never recreated there either.
|
||||
|
||||
**Desired state — who owns it (R-166, v0.189.0).** `app.yaml` gains `desired_state`, a tri-state
|
||||
`""` / `running` / `stopped`. It is written by **the customer's own action and nothing else**: the
|
||||
`/api/stacks/{name}/{action}` switch (`start`/`restart`/`update` → running, `stop` → stopped),
|
||||
|
||||
Reference in New Issue
Block a user