v0.189.0 — desired state + the app-stop crash marker (R-166 / D-b)
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
The box stops inferring the customer's intent from a container count and reads
what they actually asked for.
Part 1 — desired state. AppConfig gains a tri-state `desired_state`
(""/running/stopped), written ONLY by the customer's own action: the API action
switch, DeployStack, UpdateOptionalConfig's redeploy branch, and the .fab
import. Intent is written BEFORE the act and a failed write REFUSES the act.
StartStack/StopStack are deliberately not writers — 14 callers, only 2 are the
customer. bootrecon.isBootOrphan now reads intent instead of len(Containers)>0,
which closes R-157 mechanism B (a power cut or interrupted deploy left an app
with zero containers, read as a deliberate stop, and stranded silently).
ABSENT MEANS UNKNOWN, NEVER "running": every pre-v0.189.0 app.yaml reads absent,
so the legacy fallback is byte-identical to the old rule. A running-only startup
backfill converges the unambiguous cases; `stopped` is never inferred.
Part 2 — backup.AppStopGuard, a persisted marker over every stop→work→start
window (volume dump, offbox reconstitute, .fab export). Its own file, never
quiesce's. Written before the stop, cleared only after a restart that succeeded,
kept when one fails. Recover() completes before the boot reconciler is launched
and returns its outcome, which main.go reports on the existing backup_failed
event once the notifier exists. A defer is not the mechanism — a SIGKILL runs
none (Campaign 8 fault 10).
Also: SaveAppConfig rebuilt AppConfig field-by-field (the R-100 shape) and would
have dropped desired_state on every save across nine call sites. Replaced with
copy-and-overlay. Measured: app.yaml does not round-trip unknown YAML keys.
No hub change, no agent coupling, no user-visible string. 27/27 packages green;
7 red-proofs observed FAIL then restored.
This commit is contained in:
@@ -11,10 +11,15 @@
|
||||
// - **Bounded, never a loop.** At most `attempts` tries, `retryDelay` apart, then it stops and the
|
||||
// alarm owns the problem. A restart loop would paper over a genuinely broken app forever and
|
||||
// hammer docker while doing it.
|
||||
// - **A user's Stop survives a reboot.** The UI's Stop is `docker compose down`, which REMOVES the
|
||||
// containers; a boot interruption leaves them behind as Exited. So "has containers on disk that
|
||||
// are down" is the boot-orphan signature, and a stack with ZERO containers is deliberately never
|
||||
// touched. This distinction is the whole safety argument — see TestReconcile_UserStoppedAppIsNeverStarted.
|
||||
// - **A user's Stop survives a reboot.** This is still the whole safety argument; only the way it
|
||||
// is established changed. Until v0.189.0 it was inferred — the UI's Stop is `docker compose
|
||||
// down`, which REMOVES containers, so "zero containers" was read as "the customer stopped it"
|
||||
// and left alone. Since v0.189.0 (R-166) the customer's intent is RECORDED in app.yaml and read
|
||||
// directly, because the inference could not distinguish a deliberate Stop from a power cut or an
|
||||
// interrupted backup, and silently stranded both. An app.yaml with no recorded intent — every
|
||||
// app on every box predating the field — keeps the old inference exactly. See isBootOrphan,
|
||||
// TestReconcile_UserStoppedAppIsNeverStarted and
|
||||
// TestReconcile_LegacyNoDesiredState_BehavesExactlyAsBefore.
|
||||
//
|
||||
// It runs inside the notifier's boot grace (cmd/controller/main.go `deadAppBootGrace`), so a
|
||||
// successful recovery never fires an alert and a failed one alerts honestly once the grace expires.
|
||||
@@ -87,17 +92,66 @@ func sleepCtx(ctx context.Context, d time.Duration) {
|
||||
// isBootOrphan reports whether a stack is an app the boot left behind.
|
||||
//
|
||||
// The gate, term by term:
|
||||
// - Deployed — an app the customer asked to have running.
|
||||
// - Deployed — an app that is installed. NOTE: `Deployed` means INSTALLED, not "wanted running";
|
||||
// the two were conflated until v0.189.0 and that conflation is what the desired-state term below
|
||||
// repairs.
|
||||
// - not Protected — traefik/cloudflared/felhom-controller have their own supervision; this must
|
||||
// never race the base-stack self-heal.
|
||||
// - not Deploying — mid-deploy is not a fault.
|
||||
// - has containers — the D-case guard: a UI Stop removes them, and a deliberate stop must survive
|
||||
// a reboot.
|
||||
// - desired state — see below. REPLACES the old container-count term.
|
||||
// - IsDownState — stopped/exited/degraded (R-51 included: a boot that half-started a stack is the
|
||||
// same interrupted-boot shape).
|
||||
//
|
||||
// ── WHY INTENT REPLACED THE CONTAINER COUNT (R-166, closing R-157 mechanism B) ────────────────────
|
||||
//
|
||||
// This gate used to end in `len(s.Containers) > 0`, and its comment called that "the D-case guard":
|
||||
// a UI Stop is `compose down`, which REMOVES containers, so zero containers was read as "the
|
||||
// customer stopped this" and left alone. The safety goal was right and still holds. The SIGNAL was
|
||||
// wrong, because zero containers has at least three causes and the count cannot tell them apart:
|
||||
//
|
||||
// a deliberate Stop → must stay down
|
||||
// a power cut mid-compose, or an interrupted deploy → must come back
|
||||
// a backup that stopped the app and died before restarting it → must come back
|
||||
//
|
||||
// Two of those three were silently unrecoverable: the app simply stayed gone until a human noticed.
|
||||
// The count was never capable of separating them, so the fix is not a better inference — it is to
|
||||
// stop inferring and read what the customer actually asked for, which app.yaml now records.
|
||||
//
|
||||
// ── WHAT ABSENT STILL MEANS, AND WHY THE OLD BEHAVIOUR IS KEPT ────────────────────────────────────
|
||||
//
|
||||
// DesiredStateUnknown falls back to the ORIGINAL container-count rule, byte-for-byte. This is the
|
||||
// single most important line in the change. Every app.yaml on every existing box predates the field,
|
||||
// so absent is what the whole fleet reads on upgrade; treating absent as "running" would start, on
|
||||
// the first boot after the upgrade, every app its owner had deliberately stopped. The fallback is
|
||||
// what makes this feature inert for an app nobody has pressed a button on since — see
|
||||
// TestReconcile_LegacyNoDesiredState_BehavesExactlyAsBefore and its red-proof.
|
||||
//
|
||||
// The full decision table (§8.1):
|
||||
//
|
||||
// desired containers state → result
|
||||
// stopped any any → never an orphan (the customer said so)
|
||||
// running 0 — → ORPHAN ← the R-157 case, invisible before v0.189.0
|
||||
// running >0 IsDownState → ORPHAN (unchanged)
|
||||
// running >0 up → not an orphan
|
||||
// absent 0 — → not an orphan (exactly the pre-v0.189.0 behaviour)
|
||||
// absent >0 IsDownState → ORPHAN (exactly the pre-v0.189.0 behaviour)
|
||||
func isBootOrphan(s stacks.Stack) bool {
|
||||
return s.Deployed && !s.Protected && !s.Deploying &&
|
||||
len(s.Containers) > 0 && stacks.IsDownState(s.State)
|
||||
if !s.Deployed || s.Protected || s.Deploying {
|
||||
return false
|
||||
}
|
||||
switch stacks.DesiredStateOf(s) {
|
||||
case stacks.DesiredStateStopped:
|
||||
// The customer pressed Stop. No observation may overturn that — not a missing container, not
|
||||
// a down state, not a reboot. Nothing else in this package starts an app.
|
||||
return false
|
||||
case stacks.DesiredStateRunning:
|
||||
// Wanted running. ANY way of not being up is a fault to repair, including having no
|
||||
// containers at all — which is the case the old count term structurally could not see.
|
||||
return len(s.Containers) == 0 || stacks.IsDownState(s.State)
|
||||
default:
|
||||
// DesiredStateUnknown — legacy. Keep the pre-R-166 rule exactly.
|
||||
return len(s.Containers) > 0 && stacks.IsDownState(s.State)
|
||||
}
|
||||
}
|
||||
|
||||
// Run performs the sweep once and returns what happened. It is safe to call with no boot orphans
|
||||
|
||||
Reference in New Issue
Block a user