v0.189.0 — desired state + the app-stop crash marker (R-166 / D-b)
gates / gates (push) Successful in 8s

The box stops inferring the customer's intent from a container count and reads
what they actually asked for.

Part 1 — desired state. AppConfig gains a tri-state `desired_state`
(""/running/stopped), written ONLY by the customer's own action: the API action
switch, DeployStack, UpdateOptionalConfig's redeploy branch, and the .fab
import. Intent is written BEFORE the act and a failed write REFUSES the act.
StartStack/StopStack are deliberately not writers — 14 callers, only 2 are the
customer. bootrecon.isBootOrphan now reads intent instead of len(Containers)>0,
which closes R-157 mechanism B (a power cut or interrupted deploy left an app
with zero containers, read as a deliberate stop, and stranded silently).

ABSENT MEANS UNKNOWN, NEVER "running": every pre-v0.189.0 app.yaml reads absent,
so the legacy fallback is byte-identical to the old rule. A running-only startup
backfill converges the unambiguous cases; `stopped` is never inferred.

Part 2 — backup.AppStopGuard, a persisted marker over every stop→work→start
window (volume dump, offbox reconstitute, .fab export). Its own file, never
quiesce's. Written before the stop, cleared only after a restart that succeeded,
kept when one fails. Recover() completes before the boot reconciler is launched
and returns its outcome, which main.go reports on the existing backup_failed
event once the notifier exists. A defer is not the mechanism — a SIGKILL runs
none (Campaign 8 fault 10).

Also: SaveAppConfig rebuilt AppConfig field-by-field (the R-100 shape) and would
have dropped desired_state on every save across nine call sites. Replaced with
copy-and-overlay. Measured: app.yaml does not round-trip unknown YAML keys.

No hub change, no agent coupling, no user-visible string. 27/27 packages green;
7 red-proofs observed FAIL then restored.
This commit is contained in:
2026-08-02 18:40:17 +02:00
parent e7c44c0e0f
commit dbcb306fcf
17 changed files with 2211 additions and 33 deletions
+63 -9
View File
@@ -11,10 +11,15 @@
// - **Bounded, never a loop.** At most `attempts` tries, `retryDelay` apart, then it stops and the
// alarm owns the problem. A restart loop would paper over a genuinely broken app forever and
// hammer docker while doing it.
// - **A user's Stop survives a reboot.** The UI's Stop is `docker compose down`, which REMOVES the
// containers; a boot interruption leaves them behind as Exited. So "has containers on disk that
// are down" is the boot-orphan signature, and a stack with ZERO containers is deliberately never
// touched. This distinction is the whole safety argument — see TestReconcile_UserStoppedAppIsNeverStarted.
// - **A user's Stop survives a reboot.** This is still the whole safety argument; only the way it
// is established changed. Until v0.189.0 it was inferred — the UI's Stop is `docker compose
// down`, which REMOVES containers, so "zero containers" was read as "the customer stopped it"
// and left alone. Since v0.189.0 (R-166) the customer's intent is RECORDED in app.yaml and read
// directly, because the inference could not distinguish a deliberate Stop from a power cut or an
// interrupted backup, and silently stranded both. An app.yaml with no recorded intent — every
// app on every box predating the field — keeps the old inference exactly. See isBootOrphan,
// TestReconcile_UserStoppedAppIsNeverStarted and
// TestReconcile_LegacyNoDesiredState_BehavesExactlyAsBefore.
//
// It runs inside the notifier's boot grace (cmd/controller/main.go `deadAppBootGrace`), so a
// successful recovery never fires an alert and a failed one alerts honestly once the grace expires.
@@ -87,17 +92,66 @@ func sleepCtx(ctx context.Context, d time.Duration) {
// isBootOrphan reports whether a stack is an app the boot left behind.
//
// The gate, term by term:
// - Deployed — an app the customer asked to have running.
// - Deployed — an app that is installed. NOTE: `Deployed` means INSTALLED, not "wanted running";
// the two were conflated until v0.189.0 and that conflation is what the desired-state term below
// repairs.
// - not Protected — traefik/cloudflared/felhom-controller have their own supervision; this must
// never race the base-stack self-heal.
// - not Deploying — mid-deploy is not a fault.
// - has containers — the D-case guard: a UI Stop removes them, and a deliberate stop must survive
// a reboot.
// - desired state — see below. REPLACES the old container-count term.
// - IsDownState — stopped/exited/degraded (R-51 included: a boot that half-started a stack is the
// same interrupted-boot shape).
//
// ── WHY INTENT REPLACED THE CONTAINER COUNT (R-166, closing R-157 mechanism B) ────────────────────
//
// This gate used to end in `len(s.Containers) > 0`, and its comment called that "the D-case guard":
// a UI Stop is `compose down`, which REMOVES containers, so zero containers was read as "the
// customer stopped this" and left alone. The safety goal was right and still holds. The SIGNAL was
// wrong, because zero containers has at least three causes and the count cannot tell them apart:
//
// a deliberate Stop → must stay down
// a power cut mid-compose, or an interrupted deploy → must come back
// a backup that stopped the app and died before restarting it → must come back
//
// Two of those three were silently unrecoverable: the app simply stayed gone until a human noticed.
// The count was never capable of separating them, so the fix is not a better inference — it is to
// stop inferring and read what the customer actually asked for, which app.yaml now records.
//
// ── WHAT ABSENT STILL MEANS, AND WHY THE OLD BEHAVIOUR IS KEPT ────────────────────────────────────
//
// DesiredStateUnknown falls back to the ORIGINAL container-count rule, byte-for-byte. This is the
// single most important line in the change. Every app.yaml on every existing box predates the field,
// so absent is what the whole fleet reads on upgrade; treating absent as "running" would start, on
// the first boot after the upgrade, every app its owner had deliberately stopped. The fallback is
// what makes this feature inert for an app nobody has pressed a button on since — see
// TestReconcile_LegacyNoDesiredState_BehavesExactlyAsBefore and its red-proof.
//
// The full decision table (§8.1):
//
// desired containers state → result
// stopped any any → never an orphan (the customer said so)
// running 0 — → ORPHAN ← the R-157 case, invisible before v0.189.0
// running >0 IsDownState → ORPHAN (unchanged)
// running >0 up → not an orphan
// absent 0 — → not an orphan (exactly the pre-v0.189.0 behaviour)
// absent >0 IsDownState → ORPHAN (exactly the pre-v0.189.0 behaviour)
func isBootOrphan(s stacks.Stack) bool {
return s.Deployed && !s.Protected && !s.Deploying &&
len(s.Containers) > 0 && stacks.IsDownState(s.State)
if !s.Deployed || s.Protected || s.Deploying {
return false
}
switch stacks.DesiredStateOf(s) {
case stacks.DesiredStateStopped:
// The customer pressed Stop. No observation may overturn that — not a missing container, not
// a down state, not a reboot. Nothing else in this package starts an app.
return false
case stacks.DesiredStateRunning:
// Wanted running. ANY way of not being up is a fault to repair, including having no
// containers at all — which is the case the old count term structurally could not see.
return len(s.Containers) == 0 || stacks.IsDownState(s.State)
default:
// DesiredStateUnknown — legacy. Keep the pre-R-166 rule exactly.
return len(s.Containers) > 0 && stacks.IsDownState(s.State)
}
}
// Run performs the sweep once and returns what happened. It is safe to call with no boot orphans