v0.189.0 — desired state + the app-stop crash marker (R-166 / D-b)
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
The box stops inferring the customer's intent from a container count and reads
what they actually asked for.
Part 1 — desired state. AppConfig gains a tri-state `desired_state`
(""/running/stopped), written ONLY by the customer's own action: the API action
switch, DeployStack, UpdateOptionalConfig's redeploy branch, and the .fab
import. Intent is written BEFORE the act and a failed write REFUSES the act.
StartStack/StopStack are deliberately not writers — 14 callers, only 2 are the
customer. bootrecon.isBootOrphan now reads intent instead of len(Containers)>0,
which closes R-157 mechanism B (a power cut or interrupted deploy left an app
with zero containers, read as a deliberate stop, and stranded silently).
ABSENT MEANS UNKNOWN, NEVER "running": every pre-v0.189.0 app.yaml reads absent,
so the legacy fallback is byte-identical to the old rule. A running-only startup
backfill converges the unambiguous cases; `stopped` is never inferred.
Part 2 — backup.AppStopGuard, a persisted marker over every stop→work→start
window (volume dump, offbox reconstitute, .fab export). Its own file, never
quiesce's. Written before the stop, cleared only after a restart that succeeded,
kept when one fails. Recover() completes before the boot reconciler is launched
and returns its outcome, which main.go reports on the existing backup_failed
event once the notifier exists. A defer is not the mechanism — a SIGKILL runs
none (Campaign 8 fault 10).
Also: SaveAppConfig rebuilt AppConfig field-by-field (the R-100 shape) and would
have dropped desired_state on every save across nine call sites. Replaced with
copy-and-overlay. Measured: app.yaml does not round-trip unknown YAML keys.
No hub change, no agent coupling, no user-visible string. 27/27 packages green;
7 red-proofs observed FAIL then restored.
This commit is contained in:
+53
-11
@@ -1868,17 +1868,59 @@ invariant changes, revisit the suppression.
|
||||
callers rely on stopped counting as down). An out-of-band `docker compose stop` leaves the containers
|
||||
present → `StateExited` → still alerts, which is correct (out-of-band tampering is reportable).
|
||||
|
||||
**Boot desired-state reconciliation (R-52, v0.156.0, `internal/bootrecon`).** A `deployed: true` app
|
||||
that missed its boot start used to stay down until a human noticed — the same shutdown that produced
|
||||
F4 left immich and calibre-web `Exited` while ten sibling containers came back, and they were still
|
||||
down 18 h later (F5). At startup (5 s after the quiesce recovery, so the two never race) the
|
||||
controller performs **one bounded sweep**: every deployed, non-protected, not-mid-deploy stack that
|
||||
still HAS containers and is down gets `StartStack`, at most **2 attempts 30 s apart**, then it stops
|
||||
and the alarm owns the problem. Never a restart loop. **An app the customer stopped is never
|
||||
touched** — the UI's Stop is `compose down`, which removes the containers, so "has containers and
|
||||
they are down" is what distinguishes an interrupted boot from a deliberate stop. The whole sweep
|
||||
fits inside the 90 s boot grace, so a successful recovery is silent and a failed one still alerts.
|
||||
Outcome is logged per attempt at INFO; no new hub event (the existing alarm is the escalation).
|
||||
**Boot desired-state reconciliation (R-52, v0.156.0, `internal/bootrecon`; rebuilt on recorded intent
|
||||
in R-166, v0.189.0).** A `deployed: true` app that missed its boot start used to stay down until a
|
||||
human noticed — the same shutdown that produced F4 left immich and calibre-web `Exited` while ten
|
||||
sibling containers came back, and they were still down 18 h later (F5). At startup (5 s after the
|
||||
quiesce and app-stop recoveries, so the three never race) the controller performs **one bounded
|
||||
sweep**: every deployed, non-protected, not-mid-deploy stack that is down gets `StartStack`, at most
|
||||
**2 attempts 30 s apart**, then it stops and the alarm owns the problem. Never a restart loop. The
|
||||
whole sweep fits inside the 90 s boot grace, so a successful recovery is silent and a failed one
|
||||
still alerts. Outcome is logged per attempt at INFO; no new hub event (the existing alarm is the
|
||||
escalation).
|
||||
|
||||
**What "down" means here changed in v0.189.0.** Until then the sweep required the stack to still HAVE
|
||||
containers, because the UI's Stop is `compose down` (which removes them) and "zero containers" was
|
||||
read as a deliberate stop. That inference was wrong in two silent ways: a **power cut mid-compose**
|
||||
and an **interrupted deploy** also leave zero containers, and both were skipped as "the customer
|
||||
stopped it" and left down indefinitely. Since R-166 the sweep reads the customer's **recorded
|
||||
intent** (`desired_state` in `app.yaml`) instead:
|
||||
|
||||
| `desired_state` | containers | result |
|
||||
|---|---|---|
|
||||
| `stopped` | any | **never** started — the customer said so, and no observation overrides it |
|
||||
| `running` | 0 | **recovered** — the power-cut / interrupted-deploy case, invisible before v0.189.0 |
|
||||
| `running` | >0 and down | **recovered** (unchanged) |
|
||||
| `running` | >0 and up | left alone |
|
||||
| absent (legacy) | 0 | **not** started — byte-identical to the pre-v0.189.0 behaviour |
|
||||
| absent (legacy) | >0 and down | **recovered** — byte-identical to the pre-v0.189.0 behaviour |
|
||||
|
||||
**Absent means UNKNOWN, never "running".** Every `app.yaml` written before v0.189.0 lacks the field,
|
||||
so absent is what an upgraded box reads for every app that has not been started or stopped since;
|
||||
reading it as "running" would start every deliberately-stopped app on the first boot after the
|
||||
upgrade. Where intent is unknown the sweep falls back to the old inference rather than inventing an
|
||||
answer, and a running-only startup **backfill** converges the unambiguous cases (deployed and
|
||||
observed up) without waiting for a button press. `stopped` is never backfilled from any signal.
|
||||
|
||||
**Desired state — who owns it (R-166, v0.189.0).** `app.yaml` gains `desired_state`, a tri-state
|
||||
`""` / `running` / `stopped`. It is written by **the customer's own action and nothing else**: the
|
||||
`/api/stacks/{name}/{action}` switch (`start`/`restart`/`update` → running, `stop` → stopped),
|
||||
`DeployStack`, `UpdateOptionalConfig`'s redeploy branch, and the `.fab` import. `StartStack` and
|
||||
`StopStack` are deliberately **not** writers — a census found 14 callers of which only 2 are the
|
||||
customer, and recording intent in the primitive would make a nightly backup indistinguishable from
|
||||
the customer pressing Stop, which is the confusion the feature exists to end. Intent is written
|
||||
**before** the act, and an action whose intent cannot be recorded is **refused**.
|
||||
|
||||
**Interrupted app-data operations (R-166, v0.189.0, `backup.AppStopGuard`).** A volume dump, an
|
||||
off-site reconstitution and a `.fab` export all stop an app, work on its data, and start it again.
|
||||
A controller killed inside that window left the app down with nothing on disk recording why or that
|
||||
it was owed a restart. A persisted marker (`<data_dir>/appstop-state.json` — its **own** file, never
|
||||
quiesce's, so one file has one writer) is now written **before** the stop and cleared only after a
|
||||
restart that succeeded; a failed restart deliberately keeps it. At startup `Recover()` restarts the
|
||||
recorded apps, clears the marker, and its outcome is reported to the operator on the existing
|
||||
`backup_failed` event — an interrupted operation means the backup did not complete. **The `defer` in
|
||||
those functions is not the mechanism**: a SIGKILL runs no deferred function (Campaign 8 fault 10, on
|
||||
live hardware), which is exactly what the marker covers.
|
||||
|
||||
#### Default Enabled Events
|
||||
|
||||
|
||||
Reference in New Issue
Block a user