feat(v0.156.0): dead-primary alerting (R-51) + boot desired-state reconciliation (R-52)
R-51: aggregateState's mixed branch returned StateRunning ("partial"), so a stack whose
MAIN container was dead behind live helpers alerted on nothing — immich-server sat Exited
for 18 h, 100 % unreachable, no banner and no app_start_failed (audit F4). New
StateDegraded: a DOWN member whose docker restart policy is always/unless-stopped is a
fault (degraded, a down state); no/on-failure is a finished one-shot and stays benign; an
unreadable policy fails CLOSED. The unhealthy/restarting/paused/unknown exclusions are
byte-identical — folding unhealthy into down is the flapping fix-3 avoided.
R-52: new internal/bootrecon — one bounded start-once sweep at startup (2 attempts, 30 s
apart) for apps an interrupted boot left behind, inside the 90 s boot grace so a success
is silent and a failure still alerts. A zero-container stack is NEVER touched: the UI's
Stop is compose down, so a deliberate stop survives a reboot.
Both features carry a production-path wiring test (the v0.154.0 / v0.91.0 inert-seam
class). The main() assertion is an AST walk, not strings.Contains — the substring version
passed its own red-proof, because a commented-out call still contains the string.
Red-proofs run and restored: mix branch reverted -> "running" on the immich fixture;
boot hook commented out -> wiring test fails; zero-container gate dropped -> the
user-stopped app gets started.
NOTE: controller/cmd/controller/ is matched by .gitignore's `controller` entry, so new
files there need `git add -f` (and ripgrep silently skips main.go without --no-ignore).
This commit is contained in:
@@ -1,5 +1,64 @@
|
||||
## Changelog
|
||||
|
||||
### v0.156.0 — a dead primary alerts (R-51); a boot orphan restarts itself (R-52) (2026-07-21)
|
||||
|
||||
**No new agent coupling — MinAgent stays 0.90.0.** Two independent failures from the same live
|
||||
audit, both unattended-resilience holes: the box was broken and nobody was told, then the box could
|
||||
have fixed itself and did not.
|
||||
|
||||
**R-51 — a multi-container app whose MAIN container is dead now counts as down.** On 2026-07-20
|
||||
`immich-server` sat `Exited` for **18 hours** with the app 100 % unreachable, and the box produced no
|
||||
dead-app banner and no `app_start_failed` event — while single-container Calibre-Web, down for the
|
||||
same reason, alerted in 90 seconds (AUDIT-vacation-remote-ops-2026-07-20 F4).
|
||||
|
||||
The defect was one branch in `aggregateState`: a stack with *some* members running and *some* stopped
|
||||
returned `StateRunning` — "partial" — and `IsDownState` (correctly) does not treat running as down.
|
||||
So the alarm never had anything to fire on. *(The ROADMAP row's diagnosis — "aggregation classifies
|
||||
such a stack `unhealthy`" — is wrong at the source; corrected in the row.)*
|
||||
|
||||
- New `StateDegraded`. The mixed branch now asks each DOWN member for its restart policy: a member
|
||||
docker is supposed to keep running (`always` / `unless-stopped`) makes the stack **degraded**, a
|
||||
finished one-shot (`no` / `on-failure`) leaves it running. `IsDownState` gains `degraded` and
|
||||
**nothing else** — the `unhealthy` / `restarting` / `paused` / `unknown` exclusions are byte-
|
||||
identical, because folding `unhealthy` into down is what fix-3 removed the flapping by not doing.
|
||||
- An **unreadable** policy counts as supervised (fail-CLOSED), the opposite of the IsDownState
|
||||
fail-open rule and for a different reason: there the *state* is ambiguous, here a member is known
|
||||
dead and only the excuse is missing. The P2 census backs it — all 53 catalog templates / 78
|
||||
services are `unless-stopped`, and zero one-shot containers exist today.
|
||||
- The policy read is one `docker inspect` per down member of a *mixed* stack, cached per
|
||||
container+state and pruned to the live container set, so the 10 s refresh does not grow a docker
|
||||
call per container.
|
||||
- Everything that asks "are there live containers here" learns the state too: quiesce
|
||||
(`RunningAppStacks`), delete's stop-first guard, the export stop-first guard, telemetry, health
|
||||
probes. Everything that asks "is this app working" counts it as down: the dashboard counter, the
|
||||
stopped filter, the dead-app banner and the alarm. UI: „Részlegesen leállt", warn colour, and the
|
||||
URL is flagged unpublished (Traefik 404s when the routed member is the dead one).
|
||||
|
||||
**R-52 — an app the boot left behind now gets exactly one recovery.** The same shutdown left immich
|
||||
and calibre-web `Exited` while ten sibling containers came back; the controller *reported* them for
|
||||
18 hours and never started them (F5).
|
||||
|
||||
- New `internal/bootrecon`: one bounded sweep at startup — at most 2 attempts, 30 s apart, then it
|
||||
stops and the alarm owns the problem. **Never a restart loop.**
|
||||
- **A deliberate Stop survives a reboot.** The UI's Stop is `compose down`, which REMOVES the
|
||||
containers; an interrupted boot leaves them behind as `Exited`. So the boot-orphan signature is
|
||||
"deployed, has containers, and they are down", and a zero-container stack is never touched.
|
||||
- The whole sweep (5 s settle + one 30 s gap) fits inside the 90 s `deadAppBootGrace`, so a
|
||||
successful recovery never alerts and a failed one alerts honestly. A test asserts that arithmetic
|
||||
rather than leaving it to a comment.
|
||||
|
||||
**Seam discipline (the reason both features have a wiring test).** Two inert-seam defects shipped in
|
||||
the two days before this: controller v0.154.0 and agent v0.91.0, both a correct component with green
|
||||
tests and no production caller. So the boot sweep is asserted from `package main` — including an AST
|
||||
walk proving `func main()` actually contains the `go runBootReconcile(...)`. That test was written
|
||||
first as a `strings.Contains` and **its own red-proof passed it**, because a commented-out call still
|
||||
contains the string. Comments are not callers; the AST version fails as it should.
|
||||
|
||||
Red-proofs (all run, all failed on the pre-fix shape, all restored): the mix branch reverted to
|
||||
`return StateRunning` → the immich fixture and both production-path tests fail with `"running"`; the
|
||||
boot hook commented out → the wiring test fails; the zero-container gate dropped → the user-stopped
|
||||
app is started, which is the one thing R-52 must never do.
|
||||
|
||||
### v0.155.0 — the restore wizard read the wrong "is something running" flag (2026-07-21)
|
||||
|
||||
**No new agent coupling — MinAgent stays 0.90.0.** Fixes a defect shipped in v0.154.0 and found by
|
||||
|
||||
Reference in New Issue
Block a user