feat(v0.156.0): dead-primary alerting (R-51) + boot desired-state reconciliation (R-52)
R-51: aggregateState's mixed branch returned StateRunning ("partial"), so a stack whose
MAIN container was dead behind live helpers alerted on nothing — immich-server sat Exited
for 18 h, 100 % unreachable, no banner and no app_start_failed (audit F4). New
StateDegraded: a DOWN member whose docker restart policy is always/unless-stopped is a
fault (degraded, a down state); no/on-failure is a finished one-shot and stays benign; an
unreadable policy fails CLOSED. The unhealthy/restarting/paused/unknown exclusions are
byte-identical — folding unhealthy into down is the flapping fix-3 avoided.
R-52: new internal/bootrecon — one bounded start-once sweep at startup (2 attempts, 30 s
apart) for apps an interrupted boot left behind, inside the 90 s boot grace so a success
is silent and a failure still alerts. A zero-container stack is NEVER touched: the UI's
Stop is compose down, so a deliberate stop survives a reboot.
Both features carry a production-path wiring test (the v0.154.0 / v0.91.0 inert-seam
class). The main() assertion is an AST walk, not strings.Contains — the substring version
passed its own red-proof, because a commented-out call still contains the string.
Red-proofs run and restored: mix branch reverted -> "running" on the immich fixture;
boot hook commented out -> wiring test fails; zero-container gate dropped -> the
user-stopped app gets started.
NOTE: controller/cmd/controller/ is matched by .gitignore's `controller` entry, so new
files there need `git add -f` (and ripgrep silently skips main.go without --no-ignore).
This commit is contained in:
@@ -1517,6 +1517,33 @@ the real cooldown, the controller adds no timer). The boot grace prevents false
|
||||
controller's own startup while STILL firing for an app that never came up. This closes the campaign's
|
||||
4-hour silent CWA death.
|
||||
|
||||
**Dead-primary alerting (R-51, v0.156.0).** fix-3 above only ever saw stacks that were *entirely*
|
||||
down. A multi-container app whose MAIN container died while its helpers kept running aggregated to
|
||||
`StateRunning` ("partial") and therefore alerted on nothing — `immich-server` was `Exited` for 18 h,
|
||||
the app 100 % unreachable, with no banner and no event (F4, AUDIT-vacation-remote-ops-2026-07-20).
|
||||
`aggregateState`'s mixed branch now inspects each DOWN member's docker restart policy: `always` /
|
||||
`unless-stopped` means docker was supposed to be keeping it up, so the stack becomes
|
||||
**`StateDegraded`** — a down state, so the existing banner and the existing `app_start_failed` event
|
||||
fire unchanged. `no` / `on-failure` is a finished one-shot init/migrate container and stays benign.
|
||||
An unreadable policy counts as supervised (fail-closed: a member is known dead, only the excuse is
|
||||
missing). The `unhealthy` / `restarting` / `paused` / `unknown` exclusions are untouched — folding
|
||||
`unhealthy` into down is precisely the flapping fix-3 avoided. UI: „Részlegesen leállt", warn
|
||||
colour, counted with the stopped apps, URL flagged unpublished (Traefik withholds the route when the
|
||||
routed member is the dead one). Policy reads are one `docker inspect` per down member of a mixed
|
||||
stack, cached per container+state.
|
||||
|
||||
**Boot desired-state reconciliation (R-52, v0.156.0, `internal/bootrecon`).** A `deployed: true` app
|
||||
that missed its boot start used to stay down until a human noticed — the same shutdown that produced
|
||||
F4 left immich and calibre-web `Exited` while ten sibling containers came back, and they were still
|
||||
down 18 h later (F5). At startup (5 s after the quiesce recovery, so the two never race) the
|
||||
controller performs **one bounded sweep**: every deployed, non-protected, not-mid-deploy stack that
|
||||
still HAS containers and is down gets `StartStack`, at most **2 attempts 30 s apart**, then it stops
|
||||
and the alarm owns the problem. Never a restart loop. **An app the customer stopped is never
|
||||
touched** — the UI's Stop is `compose down`, which removes the containers, so "has containers and
|
||||
they are down" is what distinguishes an interrupted boot from a deliberate stop. The whole sweep
|
||||
fits inside the 90 s boot grace, so a successful recovery is silent and a failed one still alerts.
|
||||
Outcome is logged per attempt at INFO; no new hub event (the existing alarm is the escalation).
|
||||
|
||||
#### Default Enabled Events
|
||||
|
||||
Events the customer receives notifications for (configurable in settings):
|
||||
|
||||
Reference in New Issue
Block a user