v0.164.0: deliberately stopped apps no longer alarm (banner + email)

A UI stop (Leallitas -> compose down -> StateStopped) is the user's own
action, not a fault, and must not raise the deadapp banner OR the
app_start_failed event. Filter at the single fix-3 derivation point:
extract scanDeployedAppRunStates's pure core to classifyRunStates and
change the down predicate to IsDownState(st.State) && st.State !=
StateStopped. Suppresses StateStopped from both the banner dead-list and
the notifier Down-set at once.

Rests on two invariants (recorded at the seam, README, CONTEXT):
 I1 StopStack = compose down => zero containers => StateStopped
 I2 P2 census: all catalog services unless-stopped => faults never rest
    at stopped (they surface as exited/degraded).
IsDownState unchanged; out-of-band 'compose stop' (containers remain ->
exited) still alerts. Tests +4 (notify 3->4, main 4->7), both red-proofs
verified. No template/funcmap/notifier/counter/copy change.
This commit is contained in:
2026-07-24 10:50:14 +02:00
parent 77956d8df2
commit c23a0f6d2d
7 changed files with 257 additions and 6 deletions
+18 -2
View File
@@ -1624,8 +1624,9 @@ Each event carries typed detail structs (e.g., `BackupDetails`, `DiskDetails`, `
**Deployed-app-down alerting (fix-3, v0.120.0, CAMPAIGN-3).** A `deadapp-check` scheduler job (every
30 s, after a 90 s boot grace) scans `stackMgr.GetStacks()`: a DEPLOYED app whose containers are
`stopped`/`exited` (`stacks.IsDownState` — a Docker `created`/`dead` container, the F11 dead-at-boot
case, resolves to `stopped`) gets a **state-based WARN dashboard banner** ("Telepített alkalmazás nem
`exited`/`degraded` (`stacks.IsDownState` minus the `stopped` exclusion added in v0.164.0 — see below;
a Docker `created`/`dead` container, the F11 dead-at-boot case, resolves to `exited`) gets a
**state-based WARN dashboard banner** ("Telepített alkalmazás nem
fut: <app>", grouped above 3 so a reboot storm doesn't wall the dashboard) that self-clears when the
app runs again, AND an `app_start_failed` hub event fired **once per running→down transition**
(`Notifier.NotifyAppStartFailures` tracks per-app state; down→down cycles are silent — the hub owns
@@ -1648,6 +1649,21 @@ colour, counted with the stopped apps, URL flagged unpublished (Traefik withhold
routed member is the dead one). Policy reads are one `docker inspect` per down member of a mixed
stack, cached per container+state.
**Deliberate stops are silent (v0.164.0).** Stopping an app from the UI (Leállítás → `StopStack`
`docker compose down` → zero containers → the deployed stack aggregates to `StateStopped`) is the
user's own action, not a fault, and must not raise the banner OR the `app_start_failed` email. The
scan's pure core was extracted to `classifyRunStates([]stacks.Stack)` and its down predicate is now
`stacks.IsDownState(st.State) && st.State != stacks.StateStopped` — the SINGLE fix-3 derivation point,
so `StateStopped` is dropped from both the banner dead-list and the notifier Down-set at once (the
launcher tile still shows greyed „Leállítva"; the monitoring page and dashboard counters are factual
display, not alarms, and are unchanged). This rests on two invariants: **I1** — a UI stop always ends
at `StateStopped` (compose down removes the containers); **I2** — the P2 restart-policy census
(53 templates / 78 services, all `unless-stopped`) means a crashing app never comes to rest at
`stopped`, so faults still surface as `exited`/`degraded`/`restarting`/`unhealthy`. If either
invariant changes, revisit the suppression. `IsDownState` itself is deliberately UNCHANGED (other
callers rely on stopped counting as down). An out-of-band `docker compose stop` leaves the containers
present → `StateExited` → still alerts, which is correct (out-of-band tampering is reportable).
**Boot desired-state reconciliation (R-52, v0.156.0, `internal/bootrecon`).** A `deployed: true` app
that missed its boot start used to stay down until a human noticed — the same shutdown that produced
F4 left immich and calibre-web `Exited` while ten sibling containers came back, and they were still
@@ -0,0 +1,123 @@
package main
import (
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
"gitea.dooplex.hu/admin/felhom-controller/internal/web"
)
// v0.164.0: classifyRunStates is the single fix-3 derivation point. A deliberate user stop
// (StateStopped) must NOT alarm — it is excluded from both the banner dead-list and the notifier
// Down-set — while every genuine fault (StateExited / StateDegraded) keeps alerting byte-identically.
// Invariants behind the suppression are documented at classifyRunStates (I1: compose down ⇒ zero
// containers ⇒ StateStopped; I2: P2 census — all catalog services unless-stopped ⇒ faults never rest
// at stopped).
func stack(name string, st stacks.ContainerState, deployed, deploying bool) stacks.Stack {
return stacks.Stack{
Name: name,
Meta: stacks.Metadata{DisplayName: name},
State: st,
Deployed: deployed,
Deploying: deploying,
}
}
func downByName(states []notify.AppRunState) map[string]bool {
m := map[string]bool{}
for _, s := range states {
m[s.Name] = s.Down
}
return m
}
func deadNames(dead []web.DeadApp) map[string]bool {
m := map[string]bool{}
for _, d := range dead {
m[d.Name] = true
}
return m
}
// Group A (Scenario A) — suppression. Over a [running, stopped, exited, degraded] fixture, the dead
// list is EXACTLY {exited, degraded} and the Down flags are {false, false, true, true}: the stopped
// app is silent, the two faults still alarm.
//
// COMPANION red-proof: revert the filter to bare `stacks.IsDownState(st.State)` (drop the
// `&& st.State != stacks.StateStopped` guard) → stopped reports Down=true and enters the dead list →
// both the dead-set and the Down-flag assertions below fail. (Verified by hand-editing the seam.)
func TestClassifyRunStates_StoppedIsSuppressed(t *testing.T) {
sts := []stacks.Stack{
stack("radarr", stacks.StateRunning, true, false),
stack("cwa", stacks.StateStopped, true, false),
stack("immich", stacks.StateExited, true, false),
stack("nextcloud", stacks.StateDegraded, true, false),
}
dead, states := classifyRunStates(sts)
gotDead := deadNames(dead)
if len(gotDead) != 2 || !gotDead["immich"] || !gotDead["nextcloud"] {
t.Fatalf("dead list must be exactly {immich(exited), nextcloud(degraded)}, got %+v", dead)
}
if gotDead["cwa"] {
t.Errorf("a deliberately stopped app must NOT be in the dead list (no banner)")
}
if gotDead["radarr"] {
t.Errorf("a running app must never be in the dead list")
}
down := downByName(states)
want := map[string]bool{"radarr": false, "cwa": false, "immich": true, "nextcloud": true}
if len(down) != len(want) {
t.Fatalf("every deployed app must have a run state, got %+v", down)
}
for name, w := range want {
if down[name] != w {
t.Errorf("Down[%s] = %v, want %v (stopped ⇒ false ⇒ no app_start_failed event)", name, down[name], w)
}
}
}
// Group B (Scenario B) — fault parity. With only exited + degraded present, BOTH surface in the dead
// list AND both report Down=true — byte-identical to v0.163.1 for every non-stopped down state. The
// suppression touches stopped and nothing else.
func TestClassifyRunStates_FaultParity(t *testing.T) {
sts := []stacks.Stack{
stack("immich", stacks.StateExited, true, false),
stack("nextcloud", stacks.StateDegraded, true, false),
}
dead, states := classifyRunStates(sts)
gotDead := deadNames(dead)
if len(gotDead) != 2 || !gotDead["immich"] || !gotDead["nextcloud"] {
t.Fatalf("both faults must appear in the dead list, got %+v", dead)
}
down := downByName(states)
if !down["immich"] || !down["nextcloud"] {
t.Fatalf("both faults must report Down=true, got %+v", down)
}
// State strings must ride through to the banner unchanged (banner shows "(exited)"/"(degraded)").
byName := map[string]string{}
for _, d := range dead {
byName[d.Name] = d.State
}
if byName["immich"] != string(stacks.StateExited) || byName["nextcloud"] != string(stacks.StateDegraded) {
t.Errorf("dead-app State must carry the raw aggregate state, got %+v", byName)
}
}
// Deploying and undeployed stacks are skipped entirely (unchanged fix-3 behavior).
func TestClassifyRunStates_SkipsDeployingAndUndeployed(t *testing.T) {
sts := []stacks.Stack{
stack("mid", stacks.StateDeploying, true, true), // mid-deploy → skipped
stack("gone", stacks.StateExited, false, false), // not deployed → skipped
}
dead, states := classifyRunStates(sts)
if len(dead) != 0 || len(states) != 0 {
t.Fatalf("deploying and undeployed stacks must be skipped, got dead=%+v states=%+v", dead, states)
}
}
+24 -3
View File
@@ -1161,15 +1161,36 @@ func runBootReconcile(ctx context.Context, mgr bootrecon.StackProvider, logger *
// scanDeployedAppRunStates returns the fix-3 view of the deployed apps: the DEAD ones (for the
// state-based dashboard banner) and EVERY deployed app's run state (for the notifier's one-event-per-
// transition tracking). Deploying apps are skipped (mid-deploy is not a fault). Pure over GetStacks().
// transition tracking). Deploying apps are skipped (mid-deploy is not a fault). Pure over GetStacks()
// — the derivation itself lives in classifyRunStates so it is testable without a live Manager.
func scanDeployedAppRunStates(mgr *stacks.Manager) ([]web.DeadApp, []notify.AppRunState) {
return classifyRunStates(mgr.GetStacks())
}
// classifyRunStates is the pure fix-3 derivation over a plain stack slice. It splits the deployed
// apps into the DEAD list (dashboard banner) and the per-app run states (notifier transition tracker).
//
// v0.164.0: a deliberate user stop is NOT a fault and must not alarm anywhere (banner OR email). The
// down predicate therefore EXCLUDES StateStopped, resting on two invariants:
// - I1: the UI stop path Manager.StopStack runs `docker compose down` → containers are removed, and
// a deployed stack with zero containers aggregates to StateStopped (manager.go refreshStatusLocked).
// So StateStopped means "deployed, deliberately stopped by the user".
// - I2: the P2 restart-policy census (2026-07-21, 53 templates / 78 services) found every catalog
// service on `unless-stopped`, so a crashing app never comes to rest at `stopped` — faults surface
// as StateExited / StateDegraded (and restarting/unhealthy). StateStopped is therefore never a fault.
//
// If either invariant changes, revisit this suppression. (An out-of-band `docker compose stop` leaves
// the containers present → StateExited → still alerts, which is correct: out-of-band tampering IS
// reportable.) IsDownState is intentionally left unchanged — other callers rely on stopped counting as
// down; the suppression is a filter at this single derivation point only.
func classifyRunStates(sts []stacks.Stack) ([]web.DeadApp, []notify.AppRunState) {
var dead []web.DeadApp
var states []notify.AppRunState
for _, st := range mgr.GetStacks() {
for _, st := range sts {
if !st.Deployed || st.Deploying {
continue
}
down := stacks.IsDownState(st.State)
down := stacks.IsDownState(st.State) && st.State != stacks.StateStopped
states = append(states, notify.AppRunState{Name: st.Name, DisplayName: st.Meta.DisplayName, Down: down})
if down {
dead = append(dead, web.DeadApp{Name: st.Name, DisplayName: st.Meta.DisplayName, State: string(st.State)})
@@ -56,6 +56,42 @@ func TestNotifyAppStartFailures_FirstSeenDownFires(t *testing.T) {
}
}
// Group C (Scenario C, v0.164.0) — stop → start → crash stays correct. The stop is reported as
// Down=false (classifyRunStates suppresses StateStopped — proven in package main's
// TestClassifyRunStates_StoppedIsSuppressed), so it fires NOTHING and leaves the tracker clean; the
// later crash (StateExited → Down=true) is then a clean false→true transition → exactly ONE event.
//
// COMPANION red-proof: flip the stop cycle to Down=true (as bare IsDownState would report before the
// suppression) → the stop becomes a running→down transition and fires here, so the "zero after the
// stop" assertion below fails. (Verified by hand-editing the fixture.)
func TestNotifyAppStartFailures_StopStartCrashSequence(t *testing.T) {
n := New("http://hub", "key", "cust", nil, log.New(io.Discard, "", 0), false)
var count int
n.pushFn = func(eventType, _, _ string, _ interface{}) {
if eventType == "app_start_failed" {
count++
}
}
app := "immich"
stopped := []AppRunState{{Name: app, DisplayName: "Immich", Down: false}} // user stop: StateStopped ⇒ Down=false
running := []AppRunState{{Name: app, DisplayName: "Immich", Down: false}} // started again
exited := []AppRunState{{Name: app, DisplayName: "Immich", Down: true}} // then it crashes: StateExited
// The deliberate stop must be SILENT — the whole point of the suppression.
n.NotifyAppStartFailures(stopped)
if count != 0 {
t.Fatalf("a deliberate stop must fire no event, got %d", count)
}
// Start, then crash → exactly ONE app_start_failed for the final false→true transition.
n.NotifyAppStartFailures(running)
n.NotifyAppStartFailures(exited)
if count != 1 {
t.Fatalf("stop→start→crash must fire exactly one event for the crash, got %d", count)
}
}
// A deployed app that is up never fires.
func TestNotifyAppStartFailures_HealthyNeverFires(t *testing.T) {
n := New("http://hub", "key", "cust", nil, log.New(io.Discard, "", 0), false)