v0.164.0: deliberately stopped apps no longer alarm (banner + email)
A UI stop (Leallitas -> compose down -> StateStopped) is the user's own
action, not a fault, and must not raise the deadapp banner OR the
app_start_failed event. Filter at the single fix-3 derivation point:
extract scanDeployedAppRunStates's pure core to classifyRunStates and
change the down predicate to IsDownState(st.State) && st.State !=
StateStopped. Suppresses StateStopped from both the banner dead-list and
the notifier Down-set at once.
Rests on two invariants (recorded at the seam, README, CONTEXT):
I1 StopStack = compose down => zero containers => StateStopped
I2 P2 census: all catalog services unless-stopped => faults never rest
at stopped (they surface as exited/degraded).
IsDownState unchanged; out-of-band 'compose stop' (containers remain ->
exited) still alerts. Tests +4 (notify 3->4, main 4->7), both red-proofs
verified. No template/funcmap/notifier/counter/copy change.
This commit is contained in:
+18
-2
@@ -1624,8 +1624,9 @@ Each event carries typed detail structs (e.g., `BackupDetails`, `DiskDetails`, `
|
||||
|
||||
**Deployed-app-down alerting (fix-3, v0.120.0, CAMPAIGN-3).** A `deadapp-check` scheduler job (every
|
||||
30 s, after a 90 s boot grace) scans `stackMgr.GetStacks()`: a DEPLOYED app whose containers are
|
||||
`stopped`/`exited` (`stacks.IsDownState` — a Docker `created`/`dead` container, the F11 dead-at-boot
|
||||
case, resolves to `stopped`) gets a **state-based WARN dashboard banner** ("Telepített alkalmazás nem
|
||||
`exited`/`degraded` (`stacks.IsDownState` minus the `stopped` exclusion added in v0.164.0 — see below;
|
||||
a Docker `created`/`dead` container, the F11 dead-at-boot case, resolves to `exited`) gets a
|
||||
**state-based WARN dashboard banner** ("Telepített alkalmazás nem
|
||||
fut: <app>", grouped above 3 so a reboot storm doesn't wall the dashboard) that self-clears when the
|
||||
app runs again, AND an `app_start_failed` hub event fired **once per running→down transition**
|
||||
(`Notifier.NotifyAppStartFailures` tracks per-app state; down→down cycles are silent — the hub owns
|
||||
@@ -1648,6 +1649,21 @@ colour, counted with the stopped apps, URL flagged unpublished (Traefik withhold
|
||||
routed member is the dead one). Policy reads are one `docker inspect` per down member of a mixed
|
||||
stack, cached per container+state.
|
||||
|
||||
**Deliberate stops are silent (v0.164.0).** Stopping an app from the UI (Leállítás → `StopStack` →
|
||||
`docker compose down` → zero containers → the deployed stack aggregates to `StateStopped`) is the
|
||||
user's own action, not a fault, and must not raise the banner OR the `app_start_failed` email. The
|
||||
scan's pure core was extracted to `classifyRunStates([]stacks.Stack)` and its down predicate is now
|
||||
`stacks.IsDownState(st.State) && st.State != stacks.StateStopped` — the SINGLE fix-3 derivation point,
|
||||
so `StateStopped` is dropped from both the banner dead-list and the notifier Down-set at once (the
|
||||
launcher tile still shows greyed „Leállítva"; the monitoring page and dashboard counters are factual
|
||||
display, not alarms, and are unchanged). This rests on two invariants: **I1** — a UI stop always ends
|
||||
at `StateStopped` (compose down removes the containers); **I2** — the P2 restart-policy census
|
||||
(53 templates / 78 services, all `unless-stopped`) means a crashing app never comes to rest at
|
||||
`stopped`, so faults still surface as `exited`/`degraded`/`restarting`/`unhealthy`. If either
|
||||
invariant changes, revisit the suppression. `IsDownState` itself is deliberately UNCHANGED (other
|
||||
callers rely on stopped counting as down). An out-of-band `docker compose stop` leaves the containers
|
||||
present → `StateExited` → still alerts, which is correct (out-of-band tampering is reportable).
|
||||
|
||||
**Boot desired-state reconciliation (R-52, v0.156.0, `internal/bootrecon`).** A `deployed: true` app
|
||||
that missed its boot start used to stay down until a human noticed — the same shutdown that produced
|
||||
F4 left immich and calibre-web `Exited` while ten sibling containers came back, and they were still
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/web"
|
||||
)
|
||||
|
||||
// v0.164.0: classifyRunStates is the single fix-3 derivation point. A deliberate user stop
|
||||
// (StateStopped) must NOT alarm — it is excluded from both the banner dead-list and the notifier
|
||||
// Down-set — while every genuine fault (StateExited / StateDegraded) keeps alerting byte-identically.
|
||||
// Invariants behind the suppression are documented at classifyRunStates (I1: compose down ⇒ zero
|
||||
// containers ⇒ StateStopped; I2: P2 census — all catalog services unless-stopped ⇒ faults never rest
|
||||
// at stopped).
|
||||
|
||||
func stack(name string, st stacks.ContainerState, deployed, deploying bool) stacks.Stack {
|
||||
return stacks.Stack{
|
||||
Name: name,
|
||||
Meta: stacks.Metadata{DisplayName: name},
|
||||
State: st,
|
||||
Deployed: deployed,
|
||||
Deploying: deploying,
|
||||
}
|
||||
}
|
||||
|
||||
func downByName(states []notify.AppRunState) map[string]bool {
|
||||
m := map[string]bool{}
|
||||
for _, s := range states {
|
||||
m[s.Name] = s.Down
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func deadNames(dead []web.DeadApp) map[string]bool {
|
||||
m := map[string]bool{}
|
||||
for _, d := range dead {
|
||||
m[d.Name] = true
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Group A (Scenario A) — suppression. Over a [running, stopped, exited, degraded] fixture, the dead
|
||||
// list is EXACTLY {exited, degraded} and the Down flags are {false, false, true, true}: the stopped
|
||||
// app is silent, the two faults still alarm.
|
||||
//
|
||||
// COMPANION red-proof: revert the filter to bare `stacks.IsDownState(st.State)` (drop the
|
||||
// `&& st.State != stacks.StateStopped` guard) → stopped reports Down=true and enters the dead list →
|
||||
// both the dead-set and the Down-flag assertions below fail. (Verified by hand-editing the seam.)
|
||||
func TestClassifyRunStates_StoppedIsSuppressed(t *testing.T) {
|
||||
sts := []stacks.Stack{
|
||||
stack("radarr", stacks.StateRunning, true, false),
|
||||
stack("cwa", stacks.StateStopped, true, false),
|
||||
stack("immich", stacks.StateExited, true, false),
|
||||
stack("nextcloud", stacks.StateDegraded, true, false),
|
||||
}
|
||||
|
||||
dead, states := classifyRunStates(sts)
|
||||
|
||||
gotDead := deadNames(dead)
|
||||
if len(gotDead) != 2 || !gotDead["immich"] || !gotDead["nextcloud"] {
|
||||
t.Fatalf("dead list must be exactly {immich(exited), nextcloud(degraded)}, got %+v", dead)
|
||||
}
|
||||
if gotDead["cwa"] {
|
||||
t.Errorf("a deliberately stopped app must NOT be in the dead list (no banner)")
|
||||
}
|
||||
if gotDead["radarr"] {
|
||||
t.Errorf("a running app must never be in the dead list")
|
||||
}
|
||||
|
||||
down := downByName(states)
|
||||
want := map[string]bool{"radarr": false, "cwa": false, "immich": true, "nextcloud": true}
|
||||
if len(down) != len(want) {
|
||||
t.Fatalf("every deployed app must have a run state, got %+v", down)
|
||||
}
|
||||
for name, w := range want {
|
||||
if down[name] != w {
|
||||
t.Errorf("Down[%s] = %v, want %v (stopped ⇒ false ⇒ no app_start_failed event)", name, down[name], w)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Group B (Scenario B) — fault parity. With only exited + degraded present, BOTH surface in the dead
|
||||
// list AND both report Down=true — byte-identical to v0.163.1 for every non-stopped down state. The
|
||||
// suppression touches stopped and nothing else.
|
||||
func TestClassifyRunStates_FaultParity(t *testing.T) {
|
||||
sts := []stacks.Stack{
|
||||
stack("immich", stacks.StateExited, true, false),
|
||||
stack("nextcloud", stacks.StateDegraded, true, false),
|
||||
}
|
||||
|
||||
dead, states := classifyRunStates(sts)
|
||||
|
||||
gotDead := deadNames(dead)
|
||||
if len(gotDead) != 2 || !gotDead["immich"] || !gotDead["nextcloud"] {
|
||||
t.Fatalf("both faults must appear in the dead list, got %+v", dead)
|
||||
}
|
||||
down := downByName(states)
|
||||
if !down["immich"] || !down["nextcloud"] {
|
||||
t.Fatalf("both faults must report Down=true, got %+v", down)
|
||||
}
|
||||
// State strings must ride through to the banner unchanged (banner shows "(exited)"/"(degraded)").
|
||||
byName := map[string]string{}
|
||||
for _, d := range dead {
|
||||
byName[d.Name] = d.State
|
||||
}
|
||||
if byName["immich"] != string(stacks.StateExited) || byName["nextcloud"] != string(stacks.StateDegraded) {
|
||||
t.Errorf("dead-app State must carry the raw aggregate state, got %+v", byName)
|
||||
}
|
||||
}
|
||||
|
||||
// Deploying and undeployed stacks are skipped entirely (unchanged fix-3 behavior).
|
||||
func TestClassifyRunStates_SkipsDeployingAndUndeployed(t *testing.T) {
|
||||
sts := []stacks.Stack{
|
||||
stack("mid", stacks.StateDeploying, true, true), // mid-deploy → skipped
|
||||
stack("gone", stacks.StateExited, false, false), // not deployed → skipped
|
||||
}
|
||||
dead, states := classifyRunStates(sts)
|
||||
if len(dead) != 0 || len(states) != 0 {
|
||||
t.Fatalf("deploying and undeployed stacks must be skipped, got dead=%+v states=%+v", dead, states)
|
||||
}
|
||||
}
|
||||
@@ -1161,15 +1161,36 @@ func runBootReconcile(ctx context.Context, mgr bootrecon.StackProvider, logger *
|
||||
|
||||
// scanDeployedAppRunStates returns the fix-3 view of the deployed apps: the DEAD ones (for the
|
||||
// state-based dashboard banner) and EVERY deployed app's run state (for the notifier's one-event-per-
|
||||
// transition tracking). Deploying apps are skipped (mid-deploy is not a fault). Pure over GetStacks().
|
||||
// transition tracking). Deploying apps are skipped (mid-deploy is not a fault). Pure over GetStacks()
|
||||
// — the derivation itself lives in classifyRunStates so it is testable without a live Manager.
|
||||
func scanDeployedAppRunStates(mgr *stacks.Manager) ([]web.DeadApp, []notify.AppRunState) {
|
||||
return classifyRunStates(mgr.GetStacks())
|
||||
}
|
||||
|
||||
// classifyRunStates is the pure fix-3 derivation over a plain stack slice. It splits the deployed
|
||||
// apps into the DEAD list (dashboard banner) and the per-app run states (notifier transition tracker).
|
||||
//
|
||||
// v0.164.0: a deliberate user stop is NOT a fault and must not alarm anywhere (banner OR email). The
|
||||
// down predicate therefore EXCLUDES StateStopped, resting on two invariants:
|
||||
// - I1: the UI stop path Manager.StopStack runs `docker compose down` → containers are removed, and
|
||||
// a deployed stack with zero containers aggregates to StateStopped (manager.go refreshStatusLocked).
|
||||
// So StateStopped means "deployed, deliberately stopped by the user".
|
||||
// - I2: the P2 restart-policy census (2026-07-21, 53 templates / 78 services) found every catalog
|
||||
// service on `unless-stopped`, so a crashing app never comes to rest at `stopped` — faults surface
|
||||
// as StateExited / StateDegraded (and restarting/unhealthy). StateStopped is therefore never a fault.
|
||||
//
|
||||
// If either invariant changes, revisit this suppression. (An out-of-band `docker compose stop` leaves
|
||||
// the containers present → StateExited → still alerts, which is correct: out-of-band tampering IS
|
||||
// reportable.) IsDownState is intentionally left unchanged — other callers rely on stopped counting as
|
||||
// down; the suppression is a filter at this single derivation point only.
|
||||
func classifyRunStates(sts []stacks.Stack) ([]web.DeadApp, []notify.AppRunState) {
|
||||
var dead []web.DeadApp
|
||||
var states []notify.AppRunState
|
||||
for _, st := range mgr.GetStacks() {
|
||||
for _, st := range sts {
|
||||
if !st.Deployed || st.Deploying {
|
||||
continue
|
||||
}
|
||||
down := stacks.IsDownState(st.State)
|
||||
down := stacks.IsDownState(st.State) && st.State != stacks.StateStopped
|
||||
states = append(states, notify.AppRunState{Name: st.Name, DisplayName: st.Meta.DisplayName, Down: down})
|
||||
if down {
|
||||
dead = append(dead, web.DeadApp{Name: st.Name, DisplayName: st.Meta.DisplayName, State: string(st.State)})
|
||||
|
||||
@@ -56,6 +56,42 @@ func TestNotifyAppStartFailures_FirstSeenDownFires(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// Group C (Scenario C, v0.164.0) — stop → start → crash stays correct. The stop is reported as
|
||||
// Down=false (classifyRunStates suppresses StateStopped — proven in package main's
|
||||
// TestClassifyRunStates_StoppedIsSuppressed), so it fires NOTHING and leaves the tracker clean; the
|
||||
// later crash (StateExited → Down=true) is then a clean false→true transition → exactly ONE event.
|
||||
//
|
||||
// COMPANION red-proof: flip the stop cycle to Down=true (as bare IsDownState would report before the
|
||||
// suppression) → the stop becomes a running→down transition and fires here, so the "zero after the
|
||||
// stop" assertion below fails. (Verified by hand-editing the fixture.)
|
||||
func TestNotifyAppStartFailures_StopStartCrashSequence(t *testing.T) {
|
||||
n := New("http://hub", "key", "cust", nil, log.New(io.Discard, "", 0), false)
|
||||
var count int
|
||||
n.pushFn = func(eventType, _, _ string, _ interface{}) {
|
||||
if eventType == "app_start_failed" {
|
||||
count++
|
||||
}
|
||||
}
|
||||
|
||||
app := "immich"
|
||||
stopped := []AppRunState{{Name: app, DisplayName: "Immich", Down: false}} // user stop: StateStopped ⇒ Down=false
|
||||
running := []AppRunState{{Name: app, DisplayName: "Immich", Down: false}} // started again
|
||||
exited := []AppRunState{{Name: app, DisplayName: "Immich", Down: true}} // then it crashes: StateExited
|
||||
|
||||
// The deliberate stop must be SILENT — the whole point of the suppression.
|
||||
n.NotifyAppStartFailures(stopped)
|
||||
if count != 0 {
|
||||
t.Fatalf("a deliberate stop must fire no event, got %d", count)
|
||||
}
|
||||
|
||||
// Start, then crash → exactly ONE app_start_failed for the final false→true transition.
|
||||
n.NotifyAppStartFailures(running)
|
||||
n.NotifyAppStartFailures(exited)
|
||||
if count != 1 {
|
||||
t.Fatalf("stop→start→crash must fire exactly one event for the crash, got %d", count)
|
||||
}
|
||||
}
|
||||
|
||||
// A deployed app that is up never fires.
|
||||
func TestNotifyAppStartFailures_HealthyNeverFires(t *testing.T) {
|
||||
n := New("http://hub", "key", "cust", nil, log.New(io.Discard, "", 0), false)
|
||||
|
||||
Reference in New Issue
Block a user