v0.223.0: the app-down alarm reached nobody (R-329), and the stop nobody heard (R-386)

R-329. NotifyAppStartFailures emitted severity "warn". The hub accepts exactly
{info, warning, error, critical} and silently coerces anything else to "info",
which severityNotifies then drops BEFORE both legs. Banner shown, event stored,
POST 200, no mail sent. One word.

This is the second time: DiskAlertKind.Severity emitted "warn" until v0.215.0
and its own comment records that every warning-level disk alert went to nobody.
A comment recorded the lesson and nothing enforced it. The guard is now an AST
walk over the whole controller - grep cannot work here, since "warn" appears
legitimately nine times as a healthcheck status vocabulary.

The sweep found exactly one bad severity. Its limits are stated: the walk cannot
follow a variable, so all six dynamic call sites are registered by name with the
values each can take, and a new one fails the test. Two of the six were found by
the guard, not by the hand sweep before it.

Also pinned: fillwatch.Band.Severity() returns "" for BandOK, which would vanish
the same way. It is unreachable because Check() notifies only on escalation -
but that safety lives in a different function from the one that looks unsafe, so
the test asserts the consequence rather than the mapping.

app_start_failed gains a customer toggle, DEFAULT OFF, per operator ruling. The
operator is mailed either way: processOperator never consults customer prefs.
It is deliberately NOT in operatorOnlyEvents, which would make the toggle a lie.

R-386. classifyRunStates decided "the customer stopped this" from the STATE, so
every stopped stack was assumed deliberate. Measured on demo-hp: privatebin
stopped out of band, nine scans, zero events, zero banner - while the comment
beside it claimed an out-of-band stop still alerts.

DesiredState already records the answer and has exactly one writer. Stopped ->
no alarm; Running -> alarm; absent -> UNKNOWN, keep today's behaviour AND say
so. Absent stays silent deliberately: reading it as "nobody asked" would email
about every app anyone ever stopped, fleet-wide, on the first cycle after
upgrade. The gap is bounded not silent - IntentUnknown is set and the names are
logged at INFO on the heartbeat cadence. failedRestart still lifts a Stopped
intent, or F-CRIT-1 re-opens. No new DesiredState writer.

Two settings toggles each governed two alarms. "Lemez figyelmeztetes (90%+)"
also wrote disk_critical, the drive-is-FAILING alarm. Now four honest toggles;
12 became 15. A no-op save stores the existing slice verbatim, so byte identity
is by construction - without that guard the defaults case reorders, which the
red-proof caught.

Test count 1504 -> 1522. Five red-proofs, five seen failing; one passed first
time and is reported - that mutation was inert, not the test weak.

controller/cmd/controller/main.go

@@ -732,6 +732,7 @@ func main() {
 		notifier.NotifyAppStartFailures(states)
 		deadAppScans++
 		noteDeadAppScan(logger, deadAppScans, len(states), len(dead))
+		noteUnknownIntentSuppressions(logger, states)
 		return nil
 	})
 
@@ -1731,6 +1732,36 @@ func noteDeadAppScan(logger *log.Logger, scans, evaluated, down int) {
 		scans, evaluated, down)
 }
 
+// noteUnknownIntentSuppressions reports every app whose dead-app alarm was suppressed ONLY because
+// no customer intent was ever recorded (R-386, §4's ruling).
+//
+// **A rule without a mechanism is not a rule.** §4 chose to keep today's behaviour for the unknown
+// case, which is the safe choice — but a safe choice that is invisible is indistinguishable from the
+// defect it replaced. This line is what makes the gap BOUNDED rather than silent: an operator can
+// grep one word and answer "how many apps am I blind to, and which?".
+//
+// It rides the same cadence as the F-OBS heartbeat rather than firing every 30 s, for the reason
+// noteDeadAppScan records: a per-scan line is 2880 lines/day of noise, and noise is what made the
+// original author choose silence. The population only changes when someone starts or stops an app.
+func noteUnknownIntentSuppressions(logger *log.Logger, states []notify.AppRunState) {
+	if logger == nil || deadAppHeartbeatEvery <= 0 || deadAppScans%deadAppHeartbeatEvery != 0 {
+		return
+	}
+	var names []string
+	for _, s := range states {
+		if s.IntentUnknown {
+			names = append(names, s.Name)
+		}
+	}
+	if len(names) == 0 {
+		return
+	}
+	sort.Strings(names)
+	logger.Printf("[INFO] [deadapp] %d stopped app(s) have NO recorded customer intent, so their "+
+		"dead-app alarm is suppressed by the unknown-intent fallback (R-386): %s. This closes itself "+
+		"as each app is started or stopped through the interface.", len(names), strings.Join(names, ", "))
+}
+
 // bootReconcileSettle is the delay before the FIRST observation. It lets the initial scan, the first
 // status refresh and the two crash recoveries land before the R-52 sweep decides what "down" means.
 var bootReconcileSettle = 5 * time.Second
@@ -2169,9 +2200,58 @@ func classifyRunStates(sts []stacks.Stack, quiesced map[string]bool, failedResta
 		// brief restart never reaches it and the two suppression windows compose into one bounded
 		// delay. Quiesce suppression below still wins inside its own window.
 		crashLooping := st.CrashLooping(now)
-		userStopped := st.State == stacks.StateStopped && !failedRestart[st.Name]
+
+		// R-386: ASK THE FIELD THAT KNOWS, do not guess from the state.
+		//
+		// Until v0.223.0 this read `st.State == StateStopped && !failedRestart[...]` — i.e. EVERY
+		// stopped stack was assumed to be a deliberate customer stop. Measured on `demo-hp`
+		// 2026-08-23: `privatebin` stopped out of band, nine dead-app scans, zero events, zero
+		// banner. The comment above claimed an out-of-band stop "still alerts"; it did not, because
+		// `aggregateState` folds StateExited into the stopped counter and StateExited never survives
+		// aggregation.
+		//
+		// `DesiredState` is what the customer actually asked for, it has exactly ONE writer (the
+		// customer's own action — see the field's comment in internal/stacks/deploy.go), and it is
+		// TRI-state. The three-way ruling, operator-approved 2026-08-23:
+		//
+		//   Stopped → the customer asked        → no alarm  (unchanged)
+		//   Running → nobody asked              → ALARM     (the R-386 fix)
+		//   absent  → UNKNOWN, never "running"  → no alarm, AND say so (see IntentUnknown)
+		//
+		// The absent case keeps today's behaviour deliberately. Reading unknown as "nobody asked"
+		// would, on the first cycle after this ships, email about every app any owner has ever
+		// deliberately stopped — fleet-wide, from a field that predates the intent it is being asked
+		// about. That is the same over-correction the tri-state exists to prevent, and the backfill
+		// cannot help: it seeds Running only from an observed-UP reading, so anything stopped at
+		// upgrade time stays unknown — which is exactly the ambiguous population.
+		//
+		// The gap is BOUNDED AND NAMED, not silent: IntentUnknown is set, the caller logs it at INFO
+		// with the app name, and an operator can therefore answer "how many apps am I blind to?".
+		// It closes itself as apps are started and stopped through the interface.
+		//
+		// `failedRestart` still lifts a Stopped intent, and that ordering is load-bearing: the
+		// quiesce loop stops stacks by the same path a customer does, so a stack it stopped and could
+		// NOT restart must alarm whatever the intent says. Removing that term re-opens F-CRIT-1's
+		// indefinitely-silent dead app.
+		intentUnknown := false
+		userStopped := false
+		if st.State == stacks.StateStopped && !failedRestart[st.Name] {
+			switch stacks.DesiredStateOf(st) {
+			case stacks.DesiredStateStopped:
+				userStopped = true
+			case stacks.DesiredStateRunning:
+				userStopped = false
+			default: // DesiredStateUnknown — the §4 fallback
+				userStopped = true
+				intentUnknown = true
+			}
+		}
+
 		down := (stacks.IsDownState(st.State) || crashLooping) && !userStopped && !quiesced[st.Name]
-		states = append(states, notify.AppRunState{Name: st.Name, DisplayName: st.Meta.DisplayName, Down: down})
+		states = append(states, notify.AppRunState{
+			Name: st.Name, DisplayName: st.Meta.DisplayName, Down: down,
+			IntentUnknown: intentUnknown && !down,
+		})
 		if down {
 			dead = append(dead, web.DeadApp{Name: st.Name, DisplayName: st.Meta.DisplayName, State: string(st.State)})
 		}

controller/cmd/controller/r386_intent_test.go

@@ -0,0 +1,270 @@
+package main
+
+import (
+	"bytes"
+	"go/ast"
+	"go/parser"
+	"go/token"
+	"log"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
+	"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
+)
+
+// R-386 — the dead-app classifier asked the STATE whether the customer wanted an app stopped, when
+// the product already records the ANSWER.
+//
+// THE DEFECT. `userStopped` was `st.State == StateStopped && !failedRestart[...]` — every stopped
+// stack was assumed to be a deliberate customer stop. Measured live on `demo-hp` 2026-08-23:
+// `privatebin` stopped out of band, nine dead-app scans over four minutes, **zero events and zero
+// banner lines**, against a positive control from the same box seventeen minutes earlier. The comment
+// beside the code claimed an out-of-band stop "still alerts". It did not.
+//
+// THE LAYER. These sit at `classifyRunStates`, which is the single derivation point for "this app is
+// down" and the only place the guess was made. `classifyRunStates` is pure, so the fixture is exact
+// rather than a race against docker.
+//
+// RED-PROOF (observed, see REPORT.md): restore the pre-fix predicate
+//
+//	userStopped := st.State == stacks.StateStopped && !failedRestart[st.Name]
+//
+// and TestR386_OutOfBandStopWithRunningIntentAlarms fails with `dead-app banner = [], want privatebin`
+// — which is exactly what the live box reported.
+
+func stoppedStack(name, intent string) stacks.Stack {
+	st := stacks.Stack{Name: name, Deployed: true, State: stacks.StateStopped}
+	if intent != stacks.DesiredStateUnknown {
+		st.AppConfig = &stacks.AppConfig{DesiredState: intent}
+	}
+	return st
+}
+
+// Scenario D — R-386's measured case. Intent says Running; something stopped it anyway.
+func TestR386_OutOfBandStopWithRunningIntentAlarms(t *testing.T) {
+	dead, states := classifyRunStates(
+		[]stacks.Stack{stoppedStack("privatebin", stacks.DesiredStateRunning)}, nil, nil, time.Now())
+
+	if len(dead) != 1 || dead[0].Name != "privatebin" {
+		t.Fatalf("dead-app banner = %+v, want exactly privatebin — nobody asked for this app to be "+
+			"stopped, so its being stopped is a fault (measured silent on demo-hp 2026-08-23)", dead)
+	}
+	if !states[0].Down {
+		t.Fatalf("run state = %+v, want Down=true (this is what NotifyAppStartFailures reads)", states[0])
+	}
+	if states[0].IntentUnknown {
+		t.Errorf("IntentUnknown set for an app with a RECORDED intent — the log line would name an " +
+			"app that is not actually ambiguous")
+	}
+}
+
+// Scenario C — the customer pressed Stop. Telling them their own action was a fault is the thing
+// v0.164.0 was written to stop, and R-386 must not undo it.
+func TestR386_CustomerStopStaysSilent(t *testing.T) {
+	dead, states := classifyRunStates(
+		[]stacks.Stack{stoppedStack("docmost", stacks.DesiredStateStopped)}, nil, nil, time.Now())
+
+	if len(dead) != 0 {
+		t.Fatalf("a customer's own Stop raised an alarm: %+v", dead)
+	}
+	if states[0].Down {
+		t.Fatalf("run state = %+v, want Down=false — the customer asked for this", states[0])
+	}
+	if states[0].IntentUnknown {
+		t.Errorf("IntentUnknown set for an app whose intent is RECORDED as stopped")
+	}
+}
+
+// Scenario E — intent absent. §4's ruling: keep today's behaviour, and SAY SO.
+func TestR386_AbsentIntentSuppressesButIsAnnounced(t *testing.T) {
+	dead, states := classifyRunStates(
+		[]stacks.Stack{stoppedStack("legacy-app", stacks.DesiredStateUnknown)}, nil, nil, time.Now())
+
+	if len(dead) != 0 {
+		t.Fatalf("an app with NO recorded intent alarmed: %+v — on the first cycle after upgrade "+
+			"that is every app any owner ever deliberately stopped, fleet-wide", dead)
+	}
+	if states[0].Down {
+		t.Fatalf("run state = %+v, want Down=false", states[0])
+	}
+	// THE HALF THAT MAKES THE GAP BOUNDED RATHER THAN SILENT.
+	if !states[0].IntentUnknown {
+		t.Fatalf("IntentUnknown = false — the suppression happened but nothing records it, so an " +
+			"operator cannot answer 'how many apps am I blind to?'. A rule without a mechanism is " +
+			"not a rule")
+	}
+}
+
+// And the log line itself — the mechanism §4 demands, asserted as an OBSERVABLE, not as "the
+// function ran". The heartbeat cadence is deliberate; drive the counter to a firing scan.
+func TestR386_UnknownIntentIsLoggedWithTheAppName(t *testing.T) {
+	var buf bytes.Buffer
+	logger := log.New(&buf, "", 0)
+
+	states := []notify.AppRunState{
+		{Name: "zulip", IntentUnknown: true},
+		{Name: "legacy-app", IntentUnknown: true},
+		{Name: "docmost"}, // recorded intent — must NOT appear
+	}
+
+	saved := deadAppScans
+	t.Cleanup(func() { deadAppScans = saved })
+
+	// A non-firing scan says nothing (the 2880-lines/day lesson).
+	deadAppScans = 1
+	noteUnknownIntentSuppressions(logger, states)
+	if buf.Len() != 0 {
+		t.Fatalf("logged on a non-heartbeat scan: %q", buf.String())
+	}
+
+	// A firing scan names every ambiguous app, and only those.
+	deadAppScans = deadAppHeartbeatEvery
+	noteUnknownIntentSuppressions(logger, states)
+	out := buf.String()
+	if !strings.Contains(out, "[INFO]") {
+		t.Errorf("not logged at INFO — it must survive a default logging.level box: %q", out)
+	}
+	for _, want := range []string{"zulip", "legacy-app", "2 stopped app(s)", "R-386"} {
+		if !strings.Contains(out, want) {
+			t.Errorf("log line %q does not contain %q", out, want)
+		}
+	}
+	if strings.Contains(out, "docmost") {
+		t.Errorf("log line names an app with a RECORDED intent: %q", out)
+	}
+
+	// Nothing ambiguous → nothing said.
+	buf.Reset()
+	noteUnknownIntentSuppressions(logger, []notify.AppRunState{{Name: "docmost"}})
+	if buf.Len() != 0 {
+		t.Errorf("logged with no ambiguous apps: %q", buf.String())
+	}
+}
+
+// Scenario F — F-CRIT-1 must not re-open. The quiesce loop stops stacks by the same path a customer
+// does, so a stack it stopped and could NOT restart must alarm WHATEVER the intent says — including
+// when the intent is `Stopped`, which is the case that would silently swallow it.
+func TestR386_FailedRestartStillLiftsAStoppedIntent(t *testing.T) {
+	for _, intent := range []string{
+		stacks.DesiredStateStopped, stacks.DesiredStateRunning, stacks.DesiredStateUnknown,
+	} {
+		name := intent
+		if name == "" {
+			name = "(absent)"
+		}
+		t.Run(name, func(t *testing.T) {
+			dead, states := classifyRunStates(
+				[]stacks.Stack{stoppedStack("bookstack", intent)},
+				nil, map[string]bool{"bookstack": true}, time.Now())
+
+			if len(dead) != 1 {
+				t.Fatalf("intent %q: a stack the quiesce loop stopped and FAILED to restart did not "+
+					"alarm — this is F-CRIT-1's indefinitely-silent dead app, back through R-386's "+
+					"door: %+v", intent, dead)
+			}
+			if !states[0].Down {
+				t.Fatalf("intent %q: run state = %+v, want Down=true", intent, states[0])
+			}
+		})
+	}
+}
+
+// The quiesce suppression is unchanged and still wins inside its own window, for every intent.
+func TestR386_QuiesceSuppressionIsUntouched(t *testing.T) {
+	for _, intent := range []string{stacks.DesiredStateRunning, stacks.DesiredStateStopped} {
+		dead, states := classifyRunStates(
+			[]stacks.Stack{stoppedStack("bookstack", intent)},
+			map[string]bool{"bookstack": true}, nil, time.Now())
+		if len(dead) != 0 || states[0].Down {
+			t.Fatalf("intent %q: a quiesced stack alarmed (%+v) — R-97b's exact defect: the customer "+
+				"told their app broke during an outage the backup caused", intent, dead)
+		}
+	}
+}
+
+// A NON-stopped down state is unaffected by intent. A `degraded` stack (R-384) alarms regardless —
+// intent only ever governed the StateStopped whitelist, and widening it would silence R-384.
+func TestR386_IntentDoesNotReachNonStoppedDownStates(t *testing.T) {
+	st := stacks.Stack{
+		Name: "bookstack", Deployed: true, State: stacks.StateDegraded,
+		AppConfig: &stacks.AppConfig{DesiredState: stacks.DesiredStateStopped},
+	}
+	dead, states := classifyRunStates([]stacks.Stack{st}, nil, nil, time.Now())
+	if len(dead) != 1 || !states[0].Down {
+		t.Fatalf("a degraded stack was silenced by a Stopped intent (%+v) — R-386 must not reach "+
+			"past the StateStopped whitelist, or it undoes R-384", dead)
+	}
+}
+
+// --- production wiring (§10) --------------------------------------------------------------------
+//
+// A correct classifier the scheduler does not call is R-106's defect. This walks the AST of
+// main.go and proves the real job wires BOTH halves: the classification and the announcement.
+func TestR386_TheSchedulerWiresBothHalves(t *testing.T) {
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, "main.go", nil, 0)
+	if err != nil {
+		t.Fatalf("parse main.go: %v", err)
+	}
+
+	var scan, announce, desiredRead bool
+	ast.Inspect(f, func(n ast.Node) bool {
+		call, ok := n.(*ast.CallExpr)
+		if !ok {
+			return true
+		}
+		switch fn := call.Fun.(type) {
+		case *ast.Ident:
+			switch fn.Name {
+			case "scanDeployedAppRunStates":
+				scan = true
+			case "noteUnknownIntentSuppressions":
+				announce = true
+			}
+		case *ast.SelectorExpr:
+			// stacks.DesiredStateOf(st) — the intent read itself, inside classifyRunStates
+			if fn.Sel.Name == "DesiredStateOf" {
+				desiredRead = true
+			}
+		}
+		return true
+	})
+
+	if !scan {
+		t.Error("scanDeployedAppRunStates is never called from main.go — the detector is not wired")
+	}
+	if !announce {
+		t.Error("noteUnknownIntentSuppressions is never called from main.go — the unknown-intent gap " +
+			"is silent again, which is the half §4 required to make it bounded")
+	}
+	if !desiredRead {
+		t.Error("stacks.DesiredStateOf is never called from main.go — the classifier is back to " +
+			"guessing from the state (R-386)")
+	}
+}
+
+// THE FENCE (§12): DesiredState has exactly ONE writer — the customer's own action. This session
+// must not have added a second. Twelve of StopStack's fourteen callers are machines, so a writer in
+// the wrong place makes a nightly backup indistinguishable from the customer pressing Stop.
+func TestR386_NoNewDesiredStateWriterInMain(t *testing.T) {
+	fset := token.NewFileSet()
+	f, err := parser.ParseFile(fset, "main.go", nil, 0)
+	if err != nil {
+		t.Fatal(err)
+	}
+	ast.Inspect(f, func(n ast.Node) bool {
+		call, ok := n.(*ast.CallExpr)
+		if !ok {
+			return true
+		}
+		if sel, ok := call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "SetDesiredState" {
+			t.Errorf("%s: SetDesiredState is called from main.go — the field has ONE writer, the "+
+				"customer's own action. Reading it is fine; writing it here is the fenced act",
+				fset.Position(call.Pos()))
+		}
+		return true
+	})
+	_ = strings.TrimSpace
+}