F-OBS: the dead-app check gets a positive observable (v0.180.0)

deadapp-check had no observable at default info level: its per-cycle line goes
through Scheduler.dbg(), gated on logging.level==debug, so on a default box it is
never PRODUCED (not merely filtered) and cannot reach the always-DEBUG ring. A
30s interval also puts it on the scheduler's quiet path. 'No alarms' was
therefore indistinguishable from 'the detector never ran' — which undermines
confidence in the F-CRIT-1 fix in the field.

A periodic summary, not a line per run: at 30s a per-run line is 2880 lines/day,
which is why the original author chose silence. Every 20th scan (~10 min) emits
one INFO with the scan count, apps evaluated and apps down. A test pins the
cadence so it cannot be widened into uselessness.

Also corrects the 'unquiesce guaranteed by defer' comment — fault 10 established
the guarantee is the crash marker plus Recover().

controller/cmd/controller/main.go

@@ -468,6 +468,20 @@ func main() {
 	// quiet path (no ring spam — fix-6 friendly); the dashboard banner is state-based (self-clears) and
 	// the hub event fires once per running→down transition (notifier tracks it). A boot grace skips the
 	// controller's own startup settle so apps that legitimately take 30–60 s to come up don't alert.
+	//
+	// F-OBS (Campaign 8): this check had NO positive observable at default `info` level. Its per-cycle
+	// scheduler line is emitted through Scheduler.dbg(), which is gated on logging.level==debug and so
+	// is never PRODUCED on a default box (not merely filtered — the always-DEBUG ring cannot capture
+	// it either), and a 30 s interval also puts it on the scheduler's quiet path. So on every customer
+	// box, "no alarms" was indistinguishable from "the detector never ran" — the precise fallacy this
+	// project now has a standing rule against, and it directly undermines confidence in the F-CRIT-1
+	// fix in the field.
+	//
+	// The observable is a PERIODIC SUMMARY, not a line per run: at 30 s a per-run line is 2880
+	// lines/day of pure noise, which is what made the original author choose silence. Every
+	// deadAppHeartbeatEvery-th scan emits one INFO carrying the scan count and what it found, so an
+	// operator can always answer "is it running, and what does it see?" from a default box — and a
+	// STALLED detector is visible as the heartbeat stopping.
 	sched.Every("deadapp-check", 30*time.Second, func(ctx context.Context) error {
 		if time.Since(startTime) < deadAppBootGrace {
 			return nil // still inside the startup settle window
@@ -475,6 +489,8 @@ func main() {
 		dead, states := scanDeployedAppRunStates(stackMgr, quiesceLoop)
 		alertMgr.SetDeadAppAlerts(dead)
 		notifier.NotifyAppStartFailures(states)
+		deadAppScans++
+		noteDeadAppScan(logger, deadAppScans, len(states), len(dead))
 		return nil
 	})
 
@@ -1173,6 +1189,27 @@ func main() {
 // own boot. After the grace, an app that still isn't running alerts (the F11 dead-at-boot case).
 const deadAppBootGrace = 90 * time.Second
 
+// deadAppHeartbeatEvery is how many 30 s scans pass between deadapp heartbeat lines (F-OBS).
+// 20 scans = one line per ~10 minutes: frequent enough that a stalled detector is obvious well
+// inside the 180 s alarm grace it feeds, and 144 lines/day instead of 2880.
+const deadAppHeartbeatEvery = 20
+
+// deadAppScans counts completed deadapp scans since boot. Single-goroutine (the scheduler runs
+// each job serially), so it needs no lock.
+var deadAppScans int
+
+// noteDeadAppScan emits the F-OBS heartbeat every deadAppHeartbeatEvery-th scan, at [INFO] so it
+// survives a default `logging.level: info` box. Extracted from the job closure so a test can assert
+// the OBSERVABLE (the line, at info) rather than merely that the scan function ran — which is the
+// distinction F-OBS is about.
+func noteDeadAppScan(logger *log.Logger, scans, evaluated, down int) {
+	if logger == nil || deadAppHeartbeatEvery <= 0 || scans%deadAppHeartbeatEvery != 0 {
+		return
+	}
+	logger.Printf("[INFO] [deadapp] check alive: %d scans since boot, %d deployed app(s) evaluated, %d currently down",
+		scans, evaluated, down)
+}
+
 // bootReconcileSettle lets the initial scan, the first status refresh and the quiesce recovery
 // settle before the R-52 sweep decides what "down" means. 5 s + at most one 30 s retry gap keeps
 // the whole sweep inside deadAppBootGrace (90 s), which is what makes a successful recovery silent.