F-OBS: the dead-app check gets a positive observable (v0.180.0)

deadapp-check had no observable at default info level: its per-cycle line goes
through Scheduler.dbg(), gated on logging.level==debug, so on a default box it is
never PRODUCED (not merely filtered) and cannot reach the always-DEBUG ring. A
30s interval also puts it on the scheduler's quiet path. 'No alarms' was
therefore indistinguishable from 'the detector never ran' — which undermines
confidence in the F-CRIT-1 fix in the field.

A periodic summary, not a line per run: at 30s a per-run line is 2880 lines/day,
which is why the original author chose silence. Every 20th scan (~10 min) emits
one INFO with the scan count, apps evaluated and apps down. A test pins the
cadence so it cannot be widened into uselessness.

Also corrects the 'unquiesce guaranteed by defer' comment — fault 10 established
the guarantee is the crash marker plus Recover().
This commit is contained in:
2026-07-28 10:27:28 +02:00
parent a63409c843
commit fb91c8d766
4 changed files with 171 additions and 1 deletions
+32
View File
@@ -1,5 +1,37 @@
## Changelog
### v0.180.0 — F-OBS: the dead-app check gets a positive observable (2026-07-28)
On a default `logging.level: info` box there was **no way to tell whether `deadapp-check` had run**.
Its per-cycle scheduler line goes through `Scheduler.dbg()`, which is gated on `s.debug` — so on an
info-level box the line is never *produced*, not merely filtered, and therefore cannot reach the
always-DEBUG ring either. A 30 s interval also puts the job on the scheduler's quiet path
(`quiet := job.Interval <= 30*time.Second`).
So "no alarms" was indistinguishable from "the detector never ran" — the exact fallacy this project
now has a standing rule against, and it directly undermines confidence in the **F-CRIT-1** fix in the
field: that fix's whole value is that a genuinely dead app now alarms, and an operator had no way to
confirm the thing that alarms is alive.
**A periodic summary, not a line per run.** At 30 s a per-run line is 2880 lines/day, which is
precisely why the original author chose silence — so a fix that floods is not a fix. Every 20th scan
(≈10 minutes) emits one INFO carrying the scan count, how many deployed apps were evaluated, and how
many are currently down. An operator can answer "is it running, and what does it see?" from a default
box, and a STALLED detector shows up as the heartbeat stopping.
10 minutes is chosen to stay useful as a liveness signal: it is well inside the 180 s alarm grace this
check feeds, and a test pins the cadence so nobody can widen it to hours and quietly make the
observable useless again.
### Also
Corrected the comment claiming the quiesce unquiesce is "guaranteed by defer". Campaign 8 fault 10
established that a SIGKILL runs no deferred function — the guarantee is the crash MARKER plus
`Recover()`, which brought the stacks back 1 s after restart. The `defer` covers only the graceful
exits.
Files: `cmd/controller/main.go`, `internal/quiesce/quiesce.go` (comment),
`cmd/controller/deadapp_observable_test.go` (new).
### v0.179.0 — F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (2026-07-28)
Both Campaign 8 findings live in `internal/quiesce` and its `classifyRunStates` consumer, and both