F-OBS: the dead-app check gets a positive observable (v0.180.0)
deadapp-check had no observable at default info level: its per-cycle line goes through Scheduler.dbg(), gated on logging.level==debug, so on a default box it is never PRODUCED (not merely filtered) and cannot reach the always-DEBUG ring. A 30s interval also puts it on the scheduler's quiet path. 'No alarms' was therefore indistinguishable from 'the detector never ran' — which undermines confidence in the F-CRIT-1 fix in the field. A periodic summary, not a line per run: at 30s a per-run line is 2880 lines/day, which is why the original author chose silence. Every 20th scan (~10 min) emits one INFO with the scan count, apps evaluated and apps down. A test pins the cadence so it cannot be widened into uselessness. Also corrects the 'unquiesce guaranteed by defer' comment — fault 10 established the guarantee is the crash marker plus Recover().
This commit is contained in:
@@ -1,5 +1,37 @@
|
||||
## Changelog
|
||||
|
||||
### v0.180.0 — F-OBS: the dead-app check gets a positive observable (2026-07-28)
|
||||
|
||||
On a default `logging.level: info` box there was **no way to tell whether `deadapp-check` had run**.
|
||||
Its per-cycle scheduler line goes through `Scheduler.dbg()`, which is gated on `s.debug` — so on an
|
||||
info-level box the line is never *produced*, not merely filtered, and therefore cannot reach the
|
||||
always-DEBUG ring either. A 30 s interval also puts the job on the scheduler's quiet path
|
||||
(`quiet := job.Interval <= 30*time.Second`).
|
||||
|
||||
So "no alarms" was indistinguishable from "the detector never ran" — the exact fallacy this project
|
||||
now has a standing rule against, and it directly undermines confidence in the **F-CRIT-1** fix in the
|
||||
field: that fix's whole value is that a genuinely dead app now alarms, and an operator had no way to
|
||||
confirm the thing that alarms is alive.
|
||||
|
||||
**A periodic summary, not a line per run.** At 30 s a per-run line is 2880 lines/day, which is
|
||||
precisely why the original author chose silence — so a fix that floods is not a fix. Every 20th scan
|
||||
(≈10 minutes) emits one INFO carrying the scan count, how many deployed apps were evaluated, and how
|
||||
many are currently down. An operator can answer "is it running, and what does it see?" from a default
|
||||
box, and a STALLED detector shows up as the heartbeat stopping.
|
||||
|
||||
10 minutes is chosen to stay useful as a liveness signal: it is well inside the 180 s alarm grace this
|
||||
check feeds, and a test pins the cadence so nobody can widen it to hours and quietly make the
|
||||
observable useless again.
|
||||
|
||||
### Also
|
||||
Corrected the comment claiming the quiesce unquiesce is "guaranteed by defer". Campaign 8 fault 10
|
||||
established that a SIGKILL runs no deferred function — the guarantee is the crash MARKER plus
|
||||
`Recover()`, which brought the stacks back 1 s after restart. The `defer` covers only the graceful
|
||||
exits.
|
||||
|
||||
Files: `cmd/controller/main.go`, `internal/quiesce/quiesce.go` (comment),
|
||||
`cmd/controller/deadapp_observable_test.go` (new).
|
||||
|
||||
### v0.179.0 — F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (2026-07-28)
|
||||
|
||||
Both Campaign 8 findings live in `internal/quiesce` and its `classifyRunStates` consumer, and both
|
||||
|
||||
Reference in New Issue
Block a user