REPORT — F-OBS: the dead-app check gets a positive observable (v0.180.0) (2026-07-28)

Overwritten per the standing rule. Controller v0.179.0 → v0.180.0. One Campaign 8 finding, the smallest of the three in this task's scope — and the one the task marked droppable. It was not dropped, because it is what makes the other two fixes trustworthy in the field.

Companions this session: felhom-agent v0.107.0 → v0.109.0 (F-REBOOT) and felhom.eu host-install v1.21.0 (F-LEAK). No wire/contract change here; no agent change required.

Baselines (reconfirmed, not copied)

Controller 0.179.0 live on both boxes at start; agent 0.106.0; felhom-controller fb91c8d, felhom-agent af1c21a, felhom.eu d0cec9d, all clean.

The finding

On a default logging.level: info box there was no positive observable that deadapp-check had run. Its per-cycle scheduler line goes through Scheduler.dbg(), which is gated on logging.level == debug — so on a default box the line was never produced and therefore could not even reach the always-DEBUG ring. A 30 s interval also puts the job on the scheduler's quiet path.

So "no alarms" was indistinguishable from "the detector never ran". That is exactly the fallacy this project has a standing rule against (rule 3: an absent log line is not evidence of correct behaviour), and it directly undermines confidence in F-CRIT-1's fix — which is a fix to this same detector. Without an observable, the only field evidence that F-CRIT-1 works would be silence.

The fix

noteDeadAppScan(logger, scans, evaluated, down) — a pure, separately testable function emitting an INFO line every 20th scan (deadAppHeartbeatEvery), i.e. every 10 minutes at the job's 30 s cadence:

[INFO] [deadapp] check alive — N scans since boot, M evaluated, D currently down

Two things it deliberately does:

Carries what it saw, not that it ran. An operator needs to separate "running, everything up" from "running, 2 apps down". A bare liveness ping cannot express the second, and the second is the interesting state.
Summarises rather than floods. One line per run is 2880/day, which is precisely the pressure that made silence attractive to the original author — so a fix that floods is not a fix.

The function is pure and takes the logger as a parameter specifically so the emitted line can be asserted. The original mistake was invisible because it lived inline with nothing pinning its output.

Red-proofs — all observed failing

#	red-proof	observed failure
1	delete the `logger.Printf` (or the whole call from the job closure)	`no observable emitted at scan 20 — silence is indistinguishable from not running`
2	change the guard to `scans%1 != 0` (emit every run)	`emitted 60 lines across 60 scans — that is the flood that made silence attractive`

Two further tests are tripwires rather than red-proofs: the observable must be at INFO (a logging.level: info box would otherwise never see it) and the cadence must stay inside 15 minutes, so a stalled detector is obvious well within the 180 s alarm grace this check feeds. Off-cadence scans must stay quiet, and a nil logger must not panic — the job closure has no recovery.

go build, go vet ./..., go test ./... — green, run separately from the commit.

Live, on demo-hp

Endpoint-level/journal method (no browser on DooPlex, per the standing constraint):

[INFO] [deadapp] check alive — 20 scans since boot, 5 evaluated, 0 currently down

Present at INFO on a box whose logging.level is the default info — which is the whole point, and the thing the previous behaviour could not do. The same observable is what let F-CRIT-1's Scenario B replay earlier today assert "9 dead-app scans during the silent window" rather than merely "nothing was logged": the silence was proven to be suppression, not a dead detector.

demo-felhom received the same deploy with no injection and is quiet apart from its own heartbeat — the fix is not chattering on an untouched box.

The same shape, found in my own new code

Applying this finding's check to the agent's brand-new guest-power watchdog (shipped hours earlier in felhom-agent v0.107.0) showed it had exactly this defect: it logged at startup and when it acted, and was otherwise silent, so its health could only be inferred from absence. Fixed in agent v0.109.0 with the same pattern (INFO summary every 10th sweep, carrying what it saw, aborted sweeps not counted). Detail in felhom-agent/REPORT.md.

That is the argument for not dropping this part: the anti-pattern reproduces itself, and the fix is what makes it visible.

Deployed & fleet state

felhom-controller 0.180.0 on demo-hp (Up, healthy) and demo-felhom, both healthy. Cadences untouched. Agent 0.109.0 on both boxes.

NOT yet live-validated (carried forward)

Fault 4 — restic transport interruption; four injection approaches defeated by guest-bridged networking. Still the most valuable follow-up.
R-100 (new, filed today) — a restic offsite tier that fails nightly never goes stale on the hub: isStale reads only LastRun (hub/internal/monitor/offsite.go:120), which the controller writes unconditionally on failure (controller/internal/backup/offbox.go:716). F-CRIT-2's defect class, one layer up. Investigated, deliberately not fixed.
R-99 — prune never removes phantom snapshots (no retention bug; they accumulate).
contentionAlarmAfter (3h) — unit-tested with an injected clock, not live; a live proof needs a gate wedged for three hours.
Fault 12 (host reboot mid-backup); the three-way backup + restore-test + GC overlap; the agent's DR bring-up path (ModeDRGuestLoss), still read-only-verified.

5.8 KiB Raw Blame History