# REPORT — F-OBS: the dead-app check gets a positive observable (v0.180.0) (2026-07-28) **Overwritten** per the standing rule. Controller **v0.179.0 → v0.180.0**. One Campaign 8 finding, the smallest of the three in this task's scope — and the one the task marked droppable. It was not dropped, because it is what makes the *other two* fixes trustworthy in the field. Companions this session: `felhom-agent` v0.107.0 → v0.109.0 (F-REBOOT) and `felhom.eu` host-install v1.21.0 (F-LEAK). No wire/contract change here; no agent change required. ## Baselines (reconfirmed, not copied) Controller `0.179.0` live on both boxes at start; agent `0.106.0`; `felhom-controller fb91c8d`, `felhom-agent af1c21a`, `felhom.eu d0cec9d`, all clean. --- ## The finding On a default `logging.level: info` box there was **no positive observable that `deadapp-check` had run**. Its per-cycle scheduler line goes through `Scheduler.dbg()`, which is gated on `logging.level == debug` — so on a default box the line was never *produced* and therefore could not even reach the always-DEBUG ring. A 30 s interval also puts the job on the scheduler's quiet path. So **"no alarms" was indistinguishable from "the detector never ran"**. That is exactly the fallacy this project has a standing rule against (rule 3: an absent log line is not evidence of correct behaviour), and it directly undermines confidence in F-CRIT-1's fix — which is a fix to *this same detector*. Without an observable, the only field evidence that F-CRIT-1 works would be silence. ## The fix `noteDeadAppScan(logger, scans, evaluated, down)` — a pure, separately testable function emitting an **INFO** line every **20th** scan (`deadAppHeartbeatEvery`), i.e. every 10 minutes at the job's 30 s cadence: ``` [INFO] [deadapp] check alive — N scans since boot, M evaluated, D currently down ``` Two things it deliberately does: - **Carries what it saw, not that it ran.** An operator needs to separate "running, everything up" from "running, 2 apps down". A bare liveness ping cannot express the second, and the second is the interesting state. - **Summarises rather than floods.** One line per run is 2880/day, which is precisely the pressure that made silence attractive to the original author — so a fix that floods is not a fix. The function is pure and takes the logger as a parameter specifically so the *emitted line* can be asserted. The original mistake was invisible because it lived inline with nothing pinning its output. ## Red-proofs — all observed failing | # | red-proof | observed failure | |---|---|---| | 1 | delete the `logger.Printf` (or the whole call from the job closure) | `no observable emitted at scan 20 — silence is indistinguishable from not running` | | 2 | change the guard to `scans%1 != 0` (emit every run) | `emitted 60 lines across 60 scans — that is the flood that made silence attractive` | Two further tests are tripwires rather than red-proofs: the observable must be at **INFO** (a `logging.level: info` box would otherwise never see it) and the cadence must stay inside 15 minutes, so a stalled detector is obvious well within the 180 s alarm grace this check feeds. Off-cadence scans must stay quiet, and a nil logger must not panic — the job closure has no recovery. `go build`, `go vet ./...`, `go test ./...` — green, run separately from the commit. --- ## Live, on demo-hp Endpoint-level/journal method (no browser on DooPlex, per the standing constraint): ``` [INFO] [deadapp] check alive — 20 scans since boot, 5 evaluated, 0 currently down ``` Present at **INFO** on a box whose `logging.level` is the default `info` — which is the whole point, and the thing the previous behaviour could not do. The same observable is what let F-CRIT-1's Scenario B replay earlier today assert *"9 dead-app scans during the silent window"* rather than merely *"nothing was logged"*: the silence was proven to be suppression, not a dead detector. **demo-felhom** received the same deploy with no injection and is quiet apart from its own heartbeat — the fix is not chattering on an untouched box. --- ## The same shape, found in my own new code Applying this finding's check to the agent's brand-new guest-power watchdog (shipped hours earlier in `felhom-agent` v0.107.0) showed it had **exactly this defect**: it logged at startup and when it *acted*, and was otherwise silent, so its health could only be inferred from absence. Fixed in agent **v0.109.0** with the same pattern (INFO summary every 10th sweep, carrying what it saw, aborted sweeps not counted). Detail in `felhom-agent/REPORT.md`. That is the argument for not dropping this part: the anti-pattern reproduces itself, and the fix is what makes it visible. --- ## Deployed & fleet state `felhom-controller` **0.180.0** on **demo-hp** (`Up, healthy`) and **demo-felhom**, both healthy. Cadences untouched. Agent **0.109.0** on both boxes. ## NOT yet live-validated (carried forward) - **Fault 4** — restic transport interruption; four injection approaches defeated by guest-bridged networking. Still the most valuable follow-up. - **R-100** (new, filed today) — a restic offsite tier that fails nightly never goes stale on the hub: `isStale` reads only `LastRun` (`hub/internal/monitor/offsite.go:120`), which the controller writes **unconditionally** on failure (`controller/internal/backup/offbox.go:716`). F-CRIT-2's defect class, one layer up. Investigated, deliberately **not** fixed. - **R-99** — prune never removes phantom snapshots (no retention bug; they accumulate). - `contentionAlarmAfter` (3h) — unit-tested with an injected clock, **not** live; a live proof needs a gate wedged for three hours. - Fault 12 (host reboot mid-backup); the three-way backup + restore-test + GC overlap; the agent's DR bring-up path (`ModeDRGuestLoss`), still read-only-verified.