112 lines
5.8 KiB
Markdown
112 lines
5.8 KiB
Markdown
# REPORT — F-OBS: the dead-app check gets a positive observable (v0.180.0) (2026-07-28)
|
|
|
|
**Overwritten** per the standing rule. Controller **v0.179.0 → v0.180.0**. One Campaign 8 finding,
|
|
the smallest of the three in this task's scope — and the one the task marked droppable. It was not
|
|
dropped, because it is what makes the *other two* fixes trustworthy in the field.
|
|
|
|
Companions this session: `felhom-agent` v0.107.0 → v0.109.0 (F-REBOOT) and `felhom.eu` host-install
|
|
v1.21.0 (F-LEAK). No wire/contract change here; no agent change required.
|
|
|
|
## Baselines (reconfirmed, not copied)
|
|
Controller `0.179.0` live on both boxes at start; agent `0.106.0`; `felhom-controller fb91c8d`,
|
|
`felhom-agent af1c21a`, `felhom.eu d0cec9d`, all clean.
|
|
|
|
---
|
|
|
|
## The finding
|
|
|
|
On a default `logging.level: info` box there was **no positive observable that `deadapp-check` had
|
|
run**. Its per-cycle scheduler line goes through `Scheduler.dbg()`, which is gated on
|
|
`logging.level == debug` — so on a default box the line was never *produced* and therefore could not
|
|
even reach the always-DEBUG ring. A 30 s interval also puts the job on the scheduler's quiet path.
|
|
|
|
So **"no alarms" was indistinguishable from "the detector never ran"**. That is exactly the fallacy
|
|
this project has a standing rule against (rule 3: an absent log line is not evidence of correct
|
|
behaviour), and it directly undermines confidence in F-CRIT-1's fix — which is a fix to *this same
|
|
detector*. Without an observable, the only field evidence that F-CRIT-1 works would be silence.
|
|
|
|
## The fix
|
|
|
|
`noteDeadAppScan(logger, scans, evaluated, down)` — a pure, separately testable function emitting an
|
|
**INFO** line every **20th** scan (`deadAppHeartbeatEvery`), i.e. every 10 minutes at the job's 30 s
|
|
cadence:
|
|
|
|
```
|
|
[INFO] [deadapp] check alive — N scans since boot, M evaluated, D currently down
|
|
```
|
|
|
|
Two things it deliberately does:
|
|
|
|
- **Carries what it saw, not that it ran.** An operator needs to separate "running, everything up"
|
|
from "running, 2 apps down". A bare liveness ping cannot express the second, and the second is the
|
|
interesting state.
|
|
- **Summarises rather than floods.** One line per run is 2880/day, which is precisely the pressure
|
|
that made silence attractive to the original author — so a fix that floods is not a fix.
|
|
|
|
The function is pure and takes the logger as a parameter specifically so the *emitted line* can be
|
|
asserted. The original mistake was invisible because it lived inline with nothing pinning its output.
|
|
|
|
## Red-proofs — all observed failing
|
|
|
|
| # | red-proof | observed failure |
|
|
|---|---|---|
|
|
| 1 | delete the `logger.Printf` (or the whole call from the job closure) | `no observable emitted at scan 20 — silence is indistinguishable from not running` |
|
|
| 2 | change the guard to `scans%1 != 0` (emit every run) | `emitted 60 lines across 60 scans — that is the flood that made silence attractive` |
|
|
|
|
Two further tests are tripwires rather than red-proofs: the observable must be at **INFO** (a
|
|
`logging.level: info` box would otherwise never see it) and the cadence must stay inside 15 minutes,
|
|
so a stalled detector is obvious well within the 180 s alarm grace this check feeds. Off-cadence scans
|
|
must stay quiet, and a nil logger must not panic — the job closure has no recovery.
|
|
|
|
`go build`, `go vet ./...`, `go test ./...` — green, run separately from the commit.
|
|
|
|
---
|
|
|
|
## Live, on demo-hp
|
|
|
|
Endpoint-level/journal method (no browser on DooPlex, per the standing constraint):
|
|
|
|
```
|
|
[INFO] [deadapp] check alive — 20 scans since boot, 5 evaluated, 0 currently down
|
|
```
|
|
|
|
Present at **INFO** on a box whose `logging.level` is the default `info` — which is the whole point,
|
|
and the thing the previous behaviour could not do. The same observable is what let F-CRIT-1's Scenario
|
|
B replay earlier today assert *"9 dead-app scans during the silent window"* rather than merely
|
|
*"nothing was logged"*: the silence was proven to be suppression, not a dead detector.
|
|
|
|
**demo-felhom** received the same deploy with no injection and is quiet apart from its own heartbeat —
|
|
the fix is not chattering on an untouched box.
|
|
|
|
---
|
|
|
|
## The same shape, found in my own new code
|
|
|
|
Applying this finding's check to the agent's brand-new guest-power watchdog (shipped hours earlier in
|
|
`felhom-agent` v0.107.0) showed it had **exactly this defect**: it logged at startup and when it
|
|
*acted*, and was otherwise silent, so its health could only be inferred from absence. Fixed in agent
|
|
**v0.109.0** with the same pattern (INFO summary every 10th sweep, carrying what it saw, aborted sweeps
|
|
not counted). Detail in `felhom-agent/REPORT.md`.
|
|
|
|
That is the argument for not dropping this part: the anti-pattern reproduces itself, and the fix is
|
|
what makes it visible.
|
|
|
|
---
|
|
|
|
## Deployed & fleet state
|
|
`felhom-controller` **0.180.0** on **demo-hp** (`Up, healthy`) and **demo-felhom**, both healthy.
|
|
Cadences untouched. Agent **0.109.0** on both boxes.
|
|
|
|
## NOT yet live-validated (carried forward)
|
|
- **Fault 4** — restic transport interruption; four injection approaches defeated by guest-bridged
|
|
networking. Still the most valuable follow-up.
|
|
- **R-100** (new, filed today) — a restic offsite tier that fails nightly never goes stale on the hub:
|
|
`isStale` reads only `LastRun` (`hub/internal/monitor/offsite.go:120`), which the controller writes
|
|
**unconditionally** on failure (`controller/internal/backup/offbox.go:716`). F-CRIT-2's defect class,
|
|
one layer up. Investigated, deliberately **not** fixed.
|
|
- **R-99** — prune never removes phantom snapshots (no retention bug; they accumulate).
|
|
- `contentionAlarmAfter` (3h) — unit-tested with an injected clock, **not** live; a live proof needs a
|
|
gate wedged for three hours.
|
|
- Fault 12 (host reboot mid-backup); the three-way backup + restore-test + GC overlap; the agent's DR
|
|
bring-up path (`ModeDRGuestLoss`), still read-only-verified.
|