felhom-controller/REPORT.md

REPORT — v0.120.0: dead-app alerting (fix-3) + debug-ring revision (fix-6) — CLOSES CAMPAIGN-3

Date: 2026-07-12 · Version: controller v0.120.0 (from v0.119.0) · MinAgent: 0.81.0 (UNCHANGED) · Pairs with: hub v0.48.0 (accepts app_start_failed) · Deployed: guest 9201 (0.120.0 healthy). Source: felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md.

What shipped

• fix-3 (MED) — a dead deployed app is loud. A deadapp-check job (every 30 s, after a 90 s boot grace) scans the deployed apps: stopped/exited ones (stacks.IsDownState) raise a self-clearing WARN dashboard banner (grouped above 3) and fire an app_start_failed hub event ONCE per running→down transition (Notifier.NotifyAppStartFailures; the hub owns cooldown — no controller timer).
• fix-6 (MED) — the post-incident window survives. Ring cap 1000→5000 (display cap raised to match); periodic scheduler/refresh success lines demoted to a [TRACE] level the ring drops at write-time (failures never TRACE); atomic JSON-lines spill to <DataDir>/debug-ring.log (SSD only) every 30 s + on shutdown, loaded back on boot. Corruption-safe.
• hub v0.48.0: app_start_failed added to allowedEventTypes + customerMessages (else the event 400s at ingest — the known allowlist gotcha).

Tests + red-proofs (all green)

• fix-3: dead-app banner present + self-clears (companion: skip SetDeadAppAlerts → silent → fail); grouped above threshold; IsDownState table (stopped/exited down; starting/unhealthy/restarting/ deploying/paused/unknown NOT); notifier one-event-per-transition (companion: drop tracking → fires each cycle → fail); first-seen-down (dead-at-boot) fires; healthy never fires.
• fix-6: TRACE dropped from ring while a WARN failure is kept (companion: demote too broadly → failure lost → fail); spill→load round-trip; load keeps newest N; corrupt/truncated spill loads valid + never panics; missing file no-op.

Live validation (demo 9201 + hub)

• fix-3: docker stop seerr → within a cycle the dashboard banner "Telepített alkalmazás nem fut: Jellyseerr (stopped)" appeared AND the hub logged exactly ONE app_start_failed event (Event from demo-felhom: app_start_failed); across 3 down-cycles still ONE event (anti-spam); docker start seerr → the banner self-cleared (0 banners). This makes the campaign's 4-hour silent death impossible.
• fix-6: the ring held 0 periodic-spam lines (TRACE demotion); a controller restart PRESERVED the pre-restart window — oldest entry unchanged across systemctl restart (total 390→464, oldest 08:11:28 both sides), with a 63 KB spill on the persistent SSD data volume (survives container recreation, never a NAS path).

CAMPAIGN-3 finding ledger — RESOLVED

Finding	Fix	Ships in
F12 (CRIT) boot ordering cycle; F11/F10/F9 (HIGH) automount re-arm; F2/F1 residue	agent boot/recovery plane + appliance self-heal	agent v0.85.0
F7 (HIGH) in-place volume-dump truncation; F6/F5 (LOW) single-copy + stale dirs	atomic dumps + tier-2 for volume-only + stale sweep	controller v0.118.0
F8 (MED) storage-health contradiction; F4 (LOW) mapped_uid	share-row classifier fusion + range check	controller v0.119.0
fix-3 (MED) silent dead app; fix-6 (MED) ring window loss	dead-app alerting + ring cap/spill/spam	controller v0.120.0 (+ hub v0.48.0)

CAMPAIGN-3 is closed.

Standing follow-ups (explicit)

• Agent-ring persistence — the agent's own in-memory ring has the same restart-wipe gap; deferred (this task fixed the CONTROLLER ring only, to avoid an agent train).
• F13 (HIGH, from Task A) — an active nfs4 under the mp8 bind can fail PVE's rbind with rc255; deferred (needs a pre-start idle-unmount design or an idmapped nfs mount).
• Backup-locality option B — retarget NAS tier-1 to a local drive (operator chose A/keep locality).
• The agentless-on-proxmox2 cluster gap on Peti's box (roadmap).
• The publish train — agent 0.85 + controller 0.118/0.119/0.120 + hub 0.48 + MinAgent 0.81 + the journal-group one-liner + temp-creds deletion delivers this whole wave to Peti (on 0.113 / agent 0.81 today; reaches it at his next train).

Box state at wrap

controller 0.120.0 healthy on 9201; hub 0.48.0 live (Synced/Healthy); 8 apps healthy; seerr recovered; debug-ring spill on the SSD data volume; no test residue.