controller v0.120.0: dead-app alerting (fix-3) + ring revision (fix-6) — CLOSES CAMPAIGN-3 (docs+CHANGELOG+REPORT+CONTEXT)

fix-3: deadapp-check job -> self-clearing WARN banner + one app_start_failed hub
event per running->down transition. fix-6: ring 1000->5000, periodic spam->TRACE
(ring-dropped), atomic SSD spill/load across restart. Live-validated on 9201+hub 0.48.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
This commit is contained in:
2026-07-12 10:31:12 +02:00
parent 40498254c6
commit 92d670e8a6
4 changed files with 129 additions and 47 deletions
+27 -1
View File
@@ -1301,11 +1301,23 @@ The controller pushes structured events to the Hub's `/api/v1/event` endpoint. T
| `controller_updated` | info/error | Self-update success or failure |
| `app_deployed` | info | New app deployed via API |
| `app_removed` | info | App removed via API |
| `app_start_failed` | warn | A DEPLOYED app is not running (fix-3) — fired ONCE per running→down transition |
| `disaster_recovery_started` | warning | DR restore begins |
| `disaster_recovery_completed` | info/error | DR restore finishes (success/partial) |
Each event carries typed detail structs (e.g., `BackupDetails`, `DiskDetails`, `HealthDetails`) serialized as JSON.
**Deployed-app-down alerting (fix-3, v0.120.0, CAMPAIGN-3).** A `deadapp-check` scheduler job (every
30 s, after a 90 s boot grace) scans `stackMgr.GetStacks()`: a DEPLOYED app whose containers are
`stopped`/`exited` (`stacks.IsDownState` — a Docker `created`/`dead` container, the F11 dead-at-boot
case, resolves to `stopped`) gets a **state-based WARN dashboard banner** ("Telepített alkalmazás nem
fut: <app>", grouped above 3 so a reboot storm doesn't wall the dashboard) that self-clears when the
app runs again, AND an `app_start_failed` hub event fired **once per running→down transition**
(`Notifier.NotifyAppStartFailures` tracks per-app state; down→down cycles are silent — the hub owns
the real cooldown, the controller adds no timer). The boot grace prevents false alarms during the
controller's own startup while STILL firing for an app that never came up. This closes the campaign's
4-hour silent CWA death.
#### Default Enabled Events
Events the customer receives notifications for (configurable in settings):
@@ -1874,11 +1886,25 @@ When `logging.level: "debug"` is set in `controller.yaml`, the controller expose
| — | Telemetria teszt | `GET /api/debug/telemetry` | Run the full telemetry collection pipeline on-demand (metrics query + log scan). Returns per-app table: container list, memory current/avg/peak, CPU avg, catalog limit, log error/warning counts, and top issues. Useful for verifying container→stack mapping and testing log scanner patterns without waiting for the 15-minute report cycle. |
| 6 | Önfrissítés teszt | `POST /api/debug/selfupdate/dry-run` | Dry-run update check: current vs new image lines, compose writability, backup state. |
| 7 | DR / Telepítő varázsló | `POST /api/debug/dr/trigger-setup`, `GET /api/debug/dr/infra-status` | Infra backup status per drive. Trigger setup mode via marker file (requires "RESET" + infra backup pre-check). |
| 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 1000 entries), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). |
| 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 5000 entries, spill-persisted across restart — fix-6), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). |
#### Key Implementation Details
- **Log buffer** (`internal/web/logbuffer.go`): Ring buffer implementing `io.Writer`. Since v0.116.0 it ALWAYS exists (any `logging.level`) and captures every line INCLUDING `[DEBUG]`: the logger is `io.MultiWriter(LevelFilterWriter(os.Stdout, logging.level), logBuffer)` — stdout/docker-logs keep respecting `logging.level`, the ring holds the full detail for remote diagnostics. `logBuffer.Lines(maxBytes)` renders the newest-kept plain-text tail (the report `controller_log_tail` source). New leveled lines use `internal/logx` (`Debugf/Infof/Warnf/Errorf`); legacy `isDebug()` call sites are unchanged.
- **Ring sizing, spill persistence, periodic-noise policy (fix-6, v0.120.0, CAMPAIGN-3).** The
campaign measured the 1000-entry ring wrapping in ~6.5 min under load and dying on every restart —
the exact post-incident window was the first thing lost. Three changes: **(a) cap 1000→5000**
(`Entries`/the debug handler display cap raised to match — a larger ring is useless if the viewer
can't request more than 1000 of it; the Naplóviewer default pull is 1000). **(b) periodic-noise
policy:** a periodic job's ROUTINE success is not ring-worthy — the every-cycle scheduler "job
finished" line and `refreshStatusLocked` per-cycle enumeration are logged at a new `[TRACE]` level
that the ring DROPS at write-time (`levelPriority("TRACE") < DEBUG`). Failures and state changes are
never TRACE, so nothing load-bearing is lost; this was the biggest ring filler. **(c) spill
persistence:** `LogBuffer.SpillTo`/`LoadFrom` atomically (tmp+rename, JSON-lines) spill the ring to
`<DataDir>/debug-ring.log` on the **SSD state dir** (the persistent data volume that survives
container recreation — NEVER a NAS/HDD path) every 30 s and on clean shutdown, and load it back on
boot so a restart / recreate preserves the pre-restart window. Corruption-safe: a truncated/partial
line is skipped on load, never fatal.
- **Controller self-log pull** (`internal/report/selftail.go`): the hub's report ACK may carry `controller_log_requested` — the NEXT report ships `controller_log_tail` (ring, 128 KB cap, consume-once, the v0.111.0 app-tail pattern; additive fields, app-tail wire unchanged). Serving a pull logs the customer-visible `operator log pull served` INFO line.
- **Storage simulation**: `simulatedPaths` map in watchdog prevents the watchdog from re-probing simulated-disconnected paths. Disconnect runs all real steps except `lazyUnmount` (drive stays physically mounted).
- **DR trigger safety**: Uses marker file (`data/.needs-setup`) instead of modifying controller.yaml. Pre-checks that infra backup exists on at least one drive.