controller v0.120.0: dead-app alerting (fix-3) + ring revision (fix-6) — CLOSES CAMPAIGN-3 (docs+CHANGELOG+REPORT+CONTEXT)
fix-3: deadapp-check job -> self-clearing WARN banner + one app_start_failed hub event per running->down transition. fix-6: ring 1000->5000, periodic spam->TRACE (ring-dropped), atomic SSD spill/load across restart. Live-validated on 9201+hub 0.48. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
This commit is contained in:
+27
-1
@@ -1301,11 +1301,23 @@ The controller pushes structured events to the Hub's `/api/v1/event` endpoint. T
|
||||
| `controller_updated` | info/error | Self-update success or failure |
|
||||
| `app_deployed` | info | New app deployed via API |
|
||||
| `app_removed` | info | App removed via API |
|
||||
| `app_start_failed` | warn | A DEPLOYED app is not running (fix-3) — fired ONCE per running→down transition |
|
||||
| `disaster_recovery_started` | warning | DR restore begins |
|
||||
| `disaster_recovery_completed` | info/error | DR restore finishes (success/partial) |
|
||||
|
||||
Each event carries typed detail structs (e.g., `BackupDetails`, `DiskDetails`, `HealthDetails`) serialized as JSON.
|
||||
|
||||
**Deployed-app-down alerting (fix-3, v0.120.0, CAMPAIGN-3).** A `deadapp-check` scheduler job (every
|
||||
30 s, after a 90 s boot grace) scans `stackMgr.GetStacks()`: a DEPLOYED app whose containers are
|
||||
`stopped`/`exited` (`stacks.IsDownState` — a Docker `created`/`dead` container, the F11 dead-at-boot
|
||||
case, resolves to `stopped`) gets a **state-based WARN dashboard banner** ("Telepített alkalmazás nem
|
||||
fut: <app>", grouped above 3 so a reboot storm doesn't wall the dashboard) that self-clears when the
|
||||
app runs again, AND an `app_start_failed` hub event fired **once per running→down transition**
|
||||
(`Notifier.NotifyAppStartFailures` tracks per-app state; down→down cycles are silent — the hub owns
|
||||
the real cooldown, the controller adds no timer). The boot grace prevents false alarms during the
|
||||
controller's own startup while STILL firing for an app that never came up. This closes the campaign's
|
||||
4-hour silent CWA death.
|
||||
|
||||
#### Default Enabled Events
|
||||
|
||||
Events the customer receives notifications for (configurable in settings):
|
||||
@@ -1874,11 +1886,25 @@ When `logging.level: "debug"` is set in `controller.yaml`, the controller expose
|
||||
| — | Telemetria teszt | `GET /api/debug/telemetry` | Run the full telemetry collection pipeline on-demand (metrics query + log scan). Returns per-app table: container list, memory current/avg/peak, CPU avg, catalog limit, log error/warning counts, and top issues. Useful for verifying container→stack mapping and testing log scanner patterns without waiting for the 15-minute report cycle. |
|
||||
| 6 | Önfrissítés teszt | `POST /api/debug/selfupdate/dry-run` | Dry-run update check: current vs new image lines, compose writability, backup state. |
|
||||
| 7 | DR / Telepítő varázsló | `POST /api/debug/dr/trigger-setup`, `GET /api/debug/dr/infra-status` | Infra backup status per drive. Trigger setup mode via marker file (requires "RESET" + infra backup pre-check). |
|
||||
| 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 1000 entries), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). |
|
||||
| 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 5000 entries, spill-persisted across restart — fix-6), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). |
|
||||
|
||||
#### Key Implementation Details
|
||||
|
||||
- **Log buffer** (`internal/web/logbuffer.go`): Ring buffer implementing `io.Writer`. Since v0.116.0 it ALWAYS exists (any `logging.level`) and captures every line INCLUDING `[DEBUG]`: the logger is `io.MultiWriter(LevelFilterWriter(os.Stdout, logging.level), logBuffer)` — stdout/docker-logs keep respecting `logging.level`, the ring holds the full detail for remote diagnostics. `logBuffer.Lines(maxBytes)` renders the newest-kept plain-text tail (the report `controller_log_tail` source). New leveled lines use `internal/logx` (`Debugf/Infof/Warnf/Errorf`); legacy `isDebug()` call sites are unchanged.
|
||||
- **Ring sizing, spill persistence, periodic-noise policy (fix-6, v0.120.0, CAMPAIGN-3).** The
|
||||
campaign measured the 1000-entry ring wrapping in ~6.5 min under load and dying on every restart —
|
||||
the exact post-incident window was the first thing lost. Three changes: **(a) cap 1000→5000**
|
||||
(`Entries`/the debug handler display cap raised to match — a larger ring is useless if the viewer
|
||||
can't request more than 1000 of it; the Naplóviewer default pull is 1000). **(b) periodic-noise
|
||||
policy:** a periodic job's ROUTINE success is not ring-worthy — the every-cycle scheduler "job
|
||||
finished" line and `refreshStatusLocked` per-cycle enumeration are logged at a new `[TRACE]` level
|
||||
that the ring DROPS at write-time (`levelPriority("TRACE") < DEBUG`). Failures and state changes are
|
||||
never TRACE, so nothing load-bearing is lost; this was the biggest ring filler. **(c) spill
|
||||
persistence:** `LogBuffer.SpillTo`/`LoadFrom` atomically (tmp+rename, JSON-lines) spill the ring to
|
||||
`<DataDir>/debug-ring.log` on the **SSD state dir** (the persistent data volume that survives
|
||||
container recreation — NEVER a NAS/HDD path) every 30 s and on clean shutdown, and load it back on
|
||||
boot so a restart / recreate preserves the pre-restart window. Corruption-safe: a truncated/partial
|
||||
line is skipped on load, never fatal.
|
||||
- **Controller self-log pull** (`internal/report/selftail.go`): the hub's report ACK may carry `controller_log_requested` — the NEXT report ships `controller_log_tail` (ring, 128 KB cap, consume-once, the v0.111.0 app-tail pattern; additive fields, app-tail wire unchanged). Serving a pull logs the customer-visible `operator log pull served` INFO line.
|
||||
- **Storage simulation**: `simulatedPaths` map in watchdog prevents the watchdog from re-probing simulated-disconnected paths. Disconnect runs all real steps except `lazyUnmount` (drive stays physically mounted).
|
||||
- **DR trigger safety**: Uses marker file (`data/.needs-setup`) instead of modifying controller.yaml. Pre-checks that infra backup exists on at least one drive.
|
||||
|
||||
Reference in New Issue
Block a user