From 92d670e8a65e94c8c44ad1a25e01c41bb5c8d27a Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 12 Jul 2026 10:31:12 +0200 Subject: [PATCH] =?UTF-8?q?controller=20v0.120.0:=20dead-app=20alerting=20?= =?UTF-8?q?(fix-3)=20+=20ring=20revision=20(fix-6)=20=E2=80=94=20CLOSES=20?= =?UTF-8?q?CAMPAIGN-3=20(docs+CHANGELOG+REPORT+CONTEXT)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fix-3: deadapp-check job -> self-clearing WARN banner + one app_start_failed hub event per running->down transition. fix-6: ring 1000->5000, periodic spam->TRACE (ring-dropped), atomic SSD spill/load across restart. Live-validated on 9201+hub 0.48. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf --- CHANGELOG.md | 33 +++++++++++++++ CONTEXT.md | 16 ++++++- REPORT.md | 99 ++++++++++++++++++++++++-------------------- controller/README.md | 28 ++++++++++++- 4 files changed, 129 insertions(+), 47 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 832f59c..01ab61f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,38 @@ ## Changelog +### v0.120.0 — dead-app alerting (fix-3) + debug-ring revision (fix-6) — CLOSES CAMPAIGN-3 (2026-07-12) — MinAgent: 0.81.0 + +The last CAMPAIGN-3 findings (`felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md`). MinAgent +UNCHANGED (0.81.0). Pairs with hub v0.48.0 (accepts the new `app_start_failed` event). + +- **fix-3 (MED) — a dead deployed app is LOUD, not silent.** The campaign's CWA sat dead 4 h with no + signal; F11 then produced 4 silently-dead NAS apps per reboot. A new `deadapp-check` job (every 30 s) + scans `stackMgr.GetStacks()`: a DEPLOYED app whose containers are `stopped`/`exited` + (`stacks.IsDownState`; `created`/`dead` map to `stopped` — the F11 dead-at-boot case) raises a + state-based WARN dashboard banner ("Telepített alkalmazás nem fut: "; grouped above 3 to survive + a reboot storm) that SELF-CLEARS the moment the app runs again, AND fires an `app_start_failed` hub + event ONCE per running→down transition (`Notifier.NotifyAppStartFailures` tracks per-app state — the + hub owns the real cooldown; the controller adds no timer and does not spam). A 90 s boot grace skips + the controller's own startup settle so apps that legitimately take 30–60 s to come up don't + false-alarm; after the grace an app that never came up STILL fires (the whole point). +- **fix-6 (MED) — the post-incident window survives.** The 1000-entry ring wrapped in ~6.5 min under + the campaign's load and died on every restart. Three changes: **(a) cap 1000→5000** (viewer + + `Entries`/handler display cap raised to match — a larger ring is useless if unreadable); **(b) + periodic-noise policy** — the every-cycle scheduler "job finished" + `refreshStatusLocked` success + lines are demoted to a new `[TRACE]` level the ring DROPS at write-time (failures/transitions are + never TRACE, so nothing is lost); **(c) spill persistence** — `LogBuffer.SpillTo`/`LoadFrom` + atomically (tmp+rename, JSON-lines) spill the ring to `/debug-ring.log` on the SSD state dir + (NEVER a NAS path) every 30 s and on clean shutdown, loading it back on boot so a restart / container + recreation preserves the pre-restart window. Corruption-safe (a truncated line is skipped, never + fatal). +- **Live-validated (demo 9201 + hub):** fix-3 — `docker stop seerr` → the dashboard banner + "Telepített alkalmazás nem fut: Jellyseerr (stopped)" appeared AND the hub received exactly ONE + `app_start_failed` event across 3 down-cycles (anti-spam); `docker start` → banner self-cleared. + fix-6 — the ring showed 0 periodic-spam lines; a controller restart PRESERVED the pre-restart window + (oldest entry unchanged across the restart; 63 KB spill on the persistent SSD volume). Tests incl. + the fix-3 silent-regression + one-event-per-transition red-proofs, the fix-6 TRACE-drop-keeps-failure + + corrupt-spill-safe red-proofs, all green. + ### v0.119.0 — storage-health coherence (F8) + mapped_uid validation (F4) (2026-07-12) — MinAgent: 0.81.0 Fixes CAMPAIGN-3 (`felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md`) storage-UI findings. diff --git a/CONTEXT.md b/CONTEXT.md index c83f6a3..0062390 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,21 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-07-12 (v0.119.0 — storage-health coherence F8 + mapped_uid F4; MinAgent 0.81.0 unchanged) +Last updated: 2026-07-12 (v0.120.0 — dead-app alerting fix-3 + ring revision fix-6; CLOSES CAMPAIGN-3) + +> **2026-07-12 — v0.120.0: fix-3 + fix-6 → CAMPAIGN-3 CLOSED (LIVE on 9201 + hub 0.48.0). +> MinAgent 0.81.0 unchanged.** **fix-3:** a `deadapp-check` job (30s, 90s boot grace) flags a DEPLOYED +> app in stopped/exited state (`stacks.IsDownState`) → self-clearing WARN dashboard banner + one +> `app_start_failed` hub event per running→down transition (`Notifier.NotifyAppStartFailures`, in-memory +> tracker, hub owns cooldown). **fix-6:** ring cap 1000→5000 (display cap raised too); periodic +> scheduler/refresh success lines → `[TRACE]` (ring drops at write-time, failures never TRACE); atomic +> JSON-lines spill to `/debug-ring.log` (SSD, survives recreate) every 30s + shutdown, loaded +> on boot. **hub v0.48.0** accepts `app_start_failed` (allowlist + customerMessages). LIVE: docker stop +> seerr → banner + ONE hub event across 3 cycles (anti-spam) → docker start → banner self-cleared; ring +> 0 spam lines + restart PRESERVED the pre-restart window (oldest unchanged, 63KB spill on SSD volume). +> **CAMPAIGN-3 CLOSED** (F12/F11/F10/F9/F2/F1→agent 0.85; F7/F6/F5→0.118; F8/F4→0.119; fix-3/6→0.120). +> Follow-ups: agent-ring persistence; F13 (active-nfs-mp8 rc255); publish train (agent 0.85 + ctrl +> 0.118/0.119/0.120 + hub 0.48) to Peti. Seams: deadapp scanDeployedAppRunStates, notify.pushFn. > **2026-07-12 — v0.119.0: STORAGE-HEALTH COHERENCE (LIVE on 9201). MinAgent 0.81.0 unchanged.** Fixes > CAMPAIGN-3 F8+F4. **F8 (MED):** the share row's health came only from the agent's server-level TCP diff --git a/REPORT.md b/REPORT.md index 18a4012..df5a45d 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,59 +1,68 @@ -# REPORT — v0.119.0: storage-health coherence (F8) + mapped_uid validation (F4) +# REPORT — v0.120.0: dead-app alerting (fix-3) + debug-ring revision (fix-6) — CLOSES CAMPAIGN-3 -**Date:** 2026-07-12 · **Version:** controller v0.119.0 (from v0.118.0) · **MinAgent:** 0.81.0 (UNCHANGED) -· **Deployed:** guest 9201 (`0.119.0` healthy) · **Source:** `felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md`. - -## §3 design fork — decision - -Took the recommended **option B (controller-only)**: the share row reuses the shipped v0.117.0 -consuming-namespace classifier (`system.ClassifyPathFS`) — the exact ground truth the stacks-page stub -badge already reads. No agent change, no new probing surface, and by construction the row and the -stacks badge can never disagree (single source). Option A (agent-side export-level probe) was not built. +**Date:** 2026-07-12 · **Version:** controller v0.120.0 (from v0.119.0) · **MinAgent:** 0.81.0 (UNCHANGED) +· **Pairs with:** hub v0.48.0 (accepts `app_start_failed`) · **Deployed:** guest 9201 (`0.120.0` healthy). +**Source:** `felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md`. ## What shipped -- **F8 (MED) — one classification, two surfaces.** `networkStorageItems` now fuses the agent's health - with the namespace classification via `fuseNetHealth`: a new `stub` state overrides a benign idle/ok - when the consuming namespace sees local disk at `Where`; a whole-server `unreachable` still wins over - stub; autofs-healthy / network / `unknown` leave the agent health intact (no manufactured fault, no - force-mount). Row badge for `stub` = "Hibás — az alkalmazások nem a NAS-t látják". The share row and - the stacks/dashboard badge now derive from ONE classifier. -- **F4 (LOW) — mapped_uid/gid range check at the door.** `handleNetStorageAdd` validates the container - uid/gid (1..65533) after the `<=0` default, before the job — out of range → friendly Hungarian 400, - nothing installed. Catches the campaign's `101000` (a host-side mapped value) that used to leak a raw - `agent_error`. +- **fix-3 (MED) — a dead deployed app is loud.** A `deadapp-check` job (every 30 s, after a 90 s boot + grace) scans the deployed apps: `stopped`/`exited` ones (`stacks.IsDownState`) raise a self-clearing + WARN dashboard banner (grouped above 3) and fire an `app_start_failed` hub event ONCE per running→down + transition (`Notifier.NotifyAppStartFailures`; the hub owns cooldown — no controller timer). +- **fix-6 (MED) — the post-incident window survives.** Ring cap 1000→5000 (display cap raised to + match); periodic scheduler/refresh success lines demoted to a `[TRACE]` level the ring drops at + write-time (failures never TRACE); atomic JSON-lines spill to `/debug-ring.log` (SSD only) + every 30 s + on shutdown, loaded back on boot. Corruption-safe. +- **hub v0.48.0:** `app_start_failed` added to `allowedEventTypes` + `customerMessages` (else the event + 400s at ingest — the known allowlist gotcha). ## Tests + red-proofs (all green) -- F8 fusion table: idle+stub→stub (the contradiction resolved), ok+stub→stub, **idle+autofs→idle** - (the over-eager autofs=stub mutant fails here), unreachable+stub→unreachable (server wins), - idle+unknown→idle (no manufactured fault). End-to-end `networkStorageItems` stub fusion (companion: - drop the fuse call → row shows raw agent health → fail). -- F4: uid 101000 → 400 + friendly message, agent never reached (companion: drop the check → reaches the - agent → fail); 65534 → 400; 1000 / 65533 / 0-defaults pass the range check. +- fix-3: dead-app banner present + self-clears (companion: skip SetDeadAppAlerts → silent → fail); + grouped above threshold; `IsDownState` table (stopped/exited down; starting/unhealthy/restarting/ + deploying/paused/unknown NOT); notifier one-event-per-transition (companion: drop tracking → fires + each cycle → fail); first-seen-down (dead-at-boot) fires; healthy never fires. +- fix-6: TRACE dropped from ring while a WARN failure is kept (companion: demote too broadly → failure + lost → fail); spill→load round-trip; load keeps newest N; corrupt/truncated spill loads valid + never + panics; missing file no-op. -## Live validation (demo 9201, sim-NAS rails — exportfs only) +## Live validation (demo 9201 + hub) -- **F8 the contradiction, killed:** baseline healthy → row `ok`, no stub badge. `exportfs -u` while idle - + drop the mount → the SHARE ROW showed `health=stub` ("Hibás — az alkalmazások nem a NAS-t látják") - AND the stacks page showed the stub badge (4) — the two surfaces AGREE (previously: row "Készenlét" + - stacks stub = contradiction). `reachable:true` throughout (the server-level dial is still green — the - exact F8 blindness, now correctly overridden). Re-export → row cleared back to `ok`/"Elérhető" - (healthy idle NOT downgraded — the regression). -- **F4:** `mapped_uid:101000` → 400 + the friendly message, registry unchanged (no `c5uid`), no host - unit/dir residue; `mapped_uid:1000` → 200, passed the range check (then failed later at the - unreachable probe as designed, rolled back clean). +- **fix-3:** `docker stop seerr` → within a cycle the dashboard banner "Telepített alkalmazás nem fut: + Jellyseerr (stopped)" appeared AND the hub logged exactly ONE `app_start_failed` event + (`Event from demo-felhom: app_start_failed`); across 3 down-cycles still ONE event (anti-spam); + `docker start seerr` → the banner self-cleared (0 banners). This makes the campaign's 4-hour silent + death impossible. +- **fix-6:** the ring held 0 periodic-spam lines (TRACE demotion); a controller restart PRESERVED the + pre-restart window — oldest entry unchanged across `systemctl restart` (total 390→464, oldest + 08:11:28 both sides), with a 63 KB spill on the persistent SSD data volume (survives container + recreation, never a NAS path). -## NOT live-validated / standing items +## CAMPAIGN-3 finding ledger — RESOLVED -- `unreachable`-wins live (a genuine server-down IP on a registered share) — unit-tested only; the F8 - live proof used the export-level cut (the actual finding). -- Task D remains queued: fix-3 boot-time app-start-failure alerting + the ring wrap/count revision - (fix-6 6.5-min horizon under load). -- Peti's box (controller 0.113 / agentless-on-proxmox2) reaches 0.119 (+0.85/0.118) at his next train. -- No publish/floor movement; agent untouched (MinAgent 0.81.0). +| Finding | Fix | Ships in | +|---|---|---| +| F12 (CRIT) boot ordering cycle; F11/F10/F9 (HIGH) automount re-arm; F2/F1 residue | agent boot/recovery plane + appliance self-heal | **agent v0.85.0** | +| F7 (HIGH) in-place volume-dump truncation; F6/F5 (LOW) single-copy + stale dirs | atomic dumps + tier-2 for volume-only + stale sweep | **controller v0.118.0** | +| F8 (MED) storage-health contradiction; F4 (LOW) mapped_uid | share-row classifier fusion + range check | **controller v0.119.0** | +| fix-3 (MED) silent dead app; fix-6 (MED) ring window loss | dead-app alerting + ring cap/spill/spam | **controller v0.120.0** (+ hub v0.48.0) | + +CAMPAIGN-3 is closed. + +## Standing follow-ups (explicit) + +- **Agent-ring persistence** — the agent's own in-memory ring has the same restart-wipe gap; deferred + (this task fixed the CONTROLLER ring only, to avoid an agent train). +- **F13 (HIGH, from Task A)** — an active nfs4 under the mp8 bind can fail PVE's rbind with rc255; + deferred (needs a pre-start idle-unmount design or an idmapped nfs mount). +- **Backup-locality option B** — retarget NAS tier-1 to a local drive (operator chose A/keep locality). +- **The agentless-on-proxmox2 cluster gap** on Peti's box (roadmap). +- **The publish train** — agent 0.85 + controller 0.118/0.119/0.120 + hub 0.48 + MinAgent 0.81 + the + journal-group one-liner + temp-creds deletion delivers this whole wave to Peti (on 0.113 / agent 0.81 + today; reaches it at his next train). ## Box state at wrap -controller 0.119.0 healthy on 9201; nas-media healthy + `ok`; registry = nas-media only (no test -residue); all NAS apps healthy; NFS re-exported. +controller 0.120.0 healthy on 9201; hub 0.48.0 live (Synced/Healthy); 8 apps healthy; seerr recovered; +debug-ring spill on the SSD data volume; no test residue. diff --git a/controller/README.md b/controller/README.md index ed9cee1..6cc103f 100644 --- a/controller/README.md +++ b/controller/README.md @@ -1301,11 +1301,23 @@ The controller pushes structured events to the Hub's `/api/v1/event` endpoint. T | `controller_updated` | info/error | Self-update success or failure | | `app_deployed` | info | New app deployed via API | | `app_removed` | info | App removed via API | +| `app_start_failed` | warn | A DEPLOYED app is not running (fix-3) — fired ONCE per running→down transition | | `disaster_recovery_started` | warning | DR restore begins | | `disaster_recovery_completed` | info/error | DR restore finishes (success/partial) | Each event carries typed detail structs (e.g., `BackupDetails`, `DiskDetails`, `HealthDetails`) serialized as JSON. +**Deployed-app-down alerting (fix-3, v0.120.0, CAMPAIGN-3).** A `deadapp-check` scheduler job (every +30 s, after a 90 s boot grace) scans `stackMgr.GetStacks()`: a DEPLOYED app whose containers are +`stopped`/`exited` (`stacks.IsDownState` — a Docker `created`/`dead` container, the F11 dead-at-boot +case, resolves to `stopped`) gets a **state-based WARN dashboard banner** ("Telepített alkalmazás nem +fut: ", grouped above 3 so a reboot storm doesn't wall the dashboard) that self-clears when the +app runs again, AND an `app_start_failed` hub event fired **once per running→down transition** +(`Notifier.NotifyAppStartFailures` tracks per-app state; down→down cycles are silent — the hub owns +the real cooldown, the controller adds no timer). The boot grace prevents false alarms during the +controller's own startup while STILL firing for an app that never came up. This closes the campaign's +4-hour silent CWA death. + #### Default Enabled Events Events the customer receives notifications for (configurable in settings): @@ -1874,11 +1886,25 @@ When `logging.level: "debug"` is set in `controller.yaml`, the controller expose | — | Telemetria teszt | `GET /api/debug/telemetry` | Run the full telemetry collection pipeline on-demand (metrics query + log scan). Returns per-app table: container list, memory current/avg/peak, CPU avg, catalog limit, log error/warning counts, and top issues. Useful for verifying container→stack mapping and testing log scanner patterns without waiting for the 15-minute report cycle. | | 6 | Önfrissítés teszt | `POST /api/debug/selfupdate/dry-run` | Dry-run update check: current vs new image lines, compose writability, backup state. | | 7 | DR / Telepítő varázsló | `POST /api/debug/dr/trigger-setup`, `GET /api/debug/dr/infra-status` | Infra backup status per drive. Trigger setup mode via marker file (requires "RESET" + infra backup pre-check). | -| 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 1000 entries), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). | +| 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 5000 entries, spill-persisted across restart — fix-6), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). | #### Key Implementation Details - **Log buffer** (`internal/web/logbuffer.go`): Ring buffer implementing `io.Writer`. Since v0.116.0 it ALWAYS exists (any `logging.level`) and captures every line INCLUDING `[DEBUG]`: the logger is `io.MultiWriter(LevelFilterWriter(os.Stdout, logging.level), logBuffer)` — stdout/docker-logs keep respecting `logging.level`, the ring holds the full detail for remote diagnostics. `logBuffer.Lines(maxBytes)` renders the newest-kept plain-text tail (the report `controller_log_tail` source). New leveled lines use `internal/logx` (`Debugf/Infof/Warnf/Errorf`); legacy `isDebug()` call sites are unchanged. + - **Ring sizing, spill persistence, periodic-noise policy (fix-6, v0.120.0, CAMPAIGN-3).** The + campaign measured the 1000-entry ring wrapping in ~6.5 min under load and dying on every restart — + the exact post-incident window was the first thing lost. Three changes: **(a) cap 1000→5000** + (`Entries`/the debug handler display cap raised to match — a larger ring is useless if the viewer + can't request more than 1000 of it; the Naplóviewer default pull is 1000). **(b) periodic-noise + policy:** a periodic job's ROUTINE success is not ring-worthy — the every-cycle scheduler "job + finished" line and `refreshStatusLocked` per-cycle enumeration are logged at a new `[TRACE]` level + that the ring DROPS at write-time (`levelPriority("TRACE") < DEBUG`). Failures and state changes are + never TRACE, so nothing load-bearing is lost; this was the biggest ring filler. **(c) spill + persistence:** `LogBuffer.SpillTo`/`LoadFrom` atomically (tmp+rename, JSON-lines) spill the ring to + `/debug-ring.log` on the **SSD state dir** (the persistent data volume that survives + container recreation — NEVER a NAS/HDD path) every 30 s and on clean shutdown, and load it back on + boot so a restart / recreate preserves the pre-restart window. Corruption-safe: a truncated/partial + line is skipped on load, never fatal. - **Controller self-log pull** (`internal/report/selftail.go`): the hub's report ACK may carry `controller_log_requested` — the NEXT report ships `controller_log_tail` (ring, 128 KB cap, consume-once, the v0.111.0 app-tail pattern; additive fields, app-tail wire unchanged). Serving a pull logs the customer-visible `operator log pull served` INFO line. - **Storage simulation**: `simulatedPaths` map in watchdog prevents the watchdog from re-probing simulated-disconnected paths. Disconnect runs all real steps except `lazyUnmount` (drive stays physically mounted). - **DR trigger safety**: Uses marker file (`data/.needs-setup`) instead of modifying controller.yaml. Pre-checks that infra backup exists on at least one drive.