controller v0.120.0: dead-app alerting (fix-3) + ring revision (fix-6) — CLOSES CAMPAIGN-3 (docs+CHANGELOG+REPORT+CONTEXT)
fix-3: deadapp-check job -> self-clearing WARN banner + one app_start_failed hub event per running->down transition. fix-6: ring 1000->5000, periodic spam->TRACE (ring-dropped), atomic SSD spill/load across restart. Live-validated on 9201+hub 0.48. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017CDMFpFx84pfviCTVuGGhf
This commit is contained in:
@@ -1,5 +1,38 @@
|
||||
## Changelog
|
||||
|
||||
### v0.120.0 — dead-app alerting (fix-3) + debug-ring revision (fix-6) — CLOSES CAMPAIGN-3 (2026-07-12) — MinAgent: 0.81.0
|
||||
|
||||
The last CAMPAIGN-3 findings (`felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md`). MinAgent
|
||||
UNCHANGED (0.81.0). Pairs with hub v0.48.0 (accepts the new `app_start_failed` event).
|
||||
|
||||
- **fix-3 (MED) — a dead deployed app is LOUD, not silent.** The campaign's CWA sat dead 4 h with no
|
||||
signal; F11 then produced 4 silently-dead NAS apps per reboot. A new `deadapp-check` job (every 30 s)
|
||||
scans `stackMgr.GetStacks()`: a DEPLOYED app whose containers are `stopped`/`exited`
|
||||
(`stacks.IsDownState`; `created`/`dead` map to `stopped` — the F11 dead-at-boot case) raises a
|
||||
state-based WARN dashboard banner ("Telepített alkalmazás nem fut: <app>"; grouped above 3 to survive
|
||||
a reboot storm) that SELF-CLEARS the moment the app runs again, AND fires an `app_start_failed` hub
|
||||
event ONCE per running→down transition (`Notifier.NotifyAppStartFailures` tracks per-app state — the
|
||||
hub owns the real cooldown; the controller adds no timer and does not spam). A 90 s boot grace skips
|
||||
the controller's own startup settle so apps that legitimately take 30–60 s to come up don't
|
||||
false-alarm; after the grace an app that never came up STILL fires (the whole point).
|
||||
- **fix-6 (MED) — the post-incident window survives.** The 1000-entry ring wrapped in ~6.5 min under
|
||||
the campaign's load and died on every restart. Three changes: **(a) cap 1000→5000** (viewer +
|
||||
`Entries`/handler display cap raised to match — a larger ring is useless if unreadable); **(b)
|
||||
periodic-noise policy** — the every-cycle scheduler "job finished" + `refreshStatusLocked` success
|
||||
lines are demoted to a new `[TRACE]` level the ring DROPS at write-time (failures/transitions are
|
||||
never TRACE, so nothing is lost); **(c) spill persistence** — `LogBuffer.SpillTo`/`LoadFrom`
|
||||
atomically (tmp+rename, JSON-lines) spill the ring to `<DataDir>/debug-ring.log` on the SSD state dir
|
||||
(NEVER a NAS path) every 30 s and on clean shutdown, loading it back on boot so a restart / container
|
||||
recreation preserves the pre-restart window. Corruption-safe (a truncated line is skipped, never
|
||||
fatal).
|
||||
- **Live-validated (demo 9201 + hub):** fix-3 — `docker stop seerr` → the dashboard banner
|
||||
"Telepített alkalmazás nem fut: Jellyseerr (stopped)" appeared AND the hub received exactly ONE
|
||||
`app_start_failed` event across 3 down-cycles (anti-spam); `docker start` → banner self-cleared.
|
||||
fix-6 — the ring showed 0 periodic-spam lines; a controller restart PRESERVED the pre-restart window
|
||||
(oldest entry unchanged across the restart; 63 KB spill on the persistent SSD volume). Tests incl.
|
||||
the fix-3 silent-regression + one-event-per-transition red-proofs, the fix-6 TRACE-drop-keeps-failure
|
||||
+ corrupt-spill-safe red-proofs, all green.
|
||||
|
||||
### v0.119.0 — storage-health coherence (F8) + mapped_uid validation (F4) (2026-07-12) — MinAgent: 0.81.0
|
||||
|
||||
Fixes CAMPAIGN-3 (`felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md`) storage-UI findings.
|
||||
|
||||
+15
-1
@@ -7,7 +7,21 @@
|
||||
>
|
||||
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
|
||||
|
||||
Last updated: 2026-07-12 (v0.119.0 — storage-health coherence F8 + mapped_uid F4; MinAgent 0.81.0 unchanged)
|
||||
Last updated: 2026-07-12 (v0.120.0 — dead-app alerting fix-3 + ring revision fix-6; CLOSES CAMPAIGN-3)
|
||||
|
||||
> **2026-07-12 — v0.120.0: fix-3 + fix-6 → CAMPAIGN-3 CLOSED (LIVE on 9201 + hub 0.48.0).
|
||||
> MinAgent 0.81.0 unchanged.** **fix-3:** a `deadapp-check` job (30s, 90s boot grace) flags a DEPLOYED
|
||||
> app in stopped/exited state (`stacks.IsDownState`) → self-clearing WARN dashboard banner + one
|
||||
> `app_start_failed` hub event per running→down transition (`Notifier.NotifyAppStartFailures`, in-memory
|
||||
> tracker, hub owns cooldown). **fix-6:** ring cap 1000→5000 (display cap raised too); periodic
|
||||
> scheduler/refresh success lines → `[TRACE]` (ring drops at write-time, failures never TRACE); atomic
|
||||
> JSON-lines spill to `<DataDir>/debug-ring.log` (SSD, survives recreate) every 30s + shutdown, loaded
|
||||
> on boot. **hub v0.48.0** accepts `app_start_failed` (allowlist + customerMessages). LIVE: docker stop
|
||||
> seerr → banner + ONE hub event across 3 cycles (anti-spam) → docker start → banner self-cleared; ring
|
||||
> 0 spam lines + restart PRESERVED the pre-restart window (oldest unchanged, 63KB spill on SSD volume).
|
||||
> **CAMPAIGN-3 CLOSED** (F12/F11/F10/F9/F2/F1→agent 0.85; F7/F6/F5→0.118; F8/F4→0.119; fix-3/6→0.120).
|
||||
> Follow-ups: agent-ring persistence; F13 (active-nfs-mp8 rc255); publish train (agent 0.85 + ctrl
|
||||
> 0.118/0.119/0.120 + hub 0.48) to Peti. Seams: deadapp scanDeployedAppRunStates, notify.pushFn.
|
||||
|
||||
> **2026-07-12 — v0.119.0: STORAGE-HEALTH COHERENCE (LIVE on 9201). MinAgent 0.81.0 unchanged.** Fixes
|
||||
> CAMPAIGN-3 F8+F4. **F8 (MED):** the share row's health came only from the agent's server-level TCP
|
||||
|
||||
@@ -1,59 +1,68 @@
|
||||
# REPORT — v0.119.0: storage-health coherence (F8) + mapped_uid validation (F4)
|
||||
# REPORT — v0.120.0: dead-app alerting (fix-3) + debug-ring revision (fix-6) — CLOSES CAMPAIGN-3
|
||||
|
||||
**Date:** 2026-07-12 · **Version:** controller v0.119.0 (from v0.118.0) · **MinAgent:** 0.81.0 (UNCHANGED)
|
||||
· **Deployed:** guest 9201 (`0.119.0` healthy) · **Source:** `felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md`.
|
||||
|
||||
## §3 design fork — decision
|
||||
|
||||
Took the recommended **option B (controller-only)**: the share row reuses the shipped v0.117.0
|
||||
consuming-namespace classifier (`system.ClassifyPathFS`) — the exact ground truth the stacks-page stub
|
||||
badge already reads. No agent change, no new probing surface, and by construction the row and the
|
||||
stacks badge can never disagree (single source). Option A (agent-side export-level probe) was not built.
|
||||
**Date:** 2026-07-12 · **Version:** controller v0.120.0 (from v0.119.0) · **MinAgent:** 0.81.0 (UNCHANGED)
|
||||
· **Pairs with:** hub v0.48.0 (accepts `app_start_failed`) · **Deployed:** guest 9201 (`0.120.0` healthy).
|
||||
**Source:** `felhom.eu/documentation/audits/CAMPAIGN-3-2026-07-11.md`.
|
||||
|
||||
## What shipped
|
||||
|
||||
- **F8 (MED) — one classification, two surfaces.** `networkStorageItems` now fuses the agent's health
|
||||
with the namespace classification via `fuseNetHealth`: a new `stub` state overrides a benign idle/ok
|
||||
when the consuming namespace sees local disk at `Where`; a whole-server `unreachable` still wins over
|
||||
stub; autofs-healthy / network / `unknown` leave the agent health intact (no manufactured fault, no
|
||||
force-mount). Row badge for `stub` = "Hibás — az alkalmazások nem a NAS-t látják". The share row and
|
||||
the stacks/dashboard badge now derive from ONE classifier.
|
||||
- **F4 (LOW) — mapped_uid/gid range check at the door.** `handleNetStorageAdd` validates the container
|
||||
uid/gid (1..65533) after the `<=0` default, before the job — out of range → friendly Hungarian 400,
|
||||
nothing installed. Catches the campaign's `101000` (a host-side mapped value) that used to leak a raw
|
||||
`agent_error`.
|
||||
- **fix-3 (MED) — a dead deployed app is loud.** A `deadapp-check` job (every 30 s, after a 90 s boot
|
||||
grace) scans the deployed apps: `stopped`/`exited` ones (`stacks.IsDownState`) raise a self-clearing
|
||||
WARN dashboard banner (grouped above 3) and fire an `app_start_failed` hub event ONCE per running→down
|
||||
transition (`Notifier.NotifyAppStartFailures`; the hub owns cooldown — no controller timer).
|
||||
- **fix-6 (MED) — the post-incident window survives.** Ring cap 1000→5000 (display cap raised to
|
||||
match); periodic scheduler/refresh success lines demoted to a `[TRACE]` level the ring drops at
|
||||
write-time (failures never TRACE); atomic JSON-lines spill to `<DataDir>/debug-ring.log` (SSD only)
|
||||
every 30 s + on shutdown, loaded back on boot. Corruption-safe.
|
||||
- **hub v0.48.0:** `app_start_failed` added to `allowedEventTypes` + `customerMessages` (else the event
|
||||
400s at ingest — the known allowlist gotcha).
|
||||
|
||||
## Tests + red-proofs (all green)
|
||||
|
||||
- F8 fusion table: idle+stub→stub (the contradiction resolved), ok+stub→stub, **idle+autofs→idle**
|
||||
(the over-eager autofs=stub mutant fails here), unreachable+stub→unreachable (server wins),
|
||||
idle+unknown→idle (no manufactured fault). End-to-end `networkStorageItems` stub fusion (companion:
|
||||
drop the fuse call → row shows raw agent health → fail).
|
||||
- F4: uid 101000 → 400 + friendly message, agent never reached (companion: drop the check → reaches the
|
||||
agent → fail); 65534 → 400; 1000 / 65533 / 0-defaults pass the range check.
|
||||
- fix-3: dead-app banner present + self-clears (companion: skip SetDeadAppAlerts → silent → fail);
|
||||
grouped above threshold; `IsDownState` table (stopped/exited down; starting/unhealthy/restarting/
|
||||
deploying/paused/unknown NOT); notifier one-event-per-transition (companion: drop tracking → fires
|
||||
each cycle → fail); first-seen-down (dead-at-boot) fires; healthy never fires.
|
||||
- fix-6: TRACE dropped from ring while a WARN failure is kept (companion: demote too broadly → failure
|
||||
lost → fail); spill→load round-trip; load keeps newest N; corrupt/truncated spill loads valid + never
|
||||
panics; missing file no-op.
|
||||
|
||||
## Live validation (demo 9201, sim-NAS rails — exportfs only)
|
||||
## Live validation (demo 9201 + hub)
|
||||
|
||||
- **F8 the contradiction, killed:** baseline healthy → row `ok`, no stub badge. `exportfs -u` while idle
|
||||
+ drop the mount → the SHARE ROW showed `health=stub` ("Hibás — az alkalmazások nem a NAS-t látják")
|
||||
AND the stacks page showed the stub badge (4) — the two surfaces AGREE (previously: row "Készenlét" +
|
||||
stacks stub = contradiction). `reachable:true` throughout (the server-level dial is still green — the
|
||||
exact F8 blindness, now correctly overridden). Re-export → row cleared back to `ok`/"Elérhető"
|
||||
(healthy idle NOT downgraded — the regression).
|
||||
- **F4:** `mapped_uid:101000` → 400 + the friendly message, registry unchanged (no `c5uid`), no host
|
||||
unit/dir residue; `mapped_uid:1000` → 200, passed the range check (then failed later at the
|
||||
unreachable probe as designed, rolled back clean).
|
||||
- **fix-3:** `docker stop seerr` → within a cycle the dashboard banner "Telepített alkalmazás nem fut:
|
||||
Jellyseerr (stopped)" appeared AND the hub logged exactly ONE `app_start_failed` event
|
||||
(`Event from demo-felhom: app_start_failed`); across 3 down-cycles still ONE event (anti-spam);
|
||||
`docker start seerr` → the banner self-cleared (0 banners). This makes the campaign's 4-hour silent
|
||||
death impossible.
|
||||
- **fix-6:** the ring held 0 periodic-spam lines (TRACE demotion); a controller restart PRESERVED the
|
||||
pre-restart window — oldest entry unchanged across `systemctl restart` (total 390→464, oldest
|
||||
08:11:28 both sides), with a 63 KB spill on the persistent SSD data volume (survives container
|
||||
recreation, never a NAS path).
|
||||
|
||||
## NOT live-validated / standing items
|
||||
## CAMPAIGN-3 finding ledger — RESOLVED
|
||||
|
||||
- `unreachable`-wins live (a genuine server-down IP on a registered share) — unit-tested only; the F8
|
||||
live proof used the export-level cut (the actual finding).
|
||||
- Task D remains queued: fix-3 boot-time app-start-failure alerting + the ring wrap/count revision
|
||||
(fix-6 6.5-min horizon under load).
|
||||
- Peti's box (controller 0.113 / agentless-on-proxmox2) reaches 0.119 (+0.85/0.118) at his next train.
|
||||
- No publish/floor movement; agent untouched (MinAgent 0.81.0).
|
||||
| Finding | Fix | Ships in |
|
||||
|---|---|---|
|
||||
| F12 (CRIT) boot ordering cycle; F11/F10/F9 (HIGH) automount re-arm; F2/F1 residue | agent boot/recovery plane + appliance self-heal | **agent v0.85.0** |
|
||||
| F7 (HIGH) in-place volume-dump truncation; F6/F5 (LOW) single-copy + stale dirs | atomic dumps + tier-2 for volume-only + stale sweep | **controller v0.118.0** |
|
||||
| F8 (MED) storage-health contradiction; F4 (LOW) mapped_uid | share-row classifier fusion + range check | **controller v0.119.0** |
|
||||
| fix-3 (MED) silent dead app; fix-6 (MED) ring window loss | dead-app alerting + ring cap/spill/spam | **controller v0.120.0** (+ hub v0.48.0) |
|
||||
|
||||
CAMPAIGN-3 is closed.
|
||||
|
||||
## Standing follow-ups (explicit)
|
||||
|
||||
- **Agent-ring persistence** — the agent's own in-memory ring has the same restart-wipe gap; deferred
|
||||
(this task fixed the CONTROLLER ring only, to avoid an agent train).
|
||||
- **F13 (HIGH, from Task A)** — an active nfs4 under the mp8 bind can fail PVE's rbind with rc255;
|
||||
deferred (needs a pre-start idle-unmount design or an idmapped nfs mount).
|
||||
- **Backup-locality option B** — retarget NAS tier-1 to a local drive (operator chose A/keep locality).
|
||||
- **The agentless-on-proxmox2 cluster gap** on Peti's box (roadmap).
|
||||
- **The publish train** — agent 0.85 + controller 0.118/0.119/0.120 + hub 0.48 + MinAgent 0.81 + the
|
||||
journal-group one-liner + temp-creds deletion delivers this whole wave to Peti (on 0.113 / agent 0.81
|
||||
today; reaches it at his next train).
|
||||
|
||||
## Box state at wrap
|
||||
|
||||
controller 0.119.0 healthy on 9201; nas-media healthy + `ok`; registry = nas-media only (no test
|
||||
residue); all NAS apps healthy; NFS re-exported.
|
||||
controller 0.120.0 healthy on 9201; hub 0.48.0 live (Synced/Healthy); 8 apps healthy; seerr recovered;
|
||||
debug-ring spill on the SSD data volume; no test residue.
|
||||
|
||||
+27
-1
@@ -1301,11 +1301,23 @@ The controller pushes structured events to the Hub's `/api/v1/event` endpoint. T
|
||||
| `controller_updated` | info/error | Self-update success or failure |
|
||||
| `app_deployed` | info | New app deployed via API |
|
||||
| `app_removed` | info | App removed via API |
|
||||
| `app_start_failed` | warn | A DEPLOYED app is not running (fix-3) — fired ONCE per running→down transition |
|
||||
| `disaster_recovery_started` | warning | DR restore begins |
|
||||
| `disaster_recovery_completed` | info/error | DR restore finishes (success/partial) |
|
||||
|
||||
Each event carries typed detail structs (e.g., `BackupDetails`, `DiskDetails`, `HealthDetails`) serialized as JSON.
|
||||
|
||||
**Deployed-app-down alerting (fix-3, v0.120.0, CAMPAIGN-3).** A `deadapp-check` scheduler job (every
|
||||
30 s, after a 90 s boot grace) scans `stackMgr.GetStacks()`: a DEPLOYED app whose containers are
|
||||
`stopped`/`exited` (`stacks.IsDownState` — a Docker `created`/`dead` container, the F11 dead-at-boot
|
||||
case, resolves to `stopped`) gets a **state-based WARN dashboard banner** ("Telepített alkalmazás nem
|
||||
fut: <app>", grouped above 3 so a reboot storm doesn't wall the dashboard) that self-clears when the
|
||||
app runs again, AND an `app_start_failed` hub event fired **once per running→down transition**
|
||||
(`Notifier.NotifyAppStartFailures` tracks per-app state; down→down cycles are silent — the hub owns
|
||||
the real cooldown, the controller adds no timer). The boot grace prevents false alarms during the
|
||||
controller's own startup while STILL firing for an app that never came up. This closes the campaign's
|
||||
4-hour silent CWA death.
|
||||
|
||||
#### Default Enabled Events
|
||||
|
||||
Events the customer receives notifications for (configurable in settings):
|
||||
@@ -1874,11 +1886,25 @@ When `logging.level: "debug"` is set in `controller.yaml`, the controller expose
|
||||
| — | Telemetria teszt | `GET /api/debug/telemetry` | Run the full telemetry collection pipeline on-demand (metrics query + log scan). Returns per-app table: container list, memory current/avg/peak, CPU avg, catalog limit, log error/warning counts, and top issues. Useful for verifying container→stack mapping and testing log scanner patterns without waiting for the 15-minute report cycle. |
|
||||
| 6 | Önfrissítés teszt | `POST /api/debug/selfupdate/dry-run` | Dry-run update check: current vs new image lines, compose writability, backup state. |
|
||||
| 7 | DR / Telepítő varázsló | `POST /api/debug/dr/trigger-setup`, `GET /api/debug/dr/infra-status` | Infra backup status per drive. Trigger setup mode via marker file (requires "RESET" + infra backup pre-check). |
|
||||
| 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 1000 entries), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). |
|
||||
| 8 | Naplóviewer | `GET /api/debug/logs?level=&limit=&after=`, `GET /api/debug/agent-logs` | In-memory log viewer (last 5000 entries, spill-persisted across restart — fix-6), level filter, 2s auto-refresh, color-coded entries. Two tabs (v0.116.0): **Vezérlő** (own ring) and **Ügynök** (the agent's always-DEBUG ring proxied over the local API; a pre-0.83 agent renders the "available after the agent's next update" notice). |
|
||||
|
||||
#### Key Implementation Details
|
||||
|
||||
- **Log buffer** (`internal/web/logbuffer.go`): Ring buffer implementing `io.Writer`. Since v0.116.0 it ALWAYS exists (any `logging.level`) and captures every line INCLUDING `[DEBUG]`: the logger is `io.MultiWriter(LevelFilterWriter(os.Stdout, logging.level), logBuffer)` — stdout/docker-logs keep respecting `logging.level`, the ring holds the full detail for remote diagnostics. `logBuffer.Lines(maxBytes)` renders the newest-kept plain-text tail (the report `controller_log_tail` source). New leveled lines use `internal/logx` (`Debugf/Infof/Warnf/Errorf`); legacy `isDebug()` call sites are unchanged.
|
||||
- **Ring sizing, spill persistence, periodic-noise policy (fix-6, v0.120.0, CAMPAIGN-3).** The
|
||||
campaign measured the 1000-entry ring wrapping in ~6.5 min under load and dying on every restart —
|
||||
the exact post-incident window was the first thing lost. Three changes: **(a) cap 1000→5000**
|
||||
(`Entries`/the debug handler display cap raised to match — a larger ring is useless if the viewer
|
||||
can't request more than 1000 of it; the Naplóviewer default pull is 1000). **(b) periodic-noise
|
||||
policy:** a periodic job's ROUTINE success is not ring-worthy — the every-cycle scheduler "job
|
||||
finished" line and `refreshStatusLocked` per-cycle enumeration are logged at a new `[TRACE]` level
|
||||
that the ring DROPS at write-time (`levelPriority("TRACE") < DEBUG`). Failures and state changes are
|
||||
never TRACE, so nothing load-bearing is lost; this was the biggest ring filler. **(c) spill
|
||||
persistence:** `LogBuffer.SpillTo`/`LoadFrom` atomically (tmp+rename, JSON-lines) spill the ring to
|
||||
`<DataDir>/debug-ring.log` on the **SSD state dir** (the persistent data volume that survives
|
||||
container recreation — NEVER a NAS/HDD path) every 30 s and on clean shutdown, and load it back on
|
||||
boot so a restart / recreate preserves the pre-restart window. Corruption-safe: a truncated/partial
|
||||
line is skipped on load, never fatal.
|
||||
- **Controller self-log pull** (`internal/report/selftail.go`): the hub's report ACK may carry `controller_log_requested` — the NEXT report ships `controller_log_tail` (ring, 128 KB cap, consume-once, the v0.111.0 app-tail pattern; additive fields, app-tail wire unchanged). Serving a pull logs the customer-visible `operator log pull served` INFO line.
|
||||
- **Storage simulation**: `simulatedPaths` map in watchdog prevents the watchdog from re-probing simulated-disconnected paths. Disconnect runs all real steps except `lazyUnmount` (drive stays physically mounted).
|
||||
- **DR trigger safety**: Uses marker file (`data/.needs-setup`) instead of modifying controller.yaml. Pre-checks that infra backup exists on at least one drive.
|
||||
|
||||
Reference in New Issue
Block a user