From cac745ee048f690bcd5e1d01173eb43af45235b3 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 30 Jun 2026 14:53:53 +0200 Subject: [PATCH] docs: REPORT for hub v0.24.0 (dispatcher critical routing, Part 0) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs --- REPORT.md | 111 ++++++++++++++++++------------------------------------ 1 file changed, 36 insertions(+), 75 deletions(-) diff --git a/REPORT.md b/REPORT.md index 7584ed1..5b5f838 100644 --- a/REPORT.md +++ b/REPORT.md @@ -4,89 +4,50 @@ --- -# REPORT — hub v0.23.0: host root-disk pressure monitoring + alert +# REPORT — hub v0.24.0: dispatcher routes `critical` (NAS A2, "Part 0") **Date:** 2026-06-30 -**Task:** `TASK.md` — host root-disk pressure monitoring + alert (hub checker). Implementation class, hub-only. - -## Baseline (confirmed) +**Task:** `TASK.md` Part 0 — the hub dispatcher fix that ships alongside NAS Part A2 (controller work is in +the felhom-controller repo, v0.92.0). +## Baseline | | value | |---|---| -| Hub `main` before | v0.22.1 (commit `f133355`) | -| Hub after | **v0.23.0** | -| Commits | `897997c` (code) · `77c61ff` (manifest 0.22.1→0.23.0) | -| Live image | `gitea.dooplex.hu/admin/felhom-hub:0.23.0` (ArgoCD `felhom` Synced/Healthy) | +| Hub before | v0.23.0 | +| Hub after | **v0.24.0** — code `0ff1d3c`, manifest bump `87f0794` | +| Live image | `felhom-hub:0.24.0` (ArgoCD `felhom` Synced/Healthy) | -## What was built - -A hub monitor checker that, on the existing 60s sweep, reads each host's latest root-fs `disk_percent` -(reported by the agent, already stored) and emits an **operator alert** when it crosses a warning (default -**90%**) or critical (default **95%**) threshold — closing the silent-failure gap behind the felhom-pve -incident (vzdump piling under `/var/lib/vz/dump` until the host root filled and PVE/logging/agent writes -could start failing, with no alert). - -### Files changed (commit `897997c`) -- `internal/monitor/host_disk.go` (NEW) — `HostDiskChecker`. Sibling of `HostCapabilityChecker`/ - `HostLeafChecker`; rank-based bands (ok→warning→critical); **born/persistent** (already-breached hosts - left UNSEEDED → first `Check` emits — the F2 lesson); de-escalation/recovery re-arm silently; thresholds - hub-config overridable with a defaults/sanity guard (`normalizeDiskThresholds`). -- `internal/store/store.go` — `GetHostDiskUsage()` + `HostDiskRow` (latest report per host; `disk_percent` - column + total/used bytes from `report_json`; no schema migration). -- `internal/notify/templates.go` — Hungarian customer templates for `host_disk_warning`/`_critical`. -- `internal/api/handler.go` — both types added to `allowedEventTypes`. -- `cmd/hub/main.go` — `alerting.host_disk_warn_percent`/`host_disk_crit_percent` config; register - `hostDiskChecker` on the 60s tick. - -### Design decision worth flagging (deviation from the task) -The task said "critical → severity `critical`". The live dispatcher (`notify/dispatcher.go`) only routes -`severity == "warning" || "error"` — a `"critical"` severity is **silently dropped**. So the critical band -maps to **severity `error`** (and the operator email's 🔴), which is what actually alerts. The event *type* -is still `host_disk_critical`. Asserted by `TestHostDiskChecker_Severity`. - -### Distinct from the guest disk alert (§8/§7-D) -Event types are `host_disk_warning`/`host_disk_critical` (the Proxmox HOST root fs) — never the controller's -GUEST `disk_warning`/`disk_critical` (its own cgroup view); the two never dedup or mask each other. +## What changed +- **`internal/notify/dispatcher.go` `ProcessEvent`:** the severity gate was `!= "warning" && != "error"` — + a `critical`-severity event was **silently dropped** (never emailed). Now routes **warning / error / + critical** (`severityNotifies`); `info` stays an intentional non-notify; an **unrecognized** severity is + **logged** (`[WARN] Dispatcher: unrecognized severity …`), never silently dropped. +- **`internal/monitor/host_disk.go`:** `host_disk_critical` now emits its **natural `critical` severity** + (was forced to `error` to survive the old gate); `FormatOperatorEmail` styles `critical` 🔴 like `error`. +- **Latent crash guard:** `processCustomer` dereferenced `GetNotificationPrefs`, which returns `(nil, nil)` + for a customer with no notification row — an event for such a customer would have **panicked the dispatcher + goroutine and crashed the hub**. Now guards `prefs == nil`. +- **Seam:** `sendEmailFn` so routing is unit-tested without real HTTP. +- Verified safe first (§9 rule 5): grep confirmed **no** event currently emits `critical` (all controller + events are info/warning/error; the hub's only would-be `critical` emitter is `host_disk`) → no surprise + alert volume. ## Tests — `go build ./... && go vet ./... && go test ./...` GREEN -- `TestHostDiskChecker_Bands` — seed ok (no event), 80→nothing, ok→warning, steady (no re-emit), - warning→critical escalation, recovery clears, re-arm (a fresh breach alerts again), type is `host_disk_*`. -- `TestHostDiskChecker_Severity` — warning band → `warning`, critical band → `error`. -- `TestHostDiskChecker_BornPersistent` — a host already at 97% at init → first `Check` emits - `host_disk_critical`. **Companion red-proof:** a seed-all (transition-only) model — emulated by - pre-seeding the breached host's state to `critical` — stays **silent** on the born-breach; the real - (unseeded) design emits. That gap is the bug this design fixes. -- `TestHostDiskChecker_ThresholdDefaults` — unset/invalid/misordered config → 90/95 (no silence/inversion). +`severityNotifies` (warning/error/critical notify; info/unknown don't) + **companion red-proof** (the pre-fix +`warning||error` predicate drops `critical`); ProcessEvent routes `critical` to the operator; an unknown +severity is logged not dropped; `info` is silent and not mis-logged. -## Deploy + live validation (real operator alert) -Deployed via the repo's GitOps path (image build+push on 180; `manifests/hub.yaml` 0.22.1→0.23.0; ArgoCD -hard-refresh → `OutOfSync` → sync → `Synced/Healthy`). Used the **manifest path, not bare `kubectl set -image`**, because the repo convention is that a bare set-image is reverted on the next sync. +## Deploy + validation +Deployed via the GitOps manifest path (image build+push; `manifests/hub.yaml` 0.23.0→0.24.0; ArgoCD +hard-refresh → OutOfSync → sync → **Synced/Healthy**), per the repo convention (not bare `kubectl set +image`). Startup clean; the `host_disk` checker re-seeded `1 ok` (the demo host root is at 24% after the +prior task's prune). The critical-routing fix is unit-validated; not re-fired live (the demo disk is below +threshold). The downstream A2 NAS live e2e (add → jellyfin reads → kill → unreachable warning, no cascade → +recover → remove) was validated against an isolated sim NAS — see `felhom-controller/REPORT.md`. -**Method:** the cheapest real path from §13 — felhom-pve's host root was **already at ~96–100%** (the actual -hazard), so no artificial fill and no threshold change were needed. Startup + first-tick logs: -``` -[INFO] felhom-hub 0.23.0 starting -[INFO] Host disk checker initialized: warn=90% crit=95%, 0 ok seeded, 1 already-breached left unseeded (first Check emits) -[INFO] Host disk: demo-felhom-01 root 96% unknown→critical (host_disk_critical) -[INFO] Operator email sent for demo-felhom/host_disk_critical -``` -The born-persistent alert fired on cycle 1 and a **real operator email was sent via Resend** (the -send-success log line — same proof standard as the prior capability alert). Pod `Running 1/1`, no errors. +## Observations / follow-ups +- Per-storage `StorageTargets` worst-fill alerting remains the easy follow-on to `host_disk`. +- Part B (restic-over-SFTP NAS backup target) still pending — separate task. -## Hazard relieved (operational) -The host root was genuinely full (85G of vzdump backups — ~18 copies of demo guest 9201 at ~3.9G each, on -the root fs). I pruned the old backups, **keeping the 2 most recent** (regenerable demo-guest backups): -root **100% → 24%** (freed ~69G). The recovery/re-arm then reflects on the agent's next 15-min report (disk -now below warn); the recovery logic itself is unit-tested. - -## Observations / follow-ups (noted, not built) -- **Per-storage `StorageTargets` worst-fill alerting** is the easy follow-on (a dedicated dump/backup - storage filling). Host root `disk_percent` already covered the observed vzdump-on-root case, so root is - the v1 headline. -- **The retention *fix*** — a provisioning-side `prune-backups` (vzdump `--maxfiles`/`prune-backups`) - default so a box can't refill its own root — is a separate small operational follow-up (agent/golden - side). The manual prune above was a one-off relief, not the durable fix. - -No secrets in any committed file. Operator email address is the operator's own (not a secret); the Resend -key stays injected from `Secret/resend-api`. +No secrets committed. Operator email address is the operator's own; the Resend key stays injected from +`Secret/resend-api`.