From 34364b47c6a1bf096bf6e7f930b802fa80effcd9 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 30 Jun 2026 19:08:36 +0200 Subject: [PATCH] docs: REPORT for hub v0.25.0 (per-storage fill alerting, live-validated) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs --- REPORT.md | 87 +++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 55 insertions(+), 32 deletions(-) diff --git a/REPORT.md b/REPORT.md index 5b5f838..7df3b6d 100644 --- a/REPORT.md +++ b/REPORT.md @@ -4,50 +4,73 @@ --- -# REPORT — hub v0.24.0: dispatcher routes `critical` (NAS A2, "Part 0") +# REPORT — hub v0.25.0: per-storage worst-fill alerting (StorageFillChecker) **Date:** 2026-06-30 -**Task:** `TASK.md` Part 0 — the hub dispatcher fix that ships alongside NAS Part A2 (controller work is in -the felhom-controller repo, v0.92.0). +**Task:** `TASK.md` — per-storage worst-fill alerting (hub checker). Item 2 of 3 in the post-NAS queue +(host_disk ✅ → **this** → provisioning vzdump retention default). ## Baseline | | value | |---|---| -| Hub before | v0.23.0 | -| Hub after | **v0.24.0** — code `0ff1d3c`, manifest bump `87f0794` | -| Live image | `felhom-hub:0.24.0` (ArgoCD `felhom` Synced/Healthy) | +| Hub before | v0.24.0 | +| Hub after | **v0.25.0** — code `88073ac`, manifest bump `063fb0c` | +| Live image | `felhom-hub:0.25.0` (ArgoCD `felhom` Synced/Healthy) | ## What changed -- **`internal/notify/dispatcher.go` `ProcessEvent`:** the severity gate was `!= "warning" && != "error"` — - a `critical`-severity event was **silently dropped** (never emailed). Now routes **warning / error / - critical** (`severityNotifies`); `info` stays an intentional non-notify; an **unrecognized** severity is - **logged** (`[WARN] Dispatcher: unrecognized severity …`), never silently dropped. -- **`internal/monitor/host_disk.go`:** `host_disk_critical` now emits its **natural `critical` severity** - (was forced to `error` to survive the old gate); `FormatOperatorEmail` styles `critical` 🔴 like `error`. -- **Latent crash guard:** `processCustomer` dereferenced `GetNotificationPrefs`, which returns `(nil, nil)` - for a customer with no notification row — an event for such a customer would have **panicked the dispatcher - goroutine and crashed the hub**. Now guards `prefs == nil`. -- **Seam:** `sendEmailFn` so routing is unit-tested without real HTTP. -- Verified safe first (§9 rule 5): grep confirmed **no** event currently emits `critical` (all controller - events are info/warning/error; the hub's only would-be `critical` emitter is `host_disk`) → no surprise - alert volume. +Generalizes the host-root disk alert (v0.23.0) to **any reported storage target** — a dump/backup volume, +data drive, lvmthin pool, or PBS datastore filling pages the operator with the storage named, even when +host root is fine. +- **`internal/monitor/storage_fill.go` (NEW) — `StorageFillChecker`.** A per-target mirror of + `HostDiskChecker` on the same 60s sweep: born/persistent (already-breached `(host,target)` keys left + UNSEEDED → first `Check` emits), escalation-only emit, recovery re-arm, 1h cooldown. State keyed per + **(host, target)** so targets alert independently. Emits `storage_fill_warning`/`storage_fill_critical` + at the **natural `critical` severity** (second real caller exercising the v0.24.0 dispatcher fix). + Thresholds default 90/95, hub-config overridable (`alerting.storage_fill_warn_percent`/`_crit_percent`), + independent of the host-root thresholds. +- **Root excluded (no double-alert):** the root-backed builtin (`Type=="local"`, or a target mounted at + `/`) is skipped — `HostDiskChecker` owns root. +- **`internal/store/store.go`:** `GetHostStorageTargets()` + `HostStorageTargetRow` — parses + `report_json.storage_targets[]` of each host's latest report (percent = `used_fraction`×100); modeled on + `GetHostDiskUsage`, no denorm column / migration. +- **`internal/notify/templates.go` + `internal/api/handler.go`:** Hungarian templates + allowlist entries. +- **`cmd/hub/main.go`:** register `storageFillChecker` on the 60s tick beside `HostDiskChecker`. ## Tests — `go build ./... && go vet ./... && go test ./...` GREEN -`severityNotifies` (warning/error/critical notify; info/unknown don't) + **companion red-proof** (the pre-fix -`warning||error` predicate drops `critical`); ProcessEvent routes `critical` to the operator; an unknown -severity is logged not dropped; `info` is silent and not mis-logged. +- per-target bands (A breaches → only A warns; B silent; A escalates warning→critical; A recovers → + clears+re-arms; re-arm fires again). +- **born/persistent companion red-proof** — a seed-all/transition-only model (pre-seed the breached key) + stays silent on the born-breach; the real (unseeded) design emits. +- **root-exclusion companion** — a root-backed/`/`-mounted target emits nothing; the control asserts 96% IS + the critical band, so only the exclusion suppresses the double-alert; a real off-root storage is NOT + excluded. +- severity is natural `critical`; `GetHostStorageTargets` parse fixture. -## Deploy + validation -Deployed via the GitOps manifest path (image build+push; `manifests/hub.yaml` 0.23.0→0.24.0; ArgoCD -hard-refresh → OutOfSync → sync → **Synced/Healthy**), per the repo convention (not bare `kubectl set -image`). Startup clean; the `host_disk` checker re-seeded `1 ok` (the demo host root is at 24% after the -prior task's prune). The critical-routing fix is unit-validated; not re-fired live (the demo disk is below -threshold). The downstream A2 NAS live e2e (add → jellyfin reads → kill → unreachable warning, no cascade → -recover → remove) was validated against an isolated sim NAS — see `felhom-controller/REPORT.md`. +## Deploy + live validation +Deployed via the GitOps manifest path (build+push; `manifests/hub.yaml` 0.24.0→0.25.0; ArgoCD +hard-refresh → sync → **Synced/Healthy**). Startup confirmed the checker registered beside HostDiskChecker: +`Storage fill checker initialized: warn=90% crit=95%, 4 ok seeded, 0 already-breached, 1 root-backed +excluded` — i.e. the 5 reported storages parsed, the root-backed `local` (22%) correctly excluded, 4 +off-root tracked. + +**Method (real alert, no artificial fill):** felhom-pve's reported off-root targets were felhom-pbs 21.75%, +local-lvm 11.2%, felhom-flash/usb ~0%. I briefly set `alerting.storage_fill_warn_percent: 20` via the +hub-config ConfigMap (commit `5a3747f`) + restarted the hub, so the **real** felhom-pbs fill crossed: +``` +Storage fill checker initialized: warn=20% crit=95%, 3 ok seeded, 1 already-breached left unseeded, 1 root-backed excluded +Storage fill: demo-felhom-01 "felhom-pbs" 22% unknown→warning (storage_fill_warning) +Operator email sent for demo-felhom/storage_fill_warning +``` +The alert **named the specific target** (felhom-pbs) + its %, fired **born-persistent on cycle 1** (pbs was +over the lowered threshold at restart), and a **real operator email** was sent. The threshold was then +**restored to the default** (commit `e1dd06e`); the hub re-seeded `warn=90% … 4 ok` (felhom-pbs back to ok). ## Observations / follow-ups -- Per-storage `StorageTargets` worst-fill alerting remains the easy follow-on to `host_disk`. -- Part B (restic-over-SFTP NAS backup target) still pending — separate task. +- **Last queued item:** the provisioning-side vzdump retention default (so a box can't refill its own root) + — operational, agent/golden side. +- Follow-ons noted: **thin-pool *metadata* exhaustion** alerting (v1 alerts on data-fill %; a full thin + pool's metadata is a separate failure mode) and **per-storage-type thresholds** (e.g. tighter for PBS). +- PBS/network targets are included (a full datastore fails backups); that surfaced as the live test target. -No secrets committed. Operator email address is the operator's own; the Resend key stays injected from +No secrets committed. Operator email is the operator's own; the Resend key stays injected from `Secret/resend-api`.