docs: REPORT for hub v0.25.0 (per-storage fill alerting, live-validated)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs

REPORT.md

@@ -4,50 +4,73 @@
 
 ---
 
-# REPORT — hub v0.24.0: dispatcher routes `critical` (NAS A2, "Part 0")
+# REPORT — hub v0.25.0: per-storage worst-fill alerting (StorageFillChecker)
 
 **Date:** 2026-06-30
-**Task:** `TASK.md` Part 0 — the hub dispatcher fix that ships alongside NAS Part A2 (controller work is in
-the felhom-controller repo, v0.92.0).
+**Task:** `TASK.md` — per-storage worst-fill alerting (hub checker). Item 2 of 3 in the post-NAS queue
+(host_disk ✅ → **this** → provisioning vzdump retention default).
 
 ## Baseline
 | | value |
 |---|---|
-| Hub before | v0.23.0 |
-| Hub after | **v0.24.0** — code `0ff1d3c`, manifest bump `87f0794` |
-| Live image | `felhom-hub:0.24.0` (ArgoCD `felhom` Synced/Healthy) |
+| Hub before | v0.24.0 |
+| Hub after | **v0.25.0** — code `88073ac`, manifest bump `063fb0c` |
+| Live image | `felhom-hub:0.25.0` (ArgoCD `felhom` Synced/Healthy) |
 
 ## What changed
-- **`internal/notify/dispatcher.go` `ProcessEvent`:** the severity gate was `!= "warning" && != "error"` —
-  a `critical`-severity event was **silently dropped** (never emailed). Now routes **warning / error /
-  critical** (`severityNotifies`); `info` stays an intentional non-notify; an **unrecognized** severity is
-  **logged** (`[WARN] Dispatcher: unrecognized severity …`), never silently dropped.
-- **`internal/monitor/host_disk.go`:** `host_disk_critical` now emits its **natural `critical` severity**
-  (was forced to `error` to survive the old gate); `FormatOperatorEmail` styles `critical` 🔴 like `error`.
-- **Latent crash guard:** `processCustomer` dereferenced `GetNotificationPrefs`, which returns `(nil, nil)`
-  for a customer with no notification row — an event for such a customer would have **panicked the dispatcher
-  goroutine and crashed the hub**. Now guards `prefs == nil`.
-- **Seam:** `sendEmailFn` so routing is unit-tested without real HTTP.
-- Verified safe first (§9 rule 5): grep confirmed **no** event currently emits `critical` (all controller
-  events are info/warning/error; the hub's only would-be `critical` emitter is `host_disk`) → no surprise
-  alert volume.
+Generalizes the host-root disk alert (v0.23.0) to **any reported storage target** — a dump/backup volume,
+data drive, lvmthin pool, or PBS datastore filling pages the operator with the storage named, even when
+host root is fine.
+- **`internal/monitor/storage_fill.go` (NEW) — `StorageFillChecker`.** A per-target mirror of
+  `HostDiskChecker` on the same 60s sweep: born/persistent (already-breached `(host,target)` keys left
+  UNSEEDED → first `Check` emits), escalation-only emit, recovery re-arm, 1h cooldown. State keyed per
+  **(host, target)** so targets alert independently. Emits `storage_fill_warning`/`storage_fill_critical`
+  at the **natural `critical` severity** (second real caller exercising the v0.24.0 dispatcher fix).
+  Thresholds default 90/95, hub-config overridable (`alerting.storage_fill_warn_percent`/`_crit_percent`),
+  independent of the host-root thresholds.
+- **Root excluded (no double-alert):** the root-backed builtin (`Type=="local"`, or a target mounted at
+  `/`) is skipped — `HostDiskChecker` owns root.
+- **`internal/store/store.go`:** `GetHostStorageTargets()` + `HostStorageTargetRow` — parses
+  `report_json.storage_targets[]` of each host's latest report (percent = `used_fraction`×100); modeled on
+  `GetHostDiskUsage`, no denorm column / migration.
+- **`internal/notify/templates.go` + `internal/api/handler.go`:** Hungarian templates + allowlist entries.
+- **`cmd/hub/main.go`:** register `storageFillChecker` on the 60s tick beside `HostDiskChecker`.
 
 ## Tests — `go build ./... && go vet ./... && go test ./...` GREEN
-`severityNotifies` (warning/error/critical notify; info/unknown don't) + **companion red-proof** (the pre-fix
-`warning||error` predicate drops `critical`); ProcessEvent routes `critical` to the operator; an unknown
-severity is logged not dropped; `info` is silent and not mis-logged.
+- per-target bands (A breaches → only A warns; B silent; A escalates warning→critical; A recovers →
+  clears+re-arms; re-arm fires again).
+- **born/persistent companion red-proof** — a seed-all/transition-only model (pre-seed the breached key)
+  stays silent on the born-breach; the real (unseeded) design emits.
+- **root-exclusion companion** — a root-backed/`/`-mounted target emits nothing; the control asserts 96% IS
+  the critical band, so only the exclusion suppresses the double-alert; a real off-root storage is NOT
+  excluded.
+- severity is natural `critical`; `GetHostStorageTargets` parse fixture.
 
-## Deploy + validation
-Deployed via the GitOps manifest path (image build+push; `manifests/hub.yaml` 0.23.0→0.24.0; ArgoCD
-hard-refresh → OutOfSync → sync → **Synced/Healthy**), per the repo convention (not bare `kubectl set
-image`). Startup clean; the `host_disk` checker re-seeded `1 ok` (the demo host root is at 24% after the
-prior task's prune). The critical-routing fix is unit-validated; not re-fired live (the demo disk is below
-threshold). The downstream A2 NAS live e2e (add → jellyfin reads → kill → unreachable warning, no cascade →
-recover → remove) was validated against an isolated sim NAS — see `felhom-controller/REPORT.md`.
+## Deploy + live validation
+Deployed via the GitOps manifest path (build+push; `manifests/hub.yaml` 0.24.0→0.25.0; ArgoCD
+hard-refresh → sync → **Synced/Healthy**). Startup confirmed the checker registered beside HostDiskChecker:
+`Storage fill checker initialized: warn=90% crit=95%, 4 ok seeded, 0 already-breached, 1 root-backed
+excluded` — i.e. the 5 reported storages parsed, the root-backed `local` (22%) correctly excluded, 4
+off-root tracked.
+
+**Method (real alert, no artificial fill):** felhom-pve's reported off-root targets were felhom-pbs 21.75%,
+local-lvm 11.2%, felhom-flash/usb ~0%. I briefly set `alerting.storage_fill_warn_percent: 20` via the
+hub-config ConfigMap (commit `5a3747f`) + restarted the hub, so the **real** felhom-pbs fill crossed:
+```
+Storage fill checker initialized: warn=20% crit=95%, 3 ok seeded, 1 already-breached left unseeded, 1 root-backed excluded
+Storage fill: demo-felhom-01 "felhom-pbs" 22% unknown→warning (storage_fill_warning)
+Operator email sent for demo-felhom/storage_fill_warning
+```
+The alert **named the specific target** (felhom-pbs) + its %, fired **born-persistent on cycle 1** (pbs was
+over the lowered threshold at restart), and a **real operator email** was sent. The threshold was then
+**restored to the default** (commit `e1dd06e`); the hub re-seeded `warn=90% … 4 ok` (felhom-pbs back to ok).
 
 ## Observations / follow-ups
-- Per-storage `StorageTargets` worst-fill alerting remains the easy follow-on to `host_disk`.
-- Part B (restic-over-SFTP NAS backup target) still pending — separate task.
+- **Last queued item:** the provisioning-side vzdump retention default (so a box can't refill its own root)
+  — operational, agent/golden side.
+- Follow-ons noted: **thin-pool *metadata* exhaustion** alerting (v1 alerts on data-fill %; a full thin
+  pool's metadata is a separate failure mode) and **per-storage-type thresholds** (e.g. tighter for PBS).
+- PBS/network targets are included (a full datastore fails backups); that surfaced as the live test target.
 
-No secrets committed. Operator email address is the operator's own; the Resend key stays injected from
+No secrets committed. Operator email is the operator's own; the Resend key stays injected from
 `Secret/resend-api`.