felhom.eu/REPORT.md

felhom.eu — task reports

Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md.

---

REPORT — hub v0.25.0: per-storage worst-fill alerting (StorageFillChecker)

Date: 2026-06-30 Task: TASK.md — per-storage worst-fill alerting (hub checker). Item 2 of 3 in the post-NAS queue (host_disk ✅ → this → provisioning vzdump retention default).

Baseline

	value
Hub before	v0.24.0
Hub after	v0.25.0 — code 88073ac, manifest bump 063fb0c
Live image	felhom-hub:0.25.0 (ArgoCD felhom Synced/Healthy)

What changed

Generalizes the host-root disk alert (v0.23.0) to any reported storage target — a dump/backup volume, data drive, lvmthin pool, or PBS datastore filling pages the operator with the storage named, even when host root is fine.

• internal/monitor/storage_fill.go (NEW) — StorageFillChecker. A per-target mirror of HostDiskChecker on the same 60s sweep: born/persistent (already-breached (host,target) keys left UNSEEDED → first Check emits), escalation-only emit, recovery re-arm, 1h cooldown. State keyed per (host, target) so targets alert independently. Emits storage_fill_warning/storage_fill_critical at the natural critical severity (second real caller exercising the v0.24.0 dispatcher fix). Thresholds default 90/95, hub-config overridable (alerting.storage_fill_warn_percent/_crit_percent), independent of the host-root thresholds.
• Root excluded (no double-alert): the root-backed builtin (Type=="local", or a target mounted at /) is skipped — HostDiskChecker owns root.
• internal/store/store.go: GetHostStorageTargets() + HostStorageTargetRow — parses report_json.storage_targets[] of each host's latest report (percent = used_fraction×100); modeled on GetHostDiskUsage, no denorm column / migration.
• internal/notify/templates.go + internal/api/handler.go: Hungarian templates + allowlist entries.
• cmd/hub/main.go: register storageFillChecker on the 60s tick beside HostDiskChecker.

Tests — go build ./... && go vet ./... && go test ./... GREEN

• per-target bands (A breaches → only A warns; B silent; A escalates warning→critical; A recovers → clears+re-arms; re-arm fires again).
• born/persistent companion red-proof — a seed-all/transition-only model (pre-seed the breached key) stays silent on the born-breach; the real (unseeded) design emits.
• root-exclusion companion — a root-backed//-mounted target emits nothing; the control asserts 96% IS the critical band, so only the exclusion suppresses the double-alert; a real off-root storage is NOT excluded.
• severity is natural critical; GetHostStorageTargets parse fixture.

Deploy + live validation

Deployed via the GitOps manifest path (build+push; manifests/hub.yaml 0.24.0→0.25.0; ArgoCD hard-refresh → sync → Synced/Healthy). Startup confirmed the checker registered beside HostDiskChecker: Storage fill checker initialized: warn=90% crit=95%, 4 ok seeded, 0 already-breached, 1 root-backed excluded — i.e. the 5 reported storages parsed, the root-backed local (22%) correctly excluded, 4 off-root tracked.

Method (real alert, no artificial fill): felhom-pve's reported off-root targets were felhom-pbs 21.75%, local-lvm 11.2%, felhom-flash/usb ~0%. I briefly set alerting.storage_fill_warn_percent: 20 via the hub-config ConfigMap (commit 5a3747f) + restarted the hub, so the real felhom-pbs fill crossed:

Storage fill checker initialized: warn=20% crit=95%, 3 ok seeded, 1 already-breached left unseeded, 1 root-backed excluded
Storage fill: demo-felhom-01 "felhom-pbs" 22% unknown→warning (storage_fill_warning)
Operator email sent for demo-felhom/storage_fill_warning

The alert named the specific target (felhom-pbs) + its %, fired born-persistent on cycle 1 (pbs was over the lowered threshold at restart), and a real operator email was sent. The threshold was then restored to the default (commit e1dd06e); the hub re-seeded warn=90% … 4 ok (felhom-pbs back to ok).

Observations / follow-ups

• Last queued item: the provisioning-side vzdump retention default (so a box can't refill its own root) — operational, agent/golden side.
• Follow-ons noted: thin-pool metadata exhaustion alerting (v1 alerts on data-fill %; a full thin pool's metadata is a separate failure mode) and per-storage-type thresholds (e.g. tighter for PBS).
• PBS/network targets are included (a full datastore fails backups); that surfaced as the live test target.

No secrets committed. Operator email is the operator's own; the Resend key stays injected from Secret/resend-api.