# felhom.eu — task reports

> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md).

---

# REPORT — hub v0.25.0: per-storage worst-fill alerting (StorageFillChecker)

**Date:** 2026-06-30
**Task:** `TASK.md` — per-storage worst-fill alerting (hub checker). Item 2 of 3 in the post-NAS queue
(host_disk ✅ → **this** → provisioning vzdump retention default).

## Baseline
| | value |
|---|---|
| Hub before | v0.24.0 |
| Hub after | **v0.25.0** — code `88073ac`, manifest bump `063fb0c` |
| Live image | `felhom-hub:0.25.0` (ArgoCD `felhom` Synced/Healthy) |

## What changed
Generalizes the host-root disk alert (v0.23.0) to **any reported storage target** — a dump/backup volume,
data drive, lvmthin pool, or PBS datastore filling pages the operator with the storage named, even when
host root is fine.
- **`internal/monitor/storage_fill.go` (NEW) — `StorageFillChecker`.** A per-target mirror of
  `HostDiskChecker` on the same 60s sweep: born/persistent (already-breached `(host,target)` keys left
  UNSEEDED → first `Check` emits), escalation-only emit, recovery re-arm, 1h cooldown. State keyed per
  **(host, target)** so targets alert independently. Emits `storage_fill_warning`/`storage_fill_critical`
  at the **natural `critical` severity** (second real caller exercising the v0.24.0 dispatcher fix).
  Thresholds default 90/95, hub-config overridable (`alerting.storage_fill_warn_percent`/`_crit_percent`),
  independent of the host-root thresholds.
- **Root excluded (no double-alert):** the root-backed builtin (`Type=="local"`, or a target mounted at
  `/`) is skipped — `HostDiskChecker` owns root.
- **`internal/store/store.go`:** `GetHostStorageTargets()` + `HostStorageTargetRow` — parses
  `report_json.storage_targets[]` of each host's latest report (percent = `used_fraction`×100); modeled on
  `GetHostDiskUsage`, no denorm column / migration.
- **`internal/notify/templates.go` + `internal/api/handler.go`:** Hungarian templates + allowlist entries.
- **`cmd/hub/main.go`:** register `storageFillChecker` on the 60s tick beside `HostDiskChecker`.

## Tests — `go build ./... && go vet ./... && go test ./...` GREEN
- per-target bands (A breaches → only A warns; B silent; A escalates warning→critical; A recovers →
  clears+re-arms; re-arm fires again).
- **born/persistent companion red-proof** — a seed-all/transition-only model (pre-seed the breached key)
  stays silent on the born-breach; the real (unseeded) design emits.
- **root-exclusion companion** — a root-backed/`/`-mounted target emits nothing; the control asserts 96% IS
  the critical band, so only the exclusion suppresses the double-alert; a real off-root storage is NOT
  excluded.
- severity is natural `critical`; `GetHostStorageTargets` parse fixture.

## Deploy + live validation
Deployed via the GitOps manifest path (build+push; `manifests/hub.yaml` 0.24.0→0.25.0; ArgoCD
hard-refresh → sync → **Synced/Healthy**). Startup confirmed the checker registered beside HostDiskChecker:
`Storage fill checker initialized: warn=90% crit=95%, 4 ok seeded, 0 already-breached, 1 root-backed
excluded` — i.e. the 5 reported storages parsed, the root-backed `local` (22%) correctly excluded, 4
off-root tracked.

**Method (real alert, no artificial fill):** felhom-pve's reported off-root targets were felhom-pbs 21.75%,
local-lvm 11.2%, felhom-flash/usb ~0%. I briefly set `alerting.storage_fill_warn_percent: 20` via the
hub-config ConfigMap (commit `5a3747f`) + restarted the hub, so the **real** felhom-pbs fill crossed:
```
Storage fill checker initialized: warn=20% crit=95%, 3 ok seeded, 1 already-breached left unseeded, 1 root-backed excluded
Storage fill: demo-felhom-01 "felhom-pbs" 22% unknown→warning (storage_fill_warning)
Operator email sent for demo-felhom/storage_fill_warning
```
The alert **named the specific target** (felhom-pbs) + its %, fired **born-persistent on cycle 1** (pbs was
over the lowered threshold at restart), and a **real operator email** was sent. The threshold was then
**restored to the default** (commit `e1dd06e`); the hub re-seeded `warn=90% … 4 ok` (felhom-pbs back to ok).

## Observations / follow-ups
- **Last queued item:** the provisioning-side vzdump retention default (so a box can't refill its own root)
  — operational, agent/golden side.
- Follow-ons noted: **thin-pool *metadata* exhaustion** alerting (v1 alerts on data-fill %; a full thin
  pool's metadata is a separate failure mode) and **per-storage-type thresholds** (e.g. tighter for PBS).
- PBS/network targets are included (a full datastore fails backups); that surfaced as the live test target.

No secrets committed. Operator email is the operator's own; the Resend key stays injected from
`Secret/resend-api`.