admin
34364b47c6
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
77 lines
4.8 KiB
Markdown
77 lines
4.8 KiB
Markdown
# felhom.eu — task reports
|
||
|
||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md).
|
||
|
||
---
|
||
|
||
# REPORT — hub v0.25.0: per-storage worst-fill alerting (StorageFillChecker)
|
||
|
||
**Date:** 2026-06-30
|
||
**Task:** `TASK.md` — per-storage worst-fill alerting (hub checker). Item 2 of 3 in the post-NAS queue
|
||
(host_disk ✅ → **this** → provisioning vzdump retention default).
|
||
|
||
## Baseline
|
||
| | value |
|
||
|---|---|
|
||
| Hub before | v0.24.0 |
|
||
| Hub after | **v0.25.0** — code `88073ac`, manifest bump `063fb0c` |
|
||
| Live image | `felhom-hub:0.25.0` (ArgoCD `felhom` Synced/Healthy) |
|
||
|
||
## What changed
|
||
Generalizes the host-root disk alert (v0.23.0) to **any reported storage target** — a dump/backup volume,
|
||
data drive, lvmthin pool, or PBS datastore filling pages the operator with the storage named, even when
|
||
host root is fine.
|
||
- **`internal/monitor/storage_fill.go` (NEW) — `StorageFillChecker`.** A per-target mirror of
|
||
`HostDiskChecker` on the same 60s sweep: born/persistent (already-breached `(host,target)` keys left
|
||
UNSEEDED → first `Check` emits), escalation-only emit, recovery re-arm, 1h cooldown. State keyed per
|
||
**(host, target)** so targets alert independently. Emits `storage_fill_warning`/`storage_fill_critical`
|
||
at the **natural `critical` severity** (second real caller exercising the v0.24.0 dispatcher fix).
|
||
Thresholds default 90/95, hub-config overridable (`alerting.storage_fill_warn_percent`/`_crit_percent`),
|
||
independent of the host-root thresholds.
|
||
- **Root excluded (no double-alert):** the root-backed builtin (`Type=="local"`, or a target mounted at
|
||
`/`) is skipped — `HostDiskChecker` owns root.
|
||
- **`internal/store/store.go`:** `GetHostStorageTargets()` + `HostStorageTargetRow` — parses
|
||
`report_json.storage_targets[]` of each host's latest report (percent = `used_fraction`×100); modeled on
|
||
`GetHostDiskUsage`, no denorm column / migration.
|
||
- **`internal/notify/templates.go` + `internal/api/handler.go`:** Hungarian templates + allowlist entries.
|
||
- **`cmd/hub/main.go`:** register `storageFillChecker` on the 60s tick beside `HostDiskChecker`.
|
||
|
||
## Tests — `go build ./... && go vet ./... && go test ./...` GREEN
|
||
- per-target bands (A breaches → only A warns; B silent; A escalates warning→critical; A recovers →
|
||
clears+re-arms; re-arm fires again).
|
||
- **born/persistent companion red-proof** — a seed-all/transition-only model (pre-seed the breached key)
|
||
stays silent on the born-breach; the real (unseeded) design emits.
|
||
- **root-exclusion companion** — a root-backed/`/`-mounted target emits nothing; the control asserts 96% IS
|
||
the critical band, so only the exclusion suppresses the double-alert; a real off-root storage is NOT
|
||
excluded.
|
||
- severity is natural `critical`; `GetHostStorageTargets` parse fixture.
|
||
|
||
## Deploy + live validation
|
||
Deployed via the GitOps manifest path (build+push; `manifests/hub.yaml` 0.24.0→0.25.0; ArgoCD
|
||
hard-refresh → sync → **Synced/Healthy**). Startup confirmed the checker registered beside HostDiskChecker:
|
||
`Storage fill checker initialized: warn=90% crit=95%, 4 ok seeded, 0 already-breached, 1 root-backed
|
||
excluded` — i.e. the 5 reported storages parsed, the root-backed `local` (22%) correctly excluded, 4
|
||
off-root tracked.
|
||
|
||
**Method (real alert, no artificial fill):** felhom-pve's reported off-root targets were felhom-pbs 21.75%,
|
||
local-lvm 11.2%, felhom-flash/usb ~0%. I briefly set `alerting.storage_fill_warn_percent: 20` via the
|
||
hub-config ConfigMap (commit `5a3747f`) + restarted the hub, so the **real** felhom-pbs fill crossed:
|
||
```
|
||
Storage fill checker initialized: warn=20% crit=95%, 3 ok seeded, 1 already-breached left unseeded, 1 root-backed excluded
|
||
Storage fill: demo-felhom-01 "felhom-pbs" 22% unknown→warning (storage_fill_warning)
|
||
Operator email sent for demo-felhom/storage_fill_warning
|
||
```
|
||
The alert **named the specific target** (felhom-pbs) + its %, fired **born-persistent on cycle 1** (pbs was
|
||
over the lowered threshold at restart), and a **real operator email** was sent. The threshold was then
|
||
**restored to the default** (commit `e1dd06e`); the hub re-seeded `warn=90% … 4 ok` (felhom-pbs back to ok).
|
||
|
||
## Observations / follow-ups
|
||
- **Last queued item:** the provisioning-side vzdump retention default (so a box can't refill its own root)
|
||
— operational, agent/golden side.
|
||
- Follow-ons noted: **thin-pool *metadata* exhaustion** alerting (v1 alerts on data-fill %; a full thin
|
||
pool's metadata is a separate failure mode) and **per-storage-type thresholds** (e.g. tighter for PBS).
|
||
- PBS/network targets are included (a full datastore fails backups); that surfaced as the live test target.
|
||
|
||
No secrets committed. Operator email is the operator's own; the Resend key stays injected from
|
||
`Secret/resend-api`.
|