docs: REPORT for hub v0.23.0 (host disk monitoring, live alert proven)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs

REPORT.md

@@ -2,43 +2,91 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md).
 
-## Close the test-run findings — operator email (Part A) + the F2 closures (audit) — 2026-06-30
+---
 
-This repo's slice of the "close the TESTRUN findings" task. The agent-side fixes (F2-a, F2-b) ship in
+# REPORT — hub v0.23.0: host root-disk pressure monitoring + alert
-`felhom-agent` v0.49.0; here it is **Part A (hub config)** + the audit write-up. No hub image change.
 
-### Part A — operator email delivery (the previously-unproven hop) — PROVEN end-to-end
+**Date:** 2026-06-30
-The hub dispatcher's operator path (`Dispatcher.processOperator`) sends only when
+**Task:** `TASK.md` — host root-disk pressure monitoring + alert (hub checker). Implementation class, hub-only.
-`operatorOn && operatorEmail != ""`; the demo `hub-config` set neither, so the self-health pipeline
-(probe → report → checker → dispatch) stopped one hop short of the inbox. **Fix = config-only:**
-`manifests/hub.yaml` `hub-config` ConfigMap now sets `operator_email: admin@felhom.eu` +
-`operator_enabled: true`. Committed, ArgoCD-synced, hub pod restarted to reload; the mounted config shows
-both (Resend key NOT printed; it stays injected from `Secret/resend-api`). Live tag stays **v0.22.1**.
 
-**Proven (both channels):**
+## Baseline (confirmed)
-- **Operator:** scratch-denied the new agent `stalelock-unlock` grant → `agent_capability_degraded` →
-  `[INFO] Operator email sent for demo-felhom/agent_capability_degraded` (the send-success line that
-  never fired while the path was gated off) → restored → `agent_capability_recovered`.
-- **Customer:** `POST /api/v1/notify` (`event_type:test`) → `{"sent":true}` →
-  `Notification email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom/test`. Same `sendEmail` →
-  Resend the operator path uses.
 
-### Audit
+| | value |
-`documentation/audits/TESTRUN-fullstack-2026-06-29.md`: appended the "Findings closed" section (Part A
+|---|---|
-proof + F2-a root cause/fix + F2-b fix/live validation + the pending §D supervised reboot) and removed the
+| Hub `main` before | v0.22.1 (commit `f133355`) |
-stale duplicate "Final state" block (it showed old versions 0.45.0/0.90.0/0.20.0 and contradicted the real
+| Hub after | **v0.23.0** |
-parity).
+| Commits | `897997c` (code) · `77c61ff` (manifest 0.22.1→0.23.0) |
+| Live image | `gitea.dooplex.hu/admin/felhom-hub:0.23.0` (ArgoCD `felhom` Synced/Healthy) |
 
-### Companion (felhom-agent v0.49.0, for cross-reference)
+## What was built
-- **F2-b:** startup stale-lock recovery (`pct unlock` → delete dangling `vzdump` snapshot → start iff
-  onboot, guarded by a no-vzdump-running invariant). Live-validated on `felhom-pve` (recovery on scratch
-  9999; invariant guard proven against a REAL in-flight 9201 backup). New narrow grant + Critical capability.
-- **F2-a:** the shared-parent boot script never redeployed because `EnsureSharedParent` gated on the unit
-  only; now compares the script too. Root cause confirmed by reproducing the doubling in a `/mnt` scratch
-  dir; corrected script is staged on disk (boot-time-only). The peer-group red-proof is the §D supervised
-  reboot.
 
-### Parity — known-good
+A hub monitor checker that, on the existing 60s sweep, reads each host's latest root-fs `disk_percent`
-agent **v0.49.0** / controller v0.91.0 / hub v0.22.1 (+ operator-email hub-config); leaf==pin `60b5974d…`;
+(reported by the agent, already stored) and emits an **operator alert** when it crosses a warning (default
-caps **46/46**; both drives bound; channel up; apps healthy. Pending the §D supervised reboot: the F2-a
+**90%**) or critical (default **95%**) threshold — closing the silent-failure gap behind the felhom-pve
-host-side doubling (corrected script staged; live mount intentionally untouched). No secrets recorded.
+incident (vzdump piling under `/var/lib/vz/dump` until the host root filled and PVE/logging/agent writes
+could start failing, with no alert).
+
+### Files changed (commit `897997c`)
+- `internal/monitor/host_disk.go` (NEW) — `HostDiskChecker`. Sibling of `HostCapabilityChecker`/
+  `HostLeafChecker`; rank-based bands (ok→warning→critical); **born/persistent** (already-breached hosts
+  left UNSEEDED → first `Check` emits — the F2 lesson); de-escalation/recovery re-arm silently; thresholds
+  hub-config overridable with a defaults/sanity guard (`normalizeDiskThresholds`).
+- `internal/store/store.go` — `GetHostDiskUsage()` + `HostDiskRow` (latest report per host; `disk_percent`
+  column + total/used bytes from `report_json`; no schema migration).
+- `internal/notify/templates.go` — Hungarian customer templates for `host_disk_warning`/`_critical`.
+- `internal/api/handler.go` — both types added to `allowedEventTypes`.
+- `cmd/hub/main.go` — `alerting.host_disk_warn_percent`/`host_disk_crit_percent` config; register
+  `hostDiskChecker` on the 60s tick.
+
+### Design decision worth flagging (deviation from the task)
+The task said "critical → severity `critical`". The live dispatcher (`notify/dispatcher.go`) only routes
+`severity == "warning" || "error"` — a `"critical"` severity is **silently dropped**. So the critical band
+maps to **severity `error`** (and the operator email's 🔴), which is what actually alerts. The event *type*
+is still `host_disk_critical`. Asserted by `TestHostDiskChecker_Severity`.
+
+### Distinct from the guest disk alert (§8/§7-D)
+Event types are `host_disk_warning`/`host_disk_critical` (the Proxmox HOST root fs) — never the controller's
+GUEST `disk_warning`/`disk_critical` (its own cgroup view); the two never dedup or mask each other.
+
+## Tests — `go build ./... && go vet ./... && go test ./...` GREEN
+- `TestHostDiskChecker_Bands` — seed ok (no event), 80→nothing, ok→warning, steady (no re-emit),
+  warning→critical escalation, recovery clears, re-arm (a fresh breach alerts again), type is `host_disk_*`.
+- `TestHostDiskChecker_Severity` — warning band → `warning`, critical band → `error`.
+- `TestHostDiskChecker_BornPersistent` — a host already at 97% at init → first `Check` emits
+  `host_disk_critical`. **Companion red-proof:** a seed-all (transition-only) model — emulated by
+  pre-seeding the breached host's state to `critical` — stays **silent** on the born-breach; the real
+  (unseeded) design emits. That gap is the bug this design fixes.
+- `TestHostDiskChecker_ThresholdDefaults` — unset/invalid/misordered config → 90/95 (no silence/inversion).
+
+## Deploy + live validation (real operator alert)
+Deployed via the repo's GitOps path (image build+push on 180; `manifests/hub.yaml` 0.22.1→0.23.0; ArgoCD
+hard-refresh → `OutOfSync` → sync → `Synced/Healthy`). Used the **manifest path, not bare `kubectl set
+image`**, because the repo convention is that a bare set-image is reverted on the next sync.
+
+**Method:** the cheapest real path from §13 — felhom-pve's host root was **already at ~96–100%** (the actual
+hazard), so no artificial fill and no threshold change were needed. Startup + first-tick logs:
+```
+[INFO] felhom-hub 0.23.0 starting
+[INFO] Host disk checker initialized: warn=90% crit=95%, 0 ok seeded, 1 already-breached left unseeded (first Check emits)
+[INFO] Host disk: demo-felhom-01 root 96% unknown→critical (host_disk_critical)
+[INFO] Operator email sent for demo-felhom/host_disk_critical
+```
+The born-persistent alert fired on cycle 1 and a **real operator email was sent via Resend** (the
+send-success log line — same proof standard as the prior capability alert). Pod `Running 1/1`, no errors.
+
+## Hazard relieved (operational)
+The host root was genuinely full (85G of vzdump backups — ~18 copies of demo guest 9201 at ~3.9G each, on
+the root fs). I pruned the old backups, **keeping the 2 most recent** (regenerable demo-guest backups):
+root **100% → 24%** (freed ~69G). The recovery/re-arm then reflects on the agent's next 15-min report (disk
+now below warn); the recovery logic itself is unit-tested.
+
+## Observations / follow-ups (noted, not built)
+- **Per-storage `StorageTargets` worst-fill alerting** is the easy follow-on (a dedicated dump/backup
+  storage filling). Host root `disk_percent` already covered the observed vzdump-on-root case, so root is
+  the v1 headline.
+- **The retention *fix*** — a provisioning-side `prune-backups` (vzdump `--maxfiles`/`prune-backups`)
+  default so a box can't refill its own root — is a separate small operational follow-up (agent/golden
+  side). The manual prune above was a one-off relief, not the durable fix.
+
+No secrets in any committed file. Operator email address is the operator's own (not a secret); the Resend
+key stays injected from `Secret/resend-api`.