docs: hub REPORT for v0.26.0 (config delivery + inbound retire, live-validated)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
This commit is contained in:
@@ -4,73 +4,52 @@
|
||||
|
||||
---
|
||||
|
||||
# REPORT — hub v0.25.0: per-storage worst-fill alerting (StorageFillChecker)
|
||||
# REPORT — Hub GUI correctness + pull-based config delivery (hub v0.26.0)
|
||||
|
||||
**Date:** 2026-06-30
|
||||
**Task:** `TASK.md` — per-storage worst-fill alerting (hub checker). Item 2 of 3 in the post-NAS queue
|
||||
(host_disk ✅ → **this** → provisioning vzdump retention default).
|
||||
**Date:** 2026-06-30 · **Repo:** `felhom.eu` (`hub/`) · **Companion:** felhom-controller v0.94.0
|
||||
|
||||
## Baseline
|
||||
| | value |
|
||||
|---|---|
|
||||
| Hub before | v0.24.0 |
|
||||
| Hub after | **v0.25.0** — code `88073ac`, manifest bump `063fb0c` |
|
||||
| Live image | `felhom-hub:0.25.0` (ArgoCD `felhom` Synced/Healthy) |
|
||||
Implements the "retire the 5 inbound controls + pull-based config-refresh" task. Resolves audit
|
||||
`documentation/audits/AUDIT-hub-gui-2026-06-30.md` F-S1/F-S4 + the dead-template findings.
|
||||
|
||||
## What changed
|
||||
Generalizes the host-root disk alert (v0.23.0) to **any reported storage target** — a dump/backup volume,
|
||||
data drive, lvmthin pool, or PBS datastore filling pages the operator with the storage named, even when
|
||||
host root is fine.
|
||||
- **`internal/monitor/storage_fill.go` (NEW) — `StorageFillChecker`.** A per-target mirror of
|
||||
`HostDiskChecker` on the same 60s sweep: born/persistent (already-breached `(host,target)` keys left
|
||||
UNSEEDED → first `Check` emits), escalation-only emit, recovery re-arm, 1h cooldown. State keyed per
|
||||
**(host, target)** so targets alert independently. Emits `storage_fill_warning`/`storage_fill_critical`
|
||||
at the **natural `critical` severity** (second real caller exercising the v0.24.0 dispatcher fix).
|
||||
Thresholds default 90/95, hub-config overridable (`alerting.storage_fill_warn_percent`/`_crit_percent`),
|
||||
independent of the host-root thresholds.
|
||||
- **Root excluded (no double-alert):** the root-backed builtin (`Type=="local"`, or a target mounted at
|
||||
`/`) is skipped — `HostDiskChecker` owns root.
|
||||
- **`internal/store/store.go`:** `GetHostStorageTargets()` + `HostStorageTargetRow` — parses
|
||||
`report_json.storage_targets[]` of each host's latest report (percent = `used_fraction`×100); modeled on
|
||||
`GetHostDiskUsage`, no denorm column / migration.
|
||||
- **`internal/notify/templates.go` + `internal/api/handler.go`:** Hungarian templates + allowlist entries.
|
||||
- **`cmd/hub/main.go`:** register `storageFillChecker` on the 60s tick beside `HostDiskChecker`.
|
||||
## Baselines
|
||||
- hub `felhom.eu` was `e51e03b` (v0.25.0) → code `a3ac6c9`, manifest `d88b43a` (**v0.26.0**, live on k3s).
|
||||
- (companion) felhom-controller v0.93.0 → v0.94.0.
|
||||
|
||||
## Tests — `go build ./... && go vet ./... && go test ./...` GREEN
|
||||
- per-target bands (A breaches → only A warns; B silent; A escalates warning→critical; A recovers →
|
||||
clears+re-arms; re-arm fires again).
|
||||
- **born/persistent companion red-proof** — a seed-all/transition-only model (pre-seed the breached key)
|
||||
stays silent on the born-breach; the real (unseeded) design emits.
|
||||
- **root-exclusion companion** — a root-backed/`/`-mounted target emits nothing; the control asserts 96% IS
|
||||
the critical band, so only the exclusion suppresses the double-alert; a real off-root storage is NOT
|
||||
excluded.
|
||||
- severity is natural `critical`; `GetHostStorageTargets` parse fixture.
|
||||
## What changed (hub)
|
||||
- **Pull-based config delivery.** New `customer_configs.config_version` (stored counter, NOT a YAML
|
||||
hash — `configgen` emits a fresh `session_secret`+timestamp each call). `SaveCustomerConfig` bumps it
|
||||
on every save (new=1, updates increment). The report ACK (`handleReport`) advertises `config_version`
|
||||
beside the floor/latest; omitted for report-only customers.
|
||||
- **Retired the 5 inbound (hub→box) controls** (`01-topology-and-trust.md:11` never-inbound):
|
||||
Trigger Update, Push Config, Pull Config, Show Diff (+`compareYAMLValues`/`flattenYAML`/`maskSensitive`
|
||||
helpers + dead `ConfigSyncStatus`/`ConfigDiffCount`) — handlers + routes + buttons + JS deleted.
|
||||
Geo-disable KEEPS its hub→Cloudflare `RemoveGeoRules`; its inbound `notifyControllerGeoDisable` deleted.
|
||||
- **GUI staleness + dead templates.** Setup Commands now show `sudo ./felhom-host-install.sh
|
||||
--customer-id <id>` (no-echo passphrase) instead of `docker-setup.sh`; Option 2 relabelled
|
||||
"Manual config fetch (debug only)". Deleted orphaned `customer.html` + `config_detail.html`.
|
||||
|
||||
## Deploy + live validation
|
||||
Deployed via the GitOps manifest path (build+push; `manifests/hub.yaml` 0.24.0→0.25.0; ArgoCD
|
||||
hard-refresh → sync → **Synced/Healthy**). Startup confirmed the checker registered beside HostDiskChecker:
|
||||
`Storage fill checker initialized: warn=90% crit=95%, 4 ok seeded, 0 already-breached, 1 root-backed
|
||||
excluded` — i.e. the 5 reported storages parsed, the root-backed `local` (22%) correctly excluded, 4
|
||||
off-root tracked.
|
||||
## Files
|
||||
`internal/store/store.go` (column + bump + 3 SELECTs + struct), `internal/api/handler.go` (ACK),
|
||||
`internal/web/server.go` (routes + handleTriggerUpdate removed), `internal/web/configs.go`
|
||||
(handlers/helpers removed, geo trimmed), `internal/web/templates/customer_unified.html` (setup cmd,
|
||||
buttons/JS removed), deleted `customer.html`+`config_detail.html`, `manifests/hub.yaml` (tag).
|
||||
Tests: `internal/store/config_version_test.go`, `internal/api/config_version_ack_test.go`.
|
||||
|
||||
**Method (real alert, no artificial fill):** felhom-pve's reported off-root targets were felhom-pbs 21.75%,
|
||||
local-lvm 11.2%, felhom-flash/usb ~0%. I briefly set `alerting.storage_fill_warn_percent: 20` via the
|
||||
hub-config ConfigMap (commit `5a3747f`) + restarted the hub, so the **real** felhom-pbs fill crossed:
|
||||
```
|
||||
Storage fill checker initialized: warn=20% crit=95%, 3 ok seeded, 1 already-breached left unseeded, 1 root-backed excluded
|
||||
Storage fill: demo-felhom-01 "felhom-pbs" 22% unknown→warning (storage_fill_warning)
|
||||
Operator email sent for demo-felhom/storage_fill_warning
|
||||
```
|
||||
The alert **named the specific target** (felhom-pbs) + its %, fired **born-persistent on cycle 1** (pbs was
|
||||
over the lowered threshold at restart), and a **real operator email** was sent. The threshold was then
|
||||
**restored to the default** (commit `e1dd06e`); the hub re-seeded `warn=90% … 4 ok` (felhom-pbs back to ok).
|
||||
## Tests & green gate
|
||||
- `go build ./... && go vet ./... && go test ./...` — **all green**.
|
||||
- **Red-proof (version bump):** with the `config_version + 1` bump defeated, both
|
||||
`TestSaveCustomerConfig_BumpsConfigVersion` and `TestReportACK_ConfigVersion` FAIL
|
||||
(`config_version after edit = 1, want 2`) — the bump is load-bearing. Restored → green.
|
||||
- **No-inbound:** `grep client.Do internal/web/` → only Gitea registry/template fetches; **zero**
|
||||
ControllerURL targets (ControllerURL remains a display-only link).
|
||||
|
||||
## Observations / follow-ups
|
||||
- **Last queued item:** the provisioning-side vzdump retention default (so a box can't refill its own root)
|
||||
— operational, agent/golden side.
|
||||
- Follow-ons noted: **thin-pool *metadata* exhaustion** alerting (v1 alerts on data-fill %; a full thin
|
||||
pool's metadata is a separate failure mode) and **per-storage-type thresholds** (e.g. tighter for PBS).
|
||||
- PBS/network targets are included (a full datastore fails backups); that surfaced as the live test target.
|
||||
## Live validation (demo, hub.felhom.eu + guest 9201)
|
||||
- Hub v0.26.0 rolled out on k3s (Synced/Healthy, `[INFO] felhom-hub … starting`).
|
||||
- Live ACK for demo-felhom: `{"config_version":1,"latest_version":"0.94.0","min_controller_version":"0.87.0","status":"ok"}`.
|
||||
- Bumped demo-felhom `config_version` 1→2 in the hub DB (equivalent to a config save; the SaveCustomerConfig
|
||||
bump itself is unit-tested + red-proofed). The controller detected it, re-pulled, and self-restarted
|
||||
(see controller REPORT for the box-side trace) — full pull-based delivery, no inbound connection.
|
||||
|
||||
No secrets committed. Operator email is the operator's own; the Resend key stays injected from
|
||||
`Secret/resend-api`.
|
||||
## Open follow-ups (from the audit, not in scope here)
|
||||
Hosts page (F-M1); controller-side geo intent sync (geo-disable is now CF-only, no box notify);
|
||||
Show-Diff could return later as a read-only hub-vs-reported view.
|
||||
|
||||
Reference in New Issue
Block a user