hub v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5)

The operator sees the shared pool box's real state on the hub: total box fill vs
capacity, Σ(shared soft quotas) vs capacity (the oversubscription ratio), per-customer
usage/quota bars, and a box-level operator alert (fill % + oversub ratio) on the existing
dispatcher's operator channel. Per-customer fill alerts already existed; the box-level
aggregate was the gap. READ-ONLY against Hetzner (GET only).

Phase-0 probe (gate PASSED): the live pool box 611714 returns capacity via
storage_box_type.size (1 TiB / bx11) and usage via a stats object (size/size_data/
size_snapshots), all bytes; our token reads it (200).

- hetznerapi: additive StorageBoxType + StorageBoxStats on StorageBox (no existing field/
  method changed); fake carries them + a GetBoxCalls counter; golden decode test.
- monitor.OffsiteBoxChecker: OffsiteChecker-sibling for the box; fetch-throttled (1 GET/
  15min), cached BoxSnapshot, escalation-only + recovery re-arm. FILL (used/capacity 80/90)
  + OVERSUB (Σ shared+enabled quotas / capacity, 2.0x) — independent. Σ from the ConfigJSON
  Descriptor (offsite.ReadDescriptor, new), never the report echo; dedicated+disabled
  excluded. Scope "pool-box" -> operator channel only, no SaveEvent. Failed fetch keeps the
  last snapshot degraded; missing data never becomes 0% and never transitions a band.
- config: Alerting.OffsiteBoxFill{Warn,Crit}Percent + OffsiteOversubWarnRatio (80/90/2.0
  defaults; thresholds pending Viktor's ruling). Constructed in the HETZNER_TOKEN branch,
  60s sweep, snapshot handed to the web server.
- web: Offsite-tab panel (fill bar, Σ+ratio, per-customer usage/quota rows) + a compact
  dashboard tile; reads the cached snapshot only, never fetches; nil -> "not configured".

Tests: 10 new + 4 red-proofs (throttle, Σ filter, escalation-only, failed-fetch honesty),
all confirmed red then restored. go build/vet/test all pass; hub confirm gate OK.
This commit is contained in:
2026-07-17 20:15:34 +02:00
parent 85a14192e7
commit 4bb2df0dc4
17 changed files with 1034 additions and 17 deletions
+43
View File
@@ -1,5 +1,48 @@
# Felhom Hub — Changelog
## v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5) (2026-07-17)
Ships **R-5**: the operator sees the shared pool box's real state on the hub — **total box fill vs
capacity**, **Σ(shared soft quotas) vs capacity** (the oversubscription ratio), **per-customer usage/quota
bars**, and a **box-level operator alert** (fill % + oversubscription ratio) riding the existing
dispatcher's operator channel. Per-customer fill alerts already existed (OffsiteChecker, 90/95% of each
quota); the box-level aggregate was the gap — the operator's early warning that the *pool itself* is
filling, before any single customer breaches. All READ-ONLY against Hetzner (GET only). Green:
`go build/vet/test` all pass; hub confirm gate OK.
- **Phase-0 probe (gate, PASSED):** one authenticated GET of the live pool box (611714) pinned the API
shape — capacity is `storage_box_type.size` (1 TiB for bx11), usage is a `stats` object
(`size`/`size_data`/`size_snapshots`), all bytes; our token reads it (200). The type extension mirrors it.
- **`hetznerapi` (`hetznerapi.go`, `fake.go`):** additive `StorageBoxType` + `StorageBoxStats` sub-structs
on `StorageBox` (no existing field/method changed); fake carries them + a `GetBoxCalls` counter. Golden
decode test against the (redacted) probe capture.
- **`monitor.OffsiteBoxChecker` (`offsite_box.go`, new):** the OffsiteChecker-sibling for the box as a
whole — **fetch-throttled** (one Hetzner GET per 15 min; ≈4/hour, never per-sweep or per-page-load),
cached `BoxSnapshot`, escalation-only emits with silent recovery re-arm. Two independent signals: FILL
(used/capacity, warn 80% / crit 90%) and OVERSUBSCRIPTION (Σ shared+enabled quotas / capacity, warn
2.0×) — both can fire, neither masks the other. Σ(quota) is read from the authoritative ConfigJSON
`Descriptor` (`offsite.ReadDescriptor`, new), NEVER the report echo; dedicated + disabled customers
excluded. Events carry the customer-less scope `"pool-box"` → operator channel ONLY (`processCustomer`
no-ops on it) and are NOT SaveEvent'd (no customer row to key them). A failed fetch keeps the last
snapshot marked degraded — missing data never becomes 0% and never drives a band transition.
- **Config (`cmd/hub/main.go`):** `Alerting.OffsiteBoxFillWarnPercent` (80) / `OffsiteBoxFillCritPercent`
(90) / `OffsiteOversubWarnRatio` (2.0), plumbed like `StorageFill*`. The checker is constructed inside
the existing `HETZNER_TOKEN` branch (shares the client), registered in the 60 s sweep, and its snapshot
handed to the web server. **Thresholds are Claude's encoding of the starter suggestion — Viktor's ruling
pending; the named keys are the one-line flip.**
- **Web surfaces (`web/offsite_box.go` new, `templates/offsite.html`, `dashboard.html`, `style.css`):** an
Offsite-tab panel (capacity, used with data/snapshot split, fill bar, Σ quotas + ratio, fetched-at, +
per-customer rows sorted by usage — shared with a usage/quota bar, dedicated listed without one, no-report
customers show "no usage reported yet") and a compact Dashboard tile (`fill% · ratio`, band-colored,
linking to /offsite). The web layer reads only the cached snapshot — it NEVER fetches. Nil provider → both
render an honest "not configured". Exception color (neutral/amber/red, no green).
- **Tests + red-proofs (run-fail-restore):** 10 new tests (throttle ≈4-not-≈60, Σ/ratio truth table,
band transitions incl. in-band no-re-emit + recovery re-arm + oversub independence + pool-box operator-only
scope, failed-fetch honesty, zero-capacity guard, golden decode, panel render × "not configured"/"with
data"/"no usage reported yet"). Red-proofs: (i) drop the throttle → ≈60 calls; (ii) sum dedicated/disabled
→ wrong Σ; (iii) drop the escalation-only guard → in-band re-emit; (iv) zero the snapshot on a failed fetch
→ lost last-known. All confirmed red, then restored.
## v0.63.0 — system-initiated immediacy: wire the proven poke/bump notifiers into every mutation site that lacked one (2026-07-17)
The immediate-sync arc (Dir-1 trigger, Dir-2b wait channel, Dir-2a agent poke) covered only