hub v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5)

The operator sees the shared pool box's real state on the hub: total box fill vs
capacity, Σ(shared soft quotas) vs capacity (the oversubscription ratio), per-customer
usage/quota bars, and a box-level operator alert (fill % + oversub ratio) on the existing
dispatcher's operator channel. Per-customer fill alerts already existed; the box-level
aggregate was the gap. READ-ONLY against Hetzner (GET only).

Phase-0 probe (gate PASSED): the live pool box 611714 returns capacity via
storage_box_type.size (1 TiB / bx11) and usage via a stats object (size/size_data/
size_snapshots), all bytes; our token reads it (200).

- hetznerapi: additive StorageBoxType + StorageBoxStats on StorageBox (no existing field/
  method changed); fake carries them + a GetBoxCalls counter; golden decode test.
- monitor.OffsiteBoxChecker: OffsiteChecker-sibling for the box; fetch-throttled (1 GET/
  15min), cached BoxSnapshot, escalation-only + recovery re-arm. FILL (used/capacity 80/90)
  + OVERSUB (Σ shared+enabled quotas / capacity, 2.0x) — independent. Σ from the ConfigJSON
  Descriptor (offsite.ReadDescriptor, new), never the report echo; dedicated+disabled
  excluded. Scope "pool-box" -> operator channel only, no SaveEvent. Failed fetch keeps the
  last snapshot degraded; missing data never becomes 0% and never transitions a band.
- config: Alerting.OffsiteBoxFill{Warn,Crit}Percent + OffsiteOversubWarnRatio (80/90/2.0
  defaults; thresholds pending Viktor's ruling). Constructed in the HETZNER_TOKEN branch,
  60s sweep, snapshot handed to the web server.
- web: Offsite-tab panel (fill bar, Σ+ratio, per-customer usage/quota rows) + a compact
  dashboard tile; reads the cached snapshot only, never fetches; nil -> "not configured".

Tests: 10 new + 4 red-proofs (throttle, Σ filter, escalation-only, failed-fetch honesty),
all confirmed red then restored. go build/vet/test all pass; hub confirm gate OK.
This commit is contained in:
2026-07-17 20:15:34 +02:00
parent 85a14192e7
commit 4bb2df0dc4
17 changed files with 1034 additions and 17 deletions
+21
View File
@@ -65,6 +65,13 @@ type Config struct {
// the host-root thresholds above so a single storage's policy can be tuned separately.
StorageFillWarnPercent float64 `yaml:"storage_fill_warn_percent"`
StorageFillCritPercent float64 `yaml:"storage_fill_crit_percent"`
// Offsite POOL-BOX aggregate thresholds (v0.64.0, R-5). Fill = box used vs capacity (percent);
// oversub = Σ(shared soft quotas)/capacity (ratio). Empty/0/invalid → defaults 80/90/2.0 (the
// checker normalizes). Threshold VALUES are Claude's encoding of the starter suggestion — Viktor's
// ruling pending; these keys are the one-line flip when ruled.
OffsiteBoxFillWarnPercent float64 `yaml:"offsite_box_fill_warn_percent"`
OffsiteBoxFillCritPercent float64 `yaml:"offsite_box_fill_crit_percent"`
OffsiteOversubWarnRatio float64 `yaml:"offsite_oversub_warn_ratio"`
} `yaml:"alerting"`
Registry struct {
Image string `yaml:"image"`
@@ -291,6 +298,7 @@ func main() {
// PREREQUISITE: this MUST be a token scoped to a DEDICATED Hetzner project (the shared project token can
// delete ep0 — SPIKE §6). Base is api.hetzner.com (NOT api.hetzner.cloud). Absent token → offsite UI
// still renders, but saving with offsite enabled returns "not configured".
var offsiteBoxChecker *monitor.OffsiteBoxChecker
if tok := os.Getenv("HETZNER_TOKEN"); tok != "" {
poolBoxID, _ := strconv.ParseInt(os.Getenv("HETZNER_POOL_BOX_ID"), 10, 64)
location := os.Getenv("HETZNER_LOCATION")
@@ -302,6 +310,16 @@ func main() {
API: client, Store: dataStore, Scanner: offsite.SSHHostKeyScanner{}, PoolBoxID: poolBoxID, Location: location, Logger: logger,
})
logger.Printf("[INFO] Offsite provisioning enabled (pool_box=%d, location=%s)", poolBoxID, location)
// R-5 (v0.64.0): the pool-box aggregate checker shares the SAME client + pool box id (GET-only).
// It needs a valid box id to poll; without one, the aggregate stays unconfigured.
if poolBoxID != 0 {
offsiteBoxChecker = monitor.NewOffsiteBoxChecker(client, poolBoxID, dataStore,
cfg.Alerting.OffsiteBoxFillWarnPercent, cfg.Alerting.OffsiteBoxFillCritPercent, cfg.Alerting.OffsiteOversubWarnRatio,
dispatcher.ProcessEvent, logger)
webServer.SetOffsiteBox(offsiteBoxChecker.Snapshot)
} else {
logger.Printf("[INFO] Offsite pool-box aggregate disabled (HETZNER_POOL_BOX_ID unset)")
}
}
// v0.57.0 (F3) — clean-slate re-enrollment auto re-issues offsite credentials to the fresh box.
@@ -530,6 +548,9 @@ func main() {
hostMgmtPlaneChecker.Check()
hostOOBChecker.Check()
offsiteChecker.Check()
if offsiteBoxChecker != nil {
offsiteBoxChecker.Check() // R-5: pool-box aggregate (fetch-throttled internally)
}
// v0.46.0: pulled log bundles are transient diagnostics — 72 h TTL.
if n, perr := dataStore.PurgeExpiredLogBundles(time.Now()); perr != nil {
logger.Printf("[WARN] log-bundle TTL purge failed: %v", perr)