hub v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5)

The operator sees the shared pool box's real state on the hub: total box fill vs
capacity, Σ(shared soft quotas) vs capacity (the oversubscription ratio), per-customer
usage/quota bars, and a box-level operator alert (fill % + oversub ratio) on the existing
dispatcher's operator channel. Per-customer fill alerts already existed; the box-level
aggregate was the gap. READ-ONLY against Hetzner (GET only).

Phase-0 probe (gate PASSED): the live pool box 611714 returns capacity via
storage_box_type.size (1 TiB / bx11) and usage via a stats object (size/size_data/
size_snapshots), all bytes; our token reads it (200).

- hetznerapi: additive StorageBoxType + StorageBoxStats on StorageBox (no existing field/
  method changed); fake carries them + a GetBoxCalls counter; golden decode test.
- monitor.OffsiteBoxChecker: OffsiteChecker-sibling for the box; fetch-throttled (1 GET/
  15min), cached BoxSnapshot, escalation-only + recovery re-arm. FILL (used/capacity 80/90)
  + OVERSUB (Σ shared+enabled quotas / capacity, 2.0x) — independent. Σ from the ConfigJSON
  Descriptor (offsite.ReadDescriptor, new), never the report echo; dedicated+disabled
  excluded. Scope "pool-box" -> operator channel only, no SaveEvent. Failed fetch keeps the
  last snapshot degraded; missing data never becomes 0% and never transitions a band.
- config: Alerting.OffsiteBoxFill{Warn,Crit}Percent + OffsiteOversubWarnRatio (80/90/2.0
  defaults; thresholds pending Viktor's ruling). Constructed in the HETZNER_TOKEN branch,
  60s sweep, snapshot handed to the web server.
- web: Offsite-tab panel (fill bar, Σ+ratio, per-customer usage/quota rows) + a compact
  dashboard tile; reads the cached snapshot only, never fetches; nil -> "not configured".

Tests: 10 new + 4 red-proofs (throttle, Σ filter, escalation-only, failed-fetch honesty),
all confirmed red then restored. go build/vet/test all pass; hub confirm gate OK.
This commit is contained in:
2026-07-17 20:15:34 +02:00
parent 85a14192e7
commit 4bb2df0dc4
17 changed files with 1034 additions and 17 deletions
+6
View File
@@ -23,10 +23,12 @@ type Fake struct {
BoxResetCalls int
DeletedSubaccounts int
DeletedBoxes int
GetBoxCalls int // v0.64.0: the fetch-throttle assertion (Scenario A) counts these
// Failure injection.
FailCreate error // if set, CreateSubaccount/CreateStorageBox return this error
FailAction bool // if set, created actions come back status "error" (WaitAction fails)
FailGetBox error // v0.64.0: if set, GetStorageBox returns this (the box-poll failure path, Scenario D)
}
// NewFake returns an empty Fake.
@@ -145,6 +147,10 @@ func (f *Fake) ListStorageBoxes(_ context.Context, sel string) ([]StorageBox, er
func (f *Fake) GetStorageBox(_ context.Context, boxID int64) (StorageBox, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.GetBoxCalls++
if f.FailGetBox != nil {
return StorageBox{}, f.FailGetBox
}
b, ok := f.Boxes[boxID]
if !ok {
return StorageBox{}, fmt.Errorf("hetznerapi(fake): box %d not found", boxID)