hub v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5)

The operator sees the shared pool box's real state on the hub: total box fill vs
capacity, Σ(shared soft quotas) vs capacity (the oversubscription ratio), per-customer
usage/quota bars, and a box-level operator alert (fill % + oversub ratio) on the existing
dispatcher's operator channel. Per-customer fill alerts already existed; the box-level
aggregate was the gap. READ-ONLY against Hetzner (GET only).

Phase-0 probe (gate PASSED): the live pool box 611714 returns capacity via
storage_box_type.size (1 TiB / bx11) and usage via a stats object (size/size_data/
size_snapshots), all bytes; our token reads it (200).

- hetznerapi: additive StorageBoxType + StorageBoxStats on StorageBox (no existing field/
  method changed); fake carries them + a GetBoxCalls counter; golden decode test.
- monitor.OffsiteBoxChecker: OffsiteChecker-sibling for the box; fetch-throttled (1 GET/
  15min), cached BoxSnapshot, escalation-only + recovery re-arm. FILL (used/capacity 80/90)
  + OVERSUB (Σ shared+enabled quotas / capacity, 2.0x) — independent. Σ from the ConfigJSON
  Descriptor (offsite.ReadDescriptor, new), never the report echo; dedicated+disabled
  excluded. Scope "pool-box" -> operator channel only, no SaveEvent. Failed fetch keeps the
  last snapshot degraded; missing data never becomes 0% and never transitions a band.
- config: Alerting.OffsiteBoxFill{Warn,Crit}Percent + OffsiteOversubWarnRatio (80/90/2.0
  defaults; thresholds pending Viktor's ruling). Constructed in the HETZNER_TOKEN branch,
  60s sweep, snapshot handed to the web server.
- web: Offsite-tab panel (fill bar, Σ+ratio, per-customer usage/quota rows) + a compact
  dashboard tile; reads the cached snapshot only, never fetches; nil -> "not configured".

Tests: 10 new + 4 red-proofs (throttle, Σ filter, escalation-only, failed-fetch honesty),
all confirmed red then restored. go build/vet/test all pass; hub confirm gate OK.
This commit is contained in:
2026-07-17 20:15:34 +02:00
parent 85a14192e7
commit 4bb2df0dc4
17 changed files with 1034 additions and 17 deletions
+39
View File
@@ -34,6 +34,45 @@
<div class="flash flash-success">Endpoint deleted.</div>
{{end}}
<!-- R-5 (v0.64.0): shared pool-box aggregate — total fill, oversubscription, per-customer usage -->
<section class="card" style="margin-bottom: 1.5rem;">
<h3 style="margin: 0 0 0.75rem;">Offsite pool box</h3>
{{if not .OffsiteBox.Configured}}
<p class="text-muted" style="font-size: 0.9em;">Offsite pool metrics not configured (no Hetzner token / pool box id on this hub).</p>
{{else if .OffsiteBox.Pending}}
<p class="text-muted" style="font-size: 0.9em;">Pool metrics loading — the first refresh has not landed yet.</p>
{{else}}
<table class="detail-table">
<tr><th style="width: 12rem;">Box</th><td>{{.OffsiteBox.BoxType}}</td></tr>
<tr><th>Capacity</th><td>{{.OffsiteBox.CapacityStr}}</td></tr>
<tr><th>Used</th><td>{{.OffsiteBox.UsedStr}} &middot; {{formatFloat .OffsiteBox.FillPercent}}% full <span class="text-muted">(data {{.OffsiteBox.DataStr}} + snapshots {{.OffsiteBox.SnapshotStr}})</span></td></tr>
</table>
<div class="bar" style="margin: 0.4rem 0 0.9rem;"><div class="bar-fill bar-{{.OffsiteBox.FillBand}}" style="width: {{formatFloat .OffsiteBox.FillPercent}}%;"></div></div>
<table class="detail-table">
<tr><th style="width: 12rem;">&Sigma; shared soft quotas</th><td>{{.OffsiteBox.SumQuotaStr}}</td></tr>
<tr><th>Oversubscription</th><td><span class="status-badge status-badge-{{if eq .OffsiteBox.OversubBand "warning"}}warn{{else}}ok{{end}}">{{.OffsiteBox.RatioStr}}</span> <span class="text-muted">&Sigma;(shared quotas) &divide; capacity</span></td></tr>
<tr><th>Fetched</th><td>{{timeAgo .OffsiteBox.FetchedAt}}{{if .OffsiteBox.Degraded}} <span class="status-badge status-badge-warn">STALE — last refresh failed</span>{{end}}</td></tr>
</table>
{{if .OffsiteCusts}}
<h4 style="margin: 1rem 0 0.5rem; font-size: 0.85rem; color: var(--text-3); text-transform: uppercase; letter-spacing: 0.03em;">Per-customer usage</h4>
<table class="container-table">
<thead><tr><th>Customer</th><th>Usage</th><th>Quota</th><th style="width: 35%;"></th></tr></thead>
<tbody>
{{range .OffsiteCusts}}
<tr>
<td>{{.CustomerName}}{{if .Dedicated}} <span class="text-muted">(dedicated)</span>{{end}}</td>
<td>{{if .NoUsage}}<span class="text-muted">no usage reported yet</span>{{else}}{{.UsageStr}}{{end}}</td>
<td>{{if .Dedicated}}<span class="text-muted">dedicated</span>{{else if .QuotaStr}}{{.QuotaStr}}{{else}}<span class="text-muted"></span>{{end}}</td>
<td>{{if .HasBar}}<div class="bar"><div class="bar-fill bar-{{.UsageBand}}" style="width: {{formatFloat .UsagePercent}}%;"></div></div>{{end}}</td>
</tr>
{{end}}
</tbody>
</table>
{{end}}
{{end}}
</section>
{{if .HasEndpoints}}
{{range .Endpoints}}
<section class="card" style="margin-bottom: 1.5rem;"