hub v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5)
The operator sees the shared pool box's real state on the hub: total box fill vs
capacity, Σ(shared soft quotas) vs capacity (the oversubscription ratio), per-customer
usage/quota bars, and a box-level operator alert (fill % + oversub ratio) on the existing
dispatcher's operator channel. Per-customer fill alerts already existed; the box-level
aggregate was the gap. READ-ONLY against Hetzner (GET only).
Phase-0 probe (gate PASSED): the live pool box 611714 returns capacity via
storage_box_type.size (1 TiB / bx11) and usage via a stats object (size/size_data/
size_snapshots), all bytes; our token reads it (200).
- hetznerapi: additive StorageBoxType + StorageBoxStats on StorageBox (no existing field/
method changed); fake carries them + a GetBoxCalls counter; golden decode test.
- monitor.OffsiteBoxChecker: OffsiteChecker-sibling for the box; fetch-throttled (1 GET/
15min), cached BoxSnapshot, escalation-only + recovery re-arm. FILL (used/capacity 80/90)
+ OVERSUB (Σ shared+enabled quotas / capacity, 2.0x) — independent. Σ from the ConfigJSON
Descriptor (offsite.ReadDescriptor, new), never the report echo; dedicated+disabled
excluded. Scope "pool-box" -> operator channel only, no SaveEvent. Failed fetch keeps the
last snapshot degraded; missing data never becomes 0% and never transitions a band.
- config: Alerting.OffsiteBoxFill{Warn,Crit}Percent + OffsiteOversubWarnRatio (80/90/2.0
defaults; thresholds pending Viktor's ruling). Constructed in the HETZNER_TOKEN branch,
60s sweep, snapshot handed to the web server.
- web: Offsite-tab panel (fill bar, Σ+ratio, per-customer usage/quota rows) + a compact
dashboard tile; reads the cached snapshot only, never fetches; nil -> "not configured".
Tests: 10 new + 4 red-proofs (throttle, Σ filter, escalation-only, failed-fetch honesty),
all confirmed red then restored. go build/vet/test all pass; hub confirm gate OK.
This commit is contained in:
@@ -65,6 +65,13 @@ type Config struct {
|
||||
// the host-root thresholds above so a single storage's policy can be tuned separately.
|
||||
StorageFillWarnPercent float64 `yaml:"storage_fill_warn_percent"`
|
||||
StorageFillCritPercent float64 `yaml:"storage_fill_crit_percent"`
|
||||
// Offsite POOL-BOX aggregate thresholds (v0.64.0, R-5). Fill = box used vs capacity (percent);
|
||||
// oversub = Σ(shared soft quotas)/capacity (ratio). Empty/0/invalid → defaults 80/90/2.0 (the
|
||||
// checker normalizes). Threshold VALUES are Claude's encoding of the starter suggestion — Viktor's
|
||||
// ruling pending; these keys are the one-line flip when ruled.
|
||||
OffsiteBoxFillWarnPercent float64 `yaml:"offsite_box_fill_warn_percent"`
|
||||
OffsiteBoxFillCritPercent float64 `yaml:"offsite_box_fill_crit_percent"`
|
||||
OffsiteOversubWarnRatio float64 `yaml:"offsite_oversub_warn_ratio"`
|
||||
} `yaml:"alerting"`
|
||||
Registry struct {
|
||||
Image string `yaml:"image"`
|
||||
@@ -291,6 +298,7 @@ func main() {
|
||||
// PREREQUISITE: this MUST be a token scoped to a DEDICATED Hetzner project (the shared project token can
|
||||
// delete ep0 — SPIKE §6). Base is api.hetzner.com (NOT api.hetzner.cloud). Absent token → offsite UI
|
||||
// still renders, but saving with offsite enabled returns "not configured".
|
||||
var offsiteBoxChecker *monitor.OffsiteBoxChecker
|
||||
if tok := os.Getenv("HETZNER_TOKEN"); tok != "" {
|
||||
poolBoxID, _ := strconv.ParseInt(os.Getenv("HETZNER_POOL_BOX_ID"), 10, 64)
|
||||
location := os.Getenv("HETZNER_LOCATION")
|
||||
@@ -302,6 +310,16 @@ func main() {
|
||||
API: client, Store: dataStore, Scanner: offsite.SSHHostKeyScanner{}, PoolBoxID: poolBoxID, Location: location, Logger: logger,
|
||||
})
|
||||
logger.Printf("[INFO] Offsite provisioning enabled (pool_box=%d, location=%s)", poolBoxID, location)
|
||||
// R-5 (v0.64.0): the pool-box aggregate checker shares the SAME client + pool box id (GET-only).
|
||||
// It needs a valid box id to poll; without one, the aggregate stays unconfigured.
|
||||
if poolBoxID != 0 {
|
||||
offsiteBoxChecker = monitor.NewOffsiteBoxChecker(client, poolBoxID, dataStore,
|
||||
cfg.Alerting.OffsiteBoxFillWarnPercent, cfg.Alerting.OffsiteBoxFillCritPercent, cfg.Alerting.OffsiteOversubWarnRatio,
|
||||
dispatcher.ProcessEvent, logger)
|
||||
webServer.SetOffsiteBox(offsiteBoxChecker.Snapshot)
|
||||
} else {
|
||||
logger.Printf("[INFO] Offsite pool-box aggregate disabled (HETZNER_POOL_BOX_ID unset)")
|
||||
}
|
||||
}
|
||||
|
||||
// v0.57.0 (F3) — clean-slate re-enrollment auto re-issues offsite credentials to the fresh box.
|
||||
@@ -530,6 +548,9 @@ func main() {
|
||||
hostMgmtPlaneChecker.Check()
|
||||
hostOOBChecker.Check()
|
||||
offsiteChecker.Check()
|
||||
if offsiteBoxChecker != nil {
|
||||
offsiteBoxChecker.Check() // R-5: pool-box aggregate (fetch-throttled internally)
|
||||
}
|
||||
// v0.46.0: pulled log bundles are transient diagnostics — 72 h TTL.
|
||||
if n, perr := dataStore.PurgeExpiredLogBundles(time.Now()); perr != nil {
|
||||
logger.Printf("[WARN] log-bundle TTL purge failed: %v", perr)
|
||||
|
||||
Reference in New Issue
Block a user