diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 71946e2..3af8d53 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,48 @@ # Felhom Hub — Changelog +## v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5) (2026-07-17) + +Ships **R-5**: the operator sees the shared pool box's real state on the hub — **total box fill vs +capacity**, **Σ(shared soft quotas) vs capacity** (the oversubscription ratio), **per-customer usage/quota +bars**, and a **box-level operator alert** (fill % + oversubscription ratio) riding the existing +dispatcher's operator channel. Per-customer fill alerts already existed (OffsiteChecker, 90/95% of each +quota); the box-level aggregate was the gap — the operator's early warning that the *pool itself* is +filling, before any single customer breaches. All READ-ONLY against Hetzner (GET only). Green: +`go build/vet/test` all pass; hub confirm gate OK. + +- **Phase-0 probe (gate, PASSED):** one authenticated GET of the live pool box (611714) pinned the API + shape — capacity is `storage_box_type.size` (1 TiB for bx11), usage is a `stats` object + (`size`/`size_data`/`size_snapshots`), all bytes; our token reads it (200). The type extension mirrors it. +- **`hetznerapi` (`hetznerapi.go`, `fake.go`):** additive `StorageBoxType` + `StorageBoxStats` sub-structs + on `StorageBox` (no existing field/method changed); fake carries them + a `GetBoxCalls` counter. Golden + decode test against the (redacted) probe capture. +- **`monitor.OffsiteBoxChecker` (`offsite_box.go`, new):** the OffsiteChecker-sibling for the box as a + whole — **fetch-throttled** (one Hetzner GET per 15 min; ≈4/hour, never per-sweep or per-page-load), + cached `BoxSnapshot`, escalation-only emits with silent recovery re-arm. Two independent signals: FILL + (used/capacity, warn 80% / crit 90%) and OVERSUBSCRIPTION (Σ shared+enabled quotas / capacity, warn + 2.0×) — both can fire, neither masks the other. Σ(quota) is read from the authoritative ConfigJSON + `Descriptor` (`offsite.ReadDescriptor`, new), NEVER the report echo; dedicated + disabled customers + excluded. Events carry the customer-less scope `"pool-box"` → operator channel ONLY (`processCustomer` + no-ops on it) and are NOT SaveEvent'd (no customer row to key them). A failed fetch keeps the last + snapshot marked degraded — missing data never becomes 0% and never drives a band transition. +- **Config (`cmd/hub/main.go`):** `Alerting.OffsiteBoxFillWarnPercent` (80) / `OffsiteBoxFillCritPercent` + (90) / `OffsiteOversubWarnRatio` (2.0), plumbed like `StorageFill*`. The checker is constructed inside + the existing `HETZNER_TOKEN` branch (shares the client), registered in the 60 s sweep, and its snapshot + handed to the web server. **Thresholds are Claude's encoding of the starter suggestion — Viktor's ruling + pending; the named keys are the one-line flip.** +- **Web surfaces (`web/offsite_box.go` new, `templates/offsite.html`, `dashboard.html`, `style.css`):** an + Offsite-tab panel (capacity, used with data/snapshot split, fill bar, Σ quotas + ratio, fetched-at, + + per-customer rows sorted by usage — shared with a usage/quota bar, dedicated listed without one, no-report + customers show "no usage reported yet") and a compact Dashboard tile (`fill% · ratio`, band-colored, + linking to /offsite). The web layer reads only the cached snapshot — it NEVER fetches. Nil provider → both + render an honest "not configured". Exception color (neutral/amber/red, no green). +- **Tests + red-proofs (run-fail-restore):** 10 new tests (throttle ≈4-not-≈60, Σ/ratio truth table, + band transitions incl. in-band no-re-emit + recovery re-arm + oversub independence + pool-box operator-only + scope, failed-fetch honesty, zero-capacity guard, golden decode, panel render × "not configured"/"with + data"/"no usage reported yet"). Red-proofs: (i) drop the throttle → ≈60 calls; (ii) sum dedicated/disabled + → wrong Σ; (iii) drop the escalation-only guard → in-band re-emit; (iv) zero the snapshot on a failed fetch + → lost last-known. All confirmed red, then restored. + ## v0.63.0 — system-initiated immediacy: wire the proven poke/bump notifiers into every mutation site that lacked one (2026-07-17) The immediate-sync arc (Dir-1 trigger, Dir-2b wait channel, Dir-2a agent poke) covered only diff --git a/hub/cmd/hub/main.go b/hub/cmd/hub/main.go index f5fc22e..ab3fae7 100644 --- a/hub/cmd/hub/main.go +++ b/hub/cmd/hub/main.go @@ -65,6 +65,13 @@ type Config struct { // the host-root thresholds above so a single storage's policy can be tuned separately. StorageFillWarnPercent float64 `yaml:"storage_fill_warn_percent"` StorageFillCritPercent float64 `yaml:"storage_fill_crit_percent"` + // Offsite POOL-BOX aggregate thresholds (v0.64.0, R-5). Fill = box used vs capacity (percent); + // oversub = Σ(shared soft quotas)/capacity (ratio). Empty/0/invalid → defaults 80/90/2.0 (the + // checker normalizes). Threshold VALUES are Claude's encoding of the starter suggestion — Viktor's + // ruling pending; these keys are the one-line flip when ruled. + OffsiteBoxFillWarnPercent float64 `yaml:"offsite_box_fill_warn_percent"` + OffsiteBoxFillCritPercent float64 `yaml:"offsite_box_fill_crit_percent"` + OffsiteOversubWarnRatio float64 `yaml:"offsite_oversub_warn_ratio"` } `yaml:"alerting"` Registry struct { Image string `yaml:"image"` @@ -291,6 +298,7 @@ func main() { // PREREQUISITE: this MUST be a token scoped to a DEDICATED Hetzner project (the shared project token can // delete ep0 — SPIKE §6). Base is api.hetzner.com (NOT api.hetzner.cloud). Absent token → offsite UI // still renders, but saving with offsite enabled returns "not configured". + var offsiteBoxChecker *monitor.OffsiteBoxChecker if tok := os.Getenv("HETZNER_TOKEN"); tok != "" { poolBoxID, _ := strconv.ParseInt(os.Getenv("HETZNER_POOL_BOX_ID"), 10, 64) location := os.Getenv("HETZNER_LOCATION") @@ -302,6 +310,16 @@ func main() { API: client, Store: dataStore, Scanner: offsite.SSHHostKeyScanner{}, PoolBoxID: poolBoxID, Location: location, Logger: logger, }) logger.Printf("[INFO] Offsite provisioning enabled (pool_box=%d, location=%s)", poolBoxID, location) + // R-5 (v0.64.0): the pool-box aggregate checker shares the SAME client + pool box id (GET-only). + // It needs a valid box id to poll; without one, the aggregate stays unconfigured. + if poolBoxID != 0 { + offsiteBoxChecker = monitor.NewOffsiteBoxChecker(client, poolBoxID, dataStore, + cfg.Alerting.OffsiteBoxFillWarnPercent, cfg.Alerting.OffsiteBoxFillCritPercent, cfg.Alerting.OffsiteOversubWarnRatio, + dispatcher.ProcessEvent, logger) + webServer.SetOffsiteBox(offsiteBoxChecker.Snapshot) + } else { + logger.Printf("[INFO] Offsite pool-box aggregate disabled (HETZNER_POOL_BOX_ID unset)") + } } // v0.57.0 (F3) — clean-slate re-enrollment auto re-issues offsite credentials to the fresh box. @@ -530,6 +548,9 @@ func main() { hostMgmtPlaneChecker.Check() hostOOBChecker.Check() offsiteChecker.Check() + if offsiteBoxChecker != nil { + offsiteBoxChecker.Check() // R-5: pool-box aggregate (fetch-throttled internally) + } // v0.46.0: pulled log bundles are transient diagnostics — 72 h TTL. if n, perr := dataStore.PurgeExpiredLogBundles(time.Now()); perr != nil { logger.Printf("[WARN] log-bundle TTL purge failed: %v", perr) diff --git a/hub/internal/hetznerapi/fake.go b/hub/internal/hetznerapi/fake.go index caf43cb..02448ca 100644 --- a/hub/internal/hetznerapi/fake.go +++ b/hub/internal/hetznerapi/fake.go @@ -23,10 +23,12 @@ type Fake struct { BoxResetCalls int DeletedSubaccounts int DeletedBoxes int + GetBoxCalls int // v0.64.0: the fetch-throttle assertion (Scenario A) counts these // Failure injection. FailCreate error // if set, CreateSubaccount/CreateStorageBox return this error FailAction bool // if set, created actions come back status "error" (WaitAction fails) + FailGetBox error // v0.64.0: if set, GetStorageBox returns this (the box-poll failure path, Scenario D) } // NewFake returns an empty Fake. @@ -145,6 +147,10 @@ func (f *Fake) ListStorageBoxes(_ context.Context, sel string) ([]StorageBox, er func (f *Fake) GetStorageBox(_ context.Context, boxID int64) (StorageBox, error) { f.mu.Lock() defer f.mu.Unlock() + f.GetBoxCalls++ + if f.FailGetBox != nil { + return StorageBox{}, f.FailGetBox + } b, ok := f.Boxes[boxID] if !ok { return StorageBox{}, fmt.Errorf("hetznerapi(fake): box %d not found", boxID) diff --git a/hub/internal/hetznerapi/hetznerapi.go b/hub/internal/hetznerapi/hetznerapi.go index 0aae2fc..f931f89 100644 --- a/hub/internal/hetznerapi/hetznerapi.go +++ b/hub/internal/hetznerapi/hetznerapi.go @@ -62,6 +62,29 @@ type StorageBox struct { Status string `json:"status"` // initializing | active | … Server string `json:"server"` // uXXXXXX.your-storagebox.de Labels map[string]string `json:"labels"` + + // StorageBoxType + Stats added v0.64.0 (R-5 pool-box aggregate). Shape pinned by the Phase-0 probe + // of the live pool box 611714 (2026-07-17): GET /storage_boxes/{id} returns a `storage_box_type` + // object whose `size` is the CAPACITY in bytes (1099511627776 = 1 TiB for bx11), and a `stats` object + // carrying total/data/snapshot usage in bytes. Additive — no existing field or method changed. + StorageBoxType StorageBoxType `json:"storage_box_type"` + Stats StorageBoxStats `json:"stats"` +} + +// StorageBoxType is the box's plan; its Size is the CAPACITY in bytes (never use stats.size for capacity). +type StorageBoxType struct { + ID int64 `json:"id"` + Name string `json:"name"` // e.g. "bx11" + Description string `json:"description"` // e.g. "BX11" + Size int64 `json:"size"` // capacity in BYTES +} + +// StorageBoxStats is the box's live usage, all in BYTES (probe-confirmed): total, the data portion, and +// the snapshot portion (Size == SizeData + SizeSnapshots on the live box). +type StorageBoxStats struct { + Size int64 `json:"size"` // total used bytes + SizeData int64 `json:"size_data"` // data bytes + SizeSnapshots int64 `json:"size_snapshots"` // snapshot bytes } // CreateSubaccountRequest — POST /storage_boxes/{box}/subaccounts. Password satisfies the 4-class policy. diff --git a/hub/internal/hetznerapi/storagebox_stats_test.go b/hub/internal/hetznerapi/storagebox_stats_test.go new file mode 100644 index 0000000..427a30d --- /dev/null +++ b/hub/internal/hetznerapi/storagebox_stats_test.go @@ -0,0 +1,45 @@ +package hetznerapi + +import ( + "encoding/json" + "testing" +) + +// goldenPoolBox is the (redacted, non-secret) response captured from the Phase-0 probe of the live pool +// box 611714 (2026-07-17): GET /storage_boxes/{id}. The stats/capacity extension must decode it EXACTLY. +const goldenPoolBox = `{ + "storage_box": { + "id": 611714, + "username": "u629488", + "name": "storage-box-pool-1", + "status": "active", + "server": "u629488.your-storagebox.de", + "storage_box_type": { "id": 1333, "name": "bx11", "description": "BX11", "size": 1099511627776 }, + "stats": { "size": 2746220544, "size_data": 2746220544, "size_snapshots": 0 } + } +}` + +// TestStorageBox_DecodesProbeShape pins the type extension to the live API shape: capacity is the box +// TYPE's size (1 TiB), and stats carries the total/data/snapshot split, all in bytes. +func TestStorageBox_DecodesProbeShape(t *testing.T) { + var out struct { + StorageBox StorageBox `json:"storage_box"` + } + if err := json.Unmarshal([]byte(goldenPoolBox), &out); err != nil { + t.Fatalf("decode golden pool-box: %v", err) + } + b := out.StorageBox + if b.ID != 611714 || b.Name != "storage-box-pool-1" || b.Status != "active" { + t.Fatalf("base fields wrong: %+v", b) + } + if b.StorageBoxType.Name != "bx11" || b.StorageBoxType.Size != 1099511627776 { + t.Fatalf("capacity (storage_box_type.size) wrong: %+v", b.StorageBoxType) + } + if b.Stats.Size != 2746220544 || b.Stats.SizeData != 2746220544 || b.Stats.SizeSnapshots != 0 { + t.Fatalf("stats wrong: %+v", b.Stats) + } + // Sanity: the split sums to the total (the invariant the live box holds). + if b.Stats.SizeData+b.Stats.SizeSnapshots != b.Stats.Size { + t.Fatalf("data+snapshots (%d) != total (%d)", b.Stats.SizeData+b.Stats.SizeSnapshots, b.Stats.Size) + } +} diff --git a/hub/internal/monitor/offsite_box.go b/hub/internal/monitor/offsite_box.go new file mode 100644 index 0000000..e351958 --- /dev/null +++ b/hub/internal/monitor/offsite_box.go @@ -0,0 +1,256 @@ +package monitor + +import ( + "context" + "encoding/json" + "fmt" + "log" + "sync" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/hetznerapi" + "gitea.dooplex.hu/admin/felhom-hub/internal/offsite" + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// OffsiteBoxChecker (v0.64.0, R-5) watches the SHARED POOL BOX as a whole — the aggregate the +// per-customer OffsiteChecker cannot see. Two independent box-level signals, one checker, mirroring the +// OffsiteChecker/StorageFillChecker shape (escalation-only emit, recovery re-arms silently, born/persistent +// via an unseeded band). It is DELIBERATELY separate from OffsiteChecker — a different data source (the +// Hetzner unified API, not the controller report) and a customer-less scope. +// +// - FILL: box used vs box capacity (from the box TYPE's size, never stats.size) at warn 80% / crit 90%. +// - OVERSUBSCRIPTION: Σ(shared+enabled customers' soft quotas) vs capacity, at a warn ratio (2.0×). The +// number that says how far the sold promises exceed the disk. Independent of fill (both can fire). +// +// Fetch is throttled (boxFetchInterval); between refreshes Check() serves the cached snapshot. A failed +// fetch keeps the last snapshot (marked Degraded) and never drives a band transition — absence of data is +// NEVER treated as 0%. Events carry the customer-less scope "pool-box": the dispatcher uses it as a +// cooldown key + display string only, and processCustomer no-ops on it (no prefs → no customer email; +// verified against dispatcher.go), so a box event reaches ONLY the operator channel. NO SaveEvent — the +// scope has no customer row to key it to. +type OffsiteBoxChecker struct { + api hetznerapi.CloudAPI + boxID int64 + store *store.Store + fillWarn float64 + fillCrit float64 + oversubWarn float64 + onEvent EventNotifyFunc + logger *log.Logger + now func() time.Time // injectable clock (tests) + + mu sync.Mutex + snap BoxSnapshot + haveSnap bool + lastFetch time.Time + fillBand string // "" until first eval → born/persistent (a born-breach emits on the first sweep) + oversubBand string +} + +// offsiteBoxScope is the customer-less cooldown/display key for pool-box events. +const offsiteBoxScope = "pool-box" + +const ( + defaultOffsiteBoxFillWarnPercent = 80.0 + defaultOffsiteBoxFillCritPercent = 90.0 + defaultOffsiteOversubWarnRatio = 2.0 + // boxFetchInterval throttles the Hetzner read: Check() runs on the 60 s sweep but fetches at most + // once per this window (≈4 API reads/hour, not ≈60). A manual page load never forces a fetch. + boxFetchInterval = 15 * time.Minute +) + +// BoxSnapshot is the cached pool-box aggregate the web layer renders. All sizes are BYTES. FillBand / +// OversubBand carry the checker's own band verdict so the UI colors bars EXACTLY as the alerts fire (no +// threshold re-derivation in the web layer). +type BoxSnapshot struct { + CapacityBytes int64 + UsedBytes int64 + DataBytes int64 + SnapshotBytes int64 + FillPercent float64 + SumQuotaGB int64 + Ratio float64 // Σ(shared quota bytes) / capacity + FillBand string // "" | ok | warning | critical + OversubBand string // "" | ok | warning + FetchedAt time.Time + BoxType string + Degraded bool // last fetch failed OR capacity==0 → render stale; no alert transitions +} + +// NewOffsiteBoxChecker builds the checker. Invalid thresholds fall back to the documented defaults +// (warn 80% / crit 90% / oversub 2.0×). Nothing is seeded → the first Check that finds a breach emits +// (born/persistent; the dispatcher's 1 h cooldown dedups a hub restart). +func NewOffsiteBoxChecker(api hetznerapi.CloudAPI, boxID int64, s *store.Store, fillWarn, fillCrit, oversubWarn float64, onEvent EventNotifyFunc, logger *log.Logger) *OffsiteBoxChecker { + if fillWarn <= 0 || fillWarn >= 100 { + fillWarn = defaultOffsiteBoxFillWarnPercent + } + if fillCrit <= 0 || fillCrit > 100 || fillCrit <= fillWarn { + fillCrit = defaultOffsiteBoxFillCritPercent + } + if fillCrit <= fillWarn { + fillWarn, fillCrit = defaultOffsiteBoxFillWarnPercent, defaultOffsiteBoxFillCritPercent + } + if oversubWarn <= 0 { + oversubWarn = defaultOffsiteOversubWarnRatio + } + c := &OffsiteBoxChecker{ + api: api, boxID: boxID, store: s, + fillWarn: fillWarn, fillCrit: fillCrit, oversubWarn: oversubWarn, + onEvent: onEvent, logger: logger, now: time.Now, + } + logger.Printf("[INFO] Offsite pool-box checker initialized: box=%d fill warn=%.0f%% crit=%.0f%%, oversub warn=%.2fx, refresh %s", + boxID, fillWarn, fillCrit, oversubWarn, boxFetchInterval) + return c +} + +// Check runs on the 60 s sweep. It fetches (throttled), recomputes the snapshot, and emits on each +// band escalation. Recovery de-escalation re-arms silently. Degraded data drives no transition. +func (c *OffsiteBoxChecker) Check() { + c.mu.Lock() + defer c.mu.Unlock() + + // Fetch throttle: between refreshes serve the cache — bands only move on a fresh fetch. + if c.haveSnap && c.now().Sub(c.lastFetch) < boxFetchInterval { + return + } + c.lastFetch = c.now() // updated even on failure → a failing API is retried once per window, not per sweep + + box, err := c.api.GetStorageBox(context.Background(), c.boxID) + if err != nil { + c.logger.Printf("[WARN] Offsite pool-box: refresh failed (keeping last snapshot): %v", err) + if c.haveSnap { + c.snap.Degraded = true // serve the last-known, visibly stale; NO band transition (D) + } + return + } + + sumQuotaGB, qerr := c.sumSharedQuotaGB() + if qerr != nil { + c.logger.Printf("[WARN] Offsite pool-box: quota sum failed (ratio omitted this cycle): %v", qerr) + sumQuotaGB = 0 // a wrong sum is worse than a nominal 0.00× — never emit oversub off bad data + } + + capacity := box.StorageBoxType.Size + used := box.Stats.Size + snap := BoxSnapshot{ + CapacityBytes: capacity, UsedBytes: used, + DataBytes: box.Stats.SizeData, SnapshotBytes: box.Stats.SizeSnapshots, + SumQuotaGB: sumQuotaGB, FetchedAt: c.now(), BoxType: box.StorageBoxType.Name, + } + if capacity > 0 { + snap.FillPercent = float64(used) * 100 / float64(capacity) + snap.Ratio = float64(sumQuotaGB<<30) / float64(capacity) + snap.FillBand = bandForPercent(snap.FillPercent, c.fillWarn, c.fillCrit) + snap.OversubBand = bandOK + if snap.Ratio >= c.oversubWarn { + snap.OversubBand = bandWarning // ratio is single-threshold — no "critical" + } + } else { + snap.Degraded = true // zero capacity (initializing box / probe surprise) — guard the division + } + c.snap = snap + c.haveSnap = true + + if snap.Degraded { + return // no band transitions on degraded data + } + + // FILL band (escalation-only; recovery re-arms because bandOK has rank 0). + if bandRank(snap.FillBand) > bandRank(c.fillBand) { + c.emitFill(snap, snap.FillBand) + } + c.fillBand = snap.FillBand + + // OVERSUBSCRIPTION band — independent signal (both can fire in one sweep; neither masks the other). + if bandRank(snap.OversubBand) > bandRank(c.oversubBand) { + c.emitOversub(snap) + } + c.oversubBand = snap.OversubBand +} + +// Snapshot returns the current cached aggregate + whether one exists (mutex copy-out). The web layer's +// only source — it NEVER fetches from Hetzner in the request path. +func (c *OffsiteBoxChecker) Snapshot() (BoxSnapshot, bool) { + c.mu.Lock() + defer c.mu.Unlock() + return c.snap, c.haveSnap +} + +// FillState / OversubState expose the current bands (tests). +func (c *OffsiteBoxChecker) FillState() string { c.mu.Lock(); defer c.mu.Unlock(); return orUnknown(c.fillBand) } +func (c *OffsiteBoxChecker) OversubState() string { c.mu.Lock(); defer c.mu.Unlock(); return orUnknown(c.oversubBand) } + +func orUnknown(b string) string { + if b == "" { + return "unknown" + } + return b +} + +// sumSharedQuotaGB sums the soft quotas of every ENABLED, SHARED customer from the authoritative +// ConfigJSON descriptor (never the report echo). Dedicated + disabled customers are excluded. +func (c *OffsiteBoxChecker) sumSharedQuotaGB() (int64, error) { + cfgs, err := c.store.ListCustomerConfigs() + if err != nil { + return 0, err + } + var sum int64 + for _, cfg := range cfgs { + d, derr := offsite.ReadDescriptor(cfg.ConfigJSON) + if derr != nil || d == nil { + continue + } + if d.Enabled && d.Type == "shared" && d.QuotaGB > 0 { + sum += int64(d.QuotaGB) + } + } + return sum, nil +} + +func (c *OffsiteBoxChecker) emitFill(snap BoxSnapshot, band string) { + var severity, message string + switch band { + case bandCritical: + severity = "critical" + message = fmt.Sprintf("Offsite pool box %.0f%% full (%s of %s) — approaching capacity; add capacity or reduce retention before customer offsite runs start failing", + snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes)) + case bandWarning: + severity = "warning" + message = fmt.Sprintf("Offsite pool box %.0f%% full (%s of %s) — the shared pool is filling", + snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes)) + default: + return + } + details, _ := json.Marshal(map[string]any{ + "scope": offsiteBoxScope, "capacity_bytes": snap.CapacityBytes, "used_bytes": snap.UsedBytes, + "fill_percent": snap.FillPercent, "warn_percent": c.fillWarn, "crit_percent": c.fillCrit, + }) + c.logger.Printf("[INFO] Offsite pool-box fill: %.0f%% (%s)", snap.FillPercent, band) + // NO SaveEvent — the scope is customer-less; emit straight to the operator dispatcher. + if c.onEvent != nil { + c.onEvent(offsiteBoxScope, "offsite_box_fill", severity, message, string(details), "hub") + } +} + +func (c *OffsiteBoxChecker) emitOversub(snap BoxSnapshot) { + message := fmt.Sprintf("Offsite pool oversubscription %.2fx — Σ(shared soft quotas) %d GB against %s capacity exceeds the %.2fx threshold; a full-usage scenario would overrun the pool", + snap.Ratio, snap.SumQuotaGB, fmtSize(snap.CapacityBytes), c.oversubWarn) + details, _ := json.Marshal(map[string]any{ + "scope": offsiteBoxScope, "sum_quota_gb": snap.SumQuotaGB, "capacity_bytes": snap.CapacityBytes, + "ratio": snap.Ratio, "oversub_warn_ratio": c.oversubWarn, + }) + c.logger.Printf("[INFO] Offsite pool-box oversub: %.2fx (Σquota %d GB)", snap.Ratio, snap.SumQuotaGB) + if c.onEvent != nil { + c.onEvent(offsiteBoxScope, "offsite_box_oversub", "warning", message, string(details), "hub") + } +} + +// fmtSize renders bytes as GB (one decimal) below 1 TB, TB above — for the operator email/log only. +func fmtSize(b int64) string { + const gb = 1 << 30 + if b >= 1<<40 { + return fmt.Sprintf("%.2f TB", float64(b)/float64(int64(1)<<40)) + } + return fmt.Sprintf("%.1f GB", float64(b)/float64(gb)) +} diff --git a/hub/internal/monitor/offsite_box_test.go b/hub/internal/monitor/offsite_box_test.go new file mode 100644 index 0000000..8631075 --- /dev/null +++ b/hub/internal/monitor/offsite_box_test.go @@ -0,0 +1,228 @@ +package monitor + +import ( + "fmt" + "math" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/hetznerapi" + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +const gib = int64(1) << 30 + +// boxWith builds a fake StorageBox with the given capacity + used bytes (data-only, no snapshots). +func boxWith(capacityBytes, usedBytes int64) hetznerapi.StorageBox { + return hetznerapi.StorageBox{ + ID: 1, Status: "active", + StorageBoxType: hetznerapi.StorageBoxType{Name: "bx11", Size: capacityBytes}, + Stats: hetznerapi.StorageBoxStats{Size: usedBytes, SizeData: usedBytes}, + } +} + +// seedOffsiteCfg writes a customer config whose ConfigJSON carries an offsite descriptor. +func seedOffsiteCfg(t *testing.T, st *store.Store, id string, enabled bool, typ string, quotaGB int) { + t.Helper() + cfgJSON := fmt.Sprintf(`{"offsite":{"enabled":%v,"type":%q,"quota_gb":%d}}`, enabled, typ, quotaGB) + if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: id, APIKey: "k-" + id, RetrievalPassword: "p", ConfigJSON: cfgJSON}); err != nil { + t.Fatalf("seed config %s: %v", id, err) + } +} + +func noEvent(_, _, _, _, _, _ string) {} + +// Scenario A — fetch is throttled: ~4 API reads over an hour of 60 s sweeps, not ~60. +// Red-proof (i): drop the throttle guard in Check() → GetBoxCalls ≈ 60 → this fails. +func TestOffsiteBox_FetchThrottle(t *testing.T) { + st := newDiskStore(t) + fake := hetznerapi.NewFake() + fake.Boxes[1] = boxWith(1<<40, 300*gib) // 1 TiB, 300 GiB used + var cur time.Time + c := NewOffsiteBoxChecker(fake, 1, st, 80, 90, 2.0, noEvent, quietLog()) + c.now = func() time.Time { return cur } + + base := time.Now().UTC() + for i := 0; i < 60; i++ { // 60 sweeps at 60 s spacing = one simulated hour + cur = base.Add(time.Duration(i) * 60 * time.Second) + c.Check() + } + if fake.GetBoxCalls < 3 || fake.GetBoxCalls > 5 { + t.Fatalf("GetStorageBox called %d times over an hour of 60 s sweeps, want ≈4 (fetch-throttled to 15 min)", fake.GetBoxCalls) + } + snap, ok := c.Snapshot() + if !ok || snap.CapacityBytes != 1<<40 || snap.UsedBytes != 300*gib { + t.Fatalf("snapshot wrong: %+v ok=%v", snap, ok) + } +} + +// Scenario B — oversubscription math: Σ = shared+enabled only (A+B); dedicated (C) and disabled (D) +// excluded; ratio = Σquota / CAPACITY (not used). +// Red-proof (ii): include dedicated/disabled in the sum → this fails. +func TestOffsiteBox_OversubMath(t *testing.T) { + st := newDiskStore(t) + seedOffsiteCfg(t, st, "a", true, "shared", 500) + seedOffsiteCfg(t, st, "b", true, "shared", 700) + seedOffsiteCfg(t, st, "c", true, "dedicated", 9999) // excluded: dedicated + seedOffsiteCfg(t, st, "d", false, "shared", 300) // excluded: disabled + + fake := hetznerapi.NewFake() + fake.Boxes[1] = boxWith(1024*gib, 100*gib) // capacity 1024 GB + c := NewOffsiteBoxChecker(fake, 1, st, 80, 90, 2.0, noEvent, quietLog()) + c.Check() + + snap, _ := c.Snapshot() + if snap.SumQuotaGB != 1200 { + t.Fatalf("Σquota = %d GB, want 1200 (A 500 + B 700 only)", snap.SumQuotaGB) + } + want := 1200.0 / 1024.0 + if math.Abs(snap.Ratio-want) > 0.001 { + t.Fatalf("ratio = %.4f, want %.4f (Σquota/capacity)", snap.Ratio, want) + } +} + +// captureEvents records (eventType, severity, customerID) per emit. +type capturedBox struct{ typ, sev, cust []string } + +func (c *capturedBox) fn(cust, et, sev, _, _, _ string) { + c.cust = append(c.cust, cust) + c.typ = append(c.typ, et) + c.sev = append(c.sev, sev) +} + +// Scenario C (fill) — escalation-only, in-band no re-emit, recovery re-arms. Every emit carries the +// "pool-box" scope. Red-proof (iii): remove the escalation-only guard → C3 fails. +func TestOffsiteBox_FillBands(t *testing.T) { + st := newDiskStore(t) + fake := hetznerapi.NewFake() + cap := int64(1000) * gib + fake.Boxes[1] = boxWith(cap, 750*gib) // 75% + ev := &capturedBox{} + var cur time.Time + c := NewOffsiteBoxChecker(fake, 1, st, 80, 90, 2.0, ev.fn, quietLog()) + c.now = func() time.Time { return cur } + base := time.Now().UTC() + step := func(usedGiB int64) { + cur = cur.Add(16 * time.Minute) // bust the 15-min throttle each step + fake.Boxes[1] = boxWith(cap, usedGiB*gib) + c.Check() + } + cur = base + + c.Check() // C1: 75% → no emit + if len(ev.typ) != 0 { + t.Fatalf("C1 75%% must not emit, got %v", ev.typ) + } + step(820) // C2: 82% → one warning + if count(ev.typ, "offsite_box_fill") != 1 || ev.sev[0] != "warning" { + t.Fatalf("C2 82%% must warn once, got typ=%v sev=%v", ev.typ, ev.sev) + } + step(850) // C3: 85% same band → no second emit + if count(ev.typ, "offsite_box_fill") != 1 { + t.Fatalf("C3 same-band must NOT re-emit, got %d", count(ev.typ, "offsite_box_fill")) + } + step(920) // C4: 92% → escalate to critical + if count(ev.typ, "offsite_box_fill") != 2 || ev.sev[len(ev.sev)-1] != "critical" { + t.Fatalf("C4 92%% must escalate to critical, got typ=%v sev=%v", ev.typ, ev.sev) + } + step(700) // C5: 70% → recovery re-arm (no emit) + if c.FillState() != bandOK { + t.Fatalf("C5 recovery must re-arm to ok, got %s", c.FillState()) + } + if count(ev.typ, "offsite_box_fill") != 2 { + t.Fatalf("C5 recovery must be silent, got %d emits", count(ev.typ, "offsite_box_fill")) + } + step(920) // re-breach after recovery → emits again + if count(ev.typ, "offsite_box_fill") != 3 { + t.Fatalf("re-breach after recovery must emit again, got %d", count(ev.typ, "offsite_box_fill")) + } + for _, cust := range ev.cust { + if cust != "pool-box" { + t.Fatalf("every emit must carry the pool-box scope, got %q", cust) + } + } +} + +// Scenario C6 — oversubscription is an INDEPENDENT signal: it fires on ratio alone even when fill is +// nominal, with its own event type. +func TestOffsiteBox_OversubIndependent(t *testing.T) { + st := newDiskStore(t) + seedOffsiteCfg(t, st, "a", true, "shared", 2300) // Σ 2300 GB + fake := hetznerapi.NewFake() + fake.Boxes[1] = boxWith(1000*gib, 500*gib) // 50% fill (nominal), ratio 2.3× + ev := &capturedBox{} + c := NewOffsiteBoxChecker(fake, 1, st, 80, 90, 2.0, ev.fn, quietLog()) + c.Check() + if count(ev.typ, "offsite_box_oversub") != 1 { + t.Fatalf("2.3× oversub must warn once, got %v", ev.typ) + } + if count(ev.typ, "offsite_box_fill") != 0 { + t.Fatalf("fill is nominal (50%%) — no fill emit, got %v", ev.typ) + } + if ev.cust[0] != "pool-box" { + t.Fatalf("oversub emit scope = %q, want pool-box", ev.cust[0]) + } +} + +// Scenario D — API failure honesty: the last snapshot is KEPT (marked degraded), no band transition, +// and a failed fetch NEVER zeroes the snapshot (the recovery re-arm must not trigger off missing data). +// Red-proof (iv): let a failed fetch zero the snapshot → this fails. +func TestOffsiteBox_FailedFetchHonesty(t *testing.T) { + st := newDiskStore(t) + fake := hetznerapi.NewFake() + cap := int64(1000) * gib + fake.Boxes[1] = boxWith(cap, 920*gib) // 92% → critical + ev := &capturedBox{} + var cur time.Time + c := NewOffsiteBoxChecker(fake, 1, st, 80, 90, 2.0, ev.fn, quietLog()) + c.now = func() time.Time { return cur } + base := time.Now().UTC() + cur = base + + c.Check() // establishes a 92% critical snapshot + emit + if count(ev.typ, "offsite_box_fill") != 1 { + t.Fatalf("setup: want one critical emit, got %v", ev.typ) + } + before, _ := c.Snapshot() + + // Now the API fails. + fake.FailGetBox = fmt.Errorf("hetzner timeout") + cur = cur.Add(16 * time.Minute) + c.Check() + + after, ok := c.Snapshot() + if !ok { + t.Fatal("a failed fetch must KEEP the last snapshot, not drop it") + } + if after.UsedBytes != before.UsedBytes || after.CapacityBytes != before.CapacityBytes { + t.Fatalf("a failed fetch must not alter the last-known values (before %d/%d, after %d/%d)", + before.UsedBytes, before.CapacityBytes, after.UsedBytes, after.CapacityBytes) + } + if !after.Degraded { + t.Fatal("a failed fetch must mark the served snapshot Degraded (visible staleness)") + } + if c.FillState() != bandCritical { + t.Fatalf("a failed fetch must NOT transition the band (missing data ≠ 0%%), got %s", c.FillState()) + } + // No new emit on the failure. + if count(ev.typ, "offsite_box_fill") != 1 { + t.Fatalf("a failed fetch must not emit, got %d total", count(ev.typ, "offsite_box_fill")) + } +} + +// Not-configured / zero-capacity guard: a zero-capacity box marks degraded, never divides, never alerts. +func TestOffsiteBox_ZeroCapacityGuard(t *testing.T) { + st := newDiskStore(t) + fake := hetznerapi.NewFake() + fake.Boxes[1] = boxWith(0, 0) // initializing box — no capacity yet + ev := &capturedBox{} + c := NewOffsiteBoxChecker(fake, 1, st, 80, 90, 2.0, ev.fn, quietLog()) + c.Check() + snap, ok := c.Snapshot() + if !ok || !snap.Degraded { + t.Fatalf("zero capacity must yield a degraded snapshot, got %+v", snap) + } + if len(ev.typ) != 0 { + t.Fatalf("zero capacity must not alert, got %v", ev.typ) + } +} diff --git a/hub/internal/notify/dispatcher_test.go b/hub/internal/notify/dispatcher_test.go index cffb88d..1ba2b7c 100644 --- a/hub/internal/notify/dispatcher_test.go +++ b/hub/internal/notify/dispatcher_test.go @@ -48,6 +48,22 @@ func TestSeverityNotifies(t *testing.T) { } } +// TestProcessEvent_PoolBoxScopeOperatorOnly (v0.64.0, R-5, Scenario C7): a customer-less "pool-box" +// event reaches ONLY the operator channel — processCustomer finds no prefs for it and no-ops, so no +// customer email is ever attempted for the synthetic scope. +func TestProcessEvent_PoolBoxScopeOperatorOnly(t *testing.T) { + st := newDispStore(t) + d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + var mu sync.Mutex + var sent []string + d.sendEmailFn = func(to, _, _ string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil } + + d.ProcessEvent("pool-box", "offsite_box_fill", "warning", "Offsite pool box 82% full", `{"scope":"pool-box"}`, "hub") + if len(sent) != 1 || sent[0] != "op@felhom.eu" { + t.Fatalf("a pool-box event must reach the operator ONLY (no customer email), sent=%v", sent) + } +} + // TestProcessEvent_CriticalRoutes: a critical-severity event reaches the operator channel (Scenario E). func TestProcessEvent_CriticalRoutes(t *testing.T) { st := newDispStore(t) diff --git a/hub/internal/offsite/offsite.go b/hub/internal/offsite/offsite.go index a5cc542..30ef4d0 100644 --- a/hub/internal/offsite/offsite.go +++ b/hub/internal/offsite/offsite.go @@ -425,22 +425,37 @@ func (p *Provisioner) SetOffsiteFrozen(ctx context.Context, customerID string, f // resource) is cleared, so re-onboarding re-provisions fresh against the retained choice. No-op-safe: // an absent/empty offsite block returns the input unchanged. func ClearProvisionedDescriptor(configJSON string) (string, error) { + cur, err := ReadDescriptor(configJSON) + if err != nil { + return "", err + } + if cur == nil { + return configJSON, nil // no offsite block — nothing to clear + } + cleared := &Descriptor{Enabled: cur.Enabled, Type: cur.Type, QuotaGB: cur.QuotaGB, BoxType: cur.BoxType} + return MergeDescriptor(configJSON, cleared) +} + +// ReadDescriptor extracts the non-secret offsite Descriptor from a customer's ConfigJSON, or (nil, nil) +// when there is no offsite block (never provisioned). This is the AUTHORITATIVE tier/quota source — the +// pool-box Σ(quota) aggregate (v0.64.0, R-5) and the RESET clear both read through it, NEVER the report +// echo (a stale/absent report would undercount the sold promises; ConfigJSON is the operator's intent). +func ReadDescriptor(configJSON string) (*Descriptor, error) { obj := map[string]json.RawMessage{} if strings.TrimSpace(configJSON) != "" && configJSON != "{}" { if err := json.Unmarshal([]byte(configJSON), &obj); err != nil { - return "", fmt.Errorf("offsite: parse config_json: %w", err) + return nil, fmt.Errorf("offsite: parse config_json: %w", err) } } raw, ok := obj["offsite"] if !ok { - return configJSON, nil // no offsite block — nothing to clear + return nil, nil } - var cur Descriptor - if err := json.Unmarshal(raw, &cur); err != nil { - return "", fmt.Errorf("offsite: parse offsite descriptor: %w", err) + var d Descriptor + if err := json.Unmarshal(raw, &d); err != nil { + return nil, fmt.Errorf("offsite: parse offsite descriptor: %w", err) } - cleared := &Descriptor{Enabled: cur.Enabled, Type: cur.Type, QuotaGB: cur.QuotaGB, BoxType: cur.BoxType} - return MergeDescriptor(configJSON, cleared) + return &d, nil } func MergeDescriptor(configJSON string, d *Descriptor) (string, error) { diff --git a/hub/internal/web/offsite.go b/hub/internal/web/offsite.go index 1e380ab..4e1e498 100644 --- a/hub/internal/web/offsite.go +++ b/hub/internal/web/offsite.go @@ -139,12 +139,16 @@ func (s *Server) handleOffsite(w http.ResponseWriter, r *http.Request) { }) } + boxView, custRows := s.offsiteBoxData() // R-5 pool-box aggregate panel + data := map[string]interface{}{ - "Endpoints": cards, - "HasEndpoints": len(cards) > 0, - "Peers": rows, - "CSRFToken": s.getCSRFToken(r), - "Flash": r.URL.Query().Get("flash"), + "Endpoints": cards, + "HasEndpoints": len(cards) > 0, + "Peers": rows, + "OffsiteBox": boxView, + "OffsiteCusts": custRows, + "CSRFToken": s.getCSRFToken(r), + "Flash": r.URL.Query().Get("flash"), } if err := s.templates.ExecuteTemplate(w, "offsite.html", data); err != nil { s.logger.Printf("[ERROR] offsite.html template: %v", err) diff --git a/hub/internal/web/offsite_box.go b/hub/internal/web/offsite_box.go new file mode 100644 index 0000000..9c6d94a --- /dev/null +++ b/hub/internal/web/offsite_box.go @@ -0,0 +1,193 @@ +package web + +// Offsite pool-box aggregate surfaces (v0.64.0, R-5): the Offsite-tab panel + the Dashboard tile. +// The web layer NEVER fetches from Hetzner — it reads the checker's cached snapshot (s.offsiteBox) and +// composes per-customer rows from state the hub already holds (quota from the ConfigJSON Descriptor, +// usage from the controller report echo). + +import ( + "encoding/json" + "fmt" + "sort" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/offsite" +) + +// offsiteBoxView is the pool-box panel model. Every display value is composed handler-side; band +// strings ("ok"/"warning"/"critical") drive the bar color via templates. +type offsiteBoxView struct { + Configured bool // s.offsiteBox wired (HETZNER_TOKEN + box id present) + Pending bool // configured but the first refresh hasn't landed yet + Degraded bool // last fetch failed / zero capacity — values shown stale + CapacityStr string + UsedStr string + DataStr string + SnapshotStr string + FillPercent float64 + FillBand string + SumQuotaStr string + RatioStr string // "1.17×" + OversubBand string + BoxType string + FetchedAt time.Time +} + +// offsiteCustomerRow is one per-customer usage/quota row under the panel. +type offsiteCustomerRow struct { + CustomerName string + Dedicated bool + QuotaStr string // "" for dedicated + UsageStr string // "" when NoUsage + UsageBytes int64 // sort key + UsagePercent float64 // usage/quota (shared only) + UsageBand string // ok|warning|critical for the bar + HasBar bool // shared + quota>0 + has usage → render a bar + NoUsage bool // no offsite object in the report yet +} + +// fmtBytesGB renders bytes as GB (one decimal) below 1 TB, TB (two decimals) above. +func fmtBytesGB(b int64) string { + if b >= 1<<40 { + return fmt.Sprintf("%.2f TB", float64(b)/float64(int64(1)<<40)) + } + return fmt.Sprintf("%.1f GB", float64(b)/float64(int64(1)<<30)) +} + +// pctBand maps a percent to a band with the given warn/crit (web-side twin of monitor.bandForPercent — +// unexported there; a 3-line copy keeps web from depending on monitor internals). +func pctBand(pct, warn, crit float64) string { + switch { + case pct >= crit: + return "critical" + case pct >= warn: + return "warning" + default: + return "ok" + } +} + +// offsiteUsageBytes reads repo_size_bytes from a customer's report `offsite` object — the ONLY place the +// live repo size exists. ok=false when the report carries no offsite object (never reported). Quota is +// NEVER taken from here (the report echo would undercount promises) — only usage. Twin of monitor's +// parseOffsite, scoped to the one field the rows need. +func offsiteUsageBytes(reportJSON string) (int64, bool) { + var r struct { + Offsite *struct { + RepoSizeBytes int64 `json:"repo_size_bytes"` + } `json:"offsite"` + } + if json.Unmarshal([]byte(reportJSON), &r) != nil || r.Offsite == nil { + return 0, false + } + return r.Offsite.RepoSizeBytes, true +} + +// offsiteTile is the compact Dashboard tile: fill% + ratio, band-colored, linking to /offsite. +type offsiteTile struct { + FillPercent float64 + RatioStr string + Band string // worst of fill/oversub → tile color + Degraded bool +} + +// offsiteBoxTile builds the Dashboard tile, or nil when there is nothing to show (no provider, or the +// first refresh hasn't landed) — the caller then renders NO tile (never a zeroed one). +func (s *Server) offsiteBoxTile() *offsiteTile { + if s.offsiteBox == nil { + return nil + } + snap, ok := s.offsiteBox() + if !ok { + return nil // configured but no snapshot yet — absent, not a zeroed tile + } + band := "ok" + switch { + case snap.FillBand == "critical": + band = "critical" + case snap.FillBand == "warning" || snap.OversubBand == "warning": + band = "warning" + } + return &offsiteTile{ + FillPercent: snap.FillPercent, + RatioStr: fmt.Sprintf("%.2f×", snap.Ratio), + Band: band, + Degraded: snap.Degraded, + } +} + +// offsiteBoxData builds the panel view + per-customer rows. Nil provider → Configured:false (the panel +// renders the honest "not configured"). Never fetches. +func (s *Server) offsiteBoxData() (offsiteBoxView, []offsiteCustomerRow) { + if s.offsiteBox == nil { + return offsiteBoxView{Configured: false}, nil + } + view := offsiteBoxView{Configured: true} + if snap, ok := s.offsiteBox(); ok { + view.Pending = false + view.Degraded = snap.Degraded + view.CapacityStr = fmtBytesGB(snap.CapacityBytes) + view.UsedStr = fmtBytesGB(snap.UsedBytes) + view.DataStr = fmtBytesGB(snap.DataBytes) + view.SnapshotStr = fmtBytesGB(snap.SnapshotBytes) + view.FillPercent = snap.FillPercent + view.FillBand = snap.FillBand + view.SumQuotaStr = fmt.Sprintf("%d GB", snap.SumQuotaGB) + view.RatioStr = fmt.Sprintf("%.2f×", snap.Ratio) + view.OversubBand = snap.OversubBand + view.BoxType = snap.BoxType + view.FetchedAt = snap.FetchedAt + } else { + view.Pending = true // configured, first fetch pending + } + return view, s.offsiteCustomerRows() +} + +// offsiteCustomerRows composes the per-customer rows for every ENABLED-offsite customer: quota from the +// authoritative ConfigJSON Descriptor, usage from the report echo. Shared customers get a usage/quota +// bar; dedicated customers are listed without one. Sorted by usage descending. +func (s *Server) offsiteCustomerRows() []offsiteCustomerRow { + cfgs, err := s.store.ListCustomerConfigs() + if err != nil { + s.logger.Printf("[WARN] offsite box: list configs: %v", err) + return nil + } + usage := map[string]int64{} + hasReport := map[string]bool{} + if custs, cerr := s.store.GetCustomers(); cerr == nil { + for _, c := range custs { + if b, ok := offsiteUsageBytes(c.ReportJSON); ok { + usage[c.CustomerID], hasReport[c.CustomerID] = b, true + } + } + } + var rows []offsiteCustomerRow + for _, cfg := range cfgs { + d, derr := offsite.ReadDescriptor(cfg.ConfigJSON) + if derr != nil || d == nil || !d.Enabled { + continue // only customers actually using an offsite tier appear + } + name := cfg.CustomerName + if name == "" { + name = cfg.CustomerID + } + row := offsiteCustomerRow{CustomerName: name, Dedicated: d.Type == "dedicated"} + if d.Type == "shared" && d.QuotaGB > 0 { + row.QuotaStr = fmt.Sprintf("%d GB", d.QuotaGB) + } + if hasReport[cfg.CustomerID] { + row.UsageBytes = usage[cfg.CustomerID] + row.UsageStr = fmtBytesGB(row.UsageBytes) + if d.Type == "shared" && d.QuotaGB > 0 { + row.UsagePercent = float64(row.UsageBytes) * 100 / float64(int64(d.QuotaGB)<<30) + row.UsageBand = pctBand(row.UsagePercent, 90, 95) // matches the per-customer OffsiteChecker bands + row.HasBar = true + } + } else { + row.NoUsage = true + } + rows = append(rows, row) + } + sort.SliceStable(rows, func(i, j int) bool { return rows[i].UsageBytes > rows[j].UsageBytes }) + return rows +} diff --git a/hub/internal/web/offsite_box_render_test.go b/hub/internal/web/offsite_box_render_test.go new file mode 100644 index 0000000..f5a791a --- /dev/null +++ b/hub/internal/web/offsite_box_render_test.go @@ -0,0 +1,85 @@ +package web + +import ( + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/monitor" + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +const gib = int64(1) << 30 + +// Not configured (no HETZNER_TOKEN) → honest "not configured", never an error or fake zeros. +// (Uses renderOffsite from offsite_test.go — the real handler via httptest.) +func TestOffsiteBoxPanel_NotConfigured(t *testing.T) { + s, _ := newRenderServer(t) + // s.offsiteBox left nil + body := renderOffsite(t, s) + if !strings.Contains(body, "Offsite pool metrics not configured") { + t.Fatalf("unconfigured panel must say 'not configured':\n%s", body) + } + if strings.Contains(body, "0.0 GB") { + t.Fatal("unconfigured panel must not render fake zeros") + } +} + +// With data + per-customer rows: the panel renders capacity/fill/ratio; a shared customer with a report +// shows usage, a customer with NO report shows 'no usage reported yet' (never an asserted 0 GB). +func TestOffsiteBoxPanel_WithData(t *testing.T) { + s, st := newRenderServer(t) + s.SetOffsiteBox(func() (monitor.BoxSnapshot, bool) { + return monitor.BoxSnapshot{ + CapacityBytes: 1 << 40, UsedBytes: 300 * gib, DataBytes: 300 * gib, + FillPercent: 27, FillBand: "ok", SumQuotaGB: 1200, Ratio: 1.17, OversubBand: "ok", + BoxType: "bx11", FetchedAt: time.Now().UTC(), + }, true + }) + // "acme": shared, enabled, 500 GB, WITH a report (200 GiB used). "newbie": shared, enabled, no report. + saveCfg(t, st, "acme", `{"offsite":{"enabled":true,"type":"shared","quota_gb":500}}`) + saveCfg(t, st, "newbie", `{"offsite":{"enabled":true,"type":"shared","quota_gb":300}}`) + if err := st.SaveReport("acme", []byte(`{"customer_id":"acme","offsite":{"enabled":true,"repo_size_bytes":`+itoa(200*gib)+`}}`)); err != nil { + t.Fatal(err) + } + + body := renderOffsite(t, s) + for _, want := range []string{"Offsite pool box", "1.00 TB", "1.17×", "acme", "newbie", "no usage reported yet"} { + if !strings.Contains(body, want) { + t.Fatalf("panel missing %q:\n%s", want, body) + } + } + if strings.Contains(body, "not configured") { + t.Fatal("a configured panel must not show the not-configured message") + } +} + +func saveCfg(t *testing.T, st *store.Store, id, cfgJSON string) { + t.Helper() + if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: id, CustomerName: id, APIKey: "k-" + id, RetrievalPassword: "p", ConfigJSON: cfgJSON}); err != nil { + t.Fatalf("save config %s: %v", id, err) + } +} + +// itoa avoids importing strconv just for the report body. +func itoa(v int64) string { + if v == 0 { + return "0" + } + neg := v < 0 + if neg { + v = -v + } + var b [20]byte + i := len(b) + for v > 0 { + i-- + b[i] = byte('0' + v%10) + v /= 10 + } + if neg { + i-- + b[i] = '-' + } + return string(b[i:]) +} diff --git a/hub/internal/web/render_test.go b/hub/internal/web/render_test.go index d5ba7ed..e80304d 100644 --- a/hub/internal/web/render_test.go +++ b/hub/internal/web/render_test.go @@ -92,11 +92,15 @@ func TestTemplates_DashboardCriticalBadge(t *testing.T) { EventErrors int EventWarnings int } - data := []dashboardCustomer{{ + // v0.64.0: dashboard.html now takes {Customers, OffsiteTile}; OffsiteTile nil → no tile rendered. + data := struct { + Customers []dashboardCustomer + OffsiteTile any + }{Customers: []dashboardCustomer{{ CustomerSummary: store.CustomerSummary{CustomerID: "c1", CustomerName: "Acme", ReceivedAt: time.Now()}, OverallStatus: "ok", BackupAge: "–", EventCriticals: 2, EventErrors: 1, EventWarnings: 0, - }} + }}} var buf bytes.Buffer if err := s.templates.ExecuteTemplate(&buf, "dashboard.html", data); err != nil { t.Fatalf("render dashboard.html: %v", err) diff --git a/hub/internal/web/server.go b/hub/internal/web/server.go index 079424e..2739884 100644 --- a/hub/internal/web/server.go +++ b/hub/internal/web/server.go @@ -19,6 +19,7 @@ import ( "gitea.dooplex.hu/admin/felhom-hub/internal/claim" "gitea.dooplex.hu/admin/felhom-hub/internal/gitea" "gitea.dooplex.hu/admin/felhom-hub/internal/intent" + "gitea.dooplex.hu/admin/felhom-hub/internal/monitor" "gitea.dooplex.hu/admin/felhom-hub/internal/poke" "gitea.dooplex.hu/admin/felhom-hub/internal/offsite" "gitea.dooplex.hu/admin/felhom-hub/internal/semver" @@ -66,6 +67,7 @@ type Server struct { assetsMgr *assets.Manager gitea *gitea.Client // optional; enables the Day-0 artifact version dropdowns offsite *offsite.Provisioner // optional; enables Hetzner offsite provisioning (SLICE 1) + offsiteBox func() (monitor.BoxSnapshot, bool) // optional (v0.64.0, R-5); the pool-box aggregate snapshot accessor tenantsync tenancyProvisioner // optional; enables PBS DR tier provisioning (web/pbsdr.go) claimEngine *claim.Engine // optional; enables the customer-claim resend button (v0.50.0) // intentHub (v0.58.0, Direction-2 immediate-sync) is Bumped by every operator-intent handler @@ -181,6 +183,11 @@ func (s *Server) SetAssetManager(am *assets.Manager) { // offsite enabled returns an error (offsite not configured on this hub). func (s *Server) SetOffsiteProvisioner(p *offsite.Provisioner) { s.offsite = p } +// SetOffsiteBox wires the pool-box aggregate snapshot accessor (v0.64.0, R-5): the checker's cached +// snapshot, read on the Offsite tab + the Dashboard tile. nil (no HETZNER_TOKEN/box id) → both render an +// honest "not configured". The web layer NEVER fetches from Hetzner — it only reads this cache. +func (s *Server) SetOffsiteBox(fn func() (monitor.BoxSnapshot, bool)) { s.offsiteBox = fn } + // SetClaimEngine wires the customer-claim code engine for the Setup-tab resend button (v0.50.0). func (s *Server) SetClaimEngine(e *claim.Engine) { s.claimEngine = e } @@ -742,8 +749,13 @@ func (s *Server) handleDashboard(w http.ResponseWriter, r *http.Request) { data = append(data, dc) } + payload := struct { + Customers []dashboardCustomer + OffsiteTile *offsiteTile // R-5: nil → no tile rendered + }{Customers: data, OffsiteTile: s.offsiteBoxTile()} + w.Header().Set("Content-Type", "text/html; charset=utf-8") - if err := s.templates.ExecuteTemplate(w, "dashboard.html", data); err != nil { + if err := s.templates.ExecuteTemplate(w, "dashboard.html", payload); err != nil { s.logger.Printf("[ERROR] Template render: %v", err) } } diff --git a/hub/internal/web/templates/dashboard.html b/hub/internal/web/templates/dashboard.html index a0f7dde..9c3d75c 100644 --- a/hub/internal/web/templates/dashboard.html +++ b/hub/internal/web/templates/dashboard.html @@ -22,7 +22,15 @@ - {{if not .}} + {{with .OffsiteTile}} + + Offsite pool + {{formatFloat .FillPercent}}% · {{.RatioStr}} + {{if .Degraded}}stale{{end}} + + {{end}} + + {{if not .Customers}}

No customer reports received yet.

Configure hub.enabled: true in customer controller.yaml to start receiving reports.

@@ -44,7 +52,7 @@ - {{range .}} + {{range .Customers}} diff --git a/hub/internal/web/templates/offsite.html b/hub/internal/web/templates/offsite.html index 0839f1c..c6eb01a 100644 --- a/hub/internal/web/templates/offsite.html +++ b/hub/internal/web/templates/offsite.html @@ -34,6 +34,45 @@
Endpoint deleted.
{{end}} + +
+

Offsite pool box

+ {{if not .OffsiteBox.Configured}} +

Offsite pool metrics not configured (no Hetzner token / pool box id on this hub).

+ {{else if .OffsiteBox.Pending}} +

Pool metrics loading — the first refresh has not landed yet.

+ {{else}} + + + + +
Box{{.OffsiteBox.BoxType}}
Capacity{{.OffsiteBox.CapacityStr}}
Used{{.OffsiteBox.UsedStr}} · {{formatFloat .OffsiteBox.FillPercent}}% full (data {{.OffsiteBox.DataStr}} + snapshots {{.OffsiteBox.SnapshotStr}})
+
+ + + + +
Σ shared soft quotas{{.OffsiteBox.SumQuotaStr}}
Oversubscription{{.OffsiteBox.RatioStr}} Σ(shared quotas) ÷ capacity
Fetched{{timeAgo .OffsiteBox.FetchedAt}}{{if .OffsiteBox.Degraded}} STALE — last refresh failed{{end}}
+ + {{if .OffsiteCusts}} +

Per-customer usage

+ + + + {{range .OffsiteCusts}} + + + + + + + {{end}} + +
CustomerUsageQuota
{{.CustomerName}}{{if .Dedicated}} (dedicated){{end}}{{if .NoUsage}}no usage reported yet{{else}}{{.UsageStr}}{{end}}{{if .Dedicated}}dedicated{{else if .QuotaStr}}{{.QuotaStr}}{{else}}—{{end}}{{if .HasBar}}
{{end}}
+ {{end}} + {{end}} +
+ {{if .HasEndpoints}} {{range .Endpoints}}