hub v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5)
The operator sees the shared pool box's real state on the hub: total box fill vs
capacity, Σ(shared soft quotas) vs capacity (the oversubscription ratio), per-customer
usage/quota bars, and a box-level operator alert (fill % + oversub ratio) on the existing
dispatcher's operator channel. Per-customer fill alerts already existed; the box-level
aggregate was the gap. READ-ONLY against Hetzner (GET only).
Phase-0 probe (gate PASSED): the live pool box 611714 returns capacity via
storage_box_type.size (1 TiB / bx11) and usage via a stats object (size/size_data/
size_snapshots), all bytes; our token reads it (200).
- hetznerapi: additive StorageBoxType + StorageBoxStats on StorageBox (no existing field/
method changed); fake carries them + a GetBoxCalls counter; golden decode test.
- monitor.OffsiteBoxChecker: OffsiteChecker-sibling for the box; fetch-throttled (1 GET/
15min), cached BoxSnapshot, escalation-only + recovery re-arm. FILL (used/capacity 80/90)
+ OVERSUB (Σ shared+enabled quotas / capacity, 2.0x) — independent. Σ from the ConfigJSON
Descriptor (offsite.ReadDescriptor, new), never the report echo; dedicated+disabled
excluded. Scope "pool-box" -> operator channel only, no SaveEvent. Failed fetch keeps the
last snapshot degraded; missing data never becomes 0% and never transitions a band.
- config: Alerting.OffsiteBoxFill{Warn,Crit}Percent + OffsiteOversubWarnRatio (80/90/2.0
defaults; thresholds pending Viktor's ruling). Constructed in the HETZNER_TOKEN branch,
60s sweep, snapshot handed to the web server.
- web: Offsite-tab panel (fill bar, Σ+ratio, per-customer usage/quota rows) + a compact
dashboard tile; reads the cached snapshot only, never fetches; nil -> "not configured".
Tests: 10 new + 4 red-proofs (throttle, Σ filter, escalation-only, failed-fetch honesty),
all confirmed red then restored. go build/vet/test all pass; hub confirm gate OK.
This commit is contained in:
@@ -1,5 +1,48 @@
|
||||
# Felhom Hub — Changelog
|
||||
|
||||
## v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5) (2026-07-17)
|
||||
|
||||
Ships **R-5**: the operator sees the shared pool box's real state on the hub — **total box fill vs
|
||||
capacity**, **Σ(shared soft quotas) vs capacity** (the oversubscription ratio), **per-customer usage/quota
|
||||
bars**, and a **box-level operator alert** (fill % + oversubscription ratio) riding the existing
|
||||
dispatcher's operator channel. Per-customer fill alerts already existed (OffsiteChecker, 90/95% of each
|
||||
quota); the box-level aggregate was the gap — the operator's early warning that the *pool itself* is
|
||||
filling, before any single customer breaches. All READ-ONLY against Hetzner (GET only). Green:
|
||||
`go build/vet/test` all pass; hub confirm gate OK.
|
||||
|
||||
- **Phase-0 probe (gate, PASSED):** one authenticated GET of the live pool box (611714) pinned the API
|
||||
shape — capacity is `storage_box_type.size` (1 TiB for bx11), usage is a `stats` object
|
||||
(`size`/`size_data`/`size_snapshots`), all bytes; our token reads it (200). The type extension mirrors it.
|
||||
- **`hetznerapi` (`hetznerapi.go`, `fake.go`):** additive `StorageBoxType` + `StorageBoxStats` sub-structs
|
||||
on `StorageBox` (no existing field/method changed); fake carries them + a `GetBoxCalls` counter. Golden
|
||||
decode test against the (redacted) probe capture.
|
||||
- **`monitor.OffsiteBoxChecker` (`offsite_box.go`, new):** the OffsiteChecker-sibling for the box as a
|
||||
whole — **fetch-throttled** (one Hetzner GET per 15 min; ≈4/hour, never per-sweep or per-page-load),
|
||||
cached `BoxSnapshot`, escalation-only emits with silent recovery re-arm. Two independent signals: FILL
|
||||
(used/capacity, warn 80% / crit 90%) and OVERSUBSCRIPTION (Σ shared+enabled quotas / capacity, warn
|
||||
2.0×) — both can fire, neither masks the other. Σ(quota) is read from the authoritative ConfigJSON
|
||||
`Descriptor` (`offsite.ReadDescriptor`, new), NEVER the report echo; dedicated + disabled customers
|
||||
excluded. Events carry the customer-less scope `"pool-box"` → operator channel ONLY (`processCustomer`
|
||||
no-ops on it) and are NOT SaveEvent'd (no customer row to key them). A failed fetch keeps the last
|
||||
snapshot marked degraded — missing data never becomes 0% and never drives a band transition.
|
||||
- **Config (`cmd/hub/main.go`):** `Alerting.OffsiteBoxFillWarnPercent` (80) / `OffsiteBoxFillCritPercent`
|
||||
(90) / `OffsiteOversubWarnRatio` (2.0), plumbed like `StorageFill*`. The checker is constructed inside
|
||||
the existing `HETZNER_TOKEN` branch (shares the client), registered in the 60 s sweep, and its snapshot
|
||||
handed to the web server. **Thresholds are Claude's encoding of the starter suggestion — Viktor's ruling
|
||||
pending; the named keys are the one-line flip.**
|
||||
- **Web surfaces (`web/offsite_box.go` new, `templates/offsite.html`, `dashboard.html`, `style.css`):** an
|
||||
Offsite-tab panel (capacity, used with data/snapshot split, fill bar, Σ quotas + ratio, fetched-at, +
|
||||
per-customer rows sorted by usage — shared with a usage/quota bar, dedicated listed without one, no-report
|
||||
customers show "no usage reported yet") and a compact Dashboard tile (`fill% · ratio`, band-colored,
|
||||
linking to /offsite). The web layer reads only the cached snapshot — it NEVER fetches. Nil provider → both
|
||||
render an honest "not configured". Exception color (neutral/amber/red, no green).
|
||||
- **Tests + red-proofs (run-fail-restore):** 10 new tests (throttle ≈4-not-≈60, Σ/ratio truth table,
|
||||
band transitions incl. in-band no-re-emit + recovery re-arm + oversub independence + pool-box operator-only
|
||||
scope, failed-fetch honesty, zero-capacity guard, golden decode, panel render × "not configured"/"with
|
||||
data"/"no usage reported yet"). Red-proofs: (i) drop the throttle → ≈60 calls; (ii) sum dedicated/disabled
|
||||
→ wrong Σ; (iii) drop the escalation-only guard → in-band re-emit; (iv) zero the snapshot on a failed fetch
|
||||
→ lost last-known. All confirmed red, then restored.
|
||||
|
||||
## v0.63.0 — system-initiated immediacy: wire the proven poke/bump notifiers into every mutation site that lacked one (2026-07-17)
|
||||
|
||||
The immediate-sync arc (Dir-1 trigger, Dir-2b wait channel, Dir-2a agent poke) covered only
|
||||
|
||||
@@ -65,6 +65,13 @@ type Config struct {
|
||||
// the host-root thresholds above so a single storage's policy can be tuned separately.
|
||||
StorageFillWarnPercent float64 `yaml:"storage_fill_warn_percent"`
|
||||
StorageFillCritPercent float64 `yaml:"storage_fill_crit_percent"`
|
||||
// Offsite POOL-BOX aggregate thresholds (v0.64.0, R-5). Fill = box used vs capacity (percent);
|
||||
// oversub = Σ(shared soft quotas)/capacity (ratio). Empty/0/invalid → defaults 80/90/2.0 (the
|
||||
// checker normalizes). Threshold VALUES are Claude's encoding of the starter suggestion — Viktor's
|
||||
// ruling pending; these keys are the one-line flip when ruled.
|
||||
OffsiteBoxFillWarnPercent float64 `yaml:"offsite_box_fill_warn_percent"`
|
||||
OffsiteBoxFillCritPercent float64 `yaml:"offsite_box_fill_crit_percent"`
|
||||
OffsiteOversubWarnRatio float64 `yaml:"offsite_oversub_warn_ratio"`
|
||||
} `yaml:"alerting"`
|
||||
Registry struct {
|
||||
Image string `yaml:"image"`
|
||||
@@ -291,6 +298,7 @@ func main() {
|
||||
// PREREQUISITE: this MUST be a token scoped to a DEDICATED Hetzner project (the shared project token can
|
||||
// delete ep0 — SPIKE §6). Base is api.hetzner.com (NOT api.hetzner.cloud). Absent token → offsite UI
|
||||
// still renders, but saving with offsite enabled returns "not configured".
|
||||
var offsiteBoxChecker *monitor.OffsiteBoxChecker
|
||||
if tok := os.Getenv("HETZNER_TOKEN"); tok != "" {
|
||||
poolBoxID, _ := strconv.ParseInt(os.Getenv("HETZNER_POOL_BOX_ID"), 10, 64)
|
||||
location := os.Getenv("HETZNER_LOCATION")
|
||||
@@ -302,6 +310,16 @@ func main() {
|
||||
API: client, Store: dataStore, Scanner: offsite.SSHHostKeyScanner{}, PoolBoxID: poolBoxID, Location: location, Logger: logger,
|
||||
})
|
||||
logger.Printf("[INFO] Offsite provisioning enabled (pool_box=%d, location=%s)", poolBoxID, location)
|
||||
// R-5 (v0.64.0): the pool-box aggregate checker shares the SAME client + pool box id (GET-only).
|
||||
// It needs a valid box id to poll; without one, the aggregate stays unconfigured.
|
||||
if poolBoxID != 0 {
|
||||
offsiteBoxChecker = monitor.NewOffsiteBoxChecker(client, poolBoxID, dataStore,
|
||||
cfg.Alerting.OffsiteBoxFillWarnPercent, cfg.Alerting.OffsiteBoxFillCritPercent, cfg.Alerting.OffsiteOversubWarnRatio,
|
||||
dispatcher.ProcessEvent, logger)
|
||||
webServer.SetOffsiteBox(offsiteBoxChecker.Snapshot)
|
||||
} else {
|
||||
logger.Printf("[INFO] Offsite pool-box aggregate disabled (HETZNER_POOL_BOX_ID unset)")
|
||||
}
|
||||
}
|
||||
|
||||
// v0.57.0 (F3) — clean-slate re-enrollment auto re-issues offsite credentials to the fresh box.
|
||||
@@ -530,6 +548,9 @@ func main() {
|
||||
hostMgmtPlaneChecker.Check()
|
||||
hostOOBChecker.Check()
|
||||
offsiteChecker.Check()
|
||||
if offsiteBoxChecker != nil {
|
||||
offsiteBoxChecker.Check() // R-5: pool-box aggregate (fetch-throttled internally)
|
||||
}
|
||||
// v0.46.0: pulled log bundles are transient diagnostics — 72 h TTL.
|
||||
if n, perr := dataStore.PurgeExpiredLogBundles(time.Now()); perr != nil {
|
||||
logger.Printf("[WARN] log-bundle TTL purge failed: %v", perr)
|
||||
|
||||
@@ -23,10 +23,12 @@ type Fake struct {
|
||||
BoxResetCalls int
|
||||
DeletedSubaccounts int
|
||||
DeletedBoxes int
|
||||
GetBoxCalls int // v0.64.0: the fetch-throttle assertion (Scenario A) counts these
|
||||
|
||||
// Failure injection.
|
||||
FailCreate error // if set, CreateSubaccount/CreateStorageBox return this error
|
||||
FailAction bool // if set, created actions come back status "error" (WaitAction fails)
|
||||
FailGetBox error // v0.64.0: if set, GetStorageBox returns this (the box-poll failure path, Scenario D)
|
||||
}
|
||||
|
||||
// NewFake returns an empty Fake.
|
||||
@@ -145,6 +147,10 @@ func (f *Fake) ListStorageBoxes(_ context.Context, sel string) ([]StorageBox, er
|
||||
func (f *Fake) GetStorageBox(_ context.Context, boxID int64) (StorageBox, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.GetBoxCalls++
|
||||
if f.FailGetBox != nil {
|
||||
return StorageBox{}, f.FailGetBox
|
||||
}
|
||||
b, ok := f.Boxes[boxID]
|
||||
if !ok {
|
||||
return StorageBox{}, fmt.Errorf("hetznerapi(fake): box %d not found", boxID)
|
||||
|
||||
@@ -62,6 +62,29 @@ type StorageBox struct {
|
||||
Status string `json:"status"` // initializing | active | …
|
||||
Server string `json:"server"` // uXXXXXX.your-storagebox.de
|
||||
Labels map[string]string `json:"labels"`
|
||||
|
||||
// StorageBoxType + Stats added v0.64.0 (R-5 pool-box aggregate). Shape pinned by the Phase-0 probe
|
||||
// of the live pool box 611714 (2026-07-17): GET /storage_boxes/{id} returns a `storage_box_type`
|
||||
// object whose `size` is the CAPACITY in bytes (1099511627776 = 1 TiB for bx11), and a `stats` object
|
||||
// carrying total/data/snapshot usage in bytes. Additive — no existing field or method changed.
|
||||
StorageBoxType StorageBoxType `json:"storage_box_type"`
|
||||
Stats StorageBoxStats `json:"stats"`
|
||||
}
|
||||
|
||||
// StorageBoxType is the box's plan; its Size is the CAPACITY in bytes (never use stats.size for capacity).
|
||||
type StorageBoxType struct {
|
||||
ID int64 `json:"id"`
|
||||
Name string `json:"name"` // e.g. "bx11"
|
||||
Description string `json:"description"` // e.g. "BX11"
|
||||
Size int64 `json:"size"` // capacity in BYTES
|
||||
}
|
||||
|
||||
// StorageBoxStats is the box's live usage, all in BYTES (probe-confirmed): total, the data portion, and
|
||||
// the snapshot portion (Size == SizeData + SizeSnapshots on the live box).
|
||||
type StorageBoxStats struct {
|
||||
Size int64 `json:"size"` // total used bytes
|
||||
SizeData int64 `json:"size_data"` // data bytes
|
||||
SizeSnapshots int64 `json:"size_snapshots"` // snapshot bytes
|
||||
}
|
||||
|
||||
// CreateSubaccountRequest — POST /storage_boxes/{box}/subaccounts. Password satisfies the 4-class policy.
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
package hetznerapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// goldenPoolBox is the (redacted, non-secret) response captured from the Phase-0 probe of the live pool
|
||||
// box 611714 (2026-07-17): GET /storage_boxes/{id}. The stats/capacity extension must decode it EXACTLY.
|
||||
const goldenPoolBox = `{
|
||||
"storage_box": {
|
||||
"id": 611714,
|
||||
"username": "u629488",
|
||||
"name": "storage-box-pool-1",
|
||||
"status": "active",
|
||||
"server": "u629488.your-storagebox.de",
|
||||
"storage_box_type": { "id": 1333, "name": "bx11", "description": "BX11", "size": 1099511627776 },
|
||||
"stats": { "size": 2746220544, "size_data": 2746220544, "size_snapshots": 0 }
|
||||
}
|
||||
}`
|
||||
|
||||
// TestStorageBox_DecodesProbeShape pins the type extension to the live API shape: capacity is the box
|
||||
// TYPE's size (1 TiB), and stats carries the total/data/snapshot split, all in bytes.
|
||||
func TestStorageBox_DecodesProbeShape(t *testing.T) {
|
||||
var out struct {
|
||||
StorageBox StorageBox `json:"storage_box"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(goldenPoolBox), &out); err != nil {
|
||||
t.Fatalf("decode golden pool-box: %v", err)
|
||||
}
|
||||
b := out.StorageBox
|
||||
if b.ID != 611714 || b.Name != "storage-box-pool-1" || b.Status != "active" {
|
||||
t.Fatalf("base fields wrong: %+v", b)
|
||||
}
|
||||
if b.StorageBoxType.Name != "bx11" || b.StorageBoxType.Size != 1099511627776 {
|
||||
t.Fatalf("capacity (storage_box_type.size) wrong: %+v", b.StorageBoxType)
|
||||
}
|
||||
if b.Stats.Size != 2746220544 || b.Stats.SizeData != 2746220544 || b.Stats.SizeSnapshots != 0 {
|
||||
t.Fatalf("stats wrong: %+v", b.Stats)
|
||||
}
|
||||
// Sanity: the split sums to the total (the invariant the live box holds).
|
||||
if b.Stats.SizeData+b.Stats.SizeSnapshots != b.Stats.Size {
|
||||
t.Fatalf("data+snapshots (%d) != total (%d)", b.Stats.SizeData+b.Stats.SizeSnapshots, b.Stats.Size)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,256 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/hetznerapi"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// OffsiteBoxChecker (v0.64.0, R-5) watches the SHARED POOL BOX as a whole — the aggregate the
|
||||
// per-customer OffsiteChecker cannot see. Two independent box-level signals, one checker, mirroring the
|
||||
// OffsiteChecker/StorageFillChecker shape (escalation-only emit, recovery re-arms silently, born/persistent
|
||||
// via an unseeded band). It is DELIBERATELY separate from OffsiteChecker — a different data source (the
|
||||
// Hetzner unified API, not the controller report) and a customer-less scope.
|
||||
//
|
||||
// - FILL: box used vs box capacity (from the box TYPE's size, never stats.size) at warn 80% / crit 90%.
|
||||
// - OVERSUBSCRIPTION: Σ(shared+enabled customers' soft quotas) vs capacity, at a warn ratio (2.0×). The
|
||||
// number that says how far the sold promises exceed the disk. Independent of fill (both can fire).
|
||||
//
|
||||
// Fetch is throttled (boxFetchInterval); between refreshes Check() serves the cached snapshot. A failed
|
||||
// fetch keeps the last snapshot (marked Degraded) and never drives a band transition — absence of data is
|
||||
// NEVER treated as 0%. Events carry the customer-less scope "pool-box": the dispatcher uses it as a
|
||||
// cooldown key + display string only, and processCustomer no-ops on it (no prefs → no customer email;
|
||||
// verified against dispatcher.go), so a box event reaches ONLY the operator channel. NO SaveEvent — the
|
||||
// scope has no customer row to key it to.
|
||||
type OffsiteBoxChecker struct {
|
||||
api hetznerapi.CloudAPI
|
||||
boxID int64
|
||||
store *store.Store
|
||||
fillWarn float64
|
||||
fillCrit float64
|
||||
oversubWarn float64
|
||||
onEvent EventNotifyFunc
|
||||
logger *log.Logger
|
||||
now func() time.Time // injectable clock (tests)
|
||||
|
||||
mu sync.Mutex
|
||||
snap BoxSnapshot
|
||||
haveSnap bool
|
||||
lastFetch time.Time
|
||||
fillBand string // "" until first eval → born/persistent (a born-breach emits on the first sweep)
|
||||
oversubBand string
|
||||
}
|
||||
|
||||
// offsiteBoxScope is the customer-less cooldown/display key for pool-box events.
|
||||
const offsiteBoxScope = "pool-box"
|
||||
|
||||
const (
|
||||
defaultOffsiteBoxFillWarnPercent = 80.0
|
||||
defaultOffsiteBoxFillCritPercent = 90.0
|
||||
defaultOffsiteOversubWarnRatio = 2.0
|
||||
// boxFetchInterval throttles the Hetzner read: Check() runs on the 60 s sweep but fetches at most
|
||||
// once per this window (≈4 API reads/hour, not ≈60). A manual page load never forces a fetch.
|
||||
boxFetchInterval = 15 * time.Minute
|
||||
)
|
||||
|
||||
// BoxSnapshot is the cached pool-box aggregate the web layer renders. All sizes are BYTES. FillBand /
|
||||
// OversubBand carry the checker's own band verdict so the UI colors bars EXACTLY as the alerts fire (no
|
||||
// threshold re-derivation in the web layer).
|
||||
type BoxSnapshot struct {
|
||||
CapacityBytes int64
|
||||
UsedBytes int64
|
||||
DataBytes int64
|
||||
SnapshotBytes int64
|
||||
FillPercent float64
|
||||
SumQuotaGB int64
|
||||
Ratio float64 // Σ(shared quota bytes) / capacity
|
||||
FillBand string // "" | ok | warning | critical
|
||||
OversubBand string // "" | ok | warning
|
||||
FetchedAt time.Time
|
||||
BoxType string
|
||||
Degraded bool // last fetch failed OR capacity==0 → render stale; no alert transitions
|
||||
}
|
||||
|
||||
// NewOffsiteBoxChecker builds the checker. Invalid thresholds fall back to the documented defaults
|
||||
// (warn 80% / crit 90% / oversub 2.0×). Nothing is seeded → the first Check that finds a breach emits
|
||||
// (born/persistent; the dispatcher's 1 h cooldown dedups a hub restart).
|
||||
func NewOffsiteBoxChecker(api hetznerapi.CloudAPI, boxID int64, s *store.Store, fillWarn, fillCrit, oversubWarn float64, onEvent EventNotifyFunc, logger *log.Logger) *OffsiteBoxChecker {
|
||||
if fillWarn <= 0 || fillWarn >= 100 {
|
||||
fillWarn = defaultOffsiteBoxFillWarnPercent
|
||||
}
|
||||
if fillCrit <= 0 || fillCrit > 100 || fillCrit <= fillWarn {
|
||||
fillCrit = defaultOffsiteBoxFillCritPercent
|
||||
}
|
||||
if fillCrit <= fillWarn {
|
||||
fillWarn, fillCrit = defaultOffsiteBoxFillWarnPercent, defaultOffsiteBoxFillCritPercent
|
||||
}
|
||||
if oversubWarn <= 0 {
|
||||
oversubWarn = defaultOffsiteOversubWarnRatio
|
||||
}
|
||||
c := &OffsiteBoxChecker{
|
||||
api: api, boxID: boxID, store: s,
|
||||
fillWarn: fillWarn, fillCrit: fillCrit, oversubWarn: oversubWarn,
|
||||
onEvent: onEvent, logger: logger, now: time.Now,
|
||||
}
|
||||
logger.Printf("[INFO] Offsite pool-box checker initialized: box=%d fill warn=%.0f%% crit=%.0f%%, oversub warn=%.2fx, refresh %s",
|
||||
boxID, fillWarn, fillCrit, oversubWarn, boxFetchInterval)
|
||||
return c
|
||||
}
|
||||
|
||||
// Check runs on the 60 s sweep. It fetches (throttled), recomputes the snapshot, and emits on each
|
||||
// band escalation. Recovery de-escalation re-arms silently. Degraded data drives no transition.
|
||||
func (c *OffsiteBoxChecker) Check() {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
// Fetch throttle: between refreshes serve the cache — bands only move on a fresh fetch.
|
||||
if c.haveSnap && c.now().Sub(c.lastFetch) < boxFetchInterval {
|
||||
return
|
||||
}
|
||||
c.lastFetch = c.now() // updated even on failure → a failing API is retried once per window, not per sweep
|
||||
|
||||
box, err := c.api.GetStorageBox(context.Background(), c.boxID)
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] Offsite pool-box: refresh failed (keeping last snapshot): %v", err)
|
||||
if c.haveSnap {
|
||||
c.snap.Degraded = true // serve the last-known, visibly stale; NO band transition (D)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
sumQuotaGB, qerr := c.sumSharedQuotaGB()
|
||||
if qerr != nil {
|
||||
c.logger.Printf("[WARN] Offsite pool-box: quota sum failed (ratio omitted this cycle): %v", qerr)
|
||||
sumQuotaGB = 0 // a wrong sum is worse than a nominal 0.00× — never emit oversub off bad data
|
||||
}
|
||||
|
||||
capacity := box.StorageBoxType.Size
|
||||
used := box.Stats.Size
|
||||
snap := BoxSnapshot{
|
||||
CapacityBytes: capacity, UsedBytes: used,
|
||||
DataBytes: box.Stats.SizeData, SnapshotBytes: box.Stats.SizeSnapshots,
|
||||
SumQuotaGB: sumQuotaGB, FetchedAt: c.now(), BoxType: box.StorageBoxType.Name,
|
||||
}
|
||||
if capacity > 0 {
|
||||
snap.FillPercent = float64(used) * 100 / float64(capacity)
|
||||
snap.Ratio = float64(sumQuotaGB<<30) / float64(capacity)
|
||||
snap.FillBand = bandForPercent(snap.FillPercent, c.fillWarn, c.fillCrit)
|
||||
snap.OversubBand = bandOK
|
||||
if snap.Ratio >= c.oversubWarn {
|
||||
snap.OversubBand = bandWarning // ratio is single-threshold — no "critical"
|
||||
}
|
||||
} else {
|
||||
snap.Degraded = true // zero capacity (initializing box / probe surprise) — guard the division
|
||||
}
|
||||
c.snap = snap
|
||||
c.haveSnap = true
|
||||
|
||||
if snap.Degraded {
|
||||
return // no band transitions on degraded data
|
||||
}
|
||||
|
||||
// FILL band (escalation-only; recovery re-arms because bandOK has rank 0).
|
||||
if bandRank(snap.FillBand) > bandRank(c.fillBand) {
|
||||
c.emitFill(snap, snap.FillBand)
|
||||
}
|
||||
c.fillBand = snap.FillBand
|
||||
|
||||
// OVERSUBSCRIPTION band — independent signal (both can fire in one sweep; neither masks the other).
|
||||
if bandRank(snap.OversubBand) > bandRank(c.oversubBand) {
|
||||
c.emitOversub(snap)
|
||||
}
|
||||
c.oversubBand = snap.OversubBand
|
||||
}
|
||||
|
||||
// Snapshot returns the current cached aggregate + whether one exists (mutex copy-out). The web layer's
|
||||
// only source — it NEVER fetches from Hetzner in the request path.
|
||||
func (c *OffsiteBoxChecker) Snapshot() (BoxSnapshot, bool) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.snap, c.haveSnap
|
||||
}
|
||||
|
||||
// FillState / OversubState expose the current bands (tests).
|
||||
func (c *OffsiteBoxChecker) FillState() string { c.mu.Lock(); defer c.mu.Unlock(); return orUnknown(c.fillBand) }
|
||||
func (c *OffsiteBoxChecker) OversubState() string { c.mu.Lock(); defer c.mu.Unlock(); return orUnknown(c.oversubBand) }
|
||||
|
||||
func orUnknown(b string) string {
|
||||
if b == "" {
|
||||
return "unknown"
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// sumSharedQuotaGB sums the soft quotas of every ENABLED, SHARED customer from the authoritative
|
||||
// ConfigJSON descriptor (never the report echo). Dedicated + disabled customers are excluded.
|
||||
func (c *OffsiteBoxChecker) sumSharedQuotaGB() (int64, error) {
|
||||
cfgs, err := c.store.ListCustomerConfigs()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var sum int64
|
||||
for _, cfg := range cfgs {
|
||||
d, derr := offsite.ReadDescriptor(cfg.ConfigJSON)
|
||||
if derr != nil || d == nil {
|
||||
continue
|
||||
}
|
||||
if d.Enabled && d.Type == "shared" && d.QuotaGB > 0 {
|
||||
sum += int64(d.QuotaGB)
|
||||
}
|
||||
}
|
||||
return sum, nil
|
||||
}
|
||||
|
||||
func (c *OffsiteBoxChecker) emitFill(snap BoxSnapshot, band string) {
|
||||
var severity, message string
|
||||
switch band {
|
||||
case bandCritical:
|
||||
severity = "critical"
|
||||
message = fmt.Sprintf("Offsite pool box %.0f%% full (%s of %s) — approaching capacity; add capacity or reduce retention before customer offsite runs start failing",
|
||||
snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes))
|
||||
case bandWarning:
|
||||
severity = "warning"
|
||||
message = fmt.Sprintf("Offsite pool box %.0f%% full (%s of %s) — the shared pool is filling",
|
||||
snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes))
|
||||
default:
|
||||
return
|
||||
}
|
||||
details, _ := json.Marshal(map[string]any{
|
||||
"scope": offsiteBoxScope, "capacity_bytes": snap.CapacityBytes, "used_bytes": snap.UsedBytes,
|
||||
"fill_percent": snap.FillPercent, "warn_percent": c.fillWarn, "crit_percent": c.fillCrit,
|
||||
})
|
||||
c.logger.Printf("[INFO] Offsite pool-box fill: %.0f%% (%s)", snap.FillPercent, band)
|
||||
// NO SaveEvent — the scope is customer-less; emit straight to the operator dispatcher.
|
||||
if c.onEvent != nil {
|
||||
c.onEvent(offsiteBoxScope, "offsite_box_fill", severity, message, string(details), "hub")
|
||||
}
|
||||
}
|
||||
|
||||
func (c *OffsiteBoxChecker) emitOversub(snap BoxSnapshot) {
|
||||
message := fmt.Sprintf("Offsite pool oversubscription %.2fx — Σ(shared soft quotas) %d GB against %s capacity exceeds the %.2fx threshold; a full-usage scenario would overrun the pool",
|
||||
snap.Ratio, snap.SumQuotaGB, fmtSize(snap.CapacityBytes), c.oversubWarn)
|
||||
details, _ := json.Marshal(map[string]any{
|
||||
"scope": offsiteBoxScope, "sum_quota_gb": snap.SumQuotaGB, "capacity_bytes": snap.CapacityBytes,
|
||||
"ratio": snap.Ratio, "oversub_warn_ratio": c.oversubWarn,
|
||||
})
|
||||
c.logger.Printf("[INFO] Offsite pool-box oversub: %.2fx (Σquota %d GB)", snap.Ratio, snap.SumQuotaGB)
|
||||
if c.onEvent != nil {
|
||||
c.onEvent(offsiteBoxScope, "offsite_box_oversub", "warning", message, string(details), "hub")
|
||||
}
|
||||
}
|
||||
|
||||
// fmtSize renders bytes as GB (one decimal) below 1 TB, TB above — for the operator email/log only.
|
||||
func fmtSize(b int64) string {
|
||||
const gb = 1 << 30
|
||||
if b >= 1<<40 {
|
||||
return fmt.Sprintf("%.2f TB", float64(b)/float64(int64(1)<<40))
|
||||
}
|
||||
return fmt.Sprintf("%.1f GB", float64(b)/float64(gb))
|
||||
}
|
||||
@@ -0,0 +1,228 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/hetznerapi"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
const gib = int64(1) << 30
|
||||
|
||||
// boxWith builds a fake StorageBox with the given capacity + used bytes (data-only, no snapshots).
|
||||
func boxWith(capacityBytes, usedBytes int64) hetznerapi.StorageBox {
|
||||
return hetznerapi.StorageBox{
|
||||
ID: 1, Status: "active",
|
||||
StorageBoxType: hetznerapi.StorageBoxType{Name: "bx11", Size: capacityBytes},
|
||||
Stats: hetznerapi.StorageBoxStats{Size: usedBytes, SizeData: usedBytes},
|
||||
}
|
||||
}
|
||||
|
||||
// seedOffsiteCfg writes a customer config whose ConfigJSON carries an offsite descriptor.
|
||||
func seedOffsiteCfg(t *testing.T, st *store.Store, id string, enabled bool, typ string, quotaGB int) {
|
||||
t.Helper()
|
||||
cfgJSON := fmt.Sprintf(`{"offsite":{"enabled":%v,"type":%q,"quota_gb":%d}}`, enabled, typ, quotaGB)
|
||||
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: id, APIKey: "k-" + id, RetrievalPassword: "p", ConfigJSON: cfgJSON}); err != nil {
|
||||
t.Fatalf("seed config %s: %v", id, err)
|
||||
}
|
||||
}
|
||||
|
||||
func noEvent(_, _, _, _, _, _ string) {}
|
||||
|
||||
// Scenario A — fetch is throttled: ~4 API reads over an hour of 60 s sweeps, not ~60.
|
||||
// Red-proof (i): drop the throttle guard in Check() → GetBoxCalls ≈ 60 → this fails.
|
||||
func TestOffsiteBox_FetchThrottle(t *testing.T) {
|
||||
st := newDiskStore(t)
|
||||
fake := hetznerapi.NewFake()
|
||||
fake.Boxes[1] = boxWith(1<<40, 300*gib) // 1 TiB, 300 GiB used
|
||||
var cur time.Time
|
||||
c := NewOffsiteBoxChecker(fake, 1, st, 80, 90, 2.0, noEvent, quietLog())
|
||||
c.now = func() time.Time { return cur }
|
||||
|
||||
base := time.Now().UTC()
|
||||
for i := 0; i < 60; i++ { // 60 sweeps at 60 s spacing = one simulated hour
|
||||
cur = base.Add(time.Duration(i) * 60 * time.Second)
|
||||
c.Check()
|
||||
}
|
||||
if fake.GetBoxCalls < 3 || fake.GetBoxCalls > 5 {
|
||||
t.Fatalf("GetStorageBox called %d times over an hour of 60 s sweeps, want ≈4 (fetch-throttled to 15 min)", fake.GetBoxCalls)
|
||||
}
|
||||
snap, ok := c.Snapshot()
|
||||
if !ok || snap.CapacityBytes != 1<<40 || snap.UsedBytes != 300*gib {
|
||||
t.Fatalf("snapshot wrong: %+v ok=%v", snap, ok)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario B — oversubscription math: Σ = shared+enabled only (A+B); dedicated (C) and disabled (D)
|
||||
// excluded; ratio = Σquota / CAPACITY (not used).
|
||||
// Red-proof (ii): include dedicated/disabled in the sum → this fails.
|
||||
func TestOffsiteBox_OversubMath(t *testing.T) {
|
||||
st := newDiskStore(t)
|
||||
seedOffsiteCfg(t, st, "a", true, "shared", 500)
|
||||
seedOffsiteCfg(t, st, "b", true, "shared", 700)
|
||||
seedOffsiteCfg(t, st, "c", true, "dedicated", 9999) // excluded: dedicated
|
||||
seedOffsiteCfg(t, st, "d", false, "shared", 300) // excluded: disabled
|
||||
|
||||
fake := hetznerapi.NewFake()
|
||||
fake.Boxes[1] = boxWith(1024*gib, 100*gib) // capacity 1024 GB
|
||||
c := NewOffsiteBoxChecker(fake, 1, st, 80, 90, 2.0, noEvent, quietLog())
|
||||
c.Check()
|
||||
|
||||
snap, _ := c.Snapshot()
|
||||
if snap.SumQuotaGB != 1200 {
|
||||
t.Fatalf("Σquota = %d GB, want 1200 (A 500 + B 700 only)", snap.SumQuotaGB)
|
||||
}
|
||||
want := 1200.0 / 1024.0
|
||||
if math.Abs(snap.Ratio-want) > 0.001 {
|
||||
t.Fatalf("ratio = %.4f, want %.4f (Σquota/capacity)", snap.Ratio, want)
|
||||
}
|
||||
}
|
||||
|
||||
// captureEvents records (eventType, severity, customerID) per emit.
|
||||
type capturedBox struct{ typ, sev, cust []string }
|
||||
|
||||
func (c *capturedBox) fn(cust, et, sev, _, _, _ string) {
|
||||
c.cust = append(c.cust, cust)
|
||||
c.typ = append(c.typ, et)
|
||||
c.sev = append(c.sev, sev)
|
||||
}
|
||||
|
||||
// Scenario C (fill) — escalation-only, in-band no re-emit, recovery re-arms. Every emit carries the
|
||||
// "pool-box" scope. Red-proof (iii): remove the escalation-only guard → C3 fails.
|
||||
func TestOffsiteBox_FillBands(t *testing.T) {
|
||||
st := newDiskStore(t)
|
||||
fake := hetznerapi.NewFake()
|
||||
cap := int64(1000) * gib
|
||||
fake.Boxes[1] = boxWith(cap, 750*gib) // 75%
|
||||
ev := &capturedBox{}
|
||||
var cur time.Time
|
||||
c := NewOffsiteBoxChecker(fake, 1, st, 80, 90, 2.0, ev.fn, quietLog())
|
||||
c.now = func() time.Time { return cur }
|
||||
base := time.Now().UTC()
|
||||
step := func(usedGiB int64) {
|
||||
cur = cur.Add(16 * time.Minute) // bust the 15-min throttle each step
|
||||
fake.Boxes[1] = boxWith(cap, usedGiB*gib)
|
||||
c.Check()
|
||||
}
|
||||
cur = base
|
||||
|
||||
c.Check() // C1: 75% → no emit
|
||||
if len(ev.typ) != 0 {
|
||||
t.Fatalf("C1 75%% must not emit, got %v", ev.typ)
|
||||
}
|
||||
step(820) // C2: 82% → one warning
|
||||
if count(ev.typ, "offsite_box_fill") != 1 || ev.sev[0] != "warning" {
|
||||
t.Fatalf("C2 82%% must warn once, got typ=%v sev=%v", ev.typ, ev.sev)
|
||||
}
|
||||
step(850) // C3: 85% same band → no second emit
|
||||
if count(ev.typ, "offsite_box_fill") != 1 {
|
||||
t.Fatalf("C3 same-band must NOT re-emit, got %d", count(ev.typ, "offsite_box_fill"))
|
||||
}
|
||||
step(920) // C4: 92% → escalate to critical
|
||||
if count(ev.typ, "offsite_box_fill") != 2 || ev.sev[len(ev.sev)-1] != "critical" {
|
||||
t.Fatalf("C4 92%% must escalate to critical, got typ=%v sev=%v", ev.typ, ev.sev)
|
||||
}
|
||||
step(700) // C5: 70% → recovery re-arm (no emit)
|
||||
if c.FillState() != bandOK {
|
||||
t.Fatalf("C5 recovery must re-arm to ok, got %s", c.FillState())
|
||||
}
|
||||
if count(ev.typ, "offsite_box_fill") != 2 {
|
||||
t.Fatalf("C5 recovery must be silent, got %d emits", count(ev.typ, "offsite_box_fill"))
|
||||
}
|
||||
step(920) // re-breach after recovery → emits again
|
||||
if count(ev.typ, "offsite_box_fill") != 3 {
|
||||
t.Fatalf("re-breach after recovery must emit again, got %d", count(ev.typ, "offsite_box_fill"))
|
||||
}
|
||||
for _, cust := range ev.cust {
|
||||
if cust != "pool-box" {
|
||||
t.Fatalf("every emit must carry the pool-box scope, got %q", cust)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario C6 — oversubscription is an INDEPENDENT signal: it fires on ratio alone even when fill is
|
||||
// nominal, with its own event type.
|
||||
func TestOffsiteBox_OversubIndependent(t *testing.T) {
|
||||
st := newDiskStore(t)
|
||||
seedOffsiteCfg(t, st, "a", true, "shared", 2300) // Σ 2300 GB
|
||||
fake := hetznerapi.NewFake()
|
||||
fake.Boxes[1] = boxWith(1000*gib, 500*gib) // 50% fill (nominal), ratio 2.3×
|
||||
ev := &capturedBox{}
|
||||
c := NewOffsiteBoxChecker(fake, 1, st, 80, 90, 2.0, ev.fn, quietLog())
|
||||
c.Check()
|
||||
if count(ev.typ, "offsite_box_oversub") != 1 {
|
||||
t.Fatalf("2.3× oversub must warn once, got %v", ev.typ)
|
||||
}
|
||||
if count(ev.typ, "offsite_box_fill") != 0 {
|
||||
t.Fatalf("fill is nominal (50%%) — no fill emit, got %v", ev.typ)
|
||||
}
|
||||
if ev.cust[0] != "pool-box" {
|
||||
t.Fatalf("oversub emit scope = %q, want pool-box", ev.cust[0])
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario D — API failure honesty: the last snapshot is KEPT (marked degraded), no band transition,
|
||||
// and a failed fetch NEVER zeroes the snapshot (the recovery re-arm must not trigger off missing data).
|
||||
// Red-proof (iv): let a failed fetch zero the snapshot → this fails.
|
||||
func TestOffsiteBox_FailedFetchHonesty(t *testing.T) {
|
||||
st := newDiskStore(t)
|
||||
fake := hetznerapi.NewFake()
|
||||
cap := int64(1000) * gib
|
||||
fake.Boxes[1] = boxWith(cap, 920*gib) // 92% → critical
|
||||
ev := &capturedBox{}
|
||||
var cur time.Time
|
||||
c := NewOffsiteBoxChecker(fake, 1, st, 80, 90, 2.0, ev.fn, quietLog())
|
||||
c.now = func() time.Time { return cur }
|
||||
base := time.Now().UTC()
|
||||
cur = base
|
||||
|
||||
c.Check() // establishes a 92% critical snapshot + emit
|
||||
if count(ev.typ, "offsite_box_fill") != 1 {
|
||||
t.Fatalf("setup: want one critical emit, got %v", ev.typ)
|
||||
}
|
||||
before, _ := c.Snapshot()
|
||||
|
||||
// Now the API fails.
|
||||
fake.FailGetBox = fmt.Errorf("hetzner timeout")
|
||||
cur = cur.Add(16 * time.Minute)
|
||||
c.Check()
|
||||
|
||||
after, ok := c.Snapshot()
|
||||
if !ok {
|
||||
t.Fatal("a failed fetch must KEEP the last snapshot, not drop it")
|
||||
}
|
||||
if after.UsedBytes != before.UsedBytes || after.CapacityBytes != before.CapacityBytes {
|
||||
t.Fatalf("a failed fetch must not alter the last-known values (before %d/%d, after %d/%d)",
|
||||
before.UsedBytes, before.CapacityBytes, after.UsedBytes, after.CapacityBytes)
|
||||
}
|
||||
if !after.Degraded {
|
||||
t.Fatal("a failed fetch must mark the served snapshot Degraded (visible staleness)")
|
||||
}
|
||||
if c.FillState() != bandCritical {
|
||||
t.Fatalf("a failed fetch must NOT transition the band (missing data ≠ 0%%), got %s", c.FillState())
|
||||
}
|
||||
// No new emit on the failure.
|
||||
if count(ev.typ, "offsite_box_fill") != 1 {
|
||||
t.Fatalf("a failed fetch must not emit, got %d total", count(ev.typ, "offsite_box_fill"))
|
||||
}
|
||||
}
|
||||
|
||||
// Not-configured / zero-capacity guard: a zero-capacity box marks degraded, never divides, never alerts.
|
||||
func TestOffsiteBox_ZeroCapacityGuard(t *testing.T) {
|
||||
st := newDiskStore(t)
|
||||
fake := hetznerapi.NewFake()
|
||||
fake.Boxes[1] = boxWith(0, 0) // initializing box — no capacity yet
|
||||
ev := &capturedBox{}
|
||||
c := NewOffsiteBoxChecker(fake, 1, st, 80, 90, 2.0, ev.fn, quietLog())
|
||||
c.Check()
|
||||
snap, ok := c.Snapshot()
|
||||
if !ok || !snap.Degraded {
|
||||
t.Fatalf("zero capacity must yield a degraded snapshot, got %+v", snap)
|
||||
}
|
||||
if len(ev.typ) != 0 {
|
||||
t.Fatalf("zero capacity must not alert, got %v", ev.typ)
|
||||
}
|
||||
}
|
||||
@@ -48,6 +48,22 @@ func TestSeverityNotifies(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestProcessEvent_PoolBoxScopeOperatorOnly (v0.64.0, R-5, Scenario C7): a customer-less "pool-box"
|
||||
// event reaches ONLY the operator channel — processCustomer finds no prefs for it and no-ops, so no
|
||||
// customer email is ever attempted for the synthetic scope.
|
||||
func TestProcessEvent_PoolBoxScopeOperatorOnly(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
var mu sync.Mutex
|
||||
var sent []string
|
||||
d.sendEmailFn = func(to, _, _ string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil }
|
||||
|
||||
d.ProcessEvent("pool-box", "offsite_box_fill", "warning", "Offsite pool box 82% full", `{"scope":"pool-box"}`, "hub")
|
||||
if len(sent) != 1 || sent[0] != "op@felhom.eu" {
|
||||
t.Fatalf("a pool-box event must reach the operator ONLY (no customer email), sent=%v", sent)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProcessEvent_CriticalRoutes: a critical-severity event reaches the operator channel (Scenario E).
|
||||
func TestProcessEvent_CriticalRoutes(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
|
||||
@@ -425,22 +425,37 @@ func (p *Provisioner) SetOffsiteFrozen(ctx context.Context, customerID string, f
|
||||
// resource) is cleared, so re-onboarding re-provisions fresh against the retained choice. No-op-safe:
|
||||
// an absent/empty offsite block returns the input unchanged.
|
||||
func ClearProvisionedDescriptor(configJSON string) (string, error) {
|
||||
cur, err := ReadDescriptor(configJSON)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if cur == nil {
|
||||
return configJSON, nil // no offsite block — nothing to clear
|
||||
}
|
||||
cleared := &Descriptor{Enabled: cur.Enabled, Type: cur.Type, QuotaGB: cur.QuotaGB, BoxType: cur.BoxType}
|
||||
return MergeDescriptor(configJSON, cleared)
|
||||
}
|
||||
|
||||
// ReadDescriptor extracts the non-secret offsite Descriptor from a customer's ConfigJSON, or (nil, nil)
|
||||
// when there is no offsite block (never provisioned). This is the AUTHORITATIVE tier/quota source — the
|
||||
// pool-box Σ(quota) aggregate (v0.64.0, R-5) and the RESET clear both read through it, NEVER the report
|
||||
// echo (a stale/absent report would undercount the sold promises; ConfigJSON is the operator's intent).
|
||||
func ReadDescriptor(configJSON string) (*Descriptor, error) {
|
||||
obj := map[string]json.RawMessage{}
|
||||
if strings.TrimSpace(configJSON) != "" && configJSON != "{}" {
|
||||
if err := json.Unmarshal([]byte(configJSON), &obj); err != nil {
|
||||
return "", fmt.Errorf("offsite: parse config_json: %w", err)
|
||||
return nil, fmt.Errorf("offsite: parse config_json: %w", err)
|
||||
}
|
||||
}
|
||||
raw, ok := obj["offsite"]
|
||||
if !ok {
|
||||
return configJSON, nil // no offsite block — nothing to clear
|
||||
return nil, nil
|
||||
}
|
||||
var cur Descriptor
|
||||
if err := json.Unmarshal(raw, &cur); err != nil {
|
||||
return "", fmt.Errorf("offsite: parse offsite descriptor: %w", err)
|
||||
var d Descriptor
|
||||
if err := json.Unmarshal(raw, &d); err != nil {
|
||||
return nil, fmt.Errorf("offsite: parse offsite descriptor: %w", err)
|
||||
}
|
||||
cleared := &Descriptor{Enabled: cur.Enabled, Type: cur.Type, QuotaGB: cur.QuotaGB, BoxType: cur.BoxType}
|
||||
return MergeDescriptor(configJSON, cleared)
|
||||
return &d, nil
|
||||
}
|
||||
|
||||
func MergeDescriptor(configJSON string, d *Descriptor) (string, error) {
|
||||
|
||||
@@ -139,12 +139,16 @@ func (s *Server) handleOffsite(w http.ResponseWriter, r *http.Request) {
|
||||
})
|
||||
}
|
||||
|
||||
boxView, custRows := s.offsiteBoxData() // R-5 pool-box aggregate panel
|
||||
|
||||
data := map[string]interface{}{
|
||||
"Endpoints": cards,
|
||||
"HasEndpoints": len(cards) > 0,
|
||||
"Peers": rows,
|
||||
"CSRFToken": s.getCSRFToken(r),
|
||||
"Flash": r.URL.Query().Get("flash"),
|
||||
"Endpoints": cards,
|
||||
"HasEndpoints": len(cards) > 0,
|
||||
"Peers": rows,
|
||||
"OffsiteBox": boxView,
|
||||
"OffsiteCusts": custRows,
|
||||
"CSRFToken": s.getCSRFToken(r),
|
||||
"Flash": r.URL.Query().Get("flash"),
|
||||
}
|
||||
if err := s.templates.ExecuteTemplate(w, "offsite.html", data); err != nil {
|
||||
s.logger.Printf("[ERROR] offsite.html template: %v", err)
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
package web
|
||||
|
||||
// Offsite pool-box aggregate surfaces (v0.64.0, R-5): the Offsite-tab panel + the Dashboard tile.
|
||||
// The web layer NEVER fetches from Hetzner — it reads the checker's cached snapshot (s.offsiteBox) and
|
||||
// composes per-customer rows from state the hub already holds (quota from the ConfigJSON Descriptor,
|
||||
// usage from the controller report echo).
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
|
||||
)
|
||||
|
||||
// offsiteBoxView is the pool-box panel model. Every display value is composed handler-side; band
|
||||
// strings ("ok"/"warning"/"critical") drive the bar color via templates.
|
||||
type offsiteBoxView struct {
|
||||
Configured bool // s.offsiteBox wired (HETZNER_TOKEN + box id present)
|
||||
Pending bool // configured but the first refresh hasn't landed yet
|
||||
Degraded bool // last fetch failed / zero capacity — values shown stale
|
||||
CapacityStr string
|
||||
UsedStr string
|
||||
DataStr string
|
||||
SnapshotStr string
|
||||
FillPercent float64
|
||||
FillBand string
|
||||
SumQuotaStr string
|
||||
RatioStr string // "1.17×"
|
||||
OversubBand string
|
||||
BoxType string
|
||||
FetchedAt time.Time
|
||||
}
|
||||
|
||||
// offsiteCustomerRow is one per-customer usage/quota row under the panel.
|
||||
type offsiteCustomerRow struct {
|
||||
CustomerName string
|
||||
Dedicated bool
|
||||
QuotaStr string // "" for dedicated
|
||||
UsageStr string // "" when NoUsage
|
||||
UsageBytes int64 // sort key
|
||||
UsagePercent float64 // usage/quota (shared only)
|
||||
UsageBand string // ok|warning|critical for the bar
|
||||
HasBar bool // shared + quota>0 + has usage → render a bar
|
||||
NoUsage bool // no offsite object in the report yet
|
||||
}
|
||||
|
||||
// fmtBytesGB renders bytes as GB (one decimal) below 1 TB, TB (two decimals) above.
|
||||
func fmtBytesGB(b int64) string {
|
||||
if b >= 1<<40 {
|
||||
return fmt.Sprintf("%.2f TB", float64(b)/float64(int64(1)<<40))
|
||||
}
|
||||
return fmt.Sprintf("%.1f GB", float64(b)/float64(int64(1)<<30))
|
||||
}
|
||||
|
||||
// pctBand maps a percent to a band with the given warn/crit (web-side twin of monitor.bandForPercent —
|
||||
// unexported there; a 3-line copy keeps web from depending on monitor internals).
|
||||
func pctBand(pct, warn, crit float64) string {
|
||||
switch {
|
||||
case pct >= crit:
|
||||
return "critical"
|
||||
case pct >= warn:
|
||||
return "warning"
|
||||
default:
|
||||
return "ok"
|
||||
}
|
||||
}
|
||||
|
||||
// offsiteUsageBytes reads repo_size_bytes from a customer's report `offsite` object — the ONLY place the
|
||||
// live repo size exists. ok=false when the report carries no offsite object (never reported). Quota is
|
||||
// NEVER taken from here (the report echo would undercount promises) — only usage. Twin of monitor's
|
||||
// parseOffsite, scoped to the one field the rows need.
|
||||
func offsiteUsageBytes(reportJSON string) (int64, bool) {
|
||||
var r struct {
|
||||
Offsite *struct {
|
||||
RepoSizeBytes int64 `json:"repo_size_bytes"`
|
||||
} `json:"offsite"`
|
||||
}
|
||||
if json.Unmarshal([]byte(reportJSON), &r) != nil || r.Offsite == nil {
|
||||
return 0, false
|
||||
}
|
||||
return r.Offsite.RepoSizeBytes, true
|
||||
}
|
||||
|
||||
// offsiteTile is the compact Dashboard tile: fill% + ratio, band-colored, linking to /offsite.
|
||||
type offsiteTile struct {
|
||||
FillPercent float64
|
||||
RatioStr string
|
||||
Band string // worst of fill/oversub → tile color
|
||||
Degraded bool
|
||||
}
|
||||
|
||||
// offsiteBoxTile builds the Dashboard tile, or nil when there is nothing to show (no provider, or the
|
||||
// first refresh hasn't landed) — the caller then renders NO tile (never a zeroed one).
|
||||
func (s *Server) offsiteBoxTile() *offsiteTile {
|
||||
if s.offsiteBox == nil {
|
||||
return nil
|
||||
}
|
||||
snap, ok := s.offsiteBox()
|
||||
if !ok {
|
||||
return nil // configured but no snapshot yet — absent, not a zeroed tile
|
||||
}
|
||||
band := "ok"
|
||||
switch {
|
||||
case snap.FillBand == "critical":
|
||||
band = "critical"
|
||||
case snap.FillBand == "warning" || snap.OversubBand == "warning":
|
||||
band = "warning"
|
||||
}
|
||||
return &offsiteTile{
|
||||
FillPercent: snap.FillPercent,
|
||||
RatioStr: fmt.Sprintf("%.2f×", snap.Ratio),
|
||||
Band: band,
|
||||
Degraded: snap.Degraded,
|
||||
}
|
||||
}
|
||||
|
||||
// offsiteBoxData builds the panel view + per-customer rows. Nil provider → Configured:false (the panel
|
||||
// renders the honest "not configured"). Never fetches.
|
||||
func (s *Server) offsiteBoxData() (offsiteBoxView, []offsiteCustomerRow) {
|
||||
if s.offsiteBox == nil {
|
||||
return offsiteBoxView{Configured: false}, nil
|
||||
}
|
||||
view := offsiteBoxView{Configured: true}
|
||||
if snap, ok := s.offsiteBox(); ok {
|
||||
view.Pending = false
|
||||
view.Degraded = snap.Degraded
|
||||
view.CapacityStr = fmtBytesGB(snap.CapacityBytes)
|
||||
view.UsedStr = fmtBytesGB(snap.UsedBytes)
|
||||
view.DataStr = fmtBytesGB(snap.DataBytes)
|
||||
view.SnapshotStr = fmtBytesGB(snap.SnapshotBytes)
|
||||
view.FillPercent = snap.FillPercent
|
||||
view.FillBand = snap.FillBand
|
||||
view.SumQuotaStr = fmt.Sprintf("%d GB", snap.SumQuotaGB)
|
||||
view.RatioStr = fmt.Sprintf("%.2f×", snap.Ratio)
|
||||
view.OversubBand = snap.OversubBand
|
||||
view.BoxType = snap.BoxType
|
||||
view.FetchedAt = snap.FetchedAt
|
||||
} else {
|
||||
view.Pending = true // configured, first fetch pending
|
||||
}
|
||||
return view, s.offsiteCustomerRows()
|
||||
}
|
||||
|
||||
// offsiteCustomerRows composes the per-customer rows for every ENABLED-offsite customer: quota from the
|
||||
// authoritative ConfigJSON Descriptor, usage from the report echo. Shared customers get a usage/quota
|
||||
// bar; dedicated customers are listed without one. Sorted by usage descending.
|
||||
func (s *Server) offsiteCustomerRows() []offsiteCustomerRow {
|
||||
cfgs, err := s.store.ListCustomerConfigs()
|
||||
if err != nil {
|
||||
s.logger.Printf("[WARN] offsite box: list configs: %v", err)
|
||||
return nil
|
||||
}
|
||||
usage := map[string]int64{}
|
||||
hasReport := map[string]bool{}
|
||||
if custs, cerr := s.store.GetCustomers(); cerr == nil {
|
||||
for _, c := range custs {
|
||||
if b, ok := offsiteUsageBytes(c.ReportJSON); ok {
|
||||
usage[c.CustomerID], hasReport[c.CustomerID] = b, true
|
||||
}
|
||||
}
|
||||
}
|
||||
var rows []offsiteCustomerRow
|
||||
for _, cfg := range cfgs {
|
||||
d, derr := offsite.ReadDescriptor(cfg.ConfigJSON)
|
||||
if derr != nil || d == nil || !d.Enabled {
|
||||
continue // only customers actually using an offsite tier appear
|
||||
}
|
||||
name := cfg.CustomerName
|
||||
if name == "" {
|
||||
name = cfg.CustomerID
|
||||
}
|
||||
row := offsiteCustomerRow{CustomerName: name, Dedicated: d.Type == "dedicated"}
|
||||
if d.Type == "shared" && d.QuotaGB > 0 {
|
||||
row.QuotaStr = fmt.Sprintf("%d GB", d.QuotaGB)
|
||||
}
|
||||
if hasReport[cfg.CustomerID] {
|
||||
row.UsageBytes = usage[cfg.CustomerID]
|
||||
row.UsageStr = fmtBytesGB(row.UsageBytes)
|
||||
if d.Type == "shared" && d.QuotaGB > 0 {
|
||||
row.UsagePercent = float64(row.UsageBytes) * 100 / float64(int64(d.QuotaGB)<<30)
|
||||
row.UsageBand = pctBand(row.UsagePercent, 90, 95) // matches the per-customer OffsiteChecker bands
|
||||
row.HasBar = true
|
||||
}
|
||||
} else {
|
||||
row.NoUsage = true
|
||||
}
|
||||
rows = append(rows, row)
|
||||
}
|
||||
sort.SliceStable(rows, func(i, j int) bool { return rows[i].UsageBytes > rows[j].UsageBytes })
|
||||
return rows
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
const gib = int64(1) << 30
|
||||
|
||||
// Not configured (no HETZNER_TOKEN) → honest "not configured", never an error or fake zeros.
|
||||
// (Uses renderOffsite from offsite_test.go — the real handler via httptest.)
|
||||
func TestOffsiteBoxPanel_NotConfigured(t *testing.T) {
|
||||
s, _ := newRenderServer(t)
|
||||
// s.offsiteBox left nil
|
||||
body := renderOffsite(t, s)
|
||||
if !strings.Contains(body, "Offsite pool metrics not configured") {
|
||||
t.Fatalf("unconfigured panel must say 'not configured':\n%s", body)
|
||||
}
|
||||
if strings.Contains(body, "0.0 GB") {
|
||||
t.Fatal("unconfigured panel must not render fake zeros")
|
||||
}
|
||||
}
|
||||
|
||||
// With data + per-customer rows: the panel renders capacity/fill/ratio; a shared customer with a report
|
||||
// shows usage, a customer with NO report shows 'no usage reported yet' (never an asserted 0 GB).
|
||||
func TestOffsiteBoxPanel_WithData(t *testing.T) {
|
||||
s, st := newRenderServer(t)
|
||||
s.SetOffsiteBox(func() (monitor.BoxSnapshot, bool) {
|
||||
return monitor.BoxSnapshot{
|
||||
CapacityBytes: 1 << 40, UsedBytes: 300 * gib, DataBytes: 300 * gib,
|
||||
FillPercent: 27, FillBand: "ok", SumQuotaGB: 1200, Ratio: 1.17, OversubBand: "ok",
|
||||
BoxType: "bx11", FetchedAt: time.Now().UTC(),
|
||||
}, true
|
||||
})
|
||||
// "acme": shared, enabled, 500 GB, WITH a report (200 GiB used). "newbie": shared, enabled, no report.
|
||||
saveCfg(t, st, "acme", `{"offsite":{"enabled":true,"type":"shared","quota_gb":500}}`)
|
||||
saveCfg(t, st, "newbie", `{"offsite":{"enabled":true,"type":"shared","quota_gb":300}}`)
|
||||
if err := st.SaveReport("acme", []byte(`{"customer_id":"acme","offsite":{"enabled":true,"repo_size_bytes":`+itoa(200*gib)+`}}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
body := renderOffsite(t, s)
|
||||
for _, want := range []string{"Offsite pool box", "1.00 TB", "1.17×", "acme", "newbie", "no usage reported yet"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Fatalf("panel missing %q:\n%s", want, body)
|
||||
}
|
||||
}
|
||||
if strings.Contains(body, "not configured") {
|
||||
t.Fatal("a configured panel must not show the not-configured message")
|
||||
}
|
||||
}
|
||||
|
||||
func saveCfg(t *testing.T, st *store.Store, id, cfgJSON string) {
|
||||
t.Helper()
|
||||
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: id, CustomerName: id, APIKey: "k-" + id, RetrievalPassword: "p", ConfigJSON: cfgJSON}); err != nil {
|
||||
t.Fatalf("save config %s: %v", id, err)
|
||||
}
|
||||
}
|
||||
|
||||
// itoa avoids importing strconv just for the report body.
|
||||
func itoa(v int64) string {
|
||||
if v == 0 {
|
||||
return "0"
|
||||
}
|
||||
neg := v < 0
|
||||
if neg {
|
||||
v = -v
|
||||
}
|
||||
var b [20]byte
|
||||
i := len(b)
|
||||
for v > 0 {
|
||||
i--
|
||||
b[i] = byte('0' + v%10)
|
||||
v /= 10
|
||||
}
|
||||
if neg {
|
||||
i--
|
||||
b[i] = '-'
|
||||
}
|
||||
return string(b[i:])
|
||||
}
|
||||
@@ -92,11 +92,15 @@ func TestTemplates_DashboardCriticalBadge(t *testing.T) {
|
||||
EventErrors int
|
||||
EventWarnings int
|
||||
}
|
||||
data := []dashboardCustomer{{
|
||||
// v0.64.0: dashboard.html now takes {Customers, OffsiteTile}; OffsiteTile nil → no tile rendered.
|
||||
data := struct {
|
||||
Customers []dashboardCustomer
|
||||
OffsiteTile any
|
||||
}{Customers: []dashboardCustomer{{
|
||||
CustomerSummary: store.CustomerSummary{CustomerID: "c1", CustomerName: "Acme", ReceivedAt: time.Now()},
|
||||
OverallStatus: "ok", BackupAge: "–",
|
||||
EventCriticals: 2, EventErrors: 1, EventWarnings: 0,
|
||||
}}
|
||||
}}}
|
||||
var buf bytes.Buffer
|
||||
if err := s.templates.ExecuteTemplate(&buf, "dashboard.html", data); err != nil {
|
||||
t.Fatalf("render dashboard.html: %v", err)
|
||||
|
||||
@@ -19,6 +19,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/claim"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/gitea"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/intent"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/poke"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/semver"
|
||||
@@ -66,6 +67,7 @@ type Server struct {
|
||||
assetsMgr *assets.Manager
|
||||
gitea *gitea.Client // optional; enables the Day-0 artifact version dropdowns
|
||||
offsite *offsite.Provisioner // optional; enables Hetzner offsite provisioning (SLICE 1)
|
||||
offsiteBox func() (monitor.BoxSnapshot, bool) // optional (v0.64.0, R-5); the pool-box aggregate snapshot accessor
|
||||
tenantsync tenancyProvisioner // optional; enables PBS DR tier provisioning (web/pbsdr.go)
|
||||
claimEngine *claim.Engine // optional; enables the customer-claim resend button (v0.50.0)
|
||||
// intentHub (v0.58.0, Direction-2 immediate-sync) is Bumped by every operator-intent handler
|
||||
@@ -181,6 +183,11 @@ func (s *Server) SetAssetManager(am *assets.Manager) {
|
||||
// offsite enabled returns an error (offsite not configured on this hub).
|
||||
func (s *Server) SetOffsiteProvisioner(p *offsite.Provisioner) { s.offsite = p }
|
||||
|
||||
// SetOffsiteBox wires the pool-box aggregate snapshot accessor (v0.64.0, R-5): the checker's cached
|
||||
// snapshot, read on the Offsite tab + the Dashboard tile. nil (no HETZNER_TOKEN/box id) → both render an
|
||||
// honest "not configured". The web layer NEVER fetches from Hetzner — it only reads this cache.
|
||||
func (s *Server) SetOffsiteBox(fn func() (monitor.BoxSnapshot, bool)) { s.offsiteBox = fn }
|
||||
|
||||
// SetClaimEngine wires the customer-claim code engine for the Setup-tab resend button (v0.50.0).
|
||||
func (s *Server) SetClaimEngine(e *claim.Engine) { s.claimEngine = e }
|
||||
|
||||
@@ -742,8 +749,13 @@ func (s *Server) handleDashboard(w http.ResponseWriter, r *http.Request) {
|
||||
data = append(data, dc)
|
||||
}
|
||||
|
||||
payload := struct {
|
||||
Customers []dashboardCustomer
|
||||
OffsiteTile *offsiteTile // R-5: nil → no tile rendered
|
||||
}{Customers: data, OffsiteTile: s.offsiteBoxTile()}
|
||||
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
if err := s.templates.ExecuteTemplate(w, "dashboard.html", data); err != nil {
|
||||
if err := s.templates.ExecuteTemplate(w, "dashboard.html", payload); err != nil {
|
||||
s.logger.Printf("[ERROR] Template render: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -22,7 +22,15 @@
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
{{if not .}}
|
||||
{{with .OffsiteTile}}
|
||||
<a href="/offsite" class="offsite-tile offsite-tile-{{.Band}}">
|
||||
<span class="offsite-tile-label">Offsite pool</span>
|
||||
<span class="offsite-tile-val">{{formatFloat .FillPercent}}% · {{.RatioStr}}</span>
|
||||
{{if .Degraded}}<span class="offsite-tile-stale">stale</span>{{end}}
|
||||
</a>
|
||||
{{end}}
|
||||
|
||||
{{if not .Customers}}
|
||||
<div class="empty-state">
|
||||
<p>No customer reports received yet.</p>
|
||||
<p class="hint">Configure <code>hub.enabled: true</code> in customer controller.yaml to start receiving reports.</p>
|
||||
@@ -44,7 +52,7 @@
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{range .}}
|
||||
{{range .Customers}}
|
||||
<tr class="status-{{.OverallStatus}}" onclick="window.location='/customers/{{.CustomerID}}'">
|
||||
<td class="customer-name">
|
||||
<span class="status-dot status-dot-{{statusColor .OverallStatus}}"></span>
|
||||
|
||||
@@ -34,6 +34,45 @@
|
||||
<div class="flash flash-success">Endpoint deleted.</div>
|
||||
{{end}}
|
||||
|
||||
<!-- R-5 (v0.64.0): shared pool-box aggregate — total fill, oversubscription, per-customer usage -->
|
||||
<section class="card" style="margin-bottom: 1.5rem;">
|
||||
<h3 style="margin: 0 0 0.75rem;">Offsite pool box</h3>
|
||||
{{if not .OffsiteBox.Configured}}
|
||||
<p class="text-muted" style="font-size: 0.9em;">Offsite pool metrics not configured (no Hetzner token / pool box id on this hub).</p>
|
||||
{{else if .OffsiteBox.Pending}}
|
||||
<p class="text-muted" style="font-size: 0.9em;">Pool metrics loading — the first refresh has not landed yet.</p>
|
||||
{{else}}
|
||||
<table class="detail-table">
|
||||
<tr><th style="width: 12rem;">Box</th><td>{{.OffsiteBox.BoxType}}</td></tr>
|
||||
<tr><th>Capacity</th><td>{{.OffsiteBox.CapacityStr}}</td></tr>
|
||||
<tr><th>Used</th><td>{{.OffsiteBox.UsedStr}} · {{formatFloat .OffsiteBox.FillPercent}}% full <span class="text-muted">(data {{.OffsiteBox.DataStr}} + snapshots {{.OffsiteBox.SnapshotStr}})</span></td></tr>
|
||||
</table>
|
||||
<div class="bar" style="margin: 0.4rem 0 0.9rem;"><div class="bar-fill bar-{{.OffsiteBox.FillBand}}" style="width: {{formatFloat .OffsiteBox.FillPercent}}%;"></div></div>
|
||||
<table class="detail-table">
|
||||
<tr><th style="width: 12rem;">Σ shared soft quotas</th><td>{{.OffsiteBox.SumQuotaStr}}</td></tr>
|
||||
<tr><th>Oversubscription</th><td><span class="status-badge status-badge-{{if eq .OffsiteBox.OversubBand "warning"}}warn{{else}}ok{{end}}">{{.OffsiteBox.RatioStr}}</span> <span class="text-muted">Σ(shared quotas) ÷ capacity</span></td></tr>
|
||||
<tr><th>Fetched</th><td>{{timeAgo .OffsiteBox.FetchedAt}}{{if .OffsiteBox.Degraded}} <span class="status-badge status-badge-warn">STALE — last refresh failed</span>{{end}}</td></tr>
|
||||
</table>
|
||||
|
||||
{{if .OffsiteCusts}}
|
||||
<h4 style="margin: 1rem 0 0.5rem; font-size: 0.85rem; color: var(--text-3); text-transform: uppercase; letter-spacing: 0.03em;">Per-customer usage</h4>
|
||||
<table class="container-table">
|
||||
<thead><tr><th>Customer</th><th>Usage</th><th>Quota</th><th style="width: 35%;"></th></tr></thead>
|
||||
<tbody>
|
||||
{{range .OffsiteCusts}}
|
||||
<tr>
|
||||
<td>{{.CustomerName}}{{if .Dedicated}} <span class="text-muted">(dedicated)</span>{{end}}</td>
|
||||
<td>{{if .NoUsage}}<span class="text-muted">no usage reported yet</span>{{else}}{{.UsageStr}}{{end}}</td>
|
||||
<td>{{if .Dedicated}}<span class="text-muted">dedicated</span>{{else if .QuotaStr}}{{.QuotaStr}}{{else}}<span class="text-muted">—</span>{{end}}</td>
|
||||
<td>{{if .HasBar}}<div class="bar"><div class="bar-fill bar-{{.UsageBand}}" style="width: {{formatFloat .UsagePercent}}%;"></div></div>{{end}}</td>
|
||||
</tr>
|
||||
{{end}}
|
||||
</tbody>
|
||||
</table>
|
||||
{{end}}
|
||||
{{end}}
|
||||
</section>
|
||||
|
||||
{{if .HasEndpoints}}
|
||||
{{range .Endpoints}}
|
||||
<section class="card" style="margin-bottom: 1.5rem;"
|
||||
|
||||
@@ -298,6 +298,25 @@ header h1 {
|
||||
border-radius: var(--radius);
|
||||
transition: width 0.3s ease;
|
||||
}
|
||||
/* Fill bands (v0.64.0, R-5) — exception color: neutral when nominal, amber ≥ warn, red ≥ crit; no green. */
|
||||
.bar-fill.bar-ok { background: var(--text-3); }
|
||||
.bar-fill.bar-warning { background: var(--warn); }
|
||||
.bar-fill.bar-critical { background: var(--crit); }
|
||||
|
||||
/* Offsite pool dashboard tile (v0.64.0, R-5) — compact, band-colored, links to /offsite. */
|
||||
.offsite-tile {
|
||||
display: inline-flex; align-items: baseline; gap: 0.5rem;
|
||||
padding: 0.4rem 0.8rem; margin-bottom: 1rem;
|
||||
border: 1px solid var(--line); border-radius: var(--radius);
|
||||
text-decoration: none; font-size: 0.85rem;
|
||||
}
|
||||
.offsite-tile-label { color: var(--text-3); text-transform: uppercase; font-size: 0.72rem; letter-spacing: 0.03em; }
|
||||
.offsite-tile-val { color: var(--text-1); font-weight: 600; }
|
||||
.offsite-tile-stale { color: var(--warn); font-size: 0.72rem; }
|
||||
.offsite-tile-warning { border-color: var(--warn); }
|
||||
.offsite-tile-warning .offsite-tile-val { color: var(--warn); }
|
||||
.offsite-tile-critical { border-color: var(--crit); }
|
||||
.offsite-tile-critical .offsite-tile-val { color: var(--crit); }
|
||||
|
||||
/* Container table */
|
||||
.container-table {
|
||||
|
||||
Reference in New Issue
Block a user