diff --git a/documentation/runbooks/offsite-endpoint.md b/documentation/runbooks/offsite-endpoint.md index 35ab522..486c1eb 100644 --- a/documentation/runbooks/offsite-endpoint.md +++ b/documentation/runbooks/offsite-endpoint.md @@ -353,8 +353,18 @@ The hub's SECOND forced-command surface on the endpoint: per-customer PBS tenanc privilege-separated token + dual-grant, the §4a dance automated). Same low-priv user as peersync, its OWN keypair + script + sudoers drop-in — **the peersync files are untouched** (one script, one job; peersync script + sudoers sha256 asserted identical before/after at install). Script source of truth: -`scripts/felhom-tenantsync.sh` (**v1.1.0**; JSON stdin/stdout, ops -`provision`/`reissue`/`deprovision`/`fingerprint`). Hub client: `hub/internal/tenantsync`. +`scripts/felhom-tenantsync.sh` (**v1.2.0**; JSON stdin/stdout, ops +`provision`/`reissue`/`deprovision`/`fingerprint`/`usage`). Hub client: `hub/internal/tenantsync`. + +> **v1.2.0 (2026-07-17, hub R-5 PBS visibility):** adds the read-only **`usage`** op — +> `{"op":"usage"}` → `{"status":"ok","total":,"used":,"avail":}` (a `df` on the +> `felhom-offsite` datastore path). **No customer_id, no admin token, NO mutation** — the structural twin +> of `fingerprint`. The hub's PBS-DR checker polls it on a 15-min throttle; the Offsite "PBS DR" tab + +> dashboard PBS gauge render it. **This update needs NO sudoers / authorized_keys change** — SAME forced +> command, new op. Update = re-run the on-box script install below (`.bak-` kept). The hub tolerates +> an ep0 STILL on v1.1.0 (unknown-op → the gauge shows "n/a" until updated; no hub redeploy needed on +> update). Verify after install: `echo '{"op":"usage"}' | ssh -i tenantsync-ssh felhom-peersync@ep0` +> (or the box-local `echo '{"op":"usage"}' | sudo /usr/local/bin/felhom-tenantsync`) returns the totals. > **v1.1.0 (2026-07-17, deployed to ep0):** adds the `deprovision` op — the deliberate data-deletion > the slice-1 header reserved, now driven by the hub-side **customer RESET** (hub v0.61.0, operator diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 3af8d53..4d3cf55 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,51 @@ # Felhom Hub — Changelog +## v0.65.0 — PBS DR storage visibility (ep0 `usage` op) + Offsite tab split (Restic / PBS DR) + dual dashboard gauges (R-5) (2026-07-17) + +Makes the **PBS DR** storage visible like the restic pool box already is (v0.64.0), the two clearly +differentiated. The scoping correction: "the restic box" and "the PBS box" are NOT two Hetzner Storage +Boxes — **restic** = subaccounts on the shared Hetzner Storage Box (Hetzner API, v0.64.0); **PBS DR** = a +PBS datastore (`felhom-offsite`) on the ep0 endpoint VM (NO Hetzner API). This adds the hub's read of the +PBS datastore fill via **Option A (Viktor-ruled):** a new read-only `usage` op on the `felhom-tenantsync` +ep0 forced command (the structural twin of the existing `fingerprint` op), polled by a new hub checker on +the same 15-min throttle; splits the Offsite page into **Restic / PBS DR tabs**; and puts **two dashboard +gauges** (restic %, PBS %). READ-ONLY against ep0 and Hetzner. Green: `go build/vet/test` + `bash -n` + +the script harness; hub confirm gate OK. + +- **Phase-0 probe (gate, PASSED):** on ep0 (PBS 4.2.3), `df -B1 --output=size,used,avail ` + (path from `proxmox-backup-manager datastore list --output-format json`) yields the datastore + total/used/avail in **bytes** (live: 39990112256 / 7627939840 / 30686326784 → ~19%), read-only, in the + existing sudo context, no admin token. (PBS 4.2 has no native `datastore usage` command.) +- **`scripts/felhom-tenantsync.sh` → v1.2.0:** a read-only `usage` short-circuit (before the admin-token + generation, like `fingerprint`) → `{"status":"ok","total","used","avail"}`. **No customer_id, no admin + token, NO mutation.** +- **`tenantsync.Client.Usage()`** (`internal/tenantsync/client.go`): `BoxUsage{Total,Used,Avail}` + the + op; an endpoint ≤ v1.1.0 answers `bad_request "unknown op"` → typed `ErrUsageUnsupported` (the + graceful-degradation signal). +- **`monitor.PBSDRBoxChecker`** (`internal/monitor/pbsdr_box.go`, new): clones OffsiteBoxChecker over a + `usageReader` seam (the tenantsync client) — 15-min throttle, cached `PBSBoxSnapshot`, escalation-only + `pbsdr_box_fill` on the customer-less `"pbsdr-box"` scope (operator channel only, no SaveEvent), + recovery re-arm. FILL ONLY (PBS uses namespaces, not quotas — no oversubscription). **THREE states:** + `ok` (bands drive), `unavailable` (ErrUsageUnsupported — expected pre-update, neutral, NO alert, logged + once, the gauge shows n/a), `degraded` (exec failed — keep last snapshot, no band transition). +- **Config + wiring:** `Alerting.PBSDRBoxFill{Warn,Crit}Percent` (default 80/90, independently tunable); + the checker is built ONLY when the tenantsync client exists (shares it), registered in the 60 s sweep, + snapshot handed to the web server. **Graceful degradation: the hub deploy is INDEPENDENT of the ep0 + update** — a hub v0.65.0 against an ep0 still on v1.1.0 shows the honest "n/a", lighting up on the next + poll once ep0 is updated (no hub redeploy). +- **Web (`internal/web/pbsdr_box.go` new, `offsite.go`, `templates/offsite.html`, `dashboard.html`, + `style.css`):** the Offsite page splits into **Restic** (the v0.64.0 pool-box panel + per-customer rows) + and **PBS DR** (a new datastore panel — capacity/used/fill bar with band + the endpoint cards, which + belong here: the endpoint IS the PBS host) hash tabs (server-rendered, no JS dependency for the data). + The single dashboard tile becomes two gauges — **RESTIC** (pct·ratio) and **PBS DR** (pct; "n/a" when + unavailable) — each band-colored, each linking to its tab. +- **Tests + red-proofs:** 10 Go tests (Usage parse + unknown-op→typed; checker throttle/bands/pbsdr-box + operator-only/unavailable-no-alert/degraded-keeps-last; PBS panel render × ok/unavailable/not-configured) + + a bash script harness (usage JSON + exit 0 + **zero mutation** + provision regression). Red-proofs + (run-fail-restore): the op emitting a mutation → the harness zero-mutation assertion fails; the + escalation-only guard removed → in-band re-emit fails; unavailable driving a band → the no-alert test + fails. All confirmed red, then restored. + ## v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5) (2026-07-17) Ships **R-5**: the operator sees the shared pool box's real state on the hub — **total box fill vs diff --git a/hub/cmd/hub/main.go b/hub/cmd/hub/main.go index ab3fae7..c7df20c 100644 --- a/hub/cmd/hub/main.go +++ b/hub/cmd/hub/main.go @@ -72,6 +72,11 @@ type Config struct { OffsiteBoxFillWarnPercent float64 `yaml:"offsite_box_fill_warn_percent"` OffsiteBoxFillCritPercent float64 `yaml:"offsite_box_fill_crit_percent"` OffsiteOversubWarnRatio float64 `yaml:"offsite_oversub_warn_ratio"` + // PBS-DR datastore fill thresholds (v0.65.0). Separate keys defaulting to the SAME 80/90 as the + // restic pool box, so PBS can be tuned independently later without touching the restic policy. + // No oversubscription concept for PBS (namespaces, not quotas) — fill only. + PBSDRBoxFillWarnPercent float64 `yaml:"pbsdr_box_fill_warn_percent"` + PBSDRBoxFillCritPercent float64 `yaml:"pbsdr_box_fill_crit_percent"` } `yaml:"alerting"` Registry struct { Image string `yaml:"image"` @@ -299,6 +304,7 @@ func main() { // delete ep0 — SPIKE §6). Base is api.hetzner.com (NOT api.hetzner.cloud). Absent token → offsite UI // still renders, but saving with offsite enabled returns "not configured". var offsiteBoxChecker *monitor.OffsiteBoxChecker + var pbsdrBoxChecker *monitor.PBSDRBoxChecker // R-5 v0.65.0: PBS-DR datastore fill (constructed with the tenantsync client below) if tok := os.Getenv("HETZNER_TOKEN"); tok != "" { poolBoxID, _ := strconv.ParseInt(os.Getenv("HETZNER_POOL_BOX_ID"), 10, 64) location := os.Getenv("HETZNER_LOCATION") @@ -440,6 +446,12 @@ func main() { // the descriptor's last unmet precondition — auto-provision hands-free. apiHandler.SetWGRegisteredHook(webServer.PBSDRAutoProvision) logger.Printf("[INFO] PBS DR tenantsync enabled (endpoint %s, user %s; WG-registration auto-provision hook armed)", wgAddr, wgUser) + // R-5 (v0.65.0): the PBS-DR datastore fill checker shares the SAME tenantsync client + // (read-only usage op). Graceful vs an ep0 still on script ≤ v1.1.0 (unavailable state). + pbsdrBoxChecker = monitor.NewPBSDRBoxChecker(tsClient, + cfg.Alerting.PBSDRBoxFillWarnPercent, cfg.Alerting.PBSDRBoxFillCritPercent, + dispatcher.ProcessEvent, logger) + webServer.SetPBSDRBox(pbsdrBoxChecker.Snapshot) } } else { logger.Printf("[INFO] PBS DR tenantsync disabled (key or endpoint not configured)") @@ -549,7 +561,10 @@ func main() { hostOOBChecker.Check() offsiteChecker.Check() if offsiteBoxChecker != nil { - offsiteBoxChecker.Check() // R-5: pool-box aggregate (fetch-throttled internally) + offsiteBoxChecker.Check() // R-5: restic pool-box aggregate (fetch-throttled internally) + } + if pbsdrBoxChecker != nil { + pbsdrBoxChecker.Check() // R-5 v0.65.0: PBS-DR datastore fill (ep0 usage op, throttled) } // v0.46.0: pulled log bundles are transient diagnostics — 72 h TTL. if n, perr := dataStore.PurgeExpiredLogBundles(time.Now()); perr != nil { diff --git a/hub/internal/monitor/pbsdr_box.go b/hub/internal/monitor/pbsdr_box.go new file mode 100644 index 0000000..43eb4af --- /dev/null +++ b/hub/internal/monitor/pbsdr_box.go @@ -0,0 +1,184 @@ +package monitor + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "log" + "sync" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/tenantsync" +) + +// usageReader is the ep0 datastore-usage seam (satisfied by *tenantsync.Client). Narrow so tests inject +// a fake and the checker never depends on the full tenantsync surface. +type usageReader interface { + Usage(ctx context.Context) (tenantsync.BoxUsage, error) +} + +// PBSDRBoxChecker (v0.65.0) watches the PBS DR datastore (felhom-offsite on ep0) fill — the sibling of +// OffsiteBoxChecker over a DIFFERENT source: the ep0 read-only `usage` op (df on the datastore path), NOT +// the Hetzner API. FILL ONLY (PBS DR uses namespaces, not soft quotas — no oversubscription concept). Same +// shape: 15-min throttle, cached snapshot, escalation-only emit + silent recovery re-arm, degraded-on- +// failure honesty. THREE snapshot states: +// - "ok" → bands drive alerts. +// - "unavailable" → the endpoint script predates the usage op (≤ v1.1.0) → ErrUsageUnsupported. An +// EXPECTED pre-update condition: neutral, NO alert, logged once. The gauge shows n/a. +// - "degraded" → exec failed/timed out → keep the last snapshot, NO band transition (missing ≠ 0%). +// +// Events carry the customer-less scope "pbsdr-box" → operator channel ONLY (processCustomer no-ops on it); +// NO SaveEvent (no customer row to key it to). Deliberately NOT bolted onto OffsiteBoxChecker. +type PBSDRBoxChecker struct { + reader usageReader + fillWarn float64 + fillCrit float64 + onEvent EventNotifyFunc + logger *log.Logger + now func() time.Time // injectable clock (tests) + + mu sync.Mutex + snap PBSBoxSnapshot + haveSnap bool + lastFetch time.Time + fillBand string // "" until first ok eval → born/persistent; only an "ok" fetch updates it + unavailLogged bool // log the "unavailable" state ONCE, not per sweep +} + +// pbsdrBoxScope is the customer-less cooldown/display key for PBS-DR box events. +const pbsdrBoxScope = "pbsdr-box" + +// PBS-DR snapshot states. +const ( + PBSStateOK = "ok" + PBSStateUnavailable = "unavailable" + PBSStateDegraded = "degraded" +) + +const ( + defaultPBSDRBoxFillWarnPercent = 80.0 + defaultPBSDRBoxFillCritPercent = 90.0 +) + +// PBSBoxSnapshot is the cached PBS-DR datastore aggregate the web layer renders. Sizes in BYTES. +type PBSBoxSnapshot struct { + CapacityBytes int64 + UsedBytes int64 + FillPercent float64 + FillBand string // "" | ok | warning | critical + State string // ok | unavailable | degraded + FetchedAt time.Time +} + +// NewPBSDRBoxChecker builds the checker (defaults 80/90 on invalid thresholds). Nothing seeded → the +// first ok Check that finds a breach emits (born/persistent; the dispatcher's 1 h cooldown dedups a restart). +func NewPBSDRBoxChecker(reader usageReader, fillWarn, fillCrit float64, onEvent EventNotifyFunc, logger *log.Logger) *PBSDRBoxChecker { + if fillWarn <= 0 || fillWarn >= 100 { + fillWarn = defaultPBSDRBoxFillWarnPercent + } + if fillCrit <= 0 || fillCrit > 100 || fillCrit <= fillWarn { + fillCrit = defaultPBSDRBoxFillCritPercent + } + if fillCrit <= fillWarn { + fillWarn, fillCrit = defaultPBSDRBoxFillWarnPercent, defaultPBSDRBoxFillCritPercent + } + c := &PBSDRBoxChecker{ + reader: reader, fillWarn: fillWarn, fillCrit: fillCrit, + onEvent: onEvent, logger: logger, now: time.Now, + } + logger.Printf("[INFO] PBS-DR box checker initialized: fill warn=%.0f%% crit=%.0f%%, refresh %s", fillWarn, fillCrit, boxFetchInterval) + return c +} + +// Check runs on the 60 s sweep; fetches (throttled) and emits on each fill escalation. +func (c *PBSDRBoxChecker) Check() { + c.mu.Lock() + defer c.mu.Unlock() + + if c.haveSnap && c.now().Sub(c.lastFetch) < boxFetchInterval { + return // throttle — serve the cache + } + c.lastFetch = c.now() // updated even on failure → retried once per window, not per sweep + + usage, err := c.reader.Usage(context.Background()) + if err != nil { + if errors.Is(err, tenantsync.ErrUsageUnsupported) { + // EXPECTED pre-update condition — a distinct "unavailable" state, NOT degraded, NO alert. + c.snap = PBSBoxSnapshot{State: PBSStateUnavailable, FetchedAt: c.now()} + c.haveSnap = true + if !c.unavailLogged { + c.logger.Printf("[INFO] PBS-DR box: usage op unavailable — endpoint tenantsync update (v1.2.0) pending; gauge shows n/a until then") + c.unavailLogged = true + } + return // fillBand untouched — an expected data gap never re-arms/transitions + } + c.logger.Printf("[WARN] PBS-DR box: usage read failed (keeping last snapshot): %v", err) + if c.haveSnap { + c.snap.State = PBSStateDegraded // serve last-known, visibly stale; NO band transition + } + return + } + c.unavailLogged = false // recovered from unavailable + + snap := PBSBoxSnapshot{CapacityBytes: usage.Total, UsedBytes: usage.Used, State: PBSStateOK, FetchedAt: c.now()} + if usage.Total > 0 { + snap.FillPercent = float64(usage.Used) * 100 / float64(usage.Total) + snap.FillBand = bandForPercent(snap.FillPercent, c.fillWarn, c.fillCrit) + } else { + snap.State = PBSStateDegraded // zero total — guard the division + } + c.snap = snap + c.haveSnap = true + + if snap.State != PBSStateOK { + return + } + c.logger.Printf("[INFO] PBS-DR box refreshed: %.1f%% full (%s of %s)", snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes)) + + // FILL band (escalation-only; recovery re-arms because bandOK has rank 0). + if bandRank(snap.FillBand) > bandRank(c.fillBand) { + c.emitFill(snap, snap.FillBand) + } + c.fillBand = snap.FillBand +} + +// Snapshot returns the cached aggregate + whether one exists (mutex copy-out). The web layer's only +// source — it NEVER polls ep0 in the request path. +func (c *PBSDRBoxChecker) Snapshot() (PBSBoxSnapshot, bool) { + c.mu.Lock() + defer c.mu.Unlock() + return c.snap, c.haveSnap +} + +// FillState exposes the current band (tests). +func (c *PBSDRBoxChecker) FillState() string { + c.mu.Lock() + defer c.mu.Unlock() + return orUnknown(c.fillBand) +} + +func (c *PBSDRBoxChecker) emitFill(snap PBSBoxSnapshot, band string) { + var severity, message string + switch band { + case bandCritical: + severity = "critical" + message = fmt.Sprintf("PBS DR datastore %.0f%% full (%s of %s) — approaching capacity; the offsite DR tier's retention is at risk; free space or grow the datastore", + snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes)) + case bandWarning: + severity = "warning" + message = fmt.Sprintf("PBS DR datastore %.0f%% full (%s of %s) — the offsite DR datastore is filling", + snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes)) + default: + return + } + details, _ := json.Marshal(map[string]any{ + "scope": pbsdrBoxScope, "capacity_bytes": snap.CapacityBytes, "used_bytes": snap.UsedBytes, + "fill_percent": snap.FillPercent, "warn_percent": c.fillWarn, "crit_percent": c.fillCrit, + }) + c.logger.Printf("[INFO] PBS-DR box fill: %.0f%% (%s)", snap.FillPercent, band) + // NO SaveEvent — customer-less scope; straight to the operator dispatcher. + if c.onEvent != nil { + c.onEvent(pbsdrBoxScope, "pbsdr_box_fill", severity, message, string(details), "hub") + } +} diff --git a/hub/internal/monitor/pbsdr_box_test.go b/hub/internal/monitor/pbsdr_box_test.go new file mode 100644 index 0000000..4472513 --- /dev/null +++ b/hub/internal/monitor/pbsdr_box_test.go @@ -0,0 +1,160 @@ +package monitor + +import ( + "context" + "fmt" + "sync" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/tenantsync" +) + +// fakeUsage is a usageReader for tests: it returns a scripted usage/err and counts calls. +type fakeUsage struct { + mu sync.Mutex + usage tenantsync.BoxUsage + err error + calls int +} + +func (f *fakeUsage) Usage(_ context.Context) (tenantsync.BoxUsage, error) { + f.mu.Lock() + defer f.mu.Unlock() + f.calls++ + return f.usage, f.err +} +func (f *fakeUsage) set(u tenantsync.BoxUsage, err error) { + f.mu.Lock() + f.usage, f.err = u, err + f.mu.Unlock() +} +func (f *fakeUsage) nCalls() int { f.mu.Lock(); defer f.mu.Unlock(); return f.calls } + +func usageBytes(totalGiB, usedGiB int64) tenantsync.BoxUsage { + return tenantsync.BoxUsage{Total: totalGiB * gib, Used: usedGiB * gib, Avail: (totalGiB - usedGiB) * gib} +} + +// C1 — throttle: ≤1 ep0 usage call per 15-min window across an hour of 60 s sweeps (≈4, not ≈60). +func TestPBSDRBox_Throttle(t *testing.T) { + f := &fakeUsage{usage: usageBytes(40, 8)} + var cur time.Time + c := NewPBSDRBoxChecker(f, 80, 90, noEvent, quietLog()) + c.now = func() time.Time { return cur } + base := time.Now().UTC() + for i := 0; i < 60; i++ { + cur = base.Add(time.Duration(i) * 60 * time.Second) + c.Check() + } + if f.nCalls() < 3 || f.nCalls() > 5 { + t.Fatalf("usage op called %d times over an hour of 60 s sweeps, want ≈4 (throttled to 15 min)", f.nCalls()) + } + snap, ok := c.Snapshot() + if !ok || snap.State != PBSStateOK || snap.CapacityBytes != 40*gib { + t.Fatalf("snapshot wrong: %+v ok=%v", snap, ok) + } +} + +// C2/C3 — fill bands: escalation-only, in-band no re-emit, recovery re-arm, pbsdr-box scope. +// Red-proof (esc): remove the escalation-only guard → the C3 same-band case re-emits → fails. +func TestPBSDRBox_FillBands(t *testing.T) { + f := &fakeUsage{usage: usageBytes(1000, 750)} // 75% + ev := &capturedBox{} + var cur time.Time + c := NewPBSDRBoxChecker(f, 80, 90, ev.fn, quietLog()) + c.now = func() time.Time { return cur } + base := time.Now().UTC() + cur = base + step := func(usedGiB int64) { + cur = cur.Add(16 * time.Minute) + f.set(usageBytes(1000, usedGiB), nil) + c.Check() + } + + c.Check() // 75% → no emit + if len(ev.typ) != 0 { + t.Fatalf("75%% must not emit, got %v", ev.typ) + } + step(820) // 82% → one warning + if count(ev.typ, "pbsdr_box_fill") != 1 || ev.sev[0] != "warning" { + t.Fatalf("82%% must warn once, got typ=%v sev=%v", ev.typ, ev.sev) + } + step(850) // 85% same band → no re-emit + if count(ev.typ, "pbsdr_box_fill") != 1 { + t.Fatalf("same-band must NOT re-emit, got %d", count(ev.typ, "pbsdr_box_fill")) + } + step(920) // 92% → critical + if count(ev.typ, "pbsdr_box_fill") != 2 || ev.sev[len(ev.sev)-1] != "critical" { + t.Fatalf("92%% must escalate to critical, got typ=%v sev=%v", ev.typ, ev.sev) + } + step(700) // 70% → recovery re-arm (silent) + if c.FillState() != bandOK { + t.Fatalf("recovery must re-arm to ok, got %s", c.FillState()) + } + if count(ev.typ, "pbsdr_box_fill") != 2 { + t.Fatalf("recovery must be silent, got %d", count(ev.typ, "pbsdr_box_fill")) + } + step(920) // re-breach → emits again + if count(ev.typ, "pbsdr_box_fill") != 3 { + t.Fatalf("re-breach after recovery must emit again, got %d", count(ev.typ, "pbsdr_box_fill")) + } + for _, cust := range ev.cust { + if cust != "pbsdr-box" { + t.Fatalf("every emit must carry the pbsdr-box scope, got %q", cust) + } + } +} + +// C4 — usage-unsupported (ErrUsageUnsupported) → the distinct "unavailable" state, NOT degraded, NO +// alert, NO band transition. Red-proof (unavail-band): let unavailable drive a band → alert fires → fails. +func TestPBSDRBox_Unavailable(t *testing.T) { + f := &fakeUsage{err: tenantsync.ErrUsageUnsupported} + ev := &capturedBox{} + c := NewPBSDRBoxChecker(f, 80, 90, ev.fn, quietLog()) + c.Check() + snap, ok := c.Snapshot() + if !ok || snap.State != PBSStateUnavailable { + t.Fatalf("ErrUsageUnsupported must yield the 'unavailable' state, got %+v", snap) + } + if len(ev.typ) != 0 { + t.Fatalf("unavailable must NOT alert, got %v", ev.typ) + } + if c.FillState() != "unknown" { + t.Fatalf("unavailable must not set a fill band, got %s", c.FillState()) + } +} + +// C5 — exec error/timeout → last snapshot kept, marked degraded, NO band transition (missing ≠ 0%). +func TestPBSDRBox_DegradedKeepsLast(t *testing.T) { + f := &fakeUsage{usage: usageBytes(1000, 920)} // 92% critical + ev := &capturedBox{} + var cur time.Time + c := NewPBSDRBoxChecker(f, 80, 90, ev.fn, quietLog()) + c.now = func() time.Time { return cur } + cur = time.Now().UTC() + + c.Check() // establish 92% critical + emit + if count(ev.typ, "pbsdr_box_fill") != 1 { + t.Fatalf("setup: want one critical emit, got %v", ev.typ) + } + before, _ := c.Snapshot() + + f.set(tenantsync.BoxUsage{}, fmt.Errorf("ssh timeout")) + cur = cur.Add(16 * time.Minute) + c.Check() + + after, ok := c.Snapshot() + if !ok || after.State != PBSStateDegraded { + t.Fatalf("a failed poll must mark degraded (keeping last), got %+v", after) + } + if after.UsedBytes != before.UsedBytes || after.CapacityBytes != before.CapacityBytes { + t.Fatalf("a failed poll must keep the last-known values (before %d/%d, after %d/%d)", + before.UsedBytes, before.CapacityBytes, after.UsedBytes, after.CapacityBytes) + } + if c.FillState() != bandCritical { + t.Fatalf("a failed poll must NOT transition the band, got %s", c.FillState()) + } + if count(ev.typ, "pbsdr_box_fill") != 1 { + t.Fatalf("a failed poll must not emit, got %d", count(ev.typ, "pbsdr_box_fill")) + } +} diff --git a/hub/internal/notify/dispatcher_test.go b/hub/internal/notify/dispatcher_test.go index 1ba2b7c..5ba82f0 100644 --- a/hub/internal/notify/dispatcher_test.go +++ b/hub/internal/notify/dispatcher_test.go @@ -64,6 +64,21 @@ func TestProcessEvent_PoolBoxScopeOperatorOnly(t *testing.T) { } } +// TestProcessEvent_PBSDRBoxScopeOperatorOnly (v0.65.0, R-5, Scenario C3): a customer-less "pbsdr-box" +// event reaches ONLY the operator channel (same property as pool-box — no prefs → no customer email). +func TestProcessEvent_PBSDRBoxScopeOperatorOnly(t *testing.T) { + st := newDispStore(t) + d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + var mu sync.Mutex + var sent []string + d.sendEmailFn = func(to, _, _ string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil } + + d.ProcessEvent("pbsdr-box", "pbsdr_box_fill", "warning", "PBS DR datastore 82% full", `{"scope":"pbsdr-box"}`, "hub") + if len(sent) != 1 || sent[0] != "op@felhom.eu" { + t.Fatalf("a pbsdr-box event must reach the operator ONLY, sent=%v", sent) + } +} + // TestProcessEvent_CriticalRoutes: a critical-severity event reaches the operator channel (Scenario E). func TestProcessEvent_CriticalRoutes(t *testing.T) { st := newDispStore(t) diff --git a/hub/internal/tenantsync/client.go b/hub/internal/tenantsync/client.go index b9c2697..4eaf60c 100644 --- a/hub/internal/tenantsync/client.go +++ b/hub/internal/tenantsync/client.go @@ -28,6 +28,14 @@ import ( // treats it as a state mismatch (a descriptor should exist; re-issue is the explicit recovery). var ErrTokenExists = errors.New("tenantsync: token already exists on the endpoint (re-issue is the explicit path)") +// ErrUsageUnsupported is the typed "endpoint script predates the usage op" outcome (v0.65.0): an +// endpoint STILL on tenantsync ≤ v1.1.0 answers `{"op":"usage"}` with its generic `bad_request +// "unknown op"`. It is the GRACEFUL-DEGRADATION signal — the PBS checker maps it to an "unavailable" +// snapshot state (an expected pre-update condition), NOT an error/degraded state, so no alert fires +// and the gauge honestly says "usage not available (endpoint update pending)". The moment ep0 gets +// v1.2.0, the next poll succeeds — no hub redeploy. +var ErrUsageUnsupported = errors.New("tenantsync: endpoint does not support the usage op (script update pending)") + // customerIDRe mirrors the script's validation — refuse client-side before a wasted SSH round-trip. var customerIDRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,30}$`) @@ -137,6 +145,28 @@ func (c *Client) Fingerprint(ctx context.Context) (string, error) { return resp.Fingerprint, nil } +// BoxUsage is the endpoint PBS datastore's raw fill (bytes), from the read-only usage op (v1.2.0). +type BoxUsage struct { + Total int64 + Used int64 + Avail int64 +} + +// Usage returns the felhom-offsite datastore's total/used/avail from the endpoint — a read-only op +// (no token, no namespace, no mutation), the structural twin of Fingerprint. Against an endpoint +// still on tenantsync ≤ v1.1.0 the op is unknown → ErrUsageUnsupported (graceful degradation). +func (c *Client) Usage(ctx context.Context) (BoxUsage, error) { + stdout, stderr, runErr := c.exec(ctx, []byte(`{"op":"usage"}`)) + resp, err := parseResponse(stdout, stderr, runErr) + if err != nil { + return BoxUsage{}, err + } + if resp.Total <= 0 { + return BoxUsage{}, fmt.Errorf("tenantsync: usage op returned a non-positive total") + } + return BoxUsage{Total: resp.Total, Used: resp.Used, Avail: resp.Avail}, nil +} + func (c *Client) tenancyOp(ctx context.Context, op, customerID string) (*Result, error) { if !customerIDRe.MatchString(customerID) { return nil, fmt.Errorf("tenantsync: invalid customer_id %q", customerID) @@ -165,6 +195,10 @@ type response struct { Code string `json:"code"` Error string `json:"error"` Deleted bool `json:"deleted"` // deprovision op: whether the namespace existed (was destroyed) + // usage op (v1.2.0): the felhom-offsite datastore's raw fill, in BYTES. + Total int64 `json:"total"` + Used int64 `json:"used"` + Avail int64 `json:"avail"` Result } @@ -178,6 +212,11 @@ func parseResponse(stdout, stderr []byte, runErr error) (*response, error) { if resp.Code == "token_exists" { return nil, ErrTokenExists } + // An endpoint ≤ v1.1.0 has no usage op → its generic `bad_request "unknown op"`. Map it to the + // typed graceful-degradation signal (the exact err_json string is the script's fixed contract). + if resp.Code == "bad_request" && strings.Contains(resp.Error, "unknown op") { + return nil, ErrUsageUnsupported + } return nil, fmt.Errorf("tenantsync: endpoint refused: %s (code %s)", resp.Error, resp.Code) } if runErr != nil { diff --git a/hub/internal/tenantsync/client_test.go b/hub/internal/tenantsync/client_test.go index 98892b4..5553dfc 100644 --- a/hub/internal/tenantsync/client_test.go +++ b/hub/internal/tenantsync/client_test.go @@ -208,6 +208,38 @@ func TestFingerprint_Op(t *testing.T) { } } +// TestUsage_Op (v0.65.0): the read-only usage op parses total/used/avail (bytes). +func TestUsage_Op(t *testing.T) { + clientPEM, clientSigner := testKeys(t) + _, hostSigner := testKeys(t) + srv := startTestServer(t, hostSigner, clientSigner, + `{"status":"ok","total":39990112256,"used":7628091392,"avail":30686175232}`, "", 0) + c := newTestClient(t, srv.addr, hostKeyLine(t, hostSigner), clientPEM) + + u, err := c.Usage(context.Background()) + if err != nil { + t.Fatalf("Usage: %v", err) + } + if u.Total != 39990112256 || u.Used != 7628091392 || u.Avail != 30686175232 { + t.Errorf("usage = %+v, want total/used/avail 39990112256/7628091392/30686175232", u) + } +} + +// TestUsage_UnknownOpTypedUnsupported (v0.65.0, graceful degradation): an ep0 still on tenantsync +// ≤ v1.1.0 answers `{"op":"usage"}` with its generic `bad_request "unknown op"` (exit 1). That must +// map to the typed ErrUsageUnsupported so the PBS checker records "unavailable", not an error. +func TestUsage_UnknownOpTypedUnsupported(t *testing.T) { + clientPEM, clientSigner := testKeys(t) + _, hostSigner := testKeys(t) + srv := startTestServer(t, hostSigner, clientSigner, + `{"status":"error","code":"bad_request","error":"unknown op"}`, "", 1) + c := newTestClient(t, srv.addr, hostKeyLine(t, hostSigner), clientPEM) + + if _, err := c.Usage(context.Background()); !errors.Is(err, ErrUsageUnsupported) { + t.Fatalf("an ep0 without the usage op must map to ErrUsageUnsupported, got %v", err) + } +} + // TestErrors_NeverEmbedStdout is the secret-hygiene contract: stdout may carry the token secret, // so NO error path may quote it. A malformed-but-secret-bearing stdout must yield an error that // does not contain the marker bytes. (wgsync quotes stdout in its malformed error — this package diff --git a/hub/internal/web/offsite.go b/hub/internal/web/offsite.go index 4e1e498..18f8eb4 100644 --- a/hub/internal/web/offsite.go +++ b/hub/internal/web/offsite.go @@ -139,7 +139,8 @@ func (s *Server) handleOffsite(w http.ResponseWriter, r *http.Request) { }) } - boxView, custRows := s.offsiteBoxData() // R-5 pool-box aggregate panel + boxView, custRows := s.offsiteBoxData() // R-5 restic pool-box aggregate panel (Restic tab) + pbsView := s.pbsdrBoxData() // R-5 v0.65.0 PBS-DR datastore panel (PBS DR tab) data := map[string]interface{}{ "Endpoints": cards, @@ -147,6 +148,7 @@ func (s *Server) handleOffsite(w http.ResponseWriter, r *http.Request) { "Peers": rows, "OffsiteBox": boxView, "OffsiteCusts": custRows, + "PBSBox": pbsView, "CSRFToken": s.getCSRFToken(r), "Flash": r.URL.Query().Get("flash"), } diff --git a/hub/internal/web/offsite_box_render_test.go b/hub/internal/web/offsite_box_render_test.go index f5a791a..a20e4a7 100644 --- a/hub/internal/web/offsite_box_render_test.go +++ b/hub/internal/web/offsite_box_render_test.go @@ -49,8 +49,10 @@ func TestOffsiteBoxPanel_WithData(t *testing.T) { t.Fatalf("panel missing %q:\n%s", want, body) } } - if strings.Contains(body, "not configured") { - t.Fatal("a configured panel must not show the not-configured message") + // The RESTIC panel is configured — its specific not-configured message must be absent. (The PBS DR + // panel legitimately shows its own "not configured" here since s.pbsdrBox is unset.) + if strings.Contains(body, "Offsite pool metrics not configured") { + t.Fatal("a configured restic panel must not show the not-configured message") } } diff --git a/hub/internal/web/pbsdr_box.go b/hub/internal/web/pbsdr_box.go new file mode 100644 index 0000000..1962506 --- /dev/null +++ b/hub/internal/web/pbsdr_box.go @@ -0,0 +1,89 @@ +package web + +// PBS DR datastore fill surfaces (v0.65.0, R-5): the Offsite "PBS DR" tab panel + the Dashboard PBS +// gauge. The web layer reads only the checker's cached snapshot (s.pbsdrBox) — it never polls ep0. The +// snapshot carries its own state (ok / unavailable / degraded) so the UI renders each honestly. + +import ( + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/monitor" +) + +// pbsdrBoxView is the PBS-DR panel model. +type pbsdrBoxView struct { + Configured bool // s.pbsdrBox wired (tenantsync client present) + Pending bool // configured but the first poll hasn't landed yet + State string // ok | unavailable | degraded + Unavailable bool // endpoint script predates the usage op (≤ v1.1.0) — expected pre-update + Degraded bool // last poll failed — values shown stale + HasFill bool // numbers available (ok or degraded-with-last-known) + CapacityStr string + UsedStr string + FillPercent float64 + FillBand string + FetchedAt time.Time +} + +// pbsdrTile is the compact Dashboard PBS gauge: fill %, band-colored; "n/a" when unavailable. +type pbsdrTile struct { + Unavailable bool + FillPercent float64 + Band string + Degraded bool +} + +// pbsdrBoxData builds the PBS-DR panel view. Nil provider → Configured:false ("not configured"). Never polls. +func (s *Server) pbsdrBoxData() pbsdrBoxView { + if s.pbsdrBox == nil { + return pbsdrBoxView{Configured: false} + } + view := pbsdrBoxView{Configured: true} + snap, ok := s.pbsdrBox() + if !ok { + view.Pending = true // configured, first poll pending + return view + } + view.State = snap.State + view.FetchedAt = snap.FetchedAt + switch snap.State { + case monitor.PBSStateUnavailable: + view.Unavailable = true + case monitor.PBSStateDegraded: + view.Degraded = true + if snap.CapacityBytes > 0 { // last-known values persist across a degraded poll + view.HasFill = true + view.CapacityStr = fmtBytesGB(snap.CapacityBytes) + view.UsedStr = fmtBytesGB(snap.UsedBytes) + view.FillPercent = snap.FillPercent + view.FillBand = snap.FillBand + } + default: // ok + view.HasFill = true + view.CapacityStr = fmtBytesGB(snap.CapacityBytes) + view.UsedStr = fmtBytesGB(snap.UsedBytes) + view.FillPercent = snap.FillPercent + view.FillBand = snap.FillBand + } + return view +} + +// pbsdrBoxTile builds the Dashboard PBS gauge, or nil when there is nothing to show (no provider, or the +// first poll hasn't landed). Unavailable → a neutral "n/a" gauge (never a fake 0%). +func (s *Server) pbsdrBoxTile() *pbsdrTile { + if s.pbsdrBox == nil { + return nil + } + snap, ok := s.pbsdrBox() + if !ok { + return nil + } + if snap.State == monitor.PBSStateUnavailable { + return &pbsdrTile{Unavailable: true} + } + return &pbsdrTile{ + FillPercent: snap.FillPercent, + Band: snap.FillBand, + Degraded: snap.State == monitor.PBSStateDegraded, + } +} diff --git a/hub/internal/web/pbsdr_box_render_test.go b/hub/internal/web/pbsdr_box_render_test.go new file mode 100644 index 0000000..a1bf9bc --- /dev/null +++ b/hub/internal/web/pbsdr_box_render_test.go @@ -0,0 +1,60 @@ +package web + +import ( + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/monitor" +) + +// PBS DR panel — configured + ok: renders the datastore, capacity/used, fill bar. +func TestPBSDRPanel_OK(t *testing.T) { + s, _ := newRenderServer(t) + s.SetPBSDRBox(func() (monitor.PBSBoxSnapshot, bool) { + return monitor.PBSBoxSnapshot{ + CapacityBytes: 40 * gib, UsedBytes: 8 * gib, FillPercent: 20, FillBand: "ok", + State: monitor.PBSStateOK, FetchedAt: time.Now().UTC(), + }, true + }) + body := renderOffsite(t, s) + for _, want := range []string{"PBS DR datastore", "felhom-offsite", "40.0 GB", "8.0 GB", "20% full"} { + if !strings.Contains(body, want) { + t.Fatalf("PBS DR panel missing %q", want) + } + } +} + +// PBS DR panel — unavailable (endpoint script ≤ v1.1.0): the honest pending-update message, NOT a fake 0%. +func TestPBSDRPanel_Unavailable(t *testing.T) { + s, _ := newRenderServer(t) + s.SetPBSDRBox(func() (monitor.PBSBoxSnapshot, bool) { + return monitor.PBSBoxSnapshot{State: monitor.PBSStateUnavailable, FetchedAt: time.Now().UTC()}, true + }) + body := renderOffsite(t, s) + if !strings.Contains(body, "usage not available") || !strings.Contains(body, "v1.2.0") { + t.Fatalf("unavailable PBS panel must show the pending-update message:\n%s", pbsSection(body)) + } + // Never a fake fill for the unavailable box. + if strings.Contains(pbsSection(body), "% full") { + t.Fatal("unavailable PBS panel must not render a fill percentage") + } +} + +// PBS DR panel — not configured (no tenantsync client). +func TestPBSDRPanel_NotConfigured(t *testing.T) { + s, _ := newRenderServer(t) + // s.pbsdrBox left nil + body := renderOffsite(t, s) + if !strings.Contains(body, "PBS DR metrics not configured") { + t.Fatalf("unconfigured PBS panel must say 'not configured'") + } +} + +// pbsSection returns the slice of the body from the PBS DR datastore heading onward (for scoped asserts). +func pbsSection(body string) string { + if i := strings.Index(body, "PBS DR datastore"); i >= 0 { + return body[i:] + } + return body +} diff --git a/hub/internal/web/render_test.go b/hub/internal/web/render_test.go index e80304d..acbe1a6 100644 --- a/hub/internal/web/render_test.go +++ b/hub/internal/web/render_test.go @@ -92,10 +92,11 @@ func TestTemplates_DashboardCriticalBadge(t *testing.T) { EventErrors int EventWarnings int } - // v0.64.0: dashboard.html now takes {Customers, OffsiteTile}; OffsiteTile nil → no tile rendered. + // v0.65.0: dashboard.html takes {Customers, OffsiteTile, PBSTile}; nil tiles → no gauges rendered. data := struct { Customers []dashboardCustomer OffsiteTile any + PBSTile any }{Customers: []dashboardCustomer{{ CustomerSummary: store.CustomerSummary{CustomerID: "c1", CustomerName: "Acme", ReceivedAt: time.Now()}, OverallStatus: "ok", BackupAge: "–", diff --git a/hub/internal/web/server.go b/hub/internal/web/server.go index 2739884..16997a1 100644 --- a/hub/internal/web/server.go +++ b/hub/internal/web/server.go @@ -67,7 +67,8 @@ type Server struct { assetsMgr *assets.Manager gitea *gitea.Client // optional; enables the Day-0 artifact version dropdowns offsite *offsite.Provisioner // optional; enables Hetzner offsite provisioning (SLICE 1) - offsiteBox func() (monitor.BoxSnapshot, bool) // optional (v0.64.0, R-5); the pool-box aggregate snapshot accessor + offsiteBox func() (monitor.BoxSnapshot, bool) // optional (v0.64.0, R-5); the restic pool-box aggregate snapshot accessor + pbsdrBox func() (monitor.PBSBoxSnapshot, bool) // optional (v0.65.0, R-5); the PBS-DR datastore fill snapshot accessor tenantsync tenancyProvisioner // optional; enables PBS DR tier provisioning (web/pbsdr.go) claimEngine *claim.Engine // optional; enables the customer-claim resend button (v0.50.0) // intentHub (v0.58.0, Direction-2 immediate-sync) is Bumped by every operator-intent handler @@ -188,6 +189,11 @@ func (s *Server) SetOffsiteProvisioner(p *offsite.Provisioner) { s.offsite = p } // honest "not configured". The web layer NEVER fetches from Hetzner — it only reads this cache. func (s *Server) SetOffsiteBox(fn func() (monitor.BoxSnapshot, bool)) { s.offsiteBox = fn } +// SetPBSDRBox wires the PBS-DR datastore fill snapshot accessor (v0.65.0, R-5): read on the Offsite +// "PBS DR" tab + the Dashboard PBS gauge. nil (no tenantsync client) → "not configured". The snapshot +// carries its own state (ok/unavailable/degraded); the web layer never polls ep0. +func (s *Server) SetPBSDRBox(fn func() (monitor.PBSBoxSnapshot, bool)) { s.pbsdrBox = fn } + // SetClaimEngine wires the customer-claim code engine for the Setup-tab resend button (v0.50.0). func (s *Server) SetClaimEngine(e *claim.Engine) { s.claimEngine = e } @@ -751,8 +757,9 @@ func (s *Server) handleDashboard(w http.ResponseWriter, r *http.Request) { payload := struct { Customers []dashboardCustomer - OffsiteTile *offsiteTile // R-5: nil → no tile rendered - }{Customers: data, OffsiteTile: s.offsiteBoxTile()} + OffsiteTile *offsiteTile // R-5: restic pool box; nil → no gauge + PBSTile *pbsdrTile // R-5 v0.65.0: PBS DR datastore; nil → no gauge + }{Customers: data, OffsiteTile: s.offsiteBoxTile(), PBSTile: s.pbsdrBoxTile()} w.Header().Set("Content-Type", "text/html; charset=utf-8") if err := s.templates.ExecuteTemplate(w, "dashboard.html", payload); err != nil { diff --git a/hub/internal/web/templates/dashboard.html b/hub/internal/web/templates/dashboard.html index 9c3d75c..92cf7d2 100644 --- a/hub/internal/web/templates/dashboard.html +++ b/hub/internal/web/templates/dashboard.html @@ -22,12 +22,23 @@ - {{with .OffsiteTile}} - - Offsite pool - {{formatFloat .FillPercent}}% · {{.RatioStr}} - {{if .Degraded}}stale{{end}} - + {{if or .OffsiteTile .PBSTile}} +
+ {{with .OffsiteTile}} + + Restic + {{formatFloat .FillPercent}}% · {{.RatioStr}} + {{if .Degraded}}stale{{end}} + + {{end}} + {{with .PBSTile}} + + PBS DR + {{if .Unavailable}}n/a{{else}}{{formatFloat .FillPercent}}%{{end}} + {{if .Degraded}}stale{{end}} + + {{end}} +
{{end}} {{if not .Customers}} diff --git a/hub/internal/web/templates/offsite.html b/hub/internal/web/templates/offsite.html index c6eb01a..f7b90fd 100644 --- a/hub/internal/web/templates/offsite.html +++ b/hub/internal/web/templates/offsite.html @@ -21,11 +21,14 @@ -

Offsite connectivity

-

- Peer allocation and endpoint sync currently use the lowest endpoint id (ep0). - Per-endpoint allocation is a future work item. -

+

Offsite

+ + + {{if eq .Flash "endpoint_saved"}}
Endpoint saved.
@@ -34,6 +37,9 @@
Endpoint deleted.
{{end}} + +
+

Offsite pool box

@@ -73,6 +79,40 @@ {{end}}
+
+ + +
+ + +
+

PBS DR datastore

+ {{if not .PBSBox.Configured}} +

PBS DR metrics not configured (no offsite endpoint / tenantsync key on this hub).

+ {{else if .PBSBox.Pending}} +

PBS DR metrics loading — the first poll has not landed yet.

+ {{else if .PBSBox.Unavailable}} +

PBS DR usage not available — the endpoint script update (felhom-tenantsync v1.2.0) is pending. The gauge lights up on the next poll once ep0 is updated (no hub redeploy).

+ {{else if .PBSBox.HasFill}} + + + + +
Datastorefelhom-offsite (ep0)
Capacity{{.PBSBox.CapacityStr}}
Used{{.PBSBox.UsedStr}} · {{formatFloat .PBSBox.FillPercent}}% full
+
+ + +
Polled{{timeAgo .PBSBox.FetchedAt}}{{if .PBSBox.Degraded}} STALE — last poll failed{{end}}
+ {{else}} +

PBS DR usage temporarily unavailable (endpoint poll failed).

+ {{end}} +
+ +

+ The endpoint below IS the PBS DR host. Peer allocation and endpoint sync currently use the + lowest endpoint id (ep0); per-endpoint allocation is a future work item. +

+ {{if .HasEndpoints}} {{range .Endpoints}}
{{end}} +
+
Felhom Hub {{hubVersion}}
+ +