hub v0.65.0 — PBS DR storage visibility (ep0 usage op) + Offsite tab split + dual dashboard gauges (R-5)

Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.

Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.

- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
  path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
  cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
  no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
  neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
  60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
  the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).

Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
This commit is contained in:
2026-07-17 21:13:30 +02:00
parent 3588a31b78
commit 7f11cfb36c
19 changed files with 856 additions and 22 deletions
+16 -1
View File
@@ -72,6 +72,11 @@ type Config struct {
OffsiteBoxFillWarnPercent float64 `yaml:"offsite_box_fill_warn_percent"`
OffsiteBoxFillCritPercent float64 `yaml:"offsite_box_fill_crit_percent"`
OffsiteOversubWarnRatio float64 `yaml:"offsite_oversub_warn_ratio"`
// PBS-DR datastore fill thresholds (v0.65.0). Separate keys defaulting to the SAME 80/90 as the
// restic pool box, so PBS can be tuned independently later without touching the restic policy.
// No oversubscription concept for PBS (namespaces, not quotas) — fill only.
PBSDRBoxFillWarnPercent float64 `yaml:"pbsdr_box_fill_warn_percent"`
PBSDRBoxFillCritPercent float64 `yaml:"pbsdr_box_fill_crit_percent"`
} `yaml:"alerting"`
Registry struct {
Image string `yaml:"image"`
@@ -299,6 +304,7 @@ func main() {
// delete ep0 — SPIKE §6). Base is api.hetzner.com (NOT api.hetzner.cloud). Absent token → offsite UI
// still renders, but saving with offsite enabled returns "not configured".
var offsiteBoxChecker *monitor.OffsiteBoxChecker
var pbsdrBoxChecker *monitor.PBSDRBoxChecker // R-5 v0.65.0: PBS-DR datastore fill (constructed with the tenantsync client below)
if tok := os.Getenv("HETZNER_TOKEN"); tok != "" {
poolBoxID, _ := strconv.ParseInt(os.Getenv("HETZNER_POOL_BOX_ID"), 10, 64)
location := os.Getenv("HETZNER_LOCATION")
@@ -440,6 +446,12 @@ func main() {
// the descriptor's last unmet precondition — auto-provision hands-free.
apiHandler.SetWGRegisteredHook(webServer.PBSDRAutoProvision)
logger.Printf("[INFO] PBS DR tenantsync enabled (endpoint %s, user %s; WG-registration auto-provision hook armed)", wgAddr, wgUser)
// R-5 (v0.65.0): the PBS-DR datastore fill checker shares the SAME tenantsync client
// (read-only usage op). Graceful vs an ep0 still on script ≤ v1.1.0 (unavailable state).
pbsdrBoxChecker = monitor.NewPBSDRBoxChecker(tsClient,
cfg.Alerting.PBSDRBoxFillWarnPercent, cfg.Alerting.PBSDRBoxFillCritPercent,
dispatcher.ProcessEvent, logger)
webServer.SetPBSDRBox(pbsdrBoxChecker.Snapshot)
}
} else {
logger.Printf("[INFO] PBS DR tenantsync disabled (key or endpoint not configured)")
@@ -549,7 +561,10 @@ func main() {
hostOOBChecker.Check()
offsiteChecker.Check()
if offsiteBoxChecker != nil {
offsiteBoxChecker.Check() // R-5: pool-box aggregate (fetch-throttled internally)
offsiteBoxChecker.Check() // R-5: restic pool-box aggregate (fetch-throttled internally)
}
if pbsdrBoxChecker != nil {
pbsdrBoxChecker.Check() // R-5 v0.65.0: PBS-DR datastore fill (ep0 usage op, throttled)
}
// v0.46.0: pulled log bundles are transient diagnostics — 72 h TTL.
if n, perr := dataStore.PurgeExpiredLogBundles(time.Now()); perr != nil {