hub v0.65.0 — PBS DR storage visibility (ep0 usage op) + Offsite tab split + dual dashboard gauges (R-5)

Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.

Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.

- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
  path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
  cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
  no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
  neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
  60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
  the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).

Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
This commit is contained in:
2026-07-17 21:13:30 +02:00
parent 3588a31b78
commit 7f11cfb36c
19 changed files with 856 additions and 22 deletions
+46
View File
@@ -1,5 +1,51 @@
# Felhom Hub — Changelog
## v0.65.0 — PBS DR storage visibility (ep0 `usage` op) + Offsite tab split (Restic / PBS DR) + dual dashboard gauges (R-5) (2026-07-17)
Makes the **PBS DR** storage visible like the restic pool box already is (v0.64.0), the two clearly
differentiated. The scoping correction: "the restic box" and "the PBS box" are NOT two Hetzner Storage
Boxes — **restic** = subaccounts on the shared Hetzner Storage Box (Hetzner API, v0.64.0); **PBS DR** = a
PBS datastore (`felhom-offsite`) on the ep0 endpoint VM (NO Hetzner API). This adds the hub's read of the
PBS datastore fill via **Option A (Viktor-ruled):** a new read-only `usage` op on the `felhom-tenantsync`
ep0 forced command (the structural twin of the existing `fingerprint` op), polled by a new hub checker on
the same 15-min throttle; splits the Offsite page into **Restic / PBS DR tabs**; and puts **two dashboard
gauges** (restic %, PBS %). READ-ONLY against ep0 and Hetzner. Green: `go build/vet/test` + `bash -n` +
the script harness; hub confirm gate OK.
- **Phase-0 probe (gate, PASSED):** on ep0 (PBS 4.2.3), `df -B1 --output=size,used,avail <datastore path>`
(path from `proxmox-backup-manager datastore list --output-format json`) yields the datastore
total/used/avail in **bytes** (live: 39990112256 / 7627939840 / 30686326784 → ~19%), read-only, in the
existing sudo context, no admin token. (PBS 4.2 has no native `datastore usage` command.)
- **`scripts/felhom-tenantsync.sh` → v1.2.0:** a read-only `usage` short-circuit (before the admin-token
generation, like `fingerprint`) → `{"status":"ok","total","used","avail"}`. **No customer_id, no admin
token, NO mutation.**
- **`tenantsync.Client.Usage()`** (`internal/tenantsync/client.go`): `BoxUsage{Total,Used,Avail}` + the
op; an endpoint ≤ v1.1.0 answers `bad_request "unknown op"` → typed `ErrUsageUnsupported` (the
graceful-degradation signal).
- **`monitor.PBSDRBoxChecker`** (`internal/monitor/pbsdr_box.go`, new): clones OffsiteBoxChecker over a
`usageReader` seam (the tenantsync client) — 15-min throttle, cached `PBSBoxSnapshot`, escalation-only
`pbsdr_box_fill` on the customer-less `"pbsdr-box"` scope (operator channel only, no SaveEvent),
recovery re-arm. FILL ONLY (PBS uses namespaces, not quotas — no oversubscription). **THREE states:**
`ok` (bands drive), `unavailable` (ErrUsageUnsupported — expected pre-update, neutral, NO alert, logged
once, the gauge shows n/a), `degraded` (exec failed — keep last snapshot, no band transition).
- **Config + wiring:** `Alerting.PBSDRBoxFill{Warn,Crit}Percent` (default 80/90, independently tunable);
the checker is built ONLY when the tenantsync client exists (shares it), registered in the 60 s sweep,
snapshot handed to the web server. **Graceful degradation: the hub deploy is INDEPENDENT of the ep0
update** — a hub v0.65.0 against an ep0 still on v1.1.0 shows the honest "n/a", lighting up on the next
poll once ep0 is updated (no hub redeploy).
- **Web (`internal/web/pbsdr_box.go` new, `offsite.go`, `templates/offsite.html`, `dashboard.html`,
`style.css`):** the Offsite page splits into **Restic** (the v0.64.0 pool-box panel + per-customer rows)
and **PBS DR** (a new datastore panel — capacity/used/fill bar with band + the endpoint cards, which
belong here: the endpoint IS the PBS host) hash tabs (server-rendered, no JS dependency for the data).
The single dashboard tile becomes two gauges — **RESTIC** (pct·ratio) and **PBS DR** (pct; "n/a" when
unavailable) — each band-colored, each linking to its tab.
- **Tests + red-proofs:** 10 Go tests (Usage parse + unknown-op→typed; checker throttle/bands/pbsdr-box
operator-only/unavailable-no-alert/degraded-keeps-last; PBS panel render × ok/unavailable/not-configured)
+ a bash script harness (usage JSON + exit 0 + **zero mutation** + provision regression). Red-proofs
(run-fail-restore): the op emitting a mutation → the harness zero-mutation assertion fails; the
escalation-only guard removed → in-band re-emit fails; unavailable driving a band → the no-alert test
fails. All confirmed red, then restored.
## v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5) (2026-07-17)
Ships **R-5**: the operator sees the shared pool box's real state on the hub — **total box fill vs