hub v0.65.0 — PBS DR storage visibility (ep0 usage op) + Offsite tab split + dual dashboard gauges (R-5)

Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.

Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.

- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
  path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
  cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
  no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
  neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
  60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
  the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).

Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
This commit is contained in:
2026-07-17 21:13:30 +02:00
parent 3588a31b78
commit 7f11cfb36c
19 changed files with 856 additions and 22 deletions
+12 -2
View File
@@ -353,8 +353,18 @@ The hub's SECOND forced-command surface on the endpoint: per-customer PBS tenanc
privilege-separated token + dual-grant, the §4a dance automated). Same low-priv user as peersync, its
OWN keypair + script + sudoers drop-in — **the peersync files are untouched** (one script, one job;
peersync script + sudoers sha256 asserted identical before/after at install). Script source of truth:
`scripts/felhom-tenantsync.sh` (**v1.1.0**; JSON stdin/stdout, ops
`provision`/`reissue`/`deprovision`/`fingerprint`). Hub client: `hub/internal/tenantsync`.
`scripts/felhom-tenantsync.sh` (**v1.2.0**; JSON stdin/stdout, ops
`provision`/`reissue`/`deprovision`/`fingerprint`/`usage`). Hub client: `hub/internal/tenantsync`.
> **v1.2.0 (2026-07-17, hub R-5 PBS visibility):** adds the read-only **`usage`** op —
> `{"op":"usage"}` → `{"status":"ok","total":<bytes>,"used":<bytes>,"avail":<bytes>}` (a `df` on the
> `felhom-offsite` datastore path). **No customer_id, no admin token, NO mutation** — the structural twin
> of `fingerprint`. The hub's PBS-DR checker polls it on a 15-min throttle; the Offsite "PBS DR" tab +
> dashboard PBS gauge render it. **This update needs NO sudoers / authorized_keys change** — SAME forced
> command, new op. Update = re-run the on-box script install below (`.bak-<ver>` kept). The hub tolerates
> an ep0 STILL on v1.1.0 (unknown-op → the gauge shows "n/a" until updated; no hub redeploy needed on
> update). Verify after install: `echo '{"op":"usage"}' | ssh -i tenantsync-ssh felhom-peersync@ep0`
> (or the box-local `echo '{"op":"usage"}' | sudo /usr/local/bin/felhom-tenantsync`) returns the totals.
> **v1.1.0 (2026-07-17, deployed to ep0):** adds the `deprovision` op — the deliberate data-deletion
> the slice-1 header reserved, now driven by the hub-side **customer RESET** (hub v0.61.0, operator