hub v0.65.0 — PBS DR storage visibility (ep0 usage op) + Offsite tab split + dual dashboard gauges (R-5)

Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.

Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.

- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
  path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
  cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
  no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
  neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
  60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
  the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).

Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
This commit is contained in:
2026-07-17 21:13:30 +02:00
parent 3588a31b78
commit 7f11cfb36c
19 changed files with 856 additions and 22 deletions
+22 -1
View File
@@ -1,7 +1,12 @@
#!/usr/bin/env bash
# felhom-tenantsync v1.1.0 — the offsite endpoint's per-customer PBS tenancy surface (PBS DR tier
# felhom-tenantsync v1.2.0 — the offsite endpoint's per-customer PBS tenancy surface (PBS DR tier
# SLICE 1; spike SPIKE-pbs-tier-provisioning-2026-07-10 §3).
#
# v1.2.0 (2026-07-17, hub R-5 PBS visibility): + read-only `usage` op — the felhom-offsite datastore's
# total/used/avail (bytes, via `df` on the datastore path). No customer_id, no admin token, NO
# mutation. Structural twin of `fingerprint`. Backward-compatible: an older hub never sends it, and a
# hub that does against an OLD script gets the existing `bad_request "unknown op"` (graceful).
#
# Runs as the SSH forced command for the hub's SECOND `felhom-peersync` key (via sudo — its own
# single sudoers line; the peersync script/key are untouched: one script, one job). JSON on stdin,
# JSON on stdout. Ops:
@@ -23,6 +28,8 @@
# This is the DELIBERATE, gated data-destruction the slice-1 note reserved — the operator RESET
# confirm (typed customer-id + separate escrow-custody ack) is the human decision it demanded.
# {"op":"fingerprint"} → {"status":"ok","fingerprint":"<PBS cert sha256>"}
# {"op":"usage"} → {"status":"ok","total":<bytes>,"used":<bytes>,"avail":<bytes>}
# READ-ONLY datastore fill (df on the felhom-offsite path). No customer_id, no admin token, no mutation.
#
# Secret hygiene (load-bearing):
# - The token secret exists ONLY in memory and in the final stdout JSON — never a file, never
@@ -68,6 +75,20 @@ if [ "$OP" = "fingerprint" ]; then
exit 0
fi
# usage (v1.2.0) — READ-ONLY box-level datastore fill. No customer_id, no admin token, NO mutation
# (a pure `df` on the datastore path). The hub's PBS-DR checker polls this on its 15-min throttle.
# Short-circuits BEFORE the admin-token generation below, exactly like fingerprint.
if [ "$OP" = "usage" ]; then
DS_PATH=$(proxmox-backup-manager datastore list --output-format json \
| jq -r --arg ds "$DS" '.[] | select(.name == $ds) | .path')
[ -n "$DS_PATH" ] || err_json internal "datastore $DS not found"
# df -B1: 1-byte blocks. --output pins the columns; tail -1 skips the header row.
read -r TOTAL USED AVAIL < <(df -B1 --output=size,used,avail "$DS_PATH" | tail -1)
[ -n "$TOTAL" ] && [ -n "$USED" ] && [ -n "$AVAIL" ] || err_json internal "could not read datastore usage"
printf '{"status":"ok","total":%s,"used":%s,"avail":%s}\n' "$TOTAL" "$USED" "$AVAIL"
exit 0
fi
case "$OP" in provision|reissue|deprovision) ;; *) err_json bad_request "unknown op" ;; esac
# customer_id → the namespace AND the token name. Conservative charset (PBS ns + token grammar,