hub v0.65.0 — PBS DR storage visibility (ep0 usage op) + Offsite tab split + dual dashboard gauges (R-5)

Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.

Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.

- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
  path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
  cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
  no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
  neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
  60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
  the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).

Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
This commit is contained in:
2026-07-17 21:13:30 +02:00
parent 3588a31b78
commit 7f11cfb36c
19 changed files with 856 additions and 22 deletions
+12 -2
View File
@@ -353,8 +353,18 @@ The hub's SECOND forced-command surface on the endpoint: per-customer PBS tenanc
privilege-separated token + dual-grant, the §4a dance automated). Same low-priv user as peersync, its
OWN keypair + script + sudoers drop-in — **the peersync files are untouched** (one script, one job;
peersync script + sudoers sha256 asserted identical before/after at install). Script source of truth:
`scripts/felhom-tenantsync.sh` (**v1.1.0**; JSON stdin/stdout, ops
`provision`/`reissue`/`deprovision`/`fingerprint`). Hub client: `hub/internal/tenantsync`.
`scripts/felhom-tenantsync.sh` (**v1.2.0**; JSON stdin/stdout, ops
`provision`/`reissue`/`deprovision`/`fingerprint`/`usage`). Hub client: `hub/internal/tenantsync`.
> **v1.2.0 (2026-07-17, hub R-5 PBS visibility):** adds the read-only **`usage`** op —
> `{"op":"usage"}` → `{"status":"ok","total":<bytes>,"used":<bytes>,"avail":<bytes>}` (a `df` on the
> `felhom-offsite` datastore path). **No customer_id, no admin token, NO mutation** — the structural twin
> of `fingerprint`. The hub's PBS-DR checker polls it on a 15-min throttle; the Offsite "PBS DR" tab +
> dashboard PBS gauge render it. **This update needs NO sudoers / authorized_keys change** — SAME forced
> command, new op. Update = re-run the on-box script install below (`.bak-<ver>` kept). The hub tolerates
> an ep0 STILL on v1.1.0 (unknown-op → the gauge shows "n/a" until updated; no hub redeploy needed on
> update). Verify after install: `echo '{"op":"usage"}' | ssh -i tenantsync-ssh felhom-peersync@ep0`
> (or the box-local `echo '{"op":"usage"}' | sudo /usr/local/bin/felhom-tenantsync`) returns the totals.
> **v1.1.0 (2026-07-17, deployed to ep0):** adds the `deprovision` op — the deliberate data-deletion
> the slice-1 header reserved, now driven by the hub-side **customer RESET** (hub v0.61.0, operator
+46
View File
@@ -1,5 +1,51 @@
# Felhom Hub — Changelog
## v0.65.0 — PBS DR storage visibility (ep0 `usage` op) + Offsite tab split (Restic / PBS DR) + dual dashboard gauges (R-5) (2026-07-17)
Makes the **PBS DR** storage visible like the restic pool box already is (v0.64.0), the two clearly
differentiated. The scoping correction: "the restic box" and "the PBS box" are NOT two Hetzner Storage
Boxes — **restic** = subaccounts on the shared Hetzner Storage Box (Hetzner API, v0.64.0); **PBS DR** = a
PBS datastore (`felhom-offsite`) on the ep0 endpoint VM (NO Hetzner API). This adds the hub's read of the
PBS datastore fill via **Option A (Viktor-ruled):** a new read-only `usage` op on the `felhom-tenantsync`
ep0 forced command (the structural twin of the existing `fingerprint` op), polled by a new hub checker on
the same 15-min throttle; splits the Offsite page into **Restic / PBS DR tabs**; and puts **two dashboard
gauges** (restic %, PBS %). READ-ONLY against ep0 and Hetzner. Green: `go build/vet/test` + `bash -n` +
the script harness; hub confirm gate OK.
- **Phase-0 probe (gate, PASSED):** on ep0 (PBS 4.2.3), `df -B1 --output=size,used,avail <datastore path>`
(path from `proxmox-backup-manager datastore list --output-format json`) yields the datastore
total/used/avail in **bytes** (live: 39990112256 / 7627939840 / 30686326784 → ~19%), read-only, in the
existing sudo context, no admin token. (PBS 4.2 has no native `datastore usage` command.)
- **`scripts/felhom-tenantsync.sh` → v1.2.0:** a read-only `usage` short-circuit (before the admin-token
generation, like `fingerprint`) → `{"status":"ok","total","used","avail"}`. **No customer_id, no admin
token, NO mutation.**
- **`tenantsync.Client.Usage()`** (`internal/tenantsync/client.go`): `BoxUsage{Total,Used,Avail}` + the
op; an endpoint ≤ v1.1.0 answers `bad_request "unknown op"` → typed `ErrUsageUnsupported` (the
graceful-degradation signal).
- **`monitor.PBSDRBoxChecker`** (`internal/monitor/pbsdr_box.go`, new): clones OffsiteBoxChecker over a
`usageReader` seam (the tenantsync client) — 15-min throttle, cached `PBSBoxSnapshot`, escalation-only
`pbsdr_box_fill` on the customer-less `"pbsdr-box"` scope (operator channel only, no SaveEvent),
recovery re-arm. FILL ONLY (PBS uses namespaces, not quotas — no oversubscription). **THREE states:**
`ok` (bands drive), `unavailable` (ErrUsageUnsupported — expected pre-update, neutral, NO alert, logged
once, the gauge shows n/a), `degraded` (exec failed — keep last snapshot, no band transition).
- **Config + wiring:** `Alerting.PBSDRBoxFill{Warn,Crit}Percent` (default 80/90, independently tunable);
the checker is built ONLY when the tenantsync client exists (shares it), registered in the 60 s sweep,
snapshot handed to the web server. **Graceful degradation: the hub deploy is INDEPENDENT of the ep0
update** — a hub v0.65.0 against an ep0 still on v1.1.0 shows the honest "n/a", lighting up on the next
poll once ep0 is updated (no hub redeploy).
- **Web (`internal/web/pbsdr_box.go` new, `offsite.go`, `templates/offsite.html`, `dashboard.html`,
`style.css`):** the Offsite page splits into **Restic** (the v0.64.0 pool-box panel + per-customer rows)
and **PBS DR** (a new datastore panel — capacity/used/fill bar with band + the endpoint cards, which
belong here: the endpoint IS the PBS host) hash tabs (server-rendered, no JS dependency for the data).
The single dashboard tile becomes two gauges — **RESTIC** (pct·ratio) and **PBS DR** (pct; "n/a" when
unavailable) — each band-colored, each linking to its tab.
- **Tests + red-proofs:** 10 Go tests (Usage parse + unknown-op→typed; checker throttle/bands/pbsdr-box
operator-only/unavailable-no-alert/degraded-keeps-last; PBS panel render × ok/unavailable/not-configured)
+ a bash script harness (usage JSON + exit 0 + **zero mutation** + provision regression). Red-proofs
(run-fail-restore): the op emitting a mutation → the harness zero-mutation assertion fails; the
escalation-only guard removed → in-band re-emit fails; unavailable driving a band → the no-alert test
fails. All confirmed red, then restored.
## v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5) (2026-07-17)
Ships **R-5**: the operator sees the shared pool box's real state on the hub — **total box fill vs
+16 -1
View File
@@ -72,6 +72,11 @@ type Config struct {
OffsiteBoxFillWarnPercent float64 `yaml:"offsite_box_fill_warn_percent"`
OffsiteBoxFillCritPercent float64 `yaml:"offsite_box_fill_crit_percent"`
OffsiteOversubWarnRatio float64 `yaml:"offsite_oversub_warn_ratio"`
// PBS-DR datastore fill thresholds (v0.65.0). Separate keys defaulting to the SAME 80/90 as the
// restic pool box, so PBS can be tuned independently later without touching the restic policy.
// No oversubscription concept for PBS (namespaces, not quotas) — fill only.
PBSDRBoxFillWarnPercent float64 `yaml:"pbsdr_box_fill_warn_percent"`
PBSDRBoxFillCritPercent float64 `yaml:"pbsdr_box_fill_crit_percent"`
} `yaml:"alerting"`
Registry struct {
Image string `yaml:"image"`
@@ -299,6 +304,7 @@ func main() {
// delete ep0 — SPIKE §6). Base is api.hetzner.com (NOT api.hetzner.cloud). Absent token → offsite UI
// still renders, but saving with offsite enabled returns "not configured".
var offsiteBoxChecker *monitor.OffsiteBoxChecker
var pbsdrBoxChecker *monitor.PBSDRBoxChecker // R-5 v0.65.0: PBS-DR datastore fill (constructed with the tenantsync client below)
if tok := os.Getenv("HETZNER_TOKEN"); tok != "" {
poolBoxID, _ := strconv.ParseInt(os.Getenv("HETZNER_POOL_BOX_ID"), 10, 64)
location := os.Getenv("HETZNER_LOCATION")
@@ -440,6 +446,12 @@ func main() {
// the descriptor's last unmet precondition — auto-provision hands-free.
apiHandler.SetWGRegisteredHook(webServer.PBSDRAutoProvision)
logger.Printf("[INFO] PBS DR tenantsync enabled (endpoint %s, user %s; WG-registration auto-provision hook armed)", wgAddr, wgUser)
// R-5 (v0.65.0): the PBS-DR datastore fill checker shares the SAME tenantsync client
// (read-only usage op). Graceful vs an ep0 still on script ≤ v1.1.0 (unavailable state).
pbsdrBoxChecker = monitor.NewPBSDRBoxChecker(tsClient,
cfg.Alerting.PBSDRBoxFillWarnPercent, cfg.Alerting.PBSDRBoxFillCritPercent,
dispatcher.ProcessEvent, logger)
webServer.SetPBSDRBox(pbsdrBoxChecker.Snapshot)
}
} else {
logger.Printf("[INFO] PBS DR tenantsync disabled (key or endpoint not configured)")
@@ -549,7 +561,10 @@ func main() {
hostOOBChecker.Check()
offsiteChecker.Check()
if offsiteBoxChecker != nil {
offsiteBoxChecker.Check() // R-5: pool-box aggregate (fetch-throttled internally)
offsiteBoxChecker.Check() // R-5: restic pool-box aggregate (fetch-throttled internally)
}
if pbsdrBoxChecker != nil {
pbsdrBoxChecker.Check() // R-5 v0.65.0: PBS-DR datastore fill (ep0 usage op, throttled)
}
// v0.46.0: pulled log bundles are transient diagnostics — 72 h TTL.
if n, perr := dataStore.PurgeExpiredLogBundles(time.Now()); perr != nil {
+184
View File
@@ -0,0 +1,184 @@
package monitor
import (
"context"
"encoding/json"
"errors"
"fmt"
"log"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/tenantsync"
)
// usageReader is the ep0 datastore-usage seam (satisfied by *tenantsync.Client). Narrow so tests inject
// a fake and the checker never depends on the full tenantsync surface.
type usageReader interface {
Usage(ctx context.Context) (tenantsync.BoxUsage, error)
}
// PBSDRBoxChecker (v0.65.0) watches the PBS DR datastore (felhom-offsite on ep0) fill — the sibling of
// OffsiteBoxChecker over a DIFFERENT source: the ep0 read-only `usage` op (df on the datastore path), NOT
// the Hetzner API. FILL ONLY (PBS DR uses namespaces, not soft quotas — no oversubscription concept). Same
// shape: 15-min throttle, cached snapshot, escalation-only emit + silent recovery re-arm, degraded-on-
// failure honesty. THREE snapshot states:
// - "ok" → bands drive alerts.
// - "unavailable" → the endpoint script predates the usage op (≤ v1.1.0) → ErrUsageUnsupported. An
// EXPECTED pre-update condition: neutral, NO alert, logged once. The gauge shows n/a.
// - "degraded" → exec failed/timed out → keep the last snapshot, NO band transition (missing ≠ 0%).
//
// Events carry the customer-less scope "pbsdr-box" → operator channel ONLY (processCustomer no-ops on it);
// NO SaveEvent (no customer row to key it to). Deliberately NOT bolted onto OffsiteBoxChecker.
type PBSDRBoxChecker struct {
reader usageReader
fillWarn float64
fillCrit float64
onEvent EventNotifyFunc
logger *log.Logger
now func() time.Time // injectable clock (tests)
mu sync.Mutex
snap PBSBoxSnapshot
haveSnap bool
lastFetch time.Time
fillBand string // "" until first ok eval → born/persistent; only an "ok" fetch updates it
unavailLogged bool // log the "unavailable" state ONCE, not per sweep
}
// pbsdrBoxScope is the customer-less cooldown/display key for PBS-DR box events.
const pbsdrBoxScope = "pbsdr-box"
// PBS-DR snapshot states.
const (
PBSStateOK = "ok"
PBSStateUnavailable = "unavailable"
PBSStateDegraded = "degraded"
)
const (
defaultPBSDRBoxFillWarnPercent = 80.0
defaultPBSDRBoxFillCritPercent = 90.0
)
// PBSBoxSnapshot is the cached PBS-DR datastore aggregate the web layer renders. Sizes in BYTES.
type PBSBoxSnapshot struct {
CapacityBytes int64
UsedBytes int64
FillPercent float64
FillBand string // "" | ok | warning | critical
State string // ok | unavailable | degraded
FetchedAt time.Time
}
// NewPBSDRBoxChecker builds the checker (defaults 80/90 on invalid thresholds). Nothing seeded → the
// first ok Check that finds a breach emits (born/persistent; the dispatcher's 1 h cooldown dedups a restart).
func NewPBSDRBoxChecker(reader usageReader, fillWarn, fillCrit float64, onEvent EventNotifyFunc, logger *log.Logger) *PBSDRBoxChecker {
if fillWarn <= 0 || fillWarn >= 100 {
fillWarn = defaultPBSDRBoxFillWarnPercent
}
if fillCrit <= 0 || fillCrit > 100 || fillCrit <= fillWarn {
fillCrit = defaultPBSDRBoxFillCritPercent
}
if fillCrit <= fillWarn {
fillWarn, fillCrit = defaultPBSDRBoxFillWarnPercent, defaultPBSDRBoxFillCritPercent
}
c := &PBSDRBoxChecker{
reader: reader, fillWarn: fillWarn, fillCrit: fillCrit,
onEvent: onEvent, logger: logger, now: time.Now,
}
logger.Printf("[INFO] PBS-DR box checker initialized: fill warn=%.0f%% crit=%.0f%%, refresh %s", fillWarn, fillCrit, boxFetchInterval)
return c
}
// Check runs on the 60 s sweep; fetches (throttled) and emits on each fill escalation.
func (c *PBSDRBoxChecker) Check() {
c.mu.Lock()
defer c.mu.Unlock()
if c.haveSnap && c.now().Sub(c.lastFetch) < boxFetchInterval {
return // throttle — serve the cache
}
c.lastFetch = c.now() // updated even on failure → retried once per window, not per sweep
usage, err := c.reader.Usage(context.Background())
if err != nil {
if errors.Is(err, tenantsync.ErrUsageUnsupported) {
// EXPECTED pre-update condition — a distinct "unavailable" state, NOT degraded, NO alert.
c.snap = PBSBoxSnapshot{State: PBSStateUnavailable, FetchedAt: c.now()}
c.haveSnap = true
if !c.unavailLogged {
c.logger.Printf("[INFO] PBS-DR box: usage op unavailable — endpoint tenantsync update (v1.2.0) pending; gauge shows n/a until then")
c.unavailLogged = true
}
return // fillBand untouched — an expected data gap never re-arms/transitions
}
c.logger.Printf("[WARN] PBS-DR box: usage read failed (keeping last snapshot): %v", err)
if c.haveSnap {
c.snap.State = PBSStateDegraded // serve last-known, visibly stale; NO band transition
}
return
}
c.unavailLogged = false // recovered from unavailable
snap := PBSBoxSnapshot{CapacityBytes: usage.Total, UsedBytes: usage.Used, State: PBSStateOK, FetchedAt: c.now()}
if usage.Total > 0 {
snap.FillPercent = float64(usage.Used) * 100 / float64(usage.Total)
snap.FillBand = bandForPercent(snap.FillPercent, c.fillWarn, c.fillCrit)
} else {
snap.State = PBSStateDegraded // zero total — guard the division
}
c.snap = snap
c.haveSnap = true
if snap.State != PBSStateOK {
return
}
c.logger.Printf("[INFO] PBS-DR box refreshed: %.1f%% full (%s of %s)", snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes))
// FILL band (escalation-only; recovery re-arms because bandOK has rank 0).
if bandRank(snap.FillBand) > bandRank(c.fillBand) {
c.emitFill(snap, snap.FillBand)
}
c.fillBand = snap.FillBand
}
// Snapshot returns the cached aggregate + whether one exists (mutex copy-out). The web layer's only
// source — it NEVER polls ep0 in the request path.
func (c *PBSDRBoxChecker) Snapshot() (PBSBoxSnapshot, bool) {
c.mu.Lock()
defer c.mu.Unlock()
return c.snap, c.haveSnap
}
// FillState exposes the current band (tests).
func (c *PBSDRBoxChecker) FillState() string {
c.mu.Lock()
defer c.mu.Unlock()
return orUnknown(c.fillBand)
}
func (c *PBSDRBoxChecker) emitFill(snap PBSBoxSnapshot, band string) {
var severity, message string
switch band {
case bandCritical:
severity = "critical"
message = fmt.Sprintf("PBS DR datastore %.0f%% full (%s of %s) — approaching capacity; the offsite DR tier's retention is at risk; free space or grow the datastore",
snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes))
case bandWarning:
severity = "warning"
message = fmt.Sprintf("PBS DR datastore %.0f%% full (%s of %s) — the offsite DR datastore is filling",
snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes))
default:
return
}
details, _ := json.Marshal(map[string]any{
"scope": pbsdrBoxScope, "capacity_bytes": snap.CapacityBytes, "used_bytes": snap.UsedBytes,
"fill_percent": snap.FillPercent, "warn_percent": c.fillWarn, "crit_percent": c.fillCrit,
})
c.logger.Printf("[INFO] PBS-DR box fill: %.0f%% (%s)", snap.FillPercent, band)
// NO SaveEvent — customer-less scope; straight to the operator dispatcher.
if c.onEvent != nil {
c.onEvent(pbsdrBoxScope, "pbsdr_box_fill", severity, message, string(details), "hub")
}
}
+160
View File
@@ -0,0 +1,160 @@
package monitor
import (
"context"
"fmt"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/tenantsync"
)
// fakeUsage is a usageReader for tests: it returns a scripted usage/err and counts calls.
type fakeUsage struct {
mu sync.Mutex
usage tenantsync.BoxUsage
err error
calls int
}
func (f *fakeUsage) Usage(_ context.Context) (tenantsync.BoxUsage, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.calls++
return f.usage, f.err
}
func (f *fakeUsage) set(u tenantsync.BoxUsage, err error) {
f.mu.Lock()
f.usage, f.err = u, err
f.mu.Unlock()
}
func (f *fakeUsage) nCalls() int { f.mu.Lock(); defer f.mu.Unlock(); return f.calls }
func usageBytes(totalGiB, usedGiB int64) tenantsync.BoxUsage {
return tenantsync.BoxUsage{Total: totalGiB * gib, Used: usedGiB * gib, Avail: (totalGiB - usedGiB) * gib}
}
// C1 — throttle: ≤1 ep0 usage call per 15-min window across an hour of 60 s sweeps (≈4, not ≈60).
func TestPBSDRBox_Throttle(t *testing.T) {
f := &fakeUsage{usage: usageBytes(40, 8)}
var cur time.Time
c := NewPBSDRBoxChecker(f, 80, 90, noEvent, quietLog())
c.now = func() time.Time { return cur }
base := time.Now().UTC()
for i := 0; i < 60; i++ {
cur = base.Add(time.Duration(i) * 60 * time.Second)
c.Check()
}
if f.nCalls() < 3 || f.nCalls() > 5 {
t.Fatalf("usage op called %d times over an hour of 60 s sweeps, want ≈4 (throttled to 15 min)", f.nCalls())
}
snap, ok := c.Snapshot()
if !ok || snap.State != PBSStateOK || snap.CapacityBytes != 40*gib {
t.Fatalf("snapshot wrong: %+v ok=%v", snap, ok)
}
}
// C2/C3 — fill bands: escalation-only, in-band no re-emit, recovery re-arm, pbsdr-box scope.
// Red-proof (esc): remove the escalation-only guard → the C3 same-band case re-emits → fails.
func TestPBSDRBox_FillBands(t *testing.T) {
f := &fakeUsage{usage: usageBytes(1000, 750)} // 75%
ev := &capturedBox{}
var cur time.Time
c := NewPBSDRBoxChecker(f, 80, 90, ev.fn, quietLog())
c.now = func() time.Time { return cur }
base := time.Now().UTC()
cur = base
step := func(usedGiB int64) {
cur = cur.Add(16 * time.Minute)
f.set(usageBytes(1000, usedGiB), nil)
c.Check()
}
c.Check() // 75% → no emit
if len(ev.typ) != 0 {
t.Fatalf("75%% must not emit, got %v", ev.typ)
}
step(820) // 82% → one warning
if count(ev.typ, "pbsdr_box_fill") != 1 || ev.sev[0] != "warning" {
t.Fatalf("82%% must warn once, got typ=%v sev=%v", ev.typ, ev.sev)
}
step(850) // 85% same band → no re-emit
if count(ev.typ, "pbsdr_box_fill") != 1 {
t.Fatalf("same-band must NOT re-emit, got %d", count(ev.typ, "pbsdr_box_fill"))
}
step(920) // 92% → critical
if count(ev.typ, "pbsdr_box_fill") != 2 || ev.sev[len(ev.sev)-1] != "critical" {
t.Fatalf("92%% must escalate to critical, got typ=%v sev=%v", ev.typ, ev.sev)
}
step(700) // 70% → recovery re-arm (silent)
if c.FillState() != bandOK {
t.Fatalf("recovery must re-arm to ok, got %s", c.FillState())
}
if count(ev.typ, "pbsdr_box_fill") != 2 {
t.Fatalf("recovery must be silent, got %d", count(ev.typ, "pbsdr_box_fill"))
}
step(920) // re-breach → emits again
if count(ev.typ, "pbsdr_box_fill") != 3 {
t.Fatalf("re-breach after recovery must emit again, got %d", count(ev.typ, "pbsdr_box_fill"))
}
for _, cust := range ev.cust {
if cust != "pbsdr-box" {
t.Fatalf("every emit must carry the pbsdr-box scope, got %q", cust)
}
}
}
// C4 — usage-unsupported (ErrUsageUnsupported) → the distinct "unavailable" state, NOT degraded, NO
// alert, NO band transition. Red-proof (unavail-band): let unavailable drive a band → alert fires → fails.
func TestPBSDRBox_Unavailable(t *testing.T) {
f := &fakeUsage{err: tenantsync.ErrUsageUnsupported}
ev := &capturedBox{}
c := NewPBSDRBoxChecker(f, 80, 90, ev.fn, quietLog())
c.Check()
snap, ok := c.Snapshot()
if !ok || snap.State != PBSStateUnavailable {
t.Fatalf("ErrUsageUnsupported must yield the 'unavailable' state, got %+v", snap)
}
if len(ev.typ) != 0 {
t.Fatalf("unavailable must NOT alert, got %v", ev.typ)
}
if c.FillState() != "unknown" {
t.Fatalf("unavailable must not set a fill band, got %s", c.FillState())
}
}
// C5 — exec error/timeout → last snapshot kept, marked degraded, NO band transition (missing ≠ 0%).
func TestPBSDRBox_DegradedKeepsLast(t *testing.T) {
f := &fakeUsage{usage: usageBytes(1000, 920)} // 92% critical
ev := &capturedBox{}
var cur time.Time
c := NewPBSDRBoxChecker(f, 80, 90, ev.fn, quietLog())
c.now = func() time.Time { return cur }
cur = time.Now().UTC()
c.Check() // establish 92% critical + emit
if count(ev.typ, "pbsdr_box_fill") != 1 {
t.Fatalf("setup: want one critical emit, got %v", ev.typ)
}
before, _ := c.Snapshot()
f.set(tenantsync.BoxUsage{}, fmt.Errorf("ssh timeout"))
cur = cur.Add(16 * time.Minute)
c.Check()
after, ok := c.Snapshot()
if !ok || after.State != PBSStateDegraded {
t.Fatalf("a failed poll must mark degraded (keeping last), got %+v", after)
}
if after.UsedBytes != before.UsedBytes || after.CapacityBytes != before.CapacityBytes {
t.Fatalf("a failed poll must keep the last-known values (before %d/%d, after %d/%d)",
before.UsedBytes, before.CapacityBytes, after.UsedBytes, after.CapacityBytes)
}
if c.FillState() != bandCritical {
t.Fatalf("a failed poll must NOT transition the band, got %s", c.FillState())
}
if count(ev.typ, "pbsdr_box_fill") != 1 {
t.Fatalf("a failed poll must not emit, got %d", count(ev.typ, "pbsdr_box_fill"))
}
}
+15
View File
@@ -64,6 +64,21 @@ func TestProcessEvent_PoolBoxScopeOperatorOnly(t *testing.T) {
}
}
// TestProcessEvent_PBSDRBoxScopeOperatorOnly (v0.65.0, R-5, Scenario C3): a customer-less "pbsdr-box"
// event reaches ONLY the operator channel (same property as pool-box — no prefs → no customer email).
func TestProcessEvent_PBSDRBoxScopeOperatorOnly(t *testing.T) {
st := newDispStore(t)
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
var mu sync.Mutex
var sent []string
d.sendEmailFn = func(to, _, _ string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil }
d.ProcessEvent("pbsdr-box", "pbsdr_box_fill", "warning", "PBS DR datastore 82% full", `{"scope":"pbsdr-box"}`, "hub")
if len(sent) != 1 || sent[0] != "op@felhom.eu" {
t.Fatalf("a pbsdr-box event must reach the operator ONLY, sent=%v", sent)
}
}
// TestProcessEvent_CriticalRoutes: a critical-severity event reaches the operator channel (Scenario E).
func TestProcessEvent_CriticalRoutes(t *testing.T) {
st := newDispStore(t)
+39
View File
@@ -28,6 +28,14 @@ import (
// treats it as a state mismatch (a descriptor should exist; re-issue is the explicit recovery).
var ErrTokenExists = errors.New("tenantsync: token already exists on the endpoint (re-issue is the explicit path)")
// ErrUsageUnsupported is the typed "endpoint script predates the usage op" outcome (v0.65.0): an
// endpoint STILL on tenantsync ≤ v1.1.0 answers `{"op":"usage"}` with its generic `bad_request
// "unknown op"`. It is the GRACEFUL-DEGRADATION signal — the PBS checker maps it to an "unavailable"
// snapshot state (an expected pre-update condition), NOT an error/degraded state, so no alert fires
// and the gauge honestly says "usage not available (endpoint update pending)". The moment ep0 gets
// v1.2.0, the next poll succeeds — no hub redeploy.
var ErrUsageUnsupported = errors.New("tenantsync: endpoint does not support the usage op (script update pending)")
// customerIDRe mirrors the script's validation — refuse client-side before a wasted SSH round-trip.
var customerIDRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,30}$`)
@@ -137,6 +145,28 @@ func (c *Client) Fingerprint(ctx context.Context) (string, error) {
return resp.Fingerprint, nil
}
// BoxUsage is the endpoint PBS datastore's raw fill (bytes), from the read-only usage op (v1.2.0).
type BoxUsage struct {
Total int64
Used int64
Avail int64
}
// Usage returns the felhom-offsite datastore's total/used/avail from the endpoint — a read-only op
// (no token, no namespace, no mutation), the structural twin of Fingerprint. Against an endpoint
// still on tenantsync ≤ v1.1.0 the op is unknown → ErrUsageUnsupported (graceful degradation).
func (c *Client) Usage(ctx context.Context) (BoxUsage, error) {
stdout, stderr, runErr := c.exec(ctx, []byte(`{"op":"usage"}`))
resp, err := parseResponse(stdout, stderr, runErr)
if err != nil {
return BoxUsage{}, err
}
if resp.Total <= 0 {
return BoxUsage{}, fmt.Errorf("tenantsync: usage op returned a non-positive total")
}
return BoxUsage{Total: resp.Total, Used: resp.Used, Avail: resp.Avail}, nil
}
func (c *Client) tenancyOp(ctx context.Context, op, customerID string) (*Result, error) {
if !customerIDRe.MatchString(customerID) {
return nil, fmt.Errorf("tenantsync: invalid customer_id %q", customerID)
@@ -165,6 +195,10 @@ type response struct {
Code string `json:"code"`
Error string `json:"error"`
Deleted bool `json:"deleted"` // deprovision op: whether the namespace existed (was destroyed)
// usage op (v1.2.0): the felhom-offsite datastore's raw fill, in BYTES.
Total int64 `json:"total"`
Used int64 `json:"used"`
Avail int64 `json:"avail"`
Result
}
@@ -178,6 +212,11 @@ func parseResponse(stdout, stderr []byte, runErr error) (*response, error) {
if resp.Code == "token_exists" {
return nil, ErrTokenExists
}
// An endpoint ≤ v1.1.0 has no usage op → its generic `bad_request "unknown op"`. Map it to the
// typed graceful-degradation signal (the exact err_json string is the script's fixed contract).
if resp.Code == "bad_request" && strings.Contains(resp.Error, "unknown op") {
return nil, ErrUsageUnsupported
}
return nil, fmt.Errorf("tenantsync: endpoint refused: %s (code %s)", resp.Error, resp.Code)
}
if runErr != nil {
+32
View File
@@ -208,6 +208,38 @@ func TestFingerprint_Op(t *testing.T) {
}
}
// TestUsage_Op (v0.65.0): the read-only usage op parses total/used/avail (bytes).
func TestUsage_Op(t *testing.T) {
clientPEM, clientSigner := testKeys(t)
_, hostSigner := testKeys(t)
srv := startTestServer(t, hostSigner, clientSigner,
`{"status":"ok","total":39990112256,"used":7628091392,"avail":30686175232}`, "", 0)
c := newTestClient(t, srv.addr, hostKeyLine(t, hostSigner), clientPEM)
u, err := c.Usage(context.Background())
if err != nil {
t.Fatalf("Usage: %v", err)
}
if u.Total != 39990112256 || u.Used != 7628091392 || u.Avail != 30686175232 {
t.Errorf("usage = %+v, want total/used/avail 39990112256/7628091392/30686175232", u)
}
}
// TestUsage_UnknownOpTypedUnsupported (v0.65.0, graceful degradation): an ep0 still on tenantsync
// ≤ v1.1.0 answers `{"op":"usage"}` with its generic `bad_request "unknown op"` (exit 1). That must
// map to the typed ErrUsageUnsupported so the PBS checker records "unavailable", not an error.
func TestUsage_UnknownOpTypedUnsupported(t *testing.T) {
clientPEM, clientSigner := testKeys(t)
_, hostSigner := testKeys(t)
srv := startTestServer(t, hostSigner, clientSigner,
`{"status":"error","code":"bad_request","error":"unknown op"}`, "", 1)
c := newTestClient(t, srv.addr, hostKeyLine(t, hostSigner), clientPEM)
if _, err := c.Usage(context.Background()); !errors.Is(err, ErrUsageUnsupported) {
t.Fatalf("an ep0 without the usage op must map to ErrUsageUnsupported, got %v", err)
}
}
// TestErrors_NeverEmbedStdout is the secret-hygiene contract: stdout may carry the token secret,
// so NO error path may quote it. A malformed-but-secret-bearing stdout must yield an error that
// does not contain the marker bytes. (wgsync quotes stdout in its malformed error — this package
+3 -1
View File
@@ -139,7 +139,8 @@ func (s *Server) handleOffsite(w http.ResponseWriter, r *http.Request) {
})
}
boxView, custRows := s.offsiteBoxData() // R-5 pool-box aggregate panel
boxView, custRows := s.offsiteBoxData() // R-5 restic pool-box aggregate panel (Restic tab)
pbsView := s.pbsdrBoxData() // R-5 v0.65.0 PBS-DR datastore panel (PBS DR tab)
data := map[string]interface{}{
"Endpoints": cards,
@@ -147,6 +148,7 @@ func (s *Server) handleOffsite(w http.ResponseWriter, r *http.Request) {
"Peers": rows,
"OffsiteBox": boxView,
"OffsiteCusts": custRows,
"PBSBox": pbsView,
"CSRFToken": s.getCSRFToken(r),
"Flash": r.URL.Query().Get("flash"),
}
+4 -2
View File
@@ -49,8 +49,10 @@ func TestOffsiteBoxPanel_WithData(t *testing.T) {
t.Fatalf("panel missing %q:\n%s", want, body)
}
}
if strings.Contains(body, "not configured") {
t.Fatal("a configured panel must not show the not-configured message")
// The RESTIC panel is configured — its specific not-configured message must be absent. (The PBS DR
// panel legitimately shows its own "not configured" here since s.pbsdrBox is unset.)
if strings.Contains(body, "Offsite pool metrics not configured") {
t.Fatal("a configured restic panel must not show the not-configured message")
}
}
+89
View File
@@ -0,0 +1,89 @@
package web
// PBS DR datastore fill surfaces (v0.65.0, R-5): the Offsite "PBS DR" tab panel + the Dashboard PBS
// gauge. The web layer reads only the checker's cached snapshot (s.pbsdrBox) — it never polls ep0. The
// snapshot carries its own state (ok / unavailable / degraded) so the UI renders each honestly.
import (
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
)
// pbsdrBoxView is the PBS-DR panel model.
type pbsdrBoxView struct {
Configured bool // s.pbsdrBox wired (tenantsync client present)
Pending bool // configured but the first poll hasn't landed yet
State string // ok | unavailable | degraded
Unavailable bool // endpoint script predates the usage op (≤ v1.1.0) — expected pre-update
Degraded bool // last poll failed — values shown stale
HasFill bool // numbers available (ok or degraded-with-last-known)
CapacityStr string
UsedStr string
FillPercent float64
FillBand string
FetchedAt time.Time
}
// pbsdrTile is the compact Dashboard PBS gauge: fill %, band-colored; "n/a" when unavailable.
type pbsdrTile struct {
Unavailable bool
FillPercent float64
Band string
Degraded bool
}
// pbsdrBoxData builds the PBS-DR panel view. Nil provider → Configured:false ("not configured"). Never polls.
func (s *Server) pbsdrBoxData() pbsdrBoxView {
if s.pbsdrBox == nil {
return pbsdrBoxView{Configured: false}
}
view := pbsdrBoxView{Configured: true}
snap, ok := s.pbsdrBox()
if !ok {
view.Pending = true // configured, first poll pending
return view
}
view.State = snap.State
view.FetchedAt = snap.FetchedAt
switch snap.State {
case monitor.PBSStateUnavailable:
view.Unavailable = true
case monitor.PBSStateDegraded:
view.Degraded = true
if snap.CapacityBytes > 0 { // last-known values persist across a degraded poll
view.HasFill = true
view.CapacityStr = fmtBytesGB(snap.CapacityBytes)
view.UsedStr = fmtBytesGB(snap.UsedBytes)
view.FillPercent = snap.FillPercent
view.FillBand = snap.FillBand
}
default: // ok
view.HasFill = true
view.CapacityStr = fmtBytesGB(snap.CapacityBytes)
view.UsedStr = fmtBytesGB(snap.UsedBytes)
view.FillPercent = snap.FillPercent
view.FillBand = snap.FillBand
}
return view
}
// pbsdrBoxTile builds the Dashboard PBS gauge, or nil when there is nothing to show (no provider, or the
// first poll hasn't landed). Unavailable → a neutral "n/a" gauge (never a fake 0%).
func (s *Server) pbsdrBoxTile() *pbsdrTile {
if s.pbsdrBox == nil {
return nil
}
snap, ok := s.pbsdrBox()
if !ok {
return nil
}
if snap.State == monitor.PBSStateUnavailable {
return &pbsdrTile{Unavailable: true}
}
return &pbsdrTile{
FillPercent: snap.FillPercent,
Band: snap.FillBand,
Degraded: snap.State == monitor.PBSStateDegraded,
}
}
+60
View File
@@ -0,0 +1,60 @@
package web
import (
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
)
// PBS DR panel — configured + ok: renders the datastore, capacity/used, fill bar.
func TestPBSDRPanel_OK(t *testing.T) {
s, _ := newRenderServer(t)
s.SetPBSDRBox(func() (monitor.PBSBoxSnapshot, bool) {
return monitor.PBSBoxSnapshot{
CapacityBytes: 40 * gib, UsedBytes: 8 * gib, FillPercent: 20, FillBand: "ok",
State: monitor.PBSStateOK, FetchedAt: time.Now().UTC(),
}, true
})
body := renderOffsite(t, s)
for _, want := range []string{"PBS DR datastore", "felhom-offsite", "40.0 GB", "8.0 GB", "20% full"} {
if !strings.Contains(body, want) {
t.Fatalf("PBS DR panel missing %q", want)
}
}
}
// PBS DR panel — unavailable (endpoint script ≤ v1.1.0): the honest pending-update message, NOT a fake 0%.
func TestPBSDRPanel_Unavailable(t *testing.T) {
s, _ := newRenderServer(t)
s.SetPBSDRBox(func() (monitor.PBSBoxSnapshot, bool) {
return monitor.PBSBoxSnapshot{State: monitor.PBSStateUnavailable, FetchedAt: time.Now().UTC()}, true
})
body := renderOffsite(t, s)
if !strings.Contains(body, "usage not available") || !strings.Contains(body, "v1.2.0") {
t.Fatalf("unavailable PBS panel must show the pending-update message:\n%s", pbsSection(body))
}
// Never a fake fill for the unavailable box.
if strings.Contains(pbsSection(body), "% full") {
t.Fatal("unavailable PBS panel must not render a fill percentage")
}
}
// PBS DR panel — not configured (no tenantsync client).
func TestPBSDRPanel_NotConfigured(t *testing.T) {
s, _ := newRenderServer(t)
// s.pbsdrBox left nil
body := renderOffsite(t, s)
if !strings.Contains(body, "PBS DR metrics not configured") {
t.Fatalf("unconfigured PBS panel must say 'not configured'")
}
}
// pbsSection returns the slice of the body from the PBS DR datastore heading onward (for scoped asserts).
func pbsSection(body string) string {
if i := strings.Index(body, "PBS DR datastore"); i >= 0 {
return body[i:]
}
return body
}
+2 -1
View File
@@ -92,10 +92,11 @@ func TestTemplates_DashboardCriticalBadge(t *testing.T) {
EventErrors int
EventWarnings int
}
// v0.64.0: dashboard.html now takes {Customers, OffsiteTile}; OffsiteTile nil → no tile rendered.
// v0.65.0: dashboard.html takes {Customers, OffsiteTile, PBSTile}; nil tiles → no gauges rendered.
data := struct {
Customers []dashboardCustomer
OffsiteTile any
PBSTile any
}{Customers: []dashboardCustomer{{
CustomerSummary: store.CustomerSummary{CustomerID: "c1", CustomerName: "Acme", ReceivedAt: time.Now()},
OverallStatus: "ok", BackupAge: "",
+10 -3
View File
@@ -67,7 +67,8 @@ type Server struct {
assetsMgr *assets.Manager
gitea *gitea.Client // optional; enables the Day-0 artifact version dropdowns
offsite *offsite.Provisioner // optional; enables Hetzner offsite provisioning (SLICE 1)
offsiteBox func() (monitor.BoxSnapshot, bool) // optional (v0.64.0, R-5); the pool-box aggregate snapshot accessor
offsiteBox func() (monitor.BoxSnapshot, bool) // optional (v0.64.0, R-5); the restic pool-box aggregate snapshot accessor
pbsdrBox func() (monitor.PBSBoxSnapshot, bool) // optional (v0.65.0, R-5); the PBS-DR datastore fill snapshot accessor
tenantsync tenancyProvisioner // optional; enables PBS DR tier provisioning (web/pbsdr.go)
claimEngine *claim.Engine // optional; enables the customer-claim resend button (v0.50.0)
// intentHub (v0.58.0, Direction-2 immediate-sync) is Bumped by every operator-intent handler
@@ -188,6 +189,11 @@ func (s *Server) SetOffsiteProvisioner(p *offsite.Provisioner) { s.offsite = p }
// honest "not configured". The web layer NEVER fetches from Hetzner — it only reads this cache.
func (s *Server) SetOffsiteBox(fn func() (monitor.BoxSnapshot, bool)) { s.offsiteBox = fn }
// SetPBSDRBox wires the PBS-DR datastore fill snapshot accessor (v0.65.0, R-5): read on the Offsite
// "PBS DR" tab + the Dashboard PBS gauge. nil (no tenantsync client) → "not configured". The snapshot
// carries its own state (ok/unavailable/degraded); the web layer never polls ep0.
func (s *Server) SetPBSDRBox(fn func() (monitor.PBSBoxSnapshot, bool)) { s.pbsdrBox = fn }
// SetClaimEngine wires the customer-claim code engine for the Setup-tab resend button (v0.50.0).
func (s *Server) SetClaimEngine(e *claim.Engine) { s.claimEngine = e }
@@ -751,8 +757,9 @@ func (s *Server) handleDashboard(w http.ResponseWriter, r *http.Request) {
payload := struct {
Customers []dashboardCustomer
OffsiteTile *offsiteTile // R-5: nil → no tile rendered
}{Customers: data, OffsiteTile: s.offsiteBoxTile()}
OffsiteTile *offsiteTile // R-5: restic pool box; nil → no gauge
PBSTile *pbsdrTile // R-5 v0.65.0: PBS DR datastore; nil → no gauge
}{Customers: data, OffsiteTile: s.offsiteBoxTile(), PBSTile: s.pbsdrBoxTile()}
w.Header().Set("Content-Type", "text/html; charset=utf-8")
if err := s.templates.ExecuteTemplate(w, "dashboard.html", payload); err != nil {
+17 -6
View File
@@ -22,12 +22,23 @@
</nav>
</header>
{{with .OffsiteTile}}
<a href="/offsite" class="offsite-tile offsite-tile-{{.Band}}">
<span class="offsite-tile-label">Offsite pool</span>
<span class="offsite-tile-val">{{formatFloat .FillPercent}}% &middot; {{.RatioStr}}</span>
{{if .Degraded}}<span class="offsite-tile-stale">stale</span>{{end}}
</a>
{{if or .OffsiteTile .PBSTile}}
<div class="offsite-gauges">
{{with .OffsiteTile}}
<a href="/offsite#tab=restic" class="offsite-tile offsite-tile-{{.Band}}">
<span class="offsite-tile-label">Restic</span>
<span class="offsite-tile-val">{{formatFloat .FillPercent}}% &middot; {{.RatioStr}}</span>
{{if .Degraded}}<span class="offsite-tile-stale">stale</span>{{end}}
</a>
{{end}}
{{with .PBSTile}}
<a href="/offsite#tab=pbsdr" class="offsite-tile offsite-tile-{{.Band}}">
<span class="offsite-tile-label">PBS DR</span>
{{if .Unavailable}}<span class="offsite-tile-val" style="color:var(--text-3);">n/a</span>{{else}}<span class="offsite-tile-val">{{formatFloat .FillPercent}}%</span>{{end}}
{{if .Degraded}}<span class="offsite-tile-stale">stale</span>{{end}}
</a>
{{end}}
</div>
{{end}}
{{if not .Customers}}
+73 -5
View File
@@ -21,11 +21,14 @@
</nav>
</header>
<h2 style="margin-bottom: 0.5rem;">Offsite connectivity</h2>
<p class="text-muted" style="margin: 0 0 1rem; font-size: 0.85em;">
Peer allocation and endpoint sync currently use the lowest endpoint id (ep0).
Per-endpoint allocation is a future work item.
</p>
<h2 style="margin-bottom: 0.75rem;">Offsite</h2>
<!-- Two offsite tiers, two distinct stores (v0.65.0, R-5): RESTIC = subaccounts on the shared
Hetzner Storage Box; PBS DR = the felhom-offsite datastore on the ep0 endpoint VM. -->
<nav class="tab-nav" id="tab-nav">
<a href="#tab=restic" data-tab="restic" class="active">Restic (shared box)</a>
<a href="#tab=pbsdr" data-tab="pbsdr">PBS DR</a>
</nav>
{{if eq .Flash "endpoint_saved"}}
<div class="flash flash-success">Endpoint saved.</div>
@@ -34,6 +37,9 @@
<div class="flash flash-success">Endpoint deleted.</div>
{{end}}
<!-- ═══ Restic tab ═══ -->
<div class="tab-panel tab-panel-active" data-tab="restic">
<!-- R-5 (v0.64.0): shared pool-box aggregate — total fill, oversubscription, per-customer usage -->
<section class="card" style="margin-bottom: 1.5rem;">
<h3 style="margin: 0 0 0.75rem;">Offsite pool box</h3>
@@ -73,6 +79,40 @@
{{end}}
</section>
</div><!-- ═══ /Restic tab ═══ -->
<!-- ═══ PBS DR tab ═══ -->
<div class="tab-panel" data-tab="pbsdr">
<!-- R-5 (v0.65.0): PBS DR datastore fill (felhom-offsite on ep0, read via the tenantsync usage op) -->
<section class="card" style="margin-bottom: 1.5rem;">
<h3 style="margin: 0 0 0.75rem;">PBS DR datastore</h3>
{{if not .PBSBox.Configured}}
<p class="text-muted" style="font-size: 0.9em;">PBS DR metrics not configured (no offsite endpoint / tenantsync key on this hub).</p>
{{else if .PBSBox.Pending}}
<p class="text-muted" style="font-size: 0.9em;">PBS DR metrics loading — the first poll has not landed yet.</p>
{{else if .PBSBox.Unavailable}}
<p class="text-muted" style="font-size: 0.9em;">PBS DR usage not available — the endpoint script update (felhom-tenantsync v1.2.0) is pending. The gauge lights up on the next poll once ep0 is updated (no hub redeploy).</p>
{{else if .PBSBox.HasFill}}
<table class="detail-table">
<tr><th style="width: 12rem;">Datastore</th><td><code>felhom-offsite</code> (ep0)</td></tr>
<tr><th>Capacity</th><td>{{.PBSBox.CapacityStr}}</td></tr>
<tr><th>Used</th><td>{{.PBSBox.UsedStr}} &middot; {{formatFloat .PBSBox.FillPercent}}% full</td></tr>
</table>
<div class="bar" style="margin: 0.4rem 0 0.9rem;"><div class="bar-fill bar-{{.PBSBox.FillBand}}" style="width: {{formatFloat .PBSBox.FillPercent}}%;"></div></div>
<table class="detail-table">
<tr><th style="width: 12rem;">Polled</th><td>{{timeAgo .PBSBox.FetchedAt}}{{if .PBSBox.Degraded}} <span class="status-badge status-badge-warn">STALE — last poll failed</span>{{end}}</td></tr>
</table>
{{else}}
<p class="text-muted" style="font-size: 0.9em;">PBS DR usage temporarily unavailable (endpoint poll failed).</p>
{{end}}
</section>
<p class="text-muted" style="margin: 0 0 1rem; font-size: 0.85em;">
The endpoint below IS the PBS DR host. Peer allocation and endpoint sync currently use the
lowest endpoint id (ep0); per-endpoint allocation is a future work item.
</p>
{{if .HasEndpoints}}
{{range .Endpoints}}
<section class="card" style="margin-bottom: 1.5rem;"
@@ -185,11 +225,39 @@
</div>
{{end}}
</div><!-- ═══ /PBS DR tab ═══ -->
<footer style="margin-top: 2rem; color: var(--text-muted); font-size: 0.8rem; text-align: center;">
Felhom Hub <span style="font-family: var(--font-mono)">{{hubVersion}}</span>
</footer>
</div>
<script>
// Hash tabs (v0.65.0), mirrored from customer_unified.html. Without JS this never runs — the body
// never gets .js-tabs, so both panels stay visible and the page reads top-to-bottom (data is
// server-rendered; JS only picks which panel is shown). Default tab = restic.
(function() {
var panels = document.querySelectorAll('.tab-panel');
var links = document.querySelectorAll('#tab-nav a');
if (!panels.length || !links.length) return;
document.body.classList.add('js-tabs');
var known = {};
panels.forEach(function(p) { known[p.getAttribute('data-tab')] = true; });
function currentTab() {
var m = (location.hash || '').match(/^#tab=([a-z-]+)$/);
var t = m ? m[1] : '';
return known[t] ? t : 'restic';
}
function activate() {
var tab = currentTab();
panels.forEach(function(p) { p.classList.toggle('tab-panel-active', p.getAttribute('data-tab') === tab); });
links.forEach(function(a) { a.classList.toggle('active', a.getAttribute('data-tab') === tab); });
}
window.addEventListener('hashchange', activate);
activate();
})();
</script>
<script>
// Endpoint management JS (v0.47.0). The server enforces every guard — this layer only
// fills the edit form from a card's data attributes and adds the pubkey-change confirm.
+4
View File
@@ -303,6 +303,10 @@ header h1 {
.bar-fill.bar-warning { background: var(--warn); }
.bar-fill.bar-critical { background: var(--crit); }
/* Offsite dashboard gauges (v0.65.0, R-5) — two side-by-side tiles (Restic, PBS DR). */
.offsite-gauges { display: flex; gap: 0.75rem; flex-wrap: wrap; margin-bottom: 1rem; }
.offsite-gauges .offsite-tile { margin-bottom: 0; }
/* Offsite pool dashboard tile (v0.64.0, R-5) — compact, band-colored, links to /offsite. */
.offsite-tile {
display: inline-flex; align-items: baseline; gap: 0.5rem;
+22 -1
View File
@@ -1,7 +1,12 @@
#!/usr/bin/env bash
# felhom-tenantsync v1.1.0 — the offsite endpoint's per-customer PBS tenancy surface (PBS DR tier
# felhom-tenantsync v1.2.0 — the offsite endpoint's per-customer PBS tenancy surface (PBS DR tier
# SLICE 1; spike SPIKE-pbs-tier-provisioning-2026-07-10 §3).
#
# v1.2.0 (2026-07-17, hub R-5 PBS visibility): + read-only `usage` op — the felhom-offsite datastore's
# total/used/avail (bytes, via `df` on the datastore path). No customer_id, no admin token, NO
# mutation. Structural twin of `fingerprint`. Backward-compatible: an older hub never sends it, and a
# hub that does against an OLD script gets the existing `bad_request "unknown op"` (graceful).
#
# Runs as the SSH forced command for the hub's SECOND `felhom-peersync` key (via sudo — its own
# single sudoers line; the peersync script/key are untouched: one script, one job). JSON on stdin,
# JSON on stdout. Ops:
@@ -23,6 +28,8 @@
# This is the DELIBERATE, gated data-destruction the slice-1 note reserved — the operator RESET
# confirm (typed customer-id + separate escrow-custody ack) is the human decision it demanded.
# {"op":"fingerprint"} → {"status":"ok","fingerprint":"<PBS cert sha256>"}
# {"op":"usage"} → {"status":"ok","total":<bytes>,"used":<bytes>,"avail":<bytes>}
# READ-ONLY datastore fill (df on the felhom-offsite path). No customer_id, no admin token, no mutation.
#
# Secret hygiene (load-bearing):
# - The token secret exists ONLY in memory and in the final stdout JSON — never a file, never
@@ -68,6 +75,20 @@ if [ "$OP" = "fingerprint" ]; then
exit 0
fi
# usage (v1.2.0) — READ-ONLY box-level datastore fill. No customer_id, no admin token, NO mutation
# (a pure `df` on the datastore path). The hub's PBS-DR checker polls this on its 15-min throttle.
# Short-circuits BEFORE the admin-token generation below, exactly like fingerprint.
if [ "$OP" = "usage" ]; then
DS_PATH=$(proxmox-backup-manager datastore list --output-format json \
| jq -r --arg ds "$DS" '.[] | select(.name == $ds) | .path')
[ -n "$DS_PATH" ] || err_json internal "datastore $DS not found"
# df -B1: 1-byte blocks. --output pins the columns; tail -1 skips the header row.
read -r TOTAL USED AVAIL < <(df -B1 --output=size,used,avail "$DS_PATH" | tail -1)
[ -n "$TOTAL" ] && [ -n "$USED" ] && [ -n "$AVAIL" ] || err_json internal "could not read datastore usage"
printf '{"status":"ok","total":%s,"used":%s,"avail":%s}\n' "$TOTAL" "$USED" "$AVAIL"
exit 0
fi
case "$OP" in provision|reissue|deprovision) ;; *) err_json bad_request "unknown op" ;; esac
# customer_id → the namespace AND the token name. Conservative charset (PBS ns + token grammar,
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env bash
# tenantsync-usage-harness.sh (v0.65.0, R-5) — Group-A test for the felhom-tenantsync `usage` op.
# Stubs proxmox-backup-manager + df on PATH (real jq), runs the op, and asserts:
# 1. usage → {"status":"ok","total","used","avail"} + exit 0 + ZERO mutation calls.
# 2. provision (regression) → reaches the mutation path (admin-token generate) — the usage op did
# not disturb the case dispatch — and NEVER emits the usage JSON.
# Non-hollow: the stub LOGS every proxmox-backup-manager invocation; the zero-mutation assertion reads it.
set -uo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
SCRIPT="$HERE/felhom-tenantsync.sh"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
STUB="$WORK/bin"; mkdir -p "$STUB"
export MUTLOG="$WORK/mutations.log"; : > "$MUTLOG"
export CALLLOG="$WORK/calls.log"; : > "$CALLLOG"
DS_PATH="$WORK/ds"; mkdir -p "$DS_PATH"
# Stub proxmox-backup-manager: read-only ops answered; MUTATING subcommands logged to MUTLOG.
cat > "$STUB/proxmox-backup-manager" <<'PBM'
#!/usr/bin/env bash
echo "pbm $*" >> "$CALLLOG"
case "$1 $2" in
"cert info") echo "Fingerprint (sha256): aa:bb:cc:dd" ;;
"datastore list") printf '[{"name":"scratch","path":"/srv/pbs-scratch"},{"name":"felhom-offsite","path":"%s"}]\n' "$DS_PATH" ;;
"user generate-token") echo "MUTATION: $*" >> "$MUTLOG"; echo '{"value":"FAKE-ADMIN-TOKEN"}' ;;
"user delete-token") echo "MUTATION: $*" >> "$MUTLOG" ;;
"acl update") echo "MUTATION: $*" >> "$MUTLOG" ;;
"namespace "*) echo "MUTATION: $*" >> "$MUTLOG" ;;
*) echo "MUTATION: $*" >> "$MUTLOG" ;; # any other subcommand is mutating here
esac
PBM
chmod +x "$STUB/proxmox-backup-manager"
# Stub proxmox-backup-client (used only by the mutating ops; log if invoked).
cat > "$STUB/proxmox-backup-client" <<'PBC'
#!/usr/bin/env bash
echo "MUTATION: pbc $*" >> "$MUTLOG"
PBC
chmod +x "$STUB/proxmox-backup-client"
# Stub df: fixed bytes (total 41943040, used 8388608, avail 33554432 ⇒ 20% full).
cat > "$STUB/df" <<'DF'
#!/usr/bin/env bash
echo "1B-blocks Used Avail"
echo "41943040 8388608 33554432"
DF
chmod +x "$STUB/df"
export DS_PATH CALLLOG MUTLOG
RC=0
fail() { echo "FAIL: $*"; RC=1; }
# --- Test 1: usage op ---
OUT=$(echo '{"op":"usage"}' | PATH="$STUB:$PATH" bash "$SCRIPT" 2>/dev/null); EC=$?
[ "$EC" -eq 0 ] || fail "usage exit $EC (want 0)"
EXP='{"status":"ok","total":41943040,"used":8388608,"avail":33554432}'
[ "$OUT" = "$EXP" ] || fail "usage stdout = $OUT (want $EXP)"
if [ -s "$MUTLOG" ]; then fail "usage ran mutations: $(cat "$MUTLOG")"; else echo "PASS: usage emits totals, exit 0, ZERO mutations"; fi
# --- Test 2: provision regression (dispatch intact; usage op did not break the case) ---
: > "$MUTLOG"
POUT=$(echo '{"op":"provision","customer_id":"harness"}' | PATH="$STUB:$PATH" bash "$SCRIPT" 2>/dev/null || true)
grep -q "generate-token" "$MUTLOG" || fail "provision did not reach the mutation path (case dispatch broken by the usage op?)"
case "$POUT" in *'"total"'*) fail "provision leaked the usage shape: $POUT" ;; esac
[ "$RC" -eq 0 ] && echo "PASS: provision still dispatches to the mutation path (regression)"
exit $RC