hub v0.65.0 — PBS DR storage visibility (ep0 usage op) + Offsite tab split + dual dashboard gauges (R-5)
Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.
Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.
- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).
Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
This commit is contained in:
@@ -353,8 +353,18 @@ The hub's SECOND forced-command surface on the endpoint: per-customer PBS tenanc
|
||||
privilege-separated token + dual-grant, the §4a dance automated). Same low-priv user as peersync, its
|
||||
OWN keypair + script + sudoers drop-in — **the peersync files are untouched** (one script, one job;
|
||||
peersync script + sudoers sha256 asserted identical before/after at install). Script source of truth:
|
||||
`scripts/felhom-tenantsync.sh` (**v1.1.0**; JSON stdin/stdout, ops
|
||||
`provision`/`reissue`/`deprovision`/`fingerprint`). Hub client: `hub/internal/tenantsync`.
|
||||
`scripts/felhom-tenantsync.sh` (**v1.2.0**; JSON stdin/stdout, ops
|
||||
`provision`/`reissue`/`deprovision`/`fingerprint`/`usage`). Hub client: `hub/internal/tenantsync`.
|
||||
|
||||
> **v1.2.0 (2026-07-17, hub R-5 PBS visibility):** adds the read-only **`usage`** op —
|
||||
> `{"op":"usage"}` → `{"status":"ok","total":<bytes>,"used":<bytes>,"avail":<bytes>}` (a `df` on the
|
||||
> `felhom-offsite` datastore path). **No customer_id, no admin token, NO mutation** — the structural twin
|
||||
> of `fingerprint`. The hub's PBS-DR checker polls it on a 15-min throttle; the Offsite "PBS DR" tab +
|
||||
> dashboard PBS gauge render it. **This update needs NO sudoers / authorized_keys change** — SAME forced
|
||||
> command, new op. Update = re-run the on-box script install below (`.bak-<ver>` kept). The hub tolerates
|
||||
> an ep0 STILL on v1.1.0 (unknown-op → the gauge shows "n/a" until updated; no hub redeploy needed on
|
||||
> update). Verify after install: `echo '{"op":"usage"}' | ssh -i tenantsync-ssh felhom-peersync@ep0`
|
||||
> (or the box-local `echo '{"op":"usage"}' | sudo /usr/local/bin/felhom-tenantsync`) returns the totals.
|
||||
|
||||
> **v1.1.0 (2026-07-17, deployed to ep0):** adds the `deprovision` op — the deliberate data-deletion
|
||||
> the slice-1 header reserved, now driven by the hub-side **customer RESET** (hub v0.61.0, operator
|
||||
|
||||
@@ -1,5 +1,51 @@
|
||||
# Felhom Hub — Changelog
|
||||
|
||||
## v0.65.0 — PBS DR storage visibility (ep0 `usage` op) + Offsite tab split (Restic / PBS DR) + dual dashboard gauges (R-5) (2026-07-17)
|
||||
|
||||
Makes the **PBS DR** storage visible like the restic pool box already is (v0.64.0), the two clearly
|
||||
differentiated. The scoping correction: "the restic box" and "the PBS box" are NOT two Hetzner Storage
|
||||
Boxes — **restic** = subaccounts on the shared Hetzner Storage Box (Hetzner API, v0.64.0); **PBS DR** = a
|
||||
PBS datastore (`felhom-offsite`) on the ep0 endpoint VM (NO Hetzner API). This adds the hub's read of the
|
||||
PBS datastore fill via **Option A (Viktor-ruled):** a new read-only `usage` op on the `felhom-tenantsync`
|
||||
ep0 forced command (the structural twin of the existing `fingerprint` op), polled by a new hub checker on
|
||||
the same 15-min throttle; splits the Offsite page into **Restic / PBS DR tabs**; and puts **two dashboard
|
||||
gauges** (restic %, PBS %). READ-ONLY against ep0 and Hetzner. Green: `go build/vet/test` + `bash -n` +
|
||||
the script harness; hub confirm gate OK.
|
||||
|
||||
- **Phase-0 probe (gate, PASSED):** on ep0 (PBS 4.2.3), `df -B1 --output=size,used,avail <datastore path>`
|
||||
(path from `proxmox-backup-manager datastore list --output-format json`) yields the datastore
|
||||
total/used/avail in **bytes** (live: 39990112256 / 7627939840 / 30686326784 → ~19%), read-only, in the
|
||||
existing sudo context, no admin token. (PBS 4.2 has no native `datastore usage` command.)
|
||||
- **`scripts/felhom-tenantsync.sh` → v1.2.0:** a read-only `usage` short-circuit (before the admin-token
|
||||
generation, like `fingerprint`) → `{"status":"ok","total","used","avail"}`. **No customer_id, no admin
|
||||
token, NO mutation.**
|
||||
- **`tenantsync.Client.Usage()`** (`internal/tenantsync/client.go`): `BoxUsage{Total,Used,Avail}` + the
|
||||
op; an endpoint ≤ v1.1.0 answers `bad_request "unknown op"` → typed `ErrUsageUnsupported` (the
|
||||
graceful-degradation signal).
|
||||
- **`monitor.PBSDRBoxChecker`** (`internal/monitor/pbsdr_box.go`, new): clones OffsiteBoxChecker over a
|
||||
`usageReader` seam (the tenantsync client) — 15-min throttle, cached `PBSBoxSnapshot`, escalation-only
|
||||
`pbsdr_box_fill` on the customer-less `"pbsdr-box"` scope (operator channel only, no SaveEvent),
|
||||
recovery re-arm. FILL ONLY (PBS uses namespaces, not quotas — no oversubscription). **THREE states:**
|
||||
`ok` (bands drive), `unavailable` (ErrUsageUnsupported — expected pre-update, neutral, NO alert, logged
|
||||
once, the gauge shows n/a), `degraded` (exec failed — keep last snapshot, no band transition).
|
||||
- **Config + wiring:** `Alerting.PBSDRBoxFill{Warn,Crit}Percent` (default 80/90, independently tunable);
|
||||
the checker is built ONLY when the tenantsync client exists (shares it), registered in the 60 s sweep,
|
||||
snapshot handed to the web server. **Graceful degradation: the hub deploy is INDEPENDENT of the ep0
|
||||
update** — a hub v0.65.0 against an ep0 still on v1.1.0 shows the honest "n/a", lighting up on the next
|
||||
poll once ep0 is updated (no hub redeploy).
|
||||
- **Web (`internal/web/pbsdr_box.go` new, `offsite.go`, `templates/offsite.html`, `dashboard.html`,
|
||||
`style.css`):** the Offsite page splits into **Restic** (the v0.64.0 pool-box panel + per-customer rows)
|
||||
and **PBS DR** (a new datastore panel — capacity/used/fill bar with band + the endpoint cards, which
|
||||
belong here: the endpoint IS the PBS host) hash tabs (server-rendered, no JS dependency for the data).
|
||||
The single dashboard tile becomes two gauges — **RESTIC** (pct·ratio) and **PBS DR** (pct; "n/a" when
|
||||
unavailable) — each band-colored, each linking to its tab.
|
||||
- **Tests + red-proofs:** 10 Go tests (Usage parse + unknown-op→typed; checker throttle/bands/pbsdr-box
|
||||
operator-only/unavailable-no-alert/degraded-keeps-last; PBS panel render × ok/unavailable/not-configured)
|
||||
+ a bash script harness (usage JSON + exit 0 + **zero mutation** + provision regression). Red-proofs
|
||||
(run-fail-restore): the op emitting a mutation → the harness zero-mutation assertion fails; the
|
||||
escalation-only guard removed → in-band re-emit fails; unavailable driving a band → the no-alert test
|
||||
fails. All confirmed red, then restored.
|
||||
|
||||
## v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5) (2026-07-17)
|
||||
|
||||
Ships **R-5**: the operator sees the shared pool box's real state on the hub — **total box fill vs
|
||||
|
||||
+16
-1
@@ -72,6 +72,11 @@ type Config struct {
|
||||
OffsiteBoxFillWarnPercent float64 `yaml:"offsite_box_fill_warn_percent"`
|
||||
OffsiteBoxFillCritPercent float64 `yaml:"offsite_box_fill_crit_percent"`
|
||||
OffsiteOversubWarnRatio float64 `yaml:"offsite_oversub_warn_ratio"`
|
||||
// PBS-DR datastore fill thresholds (v0.65.0). Separate keys defaulting to the SAME 80/90 as the
|
||||
// restic pool box, so PBS can be tuned independently later without touching the restic policy.
|
||||
// No oversubscription concept for PBS (namespaces, not quotas) — fill only.
|
||||
PBSDRBoxFillWarnPercent float64 `yaml:"pbsdr_box_fill_warn_percent"`
|
||||
PBSDRBoxFillCritPercent float64 `yaml:"pbsdr_box_fill_crit_percent"`
|
||||
} `yaml:"alerting"`
|
||||
Registry struct {
|
||||
Image string `yaml:"image"`
|
||||
@@ -299,6 +304,7 @@ func main() {
|
||||
// delete ep0 — SPIKE §6). Base is api.hetzner.com (NOT api.hetzner.cloud). Absent token → offsite UI
|
||||
// still renders, but saving with offsite enabled returns "not configured".
|
||||
var offsiteBoxChecker *monitor.OffsiteBoxChecker
|
||||
var pbsdrBoxChecker *monitor.PBSDRBoxChecker // R-5 v0.65.0: PBS-DR datastore fill (constructed with the tenantsync client below)
|
||||
if tok := os.Getenv("HETZNER_TOKEN"); tok != "" {
|
||||
poolBoxID, _ := strconv.ParseInt(os.Getenv("HETZNER_POOL_BOX_ID"), 10, 64)
|
||||
location := os.Getenv("HETZNER_LOCATION")
|
||||
@@ -440,6 +446,12 @@ func main() {
|
||||
// the descriptor's last unmet precondition — auto-provision hands-free.
|
||||
apiHandler.SetWGRegisteredHook(webServer.PBSDRAutoProvision)
|
||||
logger.Printf("[INFO] PBS DR tenantsync enabled (endpoint %s, user %s; WG-registration auto-provision hook armed)", wgAddr, wgUser)
|
||||
// R-5 (v0.65.0): the PBS-DR datastore fill checker shares the SAME tenantsync client
|
||||
// (read-only usage op). Graceful vs an ep0 still on script ≤ v1.1.0 (unavailable state).
|
||||
pbsdrBoxChecker = monitor.NewPBSDRBoxChecker(tsClient,
|
||||
cfg.Alerting.PBSDRBoxFillWarnPercent, cfg.Alerting.PBSDRBoxFillCritPercent,
|
||||
dispatcher.ProcessEvent, logger)
|
||||
webServer.SetPBSDRBox(pbsdrBoxChecker.Snapshot)
|
||||
}
|
||||
} else {
|
||||
logger.Printf("[INFO] PBS DR tenantsync disabled (key or endpoint not configured)")
|
||||
@@ -549,7 +561,10 @@ func main() {
|
||||
hostOOBChecker.Check()
|
||||
offsiteChecker.Check()
|
||||
if offsiteBoxChecker != nil {
|
||||
offsiteBoxChecker.Check() // R-5: pool-box aggregate (fetch-throttled internally)
|
||||
offsiteBoxChecker.Check() // R-5: restic pool-box aggregate (fetch-throttled internally)
|
||||
}
|
||||
if pbsdrBoxChecker != nil {
|
||||
pbsdrBoxChecker.Check() // R-5 v0.65.0: PBS-DR datastore fill (ep0 usage op, throttled)
|
||||
}
|
||||
// v0.46.0: pulled log bundles are transient diagnostics — 72 h TTL.
|
||||
if n, perr := dataStore.PurgeExpiredLogBundles(time.Now()); perr != nil {
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/tenantsync"
|
||||
)
|
||||
|
||||
// usageReader is the ep0 datastore-usage seam (satisfied by *tenantsync.Client). Narrow so tests inject
|
||||
// a fake and the checker never depends on the full tenantsync surface.
|
||||
type usageReader interface {
|
||||
Usage(ctx context.Context) (tenantsync.BoxUsage, error)
|
||||
}
|
||||
|
||||
// PBSDRBoxChecker (v0.65.0) watches the PBS DR datastore (felhom-offsite on ep0) fill — the sibling of
|
||||
// OffsiteBoxChecker over a DIFFERENT source: the ep0 read-only `usage` op (df on the datastore path), NOT
|
||||
// the Hetzner API. FILL ONLY (PBS DR uses namespaces, not soft quotas — no oversubscription concept). Same
|
||||
// shape: 15-min throttle, cached snapshot, escalation-only emit + silent recovery re-arm, degraded-on-
|
||||
// failure honesty. THREE snapshot states:
|
||||
// - "ok" → bands drive alerts.
|
||||
// - "unavailable" → the endpoint script predates the usage op (≤ v1.1.0) → ErrUsageUnsupported. An
|
||||
// EXPECTED pre-update condition: neutral, NO alert, logged once. The gauge shows n/a.
|
||||
// - "degraded" → exec failed/timed out → keep the last snapshot, NO band transition (missing ≠ 0%).
|
||||
//
|
||||
// Events carry the customer-less scope "pbsdr-box" → operator channel ONLY (processCustomer no-ops on it);
|
||||
// NO SaveEvent (no customer row to key it to). Deliberately NOT bolted onto OffsiteBoxChecker.
|
||||
type PBSDRBoxChecker struct {
|
||||
reader usageReader
|
||||
fillWarn float64
|
||||
fillCrit float64
|
||||
onEvent EventNotifyFunc
|
||||
logger *log.Logger
|
||||
now func() time.Time // injectable clock (tests)
|
||||
|
||||
mu sync.Mutex
|
||||
snap PBSBoxSnapshot
|
||||
haveSnap bool
|
||||
lastFetch time.Time
|
||||
fillBand string // "" until first ok eval → born/persistent; only an "ok" fetch updates it
|
||||
unavailLogged bool // log the "unavailable" state ONCE, not per sweep
|
||||
}
|
||||
|
||||
// pbsdrBoxScope is the customer-less cooldown/display key for PBS-DR box events.
|
||||
const pbsdrBoxScope = "pbsdr-box"
|
||||
|
||||
// PBS-DR snapshot states.
|
||||
const (
|
||||
PBSStateOK = "ok"
|
||||
PBSStateUnavailable = "unavailable"
|
||||
PBSStateDegraded = "degraded"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultPBSDRBoxFillWarnPercent = 80.0
|
||||
defaultPBSDRBoxFillCritPercent = 90.0
|
||||
)
|
||||
|
||||
// PBSBoxSnapshot is the cached PBS-DR datastore aggregate the web layer renders. Sizes in BYTES.
|
||||
type PBSBoxSnapshot struct {
|
||||
CapacityBytes int64
|
||||
UsedBytes int64
|
||||
FillPercent float64
|
||||
FillBand string // "" | ok | warning | critical
|
||||
State string // ok | unavailable | degraded
|
||||
FetchedAt time.Time
|
||||
}
|
||||
|
||||
// NewPBSDRBoxChecker builds the checker (defaults 80/90 on invalid thresholds). Nothing seeded → the
|
||||
// first ok Check that finds a breach emits (born/persistent; the dispatcher's 1 h cooldown dedups a restart).
|
||||
func NewPBSDRBoxChecker(reader usageReader, fillWarn, fillCrit float64, onEvent EventNotifyFunc, logger *log.Logger) *PBSDRBoxChecker {
|
||||
if fillWarn <= 0 || fillWarn >= 100 {
|
||||
fillWarn = defaultPBSDRBoxFillWarnPercent
|
||||
}
|
||||
if fillCrit <= 0 || fillCrit > 100 || fillCrit <= fillWarn {
|
||||
fillCrit = defaultPBSDRBoxFillCritPercent
|
||||
}
|
||||
if fillCrit <= fillWarn {
|
||||
fillWarn, fillCrit = defaultPBSDRBoxFillWarnPercent, defaultPBSDRBoxFillCritPercent
|
||||
}
|
||||
c := &PBSDRBoxChecker{
|
||||
reader: reader, fillWarn: fillWarn, fillCrit: fillCrit,
|
||||
onEvent: onEvent, logger: logger, now: time.Now,
|
||||
}
|
||||
logger.Printf("[INFO] PBS-DR box checker initialized: fill warn=%.0f%% crit=%.0f%%, refresh %s", fillWarn, fillCrit, boxFetchInterval)
|
||||
return c
|
||||
}
|
||||
|
||||
// Check runs on the 60 s sweep; fetches (throttled) and emits on each fill escalation.
|
||||
func (c *PBSDRBoxChecker) Check() {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
if c.haveSnap && c.now().Sub(c.lastFetch) < boxFetchInterval {
|
||||
return // throttle — serve the cache
|
||||
}
|
||||
c.lastFetch = c.now() // updated even on failure → retried once per window, not per sweep
|
||||
|
||||
usage, err := c.reader.Usage(context.Background())
|
||||
if err != nil {
|
||||
if errors.Is(err, tenantsync.ErrUsageUnsupported) {
|
||||
// EXPECTED pre-update condition — a distinct "unavailable" state, NOT degraded, NO alert.
|
||||
c.snap = PBSBoxSnapshot{State: PBSStateUnavailable, FetchedAt: c.now()}
|
||||
c.haveSnap = true
|
||||
if !c.unavailLogged {
|
||||
c.logger.Printf("[INFO] PBS-DR box: usage op unavailable — endpoint tenantsync update (v1.2.0) pending; gauge shows n/a until then")
|
||||
c.unavailLogged = true
|
||||
}
|
||||
return // fillBand untouched — an expected data gap never re-arms/transitions
|
||||
}
|
||||
c.logger.Printf("[WARN] PBS-DR box: usage read failed (keeping last snapshot): %v", err)
|
||||
if c.haveSnap {
|
||||
c.snap.State = PBSStateDegraded // serve last-known, visibly stale; NO band transition
|
||||
}
|
||||
return
|
||||
}
|
||||
c.unavailLogged = false // recovered from unavailable
|
||||
|
||||
snap := PBSBoxSnapshot{CapacityBytes: usage.Total, UsedBytes: usage.Used, State: PBSStateOK, FetchedAt: c.now()}
|
||||
if usage.Total > 0 {
|
||||
snap.FillPercent = float64(usage.Used) * 100 / float64(usage.Total)
|
||||
snap.FillBand = bandForPercent(snap.FillPercent, c.fillWarn, c.fillCrit)
|
||||
} else {
|
||||
snap.State = PBSStateDegraded // zero total — guard the division
|
||||
}
|
||||
c.snap = snap
|
||||
c.haveSnap = true
|
||||
|
||||
if snap.State != PBSStateOK {
|
||||
return
|
||||
}
|
||||
c.logger.Printf("[INFO] PBS-DR box refreshed: %.1f%% full (%s of %s)", snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes))
|
||||
|
||||
// FILL band (escalation-only; recovery re-arms because bandOK has rank 0).
|
||||
if bandRank(snap.FillBand) > bandRank(c.fillBand) {
|
||||
c.emitFill(snap, snap.FillBand)
|
||||
}
|
||||
c.fillBand = snap.FillBand
|
||||
}
|
||||
|
||||
// Snapshot returns the cached aggregate + whether one exists (mutex copy-out). The web layer's only
|
||||
// source — it NEVER polls ep0 in the request path.
|
||||
func (c *PBSDRBoxChecker) Snapshot() (PBSBoxSnapshot, bool) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.snap, c.haveSnap
|
||||
}
|
||||
|
||||
// FillState exposes the current band (tests).
|
||||
func (c *PBSDRBoxChecker) FillState() string {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return orUnknown(c.fillBand)
|
||||
}
|
||||
|
||||
func (c *PBSDRBoxChecker) emitFill(snap PBSBoxSnapshot, band string) {
|
||||
var severity, message string
|
||||
switch band {
|
||||
case bandCritical:
|
||||
severity = "critical"
|
||||
message = fmt.Sprintf("PBS DR datastore %.0f%% full (%s of %s) — approaching capacity; the offsite DR tier's retention is at risk; free space or grow the datastore",
|
||||
snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes))
|
||||
case bandWarning:
|
||||
severity = "warning"
|
||||
message = fmt.Sprintf("PBS DR datastore %.0f%% full (%s of %s) — the offsite DR datastore is filling",
|
||||
snap.FillPercent, fmtSize(snap.UsedBytes), fmtSize(snap.CapacityBytes))
|
||||
default:
|
||||
return
|
||||
}
|
||||
details, _ := json.Marshal(map[string]any{
|
||||
"scope": pbsdrBoxScope, "capacity_bytes": snap.CapacityBytes, "used_bytes": snap.UsedBytes,
|
||||
"fill_percent": snap.FillPercent, "warn_percent": c.fillWarn, "crit_percent": c.fillCrit,
|
||||
})
|
||||
c.logger.Printf("[INFO] PBS-DR box fill: %.0f%% (%s)", snap.FillPercent, band)
|
||||
// NO SaveEvent — customer-less scope; straight to the operator dispatcher.
|
||||
if c.onEvent != nil {
|
||||
c.onEvent(pbsdrBoxScope, "pbsdr_box_fill", severity, message, string(details), "hub")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/tenantsync"
|
||||
)
|
||||
|
||||
// fakeUsage is a usageReader for tests: it returns a scripted usage/err and counts calls.
|
||||
type fakeUsage struct {
|
||||
mu sync.Mutex
|
||||
usage tenantsync.BoxUsage
|
||||
err error
|
||||
calls int
|
||||
}
|
||||
|
||||
func (f *fakeUsage) Usage(_ context.Context) (tenantsync.BoxUsage, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.calls++
|
||||
return f.usage, f.err
|
||||
}
|
||||
func (f *fakeUsage) set(u tenantsync.BoxUsage, err error) {
|
||||
f.mu.Lock()
|
||||
f.usage, f.err = u, err
|
||||
f.mu.Unlock()
|
||||
}
|
||||
func (f *fakeUsage) nCalls() int { f.mu.Lock(); defer f.mu.Unlock(); return f.calls }
|
||||
|
||||
func usageBytes(totalGiB, usedGiB int64) tenantsync.BoxUsage {
|
||||
return tenantsync.BoxUsage{Total: totalGiB * gib, Used: usedGiB * gib, Avail: (totalGiB - usedGiB) * gib}
|
||||
}
|
||||
|
||||
// C1 — throttle: ≤1 ep0 usage call per 15-min window across an hour of 60 s sweeps (≈4, not ≈60).
|
||||
func TestPBSDRBox_Throttle(t *testing.T) {
|
||||
f := &fakeUsage{usage: usageBytes(40, 8)}
|
||||
var cur time.Time
|
||||
c := NewPBSDRBoxChecker(f, 80, 90, noEvent, quietLog())
|
||||
c.now = func() time.Time { return cur }
|
||||
base := time.Now().UTC()
|
||||
for i := 0; i < 60; i++ {
|
||||
cur = base.Add(time.Duration(i) * 60 * time.Second)
|
||||
c.Check()
|
||||
}
|
||||
if f.nCalls() < 3 || f.nCalls() > 5 {
|
||||
t.Fatalf("usage op called %d times over an hour of 60 s sweeps, want ≈4 (throttled to 15 min)", f.nCalls())
|
||||
}
|
||||
snap, ok := c.Snapshot()
|
||||
if !ok || snap.State != PBSStateOK || snap.CapacityBytes != 40*gib {
|
||||
t.Fatalf("snapshot wrong: %+v ok=%v", snap, ok)
|
||||
}
|
||||
}
|
||||
|
||||
// C2/C3 — fill bands: escalation-only, in-band no re-emit, recovery re-arm, pbsdr-box scope.
|
||||
// Red-proof (esc): remove the escalation-only guard → the C3 same-band case re-emits → fails.
|
||||
func TestPBSDRBox_FillBands(t *testing.T) {
|
||||
f := &fakeUsage{usage: usageBytes(1000, 750)} // 75%
|
||||
ev := &capturedBox{}
|
||||
var cur time.Time
|
||||
c := NewPBSDRBoxChecker(f, 80, 90, ev.fn, quietLog())
|
||||
c.now = func() time.Time { return cur }
|
||||
base := time.Now().UTC()
|
||||
cur = base
|
||||
step := func(usedGiB int64) {
|
||||
cur = cur.Add(16 * time.Minute)
|
||||
f.set(usageBytes(1000, usedGiB), nil)
|
||||
c.Check()
|
||||
}
|
||||
|
||||
c.Check() // 75% → no emit
|
||||
if len(ev.typ) != 0 {
|
||||
t.Fatalf("75%% must not emit, got %v", ev.typ)
|
||||
}
|
||||
step(820) // 82% → one warning
|
||||
if count(ev.typ, "pbsdr_box_fill") != 1 || ev.sev[0] != "warning" {
|
||||
t.Fatalf("82%% must warn once, got typ=%v sev=%v", ev.typ, ev.sev)
|
||||
}
|
||||
step(850) // 85% same band → no re-emit
|
||||
if count(ev.typ, "pbsdr_box_fill") != 1 {
|
||||
t.Fatalf("same-band must NOT re-emit, got %d", count(ev.typ, "pbsdr_box_fill"))
|
||||
}
|
||||
step(920) // 92% → critical
|
||||
if count(ev.typ, "pbsdr_box_fill") != 2 || ev.sev[len(ev.sev)-1] != "critical" {
|
||||
t.Fatalf("92%% must escalate to critical, got typ=%v sev=%v", ev.typ, ev.sev)
|
||||
}
|
||||
step(700) // 70% → recovery re-arm (silent)
|
||||
if c.FillState() != bandOK {
|
||||
t.Fatalf("recovery must re-arm to ok, got %s", c.FillState())
|
||||
}
|
||||
if count(ev.typ, "pbsdr_box_fill") != 2 {
|
||||
t.Fatalf("recovery must be silent, got %d", count(ev.typ, "pbsdr_box_fill"))
|
||||
}
|
||||
step(920) // re-breach → emits again
|
||||
if count(ev.typ, "pbsdr_box_fill") != 3 {
|
||||
t.Fatalf("re-breach after recovery must emit again, got %d", count(ev.typ, "pbsdr_box_fill"))
|
||||
}
|
||||
for _, cust := range ev.cust {
|
||||
if cust != "pbsdr-box" {
|
||||
t.Fatalf("every emit must carry the pbsdr-box scope, got %q", cust)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// C4 — usage-unsupported (ErrUsageUnsupported) → the distinct "unavailable" state, NOT degraded, NO
|
||||
// alert, NO band transition. Red-proof (unavail-band): let unavailable drive a band → alert fires → fails.
|
||||
func TestPBSDRBox_Unavailable(t *testing.T) {
|
||||
f := &fakeUsage{err: tenantsync.ErrUsageUnsupported}
|
||||
ev := &capturedBox{}
|
||||
c := NewPBSDRBoxChecker(f, 80, 90, ev.fn, quietLog())
|
||||
c.Check()
|
||||
snap, ok := c.Snapshot()
|
||||
if !ok || snap.State != PBSStateUnavailable {
|
||||
t.Fatalf("ErrUsageUnsupported must yield the 'unavailable' state, got %+v", snap)
|
||||
}
|
||||
if len(ev.typ) != 0 {
|
||||
t.Fatalf("unavailable must NOT alert, got %v", ev.typ)
|
||||
}
|
||||
if c.FillState() != "unknown" {
|
||||
t.Fatalf("unavailable must not set a fill band, got %s", c.FillState())
|
||||
}
|
||||
}
|
||||
|
||||
// C5 — exec error/timeout → last snapshot kept, marked degraded, NO band transition (missing ≠ 0%).
|
||||
func TestPBSDRBox_DegradedKeepsLast(t *testing.T) {
|
||||
f := &fakeUsage{usage: usageBytes(1000, 920)} // 92% critical
|
||||
ev := &capturedBox{}
|
||||
var cur time.Time
|
||||
c := NewPBSDRBoxChecker(f, 80, 90, ev.fn, quietLog())
|
||||
c.now = func() time.Time { return cur }
|
||||
cur = time.Now().UTC()
|
||||
|
||||
c.Check() // establish 92% critical + emit
|
||||
if count(ev.typ, "pbsdr_box_fill") != 1 {
|
||||
t.Fatalf("setup: want one critical emit, got %v", ev.typ)
|
||||
}
|
||||
before, _ := c.Snapshot()
|
||||
|
||||
f.set(tenantsync.BoxUsage{}, fmt.Errorf("ssh timeout"))
|
||||
cur = cur.Add(16 * time.Minute)
|
||||
c.Check()
|
||||
|
||||
after, ok := c.Snapshot()
|
||||
if !ok || after.State != PBSStateDegraded {
|
||||
t.Fatalf("a failed poll must mark degraded (keeping last), got %+v", after)
|
||||
}
|
||||
if after.UsedBytes != before.UsedBytes || after.CapacityBytes != before.CapacityBytes {
|
||||
t.Fatalf("a failed poll must keep the last-known values (before %d/%d, after %d/%d)",
|
||||
before.UsedBytes, before.CapacityBytes, after.UsedBytes, after.CapacityBytes)
|
||||
}
|
||||
if c.FillState() != bandCritical {
|
||||
t.Fatalf("a failed poll must NOT transition the band, got %s", c.FillState())
|
||||
}
|
||||
if count(ev.typ, "pbsdr_box_fill") != 1 {
|
||||
t.Fatalf("a failed poll must not emit, got %d", count(ev.typ, "pbsdr_box_fill"))
|
||||
}
|
||||
}
|
||||
@@ -64,6 +64,21 @@ func TestProcessEvent_PoolBoxScopeOperatorOnly(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestProcessEvent_PBSDRBoxScopeOperatorOnly (v0.65.0, R-5, Scenario C3): a customer-less "pbsdr-box"
|
||||
// event reaches ONLY the operator channel (same property as pool-box — no prefs → no customer email).
|
||||
func TestProcessEvent_PBSDRBoxScopeOperatorOnly(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "from@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
var mu sync.Mutex
|
||||
var sent []string
|
||||
d.sendEmailFn = func(to, _, _ string) error { mu.Lock(); defer mu.Unlock(); sent = append(sent, to); return nil }
|
||||
|
||||
d.ProcessEvent("pbsdr-box", "pbsdr_box_fill", "warning", "PBS DR datastore 82% full", `{"scope":"pbsdr-box"}`, "hub")
|
||||
if len(sent) != 1 || sent[0] != "op@felhom.eu" {
|
||||
t.Fatalf("a pbsdr-box event must reach the operator ONLY, sent=%v", sent)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProcessEvent_CriticalRoutes: a critical-severity event reaches the operator channel (Scenario E).
|
||||
func TestProcessEvent_CriticalRoutes(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
|
||||
@@ -28,6 +28,14 @@ import (
|
||||
// treats it as a state mismatch (a descriptor should exist; re-issue is the explicit recovery).
|
||||
var ErrTokenExists = errors.New("tenantsync: token already exists on the endpoint (re-issue is the explicit path)")
|
||||
|
||||
// ErrUsageUnsupported is the typed "endpoint script predates the usage op" outcome (v0.65.0): an
|
||||
// endpoint STILL on tenantsync ≤ v1.1.0 answers `{"op":"usage"}` with its generic `bad_request
|
||||
// "unknown op"`. It is the GRACEFUL-DEGRADATION signal — the PBS checker maps it to an "unavailable"
|
||||
// snapshot state (an expected pre-update condition), NOT an error/degraded state, so no alert fires
|
||||
// and the gauge honestly says "usage not available (endpoint update pending)". The moment ep0 gets
|
||||
// v1.2.0, the next poll succeeds — no hub redeploy.
|
||||
var ErrUsageUnsupported = errors.New("tenantsync: endpoint does not support the usage op (script update pending)")
|
||||
|
||||
// customerIDRe mirrors the script's validation — refuse client-side before a wasted SSH round-trip.
|
||||
var customerIDRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]{0,30}$`)
|
||||
|
||||
@@ -137,6 +145,28 @@ func (c *Client) Fingerprint(ctx context.Context) (string, error) {
|
||||
return resp.Fingerprint, nil
|
||||
}
|
||||
|
||||
// BoxUsage is the endpoint PBS datastore's raw fill (bytes), from the read-only usage op (v1.2.0).
|
||||
type BoxUsage struct {
|
||||
Total int64
|
||||
Used int64
|
||||
Avail int64
|
||||
}
|
||||
|
||||
// Usage returns the felhom-offsite datastore's total/used/avail from the endpoint — a read-only op
|
||||
// (no token, no namespace, no mutation), the structural twin of Fingerprint. Against an endpoint
|
||||
// still on tenantsync ≤ v1.1.0 the op is unknown → ErrUsageUnsupported (graceful degradation).
|
||||
func (c *Client) Usage(ctx context.Context) (BoxUsage, error) {
|
||||
stdout, stderr, runErr := c.exec(ctx, []byte(`{"op":"usage"}`))
|
||||
resp, err := parseResponse(stdout, stderr, runErr)
|
||||
if err != nil {
|
||||
return BoxUsage{}, err
|
||||
}
|
||||
if resp.Total <= 0 {
|
||||
return BoxUsage{}, fmt.Errorf("tenantsync: usage op returned a non-positive total")
|
||||
}
|
||||
return BoxUsage{Total: resp.Total, Used: resp.Used, Avail: resp.Avail}, nil
|
||||
}
|
||||
|
||||
func (c *Client) tenancyOp(ctx context.Context, op, customerID string) (*Result, error) {
|
||||
if !customerIDRe.MatchString(customerID) {
|
||||
return nil, fmt.Errorf("tenantsync: invalid customer_id %q", customerID)
|
||||
@@ -165,6 +195,10 @@ type response struct {
|
||||
Code string `json:"code"`
|
||||
Error string `json:"error"`
|
||||
Deleted bool `json:"deleted"` // deprovision op: whether the namespace existed (was destroyed)
|
||||
// usage op (v1.2.0): the felhom-offsite datastore's raw fill, in BYTES.
|
||||
Total int64 `json:"total"`
|
||||
Used int64 `json:"used"`
|
||||
Avail int64 `json:"avail"`
|
||||
Result
|
||||
}
|
||||
|
||||
@@ -178,6 +212,11 @@ func parseResponse(stdout, stderr []byte, runErr error) (*response, error) {
|
||||
if resp.Code == "token_exists" {
|
||||
return nil, ErrTokenExists
|
||||
}
|
||||
// An endpoint ≤ v1.1.0 has no usage op → its generic `bad_request "unknown op"`. Map it to the
|
||||
// typed graceful-degradation signal (the exact err_json string is the script's fixed contract).
|
||||
if resp.Code == "bad_request" && strings.Contains(resp.Error, "unknown op") {
|
||||
return nil, ErrUsageUnsupported
|
||||
}
|
||||
return nil, fmt.Errorf("tenantsync: endpoint refused: %s (code %s)", resp.Error, resp.Code)
|
||||
}
|
||||
if runErr != nil {
|
||||
|
||||
@@ -208,6 +208,38 @@ func TestFingerprint_Op(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestUsage_Op (v0.65.0): the read-only usage op parses total/used/avail (bytes).
|
||||
func TestUsage_Op(t *testing.T) {
|
||||
clientPEM, clientSigner := testKeys(t)
|
||||
_, hostSigner := testKeys(t)
|
||||
srv := startTestServer(t, hostSigner, clientSigner,
|
||||
`{"status":"ok","total":39990112256,"used":7628091392,"avail":30686175232}`, "", 0)
|
||||
c := newTestClient(t, srv.addr, hostKeyLine(t, hostSigner), clientPEM)
|
||||
|
||||
u, err := c.Usage(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("Usage: %v", err)
|
||||
}
|
||||
if u.Total != 39990112256 || u.Used != 7628091392 || u.Avail != 30686175232 {
|
||||
t.Errorf("usage = %+v, want total/used/avail 39990112256/7628091392/30686175232", u)
|
||||
}
|
||||
}
|
||||
|
||||
// TestUsage_UnknownOpTypedUnsupported (v0.65.0, graceful degradation): an ep0 still on tenantsync
|
||||
// ≤ v1.1.0 answers `{"op":"usage"}` with its generic `bad_request "unknown op"` (exit 1). That must
|
||||
// map to the typed ErrUsageUnsupported so the PBS checker records "unavailable", not an error.
|
||||
func TestUsage_UnknownOpTypedUnsupported(t *testing.T) {
|
||||
clientPEM, clientSigner := testKeys(t)
|
||||
_, hostSigner := testKeys(t)
|
||||
srv := startTestServer(t, hostSigner, clientSigner,
|
||||
`{"status":"error","code":"bad_request","error":"unknown op"}`, "", 1)
|
||||
c := newTestClient(t, srv.addr, hostKeyLine(t, hostSigner), clientPEM)
|
||||
|
||||
if _, err := c.Usage(context.Background()); !errors.Is(err, ErrUsageUnsupported) {
|
||||
t.Fatalf("an ep0 without the usage op must map to ErrUsageUnsupported, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestErrors_NeverEmbedStdout is the secret-hygiene contract: stdout may carry the token secret,
|
||||
// so NO error path may quote it. A malformed-but-secret-bearing stdout must yield an error that
|
||||
// does not contain the marker bytes. (wgsync quotes stdout in its malformed error — this package
|
||||
|
||||
@@ -139,7 +139,8 @@ func (s *Server) handleOffsite(w http.ResponseWriter, r *http.Request) {
|
||||
})
|
||||
}
|
||||
|
||||
boxView, custRows := s.offsiteBoxData() // R-5 pool-box aggregate panel
|
||||
boxView, custRows := s.offsiteBoxData() // R-5 restic pool-box aggregate panel (Restic tab)
|
||||
pbsView := s.pbsdrBoxData() // R-5 v0.65.0 PBS-DR datastore panel (PBS DR tab)
|
||||
|
||||
data := map[string]interface{}{
|
||||
"Endpoints": cards,
|
||||
@@ -147,6 +148,7 @@ func (s *Server) handleOffsite(w http.ResponseWriter, r *http.Request) {
|
||||
"Peers": rows,
|
||||
"OffsiteBox": boxView,
|
||||
"OffsiteCusts": custRows,
|
||||
"PBSBox": pbsView,
|
||||
"CSRFToken": s.getCSRFToken(r),
|
||||
"Flash": r.URL.Query().Get("flash"),
|
||||
}
|
||||
|
||||
@@ -49,8 +49,10 @@ func TestOffsiteBoxPanel_WithData(t *testing.T) {
|
||||
t.Fatalf("panel missing %q:\n%s", want, body)
|
||||
}
|
||||
}
|
||||
if strings.Contains(body, "not configured") {
|
||||
t.Fatal("a configured panel must not show the not-configured message")
|
||||
// The RESTIC panel is configured — its specific not-configured message must be absent. (The PBS DR
|
||||
// panel legitimately shows its own "not configured" here since s.pbsdrBox is unset.)
|
||||
if strings.Contains(body, "Offsite pool metrics not configured") {
|
||||
t.Fatal("a configured restic panel must not show the not-configured message")
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
package web
|
||||
|
||||
// PBS DR datastore fill surfaces (v0.65.0, R-5): the Offsite "PBS DR" tab panel + the Dashboard PBS
|
||||
// gauge. The web layer reads only the checker's cached snapshot (s.pbsdrBox) — it never polls ep0. The
|
||||
// snapshot carries its own state (ok / unavailable / degraded) so the UI renders each honestly.
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
|
||||
)
|
||||
|
||||
// pbsdrBoxView is the PBS-DR panel model.
|
||||
type pbsdrBoxView struct {
|
||||
Configured bool // s.pbsdrBox wired (tenantsync client present)
|
||||
Pending bool // configured but the first poll hasn't landed yet
|
||||
State string // ok | unavailable | degraded
|
||||
Unavailable bool // endpoint script predates the usage op (≤ v1.1.0) — expected pre-update
|
||||
Degraded bool // last poll failed — values shown stale
|
||||
HasFill bool // numbers available (ok or degraded-with-last-known)
|
||||
CapacityStr string
|
||||
UsedStr string
|
||||
FillPercent float64
|
||||
FillBand string
|
||||
FetchedAt time.Time
|
||||
}
|
||||
|
||||
// pbsdrTile is the compact Dashboard PBS gauge: fill %, band-colored; "n/a" when unavailable.
|
||||
type pbsdrTile struct {
|
||||
Unavailable bool
|
||||
FillPercent float64
|
||||
Band string
|
||||
Degraded bool
|
||||
}
|
||||
|
||||
// pbsdrBoxData builds the PBS-DR panel view. Nil provider → Configured:false ("not configured"). Never polls.
|
||||
func (s *Server) pbsdrBoxData() pbsdrBoxView {
|
||||
if s.pbsdrBox == nil {
|
||||
return pbsdrBoxView{Configured: false}
|
||||
}
|
||||
view := pbsdrBoxView{Configured: true}
|
||||
snap, ok := s.pbsdrBox()
|
||||
if !ok {
|
||||
view.Pending = true // configured, first poll pending
|
||||
return view
|
||||
}
|
||||
view.State = snap.State
|
||||
view.FetchedAt = snap.FetchedAt
|
||||
switch snap.State {
|
||||
case monitor.PBSStateUnavailable:
|
||||
view.Unavailable = true
|
||||
case monitor.PBSStateDegraded:
|
||||
view.Degraded = true
|
||||
if snap.CapacityBytes > 0 { // last-known values persist across a degraded poll
|
||||
view.HasFill = true
|
||||
view.CapacityStr = fmtBytesGB(snap.CapacityBytes)
|
||||
view.UsedStr = fmtBytesGB(snap.UsedBytes)
|
||||
view.FillPercent = snap.FillPercent
|
||||
view.FillBand = snap.FillBand
|
||||
}
|
||||
default: // ok
|
||||
view.HasFill = true
|
||||
view.CapacityStr = fmtBytesGB(snap.CapacityBytes)
|
||||
view.UsedStr = fmtBytesGB(snap.UsedBytes)
|
||||
view.FillPercent = snap.FillPercent
|
||||
view.FillBand = snap.FillBand
|
||||
}
|
||||
return view
|
||||
}
|
||||
|
||||
// pbsdrBoxTile builds the Dashboard PBS gauge, or nil when there is nothing to show (no provider, or the
|
||||
// first poll hasn't landed). Unavailable → a neutral "n/a" gauge (never a fake 0%).
|
||||
func (s *Server) pbsdrBoxTile() *pbsdrTile {
|
||||
if s.pbsdrBox == nil {
|
||||
return nil
|
||||
}
|
||||
snap, ok := s.pbsdrBox()
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
if snap.State == monitor.PBSStateUnavailable {
|
||||
return &pbsdrTile{Unavailable: true}
|
||||
}
|
||||
return &pbsdrTile{
|
||||
FillPercent: snap.FillPercent,
|
||||
Band: snap.FillBand,
|
||||
Degraded: snap.State == monitor.PBSStateDegraded,
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
|
||||
)
|
||||
|
||||
// PBS DR panel — configured + ok: renders the datastore, capacity/used, fill bar.
|
||||
func TestPBSDRPanel_OK(t *testing.T) {
|
||||
s, _ := newRenderServer(t)
|
||||
s.SetPBSDRBox(func() (monitor.PBSBoxSnapshot, bool) {
|
||||
return monitor.PBSBoxSnapshot{
|
||||
CapacityBytes: 40 * gib, UsedBytes: 8 * gib, FillPercent: 20, FillBand: "ok",
|
||||
State: monitor.PBSStateOK, FetchedAt: time.Now().UTC(),
|
||||
}, true
|
||||
})
|
||||
body := renderOffsite(t, s)
|
||||
for _, want := range []string{"PBS DR datastore", "felhom-offsite", "40.0 GB", "8.0 GB", "20% full"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Fatalf("PBS DR panel missing %q", want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// PBS DR panel — unavailable (endpoint script ≤ v1.1.0): the honest pending-update message, NOT a fake 0%.
|
||||
func TestPBSDRPanel_Unavailable(t *testing.T) {
|
||||
s, _ := newRenderServer(t)
|
||||
s.SetPBSDRBox(func() (monitor.PBSBoxSnapshot, bool) {
|
||||
return monitor.PBSBoxSnapshot{State: monitor.PBSStateUnavailable, FetchedAt: time.Now().UTC()}, true
|
||||
})
|
||||
body := renderOffsite(t, s)
|
||||
if !strings.Contains(body, "usage not available") || !strings.Contains(body, "v1.2.0") {
|
||||
t.Fatalf("unavailable PBS panel must show the pending-update message:\n%s", pbsSection(body))
|
||||
}
|
||||
// Never a fake fill for the unavailable box.
|
||||
if strings.Contains(pbsSection(body), "% full") {
|
||||
t.Fatal("unavailable PBS panel must not render a fill percentage")
|
||||
}
|
||||
}
|
||||
|
||||
// PBS DR panel — not configured (no tenantsync client).
|
||||
func TestPBSDRPanel_NotConfigured(t *testing.T) {
|
||||
s, _ := newRenderServer(t)
|
||||
// s.pbsdrBox left nil
|
||||
body := renderOffsite(t, s)
|
||||
if !strings.Contains(body, "PBS DR metrics not configured") {
|
||||
t.Fatalf("unconfigured PBS panel must say 'not configured'")
|
||||
}
|
||||
}
|
||||
|
||||
// pbsSection returns the slice of the body from the PBS DR datastore heading onward (for scoped asserts).
|
||||
func pbsSection(body string) string {
|
||||
if i := strings.Index(body, "PBS DR datastore"); i >= 0 {
|
||||
return body[i:]
|
||||
}
|
||||
return body
|
||||
}
|
||||
@@ -92,10 +92,11 @@ func TestTemplates_DashboardCriticalBadge(t *testing.T) {
|
||||
EventErrors int
|
||||
EventWarnings int
|
||||
}
|
||||
// v0.64.0: dashboard.html now takes {Customers, OffsiteTile}; OffsiteTile nil → no tile rendered.
|
||||
// v0.65.0: dashboard.html takes {Customers, OffsiteTile, PBSTile}; nil tiles → no gauges rendered.
|
||||
data := struct {
|
||||
Customers []dashboardCustomer
|
||||
OffsiteTile any
|
||||
PBSTile any
|
||||
}{Customers: []dashboardCustomer{{
|
||||
CustomerSummary: store.CustomerSummary{CustomerID: "c1", CustomerName: "Acme", ReceivedAt: time.Now()},
|
||||
OverallStatus: "ok", BackupAge: "–",
|
||||
|
||||
@@ -67,7 +67,8 @@ type Server struct {
|
||||
assetsMgr *assets.Manager
|
||||
gitea *gitea.Client // optional; enables the Day-0 artifact version dropdowns
|
||||
offsite *offsite.Provisioner // optional; enables Hetzner offsite provisioning (SLICE 1)
|
||||
offsiteBox func() (monitor.BoxSnapshot, bool) // optional (v0.64.0, R-5); the pool-box aggregate snapshot accessor
|
||||
offsiteBox func() (monitor.BoxSnapshot, bool) // optional (v0.64.0, R-5); the restic pool-box aggregate snapshot accessor
|
||||
pbsdrBox func() (monitor.PBSBoxSnapshot, bool) // optional (v0.65.0, R-5); the PBS-DR datastore fill snapshot accessor
|
||||
tenantsync tenancyProvisioner // optional; enables PBS DR tier provisioning (web/pbsdr.go)
|
||||
claimEngine *claim.Engine // optional; enables the customer-claim resend button (v0.50.0)
|
||||
// intentHub (v0.58.0, Direction-2 immediate-sync) is Bumped by every operator-intent handler
|
||||
@@ -188,6 +189,11 @@ func (s *Server) SetOffsiteProvisioner(p *offsite.Provisioner) { s.offsite = p }
|
||||
// honest "not configured". The web layer NEVER fetches from Hetzner — it only reads this cache.
|
||||
func (s *Server) SetOffsiteBox(fn func() (monitor.BoxSnapshot, bool)) { s.offsiteBox = fn }
|
||||
|
||||
// SetPBSDRBox wires the PBS-DR datastore fill snapshot accessor (v0.65.0, R-5): read on the Offsite
|
||||
// "PBS DR" tab + the Dashboard PBS gauge. nil (no tenantsync client) → "not configured". The snapshot
|
||||
// carries its own state (ok/unavailable/degraded); the web layer never polls ep0.
|
||||
func (s *Server) SetPBSDRBox(fn func() (monitor.PBSBoxSnapshot, bool)) { s.pbsdrBox = fn }
|
||||
|
||||
// SetClaimEngine wires the customer-claim code engine for the Setup-tab resend button (v0.50.0).
|
||||
func (s *Server) SetClaimEngine(e *claim.Engine) { s.claimEngine = e }
|
||||
|
||||
@@ -751,8 +757,9 @@ func (s *Server) handleDashboard(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
payload := struct {
|
||||
Customers []dashboardCustomer
|
||||
OffsiteTile *offsiteTile // R-5: nil → no tile rendered
|
||||
}{Customers: data, OffsiteTile: s.offsiteBoxTile()}
|
||||
OffsiteTile *offsiteTile // R-5: restic pool box; nil → no gauge
|
||||
PBSTile *pbsdrTile // R-5 v0.65.0: PBS DR datastore; nil → no gauge
|
||||
}{Customers: data, OffsiteTile: s.offsiteBoxTile(), PBSTile: s.pbsdrBoxTile()}
|
||||
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
if err := s.templates.ExecuteTemplate(w, "dashboard.html", payload); err != nil {
|
||||
|
||||
@@ -22,12 +22,23 @@
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
{{with .OffsiteTile}}
|
||||
<a href="/offsite" class="offsite-tile offsite-tile-{{.Band}}">
|
||||
<span class="offsite-tile-label">Offsite pool</span>
|
||||
<span class="offsite-tile-val">{{formatFloat .FillPercent}}% · {{.RatioStr}}</span>
|
||||
{{if .Degraded}}<span class="offsite-tile-stale">stale</span>{{end}}
|
||||
</a>
|
||||
{{if or .OffsiteTile .PBSTile}}
|
||||
<div class="offsite-gauges">
|
||||
{{with .OffsiteTile}}
|
||||
<a href="/offsite#tab=restic" class="offsite-tile offsite-tile-{{.Band}}">
|
||||
<span class="offsite-tile-label">Restic</span>
|
||||
<span class="offsite-tile-val">{{formatFloat .FillPercent}}% · {{.RatioStr}}</span>
|
||||
{{if .Degraded}}<span class="offsite-tile-stale">stale</span>{{end}}
|
||||
</a>
|
||||
{{end}}
|
||||
{{with .PBSTile}}
|
||||
<a href="/offsite#tab=pbsdr" class="offsite-tile offsite-tile-{{.Band}}">
|
||||
<span class="offsite-tile-label">PBS DR</span>
|
||||
{{if .Unavailable}}<span class="offsite-tile-val" style="color:var(--text-3);">n/a</span>{{else}}<span class="offsite-tile-val">{{formatFloat .FillPercent}}%</span>{{end}}
|
||||
{{if .Degraded}}<span class="offsite-tile-stale">stale</span>{{end}}
|
||||
</a>
|
||||
{{end}}
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
{{if not .Customers}}
|
||||
|
||||
@@ -21,11 +21,14 @@
|
||||
</nav>
|
||||
</header>
|
||||
|
||||
<h2 style="margin-bottom: 0.5rem;">Offsite connectivity</h2>
|
||||
<p class="text-muted" style="margin: 0 0 1rem; font-size: 0.85em;">
|
||||
Peer allocation and endpoint sync currently use the lowest endpoint id (ep0).
|
||||
Per-endpoint allocation is a future work item.
|
||||
</p>
|
||||
<h2 style="margin-bottom: 0.75rem;">Offsite</h2>
|
||||
|
||||
<!-- Two offsite tiers, two distinct stores (v0.65.0, R-5): RESTIC = subaccounts on the shared
|
||||
Hetzner Storage Box; PBS DR = the felhom-offsite datastore on the ep0 endpoint VM. -->
|
||||
<nav class="tab-nav" id="tab-nav">
|
||||
<a href="#tab=restic" data-tab="restic" class="active">Restic (shared box)</a>
|
||||
<a href="#tab=pbsdr" data-tab="pbsdr">PBS DR</a>
|
||||
</nav>
|
||||
|
||||
{{if eq .Flash "endpoint_saved"}}
|
||||
<div class="flash flash-success">Endpoint saved.</div>
|
||||
@@ -34,6 +37,9 @@
|
||||
<div class="flash flash-success">Endpoint deleted.</div>
|
||||
{{end}}
|
||||
|
||||
<!-- ═══ Restic tab ═══ -->
|
||||
<div class="tab-panel tab-panel-active" data-tab="restic">
|
||||
|
||||
<!-- R-5 (v0.64.0): shared pool-box aggregate — total fill, oversubscription, per-customer usage -->
|
||||
<section class="card" style="margin-bottom: 1.5rem;">
|
||||
<h3 style="margin: 0 0 0.75rem;">Offsite pool box</h3>
|
||||
@@ -73,6 +79,40 @@
|
||||
{{end}}
|
||||
</section>
|
||||
|
||||
</div><!-- ═══ /Restic tab ═══ -->
|
||||
|
||||
<!-- ═══ PBS DR tab ═══ -->
|
||||
<div class="tab-panel" data-tab="pbsdr">
|
||||
|
||||
<!-- R-5 (v0.65.0): PBS DR datastore fill (felhom-offsite on ep0, read via the tenantsync usage op) -->
|
||||
<section class="card" style="margin-bottom: 1.5rem;">
|
||||
<h3 style="margin: 0 0 0.75rem;">PBS DR datastore</h3>
|
||||
{{if not .PBSBox.Configured}}
|
||||
<p class="text-muted" style="font-size: 0.9em;">PBS DR metrics not configured (no offsite endpoint / tenantsync key on this hub).</p>
|
||||
{{else if .PBSBox.Pending}}
|
||||
<p class="text-muted" style="font-size: 0.9em;">PBS DR metrics loading — the first poll has not landed yet.</p>
|
||||
{{else if .PBSBox.Unavailable}}
|
||||
<p class="text-muted" style="font-size: 0.9em;">PBS DR usage not available — the endpoint script update (felhom-tenantsync v1.2.0) is pending. The gauge lights up on the next poll once ep0 is updated (no hub redeploy).</p>
|
||||
{{else if .PBSBox.HasFill}}
|
||||
<table class="detail-table">
|
||||
<tr><th style="width: 12rem;">Datastore</th><td><code>felhom-offsite</code> (ep0)</td></tr>
|
||||
<tr><th>Capacity</th><td>{{.PBSBox.CapacityStr}}</td></tr>
|
||||
<tr><th>Used</th><td>{{.PBSBox.UsedStr}} · {{formatFloat .PBSBox.FillPercent}}% full</td></tr>
|
||||
</table>
|
||||
<div class="bar" style="margin: 0.4rem 0 0.9rem;"><div class="bar-fill bar-{{.PBSBox.FillBand}}" style="width: {{formatFloat .PBSBox.FillPercent}}%;"></div></div>
|
||||
<table class="detail-table">
|
||||
<tr><th style="width: 12rem;">Polled</th><td>{{timeAgo .PBSBox.FetchedAt}}{{if .PBSBox.Degraded}} <span class="status-badge status-badge-warn">STALE — last poll failed</span>{{end}}</td></tr>
|
||||
</table>
|
||||
{{else}}
|
||||
<p class="text-muted" style="font-size: 0.9em;">PBS DR usage temporarily unavailable (endpoint poll failed).</p>
|
||||
{{end}}
|
||||
</section>
|
||||
|
||||
<p class="text-muted" style="margin: 0 0 1rem; font-size: 0.85em;">
|
||||
The endpoint below IS the PBS DR host. Peer allocation and endpoint sync currently use the
|
||||
lowest endpoint id (ep0); per-endpoint allocation is a future work item.
|
||||
</p>
|
||||
|
||||
{{if .HasEndpoints}}
|
||||
{{range .Endpoints}}
|
||||
<section class="card" style="margin-bottom: 1.5rem;"
|
||||
@@ -185,11 +225,39 @@
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
</div><!-- ═══ /PBS DR tab ═══ -->
|
||||
|
||||
<footer style="margin-top: 2rem; color: var(--text-muted); font-size: 0.8rem; text-align: center;">
|
||||
Felhom Hub <span style="font-family: var(--font-mono)">{{hubVersion}}</span>
|
||||
</footer>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
// Hash tabs (v0.65.0), mirrored from customer_unified.html. Without JS this never runs — the body
|
||||
// never gets .js-tabs, so both panels stay visible and the page reads top-to-bottom (data is
|
||||
// server-rendered; JS only picks which panel is shown). Default tab = restic.
|
||||
(function() {
|
||||
var panels = document.querySelectorAll('.tab-panel');
|
||||
var links = document.querySelectorAll('#tab-nav a');
|
||||
if (!panels.length || !links.length) return;
|
||||
document.body.classList.add('js-tabs');
|
||||
var known = {};
|
||||
panels.forEach(function(p) { known[p.getAttribute('data-tab')] = true; });
|
||||
function currentTab() {
|
||||
var m = (location.hash || '').match(/^#tab=([a-z-]+)$/);
|
||||
var t = m ? m[1] : '';
|
||||
return known[t] ? t : 'restic';
|
||||
}
|
||||
function activate() {
|
||||
var tab = currentTab();
|
||||
panels.forEach(function(p) { p.classList.toggle('tab-panel-active', p.getAttribute('data-tab') === tab); });
|
||||
links.forEach(function(a) { a.classList.toggle('active', a.getAttribute('data-tab') === tab); });
|
||||
}
|
||||
window.addEventListener('hashchange', activate);
|
||||
activate();
|
||||
})();
|
||||
</script>
|
||||
|
||||
<script>
|
||||
// Endpoint management JS (v0.47.0). The server enforces every guard — this layer only
|
||||
// fills the edit form from a card's data attributes and adds the pubkey-change confirm.
|
||||
|
||||
@@ -303,6 +303,10 @@ header h1 {
|
||||
.bar-fill.bar-warning { background: var(--warn); }
|
||||
.bar-fill.bar-critical { background: var(--crit); }
|
||||
|
||||
/* Offsite dashboard gauges (v0.65.0, R-5) — two side-by-side tiles (Restic, PBS DR). */
|
||||
.offsite-gauges { display: flex; gap: 0.75rem; flex-wrap: wrap; margin-bottom: 1rem; }
|
||||
.offsite-gauges .offsite-tile { margin-bottom: 0; }
|
||||
|
||||
/* Offsite pool dashboard tile (v0.64.0, R-5) — compact, band-colored, links to /offsite. */
|
||||
.offsite-tile {
|
||||
display: inline-flex; align-items: baseline; gap: 0.5rem;
|
||||
|
||||
@@ -1,7 +1,12 @@
|
||||
#!/usr/bin/env bash
|
||||
# felhom-tenantsync v1.1.0 — the offsite endpoint's per-customer PBS tenancy surface (PBS DR tier
|
||||
# felhom-tenantsync v1.2.0 — the offsite endpoint's per-customer PBS tenancy surface (PBS DR tier
|
||||
# SLICE 1; spike SPIKE-pbs-tier-provisioning-2026-07-10 §3).
|
||||
#
|
||||
# v1.2.0 (2026-07-17, hub R-5 PBS visibility): + read-only `usage` op — the felhom-offsite datastore's
|
||||
# total/used/avail (bytes, via `df` on the datastore path). No customer_id, no admin token, NO
|
||||
# mutation. Structural twin of `fingerprint`. Backward-compatible: an older hub never sends it, and a
|
||||
# hub that does against an OLD script gets the existing `bad_request "unknown op"` (graceful).
|
||||
#
|
||||
# Runs as the SSH forced command for the hub's SECOND `felhom-peersync` key (via sudo — its own
|
||||
# single sudoers line; the peersync script/key are untouched: one script, one job). JSON on stdin,
|
||||
# JSON on stdout. Ops:
|
||||
@@ -23,6 +28,8 @@
|
||||
# This is the DELIBERATE, gated data-destruction the slice-1 note reserved — the operator RESET
|
||||
# confirm (typed customer-id + separate escrow-custody ack) is the human decision it demanded.
|
||||
# {"op":"fingerprint"} → {"status":"ok","fingerprint":"<PBS cert sha256>"}
|
||||
# {"op":"usage"} → {"status":"ok","total":<bytes>,"used":<bytes>,"avail":<bytes>}
|
||||
# READ-ONLY datastore fill (df on the felhom-offsite path). No customer_id, no admin token, no mutation.
|
||||
#
|
||||
# Secret hygiene (load-bearing):
|
||||
# - The token secret exists ONLY in memory and in the final stdout JSON — never a file, never
|
||||
@@ -68,6 +75,20 @@ if [ "$OP" = "fingerprint" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# usage (v1.2.0) — READ-ONLY box-level datastore fill. No customer_id, no admin token, NO mutation
|
||||
# (a pure `df` on the datastore path). The hub's PBS-DR checker polls this on its 15-min throttle.
|
||||
# Short-circuits BEFORE the admin-token generation below, exactly like fingerprint.
|
||||
if [ "$OP" = "usage" ]; then
|
||||
DS_PATH=$(proxmox-backup-manager datastore list --output-format json \
|
||||
| jq -r --arg ds "$DS" '.[] | select(.name == $ds) | .path')
|
||||
[ -n "$DS_PATH" ] || err_json internal "datastore $DS not found"
|
||||
# df -B1: 1-byte blocks. --output pins the columns; tail -1 skips the header row.
|
||||
read -r TOTAL USED AVAIL < <(df -B1 --output=size,used,avail "$DS_PATH" | tail -1)
|
||||
[ -n "$TOTAL" ] && [ -n "$USED" ] && [ -n "$AVAIL" ] || err_json internal "could not read datastore usage"
|
||||
printf '{"status":"ok","total":%s,"used":%s,"avail":%s}\n' "$TOTAL" "$USED" "$AVAIL"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
case "$OP" in provision|reissue|deprovision) ;; *) err_json bad_request "unknown op" ;; esac
|
||||
|
||||
# customer_id → the namespace AND the token name. Conservative charset (PBS ns + token grammar,
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env bash
|
||||
# tenantsync-usage-harness.sh (v0.65.0, R-5) — Group-A test for the felhom-tenantsync `usage` op.
|
||||
# Stubs proxmox-backup-manager + df on PATH (real jq), runs the op, and asserts:
|
||||
# 1. usage → {"status":"ok","total","used","avail"} + exit 0 + ZERO mutation calls.
|
||||
# 2. provision (regression) → reaches the mutation path (admin-token generate) — the usage op did
|
||||
# not disturb the case dispatch — and NEVER emits the usage JSON.
|
||||
# Non-hollow: the stub LOGS every proxmox-backup-manager invocation; the zero-mutation assertion reads it.
|
||||
set -uo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
SCRIPT="$HERE/felhom-tenantsync.sh"
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "$WORK"' EXIT
|
||||
STUB="$WORK/bin"; mkdir -p "$STUB"
|
||||
export MUTLOG="$WORK/mutations.log"; : > "$MUTLOG"
|
||||
export CALLLOG="$WORK/calls.log"; : > "$CALLLOG"
|
||||
DS_PATH="$WORK/ds"; mkdir -p "$DS_PATH"
|
||||
|
||||
# Stub proxmox-backup-manager: read-only ops answered; MUTATING subcommands logged to MUTLOG.
|
||||
cat > "$STUB/proxmox-backup-manager" <<'PBM'
|
||||
#!/usr/bin/env bash
|
||||
echo "pbm $*" >> "$CALLLOG"
|
||||
case "$1 $2" in
|
||||
"cert info") echo "Fingerprint (sha256): aa:bb:cc:dd" ;;
|
||||
"datastore list") printf '[{"name":"scratch","path":"/srv/pbs-scratch"},{"name":"felhom-offsite","path":"%s"}]\n' "$DS_PATH" ;;
|
||||
"user generate-token") echo "MUTATION: $*" >> "$MUTLOG"; echo '{"value":"FAKE-ADMIN-TOKEN"}' ;;
|
||||
"user delete-token") echo "MUTATION: $*" >> "$MUTLOG" ;;
|
||||
"acl update") echo "MUTATION: $*" >> "$MUTLOG" ;;
|
||||
"namespace "*) echo "MUTATION: $*" >> "$MUTLOG" ;;
|
||||
*) echo "MUTATION: $*" >> "$MUTLOG" ;; # any other subcommand is mutating here
|
||||
esac
|
||||
PBM
|
||||
chmod +x "$STUB/proxmox-backup-manager"
|
||||
|
||||
# Stub proxmox-backup-client (used only by the mutating ops; log if invoked).
|
||||
cat > "$STUB/proxmox-backup-client" <<'PBC'
|
||||
#!/usr/bin/env bash
|
||||
echo "MUTATION: pbc $*" >> "$MUTLOG"
|
||||
PBC
|
||||
chmod +x "$STUB/proxmox-backup-client"
|
||||
|
||||
# Stub df: fixed bytes (total 41943040, used 8388608, avail 33554432 ⇒ 20% full).
|
||||
cat > "$STUB/df" <<'DF'
|
||||
#!/usr/bin/env bash
|
||||
echo "1B-blocks Used Avail"
|
||||
echo "41943040 8388608 33554432"
|
||||
DF
|
||||
chmod +x "$STUB/df"
|
||||
|
||||
export DS_PATH CALLLOG MUTLOG
|
||||
RC=0
|
||||
fail() { echo "FAIL: $*"; RC=1; }
|
||||
|
||||
# --- Test 1: usage op ---
|
||||
OUT=$(echo '{"op":"usage"}' | PATH="$STUB:$PATH" bash "$SCRIPT" 2>/dev/null); EC=$?
|
||||
[ "$EC" -eq 0 ] || fail "usage exit $EC (want 0)"
|
||||
EXP='{"status":"ok","total":41943040,"used":8388608,"avail":33554432}'
|
||||
[ "$OUT" = "$EXP" ] || fail "usage stdout = $OUT (want $EXP)"
|
||||
if [ -s "$MUTLOG" ]; then fail "usage ran mutations: $(cat "$MUTLOG")"; else echo "PASS: usage emits totals, exit 0, ZERO mutations"; fi
|
||||
|
||||
# --- Test 2: provision regression (dispatch intact; usage op did not break the case) ---
|
||||
: > "$MUTLOG"
|
||||
POUT=$(echo '{"op":"provision","customer_id":"harness"}' | PATH="$STUB:$PATH" bash "$SCRIPT" 2>/dev/null || true)
|
||||
grep -q "generate-token" "$MUTLOG" || fail "provision did not reach the mutation path (case dispatch broken by the usage op?)"
|
||||
case "$POUT" in *'"total"'*) fail "provision leaked the usage shape: $POUT" ;; esac
|
||||
[ "$RC" -eq 0 ] && echo "PASS: provision still dispatches to the mutation path (regression)"
|
||||
|
||||
exit $RC
|
||||
Reference in New Issue
Block a user