hub v0.65.0 — PBS DR storage visibility (ep0 usage op) + Offsite tab split + dual dashboard gauges (R-5)

Makes PBS DR storage visible like the restic pool box (v0.64.0), differentiated. Scoping
correction: restic = subaccounts on the shared Hetzner Storage Box (Hetzner API); PBS DR =
the felhom-offsite PBS datastore on the ep0 endpoint VM (NO Hetzner API). Option A
(Viktor-ruled): a read-only `usage` op on the felhom-tenantsync ep0 forced command (twin of
fingerprint), polled by a new hub checker on the 15-min throttle. READ-ONLY throughout.

Phase-0 (gate PASSED): on ep0 (PBS 4.2.3), df -B1 --output=size,used,avail <datastore path>
yields bytes (39990112256/7627939840/... ~19%), read-only, existing sudo context, no admin token.

- scripts/felhom-tenantsync.sh -> v1.2.0: read-only `usage` short-circuit (df on the datastore
  path), no customer_id, no admin token, NO mutation. + a bash harness proving zero mutation.
- tenantsync.Client.Usage() + BoxUsage; unknown-op -> typed ErrUsageUnsupported (graceful).
- monitor.PBSDRBoxChecker: OffsiteBoxChecker clone over a usageReader seam; 15-min throttle,
  cached PBSBoxSnapshot, escalation-only pbsdr_box_fill on the "pbsdr-box" scope (operator only,
  no SaveEvent), recovery re-arm. Fill only. THREE states: ok / unavailable (ep0 <=v1.1.0,
  neutral no-alert) / degraded (exec failed, keep last).
- config: Alerting.PBSDRBoxFill{Warn,Crit}Percent (80/90); built with the tenantsync client,
  60s sweep, SetPBSDRBox. Hub deploy INDEPENDENT of the ep0 update (graceful degradation).
- web: /offsite splits into Restic + PBS DR hash tabs (endpoint cards under PBS DR); PBS panel;
  the single dashboard tile becomes two gauges (RESTIC pct.ratio, PBS DR pct / n/a).
- runbook offsite-endpoint.md 10: v1.2.0 update steps (no sudoers/authorized_keys change).

Tests: 10 Go + the harness; 3 red-proofs (usage mutation, escalation-only, unavailable-drives-band)
confirmed red then restored. go build/vet/test + bash -n + hub confirm gate all pass.
This commit is contained in:
2026-07-17 21:13:30 +02:00
parent 3588a31b78
commit 7f11cfb36c
19 changed files with 856 additions and 22 deletions
+17 -6
View File
@@ -22,12 +22,23 @@
</nav>
</header>
{{with .OffsiteTile}}
<a href="/offsite" class="offsite-tile offsite-tile-{{.Band}}">
<span class="offsite-tile-label">Offsite pool</span>
<span class="offsite-tile-val">{{formatFloat .FillPercent}}% &middot; {{.RatioStr}}</span>
{{if .Degraded}}<span class="offsite-tile-stale">stale</span>{{end}}
</a>
{{if or .OffsiteTile .PBSTile}}
<div class="offsite-gauges">
{{with .OffsiteTile}}
<a href="/offsite#tab=restic" class="offsite-tile offsite-tile-{{.Band}}">
<span class="offsite-tile-label">Restic</span>
<span class="offsite-tile-val">{{formatFloat .FillPercent}}% &middot; {{.RatioStr}}</span>
{{if .Degraded}}<span class="offsite-tile-stale">stale</span>{{end}}
</a>
{{end}}
{{with .PBSTile}}
<a href="/offsite#tab=pbsdr" class="offsite-tile offsite-tile-{{.Band}}">
<span class="offsite-tile-label">PBS DR</span>
{{if .Unavailable}}<span class="offsite-tile-val" style="color:var(--text-3);">n/a</span>{{else}}<span class="offsite-tile-val">{{formatFloat .FillPercent}}%</span>{{end}}
{{if .Degraded}}<span class="offsite-tile-stale">stale</span>{{end}}
</a>
{{end}}
</div>
{{end}}
{{if not .Customers}}
+73 -5
View File
@@ -21,11 +21,14 @@
</nav>
</header>
<h2 style="margin-bottom: 0.5rem;">Offsite connectivity</h2>
<p class="text-muted" style="margin: 0 0 1rem; font-size: 0.85em;">
Peer allocation and endpoint sync currently use the lowest endpoint id (ep0).
Per-endpoint allocation is a future work item.
</p>
<h2 style="margin-bottom: 0.75rem;">Offsite</h2>
<!-- Two offsite tiers, two distinct stores (v0.65.0, R-5): RESTIC = subaccounts on the shared
Hetzner Storage Box; PBS DR = the felhom-offsite datastore on the ep0 endpoint VM. -->
<nav class="tab-nav" id="tab-nav">
<a href="#tab=restic" data-tab="restic" class="active">Restic (shared box)</a>
<a href="#tab=pbsdr" data-tab="pbsdr">PBS DR</a>
</nav>
{{if eq .Flash "endpoint_saved"}}
<div class="flash flash-success">Endpoint saved.</div>
@@ -34,6 +37,9 @@
<div class="flash flash-success">Endpoint deleted.</div>
{{end}}
<!-- ═══ Restic tab ═══ -->
<div class="tab-panel tab-panel-active" data-tab="restic">
<!-- R-5 (v0.64.0): shared pool-box aggregate — total fill, oversubscription, per-customer usage -->
<section class="card" style="margin-bottom: 1.5rem;">
<h3 style="margin: 0 0 0.75rem;">Offsite pool box</h3>
@@ -73,6 +79,40 @@
{{end}}
</section>
</div><!-- ═══ /Restic tab ═══ -->
<!-- ═══ PBS DR tab ═══ -->
<div class="tab-panel" data-tab="pbsdr">
<!-- R-5 (v0.65.0): PBS DR datastore fill (felhom-offsite on ep0, read via the tenantsync usage op) -->
<section class="card" style="margin-bottom: 1.5rem;">
<h3 style="margin: 0 0 0.75rem;">PBS DR datastore</h3>
{{if not .PBSBox.Configured}}
<p class="text-muted" style="font-size: 0.9em;">PBS DR metrics not configured (no offsite endpoint / tenantsync key on this hub).</p>
{{else if .PBSBox.Pending}}
<p class="text-muted" style="font-size: 0.9em;">PBS DR metrics loading — the first poll has not landed yet.</p>
{{else if .PBSBox.Unavailable}}
<p class="text-muted" style="font-size: 0.9em;">PBS DR usage not available — the endpoint script update (felhom-tenantsync v1.2.0) is pending. The gauge lights up on the next poll once ep0 is updated (no hub redeploy).</p>
{{else if .PBSBox.HasFill}}
<table class="detail-table">
<tr><th style="width: 12rem;">Datastore</th><td><code>felhom-offsite</code> (ep0)</td></tr>
<tr><th>Capacity</th><td>{{.PBSBox.CapacityStr}}</td></tr>
<tr><th>Used</th><td>{{.PBSBox.UsedStr}} &middot; {{formatFloat .PBSBox.FillPercent}}% full</td></tr>
</table>
<div class="bar" style="margin: 0.4rem 0 0.9rem;"><div class="bar-fill bar-{{.PBSBox.FillBand}}" style="width: {{formatFloat .PBSBox.FillPercent}}%;"></div></div>
<table class="detail-table">
<tr><th style="width: 12rem;">Polled</th><td>{{timeAgo .PBSBox.FetchedAt}}{{if .PBSBox.Degraded}} <span class="status-badge status-badge-warn">STALE — last poll failed</span>{{end}}</td></tr>
</table>
{{else}}
<p class="text-muted" style="font-size: 0.9em;">PBS DR usage temporarily unavailable (endpoint poll failed).</p>
{{end}}
</section>
<p class="text-muted" style="margin: 0 0 1rem; font-size: 0.85em;">
The endpoint below IS the PBS DR host. Peer allocation and endpoint sync currently use the
lowest endpoint id (ep0); per-endpoint allocation is a future work item.
</p>
{{if .HasEndpoints}}
{{range .Endpoints}}
<section class="card" style="margin-bottom: 1.5rem;"
@@ -185,11 +225,39 @@
</div>
{{end}}
</div><!-- ═══ /PBS DR tab ═══ -->
<footer style="margin-top: 2rem; color: var(--text-muted); font-size: 0.8rem; text-align: center;">
Felhom Hub <span style="font-family: var(--font-mono)">{{hubVersion}}</span>
</footer>
</div>
<script>
// Hash tabs (v0.65.0), mirrored from customer_unified.html. Without JS this never runs — the body
// never gets .js-tabs, so both panels stay visible and the page reads top-to-bottom (data is
// server-rendered; JS only picks which panel is shown). Default tab = restic.
(function() {
var panels = document.querySelectorAll('.tab-panel');
var links = document.querySelectorAll('#tab-nav a');
if (!panels.length || !links.length) return;
document.body.classList.add('js-tabs');
var known = {};
panels.forEach(function(p) { known[p.getAttribute('data-tab')] = true; });
function currentTab() {
var m = (location.hash || '').match(/^#tab=([a-z-]+)$/);
var t = m ? m[1] : '';
return known[t] ? t : 'restic';
}
function activate() {
var tab = currentTab();
panels.forEach(function(p) { p.classList.toggle('tab-panel-active', p.getAttribute('data-tab') === tab); });
links.forEach(function(a) { a.classList.toggle('active', a.getAttribute('data-tab') === tab); });
}
window.addEventListener('hashchange', activate);
activate();
})();
</script>
<script>
// Endpoint management JS (v0.47.0). The server enforces every guard — this layer only
// fills the edit form from a card's data attributes and adds the pubkey-change confirm.
+4
View File
@@ -303,6 +303,10 @@ header h1 {
.bar-fill.bar-warning { background: var(--warn); }
.bar-fill.bar-critical { background: var(--crit); }
/* Offsite dashboard gauges (v0.65.0, R-5) — two side-by-side tiles (Restic, PBS DR). */
.offsite-gauges { display: flex; gap: 0.75rem; flex-wrap: wrap; margin-bottom: 1rem; }
.offsite-gauges .offsite-tile { margin-bottom: 0; }
/* Offsite pool dashboard tile (v0.64.0, R-5) — compact, band-colored, links to /offsite. */
.offsite-tile {
display: inline-flex; align-items: baseline; gap: 0.5rem;