felhom.eu/REPORT.md

felhom.eu — task reports

Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md; the scripts history lives in scripts/CHANGELOG.md.

Hub v0.65.0 — PBS DR storage visibility (ep0 usage op) + Offsite tab split (Restic / PBS DR) + dual dashboard gauges (R-5) — 2026-07-17

1. Baselines used

• felhom.eu @ 3588a31 (v0.64.0 REPORT), clean on main, local == origin re-confirmed.
• Hub v0.64.0 → v0.65.0. Manifest felhom-hub:0.64.0 → :0.65.0.
• scripts/felhom-tenantsync.sh v1.1.0 → v1.2.0 (adds the read-only usage op).
• READ-ONLY against ep0 and Hetzner throughout (no mutation, no admin token; the usage op is a pure df).

2. Phase-0 probe (the gate — PASSED)

On ep0 (felhom-hetzner, root SSH, PBS 4.2.3): proxmox-backup-manager datastore usage does not exist in PBS 4.2, so df on the datastore path is the authoritative read-only source. datastore list --output-format json → felhom-offsite path /srv/pbs-felhom; df -B1 --output=size,used,avail /srv/pbs-felhom:

  1B-blocks       Used       Avail
39990112256 7628091392 30686175232

All bytes (total 37.2 GiB / used 7.1 GiB → ~19%), read-only, in the existing sudo context, no admin token. Gate PASSED → the op was built (Option A). The v1.2.0 op emits exactly these as {"status":"ok","total","used","avail"} — verified against the real ep0 before writing the harness.

3. Files created / modified

Code + script + runbook (commit 7f11cfb):

• scripts/felhom-tenantsync.sh → v1.2.0 (read-only usage op) + scripts/tenantsync-usage-harness.sh (new Group-A test).
• hub/internal/tenantsync/client.go — Usage() + BoxUsage + ErrUsageUnsupported + response fields + client_test.go.
• hub/internal/monitor/pbsdr_box.go (new) + _test.go — PBSDRBoxChecker (usageReader seam, throttle, 3 states, bands).
• hub/internal/notify/dispatcher_test.go — the pbsdr-box operator-only test.
• hub/cmd/hub/main.go — Alerting.PBSDRBoxFill* config; checker construction with the tenantsync client; 60 s sweep; SetPBSDRBox.
• hub/internal/web/{server.go,offsite.go,pbsdr_box.go(new),pbsdr_box_render_test.go(new),offsite_box_render_test.go,render_test.go} — PBS view/tile, tab wiring, two-gauge payload, render tests.
• hub/internal/web/templates/{offsite.html,dashboard.html,style.css} — Restic/PBS-DR tabs, two gauges, gauge container CSS.
• documentation/runbooks/offsite-endpoint.md §10 — v1.2.0 install steps (no sudoers/authorized_keys change).
• hub/CHANGELOG.md.

Docs (commit <DOCS_HASH>): REPORT (this), CONTEXT, capability-map, ROADMAP. Manifest (commit ab91e49): manifests/hub.yaml → :0.65.0.

4. Test results + red-proof outcomes

10 Go tests + a bash harness, all green:

Test	Group	Asserts
TestUsage_Op	B	usage op parses total/used/avail (bytes)
TestUsage_UnknownOpTypedUnsupported	B	ep0 ≤ v1.1.0 unknown op → typed ErrUsageUnsupported
TestPBSDRBox_Throttle	C1	≈4 usage calls over an hour of 60 s sweeps (not ≈60)
TestPBSDRBox_FillBands	C2/C3	75→no; 82→warn; 85 in-band→no re-emit; 92→crit; 70→re-arm; re-breach→emit; scope "pbsdr-box"
TestPBSDRBox_Unavailable	C4	ErrUsageUnsupported → "unavailable" state, NO alert, no band
TestPBSDRBox_DegradedKeepsLast	C5	exec error → degraded, keeps last values, no band transition, no emit
TestProcessEvent_PBSDRBoxScopeOperatorOnly	C3	a pbsdr-box event → operator channel ONLY
TestPBSDRPanel_OK / _Unavailable / _NotConfigured	D	panel renders ok / the pending-update message / not-configured
tenantsync-usage-harness.sh	A	usage JSON + exit 0 + zero mutation (stub-logged) + provision regression

Red-proofs (run-fail-restore), all confirmed red then restored:

#	Removed / broken	Test	Result
A	the usage op emits a mutation (acl update)	harness	FAILED ("usage ran mutations: … acl update …") → restored
C-esc	the escalation-only guard	PBSDRBox_FillBands	FAILED ("same-band must NOT re-emit, got 2") → restored
C-unavail	unavailable drives a fill band	PBSDRBox_Unavailable	FAILED ("must yield 'unavailable', got State:ok … critical") → restored

Full suite: go build ./... && go vet ./... && go test ./... (17 packages) + bash -n + hub confirm gate — all green. Test count: +10 Go (+2 tenantsync, +4 monitor, +1 notify, +3 web) + 1 bash harness.

5. Deploy verification

Built felhom-hub:0.65.0 on 180 (source 7f11cfb). ep0 updated to tenantsync v1.2.0 THIS session (root SSH, .bak-1.1.0 kept; tr -d '\r' → bash -n OK → install -m0755; NO sudoers/authorized_keys change). On-box verify: echo '{"op":"usage"}' | /usr/local/bin/felhom-tenantsync → {"status":"ok","total":39990112256,"used":7628349440,"avail":30685917184}; fingerprint regression OK. Manifest bump (ab91e49) → ArgoCD hard-refresh → sync → rollout succeeded; image :0.65.0. Startup log:

[INFO] felhom-hub 0.65.0 starting
[INFO] PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync; …)
[INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, refresh 15m0s

6. Live-rendered values (the PROVEN-LIVE evidence)

Because ep0 was updated to v1.2.0, the PBS gauge shows real numbers. The hub UI is operator-password-gated (CC can't screenshot), so the in-cluster checker's first live poll is the proof — the exact numbers the PBS panel + gauge render:

[INFO] PBS-DR box refreshed: 19.1% full (7.1 GB of 37.2 GB)

Live-confirmed end-to-end (hub checker → tenantsync usage op → ep0 df → snapshot): 7.1 GB of 37.2 GB, 19.1% — matching the Phase-0 probe. The restic gauge (v0.64.0) continues to render its own live numbers. Both tabs (Restic / PBS DR) and both dashboard gauges are live; the panel-render is UNIT-verified (TestPBSDRPanel_*), the data proven live.

7. NOT yet live-validated

• The UI render (tabs + gauges) could not be screenshotted (hub UI password-gated). It is unit-verified (render tests) and the data behind both gauges is proven live (§6).
• The PBS fill alert leg (pbsdr_box_fill operator email) is NOT fired live: the datastore is nominal (19.1%, far from 80/90%). Unit + red-proof verified; a live-fire would email Viktor (offered on demand).
• The graceful "unavailable" state was exercised in unit tests, not live — ep0 was updated straight to v1.2.0, so the hub never saw the old-script path in production (the hub-independence property is verified by construction + TestUsage_UnknownOpTypedUnsupported + TestPBSDRBox_Unavailable).

8. Observations

• Separate PBS threshold pair: shipped as Alerting.PBSDRBoxFill{Warn,Crit}Percent defaulting to the same 80/90 as restic — independently tunable later without touching the restic policy. Whether Viktor wants a different PBS pair is his call (one-liner). No oversubscription concept for PBS (namespaces, not quotas) — fill only, as ruled.
• Dashboard decimal: the gauges use %.0f (e.g. "19%"), so a near-empty box shows "0%" not "0.2%" — matches the restic tile's existing rounding. The panels show the finer detail.
• The usage op is df-based because PBS 4.2 has no native datastore usage command; if a future PBS adds one with dedup-aware logical sizing, the op can switch source with no hub/client change (same JSON shape).