# felhom.eu — task reports

> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).

## Hub v0.65.0 — PBS DR storage visibility (ep0 `usage` op) + Offsite tab split (Restic / PBS DR) + dual dashboard gauges (R-5) — 2026-07-17

### 1. Baselines used

- **felhom.eu** @ `3588a31` (v0.64.0 REPORT), clean on `main`, local == origin re-confirmed.
- **Hub** v0.64.0 → **v0.65.0**. Manifest `felhom-hub:0.64.0` → `:0.65.0`.
- **`scripts/felhom-tenantsync.sh`** v1.1.0 → **v1.2.0** (adds the read-only `usage` op).
- READ-ONLY against ep0 and Hetzner throughout (no mutation, no admin token; the usage op is a pure `df`).

### 2. Phase-0 probe (the gate — PASSED)

On ep0 (`felhom-hetzner`, root SSH, PBS **4.2.3**): `proxmox-backup-manager datastore usage` does not exist
in PBS 4.2, so `df` on the datastore path is the authoritative read-only source. `datastore list
--output-format json` → `felhom-offsite` path `/srv/pbs-felhom`; `df -B1 --output=size,used,avail /srv/pbs-felhom`:

```
  1B-blocks       Used       Avail
39990112256 7628091392 30686175232
```

All **bytes** (total 37.2 GiB / used 7.1 GiB → ~19%), read-only, in the existing sudo context, no admin
token. Gate PASSED → the op was built (Option A). The v1.2.0 op emits exactly these as
`{"status":"ok","total","used","avail"}` — verified against the real ep0 before writing the harness.

### 3. Files created / modified

**Code + script + runbook (commit `7f11cfb`):**
- `scripts/felhom-tenantsync.sh` → v1.2.0 (read-only `usage` op) `+ scripts/tenantsync-usage-harness.sh` (new Group-A test).
- `hub/internal/tenantsync/client.go` — `Usage()` + `BoxUsage` + `ErrUsageUnsupported` + response fields `+ client_test.go`.
- `hub/internal/monitor/pbsdr_box.go` (new) `+ _test.go` — `PBSDRBoxChecker` (usageReader seam, throttle, 3 states, bands).
- `hub/internal/notify/dispatcher_test.go` — the pbsdr-box operator-only test.
- `hub/cmd/hub/main.go` — `Alerting.PBSDRBoxFill*` config; checker construction with the tenantsync client; 60 s sweep; `SetPBSDRBox`.
- `hub/internal/web/{server.go,offsite.go,pbsdr_box.go(new),pbsdr_box_render_test.go(new),offsite_box_render_test.go,render_test.go}` — PBS view/tile, tab wiring, two-gauge payload, render tests.
- `hub/internal/web/templates/{offsite.html,dashboard.html,style.css}` — Restic/PBS-DR tabs, two gauges, gauge container CSS.
- `documentation/runbooks/offsite-endpoint.md` §10 — v1.2.0 install steps (no sudoers/authorized_keys change).
- `hub/CHANGELOG.md`.

**Docs (commit `<DOCS_HASH>`):** REPORT (this), CONTEXT, capability-map, ROADMAP.
**Manifest (commit `ab91e49`):** `manifests/hub.yaml` → `:0.65.0`.

### 4. Test results + red-proof outcomes

10 Go tests + a bash harness, all green:

| Test | Group | Asserts |
|------|-------|---------|
| `TestUsage_Op` | B | usage op parses total/used/avail (bytes) |
| `TestUsage_UnknownOpTypedUnsupported` | B | ep0 ≤ v1.1.0 `unknown op` → typed `ErrUsageUnsupported` |
| `TestPBSDRBox_Throttle` | C1 | ≈4 usage calls over an hour of 60 s sweeps (not ≈60) |
| `TestPBSDRBox_FillBands` | C2/C3 | 75→no; 82→warn; 85 in-band→no re-emit; 92→crit; 70→re-arm; re-breach→emit; scope "pbsdr-box" |
| `TestPBSDRBox_Unavailable` | C4 | ErrUsageUnsupported → "unavailable" state, NO alert, no band |
| `TestPBSDRBox_DegradedKeepsLast` | C5 | exec error → degraded, keeps last values, no band transition, no emit |
| `TestProcessEvent_PBSDRBoxScopeOperatorOnly` | C3 | a pbsdr-box event → operator channel ONLY |
| `TestPBSDRPanel_OK` / `_Unavailable` / `_NotConfigured` | D | panel renders ok / the pending-update message / not-configured |
| `tenantsync-usage-harness.sh` | A | usage JSON + exit 0 + **zero mutation** (stub-logged) + provision regression |

**Red-proofs (run-fail-restore), all confirmed red then restored:**

| # | Removed / broken | Test | Result |
|---|------------------|------|--------|
| A | the usage op emits a mutation (`acl update`) | harness | FAILED ("usage ran mutations: … acl update …") → restored |
| C-esc | the escalation-only guard | `PBSDRBox_FillBands` | FAILED ("same-band must NOT re-emit, got 2") → restored |
| C-unavail | unavailable drives a fill band | `PBSDRBox_Unavailable` | FAILED ("must yield 'unavailable', got State:ok … critical") → restored |

Full suite: `go build ./... && go vet ./... && go test ./...` (17 packages) + `bash -n` + hub confirm gate —
all green. Test count: +10 Go (+2 tenantsync, +4 monitor, +1 notify, +3 web) + 1 bash harness.

### 5. Deploy verification

Built `felhom-hub:0.65.0` on 180 (source `7f11cfb`). **ep0 updated to tenantsync v1.2.0 THIS session** (root
SSH, `.bak-1.1.0` kept; `tr -d '\r'` → `bash -n` OK → `install -m0755`; NO sudoers/authorized_keys change).
On-box verify: `echo '{"op":"usage"}' | /usr/local/bin/felhom-tenantsync` →
`{"status":"ok","total":39990112256,"used":7628349440,"avail":30685917184}`; `fingerprint` regression OK.
Manifest bump (`ab91e49`) → ArgoCD hard-refresh → sync → rollout succeeded; image `:0.65.0`. Startup log:

```
[INFO] felhom-hub 0.65.0 starting
[INFO] PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync; …)
[INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, refresh 15m0s
```

### 6. Live-rendered values (the PROVEN-LIVE evidence)

Because ep0 was updated to v1.2.0, the PBS gauge shows **real** numbers. The hub UI is operator-password-gated
(CC can't screenshot), so the in-cluster checker's first live poll is the proof — the exact numbers the PBS
panel + gauge render:

```
[INFO] PBS-DR box refreshed: 19.1% full (7.1 GB of 37.2 GB)
```

Live-confirmed end-to-end (hub checker → tenantsync `usage` op → ep0 `df` → snapshot): **7.1 GB of 37.2 GB,
19.1%** — matching the Phase-0 probe. The restic gauge (v0.64.0) continues to render its own live numbers.
Both tabs (Restic / PBS DR) and both dashboard gauges are live; the panel-render is UNIT-verified
(`TestPBSDRPanel_*`), the data proven live.

### 7. NOT yet live-validated

- **The UI render** (tabs + gauges) could not be screenshotted (hub UI password-gated). It is unit-verified
  (render tests) and the data behind both gauges is proven live (§6).
- **The PBS fill alert leg** (`pbsdr_box_fill` operator email) is NOT fired live: the datastore is nominal
  (19.1%, far from 80/90%). Unit + red-proof verified; a live-fire would email Viktor (offered on demand).
- **The graceful "unavailable" state** was exercised in unit tests, not live — ep0 was updated straight to
  v1.2.0, so the hub never saw the old-script path in production (the hub-independence property is verified
  by construction + `TestUsage_UnknownOpTypedUnsupported` + `TestPBSDRBox_Unavailable`).

### 8. Observations

- **Separate PBS threshold pair:** shipped as `Alerting.PBSDRBoxFill{Warn,Crit}Percent` defaulting to the
  same 80/90 as restic — independently tunable later without touching the restic policy. Whether Viktor
  wants a different PBS pair is his call (one-liner). No oversubscription concept for PBS (namespaces, not
  quotas) — fill only, as ruled.
- **Dashboard decimal:** the gauges use `%.0f` (e.g. "19%"), so a near-empty box shows "0%" not "0.2%" —
  matches the restic tile's existing rounding. The panels show the finer detail.
- The `usage` op is `df`-based because PBS 4.2 has no native `datastore usage` command; if a future PBS adds
  one with dedup-aware logical sizing, the op can switch source with no hub/client change (same JSON shape).