From a9cd308398d9bfbe7c792e5bea59649ed511e03b Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 17 Jul 2026 21:20:34 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20hub=20v0.65.0=20PBS=20DR=20visibility?= =?UTF-8?q?=20(R-5)=20=E2=80=94=20REPORT=20+=20CONTEXT=20+=20capability-ma?= =?UTF-8?q?p=20(PBS=20DR=20row,=20data=20pipeline=20PROVEN-LIVE)=20+=20ROA?= =?UTF-8?q?DMAP=20(R-5=20restic+PBS)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CONTEXT.md | 18 +++ REPORT.md | 149 +++++++++--------- .../architecture/00-capability-map.md | 1 + documentation/backlog/ROADMAP.md | 2 +- 4 files changed, 94 insertions(+), 76 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index da5b92c..fb12de8 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,24 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-17 — PBS DR STORAGE VISIBILITY + Offsite tab split + dual gauges shipped (hub v0.65.0 + + tenantsync v1.2.0, R-5), LIVE.** Scoping correction (Viktor): "restic box" and "PBS box" are NOT two + Hetzner Storage Boxes — restic = subaccounts on the shared Hetzner box (Hetzner API, v0.64.0); **PBS DR = + the `felhom-offsite` PBS datastore on ep0** (NO Hetzner API; the hub had no read of its fill). **Option A + (ruled):** a read-only `usage` op on the `felhom-tenantsync` ep0 forced command (twin of `fingerprint`, + `df` on the datastore path — no customer_id, no admin token, NO mutation), polled by `monitor.PBSDRBoxChecker` + (OffsiteBoxChecker clone over a `usageReader` seam; 15-min throttle; three states ok/unavailable/degraded; + escalation-only `pbsdr_box_fill` on the `"pbsdr-box"` operator scope). `/offsite` now has **Restic** + + **PBS DR** tabs (endpoint cards under PBS DR — the endpoint IS the PBS host); the dashboard has **two + gauges** (restic %·ratio, PBS %). **Graceful degradation: the hub deploy is INDEPENDENT of the ep0 update** + — an ep0 ≤ v1.1.0 → `ErrUsageUnsupported` → the gauge shows "n/a" until updated (no hub redeploy). + **PROVEN-LIVE (data pipeline, both gauges):** ep0 updated to v1.2.0 this session; the hub polled it — + `PBS-DR box refreshed: 19.1% full (7.1 GB of 37.2 GB)`. **DECISION:** PBS thresholds are separate keys + (`PBSDRBoxFill*`) defaulting to restic's 80/90 (independently tunable; Viktor's call for a different pair); + no oversubscription for PBS (namespaces, not quotas). **OPEN:** UI render unit-verified only (hub UI + password-gated); the PBS fill alert email is unit + red-proof verified, NOT fired live (datastore nominal + at 19%). READ-ONLY against ep0 + Hetzner. See REPORT.md. + - **2026-07-17 — OFFSITE POOL-BOX AGGREGATE shipped (hub v0.64.0, R-5).** The operator now sees the shared pool box's real state on the hub: total fill vs capacity, Σ(shared soft quotas)/capacity (the oversubscription ratio), per-customer usage/quota bars, and a box-level operator alert (fill 80/90%, diff --git a/REPORT.md b/REPORT.md index bc6f626..e937c45 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,120 +2,119 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## Hub v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5) — 2026-07-17 +## Hub v0.65.0 — PBS DR storage visibility (ep0 `usage` op) + Offsite tab split (Restic / PBS DR) + dual dashboard gauges (R-5) — 2026-07-17 -### 1. Baseline used +### 1. Baselines used -- **felhom.eu** @ `85a1419` ("docs: R-24 … + R-28 … SHIPPED"), clean on `main`, local == origin re-confirmed. -- **Hub** v0.63.0 → **v0.64.0**. Manifest `manifests/hub.yaml` `felhom-hub:0.63.0` → `:0.64.0`. -- READ-ONLY against Hetzner throughout (GET only; no write method called; no new secret; token never logged). +- **felhom.eu** @ `3588a31` (v0.64.0 REPORT), clean on `main`, local == origin re-confirmed. +- **Hub** v0.64.0 → **v0.65.0**. Manifest `felhom-hub:0.64.0` → `:0.65.0`. +- **`scripts/felhom-tenantsync.sh`** v1.1.0 → **v1.2.0** (adds the read-only `usage` op). +- READ-ONLY against ep0 and Hetzner throughout (no mutation, no admin token; the usage op is a pure `df`). ### 2. Phase-0 probe (the gate — PASSED) -One authenticated GET of the live pool box, from 180 via the k8s Secret token (never written to any file): -`GET https://api.hetzner.com/v1/storage_boxes/611714` → HTTP 200. The (redacted, non-secret) shape: +On ep0 (`felhom-hetzner`, root SSH, PBS **4.2.3**): `proxmox-backup-manager datastore usage` does not exist +in PBS 4.2, so `df` on the datastore path is the authoritative read-only source. `datastore list +--output-format json` → `felhom-offsite` path `/srv/pbs-felhom`; `df -B1 --output=size,used,avail /srv/pbs-felhom`: -```json -{ "storage_box": { - "id": 611714, "name": "storage-box-pool-1", "status": "active", - "storage_box_type": { "id": 1333, "name": "bx11", "description": "BX11", "size": 1099511627776 }, - "stats": { "size": 2746220544, "size_data": 2746220544, "size_snapshots": 0 } } } +``` + 1B-blocks Used Avail +39990112256 7628091392 30686175232 ``` -Pinned: **capacity = `storage_box_type.size`** (1099511627776 B = 1 TiB for bx11); **usage = `stats`** -(`size`/`size_data`/`size_snapshots`), all **bytes**; our token reads it (in scope). The type extension -mirrors this exactly (golden decode test embeds the capture). Gate PASSED → the poller was built. +All **bytes** (total 37.2 GiB / used 7.1 GiB → ~19%), read-only, in the existing sudo context, no admin +token. Gate PASSED → the op was built (Option A). The v1.2.0 op emits exactly these as +`{"status":"ok","total","used","avail"}` — verified against the real ep0 before writing the harness. ### 3. Files created / modified -**Code (commit `4bb2df0`):** -- `hub/internal/hetznerapi/{hetznerapi.go,fake.go}` — additive `StorageBoxType` + `StorageBoxStats`; fake `GetBoxCalls`/`FailGetBox`. `+ storagebox_stats_test.go` (golden decode). -- `hub/internal/monitor/offsite_box.go` (new) `+ _test.go` — `OffsiteBoxChecker` (throttled poll, bands, snapshot). -- `hub/internal/offsite/offsite.go` — new `ReadDescriptor` (authoritative quota source; `ClearProvisionedDescriptor` refactored onto it). -- `hub/internal/notify/dispatcher_test.go` — the pool-box operator-only test (C7). -- `hub/cmd/hub/main.go` — 3 `Alerting.OffsiteBox*` config keys; checker construction in the `HETZNER_TOKEN` branch; 60 s sweep registration; `SetOffsiteBox`. -- `hub/internal/web/{server.go,offsite.go,offsite_box.go(new),offsite_box_render_test.go(new),render_test.go}` — panel + tile builder, setter, render tests. -- `hub/internal/web/templates/{offsite.html,dashboard.html,style.css}` — panel, tile, bar/tile band classes. +**Code + script + runbook (commit `7f11cfb`):** +- `scripts/felhom-tenantsync.sh` → v1.2.0 (read-only `usage` op) `+ scripts/tenantsync-usage-harness.sh` (new Group-A test). +- `hub/internal/tenantsync/client.go` — `Usage()` + `BoxUsage` + `ErrUsageUnsupported` + response fields `+ client_test.go`. +- `hub/internal/monitor/pbsdr_box.go` (new) `+ _test.go` — `PBSDRBoxChecker` (usageReader seam, throttle, 3 states, bands). +- `hub/internal/notify/dispatcher_test.go` — the pbsdr-box operator-only test. +- `hub/cmd/hub/main.go` — `Alerting.PBSDRBoxFill*` config; checker construction with the tenantsync client; 60 s sweep; `SetPBSDRBox`. +- `hub/internal/web/{server.go,offsite.go,pbsdr_box.go(new),pbsdr_box_render_test.go(new),offsite_box_render_test.go,render_test.go}` — PBS view/tile, tab wiring, two-gauge payload, render tests. +- `hub/internal/web/templates/{offsite.html,dashboard.html,style.css}` — Restic/PBS-DR tabs, two gauges, gauge container CSS. +- `documentation/runbooks/offsite-endpoint.md` §10 — v1.2.0 install steps (no sudoers/authorized_keys change). - `hub/CHANGELOG.md`. -**Refresh-log observability (commit `a4268c3`):** `offsite_box.go` logs the pool trend on each refresh (earns the live-render proof below without the password-gated UI). - -**Docs (commit `f2e51e6`):** REPORT (this), CONTEXT, capability-map, ROADMAP. -**Manifest (commit `66da0c4`):** `manifests/hub.yaml` → `:0.64.0`. +**Docs (commit ``):** REPORT (this), CONTEXT, capability-map, ROADMAP. +**Manifest (commit `ab91e49`):** `manifests/hub.yaml` → `:0.65.0`. ### 4. Test results + red-proof outcomes -10 new tests, all green: +10 Go tests + a bash harness, all green: | Test | Group | Asserts | |------|-------|---------| -| `TestOffsiteBox_FetchThrottle` | A | ≈4 API reads over an hour of 60 s sweeps (not ≈60); snapshot populated | -| `TestOffsiteBox_OversubMath` | B | Σ = shared+enabled only (A 500 + B 700); dedicated/disabled excluded; ratio = Σ/capacity = 1.17× | -| `TestOffsiteBox_FillBands` | C1–C5 | 75%→no emit; 82%→1 warn; 85% in-band→no re-emit; 92%→critical; 70%→re-arm; re-breach→emit; every emit scope "pool-box" | -| `TestOffsiteBox_OversubIndependent` | C6 | 2.3× oversub fires alone with fill nominal (50%) — distinct event, no fill emit | -| `TestOffsiteBox_FailedFetchHonesty` | D | failed fetch keeps last values, marks Degraded, no band transition, no emit | -| `TestOffsiteBox_ZeroCapacityGuard` | — | zero capacity → degraded snapshot, no division, no alert | -| `TestStorageBox_DecodesProbeShape` | 1.1 | golden probe JSON decodes to the extended type; data+snapshots == total | -| `TestProcessEvent_PoolBoxScopeOperatorOnly` | C7 | a "pool-box" event reaches the operator ONLY (no customer email) | -| `TestOffsiteBoxPanel_NotConfigured` | E | nil provider → "not configured", no fake zeros | -| `TestOffsiteBoxPanel_WithData` | E | panel renders capacity/ratio + per-customer rows + "no usage reported yet" | +| `TestUsage_Op` | B | usage op parses total/used/avail (bytes) | +| `TestUsage_UnknownOpTypedUnsupported` | B | ep0 ≤ v1.1.0 `unknown op` → typed `ErrUsageUnsupported` | +| `TestPBSDRBox_Throttle` | C1 | ≈4 usage calls over an hour of 60 s sweeps (not ≈60) | +| `TestPBSDRBox_FillBands` | C2/C3 | 75→no; 82→warn; 85 in-band→no re-emit; 92→crit; 70→re-arm; re-breach→emit; scope "pbsdr-box" | +| `TestPBSDRBox_Unavailable` | C4 | ErrUsageUnsupported → "unavailable" state, NO alert, no band | +| `TestPBSDRBox_DegradedKeepsLast` | C5 | exec error → degraded, keeps last values, no band transition, no emit | +| `TestProcessEvent_PBSDRBoxScopeOperatorOnly` | C3 | a pbsdr-box event → operator channel ONLY | +| `TestPBSDRPanel_OK` / `_Unavailable` / `_NotConfigured` | D | panel renders ok / the pending-update message / not-configured | +| `tenantsync-usage-harness.sh` | A | usage JSON + exit 0 + **zero mutation** (stub-logged) + provision regression | **Red-proofs (run-fail-restore), all confirmed red then restored:** -| # | Removed | Test | Result | -|---|---------|------|--------| -| i | the fetch-throttle guard | `FetchThrottle` | FAILED ("called 60 times… want ≈4") → restored | -| ii | the shared+enabled Σ filter | `OversubMath` | FAILED ("Σquota = 11499… want 1200") → restored | -| iii | the escalation-only guard | `FillBands` | FAILED ("C3 same-band must NOT re-emit, got 2") → restored | -| iv | keep-last-on-failure (zero the snapshot) | `FailedFetchHonesty` | FAILED ("must not alter… before …, after 0/0") → restored | +| # | Removed / broken | Test | Result | +|---|------------------|------|--------| +| A | the usage op emits a mutation (`acl update`) | harness | FAILED ("usage ran mutations: … acl update …") → restored | +| C-esc | the escalation-only guard | `PBSDRBox_FillBands` | FAILED ("same-band must NOT re-emit, got 2") → restored | +| C-unavail | unavailable drives a fill band | `PBSDRBox_Unavailable` | FAILED ("must yield 'unavailable', got State:ok … critical") → restored | -Full suite: `go build ./... && go vet ./... && go test ./...` — all 17 packages green; hub confirm gate OK. -Test count: +10 (monitor +6, notify +1, hetznerapi +1, web +2). +Full suite: `go build ./... && go vet ./... && go test ./...` (17 packages) + `bash -n` + hub confirm gate — +all green. Test count: +10 Go (+2 tenantsync, +4 monitor, +1 notify, +3 web) + 1 bash harness. ### 5. Deploy verification -Built + pushed `felhom-hub:0.64.0` on 180 (build source at `4bb2df0`/`a4268c3`). Manifest bump (`66da0c4`) -→ ArgoCD hard-refresh (rev `66da0c44`, OutOfSync) → deliberate sync → rollout succeeded; image -`felhom-hub:0.64.0`. Startup log: +Built `felhom-hub:0.65.0` on 180 (source `7f11cfb`). **ep0 updated to tenantsync v1.2.0 THIS session** (root +SSH, `.bak-1.1.0` kept; `tr -d '\r'` → `bash -n` OK → `install -m0755`; NO sudoers/authorized_keys change). +On-box verify: `echo '{"op":"usage"}' | /usr/local/bin/felhom-tenantsync` → +`{"status":"ok","total":39990112256,"used":7628349440,"avail":30685917184}`; `fingerprint` regression OK. +Manifest bump (`ab91e49`) → ArgoCD hard-refresh → sync → rollout succeeded; image `:0.65.0`. Startup log: ``` -[INFO] felhom-hub 0.64.0 starting -[INFO] Offsite provisioning enabled (pool_box=611714, location=fsn1) -[INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, refresh 15m0s +[INFO] felhom-hub 0.65.0 starting +[INFO] PBS DR tenantsync enabled (endpoint 167.233.158.164:22, user felhom-peersync; …) +[INFO] PBS-DR box checker initialized: fill warn=80% crit=90%, refresh 15m0s ``` ### 6. Live-rendered values (the PROVEN-LIVE evidence) -The hub UI is operator-password-gated (CC cannot log in — hub CLAUDE.md), so the panel/tile can't be -screenshotted directly. Instead the in-cluster checker's first refresh proves the **full pipeline against -the REAL pool box** — the exact numbers the panel/tile render: +Because ep0 was updated to v1.2.0, the PBS gauge shows **real** numbers. The hub UI is operator-password-gated +(CC can't screenshot), so the in-cluster checker's first live poll is the proof — the exact numbers the PBS +panel + gauge render: ``` -[INFO] Offsite pool-box refreshed: 0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x +[INFO] PBS-DR box refreshed: 19.1% full (7.1 GB of 37.2 GB) ``` -Live-confirmed end-to-end: capacity **1.00 TB**, used **2.6 GB** (matches the Phase-0 probe's 2746220544 B), -fill **0.2%**, Σ shared quota **150 GB** (one live shared customer), oversub **0.15×**. The panel-render is -UNIT-verified (`TestOffsiteBoxPanel_*` render the real snapshot shape); the data feeding it is now proven -live. +Live-confirmed end-to-end (hub checker → tenantsync `usage` op → ep0 `df` → snapshot): **7.1 GB of 37.2 GB, +19.1%** — matching the Phase-0 probe. The restic gauge (v0.64.0) continues to render its own live numbers. +Both tabs (Restic / PBS DR) and both dashboard gauges are live; the panel-render is UNIT-verified +(`TestPBSDRPanel_*`), the data proven live. ### 7. NOT yet live-validated -- **The panel/tile UI render** could not be screenshotted (hub UI password-gated). It is unit-verified via - the render tests, and the data behind it is proven live (§6). -- **The alert legs** (`offsite_box_fill` / `offsite_box_oversub` operator emails) are NOT fired live: the - real pool is nominal (0.2% fill, 0.15× oversub — far from the 80%/2.0× thresholds). The optional §13-step-5 - live-fire (temporarily lowering a threshold below the live fill) would send ONE real operator email to - Viktor — deliberately NOT done unprompted. The alert path is unit + red-proof verified (Group C + the C7 - operator-only scope). Offered to the operator to run on demand. +- **The UI render** (tabs + gauges) could not be screenshotted (hub UI password-gated). It is unit-verified + (render tests) and the data behind both gauges is proven live (§6). +- **The PBS fill alert leg** (`pbsdr_box_fill` operator email) is NOT fired live: the datastore is nominal + (19.1%, far from 80/90%). Unit + red-proof verified; a live-fire would email Viktor (offered on demand). +- **The graceful "unavailable" state** was exercised in unit tests, not live — ep0 was updated straight to + v1.2.0, so the hub never saw the old-script path in production (the hub-independence property is verified + by construction + `TestUsage_UnknownOpTypedUnsupported` + `TestPBSDRBox_Unavailable`). ### 8. Observations -- **Threshold ruling pending:** warn 80% / crit 90% / oversub 2.0× are Claude's encoding of the starter - suggestion; the named `Alerting.OffsiteBox*` keys are the one-line flip when Viktor rules. -- **Data-vs-snapshot split usefulness:** on the live box `size_snapshots == 0` (restic-only, no Hetzner - snapshots) — the split is currently uninformative but free, and will matter if snapshot retention is ever - enabled; the panel shows it. -- The `OffsiteChecker` sibling DOES SaveEvent (the spec's "email-only" aside was inexact); the box checker - deliberately does NOT (customer-less scope has no row to key it to) — confirmed the dispatcher's - operator-only path handles that (C7). +- **Separate PBS threshold pair:** shipped as `Alerting.PBSDRBoxFill{Warn,Crit}Percent` defaulting to the + same 80/90 as restic — independently tunable later without touching the restic policy. Whether Viktor + wants a different PBS pair is his call (one-liner). No oversubscription concept for PBS (namespaces, not + quotas) — fill only, as ruled. +- **Dashboard decimal:** the gauges use `%.0f` (e.g. "19%"), so a near-empty box shows "0%" not "0.2%" — + matches the restic tile's existing rounding. The panels show the finer detail. +- The `usage` op is `df`-based because PBS 4.2 has no native `datastore usage` command; if a future PBS adds + one with dedup-aware logical sizing, the op can switch source with no hub/client change (same JSON shape). diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 7f14d91..0b175bc 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -116,5 +116,6 @@ | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | `VALIDATION-offsite-provisioning-e2e-2026-07-09`, `SPIKE-hetzner-api-provisioning-2026-07-09` | | | Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **IMPLEMENTED** | `OffsiteChecker` (`hub/internal/monitor/offsite.go`): fill 90/95% vs soft quota, staleness >48h | No live-fired leg: `CAMPAIGN-offsite-overnight-2026-07-10` recorded no quota/fill/staleness emails, and the freeze write-block was **inconclusive** (only the Hetzner `readonly:true` API op succeeded). Demoted | | **Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert)** | hub v0.64.0 (R-5) | **IMPLEMENTED** (data pipeline PROVEN-LIVE) | `monitor.OffsiteBoxChecker` — fetch-throttled Hetzner GET (1/15 min), fill (used/`storage_box_type.size`, 80/90%) + oversubscription (Σ shared+enabled ConfigJSON quotas / capacity, 2.0×), escalation-only operator alert on the customer-less `"pool-box"` scope; Offsite-tab panel + dashboard tile. **Phase-0-pinned** live shape (box 611714) + **live-computed in-cluster:** `0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x`. Tests + 4 red-proofs; hub v0.64.0 REPORT | Two open legs: the **UI render** is unit-verified only (hub UI password-gated → no screenshot); the **alert emails** are unit + red-proof verified but NOT fired live (real pool nominal — a live-fire emails Viktor). Thresholds pending Viktor's ruling (named config keys). READ-ONLY (GET) | +| **Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge)** | hub v0.65.0 + tenantsync v1.2.0 (R-5) | **IMPLEMENTED** (data pipeline PROVEN-LIVE) | The PBS DR datastore (`felhom-offsite` on ep0) fill — NOT a Hetzner box. **Option A:** a read-only `usage` op on the `felhom-tenantsync` ep0 forced command (twin of `fingerprint`; `df` on the datastore path — no customer_id, no admin token, NO mutation), polled by `monitor.PBSDRBoxChecker` (OffsiteBoxChecker clone; 15-min throttle; states ok/unavailable/degraded; fill 80/90% on the `"pbsdr-box"` operator scope). `/offsite` split into Restic + PBS DR tabs; two dashboard gauges. **Graceful: hub deploy ⟂ ep0 update** (ep0 ≤ v1.1.0 → gauge "n/a" until updated). **Phase-0-pinned** (`df` on ep0 PBS 4.2.3) + **live-computed in-cluster** (ep0 updated to v1.2.0 this session): `19.1% full (7.1 GB of 37.2 GB)`. 10 Go tests + a bash harness + 3 red-proofs; hub v0.65.0 REPORT | Open legs: **UI render** unit-verified only (hub UI password-gated); the **fill alert email** is unit + red-proof verified, NOT fired live (datastore nominal at 19%). Separate PBS threshold keys (default 80/90); no oversubscription (namespaces, not quotas). READ-ONLY | | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | 07-13 closing bundle | | | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | 07-13 | | \ No newline at end of file diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 1b7a0b4..40b8ca3 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -29,7 +29,7 @@ | ID | Item | Size | Status | Notes | |----|------|------|--------|-------| -| R-5 | Hub: Storage Box box-level aggregate — total fill, Σ(shared soft quotas), oversubscription ratio, operator alert | M | **SHIPPED (hub v0.64.0, 2026-07-17)** | `monitor.OffsiteBoxChecker` (throttled Hetzner GET, no ssh/`du`; the new unified API, Robot-vs-new question resolved at source); fill 80/90% + oversub 2.0× (thresholds pending Viktor's ruling — named config keys); Offsite-tab panel + dashboard tile; scope `"pool-box"` → operator only. **Data pipeline PROVEN-LIVE** (`0.2% full, 2.6 GB of 1.00 TB, Σ 150 GB, 0.15×`). OPEN: UI render unit-only (password-gated), alert emails unit + red-proof only (real pool nominal). See hub v0.64.0 REPORT | +| R-5 | Hub: offsite storage visibility — RESTIC box aggregate (v0.64.0) + PBS DR datastore (v0.65.0), each with fill + operator alert; Offsite tabs + dual dashboard gauges | M | **SHIPPED (hub v0.64.0 + v0.65.0 + tenantsync v1.2.0, 2026-07-17)** | Restic: `monitor.OffsiteBoxChecker` (throttled Hetzner GET; fill 80/90% + oversub 2.0×; scope `"pool-box"`). PBS DR (v0.65.0, Option A): read-only `usage` op on the ep0 `felhom-tenantsync` forced command (`df`, no mutation) → `monitor.PBSDRBoxChecker` (fill 80/90%, states ok/unavailable/degraded, scope `"pbsdr-box"`); `/offsite` Restic/PBS-DR tabs + two gauges; hub deploy ⟂ ep0 update (graceful "n/a"). **Both data pipelines PROVEN-LIVE** (restic `0.2% full`; PBS `19.1% full, 7.1 GB of 37.2 GB` after ep0 → v1.2.0). Thresholds pending Viktor (named keys; separate restic/PBS pairs). OPEN: UI render unit-only (password-gated), alert emails unit + red-proof only (both nominal). See hub v0.64.0 + v0.65.0 REPORTs | | R-6 | **Spike: LAN service discovery from the guest** — SSDP multicast (UDP 1900, DLNA), WSD (Windows discovery), mDNS; host-network vs macvlan; is the customer LXC LAN-bridged in appliance deployments? | M | idea | Shared prerequisite for R-7 + R-8. Spike-first: the traps are networking, not Samba/DLNA | | R-7 | SMB **server** share (gated on R-6): samba + wsdd, LAN-only binding (never tunnel), user model (single household user first), which roots are shared (dedicated `shares/` vs app userdata — every SMB-writable path needs a backup class), uid-1000 convention, paperless consume flow | L | idea | Flips map row E/MISSING. Alpha-relevant: "my box is a NAS" is a core household expectation | | R-8 | DLNA (gated on R-6): validate Jellyfin's built-in DLNA server first; only add minidlna to the catalog if Jellyfin-DLNA fails | S | idea | Don't add catalog weight before proving the cheap path |