hub v0.106.0: report loss of visibility into the off-site stores (R-339)

THE GAP, measured not supposed. On 2026-08-18 ep0's PBS proxy was wedged for
9 h 37 m and the hub emitted NOTHING on the operator channel. Both box
checkers hold their last snapshot and return silently on a failed fetch --
correct for a FILL signal, since a missing reading must never be read as 0%,
but it makes a dead off-site endpoint and a healthy one indistinguishable.
The only mails that morning came from the boxes' own backup failures, and
only because the WEEKLY offsite run happened to land inside the window. Two
days earlier nothing would have fired at all.

REACHABILITY is now a second, independent signal on both checkers:
consecutive failed fetch windows, reported past a default 3 windows
(~30-45 min) as pbsdr_box_unreachable / offsite_box_unreachable (warning) on
the customer-less pbsdr-box / pool-box scopes, each with a paired *_recovered
all-clear. Tunable via alerting.box_unreachable_windows (0/invalid -> 3).

THE FILL LOGIC IS UNTOUCHED. No threshold, throttle, band or escalate-once
behaviour changed; a degraded read still drives no transition.

Three decisions a later reader would otherwise "fix" back, so each is
argued in-code:
  - the unreachable event REPEATS rather than escalating once. The band shape
    would give exactly ONE mail at ~minute 30 of a nine-hour outage, and one
    mail is missable. It leans on the dispatcher's 1 h operator cooldown to
    become an hourly "still blind" heartbeat.
  - ErrUsageUnsupported is NOT blindness: an old ep0 answers "no such op",
    which means we reached it. Counting it would alert for days on a healthy
    pre-update endpoint.
  - born-blind is reported: the counter is not gated on having a snapshot, so
    a hub restarted INTO an outage still speaks. last_ok is OMITTED rather
    than zero-valued -- a fabricated timestamp reads as "it was fine until
    then".

Both recoveries are severity "info" and severityNotifies drops "info", so
they are registered in recoveredPairedDownTypes or the operator hears that
the tier broke and never that it healed. A cross-package test drives
ProcessEvent and asserts an actual operator MAIL, not a map entry -- a green
checker test proves nothing about the seam (agent v0.91.0 shipped fully green
with SetAuthSink never called).

Tests: box_reachability_test.go (Scenarios A-F) + dispatcher_box_reachability
_test.go (wiring). Three red-proofs run and reverted, each seen failing with a
message naming the right cause: threshold 3->1, the sentinel counter guard,
the pairing entry.

Register: R-339 filed and marked SHIPPED (PROVEN-LIVE still owed -- no real or
constructed outage has exercised the emit path, and one cannot be manufactured
against Tier-2 ep0). R-340 filed: the reachability read rides ep0's LOCAL API
daemon, which the incident explicitly cleared, so this check would have shown
GREEN for all 9 h 37 m -- the honest boundary, recorded rather than glossed.
R-336's next-step corrected: pvestatd's interval is NOT tunable (Proxmox staff
have said so); the only lever is disabling the storage entry, which collides
with the agent's consume-the-one-time-secret path. Doc-only, no agent code
touched.

documentation/architecture/00-capability-map.md

@@ -157,5 +157,6 @@
 | Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **IMPLEMENTED** | `OffsiteChecker` (`hub/internal/monitor/offsite.go`): fill 90/95% vs soft quota, staleness >48h | No live-fired leg: `CAMPAIGN-offsite-overnight-2026-07-10` recorded no quota/fill/staleness emails, and the freeze write-block was **inconclusive** (only the Hetzner `readonly:true` API op succeeded). Demoted |
 | **Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert)** | hub v0.64.0 (R-5) | **IMPLEMENTED** (data pipeline PROVEN-LIVE) | `monitor.OffsiteBoxChecker` — fetch-throttled Hetzner GET (1/15 min), fill (used/`storage_box_type.size`, 80/90%) + oversubscription (Σ shared+enabled ConfigJSON quotas / capacity, 2.0×), escalation-only operator alert on the customer-less `"pool-box"` scope; Offsite-tab panel + dashboard tile. **Phase-0-pinned** live shape (box 611714) + **live-computed in-cluster:** `0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x`. Tests + 4 red-proofs; hub v0.64.0 REPORT | Two open legs: the **UI render** is unit-verified only (hub UI password-gated → no screenshot); the **alert emails** are unit + red-proof verified but NOT fired live (real pool nominal — a live-fire emails Viktor). Thresholds pending Viktor's ruling (named config keys). READ-ONLY (GET) |
 | **Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge)** | hub v0.65.0 + tenantsync v1.2.0 (R-5) | **IMPLEMENTED** (data pipeline PROVEN-LIVE) | The PBS DR datastore (`felhom-offsite` on ep0) fill — NOT a Hetzner box. **Option A:** a read-only `usage` op on the `felhom-tenantsync` ep0 forced command (twin of `fingerprint`; `df` on the datastore path — no customer_id, no admin token, NO mutation), polled by `monitor.PBSDRBoxChecker` (OffsiteBoxChecker clone; 15-min throttle; states ok/unavailable/degraded; fill 80/90% on the `"pbsdr-box"` operator scope). `/offsite` split into Restic + PBS DR tabs; two dashboard gauges. **Graceful: hub deploy ⟂ ep0 update** (ep0 ≤ v1.1.0 → gauge "n/a" until updated). **Phase-0-pinned** (`df` on ep0 PBS 4.2.3) + **live-computed in-cluster** (ep0 updated to v1.2.0 this session): `19.1% full (7.1 GB of 37.2 GB)`. 10 Go tests + a bash harness + 3 red-proofs; hub v0.65.0 REPORT | Open legs: **UI render** unit-verified only (hub UI password-gated); the **fill alert email** is unit + red-proof verified, NOT fired live (datastore nominal at 19%). Separate PBS threshold keys (default 80/90); no oversubscription (namespaces, not quotas). READ-ONLY |
+| **The hub reports LOSS OF VISIBILITY into either off-site store (not just how full it is)** | hub **v0.106.0** (R-339) | **IMPLEMENTED — deliberately NOT proven-live** | Both box checkers count consecutive failed fetch windows and emit `pbsdr_box_unreachable` / `offsite_box_unreachable` (severity `warning`) past a default 3 windows (≈30–45 min), each with a paired `*_recovered` all-clear routed via `recoveredPairedDownTypes` — required because the recoveries are severity `info`, which `severityNotifies` drops. Scopes stay customer-less (`pbsdr-box` / `pool-box`) → operator channel only. Fill logic untouched: a degraded read still drives no band transition. Evidence: `internal/monitor/box_reachability_test.go` + the cross-package wiring test in `internal/notify/`, which asserts an actual operator mail rather than a map entry. **Filed BECAUSE of a measured gap**, not a hypothesis: the 2026-08-18 ep0 outage ran 9 h 37 m with the hub silent | **The gap that remains is R-340**, and it is not small: the ep0 read is the `usage` op, which rides the LOCAL API daemon — the daemon that incident explicitly cleared — so this check would have shown GREEN for that entire outage. It closes "ep0 is unreachable as a host"; it does not close what actually happened. **No live or constructed outage has exercised the emit path**, and one cannot be manufactured against ep0 (Tier 2, protected) |
 | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | 07-13 closing bundle | |
 | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | 07-13 | |