hub v0.106.0: report loss of visibility into the off-site stores (R-339)

THE GAP, measured not supposed. On 2026-08-18 ep0's PBS proxy was wedged for
9 h 37 m and the hub emitted NOTHING on the operator channel. Both box
checkers hold their last snapshot and return silently on a failed fetch --
correct for a FILL signal, since a missing reading must never be read as 0%,
but it makes a dead off-site endpoint and a healthy one indistinguishable.
The only mails that morning came from the boxes' own backup failures, and
only because the WEEKLY offsite run happened to land inside the window. Two
days earlier nothing would have fired at all.

REACHABILITY is now a second, independent signal on both checkers:
consecutive failed fetch windows, reported past a default 3 windows
(~30-45 min) as pbsdr_box_unreachable / offsite_box_unreachable (warning) on
the customer-less pbsdr-box / pool-box scopes, each with a paired *_recovered
all-clear. Tunable via alerting.box_unreachable_windows (0/invalid -> 3).

THE FILL LOGIC IS UNTOUCHED. No threshold, throttle, band or escalate-once
behaviour changed; a degraded read still drives no transition.

Three decisions a later reader would otherwise "fix" back, so each is
argued in-code:
  - the unreachable event REPEATS rather than escalating once. The band shape
    would give exactly ONE mail at ~minute 30 of a nine-hour outage, and one
    mail is missable. It leans on the dispatcher's 1 h operator cooldown to
    become an hourly "still blind" heartbeat.
  - ErrUsageUnsupported is NOT blindness: an old ep0 answers "no such op",
    which means we reached it. Counting it would alert for days on a healthy
    pre-update endpoint.
  - born-blind is reported: the counter is not gated on having a snapshot, so
    a hub restarted INTO an outage still speaks. last_ok is OMITTED rather
    than zero-valued -- a fabricated timestamp reads as "it was fine until
    then".

Both recoveries are severity "info" and severityNotifies drops "info", so
they are registered in recoveredPairedDownTypes or the operator hears that
the tier broke and never that it healed. A cross-package test drives
ProcessEvent and asserts an actual operator MAIL, not a map entry -- a green
checker test proves nothing about the seam (agent v0.91.0 shipped fully green
with SetAuthSink never called).

Tests: box_reachability_test.go (Scenarios A-F) + dispatcher_box_reachability
_test.go (wiring). Three red-proofs run and reverted, each seen failing with a
message naming the right cause: threshold 3->1, the sentinel counter guard,
the pairing entry.

Register: R-339 filed and marked SHIPPED (PROVEN-LIVE still owed -- no real or
constructed outage has exercised the emit path, and one cannot be manufactured
against Tier-2 ep0). R-340 filed: the reachability read rides ep0's LOCAL API
daemon, which the incident explicitly cleared, so this check would have shown
GREEN for all 9 h 37 m -- the honest boundary, recorded rather than glossed.
R-336's next-step corrected: pvestatd's interval is NOT tunable (Proxmox staff
have said so); the only lever is disabling the storage entry, which collides
with the agent's consume-the-one-time-secret path. Doc-only, no agent code
touched.

hub/internal/monitor/offsite_box.go

@@ -40,12 +40,24 @@ type OffsiteBoxChecker struct {
 	logger      *log.Logger
 	now         func() time.Time // injectable clock (tests)
 
+	reachFailThreshold int // consecutive failed fetch windows before reporting unreachable
+
 	mu          sync.Mutex
 	snap        BoxSnapshot
 	haveSnap    bool
 	lastFetch   time.Time
 	fillBand    string // "" until first eval → born/persistent (a born-breach emits on the first sweep)
 	oversubBand string
+
+	// REACHABILITY (v0.106.0, R-339) — the sibling of PBSDRBoxChecker's, same rules over a different
+	// source. NOTE the one structural difference, stated so a reader does not go looking for a branch
+	// that was forgotten: there is NO ErrUsageUnsupported equivalent here. The Hetzner API has no
+	// version-gated op, so every non-nil error from GetStorageBox is genuine blindness and there is no
+	// "expected data gap" case to exclude.
+	reachFails     int
+	reachAlerted   bool
+	reachFirstFail time.Time
+	lastOK         time.Time // zero = never (born blind — do NOT fabricate one)
 }
 
 // offsiteBoxScope is the customer-less cooldown/display key for pool-box events.
@@ -81,7 +93,7 @@ type BoxSnapshot struct {
 // NewOffsiteBoxChecker builds the checker. Invalid thresholds fall back to the documented defaults
 // (warn 80% / crit 90% / oversub 2.0×). Nothing is seeded → the first Check that finds a breach emits
 // (born/persistent; the dispatcher's 1 h cooldown dedups a hub restart).
-func NewOffsiteBoxChecker(api hetznerapi.CloudAPI, boxID int64, s *store.Store, fillWarn, fillCrit, oversubWarn float64, onEvent EventNotifyFunc, logger *log.Logger) *OffsiteBoxChecker {
+func NewOffsiteBoxChecker(api hetznerapi.CloudAPI, boxID int64, s *store.Store, fillWarn, fillCrit, oversubWarn float64, reachWindows int, onEvent EventNotifyFunc, logger *log.Logger) *OffsiteBoxChecker {
 	if fillWarn <= 0 || fillWarn >= 100 {
 		fillWarn = defaultOffsiteBoxFillWarnPercent
 	}
@@ -94,13 +106,19 @@ func NewOffsiteBoxChecker(api hetznerapi.CloudAPI, boxID int64, s *store.Store,
 	if oversubWarn <= 0 {
 		oversubWarn = defaultOffsiteOversubWarnRatio
 	}
+	if reachWindows <= 0 {
+		reachWindows = defaultBoxUnreachableWindows
+	}
 	c := &OffsiteBoxChecker{
 		api: api, boxID: boxID, store: s,
 		fillWarn: fillWarn, fillCrit: fillCrit, oversubWarn: oversubWarn,
-		onEvent: onEvent, logger: logger, now: time.Now,
+		reachFailThreshold: reachWindows,
+		onEvent:            onEvent, logger: logger, now: time.Now,
 	}
-	logger.Printf("[INFO] Offsite pool-box checker initialized: box=%d fill warn=%.0f%% crit=%.0f%%, oversub warn=%.2fx, refresh %s",
-		boxID, fillWarn, fillCrit, oversubWarn, boxFetchInterval)
+	// Reachability threshold in the init line for the same reason as the PBS-DR checker: its absence
+	// from the pod log is how you notice the parameter never arrived.
+	logger.Printf("[INFO] Offsite pool-box checker initialized: box=%d fill warn=%.0f%% crit=%.0f%%, oversub warn=%.2fx, unreachable after %d consecutive failed reads, refresh %s",
+		boxID, fillWarn, fillCrit, oversubWarn, reachWindows, boxFetchInterval)
 	return c
 }
 
@@ -122,9 +140,28 @@ func (c *OffsiteBoxChecker) Check() {
 		if c.haveSnap {
 			c.snap.Degraded = true // serve the last-known, visibly stale; NO band transition (D)
 		}
+		// REACHABILITY — not gated on haveSnap (a hub restarted into an outage must still report).
+		if c.reachFails == 0 {
+			c.reachFirstFail = c.now()
+		}
+		c.reachFails++
+		if c.reachFails >= c.reachFailThreshold {
+			c.emitUnreachable(err)
+		}
 		return
 	}
 
+	// REACHABILITY: the API answered — a success for this signal even if the payload turns out to be
+	// unusable for a fill percentage (zero capacity below). Handled here, before the degraded early
+	// return, so a zero-capacity read still clears an outstanding blindness alert.
+	if c.reachAlerted {
+		c.emitRecovered()
+		c.reachAlerted = false
+	}
+	c.reachFails = 0
+	c.reachFirstFail = time.Time{}
+	c.lastOK = c.now()
+
 	sumQuotaGB, qerr := c.sumSharedQuotaGB()
 	if qerr != nil {
 		c.logger.Printf("[WARN] Offsite pool-box: quota sum failed (ratio omitted this cycle): %v", qerr)
@@ -182,8 +219,16 @@ func (c *OffsiteBoxChecker) Snapshot() (BoxSnapshot, bool) {
 }
 
 // FillState / OversubState expose the current bands (tests).
-func (c *OffsiteBoxChecker) FillState() string    { c.mu.Lock(); defer c.mu.Unlock(); return orUnknown(c.fillBand) }
-func (c *OffsiteBoxChecker) OversubState() string  { c.mu.Lock(); defer c.mu.Unlock(); return orUnknown(c.oversubBand) }
+func (c *OffsiteBoxChecker) FillState() string {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	return orUnknown(c.fillBand)
+}
+func (c *OffsiteBoxChecker) OversubState() string {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	return orUnknown(c.oversubBand)
+}
 
 func orUnknown(b string) string {
 	if b == "" {
@@ -212,6 +257,51 @@ func (c *OffsiteBoxChecker) sumSharedQuotaGB() (int64, error) {
 	return sum, nil
 }
 
+// emitUnreachable reports that the hub cannot see the shared storage box.
+//
+// ⚠ CADENCE — THIS REPEATS BY DESIGN; see the twin in pbsdr_box.go for the full reasoning. In short:
+// emitFill/emitOversub are escalation-only, which applied here would yield exactly one mail at ~30
+// minutes into a multi-hour outage. This fires on every failed window past the threshold and relies on
+// the dispatcher's 1-hour operator cooldown to become an hourly "still blind" heartbeat.
+func (c *OffsiteBoxChecker) emitUnreachable(cause error) {
+	c.reachAlerted = true
+	lastOK := "never"
+	if !c.lastOK.IsZero() {
+		lastOK = c.lastOK.Format(time.RFC3339)
+	}
+	d := map[string]any{
+		"scope": offsiteBoxScope, "consecutive_failures": c.reachFails, "error": cause.Error(),
+	}
+	// Omitted, never zero-valued, when there has never been a successful read — see Scenario E.
+	if !c.lastOK.IsZero() {
+		d["last_ok"] = c.lastOK.Format(time.RFC3339)
+	}
+	details, _ := json.Marshal(d)
+	msg := fmt.Sprintf("Offsite pool box unreachable — %d consecutive 15-minute checks failed (last successful read: %s); the shared storage box cannot be seen from the hub",
+		c.reachFails, lastOK)
+	c.logger.Printf("[WARN] Offsite pool-box UNREACHABLE: %d consecutive failed reads (last ok: %s)", c.reachFails, lastOK)
+	if c.onEvent != nil {
+		c.onEvent(offsiteBoxScope, "offsite_box_unreachable", "warning", msg, string(details), "hub")
+	}
+}
+
+// emitRecovered is the paired all-clear (severity "info"; reaches the operator only via the
+// dispatcher's recoveredPairedDownTypes entry, which short-circuits the severity gate).
+func (c *OffsiteBoxChecker) emitRecovered() {
+	blindFor := "unknown"
+	if !c.reachFirstFail.IsZero() {
+		blindFor = c.now().Sub(c.reachFirstFail).Round(time.Second).String()
+	}
+	details, _ := json.Marshal(map[string]any{
+		"scope": offsiteBoxScope, "blind_for": blindFor, "consecutive_failures": c.reachFails,
+	})
+	msg := fmt.Sprintf("Offsite pool box reachable again after %s — the shared storage box is visible to the hub", blindFor)
+	c.logger.Printf("[INFO] Offsite pool-box RECOVERED after %s (%d failed reads)", blindFor, c.reachFails)
+	if c.onEvent != nil {
+		c.onEvent(offsiteBoxScope, "offsite_box_recovered", "info", msg, string(details), "hub")
+	}
+}
+
 func (c *OffsiteBoxChecker) emitFill(snap BoxSnapshot, band string) {
 	var severity, message string
 	switch band {