felhom.eu/REPORT.md

# felhom.eu — task reports

> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).

## Hub v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5) — 2026-07-17

### 1. Baseline used

- **felhom.eu** @ `85a1419` ("docs: R-24 … + R-28 … SHIPPED"), clean on `main`, local == origin re-confirmed.
- **Hub** v0.63.0 → **v0.64.0**. Manifest `manifests/hub.yaml` `felhom-hub:0.63.0` → `:0.64.0`.
- READ-ONLY against Hetzner throughout (GET only; no write method called; no new secret; token never logged).

### 2. Phase-0 probe (the gate — PASSED)

One authenticated GET of the live pool box, from 180 via the k8s Secret token (never written to any file):
`GET https://api.hetzner.com/v1/storage_boxes/611714` → HTTP 200. The (redacted, non-secret) shape:

```json
{ "storage_box": {
  "id": 611714, "name": "storage-box-pool-1", "status": "active",
  "storage_box_type": { "id": 1333, "name": "bx11", "description": "BX11", "size": 1099511627776 },
  "stats": { "size": 2746220544, "size_data": 2746220544, "size_snapshots": 0 } } }
```

Pinned: **capacity = `storage_box_type.size`** (1099511627776 B = 1 TiB for bx11); **usage = `stats`**
(`size`/`size_data`/`size_snapshots`), all **bytes**; our token reads it (in scope). The type extension
mirrors this exactly (golden decode test embeds the capture). Gate PASSED → the poller was built.

### 3. Files created / modified

**Code (commit `4bb2df0`):**
- `hub/internal/hetznerapi/{hetznerapi.go,fake.go}` — additive `StorageBoxType` + `StorageBoxStats`; fake `GetBoxCalls`/`FailGetBox`. `+ storagebox_stats_test.go` (golden decode).
- `hub/internal/monitor/offsite_box.go` (new) `+ _test.go` — `OffsiteBoxChecker` (throttled poll, bands, snapshot).
- `hub/internal/offsite/offsite.go` — new `ReadDescriptor` (authoritative quota source; `ClearProvisionedDescriptor` refactored onto it).
- `hub/internal/notify/dispatcher_test.go` — the pool-box operator-only test (C7).
- `hub/cmd/hub/main.go` — 3 `Alerting.OffsiteBox*` config keys; checker construction in the `HETZNER_TOKEN` branch; 60 s sweep registration; `SetOffsiteBox`.
- `hub/internal/web/{server.go,offsite.go,offsite_box.go(new),offsite_box_render_test.go(new),render_test.go}` — panel + tile builder, setter, render tests.
- `hub/internal/web/templates/{offsite.html,dashboard.html,style.css}` — panel, tile, bar/tile band classes.
- `hub/CHANGELOG.md`.

**Refresh-log observability (commit `a4268c3`):** `offsite_box.go` logs the pool trend on each refresh (earns the live-render proof below without the password-gated UI).

**Docs (commit `f2e51e6`):** REPORT (this), CONTEXT, capability-map, ROADMAP.
**Manifest (commit `66da0c4`):** `manifests/hub.yaml` → `:0.64.0`.

### 4. Test results + red-proof outcomes

10 new tests, all green:

| Test | Group | Asserts |
|------|-------|---------|
| `TestOffsiteBox_FetchThrottle` | A | ≈4 API reads over an hour of 60 s sweeps (not ≈60); snapshot populated |
| `TestOffsiteBox_OversubMath` | B | Σ = shared+enabled only (A 500 + B 700); dedicated/disabled excluded; ratio = Σ/capacity = 1.17× |
| `TestOffsiteBox_FillBands` | C1–C5 | 75%→no emit; 82%→1 warn; 85% in-band→no re-emit; 92%→critical; 70%→re-arm; re-breach→emit; every emit scope "pool-box" |
| `TestOffsiteBox_OversubIndependent` | C6 | 2.3× oversub fires alone with fill nominal (50%) — distinct event, no fill emit |
| `TestOffsiteBox_FailedFetchHonesty` | D | failed fetch keeps last values, marks Degraded, no band transition, no emit |
| `TestOffsiteBox_ZeroCapacityGuard` | — | zero capacity → degraded snapshot, no division, no alert |
| `TestStorageBox_DecodesProbeShape` | 1.1 | golden probe JSON decodes to the extended type; data+snapshots == total |
| `TestProcessEvent_PoolBoxScopeOperatorOnly` | C7 | a "pool-box" event reaches the operator ONLY (no customer email) |
| `TestOffsiteBoxPanel_NotConfigured` | E | nil provider → "not configured", no fake zeros |
| `TestOffsiteBoxPanel_WithData` | E | panel renders capacity/ratio + per-customer rows + "no usage reported yet" |

**Red-proofs (run-fail-restore), all confirmed red then restored:**

| # | Removed | Test | Result |
|---|---------|------|--------|
| i | the fetch-throttle guard | `FetchThrottle` | FAILED ("called 60 times… want ≈4") → restored |
| ii | the shared+enabled Σ filter | `OversubMath` | FAILED ("Σquota = 11499… want 1200") → restored |
| iii | the escalation-only guard | `FillBands` | FAILED ("C3 same-band must NOT re-emit, got 2") → restored |
| iv | keep-last-on-failure (zero the snapshot) | `FailedFetchHonesty` | FAILED ("must not alter… before …, after 0/0") → restored |

Full suite: `go build ./... && go vet ./... && go test ./...` — all 17 packages green; hub confirm gate OK.
Test count: +10 (monitor +6, notify +1, hetznerapi +1, web +2).

### 5. Deploy verification

Built + pushed `felhom-hub:0.64.0` on 180 (build source at `4bb2df0`/`a4268c3`). Manifest bump (`66da0c4`)
→ ArgoCD hard-refresh (rev `66da0c44`, OutOfSync) → deliberate sync → rollout succeeded; image
`felhom-hub:0.64.0`. Startup log:

```
[INFO] felhom-hub 0.64.0 starting
[INFO] Offsite provisioning enabled (pool_box=611714, location=fsn1)
[INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, refresh 15m0s
```

### 6. Live-rendered values (the PROVEN-LIVE evidence)

The hub UI is operator-password-gated (CC cannot log in — hub CLAUDE.md), so the panel/tile can't be
screenshotted directly. Instead the in-cluster checker's first refresh proves the **full pipeline against
the REAL pool box** — the exact numbers the panel/tile render:

```
[INFO] Offsite pool-box refreshed: 0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x
```

Live-confirmed end-to-end: capacity **1.00 TB**, used **2.6 GB** (matches the Phase-0 probe's 2746220544 B),
fill **0.2%**, Σ shared quota **150 GB** (one live shared customer), oversub **0.15×**. The panel-render is
UNIT-verified (`TestOffsiteBoxPanel_*` render the real snapshot shape); the data feeding it is now proven
live.

### 7. NOT yet live-validated

- **The panel/tile UI render** could not be screenshotted (hub UI password-gated). It is unit-verified via
  the render tests, and the data behind it is proven live (§6).
- **The alert legs** (`offsite_box_fill` / `offsite_box_oversub` operator emails) are NOT fired live: the
  real pool is nominal (0.2% fill, 0.15× oversub — far from the 80%/2.0× thresholds). The optional §13-step-5
  live-fire (temporarily lowering a threshold below the live fill) would send ONE real operator email to
  Viktor — deliberately NOT done unprompted. The alert path is unit + red-proof verified (Group C + the C7
  operator-only scope). Offered to the operator to run on demand.

### 8. Observations

- **Threshold ruling pending:** warn 80% / crit 90% / oversub 2.0× are Claude's encoding of the starter
  suggestion; the named `Alerting.OffsiteBox*` keys are the one-line flip when Viktor rules.
- **Data-vs-snapshot split usefulness:** on the live box `size_snapshots == 0` (restic-only, no Hetzner
  snapshots) — the split is currently uninformative but free, and will matter if snapshot retention is ever
  enabled; the panel shows it.
- The `OffsiteChecker` sibling DOES SaveEvent (the spec's "email-only" aside was inexact); the box checker
  deliberately does NOT (customer-less scope has no row to key it to) — confirmed the dispatcher's
  operator-only path handles that (C7).