felhom.eu/REPORT.md

felhom.eu — task reports

Overwrite this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in hub/CHANGELOG.md; the scripts history lives in scripts/CHANGELOG.md.

Hub v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5) — 2026-07-17

1. Baseline used

• felhom.eu @ 85a1419 ("docs: R-24 … + R-28 … SHIPPED"), clean on main, local == origin re-confirmed.
• Hub v0.63.0 → v0.64.0. Manifest manifests/hub.yaml felhom-hub:0.63.0 → :0.64.0.
• READ-ONLY against Hetzner throughout (GET only; no write method called; no new secret; token never logged).

2. Phase-0 probe (the gate — PASSED)

One authenticated GET of the live pool box, from 180 via the k8s Secret token (never written to any file): GET https://api.hetzner.com/v1/storage_boxes/611714 → HTTP 200. The (redacted, non-secret) shape:

{ "storage_box": {
  "id": 611714, "name": "storage-box-pool-1", "status": "active",
  "storage_box_type": { "id": 1333, "name": "bx11", "description": "BX11", "size": 1099511627776 },
  "stats": { "size": 2746220544, "size_data": 2746220544, "size_snapshots": 0 } } }

Pinned: capacity = storage_box_type.size (1099511627776 B = 1 TiB for bx11); usage = stats (size/size_data/size_snapshots), all bytes; our token reads it (in scope). The type extension mirrors this exactly (golden decode test embeds the capture). Gate PASSED → the poller was built.

3. Files created / modified

Code (commit 4bb2df0):

• hub/internal/hetznerapi/{hetznerapi.go,fake.go} — additive StorageBoxType + StorageBoxStats; fake GetBoxCalls/FailGetBox. + storagebox_stats_test.go (golden decode).
• hub/internal/monitor/offsite_box.go (new) + _test.go — OffsiteBoxChecker (throttled poll, bands, snapshot).
• hub/internal/offsite/offsite.go — new ReadDescriptor (authoritative quota source; ClearProvisionedDescriptor refactored onto it).
• hub/internal/notify/dispatcher_test.go — the pool-box operator-only test (C7).
• hub/cmd/hub/main.go — 3 Alerting.OffsiteBox* config keys; checker construction in the HETZNER_TOKEN branch; 60 s sweep registration; SetOffsiteBox.
• hub/internal/web/{server.go,offsite.go,offsite_box.go(new),offsite_box_render_test.go(new),render_test.go} — panel + tile builder, setter, render tests.
• hub/internal/web/templates/{offsite.html,dashboard.html,style.css} — panel, tile, bar/tile band classes.
• hub/CHANGELOG.md.

Refresh-log observability (commit a4268c3): offsite_box.go logs the pool trend on each refresh (earns the live-render proof below without the password-gated UI).

Docs (commit f2e51e6): REPORT (this), CONTEXT, capability-map, ROADMAP. Manifest (commit 66da0c4): manifests/hub.yaml → :0.64.0.

4. Test results + red-proof outcomes

10 new tests, all green:

Test	Group	Asserts
TestOffsiteBox_FetchThrottle	A	≈4 API reads over an hour of 60 s sweeps (not ≈60); snapshot populated
TestOffsiteBox_OversubMath	B	Σ = shared+enabled only (A 500 + B 700); dedicated/disabled excluded; ratio = Σ/capacity = 1.17×
TestOffsiteBox_FillBands	C1–C5	75%→no emit; 82%→1 warn; 85% in-band→no re-emit; 92%→critical; 70%→re-arm; re-breach→emit; every emit scope "pool-box"
TestOffsiteBox_OversubIndependent	C6	2.3× oversub fires alone with fill nominal (50%) — distinct event, no fill emit
TestOffsiteBox_FailedFetchHonesty	D	failed fetch keeps last values, marks Degraded, no band transition, no emit
TestOffsiteBox_ZeroCapacityGuard	—	zero capacity → degraded snapshot, no division, no alert
TestStorageBox_DecodesProbeShape	1.1	golden probe JSON decodes to the extended type; data+snapshots == total
TestProcessEvent_PoolBoxScopeOperatorOnly	C7	a "pool-box" event reaches the operator ONLY (no customer email)
TestOffsiteBoxPanel_NotConfigured	E	nil provider → "not configured", no fake zeros
TestOffsiteBoxPanel_WithData	E	panel renders capacity/ratio + per-customer rows + "no usage reported yet"

Red-proofs (run-fail-restore), all confirmed red then restored:

#	Removed	Test	Result
i	the fetch-throttle guard	FetchThrottle	FAILED ("called 60 times… want ≈4") → restored
ii	the shared+enabled Σ filter	OversubMath	FAILED ("Σquota = 11499… want 1200") → restored
iii	the escalation-only guard	FillBands	FAILED ("C3 same-band must NOT re-emit, got 2") → restored
iv	keep-last-on-failure (zero the snapshot)	FailedFetchHonesty	FAILED ("must not alter… before …, after 0/0") → restored

Full suite: go build ./... && go vet ./... && go test ./... — all 17 packages green; hub confirm gate OK. Test count: +10 (monitor +6, notify +1, hetznerapi +1, web +2).

5. Deploy verification

Built + pushed felhom-hub:0.64.0 on 180 (build source at 4bb2df0/a4268c3). Manifest bump (66da0c4) → ArgoCD hard-refresh (rev 66da0c44, OutOfSync) → deliberate sync → rollout succeeded; image felhom-hub:0.64.0. Startup log:

[INFO] felhom-hub 0.64.0 starting
[INFO] Offsite provisioning enabled (pool_box=611714, location=fsn1)
[INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, refresh 15m0s

6. Live-rendered values (the PROVEN-LIVE evidence)

The hub UI is operator-password-gated (CC cannot log in — hub CLAUDE.md), so the panel/tile can't be screenshotted directly. Instead the in-cluster checker's first refresh proves the full pipeline against the REAL pool box — the exact numbers the panel/tile render:

[INFO] Offsite pool-box refreshed: 0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x

Live-confirmed end-to-end: capacity 1.00 TB, used 2.6 GB (matches the Phase-0 probe's 2746220544 B), fill 0.2%, Σ shared quota 150 GB (one live shared customer), oversub 0.15×. The panel-render is UNIT-verified (TestOffsiteBoxPanel_* render the real snapshot shape); the data feeding it is now proven live.

7. NOT yet live-validated

• The panel/tile UI render could not be screenshotted (hub UI password-gated). It is unit-verified via the render tests, and the data behind it is proven live (§6).
• The alert legs (offsite_box_fill / offsite_box_oversub operator emails) are NOT fired live: the real pool is nominal (0.2% fill, 0.15× oversub — far from the 80%/2.0× thresholds). The optional §13-step-5 live-fire (temporarily lowering a threshold below the live fill) would send ONE real operator email to Viktor — deliberately NOT done unprompted. The alert path is unit + red-proof verified (Group C + the C7 operator-only scope). Offered to the operator to run on demand.

8. Observations

• Threshold ruling pending: warn 80% / crit 90% / oversub 2.0× are Claude's encoding of the starter suggestion; the named Alerting.OffsiteBox* keys are the one-line flip when Viktor rules.
• Data-vs-snapshot split usefulness: on the live box size_snapshots == 0 (restic-only, no Hetzner snapshots) — the split is currently uninformative but free, and will matter if snapshot retention is ever enabled; the panel shows it.
• The OffsiteChecker sibling DOES SaveEvent (the spec's "email-only" aside was inexact); the box checker deliberately does NOT (customer-less scope has no row to key it to) — confirmed the dispatcher's operator-only path handles that (C7).