docs: hub v0.64.0 offsite pool-box aggregate (R-5) — REPORT + CONTEXT + capability-map (IMPLEMENTED, data pipeline PROVEN-LIVE) + ROADMAP (R-5 SHIPPED)
This commit is contained in:
+16
@@ -3,6 +3,22 @@
|
||||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||||
|
||||
- **2026-07-17 — OFFSITE POOL-BOX AGGREGATE shipped (hub v0.64.0, R-5).** The operator now sees the
|
||||
shared pool box's real state on the hub: total fill vs capacity, Σ(shared soft quotas)/capacity (the
|
||||
oversubscription ratio), per-customer usage/quota bars, and a box-level operator alert (fill 80/90%,
|
||||
oversub 2.0×). New `monitor.OffsiteBoxChecker` (OffsiteChecker-sibling): fetch-throttled to one Hetzner
|
||||
GET / 15 min, cached `BoxSnapshot`, escalation-only + recovery re-arm; two INDEPENDENT signals (fill +
|
||||
oversub). Capacity = `storage_box_type.size`, usage = `stats.size` (Phase-0-pinned, bytes); Σ(quota)
|
||||
from the ConfigJSON `Descriptor` (`offsite.ReadDescriptor`, new — NEVER the report echo; dedicated +
|
||||
disabled excluded); per-customer USAGE from the report echo. Events scope `"pool-box"` → operator channel
|
||||
only, no SaveEvent. Web: Offsite-tab panel + compact dashboard tile, reading the cached snapshot only
|
||||
(never fetches). **PROVEN-LIVE (data pipeline):** the in-cluster checker computed the real box —
|
||||
`0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x`. **DECISION: thresholds
|
||||
(80/90/2.0×) are Claude's encoding — Viktor's ruling pending; named `Alerting.OffsiteBox*` keys = the
|
||||
one-line flip.** **OPEN:** the UI render is unit-verified only (hub UI password-gated → CC can't
|
||||
screenshot); the alert emails are unit + red-proof verified but NOT fired live (real pool nominal — a
|
||||
live-fire would email Viktor; offered on demand). READ-ONLY against Hetzner throughout. See REPORT.md.
|
||||
|
||||
- **2026-07-17 — SYSTEM-INITIATED IMMEDIACY wired per-plane (hub v0.63.0).** The immediate-sync arc
|
||||
covered only OPERATOR-initiated changes; **system-initiated** desired-state mutations bumped the
|
||||
generation silently, so a freshly onboarded box waited a full agent tick for state the hub had already
|
||||
|
||||
@@ -2,155 +2,120 @@
|
||||
|
||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||||
|
||||
## Hub v0.63.0 — system-initiated immediacy (poke + intent bump at every mutation site) — 2026-07-17
|
||||
## Hub v0.64.0 — offsite pool-box aggregate: fill, oversubscription, per-customer bars, operator alert (R-5) — 2026-07-17
|
||||
|
||||
### 1. Baseline used
|
||||
|
||||
- **felhom.eu** @ `4c9b0e87063278cddbe51e3966a8f8592c7c18f2` ("docs: R-21 slice C …", 2026-07-17),
|
||||
clean tree on `main`, local == `origin/main` re-confirmed before starting.
|
||||
- **Hub** v0.62.0 → **v0.63.0**. Deploy manifest `manifests/hub.yaml` image line `felhom-hub:0.62.0` → `:0.63.0`.
|
||||
- Both immediate-sync transports were already PROVEN-LIVE (agent poke: agent v0.89.0 + hub v0.59.0,
|
||||
2026-07-17; intent wait channel: hub v0.58.0 / controller v0.140.0). This task is **call-site wiring
|
||||
only** — no new mechanism, no spike.
|
||||
- **felhom.eu** @ `85a1419` ("docs: R-24 … + R-28 … SHIPPED"), clean on `main`, local == origin re-confirmed.
|
||||
- **Hub** v0.63.0 → **v0.64.0**. Manifest `manifests/hub.yaml` `felhom-hub:0.63.0` → `:0.64.0`.
|
||||
- READ-ONLY against Hetzner throughout (GET only; no write method called; no new secret; token never logged).
|
||||
|
||||
### 2. Files created / modified
|
||||
### 2. Phase-0 probe (the gate — PASSED)
|
||||
|
||||
**Implementation (commit `30972d8`):**
|
||||
- `hub/internal/web/pbsdr.go` — +3 pokes: `PBSDRAutoProvision`, `ReissuePBSDR`, `handlePBSDRReissue` (each after the successful `SetHostDesired`).
|
||||
- `hub/internal/api/handler.go` — new nil-safe `Poker` interface + `poker` field + `SetPoker`; poke in `handleAdminSetDesiredState`; intent `Bump` in `reissueOnReenroll`.
|
||||
- `hub/internal/api/wg.go` — `PokeAllHosts` in `handleAdminSetOperatorPeer` (only on a successful `BumpAllHostGenerations`).
|
||||
- `hub/cmd/hub/main.go` — capture the one `poke.Notifier` and wire it to BOTH planes (`webServer.SetPoke(n)` + `apiHandler.SetPoker(n)`); startup log "web + api admin seams armed".
|
||||
- `hub/internal/web/pbsdr_poke_test.go` (new) — 5 web tests (Groups A, B).
|
||||
- `hub/internal/api/poke_seam_test.go` (new) — 5 api tests (Groups C, D).
|
||||
- `hub/CHANGELOG.md` — v0.63.0 entry (newest on top).
|
||||
- `REUSE.md` — new canonical-pattern row: "Immediate-sync notify (per plane)".
|
||||
One authenticated GET of the live pool box, from 180 via the k8s Secret token (never written to any file):
|
||||
`GET https://api.hetzner.com/v1/storage_boxes/611714` → HTTP 200. The (redacted, non-secret) shape:
|
||||
|
||||
**Docs (commit `50c090e`):**
|
||||
- `REPORT.md` (this file, overwritten), `CONTEXT.md` (decision recorded), `documentation/architecture/00-capability-map.md` (immediacy row note + version), `documentation/backlog/ROADMAP.md` (new R-28: agent fast-tick SECONDARY).
|
||||
```json
|
||||
{ "storage_box": {
|
||||
"id": 611714, "name": "storage-box-pool-1", "status": "active",
|
||||
"storage_box_type": { "id": 1333, "name": "bx11", "description": "BX11", "size": 1099511627776 },
|
||||
"stats": { "size": 2746220544, "size_data": 2746220544, "size_snapshots": 0 } } }
|
||||
```
|
||||
|
||||
**Manifest (commit `587fcc2`):** `manifests/hub.yaml` image tag → `felhom-hub:0.63.0`.
|
||||
Pinned: **capacity = `storage_box_type.size`** (1099511627776 B = 1 TiB for bx11); **usage = `stats`**
|
||||
(`size`/`size_data`/`size_snapshots`), all **bytes**; our token reads it (in scope). The type extension
|
||||
mirrors this exactly (golden decode test embeds the capture). Gate PASSED → the poller was built.
|
||||
|
||||
### 3. Per-commit hashes pushed to `main`
|
||||
### 3. Files created / modified
|
||||
|
||||
| # | Hash | Contents |
|
||||
|---|------|----------|
|
||||
| 1 | `30972d8` | code + tests + CHANGELOG + REUSE |
|
||||
| 2 | `50c090e` | REPORT + CONTEXT + capability-map + ROADMAP |
|
||||
| 3 | `587fcc2` | manifests/hub.yaml image bump → 0.63.0 |
|
||||
**Code (commit `4bb2df0`):**
|
||||
- `hub/internal/hetznerapi/{hetznerapi.go,fake.go}` — additive `StorageBoxType` + `StorageBoxStats`; fake `GetBoxCalls`/`FailGetBox`. `+ storagebox_stats_test.go` (golden decode).
|
||||
- `hub/internal/monitor/offsite_box.go` (new) `+ _test.go` — `OffsiteBoxChecker` (throttled poll, bands, snapshot).
|
||||
- `hub/internal/offsite/offsite.go` — new `ReadDescriptor` (authoritative quota source; `ClearProvisionedDescriptor` refactored onto it).
|
||||
- `hub/internal/notify/dispatcher_test.go` — the pool-box operator-only test (C7).
|
||||
- `hub/cmd/hub/main.go` — 3 `Alerting.OffsiteBox*` config keys; checker construction in the `HETZNER_TOKEN` branch; 60 s sweep registration; `SetOffsiteBox`.
|
||||
- `hub/internal/web/{server.go,offsite.go,offsite_box.go(new),offsite_box_render_test.go(new),render_test.go}` — panel + tile builder, setter, render tests.
|
||||
- `hub/internal/web/templates/{offsite.html,dashboard.html,style.css}` — panel, tile, bar/tile band classes.
|
||||
- `hub/CHANGELOG.md`.
|
||||
|
||||
**Refresh-log observability (commit `a4268c3`):** `offsite_box.go` logs the pool trend on each refresh (earns the live-render proof below without the password-gated UI).
|
||||
|
||||
**Docs (commit `<DOCS_HASH>`):** REPORT (this), CONTEXT, capability-map, ROADMAP.
|
||||
**Manifest (commit `66da0c4`):** `manifests/hub.yaml` → `:0.64.0`.
|
||||
|
||||
### 4. Test results + red-proof outcomes
|
||||
|
||||
**New tests (10), all green:**
|
||||
10 new tests, all green:
|
||||
|
||||
| Test | Group | Asserts |
|
||||
|------|-------|---------|
|
||||
| `TestPBSDR_AutoProvisionPokes` | A | auto-provision success → exactly one poke to the resolved WG /32 (10.77.0.2) within 2 s |
|
||||
| `TestPBSDR_AutoProvisionBlockedDoesNotPoke` | A (neg) | blocked precondition (no WG peer) → zero pokes |
|
||||
| `TestPBSDR_ReissueCorePokes` | B1 | `ReissuePBSDR` success → one poke to the host's /32 |
|
||||
| `TestPBSDR_ReissueCoreErrorDoesNotPoke` | B (neg) | tenantsync-error reissue → zero pokes (fire-after-commit) |
|
||||
| `TestPBSDR_HandleReissuePokes` | B2 | operator button success → one poke |
|
||||
| `TestAdminSetDesiredState_PokesTargetHostOnly` | C1 | admin-set success → PokeHost==1 (target only), PokeAllHosts==0; invalid-JSON path → 0 |
|
||||
| `TestAdminSetDesiredState_NilPokerNoPanic` | C1 (nil) | nil poker → mutation still 200, no panic |
|
||||
| `TestAdminSetOperatorPeer_PokesFleet` | C2 | operator-peer success → PokeAllHosts==1, PokeHost==0 |
|
||||
| `TestReenroll_BumpsIntent` | D | re-enroll → intent generation advances (synchronous) |
|
||||
| `TestReenroll_NilIntentHubNoPanic` | D (nil) | nil intent hub → enroll still 201, no panic |
|
||||
| `TestOffsiteBox_FetchThrottle` | A | ≈4 API reads over an hour of 60 s sweeps (not ≈60); snapshot populated |
|
||||
| `TestOffsiteBox_OversubMath` | B | Σ = shared+enabled only (A 500 + B 700); dedicated/disabled excluded; ratio = Σ/capacity = 1.17× |
|
||||
| `TestOffsiteBox_FillBands` | C1–C5 | 75%→no emit; 82%→1 warn; 85% in-band→no re-emit; 92%→critical; 70%→re-arm; re-breach→emit; every emit scope "pool-box" |
|
||||
| `TestOffsiteBox_OversubIndependent` | C6 | 2.3× oversub fires alone with fill nominal (50%) — distinct event, no fill emit |
|
||||
| `TestOffsiteBox_FailedFetchHonesty` | D | failed fetch keeps last values, marks Degraded, no band transition, no emit |
|
||||
| `TestOffsiteBox_ZeroCapacityGuard` | — | zero capacity → degraded snapshot, no division, no alert |
|
||||
| `TestStorageBox_DecodesProbeShape` | 1.1 | golden probe JSON decodes to the extended type; data+snapshots == total |
|
||||
| `TestProcessEvent_PoolBoxScopeOperatorOnly` | C7 | a "pool-box" event reaches the operator ONLY (no customer email) |
|
||||
| `TestOffsiteBoxPanel_NotConfigured` | E | nil provider → "not configured", no fake zeros |
|
||||
| `TestOffsiteBoxPanel_WithData` | E | panel renders capacity/ratio + per-customer rows + "no usage reported yet" |
|
||||
|
||||
Async web assertions synchronize on a buffered channel in the fake sender with a bounded `select`
|
||||
(≤2 s), never sleep-poll. The web fake sender structurally satisfies poke's unexported `pokeSender`
|
||||
seam (value passed, type never named); the notifier is a REAL `poke.Notifier` over the test store —
|
||||
exactly as `main.go` builds it. The api fake `Poker` satisfies the exported `api.Poker` interface and
|
||||
counts calls synchronously.
|
||||
**Red-proofs (run-fail-restore), all confirmed red then restored:**
|
||||
|
||||
**Red-proofs (run-fail-restore), one representative per scenario group:**
|
||||
| # | Removed | Test | Result |
|
||||
|---|---------|------|--------|
|
||||
| i | the fetch-throttle guard | `FetchThrottle` | FAILED ("called 60 times… want ≈4") → restored |
|
||||
| ii | the shared+enabled Σ filter | `OversubMath` | FAILED ("Σquota = 11499… want 1200") → restored |
|
||||
| iii | the escalation-only guard | `FillBands` | FAILED ("C3 same-band must NOT re-emit, got 2") → restored |
|
||||
| iv | keep-last-on-failure (zero the snapshot) | `FailedFetchHonesty` | FAILED ("must not alter… before …, after 0/0") → restored |
|
||||
|
||||
| Group | Removed | Test | Result |
|
||||
|-------|---------|------|--------|
|
||||
| A | `PBSDRAutoProvision` poke (Part 1.1) | `TestPBSDR_AutoProvisionPokes` | FAILED red ("no poke within 2s… 10.77.0.2") → restored |
|
||||
| B | `ReissuePBSDR` poke (Part 1.2) | `TestPBSDR_ReissueCorePokes` | FAILED red ("no poke within 2s… 10.77.0.2") → restored |
|
||||
| C | `handleAdminSetDesiredState` poke (Part 2.2) | `TestAdminSetDesiredState_PokesTargetHostOnly` | FAILED red ("PokeHost targets = [], want [h1]") → restored |
|
||||
| D | `reissueOnReenroll` bump (Part 3) | `TestReenroll_BumpsIntent` | FAILED red ("intent generation = 0, want > 0") → restored |
|
||||
Full suite: `go build ./... && go vet ./... && go test ./...` — all 17 packages green; hub confirm gate OK.
|
||||
Test count: +10 (monitor +6, notify +1, hetznerapi +1, web +2).
|
||||
|
||||
These tests fail on the pre-fix tree by construction; a representative removal per group was
|
||||
demonstrated (B2 and C2 share their group's wiring shape with the demonstrated B1/C1 removals).
|
||||
### 5. Deploy verification
|
||||
|
||||
### 5. Test count before/after
|
||||
|
||||
- Package `internal/web`: +5 tests (new `pbsdr_poke_test.go`).
|
||||
- Package `internal/api`: +5 tests (new `poke_seam_test.go`).
|
||||
- Full suite after: `go build ./... && go vet ./... && go test ./...` — **all packages green** (api + web
|
||||
are the only two exercising the new code; every other package unchanged).
|
||||
|
||||
### 6. Deployed version + pod/log verification
|
||||
|
||||
Built on 180 from `~/git/felhom.eu` @ `50c090e` (build.sh `git pull --ff-only` confirmed the wiring is
|
||||
present in the built source) → `gitea.dooplex.hu/admin/felhom-hub:0.63.0` pushed. Manifest bump
|
||||
(`587fcc2`) → ArgoCD app `felhom` hard-refresh (revision `587fcc27ca03`, OutOfSync) → deliberate sync:
|
||||
Built + pushed `felhom-hub:0.64.0` on 180 (build source at `4bb2df0`/`a4268c3`). Manifest bump (`66da0c4`)
|
||||
→ ArgoCD hard-refresh (rev `66da0c44`, OutOfSync) → deliberate sync → rollout succeeded; image
|
||||
`felhom-hub:0.64.0`. Startup log:
|
||||
|
||||
```
|
||||
sync=Synced health=Progressing
|
||||
deployment "hub" successfully rolled out
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.63.0
|
||||
|
||||
# startup log (poke ENABLED — POKE_SSH_KEY_FILE Secret present in this environment):
|
||||
[INFO] felhom-hub 0.63.0 starting
|
||||
[INFO] agent-plane poke enabled (endpoint 167.233.158.164:22, user felhom-peersync; web + api admin seams armed)
|
||||
[INFO] felhom-hub 0.64.0 starting
|
||||
[INFO] Offsite provisioning enabled (pool_box=611714, location=fsn1)
|
||||
[INFO] Offsite pool-box checker initialized: box=611714 fill warn=80% crit=90%, oversub warn=2.00x, refresh 15m0s
|
||||
```
|
||||
|
||||
The startup line is the new one — "web + api admin seams armed" confirms `apiHandler.SetPoker(n)` was
|
||||
wired alongside `webServer.SetPoke(n)` over the one notifier. Poke is enabled (not the disabled
|
||||
fallback), so all wired sites are armed live.
|
||||
### 6. Live-rendered values (the PROVEN-LIVE evidence)
|
||||
|
||||
### 7. NOT yet live-validated (real system-initiated events only)
|
||||
The hub UI is operator-password-gated (CC cannot log in — hub CLAUDE.md), so the panel/tile can't be
|
||||
screenshotted directly. Instead the in-cluster checker's first refresh proves the **full pipeline against
|
||||
the REAL pool box** — the exact numbers the panel/tile render:
|
||||
|
||||
The new pokes/bumps fire only on real system events, none manufacturable non-destructively from this
|
||||
task. Acceptance bar here = **unit tests + deploy + startup-log verification**. Unfired live legs, to be
|
||||
observed at the rehearsal's bind sequence (`RUNBOOK-rehearsal-v3 S5`):
|
||||
```
|
||||
[INFO] Offsite pool-box refreshed: 0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x
|
||||
```
|
||||
|
||||
- **Auto-provision poke** — fires when a DR-ON customer's host registers its WG peer for the first time
|
||||
and the atom provisions (the exact observed slice-C lag).
|
||||
- **Reconciler-escalation poke** — fires when the pbsdrheal reconciler escalates to `ReissuePBSDR` (a box
|
||||
stuck in `waiting_secret`/`consumed_failed` after losing its converged marker).
|
||||
- **Clean-slate re-enroll bump** — fires when a wiped, re-minted box re-enrolls for a claimed/offsite
|
||||
customer (on that path the controller usually does not exist yet; its startup fetch covers that shape).
|
||||
- **Operator-button / admin-api pokes** (`handlePBSDRReissue`, admin desired-state / operator-peer) — the
|
||||
operator-triggered legs; exercisable in a supervised session but not fired here.
|
||||
Live-confirmed end-to-end: capacity **1.00 TB**, used **2.6 GB** (matches the Phase-0 probe's 2746220544 B),
|
||||
fill **0.2%**, Σ shared quota **150 GB** (one live shared customer), oversub **0.15×**. The panel-render is
|
||||
UNIT-verified (`TestOffsiteBoxPanel_*` render the real snapshot shape); the data feeding it is now proven
|
||||
live.
|
||||
|
||||
The **first-hold-baseline race** stays OPEN by design: a bump landing during a fresh controller's FIRST
|
||||
wait is recorded as baseline without firing (the ruled fix is carrying `intent_gen` in the report ACK —
|
||||
its own future task, out of scope).
|
||||
### 7. NOT yet live-validated
|
||||
|
||||
### 8. Part-4 audit table — EVERY desired-state mutation site
|
||||
- **The panel/tile UI render** could not be screenshotted (hub UI password-gated). It is unit-verified via
|
||||
the render tests, and the data behind it is proven live (§6).
|
||||
- **The alert legs** (`offsite_box_fill` / `offsite_box_oversub` operator emails) are NOT fired live: the
|
||||
real pool is nominal (0.2% fill, 0.15× oversub — far from the 80%/2.0× thresholds). The optional §13-step-5
|
||||
live-fire (temporarily lowering a threshold below the live fill) would send ONE real operator email to
|
||||
Viktor — deliberately NOT done unprompted. The alert path is unit + red-proof verified (Group C + the C7
|
||||
operator-only scope). Offered to the operator to run on demand.
|
||||
|
||||
Re-grepped at implementation time for all non-test callers of `SetHostDesired`, `BumpHostDesired`,
|
||||
`BumpAllHostGenerations`, plus the system-initiated customer-deliverable writers. No forgotten fourth
|
||||
caller — every site below is accounted for.
|
||||
### 8. Observations
|
||||
|
||||
| Site (file:landmark) | Initiator | Plane | Notification after this task | Reason if none |
|
||||
|---|---|---|---|---|
|
||||
| `web/pbsdr.go` `applyPBSDR` disable (~L143→148) | operator (form) | agent | poke (pre-existing) | — |
|
||||
| `web/pbsdr.go` `applyPBSDR` re-enable/storage-id (~L168→173) | operator (form) | agent | poke (pre-existing) | — |
|
||||
| `web/pbsdr.go` `applyPBSDR` fresh-provision (~L179→186) | operator (form) | agent | poke (pre-existing) | — |
|
||||
| `web/pbsdr.go` `pbsdrProvisionAtom` `SetHostDesired` (~L273) | — (shared atom) | agent | poke at each caller | callers poke: `applyPBSDR` (L186) + `PBSDRAutoProvision` (NEW) |
|
||||
| `web/pbsdr.go` `PBSDRAutoProvision` (~L312) | **system** (WG-reg hook) | agent | **NEW poke** | — (the observed lag) |
|
||||
| `web/pbsdr.go` `ReissuePBSDR` `SetHostDesired` (~L357) | **system** (reconciler escalation) | agent | **NEW poke** | — (also covers pbsdrheal, zero reconciler change) |
|
||||
| `web/pbsdr.go` `handlePBSDRReissue` `SetHostDesired` (~L409) | operator (button) | agent | **NEW poke** | — |
|
||||
| `api/handler.go` `handleAdminSetDesiredState` `SetHostDesired` (~L1419) | operator/system (admin api) | agent | **NEW poke** (`PokeHost`) | — |
|
||||
| `api/wg.go` `handleAdminSetOperatorPeer` `BumpAllHostGenerations` (~L504) | operator (admin api) | agent | **NEW poke** (`PokeAllHosts`, fire-after-commit) | — |
|
||||
| `api/wg.go` `handleRegisterWGPeer` `BumpHostDesired` (~L291) | system (agent registration) | agent | **none (deliberate)** | box's tunnel does not exist until it fetches the WG block — a poke is undeliverable by construction; the agent fast-tick SECONDARY (R-28) owns this leg |
|
||||
| `api/wg.go` WG peer delete `BumpHostDesired` (~L422) | operator | agent | **none (deliberate)** | the mutation removes the transport |
|
||||
| `pbsdrheal` **Restage** path | system (reconciler) | agent | **none (deliberate)** | no generation bump → the agent's 60 s pbsdr ticker is the pickup path (a poke is a verified no-op); `internal/pbsdrheal/` byte-unchanged |
|
||||
| `api/handler.go` `reissueOnReenroll` (F2 claim + F3 offsite, ~L1071) | **system** (clean-slate re-enroll) | controller | **NEW intent bump** | — |
|
||||
| `web/configs.go` bumpIntent ×11 (L564/633/657/702/742/779/796/842/859/876/1055) | operator | controller | intent bump (pre-existing) | — |
|
||||
| `web/customer_reset.go` bumpIntent (~L226) | operator | controller | intent bump (pre-existing) | — |
|
||||
| `web/logtail.go` bumpIntent (~L35) / `web/logbundle.go` bumpIntent (~L68) | operator | controller | intent bump (pre-existing) | — |
|
||||
|
||||
### 9. Observations (not acted on)
|
||||
|
||||
- **`handlePBSDRReissue` ↔ `ReissuePBSDR` duplication.** The operator button re-implements the reissue
|
||||
atom's tail (reissue → store secret → descriptor bump) instead of calling `ReissuePBSDR`, deliberately,
|
||||
to keep granular per-step HTTP error strings (the "keep in lockstep" comment marks the known
|
||||
duplication). Both now poke after their `SetHostDesired`. Not deduped (per spec forbidden-dedup); if
|
||||
ever unified, the handler's error-string granularity is the constraint to preserve.
|
||||
- **`pbsdrProvisionAtom` is poke-free by design** — its two callers poke on success (form path at L186,
|
||||
hook path at the new L312). This keeps the poke at the decision point (blocked vs minted) rather than
|
||||
inside the shared atom, matching the existing L186 structure.
|
||||
- **Threshold ruling pending:** warn 80% / crit 90% / oversub 2.0× are Claude's encoding of the starter
|
||||
suggestion; the named `Alerting.OffsiteBox*` keys are the one-line flip when Viktor rules.
|
||||
- **Data-vs-snapshot split usefulness:** on the live box `size_snapshots == 0` (restic-only, no Hetzner
|
||||
snapshots) — the split is currently uninformative but free, and will matter if snapshot retention is ever
|
||||
enabled; the panel shows it.
|
||||
- The `OffsiteChecker` sibling DOES SaveEvent (the spec's "email-only" aside was inexact); the box checker
|
||||
deliberately does NOT (customer-less scope has no row to key it to) — confirmed the dispatcher's
|
||||
operator-only path handles that (C7).
|
||||
|
||||
@@ -115,6 +115,6 @@
|
||||
| Controller self-update: anonymous registry, no credentials in guest | controller v0.112 | **PROVEN-LIVE** (demo) | 07-10 arc | |
|
||||
| Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | `VALIDATION-offsite-provisioning-e2e-2026-07-09`, `SPIKE-hetzner-api-provisioning-2026-07-09` | |
|
||||
| Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **IMPLEMENTED** | `OffsiteChecker` (`hub/internal/monitor/offsite.go`): fill 90/95% vs soft quota, staleness >48h | No live-fired leg: `CAMPAIGN-offsite-overnight-2026-07-10` recorded no quota/fill/staleness emails, and the freeze write-block was **inconclusive** (only the Hetzner `readonly:true` API op succeeded). Demoted |
|
||||
| **Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert)** | — | **MISSING** | — | → R-5 |
|
||||
| **Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert)** | hub v0.64.0 (R-5) | **IMPLEMENTED** (data pipeline PROVEN-LIVE) | `monitor.OffsiteBoxChecker` — fetch-throttled Hetzner GET (1/15 min), fill (used/`storage_box_type.size`, 80/90%) + oversubscription (Σ shared+enabled ConfigJSON quotas / capacity, 2.0×), escalation-only operator alert on the customer-less `"pool-box"` scope; Offsite-tab panel + dashboard tile. **Phase-0-pinned** live shape (box 611714) + **live-computed in-cluster:** `0.2% full (2.6 GB of 1.00 TB), Σ shared quota 150 GB, oversub 0.15x`. Tests + 4 red-proofs; hub v0.64.0 REPORT | Two open legs: the **UI render** is unit-verified only (hub UI password-gated → no screenshot); the **alert emails** are unit + red-proof verified but NOT fired live (real pool nominal — a live-fire emails Viktor). Thresholds pending Viktor's ruling (named config keys). READ-ONLY (GET) |
|
||||
| Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | 07-13 closing bundle | |
|
||||
| Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | 07-13 | |
|
||||
@@ -29,7 +29,7 @@
|
||||
|
||||
| ID | Item | Size | Status | Notes |
|
||||
|----|------|------|--------|-------|
|
||||
| R-5 | Hub: Storage Box box-level aggregate — total BX11 fill, Σ(customer soft quotas), oversubscription ratio, operator alert threshold | M | idea | Per-customer OffsiteChecker exists (hub v0.41). Source: Hetzner Storage Box API (verify Robot-legacy vs. new API endpoint for our box before speccing — 2025 migration); NOT ssh `du` over restic repos. Token → k8s Secret ref. Flips map row G/MISSING |
|
||||
| R-5 | Hub: Storage Box box-level aggregate — total fill, Σ(shared soft quotas), oversubscription ratio, operator alert | M | **SHIPPED (hub v0.64.0, 2026-07-17)** | `monitor.OffsiteBoxChecker` (throttled Hetzner GET, no ssh/`du`; the new unified API, Robot-vs-new question resolved at source); fill 80/90% + oversub 2.0× (thresholds pending Viktor's ruling — named config keys); Offsite-tab panel + dashboard tile; scope `"pool-box"` → operator only. **Data pipeline PROVEN-LIVE** (`0.2% full, 2.6 GB of 1.00 TB, Σ 150 GB, 0.15×`). OPEN: UI render unit-only (password-gated), alert emails unit + red-proof only (real pool nominal). See hub v0.64.0 REPORT |
|
||||
| R-6 | **Spike: LAN service discovery from the guest** — SSDP multicast (UDP 1900, DLNA), WSD (Windows discovery), mDNS; host-network vs macvlan; is the customer LXC LAN-bridged in appliance deployments? | M | idea | Shared prerequisite for R-7 + R-8. Spike-first: the traps are networking, not Samba/DLNA |
|
||||
| R-7 | SMB **server** share (gated on R-6): samba + wsdd, LAN-only binding (never tunnel), user model (single household user first), which roots are shared (dedicated `shares/` vs app userdata — every SMB-writable path needs a backup class), uid-1000 convention, paperless consume flow | L | idea | Flips map row E/MISSING. Alpha-relevant: "my box is a NAS" is a core household expectation |
|
||||
| R-8 | DLNA (gated on R-6): validate Jellyfin's built-in DLNA server first; only add minidlna to the catalog if Jellyfin-DLNA fails | S | idea | Don't add catalog weight before proving the cheap path |
|
||||
|
||||
Reference in New Issue
Block a user