From 1133aade73d7d426a1da3876a5afa7255156e01f Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 23 Jul 2026 12:59:04 +0200 Subject: [PATCH] =?UTF-8?q?hub=20v0.72.0=20=E2=80=94=20R-70=20+=20R-71c:?= =?UTF-8?q?=20offsite=20delivery-state=20detector,=20card,=20stuck=20event?= =?UTF-8?q?,=20R-39(a)-guarded=20self-heal=20restage?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR --- REUSE.md | 3 + documentation/backlog/ROADMAP.md | 4 +- hub/CHANGELOG.md | 42 +++ hub/cmd/hub/main.go | 12 + hub/internal/monitor/offsite_delivery.go | 188 +++++++++++ hub/internal/monitor/offsite_delivery_test.go | 302 ++++++++++++++++++ .../dispatcher_offsite_delivery_test.go | 44 +++ hub/internal/offsite/delivery.go | 97 ++++++ hub/internal/offsite/delivery_test.go | 172 ++++++++++ hub/internal/store/store.go | 112 +++++++ hub/internal/web/configs.go | 75 +++++ .../web/configs_delivery_render_test.go | 122 +++++++ .../web/templates/config_form_body.html | 9 +- 13 files changed, 1179 insertions(+), 3 deletions(-) create mode 100644 hub/internal/monitor/offsite_delivery.go create mode 100644 hub/internal/monitor/offsite_delivery_test.go create mode 100644 hub/internal/notify/dispatcher_offsite_delivery_test.go create mode 100644 hub/internal/offsite/delivery.go create mode 100644 hub/internal/offsite/delivery_test.go create mode 100644 hub/internal/web/configs_delivery_render_test.go diff --git a/REUSE.md b/REUSE.md index f9b63ff..89e8004 100644 --- a/REUSE.md +++ b/REUSE.md @@ -99,6 +99,9 @@ |---|---|---|---|---| | `tenantsync.Client` (`Provision`/`Reissue`/`Fingerprint`) | hub/internal/tenantsync/client.go | `(ctx, customerID) (*Result, error)` | ep0 per-customer PBS tenancy over the pinned-SSH forced-command channel (the wgsync twin) | `Result.TokenSecret` is transient custody → `SaveHostPBSSecret` immediately, never log the struct. Error paths NEVER embed stdout (the secret channel) — do not "improve" diagnostics by quoting the response. `ErrTokenExists` is typed: provision refuses an existing token; re-issue is the explicit path. | | `(*Store).SaveHostPBSSecret` / `ConsumeHostPBSSecret` | hub/internal/store/pbsdr.go | `(hostID, value)` / `(hostID) (string, error)` | HOST-scoped consume-once secret (the one_time_secrets host twin) | Same-tx mark-consumed; re-save resets consumption (re-issue supersedes). The agent consumes via `POST /api/v1/hosts/{id}/pbs/consume-token` (hub/internal/api/pbsdr.go). | +| `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. | +| `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). | +| `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. | | `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. | | `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. | | `cfg.DRTier` + offsite coupling | hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) | bool | Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) | One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field `dr_tier` (formBool helper). | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index ee1eebe..16f58c0 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -81,8 +81,8 @@ | R-67 | **The NAS share appears in FileBrowser — browse what you mounted.** | S | **SHIPPED (controller v0.160.0, 2026-07-22)** | Origin: the R-64 pairing drill — the share said „Elérhető" and the customer had no way to BROWSE it (FileBrowser synced drives only). **Couples to R-64: browsing was its missing UX half.** A registered network storage now binds its share ROOT into FileBrowser (`/mnt/felhom-drives/:/srv/:rslave`) with its display label as the sidebar source; NAS add/remove trigger the same debounced sync. Two classes, two gates: drives keep the drive-absent gate byte-identically (proven live: the drives-only box logged a no-op sync); network shares gate on the STUB classifier instead — idle autofs is HEALTHY and included (Phase-0 probe on demo-hp: an in-container access through an rslave bind WAKES the idle trigger), while a stub verdict excludes the share from mounts AND sources with a WARN (an exposed stub swallows uploads the real mount later shadows). Nothing is ever written toward the NAS (no skeleton — red-proven). Live leg: cross-box upload round-trip demo-hp → demo-felhom + dead-NAS check (`Host is down` in seconds, unaided recovery after samba restart). Operator residual: the FileBrowser UI click-through (its admin credential is customer-held by design). Evidence: `felhom-controller/REPORT.md` (2026-07-22) | | R-68 | **Notification train: paired recovery mails + prefs seeding at claim + priority headers (power-outage audit F11+F12+F14-light).** The dead-man's-switch fired perfectly on 07-22 and the customer who got „A szerver nem elérhető!" was never told it recovered (F11); a customer without a `customer_notifications` row is silently unnotifiable (F12, demo-hp live); delivered ≠ noticed (F14). | M | **SHIPPED (hub v0.71.0, 2026-07-22)** | Origin: `AUDIT-power-outage-recovery-2026-07-22.md`. Recovery = explicit eventType branch (severity semantics frozen; `severityNotifies` untouched): operator always hears both edges, customer iff PAIRED (customer-channel `sent` stale/down row newer than the last sent recovery — `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for recovery; ties → no mail, flap-safe). Seed-at-claim: `MarkClaimed` → `SeedNotificationPrefs` (INSERT-if-absent, never upsert — red-proofed; empty email no-op; never fails the claim; default critical-only set). Hub-side empty-email no-clobber belt in `handleSavePreferences` (controller 0.160.0 already guards its own two push legs — latent, not live). `X-Priority: 1` + `Importance: high` on error/critical via Resend `headers` (live-probed HTTP 200 before implementation); the `test` event now also mails the operator with those headers (one click proves both channels + rendering); latent `sendTestEmail` nil-prefs panic fixed. 17 tests + 4 red-proofs. **Live legs pending:** natural `*_recovered` mail on the next real staleness cycle (or the reboot-drill arc — NEVER fabricated by blocking reports, that is F9-bypass-shaped) and seed-at-claim on a real claim (Peti Friday reinstall is the natural candidate) | | R-69 | **F14-full: an operator push channel that actually interrupts (ntfy / Telegram / similar), beyond mail-client priority flags.** F14-light (v0.71.0 headers + Gmail filter) nudges a mail client; a 15:29 node_down should reach the operator's pocket in seconds regardless of inbox hygiene. Needs: channel choice (self-hosted ntfy on k3s vs Telegram bot), dispatcher fan-out seam, per-severity routing, quiet hours. | M | idea | Origin: `AUDIT-power-outage-recovery-2026-07-22.md` F14. Deliberately NOT built in the v0.71.0 train (scope-forked per the task spec) | -| R-70 | **[P2-HIGH] The offsite last mile is invisible on BOTH surfaces — the hub cannot tell "staged" from "delivered" from "applied".** demo-hp sat 2 days with the hub customer page saying "Provisioned: … the transient password is delivered to the controller once" while the box said „Még nincs beállítva távoli mentési cél" — and a real customer would sit unprotected indefinitely believing otherwise. The hub HAS the signal (`one_time_secrets.consumed_at` + 153 consecutive reports carrying no offbox object) and reads none of it: the "Provisioned" line is static copy gated only on `offsite.host` in ConfigJSON (`config_form_body.html:119–120`). | S–M | idea | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. Two legs: **hub customer card** shows the real delivery state ("provisioned, awaiting box consumption" / "consumed, awaiting apply" / "applied" — consumed_at × report-offbox-presence is enough for all three), and **controller banner** when the descriptor is enabled but no target is configured („Felhom offsite készen áll — a beállítás automatikus, folyamatban"). Couple to **R-31**'s async/status-card idiom (same surface likely serves both) and to the **R-39 `consumed_at` honesty gauge** precedent on the PBS side — a consumed secret + N report cycles with no offbox status is the same "disagreement no single tier can see" shape and deserves the same loud event. Supporting live datum: demo-felhom's 07-21 staged secret is still unconsumed today (key-auth-first path never consumes) — invisible for the same reason. | -| R-71 | **[P1] Day-0 race: the managed floor-update kills the offsite apply-bridge between password-consume and persist — the one-shot credential is burned and the box lands in the silent consume-404 dead-end forever.** Proven on demo-hp (07-21): consume 16:27:42 → managed update 0.153.0→0.156.0 replaces the container 16:28:17, ~35 s later, mid `ssh-copy-id` window; nothing persisted, no installed key ⇒ the key-auth-first recovery path can never engage, and every subsequent start logs the WARN and gives up until an operator Re-issue. **This recurs structurally on every fresh onboarding whose ISO floor lags the managed floor** — the update fires minutes after first boot, exactly when the bridge first runs. demo-felhom escaped by timing only. | M | idea | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` (mechanism cites: bridge order `offsiteapply.go:106–187` consume-then-persist, retry only on process start; the dead-end is even documented in source l.168–173 "the password is spent; reset it on the hub to retry"). Candidate directions, spec-first: (a) **order** — first-boot path lets the managed update settle before the bridge's consume step (cheapest; the race window is the update, not the reboot); (b) **two-phase consume** — hub marks consumed only on a controller ack-after-persist (touches the 404-no-oracle contract, design carefully); (c) **hub-side self-heal** — auto-restage a fresh secret when `consumed_at` is set but K consecutive reports show offsite enabled with no offbox status (the R-39(a) mint-race lesson applies: never restage on top of an UNCONSUMED secret). Pairs with R-70 (visibility) — but visibility alone is not the fix; the burned credential needs an unattended recovery path. | +| R-70 | **[P2-HIGH] The offsite last mile is invisible on BOTH surfaces — the hub cannot tell "staged" from "delivered" from "applied".** demo-hp sat 2 days with the hub customer page saying "Provisioned: … the transient password is delivered to the controller once" while the box said „Még nincs beállítva távoli mentési cél" — and a real customer would sit unprotected indefinitely believing otherwise. The hub HAS the signal (`one_time_secrets.consumed_at` + 153 consecutive reports carrying no offbox object) and reads none of it: the "Provisioned" line is static copy gated only on `offsite.host` in ConfigJSON (`config_form_body.html:119–120`). | S–M | **SHIPPED (hub v0.72.0 + controller v0.161.0, 2026-07-23)** — detector `offsite.DeliveryStateFor` (one impl, all consumers), customer-card state line with age (static "delivered once" copy GONE), `offsite_delivery_stuck` warning event (24h durable cooldown), controller truthful empty-state banner. Live validation on the two fixtures (demo-hp `applied`; demo-felhom `applied` + stale-staged info) recorded in `felhom.eu/REPORT.md`; the banner leg is unit-proven/live-pending (no box occupies the enabled+no-offbox window). | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. Two legs: **hub customer card** shows the real delivery state ("provisioned, awaiting box consumption" / "consumed, awaiting apply" / "applied" — consumed_at × report-offbox-presence is enough for all three), and **controller banner** when the descriptor is enabled but no target is configured („Felhom offsite készen áll — a beállítás automatikus, folyamatban"). Couple to **R-31**'s async/status-card idiom (same surface likely serves both) and to the **R-39 `consumed_at` honesty gauge** precedent on the PBS side — a consumed secret + N report cycles with no offbox status is the same "disagreement no single tier can see" shape and deserves the same loud event. Supporting live datum: demo-felhom's 07-21 staged secret is still unconsumed today (key-auth-first path never consumes) — invisible for the same reason. | +| R-71 | **[P1] Day-0 race: the managed floor-update kills the offsite apply-bridge between password-consume and persist — the one-shot credential is burned and the box lands in the silent consume-404 dead-end forever.** Proven on demo-hp (07-21): consume 16:27:42 → managed update 0.153.0→0.156.0 replaces the container 16:28:17, ~35 s later, mid `ssh-copy-id` window; nothing persisted, no installed key ⇒ the key-auth-first recovery path can never engage, and every subsequent start logs the WARN and gives up until an operator Re-issue. **This recurs structurally on every fresh onboarding whose ISO floor lags the managed floor** — the update fires minutes after first boot, exactly when the bridge first runs. demo-felhom escaped by timing only. | M | **PARTIAL — (c) SHIPPED (hub v0.72.0, 2026-07-23): self-heal restage via the EXISTING Re-issue path (`monitor.OffsiteDeliveryChecker`), trigger = consumed ≥1h + ≥4 consecutive offbox-less reports + zero offbox evidence, one restage/customer/24h (durable via events table), every firing = `offsite_credential_restaged` warning; R-39(a) act-time guard red-proofed (clobber observed with guard removed). Ships unit-proven, NOT live-fired (no broken box existed; arms on next natural occurrence or a staged drill — never PROVEN-LIVE until then). (a) day-0 ordering stays OPEN → its own upcoming spec. | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` (mechanism cites: bridge order `offsiteapply.go:106–187` consume-then-persist, retry only on process start; the dead-end is even documented in source l.168–173 "the password is spent; reset it on the hub to retry"). Candidate directions, spec-first: (a) **order** — first-boot path lets the managed update settle before the bridge's consume step (cheapest; the race window is the update, not the reboot); (b) **two-phase consume** — hub marks consumed only on a controller ack-after-persist (touches the 404-no-oracle contract, design carefully); (c) **hub-side self-heal** — auto-restage a fresh secret when `consumed_at` is set but K consecutive reports show offsite enabled with no offbox status (the R-39(a) mint-race lesson applies: never restage on top of an UNCONSUMED secret). Pairs with R-70 (visibility) — but visibility alone is not the fix; the burned credential needs an unattended recovery path. | | R-53 | **`app_export.html` substituted the CSRF token where the customer domain belongs** - the open-in-browser link was wrong for every app with a subdomain, and a session CSRF token landed in a URL. | XS | **SHIPPED (controller v0.150.0, 2026-07-20)** | One template token (`{{$.CSRFToken}}` -> `{{$.Domain}}`) plus the `Domain` key in `exportPageHandler`'s data map - that handler does not go through `baseData`, which is where every other page gets it, so the template had no domain to read. Render tests assert the joined `.` and that the token appears nowhere in that line; red-proofed against the pre-fix template. Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F7) | ## P3 — post-alpha diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 80e833c..6ae1347 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,47 @@ # Felhom Hub — Changelog +## v0.72.0 — R-70 + R-71(c): the offsite last mile becomes visible, and burned credentials heal themselves (2026-07-23) + +Origin: `documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` — demo-hp sat 2 days with the +customer card claiming "Provisioned … delivered to the controller once" (static copy) while the box +had NOTHING: the day-0 managed update killed the apply-bridge after password-consume, and the hub — +holding both signals (`one_time_secrets.consumed_at`, 153 offbox-less reports) — read neither. + +**One detector, four consumers:** + +- **Detector** (`internal/offsite/delivery.go`, `DeliveryStateFor`): per-customer + `OffsiteDeliveryState` from the secret row × report offsite-presence — `applied` (precedence: + the box's own report wins) / `consumed_awaiting_apply` (the burned-credential shape) / + `staged_awaiting_consume` / `no_secret`. The applied+stale-staged edge (demo-felhom's live + shape: key-auth-first never consumes) stays `applied` PLUS a visible stale-staged flag. New + store reads: `GetOneTimeSecretInfo` (timestamps only, value never selected), + `LatestReportOffsitePresence`, `CountReportsOffsiteSince`, `LastEventAt`, + a test-only + timestamp back-dater (the PBSDR seam pattern). +- **Customer card** (`config_form_body.html` + `deliveryViewFor`): the static "delivered once" + claim is GONE; the card renders the derived state with its age (badge idiom `n-ok/n-warn/ + n-neutral`; consumed goes amber past 30 min; the stale-staged info line names the specimen's + age). Render test per branch (the v0.70.1 template-gate rule). +- **Loud event**: `offsite_delivery_stuck` (WARNING → operator email; severity gate untouched — + explicit tests pin that info would be silent) when consumed_awaiting_apply persists ≥ 1 h; + per-customer 24 h cooldown, DURABLE via `LastEventAt` over the events table (a hub restart + neither floods nor resets). +- **Self-heal (R-71c)**: `OffsiteDeliveryChecker` (shared 60 s ticker) invokes the EXISTING + Re-issue path (`web.Server.ReissueOffsiteForCustomer` behind the narrow `monitor.OffsiteReissuer` + interface — the pbsdrheal precedent; armed only when the provisioner is configured) when the + shape is unambiguous: consumed ≥ 1 h + ≥ 4 consecutive offbox-less reports since consume + zero + offbox evidence. One restage per customer per 24 h (durable); every firing surfaces as + `offsite_credential_restaged` (WARNING → operator) — repeats are repeating events, never a + silent retry loop. **The R-39(a) guard**: `SaveOneTimeSecret` clobbers by design (Re-issue + depends on supersede), so the heal RE-READS the secret row immediately before acting and refuses + unless it is still a consumed row — the TOCTOU (operator Re-issue landing mid-tick) is the shape + the guard kills. Red-proofs run and recorded (guard removed → the operator's fresh secret gets + clobbered, test fails on `reissue calls = 1`; both rate limits removed → duplicate fire). + +Companion: felhom-controller v0.161.0 (the box-side truthful empty state). The self-heal ships +unit-proven + red-proofed, NOT live-fired — no broken box exists and none was broken for it; it +arms on the next natural occurrence (or a staged drill). R-71(a) (day-0 ordering) stays open — +separate spec. + ## v0.71.0 — paired recovery mails, prefs seeding at claim, priority headers, operator test leg (2026-07-22) Origin: `documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md` F11 (recovery is silent), diff --git a/hub/cmd/hub/main.go b/hub/cmd/hub/main.go index e9b875a..6a29dd2 100644 --- a/hub/cmd/hub/main.go +++ b/hub/cmd/hub/main.go @@ -306,6 +306,11 @@ func main() { // still renders, but saving with offsite enabled returns "not configured". var offsiteBoxChecker *monitor.OffsiteBoxChecker var pbsdrBoxChecker *monitor.PBSDRBoxChecker // R-5 v0.65.0: PBS-DR datastore fill (constructed with the tenantsync client below) + // R-71c: the delivery checker's self-heal invokes the SAME Re-issue path the operator button + // uses (webServer satisfies monitor.OffsiteReissuer). Wired ONLY when the provisioner exists — + // a heal that cannot actually restage must never run (it would emit a restaged event for a + // silent no-op: ReissueOffsiteForCustomer returns nil when offsite is unconfigured). + var offsiteHealReissuer monitor.OffsiteReissuer if tok := os.Getenv("HETZNER_TOKEN"); tok != "" { poolBoxID, _ := strconv.ParseInt(os.Getenv("HETZNER_POOL_BOX_ID"), 10, 64) location := os.Getenv("HETZNER_LOCATION") @@ -317,6 +322,7 @@ func main() { API: client, Store: dataStore, Scanner: offsite.SSHHostKeyScanner{}, PoolBoxID: poolBoxID, Location: location, Logger: logger, }) logger.Printf("[INFO] Offsite provisioning enabled (pool_box=%d, location=%s)", poolBoxID, location) + offsiteHealReissuer = webServer // R-71c heal armed (provisioner present) // R-5 (v0.64.0): the pool-box aggregate checker shares the SAME client + pool box id (GET-only). // It needs a valid box id to poll; without one, the aggregate stays unconfigured. if poolBoxID != 0 { @@ -544,6 +550,11 @@ func main() { // (90/95% of quota_gb) + staleness (enabled+escrowed but no run >48h — the silently-stuck detector; // run FAILURES already alert via backup_failed). Nil-safe on pre-v0.109 reports. Same sweep. offsiteChecker := monitor.NewOffsiteChecker(dataStore, 0, dispatcher.ProcessEvent, logger) + // R-70 + R-71c: the delivery-state checker — surfaces the burned-credential shape as + // offsite_delivery_stuck (warning, 24h/customer) and self-heals it via the Re-issue path + // (offsite_credential_restaged, one restage/customer/24h, R-39(a)-guarded). Cooldowns are + // durable (events table), so a hub restart neither floods nor silently re-heals. + offsiteDeliveryChecker := monitor.NewOffsiteDeliveryChecker(dataStore, offsiteHealReissuer, dispatcher.ProcessEvent, logger) go func() { ticker := time.NewTicker(60 * time.Second) defer ticker.Stop() @@ -561,6 +572,7 @@ func main() { hostMgmtPlaneChecker.Check() hostOOBChecker.Check() offsiteChecker.Check() + offsiteDeliveryChecker.Check() if offsiteBoxChecker != nil { offsiteBoxChecker.Check() // R-5: restic pool-box aggregate (fetch-throttled internally) } diff --git a/hub/internal/monitor/offsite_delivery.go b/hub/internal/monitor/offsite_delivery.go new file mode 100644 index 0000000..13ec5b2 --- /dev/null +++ b/hub/internal/monitor/offsite_delivery.go @@ -0,0 +1,188 @@ +package monitor + +import ( + "context" + "encoding/json" + "fmt" + "log" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/offsite" + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// R-70 + R-71(c): the offsite delivery-state checker. Reads the shared detector +// (offsite.DeliveryStateFor) for every offsite-enabled customer and drives two consumers: +// +// - the LOUD EVENT: `offsite_delivery_stuck` (warning → operator email per existing dispatcher +// rules) when the burned-credential shape persists past stuckAfter; +// - the SELF-HEAL (R-71c): invoke the EXISTING Re-issue path — never a second delivery +// mechanism — when the shape is unambiguous, then surface `offsite_credential_restaged` +// (warning) so the operator ALWAYS knows it fired. +// +// Cooldowns are durable: both events rate-limit off store.LastEventAt (the events table), so a hub +// restart cannot flood or silently re-heal. A repeating pattern surfaces as repeating events on a +// 24 h cadence, never as a silent retry loop. +// +// THE R-39(a) GUARD (mandatory, enforced HERE because the store deliberately clobbers): +// SaveOneTimeSecret is last-write-wins by design — Re-issue depends on supersede. Restaging on top +// of an UNCONSUMED secret would clobber a password a box may be about to consume (the operator may +// have clicked Re-issue between this checker's derive and its act). So the heal re-reads the +// secret row IMMEDIATELY before acting and refuses unless it is still a CONSUMED row. + +const ( + eventDeliveryStuck = "offsite_delivery_stuck" // hub-internal (not in allowedEventTypes, like pbsdr_*) + eventCredentialRestaged = "offsite_credential_restaged" // hub-internal + // stuckAfter: consumed_awaiting_apply is normal for seconds (F10 repair: consume→applied in + // 4 s). An hour of it means the apply will never come without intervention. + stuckAfter = time.Hour + // stuckCooldown / healCooldown: per-customer, durable via the events table. + stuckCooldown = 24 * time.Hour + healCooldown = 24 * time.Hour + // healMinReports: at least this many consecutive offbox-less reports after consumed_at before + // the self-heal may fire — the box must be alive and reporting, just not applied. + healMinReports = 4 +) + +// OffsiteReissuer is the narrow reissue surface the self-heal needs — satisfied by +// *web.Server.ReissueOffsiteForCustomer (the pbsdrheal.Reissuer precedent; avoids an import +// cycle and guarantees the heal IS the designed Re-issue path, not a sibling mechanism). +type OffsiteReissuer interface { + ReissueOffsiteForCustomer(ctx context.Context, customerID string) error +} + +// OffsiteDeliveryChecker runs on the shared monitor ticker. +type OffsiteDeliveryChecker struct { + store *store.Store + reissuer OffsiteReissuer // nil → self-heal disabled (no provisioner configured); detector+event still run + onEvent EventNotifyFunc + logger *log.Logger + now func() time.Time +} + +// NewOffsiteDeliveryChecker constructs the checker. reissuer may be nil (heal disabled). +func NewOffsiteDeliveryChecker(s *store.Store, reissuer OffsiteReissuer, onEvent EventNotifyFunc, logger *log.Logger) *OffsiteDeliveryChecker { + return &OffsiteDeliveryChecker{store: s, reissuer: reissuer, onEvent: onEvent, logger: logger, now: time.Now} +} + +// Check derives the delivery state for every offsite-enabled active customer and applies the +// event + self-heal rules. Never returns an error — a checker failure must not take the ticker +// down (log-and-continue, like every sibling checker). +func (c *OffsiteDeliveryChecker) Check() { + configs, err := c.store.ListCustomerConfigs() + if err != nil { + c.logger.Printf("[WARN] offsite-delivery: list configs: %v", err) + return + } + for _, cfg := range configs { + if cfg.Status != "active" { + continue // blocked/inactive customers are not delivery-monitored (and never healed) + } + d, err := offsite.ReadDescriptor(cfg.ConfigJSON) + if err != nil || d == nil || !d.Enabled { + continue // unparseable config never drives a heal; the config UI owns that failure + } + status, err := offsite.DeliveryStateFor(c.store, cfg.CustomerID) + if err != nil { + c.logger.Printf("[WARN] offsite-delivery: %s: derive: %v", cfg.CustomerID, err) + continue + } + if status.State != offsite.DeliveryConsumedAwaitingApply { + continue // applied / staged / no_secret: card-rendered states, no event or heal (yet) + } + age := c.now().Sub(status.Since) + if age < stuckAfter { + continue // normal convergence window + } + c.maybeEmitStuck(cfg.CustomerID, status, age) + c.maybeHeal(cfg.CustomerID, status) + } +} + +// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer. +func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) { + last, err := c.store.LastEventAt(customerID, eventDeliveryStuck) + if err != nil { + c.logger.Printf("[WARN] offsite-delivery: %s: cooldown read: %v", customerID, err) + return + } + if !last.IsZero() && c.now().Sub(last) < stuckCooldown { + return + } + msg := fmt.Sprintf("Offsite delivery stuck: one-time password consumed %s ago and %d report(s) since carry no offbox target — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one.", + age.Round(time.Minute), status.ReportsSinceConsume) + details, _ := json.Marshal(map[string]any{ + "state": string(status.State), + "consumed_at": status.Since.UTC().Format(time.RFC3339), + "reports_since_consume": status.ReportsSinceConsume, + }) + c.emit(customerID, eventDeliveryStuck, "warning", msg, string(details)) +} + +// maybeHeal fires the R-71c self-heal when the burned-credential shape is unambiguous: +// consumed ≥ stuckAfter ago, ≥ healMinReports consecutive reports since with ZERO offbox evidence, +// one heal per healCooldown — and the R-39(a) guard holds at act time. +func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus) { + if c.reissuer == nil { + return + } + if status.ReportsSinceConsume < healMinReports || status.OffsiteReportsSinceConsume != 0 { + return // box not reporting enough, or offbox evidence exists (regressed-apply shape) → operator's call + } + last, err := c.store.LastEventAt(customerID, eventCredentialRestaged) + if err != nil { + c.logger.Printf("[WARN] offsite-delivery: %s: heal rate-limit read: %v", customerID, err) + return + } + if !last.IsZero() && c.now().Sub(last) < healCooldown { + return // one restage per customer per 24 h — repeats surface as repeated events only + } + // THE R-39(a) GUARD — re-read the secret row immediately before acting. The derive above is a + // snapshot; an operator Re-issue may have staged a FRESH UNCONSUMED secret since (TOCTOU). + // SaveOneTimeSecret clobbers by design, so acting now would burn that fresh password. + info, err := c.store.GetOneTimeSecretInfo(customerID) + if err != nil { + c.logger.Printf("[WARN] offsite-delivery: %s: guard read: %v", customerID, err) + return + } + if info == nil || info.ConsumedAt.IsZero() { + c.logger.Printf("[INFO] offsite-delivery: %s: heal refused — secret row is now %s (R-39(a) guard: never restage over an unconsumed secret)", + customerID, secretShape(info)) + return + } + ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute) + defer cancel() + if err := c.reissuer.ReissueOffsiteForCustomer(ctx, customerID); err != nil { + c.logger.Printf("[WARN] offsite-delivery: %s: self-heal reissue failed: %v", customerID, err) + return + } + c.logger.Printf("[INFO] offsite-delivery: %s: self-heal restage fired (consumed_at %s, %d offbox-less reports since)", + customerID, status.Since.UTC().Format(time.RFC3339), status.ReportsSinceConsume) + details, _ := json.Marshal(map[string]any{ + "burned_consumed_at": status.Since.UTC().Format(time.RFC3339), + "reports_since_consume": status.ReportsSinceConsume, + }) + c.emit(customerID, eventCredentialRestaged, "warning", + "Offsite credential re-staged automatically: the previous one-time password was consumed but never applied (burned mid-delivery). The box picks the fresh password up on its next config refresh.", + string(details)) +} + +func secretShape(info *store.OneTimeSecretInfo) string { + if info == nil { + return "absent" + } + return "unconsumed (staged " + info.CreatedAt.UTC().Format(time.RFC3339) + ")" +} + +// emit saves the event (audit trail first) and then notifies — the OffsiteChecker convention: +// SaveEvent failure logs and SKIPS the notification (an email without its audit row lies). +func (c *OffsiteDeliveryChecker) emit(customerID, eventType, severity, message, details string) { + if _, err := c.store.SaveEvent(customerID, eventType, severity, message, details, "hub"); err != nil { + c.logger.Printf("[WARN] offsite-delivery: %s: save %s: %v", customerID, eventType, err) + return + } + c.logger.Printf("[INFO] offsite-delivery: %s: %s (%s)", customerID, eventType, severity) + if c.onEvent != nil { + c.onEvent(customerID, eventType, severity, message, details, "hub") + } +} diff --git a/hub/internal/monitor/offsite_delivery_test.go b/hub/internal/monitor/offsite_delivery_test.go new file mode 100644 index 0000000..bbdd1fb --- /dev/null +++ b/hub/internal/monitor/offsite_delivery_test.go @@ -0,0 +1,302 @@ +package monitor + +import ( + "context" + "encoding/json" + "io" + "log" + "path/filepath" + "sync" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// R-70/R-71c checker tests. Real store, fake reissuer (records calls AND mimics the production +// side effect — a restage IS a SaveOneTimeSecret — so a clobber is observable on the row itself), +// captured onEvent, injected clock. + +const ( + dtReportNoOffsite = `{"health":{"status":"ok"}}` + dtReportWithOffsite = `{"health":{"status":"ok"},"offsite":{"enabled":true}}` + dtOffsiteConfig = `{"offsite":{"enabled":true,"type":"shared","host":"h","user":"u","repo_path":"/home/felhom-repo","quota_gb":50}}` +) + +type fakeReissuer struct { + mu sync.Mutex + st *store.Store + calls []string +} + +func (f *fakeReissuer) ReissueOffsiteForCustomer(_ context.Context, customerID string) error { + f.mu.Lock() + f.calls = append(f.calls, customerID) + f.mu.Unlock() + // The production path's essential side effect: ReissueCredentials → SaveOneTimeSecret + // (last-write-wins clobber). Mimicked so the R-39(a) tests can observe what a wrongly-fired + // heal would DO to the row, not merely that it was called. + return f.st.SaveOneTimeSecret(customerID, "fresh-from-heal") +} + +func (f *fakeReissuer) count() int { + f.mu.Lock() + defer f.mu.Unlock() + return len(f.calls) +} + +type dtHarness struct { + st *store.Store + reissuer *fakeReissuer + checker *OffsiteDeliveryChecker + events *[]string // "type:severity" +} + +func newDTHarness(t *testing.T, withReissuer bool) dtHarness { + t.Helper() + st, err := store.New(filepath.Join(t.TempDir(), "d.db"), log.New(io.Discard, "", 0)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + if err := st.SaveCustomerConfig(&store.CustomerConfig{ + CustomerID: "c1", CustomerName: "C", Domain: "c1.hu", APIKey: "k", RetrievalPassword: "p", + ConfigJSON: dtOffsiteConfig, + }); err != nil { + t.Fatal(err) + } + events := &[]string{} + var mu sync.Mutex + onEvent := func(_, eventType, severity, _, _, _ string) { + mu.Lock() + *events = append(*events, eventType+":"+severity) + mu.Unlock() + } + var ri *fakeReissuer + var riIface OffsiteReissuer + if withReissuer { + ri = &fakeReissuer{st: st} + riIface = ri + } + c := NewOffsiteDeliveryChecker(st, riIface, onEvent, log.New(io.Discard, "", 0)) + return dtHarness{st: st, reissuer: ri, checker: c, events: events} +} + +// burnedFixture puts c1 into the F10 shape: consumed >1h ago, N offbox-less reports since. +func (h dtHarness) burnedFixture(t *testing.T, reports int) { + t.Helper() + if err := h.st.SaveOneTimeSecret("c1", "x"); err != nil { + t.Fatal(err) + } + consumed := time.Now().UTC().Add(-2 * time.Hour).Format("2006-01-02 15:04:05") + staged := time.Now().UTC().Add(-3 * time.Hour).Format("2006-01-02 15:04:05") + if err := h.st.SetOneTimeSecretTimesForTest("c1", staged, consumed); err != nil { + t.Fatal(err) + } + for i := 0; i < reports; i++ { + if err := h.st.SaveReport("c1", []byte(dtReportNoOffsite)); err != nil { + t.Fatal(err) + } + } +} + +func (h dtHarness) savedEvents(t *testing.T, eventType string) []store.Event { + t.Helper() + all, err := h.st.GetRecentEvents("c1", 50) + if err != nil { + t.Fatal(err) + } + var out []store.Event + for _, e := range all { + if e.EventType == eventType { + out = append(out, e) + } + } + return out +} + +// Scenario: the stuck event fires once with severity WARNING, and the durable 24h cooldown holds +// across a second pass (delete the LastEventAt guard in maybeEmitStuck → this fails with 2 events — +// the executable red-proof of the cooldown). +func TestDeliveryChecker_StuckEvent_WarningOncePer24h(t *testing.T) { + h := newDTHarness(t, false) + h.burnedFixture(t, 5) + + h.checker.Check() + h.checker.Check() // same tick shape again — cooldown must swallow it + + saved := h.savedEvents(t, "offsite_delivery_stuck") + if len(saved) != 1 { + t.Fatalf("stuck events = %d, want exactly 1 (24h per-customer cooldown)", len(saved)) + } + if saved[0].Severity != "warning" { + t.Fatalf("severity = %q, want warning (operator email tier; never info-silent, never critical)", saved[0].Severity) + } + var details map[string]any + if err := json.Unmarshal([]byte(saved[0].DetailsJSON), &details); err != nil || details["consumed_at"] == "" { + t.Fatalf("details must carry consumed_at (mióta), got %s err %v", saved[0].DetailsJSON, err) + } + if got := *h.events; len(got) != 1 || got[0] != "offsite_delivery_stuck:warning" { + t.Fatalf("dispatched = %v, want exactly [offsite_delivery_stuck:warning]", got) + } +} + +// Scenario: the R-71c self-heal fires EXACTLY once — one reissue call, the restaged event +// (warning), and the 24h rate limit blocks a re-trigger even when the shape recurs (delete the +// LastEventAt guard in maybeHeal → the second pass fires again and this fails — the rate-limit +// red-proof). +func TestDeliveryChecker_SelfHeal_FiresOnceAndRateLimits(t *testing.T) { + h := newDTHarness(t, true) + h.burnedFixture(t, 4) + + h.checker.Check() + if h.reissuer.count() != 1 { + t.Fatalf("reissue calls = %d, want 1", h.reissuer.count()) + } + restaged := h.savedEvents(t, "offsite_credential_restaged") + if len(restaged) != 1 || restaged[0].Severity != "warning" { + t.Fatalf("restaged events = %v, want exactly 1 with severity warning (the operator ALWAYS learns a heal fired)", restaged) + } + + // The shape recurs (box burned the fresh one too): back into consumed_awaiting_apply. + h.burnedFixture(t, 4) + h.checker.Check() + if h.reissuer.count() != 1 { + t.Fatalf("reissue calls after recurrence = %d, want STILL 1 (one restage per customer per 24h; repeats surface as events only)", h.reissuer.count()) + } +} + +// Not-enough-evidence gates: under healMinReports offbox-less reports → no heal; any offbox +// evidence since consume (regressed-apply shape) → no heal. The stuck EVENT still fires (age gate +// alone) — visibility never waits for the heal's stricter bar. +func TestDeliveryChecker_SelfHeal_EvidenceGates(t *testing.T) { + h := newDTHarness(t, true) + h.burnedFixture(t, 3) // 3 < healMinReports + h.checker.Check() + if h.reissuer.count() != 0 { + t.Fatalf("reissue with 3 reports = %d calls, want 0 (needs >= 4)", h.reissuer.count()) + } + if len(h.savedEvents(t, "offsite_delivery_stuck")) != 1 { + t.Fatal("stuck event must fire regardless of the heal's stricter evidence bar") + } + + // add offbox evidence AFTER consume, then more offbox-less reports — mixed history: no heal + if err := h.st.SaveReport("c1", []byte(dtReportWithOffsite)); err != nil { + t.Fatal(err) + } + if err := h.st.SaveReport("c1", []byte(dtReportNoOffsite)); err != nil { + t.Fatal(err) + } + h.checker.Check() + if h.reissuer.count() != 0 { + t.Fatalf("reissue on mixed offbox history = %d calls, want 0 (regressed-apply is the operator's call)", h.reissuer.count()) + } +} + +// THE CLOBBER RED-PROOF (R-39(a), mandatory per spec): an operator Re-issue lands between the +// checker's derive and its act (simulated via the onEvent hook, which runs after the stuck event +// and before maybeHeal). The act-time guard re-reads the row, finds it UNCONSUMED, and refuses — +// zero reissue calls, row untouched. Remove the `info.ConsumedAt.IsZero()` refusal in maybeHeal → +// the fake fires SaveOneTimeSecret and this test FAILS on both assertions (calls=1, row clobbered) +// — proving the fixture would have been clobbered. +func TestDeliveryChecker_R39aGuard_NeverRestagesOverUnconsumed(t *testing.T) { + h := newDTHarness(t, true) + h.burnedFixture(t, 4) + + // The TOCTOU: the moment the stuck event dispatches, the "operator" stages a fresh secret. + operatorStaged := "2026-07-23 12:00:00" + *h.events = nil + base := h.checker.onEvent + h.checker.onEvent = func(cid, et, sev, msg, det, src string) { + if et == "offsite_delivery_stuck" { + if err := h.st.SaveOneTimeSecret("c1", "operator-fresh"); err != nil { + t.Errorf("mid-tick stage: %v", err) + } + if err := h.st.SetOneTimeSecretTimesForTest("c1", operatorStaged, ""); err != nil { + t.Errorf("mid-tick stamp: %v", err) + } + } + base(cid, et, sev, msg, det, src) + } + + h.checker.Check() + + if h.reissuer.count() != 0 { + t.Fatalf("reissue calls = %d, want 0 — the R-39(a) guard must refuse over an unconsumed secret", h.reissuer.count()) + } + info, err := h.st.GetOneTimeSecretInfo("c1") + if err != nil || info == nil { + t.Fatalf("secret row: %v / %v", info, err) + } + if !info.ConsumedAt.IsZero() || !info.CreatedAt.Equal(time.Date(2026, 7, 23, 12, 0, 0, 0, time.UTC)) { + t.Fatalf("the operator's fresh secret was CLOBBERED (created_at=%v consumed_at=%v) — R-39(a) violated", info.CreatedAt, info.ConsumedAt) + } + if len(h.savedEvents(t, "offsite_credential_restaged")) != 0 { + t.Fatal("no restaged event may exist for a refused heal") + } +} + +// The demo-felhom live shape, full Check(): applied + stale unconsumed staged secret → ZERO events, +// ZERO reissue calls, row untouched. Precedence (applied wins) is the first line of defense; the +// R-39(a) guard is the second. +func TestDeliveryChecker_AppliedWithStaleStaged_Untouched(t *testing.T) { + h := newDTHarness(t, true) + if err := h.st.SaveReport("c1", []byte(dtReportWithOffsite)); err != nil { + t.Fatal(err) + } + if err := h.st.SaveOneTimeSecret("c1", "stale"); err != nil { + t.Fatal(err) + } + if err := h.st.SetOneTimeSecretTimesForTest("c1", "2026-07-21 08:29:29", ""); err != nil { + t.Fatal(err) + } + + h.checker.Check() + + if h.reissuer.count() != 0 { + t.Fatalf("reissue calls = %d, want 0 (demo-felhom shape is healthy)", h.reissuer.count()) + } + if len(*h.events) != 0 { + t.Fatalf("events = %v, want none", *h.events) + } + info, _ := h.st.GetOneTimeSecretInfo("c1") + if info == nil || !info.ConsumedAt.IsZero() || !info.CreatedAt.Equal(time.Date(2026, 7, 21, 8, 29, 29, 0, time.UTC)) { + t.Fatalf("fixture row mutated: %+v — the live specimen must survive the checker untouched", info) + } +} + +// Guard rails: young consumed state (inside stuckAfter) is silent; offsite-disabled and +// non-active customers are skipped entirely. +func TestDeliveryChecker_QuietShapes(t *testing.T) { + h := newDTHarness(t, true) + // consumed 5 minutes ago — normal convergence window + if err := h.st.SaveOneTimeSecret("c1", "x"); err != nil { + t.Fatal(err) + } + recent := time.Now().UTC().Add(-5 * time.Minute).Format("2006-01-02 15:04:05") + if err := h.st.SetOneTimeSecretTimesForTest("c1", recent, recent); err != nil { + t.Fatal(err) + } + for i := 0; i < 5; i++ { + if err := h.st.SaveReport("c1", []byte(dtReportNoOffsite)); err != nil { + t.Fatal(err) + } + } + h.checker.Check() + if len(*h.events) != 0 || h.reissuer.count() != 0 { + t.Fatalf("young consumed state must be silent, got events=%v calls=%d", *h.events, h.reissuer.count()) + } + + // offsite disabled → skipped even in a stuck-looking shape + if err := h.st.SaveCustomerConfig(&store.CustomerConfig{ + CustomerID: "c1", CustomerName: "C", Domain: "c1.hu", APIKey: "k", RetrievalPassword: "p", + ConfigJSON: `{"offsite":{"enabled":false}}`, + }); err != nil { + t.Fatal(err) + } + h.burnedFixture(t, 5) + h.checker.Check() + if len(*h.events) != 0 || h.reissuer.count() != 0 { + t.Fatalf("disabled offsite must be skipped, got events=%v calls=%d", *h.events, h.reissuer.count()) + } +} diff --git a/hub/internal/notify/dispatcher_offsite_delivery_test.go b/hub/internal/notify/dispatcher_offsite_delivery_test.go new file mode 100644 index 0000000..6351c44 --- /dev/null +++ b/hub/internal/notify/dispatcher_offsite_delivery_test.go @@ -0,0 +1,44 @@ +package notify + +import ( + "io" + "log" + "testing" +) + +// R-70/R-71c severity contract, alongside the v0.71.0 guards: the two new detector events carry +// `warning` — they route to the OPERATOR through the existing severity gate, with NO widening of +// severityNotifies and NO customer email (neither type is in any default enabled_events set, and +// neither has a customerMessages entry — operator-only until the mechanism has history). +func TestOffsiteDeliveryEvents_WarningRoutesToOperatorOnly(t *testing.T) { + st := newDispStore(t) + d := NewDispatcher(st, "test-key", "hub@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + mails := captureSeam(d) + + d.ProcessEvent("c1", "offsite_delivery_stuck", "warning", "stuck", "{}", "hub") + d.ProcessEvent("c1", "offsite_credential_restaged", "warning", "restaged", "{}", "hub") + + op := mailsFor(*mails, "op@felhom.eu") + if len(op) != 2 { + t.Fatalf("operator mails = %d, want 2 (warning must notify the operator)", len(op)) + } + // customer leg: c1 has no notification prefs row → nothing may go anywhere else + if len(*mails) != 2 { + t.Fatalf("total mails = %d, want 2 — the detector events are operator-only", len(*mails)) + } +} + +// The deliberate decision, as an explicit test beside TestRecovery_SeverityStaysInfo: had these +// events shipped as `info`, the dispatcher would silently drop them — the exact invisibility R-70 +// exists to kill. This pins the severity choice against a future "quiet them down" edit. +func TestOffsiteDeliveryEvents_InfoWouldBeSilent(t *testing.T) { + st := newDispStore(t) + d := NewDispatcher(st, "test-key", "hub@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0)) + mails := captureSeam(d) + + d.ProcessEvent("c1", "offsite_delivery_stuck", "info", "stuck", "{}", "hub") + + if len(*mails) != 0 { + t.Fatalf("mails = %d, want 0 — info stays silent (severityNotifies untouched); the checker MUST emit warning", len(*mails)) + } +} diff --git a/hub/internal/offsite/delivery.go b/hub/internal/offsite/delivery.go new file mode 100644 index 0000000..2c2aa70 --- /dev/null +++ b/hub/internal/offsite/delivery.go @@ -0,0 +1,97 @@ +package offsite + +import ( + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// R-70 delivery-state detector — ONE implementation, all consumers read it (customer card, the +// monitor checker's stuck event + R-71c self-heal, and any future surface). Derived from the two +// signals the hub already holds and previously read nowhere: +// - the one_time_secrets row (exists? consumed_at?) — the delivery ledger; +// - report offsite-object presence — the box's own testimony that an offbox target is applied +// (the report builder attaches `offsite` only when a target is configured). +// +// Origin: DIAG-f10-demo-hp-offsite-2026-07-23 — demo-hp sat 2 days in consumed_awaiting_apply +// (the burned-credential shape) while the operator card said "Provisioned" from static copy. + +// DeliveryState is the customer's offsite last-mile state. +type DeliveryState string + +const ( + // DeliveryApplied — the latest report carries an offsite status object: the tier is live on + // the box. Wins over every secret-row shape (precedence rule; an applied box with a stale + // staged secret is APPLIED — see StaleStagedSince). + DeliveryApplied DeliveryState = "applied" + // DeliveryConsumedAwaitingApply — the one-time password was consumed but no report since has + // ever shown an offbox target: the burned-credential shape (normal for seconds, wrong for + // hours; the F10 box sat here for 2 days). + DeliveryConsumedAwaitingApply DeliveryState = "consumed_awaiting_apply" + // DeliveryStagedAwaitingConsume — a secret is staged and not yet consumed: delivery pending + // (normal for minutes while the box re-pulls config, wrong for days). + DeliveryStagedAwaitingConsume DeliveryState = "staged_awaiting_consume" + // DeliveryNoSecret — offsite is enabled in the config but no secret row exists at all: a + // mis-state (e.g. a RESET purged the row without deprovisioning). Rendered needs-attention. + DeliveryNoSecret DeliveryState = "no_secret" +) + +// DeliveryStatus is the derived last-mile state plus the timestamps every consumer must surface +// ("mióta" — R-70's whole point is that nothing about this plumbing stays invisible). +type DeliveryStatus struct { + State DeliveryState + // Since anchors the state's age: consumed_at for consumed_awaiting_apply, created_at for + // staged_awaiting_consume, the latest report time for applied, zero for no_secret. + Since time.Time + // StaleStagedSince is non-zero ONLY in the applied+stale-staged edge (the demo-felhom shape): + // the box is applied via an earlier generation while an unconsumed secret sits staged. The + // caller renders an info line; the self-heal must NEVER touch this shape (R-39(a)). + StaleStagedSince time.Time + // ReportsSinceConsume / OffsiteReportsSinceConsume count reports received after consumed_at + // (consumed_awaiting_apply only; zero otherwise). The R-71c trigger requires + // ReportsSinceConsume >= N with OffsiteReportsSinceConsume == 0 — "consecutive without". + ReportsSinceConsume int + OffsiteReportsSinceConsume int +} + +// DeliveryStateFor derives the customer's offsite delivery state. Callers gate on the descriptor +// (offsite enabled) themselves — this function only reads the ledger and the reports; age math is +// the caller's (against its own clock seam). +func DeliveryStateFor(st *store.Store, customerID string) (DeliveryStatus, error) { + found, receivedAt, hasOffsite, err := st.LatestReportOffsitePresence(customerID) + if err != nil { + return DeliveryStatus{}, err + } + secret, err := st.GetOneTimeSecretInfo(customerID) + if err != nil { + return DeliveryStatus{}, err + } + + // Precedence: applied wins. The box's own report is the strongest evidence there is. + if found && hasOffsite { + status := DeliveryStatus{State: DeliveryApplied, Since: receivedAt} + if secret != nil && secret.ConsumedAt.IsZero() { + status.StaleStagedSince = secret.CreatedAt // demo-felhom shape: applied + stale staged + } + return status, nil + } + + if secret == nil { + return DeliveryStatus{State: DeliveryNoSecret}, nil + } + + if !secret.ConsumedAt.IsZero() { + total, withOffsite, err := st.CountReportsOffsiteSince(customerID, secret.ConsumedAt) + if err != nil { + return DeliveryStatus{}, err + } + return DeliveryStatus{ + State: DeliveryConsumedAwaitingApply, + Since: secret.ConsumedAt, + ReportsSinceConsume: total, + OffsiteReportsSinceConsume: withOffsite, + }, nil + } + + return DeliveryStatus{State: DeliveryStagedAwaitingConsume, Since: secret.CreatedAt}, nil +} diff --git a/hub/internal/offsite/delivery_test.go b/hub/internal/offsite/delivery_test.go new file mode 100644 index 0000000..866be6d --- /dev/null +++ b/hub/internal/offsite/delivery_test.go @@ -0,0 +1,172 @@ +package offsite + +import ( + "io" + "log" + "path/filepath" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// R-70 detector tests — four states + the applied-wins precedence + the demo-felhom +// applied+stale-staged edge. Fixtures are the real store (t.TempDir SQLite), timestamps +// controlled via the test-only back-dater (no sleeps). + +func newDeliveryStore(t *testing.T) *store.Store { + t.Helper() + st, err := store.New(filepath.Join(t.TempDir(), "d.db"), log.New(io.Discard, "", 0)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + return st +} + +const ( + reportWithOffsite = `{"health":{"status":"ok"},"offsite":{"enabled":true,"escrow_state":"escrowed"}}` + reportNoOffsite = `{"health":{"status":"ok"}}` +) + +func TestDeliveryState_Applied(t *testing.T) { + st := newDeliveryStore(t) + if err := st.SaveReport("c1", []byte(reportWithOffsite)); err != nil { + t.Fatal(err) + } + if err := st.SaveOneTimeSecret("c1", "x"); err != nil { + t.Fatal(err) + } + if _, err := st.ConsumeOneTimeSecret("c1"); err != nil { + t.Fatal(err) + } + got, err := DeliveryStateFor(st, "c1") + if err != nil { + t.Fatal(err) + } + if got.State != DeliveryApplied { + t.Fatalf("state = %s, want applied", got.State) + } + if !got.StaleStagedSince.IsZero() { + t.Fatalf("consumed secret must not flag stale-staged, got %v", got.StaleStagedSince) + } + if got.Since.IsZero() { + t.Fatal("applied state must carry the report timestamp (mióta)") + } +} + +// The demo-felhom shape: applied via an earlier generation + an unconsumed secret staged since. +// Precedence: applied WINS, and the stale staged secret becomes a VISIBLE info flag — never a +// state downgrade, never a heal trigger. +func TestDeliveryState_AppliedWithStaleStaged(t *testing.T) { + st := newDeliveryStore(t) + if err := st.SaveReport("c1", []byte(reportWithOffsite)); err != nil { + t.Fatal(err) + } + if err := st.SaveOneTimeSecret("c1", "stale"); err != nil { + t.Fatal(err) + } + if err := st.SetOneTimeSecretTimesForTest("c1", "2026-07-21 08:29:29", ""); err != nil { + t.Fatal(err) + } + got, err := DeliveryStateFor(st, "c1") + if err != nil { + t.Fatal(err) + } + if got.State != DeliveryApplied { + t.Fatalf("precedence broken: state = %s, want applied (applied wins over staged)", got.State) + } + want := time.Date(2026, 7, 21, 8, 29, 29, 0, time.UTC) + if !got.StaleStagedSince.Equal(want) { + t.Fatalf("StaleStagedSince = %v, want %v (the invisible plumbing must be visible)", got.StaleStagedSince, want) + } +} + +// The burned-credential shape (F10): consumed, and every report since carries no offbox. +func TestDeliveryState_ConsumedAwaitingApply(t *testing.T) { + st := newDeliveryStore(t) + if err := st.SaveOneTimeSecret("c1", "x"); err != nil { + t.Fatal(err) + } + // consumed 2h ago; the reports below (received "now") all come AFTER it + if err := st.SetOneTimeSecretTimesForTest("c1", "2026-01-01 00:00:00", "2026-01-01 00:03:00"); err != nil { + t.Fatal(err) + } + for i := 0; i < 5; i++ { + if err := st.SaveReport("c1", []byte(reportNoOffsite)); err != nil { + t.Fatal(err) + } + } + got, err := DeliveryStateFor(st, "c1") + if err != nil { + t.Fatal(err) + } + if got.State != DeliveryConsumedAwaitingApply { + t.Fatalf("state = %s, want consumed_awaiting_apply", got.State) + } + if got.ReportsSinceConsume != 5 || got.OffsiteReportsSinceConsume != 0 { + t.Fatalf("counts = %d/%d, want 5/0", got.ReportsSinceConsume, got.OffsiteReportsSinceConsume) + } + if want := time.Date(2026, 1, 1, 0, 3, 0, 0, time.UTC); !got.Since.Equal(want) { + t.Fatalf("Since = %v, want consumed_at %v", got.Since, want) + } +} + +// The since-filter: reports received BEFORE consumed_at must not count toward the trigger. +func TestDeliveryState_ReportsBeforeConsumeNotCounted(t *testing.T) { + st := newDeliveryStore(t) + for i := 0; i < 3; i++ { + if err := st.SaveReport("c1", []byte(reportNoOffsite)); err != nil { + t.Fatal(err) + } + } + if err := st.SaveOneTimeSecret("c1", "x"); err != nil { + t.Fatal(err) + } + // consumed FAR in the future relative to the rows above → zero reports since + if err := st.SetOneTimeSecretTimesForTest("c1", "2099-01-01 00:00:00", "2099-01-01 00:00:01"); err != nil { + t.Fatal(err) + } + got, err := DeliveryStateFor(st, "c1") + if err != nil { + t.Fatal(err) + } + if got.State != DeliveryConsumedAwaitingApply || got.ReportsSinceConsume != 0 { + t.Fatalf("state/count = %s/%d, want consumed_awaiting_apply/0 (pre-consume reports must not count)", + got.State, got.ReportsSinceConsume) + } +} + +func TestDeliveryState_StagedAwaitingConsume(t *testing.T) { + st := newDeliveryStore(t) + if err := st.SaveReport("c1", []byte(reportNoOffsite)); err != nil { + t.Fatal(err) + } + if err := st.SaveOneTimeSecret("c1", "x"); err != nil { + t.Fatal(err) + } + got, err := DeliveryStateFor(st, "c1") + if err != nil { + t.Fatal(err) + } + if got.State != DeliveryStagedAwaitingConsume { + t.Fatalf("state = %s, want staged_awaiting_consume", got.State) + } + if got.Since.IsZero() { + t.Fatal("staged state must carry created_at (mióta)") + } +} + +func TestDeliveryState_NoSecret(t *testing.T) { + st := newDeliveryStore(t) + if err := st.SaveReport("c1", []byte(reportNoOffsite)); err != nil { + t.Fatal(err) + } + got, err := DeliveryStateFor(st, "c1") + if err != nil { + t.Fatal(err) + } + if got.State != DeliveryNoSecret { + t.Fatalf("state = %s, want no_secret (mis-state must render needs-attention, never applied)", got.State) + } +} diff --git a/hub/internal/store/store.go b/hub/internal/store/store.go index 9113c21..dbbb4a1 100644 --- a/hub/internal/store/store.go +++ b/hub/internal/store/store.go @@ -913,6 +913,77 @@ func (s *Store) SaveReport(customerID string, reportJSON []byte) error { return err } +// reportOffsitePresence is the minimal parse for "does this controller report carry an offsite +// status object" — the R-70 delivery-state signal. The report builder attaches `offsite` only when +// the box actually has an offbox target configured, so presence == applied-on-the-box. +type reportOffsitePresence struct { + Offsite json.RawMessage `json:"offsite"` +} + +func reportHasOffsite(reportJSON string) bool { + var p reportOffsitePresence + if err := json.Unmarshal([]byte(reportJSON), &p); err != nil { + return false // unparseable report → no offsite evidence + } + return len(p.Offsite) > 0 && string(p.Offsite) != "null" +} + +// LatestReportOffsitePresence reports whether the customer's most recent controller report exists +// and whether it carries an offsite status object (R-70 detector input). +func (s *Store) LatestReportOffsitePresence(customerID string) (found bool, receivedAt time.Time, hasOffsite bool, err error) { + var recv, reportJSON string + err = s.db.QueryRow(`SELECT received_at, report_json FROM reports WHERE customer_id = ? ORDER BY id DESC LIMIT 1`, + customerID).Scan(&recv, &reportJSON) + if err == sql.ErrNoRows { + return false, time.Time{}, false, nil + } + if err != nil { + return false, time.Time{}, false, err + } + return true, parseSQLiteTime(recv), reportHasOffsite(reportJSON), nil +} + +// CountReportsOffsiteSince counts the customer's controller reports received strictly after `since` +// (UTC) and how many of them carry an offsite status object (R-70 detector input: "N consecutive +// reports since consume without offbox" == total>0 && withOffsite==0). Capped at 500 rows per call — +// far beyond any detector threshold; the cap only bounds memory. +func (s *Store) CountReportsOffsiteSince(customerID string, since time.Time) (total, withOffsite int, err error) { + rows, err := s.db.Query(`SELECT report_json FROM reports WHERE customer_id = ? AND received_at > ? ORDER BY id LIMIT 500`, + customerID, since.UTC().Format("2006-01-02 15:04:05")) + if err != nil { + return 0, 0, err + } + defer rows.Close() + for rows.Next() { + var reportJSON string + if err := rows.Scan(&reportJSON); err != nil { + return 0, 0, err + } + total++ + if reportHasOffsite(reportJSON) { + withOffsite++ + } + } + return total, withOffsite, rows.Err() +} + +// LastEventAt returns the created_at of the most recent event of the given type for a customer +// (zero time when none). Durable across hub restarts — used as the cooldown/rate-limit source for +// hub-emitted detector events (R-70/R-71c: a repeating pattern must surface as repeating events on +// a bounded cadence, never as a silent retry loop OR a restart-reset flood). +func (s *Store) LastEventAt(customerID, eventType string) (time.Time, error) { + var createdAt string + err := s.db.QueryRow(`SELECT created_at FROM events WHERE customer_id = ? AND event_type = ? ORDER BY id DESC LIMIT 1`, + customerID, eventType).Scan(&createdAt) + if err == sql.ErrNoRows { + return time.Time{}, nil + } + if err != nil { + return time.Time{}, err + } + return parseSQLiteTime(createdAt), nil +} + // GetCustomers returns the latest report summary for each customer. func (s *Store) GetCustomers() ([]CustomerSummary, error) { rows, err := s.db.Query(` @@ -1249,6 +1320,47 @@ func (s *Store) ConsumeOneTimeSecret(customerID string) (string, error) { return value, nil } +// OneTimeSecretInfo is the delivery-state metadata of a customer's one-time offsite secret — +// timestamps ONLY, the value column is deliberately never selected (R-70 detector input; the +// customer-scoped sibling of PBSDRHealStates' SecretUnconsumedFor). +type OneTimeSecretInfo struct { + CustomerID string + CreatedAt time.Time + ConsumedAt time.Time // zero = staged, not yet consumed +} + +// GetOneTimeSecretInfo returns the timestamps of a customer's one-time offsite secret row, or nil +// when none exists. Never reads the value. +func (s *Store) GetOneTimeSecretInfo(customerID string) (*OneTimeSecretInfo, error) { + var createdAt string + var consumedAt sql.NullString + err := s.db.QueryRow(`SELECT created_at, consumed_at FROM one_time_secrets WHERE customer_id = ?`, + customerID).Scan(&createdAt, &consumedAt) + if err == sql.ErrNoRows { + return nil, nil + } + if err != nil { + return nil, err + } + info := &OneTimeSecretInfo{CustomerID: customerID, CreatedAt: parseSQLiteTime(createdAt)} + if consumedAt.Valid { + info.ConsumedAt = parseSQLiteTime(consumedAt.String) + } + return info, nil +} + +// SetOneTimeSecretTimesForTest back-dates a one-time secret's timestamps (SQLite datetime strings; +// consumedAt "" leaves it NULL) so grace/age behavior is testable without sleeping. TEST-ONLY — +// mirrors SetHostPBSSecretCreatedAtForTest. +func (s *Store) SetOneTimeSecretTimesForTest(customerID, createdAt, consumedAt string) error { + if consumedAt == "" { + _, err := s.db.Exec(`UPDATE one_time_secrets SET created_at = ?, consumed_at = NULL WHERE customer_id = ?`, createdAt, customerID) + return err + } + _, err := s.db.Exec(`UPDATE one_time_secrets SET created_at = ?, consumed_at = ? WHERE customer_id = ?`, createdAt, consumedAt, customerID) + return err +} + // ListCustomerConfigs returns all customer configurations ordered by ID. func (s *Store) ListCustomerConfigs() ([]CustomerConfig, error) { rows, err := s.db.Query(` diff --git a/hub/internal/web/configs.go b/hub/internal/web/configs.go index 34950ef..6c0b3b8 100644 --- a/hub/internal/web/configs.go +++ b/hub/internal/web/configs.go @@ -518,6 +518,80 @@ type configFormView struct { Error string CSRFField template.HTML PBSDR pbsDRView + Delivery *deliveryView +} + +// deliveryView is the R-70 customer-card rendering of offsite.DeliveryStateFor — the hub's real +// delivery knowledge replacing the static "delivered to the controller once" copy that let a +// burned credential hide for 2 days (DIAG-f10-demo-hp-offsite-2026-07-23). All strings are +// precomputed operator-tier English (this page's existing language). +type deliveryView struct { + State string // offsite.DeliveryState (template branch key + test anchor) + Badge string // short badge text + BadgeClass string // n-ok | n-warn | n-neutral + Line string // the state sentence, with age ("mióta" — every state carries its timestamp) + StaleLine string // non-empty ONLY for applied+stale-staged (the demo-felhom shape) +} + +// agoHuman renders a duration as a coarse operator-friendly age. +func agoHuman(d time.Duration) string { + switch { + case d < time.Minute: + return "under a minute" + case d < time.Hour: + return fmt.Sprintf("%d min", int(d.Minutes())) + case d < 48*time.Hour: + return fmt.Sprintf("%.1f h", d.Hours()) + default: + return fmt.Sprintf("%d days", int(d.Hours()/24)) + } +} + +// deliveryViewFor derives the card view; nil for brand-new configs (nothing staged yet) or on a +// detector error (the card then simply omits the state line — never a fabricated one). +func (s *Server) deliveryViewFor(customerID string) *deliveryView { + if customerID == "" { + return nil + } + now := time.Now() + status, err := offsite.DeliveryStateFor(s.store, customerID) + if err != nil { + s.logger.Printf("[WARN] delivery view %s: %v", customerID, err) + return nil + } + v := &deliveryView{State: string(status.State)} + age := agoHuman(now.Sub(status.Since)) + switch status.State { + case offsite.DeliveryApplied: + v.Badge, v.BadgeClass = "applied", "n-ok" + v.Line = "offsite active on the box (last report " + age + " ago)" + if !status.StaleStagedSince.IsZero() { + v.StaleLine = fmt.Sprintf("Note: an unconsumed one-time secret has been staged since %s (%s ago) — superseded by the working install (key-auth-first never consumes); harmless, replaced by the next re-issue.", + status.StaleStagedSince.UTC().Format("2006-01-02 15:04 UTC"), agoHuman(now.Sub(status.StaleStagedSince))) + } + case offsite.DeliveryConsumedAwaitingApply: + // Amber past 30 min: consume→apply is a seconds-scale hop; half an hour of it is the + // burned-credential shape taking form (the monitor turns it into an event at 1 h). + v.Badge = "consumed" + if now.Sub(status.Since) > 30*time.Minute { + v.BadgeClass = "n-warn" + v.Line = fmt.Sprintf("password consumed %s ago and the box still reports no offsite target — likely burned mid-apply; Re-issue delivers a fresh one", age) + } else { + v.BadgeClass = "n-neutral" + v.Line = "password consumed — apply in progress (" + age + ")" + } + case offsite.DeliveryStagedAwaitingConsume: + v.Badge, v.BadgeClass = "staged", "n-neutral" + v.Line = "one-time password staged " + age + " ago — the box consumes it on its next config refresh" + if now.Sub(status.Since) > 30*time.Minute { + v.BadgeClass = "n-warn" + v.Line = "one-time password staged " + age + " ago and NOT yet consumed — the box has not re-pulled its config (is it reporting?)" + } + default: // DeliveryNoSecret + v.Badge, v.BadgeClass = "missing", "n-warn" + v.Line = "offsite is enabled but no credential is staged — needs attention (Re-issue stages a fresh one)" + } + return v } // configFormData assembles the config form's view model. overrides carries SUBMITTED form values @@ -538,6 +612,7 @@ func (s *Server) configFormData(r *http.Request, isNew bool, cfg *store.Customer Error: errMsg, CSRFField: s.csrfField(r), PBSDR: s.pbsDRViewFor(cfg.CustomerID, cfg.DRTier), + Delivery: s.deliveryViewFor(cfg.CustomerID), } } diff --git a/hub/internal/web/configs_delivery_render_test.go b/hub/internal/web/configs_delivery_render_test.go new file mode 100644 index 0000000..5850582 --- /dev/null +++ b/hub/internal/web/configs_delivery_render_test.go @@ -0,0 +1,122 @@ +package web + +import ( + "strings" + "testing" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// R-70 card render tests — one per template branch (the v0.70.1 seam-wiring lesson: a +// conditional affordance ships with a render test per branch of its gate; handler tests prove +// nothing about reachability). The card replaces the static "delivered to the controller once" +// copy, so the tests also pin that the old static claim is GONE. + +func renderConfigFormWith(t *testing.T, delivery *deliveryView) string { + t.Helper() + s, _ := newTestServer(t) + data := configFormView{ + Config: &store.CustomerConfig{CustomerID: "c1"}, + Overrides: map[string]interface{}{ + "offsite": map[string]interface{}{ + "enabled": true, "type": "shared", "host": "u-sub1.example.de", + "user": "u-sub1", "repo_path": "/home/felhom-repo", + }, + }, + ActiveNav: "configs", + Delivery: delivery, + } + var b strings.Builder + if err := s.templates.ExecuteTemplate(&b, "config_form.html", data); err != nil { + t.Fatalf("render: %v", err) + } + return b.String() +} + +func TestDeliveryCard_AppliedRenders(t *testing.T) { + out := renderConfigFormWith(t, &deliveryView{ + State: "applied", Badge: "applied", BadgeClass: "n-ok", + Line: "offsite active on the box (last report 2 min ago)", + }) + if !strings.Contains(out, `data-delivery-state="applied"`) || !strings.Contains(out, "offsite active on the box") { + t.Fatalf("applied state not rendered:\n%s", out) + } + if !strings.Contains(out, `class="n n-ok"`) { + t.Fatal("applied badge must use n-ok") + } + if strings.Contains(out, "the transient password is delivered to the controller once") { + t.Fatal("the old static 'delivered once' claim must be GONE — it hid a burned credential for 2 days (DIAG-f10)") + } +} + +func TestDeliveryCard_ConsumedAwaitingApplyAmberRenders(t *testing.T) { + out := renderConfigFormWith(t, &deliveryView{ + State: "consumed_awaiting_apply", Badge: "consumed", BadgeClass: "n-warn", + Line: "password consumed 2.0 h ago and the box still reports no offsite target — likely burned mid-apply; Re-issue delivers a fresh one", + }) + if !strings.Contains(out, `data-delivery-state="consumed_awaiting_apply"`) || + !strings.Contains(out, "likely burned mid-apply") || !strings.Contains(out, `class="n n-warn"`) { + t.Fatalf("amber consumed_awaiting_apply state not rendered:\n%s", out) + } +} + +func TestDeliveryCard_StagedRenders(t *testing.T) { + out := renderConfigFormWith(t, &deliveryView{ + State: "staged_awaiting_consume", Badge: "staged", BadgeClass: "n-neutral", + Line: "one-time password staged 3 min ago — the box consumes it on its next config refresh", + }) + if !strings.Contains(out, `data-delivery-state="staged_awaiting_consume"`) || + !strings.Contains(out, "the box consumes it on its next config refresh") { + t.Fatalf("staged state not rendered:\n%s", out) + } +} + +// The demo-felhom shape: applied + the stale-staged info line — BOTH must render. +func TestDeliveryCard_AppliedWithStaleStagedInfoLine(t *testing.T) { + out := renderConfigFormWith(t, &deliveryView{ + State: "applied", Badge: "applied", BadgeClass: "n-ok", + Line: "offsite active on the box (last report 2 min ago)", + StaleLine: "Note: an unconsumed one-time secret has been staged since 2026-07-21 08:29 UTC (2 days ago) — superseded by the working install (key-auth-first never consumes); harmless, replaced by the next re-issue.", + }) + if !strings.Contains(out, `data-delivery-state="applied"`) || + !strings.Contains(out, "an unconsumed one-time secret has been staged since 2026-07-21") { + t.Fatalf("applied + stale-staged info line not rendered:\n%s", out) + } +} + +// Nil Delivery (brand-new config / detector error): the card is simply absent — never a +// fabricated state line. +func TestDeliveryCard_NilDeliveryOmitted(t *testing.T) { + out := renderConfigFormWith(t, nil) + if strings.Contains(out, "data-delivery-state") { + t.Fatalf("nil Delivery must render no state line:\n%s", out) + } + // the target line survives (it is descriptor truth, not delivery state) + if !strings.Contains(out, "u-sub1@u-sub1.example.de:/home/felhom-repo") { + t.Fatal("provisioned target line must still render") + } +} + +// deliveryViewFor derives from the real detector: the burned shape past 30 min renders amber. +func TestDeliveryViewFor_BurnedShapeGoesAmber(t *testing.T) { + s, st := newTestServer(t) + if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c9", APIKey: "k", RetrievalPassword: "p"}); err != nil { + t.Fatal(err) + } + if err := st.SaveOneTimeSecret("c9", "x"); err != nil { + t.Fatal(err) + } + if err := st.SetOneTimeSecretTimesForTest("c9", "2026-01-01 00:00:00", "2026-01-01 00:03:00"); err != nil { + t.Fatal(err) + } + if err := st.SaveReport("c9", []byte(`{"health":{"status":"ok"}}`)); err != nil { + t.Fatal(err) + } + v := s.deliveryViewFor("c9") + if v == nil || v.State != "consumed_awaiting_apply" || v.BadgeClass != "n-warn" { + t.Fatalf("view = %+v, want consumed_awaiting_apply with n-warn (amber past 30 min)", v) + } + if !strings.Contains(v.Line, "Re-issue") { + t.Fatalf("amber line must point at the remedy, got %q", v.Line) + } +} diff --git a/hub/internal/web/templates/config_form_body.html b/hub/internal/web/templates/config_form_body.html index a6a4aab..94edc66 100644 --- a/hub/internal/web/templates/config_form_body.html +++ b/hub/internal/web/templates/config_form_body.html @@ -117,7 +117,14 @@ {{with .Overrides}}{{with index . "offsite"}}{{if index . "host"}} -

Provisioned: {{index . "user"}}@{{index . "host"}}:{{index . "repo_path"}} — the transient password is delivered to the controller once (never shown here).

+

Provisioned: {{index . "user"}}@{{index . "host"}}:{{index . "repo_path"}} (the transient password is never shown here).

+ + {{with $.Delivery}} +

{{.Badge}} {{.Line}}

+ {{if .StaleLine}}

{{.StaleLine}}

{{end}} + {{end}}