hub v0.72.0 — R-70 + R-71c: offsite delivery-state detector, card, stuck event, R-39(a)-guarded self-heal restage
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NKSN3gSg4TKVBBqkwW2djR
This commit is contained in:
@@ -99,6 +99,9 @@
|
||||
|---|---|---|---|---|
|
||||
| `tenantsync.Client` (`Provision`/`Reissue`/`Fingerprint`) | hub/internal/tenantsync/client.go | `(ctx, customerID) (*Result, error)` | ep0 per-customer PBS tenancy over the pinned-SSH forced-command channel (the wgsync twin) | `Result.TokenSecret` is transient custody → `SaveHostPBSSecret` immediately, never log the struct. Error paths NEVER embed stdout (the secret channel) — do not "improve" diagnostics by quoting the response. `ErrTokenExists` is typed: provision refuses an existing token; re-issue is the explicit path. |
|
||||
| `(*Store).SaveHostPBSSecret` / `ConsumeHostPBSSecret` | hub/internal/store/pbsdr.go | `(hostID, value)` / `(hostID) (string, error)` | HOST-scoped consume-once secret (the one_time_secrets host twin) | Same-tx mark-consumed; re-save resets consumption (re-issue supersedes). The agent consumes via `POST /api/v1/hosts/{id}/pbs/consume-token` (hub/internal/api/pbsdr.go). |
|
||||
| `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
|
||||
| `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). |
|
||||
| `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
|
||||
| `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
|
||||
| `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
|
||||
| `cfg.DRTier` + offsite coupling | hub/internal/store/store.go (CustomerConfig), hub/internal/web/configs.go (applyOffsite guard) | bool | Per-customer DR-tier flag: new-customer default ON (handleConfigNewForm); offsite REFUSED without it (exact F-6 message) | One-time migration backfill initializes legacy rows from descriptor reality — never re-runs (opt-outs survive re-open; store test pins it). Form field `dr_tier` (formBool helper). |
|
||||
|
||||
@@ -81,8 +81,8 @@
|
||||
| R-67 | **The NAS share appears in FileBrowser — browse what you mounted.** | S | **SHIPPED (controller v0.160.0, 2026-07-22)** | Origin: the R-64 pairing drill — the share said „Elérhető" and the customer had no way to BROWSE it (FileBrowser synced drives only). **Couples to R-64: browsing was its missing UX half.** A registered network storage now binds its share ROOT into FileBrowser (`/mnt/felhom-drives/<name>:/srv/<name>:rslave`) with its display label as the sidebar source; NAS add/remove trigger the same debounced sync. Two classes, two gates: drives keep the drive-absent gate byte-identically (proven live: the drives-only box logged a no-op sync); network shares gate on the STUB classifier instead — idle autofs is HEALTHY and included (Phase-0 probe on demo-hp: an in-container access through an rslave bind WAKES the idle trigger), while a stub verdict excludes the share from mounts AND sources with a WARN (an exposed stub swallows uploads the real mount later shadows). Nothing is ever written toward the NAS (no skeleton — red-proven). Live leg: cross-box upload round-trip demo-hp → demo-felhom + dead-NAS check (`Host is down` in seconds, unaided recovery after samba restart). Operator residual: the FileBrowser UI click-through (its admin credential is customer-held by design). Evidence: `felhom-controller/REPORT.md` (2026-07-22) |
|
||||
| R-68 | **Notification train: paired recovery mails + prefs seeding at claim + priority headers (power-outage audit F11+F12+F14-light).** The dead-man's-switch fired perfectly on 07-22 and the customer who got „A szerver nem elérhető!" was never told it recovered (F11); a customer without a `customer_notifications` row is silently unnotifiable (F12, demo-hp live); delivered ≠ noticed (F14). | M | **SHIPPED (hub v0.71.0, 2026-07-22)** | Origin: `AUDIT-power-outage-recovery-2026-07-22.md`. Recovery = explicit eventType branch (severity semantics frozen; `severityNotifies` untouched): operator always hears both edges, customer iff PAIRED (customer-channel `sent` stale/down row newer than the last sent recovery — `store.LastCustomerSentAt`; `enabled_events` deliberately ignored for recovery; ties → no mail, flap-safe). Seed-at-claim: `MarkClaimed` → `SeedNotificationPrefs` (INSERT-if-absent, never upsert — red-proofed; empty email no-op; never fails the claim; default critical-only set). Hub-side empty-email no-clobber belt in `handleSavePreferences` (controller 0.160.0 already guards its own two push legs — latent, not live). `X-Priority: 1` + `Importance: high` on error/critical via Resend `headers` (live-probed HTTP 200 before implementation); the `test` event now also mails the operator with those headers (one click proves both channels + rendering); latent `sendTestEmail` nil-prefs panic fixed. 17 tests + 4 red-proofs. **Live legs pending:** natural `*_recovered` mail on the next real staleness cycle (or the reboot-drill arc — NEVER fabricated by blocking reports, that is F9-bypass-shaped) and seed-at-claim on a real claim (Peti Friday reinstall is the natural candidate) |
|
||||
| R-69 | **F14-full: an operator push channel that actually interrupts (ntfy / Telegram / similar), beyond mail-client priority flags.** F14-light (v0.71.0 headers + Gmail filter) nudges a mail client; a 15:29 node_down should reach the operator's pocket in seconds regardless of inbox hygiene. Needs: channel choice (self-hosted ntfy on k3s vs Telegram bot), dispatcher fan-out seam, per-severity routing, quiet hours. | M | idea | Origin: `AUDIT-power-outage-recovery-2026-07-22.md` F14. Deliberately NOT built in the v0.71.0 train (scope-forked per the task spec) |
|
||||
| R-70 | **[P2-HIGH] The offsite last mile is invisible on BOTH surfaces — the hub cannot tell "staged" from "delivered" from "applied".** demo-hp sat 2 days with the hub customer page saying "Provisioned: … the transient password is delivered to the controller once" while the box said „Még nincs beállítva távoli mentési cél" — and a real customer would sit unprotected indefinitely believing otherwise. The hub HAS the signal (`one_time_secrets.consumed_at` + 153 consecutive reports carrying no offbox object) and reads none of it: the "Provisioned" line is static copy gated only on `offsite.host` in ConfigJSON (`config_form_body.html:119–120`). | S–M | idea | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. Two legs: **hub customer card** shows the real delivery state ("provisioned, awaiting box consumption" / "consumed, awaiting apply" / "applied" — consumed_at × report-offbox-presence is enough for all three), and **controller banner** when the descriptor is enabled but no target is configured („Felhom offsite készen áll — a beállítás automatikus, folyamatban"). Couple to **R-31**'s async/status-card idiom (same surface likely serves both) and to the **R-39 `consumed_at` honesty gauge** precedent on the PBS side — a consumed secret + N report cycles with no offbox status is the same "disagreement no single tier can see" shape and deserves the same loud event. Supporting live datum: demo-felhom's 07-21 staged secret is still unconsumed today (key-auth-first path never consumes) — invisible for the same reason. |
|
||||
| R-71 | **[P1] Day-0 race: the managed floor-update kills the offsite apply-bridge between password-consume and persist — the one-shot credential is burned and the box lands in the silent consume-404 dead-end forever.** Proven on demo-hp (07-21): consume 16:27:42 → managed update 0.153.0→0.156.0 replaces the container 16:28:17, ~35 s later, mid `ssh-copy-id` window; nothing persisted, no installed key ⇒ the key-auth-first recovery path can never engage, and every subsequent start logs the WARN and gives up until an operator Re-issue. **This recurs structurally on every fresh onboarding whose ISO floor lags the managed floor** — the update fires minutes after first boot, exactly when the bridge first runs. demo-felhom escaped by timing only. | M | idea | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` (mechanism cites: bridge order `offsiteapply.go:106–187` consume-then-persist, retry only on process start; the dead-end is even documented in source l.168–173 "the password is spent; reset it on the hub to retry"). Candidate directions, spec-first: (a) **order** — first-boot path lets the managed update settle before the bridge's consume step (cheapest; the race window is the update, not the reboot); (b) **two-phase consume** — hub marks consumed only on a controller ack-after-persist (touches the 404-no-oracle contract, design carefully); (c) **hub-side self-heal** — auto-restage a fresh secret when `consumed_at` is set but K consecutive reports show offsite enabled with no offbox status (the R-39(a) mint-race lesson applies: never restage on top of an UNCONSUMED secret). Pairs with R-70 (visibility) — but visibility alone is not the fix; the burned credential needs an unattended recovery path. |
|
||||
| R-70 | **[P2-HIGH] The offsite last mile is invisible on BOTH surfaces — the hub cannot tell "staged" from "delivered" from "applied".** demo-hp sat 2 days with the hub customer page saying "Provisioned: … the transient password is delivered to the controller once" while the box said „Még nincs beállítva távoli mentési cél" — and a real customer would sit unprotected indefinitely believing otherwise. The hub HAS the signal (`one_time_secrets.consumed_at` + 153 consecutive reports carrying no offbox object) and reads none of it: the "Provisioned" line is static copy gated only on `offsite.host` in ConfigJSON (`config_form_body.html:119–120`). | S–M | **SHIPPED (hub v0.72.0 + controller v0.161.0, 2026-07-23)** — detector `offsite.DeliveryStateFor` (one impl, all consumers), customer-card state line with age (static "delivered once" copy GONE), `offsite_delivery_stuck` warning event (24h durable cooldown), controller truthful empty-state banner. Live validation on the two fixtures (demo-hp `applied`; demo-felhom `applied` + stale-staged info) recorded in `felhom.eu/REPORT.md`; the banner leg is unit-proven/live-pending (no box occupies the enabled+no-offbox window). | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. Two legs: **hub customer card** shows the real delivery state ("provisioned, awaiting box consumption" / "consumed, awaiting apply" / "applied" — consumed_at × report-offbox-presence is enough for all three), and **controller banner** when the descriptor is enabled but no target is configured („Felhom offsite készen áll — a beállítás automatikus, folyamatban"). Couple to **R-31**'s async/status-card idiom (same surface likely serves both) and to the **R-39 `consumed_at` honesty gauge** precedent on the PBS side — a consumed secret + N report cycles with no offbox status is the same "disagreement no single tier can see" shape and deserves the same loud event. Supporting live datum: demo-felhom's 07-21 staged secret is still unconsumed today (key-auth-first path never consumes) — invisible for the same reason. |
|
||||
| R-71 | **[P1] Day-0 race: the managed floor-update kills the offsite apply-bridge between password-consume and persist — the one-shot credential is burned and the box lands in the silent consume-404 dead-end forever.** Proven on demo-hp (07-21): consume 16:27:42 → managed update 0.153.0→0.156.0 replaces the container 16:28:17, ~35 s later, mid `ssh-copy-id` window; nothing persisted, no installed key ⇒ the key-auth-first recovery path can never engage, and every subsequent start logs the WARN and gives up until an operator Re-issue. **This recurs structurally on every fresh onboarding whose ISO floor lags the managed floor** — the update fires minutes after first boot, exactly when the bridge first runs. demo-felhom escaped by timing only. | M | **PARTIAL — (c) SHIPPED (hub v0.72.0, 2026-07-23): self-heal restage via the EXISTING Re-issue path (`monitor.OffsiteDeliveryChecker`), trigger = consumed ≥1h + ≥4 consecutive offbox-less reports + zero offbox evidence, one restage/customer/24h (durable via events table), every firing = `offsite_credential_restaged` warning; R-39(a) act-time guard red-proofed (clobber observed with guard removed). Ships unit-proven, NOT live-fired (no broken box existed; arms on next natural occurrence or a staged drill — never PROVEN-LIVE until then). (a) day-0 ordering stays OPEN → its own upcoming spec. | Origin: `audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` (mechanism cites: bridge order `offsiteapply.go:106–187` consume-then-persist, retry only on process start; the dead-end is even documented in source l.168–173 "the password is spent; reset it on the hub to retry"). Candidate directions, spec-first: (a) **order** — first-boot path lets the managed update settle before the bridge's consume step (cheapest; the race window is the update, not the reboot); (b) **two-phase consume** — hub marks consumed only on a controller ack-after-persist (touches the 404-no-oracle contract, design carefully); (c) **hub-side self-heal** — auto-restage a fresh secret when `consumed_at` is set but K consecutive reports show offsite enabled with no offbox status (the R-39(a) mint-race lesson applies: never restage on top of an UNCONSUMED secret). Pairs with R-70 (visibility) — but visibility alone is not the fix; the burned credential needs an unattended recovery path. |
|
||||
| R-53 | **`app_export.html` substituted the CSRF token where the customer domain belongs** - the open-in-browser link was wrong for every app with a subdomain, and a session CSRF token landed in a URL. | XS | **SHIPPED (controller v0.150.0, 2026-07-20)** | One template token (`{{$.CSRFToken}}` -> `{{$.Domain}}`) plus the `Domain` key in `exportPageHandler`'s data map - that handler does not go through `baseData`, which is where every other page gets it, so the template had no domain to read. Render tests assert the joined `<sub>.<domain>` and that the token appears nowhere in that line; red-proofed against the pre-fix template. Origin: `audits/AUDIT-vacation-remote-ops-2026-07-20.md` (F7) |
|
||||
|
||||
## P3 — post-alpha
|
||||
|
||||
@@ -1,5 +1,47 @@
|
||||
# Felhom Hub — Changelog
|
||||
|
||||
## v0.72.0 — R-70 + R-71(c): the offsite last mile becomes visible, and burned credentials heal themselves (2026-07-23)
|
||||
|
||||
Origin: `documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` — demo-hp sat 2 days with the
|
||||
customer card claiming "Provisioned … delivered to the controller once" (static copy) while the box
|
||||
had NOTHING: the day-0 managed update killed the apply-bridge after password-consume, and the hub —
|
||||
holding both signals (`one_time_secrets.consumed_at`, 153 offbox-less reports) — read neither.
|
||||
|
||||
**One detector, four consumers:**
|
||||
|
||||
- **Detector** (`internal/offsite/delivery.go`, `DeliveryStateFor`): per-customer
|
||||
`OffsiteDeliveryState` from the secret row × report offsite-presence — `applied` (precedence:
|
||||
the box's own report wins) / `consumed_awaiting_apply` (the burned-credential shape) /
|
||||
`staged_awaiting_consume` / `no_secret`. The applied+stale-staged edge (demo-felhom's live
|
||||
shape: key-auth-first never consumes) stays `applied` PLUS a visible stale-staged flag. New
|
||||
store reads: `GetOneTimeSecretInfo` (timestamps only, value never selected),
|
||||
`LatestReportOffsitePresence`, `CountReportsOffsiteSince`, `LastEventAt`, + a test-only
|
||||
timestamp back-dater (the PBSDR seam pattern).
|
||||
- **Customer card** (`config_form_body.html` + `deliveryViewFor`): the static "delivered once"
|
||||
claim is GONE; the card renders the derived state with its age (badge idiom `n-ok/n-warn/
|
||||
n-neutral`; consumed goes amber past 30 min; the stale-staged info line names the specimen's
|
||||
age). Render test per branch (the v0.70.1 template-gate rule).
|
||||
- **Loud event**: `offsite_delivery_stuck` (WARNING → operator email; severity gate untouched —
|
||||
explicit tests pin that info would be silent) when consumed_awaiting_apply persists ≥ 1 h;
|
||||
per-customer 24 h cooldown, DURABLE via `LastEventAt` over the events table (a hub restart
|
||||
neither floods nor resets).
|
||||
- **Self-heal (R-71c)**: `OffsiteDeliveryChecker` (shared 60 s ticker) invokes the EXISTING
|
||||
Re-issue path (`web.Server.ReissueOffsiteForCustomer` behind the narrow `monitor.OffsiteReissuer`
|
||||
interface — the pbsdrheal precedent; armed only when the provisioner is configured) when the
|
||||
shape is unambiguous: consumed ≥ 1 h + ≥ 4 consecutive offbox-less reports since consume + zero
|
||||
offbox evidence. One restage per customer per 24 h (durable); every firing surfaces as
|
||||
`offsite_credential_restaged` (WARNING → operator) — repeats are repeating events, never a
|
||||
silent retry loop. **The R-39(a) guard**: `SaveOneTimeSecret` clobbers by design (Re-issue
|
||||
depends on supersede), so the heal RE-READS the secret row immediately before acting and refuses
|
||||
unless it is still a consumed row — the TOCTOU (operator Re-issue landing mid-tick) is the shape
|
||||
the guard kills. Red-proofs run and recorded (guard removed → the operator's fresh secret gets
|
||||
clobbered, test fails on `reissue calls = 1`; both rate limits removed → duplicate fire).
|
||||
|
||||
Companion: felhom-controller v0.161.0 (the box-side truthful empty state). The self-heal ships
|
||||
unit-proven + red-proofed, NOT live-fired — no broken box exists and none was broken for it; it
|
||||
arms on the next natural occurrence (or a staged drill). R-71(a) (day-0 ordering) stays open —
|
||||
separate spec.
|
||||
|
||||
## v0.71.0 — paired recovery mails, prefs seeding at claim, priority headers, operator test leg (2026-07-22)
|
||||
|
||||
Origin: `documentation/audits/AUDIT-power-outage-recovery-2026-07-22.md` F11 (recovery is silent),
|
||||
|
||||
@@ -306,6 +306,11 @@ func main() {
|
||||
// still renders, but saving with offsite enabled returns "not configured".
|
||||
var offsiteBoxChecker *monitor.OffsiteBoxChecker
|
||||
var pbsdrBoxChecker *monitor.PBSDRBoxChecker // R-5 v0.65.0: PBS-DR datastore fill (constructed with the tenantsync client below)
|
||||
// R-71c: the delivery checker's self-heal invokes the SAME Re-issue path the operator button
|
||||
// uses (webServer satisfies monitor.OffsiteReissuer). Wired ONLY when the provisioner exists —
|
||||
// a heal that cannot actually restage must never run (it would emit a restaged event for a
|
||||
// silent no-op: ReissueOffsiteForCustomer returns nil when offsite is unconfigured).
|
||||
var offsiteHealReissuer monitor.OffsiteReissuer
|
||||
if tok := os.Getenv("HETZNER_TOKEN"); tok != "" {
|
||||
poolBoxID, _ := strconv.ParseInt(os.Getenv("HETZNER_POOL_BOX_ID"), 10, 64)
|
||||
location := os.Getenv("HETZNER_LOCATION")
|
||||
@@ -317,6 +322,7 @@ func main() {
|
||||
API: client, Store: dataStore, Scanner: offsite.SSHHostKeyScanner{}, PoolBoxID: poolBoxID, Location: location, Logger: logger,
|
||||
})
|
||||
logger.Printf("[INFO] Offsite provisioning enabled (pool_box=%d, location=%s)", poolBoxID, location)
|
||||
offsiteHealReissuer = webServer // R-71c heal armed (provisioner present)
|
||||
// R-5 (v0.64.0): the pool-box aggregate checker shares the SAME client + pool box id (GET-only).
|
||||
// It needs a valid box id to poll; without one, the aggregate stays unconfigured.
|
||||
if poolBoxID != 0 {
|
||||
@@ -544,6 +550,11 @@ func main() {
|
||||
// (90/95% of quota_gb) + staleness (enabled+escrowed but no run >48h — the silently-stuck detector;
|
||||
// run FAILURES already alert via backup_failed). Nil-safe on pre-v0.109 reports. Same sweep.
|
||||
offsiteChecker := monitor.NewOffsiteChecker(dataStore, 0, dispatcher.ProcessEvent, logger)
|
||||
// R-70 + R-71c: the delivery-state checker — surfaces the burned-credential shape as
|
||||
// offsite_delivery_stuck (warning, 24h/customer) and self-heals it via the Re-issue path
|
||||
// (offsite_credential_restaged, one restage/customer/24h, R-39(a)-guarded). Cooldowns are
|
||||
// durable (events table), so a hub restart neither floods nor silently re-heals.
|
||||
offsiteDeliveryChecker := monitor.NewOffsiteDeliveryChecker(dataStore, offsiteHealReissuer, dispatcher.ProcessEvent, logger)
|
||||
go func() {
|
||||
ticker := time.NewTicker(60 * time.Second)
|
||||
defer ticker.Stop()
|
||||
@@ -561,6 +572,7 @@ func main() {
|
||||
hostMgmtPlaneChecker.Check()
|
||||
hostOOBChecker.Check()
|
||||
offsiteChecker.Check()
|
||||
offsiteDeliveryChecker.Check()
|
||||
if offsiteBoxChecker != nil {
|
||||
offsiteBoxChecker.Check() // R-5: restic pool-box aggregate (fetch-throttled internally)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/offsite"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-70 + R-71(c): the offsite delivery-state checker. Reads the shared detector
|
||||
// (offsite.DeliveryStateFor) for every offsite-enabled customer and drives two consumers:
|
||||
//
|
||||
// - the LOUD EVENT: `offsite_delivery_stuck` (warning → operator email per existing dispatcher
|
||||
// rules) when the burned-credential shape persists past stuckAfter;
|
||||
// - the SELF-HEAL (R-71c): invoke the EXISTING Re-issue path — never a second delivery
|
||||
// mechanism — when the shape is unambiguous, then surface `offsite_credential_restaged`
|
||||
// (warning) so the operator ALWAYS knows it fired.
|
||||
//
|
||||
// Cooldowns are durable: both events rate-limit off store.LastEventAt (the events table), so a hub
|
||||
// restart cannot flood or silently re-heal. A repeating pattern surfaces as repeating events on a
|
||||
// 24 h cadence, never as a silent retry loop.
|
||||
//
|
||||
// THE R-39(a) GUARD (mandatory, enforced HERE because the store deliberately clobbers):
|
||||
// SaveOneTimeSecret is last-write-wins by design — Re-issue depends on supersede. Restaging on top
|
||||
// of an UNCONSUMED secret would clobber a password a box may be about to consume (the operator may
|
||||
// have clicked Re-issue between this checker's derive and its act). So the heal re-reads the
|
||||
// secret row IMMEDIATELY before acting and refuses unless it is still a CONSUMED row.
|
||||
|
||||
const (
|
||||
eventDeliveryStuck = "offsite_delivery_stuck" // hub-internal (not in allowedEventTypes, like pbsdr_*)
|
||||
eventCredentialRestaged = "offsite_credential_restaged" // hub-internal
|
||||
// stuckAfter: consumed_awaiting_apply is normal for seconds (F10 repair: consume→applied in
|
||||
// 4 s). An hour of it means the apply will never come without intervention.
|
||||
stuckAfter = time.Hour
|
||||
// stuckCooldown / healCooldown: per-customer, durable via the events table.
|
||||
stuckCooldown = 24 * time.Hour
|
||||
healCooldown = 24 * time.Hour
|
||||
// healMinReports: at least this many consecutive offbox-less reports after consumed_at before
|
||||
// the self-heal may fire — the box must be alive and reporting, just not applied.
|
||||
healMinReports = 4
|
||||
)
|
||||
|
||||
// OffsiteReissuer is the narrow reissue surface the self-heal needs — satisfied by
|
||||
// *web.Server.ReissueOffsiteForCustomer (the pbsdrheal.Reissuer precedent; avoids an import
|
||||
// cycle and guarantees the heal IS the designed Re-issue path, not a sibling mechanism).
|
||||
type OffsiteReissuer interface {
|
||||
ReissueOffsiteForCustomer(ctx context.Context, customerID string) error
|
||||
}
|
||||
|
||||
// OffsiteDeliveryChecker runs on the shared monitor ticker.
|
||||
type OffsiteDeliveryChecker struct {
|
||||
store *store.Store
|
||||
reissuer OffsiteReissuer // nil → self-heal disabled (no provisioner configured); detector+event still run
|
||||
onEvent EventNotifyFunc
|
||||
logger *log.Logger
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
// NewOffsiteDeliveryChecker constructs the checker. reissuer may be nil (heal disabled).
|
||||
func NewOffsiteDeliveryChecker(s *store.Store, reissuer OffsiteReissuer, onEvent EventNotifyFunc, logger *log.Logger) *OffsiteDeliveryChecker {
|
||||
return &OffsiteDeliveryChecker{store: s, reissuer: reissuer, onEvent: onEvent, logger: logger, now: time.Now}
|
||||
}
|
||||
|
||||
// Check derives the delivery state for every offsite-enabled active customer and applies the
|
||||
// event + self-heal rules. Never returns an error — a checker failure must not take the ticker
|
||||
// down (log-and-continue, like every sibling checker).
|
||||
func (c *OffsiteDeliveryChecker) Check() {
|
||||
configs, err := c.store.ListCustomerConfigs()
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] offsite-delivery: list configs: %v", err)
|
||||
return
|
||||
}
|
||||
for _, cfg := range configs {
|
||||
if cfg.Status != "active" {
|
||||
continue // blocked/inactive customers are not delivery-monitored (and never healed)
|
||||
}
|
||||
d, err := offsite.ReadDescriptor(cfg.ConfigJSON)
|
||||
if err != nil || d == nil || !d.Enabled {
|
||||
continue // unparseable config never drives a heal; the config UI owns that failure
|
||||
}
|
||||
status, err := offsite.DeliveryStateFor(c.store, cfg.CustomerID)
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] offsite-delivery: %s: derive: %v", cfg.CustomerID, err)
|
||||
continue
|
||||
}
|
||||
if status.State != offsite.DeliveryConsumedAwaitingApply {
|
||||
continue // applied / staged / no_secret: card-rendered states, no event or heal (yet)
|
||||
}
|
||||
age := c.now().Sub(status.Since)
|
||||
if age < stuckAfter {
|
||||
continue // normal convergence window
|
||||
}
|
||||
c.maybeEmitStuck(cfg.CustomerID, status, age)
|
||||
c.maybeHeal(cfg.CustomerID, status)
|
||||
}
|
||||
}
|
||||
|
||||
// maybeEmitStuck emits offsite_delivery_stuck (warning) once per stuckCooldown per customer.
|
||||
func (c *OffsiteDeliveryChecker) maybeEmitStuck(customerID string, status offsite.DeliveryStatus, age time.Duration) {
|
||||
last, err := c.store.LastEventAt(customerID, eventDeliveryStuck)
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] offsite-delivery: %s: cooldown read: %v", customerID, err)
|
||||
return
|
||||
}
|
||||
if !last.IsZero() && c.now().Sub(last) < stuckCooldown {
|
||||
return
|
||||
}
|
||||
msg := fmt.Sprintf("Offsite delivery stuck: one-time password consumed %s ago and %d report(s) since carry no offbox target — the credential is likely burned (apply died between consume and persist). Re-issue delivers a fresh one.",
|
||||
age.Round(time.Minute), status.ReportsSinceConsume)
|
||||
details, _ := json.Marshal(map[string]any{
|
||||
"state": string(status.State),
|
||||
"consumed_at": status.Since.UTC().Format(time.RFC3339),
|
||||
"reports_since_consume": status.ReportsSinceConsume,
|
||||
})
|
||||
c.emit(customerID, eventDeliveryStuck, "warning", msg, string(details))
|
||||
}
|
||||
|
||||
// maybeHeal fires the R-71c self-heal when the burned-credential shape is unambiguous:
|
||||
// consumed ≥ stuckAfter ago, ≥ healMinReports consecutive reports since with ZERO offbox evidence,
|
||||
// one heal per healCooldown — and the R-39(a) guard holds at act time.
|
||||
func (c *OffsiteDeliveryChecker) maybeHeal(customerID string, status offsite.DeliveryStatus) {
|
||||
if c.reissuer == nil {
|
||||
return
|
||||
}
|
||||
if status.ReportsSinceConsume < healMinReports || status.OffsiteReportsSinceConsume != 0 {
|
||||
return // box not reporting enough, or offbox evidence exists (regressed-apply shape) → operator's call
|
||||
}
|
||||
last, err := c.store.LastEventAt(customerID, eventCredentialRestaged)
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] offsite-delivery: %s: heal rate-limit read: %v", customerID, err)
|
||||
return
|
||||
}
|
||||
if !last.IsZero() && c.now().Sub(last) < healCooldown {
|
||||
return // one restage per customer per 24 h — repeats surface as repeated events only
|
||||
}
|
||||
// THE R-39(a) GUARD — re-read the secret row immediately before acting. The derive above is a
|
||||
// snapshot; an operator Re-issue may have staged a FRESH UNCONSUMED secret since (TOCTOU).
|
||||
// SaveOneTimeSecret clobbers by design, so acting now would burn that fresh password.
|
||||
info, err := c.store.GetOneTimeSecretInfo(customerID)
|
||||
if err != nil {
|
||||
c.logger.Printf("[WARN] offsite-delivery: %s: guard read: %v", customerID, err)
|
||||
return
|
||||
}
|
||||
if info == nil || info.ConsumedAt.IsZero() {
|
||||
c.logger.Printf("[INFO] offsite-delivery: %s: heal refused — secret row is now %s (R-39(a) guard: never restage over an unconsumed secret)",
|
||||
customerID, secretShape(info))
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
|
||||
defer cancel()
|
||||
if err := c.reissuer.ReissueOffsiteForCustomer(ctx, customerID); err != nil {
|
||||
c.logger.Printf("[WARN] offsite-delivery: %s: self-heal reissue failed: %v", customerID, err)
|
||||
return
|
||||
}
|
||||
c.logger.Printf("[INFO] offsite-delivery: %s: self-heal restage fired (consumed_at %s, %d offbox-less reports since)",
|
||||
customerID, status.Since.UTC().Format(time.RFC3339), status.ReportsSinceConsume)
|
||||
details, _ := json.Marshal(map[string]any{
|
||||
"burned_consumed_at": status.Since.UTC().Format(time.RFC3339),
|
||||
"reports_since_consume": status.ReportsSinceConsume,
|
||||
})
|
||||
c.emit(customerID, eventCredentialRestaged, "warning",
|
||||
"Offsite credential re-staged automatically: the previous one-time password was consumed but never applied (burned mid-delivery). The box picks the fresh password up on its next config refresh.",
|
||||
string(details))
|
||||
}
|
||||
|
||||
func secretShape(info *store.OneTimeSecretInfo) string {
|
||||
if info == nil {
|
||||
return "absent"
|
||||
}
|
||||
return "unconsumed (staged " + info.CreatedAt.UTC().Format(time.RFC3339) + ")"
|
||||
}
|
||||
|
||||
// emit saves the event (audit trail first) and then notifies — the OffsiteChecker convention:
|
||||
// SaveEvent failure logs and SKIPS the notification (an email without its audit row lies).
|
||||
func (c *OffsiteDeliveryChecker) emit(customerID, eventType, severity, message, details string) {
|
||||
if _, err := c.store.SaveEvent(customerID, eventType, severity, message, details, "hub"); err != nil {
|
||||
c.logger.Printf("[WARN] offsite-delivery: %s: save %s: %v", customerID, eventType, err)
|
||||
return
|
||||
}
|
||||
c.logger.Printf("[INFO] offsite-delivery: %s: %s (%s)", customerID, eventType, severity)
|
||||
if c.onEvent != nil {
|
||||
c.onEvent(customerID, eventType, severity, message, details, "hub")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,302 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-70/R-71c checker tests. Real store, fake reissuer (records calls AND mimics the production
|
||||
// side effect — a restage IS a SaveOneTimeSecret — so a clobber is observable on the row itself),
|
||||
// captured onEvent, injected clock.
|
||||
|
||||
const (
|
||||
dtReportNoOffsite = `{"health":{"status":"ok"}}`
|
||||
dtReportWithOffsite = `{"health":{"status":"ok"},"offsite":{"enabled":true}}`
|
||||
dtOffsiteConfig = `{"offsite":{"enabled":true,"type":"shared","host":"h","user":"u","repo_path":"/home/felhom-repo","quota_gb":50}}`
|
||||
)
|
||||
|
||||
type fakeReissuer struct {
|
||||
mu sync.Mutex
|
||||
st *store.Store
|
||||
calls []string
|
||||
}
|
||||
|
||||
func (f *fakeReissuer) ReissueOffsiteForCustomer(_ context.Context, customerID string) error {
|
||||
f.mu.Lock()
|
||||
f.calls = append(f.calls, customerID)
|
||||
f.mu.Unlock()
|
||||
// The production path's essential side effect: ReissueCredentials → SaveOneTimeSecret
|
||||
// (last-write-wins clobber). Mimicked so the R-39(a) tests can observe what a wrongly-fired
|
||||
// heal would DO to the row, not merely that it was called.
|
||||
return f.st.SaveOneTimeSecret(customerID, "fresh-from-heal")
|
||||
}
|
||||
|
||||
func (f *fakeReissuer) count() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return len(f.calls)
|
||||
}
|
||||
|
||||
type dtHarness struct {
|
||||
st *store.Store
|
||||
reissuer *fakeReissuer
|
||||
checker *OffsiteDeliveryChecker
|
||||
events *[]string // "type:severity"
|
||||
}
|
||||
|
||||
func newDTHarness(t *testing.T, withReissuer bool) dtHarness {
|
||||
t.Helper()
|
||||
st, err := store.New(filepath.Join(t.TempDir(), "d.db"), log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { st.Close() })
|
||||
if err := st.SaveCustomerConfig(&store.CustomerConfig{
|
||||
CustomerID: "c1", CustomerName: "C", Domain: "c1.hu", APIKey: "k", RetrievalPassword: "p",
|
||||
ConfigJSON: dtOffsiteConfig,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
events := &[]string{}
|
||||
var mu sync.Mutex
|
||||
onEvent := func(_, eventType, severity, _, _, _ string) {
|
||||
mu.Lock()
|
||||
*events = append(*events, eventType+":"+severity)
|
||||
mu.Unlock()
|
||||
}
|
||||
var ri *fakeReissuer
|
||||
var riIface OffsiteReissuer
|
||||
if withReissuer {
|
||||
ri = &fakeReissuer{st: st}
|
||||
riIface = ri
|
||||
}
|
||||
c := NewOffsiteDeliveryChecker(st, riIface, onEvent, log.New(io.Discard, "", 0))
|
||||
return dtHarness{st: st, reissuer: ri, checker: c, events: events}
|
||||
}
|
||||
|
||||
// burnedFixture puts c1 into the F10 shape: consumed >1h ago, N offbox-less reports since.
|
||||
func (h dtHarness) burnedFixture(t *testing.T, reports int) {
|
||||
t.Helper()
|
||||
if err := h.st.SaveOneTimeSecret("c1", "x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
consumed := time.Now().UTC().Add(-2 * time.Hour).Format("2006-01-02 15:04:05")
|
||||
staged := time.Now().UTC().Add(-3 * time.Hour).Format("2006-01-02 15:04:05")
|
||||
if err := h.st.SetOneTimeSecretTimesForTest("c1", staged, consumed); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < reports; i++ {
|
||||
if err := h.st.SaveReport("c1", []byte(dtReportNoOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (h dtHarness) savedEvents(t *testing.T, eventType string) []store.Event {
|
||||
t.Helper()
|
||||
all, err := h.st.GetRecentEvents("c1", 50)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out []store.Event
|
||||
for _, e := range all {
|
||||
if e.EventType == eventType {
|
||||
out = append(out, e)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Scenario: the stuck event fires once with severity WARNING, and the durable 24h cooldown holds
|
||||
// across a second pass (delete the LastEventAt guard in maybeEmitStuck → this fails with 2 events —
|
||||
// the executable red-proof of the cooldown).
|
||||
func TestDeliveryChecker_StuckEvent_WarningOncePer24h(t *testing.T) {
|
||||
h := newDTHarness(t, false)
|
||||
h.burnedFixture(t, 5)
|
||||
|
||||
h.checker.Check()
|
||||
h.checker.Check() // same tick shape again — cooldown must swallow it
|
||||
|
||||
saved := h.savedEvents(t, "offsite_delivery_stuck")
|
||||
if len(saved) != 1 {
|
||||
t.Fatalf("stuck events = %d, want exactly 1 (24h per-customer cooldown)", len(saved))
|
||||
}
|
||||
if saved[0].Severity != "warning" {
|
||||
t.Fatalf("severity = %q, want warning (operator email tier; never info-silent, never critical)", saved[0].Severity)
|
||||
}
|
||||
var details map[string]any
|
||||
if err := json.Unmarshal([]byte(saved[0].DetailsJSON), &details); err != nil || details["consumed_at"] == "" {
|
||||
t.Fatalf("details must carry consumed_at (mióta), got %s err %v", saved[0].DetailsJSON, err)
|
||||
}
|
||||
if got := *h.events; len(got) != 1 || got[0] != "offsite_delivery_stuck:warning" {
|
||||
t.Fatalf("dispatched = %v, want exactly [offsite_delivery_stuck:warning]", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario: the R-71c self-heal fires EXACTLY once — one reissue call, the restaged event
|
||||
// (warning), and the 24h rate limit blocks a re-trigger even when the shape recurs (delete the
|
||||
// LastEventAt guard in maybeHeal → the second pass fires again and this fails — the rate-limit
|
||||
// red-proof).
|
||||
func TestDeliveryChecker_SelfHeal_FiresOnceAndRateLimits(t *testing.T) {
|
||||
h := newDTHarness(t, true)
|
||||
h.burnedFixture(t, 4)
|
||||
|
||||
h.checker.Check()
|
||||
if h.reissuer.count() != 1 {
|
||||
t.Fatalf("reissue calls = %d, want 1", h.reissuer.count())
|
||||
}
|
||||
restaged := h.savedEvents(t, "offsite_credential_restaged")
|
||||
if len(restaged) != 1 || restaged[0].Severity != "warning" {
|
||||
t.Fatalf("restaged events = %v, want exactly 1 with severity warning (the operator ALWAYS learns a heal fired)", restaged)
|
||||
}
|
||||
|
||||
// The shape recurs (box burned the fresh one too): back into consumed_awaiting_apply.
|
||||
h.burnedFixture(t, 4)
|
||||
h.checker.Check()
|
||||
if h.reissuer.count() != 1 {
|
||||
t.Fatalf("reissue calls after recurrence = %d, want STILL 1 (one restage per customer per 24h; repeats surface as events only)", h.reissuer.count())
|
||||
}
|
||||
}
|
||||
|
||||
// Not-enough-evidence gates: under healMinReports offbox-less reports → no heal; any offbox
|
||||
// evidence since consume (regressed-apply shape) → no heal. The stuck EVENT still fires (age gate
|
||||
// alone) — visibility never waits for the heal's stricter bar.
|
||||
func TestDeliveryChecker_SelfHeal_EvidenceGates(t *testing.T) {
|
||||
h := newDTHarness(t, true)
|
||||
h.burnedFixture(t, 3) // 3 < healMinReports
|
||||
h.checker.Check()
|
||||
if h.reissuer.count() != 0 {
|
||||
t.Fatalf("reissue with 3 reports = %d calls, want 0 (needs >= 4)", h.reissuer.count())
|
||||
}
|
||||
if len(h.savedEvents(t, "offsite_delivery_stuck")) != 1 {
|
||||
t.Fatal("stuck event must fire regardless of the heal's stricter evidence bar")
|
||||
}
|
||||
|
||||
// add offbox evidence AFTER consume, then more offbox-less reports — mixed history: no heal
|
||||
if err := h.st.SaveReport("c1", []byte(dtReportWithOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := h.st.SaveReport("c1", []byte(dtReportNoOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.checker.Check()
|
||||
if h.reissuer.count() != 0 {
|
||||
t.Fatalf("reissue on mixed offbox history = %d calls, want 0 (regressed-apply is the operator's call)", h.reissuer.count())
|
||||
}
|
||||
}
|
||||
|
||||
// THE CLOBBER RED-PROOF (R-39(a), mandatory per spec): an operator Re-issue lands between the
|
||||
// checker's derive and its act (simulated via the onEvent hook, which runs after the stuck event
|
||||
// and before maybeHeal). The act-time guard re-reads the row, finds it UNCONSUMED, and refuses —
|
||||
// zero reissue calls, row untouched. Remove the `info.ConsumedAt.IsZero()` refusal in maybeHeal →
|
||||
// the fake fires SaveOneTimeSecret and this test FAILS on both assertions (calls=1, row clobbered)
|
||||
// — proving the fixture would have been clobbered.
|
||||
func TestDeliveryChecker_R39aGuard_NeverRestagesOverUnconsumed(t *testing.T) {
|
||||
h := newDTHarness(t, true)
|
||||
h.burnedFixture(t, 4)
|
||||
|
||||
// The TOCTOU: the moment the stuck event dispatches, the "operator" stages a fresh secret.
|
||||
operatorStaged := "2026-07-23 12:00:00"
|
||||
*h.events = nil
|
||||
base := h.checker.onEvent
|
||||
h.checker.onEvent = func(cid, et, sev, msg, det, src string) {
|
||||
if et == "offsite_delivery_stuck" {
|
||||
if err := h.st.SaveOneTimeSecret("c1", "operator-fresh"); err != nil {
|
||||
t.Errorf("mid-tick stage: %v", err)
|
||||
}
|
||||
if err := h.st.SetOneTimeSecretTimesForTest("c1", operatorStaged, ""); err != nil {
|
||||
t.Errorf("mid-tick stamp: %v", err)
|
||||
}
|
||||
}
|
||||
base(cid, et, sev, msg, det, src)
|
||||
}
|
||||
|
||||
h.checker.Check()
|
||||
|
||||
if h.reissuer.count() != 0 {
|
||||
t.Fatalf("reissue calls = %d, want 0 — the R-39(a) guard must refuse over an unconsumed secret", h.reissuer.count())
|
||||
}
|
||||
info, err := h.st.GetOneTimeSecretInfo("c1")
|
||||
if err != nil || info == nil {
|
||||
t.Fatalf("secret row: %v / %v", info, err)
|
||||
}
|
||||
if !info.ConsumedAt.IsZero() || !info.CreatedAt.Equal(time.Date(2026, 7, 23, 12, 0, 0, 0, time.UTC)) {
|
||||
t.Fatalf("the operator's fresh secret was CLOBBERED (created_at=%v consumed_at=%v) — R-39(a) violated", info.CreatedAt, info.ConsumedAt)
|
||||
}
|
||||
if len(h.savedEvents(t, "offsite_credential_restaged")) != 0 {
|
||||
t.Fatal("no restaged event may exist for a refused heal")
|
||||
}
|
||||
}
|
||||
|
||||
// The demo-felhom live shape, full Check(): applied + stale unconsumed staged secret → ZERO events,
|
||||
// ZERO reissue calls, row untouched. Precedence (applied wins) is the first line of defense; the
|
||||
// R-39(a) guard is the second.
|
||||
func TestDeliveryChecker_AppliedWithStaleStaged_Untouched(t *testing.T) {
|
||||
h := newDTHarness(t, true)
|
||||
if err := h.st.SaveReport("c1", []byte(dtReportWithOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := h.st.SaveOneTimeSecret("c1", "stale"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := h.st.SetOneTimeSecretTimesForTest("c1", "2026-07-21 08:29:29", ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
h.checker.Check()
|
||||
|
||||
if h.reissuer.count() != 0 {
|
||||
t.Fatalf("reissue calls = %d, want 0 (demo-felhom shape is healthy)", h.reissuer.count())
|
||||
}
|
||||
if len(*h.events) != 0 {
|
||||
t.Fatalf("events = %v, want none", *h.events)
|
||||
}
|
||||
info, _ := h.st.GetOneTimeSecretInfo("c1")
|
||||
if info == nil || !info.ConsumedAt.IsZero() || !info.CreatedAt.Equal(time.Date(2026, 7, 21, 8, 29, 29, 0, time.UTC)) {
|
||||
t.Fatalf("fixture row mutated: %+v — the live specimen must survive the checker untouched", info)
|
||||
}
|
||||
}
|
||||
|
||||
// Guard rails: young consumed state (inside stuckAfter) is silent; offsite-disabled and
|
||||
// non-active customers are skipped entirely.
|
||||
func TestDeliveryChecker_QuietShapes(t *testing.T) {
|
||||
h := newDTHarness(t, true)
|
||||
// consumed 5 minutes ago — normal convergence window
|
||||
if err := h.st.SaveOneTimeSecret("c1", "x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
recent := time.Now().UTC().Add(-5 * time.Minute).Format("2006-01-02 15:04:05")
|
||||
if err := h.st.SetOneTimeSecretTimesForTest("c1", recent, recent); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < 5; i++ {
|
||||
if err := h.st.SaveReport("c1", []byte(dtReportNoOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
h.checker.Check()
|
||||
if len(*h.events) != 0 || h.reissuer.count() != 0 {
|
||||
t.Fatalf("young consumed state must be silent, got events=%v calls=%d", *h.events, h.reissuer.count())
|
||||
}
|
||||
|
||||
// offsite disabled → skipped even in a stuck-looking shape
|
||||
if err := h.st.SaveCustomerConfig(&store.CustomerConfig{
|
||||
CustomerID: "c1", CustomerName: "C", Domain: "c1.hu", APIKey: "k", RetrievalPassword: "p",
|
||||
ConfigJSON: `{"offsite":{"enabled":false}}`,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
h.burnedFixture(t, 5)
|
||||
h.checker.Check()
|
||||
if len(*h.events) != 0 || h.reissuer.count() != 0 {
|
||||
t.Fatalf("disabled offsite must be skipped, got events=%v calls=%d", *h.events, h.reissuer.count())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-70/R-71c severity contract, alongside the v0.71.0 guards: the two new detector events carry
|
||||
// `warning` — they route to the OPERATOR through the existing severity gate, with NO widening of
|
||||
// severityNotifies and NO customer email (neither type is in any default enabled_events set, and
|
||||
// neither has a customerMessages entry — operator-only until the mechanism has history).
|
||||
func TestOffsiteDeliveryEvents_WarningRoutesToOperatorOnly(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "hub@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
mails := captureSeam(d)
|
||||
|
||||
d.ProcessEvent("c1", "offsite_delivery_stuck", "warning", "stuck", "{}", "hub")
|
||||
d.ProcessEvent("c1", "offsite_credential_restaged", "warning", "restaged", "{}", "hub")
|
||||
|
||||
op := mailsFor(*mails, "op@felhom.eu")
|
||||
if len(op) != 2 {
|
||||
t.Fatalf("operator mails = %d, want 2 (warning must notify the operator)", len(op))
|
||||
}
|
||||
// customer leg: c1 has no notification prefs row → nothing may go anywhere else
|
||||
if len(*mails) != 2 {
|
||||
t.Fatalf("total mails = %d, want 2 — the detector events are operator-only", len(*mails))
|
||||
}
|
||||
}
|
||||
|
||||
// The deliberate decision, as an explicit test beside TestRecovery_SeverityStaysInfo: had these
|
||||
// events shipped as `info`, the dispatcher would silently drop them — the exact invisibility R-70
|
||||
// exists to kill. This pins the severity choice against a future "quiet them down" edit.
|
||||
func TestOffsiteDeliveryEvents_InfoWouldBeSilent(t *testing.T) {
|
||||
st := newDispStore(t)
|
||||
d := NewDispatcher(st, "test-key", "hub@felhom.eu", "op@felhom.eu", true, log.New(io.Discard, "", 0))
|
||||
mails := captureSeam(d)
|
||||
|
||||
d.ProcessEvent("c1", "offsite_delivery_stuck", "info", "stuck", "{}", "hub")
|
||||
|
||||
if len(*mails) != 0 {
|
||||
t.Fatalf("mails = %d, want 0 — info stays silent (severityNotifies untouched); the checker MUST emit warning", len(*mails))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,97 @@
|
||||
package offsite
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-70 delivery-state detector — ONE implementation, all consumers read it (customer card, the
|
||||
// monitor checker's stuck event + R-71c self-heal, and any future surface). Derived from the two
|
||||
// signals the hub already holds and previously read nowhere:
|
||||
// - the one_time_secrets row (exists? consumed_at?) — the delivery ledger;
|
||||
// - report offsite-object presence — the box's own testimony that an offbox target is applied
|
||||
// (the report builder attaches `offsite` only when a target is configured).
|
||||
//
|
||||
// Origin: DIAG-f10-demo-hp-offsite-2026-07-23 — demo-hp sat 2 days in consumed_awaiting_apply
|
||||
// (the burned-credential shape) while the operator card said "Provisioned" from static copy.
|
||||
|
||||
// DeliveryState is the customer's offsite last-mile state.
|
||||
type DeliveryState string
|
||||
|
||||
const (
|
||||
// DeliveryApplied — the latest report carries an offsite status object: the tier is live on
|
||||
// the box. Wins over every secret-row shape (precedence rule; an applied box with a stale
|
||||
// staged secret is APPLIED — see StaleStagedSince).
|
||||
DeliveryApplied DeliveryState = "applied"
|
||||
// DeliveryConsumedAwaitingApply — the one-time password was consumed but no report since has
|
||||
// ever shown an offbox target: the burned-credential shape (normal for seconds, wrong for
|
||||
// hours; the F10 box sat here for 2 days).
|
||||
DeliveryConsumedAwaitingApply DeliveryState = "consumed_awaiting_apply"
|
||||
// DeliveryStagedAwaitingConsume — a secret is staged and not yet consumed: delivery pending
|
||||
// (normal for minutes while the box re-pulls config, wrong for days).
|
||||
DeliveryStagedAwaitingConsume DeliveryState = "staged_awaiting_consume"
|
||||
// DeliveryNoSecret — offsite is enabled in the config but no secret row exists at all: a
|
||||
// mis-state (e.g. a RESET purged the row without deprovisioning). Rendered needs-attention.
|
||||
DeliveryNoSecret DeliveryState = "no_secret"
|
||||
)
|
||||
|
||||
// DeliveryStatus is the derived last-mile state plus the timestamps every consumer must surface
|
||||
// ("mióta" — R-70's whole point is that nothing about this plumbing stays invisible).
|
||||
type DeliveryStatus struct {
|
||||
State DeliveryState
|
||||
// Since anchors the state's age: consumed_at for consumed_awaiting_apply, created_at for
|
||||
// staged_awaiting_consume, the latest report time for applied, zero for no_secret.
|
||||
Since time.Time
|
||||
// StaleStagedSince is non-zero ONLY in the applied+stale-staged edge (the demo-felhom shape):
|
||||
// the box is applied via an earlier generation while an unconsumed secret sits staged. The
|
||||
// caller renders an info line; the self-heal must NEVER touch this shape (R-39(a)).
|
||||
StaleStagedSince time.Time
|
||||
// ReportsSinceConsume / OffsiteReportsSinceConsume count reports received after consumed_at
|
||||
// (consumed_awaiting_apply only; zero otherwise). The R-71c trigger requires
|
||||
// ReportsSinceConsume >= N with OffsiteReportsSinceConsume == 0 — "consecutive without".
|
||||
ReportsSinceConsume int
|
||||
OffsiteReportsSinceConsume int
|
||||
}
|
||||
|
||||
// DeliveryStateFor derives the customer's offsite delivery state. Callers gate on the descriptor
|
||||
// (offsite enabled) themselves — this function only reads the ledger and the reports; age math is
|
||||
// the caller's (against its own clock seam).
|
||||
func DeliveryStateFor(st *store.Store, customerID string) (DeliveryStatus, error) {
|
||||
found, receivedAt, hasOffsite, err := st.LatestReportOffsitePresence(customerID)
|
||||
if err != nil {
|
||||
return DeliveryStatus{}, err
|
||||
}
|
||||
secret, err := st.GetOneTimeSecretInfo(customerID)
|
||||
if err != nil {
|
||||
return DeliveryStatus{}, err
|
||||
}
|
||||
|
||||
// Precedence: applied wins. The box's own report is the strongest evidence there is.
|
||||
if found && hasOffsite {
|
||||
status := DeliveryStatus{State: DeliveryApplied, Since: receivedAt}
|
||||
if secret != nil && secret.ConsumedAt.IsZero() {
|
||||
status.StaleStagedSince = secret.CreatedAt // demo-felhom shape: applied + stale staged
|
||||
}
|
||||
return status, nil
|
||||
}
|
||||
|
||||
if secret == nil {
|
||||
return DeliveryStatus{State: DeliveryNoSecret}, nil
|
||||
}
|
||||
|
||||
if !secret.ConsumedAt.IsZero() {
|
||||
total, withOffsite, err := st.CountReportsOffsiteSince(customerID, secret.ConsumedAt)
|
||||
if err != nil {
|
||||
return DeliveryStatus{}, err
|
||||
}
|
||||
return DeliveryStatus{
|
||||
State: DeliveryConsumedAwaitingApply,
|
||||
Since: secret.ConsumedAt,
|
||||
ReportsSinceConsume: total,
|
||||
OffsiteReportsSinceConsume: withOffsite,
|
||||
}, nil
|
||||
}
|
||||
|
||||
return DeliveryStatus{State: DeliveryStagedAwaitingConsume, Since: secret.CreatedAt}, nil
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
package offsite
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-70 detector tests — four states + the applied-wins precedence + the demo-felhom
|
||||
// applied+stale-staged edge. Fixtures are the real store (t.TempDir SQLite), timestamps
|
||||
// controlled via the test-only back-dater (no sleeps).
|
||||
|
||||
func newDeliveryStore(t *testing.T) *store.Store {
|
||||
t.Helper()
|
||||
st, err := store.New(filepath.Join(t.TempDir(), "d.db"), log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { st.Close() })
|
||||
return st
|
||||
}
|
||||
|
||||
const (
|
||||
reportWithOffsite = `{"health":{"status":"ok"},"offsite":{"enabled":true,"escrow_state":"escrowed"}}`
|
||||
reportNoOffsite = `{"health":{"status":"ok"}}`
|
||||
)
|
||||
|
||||
func TestDeliveryState_Applied(t *testing.T) {
|
||||
st := newDeliveryStore(t)
|
||||
if err := st.SaveReport("c1", []byte(reportWithOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveOneTimeSecret("c1", "x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := st.ConsumeOneTimeSecret("c1"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := DeliveryStateFor(st, "c1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.State != DeliveryApplied {
|
||||
t.Fatalf("state = %s, want applied", got.State)
|
||||
}
|
||||
if !got.StaleStagedSince.IsZero() {
|
||||
t.Fatalf("consumed secret must not flag stale-staged, got %v", got.StaleStagedSince)
|
||||
}
|
||||
if got.Since.IsZero() {
|
||||
t.Fatal("applied state must carry the report timestamp (mióta)")
|
||||
}
|
||||
}
|
||||
|
||||
// The demo-felhom shape: applied via an earlier generation + an unconsumed secret staged since.
|
||||
// Precedence: applied WINS, and the stale staged secret becomes a VISIBLE info flag — never a
|
||||
// state downgrade, never a heal trigger.
|
||||
func TestDeliveryState_AppliedWithStaleStaged(t *testing.T) {
|
||||
st := newDeliveryStore(t)
|
||||
if err := st.SaveReport("c1", []byte(reportWithOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveOneTimeSecret("c1", "stale"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SetOneTimeSecretTimesForTest("c1", "2026-07-21 08:29:29", ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := DeliveryStateFor(st, "c1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.State != DeliveryApplied {
|
||||
t.Fatalf("precedence broken: state = %s, want applied (applied wins over staged)", got.State)
|
||||
}
|
||||
want := time.Date(2026, 7, 21, 8, 29, 29, 0, time.UTC)
|
||||
if !got.StaleStagedSince.Equal(want) {
|
||||
t.Fatalf("StaleStagedSince = %v, want %v (the invisible plumbing must be visible)", got.StaleStagedSince, want)
|
||||
}
|
||||
}
|
||||
|
||||
// The burned-credential shape (F10): consumed, and every report since carries no offbox.
|
||||
func TestDeliveryState_ConsumedAwaitingApply(t *testing.T) {
|
||||
st := newDeliveryStore(t)
|
||||
if err := st.SaveOneTimeSecret("c1", "x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// consumed 2h ago; the reports below (received "now") all come AFTER it
|
||||
if err := st.SetOneTimeSecretTimesForTest("c1", "2026-01-01 00:00:00", "2026-01-01 00:03:00"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < 5; i++ {
|
||||
if err := st.SaveReport("c1", []byte(reportNoOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
got, err := DeliveryStateFor(st, "c1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.State != DeliveryConsumedAwaitingApply {
|
||||
t.Fatalf("state = %s, want consumed_awaiting_apply", got.State)
|
||||
}
|
||||
if got.ReportsSinceConsume != 5 || got.OffsiteReportsSinceConsume != 0 {
|
||||
t.Fatalf("counts = %d/%d, want 5/0", got.ReportsSinceConsume, got.OffsiteReportsSinceConsume)
|
||||
}
|
||||
if want := time.Date(2026, 1, 1, 0, 3, 0, 0, time.UTC); !got.Since.Equal(want) {
|
||||
t.Fatalf("Since = %v, want consumed_at %v", got.Since, want)
|
||||
}
|
||||
}
|
||||
|
||||
// The since-filter: reports received BEFORE consumed_at must not count toward the trigger.
|
||||
func TestDeliveryState_ReportsBeforeConsumeNotCounted(t *testing.T) {
|
||||
st := newDeliveryStore(t)
|
||||
for i := 0; i < 3; i++ {
|
||||
if err := st.SaveReport("c1", []byte(reportNoOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := st.SaveOneTimeSecret("c1", "x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// consumed FAR in the future relative to the rows above → zero reports since
|
||||
if err := st.SetOneTimeSecretTimesForTest("c1", "2099-01-01 00:00:00", "2099-01-01 00:00:01"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := DeliveryStateFor(st, "c1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.State != DeliveryConsumedAwaitingApply || got.ReportsSinceConsume != 0 {
|
||||
t.Fatalf("state/count = %s/%d, want consumed_awaiting_apply/0 (pre-consume reports must not count)",
|
||||
got.State, got.ReportsSinceConsume)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeliveryState_StagedAwaitingConsume(t *testing.T) {
|
||||
st := newDeliveryStore(t)
|
||||
if err := st.SaveReport("c1", []byte(reportNoOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveOneTimeSecret("c1", "x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := DeliveryStateFor(st, "c1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.State != DeliveryStagedAwaitingConsume {
|
||||
t.Fatalf("state = %s, want staged_awaiting_consume", got.State)
|
||||
}
|
||||
if got.Since.IsZero() {
|
||||
t.Fatal("staged state must carry created_at (mióta)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeliveryState_NoSecret(t *testing.T) {
|
||||
st := newDeliveryStore(t)
|
||||
if err := st.SaveReport("c1", []byte(reportNoOffsite)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := DeliveryStateFor(st, "c1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.State != DeliveryNoSecret {
|
||||
t.Fatalf("state = %s, want no_secret (mis-state must render needs-attention, never applied)", got.State)
|
||||
}
|
||||
}
|
||||
@@ -913,6 +913,77 @@ func (s *Store) SaveReport(customerID string, reportJSON []byte) error {
|
||||
return err
|
||||
}
|
||||
|
||||
// reportOffsitePresence is the minimal parse for "does this controller report carry an offsite
|
||||
// status object" — the R-70 delivery-state signal. The report builder attaches `offsite` only when
|
||||
// the box actually has an offbox target configured, so presence == applied-on-the-box.
|
||||
type reportOffsitePresence struct {
|
||||
Offsite json.RawMessage `json:"offsite"`
|
||||
}
|
||||
|
||||
func reportHasOffsite(reportJSON string) bool {
|
||||
var p reportOffsitePresence
|
||||
if err := json.Unmarshal([]byte(reportJSON), &p); err != nil {
|
||||
return false // unparseable report → no offsite evidence
|
||||
}
|
||||
return len(p.Offsite) > 0 && string(p.Offsite) != "null"
|
||||
}
|
||||
|
||||
// LatestReportOffsitePresence reports whether the customer's most recent controller report exists
|
||||
// and whether it carries an offsite status object (R-70 detector input).
|
||||
func (s *Store) LatestReportOffsitePresence(customerID string) (found bool, receivedAt time.Time, hasOffsite bool, err error) {
|
||||
var recv, reportJSON string
|
||||
err = s.db.QueryRow(`SELECT received_at, report_json FROM reports WHERE customer_id = ? ORDER BY id DESC LIMIT 1`,
|
||||
customerID).Scan(&recv, &reportJSON)
|
||||
if err == sql.ErrNoRows {
|
||||
return false, time.Time{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return false, time.Time{}, false, err
|
||||
}
|
||||
return true, parseSQLiteTime(recv), reportHasOffsite(reportJSON), nil
|
||||
}
|
||||
|
||||
// CountReportsOffsiteSince counts the customer's controller reports received strictly after `since`
|
||||
// (UTC) and how many of them carry an offsite status object (R-70 detector input: "N consecutive
|
||||
// reports since consume without offbox" == total>0 && withOffsite==0). Capped at 500 rows per call —
|
||||
// far beyond any detector threshold; the cap only bounds memory.
|
||||
func (s *Store) CountReportsOffsiteSince(customerID string, since time.Time) (total, withOffsite int, err error) {
|
||||
rows, err := s.db.Query(`SELECT report_json FROM reports WHERE customer_id = ? AND received_at > ? ORDER BY id LIMIT 500`,
|
||||
customerID, since.UTC().Format("2006-01-02 15:04:05"))
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var reportJSON string
|
||||
if err := rows.Scan(&reportJSON); err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
total++
|
||||
if reportHasOffsite(reportJSON) {
|
||||
withOffsite++
|
||||
}
|
||||
}
|
||||
return total, withOffsite, rows.Err()
|
||||
}
|
||||
|
||||
// LastEventAt returns the created_at of the most recent event of the given type for a customer
|
||||
// (zero time when none). Durable across hub restarts — used as the cooldown/rate-limit source for
|
||||
// hub-emitted detector events (R-70/R-71c: a repeating pattern must surface as repeating events on
|
||||
// a bounded cadence, never as a silent retry loop OR a restart-reset flood).
|
||||
func (s *Store) LastEventAt(customerID, eventType string) (time.Time, error) {
|
||||
var createdAt string
|
||||
err := s.db.QueryRow(`SELECT created_at FROM events WHERE customer_id = ? AND event_type = ? ORDER BY id DESC LIMIT 1`,
|
||||
customerID, eventType).Scan(&createdAt)
|
||||
if err == sql.ErrNoRows {
|
||||
return time.Time{}, nil
|
||||
}
|
||||
if err != nil {
|
||||
return time.Time{}, err
|
||||
}
|
||||
return parseSQLiteTime(createdAt), nil
|
||||
}
|
||||
|
||||
// GetCustomers returns the latest report summary for each customer.
|
||||
func (s *Store) GetCustomers() ([]CustomerSummary, error) {
|
||||
rows, err := s.db.Query(`
|
||||
@@ -1249,6 +1320,47 @@ func (s *Store) ConsumeOneTimeSecret(customerID string) (string, error) {
|
||||
return value, nil
|
||||
}
|
||||
|
||||
// OneTimeSecretInfo is the delivery-state metadata of a customer's one-time offsite secret —
|
||||
// timestamps ONLY, the value column is deliberately never selected (R-70 detector input; the
|
||||
// customer-scoped sibling of PBSDRHealStates' SecretUnconsumedFor).
|
||||
type OneTimeSecretInfo struct {
|
||||
CustomerID string
|
||||
CreatedAt time.Time
|
||||
ConsumedAt time.Time // zero = staged, not yet consumed
|
||||
}
|
||||
|
||||
// GetOneTimeSecretInfo returns the timestamps of a customer's one-time offsite secret row, or nil
|
||||
// when none exists. Never reads the value.
|
||||
func (s *Store) GetOneTimeSecretInfo(customerID string) (*OneTimeSecretInfo, error) {
|
||||
var createdAt string
|
||||
var consumedAt sql.NullString
|
||||
err := s.db.QueryRow(`SELECT created_at, consumed_at FROM one_time_secrets WHERE customer_id = ?`,
|
||||
customerID).Scan(&createdAt, &consumedAt)
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
info := &OneTimeSecretInfo{CustomerID: customerID, CreatedAt: parseSQLiteTime(createdAt)}
|
||||
if consumedAt.Valid {
|
||||
info.ConsumedAt = parseSQLiteTime(consumedAt.String)
|
||||
}
|
||||
return info, nil
|
||||
}
|
||||
|
||||
// SetOneTimeSecretTimesForTest back-dates a one-time secret's timestamps (SQLite datetime strings;
|
||||
// consumedAt "" leaves it NULL) so grace/age behavior is testable without sleeping. TEST-ONLY —
|
||||
// mirrors SetHostPBSSecretCreatedAtForTest.
|
||||
func (s *Store) SetOneTimeSecretTimesForTest(customerID, createdAt, consumedAt string) error {
|
||||
if consumedAt == "" {
|
||||
_, err := s.db.Exec(`UPDATE one_time_secrets SET created_at = ?, consumed_at = NULL WHERE customer_id = ?`, createdAt, customerID)
|
||||
return err
|
||||
}
|
||||
_, err := s.db.Exec(`UPDATE one_time_secrets SET created_at = ?, consumed_at = ? WHERE customer_id = ?`, createdAt, consumedAt, customerID)
|
||||
return err
|
||||
}
|
||||
|
||||
// ListCustomerConfigs returns all customer configurations ordered by ID.
|
||||
func (s *Store) ListCustomerConfigs() ([]CustomerConfig, error) {
|
||||
rows, err := s.db.Query(`
|
||||
|
||||
@@ -518,6 +518,80 @@ type configFormView struct {
|
||||
Error string
|
||||
CSRFField template.HTML
|
||||
PBSDR pbsDRView
|
||||
Delivery *deliveryView
|
||||
}
|
||||
|
||||
// deliveryView is the R-70 customer-card rendering of offsite.DeliveryStateFor — the hub's real
|
||||
// delivery knowledge replacing the static "delivered to the controller once" copy that let a
|
||||
// burned credential hide for 2 days (DIAG-f10-demo-hp-offsite-2026-07-23). All strings are
|
||||
// precomputed operator-tier English (this page's existing language).
|
||||
type deliveryView struct {
|
||||
State string // offsite.DeliveryState (template branch key + test anchor)
|
||||
Badge string // short badge text
|
||||
BadgeClass string // n-ok | n-warn | n-neutral
|
||||
Line string // the state sentence, with age ("mióta" — every state carries its timestamp)
|
||||
StaleLine string // non-empty ONLY for applied+stale-staged (the demo-felhom shape)
|
||||
}
|
||||
|
||||
// agoHuman renders a duration as a coarse operator-friendly age.
|
||||
func agoHuman(d time.Duration) string {
|
||||
switch {
|
||||
case d < time.Minute:
|
||||
return "under a minute"
|
||||
case d < time.Hour:
|
||||
return fmt.Sprintf("%d min", int(d.Minutes()))
|
||||
case d < 48*time.Hour:
|
||||
return fmt.Sprintf("%.1f h", d.Hours())
|
||||
default:
|
||||
return fmt.Sprintf("%d days", int(d.Hours()/24))
|
||||
}
|
||||
}
|
||||
|
||||
// deliveryViewFor derives the card view; nil for brand-new configs (nothing staged yet) or on a
|
||||
// detector error (the card then simply omits the state line — never a fabricated one).
|
||||
func (s *Server) deliveryViewFor(customerID string) *deliveryView {
|
||||
if customerID == "" {
|
||||
return nil
|
||||
}
|
||||
now := time.Now()
|
||||
status, err := offsite.DeliveryStateFor(s.store, customerID)
|
||||
if err != nil {
|
||||
s.logger.Printf("[WARN] delivery view %s: %v", customerID, err)
|
||||
return nil
|
||||
}
|
||||
v := &deliveryView{State: string(status.State)}
|
||||
age := agoHuman(now.Sub(status.Since))
|
||||
switch status.State {
|
||||
case offsite.DeliveryApplied:
|
||||
v.Badge, v.BadgeClass = "applied", "n-ok"
|
||||
v.Line = "offsite active on the box (last report " + age + " ago)"
|
||||
if !status.StaleStagedSince.IsZero() {
|
||||
v.StaleLine = fmt.Sprintf("Note: an unconsumed one-time secret has been staged since %s (%s ago) — superseded by the working install (key-auth-first never consumes); harmless, replaced by the next re-issue.",
|
||||
status.StaleStagedSince.UTC().Format("2006-01-02 15:04 UTC"), agoHuman(now.Sub(status.StaleStagedSince)))
|
||||
}
|
||||
case offsite.DeliveryConsumedAwaitingApply:
|
||||
// Amber past 30 min: consume→apply is a seconds-scale hop; half an hour of it is the
|
||||
// burned-credential shape taking form (the monitor turns it into an event at 1 h).
|
||||
v.Badge = "consumed"
|
||||
if now.Sub(status.Since) > 30*time.Minute {
|
||||
v.BadgeClass = "n-warn"
|
||||
v.Line = fmt.Sprintf("password consumed %s ago and the box still reports no offsite target — likely burned mid-apply; Re-issue delivers a fresh one", age)
|
||||
} else {
|
||||
v.BadgeClass = "n-neutral"
|
||||
v.Line = "password consumed — apply in progress (" + age + ")"
|
||||
}
|
||||
case offsite.DeliveryStagedAwaitingConsume:
|
||||
v.Badge, v.BadgeClass = "staged", "n-neutral"
|
||||
v.Line = "one-time password staged " + age + " ago — the box consumes it on its next config refresh"
|
||||
if now.Sub(status.Since) > 30*time.Minute {
|
||||
v.BadgeClass = "n-warn"
|
||||
v.Line = "one-time password staged " + age + " ago and NOT yet consumed — the box has not re-pulled its config (is it reporting?)"
|
||||
}
|
||||
default: // DeliveryNoSecret
|
||||
v.Badge, v.BadgeClass = "missing", "n-warn"
|
||||
v.Line = "offsite is enabled but no credential is staged — needs attention (Re-issue stages a fresh one)"
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// configFormData assembles the config form's view model. overrides carries SUBMITTED form values
|
||||
@@ -538,6 +612,7 @@ func (s *Server) configFormData(r *http.Request, isNew bool, cfg *store.Customer
|
||||
Error: errMsg,
|
||||
CSRFField: s.csrfField(r),
|
||||
PBSDR: s.pbsDRViewFor(cfg.CustomerID, cfg.DRTier),
|
||||
Delivery: s.deliveryViewFor(cfg.CustomerID),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// R-70 card render tests — one per template branch (the v0.70.1 seam-wiring lesson: a
|
||||
// conditional affordance ships with a render test per branch of its gate; handler tests prove
|
||||
// nothing about reachability). The card replaces the static "delivered to the controller once"
|
||||
// copy, so the tests also pin that the old static claim is GONE.
|
||||
|
||||
func renderConfigFormWith(t *testing.T, delivery *deliveryView) string {
|
||||
t.Helper()
|
||||
s, _ := newTestServer(t)
|
||||
data := configFormView{
|
||||
Config: &store.CustomerConfig{CustomerID: "c1"},
|
||||
Overrides: map[string]interface{}{
|
||||
"offsite": map[string]interface{}{
|
||||
"enabled": true, "type": "shared", "host": "u-sub1.example.de",
|
||||
"user": "u-sub1", "repo_path": "/home/felhom-repo",
|
||||
},
|
||||
},
|
||||
ActiveNav: "configs",
|
||||
Delivery: delivery,
|
||||
}
|
||||
var b strings.Builder
|
||||
if err := s.templates.ExecuteTemplate(&b, "config_form.html", data); err != nil {
|
||||
t.Fatalf("render: %v", err)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func TestDeliveryCard_AppliedRenders(t *testing.T) {
|
||||
out := renderConfigFormWith(t, &deliveryView{
|
||||
State: "applied", Badge: "applied", BadgeClass: "n-ok",
|
||||
Line: "offsite active on the box (last report 2 min ago)",
|
||||
})
|
||||
if !strings.Contains(out, `data-delivery-state="applied"`) || !strings.Contains(out, "offsite active on the box") {
|
||||
t.Fatalf("applied state not rendered:\n%s", out)
|
||||
}
|
||||
if !strings.Contains(out, `class="n n-ok"`) {
|
||||
t.Fatal("applied badge must use n-ok")
|
||||
}
|
||||
if strings.Contains(out, "the transient password is delivered to the controller once") {
|
||||
t.Fatal("the old static 'delivered once' claim must be GONE — it hid a burned credential for 2 days (DIAG-f10)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeliveryCard_ConsumedAwaitingApplyAmberRenders(t *testing.T) {
|
||||
out := renderConfigFormWith(t, &deliveryView{
|
||||
State: "consumed_awaiting_apply", Badge: "consumed", BadgeClass: "n-warn",
|
||||
Line: "password consumed 2.0 h ago and the box still reports no offsite target — likely burned mid-apply; Re-issue delivers a fresh one",
|
||||
})
|
||||
if !strings.Contains(out, `data-delivery-state="consumed_awaiting_apply"`) ||
|
||||
!strings.Contains(out, "likely burned mid-apply") || !strings.Contains(out, `class="n n-warn"`) {
|
||||
t.Fatalf("amber consumed_awaiting_apply state not rendered:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeliveryCard_StagedRenders(t *testing.T) {
|
||||
out := renderConfigFormWith(t, &deliveryView{
|
||||
State: "staged_awaiting_consume", Badge: "staged", BadgeClass: "n-neutral",
|
||||
Line: "one-time password staged 3 min ago — the box consumes it on its next config refresh",
|
||||
})
|
||||
if !strings.Contains(out, `data-delivery-state="staged_awaiting_consume"`) ||
|
||||
!strings.Contains(out, "the box consumes it on its next config refresh") {
|
||||
t.Fatalf("staged state not rendered:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
// The demo-felhom shape: applied + the stale-staged info line — BOTH must render.
|
||||
func TestDeliveryCard_AppliedWithStaleStagedInfoLine(t *testing.T) {
|
||||
out := renderConfigFormWith(t, &deliveryView{
|
||||
State: "applied", Badge: "applied", BadgeClass: "n-ok",
|
||||
Line: "offsite active on the box (last report 2 min ago)",
|
||||
StaleLine: "Note: an unconsumed one-time secret has been staged since 2026-07-21 08:29 UTC (2 days ago) — superseded by the working install (key-auth-first never consumes); harmless, replaced by the next re-issue.",
|
||||
})
|
||||
if !strings.Contains(out, `data-delivery-state="applied"`) ||
|
||||
!strings.Contains(out, "an unconsumed one-time secret has been staged since 2026-07-21") {
|
||||
t.Fatalf("applied + stale-staged info line not rendered:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
// Nil Delivery (brand-new config / detector error): the card is simply absent — never a
|
||||
// fabricated state line.
|
||||
func TestDeliveryCard_NilDeliveryOmitted(t *testing.T) {
|
||||
out := renderConfigFormWith(t, nil)
|
||||
if strings.Contains(out, "data-delivery-state") {
|
||||
t.Fatalf("nil Delivery must render no state line:\n%s", out)
|
||||
}
|
||||
// the target line survives (it is descriptor truth, not delivery state)
|
||||
if !strings.Contains(out, "u-sub1@u-sub1.example.de:/home/felhom-repo") {
|
||||
t.Fatal("provisioned target line must still render")
|
||||
}
|
||||
}
|
||||
|
||||
// deliveryViewFor derives from the real detector: the burned shape past 30 min renders amber.
|
||||
func TestDeliveryViewFor_BurnedShapeGoesAmber(t *testing.T) {
|
||||
s, st := newTestServer(t)
|
||||
if err := st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c9", APIKey: "k", RetrievalPassword: "p"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveOneTimeSecret("c9", "x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SetOneTimeSecretTimesForTest("c9", "2026-01-01 00:00:00", "2026-01-01 00:03:00"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveReport("c9", []byte(`{"health":{"status":"ok"}}`)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
v := s.deliveryViewFor("c9")
|
||||
if v == nil || v.State != "consumed_awaiting_apply" || v.BadgeClass != "n-warn" {
|
||||
t.Fatalf("view = %+v, want consumed_awaiting_apply with n-warn (amber past 30 min)", v)
|
||||
}
|
||||
if !strings.Contains(v.Line, "Re-issue") {
|
||||
t.Fatalf("amber line must point at the remedy, got %q", v.Line)
|
||||
}
|
||||
}
|
||||
@@ -117,7 +117,14 @@
|
||||
</div>
|
||||
</div>
|
||||
{{with .Overrides}}{{with index . "offsite"}}{{if index . "host"}}
|
||||
<p class="form-hint" style="margin-top:.5rem">Provisioned: {{index . "user"}}@{{index . "host"}}:{{index . "repo_path"}} — the transient password is delivered to the controller once (never shown here).</p>
|
||||
<p class="form-hint" style="margin-top:.5rem">Provisioned: {{index . "user"}}@{{index . "host"}}:{{index . "repo_path"}} (the transient password is never shown here).</p>
|
||||
<!-- R-70: the delivery state replaces the old static "delivered once" claim — the hub
|
||||
now RENDERS what it knows (one_time_secrets × report offbox presence) instead of
|
||||
asserting a delivery it never checked (DIAG-f10: a burned credential hid 2 days). -->
|
||||
{{with $.Delivery}}
|
||||
<p class="form-hint" style="margin-top:.35rem" data-delivery-state="{{.State}}"><span class="n {{.BadgeClass}}">{{.Badge}}</span> {{.Line}}</p>
|
||||
{{if .StaleLine}}<p class="form-hint" style="margin-top:.35rem">{{.StaleLine}}</p>{{end}}
|
||||
{{end}}
|
||||
<!-- F4: explicit operator recovery for a consumed-password dead-end (fresh-guest DR).
|
||||
Rides the parent form (nested forms are invalid HTML) via formaction; the _csrf field
|
||||
submits with it. Resets the box credential + stages a fresh one-time password. -->
|
||||
|
||||
Reference in New Issue
Block a user