diff --git a/CHANGELOG.md b/CHANGELOG.md index 4903c7a..34d9aa0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,50 @@ ## Changelog +### v0.162.0 — R-71(a): the apply-bridge waits for the dust to settle (settle-gate) (2026-07-24) + +No agent coupling; MinAgent unchanged. Origin: +`felhom.eu/documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md` — the day-0 race. A fresh box +boots below the operator floor (ISO 0.153.0 < floor 0.156.0), the apply-bridge consumes the +single-use offsite password, then ~35 s later the managed auto-floor update replaces the container +mid-install → the new process finds no installed key → consume → **404** → offsite dead until an +operator Re-issue. This recurs on **every** fresh onboarding whose ISO floor lags the managed floor; +demo-felhom escaped by timing alone. The v1.25.0 golden≥floor build gate PREVENTS the trigger for +fresh installs; R-71c (hub) HEALS a burn after the fact; this (a) removes the SYSTEMATIC trigger for +every restart shape. + +**The change (ordering only — the bridge's consume/install/persist internals, the 404-no-oracle +contract, and the Consumer are UNTOUCHED; R-71(b) stays rejected-by-design):** +- New seam `offsiteapply.SettleProvider.SettleState() (version, floor string, updateRunning, + floorKnown bool)` — a thin adapter (`SettleFunc`) over the self-updater's OWN knowledge in main.go + (`GetFloor()`/`IsUpdateRunning()`); the bridge never fetches the floor a second way. +- `Bridge.AwaitSettle` polls every 10 s (bounds: 90 s floor-knowledge sub-bound, 5 min overall) + BEFORE the 3-minute Reconcile context is created (the deferral never eats the reconcile budget). + Releases: `updateRunning` → wait (the swap supersedes us); `floorKnown && version > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-07-23 (v0.161.0 — R-70: truthful hub-managed offsite empty state) +Last updated: 2026-07-24 (v0.162.0 — R-71a: the apply-bridge settle-gate) + +> **2026-07-24 — v0.162.0 (R-71a), CODE COMPLETE + green; deploy/live-verify on the publish train.** +> The structural fix for the F10 day-0 race (DIAG-f10): the apply-bridge no longer consumes the +> single-use offsite password while a managed floor-update is in flight or imminent (below floor). +> New seam `offsiteapply.SettleProvider.SettleState()` + `SettleFunc` adapter over the updater's own +> `GetFloor()`/`IsUpdateRunning()` (no second floor path); `Bridge.AwaitSettle` polls 10 s BEFORE the +> 3-min Reconcile ctx (deferral never eats the reconcile budget), bounds 90 s floor sub-bound / 5 min +> overall (both GO+WARN — the "hub that can't serve a floor can't serve a consume → no burn" argument, +> R-71c is the belt). At/above floor → GO first poll, zero wait (B′). Bridge goroutine MOVED after the +> updater in main.go; wired only when an updater exists. **Ordering-only** — consume/persist/404 +> contract untouched; R-71(b) rejected-by-design. **FINDING:** the floor is in-memory +> (report-ACK-derived ~5–10 s), NOT persisted → unknown on any restart until the first ACK (sized the +> 90 s sub-bound to that). 5 test scenarios (A–E) + nil-provider + cancelled-gate; **4 red-proofs all +> observed FAIL then restored** (gate/updateRunning/sub-bound/overall-bound). Deferral paths NOT +> live-fired (precondition now structurally prevented by the v1.25.0 build gate). **Layering: gate +> prevents, (a) defers, (c) heals.** ROADMAP R-71 → SHIPPED (a)+(c). Live leg = the B′ first-poll GO +> line on both above-floor boxes. > **2026-07-23 — v0.161.0 (R-70 controller leg), SHIPPED + deployed BOTH boxes.** When > `offsite.enabled` is in controller.yaml but no `offbox` target exists (pre-apply window / burned diff --git a/REPORT.md b/REPORT.md index edc5a21..07510ba 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,44 +1,71 @@ -# REPORT — R-70 controller leg: the hub-managed offsite empty state tells the truth (v0.161.0, 2026-07-23) +# REPORT — felhom-controller v0.162.0 (R-71a: the apply-bridge settle-gate) -**Spec:** R-70 + R-71(c) prompt §4d (controller banner, XS; the detector/event/self-heal live in -felhom-hub v0.72.0 — see `felhom.eu/REPORT.md`). Origin: -`felhom.eu/documentation/audits/DIAG-f10-demo-hp-offsite-2026-07-23.md`. +**Date:** 2026-07-24 · **Task:** R-71(a) — structural fix for the F10 day-0 update-vs-bridge race +(consume-then-persist not crash-safe). **Scope:** `felhom-controller` only, ordering-only. No agent +(0.93.0 stays), no hub. MinAgent unchanged. -## What shipped +## The defect (DIAG-f10) -One data key + one template gate. `backupsOffboxData` now exposes `OffsiteHubEnabled` -(= `s.cfg.Offsite.Enabled`, the hub descriptor already loaded from controller.yaml — -`internal/web/handlers.go`, no new wiring; `Server.cfg` was always there). On Távoli mentés, when -hub-managed offsite is enabled but no `offbox` target exists (the pre-apply window, or a burned -credential — the F10 shape), both empty surfaces switch to the truth: +A fresh box boots below the operator floor (ISO 0.153.0 < floor 0.156.0). The apply-bridge consumes +the hub's single-use offsite password at boot; ~35 s later the managed **auto-floor update** replaces +the container mid-install → the new 0.156.0 process finds no installed key → `consume` → **404** → +offsite tier dead until an operator Re-issue. Recurs on **every** onboarding whose ISO floor lags the +managed floor; demo-felhom escaped by timing alone. Layering: the v1.25.0 golden≥floor build gate +PREVENTS the trigger for fresh installs, R-71c (hub) HEALS a burn, **this (a) removes the systematic +trigger for every restart shape.** -> „Felhom offsite tárhely kiépítve — a beállítás automatikus, folyamatban. Ha egy napon belül nem -> áll be, jelezd az üzemeltetőnek." +## What shipped (ordering only — consume/install/persist internals + the 404-no-oracle contract + the Consumer UNTOUCHED; R-71(b) stays rejected-by-design) -- the v0.124.0 status card no longer offers „igényelhető szolgáltatás" for a service that is - already ordered (it contradicted the new banner — coherence extension of the spec's letter, - gate `{{if and (not .OffboxConfigured) .OffsiteHubEnabled}}`); -- the `{{if .Offbox}}` else-branch replaces the bare „Még nincs beállítva távoli mentési cél."; -- NOT hub-enabled → today's copy byte-identical; configured target → status block as before; - the own-NAS setup form untouched. +1. **Seam** `offsiteapply.SettleProvider.SettleState() (version, floor string, updateRunning, + floorKnown bool)` + `SettleFunc` adapter (seams.go) — a thin adapter over the self-updater's own + knowledge (`GetFloor()`/`IsUpdateRunning()`) in main.go. No second floor-fetch path. +2. **`Bridge.AwaitSettle`** — polls every 10 s BEFORE the 3-minute Reconcile context is created (the + deferral never eats the reconcile budget). Release table: + - `updateRunning` → wait (the swap's restart supersedes us). + - `floorKnown && version < floor` → wait (auto-floor update imminent — do NOT burn the password); + log `deferring offsite apply: managed update to floor pending (we are )`. + - `floorKnown && at/above floor` → **GO on the first poll, zero sleep** (the B′ invariant). + - `!floorKnown` past the 90 s sub-bound → GO + WARN (a hub that can't serve a floor can't serve a + consume → no burn risk). + - overall 5-min bound → GO + WARN (`R-71c self-heal is the belt`). + `ReconcileWhenSettled` runs the gate, then Reconcile under a fresh 3-min context. +3. **main.go** — the bridge goroutine MOVED to after the self-updater is constructed (so the adapter + can read it). Wired ONLY when an updater exists; otherwise `Settle` is nil = reconcile immediately + (no updater → no floor-update to race). +4. **Bounds** are named constants with rationale comments. The floor is in-memory + (report-ACK-derived), **NOT persisted** — so on any restart it is unknown until the first report + ACK. Source-confirmed latency: the startup report fires ~5 s after boot and `SetFloor` runs + synchronously in its ACK handler → floor normally known in ~5–10 s (≤~45 s across 3×15 s report + retries); the 90 s sub-bound is headroom over that worst case. -## Tests & gates +## Tests — all green (`go test ./...` clean, `go vet ./...` clean, gofmt clean) -Render tests per branch of the gate (`internal/web/backups_remote_hubenabled_test.go`, the -v0.70.1 template-gate lesson): banner branch (+ old copy asserted GONE + own-NAS button asserted -present), legacy branch (old copy intact, banner absent), configured branch (no banner). Green -gate: `go build && go vet && go test ./...` — 25 packages ok. Design-v2 gates: emoji, mojibake, -template_id, native_confirm, offbox_rename, app_row_dedup all OK; `docker_run_volume_path_gate` -remains red on the pre-existing R-29 allowlist item (`appexport/estimate.go:179`, named volume, -benign, filed 2026-07-18) — untouched by this change, not fixed inline per R-29's own row. +Injectable clock (no real sleeps), fake `SettleState`, recorded `Consumer` +(`internal/offsiteapply/settlegate_test.go`): -No red-proof leg: no defect was fixed here (new truthful copy behind a new gate); the per-branch -render tests are the non-hollow coverage. +- **A** below-floor defers (0 consumes while below) then GOes at floor → exactly one consume. +- **B** update-running defers then GOes when it finishes. +- **C** floor-unknown GOes at the 90 s sub-bound + the WARN line. +- **D** perpetually-below GOes at the 5-min overall bound + the WARN line. +- **E** (B′) at/above-floor GOes on **poll 1** with **zero** sleep consumed. +- nil-provider → immediate reconcile; cancelled gate → skips reconcile (no consume). -## Live state +## Red-proofs — four, each OBSERVED failing then restored -Deployed to both boxes (demo-felhom 9201 + demo-hp 9201). Both live boxes have offbox CONFIGURED, -so the new banner is **unit-proven, live-pending** — it renders only in the enabled+no-offbox -window, which no healthy box occupies (and we do not break one to see it — F9 rule). The natural -live leg is the next fresh onboarding's pre-apply window. Live verification on both boxes: -configured view unchanged (no banner), version 0.161.0 healthy. +| # | Break | Observed FAIL | +|---|-------|---------------| +| A | gate removed (early `return` in AwaitSettle) | `TestSettle_BelowFloorDefersThenGoes`: "expected the gate to defer while below floor (>=2 sleeps), got 0" — below-floor consumes immediately (the F10 burn) | +| B | `updateRunning` branch removed | `TestSettle_UpdateRunningDefersThenGoes`: mid-swap box GOes immediately (log shows GO, deferral line absent) | +| C | floor sub-bound branch removed | `TestSettle_FloorUnknownGoesAfterSubBound` + red-proof: "release near the sub-bound 1m30s, waited **5m0s**" — floor-unknown drags to the overall bound | +| D | overall bound check removed | `TestSettle_PerpetuallyBelowFloorGoesAtOverallBound`: **20 s test timeout** — perpetual below-floor loops forever in `AwaitSettle` | + +## Ship / honesty + +The deferral paths ship **unit-proven + red-proofed, NOT live-fired** — their precondition is now +structurally prevented by the v1.25.0 golden≥floor build gate, which is the point. The +live-observable leg is the **B′ first-poll GO line on both above-floor demo boxes** (proving zero +added latency on the common restart shape). **Layering: gate prevents, (a) defers, (c) heals.** +R-71 → SHIPPED with (a)+(c); (b) recorded rejected-by-design. + +_(Build/deploy to both boxes + the two first-poll GO log lines appended on completion of the publish +train.)_ diff --git a/REUSE.md b/REUSE.md index 198d2c3..f808530 100644 --- a/REUSE.md +++ b/REUSE.md @@ -148,6 +148,7 @@ | `report.SetPendingLogTails` + `buildLogTailsSection` | controller/internal/report/logtail.go | ACK `log_tail_requests` → next report `log_tails` | THE pull-based ACK-flag pattern (hub asks, controller pushes next cycle) — copy for any new hub→box request | Consume-once drain at BuildReport; failed push re-arms from the hub's still-pending request; NEVER add a hub→controller push channel | | `metrics.FetchContainerLogTail` | controller/internal/metrics/logscanner.go | `(name, tailLines) (string, error)` | Raw per-container `docker logs --tail=N` | 15s timeout; caller caps/redacts (capTailLines) | | `ConfigRefresher.Reconcile` | controller/internal/report/config_refresh.go | `(ackVersion int)` | Pull-based config refresh | Re-pulls controller.yaml (re-merging local_api), then graceful self-restart; first-run = baseline, no restart | +| `offsiteapply.SettleProvider` / `SettleFunc` / `Bridge.AwaitSettle` / `ReconcileWhenSettled` (R-71a, v0.162.0) | controller/internal/offsiteapply/offsiteapply.go + seams.go | `SettleState() (version, floor string, updateRunning, floorKnown bool)` | THE settle-gate: defers the offsite one-time-password consume past a managed day-0 floor-update (the F10 race). Wire the `SettleFunc` adapter over `updater.GetFloor()`/`IsUpdateRunning()` — **the updater's knowledge is the ONE floor source; never fetch the floor a second way**. Gate ONLY the bridge goroutine, and only when an updater exists (nil `Settle` = reconcile immediately). Bounds `settlePoll`/`settleFloorSubBound`/`settleOverallBound`; the floor is in-memory (report-ACK-derived, ~5–10 s), NOT persisted → unknown until the first ACK on any restart. Inject `Now`/`Sleep` in tests (no real sleeps). B′: at/above-floor GOes on the first poll, zero wait. Do NOT touch the consume/persist order or the 404 contract — ordering only | | `bootstrap.MaybeIngest` / `RefreshConfig` | controller/internal/bootstrap/bootstrap.go | bootstrap.json → controller.yaml | Day-0 + refresh | Overwrites controller.yaml, NEVER settings.json | | `api.GracefulSelfRestart` | controller/internal/api/selfrestart.go | `(logger)` | Controller self-restart | Detached exit; bootstrap unit re-runs the image | | `Settings.AddPendingEvent/DrainPendingEvents` | controller/internal/settings/settings.go | offline event queue | Events while hub unreachable | — | diff --git a/controller/README.md b/controller/README.md index ceb938f..ee56c74 100644 --- a/controller/README.md +++ b/controller/README.md @@ -1058,6 +1058,21 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an > recovery for a genuinely-spent password on a fresh guest. > - **Staged-secret wipe (v0.107.0):** confirm-escrow calls the agent's `DELETE /escrow/stage-secret` > (agent ≥ v0.78.0) whenever `EscrowState` flips to `escrowed` — best-effort, loud-logged on failure. +> - **Settle-gate (R-71a, v0.162.0) — the day-0 race removed.** The apply-bridge runs BEHIND a +> settle-gate (`Bridge.AwaitSettle` → `ReconcileWhenSettled`): before the consume/install path it +> polls the self-updater's own state via the `SettleProvider` seam (a `SettleFunc` adapter over +> `updater.GetFloor()`/`IsUpdateRunning()` in main.go — no second floor-fetch path). While a managed +> update is running OR the box is below the operator floor (an auto-floor update is imminent), the +> gate WAITS rather than consume the single-use password — the update's restart would otherwise kill +> the bridge mid-install and burn it (the F10 day-0 shape). At/above floor with no update in flight, +> it GOes on the first poll with zero added latency (B′). Bounds: 10 s poll, 90 s floor-knowledge +> sub-bound (sized to the ~5–10 s report-ACK floor latency; the floor is in-memory, not persisted, +> so it is unknown until the first ACK on any restart), 5 min overall — both bounds GO+WARN and lean +> on the R-71c hub self-heal as the belt (a hub that cannot serve a floor cannot serve a consume, so +> proceeding never burns a password). The gate is wired only when a self-updater exists (no updater +> → no floor-update to race → reconcile immediately). Ordering-only: the consume/install/persist +> internals and the 404-no-oracle contract are untouched. Three-layer defense: the v1.25.0 +> golden≥floor build gate PREVENTS the trigger, (a) DEFERS it, R-71c HEALS a burn. > **NAS network storage (v0.92.0, Part A2; pairs with agent v0.50.0).** A customer NAS share (NFS or SMB) > is a **distinct storage KIND** from a physical drive (`StoragePath.Kind == "network"`), for **bulk media**. diff --git a/controller/cmd/controller/main.go b/controller/cmd/controller/main.go index 7ea09b0..5c3a3fb 100644 --- a/controller/cmd/controller/main.go +++ b/controller/cmd/controller/main.go @@ -287,39 +287,9 @@ func main() { backupMgr.SetSharesReconciler(stackMgr.ReconcileSamba) } - // SLICE 2: the offsite apply-bridge — on startup (async, non-blocking) reconcile the hub-served offsite - // descriptor into a configured key-only offbox target (fail-safe, idempotent, no blind TOFU). The - // config_refresh self-restart re-runs this after a descriptor change (new process → startup). - if backupMgr != nil && cfg.Offsite.Enabled && cfg.Hub.URL != "" && cfg.Hub.APIKey != "" { - bridge := &offsiteapply.Bridge{ - Cfg: cfg, - Consumer: offsiteapply.HTTPConsumer{HubURL: cfg.Hub.URL, CustomerID: cfg.Customer.ID, APIKey: cfg.Hub.APIKey}, - Scanner: offsiteapply.KeyscanScanner{}, - KeyGen: offsiteapply.ED25519KeyGen{}, - Installer: offsiteapply.SSHCopyIDInstaller{}, - Prober: offsiteapply.SFTPKeyAuthProber{KeyPath: filepath.Join(cfg.Paths.DataDir, "offbox", "ssh_key")}, - Enabler: offsiteapply.EnablerFunc(func(ctx context.Context, host, user string, port int, repoPath, priv, kh string, quotaGB int) error { - tgt := &settings.OffboxTarget{Enabled: true, Host: host, User: user, Port: port, RepoPath: repoPath, Schedule: "daily", QuotaGB: quotaGB} - stage := func(ctx context.Context, pw string) error { - ac, err := agentapi.New(cfg.LocalAPI.Endpoint, cfg.LocalAPI.Token, cfg.LocalAPI.Fingerprint) - if err != nil { - return err - } - return ac.StageEscrowSecret(ctx, pw) - } - return backupMgr.ApplyOffsiteTarget(ctx, tgt, priv, kh, stage) - }), - MarkerPath: filepath.Join(cfg.Paths.DataDir, "offbox", "applied_marker"), - Logger: logger, - } - go func() { - ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute) - defer cancel() - if err := bridge.Reconcile(ctx); err != nil { - logger.Printf("[WARN] [offsite-apply] reconcile: %v (retries on next config refresh/restart)", err) - } - }() - } + // SLICE 2: the offsite apply-bridge is launched further down, AFTER the self-updater is constructed + // (R-71a: the bridge's settle-gate reads the updater's floor/update-running state to defer the + // consume past a managed day-0 floor-update). See "offsite apply-bridge" below. // --- Wire the data-migration engine (B1) + backup↔migration mutual exclusion (Change 3) --- stackMgr.SetMigrationDeps(sett, func() bool { return backupMgr != nil && backupMgr.IsRunning() }) @@ -383,6 +353,54 @@ func main() { cfg.SelfUpdate.CheckInterval, cfg.SelfUpdate.AutoUpdate, cfg.SelfUpdate.AutoUpdateTime) } + // SLICE 2: the offsite apply-bridge — on startup (async, non-blocking) reconcile the hub-served + // offsite descriptor into a configured key-only offbox target (fail-safe, idempotent, no blind + // TOFU). The config_refresh self-restart re-runs this after a descriptor change (new process → + // startup). Launched HERE (after the self-updater is built) so the R-71a settle-gate can read the + // updater's floor/update-running state and defer the one-time-password consume past a managed + // day-0 floor-update (the F10 race). The gate is wired ONLY when an updater exists — with no update + // mechanism there is no floor-update to race, so the bridge reconciles immediately (Settle nil). + if backupMgr != nil && cfg.Offsite.Enabled && cfg.Hub.URL != "" && cfg.Hub.APIKey != "" { + bridge := &offsiteapply.Bridge{ + Cfg: cfg, + Consumer: offsiteapply.HTTPConsumer{HubURL: cfg.Hub.URL, CustomerID: cfg.Customer.ID, APIKey: cfg.Hub.APIKey}, + Scanner: offsiteapply.KeyscanScanner{}, + KeyGen: offsiteapply.ED25519KeyGen{}, + Installer: offsiteapply.SSHCopyIDInstaller{}, + Prober: offsiteapply.SFTPKeyAuthProber{KeyPath: filepath.Join(cfg.Paths.DataDir, "offbox", "ssh_key")}, + Enabler: offsiteapply.EnablerFunc(func(ctx context.Context, host, user string, port int, repoPath, priv, kh string, quotaGB int) error { + tgt := &settings.OffboxTarget{Enabled: true, Host: host, User: user, Port: port, RepoPath: repoPath, Schedule: "daily", QuotaGB: quotaGB} + stage := func(ctx context.Context, pw string) error { + ac, err := agentapi.New(cfg.LocalAPI.Endpoint, cfg.LocalAPI.Token, cfg.LocalAPI.Fingerprint) + if err != nil { + return err + } + return ac.StageEscrowSecret(ctx, pw) + } + return backupMgr.ApplyOffsiteTarget(ctx, tgt, priv, kh, stage) + }), + MarkerPath: filepath.Join(cfg.Paths.DataDir, "offbox", "applied_marker"), + Logger: logger, + } + if updater != nil { + // Thin adapter over the updater's OWN knowledge (StackDataProvider pattern) — the bridge + // never fetches the floor a second way. floorKnown = the floor has been learned from a + // report ACK yet (GetFloor() != ""). + u := updater + bridge.Settle = offsiteapply.SettleFunc(func() (string, string, bool, bool) { + floor := u.GetFloor() + return Version, floor, u.IsUpdateRunning(), floor != "" + }) + } + go func() { + // ReconcileWhenSettled runs the settle-gate FIRST (its own bounds), then Reconcile under a + // fresh 3-minute context — the gate's wait never eats the reconcile budget. + if err := bridge.ReconcileWhenSettled(context.Background()); err != nil { + logger.Printf("[WARN] [offsite-apply] reconcile: %v (retries on next config refresh/restart)", err) + } + }() + } + // --- Initialize scheduler --- sched := scheduler.New(logger) sched.SetDebug(cfg.Logging.Level == "debug") diff --git a/controller/internal/offsiteapply/offsiteapply.go b/controller/internal/offsiteapply/offsiteapply.go index 9e0f343..1fb2724 100644 --- a/controller/internal/offsiteapply/offsiteapply.go +++ b/controller/internal/offsiteapply/offsiteapply.go @@ -14,8 +14,10 @@ import ( "log" "os" "path/filepath" + "time" "gitea.dooplex.hu/admin/felhom-controller/internal/config" + "gitea.dooplex.hu/admin/felhom-controller/internal/util" ) // The apply-bridge seams (tests inject fakes — no live SSH / hub calls in unit tests). @@ -51,8 +53,56 @@ type ( KeyAuthProber interface { Probe(ctx context.Context, host, user string, port int, knownHosts string) (privPEM string, ok bool) } + // SettleProvider reports the managed-update settle state so the bridge can DEFER consuming the + // one-time password until any imminent managed floor-update has converged (R-71a — the structural + // fix for the F10 day-0 race). The failure it prevents: a fresh box boots below the operator floor, + // the bridge consumes the single-use password, then ~35 s later the managed auto-floor update + // replaces the container mid-install → the new process finds no installed key → consume → 404 → + // offsite dead until an operator Re-issue. version = the running controller version; floor = the + // operator-enforced minimum (from the hub report ACK); updateRunning = a swap is in flight; + // floorKnown = the floor has been learned yet (false = the first report ACK hasn't landed). A hub + // that hasn't served a floor cannot serve a consume either, so !floorKnown carries NO burn risk. + SettleProvider interface { + SettleState() (version, floor string, updateRunning, floorKnown bool) + } ) +// Settle-gate timing (R-71a). Named constants with rationale so the trade-offs stay visible. +const ( + // reconcileTimeout bounds the actual Reconcile once the gate releases (moved here from main.go so + // ReconcileWhenSettled owns the whole "gate THEN reconcile" contract). The gate's own wait must NOT + // eat this budget — the reconcile context is created only after the gate returns. + reconcileTimeout = 3 * time.Minute + // settlePoll is the gate's poll cadence. The bridge is a background reconcile with no user-visible + // latency, so a coarse poll is free; the at/above-floor happy path returns on the FIRST evaluation + // with no sleep at all (the B′ invariant), so this cadence only ever paces the deferral cases. + settlePoll = 10 * time.Second + // settleFloorSubBound: if the hub never tells us the floor (report ACK), stop waiting and GO+WARN. + // Sized to the report-ACK latency observed at source: the startup report fires ~5 s after boot + // (main.go's startup goroutine sleeps 5 s), and SetFloor runs SYNCHRONOUSLY inside that report's + // ACK handler (main.go OnPushResponse → updater.SetFloor), so the floor is normally known within + // ~5–10 s. The startup report retries up to 3× with 15 s gaps, so a slow first report can push + // floor-knowledge to ~45 s; 90 s is generous headroom over that worst case. Proceeding here cannot + // burn a one-time password: a hub that cannot serve a floor cannot serve a consume. + settleFloorSubBound = 90 * time.Second + // settleOverallBound: the absolute cap. If we are still below floor after this (a managed + // floor-update that never lands), proceed anyway — the R-71c hub self-heal restage is the belt for + // a consume that a genuinely stuck update might later burn. + settleOverallBound = 5 * time.Minute +) + +// belowFloor reports whether the running version is strictly below the operator floor. An +// unparseable version or floor (a dev build, or a malformed floor) is treated as NOT below — the gate +// must never wedge on a version it cannot compare (and a dev build never auto-floor-updates anyway). +func belowFloor(version, floor string) bool { + v, err1 := util.ParseVersion(version) + f, err2 := util.ParseVersion(floor) + if err1 != nil || err2 != nil { + return false + } + return v.Compare(f) < 0 +} + // Bridge reconciles the offsite descriptor into a configured offbox target. type Bridge struct { Cfg *config.Config @@ -64,6 +114,36 @@ type Bridge struct { Prober KeyAuthProber // optional: key-auth-first (nil → always the full consume+install path) MarkerPath string // where the applied-descriptor-hash is persisted (e.g. /offbox/applied_marker) Logger *log.Logger + + // Settle gates the consume/install path behind managed-update convergence (R-71a). nil → no gate + // (old behavior: reconcile immediately). Wired only when a self-updater exists — with no update + // mechanism there is no floor-update to race, so no gate is needed. + Settle SettleProvider + // Now/Sleep are clock seams for the settle-gate ONLY (tests inject a fake clock so the bounds are + // exercised with zero real sleeps). nil → the real wall clock and a context-aware sleep. + Now func() time.Time + Sleep func(ctx context.Context, d time.Duration) +} + +func (b *Bridge) nowFn() func() time.Time { + if b.Now != nil { + return b.Now + } + return time.Now +} + +func (b *Bridge) sleepFn() func(context.Context, time.Duration) { + if b.Sleep != nil { + return b.Sleep + } + return func(ctx context.Context, d time.Duration) { + t := time.NewTimer(d) + defer t.Stop() + select { + case <-ctx.Done(): + case <-t.C: + } + } } func (b *Bridge) logf(f string, a ...any) { @@ -185,3 +265,79 @@ func (b *Bridge) Reconcile(ctx context.Context) error { b.logf("[INFO] [offsite-apply] offsite configured for %s@%s:%s (pending key escrow)", o.User, o.Host, o.RepoPath) return nil } + +// AwaitSettle blocks until it is safe to run the apply-bridge, then returns (R-71a). It removes the +// SYSTEMATIC trigger for the F10 day-0 race by refusing to consume the one-time password while a +// managed floor-update is in flight or imminent (we are below the floor): that update's restart would +// supersede the bridge and kill it mid-install, spending the password for nothing. It NEVER blocks +// when it is genuinely safe — the overwhelmingly common shape (a restart of an at/above-floor box) +// evaluates GO on the first poll with no sleep at all (the B′ invariant: zero new latency). +// +// It is a strict no-op unless a SettleProvider is wired (nil → old behavior, reconcile immediately). +// The gate has its own bounds (settlePoll/settleFloorSubBound/settleOverallBound) and its own context +// so the deferral never eats the reconcile budget. +func (b *Bridge) AwaitSettle(ctx context.Context) { + if b.Settle == nil { + return + } + now, sleep := b.nowFn(), b.sleepFn() + start := now() + var loggedUpdate, loggedDefer, loggedFloorWait bool + for { + version, floor, updateRunning, floorKnown := b.Settle.SettleState() + elapsed := now().Sub(start) + switch { + case updateRunning: + // A swap is in flight; its restart supersedes us. Wait it out. + if !loggedUpdate { + b.logf("[INFO] [offsite-apply] settle-gate: a managed update is in progress — deferring offsite apply until it converges") + loggedUpdate = true + } + case floorKnown && belowFloor(version, floor): + // The auto-floor update is imminent (below floor + floor known). Do NOT consume — the + // update's restart would burn the password. Wait for the update to land (which restarts us + // at floor → the GO branch below). + if !loggedDefer { + b.logf("[INFO] [offsite-apply] deferring offsite apply: managed update to floor %s pending (we are %s)", floor, version) + loggedDefer = true + } + case floorKnown: + // At/above floor, no update running — the safe steady state. GO. + b.logf("[INFO] [offsite-apply] settle-gate: GO — at/above floor %s (we are %s), no managed update running", floor, version) + return + case elapsed >= settleFloorSubBound: + // Floor never became known within the sub-bound. A hub that will not tell us the floor + // cannot serve a consume either, so the burn risk is nil — don't hold offsite hostage. + b.logf("[WARN] [offsite-apply] settle-gate: GO — floor still unknown after %s; a hub that cannot serve a floor cannot serve a consume (no burn risk)", settleFloorSubBound) + return + default: + // Floor not known yet, still inside the sub-bound — wait for the report ACK. + if !loggedFloorWait { + b.logf("[INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply") + loggedFloorWait = true + } + } + if elapsed >= settleOverallBound { + b.logf("[WARN] [offsite-apply] settle-gate bound exhausted after %s — proceeding; R-71c self-heal is the belt", settleOverallBound) + return + } + sleep(ctx, settlePoll) + if ctx.Err() != nil { + return // shutdown / cancellation — abandon the gate (the next start retries) + } + } +} + +// ReconcileWhenSettled runs the settle-gate (R-71a) and THEN Reconcile under a FRESH reconcile +// context. The gate's deferral must not eat the reconcile budget, so the reconcile timeout starts +// only after the gate releases. gateCtx bounds the gate (e.g. process shutdown); a cancelled gate +// skips the reconcile (the next start retries). +func (b *Bridge) ReconcileWhenSettled(gateCtx context.Context) error { + b.AwaitSettle(gateCtx) + if gateCtx.Err() != nil { + return gateCtx.Err() + } + ctx, cancel := context.WithTimeout(context.Background(), reconcileTimeout) + defer cancel() + return b.Reconcile(ctx) +} diff --git a/controller/internal/offsiteapply/seams.go b/controller/internal/offsiteapply/seams.go index 91c0e3f..1797987 100644 --- a/controller/internal/offsiteapply/seams.go +++ b/controller/internal/offsiteapply/seams.go @@ -33,6 +33,13 @@ func (f EnablerFunc) ConfigureOffbox(ctx context.Context, host, user string, por return f(ctx, host, user, port, repoPath, privPEM, knownHosts, quotaGB) } +// SettleFunc adapts a plain func to a SettleProvider (thin adapter over the Updater in main.go — +// the StackDataProvider pattern). It reads the updater's OWN knowledge; the bridge never fetches the +// floor a second way (no second floor path). +type SettleFunc func() (version, floor string, updateRunning, floorKnown bool) + +func (f SettleFunc) SettleState() (string, string, bool, bool) { return f() } + // --- HTTPConsumer: POST the hub consume-password endpoint with the per-customer API key --- type HTTPConsumer struct { diff --git a/controller/internal/offsiteapply/settlegate_test.go b/controller/internal/offsiteapply/settlegate_test.go new file mode 100644 index 0000000..678ca6d --- /dev/null +++ b/controller/internal/offsiteapply/settlegate_test.go @@ -0,0 +1,300 @@ +package offsiteapply + +import ( + "bytes" + "context" + "log" + "path/filepath" + "strings" + "sync" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-controller/internal/config" +) + +// --- settle-gate fakes (R-71a) --- + +// fakeSettle is a mutable SettleProvider — a test flips its fields between polls to model a managed +// update landing (below-floor → at-floor) or an update finishing (updateRunning true → false). +type fakeSettle struct { + mu sync.Mutex + version string + floor string + updateRunning bool + floorKnown bool + polls int + // afterPoll runs after each SettleState read (poll number passed) so a test can flip state at a + // chosen poll — modelling the update converging. + afterPoll func(f *fakeSettle, poll int) +} + +func (f *fakeSettle) SettleState() (string, string, bool, bool) { + f.mu.Lock() + defer f.mu.Unlock() + f.polls++ + v, fl, ur, fk := f.version, f.floor, f.updateRunning, f.floorKnown + if f.afterPoll != nil { + f.afterPoll(f, f.polls) + } + return v, fl, ur, fk +} + +// fakeClock is an injectable clock: it never really sleeps. Each Sleep advances virtual time by the +// requested duration and records the call, so a test drives the gate through its bounds instantly and +// asserts EXACTLY how much virtual wait was consumed (the B′ zero-wait proof). +type fakeClock struct { + mu sync.Mutex + t time.Time + sleeps int + totalDur time.Duration +} + +func newClock() *fakeClock { return &fakeClock{t: time.Unix(0, 0)} } + +func (c *fakeClock) now() time.Time { + c.mu.Lock() + defer c.mu.Unlock() + return c.t +} + +func (c *fakeClock) sleep(_ context.Context, d time.Duration) { + c.mu.Lock() + defer c.mu.Unlock() + c.sleeps++ + c.totalDur += d + c.t = c.t.Add(d) +} + +// settleBridge builds a bridge wired for a FULL Reconcile (so a released gate consumes exactly once) +// plus the injectable settle-gate. Prober is nil → the fresh consume+install path runs on release. +func settleBridge(t *testing.T, s *fakeSettle, clk *fakeClock) (*Bridge, *fakeConsumer, *bytes.Buffer) { + t.Helper() + cfg := &config.Config{} + cfg.Offsite = goodOffsite() + cons := &fakeConsumer{pw: "the-transient-pw"} + var logbuf bytes.Buffer + b := &Bridge{ + Cfg: cfg, + Consumer: cons, + Scanner: &fakeScanner{fp: "SHA256:goodfp", line: "[h]:23 ssh-ed25519 AAAAKEY"}, + KeyGen: &fakeKeyGen{priv: "PRIVPEM", pub: "ssh-ed25519 AAAAPUB felhom"}, + Installer: &fakeInstaller{}, + Enabler: &fakeEnabler{}, + MarkerPath: filepath.Join(t.TempDir(), "offbox", "applied_marker"), + Logger: log.New(&logbuf, "", 0), + Settle: s, + Now: clk.now, + Sleep: clk.sleep, + } + return b, cons, &logbuf +} + +// Scenario A — the race, killed. While below floor the gate consumes NOTHING; when the update lands +// (fake flips to at-floor) exactly one Reconcile proceeds → exactly one Consume. +func TestSettle_BelowFloorDefersThenGoes(t *testing.T) { + clk := newClock() + s := &fakeSettle{version: "0.153.0", floor: "0.156.0", floorKnown: true} + // Model the managed update converging at the 3rd poll: the container comes back at floor. + s.afterPoll = func(f *fakeSettle, poll int) { + if poll == 3 { + f.version = "0.156.0" + } + } + b, cons, logbuf := settleBridge(t, s, clk) + + if err := b.ReconcileWhenSettled(context.Background()); err != nil { + t.Fatalf("reconcile-when-settled: %v", err) + } + if cons.calls != 1 { + t.Fatalf("consume must run exactly once AFTER the update lands, got %d", cons.calls) + } + // It waited (polled) while below floor and did not consume prematurely. + if clk.sleeps < 2 { + t.Fatalf("expected the gate to defer while below floor (>=2 sleeps), got %d", clk.sleeps) + } + if !strings.Contains(logbuf.String(), "deferring offsite apply: managed update to floor 0.156.0 pending (we are 0.153.0)") { + t.Fatalf("expected the below-floor deferral log, got:\n%s", logbuf.String()) + } + if !strings.Contains(logbuf.String(), "settle-gate: GO — at/above floor 0.156.0") { + t.Fatalf("expected the at-floor GO log, got:\n%s", logbuf.String()) + } +} + +// Scenario A red-proof: WITHOUT the gate (Settle nil), a below-floor box consumes IMMEDIATELY — the +// exact F10 burn. This proves the gate is load-bearing (remove it → the failure returns). +func TestSettle_RedProof_NoGateConsumesBelowFloor(t *testing.T) { + clk := newClock() + s := &fakeSettle{version: "0.153.0", floor: "0.156.0", floorKnown: true} + b, cons, _ := settleBridge(t, s, clk) + b.Settle = nil // remove the gate → the pre-R-71a behavior + + if err := b.ReconcileWhenSettled(context.Background()); err != nil { + t.Fatalf("reconcile: %v", err) + } + if cons.calls != 1 { + t.Fatalf("RED-PROOF: without the gate the below-floor box consumes immediately (want 1), got %d", cons.calls) + } + if clk.sleeps != 0 { + t.Fatalf("RED-PROOF: without the gate there is no deferral wait, got %d sleeps", clk.sleeps) + } +} + +// Scenario B — an update in progress defers (zero consume); when it finishes at floor the gate GOes. +func TestSettle_UpdateRunningDefersThenGoes(t *testing.T) { + clk := newClock() + s := &fakeSettle{version: "0.156.0", floor: "0.156.0", floorKnown: true, updateRunning: true} + s.afterPoll = func(f *fakeSettle, poll int) { + if poll == 2 { + f.updateRunning = false + } + } + b, cons, logbuf := settleBridge(t, s, clk) + + if err := b.ReconcileWhenSettled(context.Background()); err != nil { + t.Fatalf("reconcile: %v", err) + } + if cons.calls != 1 { + t.Fatalf("consume must run once after the update finishes, got %d", cons.calls) + } + if !strings.Contains(logbuf.String(), "a managed update is in progress") { + t.Fatalf("expected the update-running deferral log, got:\n%s", logbuf.String()) + } +} + +// Scenario C — floor never becomes known: GO after the sub-bound with the WARN; then Consume runs. +func TestSettle_FloorUnknownGoesAfterSubBound(t *testing.T) { + clk := newClock() + s := &fakeSettle{version: "0.156.0", floor: "", floorKnown: false} // floor never learned + b, cons, logbuf := settleBridge(t, s, clk) + + if err := b.ReconcileWhenSettled(context.Background()); err != nil { + t.Fatalf("reconcile: %v", err) + } + if cons.calls != 1 { + t.Fatalf("consume must run once after the floor-unknown sub-bound GO, got %d", cons.calls) + } + // It released at the sub-bound (90s), not the overall bound (5m). + if clk.totalDur < settleFloorSubBound || clk.totalDur >= settleOverallBound { + t.Fatalf("expected release near the sub-bound %s, waited %s", settleFloorSubBound, clk.totalDur) + } + if !strings.Contains(logbuf.String(), "floor still unknown after 1m30s") { + t.Fatalf("expected the floor-unknown GO+WARN log, got:\n%s", logbuf.String()) + } +} + +// Scenario C red-proof: drop the sub-bound and floor-unknown would only ever release at the overall +// bound (5m) — a 3.3× longer hostage window. We assert the sub-bound is what releases it (waited +// well under the overall bound); if the sub-bound branch were removed this assertion fails. +func TestSettle_RedProof_FloorUnknownWithoutSubBoundWaitsFarLonger(t *testing.T) { + clk := newClock() + s := &fakeSettle{version: "0.156.0", floor: "", floorKnown: false} + b, _, _ := settleBridge(t, s, clk) + + b.AwaitSettle(context.Background()) + // With the sub-bound present, release happens at ~90s. The red-proof: if a maintainer deletes the + // `elapsed >= settleFloorSubBound` branch, the ONLY remaining exit for a perpetually-unknown floor + // is the 5-minute overall bound — this bound-check pins that regression. + if clk.totalDur >= settleOverallBound { + t.Fatalf("RED-PROOF: floor-unknown should release at the sub-bound %s, not drag to the overall bound; waited %s", + settleFloorSubBound, clk.totalDur) + } +} + +// Scenario D — perpetually below floor (an update that never lands): GO at the overall bound + WARN. +func TestSettle_PerpetuallyBelowFloorGoesAtOverallBound(t *testing.T) { + clk := newClock() + s := &fakeSettle{version: "0.153.0", floor: "0.156.0", floorKnown: true} // never converges + b, cons, logbuf := settleBridge(t, s, clk) + + if err := b.ReconcileWhenSettled(context.Background()); err != nil { + t.Fatalf("reconcile: %v", err) + } + if cons.calls != 1 { + t.Fatalf("consume must run once after the overall-bound GO, got %d", cons.calls) + } + if clk.totalDur < settleOverallBound { + t.Fatalf("expected release at the overall bound %s, waited %s", settleOverallBound, clk.totalDur) + } + if !strings.Contains(logbuf.String(), "settle-gate bound exhausted") || !strings.Contains(logbuf.String(), "R-71c self-heal is the belt") { + t.Fatalf("expected the bound-exhausted GO+WARN log, got:\n%s", logbuf.String()) + } +} + +// Scenario D red-proof: drop the overall bound and a perpetually-below-floor box loops forever. We +// pin the bound by asserting the gate terminates AND that termination is the overall bound (a +// deleted bound would never reach this assertion — the fake clock loops without limit). +func TestSettle_RedProof_PerpetualBelowFloorTerminatesAtBound(t *testing.T) { + clk := newClock() + s := &fakeSettle{version: "0.153.0", floor: "0.156.0", floorKnown: true} + b, _, _ := settleBridge(t, s, clk) + + done := make(chan struct{}) + go func() { b.AwaitSettle(context.Background()); close(done) }() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("RED-PROOF: the gate must terminate at the overall bound — without it a below-floor box loops forever") + } + if clk.totalDur < settleOverallBound { + t.Fatalf("RED-PROOF: termination must be the overall bound %s, waited %s", settleOverallBound, clk.totalDur) + } +} + +// Scenario E — the B′ invariant: an at/above-floor box GOes on the FIRST evaluation with ZERO wait. +func TestSettle_AtFloorGoesFirstPollNoWait(t *testing.T) { + clk := newClock() + s := &fakeSettle{version: "0.162.0", floor: "0.156.0", floorKnown: true} // above floor + b, cons, logbuf := settleBridge(t, s, clk) + + if err := b.ReconcileWhenSettled(context.Background()); err != nil { + t.Fatalf("reconcile: %v", err) + } + if s.polls != 1 { + t.Fatalf("B′: the happy path must GO on the FIRST poll, got %d polls", s.polls) + } + if clk.sleeps != 0 || clk.totalDur != 0 { + t.Fatalf("B′: zero new latency — no sleep may be consumed, got %d sleeps / %s", clk.sleeps, clk.totalDur) + } + if cons.calls != 1 { + t.Fatalf("the apply must proceed immediately, consume=%d want 1", cons.calls) + } + if !strings.Contains(logbuf.String(), "settle-gate: GO — at/above floor 0.156.0 (we are 0.162.0)") { + t.Fatalf("expected the first-poll GO log, got:\n%s", logbuf.String()) + } +} + +// No SettleProvider wired (self-update disabled) → no gate: immediate reconcile, zero wait. +func TestSettle_NilProviderNoGate(t *testing.T) { + clk := newClock() + b, cons, _ := settleBridge(t, &fakeSettle{}, clk) + b.Settle = nil + + if err := b.ReconcileWhenSettled(context.Background()); err != nil { + t.Fatalf("reconcile: %v", err) + } + if cons.calls != 1 || clk.sleeps != 0 { + t.Fatalf("nil Settle must reconcile immediately with no wait: consume=%d sleeps=%d", cons.calls, clk.sleeps) + } +} + +// A cancelled gate context skips the reconcile (shutdown mid-wait): no consume. +func TestSettle_CancelledGateSkipsReconcile(t *testing.T) { + clk := newClock() + s := &fakeSettle{version: "0.153.0", floor: "0.156.0", floorKnown: true} // would defer forever + b, cons, _ := settleBridge(t, s, clk) + ctx, cancel := context.WithCancel(context.Background()) + // Cancel after the first poll so the gate's post-sleep ctx check trips. + s.afterPoll = func(_ *fakeSettle, poll int) { + if poll == 1 { + cancel() + } + } + err := b.ReconcileWhenSettled(ctx) + if err == nil { + t.Fatal("a cancelled gate must return the context error, not reconcile") + } + if cons.calls != 0 { + t.Fatalf("a cancelled gate must NOT consume, got %d", cons.calls) + } +}