v0.162.0 — R-71(a): the apply-bridge settle-gate (kills the F10 day-0 race)

The day-0 race (DIAG-f10): a fresh box boots below the operator floor, the
apply-bridge consumes the single-use offsite password, then ~35s later the
managed auto-floor update replaces the container mid-install -> the new process
finds no installed key -> consume -> 404 -> offsite dead until an operator
Re-issue. Recurs on every onboarding whose ISO floor lags the managed floor.

Ordering-only fix (consume/install/persist internals + the 404-no-oracle
contract + the Consumer UNTOUCHED; R-71(b) rejected-by-design):
- New seam offsiteapply.SettleProvider.SettleState() + SettleFunc adapter over
  the self-updater's own GetFloor()/IsUpdateRunning() (no second floor path).
- Bridge.AwaitSettle polls 10s BEFORE the 3-min Reconcile ctx: defers while an
  update runs or the box is below the known floor; GOes at/above floor on the
  first poll with zero added latency (B'). Bounds 90s floor sub-bound / 5min
  overall, both GO+WARN (hub that can't serve a floor can't serve a consume ->
  no burn risk; R-71c is the belt). ReconcileWhenSettled = gate then reconcile.
- main.go: bridge goroutine moved after the updater is built; wired only when an
  updater exists (nil Settle = reconcile immediately, old behavior).

Finding: the floor is in-memory (report-ACK ~5-10s), NOT persisted -> unknown on
any restart until the first ACK; the 90s sub-bound is sized to that.

Tests (injectable clock, fake SettleState, recorded Consumer): A-E + nil-provider
+ cancelled-gate. Four red-proofs all observed FAIL then restored: gate removed /
updateRunning branch / floor sub-bound / overall bound. Deferral paths ship
unit-proven + red-proofed, NOT live-fired -- their precondition is now
structurally prevented by the v1.25.0 build gate. Layering: gate prevents, (a)
defers, (c) heals.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N7Drmtm2RzoqbkJZCNSFNQ
This commit is contained in:
2026-07-24 07:48:45 +02:00
parent ce8531426c
commit cb8bf14599
9 changed files with 654 additions and 68 deletions
+15
View File
@@ -1058,6 +1058,21 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an
> recovery for a genuinely-spent password on a fresh guest.
> - **Staged-secret wipe (v0.107.0):** confirm-escrow calls the agent's `DELETE /escrow/stage-secret`
> (agent ≥ v0.78.0) whenever `EscrowState` flips to `escrowed` — best-effort, loud-logged on failure.
> - **Settle-gate (R-71a, v0.162.0) — the day-0 race removed.** The apply-bridge runs BEHIND a
> settle-gate (`Bridge.AwaitSettle` → `ReconcileWhenSettled`): before the consume/install path it
> polls the self-updater's own state via the `SettleProvider` seam (a `SettleFunc` adapter over
> `updater.GetFloor()`/`IsUpdateRunning()` in main.go — no second floor-fetch path). While a managed
> update is running OR the box is below the operator floor (an auto-floor update is imminent), the
> gate WAITS rather than consume the single-use password — the update's restart would otherwise kill
> the bridge mid-install and burn it (the F10 day-0 shape). At/above floor with no update in flight,
> it GOes on the first poll with zero added latency (B). Bounds: 10 s poll, 90 s floor-knowledge
> sub-bound (sized to the ~510 s report-ACK floor latency; the floor is in-memory, not persisted,
> so it is unknown until the first ACK on any restart), 5 min overall — both bounds GO+WARN and lean
> on the R-71c hub self-heal as the belt (a hub that cannot serve a floor cannot serve a consume, so
> proceeding never burns a password). The gate is wired only when a self-updater exists (no updater
> → no floor-update to race → reconcile immediately). Ordering-only: the consume/install/persist
> internals and the 404-no-oracle contract are untouched. Three-layer defense: the v1.25.0
> golden≥floor build gate PREVENTS the trigger, (a) DEFERS it, R-71c HEALS a burn.
> **NAS network storage (v0.92.0, Part A2; pairs with agent v0.50.0).** A customer NAS share (NFS or SMB)
> is a **distinct storage KIND** from a physical drive (`StoragePath.Kind == "network"`), for **bulk media**.
+51 -33
View File
@@ -287,39 +287,9 @@ func main() {
backupMgr.SetSharesReconciler(stackMgr.ReconcileSamba)
}
// SLICE 2: the offsite apply-bridge — on startup (async, non-blocking) reconcile the hub-served offsite
// descriptor into a configured key-only offbox target (fail-safe, idempotent, no blind TOFU). The
// config_refresh self-restart re-runs this after a descriptor change (new process → startup).
if backupMgr != nil && cfg.Offsite.Enabled && cfg.Hub.URL != "" && cfg.Hub.APIKey != "" {
bridge := &offsiteapply.Bridge{
Cfg: cfg,
Consumer: offsiteapply.HTTPConsumer{HubURL: cfg.Hub.URL, CustomerID: cfg.Customer.ID, APIKey: cfg.Hub.APIKey},
Scanner: offsiteapply.KeyscanScanner{},
KeyGen: offsiteapply.ED25519KeyGen{},
Installer: offsiteapply.SSHCopyIDInstaller{},
Prober: offsiteapply.SFTPKeyAuthProber{KeyPath: filepath.Join(cfg.Paths.DataDir, "offbox", "ssh_key")},
Enabler: offsiteapply.EnablerFunc(func(ctx context.Context, host, user string, port int, repoPath, priv, kh string, quotaGB int) error {
tgt := &settings.OffboxTarget{Enabled: true, Host: host, User: user, Port: port, RepoPath: repoPath, Schedule: "daily", QuotaGB: quotaGB}
stage := func(ctx context.Context, pw string) error {
ac, err := agentapi.New(cfg.LocalAPI.Endpoint, cfg.LocalAPI.Token, cfg.LocalAPI.Fingerprint)
if err != nil {
return err
}
return ac.StageEscrowSecret(ctx, pw)
}
return backupMgr.ApplyOffsiteTarget(ctx, tgt, priv, kh, stage)
}),
MarkerPath: filepath.Join(cfg.Paths.DataDir, "offbox", "applied_marker"),
Logger: logger,
}
go func() {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
defer cancel()
if err := bridge.Reconcile(ctx); err != nil {
logger.Printf("[WARN] [offsite-apply] reconcile: %v (retries on next config refresh/restart)", err)
}
}()
}
// SLICE 2: the offsite apply-bridge is launched further down, AFTER the self-updater is constructed
// (R-71a: the bridge's settle-gate reads the updater's floor/update-running state to defer the
// consume past a managed day-0 floor-update). See "offsite apply-bridge" below.
// --- Wire the data-migration engine (B1) + backup↔migration mutual exclusion (Change 3) ---
stackMgr.SetMigrationDeps(sett, func() bool { return backupMgr != nil && backupMgr.IsRunning() })
@@ -383,6 +353,54 @@ func main() {
cfg.SelfUpdate.CheckInterval, cfg.SelfUpdate.AutoUpdate, cfg.SelfUpdate.AutoUpdateTime)
}
// SLICE 2: the offsite apply-bridge — on startup (async, non-blocking) reconcile the hub-served
// offsite descriptor into a configured key-only offbox target (fail-safe, idempotent, no blind
// TOFU). The config_refresh self-restart re-runs this after a descriptor change (new process →
// startup). Launched HERE (after the self-updater is built) so the R-71a settle-gate can read the
// updater's floor/update-running state and defer the one-time-password consume past a managed
// day-0 floor-update (the F10 race). The gate is wired ONLY when an updater exists — with no update
// mechanism there is no floor-update to race, so the bridge reconciles immediately (Settle nil).
if backupMgr != nil && cfg.Offsite.Enabled && cfg.Hub.URL != "" && cfg.Hub.APIKey != "" {
bridge := &offsiteapply.Bridge{
Cfg: cfg,
Consumer: offsiteapply.HTTPConsumer{HubURL: cfg.Hub.URL, CustomerID: cfg.Customer.ID, APIKey: cfg.Hub.APIKey},
Scanner: offsiteapply.KeyscanScanner{},
KeyGen: offsiteapply.ED25519KeyGen{},
Installer: offsiteapply.SSHCopyIDInstaller{},
Prober: offsiteapply.SFTPKeyAuthProber{KeyPath: filepath.Join(cfg.Paths.DataDir, "offbox", "ssh_key")},
Enabler: offsiteapply.EnablerFunc(func(ctx context.Context, host, user string, port int, repoPath, priv, kh string, quotaGB int) error {
tgt := &settings.OffboxTarget{Enabled: true, Host: host, User: user, Port: port, RepoPath: repoPath, Schedule: "daily", QuotaGB: quotaGB}
stage := func(ctx context.Context, pw string) error {
ac, err := agentapi.New(cfg.LocalAPI.Endpoint, cfg.LocalAPI.Token, cfg.LocalAPI.Fingerprint)
if err != nil {
return err
}
return ac.StageEscrowSecret(ctx, pw)
}
return backupMgr.ApplyOffsiteTarget(ctx, tgt, priv, kh, stage)
}),
MarkerPath: filepath.Join(cfg.Paths.DataDir, "offbox", "applied_marker"),
Logger: logger,
}
if updater != nil {
// Thin adapter over the updater's OWN knowledge (StackDataProvider pattern) — the bridge
// never fetches the floor a second way. floorKnown = the floor has been learned from a
// report ACK yet (GetFloor() != "").
u := updater
bridge.Settle = offsiteapply.SettleFunc(func() (string, string, bool, bool) {
floor := u.GetFloor()
return Version, floor, u.IsUpdateRunning(), floor != ""
})
}
go func() {
// ReconcileWhenSettled runs the settle-gate FIRST (its own bounds), then Reconcile under a
// fresh 3-minute context — the gate's wait never eats the reconcile budget.
if err := bridge.ReconcileWhenSettled(context.Background()); err != nil {
logger.Printf("[WARN] [offsite-apply] reconcile: %v (retries on next config refresh/restart)", err)
}
}()
}
// --- Initialize scheduler ---
sched := scheduler.New(logger)
sched.SetDebug(cfg.Logging.Level == "debug")
@@ -14,8 +14,10 @@ import (
"log"
"os"
"path/filepath"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
)
// The apply-bridge seams (tests inject fakes — no live SSH / hub calls in unit tests).
@@ -51,8 +53,56 @@ type (
KeyAuthProber interface {
Probe(ctx context.Context, host, user string, port int, knownHosts string) (privPEM string, ok bool)
}
// SettleProvider reports the managed-update settle state so the bridge can DEFER consuming the
// one-time password until any imminent managed floor-update has converged (R-71a — the structural
// fix for the F10 day-0 race). The failure it prevents: a fresh box boots below the operator floor,
// the bridge consumes the single-use password, then ~35 s later the managed auto-floor update
// replaces the container mid-install → the new process finds no installed key → consume → 404 →
// offsite dead until an operator Re-issue. version = the running controller version; floor = the
// operator-enforced minimum (from the hub report ACK); updateRunning = a swap is in flight;
// floorKnown = the floor has been learned yet (false = the first report ACK hasn't landed). A hub
// that hasn't served a floor cannot serve a consume either, so !floorKnown carries NO burn risk.
SettleProvider interface {
SettleState() (version, floor string, updateRunning, floorKnown bool)
}
)
// Settle-gate timing (R-71a). Named constants with rationale so the trade-offs stay visible.
const (
// reconcileTimeout bounds the actual Reconcile once the gate releases (moved here from main.go so
// ReconcileWhenSettled owns the whole "gate THEN reconcile" contract). The gate's own wait must NOT
// eat this budget — the reconcile context is created only after the gate returns.
reconcileTimeout = 3 * time.Minute
// settlePoll is the gate's poll cadence. The bridge is a background reconcile with no user-visible
// latency, so a coarse poll is free; the at/above-floor happy path returns on the FIRST evaluation
// with no sleep at all (the B invariant), so this cadence only ever paces the deferral cases.
settlePoll = 10 * time.Second
// settleFloorSubBound: if the hub never tells us the floor (report ACK), stop waiting and GO+WARN.
// Sized to the report-ACK latency observed at source: the startup report fires ~5 s after boot
// (main.go's startup goroutine sleeps 5 s), and SetFloor runs SYNCHRONOUSLY inside that report's
// ACK handler (main.go OnPushResponse → updater.SetFloor), so the floor is normally known within
// ~510 s. The startup report retries up to 3× with 15 s gaps, so a slow first report can push
// floor-knowledge to ~45 s; 90 s is generous headroom over that worst case. Proceeding here cannot
// burn a one-time password: a hub that cannot serve a floor cannot serve a consume.
settleFloorSubBound = 90 * time.Second
// settleOverallBound: the absolute cap. If we are still below floor after this (a managed
// floor-update that never lands), proceed anyway — the R-71c hub self-heal restage is the belt for
// a consume that a genuinely stuck update might later burn.
settleOverallBound = 5 * time.Minute
)
// belowFloor reports whether the running version is strictly below the operator floor. An
// unparseable version or floor (a dev build, or a malformed floor) is treated as NOT below — the gate
// must never wedge on a version it cannot compare (and a dev build never auto-floor-updates anyway).
func belowFloor(version, floor string) bool {
v, err1 := util.ParseVersion(version)
f, err2 := util.ParseVersion(floor)
if err1 != nil || err2 != nil {
return false
}
return v.Compare(f) < 0
}
// Bridge reconciles the offsite descriptor into a configured offbox target.
type Bridge struct {
Cfg *config.Config
@@ -64,6 +114,36 @@ type Bridge struct {
Prober KeyAuthProber // optional: key-auth-first (nil → always the full consume+install path)
MarkerPath string // where the applied-descriptor-hash is persisted (e.g. <dataDir>/offbox/applied_marker)
Logger *log.Logger
// Settle gates the consume/install path behind managed-update convergence (R-71a). nil → no gate
// (old behavior: reconcile immediately). Wired only when a self-updater exists — with no update
// mechanism there is no floor-update to race, so no gate is needed.
Settle SettleProvider
// Now/Sleep are clock seams for the settle-gate ONLY (tests inject a fake clock so the bounds are
// exercised with zero real sleeps). nil → the real wall clock and a context-aware sleep.
Now func() time.Time
Sleep func(ctx context.Context, d time.Duration)
}
func (b *Bridge) nowFn() func() time.Time {
if b.Now != nil {
return b.Now
}
return time.Now
}
func (b *Bridge) sleepFn() func(context.Context, time.Duration) {
if b.Sleep != nil {
return b.Sleep
}
return func(ctx context.Context, d time.Duration) {
t := time.NewTimer(d)
defer t.Stop()
select {
case <-ctx.Done():
case <-t.C:
}
}
}
func (b *Bridge) logf(f string, a ...any) {
@@ -185,3 +265,79 @@ func (b *Bridge) Reconcile(ctx context.Context) error {
b.logf("[INFO] [offsite-apply] offsite configured for %s@%s:%s (pending key escrow)", o.User, o.Host, o.RepoPath)
return nil
}
// AwaitSettle blocks until it is safe to run the apply-bridge, then returns (R-71a). It removes the
// SYSTEMATIC trigger for the F10 day-0 race by refusing to consume the one-time password while a
// managed floor-update is in flight or imminent (we are below the floor): that update's restart would
// supersede the bridge and kill it mid-install, spending the password for nothing. It NEVER blocks
// when it is genuinely safe — the overwhelmingly common shape (a restart of an at/above-floor box)
// evaluates GO on the first poll with no sleep at all (the B invariant: zero new latency).
//
// It is a strict no-op unless a SettleProvider is wired (nil → old behavior, reconcile immediately).
// The gate has its own bounds (settlePoll/settleFloorSubBound/settleOverallBound) and its own context
// so the deferral never eats the reconcile budget.
func (b *Bridge) AwaitSettle(ctx context.Context) {
if b.Settle == nil {
return
}
now, sleep := b.nowFn(), b.sleepFn()
start := now()
var loggedUpdate, loggedDefer, loggedFloorWait bool
for {
version, floor, updateRunning, floorKnown := b.Settle.SettleState()
elapsed := now().Sub(start)
switch {
case updateRunning:
// A swap is in flight; its restart supersedes us. Wait it out.
if !loggedUpdate {
b.logf("[INFO] [offsite-apply] settle-gate: a managed update is in progress — deferring offsite apply until it converges")
loggedUpdate = true
}
case floorKnown && belowFloor(version, floor):
// The auto-floor update is imminent (below floor + floor known). Do NOT consume — the
// update's restart would burn the password. Wait for the update to land (which restarts us
// at floor → the GO branch below).
if !loggedDefer {
b.logf("[INFO] [offsite-apply] deferring offsite apply: managed update to floor %s pending (we are %s)", floor, version)
loggedDefer = true
}
case floorKnown:
// At/above floor, no update running — the safe steady state. GO.
b.logf("[INFO] [offsite-apply] settle-gate: GO — at/above floor %s (we are %s), no managed update running", floor, version)
return
case elapsed >= settleFloorSubBound:
// Floor never became known within the sub-bound. A hub that will not tell us the floor
// cannot serve a consume either, so the burn risk is nil — don't hold offsite hostage.
b.logf("[WARN] [offsite-apply] settle-gate: GO — floor still unknown after %s; a hub that cannot serve a floor cannot serve a consume (no burn risk)", settleFloorSubBound)
return
default:
// Floor not known yet, still inside the sub-bound — wait for the report ACK.
if !loggedFloorWait {
b.logf("[INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply")
loggedFloorWait = true
}
}
if elapsed >= settleOverallBound {
b.logf("[WARN] [offsite-apply] settle-gate bound exhausted after %s — proceeding; R-71c self-heal is the belt", settleOverallBound)
return
}
sleep(ctx, settlePoll)
if ctx.Err() != nil {
return // shutdown / cancellation — abandon the gate (the next start retries)
}
}
}
// ReconcileWhenSettled runs the settle-gate (R-71a) and THEN Reconcile under a FRESH reconcile
// context. The gate's deferral must not eat the reconcile budget, so the reconcile timeout starts
// only after the gate releases. gateCtx bounds the gate (e.g. process shutdown); a cancelled gate
// skips the reconcile (the next start retries).
func (b *Bridge) ReconcileWhenSettled(gateCtx context.Context) error {
b.AwaitSettle(gateCtx)
if gateCtx.Err() != nil {
return gateCtx.Err()
}
ctx, cancel := context.WithTimeout(context.Background(), reconcileTimeout)
defer cancel()
return b.Reconcile(ctx)
}
@@ -33,6 +33,13 @@ func (f EnablerFunc) ConfigureOffbox(ctx context.Context, host, user string, por
return f(ctx, host, user, port, repoPath, privPEM, knownHosts, quotaGB)
}
// SettleFunc adapts a plain func to a SettleProvider (thin adapter over the Updater in main.go —
// the StackDataProvider pattern). It reads the updater's OWN knowledge; the bridge never fetches the
// floor a second way (no second floor path).
type SettleFunc func() (version, floor string, updateRunning, floorKnown bool)
func (f SettleFunc) SettleState() (string, string, bool, bool) { return f() }
// --- HTTPConsumer: POST the hub consume-password endpoint with the per-customer API key ---
type HTTPConsumer struct {
@@ -0,0 +1,300 @@
package offsiteapply
import (
"bytes"
"context"
"log"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
)
// --- settle-gate fakes (R-71a) ---
// fakeSettle is a mutable SettleProvider — a test flips its fields between polls to model a managed
// update landing (below-floor → at-floor) or an update finishing (updateRunning true → false).
type fakeSettle struct {
mu sync.Mutex
version string
floor string
updateRunning bool
floorKnown bool
polls int
// afterPoll runs after each SettleState read (poll number passed) so a test can flip state at a
// chosen poll — modelling the update converging.
afterPoll func(f *fakeSettle, poll int)
}
func (f *fakeSettle) SettleState() (string, string, bool, bool) {
f.mu.Lock()
defer f.mu.Unlock()
f.polls++
v, fl, ur, fk := f.version, f.floor, f.updateRunning, f.floorKnown
if f.afterPoll != nil {
f.afterPoll(f, f.polls)
}
return v, fl, ur, fk
}
// fakeClock is an injectable clock: it never really sleeps. Each Sleep advances virtual time by the
// requested duration and records the call, so a test drives the gate through its bounds instantly and
// asserts EXACTLY how much virtual wait was consumed (the B zero-wait proof).
type fakeClock struct {
mu sync.Mutex
t time.Time
sleeps int
totalDur time.Duration
}
func newClock() *fakeClock { return &fakeClock{t: time.Unix(0, 0)} }
func (c *fakeClock) now() time.Time {
c.mu.Lock()
defer c.mu.Unlock()
return c.t
}
func (c *fakeClock) sleep(_ context.Context, d time.Duration) {
c.mu.Lock()
defer c.mu.Unlock()
c.sleeps++
c.totalDur += d
c.t = c.t.Add(d)
}
// settleBridge builds a bridge wired for a FULL Reconcile (so a released gate consumes exactly once)
// plus the injectable settle-gate. Prober is nil → the fresh consume+install path runs on release.
func settleBridge(t *testing.T, s *fakeSettle, clk *fakeClock) (*Bridge, *fakeConsumer, *bytes.Buffer) {
t.Helper()
cfg := &config.Config{}
cfg.Offsite = goodOffsite()
cons := &fakeConsumer{pw: "the-transient-pw"}
var logbuf bytes.Buffer
b := &Bridge{
Cfg: cfg,
Consumer: cons,
Scanner: &fakeScanner{fp: "SHA256:goodfp", line: "[h]:23 ssh-ed25519 AAAAKEY"},
KeyGen: &fakeKeyGen{priv: "PRIVPEM", pub: "ssh-ed25519 AAAAPUB felhom"},
Installer: &fakeInstaller{},
Enabler: &fakeEnabler{},
MarkerPath: filepath.Join(t.TempDir(), "offbox", "applied_marker"),
Logger: log.New(&logbuf, "", 0),
Settle: s,
Now: clk.now,
Sleep: clk.sleep,
}
return b, cons, &logbuf
}
// Scenario A — the race, killed. While below floor the gate consumes NOTHING; when the update lands
// (fake flips to at-floor) exactly one Reconcile proceeds → exactly one Consume.
func TestSettle_BelowFloorDefersThenGoes(t *testing.T) {
clk := newClock()
s := &fakeSettle{version: "0.153.0", floor: "0.156.0", floorKnown: true}
// Model the managed update converging at the 3rd poll: the container comes back at floor.
s.afterPoll = func(f *fakeSettle, poll int) {
if poll == 3 {
f.version = "0.156.0"
}
}
b, cons, logbuf := settleBridge(t, s, clk)
if err := b.ReconcileWhenSettled(context.Background()); err != nil {
t.Fatalf("reconcile-when-settled: %v", err)
}
if cons.calls != 1 {
t.Fatalf("consume must run exactly once AFTER the update lands, got %d", cons.calls)
}
// It waited (polled) while below floor and did not consume prematurely.
if clk.sleeps < 2 {
t.Fatalf("expected the gate to defer while below floor (>=2 sleeps), got %d", clk.sleeps)
}
if !strings.Contains(logbuf.String(), "deferring offsite apply: managed update to floor 0.156.0 pending (we are 0.153.0)") {
t.Fatalf("expected the below-floor deferral log, got:\n%s", logbuf.String())
}
if !strings.Contains(logbuf.String(), "settle-gate: GO — at/above floor 0.156.0") {
t.Fatalf("expected the at-floor GO log, got:\n%s", logbuf.String())
}
}
// Scenario A red-proof: WITHOUT the gate (Settle nil), a below-floor box consumes IMMEDIATELY — the
// exact F10 burn. This proves the gate is load-bearing (remove it → the failure returns).
func TestSettle_RedProof_NoGateConsumesBelowFloor(t *testing.T) {
clk := newClock()
s := &fakeSettle{version: "0.153.0", floor: "0.156.0", floorKnown: true}
b, cons, _ := settleBridge(t, s, clk)
b.Settle = nil // remove the gate → the pre-R-71a behavior
if err := b.ReconcileWhenSettled(context.Background()); err != nil {
t.Fatalf("reconcile: %v", err)
}
if cons.calls != 1 {
t.Fatalf("RED-PROOF: without the gate the below-floor box consumes immediately (want 1), got %d", cons.calls)
}
if clk.sleeps != 0 {
t.Fatalf("RED-PROOF: without the gate there is no deferral wait, got %d sleeps", clk.sleeps)
}
}
// Scenario B — an update in progress defers (zero consume); when it finishes at floor the gate GOes.
func TestSettle_UpdateRunningDefersThenGoes(t *testing.T) {
clk := newClock()
s := &fakeSettle{version: "0.156.0", floor: "0.156.0", floorKnown: true, updateRunning: true}
s.afterPoll = func(f *fakeSettle, poll int) {
if poll == 2 {
f.updateRunning = false
}
}
b, cons, logbuf := settleBridge(t, s, clk)
if err := b.ReconcileWhenSettled(context.Background()); err != nil {
t.Fatalf("reconcile: %v", err)
}
if cons.calls != 1 {
t.Fatalf("consume must run once after the update finishes, got %d", cons.calls)
}
if !strings.Contains(logbuf.String(), "a managed update is in progress") {
t.Fatalf("expected the update-running deferral log, got:\n%s", logbuf.String())
}
}
// Scenario C — floor never becomes known: GO after the sub-bound with the WARN; then Consume runs.
func TestSettle_FloorUnknownGoesAfterSubBound(t *testing.T) {
clk := newClock()
s := &fakeSettle{version: "0.156.0", floor: "", floorKnown: false} // floor never learned
b, cons, logbuf := settleBridge(t, s, clk)
if err := b.ReconcileWhenSettled(context.Background()); err != nil {
t.Fatalf("reconcile: %v", err)
}
if cons.calls != 1 {
t.Fatalf("consume must run once after the floor-unknown sub-bound GO, got %d", cons.calls)
}
// It released at the sub-bound (90s), not the overall bound (5m).
if clk.totalDur < settleFloorSubBound || clk.totalDur >= settleOverallBound {
t.Fatalf("expected release near the sub-bound %s, waited %s", settleFloorSubBound, clk.totalDur)
}
if !strings.Contains(logbuf.String(), "floor still unknown after 1m30s") {
t.Fatalf("expected the floor-unknown GO+WARN log, got:\n%s", logbuf.String())
}
}
// Scenario C red-proof: drop the sub-bound and floor-unknown would only ever release at the overall
// bound (5m) — a 3.3× longer hostage window. We assert the sub-bound is what releases it (waited
// well under the overall bound); if the sub-bound branch were removed this assertion fails.
func TestSettle_RedProof_FloorUnknownWithoutSubBoundWaitsFarLonger(t *testing.T) {
clk := newClock()
s := &fakeSettle{version: "0.156.0", floor: "", floorKnown: false}
b, _, _ := settleBridge(t, s, clk)
b.AwaitSettle(context.Background())
// With the sub-bound present, release happens at ~90s. The red-proof: if a maintainer deletes the
// `elapsed >= settleFloorSubBound` branch, the ONLY remaining exit for a perpetually-unknown floor
// is the 5-minute overall bound — this bound-check pins that regression.
if clk.totalDur >= settleOverallBound {
t.Fatalf("RED-PROOF: floor-unknown should release at the sub-bound %s, not drag to the overall bound; waited %s",
settleFloorSubBound, clk.totalDur)
}
}
// Scenario D — perpetually below floor (an update that never lands): GO at the overall bound + WARN.
func TestSettle_PerpetuallyBelowFloorGoesAtOverallBound(t *testing.T) {
clk := newClock()
s := &fakeSettle{version: "0.153.0", floor: "0.156.0", floorKnown: true} // never converges
b, cons, logbuf := settleBridge(t, s, clk)
if err := b.ReconcileWhenSettled(context.Background()); err != nil {
t.Fatalf("reconcile: %v", err)
}
if cons.calls != 1 {
t.Fatalf("consume must run once after the overall-bound GO, got %d", cons.calls)
}
if clk.totalDur < settleOverallBound {
t.Fatalf("expected release at the overall bound %s, waited %s", settleOverallBound, clk.totalDur)
}
if !strings.Contains(logbuf.String(), "settle-gate bound exhausted") || !strings.Contains(logbuf.String(), "R-71c self-heal is the belt") {
t.Fatalf("expected the bound-exhausted GO+WARN log, got:\n%s", logbuf.String())
}
}
// Scenario D red-proof: drop the overall bound and a perpetually-below-floor box loops forever. We
// pin the bound by asserting the gate terminates AND that termination is the overall bound (a
// deleted bound would never reach this assertion — the fake clock loops without limit).
func TestSettle_RedProof_PerpetualBelowFloorTerminatesAtBound(t *testing.T) {
clk := newClock()
s := &fakeSettle{version: "0.153.0", floor: "0.156.0", floorKnown: true}
b, _, _ := settleBridge(t, s, clk)
done := make(chan struct{})
go func() { b.AwaitSettle(context.Background()); close(done) }()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("RED-PROOF: the gate must terminate at the overall bound — without it a below-floor box loops forever")
}
if clk.totalDur < settleOverallBound {
t.Fatalf("RED-PROOF: termination must be the overall bound %s, waited %s", settleOverallBound, clk.totalDur)
}
}
// Scenario E — the B invariant: an at/above-floor box GOes on the FIRST evaluation with ZERO wait.
func TestSettle_AtFloorGoesFirstPollNoWait(t *testing.T) {
clk := newClock()
s := &fakeSettle{version: "0.162.0", floor: "0.156.0", floorKnown: true} // above floor
b, cons, logbuf := settleBridge(t, s, clk)
if err := b.ReconcileWhenSettled(context.Background()); err != nil {
t.Fatalf("reconcile: %v", err)
}
if s.polls != 1 {
t.Fatalf("B: the happy path must GO on the FIRST poll, got %d polls", s.polls)
}
if clk.sleeps != 0 || clk.totalDur != 0 {
t.Fatalf("B: zero new latency — no sleep may be consumed, got %d sleeps / %s", clk.sleeps, clk.totalDur)
}
if cons.calls != 1 {
t.Fatalf("the apply must proceed immediately, consume=%d want 1", cons.calls)
}
if !strings.Contains(logbuf.String(), "settle-gate: GO — at/above floor 0.156.0 (we are 0.162.0)") {
t.Fatalf("expected the first-poll GO log, got:\n%s", logbuf.String())
}
}
// No SettleProvider wired (self-update disabled) → no gate: immediate reconcile, zero wait.
func TestSettle_NilProviderNoGate(t *testing.T) {
clk := newClock()
b, cons, _ := settleBridge(t, &fakeSettle{}, clk)
b.Settle = nil
if err := b.ReconcileWhenSettled(context.Background()); err != nil {
t.Fatalf("reconcile: %v", err)
}
if cons.calls != 1 || clk.sleeps != 0 {
t.Fatalf("nil Settle must reconcile immediately with no wait: consume=%d sleeps=%d", cons.calls, clk.sleeps)
}
}
// A cancelled gate context skips the reconcile (shutdown mid-wait): no consume.
func TestSettle_CancelledGateSkipsReconcile(t *testing.T) {
clk := newClock()
s := &fakeSettle{version: "0.153.0", floor: "0.156.0", floorKnown: true} // would defer forever
b, cons, _ := settleBridge(t, s, clk)
ctx, cancel := context.WithCancel(context.Background())
// Cancel after the first poll so the gate's post-sleep ctx check trips.
s.afterPoll = func(_ *fakeSettle, poll int) {
if poll == 1 {
cancel()
}
}
err := b.ReconcileWhenSettled(ctx)
if err == nil {
t.Fatal("a cancelled gate must return the context error, not reconcile")
}
if cons.calls != 0 {
t.Fatalf("a cancelled gate must NOT consume, got %d", cons.calls)
}
}