v0.162.0 — R-71(a): the apply-bridge settle-gate (kills the F10 day-0 race)
The day-0 race (DIAG-f10): a fresh box boots below the operator floor, the apply-bridge consumes the single-use offsite password, then ~35s later the managed auto-floor update replaces the container mid-install -> the new process finds no installed key -> consume -> 404 -> offsite dead until an operator Re-issue. Recurs on every onboarding whose ISO floor lags the managed floor. Ordering-only fix (consume/install/persist internals + the 404-no-oracle contract + the Consumer UNTOUCHED; R-71(b) rejected-by-design): - New seam offsiteapply.SettleProvider.SettleState() + SettleFunc adapter over the self-updater's own GetFloor()/IsUpdateRunning() (no second floor path). - Bridge.AwaitSettle polls 10s BEFORE the 3-min Reconcile ctx: defers while an update runs or the box is below the known floor; GOes at/above floor on the first poll with zero added latency (B'). Bounds 90s floor sub-bound / 5min overall, both GO+WARN (hub that can't serve a floor can't serve a consume -> no burn risk; R-71c is the belt). ReconcileWhenSettled = gate then reconcile. - main.go: bridge goroutine moved after the updater is built; wired only when an updater exists (nil Settle = reconcile immediately, old behavior). Finding: the floor is in-memory (report-ACK ~5-10s), NOT persisted -> unknown on any restart until the first ACK; the 90s sub-bound is sized to that. Tests (injectable clock, fake SettleState, recorded Consumer): A-E + nil-provider + cancelled-gate. Four red-proofs all observed FAIL then restored: gate removed / updateRunning branch / floor sub-bound / overall bound. Deferral paths ship unit-proven + red-proofed, NOT live-fired -- their precondition is now structurally prevented by the v1.25.0 build gate. Layering: gate prevents, (a) defers, (c) heals. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N7Drmtm2RzoqbkJZCNSFNQ
This commit is contained in:
@@ -14,8 +14,10 @@ import (
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
|
||||
)
|
||||
|
||||
// The apply-bridge seams (tests inject fakes — no live SSH / hub calls in unit tests).
|
||||
@@ -51,8 +53,56 @@ type (
|
||||
KeyAuthProber interface {
|
||||
Probe(ctx context.Context, host, user string, port int, knownHosts string) (privPEM string, ok bool)
|
||||
}
|
||||
// SettleProvider reports the managed-update settle state so the bridge can DEFER consuming the
|
||||
// one-time password until any imminent managed floor-update has converged (R-71a — the structural
|
||||
// fix for the F10 day-0 race). The failure it prevents: a fresh box boots below the operator floor,
|
||||
// the bridge consumes the single-use password, then ~35 s later the managed auto-floor update
|
||||
// replaces the container mid-install → the new process finds no installed key → consume → 404 →
|
||||
// offsite dead until an operator Re-issue. version = the running controller version; floor = the
|
||||
// operator-enforced minimum (from the hub report ACK); updateRunning = a swap is in flight;
|
||||
// floorKnown = the floor has been learned yet (false = the first report ACK hasn't landed). A hub
|
||||
// that hasn't served a floor cannot serve a consume either, so !floorKnown carries NO burn risk.
|
||||
SettleProvider interface {
|
||||
SettleState() (version, floor string, updateRunning, floorKnown bool)
|
||||
}
|
||||
)
|
||||
|
||||
// Settle-gate timing (R-71a). Named constants with rationale so the trade-offs stay visible.
|
||||
const (
|
||||
// reconcileTimeout bounds the actual Reconcile once the gate releases (moved here from main.go so
|
||||
// ReconcileWhenSettled owns the whole "gate THEN reconcile" contract). The gate's own wait must NOT
|
||||
// eat this budget — the reconcile context is created only after the gate returns.
|
||||
reconcileTimeout = 3 * time.Minute
|
||||
// settlePoll is the gate's poll cadence. The bridge is a background reconcile with no user-visible
|
||||
// latency, so a coarse poll is free; the at/above-floor happy path returns on the FIRST evaluation
|
||||
// with no sleep at all (the B′ invariant), so this cadence only ever paces the deferral cases.
|
||||
settlePoll = 10 * time.Second
|
||||
// settleFloorSubBound: if the hub never tells us the floor (report ACK), stop waiting and GO+WARN.
|
||||
// Sized to the report-ACK latency observed at source: the startup report fires ~5 s after boot
|
||||
// (main.go's startup goroutine sleeps 5 s), and SetFloor runs SYNCHRONOUSLY inside that report's
|
||||
// ACK handler (main.go OnPushResponse → updater.SetFloor), so the floor is normally known within
|
||||
// ~5–10 s. The startup report retries up to 3× with 15 s gaps, so a slow first report can push
|
||||
// floor-knowledge to ~45 s; 90 s is generous headroom over that worst case. Proceeding here cannot
|
||||
// burn a one-time password: a hub that cannot serve a floor cannot serve a consume.
|
||||
settleFloorSubBound = 90 * time.Second
|
||||
// settleOverallBound: the absolute cap. If we are still below floor after this (a managed
|
||||
// floor-update that never lands), proceed anyway — the R-71c hub self-heal restage is the belt for
|
||||
// a consume that a genuinely stuck update might later burn.
|
||||
settleOverallBound = 5 * time.Minute
|
||||
)
|
||||
|
||||
// belowFloor reports whether the running version is strictly below the operator floor. An
|
||||
// unparseable version or floor (a dev build, or a malformed floor) is treated as NOT below — the gate
|
||||
// must never wedge on a version it cannot compare (and a dev build never auto-floor-updates anyway).
|
||||
func belowFloor(version, floor string) bool {
|
||||
v, err1 := util.ParseVersion(version)
|
||||
f, err2 := util.ParseVersion(floor)
|
||||
if err1 != nil || err2 != nil {
|
||||
return false
|
||||
}
|
||||
return v.Compare(f) < 0
|
||||
}
|
||||
|
||||
// Bridge reconciles the offsite descriptor into a configured offbox target.
|
||||
type Bridge struct {
|
||||
Cfg *config.Config
|
||||
@@ -64,6 +114,36 @@ type Bridge struct {
|
||||
Prober KeyAuthProber // optional: key-auth-first (nil → always the full consume+install path)
|
||||
MarkerPath string // where the applied-descriptor-hash is persisted (e.g. <dataDir>/offbox/applied_marker)
|
||||
Logger *log.Logger
|
||||
|
||||
// Settle gates the consume/install path behind managed-update convergence (R-71a). nil → no gate
|
||||
// (old behavior: reconcile immediately). Wired only when a self-updater exists — with no update
|
||||
// mechanism there is no floor-update to race, so no gate is needed.
|
||||
Settle SettleProvider
|
||||
// Now/Sleep are clock seams for the settle-gate ONLY (tests inject a fake clock so the bounds are
|
||||
// exercised with zero real sleeps). nil → the real wall clock and a context-aware sleep.
|
||||
Now func() time.Time
|
||||
Sleep func(ctx context.Context, d time.Duration)
|
||||
}
|
||||
|
||||
func (b *Bridge) nowFn() func() time.Time {
|
||||
if b.Now != nil {
|
||||
return b.Now
|
||||
}
|
||||
return time.Now
|
||||
}
|
||||
|
||||
func (b *Bridge) sleepFn() func(context.Context, time.Duration) {
|
||||
if b.Sleep != nil {
|
||||
return b.Sleep
|
||||
}
|
||||
return func(ctx context.Context, d time.Duration) {
|
||||
t := time.NewTimer(d)
|
||||
defer t.Stop()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
case <-t.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (b *Bridge) logf(f string, a ...any) {
|
||||
@@ -185,3 +265,79 @@ func (b *Bridge) Reconcile(ctx context.Context) error {
|
||||
b.logf("[INFO] [offsite-apply] offsite configured for %s@%s:%s (pending key escrow)", o.User, o.Host, o.RepoPath)
|
||||
return nil
|
||||
}
|
||||
|
||||
// AwaitSettle blocks until it is safe to run the apply-bridge, then returns (R-71a). It removes the
|
||||
// SYSTEMATIC trigger for the F10 day-0 race by refusing to consume the one-time password while a
|
||||
// managed floor-update is in flight or imminent (we are below the floor): that update's restart would
|
||||
// supersede the bridge and kill it mid-install, spending the password for nothing. It NEVER blocks
|
||||
// when it is genuinely safe — the overwhelmingly common shape (a restart of an at/above-floor box)
|
||||
// evaluates GO on the first poll with no sleep at all (the B′ invariant: zero new latency).
|
||||
//
|
||||
// It is a strict no-op unless a SettleProvider is wired (nil → old behavior, reconcile immediately).
|
||||
// The gate has its own bounds (settlePoll/settleFloorSubBound/settleOverallBound) and its own context
|
||||
// so the deferral never eats the reconcile budget.
|
||||
func (b *Bridge) AwaitSettle(ctx context.Context) {
|
||||
if b.Settle == nil {
|
||||
return
|
||||
}
|
||||
now, sleep := b.nowFn(), b.sleepFn()
|
||||
start := now()
|
||||
var loggedUpdate, loggedDefer, loggedFloorWait bool
|
||||
for {
|
||||
version, floor, updateRunning, floorKnown := b.Settle.SettleState()
|
||||
elapsed := now().Sub(start)
|
||||
switch {
|
||||
case updateRunning:
|
||||
// A swap is in flight; its restart supersedes us. Wait it out.
|
||||
if !loggedUpdate {
|
||||
b.logf("[INFO] [offsite-apply] settle-gate: a managed update is in progress — deferring offsite apply until it converges")
|
||||
loggedUpdate = true
|
||||
}
|
||||
case floorKnown && belowFloor(version, floor):
|
||||
// The auto-floor update is imminent (below floor + floor known). Do NOT consume — the
|
||||
// update's restart would burn the password. Wait for the update to land (which restarts us
|
||||
// at floor → the GO branch below).
|
||||
if !loggedDefer {
|
||||
b.logf("[INFO] [offsite-apply] deferring offsite apply: managed update to floor %s pending (we are %s)", floor, version)
|
||||
loggedDefer = true
|
||||
}
|
||||
case floorKnown:
|
||||
// At/above floor, no update running — the safe steady state. GO.
|
||||
b.logf("[INFO] [offsite-apply] settle-gate: GO — at/above floor %s (we are %s), no managed update running", floor, version)
|
||||
return
|
||||
case elapsed >= settleFloorSubBound:
|
||||
// Floor never became known within the sub-bound. A hub that will not tell us the floor
|
||||
// cannot serve a consume either, so the burn risk is nil — don't hold offsite hostage.
|
||||
b.logf("[WARN] [offsite-apply] settle-gate: GO — floor still unknown after %s; a hub that cannot serve a floor cannot serve a consume (no burn risk)", settleFloorSubBound)
|
||||
return
|
||||
default:
|
||||
// Floor not known yet, still inside the sub-bound — wait for the report ACK.
|
||||
if !loggedFloorWait {
|
||||
b.logf("[INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply")
|
||||
loggedFloorWait = true
|
||||
}
|
||||
}
|
||||
if elapsed >= settleOverallBound {
|
||||
b.logf("[WARN] [offsite-apply] settle-gate bound exhausted after %s — proceeding; R-71c self-heal is the belt", settleOverallBound)
|
||||
return
|
||||
}
|
||||
sleep(ctx, settlePoll)
|
||||
if ctx.Err() != nil {
|
||||
return // shutdown / cancellation — abandon the gate (the next start retries)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ReconcileWhenSettled runs the settle-gate (R-71a) and THEN Reconcile under a FRESH reconcile
|
||||
// context. The gate's deferral must not eat the reconcile budget, so the reconcile timeout starts
|
||||
// only after the gate releases. gateCtx bounds the gate (e.g. process shutdown); a cancelled gate
|
||||
// skips the reconcile (the next start retries).
|
||||
func (b *Bridge) ReconcileWhenSettled(gateCtx context.Context) error {
|
||||
b.AwaitSettle(gateCtx)
|
||||
if gateCtx.Err() != nil {
|
||||
return gateCtx.Err()
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), reconcileTimeout)
|
||||
defer cancel()
|
||||
return b.Reconcile(ctx)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user