controller v0.71.0: fix guest-reboot recovery (boot-race + agent-path blocker)
Live diagnosis of drive-backed apps stuck Exited after a pct reboot pinned THREE sub-causes, fixed together (hardening the existing processGuestBootChange, not a parallel mechanism): 1. Agent-path blocker (live root cause): agentClient() returned "agent not configured" (cfg.LocalAPI.Endpoint empty), so processGuestBootChange AND the whole drive gate bailed at the first guard. bootstrap.json had a complete local_api block, but MaybeIngest returned immediately on "already configured" so a controller.yaml seeded before local_api existed never got the agent path. Fix: MaybeIngest now calls ensureLocalAPI on the already-configured path, merging local_api from bootstrap.json into the existing controller.yaml when missing (no hub re-pull, config preserved; idempotent + fail-safe). 2. Boot-race readiness gate: processGuestBootChange sampled BoundUnderParent once during fast startup, racing the ~18s rebind, recreated nothing, burned its boot-id one-shot. Fix: gate on the REAL live in-guest bind -- driveBindLive checks /mnt/felhom-drives/<drive> is a mountpoint in the controller's own /mnt rslave /proc/self/mountinfo; pollLiveBinds waits for it (bounded ~120s) before recreating via the normal pipeline. shouldRecreateOnBoot stays state-independent so stuck-Exited create-time-failure apps are included. 3. Single-shot fragility: processGuestBootChange ran only once at startup; a briefly-unreachable agent right after a guest reboot stranded recovery. Fix: driveGateLoop runs it every periodic tick too (idempotent, boot-id gated). Tests (non-hollow, pre-fix companions, red-proofed): pollLiveBinds waits then reports live / never-live stays absent / single early sample misses; ensureLocalAPI merges local_api into a configured controller.yaml that lacks it / no-ops when present. Live-accepted with repeated pct reboot 9201. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -99,7 +99,12 @@ func Path() string {
|
||||
// - On success: writes controller.yaml (0600, atomic), reloads it, and returns the reloaded cfg.
|
||||
func MaybeIngest(configPath string, cfg *config.Config, logger *log.Logger, pull PullFunc) *config.Config {
|
||||
if cfg != nil && cfg.Customer.ID != "" {
|
||||
return cfg // already configured — do not clobber, do not pull (idempotent)
|
||||
// Already configured — do NOT re-pull the hub config or clobber controller.yaml. But STILL
|
||||
// ensure the per-guest local_api block is present: a controller.yaml that was seeded/pulled
|
||||
// before local_api existed (or by the setup wizard) is "configured" yet has no agent path, so
|
||||
// agentClient() returns "agent not configured" and the ENTIRE drive gate + guest-reboot
|
||||
// recovery silently die. ensureLocalAPI merges it in from bootstrap.json if missing.
|
||||
return ensureLocalAPI(configPath, cfg, logger)
|
||||
}
|
||||
bpath := Path()
|
||||
data, err := os.ReadFile(bpath)
|
||||
@@ -184,6 +189,50 @@ func pullWithRetry(pull PullFunc, hubURL, customerID, password string, logger *l
|
||||
return "", lastErr
|
||||
}
|
||||
|
||||
// ensureLocalAPI handles an ALREADY-configured controller whose controller.yaml lacks the per-guest
|
||||
// local_api block (seeded/pulled before local_api existed, or set up via the wizard). Without it the
|
||||
// controller cannot reach the host agent at all (agentClient → "agent not configured"), which silently
|
||||
// kills the whole drive gate + guest-reboot recovery. If bootstrap.json carries a complete local_api
|
||||
// block, this merges it into the existing controller.yaml in place and reloads. Idempotent + fail-safe:
|
||||
// returns cfg unchanged when local_api is already present, the bootstrap is absent/incomplete, or any
|
||||
// step fails (it must never brick a configured guest).
|
||||
func ensureLocalAPI(configPath string, cfg *config.Config, logger *log.Logger) *config.Config {
|
||||
if cfg == nil || cfg.LocalAPI.Endpoint != "" {
|
||||
return cfg // already has the agent path → nothing to do
|
||||
}
|
||||
data, err := os.ReadFile(Path())
|
||||
if err != nil {
|
||||
return cfg // no bootstrap → nothing to merge (legacy/manually-configured guest)
|
||||
}
|
||||
var b Bootstrap
|
||||
if err := json.Unmarshal(data, &b); err != nil {
|
||||
return cfg
|
||||
}
|
||||
if b.LocalAPI.Endpoint == "" || b.LocalAPI.Fingerprint == "" || b.LocalAPI.Token == "" {
|
||||
return cfg // bootstrap has no usable local_api to merge
|
||||
}
|
||||
current, err := os.ReadFile(configPath)
|
||||
if err != nil {
|
||||
return cfg
|
||||
}
|
||||
merged, err := mergeLocalAPI(string(current), b.LocalAPI)
|
||||
if err != nil {
|
||||
logger.Printf("[WARN] bootstrap: merging local_api into existing config failed: %v — agent path stays unconfigured", err)
|
||||
return cfg
|
||||
}
|
||||
if err := writeFileAtomic(configPath, merged); err != nil {
|
||||
logger.Printf("[WARN] bootstrap: could not write %s with local_api: %v", configPath, err)
|
||||
return cfg
|
||||
}
|
||||
reloaded, err := config.LoadPermissive(configPath)
|
||||
if err != nil {
|
||||
logger.Printf("[WARN] bootstrap: wrote local_api but reload failed: %v", err)
|
||||
return cfg
|
||||
}
|
||||
logger.Printf("[INFO] bootstrap: existing config was missing local_api — merged from %s (%s); agent path now configured", Path(), b.LocalAPI.Endpoint)
|
||||
return reloaded
|
||||
}
|
||||
|
||||
// mergeLocalAPI parses the pulled controller.yaml as a generic map, sets the local_api block from the
|
||||
// bootstrap (overwriting any hub-emitted placeholder), and re-marshals. local_api.enabled is NOT set
|
||||
// — it defaults on once endpoint is present (config.LocalAPIConfig).
|
||||
|
||||
Reference in New Issue
Block a user