controller v0.71.0: fix guest-reboot recovery (boot-race + agent-path blocker)

Live diagnosis of drive-backed apps stuck Exited after a pct reboot pinned THREE
sub-causes, fixed together (hardening the existing processGuestBootChange, not a
parallel mechanism):

1. Agent-path blocker (live root cause): agentClient() returned "agent not
   configured" (cfg.LocalAPI.Endpoint empty), so processGuestBootChange AND the
   whole drive gate bailed at the first guard. bootstrap.json had a complete
   local_api block, but MaybeIngest returned immediately on "already configured"
   so a controller.yaml seeded before local_api existed never got the agent path.
   Fix: MaybeIngest now calls ensureLocalAPI on the already-configured path,
   merging local_api from bootstrap.json into the existing controller.yaml when
   missing (no hub re-pull, config preserved; idempotent + fail-safe).

2. Boot-race readiness gate: processGuestBootChange sampled BoundUnderParent once
   during fast startup, racing the ~18s rebind, recreated nothing, burned its
   boot-id one-shot. Fix: gate on the REAL live in-guest bind -- driveBindLive
   checks /mnt/felhom-drives/<drive> is a mountpoint in the controller's own /mnt
   rslave /proc/self/mountinfo; pollLiveBinds waits for it (bounded ~120s) before
   recreating via the normal pipeline. shouldRecreateOnBoot stays state-independent
   so stuck-Exited create-time-failure apps are included.

3. Single-shot fragility: processGuestBootChange ran only once at startup; a
   briefly-unreachable agent right after a guest reboot stranded recovery. Fix:
   driveGateLoop runs it every periodic tick too (idempotent, boot-id gated).

Tests (non-hollow, pre-fix companions, red-proofed): pollLiveBinds waits then
reports live / never-live stays absent / single early sample misses; ensureLocalAPI
merges local_api into a configured controller.yaml that lacks it / no-ops when
present. Live-accepted with repeated pct reboot 9201.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-16 16:12:49 +02:00
parent 25e5cb5850
commit e2de234325
5 changed files with 172 additions and 26 deletions
+37 -24
View File
@@ -1,33 +1,46 @@
## Changelog
### v0.71.0 — fix the guest-reboot boot-race that strands drive-backed apps (2026-06-16)
### v0.71.0 — fix guest-reboot recovery of drive-backed apps (boot-race + the agent-path blocker) (2026-06-16)
A `pct reboot` of the guest left drive-backed apps (audiobookshelf, calibre-web, immich-server,
jellyfin, komga, radarr, romm, paperless-webserver) stuck `Exited` forever. **Sub-cause (diagnosed,
not guessed):** on guest boot the in-guest dockerd auto-starts the `unless-stopped` apps **~18s before**
the agent re-binds the drive under the stable parent; the create-time volume bind fails
(`mkdir /mnt/felhom-drives/<drive>/userdata: permission denied` on the empty fail-closed placeholder)
and, being a create-time failure (`RestartCount=0`), is **never retried**. The existing recovery,
`processGuestBootChange`, *ran* but **raced the rebind**: it sampled the agent's `BoundUnderParent`
**once** during fast controller startup (bind not live yet) → recreated nothing → **persisted the new
boot-id**, burning its one-shot. The periodic drive-gate never recovered them either (its first
observation was *after* the rebind: present + not-disconnected → no transition).
jellyfin, komga, radarr, romm, paperless-webserver) stuck `Exited` forever. On guest boot the in-guest
dockerd auto-starts the `unless-stopped` apps **~18s before** the agent re-binds the drive under the
stable parent; the create-time volume bind fails (`mkdir /mnt/felhom-drives/<drive>/userdata:
permission denied` on the empty fail-closed placeholder) and, being a create-time failure
(`RestartCount=0`), is **never retried**. The intended recovery (`processGuestBootChange`) did not fire.
Live diagnosis pinned **three** sub-causes, fixed together (harden the existing mechanism — no parallel
one):
**Fix (harden the existing mechanism — no parallel one):** `processGuestBootChange` now gates on the
**REAL live in-guest bind** instead of the once-sampled agent view. New `driveBindLive` checks whether
`/mnt/felhom-drives/<drive>` is an actual mountpoint in the controller's own `/mnt` (rslave)
`/proc/self/mountinfo` — true only once the agent's bind has propagated, exactly when docker can
recreate the app. New `pollLiveBinds` waits for that (bounded ~120s, polling 2s; the rebind lands ~18s)
and only then recreates the deployed drive-backed apps via the normal pipeline (`compose down``up -d`).
`shouldRecreateOnBoot` is unchanged and state-independent, so a stuck-`Exited` create-time-failure app
is included. Single-flight (runs once before the periodic gate); apps on a drive that never goes live
in the window are left to the gate (drive-absent → stop→return→restart). The host-reboot path the
earlier sweep validated is unaffected (same code path, now strictly more robust — it waits for the
bind). The **guest-only reboot path** (which the host-reboot sweep never exercised) is now covered.
1. **The agent-path blocker (the live root cause).** `agentClient()` returned **"agent not configured"**
`cfg.LocalAPI.Endpoint` was empty — so `processGuestBootChange` (and the **entire** drive gate)
bailed at its first guard, never reaching any boot-id/bind logic. `bootstrap.json` *had* a complete
`local_api` block, but `MaybeIngest` returned immediately on "already configured" (customer.id set),
so a controller.yaml seeded before `local_api` existed never got the agent path merged. **Fix:**
`MaybeIngest` now calls new **`ensureLocalAPI`** on the already-configured path — it merges `local_api`
from bootstrap.json into the existing controller.yaml when missing (no hub re-pull, config preserved),
idempotent + fail-safe.
2. **The boot-race readiness gate.** `processGuestBootChange` sampled the agent's `BoundUnderParent`
**once** during fast startup, racing the ~18s rebind, recreated nothing, and burned its boot-id
one-shot. **Fix:** it now gates on the **REAL live in-guest bind** new `driveBindLive` checks
whether `/mnt/felhom-drives/<drive>` is an actual mountpoint in the controller's own `/mnt` (rslave)
`/proc/self/mountinfo` (true only once the agent's bind propagated, exactly when docker can recreate
the app), and new `pollLiveBinds` **waits** for it (bounded ~120s, poll 2s) before recreating via the
normal pipeline (`compose down``up -d`). `shouldRecreateOnBoot` is unchanged and state-independent,
so a stuck-`Exited` create-time-failure app is included.
3. **The single-shot fragility.** `processGuestBootChange` ran only once at startup; right after a guest
reboot the agent's local API can be briefly unreachable/stale, so the one attempt bailed and never
retried. **Fix:** `driveGateLoop` now runs it on every periodic tick too — idempotent (boot-id
gated), so it retries until the agent is reachable.
Tests: `pollLiveBinds` waits through the rebind window then reports live (recreate fires);
never-live drive stays absent (no spurious recreate); plus an explicit pre-fix companion that a single
early sample misses the not-yet-live bind. Live-accepted with repeated `pct reboot 9201`.
Apps on a drive that never goes live in the window are left to the normal gate. The host-reboot path the
earlier sweep validated is unaffected (same code path, strictly more robust); the **guest-only reboot
path** (never exercised by host-reboot sweeps) is now covered.
Tests (non-hollow, with pre-fix companions, red-proofed): `pollLiveBinds` waits through the rebind then
reports live (recreate fires) / never-live stays absent / a single early sample misses the not-yet-live
bind; `ensureLocalAPI` merges `local_api` into an already-configured controller.yaml that lacks it
(companion: pre-fix MaybeIngest left it empty) and no-ops when already present. Live-accepted with
repeated `pct reboot 9201`.
### v0.70.0 — config-apply self-restart + geo-restriction UX fixes (2026-06-16)