controller v0.71.0: fix guest-reboot boot-race stranding drive-backed apps

Sub-cause: on guest pct reboot, in-guest dockerd auto-starts unless-stopped
drive-backed apps ~18s BEFORE the agent re-binds the drive; the create-time
volume bind fails (mkdir /mnt/felhom-drives/<drive>/userdata: permission denied)
and RestartCount=0 means it's never retried -> stuck Exited. The existing
recovery (processGuestBootChange) RAN but raced the rebind: it sampled the
agent's BoundUnderParent once during fast startup (not live yet), recreated
nothing, and persisted the new boot-id -> burned its one-shot. The periodic gate
never recovered them either (first observation after the rebind -> no transition).

Fix (harden the existing mechanism, no parallel one): processGuestBootChange now
gates on the REAL live in-guest bind. driveBindLive checks whether
/mnt/felhom-drives/<drive> is an actual mountpoint in the controller's own /mnt
(rslave) /proc/self/mountinfo -- true only once the agent's bind propagated,
exactly when docker can recreate the app. pollLiveBinds waits for that (bounded
~120s, poll 2s; rebind lands ~18s) and only then recreates via the normal
pipeline, including stuck-Exited create-time-failure apps (shouldRecreateOnBoot
is state-independent). Single-flight; absent-after-window drives left to the gate;
host-reboot path unaffected; guest-only reboot path now covered.

Tests: pollLiveBinds waits through the rebind then reports live (recreate fires);
never-live drive stays absent; pre-fix companion (single early sample misses the
not-yet-live bind). Red-proofed against a no-wait single-sample.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-16 15:46:51 +02:00
parent 38c82db522
commit 25e5cb5850
4 changed files with 213 additions and 13 deletions
+29
View File
@@ -1,5 +1,34 @@
## Changelog
### v0.71.0 — fix the guest-reboot boot-race that strands drive-backed apps (2026-06-16)
A `pct reboot` of the guest left drive-backed apps (audiobookshelf, calibre-web, immich-server,
jellyfin, komga, radarr, romm, paperless-webserver) stuck `Exited` forever. **Sub-cause (diagnosed,
not guessed):** on guest boot the in-guest dockerd auto-starts the `unless-stopped` apps **~18s before**
the agent re-binds the drive under the stable parent; the create-time volume bind fails
(`mkdir /mnt/felhom-drives/<drive>/userdata: permission denied` on the empty fail-closed placeholder)
and, being a create-time failure (`RestartCount=0`), is **never retried**. The existing recovery,
`processGuestBootChange`, *ran* but **raced the rebind**: it sampled the agent's `BoundUnderParent`
**once** during fast controller startup (bind not live yet) → recreated nothing → **persisted the new
boot-id**, burning its one-shot. The periodic drive-gate never recovered them either (its first
observation was *after* the rebind: present + not-disconnected → no transition).
**Fix (harden the existing mechanism — no parallel one):** `processGuestBootChange` now gates on the
**REAL live in-guest bind** instead of the once-sampled agent view. New `driveBindLive` checks whether
`/mnt/felhom-drives/<drive>` is an actual mountpoint in the controller's own `/mnt` (rslave)
`/proc/self/mountinfo` — true only once the agent's bind has propagated, exactly when docker can
recreate the app. New `pollLiveBinds` waits for that (bounded ~120s, polling 2s; the rebind lands ~18s)
and only then recreates the deployed drive-backed apps via the normal pipeline (`compose down``up -d`).
`shouldRecreateOnBoot` is unchanged and state-independent, so a stuck-`Exited` create-time-failure app
is included. Single-flight (runs once before the periodic gate); apps on a drive that never goes live
in the window are left to the gate (drive-absent → stop→return→restart). The host-reboot path the
earlier sweep validated is unaffected (same code path, now strictly more robust — it waits for the
bind). The **guest-only reboot path** (which the host-reboot sweep never exercised) is now covered.
Tests: `pollLiveBinds` waits through the rebind window then reports live (recreate fires);
never-live drive stays absent (no spurious recreate); plus an explicit pre-fix companion that a single
early sample misses the not-yet-live bind. Live-accepted with repeated `pct reboot 9201`.
### v0.70.0 — config-apply self-restart + geo-restriction UX fixes (2026-06-16)
Fixes found during live geo testing (rotating the Cloudflare API token).