felhom-controller/REPORT.md

REPORT — controller v0.71.0: fix guest-reboot recovery of drive-backed apps (2026-06-16)

Deployed: controller v0.71.0 on guest 9201 / felhom-pve (bootstrap-managed; healthy). Commits (trunk): 25e5cb5 (boot-race fix, first cut) → e2de234 (full fix: + agent-path blocker + periodic retry + docs/tests). Live-accepted: two pct reboot 9201 cycles, all 8 drive-backed apps recovered automatically, zero manual starts.

Phase A — diagnosis (pinned live, not guessed): THREE sub-causes

A guest pct reboot strands drive-backed apps because in-guest dockerd auto-starts the unless-stopped apps ~18s BEFORE the agent re-binds the drive, so the create-time volume bind fails (mkdir /mnt/felhom-drives/<drive>/userdata: permission denied) and RestartCount=0 means it is never retried → stuck Exited. The intended recovery (processGuestBootChange) did not fire. Live repro (paperless-webserver: exit=128, RestartCount=0, State.Error=mkdir…permission denied) was the fixture. Diagnosis pinned three distinct sub-causes:

1. AGENT-PATH BLOCKER (the live root cause). /api/disks → {"error":"agent not configured"}: agentClient() requires cfg.LocalAPI.Endpoint, which was empty, so processGuestBootChange (and the entire drive gate) returned at its first guard — never reaching any boot-id/bind logic. The authoritative bootstrap.json had a complete local_api block (endpoint/fingerprint/token), but bootstrap.MaybeIngest (bootstrap.go:101) returned immediately on "already configured" (cfg.Customer.ID != ""), so a controller.yaml seeded before local_api existed never got it merged. Evidence: LastGuestBootID was stuck at the guest's first-boot value across every reboot (it was never updated because the function bailed before reaching the persist).
2. BOOT-RACE READINESS GATE. processGuestBootChange sampled the agent's BoundUnderParent once during fast startup (racing the ~18s rebind), recreated nothing, and persisted the new boot-id — burning its one-shot. (Confirmed by manually wiring local_api: the old sample-once would still have missed; the new poll caught it.)
3. SINGLE-SHOT FRAGILITY. processGuestBootChange ran only once at startup; right after a guest reboot the agent's local API can be briefly unreachable/stale, so the single attempt bailed with no retry.

(The periodic drive-gate never recovered them either: its first observation was after the rebind → present + not-disconnected → no transition; settings showed felhom-flash disconnected=None.)

Phase B — fix (harden the existing mechanism, no parallel one)

1. ensureLocalAPI (internal/bootstrap/bootstrap.go): MaybeIngest now calls it on the already-configured path — when cfg.LocalAPI.Endpoint is empty it merges local_api from bootstrap.json into the existing controller.yaml in place (no hub re-pull, existing config preserved), idempotent + fail-safe.
2. driveBindLive + pollLiveBinds (internal/web/intermediary.go): processGuestBootChange now gates on the real live in-guest bind — driveBindLive checks whether /mnt/felhom-drives/<drive> is an actual mountpoint in the controller's own /mnt (rslave) /proc/self/mountinfo (true only once the agent's bind propagated, exactly when docker can recreate the app); pollLiveBinds waits for it (bounded ~120s, poll 2s) before recreating via the normal pipeline (compose down→up -d). shouldRecreateOnBoot is unchanged and state-independent → stuck-Exited create-time-failure apps are included. Drives that never go live in the window are left to the gate.
3. Periodic retry (driveGateLoop): processGuestBootChange now runs on every periodic tick too — idempotent (boot-id gated) — so a momentarily-unreachable agent right after a reboot no longer permanently strands recovery.

The host-reboot path the earlier sweep validated is unaffected (same code path, strictly more robust); the guest-only reboot path (never exercised by host-reboot sweeps) is now covered.

Phase C — tests (non-hollow, pre-fix companions, red-proofed)

• internal/web/intermediary_test.go: pollLiveBinds waits through the rebind window then reports live (recreate fires); a never-live drive stays absent (no spurious recreate); an explicit companion that a single early sample misses the not-yet-live bind. Red-proofed against a no-wait single-sample.
• internal/bootstrap/bootstrap_test.go: ensureLocalAPI merges local_api into an already-configured controller.yaml that lacks it (companion: pre-fix MaybeIngest left LocalAPI.Endpoint empty — red-proofed) and no-ops when already present. Full controller suite green; go vet clean.

Phase D — live acceptance (the real gate)

Built + deployed felhom-controller:0.71.0 to guest 9201 (the redeploy itself validated ensureLocalAPI: the container recreate reset controller.yaml, the code re-merged local_api, /api/disks → 200). Then two pct reboot 9201 cycles (zero manual intervention):

reboot	boot-id	gate log	result
#1	…7348791	"waiting (≤2m0s) for live drive bind(s) … → live bind confirmed — recreating" ×8	all 8 Up
#2	…7367438	same full sequence ×8	all 8 Up

Both recovered audiobookshelf, calibre-web, immich-server, jellyfin, komga, radarr, romm, paperless-webserver automatically. (komga reports its container healthcheck "unhealthy" but is up and serving — a pre-existing, unrelated issue.) The boot-id now advances correctly on each reboot.

Notes

• The Komga healthcheck quirk is pre-existing and out of scope.
• The local_api-merge gap likely affected the whole drive gate on this golden, not just boot recovery — the ensureLocalAPI fix restores the agent path generally.