felhom-controller/REPORT.md

REPORT — controller v0.71.0: fix guest-reboot recovery of drive-backed apps (2026-06-16)

Deployed: controller v0.71.0 on guest 9201 / felhom-pve (bootstrap-managed; healthy). Commits (trunk): 25e5cb5 (boot-race fix, first cut) → e2de234 (full fix: + agent-path blocker + periodic retry + docs/tests). Live-accepted: two pct reboot 9201 cycles, all 8 drive-backed apps recovered automatically, zero manual starts.

Phase A — diagnosis (pinned live, not guessed): THREE sub-causes

A guest pct reboot strands drive-backed apps because in-guest dockerd auto-starts the unless-stopped apps ~18s BEFORE the agent re-binds the drive, so the create-time volume bind fails (mkdir /mnt/felhom-drives/<drive>/userdata: permission denied) and RestartCount=0 means it is never retried → stuck Exited. The intended recovery (processGuestBootChange) did not fire. Live repro (paperless-webserver: exit=128, RestartCount=0, State.Error=mkdir…permission denied) was the fixture. Diagnosis pinned three distinct sub-causes:

1. AGENT-PATH BLOCKER (the live root cause). /api/disks → {"error":"agent not configured"}: agentClient() requires cfg.LocalAPI.Endpoint, which was empty, so processGuestBootChange (and the entire drive gate) returned at its first guard — never reaching any boot-id/bind logic. The authoritative bootstrap.json had a complete local_api block (endpoint/fingerprint/token), but bootstrap.MaybeIngest (bootstrap.go:101) returned immediately on "already configured" (cfg.Customer.ID != ""), so a controller.yaml seeded before local_api existed never got it merged. Evidence: LastGuestBootID was stuck at the guest's first-boot value across every reboot (it was never updated because the function bailed before reaching the persist).
2. BOOT-RACE READINESS GATE. processGuestBootChange sampled the agent's BoundUnderParent once during fast startup (racing the ~18s rebind), recreated nothing, and persisted the new boot-id — burning its one-shot. (Confirmed by manually wiring local_api: the old sample-once would still have missed; the new poll caught it.)
3. SINGLE-SHOT FRAGILITY. processGuestBootChange ran only once at startup; right after a guest reboot the agent's local API can be briefly unreachable/stale, so the single attempt bailed with no retry.

(The periodic drive-gate never recovered them either: its first observation was after the rebind → present + not-disconnected → no transition; settings showed felhom-flash disconnected=None.)

Phase B — fix (harden the existing mechanism, no parallel one)

1. ensureLocalAPI (internal/bootstrap/bootstrap.go): MaybeIngest now calls it on the already-configured path — when cfg.LocalAPI.Endpoint is empty it merges local_api from bootstrap.json into the existing controller.yaml in place (no hub re-pull, existing config preserved), idempotent + fail-safe.
2. driveBindLive + pollLiveBinds (internal/web/intermediary.go): processGuestBootChange now gates on the real live in-guest bind — driveBindLive checks whether /mnt/felhom-drives/<drive> is an actual mountpoint in the controller's own /mnt (rslave) /proc/self/mountinfo (true only once the agent's bind propagated, exactly when docker can recreate the app); pollLiveBinds waits for it (bounded ~120s, poll 2s) before recreating via the normal pipeline (compose down→up -d). shouldRecreateOnBoot is unchanged and state-independent → stuck-Exited create-time-failure apps are included. Drives that never go live in the window are left to the gate.
3. Periodic retry (driveGateLoop): processGuestBootChange now runs on every periodic tick too — idempotent (boot-id gated) — so a momentarily-unreachable agent right after a reboot no longer permanently strands recovery.

Both reboot paths share this code, the same agent dependency, and the same boot-race — so both were broken by the regression and both are fixed here (see the regression analysis + host-reboot drill below).

Phase C — tests (non-hollow, pre-fix companions, red-proofed)

• internal/web/intermediary_test.go: pollLiveBinds waits through the rebind window then reports live (recreate fires); a never-live drive stays absent (no spurious recreate); an explicit companion that a single early sample misses the not-yet-live bind. Red-proofed against a no-wait single-sample.
• internal/bootstrap/bootstrap_test.go: ensureLocalAPI merges local_api into an already-configured controller.yaml that lacks it (companion: pre-fix MaybeIngest left LocalAPI.Endpoint empty — red-proofed) and no-ops when already present. Full controller suite green; go vet clean.

Phase D — live acceptance (the real gate)

Built + deployed felhom-controller:0.71.0 to guest 9201 (the redeploy itself validated ensureLocalAPI: the container recreate reset controller.yaml, the code re-merged local_api, /api/disks → 200). Then two pct reboot 9201 cycles (zero manual intervention):

reboot	boot-id	gate log	result
#1	…7348791	"waiting (≤2m0s) for live drive bind(s) … → live bind confirmed — recreating" ×8	all 8 Up
#2	…7367438	same full sequence ×8	all 8 Up

Both recovered all 8 drive-backed apps automatically. (komga reports its container healthcheck "unhealthy" but is up and serving — a pre-existing, unrelated issue.)

HOST reboot (re-drilled after the follow-up challenge — the guest reboots only prove the guest path): two systemctl reboot of felhom-pve itself.

host reboot	host btime	boot-id	result
#1	1781545822→1781620637	1781620637-1443 — recreate ×8 (apps were state=stopped)	all 8 Up
#2	1781620637→1781620928	1781620928-1516 — recreate ×8	all 8 Up

Both recovered automatically, zero manual starts. Two findings: (a) the host-btime prefix advances, so the host-reboot path triggers processGuestBootChange (the persisted LastGuestBootID now tracks each boot — it was frozen at the first-boot value before the fix); (b) the boot-race manifests on host reboots too (the gate recreated state=stopped apps), so the host path is not immune — it has the same race and the same agent-path dependency as the guest path.

Regression analysis — "how did the earlier (v0.68) host-reboot sweep pass?"

It genuinely passed and genuinely exercised processGuestBootChange: documentation/audits/storage-lifecycle-acceptance-2026-06-15.md records the host reboot surfacing a real bug (5 apps stayed exited because the recreate filtered on container state, fixed in v0.68.1). If agentClient() had been failing then, processGuestBootChange would have bailed and that bug could never have appeared. So the agent path worked at v0.68 and regressed afterward — "boot-id determinism was never exercising" is false; it was.

The regression: controller.yaml is reset to the golden's "configured-but-no-local_api" baseline on every container recreate (each deploy), and the old MaybeIngest returned immediately on "already configured" → local_api was never re-merged → agentClient() → "agent not configured" → the whole drive gate + boot recovery silently died, on both reboot paths. (My v0.70.0 config-apply round-trip is exonerated: GET /api/config returns the file verbatim — router.go:1150, no redaction — so it preserved local_api.) ensureLocalAPI re-merges local_api on every startup, closing the regression permanently.

Follow-up — FileBrowser was NOT recovered (host-reboot recovery was incomplete)

After the host-reboot drills, FileBrowser was found down (state=created, exit=128, RestartCount=0, mkdir /mnt/felhom-drives/felhom-usb/userdata: permission denied). My "all recovered" claim above was incomplete: it only checked the 8 app stacks, which are all on felhom-flash. Two distinct gaps:

• (A) Agent-side — felhom-usb did not re-mount after the host reboot. The host reboot swapped the USB device letters (sdb↔sdc); felhom-flash re-mounted (now sdc1), but felhom-usb (sdb1, da9e7089) did not — its systemd mount unit (mnt-felhom\x2dusb.mount) existed and the by-uuid symlink resolved, yet the unit was not active, so the agent reported the (present) drive as enrolled drive not present (durable-id absent) every reconcile and never bound it. So the host-reboot recovery covered felhom-flash apps but left felhom-usb entirely unmounted — a felhom-agent bug (it must re-activate/retry the mount once the USB enumerates after a host reboot, not give up on "absent"). Out of the controller's scope.
• (B) Controller-side — FileBrowser is not covered by the boot-recovery. FileBrowser binds all three drives' userdata but is base-infra (no HDD_PATH), so processGuestBootChange/shouldRecreateOnBoot skip it. Its recovery relies on SyncFileBrowserMounts, which runs once at controller startup — racing the bind exactly like the app boot-race — and is not retried after the binds go live. So FileBrowser is stranded by a guest/host reboot independently of (A). Recommended fix (separate, small): have processGuestBootChange call SyncFileBrowserMounts after the live-bind poll confirms the binds, so FileBrowser converges with the apps. Not done here (it's a code change + version bump; flagged for a follow-up slice). Note it would only fully help once (A) is fixed — on this reboot felhom-usb was absent, so FB could not bind it regardless.

Recovered live (non-destructive): started the felhom-usb mount unit (systemctl start /mnt/felhom-usb) → agent bound it into the guest → restarted the controller → SyncFileBrowserMounts synced 3 storage paths → FileBrowser Up (healthy). felhom-usb's real userdata (drwxrwsr-x root:1000) was intact all along — only hidden behind the unmounted placeholder.

Notes

• The Komga healthcheck quirk is pre-existing and out of scope.
• The local_api-merge gap affected the whole drive gate on this golden, not just boot recovery — ensureLocalAPI restores the agent path generally.
• Trust caveat: the reboot drills prove the felhom-flash app recovery path (guest ×2, host ×2). They do not yet prove a clean felhom-usb host-reboot remount (gap A) or FileBrowser auto-recovery (gap B) — both are now documented and recommended for follow-up.