felhom-controller/REPORT.md

# REPORT — controller v0.71.0: fix guest-reboot recovery of drive-backed apps (2026-06-16)

**Deployed:** controller **v0.71.0** on guest 9201 / felhom-pve (bootstrap-managed; healthy).
**Commits (trunk):** `25e5cb5` (boot-race fix, first cut) → `e2de234` (full fix: + agent-path blocker + periodic retry + docs/tests).
**Live-accepted:** two `pct reboot 9201` cycles, all 8 drive-backed apps recovered automatically, zero manual starts.

## Phase A — diagnosis (pinned live, not guessed): THREE sub-causes

A guest `pct reboot` strands drive-backed apps because in-guest dockerd auto-starts the `unless-stopped`
apps ~18s BEFORE the agent re-binds the drive, so the create-time volume bind fails
(`mkdir /mnt/felhom-drives/<drive>/userdata: permission denied`) and `RestartCount=0` means it is never
retried → stuck `Exited`. The intended recovery (`processGuestBootChange`) did **not** fire. Live repro
(`paperless-webserver`: `exit=128, RestartCount=0, State.Error=mkdir…permission denied`) was the fixture.
Diagnosis pinned three distinct sub-causes:

1. **AGENT-PATH BLOCKER (the live root cause).** `/api/disks` → `{"error":"agent not configured"}`:
   `agentClient()` requires `cfg.LocalAPI.Endpoint`, which was **empty**, so `processGuestBootChange`
   (and the **entire** drive gate) returned at its first guard — never reaching any boot-id/bind logic.
   The authoritative `bootstrap.json` *had* a complete `local_api` block (`endpoint/fingerprint/token`),
   but `bootstrap.MaybeIngest` (bootstrap.go:101) returned immediately on "already configured"
   (`cfg.Customer.ID != ""`), so a controller.yaml seeded before `local_api` existed never got it merged.
   Evidence: `LastGuestBootID` was stuck at the guest's first-boot value across every reboot (it was never
   updated because the function bailed before reaching the persist).
2. **BOOT-RACE READINESS GATE.** `processGuestBootChange` sampled the agent's `BoundUnderParent` **once**
   during fast startup (racing the ~18s rebind), recreated nothing, and **persisted the new boot-id** —
   burning its one-shot. (Confirmed by manually wiring `local_api`: the *old* sample-once would still have
   missed; the new poll caught it.)
3. **SINGLE-SHOT FRAGILITY.** `processGuestBootChange` ran only once at startup; right after a guest reboot
   the agent's local API can be briefly unreachable/stale, so the single attempt bailed with no retry.

(The periodic drive-gate never recovered them either: its first observation was *after* the rebind →
present + not-disconnected → no transition; settings showed `felhom-flash disconnected=None`.)

## Phase B — fix (harden the existing mechanism, no parallel one)

1. **`ensureLocalAPI`** (`internal/bootstrap/bootstrap.go`): `MaybeIngest` now calls it on the
   already-configured path — when `cfg.LocalAPI.Endpoint` is empty it merges `local_api` from
   bootstrap.json into the existing controller.yaml in place (no hub re-pull, existing config preserved),
   idempotent + fail-safe.
2. **`driveBindLive` + `pollLiveBinds`** (`internal/web/intermediary.go`): `processGuestBootChange` now
   gates on the **real live in-guest bind** — `driveBindLive` checks whether `/mnt/felhom-drives/<drive>`
   is an actual mountpoint in the controller's own `/mnt` (rslave) `/proc/self/mountinfo` (true only once
   the agent's bind propagated, exactly when docker can recreate the app); `pollLiveBinds` waits for it
   (bounded ~120s, poll 2s) before recreating via the normal pipeline (`compose down`→`up -d`).
   `shouldRecreateOnBoot` is unchanged and state-independent → stuck-`Exited` create-time-failure apps are
   included. Drives that never go live in the window are left to the gate.
3. **Periodic retry** (`driveGateLoop`): `processGuestBootChange` now runs on every periodic tick too —
   idempotent (boot-id gated) — so a momentarily-unreachable agent right after a reboot no longer
   permanently strands recovery.

The host-reboot path the earlier sweep validated is unaffected (same code path, strictly more robust); the
**guest-only reboot path** (never exercised by host-reboot sweeps) is now covered.

## Phase C — tests (non-hollow, pre-fix companions, red-proofed)

- `internal/web/intermediary_test.go`: `pollLiveBinds` waits through the rebind window then reports live
  (recreate fires); a never-live drive stays absent (no spurious recreate); an explicit companion that a
  **single early sample misses** the not-yet-live bind. Red-proofed against a no-wait single-sample.
- `internal/bootstrap/bootstrap_test.go`: `ensureLocalAPI` merges `local_api` into an already-configured
  controller.yaml that lacks it (companion: pre-fix `MaybeIngest` left `LocalAPI.Endpoint` empty —
  red-proofed) and no-ops when already present. Full controller suite green; `go vet` clean.

## Phase D — live acceptance (the real gate)

Built + deployed `felhom-controller:0.71.0` to guest 9201 (the redeploy itself validated `ensureLocalAPI`:
the container recreate reset controller.yaml, the code re-merged `local_api`, `/api/disks` → 200). Then
**two** `pct reboot 9201` cycles (zero manual intervention):

| reboot | boot-id | gate log | result |
|---|---|---|---|
| #1 | `…7348791` | "waiting (≤2m0s) for live drive bind(s) … → live bind confirmed — recreating" ×8 | all 8 `Up` |
| #2 | `…7367438` | same full sequence ×8 | all 8 `Up` |

Both recovered audiobookshelf, calibre-web, immich-server, jellyfin, komga, radarr, romm,
paperless-webserver automatically. (komga reports its container healthcheck "unhealthy" but is up and
serving — a pre-existing, unrelated issue.) The boot-id now advances correctly on each reboot.

## Notes

- The Komga healthcheck quirk is pre-existing and out of scope.
- The `local_api`-merge gap likely affected the whole drive gate on this golden, not just boot recovery —
  the `ensureLocalAPI` fix restores the agent path generally.