admin
14f200c9a1
Found FileBrowser down after the host-reboot drills (my 'all recovered' claim only
checked the 8 felhom-flash apps). Two distinct gaps:
(A) agent-side: felhom-usb did not re-mount after the host-reboot device-letter
swap (mount unit inactive; agent reports present drive as durable-id absent) —
a felhom-agent bug, out of controller scope.
(B) controller-side: FileBrowser is base-infra (no HDD_PATH) so processGuestBootChange
skips it; its SyncFileBrowserMounts runs once at startup, racing the bind, not
retried. Recommended fix: call SyncFileBrowserMounts after the live-bind poll.
Recovered FileBrowser live (started felhom-usb mount unit -> agent bound it ->
restarted controller -> FB Up healthy). felhom-usb userdata was intact, just hidden
behind the unmounted placeholder. Corrected the host-reboot trust caveat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
145 lines
10 KiB
Markdown
145 lines
10 KiB
Markdown
# REPORT — controller v0.71.0: fix guest-reboot recovery of drive-backed apps (2026-06-16)
|
||
|
||
**Deployed:** controller **v0.71.0** on guest 9201 / felhom-pve (bootstrap-managed; healthy).
|
||
**Commits (trunk):** `25e5cb5` (boot-race fix, first cut) → `e2de234` (full fix: + agent-path blocker + periodic retry + docs/tests).
|
||
**Live-accepted:** two `pct reboot 9201` cycles, all 8 drive-backed apps recovered automatically, zero manual starts.
|
||
|
||
## Phase A — diagnosis (pinned live, not guessed): THREE sub-causes
|
||
|
||
A guest `pct reboot` strands drive-backed apps because in-guest dockerd auto-starts the `unless-stopped`
|
||
apps ~18s BEFORE the agent re-binds the drive, so the create-time volume bind fails
|
||
(`mkdir /mnt/felhom-drives/<drive>/userdata: permission denied`) and `RestartCount=0` means it is never
|
||
retried → stuck `Exited`. The intended recovery (`processGuestBootChange`) did **not** fire. Live repro
|
||
(`paperless-webserver`: `exit=128, RestartCount=0, State.Error=mkdir…permission denied`) was the fixture.
|
||
Diagnosis pinned three distinct sub-causes:
|
||
|
||
1. **AGENT-PATH BLOCKER (the live root cause).** `/api/disks` → `{"error":"agent not configured"}`:
|
||
`agentClient()` requires `cfg.LocalAPI.Endpoint`, which was **empty**, so `processGuestBootChange`
|
||
(and the **entire** drive gate) returned at its first guard — never reaching any boot-id/bind logic.
|
||
The authoritative `bootstrap.json` *had* a complete `local_api` block (`endpoint/fingerprint/token`),
|
||
but `bootstrap.MaybeIngest` (bootstrap.go:101) returned immediately on "already configured"
|
||
(`cfg.Customer.ID != ""`), so a controller.yaml seeded before `local_api` existed never got it merged.
|
||
Evidence: `LastGuestBootID` was stuck at the guest's first-boot value across every reboot (it was never
|
||
updated because the function bailed before reaching the persist).
|
||
2. **BOOT-RACE READINESS GATE.** `processGuestBootChange` sampled the agent's `BoundUnderParent` **once**
|
||
during fast startup (racing the ~18s rebind), recreated nothing, and **persisted the new boot-id** —
|
||
burning its one-shot. (Confirmed by manually wiring `local_api`: the *old* sample-once would still have
|
||
missed; the new poll caught it.)
|
||
3. **SINGLE-SHOT FRAGILITY.** `processGuestBootChange` ran only once at startup; right after a guest reboot
|
||
the agent's local API can be briefly unreachable/stale, so the single attempt bailed with no retry.
|
||
|
||
(The periodic drive-gate never recovered them either: its first observation was *after* the rebind →
|
||
present + not-disconnected → no transition; settings showed `felhom-flash disconnected=None`.)
|
||
|
||
## Phase B — fix (harden the existing mechanism, no parallel one)
|
||
|
||
1. **`ensureLocalAPI`** (`internal/bootstrap/bootstrap.go`): `MaybeIngest` now calls it on the
|
||
already-configured path — when `cfg.LocalAPI.Endpoint` is empty it merges `local_api` from
|
||
bootstrap.json into the existing controller.yaml in place (no hub re-pull, existing config preserved),
|
||
idempotent + fail-safe.
|
||
2. **`driveBindLive` + `pollLiveBinds`** (`internal/web/intermediary.go`): `processGuestBootChange` now
|
||
gates on the **real live in-guest bind** — `driveBindLive` checks whether `/mnt/felhom-drives/<drive>`
|
||
is an actual mountpoint in the controller's own `/mnt` (rslave) `/proc/self/mountinfo` (true only once
|
||
the agent's bind propagated, exactly when docker can recreate the app); `pollLiveBinds` waits for it
|
||
(bounded ~120s, poll 2s) before recreating via the normal pipeline (`compose down`→`up -d`).
|
||
`shouldRecreateOnBoot` is unchanged and state-independent → stuck-`Exited` create-time-failure apps are
|
||
included. Drives that never go live in the window are left to the gate.
|
||
3. **Periodic retry** (`driveGateLoop`): `processGuestBootChange` now runs on every periodic tick too —
|
||
idempotent (boot-id gated) — so a momentarily-unreachable agent right after a reboot no longer
|
||
permanently strands recovery.
|
||
|
||
Both reboot paths share this code, the same agent dependency, and the same boot-race — so both were
|
||
broken by the regression and both are fixed here (see the regression analysis + host-reboot drill below).
|
||
|
||
## Phase C — tests (non-hollow, pre-fix companions, red-proofed)
|
||
|
||
- `internal/web/intermediary_test.go`: `pollLiveBinds` waits through the rebind window then reports live
|
||
(recreate fires); a never-live drive stays absent (no spurious recreate); an explicit companion that a
|
||
**single early sample misses** the not-yet-live bind. Red-proofed against a no-wait single-sample.
|
||
- `internal/bootstrap/bootstrap_test.go`: `ensureLocalAPI` merges `local_api` into an already-configured
|
||
controller.yaml that lacks it (companion: pre-fix `MaybeIngest` left `LocalAPI.Endpoint` empty —
|
||
red-proofed) and no-ops when already present. Full controller suite green; `go vet` clean.
|
||
|
||
## Phase D — live acceptance (the real gate)
|
||
|
||
Built + deployed `felhom-controller:0.71.0` to guest 9201 (the redeploy itself validated `ensureLocalAPI`:
|
||
the container recreate reset controller.yaml, the code re-merged `local_api`, `/api/disks` → 200). Then
|
||
**two** `pct reboot 9201` cycles (zero manual intervention):
|
||
|
||
| reboot | boot-id | gate log | result |
|
||
|---|---|---|---|
|
||
| #1 | `…7348791` | "waiting (≤2m0s) for live drive bind(s) … → live bind confirmed — recreating" ×8 | all 8 `Up` |
|
||
| #2 | `…7367438` | same full sequence ×8 | all 8 `Up` |
|
||
|
||
Both recovered all 8 drive-backed apps automatically. (komga reports its container healthcheck
|
||
"unhealthy" but is up and serving — a pre-existing, unrelated issue.)
|
||
|
||
**HOST reboot (re-drilled after the follow-up challenge — the guest reboots only prove the guest path):**
|
||
two `systemctl reboot` of felhom-pve itself.
|
||
|
||
| host reboot | host btime | boot-id | result |
|
||
|---|---|---|---|
|
||
| #1 | `1781545822`→`1781620637` | `1781620637-1443` — recreate ×8 (apps were `state=stopped`) | all 8 `Up` |
|
||
| #2 | `1781620637`→`1781620928` | `1781620928-1516` — recreate ×8 | all 8 `Up` |
|
||
|
||
Both recovered automatically, zero manual starts. Two findings: (a) the **host-btime prefix advances**, so
|
||
the host-reboot path triggers `processGuestBootChange` (the persisted `LastGuestBootID` now tracks each
|
||
boot — it was frozen at the first-boot value before the fix); (b) the **boot-race manifests on host
|
||
reboots too** (the gate recreated `state=stopped` apps), so the host path is **not** immune — it has the
|
||
same race and the same agent-path dependency as the guest path.
|
||
|
||
## Regression analysis — "how did the earlier (v0.68) host-reboot sweep pass?"
|
||
|
||
It genuinely passed and genuinely exercised `processGuestBootChange`:
|
||
`documentation/audits/storage-lifecycle-acceptance-2026-06-15.md` records the host reboot **surfacing a
|
||
real bug** (5 apps stayed exited because the recreate filtered on container state, fixed in v0.68.1). If
|
||
`agentClient()` had been failing then, `processGuestBootChange` would have bailed and that bug could never
|
||
have appeared. **So the agent path worked at v0.68 and regressed afterward** — "boot-id determinism was
|
||
never exercising" is false; it was.
|
||
|
||
The regression: `controller.yaml` is reset to the golden's "configured-but-no-`local_api`" baseline on
|
||
every container recreate (each deploy), and the old `MaybeIngest` returned immediately on "already
|
||
configured" → `local_api` was never re-merged → `agentClient()` → "agent not configured" → the whole
|
||
drive gate + boot recovery silently died, on **both** reboot paths. (My v0.70.0 config-apply round-trip is
|
||
**exonerated**: `GET /api/config` returns the file verbatim — `router.go:1150`, no redaction — so it
|
||
preserved `local_api`.) `ensureLocalAPI` re-merges `local_api` on every startup, closing the regression
|
||
permanently.
|
||
|
||
## Follow-up — FileBrowser was NOT recovered (host-reboot recovery was incomplete)
|
||
|
||
After the host-reboot drills, **FileBrowser was found down** (`state=created, exit=128, RestartCount=0`,
|
||
`mkdir /mnt/felhom-drives/felhom-usb/userdata: permission denied`). My "all recovered" claim above was
|
||
**incomplete**: it only checked the 8 app stacks, which are all on **felhom-flash**. Two distinct gaps:
|
||
|
||
- **(A) Agent-side — felhom-usb did not re-mount after the host reboot.** The host reboot swapped the USB
|
||
device letters (`sdb↔sdc`); felhom-flash re-mounted (now `sdc1`), but **felhom-usb (`sdb1`,
|
||
`da9e7089`) did not** — its systemd mount unit (`mnt-felhom\x2dusb.mount`) existed and the by-uuid
|
||
symlink resolved, yet the unit was **not active**, so the agent reported the (present) drive as
|
||
`enrolled drive not present (durable-id absent)` every reconcile and never bound it. So the
|
||
host-reboot recovery covered felhom-flash apps but left **felhom-usb entirely unmounted** — a
|
||
**felhom-agent** bug (it must re-activate/retry the mount once the USB enumerates after a host reboot,
|
||
not give up on "absent"). **Out of the controller's scope.**
|
||
- **(B) Controller-side — FileBrowser is not covered by the boot-recovery.** FileBrowser binds all three
|
||
drives' `userdata` but is **base-infra (no `HDD_PATH`)**, so `processGuestBootChange`/`shouldRecreateOnBoot`
|
||
skip it. Its recovery relies on `SyncFileBrowserMounts`, which runs once at controller startup — racing
|
||
the bind exactly like the app boot-race — and is not retried after the binds go live. So FileBrowser is
|
||
stranded by a guest/host reboot independently of (A). **Recommended fix (separate, small): have
|
||
`processGuestBootChange` call `SyncFileBrowserMounts` after the live-bind poll confirms the binds, so
|
||
FileBrowser converges with the apps.** Not done here (it's a code change + version bump; flagged for a
|
||
follow-up slice). Note it would only fully help once (A) is fixed — on this reboot felhom-usb was absent,
|
||
so FB could not bind it regardless.
|
||
|
||
**Recovered live (non-destructive):** started the felhom-usb mount unit (`systemctl start /mnt/felhom-usb`)
|
||
→ agent bound it into the guest → restarted the controller → `SyncFileBrowserMounts` synced 3 storage
|
||
paths → **FileBrowser `Up (healthy)`**. felhom-usb's real `userdata` (`drwxrwsr-x root:1000`) was intact
|
||
all along — only hidden behind the unmounted placeholder.
|
||
|
||
## Notes
|
||
|
||
- The Komga healthcheck quirk is pre-existing and out of scope.
|
||
- The `local_api`-merge gap affected the **whole drive gate** on this golden, not just boot recovery —
|
||
`ensureLocalAPI` restores the agent path generally.
|
||
- **Trust caveat:** the reboot drills prove the **felhom-flash app** recovery path (guest ×2, host ×2).
|
||
They do **not** yet prove a clean felhom-usb host-reboot remount (gap A) or FileBrowser auto-recovery
|
||
(gap B) — both are now documented and recommended for follow-up.
|