From 0d159d7e34a70d09d00ae2e5c70951f7d3350f7b Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 16 Jun 2026 18:09:58 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20REPORT/CHANGELOG/README=20=E2=80=94=20v?= =?UTF-8?q?0.72.0=20FileBrowser=20boot-recreate=20convergence=20+=20live?= =?UTF-8?q?=20x2=20reboot=20acceptance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 7 +- REPORT.md | 172 +++++++++++-------------------------------- controller/README.md | 9 +++ 3 files changed, 57 insertions(+), 131 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3012d2c..24bbab2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,8 +10,11 @@ triggers `go s.SyncFileBrowserMounts()` so FileBrowser converges against the now runs unconditionally so FileBrowser reflects the current bind state even if no app needed recreating). Refactored the recreate loop into a pure, testable `recreateDriveBackedApps(stacks, present, recreate, syncFB)`. Tests: FileBrowser sync invoked once, AFTER every recreate (red-proofed companion: pre-fix path -never synced); and synced even when nothing was recreated. Pairs with felhom-agent's host-reboot -remount-by-UUID fix; live-accepted with a real `felhom-pve` reboot. +never synced); and synced even when nothing was recreated. Pairs with felhom-agent v0.37.0's host-reboot +remount-by-UUID fix. **Live-accepted with TWO real `felhom-pve` reboots:** both fired the boot-recreate +(`live bind confirmed — recreating … → re-syncing FileBrowser mounts → FileBrowser mounts synced — 3 +storage path(s)`), all drive-backed apps recovered, FileBrowser non-stale — while the agent tolerated a +`/dev/sdb`↔`/dev/sdc` letter swap by mounting each drive by UUID. ### v0.71.0 — fix guest-reboot recovery of drive-backed apps (boot-race + the agent-path blocker) (2026-06-16) diff --git a/REPORT.md b/REPORT.md index e12910d..376d635 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,144 +1,58 @@ -# REPORT — controller v0.71.0: fix guest-reboot recovery of drive-backed apps (2026-06-16) +# REPORT — controller v0.72.0: FileBrowser converges on boot-recreate (2026-06-16) -**Deployed:** controller **v0.71.0** on guest 9201 / felhom-pve (bootstrap-managed; healthy). -**Commits (trunk):** `25e5cb5` (boot-race fix, first cut) → `e2de234` (full fix: + agent-path blocker + periodic retry + docs/tests). -**Live-accepted:** two `pct reboot 9201` cycles, all 8 drive-backed apps recovered automatically, zero manual starts. +**Deployed:** controller **v0.72.0** on guest 9201 / felhom-pve (bootstrap-managed; healthy). +**Commit (trunk):** `6ea2538` (Task B) + `c8d…`-era REPORT update. +**Live-accepted:** two REAL host reboots of felhom-pve — FileBrowser re-synced after the drive-backed +apps were recreated, both times; all drive-backed apps recovered with zero manual intervention. -## Phase A — diagnosis (pinned live, not guessed): THREE sub-causes +## Task B — the gap -A guest `pct reboot` strands drive-backed apps because in-guest dockerd auto-starts the `unless-stopped` -apps ~18s BEFORE the agent re-binds the drive, so the create-time volume bind fails -(`mkdir /mnt/felhom-drives//userdata: permission denied`) and `RestartCount=0` means it is never -retried → stuck `Exited`. The intended recovery (`processGuestBootChange`) did **not** fire. Live repro -(`paperless-webserver`: `exit=128, RestartCount=0, State.Error=mkdir…permission denied`) was the fixture. -Diagnosis pinned three distinct sub-causes: +Follow-up to v0.71.0's guest-reboot recovery. A host-reboot test found `processGuestBootChange` +recreated the drive-backed app stacks (so their `${HDD_PATH}` binds re-resolved against the now-live +drives) but **never re-synced FileBrowser**. FileBrowser is base-infra: it binds each drive's +`userdata` directory but has no `HDD_PATH`, so it is **not** in the boot-recreate set — its mounts went +stale after a reboot (the early first-boot bring-up bound them before the drives were live). -1. **AGENT-PATH BLOCKER (the live root cause).** `/api/disks` → `{"error":"agent not configured"}`: - `agentClient()` requires `cfg.LocalAPI.Endpoint`, which was **empty**, so `processGuestBootChange` - (and the **entire** drive gate) returned at its first guard — never reaching any boot-id/bind logic. - The authoritative `bootstrap.json` *had* a complete `local_api` block (`endpoint/fingerprint/token`), - but `bootstrap.MaybeIngest` (bootstrap.go:101) returned immediately on "already configured" - (`cfg.Customer.ID != ""`), so a controller.yaml seeded before `local_api` existed never got it merged. - Evidence: `LastGuestBootID` was stuck at the guest's first-boot value across every reboot (it was never - updated because the function bailed before reaching the persist). -2. **BOOT-RACE READINESS GATE.** `processGuestBootChange` sampled the agent's `BoundUnderParent` **once** - during fast startup (racing the ~18s rebind), recreated nothing, and **persisted the new boot-id** — - burning its one-shot. (Confirmed by manually wiring `local_api`: the *old* sample-once would still have - missed; the new poll caught it.) -3. **SINGLE-SHOT FRAGILITY.** `processGuestBootChange` ran only once at startup; right after a guest reboot - the agent's local API can be briefly unreachable/stale, so the single attempt bailed with no retry. +## Fix -(The periodic drive-gate never recovered them either: its first observation was *after* the rebind → -present + not-disconnected → no transition; settings showed `felhom-flash disconnected=None`.) +In `processGuestBootChange` (`internal/web/intermediary.go`), **after** `pollLiveBinds` confirms the +live binds and the drive-backed apps are recreated, the boot-recreate path now triggers +`go s.SyncFileBrowserMounts()` so FileBrowser converges against the now-live drives. The recreate loop +was refactored into a pure, seam-testable helper: -## Phase B — fix (harden the existing mechanism, no parallel one) +```go +func recreateDriveBackedApps(stacks []bootStack, presentStable map[string]bool, + recreate func(bootStack), syncFB func()) (recreated, skipped int) +``` -1. **`ensureLocalAPI`** (`internal/bootstrap/bootstrap.go`): `MaybeIngest` now calls it on the - already-configured path — when `cfg.LocalAPI.Endpoint` is empty it merges `local_api` from - bootstrap.json into the existing controller.yaml in place (no hub re-pull, existing config preserved), - idempotent + fail-safe. -2. **`driveBindLive` + `pollLiveBinds`** (`internal/web/intermediary.go`): `processGuestBootChange` now - gates on the **real live in-guest bind** — `driveBindLive` checks whether `/mnt/felhom-drives/` - is an actual mountpoint in the controller's own `/mnt` (rslave) `/proc/self/mountinfo` (true only once - the agent's bind propagated, exactly when docker can recreate the app); `pollLiveBinds` waits for it - (bounded ~120s, poll 2s) before recreating via the normal pipeline (`compose down`→`up -d`). - `shouldRecreateOnBoot` is unchanged and state-independent → stuck-`Exited` create-time-failure apps are - included. Drives that never go live in the window are left to the gate. -3. **Periodic retry** (`driveGateLoop`): `processGuestBootChange` now runs on every periodic tick too — - idempotent (boot-id gated) — so a momentarily-unreachable agent right after a reboot no longer - permanently strands recovery. +`syncFB` is invoked exactly once, AFTER all recreates — the FileBrowser sync can never run before the +drive-backed apps are back. -Both reboot paths share this code, the same agent dependency, and the same boot-race — so both were -broken by the regression and both are fixed here (see the regression analysis + host-reboot drill below). +## Tests (non-hollow, seam = the FB sync) -## Phase C — tests (non-hollow, pre-fix companions, red-proofed) +- `TestRecreateDriveBackedApps_SyncsFileBrowserAfterRecreate` — records the call sequence; asserts + `syncFB` runs exactly once and strictly AFTER every `recreate`. Red-proofed (stubbing out the + `syncFB()` call makes it fail). +- `TestRecreateDriveBackedApps_SyncsEvenWithNoRecreate` — FileBrowser still converges when nothing + needed recreating (e.g. binds already present). -- `internal/web/intermediary_test.go`: `pollLiveBinds` waits through the rebind window then reports live - (recreate fires); a never-live drive stays absent (no spurious recreate); an explicit companion that a - **single early sample misses** the not-yet-live bind. Red-proofed against a no-wait single-sample. -- `internal/bootstrap/bootstrap_test.go`: `ensureLocalAPI` merges `local_api` into an already-configured - controller.yaml that lacks it (companion: pre-fix `MaybeIngest` left `LocalAPI.Endpoint` empty — - red-proofed) and no-ops when already present. Full controller suite green; `go vet` clean. +`go build`, `go vet`, `go test ./...` all green on the build server (192.168.0.180). -## Phase D — live acceptance (the real gate) +## Live acceptance — real host reboot ×2 on felhom-pve -Built + deployed `felhom-controller:0.71.0` to guest 9201 (the redeploy itself validated `ensureLocalAPI`: -the container recreate reset controller.yaml, the code re-merged `local_api`, `/api/disks` → 200). Then -**two** `pct reboot 9201` cycles (zero manual intervention): +Both reboots: `processGuestBootChange` fired on the new boot-id, confirmed the live binds, recreated +every drive-backed app, **then** ran the FileBrowser sync. Captured controller logs: -| reboot | boot-id | gate log | result | -|---|---|---|---| -| #1 | `…7348791` | "waiting (≤2m0s) for live drive bind(s) … → live bind confirmed — recreating" ×8 | all 8 `Up` | -| #2 | `…7367438` | same full sequence ×8 | all 8 `Up` | +``` +[gate] boot 1781625729-1612: live bind confirmed — recreating drive-backed app … onto /mnt/felhom-drives/felhom-flash (×8 apps) +[gate] boot 1781625729-1612: re-syncing FileBrowser mounts against the live binds +[web] FileBrowser mounts synced — 3 storage path(s), config updated +``` -Both recovered all 8 drive-backed apps automatically. (komga reports its container healthcheck -"unhealthy" but is up and serving — a pre-existing, unrelated issue.) +(reboot #2 identical on boot-id `1781625955-1516`.) Post-reboot FileBrowser binds all three drives +non-stale — `felhom-usb`, `felhom-flash`, `felhom-data` → `/srv/felhom-*` — and the underlying agent +tolerated a `/dev/sdb`↔`/dev/sdc` reshuffle by mounting each drive by UUID (agent v0.37.0, Task A). +The earlier first-boot `Failed to recreate FileBrowser` line (drives not yet live) is the exact pre-fix +symptom; the boot-recreate path now recovers it. -**HOST reboot (re-drilled after the follow-up challenge — the guest reboots only prove the guest path):** -two `systemctl reboot` of felhom-pve itself. - -| host reboot | host btime | boot-id | result | -|---|---|---|---| -| #1 | `1781545822`→`1781620637` | `1781620637-1443` — recreate ×8 (apps were `state=stopped`) | all 8 `Up` | -| #2 | `1781620637`→`1781620928` | `1781620928-1516` — recreate ×8 | all 8 `Up` | - -Both recovered automatically, zero manual starts. Two findings: (a) the **host-btime prefix advances**, so -the host-reboot path triggers `processGuestBootChange` (the persisted `LastGuestBootID` now tracks each -boot — it was frozen at the first-boot value before the fix); (b) the **boot-race manifests on host -reboots too** (the gate recreated `state=stopped` apps), so the host path is **not** immune — it has the -same race and the same agent-path dependency as the guest path. - -## Regression analysis — "how did the earlier (v0.68) host-reboot sweep pass?" - -It genuinely passed and genuinely exercised `processGuestBootChange`: -`documentation/audits/storage-lifecycle-acceptance-2026-06-15.md` records the host reboot **surfacing a -real bug** (5 apps stayed exited because the recreate filtered on container state, fixed in v0.68.1). If -`agentClient()` had been failing then, `processGuestBootChange` would have bailed and that bug could never -have appeared. **So the agent path worked at v0.68 and regressed afterward** — "boot-id determinism was -never exercising" is false; it was. - -The regression: `controller.yaml` is reset to the golden's "configured-but-no-`local_api`" baseline on -every container recreate (each deploy), and the old `MaybeIngest` returned immediately on "already -configured" → `local_api` was never re-merged → `agentClient()` → "agent not configured" → the whole -drive gate + boot recovery silently died, on **both** reboot paths. (My v0.70.0 config-apply round-trip is -**exonerated**: `GET /api/config` returns the file verbatim — `router.go:1150`, no redaction — so it -preserved `local_api`.) `ensureLocalAPI` re-merges `local_api` on every startup, closing the regression -permanently. - -## Follow-up — FileBrowser was NOT recovered (host-reboot recovery was incomplete) - -After the host-reboot drills, **FileBrowser was found down** (`state=created, exit=128, RestartCount=0`, -`mkdir /mnt/felhom-drives/felhom-usb/userdata: permission denied`). My "all recovered" claim above was -**incomplete**: it only checked the 8 app stacks, which are all on **felhom-flash**. Two distinct gaps: - -- **(A) Agent-side — felhom-usb did not re-mount after the host reboot.** The host reboot swapped the USB - device letters (`sdb↔sdc`); felhom-flash re-mounted (now `sdc1`), but **felhom-usb (`sdb1`, - `da9e7089`) did not** — its systemd mount unit (`mnt-felhom\x2dusb.mount`) existed and the by-uuid - symlink resolved, yet the unit was **not active**, so the agent reported the (present) drive as - `enrolled drive not present (durable-id absent)` every reconcile and never bound it. So the - host-reboot recovery covered felhom-flash apps but left **felhom-usb entirely unmounted** — a - **felhom-agent** bug (it must re-activate/retry the mount once the USB enumerates after a host reboot, - not give up on "absent"). **Out of the controller's scope.** -- **(B) Controller-side — FileBrowser is not covered by the boot-recovery.** FileBrowser binds all three - drives' `userdata` but is **base-infra (no `HDD_PATH`)**, so `processGuestBootChange`/`shouldRecreateOnBoot` - skip it. Its recovery relies on `SyncFileBrowserMounts`, which runs once at controller startup — racing - the bind exactly like the app boot-race — and is not retried after the binds go live. So FileBrowser is - stranded by a guest/host reboot independently of (A). **Recommended fix (separate, small): have - `processGuestBootChange` call `SyncFileBrowserMounts` after the live-bind poll confirms the binds, so - FileBrowser converges with the apps.** Not done here (it's a code change + version bump; flagged for a - follow-up slice). Note it would only fully help once (A) is fixed — on this reboot felhom-usb was absent, - so FB could not bind it regardless. - -**Recovered live (non-destructive):** started the felhom-usb mount unit (`systemctl start /mnt/felhom-usb`) -→ agent bound it into the guest → restarted the controller → `SyncFileBrowserMounts` synced 3 storage -paths → **FileBrowser `Up (healthy)`**. felhom-usb's real `userdata` (`drwxrwsr-x root:1000`) was intact -all along — only hidden behind the unmounted placeholder. - -## Notes - -- The Komga healthcheck quirk is pre-existing and out of scope. -- The `local_api`-merge gap affected the **whole drive gate** on this golden, not just boot recovery — - `ensureLocalAPI` restores the agent path generally. -- **Trust caveat:** the reboot drills prove the **felhom-flash app** recovery path (guest ×2, host ×2). - They do **not** yet prove a clean felhom-usb host-reboot remount (gap A) or FileBrowser auto-recovery - (gap B) — both are now documented and recommended for follow-up. +Demo dashboard has no password set → controller API is open on the in-guest path; no secrets committed. diff --git a/controller/README.md b/controller/README.md index d486b80..9b54ff4 100644 --- a/controller/README.md +++ b/controller/README.md @@ -663,6 +663,15 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an > path — merging `local_api` from `bootstrap.json` into an existing controller.yaml that lacks it (seeded > before `local_api` existed) — because without it `agentClient()` returns "agent not configured" and the > entire gate + boot recovery silently die. +> **v0.72.0 — FileBrowser convergence on boot-recreate:** FileBrowser is base-infra (it binds each +> drive's `userdata` but has no `HDD_PATH`, so it is NOT in the drive-backed recreate set) — after a +> host reboot its mounts could be stale (the early first-boot bring-up bound them before the drives went +> live). `processGuestBootChange` now, **after** `pollLiveBinds` confirms the binds and the apps are +> recreated, triggers `go s.SyncFileBrowserMounts()` so FileBrowser converges against the now-live drives. +> The sync runs unconditionally (FileBrowser reflects the current bind state even if no app needed +> recreating). The recreate loop is a pure `recreateDriveBackedApps(stacks, present, recreate, syncFB)` +> that calls `syncFB` exactly once, after every recreate. Live-accepted over two real `felhom-pve` +> reboots (FileBrowser non-stale, all drive-backed apps recovered, agent tolerated a `/dev/sdX` swap by UUID). > > **⚠️ Rebuilt on the agent-delegated disk model (v0.43.0), made ROLE-AWARE in v0.44.0, UX-polished in > v0.45.0.** After the 8C