Files
felhom.eu/documentation/audits/DIAGNOSE-drive-bind-after-reprovision-2026-06-23.md
T
admin e8a5bdc492 docs(audit): DIAGNOSE re-enrolled drives not binding after 9201 re-provision
Root cause: controller handleStorageRegister registers the raw /mnt/<name> path
instead of the stable /mnt/felhom-drives/<name> path the agent binds the drive at.
Drives ARE live in guest+container at the stable path; controller watches an empty
rootfs placeholder. Not v0.40.0 mp1; not stale agent state. Read-only diagnosis,
no fix applied. Recommended fix direction documented.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
2026-06-23 17:15:30 +02:00

12 KiB

DIAGNOSE — re-enrolled external drives not binding into guest 9201 (after re-provision)

Date: 2026-06-23 · Class: diagnostic (read-only; no fix, no mutation) · Guest: 9201 (demo-felhom) Versions live: agent v0.40.0, controller v0.77.0 (66d8462). Verdict: controller code bug in ONE of the three drive-enroll paths (handleStorageRegister) — it registers the raw /mnt/<name> path instead of the stable /mnt/felhom-drives/<name> path the agent actually binds the drive at. NOT the v0.40.0 mp1 change. NOT stale agent state.


1. Intended path model (from source)

Post the intermediary-mount migration (SPIKE-intermediary-mount-2026-06-15.md), an external drive is visible in the guest ONLY at its STABLE path /mnt/felhom-drives/<name>:

  • Agent (felhom-agent/internal/localapi/intermediary.go): StableParentDir = "/mnt/felhom-drives". A single permanent parent bind (mp8 /mnt/felhom-drives) is set into the guest once; the agent then binds each drive's felhom-data namespace under it host-side (mount --bind /mnt/<name>/felhom-data /mnt/felhom-drives/<name>), and host→guest mount propagation carries it live. StablePathForRaw("/mnt/<name>")/mnt/felhom-drives/<name> (intermediary.go:70).
  • The agent's /disks list reports both paths per drive (disks.go): mount_path = the RAW host PVE mount /mnt/<name> (disks.go:113), and guest_path = the STABLE in-guest path (disks.go:146, set via StablePathForRaw), plus bound_under_parent (disks.go:150).
  • Controller (felhom-controller/internal/web/intermediary.go:73-98, verbatim): "the REGISTERED storage path + every app's HDD_PATH + FileBrowser source = the STABLE path; the AGENT still operates on the RAW /mnt/ host mount, so controller→agent where is mapped back to raw via agentWhere()." stablePathForName(name)/mnt/felhom-drives/<name> (intermediary.go:87).

So the intended registered path is the STABLE /mnt/felhom-drives/<name>. The drive-absent gate planDriveGates only manages paths under /mnt/felhom-drives/ (intermediary.go:220) — a path registered any other way is invisible to the gate.


2. Evidence (live, read-only — host / guest / container / state / logs)

# Where Observation
E1 HOST pct config 9201 mp0 /var/lib/docker(200G), mp1 /mnt/sys_drive(50G), mp8 /mnt/felhom-drives, mp9 bootstrap. No per-drive mpN — correct (intermediary model).
E2 HOST findmnt /mnt/felhom-drives/felhom-flash/dev/sdc1[/felhom-data]; /mnt/felhom-drives/felhom-usb/dev/sdb1[/felhom-data] (each twice — the known double-bind). Raw /mnt/felhom-flash←sdc1, /mnt/felhom-usb←sdb1 also present (host-level, dated Jun 14-15, survived the destroy).
E3 HOST ls …/felhom-data Both drives' felhom-data namespace exists + is guest-owned (100000:100000). Bind source present — H4 refuted.
E4 GUEST findmnt /mnt/felhom-drives/felhom-flashsdc1[/felhom-data] and /mnt/felhom-drives/felhom-usbsdb1[/felhom-data] — the drives are live in the guest at the stable path. /mnt/felhom-flash, /mnt/felhom-usb are plain dirs on rootfs disk-0 (created Jun 23 14:52, this boot).
E5 CONTROLLER container findmnt /mnt/felhom-drives/felhom-flashsdc1[/felhom-data] ; /mnt/felhom-drives/felhom-usbsdb1[/felhom-data] . /mnt/felhom-flashnotmount; /mnt/felhom-usbnotmount. The real drives ARE visible in-container — at the stable path.
E6 CONTROLLER settings.json storage_paths = /mnt/felhom-flash (added 2026-06-23T14:52:10Z) + /mnt/felhom-usb (14:52:16Z), labels Tárhely (felhom-flash/usb) (auto-inferred). The registry holds the RAW paths.
E7 CONTROLLER selftest [WARN] Storage paths: 0 connected, 2 disconnected; [PASS] System data path: /mnt/sys_drive.
E8 AGENT journal (this boot) Repeated guest-attach: drive bound under shared parent (normalized to one bind, live) … where=/mnt/felhom-usb stable=/mnt/felhom-drives/felhom-usb prior_binds=2 and …where=/mnt/felhom-flash stable=/mnt/felhom-drives/felhom-flash. The agent binds correctly at the stable path every reconcile tick.
E9 AGENT guest-binds.json / drive-intents.json (Jun 22) 9201 → {81a26531, da9e7089 enrolled; ce9d ejected}. ReassertGuestBinds re-bound the enrolled drives at the stable path; intent-aware skip of the ejected ce9d. Agent state is correct — H3 refuted as a cause.

The chain: drive present + host-bound + parent-propagated → live in guest AND container at /mnt/felhom-drives/<name> (E2/E4/E5/E8). The controller registered /mnt/<name> (E6) → it watches an empty placeholder dir on rootfs disk-0 → "0 connected, 2 disconnected" / "Rendszermeghajtón" / 31.2 GB.


3. Hypothesis verdicts

  • H1 — intermediary bind never placed: REFUTED. E2/E4/E5/E8 show the binds placed and live all the way into the container.
  • H2 — path-model mismatch (controller watches /mnt/<name>, drive lives at /mnt/felhom-drives/<name>): CONFIRMED — primary cause. E6 (raw in registry) vs E4/E5 (drive at stable). Source: handleStorageRegister registers raw (see §4).
  • H3 — stale agent drive-state vs fresh guest: REFUTED as cause. The Jun-22 agent state is stale but correct — it drove the right stable binds (E8/E9). It is not what mis-registered the path.
  • H4 — felhom-data namespace absent → placeholder: REFUTED. Namespaces present (E3). (The rootfs placeholder dirs DO exist, but the controller created them itself — see §4 — not the guest-hook for an absent bind source.)
  • H5 — "Újraindítás most" is a no-op for activation: CONFIRMED as a downstream symptom. The button calls agent GuestReboot (agentapi/client.go:366). A reboot correctly re-binds the drive at the stable path, but the controller's registered path is the raw rootfs placeholder, so pendingActivationDrives (storage_handlers.go:172) keeps flagging it (attached[/mnt/felhom-usb] raw key happens to match the raw registered sp.Path, and system.IsMountPoint("/mnt/felhom-usb") is false because it's a same-device rootfs dir) → the "2 meghajtó aktiválásra vár" banner persists no matter how many reboots. A reboot cannot fix a wrong registered path.
  • H6 — propagation broken by an unprivileged edge: REFUTED. Propagation works (E5: the stable path is live inside the container).

Single primary cause: H2. H5 and the badge are downstream consequences of it.


4. Root cause in source

felhom-controller/internal/web/storage_handlers.go:

  • runStorageInit (line 127-128) and runStorageAttach (line 160-161) both do stable := stablePathForName(path.Base(where))registerStoragePath(stable, …)correct.
  • handleStorageRegister (line 667-694) — the "register an already-mounted, unregistered drive" action — does s.registerStoragePath(req.Where, …) with req.Where = the raw /mnt/<name> verbatim (line 682). It is missing the stablePathForName mapping its two sibling enroll paths have. attachIntoGuest(req.Where) still (correctly) hands the agent the raw path, so the agent binds the drive at the stable path — but the controller registers and watches the raw one.
  • registerStoragePath then calls appbackup.EnsureUserdataSkeleton(where=/mnt/felhom-flash) (storage_handlers.go:230), which mkdirs the path → this is what created the empty /mnt/felhom-flash / /mnt/felhom-usb placeholder dirs on the guest rootfs (E4, Jun 23 14:52).

Why this path ran: after a destroy + re-provision the HOST drive mounts survive (E2), so on the fresh controller (registry wiped) the drives appear mounted-but-unregistered → the natural UI action is "Regisztrálás" → POST /api/storage/registerhandleStorageRegister → the raw-path bug. The auto-inferred label in E6 confirms this handler (not the user-labelled free-form add at handlers.go:1310).


5. The five required answers

  1. Root cause (one sentence): The controller's handleStorageRegister enroll path registers the drive's raw /mnt/<name> mount path instead of the stable /mnt/felhom-drives/<name> path the agent binds it at, so the controller watches an empty placeholder dir on the guest rootfs while the real drive is live (and correctly bound) at the stable path. Evidence chain: E2/E4/E5/E8 (drive live at stable, host→guest→container) vs E6 (raw path in registry) → source storage_handlers.go:682.

  2. Is v0.40.0 mp1 implicated? No. /mnt/sys_drive = disk-2, [PASS] (E1/E7); the drive-enroll registration path is wholly independent of the golden's mp1 volume and of bringup.go. The bug lives in controller v0.77.0 handleStorageRegister, unchanged by the agent v0.40.0 work.

  3. Re-provision-specific or fresh-enroll too? It is a genuine code bug in the "register-already-mounted-drive" path, not a stale-state artifact. It is surfaced by the re-provision because surviving host mounts make "Regisztrálás" the natural action. A first-time enroll of a new drive uses "Inicializálás" (runStorageInit) or "Csatolás" (runStorageAttach), which map to the stable path correctly and do not hit it. So: any use of the register existing mount action is broken (Day-0 relevant for that flow), most reliably triggered by re-provision.

  4. "Rendszermeghajtón" label: Purely downstream, not an independent bug. CheckBackupDestination / the storage view use isSameBlockDevice(path, "/") (system/mounts_linux.go:173,225) — stat() on /mnt/felhom-flash (an empty dir on rootfs) yields the same st_dev as /SystemDrive=true; and GetDiskUsage returns the 32 GB rootfs (≈31.2 GB). The label faithfully reports that the registered path is on the system drive — because it is the wrong path. Register the stable path and stat resolves to sdc1/sdb1 (different device) → badge clears.

  5. Recommended fix direction (NOT implemented): In handleStorageRegister (storage_handlers.go:667-694), map the incoming raw path to the stable path before registering — stable := stablePathForName(path.Base(req.Where)); s.registerStoragePath(stable, req.Label, …) — exactly as runStorageInit/runStorageAttach do; keep attachIntoGuest(req.Where) on the RAW path (the agent operates on raw). Consider the same guard for the free-form add at handlers.go:1310 when the path is a /mnt/<name> drive. Remediation for 9201 (separate from the code fix): deregister the two raw storage_paths, re-register at the stable paths (or re-enroll once the fix ships), and remove the stray /mnt/felhom-flash /mnt/felhom-usb skeleton dirs on the rootfs. What NOT to do: do not "fix" by auto-creating felhom-data under /mnt/<name> or by making the agent bind the drive at /mnt/<name> — that breaks the intermediary model (live hot-swap + confinement). The drive is already correctly at /mnt/felhom-drives/<name>; the only thing wrong is which path the controller wrote down. This is a normal controller change (Docker-only, build+deploy) — not a risky/supervised agent/golden task.


6. Secondary observations (not acted on)

  • Double-bind churn: the agent normalizes prior_binds=2 → 1 on /mnt/felhom-drives/<name> every ~20s tick (E8) — the pre-existing double-bind under the shared parent (each drive appears twice in /proc/mounts). Cosmetic here; flagged in prior memory as the v0.36.x peer-group area. Out of scope.
  • pendingActivationDrives raw/stable coupling (storage_handlers.go:188-199): it keys attached by the agent's raw MountPath and compares against the registered sp.Path. With the fix (stable registration) this comparison would never match a raw key, so the "pending activation" detection for stable paths should be revisited to key off guest_path / bound_under_parent (as planDriveGates already does). Worth checking when fixing #5 so the activation banner reflects the stable model.