diff --git a/documentation/audits/DIAGNOSE-drive-bind-after-reprovision-2026-06-23.md b/documentation/audits/DIAGNOSE-drive-bind-after-reprovision-2026-06-23.md new file mode 100644 index 0000000..5f953f3 --- /dev/null +++ b/documentation/audits/DIAGNOSE-drive-bind-after-reprovision-2026-06-23.md @@ -0,0 +1,154 @@ +# DIAGNOSE — re-enrolled external drives not binding into guest 9201 (after re-provision) + +**Date:** 2026-06-23 · **Class:** diagnostic (read-only; no fix, no mutation) · **Guest:** 9201 (demo-felhom) +**Versions live:** agent v0.40.0, controller v0.77.0 (`66d8462`). +**Verdict:** controller code bug in ONE of the three drive-enroll paths (`handleStorageRegister`) — it +registers the **raw** `/mnt/` path instead of the **stable** `/mnt/felhom-drives/` path the +agent actually binds the drive at. NOT the v0.40.0 `mp1` change. NOT stale agent state. + +--- + +## 1. Intended path model (from source) + +Post the intermediary-mount migration (`SPIKE-intermediary-mount-2026-06-15.md`), an external drive is +visible **in the guest ONLY at its STABLE path** `/mnt/felhom-drives/`: + +- **Agent** (`felhom-agent/internal/localapi/intermediary.go`): `StableParentDir = "/mnt/felhom-drives"`. + A single permanent parent bind (`mp8 /mnt/felhom-drives`) is set into the guest once; the agent then + binds each drive's `felhom-data` namespace **under** it host-side + (`mount --bind /mnt//felhom-data /mnt/felhom-drives/`), and host→guest mount propagation + carries it live. `StablePathForRaw("/mnt/")` → `/mnt/felhom-drives/` (intermediary.go:70). +- The agent's `/disks` list reports **both** paths per drive (`disks.go`): `mount_path` = the RAW host + PVE mount `/mnt/` (disks.go:113), and `guest_path` = the STABLE in-guest path + (disks.go:146, set via `StablePathForRaw`), plus `bound_under_parent` (disks.go:150). +- **Controller** (`felhom-controller/internal/web/intermediary.go:73-98`, verbatim): + *"the REGISTERED storage path + every app's HDD_PATH + FileBrowser source = the STABLE path; the AGENT + still operates on the RAW /mnt/ host mount, so controller→agent `where` is mapped back to raw via + `agentWhere()`."* `stablePathForName(name)` → `/mnt/felhom-drives/` (intermediary.go:87). + +**So the intended registered path is the STABLE `/mnt/felhom-drives/`.** The drive-absent gate +`planDriveGates` only manages paths under `/mnt/felhom-drives/` (intermediary.go:220) — a path registered +any other way is invisible to the gate. + +--- + +## 2. Evidence (live, read-only — host / guest / container / state / logs) + +| # | Where | Observation | +|---|-------|-------------| +| E1 | HOST `pct config 9201` | `mp0 /var/lib/docker`(200G), `mp1 /mnt/sys_drive`(50G), **`mp8 /mnt/felhom-drives`**, `mp9 bootstrap`. **No per-drive mpN** — correct (intermediary model). | +| E2 | HOST `findmnt` | `/mnt/felhom-drives/felhom-flash` ← `/dev/sdc1[/felhom-data]`; `/mnt/felhom-drives/felhom-usb` ← `/dev/sdb1[/felhom-data]` (each twice — the known double-bind). Raw `/mnt/felhom-flash`←sdc1, `/mnt/felhom-usb`←sdb1 also present (host-level, dated Jun 14-15, survived the destroy). | +| E3 | HOST `ls …/felhom-data` | Both drives' `felhom-data` namespace exists + is guest-owned (100000:100000). Bind source present — **H4 refuted**. | +| E4 | GUEST `findmnt` | `/mnt/felhom-drives/felhom-flash` ← `sdc1[/felhom-data]` ✅ and `/mnt/felhom-drives/felhom-usb` ← `sdb1[/felhom-data]` ✅ — the drives **are live in the guest at the stable path**. `/mnt/felhom-flash`, `/mnt/felhom-usb` are **plain dirs on rootfs disk-0** (created Jun 23 14:52, this boot). | +| E5 | CONTROLLER container `findmnt` | `/mnt/felhom-drives/felhom-flash` → `sdc1[/felhom-data]` ✅; `/mnt/felhom-drives/felhom-usb` → `sdb1[/felhom-data]` ✅. `/mnt/felhom-flash` → **notmount**; `/mnt/felhom-usb` → **notmount**. The real drives ARE visible in-container — at the stable path. | +| E6 | CONTROLLER `settings.json` | `storage_paths` = **`/mnt/felhom-flash`** (added `2026-06-23T14:52:10Z`) + **`/mnt/felhom-usb`** (`14:52:16Z`), labels `Tárhely (felhom-flash/usb)` (auto-inferred). **The registry holds the RAW paths.** | +| E7 | CONTROLLER selftest | `[WARN] Storage paths: 0 connected, 2 disconnected`; `[PASS] System data path: /mnt/sys_drive`. | +| E8 | AGENT journal (this boot) | Repeated `guest-attach: drive bound under shared parent (normalized to one bind, live) … where=/mnt/felhom-usb stable=/mnt/felhom-drives/felhom-usb prior_binds=2` and `…where=/mnt/felhom-flash stable=/mnt/felhom-drives/felhom-flash`. The agent binds correctly at the **stable** path every reconcile tick. | +| E9 | AGENT `guest-binds.json` / `drive-intents.json` | (Jun 22) 9201 → {81a26531, da9e7089 enrolled; ce9d ejected}. `ReassertGuestBinds` re-bound the enrolled drives at the stable path; intent-aware skip of the ejected `ce9d`. Agent state is **correct** — H3 refuted as a *cause*. | + +**The chain:** drive present + host-bound + parent-propagated → live in guest AND container at +`/mnt/felhom-drives/` (E2/E4/E5/E8). The controller registered `/mnt/` (E6) → it watches an +empty placeholder dir on rootfs disk-0 → "0 connected, 2 disconnected" / "Rendszermeghajtón" / 31.2 GB. + +--- + +## 3. Hypothesis verdicts + +- **H1 — intermediary bind never placed:** **REFUTED.** E2/E4/E5/E8 show the binds placed and live all + the way into the container. +- **H2 — path-model mismatch (controller watches `/mnt/`, drive lives at + `/mnt/felhom-drives/`):** **CONFIRMED — primary cause.** E6 (raw in registry) vs E4/E5 (drive at + stable). Source: `handleStorageRegister` registers raw (see §4). +- **H3 — stale agent drive-state vs fresh guest:** **REFUTED as cause.** The Jun-22 agent state is stale + but *correct* — it drove the right stable binds (E8/E9). It is not what mis-registered the path. +- **H4 — felhom-data namespace absent → placeholder:** **REFUTED.** Namespaces present (E3). (The + rootfs placeholder dirs DO exist, but the controller created them itself — see §4 — not the guest-hook + for an absent bind source.) +- **H5 — "Újraindítás most" is a no-op for activation:** **CONFIRMED as a downstream symptom.** The + button calls agent `GuestReboot` (`agentapi/client.go:366`). A reboot correctly re-binds the drive at + the stable path, but the controller's *registered* path is the raw rootfs placeholder, so + `pendingActivationDrives` (storage_handlers.go:172) keeps flagging it (`attached[/mnt/felhom-usb]` raw + key happens to match the raw registered `sp.Path`, and `system.IsMountPoint("/mnt/felhom-usb")` is + false because it's a same-device rootfs dir) → the "2 meghajtó aktiválásra vár" banner persists no + matter how many reboots. A reboot cannot fix a wrong registered path. +- **H6 — propagation broken by an unprivileged edge:** **REFUTED.** Propagation works (E5: the stable + path is live inside the container). + +**Single primary cause: H2.** H5 and the badge are downstream consequences of it. + +--- + +## 4. Root cause in source + +`felhom-controller/internal/web/storage_handlers.go`: + +- `runStorageInit` (line 127-128) and `runStorageAttach` (line 160-161) both do + `stable := stablePathForName(path.Base(where))` → `registerStoragePath(stable, …)` — **correct**. +- **`handleStorageRegister` (line 667-694)** — the *"register an already-mounted, unregistered drive"* + action — does `s.registerStoragePath(req.Where, …)` with `req.Where` = the **raw** `/mnt/` + **verbatim** (line 682). It is **missing the `stablePathForName` mapping** its two sibling enroll paths + have. `attachIntoGuest(req.Where)` still (correctly) hands the agent the raw path, so the agent binds + the drive at the stable path — but the controller registers and watches the raw one. +- `registerStoragePath` then calls `appbackup.EnsureUserdataSkeleton(where=/mnt/felhom-flash)` + (storage_handlers.go:230), which `mkdir`s the path → **this is what created the empty + `/mnt/felhom-flash` / `/mnt/felhom-usb` placeholder dirs on the guest rootfs** (E4, Jun 23 14:52). + +Why this path ran: after a destroy + re-provision the HOST drive mounts survive (E2), so on the fresh +controller (registry wiped) the drives appear *mounted-but-unregistered* → the natural UI action is +"Regisztrálás" → `POST /api/storage/register` → `handleStorageRegister` → the raw-path bug. The +auto-inferred label in E6 confirms this handler (not the user-labelled free-form add at `handlers.go:1310`). + +--- + +## 5. The five required answers + +1. **Root cause (one sentence):** The controller's `handleStorageRegister` enroll path registers the + drive's **raw** `/mnt/` mount path instead of the **stable** `/mnt/felhom-drives/` path the + agent binds it at, so the controller watches an empty placeholder dir on the guest rootfs while the + real drive is live (and correctly bound) at the stable path. Evidence chain: E2/E4/E5/E8 (drive live at + stable, host→guest→container) vs E6 (raw path in registry) → source storage_handlers.go:682. + +2. **Is v0.40.0 `mp1` implicated?** **No.** `/mnt/sys_drive` = `disk-2`, `[PASS]` (E1/E7); the + drive-enroll registration path is wholly independent of the golden's `mp1` volume and of `bringup.go`. + The bug lives in controller v0.77.0 `handleStorageRegister`, unchanged by the agent v0.40.0 work. + +3. **Re-provision-specific or fresh-enroll too?** It is a **genuine code bug in the + "register-already-mounted-drive" path**, not a stale-state artifact. It is *surfaced* by the + re-provision because surviving host mounts make "Regisztrálás" the natural action. A **first-time** + enroll of a **new** drive uses "Inicializálás" (`runStorageInit`) or "Csatolás" (`runStorageAttach`), + which map to the stable path correctly and do **not** hit it. So: any use of the *register existing + mount* action is broken (Day-0 relevant for that flow), most reliably triggered by re-provision. + +4. **"Rendszermeghajtón" label:** **Purely downstream**, not an independent bug. `CheckBackupDestination` + / the storage view use `isSameBlockDevice(path, "/")` (`system/mounts_linux.go:173,225`) — `stat()` on + `/mnt/felhom-flash` (an empty dir on rootfs) yields the same `st_dev` as `/` → `SystemDrive=true`; and + `GetDiskUsage` returns the 32 GB rootfs (≈31.2 GB). The label faithfully reports that the *registered + path* is on the system drive — because it is the wrong path. Register the stable path and `stat` + resolves to `sdc1`/`sdb1` (different device) → badge clears. + +5. **Recommended fix direction (NOT implemented):** In `handleStorageRegister` + (storage_handlers.go:667-694), map the incoming raw path to the stable path before registering — + `stable := stablePathForName(path.Base(req.Where)); s.registerStoragePath(stable, req.Label, …)` — + exactly as `runStorageInit`/`runStorageAttach` do; keep `attachIntoGuest(req.Where)` on the RAW path + (the agent operates on raw). Consider the same guard for the free-form add at `handlers.go:1310` when + the path is a `/mnt/` drive. **Remediation for 9201** (separate from the code fix): deregister + the two raw `storage_paths`, re-register at the stable paths (or re-enroll once the fix ships), and + remove the stray `/mnt/felhom-flash` `/mnt/felhom-usb` skeleton dirs on the rootfs. + **What NOT to do:** do **not** "fix" by auto-creating `felhom-data` under `/mnt/` or by making + the agent bind the drive at `/mnt/` — that breaks the intermediary model (live hot-swap + + confinement). The drive is already correctly at `/mnt/felhom-drives/`; the only thing wrong is + which path the controller wrote down. This is a **normal controller change** (Docker-only, + build+deploy) — **not** a risky/supervised agent/golden task. + +--- + +## 6. Secondary observations (not acted on) + +- **Double-bind churn:** the agent normalizes `prior_binds=2 → 1` on `/mnt/felhom-drives/` every + ~20s tick (E8) — the pre-existing double-bind under the shared parent (each drive appears twice in + `/proc/mounts`). Cosmetic here; flagged in prior memory as the v0.36.x peer-group area. Out of scope. +- **`pendingActivationDrives` raw/stable coupling** (storage_handlers.go:188-199): it keys `attached` + by the agent's raw `MountPath` and compares against the registered `sp.Path`. With the fix (stable + registration) this comparison would never match a raw key, so the "pending activation" detection for + stable paths should be revisited to key off `guest_path` / `bound_under_parent` (as `planDriveGates` + already does). Worth checking when fixing #5 so the activation banner reflects the stable model.