# SPIKE — 2nd external drive "drops" the 1st (multi-drive mutual exclusion / flapping) - **Date:** 2026-06-29 - **Class:** Diagnostic / read-only. **No fix.** No deliberate reproduce was needed — the symptom is **continuously live** in the controller gate logs, so the host-vs-guest `mountinfo` diff was read directly from the running state (an attach would only re-stop apps, adding no evidence). - **Host:** felhom-pve (`demo-felhom`, 192.168.0.162); guest LXC 9201 (init pid 2137736). - **Live versions:** agent **0.43.0** (NON-ROOT `felhom-agent` since 2026-06-28), controller **0.89.0**. - **Verdict:** **Detection bug (matrix row 3), NOT a propagation fault and NOT a controller-gate bug.** The guest demonstrably keeps **both** drive submounts; the agent mis-reports `bound_under_parent=false` for **every** drive because the non-root agent cannot resolve the guest init PID (`lxc-info` is denied / unusable under the `felhom-agent` user). The controller gate is the **messenger** — it correctly stops apps for a drive its report says is absent. Root cause is the **2026-06-28 root→non-root agent migration** (same regression family as the leaf-pin + PBS-key issues): `lxc-info` was never added to the agent sudoers allowlist. --- ## 1. The question (§3) — answered > Does the guest actually lose drive A's submount (propagation fault), or does the guest keep it > while the agent mis-reports `BoundUnderParent=false` (detection bug)? **The guest keeps BOTH submounts; the agent mis-reports.** Propagation is intact (the guest's parent bind is `master:1`, a live slave of the host's `shared:1` parent, and both drives are visible in the guest namespace right now). The fault is purely in the agent's *guest-sees* detection. --- ## 2. Raw evidence ### A1/A2 — host parent + submounts (both drives bound, propagation `shared`) ``` findmnt /mnt/felhom-drives: /mnt/felhom-drives /dev/mapper/pve-root[/mnt/felhom-drives] shared ├─/mnt/felhom-drives/felhom-flash /dev/sdc1[/felhom-data] shared └─/mnt/felhom-drives/felhom-usb /dev/sdb1[/felhom-data] shared /proc/self/mountinfo (host): 137 33 252:1 /mnt/felhom-drives /mnt/felhom-drives shared:1 99 137 8:33 /felhom-data /mnt/felhom-drives/felhom-flash shared:51 ← child of parent 137 100 33 8:33 /felhom-data /mnt/felhom-drives/felhom-flash shared:51 ← ALSO child of root 33 (doubling) 109 137 8:17 /felhom-data /mnt/felhom-drives/felhom-usb shared:49 110 33 8:17 /felhom-data /mnt/felhom-drives/felhom-usb shared:49 ``` ### A3 — GUEST view (the `BoundUnderParent` truth) — **both drives present** ``` guest_pid=2137736 571 324 252:1 /mnt/felhom-drives /mnt/felhom-drives shared:482 master:1 ← slave of host group 1 (propagation LIVE) 108 571 8:33 /felhom-data /mnt/felhom-drives/felhom-flash shared:195 master:51 344 571 8:17 /felhom-data /mnt/felhom-drives/felhom-usb shared:210 master:49 (+ duplicate entries 572/111/345 via the guest's second parent path — see §4 doubling) ``` The guest **sees both** `felhom-flash` and `felhom-usb` as mountpoints under the parent. ### Agent `/disks` report — **both reported absent** (the contradiction) ``` felhom-flash guest_path=/mnt/felhom-drives/felhom-flash bound_under_parent=False guest_attached=False felhom-usb guest_path=/mnt/felhom-drives/felhom-usb bound_under_parent=False guest_attached=False ``` Guest mountinfo says present; agent says `bound_under_parent=False`. **The report is wrong.** ### The cause — `lxc-info` fails under the non-root agent ``` # as root (what the agent did PRE-migration): works lxc-info -n 9201 -p -H → 2137736 # as the felhom-agent user, directly: FAILS (no usable $HOME) sudo -u felhom-agent lxc-info -n 9201 -p -H → lxc-info: Permission denied - Failed to create directory "/home/felhom-agent/" # as the felhom-agent user, via sudo (the runner's path): DENIED (not in NOPASSWD allowlist) sudo -u felhom-agent sudo -n lxc-info -n 9201 -p -H → sudo: a password is required # NB: the non-root user CAN read /proc//mountinfo (exit 0, sees both drives) — # the ONLY broken step is resolving the guest PID via lxc-info. ``` ### Controller gate — the live symptom (every cycle, both drives) ``` [gate] drive ABSENT /mnt/felhom-drives/felhom-usb — stopped+blocked 1 app(s): [nextcloud] [gate] drive ABSENT /mnt/felhom-drives/felhom-flash — stopped+blocked 0 app(s): [] [stacks] Stopping stack: nextcloud (repeats ~every minute) ``` --- ## 3. Causal chain (code-verified) 1. `boundUnderParent()` (`felhom-agent/internal/localapi/disks.go:703`) → `guestAttach.GuestSeesMount(ctx, vmid, stablePath)`. 2. `GuestSeesMount` (`intermediary.go:236`) calls `guestInitPID` → `b.runner.Run(ctx, "lxc-info", "-n", vmid, "-p", "-H")` (`intermediary.go:256`). 3. The agent runs **non-root** with `privileged.mode: sudo`; the runner shells every command via `sudo -n`. `lxc-info` is **absent from `configs/felhom-agent.sudoers`** (line 93 grants only `FELHOM_MOUNT/DISK/PROVISION/FORMAT/DNSMASQ/GUESTHOOK/INTERMEDIARY` — none lists `lxc-info`), so `sudo -n lxc-info` → "a password is required" → error. (Run un-sudo'd it also fails: the `felhom-agent` user has no writable `$HOME` for lxc-info's lock dir.) 4. `guestInitPID` returns `""` → `GuestSeesMount` returns `false` (pid empty) for **every** drive → `bound_under_parent=false` for all. 5. Controller `planDriveGates` (`felhom-controller/.../web/intermediary.go:202`) sets `present[guestPath] = … || d.BoundUnderParent` → `false`; the gate (line 224) `!present && !Disconnected → Stop apps + SetDisconnected(true)`. Every external drive is judged ABSENT and its apps stopped. **The gate is correct given its input** — do NOT "fix" it (the task's own warning: it is a real data-availability signal, here fed a false reading). **Why it looks like "attaching B drops A" (mutual exclusion):** with `bound_under_parent` permanently false, the gate marks *both* drives ABSENT on every cycle. An attach/enroll transiently re-binds and clears the just-touched drive's `disconnected` flag (so it briefly shows Aktív), while the gate continues to stop the other drive's apps — and re-stops the first on the next cycle. The net effect is perpetual flapping (nextcloud stopped ~every minute); the user perceives the most-recently-attached drive as "the active one" and the other as having "dropped." It is not single-active by design — it is both-absent by mis-detection. --- ## 4. Secondary observation (latent, NOT the cause) — host-side "doubling" Each drive submount appears **twice** on the host: once under the parent (mount 137) and once under root (mount 33), because the parent `/mnt/felhom-drives` carries `shared:1` — **the same peer group as `/`** (root id 33 is also `shared:1`). It was never isolated into its own group. Cause: `EnsureSharedParent` calls `mount --make-private` (`intermediary.go:110`), but **`make-private` is NOT in the sudoers** (line 85 allows only `mount --make-shared /mnt/felhom-drives`, not `--make-private`), so the agent's isolation step is denied. (The root-run boot script `felhom-shared-parent.sh` does run make-private, but the live parent is currently in root's group regardless — worth confirming at the next host boot.) This doubling makes `AttachDrive` always see `countHostMounts==2` and take the umount-all→rebind normalize branch, and propagates binds/umounts into root's namespace. It does **not** cause the reported symptom (the guest still sees both drives), but it is a real correctness smell from the same non-root-allowlist gap and should be fixed alongside. --- ## 5. Interpretation matrix — matched row | Observation | Root cause | |---|---| | **Host keeps both submounts, guest keeps both, but agent reports `BoundUnderParent=false`** | **Detection bug** in `GuestSeesMount`/`boundUnderParent` — specifically **PID resolution**: `lxc-info` denied/unusable under the non-root agent (sudoers gap + homeless user) → `guestInitPID=""` | Rows ruled out: no propagation churn (parent is a stable `shared:1` mountpoint, guest is `master:1`, peer groups did not change); guest parent is **not** `private` (it is a live slave); host did **not** drop any submount. --- ## 6. Remedy options for the operator (NOT executed) 1. **Grant `lxc-info` in the agent sudoers** (minimal, restores pre-migration behavior). Add a fixed vector e.g. `/usr/bin/lxc-info -n [0-9]* -p -H` to a `Cmnd_Alias` and the NOPASSWD line, `visudo -cf`, reload. Verified `sudo lxc-info -n 9201 -p -H` works as root → returns `2137736`. Trade-off: widens the host-root surface by one read-only command (low risk; read-only, fixed args). 2. **Remove the `lxc-info` dependency in `guestInitPID`** (code fix, no new privilege): resolve the guest init PID without lxc-info — e.g. read it from the LXC cgroup / `pct` pidfile / a `/proc` scan the non-root user can already do (it *can* read `/proc//mountinfo`; only PID *discovery* is the gap). Cleaner long-term; keeps the allowlist tight. 3. **Give `felhom-agent` a writable `$HOME`** so direct `lxc-info` works — weakest option (lxc-info on system containers still wants privilege for `/run/lxc` locks; #1 or #2 are the real fixes). **Also fix alongside (the §4 doubling):** add `mount --make-private /mnt/felhom-drives` to the sudoers `FELHOM_INTERMEDIARY` alias so `EnsureSharedParent` can isolate the parent's peer group. > Whichever path: this is a **migration-completeness** gap — the root→non-root cutover must carry the > agent's full privileged-command set (here `lxc-info`, `make-private`) into the sudoers, the same way > the leaf/token/PBS-key issues were migration carry-over gaps. --- ## 7. Confidence + gaps - **Root cause: very high.** Three independent confirmations: (a) guest mountinfo shows both drives; (b) agent `/disks` reports both `bound_under_parent=false`; (c) `lxc-info` reproducibly fails for the `felhom-agent` user (sudo-denied + homeless) yet works as root, and the code path `boundUnderParent→GuestSeesMount→guestInitPID→lxc-info` is direct. - **Not separately pinned:** the exact controller-side trigger that re-clears `disconnected` between gate cycles (enroll/re-attach reconcile) — observed in logs but not traced to a single function; it governs the *cadence* of the flapping, not the root cause. - **`guest_attached=False`** is also reported for both (a sibling `GuestConfig` path); likely the same non-root regression family but via `pct config`, not traced here — surfaces in the report but the gate keys on `BoundUnderParent`, which is the proven culprit. - **No reproduce performed** (symptom already continuously live); no mounts/config/services changed. *No secrets recorded (the per-guest token used for the read-only `/disks` check is stored out-of-band; never printed).* --- ## 8. Outcome — FIXED 2026-06-29 (supervised sudoers completeness pass) A full audit of every privileged command vs the allowlist (felhom-agent `55888e5`; full table in `felhom-agent/REPORT.md`) closed the gaps. **Sudoers-only, no binary change, no agent restart.** Grants added: `lxc-info -n [0-9]* -p -H` (the root-cause fix) + `mount --make-private /mnt/felhom-drives` → FELHOM_INTERMEDIARY; `systemctl restart dnsmasq` → FELHOM_DNSMASQ; `pct set [0-9]* -onboot 1` → FELHOM_PROVISION; `pct reboot [0-9]*` → FELHOM_GUESTHOOK. Deployed via the install-script mechanism (`visudo -cf` → `install -m 0440 -o root -g root` → re-validate `/etc/sudoers`). **Validation (all pass):** `sudo -n lxc-info -n 9201 -p -H` now returns PID `2137736`; agent `/disks` reports `bound_under_parent=true` for **both** felhom-usb + felhom-flash; the controller gate logged `drive RETURNED … re-attached + restarted` for both and **nextcloud auto-restarted** (held healthy, zero `drive ABSENT` after); the real UI (`/settings`) shows **both drives Aktív concurrently**. Symptom gone. **Surfaced, then RESOLVED (2026-06-29, agent v0.45.0):** `GuestExec`'s general `pct exec -- …` (controller-swap / managed auto-update) needed arbitrary root-in-guest. Rather than grant it, controller-swap's only shell vector (`writeImage`'s `bash -c "printf … >"`) was replaced by a stdin-fed `tee`, making all 5 swap vectors fixed/read-only and allowlistable as `FELHOM_CONTROLLERSWAP` (no `bash -c`, no general `pct exec`). Spike-proven (`SPIKE-controllerswap-narrow-grants-2026-06-29.md`) + shipped; controller-swap works under the non-root agent again (live: 45/45 capability ok, negative controls denied). **Deferred:** `sensors -j` (no caller + lm-sensors not installed). **Left as-is:** the §4 host-side doubling (re-isolating the live parent would orphan the working guest slave — fixed grant takes effect on the next clean host boot). The §7 "guest_attached / pct config" note is **resolved**: `GuestConfig` is a Proxmox **API** call, not a sudo CLI command, so `guest_attached=false` is not a sudoers gap (and is benign in the intermediary model). **Durable countermeasure SHIPPED (2026-06-29):** the agent now self-probes its privileged capabilities — felhom-agent **v0.44.0** `internal/capability` (build-time manifest⊆sudoers test + runtime `sudo -n -l` probe → `HostReport.Capabilities`) + hub **v0.19.0** `HostCapabilityChecker` (operator alert on a Critical grant going degraded). This exact incident's missing `lxc-info` grant is now a CI build failure and a live `capabilities self-check` degraded signal, not a silent multi-day regression. Live: agent reports 40/40 ok. See `felhom-agent/REPORT.md`.