# SPIKE — 2nd external drive "drops" the 1st (multi-drive mutual exclusion / flapping)

- **Date:** 2026-06-29
- **Class:** Diagnostic / read-only. **No fix.** No deliberate reproduce was needed — the symptom is
  **continuously live** in the controller gate logs, so the host-vs-guest `mountinfo` diff was read
  directly from the running state (an attach would only re-stop apps, adding no evidence).
- **Host:** felhom-pve (`demo-felhom`, 192.168.0.162); guest LXC 9201 (init pid 2137736).
- **Live versions:** agent **0.43.0** (NON-ROOT `felhom-agent` since 2026-06-28), controller **0.89.0**.
- **Verdict:** **Detection bug (matrix row 3), NOT a propagation fault and NOT a controller-gate
  bug.** The guest demonstrably keeps **both** drive submounts; the agent mis-reports
  `bound_under_parent=false` for **every** drive because the non-root agent cannot resolve the guest
  init PID (`lxc-info` is denied / unusable under the `felhom-agent` user). The controller gate is the
  **messenger** — it correctly stops apps for a drive its report says is absent. Root cause is the
  **2026-06-28 root→non-root agent migration** (same regression family as the leaf-pin + PBS-key
  issues): `lxc-info` was never added to the agent sudoers allowlist.

---

## 1. The question (§3) — answered

> Does the guest actually lose drive A's submount (propagation fault), or does the guest keep it
> while the agent mis-reports `BoundUnderParent=false` (detection bug)?

**The guest keeps BOTH submounts; the agent mis-reports.** Propagation is intact (the guest's parent
bind is `master:1`, a live slave of the host's `shared:1` parent, and both drives are visible in the
guest namespace right now). The fault is purely in the agent's *guest-sees* detection.

---

## 2. Raw evidence

### A1/A2 — host parent + submounts (both drives bound, propagation `shared`)
```
findmnt /mnt/felhom-drives:
  /mnt/felhom-drives                /dev/mapper/pve-root[/mnt/felhom-drives] shared
  ├─/mnt/felhom-drives/felhom-flash /dev/sdc1[/felhom-data]                  shared
  └─/mnt/felhom-drives/felhom-usb   /dev/sdb1[/felhom-data]                  shared

/proc/self/mountinfo (host):
  137 33  252:1 /mnt/felhom-drives        /mnt/felhom-drives         shared:1
   99 137 8:33  /felhom-data /mnt/felhom-drives/felhom-flash shared:51   ← child of parent 137
  100 33  8:33  /felhom-data /mnt/felhom-drives/felhom-flash shared:51   ← ALSO child of root 33 (doubling)
  109 137 8:17  /felhom-data /mnt/felhom-drives/felhom-usb   shared:49
  110 33  8:17  /felhom-data /mnt/felhom-drives/felhom-usb   shared:49
```

### A3 — GUEST view (the `BoundUnderParent` truth) — **both drives present**
```
guest_pid=2137736
571 324 252:1 /mnt/felhom-drives                /mnt/felhom-drives         shared:482 master:1   ← slave of host group 1 (propagation LIVE)
108 571 8:33  /felhom-data /mnt/felhom-drives/felhom-flash shared:195 master:51
344 571 8:17  /felhom-data /mnt/felhom-drives/felhom-usb   shared:210 master:49
   (+ duplicate entries 572/111/345 via the guest's second parent path — see §4 doubling)
```
The guest **sees both** `felhom-flash` and `felhom-usb` as mountpoints under the parent.

### Agent `/disks` report — **both reported absent** (the contradiction)
```
felhom-flash  guest_path=/mnt/felhom-drives/felhom-flash  bound_under_parent=False  guest_attached=False
felhom-usb    guest_path=/mnt/felhom-drives/felhom-usb    bound_under_parent=False  guest_attached=False
```
Guest mountinfo says present; agent says `bound_under_parent=False`. **The report is wrong.**

### The cause — `lxc-info` fails under the non-root agent
```
# as root (what the agent did PRE-migration):           works
  lxc-info -n 9201 -p -H            → 2137736

# as the felhom-agent user, directly:                    FAILS (no usable $HOME)
  sudo -u felhom-agent lxc-info -n 9201 -p -H
    → lxc-info: Permission denied - Failed to create directory "/home/felhom-agent/"

# as the felhom-agent user, via sudo (the runner's path): DENIED (not in NOPASSWD allowlist)
  sudo -u felhom-agent sudo -n lxc-info -n 9201 -p -H
    → sudo: a password is required

# NB: the non-root user CAN read /proc/<pid>/mountinfo (exit 0, sees both drives) —
#     the ONLY broken step is resolving the guest PID via lxc-info.
```

### Controller gate — the live symptom (every cycle, both drives)
```
[gate] drive ABSENT /mnt/felhom-drives/felhom-usb  — stopped+blocked 1 app(s): [nextcloud]
[gate] drive ABSENT /mnt/felhom-drives/felhom-flash — stopped+blocked 0 app(s): []
[stacks] Stopping stack: nextcloud   (repeats ~every minute)
```

---

## 3. Causal chain (code-verified)

1. `boundUnderParent()` (`felhom-agent/internal/localapi/disks.go:703`) →
   `guestAttach.GuestSeesMount(ctx, vmid, stablePath)`.
2. `GuestSeesMount` (`intermediary.go:236`) calls `guestInitPID` →
   `b.runner.Run(ctx, "lxc-info", "-n", vmid, "-p", "-H")` (`intermediary.go:256`).
3. The agent runs **non-root** with `privileged.mode: sudo`; the runner shells every command via
   `sudo -n`. `lxc-info` is **absent from `configs/felhom-agent.sudoers`** (line 93 grants only
   `FELHOM_MOUNT/DISK/PROVISION/FORMAT/DNSMASQ/GUESTHOOK/INTERMEDIARY` — none lists `lxc-info`), so
   `sudo -n lxc-info` → "a password is required" → error. (Run un-sudo'd it also fails: the
   `felhom-agent` user has no writable `$HOME` for lxc-info's lock dir.)
4. `guestInitPID` returns `""` → `GuestSeesMount` returns `false` (pid empty) for **every** drive →
   `bound_under_parent=false` for all.
5. Controller `planDriveGates` (`felhom-controller/.../web/intermediary.go:202`) sets
   `present[guestPath] = … || d.BoundUnderParent` → `false`; the gate (line 224)
   `!present && !Disconnected → Stop apps + SetDisconnected(true)`. Every external drive is judged
   ABSENT and its apps stopped. **The gate is correct given its input** — do NOT "fix" it (the task's
   own warning: it is a real data-availability signal, here fed a false reading).

**Why it looks like "attaching B drops A" (mutual exclusion):** with `bound_under_parent` permanently
false, the gate marks *both* drives ABSENT on every cycle. An attach/enroll transiently re-binds and
clears the just-touched drive's `disconnected` flag (so it briefly shows Aktív), while the gate
continues to stop the other drive's apps — and re-stops the first on the next cycle. The net effect is
perpetual flapping (nextcloud stopped ~every minute); the user perceives the most-recently-attached
drive as "the active one" and the other as having "dropped." It is not single-active by design — it is
both-absent by mis-detection.

---

## 4. Secondary observation (latent, NOT the cause) — host-side "doubling"

Each drive submount appears **twice** on the host: once under the parent (mount 137) and once under
root (mount 33), because the parent `/mnt/felhom-drives` carries `shared:1` — **the same peer group as
`/`** (root id 33 is also `shared:1`). It was never isolated into its own group. Cause:
`EnsureSharedParent` calls `mount --make-private` (`intermediary.go:110`), but **`make-private` is NOT
in the sudoers** (line 85 allows only `mount --make-shared /mnt/felhom-drives`, not `--make-private`),
so the agent's isolation step is denied. (The root-run boot script `felhom-shared-parent.sh` does run
make-private, but the live parent is currently in root's group regardless — worth confirming at the
next host boot.) This doubling makes `AttachDrive` always see `countHostMounts==2` and take the
umount-all→rebind normalize branch, and propagates binds/umounts into root's namespace. It does **not**
cause the reported symptom (the guest still sees both drives), but it is a real correctness smell from
the same non-root-allowlist gap and should be fixed alongside.

---

## 5. Interpretation matrix — matched row

| Observation | Root cause |
|---|---|
| **Host keeps both submounts, guest keeps both, but agent reports `BoundUnderParent=false`** | **Detection bug** in `GuestSeesMount`/`boundUnderParent` — specifically **PID resolution**: `lxc-info` denied/unusable under the non-root agent (sudoers gap + homeless user) → `guestInitPID=""` |

Rows ruled out: no propagation churn (parent is a stable `shared:1` mountpoint, guest is `master:1`,
peer groups did not change); guest parent is **not** `private` (it is a live slave); host did **not**
drop any submount.

---

## 6. Remedy options for the operator (NOT executed)

1. **Grant `lxc-info` in the agent sudoers** (minimal, restores pre-migration behavior). Add a fixed
   vector e.g. `/usr/bin/lxc-info -n [0-9]* -p -H` to a `Cmnd_Alias` and the NOPASSWD line, `visudo
   -cf`, reload. Verified `sudo lxc-info -n 9201 -p -H` works as root → returns `2137736`. Trade-off:
   widens the host-root surface by one read-only command (low risk; read-only, fixed args).
2. **Remove the `lxc-info` dependency in `guestInitPID`** (code fix, no new privilege): resolve the
   guest init PID without lxc-info — e.g. read it from the LXC cgroup / `pct` pidfile / a `/proc` scan
   the non-root user can already do (it *can* read `/proc/<pid>/mountinfo`; only PID *discovery* is the
   gap). Cleaner long-term; keeps the allowlist tight.
3. **Give `felhom-agent` a writable `$HOME`** so direct `lxc-info` works — weakest option (lxc-info on
   system containers still wants privilege for `/run/lxc` locks; #1 or #2 are the real fixes).

**Also fix alongside (the §4 doubling):** add `mount --make-private /mnt/felhom-drives` to the sudoers
`FELHOM_INTERMEDIARY` alias so `EnsureSharedParent` can isolate the parent's peer group.

> Whichever path: this is a **migration-completeness** gap — the root→non-root cutover must carry the
> agent's full privileged-command set (here `lxc-info`, `make-private`) into the sudoers, the same way
> the leaf/token/PBS-key issues were migration carry-over gaps.

---

## 7. Confidence + gaps

- **Root cause: very high.** Three independent confirmations: (a) guest mountinfo shows both drives;
  (b) agent `/disks` reports both `bound_under_parent=false`; (c) `lxc-info` reproducibly fails for the
  `felhom-agent` user (sudo-denied + homeless) yet works as root, and the code path
  `boundUnderParent→GuestSeesMount→guestInitPID→lxc-info` is direct.
- **Not separately pinned:** the exact controller-side trigger that re-clears `disconnected` between
  gate cycles (enroll/re-attach reconcile) — observed in logs but not traced to a single function; it
  governs the *cadence* of the flapping, not the root cause.
- **`guest_attached=False`** is also reported for both (a sibling `GuestConfig` path); likely the same
  non-root regression family but via `pct config`, not traced here — surfaces in the report but the
  gate keys on `BoundUnderParent`, which is the proven culprit.
- **No reproduce performed** (symptom already continuously live); no mounts/config/services changed.

*No secrets recorded (the per-guest token used for the read-only `/disks` check is stored
out-of-band; never printed).*

---

## 8. Outcome — FIXED 2026-06-29 (supervised sudoers completeness pass)

A full audit of every privileged command vs the allowlist (felhom-agent `55888e5`; full table in
`felhom-agent/REPORT.md`) closed the gaps. **Sudoers-only, no binary change, no agent restart.**

Grants added: `lxc-info -n [0-9]* -p -H` (the root-cause fix) + `mount --make-private
/mnt/felhom-drives` → FELHOM_INTERMEDIARY; `systemctl restart dnsmasq` → FELHOM_DNSMASQ; `pct set
[0-9]* -onboot 1` → FELHOM_PROVISION; `pct reboot [0-9]*` → FELHOM_GUESTHOOK. Deployed via the
install-script mechanism (`visudo -cf` → `install -m 0440 -o root -g root` → re-validate `/etc/sudoers`).

**Validation (all pass):** `sudo -n lxc-info -n 9201 -p -H` now returns PID `2137736`; agent `/disks`
reports `bound_under_parent=true` for **both** felhom-usb + felhom-flash; the controller gate logged
`drive RETURNED … re-attached + restarted` for both and **nextcloud auto-restarted** (held healthy,
zero `drive ABSENT` after); the real UI (`/settings`) shows **both drives Aktív concurrently**. Symptom
gone.

**Surfaced, then RESOLVED (2026-06-29, agent v0.45.0):** `GuestExec`'s general `pct exec <vmid> -- …`
(controller-swap / managed auto-update) needed arbitrary root-in-guest. Rather than grant it,
controller-swap's only shell vector (`writeImage`'s `bash -c "printf … >"`) was replaced by a
stdin-fed `tee`, making all 5 swap vectors fixed/read-only and allowlistable as `FELHOM_CONTROLLERSWAP`
(no `bash -c`, no general `pct exec`). Spike-proven (`SPIKE-controllerswap-narrow-grants-2026-06-29.md`)
+ shipped; controller-swap works under the non-root agent again (live: 45/45 capability ok, negative
controls denied). **Deferred:** `sensors -j`
(no caller + lm-sensors not installed). **Left as-is:** the §4 host-side doubling (re-isolating the
live parent would orphan the working guest slave — fixed grant takes effect on the next clean host
boot). The §7 "guest_attached / pct config" note is **resolved**: `GuestConfig` is a Proxmox **API**
call, not a sudo CLI command, so `guest_attached=false` is not a sudoers gap (and is benign in the
intermediary model).

**Durable countermeasure SHIPPED (2026-06-29):** the agent now self-probes its privileged
capabilities — felhom-agent **v0.44.0** `internal/capability` (build-time manifest⊆sudoers test +
runtime `sudo -n -l` probe → `HostReport.Capabilities`) + hub **v0.19.0** `HostCapabilityChecker`
(operator alert on a Critical grant going degraded). This exact incident's missing `lxc-info` grant
is now a CI build failure and a live `capabilities self-check` degraded signal, not a silent
multi-day regression. Live: agent reports 40/40 ok. See `felhom-agent/REPORT.md`.
