spike: multi-drive mutual-exclusion — root cause is bound_under_parent detection bug (lxc-info denied under non-root agent), not propagation/gate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
This commit is contained in:
@@ -0,0 +1,184 @@
|
||||
# SPIKE — 2nd external drive "drops" the 1st (multi-drive mutual exclusion / flapping)
|
||||
|
||||
- **Date:** 2026-06-29
|
||||
- **Class:** Diagnostic / read-only. **No fix.** No deliberate reproduce was needed — the symptom is
|
||||
**continuously live** in the controller gate logs, so the host-vs-guest `mountinfo` diff was read
|
||||
directly from the running state (an attach would only re-stop apps, adding no evidence).
|
||||
- **Host:** felhom-pve (`demo-felhom`, 192.168.0.162); guest LXC 9201 (init pid 2137736).
|
||||
- **Live versions:** agent **0.43.0** (NON-ROOT `felhom-agent` since 2026-06-28), controller **0.89.0**.
|
||||
- **Verdict:** **Detection bug (matrix row 3), NOT a propagation fault and NOT a controller-gate
|
||||
bug.** The guest demonstrably keeps **both** drive submounts; the agent mis-reports
|
||||
`bound_under_parent=false` for **every** drive because the non-root agent cannot resolve the guest
|
||||
init PID (`lxc-info` is denied / unusable under the `felhom-agent` user). The controller gate is the
|
||||
**messenger** — it correctly stops apps for a drive its report says is absent. Root cause is the
|
||||
**2026-06-28 root→non-root agent migration** (same regression family as the leaf-pin + PBS-key
|
||||
issues): `lxc-info` was never added to the agent sudoers allowlist.
|
||||
|
||||
---
|
||||
|
||||
## 1. The question (§3) — answered
|
||||
|
||||
> Does the guest actually lose drive A's submount (propagation fault), or does the guest keep it
|
||||
> while the agent mis-reports `BoundUnderParent=false` (detection bug)?
|
||||
|
||||
**The guest keeps BOTH submounts; the agent mis-reports.** Propagation is intact (the guest's parent
|
||||
bind is `master:1`, a live slave of the host's `shared:1` parent, and both drives are visible in the
|
||||
guest namespace right now). The fault is purely in the agent's *guest-sees* detection.
|
||||
|
||||
---
|
||||
|
||||
## 2. Raw evidence
|
||||
|
||||
### A1/A2 — host parent + submounts (both drives bound, propagation `shared`)
|
||||
```
|
||||
findmnt /mnt/felhom-drives:
|
||||
/mnt/felhom-drives /dev/mapper/pve-root[/mnt/felhom-drives] shared
|
||||
├─/mnt/felhom-drives/felhom-flash /dev/sdc1[/felhom-data] shared
|
||||
└─/mnt/felhom-drives/felhom-usb /dev/sdb1[/felhom-data] shared
|
||||
|
||||
/proc/self/mountinfo (host):
|
||||
137 33 252:1 /mnt/felhom-drives /mnt/felhom-drives shared:1
|
||||
99 137 8:33 /felhom-data /mnt/felhom-drives/felhom-flash shared:51 ← child of parent 137
|
||||
100 33 8:33 /felhom-data /mnt/felhom-drives/felhom-flash shared:51 ← ALSO child of root 33 (doubling)
|
||||
109 137 8:17 /felhom-data /mnt/felhom-drives/felhom-usb shared:49
|
||||
110 33 8:17 /felhom-data /mnt/felhom-drives/felhom-usb shared:49
|
||||
```
|
||||
|
||||
### A3 — GUEST view (the `BoundUnderParent` truth) — **both drives present**
|
||||
```
|
||||
guest_pid=2137736
|
||||
571 324 252:1 /mnt/felhom-drives /mnt/felhom-drives shared:482 master:1 ← slave of host group 1 (propagation LIVE)
|
||||
108 571 8:33 /felhom-data /mnt/felhom-drives/felhom-flash shared:195 master:51
|
||||
344 571 8:17 /felhom-data /mnt/felhom-drives/felhom-usb shared:210 master:49
|
||||
(+ duplicate entries 572/111/345 via the guest's second parent path — see §4 doubling)
|
||||
```
|
||||
The guest **sees both** `felhom-flash` and `felhom-usb` as mountpoints under the parent.
|
||||
|
||||
### Agent `/disks` report — **both reported absent** (the contradiction)
|
||||
```
|
||||
felhom-flash guest_path=/mnt/felhom-drives/felhom-flash bound_under_parent=False guest_attached=False
|
||||
felhom-usb guest_path=/mnt/felhom-drives/felhom-usb bound_under_parent=False guest_attached=False
|
||||
```
|
||||
Guest mountinfo says present; agent says `bound_under_parent=False`. **The report is wrong.**
|
||||
|
||||
### The cause — `lxc-info` fails under the non-root agent
|
||||
```
|
||||
# as root (what the agent did PRE-migration): works
|
||||
lxc-info -n 9201 -p -H → 2137736
|
||||
|
||||
# as the felhom-agent user, directly: FAILS (no usable $HOME)
|
||||
sudo -u felhom-agent lxc-info -n 9201 -p -H
|
||||
→ lxc-info: Permission denied - Failed to create directory "/home/felhom-agent/"
|
||||
|
||||
# as the felhom-agent user, via sudo (the runner's path): DENIED (not in NOPASSWD allowlist)
|
||||
sudo -u felhom-agent sudo -n lxc-info -n 9201 -p -H
|
||||
→ sudo: a password is required
|
||||
|
||||
# NB: the non-root user CAN read /proc/<pid>/mountinfo (exit 0, sees both drives) —
|
||||
# the ONLY broken step is resolving the guest PID via lxc-info.
|
||||
```
|
||||
|
||||
### Controller gate — the live symptom (every cycle, both drives)
|
||||
```
|
||||
[gate] drive ABSENT /mnt/felhom-drives/felhom-usb — stopped+blocked 1 app(s): [nextcloud]
|
||||
[gate] drive ABSENT /mnt/felhom-drives/felhom-flash — stopped+blocked 0 app(s): []
|
||||
[stacks] Stopping stack: nextcloud (repeats ~every minute)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Causal chain (code-verified)
|
||||
|
||||
1. `boundUnderParent()` (`felhom-agent/internal/localapi/disks.go:703`) →
|
||||
`guestAttach.GuestSeesMount(ctx, vmid, stablePath)`.
|
||||
2. `GuestSeesMount` (`intermediary.go:236`) calls `guestInitPID` →
|
||||
`b.runner.Run(ctx, "lxc-info", "-n", vmid, "-p", "-H")` (`intermediary.go:256`).
|
||||
3. The agent runs **non-root** with `privileged.mode: sudo`; the runner shells every command via
|
||||
`sudo -n`. `lxc-info` is **absent from `configs/felhom-agent.sudoers`** (line 93 grants only
|
||||
`FELHOM_MOUNT/DISK/PROVISION/FORMAT/DNSMASQ/GUESTHOOK/INTERMEDIARY` — none lists `lxc-info`), so
|
||||
`sudo -n lxc-info` → "a password is required" → error. (Run un-sudo'd it also fails: the
|
||||
`felhom-agent` user has no writable `$HOME` for lxc-info's lock dir.)
|
||||
4. `guestInitPID` returns `""` → `GuestSeesMount` returns `false` (pid empty) for **every** drive →
|
||||
`bound_under_parent=false` for all.
|
||||
5. Controller `planDriveGates` (`felhom-controller/.../web/intermediary.go:202`) sets
|
||||
`present[guestPath] = … || d.BoundUnderParent` → `false`; the gate (line 224)
|
||||
`!present && !Disconnected → Stop apps + SetDisconnected(true)`. Every external drive is judged
|
||||
ABSENT and its apps stopped. **The gate is correct given its input** — do NOT "fix" it (the task's
|
||||
own warning: it is a real data-availability signal, here fed a false reading).
|
||||
|
||||
**Why it looks like "attaching B drops A" (mutual exclusion):** with `bound_under_parent` permanently
|
||||
false, the gate marks *both* drives ABSENT on every cycle. An attach/enroll transiently re-binds and
|
||||
clears the just-touched drive's `disconnected` flag (so it briefly shows Aktív), while the gate
|
||||
continues to stop the other drive's apps — and re-stops the first on the next cycle. The net effect is
|
||||
perpetual flapping (nextcloud stopped ~every minute); the user perceives the most-recently-attached
|
||||
drive as "the active one" and the other as having "dropped." It is not single-active by design — it is
|
||||
both-absent by mis-detection.
|
||||
|
||||
---
|
||||
|
||||
## 4. Secondary observation (latent, NOT the cause) — host-side "doubling"
|
||||
|
||||
Each drive submount appears **twice** on the host: once under the parent (mount 137) and once under
|
||||
root (mount 33), because the parent `/mnt/felhom-drives` carries `shared:1` — **the same peer group as
|
||||
`/`** (root id 33 is also `shared:1`). It was never isolated into its own group. Cause:
|
||||
`EnsureSharedParent` calls `mount --make-private` (`intermediary.go:110`), but **`make-private` is NOT
|
||||
in the sudoers** (line 85 allows only `mount --make-shared /mnt/felhom-drives`, not `--make-private`),
|
||||
so the agent's isolation step is denied. (The root-run boot script `felhom-shared-parent.sh` does run
|
||||
make-private, but the live parent is currently in root's group regardless — worth confirming at the
|
||||
next host boot.) This doubling makes `AttachDrive` always see `countHostMounts==2` and take the
|
||||
umount-all→rebind normalize branch, and propagates binds/umounts into root's namespace. It does **not**
|
||||
cause the reported symptom (the guest still sees both drives), but it is a real correctness smell from
|
||||
the same non-root-allowlist gap and should be fixed alongside.
|
||||
|
||||
---
|
||||
|
||||
## 5. Interpretation matrix — matched row
|
||||
|
||||
| Observation | Root cause |
|
||||
|---|---|
|
||||
| **Host keeps both submounts, guest keeps both, but agent reports `BoundUnderParent=false`** | **Detection bug** in `GuestSeesMount`/`boundUnderParent` — specifically **PID resolution**: `lxc-info` denied/unusable under the non-root agent (sudoers gap + homeless user) → `guestInitPID=""` |
|
||||
|
||||
Rows ruled out: no propagation churn (parent is a stable `shared:1` mountpoint, guest is `master:1`,
|
||||
peer groups did not change); guest parent is **not** `private` (it is a live slave); host did **not**
|
||||
drop any submount.
|
||||
|
||||
---
|
||||
|
||||
## 6. Remedy options for the operator (NOT executed)
|
||||
|
||||
1. **Grant `lxc-info` in the agent sudoers** (minimal, restores pre-migration behavior). Add a fixed
|
||||
vector e.g. `/usr/bin/lxc-info -n [0-9]* -p -H` to a `Cmnd_Alias` and the NOPASSWD line, `visudo
|
||||
-cf`, reload. Verified `sudo lxc-info -n 9201 -p -H` works as root → returns `2137736`. Trade-off:
|
||||
widens the host-root surface by one read-only command (low risk; read-only, fixed args).
|
||||
2. **Remove the `lxc-info` dependency in `guestInitPID`** (code fix, no new privilege): resolve the
|
||||
guest init PID without lxc-info — e.g. read it from the LXC cgroup / `pct` pidfile / a `/proc` scan
|
||||
the non-root user can already do (it *can* read `/proc/<pid>/mountinfo`; only PID *discovery* is the
|
||||
gap). Cleaner long-term; keeps the allowlist tight.
|
||||
3. **Give `felhom-agent` a writable `$HOME`** so direct `lxc-info` works — weakest option (lxc-info on
|
||||
system containers still wants privilege for `/run/lxc` locks; #1 or #2 are the real fixes).
|
||||
|
||||
**Also fix alongside (the §4 doubling):** add `mount --make-private /mnt/felhom-drives` to the sudoers
|
||||
`FELHOM_INTERMEDIARY` alias so `EnsureSharedParent` can isolate the parent's peer group.
|
||||
|
||||
> Whichever path: this is a **migration-completeness** gap — the root→non-root cutover must carry the
|
||||
> agent's full privileged-command set (here `lxc-info`, `make-private`) into the sudoers, the same way
|
||||
> the leaf/token/PBS-key issues were migration carry-over gaps.
|
||||
|
||||
---
|
||||
|
||||
## 7. Confidence + gaps
|
||||
|
||||
- **Root cause: very high.** Three independent confirmations: (a) guest mountinfo shows both drives;
|
||||
(b) agent `/disks` reports both `bound_under_parent=false`; (c) `lxc-info` reproducibly fails for the
|
||||
`felhom-agent` user (sudo-denied + homeless) yet works as root, and the code path
|
||||
`boundUnderParent→GuestSeesMount→guestInitPID→lxc-info` is direct.
|
||||
- **Not separately pinned:** the exact controller-side trigger that re-clears `disconnected` between
|
||||
gate cycles (enroll/re-attach reconcile) — observed in logs but not traced to a single function; it
|
||||
governs the *cadence* of the flapping, not the root cause.
|
||||
- **`guest_attached=False`** is also reported for both (a sibling `GuestConfig` path); likely the same
|
||||
non-root regression family but via `pct config`, not traced here — surfaces in the report but the
|
||||
gate keys on `BoundUnderParent`, which is the proven culprit.
|
||||
- **No reproduce performed** (symptom already continuously live); no mounts/config/services changed.
|
||||
|
||||
*No secrets recorded (the per-guest token used for the read-only `/disks` check is stored
|
||||
out-of-band; never printed).*
|
||||
Reference in New Issue
Block a user