From 66a001de0bb78d27803d6756995c6ec2c591caa7 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 29 Jun 2026 15:33:44 +0200 Subject: [PATCH] =?UTF-8?q?spike:=20multi-drive=20mutual-exclusion=20?= =?UTF-8?q?=E2=80=94=20root=20cause=20is=20bound=5Funder=5Fparent=20detect?= =?UTF-8?q?ion=20bug=20(lxc-info=20denied=20under=20non-root=20agent),=20n?= =?UTF-8?q?ot=20propagation/gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt --- ...-multidrive-mutual-exclusion-2026-06-29.md | 184 ++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 documentation/audits/SPIKE-multidrive-mutual-exclusion-2026-06-29.md diff --git a/documentation/audits/SPIKE-multidrive-mutual-exclusion-2026-06-29.md b/documentation/audits/SPIKE-multidrive-mutual-exclusion-2026-06-29.md new file mode 100644 index 0000000..75d4ddf --- /dev/null +++ b/documentation/audits/SPIKE-multidrive-mutual-exclusion-2026-06-29.md @@ -0,0 +1,184 @@ +# SPIKE — 2nd external drive "drops" the 1st (multi-drive mutual exclusion / flapping) + +- **Date:** 2026-06-29 +- **Class:** Diagnostic / read-only. **No fix.** No deliberate reproduce was needed — the symptom is + **continuously live** in the controller gate logs, so the host-vs-guest `mountinfo` diff was read + directly from the running state (an attach would only re-stop apps, adding no evidence). +- **Host:** felhom-pve (`demo-felhom`, 192.168.0.162); guest LXC 9201 (init pid 2137736). +- **Live versions:** agent **0.43.0** (NON-ROOT `felhom-agent` since 2026-06-28), controller **0.89.0**. +- **Verdict:** **Detection bug (matrix row 3), NOT a propagation fault and NOT a controller-gate + bug.** The guest demonstrably keeps **both** drive submounts; the agent mis-reports + `bound_under_parent=false` for **every** drive because the non-root agent cannot resolve the guest + init PID (`lxc-info` is denied / unusable under the `felhom-agent` user). The controller gate is the + **messenger** — it correctly stops apps for a drive its report says is absent. Root cause is the + **2026-06-28 root→non-root agent migration** (same regression family as the leaf-pin + PBS-key + issues): `lxc-info` was never added to the agent sudoers allowlist. + +--- + +## 1. The question (§3) — answered + +> Does the guest actually lose drive A's submount (propagation fault), or does the guest keep it +> while the agent mis-reports `BoundUnderParent=false` (detection bug)? + +**The guest keeps BOTH submounts; the agent mis-reports.** Propagation is intact (the guest's parent +bind is `master:1`, a live slave of the host's `shared:1` parent, and both drives are visible in the +guest namespace right now). The fault is purely in the agent's *guest-sees* detection. + +--- + +## 2. Raw evidence + +### A1/A2 — host parent + submounts (both drives bound, propagation `shared`) +``` +findmnt /mnt/felhom-drives: + /mnt/felhom-drives /dev/mapper/pve-root[/mnt/felhom-drives] shared + ├─/mnt/felhom-drives/felhom-flash /dev/sdc1[/felhom-data] shared + └─/mnt/felhom-drives/felhom-usb /dev/sdb1[/felhom-data] shared + +/proc/self/mountinfo (host): + 137 33 252:1 /mnt/felhom-drives /mnt/felhom-drives shared:1 + 99 137 8:33 /felhom-data /mnt/felhom-drives/felhom-flash shared:51 ← child of parent 137 + 100 33 8:33 /felhom-data /mnt/felhom-drives/felhom-flash shared:51 ← ALSO child of root 33 (doubling) + 109 137 8:17 /felhom-data /mnt/felhom-drives/felhom-usb shared:49 + 110 33 8:17 /felhom-data /mnt/felhom-drives/felhom-usb shared:49 +``` + +### A3 — GUEST view (the `BoundUnderParent` truth) — **both drives present** +``` +guest_pid=2137736 +571 324 252:1 /mnt/felhom-drives /mnt/felhom-drives shared:482 master:1 ← slave of host group 1 (propagation LIVE) +108 571 8:33 /felhom-data /mnt/felhom-drives/felhom-flash shared:195 master:51 +344 571 8:17 /felhom-data /mnt/felhom-drives/felhom-usb shared:210 master:49 + (+ duplicate entries 572/111/345 via the guest's second parent path — see §4 doubling) +``` +The guest **sees both** `felhom-flash` and `felhom-usb` as mountpoints under the parent. + +### Agent `/disks` report — **both reported absent** (the contradiction) +``` +felhom-flash guest_path=/mnt/felhom-drives/felhom-flash bound_under_parent=False guest_attached=False +felhom-usb guest_path=/mnt/felhom-drives/felhom-usb bound_under_parent=False guest_attached=False +``` +Guest mountinfo says present; agent says `bound_under_parent=False`. **The report is wrong.** + +### The cause — `lxc-info` fails under the non-root agent +``` +# as root (what the agent did PRE-migration): works + lxc-info -n 9201 -p -H → 2137736 + +# as the felhom-agent user, directly: FAILS (no usable $HOME) + sudo -u felhom-agent lxc-info -n 9201 -p -H + → lxc-info: Permission denied - Failed to create directory "/home/felhom-agent/" + +# as the felhom-agent user, via sudo (the runner's path): DENIED (not in NOPASSWD allowlist) + sudo -u felhom-agent sudo -n lxc-info -n 9201 -p -H + → sudo: a password is required + +# NB: the non-root user CAN read /proc//mountinfo (exit 0, sees both drives) — +# the ONLY broken step is resolving the guest PID via lxc-info. +``` + +### Controller gate — the live symptom (every cycle, both drives) +``` +[gate] drive ABSENT /mnt/felhom-drives/felhom-usb — stopped+blocked 1 app(s): [nextcloud] +[gate] drive ABSENT /mnt/felhom-drives/felhom-flash — stopped+blocked 0 app(s): [] +[stacks] Stopping stack: nextcloud (repeats ~every minute) +``` + +--- + +## 3. Causal chain (code-verified) + +1. `boundUnderParent()` (`felhom-agent/internal/localapi/disks.go:703`) → + `guestAttach.GuestSeesMount(ctx, vmid, stablePath)`. +2. `GuestSeesMount` (`intermediary.go:236`) calls `guestInitPID` → + `b.runner.Run(ctx, "lxc-info", "-n", vmid, "-p", "-H")` (`intermediary.go:256`). +3. The agent runs **non-root** with `privileged.mode: sudo`; the runner shells every command via + `sudo -n`. `lxc-info` is **absent from `configs/felhom-agent.sudoers`** (line 93 grants only + `FELHOM_MOUNT/DISK/PROVISION/FORMAT/DNSMASQ/GUESTHOOK/INTERMEDIARY` — none lists `lxc-info`), so + `sudo -n lxc-info` → "a password is required" → error. (Run un-sudo'd it also fails: the + `felhom-agent` user has no writable `$HOME` for lxc-info's lock dir.) +4. `guestInitPID` returns `""` → `GuestSeesMount` returns `false` (pid empty) for **every** drive → + `bound_under_parent=false` for all. +5. Controller `planDriveGates` (`felhom-controller/.../web/intermediary.go:202`) sets + `present[guestPath] = … || d.BoundUnderParent` → `false`; the gate (line 224) + `!present && !Disconnected → Stop apps + SetDisconnected(true)`. Every external drive is judged + ABSENT and its apps stopped. **The gate is correct given its input** — do NOT "fix" it (the task's + own warning: it is a real data-availability signal, here fed a false reading). + +**Why it looks like "attaching B drops A" (mutual exclusion):** with `bound_under_parent` permanently +false, the gate marks *both* drives ABSENT on every cycle. An attach/enroll transiently re-binds and +clears the just-touched drive's `disconnected` flag (so it briefly shows Aktív), while the gate +continues to stop the other drive's apps — and re-stops the first on the next cycle. The net effect is +perpetual flapping (nextcloud stopped ~every minute); the user perceives the most-recently-attached +drive as "the active one" and the other as having "dropped." It is not single-active by design — it is +both-absent by mis-detection. + +--- + +## 4. Secondary observation (latent, NOT the cause) — host-side "doubling" + +Each drive submount appears **twice** on the host: once under the parent (mount 137) and once under +root (mount 33), because the parent `/mnt/felhom-drives` carries `shared:1` — **the same peer group as +`/`** (root id 33 is also `shared:1`). It was never isolated into its own group. Cause: +`EnsureSharedParent` calls `mount --make-private` (`intermediary.go:110`), but **`make-private` is NOT +in the sudoers** (line 85 allows only `mount --make-shared /mnt/felhom-drives`, not `--make-private`), +so the agent's isolation step is denied. (The root-run boot script `felhom-shared-parent.sh` does run +make-private, but the live parent is currently in root's group regardless — worth confirming at the +next host boot.) This doubling makes `AttachDrive` always see `countHostMounts==2` and take the +umount-all→rebind normalize branch, and propagates binds/umounts into root's namespace. It does **not** +cause the reported symptom (the guest still sees both drives), but it is a real correctness smell from +the same non-root-allowlist gap and should be fixed alongside. + +--- + +## 5. Interpretation matrix — matched row + +| Observation | Root cause | +|---|---| +| **Host keeps both submounts, guest keeps both, but agent reports `BoundUnderParent=false`** | **Detection bug** in `GuestSeesMount`/`boundUnderParent` — specifically **PID resolution**: `lxc-info` denied/unusable under the non-root agent (sudoers gap + homeless user) → `guestInitPID=""` | + +Rows ruled out: no propagation churn (parent is a stable `shared:1` mountpoint, guest is `master:1`, +peer groups did not change); guest parent is **not** `private` (it is a live slave); host did **not** +drop any submount. + +--- + +## 6. Remedy options for the operator (NOT executed) + +1. **Grant `lxc-info` in the agent sudoers** (minimal, restores pre-migration behavior). Add a fixed + vector e.g. `/usr/bin/lxc-info -n [0-9]* -p -H` to a `Cmnd_Alias` and the NOPASSWD line, `visudo + -cf`, reload. Verified `sudo lxc-info -n 9201 -p -H` works as root → returns `2137736`. Trade-off: + widens the host-root surface by one read-only command (low risk; read-only, fixed args). +2. **Remove the `lxc-info` dependency in `guestInitPID`** (code fix, no new privilege): resolve the + guest init PID without lxc-info — e.g. read it from the LXC cgroup / `pct` pidfile / a `/proc` scan + the non-root user can already do (it *can* read `/proc//mountinfo`; only PID *discovery* is the + gap). Cleaner long-term; keeps the allowlist tight. +3. **Give `felhom-agent` a writable `$HOME`** so direct `lxc-info` works — weakest option (lxc-info on + system containers still wants privilege for `/run/lxc` locks; #1 or #2 are the real fixes). + +**Also fix alongside (the §4 doubling):** add `mount --make-private /mnt/felhom-drives` to the sudoers +`FELHOM_INTERMEDIARY` alias so `EnsureSharedParent` can isolate the parent's peer group. + +> Whichever path: this is a **migration-completeness** gap — the root→non-root cutover must carry the +> agent's full privileged-command set (here `lxc-info`, `make-private`) into the sudoers, the same way +> the leaf/token/PBS-key issues were migration carry-over gaps. + +--- + +## 7. Confidence + gaps + +- **Root cause: very high.** Three independent confirmations: (a) guest mountinfo shows both drives; + (b) agent `/disks` reports both `bound_under_parent=false`; (c) `lxc-info` reproducibly fails for the + `felhom-agent` user (sudo-denied + homeless) yet works as root, and the code path + `boundUnderParent→GuestSeesMount→guestInitPID→lxc-info` is direct. +- **Not separately pinned:** the exact controller-side trigger that re-clears `disconnected` between + gate cycles (enroll/re-attach reconcile) — observed in logs but not traced to a single function; it + governs the *cadence* of the flapping, not the root cause. +- **`guest_attached=False`** is also reported for both (a sibling `GuestConfig` path); likely the same + non-root regression family but via `pct config`, not traced here — surfaces in the report but the + gate keys on `BoundUnderParent`, which is the proven culprit. +- **No reproduce performed** (symptom already continuously live); no mounts/config/services changed. + +*No secrets recorded (the per-guest token used for the read-only `/disks` check is stored +out-of-band; never printed).*