admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
259f48aedc05e38c00a67674357efbce6e692e2f
felhom.eu/documentation/audits/SPIKE-multidrive-mutual-exclusion-2026-06-29.md
T

14 KiB
Raw Blame History

SPIKE — 2nd external drive "drops" the 1st (multi-drive mutual exclusion / flapping)

  • Date: 2026-06-29
  • Class: Diagnostic / read-only. No fix. No deliberate reproduce was needed — the symptom is continuously live in the controller gate logs, so the host-vs-guest mountinfo diff was read directly from the running state (an attach would only re-stop apps, adding no evidence).
  • Host: felhom-pve (demo-felhom, 192.168.0.162); guest LXC 9201 (init pid 2137736).
  • Live versions: agent 0.43.0 (NON-ROOT felhom-agent since 2026-06-28), controller 0.89.0.
  • Verdict: Detection bug (matrix row 3), NOT a propagation fault and NOT a controller-gate bug. The guest demonstrably keeps both drive submounts; the agent mis-reports bound_under_parent=false for every drive because the non-root agent cannot resolve the guest init PID (lxc-info is denied / unusable under the felhom-agent user). The controller gate is the messenger — it correctly stops apps for a drive its report says is absent. Root cause is the 2026-06-28 root→non-root agent migration (same regression family as the leaf-pin + PBS-key issues): lxc-info was never added to the agent sudoers allowlist.

1. The question (§3) — answered

Does the guest actually lose drive A's submount (propagation fault), or does the guest keep it while the agent mis-reports BoundUnderParent=false (detection bug)?

The guest keeps BOTH submounts; the agent mis-reports. Propagation is intact (the guest's parent bind is master:1, a live slave of the host's shared:1 parent, and both drives are visible in the guest namespace right now). The fault is purely in the agent's guest-sees detection.

2. Raw evidence

A1/A2 — host parent + submounts (both drives bound, propagation shared)

findmnt /mnt/felhom-drives:
  /mnt/felhom-drives                /dev/mapper/pve-root[/mnt/felhom-drives] shared
  ├─/mnt/felhom-drives/felhom-flash /dev/sdc1[/felhom-data]                  shared
  └─/mnt/felhom-drives/felhom-usb   /dev/sdb1[/felhom-data]                  shared

/proc/self/mountinfo (host):
  137 33  252:1 /mnt/felhom-drives        /mnt/felhom-drives         shared:1
   99 137 8:33  /felhom-data /mnt/felhom-drives/felhom-flash shared:51   ← child of parent 137
  100 33  8:33  /felhom-data /mnt/felhom-drives/felhom-flash shared:51   ← ALSO child of root 33 (doubling)
  109 137 8:17  /felhom-data /mnt/felhom-drives/felhom-usb   shared:49
  110 33  8:17  /felhom-data /mnt/felhom-drives/felhom-usb   shared:49

A3 — GUEST view (the BoundUnderParent truth) — both drives present

guest_pid=2137736
571 324 252:1 /mnt/felhom-drives                /mnt/felhom-drives         shared:482 master:1   ← slave of host group 1 (propagation LIVE)
108 571 8:33  /felhom-data /mnt/felhom-drives/felhom-flash shared:195 master:51
344 571 8:17  /felhom-data /mnt/felhom-drives/felhom-usb   shared:210 master:49
   (+ duplicate entries 572/111/345 via the guest's second parent path — see §4 doubling)

The guest sees both felhom-flash and felhom-usb as mountpoints under the parent.

Agent /disks report — both reported absent (the contradiction)

felhom-flash  guest_path=/mnt/felhom-drives/felhom-flash  bound_under_parent=False  guest_attached=False
felhom-usb    guest_path=/mnt/felhom-drives/felhom-usb    bound_under_parent=False  guest_attached=False

Guest mountinfo says present; agent says bound_under_parent=False. The report is wrong.

The cause — lxc-info fails under the non-root agent

# as root (what the agent did PRE-migration):           works
  lxc-info -n 9201 -p -H            → 2137736

# as the felhom-agent user, directly:                    FAILS (no usable $HOME)
  sudo -u felhom-agent lxc-info -n 9201 -p -H
    → lxc-info: Permission denied - Failed to create directory "/home/felhom-agent/"

# as the felhom-agent user, via sudo (the runner's path): DENIED (not in NOPASSWD allowlist)
  sudo -u felhom-agent sudo -n lxc-info -n 9201 -p -H
    → sudo: a password is required

# NB: the non-root user CAN read /proc/<pid>/mountinfo (exit 0, sees both drives) —
#     the ONLY broken step is resolving the guest PID via lxc-info.

Controller gate — the live symptom (every cycle, both drives)

[gate] drive ABSENT /mnt/felhom-drives/felhom-usb  — stopped+blocked 1 app(s): [nextcloud]
[gate] drive ABSENT /mnt/felhom-drives/felhom-flash — stopped+blocked 0 app(s): []
[stacks] Stopping stack: nextcloud   (repeats ~every minute)

3. Causal chain (code-verified)

  1. boundUnderParent() (felhom-agent/internal/localapi/disks.go:703) → guestAttach.GuestSeesMount(ctx, vmid, stablePath).
  2. GuestSeesMount (intermediary.go:236) calls guestInitPID → b.runner.Run(ctx, "lxc-info", "-n", vmid, "-p", "-H") (intermediary.go:256).
  3. The agent runs non-root with privileged.mode: sudo; the runner shells every command via sudo -n. lxc-info is absent from configs/felhom-agent.sudoers (line 93 grants only FELHOM_MOUNT/DISK/PROVISION/FORMAT/DNSMASQ/GUESTHOOK/INTERMEDIARY — none lists lxc-info), so sudo -n lxc-info → "a password is required" → error. (Run un-sudo'd it also fails: the felhom-agent user has no writable $HOME for lxc-info's lock dir.)
  4. guestInitPID returns "" → GuestSeesMount returns false (pid empty) for every drive → bound_under_parent=false for all.
  5. Controller planDriveGates (felhom-controller/.../web/intermediary.go:202) sets present[guestPath] = … || d.BoundUnderParent → false; the gate (line 224) !present && !Disconnected → Stop apps + SetDisconnected(true). Every external drive is judged ABSENT and its apps stopped. The gate is correct given its input — do NOT "fix" it (the task's own warning: it is a real data-availability signal, here fed a false reading).

Why it looks like "attaching B drops A" (mutual exclusion): with bound_under_parent permanently false, the gate marks both drives ABSENT on every cycle. An attach/enroll transiently re-binds and clears the just-touched drive's disconnected flag (so it briefly shows Aktív), while the gate continues to stop the other drive's apps — and re-stops the first on the next cycle. The net effect is perpetual flapping (nextcloud stopped ~every minute); the user perceives the most-recently-attached drive as "the active one" and the other as having "dropped." It is not single-active by design — it is both-absent by mis-detection.

4. Secondary observation (latent, NOT the cause) — host-side "doubling"

Each drive submount appears twice on the host: once under the parent (mount 137) and once under root (mount 33), because the parent /mnt/felhom-drives carries shared:1 — the same peer group as / (root id 33 is also shared:1). It was never isolated into its own group. Cause: EnsureSharedParent calls mount --make-private (intermediary.go:110), but make-private is NOT in the sudoers (line 85 allows only mount --make-shared /mnt/felhom-drives, not --make-private), so the agent's isolation step is denied. (The root-run boot script felhom-shared-parent.sh does run make-private, but the live parent is currently in root's group regardless — worth confirming at the next host boot.) This doubling makes AttachDrive always see countHostMounts==2 and take the umount-all→rebind normalize branch, and propagates binds/umounts into root's namespace. It does not cause the reported symptom (the guest still sees both drives), but it is a real correctness smell from the same non-root-allowlist gap and should be fixed alongside.

5. Interpretation matrix — matched row

Observation Root cause
Host keeps both submounts, guest keeps both, but agent reports BoundUnderParent=false Detection bug in GuestSeesMount/boundUnderParent — specifically PID resolution: lxc-info denied/unusable under the non-root agent (sudoers gap + homeless user) → guestInitPID=""

Rows ruled out: no propagation churn (parent is a stable shared:1 mountpoint, guest is master:1, peer groups did not change); guest parent is not private (it is a live slave); host did not drop any submount.

6. Remedy options for the operator (NOT executed)

  1. Grant lxc-info in the agent sudoers (minimal, restores pre-migration behavior). Add a fixed vector e.g. /usr/bin/lxc-info -n [0-9]* -p -H to a Cmnd_Alias and the NOPASSWD line, visudo -cf, reload. Verified sudo lxc-info -n 9201 -p -H works as root → returns 2137736. Trade-off: widens the host-root surface by one read-only command (low risk; read-only, fixed args).
  2. Remove the lxc-info dependency in guestInitPID (code fix, no new privilege): resolve the guest init PID without lxc-info — e.g. read it from the LXC cgroup / pct pidfile / a /proc scan the non-root user can already do (it can read /proc/<pid>/mountinfo; only PID discovery is the gap). Cleaner long-term; keeps the allowlist tight.
  3. Give felhom-agent a writable $HOME so direct lxc-info works — weakest option (lxc-info on system containers still wants privilege for /run/lxc locks; #1 or #2 are the real fixes).

Also fix alongside (the §4 doubling): add mount --make-private /mnt/felhom-drives to the sudoers FELHOM_INTERMEDIARY alias so EnsureSharedParent can isolate the parent's peer group.

Whichever path: this is a migration-completeness gap — the root→non-root cutover must carry the agent's full privileged-command set (here lxc-info, make-private) into the sudoers, the same way the leaf/token/PBS-key issues were migration carry-over gaps.

7. Confidence + gaps

  • Root cause: very high. Three independent confirmations: (a) guest mountinfo shows both drives; (b) agent /disks reports both bound_under_parent=false; (c) lxc-info reproducibly fails for the felhom-agent user (sudo-denied + homeless) yet works as root, and the code path boundUnderParent→GuestSeesMount→guestInitPID→lxc-info is direct.
  • Not separately pinned: the exact controller-side trigger that re-clears disconnected between gate cycles (enroll/re-attach reconcile) — observed in logs but not traced to a single function; it governs the cadence of the flapping, not the root cause.
  • guest_attached=False is also reported for both (a sibling GuestConfig path); likely the same non-root regression family but via pct config, not traced here — surfaces in the report but the gate keys on BoundUnderParent, which is the proven culprit.
  • No reproduce performed (symptom already continuously live); no mounts/config/services changed.

No secrets recorded (the per-guest token used for the read-only /disks check is stored out-of-band; never printed).

8. Outcome — FIXED 2026-06-29 (supervised sudoers completeness pass)

A full audit of every privileged command vs the allowlist (felhom-agent 55888e5; full table in felhom-agent/REPORT.md) closed the gaps. Sudoers-only, no binary change, no agent restart.

Grants added: lxc-info -n [0-9]* -p -H (the root-cause fix) + mount --make-private /mnt/felhom-drives → FELHOM_INTERMEDIARY; systemctl restart dnsmasq → FELHOM_DNSMASQ; pct set [0-9]* -onboot 1 → FELHOM_PROVISION; pct reboot [0-9]* → FELHOM_GUESTHOOK. Deployed via the install-script mechanism (visudo -cf → install -m 0440 -o root -g root → re-validate /etc/sudoers).

Validation (all pass): sudo -n lxc-info -n 9201 -p -H now returns PID 2137736; agent /disks reports bound_under_parent=true for both felhom-usb + felhom-flash; the controller gate logged drive RETURNED … re-attached + restarted for both and nextcloud auto-restarted (held healthy, zero drive ABSENT after); the real UI (/settings) shows both drives Aktív concurrently. Symptom gone.

Surfaced, then RESOLVED (2026-06-29, agent v0.45.0): GuestExec's general pct exec <vmid> -- … (controller-swap / managed auto-update) needed arbitrary root-in-guest. Rather than grant it, controller-swap's only shell vector (writeImage's bash -c "printf … >") was replaced by a stdin-fed tee, making all 5 swap vectors fixed/read-only and allowlistable as FELHOM_CONTROLLERSWAP (no bash -c, no general pct exec). Spike-proven (SPIKE-controllerswap-narrow-grants-2026-06-29.md)

  • shipped; controller-swap works under the non-root agent again (live: 45/45 capability ok, negative controls denied). Deferred: sensors -j (no caller + lm-sensors not installed). Left as-is: the §4 host-side doubling (re-isolating the live parent would orphan the working guest slave — fixed grant takes effect on the next clean host boot). The §7 "guest_attached / pct config" note is resolved: GuestConfig is a Proxmox API call, not a sudo CLI command, so guest_attached=false is not a sudoers gap (and is benign in the intermediary model).

Durable countermeasure SHIPPED (2026-06-29): the agent now self-probes its privileged capabilities — felhom-agent v0.44.0 internal/capability (build-time manifest⊆sudoers test + runtime sudo -n -l probe → HostReport.Capabilities) + hub v0.19.0 HostCapabilityChecker (operator alert on a Critical grant going degraded). This exact incident's missing lxc-info grant is now a CI build failure and a live capabilities self-check degraded signal, not a silent multi-day regression. Live: agent reports 40/40 ok. See felhom-agent/REPORT.md.

Reference in New Issue View Git Blame Copy Permalink