admin
6b1c2892d6
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
223 lines
14 KiB
Markdown
223 lines
14 KiB
Markdown
# SPIKE — 2nd external drive "drops" the 1st (multi-drive mutual exclusion / flapping)
|
|
|
|
- **Date:** 2026-06-29
|
|
- **Class:** Diagnostic / read-only. **No fix.** No deliberate reproduce was needed — the symptom is
|
|
**continuously live** in the controller gate logs, so the host-vs-guest `mountinfo` diff was read
|
|
directly from the running state (an attach would only re-stop apps, adding no evidence).
|
|
- **Host:** felhom-pve (`demo-felhom`, 192.168.0.162); guest LXC 9201 (init pid 2137736).
|
|
- **Live versions:** agent **0.43.0** (NON-ROOT `felhom-agent` since 2026-06-28), controller **0.89.0**.
|
|
- **Verdict:** **Detection bug (matrix row 3), NOT a propagation fault and NOT a controller-gate
|
|
bug.** The guest demonstrably keeps **both** drive submounts; the agent mis-reports
|
|
`bound_under_parent=false` for **every** drive because the non-root agent cannot resolve the guest
|
|
init PID (`lxc-info` is denied / unusable under the `felhom-agent` user). The controller gate is the
|
|
**messenger** — it correctly stops apps for a drive its report says is absent. Root cause is the
|
|
**2026-06-28 root→non-root agent migration** (same regression family as the leaf-pin + PBS-key
|
|
issues): `lxc-info` was never added to the agent sudoers allowlist.
|
|
|
|
---
|
|
|
|
## 1. The question (§3) — answered
|
|
|
|
> Does the guest actually lose drive A's submount (propagation fault), or does the guest keep it
|
|
> while the agent mis-reports `BoundUnderParent=false` (detection bug)?
|
|
|
|
**The guest keeps BOTH submounts; the agent mis-reports.** Propagation is intact (the guest's parent
|
|
bind is `master:1`, a live slave of the host's `shared:1` parent, and both drives are visible in the
|
|
guest namespace right now). The fault is purely in the agent's *guest-sees* detection.
|
|
|
|
---
|
|
|
|
## 2. Raw evidence
|
|
|
|
### A1/A2 — host parent + submounts (both drives bound, propagation `shared`)
|
|
```
|
|
findmnt /mnt/felhom-drives:
|
|
/mnt/felhom-drives /dev/mapper/pve-root[/mnt/felhom-drives] shared
|
|
├─/mnt/felhom-drives/felhom-flash /dev/sdc1[/felhom-data] shared
|
|
└─/mnt/felhom-drives/felhom-usb /dev/sdb1[/felhom-data] shared
|
|
|
|
/proc/self/mountinfo (host):
|
|
137 33 252:1 /mnt/felhom-drives /mnt/felhom-drives shared:1
|
|
99 137 8:33 /felhom-data /mnt/felhom-drives/felhom-flash shared:51 ← child of parent 137
|
|
100 33 8:33 /felhom-data /mnt/felhom-drives/felhom-flash shared:51 ← ALSO child of root 33 (doubling)
|
|
109 137 8:17 /felhom-data /mnt/felhom-drives/felhom-usb shared:49
|
|
110 33 8:17 /felhom-data /mnt/felhom-drives/felhom-usb shared:49
|
|
```
|
|
|
|
### A3 — GUEST view (the `BoundUnderParent` truth) — **both drives present**
|
|
```
|
|
guest_pid=2137736
|
|
571 324 252:1 /mnt/felhom-drives /mnt/felhom-drives shared:482 master:1 ← slave of host group 1 (propagation LIVE)
|
|
108 571 8:33 /felhom-data /mnt/felhom-drives/felhom-flash shared:195 master:51
|
|
344 571 8:17 /felhom-data /mnt/felhom-drives/felhom-usb shared:210 master:49
|
|
(+ duplicate entries 572/111/345 via the guest's second parent path — see §4 doubling)
|
|
```
|
|
The guest **sees both** `felhom-flash` and `felhom-usb` as mountpoints under the parent.
|
|
|
|
### Agent `/disks` report — **both reported absent** (the contradiction)
|
|
```
|
|
felhom-flash guest_path=/mnt/felhom-drives/felhom-flash bound_under_parent=False guest_attached=False
|
|
felhom-usb guest_path=/mnt/felhom-drives/felhom-usb bound_under_parent=False guest_attached=False
|
|
```
|
|
Guest mountinfo says present; agent says `bound_under_parent=False`. **The report is wrong.**
|
|
|
|
### The cause — `lxc-info` fails under the non-root agent
|
|
```
|
|
# as root (what the agent did PRE-migration): works
|
|
lxc-info -n 9201 -p -H → 2137736
|
|
|
|
# as the felhom-agent user, directly: FAILS (no usable $HOME)
|
|
sudo -u felhom-agent lxc-info -n 9201 -p -H
|
|
→ lxc-info: Permission denied - Failed to create directory "/home/felhom-agent/"
|
|
|
|
# as the felhom-agent user, via sudo (the runner's path): DENIED (not in NOPASSWD allowlist)
|
|
sudo -u felhom-agent sudo -n lxc-info -n 9201 -p -H
|
|
→ sudo: a password is required
|
|
|
|
# NB: the non-root user CAN read /proc/<pid>/mountinfo (exit 0, sees both drives) —
|
|
# the ONLY broken step is resolving the guest PID via lxc-info.
|
|
```
|
|
|
|
### Controller gate — the live symptom (every cycle, both drives)
|
|
```
|
|
[gate] drive ABSENT /mnt/felhom-drives/felhom-usb — stopped+blocked 1 app(s): [nextcloud]
|
|
[gate] drive ABSENT /mnt/felhom-drives/felhom-flash — stopped+blocked 0 app(s): []
|
|
[stacks] Stopping stack: nextcloud (repeats ~every minute)
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Causal chain (code-verified)
|
|
|
|
1. `boundUnderParent()` (`felhom-agent/internal/localapi/disks.go:703`) →
|
|
`guestAttach.GuestSeesMount(ctx, vmid, stablePath)`.
|
|
2. `GuestSeesMount` (`intermediary.go:236`) calls `guestInitPID` →
|
|
`b.runner.Run(ctx, "lxc-info", "-n", vmid, "-p", "-H")` (`intermediary.go:256`).
|
|
3. The agent runs **non-root** with `privileged.mode: sudo`; the runner shells every command via
|
|
`sudo -n`. `lxc-info` is **absent from `configs/felhom-agent.sudoers`** (line 93 grants only
|
|
`FELHOM_MOUNT/DISK/PROVISION/FORMAT/DNSMASQ/GUESTHOOK/INTERMEDIARY` — none lists `lxc-info`), so
|
|
`sudo -n lxc-info` → "a password is required" → error. (Run un-sudo'd it also fails: the
|
|
`felhom-agent` user has no writable `$HOME` for lxc-info's lock dir.)
|
|
4. `guestInitPID` returns `""` → `GuestSeesMount` returns `false` (pid empty) for **every** drive →
|
|
`bound_under_parent=false` for all.
|
|
5. Controller `planDriveGates` (`felhom-controller/.../web/intermediary.go:202`) sets
|
|
`present[guestPath] = … || d.BoundUnderParent` → `false`; the gate (line 224)
|
|
`!present && !Disconnected → Stop apps + SetDisconnected(true)`. Every external drive is judged
|
|
ABSENT and its apps stopped. **The gate is correct given its input** — do NOT "fix" it (the task's
|
|
own warning: it is a real data-availability signal, here fed a false reading).
|
|
|
|
**Why it looks like "attaching B drops A" (mutual exclusion):** with `bound_under_parent` permanently
|
|
false, the gate marks *both* drives ABSENT on every cycle. An attach/enroll transiently re-binds and
|
|
clears the just-touched drive's `disconnected` flag (so it briefly shows Aktív), while the gate
|
|
continues to stop the other drive's apps — and re-stops the first on the next cycle. The net effect is
|
|
perpetual flapping (nextcloud stopped ~every minute); the user perceives the most-recently-attached
|
|
drive as "the active one" and the other as having "dropped." It is not single-active by design — it is
|
|
both-absent by mis-detection.
|
|
|
|
---
|
|
|
|
## 4. Secondary observation (latent, NOT the cause) — host-side "doubling"
|
|
|
|
Each drive submount appears **twice** on the host: once under the parent (mount 137) and once under
|
|
root (mount 33), because the parent `/mnt/felhom-drives` carries `shared:1` — **the same peer group as
|
|
`/`** (root id 33 is also `shared:1`). It was never isolated into its own group. Cause:
|
|
`EnsureSharedParent` calls `mount --make-private` (`intermediary.go:110`), but **`make-private` is NOT
|
|
in the sudoers** (line 85 allows only `mount --make-shared /mnt/felhom-drives`, not `--make-private`),
|
|
so the agent's isolation step is denied. (The root-run boot script `felhom-shared-parent.sh` does run
|
|
make-private, but the live parent is currently in root's group regardless — worth confirming at the
|
|
next host boot.) This doubling makes `AttachDrive` always see `countHostMounts==2` and take the
|
|
umount-all→rebind normalize branch, and propagates binds/umounts into root's namespace. It does **not**
|
|
cause the reported symptom (the guest still sees both drives), but it is a real correctness smell from
|
|
the same non-root-allowlist gap and should be fixed alongside.
|
|
|
|
---
|
|
|
|
## 5. Interpretation matrix — matched row
|
|
|
|
| Observation | Root cause |
|
|
|---|---|
|
|
| **Host keeps both submounts, guest keeps both, but agent reports `BoundUnderParent=false`** | **Detection bug** in `GuestSeesMount`/`boundUnderParent` — specifically **PID resolution**: `lxc-info` denied/unusable under the non-root agent (sudoers gap + homeless user) → `guestInitPID=""` |
|
|
|
|
Rows ruled out: no propagation churn (parent is a stable `shared:1` mountpoint, guest is `master:1`,
|
|
peer groups did not change); guest parent is **not** `private` (it is a live slave); host did **not**
|
|
drop any submount.
|
|
|
|
---
|
|
|
|
## 6. Remedy options for the operator (NOT executed)
|
|
|
|
1. **Grant `lxc-info` in the agent sudoers** (minimal, restores pre-migration behavior). Add a fixed
|
|
vector e.g. `/usr/bin/lxc-info -n [0-9]* -p -H` to a `Cmnd_Alias` and the NOPASSWD line, `visudo
|
|
-cf`, reload. Verified `sudo lxc-info -n 9201 -p -H` works as root → returns `2137736`. Trade-off:
|
|
widens the host-root surface by one read-only command (low risk; read-only, fixed args).
|
|
2. **Remove the `lxc-info` dependency in `guestInitPID`** (code fix, no new privilege): resolve the
|
|
guest init PID without lxc-info — e.g. read it from the LXC cgroup / `pct` pidfile / a `/proc` scan
|
|
the non-root user can already do (it *can* read `/proc/<pid>/mountinfo`; only PID *discovery* is the
|
|
gap). Cleaner long-term; keeps the allowlist tight.
|
|
3. **Give `felhom-agent` a writable `$HOME`** so direct `lxc-info` works — weakest option (lxc-info on
|
|
system containers still wants privilege for `/run/lxc` locks; #1 or #2 are the real fixes).
|
|
|
|
**Also fix alongside (the §4 doubling):** add `mount --make-private /mnt/felhom-drives` to the sudoers
|
|
`FELHOM_INTERMEDIARY` alias so `EnsureSharedParent` can isolate the parent's peer group.
|
|
|
|
> Whichever path: this is a **migration-completeness** gap — the root→non-root cutover must carry the
|
|
> agent's full privileged-command set (here `lxc-info`, `make-private`) into the sudoers, the same way
|
|
> the leaf/token/PBS-key issues were migration carry-over gaps.
|
|
|
|
---
|
|
|
|
## 7. Confidence + gaps
|
|
|
|
- **Root cause: very high.** Three independent confirmations: (a) guest mountinfo shows both drives;
|
|
(b) agent `/disks` reports both `bound_under_parent=false`; (c) `lxc-info` reproducibly fails for the
|
|
`felhom-agent` user (sudo-denied + homeless) yet works as root, and the code path
|
|
`boundUnderParent→GuestSeesMount→guestInitPID→lxc-info` is direct.
|
|
- **Not separately pinned:** the exact controller-side trigger that re-clears `disconnected` between
|
|
gate cycles (enroll/re-attach reconcile) — observed in logs but not traced to a single function; it
|
|
governs the *cadence* of the flapping, not the root cause.
|
|
- **`guest_attached=False`** is also reported for both (a sibling `GuestConfig` path); likely the same
|
|
non-root regression family but via `pct config`, not traced here — surfaces in the report but the
|
|
gate keys on `BoundUnderParent`, which is the proven culprit.
|
|
- **No reproduce performed** (symptom already continuously live); no mounts/config/services changed.
|
|
|
|
*No secrets recorded (the per-guest token used for the read-only `/disks` check is stored
|
|
out-of-band; never printed).*
|
|
|
|
---
|
|
|
|
## 8. Outcome — FIXED 2026-06-29 (supervised sudoers completeness pass)
|
|
|
|
A full audit of every privileged command vs the allowlist (felhom-agent `55888e5`; full table in
|
|
`felhom-agent/REPORT.md`) closed the gaps. **Sudoers-only, no binary change, no agent restart.**
|
|
|
|
Grants added: `lxc-info -n [0-9]* -p -H` (the root-cause fix) + `mount --make-private
|
|
/mnt/felhom-drives` → FELHOM_INTERMEDIARY; `systemctl restart dnsmasq` → FELHOM_DNSMASQ; `pct set
|
|
[0-9]* -onboot 1` → FELHOM_PROVISION; `pct reboot [0-9]*` → FELHOM_GUESTHOOK. Deployed via the
|
|
install-script mechanism (`visudo -cf` → `install -m 0440 -o root -g root` → re-validate `/etc/sudoers`).
|
|
|
|
**Validation (all pass):** `sudo -n lxc-info -n 9201 -p -H` now returns PID `2137736`; agent `/disks`
|
|
reports `bound_under_parent=true` for **both** felhom-usb + felhom-flash; the controller gate logged
|
|
`drive RETURNED … re-attached + restarted` for both and **nextcloud auto-restarted** (held healthy,
|
|
zero `drive ABSENT` after); the real UI (`/settings`) shows **both drives Aktív concurrently**. Symptom
|
|
gone.
|
|
|
|
**Surfaced, then RESOLVED (2026-06-29, agent v0.45.0):** `GuestExec`'s general `pct exec <vmid> -- …`
|
|
(controller-swap / managed auto-update) needed arbitrary root-in-guest. Rather than grant it,
|
|
controller-swap's only shell vector (`writeImage`'s `bash -c "printf … >"`) was replaced by a
|
|
stdin-fed `tee`, making all 5 swap vectors fixed/read-only and allowlistable as `FELHOM_CONTROLLERSWAP`
|
|
(no `bash -c`, no general `pct exec`). Spike-proven (`SPIKE-controllerswap-narrow-grants-2026-06-29.md`)
|
|
+ shipped; controller-swap works under the non-root agent again (live: 45/45 capability ok, negative
|
|
controls denied). **Deferred:** `sensors -j`
|
|
(no caller + lm-sensors not installed). **Left as-is:** the §4 host-side doubling (re-isolating the
|
|
live parent would orphan the working guest slave — fixed grant takes effect on the next clean host
|
|
boot). The §7 "guest_attached / pct config" note is **resolved**: `GuestConfig` is a Proxmox **API**
|
|
call, not a sudo CLI command, so `guest_attached=false` is not a sudoers gap (and is benign in the
|
|
intermediary model).
|
|
|
|
**Durable countermeasure SHIPPED (2026-06-29):** the agent now self-probes its privileged
|
|
capabilities — felhom-agent **v0.44.0** `internal/capability` (build-time manifest⊆sudoers test +
|
|
runtime `sudo -n -l` probe → `HostReport.Capabilities`) + hub **v0.19.0** `HostCapabilityChecker`
|
|
(operator alert on a Critical grant going degraded). This exact incident's missing `lxc-info` grant
|
|
is now a CI build failure and a live `capabilities self-check` degraded signal, not a silent
|
|
multi-day regression. Live: agent reports 40/40 ok. See `felhom-agent/REPORT.md`.
|