felhom.eu/documentation/audits/SPIKE-intermediary-mount-2026-06-15.md

SPIKE — Stable intermediary mount (swap the backing drive underneath)

Date: 2026-06-15 Type: Architecture spike — report-only + live throwaway experiments on felhom-flash (/dev/sdc1). Host: felhom-pve (node demo-felhom), PVE 9.2.2, kernel 6.x, unprivileged LXC guest 9201. Status: findings complete; live propagation proven no-restart both directions; guest 9201 restored to its original healthy config (mp0/1/2/9, 25 containers up). No production code, no version bump. The build spec is gated on the fork picks at the end.

Concept under test (operator's vision): bind a permanent stable host dir into the guest once (never removed until decommission); mount/unmount the real drive UNDERNEATH it host-side so changes reach the guest live — no mp churn, no guest restart, and the bind source never disappears. The last property makes it inherently immune to C1 (the "drive absent at guest boot → guest bricks" failure), because the bind source is the always-present stable dir, not <drive>/felhom-data.

---

One-screen summary

It works, live, both directions, with zero guest restart and zero pct change — provided the permanent bind is set up with shared→slave mount propagation, which PVE/LXC gives us essentially for free. The guest receives the bind as shared,slave; a host mount under the stable dir appears in the running guest instantly, and a host unmount makes it disappear instantly.

The architecture is viable and strictly better than today's per-drive pct set -mpN model on every axis tested: live attach/detach, C1-immunity, fail-closed safety, idmap, confinement, migration, FileBrowser.

Proven live on guest 9201 (all transitions with the guest running, no pct set, no reboot unless noted):

#	Test	Result
Q1	Mount real /dev/sdc1 under the stable dir → guest sees it	✅ live, no restart (both parent-bind and per-drive forms)
Q1	Unmount host-side → guest loses it	✅ live, no restart
Q2	nsenter inject the mount into the unprivileged guest ns	❌ fails ("must be superuser to use mount"); block device absent from guest /dev — not viable
Q3	Guest uid/gid 1000 → host 101000 read/write through the intermediary	✅ no idmap regression (matches today's direct bind)
Q4	Fail-closed when the drive is absent	✅ naturally fail-closed + ro-placeholder gives CAP_DAC_OVERRIDE-proof EROFS
Q5	Reboot guest with the drive absent	✅ boots clean — C1-immune (bind source = stable dir, exists empty)
Q6	Full eject → reconnect round-trip	✅ no restart either direction; reboot mid-eject = no brick
Q7	Decommission / plug-and-enroll fit	parent-bind form makes enroll/plug/eject/reconnect all restart-free
Q8	Migration + FileBrowser fit + current→intermediary migration	unchanged/cleaner; migration sketch below, no data loss / no HDD_PATH break

The one design correction the spike forced: the stable bind must carry only the drive's felhom-data subtree, not the raw whole-drive mount — otherwise the customer's other top-level dirs (dump, images, private, lost+found) would cross into the guest and break today's confinement invariant. Binding …/felhom-data onto the stable path propagates identically and keeps confinement (Q8, proven live).

---

Current model (what we are replacing), confirmed in source

felhom-agent/internal/localapi/guestbind.go — GuestBinder.AttachBind:

src := where + "/" + felhomDataNS                 // host source = <drive>/felhom-data
// ... mkdir + chown 100000:100000 src ...
spec := fmt.Sprintf("%s,mp=%s", src, where)       // e.g. /mnt/felhom-usb/felhom-data,mp=/mnt/felhom-usb
b.run(ctx, "pct", "set", strconv.Itoa(vmid), "-"+mountKey, spec)

Live config of guest 9201 today:

mp1: /mnt/felhom-usb/felhom-data,mp=/mnt/felhom-usb
mp2: /mnt/felhom-flash/felhom-data,mp=/mnt/felhom-flash

Two structural problems this spike addresses:

1. C1 brick. The mp bind source is <drive>/felhom-data. If the drive is absent at guest boot, that path does not exist → the guest fails to start / comes up broken. The whole GuestBindStore + intent-aware ReassertGuestBinds machinery (disks.go:693) exists to paper over the re-provision variant of this; the absent-drive-at-boot variant is the C1 brick the self-heal hook is shipping to patch.
2. No live attach. AttachBind is pct set, which on an unprivileged guest does not activate a new mountpoint live — RebootGuest exists precisely because "the host-side live inject is blocked on an unprivileged guest, so a drive enrolled into a RUNNING guest activates only at the next boot" (guestbind.go:73-76). Every enroll/reconnect today costs a guest reboot (all apps bounce).

The intermediary model dissolves both: the pct bind is set up once (the only reboot, at provision), and thereafter every attach/detach is a pure host-side mount/umount that propagates live.

---

Q1 — PROPAGATION (the make-or-break) — ✅ PASS, both forms

Setup (one-time, host-side)

mkdir -p /mnt/felhom-drives/flash-1
mount --bind /mnt/felhom-drives /mnt/felhom-drives   # make the parent a mountpoint
mount --make-shared /mnt/felhom-drives               # → peer group; sub-mounts propagate

findmnt host-side: /mnt/felhom-drives pve-root[/mnt/felhom-drives] shared.

One-time guest binds (then ONE reboot to activate)

pct set 9201 -mp3 /mnt/felhom-drives,mp=/mnt/felhom-drives            # (b) parent-bind form
pct set 9201 -mp4 /mnt/felhom-drives/flash-1,mp=/mnt/felhom-flash-1   # (a) per-drive form
pct reboot 9201

Critical observation — the guest receives the binds as shared,slave:

# pct exec 9201 -- findmnt -o TARGET,SOURCE,PROPAGATION /mnt/felhom-drives
/mnt/felhom-drives   pve-root[/mnt/felhom-drives]            shared,slave
# pct exec 9201 -- findmnt -o TARGET,SOURCE,PROPAGATION /mnt/felhom-flash-1
/mnt/felhom-flash-1  pve-root[/mnt/felhom-drives/flash-1]    shared,slave

slave = the guest mount receives propagation from the host's shared peer group. This is exactly the host→guest direction we need, and LXC sets it up automatically (the container root is rslave of the host). No custom lxc.mount.entry propagation flags required — a plain pct set bind of a host path that is itself a shared mount comes through as shared,slave.

The live test — guest RUNNING, mount the real drive underneath

echo FLASH-DRIVE-CONTENT > /mnt/felhom-flash/SPIKE-MARKER.txt   # marker on the real drive
mount /dev/sdc1 /mnt/felhom-drives/flash-1                      # ← the only action

Result, immediately, no restart, no pct change:

# (a) per-drive  — guest /mnt/felhom-flash-1
SPIKE-MARKER.txt … FLASH-DRIVE-CONTENT          ✅ visible
# (b) parent     — guest /mnt/felhom-drives/flash-1
SPIKE-MARKER.txt … FLASH-DRIVE-CONTENT          ✅ visible
# guest findmnt shows the /dev/sdc1 sub-mount stacked, shared,slave

Note the per-drive form (a): the drive is mounted at the exact path LXC bound from, and the new sub-mount still propagates into the guest's /mnt/felhom-flash-1. Both forms work.

Unmount → disappears live

umount /mnt/felhom-drives/flash-1      # guest running

→ guest /mnt/felhom-flash-1 and /mnt/felhom-drives/flash-1 are empty again, marker gone, sub-mount gone — live, no restart. Propagation is bidirectional (mount appears, unmount disappears).

Verdict Q1: live propagation works, no restart, for both the parent-bind and per-drive forms. This is the make-or-break and it passed cleanly.

---

Q2 — nsenter FALLBACK — ❌ NOT VIABLE on the unprivileged LXC

If propagation had been flaky, the fallback was to have the agent enter the guest mount namespace and bind the drive directly. Tested three ways against the running guest (init PID from lxc-info -n 9201 -p -H):

nsenter -t <pid> -m -- mount -t ext4 /dev/sdc1 /mnt/felhom-flash-1
   → mount: must be superuser to use mount.        (rc=32)
   (and /dev/sdc1 does not exist in the guest's restricted /dev anyway)

nsenter -t <pid> -m -- mount --bind /mnt/felhom-flash /mnt/felhom-flash-1
   → mount: must be superuser to use mount.        (rc=32)

mount --bind /mnt/felhom-flash /proc/<pid>/root/mnt/felhom-flash-1   # host-ns trick
   → errored; guest then "Permission denied" — did not cleanly inject

Why: the guest's mount namespace is owned by the container's user namespace. Entering only the mount ns (-m) keeps host-root in the init userns, and the kernel refuses the mount as unprivileged-for-that-ns; the block device is not present in the guest /dev; and a host-ns mount on /proc/<pid>/root/... lands in the host's mount tree, not the guest's. This is a known limitation of unprivileged LXC and is precisely why the existing code says host-side live inject "is blocked on an unprivileged guest."

Recommendation: use (a) propagation, not (b) nsenter. Propagation is clean, declarative, and survives reboot (Q5); nsenter does not work here at all. No leaks were left by the attempts.

---

Q3 — UNPRIVILEGED IDMAP — ✅ no regression

With the drive mounted under the stable dir, simulated a deploy-chowned app subdir (chown 101000:101000 host-side = guest uid/gid 1000) and wrote from inside the guest as that mapped identity:

# host:  /mnt/felhom-drives/flash-1/felhom-data/appdata/spiketest  owner 101000:101000
# guest: setpriv --reuid 1000 --regid 1000 -- sh -c 'echo … > …/from-guest.txt'   → WROTE-OK
# host:  from-guest.txt owner 101000:101000                                         ✅
# guest: reads it back via the OTHER bind path (/mnt/felhom-flash-1/…)              ✅ same data

Idmap through the intermediary mount is identical to today's direct bind (felhom-usb = gid 101000). Files written by guest uid 1000 land as host 101000; the felhom-data root owned by host 100000 shows as guest root:root. No idmap regression.

---

Q4 — ABSENT-DRIVE FAIL-CLOSED (the safety core) — ✅ naturally fail-closed; ro-placeholder is the strongest belt

(a) The hazard — does a guest write to the bare path leak onto host root fs?

The stable dir lives on the host root fs (df → /dev/mapper/pve-root). With the drive unmounted, the bare /mnt/felhom-drives/flash-1 is owned by host root (uid 0), which is unmapped in the container idmap (100000–165535) and appears as nobody:nogroup. Result:

guest ROOT   : mkdir /mnt/felhom-flash-1/felhom-data → Permission denied  → ROOT-WRITE-DENIED
guest uid1000: write /mnt/felhom-flash-1/…           → Permission denied  → UID1000-WRITE-DENIED
host check   : find … -type f → (nothing)            nothing leaked onto pve-root

The model is fail-closed by construction. The felhom-data namespace (owned by mapped 100000) lives on the drive; the permanent stable mountpoint is host-root-owned, so when the drive is absent the guest — even as root — cannot create felhom-data under it. The whole leak hazard is naturally closed without any extra step.

(b) chmod 0000 root:root — and the CAP_DAC_OVERRIDE caveat (the crux)

bare dir host-root-owned, chmod 0000 → guest ROOT write  → ROOT-DENIED          ✅

But a contrast test exposes the trap. A dir owned by a mapped id (100000 = guest root), chmod 0000:

guest ROOT write into 0000 dir owned by 100000  → ROOT-BYPASSED-0000   ⚠️ landed on host fs
guest ROOT write into 0000 dir owned by host-0  → ROOT-DENIED          ✅

chmod 0000 is reliable only on a host-root-owned (unmapped) dir. Container-root holds CAP_DAC_OVERRIDE over its userns and bypasses 0000 whenever the file owner is mapped into the namespace. So a chmod-based fail-close must guarantee the bare dir stays host-root-owned — which the natural design already does, but it's a fragile invariant to rely on (one stray chown breaks it).

(c) Read-only placeholder mount — CAP_DAC_OVERRIDE-proof — ✅ EROFS for everyone

mount -t tmpfs -o ro,size=1M,mode=0000 felhom-absent /mnt/felhom-drives/flash-1

Propagates into the guest live (shared). Guest writes are blocked, and — the decisive test — even when the placeholder is deliberately made mapped-owned (100000) and mode 0777, a remount,ro still gives:

guest ROOT write → Read-only file system   → ROOT-DENIED-EROFS   ✅

EROFS is a mount-level restriction; CAP_DAC_OVERRIDE cannot bypass it. This is the ownership-independent, capability-proof fail-close.

(d) Real drive SHADOWS the hardened placeholder — no "restore" step on reconnect — ✅

chmod 0000 /mnt/felhom-drives/flash-1
mount -t tmpfs -o ro,… felhom-absent /mnt/felhom-drives/flash-1   # placeholder
mount /dev/sdc1 /mnt/felhom-drives/flash-1                        # drive ON TOP

Host stack: bare 0000 dir → ro tmpfs → /dev/sdc1 (top). The guest sees /dev/sdc1 rw, marker readable, uid-1000 write OK — no un-harden step needed. On eject (umount the drive), the ro placeholder is re-exposed automatically and the guest is denied again. The fail-close lives permanently underneath; reconnect/eject just stack/unstack the drive on top of it.

Recommendation Q4: use the ro tmpfs placeholder as the primary fail-close (capability-proof, ownership-independent, auto-shadowed by the drive, auto-re-exposed on eject), and keep the stable dir host-root-owned as a second, cheaper layer. App-stop remains the primary safety (the controller stops apps on a drive before eject); this mount-level fail-close is the secondary net for the "drive vanished without a clean eject" race.

---

Q5 — C1 IMMUNITY — ✅ the headline result

With the permanent stable bind in place and the drive absent (ro placeholder in place, nothing real mounted):

# pre-reboot: drive ABSENT, mp3/mp4 still in config (sources = stable dirs, exist empty)
pct reboot 9201      # → real 0m12s, status: running

The guest boots clean in 12 s with the drive absent. Because the mp bind source is the stable dir (which always exists, empty), there is nothing for pct start to fail on. Post-boot:

• the ro placeholder survived into the fresh guest namespace (a bind of the shared parent subtree carries its existing sub-mounts at bind time), so the guest is still fail-closed: guest write → DENIED-failclosed;
• then mount /dev/sdc1 … host-side → guest sees the drive live, no second restart.

This is the direct test that the architecture is inherently C1-immune. Today's model bricks here because its bind source <drive>/felhom-data is gone; the intermediary model does not even notice the drive is missing at boot.

---

Q6 — EJECT / RECONNECT ROUND-TRIP (no restart) — ✅

Full lifecycle on the stable bind, guest running throughout:

Phase	Host action	Guest effect	Restart?
EJECT	(controller stops apps on the drive — simulated) → umount drive → ro placeholder re-exposed	data gone; drive safely removable; writes fail-closed	none
RECONNECT	mount /dev/sdc1 (on top of placeholder) → re-enable apps	data visible; uid-1000 writes OK	none
Reboot mid-eject	reboot with drive absent (Q5)	boots clean, stays fail-closed	reboot, no brick

Eject and reconnect are pure host-side mount/umount. The only restart anywhere in the lifecycle is the one-time provision reboot that activates the permanent pct bind.

---

Q7 — DECOMMISSION + PLUG-AND-ENROLL fit

With the parent-bind form (Q1b) — one permanent pct set -mpN /mnt/felhom-drives,mp=/mnt/felhom-drives per guest, set at provision — the full drive lifecycle becomes restart-free:

Operation	Today (per-drive pct set)	Intermediary (parent-bind)
Enroll	pct set -mpN + reboot	host mount …/felhom-data /mnt/felhom-drives/<name> → live
Plug (raw-drive enroll, see SPIKE-raw-drive-enroll)	format → pct set + reboot	format → host mount under the parent → live
Eject	pct set -delete + reboot	umount → placeholder re-exposed → live
Reconnect	pct set + reboot	host mount → live
Decommission	pct set -delete + reboot + GuestBindStore.Remove	umount + un-enroll + GuestBindStore.Remove; the parent bind stays

The parent-bind is created once and only decommission of the whole guest ever removes it. Per-drive decommission is just "unmount + un-enroll, leave the parent bind" — no mp churn, no reboot. The GuestBindStore/ReassertGuestBinds re-assert machinery shrinks to "for each enrolled-and-present durable-id, ensure its felhom-data is host-mounted under the parent" — a host-side reconcile with no guest config writes and no reboot, which also fixes F9 (re-provision) for free.

The per-drive form (Q1a) also works but needs a pct set + reboot to add each new drive's stable slot — so it keeps the reboot-on-enroll cost. Parent-bind is the form that makes enroll/plug/eject/reconnect all restart-free.

---

Q8 — MIGRATION + FILEBROWSER fit + CURRENT→INTERMEDIARY migration

Confinement (the design correction) — ✅ proven live

The spike first mounted the whole drive under the stable dir for mechanism-testing — which exposed dump/images/private/lost+found to the guest and would break today's confinement invariant. The production shape binds only the felhom-data subtree:

mount /dev/sdc1 /mnt/_raw/<name>                                   # raw drive, host-private
mount --bind /mnt/_raw/<name>/felhom-data /mnt/felhom-drives/<name>   # only felhom-data → stable path

Proven live: guest sees only appdata/backups/media/userdata (the namespace), and ls /mnt/felhom-flash-1/dump → No such file or directory (CONFINED-no-dump-visible). A bind-mount of felhom-data propagates identically to a raw block-device mount.

Migration & FileBrowser — unchanged / cleaner

• Migration (controller/internal/stacks/migrate.go) moves data between stable guest paths (/mnt/<a> → /mnt/<b>); those paths are now rock-stable mountpoints that never change identity, so the migrate→flip→cleanup pipeline is easier (no mp re-point, no reboot to swap a drive in/out under a path).
• FileBrowser (syncFileBrowserMounts) binds <drive>/appdata etc. by stable path; nothing changes — the path is stable by definition. If anything it's safer: when a drive is ejected the path goes fail-closed rather than disappearing.

Current→intermediary migration sketch (the real adoption cost)

Goal: move guest 9201's existing felhom-usb (live apps) + felhom-flash from today's direct mp1/mp2 to the intermediary model without data loss or HDD_PATH break. HDD_PATH is the guest path (/mnt/felhom-usb, /mnt/felhom-flash) — keep it identical so no app/catalog/controller config changes.

Per guest, one maintenance reboot:

1. Pre-stage host-side (no guest impact): create /mnt/felhom-drives, mount --bind self, mount --make-shared. For each enrolled drive currently at /mnt/<name> (where /mnt/<name> is the raw host mount of the drive): mount --bind /mnt/<name>/felhom-data /mnt/felhom-drives/<name>. (felhom-usb and felhom-flash are already host-mounted at /mnt/felhom-usb / /mnt/felhom-flash.)
2. Stop apps (existing quiesce), or accept a short bounce.
3. Swap the guest config — replace the per-drive binds with the intermediary set, keeping the guest target path identical:
   • pct set 9201 -delete mp1 -delete mp2
   • pct set 9201 -mp1 /mnt/felhom-drives,mp=/mnt/felhom-drives (parent bind — one slot for all drives)
   • keep each drive visible at its old HDD_PATH by binding /mnt/felhom-drives/<name> → /mnt/<name> inside the guest at boot, or repoint HDD_PATH to /mnt/felhom-drives/<name>. (Fork: see below — symlink-preserve vs HDD_PATH-repoint.)
4. One reboot to activate the new permanent bind. Guest comes up; the controller's host-side reconcile ensures every enrolled drive's felhom-data is mounted under the parent.
5. Verify: each app's HDD_PATH resolves to the same data; findmnt shows …/felhom-data propagated under the stable path. Then the model is live and all future attach/detach/eject/reconnect is reboot-free.

No bytes move (it's a remount, not a copy), so there is no data-loss window — only the one-time reboot. The only real decision is how to keep HDD_PATH identical (fork below).

---

VERDICT

• Does live propagation work, no restart? Yes — unambiguously. Host→guest mount propagation via shared(host)/slave(guest) is delivered automatically by PVE/LXC; mounting/unmounting the drive under the stable dir appears/disappears in the running guest instantly, both forms, no pct change, no reboot.
• Is the architecture viable? Yes, and it is strictly better than today's per-drive pct set model. It eliminates the reboot-on-enroll/reconnect cost and is inherently C1-immune (Q5: guest boots clean with the drive absent because the bind source is the permanent stable dir).
• Which mechanism? Propagation (a). nsenter (b) does not work on the unprivileged LXC.
• Which fail-close? Read-only tmpfs placeholder (c) — the only CAP_DAC_OVERRIDE-proof, ownership-independent option — backed by keeping the stable dir host-root-owned as a cheap second layer. chmod 0000 alone (b) is unsafe if the dir is ever mapped-owned. App-stop stays the primary safety.
• Which bind form? Parent-bind (Q1b) — makes enroll/plug/eject/reconnect all restart-free with a single permanent mp per guest.

The self-heal hook shipping separately remains the correct immediate C1 net for the current model; this spike establishes the bigger architecture that makes C1 a non-event by design.

FORK LIST (decisions for the build spec)

1. Bind granularity: parent-bind (/mnt/felhom-drives → guest, one slot, fully restart-free) ✅ recommended — vs per-drive-dir (/mnt/felhom-drives/<name> → guest, needs pct set+reboot per new drive). Parent-bind is the whole point; per-drive keeps the reboot cost.
2. Inject mechanism: shared/slave propagation ✅ recommended — vs nsenter (proven non-viable here).
3. Fail-close: ro tmpfs placeholder ✅ recommended (capability-proof) — vs chmod 0000 (only safe on host-root-owned dir; fragile). Likely both (placeholder primary, host-root-ownership second layer).
4. HDD_PATH preservation on migration: keep the guest path identical via an in-guest bind/symlink /mnt/felhom-drives/<name> → /mnt/<name> (zero app/catalog change) — vs repoint HDD_PATH to /mnt/felhom-drives/<name> (cleaner long-term, but touches controller/catalog/app config). Decide before the migration spec.
5. What the agent mounts under the stable path: bind of <raw>/felhom-data ✅ required for confinement (proven) — not the raw whole-drive mount. Implies the agent gains a host-private raw mountpoint (/mnt/_raw/<name> or similar) plus the felhom-data bind onto the stable path.

---

Restore / end-state

Guest 9201 returned to its original healthy config and verified:

mp0: local-lvm:vm-9201-disk-1,mp=/var/lib/docker,backup=1,size=256G
mp1: /mnt/felhom-usb/felhom-data,mp=/mnt/felhom-usb     (sdb1[/felhom-data], rw)
mp2: /mnt/felhom-flash/felhom-data,mp=/mnt/felhom-flash (sdc1[/felhom-data], rw)
mp9: …/bootstrap,mp=/etc/felhom-bootstrap,ro=1
→ 25 containers running (filebrowser/felhom-controller/immich/paperless/romm/jellyfin/… healthy)

All experimental mp3/mp4 removed; host /mnt/felhom-drives (+ flash-1, mapped-test, placeholder tmpfs) unmounted and deleted; test artifacts (SPIKE-MARKER.txt, felhom-data/appdata/spiketest) removed from felhom-flash. nsenter attempts left no leaked mounts. No production code changed; no version bump. felhom-usb (8 live apps) and /dev/sda/felhom-pbs/local/local-lvm were never touched.