felhom-agent/REPORT.md

REPORT — felhom-agent v0.37.0 (host-reboot remount by durable-id/UUID)

TASK — Close out the reboot story (agent half: Task A + Task A2).

Task A — host-reboot remount resolves by filesystem UUID, never a cached node

Diagnose-first (root cause, pinned LIVE on felhom-pve)

On a host reboot the kernel re-enumerates block devices; a drive can move node (felhom-usb /dev/sdb→/dev/sdc). Two things could leave an enrolled drive unmounted after a reboot:

1. A systemd .mount unit left disabled by a prior detach (Unmount = stop + disable) never gets a multi-user.target.wants symlink, so it does not auto-mount at boot — and nothing in the agent re-asserted it at startup.
2. Any remount that trusted a remembered /dev/sdX node would target the WRONG device after a letter move.

Confirmed live: felhom-usb's mount unit was disabled (systemctl is-enabled → disabled, no wants-symlink) while felhom-flash's was enabled. The What= line was already correct (/dev/disk/by-uuid/<UUID>), so the fix is to re-assert the unit by UUID at startup, not to rewrite the unit.

Fix

• storage.ResolveStorageDevice(durableID) (durable_device.go) — resolves the enrolled uuid:<fs-uuid> storage scheme to its current backing /dev node by re-scanning /dev/disk/by-uuid (EvalSymlinks); rejects any non-uuid: scheme (anti-node-binding) and errors when the UUID is absent so a caller skips a gone drive instead of fail-mounting.
• storage.parseFelhomMountUnit (mountunit.go) — pure inverse of renderMountUnit, gated on a Managed by felhom-agent marker; extracts Name/UUID/Where/Type/Options, ignores foreign units.
• (*SudoHostOps).ReassertEnrolledMounts(ctx) (hostops.go) — for each enrolled .mount unit, re-resolve by UUID and re-run EnsureMount (idempotent systemctl enable --now). Skips ONLY the durable steady state (mounted AND enabled), via the pure shouldReassertMount; a mounted-but-DISABLED unit (the exact live felhom-usb state — serves now, but a reboot would not auto-mount it) is still re-asserted so the wants-symlink is re-created. Enabled-state read with a privilege-free os.Lstat of the multi-user.target.wants symlink (unitEnabled) — no systemctl is-enabled subprocess / no new sudoers entry. Absent UUIDs skipped (retry next tick).
• main.go — runs ReassertEnrolledMounts at startup before ReassertGuestBinds, and again ahead of the bind re-assert on the periodic 20s tick, so host mounts are live before the guest binds re-establish.

Tests (non-hollow, seam = device resolution; Linux)

• TestResolveStorageDevice_ToleratesDeviceLetterMove — fake /dev/disk/by-uuid/<uuid> symlink pointed at sdc (moved from the enroll-time sdb); asserts ResolveStorageDevice returns the CURRENT node sdc. Companion (red-proof): asserts the cached enroll-time node (sdb) differs from the freshly-resolved node — a node-trusting remount would mount the wrong device.
• TestResolveStorageDevice_AbsentAndScheme — absent UUID errors; only uuid: is resolvable (/dev/sdb1, store:, byid:, "" all rejected).
• TestParseFelhomMountUnit — renderMountUnit→parseFelhomMountUnit round-trip + rejects a non-felhom unit.
• TestShouldReassertMount — the four mounted/enabled combinations; pins that mounted-but-disabled (the live bug) re-asserts and only mounted+enabled is skipped.
• TestUnitEnabled — wants-symlink presence ⇒ enabled (privilege-free Lstat).

Task A2 — verdict: enrolling a NEW drive does NOT need an LXC restart

NO. Source-verified the enroll path: /disks/guest-attach → handleDiskGuestAttach → GuestBinder.AttachDrive (intermediary-mount model, "no pct, no reboot") — a live bind under the single shared parent /mnt/felhom-drives, which provides unbounded named live slots. It does NOT take the legacy RebootGuest branch. The operator's pre-created-slot-pool idea is unnecessary.

Gate

go build ./..., go vet ./..., go test ./... all green locally (Windows: the symlink tests skip — they assert on Linux, the agent's OS). Linux green gate + companion red-proof run on the build server (192.168.0.180, go1.26.0) — see live-acceptance section.

Live acceptance — real host reboot ×2 on felhom-pve (agent v0.37.0 deployed)

Pre-reboot live proof of the root cause + fix: felhom-usb's mnt-felhom\x2dusb.mount was disabled while felhom-flash's was enabled. After deploying v0.37.0 and restarting the agent, ReassertEnrolledMounts logged re-asserted enrolled mount by UUID (enable --now) … wasMounted=true and the unit went enabled (wants-symlink created) — the mounted-but-disabled case handled.

The /dev/sdX reshuffle actually occurred and was tolerated as a no-op (the headline result):

Drive (UUID)	before #1	after #1	after #2
felhom-usb (da9e7089…, 931G)	sdb	sdc	sdb
felhom-flash (81a26531…, 119G)	sdc	sdb	sdc

The two drives swapped letters on reboot #1 and swapped back on reboot #2. Both times each landed at its correct mountpoint by UUID — findmnt /mnt/felhom-usb → the da9e7089 device, /mnt/felhom-flash → the 81a26531 device — never the wrong device. Both units enabled+active, no agent storage errors. On a clean boot the (now-enabled) units auto-mount, so ReassertEnrolledMounts finds them mounted+enabled and skips (the intended no-churn steady state).

• Ordering: felhom-shared-parent.service Finished 18:02:16 / 18:06:01; pve-guests.service Starting 18:02:22 / 18:06:07 — shared parent re-establishes ~6s BEFORE the guests, both reboots.
• Guest recovery: guest 9201 came up and all drive-backed apps recovered (immich, paperless, jellyfin, calibre, audiobookshelf, radarr, romm, komga) both reboots; romm's backing mount /mnt/felhom-drives/felhom-flash/userdata/roms resolved correctly across the swap.

Secrets (local_api token / leaf fingerprint) handled out-of-band — none committed.