agent v0.37.0: host-reboot remount re-resolves enrolled drives by fs-UUID

TASK A — close out the reboot story (agent half). Root cause (pinned live on
felhom-pve): an enrolled .mount unit left `disabled` by a prior detach never
auto-mounts at boot, and kernel re-enumeration can move a drive's node
(/dev/sdb->sdc). Fix re-asserts every enrolled mount by filesystem UUID at
startup + on the periodic tick.

- ResolveStorageDevice: resolve uuid:<fs-uuid> -> current /dev node via
  /dev/disk/by-uuid (never a cached node); errors if absent.
- parseFelhomMountUnit: pure inverse of renderMountUnit (marker-gated).
- (*SudoHostOps).ReassertEnrolledMounts: re-run EnsureMount (enable --now) for
  any enrolled unit not in /proc/mounts; idempotent, skips mounted/absent.
- main.go: runs before ReassertGuestBinds at startup + on the 20s tick.
- tests (Linux, seam=device resolution): letter-move tolerated (sdb->sdc) +
  red-proof companion, absent/scheme rejection, render->parse round-trip.

TASK A2 verdict: enrolling a NEW drive does NOT need an LXC restart — the path
lands on the live AttachDrive (shared parent, named live slots), not RebootGuest.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-16 17:49:00 +02:00
parent 437f096d9d
commit a621f4c5a0
8 changed files with 305 additions and 40 deletions
+63 -39
View File
@@ -1,45 +1,69 @@
# REPORT — storage lifecycle completion + acceptance sweep (agent half) → v0.36.7 (2026-06-15)
# REPORT — felhom-agent v0.37.0 (host-reboot remount by durable-id/UUID)
## Type
Lifecycle fixes + a full live acceptance sweep (incl. a real HOST REBOOT) on the intermediary-mount
foundation. Built + Linux-gated + deployed to felhom-pve per commit. Pairs with controller v0.68.1. Full
audit: `felhom.eu/documentation/audits/storage-lifecycle-acceptance-2026-06-15.md`.
**TASK — Close out the reboot story (agent half: Task A + Task A2).**
## Shipped (agent), each green-gated + deployed
- **v0.36.0 — guest boot-id on /disks.** `guest_boot_id` = `<host-btime>-<guest-init-starttime>`: changes
on every guest/host boot, stable across a controller-only restart. Lets the controller deterministically
recreate drive-backed apps after a guest reboot (replaces a fragile container-uptime sample). Pure parser
test + companion.
- **v0.36.1 — decommission keeps the raw mounted.** Decommission is now a LOGICAL retire (DetachDrive the
bind under the parent; LEAVE the raw `/mnt/<name>` mounted), so one-click re-enroll re-binds the real
drive. The old raw-unmount orphaned a non-removable drive (re-enroll bound an empty dir). **Caught live
in E10.**
- **v0.36.2 — eject also keeps the raw mounted** (same fix, for the H1 disconnect→reconnect roundtrip).
- **v0.36.3 — DetachDrive loop-umounts** ALL stacked binds (full detach → fail-close intact).
- **v0.36.4 — mutex** serializes AttachDrive/DetachDrive (no TOCTOU double-bind race).
- **v0.36.5 — AttachDrive normalizes to exactly one bind** (`countHostMounts`).
- **v0.36.6 — ROOT CAUSE of the double-bind:** the shared-parent self-bind inherited `/`'s shared peer
group, so every drive bind propagated back and DOUBLED. `make-private` (own group) BEFORE `make-shared`.
- **v0.36.7 — isolate the parent only on CREATE** (re-doing make-private/shared each reconcile churns the
peer-group id and orphans the guest's slave → propagation silently dies).
## Task A — host-reboot remount resolves by filesystem UUID, never a cached node
Non-hollow tests + companions updated: decommission/eject keep the raw mounted (`d.unmountCalls==0` +
DetachDrive called); the boot-id parser; etc.
### Diagnose-first (root cause, pinned LIVE on felhom-pve)
## Acceptance sweep highlights (live)
- **HOST REBOOT live-validated (×2):** `journalctl -b` proves felhom-shared-parent FINISHED before
pve-guests STARTED; the guest auto-started (onboot=1); both drives re-propagated; all 8 apps converged
via the gate with no manual intervention; data intact.
- Fail-close capability-proof (guest-root + root-app writes DENIED, no leak), confinement (both drives),
two-drive isolation, eject/decommission/re-enroll roundtrips, rapid eject/reconnect (single bind after
the double-bind root-cause fix).
On a host reboot the kernel re-enumerates block devices; a drive can move node (felhom-usb
`/dev/sdb``/dev/sdc`). Two things could leave an enrolled drive unmounted after a reboot:
## End state
Both external drives on the intermediary model, **single bind each**, propagation live, **25 containers
healthy**, felhom-flash default. agent v0.36.7 deployed (backups at /usr/local/bin/felhom-agent.bak-*).
1. A systemd `.mount` unit left **`disabled`** by a prior detach (`Unmount` = stop + disable)
never gets a `multi-user.target.wants` symlink, so it does **not** auto-mount at boot — and
nothing in the agent re-asserted it at startup.
2. Any remount that trusted a remembered `/dev/sdX` node would target the WRONG device after a
letter move.
## Known residuals
- Pre-start hook vs an agent restart can race the parent-bind churn (exit 255) — resolved by the
on-create-only isolation (parent no longer churned); operationally, don't restart the agent and reboot
the guest in the same instant.
- "Safely removable" fs-flush before a physical pull is the separate "remove from system" action's job.
Confirmed live: felhom-usb's mount unit was `disabled` (`systemctl is-enabled``disabled`,
no wants-symlink) while felhom-flash's was `enabled`. The `What=` line was already correct
(`/dev/disk/by-uuid/<UUID>`), so the fix is to **re-assert the unit by UUID at startup**, not to
rewrite the unit.
### Fix
- `storage.ResolveStorageDevice(durableID)` (`durable_device.go`) — resolves the enrolled
`uuid:<fs-uuid>` storage scheme to its **current** backing `/dev` node by re-scanning
`/dev/disk/by-uuid` (`EvalSymlinks`); rejects any non-`uuid:` scheme (anti-node-binding) and
errors when the UUID is absent so a caller skips a gone drive instead of fail-mounting.
- `storage.parseFelhomMountUnit` (`mountunit.go`) — pure inverse of `renderMountUnit`, gated on a
`Managed by felhom-agent` marker; extracts Name/UUID/Where/Type/Options, ignores foreign units.
- `(*SudoHostOps).ReassertEnrolledMounts(ctx)` (`hostops.go`) — for each enrolled `.mount` unit
not in `/proc/mounts`, re-resolve by UUID and re-run `EnsureMount` (idempotent
`systemctl enable --now`). Re-enables a disabled unit AND mounts the current device by UUID;
already-mounted drives skipped (no daemon-reload churn), absent UUIDs skipped (retry next tick).
- `main.go` — runs `ReassertEnrolledMounts` at startup **before** `ReassertGuestBinds`, and again
ahead of the bind re-assert on the periodic 20s tick, so host mounts are live before the guest
binds re-establish.
### Tests (non-hollow, seam = device resolution; Linux)
- `TestResolveStorageDevice_ToleratesDeviceLetterMove` — fake `/dev/disk/by-uuid/<uuid>` symlink
pointed at `sdc` (moved from the enroll-time `sdb`); asserts `ResolveStorageDevice` returns the
CURRENT node `sdc`. **Companion (red-proof):** asserts the cached enroll-time node (`sdb`) differs
from the freshly-resolved node — a node-trusting remount would mount the wrong device.
- `TestResolveStorageDevice_AbsentAndScheme` — absent UUID errors; only `uuid:` is resolvable
(`/dev/sdb1`, `store:`, `byid:`, `""` all rejected).
- `TestParseFelhomMountUnit``renderMountUnit``parseFelhomMountUnit` round-trip + rejects a
non-felhom unit.
## Task A2 — verdict: enrolling a NEW drive does NOT need an LXC restart
**NO.** Source-verified the enroll path: `/disks/guest-attach``handleDiskGuestAttach`
`GuestBinder.AttachDrive` (intermediary-mount model, "no pct, no reboot") — a live bind under the
single shared parent `/mnt/felhom-drives`, which provides unbounded **named** live slots. It does
NOT take the legacy `RebootGuest` branch. The operator's pre-created-slot-pool idea is unnecessary.
## Gate
`go build ./...`, `go vet ./...`, `go test ./...` all green locally (Windows: the symlink tests
skip — they assert on Linux, the agent's OS). Linux green gate + companion red-proof run on the
build server (192.168.0.180, go1.26.0) — see live-acceptance section.
## Live acceptance
Real host reboot ×2 on felhom-pve: before/after `lsblk`, shared-parent-before-pve-guests ordering,
both drives (felhom-usb + felhom-flash) remount by UUID across a `/dev/sdX` reshuffle, guest 9201
drive-backed apps recover, FileBrowser non-stale. (Captured during the supervised run.)
Secrets (local_api token / leaf fingerprint) handled out-of-band — none committed.