diff --git a/documentation/audits/storage-lifecycle-acceptance-2026-06-15.md b/documentation/audits/storage-lifecycle-acceptance-2026-06-15.md new file mode 100644 index 0000000..718a3f0 --- /dev/null +++ b/documentation/audits/storage-lifecycle-acceptance-2026-06-15.md @@ -0,0 +1,159 @@ +# Storage lifecycle completion + acceptance sweep (intermediary-mount) — 2026-06-15 + +**Scope:** finish the storage lifecycle on the intermediary-mount foundation (Phase A), migrate the +second drive + confirm rslave (Phase B), and run a full edge-case acceptance sweep **including a real +host reboot** (Phase C). Unattended, live on guest 9201 / felhom-pve. Trunk-based, per-commit green gate, +non-hollow tests with pre-fix companions. + +**Deployed at close:** agent **v0.36.7** (`felhom-agent@1e20584`), controller **v0.68.1** +(`felhom-controller@7a85732`), catalog `app-catalog-felhom.eu@939864f`. Both external drives on the new +model, **25 containers healthy**, felhom-flash default. + +**Headline:** the **host-boot ordering is now LIVE-validated** — on a real `reboot` of felhom-pve the +shared-parent unit ran *before* pve-guests, the guest auto-started (onboot=1), both drives re-propagated +and all 8 apps converged with **no manual intervention**. + +--- + +## Phase A — lifecycle fixes + +| Item | Result | Where | +|---|---|---| +| **Boot-id determinism** | ✅ agent v0.36.0 emits `guest_boot_id` (`-`); controller v0.68.0 persists `LastGuestBootID` + `processGuestBootChange` recreates **all** deployed drive-backed present apps on change. v0.68.1: state-independent (was missing apps docker hadn't restarted yet). | `internal/localapi/intermediary.go` (`GuestBootID`), `internal/web/intermediary.go` (`shouldRecreateOnBoot`), `settings.go` | +| **H2 decommission button** | ✅ "Leszerelés" on every connected drive → migrate-then-decommission (target select) OR decommission-anyway (type-to-confirm). Server side was already complete; new model never touches the parent mp. | `templates/settings.html` (`storageDecommission`) | +| **H3 re-enroll button** | ✅ "Visszacsatlakoztatás" on decommissioned/ejected drives → clears marker + re-attaches under the parent + restarts gate-stopped apps. | `handleStorageReconnect` (decommissioned branch) | +| **M1 default reassignment** | ✅ `defaultPromotionTarget`: decommissioning the default auto-promotes another schedulable drive (prefers the migrate target); BLOCKS if none. | `internal/web/intermediary.go`, `finalizeDecommissionWith` | +| **M3 userdata setgid** | ✅ skeleton already 2775 on every nested dir; the migrate merge-walk now RE-ASSERTS 2775 (`EnsureUserdataDir`) on userdata dirs (`isUserdataDir`) instead of merely preserving the source mode. | `internal/stacks/migrate.go` | + +Non-hollow tests (each with a companion that fails the pre-fix impl): `TestShouldRecreateOnBoot`, +`TestDefaultPromotionTarget`, `TestIsUserdataDir`, `TestStarttimeFromStat`. Also fixed the existing H1 +`{path}` vs `{where}` JS body mismatch. + +## Phase B — second drive + rslave + +- **felhom-usb migrated** off its stranded legacy bind (stale `decommissioned` intent cleared via the H3 + reconnect → agent guest-attach set intent=enrolled + recorded the bind + bound felhom-data under the + parent); registry repointed to `/mnt/felhom-drives/felhom-usb`; legacy `mp1` deleted. **Both** drives + now on the new model. +- **rslave is EXPLICIT** (not incidental): `-v /mnt:/mnt:rslave` in the controller bootstrap docker-run + (line 50). Verified it survives a controller redeploy (E7). + +--- + +## Phase C — acceptance sweep (per-item, with evidence) + +### E1 — HOST REBOOT (the headline) ✅ LIVE-VALIDATED (twice) +`reboot` issued over SSH; polled SSH return (~70–80 s). From `journalctl -b`: +``` +felhom-shared-parent.service Finished 19:50:29 (ExecMainStatus=0) +pve-guests.service Starting 19:50:35 → "Starting CT 9201" (onboot=1 auto-start) +``` +The shared parent was made shared **before** pve-guests; the guest auto-started; both drives +re-propagated (`BoundUnderParent`); `/mnt/felhom-usb` (legacy) gone. Apps converged via the gate with no +manual intervention: +``` +[gate] drive ABSENT /mnt/felhom-drives/felhom-flash — stopped+blocked 8 app(s) +[gate] drive RETURNED /mnt/felhom-drives/felhom-flash — re-attached + restarted gate-stopped apps +``` +Data intact (romm bound to `/mnt/felhom-drives/felhom-flash/...`, immich-postgres `base`). The first +host reboot (v0.68.0) surfaced a bug — 5 apps stayed exited because the recreate filtered on container +state; fixed in v0.68.1 (recreate ALL deployed present apps) + re-validated. + +### E2 — guest reboot ×3 ✅ +Multiple guest reboots; each converged all 8 apps with zero manual starts (the deterministic `guest_boot_id` +recreate + the gate). The final clean reboots showed **single bind per drive** + no 255. + +### E3 — drive absent at guest boot ✅ (covered by E1) +The gate's ABSENT→RETURNED path (E1) is exactly this: a drive absent at boot stops+blocks its apps; the +guesthook (C1 net) lets the guest boot regardless. The agent re-binds on return → gate restarts. + +### E4 — drive yanked while running ✅ +`umount` of a running drive → immediate guest-root write to the bare stable path **DENIED**, **no leak** +to host root. The agent reconcile auto-rebound it; apps stayed healthy. (Gate stop/restart proven by E1.) + +### E5 — fail-close, capability-proof ✅ +With felhom-usb detached, a **guest-root** write AND a **root app container** (komga) write to the bare +stable path both returned **Permission denied** — the host-root-owned (unmapped) dir defeats guest-root +CAP_DAC_OVERRIDE (what chmod-0000 could not). `find` showed no file leaked onto host root. + +### E6 — confinement ✅ (both drives) +Guest sees only `userdata` (felhom-usb) / `appdata backups media userdata` (felhom-flash); `ls .../dump` +→ not found. The host raw drives carry `dump/images/lost+found/...` which never cross in. + +### E7 — controller redeploy ✅ +After the controller container restarts, `/mnt/felhom-drives` is `private,slave` inside it and it still +sees the drive data — proving the `:rslave` is explicit, not incidental. + +### E8 — two drives ✅ +Detaching felhom-usb left felhom-flash + its 8 apps fully unaffected. + +### E9 — decommission button (H2) ✅ +`decommission-anyway` on felhom-usb → `decommissioned:true`, intent decommissioned, bind detached, **parent +mp untouched** (reboot-safe, no brick). Migrate-then-decommission shares the proven `handleStorageDecommission`. + +### E10 — re-enroll (H3) ✅ (after a fix) +**First attempt FAILED** and exposed a real bug: decommission unmounted the RAW drive, so re-enroll bound +an empty pve-root dir. **Fix (agent v0.36.1):** decommission is now a *logical* retire — DetachDrive the +bind but LEAVE the raw mounted, so re-enroll re-binds the real drive. Re-tested: re-enroll → felhom-usb +backed by `/dev/sdc1[/felhom-data]`, data intact. + +### E11 — default reassignment (M1) ✅ +Made felhom-usb default, decommissioned it → `[gate] default drive reassigned … → … (M1)`; a default +always remained (never zero). The only-drive BLOCK is unit-tested (`TestDefaultPromotionTarget`). + +### E12 — eject the drive holding ALL apps ✅ (covered by E1) +E1's felhom-flash ABSENT→RETURNED is exactly this: all 8 apps gated, then auto-restarted on return, +reboot-free. + +### E13 — rapid eject/reconnect ×3 ✅ (after the deepest fix) +Surfaced a **double-bind** (2 stacked binds per drive). Root-caused to the shared-parent self-bind +**inheriting `/`'s shared peer group** (`/mnt/felhom-drives` was `shared:1` like `/`), so every drive +bind propagated back and doubled. Fixed across agent v0.36.3–v0.36.7: +- v0.36.3 `DetachDrive` loop-umounts all stacked layers (full detach → fail-close intact); +- v0.36.4 a mutex serializes Attach/Detach (no TOCTOU double-bind); +- v0.36.5 `AttachDrive` normalizes to exactly one bind (`countHostMounts`); +- **v0.36.6 the real root cause** — `make-private` before `make-shared` so the parent owns its own peer + group (now `shared:51`), binds propagate to the guest exactly once; +- v0.36.7 isolate **only on create** (re-doing it churns the peer-group id and orphans the guest's slave). + +End state: **single bind per drive**, guest sees both, all apps healthy, no leak. + +### E14 / E15 — migration + setgid (M3) ◑ partial +M3's routing (`isUserdataDir`) is unit-tested and `EnsureUserdataDir` enforces 2775; the **convention was +verified live** (`/mnt/felhom-drives/felhom-flash/userdata` = `2775`, gid 1000; FileBrowser bound there). +A full live migrate-all round-trip between the two drives was **NOT re-run** to avoid merging both drives' +userdata; the migrate/checksum machinery is the pre-existing, previously-validated path (only the M3 +re-assert is new, and it's unit-tested). + +### Regression ✅ +FileBrowser bound to `userdata` on both new-model drives (2775 group-write); immich-postgres data intact; +dashboard + monitoring HTTP 200; no exited containers. + +--- + +## Surprises / breaks found (and fixed) this run +1. **E1 (v0.68.0):** boot-id recreate filtered on container state → 5 apps stayed exited after a host + reboot. → v0.68.1 recreate ALL deployed present apps (state-independent). +2. **E10:** decommission unmounted the raw drive → re-enroll bound an empty dir. → v0.36.1 keep raw mounted + (logical retire); v0.36.2 same for eject. +3. **E13 double-bind:** shared-parent inherited `/`'s peer group → every drive bind doubled. → v0.36.6 + make-private before make-shared (own group); v0.36.7 only-on-create (avoid peer-group churn that + orphans the guest slave). v0.36.3–.5 are defense-in-depth (loop-detach, mutex, normalize-to-one). +4. **Pre-start hook vs agent churn (transient):** rebooting the guest *immediately after* an agent restart + raced the agent's parent-bind churn against the guest's mp3 setup → `lxc.hook.pre-start` exit 255. + Resolved by the on-create-only isolation (the parent is no longer churned); a clean guest reboot + (no concurrent agent op) returned exit 0. Operational note: don't restart the agent and reboot the + guest in the same instant. +5. **Device-letter swap:** on a host reboot the kernel re-enumerated sdb↔sdc; harmless because all mounts + are by fs-UUID. + +## Residuals (documented, not blocking) +- **E14/E15** full live migrate-all not re-run (M3 unit-tested + convention verified live; data-merge risk). +- **Boot-id first-sight bounce:** a fresh controller (empty `LastGuestBootID`) recreates apps once on its + first start — a one-time cost per data-volume lifetime; subsequent restarts don't bounce. +- **"Safely removable" semantics:** eject/decommission now leave the raw mounted (re-enrollable); + fs-flush-before-physical-pull is the separate "remove from system" action's job (future refinement). + +## Commit hashes +agent `1e20584` (v0.36.7) · controller `7a85732` (v0.68.1) · catalog `939864f`. +Agent versions this run: v0.36.0→v0.36.7. Controller: v0.68.0→v0.68.1.