felhom.eu/documentation/audits/B3-storage-lifecycle-2026-06-15.md

B3 — Live unattended exercise of the storage lifecycle (migration / decommission / re-enroll / eject) + feature sweep

Date: 2026-06-15 Target: shipped controller v0.66.2 + agent v0.32.0, demo guest 9201 on felhom-pve (node demo-felhom). Mode: live, unattended, all 8 phases run without pausing. No production code changed. Drive method: the real operations were driven through the dashboard UI via the browser tool wherever the UI exposes them (per the A4 strict-UI requirement). Where the UI has no button or calls an unrouted endpoint (documented findings below), the operation was driven via the controller's own JSON API hit in-guest at the container IP with the correct Host header (bypassing the cloudflared/traefik proxy that mangles bodies) — never a raw shortcut around the controller's logic. Host-level captures/simulations used SSH (root@felhom-pve). Safety rails honoured: /dev/sda, the felhom-controller container, the bootstrap unit, felhom-pbs, local/local-lvm, and guest 9201's root disk were never touched. All operations confined to the two user-data drives (felhom-usb /dev/sdb1, felhom-flash /dev/sdc1) + the demo apps.

---

VERDICT

The migration engine is solid (checksums verified identical across every move, sources cleaned, HDD_PATH flipped, apps healthy). The decommission / eject / re-enroll lifecycle has serious gaps, including one critical guest-bricking bug and three unrouted endpoints the UI calls. The B3 migration target (felhom-flash) now holds all apps, healthy, and is the default drive.

🔴 Critical (fix before any customer uses decommission)

• C1 — Decommission bricks the guest on next reboot. The agent decommission removes its bind-tracking JSON + host-unmounts, but never runs pct set --delete mpN, so the stale mp bind entry survives in the LXC config. On the next guest reboot the LXC pre-start hook fails (exit 255) because the bind source (/mnt/<drive>/felhom-data) no longer exists → the whole guest fails to start — ALL apps down, not just the decommissioned drive's. Proven live (Phase 3): guest 9201 went down on reboot; only pct set 9201 --delete mp1 + pct start recovered it.

🟠 High (UI lifecycle largely non-functional)

• H1 — Three endpoints the settings JS calls are unrouted → HTTP 404: /api/storage/disconnect, /api/storage/reconnect, /api/storage/restart-apps. The "Leválasztás" (disconnect), "Csatlakoztatás" (reconnect), and "Alkalmazások indítása" (restart-apps) buttons are dead.
• H2 — No UI trigger for /api/storage/decommission. The B2b decommission flow (Phases 2 & 4 of this exercise) has no dashboard button — settings.html only exposes migrate-all (/api/storage/migrate), eject (/api/storage/eject), and remove. The routed /api/storage/decommission (mode=migrate|anyway) is reachable only by API.
• H3 — No one-click re-enroll for a decommissioned drive; its only UI action is "Eltávolítás a rendszerből" (remove). The Change-4 marker-clear fires only on re-register, which has no button on the decommissioned entry.

🟡 Medium

• M1 — Decommissioning the default drive does not reassign the default. After decommissioning felhom-usb (the default), no drive held the default badge (both remaining drives showed "Legyen alapértelmezett") — the default was orphaned on the decommissioned drive, no failover. New HDD-app deploys would have had no valid default target.
• M2 — Eject does NOT auto-remount on reconnect (contradicts the expected eject≠decommission distinction). An ejected drive is intent-gated-skipped by ReassertGuestBinds exactly like a decommissioned one; its systemd unit is disabled and it stays unmounted. The real distinction is: eject keeps the bind record (reassert logs a skip line) + sets no soft-marker; decommission removes the bind record + sets the "Kiváltva" soft-marker. Neither auto-remounts, and the only "reconnect" path (/api/storage/reconnect) is 404 (H1).
• M3 — import/calibre userdata dir loses setgid. After migrate-all, userdata/import/calibre was 755 on the target while it was 2775 on the source (and 16/17 other dirs preserved 2775). Root cause: the enroll-time userdata skeleton created import/calibre as 755 (mtime = enroll time), and the migrate-all merge-walk does not re-assert setgid on a pre-existing target dir (it correctly creates new dirs like import/paperless as 2775). calibre-web binds that dir — group-write/setgid is broken there.
• M4 — felhom-usb's systemd mount unit had a stale UUID (277a2179… vs the real da9e7089… from the 2026-06-14 reformat), so it could not auto-mount and a role-gated eject 403'd. Fixed in restore.
• M5 — Decommission's host-unmount does not release the drive from the running guest. The unprivileged LXC bind keeps /dev/sdX1 live-mounted inside the guest until reboot, so "safely removable" is only true post-reboot.
• M6 — Test notification email fails (returns "Hiba"), likely no SMTP transport configured on the demo.

---

Per-phase results (PASS/FAIL + evidence)

Phase 0 — baseline + seed — PASS

• Baseline: 8 HDD apps on felhom-usb (audiobookshelf, calibre-web, immich, jellyfin, komga, paperless-ngx, radarr, romm); felhom-usb = default; flash = empty skeleton.
• Userdata ownership baseline: all 17 dirs 2775 setgid, gid 101000 (= guest gid 1000). ✓
• Seeded checksummed files (source of truth):
  • userdata/media/movies/b3-movie.txt → 6155f6df…833e8
  • userdata/documents/b3-document.txt → 1f2bb9a5…3764
  • userdata/media/photos/b3-photo.txt → 162a7fa1…6e4f
  • appdata/romm/config/b3-romm.txt → 3643c7cc…60f1

Phase 1 — per-app migration (romm, felhom-usb → felhom-flash) — PASS (UI-driven)

Driven via the romm app page → "Áthelyezés másik tárhelyre" → felhom-flash → Áthelyezés button (native confirm() auto-accepted to clear the blocking dialog; the real appMigrate → POST /api/storage/migrate-app flow ran). Job mig-20260615-094757.

• stop → copy → verify → flip → redeploy healthy ✓ (romm Up … (healthy)).
• Migrated appdata checksum on flash = 3643c7cc…60f1 — identical ✓.
• Source subtree …/felhom-usb/…/appdata/romm removed ✓.
• HDD_PATH flipped: docker inspect romm mounts now /mnt/felhom-flash/...; app.yaml HDD_PATH: /mnt/felhom-flash ✓.
• UI badge updated to "Adattároló: felhom-flash" ✓.

Phase 2 — migrate-all + decommission felhom-usb → felhom-flash — PASS (migrate via UI; decommission via API per H2)

Migrate-all driven via settings → felhom-usb → "Összes adat áthelyezése" → felhom-flash (real storageMigrateAll → POST /api/storage/migrate). Job mig-20260615-095545, all 7 apps cleaned.

• All 7 apps healthy on flash ✓ (komga unhealthy = pre-existing healthcheck quirk, still serving).
• All 3 seeded userdata checksums identical on flash (6155f6df / 1f2bb9a5 / 162a7fa1) ✓.
• Source userdata files + appdata/ removed ✓.
• #8 setgid proof: 16/17 migrated userdata dirs 2775 gid 101000; 1 exception → import/calibre = 755 (finding M3).
• Decommission (no UI button → routed POST /api/storage/decommission {mode:anyway}): decommissioned:true; agent intent=decommissioned; host unmounted; guest-binds.json pruned (da9e7089 removed); controller badge "Kiváltva" ✓. But pct config still showed mp1 (stale — see C1).
• Default edge (M1): felhom-usb was default; afterwards no drive held the default badge — not reassigned.

Phase 3 — reconnect felhom-usb → NO auto-rebind — PASS (invariant holds) + 🔴 C1 surfaced

• Agent restart: host /mnt/felhom-usb not remounted; reassert log F9 re-assert: skipping non-enrolled drive (intent-gated) intent=ejected (demonstrated on the lingering ce9d drive; felhom-usb was pruned from the bind list entirely so it isn't even reattempted). No auto-rebind ✓.
• Guest reboot → C1 fired: lxc_init … Failed to run lxc.hook.pre-start … startup for container '9201' failed (exit 255) due to stale mp1. Recovery: pct set 9201 --delete mp1 → pct start 9201 → booted first try, felhom-usb correctly absent, 24→25 containers back.

Phase 4 — decommission-anyway felhom-flash → missing-storage — PASS (decommission via API per H2)

POST /api/storage/decommission {mode:anyway, mount_name:felhom-flash} → stopped_apps = all 8.

• Apps stopped, keep HDD_PATH=/mnt/felhom-flash ✓; flash intent=decommissioned + unmounted ✓.
• Missing-storage badge "Hiányzó tárhely: felhom-flash" rendered 8× on dashboard, 8× on stacks, on the app card ✓. Dashboard screenshot: 9 running / 8 stopped, Audiobookshelf "Leállítva" + the orange badge.

Phase 5 — re-enroll felhom-flash clears decommissioned + badge — PASS (re-mount via SSH + re-register via API per H3)

Re-mounted flash (data intact, seed checksum 6155f6df still matched ✓), POST /api/storage/register → reEnrollClearMarker.

• Agent intent → enrolled (Change 4 marker cleared) ✓; flash badge no longer Kiváltva; 0 "Hiányzó tárhely" on dashboard ✓.
• Apps restarted via POST /api/stacks/{name}/start (because /api/storage/restart-apps returned 404 — H1); 25 containers healthy ✓.

Phase 6 — eject vs decommission — PASS (finding M2: premise contradicted)

Eject flash via POST /api/storage/eject: intent→ejected, host unmounted, systemd unit disabled, bind record kept. Agent restart (reconnect sim): NOT auto-remounted; reassert logged skipping non-enrolled drive (intent-gated) intent=ejected. Apps stayed up (guest bind persisted). Flash restored (re-mount + register → enrolled, 25 containers).

Phase 7 — feature sweep

Feature	Result
Manual backup (whole-guest → felhom-pbs)	PASS — real encrypted vzdump→proxmox-backup-client; phase reached done; new PBS snapshot ct/9201/2026-06-15T10:29:13Z landed (4 total). A 2nd trigger correctly returned 409 "already in progress".
Fresh app deploy + lifecycle + logs (Glance)	PASS — deploy 202 → healthy; stop/start/restart all HTTP 200; logs endpoint returned app output.
FileBrowser umask fix	PASS — PID1 Umask: 0002, entrypoint [sh -c umask 002; exec …]; umask-002 folder = 2775 group-writable (gid 101000); a uid-1000 process wrote into it ("WRITE OK") → FileBrowser/app group collaboration confirmed. (Note: FileBrowser web session had expired; folder-creation via the UI was not re-driven because logging in would require entering a password — prohibited. The fix's effect was verified by replicating PID1's exact umask-002 environment + the group-write test.)
Monitoring page	PASS — renders host CPU/mem/temp/uptime + storage bars; felhom-usb correctly shown "Nem elérhető" while decommissioned.
Test notification email	FAIL (M6) — /settings/notifications/test returned "Hiba" (likely no SMTP configured).
Mem-gate	Gate allows within budget (Glance deployed cleanly); a refusal was not reproducible with ~10 GB free and no catalog app large enough to exceed 12 GB — would require artificial RAM exhaustion (not done, would destabilise the demo).

Phase 8 — restore — DONE (sane end-state)

• felhom-flash: Alapértelmezett | Aktív | USB — default, all apps healthy.
• felhom-usb: Aktív (stale-UUID unit fixed → da9e7089, re-mounted, soft-marker cleared; mp1 re-bound and valid). Note: its agent intent remains decommissioned (a latent cosmetic inconsistency — harmless because the drive is mounted and its mp is valid; the reassert would merely skip re-adding an already-present bind).
• Belső SSD: Aktív. No "Kiváltva" anywhere; no stale mps (mp1=usb, mp2=flash both mounted → reboot-safe); 25 containers running; Glance test app removed. sda / PBS / controller / local / local-lvm untouched throughout.

---

Code-fix candidates for the next slice

1. (C1, critical) On decommission/eject, the agent must pct set <vmid> --delete mpN for the drive's bind (or the controller must, post-finalize) so the stale bind cannot brick the guest on reboot. Add a regression test: decommission a data drive → reboot guest → assert it starts and the drive is absent. Alternatively, make the agent's pre-start hook self-heal by dropping config binds whose source path is missing.
2. (H1) Route /api/storage/disconnect, /api/storage/reconnect, /api/storage/restart-apps (or remove the dead buttons). These are called by settings.html JS but absent from ServeStorageAPI.
3. (H2) Wire a decommission button (migrate-all-then-decommission / decommission-anyway with type-to-confirm) into the settings storage actions → /api/storage/decommission.
4. (H3) Add a re-enroll action on a decommissioned drive entry (re-mount + re-register in one click), instead of only "Eltávolítás a rendszerből".
5. (M1) Reassign the default to another schedulable drive when the current default is decommissioned/removed (or block decommissioning the default until reassigned).
6. (M2) Decide eject semantics: either make eject auto-remount on reconnect (and route /api/storage/reconnect), or relabel the UI so "eject" reads as the permanent-until-re-enroll action it actually is.
7. (M3) Fix the userdata skeleton so every nested dir (incl. import/calibre) is created 2775 setgid at enroll, and have the migrate-all merge-walk re-assert setgid on pre-existing target dirs.
8. (M4) Update the systemd mount unit's What= UUID when a drive is reformatted (or key the unit on a durable id), so re-mount after reformat works.

---

Evidence index (selected)

• Checksums (source-of-truth → verified-identical-on-target): movie 6155f6df…, document 1f2bb9a5…, photo 162a7fa1…, romm-appdata 3643c7cc….
• Ownership: source 17/17 2775 gid 101000; post-migrate-all target 16/17 2775, import/calibre 755 (M3).
• C1 brick: __lxc_start: 2208 Failed to initialize container "9201"; recovery via pct set 9201 --delete mp1.
• Intent transitions (agent drive-intents.json): usb da9e7089 enrolled→decommissioned→(restore, controller-cleared); flash 81a26531 enrolled→ejected→enrolled, →decommissioned→enrolled.
• PBS snapshot landed: felhom-pbs:backup/ct/9201/2026-06-15T10:29:13Z.
• Screenshots saved (browser): romm-on-flash badge, Phase-4 dashboard missing-storage badges, monitoring page, settings storage states.