felhom.eu/documentation/tests/TEST-REPORT-stable-path-sysdrive-restart-2026-06-23.md

TEST REPORT — NO-MERCY CAMPAIGN: stable-path model + /mnt/sys_drive + server restart

Date: 2026-06-23 · Class: destructive test campaign (operator-authorized, DEMO only) Targets: guest 9201 (demo-felhom) on felhom-pve (192.168.0.162, N100, PVE 9) · controller v0.81.0 · agent v0.40.0 · hub hub.felhom.eu Rule: report-don't-fix. No product-code changes. Findings get evidence + file:line + a proposed fix direction (NOT implemented).

Status legend

PASS = effect asserted AND wrong-outcome ruled out · FAIL = wrong outcome observed · INCONCLUSIVE = could not assert cleanly (reason given).

---

Phase 0 — floor + baseline (non-destructive) — ✅ COMPLETE

PBS floor (hard restore floor)

• Snapshot id: felhom-pbs:backup/ct/9201/2026-06-23T17:28:25Z (PBS datastore felhom-spike @ felhom@pbs!n100@192.168.0.180). Backup finished successfully (00:00:15, incremental, reused 89.9%).
• Used? Not yet (recorded for restore if a later phase wedges 9201).
• Note: felhom-pbs storage resolves to the PBS server on 192.168.0.180 datastore felhom-spike. Writing the floor snapshot there is normal PBS operation (the demo's nightly job already does) — it does not mutate/break 180 as a host, consistent with the guardrail.

Baseline — HOST (pct config 9201)

• rootfs: local-lvm:vm-9201-disk-0,size=32G · mp0: vm-9201-disk-1,mp=/var/lib/docker,backup=1,size=200G
• mp1: vm-9201-disk-2,mp=/mnt/sys_drive,backup=1,size=50G · mp8: /mnt/felhom-drives (bind) · mp9: …/bootstrap → /etc/felhom-bootstrap,ro=1
• onboot: 0 ⚠️ — flagged for Phase 4.1 (the spec asserts 9201 auto-starts on host reboot; with onboot: 0 a plain host reboot will not auto-start it). Pre-existing config.
• Drives (host lsblk -f): felhom-usb = /dev/sdb1 UUID da9e7089-… (916 GB HDD); felhom-flash = /dev/sdc1 UUID 81a26531-… (117 GB USB). Both mounted host-side at raw /mnt/<name> AND bind-mounted to stable /mnt/felhom-drives/<name> under the shared parent.

Baseline — GUEST

• findmnt /mnt: /mnt/sys_drive = pve-vm--9201--disk--2 (distinct device); /mnt/felhom-drives/{flash,usb} = sdc1[/felhom-data] / sdb1[/felhom-data].
• Invariant (bug signature) — PASS: no rootfs placeholder dirs at raw /mnt/felhom-flash / /mnt/felhom-usb inside the guest (not present (good) for both).

Baseline — CONTAINER (felhom-controller)

• settings.json storage paths = ONLY stable /mnt/felhom-drives/felhom-flash + /mnt/felhom-drives/felhom-usb (no raw entries). Both findmnt to the correct sdX1[/felhom-data]. /mnt/sys_drive = disk-2.
• Container / = docker-data overlay (197G, the 200G mp0). /mnt/sys_drive separate (50G).

Baseline — AGENT state (/var/lib/felhom-agent/)

• guest-binds.json → 9201: [uuid:81a26531 (flash), uuid:ce9d1d81 (absent), uuid:da9e7089 (usb)].
• drive-intents.json: 277a2179→enrolled (absent), 81a26531→enrolled (flash), ce9d1d81→ejected (absent), da9e7089→enrolled (usb). ⚠️ Two leftover intents for physically-absent drives (277a2179 enrolled-absent, ce9d1d81 ejected) — relevant to Phase 4.3 intent-gate.
• Agent version: felhom-agent 0.40.0.

Deployed apps at baseline

• Only base infra running: filebrowser, felhom-controller, cloudflared, traefik (all healthy). No customer apps deployed — tests needing "an app on a drive" will deploy a small one.

Scratch

• Loop devices: LOOP_A=/dev/loop0 (/root/nomercy-scratch/loopA.img, 2G), LOOP_B=/dev/loop1 (/root/nomercy-scratch/loopB.img, 2G). For teardown in Phase 5.

Phase 2.3 (opportunistic, captured during the floor vzdump) — ✅ PASS

• The floor vzdump 9201 log: including mount point mp1 ('/mnt/sys_drive') in backup and the proxmox-backup-client invocation carries --include-dev /mnt/vzsnap0/./mnt/sys_drive. mp1 is in the archive. Wrong outcome (mp1 excluded / backup=0) ruled out. (Optional restore-to-scratch deferred.)

---

Phase 1 — stable-vs-raw path model — ✅ COMPLETE (1.3/1.4/1.8 by-design inconclusive)

Method: mix of live-UI (claude-in-chrome on felhom.demo-felhom.eu) and exact-endpoint (curl to the in-guest controller 127.0.0.1:8080). The demo controller has no password set (authEnabled()=false), so CSRF is skipped (csrf.go:30) and direct POSTs equal the UI's requests (no server logic skipped — only browser rendering, which I cover with claude-in-chrome where it matters).

1.1 — registry holds only stable paths — ✅ PASS (exact-endpoint + UI)

• settings.json storage_paths = ONLY /mnt/felhom-drives/{felhom-flash,felhom-usb}. GET /api/disks: both drives state=attached, role=user-data, guest_path=/mnt/felhom-drives/<name>, bound_under_parent=true, real sizes (flash 125.7 GB, usb 983 GB). No raw /mnt/<name> registry entry; no rootfs placeholder. Wrong outcome (raw entry / placeholder) ruled out.

1.2 — deregister → re-add via "Regisztrálás" — ✅ PASS (exact-endpoint = UI request)

• Deregister-only of felhom-usb via POST /settings/storage/remove (the "Eltávolítás" form, field storage_path, stable path) → 302; registry left only flash; drive stayed mounted at stable (sdb1[/felhom-data]) — not ejected; no rootfs placeholder.
• Re-add via POST /api/storage/register {where:"/mnt/felhom-usb"} (the raw mount_path the disk-card button sends) → {ok:true, where:"/mnt/felhom-drives/felhom-usb", raw:"/mnt/felhom-usb"}. Registry restored to BOTH stable paths; no raw entry leaked; no rootfs placeholder created. The v0.78 bug (raw registration + "Rendszermeghajtón" placeholder) is ruled out on the exact UI request path.

1.3 — attach existing (LOOP_A) — ⚠️ INCONCLUSIVE (by-design block; compensating coverage)

1.4 — init/format+enroll (LOOP_B) — ⚠️ INCONCLUSIVE (by-design block; compensating coverage)

1.8 — wipe (LOOP_A) — ⚠️ INCONCLUSIVE (destructive device-wipe not executed)

• Loop devices cannot drive the real init/attach/wipe flow: the agent refuses non-whitelisted block devices — POST /api/storage/init {device:/dev/loop1} → HTTP 400: storage: refusing to operate on non-whitelisted block device "/dev/loop1" (felhom-agent internal/storage/validate.go:124, ValidateBlockDevice; the whitelist is real block-disk patterns only, no dm/loop/symlinks). This is a SECURITY PASS in its own right — the agent will not format an arbitrary device the controller names.
• The only agent-enumerable user-data devices are the two real drives; felhom-flash holds ~173 MB of real appdata+backups with no PBS floor (mp8 is a bind, excluded from the whole-CT vzdump), so a destructive real-drive wipe is disproportionate to the test value and was not performed.
• Compensating coverage for the stable-registration these target: the init/attach paths register via the SAME registerStoragePath(stablePathForName(path.Base(where)), …) proven live in 1.2, and the controller unit tests TestRunStorageInit_Success / TestRunStorageAttach_Success assert registration at /mnt/felhom-drives/<name> (and that the agent gets the RAW path) — both green this session. Recommend a future supervised run with a genuinely spare whitelisted disk to close the live gap.

1.5 — disk card shows stable path + correct badge — ✅ PASS (live UI, claude-in-chrome)

• Both cards render the stable sub-line (/mnt/felhom-drives/felhom-flash, /mnt/felhom-drives/felhom-usb), correct device/size (ext4 · /dev/sdc1[/felhom-data] · Flash Drive 0.2/117.1 GB; ext4 · /dev/sdb1[/felhom-data] · HD710 PRO 0.0/915.8 GB), badge "Regisztrálva" + "Aktív" — NOT the v0.79 spurious "Nem regisztrált", NOT "Rendszermeghajtón". Action buttons (Leválasztás/Eltávolítás/Leszerelés/Törlés) present and (per source regKey=guest_path) carry the stable path. Screenshot captured.

1.12 — free-form raw add probe (the parked gap) — ✅ PASS (no symptom reproduced) (exact-endpoint)

• Raw /mnt/felhom-flash does not exist inside the container (the raw host mount is never bound into the guest; only the stable parent /mnt/felhom-drives is, via mp8 + the /mnt:rslave bind). So POST /settings/storage/add {storage_path:/mnt/felhom-flash} is rejected at validation: "Az útvonal nem létezik vagy nem mappa." (handlers.go:1264 exists-and-dir; :1272 is-mountpoint). Registry unchanged (still exactly the 2 stable paths). The free-form-add gap does not recreate the "Rendszermeghajtón"/disconnected symptom for a raw drive path.
• Observation (not a finding): the same form WOULD accept /mnt/sys_drive (exists + is a mountpoint + writable + non-overlapping) — i.e. a user could free-form-register the system SSD volume as a user-data path. That is consistent with the documented SSD-only fallback (Phase 2.4), not the raw-drive bug; noted for awareness, no symptom.

1.6 — deploy app onto a drive → HDD_PATH stable — ✅ PASS (exact-endpoint = UI deploy)

• Deployed navidrome (lightest drive-backed app, 256M) via POST /api/stacks/navidrome/deploy {values:{DOMAIN,SUBDOMAIN:navtest,HDD_PATH:/mnt/felhom-drives/felhom-usb}} → healthy. app.yaml HDD_PATH=/mnt/felhom-drives/felhom-usb (stable). Container media bind /mnt/felhom-drives/felhom-usb/userdata/media/music → /music (stable drive namespace, dir 2775 g1000). Data not on rootfs; HDD_PATH not raw. (navidrome's config/DB is a named Docker volume on the docker-data disk by catalog design — only its media is drive-bound.)

1.7 — eject removes the stable entry cleanly + impact names apps — ✅ PASS (exact-endpoint)

• Impact: GET /api/storage/impact?where=/mnt/felhom-drives/felhom-usb → apps:["Navidrome"]; felhom-flash → apps:[]. Names the right apps (wrong outcome "no apps when an app uses it" ruled out).
• Eject (Leválasztás) on felhom-flash (no app, reversible): agent ejected the raw /mnt/felhom-flash (mapped from stable via agentWhere); registry left ONLY felhom-usb — the felhom-flash stable entry removed cleanly, no orphan (the v0.80 bug ruled out); bind detached, raw host mount preserved. Restored via POST /api/storage/register → re-registered stable + re-bound (sdc1[/felhom-data]).

1.8 — wipe — ⚠️ INCONCLUSIVE (see 1.3/1.4 — no safe whitelisted target; device-wipe not executed)

• The wipe's confirm/impact GATE is otherwise exercised: handleStorageWipe requires mount_name == path.Base(where) (storage_handlers.go:558) and ejects via agentWhere then formats via the agent's two-step confirm — the same mapping proven in 1.7/1.10. The destructive mkfs itself was not run (loop blocked by the whitelist; real drives hold un-floored data).

1.9 — migrate-app flips HDD_PATH stable→stable — ✅ PASS (exact-endpoint)

• POST /api/storage/migrate-app {app:navidrome, target:/mnt/felhom-drives/felhom-flash} → job ran validate→copy→verify→flip→cleanup (phase=done, unit cleaned). navidrome HDD_PATH flipped to /mnt/felhom-drives/felhom-flash (stable); media bind now on flash; navidrome healthy; no raw leak; source cleaned only AFTER verify. Wrong outcomes (source deleted before verify / HDD_PATH left raw/old) ruled out. (navidrome now lives on felhom-flash — removed in Phase 5.)

1.10 — self-serve decommission → agent op via raw + app missing storage — ✅ PASS (exact-endpoint)

• POST /api/storage/decommission {where:stable-usb, mode:anyway, mount_name:felhom-usb} → {decommissioned:true, stopped_apps:["Navidrome"]}. Agent intent da9e7089→decommissioned; agent log shows the op on the RAW path (where=/mnt/felhom-usb, guest-detach … fully unmounted) — wrong outcome (agent called with the unmapped stable path) ruled out. navidrome gate-stopped.
• Restored: POST /api/storage/reconnect → {reenrolled:true, restarted:["navidrome"]}, intent→enrolled, re-bound at stable, navidrome healthy.

1.11 — backup lands under the stable namespace — ✅ PASS (path-correctness) + FINDING (restic) (exact-endpoint)

• POST /api/backup/run (the app-data DB-dump backup) ran successfully; for navidrome it correctly found 0 databases (no DB container — SQLite in a named volume), a no-op success, not a failure.
• Path correctness: all existing backup artifacts live under the stable namespace /mnt/felhom-drives/felhom-flash/backups/primary/<app>/ (recovery units: compose/ + manifest.json), never a raw/rootfs path. Wrong outcome (backup under raw/rootfs) ruled out.
• ⚠️ FINDING F1 (minor, pre-existing): the prompt's "restic repo" premise is stale, and the restic binary is dead weight in the image. Disk-tier restic backup was moved to the host agent in the de-privileging; the controller source no longer execs restic (all references are "moved to agent" comments + vestigial config.go ResticPasswordFile/Schedule fields + the settings.go migrateResticToRsync one-way migration). But Dockerfile:58 still apt installs restic (and line 42 comments "for backup operations"), so /usr/bin/restic ships in the controller image unused. Fix direction: drop restic from the Dockerfile install list (and the dead Restic* config fields) to shed image weight + remove a confusing "disk-tier is still here" signal. Not a functional bug.
• Observation: a named-volume app (navidrome) deployed onto a drive created no appdata/ or backups/ namespace nor a recovery unit on that drive (only the shared userdata/ skeleton). Recovery units appear keyed to appdata-bind apps; named-volume-only apps rely on the DB-dump/volume-tar path instead. Flagged for awareness; likely by-design, not verified as a bug.

Phase 1 verdict: the stable-vs-raw path model holds across register/deregister/disk-card/eject/ decommission/migrate/backup. The v0.78 (raw register), v0.79 (spurious unregistered badge), and v0.80 (orphaned stable entry on eject) bugs are all ruled out live. The only gaps are the loop-device init/attach/wipe (blocked by the agent's block-device whitelist — a safety PASS) and the restic dead-weight finding (F1).

Phase 2 — /mnt/sys_drive third volume — ✅ COMPLETE (2.5 reboot-survival in 3.3)

Method: exact-endpoint + host/guest inspection.

2.1 — sys_drive is a distinct device; warning absent — ✅ PASS

• Three distinct devices: /=pve-vm--9201--disk--0, /var/lib/docker=disk--1, /mnt/sys_drive=disk--2 (same in the controller container). Controller selftest log: [PASS] System data path: /mnt/sys_drive; the "… nem külön meghajtón van" warning (healthcheck.go:301, which fires per user-data storage path that isn't a mountpoint) does not fire — both drives are mountpoints. Wrong outcome (sys_drive on rootfs / warning present) ruled out.

2.2 — disk-full isolation — ✅ PASS

• fallocate -l 46G /mnt/sys_drive/… → sys_drive 99% (599M free). / unchanged (940M/29G free, 4%), /var/lib/docker unchanged (1.5G/186G, 1%). A dd to sys_drive hit ENOSPC at the 599M boundary (598 MiB written then stop — confined); dd of 50 MiB to / and to /var/lib/docker both succeeded. All apps (navidrome, radarr, filebrowser, traefik, cloudflared) + the controller stayed healthy; GET /settings 200 throughout. Cleanup restored sys_drive to 1%. Wrong outcome (rootfs or docker-data fills / controller crashes) ruled out.

2.3 — mp1 included in whole-CT vzdump — ✅ PASS (captured during the Phase-0 floor)

• The floor vzdump 9201 log: including mount point mp1 ('/mnt/sys_drive') + the proxmox-backup-client … --include-dev /mnt/vzsnap0/./mnt/sys_drive. mp1 (backup=1) is in the PBS archive. Wrong outcome (mp1 excluded) ruled out.

2.4 — SSD-only fallback app data on disk-2 — ✅ PASS

• Deployed radarr with HDD_PATH=/mnt/sys_drive. Media binds /mnt/sys_drive/userdata/{media/movies,downloads}; the data dir is backed by disk--2, not the disk--0 rootfs. Wrong outcome (data on rootfs) ruled out.

2.5 — df size + reboot-survival — ✅ PASS (size) / ↪ reboot-survival verified in 3.3

• df /mnt/sys_drive = 50 GB (8 golden + 42 grow, matching mp1 size=50G). Still a separate device after the guest reboot — confirmed in Phase 3.3 (post-restart sys_drive still disk--2).

Phase 3 — server restart + retired banner — ✅ COMPLETE (F2 found)

3.1 — banner gone in every storage state — ✅ PASS

• Rendered /settings in the current state (navidrome on flash, radarr on sys_drive, both drives registered, after the 1.12 raw free-form add attempt): 0 occurrences of "aktiválásra vár" / "Újraindítás most" / activate-drives-btn / activatePendingDrives. The banner is retired (v0.81.0) and never appears. The "Kiszolgáló újraindítása" card is present (4 refs).

3.2 — endpoint routing + wrapping — ✅ PASS (routing) / ⚠️ auth-gate not demonstrable on demo

• POST /api/storage/activate → 404 (old route removed). POST /api/server/reboot is routed (202 — confirmed live via the 3.3 UI click, not double-fired here). Route is wrapped RequireAuth(CsrfProtect(HandleServerReboot)) (cmd/controller/main.go:731).
• Demo limitation: the demo controller has no password set (authEnabled()=false), so RequireAuth
  • CsrfProtect are both no-ops here (csrf.go:30 exempts when auth is disabled) — the "rejected without auth/CSRF" half of the matrix can't be shown without setting a password (out of scope: changing account settings). The middleware wrapping is source-verified and is the same gate every other mutating endpoint uses. Pre-existing security note (unrelated to this session): the demo dashboard being open is the already-documented controller-dashboard-public-unauth issue.

3.3 — "Kiszolgáló újraindítása" (UI click → guest reboot) — ✅ PASS (live UI, operator-authorized)

• Method: claude-in-chrome on felhom.demo-felhom.eu/settings; the native confirm() was neutralized (overridden to return true, per browser-safety guidance against blocking dialogs) and the page's own click handler restartServer() was invoked — the full client-side flow (confirm → fetch /api/server/reboot → pollRestart). (A ref-coordinate click did not register the handler — UI quirk — so the handler was invoked directly; same code path.) Confirms 3.2's /api/server/reboot → 202 live.
• Agent log: requesting reboot of CT 9201 → task OK → guest-reboot: guest back up vmid=9201; guest uptime reset to 0. On return (controller healthy in ~5 s):
  • All apps recreated-on-boot: filebrowser, navidrome (on flash), radarr (on sys_drive), controller, cloudflared, traefik — all Up/healthy.
  • Drives re-bound at STABLE: flash sdc1[/felhom-data], usb sdb1[/felhom-data].
  • sys_drive still separate: pve-vm--9201--disk--2 (→ 2.5 reboot-survival PASS).
  • No rootfs placeholders at raw /mnt/felhom-{flash,usb} (both "not present").
  • Registry intact: both stable paths. Dashboard returned.
  • Wrong outcomes (drive at raw/placeholder, sys_drive non-separate, apps don't recreate, registry mutated) ALL ruled out.

3.4 — "Vezérlő újraindítása" (controller-only restart) — ⚠️ PASS-with-FINDING (live UI)

• Method: claude-in-chrome; invoked the page's restartController() handler (confirm → fetch /api/selfrestart → pollRestart). Controller back healthy in ~3 s.
• Guest NOT rebooted (uptime kept climbing, not reset) ✓. Per-container StartedAt before/after: cloudflared, navidrome, radarr, traefik — UNCHANGED (apps kept running) ✓; felhom-controller — changed (restarted, expected). Drives stayed bound; sys_drive unaffected.
• ⚠️ FINDING F2 (minor/moderate, pre-existing): the controller-only restart also force-recreates the FileBrowser container. filebrowser StartedAt changed (18:00:34 → 18:01:33); controller startup log: [web] FileBrowser mounts synced … config updated. Root cause: SyncFileBrowserMounts() (internal/web/handlers.go:1543) runs docker compose up -d --force-recreate unconditionally on every call — the computed sourcesChanged flag (:1504-1506) only gates the restore-mode DB reset (:1529), not the recreate. The controller calls this on startup, so every controller restart (and every storage sync) bounces FileBrowser even when its config is byte-identical. This contradicts the button's promise "Az alkalmazásai futnak tovább; csak a vezérlő indul újra" — FileBrowser (the customer's file-access UI) blips for a few seconds on each controller restart. Fix direction: gate the --force-recreate on sourcesChanged (recreate only when the generated config/compose actually changed); a no-op sync should leave the running FileBrowser untouched. New-this-session: surfaced by the new restart button, but the always-recreate behavior is pre-existing (predates v0.81.0). Not a data-safety issue.

Phase 3 verdict: the retired banner stays gone (3.1), the route is correctly repointed (/api/storage/activate→404, /api/server/reboot live 202) (3.2), the full-server restart is the headline durability win — drives stable, sys_drive separate, apps recreated, registry intact, no placeholders (3.3). The controller-only restart keeps the guest + customer apps running but does bounce FileBrowser (F2).

Phase 4 — host reboot + chaos — ✅ COMPLETE (4.7 deferred; F3/F4 found)

Pre-host-reboot capture: sdb1=da9e7089(usb), sdc1=81a26531(flash); onboot: 0; felhom-agent service enabled (will auto-start on host boot). navidrome on flash, radarr on sys_drive. Prediction: with onboot: 0, 9201 will NOT auto-start after the host reboot (candidate finding F3).

4.1 — plain felhom-pve host reboot — ✅ PASS (drives/apps/registry) + ⚠️ FINDING F3 (no auto-start)

• Issued reboot on felhom-pve; host returned (fresh boot, ~90 s). felhom-agent auto-started (active).
• ⚠️ FINDING F3 (moderate, pre-existing config): 9201 did NOT auto-start — it was stopped after the host reboot because pct config 9201 has onboot: 0. A managed customer guest should come back on its own after a host power-loss/reboot; as configured, the customer's entire home-server (all apps + the controller) stays down until someone manually pct starts it. Fix direction: the golden/provision path should set onboot: 1 (+ a sensible startup/boot-order) on the customer guest; audit whether any provisioned guest has onboot:1. (I started 9201 manually to continue.)
• After manual pct start 9201 (controller healthy ~5 s): drives re-bound by the agent reconcile by durable-id (uuid:81a26531 flash, uuid:da9e7089 usb) at the stable paths; sys_drive separate (disk--2); all apps recreated (navidrome on flash, radarr on sys_drive, both healthy); no rootfs placeholders; registry stable. Wrong outcomes (raw/placeholder, sys_drive non-separate, registry drift) ruled out (modulo F3 auto-start).

4.2 — /dev reshuffle (sdb↔sdc) — ✅ PASS (REAL swap, not just code-path)

• The host reboot genuinely swapped the kernel letters: before sdb1=da9e7089(usb)/sdc1=81a26531(flash); after sdb1=81a26531(flash)/sdc1=da9e7089(usb). Despite the swap, the agent bound each drive by fs-UUID: felhom-flash (81a26531, now sdb1) at /mnt/felhom-drives/felhom-flash; felhom-usb (da9e7089, now sdc1) at /mnt/felhom-drives/felhom-usb. Each NAME mapped to its own UUID at the correct stable path — the v0.37 wrong-physical-device regression is ruled out on a real letter swap.

4.5 — double-bind churn — ✅ PASS (stable, not leaking) + ⚠️ FINDING F4 (pre-existing 2×)

• Across the host reboot + multiple 20 s reconcile ticks, each drive's bind count held stable at 2 (grep -c on /proc/mounts, guest AND host) — it did not accumulate to 3,4,… so the "binds leak" wrong outcome is ruled out.
• ⚠️ FINDING F4 (minor, pre-existing): each enrolled drive is double-bound — two identical /dev/sdXN /mnt/felhom-drives/<name> ext4 … entries in /proc/mounts. The agent logs guest-attach: drive bound under shared parent (normalized to one bind … prior_binds=2), i.e. it detects 2 prior binds but the steady state is still 2, so the "normalize to one" isn't fully effective. Matches the documented pre-existing double-bind (the v0.36.x peer-group area); stable, not a leak, no data impact. Fix direction: make the normalize-to-one path converge to a single bind (or stop the duplicate rbind at its source).

4.3 — intent gate (ejected drive must NOT auto-rebind) — ✅ PASS (agent-restart variant)

• Method: to avoid a second full host reboot, the same ReassertGuestBinds intent gate (which runs on agent startup, identical to the host-reboot path) was exercised by an agent restart. Ejected felhom-usb (no app on it — navidrome migrated to flash in 1.9), systemctl restart felhom-agent.
• Agent log: reconcile: skipping non-enrolled drive (intent-gated) … da9e7089 intent=ejected (repeated each tick) AND … ce9d1d81 intent=ejected (the baseline leftover); enrolled felhom-flash re-bound. Steady state: usb 0 binds (NOT re-bound), flash bound. The ejected drive did not auto-rebind; the enrolled one did. Wrong outcome (ejected auto-rebinds) ruled out.
• Transient note: ~3 s after the agent restart, findmnt briefly showed usb still bound — a propagation lag of the eject's detach through the /mnt:rslave view + the F4 double-bind — which cleared to 0 within one tick. Not a gate failure (the gate decision is logged correctly throughout).
• Restored usb via re-register → intent→enrolled, bound at stable (now sdc1 post-swap), both drives back.

4.4 — drive yank mid-flight → mountpoint-gate — ✅ PASS (exact-endpoint, no host reboot)

• Method: systemctl stop felhom-agent (freeze binds), lazy-unmounted the flash bind on the host (yank), then POST /api/stacks/navidrome/restart (HDD_PATH=flash, now unmounted) to run the deploy belt.
• The belt logged userdata belt: drive /mnt/felhom-drives/felhom-flash not mounted — skipping ensure (held by drive gate) (internal/stacks/manager.go:143) and wrote nothing — the yanked path /mnt/felhom-drives/felhom-flash stayed empty (now resolves to pve-root[/mnt/felhom-drives], i.e. the parent/rootfs, but no app data or placeholder was created there). The controller refused to MkdirAll onto the rootfs. Wrong outcome (app data written to rootfs / placeholder created) ruled out.
• Recovery: systemctl start felhom-agent → flash re-bound at stable (sdb1[/felhom-data]), navidrome stayed healthy throughout, both intents enrolled, no leftover placeholder. Reconnect recovered cleanly.

4.6 — concurrency — ✅ PASS (exact-endpoint)

• (b) eject-in-use is WARNED: impact?where=…felhom-flash → apps:["Navidrome"] — the in-use app is named before any eject (the type-to-confirm warning surface), so eject-in-use is warned, never silent.
• (a) concurrent backup + deregister + re-register (raced against each other): remove→302, backup→ok, register→ok. Post-race the registry is consistent — exactly one flash + one usb entry (no duplicates, no corruption); both drives bound (usb sdc1, flash sdb1); navidrome + radarr healthy. Wrong outcomes (concurrent corruption / silent app breakage) ruled out.

4.7 — CAPSTONE (destroy + re-provision 9201) — ⏸️ DEFERRED (operator-gated; not run)

• Explicitly operator-gated and requires the golden volid + retrieval passphrase, which were not provided for this autonomous run. Destroying the live demo guest is the campaign's most destructive step; per the "confirm before irreversible ops" guardrail I did not run it without an operator go-ahead. Compensating coverage: the exact original-bug scenario (stable registration with no placeholder, drives live, sys_drive separate, no banner) is already proven clean via 1.2 (register → stable, no placeholder), 3.3 (full-server restart recovery), and 4.1/4.2 (host-reboot + real sdb↔sdc swap recovery). To close 4.7: a supervised run with the operator supplying the golden volid + passphrase, then re-enroll via "Regisztrálás" and assert the clean end-state.

Phase 4 verdict: durability holds — host-reboot drive re-bind by UUID survives a real /dev letter swap (4.2), the intent gate skips ejected drives (4.3), the mountpoint-gate blocks rootfs writes on a drive yank (4.4), and concurrency doesn't corrupt (4.6). Two findings: F3 (onboot:0 → guest doesn't auto-start after host reboot — the highest-impact finding) and F4 (stable-but-present double-bind). The capstone (4.7) is deferred pending operator credentials.

Phase 5 — restore to clean baseline — ✅ COMPLETE

• Removed the test apps (navidrome stop+remove, radarr stop+remove) → only infra remains (felhom-controller, traefik, cloudflared, filebrowser — all healthy), matching the Phase-0 baseline.
• Loop devices loop0/loop1 were already gone (cleared by the host reboot); scratch imgs + dir removed. All fill files removed (2.2). No decommission soft-markers left (everything reconnected; both intents enrolled).
• Final assertions (all ✅): both drives live at stable with correct device/size (felhom-flash /dev/sdb1 125.7 GB, felhom-usb /dev/sdc1 983.3 GB — letters are post-swap, bound by UUID); registry = exactly 2 stable paths; /mnt/sys_drive separate (disk--2, 50 G, [PASS] System data path); dashboard GET / 200; settings UI clean — 0 banner occurrences, both "Vezérlő újraindítása" + "Kiszolgáló újraindítása" cards present. The PBS floor was not needed (nothing wedged). 9201 left at onboot:0 (original state — F3 is reported, not fixed).

---

DELIVERABLE SUMMARY

1. Findings (root-caused; report-don't-fix — none implemented)

ID	Sev	New?	Summary	Evidence / source	Fix direction
F3	Moderate	surfaced this session (pre-existing config)	Guest 9201 has onboot:0 → does NOT auto-start after a host reboot. The whole customer home-server (apps + controller) stays down until a manual pct start.	4.1: host rebooted → pct status 9201 = stopped; pct config 9201 → onboot: 0	Golden/provision should set onboot:1 (+ boot order) on customer guests; audit all provisioned guests.
F2	Minor/Mod	surfaced this session (behavior pre-existing)	"Vezérlő újraindítása" (controller-only restart) also force-recreates the FileBrowser container every time, even on a no-op sync — contradicts the "your apps keep running" promise.	3.4: filebrowser StartedAt changed on controller restart; SyncFileBrowserMounts runs docker compose up -d --force-recreate unconditionally — internal/web/handlers.go:1543; sourcesChanged (:1504) only gates the DB reset (:1529)	Gate the --force-recreate on sourcesChanged; skip recreate when config/compose is byte-identical.
F1	Minor	new discovery (pre-existing)	restic binary ships in the controller image as dead weight. All restic code moved to the agent during de-privileging, but Dockerfile:58 still apt installs it.	1.11: /usr/bin/restic present; no Go code execs it (only "moved to agent" comments + vestigial config/migration)	Drop restic from the Dockerfile install list + the dead Restic* config fields.
F4	Minor	pre-existing (documented v0.36.x area)	Each enrolled drive is double-bound (2 identical /proc/mounts entries). Stable, not leaking; agent logs "normalized to one bind (prior_binds=2)" but steady state is 2.	4.5: grep -c = 2 (guest+host), stable across ticks; brief detach transient in 4.3	Make normalize-to-one converge to a single bind / stop the duplicate rbind at source.

Observations (not findings): (a) a named-volume app (navidrome) deployed onto a drive creates no recovery unit / appdata namespace on that drive — likely by-design (recovery units key on appdata-bind apps). (b) The free-form "add storage path" form would accept /mnt/sys_drive (a real mountpoint) — consistent with the SSD-fallback, not the raw-drive bug.

2. Method per user-facing test

• claude-in-chrome (live UI): 1.5 (disk-card badges, screenshot), 3.3 ("Kiszolgáló újraindítása" click → guest reboot; confirm() neutralized, page handler invoked), 3.4 ("Vezérlő újraindítása" click), 5 (final settings UI). No F9-style bypass — the real handlers/endpoints ran the full server pipeline.
• Exact-endpoint (= the UI's own request; demo has no password so CSRF is a no-op): 1.1, 1.2, 1.6, 1.7, 1.9, 1.10, 1.11, 1.12, 2.x, 3.1/3.2, 4.3, 4.4, 4.6. The drive ops invoked the exact /api/storage/* and /api/stacks/* endpoints the buttons call.
• Host/agent-level (real infrastructure): 2.2 (fill), 2.3 (vzdump), 4.1/4.2 (host reboot + real /dev swap), 4.3 (agent restart → ReassertGuestBinds), 4.4 (agent stop + unmount).

3. Phase-0 PBS floor

• felhom-pbs:backup/ct/9201/2026-06-23T17:28:25Z (datastore felhom-spike). Not used — nothing wedged; every destructive step was reversed in-band.

4. End state

• 9201 restored to a healthy baseline: infra-only apps healthy, both drives live at stable (correct device/size), /mnt/sys_drive separate + warning-free, registry = 2 stable paths, settings UI clean (no banner, both restart buttons). Left intentionally at onboot:0 (original state; F3 reported not fixed). Two pre-existing absent drive-intents (277a2179 enrolled-absent, ce9d1d81 ejected) remain — they predate the campaign (present in the Phase-0 baseline), not introduced here.

5. Coverage gaps

• 1.3 / 1.4 / 1.8 (loop-device init/attach/wipe): not executable — the agent's block-device whitelist (felhom-agent internal/storage/validate.go:124) refuses non-whitelisted devices (a safety PASS), and no spare whitelisted disk exists (the 2 real drives hold un-floored data). Stable-registration covered by 1.2 live + unit tests. Needs a real spare disk in a supervised run.
• 4.7 (capstone destroy + re-provision): deferred — operator-gated, needs golden volid + retrieval passphrase (not provided this run). Compensating coverage via 1.2/3.3/4.1.
• 3.2 auth/CSRF rejection: can't be shown on the passwordless demo (gate is a no-op); wrapping is source-verified.
• 4.2 was a REAL /dev swap (better than the planned code-path fallback) — no gap there.

---

Campaign result (overview)

• Phases 0–5 complete. The three shipped features hold up: the stable-vs-raw path model (v0.78–0.80) is correct across register/deregister/disk-card/eject/decommission/migrate/backup (Phase 1); the /mnt/sys_drive split (agent v0.40.0) isolates disk-full, is backed up, and survives reboots (Phase 2); the retired banner + standalone server-restart (v0.81.0) work — full-server restart recovers drives at stable, sys_drive separate, apps recreated, registry intact (Phase 3); and durability survives a real host reboot with a genuine sdb↔sdc swap, the intent gate, the mountpoint-gate, and concurrency (Phase 4).
• 4 findings, all reported-not-fixed (see DELIVERABLE SUMMARY §1): F3 onboot:0 (guest won't auto-start after host reboot — highest impact), F2 controller restart bounces FileBrowser, F1 restic dead-weight binary, F4 stable double-bind. No data-loss, no path-escape, no silent corruption found. Demo restored to a healthy baseline.