felhom.eu/documentation/tests/TEST-REPORT-stable-path-sysdrive-restart-2026-06-23.md

TEST REPORT — NO-MERCY CAMPAIGN: stable-path model + /mnt/sys_drive + server restart

Date: 2026-06-23 · Class: destructive test campaign (operator-authorized, DEMO only) Targets: guest 9201 (demo-felhom) on felhom-pve (192.168.0.162, N100, PVE 9) · controller v0.81.0 · agent v0.40.0 · hub hub.felhom.eu Rule: report-don't-fix. No product-code changes. Findings get evidence + file:line + a proposed fix direction (NOT implemented).

Status legend

PASS = effect asserted AND wrong-outcome ruled out · FAIL = wrong outcome observed · INCONCLUSIVE = could not assert cleanly (reason given).

---

Phase 0 — floor + baseline (non-destructive) — ✅ COMPLETE

PBS floor (hard restore floor)

• Snapshot id: felhom-pbs:backup/ct/9201/2026-06-23T17:28:25Z (PBS datastore felhom-spike @ felhom@pbs!n100@192.168.0.180). Backup finished successfully (00:00:15, incremental, reused 89.9%).
• Used? Not yet (recorded for restore if a later phase wedges 9201).
• Note: felhom-pbs storage resolves to the PBS server on 192.168.0.180 datastore felhom-spike. Writing the floor snapshot there is normal PBS operation (the demo's nightly job already does) — it does not mutate/break 180 as a host, consistent with the guardrail.

Baseline — HOST (pct config 9201)

• rootfs: local-lvm:vm-9201-disk-0,size=32G · mp0: vm-9201-disk-1,mp=/var/lib/docker,backup=1,size=200G
• mp1: vm-9201-disk-2,mp=/mnt/sys_drive,backup=1,size=50G · mp8: /mnt/felhom-drives (bind) · mp9: …/bootstrap → /etc/felhom-bootstrap,ro=1
• onboot: 0 ⚠️ — flagged for Phase 4.1 (the spec asserts 9201 auto-starts on host reboot; with onboot: 0 a plain host reboot will not auto-start it). Pre-existing config.
• Drives (host lsblk -f): felhom-usb = /dev/sdb1 UUID da9e7089-… (916 GB HDD); felhom-flash = /dev/sdc1 UUID 81a26531-… (117 GB USB). Both mounted host-side at raw /mnt/<name> AND bind-mounted to stable /mnt/felhom-drives/<name> under the shared parent.

Baseline — GUEST

• findmnt /mnt: /mnt/sys_drive = pve-vm--9201--disk--2 (distinct device); /mnt/felhom-drives/{flash,usb} = sdc1[/felhom-data] / sdb1[/felhom-data].
• Invariant (bug signature) — PASS: no rootfs placeholder dirs at raw /mnt/felhom-flash / /mnt/felhom-usb inside the guest (not present (good) for both).

Baseline — CONTAINER (felhom-controller)

• settings.json storage paths = ONLY stable /mnt/felhom-drives/felhom-flash + /mnt/felhom-drives/felhom-usb (no raw entries). Both findmnt to the correct sdX1[/felhom-data]. /mnt/sys_drive = disk-2.
• Container / = docker-data overlay (197G, the 200G mp0). /mnt/sys_drive separate (50G).

Baseline — AGENT state (/var/lib/felhom-agent/)

• guest-binds.json → 9201: [uuid:81a26531 (flash), uuid:ce9d1d81 (absent), uuid:da9e7089 (usb)].
• drive-intents.json: 277a2179→enrolled (absent), 81a26531→enrolled (flash), ce9d1d81→ejected (absent), da9e7089→enrolled (usb). ⚠️ Two leftover intents for physically-absent drives (277a2179 enrolled-absent, ce9d1d81 ejected) — relevant to Phase 4.3 intent-gate.
• Agent version: felhom-agent 0.40.0.

Deployed apps at baseline

• Only base infra running: filebrowser, felhom-controller, cloudflared, traefik (all healthy). No customer apps deployed — tests needing "an app on a drive" will deploy a small one.

Scratch

• Loop devices: LOOP_A=/dev/loop0 (/root/nomercy-scratch/loopA.img, 2G), LOOP_B=/dev/loop1 (/root/nomercy-scratch/loopB.img, 2G). For teardown in Phase 5.

Phase 2.3 (opportunistic, captured during the floor vzdump) — ✅ PASS

• The floor vzdump 9201 log: including mount point mp1 ('/mnt/sys_drive') in backup and the proxmox-backup-client invocation carries --include-dev /mnt/vzsnap0/./mnt/sys_drive. mp1 is in the archive. Wrong outcome (mp1 excluded / backup=0) ruled out. (Optional restore-to-scratch deferred.)

---

Phase 1 — stable-vs-raw path model — ⏳ IN PROGRESS

Method: mix of live-UI (claude-in-chrome on felhom.demo-felhom.eu) and exact-endpoint (curl to the in-guest controller 127.0.0.1:8080). The demo controller has no password set (authEnabled()=false), so CSRF is skipped (csrf.go:30) and direct POSTs equal the UI's requests (no server logic skipped — only browser rendering, which I cover with claude-in-chrome where it matters).

1.1 — registry holds only stable paths — ✅ PASS (exact-endpoint + UI)

• settings.json storage_paths = ONLY /mnt/felhom-drives/{felhom-flash,felhom-usb}. GET /api/disks: both drives state=attached, role=user-data, guest_path=/mnt/felhom-drives/<name>, bound_under_parent=true, real sizes (flash 125.7 GB, usb 983 GB). No raw /mnt/<name> registry entry; no rootfs placeholder. Wrong outcome (raw entry / placeholder) ruled out.

1.2 — deregister → re-add via "Regisztrálás" — ✅ PASS (exact-endpoint = UI request)

• Deregister-only of felhom-usb via POST /settings/storage/remove (the "Eltávolítás" form, field storage_path, stable path) → 302; registry left only flash; drive stayed mounted at stable (sdb1[/felhom-data]) — not ejected; no rootfs placeholder.
• Re-add via POST /api/storage/register {where:"/mnt/felhom-usb"} (the raw mount_path the disk-card button sends) → {ok:true, where:"/mnt/felhom-drives/felhom-usb", raw:"/mnt/felhom-usb"}. Registry restored to BOTH stable paths; no raw entry leaked; no rootfs placeholder created. The v0.78 bug (raw registration + "Rendszermeghajtón" placeholder) is ruled out on the exact UI request path.

1.3 — attach existing (LOOP_A) — ⚠️ INCONCLUSIVE (by-design block; compensating coverage)

1.4 — init/format+enroll (LOOP_B) — ⚠️ INCONCLUSIVE (by-design block; compensating coverage)

1.8 — wipe (LOOP_A) — ⚠️ INCONCLUSIVE (destructive device-wipe not executed)

• Loop devices cannot drive the real init/attach/wipe flow: the agent refuses non-whitelisted block devices — POST /api/storage/init {device:/dev/loop1} → HTTP 400: storage: refusing to operate on non-whitelisted block device "/dev/loop1" (felhom-agent internal/storage/validate.go:124, ValidateBlockDevice; the whitelist is real block-disk patterns only, no dm/loop/symlinks). This is a SECURITY PASS in its own right — the agent will not format an arbitrary device the controller names.
• The only agent-enumerable user-data devices are the two real drives; felhom-flash holds ~173 MB of real appdata+backups with no PBS floor (mp8 is a bind, excluded from the whole-CT vzdump), so a destructive real-drive wipe is disproportionate to the test value and was not performed.
• Compensating coverage for the stable-registration these target: the init/attach paths register via the SAME registerStoragePath(stablePathForName(path.Base(where)), …) proven live in 1.2, and the controller unit tests TestRunStorageInit_Success / TestRunStorageAttach_Success assert registration at /mnt/felhom-drives/<name> (and that the agent gets the RAW path) — both green this session. Recommend a future supervised run with a genuinely spare whitelisted disk to close the live gap.

1.5 — disk card shows stable path + correct badge — ✅ PASS (live UI, claude-in-chrome)

• Both cards render the stable sub-line (/mnt/felhom-drives/felhom-flash, /mnt/felhom-drives/felhom-usb), correct device/size (ext4 · /dev/sdc1[/felhom-data] · Flash Drive 0.2/117.1 GB; ext4 · /dev/sdb1[/felhom-data] · HD710 PRO 0.0/915.8 GB), badge "Regisztrálva" + "Aktív" — NOT the v0.79 spurious "Nem regisztrált", NOT "Rendszermeghajtón". Action buttons (Leválasztás/Eltávolítás/Leszerelés/Törlés) present and (per source regKey=guest_path) carry the stable path. Screenshot captured.

1.12 — free-form raw add probe (the parked gap) — ✅ PASS (no symptom reproduced) (exact-endpoint)

• Raw /mnt/felhom-flash does not exist inside the container (the raw host mount is never bound into the guest; only the stable parent /mnt/felhom-drives is, via mp8 + the /mnt:rslave bind). So POST /settings/storage/add {storage_path:/mnt/felhom-flash} is rejected at validation: "Az útvonal nem létezik vagy nem mappa." (handlers.go:1264 exists-and-dir; :1272 is-mountpoint). Registry unchanged (still exactly the 2 stable paths). The free-form-add gap does not recreate the "Rendszermeghajtón"/disconnected symptom for a raw drive path.
• Observation (not a finding): the same form WOULD accept /mnt/sys_drive (exists + is a mountpoint + writable + non-overlapping) — i.e. a user could free-form-register the system SSD volume as a user-data path. That is consistent with the documented SSD-only fallback (Phase 2.4), not the raw-drive bug; noted for awareness, no symptom.

1.6 — deploy app onto a drive → HDD_PATH stable — ✅ PASS (exact-endpoint = UI deploy)

• Deployed navidrome (lightest drive-backed app, 256M) via POST /api/stacks/navidrome/deploy {values:{DOMAIN,SUBDOMAIN:navtest,HDD_PATH:/mnt/felhom-drives/felhom-usb}} → healthy. app.yaml HDD_PATH=/mnt/felhom-drives/felhom-usb (stable). Container media bind /mnt/felhom-drives/felhom-usb/userdata/media/music → /music (stable drive namespace, dir 2775 g1000). Data not on rootfs; HDD_PATH not raw. (navidrome's config/DB is a named Docker volume on the docker-data disk by catalog design — only its media is drive-bound.)

1.7 — eject removes the stable entry cleanly + impact names apps — ✅ PASS (exact-endpoint)

• Impact: GET /api/storage/impact?where=/mnt/felhom-drives/felhom-usb → apps:["Navidrome"]; felhom-flash → apps:[]. Names the right apps (wrong outcome "no apps when an app uses it" ruled out).
• Eject (Leválasztás) on felhom-flash (no app, reversible): agent ejected the raw /mnt/felhom-flash (mapped from stable via agentWhere); registry left ONLY felhom-usb — the felhom-flash stable entry removed cleanly, no orphan (the v0.80 bug ruled out); bind detached, raw host mount preserved. Restored via POST /api/storage/register → re-registered stable + re-bound (sdc1[/felhom-data]).

1.8 — wipe — ⚠️ INCONCLUSIVE (see 1.3/1.4 — no safe whitelisted target; device-wipe not executed)

• The wipe's confirm/impact GATE is otherwise exercised: handleStorageWipe requires mount_name == path.Base(where) (storage_handlers.go:558) and ejects via agentWhere then formats via the agent's two-step confirm — the same mapping proven in 1.7/1.10. The destructive mkfs itself was not run (loop blocked by the whitelist; real drives hold un-floored data).

1.9 — migrate-app flips HDD_PATH stable→stable — ✅ PASS (exact-endpoint)

• POST /api/storage/migrate-app {app:navidrome, target:/mnt/felhom-drives/felhom-flash} → job ran validate→copy→verify→flip→cleanup (phase=done, unit cleaned). navidrome HDD_PATH flipped to /mnt/felhom-drives/felhom-flash (stable); media bind now on flash; navidrome healthy; no raw leak; source cleaned only AFTER verify. Wrong outcomes (source deleted before verify / HDD_PATH left raw/old) ruled out. (navidrome now lives on felhom-flash — removed in Phase 5.)

1.10 — self-serve decommission → agent op via raw + app missing storage — ✅ PASS (exact-endpoint)

• POST /api/storage/decommission {where:stable-usb, mode:anyway, mount_name:felhom-usb} → {decommissioned:true, stopped_apps:["Navidrome"]}. Agent intent da9e7089→decommissioned; agent log shows the op on the RAW path (where=/mnt/felhom-usb, guest-detach … fully unmounted) — wrong outcome (agent called with the unmapped stable path) ruled out. navidrome gate-stopped.
• Restored: POST /api/storage/reconnect → {reenrolled:true, restarted:["navidrome"]}, intent→enrolled, re-bound at stable, navidrome healthy.

1.11 — backup lands under the stable namespace — ✅ PASS (path-correctness) + FINDING (restic) (exact-endpoint)

• POST /api/backup/run (the app-data DB-dump backup) ran successfully; for navidrome it correctly found 0 databases (no DB container — SQLite in a named volume), a no-op success, not a failure.
• Path correctness: all existing backup artifacts live under the stable namespace /mnt/felhom-drives/felhom-flash/backups/primary/<app>/ (recovery units: compose/ + manifest.json), never a raw/rootfs path. Wrong outcome (backup under raw/rootfs) ruled out.
• ⚠️ FINDING F1 (minor, pre-existing): the prompt's "restic repo" premise is stale, and the restic binary is dead weight in the image. Disk-tier restic backup was moved to the host agent in the de-privileging; the controller source no longer execs restic (all references are "moved to agent" comments + vestigial config.go ResticPasswordFile/Schedule fields + the settings.go migrateResticToRsync one-way migration). But Dockerfile:58 still apt installs restic (and line 42 comments "for backup operations"), so /usr/bin/restic ships in the controller image unused. Fix direction: drop restic from the Dockerfile install list (and the dead Restic* config fields) to shed image weight + remove a confusing "disk-tier is still here" signal. Not a functional bug.
• Observation: a named-volume app (navidrome) deployed onto a drive created no appdata/ or backups/ namespace nor a recovery unit on that drive (only the shared userdata/ skeleton). Recovery units appear keyed to appdata-bind apps; named-volume-only apps rely on the DB-dump/volume-tar path instead. Flagged for awareness; likely by-design, not verified as a bug.

Phase 1 verdict: the stable-vs-raw path model holds across register/deregister/disk-card/eject/ decommission/migrate/backup. The v0.78 (raw register), v0.79 (spurious unregistered badge), and v0.80 (orphaned stable entry on eject) bugs are all ruled out live. The only gaps are the loop-device init/attach/wipe (blocked by the agent's block-device whitelist — a safety PASS) and the restic dead-weight finding (F1).

Phase 2 — /mnt/sys_drive third volume — ⏳ PENDING (2.3 PASS captured)

Phase 3 — server restart + retired banner — ⏳ PENDING

Phase 4 — host reboot + chaos — ⏳ PENDING

Phase 5 — restore to clean baseline — ⏳ PENDING

---

Findings (running)

None yet beyond the flagged baseline observations (onboot:0, leftover absent intents).

Method notes

• User-facing actions: method stated per test (claude-in-chrome vs exact-endpoint). The demo controller renders /settings unauthenticated and is reachable in-guest at 127.0.0.1:8080 (curl in-container).