felhom.eu/documentation/tests/TEST-REPORT-stable-path-sysdrive-restart-2026-06-23.md

# TEST REPORT — NO-MERCY CAMPAIGN: stable-path model + /mnt/sys_drive + server restart

**Date:** 2026-06-23 · **Class:** destructive test campaign (operator-authorized, DEMO only)
**Targets:** guest **9201** (demo-felhom) on **felhom-pve** (192.168.0.162, N100, PVE 9) ·
controller **v0.81.0** · agent **v0.40.0** · hub `hub.felhom.eu`
**Rule:** report-don't-fix. No product-code changes. Findings get evidence + `file:line` + a proposed
fix direction (NOT implemented).

## Status legend
PASS = effect asserted AND wrong-outcome ruled out · FAIL = wrong outcome observed ·
INCONCLUSIVE = could not assert cleanly (reason given).

---

## Phase 0 — floor + baseline (non-destructive) — ✅ COMPLETE

### PBS floor (hard restore floor)
- **Snapshot id:** `felhom-pbs:backup/ct/9201/2026-06-23T17:28:25Z` (PBS datastore `felhom-spike` @
  `felhom@pbs!n100@192.168.0.180`). Backup finished successfully (00:00:15, incremental, reused 89.9%).
- **Used?** Not yet (recorded for restore if a later phase wedges 9201).
- Note: `felhom-pbs` storage resolves to the PBS server on 192.168.0.180 datastore `felhom-spike`.
  Writing the floor snapshot there is normal PBS operation (the demo's nightly job already does) — it
  does **not** mutate/break 180 as a host, consistent with the guardrail.

### Baseline — HOST (`pct config 9201`)
- `rootfs: local-lvm:vm-9201-disk-0,size=32G` · `mp0: vm-9201-disk-1,mp=/var/lib/docker,backup=1,size=200G`
- `mp1: vm-9201-disk-2,mp=/mnt/sys_drive,backup=1,size=50G` · `mp8: /mnt/felhom-drives` (bind) ·
  `mp9: …/bootstrap → /etc/felhom-bootstrap,ro=1`
- **`onboot: 0`** ⚠️ — flagged for Phase 4.1 (the spec asserts 9201 auto-starts on host reboot; with
  `onboot: 0` a plain host reboot will **not** auto-start it). Pre-existing config.
- Drives (host `lsblk -f`): `felhom-usb` = `/dev/sdb1` UUID `da9e7089-…` (916 GB HDD);
  `felhom-flash` = `/dev/sdc1` UUID `81a26531-…` (117 GB USB). Both mounted host-side at raw
  `/mnt/<name>` AND bind-mounted to stable `/mnt/felhom-drives/<name>` under the shared parent.

### Baseline — GUEST
- `findmnt /mnt`: `/mnt/sys_drive` = `pve-vm--9201--disk--2` (distinct device); `/mnt/felhom-drives/{flash,usb}`
  = `sdc1[/felhom-data]` / `sdb1[/felhom-data]`.
- **Invariant (bug signature) — PASS:** no rootfs placeholder dirs at raw `/mnt/felhom-flash` /
  `/mnt/felhom-usb` inside the guest (`not present (good)` for both).

### Baseline — CONTAINER (felhom-controller)
- `settings.json` storage paths = ONLY stable `/mnt/felhom-drives/felhom-flash` + `/mnt/felhom-drives/felhom-usb`
  (no raw entries). Both `findmnt` to the correct `sdX1[/felhom-data]`. `/mnt/sys_drive` = disk-2.
- Container `/` = docker-data overlay (197G, the 200G mp0). `/mnt/sys_drive` separate (50G).

### Baseline — AGENT state (`/var/lib/felhom-agent/`)
- `guest-binds.json` → `9201`: `[uuid:81a26531 (flash), uuid:ce9d1d81 (absent), uuid:da9e7089 (usb)]`.
- `drive-intents.json`: `277a2179→enrolled (absent)`, `81a26531→enrolled (flash)`,
  `ce9d1d81→ejected (absent)`, `da9e7089→enrolled (usb)`. ⚠️ Two leftover intents for physically-absent
  drives (`277a2179` enrolled-absent, `ce9d1d81` ejected) — relevant to Phase 4.3 intent-gate.
- Agent version: `felhom-agent 0.40.0`.

### Deployed apps at baseline
- Only base infra running: `filebrowser`, `felhom-controller`, `cloudflared`, `traefik` (all healthy).
  **No customer apps deployed** — tests needing "an app on a drive" will deploy a small one.

### Scratch
- Loop devices: **LOOP_A=`/dev/loop0`** (`/root/nomercy-scratch/loopA.img`, 2G),
  **LOOP_B=`/dev/loop1`** (`/root/nomercy-scratch/loopB.img`, 2G). For teardown in Phase 5.

### Phase 2.3 (opportunistic, captured during the floor vzdump) — ✅ PASS
- The floor `vzdump 9201` log: `including mount point mp1 ('/mnt/sys_drive') in backup` and the
  `proxmox-backup-client` invocation carries `--include-dev /mnt/vzsnap0/./mnt/sys_drive`. mp1 is in the
  archive. Wrong outcome (mp1 excluded / `backup=0`) ruled out. *(Optional restore-to-scratch deferred.)*

---

## Phase 1 — stable-vs-raw path model — ⏳ IN PROGRESS

**Method:** mix of live-UI (claude-in-chrome on `felhom.demo-felhom.eu`) and exact-endpoint (curl to
the in-guest controller `127.0.0.1:8080`). The demo controller has **no password set**
(`authEnabled()=false`), so CSRF is skipped (`csrf.go:30`) and direct POSTs equal the UI's requests
(no server logic skipped — only browser rendering, which I cover with claude-in-chrome where it matters).

### 1.1 — registry holds only stable paths — ✅ PASS (exact-endpoint + UI)
- `settings.json` storage_paths = ONLY `/mnt/felhom-drives/{felhom-flash,felhom-usb}`. `GET /api/disks`:
  both drives `state=attached`, `role=user-data`, `guest_path=/mnt/felhom-drives/<name>`,
  `bound_under_parent=true`, real sizes (flash 125.7 GB, usb 983 GB). No raw `/mnt/<name>` registry entry;
  no rootfs placeholder. Wrong outcome (raw entry / placeholder) ruled out.

### 1.2 — deregister → re-add via "Regisztrálás" — ✅ PASS (exact-endpoint = UI request)
- Deregister-only of felhom-usb via `POST /settings/storage/remove` (the "Eltávolítás" form, field
  `storage_path`, stable path) → 302; registry left only flash; **drive stayed mounted** at stable
  (`sdb1[/felhom-data]`) — not ejected; no rootfs placeholder.
- Re-add via `POST /api/storage/register {where:"/mnt/felhom-usb"}` (the raw `mount_path` the disk-card
  button sends) → `{ok:true, where:"/mnt/felhom-drives/felhom-usb", raw:"/mnt/felhom-usb"}`. Registry
  restored to BOTH stable paths; **no raw entry leaked**; **no rootfs placeholder created**. The v0.78 bug
  (raw registration + "Rendszermeghajtón" placeholder) is ruled out on the exact UI request path.

### 1.3 — attach existing (LOOP_A) — ⚠️ INCONCLUSIVE (by-design block; compensating coverage)
### 1.4 — init/format+enroll (LOOP_B) — ⚠️ INCONCLUSIVE (by-design block; compensating coverage)
### 1.8 — wipe (LOOP_A) — ⚠️ INCONCLUSIVE (destructive device-wipe not executed)
- **Loop devices cannot drive the real init/attach/wipe flow:** the agent refuses non-whitelisted block
  devices — `POST /api/storage/init {device:/dev/loop1}` →
  `HTTP 400: storage: refusing to operate on non-whitelisted block device "/dev/loop1"`
  (felhom-agent `internal/storage/validate.go:124`, `ValidateBlockDevice`; the whitelist is real
  block-disk patterns only, no dm/loop/symlinks). **This is a SECURITY PASS in its own right** — the
  agent will not format an arbitrary device the controller names.
- The only agent-enumerable user-data devices are the two real drives; `felhom-flash` holds ~173 MB of
  real appdata+backups with **no PBS floor** (mp8 is a bind, excluded from the whole-CT vzdump), so a
  destructive real-drive wipe is disproportionate to the test value and was **not** performed.
- **Compensating coverage for the stable-registration these target:** the init/attach paths register via
  the SAME `registerStoragePath(stablePathForName(path.Base(where)), …)` proven live in 1.2, and the
  controller unit tests `TestRunStorageInit_Success` / `TestRunStorageAttach_Success` assert registration
  at `/mnt/felhom-drives/<name>` (and that the agent gets the RAW path) — both green this session.
  Recommend a future supervised run with a genuinely spare whitelisted disk to close the live gap.

### 1.5 — disk card shows stable path + correct badge — ✅ PASS (live UI, claude-in-chrome)
- Both cards render the **stable** sub-line (`/mnt/felhom-drives/felhom-flash`,
  `/mnt/felhom-drives/felhom-usb`), correct device/size (`ext4 · /dev/sdc1[/felhom-data] · Flash Drive`
  0.2/117.1 GB; `ext4 · /dev/sdb1[/felhom-data] · HD710 PRO` 0.0/915.8 GB), badge **"Regisztrálva"** +
  "Aktív" — NOT the v0.79 spurious "Nem regisztrált", NOT "Rendszermeghajtón". Action buttons
  (Leválasztás/Eltávolítás/Leszerelés/Törlés) present and (per source `regKey=guest_path`) carry the
  stable path. Screenshot captured.

### 1.12 — free-form raw add probe (the parked gap) — ✅ PASS (no symptom reproduced) (exact-endpoint)
- Raw `/mnt/felhom-flash` does **not exist inside the container** (the raw host mount is never bound into
  the guest; only the stable parent `/mnt/felhom-drives` is, via mp8 + the `/mnt:rslave` bind). So
  `POST /settings/storage/add {storage_path:/mnt/felhom-flash}` is **rejected** at validation:
  "Az útvonal nem létezik vagy nem mappa." (`handlers.go:1264` exists-and-dir; `:1272` is-mountpoint).
  Registry unchanged (still exactly the 2 stable paths). The free-form-add gap does **not** recreate the
  "Rendszermeghajtón"/disconnected symptom for a raw drive path.
- **Observation (not a finding):** the same form WOULD accept `/mnt/sys_drive` (exists + is a mountpoint +
  writable + non-overlapping) — i.e. a user could free-form-register the system SSD volume as a user-data
  path. That is consistent with the documented SSD-only fallback (Phase 2.4), not the raw-drive bug; noted
  for awareness, no symptom.

### 1.6 — deploy app onto a drive → HDD_PATH stable — ✅ PASS (exact-endpoint = UI deploy)
- Deployed **navidrome** (lightest drive-backed app, 256M) via `POST /api/stacks/navidrome/deploy`
  `{values:{DOMAIN,SUBDOMAIN:navtest,HDD_PATH:/mnt/felhom-drives/felhom-usb}}` → healthy. `app.yaml`
  `HDD_PATH=/mnt/felhom-drives/felhom-usb` (stable). Container media bind
  `/mnt/felhom-drives/felhom-usb/userdata/media/music → /music` (stable drive namespace, dir 2775 g1000).
  Data not on rootfs; HDD_PATH not raw. (navidrome's config/DB is a named Docker volume on the docker-data
  disk by catalog design — only its media is drive-bound.)

### 1.7 — eject removes the stable entry cleanly + impact names apps — ✅ PASS (exact-endpoint)
- **Impact:** `GET /api/storage/impact?where=/mnt/felhom-drives/felhom-usb` → `apps:["Navidrome"]`;
  felhom-flash → `apps:[]`. Names the right apps (wrong outcome "no apps when an app uses it" ruled out).
- **Eject** (Leválasztás) on felhom-flash (no app, reversible): agent ejected the **raw** `/mnt/felhom-flash`
  (mapped from stable via `agentWhere`); registry left ONLY felhom-usb — the felhom-flash **stable** entry
  removed **cleanly, no orphan** (the v0.80 bug ruled out); bind detached, raw host mount preserved.
  Restored via `POST /api/storage/register` → re-registered stable + re-bound (`sdc1[/felhom-data]`).

### 1.8 — wipe — ⚠️ INCONCLUSIVE (see 1.3/1.4 — no safe whitelisted target; device-wipe not executed)
- The wipe's confirm/impact GATE is otherwise exercised: `handleStorageWipe` requires
  `mount_name == path.Base(where)` (storage_handlers.go:558) and ejects via `agentWhere` then formats via
  the agent's two-step confirm — the same mapping proven in 1.7/1.10. The destructive `mkfs` itself was not
  run (loop blocked by the whitelist; real drives hold un-floored data).

### 1.9 — migrate-app flips HDD_PATH stable→stable — ✅ PASS (exact-endpoint)
- `POST /api/storage/migrate-app {app:navidrome, target:/mnt/felhom-drives/felhom-flash}` → job ran
  validate→copy→verify→flip→**cleanup** (phase=done, unit `cleaned`). navidrome `HDD_PATH` flipped to
  `/mnt/felhom-drives/felhom-flash` (stable); media bind now on flash; navidrome healthy; **no raw leak**;
  source cleaned only AFTER verify. Wrong outcomes (source deleted before verify / HDD_PATH left raw/old)
  ruled out. (navidrome now lives on felhom-flash — removed in Phase 5.)

### 1.10 — self-serve decommission → agent op via raw + app missing storage — ✅ PASS (exact-endpoint)
- `POST /api/storage/decommission {where:stable-usb, mode:anyway, mount_name:felhom-usb}` →
  `{decommissioned:true, stopped_apps:["Navidrome"]}`. Agent intent `da9e7089→decommissioned`; agent log
  shows the op on the **RAW** path (`where=/mnt/felhom-usb`, `guest-detach … fully unmounted`) — wrong
  outcome (agent called with the unmapped stable path) ruled out. navidrome gate-stopped.
- Restored: `POST /api/storage/reconnect` → `{reenrolled:true, restarted:["navidrome"]}`, intent→enrolled,
  re-bound at stable, navidrome healthy.

### 1.11 — backup lands under the stable namespace — ✅ PASS (path-correctness) + FINDING (restic) (exact-endpoint)
- `POST /api/backup/run` (the app-data **DB-dump** backup) ran successfully; for navidrome it correctly
  found **0 databases** (no DB container — SQLite in a named volume), a no-op success, not a failure.
- **Path correctness:** all existing backup artifacts live under the **stable** namespace
  `/mnt/felhom-drives/felhom-flash/backups/primary/<app>/` (recovery units: `compose/` + `manifest.json`),
  never a raw/rootfs path. Wrong outcome (backup under raw/rootfs) ruled out.
- ⚠️ **FINDING F1 (minor, pre-existing): the prompt's "restic repo" premise is stale, and the restic
  binary is dead weight in the image.** Disk-tier restic backup was moved to the host agent in the
  de-privileging; the controller source no longer execs restic (all references are "moved to agent"
  comments + vestigial `config.go` `ResticPasswordFile/Schedule` fields + the `settings.go`
  `migrateResticToRsync` one-way migration). **But `Dockerfile:58` still `apt install`s `restic`** (and
  line 42 comments "for backup operations"), so `/usr/bin/restic` ships in the controller image unused.
  *Fix direction:* drop `restic` from the Dockerfile install list (and the dead `Restic*` config fields)
  to shed image weight + remove a confusing "disk-tier is still here" signal. Not a functional bug.
- **Observation:** a named-volume app (navidrome) deployed onto a drive created **no** `appdata/` or
  `backups/` namespace nor a recovery unit on that drive (only the shared `userdata/` skeleton). Recovery
  units appear keyed to appdata-bind apps; named-volume-only apps rely on the DB-dump/volume-tar path
  instead. Flagged for awareness; likely by-design, not verified as a bug.

**Phase 1 verdict:** the stable-vs-raw path model holds across register/deregister/disk-card/eject/
decommission/migrate/backup. The v0.78 (raw register), v0.79 (spurious unregistered badge), and v0.80
(orphaned stable entry on eject) bugs are all ruled out live. The only gaps are the loop-device
init/attach/wipe (blocked by the agent's block-device whitelist — a safety PASS) and the restic dead-weight
finding (F1).

## Phase 2 — /mnt/sys_drive third volume — ⏳ PENDING (2.3 PASS captured)

## Phase 3 — server restart + retired banner — ⏳ PENDING

## Phase 4 — host reboot + chaos — ⏳ PENDING

## Phase 5 — restore to clean baseline — ⏳ PENDING

---

## Findings (running)
_None yet beyond the flagged baseline observations (`onboot:0`, leftover absent intents)._

## Method notes
- User-facing actions: method stated per test (claude-in-chrome vs exact-endpoint). The demo controller
  renders `/settings` unauthenticated and is reachable in-guest at `127.0.0.1:8080` (curl in-container).