docs(spike): userdata/ layout + shared-storage ownership mapping (report-only gate)

Maps current namespace dir creation/ownership (live on 9201), the 9-app/13-mount
media blast radius, run-identity table (FileBrowser=uid1000, komga/audiobookshelf=root),
LXC idmap, and proposes the setgid GID-1000 convention. Flags: MigrateAll's merge walk
drops setgid+owner on userdata. Ends with a 7-item fork list. No code/deploy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

documentation/audits/SPIKE-userdata-layout-2026-06-14.md

@@ -0,0 +1,276 @@
+# SPIKE — userdata/ layout + shared-storage ownership convention (2026-06-14)
+
+**Report-only. NO code, NO catalog edits, NO deploy, NO version bump.** This is the GATE before any
+implementation; it MAPS the current state and PROPOSES the change with file:line + live evidence, and
+ends with a FORK LIST for the operator.
+
+**Goal (operator-decided):** introduce a customer-facing `userdata/` dir under each drive's felhom-data
+namespace, sibling to `appdata/` and `backups/`. FileBrowser mounts `userdata/`. Apps that write to
+`${HDD_PATH}/media/...` move into the userdata layout. A shared-ownership convention must let FileBrowser
++ apps collaborate on userdata without permission collisions.
+
+**Access path used (operator can't login to the LXC):** `ssh root@felhom-pve` (the Proxmox host,
+192.168.0.162) → `pct exec 9201 -- <cmd>`; in-guest apps are Docker containers reached via `pct exec 9201
+-- docker exec <name> …`. All "live:" evidence below was captured this way on guest **9201**.
+
+---
+
+## 1. Current layout (LIVE) + who creates what
+
+### Live evidence
+
+In-guest (Model A — `/mnt/felhom-usb` IS the felhom-data namespace root):
+```
+$ pct exec 9201 -- ls -la /mnt/felhom-usb/
+drwxr-xr-x 3 root root … appdata
+drwxr-xr-x 3 root root … backups
+drwxr-xr-x 3 root root … media          ← media/books (calibre-web), root:root mode 755
+$ pct exec 9201 -- ls -la /mnt/felhom-usb/media/      → drwxr-xr-x root root books
+$ pct exec 9201 -- ls -la /mnt/felhom-usb/appdata/    → drwxr-xr-x root root romm
+```
+Host view of the same drive (host `/mnt/felhom-usb/felhom-data/` = guest `/mnt/felhom-usb/`):
+```
+$ ls -la /mnt/felhom-usb/felhom-data/
+drwxr-xr-x 100000 100000 … appdata / backups / media     ← all guest-root (uid 0 → host 100000)
+```
+System drive (in guest): namespace root is `/mnt/sys_drive/felhom-data/` (NOT `/mnt/sys_drive`):
+```
+$ pct exec 9201 -- ls -la /mnt/sys_drive/felhom-data/  → appdata / backups / exports   (all root:root)
+```
+**No `userdata/` exists anywhere yet** (`find /mnt -maxdepth 3 -name userdata` → none).
+
+### Who creates each dir (code)
+
+| Dir | Created by | Where | Owner / mode |
+|---|---|---|---|
+| `<drive>/felhom-data` (additional drive) | **AGENT** at guest-attach | `felhom-agent/internal/localapi/guestbind.go:55-61`: `mkdir -p <drive>/felhom-data` then `chown 100000:100000` (NOT `-R`) | guest-root 0:0, 0755 |
+| system-drive `felhom-data/…` | provisioning + controller | `scripts/docker-setup.sh` does **NOT** create namespace dirs; the controller creates `appdata` on FileBrowser sync | guest-root 0:0, 0755 |
+| `appdata/` | **CONTROLLER** | `internal/web/handlers.go:~1464` `syncFileBrowserMounts`: `os.MkdirAll(<drive>/appdata, 0755)` | guest-root 0:0, 0755 |
+| `appdata/<app>`, `media/…`, `backups/<app>` | **Docker auto-create** (bind-mount source) on first app deploy, OR the backup code | docker daemon (runs as guest-root) | **guest-root 0:0, 0755** |
+
+**Why `media/` is guest-root (confirmed):** the agent creates only the `felhom-data` ROOT (comment
+`guestbind.go:20-21`: "per-app subdirs are chowned at deploy — NOT here"); nothing pre-creates `media/`,
+so when calibre-web first deployed, the **Docker daemon auto-created** `${HDD_PATH}/media/books/...` as
+its bind-mount source — owned by the daemon (guest-root). This is the collision root cause: a
+guest-uid-1000 app (FileBrowser, calibre) gets only `r-x` on a 0755 root-owned dir, never write.
+
+---
+
+## 2. HDD_PATH + namespace resolution
+
+- `appbackup/paths.go:11` `const FelhomDataDir = "felhom-data"`; `:24` `NamespaceRoot(drivePath,
+  inGuestDrive)` → returns `drivePath` as-is when `inGuestDrive` (Model-A user drive: the in-guest mount
+  IS the namespace root), else `filepath.Join(drivePath, "felhom-data")` (SSD/system). `:68`
+  `AppDataDir(nsRoot, app)` = `<nsRoot>/appdata/<app>`.
+- **`${HDD_PATH}` at deploy time = the registered StoragePath, which IS the in-guest namespace root.**
+  Live: `romm/app.yaml` → `HDD_PATH: /mnt/felhom-usb`; SSD apps (mealie, actualbudget) have **no
+  HDD_PATH** (named volumes only). So `${HDD_PATH}/media` expands to:
+  - user drive: `/mnt/felhom-usb/media` (guest) = host `/mnt/felhom-usb/felhom-data/media`
+  - system drive (a media app pinned there): `/mnt/sys_drive/felhom-data/media`
+- Host→guest bind (`guestbind.go:64-66`): `pct set <vmid> -mpN <drive>/felhom-data,mp=/mnt/<name>` — the
+  `felhom-data` segment is consumed by the bind, so the guest sees the namespace at `/mnt/<name>`.
+- **Therefore `${HDD_PATH}/userdata` works uniformly** on both drive types (it's always namespace-relative).
+
+---
+
+## 3. App-catalog blast radius (the apps that must change)
+
+Grepped `app-catalog-felhom.eu/templates/*/docker-compose.yml`. Apps writing customer content to
+`${HDD_PATH}/media/...` (the ones the userdata move touches):
+
+| App | file:line | Mount | RW/RO |
+|---|---|---|---|
+| jellyfin | jellyfin/docker-compose.yml:20 | `${HDD_PATH}/media:/media` | **:ro** |
+| emby | emby/docker-compose.yml:21 | `${HDD_PATH}/media:/media` | **:ro** |
+| plex | plex/docker-compose.yml:22 | `${HDD_PATH}/media:/media` | **:ro** |
+| navidrome | navidrome/docker-compose.yml:22 | `${HDD_PATH}/media/music:/music` | **:ro** |
+| sonarr | sonarr/docker-compose.yml:21-22 | `${HDD_PATH}/media:/media` + `${HDD_PATH}/downloads:/downloads` | RW |
+| radarr | radarr/docker-compose.yml:21-22 | `${HDD_PATH}/media:/media` + `${HDD_PATH}/downloads:/downloads` | RW |
+| audiobookshelf | audiobookshelf/docker-compose.yml:20-21 | `${HDD_PATH}/media/audiobooks` + `${HDD_PATH}/media/podcasts` | RW |
+| calibre-web | calibre-web/docker-compose.yml:29-30 | `${HDD_PATH}/media/books/ingest` + `${HDD_PATH}/media/books/library` | RW |
+| komga | komga/docker-compose.yml:19 | `${HDD_PATH}/media/comics:/data` | RW |
+
+**Total: 9 apps / ~13 media mounts** must change (`${HDD_PATH}/media` → the userdata layout).
+
+**NOT in scope (app-exclusive `appdata/` data, not shared media):** immich
+(`${HDD_PATH}/appdata/immich`), nextcloud (`${HDD_PATH}/appdata/nextcloud`), paperless-ngx
+(`${HDD_PATH}/appdata/paperless/{media,consume,export}`), romm (`${HDD_PATH}/appdata/romm/{library,
+resources}`). These are single-app subtrees under `appdata/` — moving them to userdata is a *separate*
+decision (see ROMM + fork list). `sonarr/radarr` also use `${HDD_PATH}/downloads` (staging, not final
+library) — decide whether downloads moves under userdata too.
+
+**ROMM (flagged):** `romm/docker-compose.yml:65-66` →
+`${HDD_PATH}/appdata/romm/library:/romm/library` + `…/resources:/romm/resources`. **Compose-path only —
+no in-app/env override** (`.felhom.yml` only exposes `HDD_PATH`). The library lives under `appdata/`
+today, so it is **not customer-browsable**. The operator wants a ROM import/library location *under
+userdata* — that means repointing the `library` mount to `${HDD_PATH}/userdata/<…>` (fork: exact path).
+`resources` (cover art/metadata) is app-internal — leave under appdata.
+
+---
+
+## 4. App run-identity (decides the shared-group design)
+
+| App | image | run identity | writes shared media? |
+|---|---|---|---|
+| **filebrowser** | gtstef/filebrowser:1.3.3-stable | **uid 1000 (filebrowser)** by image default — LIVE `id`=`uid=1000 gid=1000`; compose sets no `user:` | RW (the customer's file manager) |
+| calibre-web | crocodilestick/calibre-web-automated | PUID/PGID **1000** (linuxserver-style) | RW (books) |
+| sonarr | lscr.io/linuxserver/sonarr | PUID/PGID **1000** | RW (media+downloads) |
+| radarr | lscr.io/linuxserver/radarr | PUID/PGID **1000** | RW (media+downloads) |
+| emby | emby/embyserver | UID/GID **1000** | :ro |
+| paperless-ngx | paperless-ngx | USERMAP_UID/GID **1000** | (appdata only) |
+| jellyfin | jellyfin/jellyfin | **root** (no PUID) | :ro |
+| plex | plexinc/pms-docker | **root** (no PUID) | :ro |
+| navidrome | deluan/navidrome | **root** (no PUID) | :ro |
+| komga | gotson/komga | **root** (no PUID) — LIVE pattern | **RW (comics)** |
+| audiobookshelf | advplyr/audiobookshelf | **root** (no PUID) | **RW (audiobooks/podcasts)** |
+| romm | rommapp/romm | **root** — LIVE `docker exec romm id`=`uid=0(root)`, `Config.User`="" | (appdata; would be RW if moved) |
+
+**The 1000 cluster is large and natural:** FileBrowser + the 5 PUID/PGID-1000 apps already share GID
+1000. The :ro media apps (jellyfin/plex/emby/navidrome) only READ — they need group/other **read**, no
+write concern. **The problem set = root-running apps that WRITE shared media: komga + audiobookshelf**
+(and romm if its library moves to userdata).
+
+---
+
+## 5. FileBrowser — run UID + mount-change plan
+
+- **LIVE:** `docker exec filebrowser id` → `uid=1000(filebrowser) gid=1000(filebrowser)`;
+  `Config.User`=`filebrowser`. The gtstef image bakes uid 1000 even though the controller compose sets no
+  `user:` — so FileBrowser is **already gid 1000**.
+- Current mounts (LIVE `docker inspect`): `/mnt/sys_drive/felhom-data/appdata → /srv/felhom-data` and
+  `/mnt/felhom-usb/appdata → /srv/felhom-usb`. Renderers: `internal/web/handlers.go:~1464`
+  `syncFileBrowserMounts` (`os.MkdirAll(<drive>/appdata,0755)` + bind `…/appdata:/srv/<name>`);
+  `internal/infra/infra.go:~116` `RenderFileBrowserCompose` (image const `FileBrowserImage =
+  "gtstef/filebrowser:1.3.3-stable"`, **no `user:`**, no PUID), `:~211` `RenderFileBrowserConfig`.
+- **To mount `userdata/` instead of `appdata/`:** change `syncFileBrowserMounts` to
+  `<drive>/userdata:/srv/<name>` (and `MkdirAll userdata`). (Scoping note: it was deliberately scoped to
+  `appdata/` in Phase 4A to hide backups/recovery-units; pointing at `userdata/` keeps backups hidden and
+  exposes only customer media — strictly better.)
+- **To run as the shared GID:** FileBrowser is already uid/gid 1000. If the shared GID is 1000, **no
+  change needed**. If a dedicated felhom GID is chosen, add `user: "1000:<GID>"` (or `group_add`) to
+  `RenderFileBrowserCompose`. With `userdata` dirs at `:1000`/`:GID` mode 2775, FileBrowser then has
+  full RW.
+
+---
+
+## 6. LXC userns mapping
+
+- **LIVE** `/etc/pve/lxc/9201.conf`: `unprivileged: 1`, `features: nesting=1,keyctl=1`, **no custom
+  `lxc.idmap`** → the default unprivileged map applies: **guest uid/gid 0 → host 100000**, **guest 1000 →
+  host 101000** (range 65536). Confirmed by the host `ls` showing guest-root dirs as `100000:100000`.
+- The convention only needs to be consistent **inside the guest** (guest uids/gids); the host-side numbers
+  are the +100000 image. The agent, running as host-root, can `chown` to any host uid:gid, i.e. it can
+  stamp guest `0:1000` as host `100000:101000`.
+
+---
+
+## 7. Ownership-convention recommendation (+ umask caveats)
+
+**Recommended convention:** `userdata/` and its subtree owned `root:<sharedGID>` (guest), **mode 2775**
+(setgid + group-rwx). Setgid makes every file/dir created inside inherit `<sharedGID>` regardless of who
+creates it. Members of `<sharedGID>`: FileBrowser + every customer-content app. **Recommend
+`<sharedGID> = 1000`** — it is already FileBrowser's gid and the PUID/PGID of calibre/sonarr/radarr/emby/
+paperless, so 5 apps + FileBrowser need zero identity change.
+
+**The setgid bit fixes the GROUP; it does NOT fix the WRITE BIT — umask still applies.** Caveats per app:
+- **linuxserver apps (calibre-web, sonarr, radarr):** run as 1000 already; default `UMASK=022` → files
+  0644 (group can't write). Set **`UMASK=002`** (linuxserver supports it) so group keeps write.
+- **root-running RW-to-shared apps (komga, audiobookshelf):** write as root with umask 022 → group-read
+  only. Setgid stamps gid 1000 but FileBrowser still can't modify/delete. Each needs **either** a
+  `user: "1000:1000"` pin **or** a `UMASK`/umask-equivalent (komga: `user:` works; audiobookshelf: no
+  PUID — `user: "1000:1000"` is the lever). Flag: verify each image starts cleanly when pinned to 1000.
+- **:ro media apps (jellyfin/plex/emby/navidrome):** read-only mounts — group/other **read** is enough;
+  no umask concern.
+- **romm (if library moves to userdata):** runs as root; same caveat as komga/audiobookshelf.
+
+**Pre-create userdata so Docker never root-auto-creates it again** (the #1 fix). Creation points to
+patch:
+- **Additional drives:** `felhom-agent/internal/localapi/guestbind.go:55-66` — after creating
+  `felhom-data`, also `mkdir -p <drive>/felhom-data/userdata/{media subdirs}` + `chown 100000:101000` +
+  `chmod 2775` (agent is host-root → can set guest `0:1000`).
+- **System drive + per-deploy safety net:** the controller — at app deploy (`internal/stacks/deploy.go`)
+  and/or the FileBrowser-sync `MkdirAll` site (`handlers.go:~1464`) — ensure `userdata/<subdirs>` exist
+  with the shared GID + setgid **before** `docker compose up` so the daemon never auto-creates them root.
+  (The controller runs as guest-root, so it can chown to `0:1000` + chmod 2775.)
+
+---
+
+## 8. Migration interaction (controller v0.65.0) — ⚠ FLAG
+
+`MigrateAll` copies "everything under felhom-data", so `userdata/` is covered. **But ownership/setgid
+survival is split:**
+- **App subtrees** (`appdata/<app>` + recovery unit) go via `rsyncCopy` = `rsync -a --checksum` —
+  `-a` includes `-o -g -p`, and the controller runs **as root in the guest** (Dockerfile has no `USER`),
+  so owner/group **and the setgid bit are preserved**. ✓
+- **Non-app content** (which is exactly `userdata/`, `media/`, customer dirs) goes via the **custom
+  conflict-merge walk** (`migrate.go` `walkMerge`/`copyFile`), NOT rsync. That code does
+  `os.MkdirAll(dst, 0o755)` for dirs (**drops the setgid bit + sets root:root**) and `copyFile` preserves
+  only `fi.Mode().Perm()` (**`.Perm()` masks off setgid/setuid**) and does **not** chown. **So a
+  MigrateAll would land userdata/ at the target as `root:root` 0755 — the shared-ownership convention
+  would NOT survive a whole-drive migration.** This must be fixed (re-apply the convention post-copy, or
+  teach `walkMerge`/`copyFile` to preserve setgid+group) as part of implementation. Per-app `MigrateApp`
+  doesn't touch userdata (it only moves the app's appdata+recovery-unit), so it's unaffected.
+
+---
+
+## 9. Transition (existing demo has `media/` in the old layout)
+
+- Live: `/mnt/felhom-usb/media/books` (calibre-web) already exists, guest-root 0755. New deploys would
+  use `userdata/` but existing media stays at `media/`.
+- Options: **(a) new-layout-for-new-deploys-only**, leave existing `media/` (simplest; existing apps keep
+  working until re-deployed). **(b) one-time move** `media/ → userdata/media/` + re-chown to the
+  convention + redeploy affected apps. The just-shipped per-app migrate does NOT do this (it moves
+  *appdata* between drives, not `media→userdata` within a drive); it would need a small dedicated
+  migration step (mv + chown -R + chmod -R, then redeploy). **Recommend (a) for the cutover** + a small
+  opt-in one-time fixup script for the demo, executed under supervision — not auto-run.
+
+---
+
+## Summary table
+
+| # | Finding | Evidence |
+|---|---|---|
+| 1 | `media/`/`appdata/` are **guest-root 0:0 mode 755**; Docker auto-creates bind sources as root (calibre's `media/books` proves it) — the collision cause | live `ls -la` (guest + host 100000) |
+| 2 | `${HDD_PATH}` = the in-guest namespace root (`/mnt/felhom-usb`); `${HDD_PATH}/userdata` works uniformly | `romm/app.yaml HDD_PATH=/mnt/felhom-usb`; `paths.go:11,24,68` |
+| 3 | **9 apps / ~13 mounts** use `${HDD_PATH}/media`; ROMM library is compose-path-only under appdata | catalog grep (table above) |
+| 4 | FileBrowser + 5 apps already at **uid/gid 1000**; only komga + audiobookshelf (root, RW) are the problem set | live `id` + catalog |
+| 5 | FileBrowser **uid 1000** by image default, mounts `appdata/` today; switch to `userdata/` is a one-line mount change | live `docker exec filebrowser id`; `handlers.go:~1464`, `infra.go:~116` |
+| 6 | unprivileged LXC, default idmap (0→100000, 1000→101000); convention is guest-internal | `/etc/pve/lxc/9201.conf` |
+| 7 | Convention: `userdata` `:1000` mode **2775** (setgid); umask caveat → komga/audiobookshelf need `user:1000` or UMASK; linuxserver apps need `UMASK=002` | synthesis |
+| 8 | ⚠ MigrateAll's **non-app merge walk drops setgid+owner** on userdata — convention won't survive a migration as-is | `migrate.go walkMerge`/`copyFile` |
+| 9 | Existing `media/` stays; recommend new-layout-for-new + supervised one-time fixup for the demo | live |
+
+---
+
+## FORK LIST (operator's call — GATE for implementation)
+
+1. **Shared GID value** — **`1000`** (recommended: already FileBrowser's gid + the PUID/PGID of
+   calibre/sonarr/radarr/emby/paperless; zero identity change for 6 of the players) **vs** a dedicated
+   `felhom` group (cleaner semantics, but every app needs `group_add`/PUID set to it).
+2. **Common-group-only vs one-UID-for-all-apps** (operator's open question):
+   - *Common-group (setgid 2775 + per-app UMASK/user tweaks):* preserves each app's native identity
+     (linuxserver PUID dance, root-init images work), but needs the umask fix on root-RW apps
+     (komga/audiobookshelf) and `UMASK=002` on linuxserver apps. More per-app tuning, fewer image
+     surprises.
+   - *One-UID-for-all (`user: "1000:1000"` on every content app):* simplest permissions (everything is
+     1000, no setgid/umask reasoning), but **breaks images that must initialize as root** — linuxserver
+     images run their PUID/PGID setup as root then drop; pinning `user:` skips that; some official images
+     (jellyfin/plex) and root-only apps may misbehave. Higher image-compat risk.
+   - *Recommendation:* **common-group (setgid) with GID 1000**, pin `user:`/UMASK only on the few
+     root-RW-to-shared apps.
+3. **userdata subfolder set** — confirm `media/{movies, "tv shows", music, audiobooks, books}`. The space
+   in `"tv shows"` is cosmetic and workable (quote in compose/scripts); a hyphen/underscore avoids
+   quoting. Decide: keep the space, or `tv-shows`. Also: do `downloads/` (sonarr/radarr staging) and
+   `comics/` (komga) live under `userdata/media/` too?
+4. **ROMM import location under userdata** — repoint `library` to e.g. `${HDD_PATH}/userdata/roms`
+   (browsable) while keeping `resources` under `appdata/romm`? Confirm the exact path + that RomM tolerates
+   an empty/new library dir.
+5. **Media path scheme** — edit each compose to `${HDD_PATH}/userdata/media/...` **vs** introduce a single
+   **`USERDATA_PATH`** env (one var, set at deploy like HDD_PATH; cleaner, one place to change, and makes a
+   future layout change a one-liner). *Recommendation: `USERDATA_PATH`.*
+6. **Transition** — new-layout-for-new-deploys-only (recommended) **vs** migrate existing `media/ →
+   userdata/media/` now (needs a supervised one-time fixup; the v0.65.0 per-app migrate does NOT cover
+   this in-drive move).
+7. **(Surfaced, needs a call)** **Fix MigrateAll's merge walk to preserve setgid+group** (or re-stamp the
+   convention post-migration) — otherwise the shared-ownership convention silently degrades to root:root
+   on any whole-drive migration (#8). Decide whether this rides in the same slice.