docs: README userdata convention + REPORT for v0.66.x (userdata layout)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

REPORT.md

@@ -1,69 +1,88 @@
-# REPORT — TASK B1+B2 data migration + self-serve decommission → controller v0.65.0 (2026-06-14)
+# REPORT — userdata layout + shared-storage ownership convention → controller v0.66.1 + catalog (2026-06-14)
 
-Customer-self-serve storage **migration** (move app data between drives) and **decommission** (retire a
+Customer-facing `userdata/` tree with a shared-ownership convention so FileBrowser + content apps
-drive), implemented trunk-based on `main` against the locked spike design
+collaborate without permission collisions. Spike:
-(`felhom.eu/documentation/audits/SPIKE-decommission-migration-2026-06-14.md`). Pairs with **agent
+`felhom.eu/documentation/audits/SPIKE-userdata-layout-2026-06-14.md`. Trunk-based on `main` in both the
-v0.32.0**. Each change carries non-hollow tests; the load-bearing companions were mutation-proven to
+controller and the app-catalog. No back-compat (existing demo apps moved to the new layout — no real
-fail on the pre-fix code. Built + deployed both images to demo guest 9201 / felhom-pve.
+data). Built + deployed + live-verified on demo guest 9201.
 
-> Validate against pushed source, not this report. **Live decommission/migration of real data is NOT
+> Validate against pushed source at file:line, not this report.
-> yet validated — that is the supervised B3 session (see the end).**
 
-## Confirmed baselines
+## Confirmed baseline
-- Controller `main` was `c37ee6d` = **v0.64.0** → shipped **v0.65.0**.
+- Controller `main` was `cbaa53f` = **v0.65.0** → shipped **v0.66.0** then **v0.66.1** (first-deploy fix).
-- Agent `main` was `8e6d00a` = **v0.31.0** → shipped **v0.32.0**.
+- Catalog `main` was `5ec2b2c` (commits, no semver).
 
-## Commits (controller repo unless noted)
+## Commit hashes
-| # | Commit | Scope |
+| Repo | Commit | Scope |
 |---|---|---|
-| 1 | `b57150e` | B1 migration engine (`internal/stacks/migrate.go`, MigrateAll+MigrateApp) + tests + main wiring + backup mutual-exclusion (Change 3) |
+| app-catalog | `69611ce` | repoint 12 composes off `${HDD_PATH}/media` → `${USERDATA_PATH}/...` + .felhom.yml |
-| 2 | `16a4c3e` | B1 UI — `/api/storage/migrate{,-app,/status}` + settings.html & app_info.html + progress panel |
+| app-catalog | `0d60a5c` | komga + audiobookshelf revert to root (user:1000 fallback) |
-| 3 | `f43697c` (felhom-agent) | v0.32.0 — `POST /disks/decommission` + intent-aware `ReassertGuestBinds` + `GuestBindStore.Remove` + tests |
+| app-catalog | `c99070c` | komga + audiobookshelf `no-new-privileges:true` (root hardening) |
-| 4 | `f2596ea` | B2b decommission orchestration + "Hiányzó tárhely" indicator + Change 4 re-enroll fix + tests |
+| controller | `c48f95f` | v0.66.0 — userdata helper, USERDATA_PATH inject, skeleton, deploy belt, FileBrowser switch, migrate setgid fix + tests |
+| controller | `3b7d089` | v0.66.1 — fix USERDATA_PATH on first deploy (shared `withUserdataPath`) + regression test |
 
 ## What shipped
-- **B1 engine** — crash-safe, resumable, journaled (`<dataDir>/migration.json`). Pipeline validate → stop
+- **Catalog**: `${USERDATA_PATH}` introduced; media/ingest mounts repointed per app (jellyfin/emby/plex
-  → copy (`rsync -a --checksum`, additive, **no `--delete`**; conflict-merge walk for non-app content) →
+  `media:/media:ro`; navidrome `media/music:ro`; audiobookshelf `media/{audiobooks,podcasts}`; komga
-  verify (`rsync -ani --checksum`, zero pending) → flip+redeploy (`RedeployFromEnv`) → **cleanup gated on
+  `media/comics`; calibre `media/books` + `import/calibre`, `UMASK=002`; radarr `media/movies`+`downloads`,
-  all-verified AND all-redeployed**. Two entry points (whole-namespace + per-app incl. SSD→drive) over one
+  sonarr `media/tv`+`downloads`, both `UMASK=002`; romm library→`userdata/roms`; immich +external
-  pipeline. Single-flight; mutually exclusive with the backup orchestrator (migration refuses while a
+  `media/photos:/external/photos:ro`; paperless consume→`import/paperless`). nextcloud unchanged.
-  backup runs; scheduled DB-dump/Tier-2 skip while a migration runs).
+- **Controller** (`internal/appbackup/userdata.go`): `EnsureUserdataDir`/`EnsureDirOwned` (MkdirAll →
-- **B1 UI** — migrate-all (settings page) + per-app "Áthelyezés másik tárhelyre" (app-info page) + shared
+  explicit `Chmod(ModeSetgid|0775)` → chown gid 1000), `UserdataSkeleton`, `EnsureUserdataSkeleton`;
-  status poll/progress panel.
+  Linux `chownGID`/`StatGID` + non-Linux stubs. `withUserdataPath` injects `USERDATA_PATH=<HDD_PATH>/userdata`
-- **B2a (agent)** — self-serve `POST /disks/decommission` mirroring eject (user-data role gate, no
+  in BOTH env builders. Skeleton pre-create on register + FB sync; deploy belt in `composeExecCustomEnv`
-  signature, never formats): intent=decommissioned + bind-prune + unmount. **`ReassertGuestBinds` is now
+  (`up`). FileBrowser mounts `userdata` (uid 1000). `migrate.go` preserves setgid+group (#8).
-  intent-aware** (skips non-enrolled) — the F9-reconnect correctness fix. Operator-signed path untouched.
-- **B2b** — `handleStorageDecommission` (migrate-all-or-none): migrate-then-decommission via the migration
-  done-hook, or decommission-anyway (stop apps, keep HDD_PATH). `agentapi.Decommission` added. Soft-marker
-  retained. **Change 4:** re-enroll clears the marker (`ClearDecommissioned` had zero callers before).
-  **"Hiányzó tárhely"** badge for apps on a missing drive.
 
-## Tests (per-commit green: `go build && go vet && go test ./...`)
+## komga / audiobookshelf — user:1000 outcome: FELL BACK TO ROOT
-- `internal/stacks/migrate_test.go` — siblingName, conflict-merge (differ→`(1)`, identical→dedup, re-run
+Tried `user: "1000:1000"` on both (catalog `0d60a5c`). **Both crash-looped** — their named config/metadata
-  idempotent no `(1)(1)`, 3rd→`(2)`), assert-only verify, skip-dirs, cleanup-only-after-redeploy
+volumes are Docker-created root-owned and the pinned-1000 process can't write them (live: komga SQLite
-  (+companion: redeploy-fail → source intact), verify-catches-corruption, resume (no re-copy),
+`/config` open failure, 14 restarts; audiobookshelf `EACCES mkdir '/metadata/logs'`; neither image has a
-  collision-refused, backup-exclusion, single-flight, SSD→drive resolution, cleanup gate.
+PUID-style root-init chown). **Reverted to root** (the spec's accepted fallback) + added
-- `internal/backup/migration_exclusion_test.go` — scheduled DB-dump **skipped** while migrating.
+`no-new-privileges:true` (catalog `c99070c`) to restore an escalation boundary (full `cap_drop` NOT
-- `internal/web/decommission_test.go` — finalize soft-marks + calls agent (entry retained), re-enroll
+applied — the root-init needs CHOWN/SETUID/SETGID). Both then deploy **healthy as root** (audiobookshelf
-  clears marker (Change 4), missing-storage label, unflagged-migration no-op.
+healthy; komga serving — see healthcheck note). Consequence: root+setgid → their output is **group-
-- **Mutation-proven companions** (fail on pre-fix, pass after revert): migration collision guard, cleanup
+readable** (FileBrowser browses/reads), not group-writable; the app manages its own files. PUID-1000 apps
-  gate, Change-4 re-enroll clearing, and (agent) the intent-aware reassert (`disks.go` guard removed →
+(radarr/sonarr/calibre + UMASK 002) give **full group-write**.
-  decommissioned drive re-bound → test FAILS).
 
-## Build / deploy / verify
+## immich / paperless — DEPLOY DEFERRED (guest RAM), design verified
-- Built+pushed `gitea.dooplex.hu/admin/felhom-controller:0.65.0` (192.168.0.180); deployed to guest 9201
+Guest 9201 has **2 GiB RAM total**. immich needs ~4 GiB → cannot deploy. paperless (~1.1 GiB:
-  via golden/bootstrap → `docker ps`: `:0.65.0 Up (healthy)`, clean startup (`controller_started (0.65.0)`,
+app+postgres+redis) does not fit alongside the running apps; the controller's memory gate refuses it
-  no errors), `GET /api/storage/migrate/status` → `{"job":null}` (idle, wired).
+(observed: komga refused at 92 MB free until non-media apps were temporarily stopped). Their compose +
-- Built agent v0.32.0 on 180; installed on felhom-pve (`/usr/local/bin/felhom-agent`, backup
+run-identity design is shipped + statically verified (paperless `USERMAP 1000` consumes from
-  `.bak-0.31.0`), `systemctl restart felhom-agent` → active, `--version` 0.32.0.
+`import/paperless`; immich `media/photos:/external/photos:ro`). **immich external library: MOUNT READY,
-- **Live-proven (incidental, safe):** on agent restart the new intent-aware re-assert logged
+REGISTRATION PENDING** — making /external/photos visible is only the mount; surfacing it in Immich
-  `F9 re-assert: skipping non-enrolled drive (intent-gated) vmid=9201 durable_id=uuid:ce9d… intent=ejected`
+requires registering an External Library at `Administration → External Libraries` (admin UI/API), a
-  — the exact drive that lingered in `guest-binds.json` (B0 spike finding) is now correctly NOT re-bound.
+post-deploy step compose cannot do. Flagged in the immich compose + `.felhom.yml` first_steps. Not
+overclaimed.
 
-## NOT yet live-validated — awaiting supervised B3
+## Live-verify evidence (guest 9201) — all three fixes PASS
-The real end-to-end UI flow with actual data was deliberately NOT run (per the task STOP):
+- **(c) convention**: `ls -la`/`stat` of `/mnt/felhom-usb/userdata` and every subdir → **mode 2775
-- put data on a drive → decommission (migrate-to-another OR decommission-anyway) → verify data integrity
+  (setgid) group 1000** throughout (media/{movies,comics,...}, roms, import/{paperless,calibre}, downloads,
-  end-to-end → drive un-enrolled and does NOT auto-mount/re-bind on reconnect → orphaned apps show
+  documents). Same on the system drive.
-  "Hiányzó tárhely" → re-enroll clears it. Operator-present, real UI (no agent/CLI shortcuts, per the A4
+- **(a) FileBrowser writes userdata**: `docker exec filebrowser` (uid 1000) `mkdir` + write a file under
-  rule). The engine/endpoint logic is unit-proven and the intent-aware reassert is already proven live by
+  `/srv/felhom-usb/documents/...` → `MKDIR_OK`/`WRITE_OK`, new dir inherited gid 1000 via setgid. The
-  the restart log above; the data-bearing round-trip is what B3 covers.
+  original permission-denied (root-owned `appdata` mount) is gone.
+- **(b) shared media across apps**: radarr (uid 1000, UMASK 002) wrote `media/movies/Inception/Inception.mkv`
+  → dir **2775**, file **664 group 1000**; jellyfin (root, `media:/media:ro`) `cat` it → `MOVIEDATA`;
+  FileBrowser (uid 1000) listed **and deleted** it → full RW collaboration. komga (root fallback) wrote
+  `media/comics/Batman` group 1000 → FileBrowser **lists/reads** but cannot delete (the documented
+  root-fallback group-read tradeoff).
+- App health after moving to the new layout: jellyfin, calibre-web, romm, radarr, audiobookshelf all
+  **Up healthy**; komga serving (HTTP up, tasks running, writes/reads OK) but its Docker healthcheck
+  (`/api/v1/actuator/health`) reports **unhealthy** — a pre-existing healthcheck-probe nuance in the komga
+  template, NOT a userdata/permission issue (no OOM, 0 restarts, logs clean). Flagged for a separate
+  healthcheck fix.
+
+## Tests (per-commit green; authoritative Linux gate on 192.168.0.180)
+`internal/appbackup`: SharedContentGID==1000, skeleton list+structure, `EnsureDirOwned` setgid+gid
+(Linux; in-test companion proves a plain MkdirAll has NO setgid). `internal/stacks`:
+`ParseComposeUserdataMounts` selectivity, deploy belt creates declared dirs, `withUserdataPath`
+on/off-by-HDD_PATH, **migration preserves setgid+group** (Linux — mutation-proven: reverting to
+0755/.Perm() drops setgid → test FAILS). Full `go build && go vet && go test ./...` green.
+
+## Follow-ups (small, noted not done)
+- FileBrowser-created dirs are 2755 (its umask 022) — group-read not group-write for OTHER actors;
+  consider a FileBrowser `UMASK`/`user:1000:1000`-with-002 so customer-made folders are group-writable.
+- komga healthcheck endpoint (`/api/v1/actuator/health`) returns non-200 on komga 1.20 → "unhealthy"
+  despite serving; fix the healthcheck definition.
+- immich/paperless live deploy needs a larger guest (the 2 GiB demo can't host them).

controller/README.md

@@ -698,6 +698,31 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an
 >   restores `Schedulable`. A deployed app whose drive is decommissioned/disconnected/absent shows the
 >   **"Hiányzó tárhely"** warning badge on the dashboard/stacks/app card.
 >
+> - **(v0.66.0) Userdata layout + shared-storage ownership convention** (`internal/appbackup/userdata.go`).
+>   Each drive's felhom-data namespace gains a customer-facing **`userdata/`** tree (sibling of
+>   `appdata/`/`backups/`) — the ONLY thing FileBrowser mounts. Skeleton:
+>   `media/{movies,tv,music,audiobooks,books,comics,photos}`, `downloads`, `import/{paperless,calibre}`,
+>   `roms`, `documents`. **Convention:** every userdata dir is group-owned by `SharedContentGID` (1000),
+>   mode **2775** (setgid + group-rwx) — `EnsureUserdataDir` = MkdirAll → explicit `Chmod(ModeSetgid|0775)`
+>   (MkdirAll's mode is umask-masked AND drops setgid) → chown group 1000. Setgid makes new files inherit
+>   group 1000 so FileBrowser (uid 1000) + the content apps collaborate without permission collisions.
+>   - **`USERDATA_PATH`** = `<HDD_PATH>/userdata` is injected into the compose env (the shared
+>     `withUserdataPath`, used by BOTH `stackEnv` and `composeExecWithEnv` — the initial deploy builds env
+>     from values, not app.yaml). Catalog media mounts use `${USERDATA_PATH}/...`.
+>   - **Pre-create**: the full skeleton is created with the convention on `registerStoragePath` +
+>     `syncFileBrowserMounts` (system + additional drives); a **deploy belt** in `composeExecCustomEnv`
+>     (on `up`) pre-creates every `${USERDATA_PATH}/...` bind source the app declares, so Docker never
+>     auto-creates a userdata dir as guest-root (covers apps not in the skeleton).
+>   - **FileBrowser** mounts `<drive>/userdata` (was `appdata`) and runs as uid 1000 → it can create
+>     folders + upload into the 2775 setgid tree (fixes the old permission-denied); app internals
+>     (`appdata/`) are no longer browsable.
+>   - **Run-identity**: PUID/PGID-1000 apps (radarr/sonarr/calibre with `UMASK=002`) write **group-
+>     writable**, so FileBrowser can fully manage their output. Root-only apps (jellyfin :ro; komga +
+>     audiobookshelf RW after the `user:1000` fallback) write **group-readable** via setgid (FileBrowser
+>     browses/reads, the app manages its own files).
+>   - **Migration-safe**: `migrate.go`'s merge walk preserves the source dir's setgid + group (and
+>     `copyFile` the full file mode + group), so the convention survives a whole-drive `MigrateAll`.
+>
 > The privileged controller-side disk subsections **below are historical** (the `internal/storage/*` scan/
 > format code was removed in 8C — execution is the agent's now).