diff --git a/REPORT.md b/REPORT.md index 559ff55..6bc0032 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,88 +1,76 @@ -# REPORT — userdata layout + shared-storage ownership convention → controller v0.66.1 + catalog (2026-06-14) +# REPORT — userdata arc close-out: FileBrowser umask + Immich/Paperless deploy → controller v0.66.2 + catalog (2026-06-15) -Customer-facing `userdata/` tree with a shared-ownership convention so FileBrowser + content apps -collaborate without permission collisions. Spike: -`felhom.eu/documentation/audits/SPIKE-userdata-layout-2026-06-14.md`. Trunk-based on `main` in both the -controller and the app-catalog. No back-compat (existing demo apps moved to the new layout — no real -data). Built + deployed + live-verified on demo guest 9201. +Finished the userdata arc: (1) FileBrowser-created folders are now group-writable; (2) Immich deployed +with its external photo library registered + verified; (3) Paperless deployed and consume verified +end-to-end. Trunk-based on `main` (controller + app-catalog). Built + deployed + live-verified on demo +guest 9201 (now 12 GiB RAM). > Validate against pushed source at file:line, not this report. ## Confirmed baseline -- Controller `main` was `cbaa53f` = **v0.65.0** → shipped **v0.66.0** then **v0.66.1** (first-deploy fix). -- Catalog `main` was `5ec2b2c` (commits, no semver). +- Controller `main` was `d0718e8` = **v0.66.1** → shipped **v0.66.2**. +- Guest 9201: `pct config` memory **12288** (12 GiB) — confirmed before the deploy phases (was 2 GiB). ## Commit hashes | Repo | Commit | Scope | |---|---|---| -| app-catalog | `69611ce` | repoint 12 composes off `${HDD_PATH}/media` → `${USERDATA_PATH}/...` + .felhom.yml | -| app-catalog | `0d60a5c` | komga + audiobookshelf revert to root (user:1000 fallback) | -| app-catalog | `c99070c` | komga + audiobookshelf `no-new-privileges:true` (root hardening) | -| controller | `c48f95f` | v0.66.0 — userdata helper, USERDATA_PATH inject, skeleton, deploy belt, FileBrowser switch, migrate setgid fix + tests | -| controller | `3b7d089` | v0.66.1 — fix USERDATA_PATH on first deploy (shared `withUserdataPath`) + regression test | +| controller | `98d5504` | v0.66.2 — FileBrowser umask 002 entrypoint wrapper + test | +| controller | (this) | docs: README FB umask + REPORT | +| app-catalog | `67037cc` | paperless OCR lang-pack fix (space-separated `PAPERLESS_OCR_LANGUAGES`) | -## What shipped -- **Catalog**: `${USERDATA_PATH}` introduced; media/ingest mounts repointed per app (jellyfin/emby/plex - `media:/media:ro`; navidrome `media/music:ro`; audiobookshelf `media/{audiobooks,podcasts}`; komga - `media/comics`; calibre `media/books` + `import/calibre`, `UMASK=002`; radarr `media/movies`+`downloads`, - sonarr `media/tv`+`downloads`, both `UMASK=002`; romm library→`userdata/roms`; immich +external - `media/photos:/external/photos:ro`; paperless consume→`import/paperless`). nextcloud unchanged. -- **Controller** (`internal/appbackup/userdata.go`): `EnsureUserdataDir`/`EnsureDirOwned` (MkdirAll → - explicit `Chmod(ModeSetgid|0775)` → chown gid 1000), `UserdataSkeleton`, `EnsureUserdataSkeleton`; - Linux `chownGID`/`StatGID` + non-Linux stubs. `withUserdataPath` injects `USERDATA_PATH=/userdata` - in BOTH env builders. Skeleton pre-create on register + FB sync; deploy belt in `composeExecCustomEnv` - (`up`). FileBrowser mounts `userdata` (uid 1000). `migrate.go` preserves setgid+group (#8). +## Phase 1 — FileBrowser umask (mechanism: ENTRYPOINT WRAPPER, not UMASK env) +Verified the gtstef/filebrowser:1.3.3-stable image is a single Go binary (`entrypoint ./filebrowser`, +runs uid 1000) that does **NOT** honor a `UMASK` env — live test: `docker run -e UMASK=002 … umask` → +`0022`. So `RenderFileBrowserCompose` (`internal/infra/infra.go`) wraps the entrypoint: +`["sh","-c","umask 002; exec /home/filebrowser/filebrowser"]`. Test asserts the rendered compose carries +it. **Live proof on 9201** (after deploying v0.66.2 → FileBrowser re-rendered): PID1 `Umask: 0002`; a +folder created at that umask under the setgid parent → **2775 group 1000**; a uid/gid-1000 app process +(busybox `--user 1000:1000`) wrote a file INTO that customer folder (664 group 1000) → `APP_WRITE_OK`; +FileBrowser then deleted it. The pre-fix `Teszt` folder (2755) was the only stale dir — normalized with a +one-time `find -type d -exec chmod g+rwxs` (no data); no 2755 dirs remain. -## komga / audiobookshelf — user:1000 outcome: FELL BACK TO ROOT -Tried `user: "1000:1000"` on both (catalog `0d60a5c`). **Both crash-looped** — their named config/metadata -volumes are Docker-created root-owned and the pinned-1000 process can't write them (live: komga SQLite -`/config` open failure, 14 restarts; audiobookshelf `EACCES mkdir '/metadata/logs'`; neither image has a -PUID-style root-init chown). **Reverted to root** (the spec's accepted fallback) + added -`no-new-privileges:true` (catalog `c99070c`) to restore an escalation boundary (full `cap_drop` NOT -applied — the root-init needs CHOWN/SETUID/SETGID). Both then deploy **healthy as root** (audiobookshelf -healthy; komga serving — see healthcheck note). Consequence: root+setgid → their output is **group- -readable** (FileBrowser browses/reads), not group-writable; the app manages its own files. PUID-1000 apps -(radarr/sonarr/calibre + UMASK 002) give **full group-write**. +## Phase 2 — Immich: DEPLOYED + EXTERNAL LIBRARY REGISTERED + verified +- Deployed via the controller deploy flow (the dashboard's `/api/stacks/immich/deploy` over the public + URL — mem-gate + USERDATA_PATH + belt all run; NOT a raw compose/CLI shortcut). **Mem-gate PASSED** + (it refused at 2 GiB; cleared at 12 GiB). All 4 containers **healthy**; memory: server 916 MiB, ML + 231 MiB, postgres 340 MiB, redis 5 MiB (~1.5 GiB total; 8.7 GiB available after). +- External mount confirmed: `/mnt/felhom-usb/userdata/media/photos → /external/photos (ro)`, 2775 setgid. +- **External library REGISTERED via the Immich API** (no browser tool in this env — see Note): admin + account created (creds stored out-of-band), `POST /api/libraries` with `importPaths:["/external/photos"]`. +- **End-to-end VERIFIED**: a JPEG placed in `media/photos` by a uid-1000 process → `POST /api/libraries/{id}/scan` + → `GET …/statistics` returned `{"photos":1,"total":1,"usage":631}`. The photo appears in Immich. -## immich / paperless — DEPLOY DEFERRED (guest RAM), design verified -Guest 9201 has **2 GiB RAM total**. immich needs ~4 GiB → cannot deploy. paperless (~1.1 GiB: -app+postgres+redis) does not fit alongside the running apps; the controller's memory gate refuses it -(observed: komga refused at 92 MB free until non-media apps were temporarily stopped). Their compose + -run-identity design is shipped + statically verified (paperless `USERMAP 1000` consumes from -`import/paperless`; immich `media/photos:/external/photos:ro`). **immich external library: MOUNT READY, -REGISTRATION PENDING** — making /external/photos visible is only the mount; surfacing it in Immich -requires registering an External Library at `Administration → External Libraries` (admin UI/API), a -post-deploy step compose cannot do. Flagged in the immich compose + `.felhom.yml` first_steps. Not -overclaimed. +## Phase 3 — Paperless: DEPLOYED (UID 1000) + consume verified + catalog OCR bug fixed +- Deployed via the controller deploy flow; mem-gate passed. Runs as **uid 1000 (USERMAP_UID/GID 1000)**; + consume mount `/mnt/felhom-usb/userdata/import/paperless → /usr/src/paperless/consume`. +- **Found + fixed a real catalog bug** (`67037cc`): the default `PAPERLESS_OCR_LANGUAGE=hun+eng` crash-looped + — the compose fed the `+`-joined value into `PAPERLESS_OCR_LANGUAGES` (the install list, which must be + space-separated), so the image tried to apt-install a bogus `tesseract-ocr-hun+eng` → `hun` pack missing + → Django check crash-loop. Confirmed it's a name bug not network (`tesseract-ocr-hun` installs cleanly). + Fix: `PAPERLESS_OCR_LANGUAGES=eng hun deu` (fixed space-separated superset). **Verified**: redeployed with + the default `hun+eng` → boots **healthy**, `tesseract --list-langs` = `deu eng hun`. +- **Consume VERIFIED end-to-end** (on the final hun+eng deploy): a PDF dropped into `import/paperless` by a + uid-1000 process → within ~45 s `consume_files_left=0` (paperless **read + deleted** it from the 2775 + shared dir) and `doc_count=1` (ingested/OCR'd). Proves write+delete in the shared dir works. -## Live-verify evidence (guest 9201) — all three fixes PASS -- **(c) convention**: `ls -la`/`stat` of `/mnt/felhom-usb/userdata` and every subdir → **mode 2775 - (setgid) group 1000** throughout (media/{movies,comics,...}, roms, import/{paperless,calibre}, downloads, - documents). Same on the system drive. -- **(a) FileBrowser writes userdata**: `docker exec filebrowser` (uid 1000) `mkdir` + write a file under - `/srv/felhom-usb/documents/...` → `MKDIR_OK`/`WRITE_OK`, new dir inherited gid 1000 via setgid. The - original permission-denied (root-owned `appdata` mount) is gone. -- **(b) shared media across apps**: radarr (uid 1000, UMASK 002) wrote `media/movies/Inception/Inception.mkv` - → dir **2775**, file **664 group 1000**; jellyfin (root, `media:/media:ro`) `cat` it → `MOVIEDATA`; - FileBrowser (uid 1000) listed **and deleted** it → full RW collaboration. komga (root fallback) wrote - `media/comics/Batman` group 1000 → FileBrowser **lists/reads** but cannot delete (the documented - root-fallback group-read tradeoff). -- App health after moving to the new layout: jellyfin, calibre-web, romm, radarr, audiobookshelf all - **Up healthy**; komga serving (HTTP up, tasks running, writes/reads OK) but its Docker healthcheck - (`/api/v1/actuator/health`) reports **unhealthy** — a pre-existing healthcheck-probe nuance in the komga - template, NOT a userdata/permission issue (no OOM, 0 restarts, logs clean). Flagged for a separate - healthcheck fix. +## Phase 4 — close-out +Controller **v0.66.2 Up (healthy)** on 9201; FileBrowser re-rendered with the umask wrapper; clean logs; +8.5 GiB free. Full userdata collaboration story now holds with the heavy apps live: customer-created +folder usable by apps (P1), Immich photos via external library (P2), Paperless consume (P3). -## Tests (per-commit green; authoritative Linux gate on 192.168.0.180) -`internal/appbackup`: SharedContentGID==1000, skeleton list+structure, `EnsureDirOwned` setgid+gid -(Linux; in-test companion proves a plain MkdirAll has NO setgid). `internal/stacks`: -`ParseComposeUserdataMounts` selectivity, deploy belt creates declared dirs, `withUserdataPath` -on/off-by-HDD_PATH, **migration preserves setgid+group** (Linux — mutation-proven: reverting to -0755/.Perm() drops setgid → test FAILS). Full `go build && go vet && go test ./...` green. +## Tests (Linux gate green on 192.168.0.180) +`internal/infra`: `RenderFileBrowserCompose` emits the umask-002 entrypoint wrapper. Full +`go build && go vet && go test ./...` green. -## Follow-ups (small, noted not done) -- FileBrowser-created dirs are 2755 (its umask 022) — group-read not group-write for OTHER actors; - consider a FileBrowser `UMASK`/`user:1000:1000`-with-002 so customer-made folders are group-writable. -- komga healthcheck endpoint (`/api/v1/actuator/health`) returns non-200 on komga 1.20 → "unhealthy" - despite serving; fix the healthcheck definition. -- immich/paperless live deploy needs a larger guest (the 2 GiB demo can't host them). +## Note on the deploy method (transparency) +The task specified deploying via the dashboard with Claude-in-Chrome, but no browser-automation tool was +available in this environment. I used the controller's **real deploy endpoint** (the same JSON API the +dashboard's deploy button calls, over the public URL through traefik/cloudflared) — so the mem-gate, +USERDATA_PATH injection, and deploy belt all executed; this is NOT a raw `docker compose`/CLI shortcut. +The Immich external-library registration used the Immich REST API (admin sign-up → login → create library +→ scan), fully completed and verified, not left as a documented manual step. + +## Pending / follow-ups (noted, not done) +- **komga** reports `unhealthy` via its `/api/v1/actuator/health` probe on komga 1.20 despite serving + (pre-existing healthcheck-definition issue, unrelated to userdata) — fix the healthcheck. +- Demo admin accounts created for Immich + Paperless to drive verification (creds out-of-band). diff --git a/controller/README.md b/controller/README.md index 0bc14f1..3a7079f 100644 --- a/controller/README.md +++ b/controller/README.md @@ -715,7 +715,11 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an > auto-creates a userdata dir as guest-root (covers apps not in the skeleton). > - **FileBrowser** mounts `/userdata` (was `appdata`) and runs as uid 1000 → it can create > folders + upload into the 2775 setgid tree (fixes the old permission-denied); app internals -> (`appdata/`) are no longer browsable. +> (`appdata/`) are no longer browsable. **(v0.66.2)** its entrypoint is wrapped +> `["sh","-c","umask 002; exec /home/filebrowser/filebrowser"]` so folders the customer creates come +> out **2775 (group-writable)** — the gtstef image is a single Go binary that ignores a `UMASK` env +> (verified), so the wrapper is the mechanism. Without it, customer folders were 2755 (group-read +> only) and apps in group 1000 couldn't write into them. > - **Run-identity**: PUID/PGID-1000 apps (radarr/sonarr/calibre with `UMASK=002`) write **group- > writable**, so FileBrowser can fully manage their output. Root-only apps (jellyfin :ro; komga + > audiobookshelf RW after the `user:1000` fallback) write **group-readable** via setgid (FileBrowser