docs: REPORT + README — intermediary-mount model (controller half)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,76 +1,42 @@
|
||||
# REPORT — userdata arc close-out: FileBrowser umask + Immich/Paperless deploy → controller v0.66.2 + catalog (2026-06-15)
|
||||
# REPORT — intermediary-mount re-architecture (controller half) → v0.67.5 (2026-06-15)
|
||||
|
||||
Finished the userdata arc: (1) FileBrowser-created folders are now group-writable; (2) Immich deployed
|
||||
with its external photo library registered + verified; (3) Paperless deployed and consume verified
|
||||
end-to-end. Trunk-based on `main` (controller + app-catalog). Built + deployed + live-verified on demo
|
||||
guest 9201 (now 12 GiB RAM).
|
||||
## Type
|
||||
Controller half of the intermediary-mount re-architecture (pairs with agent v0.35.1 + catalog repoint).
|
||||
Built + Linux-gated (`go vet`/`go test ./...` green) + deployed to demo guest 9201, which was then
|
||||
migrated live. Drives are now visible in-guest ONLY at the STABLE path `/mnt/felhom-drives/<name>` (the
|
||||
host swaps the backing drive underneath it — no per-drive `pct` mp, no guest reboot for attach/detach).
|
||||
Design: `felhom.eu/documentation/audits/SPIKE-intermediary-mount-2026-06-15.md`.
|
||||
|
||||
> Validate against pushed source at file:line, not this report.
|
||||
## What changed (internal/web/intermediary.go + storage_handlers.go + api/router.go + settings.go)
|
||||
- **HDD_PATH repoint.** The registered storage path + every app's HDD_PATH + FileBrowser source are now
|
||||
the stable `/mnt/felhom-drives/<name>`; the AGENT still operates on the raw `/mnt/<name>` (mapped via
|
||||
`agentWhere()` at the assign/attach/eject/decommission call sites). Enroll binds-under-the-parent
|
||||
BEFORE register (the controller only sees the drive at the stable path post-attach). New
|
||||
`settings.RepointStoragePath`; `agentapi.DiskInfo` gains `GuestPath`/`BoundUnderParent`.
|
||||
- **Drive-absent GATE** (`planDriveGates` pure core + `driveGateLoop` 30s, replacing the retired
|
||||
watchdog). An ABSENT external drive (only `/mnt/felhom-drives/<name>` paths — never SSD/system) stops +
|
||||
blocks its apps (`StoppedStacks` = gate-stopped set); a RETURNED drive re-attaches + auto-restarts
|
||||
them. `present` keys on `BoundUnderParent` (guest-usable), not host-mount. Start-gate in `actionStack`
|
||||
refuses to start an app whose drive is absent.
|
||||
- **Guest-reboot convergence** (`recreateBootStaleApps`, one-time at startup after a bounded stack-scan
|
||||
wait): docker auto-starts app containers on a guest reboot possibly before the agent re-propagates the
|
||||
drive, so they bind the empty stable dir; the controller (which restarts with the guest) recreates
|
||||
present drive-backed apps that are boot-stale/exited/restarting onto the populated path.
|
||||
- **H1 endpoints routed** (were 404): `POST /api/storage/{disconnect,reconnect,restart-apps}` → host-side
|
||||
eject/reconnect (no guest reboot).
|
||||
|
||||
## Confirmed baseline
|
||||
- Controller `main` was `d0718e8` = **v0.66.1** → shipped **v0.66.2**.
|
||||
- Guest 9201: `pct config` memory **12288** (12 GiB) — confirmed before the deploy phases (was 2 GiB).
|
||||
Version path v0.67.0 → .5 (`.1` gate-only-externals, `.2` BoundUnderParent, `.3`/`.4`/`.5` startup
|
||||
recreate hardening). Non-hollow tests + companions: `TestPlanDriveGates` (5 states incl. SSD-never-gated),
|
||||
`TestAgentWhere`, `TestRunStorageInit_Success` (agent gets RAW, registry gets STABLE).
|
||||
|
||||
## Commit hashes
|
||||
| Repo | Commit | Scope |
|
||||
|---|---|---|
|
||||
| controller | `98d5504` | v0.66.2 — FileBrowser umask 002 entrypoint wrapper + test |
|
||||
| controller | (this) | docs: README FB umask + REPORT |
|
||||
| app-catalog | `67037cc` | paperless OCR lang-pack fix (space-separated `PAPERLESS_OCR_LANGUAGES`) |
|
||||
## Live migration (guest 9201) + verification
|
||||
8 deployed apps (all on felhom-flash) repointed (app.yaml HDD_PATH rewrite + controller-driven
|
||||
down/up recreate, which decrypts the `ENC:` secrets) + registry repointed + FileBrowser re-synced + the
|
||||
legacy `mp2` deleted. **A guest reboot self-converges**: the startup recreate logged + rebound the
|
||||
drive-backed apps onto `/mnt/felhom-drives/felhom-flash` with no manual intervention. End state: **25
|
||||
containers healthy** (komga healthcheck pre-existing quirk), confinement + data integrity verified.
|
||||
|
||||
## Phase 1 — FileBrowser umask (mechanism: ENTRYPOINT WRAPPER, not UMASK env)
|
||||
Verified the gtstef/filebrowser:1.3.3-stable image is a single Go binary (`entrypoint ./filebrowser`,
|
||||
runs uid 1000) that does **NOT** honor a `UMASK` env — live test: `docker run -e UMASK=002 … umask` →
|
||||
`0022`. So `RenderFileBrowserCompose` (`internal/infra/infra.go`) wraps the entrypoint:
|
||||
`["sh","-c","umask 002; exec /home/filebrowser/filebrowser"]`. Test asserts the rendered compose carries
|
||||
it. **Live proof on 9201** (after deploying v0.66.2 → FileBrowser re-rendered): PID1 `Umask: 0002`; a
|
||||
folder created at that umask under the setgid parent → **2775 group 1000**; a uid/gid-1000 app process
|
||||
(busybox `--user 1000:1000`) wrote a file INTO that customer folder (664 group 1000) → `APP_WRITE_OK`;
|
||||
FileBrowser then deleted it. The pre-fix `Teszt` folder (2755) was the only stale dir — normalized with a
|
||||
one-time `find -type d -exec chmod g+rwxs` (no data); no 2755 dirs remain.
|
||||
|
||||
## Phase 2 — Immich: DEPLOYED + EXTERNAL LIBRARY REGISTERED + verified
|
||||
- Deployed via the controller deploy flow (the dashboard's `/api/stacks/immich/deploy` over the public
|
||||
URL — mem-gate + USERDATA_PATH + belt all run; NOT a raw compose/CLI shortcut). **Mem-gate PASSED**
|
||||
(it refused at 2 GiB; cleared at 12 GiB). All 4 containers **healthy**; memory: server 916 MiB, ML
|
||||
231 MiB, postgres 340 MiB, redis 5 MiB (~1.5 GiB total; 8.7 GiB available after).
|
||||
- External mount confirmed: `/mnt/felhom-usb/userdata/media/photos → /external/photos (ro)`, 2775 setgid.
|
||||
- **External library REGISTERED via the Immich API** (no browser tool in this env — see Note): admin
|
||||
account created (creds stored out-of-band), `POST /api/libraries` with `importPaths:["/external/photos"]`.
|
||||
- **End-to-end VERIFIED**: a JPEG placed in `media/photos` by a uid-1000 process → `POST /api/libraries/{id}/scan`
|
||||
→ `GET …/statistics` returned `{"photos":1,"total":1,"usage":631}`. The photo appears in Immich.
|
||||
|
||||
## Phase 3 — Paperless: DEPLOYED (UID 1000) + consume verified + catalog OCR bug fixed
|
||||
- Deployed via the controller deploy flow; mem-gate passed. Runs as **uid 1000 (USERMAP_UID/GID 1000)**;
|
||||
consume mount `/mnt/felhom-usb/userdata/import/paperless → /usr/src/paperless/consume`.
|
||||
- **Found + fixed a real catalog bug** (`67037cc`): the default `PAPERLESS_OCR_LANGUAGE=hun+eng` crash-looped
|
||||
— the compose fed the `+`-joined value into `PAPERLESS_OCR_LANGUAGES` (the install list, which must be
|
||||
space-separated), so the image tried to apt-install a bogus `tesseract-ocr-hun+eng` → `hun` pack missing
|
||||
→ Django check crash-loop. Confirmed it's a name bug not network (`tesseract-ocr-hun` installs cleanly).
|
||||
Fix: `PAPERLESS_OCR_LANGUAGES=eng hun deu` (fixed space-separated superset). **Verified**: redeployed with
|
||||
the default `hun+eng` → boots **healthy**, `tesseract --list-langs` = `deu eng hun`.
|
||||
- **Consume VERIFIED end-to-end** (on the final hun+eng deploy): a PDF dropped into `import/paperless` by a
|
||||
uid-1000 process → within ~45 s `consume_files_left=0` (paperless **read + deleted** it from the 2775
|
||||
shared dir) and `doc_count=1` (ingested/OCR'd). Proves write+delete in the shared dir works.
|
||||
|
||||
## Phase 4 — close-out
|
||||
Controller **v0.66.2 Up (healthy)** on 9201; FileBrowser re-rendered with the umask wrapper; clean logs;
|
||||
8.5 GiB free. Full userdata collaboration story now holds with the heavy apps live: customer-created
|
||||
folder usable by apps (P1), Immich photos via external library (P2), Paperless consume (P3).
|
||||
|
||||
## Tests (Linux gate green on 192.168.0.180)
|
||||
`internal/infra`: `RenderFileBrowserCompose` emits the umask-002 entrypoint wrapper. Full
|
||||
`go build && go vet && go test ./...` green.
|
||||
|
||||
## Note on the deploy method (transparency)
|
||||
The task specified deploying via the dashboard with Claude-in-Chrome, but no browser-automation tool was
|
||||
available in this environment. I used the controller's **real deploy endpoint** (the same JSON API the
|
||||
dashboard's deploy button calls, over the public URL through traefik/cloudflared) — so the mem-gate,
|
||||
USERDATA_PATH injection, and deploy belt all executed; this is NOT a raw `docker compose`/CLI shortcut.
|
||||
The Immich external-library registration used the Immich REST API (admin sign-up → login → create library
|
||||
→ scan), fully completed and verified, not left as a documented manual step.
|
||||
|
||||
## Pending / follow-ups (noted, not done)
|
||||
- **komga** reports `unhealthy` via its `/api/v1/actuator/health` probe on komga 1.20 despite serving
|
||||
(pre-existing healthcheck-definition issue, unrelated to userdata) — fix the healthcheck.
|
||||
- Demo admin accounts created for Immich + Paperless to drive verification (creds out-of-band).
|
||||
## Known residuals (documented)
|
||||
- Startup recreate is state-sampled; an app cleanly stopped at the exact sample instant may need one
|
||||
manual start (1/8 once). Deterministic fix = a guest-boot-id signal from the agent (future).
|
||||
- felhom-usb left on its legacy bind (no deployed apps + stale `decommissioned` intent).
|
||||
|
||||
@@ -621,6 +621,24 @@ not just those with HDD data. Non-HDD apps can configure destination, method, an
|
||||
|
||||
### 4. Storage Management
|
||||
|
||||
> **⚠️ INTERMEDIARY-MOUNT model (v0.67.x, pairs with agent v0.35.x).** External data drives are now
|
||||
> visible in the guest at a STABLE path **`/mnt/felhom-drives/<name>`**, NOT the raw `/mnt/<name>`. The
|
||||
> agent keeps a single permanent parent bind `/mnt/felhom-drives` in the guest and swaps each drive's
|
||||
> `felhom-data` namespace **underneath it host-side** (`mount --bind`), so the guest sees attach/detach
|
||||
> **live with no reboot** (mount propagation), the bind source never disappears (C1-immune), and only
|
||||
> `felhom-data` crosses in (confinement). The per-drive `pct set -mpN` bind is **deprecated**.
|
||||
> - The registered storage path + every app's **`HDD_PATH`** + the FileBrowser source = the stable
|
||||
> `/mnt/felhom-drives/<name>`. The controller maps it back to the raw `/mnt/<name>` (`agentWhere()`)
|
||||
> only for agent calls (assign/attach/eject/decommission). `GET /api/disks` carries `guest_path` +
|
||||
> `bound_under_parent` (the guest-visible signal).
|
||||
> - **Drive-absent gate** (`internal/web/intermediary.go`, `driveGateLoop` 30s): an absent
|
||||
> `/mnt/felhom-drives/<name>` drive stops + blocks its apps (`StoppedStacks` = the gate-stopped set);
|
||||
> a returned drive re-attaches + auto-restarts them; `actionStack` refuses to start an app whose drive
|
||||
> is absent ("tárhely nem elérhető"). SSD/system paths are never gated.
|
||||
> - **H1 endpoints** `POST /api/storage/{disconnect,reconnect,restart-apps}` drive the host-side
|
||||
> eject/reconnect (no guest reboot). A guest reboot self-converges (`recreateBootStaleApps` recreates
|
||||
> boot-stale drive-backed apps onto the re-propagated drive once the agent re-binds it).
|
||||
>
|
||||
> **⚠️ Rebuilt on the agent-delegated disk model (v0.43.0), made ROLE-AWARE in v0.44.0, UX-polished in
|
||||
> v0.45.0.** After the 8C
|
||||
> de-privileging, the controller holds **no Proxmox/disk credentials and no destructive authority** — disk
|
||||
|
||||
Reference in New Issue
Block a user