From 89d64f64b33eb22d70bd08d90b8b8d97aefb915c Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 26 Jul 2026 08:30:45 +0200 Subject: [PATCH] R-75 SHIPPED (controller v0.172.0): docs, capability-map note, ROADMAP collapse MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New documentation/controller/import-and-data-paths.md: the canonical import root (and why it is NOT a registered StoragePath), the three data_paths roles, the Fork-3 validation asymmetry, the class-driven copy rule, and the seven invariants a future change must not break. Capability map "File access via browser" — status DELIBERATELY UNCHANGED. The drop-zone now has its own FileBrowser source and the app page carries a deep link, both verified live, but nothing drove the FileBrowser HTTP UI (no browser on DooPlex), so the row's standing "browse is exercised in no doc" caveat still holds and PROVEN-LIVE remains unearned. R-75 collapsed to its shipped one-liner. R-76 left open — this task does not fix it, and nothing built here assumes an import/* directory stays 2775. --- .../architecture/00-capability-map.md | 2 +- documentation/backlog/ROADMAP.md | 2 +- .../controller/import-and-data-paths.md | 117 ++++++++++++++++++ 3 files changed, 119 insertions(+), 2 deletions(-) create mode 100644 documentation/controller/import-and-data-paths.md diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index a834aa2..0e4d7cc 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -93,7 +93,7 @@ | Documents/OCR | paperless-ngx (classified) | **PROVEN-LIVE** | `CAMPAIGN-6C` 4P-1 (deploy paperless-ngx, ingest 3 docs via consume flow, OCR + PDF/A ~90s) + 4P-2/3/5 | Consume-folder ingestion awkward without SMB → R-7 | | Files from Windows Explorer / Mac Finder (SMB **server**) | controller v0.145.0 + `felhom-samba:1.0.0` | **PROVEN-LIVE** | `felhom-controller` REPORT.md (v0.144.0) + `controller/sharing.md`; transport verdict `audits/SPIKE-lan-discovery-2026-07-18.md` | „Megosztás" page: enable + one household password + shares (new folder or picked existing, per-share read-only). Fourth protected infra stack (host-net, smbd+**nmbd**+wsdd). Live on demo: 445 reachable, NetBIOS `FELHOM` resolves, write/read byte-compare PASS, write to a read-only share REFUSED, SMB writes land as uid 1000. **Explorer leg PASSED 2026-07-18 (Viktor):** Network → FELHOM → both shares open; a real Explorer *save* into `dokumentumok` landed owned uid 1000, and a write into the read-only `filmek` was refused by Windows with the folder left untouched. Share data RIDES BOTH BACKUP TIERS (R-7b, controller v0.145.0, Model B′ sibling shares source): tier-2 cross-drive legs + an offsite `_shares` restic snapshot carrying the share definitions and the credential copy, with a „Megosztások" restore. **All four legs PROVEN-LIVE on demo 2026-07-18** — tier-2 tree md5-verified; offsite snapshots `e0b9d723` (Viktor 12:18:16Z) and `4e2b15ec` both carrying manifest + `passdb.tar`; restore round-trip returned a deleted probe file byte-identical and a deleted share DEFINITION with its original flags without overwriting live files; samba liveness → hub-accepted `health_critical`. Remaining human leg: SMB positive auth with the real household password | | Media to TV via DLNA | — | **MISSING** | — | Jellyfin app exists; DLNA/SSDP unvalidated → R-6, R-8 | -| File access via browser | FileBrowser (infra app, auto-mount sync) | **IMPLEMENTED** | FileBrowser runs healthy + userdata-bound (`storage-lifecycle-acceptance-2026-06-15`, `CAMPAIGN-3`) | Actual browse/download **through FileBrowser** is exercised in no doc. (Cited `CAMPAIGN-2` T-PAGE-ALL renders only the controller dashboard pages, not FileBrowser.) Demoted | +| File access via browser | FileBrowser (infra app, auto-mount sync) | **IMPLEMENTED** | FileBrowser runs healthy + userdata-bound (`storage-lifecycle-acceptance-2026-06-15`, `CAMPAIGN-3`) | Actual browse/download **through FileBrowser** is exercised in no doc. (Cited `CAMPAIGN-2` T-PAGE-ALL renders only the controller dashboard pages, not FileBrowser.) Demoted. **2026-07-26, controller v0.172.0 (R-75) — status DELIBERATELY UNCHANGED.** The canonical drop-zone now has its own FileBrowser source („Beolvasás" → `/srv/beolvasas`, a separate bind of `/userdata/import`) and the app page carries a per-app deep link into it. Verified live on demo-hp: the source and bind are in the generated config, the app page renders `https://files.enkisfelhom.hu/files/Beolvas%C3%A1s/paperless`, and a file written through FileBrowser's OWN mount was consumed and deleted by paperless in ~30 s. **That is still not a browse.** Nothing in this arc drove the FileBrowser HTTP UI — no browser exists on DooPlex — so the row's standing caveat survives intact and the upgrade to PROVEN-LIVE remains unearned. What it would take: a human click-through, or an authenticated `/api/resources` round-trip against the live instance. See `controller/import-and-data-paths.md` | | **Indítópult (app launcher) — one-tap grid of the household's openable apps** | controller v0.163.0 | **IMPLEMENTED** | New FIRST sidebar page `/launcher`: colored tiles (deterministic slug color or `.felhom.yml` `brand_color`) + white glyph/monogram, one per openable app (tile ⟺ „Megnyitás" — subdomain presence is the single criterion; controller excluded). Operational → `` to the public URL; stopped → greyed + state badge, no link. `/` stays the Vezérlőpult. Endpoint-level + render-test verified; `felhom-controller/REPORT.md` (2026-07-24) | Live operator click-through of a real tile → app pending (browser automation not available on DooPlex). Follow-up: curate `brand_color` for top catalog apps (R-72). Sharing the launcher outside the household is now the capability-URL guest link — see the row below | | **Indítópult megosztás (vendég link)** — capability URL `/s/` serves a standalone read-only guest launcher (no account, no admin session); optional per-share password; QR | controller v0.165.0 | **IMPLEMENTED** | 160-bit `crypto/rand` token, constant-time match (empty stored = disabled = byte-identical to the mux default 404); guest headers `noindex`/`no-referrer`/`no-store`; optional SEPARATE bcrypt share password + its own per-IP attempt map; signed cookie `HMAC(token`\|`passwordHash)` keyed with `session_secret` (rotate-token OR change-password invalidates all cookies); token redacted in logs (`/s/`). Groups A–G (14 tests) + 3 red-proofs; **§13 endpoint-level live validation on 9201 all-pass** (`felhom-controller/REPORT.md` 2026-07-24). **Design ruling: member accounts SUPERSEDED by this capability-URL model; per-member tile visibility parked under the SSO arc (R-15).** | Full operator browser click-through + a validation doc pending → then PROVEN-LIVE. Accepted residuals: link-preview crawlers fetch once (noindex prevents indexing); reverse-proxy/CF access logs hold the path (ops-tier); the modal link carries the request Host (LAN-IP admin ⇒ LAN-IP link) | | Forgot dashboard password → instant reset code | controller v0.123, hub | **PROVEN-LIVE** | `DRILL-day0-take2-2026-07-12` F-15 (live re-run of the exact failure path: hash applied 1s after request, code accepted first try) | | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index eab3f49..bdae190 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -104,7 +104,7 @@ | R-72 | Curate `brand_color` for the top catalog apps | XS | idea | Parked follow-up to the v0.163.0 launcher. `.felhom.yml` `brand_color` (`#rgb`/`#rrggbb`) overrides the deterministic slug-hash tile color; no catalog app sets it yet. Pick brand-accurate colors for the most-installed apps so their launcher tiles match their real brand. Catalog-only change (`app-catalog-felhom.eu`), `brand_color` is already `omitempty` and consumed by the controller | | R-73 | **SMART history / trending (hub-side)** | M | idea (parked) | Parked follow-up to the v0.169.0 disk-health card + degradation alert (agent v0.94.0 serializes SMART into `/disks`; controller renders „Lemezek állapota" + fires `disk_health_degraded` on a degradation). Today the verdict is point-in-time and the baseline is in-memory (a controller restart re-baselines silently). Hub-side would let the operator SEE a disk's SMART counters trend over time (reallocated/pending sectors climbing, NVMe wear %), catch slow degradation the 6h transition check can miss between reboots, and persist the baseline across restarts. Needs a hub ingest surface for periodic SMART snapshots + a small time-series store + an operator trend view. **No demo hardware exposes real SMART** (virtualized), so this only earns evidence on a real box | | R-74 | **Island control plane on a CLUSTER (Peti's 2 nodes)** — bring R-50's island bridge to a multi-node PVE cluster. | M | idea (Phase C of R-50, parked) | R-50 shipped the island for the ONE-host fleet (demo-hp, demo-felhom). A cluster needs **bridge parity on every node**: either per-node identical `/etc/network/interfaces` `vmbr9` stanzas (simplest, drift-prone) or — preferred at ≥2 nodes — a Proxmox **SDN zone/vnet** defined cluster-wide (one definition, auto-applied per node). The guest island IP is per-guest + node-independent; the **agent-follows-guest** rule holds (each node's agent binds its own `vmbr9` `169.254.253.1`). Migration order per the spike: drill-proven → demo (done) → **Peti (this row)**. Its own supervised runbook, coordinated with Peti (a live customer). Completes the capability-map "site/network change" row for clustered installs. Source: `audits/SPIKE-island-bridge-2026-07-25.md` (cluster-parity finding) + `RUNBOOK-island-migration.md` (single-host procedure to generalise) | -| R-75 | **Catalog-derived userdata skeleton + import surfaces** — replace the hardcoded `appbackup.UserdataSkeleton()` Go list with a set DERIVED from the catalog, and give the customer a way to find an app's folders: a named FileBrowser sidebar source for the `import/` tree, a per-app deep link from the controller's app page, and `data_paths:` in `.felhom.yml` as a role+Hungarian-label ANNOTATION over paths that must already exist as userdata-rooted compose binds. | M | **spiked (2026-07-26) — GO, one mandatory constraint** | `audits/SPIKE-catalog-data-paths-2026-07-26.md`. **Phase 0 HOLDS:** the path set is fully derivable from `ParseComposeClassifiableBinds` (`stacks/classify_binds.go:38`) and `ValidateBackupSpec`'s refusal set (`appbackup/classify.go:97`) already covers the path-safety class `data_paths:` needs — annotation-only introduces **no new filesystem-write primitive**. Every non-`${VAR}` host token in all 53 templates is a named docker volume or the docker socket; **no** app's folder is reachable only via env indirection; **zero** templates use long `type: bind` syntax. **The derivation is ALREADY LIVE at deploy time** — `stacks/manager.go:183` `ensureUserdataMounts` ("the deploy belt") derives userdata dirs from compose and applies 2775/GID-1000 per app; proven by a clean two-box control (P5: `media/podcasts` exists on demo-felhom where audiobookshelf is deployed and NOT on demo-hp, and it is in no skeleton). So this item is mostly *promoting a shipped lazy mechanism to an eager, annotated one*. **Diff is tiny:** derived set 14 dirs vs hardcoded 14 — `+media/podcasts`, `−documents`; `documents` is implied by no app, exists on BOTH boxes, and is the entire removal risk → **zero-removals invariant is mandatory**, proven against the P5 before-state. Only **2 of 53** apps have an `import/*` path (paperless-ngx, calibre-web), and **both are `class: excluded`** — any "put your files here" copy must say temporary+unbacked, drivable from data since every catalog class is `explicit`. **MANDATORY CONSTRAINT (P6):** the naive map-based derivation produced **20 distinct outputs from 20 identical runs** (1/20 identical); `fbNeedsRecreate` force-recreates on any byte diff, so that is a guaranteed FileBrowser restart loop on all ~14 sync call sites — fleet-wide, v0.151-class. `sort.Strings` → 20/20. **The spec must state the sort + a determinism red-proof as a requirement.** Mitigating: the catalog sync does **not** currently trigger `SyncFileBrowserMounts` (`main.go:215` wires only `ScanStacks` + `InjectMissingFields`, and even that is change-gated), so the 15-min storm only appears if the impl wires them together. **FileBrowser Quantum 1.3.3 carries the UI half:** 4 sources incl. nested + accented/spaced/`&` names all index `ready` with no warning; deep-link template `{baseURL}files/{encodeURIComponent(source)}/{encodeURIComponent'd segments}` is constructible in Go from `(sourceName, relPath)` alone, and the router guard preserves `?redirect=fullPath` so a cold deep link survives login. Use `url.PathEscape` — **not** `QueryEscape` (space→`+` breaks a path segment). Scale safe to 12 sources server-side. **Share guard needs no change:** `userdata/import` is not in `SharingDeniedRoots` and the real `sharingResolvePath` accepts it while still refusing appdata/root/relative/traversal/symlink-escape (P4, all green, no live share created). **Two pre-existing defects surfaced, deliberately NOT fixed here** — see R-76. Flips capability-map row *File access via browser* (line 96, currently **IMPLEMENTED** with the honest caveat "actual browse/download through FileBrowser is exercised in no doc") toward PROVEN-LIVE, and is the natural home for R-7 slice 2's "curated app-folder presets (one-click paperless `consume/`)". **Design forks recorded, awaiting operator ruling:** per-drive vs canonical import root (rec: per-drive); all-apps vs deployed-only (rec: all-apps skeleton, deployed-only in the UI); unknown-role fails OPEN per the R-57 lifecycle precedent vs whole-block reject per the `backup:` precedent (rec: OPEN for roles, reject for malformed paths); backup-class collision copy (rec: data-driven from the class) | +| R-75 | **Catalog-derived userdata skeleton + import surfaces** | M | **SHIPPED (controller v0.172.0 + catalog, 2026-07-26)** | `${IMPORT_PATH}` = `/userdata/import` — ONE canonical drop-zone per box on the system drive, with **no per-drive fallback** (unresolvable leaves the var unset so compose fails loudly). *Operator ruling overriding the spike's Fork-1 per-drive recommendation:* each drop-zone app has exactly one ingest bind, so a per-drive `import/` would put a dead lookalike on every other drive — unbacked too, since `import/*` is `class: excluded`. Third `BindRoot` (`RootImport`) threaded through the classifier AND every resolver (`resolveAbs` would otherwise have named a directory on the WRONG DRIVE); catalog-derived skeleton, **sorted** (the naive map-order derivation measured 20 distinct outputs from 20 identical runs — a fleet-wide FileBrowser restart loop via `fbNeedsRecreate`); one authoritative compose parser; FileBrowser „Beolvasás" source; app-page „Hova tegyem a fájlokat?" with `PathEscape` deep links and **class-driven** copy; `data_paths:` annotation (malformed path ⇒ whole-block reject, unknown role ⇒ fails OPEN); system-owned `beolvasas` SMB share refused server-side at handler AND store. **Live-proven on demo-hp:** consume bind resolves to the system drive, a file dropped through FileBrowser's mount was ingested and deleted in ~30 s, the share auto-appeared on enable and survived a direct delete POST, and the before/after directory diff on BOTH boxes shows **zero removals** (demo-hp gained `media/podcasts`, the one genuinely derived entry). Two defects found and fixed during the live legs: `EnsureImportRoot` left the parent `userdata` at 755, and the carry-list initially re-created the per-drive drop-zone forever. Capability-map row *File access via browser* **deliberately NOT upgraded** — nothing drove the FileBrowser HTTP UI. Docs: `controller/import-and-data-paths.md`, spike `audits/SPIKE-catalog-data-paths-2026-07-26.md` | | R-76 | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** | S | idea (surfaced by the R-75 spike, 2026-07-26) | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say the wrapper fails to achieve the intended mode either. **(b)** `import/calibre` is live on demo-felhom at `755 1000:1000` where every sibling is `2775 root:1000` — with `media/books` on the same box, same app, same deploy, at `2775` as the control, and no parser asymmetry (checked: `ParseComposeUserdataMounts` picks up both calibre-web binds). Consistent with the consuming app rewriting the mode of its own ingest dir after the deploy belt sets it; **not confirmed causally**. Together they mean **a drop-zone directory's mode is not stable against either the customer or the consuming app** — verify before building anything that assumes `import/*` stays 2775. Flips no capability-map row today (latent); would become customer-visible the moment a non-1000 content app enters the catalog | | R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | **SECOND-HARDWARE PROOF 2026-07-21 (slice C, demo-hp): the pairing flow is no longer a one-board result.** The universal secret-free ISO was booted on a completely different machine — an **HP t740 (Ryzen V1756B, AMI M42 firmware)** versus the N100 it was proven on — and the whole chain ran on **virgin hardware in one pass**: armed install → self-registration as an unclaimed appliance → operator bind → day-0, ending with a running customer guest 9201 and agent 0.92.1 checking in as host `demo-hp-bb76ea`. **Two things generalise from the second board specifically:** the **shim** loader booted with **Secure Boot ENABLED** (`mokutil --sb-state` → `SecureBoot enabled`), confirming the mkimage/SB-off dance is an N100-firmware workaround and NOT a Felhom requirement; and the exact-serial disk filter selected the SanDisk system SSD while leaving the box's **1TB NVMe untouched and unenrolled** (its prior NTFS partition is still intact, unmounted, in no LVM/ZFS) — the destructive path stayed inside its filter on hardware it had never seen. **Not clean, and the failures are filed:** the install got no DHCP on the 4-port NIC and baked a static fallback rather than aborting (**R-59**), which cost a cable move and a hand-repair; and the console was unreachable because the baked root password is unknowable (**R-61**) | R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** | diff --git a/documentation/controller/import-and-data-paths.md b/documentation/controller/import-and-data-paths.md new file mode 100644 index 0000000..c825b70 --- /dev/null +++ b/documentation/controller/import-and-data-paths.md @@ -0,0 +1,117 @@ +# Import root & `data_paths` (R-75) + +Shipped in controller **v0.172.0** (2026-07-26). Evidence: +`audits/SPIKE-catalog-data-paths-2026-07-26.md`. + +--- + +## 1. The canonical import root + +``` +${IMPORT_PATH} = /userdata/import + = /mnt/sys_drive/felhom-data/userdata/import (both demo boxes) +``` + +**One drop-zone per box, on the system drive — never per data drive.** It is resolved from +`cfg.Paths.SystemDataPath` via `appbackup.NamespaceRoot(sys, false)` (the system drive holds a +`felhom-data` SUBDIR, so it is the `inGuestDrive=false` case) and exposed as +`stacks.Manager.GetImportRoot()`. + +**Why canonical, not per-drive** — this overrides the spike's Fork-1 recommendation, on a stronger +argument the spike did not weigh. Each drop-zone app has exactly **one** ingest bind. On a two-drive +box a per-drive `import/` would therefore put a folder that *looks* like a drop-zone on every drive +while only one of them does anything; and because `import/*` is `class: excluded`, files stranded in +a dead one are never backed up either. A canonical root is the only shape with no dead drop-zone. + +**No fallback, deliberately.** If the system root is unresolvable, `IMPORT_PATH` is left **unset** and +compose fails loudly on `${IMPORT_PATH}`. It must never fall back to `/userdata/import` — +that silently recreates the shape this design removes. + +**The system drive is NOT a registered `StoragePath`** (verified on both demo boxes, 2026-07-26). +Registering it would make a 50 GB volume holding the recovery units a customer-visible drive, a deploy +target and a wipe candidate. Everything that must reach the import root therefore does so explicitly: + +| Surface | How it reaches the import root | +|---|---| +| Deploy belt | `ensureUserdataMounts` — an import loop that is **not** drive-absent-gated | +| Skeleton | `Manager.EnsureImportRoot()`, not the per-storage-path skeleton loop | +| FileBrowser | an explicit extra bind `:/srv/beolvasas`, outside `buildFileBrowserPaths` | +| SMB share | written directly to the store by `ensureImportShare` | + +That last one matters: `sharingResolvePath` **refuses** the import root, correctly — it validates +paths a *customer* supplied through the picker and requires containment in a registered live root. A +controller-generated constant is a different trust class. (`TestImportRoot_NotReachableViaTheCustomerPicker` +pins both halves.) + +--- + +## 2. The three roles + +`data_paths:` is a **pure annotation** over paths that already exist as compose binds. It can never +declare a path — `ValidateDataPaths` rejects any entry matching no bind. That is the property which +keeps the design cheap: **no new filesystem-write primitive is introduced from catalog data.** + +```yaml +data_paths: + - path: paperless # relative to the bind's root + root: import # import | userdata | hdd + role: import # import | library | export + label: "Beolvasandó dokumentumok" +``` + +| role | meaning | rendered? | +|---|---|---| +| `import` | drop-zone; the app consumes and usually DELETES from it | yes, + free space | +| `library` | where the customer's files live | yes | +| `export` | the app writes results here for the customer | yes | + +`root: hdd` entries are **not** rendered: `appdata/` is app-internal, FileBrowser does not mount it, +and a link would 404. + +### Validation is asymmetric, and that is a decision + +| defect | outcome | precedent | +|---|---|---| +| malformed **path** (absolute, `..`, backslash, non-`Clean`'d, matches no bind, duplicate, unknown root) | **whole-block reject**, one ERROR, no folder card | `backup:` — paths govern DATA HANDLING | +| unknown **role** | **fails OPEN** — that entry is not surfaced, one WARN, the rest renders | `Lifecycle` — roles govern PRESENTATION; a typo must never brick a template | + +The path rules reuse `appbackup.ValidateRelPath`, the single refusal set shared with `backup:`. +**Do not write a second path validator.** + +--- + +## 3. The class-driven copy rule + +The consequence sentence on the app page is driven by the **derived backup class**, never by the role +and never by a per-app string, so the promise the UI makes cannot drift from what the backup engines +actually do: + +| class | Hungarian copy | +|---|---| +| `excluded` + `role: import` | „Ide másold a feldolgozandó fájlokat. Az alkalmazás beolvassa, majd törli innen — ez a mappa átmeneti, és nem készül róla biztonsági mentés." | +| `excluded` (other) | „Ez a mappa átmeneti, és nem készül róla biztonsági mentés." | +| `mandatory` / `optional` | „Itt tárolódnak a fájljaid. Biztonsági mentés készül róla." | +| unclassified | **nothing** — an unverified backup promise is worse than no sentence | + +Both catalog import paths are `class: excluded`, so both must carry the temporary-and-unbacked +warning. `TestConsequenceIsClassDriven` pins it. + +--- + +## 4. Invariants a future change must not break + +1. **Zero removals.** `UserdataSkeletonCarry()` is the v0.171.0 hardcoded list, retained forever, so + derivation can only ADD. `documents` is implied by no catalog app and exists on both demo boxes. +2. **The skeleton set is SORTED.** `fbNeedsRecreate` force-recreates FileBrowser on any byte diff + across ~14 `SyncFileBrowserMounts` call sites; the naive map-order derivation measured **20 + distinct outputs from 20 identical runs**. +3. **The catalog sync is NOT wired to `SyncFileBrowserMounts`.** Connecting them turns a 15-minute + catalog cycle into a 15-minute container-restart cycle. +4. **`url.PathEscape` per path segment for deep links — never `url.QueryEscape`** (it encodes a space + as `+`, which in a path segment is a literal plus). Let `html/template` do the attribute escaping. +5. **Sharing stays opt-in.** The `beolvasas` share is created when sharing is ENABLED, never as a side + effect of deploying an app. +6. **`${IMPORT_PATH}` binds do not resolve against `hddPath`.** Any new consumer of `BindRoot` needs + an explicit `RootImport` case with the import root passed in. +7. **Do not assume an `import/*` directory stays `2775`** — see R-76: FileBrowser creates `0755` + without propagating setgid, and a live `import/calibre` was found at `755`.