diff --git a/REPORT.md b/REPORT.md index c99dc4c..4d89108 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,126 +1,127 @@ -# REPORT — F-S2 + F-S3: compose-derived appdata dir resolution (controller v0.131.0) +# REPORT — Backup classification: schema + parser + classifier + catalog migration (controller v0.132.0) ## Summary -The controller assumed an app's HDD appdata dir is always `appdata/`. paperless-ngx binds -`${HDD_PATH}/appdata/paperless/...` (stack `paperless-ngx`, dir `paperless`), so every consumer that -keyed by stack name silently missed it via a stat-and-skip. One canonical resolver -(`appbackup.AppDataDirNames`) now derives the real dir name(s) from the app's compose `${HDD_PATH}` -binds; all consumers use it. Task 1 of the backup-classification-redesign arc -(`felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`), independent of the -classification schema. +Task 2 of the backup-classification-redesign arc +(`felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`). Ships the +referential-coupling classification as **DATA + PARSER + PURE CLASSIFIER — deliberately INERT**: no +backup tier changes behavior. The `backup:` block now lands in `.felhom.yml` for all 13 bind-bearing +catalog apps; the controller parses + validates it loudly and a pure classifier resolves per-bind +classes with the SQ5 two-level default. Task 3 (tier policy engine) and Task 4 (manual `.fab` UI) +consume this — nothing does yet except a validation log pass. ## Confirmed baselines -| Repo | `main` @ start | Version | → | -|------|----------------|---------|---| -| felhom-controller | `b42904b` (verified 2026-07-14) | v0.130.0 | **v0.131.0** (commit `68f0e0c`, pushed to `main`) | +| Repo | `main` @ start | → | Commit(s) on `main` | +|------|----------------|---|---------------------| +| felhom-controller | `af98c53` (v0.131.0) | **v0.132.0** | `0649f9a` (code+tests+docs) | +| app-catalog-felhom.eu | `365a017` | (no version scheme) | `21e8df1` (13 blocks + CHANGELOG) | + +Trunk-based, direct to `main`. Controller committed + deployed FIRST, then the catalog blocks (so the +parser validates them on first sync). ## Files created / modified -**Part 1 — resolver + helpers** -- `controller/internal/appbackup/paths.go` — NEW pure `AppDataDirNames(hddPath, stackName, hddMounts) []string` - (dedup+sort; fallback `[stackName]`) + `AppDataBindsPresent(hddPath, hddMounts) bool` (WARN predicate); - `AppDataDir` doc updated. -- `controller/internal/backup/tier2.go` — `appDataDirNames` / `tier2AppDataName` (the one place the N>1 - refusal error `errTier2MultiDir` is built) / `tier2AppDataBindsPresent`. -- `controller/internal/backup/appbackup_bridge.go` — forwarders for the two new appbackup funcs. -- `controller/internal/stacks/migrate.go` — `ResolveAppDataDirNames` (exported; used by handlers) + - `resolveAppDataDirs` (names + `declared` bool for the WARN); `migSeams.resolveNames` test seam. +**Controller (`0649f9a`)** +- `internal/appbackup/classify.go` (NEW) — schema (`BindClass`, `BindRoot`, `BackupSpec`, `BindSpec`, + `ComposeBind`, `ClassOrigin`, `ClassifiedBind`) + `ValidateBackupSpec` (whole-block-reject) + + `ClassifyBinds` (two-level default; explicit beats `:ro`; nil spec → legacy/false). +- `internal/stacks/classify_binds.go` (NEW) — `ParseComposeClassifiableBinds`: `${VAR}`-relative binds + + `:ro` flag, `ParseComposeUserdataMounts` scanner shape, dedupe (first-`:ro`-wins). +- `internal/stacks/metadata.go` — `Metadata.Backup` field + the LoadMetadata validation choke point + (bad block → nil + one `[ERROR]` → legacy) + `Manager.ClassifiedBinds`. +- `internal/appbackup/appdata.go` — `StackDataProvider.GetStackClassifiedBinds`. +- `internal/backup/appbackup_bridge.go` — `ClassifiedBind` alias. +- `cmd/controller/main.go` — `stackAdapter.GetStackClassifiedBinds` delegation. +- 5 test fakes (`recovery_unit_test.go`, `tier2_restore_test.go`, `volume_dumps_test.go`, + `api/backup_snapshots_test.go`, `web/async_restore_test.go`) — nil-stub method ONLY (no logic edits). +- Tests: `appbackup/classify_test.go`, `stacks/classify_binds_test.go`, `stacks/classify_wiring_test.go`. +- Docs: `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `controller/README.md`. -**Part 2 — tier-2 backup/info/restore** -- `controller/internal/backup/tier2.go` — `RunTier2` resolves the appdata dir name (L146 site), refuses - N>1 (honest `no_target` status + `[ERROR]`), WARNs on a declared-but-absent dir; new `tier2Mirror` - seam on both rsync legs; rewrote the lying package header (no userdata/no-wholesale truth). -- `controller/internal/backup/backup.go` — `tier2Mirror` seam field. -- `controller/internal/backup/tier2_restore.go` — `RestoreTier2Files` targets the resolved live dir; - `errTier2MultiDirRestore` sentinel, refused BEFORE `StopStack`. - -**Part 3 — migrate (F-S3)** — `controller/internal/stacks/migrate.go`: all six per-app appdata legs -(collision `:328`, size `:354`, copy `:449`, verify `:491`, cleanup `:556`, skip-set `:605`) loop the -resolved name(s); copy leg WARNs on a missing declared dir. Merge walk / scope gating / journal untouched. - -**Part 4 — display** — `controller/internal/web/handlers.go`: `appDetailsForPath` sums the resolved -dir(s); `dirSizeHuman` split into `dirSizeBytesWalk` + `humanizeDirBytes`. - -**Part 5 — truth repairs** — `controller/cmd/controller/main.go:1357` export-adapter comment corrected; -the v0.130.0 CHANGELOG's false "copies the namespace wholesale" sentence is explicitly noted false in the -v0.131.0 entry (old entry left intact as the record). - -**Docs** — `CHANGELOG.md` (v0.131.0, newest-on-top, incl. v0.130.0 correction), `REUSE.md` -(AppDataDirNames/BindsPresent + tier2Mirror/resolveNames seams), `CONTEXT.md`, `controller/README.md`. - -## Commits on `main` - -| Hash | Contents | -|------|----------| -| `68f0e0c` | F-S2 + F-S3 code + all tests + docs (single commit) | +**Catalog (`21e8df1`)** — `backup:` block added after `resources:` in 13 `.felhom.yml` files (immich, +paperless-ngx, nextcloud, calibre-web, audiobookshelf, komga, navidrome, radarr, sonarr, emby, +jellyfin, plex, romm) + `CHANGELOG.md`. ## Tests -`go build ./... && go vet ./... && go test ./...` — **all green** (every package `ok`; vet clean). -**+9 test functions** (before → after in the touched packages): +`go build ./... && go vet ./... && go test ./...` — **all green**, both repos' content parses clean. +**+14 test functions** (before → after in the touched packages; a 9-case validation subtable inside): -- `internal/appbackup/appdatadirnames_test.go` (NEW, +2): `TestAppDataDirNames` (Group A table: - paperless mismatch, matching name, two-name sorted, media+export dedupe, non-appdata + foreign-drive - filtered, whole-root bind ignored, empty→fallback, unclean path), `TestAppDataBindsPresent`. -- `internal/backup/tier2_appdata_test.go` (NEW, +6): `TestRunTier2_PaperlessShape` (Scenario A), - `TestRunTier2_LegacyShape` (Scenario B: match + no-binds fallback), `TestRunTier2_MultiDirRefusal` - (Scenario C, exact Hungarian reason, mirror-count 0), `TestTier2Info_MultiDirRefusal`, - `TestRestoreTier2Files_ResolvedLiveDir` (Scenario E), `TestRestoreTier2Files_MultiDirRefusal` - (refused before StopStack). `t2rFakeProvider` extended with configurable `GetStackHDDMounts`. -- `internal/stacks/migrate_fs3_test.go` (NEW, +1): `TestMigrate_PaperlessShape_ScopeApp` (Scenario D: - copy+verify captured src/dst = appdata/paperless; size/collision/cleanup/skip-set all on the resolved - dir; never appdata/paperless-ngx). +- **Group A (classifier):** `TestClassify_ImmichShape` (explicit beats `:ro`), `_TwoLevelDefault` + (unlisted writable→mandatory, `:ro`→excluded), `_NilSpecLegacy`, `_BareRootFallsToDefault`, + `_SameRelPathBothRoots`. +- **Group B (validation):** `TestValidateBackupSpec_Defects` (9 cases: unknown/empty class, empty/ + absolute/`..`/backslash/non-clean path, duplicate, no-matching-bind — each names the entry) + `_ValidAndNil`. +- **Group C (parser):** `TestParseClassifiableBinds_Variants` (ro/rw/ro,z/z, dedupe, bare root, both + roots, non-volume sections), `_CleanAndSubpaths` (false-prefix rejected), `_Missing`. +- **Group D (end-to-end, NO seams):** `TestClassifiedBinds_ValidFixture`, `_RejectedFixture` (RP-4), + `_NoBlockLegacy` — real files through the REAL LoadMetadata path. +- **Group E (catalog fidelity):** `TestCatalogBlocks_Fidelity` (immich, paperless-ngx, sonarr exact class sets). + +**Inertness proof (Scenario C):** the entire pre-existing suite is green with **zero test-logic +edits** — only the mandated nil-stub method was added to each fake (verified: `+14 −1` across the 5 +fake files, the `−1` being a gofmt re-alignment). ### Red-proofs (run → fail → revert → green) — all confirmed -| RP | Mutation | Failing assertion observed | Reverted | -|----|----------|----------------------------|----------| -| RP-1 | `AppDataDirNames` → `return []string{stackName}` | `TestAppDataDirNames` paperless/two-name/unclean cases FAIL | green | -| RP-2 | RunTier2 L146 → `AppDataDir(nsRoot, stackName)` | `TestRunTier2_PaperlessShape`: appdata leg never mirrored | green | -| RP-3 | restore `liveDir` → stack-name keying | `TestRestoreTier2Files_ResolvedLiveDir`: dst = appdata/app not appdata/paperless | green | -| RP-4 | migCopy site → key by `app` | `TestMigrate_PaperlessShape_ScopeApp`: copy seam never saw appdata/paperless | green | -| RP-5 | delete N>1 guard (`>1` → `>999`) | all three MultiDir refusal tests FAIL (mirror fired, no refusal) | green | +| RP | Mutation | Failing assertion | Reverted | +|----|----------|-------------------|----------| +| RP-1 | disable the class-value check | `TestValidateBackupSpec_Defects/unknown_class` + `/empty_class` accept silently | green | +| RP-2 | `:ro` default overrides explicit | `TestClassify_ImmichShape`: media/photos forced excluded | green | +| RP-3 | flip unlisted-writable default to excluded | `TestClassify_TwoLevelDefault`: unlisted writable ≠ mandatory | green | +| RP-4 | LoadMetadata accepts `Backup` without validating | `TestClassifiedBinds_RejectedFixture`: bad block not degraded to legacy | green | ## Deploy + verify -- Built `v0.131.0` on 180 (`build.sh 0.131.0 --push`, image 145M, digest `sha256:7bba3b54…`). -- Deployed via the bootstrap mechanism to **BOTH guests**: - - **demo 9201** (felhom-pve): `felhom-controller:0.131.0 … Up (healthy)`; logs show a clean status - refresh (9 containers / 56 stacks) + hub report pushed. - - **drill guest 9201** (nested PVE `drill-day0`, 192.168.0.152): `felhom-controller:0.131.0 … Up - (healthy)`; logs clean (scheduler + status refresh). +- Built `v0.132.0` on 180 (`build.sh 0.132.0 --push`, image 145M, digest `sha256:00c6f83e…`). +- Deployed via the bootstrap mechanism to **BOTH guests** — `felhom-controller:0.132.0 … Up (healthy)` + on demo 9201 (felhom-pve) and the drill guest 9201 (nested PVE `drill-day0`, 192.168.0.152). -## NOT live-validated here (deferred to Viktor's supervised session) +## Live sync check (demo guest — the real server-side pipeline) -The functional live legs all require a **deployed paperless-ngx** (the only catalog app with the -appdata mismatch shape). On the current demo, paperless-ngx is **undeployed** — only leftover skeleton -data (`appdata/paperless`, ~16 KB) + a stale recovery unit remain from a prior session; every -currently-deployed HDD app (jellyfin/radarr/navidrome/calibre-web) uses userdata/media binds, not -`appdata/`. Standing up paperless-ngx is a 5-container, Postgres-backed deploy, and the controller's -API (`:8080`) is container-internal (reachable only via traefik + Host header), so a clean unsupervised -deploy→backup→restore→teardown was disproportionate/risky on the shared demo. Deferred, to run through -the real dashboard UI under supervision: +Catalog `21e8df1` pushed AFTER the controller was live, then the controller's `doSync` pipeline +(same code path as the "Sablonok frissítése" button) was triggered by a bootstrap restart: -1. **Tier-2 backup (F-S2 core):** deploy paperless-ngx on felhom-usb, trigger "2. mentés", verify - `…/secondary/paperless-ngx/appdata/` mirrors `appdata/paperless` (count + size). -2. **Marker restore leg:** inject a marker in `appdata/paperless`, re-run tier-2, delete it live, run the - UI file-restore, verify byte-identical return + app healthy. -3. **Storage page:** paperless-ngx shows a non-empty size. -4. **F-S3 migration leg:** scope="app" migration of a real paperless-ngx between drives (supervised — - destructive cleanup step). +- `[sync] Updated /.felhom.yml` for **exactly the 13 apps**, then `[sync] Catalog sync complete`. +- **`backup block rejected` count: 0.** Zero `[ERROR] [stacks]` lines anywhere in the log. Controller + healthy. → **live proof that all 13 committed blocks parse, validate, and match their real composes.** -Confidence for these rests on the exhaustive unit coverage + RP-1..RP-5 above and the confirmed-live -compose shape (paperless-ngx's on-box `docker-compose.yml` binds `${HDD_PATH}/appdata/paperless/media` -and `.../export` — the exact mismatch the fix resolves). +## 13-app cross-check (shipped parser against the real catalog clone — every bind resolved `explicit`) + +| App | Resolved class set (root/relpath = class) | +|-----|-------------------------------------------| +| immich | hdd/appdata/immich=mandatory · userdata/media/photos=optional | +| paperless-ngx | hdd/appdata/paperless/media=mandatory · hdd/appdata/paperless/export=excluded · userdata/import/paperless=excluded | +| nextcloud | hdd/appdata/nextcloud=mandatory | +| calibre-web | userdata/media/books=mandatory · userdata/import/calibre=excluded | +| audiobookshelf | userdata/media/audiobooks=optional · userdata/media/podcasts=excluded | +| komga | userdata/media/comics=optional | +| navidrome | userdata/media/music=excluded | +| radarr | userdata/media/movies=excluded · userdata/downloads=excluded | +| sonarr | userdata/media/tv=excluded · userdata/downloads=excluded | +| emby | userdata/media=excluded | +| jellyfin | userdata/media=excluded | +| plex | userdata/media=excluded | +| romm | userdata/roms=optional · hdd/appdata/romm/resources=excluded | + +Every bind resolves with origin `explicit` (full coverage — no writable bind left to the mandatory +default). Verified via a throwaway `go run` harness over the shipped `LoadMetadata → +ParseComposeClassifiableBinds → ClassifyBinds`, then removed (not committed). + +## PENDING-VETO resolution + +**audiobookshelf `media/audiobooks` = `optional`** (spike SQ2 proposed `excluded`). Committed optional +for consistency with komga/romm (curated, often personally-ripped → precious-decoupled). If Viktor +rules `excluded`, flip that ONE line in `templates/audiobookshelf/.felhom.yml`. ## Observations (noticed, NOT acted on) -- **F-S1 (userdata not backed up at any tier)** remains open — deliberately out of scope; owned by the - classification redesign (Task 2/3), which this task's tier-2 header + comments now point to. -- **Multi-dir (N>1) limitation** is a deliberate tier-2 refusal to be lifted by the tier-policy engine - (Task 3, which owns the flat destination layout). No catalog app hits it today. -- Leftover undeployed paperless/immich/nextcloud/romm skeleton dirs + a stale - `felhom-flash/backups/secondary/paperless-ngx` exist on the demo drives (prior-session residue) — not - touched. +- The classification is INERT by contract — offsite/tier-2/`.fab` still capture per today's legacy + behavior. **Task 3** (tier policy engine) is the next step: consume `Manager.ClassifiedBinds` / + `GetStackClassifiedBinds` to scope offsite/tier-2/`.fab` capture by class (mandatory→offsite, + excluded→never, optional→customer-selectable). The seam is wired + tested for it. +- Recovery units already carry `.felhom.yml` and git-sync already whitelists it, so the block reaches + deployed stacks + units with zero plumbing changes; no recovery-unit SchemaVersion bump (Task 3 owns + any content-semantics bump). +- All 13 catalog apps list every classifiable bind explicitly — the two-level *default* branch + (unlisted writable→mandatory) is exercised only by tests today, never by a real catalog app.