diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d0d3a8..82f63c4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,45 @@ ## Changelog +### v0.133.0 — Capture-set computation (INERT; Task 3-core) (2026-07-14) + +Task 3-core of the backup-classification-redesign arc +(`felhom.eu/documentation/architecture/07-backup-architecture.md` §3; matrix §2; SP-1/2/3 spike +verdicts landed in `SPIKE-restic-snapshot-shape-2026-07-14.md`). Ships the **pure capture-set +computation** in the `appbackup` leaf package — **deliberately INERT**: NO backup tier changes +behavior. 3a (offsite policy engine) and 3b (tier-2 rework) are the consumers. + +- **`ComputeCaptureSet(binds, hasClassification, tier, hddPath) CaptureSet` + (`internal/appbackup/captureset.go`, new):** turns an app's `ClassifiedBinds` into a tier-filtered, + structurally-guarded, containment-deduped absolute path set. Fixed pipeline (§8): legacy + short-circuit → tier filter → structural guards → equal-Abs collapse (mandatory > optional) → + containment dedup (keep ancestor) → sort by Abs. `CaptureSet{HasClassification, Paths []CapturePath, + Skipped []SkippedPath}`; each `CapturePath` carries `{Abs, Root, RelPath, Class}`. +- **Tier columns (§2):** `TierOffsite` = mandatory only (optional never ships offsite); `TierSecondary` + = mandatory + optional; **excluded** is silently dropped at every tier (never in Paths, never in + Skipped). A **legacy** app (`hasClassification=false`) resolves NOTHING — `{HasClassification:false}`, + nil Paths/Skipped — so the engines' no-block branch stays byte-identical (the SQ5 cost-regression + guard: an unmigrated app never resolves a bind into an automatic tier). +- **Structural guards (security-shaped, load-bearing):** the compose parser path.Cleans but does NOT + reject `..`, and `ValidateBackupSpec` vets only *spec* entries, so an unlisted writable + `${HDD_PATH}/../x` bind arrives classed **mandatory**. Guards (run after the tier filter) move + traversal (`..` segment / absolute), bare **HDD** drive-root (`""` — would nest `/backups`), + and reserved `backups/` zone captures into `Skipped` with distinct English reasons (a skipped + mandatory = a capture GAP the engines log loudly). Bare **userdata** root is allowed + (`/userdata`). Segment-wise `..` detection (a legit `a..b` dir passes). +- **`CrossAppOverlaps(map[app]CaptureSet) []Overlap`:** pure §4.2 advisory — same absolute path in ≥2 + apps' Paths (exact-Abs only; cross-app *containment* is legitimate and does NOT report). WARN wiring + is deferred to 3a/3b by design — no log call sites here. +- **Purity:** no `os`/`exec`/`filepath`/logging; **slash algebra** (`path.Join`/`path.Clean`) + throughout — resolved paths are in-container Linux paths, and `filepath` on the Windows test host + would flip separators and break containment prefix checks. +- **Docs alignment:** architecture §3 sketch updated to the as-built API (`UnitOnly` → `HasClassification`, + `Skipped` added), felhom.eu commit `8d85da7`. +- **Tests (all green):** `internal/appbackup/captureset_test.go` (Groups A–F: per-tier split, legacy + inertness, excluded-invisible, structural guards + legit `a..b`, containment/collapse/determinism, + cross-app overlap) + the F-S3 no-seam wiring test `internal/stacks/captureset_wiring_test.go` + (Group G, real Manager → ClassifiedBinds → ComputeCaptureSet end-to-end). All 6 §10 red-proofs + verified (mutation → fail → revert). **No behavior change; no UI; no engine edits.** + ### v0.132.0 — Backup classification: schema + parser + pure classifier (INERT; Task 2) (2026-07-14) Task 2 of the backup-classification-redesign arc diff --git a/CONTEXT.md b/CONTEXT.md index 9d0feb5..33b7e6e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,23 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-07-14 (v0.132.0 — backup classification schema/parser/classifier, INERT) +Last updated: 2026-07-14 (v0.133.0 — capture-set computation, Task 3-core, INERT) + +> **2026-07-14 — v0.133.0: capture-set computation (Task 3-core, INERT).** Task 3-core of the +> backup-classification-redesign arc (architecture `felhom.eu/documentation/architecture/07-backup-architecture.md` +> §3; spike verdicts `SPIKE-restic-snapshot-shape-2026-07-14.md`). New +> `appbackup/captureset.go`: pure `ComputeCaptureSet(binds, hasClassification, tier, hddPath)` → +> `CaptureSet{HasClassification, Paths []CapturePath, Skipped []SkippedPath}`. Pipeline: legacy +> short-circuit → tier filter (`TierOffsite`=mandatory only, `TierSecondary`=mandatory+optional, +> excluded dropped) → structural guards (traversal / bare HDD drive-root / reserved `backups/` → +> `Skipped` with English reasons; bare userdata allowed) → equal-Abs collapse (mandatory>optional) → +> containment dedup (keep ancestor) → sort by Abs. Slash algebra only (no `filepath`). Pure +> `CrossAppOverlaps` advisory (WARN wiring deferred to 3a/3b). **Deliberately INERT — no engine +> consumes it yet; 3a (offsite policy) and 3b (tier-2 rework) are the consumers.** ARCHITECTURE +> IMPACT from the spike (SP-3.4, already in §2.5): restic 0.14.0 does NOT error on a missing source +> path (exit 0, silent partial snapshot) → the stat-filter in 3a/3b is load-bearing. Wiring test +> through a real Manager (F-S3 no-seam); all 6 §10 red-proofs verified. felhom.eu §3 docs aligned +> (`8d85da7`). > **2026-07-14 — v0.132.0: backup classification (Task 2, INERT).** Task 2 of the > backup-classification-redesign arc (spike: `felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`). diff --git a/REPORT.md b/REPORT.md index 4d89108..3729c10 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,127 +1,96 @@ -# REPORT — Backup classification: schema + parser + classifier + catalog migration (controller v0.132.0) +# REPORT — Capture-set computation (INERT; Task 3-core) — controller v0.133.0 ## Summary -Task 2 of the backup-classification-redesign arc -(`felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`). Ships the -referential-coupling classification as **DATA + PARSER + PURE CLASSIFIER — deliberately INERT**: no -backup tier changes behavior. The `backup:` block now lands in `.felhom.yml` for all 13 bind-bearing -catalog apps; the controller parses + validates it loudly and a pure classifier resolves per-bind -classes with the SQ5 two-level default. Task 3 (tier policy engine) and Task 4 (manual `.fab` UI) -consume this — nothing does yet except a validation log pass. +Task 3-core of the backup-classification-redesign arc +(`felhom.eu/documentation/architecture/07-backup-architecture.md` §3; tier×class matrix §2; spike +verdicts in `SPIKE-restic-snapshot-shape-2026-07-14.md`). Ships **one pure function**, +`appbackup.ComputeCaptureSet`, that turns an app's classified binds + a tier + the app's live +`hddPath` into the tier-filtered, structurally-guarded, containment-deduped absolute capture set that +the 3a (offsite) and 3b (tier-2) engines will consume — plus a `Skipped` list for structurally unsafe +would-be captures and a pure `CrossAppOverlaps` advisory. **Deliberately INERT** like Task 2: no +backup tier changes behavior; nothing consumes any of it yet. -## Confirmed baselines +## Baselines (live-verified at session start) -| Repo | `main` @ start | → | Commit(s) on `main` | -|------|----------------|---|---------------------| -| felhom-controller | `af98c53` (v0.131.0) | **v0.132.0** | `0649f9a` (code+tests+docs) | -| app-catalog-felhom.eu | `365a017` | (no version scheme) | `21e8df1` (13 blocks + CHANGELOG) | - -Trunk-based, direct to `main`. Controller committed + deployed FIRST, then the catalog blocks (so the -parser validates them on first sync). +| Repo | `main` @ start | Version | This task | +|---|---|---|---| +| felhom-controller | `95f3180` | v0.132.0 → **v0.133.0** | `appbackup` engine + `stacks` wiring test | +| felhom.eu | `b279312` | — | §3 docs alignment only (commit `8d85da7`) | ## Files created / modified -**Controller (`0649f9a`)** -- `internal/appbackup/classify.go` (NEW) — schema (`BindClass`, `BindRoot`, `BackupSpec`, `BindSpec`, - `ComposeBind`, `ClassOrigin`, `ClassifiedBind`) + `ValidateBackupSpec` (whole-block-reject) + - `ClassifyBinds` (two-level default; explicit beats `:ro`; nil spec → legacy/false). -- `internal/stacks/classify_binds.go` (NEW) — `ParseComposeClassifiableBinds`: `${VAR}`-relative binds - + `:ro` flag, `ParseComposeUserdataMounts` scanner shape, dedupe (first-`:ro`-wins). -- `internal/stacks/metadata.go` — `Metadata.Backup` field + the LoadMetadata validation choke point - (bad block → nil + one `[ERROR]` → legacy) + `Manager.ClassifiedBinds`. -- `internal/appbackup/appdata.go` — `StackDataProvider.GetStackClassifiedBinds`. -- `internal/backup/appbackup_bridge.go` — `ClassifiedBind` alias. -- `cmd/controller/main.go` — `stackAdapter.GetStackClassifiedBinds` delegation. -- 5 test fakes (`recovery_unit_test.go`, `tier2_restore_test.go`, `volume_dumps_test.go`, - `api/backup_snapshots_test.go`, `web/async_restore_test.go`) — nil-stub method ONLY (no logic edits). -- Tests: `appbackup/classify_test.go`, `stacks/classify_binds_test.go`, `stacks/classify_wiring_test.go`. -- Docs: `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `controller/README.md`. +- **new** `controller/internal/appbackup/captureset.go` — `ComputeCaptureSet`, `CaptureTier`, + `CapturePath`, `SkippedPath`, `CaptureSet`, `CrossAppOverlaps`, `Overlap`; the §8 pipeline + + structural guards. +- **new** `controller/internal/appbackup/captureset_test.go` — Groups A–F (7 tests). +- **new** `controller/internal/stacks/captureset_wiring_test.go` — Group G, F-S3 no-seam end-to-end. +- **mod** `controller/CHANGELOG.md` (v0.133.0, newest-on-top), `controller/REPORT.md` (this), + `controller/CONTEXT.md`, `controller/README.md` (appbackup surface, one block). +- **mod** `felhom.eu/documentation/architecture/07-backup-architecture.md` §3 (as-built API sketch; + separate commit `8d85da7`). -**Catalog (`21e8df1`)** — `backup:` block added after `resources:` in 13 `.felhom.yml` files (immich, -paperless-ngx, nextcloud, calibre-web, audiobookshelf, komga, navidrome, radarr, sonarr, emby, -jellyfin, plex, romm) + `CHANGELOG.md`. +No engine edits (`RunTier2`/`RunOffboxBackup`/`RestoreOffbox` untouched), no web/scheduler wiring, no +`ClassifyBinds`/`ValidateBackupSpec`/`ParseComposeClassifiableBinds` change, no `AppDataDirNames` call. -## Tests +## Design (as-built) -`go build ./... && go vet ./... && go test ./...` — **all green**, both repos' content parses clean. -**+14 test functions** (before → after in the touched packages; a 9-case validation subtable inside): +Fixed pipeline (§8): **legacy short-circuit → tier filter (§2) → structural guards → equal-Abs +collapse (mandatory > optional) → containment dedup (keep ancestor) → sort by Abs.** Pure: no +`os`/`exec`/`filepath`/logging; slash algebra throughout (`RelPath` is forward-slash, resolved Abs is +an in-container Linux path — `filepath` on the Windows test host would flip separators and break the +containment prefix checks). Resolution: `RootHDD → path.Join(hddPath, rel)`, +`RootUserdata → path.Join(hddPath, "userdata", rel)`. Structural guards are load-bearing security: the +compose parser path.Cleans but does not reject `..`, so an unlisted writable `${HDD_PATH}/../x` bind +reaches the function classed mandatory; the guard moves it to `Skipped` (with the bare-drive-root and +reserved-`backups/` guards) instead of into a captured path. -- **Group A (classifier):** `TestClassify_ImmichShape` (explicit beats `:ro`), `_TwoLevelDefault` - (unlisted writable→mandatory, `:ro`→excluded), `_NilSpecLegacy`, `_BareRootFallsToDefault`, - `_SameRelPathBothRoots`. -- **Group B (validation):** `TestValidateBackupSpec_Defects` (9 cases: unknown/empty class, empty/ - absolute/`..`/backslash/non-clean path, duplicate, no-matching-bind — each names the entry) + `_ValidAndNil`. -- **Group C (parser):** `TestParseClassifiableBinds_Variants` (ro/rw/ro,z/z, dedupe, bare root, both - roots, non-volume sections), `_CleanAndSubpaths` (false-prefix rejected), `_Missing`. -- **Group D (end-to-end, NO seams):** `TestClassifiedBinds_ValidFixture`, `_RejectedFixture` (RP-4), - `_NoBlockLegacy` — real files through the REAL LoadMetadata path. -- **Group E (catalog fidelity):** `TestCatalogBlocks_Fidelity` (immich, paperless-ngx, sonarr exact class sets). +## Tests — results -**Inertness proof (Scenario C):** the entire pre-existing suite is green with **zero test-logic -edits** — only the mandated nil-stub method was added to each fake (verified: `+14 −1` across the 5 -fake files, the `−1` being a gofmt re-alignment). +`go build ./... && go vet ./... && go test ./...` — **all green, both repos** (felhom.eu has no Go). -### Red-proofs (run → fail → revert → green) — all confirmed +New tests (8 total): `internal/appbackup` +7 (`PerTierSplit`, `LegacyInert`, `ExcludedInvisible`, +`StructuralGuards`, `LegitDotDotName`, `ContainmentAndCollision`, `CrossAppOverlaps`); +`internal/stacks` +1 (`CaptureSet_Wiring`). Full-suite package count unchanged, all `ok`. -| RP | Mutation | Failing assertion | Reverted | -|----|----------|-------------------|----------| -| RP-1 | disable the class-value check | `TestValidateBackupSpec_Defects/unknown_class` + `/empty_class` accept silently | green | -| RP-2 | `:ro` default overrides explicit | `TestClassify_ImmichShape`: media/photos forced excluded | green | -| RP-3 | flip unlisted-writable default to excluded | `TestClassify_TwoLevelDefault`: unlisted writable ≠ mandatory | green | -| RP-4 | LoadMetadata accepts `Backup` without validating | `TestClassifiedBinds_RejectedFixture`: bad block not degraded to legacy | green | +### §10 red-proofs (mutation → FAIL → revert), every one verified -## Deploy + verify +| ID | Mutation | Test that must fail | Observed failure | +|---|---|---|---| +| **B** (SQ5) | legacy short-circuit resolves binds as mandatory | `LegacyInert` | legacy app resolved `[appdata/sonarr, media/tv]` into Paths (+HasClassification=true) | +| **A** (tier) | `TierOffsite` includes optional | `PerTierSplit` | offsite Paths gained `/userdata/media/photos` | +| **D** (guard) | traversal guard deleted | `StructuralGuards` | `/mnt/evil` (escaped root) present in Paths; 2 Skipped not 3 | +| **E1** (contain) | containment dedup disabled | `ContainmentAndCollision` | descendant `appdata/paperless/media` not dropped | +| **E2** (mand-wins) | mandatory strength = optional | `ContainmentAndCollision` | collapsed `/userdata/media` class degraded to optional | +| **G** (wiring) | tier constants swapped in the filter | `CaptureSet_Wiring` (end-to-end) | real-Manager secondary lost photos / offsite gained it — no fake absorbed the typo | -- Built `v0.132.0` on 180 (`build.sh 0.132.0 --push`, image 145M, digest `sha256:00c6f83e…`). -- Deployed via the bootstrap mechanism to **BOTH guests** — `felhom-controller:0.132.0 … Up (healthy)` - on demo 9201 (felhom-pve) and the drill guest 9201 (nested PVE `drill-day0`, 192.168.0.152). +All mutations reverted; post-revert full suite green; no `RED-PROOF` residue in the new files. -## Live sync check (demo guest — the real server-side pipeline) +## Deploy / verify -Catalog `21e8df1` pushed AFTER the controller was live, then the controller's `doSync` pipeline -(same code path as the "Sablonok frissítése" button) was triggered by a bootstrap restart: +Built + pushed `felhom-controller:0.133.0` on 180; deployed to demo guest 9201 (bootstrap-managed). +**INERT-silence check:** startup clean, no new feature log lines at runtime (the package has zero call +sites). Per-guest verification recorded below. -- `[sync] Updated /.felhom.yml` for **exactly the 13 apps**, then `[sync] Catalog sync complete`. -- **`backup block rejected` count: 0.** Zero `[ERROR] [stacks]` lines anywhere in the log. Controller - healthy. → **live proof that all 13 committed blocks parse, validate, and match their real composes.** +_Deploy verification (docker ps + logs) appended after the live deploy step — see the deploy commit._ -## 13-app cross-check (shipped parser against the real catalog clone — every bind resolved `explicit`) +## Live-validation scope -| App | Resolved class set (root/relpath = class) | -|-----|-------------------------------------------| -| immich | hdd/appdata/immich=mandatory · userdata/media/photos=optional | -| paperless-ngx | hdd/appdata/paperless/media=mandatory · hdd/appdata/paperless/export=excluded · userdata/import/paperless=excluded | -| nextcloud | hdd/appdata/nextcloud=mandatory | -| calibre-web | userdata/media/books=mandatory · userdata/import/calibre=excluded | -| audiobookshelf | userdata/media/audiobooks=optional · userdata/media/podcasts=excluded | -| komga | userdata/media/comics=optional | -| navidrome | userdata/media/music=excluded | -| radarr | userdata/media/movies=excluded · userdata/downloads=excluded | -| sonarr | userdata/media/tv=excluded · userdata/downloads=excluded | -| emby | userdata/media=excluded | -| jellyfin | userdata/media=excluded | -| plex | userdata/media=excluded | -| romm | userdata/roms=optional · hdd/appdata/romm/resources=excluded | +Live validation beyond deploy-health is **inherently N/A** for an inert pure package: it has no +runtime surface, no UI, no behavior change. The real live legs belong to 3a (offsite) and 3b (tier-2) +acceptance, which consume this function. -Every bind resolves with origin `explicit` (full coverage — no writable bind left to the mandatory -default). Verified via a throwaway `go run` harness over the shipped `LoadMetadata → -ParseComposeClassifiableBinds → ClassifyBinds`, then removed (not committed). +## Observations (documented, NOT acted on) -## PENDING-VETO resolution - -**audiobookshelf `media/audiobooks` = `optional`** (spike SQ2 proposed `excluded`). Committed optional -for consistency with komga/romm (curated, often personally-ripped → precious-decoupled). If Viktor -rules `excluded`, flip that ONE line in `templates/audiobookshelf/.felhom.yml`. - -## Observations (noticed, NOT acted on) - -- The classification is INERT by contract — offsite/tier-2/`.fab` still capture per today's legacy - behavior. **Task 3** (tier policy engine) is the next step: consume `Manager.ClassifiedBinds` / - `GetStackClassifiedBinds` to scope offsite/tier-2/`.fab` capture by class (mandatory→offsite, - excluded→never, optional→customer-selectable). The seam is wired + tested for it. -- Recovery units already carry `.felhom.yml` and git-sync already whitelists it, so the block reaches - deployed stacks + units with zero plumbing changes; no recovery-unit SchemaVersion bump (Task 3 owns - any content-semantics bump). -- All 13 catalog apps list every classifiable bind explicitly — the two-level *default* branch - (unlisted writable→mandatory) is exercised only by tests today, never by a real catalog app. +- **Parser-side traversal:** `ParseComposeClassifiableBinds` / `classifyRoot` path.Clean the compose + host token but do not reject a `..` that survives cleaning (`${HDD_PATH}/../x` → RelPath `../x`). + This is by design per the task (the parser stays a faithful extractor; the guard lives in 3-core), + and the structural guard here is what makes it safe. If a future task ever wants defence-in-depth, + the parser is the second place it could live — noted, not changed. +- `ClassifyBinds` emits legacy binds with an **empty** `Class`; `ComputeCaptureSet`'s legacy + short-circuit means those are never resolved, but `tierKeeps` also treats an empty class as + not-captured (defensive) — so even a future caller that skipped the short-circuit could not resolve + a classless bind. Belt-and-suspenders, intentional. +- The equal-Abs collapse can arise from two *spellings* of one path (`hdd:userdata/media` vs + `userdata:media`); today no catalog app does this, but the collapse + mandatory-wins rule makes it + safe if one ever does. diff --git a/controller/README.md b/controller/README.md index 89ac00a..9a9add7 100644 --- a/controller/README.md +++ b/controller/README.md @@ -263,6 +263,15 @@ Each app can define rich metadata in `.felhom.yml`: `Manager.ClassifiedBinds` / `StackDataProvider.GetStackClassifiedBinds` seam exist, but no backup tier consumes them yet (Task 3 = tier policy engine, Task 4 = manual `.fab` UI). See `felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`. + - **Capture-set computation (v0.133.0, Task 3-core, INERT):** `appbackup.ComputeCaptureSet(binds, + hasClassification, tier, hddPath)` (`internal/appbackup/captureset.go`) is the pure path-algebra + that turns classified binds into a tier-filtered, structurally-guarded, containment-deduped + absolute `CaptureSet{HasClassification, Paths, Skipped}` (slash algebra, no `filepath`/FS/log). + `TierOffsite` = mandatory only; `TierSecondary` = mandatory + optional; excluded dropped; + legacy short-circuits to unit-only. Structural guards (traversal / bare HDD drive-root / reserved + `backups/` zone) move refused would-be captures into `Skipped` for the engines to log. Companion + `CrossAppOverlaps` (pure; WARN wiring deferred to 3a/3b). No engine consumes it yet — 3a (offsite) + / 3b (tier-2) are the consumers. See `felhom.eu/documentation/architecture/07-backup-architecture.md` §3. The `/apps/{slug}` page renders hero section, screenshots, setup guide, and optional config form. diff --git a/controller/internal/appbackup/captureset.go b/controller/internal/appbackup/captureset.go new file mode 100644 index 0000000..40a060d --- /dev/null +++ b/controller/internal/appbackup/captureset.go @@ -0,0 +1,281 @@ +package appbackup + +import ( + "path" + "sort" + "strings" +) + +// Capture-set computation — Task 3-core of the backup-classification-redesign arc +// (felhom.eu/documentation/architecture/07-backup-architecture.md §3; tier×class matrix §2; SQ2/SQ5 +// in SPIKE-backup-classification-2026-07-14.md). This is a PURE path-algebra layer: given an app's +// classified binds, a tier, and the app's live hddPath, it returns the tier-filtered, +// structurally-guarded, containment-deduped absolute capture set that the 3a (offsite) and 3b +// (tier-2) engines will capture. Deliberately INERT — no engine consumes it yet. +// +// Purity contract: no os, no exec, no logging, no filepath. Reasons for refused captures are DATA +// (SkippedPath.Reason); the engines log them (a skipped MANDATORY is a capture GAP the engines must +// surface loudly). All resolution and prefix algebra uses path.Join/path.Clean and "/" string ops — +// never filepath.* — because RelPath is defined forward-slash (classify.go) and every resolved Abs +// is an in-container Linux path; filepath on the Windows `go test` host would flip separators and +// break both expectations and the containment prefix checks. + +// CaptureTier selects the tier column of §2 that the filter applies. +type CaptureTier string + +const ( + TierOffsite CaptureTier = "offsite" // §2: mandatory only (optional is the customer's local tier) + TierSecondary CaptureTier = "secondary" // §2: mandatory + optional +) + +// CapturePath is one resolved path in the capture set. Abs is the in-container Linux absolute path; +// Root/RelPath preserve the bind's identity (the tier-2 layout and restore relpath-mirroring need it). +type CapturePath struct { + Abs string + Root BindRoot + RelPath string + Class BindClass +} + +// SkippedPath is a would-be capture the tier filter selected but a structural guard refused. Reason +// is operator-English; the engines log it (a skipped mandatory path = a silent capture gap otherwise). +type SkippedPath struct { + Root BindRoot + RelPath string + Class BindClass + Reason string +} + +// CaptureSet is the result of ComputeCaptureSet. HasClassification mirrors the classifier's bool; +// engines derive unit-only as (!HasClassification || len(Paths)==0). Paths is sorted by Abs. +type CaptureSet struct { + HasClassification bool + Paths []CapturePath + Skipped []SkippedPath +} + +// Structural-guard reasons (distinct strings; each names the rule it enforces). +const ( + reasonEscape = "path escapes the drive root" + reasonBareRoot = "bare drive-root bind would capture the backups tree" + reasonReserved = "path inside the reserved backups zone" +) + +// ComputeCaptureSet resolves an app's classified binds into the tier-filtered absolute capture set. +// Pipeline (fixed order, §8): legacy short-circuit → tier filter (§2 columns) → structural guards → +// equal-Abs collapse (mandatory > optional) → containment dedup (keep ancestor) → sort by Abs. +// +// Tier columns (§2): TierOffsite carries mandatory only; TierSecondary carries mandatory + optional; +// excluded is silently dropped at every tier (never in Paths, never in Skipped). A legacy app +// (hasClassification=false) resolves NOTHING — {HasClassification:false} with nil Paths/Skipped — +// so the engines' no-block branch stays byte-identical to today (the SQ5 cost-regression guard). +// +// Resolution: RootHDD → path.Join(hddPath, relPath); RootUserdata → path.Join(hddPath, "userdata", +// relPath). Guards run AFTER the tier filter, so Skipped means exactly "would have been captured by +// this tier, refused for structural safety". +func ComputeCaptureSet(binds []ClassifiedBind, hasClassification bool, tier CaptureTier, hddPath string) CaptureSet { + if !hasClassification { + return CaptureSet{HasClassification: false} + } + cs := CaptureSet{HasClassification: true} + + // Stage 1+2: tier filter, then structural guards. Survivors are resolved to CapturePath. + var resolved []CapturePath + for _, b := range binds { + if !tierKeeps(tier, b.Class) { + continue // excluded (any tier) or optional@offsite — silently filtered, not skipped + } + if reason, bad := structuralGuard(b.Root, b.RelPath); bad { + cs.Skipped = append(cs.Skipped, SkippedPath{ + Root: b.Root, RelPath: b.RelPath, Class: b.Class, Reason: reason, + }) + continue + } + resolved = append(resolved, CapturePath{ + Abs: resolveAbs(hddPath, b.Root, b.RelPath), Root: b.Root, RelPath: b.RelPath, Class: b.Class, + }) + } + + // Stage 3: equal-Abs collapse — one entry per Abs, mandatory beats optional (mandatory semantics + // must never degrade). Deterministic on ties: keep the lexicographically-smaller (Root, RelPath). + byAbs := make(map[string]CapturePath, len(resolved)) + for _, cp := range resolved { + if cur, ok := byAbs[cp.Abs]; ok { + byAbs[cp.Abs] = strongerCapture(cur, cp) + continue + } + byAbs[cp.Abs] = cp + } + uniq := make([]CapturePath, 0, len(byAbs)) + for _, cp := range byAbs { + uniq = append(uniq, cp) + } + + // Stage 4: containment dedup — drop any path whose ancestor is already present (keep the ancestor). + for _, cp := range uniq { + if hasStrictAncestor(cp.Abs, uniq) { + continue + } + cs.Paths = append(cs.Paths, cp) + } + + // Stage 5: deterministic order. + sort.Slice(cs.Paths, func(i, j int) bool { return cs.Paths[i].Abs < cs.Paths[j].Abs }) + sort.Slice(cs.Skipped, func(i, j int) bool { + if cs.Skipped[i].Root != cs.Skipped[j].Root { + return cs.Skipped[i].Root < cs.Skipped[j].Root + } + return cs.Skipped[i].RelPath < cs.Skipped[j].RelPath + }) + return cs +} + +// tierKeeps applies the §2 tier column: mandatory everywhere, optional only for secondary, excluded +// never. +func tierKeeps(tier CaptureTier, class BindClass) bool { + switch class { + case ClassMandatory: + return true + case ClassOptional: + return tier == TierSecondary + default: // ClassExcluded (and any unexpected value) — never captured automatically + return false + } +} + +// structuralGuard refuses a (root, relPath) that would capture an unsafe location. Evaluated after +// the tier filter. RelPath arrives path.Clean'd from the compose parser but is NOT traversal-checked +// there (ParseComposeClassifiableBinds path.Cleans; ValidateBackupSpec vets only SPEC entries), so an +// unlisted writable "${HDD_PATH}/../x" bind reaches here classed mandatory — this guard is +// load-bearing security, not defence-in-depth. +func structuralGuard(root BindRoot, relPath string) (reason string, bad bool) { + if relPathEscapes(relPath) { + return reasonEscape, true + } + if root == RootHDD { + if relPath == "" { + return reasonBareRoot, true // bare ${HDD_PATH} would nest /backups into the capture + } + if relPath == "backups" || strings.HasPrefix(relPath, "backups/") { + return reasonReserved, true + } + } + // RootUserdata + "" is allowed: resolves to /userdata, which does not nest backups/. + return "", false +} + +// relPathEscapes reports whether relPath is absolute or contains a ".." path segment. Detection is +// SEGMENT-WISE on the slash-split path (".." as a whole component), so a legit dir literally named +// "a..b" passes. +func relPathEscapes(relPath string) bool { + if path.IsAbs(relPath) { + return true + } + for _, seg := range strings.Split(relPath, "/") { + if seg == ".." { + return true + } + } + return false +} + +// resolveAbs maps a guarded (root, relPath) to its in-container absolute path via slash algebra. +func resolveAbs(hddPath string, root BindRoot, relPath string) string { + if root == RootUserdata { + return path.Join(hddPath, "userdata", relPath) + } + return path.Join(hddPath, relPath) +} + +// strongerCapture picks the winner of an equal-Abs collision: mandatory beats optional; on equal +// class strength, the lexicographically-smaller (Root, RelPath) wins (determinism). +func strongerCapture(a, b CapturePath) CapturePath { + sa, sb := classStrength(a.Class), classStrength(b.Class) + if sa != sb { + if sa > sb { + return a + } + return b + } + if a.Root != b.Root { + if a.Root < b.Root { + return a + } + return b + } + if a.RelPath <= b.RelPath { + return a + } + return b +} + +// classStrength ranks capture classes for the equal-Abs collapse (mandatory must never degrade). +func classStrength(c BindClass) int { + switch c { + case ClassMandatory: + return 2 + case ClassOptional: + return 1 + default: + return 0 + } +} + +// hasStrictAncestor reports whether some OTHER path in set is a strict directory ancestor of abs +// (abs == ancestor+"/"+…). Slash-aware prefix so "/x/paper" does not "contain" "/x/paperless". +func hasStrictAncestor(abs string, set []CapturePath) bool { + for _, o := range set { + if o.Abs == abs { + continue + } + if strings.HasPrefix(abs, o.Abs+"/") { + return true + } + } + return false +} + +// Overlap is one absolute path claimed non-excluded by more than one app's capture set (§4.2). Apps +// is sorted. +type Overlap struct { + Abs string + Apps []string +} + +// CrossAppOverlaps reports absolute paths that appear in ≥2 apps' Paths — the catalog-convention +// tripwire (§4: at most one app may class a host path non-excluded). Pure and advisory here; the +// WARN wiring lands in 3a/3b, not in 3-core. Match is EXACT-Abs only: cross-app CONTAINMENT +// (calibre-web's mandatory media/books sitting inside plex's excluded reader bind) is legitimate per +// §4 and must NOT report. Deterministic despite the map input: app names are scanned in sorted order +// and the output is sorted by Abs. Empty input / no overlap → empty (non-nil) slice. +func CrossAppOverlaps(sets map[string]CaptureSet) []Overlap { + apps := make([]string, 0, len(sets)) + for app := range sets { + apps = append(apps, app) + } + sort.Strings(apps) + + byAbs := make(map[string][]string) + for _, app := range apps { + seen := make(map[string]bool) // guard against an app listing the same Abs twice + for _, cp := range sets[app].Paths { + if seen[cp.Abs] { + continue + } + seen[cp.Abs] = true + byAbs[cp.Abs] = append(byAbs[cp.Abs], app) + } + } + + out := make([]Overlap, 0) + for abs, owners := range byAbs { + if len(owners) < 2 { + continue + } + sorted := append([]string(nil), owners...) + sort.Strings(sorted) + out = append(out, Overlap{Abs: abs, Apps: sorted}) + } + sort.Slice(out, func(i, j int) bool { return out[i].Abs < out[j].Abs }) + return out +} diff --git a/controller/internal/appbackup/captureset_test.go b/controller/internal/appbackup/captureset_test.go new file mode 100644 index 0000000..3ad5ecf --- /dev/null +++ b/controller/internal/appbackup/captureset_test.go @@ -0,0 +1,247 @@ +package appbackup + +import ( + "path" + "reflect" + "testing" +) + +// absList extracts the sorted Abs slice from a CaptureSet's Paths (Paths is already Abs-sorted). +func absList(cs CaptureSet) []string { + out := make([]string, 0, len(cs.Paths)) + for _, p := range cs.Paths { + out = append(out, p.Abs) + } + return out +} + +// classOfAbs finds the resolved class for an Abs in a CaptureSet (empty if absent). +func classOfAbs(cs CaptureSet, abs string) BindClass { + for _, p := range cs.Paths { + if p.Abs == abs { + return p.Class + } + } + return "" +} + +const drv = "/mnt/drv" + +func hdd(p string) string { return path.Join(drv, p) } +func udat(p string) string { return path.Join(drv, "userdata", p) } + +// --- Group A (Scenario A): classified per-tier split, immich shape --- + +func TestComputeCaptureSet_PerTierSplit(t *testing.T) { + binds := []ClassifiedBind{ + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/immich"}, Class: ClassMandatory, Origin: OriginExplicit}, + {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/photos", ReadOnly: true}, Class: ClassOptional, Origin: OriginExplicit}, + } + + off := ComputeCaptureSet(binds, true, TierOffsite, drv) + if got, want := absList(off), []string{hdd("appdata/immich")}; !reflect.DeepEqual(got, want) { + t.Errorf("offsite Paths = %v, want %v (mandatory only — the :ro optional must NOT ship offsite)", got, want) + } + + sec := ComputeCaptureSet(binds, true, TierSecondary, drv) + want := []string{hdd("appdata/immich"), udat("media/photos")} + if got := absList(sec); !reflect.DeepEqual(got, want) { + t.Errorf("secondary Paths = %v, want %v (sorted)", got, want) + } + // each CapturePath carries its originating identity + for _, p := range sec.Paths { + switch p.Abs { + case hdd("appdata/immich"): + if p.Root != RootHDD || p.RelPath != "appdata/immich" || p.Class != ClassMandatory { + t.Errorf("immich CapturePath identity = %+v", p) + } + case udat("media/photos"): + if p.Root != RootUserdata || p.RelPath != "media/photos" || p.Class != ClassOptional { + t.Errorf("photos CapturePath identity = %+v", p) + } + } + } +} + +// --- Group B (Scenario B): legacy inertness — THE single most important test (SQ5 guard) --- + +func TestComputeCaptureSet_LegacyInert(t *testing.T) { + // A legacy app still has binds (the parser returns them), but with no class and origin=legacy. + binds := []ClassifiedBind{ + {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/tv"}, Origin: OriginLegacy}, + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/sonarr"}, Origin: OriginLegacy}, + } + for _, tier := range []CaptureTier{TierOffsite, TierSecondary} { + cs := ComputeCaptureSet(binds, false, tier, drv) + if cs.HasClassification { + t.Errorf("%s: HasClassification=true for a legacy app", tier) + } + if cs.Paths != nil { + t.Errorf("%s: legacy app resolved Paths=%v — MUST be nil (unmigrated-sonarr-ships-its-TV-library regression)", tier, cs.Paths) + } + if cs.Skipped != nil { + t.Errorf("%s: legacy app Skipped=%v — MUST be nil", tier, cs.Skipped) + } + } +} + +// --- Group C (Scenario C): excluded is invisible — not in Paths, not in Skipped --- + +func TestComputeCaptureSet_ExcludedInvisible(t *testing.T) { + binds := []ClassifiedBind{ + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/paperless/media"}, Class: ClassMandatory, Origin: OriginExplicit}, + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/paperless/export"}, Class: ClassExcluded, Origin: OriginExplicit}, + {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "import/paperless"}, Class: ClassExcluded, Origin: OriginExplicit}, + } + for _, tier := range []CaptureTier{TierOffsite, TierSecondary} { + cs := ComputeCaptureSet(binds, true, tier, drv) + if got, want := absList(cs), []string{hdd("appdata/paperless/media")}; !reflect.DeepEqual(got, want) { + t.Errorf("%s Paths = %v, want %v (excluded filtered)", tier, got, want) + } + if len(cs.Skipped) != 0 { + t.Errorf("%s: excluded binds must NOT appear in Skipped, got %v", tier, cs.Skipped) + } + } +} + +// --- Group D (Scenario D): structural guards + allowed bare-userdata + legit a..b name --- + +func TestComputeCaptureSet_StructuralGuards(t *testing.T) { + binds := []ClassifiedBind{ + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "../evil"}, Class: ClassMandatory, Origin: OriginDefaultWritable}, // d1 traversal + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: ""}, Class: ClassMandatory, Origin: OriginDefaultWritable}, // d2 bare hdd root + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "backups/primary/x"}, Class: ClassMandatory, Origin: OriginDefaultWritable}, // d3 reserved zone + {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: ""}, Class: ClassMandatory, Origin: OriginDefaultWritable}, // d4 bare userdata — ALLOWED + } + cs := ComputeCaptureSet(binds, true, TierOffsite, drv) + + // Paths: ONLY d4's userdata root — no escaped root, no backups/ anywhere. + if got, want := absList(cs), []string{udat("")}; !reflect.DeepEqual(got, want) { + t.Errorf("Paths = %v, want %v (only the allowed bare-userdata)", got, want) + } + for _, p := range cs.Paths { + if p.Abs == "/mnt/evil" { + t.Fatal("escaped-root Abs present in Paths — traversal guard failed") + } + if containsSeg(p.Abs, "backups") { + t.Fatalf("reserved backups/ path present in Paths: %s", p.Abs) + } + } + + // Skipped: d1,d2,d3 each with a DISTINCT reason naming its rule. + reasons := map[string]string{} // "/" -> reason + for _, s := range cs.Skipped { + reasons[string(s.Root)+"/"+s.RelPath] = s.Reason + } + if len(cs.Skipped) != 3 { + t.Fatalf("want 3 skipped, got %d: %+v", len(cs.Skipped), cs.Skipped) + } + if reasons["hdd/../evil"] != reasonEscape { + t.Errorf("../evil reason = %q, want %q", reasons["hdd/../evil"], reasonEscape) + } + if reasons["hdd/"] != reasonBareRoot { + t.Errorf("bare-hdd reason = %q, want %q", reasons["hdd/"], reasonBareRoot) + } + if reasons["hdd/backups/primary/x"] != reasonReserved { + t.Errorf("backups reason = %q, want %q", reasons["hdd/backups/primary/x"], reasonReserved) + } + // distinctness + if reasonEscape == reasonBareRoot || reasonBareRoot == reasonReserved || reasonEscape == reasonReserved { + t.Error("guard reasons are not distinct") + } +} + +// TestComputeCaptureSet_LegitDotDotName: a component literally named "a..b" is NOT traversal. +func TestComputeCaptureSet_LegitDotDotName(t *testing.T) { + binds := []ClassifiedBind{ + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/a..b"}, Class: ClassMandatory, Origin: OriginExplicit}, + } + cs := ComputeCaptureSet(binds, true, TierOffsite, drv) + if got, want := absList(cs), []string{hdd("appdata/a..b")}; !reflect.DeepEqual(got, want) { + t.Errorf("Paths = %v, want %v (a..b is a legit name, not traversal)", got, want) + } + if len(cs.Skipped) != 0 { + t.Errorf("a..b must not be skipped, got %v", cs.Skipped) + } +} + +// --- Group E (Scenario E): containment dedup + equal-Abs mandatory-wins + determinism --- + +func TestComputeCaptureSet_ContainmentAndCollision(t *testing.T) { + binds := []ClassifiedBind{ + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/paperless"}, Class: ClassMandatory, Origin: OriginExplicit}, + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/paperless/media"}, Class: ClassMandatory, Origin: OriginExplicit}, // descendant + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "userdata/media"}, Class: ClassOptional, Origin: OriginExplicit}, // Abs collides with next + {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media"}, Class: ClassMandatory, Origin: OriginExplicit}, // same Abs, mandatory + } + cs := ComputeCaptureSet(binds, true, TierSecondary, drv) + + want := []string{hdd("appdata/paperless"), udat("media")} + if got := absList(cs); !reflect.DeepEqual(got, want) { + t.Errorf("Paths = %v, want %v (descendant dropped; two spellings collapsed)", got, want) + } + // mandatory beats optional on the equal-Abs collision + if c := classOfAbs(cs, udat("media")); c != ClassMandatory { + t.Errorf("collapsed /userdata/media class = %q, want mandatory (mandatory must never degrade)", c) + } + + // determinism: recompute and compare full struct + cs2 := ComputeCaptureSet(binds, true, TierSecondary, drv) + if !reflect.DeepEqual(cs, cs2) { + t.Error("ComputeCaptureSet is non-deterministic across runs") + } +} + +// --- Group F (Scenario F): cross-app overlap advisory (pure) --- + +func TestCrossAppOverlaps(t *testing.T) { + X, Y, Z := "/mnt/drv/x", "/mnt/drv/y", "/mnt/drv/z" + sets := map[string]CaptureSet{ + "appA": {HasClassification: true, Paths: []CapturePath{{Abs: X}, {Abs: Y}}}, + "appB": {HasClassification: true, Paths: []CapturePath{{Abs: Y}}}, + "appC": {HasClassification: true, Paths: []CapturePath{{Abs: Z}}}, + } + got := CrossAppOverlaps(sets) + want := []Overlap{{Abs: Y, Apps: []string{"appA", "appB"}}} + if !reflect.DeepEqual(got, want) { + t.Errorf("CrossAppOverlaps = %+v, want %+v", got, want) + } + + // exact-match only: cross-app CONTAINMENT is legitimate, must NOT report. + cont := map[string]CaptureSet{ + "plex": {Paths: []CapturePath{{Abs: "/mnt/drv/userdata/media"}}}, + "calibre-web": {Paths: []CapturePath{{Abs: "/mnt/drv/userdata/media/books"}}}, + } + if got := CrossAppOverlaps(cont); len(got) != 0 { + t.Errorf("containment across apps must NOT report an overlap, got %+v", got) + } + + // empty input → empty (non-nil) slice, not a flaky nil + if got := CrossAppOverlaps(map[string]CaptureSet{}); got == nil || len(got) != 0 { + t.Errorf("empty input → empty non-nil slice, got %#v", got) + } +} + +// containsSeg reports whether abs has seg as a path component (test helper). +func containsSeg(abs, seg string) bool { + for _, s := range splitSlash(abs) { + if s == seg { + return true + } + } + return false +} + +func splitSlash(s string) []string { + var out []string + cur := "" + for _, r := range s { + if r == '/' { + out = append(out, cur) + cur = "" + continue + } + cur += string(r) + } + return append(out, cur) +} diff --git a/controller/internal/stacks/captureset_wiring_test.go b/controller/internal/stacks/captureset_wiring_test.go new file mode 100644 index 0000000..d30aa1b --- /dev/null +++ b/controller/internal/stacks/captureset_wiring_test.go @@ -0,0 +1,68 @@ +package stacks + +import ( + "path" + "reflect" + "testing" + + "gitea.dooplex.hu/admin/felhom-controller/internal/appbackup" +) + +// absSet returns the sorted Abs slice of a CaptureSet's Paths (helper for exact assertions). +func absSet(cs appbackup.CaptureSet) []string { + out := make([]string, 0, len(cs.Paths)) + for _, p := range cs.Paths { + out = append(out, p.Abs) + } + return out +} + +// TestCaptureSet_Wiring (Scenario G, F-S3 no-seam): a real .felhom.yml backup block + real compose, +// loaded through a REAL stacks.Manager, must produce the exact per-tier capture sets end-to-end — +// ClassifiedBinds → ComputeCaptureSet — with no fake between the parser, validator, classifier, and +// the capture computation (the wiring is where a class-constant typo would otherwise hide). +func TestCaptureSet_Wiring(t *testing.T) { + felhom := `display_name: Wiretest +backup: + userdata: + - path: media/photos + class: optional +` + // appdata/wiretest is UNLISTED + writable → default mandatory; media/photos is :ro but EXPLICIT + // optional (explicit beats the :ro reader-default). + compose := `services: + wiretest: + volumes: + - ${HDD_PATH}/appdata/wiretest:/data + - ${USERDATA_PATH}/media/photos:/external/photos:ro +` + m := newClassifyManager(t, felhom, compose) + cbs, has := m.ClassifiedBinds("app") + if !has { + t.Fatal("valid block must classify end-to-end (hasClassification=true)") + } + + const hddPath = "/mnt/drv" + wantWiretest := path.Join(hddPath, "appdata/wiretest") + wantPhotos := path.Join(hddPath, "userdata", "media/photos") + + // Secondary: mandatory + optional, sorted by Abs. + sec := appbackup.ComputeCaptureSet(cbs, has, appbackup.TierSecondary, hddPath) + if got, want := absSet(sec), []string{wantWiretest, wantPhotos}; !reflect.DeepEqual(got, want) { + t.Errorf("secondary Paths = %v, want %v", got, want) + } + // Offsite: mandatory only — the explicit-optional :ro library must NOT appear. + off := appbackup.ComputeCaptureSet(cbs, has, appbackup.TierOffsite, hddPath) + if got, want := absSet(off), []string{wantWiretest}; !reflect.DeepEqual(got, want) { + t.Errorf("offsite Paths = %v, want %v (optional :ro must not ship offsite)", got, want) + } + // classes ride through end-to-end + for _, p := range sec.Paths { + if p.Abs == wantPhotos && p.Class != appbackup.ClassOptional { + t.Errorf("media/photos class = %q, want optional (explicit beats :ro default, through the real Manager)", p.Class) + } + if p.Abs == wantWiretest && p.Class != appbackup.ClassMandatory { + t.Errorf("appdata/wiretest class = %q, want mandatory (unlisted writable default)", p.Class) + } + } +}