From cf9ce019171eadb53af7f9bfd51046aa1a76aa27 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 15 Jul 2026 11:28:40 +0200 Subject: [PATCH] .fab exclusion scoping: classes in the manual export (Task 4, v0.136.0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SQ6 over-capture FIXED for classified apps: the userdata root tar is exclude-scoped (keeps only ancestors/descendants of a SELECTED bind relpath — R1-C, the tier2Reconcile keep-rule); no selected userdata bind → no root tar (radarr state-only). New appbackup.ComputeFabBuckets (shared resolveGuardCollapse pipeline; class buckets; guards over ALL classes; no cross-bucket containment). appexport/fabplan.go: computeFabPlan + tarDirectoryExcluding + fabEstimateSplit. ExportRequest gains DeselectOptional/OptInExcluded (both start handlers — two-call-site); mandatory is a server-side floor. Manifest v1 + import UNTOUCHED; legacy apps byte-identical to v0.130.0. Estimate additive class split; export UI: locked-mandatory/optional-checkboxes/excluded-opt-in + two-number warning. All 6 §10 red-proofs verified. 6D Accept #1 now runs against this shape. --- CHANGELOG.md | 38 +++ CONTEXT.md | 14 +- REPORT.md | 116 +++---- controller/README.md | 2 + controller/cmd/controller/main.go | 6 + controller/internal/appbackup/captureset.go | 115 +++++-- .../internal/appbackup/fabbuckets_test.go | 96 ++++++ controller/internal/appexport/estimate.go | 27 ++ .../appexport/estimate_volsize_test.go | 11 +- controller/internal/appexport/export.go | 86 +++-- .../internal/appexport/fabexport_test.go | 138 ++++++++ controller/internal/appexport/fabplan.go | 304 ++++++++++++++++++ controller/internal/appexport/fabplan_test.go | 265 +++++++++++++++ controller/internal/appexport/provider.go | 6 + .../internal/appexport/roundtrip_test.go | 4 + controller/internal/web/fab_export_test.go | 208 ++++++++++++ controller/internal/web/handler_export.go | 20 +- .../internal/web/handler_export_download.go | 18 +- .../internal/web/templates/app_export.html | 95 +++++- 19 files changed, 1392 insertions(+), 177 deletions(-) create mode 100644 controller/internal/appbackup/fabbuckets_test.go create mode 100644 controller/internal/appexport/fabexport_test.go create mode 100644 controller/internal/appexport/fabplan.go create mode 100644 controller/internal/appexport/fabplan_test.go create mode 100644 controller/internal/web/fab_export_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 373be69..c2d7942 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,43 @@ ## Changelog +### v0.136.0 — `.fab` exclusion scoping: classes in the manual export (Task 4) (2026-07-15) + +Task 4 — the `.fab` column of the matrix (architecture §2; the SQ5 exclusion-scoping verdict + Viktor +ruling #1 + R1-C). **The SQ6 over-capture is fixed for classified apps:** a manual export no longer +drags every sibling app's content along in the userdata root tar. Mechanically unchanged from +v0.130.0 — ONE exclude-scoped userdata-root tar + per-mount skip — so the **manifest stays v1** +(basename keying, the `userdata` fallback), the **import side is untouched**, and old controllers +import new bundles correctly (they just extract a root tar containing less). **Legacy (no-block) apps +export byte-identically to v0.130.0** (the SQ5 safety net). + +- **`appbackup.ComputeFabBuckets` (new, pure):** the same resolution + structural guards + equal-Abs + collapse as `ComputeCaptureSet` (shared pipeline extracted, not duplicated), but bucketed by class + (mandatory / optional / excluded), guards over ALL classes (a traversal path is never plannable — + opt-in or not), and NO cross-bucket containment dedup (a mandatory child inside an excluded parent + stays independently addressable). +- **The export plan (`appexport/fabplan.go`):** `ExportRequest` gains `DeselectOptional` + + `OptInExcluded`; `computeFabPlan` resolves the selection (mandatory forced-in — a **server-side + floor** ignores a client trying to deselect a mandatory path; optional default-in; excluded + default-out) into `{SkipMounts, SkipUserdataTar, UserdataExcludeRels}`. The userdata root tar is + **skipped entirely** when no userdata bind is selected (radarr → state-only, the SQ6 headline); else + the exclude list is the **topmost dirs neither ancestor nor descendant of a selected relpath** + (R1-C — the exact `tier2Reconcile` keep-rule). An HDD mount matching no classified bind is KEPT + (fail toward capture). `tarDirectoryExcluding` skips excluded subtrees in the walk (`tarDirectory` + is now a thin wrapper). +- **Estimate split (additive):** `ExportEstimate` gains `HasClassification` + `BaseBytes` + + per-path `MandatoryItems`/`OptionalItems`/`ExcludedItems` (each with its du size) — existing fields + and the fits gate are unchanged. Both estimate pipelines surface it (shared `EstimateExport`). +- **UI (`app_export.html`, classified apps only):** locked mandatory list, pre-selected optional + checkboxes, a collapsed excluded opt-in behind the two-number warning + ("Alap mentés: ~X. A kihagyott, nagy méretű tartalommal együtt: ~Y.") + the FileBrowser pointer; + totals recompute client-side per toggle. Selections POST through BOTH start endpoints (the + two-call-site). +- **Tests:** `ComputeFabBuckets` bucket/guard/containment tests; `computeFabPlan` scenarios A–F + + §8 edge cases; `tarDirectoryExcluding` FS-level; export-level bundle tests (exclude-scoped tar, + legacy full-root, all-excluded no-tar); the two-call-site bundle test across both start pipelines. + All 6 §10 red-proofs verified. **CAMPAIGN-6D Accept #1** (the ≥1 GiB `.fab` full circle) now runs + against this final capture shape. + ### v0.135.0 — Tier-2 engine rework: class-driven legs, v2 layout, NAS-target exclusion (Task 3b) (2026-07-15) Task 3b of the backup-classification-redesign arc — the tier-2 column of the matrix (architecture diff --git a/CONTEXT.md b/CONTEXT.md index 43f2ab8..762c0d9 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,19 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-07-15 (v0.135.0 — tier-2 engine rework, Task 3b) +Last updated: 2026-07-15 (v0.136.0 — .fab exclusion scoping, Task 4) + +> **2026-07-15 — v0.136.0: `.fab` exclusion scoping (Task 4).** Architecture §2 `.fab` row + SQ5 +> verdict + R1-C. SQ6 over-capture FIXED for classified apps: the userdata root tar is exclude-scoped +> (keeps only ancestors/descendants of a SELECTED bind relpath — the tier2Reconcile keep-rule); no +> selected userdata bind ⇒ no root tar (radarr state-only). New `appbackup.ComputeFabBuckets` (shared +> `resolveGuardCollapse` pipeline; class buckets; guards over ALL classes; no cross-bucket containment). +> New `appexport/fabplan.go`: `computeFabPlan` (SkipMounts/SkipUserdataTar/UserdataExcludeRels) + +> `tarDirectoryExcluding` + `fabEstimateSplit`. `ExportRequest` += DeselectOptional/OptInExcluded (both +> start handlers — two-call-site); mandatory is a SERVER-SIDE floor. Manifest v1 + import UNTOUCHED; +> legacy apps byte-identical to v0.130.0. Estimate gains an additive class split; export UI shows +> locked-mandatory / optional-checkboxes / excluded-opt-in + the two-number warning. All 6 §10 +> red-proofs verified. CAMPAIGN-6D Accept #1 (≥1 GiB .fab full circle) now runs against this shape. > **2026-07-15 — v0.135.0: tier-2 engine rework (Task 3b).** Architecture §2/§8. New > `tier2_capture.go`: classified apps get the `TierSecondary` per-bind legs (paperless copy shrinks — diff --git a/REPORT.md b/REPORT.md index 6aa602e..2263b12 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,113 +1,71 @@ -# REPORT — Tier-2 engine rework (Task 3b) — controller v0.135.0 +# REPORT — `.fab` exclusion scoping (Task 4) — controller v0.136.0 ## Summary -Task 3b: the tier-2 column of the matrix (architecture §2/§8). The tier-2 appdata leg becomes -**class-driven** for classified apps (per-bind mandatory + optional; paperless's copy shrinks as -`export` drops out), legacy apps keep a byte-identical resolver set — both mapped into a new **v2 -relpath-mirroring layout** that lifts the N>1 refusal structurally and gains a reconcile step; the SSD -fallback is an enforced **state-only** tier; and `selectTier2Target` gains the F-6C-1 **network-target -exclusion** (pinned + auto). Restore reads v2 behind a marker gate. Plus **Part 0** (persisted prefs -seed — fixes the 3a-fix un-disableable checkbox) and **Part 0.5** (offsite scratch prefers local). -Every destructive write lands only under `backups/secondary//` (code-asserted). +Task 4: the `.fab` column of the matrix (SQ5 verdict + ruling #1 + Viktor's R1-C). **The SQ6 +over-capture is fixed for classified apps** — a manual export's userdata root tar is now +exclude-scoped (keeps only ancestors/descendants of a selected bind), so sibling apps' content no +longer rides along; an all-excluded app produces no userdata tar at all. Mechanics unchanged from +v0.130.0 (one root tar + per-mount skip), so the **manifest stays v1 and the import side is +untouched**; **legacy apps export byte-identically**. Mandatory is a server-side floor; optional is a +pre-selected checkbox; excluded is opt-in behind the two-number warning. ## Baselines (live-verified at session start) | Repo | `main` @ start | Version | → | |---|---|---|---| -| felhom-controller | `1245a6c` | v0.134.1 | **v0.135.0** | +| felhom-controller | `5f21613` | v0.135.0 | **v0.136.0** | ## Files -- **new** `internal/backup/tier2_capture.go` — `tier2CaptureSet` (TierSecondary legs + gaps), `tier2Leg`, `tier2DestRel`. -- **mod** `internal/backup/tier2.go` — `selectTier2Target` (network exclusion, two sizes, `StateOnly`), `RunTier2` v2 body (migration/mirror/reconcile/marker-last), `tier2SafeRemove` + `tier2Reconcile` + `classifyTier2Rel`, `Tier2Info` rewired, header + `RunTier2` comments truthful; deleted `errTier2MultiDir`/`tier2AppDataName`. -- **mod** `internal/backup/tier2_restore.go` — v2 marker gate + two-subtree missing-only merge; deleted `errTier2MultiDirRestore`. -- **mod** `internal/backup/offbox_restore.go` — Part 0.5 scratch local-preference. -- **mod** `internal/backup/backup.go` — `tier2SSDFits` seam. -- **mod** `internal/backup/appbackup_bridge.go` — `ClassMandatory/Optional/Excluded` aliases. -- **mod** `internal/settings/settings.go` — Part 0 seed (`OffboxEnlargeNoticeSeeded` + `seedOffboxEnlargeNotice` at Load; getter append removed); `CrossDriveBackup.LastWarning`. -- **mod** `internal/web/handlers.go` + `templates/backups_apps.html` — surface `Tier2LastWarning`. -- **tests** new `tier2_v2_test.go`, `offbox_scratch_test.go`; rewrote `notif_migration_test.go` (Part 0), `tier2_restore_test.go` fixture (v2), `async_restore_test.go` fixture (v2), `tier2_test.go` signatures; **deleted** obsolete `tier2_appdata_test.go`. +- **mod** `internal/appbackup/captureset.go` — extracted `resolveGuardCollapse` (shared pipeline); new `ComputeFabBuckets` + `FabBuckets`. +- **new** `internal/appexport/fabplan.go` — `fabPlan`, `computeFabPlan`, `fabUserdataExcludes` + `classifyFabRel`, `tarDirectoryExcluding`, `fabEstimateSplit`. +- **mod** `internal/appexport/export.go` — `ExportRequest` += `DeselectOptional`/`OptInExcluded`; `dirLister` seam; `exportHDDData` consumes the plan; `tarDirectory` → thin wrapper. +- **mod** `internal/appexport/estimate.go` — `ExportEstimate` class-split fields (additive); `FabItem`. +- **mod** `internal/appexport/provider.go` + `cmd/controller/main.go` — `GetStackClassifiedBinds` on the export seam + adapter. +- **mod** `internal/web/handler_export.go` + `handler_export_download.go` — both start handlers carry the selections (two-call-site). +- **mod** `internal/web/templates/app_export.html` — the class-selection UI. +- **tests** new `fabbuckets_test.go`, `fabplan_test.go`, `fabexport_test.go`, `fab_export_test.go`; provider fakes updated. - CHANGELOG / REPORT / CONTEXT / README. -Untouched: offbox engine (except `offboxRestoreScratchDir`), appexport, hub, agent; `rsyncMirror`/`rsyncRestoreMissing` unchanged. +Untouched (§5/§12): `manifest.go`, `validate.go`, `restore.go`, every import handler, `ExportDataMounts` — the verdict rests on the import side staying byte-identical. ## Tests — results -`go build ./... && go vet ./... && go test ./...` — **green.** New: v2 suite (§7 A–H + reconcile -keep/remove + safe-remove boundary proof + `classifyTier2Rel` table), Part 0 (idempotent seed + -opt-out-sticks + fresh-customer), Part 0.5 (local-preference + network-only WARN). +`go build ./... && go vet ./... && go test ./...` — **green.** New: bucket/guard/no-containment +(`ComputeFabBuckets`); plan scenarios A–F + §8 (unmatched-mount-kept, skip-when-none, mandatory floor, +nested excludes); `tarDirectoryExcluding` FS-level; export-level bundle tests (exclude-scoped tar, +legacy full-root, all-excluded no-tar); the two-call-site bundle test across BOTH start pipelines. ### §10 red-proofs (mutate → FAIL → revert), all verified | ID | Mutation | Test | |---|---|---| -| A | classified app routed to the legacy resolver | `ClassifiedPaperless` (`export` appears in dest) | -| B | legacy routed through ComputeCaptureSet | `LegacyMultiDir` (empty legs) | -| C | marker written before the legs | `MigrationAndMarkerLast` (marker present after a leg failure) | -| D-reconcile | disable stale detection | `Reconcile` (stale dir survives) | -| D-boundary | neuter the safe-remove prefix assert | `Reconcile` (removal outside destBase not refused) | -| E | SSD copies optional (no state-only filter) | `SSDStateOnly` (optional on the SSD) | -| F1 | drop the pinned IsNetwork check | `NetworkExclusion/pinned` (NAS pin honored) | -| F2 | drop the auto IsNetwork check (NFS iterated first) | `NetworkExclusion/auto` (NAS selected) | -| G2 | neuter the restore marker gate | `RestoreRefusesOldLayout` (merge runs on old layout) | -| Part 0 | don't persist the seed marker | `SeedOffboxEnlargeNotice_OptOutSticks` (type re-enabled) | +| A | legacy routed through the classified plan (drop short-circuit) | `FabExport_LegacyFullRoot` (userdata tar skipped) | +| B | always-tar the userdata root | `FabExport_AllExcludedNoUserdataTar` (tar present) | +| C | invert the keep-rule | `FabExport_ExcludeScopedUserdataTar` (mandatory excluded) | +| D3 | drop the server-side mandatory floor | `FabPlan_MandatoryFloor` (mandatory deselected) | +| E | skip guards for excluded buckets | `ComputeFabBuckets_GuardsAllClasses` (traversal enters a bucket) | +| F | drop selection fields from one start handler | `Fab_SelectionsRideBothStartPipelines/download` (opted-in content absent) | All reverted; post-revert full suite green; no residue. ## Deploy / verify -Built + pushed `felhom-controller:0.135.0` on 180; deployed to guest 9201 → `Up (healthy)`. Commit -`3603d1f`. +`` ## §13 live validation -- **Leg 0 (pre-deploy diagnose) — the prompt's expected cause was WRONG.** The nightly - `crossdrive_failed` was **calibre-web** with `LastError = "mkdir …/teszt_enroll/backups/secondary/ - calibre-web/recovery-unit…"`. teszt_enroll is a **local** ext4 drive, now mounted + writable — the - failure was a **transient disconnect** at run time, **not F-6C-1**. Confirmed read-only that **no - app auto-targets the `nas-media` NFS path** (all dests are felhom-usb / teszt_enroll). So F-6C-1 is - not the demo's failure cause. -- **Leg 1 — classified v2 tree (calibre-web):** `Tier 2 copied calibre-web → teszt_enroll (258.4 MB, - 1 leg, 33s)`; dest tree = `.felhom-tier2-layout`=**2** + `recovery-unit/` + the class-driven - **`userdata/media/books/`** leg (its mandatory bind — not the whole userdata, no flat appdata). Also - the leg-0 before/after: calibre-web was `error` → now `ok`. Migration clean (no old flat `appdata/`). -- **Leg 2 — v2 layout (radarr):** marker=2 + `recovery-unit/` only, **0 legs** (radarr is classified - all-excluded → correctly unit-only). Migration clean. -- **Leg 3 — NAS exclusion:** read-only confirmed no app targets the nas-media NFS path (leg 0); - pinned + auto exclusion is unit-tested (F1/F2 red-proofs) and live in the deployed code. -- **Leg 4 — restore round-trip (calibre-web):** planted a sentinel in live userdata → tier-2 copied it - to the v2 secondary (`userdata/media/books/.felhom-t2probe`) → deleted it live → `RestoreTier2Files` - → **restored** (the v2 `userdata/` subtree missing-only merge, end-to-end). Sentinel cleaned up. -- **Leg 5 — Part 0 opt-out sticks (LIVE):** unchecked `offbox_enlarge_blocked` + saved + **restarted - the controller** → the checkbox is **still unchecked** and the stored `enabled_events` count is 0 — - the seed did NOT re-enable it (vs the 3a-fix getter-append bug). Pref restored afterward. -- **Leg 6 — customer-email diagnosis:** the controller's `notifications.email` is **EMPTY** (synced as - `email=` to the hub), so the dispatcher has no customer address — the synthetic event's operator-only - delivery is **expected, not a bug** (the customer whitelist has the type, the hub accepts it). Fix = - configure a customer email (operator/customer action); **no hub code change** (flagged per §13-6). -- **Leg 7 — NAS hygiene:** removed the empty F-6C-1-era `nas-media/backups/secondary/` dir (rmdir; - nas-media/backups/ now holds only `primary/` + `offsite-restore/`); the NAS is never a tier-2 target - again. +`` -## 6D-pending (shrunk) +## 6D-pending -- `PlaceOffsiteRestore` against live data; organic enlarge-block firing; the SQ3 immich offsite-only - full circle (all from Task 3a — unchanged by 3b). -- A pure **legacy (no-block)** app's live tier-2 `hdd/appdata/` leg — not found on the demo (all - toggled apps carry classification blocks); covered by unit tests (`LegacyMultiDir`). +`` ## Observations -- The `Tier2LastWarning` surfacing is minimal (one line on the app's tier-2 row); the SSD state-only - hint was already rendered via `EffectiveIsSSD`/`EffectiveDesc` (no new template needed there). -- `tier2SSDFits` seam added because `system.GetDiskUsage` is Linux-only (nil on the Windows test host) - — production uses the real predicate. -- **Leg-0 finding contradicts the prompt:** the demo's nightly `crossdrive_failed` is a transient - teszt_enroll mkdir failure, not F-6C-1 (no app auto-targets the NFS path). Recorded, not acted on. -- `nas-media/backups/offsite-restore/` (Jul 15) remains — a pre-Part-0.5 offsite scratch on the network - drive (3a-fix era). Out of leg-7 scope (that leg is tier-2 `backups/secondary/` residue); Part 0.5 - prevents NEW offsite scratch landing there. Noted for a future offsite-scratch hygiene pass. -- Leg 6: the demo customer has no notification email configured — the customer-email delivery leg is - blocked on that config gap (not code); setting a customer email + re-firing one synthetic event would - complete it. +- `ComputeCaptureSet` was refactored to share `resolveGuardCollapse` with `ComputeFabBuckets` (no + duplicate pipeline); its existing tests stayed green through the refactor. +- The estimate split is in the shared `EstimateExport`, so both estimate pipelines surface it by + construction; the genuine two-call-site risk (and the F red-proof) is on the START selections, which + the bundle test exercises per-handler. diff --git a/controller/README.md b/controller/README.md index 54c6c1c..caaef85 100644 --- a/controller/README.md +++ b/controller/README.md @@ -382,6 +382,8 @@ Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypte **Mount discovery (v0.130.0, C6B-F1):** `stacks.ExportDataMounts` — the `${HDD_PATH}` binds unioned with the `${USERDATA_PATH}` **root** (single `userdata` entry) when the compose uses the standard userdata convention; pre-fix the adapter was `${HDD_PATH}`-only, so 12/13 `needs_hdd` catalog apps exported hollow (config-only) bundles. The root (not per-bind) keying is what round-trips through the import's basename→`/` mapping. A basename collision between mounts fails the export loudly. The share-removal endpoint also refuses while a deployed app's HDD_PATH is on the share (C6B-F2 guard). +**Class-scoped export (v0.136.0, Task 4 — the SQ6 fix):** for a **classified** app the userdata root tar is **exclude-scoped** — it keeps only dirs that are an ancestor-or-descendant of a SELECTED bind relpath (mandatory ∪ checked-optional ∪ opted-in-excluded; R1-C, the `tier2Reconcile` keep-rule), so sibling apps' content no longer rides along. No selected userdata bind ⇒ the root tar is **skipped entirely** (radarr → state-only). Non-selected HDD bind mounts are skipped; a mount matching no classified bind is kept (fail toward capture). Mechanics unchanged: **ONE** userdata tar, per-mount skip, **manifest v1 + import untouched**. The plan is pure (`appexport/fabplan.go` `computeFabPlan` over `appbackup.ComputeFabBuckets`); `tarDirectoryExcluding` prunes excluded subtrees in the walk. **Legacy (no-block) apps export byte-identically to v0.130.0.** Mandatory paths are a **server-side floor** (a client cannot deselect them). The export page shows the class selection UI (locked mandatory, pre-selected optional checkboxes, opt-in excluded behind the two-number warning + FileBrowser pointer); the estimate carries an additive class split (`ExportEstimate.MandatoryItems/OptionalItems/ExcludedItems` + `BaseBytes`). + **Import flow:** Decrypt if needed → extract → **validate-before-destroy** (v0.125.0: every manifest-claimed data tar must be present non-empty BEFORE the app is stopped or any volume removed — hollow bundles from containerized ≤0.124.0 exporters are refused with the app untouched) → prepare stack dir (create new or `compose down --volumes` for existing) → restore config (re-encrypt app.yaml with current server key) → restore user data (HDD tars; volumes populate via `docker cp` streaming) → restore DB (start DB service, wait for ready, import dump) → start full stack → refresh UI. **Architecture:** `internal/appexport/` package with `ExportStackProvider` adapter interface (same pattern as `backup.StackDataProvider`). `exportAdapter` in `main.go` bridges `stacks.Manager` to the provider. diff --git a/controller/cmd/controller/main.go b/controller/cmd/controller/main.go index 1d3bbff..3af1896 100644 --- a/controller/cmd/controller/main.go +++ b/controller/cmd/controller/main.go @@ -1353,6 +1353,12 @@ func (a *exportAdapter) GetStackDir(name string) (string, bool) { return filepath.Dir(s.ComposePath), true } +// GetStackClassifiedBinds delegates to the stacks manager's backup-classification resolver (Task 2), +// used by the `.fab` class-scoped export plan (Task 4). +func (a *exportAdapter) GetStackClassifiedBinds(name string) ([]appbackup.ClassifiedBind, bool) { + return a.mgr.ClassifiedBinds(name) +} + func (a *exportAdapter) GetStackComposePath(name string) (string, bool) { s, ok := a.mgr.GetStack(name) if !ok { diff --git a/controller/internal/appbackup/captureset.go b/controller/internal/appbackup/captureset.go index 40a060d..fc7cd8d 100644 --- a/controller/internal/appbackup/captureset.go +++ b/controller/internal/appbackup/captureset.go @@ -79,37 +79,9 @@ func ComputeCaptureSet(binds []ClassifiedBind, hasClassification bool, tier Capt } cs := CaptureSet{HasClassification: true} - // Stage 1+2: tier filter, then structural guards. Survivors are resolved to CapturePath. - var resolved []CapturePath - for _, b := range binds { - if !tierKeeps(tier, b.Class) { - continue // excluded (any tier) or optional@offsite — silently filtered, not skipped - } - if reason, bad := structuralGuard(b.Root, b.RelPath); bad { - cs.Skipped = append(cs.Skipped, SkippedPath{ - Root: b.Root, RelPath: b.RelPath, Class: b.Class, Reason: reason, - }) - continue - } - resolved = append(resolved, CapturePath{ - Abs: resolveAbs(hddPath, b.Root, b.RelPath), Root: b.Root, RelPath: b.RelPath, Class: b.Class, - }) - } - - // Stage 3: equal-Abs collapse — one entry per Abs, mandatory beats optional (mandatory semantics - // must never degrade). Deterministic on ties: keep the lexicographically-smaller (Root, RelPath). - byAbs := make(map[string]CapturePath, len(resolved)) - for _, cp := range resolved { - if cur, ok := byAbs[cp.Abs]; ok { - byAbs[cp.Abs] = strongerCapture(cur, cp) - continue - } - byAbs[cp.Abs] = cp - } - uniq := make([]CapturePath, 0, len(byAbs)) - for _, cp := range byAbs { - uniq = append(uniq, cp) - } + // Stages 1–3: tier filter → structural guards → equal-Abs collapse (shared with ComputeFabBuckets). + uniq, skipped := resolveGuardCollapse(binds, hddPath, func(c BindClass) bool { return tierKeeps(tier, c) }) + cs.Skipped = skipped // Stage 4: containment dedup — drop any path whose ancestor is already present (keep the ancestor). for _, cp := range uniq { @@ -130,6 +102,87 @@ func ComputeCaptureSet(binds []ClassifiedBind, hasClassification bool, tier Capt return cs } +// resolveGuardCollapse is the pipeline shared by ComputeCaptureSet and ComputeFabBuckets: keep-filter +// (the caller's predicate over class) → structural guards (Skipped) → resolve to Abs → equal-Abs +// collapse (mandatory > optional > excluded; ties by smaller Root/RelPath). It does NOT apply +// containment dedup — the caller decides (ComputeCaptureSet does; ComputeFabBuckets must not, so a +// mandatory child inside an excluded parent stays independently addressable). +func resolveGuardCollapse(binds []ClassifiedBind, hddPath string, keep func(BindClass) bool) (uniq []CapturePath, skipped []SkippedPath) { + var resolved []CapturePath + for _, b := range binds { + if !keep(b.Class) { + continue + } + if reason, bad := structuralGuard(b.Root, b.RelPath); bad { + skipped = append(skipped, SkippedPath{Root: b.Root, RelPath: b.RelPath, Class: b.Class, Reason: reason}) + continue + } + resolved = append(resolved, CapturePath{ + Abs: resolveAbs(hddPath, b.Root, b.RelPath), Root: b.Root, RelPath: b.RelPath, Class: b.Class, + }) + } + byAbs := make(map[string]CapturePath, len(resolved)) + for _, cp := range resolved { + if cur, ok := byAbs[cp.Abs]; ok { + byAbs[cp.Abs] = strongerCapture(cur, cp) + continue + } + byAbs[cp.Abs] = cp + } + uniq = make([]CapturePath, 0, len(byAbs)) + for _, cp := range byAbs { + uniq = append(uniq, cp) + } + return uniq, skipped +} + +// FabBuckets is the class-bucketed capture set for the manual `.fab` export (Task 4). Unlike +// ComputeCaptureSet it keeps ALL classes (structural guards run over every class — a traversal path is +// never plannable, opt-in or not) and does NOT collapse across containment (mandatory `media/books` +// inside excluded `media` both survive, in their own buckets). HasClassification=false ⇒ empty (the +// legacy full-root capture, unchanged). +type FabBuckets struct { + HasClassification bool + Mandatory []CapturePath + Optional []CapturePath + Excluded []CapturePath + Skipped []SkippedPath +} + +// ComputeFabBuckets resolves an app's classified binds into per-class buckets for the `.fab` export +// selection UI + plan. Same resolution + structural guards + equal-Abs collapse as ComputeCaptureSet +// (via resolveGuardCollapse), bucketed by class, no cross-bucket containment dedup. Each bucket is +// Abs-sorted (deterministic). +func ComputeFabBuckets(binds []ClassifiedBind, hasClassification bool, hddPath string) FabBuckets { + if !hasClassification { + return FabBuckets{HasClassification: false} + } + fb := FabBuckets{HasClassification: true} + uniq, skipped := resolveGuardCollapse(binds, hddPath, func(BindClass) bool { return true }) + fb.Skipped = skipped + for _, cp := range uniq { + switch cp.Class { + case ClassMandatory: + fb.Mandatory = append(fb.Mandatory, cp) + case ClassOptional: + fb.Optional = append(fb.Optional, cp) + default: + fb.Excluded = append(fb.Excluded, cp) + } + } + for _, b := range []*[]CapturePath{&fb.Mandatory, &fb.Optional, &fb.Excluded} { + bk := *b + sort.Slice(bk, func(i, j int) bool { return bk[i].Abs < bk[j].Abs }) + } + sort.Slice(fb.Skipped, func(i, j int) bool { + if fb.Skipped[i].Root != fb.Skipped[j].Root { + return fb.Skipped[i].Root < fb.Skipped[j].Root + } + return fb.Skipped[i].RelPath < fb.Skipped[j].RelPath + }) + return fb +} + // tierKeeps applies the §2 tier column: mandatory everywhere, optional only for secondary, excluded // never. func tierKeeps(tier CaptureTier, class BindClass) bool { diff --git a/controller/internal/appbackup/fabbuckets_test.go b/controller/internal/appbackup/fabbuckets_test.go new file mode 100644 index 0000000..5ab00f8 --- /dev/null +++ b/controller/internal/appbackup/fabbuckets_test.go @@ -0,0 +1,96 @@ +package appbackup + +import ( + "reflect" + "testing" +) + +func bucketAbs(b []CapturePath) []string { + out := make([]string, 0, len(b)) + for _, p := range b { + out = append(out, p.Abs) + } + return out +} + +// classified app → three buckets, resolved + Abs-sorted. +func TestComputeFabBuckets_Classified(t *testing.T) { + binds := []ClassifiedBind{ + {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/books"}, Class: ClassMandatory}, + {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/comics"}, Class: ClassOptional}, + {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/movies"}, Class: ClassExcluded}, + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/app"}, Class: ClassMandatory}, + } + fb := ComputeFabBuckets(binds, true, drv) + if !fb.HasClassification { + t.Fatal("HasClassification must be true") + } + if got, want := bucketAbs(fb.Mandatory), []string{hdd("appdata/app"), udat("media/books")}; !reflect.DeepEqual(got, want) { + t.Errorf("Mandatory = %v, want %v", got, want) + } + if got, want := bucketAbs(fb.Optional), []string{udat("media/comics")}; !reflect.DeepEqual(got, want) { + t.Errorf("Optional = %v, want %v", got, want) + } + if got, want := bucketAbs(fb.Excluded), []string{udat("media/movies")}; !reflect.DeepEqual(got, want) { + t.Errorf("Excluded = %v, want %v", got, want) + } +} + +// legacy (no block) → empty buckets (the full-root capture stays out of the classified plan). +func TestComputeFabBuckets_LegacyEmpty(t *testing.T) { + binds := []ClassifiedBind{{ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/tv"}, Origin: OriginLegacy}} + fb := ComputeFabBuckets(binds, false, drv) + if fb.HasClassification || fb.Mandatory != nil || fb.Optional != nil || fb.Excluded != nil { + t.Errorf("legacy app must yield empty buckets, got %+v", fb) + } +} + +// Scenario E: structural guards run over ALL classes — a traversal path in an EXCLUDED bind is Skipped, +// never plannable (opt-in or not). +func TestComputeFabBuckets_GuardsAllClasses(t *testing.T) { + binds := []ClassifiedBind{ + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "../evil"}, Class: ClassExcluded}, + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/ok"}, Class: ClassMandatory}, + } + fb := ComputeFabBuckets(binds, true, drv) + for _, b := range [][]CapturePath{fb.Mandatory, fb.Optional, fb.Excluded} { + for _, p := range b { + if p.RelPath == "../evil" { + t.Fatal("a traversal path must never enter a bucket (guards run over all classes)") + } + } + } + if len(fb.Skipped) != 1 || fb.Skipped[0].RelPath != "../evil" { + t.Errorf("traversal excluded path must be Skipped, got %+v", fb.Skipped) + } +} + +// no cross-bucket containment dedup: a mandatory CHILD inside an excluded PARENT both survive. +func TestComputeFabBuckets_NoCrossBucketContainment(t *testing.T) { + binds := []ClassifiedBind{ + {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media"}, Class: ClassExcluded}, + {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/books"}, Class: ClassMandatory}, + } + fb := ComputeFabBuckets(binds, true, drv) + if got, want := bucketAbs(fb.Mandatory), []string{udat("media/books")}; !reflect.DeepEqual(got, want) { + t.Errorf("mandatory child must survive independently: Mandatory = %v, want %v", got, want) + } + if got, want := bucketAbs(fb.Excluded), []string{udat("media")}; !reflect.DeepEqual(got, want) { + t.Errorf("excluded parent must survive: Excluded = %v, want %v", got, want) + } +} + +// equal-Abs collapse: two spellings of one path collapse, mandatory wins (into the mandatory bucket). +func TestComputeFabBuckets_EqualAbsMandatoryWins(t *testing.T) { + binds := []ClassifiedBind{ + {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "userdata/media"}, Class: ClassOptional}, + {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media"}, Class: ClassMandatory}, + } + fb := ComputeFabBuckets(binds, true, drv) + if got, want := bucketAbs(fb.Mandatory), []string{udat("media")}; !reflect.DeepEqual(got, want) { + t.Errorf("collapsed path must land in Mandatory, got Mandatory=%v", got) + } + if len(fb.Optional) != 0 { + t.Errorf("optional spelling must collapse away, got %v", bucketAbs(fb.Optional)) + } +} diff --git a/controller/internal/appexport/estimate.go b/controller/internal/appexport/estimate.go index 51fd469..3d36fa7 100644 --- a/controller/internal/appexport/estimate.go +++ b/controller/internal/appexport/estimate.go @@ -27,6 +27,27 @@ type ExportEstimate struct { // failed). When true, DataSizeBytes is a partial/understated sum and FitsOnDest is FORCED false // — a failed read must NEVER render as "fits". The UI shows "ismeretlen méret". SizeUnknown bool `json:"size_unknown"` + + // Task 4 class split (classified apps only; empty for legacy — existing fields above are + // unchanged, so old JSON consumers keep working). BaseBytes = config + DB + volumes + mandatory + // (always in the bundle). OptionalItems are pre-selected, ExcludedItems are opt-in — each carries + // its own size so the UI recomputes the total client-side per checkbox toggle (no extra du calls). + HasClassification bool `json:"has_classification"` + BaseBytes int64 `json:"base_bytes"` + BaseHuman string `json:"base_human"` + MandatoryItems []FabItem `json:"mandatory_items,omitempty"` + OptionalItems []FabItem `json:"optional_items,omitempty"` + ExcludedItems []FabItem `json:"excluded_items,omitempty"` +} + +// FabItem is one class-scoped path in the `.fab` selection UI: Key is the DeselectOptional/OptInExcluded +// value ("root/rel"), RelPath is the display path, Bytes/Human its du size. +type FabItem struct { + Key string `json:"key"` + Root string `json:"root"` + RelPath string `json:"rel_path"` + Bytes int64 `json:"bytes"` + Human string `json:"human"` } // EstimateExport calculates size estimates for an app export. @@ -57,6 +78,7 @@ func (e *Exporter) EstimateExport(stackName, destDrive string) (*ExportEstimate, } volumes := e.provider.GetDockerVolumes(stackName) e.debugf("EstimateExport: Docker volumes: %v", volumes) + var volumeBytes int64 for _, vol := range volumes { volSize, err := volumeSizer(vol) if err != nil { @@ -68,6 +90,7 @@ func (e *Exporter) EstimateExport(stackName, destDrive string) (*ExportEstimate, } e.debugf("EstimateExport: volume %s = %s", vol, humanizeBytes(volSize)) est.DataSizeBytes += volSize + volumeBytes += volSize } if est.SizeUnknown { est.DataSizeHuman = "ismeretlen méret" @@ -78,6 +101,10 @@ func (e *Exporter) EstimateExport(stackName, destDrive string) (*ExportEstimate, est.TotalSizeBytes = est.ConfigSizeBytes + est.DataSizeBytes est.TotalSizeHuman = humanizeBytes(est.TotalSizeBytes) + // Task 4: the class split (classified apps only). Independent du over each bucket path — additive, + // never touches the fields/fits gate above. + e.fabEstimateSplit(stackName, est, volumeBytes) + // Rough time estimate: ~500 MB/min for HDDs, minimum 1 minute minutes := int(est.TotalSizeBytes / (500 * 1024 * 1024)) if minutes < 1 { diff --git a/controller/internal/appexport/estimate_volsize_test.go b/controller/internal/appexport/estimate_volsize_test.go index 386a766..727e9c1 100644 --- a/controller/internal/appexport/estimate_volsize_test.go +++ b/controller/internal/appexport/estimate_volsize_test.go @@ -6,19 +6,26 @@ import ( "log" "strings" "testing" + + "gitea.dooplex.hu/admin/felhom-controller/internal/appbackup" ) // hddProvider is an rtProvider that reports an HDD-backed stack (estimate scenario H + the // v0.130.0 additive-export tests in export_additive_test.go). type hddProvider struct { *rtProvider - mounts []string - hddPath string + mounts []string + hddPath string + binds []appbackup.ClassifiedBind + hasBinds bool } func (p *hddProvider) GetStackNeedsHDD(string) bool { return true } func (p *hddProvider) GetStackHDDMounts(string) []string { return p.mounts } func (p *hddProvider) GetStackHDDPath(string) string { return p.hddPath } +func (p *hddProvider) GetStackClassifiedBinds(string) ([]appbackup.ClassifiedBind, bool) { + return p.binds, p.hasBinds +} func newEstimator(t *testing.T, provider ExportStackProvider) *Exporter { t.Helper() diff --git a/controller/internal/appexport/export.go b/controller/internal/appexport/export.go index 4a9b69f..f4ea8f3 100644 --- a/controller/internal/appexport/export.go +++ b/controller/internal/appexport/export.go @@ -75,6 +75,14 @@ type ExportRequest struct { DestDrive string // drive mount path (e.g., "/mnt/hdd_1") Password string // empty = no encryption StopApp bool // stop app before export + + // `.fab` class-scoped selection (Task 4; classified apps only — legacy apps ignore these). Both + // empty = ruling #1 defaults (mandatory in, optional in, excluded out). Values are "root/rel" keys + // (matching a CapturePath: "hdd/appdata/x" | "userdata/media/y"). The server enforces the floor: + // a DeselectOptional entry naming a MANDATORY path is ignored with a WARN (the client cannot weaken + // the mandatory floor). OptInExcluded pulls an excluded bind into the bundle. + DeselectOptional []string + OptInExcluded []string } // Exporter manages app export/import operations. @@ -84,6 +92,10 @@ type Exporter struct { version string debug bool + // dirLister (Task 4) lists child DIR names of a path — the seam the `.fab` userdata-exclude + // computation walks. Nil → the real os.ReadDir-based lister. + dirLister func(dir string) []string + mu sync.Mutex activeJob *Job } @@ -330,7 +342,7 @@ func (e *Exporter) executeExport(req ExportRequest, job *Job) { // dropped every named volume of every needs_hdd app. if e.provider.GetStackNeedsHDD(req.StackName) { e.debugf("exporting HDD data for %s", req.StackName) - if err := e.exportHDDData(req.StackName, dataDir, manifest); err != nil { + if err := e.exportHDDData(req, dataDir, manifest); err != nil { e.failJob(job, step, fmt.Sprintf("Felhasználói adatok mentése sikertelen: %v", err)) return } @@ -591,7 +603,8 @@ func (e *Exporter) dumpDatabase(stackName, dbDir string, manifest *Manifest) boo // collision between two mounts is a FATAL error (the manifest keys tars by basename; the old // code silently overwrote the first tar). A non-existent mount is still soft-skipped (honestly // absent from the manifest — the anti-hollow guard catches total emptiness). -func (e *Exporter) exportHDDData(stackName, dataDir string, manifest *Manifest) error { +func (e *Exporter) exportHDDData(req ExportRequest, dataDir string, manifest *Manifest) error { + stackName := req.StackName hddDir := filepath.Join(dataDir, "hdd") os.MkdirAll(hddDir, 0755) @@ -602,8 +615,22 @@ func (e *Exporter) exportHDDData(stackName, dataDir string, manifest *Manifest) return nil } + // Task 4: the class-scoped plan. Legacy / no-block apps get an EMPTY plan (all mounts kept, root + // tar with zero excludes) → byte-identical v0.130.0 capture. + plan := e.computeFabPlan(req, mounts) + ud := appbackup.UserdataDir(filepath.Clean(e.provider.GetStackHDDPath(stackName))) + claimed := make(map[string]string) // subdir → mount that claimed it for _, mount := range mounts { + if plan.SkipMounts[filepath.Clean(mount)] { + e.debugf("HDD mount %s skipped — not selected (class-scoped plan)", mount) + continue + } + isUserdataRoot := filepath.Clean(mount) == filepath.Clean(ud) + if isUserdataRoot && plan.SkipUserdataTar { + e.debugf("userdata root %s skipped — no selected userdata bind (Scenario B)", mount) + continue + } if _, err := os.Stat(mount); os.IsNotExist(err) { e.debugf("HDD mount %s does not exist — skipping", mount) continue @@ -620,9 +647,13 @@ func (e *Exporter) exportHDDData(stackName, dataDir string, manifest *Manifest) claimed[subdir] = mount tarPath := filepath.Join(hddDir, subdir+".tar") - e.debugf("tarring HDD mount: %s → %s", mount, tarPath) + var excludes []string + if isUserdataRoot { + excludes = plan.UserdataExcludeRels // R1-C: exclude-scoped root tar (empty for legacy) + } + e.debugf("tarring HDD mount: %s → %s (%d exclude(s))", mount, tarPath, len(excludes)) tarStart := time.Now() - if err := tarDirectory(mount, tarPath); err != nil { + if err := tarDirectoryExcluding(mount, tarPath, excludes); err != nil { // v0.125.0: claim the subdir ONLY on success — a claimed-but-absent tar would trip // the packaging assertion; an honestly-skipped mount stays out of the manifest. e.logger.Printf("[WARN] Export: failed to tar %s (excluded from the bundle): %v", mount, err) @@ -817,51 +848,10 @@ func createTarGz(outputPath, sourceDir string) error { }) } -// tarDirectory creates a tar (not gzipped) of a directory's contents. +// tarDirectory creates a tar (not gzipped) of a directory's contents. Thin wrapper over +// tarDirectoryExcluding (Task 4) with no excludes — its existing callers are unchanged. func tarDirectory(sourceDir, outputPath string) error { - outFile, err := os.Create(outputPath) - if err != nil { - return err - } - defer outFile.Close() - - tw := tar.NewWriter(outFile) - defer tw.Close() - - return filepath.Walk(sourceDir, func(path string, info os.FileInfo, err error) error { - if err != nil { - return err - } - relPath, err := filepath.Rel(sourceDir, path) - if err != nil { - return err - } - if relPath == "." { - return nil - } - - header, err := tar.FileInfoHeader(info, "") - if err != nil { - return err - } - header.Name = relPath - - if err := tw.WriteHeader(header); err != nil { - return err - } - - if info.IsDir() { - return nil - } - - f, err := os.Open(path) - if err != nil { - return err - } - defer f.Close() - _, err = io.Copy(tw, f) - return err - }) + return tarDirectoryExcluding(sourceDir, outputPath, nil) } // gzipFile compresses a file with gzip. diff --git a/controller/internal/appexport/fabexport_test.go b/controller/internal/appexport/fabexport_test.go new file mode 100644 index 0000000..ba4ba49 --- /dev/null +++ b/controller/internal/appexport/fabexport_test.go @@ -0,0 +1,138 @@ +package appexport + +import ( + "io" + "log" + "os" + "path/filepath" + "testing" + + "gitea.dooplex.hu/admin/felhom-controller/internal/appbackup" +) + +func fabWrite(t *testing.T, root, rel, content string) { + t.Helper() + p := filepath.Join(root, filepath.FromSlash(rel)) + if err := os.MkdirAll(filepath.Dir(p), 0755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(content), 0644); err != nil { + t.Fatal(err) + } +} + +// Scenario C at the export level: the userdata.tar keeps the mandatory subtree and NOT the siblings, +// and the manifest still lists the single `userdata` basename (v1 unchanged). The bundle-level anchor +// for the SQ6 fix (mirrors the §13 before/after). +func TestFabExport_ExcludeScopedUserdataTar(t *testing.T) { + drive := t.TempDir() // hddPath + fabWrite(t, drive, "userdata/media/books/a.epub", "BOOK") + fabWrite(t, drive, "userdata/media/movies/big.mkv", "MOVIE") + fabWrite(t, drive, "userdata/music/s.flac", "SONG") + + ud := appbackup.UserdataDir(filepath.Clean(drive)) + prov := &fabProv{ + rtProvider: &rtProvider{}, hddPath: drive, has: true, + binds: []appbackup.ClassifiedBind{mUD("media/books"), xUD("media/movies")}, + mounts: []string{ud}, + } + e := NewExporter(prov, log.New(io.Discard, "", 0), "test") + + dataDir := t.TempDir() + man := &Manifest{} + if err := e.exportHDDData(ExportRequest{StackName: "calibre-web"}, dataDir, man); err != nil { + t.Fatalf("exportHDDData: %v", err) + } + + entries := tarEntries(t, filepath.Join(dataDir, "hdd", "userdata.tar")) + if !containsSuffix(entries, "media/books/a.epub") { + t.Errorf("mandatory media/books missing from userdata.tar: %v", entries) + } + for _, sib := range []string{"media/movies/big.mkv", "media/movies", "music/s.flac", "music"} { + if containsSuffix(entries, sib) { + t.Errorf("sibling %q must NOT ride along (SQ6): %v", sib, entries) + } + } + // v1 manifest: the single `userdata` basename, unchanged. + if len(man.HDDSubdirs) != 1 || man.HDDSubdirs[0] != "userdata" { + t.Errorf("manifest must list the single v1 `userdata` basename, got %v", man.HDDSubdirs) + } +} + +// Scenario A at the export level: a legacy (no-block) app tars the FULL userdata root (every sibling) +// — byte-identical to v0.130.0 (the SQ5 safety net). +func TestFabExport_LegacyFullRoot(t *testing.T) { + drive := t.TempDir() + fabWrite(t, drive, "userdata/media/books/a.epub", "BOOK") + fabWrite(t, drive, "userdata/media/movies/big.mkv", "MOVIE") + + ud := appbackup.UserdataDir(filepath.Clean(drive)) + prov := &fabProv{rtProvider: &rtProvider{}, hddPath: drive, has: false, mounts: []string{ud}} + e := NewExporter(prov, log.New(io.Discard, "", 0), "test") + + dataDir := t.TempDir() + man := &Manifest{} + if err := e.exportHDDData(ExportRequest{StackName: "sonarr"}, dataDir, man); err != nil { + t.Fatalf("exportHDDData: %v", err) + } + entries := tarEntries(t, filepath.Join(dataDir, "hdd", "userdata.tar")) + for _, want := range []string{"media/books/a.epub", "media/movies/big.mkv"} { + if !containsSuffix(entries, want) { + t.Errorf("legacy app must capture the FULL root — %q missing: %v", want, entries) + } + } +} + +// Scenario B at the export level: an all-excluded app produces NO userdata.tar (root tar skipped). +func TestFabExport_AllExcludedNoUserdataTar(t *testing.T) { + drive := t.TempDir() + fabWrite(t, drive, "userdata/media/movies/big.mkv", "MOVIE") + ud := appbackup.UserdataDir(filepath.Clean(drive)) + prov := &fabProv{ + rtProvider: &rtProvider{}, hddPath: drive, has: true, + binds: []appbackup.ClassifiedBind{xUD("media/movies")}, + mounts: []string{ud}, + } + e := NewExporter(prov, log.New(io.Discard, "", 0), "test") + dataDir := t.TempDir() + man := &Manifest{} + if err := e.exportHDDData(ExportRequest{StackName: "radarr"}, dataDir, man); err != nil { + t.Fatalf("exportHDDData: %v", err) + } + if _, err := os.Stat(filepath.Join(dataDir, "hdd", "userdata.tar")); !os.IsNotExist(err) { + t.Errorf("all-excluded app must produce NO userdata.tar (Scenario B), stat err=%v", err) + } + if len(man.HDDSubdirs) != 0 { + t.Errorf("no userdata leg → no manifest subdir, got %v", man.HDDSubdirs) + } +} + +// §7-F: EstimateExport populates the class split for a classified app (both web estimate pipelines +// call this shared function, so both surface it). du returns 0 on the Windows test host, so this +// asserts the STRUCTURE (HasClassification + item keys), not byte values. +func TestEstimateExport_ClassifiedSplit(t *testing.T) { + drive := t.TempDir() + stackDir := t.TempDir() + os.WriteFile(filepath.Join(stackDir, "docker-compose.yml"), []byte("services: {}\n"), 0644) + prov := &fabProv{ + rtProvider: &rtProvider{stackDir: stackDir, deployed: true}, hddPath: drive, has: true, + binds: []appbackup.ClassifiedBind{mUD("media/books"), oUD("media/comics"), xUD("media/movies")}, + } + e := NewExporter(prov, log.New(io.Discard, "", 0), "test") + est, err := e.EstimateExport("calibre-web", drive) + if err != nil { + t.Fatalf("EstimateExport: %v", err) + } + if !est.HasClassification { + t.Fatal("classified app estimate must carry HasClassification") + } + if len(est.MandatoryItems) != 1 || est.MandatoryItems[0].Key != "userdata/media/books" { + t.Errorf("MandatoryItems = %+v", est.MandatoryItems) + } + if len(est.OptionalItems) != 1 || est.OptionalItems[0].Key != "userdata/media/comics" { + t.Errorf("OptionalItems = %+v", est.OptionalItems) + } + if len(est.ExcludedItems) != 1 || est.ExcludedItems[0].Key != "userdata/media/movies" { + t.Errorf("ExcludedItems = %+v", est.ExcludedItems) + } +} diff --git a/controller/internal/appexport/fabplan.go b/controller/internal/appexport/fabplan.go new file mode 100644 index 0000000..efa74d8 --- /dev/null +++ b/controller/internal/appexport/fabplan.go @@ -0,0 +1,304 @@ +package appexport + +import ( + "archive/tar" + "io" + "os" + "path/filepath" + "sort" + "strings" + + "gitea.dooplex.hu/admin/felhom-controller/internal/appbackup" +) + +// `.fab` class-scoped export plan (Task 4, architecture §2 `.fab` row + SQ5 exclusion-scoping verdict, +// R1-C). Mechanics unchanged from v0.130.0: ONE exclude-scoped userdata-root tar + per-mount skip for +// non-selected HDD binds. The manifest stays v1 (basename keying) and the import side is untouched. + +// fabPlan is the class-scoped adjustment to the v0.130.0 mount/tar set. Empty (all zero) = the legacy +// full capture (Scenario A — a no-block app produces this). +type fabPlan struct { + SkipMounts map[string]bool // absolute HDD mount paths to skip entirely + SkipUserdataTar bool // no selected userdata bind ⇒ the whole root tar is skipped (Scenario B) + UserdataExcludeRels []string // rels (relative to the userdata root) excluded from its tar (R1-C, Scenario C) +} + +func relKey(root appbackup.BindRoot, rel string) string { return string(root) + "/" + rel } +func relKeyOf(cp appbackup.CapturePath) string { return relKey(cp.Root, cp.RelPath) } + +// computeFabPlan resolves the class buckets + the caller's selection into the mount/userdata plan +// (§8). Legacy / no-block ⇒ empty plan. The mandatory floor is enforced here: DeselectOptional can +// never drop a mandatory path. +func (e *Exporter) computeFabPlan(req ExportRequest, mounts []string) fabPlan { + binds, has := e.provider.GetStackClassifiedBinds(req.StackName) + if !has { + return fabPlan{} // legacy: byte-identical v0.130.0 capture + } + hddPath := filepath.Clean(e.provider.GetStackHDDPath(req.StackName)) + fb := appbackup.ComputeFabBuckets(binds, has, hddPath) + + deselect := sliceSet(req.DeselectOptional) + optIn := sliceSet(req.OptInExcluded) + + // Server-side floor: a request naming a mandatory path in DeselectOptional is ignored (loud WARN). + for _, cp := range fb.Mandatory { + if deselect[relKeyOf(cp)] { + e.logger.Printf("[WARN] appexport: %s: request tried to deselect a MANDATORY path %s — ignored (floor enforced)", req.StackName, relKeyOf(cp)) + } + } + + // Resolve the selected set (mandatory always; optional default-in; excluded default-out). + var selectedHDD, selectedUD []appbackup.CapturePath + add := func(cp appbackup.CapturePath) { + if cp.Root == appbackup.RootUserdata { + selectedUD = append(selectedUD, cp) + } else { + selectedHDD = append(selectedHDD, cp) + } + } + for _, cp := range fb.Mandatory { + add(cp) + } + for _, cp := range fb.Optional { + if !deselect[relKeyOf(cp)] { + add(cp) + } + } + for _, cp := range fb.Excluded { + if optIn[relKeyOf(cp)] { + add(cp) + } + } + + // Every classified HDD bind (any class) — a mount matching NONE of these is "unclassified" and kept + // (fail toward capture, the C6B-F1 direction). + var classifiedHDD []string + for _, bucket := range [][]appbackup.CapturePath{fb.Mandatory, fb.Optional, fb.Excluded} { + for _, cp := range bucket { + if cp.Root == appbackup.RootHDD { + classifiedHDD = append(classifiedHDD, cp.Abs) + } + } + } + + plan := fabPlan{SkipMounts: map[string]bool{}} + ud := appbackup.UserdataDir(hddPath) + for _, m := range mounts { + mc := filepath.Clean(m) + if mc == filepath.Clean(ud) { + if len(selectedUD) == 0 { + plan.SkipUserdataTar = true // Scenario B: no selected userdata bind → no root tar + } + continue + } + // HDD mount. Unmatched by ANY classified bind → keep (fail toward capture), log it. + if !relatedToAny(mc, classifiedHDD) { + e.logger.Printf("[INFO] appexport: %s: HDD mount %s matches no classified bind — kept (fail toward capture)", req.StackName, mc) + continue + } + // Matched a classified bind: keep iff ancestor-or-descendant of a SELECTED HDD path. + if !relatedToAny(mc, absList(selectedHDD)) { + plan.SkipMounts[mc] = true + } + } + + if !plan.SkipUserdataTar && len(selectedUD) > 0 { + plan.UserdataExcludeRels = e.fabUserdataExcludes(ud, udRels(selectedUD)) + } + return plan +} + +// fabEstimateSplit populates the class-split estimate fields for a classified app (Task 4). Legacy / +// no-block apps leave HasClassification=false (the UI shows the plain estimate). BaseBytes = config + +// volumes + mandatory; optional/excluded carry per-path sizes for client-side total recomputation. +func (e *Exporter) fabEstimateSplit(stackName string, est *ExportEstimate, volumeBytes int64) { + binds, has := e.provider.GetStackClassifiedBinds(stackName) + if !has { + return + } + hddPath := filepath.Clean(e.provider.GetStackHDDPath(stackName)) + fb := appbackup.ComputeFabBuckets(binds, has, hddPath) + est.HasClassification = true + + toItems := func(cps []appbackup.CapturePath) ([]FabItem, int64) { + var items []FabItem + var sum int64 + for _, cp := range cps { + sz := duBytes(cp.Abs) + items = append(items, FabItem{Key: relKeyOf(cp), Root: string(cp.Root), RelPath: cp.RelPath, Bytes: sz, Human: humanizeBytes(sz)}) + sum += sz + } + return items, sum + } + var mandSum int64 + est.MandatoryItems, mandSum = toItems(fb.Mandatory) + est.OptionalItems, _ = toItems(fb.Optional) + est.ExcludedItems, _ = toItems(fb.Excluded) + est.BaseBytes = est.ConfigSizeBytes + volumeBytes + mandSum + est.BaseHuman = humanizeBytes(est.BaseBytes) +} + +func sliceSet(ss []string) map[string]bool { + m := make(map[string]bool, len(ss)) + for _, s := range ss { + m[s] = true + } + return m +} + +func absList(cps []appbackup.CapturePath) []string { + out := make([]string, len(cps)) + for i, cp := range cps { + out[i] = cp.Abs + } + return out +} + +// udRels returns the userdata rels (relative to ${USERDATA_PATH}) of selected userdata paths. +func udRels(cps []appbackup.CapturePath) []string { + out := make([]string, 0, len(cps)) + for _, cp := range cps { + out = append(out, cp.RelPath) + } + return out +} + +// relatedToAny reports whether path p is an ancestor OR descendant (or equal) of any path in set. +func relatedToAny(p string, set []string) bool { + pc := filepath.Clean(p) + for _, s := range set { + sc := filepath.Clean(s) + if pc == sc || strings.HasPrefix(pc, sc+string(filepath.Separator)) || strings.HasPrefix(sc, pc+string(filepath.Separator)) { + return true + } + } + return false +} + +// fabRelClass classifies a dir rel (slash-form, relative to the userdata root) against the selected +// userdata rels — the R1-C keep-rule (mirrors backup.classifyTier2Rel, copied not imported): +// keepInside = a selected rel or inside one (keep, don't descend); keepAncestor = on the path to a +// selected rel (keep, descend); else stale (exclude the topmost). +type fabRelClass int + +const ( + fabStale fabRelClass = iota + fabKeepInside + fabKeepAncestor +) + +func classifyFabRel(dirRel string, selectedRels []string) fabRelClass { + for _, sr := range selectedRels { + if dirRel == sr || strings.HasPrefix(dirRel, sr+"/") { + return fabKeepInside + } + } + for _, sr := range selectedRels { + if strings.HasPrefix(sr, dirRel+"/") { + return fabKeepAncestor + } + } + return fabStale +} + +// fabUserdataExcludes walks the userdata root (via the dirLister seam) and returns the topmost rels +// (relative to the root, slash-form) that are neither an ancestor nor a descendant of a selected +// userdata rel — the exclude list for the root tar (R1-C). Deterministic (sorted). +func (e *Exporter) fabUserdataExcludes(udRoot string, selectedRels []string) []string { + lister := e.dirLister + if lister == nil { + lister = realDirLister + } + var excludes []string + var walk func(dirAbs, dirRel string) + walk = func(dirAbs, dirRel string) { + for _, name := range lister(dirAbs) { + childRel := name + if dirRel != "" { + childRel = dirRel + "/" + name + } + switch classifyFabRel(childRel, selectedRels) { + case fabKeepInside: + // selected leg or content inside it — keep, no descent + case fabKeepAncestor: + walk(filepath.Join(dirAbs, name), childRel) + default: + excludes = append(excludes, childRel) // topmost neither-ancestor-nor-descendant + } + } + } + walk(udRoot, "") + sort.Strings(excludes) + return excludes +} + +func realDirLister(dir string) []string { + entries, err := os.ReadDir(dir) + if err != nil { + return nil + } + var names []string + for _, en := range entries { + if en.IsDir() { + names = append(names, en.Name()) + } + } + return names +} + +// tarDirectoryExcluding is tarDirectory with an exclude list: any path whose rel (relative to +// sourceDir, slash-form) equals or descends from an exclude rel is skipped (a dir is pruned whole). +// Empty excludes == tarDirectory. Anchored to the tar root exactly like tarDirectory's rel names. +func tarDirectoryExcluding(sourceDir, outputPath string, excludeRels []string) error { + excl := make([]string, len(excludeRels)) + for i, r := range excludeRels { + excl[i] = filepath.ToSlash(r) + } + outFile, err := os.Create(outputPath) + if err != nil { + return err + } + defer outFile.Close() + tw := tar.NewWriter(outFile) + defer tw.Close() + + return filepath.Walk(sourceDir, func(path string, info os.FileInfo, err error) error { + if err != nil { + return err + } + relPath, err := filepath.Rel(sourceDir, path) + if err != nil { + return err + } + if relPath == "." { + return nil + } + rel := filepath.ToSlash(relPath) + for _, e := range excl { + if rel == e || strings.HasPrefix(rel, e+"/") { + if info.IsDir() { + return filepath.SkipDir + } + return nil + } + } + header, err := tar.FileInfoHeader(info, "") + if err != nil { + return err + } + header.Name = relPath + if err := tw.WriteHeader(header); err != nil { + return err + } + if info.IsDir() { + return nil + } + f, err := os.Open(path) + if err != nil { + return err + } + defer f.Close() + _, err = io.Copy(tw, f) + return err + }) +} diff --git a/controller/internal/appexport/fabplan_test.go b/controller/internal/appexport/fabplan_test.go new file mode 100644 index 0000000..6d249e4 --- /dev/null +++ b/controller/internal/appexport/fabplan_test.go @@ -0,0 +1,265 @@ +package appexport + +import ( + "archive/tar" + "io" + "log" + "os" + "path/filepath" + "reflect" + "sort" + "testing" + + "gitea.dooplex.hu/admin/felhom-controller/internal/appbackup" +) + +// fabProv is a minimal ExportStackProvider for the plan tests: configurable classified binds + hddPath. +type fabProv struct { + *rtProvider + hddPath string + binds []appbackup.ClassifiedBind + has bool + mounts []string +} + +func (p *fabProv) GetStackHDDPath(string) string { return p.hddPath } +func (p *fabProv) GetStackHDDMounts(string) []string { return p.mounts } +func (p *fabProv) GetStackClassifiedBinds(string) ([]appbackup.ClassifiedBind, bool) { + return p.binds, p.has +} + +func newFabExporter(binds []appbackup.ClassifiedBind, has bool, hddPath string, tree map[string][]string) *Exporter { + e := NewExporter(&fabProv{rtProvider: &rtProvider{}, hddPath: hddPath, binds: binds, has: has}, log.New(io.Discard, "", 0), "test") + e.dirLister = func(dir string) []string { return tree[dir] } + return e +} + +func mHDD(rel string) appbackup.ClassifiedBind { + return appbackup.ClassifiedBind{ComposeBind: appbackup.ComposeBind{Root: appbackup.RootHDD, RelPath: rel}, Class: appbackup.ClassMandatory} +} +func mUD(rel string) appbackup.ClassifiedBind { + return appbackup.ClassifiedBind{ComposeBind: appbackup.ComposeBind{Root: appbackup.RootUserdata, RelPath: rel}, Class: appbackup.ClassMandatory} +} +func oUD(rel string) appbackup.ClassifiedBind { + return appbackup.ClassifiedBind{ComposeBind: appbackup.ComposeBind{Root: appbackup.RootUserdata, RelPath: rel}, Class: appbackup.ClassOptional} +} +func xUD(rel string) appbackup.ClassifiedBind { + return appbackup.ClassifiedBind{ComposeBind: appbackup.ComposeBind{Root: appbackup.RootUserdata, RelPath: rel}, Class: appbackup.ClassExcluded} +} + +const hp = "/srv/data" // hddPath (plan is string-only; the lister is injected) + +// udDir builds an OS-native userdata dir path (matches appbackup.UserdataDir + the walk's +// filepath.Join, so injected-lister keys line up on any host). +func udDir(rel ...string) string { + return filepath.Join(append([]string{hp, "userdata"}, rel...)...) +} + +// A — legacy app: empty plan (byte-identical v0.130.0 capture). +func TestFabPlan_LegacyEmpty(t *testing.T) { + e := newFabExporter(nil, false, hp, nil) + mounts := []string{hp + "/appdata/app", hp + "/userdata"} + plan := e.computeFabPlan(ExportRequest{StackName: "x"}, mounts) + if len(plan.SkipMounts) != 0 || plan.SkipUserdataTar || plan.UserdataExcludeRels != nil { + t.Errorf("legacy app must yield an EMPTY plan (every mount kept, no excludes), got %+v", plan) + } +} + +// B — all-excluded app: the userdata root tar is skipped entirely. +func TestFabPlan_AllExcludedSkipsUserdataTar(t *testing.T) { + binds := []appbackup.ClassifiedBind{xUD("media/movies"), xUD("downloads")} + e := newFabExporter(binds, true, hp, map[string][]string{"/srv/data/userdata": {"media", "downloads"}}) + plan := e.computeFabPlan(ExportRequest{StackName: "radarr"}, []string{hp + "/userdata"}) + if !plan.SkipUserdataTar { + t.Error("no selected userdata bind → the whole root tar must be skipped (Scenario B)") + } +} + +// C — selected-complement: media/books kept, siblings excluded (R1-C). +func TestFabPlan_SelectedComplement(t *testing.T) { + binds := []appbackup.ClassifiedBind{mUD("media/books"), xUD("media/movies")} + tree := map[string][]string{ + udDir(): {"media", "music"}, + udDir("media"): {"books", "movies", "comics"}, + } + e := newFabExporter(binds, true, hp, tree) + plan := e.computeFabPlan(ExportRequest{StackName: "calibre-web"}, []string{hp + "/userdata"}) + if plan.SkipUserdataTar { + t.Fatal("a selected userdata bind exists — the root tar must NOT be skipped") + } + want := []string{"media/comics", "media/movies", "music"} + if got := plan.UserdataExcludeRels; !reflect.DeepEqual(got, want) { + t.Errorf("excludes = %v, want %v (media/books kept, siblings excluded)", got, want) + } +} + +// D — optional default-in / uncheck-out / excluded opt-in. +func TestFabPlan_OptionalAndOptIn(t *testing.T) { + // A mandatory anchor (data) keeps the root tar always produced, so comics/podcasts inclusion is + // exercised via the EXCLUDE LIST (not the whole-tar skip). + binds := []appbackup.ClassifiedBind{mUD("data"), oUD("media/comics"), xUD("media/podcasts")} + tree := map[string][]string{ + udDir(): {"data", "media"}, + udDir("media"): {"comics", "podcasts", "junk"}, + } + // D1 default: optional comics IN → not excluded; podcasts (excluded) + junk (unselected) excluded. + e := newFabExporter(binds, true, hp, tree) + p1 := e.computeFabPlan(ExportRequest{StackName: "komga"}, []string{hp + "/userdata"}) + if p1.SkipUserdataTar { + t.Fatal("mandatory anchor selected — tar must be produced") + } + if effExcluded(p1.UserdataExcludeRels, "media/comics") { + t.Error("D1: default → optional comics must be INCLUDED (not excluded)") + } + if !effExcluded(p1.UserdataExcludeRels, "media/podcasts") { + t.Error("D1: excluded podcasts must be excluded by default") + } + // D2 uncheck the optional → excluded (effectively, via a topmost exclude covering it). + p2 := e.computeFabPlan(ExportRequest{StackName: "komga", DeselectOptional: []string{"userdata/media/comics"}}, []string{hp + "/userdata"}) + if !effExcluded(p2.UserdataExcludeRels, "media/comics") { + t.Errorf("D2: unchecked optional must be excluded, excludes=%v", p2.UserdataExcludeRels) + } + // D3 opt-in the excluded → included. + p3 := e.computeFabPlan(ExportRequest{StackName: "komga", OptInExcluded: []string{"userdata/media/podcasts"}}, []string{hp + "/userdata"}) + if effExcluded(p3.UserdataExcludeRels, "media/podcasts") { + t.Error("D3: opted-in excluded must be INCLUDED (not excluded)") + } +} + +// effExcluded reports whether rel (or an ancestor of it) is in the topmost exclude list. +func effExcluded(excludes []string, rel string) bool { + for _, e := range excludes { + if rel == e || len(rel) > len(e) && rel[:len(e)+1] == e+"/" { + return true + } + } + return false +} + +// D floor — a request deselecting a MANDATORY path is IGNORED (mandatory stays in). +func TestFabPlan_MandatoryFloor(t *testing.T) { + binds := []appbackup.ClassifiedBind{mUD("media/books")} + tree := map[string][]string{udDir(): {"media"}, udDir("media"): {"books"}} + e := newFabExporter(binds, true, hp, tree) + // client tries to deselect the mandatory path — must be ignored (books NOT excluded). + plan := e.computeFabPlan(ExportRequest{StackName: "x", DeselectOptional: []string{"userdata/media/books"}}, []string{hp + "/userdata"}) + if contains(plan.UserdataExcludeRels, "media/books") || plan.SkipUserdataTar { + t.Errorf("mandatory floor breached — media/books must stay in the bundle; plan=%+v", plan) + } +} + +// §8 — an HDD mount matching NO classified bind is KEPT (fail toward capture). +func TestFabPlan_UnmatchedMountKept(t *testing.T) { + binds := []appbackup.ClassifiedBind{mHDD("appdata/known")} + e := newFabExporter(binds, true, hp, nil) + mounts := []string{hp + "/appdata/known", hp + "/appdata/mystery"} + plan := e.computeFabPlan(ExportRequest{StackName: "x"}, mounts) + if plan.SkipMounts[filepath.Clean(hp+"/appdata/mystery")] { + t.Error("an unmatched mount must be KEPT (fail toward capture, C6B-F1)") + } + if plan.SkipMounts[filepath.Clean(hp+"/appdata/known")] { + t.Error("a mandatory-matched mount must be kept") + } +} + +// §8 — a classified HDD mount that is NOT selected is skipped. +func TestFabPlan_UnselectedHDDMountSkipped(t *testing.T) { + binds := []appbackup.ClassifiedBind{xHDD("appdata/cache")} + e := newFabExporter(binds, true, hp, nil) + mounts := []string{hp + "/appdata/cache"} + plan := e.computeFabPlan(ExportRequest{StackName: "x"}, mounts) + if !plan.SkipMounts[filepath.Clean(hp+"/appdata/cache")] { + t.Error("an excluded, un-opted-in HDD mount must be skipped") + } +} + +func xHDD(rel string) appbackup.ClassifiedBind { + return appbackup.ClassifiedBind{ComposeBind: appbackup.ComposeBind{Root: appbackup.RootHDD, RelPath: rel}, Class: appbackup.ClassExcluded} +} + +// classifyFabRel keep-rule truth table (the R1-C core, pure). +func TestClassifyFabRel(t *testing.T) { + sel := []string{"media/books"} + cases := []struct { + rel string + want fabRelClass + }{ + {"media/books", fabKeepInside}, + {"media/books/covers", fabKeepInside}, + {"media", fabKeepAncestor}, + {"media/movies", fabStale}, + {"music", fabStale}, + } + for _, c := range cases { + if got := classifyFabRel(c.rel, sel); got != c.want { + t.Errorf("classifyFabRel(%q) = %d, want %d", c.rel, got, c.want) + } + } +} + +// tarDirectoryExcluding FS-level: excluded subtrees are absent, kept content present. +func TestTarDirectoryExcluding(t *testing.T) { + src := t.TempDir() + write := func(rel, content string) { + p := filepath.Join(src, filepath.FromSlash(rel)) + os.MkdirAll(filepath.Dir(p), 0755) + os.WriteFile(p, []byte(content), 0644) + } + write("media/books/a.epub", "BOOK") + write("media/movies/big.mkv", "MOVIE") + write("music/song.flac", "SONG") + + out := filepath.Join(t.TempDir(), "userdata.tar") + if err := tarDirectoryExcluding(src, out, []string{"media/movies", "music"}); err != nil { + t.Fatal(err) + } + got := tarEntries(t, out) + if !containsSuffix(got, "media/books/a.epub") { + t.Errorf("kept content missing: %v", got) + } + for _, bad := range []string{"media/movies/big.mkv", "music/song.flac", "media/movies", "music"} { + if containsSuffix(got, bad) { + t.Errorf("excluded path %q present in tar: %v", bad, got) + } + } +} + +func tarEntries(t *testing.T, tarPath string) []string { + t.Helper() + f, err := os.Open(tarPath) + if err != nil { + t.Fatal(err) + } + defer f.Close() + tr := tar.NewReader(f) + var names []string + for { + h, err := tr.Next() + if err == io.EOF { + break + } + if err != nil { + t.Fatal(err) + } + names = append(names, filepath.ToSlash(h.Name)) + } + sort.Strings(names) + return names +} + +func contains(ss []string, want string) bool { + for _, s := range ss { + if s == want { + return true + } + } + return false +} +func containsSuffix(ss []string, suffix string) bool { + for _, s := range ss { + if s == suffix || filepath.ToSlash(s) == suffix { + return true + } + } + return false +} diff --git a/controller/internal/appexport/provider.go b/controller/internal/appexport/provider.go index 47968b1..82f6af0 100644 --- a/controller/internal/appexport/provider.go +++ b/controller/internal/appexport/provider.go @@ -4,6 +4,8 @@ // the app to its current state. package appexport +import "gitea.dooplex.hu/admin/felhom-controller/internal/appbackup" + // ExportStackProvider provides stack data without circular imports. // Implemented by exportAdapter in main.go (same pattern as backup.StackDataProvider). type ExportStackProvider interface { @@ -15,6 +17,10 @@ type ExportStackProvider interface { GetStackHDDMounts(name string) []string // GetStackHDDPath returns the raw HDD_PATH env var from app.yaml. GetStackHDDPath(name string) string + // GetStackClassifiedBinds returns the app's backup-classified compose binds + whether it carries a + // (valid) backup block (Task 2). Drives the `.fab` class-scoped export plan (Task 4); a legacy app + // (false) exports the v0.130.0 full-root capture unchanged. + GetStackClassifiedBinds(name string) ([]appbackup.ClassifiedBind, bool) // IsStackRunning returns true if the stack has running containers. IsStackRunning(name string) bool // StopStack stops the stack via docker compose down. diff --git a/controller/internal/appexport/roundtrip_test.go b/controller/internal/appexport/roundtrip_test.go index f747953..0b49864 100644 --- a/controller/internal/appexport/roundtrip_test.go +++ b/controller/internal/appexport/roundtrip_test.go @@ -1,6 +1,7 @@ package appexport import ( + "gitea.dooplex.hu/admin/felhom-controller/internal/appbackup" "io" "log" "os" @@ -36,6 +37,9 @@ func (p *rtProvider) GetStackComposePath(string) (string, bool) { } func (p *rtProvider) GetStackHDDMounts(string) []string { return nil } func (p *rtProvider) GetStackHDDPath(string) string { return "" } +func (p *rtProvider) GetStackClassifiedBinds(string) ([]appbackup.ClassifiedBind, bool) { + return nil, false +} func (p *rtProvider) IsStackRunning(string) bool { return p.running } func (p *rtProvider) StopStack(string) error { p.stopped++; return nil } func (p *rtProvider) StartStack(string) error { p.started = true; return nil } diff --git a/controller/internal/web/fab_export_test.go b/controller/internal/web/fab_export_test.go new file mode 100644 index 0000000..d8eda50 --- /dev/null +++ b/controller/internal/web/fab_export_test.go @@ -0,0 +1,208 @@ +package web + +import ( + "archive/tar" + "compress/gzip" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-controller/internal/appbackup" + "gitea.dooplex.hu/admin/felhom-controller/internal/appexport" + "gitea.dooplex.hu/admin/felhom-controller/internal/settings" +) + +// fabWebProvider is a minimal appexport.ExportStackProvider: a config+userdata app (no volumes, no DB) +// so a real export runs without docker, and classified so the plan applies. +type fabWebProvider struct { + stackDir, stacksDir, hddPath string + binds []appbackup.ClassifiedBind +} + +func (p *fabWebProvider) GetStackDir(string) (string, bool) { return p.stackDir, true } +func (p *fabWebProvider) GetStackComposePath(string) (string, bool) { + return filepath.Join(p.stackDir, "docker-compose.yml"), true +} +func (p *fabWebProvider) GetStackHDDMounts(string) []string { return []string{appbackup.UserdataDir(p.hddPath)} } +func (p *fabWebProvider) GetStackHDDPath(string) string { return p.hddPath } +func (p *fabWebProvider) GetStackClassifiedBinds(string) ([]appbackup.ClassifiedBind, bool) { + return p.binds, true +} +func (p *fabWebProvider) IsStackRunning(string) bool { return false } +func (p *fabWebProvider) StopStack(string) error { return nil } +func (p *fabWebProvider) StartStack(string) error { return nil } +func (p *fabWebProvider) GetStackDisplayName(n string) string { return n } +func (p *fabWebProvider) GetStackNeedsHDD(string) bool { return true } +func (p *fabWebProvider) GetDockerVolumes(string) []string { return nil } +func (p *fabWebProvider) IsStackDeployed(string) bool { return true } +func (p *fabWebProvider) GetDecryptedEnv(string) map[string]string { return nil } +func (p *fabWebProvider) GetStacksBaseDir() string { return p.stacksDir } +func (p *fabWebProvider) SaveEncryptedAppConfig(string, map[string]string) error { return nil } +func (p *fabWebProvider) RefreshStacks() error { return nil } +func (p *fabWebProvider) RemoveStackVolumes(string) error { return nil } + +// waitExportDone polls the exporter until the active job finishes. +func waitExportDone(t *testing.T, e *appexport.Exporter) { + t.Helper() + deadline := time.Now().Add(30 * time.Second) + for time.Now().Before(deadline) { + if j := e.GetActiveJob(); j != nil { + if done, _ := j.Snapshot()["done"].(bool); done { + return + } + } + time.Sleep(50 * time.Millisecond) + } + t.Fatal("export did not finish") +} + +// findFab returns the newest .fab under dir. +func findFab(t *testing.T, dir string) string { + t.Helper() + var found string + filepath.Walk(dir, func(p string, info os.FileInfo, err error) error { + if err == nil && strings.HasSuffix(p, ".fab") { + found = p + } + return nil + }) + if found == "" { + t.Fatalf("no .fab under %s", dir) + } + return found +} + +// userdataTarEntries extracts data/hdd/userdata.tar from an UNENCRYPTED .fab and lists its entries. +func userdataTarEntries(t *testing.T, fabPath string) []string { + t.Helper() + f, err := os.Open(fabPath) + if err != nil { + t.Fatal(err) + } + defer f.Close() + gz, err := gzip.NewReader(f) + if err != nil { + t.Fatal(err) + } + tr := tar.NewReader(gz) + var inner []byte + for { + h, err := tr.Next() + if err == io.EOF { + break + } + if err != nil { + t.Fatal(err) + } + if strings.HasSuffix(filepath.ToSlash(h.Name), "hdd/userdata.tar") { + inner, _ = io.ReadAll(tr) + } + } + if inner == nil { + return nil // no userdata tar in the bundle + } + itr := tar.NewReader(strings.NewReader(string(inner))) + var names []string + for { + h, err := itr.Next() + if err == io.EOF { + break + } + if err != nil { + t.Fatal(err) + } + names = append(names, filepath.ToSlash(h.Name)) + } + return names +} + +// §7-F / red-proof F: BOTH start pipelines must carry the class selection. Each produces a bundle; +// with OptInExcluded, the excluded content must be present in the userdata tar. Red-proof: drop the +// selection fields from one start handler → that pipeline's assertion fails (two-call-site). +func TestFab_SelectionsRideBothStartPipelines(t *testing.T) { + build := func(t *testing.T) (*Server, *fabWebProvider, *appexport.Exporter, string) { + s := testServer(t) + s.cfg.Paths.DataDir = t.TempDir() + drive := t.TempDir() + stackDir := t.TempDir() + os.WriteFile(filepath.Join(stackDir, "docker-compose.yml"), []byte("services: {}\n"), 0644) + fabWrite(t, drive, "userdata/media/books/a.epub", "BOOK") + fabWrite(t, drive, "userdata/media/movies/big.mkv", "MOVIE") // excluded, opted-in below + prov := &fabWebProvider{stackDir: stackDir, stacksDir: t.TempDir(), hddPath: drive, + binds: []appbackup.ClassifiedBind{ + {ComposeBind: appbackup.ComposeBind{Root: appbackup.RootUserdata, RelPath: "media/books"}, Class: appbackup.ClassMandatory}, + {ComposeBind: appbackup.ComposeBind{Root: appbackup.RootUserdata, RelPath: "media/movies"}, Class: appbackup.ClassExcluded}, + }} + e := appexport.NewExporter(prov, s.logger, "test") + s.appExporter = e + if err := s.settings.AddStoragePath(settings.StoragePath{Path: drive, Label: "d", Schedulable: true}); err != nil { + t.Fatal(err) + } + return s, prov, e, drive + } + + post := func(s *Server, path, body string, h func(http.ResponseWriter, *http.Request)) { + rr := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodPost, path, strings.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + h(rr, req) + var resp map[string]interface{} + json.Unmarshal(rr.Body.Bytes(), &resp) + if resp["ok"] != true { + // jsonResponse uses {"ok":true}; some paths wrap in {"data":..}. Accept 200. + if rr.Code != http.StatusOK { + panic("start failed: " + rr.Body.String()) + } + } + } + + // Pipeline 1: /api/export/start (dest = the registered drive). + t.Run("api/export/start", func(t *testing.T) { + s, _, e, drive := build(t) + body, _ := json.Marshal(map[string]interface{}{"stack_name": "calibre-web", "dest_drive": drive, "opt_in_excluded": []string{"userdata/media/movies"}}) + post(s, "/api/export/start", string(body), s.apiExportStart) + waitExportDone(t, e) + got := userdataTarEntries(t, findFab(t, filepath.Join(drive, "exports"))) + if !hasSuffixIn(got, "media/movies/big.mkv") { + t.Errorf("start pipeline dropped opt_in_excluded — movies absent: %v", got) + } + }) + + // Pipeline 2: /api/export/download/start (dest = the staging download dir). + t.Run("api/export/download/start", func(t *testing.T) { + s, _, e, _ := build(t) + body := `{"stack_name":"calibre-web","opt_in_excluded":["userdata/media/movies"]}` + post(s, "/api/export/download/start", body, s.apiExportDownloadStart) + waitExportDone(t, e) + got := userdataTarEntries(t, findFab(t, s.fabDownloadDir())) + if !hasSuffixIn(got, "media/movies/big.mkv") { + t.Errorf("download pipeline dropped opt_in_excluded — movies absent: %v", got) + } + }) +} + +func fabWrite(t *testing.T, root, rel, content string) { + t.Helper() + p := filepath.Join(root, filepath.FromSlash(rel)) + if err := os.MkdirAll(filepath.Dir(p), 0755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(content), 0644); err != nil { + t.Fatal(err) + } +} + +func hasSuffixIn(ss []string, suffix string) bool { + for _, s := range ss { + if strings.HasSuffix(filepath.ToSlash(s), suffix) { + return true + } + } + return false +} diff --git a/controller/internal/web/handler_export.go b/controller/internal/web/handler_export.go index 84ace83..16b1bb2 100644 --- a/controller/internal/web/handler_export.go +++ b/controller/internal/web/handler_export.go @@ -168,10 +168,12 @@ func (s *Server) apiExportStart(w http.ResponseWriter, r *http.Request) { } var req struct { - StackName string `json:"stack_name"` - DestDrive string `json:"dest_drive"` - Password string `json:"password"` - StopApp bool `json:"stop_app"` + StackName string `json:"stack_name"` + DestDrive string `json:"dest_drive"` + Password string `json:"password"` + StopApp bool `json:"stop_app"` + DeselectOptional []string `json:"deselect_optional"` // Task 4: `.fab` class selection + OptInExcluded []string `json:"opt_in_excluded"` } if err := json.NewDecoder(r.Body).Decode(&req); err != nil { s.logger.Printf("[DEBUG] [web] apiExportStart: invalid body: %v", err) @@ -199,10 +201,12 @@ func (s *Server) apiExportStart(w http.ResponseWriter, r *http.Request) { } err := s.appExporter.StartExport(appexport.ExportRequest{ - StackName: req.StackName, - DestDrive: req.DestDrive, - Password: req.Password, - StopApp: req.StopApp, + StackName: req.StackName, + DestDrive: req.DestDrive, + Password: req.Password, + StopApp: req.StopApp, + DeselectOptional: req.DeselectOptional, + OptInExcluded: req.OptInExcluded, }) if err != nil { s.logger.Printf("[ERROR] [web] Export start failed for %s: %v", req.StackName, err) diff --git a/controller/internal/web/handler_export_download.go b/controller/internal/web/handler_export_download.go index fbb1741..04c5980 100644 --- a/controller/internal/web/handler_export_download.go +++ b/controller/internal/web/handler_export_download.go @@ -78,9 +78,11 @@ func (s *Server) apiExportDownloadStart(w http.ResponseWriter, r *http.Request) return } var req struct { - StackName string `json:"stack_name"` - Password string `json:"password"` - StopApp bool `json:"stop_app"` + StackName string `json:"stack_name"` + Password string `json:"password"` + StopApp bool `json:"stop_app"` + DeselectOptional []string `json:"deselect_optional"` // Task 4: `.fab` class selection (two-call-site) + OptInExcluded []string `json:"opt_in_excluded"` } if err := json.NewDecoder(r.Body).Decode(&req); err != nil { jsonError(w, "Invalid request body", http.StatusBadRequest) @@ -98,10 +100,12 @@ func (s *Server) apiExportDownloadStart(w http.ResponseWriter, r *http.Request) } // A concurrent export/import gets the exporter's own busy answer (single-flight). if err := s.appExporter.StartExport(appexport.ExportRequest{ - StackName: req.StackName, - DestDrive: s.fabDownloadRoot(), - Password: req.Password, - StopApp: req.StopApp, + StackName: req.StackName, + DestDrive: s.fabDownloadRoot(), + Password: req.Password, + StopApp: req.StopApp, + DeselectOptional: req.DeselectOptional, + OptInExcluded: req.OptInExcluded, }); err != nil { s.logger.Printf("[ERROR] [web] download-export start failed for %s: %v", req.StackName, err) jsonError(w, err.Error(), http.StatusConflict) diff --git a/controller/internal/web/templates/app_export.html b/controller/internal/web/templates/app_export.html index c5d8ed2..c9100af 100644 --- a/controller/internal/web/templates/app_export.html +++ b/controller/internal/web/templates/app_export.html @@ -37,6 +37,23 @@ + +

Jelszó (opcionális)

@@ -122,11 +139,84 @@ async function loadEstimate() { btn.disabled = false; } box.style.display = 'block'; + renderClassSelect(est); } catch(e) { console.error('Estimate error:', e); } } +// Task 4: the class-scoped selection UI (classified apps only; legacy apps see the plain estimate). +var lastEst = null; +function humanBytes(b) { + if (b >= 1073741824) return (b/1073741824).toFixed(1) + ' GB'; + if (b >= 1048576) return (b/1048576).toFixed(1) + ' MB'; + if (b >= 1024) return (b/1024).toFixed(1) + ' KB'; + return b + ' B'; +} +function renderClassSelect(est) { + lastEst = est; + var wrap = document.getElementById('classSelect'); + if (!est.has_classification) { wrap.style.display = 'none'; return; } + wrap.style.display = 'block'; + + var mand = est.mandatory_items || [], opt = est.optional_items || [], excl = est.excluded_items || []; + // Mandatory (locked). + var mb = document.getElementById('mandBox'); + if (mand.length) { + mb.style.display = 'block'; + document.getElementById('mandList').innerHTML = mand.map(function(i){ + return '
' + esc(i.rel_path) + ' (' + i.human + ')
'; + }).join(''); + } else { mb.style.display = 'none'; } + // Optional (pre-selected checkboxes). + var ob = document.getElementById('optBox'); + if (opt.length) { + ob.style.display = 'block'; + document.getElementById('optList').innerHTML = opt.map(function(i){ + return ''; + }).join(''); + } else { ob.style.display = 'none'; } + // Excluded (opt-in, collapsed, behind the two-number warning). + var eb = document.getElementById('exclBox'); + if (excl.length) { + eb.style.display = 'block'; + document.getElementById('exclList').innerHTML = excl.map(function(i){ + return ''; + }).join(''); + } else { eb.style.display = 'none'; } + recomputeTotal(); +} +function recomputeTotal() { + if (!lastEst) return; + var base = lastEst.base_bytes || 0; + var optSum = 0, exclSum = 0, exclAll = 0; + document.querySelectorAll('.fab-opt').forEach(function(cb){ if (cb.checked) optSum += parseInt(cb.dataset.bytes,10)||0; }); + document.querySelectorAll('.fab-excl').forEach(function(cb){ var b = parseInt(cb.dataset.bytes,10)||0; exclAll += b; if (cb.checked) exclSum += b; }); + var alap = base + optSum; + var full = alap + exclAll; + var s = document.getElementById('exclSummary'); + if ((lastEst.excluded_items||[]).length) { + s.innerHTML = 'Alap mentés: ~' + humanBytes(alap) + '. A kihagyott, nagy méretű tartalommal együtt: ~' + humanBytes(full) + '.'; + } +} +function toggleExcl() { + var l = document.getElementById('exclList'); + var shown = l.style.display !== 'none'; + l.style.display = shown ? 'none' : 'block'; + document.getElementById('exclToggle').textContent = shown ? 'Kihagyott tartalom megjelenítése' : 'Kihagyott tartalom elrejtése'; +} +function gatherSelections() { + var deselect = [], optIn = []; + document.querySelectorAll('.fab-opt').forEach(function(cb){ if (!cb.checked) deselect.push(cb.dataset.key); }); + document.querySelectorAll('.fab-excl').forEach(function(cb){ if (cb.checked) optIn.push(cb.dataset.key); }); + return { deselect_optional: deselect, opt_in_excluded: optIn }; +} +function esc(s) { return String(s).replace(/[&<>"]/g, function(c){ return {'&':'&','<':'<','>':'>','"':'"'}[c]; }); } + async function startExport() { var drive = document.getElementById('destDrive').value; var password = document.getElementById('exportPassword').value; @@ -137,6 +227,7 @@ async function startExport() { document.getElementById('doneCard').style.display = 'none'; try { + var sel = gatherSelections(); var resp = await fetch('/api/export/start', { method: 'POST', headers: csrfH(), @@ -144,7 +235,9 @@ async function startExport() { stack_name: stackName, dest_drive: drive, password: password, - stop_app: stopApp + stop_app: stopApp, + deselect_optional: sel.deselect_optional, + opt_in_excluded: sel.opt_in_excluded }) }); var data = await resp.json();