F-S2 + F-S3: compose-derived appdata dir resolution (v0.131.0)

The controller assumed an app's HDD appdata dir is always appdata/<stackName>.
paperless-ngx writes appdata/paperless (stack paperless-ngx), so every consumer
keying by stack name silently missed it via a stat-and-skip. One canonical
resolver appbackup.AppDataDirNames derives the real dir name(s) from the app's
compose ${HDD_PATH} binds; all consumers use it.

- F-S2 (tier-2): RunTier2 mirrors the resolved appdata/<name> (paperless docs
  got NO tier-2 copy before). Tier2Info size + RestoreTier2Files live dir use it.
  WARN when a declared appdata dir is absent. New tier2Mirror seam.
- F-S3 (migrate, NEW): all six per-app appdata legs (collision/size/copy/verify/
  cleanup/skip-set) now loop resolved names. scope="app" migration of paperless
  previously copied nothing and left an empty media dir (scope="all" was saved by
  the merge walk). WARN on missing declared dir in the copy leg.
- Multi-dir (N>1) refusal: tier-2 backup/info/restore refuse loudly (Hungarian);
  migrate supports N. No catalog app hits it today; lifted by Task 3.
- Display: storage page sums resolved dirs.
- Truth repair: the v0.130.0 "tier-2 copies the namespace wholesale" claim is
  false; corrected in CHANGELOG + main.go export-adapter comment.

+9 tests; red-proofs RP-1..RP-5 all confirmed. Controller-only, no agent/hub
coupling. Task 1 of the backup-classification-redesign arc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
This commit is contained in:
2026-07-14 17:45:53 +02:00
parent b42904bbab
commit 68f0e0cf5c
16 changed files with 765 additions and 44 deletions
+43
View File
@@ -1,5 +1,48 @@
## Changelog ## Changelog
### v0.131.0 — F-S2 + F-S3: compose-derived appdata dir resolution (paperless-ngx → appdata/paperless) (2026-07-14)
The controller assumed an app's HDD appdata dir is always `appdata/<stackName>`. paperless-ngx binds
`${HDD_PATH}/appdata/paperless/...` — stack `paperless-ngx`, dir `paperless` — so every consumer that
keyed by stack name silently missed it via a stat-and-skip. One canonical resolver
(`appbackup.AppDataDirNames`) now derives the real dir name(s) from the app's compose `${HDD_PATH}`
binds, and all consumers use it. Task 1 of the backup-classification-redesign arc
(`felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`), deliberately independent
of the classification schema.
- **F-S2 (spike-proven live) — tier-2 backup/info/restore.** `RunTier2` now mirrors the resolved
`appdata/<name>` dir, so paperless documents get their off-drive copy (previously: the appdata leg's
`os.Stat` gate skipped `appdata/paperless-ngx`, which never existed — **silent, no copy**).
`Tier2Info`'s size + the SSD-headroom guard use the resolved dir. `RestoreTier2Files` targets the
resolved live dir (was restoring into a wrong/empty `appdata/paperless-ngx`). A `[WARN]` now fires
when the compose DECLARES an appdata dir but it is absent on disk (the silence that hid F-S2). Every
rsync leg goes through a new `tier2Mirror` seam (prod behavior unchanged).
- **F-S3 (NEW, found this session) — scope="app" migration.** `migrate.go` keyed all six per-app
appdata legs (collision check, source-size, copy, verify, cleanup, skip-set) by stack name. For
**scope="app"** there is no merge walk, so migrating paperless-ngx copied nothing, "verified"
vacuously, flipped `HDD_PATH`, and the app came up with an **empty media dir**. (scope="all" was
saved by the merge walk — data safe, accounting off.) All six legs now loop the resolved name(s);
the copy leg WARNs on a missing declared dir.
- **Multi-dir refusal (defensive; no catalog app hits it today).** An app resolving to N>1 distinct
appdata dirs is refused loudly by tier-2 backup/info (honest `no_target` status +
`"az alkalmazáshoz több adatkönyvtár tartozik — a 2. mentés jelenleg alkalmazásonként egy könyvtárat
támogat"`) and tier-2 file-restore (refused BEFORE the app is stopped). Migrate supports N naturally.
This limitation is lifted by the tier-policy engine (Task 3).
- **Display.** The storage-detail page sums the resolved appdata dir(s), so paperless-ngx shows a
non-empty size.
- **Truth repair.** The v0.130.0 entry below states "The scheduled/tier-2 backup path was NOT affected
(it copies the felhom-data namespace wholesale)" — **that sentence is false** and is left in place
only as the historical record it corrects here: tier-2 copies the recovery unit + the resolved
`appdata/<name>` dir(s) ONLY (never the userdata tree — F-S1, unaddressed here — and never the
namespace wholesale). The `main.go` export-adapter comment that repeated the claim is fixed in code.
Scope guards: destination layout unchanged (`<destBase>/appdata` stays flat); no userdata copying at
any tier (F-S1 is the classification redesign's, not this task's); `ExportDataMounts` / `.fab` /
offbox untouched. Tests: +9 (resolver table incl. dedupe/foreign-drive/whole-root; RunTier2
paperless/legacy/multi-dir; Tier2Info + restore refusals + resolved live dir; scope="app" paperless
migration). Red-proofs RP-1..RP-5 all confirmed (resolver, RunTier2 leg, restore dst, migrate copy
leg, N>1 guard). Controller-only; no agent/hub coupling; MinAgent unchanged.
### v0.130.0 — CRITICAL C6B-F1: hollow .fab export (three compounding defects) + C6B-F2 share-removal guard (2026-07-14) ### v0.130.0 — CRITICAL C6B-F1: hollow .fab export (three compounding defects) + C6B-F2 share-removal guard (2026-07-14)
CAMPAIGN-6B surfaced that `.fab` export produced a **config-only, data-free bundle** for 12/13 CAMPAIGN-6B surfaced that `.fab` export produced a **config-only, data-free bundle** for 12/13
+22 -1
View File
@@ -7,7 +7,28 @@
> >
> Ask Claude Code: "Please update CONTEXT.md with what we did today" > Ask Claude Code: "Please update CONTEXT.md with what we did today"
Last updated: 2026-07-14 (v0.130.0 — CRITICAL C6B-F1 hollow-.fab-export fix + C6B-F2 share-removal guard) Last updated: 2026-07-14 (v0.131.0 — F-S2 + F-S3 compose-derived appdata dir resolution)
> **2026-07-14 — v0.131.0: F-S2 + F-S3 (compose-derived appdata dir resolution).** Task 1 of the
> backup-classification-redesign arc (spike: `felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md`).
> The controller assumed `appdata/<stackName>`; paperless-ngx writes `appdata/paperless` (stack
> `paperless-ngx`). ONE canonical resolver `appbackup.AppDataDirNames(hddPath, stackName, mounts)`
> derives the real dir name(s) from compose `${HDD_PATH}` binds (deduped/sorted; fallback `[stackName]`);
> all consumers use it. **F-S2** (spike-proven): `RunTier2`/`Tier2Info`/`RestoreTier2Files` now hit the
> resolved dir (paperless documents got NO tier-2 copy before — the appdata leg stat-skipped a dir that
> never existed). **F-S3 (NEW, found this session):** `migrate.go` keyed all six per-app appdata legs by
> stack name; **scope="app"** has no merge walk, so migrating paperless-ngx copied nothing, verified
> vacuously, flipped HDD_PATH → **empty media dir** (scope="all" was saved by the merge walk — data safe,
> accounting off). All six legs now loop resolved names. **Multi-dir (N>1) refusal** is defensive (no
> catalog app hits it today: immich/nextcloud/romm match, paperless mismatches, each app = exactly ONE
> dir): tier-2 backup/info/restore refuse loudly (Hungarian); **migrate supports N naturally**. This
> limitation is **deferred to Task 3 (tier-policy engine)**, which owns the destination layout. Storage
> page sums resolved dirs. Truth repairs: the v0.130.0 CHANGELOG/CONTEXT "tier-2 copies the namespace
> wholesale" claim is FALSE — corrected in the v0.131.0 CHANGELOG entry + `main.go` export-adapter
> comment; tier-2 copies the recovery unit + resolved `appdata/<name>` ONLY (NOT userdata — F-S1,
> unaddressed here). New seam `tier2Mirror`; `migSeams.resolveNames`. +9 tests, RP-1..RP-5 all
> confirmed. Controller-only, no agent/hub coupling. **NOT live-validated here:** scope="app" migration
> of a real app between drives (F-S3 live proof — supervised leg, Viktor's session).
> **2026-07-14 — v0.130.0: CRITICAL C6B-F1 (hollow .fab export) + C6B-F2 (share-removal guard).** > **2026-07-14 — v0.130.0: CRITICAL C6B-F1 (hollow .fab export) + C6B-F2 (share-removal guard).**
> CAMPAIGN-6B proved `.fab` export shipped **config-only, data-free bundles** for 12/13 `needs_hdd` > CAMPAIGN-6B proved `.fab` export shipped **config-only, data-free bundles** for 12/13 `needs_hdd`
+4 -1
View File
@@ -12,7 +12,8 @@
|---|---|---|---|---| |---|---|---|---|---|
| `NamespaceRoot` | controller/internal/appbackup/paths.go | `(drivePath string, inGuestDrive bool) string` | Resolve felhom-data root for a drive | `inGuestDrive=true` returns path AS-IS (Model A: guest mount IS the ns root); false appends `felhom-data`. Never double-nest | | `NamespaceRoot` | controller/internal/appbackup/paths.go | `(drivePath string, inGuestDrive bool) string` | Resolve felhom-data root for a drive | `inGuestDrive=true` returns path AS-IS (Model A: guest mount IS the ns root); false appends `felhom-data`. Never double-nest |
| `PrimaryBackupPath` / `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath` | controller/internal/appbackup/paths.go | `(nsRoot[, stackName]) string` | All backup dir layout | Take the NAMESPACE ROOT, not a bare drive path | | `PrimaryBackupPath` / `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath` | controller/internal/appbackup/paths.go | `(nsRoot[, stackName]) string` | All backup dir layout | Take the NAMESPACE ROOT, not a bare drive path |
| `AppDBDumpPath` / `AppVolumeDumpPath` / `AppDataDir` | controller/internal/appbackup/paths.go | `(nsRoot, stackName) string` | Per-app dump/data dirs | Same nsRoot contract | | `AppDBDumpPath` / `AppVolumeDumpPath` / `AppDataDir` | controller/internal/appbackup/paths.go | `(nsRoot, stackName) string` | Per-app dump/data dirs | Same nsRoot contract. `AppDataDir`'s final segment is the app's real appdata dir NAME — NOT always the stack name (paperless-ngx → `paperless`); resolve via `AppDataDirNames` first (F-S2/F-S3) |
| `AppDataDirNames` / `AppDataBindsPresent` | controller/internal/appbackup/paths.go | `(hddPath, stackName string, hddMounts []string) []string` / `(hddPath, hddMounts) bool` | Resolve the real `appdata/<name>` dir(s) from compose `${HDD_PATH}` binds (F-S2/F-S3) | `hddMounts` = ParseComposeHDDMounts shape. Deduped+sorted; falls back to `[stackName]` when no appdata bind. Tier-2 (`backup.Manager.tier2AppDataName`) refuses N>1; migrate (`stacks.Manager.ResolveAppDataDirNames`) loops N. `BindsPresent` drives the WARN-on-missing-declared-dir |
| `UserdataDir` / `EnsureUserdataSkeleton` / `EnsureDirOwned` | controller/internal/appbackup/userdata.go | `(nsRoot)` / `(path, gid int)` | userdata/ tree w/ 2775 setgid gid-1000 convention | Linux-only chown via build-tag twin userdata_linux.go | | `UserdataDir` / `EnsureUserdataSkeleton` / `EnsureDirOwned` | controller/internal/appbackup/userdata.go | `(nsRoot)` / `(path, gid int)` | userdata/ tree w/ 2775 setgid gid-1000 convention | Linux-only chown via build-tag twin userdata_linux.go |
| `HumanizeBytes` | controller/internal/appbackup/appdata.go | `(b int64) string` | Human byte sizes | Exported canonical; private clones exist (§6) | | `HumanizeBytes` | controller/internal/appbackup/appdata.go | `(b int64) string` | Human byte sizes | Exported canonical; private clones exist (§6) |
| `stablePathForName` / `agentWhere` | controller/internal/web/intermediary.go | `(name/registeredPath) string` | Map registry stable path `/mnt/felhom-drives/<n>` ↔ raw agent mount | Registry stores STABLE path; agent ops take the RAW mount — always convert | | `stablePathForName` / `agentWhere` | controller/internal/web/intermediary.go | `(name/registeredPath) string` | Map registry stable path `/mnt/felhom-drives/<n>` ↔ raw agent mount | Registry stores STABLE path; agent ops take the RAW mount — always convert |
@@ -199,6 +200,8 @@
| `dumpVolumesSafe` (func seam) | controller/internal/backup/backup.go | nil → real `DumpAppVolumesSafe` | injected in controller/internal/backup/volume_dumps_test.go (gating tests without Docker) | | `dumpVolumesSafe` (func seam) | controller/internal/backup/backup.go | nil → real `DumpAppVolumesSafe` | injected in controller/internal/backup/volume_dumps_test.go (gating tests without Docker) |
| `generateSecret` (func seam) | controller/internal/backup/backup.go | `stacks.Manager.GenerateSecretForField` via `SetSecretGenerator` (main.go) | injected in controller/internal/backup/restore_secrets_gen_test.go | | `generateSecret` (func seam) | controller/internal/backup/backup.go | `stacks.Manager.GenerateSecretForField` via `SetSecretGenerator` (main.go) | injected in controller/internal/backup/restore_secrets_gen_test.go |
| `restoreFilesCopier` (func seam) | controller/internal/backup/backup.go | nil → real `rsyncRestoreMissing` | injected in controller/internal/backup/tier2_restore_test.go (orchestration without rsync) | | `restoreFilesCopier` (func seam) | controller/internal/backup/backup.go | nil → real `rsyncRestoreMissing` | injected in controller/internal/backup/tier2_restore_test.go (orchestration without rsync) |
| `tier2Mirror` (func seam) | controller/internal/backup/backup.go | nil → real `rsyncMirror` | both RunTier2 rsync legs; injected in controller/internal/backup/tier2_appdata_test.go (resolve→mirror without rsync) |
| `migSeams.resolveNames` (func seam) | controller/internal/stacks/migrate.go | nil → real `ResolveAppDataDirNames` (compose-derived) | injected in controller/internal/stacks/migrate_fs3_test.go (F-S3 appdata dir-name resolution) |
Cross-repo edges: Cross-repo edges:
- `controller/internal/agentapi/client.go`**felhom-agent** local API (`/storage`, `/disks*`, `/backup*`, `/netstorage*`, `/guest/*`): pinned leaf SHA-256 + per-guest bearer token from bootstrap.json. - `controller/internal/agentapi/client.go`**felhom-agent** local API (`/storage`, `/disks*`, `/backup*`, `/netstorage*`, `/guest/*`): pinned leaf SHA-256 + per-guest bearer token from bootstrap.json.
+11 -5
View File
@@ -476,7 +476,7 @@ Path computation is centralized in `backup/paths.go` via the `FelhomDataDir = "f
- `PrimaryResticRepoPath(drivePath)``<drive>/felhom-data/backups/primary/restic/` - `PrimaryResticRepoPath(drivePath)``<drive>/felhom-data/backups/primary/restic/`
- `AppDBDumpPath(drivePath, stackName)``<drive>/felhom-data/backups/primary/<stack>/db-dumps/` - `AppDBDumpPath(drivePath, stackName)``<drive>/felhom-data/backups/primary/<stack>/db-dumps/`
- `AppVolumeDumpPath(drivePath, stackName)``<drive>/felhom-data/backups/primary/<stack>/volume-dumps/` - `AppVolumeDumpPath(drivePath, stackName)``<drive>/felhom-data/backups/primary/<stack>/volume-dumps/`
- `AppDataDir(drivePath, stackName)``<drive>/felhom-data/appdata/<stack>/` - `AppDataDir(drivePath, name)``<drive>/felhom-data/appdata/<name>/` (final segment is the app's real appdata dir NAME, resolved via `AppDataDirNames` from compose binds — NOT always the stack name; F-S2)
- `SecondaryResticRepoPath(drivePath)``<drive>/felhom-data/backups/secondary/restic/` - `SecondaryResticRepoPath(drivePath)``<drive>/felhom-data/backups/secondary/restic/`
- `AppSecondaryRsyncPath(drivePath, stackName)``<drive>/felhom-data/backups/secondary/<stack>/rsync/` - `AppSecondaryRsyncPath(drivePath, stackName)``<drive>/felhom-data/backups/secondary/<stack>/rsync/`
- `SecondaryInfraPath(drivePath)``<drive>/felhom-data/backups/secondary/_infra/` - `SecondaryInfraPath(drivePath)``<drive>/felhom-data/backups/secondary/_infra/`
@@ -529,9 +529,14 @@ backups/primary/<app>/
#### Tier 2 — off-drive copy (Phase 3, v0.55.x) #### Tier 2 — off-drive copy (Phase 3, v0.55.x)
For every HDD app, Tier 2 (`internal/backup/tier2.go`) rsync-mirrors the recovery unit For every HDD app, Tier 2 (`internal/backup/tier2.go`) rsync-mirrors the recovery unit
(`backups/primary/<app>/`) + the app's `appdata/<app>/` to `<target>/backups/secondary/<app>/` on a (`backups/primary/<app>/`) + the app's resolved `appdata/<name>/` to `<target>/backups/secondary/<app>/`
**different physical disk** — the only off-drive protection bind-mounted HDD userdata can get (PBS can't on a **different physical disk** — the only off-drive protection bind-mounted HDD app data can get (PBS
reach bind mounts). Auto-targeted: **prefer another registered user-data drive** (off-disk via can't reach bind mounts). **The appdata dir NAME is derived from the app's compose `${HDD_PATH}` binds,
not assumed to be the stack name** (F-S2, v0.131.0: paperless-ngx writes `appdata/paperless`;
`tier2AppDataName``appbackup.AppDataDirNames`); an app resolving to >1 distinct appdata dir is
refused loudly. This copies the recovery unit + `appdata/<name>` ONLY — **not** the browsable
`userdata/` tree (F-S1, owned by the backup-classification redesign) and **not** the namespace
wholesale. Auto-targeted: **prefer another registered user-data drive** (off-disk via
`system.SamePhysicalDevice`); else the **internal SSD for small units only**, behind a size-aware `system.SamePhysicalDevice`); else the **internal SSD for small units only**, behind a size-aware
**rootfs-headroom guard** (`tier2FitsHeadroom`) that **refuses rather than fills** the ~8 GB guest rootfs **rootfs-headroom guard** (`tier2FitsHeadroom`) that **refuses rather than fills** the ~8 GB guest rootfs
(reserve = `max(2 GB, 20%)`), recording an honest "needs a 2nd HDD" status. Status persists via (reserve = `max(2 GB, 20%)`), recording an honest "needs a 2nd HDD" status. Status persists via
@@ -541,7 +546,8 @@ reach bind mounts). Auto-targeted: **prefer another registered user-data drive**
**In-place file restore from the Tier-2 copy (C2, v0.100.0 — closes drill finding F2)** — **In-place file restore from the Tier-2 copy (C2, v0.100.0 — closes drill finding F2)** —
`POST /backup/tier2/restore` (`backup.RestoreTier2Files`, `internal/backup/tier2_restore.go`) + the `POST /backup/tier2/restore` (`backup.RestoreTier2Files`, `internal/backup/tier2_restore.go`) + the
**"Fájlok visszaállítása"** button on the healthy Tier-2 layer row. **Additive-only** semantics **"Fájlok visszaállítása"** button on the healthy Tier-2 layer row. **Additive-only** semantics
(`rsyncRestoreMissing`: `rsync -a --ignore-existing`): files missing from the live `appdata/<stack>` (`rsyncRestoreMissing`: `rsync -a --ignore-existing`): files missing from the live resolved
`appdata/<name>` dir (F-S2 — compose-derived, not the stack name)
are copied back from the RECORDED Tier-2 copy; existing live files are **never overwritten** (a are copied back from the RECORDED Tier-2 copy; existing live files are **never overwritten** (a
customer edit after the last copy wins) and **nothing is ever deleted** — this exactly serves the customer edit after the last copy wins) and **nothing is ever deleted** — this exactly serves the
"I deleted my files" scenario with zero risk to newer data. Source = the recorded "I deleted my files" scenario with zero risk to newer data. Source = the recorded
+5 -2
View File
@@ -1357,8 +1357,11 @@ func (a *exportAdapter) GetStackHDDMounts(name string) []string {
// C6B-F1 (v0.130.0): union ${HDD_PATH} binds + the ${USERDATA_PATH} root. The old // C6B-F1 (v0.130.0): union ${HDD_PATH} binds + the ${USERDATA_PATH} root. The old
// ParseComposeHDDMounts-only call was blind to the standard userdata convention, so // ParseComposeHDDMounts-only call was blind to the standard userdata convention, so
// 12/13 needs_hdd catalog apps exported hollow (config-only) bundles. The backup-side // 12/13 needs_hdd catalog apps exported hollow (config-only) bundles. The backup-side
// stackAdapter is intentionally NOT changed — the scheduled/tier-2 path copies the // stackAdapter is intentionally NOT changed here — the scheduled/tier-2 path copies the
// felhom-data namespace wholesale and its mount list is size/inventory display only. // recovery unit + the app's resolved appdata/<name> dir(s) only (NOT the userdata tree —
// F-S1; NOT the namespace wholesale), and derives that dir name from the compose binds
// (F-S2, see backup.tier2AppDataName). Spec:
// felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md.
return stacks.ExportDataMounts(s.ComposePath, appCfg.Env["HDD_PATH"]) return stacks.ExportDataMounts(s.ComposePath, appCfg.Env["HDD_PATH"])
} }
return nil return nil
@@ -0,0 +1,104 @@
package appbackup
import (
"path/filepath"
"reflect"
"testing"
)
// fp joins the elements under an HDD path with OS separators — mounts in the ParseComposeHDDMounts
// shape are already filepath.Clean'd, so tests build them the same way.
func fp(elems ...string) string { return filepath.Join(elems...) }
// TestAppDataDirNames is the pure derivation table (Group A). Every case asserts the RESOLVED name
// list, never mere absence of error. Companion RP-1: a resolver that ignores mounts and returns
// []string{stackName} fails the paperless, two-name, and dedupe cases.
func TestAppDataDirNames(t *testing.T) {
const hdd = "/mnt/felhom-usb"
cases := []struct {
name string
stack string
mounts []string
want []string
}{
{
// paperless shape: stack "paperless-ngx", dir "paperless" (F-S2/F-S3 core).
name: "paperless mismatch",
stack: "paperless-ngx",
mounts: []string{
fp(hdd, "appdata", "paperless", "media"),
fp(hdd, "appdata", "paperless", "export"),
},
want: []string{"paperless"}, // media+export dedupe to one name
},
{
// match shape: dir name == stack name (immich/nextcloud/romm).
name: "matching name",
stack: "nextcloud",
mounts: []string{fp(hdd, "appdata", "nextcloud")},
want: []string{"nextcloud"},
},
{
// two DISTINCT names → both, sorted (no catalog app does this today).
name: "two distinct names sorted",
stack: "weird",
mounts: []string{
fp(hdd, "appdata", "zebra", "x"),
fp(hdd, "appdata", "alpha", "y"),
},
want: []string{"alpha", "zebra"},
},
{
// non-appdata HDD binds + a foreign-drive mount are filtered → fallback.
name: "non-appdata and foreign filtered",
stack: "romm",
mounts: []string{
fp(hdd, "roms"), // under HDD but not appdata/
fp("/mnt/other-drive", "appdata", "ghost"), // foreign drive — wrong prefix
},
want: []string{"romm"},
},
{
// whole-appdata-root bind (no name derivable) → ignored → fallback.
name: "whole appdata root bind",
stack: "root-binder",
mounts: []string{fp(hdd, "appdata")},
want: []string{"root-binder"},
},
{
name: "empty mounts fallback",
stack: "vaultwarden",
mounts: nil,
want: []string{"vaultwarden"},
},
{
// unclean paths still resolve (Clean applied both sides).
name: "unclean path",
stack: "paperless-ngx",
mounts: []string{hdd + "/appdata/paperless/../paperless/media"},
want: []string{"paperless"},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := AppDataDirNames(hdd, tc.stack, tc.mounts)
if !reflect.DeepEqual(got, tc.want) {
t.Errorf("AppDataDirNames(%q, %q, %v) = %v, want %v", hdd, tc.stack, tc.mounts, got, tc.want)
}
})
}
}
// TestAppDataBindsPresent pins the WARN predicate: true only when a mount sits under appdata/.
func TestAppDataBindsPresent(t *testing.T) {
const hdd = "/mnt/felhom-usb"
if !AppDataBindsPresent(hdd, []string{fp(hdd, "appdata", "paperless", "media")}) {
t.Error("declared appdata bind should report present")
}
if AppDataBindsPresent(hdd, []string{fp(hdd, "roms")}) {
t.Error("non-appdata bind should NOT report present")
}
if AppDataBindsPresent(hdd, nil) {
t.Error("no mounts should NOT report present")
}
}
+60 -2
View File
@@ -5,7 +5,11 @@
// cross-drive, or drive-mount code in the backup package. // cross-drive, or drive-mount code in the backup package.
package appbackup package appbackup
import "path/filepath" import (
"path/filepath"
"sort"
"strings"
)
// FelhomDataDir is the namespace directory on storage drives for all felhom-managed data. // FelhomDataDir is the namespace directory on storage drives for all felhom-managed data.
const FelhomDataDir = "felhom-data" const FelhomDataDir = "felhom-data"
@@ -64,7 +68,61 @@ func AppVolumeDumpPath(nsRoot, stackName string) string {
return filepath.Join(RecoveryUnitPath(nsRoot, stackName), "volume-dumps") return filepath.Join(RecoveryUnitPath(nsRoot, stackName), "volume-dumps")
} }
// AppDataDir returns the app data directory under a felhom-data namespace root. // AppDataDir returns the app data directory under a felhom-data namespace root. The final segment
// is the app's real appdata dir NAME — usually the stack name, but NOT always: paperless-ngx writes
// appdata/paperless (F-S2/F-S3). Callers that key by stack name silently miss such apps; use
// AppDataDirNames to resolve the real name(s) from the app's compose binds and pass them here.
func AppDataDir(nsRoot, stackName string) string { func AppDataDir(nsRoot, stackName string) string {
return filepath.Join(nsRoot, "appdata", stackName) return filepath.Join(nsRoot, "appdata", stackName)
} }
// AppDataDirNames returns the app's real directory name(s) under <hddPath>/appdata, derived from its
// compose HDD bind mounts (F-S2/F-S3: the dir name is NOT always the stack name — paperless-ngx
// writes appdata/paperless). hddMounts are resolved host paths in the ParseComposeHDDMounts shape
// (each is <hddPath> itself or a subpath, filepath.Clean'd). The first path element under
// <hddPath>/appdata/ is taken as the dir name; results are deduped and sorted. Falls back to
// []string{stackName} when no appdata-prefixed mount is derivable (no HDD appdata binds, unreadable
// compose, nil provider) — the exact legacy behavior.
//
// Today every catalog app resolves to exactly ONE name (immich→immich, nextcloud→nextcloud,
// romm→romm, paperless-ngx→paperless). The N>1 return is defensive: tier-2 refuses it loudly,
// migrate handles it naturally.
func AppDataDirNames(hddPath, stackName string, hddMounts []string) []string {
prefix := filepath.Clean(hddPath) + string(filepath.Separator) + "appdata" + string(filepath.Separator)
seen := make(map[string]bool)
var names []string
for _, mnt := range hddMounts {
cm := filepath.Clean(mnt)
if !strings.HasPrefix(cm, prefix) {
continue // not under appdata/ (a whole-root bind, a different subtree, a foreign drive)
}
rem := strings.TrimPrefix(cm, prefix)
first := strings.Split(rem, string(filepath.Separator))[0]
if first == "" {
continue
}
if !seen[first] {
seen[first] = true
names = append(names, first)
}
}
if len(names) == 0 {
return []string{stackName}
}
sort.Strings(names)
return names
}
// AppDataBindsPresent reports whether any of the app's resolved HDD mounts sits under
// <hddPath>/appdata/ — i.e. the compose actually DECLARES an appdata bind. Callers use it to
// distinguish "no appdata to back up" (silent skip is correct) from "declared appdata dir missing
// on disk" (the silence that hid F-S2 — worth a WARN). Same prefix rule as AppDataDirNames.
func AppDataBindsPresent(hddPath string, hddMounts []string) bool {
prefix := filepath.Clean(hddPath) + string(filepath.Separator) + "appdata" + string(filepath.Separator)
for _, mnt := range hddMounts {
if strings.HasPrefix(filepath.Clean(mnt), prefix) {
return true
}
}
return false
}
@@ -133,3 +133,11 @@ func RecoveryUnitManifestPath(nsRoot, stackName string) string {
func AppDataDir(nsRoot, stackName string) string { func AppDataDir(nsRoot, stackName string) string {
return appbackup.AppDataDir(nsRoot, stackName) return appbackup.AppDataDir(nsRoot, stackName)
} }
func AppDataDirNames(hddPath, stackName string, hddMounts []string) []string {
return appbackup.AppDataDirNames(hddPath, stackName, hddMounts)
}
func AppDataBindsPresent(hddPath string, hddMounts []string) bool {
return appbackup.AppDataBindsPresent(hddPath, hddMounts)
}
+5
View File
@@ -65,6 +65,11 @@ type Manager struct {
// the orchestration never shells out. Nil → the real rsyncRestoreMissing (additive-only). // the orchestration never shells out. Nil → the real rsyncRestoreMissing (additive-only).
restoreFilesCopier func(src, dst string) (filesRestored int, err error) restoreFilesCopier func(src, dst string) (filesRestored int, err error)
// tier2Mirror (F-S2) — the Tier-2 backup mirror seam (both rsync legs in RunTier2), overridable
// so the resolve→mirror→record flow is unit-testable without rsync. Nil → the real rsyncMirror
// (`-a --delete`, contents-of-src semantics).
tier2Mirror func(src, dst string) error
// migrationRunning, if set, reports whether a data migration is in progress. The scheduled // migrationRunning, if set, reports whether a data migration is in progress. The scheduled
// backup paths skip when it returns true (Change 3 — backup ↔ migration mutual exclusion), so a // backup paths skip when it returns true (Change 3 — backup ↔ migration mutual exclusion), so a
// nightly dump/Tier-2 can't race a migration copy/cleanup on the same drive. // nightly dump/Tier-2 can't race a migration copy/cleanup on the same drive.
+75 -11
View File
@@ -14,21 +14,59 @@ import (
"gitea.dooplex.hu/admin/felhom-controller/internal/system" "gitea.dooplex.hu/admin/felhom-controller/internal/system"
) )
// Tier 2 = an off-drive (different physical disk) copy of an HDD app's recovery unit + bulk userdata. // Tier 2 = an off-drive (different physical disk) copy of an HDD app's recovery unit + its resolved
// It is the ONLY off-drive protection that browsable HDD userdata can get — PBS can't reach bind // appdata/<name> dir(s). It does NOT copy the browsable userdata tree (F-S1: userdata is not backed
// mounts. Auto-enabled for every HDD app; the target is auto-picked: prefer another registered // up at any tier yet — that gap is owned by the classification redesign, see
// user-data drive (can hold bulk), else the internal SSD for SMALL units only — and the SSD is the // felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md). The appdata dir NAME is
// guest rootfs (~8 GB), so we REFUSE rather than fill it (a size-aware headroom guard). When no // derived from the app's compose binds, NOT assumed to be the stack name (F-S2: paperless-ngx writes
// off-drive target fits, we record an honest "needs a 2nd HDD" status instead of silently doing // appdata/paperless); see tier2AppDataName. Auto-enabled for every HDD app; the target is auto-picked:
// nothing useful. // prefer another registered user-data drive (can hold bulk), else the internal SSD for SMALL units
// only — and the SSD is the guest rootfs (~8 GB), so we REFUSE rather than fill it (a size-aware
// headroom guard). When no off-drive target fits, we record an honest "needs a 2nd HDD" status
// instead of silently doing nothing useful.
const gibibyte = 1024 * 1024 * 1024 const gibibyte = 1024 * 1024 * 1024
var ( var (
errNoOffDiskTarget = errors.New("no off-drive target (single drive, app already on the system disk)") errNoOffDiskTarget = errors.New("no off-drive target (single drive, app already on the system disk)")
errSSDNoHeadroom = errors.New("the internal SSD lacks headroom for this app's data — a 2nd drive is required for off-drive backup") errSSDNoHeadroom = errors.New("the internal SSD lacks headroom for this app's data — a 2nd drive is required for off-drive backup")
// errTier2MultiDir is raised when an app's compose resolves to MORE THAN ONE distinct appdata
// dir under <hddPath>/appdata (no catalog app does today). Tier 2's destination layout is flat
// (<destBase>/appdata), so it refuses rather than silently collapse two source dirs into one.
errTier2MultiDir = errors.New("az alkalmazáshoz több adatkönyvtár tartozik — a 2. mentés jelenleg alkalmazásonként egy könyvtárat támogat")
) )
// appDataDirNames resolves the app's real appdata dir name(s) under hddPath from its compose HDD
// binds, via the stack provider (nil provider → legacy [stackName] fallback). See
// appbackup.AppDataDirNames.
func (m *Manager) appDataDirNames(stackName, hddPath string) []string {
var mounts []string
if m.stackProvider != nil {
mounts = m.stackProvider.GetStackHDDMounts(stackName)
}
return AppDataDirNames(hddPath, stackName, mounts)
}
// tier2AppDataName resolves the SINGLE appdata dir name for tier-2's flat destination. N>1 distinct
// names → errTier2MultiDir (the one place the tier-2 multi-dir refusal is built). It always returns
// at least one name from appDataDirNames' fallback, so name is meaningful only when err == nil.
func (m *Manager) tier2AppDataName(stackName, hddPath string) (string, error) {
names := m.appDataDirNames(stackName, hddPath)
if len(names) > 1 {
return "", errTier2MultiDir
}
return names[0], nil
}
// tier2AppDataBindsPresent reports whether the app's compose declares an appdata bind (drives the
// WARN-on-missing-declared-dir rule; nil provider → false).
func (m *Manager) tier2AppDataBindsPresent(stackName, hddPath string) bool {
if m.stackProvider == nil {
return false
}
return AppDataBindsPresent(hddPath, m.stackProvider.GetStackHDDMounts(stackName))
}
// Tier2Target is a resolved off-drive destination for an app's Tier 2 copy. // Tier2Target is a resolved off-drive destination for an app's Tier 2 copy.
type Tier2Target struct { type Tier2Target struct {
NamespaceRoot string // felhom-data namespace root on the target drive NamespaceRoot string // felhom-data namespace root on the target drive
@@ -143,7 +181,16 @@ func (m *Manager) RunTier2(stackName string) error {
} }
sourceNsRoot := m.namespaceRoot(sourceDrive) sourceNsRoot := m.namespaceRoot(sourceDrive)
unitDir := RecoveryUnitPath(sourceNsRoot, stackName) unitDir := RecoveryUnitPath(sourceNsRoot, stackName)
appDataDir := AppDataDir(sourceNsRoot, stackName) // F-S2: resolve the app's REAL appdata dir name from its compose binds (paperless-ngx writes
// appdata/paperless, not appdata/paperless-ngx). For an HDD app HDD_PATH == nsRoot (Model A), so
// the mounts (resolved against HDD_PATH) share the nsRoot prefix. N>1 distinct names → refuse.
appDataName, resErr := m.tier2AppDataName(stackName, sourceNsRoot)
if resErr != nil {
m.recordTier2NoTarget(stackName, resErr.Error())
m.logger.Printf("[ERROR] [backup] Tier 2 for %s refused: %v", stackName, resErr)
return nil
}
appDataDir := AppDataDir(sourceNsRoot, appDataName)
if _, err := os.Stat(unitDir); err != nil { if _, err := os.Stat(unitDir); err != nil {
return nil // no recovery unit yet — nothing to copy return nil // no recovery unit yet — nothing to copy
} }
@@ -166,7 +213,12 @@ func (m *Manager) RunTier2(stackName string) error {
destBase := filepath.Join(target.NamespaceRoot, "backups", "secondary", stackName) destBase := filepath.Join(target.NamespaceRoot, "backups", "secondary", stackName)
start := time.Now() start := time.Now()
if err := rsyncMirror(unitDir, filepath.Join(destBase, "recovery-unit")); err != nil { mirror := m.tier2Mirror
if mirror == nil {
mirror = rsyncMirror
}
if err := mirror(unitDir, filepath.Join(destBase, "recovery-unit")); err != nil {
m.recordTier2Failure(stackName, target, err) m.recordTier2Failure(stackName, target, err)
if m.tier2Notify != nil { if m.tier2Notify != nil {
m.tier2Notify(stackName, target.Label, time.Since(start), err) m.tier2Notify(stackName, target.Label, time.Since(start), err)
@@ -174,13 +226,18 @@ func (m *Manager) RunTier2(stackName string) error {
return fmt.Errorf("tier2 rsync unit for %s: %w", stackName, err) return fmt.Errorf("tier2 rsync unit for %s: %w", stackName, err)
} }
if _, e := os.Stat(appDataDir); e == nil { if _, e := os.Stat(appDataDir); e == nil {
if err := rsyncMirror(appDataDir, filepath.Join(destBase, "appdata")); err != nil { if err := mirror(appDataDir, filepath.Join(destBase, "appdata")); err != nil {
m.recordTier2Failure(stackName, target, err) m.recordTier2Failure(stackName, target, err)
if m.tier2Notify != nil { if m.tier2Notify != nil {
m.tier2Notify(stackName, target.Label, time.Since(start), err) m.tier2Notify(stackName, target.Label, time.Since(start), err)
} }
return fmt.Errorf("tier2 rsync appdata for %s: %w", stackName, err) return fmt.Errorf("tier2 rsync appdata for %s: %w", stackName, err)
} }
} else if m.tier2AppDataBindsPresent(stackName, sourceNsRoot) {
// F-S2: the compose DECLARES an appdata bind but the dir is missing on disk. Skipping is kept
// (nothing to copy) but the silence that hid F-S2 for months is now a loud WARN.
m.logger.Printf("[WARN] [backup] Tier 2 for %s: compose declares appdata dir %q but it is absent at %s — appdata leg skipped",
stackName, appDataName, appDataDir)
} }
dur := time.Since(start) dur := time.Since(start)
@@ -279,8 +336,15 @@ func (m *Manager) Tier2Info(stackName string) Tier2Info {
} }
// Resolve what the runner WOULD pick right now (real unit size feeds the SSD headroom guard). // Resolve what the runner WOULD pick right now (real unit size feeds the SSD headroom guard).
// F-S2: N>1 distinct appdata dirs → the same honest refusal the runner records.
sourceNsRoot := m.namespaceRoot(source) sourceNsRoot := m.namespaceRoot(source)
unitSize := dirSizeBytes(RecoveryUnitPath(sourceNsRoot, stackName)) + dirSizeBytes(AppDataDir(sourceNsRoot, stackName)) appDataName, resErr := m.tier2AppDataName(stackName, sourceNsRoot)
if resErr != nil {
info.NoTarget = true
info.NoTargetReason = resErr.Error()
return info
}
unitSize := dirSizeBytes(RecoveryUnitPath(sourceNsRoot, stackName)) + dirSizeBytes(AppDataDir(sourceNsRoot, appDataName))
target, err := m.selectTier2Target(stackName, unitSize) target, err := m.selectTier2Target(stackName, unitSize)
if err != nil { if err != nil {
info.NoTarget = true info.NoTarget = true
@@ -0,0 +1,215 @@
package backup
import (
"errors"
"io"
"log"
"path/filepath"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// newRunTier2Manager builds a Manager wired to a fake provider, a real temp-dir source drive holding
// a recovery unit + an appdata/<appDataName> dir (created iff appDataName != ""), and a (non-existent)
// schedulable off-drive target so selectTier2Target resolves a real target without touching the disk
// (SamePhysicalDevice returns false for an unstattable path on both Linux and Windows). Compose
// mounts default to appdata/<appDataName>/media + /export; callers override fake.mounts for the
// legacy/multi-dir shapes. The tier2Mirror seam captures the (src,dst) of each leg.
func newRunTier2Manager(t *testing.T, stack, appDataName string) (m *Manager, src string, captured *[][2]string) {
t.Helper()
tmp := t.TempDir()
src = filepath.Join(tmp, "usb") // the source drive == HDD_PATH == namespace root (Model A)
sysPath := filepath.Join(tmp, "sys")
target := filepath.Join(tmp, "off-drive-target") // never created → treated as a different device
sett, err := settings.Load(filepath.Join(tmp, "settings.json"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
if err := sett.AddStoragePath(settings.StoragePath{Path: target, Label: "off", Schedulable: true}); err != nil {
t.Fatal(err)
}
// A recovery unit (gates RunTier2) + the resolved appdata dir, both with real bytes.
mustWrite(t, filepath.Join(RecoveryUnitPath(src, stack), "manifest.json"), "{}")
var mounts []string
if appDataName != "" {
mustWrite(t, filepath.Join(AppDataDir(src, appDataName), "media", "a.jpg"), "JPEGDATA")
mounts = []string{
filepath.Join(src, "appdata", appDataName, "media"),
filepath.Join(src, "appdata", appDataName, "export"),
}
}
fake := &t2rFakeProvider{hdd: src, mounts: mounts}
cfg := &config.Config{}
cfg.Paths.SystemDataPath = sysPath
m = NewManager(cfg, sett, log.New(io.Discard, "", 0))
m.stackProvider = fake
m.systemDataPath = sysPath
pairs := &[][2]string{}
m.tier2Mirror = func(s, d string) error {
*pairs = append(*pairs, [2]string{s, d})
return nil
}
return m, src, pairs
}
// TestRunTier2_PaperlessShape (Scenario A / RP-2): the appdata leg is mirrored from the REAL
// compose-derived dir (appdata/paperless), and the recorded size includes its bytes. Companion
// RP-2: reverting the L146 site to AppDataDir(nsRoot, stackName) makes the appdata leg mirror a
// non-existent dir — the appdata capture below (dst ".../appdata") never fires.
func TestRunTier2_PaperlessShape(t *testing.T) {
m, srcDrive, captured := newRunTier2Manager(t, "paperless-ngx", "paperless")
if err := m.RunTier2("paperless-ngx"); err != nil {
t.Fatalf("RunTier2: %v", err)
}
// The appdata leg must mirror appdata/paperless → <destBase>/appdata.
wantSrc := AppDataDir(srcDrive, "paperless")
var appdataLeg *[2]string
for i := range *captured {
if filepath.Base((*captured)[i][1]) == "appdata" {
appdataLeg = &(*captured)[i]
}
}
if appdataLeg == nil {
t.Fatalf("appdata leg was never mirrored (F-S2 regression); captured=%v", *captured)
}
if appdataLeg[0] != wantSrc {
t.Errorf("appdata mirror src = %q, want the resolved dir %q", appdataLeg[0], wantSrc)
}
if filepath.Base(appdataLeg[1]) != "appdata" {
t.Errorf("appdata mirror dst = %q, want flat <destBase>/appdata", appdataLeg[1])
}
// Recorded success size must include the paperless bytes (non-empty, > 0).
cd := m.settings.GetCrossDriveConfig("paperless-ngx")
if cd == nil || cd.LastStatus != "ok" {
t.Fatalf("expected recorded ok status, got %+v", cd)
}
if cd.LastSizeHuman == "" || cd.LastSizeHuman == "0 B" {
t.Errorf("recorded size = %q, want it to include the appdata bytes", cd.LastSizeHuman)
}
}
// TestRunTier2_LegacyShape (Scenario B): a match-name app AND a no-binds app both mirror the exact
// same src the pre-fix code keyed by stack name — byte-identical behavior.
func TestRunTier2_LegacyShape(t *testing.T) {
t.Run("matching name (nextcloud)", func(t *testing.T) {
m, srcDrive, captured := newRunTier2Manager(t, "nextcloud", "nextcloud")
m.stackProvider.(*t2rFakeProvider).mounts = []string{filepath.Join(srcDrive, "appdata", "nextcloud")}
if err := m.RunTier2("nextcloud"); err != nil {
t.Fatalf("RunTier2: %v", err)
}
assertAppdataSrc(t, captured, AppDataDir(srcDrive, "nextcloud"))
})
t.Run("no appdata binds (fallback == stack name)", func(t *testing.T) {
// The appdata dir is created under the stack name; mounts are cleared → resolver falls back.
m, srcDrive, captured := newRunTier2Manager(t, "vaultwarden", "vaultwarden")
m.stackProvider.(*t2rFakeProvider).mounts = nil
if err := m.RunTier2("vaultwarden"); err != nil {
t.Fatalf("RunTier2: %v", err)
}
assertAppdataSrc(t, captured, AppDataDir(srcDrive, "vaultwarden"))
})
}
func assertAppdataSrc(t *testing.T, captured *[][2]string, want string) {
t.Helper()
for _, p := range *captured {
if filepath.Base(p[1]) == "appdata" {
if p[0] != want {
t.Errorf("appdata mirror src = %q, want %q", p[0], want)
}
return
}
}
t.Fatalf("appdata leg not mirrored; captured=%v", *captured)
}
// TestRunTier2_MultiDirRefusal (Scenario C / RP-5): two distinct appdata dirs → an honest no_target
// status with the EXACT Hungarian reason, an [ERROR] log, and NO mirror call. Companion RP-5:
// deleting the N>1 guard in tier2AppDataName makes the mirror fire (call-count > 0).
func TestRunTier2_MultiDirRefusal(t *testing.T) {
m, srcDrive, captured := newRunTier2Manager(t, "twodir", "alpha")
m.stackProvider.(*t2rFakeProvider).mounts = []string{
filepath.Join(srcDrive, "appdata", "alpha", "x"),
filepath.Join(srcDrive, "appdata", "beta", "y"),
}
if err := m.RunTier2("twodir"); err != nil {
t.Fatalf("RunTier2 must record a status, not error: %v", err)
}
if n := len(*captured); n != 0 {
t.Errorf("mirror was called %d time(s) on a multi-dir refusal — want 0", n)
}
cd := m.settings.GetCrossDriveConfig("twodir")
if cd == nil || cd.LastStatus != "no_target" {
t.Fatalf("expected no_target status, got %+v", cd)
}
if cd.LastError != errTier2MultiDir.Error() {
t.Errorf("reason = %q, want %q", cd.LastError, errTier2MultiDir.Error())
}
}
// TestTier2Info_MultiDirRefusal (Scenario C, info tier): the config-panel view surfaces the same
// refusal reason.
func TestTier2Info_MultiDirRefusal(t *testing.T) {
m, srcDrive, _ := newRunTier2Manager(t, "twodir", "alpha")
m.stackProvider.(*t2rFakeProvider).mounts = []string{
filepath.Join(srcDrive, "appdata", "alpha", "x"),
filepath.Join(srcDrive, "appdata", "beta", "y"),
}
info := m.Tier2Info("twodir")
if !info.NoTarget {
t.Fatal("expected NoTarget on a multi-dir app")
}
if info.NoTargetReason != errTier2MultiDir.Error() {
t.Errorf("reason = %q, want %q", info.NoTargetReason, errTier2MultiDir.Error())
}
}
// TestRestoreTier2Files_ResolvedLiveDir (Scenario E / RP-3): the copier's dst is the RESOLVED live
// dir (appdata/paperless), not appdata/<stackName>. Companion RP-3: reverting liveDir to stack-name
// keying makes dst appdata/paperless-ngx and this fails.
func TestRestoreTier2Files_ResolvedLiveDir(t *testing.T) {
m, fake, liveDrive, _ := newT2RManager(t)
fake.mounts = []string{
filepath.Join(liveDrive, "appdata", "paperless", "media"),
}
var gotDst string
m.restoreFilesCopier = func(_, dst string) (int, error) { gotDst = dst; return 0, nil }
if _, err := m.RestoreTier2Files("app"); err != nil {
t.Fatalf("restore: %v", err)
}
if want := AppDataDir(liveDrive, "paperless"); gotDst != want {
t.Errorf("restore dst = %q, want resolved live dir %q", gotDst, want)
}
}
// TestRestoreTier2Files_MultiDirRefusal (Scenario C, restore tier): two dirs → errTier2MultiDirRestore
// BEFORE the app is stopped (effect assertion: StopStack never invoked).
func TestRestoreTier2Files_MultiDirRefusal(t *testing.T) {
m, fake, liveDrive, _ := newT2RManager(t)
fake.mounts = []string{
filepath.Join(liveDrive, "appdata", "alpha", "x"),
filepath.Join(liveDrive, "appdata", "beta", "y"),
}
called := false
m.restoreFilesCopier = func(string, string) (int, error) { called = true; return 0, nil }
_, err := m.RestoreTier2Files("app")
if !errors.Is(err, errTier2MultiDirRestore) {
t.Fatalf("err = %v, want errTier2MultiDirRestore", err)
}
if len(fake.stopped) != 0 {
t.Errorf("app was STOPPED on a multi-dir refusal: %v", fake.stopped)
}
if called {
t.Error("copier invoked on a refusal")
}
}
+11 -1
View File
@@ -29,6 +29,9 @@ var (
errTier2DriveGone = errors.New("a másodlagos meghajtó nincs csatlakoztatva") errTier2DriveGone = errors.New("a másodlagos meghajtó nincs csatlakoztatva")
errLiveDriveGone = errors.New("az alkalmazás meghajtója nincs csatlakoztatva") errLiveDriveGone = errors.New("az alkalmazás meghajtója nincs csatlakoztatva")
errLiveDriveDecommed = errors.New("az alkalmazás meghajtója le van szerelve") errLiveDriveDecommed = errors.New("az alkalmazás meghajtója le van szerelve")
// errTier2MultiDirRestore (F-S2): the app resolves to more than one appdata dir, which the flat
// tier-2 copy layout does not represent — refused BEFORE the app is stopped.
errTier2MultiDirRestore = errors.New("az alkalmazáshoz több adatkönyvtár tartozik — a fájl-visszaállítás jelenleg nem támogatott")
) )
// RestoreTier2Files restores the app's MISSING user files in place from its recorded Tier-2 copy // RestoreTier2Files restores the app's MISSING user files in place from its recorded Tier-2 copy
@@ -60,7 +63,14 @@ func (m *Manager) RestoreTier2Files(stackName string) (filesRestored int, err er
return 0, fmt.Errorf("%w (%s)", errLiveDriveDecommed, drive) return 0, fmt.Errorf("%w (%s)", errLiveDriveDecommed, drive)
} }
} }
liveDir := AppDataDir(m.namespaceRoot(drive), stackName) // F-S2: the live appdata dir is the app's REAL compose-derived dir (paperless-ngx → paperless),
// not the stack name. N>1 distinct dirs → refuse here, BEFORE the app is stopped.
liveNsRoot := m.namespaceRoot(drive)
appDataName, resErr := m.tier2AppDataName(stackName, liveNsRoot)
if resErr != nil {
return 0, errTier2MultiDirRestore
}
liveDir := AppDataDir(liveNsRoot, appDataName)
// Source side: the RECORDED Tier-2 copy must exist and its drive must be connected. // Source side: the RECORDED Tier-2 copy must exist and its drive must be connected.
var srcDir string var srcDir string
@@ -17,6 +17,7 @@ import (
// the NON-effect (never stopped) and the happy path asserts the stop→copy→start order. // the NON-effect (never stopped) and the happy path asserts the stop→copy→start order.
type t2rFakeProvider struct { type t2rFakeProvider struct {
hdd string hdd string
mounts []string // F-S2: configurable compose-derived HDD mounts (drives appdata dir-name resolution)
stopped []string stopped []string
started []string started []string
order []string // interleaved event log: "stop", "copy" (appended by the copier seam), "start" order []string // interleaved event log: "stop", "copy" (appended by the copier seam), "start"
@@ -24,7 +25,7 @@ type t2rFakeProvider struct {
func (f *t2rFakeProvider) GetStackComposePath(string) (string, bool) { return "", false } func (f *t2rFakeProvider) GetStackComposePath(string) (string, bool) { return "", false }
func (f *t2rFakeProvider) ListDeployedStacks() []StackSummary { return nil } func (f *t2rFakeProvider) ListDeployedStacks() []StackSummary { return nil }
func (f *t2rFakeProvider) GetStackHDDMounts(string) []string { return nil } func (f *t2rFakeProvider) GetStackHDDMounts(string) []string { return f.mounts }
func (f *t2rFakeProvider) GetStackHDDPath(string) string { return f.hdd } func (f *t2rFakeProvider) GetStackHDDPath(string) string { return f.hdd }
func (f *t2rFakeProvider) GetDockerVolumes(string) []string { return nil } func (f *t2rFakeProvider) GetDockerVolumes(string) []string { return nil }
func (f *t2rFakeProvider) StopStack(name string) error { func (f *t2rFakeProvider) StopStack(name string) error {
+64 -8
View File
@@ -139,6 +139,7 @@ type migSeams struct {
verify func(ctx context.Context, src, dst string) error verify func(ctx context.Context, src, dst string) error
stop func(name string) error stop func(name string) error
flipRedeploy func(name, target string) error flipRedeploy func(name, target string) error
resolveNames func(app string) []string // F-S3: override appdata dir-name resolution in tests
} }
// SetMigrationDeps wires the registry + the backup-running check (mutual exclusion, Change 3). // SetMigrationDeps wires the registry + the backup-running check (mutual exclusion, Change 3).
@@ -291,6 +292,39 @@ func (m *Manager) appSourceNS(cfg *AppConfig) (src, ns string) {
return src, appbackup.NamespaceRoot(src, m.inGuest(src)) return src, appbackup.NamespaceRoot(src, m.inGuest(src))
} }
// resolveAppDataDirNames returns an app's real appdata dir name(s) under its HDD_PATH, derived from
// its compose binds (F-S2/F-S3: paperless-ngx writes appdata/paperless, not appdata/paperless-ngx).
// Mirrors the stackAdapter Priority-1 shape (app's own HDD_PATH + ParseComposeHDDMounts); the
// Priority-2 multi-storage union is NOT needed here — migration/tier-2 only apply to real-HDD apps.
// A test seam (testSeams.resolveNames) overrides it so the phase machine can be driven without a
// live stack registry. Falls back to []string{app} when nothing is derivable (legacy behavior).
// ResolveAppDataDirNames returns an app's real appdata dir name(s) under its HDD_PATH (F-S2/F-S3),
// for callers outside the migrate flow (the storage-page size display). Exported thin wrapper.
func (m *Manager) ResolveAppDataDirNames(app string) []string {
names, _ := m.resolveAppDataDirs(app)
return names
}
// resolveAppDataDirs also reports whether the app's compose DECLARES an appdata bind (declared) —
// migCopy uses it to WARN when a declared dir is missing on disk (the silence that hid F-S2), while
// distinguishing that from an app that genuinely has no appdata (fallback, no WARN).
func (m *Manager) resolveAppDataDirs(app string) (names []string, declared bool) {
if m.testSeams != nil && m.testSeams.resolveNames != nil {
return m.testSeams.resolveNames(app), true
}
stack, ok := m.GetStack(app)
if !ok {
return []string{app}, false
}
cfg := LoadAppConfig(filepath.Dir(stack.ComposePath))
if cfg == nil || cfg.Env["HDD_PATH"] == "" {
return []string{app}, false
}
hddPath := cfg.Env["HDD_PATH"]
mounts := ParseComposeHDDMounts(stack.ComposePath, hddPath)
return appbackup.AppDataDirNames(hddPath, app, mounts), appbackup.AppDataBindsPresent(hddPath, mounts)
}
// appsOnDrive returns the names of deployed apps whose HDD_PATH equals sourcePath. // appsOnDrive returns the names of deployed apps whose HDD_PATH equals sourcePath.
func (m *Manager) appsOnDrive(sourcePath string) []string { func (m *Manager) appsOnDrive(sourcePath string) []string {
var out []string var out []string
@@ -322,11 +356,14 @@ func (m *Manager) migValidate(j *MigrationJob) error {
if m.settings == nil || !m.settings.IsStoragePathSchedulable(j.Target) { if m.settings == nil || !m.settings.IsStoragePathSchedulable(j.Target) {
return fmt.Errorf("a céltároló nem elérhető vagy nem választható") return fmt.Errorf("a céltároló nem elérhető vagy nem választható")
} }
// App-dir collision: refuse if the same app dir already exists at the target. // App-dir collision: refuse if any of the app's resolved appdata dir(s) already exists at target.
var collide []string var collide []string
for _, app := range j.Apps { for _, app := range j.Apps {
if pathExists(appbackup.AppDataDir(j.TargetNS, app)) { for _, name := range m.ResolveAppDataDirNames(app) {
if pathExists(appbackup.AppDataDir(j.TargetNS, name)) {
collide = append(collide, app) collide = append(collide, app)
break
}
} }
} }
if len(collide) > 0 { if len(collide) > 0 {
@@ -351,7 +388,9 @@ func (m *Manager) migSourceSize(j *MigrationJob) int64 {
} }
var total int64 var total int64
for _, app := range j.Apps { for _, app := range j.Apps {
total += dirBytes(appbackup.AppDataDir(j.SourceNS, app)) for _, name := range m.ResolveAppDataDirNames(app) {
total += dirBytes(appbackup.AppDataDir(j.SourceNS, name))
}
total += dirBytes(appbackup.RecoveryUnitPath(j.SourceNS, app)) total += dirBytes(appbackup.RecoveryUnitPath(j.SourceNS, app))
} }
return total return total
@@ -445,11 +484,22 @@ func (m *Manager) migCopy(ctx context.Context, j *MigrationJob) error {
} }
j.CurrentApp = app j.CurrentApp = app
_ = m.persistJob(j) _ = m.persistJob(j)
// appdata subtree (collision-free post-validate) // appdata subtree(s) — the app's REAL compose-derived dir name(s) (F-S3: paperless-ngx writes
if err := m.copySubtree(ctx, j, appbackup.AppDataDir(j.SourceNS, app), appbackup.AppDataDir(j.TargetNS, app)); err != nil { // appdata/paperless, not appdata/paperless-ngx); collision-free post-validate.
names, declared := m.resolveAppDataDirs(app)
for _, name := range names {
src := appbackup.AppDataDir(j.SourceNS, name)
if declared && !pathExists(src) {
// The compose DECLARES this appdata dir but it is absent on disk — WARN instead of the
// silent no-op that hid F-S2. copySubtree still no-ops safely below.
m.logger.Printf("[WARN] [migrate] %s: compose declares appdata dir %q but it is absent at %s — nothing to copy",
app, name, src)
}
if err := m.copySubtree(ctx, j, src, appbackup.AppDataDir(j.TargetNS, name)); err != nil {
u.Error = err.Error() u.Error = err.Error()
return fmt.Errorf("másolás sikertelen (%s appdata): %w", app, err) return fmt.Errorf("másolás sikertelen (%s appdata): %w", app, err)
} }
}
// the app's recovery unit (db-dumps + volume-dumps + compose + manifest) // the app's recovery unit (db-dumps + volume-dumps + compose + manifest)
if err := m.copySubtree(ctx, j, appbackup.RecoveryUnitPath(j.SourceNS, app), appbackup.RecoveryUnitPath(j.TargetNS, app)); err != nil { if err := m.copySubtree(ctx, j, appbackup.RecoveryUnitPath(j.SourceNS, app), appbackup.RecoveryUnitPath(j.TargetNS, app)); err != nil {
u.Error = err.Error() u.Error = err.Error()
@@ -488,10 +538,12 @@ func (m *Manager) migVerify(ctx context.Context, j *MigrationJob) error {
if stateRank(u.State) >= stateRank(UnitVerified) { if stateRank(u.State) >= stateRank(UnitVerified) {
continue continue
} }
if err := m.verifySubtree(ctx, appbackup.AppDataDir(j.SourceNS, app), appbackup.AppDataDir(j.TargetNS, app)); err != nil { for _, name := range m.ResolveAppDataDirNames(app) {
if err := m.verifySubtree(ctx, appbackup.AppDataDir(j.SourceNS, name), appbackup.AppDataDir(j.TargetNS, name)); err != nil {
u.Error = err.Error() u.Error = err.Error()
return fmt.Errorf("ellenőrzés sikertelen (%s appdata): %w", app, err) return fmt.Errorf("ellenőrzés sikertelen (%s appdata): %w", app, err)
} }
}
if err := m.verifySubtree(ctx, appbackup.RecoveryUnitPath(j.SourceNS, app), appbackup.RecoveryUnitPath(j.TargetNS, app)); err != nil { if err := m.verifySubtree(ctx, appbackup.RecoveryUnitPath(j.SourceNS, app), appbackup.RecoveryUnitPath(j.TargetNS, app)); err != nil {
u.Error = err.Error() u.Error = err.Error()
return fmt.Errorf("ellenőrzés sikertelen (%s mentés): %w", app, err) return fmt.Errorf("ellenőrzés sikertelen (%s mentés): %w", app, err)
@@ -553,9 +605,11 @@ func (m *Manager) migCleanup(j *MigrationJob) error {
if u.State == UnitCleaned { if u.State == UnitCleaned {
continue continue
} }
if err := os.RemoveAll(appbackup.AppDataDir(j.SourceNS, app)); err != nil { for _, name := range m.ResolveAppDataDirNames(app) {
if err := os.RemoveAll(appbackup.AppDataDir(j.SourceNS, name)); err != nil {
return fmt.Errorf("forrás törlése sikertelen (%s appdata): %w", app, err) return fmt.Errorf("forrás törlése sikertelen (%s appdata): %w", app, err)
} }
}
if err := os.RemoveAll(appbackup.RecoveryUnitPath(j.SourceNS, app)); err != nil { if err := os.RemoveAll(appbackup.RecoveryUnitPath(j.SourceNS, app)); err != nil {
return fmt.Errorf("forrás törlése sikertelen (%s mentés): %w", app, err) return fmt.Errorf("forrás törlése sikertelen (%s mentés): %w", app, err)
} }
@@ -602,7 +656,9 @@ func (m *Manager) migCleanupAllowed(j *MigrationJob) error {
func (m *Manager) appDataSkipSet(j *MigrationJob) map[string]bool { func (m *Manager) appDataSkipSet(j *MigrationJob) map[string]bool {
skip := map[string]bool{} skip := map[string]bool{}
for _, app := range j.Apps { for _, app := range j.Apps {
skip[filepath.Clean(appbackup.AppDataDir(j.SourceNS, app))] = true for _, name := range m.ResolveAppDataDirNames(app) {
skip[filepath.Clean(appbackup.AppDataDir(j.SourceNS, name))] = true
}
} }
return skip return skip
} }
@@ -0,0 +1,104 @@
package stacks
import (
"context"
"path/filepath"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/appbackup"
)
// newAppJob builds a scope="app" job (no non-app merge unit — app scope has no merge walk).
func newAppJob(srcNS, dstNS, source, target string, apps ...string) *MigrationJob {
j := &MigrationJob{
Scope: "app", Phase: PhaseStop, Source: source, Target: target,
SourceNS: srcNS, TargetNS: dstNS, Apps: apps, Units: map[string]*MigUnit{},
}
for _, a := range apps {
j.Units[a] = &MigUnit{App: a, State: UnitPending}
}
return j
}
// TestMigrate_PaperlessShape_ScopeApp is Scenario D / RP-4: a scope="app" migration of paperless-ngx
// (stack name) must copy, verify, and clean the REAL appdata dir (paperless), never appdata/paperless-ngx.
// Every leg is asserted by captured src/dst pairs and on-disk effects — never mere absence of error.
// Companion RP-4: dropping the resolveNames loop (keying by stack name) makes the copy seam receive
// appdata/paperless-ngx and every assertion below fails.
func TestMigrate_PaperlessShape_ScopeApp(t *testing.T) {
m := newMigManager(t, "/mnt/target")
srcNS := t.TempDir()
dstNS := t.TempDir()
// Real source data under the REAL dir name (paperless), plus the recovery unit.
writeFile(t, filepath.Join(appbackup.AppDataDir(srcNS, "paperless"), "media", "doc.pdf"), "PDF")
writeFile(t, filepath.Join(appbackup.RecoveryUnitPath(srcNS, "paperless-ngx"), "manifest.json"), "{}")
type pair struct{ src, dst string }
var copied, verified []pair
m.testSeams = &migSeams{
resolveNames: func(app string) []string {
if app == "paperless-ngx" {
return []string{"paperless"}
}
return []string{app}
},
stop: func(string) error { return nil },
copy: func(_ context.Context, s, d string, _ func(int64)) error {
copied = append(copied, pair{s, d})
return nil
},
verify: func(_ context.Context, s, d string) error {
verified = append(verified, pair{s, d})
return nil
},
flipRedeploy: func(string, string) error { return nil },
}
// --- direct-call assertions (accounting legs) before the destructive run ---
wantSrc := appbackup.AppDataDir(srcNS, "paperless")
// migSourceSize must probe appdata/paperless (non-zero: the PDF exists there).
if got := m.migSourceSize(newAppJob(srcNS, dstNS, "/s", "/t", "paperless-ngx")); got == 0 {
t.Error("migSourceSize = 0 — it probed appdata/paperless-ngx (empty) instead of appdata/paperless")
}
// appDataSkipSet must contain the resolved source dir.
skip := m.appDataSkipSet(newAppJob(srcNS, dstNS, "/s", "/t", "paperless-ngx"))
if !skip[filepath.Clean(wantSrc)] {
t.Errorf("skip-set = %v, want it to contain %q", skip, filepath.Clean(wantSrc))
}
// Collision check must probe appdata/paperless at the target. Target must be the schedulable path
// so migValidate reaches the collision check (TargetNS stays the temp dir where the dir exists).
writeFile(t, filepath.Join(appbackup.AppDataDir(dstNS, "paperless"), "x"), "x")
cj := newAppJob(srcNS, dstNS, "/s", "/mnt/target", "paperless-ngx")
if err := m.migValidate(cj); err == nil || !contains(err.Error(), "paperless-ngx") {
t.Errorf("collision must fire on the resolved target dir, got %v", err)
}
// --- full pipeline run (copy → verify → cleanup) ---
j := newAppJob(srcNS, dstNS, "/s", "/t", "paperless-ngx")
m.runJobSync(j)
if j.Phase != PhaseDone {
t.Fatalf("phase = %s (err=%s), want done", j.Phase, j.Error)
}
wantDst := appbackup.AppDataDir(dstNS, "paperless")
assertHasPair := func(name string, got []pair) {
t.Helper()
for _, p := range got {
if p.src == wantSrc && p.dst == wantDst {
return
}
}
t.Errorf("%s seam never saw appdata/paperless: src=%q dst=%q, got %v", name, wantSrc, wantDst, got)
}
assertHasPair("copy", copied)
assertHasPair("verify", verified)
// Cleanup removed the REAL source dir.
if pathExists(wantSrc) {
t.Errorf("cleanup left the source appdata/paperless behind: %s", wantSrc)
}
// And it never probed / created appdata/paperless-ngx.
if pathExists(appbackup.AppDataDir(srcNS, "paperless-ngx")) {
t.Errorf("stale appdata/paperless-ngx dir should never exist")
}
}
+26 -6
View File
@@ -1675,18 +1675,28 @@ func (s *Server) appDetailsForPath(storagePath string) []StorageAppDetail {
Name: stack.Meta.DisplayName, Name: stack.Meta.DisplayName,
Stack: stack.Meta.Slug, Stack: stack.Meta.Slug,
} }
// Try to get data size from the storage subdirectory // Try to get data size from the storage subdirectory. F-S2: the app's real appdata dir name is
appDataDir := backup.AppDataDir(storagePath, stack.Name) // NOT always the stack name (paperless-ngx writes appdata/paperless) — sum the resolved dir(s).
if fi, err := os.Stat(appDataDir); err == nil && fi.IsDir() { // Here hddPath == storagePath (the drive's in-guest mount is the namespace root, Model A).
detail.SizeHuman = dirSizeHuman(appDataDir) var total int64
var any bool
for _, name := range s.stackMgr.ResolveAppDataDirNames(stack.Name) {
d := backup.AppDataDir(storagePath, name)
if fi, err := os.Stat(d); err == nil && fi.IsDir() {
total += dirSizeBytesWalk(d)
any = true
}
}
if any {
detail.SizeHuman = humanizeDirBytes(total)
} }
details = append(details, detail) details = append(details, detail)
} }
return details return details
} }
// dirSizeHuman returns a human-readable size for a directory. // dirSizeBytesWalk returns the total size in bytes of the regular files under path (0 if absent).
func dirSizeHuman(path string) string { func dirSizeBytesWalk(path string) int64 {
var total int64 var total int64
filepath.Walk(path, func(_ string, info os.FileInfo, err error) error { filepath.Walk(path, func(_ string, info os.FileInfo, err error) error {
if err != nil || info.IsDir() { if err != nil || info.IsDir() {
@@ -1695,6 +1705,11 @@ func dirSizeHuman(path string) string {
total += info.Size() total += info.Size()
return nil return nil
}) })
return total
}
// humanizeDirBytes formats a byte count as B/KB/MB/GB (the storage-page display convention).
func humanizeDirBytes(total int64) string {
const ( const (
KB = 1024 KB = 1024
MB = KB * 1024 MB = KB * 1024
@@ -1712,6 +1727,11 @@ func dirSizeHuman(path string) string {
} }
} }
// dirSizeHuman returns a human-readable size for a single directory.
func dirSizeHuman(path string) string {
return humanizeDirBytes(dirSizeBytesWalk(path))
}
func formatFreeSpace(gb float64) string { func formatFreeSpace(gb float64) string {
if gb >= 1000 { if gb >= 1000 {
return fmt.Sprintf("%.1f TB", gb/1024) return fmt.Sprintf("%.1f TB", gb/1024)