F-S2 + F-S3: compose-derived appdata dir resolution (v0.131.0)

The controller assumed an app's HDD appdata dir is always appdata/<stackName>.
paperless-ngx writes appdata/paperless (stack paperless-ngx), so every consumer
keying by stack name silently missed it via a stat-and-skip. One canonical
resolver appbackup.AppDataDirNames derives the real dir name(s) from the app's
compose ${HDD_PATH} binds; all consumers use it.

- F-S2 (tier-2): RunTier2 mirrors the resolved appdata/<name> (paperless docs
  got NO tier-2 copy before). Tier2Info size + RestoreTier2Files live dir use it.
  WARN when a declared appdata dir is absent. New tier2Mirror seam.
- F-S3 (migrate, NEW): all six per-app appdata legs (collision/size/copy/verify/
  cleanup/skip-set) now loop resolved names. scope="app" migration of paperless
  previously copied nothing and left an empty media dir (scope="all" was saved by
  the merge walk). WARN on missing declared dir in the copy leg.
- Multi-dir (N>1) refusal: tier-2 backup/info/restore refuse loudly (Hungarian);
  migrate supports N. No catalog app hits it today; lifted by Task 3.
- Display: storage page sums resolved dirs.
- Truth repair: the v0.130.0 "tier-2 copies the namespace wholesale" claim is
  false; corrected in CHANGELOG + main.go export-adapter comment.

+9 tests; red-proofs RP-1..RP-5 all confirmed. Controller-only, no agent/hub
coupling. Task 1 of the backup-classification-redesign arc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
This commit is contained in:
2026-07-14 17:45:53 +02:00
parent b42904bbab
commit 68f0e0cf5c
16 changed files with 765 additions and 44 deletions
@@ -133,3 +133,11 @@ func RecoveryUnitManifestPath(nsRoot, stackName string) string {
func AppDataDir(nsRoot, stackName string) string {
return appbackup.AppDataDir(nsRoot, stackName)
}
func AppDataDirNames(hddPath, stackName string, hddMounts []string) []string {
return appbackup.AppDataDirNames(hddPath, stackName, hddMounts)
}
func AppDataBindsPresent(hddPath string, hddMounts []string) bool {
return appbackup.AppDataBindsPresent(hddPath, hddMounts)
}
+5
View File
@@ -65,6 +65,11 @@ type Manager struct {
// the orchestration never shells out. Nil → the real rsyncRestoreMissing (additive-only).
restoreFilesCopier func(src, dst string) (filesRestored int, err error)
// tier2Mirror (F-S2) — the Tier-2 backup mirror seam (both rsync legs in RunTier2), overridable
// so the resolve→mirror→record flow is unit-testable without rsync. Nil → the real rsyncMirror
// (`-a --delete`, contents-of-src semantics).
tier2Mirror func(src, dst string) error
// migrationRunning, if set, reports whether a data migration is in progress. The scheduled
// backup paths skip when it returns true (Change 3 — backup ↔ migration mutual exclusion), so a
// nightly dump/Tier-2 can't race a migration copy/cleanup on the same drive.
+75 -11
View File
@@ -14,21 +14,59 @@ import (
"gitea.dooplex.hu/admin/felhom-controller/internal/system"
)
// Tier 2 = an off-drive (different physical disk) copy of an HDD app's recovery unit + bulk userdata.
// It is the ONLY off-drive protection that browsable HDD userdata can get — PBS can't reach bind
// mounts. Auto-enabled for every HDD app; the target is auto-picked: prefer another registered
// user-data drive (can hold bulk), else the internal SSD for SMALL units only — and the SSD is the
// guest rootfs (~8 GB), so we REFUSE rather than fill it (a size-aware headroom guard). When no
// off-drive target fits, we record an honest "needs a 2nd HDD" status instead of silently doing
// nothing useful.
// Tier 2 = an off-drive (different physical disk) copy of an HDD app's recovery unit + its resolved
// appdata/<name> dir(s). It does NOT copy the browsable userdata tree (F-S1: userdata is not backed
// up at any tier yet — that gap is owned by the classification redesign, see
// felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md). The appdata dir NAME is
// derived from the app's compose binds, NOT assumed to be the stack name (F-S2: paperless-ngx writes
// appdata/paperless); see tier2AppDataName. Auto-enabled for every HDD app; the target is auto-picked:
// prefer another registered user-data drive (can hold bulk), else the internal SSD for SMALL units
// only — and the SSD is the guest rootfs (~8 GB), so we REFUSE rather than fill it (a size-aware
// headroom guard). When no off-drive target fits, we record an honest "needs a 2nd HDD" status
// instead of silently doing nothing useful.
const gibibyte = 1024 * 1024 * 1024
var (
errNoOffDiskTarget = errors.New("no off-drive target (single drive, app already on the system disk)")
errSSDNoHeadroom = errors.New("the internal SSD lacks headroom for this app's data — a 2nd drive is required for off-drive backup")
// errTier2MultiDir is raised when an app's compose resolves to MORE THAN ONE distinct appdata
// dir under <hddPath>/appdata (no catalog app does today). Tier 2's destination layout is flat
// (<destBase>/appdata), so it refuses rather than silently collapse two source dirs into one.
errTier2MultiDir = errors.New("az alkalmazáshoz több adatkönyvtár tartozik — a 2. mentés jelenleg alkalmazásonként egy könyvtárat támogat")
)
// appDataDirNames resolves the app's real appdata dir name(s) under hddPath from its compose HDD
// binds, via the stack provider (nil provider → legacy [stackName] fallback). See
// appbackup.AppDataDirNames.
func (m *Manager) appDataDirNames(stackName, hddPath string) []string {
var mounts []string
if m.stackProvider != nil {
mounts = m.stackProvider.GetStackHDDMounts(stackName)
}
return AppDataDirNames(hddPath, stackName, mounts)
}
// tier2AppDataName resolves the SINGLE appdata dir name for tier-2's flat destination. N>1 distinct
// names → errTier2MultiDir (the one place the tier-2 multi-dir refusal is built). It always returns
// at least one name from appDataDirNames' fallback, so name is meaningful only when err == nil.
func (m *Manager) tier2AppDataName(stackName, hddPath string) (string, error) {
names := m.appDataDirNames(stackName, hddPath)
if len(names) > 1 {
return "", errTier2MultiDir
}
return names[0], nil
}
// tier2AppDataBindsPresent reports whether the app's compose declares an appdata bind (drives the
// WARN-on-missing-declared-dir rule; nil provider → false).
func (m *Manager) tier2AppDataBindsPresent(stackName, hddPath string) bool {
if m.stackProvider == nil {
return false
}
return AppDataBindsPresent(hddPath, m.stackProvider.GetStackHDDMounts(stackName))
}
// Tier2Target is a resolved off-drive destination for an app's Tier 2 copy.
type Tier2Target struct {
NamespaceRoot string // felhom-data namespace root on the target drive
@@ -143,7 +181,16 @@ func (m *Manager) RunTier2(stackName string) error {
}
sourceNsRoot := m.namespaceRoot(sourceDrive)
unitDir := RecoveryUnitPath(sourceNsRoot, stackName)
appDataDir := AppDataDir(sourceNsRoot, stackName)
// F-S2: resolve the app's REAL appdata dir name from its compose binds (paperless-ngx writes
// appdata/paperless, not appdata/paperless-ngx). For an HDD app HDD_PATH == nsRoot (Model A), so
// the mounts (resolved against HDD_PATH) share the nsRoot prefix. N>1 distinct names → refuse.
appDataName, resErr := m.tier2AppDataName(stackName, sourceNsRoot)
if resErr != nil {
m.recordTier2NoTarget(stackName, resErr.Error())
m.logger.Printf("[ERROR] [backup] Tier 2 for %s refused: %v", stackName, resErr)
return nil
}
appDataDir := AppDataDir(sourceNsRoot, appDataName)
if _, err := os.Stat(unitDir); err != nil {
return nil // no recovery unit yet — nothing to copy
}
@@ -166,7 +213,12 @@ func (m *Manager) RunTier2(stackName string) error {
destBase := filepath.Join(target.NamespaceRoot, "backups", "secondary", stackName)
start := time.Now()
if err := rsyncMirror(unitDir, filepath.Join(destBase, "recovery-unit")); err != nil {
mirror := m.tier2Mirror
if mirror == nil {
mirror = rsyncMirror
}
if err := mirror(unitDir, filepath.Join(destBase, "recovery-unit")); err != nil {
m.recordTier2Failure(stackName, target, err)
if m.tier2Notify != nil {
m.tier2Notify(stackName, target.Label, time.Since(start), err)
@@ -174,13 +226,18 @@ func (m *Manager) RunTier2(stackName string) error {
return fmt.Errorf("tier2 rsync unit for %s: %w", stackName, err)
}
if _, e := os.Stat(appDataDir); e == nil {
if err := rsyncMirror(appDataDir, filepath.Join(destBase, "appdata")); err != nil {
if err := mirror(appDataDir, filepath.Join(destBase, "appdata")); err != nil {
m.recordTier2Failure(stackName, target, err)
if m.tier2Notify != nil {
m.tier2Notify(stackName, target.Label, time.Since(start), err)
}
return fmt.Errorf("tier2 rsync appdata for %s: %w", stackName, err)
}
} else if m.tier2AppDataBindsPresent(stackName, sourceNsRoot) {
// F-S2: the compose DECLARES an appdata bind but the dir is missing on disk. Skipping is kept
// (nothing to copy) but the silence that hid F-S2 for months is now a loud WARN.
m.logger.Printf("[WARN] [backup] Tier 2 for %s: compose declares appdata dir %q but it is absent at %s — appdata leg skipped",
stackName, appDataName, appDataDir)
}
dur := time.Since(start)
@@ -279,8 +336,15 @@ func (m *Manager) Tier2Info(stackName string) Tier2Info {
}
// Resolve what the runner WOULD pick right now (real unit size feeds the SSD headroom guard).
// F-S2: N>1 distinct appdata dirs → the same honest refusal the runner records.
sourceNsRoot := m.namespaceRoot(source)
unitSize := dirSizeBytes(RecoveryUnitPath(sourceNsRoot, stackName)) + dirSizeBytes(AppDataDir(sourceNsRoot, stackName))
appDataName, resErr := m.tier2AppDataName(stackName, sourceNsRoot)
if resErr != nil {
info.NoTarget = true
info.NoTargetReason = resErr.Error()
return info
}
unitSize := dirSizeBytes(RecoveryUnitPath(sourceNsRoot, stackName)) + dirSizeBytes(AppDataDir(sourceNsRoot, appDataName))
target, err := m.selectTier2Target(stackName, unitSize)
if err != nil {
info.NoTarget = true
@@ -0,0 +1,215 @@
package backup
import (
"errors"
"io"
"log"
"path/filepath"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// newRunTier2Manager builds a Manager wired to a fake provider, a real temp-dir source drive holding
// a recovery unit + an appdata/<appDataName> dir (created iff appDataName != ""), and a (non-existent)
// schedulable off-drive target so selectTier2Target resolves a real target without touching the disk
// (SamePhysicalDevice returns false for an unstattable path on both Linux and Windows). Compose
// mounts default to appdata/<appDataName>/media + /export; callers override fake.mounts for the
// legacy/multi-dir shapes. The tier2Mirror seam captures the (src,dst) of each leg.
func newRunTier2Manager(t *testing.T, stack, appDataName string) (m *Manager, src string, captured *[][2]string) {
t.Helper()
tmp := t.TempDir()
src = filepath.Join(tmp, "usb") // the source drive == HDD_PATH == namespace root (Model A)
sysPath := filepath.Join(tmp, "sys")
target := filepath.Join(tmp, "off-drive-target") // never created → treated as a different device
sett, err := settings.Load(filepath.Join(tmp, "settings.json"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
if err := sett.AddStoragePath(settings.StoragePath{Path: target, Label: "off", Schedulable: true}); err != nil {
t.Fatal(err)
}
// A recovery unit (gates RunTier2) + the resolved appdata dir, both with real bytes.
mustWrite(t, filepath.Join(RecoveryUnitPath(src, stack), "manifest.json"), "{}")
var mounts []string
if appDataName != "" {
mustWrite(t, filepath.Join(AppDataDir(src, appDataName), "media", "a.jpg"), "JPEGDATA")
mounts = []string{
filepath.Join(src, "appdata", appDataName, "media"),
filepath.Join(src, "appdata", appDataName, "export"),
}
}
fake := &t2rFakeProvider{hdd: src, mounts: mounts}
cfg := &config.Config{}
cfg.Paths.SystemDataPath = sysPath
m = NewManager(cfg, sett, log.New(io.Discard, "", 0))
m.stackProvider = fake
m.systemDataPath = sysPath
pairs := &[][2]string{}
m.tier2Mirror = func(s, d string) error {
*pairs = append(*pairs, [2]string{s, d})
return nil
}
return m, src, pairs
}
// TestRunTier2_PaperlessShape (Scenario A / RP-2): the appdata leg is mirrored from the REAL
// compose-derived dir (appdata/paperless), and the recorded size includes its bytes. Companion
// RP-2: reverting the L146 site to AppDataDir(nsRoot, stackName) makes the appdata leg mirror a
// non-existent dir — the appdata capture below (dst ".../appdata") never fires.
func TestRunTier2_PaperlessShape(t *testing.T) {
m, srcDrive, captured := newRunTier2Manager(t, "paperless-ngx", "paperless")
if err := m.RunTier2("paperless-ngx"); err != nil {
t.Fatalf("RunTier2: %v", err)
}
// The appdata leg must mirror appdata/paperless → <destBase>/appdata.
wantSrc := AppDataDir(srcDrive, "paperless")
var appdataLeg *[2]string
for i := range *captured {
if filepath.Base((*captured)[i][1]) == "appdata" {
appdataLeg = &(*captured)[i]
}
}
if appdataLeg == nil {
t.Fatalf("appdata leg was never mirrored (F-S2 regression); captured=%v", *captured)
}
if appdataLeg[0] != wantSrc {
t.Errorf("appdata mirror src = %q, want the resolved dir %q", appdataLeg[0], wantSrc)
}
if filepath.Base(appdataLeg[1]) != "appdata" {
t.Errorf("appdata mirror dst = %q, want flat <destBase>/appdata", appdataLeg[1])
}
// Recorded success size must include the paperless bytes (non-empty, > 0).
cd := m.settings.GetCrossDriveConfig("paperless-ngx")
if cd == nil || cd.LastStatus != "ok" {
t.Fatalf("expected recorded ok status, got %+v", cd)
}
if cd.LastSizeHuman == "" || cd.LastSizeHuman == "0 B" {
t.Errorf("recorded size = %q, want it to include the appdata bytes", cd.LastSizeHuman)
}
}
// TestRunTier2_LegacyShape (Scenario B): a match-name app AND a no-binds app both mirror the exact
// same src the pre-fix code keyed by stack name — byte-identical behavior.
func TestRunTier2_LegacyShape(t *testing.T) {
t.Run("matching name (nextcloud)", func(t *testing.T) {
m, srcDrive, captured := newRunTier2Manager(t, "nextcloud", "nextcloud")
m.stackProvider.(*t2rFakeProvider).mounts = []string{filepath.Join(srcDrive, "appdata", "nextcloud")}
if err := m.RunTier2("nextcloud"); err != nil {
t.Fatalf("RunTier2: %v", err)
}
assertAppdataSrc(t, captured, AppDataDir(srcDrive, "nextcloud"))
})
t.Run("no appdata binds (fallback == stack name)", func(t *testing.T) {
// The appdata dir is created under the stack name; mounts are cleared → resolver falls back.
m, srcDrive, captured := newRunTier2Manager(t, "vaultwarden", "vaultwarden")
m.stackProvider.(*t2rFakeProvider).mounts = nil
if err := m.RunTier2("vaultwarden"); err != nil {
t.Fatalf("RunTier2: %v", err)
}
assertAppdataSrc(t, captured, AppDataDir(srcDrive, "vaultwarden"))
})
}
func assertAppdataSrc(t *testing.T, captured *[][2]string, want string) {
t.Helper()
for _, p := range *captured {
if filepath.Base(p[1]) == "appdata" {
if p[0] != want {
t.Errorf("appdata mirror src = %q, want %q", p[0], want)
}
return
}
}
t.Fatalf("appdata leg not mirrored; captured=%v", *captured)
}
// TestRunTier2_MultiDirRefusal (Scenario C / RP-5): two distinct appdata dirs → an honest no_target
// status with the EXACT Hungarian reason, an [ERROR] log, and NO mirror call. Companion RP-5:
// deleting the N>1 guard in tier2AppDataName makes the mirror fire (call-count > 0).
func TestRunTier2_MultiDirRefusal(t *testing.T) {
m, srcDrive, captured := newRunTier2Manager(t, "twodir", "alpha")
m.stackProvider.(*t2rFakeProvider).mounts = []string{
filepath.Join(srcDrive, "appdata", "alpha", "x"),
filepath.Join(srcDrive, "appdata", "beta", "y"),
}
if err := m.RunTier2("twodir"); err != nil {
t.Fatalf("RunTier2 must record a status, not error: %v", err)
}
if n := len(*captured); n != 0 {
t.Errorf("mirror was called %d time(s) on a multi-dir refusal — want 0", n)
}
cd := m.settings.GetCrossDriveConfig("twodir")
if cd == nil || cd.LastStatus != "no_target" {
t.Fatalf("expected no_target status, got %+v", cd)
}
if cd.LastError != errTier2MultiDir.Error() {
t.Errorf("reason = %q, want %q", cd.LastError, errTier2MultiDir.Error())
}
}
// TestTier2Info_MultiDirRefusal (Scenario C, info tier): the config-panel view surfaces the same
// refusal reason.
func TestTier2Info_MultiDirRefusal(t *testing.T) {
m, srcDrive, _ := newRunTier2Manager(t, "twodir", "alpha")
m.stackProvider.(*t2rFakeProvider).mounts = []string{
filepath.Join(srcDrive, "appdata", "alpha", "x"),
filepath.Join(srcDrive, "appdata", "beta", "y"),
}
info := m.Tier2Info("twodir")
if !info.NoTarget {
t.Fatal("expected NoTarget on a multi-dir app")
}
if info.NoTargetReason != errTier2MultiDir.Error() {
t.Errorf("reason = %q, want %q", info.NoTargetReason, errTier2MultiDir.Error())
}
}
// TestRestoreTier2Files_ResolvedLiveDir (Scenario E / RP-3): the copier's dst is the RESOLVED live
// dir (appdata/paperless), not appdata/<stackName>. Companion RP-3: reverting liveDir to stack-name
// keying makes dst appdata/paperless-ngx and this fails.
func TestRestoreTier2Files_ResolvedLiveDir(t *testing.T) {
m, fake, liveDrive, _ := newT2RManager(t)
fake.mounts = []string{
filepath.Join(liveDrive, "appdata", "paperless", "media"),
}
var gotDst string
m.restoreFilesCopier = func(_, dst string) (int, error) { gotDst = dst; return 0, nil }
if _, err := m.RestoreTier2Files("app"); err != nil {
t.Fatalf("restore: %v", err)
}
if want := AppDataDir(liveDrive, "paperless"); gotDst != want {
t.Errorf("restore dst = %q, want resolved live dir %q", gotDst, want)
}
}
// TestRestoreTier2Files_MultiDirRefusal (Scenario C, restore tier): two dirs → errTier2MultiDirRestore
// BEFORE the app is stopped (effect assertion: StopStack never invoked).
func TestRestoreTier2Files_MultiDirRefusal(t *testing.T) {
m, fake, liveDrive, _ := newT2RManager(t)
fake.mounts = []string{
filepath.Join(liveDrive, "appdata", "alpha", "x"),
filepath.Join(liveDrive, "appdata", "beta", "y"),
}
called := false
m.restoreFilesCopier = func(string, string) (int, error) { called = true; return 0, nil }
_, err := m.RestoreTier2Files("app")
if !errors.Is(err, errTier2MultiDirRestore) {
t.Fatalf("err = %v, want errTier2MultiDirRestore", err)
}
if len(fake.stopped) != 0 {
t.Errorf("app was STOPPED on a multi-dir refusal: %v", fake.stopped)
}
if called {
t.Error("copier invoked on a refusal")
}
}
+11 -1
View File
@@ -29,6 +29,9 @@ var (
errTier2DriveGone = errors.New("a másodlagos meghajtó nincs csatlakoztatva")
errLiveDriveGone = errors.New("az alkalmazás meghajtója nincs csatlakoztatva")
errLiveDriveDecommed = errors.New("az alkalmazás meghajtója le van szerelve")
// errTier2MultiDirRestore (F-S2): the app resolves to more than one appdata dir, which the flat
// tier-2 copy layout does not represent — refused BEFORE the app is stopped.
errTier2MultiDirRestore = errors.New("az alkalmazáshoz több adatkönyvtár tartozik — a fájl-visszaállítás jelenleg nem támogatott")
)
// RestoreTier2Files restores the app's MISSING user files in place from its recorded Tier-2 copy
@@ -60,7 +63,14 @@ func (m *Manager) RestoreTier2Files(stackName string) (filesRestored int, err er
return 0, fmt.Errorf("%w (%s)", errLiveDriveDecommed, drive)
}
}
liveDir := AppDataDir(m.namespaceRoot(drive), stackName)
// F-S2: the live appdata dir is the app's REAL compose-derived dir (paperless-ngx → paperless),
// not the stack name. N>1 distinct dirs → refuse here, BEFORE the app is stopped.
liveNsRoot := m.namespaceRoot(drive)
appDataName, resErr := m.tier2AppDataName(stackName, liveNsRoot)
if resErr != nil {
return 0, errTier2MultiDirRestore
}
liveDir := AppDataDir(liveNsRoot, appDataName)
// Source side: the RECORDED Tier-2 copy must exist and its drive must be connected.
var srcDir string
@@ -17,6 +17,7 @@ import (
// the NON-effect (never stopped) and the happy path asserts the stop→copy→start order.
type t2rFakeProvider struct {
hdd string
mounts []string // F-S2: configurable compose-derived HDD mounts (drives appdata dir-name resolution)
stopped []string
started []string
order []string // interleaved event log: "stop", "copy" (appended by the copier seam), "start"
@@ -24,7 +25,7 @@ type t2rFakeProvider struct {
func (f *t2rFakeProvider) GetStackComposePath(string) (string, bool) { return "", false }
func (f *t2rFakeProvider) ListDeployedStacks() []StackSummary { return nil }
func (f *t2rFakeProvider) GetStackHDDMounts(string) []string { return nil }
func (f *t2rFakeProvider) GetStackHDDMounts(string) []string { return f.mounts }
func (f *t2rFakeProvider) GetStackHDDPath(string) string { return f.hdd }
func (f *t2rFakeProvider) GetDockerVolumes(string) []string { return nil }
func (f *t2rFakeProvider) StopStack(name string) error {