Tier-2 engine rework: class-driven legs, v2 layout, NAS-target exclusion (Task 3b, v0.135.0)

tier2_capture.go: classified apps get TierSecondary per-bind legs (paperless copy shrinks — export
drops); legacy apps keep the byte-identical resolver set. v2 relpath-mirroring layout
(backups/secondary/<stack>/{marker LAST, recovery-unit/, hdd/<rel>/, userdata/<rel>/}); N>1 native
(errTier2MultiDir/tier2AppDataName deleted). Migration=delete-and-rebuild + reconcile; all RemoveAll
via tier2SafeRemove (refuses outside backups/secondary/). SSD=state-only tier. selectTier2Target
never picks network storage (pinned+auto, F-6C-1). Restore reads v2 behind a marker gate.
Part 0: offbox_enlarge_blocked is a persisted one-time Load seed (opt-out sticks), not a getter
append. Part 0.5: offsite restore scratch prefers a local (non-network) path.
Full v2 test suite + all 10 §10 red-proofs verified. Destructive writes bounded to backups/secondary/.
This commit is contained in:
2026-07-15 10:10:20 +02:00
parent 1245a6c46e
commit 3603d1fc7f
20 changed files with 1014 additions and 484 deletions
+211 -73
View File
@@ -14,26 +14,29 @@ import (
"gitea.dooplex.hu/admin/felhom-controller/internal/system"
)
// Tier 2 = an off-drive (different physical disk) copy of an HDD app's recovery unit + its resolved
// appdata/<name> dir(s). It does NOT copy the browsable userdata tree (F-S1: userdata is not backed
// up at any tier yet — that gap is owned by the classification redesign, see
// felhom.eu/documentation/audits/SPIKE-backup-classification-2026-07-14.md). The appdata dir NAME is
// derived from the app's compose binds, NOT assumed to be the stack name (F-S2: paperless-ngx writes
// appdata/paperless); see tier2AppDataName. Auto-enabled for every HDD app; the target is auto-picked:
// prefer another registered user-data drive (can hold bulk), else the internal SSD for SMALL units
// only — and the SSD is the guest rootfs (~8 GB), so we REFUSE rather than fill it (a size-aware
// headroom guard). When no off-drive target fits, we record an honest "needs a 2nd HDD" status
// instead of silently doing nothing useful.
// Tier 2 = an off-drive (different physical disk) copy of an app's recovery unit + its CLASS-DRIVEN
// capture legs (Task 3b, architecture §2/§8). For a classified app the legs are the Task-3-core
// TierSecondary set — per-bind mandatory + optional HDD/userdata paths (so a classified app's copy
// tracks exactly what the catalog classed non-excluded; paperless's copy legitimately shrinks as
// `export` drops out). For a legacy app the legs are the resolver appdata dir(s), byte-identical to
// v0.134.x (the SQ5 footprint promise), mapped into the same layout. The dest is the v2
// relpath-mirroring layout (hdd/<rel>, userdata/<rel>, recovery-unit/, .felhom-tier2-layout marker) —
// N>1 dirs + nested binds represented natively (the old flat-appdata N>1 refusal is gone), migration
// is delete-and-rebuild of the derived copy, and a reconcile step prunes dest dirs a bind no longer
// covers. Target auto-pick: another registered NON-NETWORK user-data drive (holds the full set), else
// the internal SSD for the STATE-ONLY set (unit + mandatory), rootfs-headroom-guarded. NETWORK (NAS)
// storage is never a target (F-6C-1: rsync -og under root_squash → silently-wrong-owner restore). No
// off-drive target ⇒ an honest recorded status, not a silent no-op.
const gibibyte = 1024 * 1024 * 1024
var (
errNoOffDiskTarget = errors.New("no off-drive target (single drive, app already on the system disk)")
errSSDNoHeadroom = errors.New("the internal SSD lacks headroom for this app's data — a 2nd drive is required for off-drive backup")
// errTier2MultiDir is raised when an app's compose resolves to MORE THAN ONE distinct appdata
// dir under <hddPath>/appdata (no catalog app does today). Tier 2's destination layout is flat
// (<destBase>/appdata), so it refuses rather than silently collapse two source dirs into one.
errTier2MultiDir = errors.New("az alkalmazáshoz több adatkönyvtár tartozik — a 2. mentés jelenleg alkalmazásonként egy könyvtárat támogat")
// errTier2NetworkOnly (F-6C-1): the only off-disk candidate is a NAS network share, which can never
// be a tier-2 target — rsync -og chowns under a root_squash export and a wrong-owner restore is
// silently broken. The message IS the customer-facing reason.
errTier2NetworkOnly = errors.New("Hálózati tároló nem lehet a 2. mentés célja — a fájltulajdonos-adatok megőrzése ott nem garantálható.")
)
// appDataDirNames resolves the app's real appdata dir name(s) under hddPath from its compose HDD
@@ -47,17 +50,6 @@ func (m *Manager) appDataDirNames(stackName, hddPath string) []string {
return AppDataDirNames(hddPath, stackName, mounts)
}
// tier2AppDataName resolves the SINGLE appdata dir name for tier-2's flat destination. N>1 distinct
// names → errTier2MultiDir (the one place the tier-2 multi-dir refusal is built). It always returns
// at least one name from appDataDirNames' fallback, so name is meaningful only when err == nil.
func (m *Manager) tier2AppDataName(stackName, hddPath string) (string, error) {
names := m.appDataDirNames(stackName, hddPath)
if len(names) > 1 {
return "", errTier2MultiDir
}
return names[0], nil
}
// tier2AppDataBindsPresent reports whether the app's compose declares an appdata bind (drives the
// WARN-on-missing-declared-dir rule; nil provider → false).
func (m *Manager) tier2AppDataBindsPresent(stackName, hddPath string) bool {
@@ -72,6 +64,7 @@ type Tier2Target struct {
NamespaceRoot string // felhom-data namespace root on the target drive
Label string // human label (UI)
IsSystemDrive bool // target is the internal SSD/system drive (DB/config only)
StateOnly bool // §2.2: this target carries unit + MANDATORY only (optional legs skipped) — the SSD tier
Reason string // why this target (Hungarian, for UI/logs)
}
@@ -87,22 +80,30 @@ func tier2FitsHeadroom(availGB, totalGB, unitGB float64) bool {
}
// selectTier2Target picks the off-drive destination for an app's Tier 2 copy. A customer-pinned
// target (PreferredTarget, set from the config panel) wins when it is still valid; otherwise it
// auto-picks: another user-data drive, else the internal SSD for small units (headroom-guarded).
func (m *Manager) selectTier2Target(stackName string, unitSizeBytes int64) (*Tier2Target, error) {
// target wins when still valid; otherwise it auto-picks: another user-data drive (holds the FULL set),
// else the internal SSD for the STATE-ONLY set (unit + mandatory), headroom-guarded. NETWORK storage
// (NAS) is never a valid target at any step — pinned or auto (F-6C-1: rsync -og under root_squash →
// silently-wrong-owner restore). fullSize sizes the real-drive path; stateOnlySize sizes the SSD path.
func (m *Manager) selectTier2Target(stackName string, fullSize, stateOnlySize int64) (*Tier2Target, error) {
sourceDrive := m.GetAppDrivePath(stackName)
if sourceDrive == "" {
return nil, fmt.Errorf("no source drive for %s", stackName)
}
sawNetworkCandidate := false // an off-disk candidate existed but was network-only → better reason
// 0. Honor a customer-pinned target if it is still valid (registered, schedulable, off-disk).
// An invalid pin (gone / same physical disk) silently falls through to the auto-pick.
// 0. Honor a customer-pinned target if it is still valid (registered, schedulable, off-disk,
// NON-network). An invalid pin (gone / same disk / network) silently falls through to auto.
if m.settings != nil {
if cd := m.settings.GetCrossDriveConfig(stackName); cd != nil && cd.PreferredTarget != "" {
for _, sp := range m.settings.GetSchedulableStoragePaths() {
if sp.Path != cd.PreferredTarget {
continue
}
if sp.IsNetwork() {
m.logger.Printf("[INFO] [backup] Tier 2 %s: pinned target %s is network storage — invalid (F-6C-1); falling through to auto", stackName, sp.Path)
sawNetworkCandidate = true
break
}
if sp.Path == sourceDrive || system.SamePhysicalDevice(sourceDrive, sp.Path) {
break // pinned target is on the same physical disk — not off-drive; fall through
}
@@ -113,19 +114,22 @@ func (m *Manager) selectTier2Target(stackName string, unitSizeBytes int64) (*Tie
return &Tier2Target{
NamespaceRoot: NamespaceRoot(sp.Path, true),
Label: label,
IsSystemDrive: false,
Reason: "kézi választás",
}, nil
}
}
}
// 1. Prefer another registered user-data drive on a DIFFERENT physical disk (can hold bulk userdata).
// 1. Prefer another registered user-data drive on a DIFFERENT physical disk, NON-network.
if m.settings != nil {
for _, sp := range m.settings.GetSchedulableStoragePaths() {
if sp.Path == sourceDrive || system.SamePhysicalDevice(sourceDrive, sp.Path) {
continue
}
if sp.IsNetwork() {
sawNetworkCandidate = true // F-6C-1: never a tier-2 target
continue
}
label := sp.Label
if label == "" {
label = filepath.Base(sp.Path)
@@ -133,24 +137,31 @@ func (m *Manager) selectTier2Target(stackName string, unitSizeBytes int64) (*Tie
return &Tier2Target{
NamespaceRoot: NamespaceRoot(sp.Path, true), // Model A: in-guest mount IS the namespace root
Label: label,
IsSystemDrive: false,
Reason: "másik adatmeghajtó",
}, nil
}
}
// 2. Fall back to the internal SSD (system data path) — SMALL units only.
// 2. Fall back to the internal SSD (system data path) — STATE-ONLY set only.
sys := m.systemDataPath
if sys == "" || system.SamePhysicalDevice(sourceDrive, sys) {
if sawNetworkCandidate {
return nil, errTier2NetworkOnly // the only off-disk candidate was a NAS
}
return nil, errNoOffDiskTarget // single drive / app already on the system disk
}
if !m.tier2FitsSystemDrive(sys, unitSizeBytes) {
fits := m.tier2FitsSystemDrive
if m.tier2SSDFits != nil {
fits = m.tier2SSDFits
}
if !fits(sys, stateOnlySize) {
return nil, errSSDNoHeadroom // would fill the ~8 GB rootfs — refuse, don't fill
}
return &Tier2Target{
NamespaceRoot: NamespaceRoot(sys, false), // system path is a real root → felhom-data appended
Label: "belső SSD (rendszer)",
IsSystemDrive: true,
StateOnly: true,
Reason: "nincs 2. adatmeghajtó — csak az adatbázis/konfiguráció fér a belső SSD-re; a nagy fájlokhoz 2. meghajtó kell",
}, nil
}
@@ -164,9 +175,97 @@ func (m *Manager) tier2FitsSystemDrive(sys string, unitSizeBytes int64) bool {
return tier2FitsHeadroom(di.AvailGB, di.TotalGB, float64(unitSizeBytes)/gibibyte)
}
// RunTier2 makes/refreshes the off-drive copy of a single HDD app's recovery unit + userdata.
// Best-effort and idempotent (rsync mirror). Records status into settings for the UI; returns an
// error only on an actual copy failure (no valid target is a recorded status, not an error).
// Tier-2 v2 layout (Task 3b, architecture §8): backups/secondary/<stack>/ holds
// .felhom-tier2-layout (marker file, content "2" — written LAST, after all legs + reconcile)
// recovery-unit/ (the unit leg, layout-identical to v1)
// hdd/<relpath>/ (per-bind HDD legs, relpath-mirroring)
// userdata/<relpath>/ (per-bind USERDATA legs)
// Relpath-mirroring represents N>1 dirs + nested binds natively (the v1 flat-appdata N>1 refusal is
// lifted structurally) and makes restore position-derivable (dest relpath → live path under the app's
// current HDD_PATH). The whole tree is DERIVED from live data — migration is delete-and-rebuild.
const (
tier2LayoutMarker = ".felhom-tier2-layout"
tier2LayoutVersion = "2"
)
// tier2SafeRemove os.RemoveAll's target ONLY if it is strictly WITHIN a backups/secondary/<...>
// destBase (§9.2 destructive-write boundary — defense in depth against a mapping bug ever pointing a
// removal at live data; red-proofed). Never removes destBase itself.
func tier2SafeRemove(destBase, target string) error {
cb := filepath.Clean(destBase)
ct := filepath.Clean(target)
if !strings.Contains(filepath.ToSlash(cb), "/backups/secondary/") {
return fmt.Errorf("refusing tier-2 removal: destBase %q is not under backups/secondary/", cb)
}
if !strings.HasPrefix(ct, cb+string(filepath.Separator)) {
return fmt.Errorf("refusing tier-2 removal: %q is not strictly within %q", ct, cb)
}
return os.RemoveAll(ct)
}
// tier2RelClass classifies a dest dir (relpath dirRel, slash-form) against the current leg relpaths:
// keepInside = the dir is a leg or inside one (keep, don't descend); keepAncestor = on the path to a
// leg (keep, descend to find stale deeper); stale = neither (remove the topmost).
type tier2RelClass int
const (
tier2Stale tier2RelClass = iota
tier2KeepInside
tier2KeepAncestor
)
func classifyTier2Rel(dirRel string, legRels []string) tier2RelClass {
for _, lr := range legRels {
if dirRel == lr || strings.HasPrefix(dirRel, lr+"/") {
return tier2KeepInside // exact leg or descendant content
}
}
for _, lr := range legRels {
if strings.HasPrefix(lr, dirRel+"/") {
return tier2KeepAncestor // on the path to a deeper leg
}
}
return tier2Stale
}
// tier2Reconcile removes dest subdirs under hdd/ and userdata/ that are neither an ancestor nor a
// descendant of any current leg relpath (§7-D — the deferred-pruning answer: a bind removed/re-classed
// stops occupying the secondary drive within one run). Runs strictly inside destBase.
func (m *Manager) tier2Reconcile(destBase string, legRels []string) {
var walk func(dirAbs, dirRel string)
walk = func(dirAbs, dirRel string) {
entries, err := os.ReadDir(dirAbs)
if err != nil {
return
}
for _, e := range entries {
if !e.IsDir() {
continue
}
childRel := dirRel + "/" + e.Name()
childAbs := filepath.Join(dirAbs, e.Name())
switch classifyTier2Rel(childRel, legRels) {
case tier2KeepInside:
// mirrored leg content — keep, no descent
case tier2KeepAncestor:
walk(childAbs, childRel)
default:
if err := tier2SafeRemove(destBase, childAbs); err != nil {
m.logger.Printf("[WARN] [backup] Tier 2 reconcile: %v", err)
} else {
m.logger.Printf("[INFO] [backup] Tier 2 reconcile: removed stale dest dir %s", childRel)
}
}
}
}
for _, root := range []string{"hdd", "userdata"} {
walk(filepath.Join(destBase, root), root)
}
}
// RunTier2 makes/refreshes the off-drive v2 copy of a single app's recovery unit + its class-driven
// capture legs (§8). Best-effort and idempotent. Records status for the UI; returns an error only on
// an actual copy failure (no valid target is a recorded status, not an error).
func (m *Manager) RunTier2(stackName string) error {
// Customer turned Tier 2 off for this app (config panel) — skip without touching status.
if m.settings != nil {
@@ -181,23 +280,25 @@ func (m *Manager) RunTier2(stackName string) error {
}
sourceNsRoot := m.namespaceRoot(sourceDrive)
unitDir := RecoveryUnitPath(sourceNsRoot, stackName)
// F-S2: resolve the app's REAL appdata dir name from its compose binds (paperless-ngx writes
// appdata/paperless, not appdata/paperless-ngx). For an HDD app HDD_PATH == nsRoot (Model A), so
// the mounts (resolved against HDD_PATH) share the nsRoot prefix. N>1 distinct names → refuse.
appDataName, resErr := m.tier2AppDataName(stackName, sourceNsRoot)
if resErr != nil {
m.recordTier2NoTarget(stackName, resErr.Error())
m.logger.Printf("[ERROR] [backup] Tier 2 for %s refused: %v", stackName, resErr)
return nil
}
appDataDir := AppDataDir(sourceNsRoot, appDataName)
if _, err := os.Stat(unitDir); err != nil {
return nil // no recovery unit yet — nothing to copy
}
unitSize := dirSizeBytes(unitDir) + dirSizeBytes(appDataDir)
// Class-driven legs (classified) / resolver legs (legacy) + loud gap warnings (§7-H).
legs, warns := m.tier2CaptureSet(stackName, sourceNsRoot)
target, err := m.selectTier2Target(stackName, unitSize)
// Two sizes: full = unit + all legs; state-only = unit + mandatory legs (the SSD ceiling).
unitSize := dirSizeBytes(unitDir)
var fullSize, stateOnlySize int64 = unitSize, unitSize
for _, lg := range legs {
sz := dirSizeBytes(lg.Src)
fullSize += sz
if lg.Class == ClassMandatory {
stateOnlySize += sz
}
}
target, err := m.selectTier2Target(stackName, fullSize, stateOnlySize)
if err != nil {
reason := tier2NoTargetReason(err)
m.recordTier2NoTarget(stackName, reason)
@@ -210,14 +311,41 @@ func (m *Manager) RunTier2(stackName string) error {
return nil
}
// §2.2: the SSD is a STATE-ONLY tier — drop optional legs, tell the customer honestly.
if target.StateOnly {
kept := legs[:0]
droppedOptional := false
for _, lg := range legs {
if lg.Class == ClassMandatory {
kept = append(kept, lg)
} else {
droppedOptional = true
}
}
legs = kept
if droppedOptional {
warns = append(warns, "A belső SSD-n csak a konfiguráció, adatbázis és a kötelező adatok férnek el — a választható tartalom nem került másolásra.")
}
}
destBase := filepath.Join(target.NamespaceRoot, "backups", "secondary", stackName)
start := time.Now()
mirror := m.tier2Mirror
if mirror == nil {
mirror = rsyncMirror
}
// Migration = delete-and-rebuild (marker LAST). First v2 run (marker absent): remove the old flat
// appdata/ leg (recovery-unit/ is layout-identical — untouched). A half-migrated dest self-heals on
// the next run because the marker is only written after every leg + reconcile succeed.
markerPath := filepath.Join(destBase, tier2LayoutMarker)
if _, mErr := os.Stat(markerPath); mErr != nil {
if rmErr := tier2SafeRemove(destBase, filepath.Join(destBase, "appdata")); rmErr != nil && !os.IsNotExist(rmErr) {
m.logger.Printf("[WARN] [backup] Tier 2 %s: migration cleanup of old flat appdata failed: %v", stackName, rmErr)
}
}
// Unit leg (always).
if err := mirror(unitDir, filepath.Join(destBase, "recovery-unit")); err != nil {
m.recordTier2Failure(stackName, target, err)
if m.tier2Notify != nil {
@@ -225,29 +353,35 @@ func (m *Manager) RunTier2(stackName string) error {
}
return fmt.Errorf("tier2 rsync unit for %s: %w", stackName, err)
}
if _, e := os.Stat(appDataDir); e == nil {
if err := mirror(appDataDir, filepath.Join(destBase, "appdata")); err != nil {
// Capture legs.
legRels := make([]string, 0, len(legs))
mirroredSize := unitSize
for _, lg := range legs {
if err := mirror(lg.Src, filepath.Join(destBase, filepath.FromSlash(lg.DestRel))); err != nil {
m.recordTier2Failure(stackName, target, err)
if m.tier2Notify != nil {
m.tier2Notify(stackName, target.Label, time.Since(start), err)
}
return fmt.Errorf("tier2 rsync appdata for %s: %w", stackName, err)
return fmt.Errorf("tier2 rsync leg %s for %s: %w", lg.DestRel, stackName, err)
}
} else if m.tier2AppDataBindsPresent(stackName, sourceNsRoot) {
// F-S2: the compose DECLARES an appdata bind but the dir is missing on disk. Skipping is kept
// (nothing to copy) but the silence that hid F-S2 for months is now a loud WARN.
m.logger.Printf("[WARN] [backup] Tier 2 for %s: compose declares appdata dir %q but it is absent at %s — appdata leg skipped",
stackName, appDataName, appDataDir)
legRels = append(legRels, lg.DestRel)
mirroredSize += dirSizeBytes(lg.Src)
}
// Reconcile stale dest dirs (a bind removed / re-classed excluded), then write the marker LAST.
m.tier2Reconcile(destBase, legRels)
if err := os.WriteFile(markerPath, []byte(tier2LayoutVersion), 0644); err != nil {
m.logger.Printf("[WARN] [backup] Tier 2 %s: layout marker write failed (restore will refuse until next run): %v", stackName, err)
}
dur := time.Since(start)
m.recordTier2Success(stackName, target, unitSize, dur)
m.recordTier2Success(stackName, target, mirroredSize, strings.Join(warns, " "), dur)
if m.tier2Notify != nil {
m.tier2Notify(stackName, target.Label, dur, nil)
}
m.logger.Printf("[INFO] [backup] Tier 2 copied %s → %s (%s, %s)%s",
stackName, destBase, humanizeBytes(unitSize), dur.Round(time.Second),
map[bool]string{true: " [SSD: DB/config only]", false: ""}[target.IsSystemDrive])
m.logger.Printf("[INFO] [backup] Tier 2 copied %s → %s (%s, %d leg(s), %s)%s",
stackName, destBase, humanizeBytes(mirroredSize), len(legs), dur.Round(time.Second),
map[bool]string{true: " [SSD: state-only]", false: ""}[target.StateOnly])
return nil
}
@@ -335,17 +469,20 @@ func (m *Manager) Tier2Info(stackName string) Tier2Info {
}
}
// Resolve what the runner WOULD pick right now (real unit size feeds the SSD headroom guard).
// F-S2: N>1 distinct appdata dirs → the same honest refusal the runner records.
// Resolve what the runner WOULD pick right now: the v2 capture legs feed both sizes (the SSD
// headroom guard tests the state-only set).
sourceNsRoot := m.namespaceRoot(source)
appDataName, resErr := m.tier2AppDataName(stackName, sourceNsRoot)
if resErr != nil {
info.NoTarget = true
info.NoTargetReason = resErr.Error()
return info
legs, _ := m.tier2CaptureSet(stackName, sourceNsRoot)
unitSize := dirSizeBytes(RecoveryUnitPath(sourceNsRoot, stackName))
fullSize, stateOnlySize := unitSize, unitSize
for _, lg := range legs {
sz := dirSizeBytes(lg.Src)
fullSize += sz
if lg.Class == ClassMandatory {
stateOnlySize += sz
}
}
unitSize := dirSizeBytes(RecoveryUnitPath(sourceNsRoot, stackName)) + dirSizeBytes(AppDataDir(sourceNsRoot, appDataName))
target, err := m.selectTier2Target(stackName, unitSize)
target, err := m.selectTier2Target(stackName, fullSize, stateOnlySize)
if err != nil {
info.NoTarget = true
info.NoTargetReason = tier2NoTargetReason(err)
@@ -371,7 +508,7 @@ func (m *Manager) withTier2Prefs(stackName string, cfg *settings.CrossDriveBacku
return cfg
}
func (m *Manager) recordTier2Success(stackName string, target *Tier2Target, sizeBytes int64, dur time.Duration) {
func (m *Manager) recordTier2Success(stackName string, target *Tier2Target, sizeBytes int64, warning string, dur time.Duration) {
if m.settings == nil {
return
}
@@ -382,6 +519,7 @@ func (m *Manager) recordTier2Success(stackName string, target *Tier2Target, size
Schedule: "daily",
LastRun: time.Now().Format(time.RFC3339),
LastStatus: "ok",
LastWarning: strings.TrimSpace(warning),
LastDuration: dur.Round(time.Second).String(),
LastSizeHuman: humanizeBytes(sizeBytes),
})); err != nil {