v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (R-43 + R-44)
Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.
R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.
New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
- nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
created after the snapshot survives as an extra;
- the undo exists before the act — the pre-restore- dump is verified ON DISK
before anything is stopped, overwritten or replayed; if it cannot be taken
the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.
R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.
Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.
11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.
NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.
Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
This commit is contained in:
@@ -59,6 +59,16 @@ type Manager struct {
|
||||
// offboxPlaceCopier (3a) — the place-to-live missing-only merge seam (nil → rsyncRestoreMissing,
|
||||
// the `-a --ignore-existing` additive copy). Never rsyncMirror (--delete trap).
|
||||
offboxPlaceCopier func(src, dst string) (int, error)
|
||||
// offboxFullPlaceCopier (R-43, v0.148.0) — the FULL-restore overwrite seam (nil →
|
||||
// rsyncRestoreOverwrite: `-a` with NO --ignore-existing and NO --delete). Distinct from
|
||||
// offboxPlaceCopier on purpose: the two have opposite semantics for an existing file.
|
||||
offboxFullPlaceCopier func(src, dst string) (int, error)
|
||||
// safetyDumpFn (R-43) — the pre-restore safety-dump seam (nil → the real DumpOne), so the
|
||||
// "never replay without an undo on disk" refusal is unit-testable without Docker.
|
||||
safetyDumpFn func(ctx context.Context, db DiscoveredDB, dumpDir string) DumpResult
|
||||
// offsitePreDumpFn (R-44) — the offsite dump pre-phase seam (nil → runDBDumpsInternal), so the
|
||||
// dumps-strictly-before-capture ordering is observable in a test without Docker or restic.
|
||||
offsitePreDumpFn func(ctx context.Context) error
|
||||
// offboxFreeFn (3a) — the free-space probe for the restore headroom gate, overridable in tests (the
|
||||
// Windows `go test` host has no `df`). Nil → the real diskFreeBytes (df --output=avail).
|
||||
offboxFreeFn func(path string) int64
|
||||
@@ -126,6 +136,13 @@ type Manager struct {
|
||||
lastDBDump *DBDumpStatus
|
||||
running bool
|
||||
|
||||
// R-43/R-44 (v0.148.0) — the coherence stamp of the offsite run in flight, read by
|
||||
// CaptureRecoveryUnit so each unit records WHICH run took the dumps sitting beside its files.
|
||||
// Set for the duration of the dump pre-phase + capture, cleared after; "" means "no offsite run
|
||||
// is establishing coherence right now" (the periodic refresh and the local 02:30 dump leg).
|
||||
offsiteRunID string
|
||||
offsiteRunDumpAt string
|
||||
|
||||
// Restore op-status (Part B, opstatus.go) — display-only async-restore progress, under `mu`.
|
||||
opRunning bool
|
||||
opName string
|
||||
@@ -310,6 +327,28 @@ func (m *Manager) RunDBDumps(ctx context.Context) error {
|
||||
return m.runDBDumpsInternal(ctx)
|
||||
}
|
||||
|
||||
// offsiteRunStamp returns the in-flight offsite run's coherence stamp ("" when none).
|
||||
func (m *Manager) offsiteRunStamp() (runID, dumpsAt string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.offsiteRunID, m.offsiteRunDumpAt
|
||||
}
|
||||
|
||||
// beginOffsiteRunStamp marks the start of an offsite run's coherence window and returns the cleanup.
|
||||
// The stamp is what CaptureRecoveryUnit writes into each unit manifest, so it must be live across
|
||||
// BOTH the dump leg and the unit capture that follows it — those two together are the pair.
|
||||
func (m *Manager) beginOffsiteRunStamp(runID string) func() {
|
||||
m.mu.Lock()
|
||||
m.offsiteRunID = runID
|
||||
m.offsiteRunDumpAt = time.Now().UTC().Format(time.RFC3339)
|
||||
m.mu.Unlock()
|
||||
return func() {
|
||||
m.mu.Lock()
|
||||
m.offsiteRunID, m.offsiteRunDumpAt = "", ""
|
||||
m.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// runDBDumpsInternal is the implementation of RunDBDumps. Caller must hold the running flag.
|
||||
func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
|
||||
start := time.Now()
|
||||
|
||||
@@ -658,9 +658,40 @@ func (m *Manager) runOffboxBackup(ctx context.Context, withProgress bool) error
|
||||
m.offboxRecordStats(ctx, base, env) // the prune may have brought the size back down — refresh
|
||||
runErr = fmt.Errorf("A távoli mentés túllépte a tárhelykeretet (%d/%d GB) — törölj régi mentéseket vagy kérj nagyobb keretet.", usedGB, quota)
|
||||
} else {
|
||||
// R-43/R-44 (v0.148.0) — THE COHERENCE PRE-PHASE. Refresh the DB/volume dumps and the recovery
|
||||
// units BEFORE capturing, so the snapshot restic is about to write is an internally coherent
|
||||
// {DB@T, files@T} pair. Before this, a push shipped live files beside whatever dump the 02:30
|
||||
// local run happened to leave — on 2026-07-19 that was a dump taken four hours before the
|
||||
// customer's account even existed, so the "backup" of the photos contained zero of them
|
||||
// (DIAG-immich-restore-2026-07-19).
|
||||
//
|
||||
// Order matters and is the whole mechanism: dumps FIRST, then files. The gap between the two
|
||||
// can only ADD files the DB does not reference yet (an upload landing mid-run is a harmless
|
||||
// orphan blob), never remove one the DB DOES reference — so the file set is always a superset
|
||||
// of what the restored DB points at. The reverse order would produce dangling rows.
|
||||
//
|
||||
// This runs on the NIGHTLY path too, not just the manual one: "every snapshot is a coherent
|
||||
// pair" is the property that makes retention a history of restorable points rather than a
|
||||
// history of skewed ones. It also makes the nightly ordering structural instead of a
|
||||
// coincidence of two independent scheduler entries at 02:30 and 04:15.
|
||||
endStamp := m.beginOffsiteRunStamp(start.UTC().Format("20060102T150405Z"))
|
||||
if withProgress {
|
||||
m.offboxProgress.setPhase(OffboxPhaseDump)
|
||||
}
|
||||
dumpStart := time.Now()
|
||||
if dErr := m.offsitePreDump(ctx); dErr != nil {
|
||||
// Data-first: a dump failure must NOT abort the push. The files are still worth shipping,
|
||||
// and refusing to ship them would turn a degraded backup into no backup at all. It is a
|
||||
// loud WARN, and the unit manifest simply carries the older dump set — which the restore
|
||||
// confirm then surfaces as a skewed pair (P2) rather than silently pretending otherwise.
|
||||
m.logger.Printf("[WARN] [offbox] pre-push dump leg failed (%v) — continuing with the existing dumps; the snapshot's DB half may be older than its files", dErr)
|
||||
} else {
|
||||
m.logger.Printf("[INFO] [offbox] pre-push dump leg completed in %s — snapshot pair is coherent", time.Since(dumpStart).Round(time.Millisecond))
|
||||
}
|
||||
runResult, runErr = m.runOffboxInternal(ctx, apps, base, env, t)
|
||||
backedUp = runResult.backedUp
|
||||
missing = runResult.missing
|
||||
endStamp()
|
||||
}
|
||||
// Sorted names of apps whose enlargement was blocked this run (replaces the persisted set; empty clears).
|
||||
var blockedNames []string
|
||||
|
||||
@@ -179,8 +179,10 @@ func (p *offboxProgressState) setPhase(phase string) {
|
||||
p.mu.Unlock()
|
||||
}
|
||||
|
||||
// OffboxPhaseShares / OffboxPhaseRetention are the post-app-loop stages.
|
||||
// OffboxPhaseDump is the PRE-app-loop stage (R-44, v0.148.0); OffboxPhaseShares /
|
||||
// OffboxPhaseRetention are the post-app-loop stages.
|
||||
const (
|
||||
OffboxPhaseDump = "dump"
|
||||
OffboxPhaseShares = "shares"
|
||||
OffboxPhaseRetention = "retention"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,400 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Offsite reconstitution (R-43, v0.148.0) — the leg that was missing.
|
||||
//
|
||||
// Until v0.148.0 NO offsite path could restore a database. The two „visszaállítás" buttons staged
|
||||
// files into a scratch folder and never touched postgres; the place-to-live button merged only the
|
||||
// files MISSING from the live tree (`rsync --ignore-existing`) and never replayed a dump. For a
|
||||
// DB-indexed app — most of the catalog — that combination cannot bring content back: the bytes
|
||||
// return and the application still cannot see them, because its index lives in the database.
|
||||
// Measured live on 2026-07-19 (DIAG-immich-restore-2026-07-19): 11 photos, files intact on disk,
|
||||
// timeline empty, two "successful" restores that merged 0 files.
|
||||
//
|
||||
// ReconstituteFromOffsite is the honest version of that operation: it takes the CHOSEN snapshot's
|
||||
// coherent pair and makes the live app equal to it — files overwritten to the snapshot's version,
|
||||
// database replayed from the same snapshot's dump, app restarted. It is deliberately a different
|
||||
// function from PlaceOffsiteRestore rather than a flag on it, because the two have opposite file
|
||||
// semantics and conflating them is exactly how the missing-only merge came to be presented as a
|
||||
// restore.
|
||||
//
|
||||
// Two invariants hold throughout:
|
||||
//
|
||||
// - NOTHING IS EVER DELETED. The file copy overwrites and adds; it never carries `--delete`. A
|
||||
// file the customer created after the snapshot survives the restore as an extra. That is the
|
||||
// house boundary — a restore that silently removed newer work would be a data-loss event
|
||||
// wearing a recovery button's label.
|
||||
// - THE UNDO EXISTS BEFORE THE ACT. A safety dump of the live database is written, and verified
|
||||
// present on disk, BEFORE anything is stopped, overwritten or replayed. If that dump cannot be
|
||||
// taken, the whole operation refuses with zero changes — a replay whose previous state was not
|
||||
// captured is not a restore, it is an overwrite with no way back.
|
||||
|
||||
// offsitePreDump runs the coherence pre-phase's dump leg (nil seam → runDBDumpsInternal, which also
|
||||
// refreshes the recovery units so the manifests enumerate the dumps just written). Extracted as a
|
||||
// seam because the ORDER — dumps strictly before the restic capture — is the entire mechanism of
|
||||
// R-44, and an ordering guarantee that no test can observe is one refactor away from silently
|
||||
// reverting to the behaviour that produced DIAG-immich-restore-2026-07-19.
|
||||
func (m *Manager) offsitePreDump(ctx context.Context) error {
|
||||
if m.offsitePreDumpFn != nil {
|
||||
return m.offsitePreDumpFn(ctx)
|
||||
}
|
||||
return m.runDBDumpsInternal(ctx)
|
||||
}
|
||||
|
||||
// SetOffsitePreDumpFn overrides the offsite dump pre-phase (tests; no Docker needed).
|
||||
func (m *Manager) SetOffsitePreDumpFn(fn func(ctx context.Context) error) { m.offsitePreDumpFn = fn }
|
||||
|
||||
// preRestoreDumpPrefix marks the safety dumps taken immediately before a reconstitution. They live
|
||||
// in the app's own unit db-dumps dir so `ListDumpFiles` surfaces them beside the regular dumps —
|
||||
// they ARE the undo, and an undo the customer cannot see is not much of one. The regular replay
|
||||
// loop matches `<stack>-<dbtype>.sql` exactly, so a prefixed file is never mistaken for a source.
|
||||
const preRestoreDumpPrefix = "pre-restore-"
|
||||
|
||||
// OffsiteReconstituteResult reports what a reconstitution actually did, so the flash can state an
|
||||
// OUTCOME instead of a mechanism. Every field here exists because the v0.147 flash could not say it.
|
||||
type OffsiteReconstituteResult struct {
|
||||
SnapshotID string
|
||||
FilesPlaced int
|
||||
DBsReplayed int
|
||||
SafetyDump string // path of the pre-restore dump (the undo), "" when the app has no DB
|
||||
DumpsAt time.Time // when the snapshot's DB half was taken (zero = unknown/legacy unit)
|
||||
OffsiteRunID string // "" for a pre-v0.148 snapshot — an unverified pair
|
||||
Skewed bool // the snapshot carries no coherence stamp: files and DB may differ in age
|
||||
LooksEmpty bool // R-44 sniff on the dump about to be replayed
|
||||
}
|
||||
|
||||
// fullPlaceCopier returns the FULL-restore file copier (nil seam → rsyncRestoreOverwrite).
|
||||
// Deliberately NOT placeCopier(): that one is `--ignore-existing`, whose whole purpose is to leave
|
||||
// live files alone, which is precisely what a full restore must not do.
|
||||
func (m *Manager) fullPlaceCopier() func(src, dst string) (int, error) {
|
||||
if m.offboxFullPlaceCopier != nil {
|
||||
return m.offboxFullPlaceCopier
|
||||
}
|
||||
return rsyncRestoreOverwrite
|
||||
}
|
||||
|
||||
// rsyncRestoreOverwrite copies src over dst: `rsync -a --itemize-changes`, with NO
|
||||
// `--ignore-existing` (a changed file becomes the snapshot's version) and NO `--delete` (an extra
|
||||
// file at dst survives). Returns the number of regular files transferred.
|
||||
func rsyncRestoreOverwrite(src, dst string) (int, error) {
|
||||
if err := os.MkdirAll(dst, 0755); err != nil {
|
||||
return 0, fmt.Errorf("mkdir %s: %w", dst, err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Minute)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, "rsync", "-a", "--itemize-changes",
|
||||
strings.TrimRight(src, "/")+"/", strings.TrimRight(dst, "/")+"/")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return countRestoredFiles(string(out)), nil
|
||||
}
|
||||
|
||||
// writeSafetyDump dumps every live database of stack into the app's unit db-dumps dir under the
|
||||
// `pre-restore-` prefix, and returns the first dump's path. Returns ("", nil) when the app has no
|
||||
// database at all — a no-DB app has nothing to undo and must flow exactly as it did before
|
||||
// v0.148.0 (no dump, no replay, no behaviour change).
|
||||
//
|
||||
// A discovered database that CANNOT be dumped is a hard error: it means the undo would not exist.
|
||||
func (m *Manager) writeSafetyDump(ctx context.Context, stackName, nsRoot string) (string, error) {
|
||||
discover := m.discoverDBs
|
||||
if discover == nil {
|
||||
discover = func(ctx context.Context) ([]DiscoveredDB, error) {
|
||||
return DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames())
|
||||
}
|
||||
}
|
||||
dbs, err := discover(ctx)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("a biztonsági mentés előtt nem sikerült felderíteni az adatbázisokat: %w", err)
|
||||
}
|
||||
var mine []DiscoveredDB
|
||||
for _, db := range dbs {
|
||||
if db.StackName == stackName {
|
||||
mine = append(mine, db)
|
||||
}
|
||||
}
|
||||
if len(mine) == 0 {
|
||||
return "", nil // no DB → nothing to undo → scenario E flows unchanged
|
||||
}
|
||||
|
||||
dumpDir := AppDBDumpPath(nsRoot, stackName)
|
||||
if err := os.MkdirAll(dumpDir, 0755); err != nil {
|
||||
return "", fmt.Errorf("a biztonsági mentés könyvtára nem hozható létre: %w", err)
|
||||
}
|
||||
stamp := time.Now().UTC().Format("20060102T150405Z")
|
||||
first := ""
|
||||
for _, db := range mine {
|
||||
res := m.dumpForSafety(ctx, db, dumpDir)
|
||||
if res.Error != nil {
|
||||
return "", fmt.Errorf("a jelenlegi adatbázis biztonsági mentése sikertelen (%s): %w — a visszaállítás nem indult el", db.ContainerName, res.Error)
|
||||
}
|
||||
// DumpOne writes `<stack>-<dbtype>.sql`; rename it under the safety prefix so it can never be
|
||||
// picked up as a replay SOURCE and can never overwrite the app's real dump.
|
||||
safe := filepath.Join(dumpDir, fmt.Sprintf("%s%s-%s-%s.sql", preRestoreDumpPrefix, stamp, stackName, db.DBType))
|
||||
if res.FilePath != safe {
|
||||
if err := os.Rename(res.FilePath, safe); err != nil {
|
||||
return "", fmt.Errorf("a biztonsági mentés véglegesítése sikertelen: %w", err)
|
||||
}
|
||||
}
|
||||
if first == "" {
|
||||
first = safe
|
||||
}
|
||||
m.logger.Printf("[INFO] [offbox] %s: pre-restore safety dump written → %s (%s)", stackName, filepath.Base(safe), humanizeBytes(res.Size))
|
||||
}
|
||||
return first, nil
|
||||
}
|
||||
|
||||
// dumpForSafety is the DumpOne seam for the safety dump (tests inject; nil → the real DumpOne).
|
||||
func (m *Manager) dumpForSafety(ctx context.Context, db DiscoveredDB, dumpDir string) DumpResult {
|
||||
if m.safetyDumpFn != nil {
|
||||
return m.safetyDumpFn(ctx, db, dumpDir)
|
||||
}
|
||||
return DumpOne(ctx, db, dumpDir, m.logger, m.isDebug())
|
||||
}
|
||||
|
||||
// ReconstituteFromOffsite makes the live app equal to a restored full-scratch snapshot: files
|
||||
// overwritten to the snapshot's version (extras survive, nothing deleted), then the snapshot's own
|
||||
// DB dump replayed, with a safety dump of the current database taken first. Requires a completed
|
||||
// FULL scratch restore (RestoreOffboxScratch with full=true). Single-flight.
|
||||
func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (OffsiteReconstituteResult, error) {
|
||||
var res OffsiteReconstituteResult
|
||||
if !m.OffboxConfigured() {
|
||||
return res, fmt.Errorf("off-box backup not configured")
|
||||
}
|
||||
if !isSafeStackName(stack) {
|
||||
return res, fmt.Errorf("invalid stack name")
|
||||
}
|
||||
if m.stackProvider == nil {
|
||||
return res, fmt.Errorf("stack provider not configured")
|
||||
}
|
||||
if err := m.acquireRunning(); err != nil {
|
||||
return res, fmt.Errorf("egy másik mentési/visszaállítási művelet már fut")
|
||||
}
|
||||
defer m.releaseRunning()
|
||||
|
||||
scratch, _, err := m.offboxRestoreScratchDir(stack)
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
if _, sErr := os.Stat(scratch); sErr != nil {
|
||||
return res, fmt.Errorf("nincs előkészített teljes visszaállítás — futtass előbb egy teljes visszaállítást")
|
||||
}
|
||||
id, paths, err := m.offboxLatestSnapshot(ctx, stack)
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
res.SnapshotID = id
|
||||
|
||||
hdd := strings.TrimSpace(m.stackProvider.GetStackHDDPath(stack))
|
||||
if hdd == "" {
|
||||
return res, fmt.Errorf("a(z) %s nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat", stack)
|
||||
}
|
||||
liveNs := m.namespaceRoot(hdd)
|
||||
|
||||
placements, err := mapOffsiteRestorePaths(paths, stack, scratch, liveNs)
|
||||
if err != nil {
|
||||
return res, err // whole-placement refusal (no partial writes)
|
||||
}
|
||||
// Stat pre-pass over EVERY placement before the first copy — an incomplete scratch (e.g. only a
|
||||
// unit-only restore was run) refuses with ZERO copies.
|
||||
for _, pl := range placements {
|
||||
if _, sErr := os.Stat(pl.src); sErr != nil {
|
||||
return res, fmt.Errorf("a teljes visszaállítás hiányos (%s nincs meg) — futtass előbb egy teljes visszaállítást", filepath.Base(pl.src))
|
||||
}
|
||||
}
|
||||
|
||||
// The snapshot's coherence stamp, read from the RESTORED unit manifest (not the live one).
|
||||
scratchUnit := ""
|
||||
for _, pl := range placements {
|
||||
if pl.isUnit {
|
||||
scratchUnit = pl.src
|
||||
break
|
||||
}
|
||||
}
|
||||
if scratchUnit == "" {
|
||||
return res, fmt.Errorf("a pillanatképben nincs mentési egység — a visszaállítás nem indítható")
|
||||
}
|
||||
scratchDumpDir := filepath.Join(scratchUnit, "db-dumps")
|
||||
if man := readManifest(filepath.Join(scratchUnit, "manifest.json")); man != nil {
|
||||
res.OffsiteRunID = man.OffsiteRunID
|
||||
if man.DumpsAt != "" {
|
||||
if t, pErr := time.Parse(time.RFC3339, man.DumpsAt); pErr == nil {
|
||||
res.DumpsAt = t
|
||||
}
|
||||
}
|
||||
}
|
||||
// A pre-v0.148 snapshot carries no stamp: its dump was whatever the 02:30 local run left behind,
|
||||
// so the pair's two halves may be hours or days apart. Surfaced, never blocked — the confirm
|
||||
// dialog says so and the safety dump makes it reversible.
|
||||
res.Skewed = res.OffsiteRunID == ""
|
||||
res.LooksEmpty = m.sniffScratchDump(scratchDumpDir, stack)
|
||||
|
||||
// --- THE UNDO, BEFORE THE ACT ---------------------------------------------------------------
|
||||
// Taken while the stack is still UP (a stopped database cannot be dumped) and before a single
|
||||
// byte is overwritten, so a failure here aborts with the live app completely untouched.
|
||||
safety, err := m.writeSafetyDump(ctx, stack, liveNs)
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
res.SafetyDump = safety
|
||||
hasDB := safety != ""
|
||||
if hasDB {
|
||||
if _, sErr := os.Stat(safety); sErr != nil {
|
||||
// Fail-closed: never replay when the undo is not verifiably on disk.
|
||||
return res, fmt.Errorf("a biztonsági mentés nem található a lemezen — a visszaállítás biztonsági okból nem indult el")
|
||||
}
|
||||
}
|
||||
|
||||
// --- FILES ----------------------------------------------------------------------------------
|
||||
if err := m.stackProvider.StopStack(stack); err != nil {
|
||||
m.logger.Printf("[WARN] [offbox] could not stop %s before reconstitution: %v (continuing)", stack, err)
|
||||
}
|
||||
copier := m.fullPlaceCopier()
|
||||
for _, pl := range placements {
|
||||
if pl.isUnit {
|
||||
// The live recovery unit is still never overwritten — it is the LOCAL restore path's
|
||||
// source and clobbering it would trade one recovery route for another. The snapshot's
|
||||
// dump is replayed from the scratch unit instead, so nothing is lost by skipping it.
|
||||
continue
|
||||
}
|
||||
n, cErr := copier(pl.src, pl.dst)
|
||||
if cErr != nil {
|
||||
// Best-effort bring-up: leaving the app stopped after a partial copy would turn a failed
|
||||
// restore into an outage.
|
||||
if sErr := m.stackProvider.StartStack(stack); sErr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] %s: restart after failed placement also failed: %v", stack, sErr)
|
||||
}
|
||||
return res, fmt.Errorf("a(z) %s fájljainak visszaállítása sikertelen: %w", stack, cErr)
|
||||
}
|
||||
res.FilesPlaced += n
|
||||
}
|
||||
|
||||
// --- DATABASE -------------------------------------------------------------------------------
|
||||
// The stack must be UP for the replay: ImportDump talks to the running container using its own
|
||||
// discovered credentials (the same precedence RestoreFromRecoveryUnit uses — the logical dump
|
||||
// wins over whatever the file copy just laid down for the DB's own data dir).
|
||||
if err := m.stackProvider.StartStack(stack); err != nil {
|
||||
return res, fmt.Errorf("a(z) %s újraindítása sikertelen a fájlok visszaállítása után: %w", stack, err)
|
||||
}
|
||||
if hasDB {
|
||||
n, iErr := m.reimportDBDumpsFrom(ctx, stack, scratchDumpDir)
|
||||
res.DBsReplayed = n
|
||||
if iErr != nil {
|
||||
return res, fmt.Errorf("az adatbázis visszaállítása sikertelen: %w — a korábbi állapot mentése megvan: %s", iErr, filepath.Base(safety))
|
||||
}
|
||||
}
|
||||
if err := m.waitForHealthy(stack, 90*time.Second); err != nil {
|
||||
m.logger.Printf("[WARN] [offbox] %s reconstituted but health check failed: %v", stack, err)
|
||||
}
|
||||
|
||||
m.logger.Printf("[INFO] [offbox] reconstituted %s from snapshot %s: %d file(s) placed, %d DB dump(s) replayed, safety dump=%s, skewed=%v",
|
||||
stack, id, res.FilesPlaced, res.DBsReplayed, filepath.Base(safety), res.Skewed)
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// OffsitePairInfo describes the {DB, files} pair sitting in a prepared full-restore scratch, so the
|
||||
// confirm dialog can tell the customer what they are about to restore BEFORE they commit to it.
|
||||
// Everything here is honesty-surface: none of it blocks the operation.
|
||||
type OffsitePairInfo struct {
|
||||
Ready bool
|
||||
DumpsAt time.Time // when the DB half was taken (zero = legacy unit, age unknown)
|
||||
Skewed bool // no coherence stamp → the two halves may be from different times
|
||||
LooksEmpty bool // R-44 sniff: the dump has an accounts table with no rows
|
||||
HasDump bool
|
||||
}
|
||||
|
||||
// OffsiteScratchPair reads the prepared scratch's unit manifest and reports what the pair looks
|
||||
// like. Cheap and read-only — safe to call from a page render.
|
||||
func (m *Manager) OffsiteScratchPair(stack string) OffsitePairInfo {
|
||||
var info OffsitePairInfo
|
||||
if !isSafeStackName(stack) {
|
||||
return info
|
||||
}
|
||||
scratch, _, err := m.offboxRestoreScratchDir(stack)
|
||||
if err != nil {
|
||||
return info
|
||||
}
|
||||
// The unit sits at <scratch>/<oldNs>/backups/primary/<stack>; the old namespace is unknown here,
|
||||
// so find it rather than reconstructing it.
|
||||
unit := findScratchUnitDir(scratch, stack)
|
||||
if unit == "" {
|
||||
return info
|
||||
}
|
||||
info.Ready = true
|
||||
dumpDir := filepath.Join(unit, "db-dumps")
|
||||
if entries, rErr := os.ReadDir(dumpDir); rErr == nil {
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() && filepath.Ext(e.Name()) == ".sql" && !strings.HasPrefix(e.Name(), preRestoreDumpPrefix) {
|
||||
info.HasDump = true
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if man := readManifest(filepath.Join(unit, "manifest.json")); man != nil {
|
||||
if man.DumpsAt != "" {
|
||||
if t, pErr := time.Parse(time.RFC3339, man.DumpsAt); pErr == nil {
|
||||
info.DumpsAt = t
|
||||
}
|
||||
}
|
||||
info.Skewed = man.OffsiteRunID == ""
|
||||
} else {
|
||||
info.Skewed = true
|
||||
}
|
||||
if info.HasDump {
|
||||
info.LooksEmpty = m.sniffScratchDump(dumpDir, stack)
|
||||
}
|
||||
return info
|
||||
}
|
||||
|
||||
// findScratchUnitDir locates `backups/primary/<stack>` anywhere under a restored scratch. restic
|
||||
// rebuilds absolute source paths under the target, and the snapshot may have come from a drive that
|
||||
// no longer exists on this box, so the prefix cannot be assumed.
|
||||
func findScratchUnitDir(scratch, stack string) string {
|
||||
found := ""
|
||||
suffix := filepath.Join("backups", "primary", stack)
|
||||
_ = filepath.Walk(scratch, func(path string, fi os.FileInfo, err error) error {
|
||||
if err != nil || found != "" {
|
||||
return nil //nolint:nilerr // a walk error on one branch must not abort the search
|
||||
}
|
||||
if fi.IsDir() && strings.HasSuffix(path, suffix) {
|
||||
found = path
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return found
|
||||
}
|
||||
|
||||
// sniffScratchDump runs the R-44 content sniff over the dump about to be replayed. Best-effort and
|
||||
// warn-level: any failure to read simply reports "no warning", because a sniff that blocks a
|
||||
// restore is worse than the skew it describes.
|
||||
func (m *Manager) sniffScratchDump(dumpDir, stack string) bool {
|
||||
entries, err := os.ReadDir(dumpDir)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
for _, e := range entries {
|
||||
name := e.Name()
|
||||
if e.IsDir() || filepath.Ext(name) != ".sql" || strings.HasPrefix(name, preRestoreDumpPrefix) {
|
||||
continue
|
||||
}
|
||||
dbType := DBTypePostgres
|
||||
if strings.Contains(name, string(DBTypeMariaDB)) {
|
||||
dbType = DBTypeMariaDB
|
||||
}
|
||||
if v := ValidateDump(filepath.Join(dumpDir, name), dbType); v.LooksEmpty {
|
||||
m.logger.Printf("[WARN] [offbox] %s: the snapshot dump %s has no account rows — it may predate the customer's data", stack, name)
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,424 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// R-43/R-44 (v0.148.0) — the coherent-pair + true-restore tests.
|
||||
//
|
||||
// These exist because the product shipped a restore button for months that could not restore.
|
||||
// DIAG-immich-restore-2026-07-19: 11 photos, files intact, timeline empty, two "successful"
|
||||
// restores that merged 0 files and never touched postgres. Every test below asserts a behaviour
|
||||
// whose absence produced that outcome, so each one is a regression guard for a real incident
|
||||
// rather than a description of the current implementation.
|
||||
|
||||
// recordingProvider records stop/start call ORDER so the reconstitution sequence can be asserted.
|
||||
type recordingProvider struct {
|
||||
offbox3aProvider
|
||||
calls []string
|
||||
}
|
||||
|
||||
func (p *recordingProvider) StopStack(string) error { p.calls = append(p.calls, "stop"); return nil }
|
||||
func (p *recordingProvider) StartStack(string) error { p.calls = append(p.calls, "start"); return nil }
|
||||
|
||||
// The app really is up again after StartStack, so the post-restore health wait returns at once.
|
||||
// Leaving it false would make each test sit through the full 90s deadline.
|
||||
func (p *recordingProvider) RefreshAndIsRunning(string) bool { return true }
|
||||
|
||||
// recoveryProvider adds the recovery info CaptureRecoveryUnit needs (the shared 3a provider has none).
|
||||
type recoveryProvider struct {
|
||||
offbox3aProvider
|
||||
stackDir string
|
||||
}
|
||||
|
||||
func (p *recoveryProvider) GetStackRecoveryInfo(name string) (RecoveryInfo, bool) {
|
||||
return RecoveryInfo{DisplayName: "Immich", StackDir: p.stackDir}, name == "immich"
|
||||
}
|
||||
|
||||
// pgDump builds a structurally valid postgres dump big enough to clear ValidateDump's 100-byte
|
||||
// floor, with the accounts-table COPY block carrying `rows` rows. The R-44 sniff runs only on a
|
||||
// dump that already passes structural validation, so a toy fixture would silently skip it.
|
||||
func pgDump(rows int) string {
|
||||
const head = `-- PostgreSQL database dump
|
||||
-- Dumped from database version 16.10
|
||||
SET statement_timeout = 0;
|
||||
SET lock_timeout = 0;
|
||||
SET client_encoding = 'UTF8';
|
||||
CREATE TABLE public.asset (id uuid NOT NULL);
|
||||
CREATE TABLE public."user" (id uuid NOT NULL, email text);
|
||||
COPY public."user" (id, email) FROM stdin;
|
||||
`
|
||||
var b strings.Builder
|
||||
b.WriteString(head)
|
||||
for i := 0; i < rows; i++ {
|
||||
b.WriteString("id-x\tuser@example.invalid\n")
|
||||
}
|
||||
b.WriteString("\\.\n") // the COPY-block terminator
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// reconFixture builds a manager with a COMPLETED full scratch for `immich`, a snapshot whose unit
|
||||
// carries the given coherence stamp, and injectable copy/dump/import seams.
|
||||
func reconFixture(t *testing.T, runID, dumpsAt string, dumpBody string) (*Manager, *recordingProvider, *[]string) {
|
||||
t.Helper()
|
||||
drive := t.TempDir()
|
||||
m, sett := newOffboxManager(t)
|
||||
prov := &recordingProvider{offbox3aProvider: offbox3aProvider{
|
||||
hdd: map[string]string{"immich": drive}, binds: map[string][]ClassifiedBind{}, has: map[string]bool{},
|
||||
}}
|
||||
m.SetStackProvider(prov)
|
||||
if err := sett.AddStoragePath(settings.StoragePath{Path: drive, Label: "USB", Schedulable: true}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
scratch, liveNs, err := m.offboxRestoreScratchDir("immich")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
oldNs := "/felhomdata/ns"
|
||||
unitP := oldNs + "/backups/primary/immich"
|
||||
dataP := oldNs + "/appdata/immich"
|
||||
placements, err := mapOffsiteRestorePaths([]string{unitP, dataP}, "immich", scratch, liveNs)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, pl := range placements {
|
||||
if err := os.MkdirAll(pl.src, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pl.isUnit {
|
||||
dd := filepath.Join(pl.src, "db-dumps")
|
||||
if err := os.MkdirAll(dd, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if dumpBody != "" {
|
||||
if err := os.WriteFile(filepath.Join(dd, "immich-postgres.sql"), []byte(dumpBody), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
man := &RecoveryManifest{SchemaVersion: 1, AppName: "immich", OffsiteRunID: runID, DumpsAt: dumpsAt}
|
||||
if err := writeManifest(filepath.Join(pl.src, "manifest.json"), man); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
m.SetOffboxFreeFn(func(string) int64 { return 100 << 30 })
|
||||
m.SetOffboxSizer(func(string) int64 { return 1 << 20 })
|
||||
m.SetOffboxRunner(func(_ context.Context, _ []string, args ...string) ([]byte, error) {
|
||||
if contains(args, "snapshots") {
|
||||
return []byte(`[{"short_id":"snap1","time":"2026-07-19T06:00:00Z","paths":["` + unitP + `","` + dataP + `"]}]`), nil
|
||||
}
|
||||
return nil, nil
|
||||
})
|
||||
|
||||
// Seams: one DB, a safety dump that really writes a file, and a recording importer.
|
||||
db := DiscoveredDB{StackName: "immich", ContainerName: "immich-postgres", DBType: DBTypePostgres}
|
||||
m.discoverDBs = func(context.Context) ([]DiscoveredDB, error) { return []DiscoveredDB{db}, nil }
|
||||
m.SetSafetyDumpFn(func(_ context.Context, d DiscoveredDB, dir string) DumpResult {
|
||||
p := filepath.Join(dir, "immich-postgres.sql")
|
||||
_ = os.MkdirAll(dir, 0o755)
|
||||
_ = os.WriteFile(p, []byte(pgDump(1)), 0o644)
|
||||
return DumpResult{DB: d, FilePath: p, Size: 42}
|
||||
})
|
||||
var imported []string
|
||||
m.importDBDump = func(_ context.Context, _ DiscoveredDB, p string) error {
|
||||
imported = append(imported, p)
|
||||
return nil
|
||||
}
|
||||
m.SetOffboxFullPlaceCopier(func(_, _ string) (int, error) { return 3, nil })
|
||||
return m, prov, &imported
|
||||
}
|
||||
|
||||
// TestReconstituteReplaysDBAndOrdersOperations is Scenario C: the whole point of R-43. A restore of
|
||||
// a DB-indexed app must stop the app, place files, restart it and REPLAY the snapshot's dump — and
|
||||
// the safety dump must exist before any of it. Before v0.148.0 the replay simply did not happen,
|
||||
// which is why the photos never came back.
|
||||
func TestReconstituteReplaysDBAndOrdersOperations(t *testing.T) {
|
||||
m, prov, imported := reconFixture(t, "20260719T060000Z", "2026-07-19T06:00:00Z", pgDump(1))
|
||||
|
||||
res, err := m.ReconstituteFromOffsite(context.Background(), "immich")
|
||||
if err != nil {
|
||||
t.Fatalf("reconstitute: %v", err)
|
||||
}
|
||||
if res.DBsReplayed != 1 {
|
||||
t.Fatalf("expected the snapshot dump to be replayed exactly once, got %d — this is the R-43 defect", res.DBsReplayed)
|
||||
}
|
||||
if len(*imported) != 1 || !strings.Contains((*imported)[0], "immich-postgres.sql") {
|
||||
t.Fatalf("expected an import of the snapshot dump, got %v", *imported)
|
||||
}
|
||||
// The dump replayed must come from the SCRATCH unit, never the live one: the live unit is
|
||||
// deliberately not overwritten, so replaying from it would replay the CURRENT database back over
|
||||
// itself and restore nothing.
|
||||
if !strings.Contains((*imported)[0], "offsite-restore") {
|
||||
t.Fatalf("replay source must be the restored scratch unit, got %s", (*imported)[0])
|
||||
}
|
||||
if res.FilesPlaced != 3 {
|
||||
t.Fatalf("expected the userdata placement to be counted, got %d", res.FilesPlaced)
|
||||
}
|
||||
// stop BEFORE the file copy, start BEFORE the replay (ImportDump needs a live container).
|
||||
if got := strings.Join(prov.calls, ","); got != "stop,start" {
|
||||
t.Fatalf("expected stop then start around the restore, got %q", got)
|
||||
}
|
||||
if res.SafetyDump == "" {
|
||||
t.Fatal("no safety dump recorded — the undo must exist")
|
||||
}
|
||||
if _, err := os.Stat(res.SafetyDump); err != nil {
|
||||
t.Fatalf("safety dump not on disk: %v", err)
|
||||
}
|
||||
if !strings.HasPrefix(filepath.Base(res.SafetyDump), preRestoreDumpPrefix) {
|
||||
t.Fatalf("safety dump must carry the pre-restore prefix so it is never replayed as a source, got %s", filepath.Base(res.SafetyDump))
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconstituteRefusesWhenSafetyDumpFails is the RED-PROOF for the undo invariant: a replay whose
|
||||
// previous state was not captured is an overwrite with no way back, so it must not happen at all —
|
||||
// and it must abort with the live app untouched (no stop, no copy).
|
||||
func TestReconstituteRefusesWhenSafetyDumpFails(t *testing.T) {
|
||||
m, prov, imported := reconFixture(t, "run1", "2026-07-19T06:00:00Z", pgDump(1))
|
||||
m.SetSafetyDumpFn(func(_ context.Context, d DiscoveredDB, _ string) DumpResult {
|
||||
return DumpResult{DB: d, Error: context.DeadlineExceeded}
|
||||
})
|
||||
var copied bool
|
||||
m.SetOffboxFullPlaceCopier(func(_, _ string) (int, error) { copied = true; return 1, nil })
|
||||
|
||||
_, err := m.ReconstituteFromOffsite(context.Background(), "immich")
|
||||
if err == nil {
|
||||
t.Fatal("expected a refusal when the safety dump cannot be taken")
|
||||
}
|
||||
if len(*imported) != 0 {
|
||||
t.Fatalf("REPLAYED WITHOUT AN UNDO — the exact thing the invariant forbids: %v", *imported)
|
||||
}
|
||||
if copied {
|
||||
t.Fatal("files were overwritten despite the refusal — the abort must leave live data untouched")
|
||||
}
|
||||
if len(prov.calls) != 0 {
|
||||
t.Fatalf("the app was stopped despite the refusal, got %v", prov.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconstituteNoDBAppMakesNoDumpOrImportCalls is Scenario E: an app without a database must flow
|
||||
// exactly as before — no safety dump, no replay — so the new leg cannot regress the simple case.
|
||||
func TestReconstituteNoDBAppMakesNoDumpOrImportCalls(t *testing.T) {
|
||||
m, _, imported := reconFixture(t, "run1", "2026-07-19T06:00:00Z", "")
|
||||
m.discoverDBs = func(context.Context) ([]DiscoveredDB, error) { return nil, nil }
|
||||
dumped := 0
|
||||
m.SetSafetyDumpFn(func(_ context.Context, d DiscoveredDB, _ string) DumpResult {
|
||||
dumped++
|
||||
return DumpResult{DB: d}
|
||||
})
|
||||
|
||||
res, err := m.ReconstituteFromOffsite(context.Background(), "immich")
|
||||
if err != nil {
|
||||
t.Fatalf("reconstitute: %v", err)
|
||||
}
|
||||
if dumped != 0 {
|
||||
t.Fatalf("a no-DB app must not produce a safety dump, got %d call(s)", dumped)
|
||||
}
|
||||
if len(*imported) != 0 {
|
||||
t.Fatalf("a no-DB app must not import anything, got %v", *imported)
|
||||
}
|
||||
if res.SafetyDump != "" || res.DBsReplayed != 0 {
|
||||
t.Fatalf("unexpected DB activity: safety=%q replayed=%d", res.SafetyDump, res.DBsReplayed)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconstituteSurfacesLegacySkewedPair is Scenario D: a pre-v0.148 snapshot carries no coherence
|
||||
// stamp, so its two halves may be from different times. That must be SURFACED (and reversible), never
|
||||
// blocked — the customer's own judgement is the gate, and refusing would deny a legitimate restore.
|
||||
func TestReconstituteSurfacesLegacySkewedPair(t *testing.T) {
|
||||
m, _, imported := reconFixture(t, "", "", pgDump(1))
|
||||
|
||||
res, err := m.ReconstituteFromOffsite(context.Background(), "immich")
|
||||
if err != nil {
|
||||
t.Fatalf("a legacy pair must still be restorable, got refusal: %v", err)
|
||||
}
|
||||
if !res.Skewed {
|
||||
t.Fatal("an unstamped (pre-v0.148) snapshot must report Skewed so the confirm can say so")
|
||||
}
|
||||
if len(*imported) != 1 {
|
||||
t.Fatalf("the legacy restore must still replay, got %v", *imported)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconstituteFlagsCustomerEmptyDump is the R-44 sniff at the restore end: the immich dump that
|
||||
// started all of this was structurally valid and contained zero users. Restoring it is allowed, but
|
||||
// the customer must be told before they commit.
|
||||
func TestReconstituteFlagsCustomerEmptyDump(t *testing.T) {
|
||||
// A valid postgres dump whose accounts table has NO rows — the 2026-07-19 shape exactly.
|
||||
m, _, _ := reconFixture(t, "run1", "2026-07-19T06:00:00Z", pgDump(0))
|
||||
|
||||
res, err := m.ReconstituteFromOffsite(context.Background(), "immich")
|
||||
if err != nil {
|
||||
t.Fatalf("the sniff must never block a restore: %v", err)
|
||||
}
|
||||
if !res.LooksEmpty {
|
||||
t.Fatal("a dump with an empty accounts table must raise the warn-level signal")
|
||||
}
|
||||
}
|
||||
|
||||
// TestOffsiteScratchPairReportsWhatTheConfirmNeeds covers the page-render surface: the confirm can
|
||||
// only be honest if this reports the pair's age and warnings before anything is started.
|
||||
func TestOffsiteScratchPairReportsWhatTheConfirmNeeds(t *testing.T) {
|
||||
m, _, _ := reconFixture(t, "run1", "2026-07-19T06:00:00Z",
|
||||
"-- PostgreSQL database dump\nCREATE TABLE a();\nCOPY public.\"user\" (id) FROM stdin;\n7\n\\.\n")
|
||||
|
||||
info := m.OffsiteScratchPair("immich")
|
||||
if !info.Ready || !info.HasDump {
|
||||
t.Fatalf("expected a ready pair with a dump, got %+v", info)
|
||||
}
|
||||
if info.Skewed {
|
||||
t.Fatal("a stamped snapshot must not be reported as skewed")
|
||||
}
|
||||
if info.LooksEmpty {
|
||||
t.Fatal("a dump with account rows must not be flagged empty")
|
||||
}
|
||||
want, _ := time.Parse(time.RFC3339, "2026-07-19T06:00:00Z")
|
||||
if !info.DumpsAt.Equal(want) {
|
||||
t.Fatalf("DumpsAt = %v, want %v", info.DumpsAt, want)
|
||||
}
|
||||
}
|
||||
|
||||
// --- R-44: the coherence pre-phase -----------------------------------------------------------
|
||||
|
||||
// TestOffsiteRunDumpsBeforeCapture is Scenarios A + B. The ORDER is the entire mechanism: dumps
|
||||
// must be refreshed BEFORE restic captures, so the snapshot pairs this run's database with this
|
||||
// run's files. Reversed, the snapshot would hold rows pointing at files that were never captured.
|
||||
//
|
||||
// It also asserts the ordering on the NIGHTLY entry point (RunOffboxBackup, no progress sink), not
|
||||
// just the manual one — before v0.148.0 the nightly ordering was an accident of two independent
|
||||
// scheduler entries at 02:30 and 04:15, which a schedule edit could silently invert.
|
||||
func TestOffsiteRunDumpsBeforeCapture(t *testing.T) {
|
||||
drive := t.TempDir()
|
||||
m, sett, prov := classifiedOffboxManager(t, drive)
|
||||
mkUnit(t, drive, "immich")
|
||||
if err := os.MkdirAll(filepath.Join(drive, "appdata", "immich"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prov.hdd["immich"] = drive
|
||||
prov.has["immich"] = true
|
||||
prov.binds["immich"] = []ClassifiedBind{mandatoryHDD("appdata/immich")}
|
||||
_ = sett.SetAppOffbox("immich", true)
|
||||
|
||||
var order []string
|
||||
m.SetOffsitePreDumpFn(func(context.Context) error {
|
||||
order = append(order, "dump")
|
||||
return nil
|
||||
})
|
||||
m.SetOffboxRunner(func(_ context.Context, _ []string, args ...string) ([]byte, error) {
|
||||
switch {
|
||||
case contains(args, "cat") && contains(args, "config"):
|
||||
return []byte(`{"version":2}`), nil
|
||||
case contains(args, "backup"):
|
||||
order = append(order, "capture")
|
||||
return nil, nil
|
||||
case contains(args, "snapshots"):
|
||||
return []byte(`[]`), nil
|
||||
case contains(args, "stats"):
|
||||
return []byte(`{"total_size":123}`), nil
|
||||
}
|
||||
return nil, nil
|
||||
})
|
||||
|
||||
if err := m.RunOffboxBackup(context.Background()); err != nil {
|
||||
t.Fatalf("run: %v", err)
|
||||
}
|
||||
if len(order) < 2 {
|
||||
t.Fatalf("expected both a dump and a capture, got %v", order)
|
||||
}
|
||||
if order[0] != "dump" {
|
||||
t.Fatalf("the dump leg MUST precede the capture (R-44); got %v", order)
|
||||
}
|
||||
if order[1] != "capture" {
|
||||
t.Fatalf("expected the capture immediately after the dump, got %v", order)
|
||||
}
|
||||
}
|
||||
|
||||
// TestOffsiteRunContinuesWhenDumpLegFails is the data-first rule: a dump failure degrades the
|
||||
// snapshot's DB half but must NOT abort the push. Refusing to ship the files would turn a partial
|
||||
// backup into no backup at all — strictly worse for the customer.
|
||||
func TestOffsiteRunContinuesWhenDumpLegFails(t *testing.T) {
|
||||
drive := t.TempDir()
|
||||
m, sett, prov := classifiedOffboxManager(t, drive)
|
||||
mkUnit(t, drive, "immich")
|
||||
if err := os.MkdirAll(filepath.Join(drive, "appdata", "immich"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prov.hdd["immich"] = drive
|
||||
prov.has["immich"] = true
|
||||
prov.binds["immich"] = []ClassifiedBind{mandatoryHDD("appdata/immich")}
|
||||
_ = sett.SetAppOffbox("immich", true)
|
||||
|
||||
m.SetOffsitePreDumpFn(func(context.Context) error { return context.DeadlineExceeded })
|
||||
cap := &backupCapture{}
|
||||
m.SetOffboxRunner(cap.runner())
|
||||
|
||||
if err := m.RunOffboxBackup(context.Background()); err != nil {
|
||||
t.Fatalf("a dump failure must not fail the whole run: %v", err)
|
||||
}
|
||||
if cap.backups != 1 {
|
||||
t.Fatalf("the files must still be pushed after a dump failure, got %d capture(s)", cap.backups)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCaptureRecoveryUnitStampsAndCarriesRunID covers the stamp that makes a pair verifiable at
|
||||
// restore time, and the trap beside it: the PERIODIC refresh must neither invent a coherence claim
|
||||
// nor erase one a real run established.
|
||||
func TestCaptureRecoveryUnitStampsAndCarriesRunID(t *testing.T) {
|
||||
drive := t.TempDir()
|
||||
m, _, base := classifiedOffboxManager(t, drive)
|
||||
base.hdd["immich"] = drive
|
||||
// CaptureRecoveryUnit needs real recovery info + a compose dir to read; the shared fixture
|
||||
// provider returns none, so wrap it rather than widening a struct four other test files use.
|
||||
stackDir := filepath.Join(t.TempDir(), "immich")
|
||||
if err := os.MkdirAll(stackDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(stackDir, "docker-compose.yml"), []byte("services: {}\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.SetStackProvider(&recoveryProvider{offbox3aProvider: *base, stackDir: stackDir})
|
||||
|
||||
// 1) A run in flight stamps the manifest.
|
||||
end := m.beginOffsiteRunStamp("run-A")
|
||||
if err := m.CaptureRecoveryUnit("immich"); err != nil {
|
||||
t.Fatalf("capture: %v", err)
|
||||
}
|
||||
end()
|
||||
man := readManifest(RecoveryUnitManifestPath(drive, "immich"))
|
||||
if man == nil || man.OffsiteRunID != "run-A" {
|
||||
t.Fatalf("expected the in-flight run id to be stamped, got %+v", man)
|
||||
}
|
||||
if man.DumpsAt == "" {
|
||||
t.Fatal("a stamped unit must record when its dumps were taken")
|
||||
}
|
||||
|
||||
// 2) A periodic refresh (no run in flight) must CARRY the stamp forward, not blank it — a unit
|
||||
// that silently lost its stamp would be re-reported as a skewed legacy pair at restore time.
|
||||
if err := m.CaptureRecoveryUnit("immich"); err != nil {
|
||||
t.Fatalf("refresh: %v", err)
|
||||
}
|
||||
man2 := readManifest(RecoveryUnitManifestPath(drive, "immich"))
|
||||
if man2 == nil || man2.OffsiteRunID != "run-A" {
|
||||
t.Fatalf("the periodic refresh erased the coherence stamp: %+v", man2)
|
||||
}
|
||||
|
||||
// 3) A NEW run re-stamps even though nothing else about the unit changed — the idempotent-skip
|
||||
// must not swallow the one field the restore path reads.
|
||||
end2 := m.beginOffsiteRunStamp("run-B")
|
||||
if err := m.CaptureRecoveryUnit("immich"); err != nil {
|
||||
t.Fatalf("capture 2: %v", err)
|
||||
}
|
||||
end2()
|
||||
man3 := readManifest(RecoveryUnitManifestPath(drive, "immich"))
|
||||
if man3 == nil || man3.OffsiteRunID != "run-B" {
|
||||
t.Fatalf("a new run must re-stamp the unit, got %+v", man3)
|
||||
}
|
||||
}
|
||||
@@ -30,6 +30,16 @@ const (
|
||||
// SetOffboxFreeFn overrides the restore free-space probe (tests; the Windows go-test host has no df).
|
||||
func (m *Manager) SetOffboxFreeFn(fn func(path string) int64) { m.offboxFreeFn = fn }
|
||||
|
||||
// SetOffboxFullPlaceCopier overrides the FULL-restore overwrite copier (tests; no rsync needed).
|
||||
func (m *Manager) SetOffboxFullPlaceCopier(fn func(src, dst string) (int, error)) {
|
||||
m.offboxFullPlaceCopier = fn
|
||||
}
|
||||
|
||||
// SetSafetyDumpFn overrides the pre-restore safety dump (tests; no Docker needed).
|
||||
func (m *Manager) SetSafetyDumpFn(fn func(ctx context.Context, db DiscoveredDB, dumpDir string) DumpResult) {
|
||||
m.safetyDumpFn = fn
|
||||
}
|
||||
|
||||
// offboxFree returns the free-space probe (nil seam → the real diskFreeBytes).
|
||||
func (m *Manager) offboxFree() func(string) int64 {
|
||||
if m.offboxFreeFn != nil {
|
||||
|
||||
@@ -44,6 +44,15 @@ type RecoveryManifest struct {
|
||||
DBDumps []string `json:"db_dumps"`
|
||||
VolumeDumps []string `json:"volume_dumps"`
|
||||
Checksums map[string]string `json:"checksums"` // sha256 of captured compose/ files
|
||||
// R-43/R-44 (v0.148.0): the coherence stamp. An offsite run refreshes the dumps FIRST and then
|
||||
// captures the unit, so a manifest carrying an OffsiteRunID asserts "the db-dumps/ in this unit
|
||||
// were taken by that run" — i.e. the snapshot is an internally coherent {DB@T, files@T} pair.
|
||||
// A manifest WITHOUT these fields is a pre-v0.148 unit whose dump age is unknown and may skew
|
||||
// arbitrarily from the files beside it (the DIAG-immich-restore-2026-07-19 failure); the restore
|
||||
// confirm surfaces that honestly rather than blocking. Empty on the periodic refresh, which must
|
||||
// never claim a coherence it did not establish — it carries the prior stamp forward instead.
|
||||
OffsiteRunID string `json:"offsite_run_id,omitempty"`
|
||||
DumpsAt string `json:"dumps_at,omitempty"` // RFC3339 UTC — when this run's dump leg finished
|
||||
}
|
||||
|
||||
// SetVersion records the controller version stamped into recovery-unit manifests.
|
||||
@@ -107,13 +116,26 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
|
||||
version := m.versionLocked()
|
||||
|
||||
manifestPath := RecoveryUnitManifestPath(nsRoot, stackName)
|
||||
cur := readManifest(manifestPath)
|
||||
|
||||
// R-43/R-44: the coherence stamp of the offsite run currently in flight ("" on the periodic
|
||||
// refresh and on the local dump run). When empty we CARRY THE PRIOR STAMP FORWARD rather than
|
||||
// blanking it — a periodic refresh must neither claim a coherence it did not establish nor
|
||||
// destroy the record of one that a real run did.
|
||||
runID, dumpsAt := m.offsiteRunStamp()
|
||||
if runID == "" && cur != nil {
|
||||
runID, dumpsAt = cur.OffsiteRunID, cur.DumpsAt
|
||||
}
|
||||
|
||||
// Skip if the unit is already current — avoids needless drive writes on the periodic refresh.
|
||||
if cur := readManifest(manifestPath); cur != nil &&
|
||||
// The run-id is part of "current": an offsite run must re-stamp the manifest even when nothing
|
||||
// else changed, because the stamp is exactly the claim the restore path reads.
|
||||
if cur != nil &&
|
||||
cur.ControllerVer == version &&
|
||||
stringMapEqual(cur.Checksums, checksums) &&
|
||||
stringSliceEqual(cur.DBDumps, dbDumps) &&
|
||||
stringSliceEqual(cur.VolumeDumps, volDumps) {
|
||||
stringSliceEqual(cur.VolumeDumps, volDumps) &&
|
||||
cur.OffsiteRunID == runID {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -143,6 +165,8 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
|
||||
DBDumps: dbDumps,
|
||||
VolumeDumps: volDumps,
|
||||
Checksums: checksums,
|
||||
OffsiteRunID: runID,
|
||||
DumpsAt: dumpsAt,
|
||||
}
|
||||
if err := writeManifest(manifestPath, manifest); err != nil {
|
||||
return fmt.Errorf("writing manifest: %w", err)
|
||||
|
||||
@@ -19,7 +19,15 @@ import (
|
||||
// A dump whose DB container is not found is logged and skipped; an actual import FAILURE is returned
|
||||
// (surfaced, not swallowed) so a failed data restore cannot read as success.
|
||||
func (m *Manager) reimportDBDumps(ctx context.Context, stackName, nsRoot string) (int, error) {
|
||||
dumpDir := AppDBDumpPath(nsRoot, stackName)
|
||||
return m.reimportDBDumpsFrom(ctx, stackName, AppDBDumpPath(nsRoot, stackName))
|
||||
}
|
||||
|
||||
// reimportDBDumpsFrom is reimportDBDumps with an EXPLICIT dump directory. The offsite
|
||||
// reconstitution path (R-43) replays out of the restored SCRATCH unit rather than the live one:
|
||||
// the local unit is deliberately never overwritten by a placement, so the dump that belongs to the
|
||||
// chosen snapshot exists only under the scratch. Same discovery/import seams, same failure
|
||||
// semantics — only the source directory differs.
|
||||
func (m *Manager) reimportDBDumpsFrom(ctx context.Context, stackName, dumpDir string) (int, error) {
|
||||
entries, err := os.ReadDir(dumpDir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
|
||||
Reference in New Issue
Block a user