v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (R-43 + R-44)

Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.

R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.

New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
  - nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
    created after the snapshot survives as an extra;
  - the undo exists before the act — the pre-restore- dump is verified ON DISK
    before anything is stopped, overwritten or replayed; if it cannot be taken
    the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.

R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.

Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.

11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.

NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.

Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
This commit is contained in:
2026-07-19 12:21:05 +02:00
parent 2fcae041ae
commit 062357f778
20 changed files with 1486 additions and 154 deletions
+95 -1
View File
@@ -51,6 +51,20 @@ type DumpValidation struct {
Error string
FileSize int64
ModTime time.Time
// R-44 (v0.148.0) content sniff — a WARN-LEVEL signal, never a gate.
//
// Structural validity says nothing about whether a dump holds the customer's data. The immich
// dump of 2026-07-19 was 52MB, had a valid header and 60+ CREATE TABLEs, and contained zero
// users and zero assets: its whole bulk was the geodata reference tables immich ships. Size and
// table count are therefore both useless as emptiness heuristics — but an accounts table with
// no rows is a strong, cheap, app-agnostic hint that a dump predates the customer entirely.
//
// Deliberately NOT a refusal: plenty of legitimate apps have no users table (UserTableFound
// false → inconclusive → silent), and a false positive that blocked a restore would be far
// worse than the skew it guards against. The restore confirm shows it as one extra line.
UserTableFound bool
UserRows int
LooksEmpty bool // UserTableFound && UserRows == 0
}
// DumpFileInfo holds info about a dump file on disk.
@@ -363,6 +377,9 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation {
lineNum := 0
headerFound := false
tableCount := 0
// R-44 sniff state. inUserCopy tracks a postgres `COPY … FROM stdin;` block for an accounts
// table; rows are counted until the `\.` terminator.
inUserCopy := false
for {
lineBytes, isPrefix, err := reader.ReadLine()
if err != nil {
@@ -374,7 +391,12 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation {
break // EOF
}
if isPrefix {
// Line exceeds buffer — skip remainder (COPY data, large INSERTs)
// Line exceeds buffer — skip remainder (COPY data, large INSERTs).
// A long line inside a user COPY block is still a ROW: count it before discarding it,
// or a table whose rows happen to be wide would sniff as empty and raise a false alarm.
if inUserCopy {
v.UserRows++
}
for isPrefix && err == nil {
_, isPrefix, err = reader.ReadLine()
}
@@ -384,6 +406,23 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation {
line := string(lineBytes)
lineNum++
// R-44 content sniff (warn-level; see DumpValidation).
if inUserCopy {
if line == `\.` {
inUserCopy = false
} else {
v.UserRows++
}
} else if isUserCopyStart(line, dbType) {
inUserCopy = true
v.UserTableFound = true
} else if dbType == DBTypeMariaDB && isUserInsert(line) {
// mysqldump writes multi-row `INSERT INTO \`users\` VALUES (…),(…);` — the row count is
// not worth parsing out of it, and presence alone answers the only question asked here.
v.UserTableFound = true
v.UserRows++
}
// Header check — scan first 10 lines for expected dump header
// MariaDB 11.4+ prepends a sandbox comment before the header line
if lineNum <= 10 && !headerFound {
@@ -427,10 +466,65 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation {
return v
}
v.LooksEmpty = v.UserTableFound && v.UserRows == 0
if v.LooksEmpty {
log.Printf("[WARN] [backup] ValidateDump: %s is structurally valid (%d tables) but its accounts table has NO rows — the dump may predate the customer's data", filePath, tableCount)
}
v.Valid = true
return v
}
// userTableNames are the table names treated as "the accounts table" by the R-44 sniff. Kept
// deliberately short: a wider net (anything containing "user") would match join/audit tables like
// `user_metadata` or `album_user`, which are legitimately empty on a healthy single-user install
// and would produce exactly the false alarm this signal must not raise.
var userTableNames = []string{"user", "users", "account", "accounts"}
// isUserCopyStart reports whether a line opens a postgres `COPY <accounts-table> … FROM stdin;`
// block. pg_dump writes the table qualified and optionally quoted — `COPY public."user" (…)`,
// `COPY public.users (…)` — so both forms are matched.
func isUserCopyStart(line string, dbType DBType) bool {
if dbType != DBTypePostgres || !strings.HasPrefix(line, "COPY ") {
return false
}
rest := strings.TrimPrefix(line, "COPY ")
sp := strings.IndexByte(rest, ' ')
if sp < 0 {
return false
}
return matchesUserTable(rest[:sp])
}
// isUserInsert reports whether a line is a mysqldump INSERT into an accounts table.
func isUserInsert(line string) bool {
const pfx = "INSERT INTO "
if !strings.HasPrefix(line, pfx) {
return false
}
rest := strings.TrimPrefix(line, pfx)
sp := strings.IndexByte(rest, ' ')
if sp < 0 {
return false
}
return matchesUserTable(rest[:sp])
}
// matchesUserTable strips schema qualification and quoting from a dumped table reference and
// reports whether the bare name is an accounts table.
func matchesUserTable(ref string) bool {
if dot := strings.LastIndexByte(ref, '.'); dot >= 0 {
ref = ref[dot+1:]
}
ref = strings.Trim(ref, "\"`")
for _, n := range userTableNames {
if strings.EqualFold(ref, n) {
return true
}
}
return false
}
// ListDumpFiles returns info about SQL dump files on disk.
//
// M18: ValidateDump scans the dump line-by-line; on a customer with hundreds-of-MB dumps that is wasted
@@ -0,0 +1,141 @@
package appbackup
import (
"os"
"path/filepath"
"strings"
"testing"
)
// R-44 content-sniff tests.
//
// The dump that triggered this work (DIAG-immich-restore-2026-07-19) was 52MB, had a valid
// PostgreSQL header and 60+ CREATE TABLE statements, and contained zero users and zero assets —
// its entire bulk was immich's shipped geodata reference tables. Both of the signals the product
// already had (file size, table count) called it healthy. These tests pin the one signal that
// would have caught it, and the boundaries that keep it from crying wolf.
func writeDump(t *testing.T, body string) string {
t.Helper()
p := filepath.Join(t.TempDir(), "d.sql")
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
return p
}
const pgHead = `-- PostgreSQL database dump
-- Dumped from database version 16.10
SET statement_timeout = 0;
SET client_encoding = 'UTF8';
CREATE TABLE public.asset (id uuid NOT NULL);
CREATE TABLE public."user" (id uuid NOT NULL, email text);
`
// TestSniffFlagsEmptyAccountsTable is the 2026-07-19 shape: structurally perfect, no customer.
func TestSniffFlagsEmptyAccountsTable(t *testing.T) {
body := pgHead + "COPY public.\"user\" (id, email) FROM stdin;\n\\.\n" +
"COPY public.asset (id) FROM stdin;\n\\.\n"
v := ValidateDump(writeDump(t, body), DBTypePostgres)
if !v.Valid {
t.Fatalf("the dump is structurally valid; sniff must not change that: %s", v.Error)
}
if !v.UserTableFound {
t.Fatal("the accounts table COPY block was not recognised")
}
if v.UserRows != 0 {
t.Fatalf("UserRows = %d, want 0", v.UserRows)
}
if !v.LooksEmpty {
t.Fatal("a valid dump with zero account rows MUST raise the warn signal — this is the whole point of R-44")
}
}
// TestSniffQuietOnPopulatedDump — the common case must stay silent, or the warning becomes noise
// and gets ignored precisely when it matters.
func TestSniffQuietOnPopulatedDump(t *testing.T) {
body := pgHead + "COPY public.\"user\" (id, email) FROM stdin;\n" +
"a\tone@example.invalid\nb\ttwo@example.invalid\n\\.\n"
v := ValidateDump(writeDump(t, body), DBTypePostgres)
if v.UserRows != 2 {
t.Fatalf("UserRows = %d, want 2", v.UserRows)
}
if v.LooksEmpty {
t.Fatal("a dump with account rows must not be flagged")
}
}
// TestSniffInconclusiveWithoutAccountsTable — plenty of legitimate apps have no users table. No
// table, no claim: a false positive here would warn on every restore of such an app forever.
func TestSniffInconclusiveWithoutAccountsTable(t *testing.T) {
body := "-- PostgreSQL database dump\nCREATE TABLE public.thing (id int);\n" +
"COPY public.thing (id) FROM stdin;\n\\.\n" + strings.Repeat("-- pad\n", 20)
v := ValidateDump(writeDump(t, body), DBTypePostgres)
if v.UserTableFound {
t.Fatal("no accounts table exists — none must be reported")
}
if v.LooksEmpty {
t.Fatal("an app without an accounts table must be INCONCLUSIVE, never flagged empty")
}
}
// TestSniffIgnoresJoinAndAuditTables is the false-alarm guard that shaped the name list, and it is
// written as the case that DISCRIMINATES: an app with NO accounts table but with `user_metadata` /
// `album_user` / `user_audit` — all legitimately empty on a healthy box. Exact-matching leaves this
// inconclusive (silent, correct). A substring match on "user" would treat a join table as the
// accounts table, find zero rows, and shout "your backup looks empty" on every single restore of a
// perfectly healthy app — which is how a warning signal becomes noise and then gets ignored.
func TestSniffIgnoresJoinAndAuditTables(t *testing.T) {
body := "-- PostgreSQL database dump\nCREATE TABLE public.album (id int);\n" +
"COPY public.user_metadata (id) FROM stdin;\n\\.\n" +
"COPY public.album_user (id) FROM stdin;\n\\.\n" +
"COPY public.user_audit (id) FROM stdin;\n\\.\n" +
"COPY public.album (id) FROM stdin;\n1\n\\.\n"
v := ValidateDump(writeDump(t, body), DBTypePostgres)
if v.UserTableFound {
t.Fatal("a join/audit table must never be mistaken for the accounts table")
}
if v.LooksEmpty {
t.Fatal("empty join/audit tables must not trigger the warning — this app has no accounts table at all")
}
}
// TestSniffCountsOnlyTheAccountsTable pins the counting boundary separately: with a real accounts
// table present, rows from neighbouring user-ish tables must not inflate it.
func TestSniffCountsOnlyTheAccountsTable(t *testing.T) {
body := pgHead +
"COPY public.user_metadata (id) FROM stdin;\nm1\nm2\nm3\n\\.\n" +
"COPY public.\"user\" (id, email) FROM stdin;\na\tone@example.invalid\n\\.\n"
v := ValidateDump(writeDump(t, body), DBTypePostgres)
if v.UserRows != 1 {
t.Fatalf("only the real accounts table may be counted; UserRows = %d, want 1", v.UserRows)
}
}
// TestSniffCountsWideRows — a row wider than the read buffer is skipped by the structural scan, but
// it is still a row. Counting it wrong would flag a populated table as empty (immich asset rows are
// genuinely long, which is what makes this reachable).
func TestSniffCountsWideRows(t *testing.T) {
wide := strings.Repeat("x", 300*1024)
body := pgHead + "COPY public.\"user\" (id, email) FROM stdin;\n" + wide + "\n\\.\n"
v := ValidateDump(writeDump(t, body), DBTypePostgres)
if v.UserRows != 1 {
t.Fatalf("a buffer-exceeding row must still count; UserRows = %d, want 1", v.UserRows)
}
if v.LooksEmpty {
t.Fatal("a table whose single row is very wide must not sniff as empty")
}
}
// TestSniffMariaDBInsertForm — mysqldump writes multi-row INSERTs, not COPY blocks.
func TestSniffMariaDBInsertForm(t *testing.T) {
head := "-- MariaDB dump 10.19\nCREATE TABLE `users` (id int);\n" + strings.Repeat("-- pad\n", 20)
empty := ValidateDump(writeDump(t, head), DBTypeMariaDB)
if empty.UserTableFound {
t.Fatal("a CREATE TABLE alone is not an accounts-table row source")
}
full := ValidateDump(writeDump(t, head+"INSERT INTO `users` VALUES (1),(2);\n"), DBTypeMariaDB)
if !full.UserTableFound || full.LooksEmpty {
t.Fatalf("a populated mariadb dump must not be flagged: %+v", full)
}
}
+39
View File
@@ -59,6 +59,16 @@ type Manager struct {
// offboxPlaceCopier (3a) — the place-to-live missing-only merge seam (nil → rsyncRestoreMissing,
// the `-a --ignore-existing` additive copy). Never rsyncMirror (--delete trap).
offboxPlaceCopier func(src, dst string) (int, error)
// offboxFullPlaceCopier (R-43, v0.148.0) — the FULL-restore overwrite seam (nil →
// rsyncRestoreOverwrite: `-a` with NO --ignore-existing and NO --delete). Distinct from
// offboxPlaceCopier on purpose: the two have opposite semantics for an existing file.
offboxFullPlaceCopier func(src, dst string) (int, error)
// safetyDumpFn (R-43) — the pre-restore safety-dump seam (nil → the real DumpOne), so the
// "never replay without an undo on disk" refusal is unit-testable without Docker.
safetyDumpFn func(ctx context.Context, db DiscoveredDB, dumpDir string) DumpResult
// offsitePreDumpFn (R-44) — the offsite dump pre-phase seam (nil → runDBDumpsInternal), so the
// dumps-strictly-before-capture ordering is observable in a test without Docker or restic.
offsitePreDumpFn func(ctx context.Context) error
// offboxFreeFn (3a) — the free-space probe for the restore headroom gate, overridable in tests (the
// Windows `go test` host has no `df`). Nil → the real diskFreeBytes (df --output=avail).
offboxFreeFn func(path string) int64
@@ -126,6 +136,13 @@ type Manager struct {
lastDBDump *DBDumpStatus
running bool
// R-43/R-44 (v0.148.0) — the coherence stamp of the offsite run in flight, read by
// CaptureRecoveryUnit so each unit records WHICH run took the dumps sitting beside its files.
// Set for the duration of the dump pre-phase + capture, cleared after; "" means "no offsite run
// is establishing coherence right now" (the periodic refresh and the local 02:30 dump leg).
offsiteRunID string
offsiteRunDumpAt string
// Restore op-status (Part B, opstatus.go) — display-only async-restore progress, under `mu`.
opRunning bool
opName string
@@ -310,6 +327,28 @@ func (m *Manager) RunDBDumps(ctx context.Context) error {
return m.runDBDumpsInternal(ctx)
}
// offsiteRunStamp returns the in-flight offsite run's coherence stamp ("" when none).
func (m *Manager) offsiteRunStamp() (runID, dumpsAt string) {
m.mu.Lock()
defer m.mu.Unlock()
return m.offsiteRunID, m.offsiteRunDumpAt
}
// beginOffsiteRunStamp marks the start of an offsite run's coherence window and returns the cleanup.
// The stamp is what CaptureRecoveryUnit writes into each unit manifest, so it must be live across
// BOTH the dump leg and the unit capture that follows it — those two together are the pair.
func (m *Manager) beginOffsiteRunStamp(runID string) func() {
m.mu.Lock()
m.offsiteRunID = runID
m.offsiteRunDumpAt = time.Now().UTC().Format(time.RFC3339)
m.mu.Unlock()
return func() {
m.mu.Lock()
m.offsiteRunID, m.offsiteRunDumpAt = "", ""
m.mu.Unlock()
}
}
// runDBDumpsInternal is the implementation of RunDBDumps. Caller must hold the running flag.
func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
start := time.Now()
+31
View File
@@ -658,9 +658,40 @@ func (m *Manager) runOffboxBackup(ctx context.Context, withProgress bool) error
m.offboxRecordStats(ctx, base, env) // the prune may have brought the size back down — refresh
runErr = fmt.Errorf("A távoli mentés túllépte a tárhelykeretet (%d/%d GB) — törölj régi mentéseket vagy kérj nagyobb keretet.", usedGB, quota)
} else {
// R-43/R-44 (v0.148.0) — THE COHERENCE PRE-PHASE. Refresh the DB/volume dumps and the recovery
// units BEFORE capturing, so the snapshot restic is about to write is an internally coherent
// {DB@T, files@T} pair. Before this, a push shipped live files beside whatever dump the 02:30
// local run happened to leave — on 2026-07-19 that was a dump taken four hours before the
// customer's account even existed, so the "backup" of the photos contained zero of them
// (DIAG-immich-restore-2026-07-19).
//
// Order matters and is the whole mechanism: dumps FIRST, then files. The gap between the two
// can only ADD files the DB does not reference yet (an upload landing mid-run is a harmless
// orphan blob), never remove one the DB DOES reference — so the file set is always a superset
// of what the restored DB points at. The reverse order would produce dangling rows.
//
// This runs on the NIGHTLY path too, not just the manual one: "every snapshot is a coherent
// pair" is the property that makes retention a history of restorable points rather than a
// history of skewed ones. It also makes the nightly ordering structural instead of a
// coincidence of two independent scheduler entries at 02:30 and 04:15.
endStamp := m.beginOffsiteRunStamp(start.UTC().Format("20060102T150405Z"))
if withProgress {
m.offboxProgress.setPhase(OffboxPhaseDump)
}
dumpStart := time.Now()
if dErr := m.offsitePreDump(ctx); dErr != nil {
// Data-first: a dump failure must NOT abort the push. The files are still worth shipping,
// and refusing to ship them would turn a degraded backup into no backup at all. It is a
// loud WARN, and the unit manifest simply carries the older dump set — which the restore
// confirm then surfaces as a skewed pair (P2) rather than silently pretending otherwise.
m.logger.Printf("[WARN] [offbox] pre-push dump leg failed (%v) — continuing with the existing dumps; the snapshot's DB half may be older than its files", dErr)
} else {
m.logger.Printf("[INFO] [offbox] pre-push dump leg completed in %s — snapshot pair is coherent", time.Since(dumpStart).Round(time.Millisecond))
}
runResult, runErr = m.runOffboxInternal(ctx, apps, base, env, t)
backedUp = runResult.backedUp
missing = runResult.missing
endStamp()
}
// Sorted names of apps whose enlargement was blocked this run (replaces the persisted set; empty clears).
var blockedNames []string
@@ -179,8 +179,10 @@ func (p *offboxProgressState) setPhase(phase string) {
p.mu.Unlock()
}
// OffboxPhaseShares / OffboxPhaseRetention are the post-app-loop stages.
// OffboxPhaseDump is the PRE-app-loop stage (R-44, v0.148.0); OffboxPhaseShares /
// OffboxPhaseRetention are the post-app-loop stages.
const (
OffboxPhaseDump = "dump"
OffboxPhaseShares = "shares"
OffboxPhaseRetention = "retention"
)
@@ -0,0 +1,400 @@
package backup
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
)
// Offsite reconstitution (R-43, v0.148.0) — the leg that was missing.
//
// Until v0.148.0 NO offsite path could restore a database. The two „visszaállítás" buttons staged
// files into a scratch folder and never touched postgres; the place-to-live button merged only the
// files MISSING from the live tree (`rsync --ignore-existing`) and never replayed a dump. For a
// DB-indexed app — most of the catalog — that combination cannot bring content back: the bytes
// return and the application still cannot see them, because its index lives in the database.
// Measured live on 2026-07-19 (DIAG-immich-restore-2026-07-19): 11 photos, files intact on disk,
// timeline empty, two "successful" restores that merged 0 files.
//
// ReconstituteFromOffsite is the honest version of that operation: it takes the CHOSEN snapshot's
// coherent pair and makes the live app equal to it — files overwritten to the snapshot's version,
// database replayed from the same snapshot's dump, app restarted. It is deliberately a different
// function from PlaceOffsiteRestore rather than a flag on it, because the two have opposite file
// semantics and conflating them is exactly how the missing-only merge came to be presented as a
// restore.
//
// Two invariants hold throughout:
//
// - NOTHING IS EVER DELETED. The file copy overwrites and adds; it never carries `--delete`. A
// file the customer created after the snapshot survives the restore as an extra. That is the
// house boundary — a restore that silently removed newer work would be a data-loss event
// wearing a recovery button's label.
// - THE UNDO EXISTS BEFORE THE ACT. A safety dump of the live database is written, and verified
// present on disk, BEFORE anything is stopped, overwritten or replayed. If that dump cannot be
// taken, the whole operation refuses with zero changes — a replay whose previous state was not
// captured is not a restore, it is an overwrite with no way back.
// offsitePreDump runs the coherence pre-phase's dump leg (nil seam → runDBDumpsInternal, which also
// refreshes the recovery units so the manifests enumerate the dumps just written). Extracted as a
// seam because the ORDER — dumps strictly before the restic capture — is the entire mechanism of
// R-44, and an ordering guarantee that no test can observe is one refactor away from silently
// reverting to the behaviour that produced DIAG-immich-restore-2026-07-19.
func (m *Manager) offsitePreDump(ctx context.Context) error {
if m.offsitePreDumpFn != nil {
return m.offsitePreDumpFn(ctx)
}
return m.runDBDumpsInternal(ctx)
}
// SetOffsitePreDumpFn overrides the offsite dump pre-phase (tests; no Docker needed).
func (m *Manager) SetOffsitePreDumpFn(fn func(ctx context.Context) error) { m.offsitePreDumpFn = fn }
// preRestoreDumpPrefix marks the safety dumps taken immediately before a reconstitution. They live
// in the app's own unit db-dumps dir so `ListDumpFiles` surfaces them beside the regular dumps —
// they ARE the undo, and an undo the customer cannot see is not much of one. The regular replay
// loop matches `<stack>-<dbtype>.sql` exactly, so a prefixed file is never mistaken for a source.
const preRestoreDumpPrefix = "pre-restore-"
// OffsiteReconstituteResult reports what a reconstitution actually did, so the flash can state an
// OUTCOME instead of a mechanism. Every field here exists because the v0.147 flash could not say it.
type OffsiteReconstituteResult struct {
SnapshotID string
FilesPlaced int
DBsReplayed int
SafetyDump string // path of the pre-restore dump (the undo), "" when the app has no DB
DumpsAt time.Time // when the snapshot's DB half was taken (zero = unknown/legacy unit)
OffsiteRunID string // "" for a pre-v0.148 snapshot — an unverified pair
Skewed bool // the snapshot carries no coherence stamp: files and DB may differ in age
LooksEmpty bool // R-44 sniff on the dump about to be replayed
}
// fullPlaceCopier returns the FULL-restore file copier (nil seam → rsyncRestoreOverwrite).
// Deliberately NOT placeCopier(): that one is `--ignore-existing`, whose whole purpose is to leave
// live files alone, which is precisely what a full restore must not do.
func (m *Manager) fullPlaceCopier() func(src, dst string) (int, error) {
if m.offboxFullPlaceCopier != nil {
return m.offboxFullPlaceCopier
}
return rsyncRestoreOverwrite
}
// rsyncRestoreOverwrite copies src over dst: `rsync -a --itemize-changes`, with NO
// `--ignore-existing` (a changed file becomes the snapshot's version) and NO `--delete` (an extra
// file at dst survives). Returns the number of regular files transferred.
func rsyncRestoreOverwrite(src, dst string) (int, error) {
if err := os.MkdirAll(dst, 0755); err != nil {
return 0, fmt.Errorf("mkdir %s: %w", dst, err)
}
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Minute)
defer cancel()
cmd := exec.CommandContext(ctx, "rsync", "-a", "--itemize-changes",
strings.TrimRight(src, "/")+"/", strings.TrimRight(dst, "/")+"/")
out, err := cmd.CombinedOutput()
if err != nil {
return 0, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
}
return countRestoredFiles(string(out)), nil
}
// writeSafetyDump dumps every live database of stack into the app's unit db-dumps dir under the
// `pre-restore-` prefix, and returns the first dump's path. Returns ("", nil) when the app has no
// database at all — a no-DB app has nothing to undo and must flow exactly as it did before
// v0.148.0 (no dump, no replay, no behaviour change).
//
// A discovered database that CANNOT be dumped is a hard error: it means the undo would not exist.
func (m *Manager) writeSafetyDump(ctx context.Context, stackName, nsRoot string) (string, error) {
discover := m.discoverDBs
if discover == nil {
discover = func(ctx context.Context) ([]DiscoveredDB, error) {
return DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames())
}
}
dbs, err := discover(ctx)
if err != nil {
return "", fmt.Errorf("a biztonsági mentés előtt nem sikerült felderíteni az adatbázisokat: %w", err)
}
var mine []DiscoveredDB
for _, db := range dbs {
if db.StackName == stackName {
mine = append(mine, db)
}
}
if len(mine) == 0 {
return "", nil // no DB → nothing to undo → scenario E flows unchanged
}
dumpDir := AppDBDumpPath(nsRoot, stackName)
if err := os.MkdirAll(dumpDir, 0755); err != nil {
return "", fmt.Errorf("a biztonsági mentés könyvtára nem hozható létre: %w", err)
}
stamp := time.Now().UTC().Format("20060102T150405Z")
first := ""
for _, db := range mine {
res := m.dumpForSafety(ctx, db, dumpDir)
if res.Error != nil {
return "", fmt.Errorf("a jelenlegi adatbázis biztonsági mentése sikertelen (%s): %w — a visszaállítás nem indult el", db.ContainerName, res.Error)
}
// DumpOne writes `<stack>-<dbtype>.sql`; rename it under the safety prefix so it can never be
// picked up as a replay SOURCE and can never overwrite the app's real dump.
safe := filepath.Join(dumpDir, fmt.Sprintf("%s%s-%s-%s.sql", preRestoreDumpPrefix, stamp, stackName, db.DBType))
if res.FilePath != safe {
if err := os.Rename(res.FilePath, safe); err != nil {
return "", fmt.Errorf("a biztonsági mentés véglegesítése sikertelen: %w", err)
}
}
if first == "" {
first = safe
}
m.logger.Printf("[INFO] [offbox] %s: pre-restore safety dump written → %s (%s)", stackName, filepath.Base(safe), humanizeBytes(res.Size))
}
return first, nil
}
// dumpForSafety is the DumpOne seam for the safety dump (tests inject; nil → the real DumpOne).
func (m *Manager) dumpForSafety(ctx context.Context, db DiscoveredDB, dumpDir string) DumpResult {
if m.safetyDumpFn != nil {
return m.safetyDumpFn(ctx, db, dumpDir)
}
return DumpOne(ctx, db, dumpDir, m.logger, m.isDebug())
}
// ReconstituteFromOffsite makes the live app equal to a restored full-scratch snapshot: files
// overwritten to the snapshot's version (extras survive, nothing deleted), then the snapshot's own
// DB dump replayed, with a safety dump of the current database taken first. Requires a completed
// FULL scratch restore (RestoreOffboxScratch with full=true). Single-flight.
func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (OffsiteReconstituteResult, error) {
var res OffsiteReconstituteResult
if !m.OffboxConfigured() {
return res, fmt.Errorf("off-box backup not configured")
}
if !isSafeStackName(stack) {
return res, fmt.Errorf("invalid stack name")
}
if m.stackProvider == nil {
return res, fmt.Errorf("stack provider not configured")
}
if err := m.acquireRunning(); err != nil {
return res, fmt.Errorf("egy másik mentési/visszaállítási művelet már fut")
}
defer m.releaseRunning()
scratch, _, err := m.offboxRestoreScratchDir(stack)
if err != nil {
return res, err
}
if _, sErr := os.Stat(scratch); sErr != nil {
return res, fmt.Errorf("nincs előkészített teljes visszaállítás — futtass előbb egy teljes visszaállítást")
}
id, paths, err := m.offboxLatestSnapshot(ctx, stack)
if err != nil {
return res, err
}
res.SnapshotID = id
hdd := strings.TrimSpace(m.stackProvider.GetStackHDDPath(stack))
if hdd == "" {
return res, fmt.Errorf("a(z) %s nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat", stack)
}
liveNs := m.namespaceRoot(hdd)
placements, err := mapOffsiteRestorePaths(paths, stack, scratch, liveNs)
if err != nil {
return res, err // whole-placement refusal (no partial writes)
}
// Stat pre-pass over EVERY placement before the first copy — an incomplete scratch (e.g. only a
// unit-only restore was run) refuses with ZERO copies.
for _, pl := range placements {
if _, sErr := os.Stat(pl.src); sErr != nil {
return res, fmt.Errorf("a teljes visszaállítás hiányos (%s nincs meg) — futtass előbb egy teljes visszaállítást", filepath.Base(pl.src))
}
}
// The snapshot's coherence stamp, read from the RESTORED unit manifest (not the live one).
scratchUnit := ""
for _, pl := range placements {
if pl.isUnit {
scratchUnit = pl.src
break
}
}
if scratchUnit == "" {
return res, fmt.Errorf("a pillanatképben nincs mentési egység — a visszaállítás nem indítható")
}
scratchDumpDir := filepath.Join(scratchUnit, "db-dumps")
if man := readManifest(filepath.Join(scratchUnit, "manifest.json")); man != nil {
res.OffsiteRunID = man.OffsiteRunID
if man.DumpsAt != "" {
if t, pErr := time.Parse(time.RFC3339, man.DumpsAt); pErr == nil {
res.DumpsAt = t
}
}
}
// A pre-v0.148 snapshot carries no stamp: its dump was whatever the 02:30 local run left behind,
// so the pair's two halves may be hours or days apart. Surfaced, never blocked — the confirm
// dialog says so and the safety dump makes it reversible.
res.Skewed = res.OffsiteRunID == ""
res.LooksEmpty = m.sniffScratchDump(scratchDumpDir, stack)
// --- THE UNDO, BEFORE THE ACT ---------------------------------------------------------------
// Taken while the stack is still UP (a stopped database cannot be dumped) and before a single
// byte is overwritten, so a failure here aborts with the live app completely untouched.
safety, err := m.writeSafetyDump(ctx, stack, liveNs)
if err != nil {
return res, err
}
res.SafetyDump = safety
hasDB := safety != ""
if hasDB {
if _, sErr := os.Stat(safety); sErr != nil {
// Fail-closed: never replay when the undo is not verifiably on disk.
return res, fmt.Errorf("a biztonsági mentés nem található a lemezen — a visszaállítás biztonsági okból nem indult el")
}
}
// --- FILES ----------------------------------------------------------------------------------
if err := m.stackProvider.StopStack(stack); err != nil {
m.logger.Printf("[WARN] [offbox] could not stop %s before reconstitution: %v (continuing)", stack, err)
}
copier := m.fullPlaceCopier()
for _, pl := range placements {
if pl.isUnit {
// The live recovery unit is still never overwritten — it is the LOCAL restore path's
// source and clobbering it would trade one recovery route for another. The snapshot's
// dump is replayed from the scratch unit instead, so nothing is lost by skipping it.
continue
}
n, cErr := copier(pl.src, pl.dst)
if cErr != nil {
// Best-effort bring-up: leaving the app stopped after a partial copy would turn a failed
// restore into an outage.
if sErr := m.stackProvider.StartStack(stack); sErr != nil {
m.logger.Printf("[WARN] [offbox] %s: restart after failed placement also failed: %v", stack, sErr)
}
return res, fmt.Errorf("a(z) %s fájljainak visszaállítása sikertelen: %w", stack, cErr)
}
res.FilesPlaced += n
}
// --- DATABASE -------------------------------------------------------------------------------
// The stack must be UP for the replay: ImportDump talks to the running container using its own
// discovered credentials (the same precedence RestoreFromRecoveryUnit uses — the logical dump
// wins over whatever the file copy just laid down for the DB's own data dir).
if err := m.stackProvider.StartStack(stack); err != nil {
return res, fmt.Errorf("a(z) %s újraindítása sikertelen a fájlok visszaállítása után: %w", stack, err)
}
if hasDB {
n, iErr := m.reimportDBDumpsFrom(ctx, stack, scratchDumpDir)
res.DBsReplayed = n
if iErr != nil {
return res, fmt.Errorf("az adatbázis visszaállítása sikertelen: %w — a korábbi állapot mentése megvan: %s", iErr, filepath.Base(safety))
}
}
if err := m.waitForHealthy(stack, 90*time.Second); err != nil {
m.logger.Printf("[WARN] [offbox] %s reconstituted but health check failed: %v", stack, err)
}
m.logger.Printf("[INFO] [offbox] reconstituted %s from snapshot %s: %d file(s) placed, %d DB dump(s) replayed, safety dump=%s, skewed=%v",
stack, id, res.FilesPlaced, res.DBsReplayed, filepath.Base(safety), res.Skewed)
return res, nil
}
// OffsitePairInfo describes the {DB, files} pair sitting in a prepared full-restore scratch, so the
// confirm dialog can tell the customer what they are about to restore BEFORE they commit to it.
// Everything here is honesty-surface: none of it blocks the operation.
type OffsitePairInfo struct {
Ready bool
DumpsAt time.Time // when the DB half was taken (zero = legacy unit, age unknown)
Skewed bool // no coherence stamp → the two halves may be from different times
LooksEmpty bool // R-44 sniff: the dump has an accounts table with no rows
HasDump bool
}
// OffsiteScratchPair reads the prepared scratch's unit manifest and reports what the pair looks
// like. Cheap and read-only — safe to call from a page render.
func (m *Manager) OffsiteScratchPair(stack string) OffsitePairInfo {
var info OffsitePairInfo
if !isSafeStackName(stack) {
return info
}
scratch, _, err := m.offboxRestoreScratchDir(stack)
if err != nil {
return info
}
// The unit sits at <scratch>/<oldNs>/backups/primary/<stack>; the old namespace is unknown here,
// so find it rather than reconstructing it.
unit := findScratchUnitDir(scratch, stack)
if unit == "" {
return info
}
info.Ready = true
dumpDir := filepath.Join(unit, "db-dumps")
if entries, rErr := os.ReadDir(dumpDir); rErr == nil {
for _, e := range entries {
if !e.IsDir() && filepath.Ext(e.Name()) == ".sql" && !strings.HasPrefix(e.Name(), preRestoreDumpPrefix) {
info.HasDump = true
break
}
}
}
if man := readManifest(filepath.Join(unit, "manifest.json")); man != nil {
if man.DumpsAt != "" {
if t, pErr := time.Parse(time.RFC3339, man.DumpsAt); pErr == nil {
info.DumpsAt = t
}
}
info.Skewed = man.OffsiteRunID == ""
} else {
info.Skewed = true
}
if info.HasDump {
info.LooksEmpty = m.sniffScratchDump(dumpDir, stack)
}
return info
}
// findScratchUnitDir locates `backups/primary/<stack>` anywhere under a restored scratch. restic
// rebuilds absolute source paths under the target, and the snapshot may have come from a drive that
// no longer exists on this box, so the prefix cannot be assumed.
func findScratchUnitDir(scratch, stack string) string {
found := ""
suffix := filepath.Join("backups", "primary", stack)
_ = filepath.Walk(scratch, func(path string, fi os.FileInfo, err error) error {
if err != nil || found != "" {
return nil //nolint:nilerr // a walk error on one branch must not abort the search
}
if fi.IsDir() && strings.HasSuffix(path, suffix) {
found = path
}
return nil
})
return found
}
// sniffScratchDump runs the R-44 content sniff over the dump about to be replayed. Best-effort and
// warn-level: any failure to read simply reports "no warning", because a sniff that blocks a
// restore is worse than the skew it describes.
func (m *Manager) sniffScratchDump(dumpDir, stack string) bool {
entries, err := os.ReadDir(dumpDir)
if err != nil {
return false
}
for _, e := range entries {
name := e.Name()
if e.IsDir() || filepath.Ext(name) != ".sql" || strings.HasPrefix(name, preRestoreDumpPrefix) {
continue
}
dbType := DBTypePostgres
if strings.Contains(name, string(DBTypeMariaDB)) {
dbType = DBTypeMariaDB
}
if v := ValidateDump(filepath.Join(dumpDir, name), dbType); v.LooksEmpty {
m.logger.Printf("[WARN] [offbox] %s: the snapshot dump %s has no account rows — it may predate the customer's data", stack, name)
return true
}
}
return false
}
@@ -0,0 +1,424 @@
package backup
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-43/R-44 (v0.148.0) — the coherent-pair + true-restore tests.
//
// These exist because the product shipped a restore button for months that could not restore.
// DIAG-immich-restore-2026-07-19: 11 photos, files intact, timeline empty, two "successful"
// restores that merged 0 files and never touched postgres. Every test below asserts a behaviour
// whose absence produced that outcome, so each one is a regression guard for a real incident
// rather than a description of the current implementation.
// recordingProvider records stop/start call ORDER so the reconstitution sequence can be asserted.
type recordingProvider struct {
offbox3aProvider
calls []string
}
func (p *recordingProvider) StopStack(string) error { p.calls = append(p.calls, "stop"); return nil }
func (p *recordingProvider) StartStack(string) error { p.calls = append(p.calls, "start"); return nil }
// The app really is up again after StartStack, so the post-restore health wait returns at once.
// Leaving it false would make each test sit through the full 90s deadline.
func (p *recordingProvider) RefreshAndIsRunning(string) bool { return true }
// recoveryProvider adds the recovery info CaptureRecoveryUnit needs (the shared 3a provider has none).
type recoveryProvider struct {
offbox3aProvider
stackDir string
}
func (p *recoveryProvider) GetStackRecoveryInfo(name string) (RecoveryInfo, bool) {
return RecoveryInfo{DisplayName: "Immich", StackDir: p.stackDir}, name == "immich"
}
// pgDump builds a structurally valid postgres dump big enough to clear ValidateDump's 100-byte
// floor, with the accounts-table COPY block carrying `rows` rows. The R-44 sniff runs only on a
// dump that already passes structural validation, so a toy fixture would silently skip it.
func pgDump(rows int) string {
const head = `-- PostgreSQL database dump
-- Dumped from database version 16.10
SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
CREATE TABLE public.asset (id uuid NOT NULL);
CREATE TABLE public."user" (id uuid NOT NULL, email text);
COPY public."user" (id, email) FROM stdin;
`
var b strings.Builder
b.WriteString(head)
for i := 0; i < rows; i++ {
b.WriteString("id-x\tuser@example.invalid\n")
}
b.WriteString("\\.\n") // the COPY-block terminator
return b.String()
}
// reconFixture builds a manager with a COMPLETED full scratch for `immich`, a snapshot whose unit
// carries the given coherence stamp, and injectable copy/dump/import seams.
func reconFixture(t *testing.T, runID, dumpsAt string, dumpBody string) (*Manager, *recordingProvider, *[]string) {
t.Helper()
drive := t.TempDir()
m, sett := newOffboxManager(t)
prov := &recordingProvider{offbox3aProvider: offbox3aProvider{
hdd: map[string]string{"immich": drive}, binds: map[string][]ClassifiedBind{}, has: map[string]bool{},
}}
m.SetStackProvider(prov)
if err := sett.AddStoragePath(settings.StoragePath{Path: drive, Label: "USB", Schedulable: true}); err != nil {
t.Fatal(err)
}
scratch, liveNs, err := m.offboxRestoreScratchDir("immich")
if err != nil {
t.Fatal(err)
}
oldNs := "/felhomdata/ns"
unitP := oldNs + "/backups/primary/immich"
dataP := oldNs + "/appdata/immich"
placements, err := mapOffsiteRestorePaths([]string{unitP, dataP}, "immich", scratch, liveNs)
if err != nil {
t.Fatal(err)
}
for _, pl := range placements {
if err := os.MkdirAll(pl.src, 0o755); err != nil {
t.Fatal(err)
}
if pl.isUnit {
dd := filepath.Join(pl.src, "db-dumps")
if err := os.MkdirAll(dd, 0o755); err != nil {
t.Fatal(err)
}
if dumpBody != "" {
if err := os.WriteFile(filepath.Join(dd, "immich-postgres.sql"), []byte(dumpBody), 0o644); err != nil {
t.Fatal(err)
}
}
man := &RecoveryManifest{SchemaVersion: 1, AppName: "immich", OffsiteRunID: runID, DumpsAt: dumpsAt}
if err := writeManifest(filepath.Join(pl.src, "manifest.json"), man); err != nil {
t.Fatal(err)
}
}
}
m.SetOffboxFreeFn(func(string) int64 { return 100 << 30 })
m.SetOffboxSizer(func(string) int64 { return 1 << 20 })
m.SetOffboxRunner(func(_ context.Context, _ []string, args ...string) ([]byte, error) {
if contains(args, "snapshots") {
return []byte(`[{"short_id":"snap1","time":"2026-07-19T06:00:00Z","paths":["` + unitP + `","` + dataP + `"]}]`), nil
}
return nil, nil
})
// Seams: one DB, a safety dump that really writes a file, and a recording importer.
db := DiscoveredDB{StackName: "immich", ContainerName: "immich-postgres", DBType: DBTypePostgres}
m.discoverDBs = func(context.Context) ([]DiscoveredDB, error) { return []DiscoveredDB{db}, nil }
m.SetSafetyDumpFn(func(_ context.Context, d DiscoveredDB, dir string) DumpResult {
p := filepath.Join(dir, "immich-postgres.sql")
_ = os.MkdirAll(dir, 0o755)
_ = os.WriteFile(p, []byte(pgDump(1)), 0o644)
return DumpResult{DB: d, FilePath: p, Size: 42}
})
var imported []string
m.importDBDump = func(_ context.Context, _ DiscoveredDB, p string) error {
imported = append(imported, p)
return nil
}
m.SetOffboxFullPlaceCopier(func(_, _ string) (int, error) { return 3, nil })
return m, prov, &imported
}
// TestReconstituteReplaysDBAndOrdersOperations is Scenario C: the whole point of R-43. A restore of
// a DB-indexed app must stop the app, place files, restart it and REPLAY the snapshot's dump — and
// the safety dump must exist before any of it. Before v0.148.0 the replay simply did not happen,
// which is why the photos never came back.
func TestReconstituteReplaysDBAndOrdersOperations(t *testing.T) {
m, prov, imported := reconFixture(t, "20260719T060000Z", "2026-07-19T06:00:00Z", pgDump(1))
res, err := m.ReconstituteFromOffsite(context.Background(), "immich")
if err != nil {
t.Fatalf("reconstitute: %v", err)
}
if res.DBsReplayed != 1 {
t.Fatalf("expected the snapshot dump to be replayed exactly once, got %d — this is the R-43 defect", res.DBsReplayed)
}
if len(*imported) != 1 || !strings.Contains((*imported)[0], "immich-postgres.sql") {
t.Fatalf("expected an import of the snapshot dump, got %v", *imported)
}
// The dump replayed must come from the SCRATCH unit, never the live one: the live unit is
// deliberately not overwritten, so replaying from it would replay the CURRENT database back over
// itself and restore nothing.
if !strings.Contains((*imported)[0], "offsite-restore") {
t.Fatalf("replay source must be the restored scratch unit, got %s", (*imported)[0])
}
if res.FilesPlaced != 3 {
t.Fatalf("expected the userdata placement to be counted, got %d", res.FilesPlaced)
}
// stop BEFORE the file copy, start BEFORE the replay (ImportDump needs a live container).
if got := strings.Join(prov.calls, ","); got != "stop,start" {
t.Fatalf("expected stop then start around the restore, got %q", got)
}
if res.SafetyDump == "" {
t.Fatal("no safety dump recorded — the undo must exist")
}
if _, err := os.Stat(res.SafetyDump); err != nil {
t.Fatalf("safety dump not on disk: %v", err)
}
if !strings.HasPrefix(filepath.Base(res.SafetyDump), preRestoreDumpPrefix) {
t.Fatalf("safety dump must carry the pre-restore prefix so it is never replayed as a source, got %s", filepath.Base(res.SafetyDump))
}
}
// TestReconstituteRefusesWhenSafetyDumpFails is the RED-PROOF for the undo invariant: a replay whose
// previous state was not captured is an overwrite with no way back, so it must not happen at all —
// and it must abort with the live app untouched (no stop, no copy).
func TestReconstituteRefusesWhenSafetyDumpFails(t *testing.T) {
m, prov, imported := reconFixture(t, "run1", "2026-07-19T06:00:00Z", pgDump(1))
m.SetSafetyDumpFn(func(_ context.Context, d DiscoveredDB, _ string) DumpResult {
return DumpResult{DB: d, Error: context.DeadlineExceeded}
})
var copied bool
m.SetOffboxFullPlaceCopier(func(_, _ string) (int, error) { copied = true; return 1, nil })
_, err := m.ReconstituteFromOffsite(context.Background(), "immich")
if err == nil {
t.Fatal("expected a refusal when the safety dump cannot be taken")
}
if len(*imported) != 0 {
t.Fatalf("REPLAYED WITHOUT AN UNDO — the exact thing the invariant forbids: %v", *imported)
}
if copied {
t.Fatal("files were overwritten despite the refusal — the abort must leave live data untouched")
}
if len(prov.calls) != 0 {
t.Fatalf("the app was stopped despite the refusal, got %v", prov.calls)
}
}
// TestReconstituteNoDBAppMakesNoDumpOrImportCalls is Scenario E: an app without a database must flow
// exactly as before — no safety dump, no replay — so the new leg cannot regress the simple case.
func TestReconstituteNoDBAppMakesNoDumpOrImportCalls(t *testing.T) {
m, _, imported := reconFixture(t, "run1", "2026-07-19T06:00:00Z", "")
m.discoverDBs = func(context.Context) ([]DiscoveredDB, error) { return nil, nil }
dumped := 0
m.SetSafetyDumpFn(func(_ context.Context, d DiscoveredDB, _ string) DumpResult {
dumped++
return DumpResult{DB: d}
})
res, err := m.ReconstituteFromOffsite(context.Background(), "immich")
if err != nil {
t.Fatalf("reconstitute: %v", err)
}
if dumped != 0 {
t.Fatalf("a no-DB app must not produce a safety dump, got %d call(s)", dumped)
}
if len(*imported) != 0 {
t.Fatalf("a no-DB app must not import anything, got %v", *imported)
}
if res.SafetyDump != "" || res.DBsReplayed != 0 {
t.Fatalf("unexpected DB activity: safety=%q replayed=%d", res.SafetyDump, res.DBsReplayed)
}
}
// TestReconstituteSurfacesLegacySkewedPair is Scenario D: a pre-v0.148 snapshot carries no coherence
// stamp, so its two halves may be from different times. That must be SURFACED (and reversible), never
// blocked — the customer's own judgement is the gate, and refusing would deny a legitimate restore.
func TestReconstituteSurfacesLegacySkewedPair(t *testing.T) {
m, _, imported := reconFixture(t, "", "", pgDump(1))
res, err := m.ReconstituteFromOffsite(context.Background(), "immich")
if err != nil {
t.Fatalf("a legacy pair must still be restorable, got refusal: %v", err)
}
if !res.Skewed {
t.Fatal("an unstamped (pre-v0.148) snapshot must report Skewed so the confirm can say so")
}
if len(*imported) != 1 {
t.Fatalf("the legacy restore must still replay, got %v", *imported)
}
}
// TestReconstituteFlagsCustomerEmptyDump is the R-44 sniff at the restore end: the immich dump that
// started all of this was structurally valid and contained zero users. Restoring it is allowed, but
// the customer must be told before they commit.
func TestReconstituteFlagsCustomerEmptyDump(t *testing.T) {
// A valid postgres dump whose accounts table has NO rows — the 2026-07-19 shape exactly.
m, _, _ := reconFixture(t, "run1", "2026-07-19T06:00:00Z", pgDump(0))
res, err := m.ReconstituteFromOffsite(context.Background(), "immich")
if err != nil {
t.Fatalf("the sniff must never block a restore: %v", err)
}
if !res.LooksEmpty {
t.Fatal("a dump with an empty accounts table must raise the warn-level signal")
}
}
// TestOffsiteScratchPairReportsWhatTheConfirmNeeds covers the page-render surface: the confirm can
// only be honest if this reports the pair's age and warnings before anything is started.
func TestOffsiteScratchPairReportsWhatTheConfirmNeeds(t *testing.T) {
m, _, _ := reconFixture(t, "run1", "2026-07-19T06:00:00Z",
"-- PostgreSQL database dump\nCREATE TABLE a();\nCOPY public.\"user\" (id) FROM stdin;\n7\n\\.\n")
info := m.OffsiteScratchPair("immich")
if !info.Ready || !info.HasDump {
t.Fatalf("expected a ready pair with a dump, got %+v", info)
}
if info.Skewed {
t.Fatal("a stamped snapshot must not be reported as skewed")
}
if info.LooksEmpty {
t.Fatal("a dump with account rows must not be flagged empty")
}
want, _ := time.Parse(time.RFC3339, "2026-07-19T06:00:00Z")
if !info.DumpsAt.Equal(want) {
t.Fatalf("DumpsAt = %v, want %v", info.DumpsAt, want)
}
}
// --- R-44: the coherence pre-phase -----------------------------------------------------------
// TestOffsiteRunDumpsBeforeCapture is Scenarios A + B. The ORDER is the entire mechanism: dumps
// must be refreshed BEFORE restic captures, so the snapshot pairs this run's database with this
// run's files. Reversed, the snapshot would hold rows pointing at files that were never captured.
//
// It also asserts the ordering on the NIGHTLY entry point (RunOffboxBackup, no progress sink), not
// just the manual one — before v0.148.0 the nightly ordering was an accident of two independent
// scheduler entries at 02:30 and 04:15, which a schedule edit could silently invert.
func TestOffsiteRunDumpsBeforeCapture(t *testing.T) {
drive := t.TempDir()
m, sett, prov := classifiedOffboxManager(t, drive)
mkUnit(t, drive, "immich")
if err := os.MkdirAll(filepath.Join(drive, "appdata", "immich"), 0o755); err != nil {
t.Fatal(err)
}
prov.hdd["immich"] = drive
prov.has["immich"] = true
prov.binds["immich"] = []ClassifiedBind{mandatoryHDD("appdata/immich")}
_ = sett.SetAppOffbox("immich", true)
var order []string
m.SetOffsitePreDumpFn(func(context.Context) error {
order = append(order, "dump")
return nil
})
m.SetOffboxRunner(func(_ context.Context, _ []string, args ...string) ([]byte, error) {
switch {
case contains(args, "cat") && contains(args, "config"):
return []byte(`{"version":2}`), nil
case contains(args, "backup"):
order = append(order, "capture")
return nil, nil
case contains(args, "snapshots"):
return []byte(`[]`), nil
case contains(args, "stats"):
return []byte(`{"total_size":123}`), nil
}
return nil, nil
})
if err := m.RunOffboxBackup(context.Background()); err != nil {
t.Fatalf("run: %v", err)
}
if len(order) < 2 {
t.Fatalf("expected both a dump and a capture, got %v", order)
}
if order[0] != "dump" {
t.Fatalf("the dump leg MUST precede the capture (R-44); got %v", order)
}
if order[1] != "capture" {
t.Fatalf("expected the capture immediately after the dump, got %v", order)
}
}
// TestOffsiteRunContinuesWhenDumpLegFails is the data-first rule: a dump failure degrades the
// snapshot's DB half but must NOT abort the push. Refusing to ship the files would turn a partial
// backup into no backup at all — strictly worse for the customer.
func TestOffsiteRunContinuesWhenDumpLegFails(t *testing.T) {
drive := t.TempDir()
m, sett, prov := classifiedOffboxManager(t, drive)
mkUnit(t, drive, "immich")
if err := os.MkdirAll(filepath.Join(drive, "appdata", "immich"), 0o755); err != nil {
t.Fatal(err)
}
prov.hdd["immich"] = drive
prov.has["immich"] = true
prov.binds["immich"] = []ClassifiedBind{mandatoryHDD("appdata/immich")}
_ = sett.SetAppOffbox("immich", true)
m.SetOffsitePreDumpFn(func(context.Context) error { return context.DeadlineExceeded })
cap := &backupCapture{}
m.SetOffboxRunner(cap.runner())
if err := m.RunOffboxBackup(context.Background()); err != nil {
t.Fatalf("a dump failure must not fail the whole run: %v", err)
}
if cap.backups != 1 {
t.Fatalf("the files must still be pushed after a dump failure, got %d capture(s)", cap.backups)
}
}
// TestCaptureRecoveryUnitStampsAndCarriesRunID covers the stamp that makes a pair verifiable at
// restore time, and the trap beside it: the PERIODIC refresh must neither invent a coherence claim
// nor erase one a real run established.
func TestCaptureRecoveryUnitStampsAndCarriesRunID(t *testing.T) {
drive := t.TempDir()
m, _, base := classifiedOffboxManager(t, drive)
base.hdd["immich"] = drive
// CaptureRecoveryUnit needs real recovery info + a compose dir to read; the shared fixture
// provider returns none, so wrap it rather than widening a struct four other test files use.
stackDir := filepath.Join(t.TempDir(), "immich")
if err := os.MkdirAll(stackDir, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(stackDir, "docker-compose.yml"), []byte("services: {}\n"), 0o644); err != nil {
t.Fatal(err)
}
m.SetStackProvider(&recoveryProvider{offbox3aProvider: *base, stackDir: stackDir})
// 1) A run in flight stamps the manifest.
end := m.beginOffsiteRunStamp("run-A")
if err := m.CaptureRecoveryUnit("immich"); err != nil {
t.Fatalf("capture: %v", err)
}
end()
man := readManifest(RecoveryUnitManifestPath(drive, "immich"))
if man == nil || man.OffsiteRunID != "run-A" {
t.Fatalf("expected the in-flight run id to be stamped, got %+v", man)
}
if man.DumpsAt == "" {
t.Fatal("a stamped unit must record when its dumps were taken")
}
// 2) A periodic refresh (no run in flight) must CARRY the stamp forward, not blank it — a unit
// that silently lost its stamp would be re-reported as a skewed legacy pair at restore time.
if err := m.CaptureRecoveryUnit("immich"); err != nil {
t.Fatalf("refresh: %v", err)
}
man2 := readManifest(RecoveryUnitManifestPath(drive, "immich"))
if man2 == nil || man2.OffsiteRunID != "run-A" {
t.Fatalf("the periodic refresh erased the coherence stamp: %+v", man2)
}
// 3) A NEW run re-stamps even though nothing else about the unit changed — the idempotent-skip
// must not swallow the one field the restore path reads.
end2 := m.beginOffsiteRunStamp("run-B")
if err := m.CaptureRecoveryUnit("immich"); err != nil {
t.Fatalf("capture 2: %v", err)
}
end2()
man3 := readManifest(RecoveryUnitManifestPath(drive, "immich"))
if man3 == nil || man3.OffsiteRunID != "run-B" {
t.Fatalf("a new run must re-stamp the unit, got %+v", man3)
}
}
@@ -30,6 +30,16 @@ const (
// SetOffboxFreeFn overrides the restore free-space probe (tests; the Windows go-test host has no df).
func (m *Manager) SetOffboxFreeFn(fn func(path string) int64) { m.offboxFreeFn = fn }
// SetOffboxFullPlaceCopier overrides the FULL-restore overwrite copier (tests; no rsync needed).
func (m *Manager) SetOffboxFullPlaceCopier(fn func(src, dst string) (int, error)) {
m.offboxFullPlaceCopier = fn
}
// SetSafetyDumpFn overrides the pre-restore safety dump (tests; no Docker needed).
func (m *Manager) SetSafetyDumpFn(fn func(ctx context.Context, db DiscoveredDB, dumpDir string) DumpResult) {
m.safetyDumpFn = fn
}
// offboxFree returns the free-space probe (nil seam → the real diskFreeBytes).
func (m *Manager) offboxFree() func(string) int64 {
if m.offboxFreeFn != nil {
+26 -2
View File
@@ -44,6 +44,15 @@ type RecoveryManifest struct {
DBDumps []string `json:"db_dumps"`
VolumeDumps []string `json:"volume_dumps"`
Checksums map[string]string `json:"checksums"` // sha256 of captured compose/ files
// R-43/R-44 (v0.148.0): the coherence stamp. An offsite run refreshes the dumps FIRST and then
// captures the unit, so a manifest carrying an OffsiteRunID asserts "the db-dumps/ in this unit
// were taken by that run" — i.e. the snapshot is an internally coherent {DB@T, files@T} pair.
// A manifest WITHOUT these fields is a pre-v0.148 unit whose dump age is unknown and may skew
// arbitrarily from the files beside it (the DIAG-immich-restore-2026-07-19 failure); the restore
// confirm surfaces that honestly rather than blocking. Empty on the periodic refresh, which must
// never claim a coherence it did not establish — it carries the prior stamp forward instead.
OffsiteRunID string `json:"offsite_run_id,omitempty"`
DumpsAt string `json:"dumps_at,omitempty"` // RFC3339 UTC — when this run's dump leg finished
}
// SetVersion records the controller version stamped into recovery-unit manifests.
@@ -107,13 +116,26 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
version := m.versionLocked()
manifestPath := RecoveryUnitManifestPath(nsRoot, stackName)
cur := readManifest(manifestPath)
// R-43/R-44: the coherence stamp of the offsite run currently in flight ("" on the periodic
// refresh and on the local dump run). When empty we CARRY THE PRIOR STAMP FORWARD rather than
// blanking it — a periodic refresh must neither claim a coherence it did not establish nor
// destroy the record of one that a real run did.
runID, dumpsAt := m.offsiteRunStamp()
if runID == "" && cur != nil {
runID, dumpsAt = cur.OffsiteRunID, cur.DumpsAt
}
// Skip if the unit is already current — avoids needless drive writes on the periodic refresh.
if cur := readManifest(manifestPath); cur != nil &&
// The run-id is part of "current": an offsite run must re-stamp the manifest even when nothing
// else changed, because the stamp is exactly the claim the restore path reads.
if cur != nil &&
cur.ControllerVer == version &&
stringMapEqual(cur.Checksums, checksums) &&
stringSliceEqual(cur.DBDumps, dbDumps) &&
stringSliceEqual(cur.VolumeDumps, volDumps) {
stringSliceEqual(cur.VolumeDumps, volDumps) &&
cur.OffsiteRunID == runID {
return nil
}
@@ -143,6 +165,8 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
DBDumps: dbDumps,
VolumeDumps: volDumps,
Checksums: checksums,
OffsiteRunID: runID,
DumpsAt: dumpsAt,
}
if err := writeManifest(manifestPath, manifest); err != nil {
return fmt.Errorf("writing manifest: %w", err)
+9 -1
View File
@@ -19,7 +19,15 @@ import (
// A dump whose DB container is not found is logged and skipped; an actual import FAILURE is returned
// (surfaced, not swallowed) so a failed data restore cannot read as success.
func (m *Manager) reimportDBDumps(ctx context.Context, stackName, nsRoot string) (int, error) {
dumpDir := AppDBDumpPath(nsRoot, stackName)
return m.reimportDBDumpsFrom(ctx, stackName, AppDBDumpPath(nsRoot, stackName))
}
// reimportDBDumpsFrom is reimportDBDumps with an EXPLICIT dump directory. The offsite
// reconstitution path (R-43) replays out of the restored SCRATCH unit rather than the live one:
// the local unit is deliberately never overwritten by a placement, so the dump that belongs to the
// chosen snapshot exists only under the scratch. Same discovery/import seams, same failure
// semantics — only the source directory differs.
func (m *Manager) reimportDBDumpsFrom(ctx context.Context, stackName, dumpDir string) (int, error) {
entries, err := os.ReadDir(dumpDir)
if err != nil {
if os.IsNotExist(err) {
+11
View File
@@ -815,6 +815,17 @@ func (s *Server) backupsRestoreHandler(w http.ResponseWriter, r *http.Request) {
}
}
data["OffboxScratchReady"] = ready
// R-43: the same prepared scratch also enables the TRUE restore (files + database). Its confirm
// has to state what the pair actually IS — how old the DB half is, whether the two halves even
// come from the same run, and whether the dump looks customer-empty — because a restore is the
// one operation whose result the customer cannot inspect until after committing to it.
pairs := map[string]backup.OffsitePairInfo{}
if s.backupMgr != nil {
for name := range ready {
pairs[name] = s.backupMgr.OffsiteScratchPair(name)
}
}
data["OffboxPairInfo"] = pairs
// R-7b: the shares source is not an app — it has no per-app toggle and no recovery unit — so it
// gets its own restore entry rather than a synthetic row in OffboxApps (which would also make it
// appear in the per-app offsite TOGGLE list on /backups/remote, where it does not belong).
@@ -3,6 +3,7 @@ package web
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/url"
"strconv"
@@ -340,6 +341,69 @@ func (s *Server) offboxRestoreHandler(w http.ResponseWriter, r *http.Request) {
offboxRedirectTo(w, r, "/backups/restore", "A távoli visszaállítás elindult — az állapot itt frissül.", false)
}
// offboxReconstituteHandler is the TRUE offsite restore (R-43, v0.148.0): files overwritten to the
// snapshot's version + that same snapshot's database replayed + the app restarted, with a safety
// dump of the current database taken first.
//
// It is a separate button from the missing-only place, not a flag on it. The two do opposite things
// to an existing file, and the v0.147 flash („hiányzó fájljai helyreállítva") described a mechanism
// that could report success after merging zero files while the customer's photos stayed invisible.
// The flash here states the OUTCOME instead — file count, database, backup timestamp, restart —
// because that is the only part the customer can check against what they see in the app.
func (s *Server) offboxReconstituteHandler(w http.ResponseWriter, r *http.Request) {
if s.backupMgr == nil || !s.backupMgr.OffboxConfigured() {
offboxRedirectTo(w, r, "/backups/restore", "A távoli mentési cél nincs beállítva.", true)
return
}
_ = r.ParseForm()
app := strings.TrimSpace(r.FormValue("app"))
if app == "" {
offboxRedirectTo(w, r, "/backups/restore", "Hiányzó alkalmazás.", true)
return
}
// This one overwrites live files and replays a database — it must never happen on a stray click.
if r.FormValue("confirm") != "1" {
offboxRedirectTo(w, r, "/backups/restore", "A teljes visszaállítás megerősítés nélkül nem hajtható végre.", true)
return
}
if s.backupMgr.IsRunning() {
offboxRedirectTo(w, r, "/backups/restore", "Egy mentési/visszaállítási művelet már fut.", true)
return
}
s.backupMgr.BeginRestoreOp("offbox-reconstitute", app)
go func() {
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Minute)
defer cancel()
res, err := s.backupMgr.ReconstituteFromOffsite(ctx, app)
if err != nil {
s.logger.Printf("[ERROR] [web] off-box reconstitute %s (async): %v", app, err)
s.backupMgr.EndRestoreOp(false, "A teljes visszaállítás sikertelen: "+err.Error())
return
}
s.logger.Printf("[INFO] [web] off-box reconstitute %s completed (async): files=%d dbs=%d snapshot=%s",
app, res.FilesPlaced, res.DBsReplayed, res.SnapshotID)
s.backupMgr.EndRestoreOp(true, reconstituteOutcomeMsg(app, res))
}()
offboxRedirectTo(w, r, "/backups/restore", "A teljes visszaállítás elindult — az állapot itt frissül.", false)
}
// reconstituteOutcomeMsg builds the OUTCOME flash for a completed reconstitution. Pure, so the
// wording is unit-testable — this string is the customer's only evidence that the operation did
// what its label promised, and the zero-file and no-database cases must each read truthfully rather
// than borrowing the confident sentence that belongs to the full case.
func reconstituteOutcomeMsg(app string, res backup.OffsiteReconstituteResult) string {
when := ""
if !res.DumpsAt.IsZero() {
when = " (mentés: " + res.DumpsAt.In(getTimezone()).Format("2006-01-02 15:04") + ")"
}
if res.DBsReplayed == 0 {
// A no-database app: saying "és az adatbázis" here would be a lie, and this is precisely the
// class of sentence the DIAG found being printed over a no-op.
return fmt.Sprintf("A(z) %s: %d fájl visszaállítva%s — az alkalmazás újraindult. Ennek az alkalmazásnak nincs adatbázisa.", app, res.FilesPlaced, when)
}
return fmt.Sprintf("A(z) %s: %d fájl és az adatbázis visszaállítva%s — az alkalmazás újraindult.", app, res.FilesPlaced, when)
}
// offboxVerifyCopyDeleteHandler removes ONE verification copy (v0.147.0, 4a).
//
// The only delete this slice adds, so it is deliberately narrow: it names a STACK, never a path — the
+2
View File
@@ -422,6 +422,8 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.offboxRestoreHandler(w, r)
case path == "/backup/offbox/place" && r.Method == http.MethodPost:
s.offboxPlaceHandler(w, r)
case path == "/backup/offbox/reconstitute" && r.Method == http.MethodPost:
s.offboxReconstituteHandler(w, r)
// v0.147.0 (4a): remove ONE verification copy. The only delete path this slice adds — see
// DeleteOffsiteRestoreCopy for the prefix guard.
case path == "/backup/offbox/verify-copy/delete" && r.Method == http.MethodPost:
@@ -215,6 +215,16 @@
/* A run is not only the per-app loop: the shares leg and the retention/prune step follow
it and can dominate the wall clock (40 of 57 seconds, measured). Name them, or the
card freezes on the last app's finished counters for that whole tail. */
/* The dump pre-phase runs BEFORE the per-app loop (R-44): it refreshes every app's
database dump so the snapshot pairs this run's files with this run's DB. On an app
with a large database it can dominate the early wall clock, and an unnamed silent
stretch at the start reads as a hang. */
if(p.phase === 'dump'){
text.textContent = 'Adatbázisok mentése a pillanatképhez…';
bar.style.width = '100%';
box.className = 'alert alert-info'; box.style.display = '';
return;
}
if(p.phase === 'retention'){
text.textContent = 'Karbantartás: régi mentések rendezése a távoli tárolón…';
bar.style.width = '100%';
@@ -93,7 +93,25 @@
<input type="hidden" name="app" value="{{.Name}}">
<button type="submit" class="btn btn-xs btn-outline">Helyreállítás az élő adatok közé (csak a hiányzó fájlok)</button>
</form>
<span class="form-hint" style="display:block;margin-top:.25rem">A meglévő fájlokat nem írja felül.</span>
<span class="form-hint" style="display:block;margin-top:.25rem">A meglévő fájlokat nem írja felül. Adatbázist nem állít vissza — törölt tartalom ettől nem jelenik meg újra.</span>
{{$pair := index $.OffboxPairInfo .Name}}
<form method="POST" action="/backup/offbox/reconstitute" style="display:inline">{{$.CSRFField}}
<input type="hidden" name="app" value="{{.Name}}">
<input type="hidden" name="confirm" value="1">
<button type="button" class="btn btn-xs btn-primary"
data-restore-app="{{.Name}}"
data-restore-when="{{if not $pair.DumpsAt.IsZero}}{{fmtTime $pair.DumpsAt}}{{end}}"
data-restore-skewed="{{if $pair.Skewed}}1{{end}}"
data-restore-empty="{{if $pair.LooksEmpty}}1{{end}}"
onclick="confirmFullRestore(this)">Teljes visszaállítás (fájlok + adatbázis)</button>
</form>
<span class="form-hint" style="display:block;margin-top:.25rem">A fájlokat a mentés szerinti változatra állítja vissza és az adatbázist is visszatölti. Semmit nem töröl: a mentés óta létrejött fájlok megmaradnak. A jelenlegi adatbázisról előtte biztonsági mentés készül.</span>
{{if $pair.Skewed}}
<span class="form-hint" style="display:block;margin-top:.25rem;color:var(--warn)">Az adatbázis-mentés régebbi{{if not $pair.DumpsAt.IsZero}} ({{fmtTime $pair.DumpsAt}}){{end}} — a fájlok és az adatbázis eltérő időpontból származnak.</span>
{{end}}
{{if $pair.LooksEmpty}}
<span class="form-hint" style="display:block;margin-top:.25rem;color:var(--warn)">A mentett adatbázis üresnek tűnik (nincs benne felhasználói fiók) — elképzelhető, hogy a mentés korábbi, mint az adataid.</span>
{{end}}
{{end}}{{end}}
{{template "app_list_row_end"}}
{{end}}
@@ -217,6 +235,26 @@ function confirmDeleteVerifyCopy(btn){
});
});
}
/* R-43: the true offsite restore overwrites live files and replays a database, so it double-confirms
and — unlike the old missing-only merge — states the DB half's age and any warning BEFORE the
customer commits. The honesty lines are already rendered under the button; repeating the decisive
ones here means the person clicking "Igen" has read them. */
function confirmFullRestore(btn){
var app = btn.getAttribute('data-restore-app') || '';
var when = btn.getAttribute('data-restore-when') || '';
var skewed = btn.getAttribute('data-restore-skewed') === '1';
var empty = btn.getAttribute('data-restore-empty') === '1';
var q = 'Teljes visszaállítás: ' + app + (when ? ' — a mentés ideje: ' + when : '') + '.';
if (skewed) { q += ' FIGYELEM: a fájlok és az adatbázis eltérő időpontból származnak.'; }
if (empty) { q += ' FIGYELEM: a mentett adatbázis üresnek tűnik.'; }
q += ' A fájlok a mentés szerinti változatra állnak vissza, semmi nem törlődik.';
felhomConfirm(btn, q, function(){
felhomConfirm(btn, 'UTOLSÓ MEGERŐSÍTÉS: az alkalmazás leáll, az adatbázis visszatöltődik, majd újraindul. A jelenlegi adatbázisról biztonsági mentés készül.', function(){
var f = btn.closest('form');
if (f) { if (f.requestSubmit) f.requestSubmit(); else f.submit(); }
});
});
}
function fabStart(stack, next){
fetch('/api/export/download/start', {method:'POST', headers:Object.assign({'Content-Type':'application/json'}, csrfHeaders()), body: JSON.stringify({stack_name: stack, password: fabPassword()})})
.then(function(r){ return r.json(); })