v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (R-43 + R-44)

Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.

R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.

New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
  - nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
    created after the snapshot survives as an extra;
  - the undo exists before the act — the pre-restore- dump is verified ON DISK
    before anything is stopped, overwritten or replayed; if it cannot be taken
    the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.

R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.

Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.

11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.

NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.

Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
This commit is contained in:
2026-07-19 12:21:05 +02:00
parent 2fcae041ae
commit 062357f778
20 changed files with 1486 additions and 154 deletions
+95 -1
View File
@@ -51,6 +51,20 @@ type DumpValidation struct {
Error string
FileSize int64
ModTime time.Time
// R-44 (v0.148.0) content sniff — a WARN-LEVEL signal, never a gate.
//
// Structural validity says nothing about whether a dump holds the customer's data. The immich
// dump of 2026-07-19 was 52MB, had a valid header and 60+ CREATE TABLEs, and contained zero
// users and zero assets: its whole bulk was the geodata reference tables immich ships. Size and
// table count are therefore both useless as emptiness heuristics — but an accounts table with
// no rows is a strong, cheap, app-agnostic hint that a dump predates the customer entirely.
//
// Deliberately NOT a refusal: plenty of legitimate apps have no users table (UserTableFound
// false → inconclusive → silent), and a false positive that blocked a restore would be far
// worse than the skew it guards against. The restore confirm shows it as one extra line.
UserTableFound bool
UserRows int
LooksEmpty bool // UserTableFound && UserRows == 0
}
// DumpFileInfo holds info about a dump file on disk.
@@ -363,6 +377,9 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation {
lineNum := 0
headerFound := false
tableCount := 0
// R-44 sniff state. inUserCopy tracks a postgres `COPY … FROM stdin;` block for an accounts
// table; rows are counted until the `\.` terminator.
inUserCopy := false
for {
lineBytes, isPrefix, err := reader.ReadLine()
if err != nil {
@@ -374,7 +391,12 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation {
break // EOF
}
if isPrefix {
// Line exceeds buffer — skip remainder (COPY data, large INSERTs)
// Line exceeds buffer — skip remainder (COPY data, large INSERTs).
// A long line inside a user COPY block is still a ROW: count it before discarding it,
// or a table whose rows happen to be wide would sniff as empty and raise a false alarm.
if inUserCopy {
v.UserRows++
}
for isPrefix && err == nil {
_, isPrefix, err = reader.ReadLine()
}
@@ -384,6 +406,23 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation {
line := string(lineBytes)
lineNum++
// R-44 content sniff (warn-level; see DumpValidation).
if inUserCopy {
if line == `\.` {
inUserCopy = false
} else {
v.UserRows++
}
} else if isUserCopyStart(line, dbType) {
inUserCopy = true
v.UserTableFound = true
} else if dbType == DBTypeMariaDB && isUserInsert(line) {
// mysqldump writes multi-row `INSERT INTO \`users\` VALUES (…),(…);` — the row count is
// not worth parsing out of it, and presence alone answers the only question asked here.
v.UserTableFound = true
v.UserRows++
}
// Header check — scan first 10 lines for expected dump header
// MariaDB 11.4+ prepends a sandbox comment before the header line
if lineNum <= 10 && !headerFound {
@@ -427,10 +466,65 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation {
return v
}
v.LooksEmpty = v.UserTableFound && v.UserRows == 0
if v.LooksEmpty {
log.Printf("[WARN] [backup] ValidateDump: %s is structurally valid (%d tables) but its accounts table has NO rows — the dump may predate the customer's data", filePath, tableCount)
}
v.Valid = true
return v
}
// userTableNames are the table names treated as "the accounts table" by the R-44 sniff. Kept
// deliberately short: a wider net (anything containing "user") would match join/audit tables like
// `user_metadata` or `album_user`, which are legitimately empty on a healthy single-user install
// and would produce exactly the false alarm this signal must not raise.
var userTableNames = []string{"user", "users", "account", "accounts"}
// isUserCopyStart reports whether a line opens a postgres `COPY <accounts-table> … FROM stdin;`
// block. pg_dump writes the table qualified and optionally quoted — `COPY public."user" (…)`,
// `COPY public.users (…)` — so both forms are matched.
func isUserCopyStart(line string, dbType DBType) bool {
if dbType != DBTypePostgres || !strings.HasPrefix(line, "COPY ") {
return false
}
rest := strings.TrimPrefix(line, "COPY ")
sp := strings.IndexByte(rest, ' ')
if sp < 0 {
return false
}
return matchesUserTable(rest[:sp])
}
// isUserInsert reports whether a line is a mysqldump INSERT into an accounts table.
func isUserInsert(line string) bool {
const pfx = "INSERT INTO "
if !strings.HasPrefix(line, pfx) {
return false
}
rest := strings.TrimPrefix(line, pfx)
sp := strings.IndexByte(rest, ' ')
if sp < 0 {
return false
}
return matchesUserTable(rest[:sp])
}
// matchesUserTable strips schema qualification and quoting from a dumped table reference and
// reports whether the bare name is an accounts table.
func matchesUserTable(ref string) bool {
if dot := strings.LastIndexByte(ref, '.'); dot >= 0 {
ref = ref[dot+1:]
}
ref = strings.Trim(ref, "\"`")
for _, n := range userTableNames {
if strings.EqualFold(ref, n) {
return true
}
}
return false
}
// ListDumpFiles returns info about SQL dump files on disk.
//
// M18: ValidateDump scans the dump line-by-line; on a customer with hundreds-of-MB dumps that is wasted