v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (R-43 + R-44)
Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.
R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.
New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
- nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
created after the snapshot survives as an extra;
- the undo exists before the act — the pre-restore- dump is verified ON DISK
before anything is stopped, overwritten or replayed; if it cannot be taken
the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.
R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.
Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.
11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.
NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.
Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
This commit is contained in:
@@ -658,9 +658,40 @@ func (m *Manager) runOffboxBackup(ctx context.Context, withProgress bool) error
|
||||
m.offboxRecordStats(ctx, base, env) // the prune may have brought the size back down — refresh
|
||||
runErr = fmt.Errorf("A távoli mentés túllépte a tárhelykeretet (%d/%d GB) — törölj régi mentéseket vagy kérj nagyobb keretet.", usedGB, quota)
|
||||
} else {
|
||||
// R-43/R-44 (v0.148.0) — THE COHERENCE PRE-PHASE. Refresh the DB/volume dumps and the recovery
|
||||
// units BEFORE capturing, so the snapshot restic is about to write is an internally coherent
|
||||
// {DB@T, files@T} pair. Before this, a push shipped live files beside whatever dump the 02:30
|
||||
// local run happened to leave — on 2026-07-19 that was a dump taken four hours before the
|
||||
// customer's account even existed, so the "backup" of the photos contained zero of them
|
||||
// (DIAG-immich-restore-2026-07-19).
|
||||
//
|
||||
// Order matters and is the whole mechanism: dumps FIRST, then files. The gap between the two
|
||||
// can only ADD files the DB does not reference yet (an upload landing mid-run is a harmless
|
||||
// orphan blob), never remove one the DB DOES reference — so the file set is always a superset
|
||||
// of what the restored DB points at. The reverse order would produce dangling rows.
|
||||
//
|
||||
// This runs on the NIGHTLY path too, not just the manual one: "every snapshot is a coherent
|
||||
// pair" is the property that makes retention a history of restorable points rather than a
|
||||
// history of skewed ones. It also makes the nightly ordering structural instead of a
|
||||
// coincidence of two independent scheduler entries at 02:30 and 04:15.
|
||||
endStamp := m.beginOffsiteRunStamp(start.UTC().Format("20060102T150405Z"))
|
||||
if withProgress {
|
||||
m.offboxProgress.setPhase(OffboxPhaseDump)
|
||||
}
|
||||
dumpStart := time.Now()
|
||||
if dErr := m.offsitePreDump(ctx); dErr != nil {
|
||||
// Data-first: a dump failure must NOT abort the push. The files are still worth shipping,
|
||||
// and refusing to ship them would turn a degraded backup into no backup at all. It is a
|
||||
// loud WARN, and the unit manifest simply carries the older dump set — which the restore
|
||||
// confirm then surfaces as a skewed pair (P2) rather than silently pretending otherwise.
|
||||
m.logger.Printf("[WARN] [offbox] pre-push dump leg failed (%v) — continuing with the existing dumps; the snapshot's DB half may be older than its files", dErr)
|
||||
} else {
|
||||
m.logger.Printf("[INFO] [offbox] pre-push dump leg completed in %s — snapshot pair is coherent", time.Since(dumpStart).Round(time.Millisecond))
|
||||
}
|
||||
runResult, runErr = m.runOffboxInternal(ctx, apps, base, env, t)
|
||||
backedUp = runResult.backedUp
|
||||
missing = runResult.missing
|
||||
endStamp()
|
||||
}
|
||||
// Sorted names of apps whose enlargement was blocked this run (replaces the persisted set; empty clears).
|
||||
var blockedNames []string
|
||||
|
||||
Reference in New Issue
Block a user