v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (R-43 + R-44)

Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.

R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.

New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
  - nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
    created after the snapshot survives as an extra;
  - the undo exists before the act — the pre-restore- dump is verified ON DISK
    before anything is stopped, overwritten or replayed; if it cannot be taken
    the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.

R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.

Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.

11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.

NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.

Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
This commit is contained in:
2026-07-19 12:21:05 +02:00
parent 2fcae041ae
commit 062357f778
20 changed files with 1486 additions and 154 deletions
+31
View File
@@ -658,9 +658,40 @@ func (m *Manager) runOffboxBackup(ctx context.Context, withProgress bool) error
m.offboxRecordStats(ctx, base, env) // the prune may have brought the size back down — refresh
runErr = fmt.Errorf("A távoli mentés túllépte a tárhelykeretet (%d/%d GB) — törölj régi mentéseket vagy kérj nagyobb keretet.", usedGB, quota)
} else {
// R-43/R-44 (v0.148.0) — THE COHERENCE PRE-PHASE. Refresh the DB/volume dumps and the recovery
// units BEFORE capturing, so the snapshot restic is about to write is an internally coherent
// {DB@T, files@T} pair. Before this, a push shipped live files beside whatever dump the 02:30
// local run happened to leave — on 2026-07-19 that was a dump taken four hours before the
// customer's account even existed, so the "backup" of the photos contained zero of them
// (DIAG-immich-restore-2026-07-19).
//
// Order matters and is the whole mechanism: dumps FIRST, then files. The gap between the two
// can only ADD files the DB does not reference yet (an upload landing mid-run is a harmless
// orphan blob), never remove one the DB DOES reference — so the file set is always a superset
// of what the restored DB points at. The reverse order would produce dangling rows.
//
// This runs on the NIGHTLY path too, not just the manual one: "every snapshot is a coherent
// pair" is the property that makes retention a history of restorable points rather than a
// history of skewed ones. It also makes the nightly ordering structural instead of a
// coincidence of two independent scheduler entries at 02:30 and 04:15.
endStamp := m.beginOffsiteRunStamp(start.UTC().Format("20060102T150405Z"))
if withProgress {
m.offboxProgress.setPhase(OffboxPhaseDump)
}
dumpStart := time.Now()
if dErr := m.offsitePreDump(ctx); dErr != nil {
// Data-first: a dump failure must NOT abort the push. The files are still worth shipping,
// and refusing to ship them would turn a degraded backup into no backup at all. It is a
// loud WARN, and the unit manifest simply carries the older dump set — which the restore
// confirm then surfaces as a skewed pair (P2) rather than silently pretending otherwise.
m.logger.Printf("[WARN] [offbox] pre-push dump leg failed (%v) — continuing with the existing dumps; the snapshot's DB half may be older than its files", dErr)
} else {
m.logger.Printf("[INFO] [offbox] pre-push dump leg completed in %s — snapshot pair is coherent", time.Since(dumpStart).Round(time.Millisecond))
}
runResult, runErr = m.runOffboxInternal(ctx, apps, base, env, t)
backedUp = runResult.backedUp
missing = runResult.missing
endStamp()
}
// Sorted names of apps whose enlargement was blocked this run (replaces the persisted set; empty clears).
var blockedNames []string