v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (R-43 + R-44)
Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.
R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.
New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
- nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
created after the snapshot survives as an extra;
- the undo exists before the act — the pre-restore- dump is verified ON DISK
before anything is stopped, overwritten or replayed; if it cannot be taken
the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.
R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.
Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.
11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.
NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.
Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
This commit is contained in:
@@ -44,6 +44,15 @@ type RecoveryManifest struct {
|
||||
DBDumps []string `json:"db_dumps"`
|
||||
VolumeDumps []string `json:"volume_dumps"`
|
||||
Checksums map[string]string `json:"checksums"` // sha256 of captured compose/ files
|
||||
// R-43/R-44 (v0.148.0): the coherence stamp. An offsite run refreshes the dumps FIRST and then
|
||||
// captures the unit, so a manifest carrying an OffsiteRunID asserts "the db-dumps/ in this unit
|
||||
// were taken by that run" — i.e. the snapshot is an internally coherent {DB@T, files@T} pair.
|
||||
// A manifest WITHOUT these fields is a pre-v0.148 unit whose dump age is unknown and may skew
|
||||
// arbitrarily from the files beside it (the DIAG-immich-restore-2026-07-19 failure); the restore
|
||||
// confirm surfaces that honestly rather than blocking. Empty on the periodic refresh, which must
|
||||
// never claim a coherence it did not establish — it carries the prior stamp forward instead.
|
||||
OffsiteRunID string `json:"offsite_run_id,omitempty"`
|
||||
DumpsAt string `json:"dumps_at,omitempty"` // RFC3339 UTC — when this run's dump leg finished
|
||||
}
|
||||
|
||||
// SetVersion records the controller version stamped into recovery-unit manifests.
|
||||
@@ -107,13 +116,26 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
|
||||
version := m.versionLocked()
|
||||
|
||||
manifestPath := RecoveryUnitManifestPath(nsRoot, stackName)
|
||||
cur := readManifest(manifestPath)
|
||||
|
||||
// R-43/R-44: the coherence stamp of the offsite run currently in flight ("" on the periodic
|
||||
// refresh and on the local dump run). When empty we CARRY THE PRIOR STAMP FORWARD rather than
|
||||
// blanking it — a periodic refresh must neither claim a coherence it did not establish nor
|
||||
// destroy the record of one that a real run did.
|
||||
runID, dumpsAt := m.offsiteRunStamp()
|
||||
if runID == "" && cur != nil {
|
||||
runID, dumpsAt = cur.OffsiteRunID, cur.DumpsAt
|
||||
}
|
||||
|
||||
// Skip if the unit is already current — avoids needless drive writes on the periodic refresh.
|
||||
if cur := readManifest(manifestPath); cur != nil &&
|
||||
// The run-id is part of "current": an offsite run must re-stamp the manifest even when nothing
|
||||
// else changed, because the stamp is exactly the claim the restore path reads.
|
||||
if cur != nil &&
|
||||
cur.ControllerVer == version &&
|
||||
stringMapEqual(cur.Checksums, checksums) &&
|
||||
stringSliceEqual(cur.DBDumps, dbDumps) &&
|
||||
stringSliceEqual(cur.VolumeDumps, volDumps) {
|
||||
stringSliceEqual(cur.VolumeDumps, volDumps) &&
|
||||
cur.OffsiteRunID == runID {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -143,6 +165,8 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
|
||||
DBDumps: dbDumps,
|
||||
VolumeDumps: volDumps,
|
||||
Checksums: checksums,
|
||||
OffsiteRunID: runID,
|
||||
DumpsAt: dumpsAt,
|
||||
}
|
||||
if err := writeManifest(manifestPath, manifest); err != nil {
|
||||
return fmt.Errorf("writing manifest: %w", err)
|
||||
|
||||
Reference in New Issue
Block a user