v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (R-43 + R-44)

Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.

R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.

New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
  - nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
    created after the snapshot survives as an extra;
  - the undo exists before the act — the pre-restore- dump is verified ON DISK
    before anything is stopped, overwritten or replayed; if it cannot be taken
    the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.

R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.

Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.

11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.

NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.

Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
This commit is contained in:
2026-07-19 12:21:05 +02:00
parent 2fcae041ae
commit 062357f778
20 changed files with 1486 additions and 154 deletions
+74
View File
@@ -1,5 +1,79 @@
## Changelog
### v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (2026-07-19)
Closes **R-43** and **R-44**, the two findings from `DIAG-immich-restore-2026-07-19`. The short
version of that diagnosis: Viktor deleted 11 immich photos to test offsite restore, both restore
runs flashed success, and the photos stayed gone. Two independent defects, both fixed here.
**R-43 — no offsite path could restore a database.** All three offsite buttons were file-only.
The two „visszaállítás" actions staged into a scratch folder and never touched postgres; the
place-to-live action merged only files MISSING from the live tree and never replayed a dump. For a
DB-indexed app — most of the catalog — that combination cannot bring content back: the bytes
return and the app still cannot see them, because its index lives in the database. The dump was
faithfully carried INTO every snapshot and could never be replayed OUT of one.
New: **„Teljes visszaállítás (fájlok + adatbázis)"** (`ReconstituteFromOffsite`,
`/backup/offbox/reconstitute`). Safety dump → stop → files overwritten to the snapshot's version →
start → the snapshot's own dump replayed → health wait. Two invariants:
- **Nothing is ever deleted.** The full-restore copier is `rsync -a` with NO `--ignore-existing`
(a changed file becomes the snapshot's version) and NO `--delete` (a file created after the
snapshot survives as an extra). A restore that silently removed newer work would be a data-loss
event wearing a recovery button's label.
- **The undo exists before the act.** A `pre-restore-` dump of the live database is written and
verified on disk BEFORE anything is stopped, overwritten or replayed; if it cannot be taken the
whole operation refuses with zero changes. The safety dumps live in the app's own unit and
appear in `ListDumpFiles` — an undo the customer cannot see is not much of one.
The replay reads the SCRATCH unit, not the live one: the live recovery unit is still never
overwritten (it is the local restore path's source), so replaying from it would replay the current
database back over itself and restore nothing.
**R-44 — a manual push shipped an unrefreshed dump.** `RunOffboxBackup` went straight to the
restic push; dumps came only from the separate 02:30 local run, so a manual push at any other hour
shipped a dump up to ~24h old. On 2026-07-19 that dump was taken four hours before the customer's
account existed and probed to `asset: 0 / user: 0 / album: 0` — a 52MB file whose entire bulk was
immich's shipped geodata tables. Size and table count both called it healthy.
Every offsite run — **manual and nightly** — now refreshes the dumps and recovery units FIRST, then
captures. Order is the mechanism: the gap can only ADD files the DB does not reference yet (a
harmless orphan blob), never remove one it does, so the file set is always a superset of what the
restored DB points at. This also makes the nightly ordering structural instead of a coincidence of
two scheduler entries at 02:30 and 04:15. Each unit manifest carries the run's `offsite_run_id` +
`dumps_at`, so a snapshot's coherence is verifiable at restore time rather than assumed.
**Honesty surfaces** (warn-level, never gates — a false positive that blocked a restore would be
worse than the skew it guards against):
- A pre-v0.148 snapshot has no stamp → the confirm says „Az adatbázis-mentés régebbi (<ts>) — a
fájlok és az adatbázis eltérő időpontból származnak." It still restores.
- `ValidateDump` gained a content sniff: a structurally valid dump whose accounts table has zero
rows raises „A mentett adatbázis üresnek tűnik". Exact table-name matching, deliberately — a
substring match on "user" would flag `user_metadata` / `album_user` / `user_audit` on every
healthy single-user box and turn the signal into noise.
- The completion flash states an OUTCOME, not a mechanism: „A(z) X: N fájl és az adatbázis
visszaállítva (mentés: <ts>) — az alkalmazás újraindult." A no-database app says so explicitly
rather than borrowing the confident sentence.
- The old missing-only button now says what it does NOT do: „Adatbázist nem állít vissza — törölt
tartalom ettől nem jelenik meg újra."
A new `dump` progress phase („Adatbázisok mentése a pillanatképhez…") names the pre-phase, which on
a large database dominates the early wall clock and would otherwise read as a hang.
Tests: 11 new, with **5 red-proofs run and reverted** — replay removed (0 replayed), capture moved
before the dump (`[capture dump]`), both undo guards removed (no refusal), substring table matching
(join tables mistaken for accounts), buffer-exceeding rows uncounted (a wide row sniffed as empty).
Two of those red-proofs found real test weaknesses rather than confirming strength: the first undo
mutation was caught by a second guard, and the first table-matching test did not discriminate
between the two matchers at all — both tests were rewritten to the cases that actually separate
them.
**NOT in this slice:** the catalog-wide invariant check (stays on R-41), nightly cadence, retention,
quota math, tier-2, and the v0.147.x progress semantics beyond the one added phase line. The
missing-only place button's own zero-file flash is also untouched — that is the v0.147 feedback arc's
item, not R-43/R-44.
### docs — the workflow moved to DooPlex-local execution (2026-07-19)
**Docs only, no version bump, no code change.** Claude Code now runs on DooPlex