Files

T

admin 062357f778 v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (R-43 + R-44)

Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.

R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.

New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
  - nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
    created after the snapshot survives as an extra;
  - the undo exists before the act — the pre-restore- dump is verified ON DISK
    before anything is stopped, overwritten or replayed; if it cannot be taken
    the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.

R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.

Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.

11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.

NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.

Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s

2026-07-19 12:21:16 +02:00

4.7 KiB

Raw Blame History

REPORT — v0.148.0: coherent snapshot pairs + a real offsite restore (R-43 + R-44)

Date: 2026-07-19 · Repo: felhom-controller (v0.147.3 → v0.148.0) · Trunk, pushed to main.

Closes the two findings from felhom.eu/documentation/audits/DIAG-immich-restore-2026-07-19.md.

What was broken

Viktor deleted 11 immich photos to test offsite restore. Both restore runs flashed success; the photos stayed gone. Two independent defects:

R-43 — no offsite path could restore a database. All three buttons were file-only, so for a DB-indexed app (most of the catalog) the bytes came back and the app still could not see them.
R-44 — a manual push shipped whatever dump the 02:30 local run had left. That day's dump predated the customer's account by four hours and contained asset: 0 / user: 0 / album: 0, in a 52 MB file whose bulk was immich's shipped geodata tables.

What shipped

P1 — coherence pre-phase. Every offsite run (manual AND nightly) refreshes dumps + recovery units before capturing. Each manifest carries offsite_run_id + dumps_at. The periodic refresh carries a prior stamp forward and never invents one. A dump failure is a loud WARN that does not abort the push (data-first). New dump progress phase.

P2 — honesty surfaces. ValidateDump gained an exact-match accounts-table sniff (LooksEmpty); the confirm shows dump age, an unstamped-pair warning and the empty-dump warning. All warn-level, none a gate.

P3 — ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop → files overwritten → start → snapshot's dump replayed → health wait. Outcome-shaped flash. Two invariants: nothing is ever deleted (-a, no --ignore-existing, no --delete), and the pre-restore- undo is verified on disk before anything is touched — otherwise refuse with zero changes.

P4 — tests. 11 new; full suite green: go build ./... && go vet ./... && go test ./.... Design gates pass: template_id, emoji, native_confirm, offbox_rename, app_row_dedup, mojibake.

docker_run_volume_path_gate still fails on internal/appexport/estimate.go:179. Pre-existing on HEAD and untouched by this release — same finding as the v0.147.x report, same file, not a path this work goes near.

Red-proofs — 5 run and reverted

Mutation	Result
Replay removed	`0 DB dump(s) replayed` → FAIL
Capture moved before the dump	`got [capture dump]` → FAIL
Both undo guards removed	no refusal → FAIL
Substring table matching	join table mistaken for accounts; `UserRows = 4, want 1` → FAIL
Buffer-exceeding rows uncounted	wide row sniffs as empty → FAIL

Two of these found real test weaknesses rather than confirming strength, and both tests were rewritten:

The first undo mutation still refused — a second guard (os.Stat on the empty path) held. The red-proof only goes red when BOTH guards are removed, which is now recorded in REUSE.md.
The first table-matching test did not discriminate between the two matchers at all: its aggregate row count was identical either way. Rewritten to the case that separates them — an app with no accounts table but with user_metadata / album_user / user_audit, where a substring match would shout "your backup looks empty" on every restore of a healthy app.

Deliberately NOT done

Per §10 of the task: no catalog-wide invariant check (stays R-41), no changes to nightly cadence, retention, quota math or tier-2, and no touching v0.147.x progress semantics beyond the one added phase line. The missing-only button's own zero-file flash is untouched — that belongs to the v0.147 feedback arc, not R-43/R-44. Its label now states what it does not do, which is in scope.

Live acceptance — NOT YET DONE (§9, needs Viktor)

The genuine red-proof is the joint run over the tunnel and has not happened: upload → „Távoli mentés most" → empty the trash for real → „Teljes visszaállítás (fájlok + adatbázis)" → photos visible in the immich timeline. Until that evidence exists:

the capability-map offsite row stays PARTIAL / scope-contested (no PROVEN-LIVE flip),
the customer-restore row stays MISSING,
R-3 stays DRAFT.

Floor raise to v0.148.0 is Viktor's click and must come BEFORE the acceptance run.

Open question for Viktor

The capability-map row at 00-capability-map.md:61 claims PROVEN-LIVE citing CAMPAIGN-6D's "immich end-to-end from offsite alone". This diagnosis contradicts that phrase. I flagged it scope-contested rather than downgrading a proven row — the ruling (did 6D exercise the DB half, or only the file half?) is still outstanding and is not something this implementation can settle.

4.7 KiB Raw Blame History