felhom.eu/documentation/audits/DIAG-immich-restore-round2-2026-07-19.md

DIAGNOSE — v0.148.0 full-restore acceptance: files back, timeline empty (round 2, 2026-07-19)

Class: diagnosis. No product code changed. Access mode: SSH (the box was reachable from the workstation; every step below is direct evidence, none inferred). Companion: DIAG-immich-restore-2026-07-19.md (round 1, which produced R-43 + R-44).

---

TL;DR

H1 confirmed: the reconstitution never ran. The operator clicked the old missing-only button; /backup/offbox/reconstitute was never hit (reconstituted lines: 0, safety dump lines: 0, replay lines: 0). Files came back (34 merged — hence the disk growth), the database did not.

Then Phase-3 recovery found a NEW defect (H4), which is the more important result. Running the real sequence deliberately, the v0.148.0 path executed correctly — safety dump → stop → start → replay — and the replay aborted: ERROR: relation "clip_index" already exists.

Root cause, proven to the second: the replay races the application's own schema repair. The reconstitution starts the stack before replaying (ImportDump needs a live container), which gives immich-server a window to recreate schema objects the dump is about to create.

10:58:25  controller: replaying DB dump into immich-postgres
10:58:33  immich-server: "Reindexing clip_index" → "Reindexed clip_index"   ← app recreates it
10:58:35  controller: ERROR relation "clip_index" already exists — exit status 3

The photos ARE back (11 assets, status=active, deletedAt null, all 11 files resolve, owned by the live user) — because pg_dump emits COPY data before CREATE INDEX, so the abort landed after the data. That success is accidental. A collision earlier in the script would abort before the data and leave a genuinely half-restored database, reported identically.

This is NOT clean §9 acceptance evidence. The run reported failure, and immich now reports schema drift (indexes the aborted script never created). Recovery is PARTIAL.

---

Phase 1 — facts

1. Endpoint timeline (controller log, guest UTC)

UTC	Event
10:23:13	controller 0.148.0 starts
10:28:51	manual offsite run starts — v0.148 dump pre-phase runs
10:29:56	pre-push dump leg completed in 1m5.04s — snapshot pair is coherent
10:32:52	backed up immich (1 mandatory path(s))
10:33:41	backup OK: 3 app(s), 6 snapshot(s), 4m45s
10:38:32	restored immich (49e7cb46, full=true) → …/backups/offsite-restore/immich (staging)
10:39:55	placed immich from offsite scratch: 34 file(s) merged (missing-only)

Counted over the same window: reconstituted: 0 · safety dump: 0 · replaying DB dump: 0 · missing-only: 1. The reconstitute endpoint appears nowhere in the log. H1.

2. Which snapshot / which dump

Only two snapshots exist for immich:

• 6df12205 — 2026-07-18T17:13:46Z
• 49e7cb46 — 2026-07-19T10:30:01Z ← staged, correct

No snapshot was pushed after the deletion, so there is no post-delete empty-state snapshot to restore by accident. 49e7cb46's unit carries offsite_run_id: 20260719T102851Z / dumps_at: 2026-07-19T10:28:51Z, and its dump (immich-postgres.sql, mtime 10:28:52, 52 393 708 B) probes to asset: 11 / user: 1.

R-44 is working exactly as designed. Round 1's dump was asset: 0 / user: 0. This one holds the customer's data and is stamped as coherent with the files beside it. The pair mechanism is not implicated in this failure.

The pre-restore- safety dump exists: pre-restore-20260719T105811Z-immich-postgres.sql, 51 964 807 B, 10:58:11 — written before anything was stopped or overwritten, as designed.

3. DB truth

Before Phase 3: total 0 / trashed 0 / live 0, 1 user — the trash was genuinely emptied, this was real data loss (unlike round 1, where the rows survived as trashed).

After Phase 3: asset: 11, all status=active, deletedAt null, isOffline false, all 11 owned by the live user; all 11 originalPath values resolve to files on disk. Every timeline-visibility condition is satisfied.

4. File truth — what the 1.2 GB actually is

The operator's suspicion (immich's own DB-backup feature accumulating) is a minor contributor: one 18 MB file. The dominant item is something else entirely.

Item	Bytes	Share
volume-dumps/immich_immich_ml_cache.tar	823 660 032	~60%
volume-dumps/immich_immich_postgres_data.tar	308 251 136	~23%
db-dumps/immich-postgres.sql	52 393 708	~4%
volume-dumps/immich_immich_redis_data.tar	6 358 016	<1%
appdata/immich tree (below)	~127 MB	~10%
— of which upload/ (originals, 2 user trees)	72 MB
— of which encoded-video/	30 MB
— of which backups/ (immich's OWN nightly dump)	18 MB
— of which thumbs/	8 MB

The single biggest thing in the customer's offsite backup is immich's machine-learning model cache — 786 MiB of re-downloadable model weights. Second is a raw tar of the postgres data directory, which duplicates the logical .sql dump captured beside it (both are in every snapshot). Actual irreplaceable customer content — the originals — is 72 MB, and even that includes the ~36 MB stranded pre-v3 user tree (dccc13fe…) from round 1.

So roughly 1.1 GB of a 1.2 GB "photo backup" is cache and duplication, at the customer's offsite quota and transfer cost.

5. Phase-3 run — the new defect

pg_dump is invoked --clean --if-exists, and psql with ON_ERROR_STOP=1 (so an error aborts rather than half-applying — correct). Plain-format pg_dump order is DROP → CREATE TABLE → COPY data → CREATE INDEX/constraints. The abort landed in the final stage:

• 10:58:25 replay begins; the DROP + CREATE TABLE + COPY phases succeed (11 assets land)
• 10:58:33 immich-server logs Reindexing clip_index → Reindexed clip_index
• 10:58:35 the dump's own CREATE INDEX clip_index fails: already exists → exit 3

immich-server then logs Detected schema drift and lists indexes that are missing and need to be created (user_updatedAt_id_idx, library_ownerId_idx, stack_primaryAssetId_idx, asset_id_timeline_notDeleted_idx, …) — the ones the aborted script never reached.

---

Phase 2 — verdict

Hypothesis	Verdict	Decisive evidence
H1 — mangled row caused a mis-click; missing-only ran, replay never did	CONFIRMED	placed immich … 34 file(s) merged (missing-only) at 10:39:55; reconstituted / safety dump / replaying DB dump counts all 0; the reconstitute endpoint appears nowhere in the log
H2 — full restore ran but replayed a wrong/stale unit	REJECTED	The full path never executed at all. Independently, the staging used 49e7cb46 — the correct pre-delete snapshot, stamped coherent, dump holding 11 assets
H3 — replay ran correctly, immich-side hides the assets	REJECTED	No replay occurred; the DB was genuinely asset: 0. After the Phase-3 replay the assets are visible-by-every-criterion, so nothing immich-side hides them
H4 (new) — the replay races the app's own schema repair	CONFIRMED	clip_index recreated by immich-server at 10:58:33, dump's CREATE INDEX fails at 10:58:35; resulting schema drift reported by immich itself

---

Phase 3 — recovery: PARTIAL, stopped as instructed

One deliberate run, via the exact endpoints the buttons post to (the endpoint route was used rather than a click because the mangled 4-button row makes the correct target genuinely ambiguous — finding 1 below):

1. POST /backup/offbox/restore app=immich mode=full confirm=1 → staged 49e7cb46, 1.3 G scratch.
2. POST /backup/offbox/reconstitute app=immich confirm=1 → safety dump → stop → start → replay → abort (H4).

Result: photos recovered and visible-by-every-DB-criterion; the operation reported failure; the schema is incomplete. Per the runbook — state is partially mutated, so no second attempt was made. The safety dump from 10:58:11 is intact and untouched.

Screenshot still needed from Viktor (cannot be captured from SSH): the immich timeline showing the 11 photos. Requested; not yet held.

---

Phase 4 — findings

1. [HIGH] Restore UX — adjacent buttons whose difference is "data returns vs data cannot return"

Independent of the H-verdict, and the direct cause of H1:

• The second step is hidden. „Teljes visszaállítás indítása" only appears after „…előkészítése" has been pressed, with zero signposting that a second step exists or that the first one did nothing to the live data.
• The row overflows. Four buttons plus hint text overlap into an unreadable row, so the honest labels — the ones that distinguish a file-only merge from a real restore — are exactly what gets lost.
• The proven consequence class: two adjacent controls whose difference is "your data comes back" versus "your data cannot come back" must not be separable only by layout. This is now demonstrated, not theorised: a competent operator who had read the code pressed the wrong one.

Direction (already ruled in principle, spec rides v0.149): a single „Visszaállítás…" guided dialog — one intent, visible phases, the wizard precedent.

2. [HIGH, new] The DB replay races the application's own schema repair

The reconstitution starts the stack before replaying, because ImportDump needs a live container. That hands the application a window in which it recreates schema objects the dump is about to create. Here it cost only the index phase; the data had already landed because pg_dump orders COPY before CREATE INDEX. That ordering is luck, not design — a collision earlier in the script aborts before the data and produces a half-restored database reported identically.

Note the same start-then-replay shape exists on the LOCAL restore path (RestoreFromRecoveryUnit → RecreateStackFromUnit → reimportDBDumpsCtx), so this is a class defect, not an offsite-only one. Direction for v0.149 (not decided here): bring up the DB container alone for the replay and start the app only afterwards, or quiesce the app's schema management for the duration.

3. [MED, new] A partially-successful restore reports as a flat failure

The customer was told „az adatbázis visszaállítása sikertelen" while their photos were, in fact, restored. This is the mirror image of round 1's defect (success reported over a no-op) and the same class: the message describes the mechanism's exit status, not the outcome. Worse here, because the true state — data restored, schema incomplete — is neither "success" nor "failure" and the UI has no way to say it.

4. [MED] Template classification — 1.1 GB of a 1.2 GB backup is cache and duplication

Candidates to exclude from the offsite capture set, recorded not changed:

• immich_ml_cache (786 MiB) — re-downloadable model weights, pure cache. Strong candidate.
• immich_postgres_data tar (294 MiB) — duplicates the logical .sql dump captured beside it. Keeping both doubles the DB's footprint in every snapshot; the dump is already the authoritative copy on the restore path.
• upload/backups/ (18 MiB, immich's own nightly dump) — a backup inside a backup; grows daily.
• The stranded pre-v3 dccc13fe… tree (~36 MiB) from round 1, still unreferenced by any DB.

5. Standing rulings, carried into the repo

• 00-capability-map.md:61 — ruled: CAMPAIGN-6D's destruction hit the file tree only; the database survived in its named volume (immich_postgres_data is a named volume in both eras), so "immich end-to-end from offsite alone" overclaimed scope. Row → PARTIAL, scope-corrected.
• The 704.6 MiB "discrepancy" — ruled resolved, not a defect: the figure was immich's own Tárhely widget (immich-internal accounting), never a controller page. Closed in the round-1 DIAG. The same explanation covers today's „650 MiB → 1.4 GiB" observation, which likewise does not match the controller's own measurement of the tree (127 MB).

6. Capability map — customer-restore row

NOT flipped. The Phase-3 run recovered the photos but reported failure and left schema drift, so it is not the clean acceptance evidence §9 asks for. Recorded as "partial evidence captured 2026-07-19 — blocked on H4" rather than flipped.

---

What was NOT done

• No code, label or layout fixes — all of the above rides v0.149.
• No restic forget/prune, no snapshot deletion.
• The safety dumps were read, never modified or removed.
• No second restore attempt after the partially-mutating first run.