felhom.eu/documentation/audits/DIAG-immich-restore-round2-2026-07-19.md

# DIAGNOSE — v0.148.0 full-restore acceptance: files back, timeline empty (round 2, 2026-07-19)

> **Class:** diagnosis. No product code changed. **Access mode: SSH** (the box was reachable from
> the workstation; every step below is direct evidence, none inferred).
> **Companion:** `DIAG-immich-restore-2026-07-19.md` (round 1, which produced R-43 + R-44).

---

## TL;DR

**H1 confirmed: the reconstitution never ran.** The operator clicked the old missing-only button;
`/backup/offbox/reconstitute` was **never hit** (`reconstituted lines: 0`, `safety dump lines: 0`,
`replay lines: 0`). Files came back (34 merged — hence the disk growth), the database did not.

**Then Phase-3 recovery found a NEW defect (H4), which is the more important result.** Running the
real sequence deliberately, the v0.148.0 path executed correctly — safety dump → stop → start →
replay — and the **replay aborted**: `ERROR: relation "clip_index" already exists`.

Root cause, proven to the second: **the replay races the application's own schema repair.** The
reconstitution starts the stack *before* replaying (ImportDump needs a live container), which gives
immich-server a window to recreate schema objects the dump is about to create.

```
10:58:25  controller: replaying DB dump into immich-postgres
10:58:33  immich-server: "Reindexing clip_index" → "Reindexed clip_index"   ← app recreates it
10:58:35  controller: ERROR relation "clip_index" already exists — exit status 3
```

**The photos ARE back** (11 assets, `status=active`, `deletedAt` null, all 11 files resolve, owned
by the live user) — because `pg_dump` emits COPY data *before* CREATE INDEX, so the abort landed
after the data. **That success is accidental.** A collision earlier in the script would abort before
the data and leave a genuinely half-restored database, reported identically.

**This is NOT clean §9 acceptance evidence.** The run reported failure, and immich now reports
schema drift (indexes the aborted script never created). Recovery is PARTIAL.

---

## Phase 1 — facts

### 1. Endpoint timeline (controller log, guest UTC)

| UTC | Event |
|---|---|
| 10:23:13 | controller 0.148.0 starts |
| 10:28:51 | manual offsite run starts — **v0.148 dump pre-phase runs** |
| 10:29:56 | `pre-push dump leg completed in 1m5.04s — snapshot pair is coherent` |
| 10:32:52 | `backed up immich (1 mandatory path(s))` |
| 10:33:41 | `backup OK: 3 app(s), 6 snapshot(s), 4m45s` |
| 10:38:32 | `restored immich (49e7cb46, full=true) → …/backups/offsite-restore/immich` (staging) |
| **10:39:55** | **`placed immich from offsite scratch: 34 file(s) merged (missing-only)`** |

Counted over the same window: `reconstituted: 0 · safety dump: 0 · replaying DB dump: 0 ·
missing-only: 1`. The reconstitute endpoint appears nowhere in the log. **H1.**

### 2. Which snapshot / which dump

Only two snapshots exist for immich:

- `6df12205` — 2026-07-18T17:13:46Z
- `49e7cb46` — 2026-07-19T10:30:01Z ← staged, correct

**No snapshot was pushed after the deletion**, so there is no post-delete empty-state snapshot to
restore by accident. `49e7cb46`'s unit carries `offsite_run_id: 20260719T102851Z` /
`dumps_at: 2026-07-19T10:28:51Z`, and its dump (`immich-postgres.sql`, mtime 10:28:52, 52 393 708 B)
probes to **`asset: 11` / `user: 1`**.

> **R-44 is working exactly as designed.** Round 1's dump was `asset: 0 / user: 0`. This one holds
> the customer's data and is stamped as coherent with the files beside it. The pair mechanism is
> not implicated in this failure.

The `pre-restore-` safety dump exists: `pre-restore-20260719T105811Z-immich-postgres.sql`,
51 964 807 B, 10:58:11 — written before anything was stopped or overwritten, as designed.

### 3. DB truth

Before Phase 3: `total 0 / trashed 0 / live 0`, 1 user — the trash was genuinely emptied, this was
real data loss (unlike round 1, where the rows survived as trashed).

After Phase 3: **`asset: 11, all status=active, deletedAt null, isOffline false, all 11 owned by
the live user`**; all 11 `originalPath` values resolve to files on disk. Every timeline-visibility
condition is satisfied.

### 4. File truth — what the 1.2 GB actually is

The operator's suspicion (immich's own DB-backup feature accumulating) is a **minor** contributor:
one 18 MB file. The dominant item is something else entirely.

| Item | Bytes | Share |
|---|---:|---:|
| `volume-dumps/immich_immich_ml_cache.tar` | 823 660 032 | **~60%** |
| `volume-dumps/immich_immich_postgres_data.tar` | 308 251 136 | ~23% |
| `db-dumps/immich-postgres.sql` | 52 393 708 | ~4% |
| `volume-dumps/immich_immich_redis_data.tar` | 6 358 016 | <1% |
| `appdata/immich` tree (below) | ~127 MB | ~10% |
| — of which `upload/` (originals, 2 user trees) | 72 MB | |
| — of which `encoded-video/` | 30 MB | |
| — of which `backups/` (immich's OWN nightly dump) | 18 MB | |
| — of which `thumbs/` | 8 MB | |

**The single biggest thing in the customer's offsite backup is immich's machine-learning model
cache — 786 MiB of re-downloadable model weights.** Second is a raw tar of the postgres data
directory, which duplicates the logical `.sql` dump captured beside it (both are in every
snapshot). Actual irreplaceable customer content — the originals — is 72 MB, and even that includes
the ~36 MB stranded pre-v3 user tree (`dccc13fe…`) from round 1.

So roughly **1.1 GB of a 1.2 GB "photo backup" is cache and duplication**, at the customer's offsite
quota and transfer cost.

### 5. Phase-3 run — the new defect

`pg_dump` is invoked `--clean --if-exists`, and `psql` with `ON_ERROR_STOP=1` (so an error aborts
rather than half-applying — correct). Plain-format pg_dump order is DROP → CREATE TABLE → COPY data
→ CREATE INDEX/constraints. The abort landed in the final stage:

- 10:58:25 replay begins; the DROP + CREATE TABLE + COPY phases succeed (11 assets land)
- 10:58:33 **immich-server** logs `Reindexing clip_index` → `Reindexed clip_index`
- 10:58:35 the dump's own `CREATE INDEX clip_index` fails: *already exists* → exit 3

immich-server then logs `Detected schema drift` and lists indexes that are missing and need to be
created (`user_updatedAt_id_idx`, `library_ownerId_idx`, `stack_primaryAssetId_idx`,
`asset_id_timeline_notDeleted_idx`, …) — the ones the aborted script never reached.

---

## Phase 2 — verdict

| Hypothesis | Verdict | Decisive evidence |
|---|---|---|
| **H1** — mangled row caused a mis-click; missing-only ran, replay never did | **CONFIRMED** | `placed immich … 34 file(s) merged (missing-only)` at 10:39:55; `reconstituted / safety dump / replaying DB dump` counts all **0**; the reconstitute endpoint appears nowhere in the log |
| **H2** — full restore ran but replayed a wrong/stale unit | **REJECTED** | The full path never executed at all. Independently, the staging used `49e7cb46` — the correct pre-delete snapshot, stamped coherent, dump holding 11 assets |
| **H3** — replay ran correctly, immich-side hides the assets | **REJECTED** | No replay occurred; the DB was genuinely `asset: 0`. After the Phase-3 replay the assets are visible-by-every-criterion, so nothing immich-side hides them |
| **H4 (new)** — the replay races the app's own schema repair | **CONFIRMED** | `clip_index` recreated by immich-server at 10:58:33, dump's CREATE INDEX fails at 10:58:35; resulting schema drift reported by immich itself |

---

## Phase 3 — recovery: PARTIAL, stopped as instructed

One deliberate run, via the exact endpoints the buttons post to (the endpoint route was used rather
than a click **because the mangled 4-button row makes the correct target genuinely ambiguous** —
finding 1 below):

1. `POST /backup/offbox/restore` `app=immich mode=full confirm=1` → staged `49e7cb46`, 1.3 G scratch.
2. `POST /backup/offbox/reconstitute` `app=immich confirm=1` → safety dump → stop → start → replay →
   **abort** (H4).

Result: **photos recovered and visible-by-every-DB-criterion; the operation reported failure; the
schema is incomplete.** Per the runbook — state is partially mutated, so **no second attempt was
made**. The safety dump from 10:58:11 is intact and untouched.

> **Screenshot still needed from Viktor** (cannot be captured from SSH): the immich timeline showing
> the 11 photos. Requested; not yet held.

---

## Phase 4 — findings

### 1. [HIGH] Restore UX — adjacent buttons whose difference is "data returns vs data cannot return"

Independent of the H-verdict, and the direct cause of H1:

- **The second step is hidden.** „Teljes visszaállítás indítása" only appears after
  „…előkészítése" has been pressed, with zero signposting that a second step exists or that the
  first one did nothing to the live data.
- **The row overflows.** Four buttons plus hint text overlap into an unreadable row, so the honest
  labels — the ones that distinguish a file-only merge from a real restore — are exactly what gets
  lost.
- **The proven consequence class:** *two adjacent controls whose difference is "your data comes
  back" versus "your data cannot come back" must not be separable only by layout.* This is now
  demonstrated, not theorised: a competent operator who had read the code pressed the wrong one.

Direction (already ruled in principle, spec rides v0.149): a single „Visszaállítás…" guided dialog —
one intent, visible phases, the wizard precedent.

### 2. [HIGH, new] The DB replay races the application's own schema repair

The reconstitution starts the stack before replaying, because `ImportDump` needs a live container.
That hands the application a window in which it recreates schema objects the dump is about to
create. Here it cost only the index phase; the data had already landed because pg_dump orders COPY
before CREATE INDEX. **That ordering is luck, not design** — a collision earlier in the script
aborts before the data and produces a half-restored database reported identically.

Note the same start-then-replay shape exists on the LOCAL restore path
(`RestoreFromRecoveryUnit` → `RecreateStackFromUnit` → `reimportDBDumpsCtx`), so this is a class
defect, not an offsite-only one. Direction for v0.149 (not decided here): bring up the DB container
alone for the replay and start the app only afterwards, or quiesce the app's schema management for
the duration.

### 3. [MED, new] A partially-successful restore reports as a flat failure

The customer was told „az adatbázis visszaállítása sikertelen" while their photos were, in fact,
restored. This is the mirror image of round 1's defect (success reported over a no-op) and the same
class: **the message describes the mechanism's exit status, not the outcome.** Worse here, because
the true state — data restored, schema incomplete — is neither "success" nor "failure" and the UI
has no way to say it.

### 4. [MED] Template classification — 1.1 GB of a 1.2 GB backup is cache and duplication

Candidates to exclude from the offsite capture set, **recorded not changed**:

- `immich_ml_cache` (786 MiB) — re-downloadable model weights, pure cache. Strong candidate.
- `immich_postgres_data` tar (294 MiB) — duplicates the logical `.sql` dump captured beside it.
  Keeping both doubles the DB's footprint in every snapshot; the dump is already the authoritative
  copy on the restore path.
- `upload/backups/` (18 MiB, immich's own nightly dump) — a backup inside a backup; grows daily.
- The stranded pre-v3 `dccc13fe…` tree (~36 MiB) from round 1, still unreferenced by any DB.

### 5. Standing rulings, carried into the repo

- **`00-capability-map.md:61`** — ruled: CAMPAIGN-6D's destruction hit the **file tree only**; the
  database survived in its named volume (`immich_postgres_data` is a named volume in both eras), so
  "immich end-to-end from offsite alone" **overclaimed scope**. Row → PARTIAL, scope-corrected.
- **The 704.6 MiB "discrepancy"** — ruled **resolved, not a defect**: the figure was immich's own
  Tárhely widget (immich-internal accounting), never a controller page. Closed in the round-1 DIAG.
  The same explanation covers today's „650 MiB → 1.4 GiB" observation, which likewise does not match
  the controller's own measurement of the tree (127 MB).

### 6. Capability map — customer-restore row

**NOT flipped.** The Phase-3 run recovered the photos but reported failure and left schema drift, so
it is not the clean acceptance evidence §9 asks for. Recorded as
*"partial evidence captured 2026-07-19 — blocked on H4"* rather than flipped.

---

## What was NOT done

- No code, label or layout fixes — all of the above rides v0.149.
- No `restic forget`/`prune`, no snapshot deletion.
- The safety dumps were read, never modified or removed.
- **No second restore attempt** after the partially-mutating first run.