Files

T

admin 97249099d6 docs: DIAG round 2 — H1 confirmed, H4 found; two standing rulings landed

VERDICT. H1 confirmed: the reconstitution never ran. /backup/offbox/reconstitute
was never hit (reconstituted/safety-dump/replay line counts all 0); the operator
clicked the old missing-only button, which merged 34 files and left the DB
untouched. H2 rejected (the full path never executed; staging used the correct
snapshot 49e7cb46). H3 rejected (no replay happened; DB was genuinely asset:0).

R-44 is exonerated and working: 49e7cb46 carries offsite_run_id 20260719T102851Z
and its dump probes to asset:11 / user:1 — a properly coherent pair, against
round 1's asset:0 / user:0.

H4 (NEW, the more important result). Phase-3 recovery ran the real sequence and
the v0.148.0 path executed correctly — safety dump, stop, start, replay — then
the replay ABORTED: relation "clip_index" already exists. Proven to the second:
  10:58:25 controller begins replay
  10:58:33 immich-server "Reindexing clip_index" -> "Reindexed clip_index"
  10:58:35 dump's CREATE INDEX clip_index fails, exit 3
The replay races the app's own schema repair, because reconstitution starts the
stack before replaying (ImportDump needs a live container). The photos survived
only because pg_dump emits COPY data before CREATE INDEX — luck, not design. The
same start-then-replay shape is on the LOCAL path, so it is a class defect.

Recovery: PARTIAL and STOPPED as instructed. All 11 assets are back and visible
by every DB criterion (status=active, deletedAt null, all files resolve), but the
operation reported failure and immich reports schema drift. Not clean acceptance
evidence; no second attempt made; safety dump read only.

Findings -> R-47 (the replay race), R-48 (restore controls separable only by
layout — the proven cause of this incident), R-49 (1.1 GB of a 1.2 GB immich
backup is ML model cache + a postgres data-dir tar duplicating the .sql dump +
immich's own nightly dump; recorded, not changed).

RULINGS LANDED.
  - 00-capability-map.md:61 -> PARTIAL, scope-corrected: 6D's destruction hit the
    FILE TREE only; the DB survived in its named volume, so "end-to-end from
    offsite alone" overclaimed scope.
  - The 704.6 MiB "discrepancy" -> CLOSED, not a defect: it was immich's own
    Tarhely widget, never a controller page. Same for round 2's 650MiB->1.4GiB.
  - customer-restore row: partial evidence recorded, NOT flipped.

Diagnosis only — no code, label or layout changes; no forget/prune.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s

2026-07-19 13:04:58 +02:00

12 KiB

Raw Permalink Blame History

DIAGNOSE — immich offsite restore "succeeds", photos do not reappear (2026-07-19)

FIXED IN controller v0.148.0 (2026-07-19) — findings 1 and 2 below are closed in code as R-43 (offsite reconstitution: safety dump → stop → overwrite files → start → replay the snapshot's dump) and R-44 (every run dumps before it captures; manifests carry offsite_run_id + dumps_at). Findings 4 and 5 are described-and-accepted rather than changed; 3 stays with the v0.147 feedback arc and 6 is still unreconciled. The live acceptance has not run — the capability rows below are unchanged until it does. See felhom-controller/REPORT.md.

Class: diagnosis. No product code was changed by this document. Findings + evidence only. Box: demo-felhom (felhom-pve), guest 9201, controller 0.146.0, immich v3.0.3. Trigger: Viktor deliberately deleted photos in the immich UI to test the offsite restore feature. The photos were never wanted back — the exercise was the point.

TL;DR

The restore did not fail. It was never invoked on missing data, and even if it had been, it could not have brought the photos back.

Deleting in the immich UI moves assets to Lomtár (trash). No file leaves the disk. Both „csak a hiányzó fájlok" runs therefore merged 0 files — correctly, there was nothing missing. The success flash was truthful about mechanism and empty as a test result.
A valid test (trash emptied, files genuinely gone) would also have failed to restore the photos: no offsite button loads a database. The files would return; the library would still be empty, because immich's truth lives in postgres and nothing replays the dump.
The dump that ships is additionally stale by design — it comes from the 02:30 local run and is never refreshed before a manual push. Today's dump contains 0 users, 0 assets.

One-line answer to "can a customer trust same-day offsite?" — For a DB-indexed app: no. Files come back; content does not. The backup half is honest; the restore half cannot reconstitute the app.

Timeline (guest/DB clock; host is CEST, guest is UTC — CEST shown)

CEST	Event	Evidence
07-18 17:10	old pre-v3 user tree `dccc13fe…` last written	dir mtime
07-19 02:30	local `db-dump` run writes `immich-postgres.sql`	mtime `00:30:00 UTC`
07-19 07:56:25	admin user `4dcfb81e…` created (fresh v3 deploy)	`user.createdAt`
07-19 07:57:06–09	Viktor uploads 11 assets	`asset.createdAt`
07-19 07:58:15	manual offsite backup — captures the 11 files	`[offbox] backed up immich (…1 mandatory path(s))`
07-19 08:00:08 / 08:00:15	Viktor deletes assets in UI → trash, 2 batches	`asset.deletedAt`
07-19 08:01:14	full restore → scratch folder, not live	`[offbox] restored immich (360304f8, full=true) → …/backups/offsite-restore/immich`
07-19 08:27:54	place (missing-only) → 0 files merged	`[offbox] placed immich from offsite scratch: 0 file(s) merged (missing-only)`
07-19 08:29:09	full restore → scratch again	same as above
07-19 08:29:57	place (missing-only) → 0 files merged	same as above

The backup at 07:58:15 ran after the upload and before the delete — so the snapshot's file half is correct and current. Only the DB half is not.

Evidence

DB state (live, at 08:34 CEST)

total | trashed | live
   11 |      11 |    0        -- all 11 assets trashed, none live
users: 1                      -- 4dcfb81e-073a-49fb-938e-7fffb51dbac4 / admin@felhom.eu

updatedAt == deletedAt exactly on all 11 rows — nothing has modified these rows since the 08:00 deletion. Conclusive: none of today's restore operations touched the database.

All 11 originalPath values resolve to files that exist on disk (11 OK / 0 missing, after mapping the container path /usr/src/app/upload → /mnt/felhom-drives/hdd_1/appdata/immich). The photos are fully recoverable by an ordinary in-app un-trash; left in trash at Viktor's instruction.

The decisive dump probe

The recovery unit that actually ships: /mnt/felhom-drives/hdd_1/backups/primary/immich/db-dumps/immich-postgres.sql — 51 954 452 B, mtime 02:30 CEST.

asset: 0 rows
user:  0 rows
album: 0 rows

The 52 MB is entirely geodata_places / naturalearth_countries — immich's shipped reference data. A dump that looks substantial by size contains zero customer content. It was taken at 02:30, four hours before the admin user existed (07:56:25) and five and a half hours before the photos were uploaded (07:57).

The same is true of the unit's volume dumps: immich_immich_postgres_data.tar (323 088 384 B) is also from 02:30 CEST.

Method note / correction. The first probe in this session read appdata/immich/backups/immich-db-backup-20260719T020000-…sql.gz — that is immich's own self-backup, not the controller's. It is independently also empty (asset: 0, user: 0), but the figure that matters for restore is the controller unit dump above. Both were probed; both are empty; the conclusion is unchanged.

Safety consequence

Had a full restore actually loaded that dump into postgres, it would have written an empty database over the live one — destroying the trashed rows that are the only remaining recovery path. It did not, because (a) full restore only stages to a scratch folder and never touches a DB, and (b) the place path's live recovery unit present — not overwriting guard skipped the unit entirely. The photos survived by the combination of two behaviours, neither of which is a deliberate safety feature for this case.

Mechanism (source, controller 0.146.0)

Question	Answer	Cite
Does a manual offsite run refresh DB dumps before pushing?	No. `offboxRunHandler` → `RunOffboxBackup` directly; never calls `RunDBDumps` / `captureAllRecoveryUnits`. It ships whatever `.sql` already exists.	`web/offbox_handlers.go:203-227`, `backup/offbox.go:574-759`
Where do dumps come from, then?	The separate local `db-dump` daily at 02:30; offsite runs at 04:15. A manual offsite run at any hour ships dumps up to ~24 h old.	`cmd/controller/main.go:542,611`, `config/config.go:317`
What does the recovery unit contain?	compose/, app.yaml, manifest.json — and it enumerates existing dump filenames, never creates them (`listFileNames`).	`backup/recovery_unit.go:68-154` (105-106)
„csak a hiányzó fájlok" — what does it do?	`rsync -a --ignore-existing`. Files only. Never touches a DB. Never stops/starts the stack.	`backup/offbox_restore.go:346-421`, `tier2_restore.go:148-162`
„Teljes visszaállítás" — restores to where?	A scratch/verify folder (`backups/offsite-restore/<app>`), never live data, and never loads a dump into postgres.	`backup/offbox_restore.go:171-232`, `:130-164`
Does any offsite path replay a DB dump?	No. The only code that replays into a live DB is `RestoreFromRecoveryUnit`, reachable solely from the local restore panel.	`backup/restore_unit.go:74` ← `web/handlers.go:1108`
Is there any dump-freshness / RPO check?	None anywhere. Zero `RPO` hits. `offboxUnitTime` is only a two-drive tiebreak. The `DBValidationCache` exists but no offsite/restore path reads it.	`backup/offbox.go:827-837`, `backup.go:364-370`

The place-success message

"A(z) " + app + " hiányzó fájljai helyreállítva az élő adatok közé."   // offbox_handlers.go:354

Fixed string, built from the app name only. PlaceOffsiteRestore returns error alone, so the merged-file count is structurally unavailable to the flash — it exists only in the English log (offbox_restore.go:419). With 0 files merged the customer still reads an unconditional claim that files were restored.

Tier-2 already solves this: "Nincs hiányzó fájl — minden fájl megvan a helyén." (web/handlers.go:1154). The offsite path has no such branch — a straight parity gap.

Findings

1. [HIGH → R-43] No offsite restore path can restore a database

All three offsite buttons are file-only. The DB dump is faithfully carried into the snapshot and can never be replayed out of it by any offsite action. For every DB-indexed app — which is most of the catalog — offsite restore cannot reconstitute the application. The customer is left with correct bytes on disk and an app that cannot see them.

This outranks the staleness finding: even a perfectly fresh dump would not have restored the photos today.

2. [HIGH → R-44] Manual offsite push ships an unrefreshed dump

"Backed up now" is false for the DB half. Today: files from 07:57, dump from 02:30 — containing neither the user nor the assets. A same-day restore silently loses same-day state. No UI surface discloses the DB's actual age.

3. [MED] Success semantics are mechanism-shaped, not outcome-shaped

Every flash today told the truth about what the code did while the customer watched nothing happen. The zero-file case is the sharpest form: an unconditional "helyreállítva" when nothing was restored. Feeds the v0.147.0 feedback arc — restore completions should state the outcome („N fájl visszaállítva; az adatbázis NEM — az alkalmazás tartalma nem változott").

4. [MED, new] The "live recovery unit present" guard silently skips the dump

offbox_restore.go:400-405 skips the entire unit — compose, manifest and db-dumps — whenever a local backups/primary/<app> exists, which is true on any box whose local backup has run. So on a normal box the offsite dump is never even staged to disk. Correct as "don't clobber the local unit"; invisible as "your DB was not part of this restore".

5. [LOW, new] Orphaned pre-v3 user tree

upload|thumbs|encoded-video/dccc13fe-d629-4980-98e5-f2ea88cdb790 — 10 files / 36 MB + 3.9 MB + 15 MB, mtime 07-18 17:10, stranded by the immich 2→3 redeploy. The live DB has never known about it. Dead weight, not today's issue; worth a sweep policy for major redeploys.

6. [CLOSED — not a defect] Storage figure discrepancy

The brief cites 704.6 MiB used on the library storage. Measured: 126 MB total (upload 72 M, encoded-video 30 M, backups 18 M, thumbs 8 M).

RULED 2026-07-19 (Viktor): resolved, no controller defect. The 704.6 MiB came from immich's own Tárhely widget — immich-internal accounting — not from any controller Storage page. The two numbers measure different things and were never expected to agree. The same explanation covers the „650 MiB → 1.4 GiB" figure quoted during round 2, which likewise does not match the controller's measurement of the tree. Closed; no follow-up item.

7. [Method] A UI delete cannot test restore

Deleting in an app's own UI usually means trash, not deletion. Any "restore drill" scripted that way will merge 0 files, flash success, and prove nothing. A real drill must empty the trash (or otherwise remove the files) and verify the app's content afterwards, not just the file count. This belongs in R-3's "first restore by the customer" step.

Capability map

The customer-restore row STAYS MISSING and gains this diagnosis as a note. The C6 evidence attempt found a product gap — that is the rehearsal system working, not failing. R-3 stays DRAFT.

What was NOT done

No product fixes — no changes to restore/backup code, labels, or flashes.
No restic prune/forget, no snapshot deletion, no escrow or check-folder changes.
The 11 assets were left in trash (Viktor: recovery not wanted). immich's default 30-day auto-purge applies; they are recoverable by un-trash until then.
The genuine destructive test (empty trash → attempt real restore) was offered and not run — it needs explicit go-ahead, and the dump probe already settles the question without it.

12 KiB Raw Permalink Blame History Unescape Escape