admin
97249099d6
VERDICT. H1 confirmed: the reconstitution never ran. /backup/offbox/reconstitute
was never hit (reconstituted/safety-dump/replay line counts all 0); the operator
clicked the old missing-only button, which merged 34 files and left the DB
untouched. H2 rejected (the full path never executed; staging used the correct
snapshot 49e7cb46). H3 rejected (no replay happened; DB was genuinely asset:0).
R-44 is exonerated and working: 49e7cb46 carries offsite_run_id 20260719T102851Z
and its dump probes to asset:11 / user:1 — a properly coherent pair, against
round 1's asset:0 / user:0.
H4 (NEW, the more important result). Phase-3 recovery ran the real sequence and
the v0.148.0 path executed correctly — safety dump, stop, start, replay — then
the replay ABORTED: relation "clip_index" already exists. Proven to the second:
10:58:25 controller begins replay
10:58:33 immich-server "Reindexing clip_index" -> "Reindexed clip_index"
10:58:35 dump's CREATE INDEX clip_index fails, exit 3
The replay races the app's own schema repair, because reconstitution starts the
stack before replaying (ImportDump needs a live container). The photos survived
only because pg_dump emits COPY data before CREATE INDEX — luck, not design. The
same start-then-replay shape is on the LOCAL path, so it is a class defect.
Recovery: PARTIAL and STOPPED as instructed. All 11 assets are back and visible
by every DB criterion (status=active, deletedAt null, all files resolve), but the
operation reported failure and immich reports schema drift. Not clean acceptance
evidence; no second attempt made; safety dump read only.
Findings -> R-47 (the replay race), R-48 (restore controls separable only by
layout — the proven cause of this incident), R-49 (1.1 GB of a 1.2 GB immich
backup is ML model cache + a postgres data-dir tar duplicating the .sql dump +
immich's own nightly dump; recorded, not changed).
RULINGS LANDED.
- 00-capability-map.md:61 -> PARTIAL, scope-corrected: 6D's destruction hit the
FILE TREE only; the DB survived in its named volume, so "end-to-end from
offsite alone" overclaimed scope.
- The 704.6 MiB "discrepancy" -> CLOSED, not a defect: it was immich's own
Tarhely widget, never a controller page. Same for round 2's 650MiB->1.4GiB.
- customer-restore row: partial evidence recorded, NOT flipped.
Diagnosis only — no code, label or layout changes; no forget/prune.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
213 lines
12 KiB
Markdown
213 lines
12 KiB
Markdown
# DIAGNOSE — immich offsite restore "succeeds", photos do not reappear (2026-07-19)
|
||
|
||
> **FIXED IN controller v0.148.0 (2026-07-19)** — findings 1 and 2 below are closed in code as
|
||
> **R-43** (offsite reconstitution: safety dump → stop → overwrite files → start → replay the
|
||
> snapshot's dump) and **R-44** (every run dumps before it captures; manifests carry
|
||
> `offsite_run_id` + `dumps_at`). Findings 4 and 5 are described-and-accepted rather than changed;
|
||
> 3 stays with the v0.147 feedback arc and 6 is still unreconciled. **The live acceptance has not
|
||
> run** — the capability rows below are unchanged until it does. See `felhom-controller/REPORT.md`.
|
||
>
|
||
> **Class:** diagnosis. No product code was changed *by this document*. Findings + evidence only.
|
||
> **Box:** demo-felhom (felhom-pve), guest 9201, controller 0.146.0, immich v3.0.3.
|
||
> **Trigger:** Viktor deliberately deleted photos in the immich UI to **test the offsite restore
|
||
> feature**. The photos were never wanted back — the exercise was the point.
|
||
|
||
---
|
||
|
||
## TL;DR
|
||
|
||
The restore did not fail. **It was never invoked on missing data, and even if it had been, it could
|
||
not have brought the photos back.**
|
||
|
||
1. Deleting in the immich UI moves assets to **Lomtár (trash)**. No file leaves the disk. Both
|
||
„csak a hiányzó fájlok" runs therefore merged **0 files** — correctly, there was nothing missing.
|
||
The success flash was truthful about mechanism and empty as a test result.
|
||
2. A *valid* test (trash emptied, files genuinely gone) would also have failed to restore the
|
||
photos: **no offsite button loads a database.** The files would return; the library would still
|
||
be empty, because immich's truth lives in postgres and nothing replays the dump.
|
||
3. The dump that ships is additionally **stale by design** — it comes from the 02:30 local run and
|
||
is never refreshed before a manual push. Today's dump contains **0 users, 0 assets**.
|
||
|
||
**One-line answer to "can a customer trust same-day offsite?"** — For a DB-indexed app: **no.
|
||
Files come back; content does not.** The backup half is honest; the restore half cannot
|
||
reconstitute the app.
|
||
|
||
---
|
||
|
||
## Timeline (guest/DB clock; host is CEST, guest is UTC — CEST shown)
|
||
|
||
| CEST | Event | Evidence |
|
||
|---|---|---|
|
||
| 07-18 17:10 | old pre-v3 user tree `dccc13fe…` last written | dir mtime |
|
||
| 07-19 02:30 | local `db-dump` run writes `immich-postgres.sql` | mtime `00:30:00 UTC` |
|
||
| 07-19 07:56:25 | **admin user `4dcfb81e…` created** (fresh v3 deploy) | `user.createdAt` |
|
||
| 07-19 07:57:06–09 | Viktor uploads 11 assets | `asset.createdAt` |
|
||
| 07-19 07:58:15 | **manual offsite backup** — captures the 11 files | `[offbox] backed up immich (…1 mandatory path(s))` |
|
||
| 07-19 08:00:08 / 08:00:15 | Viktor deletes assets in UI → **trash**, 2 batches | `asset.deletedAt` |
|
||
| 07-19 08:01:14 | full restore → **scratch folder**, not live | `[offbox] restored immich (360304f8, full=true) → …/backups/offsite-restore/immich` |
|
||
| 07-19 08:27:54 | place (missing-only) → **0 files merged** | `[offbox] placed immich from offsite scratch: 0 file(s) merged (missing-only)` |
|
||
| 07-19 08:29:09 | full restore → scratch again | same as above |
|
||
| 07-19 08:29:57 | place (missing-only) → **0 files merged** | same as above |
|
||
|
||
The backup at 07:58:15 ran **after** the upload and **before** the delete — so the snapshot's
|
||
*file* half is correct and current. Only the DB half is not.
|
||
|
||
---
|
||
|
||
## Evidence
|
||
|
||
### DB state (live, at 08:34 CEST)
|
||
|
||
```
|
||
total | trashed | live
|
||
11 | 11 | 0 -- all 11 assets trashed, none live
|
||
users: 1 -- 4dcfb81e-073a-49fb-938e-7fffb51dbac4 / admin@felhom.eu
|
||
```
|
||
|
||
**`updatedAt` == `deletedAt` exactly on all 11 rows** — nothing has modified these rows since the
|
||
08:00 deletion. Conclusive: **none of today's restore operations touched the database.**
|
||
|
||
All 11 `originalPath` values resolve to files that exist on disk (11 OK / 0 missing, after mapping
|
||
the container path `/usr/src/app/upload` → `/mnt/felhom-drives/hdd_1/appdata/immich`). The photos
|
||
are fully recoverable by an ordinary in-app un-trash; **left in trash at Viktor's instruction.**
|
||
|
||
### The decisive dump probe
|
||
|
||
The recovery unit that actually ships:
|
||
`/mnt/felhom-drives/hdd_1/backups/primary/immich/db-dumps/immich-postgres.sql`
|
||
— 51 954 452 B, mtime **02:30 CEST**.
|
||
|
||
```
|
||
asset: 0 rows
|
||
user: 0 rows
|
||
album: 0 rows
|
||
```
|
||
|
||
The 52 MB is entirely `geodata_places` / `naturalearth_countries` — immich's shipped reference
|
||
data. **A dump that looks substantial by size contains zero customer content.** It was taken at
|
||
02:30, four hours before the admin user existed (07:56:25) and five and a half hours before the
|
||
photos were uploaded (07:57).
|
||
|
||
The same is true of the unit's volume dumps: `immich_immich_postgres_data.tar` (323 088 384 B) is
|
||
also from 02:30 CEST.
|
||
|
||
> **Method note / correction.** The first probe in this session read
|
||
> `appdata/immich/backups/immich-db-backup-20260719T020000-…sql.gz` — that is *immich's own*
|
||
> self-backup, not the controller's. It is independently also empty (`asset: 0`, `user: 0`), but
|
||
> the figure that matters for restore is the controller unit dump above. Both were probed; both
|
||
> are empty; the conclusion is unchanged.
|
||
|
||
### Safety consequence
|
||
|
||
Had a full restore actually loaded that dump into postgres, it would have written an **empty
|
||
database over the live one — destroying the trashed rows that are the only remaining recovery
|
||
path.** It did not, because (a) full restore only stages to a scratch folder and never touches a
|
||
DB, and (b) the place path's `live recovery unit present — not overwriting` guard skipped the unit
|
||
entirely. The photos survived by the combination of two behaviours, neither of which is a
|
||
deliberate safety feature for this case.
|
||
|
||
---
|
||
|
||
## Mechanism (source, controller 0.146.0)
|
||
|
||
| Question | Answer | Cite |
|
||
|---|---|---|
|
||
| Does a manual offsite run refresh DB dumps before pushing? | **No.** `offboxRunHandler` → `RunOffboxBackup` directly; never calls `RunDBDumps` / `captureAllRecoveryUnits`. It ships whatever `.sql` already exists. | `web/offbox_handlers.go:203-227`, `backup/offbox.go:574-759` |
|
||
| Where do dumps come from, then? | The separate local `db-dump` daily at **02:30**; offsite runs at **04:15**. A *manual* offsite run at any hour ships dumps up to ~24 h old. | `cmd/controller/main.go:542,611`, `config/config.go:317` |
|
||
| What does the recovery unit contain? | compose/, app.yaml, manifest.json — and it **enumerates** existing dump filenames, never creates them (`listFileNames`). | `backup/recovery_unit.go:68-154` (105-106) |
|
||
| „csak a hiányzó fájlok" — what does it do? | `rsync -a --ignore-existing`. Files only. **Never touches a DB. Never stops/starts the stack.** | `backup/offbox_restore.go:346-421`, `tier2_restore.go:148-162` |
|
||
| „Teljes visszaállítás" — restores to where? | A **scratch/verify folder** (`backups/offsite-restore/<app>`), never live data, and **never loads a dump into postgres**. | `backup/offbox_restore.go:171-232`, `:130-164` |
|
||
| Does *any* offsite path replay a DB dump? | **No.** The only code that replays into a live DB is `RestoreFromRecoveryUnit`, reachable solely from the **local** restore panel. | `backup/restore_unit.go:74` ← `web/handlers.go:1108` |
|
||
| Is there any dump-freshness / RPO check? | **None anywhere.** Zero `RPO` hits. `offboxUnitTime` is only a two-drive tiebreak. The `DBValidationCache` exists but no offsite/restore path reads it. | `backup/offbox.go:827-837`, `backup.go:364-370` |
|
||
|
||
### The place-success message
|
||
|
||
```
|
||
"A(z) " + app + " hiányzó fájljai helyreállítva az élő adatok közé." // offbox_handlers.go:354
|
||
```
|
||
|
||
Fixed string, built from the app name only. `PlaceOffsiteRestore` returns `error` alone, so the
|
||
merged-file count is **structurally unavailable** to the flash — it exists only in the English log
|
||
(`offbox_restore.go:419`). **With 0 files merged the customer still reads an unconditional claim
|
||
that files were restored.**
|
||
|
||
Tier-2 already solves this: `"Nincs hiányzó fájl — minden fájl megvan a helyén."`
|
||
(`web/handlers.go:1154`). The offsite path has no such branch — a straight parity gap.
|
||
|
||
---
|
||
|
||
## Findings
|
||
|
||
### 1. [HIGH → R-43] No offsite restore path can restore a database
|
||
|
||
All three offsite buttons are file-only. The DB dump is faithfully carried *into* the snapshot and
|
||
can never be replayed *out of* it by any offsite action. For every DB-indexed app — which is most
|
||
of the catalog — offsite restore cannot reconstitute the application. The customer is left with
|
||
correct bytes on disk and an app that cannot see them.
|
||
|
||
This outranks the staleness finding: even a perfectly fresh dump would not have restored the
|
||
photos today.
|
||
|
||
### 2. [HIGH → R-44] Manual offsite push ships an unrefreshed dump
|
||
|
||
"Backed up now" is false for the DB half. Today: files from 07:57, dump from 02:30 — containing
|
||
neither the user nor the assets. A same-day restore silently loses same-day state. No UI surface
|
||
discloses the DB's actual age.
|
||
|
||
### 3. [MED] Success semantics are mechanism-shaped, not outcome-shaped
|
||
|
||
Every flash today told the truth about what the code did while the customer watched nothing happen.
|
||
The zero-file case is the sharpest form: an unconditional "helyreállítva" when nothing was
|
||
restored. Feeds the v0.147.0 feedback arc — restore completions should state the **outcome**
|
||
(„N fájl visszaállítva; az adatbázis NEM — az alkalmazás tartalma nem változott").
|
||
|
||
### 4. [MED, new] The "live recovery unit present" guard silently skips the dump
|
||
|
||
`offbox_restore.go:400-405` skips the **entire** unit — compose, manifest **and db-dumps** —
|
||
whenever a local `backups/primary/<app>` exists, which is true on any box whose local backup has
|
||
run. So on a normal box the offsite dump is never even staged to disk. Correct as
|
||
"don't clobber the local unit"; invisible as "your DB was not part of this restore".
|
||
|
||
### 5. [LOW, new] Orphaned pre-v3 user tree
|
||
|
||
`upload|thumbs|encoded-video/dccc13fe-d629-4980-98e5-f2ea88cdb790` — 10 files / 36 MB + 3.9 MB +
|
||
15 MB, mtime 07-18 17:10, stranded by the immich 2→3 redeploy. The live DB has never known about
|
||
it. Dead weight, not today's issue; worth a sweep policy for major redeploys.
|
||
|
||
### 6. [CLOSED — not a defect] Storage figure discrepancy
|
||
|
||
The brief cites **704.6 MiB** used on the library storage. Measured: **126 MB** total
|
||
(`upload` 72 M, `encoded-video` 30 M, `backups` 18 M, `thumbs` 8 M).
|
||
|
||
**RULED 2026-07-19 (Viktor): resolved, no controller defect.** The 704.6 MiB came from **immich's
|
||
own Tárhely widget** — immich-internal accounting — not from any controller Storage page. The two
|
||
numbers measure different things and were never expected to agree. The same explanation covers the
|
||
„650 MiB → 1.4 GiB" figure quoted during round 2, which likewise does not match the controller's
|
||
measurement of the tree. Closed; no follow-up item.
|
||
|
||
### 7. [Method] A UI delete cannot test restore
|
||
|
||
Deleting in an app's own UI usually means *trash*, not deletion. Any "restore drill" scripted that
|
||
way will merge 0 files, flash success, and prove nothing. A real drill must empty the trash (or
|
||
otherwise remove the files) **and** verify the app's content afterwards, not just the file count.
|
||
This belongs in R-3's "first restore by the customer" step.
|
||
|
||
---
|
||
|
||
## Capability map
|
||
|
||
The **customer-restore row STAYS MISSING** and gains this diagnosis as a note. The C6 evidence
|
||
attempt found a product gap — that is the rehearsal system working, not failing. **R-3 stays
|
||
DRAFT.**
|
||
|
||
---
|
||
|
||
## What was NOT done
|
||
|
||
- No product fixes — no changes to restore/backup code, labels, or flashes.
|
||
- No `restic prune`/`forget`, no snapshot deletion, no escrow or check-folder changes.
|
||
- The 11 assets were **left in trash** (Viktor: recovery not wanted). immich's default 30-day
|
||
auto-purge applies; they are recoverable by un-trash until then.
|
||
- The genuine destructive test (empty trash → attempt real restore) was **offered and not run** —
|
||
it needs explicit go-ahead, and the dump probe already settles the question without it.
|