admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: DIAGNOSE immich offsite restore — findings + ROADMAP R-43/R-44

Browse Source 
Viktor deleted 11 immich photos to test offsite restore; both place runs
merged 0 files and flashed success. Two-layered root cause:

- A UI delete means TRASH, not deletion — no file left the disk, so there
  was genuinely nothing missing to restore. The test proved nothing.
- A valid test would also have failed: no offsite path loads a database.
  All three buttons are file-only (offbox_restore.go:171-232, :346-421).
  Files return; the library stays empty. The only DB replay is
  RestoreFromRecoveryUnit, reachable solely from the LOCAL restore panel.

Compounding: the manual push never refreshes dumps (offbox_handlers.go:
203-227) — today's unit dump is from 02:30 and probes to asset:0 / user:0 /
album:0, predating both the admin user and the photos. Had a full restore
loaded it, it would have written an empty DB over the live one, destroying
the trashed rows that were the only surviving recovery path.

Answer to "can a customer trust same-day offsite?" for a DB-indexed app:
no — files come back, content does not.

No product code changed. Photos left in trash (recovery not wanted).
Capability map: customer-restore row stays MISSING by finding, not by
absence of attempt; offsite-restore row flagged scope-contested pending
a ruling on whether CAMPAIGN-6D exercised the DB half.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
This commit is contained in:
authentik Default Admin
2026-07-19 08:47:34 +02:00
parent 317037f8eb
commit f3d1ba3062
4 changed files with 269 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT-diag-immich-restore.md
+64
View File
@@ -0,0 +1,64 @@
# REPORT — DIAGNOSE immich offsite restore (felhom.eu side: docs only)
> Written as `REPORT-diag-immich-restore.md`, **not** the shared `REPORT.md`, per the CLAUDE.md
> convention — `REPORT.md` is overwritten and currently holds the 2026-07-18 website refresh.
> No code shipped in this run; findings only.
**Date:** 2026-07-19 · **Box:** demo-felhom (felhom-pve guest 9201), controller 0.146.0, immich v3.0.3
## What ran
A diagnosis of "immich offsite restore succeeds but photos do not reappear". No product code was
changed: no restore/backup logic, no labels, no flashes, no `restic prune`/`forget`, no snapshot or
escrow changes.
## Outcome
The restore did not fail. **It was never invoked on missing data, and could not have worked
if it had been.**
1. Viktor deleted the 11 photos in the immich UI to test offsite restore. A UI delete means
**trash**, not deletion — no file left the disk. Both „csak a hiányzó fájlok" runs merged
**0 files**, correctly, and flashed success. The test proved nothing.
2. A *valid* test would also have failed: **no offsite path loads a database.** All three buttons
are file-only. Files would return; the library would stay empty.
3. The shipped dump is additionally stale by design — from the 02:30 local run, never refreshed
before a manual push. Probed: **`asset: 0`, `user: 0`, `album: 0`**.
**Photos:** left in trash at Viktor's instruction (recovery not wanted). All 11 files verified
present on disk and all 11 rows intact, so an ordinary un-trash recovers them until immich's
30-day auto-purge.
**Answer to "can a customer trust same-day offsite?"** For a DB-indexed app: **no — files come
back, content does not.** The backup half is honest; the restore half cannot reconstitute the app.
## Decisive evidence
- `updatedAt` == `deletedAt` on all 11 asset rows ⇒ **no restore operation touched the DB.**
- Unit dump `immich-postgres.sql`, 51 954 452 B, mtime **02:30 CEST** ⇒ `asset: 0 / user: 0 /
album: 0`. The 52 MB is entirely immich's shipped geodata reference tables. It predates the admin
user (07:56:25) and the photos (07:57).
- **Latent hazard:** had a full restore loaded that dump, it would have written an empty DB over the
live one, destroying the trashed rows that were the only surviving recovery path.
## Files written
| File | Change |
|---|---|
| `documentation/audits/DIAG-immich-restore-2026-07-19.md` | new — timeline, evidence, source mechanism table, 7 findings |
| `documentation/backlog/ROADMAP.md` | **R-43** (P2-HIGH, offsite restore cannot restore a DB) + **R-44** (P2-HIGH, manual push ships unrefreshed dump) |
| `documentation/architecture/00-capability-map.md` | customer-restore row **stays MISSING**, gains the finding + a method note for R-3; offsite-restore row flagged *scope contested* |
## Needs Viktor
1. **Capability-map ruling (line 61).** The offsite-restore row claims PROVEN-LIVE citing
"immich end-to-end from offsite alone" (`CAMPAIGN-6D`). That phrase is contested by this
diagnosis. I flagged it rather than downgrading a proven row — did 6D's accept leg exercise the
DB half, or only the file half?
2. **Optional real red-proof.** Emptying the trash would create genuine data loss and convert the
dump-probe inference into a live proof. Offered, **not run** — irreversible, and the probe
already settles it.
3. **Unreconciled: the 704.6 MiB figure.** Measured 126 MB total on the library storage. If
704.6 MiB came off a controller Storage page, that gap is its own defect.
4. **Orphaned pre-v3 tree** `dccc13fe…` (~55 MB across upload/thumbs/encoded-video) stranded by the
immich 2→3 redeploy — worth a sweep policy for major redeploys.
documentation/architecture/00-capability-map.md
+2 -2
View File
@@ -58,13 +58,13 @@
| Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 |
| Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | |
| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` |
| Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | |
| Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** — *scope contested, ruling needed* | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | **2026-07-19:** `audits/DIAG-immich-restore-2026-07-19.md` finds **no offsite path loads a DB dump** — all three buttons are file-only (R-43). The mechanics in this row's title are each proven; the phrase "**immich end-to-end from offsite alone**" is what is contested, since a DB-indexed app cannot be reconstituted by any offsite action. Status left as-is pending Viktor's read of whether 6D's accept leg actually exercised the DB half or only the file half |
| Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill |
| PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending |
| Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) |
| Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
| **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3 |
| **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count |
## D. Storage & devices
documentation/audits/DIAG-immich-restore-2026-07-19.md
+201
View File
@@ -0,0 +1,201 @@
# DIAGNOSE — immich offsite restore "succeeds", photos do not reappear (2026-07-19)
> **Class:** diagnosis. No product code was changed. Findings + evidence only.
> **Box:** demo-felhom (felhom-pve), guest 9201, controller 0.146.0, immich v3.0.3.
> **Trigger:** Viktor deliberately deleted photos in the immich UI to **test the offsite restore
> feature**. The photos were never wanted back — the exercise was the point.
---
## TL;DR
The restore did not fail. **It was never invoked on missing data, and even if it had been, it could
not have brought the photos back.**
1. Deleting in the immich UI moves assets to **Lomtár (trash)**. No file leaves the disk. Both
„csak a hiányzó fájlok" runs therefore merged **0 files** — correctly, there was nothing missing.
The success flash was truthful about mechanism and empty as a test result.
2. A *valid* test (trash emptied, files genuinely gone) would also have failed to restore the
photos: **no offsite button loads a database.** The files would return; the library would still
be empty, because immich's truth lives in postgres and nothing replays the dump.
3. The dump that ships is additionally **stale by design** — it comes from the 02:30 local run and
is never refreshed before a manual push. Today's dump contains **0 users, 0 assets**.
**One-line answer to "can a customer trust same-day offsite?"** — For a DB-indexed app: **no.
Files come back; content does not.** The backup half is honest; the restore half cannot
reconstitute the app.
---
## Timeline (guest/DB clock; host is CEST, guest is UTC — CEST shown)
| CEST | Event | Evidence |
|---|---|---|
| 07-18 17:10 | old pre-v3 user tree `dccc13fe…` last written | dir mtime |
| 07-19 02:30 | local `db-dump` run writes `immich-postgres.sql` | mtime `00:30:00 UTC` |
| 07-19 07:56:25 | **admin user `4dcfb81e…` created** (fresh v3 deploy) | `user.createdAt` |
| 07-19 07:57:06–09 | Viktor uploads 11 assets | `asset.createdAt` |
| 07-19 07:58:15 | **manual offsite backup** — captures the 11 files | `[offbox] backed up immich (…1 mandatory path(s))` |
| 07-19 08:00:08 / 08:00:15 | Viktor deletes assets in UI → **trash**, 2 batches | `asset.deletedAt` |
| 07-19 08:01:14 | full restore → **scratch folder**, not live | `[offbox] restored immich (360304f8, full=true) → …/backups/offsite-restore/immich` |
| 07-19 08:27:54 | place (missing-only) → **0 files merged** | `[offbox] placed immich from offsite scratch: 0 file(s) merged (missing-only)` |
| 07-19 08:29:09 | full restore → scratch again | same as above |
| 07-19 08:29:57 | place (missing-only) → **0 files merged** | same as above |
The backup at 07:58:15 ran **after** the upload and **before** the delete — so the snapshot's
*file* half is correct and current. Only the DB half is not.
---
## Evidence
### DB state (live, at 08:34 CEST)
```
total | trashed | live
11 | 11 | 0 -- all 11 assets trashed, none live
users: 1 -- 4dcfb81e-073a-49fb-938e-7fffb51dbac4 / admin@felhom.eu
```
**`updatedAt` == `deletedAt` exactly on all 11 rows** — nothing has modified these rows since the
08:00 deletion. Conclusive: **none of today's restore operations touched the database.**
All 11 `originalPath` values resolve to files that exist on disk (11 OK / 0 missing, after mapping
the container path `/usr/src/app/upload` → `/mnt/felhom-drives/hdd_1/appdata/immich`). The photos
are fully recoverable by an ordinary in-app un-trash; **left in trash at Viktor's instruction.**
### The decisive dump probe
The recovery unit that actually ships:
`/mnt/felhom-drives/hdd_1/backups/primary/immich/db-dumps/immich-postgres.sql`
— 51 954 452 B, mtime **02:30 CEST**.
```
asset: 0 rows
user: 0 rows
album: 0 rows
```
The 52 MB is entirely `geodata_places` / `naturalearth_countries` — immich's shipped reference
data. **A dump that looks substantial by size contains zero customer content.** It was taken at
02:30, four hours before the admin user existed (07:56:25) and five and a half hours before the
photos were uploaded (07:57).
The same is true of the unit's volume dumps: `immich_immich_postgres_data.tar` (323 088 384 B) is
also from 02:30 CEST.
> **Method note / correction.** The first probe in this session read
> `appdata/immich/backups/immich-db-backup-20260719T020000-…sql.gz` — that is *immich's own*
> self-backup, not the controller's. It is independently also empty (`asset: 0`, `user: 0`), but
> the figure that matters for restore is the controller unit dump above. Both were probed; both
> are empty; the conclusion is unchanged.
### Safety consequence
Had a full restore actually loaded that dump into postgres, it would have written an **empty
database over the live one — destroying the trashed rows that are the only remaining recovery
path.** It did not, because (a) full restore only stages to a scratch folder and never touches a
DB, and (b) the place path's `live recovery unit present — not overwriting` guard skipped the unit
entirely. The photos survived by the combination of two behaviours, neither of which is a
deliberate safety feature for this case.
---
## Mechanism (source, controller 0.146.0)
| Question | Answer | Cite |
|---|---|---|
| Does a manual offsite run refresh DB dumps before pushing? | **No.** `offboxRunHandler` → `RunOffboxBackup` directly; never calls `RunDBDumps` / `captureAllRecoveryUnits`. It ships whatever `.sql` already exists. | `web/offbox_handlers.go:203-227`, `backup/offbox.go:574-759` |
| Where do dumps come from, then? | The separate local `db-dump` daily at **02:30**; offsite runs at **04:15**. A *manual* offsite run at any hour ships dumps up to ~24 h old. | `cmd/controller/main.go:542,611`, `config/config.go:317` |
| What does the recovery unit contain? | compose/, app.yaml, manifest.json — and it **enumerates** existing dump filenames, never creates them (`listFileNames`). | `backup/recovery_unit.go:68-154` (105-106) |
| „csak a hiányzó fájlok" — what does it do? | `rsync -a --ignore-existing`. Files only. **Never touches a DB. Never stops/starts the stack.** | `backup/offbox_restore.go:346-421`, `tier2_restore.go:148-162` |
| „Teljes visszaállítás" — restores to where? | A **scratch/verify folder** (`backups/offsite-restore/<app>`), never live data, and **never loads a dump into postgres**. | `backup/offbox_restore.go:171-232`, `:130-164` |
| Does *any* offsite path replay a DB dump? | **No.** The only code that replays into a live DB is `RestoreFromRecoveryUnit`, reachable solely from the **local** restore panel. | `backup/restore_unit.go:74` ← `web/handlers.go:1108` |
| Is there any dump-freshness / RPO check? | **None anywhere.** Zero `RPO` hits. `offboxUnitTime` is only a two-drive tiebreak. The `DBValidationCache` exists but no offsite/restore path reads it. | `backup/offbox.go:827-837`, `backup.go:364-370` |
### The place-success message
```
"A(z) " + app + " hiányzó fájljai helyreállítva az élő adatok közé." // offbox_handlers.go:354
```
Fixed string, built from the app name only. `PlaceOffsiteRestore` returns `error` alone, so the
merged-file count is **structurally unavailable** to the flash — it exists only in the English log
(`offbox_restore.go:419`). **With 0 files merged the customer still reads an unconditional claim
that files were restored.**
Tier-2 already solves this: `"Nincs hiányzó fájl — minden fájl megvan a helyén."`
(`web/handlers.go:1154`). The offsite path has no such branch — a straight parity gap.
---
## Findings
### 1. [HIGH → R-43] No offsite restore path can restore a database
All three offsite buttons are file-only. The DB dump is faithfully carried *into* the snapshot and
can never be replayed *out of* it by any offsite action. For every DB-indexed app — which is most
of the catalog — offsite restore cannot reconstitute the application. The customer is left with
correct bytes on disk and an app that cannot see them.
This outranks the staleness finding: even a perfectly fresh dump would not have restored the
photos today.
### 2. [HIGH → R-44] Manual offsite push ships an unrefreshed dump
"Backed up now" is false for the DB half. Today: files from 07:57, dump from 02:30 — containing
neither the user nor the assets. A same-day restore silently loses same-day state. No UI surface
discloses the DB's actual age.
### 3. [MED] Success semantics are mechanism-shaped, not outcome-shaped
Every flash today told the truth about what the code did while the customer watched nothing happen.
The zero-file case is the sharpest form: an unconditional "helyreállítva" when nothing was
restored. Feeds the v0.147.0 feedback arc — restore completions should state the **outcome**
(„N fájl visszaállítva; az adatbázis NEM — az alkalmazás tartalma nem változott").
### 4. [MED, new] The "live recovery unit present" guard silently skips the dump
`offbox_restore.go:400-405` skips the **entire** unit — compose, manifest **and db-dumps** —
whenever a local `backups/primary/<app>` exists, which is true on any box whose local backup has
run. So on a normal box the offsite dump is never even staged to disk. Correct as
"don't clobber the local unit"; invisible as "your DB was not part of this restore".
### 5. [LOW, new] Orphaned pre-v3 user tree
`upload|thumbs|encoded-video/dccc13fe-d629-4980-98e5-f2ea88cdb790` — 10 files / 36 MB + 3.9 MB +
15 MB, mtime 07-18 17:10, stranded by the immich 2→3 redeploy. The live DB has never known about
it. Dead weight, not today's issue; worth a sweep policy for major redeploys.
### 6. [LOW, unresolved] Storage figure discrepancy
The brief cites **704.6 MiB** used on the library storage. Measured: **126 MB** total
(`upload` 72 M, `encoded-video` 30 M, `backups` 18 M, `thumbs` 8 M). Not reconciled in this
session. If 704.6 MiB came off a controller Storage page, that gap is its own defect and needs a
separate look.
### 7. [Method] A UI delete cannot test restore
Deleting in an app's own UI usually means *trash*, not deletion. Any "restore drill" scripted that
way will merge 0 files, flash success, and prove nothing. A real drill must empty the trash (or
otherwise remove the files) **and** verify the app's content afterwards, not just the file count.
This belongs in R-3's "first restore by the customer" step.
---
## Capability map
The **customer-restore row STAYS MISSING** and gains this diagnosis as a note. The C6 evidence
attempt found a product gap — that is the rehearsal system working, not failing. **R-3 stays
DRAFT.**
---
## What was NOT done
- No product fixes — no changes to restore/backup code, labels, or flashes.
- No `restic prune`/`forget`, no snapshot deletion, no escrow or check-folder changes.
- The 11 assets were **left in trash** (Viktor: recovery not wanted). immich's default 30-day
auto-purge applies; they are recoverable by un-trash until then.
- The genuine destructive test (empty trash → attempt real restore) was **offered and not run** —
it needs explicit go-ahead, and the dump probe already settles the question without it.
documentation/backlog/ROADMAP.md
+2
View File
@@ -82,6 +82,8 @@
| R-40 | **[P2-HIGH] The update path cannot express a MULTI-HOP major upgrade.** A template pin is a single value; the customer's update button pulls whatever the catalog now says. For apps whose upstream forbids version skipping this produces a broken upgrade. Nextcloud is explicit: *"You cannot skip major releases. Please re-run the upgrade until you have reached the highest available release."* Campaign 7 moved its template **31 → 34** (a fresh deploy validates fine — 302, 3/3 healthy), so an existing 31 customer pressing update would attempt a jump Nextcloud refuses. | M | idea | Origin: CAMPAIGN 7 (`audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` §7 F7). Not nextcloud-only — any app with sequential-major rules (gitea, tandoor, outline…) has the same shape. Directions: a per-app `upgrade_path:`/`max_hop:` in `.felhom.yml` that the update button walks in stages; or refuse-and-explain when the installed major is >1 behind; or pin an intermediate "stepping-stone" tag. **Until this exists, a >1-major catalog bump is safe for NEW deploys and unsafe for the update button** — which is exactly the asymmetry the campaign's MAJOR flag was meant to record but cannot enforce |
| R-41 | **[P2-HIGH] The catalog has no standing "does every template still deploy?" check.** Campaign 7 was the first thing that ever tried to deploy all 53 apps, and found **5 that had NEVER been deployable**: papra (missing required `AUTH_SECRET`), zipline (v4 renamed `CORE_DATABASE_URL` → `DATABASE_URL`), wishlist (Docker Hub image gone; upstream moved to ghcr.io), homebox (upstream dropped the `v` tag prefix + new required env), glance (needs a seeded `glance.yml` the template never provides — PROVEN pre-existing: the pre-campaign v0.7.4 pin fails identically). Plus **7 broken healthchecks** and 2 apps whose images no longer resolve at all (plant-it, wanderer). | M | idea | Origin: CAMPAIGN 7 (§7 F5/F6). The repo already has the right pattern in `scripts/check-image-pins.py` — a mechanical gate run on every change. Cheap first slice: a **resolvability gate** (`docker manifest inspect` every pin) would alone have caught plant-it, wanderer, wishlist and homebox, and needs no box. Full slice: a periodic deploy-all sweep on the demo box reusing the campaign's engine. **Silent rot is the real risk** — an app can die upstream and nobody learns until a customer clicks Telepítés |
| R-42 | **Ruling needed: do DB/cache sidecar majors follow the app, or the newest tag?** Campaign 7 deliberately did NOT bump sidecar majors (postgres 16→18, redis 7→8, mariadb 11.6→12.3, postgis 16→17) while bumping ~40 app images to current. | S | **decision pending (Viktor)** | Origin: CAMPAIGN 7 §4. The case for not bumping: a DB major is a **data-plane decision the application owns** — immich proves it, upstream pins one specific tested `postgres:14-vectorchord…` build — and `postgres:16-alpine`/`redis:7-alpine` already track the newest patch inside their major, so they are not stale. The case for bumping: EOL majors eventually stop getting security patches, and "we never bump" silently becomes "we ship EOL databases". Suggested shape: per-app sidecar pin follows **upstream's own compose** where upstream publishes one, else stay within the current major and revisit at that major's EOL date |
| R-43 | **[P2-HIGH] No offsite restore path can restore a database — offsite restore cannot reconstitute a DB-indexed app.** All three offsite buttons are file-only. „Ellenőrző visszaállítás" and „Teljes visszaállítás" restore into a **scratch/verify folder** and never load a dump into postgres (`controller/internal/backup/offbox_restore.go:171-232`, `:130-164`); „Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" is `rsync -a --ignore-existing` over **mandatory userdata paths only** — it never touches a DB and never stops/starts the stack (`offbox_restore.go:346-421`). The DB dump is faithfully carried *into* the snapshot and can never be replayed *out of* it by any offsite action; the only code that replays into a live DB is `RestoreFromRecoveryUnit` (`backup/restore_unit.go:74`), reachable solely from the **local** restore panel (`web/handlers.go:1108`). Compounding: the `live recovery unit present — not overwriting` guard (`offbox_restore.go:400-405`) skips the **entire** unit — compose, manifest **and db-dumps** — whenever a local `backups/primary/<app>` exists, which is true on any box whose local backup has run, so on a normal box the offsite dump is never even staged to disk. | M | idea | **Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (live on demo-felhom).** Viktor deleted 11 immich photos to test offsite restore; both place runs merged **0 files** and flashed success. Root cause is two-layered — the UI delete only *trashed* the assets so nothing was missing to restore, **but a valid test would also have failed**: files return, the library stays empty, because immich's truth lives in postgres. **The customer-visible failure mode: correct bytes on disk and an app that cannot see them.** Severity case: this is the offsite tier — the one that survives losing the box — and most of the catalog is DB-indexed. Direction (needs a spec, do not improvise): either offsite gains a real DB-replay leg reusing `RestoreFromRecoveryUnit` behind a stop→restore→start sequence, or the offsite UI states plainly that it restores files only and routes the customer to the local recovery-unit flow for the DB. Pairs with R-44 (a replay leg is worthless while the dump is stale). Flips: the **customer-restore map row stays MISSING** and gains this as a note — the C6 evidence attempt found a product gap, which is the rehearsal system working |
| R-44 | **[P2-HIGH] A manual offsite push ships an unrefreshed DB dump — "backed up now" is false for the DB half.** `offboxRunHandler` → `RunOffboxBackup` goes straight to the restic push and never calls `RunDBDumps` / `captureAllRecoveryUnits` (`controller/internal/web/offbox_handlers.go:203-227`, `backup/offbox.go:574-759`); the recovery unit merely **enumerates** existing dump filenames via `listFileNames`, never creates them (`backup/recovery_unit.go:105-106`). Dumps come only from the separate local `db-dump` daily at **02:30** (`cmd/controller/main.go:542`), with the scheduled offsite at 04:15 — so a *manual* run at any other hour ships a dump up to ~24 h old. **There is no freshness check and no RPO surface anywhere:** zero `RPO` hits across `controller/`; `offboxUnitTime` is only a two-drive tiebreak (`offbox.go:827-837`); the `DBValidationCache` exists (`backup.go:364-370`) but no offsite or restore path reads it. | S–M | idea | **Evidence: `audits/DIAG-immich-restore-2026-07-19.md`.** Today's unit dump `immich-postgres.sql` (51 954 452 B, mtime **02:30 CEST**) probed to **`asset: 0 rows`, `user: 0 rows`, `album: 0 rows`** — the 52 MB is entirely immich's shipped `geodata_places`/`naturalearth_countries` reference data. It predates both the admin user (created 07:56:25) and the photos (07:57). Same for the unit's `immich_immich_postgres_data.tar` (323 MB, also 02:30). **A dump that looks substantial by size can contain zero customer content** — size is not a health signal, and nothing in the product says otherwise. **Latent hazard:** had a full restore actually loaded that dump it would have written an empty DB over the live one, destroying the trashed rows that were the only surviving recovery path. Direction: dump-before-push on manual runs (the honest fix), **or** an explicit RPO line in the UI („adatbázis-állapot: <dump time>") so the operator/customer can see what they are actually shipping. Cheap interim: surface dump mtime + row-count sanity from the existing `DBValidationCache` on `/backups/restore` |
## Pre-invite checklist — what stands between here and the first remote tester