docs: R-47 live-validated + golden 0.153.0 published

ROADMAP R-47 and the capability map's offsite-restore row now carry the live
evidence: a supervised reconstitute of immich from snapshot 49e7cb46 — the
same snapshot that aborted in round 2 — ran stop -> DB-service-only start ->
replay rc-0 -> full start, with no 'already exists', a SUCCESS outcome, and
immich's own 'No schema drift detected' twice where round 2 left it
reporting drift.

The offsite-restore row's residual for a PROVEN-LIVE flip is now only the
immich timeline screenshot (no browser on DooPlex).

Golden 0.153.0 baked and published the same day, sha256 15fdd191f3c660a6...,
first golden carrying all four infra images.

documentation/backlog/ROADMAP.md

@@ -90,7 +90,7 @@
 | R-44 | **[P2-HIGH] A manual offsite push ships an unrefreshed DB dump — "backed up now" is false for the DB half.** `offboxRunHandler` → `RunOffboxBackup` goes straight to the restic push and never calls `RunDBDumps` / `captureAllRecoveryUnits` (`controller/internal/web/offbox_handlers.go:203-227`, `backup/offbox.go:574-759`); the recovery unit merely **enumerates** existing dump filenames via `listFileNames`, never creates them (`backup/recovery_unit.go:105-106`). Dumps come only from the separate local `db-dump` daily at **02:30** (`cmd/controller/main.go:542`), with the scheduled offsite at 04:15 — so a *manual* run at any other hour ships a dump up to ~24 h old. **There is no freshness check and no RPO surface anywhere:** zero `RPO` hits across `controller/`; `offboxUnitTime` is only a two-drive tiebreak (`offbox.go:827-837`); the `DBValidationCache` exists (`backup.go:364-370`) but no offsite or restore path reads it. | S–M | **SHIPPED controller v0.148.0 (2026-07-19)** | **SHIPPED:** every offsite run — **manual AND nightly** — now refreshes the DB/volume dumps and recovery units (`offsitePreDump` → `runDBDumpsInternal`) BEFORE the restic capture, so each snapshot is an internally coherent `{DB@T, files@T}` bundle and retention becomes a history of restorable points. Order is the mechanism and is red-proofed (moving the capture first yields `[capture dump]`): the gap can only ADD files the DB does not reference yet, never remove one it does. This also makes the nightly ordering **structural** rather than a coincidence of two scheduler entries at 02:30 and 04:15. Each unit manifest carries `offsite_run_id` + `dumps_at`, so a pair's coherence is verifiable at restore time instead of assumed; the periodic refresh carries a prior stamp forward and never invents one. A dump-leg failure is a loud WARN that does NOT abort the push (data-first: a degraded backup beats none). Honesty surfaces, all warn-level and none a gate: an unstamped pre-v0.148 pair reports its skew in the confirm, and `ValidateDump` gained an **exact-match** accounts-table sniff for customer-empty dumps (a substring match on "user" would flag `user_metadata`/`album_user`/`user_audit` on every healthy single-user box — red-proofed). — **Evidence: `audits/DIAG-immich-restore-2026-07-19.md`.** Today's unit dump `immich-postgres.sql` (51 954 452 B, mtime **02:30 CEST**) probed to **`asset: 0 rows`, `user: 0 rows`, `album: 0 rows`** — the 52 MB is entirely immich's shipped `geodata_places`/`naturalearth_countries` reference data. It predates both the admin user (created 07:56:25) and the photos (07:57). Same for the unit's `immich_immich_postgres_data.tar` (323 MB, also 02:30). **A dump that looks substantial by size can contain zero customer content** — size is not a health signal, and nothing in the product says otherwise. **Latent hazard:** had a full restore actually loaded that dump it would have written an empty DB over the live one, destroying the trashed rows that were the only surviving recovery path. Direction: dump-before-push on manual runs (the honest fix), **or** an explicit RPO line in the UI („adatbázis-állapot: <dump time>") so the operator/customer can see what they are actually shipping. Cheap interim: surface dump mtime + row-count sanity from the existing `DBValidationCache` on `/backups/restore` |
 | R-45 | **[P2] Unified async-job feedback.** Every long operation invents its own progress surface, or none. Tonight produced three more one-off cards (v0.147.x: samba bring-up, offsite progress, restore result) on top of two existing patterns (deploy 3-step panel; storage-init/netstorage status poll). They agree on nothing: some use `{ok,data}` envelopes and some raw JSON, some poll 1 s / 1.5 s / 3 s, some are in-memory-only and lie after a restart, and each re-implements single-flight + snapshot + phase→Hungarian mapping. | M | idea | Origin: 2026-07-19 feedback slice 1 (controller v0.147.0). The cases to generalise from are all in-tree: `web/storage_init_job.go` (the best shape — acquire/release/set/snapshot), `web/netstorage_job.go`, `web/samba_ensure_job.go`, `backup/opstatus.go`, `backup/offbox_progress.go`. Shape: one job registry + one poll endpoint + one client-side renderer, phases declared per job. **Two lessons tonight that any framework must encode:** (1) a terminal state must be **probed, not inferred** — `compose up -d` exits 0 on a crash-loop; (2) a progress source that reports nothing is normal, not broken — restic reports 0 bytes for a whole incremental run, and a bar that sits at 0% is worse than no bar. Also fixes the restart hole: in-memory job state currently vanishes and the card silently disagrees with reality  **2026-07-20 — the first bill for NOT having this arrived, and it was customer-facing.** The samba card's poll (`web/samba_ensure_job.go` + `sharing.html`) mixed a job EDGE and a service LEVEL on one JSON field, and `/sharing` reload-looped at ~1.2 s for every customer with sharing enabled until controller v0.151.0 (`audits/DIAG-sharing-2026-07-20.md`, S-1/S-4). v0.151.0 fixed THAT card's contract only — the framework is still this item. **Third lesson for it to encode, beside the two already listed:** a phase a client answers with a one-shot action must be an EDGE the registry SERVES ONCE, and must never be synthesised from a level; if it can be re-read, it will be re-acted on. |
 | R-46 | **[P2] Verification copies need a customer-visible browse surface and an expiry.** v0.147.0 made them *visible* (listed with path/size/date, individually deletable) — but the customer still cannot LOOK INSIDE a verification restore to confirm the file they wanted is really there, which is the entire point of a verification restore, and nothing ever removes them. | S–M | idea | Origin: 2026-07-19 feedback slice 4a, registered as the explicit follow-up to it. Two gaps, deliberately designed together because they are the same object: (a) **the invisible-result gap** — a read-only browse of `backups/offsite-restore/<app>` (the FileBrowser infra stack already exists and already serves scoped roots, so this may be a mount rather than new code); (b) **the disk-lifecycle gap** — auto-expiry after N days with the count/size surfaced before it fires, so a drive is never quietly filled by verification restores nobody remembers taking. Pairs with R-43: a browse surface is also how a customer would discover that a DB-indexed app's files came back but the app still cannot see them |
-| R-47 | **[P2-HIGH] The DB replay races the application's own schema repair.** | M | **SHIPPED — controller v0.153.0, 2026-07-20** | Fixed on **BOTH** paths (it was a class defect, not an offsite-only one): offsite `ReconstituteFromOffsite` AND local `RestoreFromRecoveryUnit` now open a **DB-only window** — `appbackup.DBServiceNames` names the database SERVICE from the compose `services:` map, `stacks.Manager.StartStackServices` brings up only that service, the dump is replayed with the application still stopped, and the full start runs only after the replay exits 0. **Fail-closed:** a `.sql` dump with no identifiable DB service refuses BEFORE the first mutation. Every exit from the window (replay failure, DB-only start failure) still does a best-effort full start, so a failed restore never leaves a box with a database and no application. `--clean --if-exists` + `ON_ERROR_STOP=1` untouched — the bug was the window, not the flags. Enablers: `RedeployFromEnv` split into `PersistUnitRedeployConfig` + its unchanged tail; `StackDataProvider.RecreateStackFromUnit` → `RecreateStackDefinitionFromUnit` (the hidden `up -d` inside the old name is what carried the defect locally). **No agent coupling — MinAgent stays 0.90.0.** 19 tests + 3 red-proofs, 23/23 green. Unblocks the capability-map **offsite-restore (PARTIAL)** and **customer-restore (MISSING)** rows, both of which now await only one clean acceptance run. Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` (H4) |
+| R-47 | **[P2-HIGH] The DB replay races the application's own schema repair.** | M | **SHIPPED — controller v0.153.0, 2026-07-20** | Fixed on **BOTH** paths (it was a class defect, not an offsite-only one): offsite `ReconstituteFromOffsite` AND local `RestoreFromRecoveryUnit` now open a **DB-only window** — `appbackup.DBServiceNames` names the database SERVICE from the compose `services:` map, `stacks.Manager.StartStackServices` brings up only that service, the dump is replayed with the application still stopped, and the full start runs only after the replay exits 0. **Fail-closed:** a `.sql` dump with no identifiable DB service refuses BEFORE the first mutation. Every exit from the window (replay failure, DB-only start failure) still does a best-effort full start, so a failed restore never leaves a box with a database and no application. `--clean --if-exists` + `ON_ERROR_STOP=1` untouched — the bug was the window, not the flags. Enablers: `RedeployFromEnv` split into `PersistUnitRedeployConfig` + its unchanged tail; `StackDataProvider.RecreateStackFromUnit` → `RecreateStackDefinitionFromUnit` (the hidden `up -d` inside the old name is what carried the defect locally). **No agent coupling — MinAgent stays 0.90.0.** 19 tests + 3 red-proofs, 23/23 green. Unblocks the capability-map **offsite-restore (PARTIAL)** and **customer-restore (MISSING)** rows, both of which now await only one clean acceptance run. **LIVE-VALIDATED 2026-07-20** against the SAME snapshot that aborted in round 2 (`49e7cb46`): log shows `Starting stack immich services only: [immich-postgres]` → replay rc-0 in 20 s → full start; no `already exists`; immich's own DatabaseService logged **`No schema drift detected`** twice (round 2 left it reporting drift); 11 assets `active`, 4/4 containers healthy. **Golden 0.153.0 baked + published the same day** (sha256 `15fdd191f3c660a6…`) — the first golden carrying all FOUR infra images. Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` (H4) + `felhom-controller/REPORT.md` §4b/§4c |
 | R-48 | **[P2-HIGH] Restore controls are separable only by layout — and the difference between them is whether the data comes back.** The offsite restore row renders four buttons plus hint text into an overlapping, unreadable line, and the decisive second step („Teljes visszaállítás indítása") appears ONLY after „…előkészítése" was pressed, with no signposting that a second step exists or that the first one did nothing to live data. | M | idea | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` (finding 1) — this is not theoretical: it is the CAUSE of the round-2 incident.** An operator who had read the code pressed the missing-only button instead of the full restore; the controller log shows `/backup/offbox/reconstitute` was never hit at all. The rule this establishes, worth stating once and applying beyond this page: **two adjacent controls whose difference is "your data comes back" vs "your data cannot come back" must not be distinguishable only by layout.** Direction (ruled in principle, spec rides v0.149): collapse to a single „Visszaállítás…" guided dialog — one intent, visible phases, the escrow-wizard precedent. Pairs with R-45 (the phases are exactly the async-feedback surface) and R-46 |
 | R-49 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. | S–M | idea | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app |