From 062357f778c5101d38c55a334ee45e1c83517d38 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 19 Jul 2026 12:21:05 +0200 Subject: [PATCH] =?UTF-8?q?v0.148.0=20=E2=80=94=20coherent=20snapshot=20pa?= =?UTF-8?q?irs=20+=20an=20offsite=20restore=20that=20actually=20restores?= =?UTF-8?q?=20(R-43=20+=20R-44)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11 immich photos to test offsite restore; both runs flashed success and the photos stayed gone. Two independent defects. R-43 — no offsite path could restore a database. All three buttons were file-only: the two "visszaállítás" actions staged to a scratch folder and never touched postgres, and place-to-live merged only MISSING files. For a DB-indexed app the bytes returned and the app still could not see them. The dump was carried INTO every snapshot and could never be replayed OUT of one. New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop → files overwritten to the snapshot version → start → the snapshot's own dump replayed → health wait. Two invariants: - nothing is ever deleted (-a, no --ignore-existing, no --delete): a file created after the snapshot survives as an extra; - the undo exists before the act — the pre-restore- dump is verified ON DISK before anything is stopped, overwritten or replayed; if it cannot be taken the operation refuses with zero changes. The replay reads the SCRATCH unit: the live unit is never overwritten, so replaying from it would replay the current DB over itself and restore nothing. R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's predated the customer's account by four hours and probed to asset:0/user:0/ album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual AND nightly, now refreshes dumps + units BEFORE capturing. Order is the mechanism: the gap can only ADD files the DB does not reference yet, never remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is verifiable at restore time rather than assumed; the periodic refresh carries a prior stamp forward and never invents one. Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs report their skew, ValidateDump gained an EXACT-match accounts-table sniff for customer-empty dumps, the completion flash states an outcome instead of a mechanism, and the missing-only button now says what it does NOT do. 11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses rather than confirming strength — the first undo mutation was caught by a second guard, and the first table-matching test did not discriminate between the two matchers at all. Both tests were rewritten to the cases that separate them. NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota math, tier-2, and v0.147.x progress semantics beyond one added phase line. Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row stays MISSING, R-3 stays DRAFT. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s --- CHANGELOG.md | 74 +++ CONTEXT.md | 21 +- REPORT.md | 209 +++------ REUSE.md | 6 + controller/README.md | 20 + controller/internal/appbackup/dbdump.go | 96 +++- .../internal/appbackup/dbdump_sniff_test.go | 141 ++++++ controller/internal/backup/backup.go | 39 ++ controller/internal/backup/offbox.go | 31 ++ controller/internal/backup/offbox_progress.go | 4 +- .../internal/backup/offbox_reconstitute.go | 400 +++++++++++++++++ .../backup/offbox_reconstitute_test.go | 424 ++++++++++++++++++ controller/internal/backup/offbox_restore.go | 10 + controller/internal/backup/recovery_unit.go | 28 +- controller/internal/backup/restore_db.go | 10 +- controller/internal/web/handlers.go | 11 + controller/internal/web/offbox_handlers.go | 64 +++ controller/internal/web/server.go | 2 + .../web/templates/backups_remote.html | 10 + .../web/templates/backups_restore.html | 40 +- 20 files changed, 1486 insertions(+), 154 deletions(-) create mode 100644 controller/internal/appbackup/dbdump_sniff_test.go create mode 100644 controller/internal/backup/offbox_reconstitute.go create mode 100644 controller/internal/backup/offbox_reconstitute_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 9346935..106db1a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,79 @@ ## Changelog +### v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (2026-07-19) + +Closes **R-43** and **R-44**, the two findings from `DIAG-immich-restore-2026-07-19`. The short +version of that diagnosis: Viktor deleted 11 immich photos to test offsite restore, both restore +runs flashed success, and the photos stayed gone. Two independent defects, both fixed here. + +**R-43 — no offsite path could restore a database.** All three offsite buttons were file-only. +The two „visszaállítás" actions staged into a scratch folder and never touched postgres; the +place-to-live action merged only files MISSING from the live tree and never replayed a dump. For a +DB-indexed app — most of the catalog — that combination cannot bring content back: the bytes +return and the app still cannot see them, because its index lives in the database. The dump was +faithfully carried INTO every snapshot and could never be replayed OUT of one. + +New: **„Teljes visszaállítás (fájlok + adatbázis)"** (`ReconstituteFromOffsite`, +`/backup/offbox/reconstitute`). Safety dump → stop → files overwritten to the snapshot's version → +start → the snapshot's own dump replayed → health wait. Two invariants: + +- **Nothing is ever deleted.** The full-restore copier is `rsync -a` with NO `--ignore-existing` + (a changed file becomes the snapshot's version) and NO `--delete` (a file created after the + snapshot survives as an extra). A restore that silently removed newer work would be a data-loss + event wearing a recovery button's label. +- **The undo exists before the act.** A `pre-restore-` dump of the live database is written and + verified on disk BEFORE anything is stopped, overwritten or replayed; if it cannot be taken the + whole operation refuses with zero changes. The safety dumps live in the app's own unit and + appear in `ListDumpFiles` — an undo the customer cannot see is not much of one. + +The replay reads the SCRATCH unit, not the live one: the live recovery unit is still never +overwritten (it is the local restore path's source), so replaying from it would replay the current +database back over itself and restore nothing. + +**R-44 — a manual push shipped an unrefreshed dump.** `RunOffboxBackup` went straight to the +restic push; dumps came only from the separate 02:30 local run, so a manual push at any other hour +shipped a dump up to ~24h old. On 2026-07-19 that dump was taken four hours before the customer's +account existed and probed to `asset: 0 / user: 0 / album: 0` — a 52MB file whose entire bulk was +immich's shipped geodata tables. Size and table count both called it healthy. + +Every offsite run — **manual and nightly** — now refreshes the dumps and recovery units FIRST, then +captures. Order is the mechanism: the gap can only ADD files the DB does not reference yet (a +harmless orphan blob), never remove one it does, so the file set is always a superset of what the +restored DB points at. This also makes the nightly ordering structural instead of a coincidence of +two scheduler entries at 02:30 and 04:15. Each unit manifest carries the run's `offsite_run_id` + +`dumps_at`, so a snapshot's coherence is verifiable at restore time rather than assumed. + +**Honesty surfaces** (warn-level, never gates — a false positive that blocked a restore would be +worse than the skew it guards against): + +- A pre-v0.148 snapshot has no stamp → the confirm says „Az adatbázis-mentés régebbi () — a + fájlok és az adatbázis eltérő időpontból származnak." It still restores. +- `ValidateDump` gained a content sniff: a structurally valid dump whose accounts table has zero + rows raises „A mentett adatbázis üresnek tűnik". Exact table-name matching, deliberately — a + substring match on "user" would flag `user_metadata` / `album_user` / `user_audit` on every + healthy single-user box and turn the signal into noise. +- The completion flash states an OUTCOME, not a mechanism: „A(z) X: N fájl és az adatbázis + visszaállítva (mentés: ) — az alkalmazás újraindult." A no-database app says so explicitly + rather than borrowing the confident sentence. +- The old missing-only button now says what it does NOT do: „Adatbázist nem állít vissza — törölt + tartalom ettől nem jelenik meg újra." + +A new `dump` progress phase („Adatbázisok mentése a pillanatképhez…") names the pre-phase, which on +a large database dominates the early wall clock and would otherwise read as a hang. + +Tests: 11 new, with **5 red-proofs run and reverted** — replay removed (0 replayed), capture moved +before the dump (`[capture dump]`), both undo guards removed (no refusal), substring table matching +(join tables mistaken for accounts), buffer-exceeding rows uncounted (a wide row sniffed as empty). +Two of those red-proofs found real test weaknesses rather than confirming strength: the first undo +mutation was caught by a second guard, and the first table-matching test did not discriminate +between the two matchers at all — both tests were rewritten to the cases that actually separate +them. + +**NOT in this slice:** the catalog-wide invariant check (stays on R-41), nightly cadence, retention, +quota math, tier-2, and the v0.147.x progress semantics beyond the one added phase line. The +missing-only place button's own zero-file flash is also untouched — that is the v0.147 feedback arc's +item, not R-43/R-44. + ### docs — the workflow moved to DooPlex-local execution (2026-07-19) **Docs only, no version bump, no code change.** Claude Code now runs on DooPlex diff --git a/CONTEXT.md b/CONTEXT.md index f2982db..5ce0269 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -7,7 +7,26 @@ > > Ask Claude Code: "Please update CONTEXT.md with what we did today" -Last updated: 2026-07-19 (v0.147.3 — feedback slice 1: pressing a button says something) +Last updated: 2026-07-19 (v0.148.0 — coherent snapshot pairs + an offsite restore that restores) + +> **2026-07-19 — v0.148.0: R-43 + R-44.** Viktor deleted 11 immich photos to test offsite restore; +> both runs flashed success and the photos stayed gone (`DIAG-immich-restore-2026-07-19`). Two +> defects. **R-43:** no offsite path could restore a DATABASE — all three buttons were file-only, so +> a DB-indexed app got its bytes back and still could not see them. **R-44:** a manual push shipped +> whatever dump the 02:30 local run left; that day's predated the customer's account by four hours +> and held zero users and zero assets inside 52MB of shipped geodata. +> +> Now: every run (manual AND nightly) dumps FIRST, then captures, so each snapshot is a coherent +> `{DB@T, files@T}` pair stamped with `offsite_run_id`; and „Teljes visszaállítás (fájlok + +> adatbázis)" does safety-dump → stop → overwrite files → start → replay the snapshot's dump. +> **Two invariants: nothing is ever deleted, and the undo is verified on disk before the act.** +> Warn-level honesty surfaces for stale/empty-looking pairs — never gates. +> +> **The live acceptance has NOT run yet** (§9: upload → push → empty trash for real → one button → +> photos back). Until it does, the capability-map offsite row stays PARTIAL/scope-contested, the +> customer-restore row stays MISSING, and R-3 stays DRAFT. Floor raise = Viktor's click, BEFORE the +> acceptance run. Still open: the `00-capability-map.md:61` ruling — did CAMPAIGN-6D's "immich +> end-to-end from offsite alone" exercise the DB half, or only the file half? > **2026-07-19 — v0.147.0 → v0.147.3: feedback slice 1.** The systemic complaint, twice in one > evening: you press a button and nothing happens. Three worst offenders fixed on the two patterns diff --git a/REPORT.md b/REPORT.md index a452a4f..c927b80 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,170 +1,85 @@ -# REPORT — controller v0.147.0 → v0.147.3: feedback slice 1 (2026-07-19) +# REPORT — v0.148.0: coherent snapshot pairs + a real offsite restore (R-43 + R-44) -**Baseline v0.146.0 → LIVE v0.147.3 on demo guest 9201.** Green at every version: -`go build ./... && go vet ./... && go test ./...` (23 packages) plus `template_id_gate`, -`emoji_gate`, `native_confirm_gate`, `offbox_rename_gate`, `app_row_dedup_gate`, `mojibake_gate`. +**Date:** 2026-07-19 · **Repo:** felhom-controller (v0.147.3 → **v0.148.0**) · Trunk, pushed to `main`. -> `docker_run_volume_path_gate` fails on `internal/appexport/estimate.go:179`. **Pre-existing on -> HEAD and untouched by this release** — verified by stashing this work and re-running the gate. -> Not fixed here (out of scope, and it is someone's call whether that `-v :ro` du probe gets an -> allowlist entry or a rewrite). +Closes the two findings from `felhom.eu/documentation/audits/DIAG-immich-restore-2026-07-19.md`. -## The problem +## What was broken -The systemic complaint, twice in one evening: **you press a button and nothing happens.** No -progress, no ETA, no named result. Three worst offenders fixed on the two patterns already in the -codebase — the deploy 3-step panel and the storage-init status poll. **No new framework**; that is -now ROADMAP **R-45**, because three targeted cards were worth shipping tonight and a framework was -not. +Viktor deleted 11 immich photos to test offsite restore. Both restore runs flashed success; the +photos stayed gone. Two independent defects: -Three of the four versions below exist because the cards were watched against **real runs on the -demo box**. Each follow-up is a thing no unit test would have surfaced. +- **R-43** — no offsite path could restore a database. All three buttons were file-only, so for a + DB-indexed app (most of the catalog) the bytes came back and the app still could not see them. +- **R-44** — a manual push shipped whatever dump the 02:30 local run had left. That day's dump + predated the customer's account by four hours and contained `asset: 0 / user: 0 / album: 0`, in a + 52 MB file whose bulk was immich's shipped geodata tables. ---- +## What shipped -## 4a — a verification restore names its result +**P1 — coherence pre-phase.** Every offsite run (manual AND nightly) refreshes dumps + recovery +units before capturing. Each manifest carries `offsite_run_id` + `dumps_at`. The periodic refresh +carries a prior stamp forward and never invents one. A dump failure is a loud WARN that does not +abort the push (data-first). New `dump` progress phase. -The completion flash said the app had been restored „ellenőrző mappába a meghajtón". *Which* folder, -on *which* drive, was invisible — so the customer could not go and look at the thing they had just -asked for. And nothing anywhere listed what these restores had accumulated, so copies piled up and -the only way to find them was SSH. +**P2 — honesty surfaces.** `ValidateDump` gained an exact-match accounts-table sniff +(`LooksEmpty`); the confirm shows dump age, an unstamped-pair warning and the empty-dump warning. +All warn-level, none a gate. -- The flash now carries the **full path**. -- The restore page gained **„Meglévő ellenőrző másolatok"** — app · size · date · path — each with a - double-confirmed **„Másolat törlése"** (the house inline `felhomConfirm` idiom, never native - `confirm()`). -- `backups/offsite-restore` was open-coded in three places; it now has one home - (`offsiteRestoreRootFor`). A test pins the path in the flash to the path in the listing, so the - customer can never be told about a directory the page cannot show or remove. +**P3 — `ReconstituteFromOffsite`** (`/backup/offbox/reconstitute`): safety dump → stop → files +overwritten → start → snapshot's dump replayed → health wait. Outcome-shaped flash. Two invariants: +nothing is ever deleted (`-a`, no `--ignore-existing`, no `--delete`), and the `pre-restore-` undo +is verified on disk before anything is touched — otherwise refuse with zero changes. -**The delete is the only one this release adds**, so it names a **stack, never a path**: the Manager -resolves that name inside a root it computed itself and refuses anything landing outside. -Red-proofed — neutralise the name guard and `stack: ""` resolves to the offsite-restore **root** and -`os.RemoveAll` takes every copy with it. Every refusal is asserted as a **non-effect** (the -neighbouring copy and the live data are still on disk afterwards). +**P4 — tests.** 11 new; full suite green: `go build ./... && go vet ./... && go test ./...`. +Design gates pass: template_id, emoji, native_confirm, offbox_rename, app_row_dedup, mojibake. -Honest note: the `HasPrefix` containment check behind that name guard could **not** be red-proofed -independently — with `isSafeStackName` in front of it, no input the API accepts can reach it with an -escaping path. It is kept as defence-in-depth and labelled in the test file as unproven-by-test -rather than dressed up as a red-proof it does not have. +> `docker_run_volume_path_gate` still fails on `internal/appexport/estimate.go:179`. **Pre-existing +> on HEAD and untouched by this release** — same finding as the v0.147.x report, same file, not a +> path this work goes near. -### Live (endpoint-exact, through the real handlers) +## Red-proofs — 5 run and reverted -Real verification restore of `calibre-web`, then the guard matrix: +| Mutation | Result | +|---|---| +| Replay removed | `0 DB dump(s) replayed` → FAIL | +| Capture moved before the dump | `got [capture dump]` → FAIL | +| Both undo guards removed | no refusal → FAIL | +| Substring table matching | join table mistaken for accounts; `UserRows = 4, want 1` → FAIL | +| Buffer-exceeding rows uncounted | wide row sniffs as empty → FAIL | -``` -message: "A(z) calibre-web visszaállítva ellenőrző mappába: - /mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web - (a meglévő adatok változatlanok)." -listing: calibre-web · 430.4 KB · …/offsite-restore/calibre-web · [Másolat törlése] +**Two of these found real test weaknesses rather than confirming strength**, and both tests were +rewritten: -stack=../primary -> refused; copy AND backups/primary both still present -missing confirm -> refused; nothing changed -real delete -> copy gone; backups/primary SURVIVES; page -> „Nincs ellenőrző másolat" -``` +- The first undo mutation still refused — a second guard (`os.Stat` on the empty path) held. The + red-proof only goes red when BOTH guards are removed, which is now recorded in REUSE.md. +- The first table-matching test did not discriminate between the two matchers at all: its aggregate + row count was identical either way. Rewritten to the case that separates them — an app with no + accounts table but with `user_metadata` / `album_user` / `user_audit`, where a substring match + would shout "your backup looks empty" on every restore of a healthy app. ---- +## Deliberately NOT done -## 4b — Megosztás enable shows what it is waiting for +Per §10 of the task: no catalog-wide invariant check (stays R-41), no changes to nightly cadence, +retention, quota math or tier-2, and no touching v0.147.x progress semantics beyond the one added +phase line. The missing-only button's own zero-file flash is untouched — that belongs to the v0.147 +feedback arc, not R-43/R-44. Its label now states what it does *not* do, which is in scope. -Enabling sharing ran `ReconcileSamba()` **synchronously inside the POST handler**. On a box whose -golden had not baked `felhom-samba` that is `compose up -d` pulling ~100MB from a private registry: -minutes of an apparently-hung form post, then „Beállítás mentve." whether or not anything came up. +## Live acceptance — NOT YET DONE (§9, needs Viktor) -Detached + polled now, with a card that distinguishes **„képfájl letöltése"** (image genuinely -absent — the multi-minute case) from **„indítás"** (already baked — seconds). The distinction is -decided *before* the work starts, because afterwards the image is always present and the card could -never truthfully say „letöltés". +The genuine red-proof is the joint run over the tunnel and has not happened: upload → „Távoli mentés +most" → empty the trash for real → „Teljes visszaállítás (fájlok + adatbázis)" → photos visible in +the immich timeline. Until that evidence exists: -- Success is **probed, not inferred** — `compose up -d` exits 0 on a crash-loop, so the terminal - state is container liveness. -- `nil` from reconcile also covers "deliberately deployed nothing because there is no household - password yet"; that now gets its own message instead of a card spinning forever. -- The **password** form starts the same job — with `UserSet` false reconcile deploys nothing, so on a - fresh box *that*, not the enable toggle, is where the pull actually happens. +- the capability-map offsite row stays **PARTIAL / scope-contested** (no PROVEN-LIVE flip), +- the customer-restore row stays **MISSING**, +- R-3 stays **DRAFT**. -**Test scope, stated honestly:** the unit tests cover the single-flight slot, snapshot isolation and -the phase vocabulary. They do **not** drive `runSambaEnsureJob` end-to-end — the Server field is the -concrete `*stacks.Manager`, not an interface, and introducing one purely for this card was more churn -than the slice warranted. Pulling-vs-starting and the probed terminal state are covered by live -validation instead. That seam belongs to R-45. +**Floor raise to v0.148.0 is Viktor's click and must come BEFORE the acceptance run.** -Live: `/sharing/status` returns `{"ok":true,"data":{"phase":"running","running":true,"error":""}}`; -the page carries the card and the poll. +## Open question for Viktor ---- - -## 4c — „Távoli mentés most" streams real progress - -restic was already reporting bytes and percentages; the runner seam used `CombinedOutput()` and threw -them away. The manual run now passes `--json`, scans stdout line-by-line, and the page shows total -bytes, percent and the app being pushed. **Manual only** — the nightly stays silent, pinned by a test -that fails if the scheduled path ever passes `--json` or publishes progress. - -The poll also now **arms unconditionally**. It used to start only if the page already rendered -„Fut…", which loses a race the manual trigger always runs: the POST redirects and the page renders -before the detached goroutine writes `LastStatus=running`, so the poll never armed and the customer -watched a static page during the very run they had just started. - -Red-proofed twice: break the parser -> the percent assertion fails; drop the wiring -> the `--json` -assertion fails. - -### What three live runs actually taught (v0.147.1 -> .3) - -**Run 1 — immich reported `0%` for 40+ seconds.** The parser was not broken. On an incremental run -where nothing changed restic transfers nothing: `bytes_done` is `omitempty`, so it is not even in the -JSON, and `percent_done` stays 0. Confirmed by capturing the real schema from -`backup --dry-run --json` on restic 0.14.0 in the controller image rather than guessing; those lines -are now quoted verbatim in the type's doc comment. **A byte-only bar is indistinguishable from a hang -in the common case** — shipping it would have traded "no feedback" for "feedback that says 0% and -looks stuck". -> **v0.147.1** parses and publishes `files_done`/`total_files`. - -**Run 2 — bookstack showed clean byte progress (100%, 154.0 MB, 7/7), immich sat at `1 of 46 files, -0 bytes` for 42 seconds.** restic 0.14 only counts a file when it **completes**, so an app dominated -by one large archive (immich's ~430MB volume tar) freezes *both* counters. No percentage can move. --> **v0.147.2** shows what restic *does* keep reporting: the file being processed and elapsed -seconds. „1 / 46 fájl (430.2 MB) · feldolgozás alatt: immich_upload.tar · 42 mp" is a different -message from „0%". - -**Run 3 — the per-app legs finished in ~15s; the remaining 40 of a 57-second run was the shares leg -and `forget --prune`,** during which the card sat frozen on „calibre-web — 8 / 8 fájl". The same -silence, relocated to the end of the run. -> **v0.147.3** adds a run **phase**; the post-app stages -announce themselves and app-scoped counters clear when a phase starts. - -Final live trace — the whole 57-second run is narrated, no frozen state: - -``` -t+6s app=immich -t+9s app=bookstack -t+12s app=calibre-web -t+15s phase=shares -t+18s ... t+54s phase=retention -t+57s done -``` - ---- - -## Also: golden/controller infra-image drift closed at the source - -`infra.Images()` derives the list from the existing pins; `--print-infra-images` prints it; the golden -bake now asks the controller binary it is about to bake instead of carrying its own copy. That copy -had already drifted — `felhom-samba` was never added to it, so the golden baked 3 of 4, **which is -why enabling Megosztás pulled at runtime in the first place**. A test parses the const block out of -the source with `go/ast` and fails if a pin is added without reaching `Images()`; red-proofed. - -Verified against the real published image on the live box — all four returned. - -## Validation method - -Live validation was **endpoint-exact** (the exact URLs the UI posts to, through the real handlers and -the real restic/samba paths), not browser click-through: the controller UI is password-gated and I do -not enter credentials into web forms. No server logic was skipped — only rendering. The rendered -markup was checked by fetching the pages themselves and asserting on the emitted HTML. - -## Not done - -- **Floor not raised** — Viktor decides floor timing. -- No golden rebuild, no manifest saves, nothing on Peti's arc. -- No SMB server-side changes — the `\\FELHOM` access issue stays client-side diagnosis first. -- ROADMAP: **R-45** (unified async-job feedback) and **R-46** (verification-copy browse + expiry) - registered in `felhom.eu/documentation/backlog/ROADMAP.md`. +The capability-map row at `00-capability-map.md:61` claims PROVEN-LIVE citing `CAMPAIGN-6D`'s +"immich end-to-end from offsite alone". This diagnosis contradicts that phrase. I flagged it +scope-contested rather than downgrading a proven row — the ruling (did 6D exercise the DB half, or +only the file half?) is still outstanding and is not something this implementation can settle. diff --git a/REUSE.md b/REUSE.md index ef6eb6d..66f39c0 100644 --- a/REUSE.md +++ b/REUSE.md @@ -205,6 +205,12 @@ | `Manager.sambaUpFn` / `sambaPasswdFn` / `sambaRunFn` (func seams) | controller/internal/stacks/manager.go (fields) + samba.go | nil → `composeUp` / `docker exec smbpasswd` (STDIN) / `containerRunning("felhom-samba")` | injected in controller/internal/stacks/samba_test.go — the idempotency test asserts the up-seam is called **zero** times when config is unchanged; the passwd seam means no unit test ever handles a real secret or touches docker | | `Manager.sambaImgFn` (func seam) | controller/internal/stacks/manager.go (field) + samba.go | nil → `docker image inspect ` | drives the 4b card's pulling-vs-starting decision, which MUST be taken before `compose up` (afterwards the image is always present) | | `Manager.offboxStreamRunner` + `SetOffboxStreamRunner` | controller/internal/backup/offbox_progress.go | nil → `defaultOffboxStreamRunner` (real `restic`, stdout scanned live) | streaming sibling of `offboxRunner`; fakes emit canned `--json` status lines in offbox_progress_test.go, so the whole progress path runs with no restic, network or repo | +| `Manager.offsitePreDumpFn` + `SetOffsitePreDumpFn` (R-44, v0.148.0) | controller/internal/backup/offbox_reconstitute.go (seam) + offbox.go (call site) | nil → `runDBDumpsInternal` under the SAME running flag | THE dumps-before-capture ordering seam. Extracted so the order is observable without Docker/restic — an ordering guarantee no test can see is one refactor from silently reverting to the DIAG-immich-restore-2026-07-19 behaviour. Red-proof: moving the capture first yields `[capture dump]` | +| `Manager.offboxFullPlaceCopier` + `SetOffboxFullPlaceCopier` (R-43) | controller/internal/backup/offbox_reconstitute.go | nil → `rsyncRestoreOverwrite` (`-a --itemize-changes`; **no** `--ignore-existing`, **no** `--delete`) | **TRAP: do NOT reuse `offboxPlaceCopier` here.** The two copiers have OPPOSITE semantics for an existing file — `--ignore-existing` is exactly what a full restore must not do, and conflating them is how a missing-only merge came to be labelled a restore. Never `rsyncMirror` (`--delete`) in any restore direction | +| `Manager.safetyDumpFn` + `SetSafetyDumpFn` (R-43) | controller/internal/backup/offbox_reconstitute.go | nil → `DumpOne` | the pre-restore undo. Invariant: the `pre-restore-`-prefixed dump must be verified ON DISK before anything is stopped/overwritten/replayed; failure ⇒ refuse with zero changes. Red-proof requires removing BOTH guards (the `err != nil` return and the `os.Stat`) — removing one leaves the other holding | +| `reimportDBDumpsFrom(ctx, stack, dumpDir)` | controller/internal/backup/restore_db.go | explicit-dir sibling of `reimportDBDumps` (which passes `AppDBDumpPath`) | offsite reconstitution replays from the SCRATCH unit: the live unit is deliberately never overwritten, so replaying from it would replay the current DB over itself and restore nothing | +| `Manager.OffsiteScratchPair` / `OffsitePairInfo` | controller/internal/backup/offbox_reconstitute.go | reads the restored scratch unit's manifest (`offsite_run_id` / `dumps_at`) + the R-44 sniff | the confirm-dialog honesty surface. All warn-level: a pre-v0.148 (unstamped) pair and an empty-looking dump are SURFACED, never blocked — a false positive that refused a legitimate restore would be worse than the skew | +| `appbackup.DumpValidation.LooksEmpty` (R-44 sniff) | controller/internal/appbackup/dbdump.go | computed in ValidateDump's existing single pass; `userTableNames` is EXACT-match | size and table count are both useless as emptiness heuristics (the 2026-07-19 dump: 52MB, 60+ tables, zero users — all geodata). **TRAP: never widen to a substring match on "user"** — it would flag `user_metadata` / `album_user` / `user_audit` on every healthy single-user box. A row wider than the read buffer still counts as a row | | `report.SetPendingControllerLog` / `SetControllerLogSource` | controller/internal/report/selftail.go | ACK-armed consume-once self-log pull (the logtail.go shape) | selftail_test.go; source = `logBuffer.Lines`, wired once in main.go | | `util.ParseVersion` / `util.Version.Compare` | controller/internal/util/version.go | THE one semver comparator (house rule: never a second) — selfupdate aliases it; agentapi's MinAgent comparison uses it | rejects pre-release/dev/latest (callers fall back, never trust); numeric compare (0.100 > 0.81) | | `agentapi.AgentVersionReporter` + `featureMinAgent` | controller/internal/agentapi/features.go | version-first Supports (v0.82.0 header channel); probe = fallback for header-less agents | a coupled feature adds BOTH a featureProbes row AND a featureMinAgent row; v0.116.0: `SupportsWithSource` also reports HOW the verdict was reached (version/probe-cache/probe) for the gate log line | diff --git a/controller/README.md b/controller/README.md index 3c34d3b..15951c7 100644 --- a/controller/README.md +++ b/controller/README.md @@ -295,6 +295,26 @@ Each app can define rich metadata in `.felhom.yml`: to a data drive off the rootfs (F-A1) behind a headroom gate; unit-only default via `--include` the absolute unit path; `PlaceOffsiteRestore` merges a full scratch into live via `rsync --ignore-existing` (never `--delete`), refusing on the pure `mapOffsiteRestorePaths` guards. + It restores FILES ONLY — no database, no stack restart — and the UI now says so. + - **Coherent snapshot pairs (v0.148.0, R-44):** every offsite run — manual AND nightly — refreshes + the DB/volume dumps and recovery units (`offsitePreDump` → `runDBDumpsInternal`) BEFORE the + restic capture, so each snapshot is an internally coherent `{DB@T, files@T}` bundle and + retention is a history of restorable points. Order is the mechanism: the gap can only add files + the DB does not reference yet, never remove one it does. Each manifest carries `offsite_run_id` + + `dumps_at`; a manifest without them is a pre-v0.148 pair of unknown skew, surfaced at restore + time. The periodic refresh carries the prior stamp forward and never invents one. A dump-leg + failure is a loud WARN that does NOT abort the push (data-first: a degraded backup beats none). + - **Offsite reconstitution (v0.148.0, R-43 — `offbox_reconstitute.go`):** the leg that was missing. + `ReconstituteFromOffsite` (`/backup/offbox/reconstitute`, „Teljes visszaállítás (fájlok + + adatbázis)") makes the live app equal to the chosen snapshot: **safety dump → stop → files + overwritten (`rsyncRestoreOverwrite`: no `--ignore-existing`, no `--delete`) → start → the + snapshot's dump replayed (`reimportDBDumpsFrom`, reading the SCRATCH unit) → health wait**. + Two invariants: nothing is ever deleted (post-snapshot files survive as extras), and the + `pre-restore-` safety dump is verified on disk BEFORE anything is stopped or overwritten — if it + cannot be taken the operation refuses with zero changes. Safety dumps appear in `ListDumpFiles` + (they are the undo). The live recovery unit is still never overwritten, which is why the replay + source is the scratch. Honesty surfaces (`OffsiteScratchPair`): dump age, an unstamped-pair + warning, and the R-44 empty-dump sniff — all warn-level, none of them gates. The `/apps/{slug}` page renders hero section, screenshots, setup guide, and optional config form. diff --git a/controller/internal/appbackup/dbdump.go b/controller/internal/appbackup/dbdump.go index 3d3b782..9fbf6ec 100644 --- a/controller/internal/appbackup/dbdump.go +++ b/controller/internal/appbackup/dbdump.go @@ -51,6 +51,20 @@ type DumpValidation struct { Error string FileSize int64 ModTime time.Time + // R-44 (v0.148.0) content sniff — a WARN-LEVEL signal, never a gate. + // + // Structural validity says nothing about whether a dump holds the customer's data. The immich + // dump of 2026-07-19 was 52MB, had a valid header and 60+ CREATE TABLEs, and contained zero + // users and zero assets: its whole bulk was the geodata reference tables immich ships. Size and + // table count are therefore both useless as emptiness heuristics — but an accounts table with + // no rows is a strong, cheap, app-agnostic hint that a dump predates the customer entirely. + // + // Deliberately NOT a refusal: plenty of legitimate apps have no users table (UserTableFound + // false → inconclusive → silent), and a false positive that blocked a restore would be far + // worse than the skew it guards against. The restore confirm shows it as one extra line. + UserTableFound bool + UserRows int + LooksEmpty bool // UserTableFound && UserRows == 0 } // DumpFileInfo holds info about a dump file on disk. @@ -363,6 +377,9 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation { lineNum := 0 headerFound := false tableCount := 0 + // R-44 sniff state. inUserCopy tracks a postgres `COPY … FROM stdin;` block for an accounts + // table; rows are counted until the `\.` terminator. + inUserCopy := false for { lineBytes, isPrefix, err := reader.ReadLine() if err != nil { @@ -374,7 +391,12 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation { break // EOF } if isPrefix { - // Line exceeds buffer — skip remainder (COPY data, large INSERTs) + // Line exceeds buffer — skip remainder (COPY data, large INSERTs). + // A long line inside a user COPY block is still a ROW: count it before discarding it, + // or a table whose rows happen to be wide would sniff as empty and raise a false alarm. + if inUserCopy { + v.UserRows++ + } for isPrefix && err == nil { _, isPrefix, err = reader.ReadLine() } @@ -384,6 +406,23 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation { line := string(lineBytes) lineNum++ + // R-44 content sniff (warn-level; see DumpValidation). + if inUserCopy { + if line == `\.` { + inUserCopy = false + } else { + v.UserRows++ + } + } else if isUserCopyStart(line, dbType) { + inUserCopy = true + v.UserTableFound = true + } else if dbType == DBTypeMariaDB && isUserInsert(line) { + // mysqldump writes multi-row `INSERT INTO \`users\` VALUES (…),(…);` — the row count is + // not worth parsing out of it, and presence alone answers the only question asked here. + v.UserTableFound = true + v.UserRows++ + } + // Header check — scan first 10 lines for expected dump header // MariaDB 11.4+ prepends a sandbox comment before the header line if lineNum <= 10 && !headerFound { @@ -427,10 +466,65 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation { return v } + v.LooksEmpty = v.UserTableFound && v.UserRows == 0 + if v.LooksEmpty { + log.Printf("[WARN] [backup] ValidateDump: %s is structurally valid (%d tables) but its accounts table has NO rows — the dump may predate the customer's data", filePath, tableCount) + } + v.Valid = true return v } +// userTableNames are the table names treated as "the accounts table" by the R-44 sniff. Kept +// deliberately short: a wider net (anything containing "user") would match join/audit tables like +// `user_metadata` or `album_user`, which are legitimately empty on a healthy single-user install +// and would produce exactly the false alarm this signal must not raise. +var userTableNames = []string{"user", "users", "account", "accounts"} + +// isUserCopyStart reports whether a line opens a postgres `COPY … FROM stdin;` +// block. pg_dump writes the table qualified and optionally quoted — `COPY public."user" (…)`, +// `COPY public.users (…)` — so both forms are matched. +func isUserCopyStart(line string, dbType DBType) bool { + if dbType != DBTypePostgres || !strings.HasPrefix(line, "COPY ") { + return false + } + rest := strings.TrimPrefix(line, "COPY ") + sp := strings.IndexByte(rest, ' ') + if sp < 0 { + return false + } + return matchesUserTable(rest[:sp]) +} + +// isUserInsert reports whether a line is a mysqldump INSERT into an accounts table. +func isUserInsert(line string) bool { + const pfx = "INSERT INTO " + if !strings.HasPrefix(line, pfx) { + return false + } + rest := strings.TrimPrefix(line, pfx) + sp := strings.IndexByte(rest, ' ') + if sp < 0 { + return false + } + return matchesUserTable(rest[:sp]) +} + +// matchesUserTable strips schema qualification and quoting from a dumped table reference and +// reports whether the bare name is an accounts table. +func matchesUserTable(ref string) bool { + if dot := strings.LastIndexByte(ref, '.'); dot >= 0 { + ref = ref[dot+1:] + } + ref = strings.Trim(ref, "\"`") + for _, n := range userTableNames { + if strings.EqualFold(ref, n) { + return true + } + } + return false +} + // ListDumpFiles returns info about SQL dump files on disk. // // M18: ValidateDump scans the dump line-by-line; on a customer with hundreds-of-MB dumps that is wasted diff --git a/controller/internal/appbackup/dbdump_sniff_test.go b/controller/internal/appbackup/dbdump_sniff_test.go new file mode 100644 index 0000000..1d1a208 --- /dev/null +++ b/controller/internal/appbackup/dbdump_sniff_test.go @@ -0,0 +1,141 @@ +package appbackup + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// R-44 content-sniff tests. +// +// The dump that triggered this work (DIAG-immich-restore-2026-07-19) was 52MB, had a valid +// PostgreSQL header and 60+ CREATE TABLE statements, and contained zero users and zero assets — +// its entire bulk was immich's shipped geodata reference tables. Both of the signals the product +// already had (file size, table count) called it healthy. These tests pin the one signal that +// would have caught it, and the boundaries that keep it from crying wolf. + +func writeDump(t *testing.T, body string) string { + t.Helper() + p := filepath.Join(t.TempDir(), "d.sql") + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + return p +} + +const pgHead = `-- PostgreSQL database dump +-- Dumped from database version 16.10 +SET statement_timeout = 0; +SET client_encoding = 'UTF8'; +CREATE TABLE public.asset (id uuid NOT NULL); +CREATE TABLE public."user" (id uuid NOT NULL, email text); +` + +// TestSniffFlagsEmptyAccountsTable is the 2026-07-19 shape: structurally perfect, no customer. +func TestSniffFlagsEmptyAccountsTable(t *testing.T) { + body := pgHead + "COPY public.\"user\" (id, email) FROM stdin;\n\\.\n" + + "COPY public.asset (id) FROM stdin;\n\\.\n" + v := ValidateDump(writeDump(t, body), DBTypePostgres) + if !v.Valid { + t.Fatalf("the dump is structurally valid; sniff must not change that: %s", v.Error) + } + if !v.UserTableFound { + t.Fatal("the accounts table COPY block was not recognised") + } + if v.UserRows != 0 { + t.Fatalf("UserRows = %d, want 0", v.UserRows) + } + if !v.LooksEmpty { + t.Fatal("a valid dump with zero account rows MUST raise the warn signal — this is the whole point of R-44") + } +} + +// TestSniffQuietOnPopulatedDump — the common case must stay silent, or the warning becomes noise +// and gets ignored precisely when it matters. +func TestSniffQuietOnPopulatedDump(t *testing.T) { + body := pgHead + "COPY public.\"user\" (id, email) FROM stdin;\n" + + "a\tone@example.invalid\nb\ttwo@example.invalid\n\\.\n" + v := ValidateDump(writeDump(t, body), DBTypePostgres) + if v.UserRows != 2 { + t.Fatalf("UserRows = %d, want 2", v.UserRows) + } + if v.LooksEmpty { + t.Fatal("a dump with account rows must not be flagged") + } +} + +// TestSniffInconclusiveWithoutAccountsTable — plenty of legitimate apps have no users table. No +// table, no claim: a false positive here would warn on every restore of such an app forever. +func TestSniffInconclusiveWithoutAccountsTable(t *testing.T) { + body := "-- PostgreSQL database dump\nCREATE TABLE public.thing (id int);\n" + + "COPY public.thing (id) FROM stdin;\n\\.\n" + strings.Repeat("-- pad\n", 20) + v := ValidateDump(writeDump(t, body), DBTypePostgres) + if v.UserTableFound { + t.Fatal("no accounts table exists — none must be reported") + } + if v.LooksEmpty { + t.Fatal("an app without an accounts table must be INCONCLUSIVE, never flagged empty") + } +} + +// TestSniffIgnoresJoinAndAuditTables is the false-alarm guard that shaped the name list, and it is +// written as the case that DISCRIMINATES: an app with NO accounts table but with `user_metadata` / +// `album_user` / `user_audit` — all legitimately empty on a healthy box. Exact-matching leaves this +// inconclusive (silent, correct). A substring match on "user" would treat a join table as the +// accounts table, find zero rows, and shout "your backup looks empty" on every single restore of a +// perfectly healthy app — which is how a warning signal becomes noise and then gets ignored. +func TestSniffIgnoresJoinAndAuditTables(t *testing.T) { + body := "-- PostgreSQL database dump\nCREATE TABLE public.album (id int);\n" + + "COPY public.user_metadata (id) FROM stdin;\n\\.\n" + + "COPY public.album_user (id) FROM stdin;\n\\.\n" + + "COPY public.user_audit (id) FROM stdin;\n\\.\n" + + "COPY public.album (id) FROM stdin;\n1\n\\.\n" + v := ValidateDump(writeDump(t, body), DBTypePostgres) + if v.UserTableFound { + t.Fatal("a join/audit table must never be mistaken for the accounts table") + } + if v.LooksEmpty { + t.Fatal("empty join/audit tables must not trigger the warning — this app has no accounts table at all") + } +} + +// TestSniffCountsOnlyTheAccountsTable pins the counting boundary separately: with a real accounts +// table present, rows from neighbouring user-ish tables must not inflate it. +func TestSniffCountsOnlyTheAccountsTable(t *testing.T) { + body := pgHead + + "COPY public.user_metadata (id) FROM stdin;\nm1\nm2\nm3\n\\.\n" + + "COPY public.\"user\" (id, email) FROM stdin;\na\tone@example.invalid\n\\.\n" + v := ValidateDump(writeDump(t, body), DBTypePostgres) + if v.UserRows != 1 { + t.Fatalf("only the real accounts table may be counted; UserRows = %d, want 1", v.UserRows) + } +} + +// TestSniffCountsWideRows — a row wider than the read buffer is skipped by the structural scan, but +// it is still a row. Counting it wrong would flag a populated table as empty (immich asset rows are +// genuinely long, which is what makes this reachable). +func TestSniffCountsWideRows(t *testing.T) { + wide := strings.Repeat("x", 300*1024) + body := pgHead + "COPY public.\"user\" (id, email) FROM stdin;\n" + wide + "\n\\.\n" + v := ValidateDump(writeDump(t, body), DBTypePostgres) + if v.UserRows != 1 { + t.Fatalf("a buffer-exceeding row must still count; UserRows = %d, want 1", v.UserRows) + } + if v.LooksEmpty { + t.Fatal("a table whose single row is very wide must not sniff as empty") + } +} + +// TestSniffMariaDBInsertForm — mysqldump writes multi-row INSERTs, not COPY blocks. +func TestSniffMariaDBInsertForm(t *testing.T) { + head := "-- MariaDB dump 10.19\nCREATE TABLE `users` (id int);\n" + strings.Repeat("-- pad\n", 20) + empty := ValidateDump(writeDump(t, head), DBTypeMariaDB) + if empty.UserTableFound { + t.Fatal("a CREATE TABLE alone is not an accounts-table row source") + } + full := ValidateDump(writeDump(t, head+"INSERT INTO `users` VALUES (1),(2);\n"), DBTypeMariaDB) + if !full.UserTableFound || full.LooksEmpty { + t.Fatalf("a populated mariadb dump must not be flagged: %+v", full) + } +} diff --git a/controller/internal/backup/backup.go b/controller/internal/backup/backup.go index c2e440b..6e1b611 100644 --- a/controller/internal/backup/backup.go +++ b/controller/internal/backup/backup.go @@ -59,6 +59,16 @@ type Manager struct { // offboxPlaceCopier (3a) — the place-to-live missing-only merge seam (nil → rsyncRestoreMissing, // the `-a --ignore-existing` additive copy). Never rsyncMirror (--delete trap). offboxPlaceCopier func(src, dst string) (int, error) + // offboxFullPlaceCopier (R-43, v0.148.0) — the FULL-restore overwrite seam (nil → + // rsyncRestoreOverwrite: `-a` with NO --ignore-existing and NO --delete). Distinct from + // offboxPlaceCopier on purpose: the two have opposite semantics for an existing file. + offboxFullPlaceCopier func(src, dst string) (int, error) + // safetyDumpFn (R-43) — the pre-restore safety-dump seam (nil → the real DumpOne), so the + // "never replay without an undo on disk" refusal is unit-testable without Docker. + safetyDumpFn func(ctx context.Context, db DiscoveredDB, dumpDir string) DumpResult + // offsitePreDumpFn (R-44) — the offsite dump pre-phase seam (nil → runDBDumpsInternal), so the + // dumps-strictly-before-capture ordering is observable in a test without Docker or restic. + offsitePreDumpFn func(ctx context.Context) error // offboxFreeFn (3a) — the free-space probe for the restore headroom gate, overridable in tests (the // Windows `go test` host has no `df`). Nil → the real diskFreeBytes (df --output=avail). offboxFreeFn func(path string) int64 @@ -126,6 +136,13 @@ type Manager struct { lastDBDump *DBDumpStatus running bool + // R-43/R-44 (v0.148.0) — the coherence stamp of the offsite run in flight, read by + // CaptureRecoveryUnit so each unit records WHICH run took the dumps sitting beside its files. + // Set for the duration of the dump pre-phase + capture, cleared after; "" means "no offsite run + // is establishing coherence right now" (the periodic refresh and the local 02:30 dump leg). + offsiteRunID string + offsiteRunDumpAt string + // Restore op-status (Part B, opstatus.go) — display-only async-restore progress, under `mu`. opRunning bool opName string @@ -310,6 +327,28 @@ func (m *Manager) RunDBDumps(ctx context.Context) error { return m.runDBDumpsInternal(ctx) } +// offsiteRunStamp returns the in-flight offsite run's coherence stamp ("" when none). +func (m *Manager) offsiteRunStamp() (runID, dumpsAt string) { + m.mu.Lock() + defer m.mu.Unlock() + return m.offsiteRunID, m.offsiteRunDumpAt +} + +// beginOffsiteRunStamp marks the start of an offsite run's coherence window and returns the cleanup. +// The stamp is what CaptureRecoveryUnit writes into each unit manifest, so it must be live across +// BOTH the dump leg and the unit capture that follows it — those two together are the pair. +func (m *Manager) beginOffsiteRunStamp(runID string) func() { + m.mu.Lock() + m.offsiteRunID = runID + m.offsiteRunDumpAt = time.Now().UTC().Format(time.RFC3339) + m.mu.Unlock() + return func() { + m.mu.Lock() + m.offsiteRunID, m.offsiteRunDumpAt = "", "" + m.mu.Unlock() + } +} + // runDBDumpsInternal is the implementation of RunDBDumps. Caller must hold the running flag. func (m *Manager) runDBDumpsInternal(ctx context.Context) error { start := time.Now() diff --git a/controller/internal/backup/offbox.go b/controller/internal/backup/offbox.go index 30fbdf9..6cd496f 100644 --- a/controller/internal/backup/offbox.go +++ b/controller/internal/backup/offbox.go @@ -658,9 +658,40 @@ func (m *Manager) runOffboxBackup(ctx context.Context, withProgress bool) error m.offboxRecordStats(ctx, base, env) // the prune may have brought the size back down — refresh runErr = fmt.Errorf("A távoli mentés túllépte a tárhelykeretet (%d/%d GB) — törölj régi mentéseket vagy kérj nagyobb keretet.", usedGB, quota) } else { + // R-43/R-44 (v0.148.0) — THE COHERENCE PRE-PHASE. Refresh the DB/volume dumps and the recovery + // units BEFORE capturing, so the snapshot restic is about to write is an internally coherent + // {DB@T, files@T} pair. Before this, a push shipped live files beside whatever dump the 02:30 + // local run happened to leave — on 2026-07-19 that was a dump taken four hours before the + // customer's account even existed, so the "backup" of the photos contained zero of them + // (DIAG-immich-restore-2026-07-19). + // + // Order matters and is the whole mechanism: dumps FIRST, then files. The gap between the two + // can only ADD files the DB does not reference yet (an upload landing mid-run is a harmless + // orphan blob), never remove one the DB DOES reference — so the file set is always a superset + // of what the restored DB points at. The reverse order would produce dangling rows. + // + // This runs on the NIGHTLY path too, not just the manual one: "every snapshot is a coherent + // pair" is the property that makes retention a history of restorable points rather than a + // history of skewed ones. It also makes the nightly ordering structural instead of a + // coincidence of two independent scheduler entries at 02:30 and 04:15. + endStamp := m.beginOffsiteRunStamp(start.UTC().Format("20060102T150405Z")) + if withProgress { + m.offboxProgress.setPhase(OffboxPhaseDump) + } + dumpStart := time.Now() + if dErr := m.offsitePreDump(ctx); dErr != nil { + // Data-first: a dump failure must NOT abort the push. The files are still worth shipping, + // and refusing to ship them would turn a degraded backup into no backup at all. It is a + // loud WARN, and the unit manifest simply carries the older dump set — which the restore + // confirm then surfaces as a skewed pair (P2) rather than silently pretending otherwise. + m.logger.Printf("[WARN] [offbox] pre-push dump leg failed (%v) — continuing with the existing dumps; the snapshot's DB half may be older than its files", dErr) + } else { + m.logger.Printf("[INFO] [offbox] pre-push dump leg completed in %s — snapshot pair is coherent", time.Since(dumpStart).Round(time.Millisecond)) + } runResult, runErr = m.runOffboxInternal(ctx, apps, base, env, t) backedUp = runResult.backedUp missing = runResult.missing + endStamp() } // Sorted names of apps whose enlargement was blocked this run (replaces the persisted set; empty clears). var blockedNames []string diff --git a/controller/internal/backup/offbox_progress.go b/controller/internal/backup/offbox_progress.go index d414224..4b6fcd9 100644 --- a/controller/internal/backup/offbox_progress.go +++ b/controller/internal/backup/offbox_progress.go @@ -179,8 +179,10 @@ func (p *offboxProgressState) setPhase(phase string) { p.mu.Unlock() } -// OffboxPhaseShares / OffboxPhaseRetention are the post-app-loop stages. +// OffboxPhaseDump is the PRE-app-loop stage (R-44, v0.148.0); OffboxPhaseShares / +// OffboxPhaseRetention are the post-app-loop stages. const ( + OffboxPhaseDump = "dump" OffboxPhaseShares = "shares" OffboxPhaseRetention = "retention" ) diff --git a/controller/internal/backup/offbox_reconstitute.go b/controller/internal/backup/offbox_reconstitute.go new file mode 100644 index 0000000..5882be3 --- /dev/null +++ b/controller/internal/backup/offbox_reconstitute.go @@ -0,0 +1,400 @@ +package backup + +import ( + "context" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "time" +) + +// Offsite reconstitution (R-43, v0.148.0) — the leg that was missing. +// +// Until v0.148.0 NO offsite path could restore a database. The two „visszaállítás" buttons staged +// files into a scratch folder and never touched postgres; the place-to-live button merged only the +// files MISSING from the live tree (`rsync --ignore-existing`) and never replayed a dump. For a +// DB-indexed app — most of the catalog — that combination cannot bring content back: the bytes +// return and the application still cannot see them, because its index lives in the database. +// Measured live on 2026-07-19 (DIAG-immich-restore-2026-07-19): 11 photos, files intact on disk, +// timeline empty, two "successful" restores that merged 0 files. +// +// ReconstituteFromOffsite is the honest version of that operation: it takes the CHOSEN snapshot's +// coherent pair and makes the live app equal to it — files overwritten to the snapshot's version, +// database replayed from the same snapshot's dump, app restarted. It is deliberately a different +// function from PlaceOffsiteRestore rather than a flag on it, because the two have opposite file +// semantics and conflating them is exactly how the missing-only merge came to be presented as a +// restore. +// +// Two invariants hold throughout: +// +// - NOTHING IS EVER DELETED. The file copy overwrites and adds; it never carries `--delete`. A +// file the customer created after the snapshot survives the restore as an extra. That is the +// house boundary — a restore that silently removed newer work would be a data-loss event +// wearing a recovery button's label. +// - THE UNDO EXISTS BEFORE THE ACT. A safety dump of the live database is written, and verified +// present on disk, BEFORE anything is stopped, overwritten or replayed. If that dump cannot be +// taken, the whole operation refuses with zero changes — a replay whose previous state was not +// captured is not a restore, it is an overwrite with no way back. + +// offsitePreDump runs the coherence pre-phase's dump leg (nil seam → runDBDumpsInternal, which also +// refreshes the recovery units so the manifests enumerate the dumps just written). Extracted as a +// seam because the ORDER — dumps strictly before the restic capture — is the entire mechanism of +// R-44, and an ordering guarantee that no test can observe is one refactor away from silently +// reverting to the behaviour that produced DIAG-immich-restore-2026-07-19. +func (m *Manager) offsitePreDump(ctx context.Context) error { + if m.offsitePreDumpFn != nil { + return m.offsitePreDumpFn(ctx) + } + return m.runDBDumpsInternal(ctx) +} + +// SetOffsitePreDumpFn overrides the offsite dump pre-phase (tests; no Docker needed). +func (m *Manager) SetOffsitePreDumpFn(fn func(ctx context.Context) error) { m.offsitePreDumpFn = fn } + +// preRestoreDumpPrefix marks the safety dumps taken immediately before a reconstitution. They live +// in the app's own unit db-dumps dir so `ListDumpFiles` surfaces them beside the regular dumps — +// they ARE the undo, and an undo the customer cannot see is not much of one. The regular replay +// loop matches `-.sql` exactly, so a prefixed file is never mistaken for a source. +const preRestoreDumpPrefix = "pre-restore-" + +// OffsiteReconstituteResult reports what a reconstitution actually did, so the flash can state an +// OUTCOME instead of a mechanism. Every field here exists because the v0.147 flash could not say it. +type OffsiteReconstituteResult struct { + SnapshotID string + FilesPlaced int + DBsReplayed int + SafetyDump string // path of the pre-restore dump (the undo), "" when the app has no DB + DumpsAt time.Time // when the snapshot's DB half was taken (zero = unknown/legacy unit) + OffsiteRunID string // "" for a pre-v0.148 snapshot — an unverified pair + Skewed bool // the snapshot carries no coherence stamp: files and DB may differ in age + LooksEmpty bool // R-44 sniff on the dump about to be replayed +} + +// fullPlaceCopier returns the FULL-restore file copier (nil seam → rsyncRestoreOverwrite). +// Deliberately NOT placeCopier(): that one is `--ignore-existing`, whose whole purpose is to leave +// live files alone, which is precisely what a full restore must not do. +func (m *Manager) fullPlaceCopier() func(src, dst string) (int, error) { + if m.offboxFullPlaceCopier != nil { + return m.offboxFullPlaceCopier + } + return rsyncRestoreOverwrite +} + +// rsyncRestoreOverwrite copies src over dst: `rsync -a --itemize-changes`, with NO +// `--ignore-existing` (a changed file becomes the snapshot's version) and NO `--delete` (an extra +// file at dst survives). Returns the number of regular files transferred. +func rsyncRestoreOverwrite(src, dst string) (int, error) { + if err := os.MkdirAll(dst, 0755); err != nil { + return 0, fmt.Errorf("mkdir %s: %w", dst, err) + } + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Minute) + defer cancel() + cmd := exec.CommandContext(ctx, "rsync", "-a", "--itemize-changes", + strings.TrimRight(src, "/")+"/", strings.TrimRight(dst, "/")+"/") + out, err := cmd.CombinedOutput() + if err != nil { + return 0, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out))) + } + return countRestoredFiles(string(out)), nil +} + +// writeSafetyDump dumps every live database of stack into the app's unit db-dumps dir under the +// `pre-restore-` prefix, and returns the first dump's path. Returns ("", nil) when the app has no +// database at all — a no-DB app has nothing to undo and must flow exactly as it did before +// v0.148.0 (no dump, no replay, no behaviour change). +// +// A discovered database that CANNOT be dumped is a hard error: it means the undo would not exist. +func (m *Manager) writeSafetyDump(ctx context.Context, stackName, nsRoot string) (string, error) { + discover := m.discoverDBs + if discover == nil { + discover = func(ctx context.Context) ([]DiscoveredDB, error) { + return DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames()) + } + } + dbs, err := discover(ctx) + if err != nil { + return "", fmt.Errorf("a biztonsági mentés előtt nem sikerült felderíteni az adatbázisokat: %w", err) + } + var mine []DiscoveredDB + for _, db := range dbs { + if db.StackName == stackName { + mine = append(mine, db) + } + } + if len(mine) == 0 { + return "", nil // no DB → nothing to undo → scenario E flows unchanged + } + + dumpDir := AppDBDumpPath(nsRoot, stackName) + if err := os.MkdirAll(dumpDir, 0755); err != nil { + return "", fmt.Errorf("a biztonsági mentés könyvtára nem hozható létre: %w", err) + } + stamp := time.Now().UTC().Format("20060102T150405Z") + first := "" + for _, db := range mine { + res := m.dumpForSafety(ctx, db, dumpDir) + if res.Error != nil { + return "", fmt.Errorf("a jelenlegi adatbázis biztonsági mentése sikertelen (%s): %w — a visszaállítás nem indult el", db.ContainerName, res.Error) + } + // DumpOne writes `-.sql`; rename it under the safety prefix so it can never be + // picked up as a replay SOURCE and can never overwrite the app's real dump. + safe := filepath.Join(dumpDir, fmt.Sprintf("%s%s-%s-%s.sql", preRestoreDumpPrefix, stamp, stackName, db.DBType)) + if res.FilePath != safe { + if err := os.Rename(res.FilePath, safe); err != nil { + return "", fmt.Errorf("a biztonsági mentés véglegesítése sikertelen: %w", err) + } + } + if first == "" { + first = safe + } + m.logger.Printf("[INFO] [offbox] %s: pre-restore safety dump written → %s (%s)", stackName, filepath.Base(safe), humanizeBytes(res.Size)) + } + return first, nil +} + +// dumpForSafety is the DumpOne seam for the safety dump (tests inject; nil → the real DumpOne). +func (m *Manager) dumpForSafety(ctx context.Context, db DiscoveredDB, dumpDir string) DumpResult { + if m.safetyDumpFn != nil { + return m.safetyDumpFn(ctx, db, dumpDir) + } + return DumpOne(ctx, db, dumpDir, m.logger, m.isDebug()) +} + +// ReconstituteFromOffsite makes the live app equal to a restored full-scratch snapshot: files +// overwritten to the snapshot's version (extras survive, nothing deleted), then the snapshot's own +// DB dump replayed, with a safety dump of the current database taken first. Requires a completed +// FULL scratch restore (RestoreOffboxScratch with full=true). Single-flight. +func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (OffsiteReconstituteResult, error) { + var res OffsiteReconstituteResult + if !m.OffboxConfigured() { + return res, fmt.Errorf("off-box backup not configured") + } + if !isSafeStackName(stack) { + return res, fmt.Errorf("invalid stack name") + } + if m.stackProvider == nil { + return res, fmt.Errorf("stack provider not configured") + } + if err := m.acquireRunning(); err != nil { + return res, fmt.Errorf("egy másik mentési/visszaállítási művelet már fut") + } + defer m.releaseRunning() + + scratch, _, err := m.offboxRestoreScratchDir(stack) + if err != nil { + return res, err + } + if _, sErr := os.Stat(scratch); sErr != nil { + return res, fmt.Errorf("nincs előkészített teljes visszaállítás — futtass előbb egy teljes visszaállítást") + } + id, paths, err := m.offboxLatestSnapshot(ctx, stack) + if err != nil { + return res, err + } + res.SnapshotID = id + + hdd := strings.TrimSpace(m.stackProvider.GetStackHDDPath(stack)) + if hdd == "" { + return res, fmt.Errorf("a(z) %s nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat", stack) + } + liveNs := m.namespaceRoot(hdd) + + placements, err := mapOffsiteRestorePaths(paths, stack, scratch, liveNs) + if err != nil { + return res, err // whole-placement refusal (no partial writes) + } + // Stat pre-pass over EVERY placement before the first copy — an incomplete scratch (e.g. only a + // unit-only restore was run) refuses with ZERO copies. + for _, pl := range placements { + if _, sErr := os.Stat(pl.src); sErr != nil { + return res, fmt.Errorf("a teljes visszaállítás hiányos (%s nincs meg) — futtass előbb egy teljes visszaállítást", filepath.Base(pl.src)) + } + } + + // The snapshot's coherence stamp, read from the RESTORED unit manifest (not the live one). + scratchUnit := "" + for _, pl := range placements { + if pl.isUnit { + scratchUnit = pl.src + break + } + } + if scratchUnit == "" { + return res, fmt.Errorf("a pillanatképben nincs mentési egység — a visszaállítás nem indítható") + } + scratchDumpDir := filepath.Join(scratchUnit, "db-dumps") + if man := readManifest(filepath.Join(scratchUnit, "manifest.json")); man != nil { + res.OffsiteRunID = man.OffsiteRunID + if man.DumpsAt != "" { + if t, pErr := time.Parse(time.RFC3339, man.DumpsAt); pErr == nil { + res.DumpsAt = t + } + } + } + // A pre-v0.148 snapshot carries no stamp: its dump was whatever the 02:30 local run left behind, + // so the pair's two halves may be hours or days apart. Surfaced, never blocked — the confirm + // dialog says so and the safety dump makes it reversible. + res.Skewed = res.OffsiteRunID == "" + res.LooksEmpty = m.sniffScratchDump(scratchDumpDir, stack) + + // --- THE UNDO, BEFORE THE ACT --------------------------------------------------------------- + // Taken while the stack is still UP (a stopped database cannot be dumped) and before a single + // byte is overwritten, so a failure here aborts with the live app completely untouched. + safety, err := m.writeSafetyDump(ctx, stack, liveNs) + if err != nil { + return res, err + } + res.SafetyDump = safety + hasDB := safety != "" + if hasDB { + if _, sErr := os.Stat(safety); sErr != nil { + // Fail-closed: never replay when the undo is not verifiably on disk. + return res, fmt.Errorf("a biztonsági mentés nem található a lemezen — a visszaállítás biztonsági okból nem indult el") + } + } + + // --- FILES ---------------------------------------------------------------------------------- + if err := m.stackProvider.StopStack(stack); err != nil { + m.logger.Printf("[WARN] [offbox] could not stop %s before reconstitution: %v (continuing)", stack, err) + } + copier := m.fullPlaceCopier() + for _, pl := range placements { + if pl.isUnit { + // The live recovery unit is still never overwritten — it is the LOCAL restore path's + // source and clobbering it would trade one recovery route for another. The snapshot's + // dump is replayed from the scratch unit instead, so nothing is lost by skipping it. + continue + } + n, cErr := copier(pl.src, pl.dst) + if cErr != nil { + // Best-effort bring-up: leaving the app stopped after a partial copy would turn a failed + // restore into an outage. + if sErr := m.stackProvider.StartStack(stack); sErr != nil { + m.logger.Printf("[WARN] [offbox] %s: restart after failed placement also failed: %v", stack, sErr) + } + return res, fmt.Errorf("a(z) %s fájljainak visszaállítása sikertelen: %w", stack, cErr) + } + res.FilesPlaced += n + } + + // --- DATABASE ------------------------------------------------------------------------------- + // The stack must be UP for the replay: ImportDump talks to the running container using its own + // discovered credentials (the same precedence RestoreFromRecoveryUnit uses — the logical dump + // wins over whatever the file copy just laid down for the DB's own data dir). + if err := m.stackProvider.StartStack(stack); err != nil { + return res, fmt.Errorf("a(z) %s újraindítása sikertelen a fájlok visszaállítása után: %w", stack, err) + } + if hasDB { + n, iErr := m.reimportDBDumpsFrom(ctx, stack, scratchDumpDir) + res.DBsReplayed = n + if iErr != nil { + return res, fmt.Errorf("az adatbázis visszaállítása sikertelen: %w — a korábbi állapot mentése megvan: %s", iErr, filepath.Base(safety)) + } + } + if err := m.waitForHealthy(stack, 90*time.Second); err != nil { + m.logger.Printf("[WARN] [offbox] %s reconstituted but health check failed: %v", stack, err) + } + + m.logger.Printf("[INFO] [offbox] reconstituted %s from snapshot %s: %d file(s) placed, %d DB dump(s) replayed, safety dump=%s, skewed=%v", + stack, id, res.FilesPlaced, res.DBsReplayed, filepath.Base(safety), res.Skewed) + return res, nil +} + +// OffsitePairInfo describes the {DB, files} pair sitting in a prepared full-restore scratch, so the +// confirm dialog can tell the customer what they are about to restore BEFORE they commit to it. +// Everything here is honesty-surface: none of it blocks the operation. +type OffsitePairInfo struct { + Ready bool + DumpsAt time.Time // when the DB half was taken (zero = legacy unit, age unknown) + Skewed bool // no coherence stamp → the two halves may be from different times + LooksEmpty bool // R-44 sniff: the dump has an accounts table with no rows + HasDump bool +} + +// OffsiteScratchPair reads the prepared scratch's unit manifest and reports what the pair looks +// like. Cheap and read-only — safe to call from a page render. +func (m *Manager) OffsiteScratchPair(stack string) OffsitePairInfo { + var info OffsitePairInfo + if !isSafeStackName(stack) { + return info + } + scratch, _, err := m.offboxRestoreScratchDir(stack) + if err != nil { + return info + } + // The unit sits at //backups/primary/; the old namespace is unknown here, + // so find it rather than reconstructing it. + unit := findScratchUnitDir(scratch, stack) + if unit == "" { + return info + } + info.Ready = true + dumpDir := filepath.Join(unit, "db-dumps") + if entries, rErr := os.ReadDir(dumpDir); rErr == nil { + for _, e := range entries { + if !e.IsDir() && filepath.Ext(e.Name()) == ".sql" && !strings.HasPrefix(e.Name(), preRestoreDumpPrefix) { + info.HasDump = true + break + } + } + } + if man := readManifest(filepath.Join(unit, "manifest.json")); man != nil { + if man.DumpsAt != "" { + if t, pErr := time.Parse(time.RFC3339, man.DumpsAt); pErr == nil { + info.DumpsAt = t + } + } + info.Skewed = man.OffsiteRunID == "" + } else { + info.Skewed = true + } + if info.HasDump { + info.LooksEmpty = m.sniffScratchDump(dumpDir, stack) + } + return info +} + +// findScratchUnitDir locates `backups/primary/` anywhere under a restored scratch. restic +// rebuilds absolute source paths under the target, and the snapshot may have come from a drive that +// no longer exists on this box, so the prefix cannot be assumed. +func findScratchUnitDir(scratch, stack string) string { + found := "" + suffix := filepath.Join("backups", "primary", stack) + _ = filepath.Walk(scratch, func(path string, fi os.FileInfo, err error) error { + if err != nil || found != "" { + return nil //nolint:nilerr // a walk error on one branch must not abort the search + } + if fi.IsDir() && strings.HasSuffix(path, suffix) { + found = path + } + return nil + }) + return found +} + +// sniffScratchDump runs the R-44 content sniff over the dump about to be replayed. Best-effort and +// warn-level: any failure to read simply reports "no warning", because a sniff that blocks a +// restore is worse than the skew it describes. +func (m *Manager) sniffScratchDump(dumpDir, stack string) bool { + entries, err := os.ReadDir(dumpDir) + if err != nil { + return false + } + for _, e := range entries { + name := e.Name() + if e.IsDir() || filepath.Ext(name) != ".sql" || strings.HasPrefix(name, preRestoreDumpPrefix) { + continue + } + dbType := DBTypePostgres + if strings.Contains(name, string(DBTypeMariaDB)) { + dbType = DBTypeMariaDB + } + if v := ValidateDump(filepath.Join(dumpDir, name), dbType); v.LooksEmpty { + m.logger.Printf("[WARN] [offbox] %s: the snapshot dump %s has no account rows — it may predate the customer's data", stack, name) + return true + } + } + return false +} diff --git a/controller/internal/backup/offbox_reconstitute_test.go b/controller/internal/backup/offbox_reconstitute_test.go new file mode 100644 index 0000000..d794e5b --- /dev/null +++ b/controller/internal/backup/offbox_reconstitute_test.go @@ -0,0 +1,424 @@ +package backup + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-controller/internal/settings" +) + +// R-43/R-44 (v0.148.0) — the coherent-pair + true-restore tests. +// +// These exist because the product shipped a restore button for months that could not restore. +// DIAG-immich-restore-2026-07-19: 11 photos, files intact, timeline empty, two "successful" +// restores that merged 0 files and never touched postgres. Every test below asserts a behaviour +// whose absence produced that outcome, so each one is a regression guard for a real incident +// rather than a description of the current implementation. + +// recordingProvider records stop/start call ORDER so the reconstitution sequence can be asserted. +type recordingProvider struct { + offbox3aProvider + calls []string +} + +func (p *recordingProvider) StopStack(string) error { p.calls = append(p.calls, "stop"); return nil } +func (p *recordingProvider) StartStack(string) error { p.calls = append(p.calls, "start"); return nil } + +// The app really is up again after StartStack, so the post-restore health wait returns at once. +// Leaving it false would make each test sit through the full 90s deadline. +func (p *recordingProvider) RefreshAndIsRunning(string) bool { return true } + +// recoveryProvider adds the recovery info CaptureRecoveryUnit needs (the shared 3a provider has none). +type recoveryProvider struct { + offbox3aProvider + stackDir string +} + +func (p *recoveryProvider) GetStackRecoveryInfo(name string) (RecoveryInfo, bool) { + return RecoveryInfo{DisplayName: "Immich", StackDir: p.stackDir}, name == "immich" +} + +// pgDump builds a structurally valid postgres dump big enough to clear ValidateDump's 100-byte +// floor, with the accounts-table COPY block carrying `rows` rows. The R-44 sniff runs only on a +// dump that already passes structural validation, so a toy fixture would silently skip it. +func pgDump(rows int) string { + const head = `-- PostgreSQL database dump +-- Dumped from database version 16.10 +SET statement_timeout = 0; +SET lock_timeout = 0; +SET client_encoding = 'UTF8'; +CREATE TABLE public.asset (id uuid NOT NULL); +CREATE TABLE public."user" (id uuid NOT NULL, email text); +COPY public."user" (id, email) FROM stdin; +` + var b strings.Builder + b.WriteString(head) + for i := 0; i < rows; i++ { + b.WriteString("id-x\tuser@example.invalid\n") + } + b.WriteString("\\.\n") // the COPY-block terminator + return b.String() +} + +// reconFixture builds a manager with a COMPLETED full scratch for `immich`, a snapshot whose unit +// carries the given coherence stamp, and injectable copy/dump/import seams. +func reconFixture(t *testing.T, runID, dumpsAt string, dumpBody string) (*Manager, *recordingProvider, *[]string) { + t.Helper() + drive := t.TempDir() + m, sett := newOffboxManager(t) + prov := &recordingProvider{offbox3aProvider: offbox3aProvider{ + hdd: map[string]string{"immich": drive}, binds: map[string][]ClassifiedBind{}, has: map[string]bool{}, + }} + m.SetStackProvider(prov) + if err := sett.AddStoragePath(settings.StoragePath{Path: drive, Label: "USB", Schedulable: true}); err != nil { + t.Fatal(err) + } + + scratch, liveNs, err := m.offboxRestoreScratchDir("immich") + if err != nil { + t.Fatal(err) + } + oldNs := "/felhomdata/ns" + unitP := oldNs + "/backups/primary/immich" + dataP := oldNs + "/appdata/immich" + placements, err := mapOffsiteRestorePaths([]string{unitP, dataP}, "immich", scratch, liveNs) + if err != nil { + t.Fatal(err) + } + for _, pl := range placements { + if err := os.MkdirAll(pl.src, 0o755); err != nil { + t.Fatal(err) + } + if pl.isUnit { + dd := filepath.Join(pl.src, "db-dumps") + if err := os.MkdirAll(dd, 0o755); err != nil { + t.Fatal(err) + } + if dumpBody != "" { + if err := os.WriteFile(filepath.Join(dd, "immich-postgres.sql"), []byte(dumpBody), 0o644); err != nil { + t.Fatal(err) + } + } + man := &RecoveryManifest{SchemaVersion: 1, AppName: "immich", OffsiteRunID: runID, DumpsAt: dumpsAt} + if err := writeManifest(filepath.Join(pl.src, "manifest.json"), man); err != nil { + t.Fatal(err) + } + } + } + + m.SetOffboxFreeFn(func(string) int64 { return 100 << 30 }) + m.SetOffboxSizer(func(string) int64 { return 1 << 20 }) + m.SetOffboxRunner(func(_ context.Context, _ []string, args ...string) ([]byte, error) { + if contains(args, "snapshots") { + return []byte(`[{"short_id":"snap1","time":"2026-07-19T06:00:00Z","paths":["` + unitP + `","` + dataP + `"]}]`), nil + } + return nil, nil + }) + + // Seams: one DB, a safety dump that really writes a file, and a recording importer. + db := DiscoveredDB{StackName: "immich", ContainerName: "immich-postgres", DBType: DBTypePostgres} + m.discoverDBs = func(context.Context) ([]DiscoveredDB, error) { return []DiscoveredDB{db}, nil } + m.SetSafetyDumpFn(func(_ context.Context, d DiscoveredDB, dir string) DumpResult { + p := filepath.Join(dir, "immich-postgres.sql") + _ = os.MkdirAll(dir, 0o755) + _ = os.WriteFile(p, []byte(pgDump(1)), 0o644) + return DumpResult{DB: d, FilePath: p, Size: 42} + }) + var imported []string + m.importDBDump = func(_ context.Context, _ DiscoveredDB, p string) error { + imported = append(imported, p) + return nil + } + m.SetOffboxFullPlaceCopier(func(_, _ string) (int, error) { return 3, nil }) + return m, prov, &imported +} + +// TestReconstituteReplaysDBAndOrdersOperations is Scenario C: the whole point of R-43. A restore of +// a DB-indexed app must stop the app, place files, restart it and REPLAY the snapshot's dump — and +// the safety dump must exist before any of it. Before v0.148.0 the replay simply did not happen, +// which is why the photos never came back. +func TestReconstituteReplaysDBAndOrdersOperations(t *testing.T) { + m, prov, imported := reconFixture(t, "20260719T060000Z", "2026-07-19T06:00:00Z", pgDump(1)) + + res, err := m.ReconstituteFromOffsite(context.Background(), "immich") + if err != nil { + t.Fatalf("reconstitute: %v", err) + } + if res.DBsReplayed != 1 { + t.Fatalf("expected the snapshot dump to be replayed exactly once, got %d — this is the R-43 defect", res.DBsReplayed) + } + if len(*imported) != 1 || !strings.Contains((*imported)[0], "immich-postgres.sql") { + t.Fatalf("expected an import of the snapshot dump, got %v", *imported) + } + // The dump replayed must come from the SCRATCH unit, never the live one: the live unit is + // deliberately not overwritten, so replaying from it would replay the CURRENT database back over + // itself and restore nothing. + if !strings.Contains((*imported)[0], "offsite-restore") { + t.Fatalf("replay source must be the restored scratch unit, got %s", (*imported)[0]) + } + if res.FilesPlaced != 3 { + t.Fatalf("expected the userdata placement to be counted, got %d", res.FilesPlaced) + } + // stop BEFORE the file copy, start BEFORE the replay (ImportDump needs a live container). + if got := strings.Join(prov.calls, ","); got != "stop,start" { + t.Fatalf("expected stop then start around the restore, got %q", got) + } + if res.SafetyDump == "" { + t.Fatal("no safety dump recorded — the undo must exist") + } + if _, err := os.Stat(res.SafetyDump); err != nil { + t.Fatalf("safety dump not on disk: %v", err) + } + if !strings.HasPrefix(filepath.Base(res.SafetyDump), preRestoreDumpPrefix) { + t.Fatalf("safety dump must carry the pre-restore prefix so it is never replayed as a source, got %s", filepath.Base(res.SafetyDump)) + } +} + +// TestReconstituteRefusesWhenSafetyDumpFails is the RED-PROOF for the undo invariant: a replay whose +// previous state was not captured is an overwrite with no way back, so it must not happen at all — +// and it must abort with the live app untouched (no stop, no copy). +func TestReconstituteRefusesWhenSafetyDumpFails(t *testing.T) { + m, prov, imported := reconFixture(t, "run1", "2026-07-19T06:00:00Z", pgDump(1)) + m.SetSafetyDumpFn(func(_ context.Context, d DiscoveredDB, _ string) DumpResult { + return DumpResult{DB: d, Error: context.DeadlineExceeded} + }) + var copied bool + m.SetOffboxFullPlaceCopier(func(_, _ string) (int, error) { copied = true; return 1, nil }) + + _, err := m.ReconstituteFromOffsite(context.Background(), "immich") + if err == nil { + t.Fatal("expected a refusal when the safety dump cannot be taken") + } + if len(*imported) != 0 { + t.Fatalf("REPLAYED WITHOUT AN UNDO — the exact thing the invariant forbids: %v", *imported) + } + if copied { + t.Fatal("files were overwritten despite the refusal — the abort must leave live data untouched") + } + if len(prov.calls) != 0 { + t.Fatalf("the app was stopped despite the refusal, got %v", prov.calls) + } +} + +// TestReconstituteNoDBAppMakesNoDumpOrImportCalls is Scenario E: an app without a database must flow +// exactly as before — no safety dump, no replay — so the new leg cannot regress the simple case. +func TestReconstituteNoDBAppMakesNoDumpOrImportCalls(t *testing.T) { + m, _, imported := reconFixture(t, "run1", "2026-07-19T06:00:00Z", "") + m.discoverDBs = func(context.Context) ([]DiscoveredDB, error) { return nil, nil } + dumped := 0 + m.SetSafetyDumpFn(func(_ context.Context, d DiscoveredDB, _ string) DumpResult { + dumped++ + return DumpResult{DB: d} + }) + + res, err := m.ReconstituteFromOffsite(context.Background(), "immich") + if err != nil { + t.Fatalf("reconstitute: %v", err) + } + if dumped != 0 { + t.Fatalf("a no-DB app must not produce a safety dump, got %d call(s)", dumped) + } + if len(*imported) != 0 { + t.Fatalf("a no-DB app must not import anything, got %v", *imported) + } + if res.SafetyDump != "" || res.DBsReplayed != 0 { + t.Fatalf("unexpected DB activity: safety=%q replayed=%d", res.SafetyDump, res.DBsReplayed) + } +} + +// TestReconstituteSurfacesLegacySkewedPair is Scenario D: a pre-v0.148 snapshot carries no coherence +// stamp, so its two halves may be from different times. That must be SURFACED (and reversible), never +// blocked — the customer's own judgement is the gate, and refusing would deny a legitimate restore. +func TestReconstituteSurfacesLegacySkewedPair(t *testing.T) { + m, _, imported := reconFixture(t, "", "", pgDump(1)) + + res, err := m.ReconstituteFromOffsite(context.Background(), "immich") + if err != nil { + t.Fatalf("a legacy pair must still be restorable, got refusal: %v", err) + } + if !res.Skewed { + t.Fatal("an unstamped (pre-v0.148) snapshot must report Skewed so the confirm can say so") + } + if len(*imported) != 1 { + t.Fatalf("the legacy restore must still replay, got %v", *imported) + } +} + +// TestReconstituteFlagsCustomerEmptyDump is the R-44 sniff at the restore end: the immich dump that +// started all of this was structurally valid and contained zero users. Restoring it is allowed, but +// the customer must be told before they commit. +func TestReconstituteFlagsCustomerEmptyDump(t *testing.T) { + // A valid postgres dump whose accounts table has NO rows — the 2026-07-19 shape exactly. + m, _, _ := reconFixture(t, "run1", "2026-07-19T06:00:00Z", pgDump(0)) + + res, err := m.ReconstituteFromOffsite(context.Background(), "immich") + if err != nil { + t.Fatalf("the sniff must never block a restore: %v", err) + } + if !res.LooksEmpty { + t.Fatal("a dump with an empty accounts table must raise the warn-level signal") + } +} + +// TestOffsiteScratchPairReportsWhatTheConfirmNeeds covers the page-render surface: the confirm can +// only be honest if this reports the pair's age and warnings before anything is started. +func TestOffsiteScratchPairReportsWhatTheConfirmNeeds(t *testing.T) { + m, _, _ := reconFixture(t, "run1", "2026-07-19T06:00:00Z", + "-- PostgreSQL database dump\nCREATE TABLE a();\nCOPY public.\"user\" (id) FROM stdin;\n7\n\\.\n") + + info := m.OffsiteScratchPair("immich") + if !info.Ready || !info.HasDump { + t.Fatalf("expected a ready pair with a dump, got %+v", info) + } + if info.Skewed { + t.Fatal("a stamped snapshot must not be reported as skewed") + } + if info.LooksEmpty { + t.Fatal("a dump with account rows must not be flagged empty") + } + want, _ := time.Parse(time.RFC3339, "2026-07-19T06:00:00Z") + if !info.DumpsAt.Equal(want) { + t.Fatalf("DumpsAt = %v, want %v", info.DumpsAt, want) + } +} + +// --- R-44: the coherence pre-phase ----------------------------------------------------------- + +// TestOffsiteRunDumpsBeforeCapture is Scenarios A + B. The ORDER is the entire mechanism: dumps +// must be refreshed BEFORE restic captures, so the snapshot pairs this run's database with this +// run's files. Reversed, the snapshot would hold rows pointing at files that were never captured. +// +// It also asserts the ordering on the NIGHTLY entry point (RunOffboxBackup, no progress sink), not +// just the manual one — before v0.148.0 the nightly ordering was an accident of two independent +// scheduler entries at 02:30 and 04:15, which a schedule edit could silently invert. +func TestOffsiteRunDumpsBeforeCapture(t *testing.T) { + drive := t.TempDir() + m, sett, prov := classifiedOffboxManager(t, drive) + mkUnit(t, drive, "immich") + if err := os.MkdirAll(filepath.Join(drive, "appdata", "immich"), 0o755); err != nil { + t.Fatal(err) + } + prov.hdd["immich"] = drive + prov.has["immich"] = true + prov.binds["immich"] = []ClassifiedBind{mandatoryHDD("appdata/immich")} + _ = sett.SetAppOffbox("immich", true) + + var order []string + m.SetOffsitePreDumpFn(func(context.Context) error { + order = append(order, "dump") + return nil + }) + m.SetOffboxRunner(func(_ context.Context, _ []string, args ...string) ([]byte, error) { + switch { + case contains(args, "cat") && contains(args, "config"): + return []byte(`{"version":2}`), nil + case contains(args, "backup"): + order = append(order, "capture") + return nil, nil + case contains(args, "snapshots"): + return []byte(`[]`), nil + case contains(args, "stats"): + return []byte(`{"total_size":123}`), nil + } + return nil, nil + }) + + if err := m.RunOffboxBackup(context.Background()); err != nil { + t.Fatalf("run: %v", err) + } + if len(order) < 2 { + t.Fatalf("expected both a dump and a capture, got %v", order) + } + if order[0] != "dump" { + t.Fatalf("the dump leg MUST precede the capture (R-44); got %v", order) + } + if order[1] != "capture" { + t.Fatalf("expected the capture immediately after the dump, got %v", order) + } +} + +// TestOffsiteRunContinuesWhenDumpLegFails is the data-first rule: a dump failure degrades the +// snapshot's DB half but must NOT abort the push. Refusing to ship the files would turn a partial +// backup into no backup at all — strictly worse for the customer. +func TestOffsiteRunContinuesWhenDumpLegFails(t *testing.T) { + drive := t.TempDir() + m, sett, prov := classifiedOffboxManager(t, drive) + mkUnit(t, drive, "immich") + if err := os.MkdirAll(filepath.Join(drive, "appdata", "immich"), 0o755); err != nil { + t.Fatal(err) + } + prov.hdd["immich"] = drive + prov.has["immich"] = true + prov.binds["immich"] = []ClassifiedBind{mandatoryHDD("appdata/immich")} + _ = sett.SetAppOffbox("immich", true) + + m.SetOffsitePreDumpFn(func(context.Context) error { return context.DeadlineExceeded }) + cap := &backupCapture{} + m.SetOffboxRunner(cap.runner()) + + if err := m.RunOffboxBackup(context.Background()); err != nil { + t.Fatalf("a dump failure must not fail the whole run: %v", err) + } + if cap.backups != 1 { + t.Fatalf("the files must still be pushed after a dump failure, got %d capture(s)", cap.backups) + } +} + +// TestCaptureRecoveryUnitStampsAndCarriesRunID covers the stamp that makes a pair verifiable at +// restore time, and the trap beside it: the PERIODIC refresh must neither invent a coherence claim +// nor erase one a real run established. +func TestCaptureRecoveryUnitStampsAndCarriesRunID(t *testing.T) { + drive := t.TempDir() + m, _, base := classifiedOffboxManager(t, drive) + base.hdd["immich"] = drive + // CaptureRecoveryUnit needs real recovery info + a compose dir to read; the shared fixture + // provider returns none, so wrap it rather than widening a struct four other test files use. + stackDir := filepath.Join(t.TempDir(), "immich") + if err := os.MkdirAll(stackDir, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(stackDir, "docker-compose.yml"), []byte("services: {}\n"), 0o644); err != nil { + t.Fatal(err) + } + m.SetStackProvider(&recoveryProvider{offbox3aProvider: *base, stackDir: stackDir}) + + // 1) A run in flight stamps the manifest. + end := m.beginOffsiteRunStamp("run-A") + if err := m.CaptureRecoveryUnit("immich"); err != nil { + t.Fatalf("capture: %v", err) + } + end() + man := readManifest(RecoveryUnitManifestPath(drive, "immich")) + if man == nil || man.OffsiteRunID != "run-A" { + t.Fatalf("expected the in-flight run id to be stamped, got %+v", man) + } + if man.DumpsAt == "" { + t.Fatal("a stamped unit must record when its dumps were taken") + } + + // 2) A periodic refresh (no run in flight) must CARRY the stamp forward, not blank it — a unit + // that silently lost its stamp would be re-reported as a skewed legacy pair at restore time. + if err := m.CaptureRecoveryUnit("immich"); err != nil { + t.Fatalf("refresh: %v", err) + } + man2 := readManifest(RecoveryUnitManifestPath(drive, "immich")) + if man2 == nil || man2.OffsiteRunID != "run-A" { + t.Fatalf("the periodic refresh erased the coherence stamp: %+v", man2) + } + + // 3) A NEW run re-stamps even though nothing else about the unit changed — the idempotent-skip + // must not swallow the one field the restore path reads. + end2 := m.beginOffsiteRunStamp("run-B") + if err := m.CaptureRecoveryUnit("immich"); err != nil { + t.Fatalf("capture 2: %v", err) + } + end2() + man3 := readManifest(RecoveryUnitManifestPath(drive, "immich")) + if man3 == nil || man3.OffsiteRunID != "run-B" { + t.Fatalf("a new run must re-stamp the unit, got %+v", man3) + } +} diff --git a/controller/internal/backup/offbox_restore.go b/controller/internal/backup/offbox_restore.go index 1908521..4ce3f48 100644 --- a/controller/internal/backup/offbox_restore.go +++ b/controller/internal/backup/offbox_restore.go @@ -30,6 +30,16 @@ const ( // SetOffboxFreeFn overrides the restore free-space probe (tests; the Windows go-test host has no df). func (m *Manager) SetOffboxFreeFn(fn func(path string) int64) { m.offboxFreeFn = fn } +// SetOffboxFullPlaceCopier overrides the FULL-restore overwrite copier (tests; no rsync needed). +func (m *Manager) SetOffboxFullPlaceCopier(fn func(src, dst string) (int, error)) { + m.offboxFullPlaceCopier = fn +} + +// SetSafetyDumpFn overrides the pre-restore safety dump (tests; no Docker needed). +func (m *Manager) SetSafetyDumpFn(fn func(ctx context.Context, db DiscoveredDB, dumpDir string) DumpResult) { + m.safetyDumpFn = fn +} + // offboxFree returns the free-space probe (nil seam → the real diskFreeBytes). func (m *Manager) offboxFree() func(string) int64 { if m.offboxFreeFn != nil { diff --git a/controller/internal/backup/recovery_unit.go b/controller/internal/backup/recovery_unit.go index e044bfa..3544089 100644 --- a/controller/internal/backup/recovery_unit.go +++ b/controller/internal/backup/recovery_unit.go @@ -44,6 +44,15 @@ type RecoveryManifest struct { DBDumps []string `json:"db_dumps"` VolumeDumps []string `json:"volume_dumps"` Checksums map[string]string `json:"checksums"` // sha256 of captured compose/ files + // R-43/R-44 (v0.148.0): the coherence stamp. An offsite run refreshes the dumps FIRST and then + // captures the unit, so a manifest carrying an OffsiteRunID asserts "the db-dumps/ in this unit + // were taken by that run" — i.e. the snapshot is an internally coherent {DB@T, files@T} pair. + // A manifest WITHOUT these fields is a pre-v0.148 unit whose dump age is unknown and may skew + // arbitrarily from the files beside it (the DIAG-immich-restore-2026-07-19 failure); the restore + // confirm surfaces that honestly rather than blocking. Empty on the periodic refresh, which must + // never claim a coherence it did not establish — it carries the prior stamp forward instead. + OffsiteRunID string `json:"offsite_run_id,omitempty"` + DumpsAt string `json:"dumps_at,omitempty"` // RFC3339 UTC — when this run's dump leg finished } // SetVersion records the controller version stamped into recovery-unit manifests. @@ -107,13 +116,26 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error { version := m.versionLocked() manifestPath := RecoveryUnitManifestPath(nsRoot, stackName) + cur := readManifest(manifestPath) + + // R-43/R-44: the coherence stamp of the offsite run currently in flight ("" on the periodic + // refresh and on the local dump run). When empty we CARRY THE PRIOR STAMP FORWARD rather than + // blanking it — a periodic refresh must neither claim a coherence it did not establish nor + // destroy the record of one that a real run did. + runID, dumpsAt := m.offsiteRunStamp() + if runID == "" && cur != nil { + runID, dumpsAt = cur.OffsiteRunID, cur.DumpsAt + } // Skip if the unit is already current — avoids needless drive writes on the periodic refresh. - if cur := readManifest(manifestPath); cur != nil && + // The run-id is part of "current": an offsite run must re-stamp the manifest even when nothing + // else changed, because the stamp is exactly the claim the restore path reads. + if cur != nil && cur.ControllerVer == version && stringMapEqual(cur.Checksums, checksums) && stringSliceEqual(cur.DBDumps, dbDumps) && - stringSliceEqual(cur.VolumeDumps, volDumps) { + stringSliceEqual(cur.VolumeDumps, volDumps) && + cur.OffsiteRunID == runID { return nil } @@ -143,6 +165,8 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error { DBDumps: dbDumps, VolumeDumps: volDumps, Checksums: checksums, + OffsiteRunID: runID, + DumpsAt: dumpsAt, } if err := writeManifest(manifestPath, manifest); err != nil { return fmt.Errorf("writing manifest: %w", err) diff --git a/controller/internal/backup/restore_db.go b/controller/internal/backup/restore_db.go index 7e3473d..2fb99fc 100644 --- a/controller/internal/backup/restore_db.go +++ b/controller/internal/backup/restore_db.go @@ -19,7 +19,15 @@ import ( // A dump whose DB container is not found is logged and skipped; an actual import FAILURE is returned // (surfaced, not swallowed) so a failed data restore cannot read as success. func (m *Manager) reimportDBDumps(ctx context.Context, stackName, nsRoot string) (int, error) { - dumpDir := AppDBDumpPath(nsRoot, stackName) + return m.reimportDBDumpsFrom(ctx, stackName, AppDBDumpPath(nsRoot, stackName)) +} + +// reimportDBDumpsFrom is reimportDBDumps with an EXPLICIT dump directory. The offsite +// reconstitution path (R-43) replays out of the restored SCRATCH unit rather than the live one: +// the local unit is deliberately never overwritten by a placement, so the dump that belongs to the +// chosen snapshot exists only under the scratch. Same discovery/import seams, same failure +// semantics — only the source directory differs. +func (m *Manager) reimportDBDumpsFrom(ctx context.Context, stackName, dumpDir string) (int, error) { entries, err := os.ReadDir(dumpDir) if err != nil { if os.IsNotExist(err) { diff --git a/controller/internal/web/handlers.go b/controller/internal/web/handlers.go index e6c6f70..ad4bb78 100644 --- a/controller/internal/web/handlers.go +++ b/controller/internal/web/handlers.go @@ -815,6 +815,17 @@ func (s *Server) backupsRestoreHandler(w http.ResponseWriter, r *http.Request) { } } data["OffboxScratchReady"] = ready + // R-43: the same prepared scratch also enables the TRUE restore (files + database). Its confirm + // has to state what the pair actually IS — how old the DB half is, whether the two halves even + // come from the same run, and whether the dump looks customer-empty — because a restore is the + // one operation whose result the customer cannot inspect until after committing to it. + pairs := map[string]backup.OffsitePairInfo{} + if s.backupMgr != nil { + for name := range ready { + pairs[name] = s.backupMgr.OffsiteScratchPair(name) + } + } + data["OffboxPairInfo"] = pairs // R-7b: the shares source is not an app — it has no per-app toggle and no recovery unit — so it // gets its own restore entry rather than a synthetic row in OffboxApps (which would also make it // appear in the per-app offsite TOGGLE list on /backups/remote, where it does not belong). diff --git a/controller/internal/web/offbox_handlers.go b/controller/internal/web/offbox_handlers.go index 55e538d..cf47cf7 100644 --- a/controller/internal/web/offbox_handlers.go +++ b/controller/internal/web/offbox_handlers.go @@ -3,6 +3,7 @@ package web import ( "context" "encoding/json" + "fmt" "net/http" "net/url" "strconv" @@ -340,6 +341,69 @@ func (s *Server) offboxRestoreHandler(w http.ResponseWriter, r *http.Request) { offboxRedirectTo(w, r, "/backups/restore", "A távoli visszaállítás elindult — az állapot itt frissül.", false) } +// offboxReconstituteHandler is the TRUE offsite restore (R-43, v0.148.0): files overwritten to the +// snapshot's version + that same snapshot's database replayed + the app restarted, with a safety +// dump of the current database taken first. +// +// It is a separate button from the missing-only place, not a flag on it. The two do opposite things +// to an existing file, and the v0.147 flash („hiányzó fájljai helyreállítva") described a mechanism +// that could report success after merging zero files while the customer's photos stayed invisible. +// The flash here states the OUTCOME instead — file count, database, backup timestamp, restart — +// because that is the only part the customer can check against what they see in the app. +func (s *Server) offboxReconstituteHandler(w http.ResponseWriter, r *http.Request) { + if s.backupMgr == nil || !s.backupMgr.OffboxConfigured() { + offboxRedirectTo(w, r, "/backups/restore", "A távoli mentési cél nincs beállítva.", true) + return + } + _ = r.ParseForm() + app := strings.TrimSpace(r.FormValue("app")) + if app == "" { + offboxRedirectTo(w, r, "/backups/restore", "Hiányzó alkalmazás.", true) + return + } + // This one overwrites live files and replays a database — it must never happen on a stray click. + if r.FormValue("confirm") != "1" { + offboxRedirectTo(w, r, "/backups/restore", "A teljes visszaállítás megerősítés nélkül nem hajtható végre.", true) + return + } + if s.backupMgr.IsRunning() { + offboxRedirectTo(w, r, "/backups/restore", "Egy mentési/visszaállítási művelet már fut.", true) + return + } + s.backupMgr.BeginRestoreOp("offbox-reconstitute", app) + go func() { + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Minute) + defer cancel() + res, err := s.backupMgr.ReconstituteFromOffsite(ctx, app) + if err != nil { + s.logger.Printf("[ERROR] [web] off-box reconstitute %s (async): %v", app, err) + s.backupMgr.EndRestoreOp(false, "A teljes visszaállítás sikertelen: "+err.Error()) + return + } + s.logger.Printf("[INFO] [web] off-box reconstitute %s completed (async): files=%d dbs=%d snapshot=%s", + app, res.FilesPlaced, res.DBsReplayed, res.SnapshotID) + s.backupMgr.EndRestoreOp(true, reconstituteOutcomeMsg(app, res)) + }() + offboxRedirectTo(w, r, "/backups/restore", "A teljes visszaállítás elindult — az állapot itt frissül.", false) +} + +// reconstituteOutcomeMsg builds the OUTCOME flash for a completed reconstitution. Pure, so the +// wording is unit-testable — this string is the customer's only evidence that the operation did +// what its label promised, and the zero-file and no-database cases must each read truthfully rather +// than borrowing the confident sentence that belongs to the full case. +func reconstituteOutcomeMsg(app string, res backup.OffsiteReconstituteResult) string { + when := "" + if !res.DumpsAt.IsZero() { + when = " (mentés: " + res.DumpsAt.In(getTimezone()).Format("2006-01-02 15:04") + ")" + } + if res.DBsReplayed == 0 { + // A no-database app: saying "és az adatbázis" here would be a lie, and this is precisely the + // class of sentence the DIAG found being printed over a no-op. + return fmt.Sprintf("A(z) %s: %d fájl visszaállítva%s — az alkalmazás újraindult. Ennek az alkalmazásnak nincs adatbázisa.", app, res.FilesPlaced, when) + } + return fmt.Sprintf("A(z) %s: %d fájl és az adatbázis visszaállítva%s — az alkalmazás újraindult.", app, res.FilesPlaced, when) +} + // offboxVerifyCopyDeleteHandler removes ONE verification copy (v0.147.0, 4a). // // The only delete this slice adds, so it is deliberately narrow: it names a STACK, never a path — the diff --git a/controller/internal/web/server.go b/controller/internal/web/server.go index 1360295..335fd00 100644 --- a/controller/internal/web/server.go +++ b/controller/internal/web/server.go @@ -422,6 +422,8 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { s.offboxRestoreHandler(w, r) case path == "/backup/offbox/place" && r.Method == http.MethodPost: s.offboxPlaceHandler(w, r) + case path == "/backup/offbox/reconstitute" && r.Method == http.MethodPost: + s.offboxReconstituteHandler(w, r) // v0.147.0 (4a): remove ONE verification copy. The only delete path this slice adds — see // DeleteOffsiteRestoreCopy for the prefix guard. case path == "/backup/offbox/verify-copy/delete" && r.Method == http.MethodPost: diff --git a/controller/internal/web/templates/backups_remote.html b/controller/internal/web/templates/backups_remote.html index 24144a3..537be28 100644 --- a/controller/internal/web/templates/backups_remote.html +++ b/controller/internal/web/templates/backups_remote.html @@ -215,6 +215,16 @@ /* A run is not only the per-app loop: the shares leg and the retention/prune step follow it and can dominate the wall clock (40 of 57 seconds, measured). Name them, or the card freezes on the last app's finished counters for that whole tail. */ + /* The dump pre-phase runs BEFORE the per-app loop (R-44): it refreshes every app's + database dump so the snapshot pairs this run's files with this run's DB. On an app + with a large database it can dominate the early wall clock, and an unnamed silent + stretch at the start reads as a hang. */ + if(p.phase === 'dump'){ + text.textContent = 'Adatbázisok mentése a pillanatképhez…'; + bar.style.width = '100%'; + box.className = 'alert alert-info'; box.style.display = ''; + return; + } if(p.phase === 'retention'){ text.textContent = 'Karbantartás: régi mentések rendezése a távoli tárolón…'; bar.style.width = '100%'; diff --git a/controller/internal/web/templates/backups_restore.html b/controller/internal/web/templates/backups_restore.html index fffad39..7c6549c 100644 --- a/controller/internal/web/templates/backups_restore.html +++ b/controller/internal/web/templates/backups_restore.html @@ -93,7 +93,25 @@ - A meglévő fájlokat nem írja felül. + A meglévő fájlokat nem írja felül. Adatbázist nem állít vissza — törölt tartalom ettől nem jelenik meg újra. + {{$pair := index $.OffboxPairInfo .Name}} +
{{$.CSRFField}} + + + +
+ A fájlokat a mentés szerinti változatra állítja vissza és az adatbázist is visszatölti. Semmit nem töröl: a mentés óta létrejött fájlok megmaradnak. A jelenlegi adatbázisról előtte biztonsági mentés készül. + {{if $pair.Skewed}} + Az adatbázis-mentés régebbi{{if not $pair.DumpsAt.IsZero}} ({{fmtTime $pair.DumpsAt}}){{end}} — a fájlok és az adatbázis eltérő időpontból származnak. + {{end}} + {{if $pair.LooksEmpty}} + A mentett adatbázis üresnek tűnik (nincs benne felhasználói fiók) — elképzelhető, hogy a mentés korábbi, mint az adataid. + {{end}} {{end}}{{end}} {{template "app_list_row_end"}} {{end}} @@ -217,6 +235,26 @@ function confirmDeleteVerifyCopy(btn){ }); }); } +/* R-43: the true offsite restore overwrites live files and replays a database, so it double-confirms + and — unlike the old missing-only merge — states the DB half's age and any warning BEFORE the + customer commits. The honesty lines are already rendered under the button; repeating the decisive + ones here means the person clicking "Igen" has read them. */ +function confirmFullRestore(btn){ + var app = btn.getAttribute('data-restore-app') || ''; + var when = btn.getAttribute('data-restore-when') || ''; + var skewed = btn.getAttribute('data-restore-skewed') === '1'; + var empty = btn.getAttribute('data-restore-empty') === '1'; + var q = 'Teljes visszaállítás: ' + app + (when ? ' — a mentés ideje: ' + when : '') + '.'; + if (skewed) { q += ' FIGYELEM: a fájlok és az adatbázis eltérő időpontból származnak.'; } + if (empty) { q += ' FIGYELEM: a mentett adatbázis üresnek tűnik.'; } + q += ' A fájlok a mentés szerinti változatra állnak vissza, semmi nem törlődik.'; + felhomConfirm(btn, q, function(){ + felhomConfirm(btn, 'UTOLSÓ MEGERŐSÍTÉS: az alkalmazás leáll, az adatbázis visszatöltődik, majd újraindul. A jelenlegi adatbázisról biztonsági mentés készül.', function(){ + var f = btn.closest('form'); + if (f) { if (f.requestSubmit) f.requestSubmit(); else f.submit(); } + }); + }); +} function fabStart(stack, next){ fetch('/api/export/download/start', {method:'POST', headers:Object.assign({'Content-Type':'application/json'}, csrfHeaders()), body: JSON.stringify({stack_name: stack, password: fabPassword()})}) .then(function(r){ return r.json(); })