v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (R-43 + R-44)
Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.
R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.
New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
- nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
created after the snapshot survives as an extra;
- the undo exists before the act — the pre-restore- dump is verified ON DISK
before anything is stopped, overwritten or replayed; if it cannot be taken
the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.
R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.
Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.
11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.
NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.
Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
This commit is contained in:
@@ -1,5 +1,79 @@
|
||||
## Changelog
|
||||
|
||||
### v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (2026-07-19)
|
||||
|
||||
Closes **R-43** and **R-44**, the two findings from `DIAG-immich-restore-2026-07-19`. The short
|
||||
version of that diagnosis: Viktor deleted 11 immich photos to test offsite restore, both restore
|
||||
runs flashed success, and the photos stayed gone. Two independent defects, both fixed here.
|
||||
|
||||
**R-43 — no offsite path could restore a database.** All three offsite buttons were file-only.
|
||||
The two „visszaállítás" actions staged into a scratch folder and never touched postgres; the
|
||||
place-to-live action merged only files MISSING from the live tree and never replayed a dump. For a
|
||||
DB-indexed app — most of the catalog — that combination cannot bring content back: the bytes
|
||||
return and the app still cannot see them, because its index lives in the database. The dump was
|
||||
faithfully carried INTO every snapshot and could never be replayed OUT of one.
|
||||
|
||||
New: **„Teljes visszaállítás (fájlok + adatbázis)"** (`ReconstituteFromOffsite`,
|
||||
`/backup/offbox/reconstitute`). Safety dump → stop → files overwritten to the snapshot's version →
|
||||
start → the snapshot's own dump replayed → health wait. Two invariants:
|
||||
|
||||
- **Nothing is ever deleted.** The full-restore copier is `rsync -a` with NO `--ignore-existing`
|
||||
(a changed file becomes the snapshot's version) and NO `--delete` (a file created after the
|
||||
snapshot survives as an extra). A restore that silently removed newer work would be a data-loss
|
||||
event wearing a recovery button's label.
|
||||
- **The undo exists before the act.** A `pre-restore-` dump of the live database is written and
|
||||
verified on disk BEFORE anything is stopped, overwritten or replayed; if it cannot be taken the
|
||||
whole operation refuses with zero changes. The safety dumps live in the app's own unit and
|
||||
appear in `ListDumpFiles` — an undo the customer cannot see is not much of one.
|
||||
|
||||
The replay reads the SCRATCH unit, not the live one: the live recovery unit is still never
|
||||
overwritten (it is the local restore path's source), so replaying from it would replay the current
|
||||
database back over itself and restore nothing.
|
||||
|
||||
**R-44 — a manual push shipped an unrefreshed dump.** `RunOffboxBackup` went straight to the
|
||||
restic push; dumps came only from the separate 02:30 local run, so a manual push at any other hour
|
||||
shipped a dump up to ~24h old. On 2026-07-19 that dump was taken four hours before the customer's
|
||||
account existed and probed to `asset: 0 / user: 0 / album: 0` — a 52MB file whose entire bulk was
|
||||
immich's shipped geodata tables. Size and table count both called it healthy.
|
||||
|
||||
Every offsite run — **manual and nightly** — now refreshes the dumps and recovery units FIRST, then
|
||||
captures. Order is the mechanism: the gap can only ADD files the DB does not reference yet (a
|
||||
harmless orphan blob), never remove one it does, so the file set is always a superset of what the
|
||||
restored DB points at. This also makes the nightly ordering structural instead of a coincidence of
|
||||
two scheduler entries at 02:30 and 04:15. Each unit manifest carries the run's `offsite_run_id` +
|
||||
`dumps_at`, so a snapshot's coherence is verifiable at restore time rather than assumed.
|
||||
|
||||
**Honesty surfaces** (warn-level, never gates — a false positive that blocked a restore would be
|
||||
worse than the skew it guards against):
|
||||
|
||||
- A pre-v0.148 snapshot has no stamp → the confirm says „Az adatbázis-mentés régebbi (<ts>) — a
|
||||
fájlok és az adatbázis eltérő időpontból származnak." It still restores.
|
||||
- `ValidateDump` gained a content sniff: a structurally valid dump whose accounts table has zero
|
||||
rows raises „A mentett adatbázis üresnek tűnik". Exact table-name matching, deliberately — a
|
||||
substring match on "user" would flag `user_metadata` / `album_user` / `user_audit` on every
|
||||
healthy single-user box and turn the signal into noise.
|
||||
- The completion flash states an OUTCOME, not a mechanism: „A(z) X: N fájl és az adatbázis
|
||||
visszaállítva (mentés: <ts>) — az alkalmazás újraindult." A no-database app says so explicitly
|
||||
rather than borrowing the confident sentence.
|
||||
- The old missing-only button now says what it does NOT do: „Adatbázist nem állít vissza — törölt
|
||||
tartalom ettől nem jelenik meg újra."
|
||||
|
||||
A new `dump` progress phase („Adatbázisok mentése a pillanatképhez…") names the pre-phase, which on
|
||||
a large database dominates the early wall clock and would otherwise read as a hang.
|
||||
|
||||
Tests: 11 new, with **5 red-proofs run and reverted** — replay removed (0 replayed), capture moved
|
||||
before the dump (`[capture dump]`), both undo guards removed (no refusal), substring table matching
|
||||
(join tables mistaken for accounts), buffer-exceeding rows uncounted (a wide row sniffed as empty).
|
||||
Two of those red-proofs found real test weaknesses rather than confirming strength: the first undo
|
||||
mutation was caught by a second guard, and the first table-matching test did not discriminate
|
||||
between the two matchers at all — both tests were rewritten to the cases that actually separate
|
||||
them.
|
||||
|
||||
**NOT in this slice:** the catalog-wide invariant check (stays on R-41), nightly cadence, retention,
|
||||
quota math, tier-2, and the v0.147.x progress semantics beyond the one added phase line. The
|
||||
missing-only place button's own zero-file flash is also untouched — that is the v0.147 feedback arc's
|
||||
item, not R-43/R-44.
|
||||
|
||||
### docs — the workflow moved to DooPlex-local execution (2026-07-19)
|
||||
|
||||
**Docs only, no version bump, no code change.** Claude Code now runs on DooPlex
|
||||
|
||||
+20
-1
@@ -7,7 +7,26 @@
|
||||
>
|
||||
> Ask Claude Code: "Please update CONTEXT.md with what we did today"
|
||||
|
||||
Last updated: 2026-07-19 (v0.147.3 — feedback slice 1: pressing a button says something)
|
||||
Last updated: 2026-07-19 (v0.148.0 — coherent snapshot pairs + an offsite restore that restores)
|
||||
|
||||
> **2026-07-19 — v0.148.0: R-43 + R-44.** Viktor deleted 11 immich photos to test offsite restore;
|
||||
> both runs flashed success and the photos stayed gone (`DIAG-immich-restore-2026-07-19`). Two
|
||||
> defects. **R-43:** no offsite path could restore a DATABASE — all three buttons were file-only, so
|
||||
> a DB-indexed app got its bytes back and still could not see them. **R-44:** a manual push shipped
|
||||
> whatever dump the 02:30 local run left; that day's predated the customer's account by four hours
|
||||
> and held zero users and zero assets inside 52MB of shipped geodata.
|
||||
>
|
||||
> Now: every run (manual AND nightly) dumps FIRST, then captures, so each snapshot is a coherent
|
||||
> `{DB@T, files@T}` pair stamped with `offsite_run_id`; and „Teljes visszaállítás (fájlok +
|
||||
> adatbázis)" does safety-dump → stop → overwrite files → start → replay the snapshot's dump.
|
||||
> **Two invariants: nothing is ever deleted, and the undo is verified on disk before the act.**
|
||||
> Warn-level honesty surfaces for stale/empty-looking pairs — never gates.
|
||||
>
|
||||
> **The live acceptance has NOT run yet** (§9: upload → push → empty trash for real → one button →
|
||||
> photos back). Until it does, the capability-map offsite row stays PARTIAL/scope-contested, the
|
||||
> customer-restore row stays MISSING, and R-3 stays DRAFT. Floor raise = Viktor's click, BEFORE the
|
||||
> acceptance run. Still open: the `00-capability-map.md:61` ruling — did CAMPAIGN-6D's "immich
|
||||
> end-to-end from offsite alone" exercise the DB half, or only the file half?
|
||||
|
||||
> **2026-07-19 — v0.147.0 → v0.147.3: feedback slice 1.** The systemic complaint, twice in one
|
||||
> evening: you press a button and nothing happens. Three worst offenders fixed on the two patterns
|
||||
|
||||
@@ -1,170 +1,85 @@
|
||||
# REPORT — controller v0.147.0 → v0.147.3: feedback slice 1 (2026-07-19)
|
||||
# REPORT — v0.148.0: coherent snapshot pairs + a real offsite restore (R-43 + R-44)
|
||||
|
||||
**Baseline v0.146.0 → LIVE v0.147.3 on demo guest 9201.** Green at every version:
|
||||
`go build ./... && go vet ./... && go test ./...` (23 packages) plus `template_id_gate`,
|
||||
`emoji_gate`, `native_confirm_gate`, `offbox_rename_gate`, `app_row_dedup_gate`, `mojibake_gate`.
|
||||
**Date:** 2026-07-19 · **Repo:** felhom-controller (v0.147.3 → **v0.148.0**) · Trunk, pushed to `main`.
|
||||
|
||||
> `docker_run_volume_path_gate` fails on `internal/appexport/estimate.go:179`. **Pre-existing on
|
||||
> HEAD and untouched by this release** — verified by stashing this work and re-running the gate.
|
||||
> Not fixed here (out of scope, and it is someone's call whether that `-v :ro` du probe gets an
|
||||
> allowlist entry or a rewrite).
|
||||
Closes the two findings from `felhom.eu/documentation/audits/DIAG-immich-restore-2026-07-19.md`.
|
||||
|
||||
## The problem
|
||||
## What was broken
|
||||
|
||||
The systemic complaint, twice in one evening: **you press a button and nothing happens.** No
|
||||
progress, no ETA, no named result. Three worst offenders fixed on the two patterns already in the
|
||||
codebase — the deploy 3-step panel and the storage-init status poll. **No new framework**; that is
|
||||
now ROADMAP **R-45**, because three targeted cards were worth shipping tonight and a framework was
|
||||
not.
|
||||
Viktor deleted 11 immich photos to test offsite restore. Both restore runs flashed success; the
|
||||
photos stayed gone. Two independent defects:
|
||||
|
||||
Three of the four versions below exist because the cards were watched against **real runs on the
|
||||
demo box**. Each follow-up is a thing no unit test would have surfaced.
|
||||
- **R-43** — no offsite path could restore a database. All three buttons were file-only, so for a
|
||||
DB-indexed app (most of the catalog) the bytes came back and the app still could not see them.
|
||||
- **R-44** — a manual push shipped whatever dump the 02:30 local run had left. That day's dump
|
||||
predated the customer's account by four hours and contained `asset: 0 / user: 0 / album: 0`, in a
|
||||
52 MB file whose bulk was immich's shipped geodata tables.
|
||||
|
||||
---
|
||||
## What shipped
|
||||
|
||||
## 4a — a verification restore names its result
|
||||
**P1 — coherence pre-phase.** Every offsite run (manual AND nightly) refreshes dumps + recovery
|
||||
units before capturing. Each manifest carries `offsite_run_id` + `dumps_at`. The periodic refresh
|
||||
carries a prior stamp forward and never invents one. A dump failure is a loud WARN that does not
|
||||
abort the push (data-first). New `dump` progress phase.
|
||||
|
||||
The completion flash said the app had been restored „ellenőrző mappába a meghajtón". *Which* folder,
|
||||
on *which* drive, was invisible — so the customer could not go and look at the thing they had just
|
||||
asked for. And nothing anywhere listed what these restores had accumulated, so copies piled up and
|
||||
the only way to find them was SSH.
|
||||
**P2 — honesty surfaces.** `ValidateDump` gained an exact-match accounts-table sniff
|
||||
(`LooksEmpty`); the confirm shows dump age, an unstamped-pair warning and the empty-dump warning.
|
||||
All warn-level, none a gate.
|
||||
|
||||
- The flash now carries the **full path**.
|
||||
- The restore page gained **„Meglévő ellenőrző másolatok"** — app · size · date · path — each with a
|
||||
double-confirmed **„Másolat törlése"** (the house inline `felhomConfirm` idiom, never native
|
||||
`confirm()`).
|
||||
- `backups/offsite-restore` was open-coded in three places; it now has one home
|
||||
(`offsiteRestoreRootFor`). A test pins the path in the flash to the path in the listing, so the
|
||||
customer can never be told about a directory the page cannot show or remove.
|
||||
**P3 — `ReconstituteFromOffsite`** (`/backup/offbox/reconstitute`): safety dump → stop → files
|
||||
overwritten → start → snapshot's dump replayed → health wait. Outcome-shaped flash. Two invariants:
|
||||
nothing is ever deleted (`-a`, no `--ignore-existing`, no `--delete`), and the `pre-restore-` undo
|
||||
is verified on disk before anything is touched — otherwise refuse with zero changes.
|
||||
|
||||
**The delete is the only one this release adds**, so it names a **stack, never a path**: the Manager
|
||||
resolves that name inside a root it computed itself and refuses anything landing outside.
|
||||
Red-proofed — neutralise the name guard and `stack: ""` resolves to the offsite-restore **root** and
|
||||
`os.RemoveAll` takes every copy with it. Every refusal is asserted as a **non-effect** (the
|
||||
neighbouring copy and the live data are still on disk afterwards).
|
||||
**P4 — tests.** 11 new; full suite green: `go build ./... && go vet ./... && go test ./...`.
|
||||
Design gates pass: template_id, emoji, native_confirm, offbox_rename, app_row_dedup, mojibake.
|
||||
|
||||
Honest note: the `HasPrefix` containment check behind that name guard could **not** be red-proofed
|
||||
independently — with `isSafeStackName` in front of it, no input the API accepts can reach it with an
|
||||
escaping path. It is kept as defence-in-depth and labelled in the test file as unproven-by-test
|
||||
rather than dressed up as a red-proof it does not have.
|
||||
> `docker_run_volume_path_gate` still fails on `internal/appexport/estimate.go:179`. **Pre-existing
|
||||
> on HEAD and untouched by this release** — same finding as the v0.147.x report, same file, not a
|
||||
> path this work goes near.
|
||||
|
||||
### Live (endpoint-exact, through the real handlers)
|
||||
## Red-proofs — 5 run and reverted
|
||||
|
||||
Real verification restore of `calibre-web`, then the guard matrix:
|
||||
| Mutation | Result |
|
||||
|---|---|
|
||||
| Replay removed | `0 DB dump(s) replayed` → FAIL |
|
||||
| Capture moved before the dump | `got [capture dump]` → FAIL |
|
||||
| Both undo guards removed | no refusal → FAIL |
|
||||
| Substring table matching | join table mistaken for accounts; `UserRows = 4, want 1` → FAIL |
|
||||
| Buffer-exceeding rows uncounted | wide row sniffs as empty → FAIL |
|
||||
|
||||
```
|
||||
message: "A(z) calibre-web visszaállítva ellenőrző mappába:
|
||||
/mnt/felhom-drives/hdd_1/backups/offsite-restore/calibre-web
|
||||
(a meglévő adatok változatlanok)."
|
||||
listing: calibre-web · 430.4 KB · …/offsite-restore/calibre-web · [Másolat törlése]
|
||||
**Two of these found real test weaknesses rather than confirming strength**, and both tests were
|
||||
rewritten:
|
||||
|
||||
stack=../primary -> refused; copy AND backups/primary both still present
|
||||
missing confirm -> refused; nothing changed
|
||||
real delete -> copy gone; backups/primary SURVIVES; page -> „Nincs ellenőrző másolat"
|
||||
```
|
||||
- The first undo mutation still refused — a second guard (`os.Stat` on the empty path) held. The
|
||||
red-proof only goes red when BOTH guards are removed, which is now recorded in REUSE.md.
|
||||
- The first table-matching test did not discriminate between the two matchers at all: its aggregate
|
||||
row count was identical either way. Rewritten to the case that separates them — an app with no
|
||||
accounts table but with `user_metadata` / `album_user` / `user_audit`, where a substring match
|
||||
would shout "your backup looks empty" on every restore of a healthy app.
|
||||
|
||||
---
|
||||
## Deliberately NOT done
|
||||
|
||||
## 4b — Megosztás enable shows what it is waiting for
|
||||
Per §10 of the task: no catalog-wide invariant check (stays R-41), no changes to nightly cadence,
|
||||
retention, quota math or tier-2, and no touching v0.147.x progress semantics beyond the one added
|
||||
phase line. The missing-only button's own zero-file flash is untouched — that belongs to the v0.147
|
||||
feedback arc, not R-43/R-44. Its label now states what it does *not* do, which is in scope.
|
||||
|
||||
Enabling sharing ran `ReconcileSamba()` **synchronously inside the POST handler**. On a box whose
|
||||
golden had not baked `felhom-samba` that is `compose up -d` pulling ~100MB from a private registry:
|
||||
minutes of an apparently-hung form post, then „Beállítás mentve." whether or not anything came up.
|
||||
## Live acceptance — NOT YET DONE (§9, needs Viktor)
|
||||
|
||||
Detached + polled now, with a card that distinguishes **„képfájl letöltése"** (image genuinely
|
||||
absent — the multi-minute case) from **„indítás"** (already baked — seconds). The distinction is
|
||||
decided *before* the work starts, because afterwards the image is always present and the card could
|
||||
never truthfully say „letöltés".
|
||||
The genuine red-proof is the joint run over the tunnel and has not happened: upload → „Távoli mentés
|
||||
most" → empty the trash for real → „Teljes visszaállítás (fájlok + adatbázis)" → photos visible in
|
||||
the immich timeline. Until that evidence exists:
|
||||
|
||||
- Success is **probed, not inferred** — `compose up -d` exits 0 on a crash-loop, so the terminal
|
||||
state is container liveness.
|
||||
- `nil` from reconcile also covers "deliberately deployed nothing because there is no household
|
||||
password yet"; that now gets its own message instead of a card spinning forever.
|
||||
- The **password** form starts the same job — with `UserSet` false reconcile deploys nothing, so on a
|
||||
fresh box *that*, not the enable toggle, is where the pull actually happens.
|
||||
- the capability-map offsite row stays **PARTIAL / scope-contested** (no PROVEN-LIVE flip),
|
||||
- the customer-restore row stays **MISSING**,
|
||||
- R-3 stays **DRAFT**.
|
||||
|
||||
**Test scope, stated honestly:** the unit tests cover the single-flight slot, snapshot isolation and
|
||||
the phase vocabulary. They do **not** drive `runSambaEnsureJob` end-to-end — the Server field is the
|
||||
concrete `*stacks.Manager`, not an interface, and introducing one purely for this card was more churn
|
||||
than the slice warranted. Pulling-vs-starting and the probed terminal state are covered by live
|
||||
validation instead. That seam belongs to R-45.
|
||||
**Floor raise to v0.148.0 is Viktor's click and must come BEFORE the acceptance run.**
|
||||
|
||||
Live: `/sharing/status` returns `{"ok":true,"data":{"phase":"running","running":true,"error":""}}`;
|
||||
the page carries the card and the poll.
|
||||
## Open question for Viktor
|
||||
|
||||
---
|
||||
|
||||
## 4c — „Távoli mentés most" streams real progress
|
||||
|
||||
restic was already reporting bytes and percentages; the runner seam used `CombinedOutput()` and threw
|
||||
them away. The manual run now passes `--json`, scans stdout line-by-line, and the page shows total
|
||||
bytes, percent and the app being pushed. **Manual only** — the nightly stays silent, pinned by a test
|
||||
that fails if the scheduled path ever passes `--json` or publishes progress.
|
||||
|
||||
The poll also now **arms unconditionally**. It used to start only if the page already rendered
|
||||
„Fut…", which loses a race the manual trigger always runs: the POST redirects and the page renders
|
||||
before the detached goroutine writes `LastStatus=running`, so the poll never armed and the customer
|
||||
watched a static page during the very run they had just started.
|
||||
|
||||
Red-proofed twice: break the parser -> the percent assertion fails; drop the wiring -> the `--json`
|
||||
assertion fails.
|
||||
|
||||
### What three live runs actually taught (v0.147.1 -> .3)
|
||||
|
||||
**Run 1 — immich reported `0%` for 40+ seconds.** The parser was not broken. On an incremental run
|
||||
where nothing changed restic transfers nothing: `bytes_done` is `omitempty`, so it is not even in the
|
||||
JSON, and `percent_done` stays 0. Confirmed by capturing the real schema from
|
||||
`backup --dry-run --json` on restic 0.14.0 in the controller image rather than guessing; those lines
|
||||
are now quoted verbatim in the type's doc comment. **A byte-only bar is indistinguishable from a hang
|
||||
in the common case** — shipping it would have traded "no feedback" for "feedback that says 0% and
|
||||
looks stuck". -> **v0.147.1** parses and publishes `files_done`/`total_files`.
|
||||
|
||||
**Run 2 — bookstack showed clean byte progress (100%, 154.0 MB, 7/7), immich sat at `1 of 46 files,
|
||||
0 bytes` for 42 seconds.** restic 0.14 only counts a file when it **completes**, so an app dominated
|
||||
by one large archive (immich's ~430MB volume tar) freezes *both* counters. No percentage can move.
|
||||
-> **v0.147.2** shows what restic *does* keep reporting: the file being processed and elapsed
|
||||
seconds. „1 / 46 fájl (430.2 MB) · feldolgozás alatt: immich_upload.tar · 42 mp" is a different
|
||||
message from „0%".
|
||||
|
||||
**Run 3 — the per-app legs finished in ~15s; the remaining 40 of a 57-second run was the shares leg
|
||||
and `forget --prune`,** during which the card sat frozen on „calibre-web — 8 / 8 fájl". The same
|
||||
silence, relocated to the end of the run. -> **v0.147.3** adds a run **phase**; the post-app stages
|
||||
announce themselves and app-scoped counters clear when a phase starts.
|
||||
|
||||
Final live trace — the whole 57-second run is narrated, no frozen state:
|
||||
|
||||
```
|
||||
t+6s app=immich
|
||||
t+9s app=bookstack
|
||||
t+12s app=calibre-web
|
||||
t+15s phase=shares
|
||||
t+18s ... t+54s phase=retention
|
||||
t+57s done
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Also: golden/controller infra-image drift closed at the source
|
||||
|
||||
`infra.Images()` derives the list from the existing pins; `--print-infra-images` prints it; the golden
|
||||
bake now asks the controller binary it is about to bake instead of carrying its own copy. That copy
|
||||
had already drifted — `felhom-samba` was never added to it, so the golden baked 3 of 4, **which is
|
||||
why enabling Megosztás pulled at runtime in the first place**. A test parses the const block out of
|
||||
the source with `go/ast` and fails if a pin is added without reaching `Images()`; red-proofed.
|
||||
|
||||
Verified against the real published image on the live box — all four returned.
|
||||
|
||||
## Validation method
|
||||
|
||||
Live validation was **endpoint-exact** (the exact URLs the UI posts to, through the real handlers and
|
||||
the real restic/samba paths), not browser click-through: the controller UI is password-gated and I do
|
||||
not enter credentials into web forms. No server logic was skipped — only rendering. The rendered
|
||||
markup was checked by fetching the pages themselves and asserting on the emitted HTML.
|
||||
|
||||
## Not done
|
||||
|
||||
- **Floor not raised** — Viktor decides floor timing.
|
||||
- No golden rebuild, no manifest saves, nothing on Peti's arc.
|
||||
- No SMB server-side changes — the `\\FELHOM` access issue stays client-side diagnosis first.
|
||||
- ROADMAP: **R-45** (unified async-job feedback) and **R-46** (verification-copy browse + expiry)
|
||||
registered in `felhom.eu/documentation/backlog/ROADMAP.md`.
|
||||
The capability-map row at `00-capability-map.md:61` claims PROVEN-LIVE citing `CAMPAIGN-6D`'s
|
||||
"immich end-to-end from offsite alone". This diagnosis contradicts that phrase. I flagged it
|
||||
scope-contested rather than downgrading a proven row — the ruling (did 6D exercise the DB half, or
|
||||
only the file half?) is still outstanding and is not something this implementation can settle.
|
||||
|
||||
@@ -205,6 +205,12 @@
|
||||
| `Manager.sambaUpFn` / `sambaPasswdFn` / `sambaRunFn` (func seams) | controller/internal/stacks/manager.go (fields) + samba.go | nil → `composeUp` / `docker exec smbpasswd` (STDIN) / `containerRunning("felhom-samba")` | injected in controller/internal/stacks/samba_test.go — the idempotency test asserts the up-seam is called **zero** times when config is unchanged; the passwd seam means no unit test ever handles a real secret or touches docker |
|
||||
| `Manager.sambaImgFn` (func seam) | controller/internal/stacks/manager.go (field) + samba.go | nil → `docker image inspect <infra.SambaImage>` | drives the 4b card's pulling-vs-starting decision, which MUST be taken before `compose up` (afterwards the image is always present) |
|
||||
| `Manager.offboxStreamRunner` + `SetOffboxStreamRunner` | controller/internal/backup/offbox_progress.go | nil → `defaultOffboxStreamRunner` (real `restic`, stdout scanned live) | streaming sibling of `offboxRunner`; fakes emit canned `--json` status lines in offbox_progress_test.go, so the whole progress path runs with no restic, network or repo |
|
||||
| `Manager.offsitePreDumpFn` + `SetOffsitePreDumpFn` (R-44, v0.148.0) | controller/internal/backup/offbox_reconstitute.go (seam) + offbox.go (call site) | nil → `runDBDumpsInternal` under the SAME running flag | THE dumps-before-capture ordering seam. Extracted so the order is observable without Docker/restic — an ordering guarantee no test can see is one refactor from silently reverting to the DIAG-immich-restore-2026-07-19 behaviour. Red-proof: moving the capture first yields `[capture dump]` |
|
||||
| `Manager.offboxFullPlaceCopier` + `SetOffboxFullPlaceCopier` (R-43) | controller/internal/backup/offbox_reconstitute.go | nil → `rsyncRestoreOverwrite` (`-a --itemize-changes`; **no** `--ignore-existing`, **no** `--delete`) | **TRAP: do NOT reuse `offboxPlaceCopier` here.** The two copiers have OPPOSITE semantics for an existing file — `--ignore-existing` is exactly what a full restore must not do, and conflating them is how a missing-only merge came to be labelled a restore. Never `rsyncMirror` (`--delete`) in any restore direction |
|
||||
| `Manager.safetyDumpFn` + `SetSafetyDumpFn` (R-43) | controller/internal/backup/offbox_reconstitute.go | nil → `DumpOne` | the pre-restore undo. Invariant: the `pre-restore-`-prefixed dump must be verified ON DISK before anything is stopped/overwritten/replayed; failure ⇒ refuse with zero changes. Red-proof requires removing BOTH guards (the `err != nil` return and the `os.Stat`) — removing one leaves the other holding |
|
||||
| `reimportDBDumpsFrom(ctx, stack, dumpDir)` | controller/internal/backup/restore_db.go | explicit-dir sibling of `reimportDBDumps` (which passes `AppDBDumpPath`) | offsite reconstitution replays from the SCRATCH unit: the live unit is deliberately never overwritten, so replaying from it would replay the current DB over itself and restore nothing |
|
||||
| `Manager.OffsiteScratchPair` / `OffsitePairInfo` | controller/internal/backup/offbox_reconstitute.go | reads the restored scratch unit's manifest (`offsite_run_id` / `dumps_at`) + the R-44 sniff | the confirm-dialog honesty surface. All warn-level: a pre-v0.148 (unstamped) pair and an empty-looking dump are SURFACED, never blocked — a false positive that refused a legitimate restore would be worse than the skew |
|
||||
| `appbackup.DumpValidation.LooksEmpty` (R-44 sniff) | controller/internal/appbackup/dbdump.go | computed in ValidateDump's existing single pass; `userTableNames` is EXACT-match | size and table count are both useless as emptiness heuristics (the 2026-07-19 dump: 52MB, 60+ tables, zero users — all geodata). **TRAP: never widen to a substring match on "user"** — it would flag `user_metadata` / `album_user` / `user_audit` on every healthy single-user box. A row wider than the read buffer still counts as a row |
|
||||
| `report.SetPendingControllerLog` / `SetControllerLogSource` | controller/internal/report/selftail.go | ACK-armed consume-once self-log pull (the logtail.go shape) | selftail_test.go; source = `logBuffer.Lines`, wired once in main.go |
|
||||
| `util.ParseVersion` / `util.Version.Compare` | controller/internal/util/version.go | THE one semver comparator (house rule: never a second) — selfupdate aliases it; agentapi's MinAgent comparison uses it | rejects pre-release/dev/latest (callers fall back, never trust); numeric compare (0.100 > 0.81) |
|
||||
| `agentapi.AgentVersionReporter` + `featureMinAgent` | controller/internal/agentapi/features.go | version-first Supports (v0.82.0 header channel); probe = fallback for header-less agents | a coupled feature adds BOTH a featureProbes row AND a featureMinAgent row; v0.116.0: `SupportsWithSource` also reports HOW the verdict was reached (version/probe-cache/probe) for the gate log line |
|
||||
|
||||
@@ -295,6 +295,26 @@ Each app can define rich metadata in `.felhom.yml`:
|
||||
to a data drive off the rootfs (F-A1) behind a headroom gate; unit-only default via `--include`
|
||||
the absolute unit path; `PlaceOffsiteRestore` merges a full scratch into live via
|
||||
`rsync --ignore-existing` (never `--delete`), refusing on the pure `mapOffsiteRestorePaths` guards.
|
||||
It restores FILES ONLY — no database, no stack restart — and the UI now says so.
|
||||
- **Coherent snapshot pairs (v0.148.0, R-44):** every offsite run — manual AND nightly — refreshes
|
||||
the DB/volume dumps and recovery units (`offsitePreDump` → `runDBDumpsInternal`) BEFORE the
|
||||
restic capture, so each snapshot is an internally coherent `{DB@T, files@T}` bundle and
|
||||
retention is a history of restorable points. Order is the mechanism: the gap can only add files
|
||||
the DB does not reference yet, never remove one it does. Each manifest carries `offsite_run_id`
|
||||
+ `dumps_at`; a manifest without them is a pre-v0.148 pair of unknown skew, surfaced at restore
|
||||
time. The periodic refresh carries the prior stamp forward and never invents one. A dump-leg
|
||||
failure is a loud WARN that does NOT abort the push (data-first: a degraded backup beats none).
|
||||
- **Offsite reconstitution (v0.148.0, R-43 — `offbox_reconstitute.go`):** the leg that was missing.
|
||||
`ReconstituteFromOffsite` (`/backup/offbox/reconstitute`, „Teljes visszaállítás (fájlok +
|
||||
adatbázis)") makes the live app equal to the chosen snapshot: **safety dump → stop → files
|
||||
overwritten (`rsyncRestoreOverwrite`: no `--ignore-existing`, no `--delete`) → start → the
|
||||
snapshot's dump replayed (`reimportDBDumpsFrom`, reading the SCRATCH unit) → health wait**.
|
||||
Two invariants: nothing is ever deleted (post-snapshot files survive as extras), and the
|
||||
`pre-restore-` safety dump is verified on disk BEFORE anything is stopped or overwritten — if it
|
||||
cannot be taken the operation refuses with zero changes. Safety dumps appear in `ListDumpFiles`
|
||||
(they are the undo). The live recovery unit is still never overwritten, which is why the replay
|
||||
source is the scratch. Honesty surfaces (`OffsiteScratchPair`): dump age, an unstamped-pair
|
||||
warning, and the R-44 empty-dump sniff — all warn-level, none of them gates.
|
||||
|
||||
The `/apps/{slug}` page renders hero section, screenshots, setup guide, and optional config form.
|
||||
|
||||
|
||||
@@ -51,6 +51,20 @@ type DumpValidation struct {
|
||||
Error string
|
||||
FileSize int64
|
||||
ModTime time.Time
|
||||
// R-44 (v0.148.0) content sniff — a WARN-LEVEL signal, never a gate.
|
||||
//
|
||||
// Structural validity says nothing about whether a dump holds the customer's data. The immich
|
||||
// dump of 2026-07-19 was 52MB, had a valid header and 60+ CREATE TABLEs, and contained zero
|
||||
// users and zero assets: its whole bulk was the geodata reference tables immich ships. Size and
|
||||
// table count are therefore both useless as emptiness heuristics — but an accounts table with
|
||||
// no rows is a strong, cheap, app-agnostic hint that a dump predates the customer entirely.
|
||||
//
|
||||
// Deliberately NOT a refusal: plenty of legitimate apps have no users table (UserTableFound
|
||||
// false → inconclusive → silent), and a false positive that blocked a restore would be far
|
||||
// worse than the skew it guards against. The restore confirm shows it as one extra line.
|
||||
UserTableFound bool
|
||||
UserRows int
|
||||
LooksEmpty bool // UserTableFound && UserRows == 0
|
||||
}
|
||||
|
||||
// DumpFileInfo holds info about a dump file on disk.
|
||||
@@ -363,6 +377,9 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation {
|
||||
lineNum := 0
|
||||
headerFound := false
|
||||
tableCount := 0
|
||||
// R-44 sniff state. inUserCopy tracks a postgres `COPY … FROM stdin;` block for an accounts
|
||||
// table; rows are counted until the `\.` terminator.
|
||||
inUserCopy := false
|
||||
for {
|
||||
lineBytes, isPrefix, err := reader.ReadLine()
|
||||
if err != nil {
|
||||
@@ -374,7 +391,12 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation {
|
||||
break // EOF
|
||||
}
|
||||
if isPrefix {
|
||||
// Line exceeds buffer — skip remainder (COPY data, large INSERTs)
|
||||
// Line exceeds buffer — skip remainder (COPY data, large INSERTs).
|
||||
// A long line inside a user COPY block is still a ROW: count it before discarding it,
|
||||
// or a table whose rows happen to be wide would sniff as empty and raise a false alarm.
|
||||
if inUserCopy {
|
||||
v.UserRows++
|
||||
}
|
||||
for isPrefix && err == nil {
|
||||
_, isPrefix, err = reader.ReadLine()
|
||||
}
|
||||
@@ -384,6 +406,23 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation {
|
||||
line := string(lineBytes)
|
||||
lineNum++
|
||||
|
||||
// R-44 content sniff (warn-level; see DumpValidation).
|
||||
if inUserCopy {
|
||||
if line == `\.` {
|
||||
inUserCopy = false
|
||||
} else {
|
||||
v.UserRows++
|
||||
}
|
||||
} else if isUserCopyStart(line, dbType) {
|
||||
inUserCopy = true
|
||||
v.UserTableFound = true
|
||||
} else if dbType == DBTypeMariaDB && isUserInsert(line) {
|
||||
// mysqldump writes multi-row `INSERT INTO \`users\` VALUES (…),(…);` — the row count is
|
||||
// not worth parsing out of it, and presence alone answers the only question asked here.
|
||||
v.UserTableFound = true
|
||||
v.UserRows++
|
||||
}
|
||||
|
||||
// Header check — scan first 10 lines for expected dump header
|
||||
// MariaDB 11.4+ prepends a sandbox comment before the header line
|
||||
if lineNum <= 10 && !headerFound {
|
||||
@@ -427,10 +466,65 @@ func ValidateDump(filePath string, dbType DBType) DumpValidation {
|
||||
return v
|
||||
}
|
||||
|
||||
v.LooksEmpty = v.UserTableFound && v.UserRows == 0
|
||||
if v.LooksEmpty {
|
||||
log.Printf("[WARN] [backup] ValidateDump: %s is structurally valid (%d tables) but its accounts table has NO rows — the dump may predate the customer's data", filePath, tableCount)
|
||||
}
|
||||
|
||||
v.Valid = true
|
||||
return v
|
||||
}
|
||||
|
||||
// userTableNames are the table names treated as "the accounts table" by the R-44 sniff. Kept
|
||||
// deliberately short: a wider net (anything containing "user") would match join/audit tables like
|
||||
// `user_metadata` or `album_user`, which are legitimately empty on a healthy single-user install
|
||||
// and would produce exactly the false alarm this signal must not raise.
|
||||
var userTableNames = []string{"user", "users", "account", "accounts"}
|
||||
|
||||
// isUserCopyStart reports whether a line opens a postgres `COPY <accounts-table> … FROM stdin;`
|
||||
// block. pg_dump writes the table qualified and optionally quoted — `COPY public."user" (…)`,
|
||||
// `COPY public.users (…)` — so both forms are matched.
|
||||
func isUserCopyStart(line string, dbType DBType) bool {
|
||||
if dbType != DBTypePostgres || !strings.HasPrefix(line, "COPY ") {
|
||||
return false
|
||||
}
|
||||
rest := strings.TrimPrefix(line, "COPY ")
|
||||
sp := strings.IndexByte(rest, ' ')
|
||||
if sp < 0 {
|
||||
return false
|
||||
}
|
||||
return matchesUserTable(rest[:sp])
|
||||
}
|
||||
|
||||
// isUserInsert reports whether a line is a mysqldump INSERT into an accounts table.
|
||||
func isUserInsert(line string) bool {
|
||||
const pfx = "INSERT INTO "
|
||||
if !strings.HasPrefix(line, pfx) {
|
||||
return false
|
||||
}
|
||||
rest := strings.TrimPrefix(line, pfx)
|
||||
sp := strings.IndexByte(rest, ' ')
|
||||
if sp < 0 {
|
||||
return false
|
||||
}
|
||||
return matchesUserTable(rest[:sp])
|
||||
}
|
||||
|
||||
// matchesUserTable strips schema qualification and quoting from a dumped table reference and
|
||||
// reports whether the bare name is an accounts table.
|
||||
func matchesUserTable(ref string) bool {
|
||||
if dot := strings.LastIndexByte(ref, '.'); dot >= 0 {
|
||||
ref = ref[dot+1:]
|
||||
}
|
||||
ref = strings.Trim(ref, "\"`")
|
||||
for _, n := range userTableNames {
|
||||
if strings.EqualFold(ref, n) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// ListDumpFiles returns info about SQL dump files on disk.
|
||||
//
|
||||
// M18: ValidateDump scans the dump line-by-line; on a customer with hundreds-of-MB dumps that is wasted
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
package appbackup
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-44 content-sniff tests.
|
||||
//
|
||||
// The dump that triggered this work (DIAG-immich-restore-2026-07-19) was 52MB, had a valid
|
||||
// PostgreSQL header and 60+ CREATE TABLE statements, and contained zero users and zero assets —
|
||||
// its entire bulk was immich's shipped geodata reference tables. Both of the signals the product
|
||||
// already had (file size, table count) called it healthy. These tests pin the one signal that
|
||||
// would have caught it, and the boundaries that keep it from crying wolf.
|
||||
|
||||
func writeDump(t *testing.T, body string) string {
|
||||
t.Helper()
|
||||
p := filepath.Join(t.TempDir(), "d.sql")
|
||||
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
const pgHead = `-- PostgreSQL database dump
|
||||
-- Dumped from database version 16.10
|
||||
SET statement_timeout = 0;
|
||||
SET client_encoding = 'UTF8';
|
||||
CREATE TABLE public.asset (id uuid NOT NULL);
|
||||
CREATE TABLE public."user" (id uuid NOT NULL, email text);
|
||||
`
|
||||
|
||||
// TestSniffFlagsEmptyAccountsTable is the 2026-07-19 shape: structurally perfect, no customer.
|
||||
func TestSniffFlagsEmptyAccountsTable(t *testing.T) {
|
||||
body := pgHead + "COPY public.\"user\" (id, email) FROM stdin;\n\\.\n" +
|
||||
"COPY public.asset (id) FROM stdin;\n\\.\n"
|
||||
v := ValidateDump(writeDump(t, body), DBTypePostgres)
|
||||
if !v.Valid {
|
||||
t.Fatalf("the dump is structurally valid; sniff must not change that: %s", v.Error)
|
||||
}
|
||||
if !v.UserTableFound {
|
||||
t.Fatal("the accounts table COPY block was not recognised")
|
||||
}
|
||||
if v.UserRows != 0 {
|
||||
t.Fatalf("UserRows = %d, want 0", v.UserRows)
|
||||
}
|
||||
if !v.LooksEmpty {
|
||||
t.Fatal("a valid dump with zero account rows MUST raise the warn signal — this is the whole point of R-44")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSniffQuietOnPopulatedDump — the common case must stay silent, or the warning becomes noise
|
||||
// and gets ignored precisely when it matters.
|
||||
func TestSniffQuietOnPopulatedDump(t *testing.T) {
|
||||
body := pgHead + "COPY public.\"user\" (id, email) FROM stdin;\n" +
|
||||
"a\tone@example.invalid\nb\ttwo@example.invalid\n\\.\n"
|
||||
v := ValidateDump(writeDump(t, body), DBTypePostgres)
|
||||
if v.UserRows != 2 {
|
||||
t.Fatalf("UserRows = %d, want 2", v.UserRows)
|
||||
}
|
||||
if v.LooksEmpty {
|
||||
t.Fatal("a dump with account rows must not be flagged")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSniffInconclusiveWithoutAccountsTable — plenty of legitimate apps have no users table. No
|
||||
// table, no claim: a false positive here would warn on every restore of such an app forever.
|
||||
func TestSniffInconclusiveWithoutAccountsTable(t *testing.T) {
|
||||
body := "-- PostgreSQL database dump\nCREATE TABLE public.thing (id int);\n" +
|
||||
"COPY public.thing (id) FROM stdin;\n\\.\n" + strings.Repeat("-- pad\n", 20)
|
||||
v := ValidateDump(writeDump(t, body), DBTypePostgres)
|
||||
if v.UserTableFound {
|
||||
t.Fatal("no accounts table exists — none must be reported")
|
||||
}
|
||||
if v.LooksEmpty {
|
||||
t.Fatal("an app without an accounts table must be INCONCLUSIVE, never flagged empty")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSniffIgnoresJoinAndAuditTables is the false-alarm guard that shaped the name list, and it is
|
||||
// written as the case that DISCRIMINATES: an app with NO accounts table but with `user_metadata` /
|
||||
// `album_user` / `user_audit` — all legitimately empty on a healthy box. Exact-matching leaves this
|
||||
// inconclusive (silent, correct). A substring match on "user" would treat a join table as the
|
||||
// accounts table, find zero rows, and shout "your backup looks empty" on every single restore of a
|
||||
// perfectly healthy app — which is how a warning signal becomes noise and then gets ignored.
|
||||
func TestSniffIgnoresJoinAndAuditTables(t *testing.T) {
|
||||
body := "-- PostgreSQL database dump\nCREATE TABLE public.album (id int);\n" +
|
||||
"COPY public.user_metadata (id) FROM stdin;\n\\.\n" +
|
||||
"COPY public.album_user (id) FROM stdin;\n\\.\n" +
|
||||
"COPY public.user_audit (id) FROM stdin;\n\\.\n" +
|
||||
"COPY public.album (id) FROM stdin;\n1\n\\.\n"
|
||||
v := ValidateDump(writeDump(t, body), DBTypePostgres)
|
||||
if v.UserTableFound {
|
||||
t.Fatal("a join/audit table must never be mistaken for the accounts table")
|
||||
}
|
||||
if v.LooksEmpty {
|
||||
t.Fatal("empty join/audit tables must not trigger the warning — this app has no accounts table at all")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSniffCountsOnlyTheAccountsTable pins the counting boundary separately: with a real accounts
|
||||
// table present, rows from neighbouring user-ish tables must not inflate it.
|
||||
func TestSniffCountsOnlyTheAccountsTable(t *testing.T) {
|
||||
body := pgHead +
|
||||
"COPY public.user_metadata (id) FROM stdin;\nm1\nm2\nm3\n\\.\n" +
|
||||
"COPY public.\"user\" (id, email) FROM stdin;\na\tone@example.invalid\n\\.\n"
|
||||
v := ValidateDump(writeDump(t, body), DBTypePostgres)
|
||||
if v.UserRows != 1 {
|
||||
t.Fatalf("only the real accounts table may be counted; UserRows = %d, want 1", v.UserRows)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSniffCountsWideRows — a row wider than the read buffer is skipped by the structural scan, but
|
||||
// it is still a row. Counting it wrong would flag a populated table as empty (immich asset rows are
|
||||
// genuinely long, which is what makes this reachable).
|
||||
func TestSniffCountsWideRows(t *testing.T) {
|
||||
wide := strings.Repeat("x", 300*1024)
|
||||
body := pgHead + "COPY public.\"user\" (id, email) FROM stdin;\n" + wide + "\n\\.\n"
|
||||
v := ValidateDump(writeDump(t, body), DBTypePostgres)
|
||||
if v.UserRows != 1 {
|
||||
t.Fatalf("a buffer-exceeding row must still count; UserRows = %d, want 1", v.UserRows)
|
||||
}
|
||||
if v.LooksEmpty {
|
||||
t.Fatal("a table whose single row is very wide must not sniff as empty")
|
||||
}
|
||||
}
|
||||
|
||||
// TestSniffMariaDBInsertForm — mysqldump writes multi-row INSERTs, not COPY blocks.
|
||||
func TestSniffMariaDBInsertForm(t *testing.T) {
|
||||
head := "-- MariaDB dump 10.19\nCREATE TABLE `users` (id int);\n" + strings.Repeat("-- pad\n", 20)
|
||||
empty := ValidateDump(writeDump(t, head), DBTypeMariaDB)
|
||||
if empty.UserTableFound {
|
||||
t.Fatal("a CREATE TABLE alone is not an accounts-table row source")
|
||||
}
|
||||
full := ValidateDump(writeDump(t, head+"INSERT INTO `users` VALUES (1),(2);\n"), DBTypeMariaDB)
|
||||
if !full.UserTableFound || full.LooksEmpty {
|
||||
t.Fatalf("a populated mariadb dump must not be flagged: %+v", full)
|
||||
}
|
||||
}
|
||||
@@ -59,6 +59,16 @@ type Manager struct {
|
||||
// offboxPlaceCopier (3a) — the place-to-live missing-only merge seam (nil → rsyncRestoreMissing,
|
||||
// the `-a --ignore-existing` additive copy). Never rsyncMirror (--delete trap).
|
||||
offboxPlaceCopier func(src, dst string) (int, error)
|
||||
// offboxFullPlaceCopier (R-43, v0.148.0) — the FULL-restore overwrite seam (nil →
|
||||
// rsyncRestoreOverwrite: `-a` with NO --ignore-existing and NO --delete). Distinct from
|
||||
// offboxPlaceCopier on purpose: the two have opposite semantics for an existing file.
|
||||
offboxFullPlaceCopier func(src, dst string) (int, error)
|
||||
// safetyDumpFn (R-43) — the pre-restore safety-dump seam (nil → the real DumpOne), so the
|
||||
// "never replay without an undo on disk" refusal is unit-testable without Docker.
|
||||
safetyDumpFn func(ctx context.Context, db DiscoveredDB, dumpDir string) DumpResult
|
||||
// offsitePreDumpFn (R-44) — the offsite dump pre-phase seam (nil → runDBDumpsInternal), so the
|
||||
// dumps-strictly-before-capture ordering is observable in a test without Docker or restic.
|
||||
offsitePreDumpFn func(ctx context.Context) error
|
||||
// offboxFreeFn (3a) — the free-space probe for the restore headroom gate, overridable in tests (the
|
||||
// Windows `go test` host has no `df`). Nil → the real diskFreeBytes (df --output=avail).
|
||||
offboxFreeFn func(path string) int64
|
||||
@@ -126,6 +136,13 @@ type Manager struct {
|
||||
lastDBDump *DBDumpStatus
|
||||
running bool
|
||||
|
||||
// R-43/R-44 (v0.148.0) — the coherence stamp of the offsite run in flight, read by
|
||||
// CaptureRecoveryUnit so each unit records WHICH run took the dumps sitting beside its files.
|
||||
// Set for the duration of the dump pre-phase + capture, cleared after; "" means "no offsite run
|
||||
// is establishing coherence right now" (the periodic refresh and the local 02:30 dump leg).
|
||||
offsiteRunID string
|
||||
offsiteRunDumpAt string
|
||||
|
||||
// Restore op-status (Part B, opstatus.go) — display-only async-restore progress, under `mu`.
|
||||
opRunning bool
|
||||
opName string
|
||||
@@ -310,6 +327,28 @@ func (m *Manager) RunDBDumps(ctx context.Context) error {
|
||||
return m.runDBDumpsInternal(ctx)
|
||||
}
|
||||
|
||||
// offsiteRunStamp returns the in-flight offsite run's coherence stamp ("" when none).
|
||||
func (m *Manager) offsiteRunStamp() (runID, dumpsAt string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.offsiteRunID, m.offsiteRunDumpAt
|
||||
}
|
||||
|
||||
// beginOffsiteRunStamp marks the start of an offsite run's coherence window and returns the cleanup.
|
||||
// The stamp is what CaptureRecoveryUnit writes into each unit manifest, so it must be live across
|
||||
// BOTH the dump leg and the unit capture that follows it — those two together are the pair.
|
||||
func (m *Manager) beginOffsiteRunStamp(runID string) func() {
|
||||
m.mu.Lock()
|
||||
m.offsiteRunID = runID
|
||||
m.offsiteRunDumpAt = time.Now().UTC().Format(time.RFC3339)
|
||||
m.mu.Unlock()
|
||||
return func() {
|
||||
m.mu.Lock()
|
||||
m.offsiteRunID, m.offsiteRunDumpAt = "", ""
|
||||
m.mu.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// runDBDumpsInternal is the implementation of RunDBDumps. Caller must hold the running flag.
|
||||
func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
|
||||
start := time.Now()
|
||||
|
||||
@@ -658,9 +658,40 @@ func (m *Manager) runOffboxBackup(ctx context.Context, withProgress bool) error
|
||||
m.offboxRecordStats(ctx, base, env) // the prune may have brought the size back down — refresh
|
||||
runErr = fmt.Errorf("A távoli mentés túllépte a tárhelykeretet (%d/%d GB) — törölj régi mentéseket vagy kérj nagyobb keretet.", usedGB, quota)
|
||||
} else {
|
||||
// R-43/R-44 (v0.148.0) — THE COHERENCE PRE-PHASE. Refresh the DB/volume dumps and the recovery
|
||||
// units BEFORE capturing, so the snapshot restic is about to write is an internally coherent
|
||||
// {DB@T, files@T} pair. Before this, a push shipped live files beside whatever dump the 02:30
|
||||
// local run happened to leave — on 2026-07-19 that was a dump taken four hours before the
|
||||
// customer's account even existed, so the "backup" of the photos contained zero of them
|
||||
// (DIAG-immich-restore-2026-07-19).
|
||||
//
|
||||
// Order matters and is the whole mechanism: dumps FIRST, then files. The gap between the two
|
||||
// can only ADD files the DB does not reference yet (an upload landing mid-run is a harmless
|
||||
// orphan blob), never remove one the DB DOES reference — so the file set is always a superset
|
||||
// of what the restored DB points at. The reverse order would produce dangling rows.
|
||||
//
|
||||
// This runs on the NIGHTLY path too, not just the manual one: "every snapshot is a coherent
|
||||
// pair" is the property that makes retention a history of restorable points rather than a
|
||||
// history of skewed ones. It also makes the nightly ordering structural instead of a
|
||||
// coincidence of two independent scheduler entries at 02:30 and 04:15.
|
||||
endStamp := m.beginOffsiteRunStamp(start.UTC().Format("20060102T150405Z"))
|
||||
if withProgress {
|
||||
m.offboxProgress.setPhase(OffboxPhaseDump)
|
||||
}
|
||||
dumpStart := time.Now()
|
||||
if dErr := m.offsitePreDump(ctx); dErr != nil {
|
||||
// Data-first: a dump failure must NOT abort the push. The files are still worth shipping,
|
||||
// and refusing to ship them would turn a degraded backup into no backup at all. It is a
|
||||
// loud WARN, and the unit manifest simply carries the older dump set — which the restore
|
||||
// confirm then surfaces as a skewed pair (P2) rather than silently pretending otherwise.
|
||||
m.logger.Printf("[WARN] [offbox] pre-push dump leg failed (%v) — continuing with the existing dumps; the snapshot's DB half may be older than its files", dErr)
|
||||
} else {
|
||||
m.logger.Printf("[INFO] [offbox] pre-push dump leg completed in %s — snapshot pair is coherent", time.Since(dumpStart).Round(time.Millisecond))
|
||||
}
|
||||
runResult, runErr = m.runOffboxInternal(ctx, apps, base, env, t)
|
||||
backedUp = runResult.backedUp
|
||||
missing = runResult.missing
|
||||
endStamp()
|
||||
}
|
||||
// Sorted names of apps whose enlargement was blocked this run (replaces the persisted set; empty clears).
|
||||
var blockedNames []string
|
||||
|
||||
@@ -179,8 +179,10 @@ func (p *offboxProgressState) setPhase(phase string) {
|
||||
p.mu.Unlock()
|
||||
}
|
||||
|
||||
// OffboxPhaseShares / OffboxPhaseRetention are the post-app-loop stages.
|
||||
// OffboxPhaseDump is the PRE-app-loop stage (R-44, v0.148.0); OffboxPhaseShares /
|
||||
// OffboxPhaseRetention are the post-app-loop stages.
|
||||
const (
|
||||
OffboxPhaseDump = "dump"
|
||||
OffboxPhaseShares = "shares"
|
||||
OffboxPhaseRetention = "retention"
|
||||
)
|
||||
|
||||
@@ -0,0 +1,400 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Offsite reconstitution (R-43, v0.148.0) — the leg that was missing.
|
||||
//
|
||||
// Until v0.148.0 NO offsite path could restore a database. The two „visszaállítás" buttons staged
|
||||
// files into a scratch folder and never touched postgres; the place-to-live button merged only the
|
||||
// files MISSING from the live tree (`rsync --ignore-existing`) and never replayed a dump. For a
|
||||
// DB-indexed app — most of the catalog — that combination cannot bring content back: the bytes
|
||||
// return and the application still cannot see them, because its index lives in the database.
|
||||
// Measured live on 2026-07-19 (DIAG-immich-restore-2026-07-19): 11 photos, files intact on disk,
|
||||
// timeline empty, two "successful" restores that merged 0 files.
|
||||
//
|
||||
// ReconstituteFromOffsite is the honest version of that operation: it takes the CHOSEN snapshot's
|
||||
// coherent pair and makes the live app equal to it — files overwritten to the snapshot's version,
|
||||
// database replayed from the same snapshot's dump, app restarted. It is deliberately a different
|
||||
// function from PlaceOffsiteRestore rather than a flag on it, because the two have opposite file
|
||||
// semantics and conflating them is exactly how the missing-only merge came to be presented as a
|
||||
// restore.
|
||||
//
|
||||
// Two invariants hold throughout:
|
||||
//
|
||||
// - NOTHING IS EVER DELETED. The file copy overwrites and adds; it never carries `--delete`. A
|
||||
// file the customer created after the snapshot survives the restore as an extra. That is the
|
||||
// house boundary — a restore that silently removed newer work would be a data-loss event
|
||||
// wearing a recovery button's label.
|
||||
// - THE UNDO EXISTS BEFORE THE ACT. A safety dump of the live database is written, and verified
|
||||
// present on disk, BEFORE anything is stopped, overwritten or replayed. If that dump cannot be
|
||||
// taken, the whole operation refuses with zero changes — a replay whose previous state was not
|
||||
// captured is not a restore, it is an overwrite with no way back.
|
||||
|
||||
// offsitePreDump runs the coherence pre-phase's dump leg (nil seam → runDBDumpsInternal, which also
|
||||
// refreshes the recovery units so the manifests enumerate the dumps just written). Extracted as a
|
||||
// seam because the ORDER — dumps strictly before the restic capture — is the entire mechanism of
|
||||
// R-44, and an ordering guarantee that no test can observe is one refactor away from silently
|
||||
// reverting to the behaviour that produced DIAG-immich-restore-2026-07-19.
|
||||
func (m *Manager) offsitePreDump(ctx context.Context) error {
|
||||
if m.offsitePreDumpFn != nil {
|
||||
return m.offsitePreDumpFn(ctx)
|
||||
}
|
||||
return m.runDBDumpsInternal(ctx)
|
||||
}
|
||||
|
||||
// SetOffsitePreDumpFn overrides the offsite dump pre-phase (tests; no Docker needed).
|
||||
func (m *Manager) SetOffsitePreDumpFn(fn func(ctx context.Context) error) { m.offsitePreDumpFn = fn }
|
||||
|
||||
// preRestoreDumpPrefix marks the safety dumps taken immediately before a reconstitution. They live
|
||||
// in the app's own unit db-dumps dir so `ListDumpFiles` surfaces them beside the regular dumps —
|
||||
// they ARE the undo, and an undo the customer cannot see is not much of one. The regular replay
|
||||
// loop matches `<stack>-<dbtype>.sql` exactly, so a prefixed file is never mistaken for a source.
|
||||
const preRestoreDumpPrefix = "pre-restore-"
|
||||
|
||||
// OffsiteReconstituteResult reports what a reconstitution actually did, so the flash can state an
|
||||
// OUTCOME instead of a mechanism. Every field here exists because the v0.147 flash could not say it.
|
||||
type OffsiteReconstituteResult struct {
|
||||
SnapshotID string
|
||||
FilesPlaced int
|
||||
DBsReplayed int
|
||||
SafetyDump string // path of the pre-restore dump (the undo), "" when the app has no DB
|
||||
DumpsAt time.Time // when the snapshot's DB half was taken (zero = unknown/legacy unit)
|
||||
OffsiteRunID string // "" for a pre-v0.148 snapshot — an unverified pair
|
||||
Skewed bool // the snapshot carries no coherence stamp: files and DB may differ in age
|
||||
LooksEmpty bool // R-44 sniff on the dump about to be replayed
|
||||
}
|
||||
|
||||
// fullPlaceCopier returns the FULL-restore file copier (nil seam → rsyncRestoreOverwrite).
|
||||
// Deliberately NOT placeCopier(): that one is `--ignore-existing`, whose whole purpose is to leave
|
||||
// live files alone, which is precisely what a full restore must not do.
|
||||
func (m *Manager) fullPlaceCopier() func(src, dst string) (int, error) {
|
||||
if m.offboxFullPlaceCopier != nil {
|
||||
return m.offboxFullPlaceCopier
|
||||
}
|
||||
return rsyncRestoreOverwrite
|
||||
}
|
||||
|
||||
// rsyncRestoreOverwrite copies src over dst: `rsync -a --itemize-changes`, with NO
|
||||
// `--ignore-existing` (a changed file becomes the snapshot's version) and NO `--delete` (an extra
|
||||
// file at dst survives). Returns the number of regular files transferred.
|
||||
func rsyncRestoreOverwrite(src, dst string) (int, error) {
|
||||
if err := os.MkdirAll(dst, 0755); err != nil {
|
||||
return 0, fmt.Errorf("mkdir %s: %w", dst, err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Minute)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, "rsync", "-a", "--itemize-changes",
|
||||
strings.TrimRight(src, "/")+"/", strings.TrimRight(dst, "/")+"/")
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return countRestoredFiles(string(out)), nil
|
||||
}
|
||||
|
||||
// writeSafetyDump dumps every live database of stack into the app's unit db-dumps dir under the
|
||||
// `pre-restore-` prefix, and returns the first dump's path. Returns ("", nil) when the app has no
|
||||
// database at all — a no-DB app has nothing to undo and must flow exactly as it did before
|
||||
// v0.148.0 (no dump, no replay, no behaviour change).
|
||||
//
|
||||
// A discovered database that CANNOT be dumped is a hard error: it means the undo would not exist.
|
||||
func (m *Manager) writeSafetyDump(ctx context.Context, stackName, nsRoot string) (string, error) {
|
||||
discover := m.discoverDBs
|
||||
if discover == nil {
|
||||
discover = func(ctx context.Context) ([]DiscoveredDB, error) {
|
||||
return DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames())
|
||||
}
|
||||
}
|
||||
dbs, err := discover(ctx)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("a biztonsági mentés előtt nem sikerült felderíteni az adatbázisokat: %w", err)
|
||||
}
|
||||
var mine []DiscoveredDB
|
||||
for _, db := range dbs {
|
||||
if db.StackName == stackName {
|
||||
mine = append(mine, db)
|
||||
}
|
||||
}
|
||||
if len(mine) == 0 {
|
||||
return "", nil // no DB → nothing to undo → scenario E flows unchanged
|
||||
}
|
||||
|
||||
dumpDir := AppDBDumpPath(nsRoot, stackName)
|
||||
if err := os.MkdirAll(dumpDir, 0755); err != nil {
|
||||
return "", fmt.Errorf("a biztonsági mentés könyvtára nem hozható létre: %w", err)
|
||||
}
|
||||
stamp := time.Now().UTC().Format("20060102T150405Z")
|
||||
first := ""
|
||||
for _, db := range mine {
|
||||
res := m.dumpForSafety(ctx, db, dumpDir)
|
||||
if res.Error != nil {
|
||||
return "", fmt.Errorf("a jelenlegi adatbázis biztonsági mentése sikertelen (%s): %w — a visszaállítás nem indult el", db.ContainerName, res.Error)
|
||||
}
|
||||
// DumpOne writes `<stack>-<dbtype>.sql`; rename it under the safety prefix so it can never be
|
||||
// picked up as a replay SOURCE and can never overwrite the app's real dump.
|
||||
safe := filepath.Join(dumpDir, fmt.Sprintf("%s%s-%s-%s.sql", preRestoreDumpPrefix, stamp, stackName, db.DBType))
|
||||
if res.FilePath != safe {
|
||||
if err := os.Rename(res.FilePath, safe); err != nil {
|
||||
return "", fmt.Errorf("a biztonsági mentés véglegesítése sikertelen: %w", err)
|
||||
}
|
||||
}
|
||||
if first == "" {
|
||||
first = safe
|
||||
}
|
||||
m.logger.Printf("[INFO] [offbox] %s: pre-restore safety dump written → %s (%s)", stackName, filepath.Base(safe), humanizeBytes(res.Size))
|
||||
}
|
||||
return first, nil
|
||||
}
|
||||
|
||||
// dumpForSafety is the DumpOne seam for the safety dump (tests inject; nil → the real DumpOne).
|
||||
func (m *Manager) dumpForSafety(ctx context.Context, db DiscoveredDB, dumpDir string) DumpResult {
|
||||
if m.safetyDumpFn != nil {
|
||||
return m.safetyDumpFn(ctx, db, dumpDir)
|
||||
}
|
||||
return DumpOne(ctx, db, dumpDir, m.logger, m.isDebug())
|
||||
}
|
||||
|
||||
// ReconstituteFromOffsite makes the live app equal to a restored full-scratch snapshot: files
|
||||
// overwritten to the snapshot's version (extras survive, nothing deleted), then the snapshot's own
|
||||
// DB dump replayed, with a safety dump of the current database taken first. Requires a completed
|
||||
// FULL scratch restore (RestoreOffboxScratch with full=true). Single-flight.
|
||||
func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (OffsiteReconstituteResult, error) {
|
||||
var res OffsiteReconstituteResult
|
||||
if !m.OffboxConfigured() {
|
||||
return res, fmt.Errorf("off-box backup not configured")
|
||||
}
|
||||
if !isSafeStackName(stack) {
|
||||
return res, fmt.Errorf("invalid stack name")
|
||||
}
|
||||
if m.stackProvider == nil {
|
||||
return res, fmt.Errorf("stack provider not configured")
|
||||
}
|
||||
if err := m.acquireRunning(); err != nil {
|
||||
return res, fmt.Errorf("egy másik mentési/visszaállítási művelet már fut")
|
||||
}
|
||||
defer m.releaseRunning()
|
||||
|
||||
scratch, _, err := m.offboxRestoreScratchDir(stack)
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
if _, sErr := os.Stat(scratch); sErr != nil {
|
||||
return res, fmt.Errorf("nincs előkészített teljes visszaállítás — futtass előbb egy teljes visszaállítást")
|
||||
}
|
||||
id, paths, err := m.offboxLatestSnapshot(ctx, stack)
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
res.SnapshotID = id
|
||||
|
||||
hdd := strings.TrimSpace(m.stackProvider.GetStackHDDPath(stack))
|
||||
if hdd == "" {
|
||||
return res, fmt.Errorf("a(z) %s nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat", stack)
|
||||
}
|
||||
liveNs := m.namespaceRoot(hdd)
|
||||
|
||||
placements, err := mapOffsiteRestorePaths(paths, stack, scratch, liveNs)
|
||||
if err != nil {
|
||||
return res, err // whole-placement refusal (no partial writes)
|
||||
}
|
||||
// Stat pre-pass over EVERY placement before the first copy — an incomplete scratch (e.g. only a
|
||||
// unit-only restore was run) refuses with ZERO copies.
|
||||
for _, pl := range placements {
|
||||
if _, sErr := os.Stat(pl.src); sErr != nil {
|
||||
return res, fmt.Errorf("a teljes visszaállítás hiányos (%s nincs meg) — futtass előbb egy teljes visszaállítást", filepath.Base(pl.src))
|
||||
}
|
||||
}
|
||||
|
||||
// The snapshot's coherence stamp, read from the RESTORED unit manifest (not the live one).
|
||||
scratchUnit := ""
|
||||
for _, pl := range placements {
|
||||
if pl.isUnit {
|
||||
scratchUnit = pl.src
|
||||
break
|
||||
}
|
||||
}
|
||||
if scratchUnit == "" {
|
||||
return res, fmt.Errorf("a pillanatképben nincs mentési egység — a visszaállítás nem indítható")
|
||||
}
|
||||
scratchDumpDir := filepath.Join(scratchUnit, "db-dumps")
|
||||
if man := readManifest(filepath.Join(scratchUnit, "manifest.json")); man != nil {
|
||||
res.OffsiteRunID = man.OffsiteRunID
|
||||
if man.DumpsAt != "" {
|
||||
if t, pErr := time.Parse(time.RFC3339, man.DumpsAt); pErr == nil {
|
||||
res.DumpsAt = t
|
||||
}
|
||||
}
|
||||
}
|
||||
// A pre-v0.148 snapshot carries no stamp: its dump was whatever the 02:30 local run left behind,
|
||||
// so the pair's two halves may be hours or days apart. Surfaced, never blocked — the confirm
|
||||
// dialog says so and the safety dump makes it reversible.
|
||||
res.Skewed = res.OffsiteRunID == ""
|
||||
res.LooksEmpty = m.sniffScratchDump(scratchDumpDir, stack)
|
||||
|
||||
// --- THE UNDO, BEFORE THE ACT ---------------------------------------------------------------
|
||||
// Taken while the stack is still UP (a stopped database cannot be dumped) and before a single
|
||||
// byte is overwritten, so a failure here aborts with the live app completely untouched.
|
||||
safety, err := m.writeSafetyDump(ctx, stack, liveNs)
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
res.SafetyDump = safety
|
||||
hasDB := safety != ""
|
||||
if hasDB {
|
||||
if _, sErr := os.Stat(safety); sErr != nil {
|
||||
// Fail-closed: never replay when the undo is not verifiably on disk.
|
||||
return res, fmt.Errorf("a biztonsági mentés nem található a lemezen — a visszaállítás biztonsági okból nem indult el")
|
||||
}
|
||||
}
|
||||
|
||||
// --- FILES ----------------------------------------------------------------------------------
|
||||
if err := m.stackProvider.StopStack(stack); err != nil {
|
||||
m.logger.Printf("[WARN] [offbox] could not stop %s before reconstitution: %v (continuing)", stack, err)
|
||||
}
|
||||
copier := m.fullPlaceCopier()
|
||||
for _, pl := range placements {
|
||||
if pl.isUnit {
|
||||
// The live recovery unit is still never overwritten — it is the LOCAL restore path's
|
||||
// source and clobbering it would trade one recovery route for another. The snapshot's
|
||||
// dump is replayed from the scratch unit instead, so nothing is lost by skipping it.
|
||||
continue
|
||||
}
|
||||
n, cErr := copier(pl.src, pl.dst)
|
||||
if cErr != nil {
|
||||
// Best-effort bring-up: leaving the app stopped after a partial copy would turn a failed
|
||||
// restore into an outage.
|
||||
if sErr := m.stackProvider.StartStack(stack); sErr != nil {
|
||||
m.logger.Printf("[WARN] [offbox] %s: restart after failed placement also failed: %v", stack, sErr)
|
||||
}
|
||||
return res, fmt.Errorf("a(z) %s fájljainak visszaállítása sikertelen: %w", stack, cErr)
|
||||
}
|
||||
res.FilesPlaced += n
|
||||
}
|
||||
|
||||
// --- DATABASE -------------------------------------------------------------------------------
|
||||
// The stack must be UP for the replay: ImportDump talks to the running container using its own
|
||||
// discovered credentials (the same precedence RestoreFromRecoveryUnit uses — the logical dump
|
||||
// wins over whatever the file copy just laid down for the DB's own data dir).
|
||||
if err := m.stackProvider.StartStack(stack); err != nil {
|
||||
return res, fmt.Errorf("a(z) %s újraindítása sikertelen a fájlok visszaállítása után: %w", stack, err)
|
||||
}
|
||||
if hasDB {
|
||||
n, iErr := m.reimportDBDumpsFrom(ctx, stack, scratchDumpDir)
|
||||
res.DBsReplayed = n
|
||||
if iErr != nil {
|
||||
return res, fmt.Errorf("az adatbázis visszaállítása sikertelen: %w — a korábbi állapot mentése megvan: %s", iErr, filepath.Base(safety))
|
||||
}
|
||||
}
|
||||
if err := m.waitForHealthy(stack, 90*time.Second); err != nil {
|
||||
m.logger.Printf("[WARN] [offbox] %s reconstituted but health check failed: %v", stack, err)
|
||||
}
|
||||
|
||||
m.logger.Printf("[INFO] [offbox] reconstituted %s from snapshot %s: %d file(s) placed, %d DB dump(s) replayed, safety dump=%s, skewed=%v",
|
||||
stack, id, res.FilesPlaced, res.DBsReplayed, filepath.Base(safety), res.Skewed)
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// OffsitePairInfo describes the {DB, files} pair sitting in a prepared full-restore scratch, so the
|
||||
// confirm dialog can tell the customer what they are about to restore BEFORE they commit to it.
|
||||
// Everything here is honesty-surface: none of it blocks the operation.
|
||||
type OffsitePairInfo struct {
|
||||
Ready bool
|
||||
DumpsAt time.Time // when the DB half was taken (zero = legacy unit, age unknown)
|
||||
Skewed bool // no coherence stamp → the two halves may be from different times
|
||||
LooksEmpty bool // R-44 sniff: the dump has an accounts table with no rows
|
||||
HasDump bool
|
||||
}
|
||||
|
||||
// OffsiteScratchPair reads the prepared scratch's unit manifest and reports what the pair looks
|
||||
// like. Cheap and read-only — safe to call from a page render.
|
||||
func (m *Manager) OffsiteScratchPair(stack string) OffsitePairInfo {
|
||||
var info OffsitePairInfo
|
||||
if !isSafeStackName(stack) {
|
||||
return info
|
||||
}
|
||||
scratch, _, err := m.offboxRestoreScratchDir(stack)
|
||||
if err != nil {
|
||||
return info
|
||||
}
|
||||
// The unit sits at <scratch>/<oldNs>/backups/primary/<stack>; the old namespace is unknown here,
|
||||
// so find it rather than reconstructing it.
|
||||
unit := findScratchUnitDir(scratch, stack)
|
||||
if unit == "" {
|
||||
return info
|
||||
}
|
||||
info.Ready = true
|
||||
dumpDir := filepath.Join(unit, "db-dumps")
|
||||
if entries, rErr := os.ReadDir(dumpDir); rErr == nil {
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() && filepath.Ext(e.Name()) == ".sql" && !strings.HasPrefix(e.Name(), preRestoreDumpPrefix) {
|
||||
info.HasDump = true
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if man := readManifest(filepath.Join(unit, "manifest.json")); man != nil {
|
||||
if man.DumpsAt != "" {
|
||||
if t, pErr := time.Parse(time.RFC3339, man.DumpsAt); pErr == nil {
|
||||
info.DumpsAt = t
|
||||
}
|
||||
}
|
||||
info.Skewed = man.OffsiteRunID == ""
|
||||
} else {
|
||||
info.Skewed = true
|
||||
}
|
||||
if info.HasDump {
|
||||
info.LooksEmpty = m.sniffScratchDump(dumpDir, stack)
|
||||
}
|
||||
return info
|
||||
}
|
||||
|
||||
// findScratchUnitDir locates `backups/primary/<stack>` anywhere under a restored scratch. restic
|
||||
// rebuilds absolute source paths under the target, and the snapshot may have come from a drive that
|
||||
// no longer exists on this box, so the prefix cannot be assumed.
|
||||
func findScratchUnitDir(scratch, stack string) string {
|
||||
found := ""
|
||||
suffix := filepath.Join("backups", "primary", stack)
|
||||
_ = filepath.Walk(scratch, func(path string, fi os.FileInfo, err error) error {
|
||||
if err != nil || found != "" {
|
||||
return nil //nolint:nilerr // a walk error on one branch must not abort the search
|
||||
}
|
||||
if fi.IsDir() && strings.HasSuffix(path, suffix) {
|
||||
found = path
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return found
|
||||
}
|
||||
|
||||
// sniffScratchDump runs the R-44 content sniff over the dump about to be replayed. Best-effort and
|
||||
// warn-level: any failure to read simply reports "no warning", because a sniff that blocks a
|
||||
// restore is worse than the skew it describes.
|
||||
func (m *Manager) sniffScratchDump(dumpDir, stack string) bool {
|
||||
entries, err := os.ReadDir(dumpDir)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
for _, e := range entries {
|
||||
name := e.Name()
|
||||
if e.IsDir() || filepath.Ext(name) != ".sql" || strings.HasPrefix(name, preRestoreDumpPrefix) {
|
||||
continue
|
||||
}
|
||||
dbType := DBTypePostgres
|
||||
if strings.Contains(name, string(DBTypeMariaDB)) {
|
||||
dbType = DBTypeMariaDB
|
||||
}
|
||||
if v := ValidateDump(filepath.Join(dumpDir, name), dbType); v.LooksEmpty {
|
||||
m.logger.Printf("[WARN] [offbox] %s: the snapshot dump %s has no account rows — it may predate the customer's data", stack, name)
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,424 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
)
|
||||
|
||||
// R-43/R-44 (v0.148.0) — the coherent-pair + true-restore tests.
|
||||
//
|
||||
// These exist because the product shipped a restore button for months that could not restore.
|
||||
// DIAG-immich-restore-2026-07-19: 11 photos, files intact, timeline empty, two "successful"
|
||||
// restores that merged 0 files and never touched postgres. Every test below asserts a behaviour
|
||||
// whose absence produced that outcome, so each one is a regression guard for a real incident
|
||||
// rather than a description of the current implementation.
|
||||
|
||||
// recordingProvider records stop/start call ORDER so the reconstitution sequence can be asserted.
|
||||
type recordingProvider struct {
|
||||
offbox3aProvider
|
||||
calls []string
|
||||
}
|
||||
|
||||
func (p *recordingProvider) StopStack(string) error { p.calls = append(p.calls, "stop"); return nil }
|
||||
func (p *recordingProvider) StartStack(string) error { p.calls = append(p.calls, "start"); return nil }
|
||||
|
||||
// The app really is up again after StartStack, so the post-restore health wait returns at once.
|
||||
// Leaving it false would make each test sit through the full 90s deadline.
|
||||
func (p *recordingProvider) RefreshAndIsRunning(string) bool { return true }
|
||||
|
||||
// recoveryProvider adds the recovery info CaptureRecoveryUnit needs (the shared 3a provider has none).
|
||||
type recoveryProvider struct {
|
||||
offbox3aProvider
|
||||
stackDir string
|
||||
}
|
||||
|
||||
func (p *recoveryProvider) GetStackRecoveryInfo(name string) (RecoveryInfo, bool) {
|
||||
return RecoveryInfo{DisplayName: "Immich", StackDir: p.stackDir}, name == "immich"
|
||||
}
|
||||
|
||||
// pgDump builds a structurally valid postgres dump big enough to clear ValidateDump's 100-byte
|
||||
// floor, with the accounts-table COPY block carrying `rows` rows. The R-44 sniff runs only on a
|
||||
// dump that already passes structural validation, so a toy fixture would silently skip it.
|
||||
func pgDump(rows int) string {
|
||||
const head = `-- PostgreSQL database dump
|
||||
-- Dumped from database version 16.10
|
||||
SET statement_timeout = 0;
|
||||
SET lock_timeout = 0;
|
||||
SET client_encoding = 'UTF8';
|
||||
CREATE TABLE public.asset (id uuid NOT NULL);
|
||||
CREATE TABLE public."user" (id uuid NOT NULL, email text);
|
||||
COPY public."user" (id, email) FROM stdin;
|
||||
`
|
||||
var b strings.Builder
|
||||
b.WriteString(head)
|
||||
for i := 0; i < rows; i++ {
|
||||
b.WriteString("id-x\tuser@example.invalid\n")
|
||||
}
|
||||
b.WriteString("\\.\n") // the COPY-block terminator
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// reconFixture builds a manager with a COMPLETED full scratch for `immich`, a snapshot whose unit
|
||||
// carries the given coherence stamp, and injectable copy/dump/import seams.
|
||||
func reconFixture(t *testing.T, runID, dumpsAt string, dumpBody string) (*Manager, *recordingProvider, *[]string) {
|
||||
t.Helper()
|
||||
drive := t.TempDir()
|
||||
m, sett := newOffboxManager(t)
|
||||
prov := &recordingProvider{offbox3aProvider: offbox3aProvider{
|
||||
hdd: map[string]string{"immich": drive}, binds: map[string][]ClassifiedBind{}, has: map[string]bool{},
|
||||
}}
|
||||
m.SetStackProvider(prov)
|
||||
if err := sett.AddStoragePath(settings.StoragePath{Path: drive, Label: "USB", Schedulable: true}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
scratch, liveNs, err := m.offboxRestoreScratchDir("immich")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
oldNs := "/felhomdata/ns"
|
||||
unitP := oldNs + "/backups/primary/immich"
|
||||
dataP := oldNs + "/appdata/immich"
|
||||
placements, err := mapOffsiteRestorePaths([]string{unitP, dataP}, "immich", scratch, liveNs)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, pl := range placements {
|
||||
if err := os.MkdirAll(pl.src, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pl.isUnit {
|
||||
dd := filepath.Join(pl.src, "db-dumps")
|
||||
if err := os.MkdirAll(dd, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if dumpBody != "" {
|
||||
if err := os.WriteFile(filepath.Join(dd, "immich-postgres.sql"), []byte(dumpBody), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
man := &RecoveryManifest{SchemaVersion: 1, AppName: "immich", OffsiteRunID: runID, DumpsAt: dumpsAt}
|
||||
if err := writeManifest(filepath.Join(pl.src, "manifest.json"), man); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
m.SetOffboxFreeFn(func(string) int64 { return 100 << 30 })
|
||||
m.SetOffboxSizer(func(string) int64 { return 1 << 20 })
|
||||
m.SetOffboxRunner(func(_ context.Context, _ []string, args ...string) ([]byte, error) {
|
||||
if contains(args, "snapshots") {
|
||||
return []byte(`[{"short_id":"snap1","time":"2026-07-19T06:00:00Z","paths":["` + unitP + `","` + dataP + `"]}]`), nil
|
||||
}
|
||||
return nil, nil
|
||||
})
|
||||
|
||||
// Seams: one DB, a safety dump that really writes a file, and a recording importer.
|
||||
db := DiscoveredDB{StackName: "immich", ContainerName: "immich-postgres", DBType: DBTypePostgres}
|
||||
m.discoverDBs = func(context.Context) ([]DiscoveredDB, error) { return []DiscoveredDB{db}, nil }
|
||||
m.SetSafetyDumpFn(func(_ context.Context, d DiscoveredDB, dir string) DumpResult {
|
||||
p := filepath.Join(dir, "immich-postgres.sql")
|
||||
_ = os.MkdirAll(dir, 0o755)
|
||||
_ = os.WriteFile(p, []byte(pgDump(1)), 0o644)
|
||||
return DumpResult{DB: d, FilePath: p, Size: 42}
|
||||
})
|
||||
var imported []string
|
||||
m.importDBDump = func(_ context.Context, _ DiscoveredDB, p string) error {
|
||||
imported = append(imported, p)
|
||||
return nil
|
||||
}
|
||||
m.SetOffboxFullPlaceCopier(func(_, _ string) (int, error) { return 3, nil })
|
||||
return m, prov, &imported
|
||||
}
|
||||
|
||||
// TestReconstituteReplaysDBAndOrdersOperations is Scenario C: the whole point of R-43. A restore of
|
||||
// a DB-indexed app must stop the app, place files, restart it and REPLAY the snapshot's dump — and
|
||||
// the safety dump must exist before any of it. Before v0.148.0 the replay simply did not happen,
|
||||
// which is why the photos never came back.
|
||||
func TestReconstituteReplaysDBAndOrdersOperations(t *testing.T) {
|
||||
m, prov, imported := reconFixture(t, "20260719T060000Z", "2026-07-19T06:00:00Z", pgDump(1))
|
||||
|
||||
res, err := m.ReconstituteFromOffsite(context.Background(), "immich")
|
||||
if err != nil {
|
||||
t.Fatalf("reconstitute: %v", err)
|
||||
}
|
||||
if res.DBsReplayed != 1 {
|
||||
t.Fatalf("expected the snapshot dump to be replayed exactly once, got %d — this is the R-43 defect", res.DBsReplayed)
|
||||
}
|
||||
if len(*imported) != 1 || !strings.Contains((*imported)[0], "immich-postgres.sql") {
|
||||
t.Fatalf("expected an import of the snapshot dump, got %v", *imported)
|
||||
}
|
||||
// The dump replayed must come from the SCRATCH unit, never the live one: the live unit is
|
||||
// deliberately not overwritten, so replaying from it would replay the CURRENT database back over
|
||||
// itself and restore nothing.
|
||||
if !strings.Contains((*imported)[0], "offsite-restore") {
|
||||
t.Fatalf("replay source must be the restored scratch unit, got %s", (*imported)[0])
|
||||
}
|
||||
if res.FilesPlaced != 3 {
|
||||
t.Fatalf("expected the userdata placement to be counted, got %d", res.FilesPlaced)
|
||||
}
|
||||
// stop BEFORE the file copy, start BEFORE the replay (ImportDump needs a live container).
|
||||
if got := strings.Join(prov.calls, ","); got != "stop,start" {
|
||||
t.Fatalf("expected stop then start around the restore, got %q", got)
|
||||
}
|
||||
if res.SafetyDump == "" {
|
||||
t.Fatal("no safety dump recorded — the undo must exist")
|
||||
}
|
||||
if _, err := os.Stat(res.SafetyDump); err != nil {
|
||||
t.Fatalf("safety dump not on disk: %v", err)
|
||||
}
|
||||
if !strings.HasPrefix(filepath.Base(res.SafetyDump), preRestoreDumpPrefix) {
|
||||
t.Fatalf("safety dump must carry the pre-restore prefix so it is never replayed as a source, got %s", filepath.Base(res.SafetyDump))
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconstituteRefusesWhenSafetyDumpFails is the RED-PROOF for the undo invariant: a replay whose
|
||||
// previous state was not captured is an overwrite with no way back, so it must not happen at all —
|
||||
// and it must abort with the live app untouched (no stop, no copy).
|
||||
func TestReconstituteRefusesWhenSafetyDumpFails(t *testing.T) {
|
||||
m, prov, imported := reconFixture(t, "run1", "2026-07-19T06:00:00Z", pgDump(1))
|
||||
m.SetSafetyDumpFn(func(_ context.Context, d DiscoveredDB, _ string) DumpResult {
|
||||
return DumpResult{DB: d, Error: context.DeadlineExceeded}
|
||||
})
|
||||
var copied bool
|
||||
m.SetOffboxFullPlaceCopier(func(_, _ string) (int, error) { copied = true; return 1, nil })
|
||||
|
||||
_, err := m.ReconstituteFromOffsite(context.Background(), "immich")
|
||||
if err == nil {
|
||||
t.Fatal("expected a refusal when the safety dump cannot be taken")
|
||||
}
|
||||
if len(*imported) != 0 {
|
||||
t.Fatalf("REPLAYED WITHOUT AN UNDO — the exact thing the invariant forbids: %v", *imported)
|
||||
}
|
||||
if copied {
|
||||
t.Fatal("files were overwritten despite the refusal — the abort must leave live data untouched")
|
||||
}
|
||||
if len(prov.calls) != 0 {
|
||||
t.Fatalf("the app was stopped despite the refusal, got %v", prov.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconstituteNoDBAppMakesNoDumpOrImportCalls is Scenario E: an app without a database must flow
|
||||
// exactly as before — no safety dump, no replay — so the new leg cannot regress the simple case.
|
||||
func TestReconstituteNoDBAppMakesNoDumpOrImportCalls(t *testing.T) {
|
||||
m, _, imported := reconFixture(t, "run1", "2026-07-19T06:00:00Z", "")
|
||||
m.discoverDBs = func(context.Context) ([]DiscoveredDB, error) { return nil, nil }
|
||||
dumped := 0
|
||||
m.SetSafetyDumpFn(func(_ context.Context, d DiscoveredDB, _ string) DumpResult {
|
||||
dumped++
|
||||
return DumpResult{DB: d}
|
||||
})
|
||||
|
||||
res, err := m.ReconstituteFromOffsite(context.Background(), "immich")
|
||||
if err != nil {
|
||||
t.Fatalf("reconstitute: %v", err)
|
||||
}
|
||||
if dumped != 0 {
|
||||
t.Fatalf("a no-DB app must not produce a safety dump, got %d call(s)", dumped)
|
||||
}
|
||||
if len(*imported) != 0 {
|
||||
t.Fatalf("a no-DB app must not import anything, got %v", *imported)
|
||||
}
|
||||
if res.SafetyDump != "" || res.DBsReplayed != 0 {
|
||||
t.Fatalf("unexpected DB activity: safety=%q replayed=%d", res.SafetyDump, res.DBsReplayed)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconstituteSurfacesLegacySkewedPair is Scenario D: a pre-v0.148 snapshot carries no coherence
|
||||
// stamp, so its two halves may be from different times. That must be SURFACED (and reversible), never
|
||||
// blocked — the customer's own judgement is the gate, and refusing would deny a legitimate restore.
|
||||
func TestReconstituteSurfacesLegacySkewedPair(t *testing.T) {
|
||||
m, _, imported := reconFixture(t, "", "", pgDump(1))
|
||||
|
||||
res, err := m.ReconstituteFromOffsite(context.Background(), "immich")
|
||||
if err != nil {
|
||||
t.Fatalf("a legacy pair must still be restorable, got refusal: %v", err)
|
||||
}
|
||||
if !res.Skewed {
|
||||
t.Fatal("an unstamped (pre-v0.148) snapshot must report Skewed so the confirm can say so")
|
||||
}
|
||||
if len(*imported) != 1 {
|
||||
t.Fatalf("the legacy restore must still replay, got %v", *imported)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconstituteFlagsCustomerEmptyDump is the R-44 sniff at the restore end: the immich dump that
|
||||
// started all of this was structurally valid and contained zero users. Restoring it is allowed, but
|
||||
// the customer must be told before they commit.
|
||||
func TestReconstituteFlagsCustomerEmptyDump(t *testing.T) {
|
||||
// A valid postgres dump whose accounts table has NO rows — the 2026-07-19 shape exactly.
|
||||
m, _, _ := reconFixture(t, "run1", "2026-07-19T06:00:00Z", pgDump(0))
|
||||
|
||||
res, err := m.ReconstituteFromOffsite(context.Background(), "immich")
|
||||
if err != nil {
|
||||
t.Fatalf("the sniff must never block a restore: %v", err)
|
||||
}
|
||||
if !res.LooksEmpty {
|
||||
t.Fatal("a dump with an empty accounts table must raise the warn-level signal")
|
||||
}
|
||||
}
|
||||
|
||||
// TestOffsiteScratchPairReportsWhatTheConfirmNeeds covers the page-render surface: the confirm can
|
||||
// only be honest if this reports the pair's age and warnings before anything is started.
|
||||
func TestOffsiteScratchPairReportsWhatTheConfirmNeeds(t *testing.T) {
|
||||
m, _, _ := reconFixture(t, "run1", "2026-07-19T06:00:00Z",
|
||||
"-- PostgreSQL database dump\nCREATE TABLE a();\nCOPY public.\"user\" (id) FROM stdin;\n7\n\\.\n")
|
||||
|
||||
info := m.OffsiteScratchPair("immich")
|
||||
if !info.Ready || !info.HasDump {
|
||||
t.Fatalf("expected a ready pair with a dump, got %+v", info)
|
||||
}
|
||||
if info.Skewed {
|
||||
t.Fatal("a stamped snapshot must not be reported as skewed")
|
||||
}
|
||||
if info.LooksEmpty {
|
||||
t.Fatal("a dump with account rows must not be flagged empty")
|
||||
}
|
||||
want, _ := time.Parse(time.RFC3339, "2026-07-19T06:00:00Z")
|
||||
if !info.DumpsAt.Equal(want) {
|
||||
t.Fatalf("DumpsAt = %v, want %v", info.DumpsAt, want)
|
||||
}
|
||||
}
|
||||
|
||||
// --- R-44: the coherence pre-phase -----------------------------------------------------------
|
||||
|
||||
// TestOffsiteRunDumpsBeforeCapture is Scenarios A + B. The ORDER is the entire mechanism: dumps
|
||||
// must be refreshed BEFORE restic captures, so the snapshot pairs this run's database with this
|
||||
// run's files. Reversed, the snapshot would hold rows pointing at files that were never captured.
|
||||
//
|
||||
// It also asserts the ordering on the NIGHTLY entry point (RunOffboxBackup, no progress sink), not
|
||||
// just the manual one — before v0.148.0 the nightly ordering was an accident of two independent
|
||||
// scheduler entries at 02:30 and 04:15, which a schedule edit could silently invert.
|
||||
func TestOffsiteRunDumpsBeforeCapture(t *testing.T) {
|
||||
drive := t.TempDir()
|
||||
m, sett, prov := classifiedOffboxManager(t, drive)
|
||||
mkUnit(t, drive, "immich")
|
||||
if err := os.MkdirAll(filepath.Join(drive, "appdata", "immich"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prov.hdd["immich"] = drive
|
||||
prov.has["immich"] = true
|
||||
prov.binds["immich"] = []ClassifiedBind{mandatoryHDD("appdata/immich")}
|
||||
_ = sett.SetAppOffbox("immich", true)
|
||||
|
||||
var order []string
|
||||
m.SetOffsitePreDumpFn(func(context.Context) error {
|
||||
order = append(order, "dump")
|
||||
return nil
|
||||
})
|
||||
m.SetOffboxRunner(func(_ context.Context, _ []string, args ...string) ([]byte, error) {
|
||||
switch {
|
||||
case contains(args, "cat") && contains(args, "config"):
|
||||
return []byte(`{"version":2}`), nil
|
||||
case contains(args, "backup"):
|
||||
order = append(order, "capture")
|
||||
return nil, nil
|
||||
case contains(args, "snapshots"):
|
||||
return []byte(`[]`), nil
|
||||
case contains(args, "stats"):
|
||||
return []byte(`{"total_size":123}`), nil
|
||||
}
|
||||
return nil, nil
|
||||
})
|
||||
|
||||
if err := m.RunOffboxBackup(context.Background()); err != nil {
|
||||
t.Fatalf("run: %v", err)
|
||||
}
|
||||
if len(order) < 2 {
|
||||
t.Fatalf("expected both a dump and a capture, got %v", order)
|
||||
}
|
||||
if order[0] != "dump" {
|
||||
t.Fatalf("the dump leg MUST precede the capture (R-44); got %v", order)
|
||||
}
|
||||
if order[1] != "capture" {
|
||||
t.Fatalf("expected the capture immediately after the dump, got %v", order)
|
||||
}
|
||||
}
|
||||
|
||||
// TestOffsiteRunContinuesWhenDumpLegFails is the data-first rule: a dump failure degrades the
|
||||
// snapshot's DB half but must NOT abort the push. Refusing to ship the files would turn a partial
|
||||
// backup into no backup at all — strictly worse for the customer.
|
||||
func TestOffsiteRunContinuesWhenDumpLegFails(t *testing.T) {
|
||||
drive := t.TempDir()
|
||||
m, sett, prov := classifiedOffboxManager(t, drive)
|
||||
mkUnit(t, drive, "immich")
|
||||
if err := os.MkdirAll(filepath.Join(drive, "appdata", "immich"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prov.hdd["immich"] = drive
|
||||
prov.has["immich"] = true
|
||||
prov.binds["immich"] = []ClassifiedBind{mandatoryHDD("appdata/immich")}
|
||||
_ = sett.SetAppOffbox("immich", true)
|
||||
|
||||
m.SetOffsitePreDumpFn(func(context.Context) error { return context.DeadlineExceeded })
|
||||
cap := &backupCapture{}
|
||||
m.SetOffboxRunner(cap.runner())
|
||||
|
||||
if err := m.RunOffboxBackup(context.Background()); err != nil {
|
||||
t.Fatalf("a dump failure must not fail the whole run: %v", err)
|
||||
}
|
||||
if cap.backups != 1 {
|
||||
t.Fatalf("the files must still be pushed after a dump failure, got %d capture(s)", cap.backups)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCaptureRecoveryUnitStampsAndCarriesRunID covers the stamp that makes a pair verifiable at
|
||||
// restore time, and the trap beside it: the PERIODIC refresh must neither invent a coherence claim
|
||||
// nor erase one a real run established.
|
||||
func TestCaptureRecoveryUnitStampsAndCarriesRunID(t *testing.T) {
|
||||
drive := t.TempDir()
|
||||
m, _, base := classifiedOffboxManager(t, drive)
|
||||
base.hdd["immich"] = drive
|
||||
// CaptureRecoveryUnit needs real recovery info + a compose dir to read; the shared fixture
|
||||
// provider returns none, so wrap it rather than widening a struct four other test files use.
|
||||
stackDir := filepath.Join(t.TempDir(), "immich")
|
||||
if err := os.MkdirAll(stackDir, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(stackDir, "docker-compose.yml"), []byte("services: {}\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m.SetStackProvider(&recoveryProvider{offbox3aProvider: *base, stackDir: stackDir})
|
||||
|
||||
// 1) A run in flight stamps the manifest.
|
||||
end := m.beginOffsiteRunStamp("run-A")
|
||||
if err := m.CaptureRecoveryUnit("immich"); err != nil {
|
||||
t.Fatalf("capture: %v", err)
|
||||
}
|
||||
end()
|
||||
man := readManifest(RecoveryUnitManifestPath(drive, "immich"))
|
||||
if man == nil || man.OffsiteRunID != "run-A" {
|
||||
t.Fatalf("expected the in-flight run id to be stamped, got %+v", man)
|
||||
}
|
||||
if man.DumpsAt == "" {
|
||||
t.Fatal("a stamped unit must record when its dumps were taken")
|
||||
}
|
||||
|
||||
// 2) A periodic refresh (no run in flight) must CARRY the stamp forward, not blank it — a unit
|
||||
// that silently lost its stamp would be re-reported as a skewed legacy pair at restore time.
|
||||
if err := m.CaptureRecoveryUnit("immich"); err != nil {
|
||||
t.Fatalf("refresh: %v", err)
|
||||
}
|
||||
man2 := readManifest(RecoveryUnitManifestPath(drive, "immich"))
|
||||
if man2 == nil || man2.OffsiteRunID != "run-A" {
|
||||
t.Fatalf("the periodic refresh erased the coherence stamp: %+v", man2)
|
||||
}
|
||||
|
||||
// 3) A NEW run re-stamps even though nothing else about the unit changed — the idempotent-skip
|
||||
// must not swallow the one field the restore path reads.
|
||||
end2 := m.beginOffsiteRunStamp("run-B")
|
||||
if err := m.CaptureRecoveryUnit("immich"); err != nil {
|
||||
t.Fatalf("capture 2: %v", err)
|
||||
}
|
||||
end2()
|
||||
man3 := readManifest(RecoveryUnitManifestPath(drive, "immich"))
|
||||
if man3 == nil || man3.OffsiteRunID != "run-B" {
|
||||
t.Fatalf("a new run must re-stamp the unit, got %+v", man3)
|
||||
}
|
||||
}
|
||||
@@ -30,6 +30,16 @@ const (
|
||||
// SetOffboxFreeFn overrides the restore free-space probe (tests; the Windows go-test host has no df).
|
||||
func (m *Manager) SetOffboxFreeFn(fn func(path string) int64) { m.offboxFreeFn = fn }
|
||||
|
||||
// SetOffboxFullPlaceCopier overrides the FULL-restore overwrite copier (tests; no rsync needed).
|
||||
func (m *Manager) SetOffboxFullPlaceCopier(fn func(src, dst string) (int, error)) {
|
||||
m.offboxFullPlaceCopier = fn
|
||||
}
|
||||
|
||||
// SetSafetyDumpFn overrides the pre-restore safety dump (tests; no Docker needed).
|
||||
func (m *Manager) SetSafetyDumpFn(fn func(ctx context.Context, db DiscoveredDB, dumpDir string) DumpResult) {
|
||||
m.safetyDumpFn = fn
|
||||
}
|
||||
|
||||
// offboxFree returns the free-space probe (nil seam → the real diskFreeBytes).
|
||||
func (m *Manager) offboxFree() func(string) int64 {
|
||||
if m.offboxFreeFn != nil {
|
||||
|
||||
@@ -44,6 +44,15 @@ type RecoveryManifest struct {
|
||||
DBDumps []string `json:"db_dumps"`
|
||||
VolumeDumps []string `json:"volume_dumps"`
|
||||
Checksums map[string]string `json:"checksums"` // sha256 of captured compose/ files
|
||||
// R-43/R-44 (v0.148.0): the coherence stamp. An offsite run refreshes the dumps FIRST and then
|
||||
// captures the unit, so a manifest carrying an OffsiteRunID asserts "the db-dumps/ in this unit
|
||||
// were taken by that run" — i.e. the snapshot is an internally coherent {DB@T, files@T} pair.
|
||||
// A manifest WITHOUT these fields is a pre-v0.148 unit whose dump age is unknown and may skew
|
||||
// arbitrarily from the files beside it (the DIAG-immich-restore-2026-07-19 failure); the restore
|
||||
// confirm surfaces that honestly rather than blocking. Empty on the periodic refresh, which must
|
||||
// never claim a coherence it did not establish — it carries the prior stamp forward instead.
|
||||
OffsiteRunID string `json:"offsite_run_id,omitempty"`
|
||||
DumpsAt string `json:"dumps_at,omitempty"` // RFC3339 UTC — when this run's dump leg finished
|
||||
}
|
||||
|
||||
// SetVersion records the controller version stamped into recovery-unit manifests.
|
||||
@@ -107,13 +116,26 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
|
||||
version := m.versionLocked()
|
||||
|
||||
manifestPath := RecoveryUnitManifestPath(nsRoot, stackName)
|
||||
cur := readManifest(manifestPath)
|
||||
|
||||
// R-43/R-44: the coherence stamp of the offsite run currently in flight ("" on the periodic
|
||||
// refresh and on the local dump run). When empty we CARRY THE PRIOR STAMP FORWARD rather than
|
||||
// blanking it — a periodic refresh must neither claim a coherence it did not establish nor
|
||||
// destroy the record of one that a real run did.
|
||||
runID, dumpsAt := m.offsiteRunStamp()
|
||||
if runID == "" && cur != nil {
|
||||
runID, dumpsAt = cur.OffsiteRunID, cur.DumpsAt
|
||||
}
|
||||
|
||||
// Skip if the unit is already current — avoids needless drive writes on the periodic refresh.
|
||||
if cur := readManifest(manifestPath); cur != nil &&
|
||||
// The run-id is part of "current": an offsite run must re-stamp the manifest even when nothing
|
||||
// else changed, because the stamp is exactly the claim the restore path reads.
|
||||
if cur != nil &&
|
||||
cur.ControllerVer == version &&
|
||||
stringMapEqual(cur.Checksums, checksums) &&
|
||||
stringSliceEqual(cur.DBDumps, dbDumps) &&
|
||||
stringSliceEqual(cur.VolumeDumps, volDumps) {
|
||||
stringSliceEqual(cur.VolumeDumps, volDumps) &&
|
||||
cur.OffsiteRunID == runID {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -143,6 +165,8 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
|
||||
DBDumps: dbDumps,
|
||||
VolumeDumps: volDumps,
|
||||
Checksums: checksums,
|
||||
OffsiteRunID: runID,
|
||||
DumpsAt: dumpsAt,
|
||||
}
|
||||
if err := writeManifest(manifestPath, manifest); err != nil {
|
||||
return fmt.Errorf("writing manifest: %w", err)
|
||||
|
||||
@@ -19,7 +19,15 @@ import (
|
||||
// A dump whose DB container is not found is logged and skipped; an actual import FAILURE is returned
|
||||
// (surfaced, not swallowed) so a failed data restore cannot read as success.
|
||||
func (m *Manager) reimportDBDumps(ctx context.Context, stackName, nsRoot string) (int, error) {
|
||||
dumpDir := AppDBDumpPath(nsRoot, stackName)
|
||||
return m.reimportDBDumpsFrom(ctx, stackName, AppDBDumpPath(nsRoot, stackName))
|
||||
}
|
||||
|
||||
// reimportDBDumpsFrom is reimportDBDumps with an EXPLICIT dump directory. The offsite
|
||||
// reconstitution path (R-43) replays out of the restored SCRATCH unit rather than the live one:
|
||||
// the local unit is deliberately never overwritten by a placement, so the dump that belongs to the
|
||||
// chosen snapshot exists only under the scratch. Same discovery/import seams, same failure
|
||||
// semantics — only the source directory differs.
|
||||
func (m *Manager) reimportDBDumpsFrom(ctx context.Context, stackName, dumpDir string) (int, error) {
|
||||
entries, err := os.ReadDir(dumpDir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
|
||||
@@ -815,6 +815,17 @@ func (s *Server) backupsRestoreHandler(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
}
|
||||
data["OffboxScratchReady"] = ready
|
||||
// R-43: the same prepared scratch also enables the TRUE restore (files + database). Its confirm
|
||||
// has to state what the pair actually IS — how old the DB half is, whether the two halves even
|
||||
// come from the same run, and whether the dump looks customer-empty — because a restore is the
|
||||
// one operation whose result the customer cannot inspect until after committing to it.
|
||||
pairs := map[string]backup.OffsitePairInfo{}
|
||||
if s.backupMgr != nil {
|
||||
for name := range ready {
|
||||
pairs[name] = s.backupMgr.OffsiteScratchPair(name)
|
||||
}
|
||||
}
|
||||
data["OffboxPairInfo"] = pairs
|
||||
// R-7b: the shares source is not an app — it has no per-app toggle and no recovery unit — so it
|
||||
// gets its own restore entry rather than a synthetic row in OffboxApps (which would also make it
|
||||
// appear in the per-app offsite TOGGLE list on /backups/remote, where it does not belong).
|
||||
|
||||
@@ -3,6 +3,7 @@ package web
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
@@ -340,6 +341,69 @@ func (s *Server) offboxRestoreHandler(w http.ResponseWriter, r *http.Request) {
|
||||
offboxRedirectTo(w, r, "/backups/restore", "A távoli visszaállítás elindult — az állapot itt frissül.", false)
|
||||
}
|
||||
|
||||
// offboxReconstituteHandler is the TRUE offsite restore (R-43, v0.148.0): files overwritten to the
|
||||
// snapshot's version + that same snapshot's database replayed + the app restarted, with a safety
|
||||
// dump of the current database taken first.
|
||||
//
|
||||
// It is a separate button from the missing-only place, not a flag on it. The two do opposite things
|
||||
// to an existing file, and the v0.147 flash („hiányzó fájljai helyreállítva") described a mechanism
|
||||
// that could report success after merging zero files while the customer's photos stayed invisible.
|
||||
// The flash here states the OUTCOME instead — file count, database, backup timestamp, restart —
|
||||
// because that is the only part the customer can check against what they see in the app.
|
||||
func (s *Server) offboxReconstituteHandler(w http.ResponseWriter, r *http.Request) {
|
||||
if s.backupMgr == nil || !s.backupMgr.OffboxConfigured() {
|
||||
offboxRedirectTo(w, r, "/backups/restore", "A távoli mentési cél nincs beállítva.", true)
|
||||
return
|
||||
}
|
||||
_ = r.ParseForm()
|
||||
app := strings.TrimSpace(r.FormValue("app"))
|
||||
if app == "" {
|
||||
offboxRedirectTo(w, r, "/backups/restore", "Hiányzó alkalmazás.", true)
|
||||
return
|
||||
}
|
||||
// This one overwrites live files and replays a database — it must never happen on a stray click.
|
||||
if r.FormValue("confirm") != "1" {
|
||||
offboxRedirectTo(w, r, "/backups/restore", "A teljes visszaállítás megerősítés nélkül nem hajtható végre.", true)
|
||||
return
|
||||
}
|
||||
if s.backupMgr.IsRunning() {
|
||||
offboxRedirectTo(w, r, "/backups/restore", "Egy mentési/visszaállítási művelet már fut.", true)
|
||||
return
|
||||
}
|
||||
s.backupMgr.BeginRestoreOp("offbox-reconstitute", app)
|
||||
go func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Minute)
|
||||
defer cancel()
|
||||
res, err := s.backupMgr.ReconstituteFromOffsite(ctx, app)
|
||||
if err != nil {
|
||||
s.logger.Printf("[ERROR] [web] off-box reconstitute %s (async): %v", app, err)
|
||||
s.backupMgr.EndRestoreOp(false, "A teljes visszaállítás sikertelen: "+err.Error())
|
||||
return
|
||||
}
|
||||
s.logger.Printf("[INFO] [web] off-box reconstitute %s completed (async): files=%d dbs=%d snapshot=%s",
|
||||
app, res.FilesPlaced, res.DBsReplayed, res.SnapshotID)
|
||||
s.backupMgr.EndRestoreOp(true, reconstituteOutcomeMsg(app, res))
|
||||
}()
|
||||
offboxRedirectTo(w, r, "/backups/restore", "A teljes visszaállítás elindult — az állapot itt frissül.", false)
|
||||
}
|
||||
|
||||
// reconstituteOutcomeMsg builds the OUTCOME flash for a completed reconstitution. Pure, so the
|
||||
// wording is unit-testable — this string is the customer's only evidence that the operation did
|
||||
// what its label promised, and the zero-file and no-database cases must each read truthfully rather
|
||||
// than borrowing the confident sentence that belongs to the full case.
|
||||
func reconstituteOutcomeMsg(app string, res backup.OffsiteReconstituteResult) string {
|
||||
when := ""
|
||||
if !res.DumpsAt.IsZero() {
|
||||
when = " (mentés: " + res.DumpsAt.In(getTimezone()).Format("2006-01-02 15:04") + ")"
|
||||
}
|
||||
if res.DBsReplayed == 0 {
|
||||
// A no-database app: saying "és az adatbázis" here would be a lie, and this is precisely the
|
||||
// class of sentence the DIAG found being printed over a no-op.
|
||||
return fmt.Sprintf("A(z) %s: %d fájl visszaállítva%s — az alkalmazás újraindult. Ennek az alkalmazásnak nincs adatbázisa.", app, res.FilesPlaced, when)
|
||||
}
|
||||
return fmt.Sprintf("A(z) %s: %d fájl és az adatbázis visszaállítva%s — az alkalmazás újraindult.", app, res.FilesPlaced, when)
|
||||
}
|
||||
|
||||
// offboxVerifyCopyDeleteHandler removes ONE verification copy (v0.147.0, 4a).
|
||||
//
|
||||
// The only delete this slice adds, so it is deliberately narrow: it names a STACK, never a path — the
|
||||
|
||||
@@ -422,6 +422,8 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
s.offboxRestoreHandler(w, r)
|
||||
case path == "/backup/offbox/place" && r.Method == http.MethodPost:
|
||||
s.offboxPlaceHandler(w, r)
|
||||
case path == "/backup/offbox/reconstitute" && r.Method == http.MethodPost:
|
||||
s.offboxReconstituteHandler(w, r)
|
||||
// v0.147.0 (4a): remove ONE verification copy. The only delete path this slice adds — see
|
||||
// DeleteOffsiteRestoreCopy for the prefix guard.
|
||||
case path == "/backup/offbox/verify-copy/delete" && r.Method == http.MethodPost:
|
||||
|
||||
@@ -215,6 +215,16 @@
|
||||
/* A run is not only the per-app loop: the shares leg and the retention/prune step follow
|
||||
it and can dominate the wall clock (40 of 57 seconds, measured). Name them, or the
|
||||
card freezes on the last app's finished counters for that whole tail. */
|
||||
/* The dump pre-phase runs BEFORE the per-app loop (R-44): it refreshes every app's
|
||||
database dump so the snapshot pairs this run's files with this run's DB. On an app
|
||||
with a large database it can dominate the early wall clock, and an unnamed silent
|
||||
stretch at the start reads as a hang. */
|
||||
if(p.phase === 'dump'){
|
||||
text.textContent = 'Adatbázisok mentése a pillanatképhez…';
|
||||
bar.style.width = '100%';
|
||||
box.className = 'alert alert-info'; box.style.display = '';
|
||||
return;
|
||||
}
|
||||
if(p.phase === 'retention'){
|
||||
text.textContent = 'Karbantartás: régi mentések rendezése a távoli tárolón…';
|
||||
bar.style.width = '100%';
|
||||
|
||||
@@ -93,7 +93,25 @@
|
||||
<input type="hidden" name="app" value="{{.Name}}">
|
||||
<button type="submit" class="btn btn-xs btn-outline">Helyreállítás az élő adatok közé (csak a hiányzó fájlok)</button>
|
||||
</form>
|
||||
<span class="form-hint" style="display:block;margin-top:.25rem">A meglévő fájlokat nem írja felül.</span>
|
||||
<span class="form-hint" style="display:block;margin-top:.25rem">A meglévő fájlokat nem írja felül. Adatbázist nem állít vissza — törölt tartalom ettől nem jelenik meg újra.</span>
|
||||
{{$pair := index $.OffboxPairInfo .Name}}
|
||||
<form method="POST" action="/backup/offbox/reconstitute" style="display:inline">{{$.CSRFField}}
|
||||
<input type="hidden" name="app" value="{{.Name}}">
|
||||
<input type="hidden" name="confirm" value="1">
|
||||
<button type="button" class="btn btn-xs btn-primary"
|
||||
data-restore-app="{{.Name}}"
|
||||
data-restore-when="{{if not $pair.DumpsAt.IsZero}}{{fmtTime $pair.DumpsAt}}{{end}}"
|
||||
data-restore-skewed="{{if $pair.Skewed}}1{{end}}"
|
||||
data-restore-empty="{{if $pair.LooksEmpty}}1{{end}}"
|
||||
onclick="confirmFullRestore(this)">Teljes visszaállítás (fájlok + adatbázis)</button>
|
||||
</form>
|
||||
<span class="form-hint" style="display:block;margin-top:.25rem">A fájlokat a mentés szerinti változatra állítja vissza és az adatbázist is visszatölti. Semmit nem töröl: a mentés óta létrejött fájlok megmaradnak. A jelenlegi adatbázisról előtte biztonsági mentés készül.</span>
|
||||
{{if $pair.Skewed}}
|
||||
<span class="form-hint" style="display:block;margin-top:.25rem;color:var(--warn)">Az adatbázis-mentés régebbi{{if not $pair.DumpsAt.IsZero}} ({{fmtTime $pair.DumpsAt}}){{end}} — a fájlok és az adatbázis eltérő időpontból származnak.</span>
|
||||
{{end}}
|
||||
{{if $pair.LooksEmpty}}
|
||||
<span class="form-hint" style="display:block;margin-top:.25rem;color:var(--warn)">A mentett adatbázis üresnek tűnik (nincs benne felhasználói fiók) — elképzelhető, hogy a mentés korábbi, mint az adataid.</span>
|
||||
{{end}}
|
||||
{{end}}{{end}}
|
||||
{{template "app_list_row_end"}}
|
||||
{{end}}
|
||||
@@ -217,6 +235,26 @@ function confirmDeleteVerifyCopy(btn){
|
||||
});
|
||||
});
|
||||
}
|
||||
/* R-43: the true offsite restore overwrites live files and replays a database, so it double-confirms
|
||||
and — unlike the old missing-only merge — states the DB half's age and any warning BEFORE the
|
||||
customer commits. The honesty lines are already rendered under the button; repeating the decisive
|
||||
ones here means the person clicking "Igen" has read them. */
|
||||
function confirmFullRestore(btn){
|
||||
var app = btn.getAttribute('data-restore-app') || '';
|
||||
var when = btn.getAttribute('data-restore-when') || '';
|
||||
var skewed = btn.getAttribute('data-restore-skewed') === '1';
|
||||
var empty = btn.getAttribute('data-restore-empty') === '1';
|
||||
var q = 'Teljes visszaállítás: ' + app + (when ? ' — a mentés ideje: ' + when : '') + '.';
|
||||
if (skewed) { q += ' FIGYELEM: a fájlok és az adatbázis eltérő időpontból származnak.'; }
|
||||
if (empty) { q += ' FIGYELEM: a mentett adatbázis üresnek tűnik.'; }
|
||||
q += ' A fájlok a mentés szerinti változatra állnak vissza, semmi nem törlődik.';
|
||||
felhomConfirm(btn, q, function(){
|
||||
felhomConfirm(btn, 'UTOLSÓ MEGERŐSÍTÉS: az alkalmazás leáll, az adatbázis visszatöltődik, majd újraindul. A jelenlegi adatbázisról biztonsági mentés készül.', function(){
|
||||
var f = btn.closest('form');
|
||||
if (f) { if (f.requestSubmit) f.requestSubmit(); else f.submit(); }
|
||||
});
|
||||
});
|
||||
}
|
||||
function fabStart(stack, next){
|
||||
fetch('/api/export/download/start', {method:'POST', headers:Object.assign({'Content-Type':'application/json'}, csrfHeaders()), body: JSON.stringify({stack_name: stack, password: fabPassword()})})
|
||||
.then(function(r){ return r.json(); })
|
||||
|
||||
Reference in New Issue
Block a user