diff --git a/CHANGELOG.md b/CHANGELOG.md index e19b701..258fc51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,28 @@ -## Unreleased (2026-10-19) — the security-scan gate (R-986); no code change +## v0.336.0 — a failed off-site restore puts the app back exactly as it was and starts it (R-893 option A, operator D20); the held sentence says only what is certain (R-1169) (2026-10-20) + +**MinAgent: 0.158.1** — the floor's, carried; no agent change. No golden (`09` §3 decision 250). Task: the night brief +part 2 of 2026-10-20, Part 2 (operator answer: „D20 A"). + +- **R-893 option A — the off-site restore of one app keeps a copy of the app's live volumes and puts it back on + failure** (`internal/backup/offbox_restore_undo.go`, `07` §6.3). Before anything stops, every volume the snapshot will + replace is measured (`du` on the named volume) and the copy is added to the restore's own free-space need (R-357's + exact rule): too little room → refused with `err.backup.restore_undo_no_space`, nothing stopped; an unmeasurable + volume → `err.backup.restore_undo_unmeasured`. Right after the stop each such volume is tarred (the nightly helper, + tmp + fsync + rename) into `/backups/restore-undo///` — outside every capture set. A copy that + fails restarts the app as it was (`err.backup.restore_undo_copy_failed`). On ANY failure after the first volume or + the definition moved (file copy, volume replay — also at a count of 0, because the replay removes a volume before it + recreates it —, DB-only start, replay after a successful rollback) the box stops the app, pours the copies back (the + restore's own volume helper), removes a volume the snapshot added, writes the live definition back and starts the + app: „Nem változott semmi…" (`err.backup.restore_failed_put_back`; with placed files `…_put_back_files` says the files + stay; a start that fails `…_put_back_not_started`). The operator gets `backup_run_failures` leg `restore-put-back` + (no new event type). A put-back that fails keeps the copy for support and HOLDS the app (option C, unchanged); the + double failure (replay and rollback) holds as before and keeps the copy too. On success the copy is removed. Tests + (`r893_putback_test.go`, 7 new; `r893_mixed_restore_test.go` moved to option A's outcome) — all nine red-proved by a + mutation each (felhom.eu `audits/night2-2026-10-20/P2/`). +- **R-1169 — the held sentence** `err.backup.db_restore_failed_held_mixed` now says only what is certain (the restore + stopped half-way, the data could not be returned, the app stays stopped, the copy is kept, support helps). The + plain rolled-back case with placed files no longer says „your data is back as it was" — it uses `…_put_back_files`. +- **Earlier unreleased (2026-10-19) — the security-scan gate (R-986); no code change:** - `controller_gates.py` registers `security-scan` (FAST set, so the pre-push hook and CI run it): the SHARED `felhom.eu/scripts/security_scan_gate.py` — govulncheck (called vulnerabilities) on `controller/` and gitleaks on the diff --git a/REUSE.md b/REUSE.md index 431f27a..f413d2c 100644 --- a/REUSE.md +++ b/REUSE.md @@ -81,6 +81,7 @@ | `restoreWizardPath` / `deriveWizardStep` / `resolveWizardApp` | controller/internal/web/restore_wizard.go | `(app) string` / `(restoreWizardInput) restoreWizardView` / `([]OffboxAppRow, name) *OffboxAppRow` | R-48 offsite restore wizard: URL builder + the PURE step/unlock derivation + the app-resolution refusals | The step is **never** taken from the request. Precedence is load-bearing: op-running outranks a stale `?full_prep=`, else a commit button reappears mid-restore. Truth table + red-proof: `restore_wizard_test.go`. Adding a form here that posts anywhere new breaks `TestRestoreWizard_NoNewMutationEndpoints` **by design** — R-48 adds no mutation surface | | `restoreOpBlocked` | controller/internal/web/restore_wizard.go | `() (msg string, blocked bool)` | THE refusal gate before starting ANY restore | **Use this, never a bare `IsRunning()`.** It reads BOTH flags: `RestoreStatus().Running` (set synchronously by `BeginRestoreOp`, true for the whole off-box restore) and `IsRunning()` (the concurrency flag, the only one the nightly backup holds). **R-351: all seven handlers read only `IsRunning()`, which the goroutine acquires AFTER the handler returns — a second press started a second run and was told „…elindult".** Returns the Hungarian refusal, which names the running app and a route | **R-360 (v0.226.0): `offboxVerifyCopyDeleteHandler` was the LAST holdout and its doc comment claimed it already did this — the sentence is why nobody looked. `IsRunning()` is FALSE for the whole of a verification restore, so the copy a restore was writing into could be deleted from the UI (observed live 2026-08-21 22:35). No app-name comparison: refusing during ANY restore is stronger and uniform.** | `Manager.RestoreFromRecoveryUnitAt` / `RestoreFromRecoveryUnit` + `UnitRestoreResult` (R-353 v0.226.0; R-102 v0.229.0) | controller/internal/backup/restore_unit.go | `(stack, unitDir string) (UnitRestoreResult, error)` / `(stack string) (…)` | THE local recovery-unit restore, and the facts its surface must state | **Returns a RESULT, not just an error** — volumes replayed, DBs replayed, and what the manifest LISTED. The Manifest* counts are load-bearing: zero-replayed has two causes (the backup held no data / the backup listed data that did not come back) and they are opposite news. Pair it with `unitRestoreOutcomeMsg`; do NOT write a new sentence. **A claim about the APP is forbidden on this path** — it has no `SafetyDump` discriminator, unlike the off-site twin (CONTEXT.md ruling, 07-backup-architecture §6.3). `restoreDockerVolumes` now returns `(int, error)`; `restoreDockerVolumesFrom` is unchanged and still the shared implementation. **R-102: `…At` holds the whole body and takes the unit DIRECTORY; the one-argument form is the thin caller naming the primary unit.** THE SOURCE MOVES; THE DESTINATION DOES NOT — `GetAppDrivePath` still resolves where the data lands. The R-47 mutation order, the unit-over-guest secret precedence, the fail-closed data-key gate and the no-unit `RestoreApp` fallback are all pinned and unchanged. The volume leg goes through the R-354 `volumeReplayFrom` seam so the source directory is assertable without Docker | +| `Manager.putBackAfterFailedRestore` + `liveVolumeCopy` (R-893 option A, v0.336.0) | controller/internal/backup/offbox_restore_undo.go | `(stack, cause, live, copy, volumesMoved, filesPlaced, restart, holdKey) error` | THE off-site restore's undo: the live volumes it will replace, measured before the stop (`planLiveVolumeCopy`), copied right after it (`copyLiveVolumes`), poured back on any failure | One copy per restore under `/backups/restore-undo//` (no capture set reads `backups/`); the put-back uses the restore's own `volumeReplayFrom`/`restoreDockerVolumesFrom`, so there is one volume-replay implementation. A put-back that fails sets `keep` and HOLDS (option C) — never start on volumes that are not known to be back. Seams for tests without Docker: `undoVolumeProbeFn`, `undoVolumeTarFn`, `undoVolumeRemoveFn`; operator notice `SetRestorePutBackNotify` (leg `restore-put-back`). Pinned by `r893_putback_test.go` | | `Manager.RestoreTier2Unit` + `Tier2Coverage.CanRestoreUnit` / `Tier2CopyDate` (R-102/R-103, v0.229.0) | controller/internal/backup/tier2_restore.go | `(stack string) (UnitRestoreResult, error)` / `() bool` / `() (string, bool)` | THE full restore from the SECOND DRIVE's mirror, and the predicate that gates it | **Do NOT widen `CanRestore()`** — it answers only "can the additive file restore run?" and one predicate answering two questions is R-356. Fail-closed: `tier2UnitIsOpenable` requires a parseable manifest, because a directory is not a package. The single-writer flag is taken INSIDE `RestoreFromRecoveryUnitAt` — a second `acquireRunning()` here would refuse the restore it guards. `Tier2CopyDate` prefers the SUCCESS anchor over the attempt clock (R-101) and reports which it returned | | `unitRestoreOutcomeMsg` + its three message constants (R-353, v0.226.0) | controller/internal/web/handlers.go | `(app string, res backup.UnitRestoreResult) string` | THE customer sentence for a completed LOCAL restore | Twin of `reconstituteOutcomeMsg`; copy its SHAPE (clauses earned by having done the thing, no filesystem path, base names only), never its text. The constants are named because `r353_unit_outcome_test.go` asserts them verbatim — a silent edit is how an honest message drifts back into a comforting one, which is the documented history of the sentence it replaces | | `offsiteNoSpaceMsgFmt` + `offsiteSizeUnknownMsg` (R-357, v0.226.0) | controller/internal/backup/offbox_restore.go | two consts | EVERY headroom refusal on the off-site restore surface | **All four gates share these** (prepare, scratch, place, and the destructive reconstitute). A customer meeting one wording on one path and a different one on another has to work out whether it is the same problem. **The reconstitute gate uses NO ×1.1 margin** — it copies a measured tree; `OffboxRestorePrepareFull`'s ×1.1 predicts a download. **Fail closed when either probe reads ≤ 0**: `free < need` with `need == 0` is FALSE, so an unmeasurable input sails through — a gate present and inert | diff --git a/controller/README.md b/controller/README.md index 8acb8f6..0b6c006 100644 --- a/controller/README.md +++ b/controller/README.md @@ -1008,13 +1008,16 @@ Each app can define rich metadata in `.felhom.yml`: `--clear-restore-hold `, **which requires a controller restart**. `--single-transaction` on the Postgres import is a belt only; MariaDB DDL is not transactional, which is why the rollback is the fix. - - **Off-site restore of one app: a failed replay after a moved definition or a replaced volume HOLDS the app - (R-893 option C, `09` §3 decision 192).** The rollback puts back the database rows only; the snapshot's files, - volumes and older definition would stay. So when the snapshot's definition was written or a named volume was - replaced, the live definition is written back, the app is held stopped (`restore_mixed`, operator-cleared like - a restore hold), the household reads that it needs our help, and the operator gets a `backup_run_failures` - line (leg `restore-hold-mixed`). Otherwise the ladder above is unchanged. „Put back exactly as it was" is the - next slice. + - **Off-site restore of one app: a failed restore puts the app back EXACTLY as it was and starts it (v0.336.0, + R-893 option A, operator D20 = A; `09` §3 decision 192).** Every named volume the snapshot will replace is copied + right after the stop (tars under `/backups/restore-undo//`), after a fit check BEFORE the stop + (the copy is added to the restore's own free-space need; too little room or an unmeasurable volume refuses with + nothing stopped). On any failure after a volume or the definition moved, the copies are poured back, a volume the + snapshot added is removed, the live definition is written back and the app starts; the household reads that the + restore failed and nothing changed (or, with placed files, that its database and volumes are back and the copied + files stay); the operator gets `backup_run_failures` leg `restore-put-back`. Only a put-back that fails — or a + replay AND rollback that both fail — still HOLDS the app (`restore_mixed` / the restore hold, operator-cleared), + and then the copy is kept for support. On success the copy is removed. `offbox_restore_undo.go`. - **Where an off-site restore puts the data (v0.219.0, R-356).** `ReconstituteFromOffsite` and `PlaceOffsiteRestore` resolve the destination with `Manager.GetAppDrivePath` — **the same resolver `CaptureRecoveryUnit` wrote the snapshot with**: the app's `HDD_PATH` if it declares diff --git a/controller/cmd/controller/main.go b/controller/cmd/controller/main.go index 64ee6c1..745389b 100644 --- a/controller/cmd/controller/main.go +++ b/controller/cmd/controller/main.go @@ -1471,6 +1471,15 @@ func main() { "The app will not start from any path until the hold is cleared. Replay error: %v. Rollback error: %v", stack, replayErr, rollbackErr) notifier.NotifyBackupRunFailures(msg, d) }) + // R-893 option A (D20 = A): a failed off-site restore that was PUT BACK — the live volumes and definition poured + // back and the app started. The same operator-tier event as the hold above (no new event type), its own leg. + backupMgr.SetRestorePutBackNotify(func(stack string, replayErr error) { + d := notify.BackupRunFailuresDetails{RunKind: "offsite-reconstitute", Failed: 1, Attempted: 1} + d.Apps = append(d.Apps, notify.RunFailureDetail{App: stack, Leg: "restore-put-back", Reason: "restore failed; the pre-restore copy was put back"}) + msg := fmt.Sprintf("App %q: its off-site restore FAILED and was undone — the live volumes and definition from just before the restore were put back "+ + "and the app was started (R-893). No support step is needed; the household was told. Error: %v", stack, replayErr) + notifier.NotifyBackupRunFailures(msg, d) + }) // 3a: the pre-push enlargement gate blocked an app's userdata push (config+DB still saved). Edge- // triggered by the engine (only NEW blocks notify), so the hub's per-event-type cooldown suffices — // no controller-side timer (the hub owns cooldown). diff --git a/controller/internal/backup/backup.go b/controller/internal/backup/backup.go index 2a5bcbc..d137ee3 100644 --- a/controller/internal/backup/backup.go +++ b/controller/internal/backup/backup.go @@ -229,6 +229,15 @@ type Manager struct { // volume leg is unit-testable without Docker. Nil → the real restoreDockerVolumesFrom. volumeReplayFrom func(stackName, dumpDir string) (int, error) + // R-893 option A seams (offbox_restore_undo.go) — the live-volume copy taken before an off-site restore replaces + // the app's volumes, so a failed restore can put them back. Nil → the real docker calls. + undoVolumeProbeFn func(vol string) (bytes int64, exists bool, err error) + undoVolumeTarFn func(vol, dir string) error + undoVolumeRemoveFn func(vol string) error + // restorePutBackNotify (R-893 option A) is called once when a failed off-site restore was put back and the app + // started again. Operator-tier, the same seam shape as restoreHoldNotify. + restorePutBackNotify func(stack string, replayErr error) + // R-403 unit-REHYDRATE seam — the copy that refills the primary unit from the Tier-2 mirror after // a restore, overridable so the "only when hollow" condition and the byte-identical non-effect are // testable without rsync. Nil → the real rsyncMirror. It is a SEPARATE seam from tier2Mirror on diff --git a/controller/internal/backup/offbox_reconstitute.go b/controller/internal/backup/offbox_reconstitute.go index c0b9c96..ab9f309 100644 --- a/controller/internal/backup/offbox_reconstitute.go +++ b/controller/internal/backup/offbox_reconstitute.go @@ -796,6 +796,27 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack return res, util.MsgError("err.backup.adatbazis_masolat_csonka_nem_indult", stack) } + // --- R-893 option A (D20 = A): THE LIVE VOLUMES' COPY MUST FIT, BEFORE ANYTHING STOPS ----------- + // Every volume the snapshot will replace is copied right after the stop, so a failed restore can put it back + // (offbox_restore_undo.go). Measured here, while nothing is touched: the copy sits on the same namespace as the + // restore's own files, so it is added to the R-357 need and checked by the same exact rule (no multiplier — the + // sizes are measured, not predicted). An unmeasurable volume refuses, as an unmeasurable scratch does. + undoVols := snapshotVolumeNames(filepath.Join(scratchUnit, "volume-dumps")) + var undoExists map[string]bool + if len(undoVols) > 0 { + volNeed, ex, pErr := m.planLiveVolumeCopy(undoVols) + if pErr != nil { + m.logger.Printf("[ERROR] [offbox] %s: REFUSING the destructive restore — the live volumes could not be measured for their copy: %v; the app was NOT stopped", stack, pErr) + return res, util.MsgError("err.backup.restore_undo_unmeasured", stack) + } + if free < need+volNeed { + m.logger.Printf("[WARN] [offbox] %s: REFUSING the destructive restore — need %d B (restore %d B + copy of the live volumes %d B), free %d B on %s; the app was NOT stopped", + stack, need+volNeed, need, volNeed, free, liveNs) + return res, util.MsgError("err.backup.restore_undo_no_space", humanizeBytes(need+volNeed), humanizeBytes(free)) + } + undoExists = ex + } + // --- WHICH VERSION COMES BACK (v0.275.0, R-696, `07` §6.6) --------------------------------- // Until v0.275.0 this path never wrote the definition, so after any update the snapshot's data // (last night's version) was started by the app's NEW definition — for an engine step that is the @@ -938,6 +959,22 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack if err := m.stackProvider.StopStack(stack); err != nil { m.logger.Printf("[WARN] [offbox] could not stop %s before reconstitution: %v (continuing)", stack, err) } + // R-893 option A: the live volumes the snapshot will replace, copied with the app stopped and before anything + // moved. A copy that fails restarts the app as it was — nothing has been touched. Removed when the restore + // ends, unless a failed put-back keeps it for support. + var undo *liveVolumeCopy + if len(undoVols) > 0 { + c, cErr := m.copyLiveVolumes(stack, liveNs, undoVols, undoExists) + if cErr != nil { + m.logger.Printf("[ERROR] [offbox] %s: the copy of the live volumes failed: %v — nothing was restored, starting the app as it was", stack, cErr) + if sErr := restartStack(); sErr != nil { + m.logger.Printf("[WARN] [offbox] %s: restart after a failed volume copy also failed: %v", stack, sErr) + } + return res, util.MsgError("err.backup.restore_undo_copy_failed", stack) + } + undo = c + defer undo.drop(m, stack) + } // v0.275.0: the snapshot's own definition, written while nothing of the app's data has been touched // yet — a failure here restarts the app as it was. if defineFromSnapshot { @@ -979,9 +1016,10 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack if cErr != nil { // R-893 option C (security review 2026-10-08): once the snapshot's (possibly older) definition // is written, a start would run that version on the live data — HOLD the app instead. + // R-893 option A: no volume has moved yet, so only the live definition is written back and the app starts. + // The failing copy may have placed part of its files, so the sentence names placed files (+1). if definitionMoved { - m.holdAppAfterMixedRestore(stack, cErr, liveDef) - return res, util.MsgError("err.backup.restore_failed_held_mixed", stack) + return res, m.putBackAfterFailedRestore(stack, cErr, liveDef, undo, false, res.FilesPlaced+1, restartStack, "err.backup.restore_failed_held_mixed") } // Nothing moved yet: best-effort bring-up — a failed restore must not also be an outage. if sErr := restartStack(); sErr != nil { @@ -1013,9 +1051,10 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack // already be the snapshot's — HOLD the app rather than start it on a mix. // R-893 option C (security review 2026-10-08): a volume already replaced, or the snapshot's // definition written → HOLD (the volume detail reaches the operator through the hold notice). - if definitionMoved || nVols > 0 { - m.holdAppAfterMixedRestore(stack, vErr, liveDef) - return res, util.MsgError("err.backup.restore_failed_held_mixed", stack) + // R-893 option A: the replay removes a volume BEFORE it recreates it, so even nVols == 0 may have lost one — + // with a copy, it is always poured back; with no copy the old rule decides (and holds when one moved). + if definitionMoved || nVols > 0 || undo != nil { + return res, m.putBackAfterFailedRestore(stack, vErr, liveDef, undo, nVols > 0 || undo != nil, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed") } // Nothing replaced yet: bring the app back up rather than leaving an outage, then surface it. if sErr := restartStack(); sErr != nil { @@ -1034,8 +1073,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack if err := m.stackProvider.StartStackServices(stack, dbServices); err != nil { // R-893 option C (security review 2026-10-08): volumes or the definition already moved → HOLD. if definitionMoved || res.VolumesReplayed > 0 { - m.holdAppAfterMixedRestore(stack, err, liveDef) - return res, util.MsgError("err.backup.restore_failed_held_mixed", stack) + return res, m.putBackAfterFailedRestore(stack, err, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.restore_failed_held_mixed") } // Best-effort bring-up: a failed restore must not also be an outage. if sErr := restartStack(); sErr != nil { @@ -1065,6 +1103,9 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack // the customer type into it and makes the damage permanent. Hold it instead — // operator ruling, 2026-08-22. m.logger.Printf("[ERROR] [offbox] %s: ROLLBACK ALSO FAILED (%v) — holding the app stopped; replay error was: %v", stack, rbErr, iErr) + if undo != nil { + undo.keep = true // R-893: the volumes' copy is a second way back for support + } m.holdAppAfterFailedRollback(stack, iErr, rbErr) // R-383: the undo copy is DESCRIBED FROM DISK, never from the path alone. See // undoCopyPhrase — this sentence used to assert the file existed in exactly the @@ -1080,16 +1121,22 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack // Pinned by r893_mixed_restore_test.go; the plain case keeps TestR379_ScenarioA. // Placed FILES alone do not hold (option C as ruled): that mix is what „put back exactly as it // was" (option A, the next slice) removes. + // R-893 option A (D20 = A): the rows are back, but the snapshot's volumes (and maybe its definition) are + // under them — the copy of the live volumes is poured back over the rolled-back database, the live + // definition written back, and the app starts exactly as it was. A put-back that fails holds, as before. if definitionMoved || res.VolumesReplayed > 0 { - m.holdAppAfterMixedRestore(stack, iErr, liveDef) - return res, util.MsgError("err.backup.db_restore_failed_held_mixed", stack) + return res, m.putBackAfterFailedRestore(stack, iErr, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, "err.backup.db_restore_failed_held_mixed") } if sErr := restartStack(); sErr != nil { m.logger.Printf("[WARN] [offbox] %s: start after a successful rollback failed: %v", stack, sErr) } // Says BOTH things. A message that reported only the failure would leave the customer // believing their data was gone when it is back — the omission of a GAIN is as - // misleading as the omission of a loss. + // misleading as the omission of a loss. R-893: placed files are not taken back, so with + // placed files the sentence says that rather than „your data is back as it was". + if res.FilesPlaced > 0 { + return res, putBackMessage(stack, res.FilesPlaced) + } return res, util.MsgError("err.backup.db_restore_failed_rolled_back", stack) } } diff --git a/controller/internal/backup/offbox_restore_undo.go b/controller/internal/backup/offbox_restore_undo.go new file mode 100644 index 0000000..632eec0 --- /dev/null +++ b/controller/internal/backup/offbox_restore_undo.go @@ -0,0 +1,262 @@ +package backup + +import ( + "context" + "fmt" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + "time" + + "gitea.dooplex.hu/admin/felhom-controller/internal/dockerexec" + "gitea.dooplex.hu/admin/felhom-controller/internal/util" +) + +// R-893 option A (operator decision D20 = A, 2026-10-20; `09` §3 decision 192, second half; `07` §6.3): +// a failed off-site restore of one app puts the app back EXACTLY as it was and starts it. +// +// The off-site restore (ReconstituteFromOffsite) replaces the app's named volumes with the snapshot's BEFORE it +// replays the snapshot's database copy. Until now a failed replay put only the database ROWS back (the undo dump) +// on top of the snapshot's volume — measured 2026-10-19 on 9202: the newest rows over the backup's older volume, +// a table only the backup had left behind — and the app was held stopped for support. +// +// Now, for every volume the snapshot will replace: +// - BEFORE THE STOP the box checks that a copy of those volumes fits beside the restore (the same exact-size +// rule the restore's own gate uses, R-357) — not enough → refused, nothing stopped; +// - RIGHT AFTER THE STOP it tars each of them (the nightly volume-dump helper, tmp + fsync + rename) into +// /backups/restore-undo// — outside every capture set (captureset.go skips backups/); +// - on ANY failure after the first volume moved it stops the app, pours the copies back (the restore's own +// volume helper), removes a volume the snapshot added, writes the live definition back and starts the app; +// - a put-back that fails keeps the copy on disk and HOLDS the app (the R-893 option C hold, unchanged); +// - on success the copy is removed. +// +// Pinned by r893_putback_test.go. + +// liveVolumeCopy is the pre-restore copy of the volumes an off-site restore will replace. +type liveVolumeCopy struct { + dir string // where the tars are; removed on success + copied []string // volumes that existed and were copied (each /.tar) + absent []string // volumes the snapshot adds that did not exist before (removed on a put-back) + keep bool // set when a put-back could not be done: the copy is the support path back, never removed +} + +// restoreUndoDir is where the live-volume copy of one restore lives. Under backups/ on purpose: no capture set +// includes it (appbackup.captureset skips backups/), and backups/primary — the recovery unit — is never written. +func restoreUndoDir(nsRoot, stack string) string { + return filepath.Join(nsRoot, "backups", "restore-undo", stack) +} + +// snapshotVolumeNames lists the volumes the snapshot's unit will replace (one .tar each), sorted. +func snapshotVolumeNames(unitVolumeDir string) []string { + entries, err := os.ReadDir(unitVolumeDir) + if err != nil { + return nil + } + var out []string + for _, e := range entries { + if !e.IsDir() && strings.HasSuffix(e.Name(), ".tar") { + out = append(out, strings.TrimSuffix(e.Name(), ".tar")) + } + } + sort.Strings(out) + return out +} + +// undoVolumeProbe measures a live volume: its byte size and whether it exists at all. +func (m *Manager) undoVolumeProbe(vol string) (int64, bool, error) { + if m.undoVolumeProbeFn != nil { + return m.undoVolumeProbeFn(vol) + } + if err := dockerexec.Command("docker", "volume", "inspect", vol).Run(); err != nil { + return 0, false, nil // `docker volume inspect` fails only for a volume that does not exist + } + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + out, err := dockerexec.CommandContext(ctx, "docker", "run", "--rm", "-v", vol+":/v:ro", "alpine", "du", "-sb", "/v").Output() + if err != nil { + return 0, true, fmt.Errorf("measuring volume %s: %w", vol, err) + } + f := strings.Fields(string(out)) + if len(f) == 0 { + return 0, true, fmt.Errorf("measuring volume %s: du printed nothing", vol) + } + n, err := strconv.ParseInt(f[0], 10, 64) + return n, true, err +} + +// undoVolumeTar writes /.tar with the nightly helper's crash-safe shape (tmp, fsync, rename). +func (m *Manager) undoVolumeTar(vol, dir string) error { + if m.undoVolumeTarFn != nil { + return m.undoVolumeTarFn(vol, dir) + } + tmp := filepath.Join(dir, vol+".tar.tmp") + if out, err := m.tarVolumeOrDefault(vol, dir); err != nil { + os.Remove(tmp) + return fmt.Errorf("copying volume %s: %s: %w", vol, strings.TrimSpace(string(out)), err) + } + if err := atomicPromoteTar(tmp, filepath.Join(dir, vol+".tar")); err != nil { + os.Remove(tmp) + return fmt.Errorf("copying volume %s: %w", vol, err) + } + return nil +} + +func (m *Manager) undoVolumeRemove(vol string) error { + if m.undoVolumeRemoveFn != nil { + return m.undoVolumeRemoveFn(vol) + } + if out, err := dockerexec.Command("docker", "volume", "rm", "-f", vol).CombinedOutput(); err != nil { + return fmt.Errorf("removing volume %s: %s: %w", vol, strings.TrimSpace(string(out)), err) + } + return nil +} + +// planLiveVolumeCopy measures the volumes the snapshot will replace — BEFORE anything is stopped. It returns +// the bytes the copy needs and which of the volumes exist. A volume that cannot be measured refuses the +// restore (fail closed, the R-357 direction: an unmeasurable copy must not sail into the destructive phase). +func (m *Manager) planLiveVolumeCopy(vols []string) (need int64, exists map[string]bool, err error) { + exists = map[string]bool{} + for _, v := range vols { + n, ok, pErr := m.undoVolumeProbe(v) + if pErr != nil { + return 0, nil, pErr + } + exists[v] = ok + if ok { + need += n + } + } + return need, exists, nil +} + +// copyLiveVolumes tars every existing volume of the plan into a fresh restoreUndoDir. Runs with the app +// STOPPED, so each copy is consistent. On any error the partial copy is removed and nothing has moved yet. +func (m *Manager) copyLiveVolumes(stack, nsRoot string, vols []string, exists map[string]bool) (*liveVolumeCopy, error) { + dir := filepath.Join(restoreUndoDir(nsRoot, stack), time.Now().UTC().Format("20060102T150405Z")) + if err := os.MkdirAll(dir, 0o700); err != nil { + return nil, err + } + c := &liveVolumeCopy{dir: dir} + start := time.Now() + for _, v := range vols { + if !exists[v] { + c.absent = append(c.absent, v) + continue + } + if err := m.undoVolumeTar(v, dir); err != nil { + os.RemoveAll(dir) + return nil, err + } + c.copied = append(c.copied, v) + } + m.logger.Printf("[INFO] [offbox] %s: the live volumes are copied before the restore (%d copied, %d new from the snapshot) in %s — %s (R-893)", + stack, len(c.copied), len(c.absent), time.Since(start).Round(time.Millisecond), dir) + return c, nil +} + +// drop removes the copy (deferred by the restore). A kept copy (a put-back that failed) stays for support. +func (c *liveVolumeCopy) drop(m *Manager, stack string) { + if c == nil || c.dir == "" { + return + } + if c.keep { + m.logger.Printf("[WARN] [offbox] %s: the restore's volume copy is KEPT for support at %s", stack, c.dir) + return + } + if err := os.RemoveAll(c.dir); err != nil { + m.logger.Printf("[WARN] [offbox] %s: could not remove the restore's volume copy %s: %v", stack, c.dir, err) + return + } + _ = os.Remove(filepath.Dir(c.dir)) // the per-app folder, when it is now empty +} + +// putBack pours the copy back: every copied volume exactly as it was, every volume the snapshot added removed. +// The app must be stopped. An error means the volumes are NOT known to be as before. +func (m *Manager) putBack(stack string, c *liveVolumeCopy) error { + if c == nil { + return nil + } + volReplay := m.volumeReplayFrom + if volReplay == nil { + volReplay = m.restoreDockerVolumesFrom + } + if len(c.copied) > 0 { + n, err := volReplay(stack, c.dir) + if err != nil { + return err + } + if n != len(c.copied) { + return fmt.Errorf("put back %d of %d volume(s)", n, len(c.copied)) + } + } + for _, v := range c.absent { + if err := m.undoVolumeRemove(v); err != nil { + return err + } + } + return nil +} + +// putBackAfterFailedRestore is option A's failure branch: stop, put the volumes and the definition back, start. +// When the put-back cannot be done exactly, the app is HELD (holdAppAfterMixedRestore, unchanged; holdKey is the +// failure's own held sentence) and the copy stays on disk. volumesMoved with no copy is such a case: nothing can +// put them back. filesPlaced chooses the household's sentence: placed files are not taken back (as ruled for +// option C, a file mix does not hold), so the sentence then says so instead of „nothing changed". +func (m *Manager) putBackAfterFailedRestore(stack string, cause error, live *liveDefinition, c *liveVolumeCopy, volumesMoved bool, + filesPlaced int, restart func() error, holdKey string) error { + if volumesMoved && c == nil { + m.logger.Printf("[ERROR] [offbox] %s: volumes were replaced and there is no copy to put back — holding the app", stack) + m.holdAppAfterMixedRestore(stack, cause, live) + return util.MsgError(holdKey, stack) + } + if err := m.stackProvider.StopStack(stack); err != nil { + m.logger.Printf("[WARN] [offbox] %s: stopping the app before the put-back failed: %v", stack, err) + } + if volumesMoved { + if err := m.putBack(stack, c); err != nil { + c.keep = true + m.logger.Printf("[ERROR] [offbox] %s: putting the live volumes back FAILED: %v — the copy stays at %s; holding the app", stack, err, c.dir) + m.holdAppAfterMixedRestore(stack, cause, live) + return util.MsgError(holdKey, stack) + } + } + if live != nil { + if err := m.stackProvider.RecreateStackDefinitionFromUnit(stack, live.dir, live.env); err != nil { + m.logger.Printf("[ERROR] [offbox] %s: writing the live definition back FAILED: %v — holding the app", stack, err) + m.holdAppAfterMixedRestore(stack, cause, live) + return util.MsgError(holdKey, stack) + } + } + m.logger.Printf("[WARN] [offbox] %s: the restore failed (%v) — the live volumes and definition are put back; starting the app (R-893)", stack, cause) + if m.restorePutBackNotify != nil { + m.restorePutBackNotify(stack, cause) + } + if err := restart(); err != nil { + m.logger.Printf("[ERROR] [offbox] %s: start after the put-back failed: %v (the copy stays at %s)", stack, err, c.dirOrNone()) + return util.MsgError("err.backup.restore_failed_put_back_not_started", stack) + } + return putBackMessage(stack, filesPlaced) +} + +func (c *liveVolumeCopy) dirOrNone() string { + if c == nil { + return "(no volume copy)" + } + return c.dir +} + +// putBackMessage is the household's sentence after a failed restore whose data is back as it was before. +// Files the restore already copied are not taken back, so with placed files the sentence says that too. +func putBackMessage(stack string, filesPlaced int) error { + if filesPlaced > 0 { + return util.MsgError("err.backup.restore_failed_put_back_files", stack) + } + return util.MsgError("err.backup.restore_failed_put_back", stack) +} + +// SetRestorePutBackNotify wires the operator notification for a failed restore that was put back (main.go). +func (m *Manager) SetRestorePutBackNotify(fn func(stack string, replayErr error)) { + m.restorePutBackNotify = fn +} diff --git a/controller/internal/backup/r893_mixed_restore_test.go b/controller/internal/backup/r893_mixed_restore_test.go index 172ba0b..b3d4535 100644 --- a/controller/internal/backup/r893_mixed_restore_test.go +++ b/controller/internal/backup/r893_mixed_restore_test.go @@ -20,12 +20,14 @@ import ( // an older definition, the snapshot's files and volumes, the newer database — and the household was told // „your data is back as it was before the restore". That is true only of the database rows. // -// Now: the database is still rolled back, the LIVE definition is written back, and the app is HELD -// stopped with a sentence that says it needs support. Pinned here: +// Slice 1 (option C) HELD the app. Since option A (operator D20 = A, 2026-10-20) the box puts the app back +// EXACTLY as it was and starts it: the live definition is written back and the copy of the live volumes is +// poured back (r893_putback_test.go pins the volume half). The HOLD remains for the case nothing can undo: +// volumes replaced with no copy to put back. Pinned here: // (a) the definition write is called a second time, with the live definition; -// (b) the app is NOT started; -// (c) a hold is left (kind restore_mixed), and the operator is notified; -// (d) neither the error nor the hold sentence claims the data is back as it was (hu and en). +// (b) put back → the app IS started, no hold, the operator told once (put-back event); +// (c) no copy → a hold is left (kind restore_mixed), the app is NOT started, the operator notified; +// (d) the held sentences claim neither that the data is back nor that the app runs (hu and en). // The no-version-change, no-volume case keeps today's behaviour: TestR379_ScenarioA. // r893Provider records EVERY definition write (vtReconProvider keeps only the last). @@ -48,6 +50,12 @@ func (p *r893Provider) RecreateStackDefinitionFromUnit(name, composeDir string, } func r893Fixture(t *testing.T, versionChange bool, volumes int) (*Manager, *r893Provider, *int, *int) { + m, vp, rolled, notified, _ := r893FixtureA(t, versionChange, volumes) + return m, vp, rolled, notified +} + +// r893FixtureA is r893Fixture plus the put-back notification counter (option A). +func r893FixtureA(t *testing.T, versionChange bool, volumes int) (*Manager, *r893Provider, *int, *int, *int) { t.Helper() m, prov, _ := reconFixture(t, "20260926T021500Z", "2026-09-26T02:15:01Z", pgDump(1)) m.importDBDump = func(context.Context, DiscoveredDB, string) error { return errors.New("replay blew up") } @@ -60,6 +68,13 @@ func r893Fixture(t *testing.T, versionChange bool, volumes int) (*Manager, *r893 t.Errorf("notify got replay=%v rollback=%v — want the replay error and NO rollback error", replayErr, rollbackErr) } }) + putBack := 0 + m.SetRestorePutBackNotify(func(_ string, replayErr error) { + putBack++ + if replayErr == nil { + t.Error("the put-back notification must carry the error that failed the restore") + } + }) m.volumeReplayFrom = func(string, string) (int, error) { return volumes, nil } live := "16" if versionChange { @@ -72,7 +87,31 @@ func r893Fixture(t *testing.T, versionChange bool, volumes int) (*Manager, *r893 vp := &r893Provider{vtReconProvider: &vtReconProvider{recordingProvider: prov, livePins: ParseComposeImages(prov.composePath)}} m.SetStackProvider(vp) vtSnapshotUnit(t, m, "16", true) - return m, vp, &rolled, ¬ified + return m, vp, &rolled, ¬ified, &putBack +} + +// assertR893PutBack is option A's outcome: the app runs again at its live definition, nothing is held. +func assertR893PutBack(t *testing.T, m *Manager, vp *r893Provider, err error, notified, putBack int) { + t.Helper() + if err == nil { + t.Fatal("a failed restore must still be surfaced as a failure") + } + if !vp.fullStarted { + t.Fatalf("option A: the app must be started again after the put-back — calls %v", vp.calls) + } + if held, _ := m.RestoreHoldFor("immich"); held { + t.Fatal("option A: a put-back app must NOT be held") + } + if notified != 0 || putBack != 1 { + t.Errorf("hold notices %d (want 0), put-back notices %d (want 1)", notified, putBack) + } + if len(vp.defs) != 2 || !strings.Contains(strings.Join(vp.defs[1], " "), "postgres:18-alpine") { + t.Fatalf("definition writes %v — want the snapshot's, then the LIVE one written back", vp.defs) + } + low := strings.ToLower(err.Error()) + if !strings.Contains(low, "sikertelen") || !strings.Contains(low, "visszaker") || strings.Contains(low, "kapcsolatot") { + t.Errorf("the sentence must say the restore failed and the data is back, with no support call: %q", err.Error()) + } } func assertR893Held(t *testing.T, m *Manager, vp *r893Provider, err error, rolled, notified int) { @@ -120,20 +159,20 @@ func assertR893Held(t *testing.T, m *Manager, vp *r893Provider, err error, rolle } } -func TestR893_AVersionChangeThenAFailedReplayHoldsTheAppAtItsLiveDefinition(t *testing.T) { - m, vp, rolled, notified := r893Fixture(t, true, 0) +// RED-PROOF (option A): restore the slice-1 branch (holdAppAfterMixedRestore + return held_mixed) after the +// rollback → the app is not started and a hold is left → FAILS. +func TestR893_AVersionChangeThenAFailedReplayStartsTheAppAtItsLiveDefinition(t *testing.T) { + m, vp, rolled, notified, putBack := r893FixtureA(t, true, 0) _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false) - assertR893Held(t, m, vp, err, *rolled, *notified) - // (a) - if len(vp.defs) != 2 { - t.Fatalf("definition writes %v — want the snapshot's, then the LIVE one written back", vp.defs) - } - if got := strings.Join(vp.defs[1], " "); !strings.Contains(got, "postgres:18-alpine") { - t.Fatalf("the definition written back is %q — want the live 18", got) + if *rolled != 1 { + t.Fatalf("the database rollback must still run exactly once, ran %d", *rolled) } + assertR893PutBack(t, m, vp, err, *notified, *putBack) } -func TestR893_AReplacedVolumeThenAFailedReplayHoldsTheApp(t *testing.T) { +// A volume replaced with NO copy to pour back (no snapshot volume tar was measured) cannot be undone: the hold +// stays (option C). In production the copy exists whenever the snapshot holds a volume tar. +func TestR893_AReplacedVolumeWithNoCopyThenAFailedReplayHoldsTheApp(t *testing.T) { m, vp, rolled, notified := r893Fixture(t, false, 1) _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false) assertR893Held(t, m, vp, err, *rolled, *notified) @@ -142,29 +181,27 @@ func TestR893_AReplacedVolumeThenAFailedReplayHoldsTheApp(t *testing.T) { } } -// Security review 2026-10-08 (G1): the hold covers EVERY failure after the snapshot's definition is written, not only a -// failed replay. Here the DB-only start fails after a version change: before the fix the app was started at the -// snapshot's (older) definition on the live database. Now the live definition is written back and the app is held. -// RED-PROOF: restore `restartStack()` as the only action in the StartStackServices failure branch → started → FAILS. -func TestR893_AVersionChangeThenAFailedDBStartHoldsTheApp(t *testing.T) { - m, vp, rolled, notified := r893Fixture(t, true, 0) +// Security review 2026-10-08 (G1): EVERY failure after the snapshot's definition is written is undone, not only a +// failed replay. Here the DB-only start fails after a version change: before G1 the app was started at the +// snapshot's (older) definition on the live database. Since option A the live definition is written back FIRST and +// the app then starts at it (calls: … recreate, start — the live definition before the start). +// RED-PROOF: restore `restartStack()` as the only action in the StartStackServices failure branch → only one +// definition write (the snapshot's) → FAILS. +func TestR893_AVersionChangeThenAFailedDBStartStartsTheAppAtItsLiveDefinition(t *testing.T) { + m, vp, rolled, notified, putBack := r893FixtureA(t, true, 0) vp.startSvcErr = context.DeadlineExceeded _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false) + if *rolled != 0 { + t.Errorf("rollback ran %d (want 0: nothing was replayed)", *rolled) + } if err == nil { t.Fatal("a failed DB-only start must be surfaced") } - if vp.fullStarted { - t.Fatalf("the app was STARTED at the snapshot's definition on the live data — calls %v", vp.calls) - } - if held, _ := m.RestoreHoldFor("immich"); !held { - t.Fatal("no hold was left after a failed DB-only start that followed a version change") - } - if len(vp.defs) != 2 || !strings.Contains(strings.Join(vp.defs[1], " "), "postgres:18-alpine") { - t.Fatalf("definition writes %v — want the snapshot's, then the LIVE one written back", vp.defs) - } - if *rolled != 0 || *notified != 1 { - t.Errorf("rollback ran %d (want 0: nothing was replayed), notified %d (want 1)", *rolled, *notified) + n := len(vp.calls) + if n < 2 || vp.calls[n-2] != "recreate" || vp.calls[n-1] != "start" { + t.Fatalf("the live definition must be written back BEFORE the start — calls %v", vp.calls) } + assertR893PutBack(t, m, vp, err, *notified, *putBack) } // Security review 2026-10-08 (G3): the hold is persisted BEFORE the app is stopped, so a controller that dies in @@ -172,7 +209,7 @@ func TestR893_AVersionChangeThenAFailedDBStartHoldsTheApp(t *testing.T) { // the hold is already on disk at the moment it is called. // RED-PROOF: move SetRestoreHold after StopStack in holdAppAfterMixedRestore → FAILS. func TestR893_HoldIsPersistedBeforeTheStop(t *testing.T) { - m, vp, _, _ := r893Fixture(t, true, 0) + m, vp, _, _ := r893Fixture(t, true, 1) // a replaced volume with no copy: the case that still holds seenAtStop := []bool{} vp.onStop = func() { held, _ := m.RestoreHoldFor("immich") diff --git a/controller/internal/backup/r893_putback_test.go b/controller/internal/backup/r893_putback_test.go new file mode 100644 index 0000000..0a21ec4 --- /dev/null +++ b/controller/internal/backup/r893_putback_test.go @@ -0,0 +1,284 @@ +package backup + +import ( + "context" + "errors" + "os" + "path/filepath" + "sort" + "strings" + "testing" +) + +// R-893 option A (operator decision D20 = A, 2026-10-20): a failed off-site restore puts the app's live volumes +// back EXACTLY and starts the app. offbox_restore_undo.go. These tests pin the CONSEQUENCE — what is poured back, +// what is removed, what runs, what stays on disk — not only that a function was called. + +const r893LiveVol, r893NewVol = "immich_pgdata", "immich_newvol" + +// putBackRig records the volume legs: what the snapshot replay saw, and what the put-back poured, BYTE FOR BYTE. +type putBackRig struct { + scratchVolDir string + undoRoot string + snapReplayErr error // the snapshot's own volume replay + putBackErr error // the put-back from the copy + snapReplays int + poured map[string]string // volume → the bytes poured back from the copy + removed []string + tarred []string + tarErr error + probeErr error +} + +func r893PutBackFixture(t *testing.T) (*Manager, *r893Provider, *putBackRig, *int, *int, *int) { + t.Helper() + m, vp, rolled, notified, putBack := r893FixtureA(t, false, 0) + unit := vtSnapshotUnit(t, m, "16", true) + rig := &putBackRig{scratchVolDir: filepath.Join(unit, "volume-dumps"), poured: map[string]string{}} + // The snapshot holds two volumes: one the box has today, one it does not (the snapshot adds it). + mustWrite(t, filepath.Join(rig.scratchVolDir, r893LiveVol+".tar"), "SNAPSHOT bytes") + mustWrite(t, filepath.Join(rig.scratchVolDir, r893NewVol+".tar"), "SNAPSHOT new") + _, liveNs, err := m.offboxRestoreScratchDir("immich") + if err != nil { + t.Fatal(err) + } + rig.undoRoot = restoreUndoDir(liveNs, "immich") + + m.undoVolumeProbeFn = func(vol string) (int64, bool, error) { + if rig.probeErr != nil { + return 0, false, rig.probeErr + } + if vol == r893LiveVol { + return 1000, true, nil + } + return 0, false, nil + } + m.undoVolumeTarFn = func(vol, dir string) error { + if rig.tarErr != nil { + return rig.tarErr + } + rig.tarred = append(rig.tarred, vol) + return os.WriteFile(filepath.Join(dir, vol+".tar"), []byte("LIVE bytes of "+vol), 0o600) + } + m.undoVolumeRemoveFn = func(vol string) error { rig.removed = append(rig.removed, vol); return nil } + m.volumeReplayFrom = func(_ string, dir string) (int, error) { + entries, _ := os.ReadDir(dir) + n := 0 + if dir == rig.scratchVolDir { + rig.snapReplays++ + for _, e := range entries { + if strings.HasSuffix(e.Name(), ".tar") { + n++ + } + } + if rig.snapReplayErr != nil { + return 0, rig.snapReplayErr + } + return n, nil + } + if !strings.HasPrefix(dir, rig.undoRoot) { + t.Errorf("a volume replay from an unexpected folder %q", dir) + return 0, errors.New("unexpected folder") + } + if rig.putBackErr != nil { + return 0, rig.putBackErr + } + for _, e := range entries { + if strings.HasSuffix(e.Name(), ".tar") { + b, _ := os.ReadFile(filepath.Join(dir, e.Name())) + rig.poured[strings.TrimSuffix(e.Name(), ".tar")] = string(b) + n++ + } + } + return n, nil + } + return m, vp, rig, rolled, notified, putBack +} + +func r893Stops(vp *r893Provider) int { + n := 0 + for _, c := range vp.calls { + if c == "stop" { + n++ + } + } + return n +} + +func undoCopies(t *testing.T, root string) []string { + t.Helper() + var out []string + _ = filepath.Walk(root, func(p string, fi os.FileInfo, _ error) error { + if fi != nil && !fi.IsDir() { + out = append(out, p) + } + return nil + }) + sort.Strings(out) + return out +} + +// The whole point of option A, in one run: the replay fails after the snapshot's volumes replaced the live ones; +// the live volume comes back with EXACTLY the bytes copied before the restore, the volume the snapshot added is +// removed, the app starts, nothing is held, the copy is gone, the operator is told once. +// RED-PROOF: make putBackAfterFailedRestore skip m.putBack (volumesMoved ignored) → poured is empty → FAILS. +func TestR893A_AFailedReplayPutsTheLiveVolumesBackExactly(t *testing.T) { + m, vp, rig, rolled, notified, putBack := r893PutBackFixture(t) + _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false) + if err == nil { + t.Fatal("a failed replay must be surfaced") + } + if rig.snapReplays != 1 { + t.Fatalf("the snapshot's volumes must have been replayed once before the failure, got %d", rig.snapReplays) + } + if got := rig.poured[r893LiveVol]; got != "LIVE bytes of "+r893LiveVol { + t.Fatalf("the live volume was poured back as %q — want the exact pre-restore copy", got) + } + if len(rig.poured) != 1 { + t.Errorf("only the volume that existed before may be poured back, got %v", rig.poured) + } + if len(rig.removed) != 1 || rig.removed[0] != r893NewVol { + t.Errorf("the volume the snapshot added must be removed, got %v", rig.removed) + } + if !vp.fullStarted { + t.Fatalf("the app must run again after the put-back — calls %v", vp.calls) + } + if held, _ := m.RestoreHoldFor("immich"); held { + t.Fatal("a put-back app must not be held") + } + if *rolled != 1 || *notified != 0 || *putBack != 1 { + t.Errorf("rollback %d (want 1), hold notices %d (want 0), put-back notices %d (want 1)", *rolled, *notified, *putBack) + } + if left := undoCopies(t, rig.undoRoot); len(left) != 0 { + t.Errorf("the copy must be removed after a put-back that worked, left %v", left) + } + if !strings.Contains(err.Error(), "visszaker") { + t.Errorf("the sentence must say the data is back: %v", err) + } +} + +// RED-PROOF: drop `|| undo != nil` from the volume-replay failure branch → the app starts with nothing poured back +// (a volume the replay removed stays removed) → FAILS. +func TestR893A_AFailedVolumeReplayStillPutsBackWhenItCountedNothing(t *testing.T) { + m, vp, rig, _, _, putBack := r893PutBackFixture(t) + rig.snapReplayErr = errors.New("volume create failed") + _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false) + if err == nil { + t.Fatal("a failed volume replay must be surfaced") + } + if got := rig.poured[r893LiveVol]; got != "LIVE bytes of "+r893LiveVol { + t.Fatalf("the replay removes a volume before it recreates it; the copy must be poured back even at a count of 0, got %q", got) + } + if !vp.fullStarted || *putBack != 1 { + t.Fatalf("started=%v put-back notices=%d — want the app running and one notice", vp.fullStarted, *putBack) + } +} + +// The copy is checked BEFORE the stop. One byte short → refused, nothing stopped, nothing copied. +// RED-PROOF: remove volNeed from the comparison (`free < need`) → the restore runs → FAILS. +func TestR893A_TheCopyMustFitBeforeAnythingStops(t *testing.T) { + m, vp, rig, _, _, _ := r893PutBackFixture(t) + m.SetOffboxFreeFn(func(string) int64 { return (1 << 20) + 999 }) // the restore needs 1 MiB + the copy 1000 B + _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false) + if err == nil { + t.Fatal("a copy that does not fit must refuse the restore") + } + if n := r893Stops(vp); n != 0 { + t.Fatalf("the refusal must come before the stop — StopStack ran %d time(s)", n) + } + if len(rig.tarred) != 0 || rig.snapReplays != 0 { + t.Errorf("nothing may be copied or replayed on a refusal: tarred %v, replays %d", rig.tarred, rig.snapReplays) + } + if !strings.Contains(err.Error(), "szabad") { + t.Errorf("the refusal must name the free space: %v", err) + } + // The boundary: exactly enough room → the restore proceeds (and fails later on the fixture's replay). + m2, vp2, _, _, _, _ := r893PutBackFixture(t) + m2.SetOffboxFreeFn(func(string) int64 { return (1 << 20) + 1000 }) + _, _ = m2.ReconstituteFromOffsite(context.Background(), "immich", false) + if r893Stops(vp2) == 0 { + t.Error("with exactly enough room the restore must run") + } +} + +// RED-PROOF: let planLiveVolumeCopy skip a probe error → the restore runs on an unmeasured copy → FAILS. +func TestR893A_AnUnmeasurableVolumeRefusesBeforeAnythingStops(t *testing.T) { + m, vp, rig, _, _, _ := r893PutBackFixture(t) + rig.probeErr = errors.New("du failed") + _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false) + if err == nil { + t.Fatal("an unmeasurable volume must refuse") + } + if n := r893Stops(vp); n != 0 { + t.Fatalf("StopStack ran %d time(s) on an unmeasured copy", n) + } +} + +// A restore that WORKS removes the copy and never pours it back. +// RED-PROOF: remove `defer undo.drop(m, stack)` → the copy stays → FAILS. +func TestR893A_ASuccessfulRestoreRemovesTheCopy(t *testing.T) { + m, vp, rig, _, _, putBack := r893PutBackFixture(t) + m.importDBDump = func(context.Context, DiscoveredDB, string) error { return nil } + if _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false); err != nil { + t.Fatalf("restore: %v", err) + } + if len(rig.tarred) != 1 { + t.Fatalf("the live volume must have been copied before the restore, tarred %v", rig.tarred) + } + if len(rig.poured) != 0 || *putBack != 0 { + t.Errorf("nothing may be poured back after a restore that worked: poured %v, notices %d", rig.poured, *putBack) + } + if left := undoCopies(t, rig.undoRoot); len(left) != 0 { + t.Errorf("the copy must be removed after a restore that worked, left %v", left) + } + if !vp.fullStarted { + t.Error("the restored app must be started") + } +} + +// A put-back that fails cannot promise anything: the app is HELD (option C) and the copy STAYS for support. +// RED-PROOF: remove `c.keep = true` → the deferred drop deletes the only copy → FAILS. +func TestR893A_AFailedPutBackHoldsTheAppAndKeepsTheCopy(t *testing.T) { + m, vp, rig, _, notified, putBack := r893PutBackFixture(t) + rig.putBackErr = errors.New("tar xf failed") + _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false) + if err == nil { + t.Fatal("a failed put-back must be surfaced") + } + if vp.fullStarted { + t.Fatalf("the app must NOT start on volumes that could not be put back — calls %v", vp.calls) + } + if held, _ := m.RestoreHoldFor("immich"); !held { + t.Fatal("a failed put-back must leave the hold") + } + if *notified != 1 || *putBack != 0 { + t.Errorf("hold notices %d (want 1), put-back notices %d (want 0)", *notified, *putBack) + } + left := undoCopies(t, rig.undoRoot) + if len(left) != 1 || filepath.Base(left[0]) != r893LiveVol+".tar" { + t.Fatalf("the copy must stay on disk for support, found %v", left) + } + if low := strings.ToLower(err.Error()); strings.Contains(low, "visszaker") || !strings.Contains(low, "kapcsolatot") { + t.Errorf("the held sentence must not claim the data is back, and must send to support: %q", err.Error()) + } +} + +// A copy that fails touches nothing: no replay, no definition write, the app starts as it was. +// RED-PROOF: ignore the copy error (carry on with undo == nil) → the snapshot replay runs → FAILS. +func TestR893A_AFailedCopyRestoresNothingAndStartsTheApp(t *testing.T) { + m, vp, rig, rolled, _, _ := r893PutBackFixture(t) + rig.tarErr = errors.New("disk full") + _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false) + if err == nil { + t.Fatal("a failed copy must be surfaced") + } + if rig.snapReplays != 0 || *rolled != 0 || len(vp.defs) != 0 { + t.Fatalf("nothing may move after a failed copy: replays %d, rollbacks %d, definition writes %v", rig.snapReplays, *rolled, vp.defs) + } + if !vp.fullStarted { + t.Fatal("the app must be started again as it was") + } + if left := undoCopies(t, rig.undoRoot); len(left) != 0 { + t.Errorf("a failed copy must leave nothing behind, found %v", left) + } +} diff --git a/controller/internal/i18n/locales/en.json b/controller/internal/i18n/locales/en.json index a02b92f..cbb95d2 100644 --- a/controller/internal/i18n/locales/en.json +++ b/controller/internal/i18n/locales/en.json @@ -1303,8 +1303,14 @@ "err.backup.az_offsite_tarolo_nincs_elarvult_allapotban": "the off-site store is not orphaned", "err.backup.db_restore_and_rollback_failed": "putting back the database of %s failed, and the earlier state could not be restored either. We have left the app STOPPED, for safety, so your data cannot be damaged further. Get in touch with us — %s", "err.backup.db_restore_failed_rolled_back": "putting back the database of %s failed — your data is back as it was before the restore, and the app is still running. If you want to try again, get in touch with us first", - "err.backup.db_restore_failed_held_mixed": "putting back the database of %s failed. The database is as it was before the restore, but the files and other data of the app already come from the backup, so they do not match. We have left the app STOPPED, for safety. This needs our help: get in touch with us", + "err.backup.db_restore_failed_held_mixed": "restoring %s stopped half-way, and the data of the app could not be returned to its state from before the restore. We have left the app STOPPED, for safety. Nothing is lost: the copy from before the restore is kept. This needs our help: get in touch with us", "err.backup.restore_failed_held_mixed": "restoring %s stopped half-way. Some of the files or data of the app already come from the backup and some do not, so they do not match. We have left the app STOPPED, for safety. This needs our help: get in touch with us", + "err.backup.restore_failed_put_back": "restoring %s did not work. Nothing changed: the data of the app is back as it was before the restore, and the app runs again", + "err.backup.restore_failed_put_back_files": "restoring %s did not work. The database and data volumes of the app are back as they were before the restore, and the app runs again. The files already copied from the backup stay in place", + "err.backup.restore_failed_put_back_not_started": "restoring %s did not work. Its data is back as it was before the restore, but the app did not start. Start it on the page of the app; if it does not start, get in touch with us", + "err.backup.restore_undo_copy_failed": "a safety copy of the current data of %s could not be made, so the restore did not start. Nothing changed, and the app runs again", + "err.backup.restore_undo_no_space": "There is not enough free space for the restore: the backup and a safety copy of the current data of the app need %s together, and %s is free. Nothing was stopped", + "err.backup.restore_undo_unmeasured": "the size of the current data of %s cannot be measured, so the restore did not start, for safety. Nothing was stopped", "err.backup.egy_masik_mentesi_visszaallitasi_muvelet_mar": "another backup or restore is already running", "err.backup.ennek_az_alkalmazasnak_az_adatai_nem": "this app’s data cannot be restored from this copy", "err.backup.ervenytelen_alkalmazasnev": "that app name is not valid", diff --git a/controller/internal/i18n/locales/hu.json b/controller/internal/i18n/locales/hu.json index e37dda2..493320d 100644 --- a/controller/internal/i18n/locales/hu.json +++ b/controller/internal/i18n/locales/hu.json @@ -1298,8 +1298,14 @@ "err.backup.az_offsite_tarolo_nincs_elarvult_allapotban": "az offsite tároló nincs elárvult állapotban", "err.backup.db_restore_and_rollback_failed": "a(z) %s adatbázisának visszaállítása sikertelen, és a korábbi állapot visszatöltése sem sikerült. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk, hogy az adatai ne sérüljenek tovább. Vedd fel velünk a kapcsolatot — %s", "err.backup.db_restore_failed_rolled_back": "a(z) %s adatbázisának visszaállítása sikertelen — az adataid visszakerültek a visszaállítás előtti állapotba, az alkalmazás fut tovább. Ha újra megpróbálnád, előbb vedd fel velünk a kapcsolatot", - "err.backup.db_restore_failed_held_mixed": "a(z) %s adatbázisának visszaállítása sikertelen. Az adatbázis a visszaállítás előtti állapotban van, de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot", + "err.backup.db_restore_failed_held_mixed": "a(z) %s visszaállítása félúton megszakadt, és az alkalmazás adatait nem sikerült a visszaállítás előtti állapotukba tenni. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Nem veszett el semmi: a visszaállítás előtti másolatot megőriztük. Ehhez segítség kell: vedd fel velünk a kapcsolatot", "err.backup.restore_failed_held_mixed": "a(z) %s visszaállítása félúton megszakadt. Az alkalmazás egyes fájljai vagy adatai már a mentésből származnak, mások nem, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot", + "err.backup.restore_failed_put_back": "a(z) %s visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás előtti állapotba, és az alkalmazás újra fut", + "err.backup.restore_failed_put_back_files": "a(z) %s visszaállítása sikertelen. Az alkalmazás adatbázisa és adatkötetei visszakerültek a visszaállítás előtti állapotba, és az alkalmazás újra fut. A mentésből már átmásolt fájlok a helyükön maradtak", + "err.backup.restore_failed_put_back_not_started": "a(z) %s visszaállítása sikertelen. Az adatai visszakerültek a visszaállítás előtti állapotba, de az alkalmazás nem indult el. Indítsd el az alkalmazás oldalán; ha nem indul, vedd fel velünk a kapcsolatot", + "err.backup.restore_undo_copy_failed": "a(z) %s jelenlegi adatairól nem sikerült biztonsági másolatot készíteni, ezért a visszaállítás nem indult el. Nem változott semmi, az alkalmazás újra fut", + "err.backup.restore_undo_no_space": "Nincs elég szabad hely a visszaállításhoz: a mentésnek és az alkalmazás jelenlegi adatairól készülő biztonsági másolatnak együtt %s kell, és %s szabad. Nem állítottunk le semmit", + "err.backup.restore_undo_unmeasured": "a(z) %s jelenlegi adatainak mérete nem állapítható meg, ezért a visszaállítás biztonsági okból nem indult el. Nem állítottunk le semmit", "err.backup.egy_masik_mentesi_visszaallitasi_muvelet_mar": "egy másik mentési/visszaállítási művelet már fut", "err.backup.ennek_az_alkalmazasnak_az_adatai_nem": "ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza", "err.backup.ervenytelen_alkalmazasnev": "érvénytelen alkalmazásnév", diff --git a/controller/scripts/docker_run_volume_path_gate.py b/controller/scripts/docker_run_volume_path_gate.py index c1239c6..3449229 100644 --- a/controller/scripts/docker_run_volume_path_gate.py +++ b/controller/scripts/docker_run_volume_path_gate.py @@ -35,6 +35,9 @@ ALLOWLIST = [ ("internal/backup/restore.go", '"-v", dumpDir+":/in:ro"', "Tier-1 volume restore source: same registered-drive namespace argument as the dump " "target above — host-visible by the identical binds"), + ("internal/backup/offbox_restore_undo.go", '"-v", vol+":/v:ro"', + "the off-site restore's copy size check (R-893 option A, 2026-10-20): `du` on a NAMED volume mounted " + "read-only into a throwaway alpine — no host path, daemon-side, the estimate.go class"), ("internal/stacks/undo.go", '"-v", vol+":/v:ro"', "the undo's size check (v0.263.0): `du` on a NAMED volume mounted read-only — no host path, " "daemon-side, the estimate.go class"), diff --git a/controller/scripts/i18n_go_keys.json b/controller/scripts/i18n_go_keys.json index 35fdae9..1b8bf48 100644 --- a/controller/scripts/i18n_go_keys.json +++ b/controller/scripts/i18n_go_keys.json @@ -283,7 +283,13 @@ "err.web.decommission_backup_target": "BORN AS A KEY (R-1063, 2026-10-18) -- the agent's 409 to a drive retirement (the drive holds the system backup), in plain words; nothing changed. Pinned by internal/web TestFinalizeDecommission_AgentRefusalChangesNothing.", "err.web.decommission_protected": "BORN AS A KEY (R-1063, 2026-10-18) -- the agent's 403 to a drive retirement (a protected system/backup disk), in plain words; nothing changed. Pinned by internal/web TestFinalizeDecommission_AgentRefusalChangesNothing.", "disk.decommission.apps_restarted": "BORN AS A KEY (R-1144, 2026-10-18) -- after a refused „Végleges leszerelés — mégis\", the apps that were running are started again; this sentence says so. Pinned by internal/web TestR1144_RefusalAfterStopStartsTheRunningAppsAgain.", - "disk.decommission.apps_not_all_restarted": "BORN AS A KEY (R-1144, 2026-10-18) -- after a refused „Végleges leszerelés — mégis\", the apps that were running are started again; this sentence says so. Pinned by internal/web TestR1144_RefusalAfterStopStartsTheRunningAppsAgain." + "disk.decommission.apps_not_all_restarted": "BORN AS A KEY (R-1144, 2026-10-18) -- after a refused „Végleges leszerelés — mégis\", the apps that were running are started again; this sentence says so. Pinned by internal/web TestR1144_RefusalAfterStopStartsTheRunningAppsAgain.", + "err.backup.restore_failed_put_back": "BORN AS A KEY (R-893 option A, D20 = A, 2026-10-20) -- a failed off-site restore was put back: the live volumes and definition from before the restore, the app started; nothing changed. Pinned by internal/backup TestR893A_AFailedReplayPutsTheLiveVolumesBackExactly.", + "err.backup.restore_failed_put_back_files": "BORN AS A KEY (R-893 option A, D20 = A, 2026-10-20) -- the same put-back when files were already copied from the backup; the sentence says the files stay. Pinned by internal/backup TestR893_AVersionChangeThenAFailedReplayStartsTheAppAtItsLiveDefinition.", + "err.backup.restore_failed_put_back_not_started": "BORN AS A KEY (R-893 option A, D20 = A, 2026-10-20) -- the data is put back but the app did not start; the household is told to start it. Pinned by internal/backup offbox_restore_undo.go putBackAfterFailedRestore (the restart branch).", + "err.backup.restore_undo_copy_failed": "BORN AS A KEY (R-893 option A, D20 = A, 2026-10-20) -- the copy of the live volumes failed, nothing was restored, the app runs again. Pinned by internal/backup TestR893A_AFailedCopyRestoresNothingAndStartsTheApp.", + "err.backup.restore_undo_no_space": "BORN AS A KEY (R-893 option A, D20 = A, 2026-10-20) -- the restore and the copy of the live volumes do not fit; refused before anything stops. Pinned by internal/backup TestR893A_TheCopyMustFitBeforeAnythingStops.", + "err.backup.restore_undo_unmeasured": "BORN AS A KEY (R-893 option A, D20 = A, 2026-10-20) -- a live volume cannot be measured for its copy; refused before anything stops. Pinned by internal/backup TestR893A_AnUnmeasurableVolumeRefusesBeforeAnythingStops." }, "flash.share.already_on": "A megosztás már be van kapcsolva.", "flash.share.enable_failed": "A megosztás bekapcsolása nem sikerült.",