R-379/R-380: put the customer's undo copy back when a database restore fails
gates / gates (push) Successful in 11s

R-379 and R-380 were one failure. Both ended with a half-restored database; the
only difference was whether it looked broken. Postgres emptied and crash-looped;
MariaDB applied part of the dump and reported health=healthy with a zero-row
schema-version table. Measured live on demo-hp 2026-08-22.

The undo copy was already taken and already good - proven by hand that day on
both engines. Nothing in the product could apply it. Now it does, with the same
ImportDump call, before any restart and inside the DB-only window.

The WHOLE undo set, matched on this run's stamp. writeSafetyDump returned one
path for an app with two databases; a rollback on that would restore one and
leave the other half-written.

When the rollback also fails the app is HELD STOPPED (operator ruling): a running
app on a half-written database lets the customer make the damage permanent. Every
start path refuses it - customer button, appstop Recover, boot sweep - via the
shared driveStartGate, checked ABOVE its driveless early return because these
apps have no drive. The marker is ended so nothing auto-restarts it. The row goes
red. Cleared with --clear-restore-hold, an operator CLI route.

--single-transaction is a belt on Postgres only; MariaDB DDL is not transactional
and that is why the rollback is the fix.

R-381: the engine's stderr stops reaching the customer (615 bytes on MariaDB, its
middle rows out of their own database) and starts reaching the operator log,
which never had it.
R-382: the summary log prints the volume count it already held.
Undo copies resolve to their own app, are marked IsUndo, and are capped at 3 per
app, pruned from the capture side. The reported render-as-an-app symptom did NOT
reproduce - the live page was read first and had zero occurrences.

Tests 1468 -> 1483. Eight red-proofs; ONE PASSED and is reported: the R-381
behavioural test injected below ImportDump. A guard at that layer now convicts.
This commit is contained in:
2026-08-22 18:03:18 +02:00
parent 0f3cf0dbb2
commit 2c724c9283
15 changed files with 1288 additions and 37 deletions
@@ -264,11 +264,19 @@ func TestRestoreFromUnitIgnoresSafetyDumpsWhenDecidingToReplay(t *testing.T) {
// TestReconstituteReplayFailureStillBringsTheStackUp: the DB-only window is a deliberate half-started
// state, so EVERY exit from it must end in a full start. Otherwise a failed restore leaves the
// customer with a running database and no application — an outage caused by the recovery tool.
//
// UPDATED FOR v0.220.0 (R-379). The requirement is unchanged and still asserted; what changed is
// what happens BETWEEN the failure and the full start. A failed replay now re-applies the customer's
// own pre-restore copy first, and only then starts. The rollback seam is injected as SUCCEEDING here
// because that is this test's subject; the double-failure path — where the app is deliberately NOT
// started — is Scenario C and has its own test in r379_rollback_test.go.
func TestReconstituteReplayFailureStillBringsTheStackUp(t *testing.T) {
m, prov, _ := reconFixture(t, "run1", "2026-07-19T06:00:00Z", pgDump(1))
m.importDBDump = func(context.Context, DiscoveredDB, string) error {
return context.DeadlineExceeded
}
rolledBack := 0
m.SetRollbackImportFn(func(context.Context, DiscoveredDB, string) error { rolledBack++; return nil })
res, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
if err == nil {
@@ -280,9 +288,16 @@ func TestReconstituteReplayFailureStillBringsTheStackUp(t *testing.T) {
if got := strings.Join(prov.calls, ","); got != "stop,startsvc:immich-postgres,start" {
t.Fatalf("sequence = %q, want the best-effort full start after the failure", got)
}
// The existing message shape stays: the operator needs the undo's filename.
if !strings.Contains(err.Error(), filepath.Base(res.SafetyDump)) {
t.Fatalf("the error must name the safety dump so the operator can undo, got: %v", err)
if rolledBack != 1 {
t.Fatalf("the customer's pre-restore copy must be put back before the start; rollback calls = %d", rolledBack)
}
if !res.RolledBack {
t.Error("the outcome must record that a rollback happened — the surface has to be able to say the data is back")
}
// v0.220.0: the customer sentence now states the OUTCOME (their data is as it was) instead of a
// filename. The filename remains for the operator, in the log and on the result.
if res.SafetyDump == "" || !strings.Contains(filepath.Base(res.SafetyDump), preRestoreDumpPrefix) {
t.Fatalf("the result must still carry the undo copy for the operator, got %q", res.SafetyDump)
}
}