R-379/R-380: put the customer's undo copy back when a database restore fails
gates / gates (push) Successful in 11s
gates / gates (push) Successful in 11s
R-379 and R-380 were one failure. Both ended with a half-restored database; the only difference was whether it looked broken. Postgres emptied and crash-looped; MariaDB applied part of the dump and reported health=healthy with a zero-row schema-version table. Measured live on demo-hp 2026-08-22. The undo copy was already taken and already good - proven by hand that day on both engines. Nothing in the product could apply it. Now it does, with the same ImportDump call, before any restart and inside the DB-only window. The WHOLE undo set, matched on this run's stamp. writeSafetyDump returned one path for an app with two databases; a rollback on that would restore one and leave the other half-written. When the rollback also fails the app is HELD STOPPED (operator ruling): a running app on a half-written database lets the customer make the damage permanent. Every start path refuses it - customer button, appstop Recover, boot sweep - via the shared driveStartGate, checked ABOVE its driveless early return because these apps have no drive. The marker is ended so nothing auto-restarts it. The row goes red. Cleared with --clear-restore-hold, an operator CLI route. --single-transaction is a belt on Postgres only; MariaDB DDL is not transactional and that is why the rollback is the fix. R-381: the engine's stderr stops reaching the customer (615 bytes on MariaDB, its middle rows out of their own database) and starts reaching the operator log, which never had it. R-382: the summary log prints the volume count it already held. Undo copies resolve to their own app, are marked IsUndo, and are capped at 3 per app, pruned from the capture side. The reported render-as-an-app symptom did NOT reproduce - the live page was read first and had zero occurrences. Tests 1468 -> 1483. Eight red-proofs; ONE PASSED and is reported: the R-381 behavioural test injected below ImportDump. A guard at that layer now convicts.
This commit is contained in:
@@ -264,11 +264,19 @@ func TestRestoreFromUnitIgnoresSafetyDumpsWhenDecidingToReplay(t *testing.T) {
|
||||
// TestReconstituteReplayFailureStillBringsTheStackUp: the DB-only window is a deliberate half-started
|
||||
// state, so EVERY exit from it must end in a full start. Otherwise a failed restore leaves the
|
||||
// customer with a running database and no application — an outage caused by the recovery tool.
|
||||
//
|
||||
// UPDATED FOR v0.220.0 (R-379). The requirement is unchanged and still asserted; what changed is
|
||||
// what happens BETWEEN the failure and the full start. A failed replay now re-applies the customer's
|
||||
// own pre-restore copy first, and only then starts. The rollback seam is injected as SUCCEEDING here
|
||||
// because that is this test's subject; the double-failure path — where the app is deliberately NOT
|
||||
// started — is Scenario C and has its own test in r379_rollback_test.go.
|
||||
func TestReconstituteReplayFailureStillBringsTheStackUp(t *testing.T) {
|
||||
m, prov, _ := reconFixture(t, "run1", "2026-07-19T06:00:00Z", pgDump(1))
|
||||
m.importDBDump = func(context.Context, DiscoveredDB, string) error {
|
||||
return context.DeadlineExceeded
|
||||
}
|
||||
rolledBack := 0
|
||||
m.SetRollbackImportFn(func(context.Context, DiscoveredDB, string) error { rolledBack++; return nil })
|
||||
|
||||
res, err := m.ReconstituteFromOffsite(context.Background(), "immich", false)
|
||||
if err == nil {
|
||||
@@ -280,9 +288,16 @@ func TestReconstituteReplayFailureStillBringsTheStackUp(t *testing.T) {
|
||||
if got := strings.Join(prov.calls, ","); got != "stop,startsvc:immich-postgres,start" {
|
||||
t.Fatalf("sequence = %q, want the best-effort full start after the failure", got)
|
||||
}
|
||||
// The existing message shape stays: the operator needs the undo's filename.
|
||||
if !strings.Contains(err.Error(), filepath.Base(res.SafetyDump)) {
|
||||
t.Fatalf("the error must name the safety dump so the operator can undo, got: %v", err)
|
||||
if rolledBack != 1 {
|
||||
t.Fatalf("the customer's pre-restore copy must be put back before the start; rollback calls = %d", rolledBack)
|
||||
}
|
||||
if !res.RolledBack {
|
||||
t.Error("the outcome must record that a rollback happened — the surface has to be able to say the data is back")
|
||||
}
|
||||
// v0.220.0: the customer sentence now states the OUTCOME (their data is as it was) instead of a
|
||||
// filename. The filename remains for the operator, in the log and on the result.
|
||||
if res.SafetyDump == "" || !strings.Contains(filepath.Base(res.SafetyDump), preRestoreDumpPrefix) {
|
||||
t.Fatalf("the result must still carry the undo copy for the operator, got %q", res.SafetyDump)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user