v0.153.0 — R-47: the DB replay no longer races the app, on BOTH restore paths

Closes R-47. No new agent coupling — MinAgent stays 0.90.0.

The replay needs a running DB container, so both restore paths started the
WHOLE stack first, giving the application a window to rebuild the very schema
objects the dump was about to create. Measured live on 2026-07-19 (H4,
DIAG-immich-restore-round2): immich-server rebuilt clip_index two seconds
before the dump's CREATE INDEX, the replay aborted "already exists" under
ON_ERROR_STOP=1, and immich reported schema drift. The data survived only
because pg_dump emits COPY before CREATE INDEX.

Both paths now open a DB-ONLY window: only the stack's database service(s)
come up, the dump is replayed with the app still down, and the full start
runs only after the replay exits 0. Fail-closed: a dump with no identifiable
DB service refuses BEFORE the first mutation. Every exit from the window
still does a best-effort full start, so a failed restore never leaves a box
with a database and no application.

New: appbackup.DBServiceNames (yaml.v3 services-map parse — never a line
scan; immich's top-level volume keys are the decoy) sharing dbTypeForImage
with DiscoverDatabases; stacks.Manager.StartStackServices (refuses an empty
list — argument-less `up -d` is a full start); RedeployFromEnv split into
PersistUnitRedeployConfig + its unchanged tail. StackDataProvider's
RecreateStackFromUnit becomes RecreateStackDefinitionFromUnit — the hidden
`up -d` inside the old name is what carried the defect on the local path.

19 new tests (ordering plus state-at-replay-time, zero-mutation fail-closed
effects, replay-failure bring-up, parser decoys, empty-list refusal); three
companion red-proofs run and reverted. 23/23 packages green.

Not yet live-validated: STOP-1 supervised reconstitute, golden 0.153.0.
This commit is contained in:
2026-07-20 17:01:52 +02:00
parent fd40b29119
commit 78ff991f1c
25 changed files with 1323 additions and 201 deletions
@@ -20,13 +20,34 @@ import (
// rather than a description of the current implementation.
// recordingProvider records stop/start call ORDER so the reconstitution sequence can be asserted.
//
// R-47 widened it: it now also records the DB-ONLY bring-up and, critically, whether a FULL start
// has happened yet — the state the replay must observe as `false`. That single flag is what
// separates the fixed sequence from the one that produced H4, in which the whole stack was already
// up (and rebuilding its own schema) when the dump replay began.
type recordingProvider struct {
offbox3aProvider
calls []string
calls []string
composePath string // the LIVE compose the DB-service resolver reads
gotServices []string // services passed to StartStackServices
fullStarted bool // a FULL StartStack has happened
startSvcErr error // injected StartStackServices failure
}
func (p *recordingProvider) StopStack(string) error { p.calls = append(p.calls, "stop"); return nil }
func (p *recordingProvider) StartStack(string) error { p.calls = append(p.calls, "start"); return nil }
func (p *recordingProvider) StopStack(string) error { p.calls = append(p.calls, "stop"); return nil }
func (p *recordingProvider) StartStack(string) error {
p.fullStarted = true
p.calls = append(p.calls, "start")
return nil
}
func (p *recordingProvider) StartStackServices(_ string, services []string) error {
p.gotServices = append([]string(nil), services...)
p.calls = append(p.calls, "startsvc:"+strings.Join(services, ","))
return p.startSvcErr
}
func (p *recordingProvider) GetStackComposePath(string) (string, bool) {
return p.composePath, p.composePath != ""
}
// The app really is up again after StartStack, so the post-restore health wait returns at once.
// Leaving it false would make each test sit through the full 90s deadline.
@@ -78,6 +99,15 @@ func reconFixture(t *testing.T, runID, dumpsAt string, dumpBody string) (*Manage
t.Fatal(err)
}
// The LIVE compose the reconstitution reads to learn WHICH service holds the database (R-47).
// Immich-shaped on purpose: an app service, a redis service that must never be mistaken for a
// database, and a top-level `volumes:` key whose entry looks exactly like a service to a line scan.
liveStackDir := t.TempDir()
prov.composePath = filepath.Join(liveStackDir, "docker-compose.yml")
if err := os.WriteFile(prov.composePath, []byte(immichLikeCompose), 0o644); err != nil {
t.Fatal(err)
}
scratch, liveNs, err := m.offboxRestoreScratchDir("immich")
if err != nil {
t.Fatal(err)
@@ -163,9 +193,10 @@ func TestReconstituteReplaysDBAndOrdersOperations(t *testing.T) {
if res.FilesPlaced != 3 {
t.Fatalf("expected the userdata placement to be counted, got %d", res.FilesPlaced)
}
// stop BEFORE the file copy, start BEFORE the replay (ImportDump needs a live container).
if got := strings.Join(prov.calls, ","); got != "stop,start" {
t.Fatalf("expected stop then start around the restore, got %q", got)
// stop BEFORE the file copy; then ONLY the database service up for the replay (R-47 — a full
// start here is the H4 race); the full start comes last.
if got := strings.Join(prov.calls, ","); got != "stop,startsvc:immich-postgres,start" {
t.Fatalf("expected stop → db-only start → full start around the restore, got %q", got)
}
if res.SafetyDump == "" {
t.Fatal("no safety dump recorded — the undo must exist")