v0.153.0 — R-47: the DB replay no longer races the app, on BOTH restore paths
Closes R-47. No new agent coupling — MinAgent stays 0.90.0. The replay needs a running DB container, so both restore paths started the WHOLE stack first, giving the application a window to rebuild the very schema objects the dump was about to create. Measured live on 2026-07-19 (H4, DIAG-immich-restore-round2): immich-server rebuilt clip_index two seconds before the dump's CREATE INDEX, the replay aborted "already exists" under ON_ERROR_STOP=1, and immich reported schema drift. The data survived only because pg_dump emits COPY before CREATE INDEX. Both paths now open a DB-ONLY window: only the stack's database service(s) come up, the dump is replayed with the app still down, and the full start runs only after the replay exits 0. Fail-closed: a dump with no identifiable DB service refuses BEFORE the first mutation. Every exit from the window still does a best-effort full start, so a failed restore never leaves a box with a database and no application. New: appbackup.DBServiceNames (yaml.v3 services-map parse — never a line scan; immich's top-level volume keys are the decoy) sharing dbTypeForImage with DiscoverDatabases; stacks.Manager.StartStackServices (refuses an empty list — argument-less `up -d` is a full start); RedeployFromEnv split into PersistUnitRedeployConfig + its unchanged tail. StackDataProvider's RecreateStackFromUnit becomes RecreateStackDefinitionFromUnit — the hidden `up -d` inside the old name is what carried the defect on the local path. 19 new tests (ordering plus state-at-replay-time, zero-mutation fail-closed effects, replay-failure bring-up, parser decoys, empty-list refusal); three companion red-proofs run and reverted. 23/23 packages green. Not yet live-validated: STOP-1 supervised reconstitute, golden 0.153.0.
This commit is contained in:
@@ -39,10 +39,18 @@ type StackDataProvider interface {
|
||||
// fail-closed gate decides what to do. The unit is never the source of secrets.
|
||||
RecoverStackSecrets(name string, names []string) map[string]string
|
||||
|
||||
// RecreateStackFromUnit restores an app's definition from the unit's compose dir into the stack
|
||||
// dir, writes app.yaml from fullEnv (encrypting secret fields), and (re-)deploys it via
|
||||
// `docker compose up -d`, which re-pulls the pinned image. Secrets are NEVER regenerated.
|
||||
RecreateStackFromUnit(name, composeSrcDir string, fullEnv map[string]string) error
|
||||
// RecreateStackDefinitionFromUnit restores an app's DEFINITION from the unit's compose dir into
|
||||
// the stack dir and writes app.yaml from fullEnv (encrypting secret fields). Secrets are NEVER
|
||||
// regenerated. It starts NOTHING: the caller owns the bring-up order, because a DB-bearing app
|
||||
// must have its database service started alone for the dump replay (R-47). It was
|
||||
// `RecreateStackFromUnit` until v0.153.0 and ended in a full `docker compose up -d` — that full
|
||||
// start before the replay IS the H4 race.
|
||||
RecreateStackDefinitionFromUnit(name, composeSrcDir string, fullEnv map[string]string) error
|
||||
|
||||
// StartStackServices brings up ONLY the named compose services, leaving the rest of the stack
|
||||
// down — the DB-only window in which a dump is replayed without the application racing it.
|
||||
// Implementations must REFUSE an empty list (an argument-less `up -d` is a full start).
|
||||
StartStackServices(name string, services []string) error
|
||||
|
||||
// GetStackClassifiedBinds returns the app's backup-classified compose binds + whether it carries a
|
||||
// (valid) backup block (Task 2, referential coupling). INERT — no tier consumes it yet; wired now
|
||||
|
||||
@@ -115,12 +115,10 @@ func DiscoverDatabases(ctx context.Context, logger *log.Logger, debug bool, know
|
||||
|
||||
id, name, image := parts[0], parts[1], strings.ToLower(parts[2])
|
||||
|
||||
var dbType DBType
|
||||
if strings.Contains(image, "postgres") {
|
||||
dbType = DBTypePostgres
|
||||
} else if strings.Contains(image, "mariadb") || strings.Contains(image, "mysql") {
|
||||
dbType = DBTypeMariaDB
|
||||
} else {
|
||||
// R-47: the same predicate that DBServiceNames applies to compose `image:` values, so a dump
|
||||
// that exists is always attributable to a startable service (see dbservices.go).
|
||||
dbType, isDB := dbTypeForImage(image)
|
||||
if !isDB {
|
||||
if debug {
|
||||
logger.Printf("[DEBUG] DiscoverDatabases: skipping container %s (image=%s, not a database)", name, image)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
package appbackup
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// R-47 — naming the database SERVICE, not just the running container.
|
||||
//
|
||||
// A dump replay must never race the application's own schema management. Proven live on 2026-07-19
|
||||
// (DIAG-immich-restore-round2-2026-07-19, H4): the reconstitution started the whole stack before
|
||||
// replaying, immich-server rebuilt `clip_index` two seconds before the dump's own CREATE INDEX, and
|
||||
// the replay aborted `already exists` under ON_ERROR_STOP=1 — leaving a half-applied schema that the
|
||||
// app itself then reported as drift. The fix is to bring up ONLY the database service(s) for the
|
||||
// replay, which requires knowing their compose SERVICE names (docker `up -d <svc>` takes service
|
||||
// names, not container names).
|
||||
//
|
||||
// The symmetry that makes this safe: a `.sql` dump can only exist because DiscoverDatabases matched
|
||||
// the running container's image string, and the compose `image:` value IS that image string. So the
|
||||
// same predicate — dbTypeForImage — decides both "is there a dump" and "which service holds it".
|
||||
|
||||
// dbTypeForImage maps a container/compose image reference to the database engine the backup code
|
||||
// supports, or ok=false for anything else (redis/valkey/app images — never started in the DB-only
|
||||
// phase). Extracted from DiscoverDatabases so the discovery heuristic and the compose heuristic can
|
||||
// never drift apart; behaviour is byte-equivalent to the inline form it replaced.
|
||||
func dbTypeForImage(image string) (DBType, bool) {
|
||||
img := strings.ToLower(image)
|
||||
switch {
|
||||
case strings.Contains(img, "postgres"):
|
||||
return DBTypePostgres, true
|
||||
case strings.Contains(img, "mariadb"), strings.Contains(img, "mysql"):
|
||||
return DBTypeMariaDB, true
|
||||
}
|
||||
return DBType(""), false
|
||||
}
|
||||
|
||||
// composeServicesDoc is the minimal view of a compose file needed here: the `services:` MAP and each
|
||||
// service's `image:`. Deliberately a real YAML parse and not a line scan — a top-level `volumes:`
|
||||
// block (immich's `immich_ml_cache:`) has exactly the shape a naive scan misreads as a service, and
|
||||
// starting a phantom service, or missing the real one, both land in the wrong branch.
|
||||
type composeServicesDoc struct {
|
||||
Services map[string]struct {
|
||||
Image string `yaml:"image"`
|
||||
} `yaml:"services"`
|
||||
}
|
||||
|
||||
// DBServiceNames returns the sorted compose SERVICE names in composePath whose `image:` identifies a
|
||||
// supported database engine — the exact argument list for `docker compose up -d <svc>...`.
|
||||
//
|
||||
// A file with no (or an empty) `services:` key returns (nil, nil): an app with no identifiable DB
|
||||
// service is a legitimate, common case and the caller decides what it means. An unreadable or
|
||||
// unparseable file returns an error, because "cannot tell" must never silently read as "no database"
|
||||
// — the callers turn that into a refusal when a dump exists.
|
||||
//
|
||||
// Image values are matched literally. Catalog templates pin their images literally (enforced since
|
||||
// Campaign 7), so an interpolated `${...}` image simply does not match and lands in the caller's
|
||||
// fail-closed branch by design, rather than being guessed at.
|
||||
func DBServiceNames(composePath string) ([]string, error) {
|
||||
data, err := os.ReadFile(composePath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reading compose file: %w", err)
|
||||
}
|
||||
var doc composeServicesDoc
|
||||
if err := yaml.Unmarshal(data, &doc); err != nil {
|
||||
return nil, fmt.Errorf("parsing compose file %s: %w", composePath, err)
|
||||
}
|
||||
var names []string
|
||||
for name, svc := range doc.Services {
|
||||
if _, ok := dbTypeForImage(svc.Image); ok {
|
||||
names = append(names, name)
|
||||
}
|
||||
}
|
||||
sort.Strings(names)
|
||||
return names, nil
|
||||
}
|
||||
@@ -0,0 +1,195 @@
|
||||
package appbackup
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-47 (v0.153.0) — the DB-service resolver.
|
||||
//
|
||||
// These exist because a dump replay that starts the WHOLE stack races the application's own schema
|
||||
// management: proven live on 2026-07-19 (DIAG-immich-restore-round2-2026-07-19, H4) when
|
||||
// immich-server rebuilt `clip_index` two seconds before the dump's CREATE INDEX and the replay
|
||||
// aborted `already exists`. Closing that window means bringing up ONLY the database service, which
|
||||
// means naming it correctly — every case below is a way of naming it wrongly.
|
||||
|
||||
// writeCompose drops a compose file in a temp dir and returns its path.
|
||||
func writeCompose(t *testing.T, body string) string {
|
||||
t.Helper()
|
||||
p := filepath.Join(t.TempDir(), "docker-compose.yml")
|
||||
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// TestDBTypeForImage pins the shared heuristic. It is the SAME predicate DiscoverDatabases applies to
|
||||
// a running container's image, which is what makes "a dump exists ⇒ a service can be named" hold:
|
||||
// the compose `image:` value IS the container's image string. The table reproduces the inline form
|
||||
// this function replaced, byte for byte, including the redis/valkey negatives that must never be
|
||||
// started in the DB-only window.
|
||||
func TestDBTypeForImage(t *testing.T) {
|
||||
cases := []struct {
|
||||
image string
|
||||
want DBType
|
||||
ok bool
|
||||
}{
|
||||
{"docker.io/library/postgres:16-alpine", DBTypePostgres, true},
|
||||
// immich's real pin — a vector-extended postgres whose REPO segment carries the substring.
|
||||
{"ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0", DBTypePostgres, true},
|
||||
{"postgres", DBTypePostgres, true},
|
||||
{"POSTGRES:16", DBTypePostgres, true}, // the discovery path lowercases; so does this
|
||||
{"mariadb:11", DBTypeMariaDB, true},
|
||||
{"mysql:8.4", DBTypeMariaDB, true},
|
||||
{"docker.io/library/MySQL:8", DBTypeMariaDB, true},
|
||||
{"redis:7-alpine", "", false},
|
||||
{"valkey/valkey:8", "", false},
|
||||
{"ghcr.io/immich-app/immich-server:v1.119.0", "", false},
|
||||
{"", "", false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
got, ok := dbTypeForImage(c.image)
|
||||
if ok != c.ok || (ok && got != c.want) {
|
||||
t.Errorf("dbTypeForImage(%q) = (%q, %v), want (%q, %v)", c.image, got, ok, c.want, c.ok)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDBServiceNames(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
body string
|
||||
want []string
|
||||
}{
|
||||
{
|
||||
name: "postgres service is named",
|
||||
body: "services:\n app:\n image: ghcr.io/x/app:1\n database:\n image: postgres:16\n",
|
||||
want: []string{"database"},
|
||||
},
|
||||
{
|
||||
name: "mariadb service is named",
|
||||
body: "services:\n db:\n image: mariadb:11\n web:\n image: nextcloud:30\n",
|
||||
want: []string{"db"},
|
||||
},
|
||||
{
|
||||
name: "mysql service is named",
|
||||
body: "services:\n mysql:\n image: mysql:8.4\n",
|
||||
want: []string{"mysql"},
|
||||
},
|
||||
{
|
||||
name: "redis-only app has no database service",
|
||||
body: "services:\n app:\n image: ghcr.io/x/app:1\n redis:\n image: redis:7-alpine\n",
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
name: "multiple databases are returned SORTED (one up -d carries them all)",
|
||||
body: "services:\n zdb:\n image: postgres:16\n adb:\n image: mariadb:11\n app:\n image: x:1\n",
|
||||
want: []string{"adb", "zdb"},
|
||||
},
|
||||
{
|
||||
name: "no services key at all",
|
||||
body: "volumes:\n data:\n",
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
name: "empty services map",
|
||||
body: "services:\n",
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
name: "an interpolated image is not guessed at",
|
||||
body: "services:\n db:\n image: ${DB_IMAGE}\n",
|
||||
want: nil,
|
||||
},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
got, err := DBServiceNames(writeCompose(t, c.body))
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(got, c.want) {
|
||||
t.Errorf("DBServiceNames = %v, want %v", got, c.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestDBServiceNames_TopLevelKeysAreNotServices is the decoy test, and the reason this is a YAML
|
||||
// parse rather than a line scan. immich's real compose carries a top-level `volumes:` block whose
|
||||
// entry (`immich_ml_cache:`) sits at exactly the indentation a service name does, and a top-level
|
||||
// `networks:` block does the same. A scanner that collected "indented keys followed by image-ish
|
||||
// lines" would either invent a service that `docker compose up -d` cannot start, or — worse — match
|
||||
// the wrong one and leave the real database down while the app came up around the replay.
|
||||
func TestDBServiceNames_TopLevelKeysAreNotServices(t *testing.T) {
|
||||
// The service/volume/network names and the image pins are the catalog's real immich template.
|
||||
// `immich_postgres_data` is the trap made concrete: a top-level VOLUME key whose name contains
|
||||
// "postgres" and which no `up -d` could ever start.
|
||||
body := `services:
|
||||
immich-server:
|
||||
image: ghcr.io/immich-app/immich-server:v3.0.3
|
||||
immich-machine-learning:
|
||||
image: ghcr.io/immich-app/immich-machine-learning:v3.0.3
|
||||
immich-postgres:
|
||||
image: ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0
|
||||
immich-redis:
|
||||
image: redis:7-alpine
|
||||
volumes:
|
||||
immich_ml_cache:
|
||||
immich_postgres_data:
|
||||
immich_redis_data:
|
||||
networks:
|
||||
traefik-public:
|
||||
external: true
|
||||
immich-internal:
|
||||
`
|
||||
got, err := DBServiceNames(writeCompose(t, body))
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(got, []string{"immich-postgres"}) {
|
||||
t.Fatalf("DBServiceNames = %v, want [immich-postgres] — a top-level volume/network key was mistaken for a service", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDBServiceNames_UnreadableAndUnparseableError proves the fail-closed direction: "cannot tell"
|
||||
// must surface as an ERROR, never as the empty (= "this app has no database") answer. The callers
|
||||
// turn an empty result into a refusal only when a dump exists; if a read failure silently produced
|
||||
// the same empty slice for an app with no dump, a genuinely broken compose would flow on unnoticed.
|
||||
func TestDBServiceNames_UnreadableAndUnparseableError(t *testing.T) {
|
||||
if _, err := DBServiceNames(filepath.Join(t.TempDir(), "nope.yml")); err == nil {
|
||||
t.Fatal("a missing compose file must be an error, not an empty service list")
|
||||
}
|
||||
// Valid YAML scalar where a map is required, plus outright broken YAML.
|
||||
if _, err := DBServiceNames(writeCompose(t, "services: [1, 2, 3\n broken")); err == nil {
|
||||
t.Fatal("an unparseable compose file must be an error, not an empty service list")
|
||||
}
|
||||
}
|
||||
|
||||
// TestDiscoverAndComposeAgreeOnTheSameImages is the SYMMETRY guard: whatever image string makes
|
||||
// DiscoverDatabases produce a dump must also make DBServiceNames name a service. They now share one
|
||||
// predicate; this asserts the property that sharing is FOR, so a future edit to either side that
|
||||
// breaks it fails here rather than in a customer's restore.
|
||||
func TestDiscoverAndComposeAgreeOnTheSameImages(t *testing.T) {
|
||||
images := []string{"postgres:16", "mariadb:11", "mysql:8.4", "redis:7", "ghcr.io/x/app:1"}
|
||||
var body strings.Builder
|
||||
body.WriteString("services:\n")
|
||||
var wantDB []string
|
||||
for i, img := range images {
|
||||
svc := string(rune('a' + i))
|
||||
body.WriteString(" " + svc + ":\n image: " + img + "\n")
|
||||
if _, ok := dbTypeForImage(img); ok {
|
||||
wantDB = append(wantDB, svc)
|
||||
}
|
||||
}
|
||||
got, err := DBServiceNames(writeCompose(t, body.String()))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reflect.DeepEqual(got, wantDB) {
|
||||
t.Fatalf("compose resolver named %v but the discovery predicate says %v — the two sides have drifted", got, wantDB)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user