Files
felhom-controller/controller/internal/backup/restore_db.go
T
admin 062357f778 v0.148.0 — coherent snapshot pairs + an offsite restore that actually restores (R-43 + R-44)
Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.

R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.

New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
  - nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
    created after the snapshot survives as an extra;
  - the undo exists before the act — the pre-restore- dump is verified ON DISK
    before anything is stopped, overwritten or replayed; if it cannot be taken
    the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.

R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.

Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.

11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.

NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.

Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
2026-07-19 12:21:16 +02:00

99 lines
3.8 KiB
Go

package backup
import (
"context"
"fmt"
"os"
"path/filepath"
"time"
)
// reimportDBDumps replays the captured per-app .sql dumps back into the app's now-running database
// container(s) — the F17 fix. The per-app backup captures a logical SQL dump (DumpOne →
// <stack>-<dbtype>.sql) but the legacy restore only repopulated Docker volume tars and NEVER replayed
// the dump, so DB-resident data (e.g. rows in a DB whose data dir is a bind mount, not a named volume)
// did not come back. This runs AFTER volume restore + stack bring-up, so the dump WINS over any
// volume-tar copy of the DB (the operator-chosen precedence: the consistent logical dump is authoritative).
//
// It uses the live container's OWN discovered credentials (DiscoveredDB), so no env threading is needed.
// A dump whose DB container is not found is logged and skipped; an actual import FAILURE is returned
// (surfaced, not swallowed) so a failed data restore cannot read as success.
func (m *Manager) reimportDBDumps(ctx context.Context, stackName, nsRoot string) (int, error) {
return m.reimportDBDumpsFrom(ctx, stackName, AppDBDumpPath(nsRoot, stackName))
}
// reimportDBDumpsFrom is reimportDBDumps with an EXPLICIT dump directory. The offsite
// reconstitution path (R-43) replays out of the restored SCRATCH unit rather than the live one:
// the local unit is deliberately never overwritten by a placement, so the dump that belongs to the
// chosen snapshot exists only under the scratch. Same discovery/import seams, same failure
// semantics — only the source directory differs.
func (m *Manager) reimportDBDumpsFrom(ctx context.Context, stackName, dumpDir string) (int, error) {
entries, err := os.ReadDir(dumpDir)
if err != nil {
if os.IsNotExist(err) {
return 0, nil // no DB dumps for this app
}
return 0, fmt.Errorf("reading db-dump dir: %w", err)
}
hasDump := false
for _, e := range entries {
if !e.IsDir() && filepath.Ext(e.Name()) == ".sql" {
hasDump = true
break
}
}
if !hasDump {
return 0, nil
}
discover := m.discoverDBs
if discover == nil {
discover = func(ctx context.Context) ([]DiscoveredDB, error) {
return DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames())
}
}
imp := m.importDBDump
if imp == nil {
imp = func(ctx context.Context, db DiscoveredDB, dumpPath string) error {
return ImportDump(ctx, db, dumpPath, m.logger, m.isDebug())
}
}
dbs, err := discover(ctx)
if err != nil {
return 0, fmt.Errorf("discovering DB containers for %s: %w", stackName, err)
}
var imported int
for _, db := range dbs {
if db.StackName != stackName {
continue
}
// The dump for this DB is named "<stack>-<dbtype>.sql" (see DumpOne).
dumpPath := filepath.Join(dumpDir, fmt.Sprintf("%s-%s.sql", stackName, db.DBType))
if _, statErr := os.Stat(dumpPath); statErr != nil {
continue // no dump for this particular DB engine
}
m.logger.Printf("[INFO] [backup] Restore %s: replaying DB dump into %s (%s)", stackName, db.ContainerName, db.DBType)
if err := imp(ctx, db, dumpPath); err != nil {
return imported, fmt.Errorf("importing %s dump for %s: %w", db.DBType, stackName, err)
}
imported++
}
if imported == 0 {
m.logger.Printf("[WARN] [backup] Restore %s: a .sql dump exists but no matching running DB container was found — DB content NOT restored", stackName)
} else {
m.logger.Printf("[INFO] [backup] Restore %s: replayed %d DB dump(s)", stackName, imported)
}
return imported, nil
}
// reimportDBDumpsCtx is a small helper that runs reimportDBDumps with a bounded context so a stuck DB
// import cannot hang the restore indefinitely.
func (m *Manager) reimportDBDumpsCtx(stackName, nsRoot string) (int, error) {
ctx, cancel := context.WithTimeout(context.Background(), 35*time.Minute)
defer cancel()
return m.reimportDBDumps(ctx, stackName, nsRoot)
}