diff --git a/CHANGELOG.md b/CHANGELOG.md index a01088a9..250a37e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,30 @@ brief 2026-10-24, Part 4 (commit only; the release rides the next normal train). `…OnlyWhenTheAgentWaitsAndAnAppRuns`, `…QBLimitRuleSkipsAHeldLimit`, `…TheTwoAppsReadAndWriteTheirLimit`, `TestLineTestRunOutlastsTheClientTimeout`, `TestStartUpwatchRecoversTheHoldFirst`, `TestUpwatchHoldGetsBothAgentCalls`. +### A restore downloads first (D48) — 2026-10-25, commit only + +Operator ruling D48 (2026-10-25 08:38, `09` §3 decision 257): every restore path resolves and pulls every image its +definition names before the first change to the app; a failed pull is a refusal with the app untouched. Design: +felhom.eu `07` §6.6. + +- **Before (`e1613ae`):** the order was checks → stop → copy → volumes → definition → DB-only start (the FIRST pull) → + replay → start; a version that could not be downloaded ended in the D22 put-back, the app down meanwhile. +- **Now (`internal/backup/restore_pull.go`, `prepareRestoreImages`):** after the existing refusals and before the stop, + each image of the definition that will RUN is present or pulled now, under the image-work lock held SHARED until the + restore returns (`dockerexec.HoldImageWork`, R-863 — the clean-up's TryLock cannot run in between; nesting is safe + because nothing takes the lock exclusively and waits, pinned). Paths (each at its own line): the local unit restore + — so the second drive's unit restore, the whole restore (its `CheckOnly` pass carries the pull; the whole restore + holds the lock across its file half), a kept load (local and off-site) and a set-aside unit — the off-site restore, + the second drive's file restore and the no-unit fallback. +- **A failed pull:** the household reads „Ennek a mentésnek a verzióját (%s) nem lehet letölteni: a készítője már nem + kínálja…" when Docker says the version does not exist (`manifest unknown`, `not found`, `pull access denied`, + `repository does not exist`), „…most nem lehet letölteni… Próbáld újra később…" for anything else; the operator gets + one `backup_run_failures` line (leg `restore-image-unavailable`). Not covered: the `.fab` import (its own row). +- Tests (each seen failing with the pull switched off): `TestD48_AnUnpullablePinIsRefusedBeforeTheStop`, + `…AnUnreachableRegistryReadsDifferently`, `…APullablePinIsPulledFirstAndTheRestoreRuns`, + `…TheCheckOnlyPassCarriesThePull`, `…TheOffsiteRestoreToo`, `…PullGone`; dockerexec + `TestD48_HoldImageWorkBlocksCleanupAndNests`, `…TheOnlyExclusiveTakerNeverWaits`. + ### A restore whose app does not start is never „success" (R-1270, P2) — 2026-10-25, commit only - **Measured before (9202, 0.342.0 = `e1613ae` for these files; `audits/restore-truth-2026-10-25/before/`):** the diff --git a/controller/cmd/controller/main.go b/controller/cmd/controller/main.go index 0568fd79..194839a2 100644 --- a/controller/cmd/controller/main.go +++ b/controller/cmd/controller/main.go @@ -1541,6 +1541,15 @@ func main() { }) // R-893 option A (D20 = A): a failed off-site restore that was PUT BACK — the live volumes and definition poured // back and the app started. The same operator-tier event as the hold above (no new event type), its own leg. + // D48: a restore refused BEFORE anything was touched because an image of the backup's version could not be + // downloaded — the operator line that tells a deleted version (R-698, R-933) from a busy registry. + backupMgr.SetRestorePullRefusedNotify(func(stack, ref string, cause error) { + d := notify.BackupRunFailuresDetails{RunKind: "restore", Failed: 1, Attempted: 1} + d.Apps = append(d.Apps, notify.RunFailureDetail{App: stack, Leg: "restore-image-unavailable", Reason: cause.Error()}) + msg := fmt.Sprintf("App %q: a restore was REFUSED before anything was touched — the image %s could not be downloaded (D48). "+ + "The app runs as it was; the household was told. Cause: %v", stack, ref, cause) + notifier.NotifyBackupRunFailures(msg, d) + }) // R-1270: a restore whose data is back but whose app did not start in time — one operator line, its own leg. backupMgr.SetRestoreNotStartedNotify(func(stack, kind string, cause error) { d := notify.BackupRunFailuresDetails{RunKind: "restore", Failed: 1, Attempted: 1} diff --git a/controller/internal/backup/backup.go b/controller/internal/backup/backup.go index a39e6ab9..547f9ca3 100644 --- a/controller/internal/backup/backup.go +++ b/controller/internal/backup/backup.go @@ -243,6 +243,11 @@ type Manager struct { restorePutBackNotify func(stack string, replayErr error) // restoreNotStartedNotify (R-1270) is called once when a restore's data is back but its app did not start in time. restoreNotStartedNotify func(stack, kind string, cause error) + // D48: the restore's image probe and pull (nil → docker; tests set them, TestMain makes every image present), and + // the operator notice of a restore refused because an image could not be downloaded. + imagePresentFn func(ref string) bool + imagePullFn func(ctx context.Context, ref string) (string, error) + restorePullRefusedNotify func(stack, ref string, cause error) // R-403 unit-REHYDRATE seam — the copy that refills the primary unit from the Tier-2 mirror after // a restore, overridable so the "only when hollow" condition and the byte-identical non-effect are diff --git a/controller/internal/backup/main_test.go b/controller/internal/backup/main_test.go index 213eda36..3b47343a 100644 --- a/controller/internal/backup/main_test.go +++ b/controller/internal/backup/main_test.go @@ -19,6 +19,9 @@ func TestMain(m *testing.M) { healthSettle = 0 healthInterval = 2 * time.Millisecond healthTimeout = 50 * time.Millisecond + // D48 / R-650: no test in this package reaches Docker for the restore's pre-pull — every image is present unless a + // test says otherwise (m.imagePresentFn / m.imagePullFn). + imagePresentDefault = func(string) bool { return true } os.Exit(m.Run()) } diff --git a/controller/internal/backup/offbox_reconstitute.go b/controller/internal/backup/offbox_reconstitute.go index 2ae4e69d..c5517280 100644 --- a/controller/internal/backup/offbox_reconstitute.go +++ b/controller/internal/backup/offbox_reconstitute.go @@ -946,6 +946,20 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string, ack } } + // D48: DOWNLOAD FIRST — every image of the definition that will run, before the stop, or a refusal with the app + // untouched. Held until this restore returns (R-863). + runCompose := filepath.Join(scratchCompose, "docker-compose.yml") + if !defineFromSnapshot { + if live, ok := m.stackProvider.GetStackComposePath(stack); ok && live != "" { + runCompose = live + } + } + releaseImages, pErr := m.prepareRestoreImages(stack, runCompose) + if pErr != nil { + return res, pErr + } + defer releaseImages() + // --- FILES ---------------------------------------------------------------------------------- // R-166: mark the stop→restore→start window BEFORE stopping. A controller killed anywhere inside // it used to leave the app down with nothing on disk recording that it was owed a restart — and a diff --git a/controller/internal/backup/restore.go b/controller/internal/backup/restore.go index ab550e0e..5870a337 100644 --- a/controller/internal/backup/restore.go +++ b/controller/internal/backup/restore.go @@ -79,6 +79,15 @@ func (m *Manager) RestoreApp(stackName, snapshotID string) error { } } + // D48: the live definition's images are on the box before the stop, or the restore is refused untouched. + if composePath, ok := m.stackProvider.GetStackComposePath(stackName); ok && composePath != "" { + releaseImages, pErr := m.prepareRestoreImages(stackName, composePath) + if pErr != nil { + return pErr + } + defer releaseImages() + } + // Stop the app before restore if m.isDebug() { m.logger.Printf("[DEBUG] RestoreApp: step 1/4 — stopping app %s", stackName) diff --git a/controller/internal/backup/restore_pull.go b/controller/internal/backup/restore_pull.go new file mode 100644 index 00000000..61da0e59 --- /dev/null +++ b/controller/internal/backup/restore_pull.go @@ -0,0 +1,111 @@ +package backup + +import ( + "context" + "errors" + "fmt" + "strings" + "time" + + "gitea.dooplex.hu/admin/felhom-controller/internal/dockerexec" + "gitea.dooplex.hu/admin/felhom-controller/internal/util" +) + +// ── D48 (operator ruling 2026-10-25 08:38; `09` §3 decision 257): a restore downloads first ───────────────────────── +// +// BEFORE: every restore stopped the app and moved its data first; the first pull was inside the database-only start +// (`docker compose up -d `), after the stop. A version that could not be downloaded then ended in the D22 put-back — +// the whole stop, copy and pour-back for nothing, and an app that was down the while. +// +// THE RULE. After the existing refusals and before the first stop or change, every image the definition that will RUN +// names is resolved: present on the box, or pulled now. Under the image-work lock held SHARED from the first check +// until the restore's last start (R-863: the image clean-up cannot delete a pulled image before the start — its pass +// takes the lock exclusively with TryLock and simply does not run meanwhile). A failed pull is a REFUSAL: the app is +// not stopped and its data is not moved. The household reads which version could not be downloaded, and why — a +// version its maker deleted (Docker answers „manifest unknown", „not found", „pull access denied") reads differently +// from a registry that is busy or unreachable (anything else, „toomanyrequests" and timeouts included). + +// ErrRestoreImageUnavailable is the KIND of the refusal: an image of the backup's version could not be downloaded. +var ErrRestoreImageUnavailable = errors.New("an image of the backup's version cannot be downloaded") + +// restorePullTimeout bounds one image pull of a restore. +const restorePullTimeout = 15 * time.Minute + +// imagePresentDefault / imagePullDefault are the production probe and pull (tests replace them in TestMain, so no test +// in this package reaches Docker — R-650; a test that needs a missing image sets m.imagePresentFn / m.imagePullFn). +var ( + imagePresentDefault = func(ref string) bool { + return dockerexec.Command("docker", "image", "inspect", "--format", "{{.Id}}", ref).Run() == nil + } + imagePullDefault = func(ctx context.Context, ref string) (string, error) { + out, err := dockerexec.CommandContext(ctx, "docker", "pull", ref).CombinedOutput() + return string(out), err + } +) + +func (m *Manager) imagePresent(ref string) bool { + if m.imagePresentFn != nil { + return m.imagePresentFn(ref) + } + return imagePresentDefault(ref) +} + +func (m *Manager) imagePull(ctx context.Context, ref string) (string, error) { + if m.imagePullFn != nil { + return m.imagePullFn(ctx, ref) + } + return imagePullDefault(ctx, ref) +} + +// pullGone reports whether Docker's answer says the version does not exist (deleted by its maker, a wrong name) rather +// than that the registry could not be reached or is busy. +func pullGone(out string) bool { + low := strings.ToLower(out) + for _, s := range []string{"manifest unknown", "not found", "repository does not exist", "pull access denied", "name unknown", "no such manifest"} { + if strings.Contains(low, s) { + return true + } + } + return false +} + +// prepareRestoreImages resolves every image composePath names and pulls the absent ones, holding the image-work lock +// until release(). err = the refusal (the household's sentence); the lock is released already then. +func (m *Manager) prepareRestoreImages(stack, composePath string) (release func(), err error) { + refs := ParseComposeImages(composePath) + end := dockerexec.HoldImageWork() + for _, ref := range refs { + if m.imagePresent(ref) { + continue + } + start := time.Now() + ctx, cancel := context.WithTimeout(context.Background(), restorePullTimeout) + out, perr := m.imagePull(ctx, ref) + cancel() + if perr != nil { + end() + tail := strings.TrimSpace(out) + if len(tail) > 300 { + tail = tail[len(tail)-300:] + } + key := "err.backup.restore_image_unreachable" + why := "the registry did not answer or is busy" + if pullGone(out) { + key, why = "err.backup.restore_image_gone", "the version does not exist any more" + } + m.logger.Printf("[ERROR] [backup] Restore REFUSED for %s before anything was touched: the image %s could not be downloaded (%s; after %s): %v — %s (D48)", + stack, ref, why, time.Since(start).Round(time.Second), perr, tail) + if m.restorePullRefusedNotify != nil { + m.restorePullRefusedNotify(stack, ref, fmt.Errorf("%s: %v", why, perr)) + } + return func() {}, util.MsgErrorf(ErrRestoreImageUnavailable, key, PinsVersion([]string{ref})) + } + m.logger.Printf("[INFO] [backup] Restore %s: downloaded %s before anything was touched (%s, D48)", stack, ref, time.Since(start).Round(time.Second)) + } + return end, nil +} + +// SetRestorePullRefusedNotify wires the operator notice for a restore refused because an image could not be downloaded. +func (m *Manager) SetRestorePullRefusedNotify(fn func(stack, ref string, cause error)) { + m.restorePullRefusedNotify = fn +} diff --git a/controller/internal/backup/restore_pull_test.go b/controller/internal/backup/restore_pull_test.go new file mode 100644 index 00000000..8570bd54 --- /dev/null +++ b/controller/internal/backup/restore_pull_test.go @@ -0,0 +1,138 @@ +package backup + +import ( + "context" + "errors" + "strings" + "testing" + + "gitea.dooplex.hu/admin/felhom-controller/internal/util" +) + +// D48 (operator ruling 2026-10-25; `09` §3 decision 257) — a restore downloads first. Before (`e1613ae`): the first pull +// was inside the database-only start, AFTER the stop; a version that could not be downloaded ended in the D22 put-back. +// COMPANION RED-PROOF (REPORT.md): with prepareRestoreImages not called, TestD48_AnUnpullablePinIsRefusedBeforeTheStop +// and TestD48_TheOffsiteRestoreToo fail — the app is stopped (the calls start with "stop"). + +func d48Unpullable(m *Manager, answer string) *[]string { + var pulled []string + m.imagePresentFn = func(string) bool { return false } + m.imagePullFn = func(_ context.Context, ref string) (string, error) { + pulled = append(pulled, ref) + return answer, errors.New("exit status 1") + } + return &pulled +} + +// An unpullable pin: refused, the stop never called, nothing poured, no copy made, no definition written. +func TestD48_AnUnpullablePinIsRefusedBeforeTheStop(t *testing.T) { + m, p, rig, imported := r1177Fixture(t) + pulled := d48Unpullable(m, "Error response from daemon: manifest for example/app:9.9.9 not found: manifest unknown") + var notified []string + m.SetRestorePullRefusedNotify(func(stack, ref string, _ error) { notified = append(notified, stack+" "+ref) }) + + _, err := m.RestoreFromRecoveryUnit("app") + if !errors.Is(err, ErrRestoreImageUnavailable) || len(*pulled) == 0 { + t.Fatalf("an unpullable pin must refuse, got %v (pulled %v)", err, *pulled) + } + if want := util.MsgError("err.backup.restore_image_gone", PinsVersion([]string{(*pulled)[0]})).Error(); err.Error() != want { + t.Fatalf("a deleted version reads its own sentence, got %q", err.Error()) + } + if p.stopped || len(p.calls) != 0 { + t.Fatalf("the app must not be stopped or touched at all, calls %v", p.calls) + } + if len(rig.poured) != 0 || len(rig.defsWrote) != 0 || len(*imported) != 0 || len(undoCopies(t, rig.undoRoot)) != 0 { + t.Fatalf("nothing may move: poured %v, defs %v, imported %v", rig.poured, rig.defsWrote, *imported) + } + if len(notified) != 1 { + t.Fatalf("the operator gets one notice, got %v", notified) + } +} + +// A registry that is busy or unreachable reads differently from a deleted version. +func TestD48_AnUnreachableRegistryReadsDifferently(t *testing.T) { + m, p, _, _ := r1177Fixture(t) + pulled := d48Unpullable(m, "Error response from daemon: Get \"https://registry-1.docker.io/v2/\": dial tcp: lookup registry-1.docker.io: i/o timeout") + _, err := m.RestoreFromRecoveryUnit("app") + if len(*pulled) == 0 { + t.Fatalf("the restore never asked for the image (err %v)", err) + } + if want := util.MsgError("err.backup.restore_image_unreachable", PinsVersion([]string{(*pulled)[0]})).Error(); err == nil || err.Error() != want { + t.Fatalf("an unreachable registry reads „try again later\", got %v", err) + } + if p.stopped { + t.Fatal("the app must not be stopped") + } +} + +// A pullable pin: pulled BEFORE the stop, and the restore runs as before. +func TestD48_APullablePinIsPulledFirstAndTheRestoreRuns(t *testing.T) { + m, p, _, imported := r1177Fixture(t) + m.imagePresentFn = func(string) bool { return false } + m.imagePullFn = func(_ context.Context, ref string) (string, error) { + p.calls = append(p.calls, "pull:"+ref) + return "Status: Downloaded newer image", nil + } + if _, err := m.RestoreFromRecoveryUnit("app"); err != nil { + t.Fatalf("a pullable restore runs: %v", err) + } + if len(p.calls) == 0 || !strings.HasPrefix(p.calls[0], "pull:") { + t.Fatalf("the pull must come before the stop, calls %v", p.calls) + } + stop := -1 + for i, c := range p.calls { + if c == "stop" { + stop = i + break + } + } + for i, c := range p.calls { + if strings.HasPrefix(c, "pull:") && i > stop { + t.Fatalf("every pull before the stop, calls %v", p.calls) + } + } + if len(*imported) != 1 { + t.Fatalf("the restore ran as before (one database replayed), got %v", *imported) + } +} + +// The second drive's whole restore asks the unit half BEFORE its file half (R-1178): the pull is in that check. +func TestD48_TheCheckOnlyPassCarriesThePull(t *testing.T) { + m, p, _, _ := r1177Fixture(t) + d48Unpullable(m, "manifest unknown") + unit := m.primaryUnitDirFor("app") + if _, err := m.RestoreFromRecoveryUnitAtWith("app", unit, UnitRestoreOptions{CheckOnly: true}); !errors.Is(err, ErrRestoreImageUnavailable) { + t.Fatalf("the check must refuse an unpullable pin, got %v", err) + } + if p.stopped { + t.Fatal("the check never stops the app") + } +} + +// The off-site restore: refused before the stop too. +func TestD48_TheOffsiteRestoreToo(t *testing.T) { + m, prov, imported := reconFixture(t, "20260719T060000Z", "2026-07-19T06:00:00Z", pgDump(1)) + d48Unpullable(m, "pull access denied for example/immich, repository does not exist") + _, err := m.ReconstituteFromOffsite(context.Background(), "immich", false) + if !errors.Is(err, ErrRestoreImageUnavailable) { + t.Fatalf("an off-site restore with an unpullable image must refuse, got %v", err) + } + if len(prov.calls) != 0 || len(*imported) != 0 { + t.Fatalf("the app must not be stopped and nothing replayed: calls %v, imported %v", prov.calls, *imported) + } +} + +// Docker's answers, as the two sentences read them. +func TestD48_PullGone(t *testing.T) { + for out, want := range map[string]bool{ + "manifest for x:1 not found: manifest unknown": true, + "pull access denied for x, repository does not exist or may require": true, + "toomanyrequests: You have reached your pull rate limit": false, + "dial tcp: lookup registry-1.docker.io: i/o timeout": false, + "net/http: TLS handshake timeout": false, + } { + if got := pullGone(out); got != want { + t.Errorf("pullGone(%q) = %v, want %v", out, got, want) + } + } +} diff --git a/controller/internal/backup/restore_unit.go b/controller/internal/backup/restore_unit.go index 22cabc70..5591df07 100644 --- a/controller/internal/backup/restore_unit.go +++ b/controller/internal/backup/restore_unit.go @@ -463,6 +463,21 @@ func (m *Manager) RestoreFromRecoveryUnitAtWith(stackName, unitDir string, opt U } undoExists = ex } + // D48 (`09` §3 decision 257): DOWNLOAD FIRST. Every image the definition that will run names is on the box before + // the stop — or the restore is refused here, with the app running and its data where it was. The lock is held + // until this function returns, past the last start (R-863). The CheckOnly pass carries it too (R-1178), so the + // second drive's file half never runs before a pull that would refuse. + runCompose := filepath.Join(composeDir, "docker-compose.yml") + if !fileExists(runCompose) { + if live, ok := m.stackProvider.GetStackComposePath(stackName); ok && live != "" { + runCompose = live + } + } + releaseImages, pErr := m.prepareRestoreImages(stackName, runCompose) + if pErr != nil { + return res, pErr + } + defer releaseImages() if opt.CheckOnly { return res, nil } diff --git a/controller/internal/backup/tier2_restore.go b/controller/internal/backup/tier2_restore.go index 9339d6e0..55ec4821 100644 --- a/controller/internal/backup/tier2_restore.go +++ b/controller/internal/backup/tier2_restore.go @@ -420,6 +420,15 @@ func (m *Manager) RestoreTier2Files(stackName string) (filesRestored int, err er } m.logger.Printf("[INFO] [backup] Tier-2 file restore for %s: %s (v2) → %s (additive-only)", stackName, destBase, liveNsRoot) + // D48: the live definition's images are on the box before the stop, or the restore is refused untouched. + if composePath, ok := m.stackProvider.GetStackComposePath(stackName); ok && composePath != "" { + releaseImages, pErr := m.prepareRestoreImages(stackName, composePath) + if pErr != nil { + return 0, pErr + } + defer releaseImages() + } + // Stop → copy → start → health (the standard restore shape; F17: errors surface, never swallowed). if stopErr := m.stackProvider.StopStack(stackName); stopErr != nil { m.logger.Printf("[WARN] [backup] could not stop %s before Tier-2 file restore: %v (continuing)", stackName, stopErr) diff --git a/controller/internal/backup/tier2_whole.go b/controller/internal/backup/tier2_whole.go index 3264292d..dc77c112 100644 --- a/controller/internal/backup/tier2_whole.go +++ b/controller/internal/backup/tier2_whole.go @@ -12,6 +12,7 @@ import ( "syscall" "time" + "gitea.dooplex.hu/admin/felhom-controller/internal/dockerexec" "gitea.dooplex.hu/admin/felhom-controller/internal/util" ) @@ -307,6 +308,8 @@ func (m *Manager) RestoreTier2Whole(stackName string) (WholeRestoreResult, error // R-1178: the unit half's refusals are asked HERE, before the file half — measured before: files merged, then the // unit refused (a cut-off database copy) and the app was left STOPPED under a page that said „untouched". unitDir := tier2UnitDir(destBase) + // D48: the images the unit half pulls in its check stay on the box through the file half and the unit half. + defer dockerexec.HoldImageWork()() if _, err := m.RestoreFromRecoveryUnitAtWith(stackName, unitDir, UnitRestoreOptions{AcceptMissingFiles: true, CheckOnly: true}); err != nil { m.logger.Printf("[WARN] [backup] whole restore REFUSED for %s before the file half: the unit half would refuse (%v) — nothing moved", stackName, err) return res, err diff --git a/controller/internal/dockerexec/imagework.go b/controller/internal/dockerexec/imagework.go index dbdf4e9b..799931ad 100644 --- a/controller/internal/dockerexec/imagework.go +++ b/controller/internal/dockerexec/imagework.go @@ -52,6 +52,16 @@ func BeginImageWork(args []string) (end func()) { return imageWorkMu.RUnlock } +// HoldImageWork holds the image-work lock SHARED until the returned func runs, whatever the verb (D48, 2026-10-25): a +// restore resolves and pulls its images BEFORE it stops the app and starts them minutes later, so the clean-up must +// not run anywhere in between. A compose command inside the hold takes the lock shared again (BeginImageWork); that +// nests safely because the ONLY exclusive taker is TryImageCleanup's TryLock, which never waits — so no writer can be +// queued between the two shared holds. Pinned by TestD48_HoldImageWorkBlocksCleanupAndNests. +func HoldImageWork() (end func()) { + imageWorkMu.RLock() + return imageWorkMu.RUnlock +} + // TryImageCleanup takes the image-work lock exclusively if no image work runs now. ok=false: something is // pulling — do not clean up now. func TryImageCleanup() (end func(), ok bool) { diff --git a/controller/internal/dockerexec/imagework_d48_test.go b/controller/internal/dockerexec/imagework_d48_test.go new file mode 100644 index 00000000..26c75662 --- /dev/null +++ b/controller/internal/dockerexec/imagework_d48_test.go @@ -0,0 +1,45 @@ +package dockerexec + +import ( + "os" + "strings" + "testing" + "time" +) + +// D48 — a restore holds the image-work lock from its pre-pull until its last start. While it is held the clean-up +// cannot run, and a compose up inside the hold does not block. +func TestD48_HoldImageWorkBlocksCleanupAndNests(t *testing.T) { + end := HoldImageWork() + if _, ok := TryImageCleanup(); ok { + t.Fatal("the image clean-up must not run while a restore holds its images") + } + done := make(chan struct{}) + go func() { + defer close(done) + BeginImageWork([]string{"compose", "up", "-d"})() + }() + select { + case <-done: + case <-time.After(2 * time.Second): + t.Fatal("a compose up inside the restore's hold must not block") + } + end() + unlock, ok := TryImageCleanup() + if !ok { + t.Fatal("after the hold the clean-up runs again") + } + unlock() +} + +// The nesting is safe only while the one exclusive taker never WAITS (a waiting writer would block a nested RLock). +// Pin it: the package takes the lock exclusively nowhere but TryLock. +func TestD48_TheOnlyExclusiveTakerNeverWaits(t *testing.T) { + b, err := os.ReadFile("imagework.go") + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(b), "imageWorkMu.Lock()") { + t.Fatal("a blocking exclusive Lock() on imageWorkMu would deadlock a restore's nested shared hold (HoldImageWork)") + } +} diff --git a/controller/internal/i18n/locales/en.json b/controller/internal/i18n/locales/en.json index 0461e86d..255121e9 100644 --- a/controller/internal/i18n/locales/en.json +++ b/controller/internal/i18n/locales/en.json @@ -2862,5 +2862,7 @@ "flash.kept.restore_needs_confirmation": "Confirm the question to restore it.", "note.tier2.unit_other_install_kept": "The second drive holds the data package of an earlier install, and we could not set it aside now, so we did not overwrite it. The data package of the copy is therefore older than this backup.", "err.backup.restore_not_started": "%s: we loaded the data back, but the app has not started yet. Wait a few minutes, then open it again. If it stays like this, tell us.", - "note.restore.late_started": "%s has started: the restore is done." + "note.restore.late_started": "%s has started: the restore is done.", + "err.backup.restore_image_gone": "The version of this backup (%s) cannot be downloaded: its maker no longer offers it. We did not touch the app. Choose another backup.", + "err.backup.restore_image_unreachable": "The version of this backup (%s) cannot be downloaded right now, so we did not touch the app. Try again later, or choose another backup." } diff --git a/controller/internal/i18n/locales/hu.json b/controller/internal/i18n/locales/hu.json index de99b314..9c8214f6 100644 --- a/controller/internal/i18n/locales/hu.json +++ b/controller/internal/i18n/locales/hu.json @@ -2850,5 +2850,7 @@ "flash.kept.restore_needs_confirmation": "A visszaállításhoz erősítsd meg a kérdést.", "note.tier2.unit_other_install_kept": "A második meghajtón egy korábbi telepítés adatcsomagja van, és most nem tudtuk félretenni, ezért nem írtuk felül. A másolat adatcsomagja ezért régebbi, mint ez a mentés.", "err.backup.restore_not_started": "A(z) %s: az adatokat visszatöltöttük, de az alkalmazás még nem indult el. Várj pár percet, majd nyisd meg újra. Ha így marad, szólj nekünk.", - "note.restore.late_started": "A(z) %s elindult: a visszaállítás kész." + "note.restore.late_started": "A(z) %s elindult: a visszaállítás kész.", + "err.backup.restore_image_gone": "Ennek a mentésnek a verzióját (%s) nem lehet letölteni: a készítője már nem kínálja. Az alkalmazáshoz nem nyúltunk. Válassz egy másik mentést.", + "err.backup.restore_image_unreachable": "Ennek a mentésnek a verzióját (%s) most nem lehet letölteni, ezért az alkalmazáshoz nem nyúltunk. Próbáld újra később, vagy válassz egy másik mentést." } diff --git a/controller/scripts/i18n_go_keys.json b/controller/scripts/i18n_go_keys.json index 9d43645d..013b2656 100644 --- a/controller/scripts/i18n_go_keys.json +++ b/controller/scripts/i18n_go_keys.json @@ -324,7 +324,9 @@ "flash.kept.restore_needs_confirmation": "BORN AS A KEY (D47, operator ruling 2026-10-25; R-1269) -- the unit of an earlier install is set aside whole and restorable from the restore page; its confirm flash and the second-drive note when it cannot be set aside. Pinned by internal/web/kept_copies_test.go TestD47_* and internal/backup/unit_owner_test.go.", "note.tier2.unit_other_install_kept": "BORN AS A KEY (D47, operator ruling 2026-10-25; R-1269) -- the unit of an earlier install is set aside whole and restorable from the restore page; its confirm flash and the second-drive note when it cannot be set aside. Pinned by internal/web/kept_copies_test.go TestD47_* and internal/backup/unit_owner_test.go.", "err.backup.restore_not_started": "BORN AS A KEY (R-1270, brief 2026-10-25 Part 3) -- a restore whose data is back but whose app did not start in time is never a success; its sentence, and the late start that turns it into one. Pinned by internal/backup/restore_verdict_test.go TestR1270_*.", - "note.restore.late_started": "BORN AS A KEY (R-1270, brief 2026-10-25 Part 3) -- a restore whose data is back but whose app did not start in time is never a success; its sentence, and the late start that turns it into one. Pinned by internal/backup/restore_verdict_test.go TestR1270_*." + "note.restore.late_started": "BORN AS A KEY (R-1270, brief 2026-10-25 Part 3) -- a restore whose data is back but whose app did not start in time is never a success; its sentence, and the late start that turns it into one. Pinned by internal/backup/restore_verdict_test.go TestR1270_*.", + "err.backup.restore_image_gone": "BORN AS A KEY (D48, operator ruling 2026-10-25; 09 decision 257) -- a restore downloads first: the refusal when an image of the backup's version cannot be downloaded (deleted by its maker, or the registry busy or unreachable); the app untouched. Pinned by internal/backup/restore_pull_test.go TestD48_*.", + "err.backup.restore_image_unreachable": "BORN AS A KEY (D48, operator ruling 2026-10-25; 09 decision 257) -- a restore downloads first: the refusal when an image of the backup's version cannot be downloaded (deleted by its maker, or the registry busy or unreachable); the app untouched. Pinned by internal/backup/restore_pull_test.go TestD48_*." }, "flash.share.already_on": "A megosztás már be van kapcsolva.", "flash.share.enable_failed": "A megosztás bekapcsolása nem sikerült.",