v0.189.0 — desired state + the app-stop crash marker (R-166 / D-b)

The box stops inferring the customer's intent from a container count and reads
what they actually asked for.

Part 1 — desired state. AppConfig gains a tri-state `desired_state`
(""/running/stopped), written ONLY by the customer's own action: the API action
switch, DeployStack, UpdateOptionalConfig's redeploy branch, and the .fab
import. Intent is written BEFORE the act and a failed write REFUSES the act.
StartStack/StopStack are deliberately not writers — 14 callers, only 2 are the
customer. bootrecon.isBootOrphan now reads intent instead of len(Containers)>0,
which closes R-157 mechanism B (a power cut or interrupted deploy left an app
with zero containers, read as a deliberate stop, and stranded silently).

ABSENT MEANS UNKNOWN, NEVER "running": every pre-v0.189.0 app.yaml reads absent,
so the legacy fallback is byte-identical to the old rule. A running-only startup
backfill converges the unambiguous cases; `stopped` is never inferred.

Part 2 — backup.AppStopGuard, a persisted marker over every stop→work→start
window (volume dump, offbox reconstitute, .fab export). Its own file, never
quiesce's. Written before the stop, cleared only after a restart that succeeded,
kept when one fails. Recover() completes before the boot reconciler is launched
and returns its outcome, which main.go reports on the existing backup_failed
event once the notifier exists. A defer is not the mechanism — a SIGKILL runs
none (Campaign 8 fault 10).

Also: SaveAppConfig rebuilt AppConfig field-by-field (the R-100 shape) and would
have dropped desired_state on every save across nine call sites. Replaced with
copy-and-overlay. Measured: app.yaml does not round-trip unknown YAML keys.

No hub change, no agent coupling, no user-visible string. 27/27 packages green;
7 red-proofs observed FAIL then restored.

controller/internal/backup/appstop_marker.go

@@ -0,0 +1,281 @@
+package backup
+
+import (
+	"encoding/json"
+	"fmt"
+	"log"
+	"os"
+	"path/filepath"
+	"sort"
+	"time"
+)
+
+// ── The app-stop marker (R-166 part 2, decision D-b "in-flight operations") ───────────────────────
+//
+// Several operations stop a customer's app, do something to its data, and start it again. Between
+// the stop and the start, NOTHING ON DISK RECORDED THAT AN APP WAS OWED A RESTART. A controller that
+// died in that window left the app down with no explanation anywhere — and because a stopped app has
+// zero containers, the boot reconciler read it as a deliberate customer stop and deliberately left
+// it alone. Silently, indefinitely.
+//
+// A `defer` is NOT the fix and must never be described as one. Campaign 8 fault 10 established this
+// on live hardware: a SIGKILL runs no deferred function, and what brought the quiesce loop's stacks
+// back was its persisted marker read by Recover() one second after restart. The defer covers the
+// graceful exits; the marker covers the hard crash and the power cut. This file is that marker for
+// the app-data path, modelled directly on internal/quiesce's.
+//
+// WHY ITS OWN FILE, not quiesce's: one file, one writer. Quiesce's marker records a whole-guest
+// backup window and is written by the quiesce loop; this one records an app-data operation and is
+// written by the backup manager and the exporter. Sharing the file would give it two writers with
+// two lifetimes, and one clearing the other's record is a stranded app by a different route.
+//
+// SAFETY (D-b's binding rule): losing this file must never be worse than not having it. A lost or
+// corrupt marker means the app is not auto-restarted by THIS mechanism — which is precisely the
+// pre-v0.189.0 position, not a new hazard. It never deletes, restores, or touches a backup artifact.
+
+// AppStopReason names WHY an app was stopped, so the recovery log tells an operator which operation
+// was interrupted rather than merely that something was.
+type AppStopReason string
+
+const (
+	// ReasonVolumeDump — DumpAppVolumesSafe: stop, tar the volumes consistently, start.
+	ReasonVolumeDump AppStopReason = "volume_dump"
+	// ReasonOffboxReconstitute — a full offsite restore overwriting the app's files.
+	ReasonOffboxReconstitute AppStopReason = "offbox_reconstitute"
+	// ReasonAppExport — a .fab export taken with "stop the app first".
+	ReasonAppExport AppStopReason = "app_export"
+)
+
+// humanReason is the operator-facing phrasing for each reason.
+func (r AppStopReason) humanReason() string {
+	switch r {
+	case ReasonVolumeDump:
+		return "an app-data backup (volume dump)"
+	case ReasonOffboxReconstitute:
+		return "an off-site restore"
+	case ReasonAppExport:
+		return "an app export"
+	default:
+		return string(r)
+	}
+}
+
+// AppStopMarker is the persisted "these apps were stopped by an operation that has not reported
+// finishing — they are owed a restart" note.
+type AppStopMarker struct {
+	Active    bool          `json:"active"`
+	OpID      string        `json:"op_id"`
+	Reason    AppStopReason `json:"reason"`
+	Stacks    []string      `json:"stacks"`
+	StartedAt time.Time     `json:"started_at"`
+}
+
+// AppStopStarter is the one thing recovery needs: the ability to start a stack. StartStack must be
+// idempotent (it is — `compose up -d` on a running stack is a no-op).
+type AppStopStarter interface {
+	StartStack(name string) error
+}
+
+// AppStopGuard owns one marker file. Construct with NewAppStopGuard; the zero value is inert (every
+// method is a no-op on a nil guard), so a caller that was never wired degrades to pre-v0.189.0
+// behaviour instead of panicking.
+type AppStopGuard struct {
+	path   string
+	logger *log.Logger
+	now    func() time.Time
+	// starter is only needed by Recover; Begin/End work without one.
+	starter AppStopStarter
+}
+
+// AppStopRecovery is what Recover found and did. Returned rather than pushed through a notifier
+// seam, because of a hard ordering constraint: Recover must COMPLETE before the boot reconciler is
+// launched (§8.4, main.go:236) and the hub notifier is not constructed until main.go:307. A seam
+// wired after the fact would be a seam that never fires — the "built but never wired" shape this
+// project has now hit four times. Returning the outcome lets main.go report it the moment the
+// notifier exists, and makes the reporting decision visible at the call site instead of buried here.
+type AppStopRecovery struct {
+	Reason    AppStopReason
+	OpID      string
+	StartedAt time.Time
+	Restarted []string // apps started again by this recovery
+	Failed    []string // apps that could NOT be restarted (the marker was kept for these)
+}
+
+// Message is the operator-facing headline for an interrupted operation.
+func (r *AppStopRecovery) Message() string {
+	if r == nil {
+		return ""
+	}
+	if len(r.Failed) > 0 {
+		return fmt.Sprintf("%s was interrupted by a controller restart and %d of %d app(s) could NOT be restarted",
+			r.Reason.humanReason(), len(r.Failed), len(r.Restarted)+len(r.Failed))
+	}
+	return fmt.Sprintf("%s was interrupted by a controller restart — %d app(s) were left stopped and have been restarted",
+		r.Reason.humanReason(), len(r.Restarted))
+}
+
+// Detail is the machine-readable tail. App/stack NAMES only — never env values (§9.5).
+func (r *AppStopRecovery) Detail() string {
+	if r == nil {
+		return ""
+	}
+	d := fmt.Sprintf("op=%s reason=%s started_at=%s restarted=%v", r.OpID, r.Reason,
+		r.StartedAt.UTC().Format(time.RFC3339), r.Restarted)
+	if len(r.Failed) > 0 {
+		d += fmt.Sprintf(" restart_failed=%v", r.Failed)
+	}
+	return d
+}
+
+// NewAppStopGuard builds a guard over the given marker path.
+func NewAppStopGuard(path string, logger *log.Logger) *AppStopGuard {
+	if logger == nil {
+		logger = log.Default()
+	}
+	return &AppStopGuard{path: path, logger: logger, now: time.Now}
+}
+
+// SetStarter wires the stack-start seam used by Recover. INIT-ONLY — call once at startup, before
+// Recover. Separate from the constructor because the guard is built alongside the backup manager,
+// which learns its stack provider later (the same shape as SetStackProvider).
+func (g *AppStopGuard) SetStarter(s AppStopStarter) {
+	if g == nil {
+		return
+	}
+	g.starter = s
+}
+
+// Begin records that `stacks` are about to be stopped by `reason`. It MUST be called BEFORE the
+// first stop — an error here means the marker could not be written, and the caller must not proceed
+// to stop an app it cannot promise to restart.
+func (g *AppStopGuard) Begin(opID string, reason AppStopReason, stackNames []string) error {
+	if g == nil || g.path == "" {
+		return nil // not wired — pre-v0.189.0 behaviour, never a hard failure
+	}
+	if len(stackNames) == 0 {
+		return nil
+	}
+	return g.write(AppStopMarker{
+		Active:    true,
+		OpID:      opID,
+		Reason:    reason,
+		Stacks:    append([]string(nil), stackNames...),
+		StartedAt: g.now(),
+	})
+}
+
+// End clears the marker after a successful restart. Best-effort by contract: a failure to clear is
+// logged, never returned as the operation's error — a stale marker costs one idempotent StartStack
+// on the next boot, which is exactly D-b's "worst acceptable outcome" and far cheaper than failing
+// a backup that actually succeeded.
+func (g *AppStopGuard) End() {
+	if g == nil || g.path == "" {
+		return
+	}
+	if err := os.Remove(g.path); err != nil && !os.IsNotExist(err) {
+		g.logger.Printf("[ERROR] [appstop] could not clear the app-stop marker at %s: %v (a stale marker costs one idempotent restart at next startup)", g.path, err)
+	}
+}
+
+// Recover restarts any apps left stopped by an operation that died before restarting them, then
+// clears the marker. Call ONCE at startup, and — critically — call it to COMPLETION before the boot
+// reconciler is launched, so an app this marker explains is not also reported as an unexplained boot
+// orphan (§8.4).
+//
+// Idempotent: StartStack on a running stack is tolerated, and an absent or inactive marker is a
+// no-op. On a restart FAILURE the marker is deliberately LEFT IN PLACE — the next startup retries,
+// and in the meantime the app is down with desired_state:running, so the boot reconciler sees it as
+// an orphan and the dead-app alarm owns it. Clearing a marker whose restart failed would erase the
+// only durable record that an app is owed one.
+//
+// Returns nil when there was nothing to recover — so "no interrupted operation" and "the recovery
+// never ran" are distinguishable to the caller, not only in a log (standing rule 3).
+func (g *AppStopGuard) Recover() *AppStopRecovery {
+	if g == nil || g.path == "" {
+		return nil
+	}
+	m, ok := g.read()
+	if !ok || !m.Active || len(m.Stacks) == 0 {
+		return nil
+	}
+	if g.starter == nil {
+		g.logger.Printf("[ERROR] [appstop] crash recovery: %d app(s) were stopped by %s and are owed a restart, but no stack starter is wired — leaving the marker for the next startup: %v",
+			len(m.Stacks), m.Reason.humanReason(), m.Stacks)
+		return nil
+	}
+
+	g.logger.Printf("[WARN] [appstop] crash recovery: %s (op %q) was interrupted and left %d app(s) stopped — restarting them: %v",
+		m.Reason.humanReason(), m.OpID, len(m.Stacks), m.Stacks)
+
+	res := &AppStopRecovery{Reason: m.Reason, OpID: m.OpID, StartedAt: m.StartedAt}
+	for _, name := range m.Stacks {
+		if err := g.starter.StartStack(name); err != nil {
+			g.logger.Printf("[ERROR] [appstop] crash recovery: restart %s failed: %v", name, err)
+			res.Failed = append(res.Failed, name)
+			continue
+		}
+		g.logger.Printf("[INFO] [appstop] crash recovery: restarted %s after the interrupted %s", name, m.Reason.humanReason())
+		res.Restarted = append(res.Restarted, name)
+	}
+	sort.Strings(res.Failed)
+	sort.Strings(res.Restarted)
+
+	if len(res.Failed) > 0 {
+		g.logger.Printf("[ERROR] [appstop] crash recovery: %d app(s) could not be restarted — KEEPING the marker so the next startup retries; the dead-app alarm owns them meanwhile: %v",
+			len(res.Failed), res.Failed)
+		return res
+	}
+	g.End()
+	return res
+}
+
+// ---- marker persistence (atomic, 0600) — the quiesce shape ------------------------------------
+
+func (g *AppStopGuard) write(m AppStopMarker) error {
+	data, err := json.MarshalIndent(m, "", "  ")
+	if err != nil {
+		return err
+	}
+	if err := os.MkdirAll(filepath.Dir(g.path), 0o755); err != nil {
+		return err
+	}
+	tmp := g.path + ".tmp"
+	f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
+	if err != nil {
+		return err
+	}
+	if _, err := f.Write(data); err != nil {
+		f.Close()
+		os.Remove(tmp)
+		return err
+	}
+	// fsync before rename: the whole point is surviving a power cut, and a rename that lands ahead
+	// of the bytes it points at is a marker that reads as corrupt at exactly the wrong moment.
+	if err := f.Sync(); err != nil {
+		f.Close()
+		os.Remove(tmp)
+		return err
+	}
+	if err := f.Close(); err != nil {
+		os.Remove(tmp)
+		return err
+	}
+	return os.Rename(tmp, g.path)
+}
+
+func (g *AppStopGuard) read() (AppStopMarker, bool) {
+	data, err := os.ReadFile(g.path)
+	if err != nil {
+		return AppStopMarker{}, false
+	}
+	var m AppStopMarker
+	if err := json.Unmarshal(data, &m); err != nil {
+		// Never a silent skip (§9.4): a corrupt marker is LOUD and the bad file is quarantined, so a
+		// genuinely interrupted operation leaves a trace instead of vanishing. Still returns false —
+		// "no usable marker ⇒ no recovery" is the correct contract, and matches quiesce's.
+		g.logger.Printf("[WARN] [appstop] the app-stop marker at %s is corrupt (%v) — quarantining; apps are NOT auto-restarted from it", g.path, err)
+		_ = os.Rename(g.path, fmt.Sprintf("%s.corrupt-%d", g.path, g.now().Unix()))
+		return AppStopMarker{}, false
+	}
+	return m, true
+}

controller/internal/backup/appstop_marker_test.go

@@ -0,0 +1,395 @@
+package backup
+
+import (
+	"encoding/json"
+	"errors"
+	"io"
+	"log"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+// R-166 part 2 — the app-stop crash marker.
+//
+// THE DISCIPLINE THAT MATTERS HERE (§10): a `defer` is not crash-safety, so a test that lets the
+// deferred cleanup run proves nothing about a crash. Every "interrupted" test below simulates a
+// SIGKILL by never reaching the restart — the marker is written, the process conceptually dies, and
+// a FRESH guard over the SAME file does the recovering. That is exactly what Campaign 8 fault 10
+// established on live hardware: a SIGKILL runs no deferred function, and what brought the stacks
+// back was the marker read at startup.
+
+type fakeStarter struct {
+	starts   []string
+	failWith map[string]error
+}
+
+func (f *fakeStarter) StartStack(name string) error {
+	f.starts = append(f.starts, name)
+	if err := f.failWith[name]; err != nil {
+		return err
+	}
+	return nil
+}
+
+func newGuard(t *testing.T, dir string) (*AppStopGuard, *fakeStarter) {
+	t.Helper()
+	s := &fakeStarter{}
+	g := NewAppStopGuard(filepath.Join(dir, "appstop-state.json"), log.New(io.Discard, "", 0))
+	g.SetStarter(s)
+	return g, s
+}
+
+func markerPath(dir string) string { return filepath.Join(dir, "appstop-state.json") }
+
+func markerExists(t *testing.T, dir string) bool {
+	t.Helper()
+	_, err := os.Stat(markerPath(dir))
+	if err != nil && !os.IsNotExist(err) {
+		t.Fatal(err)
+	}
+	return err == nil
+}
+
+// --- Scenario E — a crash mid-backup brings the app back -----------------------------------------
+
+func TestRecover_InterruptedVolumeDump_RestartsTheAppAndClearsTheMarker(t *testing.T) {
+	dir := t.TempDir()
+
+	// --- process 1: an operation stops the app and is KILLED. No End(), no defer, no cleanup. ---
+	g1, _ := newGuard(t, dir)
+	if err := g1.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich"}); err != nil {
+		t.Fatalf("Begin: %v", err)
+	}
+	if !markerExists(t, dir) {
+		t.Fatal("Begin did not write a marker — nothing would survive the kill")
+	}
+	// <SIGKILL here> — g1 is abandoned deliberately; nothing else is called on it.
+
+	// --- process 2: a fresh controller starts and recovers from the file alone. ---
+	g2, starter := newGuard(t, dir)
+	res := g2.Recover()
+
+	if len(starter.starts) != 1 || starter.starts[0] != "immich" {
+		t.Fatalf("started %v, want exactly [immich] — the app was left stranded by the interrupted backup", starter.starts)
+	}
+	if res == nil || len(res.Restarted) != 1 || res.Restarted[0] != "immich" {
+		t.Fatalf("recovery result = %+v, want immich restarted", res)
+	}
+	if res.Reason != ReasonVolumeDump {
+		t.Fatalf("reason = %q, want %q — the operator must be told WHICH operation was interrupted", res.Reason, ReasonVolumeDump)
+	}
+	if markerExists(t, dir) {
+		t.Fatal("the marker survived a successful recovery — the next boot would restart the app again")
+	}
+	// The operator-facing text must name the interruption, not merely report a restart.
+	if msg := res.Message(); msg == "" || !strings.Contains(msg, "interrupted") {
+		t.Fatalf("operator message %q does not say the operation was interrupted", msg)
+	}
+}
+
+func TestRecover_NoMarker_IsASilentNoOp(t *testing.T) {
+	dir := t.TempDir()
+	g, starter := newGuard(t, dir)
+	if res := g.Recover(); res != nil {
+		t.Fatalf("Recover reported %+v on a box with no marker", res)
+	}
+	if len(starter.starts) != 0 {
+		t.Fatalf("started %v with no marker present", starter.starts)
+	}
+}
+
+func TestRecover_FailedRestart_KEEPSTheMarkerForTheNextStartup(t *testing.T) {
+	// The single most important failure behaviour: clearing a marker whose restart failed would
+	// erase the only durable record that an app is owed one. The app is genuinely still down.
+	dir := t.TempDir()
+	g1, _ := newGuard(t, dir)
+	if err := g1.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich", "nextcloud"}); err != nil {
+		t.Fatal(err)
+	}
+
+	g2, starter := newGuard(t, dir)
+	starter.failWith = map[string]error{"immich": errors.New("compose up: no such image")}
+	res := g2.Recover()
+
+	if len(res.Failed) != 1 || res.Failed[0] != "immich" {
+		t.Fatalf("failed=%v, want [immich]", res.Failed)
+	}
+	if len(res.Restarted) != 1 || res.Restarted[0] != "nextcloud" {
+		t.Fatalf("restarted=%v, want [nextcloud] — one app failing must not abort the others", res.Restarted)
+	}
+	if !markerExists(t, dir) {
+		t.Fatal("the marker was cleared even though a restart FAILED — the next startup would not retry")
+	}
+	if msg := res.Message(); !strings.Contains(msg, "NOT be restarted") {
+		t.Fatalf("operator message %q does not report the failure", msg)
+	}
+	if d := res.Detail(); !strings.Contains(d, "restart_failed") || !strings.Contains(d, "immich") {
+		t.Fatalf("detail %q does not name which app failed", d)
+	}
+}
+
+func TestRecover_IsIdempotentAcrossRepeatedStartups(t *testing.T) {
+	dir := t.TempDir()
+	g1, _ := newGuard(t, dir)
+	if err := g1.Begin("op", ReasonOffboxReconstitute, []string{"immich"}); err != nil {
+		t.Fatal(err)
+	}
+	g2, s2 := newGuard(t, dir)
+	g2.Recover()
+	g3, s3 := newGuard(t, dir)
+	g3.Recover()
+
+	if len(s2.starts) != 1 {
+		t.Fatalf("first recovery started %v", s2.starts)
+	}
+	if len(s3.starts) != 0 {
+		t.Fatalf("a SECOND startup restarted %v again — the marker was not cleared", s3.starts)
+	}
+}
+
+func TestRecover_CorruptMarkerIsQuarantinedNotSilentlySkipped(t *testing.T) {
+	// §9.4: never a silent skip. A corrupt marker cannot be acted on, but it must leave a trace —
+	// otherwise a genuinely interrupted operation vanishes without evidence.
+	dir := t.TempDir()
+	if err := os.WriteFile(markerPath(dir), []byte("{not json"), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	g, starter := newGuard(t, dir)
+	if res := g.Recover(); res != nil {
+		t.Fatalf("a corrupt marker produced a recovery result %+v", res)
+	}
+	if len(starter.starts) != 0 {
+		t.Fatalf("apps were started from a corrupt marker: %v", starter.starts)
+	}
+	if markerExists(t, dir) {
+		t.Fatal("the corrupt marker was left in place — it would be re-read forever")
+	}
+	quarantined, _ := filepath.Glob(markerPath(dir) + ".corrupt-*")
+	if len(quarantined) != 1 {
+		t.Fatalf("the corrupt marker was not quarantined (found %d) — it was silently dropped", len(quarantined))
+	}
+}
+
+func TestRecover_NoStarterWiredKeepsTheMarker(t *testing.T) {
+	// D-b's safety rule: never worse than not having the file. With no starter the guard cannot act,
+	// so it must keep the record for a startup that can, rather than clear it and lose the app.
+	dir := t.TempDir()
+	g1, _ := newGuard(t, dir)
+	if err := g1.Begin("op", ReasonVolumeDump, []string{"immich"}); err != nil {
+		t.Fatal(err)
+	}
+	g2 := NewAppStopGuard(markerPath(dir), log.New(io.Discard, "", 0)) // deliberately no SetStarter
+	if res := g2.Recover(); res != nil {
+		t.Fatalf("recovered without a starter: %+v", res)
+	}
+	if !markerExists(t, dir) {
+		t.Fatal("the marker was cleared with no starter wired — the app would never come back")
+	}
+}
+
+func TestNilGuardIsInert(t *testing.T) {
+	// A caller that was never wired must degrade to pre-v0.189.0 behaviour, not panic.
+	var g *AppStopGuard
+	if err := g.Begin("op", ReasonVolumeDump, []string{"x"}); err != nil {
+		t.Fatalf("nil guard Begin returned %v", err)
+	}
+	g.End()
+	if res := g.Recover(); res != nil {
+		t.Fatalf("nil guard recovered %+v", res)
+	}
+}
+
+func TestMarkerContentsAreDiagnosable(t *testing.T) {
+	dir := t.TempDir()
+	g, _ := newGuard(t, dir)
+	if err := g.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich"}); err != nil {
+		t.Fatal(err)
+	}
+	raw, err := os.ReadFile(markerPath(dir))
+	if err != nil {
+		t.Fatal(err)
+	}
+	var m AppStopMarker
+	if err := json.Unmarshal(raw, &m); err != nil {
+		t.Fatalf("the marker on disk is not readable JSON: %v", err)
+	}
+	if !m.Active || m.OpID != "volume-dump:immich" || m.Reason != ReasonVolumeDump ||
+		len(m.Stacks) != 1 || m.Stacks[0] != "immich" || m.StartedAt.IsZero() {
+		t.Fatalf("the marker does not record enough to diagnose the interruption: %+v", m)
+	}
+	// 0600 — it names customer apps.
+	fi, err := os.Stat(markerPath(dir))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if fi.Mode().Perm() != 0o600 {
+		t.Fatalf("marker mode = %v, want 0600", fi.Mode().Perm())
+	}
+}
+
+func TestBeginWithNoStacksWritesNothing(t *testing.T) {
+	dir := t.TempDir()
+	g, _ := newGuard(t, dir)
+	if err := g.Begin("op", ReasonVolumeDump, nil); err != nil {
+		t.Fatal(err)
+	}
+	if markerExists(t, dir) {
+		t.Fatal("a marker was written for an operation that stops nothing")
+	}
+}
+
+// --- Scenarios E/F — DumpAppVolumesSafe, the primary site ----------------------------------------
+
+// inspectingProvider is the StackDataProvider slice DumpAppVolumesSafe touches. It records whether
+// the marker file EXISTED at each step — the positive observable for the ordering property. An
+// absent log line is not evidence (standing rule 3); the file's presence at the moment of the stop
+// is.
+//
+// GetDockerVolumes returns nothing, so the dump itself is a no-op and no Docker is involved — the
+// stop/start bracket around it is what is under test.
+type inspectingProvider struct {
+	StackDataProvider
+	markerFile          string
+	events              []string
+	stopErr             error
+	startErr            error
+	markerPresentAtStop bool
+	markerAtStartCall   bool
+	// panicOnVolumes simulates a hard abort (SIGKILL/power cut) at the point the dump begins: the
+	// unwind skips the restart statement, exactly as a kill would.
+	panicOnVolumes bool
+}
+
+func (p *inspectingProvider) GetDockerVolumes(string) []string {
+	if p.panicOnVolumes {
+		panic("simulated hard abort mid-dump")
+	}
+	return nil
+}
+
+func (p *inspectingProvider) StopStack(name string) error {
+	_, err := os.Stat(p.markerFile)
+	p.markerPresentAtStop = err == nil
+	p.events = append(p.events, "stop:"+name)
+	return p.stopErr
+}
+
+func (p *inspectingProvider) StartStack(name string) error {
+	_, err := os.Stat(p.markerFile)
+	p.markerAtStartCall = err == nil
+	p.events = append(p.events, "start:"+name)
+	return p.startErr
+}
+
+func newDumpManager(t *testing.T, dir string, p *inspectingProvider) *Manager {
+	t.Helper()
+	lg := log.New(io.Discard, "", 0)
+	m := &Manager{logger: lg, stackProvider: p, systemDataPath: dir}
+	m.appStop = NewAppStopGuard(markerPath(dir), lg)
+	return m
+}
+
+func TestDumpAppVolumesSafe_MarkerCoversTheWholeStopStartWindow(t *testing.T) {
+	// Scenario F, the happy path: the marker is on disk BEFORE the stop, still on disk for the whole
+	// time the app is down, and GONE once the restart succeeds.
+	dir := t.TempDir()
+	p := &inspectingProvider{markerFile: markerPath(dir)}
+	m := newDumpManager(t, dir, p)
+
+	if err := m.DumpAppVolumesSafe("immich"); err != nil {
+		t.Fatalf("DumpAppVolumesSafe: %v", err)
+	}
+
+	if !p.markerPresentAtStop {
+		t.Fatal("the marker was NOT on disk when the app was stopped — a crash one instruction later " +
+			"strands the app, which is the entire failure this marker exists to prevent")
+	}
+	if !p.markerAtStartCall {
+		t.Fatal("the marker was already gone while the app was still down")
+	}
+	if markerExists(t, dir) {
+		t.Fatal("the marker survived a dump whose restart succeeded — the next boot would restart the app again")
+	}
+	if len(p.events) != 2 || p.events[0] != "stop:immich" || p.events[1] != "start:immich" {
+		t.Fatalf("events=%v, want [stop:immich start:immich]", p.events)
+	}
+}
+
+func TestDumpAppVolumesSafe_Interrupted_RecoveryBringsTheAppBack(t *testing.T) {
+	// Scenario E end-to-end THROUGH THE PRODUCTION PATH, and WITHOUT running any cleanup.
+	//
+	// The abort is real: GetDockerVolumes panics, which unwinds out of DumpAppVolumesSafe AFTER the
+	// marker was written and the app stopped, and BEFORE the restart statement — and because that
+	// restart is a plain statement, not a defer, it never runs. That is the shape of a hard kill.
+	//
+	// The earlier version of this test called m.appStop.Begin itself, which meant it proved the
+	// marker type worked and NOT that DumpAppVolumesSafe uses it — it survived the red-proof that
+	// deleted the production Begin call. Driving the real function is what makes the proof bite.
+	//
+	// RED-PROOF: delete the `m.appStop.Begin(...)` call from DumpAppVolumesSafe and this test fails —
+	// nothing is written, so nothing is recovered. Demonstrated in REPORT.md §5.
+	dir := t.TempDir()
+	p := &inspectingProvider{markerFile: markerPath(dir), panicOnVolumes: true}
+	m := newDumpManager(t, dir, p)
+
+	func() {
+		defer func() {
+			if recover() == nil {
+				t.Error("the simulated abort did not fire — this test proves nothing")
+			}
+		}()
+		_ = m.DumpAppVolumesSafe("immich")
+	}()
+
+	if !p.markerPresentAtStop {
+		t.Fatal("the app was stopped before any marker existed")
+	}
+	if p.markerAtStartCall {
+		t.Fatal("the restart ran despite the abort — the simulation is wrong, not the code")
+	}
+
+	// <the controller is gone> — a fresh one starts and recovers from the file alone.
+	g, starter := newGuard(t, dir)
+	res := g.Recover()
+
+	if len(starter.starts) != 1 || starter.starts[0] != "immich" {
+		t.Fatalf("started %v — the app stopped by the interrupted dump was not brought back", starter.starts)
+	}
+	if res == nil || res.Reason != ReasonVolumeDump {
+		t.Fatalf("recovery did not name the volume dump as the interrupted operation: %+v", res)
+	}
+	if markerExists(t, dir) {
+		t.Fatal("the marker was not cleared after a successful recovery")
+	}
+}
+
+func TestDumpAppVolumesSafe_FailedRestartKeepsTheMarker(t *testing.T) {
+	dir := t.TempDir()
+	p := &inspectingProvider{markerFile: markerPath(dir), startErr: errors.New("compose up failed")}
+	m := newDumpManager(t, dir, p)
+
+	if err := m.DumpAppVolumesSafe("immich"); err == nil {
+		t.Fatal("a failed restart must surface as an error")
+	}
+	if !markerExists(t, dir) {
+		t.Fatal("the marker was cleared even though the restart FAILED — the app is still down and " +
+			"nothing records that it is owed a restart")
+	}
+}
+
+func TestDumpAppVolumesSafe_FailedStopClearsTheMarker(t *testing.T) {
+	// Nothing was stopped, so nothing is owed a restart. A stranded marker here would cost a
+	// spurious restart at the next startup AND a false "a backup was interrupted" alert.
+	dir := t.TempDir()
+	p := &inspectingProvider{markerFile: markerPath(dir), stopErr: errors.New("stack is protected")}
+	m := newDumpManager(t, dir, p)
+
+	if err := m.DumpAppVolumesSafe("traefik"); err == nil {
+		t.Fatal("a failed stop must surface as an error")
+	}
+	if markerExists(t, dir) {
+		t.Fatal("a marker was left behind for an app that was never stopped")
+	}
+}

controller/internal/backup/backup.go

@@ -32,6 +32,12 @@ type Manager struct {
 	// tier2Notify, if set, is called after each Tier 2 copy (success: err==nil) for notifications.
 	tier2Notify func(stackName, destLabel string, dur time.Duration, err error)
 
+	// appStop (R-166) is the crash marker for operations that stop an app, work on its data, and
+	// start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power
+	// cut in that window leaves a durable record that Recover honours at the next startup. Built in
+	// NewManager from cfg.Paths.DataDir — see appstop_marker.go for why it is not quiesce's file.
+	appStop *AppStopGuard
+
 	// offbox (Part B): the restic-SFTP exec seam (nil → real restic) + the failure→operator-alert hook.
 	offboxRunner offboxRunner
 	offboxNotify func(dur time.Duration, snapshots int, err error)
@@ -216,10 +222,30 @@ func NewManager(cfg *config.Config, sett *settings.Settings, logger *log.Logger)
 		settings:       sett,
 		systemDataPath: cfg.Paths.SystemDataPath,
 	}
+	// R-166: its OWN file next to quiesce-state.json, never inside it — one file, one writer.
+	m.appStop = NewAppStopGuard(filepath.Join(cfg.Paths.DataDir, "appstop-state.json"), logger)
 	m.reconcileCrashedRun()
 	return m
 }
 
+// AppStopGuard exposes the app-stop crash marker so the exporter (a different package with the same
+// stop-work-start shape) can share the one marker file rather than opening a second one.
+func (m *Manager) AppStopGuard() *AppStopGuard { return m.appStop }
+
+// SetAppStopGuard injects the guard instead of using the one NewManager built. INIT-ONLY — call once
+// during single-threaded startup, before any backup runs.
+//
+// It exists because of a startup ORDERING constraint, not for testing: the guard's Recover must
+// complete before the boot reconciler is launched (main.go:~236) and this manager is not constructed
+// until ~line 272. So main.go builds the guard early, recovers, and hands the SAME object here —
+// rather than a second guard over the same file, which would be one file with two owners, the exact
+// shape this marker was kept out of quiesce's file to avoid.
+func (m *Manager) SetAppStopGuard(g *AppStopGuard) {
+	if g != nil {
+		m.appStop = g
+	}
+}
+
 // reconcileCrashedRun makes the persisted offbox status truthful after a crash (campaign C1): a controller
 // that died mid-run left LastStatus="running" on disk (the in-memory single-flight mutex is gone with the
 // process, but the persisted status keeps lying "running" forever). Flip it to error with a Hungarian
@@ -679,13 +705,28 @@ func atomicPromoteTar(tmpPath, finalPath string) error {
 // DumpAppVolumesSafe stops the stack before dumping volumes and restarts after.
 // Prevents inconsistent tars of live database volumes (e.g. PostgreSQL).
 // Protected stacks that reject StopStack will return an error — callers handle as warning.
+//
+// R-166: the stop→dump→start window is marked. Before this, a controller killed between the stop
+// and the start left the app down with NOTHING on disk saying why or that it was owed a restart —
+// and a stopped app has zero containers, which the boot reconciler then read as a deliberate
+// customer stop and left alone. The marker is the mechanism, not the restart call below: a SIGKILL
+// runs no deferred function (Campaign 8 fault 10, on live hardware), so only something already
+// written to disk can survive it.
 func (m *Manager) DumpAppVolumesSafe(stackName string) error {
 	if m.stackProvider == nil {
 		return fmt.Errorf("no stack provider")
 	}
 
+	// Intent before the act: refuse to stop an app we cannot promise to restart.
+	if err := m.appStop.Begin("volume-dump:"+stackName, ReasonVolumeDump, []string{stackName}); err != nil {
+		return fmt.Errorf("could not record the app-stop marker for %s (refusing to stop it unprotected): %w", stackName, err)
+	}
+
 	m.logger.Printf("[INFO] [backup] Stopping %s for safe volume dump", stackName)
 	if err := m.stackProvider.StopStack(stackName); err != nil {
+		// Nothing was stopped, so nothing is owed a restart — clear rather than strand a marker that
+		// would cost a spurious (if harmless) restart at the next startup.
+		m.appStop.End()
 		return fmt.Errorf("could not stop %s for volume dump: %w", stackName, err)
 	}
 
@@ -695,6 +736,10 @@ func (m *Manager) DumpAppVolumesSafe(stackName string) error {
 	startErr := m.stackProvider.StartStack(stackName)
 	if startErr != nil {
 		m.logger.Printf("[ERROR] [backup] Failed to restart %s after volume dump: %v", stackName, startErr)
+	} else {
+		// Cleared ONLY on a restart that succeeded. A failed restart keeps the marker so the next
+		// startup retries — the app really is still owed one.
+		m.appStop.End()
 	}
 
 	// Surface both errors — callers must know if the app is left stopped

controller/internal/backup/offbox_reconstitute.go

@@ -276,6 +276,22 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (Of
 	}
 
 	// --- FILES ----------------------------------------------------------------------------------
+	// R-166: mark the stop→restore→start window BEFORE stopping. A controller killed anywhere inside
+	// it used to leave the app down with nothing on disk recording that it was owed a restart — and a
+	// full offsite restore is a LONG window, so this is the shape most likely to be interrupted.
+	if err := m.appStop.Begin("offbox-reconstitute:"+stack, ReasonOffboxReconstitute, []string{stack}); err != nil {
+		return res, fmt.Errorf("a(z) %s leállítása előtti jelölő nem menthető: %w", stack, err)
+	}
+	// restartStack starts the app and clears the marker ONLY when the start actually succeeded — a
+	// failed start leaves the marker so the next startup retries. Every bring-up below goes through
+	// it; a bare StartStack here would clear nothing and strand the marker on the success path.
+	restartStack := func() error {
+		err := m.stackProvider.StartStack(stack)
+		if err == nil {
+			m.appStop.End()
+		}
+		return err
+	}
 	if err := m.stackProvider.StopStack(stack); err != nil {
 		m.logger.Printf("[WARN] [offbox] could not stop %s before reconstitution: %v (continuing)", stack, err)
 	}
@@ -291,7 +307,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (Of
 		if cErr != nil {
 			// Best-effort bring-up: leaving the app stopped after a partial copy would turn a failed
 			// restore into an outage.
-			if sErr := m.stackProvider.StartStack(stack); sErr != nil {
+			if sErr := restartStack(); sErr != nil {
 				m.logger.Printf("[WARN] [offbox] %s: restart after failed placement also failed: %v", stack, sErr)
 			}
 			return res, fmt.Errorf("a(z) %s fájljainak visszaállítása sikertelen: %w", stack, cErr)
@@ -308,7 +324,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (Of
 	if hasDB {
 		if err := m.stackProvider.StartStackServices(stack, dbServices); err != nil {
 			// Best-effort bring-up: a failed restore must not also be an outage.
-			if sErr := m.stackProvider.StartStack(stack); sErr != nil {
+			if sErr := restartStack(); sErr != nil {
 				m.logger.Printf("[WARN] [offbox] %s: full start after failed DB-only start also failed: %v", stack, sErr)
 			}
 			return res, fmt.Errorf("a(z) %s adatbázis-szolgáltatásának indítása sikertelen: %w", stack, err)
@@ -316,13 +332,13 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (Of
 		n, iErr := m.reimportDBDumpsFrom(ctx, stack, scratchDumpDir)
 		res.DBsReplayed = n
 		if iErr != nil {
-			if sErr := m.stackProvider.StartStack(stack); sErr != nil {
+			if sErr := restartStack(); sErr != nil {
 				m.logger.Printf("[WARN] [offbox] %s: full start after failed replay also failed: %v", stack, sErr)
 			}
 			return res, fmt.Errorf("az adatbázis visszaállítása sikertelen: %w — a korábbi állapot mentése megvan: %s", iErr, filepath.Base(safety))
 		}
 	}
-	if err := m.stackProvider.StartStack(stack); err != nil {
+	if err := restartStack(); err != nil {
 		return res, fmt.Errorf("a(z) %s újraindítása sikertelen a fájlok visszaállítása után: %w", stack, err)
 	}
 	if err := m.waitForHealthy(stack, 90*time.Second); err != nil {