v0.189.0 — desired state + the app-stop crash marker (R-166 / D-b)
gates / gates (push) Successful in 8s

The box stops inferring the customer's intent from a container count and reads
what they actually asked for.

Part 1 — desired state. AppConfig gains a tri-state `desired_state`
(""/running/stopped), written ONLY by the customer's own action: the API action
switch, DeployStack, UpdateOptionalConfig's redeploy branch, and the .fab
import. Intent is written BEFORE the act and a failed write REFUSES the act.
StartStack/StopStack are deliberately not writers — 14 callers, only 2 are the
customer. bootrecon.isBootOrphan now reads intent instead of len(Containers)>0,
which closes R-157 mechanism B (a power cut or interrupted deploy left an app
with zero containers, read as a deliberate stop, and stranded silently).

ABSENT MEANS UNKNOWN, NEVER "running": every pre-v0.189.0 app.yaml reads absent,
so the legacy fallback is byte-identical to the old rule. A running-only startup
backfill converges the unambiguous cases; `stopped` is never inferred.

Part 2 — backup.AppStopGuard, a persisted marker over every stop→work→start
window (volume dump, offbox reconstitute, .fab export). Its own file, never
quiesce's. Written before the stop, cleared only after a restart that succeeded,
kept when one fails. Recover() completes before the boot reconciler is launched
and returns its outcome, which main.go reports on the existing backup_failed
event once the notifier exists. A defer is not the mechanism — a SIGKILL runs
none (Campaign 8 fault 10).

Also: SaveAppConfig rebuilt AppConfig field-by-field (the R-100 shape) and would
have dropped desired_state on every save across nine call sites. Replaced with
copy-and-overlay. Measured: app.yaml does not round-trip unknown YAML keys.

No hub change, no agent coupling, no user-visible string. 27/27 packages green;
7 red-proofs observed FAIL then restored.
This commit is contained in:
2026-08-02 18:40:17 +02:00
parent e7c44c0e0f
commit dbcb306fcf
17 changed files with 2211 additions and 33 deletions
+45
View File
@@ -32,6 +32,12 @@ type Manager struct {
// tier2Notify, if set, is called after each Tier 2 copy (success: err==nil) for notifications.
tier2Notify func(stackName, destLabel string, dur time.Duration, err error)
// appStop (R-166) is the crash marker for operations that stop an app, work on its data, and
// start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power
// cut in that window leaves a durable record that Recover honours at the next startup. Built in
// NewManager from cfg.Paths.DataDir — see appstop_marker.go for why it is not quiesce's file.
appStop *AppStopGuard
// offbox (Part B): the restic-SFTP exec seam (nil → real restic) + the failure→operator-alert hook.
offboxRunner offboxRunner
offboxNotify func(dur time.Duration, snapshots int, err error)
@@ -216,10 +222,30 @@ func NewManager(cfg *config.Config, sett *settings.Settings, logger *log.Logger)
settings: sett,
systemDataPath: cfg.Paths.SystemDataPath,
}
// R-166: its OWN file next to quiesce-state.json, never inside it — one file, one writer.
m.appStop = NewAppStopGuard(filepath.Join(cfg.Paths.DataDir, "appstop-state.json"), logger)
m.reconcileCrashedRun()
return m
}
// AppStopGuard exposes the app-stop crash marker so the exporter (a different package with the same
// stop-work-start shape) can share the one marker file rather than opening a second one.
func (m *Manager) AppStopGuard() *AppStopGuard { return m.appStop }
// SetAppStopGuard injects the guard instead of using the one NewManager built. INIT-ONLY — call once
// during single-threaded startup, before any backup runs.
//
// It exists because of a startup ORDERING constraint, not for testing: the guard's Recover must
// complete before the boot reconciler is launched (main.go:~236) and this manager is not constructed
// until ~line 272. So main.go builds the guard early, recovers, and hands the SAME object here —
// rather than a second guard over the same file, which would be one file with two owners, the exact
// shape this marker was kept out of quiesce's file to avoid.
func (m *Manager) SetAppStopGuard(g *AppStopGuard) {
if g != nil {
m.appStop = g
}
}
// reconcileCrashedRun makes the persisted offbox status truthful after a crash (campaign C1): a controller
// that died mid-run left LastStatus="running" on disk (the in-memory single-flight mutex is gone with the
// process, but the persisted status keeps lying "running" forever). Flip it to error with a Hungarian
@@ -679,13 +705,28 @@ func atomicPromoteTar(tmpPath, finalPath string) error {
// DumpAppVolumesSafe stops the stack before dumping volumes and restarts after.
// Prevents inconsistent tars of live database volumes (e.g. PostgreSQL).
// Protected stacks that reject StopStack will return an error — callers handle as warning.
//
// R-166: the stop→dump→start window is marked. Before this, a controller killed between the stop
// and the start left the app down with NOTHING on disk saying why or that it was owed a restart —
// and a stopped app has zero containers, which the boot reconciler then read as a deliberate
// customer stop and left alone. The marker is the mechanism, not the restart call below: a SIGKILL
// runs no deferred function (Campaign 8 fault 10, on live hardware), so only something already
// written to disk can survive it.
func (m *Manager) DumpAppVolumesSafe(stackName string) error {
if m.stackProvider == nil {
return fmt.Errorf("no stack provider")
}
// Intent before the act: refuse to stop an app we cannot promise to restart.
if err := m.appStop.Begin("volume-dump:"+stackName, ReasonVolumeDump, []string{stackName}); err != nil {
return fmt.Errorf("could not record the app-stop marker for %s (refusing to stop it unprotected): %w", stackName, err)
}
m.logger.Printf("[INFO] [backup] Stopping %s for safe volume dump", stackName)
if err := m.stackProvider.StopStack(stackName); err != nil {
// Nothing was stopped, so nothing is owed a restart — clear rather than strand a marker that
// would cost a spurious (if harmless) restart at the next startup.
m.appStop.End()
return fmt.Errorf("could not stop %s for volume dump: %w", stackName, err)
}
@@ -695,6 +736,10 @@ func (m *Manager) DumpAppVolumesSafe(stackName string) error {
startErr := m.stackProvider.StartStack(stackName)
if startErr != nil {
m.logger.Printf("[ERROR] [backup] Failed to restart %s after volume dump: %v", stackName, startErr)
} else {
// Cleared ONLY on a restart that succeeded. A failed restart keeps the marker so the next
// startup retries — the app really is still owed one.
m.appStop.End()
}
// Surface both errors — callers must know if the app is left stopped