C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (v0.183.0)

Both are the system reporting healthy while the customer is not, and both live in the same
status-derivation code. Neither is fixed by making the system quieter.

C9-F1 (HIGH) — Tier-2 writes recovery-unit/ on EVERY run and RestoreTier2Files has never read
it (tier2_restore.go:101-104 reads hdd/ + userdata/ only). Phase 0 enumerated all 53 catalog
templates against both demo boxes: 43 apps have NO readable subtree, so the button stopped the
app, restored 0 files, restarted it and said "Nincs hiányzó fájl — minden fájl megvan a helyén."
— at the moment the customer pressed it because files were missing, with 156 MB of BookStack's
data unread in the same copy. 9 apps have file legs but never their DB or volumes, so the same
sentence was also a clean bill of health over data never opened (immich: 1.3 GB Postgres unit).

Honesty half shipped: a pre-flight coverage check refuses UP FRONT without stopping the app and
NAMES the action that works; a run that proceeds claims only what it EXAMINED and discloses that
the database and volumes are not covered. Completeness is filed as C9-F1b — routing to the
Tier-1 unit restore puts a destructive operation behind a non-destructive button, so its confirm
copy has to carry that difference. C9-F4 filed: nothing reads the Tier-2 recovery-unit/ mirror,
so the second local copy that exists for drive loss is unreachable by any customer action.

C9-F2 (HIGH) — a crash loop was counted as working. StateRestarting is deliberately NOT added to
IsDownState (that alarms on every deploy fleet-wide, the over-correction F-A1 nearly cost us); a
sustained run becomes down after crashLoopAfter = 5m, set above the 120s deploy timeout, Mealie's
60s start_period and R-97b's 180s grace. The dashboard counter uses the same predicate, so it no
longer contradicts the alarm on the same screen. README's claim that faults "still surface as
restarting" was a wish with no test — corrected in place; it is the seventh such instance.

Six red-proofs observed, including the one that matters most: adding StateRestarting to
IsDownState fails the brief-restart test with "every deploy and update would page the operator".
go test ./... rc=0, 27 packages, run and read separately from this commit.
This commit is contained in:
2026-07-28 18:53:56 +02:00
parent d8b3279731
commit fd50a73e65
12 changed files with 805 additions and 38 deletions
+12 -3
View File
@@ -1242,7 +1242,7 @@ func runBootReconcile(ctx context.Context, mgr bootrecon.StackProvider, logger *
func scanDeployedAppRunStates(mgr *stacks.Manager, q *quiesce.Loop) ([]web.DeadApp, []notify.AppRunState) {
// R-97b: a stack THIS controller stopped for a backup is not a fault. q may be nil (unprovisioned
// guest) — SuppressedStacks is nil-safe and returns nothing, i.e. suppress nothing.
return classifyRunStates(mgr.GetStacks(), q.SuppressedStacks(), q.FailedRestarts())
return classifyRunStates(mgr.GetStacks(), q.SuppressedStacks(), q.FailedRestarts(), time.Now())
}
// classifyRunStates is the pure fix-3 derivation over a plain stack slice. It splits the deployed
@@ -1274,7 +1274,9 @@ func scanDeployedAppRunStates(mgr *stacks.Manager, q *quiesce.Loop) ([]web.DeadA
// which is correct: out-of-band tampering IS reportable.) IsDownState is intentionally left unchanged
// — other callers rely on stopped counting as down; the suppression is a filter at this single
// derivation point only.
func classifyRunStates(sts []stacks.Stack, quiesced map[string]bool, failedRestart map[string]bool) ([]web.DeadApp, []notify.AppRunState) {
// `now` is injected (C9-F2) so the crash-loop threshold is a unit-testable contract rather than a
// property of the wall clock.
func classifyRunStates(sts []stacks.Stack, quiesced map[string]bool, failedRestart map[string]bool, now time.Time) ([]web.DeadApp, []notify.AppRunState) {
var dead []web.DeadApp
var states []notify.AppRunState
for _, st := range sts {
@@ -1289,8 +1291,15 @@ func classifyRunStates(sts []stacks.Stack, quiesced map[string]bool, failedResta
// F-CRIT-1: StateStopped is whitelisted as a deliberate user stop UNLESS the quiesce loop
// reports that it stopped this stack and could not restart it. That single term is what turns
// an indefinitely-silent dead app back into an alarm, without re-alarming genuine user stops.
// C9-F2: a SUSTAINED restarting is a crash loop, and a crash loop is a dead app. Deliberately
// NOT folded into IsDownState — that would alarm on every deploy and update fleet-wide, which
// is the over-correction F-A1 nearly cost us. The threshold (stacks.crashLoopAfter, 5 min) sits
// above the deploy health timeout, the slowest catalog start_period AND R-97b's grace, so a
// brief restart never reaches it and the two suppression windows compose into one bounded
// delay. Quiesce suppression below still wins inside its own window.
crashLooping := st.CrashLooping(now)
userStopped := st.State == stacks.StateStopped && !failedRestart[st.Name]
down := stacks.IsDownState(st.State) && !userStopped && !quiesced[st.Name]
down := (stacks.IsDownState(st.State) || crashLooping) && !userStopped && !quiesced[st.Name]
states = append(states, notify.AppRunState{Name: st.Name, DisplayName: st.Meta.DisplayName, Down: down})
if down {
dead = append(dead, web.DeadApp{Name: st.Name, DisplayName: st.Meta.DisplayName, State: string(st.State)})