C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (v0.183.0)

Both are the system reporting healthy while the customer is not, and both live in the same
status-derivation code. Neither is fixed by making the system quieter.

C9-F1 (HIGH) — Tier-2 writes recovery-unit/ on EVERY run and RestoreTier2Files has never read
it (tier2_restore.go:101-104 reads hdd/ + userdata/ only). Phase 0 enumerated all 53 catalog
templates against both demo boxes: 43 apps have NO readable subtree, so the button stopped the
app, restored 0 files, restarted it and said "Nincs hiányzó fájl — minden fájl megvan a helyén."
— at the moment the customer pressed it because files were missing, with 156 MB of BookStack's
data unread in the same copy. 9 apps have file legs but never their DB or volumes, so the same
sentence was also a clean bill of health over data never opened (immich: 1.3 GB Postgres unit).

Honesty half shipped: a pre-flight coverage check refuses UP FRONT without stopping the app and
NAMES the action that works; a run that proceeds claims only what it EXAMINED and discloses that
the database and volumes are not covered. Completeness is filed as C9-F1b — routing to the
Tier-1 unit restore puts a destructive operation behind a non-destructive button, so its confirm
copy has to carry that difference. C9-F4 filed: nothing reads the Tier-2 recovery-unit/ mirror,
so the second local copy that exists for drive loss is unreachable by any customer action.

C9-F2 (HIGH) — a crash loop was counted as working. StateRestarting is deliberately NOT added to
IsDownState (that alarms on every deploy fleet-wide, the over-correction F-A1 nearly cost us); a
sustained run becomes down after crashLoopAfter = 5m, set above the 120s deploy timeout, Mealie's
60s start_period and R-97b's 180s grace. The dashboard counter uses the same predicate, so it no
longer contradicts the alarm on the same screen. README's claim that faults "still surface as
restarting" was a wish with no test — corrected in place; it is the seventh such instance.

Six red-proofs observed, including the one that matters most: adding StateRestarting to
IsDownState fails the brief-restart test with "every deploy and update would page the operator".
go test ./... rc=0, 27 packages, run and read separately from this commit.
This commit is contained in:
2026-07-28 18:53:56 +02:00
parent d8b3279731
commit fd50a73e65
12 changed files with 805 additions and 38 deletions
@@ -1,6 +1,8 @@
package main
import (
"time"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
@@ -56,7 +58,7 @@ func TestClassifyRunStates_StoppedIsSuppressed(t *testing.T) {
stack("nextcloud", stacks.StateDegraded, true, false),
}
dead, states := classifyRunStates(sts, nil, nil)
dead, states := classifyRunStates(sts, nil, nil, time.Now())
gotDead := deadNames(dead)
if len(gotDead) != 2 || !gotDead["immich"] || !gotDead["nextcloud"] {
@@ -90,7 +92,7 @@ func TestClassifyRunStates_FaultParity(t *testing.T) {
stack("nextcloud", stacks.StateDegraded, true, false),
}
dead, states := classifyRunStates(sts, nil, nil)
dead, states := classifyRunStates(sts, nil, nil, time.Now())
gotDead := deadNames(dead)
if len(gotDead) != 2 || !gotDead["immich"] || !gotDead["nextcloud"] {
@@ -116,7 +118,7 @@ func TestClassifyRunStates_SkipsDeployingAndUndeployed(t *testing.T) {
stack("mid", stacks.StateDeploying, true, true), // mid-deploy → skipped
stack("gone", stacks.StateExited, false, false), // not deployed → skipped
}
dead, states := classifyRunStates(sts, nil, nil)
dead, states := classifyRunStates(sts, nil, nil, time.Now())
if len(dead) != 0 || len(states) != 0 {
t.Fatalf("deploying and undeployed stacks must be skipped, got dead=%+v states=%+v", dead, states)
}