C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (v0.183.0)

Both are the system reporting healthy while the customer is not, and both live in the same
status-derivation code. Neither is fixed by making the system quieter.

C9-F1 (HIGH) — Tier-2 writes recovery-unit/ on EVERY run and RestoreTier2Files has never read
it (tier2_restore.go:101-104 reads hdd/ + userdata/ only). Phase 0 enumerated all 53 catalog
templates against both demo boxes: 43 apps have NO readable subtree, so the button stopped the
app, restored 0 files, restarted it and said "Nincs hiányzó fájl — minden fájl megvan a helyén."
— at the moment the customer pressed it because files were missing, with 156 MB of BookStack's
data unread in the same copy. 9 apps have file legs but never their DB or volumes, so the same
sentence was also a clean bill of health over data never opened (immich: 1.3 GB Postgres unit).

Honesty half shipped: a pre-flight coverage check refuses UP FRONT without stopping the app and
NAMES the action that works; a run that proceeds claims only what it EXAMINED and discloses that
the database and volumes are not covered. Completeness is filed as C9-F1b — routing to the
Tier-1 unit restore puts a destructive operation behind a non-destructive button, so its confirm
copy has to carry that difference. C9-F4 filed: nothing reads the Tier-2 recovery-unit/ mirror,
so the second local copy that exists for drive loss is unreachable by any customer action.

C9-F2 (HIGH) — a crash loop was counted as working. StateRestarting is deliberately NOT added to
IsDownState (that alarms on every deploy fleet-wide, the over-correction F-A1 nearly cost us); a
sustained run becomes down after crashLoopAfter = 5m, set above the 120s deploy timeout, Mealie's
60s start_period and R-97b's 180s grace. The dashboard counter uses the same predicate, so it no
longer contradicts the alarm on the same screen. README's claim that faults "still surface as
restarting" was a wish with no test — corrected in place; it is the seventh such instance.

Six red-proofs observed, including the one that matters most: adding StateRestarting to
IsDownState fails the brief-restart test with "every deploy and update would page the operator".
go test ./... rc=0, 27 packages, run and read separately from this commit.
This commit is contained in:
2026-07-28 18:53:56 +02:00
parent d8b3279731
commit fd50a73e65
12 changed files with 805 additions and 38 deletions
+59 -2
View File
@@ -3,6 +3,7 @@ package web
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"net/http"
@@ -145,7 +146,17 @@ func (s *Server) dashboardHandler(w http.ResponseWriter, r *http.Request) {
// Count from the DISPLAYED set only
running, stopped := 0, 0
countNow := time.Now()
for _, st := range deployedStacks {
// C9-F2: a stack that has been `restarting` past the crash-loop threshold counts with STOPPED,
// for the same reason R-51 moved `degraded` there — this counter answers "how many of my apps
// work", and an app Docker has been restarting for five minutes does not. A BRIEF restart
// still counts as running (deploys and updates pass through it), so the counter and the
// dead-app alarm now agree instead of contradicting each other on the same screen.
if st.CrashLooping(countNow) {
stopped++
continue
}
switch st.State {
case stacks.StateRunning, stacks.StateStarting, stacks.StateUnhealthy, stacks.StateRestarting:
running++
@@ -1277,6 +1288,22 @@ func (s *Server) backupRestoreHandler(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/backups/restore?flash="+url.QueryEscape("Visszaállítás elindult — az állapot itt frissül."), http.StatusFound)
}
// C9-F1 customer-facing strings. Kept as named constants, not inlined, because both are asserted
// verbatim by tests — a silent edit to either is the way an honest message drifts back into a
// comforting one.
const (
// tier2NoCoverageMsg is shown when this app's data cannot come from the secondary copy at all.
// It NAMES the action that works rather than leaving a dead end: the keep-side recovery-unit
// restore on /backups/restore, which does restore named volumes and DB dumps (proven live,
// Campaign 9 A2). It also states plainly that no outage was taken, because the previous behaviour
// took one.
tier2NoCoverageMsg = "Ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza — az alkalmazás nem állt le. Használd a Visszaállítás indítása gombot a Biztonsági mentés → Visszaállítás oldalon."
// tier2UnitNotCoveredMsg is appended wherever the restore DID run, so a clean result never reads
// as a clean bill of health for data the operation never opened.
tier2UnitNotCoveredMsg = "Az alkalmazás adatbázisa és belső kötetei nem tartoznak ebbe a visszaállításba."
)
// backupTier2RestoreHandler (C2, closes F2) restores an app's MISSING user files in place from its
// recorded Tier-2 copy — additive-only: existing live files are never overwritten and nothing is
// ever deleted (see backup.RestoreTier2Files). Same handler shape as backupRestoreHandler.
@@ -1303,20 +1330,50 @@ func (s *Server) backupTier2RestoreHandler(w http.ResponseWriter, r *http.Reques
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape("Egy mentési/visszaállítási művelet már fut."), http.StatusFound)
return
}
// C9-F1: refuse UP FRONT — before any op is begun and before the app is stopped — when this app's
// Tier-2 copy holds nothing this restore can read (43 of the 53 catalog apps: their data lives in
// Docker named volumes, captured into recovery-unit/ and never read here). Previously the customer
// got an outage, zero files, and „Nincs hiányzó fájl — minden fájl megvan a helyén." — a claim
// about data the restore never examined, at the exact moment they pressed it BECAUSE data was
// missing. Only the no-coverage case is pre-flighted; every other refusal keeps its existing async
// path so this change cannot alter behaviour anywhere else.
cov, covErr := s.backupMgr.Tier2RestoreCoverage(stackName)
if covErr == nil && !cov.CanRestore() {
s.logger.Printf("[WARN] [web] Tier-2 file restore refused up front: stack=%s has no restorable subtree in its copy (unit_present=%v) — app NOT stopped", stackName, cov.HasUnit)
http.Redirect(w, r, "/backups/apps?flash_error="+url.QueryEscape(tier2NoCoverageMsg), http.StatusFound)
return
}
s.logger.Printf("[WARN] [web] Tier-2 file restore requested (async): stack=%s from %s", stackName, r.RemoteAddr)
s.backupMgr.BeginRestoreOp("tier2-restore", stackName)
go func() {
n, err := s.backupMgr.RestoreTier2Files(stackName)
if err != nil {
// The no-coverage refusal is not an operational failure — it means this action does not
// apply to this app. Say that, and name the one that does, instead of "sikertelen".
if errors.Is(err, backup.ErrTier2NoRestorableData) {
s.logger.Printf("[WARN] [web] Tier-2 file restore not applicable: stack=%s", stackName)
s.backupMgr.EndRestoreOp(false, tier2NoCoverageMsg)
return
}
s.logger.Printf("[ERROR] [web] Tier-2 file restore failed (async): stack=%s: %v", stackName, err)
s.backupMgr.EndRestoreOp(false, "Fájl-visszaállítás sikertelen: "+err.Error())
return
}
msg := "Nincs hiányzó fájl — minden fájl megvan a helyén."
// C9-F1 (the quiet half): even where the restore DOES cover something it covers only the
// file-based legs — never the app's database or named volumes, which sit unread in the same
// copy's recovery-unit/. „minden fájl megvan a helyén" was a blanket claim over data that was
// never opened; immich's 1.3 GB Postgres unit is the case that makes it dangerous. Claim only
// what was EXAMINED, and disclose the rest.
msg := "Minden vizsgált fájl megvan a helyén."
if n > 0 {
msg = fmt.Sprintf("%s: %d fájl visszaállítva a másodlagos másolatból.", stackName, n)
}
s.logger.Printf("[INFO] [web] Tier-2 file restore completed (async): stack=%s (%d files)", stackName, n)
if cov.HasUnit {
msg += " " + tier2UnitNotCoveredMsg
}
s.logger.Printf("[INFO] [web] Tier-2 file restore completed (async): stack=%s (%d files, legs=%v)", stackName, n, cov.Legs)
s.backupMgr.EndRestoreOp(true, msg)
}()
http.Redirect(w, r, "/backups/apps?flash="+url.QueryEscape("Fájl-visszaállítás elindult — az állapot itt frissül."), http.StatusFound)