C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (v0.183.0)

Both are the system reporting healthy while the customer is not, and both live in the same
status-derivation code. Neither is fixed by making the system quieter.

C9-F1 (HIGH) — Tier-2 writes recovery-unit/ on EVERY run and RestoreTier2Files has never read
it (tier2_restore.go:101-104 reads hdd/ + userdata/ only). Phase 0 enumerated all 53 catalog
templates against both demo boxes: 43 apps have NO readable subtree, so the button stopped the
app, restored 0 files, restarted it and said "Nincs hiányzó fájl — minden fájl megvan a helyén."
— at the moment the customer pressed it because files were missing, with 156 MB of BookStack's
data unread in the same copy. 9 apps have file legs but never their DB or volumes, so the same
sentence was also a clean bill of health over data never opened (immich: 1.3 GB Postgres unit).

Honesty half shipped: a pre-flight coverage check refuses UP FRONT without stopping the app and
NAMES the action that works; a run that proceeds claims only what it EXAMINED and discloses that
the database and volumes are not covered. Completeness is filed as C9-F1b — routing to the
Tier-1 unit restore puts a destructive operation behind a non-destructive button, so its confirm
copy has to carry that difference. C9-F4 filed: nothing reads the Tier-2 recovery-unit/ mirror,
so the second local copy that exists for drive loss is unreachable by any customer action.

C9-F2 (HIGH) — a crash loop was counted as working. StateRestarting is deliberately NOT added to
IsDownState (that alarms on every deploy fleet-wide, the over-correction F-A1 nearly cost us); a
sustained run becomes down after crashLoopAfter = 5m, set above the 120s deploy timeout, Mealie's
60s start_period and R-97b's 180s grace. The dashboard counter uses the same predicate, so it no
longer contradicts the alarm on the same screen. README's claim that faults "still surface as
restarting" was a wish with no test — corrected in place; it is the seventh such instance.

Six red-proofs observed, including the one that matters most: adding StateRestarting to
IsDownState fails the brief-restart test with "every deploy and update would page the operator".
go test ./... rc=0, 27 packages, run and read separately from this commit.
This commit is contained in:
2026-07-28 18:53:56 +02:00
parent d8b3279731
commit fd50a73e65
12 changed files with 805 additions and 38 deletions
+87 -18
View File
@@ -33,8 +33,81 @@ var (
// marker). Refuse rather than read a flat layout we no longer understand — safe, because tier-2
// restore is missing-file recovery and the live data still exists in that scenario.
errTier2OldLayout = errors.New("A 2. mentés régi formátumú — futtass előbb egy új másodlagos mentést.")
// ErrTier2NoRestorableData (C9-F1) — this app HAS a Tier-2 copy, but that copy contains no subtree
// this restore can read: its data lives entirely in Docker named volumes, which are captured into
// recovery-unit/ (db-dumps + volume-dumps) and NEVER read by this path. 43 of the 53 catalog apps
// are in this class. Exported so the handler can refuse BEFORE stopping the app and name the action
// that does work, instead of taking an outage and reporting "no missing files".
ErrTier2NoRestorableData = errors.New("ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza")
)
// Tier2Coverage says what a Tier-2 restore can and cannot return for one app — the asymmetry C9-F1
// is about. Computed from the RECORDED copy on disk, never guessed from the catalog, so an app whose
// template changed is judged by what its actual copy holds.
//
// The distinction that matters: Legs are the subtrees RestoreTier2Files reads (hdd/, userdata/);
// HasUnit means the copy ALSO holds a full recovery unit — the app's database dumps and named-volume
// tarballs — which this restore path never opens. An app can have HasUnit && no Legs (43 of 53), in
// which case the restore is a guaranteed no-op no matter how much data was lost.
type Tier2Coverage struct {
Legs []string // subtrees this restore reads and that exist in the copy: "hdd", "userdata"
HasUnit bool // recovery-unit/ present — captured, but NOT restorable by this path
}
// CanRestore reports whether the restore has any subtree to read at all.
func (c Tier2Coverage) CanRestore() bool { return len(c.Legs) > 0 }
// tier2CoverageAt inspects a resolved copy directory. Pure filesystem stat — no side effects.
func tier2CoverageAt(destBase string) Tier2Coverage {
var c Tier2Coverage
for _, leg := range []string{"hdd", "userdata"} {
if fi, err := os.Stat(filepath.Join(destBase, leg)); err == nil && fi.IsDir() {
c.Legs = append(c.Legs, leg)
}
}
if fi, err := os.Stat(filepath.Join(destBase, "recovery-unit")); err == nil && fi.IsDir() {
c.HasUnit = true
}
return c
}
// Tier2RestoreCoverage resolves the app's RECORDED Tier-2 copy and reports what a restore could
// return from it. Errors are the same refusals RestoreTier2Files itself would raise, so the caller
// can surface them before starting anything — this is what lets the handler refuse without an outage.
func (m *Manager) Tier2RestoreCoverage(stackName string) (Tier2Coverage, error) {
destBase, err := m.tier2RecordedCopyDir(stackName)
if err != nil {
return Tier2Coverage{}, err
}
return tier2CoverageAt(destBase), nil
}
// tier2RecordedCopyDir resolves the RECORDED Tier-2 copy dir for a stack, applying every
// source-side refusal in one place so the pre-flight check and the restore itself cannot drift.
func (m *Manager) tier2RecordedCopyDir(stackName string) (string, error) {
var destBase string
if m.settings != nil {
if cfg := m.settings.GetCrossDriveConfig(stackName); cfg != nil && cfg.LastRun != "" && cfg.DestinationPath != "" {
if m.settings.IsDisconnected(cfg.DestinationPath) {
return "", errTier2DriveGone
}
destBase = filepath.Join(cfg.DestinationPath, "backups", "secondary", stackName)
}
}
if destBase == "" {
return "", errNoTier2Copy
}
if _, statErr := os.Stat(destBase); statErr != nil {
return "", errNoTier2Copy // recorded but the copy dir is gone — same honest refusal
}
// §7-G2 marker gate: a pre-v2 (flat) copy has no marker → refuse rather than read a layout we no
// longer understand (live data still exists for missing-file recovery).
if _, mErr := os.Stat(filepath.Join(destBase, tier2LayoutMarker)); mErr != nil {
return "", errTier2OldLayout
}
return destBase, nil
}
// RestoreTier2Files restores the app's MISSING user files in place from its recorded Tier-2 copy
// (additive-only; see the package comment above). Returns how many regular files were copied back.
//
@@ -70,25 +143,21 @@ func (m *Manager) RestoreTier2Files(stackName string) (filesRestored int, err er
liveNsRoot := m.namespaceRoot(drive)
// Source side: the RECORDED Tier-2 copy must exist, its drive connected, and it must be v2.
var destBase string
if m.settings != nil {
if cfg := m.settings.GetCrossDriveConfig(stackName); cfg != nil && cfg.LastRun != "" && cfg.DestinationPath != "" {
if m.settings.IsDisconnected(cfg.DestinationPath) {
return 0, errTier2DriveGone
}
destBase = filepath.Join(cfg.DestinationPath, "backups", "secondary", stackName)
}
destBase, err := m.tier2RecordedCopyDir(stackName)
if err != nil {
return 0, err
}
if destBase == "" {
return 0, errNoTier2Copy
}
if _, statErr := os.Stat(destBase); statErr != nil {
return 0, errNoTier2Copy // recorded but the copy dir is gone — same honest refusal
}
// §7-G2 marker gate: a pre-v2 (flat) copy has no marker → refuse rather than read a layout we no
// longer understand (live data still exists for missing-file recovery).
if _, mErr := os.Stat(filepath.Join(destBase, tier2LayoutMarker)); mErr != nil {
return 0, errTier2OldLayout
// C9-F1: refuse BEFORE the app is stopped if this copy holds nothing this path can read. Without
// this the app was stopped, zero files were copied, it was restarted, and the customer was told
// "Nincs hiányzó fájl — minden fájl megvan a helyén." — an outage plus a claim about data the
// restore never looked at. Placed with the other source-side refusals, all of which precede the
// stop, so the promise "all refusals happen BEFORE the app is stopped" stays true.
cov := tier2CoverageAt(destBase)
if !cov.CanRestore() {
m.logger.Printf("[WARN] [backup] Tier-2 file restore refused for %s: the recorded copy has no restorable subtree (unit_present=%v) — the app was NOT stopped",
stackName, cov.HasUnit)
return 0, ErrTier2NoRestorableData
}
copier := m.restoreFilesCopier