C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (v0.183.0)
Both are the system reporting healthy while the customer is not, and both live in the same status-derivation code. Neither is fixed by making the system quieter. C9-F1 (HIGH) — Tier-2 writes recovery-unit/ on EVERY run and RestoreTier2Files has never read it (tier2_restore.go:101-104 reads hdd/ + userdata/ only). Phase 0 enumerated all 53 catalog templates against both demo boxes: 43 apps have NO readable subtree, so the button stopped the app, restored 0 files, restarted it and said "Nincs hiányzó fájl — minden fájl megvan a helyén." — at the moment the customer pressed it because files were missing, with 156 MB of BookStack's data unread in the same copy. 9 apps have file legs but never their DB or volumes, so the same sentence was also a clean bill of health over data never opened (immich: 1.3 GB Postgres unit). Honesty half shipped: a pre-flight coverage check refuses UP FRONT without stopping the app and NAMES the action that works; a run that proceeds claims only what it EXAMINED and discloses that the database and volumes are not covered. Completeness is filed as C9-F1b — routing to the Tier-1 unit restore puts a destructive operation behind a non-destructive button, so its confirm copy has to carry that difference. C9-F4 filed: nothing reads the Tier-2 recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable by any customer action. C9-F2 (HIGH) — a crash loop was counted as working. StateRestarting is deliberately NOT added to IsDownState (that alarms on every deploy fleet-wide, the over-correction F-A1 nearly cost us); a sustained run becomes down after crashLoopAfter = 5m, set above the 120s deploy timeout, Mealie's 60s start_period and R-97b's 180s grace. The dashboard counter uses the same predicate, so it no longer contradicts the alarm on the same screen. README's claim that faults "still surface as restarting" was a wish with no test — corrected in place; it is the seventh such instance. Six red-proofs observed, including the one that matters most: adding StateRestarting to IsDownState fails the brief-restart test with "every deploy and update would page the operator". go test ./... rc=0, 27 packages, run and read separately from this commit.
This commit is contained in:
@@ -33,8 +33,81 @@ var (
|
||||
// marker). Refuse rather than read a flat layout we no longer understand — safe, because tier-2
|
||||
// restore is missing-file recovery and the live data still exists in that scenario.
|
||||
errTier2OldLayout = errors.New("A 2. mentés régi formátumú — futtass előbb egy új másodlagos mentést.")
|
||||
// ErrTier2NoRestorableData (C9-F1) — this app HAS a Tier-2 copy, but that copy contains no subtree
|
||||
// this restore can read: its data lives entirely in Docker named volumes, which are captured into
|
||||
// recovery-unit/ (db-dumps + volume-dumps) and NEVER read by this path. 43 of the 53 catalog apps
|
||||
// are in this class. Exported so the handler can refuse BEFORE stopping the app and name the action
|
||||
// that does work, instead of taking an outage and reporting "no missing files".
|
||||
ErrTier2NoRestorableData = errors.New("ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza")
|
||||
)
|
||||
|
||||
// Tier2Coverage says what a Tier-2 restore can and cannot return for one app — the asymmetry C9-F1
|
||||
// is about. Computed from the RECORDED copy on disk, never guessed from the catalog, so an app whose
|
||||
// template changed is judged by what its actual copy holds.
|
||||
//
|
||||
// The distinction that matters: Legs are the subtrees RestoreTier2Files reads (hdd/, userdata/);
|
||||
// HasUnit means the copy ALSO holds a full recovery unit — the app's database dumps and named-volume
|
||||
// tarballs — which this restore path never opens. An app can have HasUnit && no Legs (43 of 53), in
|
||||
// which case the restore is a guaranteed no-op no matter how much data was lost.
|
||||
type Tier2Coverage struct {
|
||||
Legs []string // subtrees this restore reads and that exist in the copy: "hdd", "userdata"
|
||||
HasUnit bool // recovery-unit/ present — captured, but NOT restorable by this path
|
||||
}
|
||||
|
||||
// CanRestore reports whether the restore has any subtree to read at all.
|
||||
func (c Tier2Coverage) CanRestore() bool { return len(c.Legs) > 0 }
|
||||
|
||||
// tier2CoverageAt inspects a resolved copy directory. Pure filesystem stat — no side effects.
|
||||
func tier2CoverageAt(destBase string) Tier2Coverage {
|
||||
var c Tier2Coverage
|
||||
for _, leg := range []string{"hdd", "userdata"} {
|
||||
if fi, err := os.Stat(filepath.Join(destBase, leg)); err == nil && fi.IsDir() {
|
||||
c.Legs = append(c.Legs, leg)
|
||||
}
|
||||
}
|
||||
if fi, err := os.Stat(filepath.Join(destBase, "recovery-unit")); err == nil && fi.IsDir() {
|
||||
c.HasUnit = true
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// Tier2RestoreCoverage resolves the app's RECORDED Tier-2 copy and reports what a restore could
|
||||
// return from it. Errors are the same refusals RestoreTier2Files itself would raise, so the caller
|
||||
// can surface them before starting anything — this is what lets the handler refuse without an outage.
|
||||
func (m *Manager) Tier2RestoreCoverage(stackName string) (Tier2Coverage, error) {
|
||||
destBase, err := m.tier2RecordedCopyDir(stackName)
|
||||
if err != nil {
|
||||
return Tier2Coverage{}, err
|
||||
}
|
||||
return tier2CoverageAt(destBase), nil
|
||||
}
|
||||
|
||||
// tier2RecordedCopyDir resolves the RECORDED Tier-2 copy dir for a stack, applying every
|
||||
// source-side refusal in one place so the pre-flight check and the restore itself cannot drift.
|
||||
func (m *Manager) tier2RecordedCopyDir(stackName string) (string, error) {
|
||||
var destBase string
|
||||
if m.settings != nil {
|
||||
if cfg := m.settings.GetCrossDriveConfig(stackName); cfg != nil && cfg.LastRun != "" && cfg.DestinationPath != "" {
|
||||
if m.settings.IsDisconnected(cfg.DestinationPath) {
|
||||
return "", errTier2DriveGone
|
||||
}
|
||||
destBase = filepath.Join(cfg.DestinationPath, "backups", "secondary", stackName)
|
||||
}
|
||||
}
|
||||
if destBase == "" {
|
||||
return "", errNoTier2Copy
|
||||
}
|
||||
if _, statErr := os.Stat(destBase); statErr != nil {
|
||||
return "", errNoTier2Copy // recorded but the copy dir is gone — same honest refusal
|
||||
}
|
||||
// §7-G2 marker gate: a pre-v2 (flat) copy has no marker → refuse rather than read a layout we no
|
||||
// longer understand (live data still exists for missing-file recovery).
|
||||
if _, mErr := os.Stat(filepath.Join(destBase, tier2LayoutMarker)); mErr != nil {
|
||||
return "", errTier2OldLayout
|
||||
}
|
||||
return destBase, nil
|
||||
}
|
||||
|
||||
// RestoreTier2Files restores the app's MISSING user files in place from its recorded Tier-2 copy
|
||||
// (additive-only; see the package comment above). Returns how many regular files were copied back.
|
||||
//
|
||||
@@ -70,25 +143,21 @@ func (m *Manager) RestoreTier2Files(stackName string) (filesRestored int, err er
|
||||
liveNsRoot := m.namespaceRoot(drive)
|
||||
|
||||
// Source side: the RECORDED Tier-2 copy must exist, its drive connected, and it must be v2.
|
||||
var destBase string
|
||||
if m.settings != nil {
|
||||
if cfg := m.settings.GetCrossDriveConfig(stackName); cfg != nil && cfg.LastRun != "" && cfg.DestinationPath != "" {
|
||||
if m.settings.IsDisconnected(cfg.DestinationPath) {
|
||||
return 0, errTier2DriveGone
|
||||
}
|
||||
destBase = filepath.Join(cfg.DestinationPath, "backups", "secondary", stackName)
|
||||
}
|
||||
destBase, err := m.tier2RecordedCopyDir(stackName)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if destBase == "" {
|
||||
return 0, errNoTier2Copy
|
||||
}
|
||||
if _, statErr := os.Stat(destBase); statErr != nil {
|
||||
return 0, errNoTier2Copy // recorded but the copy dir is gone — same honest refusal
|
||||
}
|
||||
// §7-G2 marker gate: a pre-v2 (flat) copy has no marker → refuse rather than read a layout we no
|
||||
// longer understand (live data still exists for missing-file recovery).
|
||||
if _, mErr := os.Stat(filepath.Join(destBase, tier2LayoutMarker)); mErr != nil {
|
||||
return 0, errTier2OldLayout
|
||||
|
||||
// C9-F1: refuse BEFORE the app is stopped if this copy holds nothing this path can read. Without
|
||||
// this the app was stopped, zero files were copied, it was restarted, and the customer was told
|
||||
// "Nincs hiányzó fájl — minden fájl megvan a helyén." — an outage plus a claim about data the
|
||||
// restore never looked at. Placed with the other source-side refusals, all of which precede the
|
||||
// stop, so the promise "all refusals happen BEFORE the app is stopped" stays true.
|
||||
cov := tier2CoverageAt(destBase)
|
||||
if !cov.CanRestore() {
|
||||
m.logger.Printf("[WARN] [backup] Tier-2 file restore refused for %s: the recorded copy has no restorable subtree (unit_present=%v) — the app was NOT stopped",
|
||||
stackName, cov.HasUnit)
|
||||
return 0, ErrTier2NoRestorableData
|
||||
}
|
||||
|
||||
copier := m.restoreFilesCopier
|
||||
|
||||
Reference in New Issue
Block a user