v0.238.1: the nightly backup leaves an app alone WHILE it is being updated, not only once it is held (slice 4 follow-up)
gates / gates (push) Successful in 13s
gates / gates (push) Successful in 13s
Found live in v0.238.0 Scenario F on demo-hp: during an update's 5-minute health wait the app is not yet held, and the periodic recovery-unit capture at 10:17:09 wrote the never-started definition (alpine:3.20) into its PRIMARY unit, 53 s before the hold landed. The Tier-2 mirror the hold names survived only because Tier 2 runs daily; a nightly Tier 2 inside a verify window would have mirrored the broken definition over the copy the customer is told to restore from. backup.Manager.isHeld — consulted by the capture sweep, the Tier-2 run and the volume dump — is now also true while a guarded update is moving the app, via SetUpdatingCheck wired in main.go to stacks.Manager.IsUpdating. Test with positive control + red-proof; wiring pinned. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -594,7 +594,7 @@ the boot sweep) puts a pin back or marks an interrupted update for `ResumeInterr
|
||||
|
||||
**Every unattended start path honours a hold:** the boot sweep and the app-stop guard (as before), and
|
||||
since v0.237.0 the drive-return gate and the nightly volume dump. The nightly capture and Tier-2 run
|
||||
skip a held app so its restore point is not overwritten.
|
||||
skip a held app so its restore point is not overwritten — and, since v0.238.1, an app whose update is still in progress (the periodic capture overwrote a primary unit during a health wait, found live).
|
||||
|
||||
**The page (v0.238.0).** `Frissítés` follows the job — the button shows the phase label and the page
|
||||
reloads when the update ends. An updating card offers no lifecycle button; a held card shows the hold
|
||||
|
||||
@@ -554,6 +554,11 @@ func main() {
|
||||
// An UNWIRED manager also refuses (fail closed); TestSlice4_UpdateGuardsAreWiredAtStartup walks
|
||||
// this file for the call, because a seam built and never wired has shipped here seven times.
|
||||
stackMgr.SetUpdateGuards(&updateGuardsAdapter{b: backupMgr, q: quiesceLoop})
|
||||
// v0.238.1: the nightly legs (capture, Tier 2, volume dump) leave an app alone WHILE it is being
|
||||
// updated, not only once it is held — found live in Scenario F, see backup.Manager.isHeld.
|
||||
if backupMgr != nil {
|
||||
backupMgr.SetUpdatingCheck(stackMgr.IsUpdating)
|
||||
}
|
||||
if n := stackMgr.ResumeInterruptedUpdates(ctx); n > 0 {
|
||||
logger.Printf("[WARN] [update] resumed %d interrupted update(s)", n)
|
||||
}
|
||||
|
||||
@@ -102,3 +102,12 @@ func TestSlice4_DriveStartGate_NamesAnUpdateHold(t *testing.T) {
|
||||
t.Errorf("ok=%v why=%q", ok, why)
|
||||
}
|
||||
}
|
||||
|
||||
// v0.238.1: the backup manager must be told which apps are mid-update, or the nightly legs overwrite a
|
||||
// restore point during an update's health wait (found live, Scenario F).
|
||||
func TestSlice4_UpdatingCheckIsWiredAtStartup(t *testing.T) {
|
||||
lines, _, _ := slice4CallLines(t)
|
||||
if len(lines["SetUpdatingCheck"]) == 0 {
|
||||
t.Fatal("backupMgr.SetUpdatingCheck is never called — the nightly legs cannot see an update in progress")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -170,6 +170,9 @@ type Manager struct {
|
||||
// disconnected) can be unit-tested without Docker. Nil → the real DumpAppVolumesSafe.
|
||||
dumpVolumesSafe func(stackName string) error
|
||||
|
||||
// updatingCheck (slice 4) — nil-safe; see isHeld / SetUpdatingCheck.
|
||||
updatingCheck func(stackName string) bool
|
||||
|
||||
// R-354 volume-REPLAY seam — the mirror of the F17 DB seams above, so the off-site path's new
|
||||
// volume leg is unit-testable without Docker. Nil → the real restoreDockerVolumesFrom.
|
||||
volumeReplayFrom func(stackName, dumpDir string) (int, error)
|
||||
|
||||
@@ -166,3 +166,30 @@ func TestSlice4_NightlyLegsLeaveAHeldAppAlone(t *testing.T) {
|
||||
t.Errorf("positive control: the unheld app must still be mirrored, got %v", mirrored)
|
||||
}
|
||||
}
|
||||
|
||||
// v0.238.1 — the gap Scenario F found live: during the update's health wait the app is not yet held,
|
||||
// and the periodic capture wrote the never-started new definition into its primary unit. An app a
|
||||
// guarded update is moving must be left alone by all three nightly legs, exactly like a held one.
|
||||
//
|
||||
// COMPANION RED-PROOF (REPORT.md): delete the updatingCheck clause from isHeld — the updating app is
|
||||
// then dumped, captured and mirrored, and this test fails.
|
||||
func TestSlice4_NightlyLegsLeaveAnAppMidUpdateAlone(t *testing.T) {
|
||||
h := newAdmissionHarness(t, "updating", "free")
|
||||
h.m.settings = slice4Settings(t)
|
||||
h.m.SetUpdatingCheck(func(name string) bool { return name == "updating" })
|
||||
h.m.runVolumeDumps()
|
||||
h.m.captureAllRecoveryUnits()
|
||||
var mirrored []string
|
||||
h.m.perAppTier2 = func(name string) error { mirrored = append(mirrored, name); return nil }
|
||||
h.m.RunAllTier2()
|
||||
for _, list := range [][]string{h.volDumped, h.prov.stopped, h.prov.infoHits, mirrored} {
|
||||
for _, n := range list {
|
||||
if n == "updating" {
|
||||
t.Fatalf("a nightly leg touched an app MID-UPDATE (dumped=%v stopped=%v captured=%v mirrored=%v)", h.volDumped, h.prov.stopped, h.prov.infoHits, mirrored)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(mirrored) != 1 || mirrored[0] != "free" {
|
||||
t.Errorf("positive control: the app not being updated must still be mirrored, got %v", mirrored)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -295,10 +295,29 @@ func (m *Manager) HoldAfterFailedUpdate(stackName string, at time.Time, copyDate
|
||||
return nil
|
||||
}
|
||||
|
||||
// isHeld reports whether an app carries ANY hold. Used by the nightly legs to leave a held app alone.
|
||||
// isHeld reports whether the nightly legs must leave an app alone: it carries ANY hold, OR a guarded
|
||||
// update is moving it right now.
|
||||
//
|
||||
// THE SECOND HALF WAS FOUND LIVE, v0.238.0 Scenario F on demo-hp 2026-09-13. During the update's
|
||||
// 5-minute health wait the app is not yet held, and the periodic capture ran at 10:17:09 and wrote
|
||||
// the NEW definition (alpine:3.20, which never started) into the app's PRIMARY unit, 53 s before the
|
||||
// hold landed at 10:18:02. The Tier-2 mirror the hold names was intact only because the Tier-2 run is
|
||||
// daily — a nightly Tier-2 falling inside a verify window would have mirrored the broken definition
|
||||
// over the very copy the customer is told to restore from. An app mid-update has a restore point that
|
||||
// must not move, exactly like a held one.
|
||||
func (m *Manager) isHeld(stackName string) bool {
|
||||
held, _ := m.RestoreHoldFor(stackName)
|
||||
return held
|
||||
if held {
|
||||
return true
|
||||
}
|
||||
return m.updatingCheck != nil && m.updatingCheck(stackName)
|
||||
}
|
||||
|
||||
// SetUpdatingCheck wires the "is a guarded update moving this app" question (stacks.Manager.IsUpdating).
|
||||
// INIT-ONLY, in main.go — pinned by TestSlice4_UpdatingCheckIsWiredAtStartup. The backup package cannot
|
||||
// import stacks, which is why it is a seam.
|
||||
func (m *Manager) SetUpdatingCheck(fn func(stackName string) bool) {
|
||||
m.updatingCheck = fn
|
||||
}
|
||||
|
||||
// clearUpdateHoldAfterRestore lifts an UPDATE hold once a person has restored the app successfully.
|
||||
|
||||
Reference in New Issue
Block a user