v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s
gates / gates (push) Successful in 9s
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills. It emits the PRE-EXISTING disk_warning/disk_critical pair, which was allowlisted, copy'd, default-enabled and checkbox'd with no producer in any repo — the sixth "built but never wired" instance here. Two threshold terms (85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at both ends of this fleet's size range. Edge-triggered on escalation only, state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil usage read is never a warning and never clears one. Per filesystem, never per app. Daily 03:30, before the nightly app-data legs. R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture fails, loop continuing, carrying the target filesystem's used/free bytes. Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed, which is customer-enabled and would email the customer about a failure they cannot act on. D-c overrides R-158's own proposal here. R-174: the app-stop guard no longer starts apps onto MISSING drives — a regression in v0.189.0 code, found by review and closed the same session. SetStarter got the raw stack manager, whose StartStack has no drive gate, and Recover runs at startup. R-171 one path over. bootDriveGate could not be reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read vars assigned after Recover runs), so holder #3 is extracted into a shared driveStartGate with a test pinning the delegation. ErrStartRefused splits a refusal from a failure: both keep the marker, only Failed alarms, because routing a deliberate hold into NotifyBackupFailed is the same false alarm. Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
This commit is contained in:
@@ -4,6 +4,7 @@ import (
|
||||
"go/ast"
|
||||
"go/parser"
|
||||
"go/token"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
@@ -140,31 +141,56 @@ func TestMainReportsTheInterruptedOperation(t *testing.T) {
|
||||
}
|
||||
// It must be guarded, not unconditional: a box with nothing to recover must not email an operator
|
||||
// on every single boot.
|
||||
guarded := false
|
||||
//
|
||||
// R-174 STRENGTHENED THIS. `!= nil` alone is no longer sufficient, because Recover now returns a
|
||||
// non-nil result for a recovery that merely REFUSED starts (an absent data drive) — the drive
|
||||
// gate working as designed. `NotifyBackupFailed` sends `backup_failed`, which is customer-enabled
|
||||
// by default (settings.DefaultEnabledEvents), so a nil-only guard would email the customer
|
||||
// "A biztonsági mentés sikertelen!" about an app nothing is wrong with. The guard must consult
|
||||
// Alarming().
|
||||
guardedByNil, guardedByAlarming := false, false
|
||||
ast.Inspect(body, func(n ast.Node) bool {
|
||||
ifst, ok := n.(*ast.IfStmt)
|
||||
if !ok || ifst.Cond == nil {
|
||||
return true
|
||||
}
|
||||
bin, ok := ifst.Cond.(*ast.BinaryExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
x, ok := bin.X.(*ast.Ident)
|
||||
if !ok || x.Name != "appStopRecovery" {
|
||||
return true
|
||||
}
|
||||
carries := false
|
||||
for _, name := range callsInMain(t, ifst.Body) {
|
||||
if name == "NotifyBackupFailed" {
|
||||
guarded = true
|
||||
carries = true
|
||||
}
|
||||
}
|
||||
if !carries {
|
||||
return true
|
||||
}
|
||||
// Walk the whole condition: it may be `a != nil && a.Alarming()`.
|
||||
ast.Inspect(ifst.Cond, func(c ast.Node) bool {
|
||||
switch e := c.(type) {
|
||||
case *ast.BinaryExpr:
|
||||
if x, ok := e.X.(*ast.Ident); ok && x.Name == "appStopRecovery" && e.Op == token.NEQ {
|
||||
guardedByNil = true
|
||||
}
|
||||
case *ast.CallExpr:
|
||||
if sel, ok := e.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Alarming" {
|
||||
if x, ok := sel.X.(*ast.Ident); ok && x.Name == "appStopRecovery" {
|
||||
guardedByAlarming = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
return true
|
||||
})
|
||||
if !guarded {
|
||||
t.Fatal("the interrupted-operation alert is not guarded by `if appStopRecovery != nil` — every " +
|
||||
if !guardedByNil {
|
||||
t.Fatal("the interrupted-operation alert is not guarded by `appStopRecovery != nil` — every " +
|
||||
"healthy boot would page the operator about a backup that was never interrupted")
|
||||
}
|
||||
if !guardedByAlarming {
|
||||
t.Fatal("the interrupted-operation alert is not guarded by appStopRecovery.Alarming() — a " +
|
||||
"recovery that only REFUSED starts (drive absent) would be reported through " +
|
||||
"NotifyBackupFailed, a customer-enabled event type, telling the customer their backup " +
|
||||
"failed when the drive gate was simply doing its job (R-174)")
|
||||
}
|
||||
}
|
||||
|
||||
// --- R-171 seam: the boot drive gate must be WIRED in production -------------------------------
|
||||
@@ -213,6 +239,200 @@ func TestMainWiresBootDriveGate(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// --- R-174 seam: the app-stop guard's starter must be GATED in production -----------------------
|
||||
|
||||
// TestMainWiresGatedAppStopStarter pins Part 0's production wiring. `SetStarter(stackMgr)` — the raw
|
||||
// manager, which is what shipped in v0.189.0 — compiles, passes every behavioural test in
|
||||
// internal/backup (they inject their own gating starter), and silently starts apps onto absent
|
||||
// drives at boot. The ONLY thing that distinguishes the fixed wiring from the broken one is the
|
||||
// argument at the call site, so that is what this reads.
|
||||
//
|
||||
// AST, not strings.Contains: a commented-out call still contains the string.
|
||||
func TestMainWiresGatedAppStopStarter(t *testing.T) {
|
||||
body := mainBody(t)
|
||||
|
||||
var arg ast.Expr
|
||||
found := false
|
||||
ast.Inspect(body, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
sel, ok := call.Fun.(*ast.SelectorExpr)
|
||||
if !ok || sel.Sel.Name != "SetStarter" || len(call.Args) != 1 {
|
||||
return true
|
||||
}
|
||||
// Only the app-stop guard's SetStarter, not some other type's.
|
||||
if x, ok := sel.X.(*ast.Ident); !ok || x.Name != "appStopGuard" {
|
||||
return true
|
||||
}
|
||||
arg, found = call.Args[0], true
|
||||
return false
|
||||
})
|
||||
if !found {
|
||||
t.Fatal("func main() no longer calls appStopGuard.SetStarter — Recover would find the marker " +
|
||||
"and be unable to start anything")
|
||||
}
|
||||
|
||||
// The argument must be a gatedAppStopStarter composite literal. A bare identifier (`stackMgr`)
|
||||
// is precisely the v0.189.0 defect.
|
||||
lit, ok := arg.(*ast.CompositeLit)
|
||||
if !ok {
|
||||
t.Fatalf("appStopGuard.SetStarter is wired with %T, not a gatedAppStopStarter literal — an "+
|
||||
"un-gated starter restarts apps onto MISSING drives at boot (R-174, the R-171 defect one "+
|
||||
"path over)", arg)
|
||||
}
|
||||
id, ok := lit.Type.(*ast.Ident)
|
||||
if !ok || id.Name != "gatedAppStopStarter" {
|
||||
t.Fatalf("appStopGuard.SetStarter is wired with a %v literal, want gatedAppStopStarter", lit.Type)
|
||||
}
|
||||
|
||||
// And that gate must be a driveStartGate — the SAME predicate the boot sweep uses, so the two
|
||||
// cannot disagree about whether an app's drive is available.
|
||||
gated := false
|
||||
for _, el := range lit.Elts {
|
||||
kv, ok := el.(*ast.KeyValueExpr)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
k, ok := kv.Key.(*ast.Ident)
|
||||
if !ok || k.Name != "gate" {
|
||||
continue
|
||||
}
|
||||
if gl, ok := kv.Value.(*ast.CompositeLit); ok {
|
||||
if gid, ok := gl.Type.(*ast.Ident); ok && gid.Name == "driveStartGate" {
|
||||
gated = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if !gated {
|
||||
t.Fatal("the app-stop starter's gate is not a driveStartGate — the crash recovery and the " +
|
||||
"boot sweep would answer \"may this app start?\" from two different implementations, " +
|
||||
"which is the drift the extraction exists to prevent")
|
||||
}
|
||||
}
|
||||
|
||||
// TestBootDriveGateAndAppStopShareTheDrivePredicate pins the OTHER half of the same claim: the boot
|
||||
// sweep must keep delegating to driveStartGate rather than growing its own copy of the drive checks.
|
||||
//
|
||||
// This is the "a comment asserting an invariant needs a test pinning it" rule. The claim — that the
|
||||
// two gates cannot disagree — is true only while both call the same code.
|
||||
func TestBootDriveGateAndAppStopShareTheDrivePredicate(t *testing.T) {
|
||||
fset := token.NewFileSet()
|
||||
f, err := parser.ParseFile(fset, "main.go", nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("parse main.go: %v", err)
|
||||
}
|
||||
|
||||
var mayStart *ast.FuncDecl
|
||||
for _, decl := range f.Decls {
|
||||
fn, ok := decl.(*ast.FuncDecl)
|
||||
if !ok || fn.Name.Name != "MayStart" || fn.Recv == nil || len(fn.Recv.List) != 1 {
|
||||
continue
|
||||
}
|
||||
if id, ok := fn.Recv.List[0].Type.(*ast.Ident); ok && id.Name == "bootDriveGate" {
|
||||
mayStart = fn
|
||||
}
|
||||
}
|
||||
if mayStart == nil {
|
||||
t.Fatal("bootDriveGate.MayStart not found in main.go")
|
||||
}
|
||||
|
||||
// It must call through to the shared predicate.
|
||||
delegates := false
|
||||
ast.Inspect(mayStart.Body, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok {
|
||||
return true
|
||||
}
|
||||
sel, ok := call.Fun.(*ast.SelectorExpr)
|
||||
if !ok || sel.Sel.Name != "MayStart" {
|
||||
return true
|
||||
}
|
||||
if x, ok := sel.X.(*ast.SelectorExpr); ok && x.Sel.Name == "drive" {
|
||||
delegates = true
|
||||
}
|
||||
return true
|
||||
})
|
||||
if !delegates {
|
||||
t.Fatal("bootDriveGate.MayStart no longer delegates to the shared driveStartGate — the boot " +
|
||||
"sweep and the app-stop crash recovery would each carry their own drive logic, and the " +
|
||||
"two can then disagree about whether an app may start (R-174)")
|
||||
}
|
||||
}
|
||||
|
||||
// --- R-158 / R-167 seams: both new alerts must be WIRED in production ---------------------------
|
||||
|
||||
// TestMainWiresTheUnitCaptureAlert pins Part 1's seam. `SetUnitNotify` is nil-safe by design, so an
|
||||
// unwired seam is not a crash — it is SILENTLY the pre-v0.191.0 behaviour, in which a per-app Tier-1
|
||||
// capture failure is a `[WARN]` line and reaches no hub channel at all. Every behavioural test in
|
||||
// internal/backup injects its own callback and passes with the production wiring gone, which is
|
||||
// exactly the hole this closes. THIS PROJECT'S COUNT OF "BUILT BUT NEVER WIRED" REACHES FIVE WITH
|
||||
// R-158 — the defect being fixed here IS an instance of it.
|
||||
func TestMainWiresTheUnitCaptureAlert(t *testing.T) {
|
||||
names := callsInMain(t, mainBody(t))
|
||||
|
||||
if indexOfCall(names, "SetUnitNotify") < 0 {
|
||||
t.Fatal("func main() no longer calls backupMgr.SetUnitNotify — a per-app recovery-unit " +
|
||||
"capture failure would reach no hub channel, which is R-158 un-fixed (the seam built " +
|
||||
"and left disconnected, for the fifth time in this project)")
|
||||
}
|
||||
if indexOfCall(names, "NotifyRecoveryUnitCaptureFailed") < 0 {
|
||||
t.Fatal("main.go no longer calls NotifyRecoveryUnitCaptureFailed — the seam is wired to " +
|
||||
"something that pushes no event, which looks identical to a working alert from inside " +
|
||||
"internal/backup")
|
||||
}
|
||||
}
|
||||
|
||||
// TestMainWiresTheFillWatcher pins Part 2's seam. Three separate things can be dropped and each one
|
||||
// silently reverts the customer to "nothing warns before a disk fills": the watcher can go
|
||||
// unconstructed, its notify can go unwired (the Watcher is nil-safe), or it can never be scheduled.
|
||||
func TestMainWiresTheFillWatcher(t *testing.T) {
|
||||
body := mainBody(t)
|
||||
names := callsInMain(t, body)
|
||||
|
||||
if indexOfCall(names, "New") < 0 || !assignsIdent(body, "fillWatcher") {
|
||||
t.Fatal("func main() no longer constructs the fill watcher — nothing warns the customer " +
|
||||
"before a filesystem fills (R-167, decision D-c's customer half)")
|
||||
}
|
||||
if indexOfCall(names, "SetNotify") < 0 {
|
||||
t.Fatal("func main() no longer calls SetNotify on the fill watcher — the Watcher is nil-safe, " +
|
||||
"so it would run the checks, update its state, log, and tell the CUSTOMER nothing")
|
||||
}
|
||||
|
||||
// It must actually be scheduled: a watcher nobody calls is a watcher that never fires.
|
||||
scheduled := false
|
||||
ast.Inspect(body, func(n ast.Node) bool {
|
||||
call, ok := n.(*ast.CallExpr)
|
||||
if !ok || len(call.Args) == 0 {
|
||||
return true
|
||||
}
|
||||
sel, ok := call.Fun.(*ast.SelectorExpr)
|
||||
if !ok || (sel.Sel.Name != "Daily" && sel.Sel.Name != "Every") {
|
||||
return true
|
||||
}
|
||||
lit, ok := call.Args[0].(*ast.BasicLit)
|
||||
if ok && strings.Contains(lit.Value, "fill-watch") {
|
||||
scheduled = true
|
||||
}
|
||||
return true
|
||||
})
|
||||
if !scheduled {
|
||||
t.Fatal("the fill watcher is never registered on the scheduler — it would be constructed, " +
|
||||
"wired, and never run, which is indistinguishable from a filesystem that never fills")
|
||||
}
|
||||
}
|
||||
|
||||
// assignsIdent reports whether a block assigns to the named identifier.
|
||||
func assignsIdent(body *ast.BlockStmt, want string) bool {
|
||||
for _, n := range assignedIdentsIn(body) {
|
||||
if n == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// assignedIdentsIn returns the names assigned to in a block (plain `=` and `:=`).
|
||||
func assignedIdentsIn(body *ast.BlockStmt) []string {
|
||||
var names []string
|
||||
|
||||
@@ -33,6 +33,7 @@ import (
|
||||
cf "gitea.dooplex.hu/admin/felhom-controller/internal/cloudflare"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/crypto"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/fillwatch"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/infra"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/integrations"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/mailrelay"
|
||||
@@ -242,8 +243,17 @@ func main() {
|
||||
// constructed until ~40 lines below — and moving its construction up to suit this would be a far
|
||||
// wider change than moving one object down. It is handed to the manager (SetAppStopGuard) and to
|
||||
// the exporter later, so all three share ONE guard over ONE file.
|
||||
//
|
||||
// R-174: the starter is GATED, never the raw manager. Recover runs here, at startup — exactly
|
||||
// when an external drive may not have come back — and `Manager.StartStack` has no drive gate of
|
||||
// its own, so the un-gated version started apps onto missing drives. The gate is the SAME
|
||||
// `driveStartGate` the boot sweep uses (holder #3 of bootDriveGate), so the two cannot disagree.
|
||||
appStopGuard := backup.NewAppStopGuard(filepath.Join(cfg.Paths.DataDir, "appstop-state.json"), logger)
|
||||
appStopGuard.SetStarter(stackMgr)
|
||||
appStopGuard.SetStarter(gatedAppStopStarter{
|
||||
inner: stackMgr,
|
||||
gate: driveStartGate{mgr: stackMgr, sett: sett},
|
||||
logger: logger,
|
||||
})
|
||||
appStopRecovery := appStopGuard.Recover()
|
||||
|
||||
// --- R-166: desired-state backfill (running-only) ---
|
||||
@@ -358,8 +368,14 @@ func main() {
|
||||
// hub's allowedEventTypes + customerMessages pair changed, which is a wire change, and this
|
||||
// release ships no hub change. A controller emitting an unlisted type gets a flat 400 from
|
||||
// POST /event. Reachability is covered by TestAppStopRecoveryIsWired.
|
||||
if appStopRecovery != nil {
|
||||
// R-174: `Alarming()` and not merely `!= nil`. A recovery that ONLY refused starts is the drive
|
||||
// gate working as designed, and `backup_failed` is customer-enabled by default — reporting a
|
||||
// deliberate hold through it would email the customer "A biztonsági mentés sikertelen!" about an
|
||||
// app nothing is wrong with. The refusal is already logged at WARN with its reason.
|
||||
if appStopRecovery != nil && appStopRecovery.Alarming() {
|
||||
notifier.NotifyBackupFailed(appStopRecovery.Message(), appStopRecovery.Detail())
|
||||
} else if appStopRecovery != nil {
|
||||
logger.Printf("[WARN] [appstop] %s — not alarming: %s", appStopRecovery.Message(), appStopRecovery.Detail())
|
||||
}
|
||||
|
||||
// --- Initialize the app-email SMTP shim (mailrelay) ---
|
||||
@@ -708,6 +724,26 @@ func main() {
|
||||
backupMgr.OffsiteFailureMessage(err, dur))
|
||||
}
|
||||
})
|
||||
// R-158 / R-167 (D-c, operator half): a per-app Tier-1 recovery-unit capture failed. Until
|
||||
// v0.191.0 this was a `[WARN]` line and nothing else — /backups/apps is the page you open to
|
||||
// ask whether ONE app is backed up, and it was the one page that never said. Fires per app;
|
||||
// the capture loop continues, so three failing apps produce three events and one failing app
|
||||
// does not silence its siblings.
|
||||
//
|
||||
// OPERATOR-TIER, not backup_failed — a customer can take no action on a capture failure, and
|
||||
// backup_failed is customer-enabled by default. The space figures ride along because the
|
||||
// overwhelmingly likely cause is a full filesystem and they answer "why" without a login.
|
||||
// No controller-side cooldown: the hub owns it.
|
||||
backupMgr.SetUnitNotify(func(stackName string, err error, usage *backup.UnitSpace) {
|
||||
d := notify.RecoveryUnitFailureDetails{App: stackName, Error: err.Error()}
|
||||
if usage != nil {
|
||||
d.TargetPath, d.UsedGB, d.AvailGB = usage.Path, usage.UsedGB, usage.AvailGB
|
||||
d.TotalGB, d.UsedPercent, d.SpaceKnown = usage.TotalGB, usage.UsedPercent, true
|
||||
}
|
||||
notifier.NotifyRecoveryUnitCaptureFailed(fmt.Sprintf(
|
||||
"Recovery unit capture FAILED for %q — the app has no fresh local (Tier-1) backup, and Tier-2/Tier-3 have nothing to copy. %s. Error: %v",
|
||||
stackName, usage.String(), err), d)
|
||||
})
|
||||
// 3a: the pre-push enlargement gate blocked an app's userdata push (config+DB still saved). Edge-
|
||||
// triggered by the engine (only NEW blocks notify), so the hub's per-event-type cooldown suffices —
|
||||
// no controller-side timer (the hub owns cooldown).
|
||||
@@ -749,6 +785,49 @@ func main() {
|
||||
})
|
||||
}
|
||||
|
||||
// --- R-167 (decision D-c, customer half): warn BEFORE a filesystem fills ---
|
||||
//
|
||||
// Nothing warned before v0.191.0. The first sign of a full filesystem was a backup that did not
|
||||
// happen, and the only related signal — the healthcheck's generic `health_degraded` at 90% — looked
|
||||
// at REGISTERED STORAGE PATHS ONLY, so the docker area and the system-data area (the one holding
|
||||
// every driveless app's recovery unit) were invisible, and it never reported a free-byte figure or
|
||||
// named a drive.
|
||||
//
|
||||
// CADENCE: DAILY, at 03:30. A fill is a slow-moving quantity — the thing that fills a disk is a
|
||||
// customer's photo library or a nightly backup, not a spike — so a shorter interval buys no
|
||||
// earlier warning and only costs statfs calls. 03:30 is deliberately BEFORE the nightly app-data
|
||||
// legs (db-dump / tier2 / offbox), so a customer who is about to lose a backup to lack of space
|
||||
// hears about it while there is still a night's margin, rather than after the failure.
|
||||
// The interval is NOT a cooldown: repeats are impossible because the check is edge-triggered per
|
||||
// filesystem, and the hub owns cooldown regardless.
|
||||
fillWatcher := fillwatch.New(
|
||||
filepath.Join(cfg.Paths.DataDir, "fillwatch-state.json"), logger,
|
||||
func() []fillwatch.Target { return fillTargets(cfg, sett) },
|
||||
func(p string) *fillwatch.Usage {
|
||||
di := system.GetDiskUsage(p)
|
||||
if di == nil {
|
||||
return nil // §8.4 — unreadable is NOT full; never fabricate a zero here
|
||||
}
|
||||
return &fillwatch.Usage{
|
||||
UsedPercent: di.UsedPercent, AvailGB: di.AvailGB,
|
||||
UsedGB: di.UsedGB, TotalGB: di.TotalGB,
|
||||
}
|
||||
})
|
||||
if notifier != nil {
|
||||
fillWatcher.SetNotify(func(e fillwatch.Event) {
|
||||
// The CUSTOMER's event — deliberately not operator-only. A customer can free space,
|
||||
// delete files or add a drive, so this alert is theirs (D-c). The dynamic Hungarian
|
||||
// message carries the label and the free space; the hub has NO customerMessages entry for
|
||||
// these two types precisely so the template cannot discard them.
|
||||
notifier.PushEvent(e.Band.EventType(), e.Band.Severity(), e.Message, map[string]any{
|
||||
"path": e.Target.Path, "label": e.Target.Label,
|
||||
"used_percent": e.Usage.UsedPercent, "avail_gb": e.Usage.AvailGB,
|
||||
"total_gb": e.Usage.TotalGB, "band": e.Band.String(),
|
||||
})
|
||||
})
|
||||
}
|
||||
sched.Daily("fill-watch", "03:30", func(ctx context.Context) error { return fillWatcher.Check() })
|
||||
|
||||
// --- Central hub reporting schedule ---
|
||||
if hubPusher != nil {
|
||||
if cfg.Hub.Enabled {
|
||||
@@ -1312,7 +1391,7 @@ var bootReconcileFn = func(ctx context.Context, mgr bootrecon.StackProvider, log
|
||||
// R-171: the sweep must not start an app whose data drive is absent. Wired HERE, at the one
|
||||
// place the sweep is constructed, so there is no path that builds an ungated reconciler.
|
||||
if sm, ok := mgr.(*stacks.Manager); ok {
|
||||
r.SetDriveGate(bootDriveGate{mgr: sm, sett: bootDriveSettings})
|
||||
r.SetDriveGate(bootDriveGate{drive: driveStartGate{mgr: sm, sett: bootDriveSettings}})
|
||||
}
|
||||
return r.Run(ctx)
|
||||
}
|
||||
@@ -1352,8 +1431,7 @@ var (
|
||||
//
|
||||
// Fail-safe per bootrecon.StartGate's contract: anything that cannot be determined returns false.
|
||||
type bootDriveGate struct {
|
||||
mgr *stacks.Manager
|
||||
sett *settings.Settings
|
||||
drive driveStartGate
|
||||
}
|
||||
|
||||
func (g bootDriveGate) MayStart(stackName string) (bool, string) {
|
||||
@@ -1367,7 +1445,35 @@ func (g bootDriveGate) MayStart(stackName string) (bool, string) {
|
||||
return false, "an app-data operation is holding it — the app-stop guard restarts it when the operation ends"
|
||||
}
|
||||
}
|
||||
// 3. the drive
|
||||
// 3. the drive — the SHARED predicate, so this gate and the app-stop guard's crash recovery
|
||||
// (R-174) cannot disagree about whether an app's drive is available.
|
||||
return g.drive.MayStart(stackName)
|
||||
}
|
||||
|
||||
// driveStartGate is holder #3 of bootDriveGate, EXTRACTED so it has two callers and one
|
||||
// implementation (R-174).
|
||||
//
|
||||
// WHY IT IS SEPARATE FROM bootDriveGate RATHER THAN REUSED WHOLE. The app-stop guard's Recover needs
|
||||
// exactly this question and NOT the other two holders:
|
||||
//
|
||||
// - Holder #2 reads `bootAppStopGuard.HeldStacks()`, which during Recover is the guard's OWN
|
||||
// marker — the very stacks being recovered. Reusing bootDriveGate there would refuse every
|
||||
// recovery it was meant to perform, self-referentially.
|
||||
// - Holders #1 and #2 read package-level vars assigned in main() at the `bootDriveSettings` block,
|
||||
// which runs AFTER `appStopGuard.Recover()`. They are nil at recovery time, so a whole-gate
|
||||
// reuse would be correct only by accident of nil-safety — and would silently invert the moment
|
||||
// anyone moved either line. This project has shipped that class of accident before.
|
||||
//
|
||||
// Fail-safe per bootrecon.StartGate's contract: anything that cannot be determined returns false.
|
||||
type driveStartGate struct {
|
||||
mgr *stacks.Manager
|
||||
sett *settings.Settings
|
||||
}
|
||||
|
||||
func (g driveStartGate) MayStart(stackName string) (bool, string) {
|
||||
if g.mgr == nil {
|
||||
return false, "no stack manager wired — the drive cannot be determined"
|
||||
}
|
||||
cfg := g.mgr.LoadAppConfigByName(stackName)
|
||||
if cfg == nil {
|
||||
// CANNOT DETERMINE. A deployed app whose app.yaml will not load cannot have its drive
|
||||
@@ -1397,6 +1503,76 @@ func (g bootDriveGate) MayStart(stackName string) (bool, string) {
|
||||
return true, ""
|
||||
}
|
||||
|
||||
// gatedAppStopStarter is the app-stop guard's starter, wrapped in the drive gate (R-174).
|
||||
//
|
||||
// THE DEFECT IT CLOSES, found by review on 2026-08-02 in code shipped 2026-08-01 (v0.189.0):
|
||||
// `appStopGuard.SetStarter(stackMgr)` handed Recover the RAW stack manager, whose `StartStack` has
|
||||
// no drive gate. Recover runs at startup — precisely when an external drive may not have come back —
|
||||
// so a backup that stopped an app, followed by a power cut and a drive that did not remount, ended
|
||||
// with the app started onto a missing drive. That is R-171 one path over, and the rule is not new:
|
||||
// the API's own `startGatedByMissingDrive` already refuses this to the customer.
|
||||
//
|
||||
// The check stays HERE, per-caller, and is deliberately NOT pushed into `Manager.StartStack` — it has
|
||||
// fourteen callers and most of them legitimately start apps outside this concern.
|
||||
type gatedAppStopStarter struct {
|
||||
inner backup.AppStopStarter
|
||||
gate driveStartGate
|
||||
logger *log.Logger
|
||||
}
|
||||
|
||||
func (s gatedAppStopStarter) StartStack(name string) error {
|
||||
if ok, why := s.gate.MayStart(name); !ok {
|
||||
s.logger.Printf("[WARN] [appstop] refusing to restart %q after an interrupted operation: %s", name, why)
|
||||
return fmt.Errorf("%w: %s", backup.ErrStartRefused, why)
|
||||
}
|
||||
return s.inner.StartStack(name)
|
||||
}
|
||||
|
||||
// fillTargets is §8.1's watch list: the app-data volume, the system-data volume, and every
|
||||
// registered drive. Resolved at CHECK time, not at startup, so a drive added or decommissioned
|
||||
// between checks is picked up without a controller restart.
|
||||
//
|
||||
// WHY THESE THREE AND NOT ONLY THE REGISTERED DRIVES — which is all the healthcheck ever looked at:
|
||||
//
|
||||
// - the APP-DATA volume (`mp0`, `/var/lib/docker`) holds every app's live named volumes;
|
||||
// - the SYSTEM-DATA volume (`mp1`, `/mnt/sys_drive`) holds the retained recovery unit of every
|
||||
// DRIVELESS app (architecture/07-backup-architecture.md §7.5) and is the smaller of the two at
|
||||
// 20 G against 50 G — the very mismatch R-165 exists to remove. It is also the filesystem whose
|
||||
// silent exhaustion R-158 measured;
|
||||
// - the registered DRIVES hold the customer's own data.
|
||||
//
|
||||
// De-duplicated by path: on a box where a drive is not a separate mount these collapse, and warning
|
||||
// twice about one filesystem is exactly the noise the per-filesystem rule exists to prevent. A
|
||||
// DECOMMISSIONED drive is dropped — it is deliberately out of service, and its fill is not news. A
|
||||
// DISCONNECTED one is kept in the list but will read as unreadable and be skipped by §8.4, which is
|
||||
// the correct outcome: its absence already has its own alert.
|
||||
func fillTargets(cfg *config.Config, sett *settings.Settings) []fillwatch.Target {
|
||||
seen := make(map[string]bool)
|
||||
var out []fillwatch.Target
|
||||
add := func(path, label string) {
|
||||
if path == "" || seen[path] {
|
||||
return
|
||||
}
|
||||
seen[path] = true
|
||||
out = append(out, fillwatch.Target{Path: path, Label: label})
|
||||
}
|
||||
|
||||
// The docker data-root as seen from inside the guest, and the system-data area.
|
||||
add(system.DockerVolumePath, "Alkalmazások területe")
|
||||
if cfg != nil {
|
||||
add(cfg.Paths.SystemDataPath, "Rendszer- és mentési terület")
|
||||
}
|
||||
if sett != nil {
|
||||
for _, sp := range sett.GetStoragePaths() {
|
||||
if sp.Decommissioned {
|
||||
continue
|
||||
}
|
||||
add(sp.Path, sp.Label)
|
||||
}
|
||||
}
|
||||
return fillwatch.SortTargets(out)
|
||||
}
|
||||
|
||||
// bootFleetSample is a comparable snapshot of one deployed app — name, state and container count,
|
||||
// per §8.1. Container count is in it deliberately: a stack can go from 3 containers to 0 without its
|
||||
// aggregate state changing, and that IS the boot still moving.
|
||||
|
||||
Reference in New Issue
Block a user