92cebb8c95
gates / gates (push) Successful in 11s
Measured live on demo-hp 2026-08-30 (controller 0.223.0): the nightly db-dump
and offbox-backup legs stop each stack ~13s to tar its volumes while the
deadapp-check job scans every 30s, so the scan caught whichever stack was
mid-cycle and pushed app_start_failed to the customer. 61 e-mails about apps
that were never broken.
The defect is not a missing mechanism. quiesce/suppress.go solved exactly this
in v0.179.0 and works -- but classifyRunStates read only the quiesce loop's set,
and that loop covers the WHOLE-GUEST backup. The per-app legs stop stacks
through Manager.DumpAppVolumesSafe, which registered with nothing. Two
mechanisms stop apps on purpose; only one told the alarm. Fifth instance of the
"seam built but never wired" class, and the first where the unwired half was a
consumer.
The suppression now rides AppStopGuard, which already brackets every deliberate
stop in the product (Begin before the stop, End after a successful restart) at
all three call sites, and which main.go hands as ONE object to the backup
manager and the exporter. scanDeployedAppRunStates takes the union of both sets.
All three per-app stop paths are covered, not only the reported nightly one.
It cannot latch -- End() runs only on a restart that SUCCEEDED, so unlike the
quiesce loop an open-ended hold is a real hazard here:
1. ReleaseFailed drops the entry IMMEDIATELY on a restart that broke, wired at
every failure path, so the app alarms on the next scan;
2. Begin REPLACES the set (one marker file = one operation);
3. appStopMaxHold (6h) caps a hold nothing released, logged at WARN.
Grace is 180s, deliberately quiesce's own constant and derivation. Suppression
is NOT persisted: after a crash the guard holds nothing and a down app must
alarm. ReleaseFailed keeps the durable crash marker; a test pins that.
Three companion red-proofs, each printing the pre-fix value (REPORT.md section 5):
- drop markStopped from Begin -> "suppressed at stop = map[]"
- drop ReleaseFailed from the dump -> "map[bookstack:true] after a restart that FAILED"
- pass nil instead of appStopGuard -> the AST wiring test fails
The third is load-bearing: the component was never the broken part, so a suite
that only injected it would have been green against the shipped defect.
Green gate clean: go build + go vet + go test ./... -- 28 packages, rc 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LB8FmJaGd2cyjvy6dbEjpM
116 lines
4.9 KiB
Go
116 lines
4.9 KiB
Go
package main
|
|
|
|
import (
|
|
"go/ast"
|
|
"io"
|
|
"log"
|
|
"testing"
|
|
"time"
|
|
|
|
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
|
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
|
)
|
|
|
|
// ── R-330 — the per-app backup's suppression must actually REACH the alarm ───────────────────────
|
|
//
|
|
// Measured on demo-hp 2026-08-30 (controller 0.223.0): the nightly `db-dump` and `offbox-backup` legs
|
|
// stop each stack for ~13 s to tar its volumes, the `deadapp-check` job scans every 30 s, and the
|
|
// customer got `app_start_failed — "Telepített alkalmazás nem fut: <App>"` for apps that were never
|
|
// broken. Sixty-one e-mails.
|
|
//
|
|
// THE SHAPE OF THE ORIGINAL DEFECT IS WHY THESE TESTS EXIST. `quiesce/suppress.go` already solved
|
|
// this problem, correctly, in v0.179.0 — and the alarm still fired, because `classifyRunStates` read
|
|
// only the QUIESCE loop's set while a SECOND mechanism (the AppStopGuard's per-app operations) was
|
|
// stopping apps with nothing telling the alarm. A component that works and is not consulted is this
|
|
// project's "seam built but never wired" class, hit five times now. So one test asserts the
|
|
// CONSEQUENCE (a held app is not reported down) and one asserts the WIRING (main.go passes the
|
|
// guard), because in this defect the component was never the broken part.
|
|
|
|
// TestHeldAppIsNotReportedDown is the consequence: whatever produces the suppression set, an app the
|
|
// backup is holding must not reach the notifier's Down-set or the dashboard banner.
|
|
func TestHeldAppIsNotReportedDown(t *testing.T) {
|
|
g := backup.NewAppStopGuard(t.TempDir()+"/appstop.json", log.New(io.Discard, "", 0))
|
|
if err := g.Begin("volume-dump:bookstack", backup.ReasonVolumeDump, []string{"bookstack"}); err != nil {
|
|
t.Fatalf("Begin: %v", err)
|
|
}
|
|
|
|
sts := []stacks.Stack{
|
|
stack("bookstack", stacks.StateStopped, true, false), // held by the volume dump
|
|
stack("romm", stacks.StateExited, true, false), // genuinely broken, must still alarm
|
|
}
|
|
// The union is exactly what scanDeployedAppRunStates builds. The nil first argument is the normal
|
|
// state on a box with no whole-guest backup running — quiesce suppresses nothing, and the app-stop
|
|
// guard's set has to carry the whole answer on its own.
|
|
dead, states := classifyRunStates(sts, unionSuppressed(nil, g.SuppressedStacks()), nil, time.Now())
|
|
|
|
down := downByName(states)
|
|
if down["bookstack"] {
|
|
t.Fatal("the app the backup is holding stopped was reported DOWN — this is the false " +
|
|
"`app_start_failed` e-mail the customer received every night")
|
|
}
|
|
if !down["romm"] {
|
|
t.Fatal("a genuinely exited app stopped alarming — the fix silenced a real fault, which is " +
|
|
"the over-correction (F-CRIT-1 / R-88 Scenario D) it must never make")
|
|
}
|
|
if deadNames(dead)["bookstack"] {
|
|
t.Fatal("the held app still reached the dashboard dead-app banner")
|
|
}
|
|
if !deadNames(dead)["romm"] {
|
|
t.Fatal("the genuinely exited app vanished from the dead-app banner")
|
|
}
|
|
}
|
|
|
|
func TestUnionSuppressed_KeepsBothMechanisms(t *testing.T) {
|
|
// Two independent mechanisms stop apps on purpose. Dropping either set re-opens one of the two
|
|
// false-alarm paths, and the bug shipped because only one was being read.
|
|
got := unionSuppressed(map[string]bool{"quiesced-app": true}, map[string]bool{"dumped-app": true})
|
|
if !got["quiesced-app"] || !got["dumped-app"] {
|
|
t.Fatalf("union = %v, want both the quiesce loop's and the app-stop guard's stacks", got)
|
|
}
|
|
if got := unionSuppressed(nil, nil); len(got) != 0 {
|
|
t.Fatalf("union of two empty sets = %v, want empty", got)
|
|
}
|
|
// Neither input may be mutated: both callers hold live maps that other code reads.
|
|
a := map[string]bool{"a": true}
|
|
b := map[string]bool{"b": true}
|
|
unionSuppressed(a, b)
|
|
if len(a) != 1 || len(b) != 1 {
|
|
t.Fatalf("unionSuppressed mutated an input: a=%v b=%v", a, b)
|
|
}
|
|
}
|
|
|
|
// TestScanDeployedAppRunStatesIsGivenTheAppStopGuard walks main.go's AST. A substring search is not
|
|
// enough — the sibling wiring tests in this package record why at first hand: a commented-out call
|
|
// still satisfies strings.Contains, so the text version passed the very red-proof it existed to fail.
|
|
func TestScanDeployedAppRunStatesIsGivenTheAppStopGuard(t *testing.T) {
|
|
body := mainBody(t)
|
|
|
|
var found, withGuard bool
|
|
ast.Inspect(body, func(n ast.Node) bool {
|
|
call, ok := n.(*ast.CallExpr)
|
|
if !ok {
|
|
return true
|
|
}
|
|
id, ok := call.Fun.(*ast.Ident)
|
|
if !ok || id.Name != "scanDeployedAppRunStates" {
|
|
return true
|
|
}
|
|
found = true
|
|
for _, arg := range call.Args {
|
|
if a, ok := arg.(*ast.Ident); ok && a.Name == "appStopGuard" {
|
|
withGuard = true
|
|
}
|
|
}
|
|
return true
|
|
})
|
|
|
|
if !found {
|
|
t.Fatal("func main() never calls scanDeployedAppRunStates — the dead-app scan is not wired at all")
|
|
}
|
|
if !withGuard {
|
|
t.Fatal("scanDeployedAppRunStates is called WITHOUT appStopGuard: the suppression set exists " +
|
|
"but the alarm never reads it, which is precisely how R-330 shipped while R-97b's " +
|
|
"identical mechanism sat working three lines away")
|
|
}
|
|
}
|