9832760027
gates / gates (push) Successful in 11s
R-329. NotifyAppStartFailures emitted severity "warn". The hub accepts exactly
{info, warning, error, critical} and silently coerces anything else to "info",
which severityNotifies then drops BEFORE both legs. Banner shown, event stored,
POST 200, no mail sent. One word.
This is the second time: DiskAlertKind.Severity emitted "warn" until v0.215.0
and its own comment records that every warning-level disk alert went to nobody.
A comment recorded the lesson and nothing enforced it. The guard is now an AST
walk over the whole controller - grep cannot work here, since "warn" appears
legitimately nine times as a healthcheck status vocabulary.
The sweep found exactly one bad severity. Its limits are stated: the walk cannot
follow a variable, so all six dynamic call sites are registered by name with the
values each can take, and a new one fails the test. Two of the six were found by
the guard, not by the hand sweep before it.
Also pinned: fillwatch.Band.Severity() returns "" for BandOK, which would vanish
the same way. It is unreachable because Check() notifies only on escalation -
but that safety lives in a different function from the one that looks unsafe, so
the test asserts the consequence rather than the mapping.
app_start_failed gains a customer toggle, DEFAULT OFF, per operator ruling. The
operator is mailed either way: processOperator never consults customer prefs.
It is deliberately NOT in operatorOnlyEvents, which would make the toggle a lie.
R-386. classifyRunStates decided "the customer stopped this" from the STATE, so
every stopped stack was assumed deliberate. Measured on demo-hp: privatebin
stopped out of band, nine scans, zero events, zero banner - while the comment
beside it claimed an out-of-band stop still alerts.
DesiredState already records the answer and has exactly one writer. Stopped ->
no alarm; Running -> alarm; absent -> UNKNOWN, keep today's behaviour AND say
so. Absent stays silent deliberately: reading it as "nobody asked" would email
about every app anyone ever stopped, fleet-wide, on the first cycle after
upgrade. The gap is bounded not silent - IntentUnknown is set and the names are
logged at INFO on the heartbeat cadence. failedRestart still lifts a Stopped
intent, or F-CRIT-1 re-opens. No new DesiredState writer.
Two settings toggles each governed two alarms. "Lemez figyelmeztetes (90%+)"
also wrote disk_critical, the drive-is-FAILING alarm. Now four honest toggles;
12 became 15. A no-op save stores the existing slice verbatim, so byte identity
is by construction - without that guard the defaults case reorders, which the
red-proof caught.
Test count 1504 -> 1522. Five red-proofs, five seen failing; one passed first
time and is reported - that mutation was inert, not the test weak.
213 lines
8.7 KiB
Go
213 lines
8.7 KiB
Go
package notify
|
|
|
|
import (
|
|
"fmt"
|
|
"go/ast"
|
|
"go/parser"
|
|
"go/token"
|
|
"io/fs"
|
|
"path/filepath"
|
|
"sort"
|
|
"strconv"
|
|
"strings"
|
|
"testing"
|
|
)
|
|
|
|
// R-329 — the hub's severity vocabulary, pinned by an AST walk over the WHOLE controller.
|
|
//
|
|
// THE DEFECT THIS EXISTS TO KILL. `NotifyAppStartFailures` emitted severity `"warn"`. The hub accepts
|
|
// only {info, warning, error, critical} and **silently coerces** anything else to `info` at ingest,
|
|
// after which `severityNotifies` drops it and NEITHER the operator nor the customer leg runs. So the
|
|
// dashboard showed the app down, the hub stored the event, and no mail left the building.
|
|
//
|
|
// **This is the SECOND time.** `DiskAlertKind.Severity`'s own doc comment records that it emitted
|
|
// `"warn"` until v0.215.0, and that every Figyelmeztetés-level disk alert the product ever produced
|
|
// was delivered to nobody. A comment recorded the lesson; nothing enforced it; it happened again.
|
|
// **A comment is not a guard** — this is the guard.
|
|
//
|
|
// THE LAYER. This sits at the EMITTER, which is where the defect lives: the hub's coercion is
|
|
// deliberate and correct (a lost alarm is worse than a mis-routed one), so nothing downstream can
|
|
// detect a bad word — by design, the bad word ceases to exist at the hub's front door. The only
|
|
// place the mistake is still visible is where it is written.
|
|
//
|
|
// WHY AN AST WALK AND NOT grep. `strings.Contains` over source cannot tell an emitted severity from
|
|
// the word "warn" in a comment, a healthcheck status vocabulary (`internal/monitor`,
|
|
// `internal/selftest` both legitimately use "warn"), or a `case "warn":` in an unrelated switch. The
|
|
// 2026-08-23 sweep found nine such hits and exactly one real defect.
|
|
//
|
|
// RED-PROOF (observed, see REPORT.md): put `"warn"` back at notifier.go's app_start_failed emit and
|
|
// this fails naming the file, line, call and value.
|
|
|
|
// hubSeverityVocabulary is the hub's set, verbatim. Sources, both named so a reader can check rather
|
|
// than trust: felhom.eu/hub/internal/api/handler.go (the ingest switch) and
|
|
// felhom.eu/hub/internal/notify/dispatcher.go (severityNotifies).
|
|
var hubSeverityVocabulary = map[string]bool{
|
|
"info": true, "warning": true, "error": true, "critical": true,
|
|
}
|
|
|
|
// severityArg names the functions that take a hub severity, and which argument it is.
|
|
var severityArg = map[string]int{
|
|
"emit": 1, // (eventType, severity, message, details)
|
|
"PushEvent": 1, // (eventType, severity, message, details)
|
|
}
|
|
|
|
// knownDynamicSeveritySites are the call sites that pass a VARIABLE rather than a literal, so this
|
|
// walk cannot check their value. Each was traced by hand on 2026-08-23 and every reachable value was
|
|
// in the vocabulary. **They are listed so that a NEW dynamic site cannot appear unnoticed** — the
|
|
// honest limit of an AST check is that it cannot follow a variable, and an unlisted limit is not a
|
|
// limit, it is a hole.
|
|
var knownDynamicSeveritySites = map[string]string{
|
|
"internal/notify/notifier.go:emit": `pass-through of its own severity parameter to PushEvent — the value is checked at emit's CALLERS, above`,
|
|
"internal/notify/notifier.go:NotifyControllerUpdated": `local var: "info", or "error" when the update failed`,
|
|
"internal/notify/notifier.go:NotifyDRCompleted": `local var: "info", or "warning" when failCount > 0`,
|
|
"internal/notify/notifier.go:NotifyAgentChannelDown": `internal/channelhealth's classifier — every severity literal in checker.go is "warning" or "error"`,
|
|
"internal/notify/notifier.go:NotifyDiskHealthDegraded": `DiskAlertKind.Severity() — pinned behaviourally by TestR329_DiskAlertSeveritiesStillSatisfyTheContract`,
|
|
"cmd/controller/main.go:main": `fillwatch.Band.Severity() in the SetNotify closure — pinned by TestR329_FillwatchNeverEmitsTheEmptySeverity`,
|
|
}
|
|
|
|
func walkControllerFiles(t *testing.T, fn func(path string, fset *token.FileSet, f *ast.File)) {
|
|
t.Helper()
|
|
root, err := filepath.Abs("../..") // the controller module root
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
fset := token.NewFileSet()
|
|
err = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
|
|
if err != nil {
|
|
return err
|
|
}
|
|
if d.IsDir() {
|
|
if d.Name() == "vendor" || d.Name() == ".git" || d.Name() == "node_modules" {
|
|
return filepath.SkipDir
|
|
}
|
|
return nil
|
|
}
|
|
if !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") {
|
|
return nil
|
|
}
|
|
f, perr := parser.ParseFile(fset, path, nil, parser.ParseComments)
|
|
if perr != nil {
|
|
return fmt.Errorf("parse %s: %w", path, perr)
|
|
}
|
|
rel, _ := filepath.Rel(root, path)
|
|
fn(rel, fset, f)
|
|
return nil
|
|
})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
}
|
|
|
|
// calleeName returns the bare function name for `foo(...)` and `x.foo(...)`.
|
|
func calleeName(call *ast.CallExpr) string {
|
|
switch fn := call.Fun.(type) {
|
|
case *ast.Ident:
|
|
return fn.Name
|
|
case *ast.SelectorExpr:
|
|
return fn.Sel.Name
|
|
}
|
|
return ""
|
|
}
|
|
|
|
func TestR329_EveryEmittedSeverityIsInTheHubVocabulary(t *testing.T) {
|
|
var checked int
|
|
var dynamic []string
|
|
|
|
walkControllerFiles(t, func(rel string, fset *token.FileSet, f *ast.File) {
|
|
// Attribute each call to the FuncDecl that encloses it, by walking declarations rather than
|
|
// trusting Inspect's visit order — a closure inside main() must read as "main", and a call in
|
|
// a var block must not inherit the previous function's name.
|
|
for _, decl := range f.Decls {
|
|
fd, isFunc := decl.(*ast.FuncDecl)
|
|
enclosing := "(file scope)"
|
|
var scope ast.Node = decl
|
|
if isFunc {
|
|
enclosing = fd.Name.Name
|
|
scope = fd
|
|
}
|
|
inspectForSeverity(t, rel, enclosing, scope, fset, &checked, &dynamic)
|
|
}
|
|
})
|
|
|
|
// The walk must actually have found call sites — a guard that silently examines nothing is the
|
|
// "instrument that can drop results" failure, and it would pass forever.
|
|
if checked < 15 {
|
|
t.Fatalf("only %d severity literals examined — the AST walk is not reaching the call sites; "+
|
|
"fix this test before trusting a green from it", checked)
|
|
}
|
|
t.Logf("checked %d severity literals across the controller", checked)
|
|
|
|
// The dynamic sites are a stated limit, and the list is the mechanism that keeps it stated.
|
|
sort.Strings(dynamic)
|
|
seen := map[string]bool{}
|
|
for _, d := range dynamic {
|
|
if seen[d] {
|
|
continue
|
|
}
|
|
seen[d] = true
|
|
if _, known := knownDynamicSeveritySites[d]; !known {
|
|
t.Errorf("NEW dynamic severity call site %q — this walk cannot check a variable's value. "+
|
|
"Trace every value it can take; if all are in the hub vocabulary, add it to "+
|
|
"knownDynamicSeveritySites with the reason. Do not delete this check.", d)
|
|
}
|
|
}
|
|
for known := range knownDynamicSeveritySites {
|
|
if !seen[known] {
|
|
t.Errorf("registered dynamic site %q no longer exists — remove it from "+
|
|
"knownDynamicSeveritySites so the register stays honest", known)
|
|
}
|
|
}
|
|
}
|
|
|
|
// The vocabulary itself must match the hub's, and DiskAlertKind.Severity — the function whose doc
|
|
// comment records the first occurrence — must still satisfy it.
|
|
func TestR329_DiskAlertSeveritiesStillSatisfyTheContract(t *testing.T) {
|
|
for _, k := range []DiskAlertKind{
|
|
DiskAlertWarn, DiskAlertFailSelfReported, DiskAlertFailSectors,
|
|
DiskAlertFailTemperature, DiskAlertFailWorsened,
|
|
} {
|
|
if got := k.Severity(); !hubSeverityVocabulary[got] {
|
|
t.Errorf("DiskAlertKind(%d).Severity() = %q, not in the hub vocabulary", k, got)
|
|
}
|
|
}
|
|
// And the one that regressed: a warning must be a *warning*, not "warn" and not "info".
|
|
if got := DiskAlertWarn.Severity(); got != "warning" {
|
|
t.Errorf("DiskAlertWarn.Severity() = %q, want \"warning\"", got)
|
|
}
|
|
}
|
|
|
|
// inspectForSeverity checks every severity-taking call inside one declaration.
|
|
func inspectForSeverity(t *testing.T, rel, enclosing string, scope ast.Node, fset *token.FileSet, checked *int, dynamic *[]string) {
|
|
t.Helper()
|
|
ast.Inspect(scope, func(n ast.Node) bool {
|
|
call, ok := n.(*ast.CallExpr)
|
|
if !ok {
|
|
return true
|
|
}
|
|
idx, wanted := severityArg[calleeName(call)]
|
|
if !wanted || len(call.Args) <= idx {
|
|
return true
|
|
}
|
|
lit, isLit := call.Args[idx].(*ast.BasicLit)
|
|
if !isLit || lit.Kind != token.STRING {
|
|
*dynamic = append(*dynamic, rel+":"+enclosing)
|
|
return true
|
|
}
|
|
val, err := strconv.Unquote(lit.Value)
|
|
if err != nil {
|
|
t.Errorf("%s: unparseable severity literal %s", fset.Position(lit.Pos()), lit.Value)
|
|
return true
|
|
}
|
|
*checked++
|
|
if !hubSeverityVocabulary[val] {
|
|
t.Errorf("%s: %s(...) emits severity %q, which is NOT in the hub's vocabulary "+
|
|
"{info, warning, error, critical}.\n"+
|
|
" The hub COERCES it to \"info\" at ingest, SILENTLY, and severityNotifies then "+
|
|
"drops \"info\" — so this event is stored and emailed to NOBODY, on either leg.\n"+
|
|
" This exact mistake shipped once before (DiskAlertKind.Severity, fixed v0.215.0).",
|
|
fset.Position(lit.Pos()), calleeName(call), val)
|
|
}
|
|
return true
|
|
})
|
|
}
|