Files
felhom-controller/controller/internal/notify/r329_severity_contract_test.go
T
admin 9832760027
gates / gates (push) Successful in 11s
v0.223.0: the app-down alarm reached nobody (R-329), and the stop nobody heard (R-386)
R-329. NotifyAppStartFailures emitted severity "warn". The hub accepts exactly
{info, warning, error, critical} and silently coerces anything else to "info",
which severityNotifies then drops BEFORE both legs. Banner shown, event stored,
POST 200, no mail sent. One word.

This is the second time: DiskAlertKind.Severity emitted "warn" until v0.215.0
and its own comment records that every warning-level disk alert went to nobody.
A comment recorded the lesson and nothing enforced it. The guard is now an AST
walk over the whole controller - grep cannot work here, since "warn" appears
legitimately nine times as a healthcheck status vocabulary.

The sweep found exactly one bad severity. Its limits are stated: the walk cannot
follow a variable, so all six dynamic call sites are registered by name with the
values each can take, and a new one fails the test. Two of the six were found by
the guard, not by the hand sweep before it.

Also pinned: fillwatch.Band.Severity() returns "" for BandOK, which would vanish
the same way. It is unreachable because Check() notifies only on escalation -
but that safety lives in a different function from the one that looks unsafe, so
the test asserts the consequence rather than the mapping.

app_start_failed gains a customer toggle, DEFAULT OFF, per operator ruling. The
operator is mailed either way: processOperator never consults customer prefs.
It is deliberately NOT in operatorOnlyEvents, which would make the toggle a lie.

R-386. classifyRunStates decided "the customer stopped this" from the STATE, so
every stopped stack was assumed deliberate. Measured on demo-hp: privatebin
stopped out of band, nine scans, zero events, zero banner - while the comment
beside it claimed an out-of-band stop still alerts.

DesiredState already records the answer and has exactly one writer. Stopped ->
no alarm; Running -> alarm; absent -> UNKNOWN, keep today's behaviour AND say
so. Absent stays silent deliberately: reading it as "nobody asked" would email
about every app anyone ever stopped, fleet-wide, on the first cycle after
upgrade. The gap is bounded not silent - IntentUnknown is set and the names are
logged at INFO on the heartbeat cadence. failedRestart still lifts a Stopped
intent, or F-CRIT-1 re-opens. No new DesiredState writer.

Two settings toggles each governed two alarms. "Lemez figyelmeztetes (90%+)"
also wrote disk_critical, the drive-is-FAILING alarm. Now four honest toggles;
12 became 15. A no-op save stores the existing slice verbatim, so byte identity
is by construction - without that guard the defaults case reorders, which the
red-proof caught.

Test count 1504 -> 1522. Five red-proofs, five seen failing; one passed first
time and is reported - that mutation was inert, not the test weak.
2026-08-23 11:21:06 +02:00

213 lines
8.7 KiB
Go

package notify
import (
"fmt"
"go/ast"
"go/parser"
"go/token"
"io/fs"
"path/filepath"
"sort"
"strconv"
"strings"
"testing"
)
// R-329 — the hub's severity vocabulary, pinned by an AST walk over the WHOLE controller.
//
// THE DEFECT THIS EXISTS TO KILL. `NotifyAppStartFailures` emitted severity `"warn"`. The hub accepts
// only {info, warning, error, critical} and **silently coerces** anything else to `info` at ingest,
// after which `severityNotifies` drops it and NEITHER the operator nor the customer leg runs. So the
// dashboard showed the app down, the hub stored the event, and no mail left the building.
//
// **This is the SECOND time.** `DiskAlertKind.Severity`'s own doc comment records that it emitted
// `"warn"` until v0.215.0, and that every Figyelmeztetés-level disk alert the product ever produced
// was delivered to nobody. A comment recorded the lesson; nothing enforced it; it happened again.
// **A comment is not a guard** — this is the guard.
//
// THE LAYER. This sits at the EMITTER, which is where the defect lives: the hub's coercion is
// deliberate and correct (a lost alarm is worse than a mis-routed one), so nothing downstream can
// detect a bad word — by design, the bad word ceases to exist at the hub's front door. The only
// place the mistake is still visible is where it is written.
//
// WHY AN AST WALK AND NOT grep. `strings.Contains` over source cannot tell an emitted severity from
// the word "warn" in a comment, a healthcheck status vocabulary (`internal/monitor`,
// `internal/selftest` both legitimately use "warn"), or a `case "warn":` in an unrelated switch. The
// 2026-08-23 sweep found nine such hits and exactly one real defect.
//
// RED-PROOF (observed, see REPORT.md): put `"warn"` back at notifier.go's app_start_failed emit and
// this fails naming the file, line, call and value.
// hubSeverityVocabulary is the hub's set, verbatim. Sources, both named so a reader can check rather
// than trust: felhom.eu/hub/internal/api/handler.go (the ingest switch) and
// felhom.eu/hub/internal/notify/dispatcher.go (severityNotifies).
var hubSeverityVocabulary = map[string]bool{
"info": true, "warning": true, "error": true, "critical": true,
}
// severityArg names the functions that take a hub severity, and which argument it is.
var severityArg = map[string]int{
"emit": 1, // (eventType, severity, message, details)
"PushEvent": 1, // (eventType, severity, message, details)
}
// knownDynamicSeveritySites are the call sites that pass a VARIABLE rather than a literal, so this
// walk cannot check their value. Each was traced by hand on 2026-08-23 and every reachable value was
// in the vocabulary. **They are listed so that a NEW dynamic site cannot appear unnoticed** — the
// honest limit of an AST check is that it cannot follow a variable, and an unlisted limit is not a
// limit, it is a hole.
var knownDynamicSeveritySites = map[string]string{
"internal/notify/notifier.go:emit": `pass-through of its own severity parameter to PushEvent — the value is checked at emit's CALLERS, above`,
"internal/notify/notifier.go:NotifyControllerUpdated": `local var: "info", or "error" when the update failed`,
"internal/notify/notifier.go:NotifyDRCompleted": `local var: "info", or "warning" when failCount > 0`,
"internal/notify/notifier.go:NotifyAgentChannelDown": `internal/channelhealth's classifier — every severity literal in checker.go is "warning" or "error"`,
"internal/notify/notifier.go:NotifyDiskHealthDegraded": `DiskAlertKind.Severity() — pinned behaviourally by TestR329_DiskAlertSeveritiesStillSatisfyTheContract`,
"cmd/controller/main.go:main": `fillwatch.Band.Severity() in the SetNotify closure — pinned by TestR329_FillwatchNeverEmitsTheEmptySeverity`,
}
func walkControllerFiles(t *testing.T, fn func(path string, fset *token.FileSet, f *ast.File)) {
t.Helper()
root, err := filepath.Abs("../..") // the controller module root
if err != nil {
t.Fatal(err)
}
fset := token.NewFileSet()
err = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
if d.Name() == "vendor" || d.Name() == ".git" || d.Name() == "node_modules" {
return filepath.SkipDir
}
return nil
}
if !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") {
return nil
}
f, perr := parser.ParseFile(fset, path, nil, parser.ParseComments)
if perr != nil {
return fmt.Errorf("parse %s: %w", path, perr)
}
rel, _ := filepath.Rel(root, path)
fn(rel, fset, f)
return nil
})
if err != nil {
t.Fatal(err)
}
}
// calleeName returns the bare function name for `foo(...)` and `x.foo(...)`.
func calleeName(call *ast.CallExpr) string {
switch fn := call.Fun.(type) {
case *ast.Ident:
return fn.Name
case *ast.SelectorExpr:
return fn.Sel.Name
}
return ""
}
func TestR329_EveryEmittedSeverityIsInTheHubVocabulary(t *testing.T) {
var checked int
var dynamic []string
walkControllerFiles(t, func(rel string, fset *token.FileSet, f *ast.File) {
// Attribute each call to the FuncDecl that encloses it, by walking declarations rather than
// trusting Inspect's visit order — a closure inside main() must read as "main", and a call in
// a var block must not inherit the previous function's name.
for _, decl := range f.Decls {
fd, isFunc := decl.(*ast.FuncDecl)
enclosing := "(file scope)"
var scope ast.Node = decl
if isFunc {
enclosing = fd.Name.Name
scope = fd
}
inspectForSeverity(t, rel, enclosing, scope, fset, &checked, &dynamic)
}
})
// The walk must actually have found call sites — a guard that silently examines nothing is the
// "instrument that can drop results" failure, and it would pass forever.
if checked < 15 {
t.Fatalf("only %d severity literals examined — the AST walk is not reaching the call sites; "+
"fix this test before trusting a green from it", checked)
}
t.Logf("checked %d severity literals across the controller", checked)
// The dynamic sites are a stated limit, and the list is the mechanism that keeps it stated.
sort.Strings(dynamic)
seen := map[string]bool{}
for _, d := range dynamic {
if seen[d] {
continue
}
seen[d] = true
if _, known := knownDynamicSeveritySites[d]; !known {
t.Errorf("NEW dynamic severity call site %q — this walk cannot check a variable's value. "+
"Trace every value it can take; if all are in the hub vocabulary, add it to "+
"knownDynamicSeveritySites with the reason. Do not delete this check.", d)
}
}
for known := range knownDynamicSeveritySites {
if !seen[known] {
t.Errorf("registered dynamic site %q no longer exists — remove it from "+
"knownDynamicSeveritySites so the register stays honest", known)
}
}
}
// The vocabulary itself must match the hub's, and DiskAlertKind.Severity — the function whose doc
// comment records the first occurrence — must still satisfy it.
func TestR329_DiskAlertSeveritiesStillSatisfyTheContract(t *testing.T) {
for _, k := range []DiskAlertKind{
DiskAlertWarn, DiskAlertFailSelfReported, DiskAlertFailSectors,
DiskAlertFailTemperature, DiskAlertFailWorsened,
} {
if got := k.Severity(); !hubSeverityVocabulary[got] {
t.Errorf("DiskAlertKind(%d).Severity() = %q, not in the hub vocabulary", k, got)
}
}
// And the one that regressed: a warning must be a *warning*, not "warn" and not "info".
if got := DiskAlertWarn.Severity(); got != "warning" {
t.Errorf("DiskAlertWarn.Severity() = %q, want \"warning\"", got)
}
}
// inspectForSeverity checks every severity-taking call inside one declaration.
func inspectForSeverity(t *testing.T, rel, enclosing string, scope ast.Node, fset *token.FileSet, checked *int, dynamic *[]string) {
t.Helper()
ast.Inspect(scope, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
idx, wanted := severityArg[calleeName(call)]
if !wanted || len(call.Args) <= idx {
return true
}
lit, isLit := call.Args[idx].(*ast.BasicLit)
if !isLit || lit.Kind != token.STRING {
*dynamic = append(*dynamic, rel+":"+enclosing)
return true
}
val, err := strconv.Unquote(lit.Value)
if err != nil {
t.Errorf("%s: unparseable severity literal %s", fset.Position(lit.Pos()), lit.Value)
return true
}
*checked++
if !hubSeverityVocabulary[val] {
t.Errorf("%s: %s(...) emits severity %q, which is NOT in the hub's vocabulary "+
"{info, warning, error, critical}.\n"+
" The hub COERCES it to \"info\" at ingest, SILENTLY, and severityNotifies then "+
"drops \"info\" — so this event is stored and emailed to NOBODY, on either leg.\n"+
" This exact mistake shipped once before (DiskAlertKind.Severity, fixed v0.215.0).",
fset.Position(lit.Pos()), calleeName(call), val)
}
return true
})
}