v0.222.0: ask whether a supervised member is DEAD before whether one is UNHEALTHY (R-384), and stop promising an undo copy nobody looked for (R-383)
gates / gates (push) Successful in 11s
gates / gates (push) Successful in 11s
R-384. aggregateState returned StateUnhealthy the moment unhealthy > 0, and the R-51 mixed-case block that asks "is a supervised member dead?" sat below it. A two-container app whose database exits goes unhealthy BECAUSE it cannot reach that database - so the symptom the dead database causes was what suppressed the alarm for it. unhealthy is not a down state, so classifyRunStates never marked the app down and app_start_failed never fired. Measured live on demo-hp 2026-08-22: bookstack-db stopped at 21:27:01 and the F-OBS heartbeat printed "0 currently down" throughout. R-51's 18-hour immich failure, back through a different door. Two things moved, and either alone leaves the defect standing: the supervised test is hoisted above the unhealthy/starting/restarting returns, and "some members are up" now counts ANY member not in the down bucket. The old guard was running > 0, which made the R-51 block unreachable in exactly the case it was written for. IsDownState is byte-identical - unhealthy stays excluded, because an unhealthy container is running and folding it in reintroduces the flapping that exclusion exists to stop. No new state was minted. Only the ORDER changed. The priority comment was rewritten because it asserted an ordering the code no longer has. Three subtests in TestAggregateState_UnchangedBranches were AMENDED: they asserted an unhealthy/starting/restarting member beat an exited peer on unless-stopped, which pinned the defect as settled behaviour. They keep their intent with the down member given a benign policy. R-383. The double-failure message said the previous state's backup EXISTS, built from the returned path without asking the filesystem - and a missing file is one of the two ways that rollback fails. undoCopyPhrase now describes the copy from disk: present, partial, missing (still naming where it should be), or never written. Zero-length counts as missing. Test count 1494 -> 1504. Four red-proofs planted, four seen failing; the two halves of R-384 convict independently.
This commit is contained in:
@@ -12,9 +12,15 @@ import (
|
||||
// Part 3 of the 2026-08-22 R-361 session measured that a HELD app raises no dead-app alarm. That is
|
||||
// an ABSENCE claim, and an absence claim is worthless unless the detector is shown working. On the
|
||||
// live box it was hard to hold a stack in a down state long enough to see one: `aggregateState`
|
||||
// checks `unhealthy > 0` BEFORE the mixed-case degraded branch (internal/stacks/manager.go), so an
|
||||
// app whose database dies goes `degraded` for a moment and then `unhealthy` as its own healthcheck
|
||||
// fails — and `unhealthy` is deliberately not in `IsDownState`.
|
||||
// checked `unhealthy > 0` BEFORE the mixed-case degraded branch (internal/stacks/manager.go), so an
|
||||
// app whose database died went `degraded` for a moment and then `unhealthy` as its own healthcheck
|
||||
// failed — and `unhealthy` is deliberately not in `IsDownState`.
|
||||
//
|
||||
// **That sentence described R-384, and nobody filed it.** It was written here as an inconvenience
|
||||
// while building this control; it was in fact the mechanism by which a dead database raised no alarm
|
||||
// at all. Fixed in v0.222.0 by asking the supervised-down question FIRST — see
|
||||
// `internal/stacks/manager.go` aggregateState and TestR384_DeadSupervisedMemberIsAskedAboutFirst.
|
||||
// The past tense above is deliberate: the ordering it describes is no longer the code's.
|
||||
//
|
||||
// `classifyRunStates` is pure, so the control is exact here rather than a race against health probes.
|
||||
func TestClassifyRunStates_PositiveControl_ADownStackDoesAlarm(t *testing.T) {
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||
)
|
||||
|
||||
// R-384, the classifier half of the chain.
|
||||
//
|
||||
// `internal/stacks` TestR384_WiresTheDeadDatabaseThroughTheRealPath drives docker ps → aggregateState
|
||||
// and ends at the STATE. This starts from that state and asserts the CONSEQUENCE: the app is reported
|
||||
// down, so the banner shows it and `NotifyAppStartFailures` has something to fire on. Asserting only
|
||||
// the state would repeat case #9 of the invariant table — mechanism pinned, consequence unpinned.
|
||||
//
|
||||
// Measured live on `demo-hp` 2026-08-22: `bookstack-db` stopped at 21:27:01, the stack read
|
||||
// `unhealthy`, and the F-OBS heartbeat printed "0 currently down" throughout.
|
||||
//
|
||||
// RED-PROOF (observed): with the R-384 hoist reverted, the upstream state is `unhealthy`, and feeding
|
||||
// `unhealthy` here (the second subtest) shows exactly what the live box did — no banner, Down=false.
|
||||
func TestR384_ADeadDatabaseBehindAnUnhealthyAppAlarms(t *testing.T) {
|
||||
now := time.Now()
|
||||
|
||||
// What v0.222.0 produces for the bookstack shape.
|
||||
sts := []stacks.Stack{{Name: "bookstack", Deployed: true, State: stacks.StateDegraded}}
|
||||
dead, states := classifyRunStates(sts, nil, nil, now)
|
||||
if len(dead) != 1 || dead[0].Name != "bookstack" {
|
||||
t.Fatalf("dead-app banner = %+v, want exactly one entry for bookstack", dead)
|
||||
}
|
||||
if dead[0].State != string(stacks.StateDegraded) {
|
||||
t.Errorf("banner state = %q, want %q — the operator must see WHY", dead[0].State, stacks.StateDegraded)
|
||||
}
|
||||
if len(states) != 1 || !states[0].Down {
|
||||
t.Fatalf("run states = %+v, want Down=true (this is what NotifyAppStartFailures reads)", states)
|
||||
}
|
||||
|
||||
// The pre-fix reading, kept as the contrast that makes the fix legible: `unhealthy` reaching this
|
||||
// point is silent, and that silence is correct HERE — it is why the fix had to be upstream, in the
|
||||
// ORDER of the questions, and not by widening IsDownState.
|
||||
preFix := []stacks.Stack{{Name: "bookstack", Deployed: true, State: stacks.StateUnhealthy}}
|
||||
deadPre, statesPre := classifyRunStates(preFix, nil, nil, now)
|
||||
if len(deadPre) != 0 || statesPre[0].Down {
|
||||
t.Fatalf("unhealthy must remain silent at the classifier: dead=%+v states=%+v — R-384 must "+
|
||||
"not have folded unhealthy into the down set", deadPre, statesPre)
|
||||
}
|
||||
}
|
||||
|
||||
// Scenario E: the quiesce suppression must still hold for a stack that now reads `degraded`.
|
||||
// R-97b's defect was telling a customer their app broke during an outage the BACKUP caused, and this
|
||||
// change must not reopen that door — the suppression is cycle-keyed, so it must be state-blind.
|
||||
func TestR384_QuiesceSuppressionStillHoldsForDegraded(t *testing.T) {
|
||||
now := time.Now()
|
||||
sts := []stacks.Stack{{Name: "bookstack", Deployed: true, State: stacks.StateDegraded}}
|
||||
dead, states := classifyRunStates(sts, map[string]bool{"bookstack": true}, nil, now)
|
||||
if len(dead) != 0 {
|
||||
t.Fatalf("a quiesced stack alarmed: %+v — R-97b's exact defect, back through R-384's door", dead)
|
||||
}
|
||||
if states[0].Down {
|
||||
t.Fatalf("a quiesced stack reported Down=true — the customer would be told the backup broke their app")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user