v0.222.0: ask whether a supervised member is DEAD before whether one is UNHEALTHY (R-384), and stop promising an undo copy nobody looked for (R-383)
gates / gates (push) Successful in 11s

R-384. aggregateState returned StateUnhealthy the moment unhealthy > 0, and the
R-51 mixed-case block that asks "is a supervised member dead?" sat below it. A
two-container app whose database exits goes unhealthy BECAUSE it cannot reach
that database - so the symptom the dead database causes was what suppressed the
alarm for it. unhealthy is not a down state, so classifyRunStates never marked
the app down and app_start_failed never fired.

Measured live on demo-hp 2026-08-22: bookstack-db stopped at 21:27:01 and the
F-OBS heartbeat printed "0 currently down" throughout. R-51's 18-hour immich
failure, back through a different door.

Two things moved, and either alone leaves the defect standing: the supervised
test is hoisted above the unhealthy/starting/restarting returns, and "some
members are up" now counts ANY member not in the down bucket. The old guard was
running > 0, which made the R-51 block unreachable in exactly the case it was
written for.

IsDownState is byte-identical - unhealthy stays excluded, because an unhealthy
container is running and folding it in reintroduces the flapping that exclusion
exists to stop. No new state was minted. Only the ORDER changed. The priority
comment was rewritten because it asserted an ordering the code no longer has.

Three subtests in TestAggregateState_UnchangedBranches were AMENDED: they
asserted an unhealthy/starting/restarting member beat an exited peer on
unless-stopped, which pinned the defect as settled behaviour. They keep their
intent with the down member given a benign policy.

R-383. The double-failure message said the previous state's backup EXISTS,
built from the returned path without asking the filesystem - and a missing file
is one of the two ways that rollback fails. undoCopyPhrase now describes the
copy from disk: present, partial, missing (still naming where it should be), or
never written. Zero-length counts as missing.

Test count 1494 -> 1504. Four red-proofs planted, four seen failing; the two
halves of R-384 convict independently.
This commit is contained in:
2026-08-23 07:25:00 +02:00
parent da75603553
commit 5da11c4480
7 changed files with 638 additions and 27 deletions
@@ -12,9 +12,15 @@ import (
// Part 3 of the 2026-08-22 R-361 session measured that a HELD app raises no dead-app alarm. That is
// an ABSENCE claim, and an absence claim is worthless unless the detector is shown working. On the
// live box it was hard to hold a stack in a down state long enough to see one: `aggregateState`
// checks `unhealthy > 0` BEFORE the mixed-case degraded branch (internal/stacks/manager.go), so an
// app whose database dies goes `degraded` for a moment and then `unhealthy` as its own healthcheck
// fails — and `unhealthy` is deliberately not in `IsDownState`.
// checked `unhealthy > 0` BEFORE the mixed-case degraded branch (internal/stacks/manager.go), so an
// app whose database died went `degraded` for a moment and then `unhealthy` as its own healthcheck
// failed — and `unhealthy` is deliberately not in `IsDownState`.
//
// **That sentence described R-384, and nobody filed it.** It was written here as an inconvenience
// while building this control; it was in fact the mechanism by which a dead database raised no alarm
// at all. Fixed in v0.222.0 by asking the supervised-down question FIRST — see
// `internal/stacks/manager.go` aggregateState and TestR384_DeadSupervisedMemberIsAskedAboutFirst.
// The past tense above is deliberate: the ordering it describes is no longer the code's.
//
// `classifyRunStates` is pure, so the control is exact here rather than a race against health probes.
func TestClassifyRunStates_PositiveControl_ADownStackDoesAlarm(t *testing.T) {
@@ -0,0 +1,62 @@
package main
import (
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// R-384, the classifier half of the chain.
//
// `internal/stacks` TestR384_WiresTheDeadDatabaseThroughTheRealPath drives docker ps → aggregateState
// and ends at the STATE. This starts from that state and asserts the CONSEQUENCE: the app is reported
// down, so the banner shows it and `NotifyAppStartFailures` has something to fire on. Asserting only
// the state would repeat case #9 of the invariant table — mechanism pinned, consequence unpinned.
//
// Measured live on `demo-hp` 2026-08-22: `bookstack-db` stopped at 21:27:01, the stack read
// `unhealthy`, and the F-OBS heartbeat printed "0 currently down" throughout.
//
// RED-PROOF (observed): with the R-384 hoist reverted, the upstream state is `unhealthy`, and feeding
// `unhealthy` here (the second subtest) shows exactly what the live box did — no banner, Down=false.
func TestR384_ADeadDatabaseBehindAnUnhealthyAppAlarms(t *testing.T) {
now := time.Now()
// What v0.222.0 produces for the bookstack shape.
sts := []stacks.Stack{{Name: "bookstack", Deployed: true, State: stacks.StateDegraded}}
dead, states := classifyRunStates(sts, nil, nil, now)
if len(dead) != 1 || dead[0].Name != "bookstack" {
t.Fatalf("dead-app banner = %+v, want exactly one entry for bookstack", dead)
}
if dead[0].State != string(stacks.StateDegraded) {
t.Errorf("banner state = %q, want %q — the operator must see WHY", dead[0].State, stacks.StateDegraded)
}
if len(states) != 1 || !states[0].Down {
t.Fatalf("run states = %+v, want Down=true (this is what NotifyAppStartFailures reads)", states)
}
// The pre-fix reading, kept as the contrast that makes the fix legible: `unhealthy` reaching this
// point is silent, and that silence is correct HERE — it is why the fix had to be upstream, in the
// ORDER of the questions, and not by widening IsDownState.
preFix := []stacks.Stack{{Name: "bookstack", Deployed: true, State: stacks.StateUnhealthy}}
deadPre, statesPre := classifyRunStates(preFix, nil, nil, now)
if len(deadPre) != 0 || statesPre[0].Down {
t.Fatalf("unhealthy must remain silent at the classifier: dead=%+v states=%+v — R-384 must "+
"not have folded unhealthy into the down set", deadPre, statesPre)
}
}
// Scenario E: the quiesce suppression must still hold for a stack that now reads `degraded`.
// R-97b's defect was telling a customer their app broke during an outage the BACKUP caused, and this
// change must not reopen that door — the suppression is cycle-keyed, so it must be state-blind.
func TestR384_QuiesceSuppressionStillHoldsForDegraded(t *testing.T) {
now := time.Now()
sts := []stacks.Stack{{Name: "bookstack", Deployed: true, State: stacks.StateDegraded}}
dead, states := classifyRunStates(sts, map[string]bool{"bookstack": true}, nil, now)
if len(dead) != 0 {
t.Fatalf("a quiesced stack alarmed: %+v — R-97b's exact defect, back through R-384's door", dead)
}
if states[0].Down {
t.Fatalf("a quiesced stack reported Down=true — the customer would be told the backup broke their app")
}
}