admin
5da11c4480
gates / gates (push) Successful in 11s
R-384. aggregateState returned StateUnhealthy the moment unhealthy > 0, and the R-51 mixed-case block that asks "is a supervised member dead?" sat below it. A two-container app whose database exits goes unhealthy BECAUSE it cannot reach that database - so the symptom the dead database causes was what suppressed the alarm for it. unhealthy is not a down state, so classifyRunStates never marked the app down and app_start_failed never fired. Measured live on demo-hp 2026-08-22: bookstack-db stopped at 21:27:01 and the F-OBS heartbeat printed "0 currently down" throughout. R-51's 18-hour immich failure, back through a different door. Two things moved, and either alone leaves the defect standing: the supervised test is hoisted above the unhealthy/starting/restarting returns, and "some members are up" now counts ANY member not in the down bucket. The old guard was running > 0, which made the R-51 block unreachable in exactly the case it was written for. IsDownState is byte-identical - unhealthy stays excluded, because an unhealthy container is running and folding it in reintroduces the flapping that exclusion exists to stop. No new state was minted. Only the ORDER changed. The priority comment was rewritten because it asserted an ordering the code no longer has. Three subtests in TestAggregateState_UnchangedBranches were AMENDED: they asserted an unhealthy/starting/restarting member beat an exited peer on unless-stopped, which pinned the defect as settled behaviour. They keep their intent with the down member given a benign policy. R-383. The double-failure message said the previous state's backup EXISTS, built from the returned path without asking the filesystem - and a missing file is one of the two ways that rollback fails. undoCopyPhrase now describes the copy from disk: present, partial, missing (still naming where it should be), or never written. Zero-length counts as missing. Test count 1494 -> 1504. Four red-proofs planted, four seen failing; the two halves of R-384 convict independently.
63 lines
3.1 KiB
Go
63 lines
3.1 KiB
Go
package main
|
|
|
|
import (
|
|
"testing"
|
|
"time"
|
|
|
|
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
|
)
|
|
|
|
// R-384, the classifier half of the chain.
|
|
//
|
|
// `internal/stacks` TestR384_WiresTheDeadDatabaseThroughTheRealPath drives docker ps → aggregateState
|
|
// and ends at the STATE. This starts from that state and asserts the CONSEQUENCE: the app is reported
|
|
// down, so the banner shows it and `NotifyAppStartFailures` has something to fire on. Asserting only
|
|
// the state would repeat case #9 of the invariant table — mechanism pinned, consequence unpinned.
|
|
//
|
|
// Measured live on `demo-hp` 2026-08-22: `bookstack-db` stopped at 21:27:01, the stack read
|
|
// `unhealthy`, and the F-OBS heartbeat printed "0 currently down" throughout.
|
|
//
|
|
// RED-PROOF (observed): with the R-384 hoist reverted, the upstream state is `unhealthy`, and feeding
|
|
// `unhealthy` here (the second subtest) shows exactly what the live box did — no banner, Down=false.
|
|
func TestR384_ADeadDatabaseBehindAnUnhealthyAppAlarms(t *testing.T) {
|
|
now := time.Now()
|
|
|
|
// What v0.222.0 produces for the bookstack shape.
|
|
sts := []stacks.Stack{{Name: "bookstack", Deployed: true, State: stacks.StateDegraded}}
|
|
dead, states := classifyRunStates(sts, nil, nil, now)
|
|
if len(dead) != 1 || dead[0].Name != "bookstack" {
|
|
t.Fatalf("dead-app banner = %+v, want exactly one entry for bookstack", dead)
|
|
}
|
|
if dead[0].State != string(stacks.StateDegraded) {
|
|
t.Errorf("banner state = %q, want %q — the operator must see WHY", dead[0].State, stacks.StateDegraded)
|
|
}
|
|
if len(states) != 1 || !states[0].Down {
|
|
t.Fatalf("run states = %+v, want Down=true (this is what NotifyAppStartFailures reads)", states)
|
|
}
|
|
|
|
// The pre-fix reading, kept as the contrast that makes the fix legible: `unhealthy` reaching this
|
|
// point is silent, and that silence is correct HERE — it is why the fix had to be upstream, in the
|
|
// ORDER of the questions, and not by widening IsDownState.
|
|
preFix := []stacks.Stack{{Name: "bookstack", Deployed: true, State: stacks.StateUnhealthy}}
|
|
deadPre, statesPre := classifyRunStates(preFix, nil, nil, now)
|
|
if len(deadPre) != 0 || statesPre[0].Down {
|
|
t.Fatalf("unhealthy must remain silent at the classifier: dead=%+v states=%+v — R-384 must "+
|
|
"not have folded unhealthy into the down set", deadPre, statesPre)
|
|
}
|
|
}
|
|
|
|
// Scenario E: the quiesce suppression must still hold for a stack that now reads `degraded`.
|
|
// R-97b's defect was telling a customer their app broke during an outage the BACKUP caused, and this
|
|
// change must not reopen that door — the suppression is cycle-keyed, so it must be state-blind.
|
|
func TestR384_QuiesceSuppressionStillHoldsForDegraded(t *testing.T) {
|
|
now := time.Now()
|
|
sts := []stacks.Stack{{Name: "bookstack", Deployed: true, State: stacks.StateDegraded}}
|
|
dead, states := classifyRunStates(sts, map[string]bool{"bookstack": true}, nil, now)
|
|
if len(dead) != 0 {
|
|
t.Fatalf("a quiesced stack alarmed: %+v — R-97b's exact defect, back through R-384's door", dead)
|
|
}
|
|
if states[0].Down {
|
|
t.Fatalf("a quiesced stack reported Down=true — the customer would be told the backup broke their app")
|
|
}
|
|
}
|