feat(v0.156.0): dead-primary alerting (R-51) + boot desired-state reconciliation (R-52)

R-51: aggregateState's mixed branch returned StateRunning ("partial"), so a stack whose
MAIN container was dead behind live helpers alerted on nothing — immich-server sat Exited
for 18 h, 100 % unreachable, no banner and no app_start_failed (audit F4). New
StateDegraded: a DOWN member whose docker restart policy is always/unless-stopped is a
fault (degraded, a down state); no/on-failure is a finished one-shot and stays benign; an
unreadable policy fails CLOSED. The unhealthy/restarting/paused/unknown exclusions are
byte-identical — folding unhealthy into down is the flapping fix-3 avoided.

R-52: new internal/bootrecon — one bounded start-once sweep at startup (2 attempts, 30 s
apart) for apps an interrupted boot left behind, inside the 90 s boot grace so a success
is silent and a failure still alerts. A zero-container stack is NEVER touched: the UI's
Stop is compose down, so a deliberate stop survives a reboot.

Both features carry a production-path wiring test (the v0.154.0 / v0.91.0 inert-seam
class). The main() assertion is an AST walk, not strings.Contains — the substring version
passed its own red-proof, because a commented-out call still contains the string.

Red-proofs run and restored: mix branch reverted -> "running" on the immich fixture;
boot hook commented out -> wiring test fails; zero-container gate dropped -> the
user-stopped app gets started.

NOTE: controller/cmd/controller/ is matched by .gitignore's `controller` entry, so new
files there need `git add -f` (and ripgrep silently skips main.go without --no-ignore).
This commit is contained in:
2026-07-21 12:27:33 +02:00
parent 0f9b29a19a
commit 285dd1032f
14 changed files with 1167 additions and 22 deletions
+197
View File
@@ -0,0 +1,197 @@
// Package bootrecon implements R-52: the bounded, start-ONCE recovery of apps that were left
// behind by an interrupted boot.
//
// The live failure it closes (AUDIT-vacation-remote-ops-2026-07-20, finding F5): a pre-transport
// shutdown left `immich-server` and `calibre-web` Exited; ten sibling containers came back and
// those two did not, and they were still down ~18 hours later. The controller REPORTED them (the
// 30 s deadapp-check) but never started them — deployed-but-stopped was an alarm with no recovery.
//
// Two deliberate boundaries, both load-bearing:
//
// - **Bounded, never a loop.** At most `attempts` tries, `retryDelay` apart, then it stops and the
// alarm owns the problem. A restart loop would paper over a genuinely broken app forever and
// hammer docker while doing it.
// - **A user's Stop survives a reboot.** The UI's Stop is `docker compose down`, which REMOVES the
// containers; a boot interruption leaves them behind as Exited. So "has containers on disk that
// are down" is the boot-orphan signature, and a stack with ZERO containers is deliberately never
// touched. This distinction is the whole safety argument — see TestReconcile_UserStoppedAppIsNeverStarted.
//
// It runs inside the notifier's boot grace (cmd/controller/main.go `deadAppBootGrace`), so a
// successful recovery never fires an alert and a failed one alerts honestly once the grace expires.
package bootrecon
import (
"context"
"log"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// StackProvider is the slice of *stacks.Manager this package needs. Declared consumer-side so the
// tests can count StartStack calls without a docker anywhere near them.
type StackProvider interface {
GetStacks() []stacks.Stack
StartStack(name string) error
RefreshStatus() error
}
const (
// DefaultAttempts is the total number of start attempts per boot (not per app per retry-forever).
DefaultAttempts = 2
// DefaultRetryDelay spaces the attempts. 2 × 30 s fits comfortably inside the 90 s boot grace,
// so a recovery that works is silent and one that does not is honest.
DefaultRetryDelay = 30 * time.Second
)
// Reconciler performs the start-once sweep. Zero value is not usable — use New.
type Reconciler struct {
stacks StackProvider
logger *log.Logger
attempts int
retryDelay time.Duration
// sleep is the inter-attempt wait; injectable so tests never spend 30 real seconds.
sleep func(context.Context, time.Duration)
}
// Result is the outcome, returned for logging/testing (the hub learns about failures only through
// the existing app_start_failed alarm — this package deliberately pushes no events of its own).
type Result struct {
Candidates []string // boot-orphaned apps found
Recovered []string // running again by the end
StillDown []string // still down after the last attempt — the alarm's problem now
Attempts int // attempts actually made (0 when there was nothing to do)
}
// New builds a Reconciler with the shipped defaults.
func New(p StackProvider, logger *log.Logger) *Reconciler {
return &Reconciler{
stacks: p,
logger: logger,
attempts: DefaultAttempts,
retryDelay: DefaultRetryDelay,
sleep: sleepCtx,
}
}
func sleepCtx(ctx context.Context, d time.Duration) {
t := time.NewTimer(d)
defer t.Stop()
select {
case <-ctx.Done():
case <-t.C:
}
}
// isBootOrphan reports whether a stack is an app the boot left behind.
//
// The gate, term by term:
// - Deployed — an app the customer asked to have running.
// - not Protected — traefik/cloudflared/felhom-controller have their own supervision; this must
// never race the base-stack self-heal.
// - not Deploying — mid-deploy is not a fault.
// - has containers — the D-case guard: a UI Stop removes them, and a deliberate stop must survive
// a reboot.
// - IsDownState — stopped/exited/degraded (R-51 included: a boot that half-started a stack is the
// same interrupted-boot shape).
func isBootOrphan(s stacks.Stack) bool {
return s.Deployed && !s.Protected && !s.Deploying &&
len(s.Containers) > 0 && stacks.IsDownState(s.State)
}
// Run performs the sweep once and returns what happened. It is safe to call with no boot orphans
// (the quiet path logs one DEBUG-free INFO-free line — see below) and it never returns an error:
// a failure to start is an app-level fact the alarm reports, not a controller startup failure.
func (r *Reconciler) Run(ctx context.Context) Result {
var res Result
pending := map[string]bool{}
for _, s := range r.stacks.GetStacks() {
if isBootOrphan(s) {
pending[s.Name] = true
res.Candidates = append(res.Candidates, s.Name)
}
}
sortStrings(res.Candidates)
if len(pending) == 0 {
// The healthy path must be observable — "no alarms" and "never ran" have to be
// distinguishable in a log (the v0.91.2 lesson).
r.logger.Printf("[INFO] [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start)")
return res
}
r.logger.Printf("[INFO] [bootrecon] Boot reconciliation: %d boot-orphaned app(s) found: %v — up to %d attempt(s)",
len(res.Candidates), res.Candidates, r.attempts)
for attempt := 1; attempt <= r.attempts && len(pending) > 0; attempt++ {
res.Attempts = attempt
for _, name := range sortedKeys(pending) {
if ctx.Err() != nil {
break
}
start := time.Now()
if err := r.stacks.StartStack(name); err != nil {
r.logger.Printf("[WARN] [bootrecon] Boot reconciliation attempt %d/%d: start %q failed after %.1fs: %v",
attempt, r.attempts, name, time.Since(start).Seconds(), err)
continue
}
r.logger.Printf("[INFO] [bootrecon] Boot reconciliation attempt %d/%d: started %q (took %.1fs)",
attempt, r.attempts, name, time.Since(start).Seconds())
}
if ctx.Err() != nil {
break
}
// Re-read reality rather than trusting a nil error: `compose up -d` exits 0 on a crash-loop
// (a session-critical invariant of this repo), so only a fresh docker ps can say whether the
// app is actually up.
if err := r.stacks.RefreshStatus(); err != nil {
r.logger.Printf("[WARN] [bootrecon] Boot reconciliation: status refresh failed: %v", err)
}
for _, s := range r.stacks.GetStacks() {
if pending[s.Name] && !stacks.IsDownState(s.State) {
delete(pending, s.Name)
res.Recovered = append(res.Recovered, s.Name)
}
}
if len(pending) > 0 && attempt < r.attempts {
r.sleep(ctx, r.retryDelay)
}
}
res.StillDown = sortedKeys(pending)
sortStrings(res.Recovered)
if len(res.StillDown) == 0 {
r.logger.Printf("[INFO] [bootrecon] Boot reconciliation complete: %d app(s) recovered in %d attempt(s): %v",
len(res.Recovered), res.Attempts, res.Recovered)
} else {
// Deliberately no hub event here: the app_start_failed alarm fires on its own once the boot
// grace expires, and two events for one dead app is how an operator inbox becomes noise.
r.logger.Printf("[WARN] [bootrecon] Boot reconciliation gave up after %d attempt(s): recovered=%v still down=%v (the dead-app alarm now owns these)",
res.Attempts, res.Recovered, res.StillDown)
}
return res
}
// --- tiny local helpers (no dependency on sort ordering semantics elsewhere) ---
func sortedKeys(m map[string]bool) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sortStrings(out)
return out
}
func sortStrings(s []string) {
for i := 1; i < len(s); i++ {
for j := i; j > 0 && s[j] < s[j-1]; j-- {
s[j], s[j-1] = s[j-1], s[j]
}
}
}
@@ -0,0 +1,280 @@
package bootrecon
import (
"context"
"errors"
"io"
"log"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// fakeStacks counts StartStack calls per app — the assertion that matters in BOTH directions:
// scenario C needs exactly-bounded starts, scenario D needs a start count of ZERO.
type fakeStacks struct {
list []stacks.Stack
starts map[string]int
failWith map[string]error
// onStart mutates the world the way a real successful start would (fresh docker ps).
onStart func(f *fakeStacks, name string)
refreshN int
}
func (f *fakeStacks) GetStacks() []stacks.Stack { return f.list }
func (f *fakeStacks) RefreshStatus() error { f.refreshN++; return nil }
func (f *fakeStacks) StartStack(name string) error {
if f.starts == nil {
f.starts = map[string]int{}
}
f.starts[name]++
if err := f.failWith[name]; err != nil {
return err
}
if f.onStart != nil {
f.onStart(f, name)
}
return nil
}
func (f *fakeStacks) setState(name string, st stacks.ContainerState) {
for i := range f.list {
if f.list[i].Name == name {
f.list[i].State = st
}
}
}
// comesUp is the ordinary success behaviour: the app is running after the start.
func comesUp(f *fakeStacks, name string) { f.setState(name, stacks.StateRunning) }
func newTestReconciler(f *fakeStacks) (*Reconciler, *int) {
slept := 0
r := New(f, log.New(io.Discard, "", 0))
r.sleep = func(context.Context, time.Duration) { slept++ }
return r, &slept
}
// bootOrphan is the F5 shape: deployed, containers still present on disk, all Exited — the boot
// interrupted them, nobody stopped them.
func bootOrphan(name string) stacks.Stack {
return stacks.Stack{
Name: name,
Deployed: true,
State: stacks.StateExited,
Containers: []stacks.ContainerInfo{
{Name: name + "-app", State: stacks.StateExited, Status: "Exited (0) 3 minutes ago"},
},
}
}
// userStopped is the UI-Stop shape: `docker compose down` REMOVED the containers.
func userStopped(name string) stacks.Stack {
return stacks.Stack{Name: name, Deployed: true, State: stacks.StateStopped, Containers: nil}
}
// --- Scenario C -------------------------------------------------------------------------------
func TestReconcile_BootOrphanGetsExactlyOneRecovery(t *testing.T) {
f := &fakeStacks{
list: []stacks.Stack{bootOrphan("immich"), bootOrphan("calibre-web")},
onStart: comesUp,
}
r, slept := newTestReconciler(f)
res := r.Run(context.Background())
for _, name := range []string{"immich", "calibre-web"} {
if f.starts[name] != 1 {
t.Fatalf("StartStack(%q) called %d times, want exactly 1", name, f.starts[name])
}
}
if len(res.Recovered) != 2 || len(res.StillDown) != 0 {
t.Fatalf("recovered=%v stillDown=%v, want both apps recovered", res.Recovered, res.StillDown)
}
if res.Attempts != 1 {
t.Fatalf("attempts = %d, want 1 — a success must not retry", res.Attempts)
}
if *slept != 0 {
t.Fatalf("slept %d times after a first-attempt success, want 0", *slept)
}
}
// --- Scenario D (the WRONG case: this must assert the negative) ---------------------------------
func TestReconcile_UserStoppedAppIsNeverStarted(t *testing.T) {
f := &fakeStacks{
list: []stacks.Stack{userStopped("jellyfin"), bootOrphan("immich")},
onStart: comesUp,
}
r, _ := newTestReconciler(f)
res := r.Run(context.Background())
if n := f.starts["jellyfin"]; n != 0 {
t.Fatalf("StartStack(\"jellyfin\") called %d times, want 0 — a deliberate Stop must survive a reboot", n)
}
if f.starts["immich"] != 1 {
t.Fatalf("the real boot orphan was not started: %v", f.starts)
}
for _, c := range res.Candidates {
if c == "jellyfin" {
t.Fatalf("a zero-container stack must never be a reconciliation candidate: %v", res.Candidates)
}
}
}
// --- Bounded, never a loop ----------------------------------------------------------------------
func TestReconcile_StopsAfterTwoAttemptsAndHandsOverToTheAlarm(t *testing.T) {
f := &fakeStacks{
list: []stacks.Stack{bootOrphan("immich")},
failWith: map[string]error{"immich": errors.New("compose up: exit status 1")},
}
r, slept := newTestReconciler(f)
res := r.Run(context.Background())
if f.starts["immich"] != DefaultAttempts {
t.Fatalf("StartStack called %d times, want exactly %d (bounded, never a loop)", f.starts["immich"], DefaultAttempts)
}
if *slept != DefaultAttempts-1 {
t.Fatalf("slept %d times, want %d (one wait BETWEEN attempts, never after the last)", *slept, DefaultAttempts-1)
}
if len(res.StillDown) != 1 || res.StillDown[0] != "immich" {
t.Fatalf("stillDown = %v, want [immich] — the alarm must inherit the failure", res.StillDown)
}
}
// `compose up -d` exits 0 on a crash-loop, so a nil error is not proof the app is up. Only a
// re-read of docker ps can retire a candidate.
func TestReconcile_NilErrorIsNotProofTheAppCameUp(t *testing.T) {
f := &fakeStacks{list: []stacks.Stack{bootOrphan("immich")}} // onStart nil → stays Exited
r, _ := newTestReconciler(f)
res := r.Run(context.Background())
if f.starts["immich"] != DefaultAttempts {
t.Fatalf("StartStack called %d times, want %d — a still-down app must be retried", f.starts["immich"], DefaultAttempts)
}
if len(res.StillDown) != 1 {
t.Fatalf("stillDown = %v, want the app still listed despite StartStack returning nil", res.StillDown)
}
if f.refreshN < 1 {
t.Fatalf("RefreshStatus was never called — the outcome was taken on trust")
}
}
// A second-attempt success must still end clean (and must not alert).
func TestReconcile_SecondAttemptSucceeds(t *testing.T) {
f := &fakeStacks{list: []stacks.Stack{bootOrphan("immich")}}
f.onStart = func(fs *fakeStacks, name string) {
if fs.starts[name] >= 2 {
comesUp(fs, name)
}
}
r, slept := newTestReconciler(f)
res := r.Run(context.Background())
if f.starts["immich"] != 2 {
t.Fatalf("StartStack called %d times, want 2", f.starts["immich"])
}
if *slept != 1 {
t.Fatalf("slept %d times, want 1", *slept)
}
if len(res.StillDown) != 0 || len(res.Recovered) != 1 {
t.Fatalf("recovered=%v stillDown=%v, want a clean recovery on attempt 2", res.Recovered, res.StillDown)
}
}
// --- The gate, term by term ---------------------------------------------------------------------
func TestIsBootOrphan_Gate(t *testing.T) {
base := bootOrphan("app")
cases := []struct {
name string
mut func(s *stacks.Stack)
want bool
}{
{"boot orphan", func(*stacks.Stack) {}, true},
{"degraded counts (R-51 half-started boot)", func(s *stacks.Stack) { s.State = stacks.StateDegraded }, true},
{"not deployed", func(s *stacks.Stack) { s.Deployed = false }, false},
{"protected infra", func(s *stacks.Stack) { s.Protected = true }, false},
{"mid-deploy", func(s *stacks.Stack) { s.Deploying = true }, false},
{"no containers (UI Stop)", func(s *stacks.Stack) { s.Containers = nil }, false},
{"running", func(s *stacks.Stack) { s.State = stacks.StateRunning }, false},
{"unhealthy is not down", func(s *stacks.Stack) { s.State = stacks.StateUnhealthy }, false},
{"restarting recovers itself", func(s *stacks.Stack) { s.State = stacks.StateRestarting }, false},
{"paused is deliberate", func(s *stacks.Stack) { s.State = stacks.StatePaused }, false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
s := base
s.Containers = append([]stacks.ContainerInfo(nil), base.Containers...)
tc.mut(&s)
if got := isBootOrphan(s); got != tc.want {
t.Fatalf("isBootOrphan = %v, want %v", got, tc.want)
}
})
}
}
// A cancelled context (controller shutting down mid-boot) must abandon the sweep, not soldier on.
func TestReconcile_ContextCancellationStops(t *testing.T) {
f := &fakeStacks{list: []stacks.Stack{bootOrphan("immich")}}
r, _ := newTestReconciler(f)
ctx, cancel := context.WithCancel(context.Background())
cancel()
res := r.Run(ctx)
if f.starts["immich"] != 0 {
t.Fatalf("StartStack called %d times on a cancelled context, want 0", f.starts["immich"])
}
if len(res.Candidates) != 1 {
t.Fatalf("candidates = %v, want the app still identified", res.Candidates)
}
}
// The quiet path must be observable — "nothing to do" and "never ran" must not look identical.
func TestReconcile_QuietPathLogsAndStartsNothing(t *testing.T) {
f := &fakeStacks{list: []stacks.Stack{{Name: "immich", Deployed: true, State: stacks.StateRunning,
Containers: []stacks.ContainerInfo{{Name: "immich-app", State: stacks.StateRunning}}}}}
var buf logCapture
r := New(f, log.New(&buf, "", 0))
r.sleep = func(context.Context, time.Duration) {}
res := r.Run(context.Background())
if len(f.starts) != 0 {
t.Fatalf("a healthy box must produce zero starts, got %v", f.starts)
}
if len(res.Candidates) != 0 {
t.Fatalf("candidates = %v, want none", res.Candidates)
}
if !buf.contains("no boot-orphaned apps") {
t.Fatalf("the quiet path logged nothing identifiable: %q", buf.String())
}
}
type logCapture struct{ b []byte }
func (l *logCapture) Write(p []byte) (int, error) { l.b = append(l.b, p...); return len(p), nil }
func (l *logCapture) String() string { return string(l.b) }
func (l *logCapture) contains(s string) bool {
return len(l.b) > 0 && bytesContains(l.b, []byte(s))
}
func bytesContains(hay, needle []byte) bool {
for i := 0; i+len(needle) <= len(hay); i++ {
if string(hay[i:i+len(needle)]) == string(needle) {
return true
}
}
return false
}
var _ io.Writer = (*logCapture)(nil)
+1 -1
View File
@@ -228,7 +228,7 @@ func buildControllerTelemetry(telemetry []metrics.ContainerTelemetry, logs []met
// etc. are excluded to avoid sending zero-value telemetry to the hub.
func isStackRunning(state stacks.ContainerState) bool {
switch state {
case stacks.StateRunning, stacks.StateStarting, stacks.StateUnhealthy, stacks.StateRestarting:
case stacks.StateRunning, stacks.StateStarting, stacks.StateUnhealthy, stacks.StateRestarting, stacks.StateDegraded:
return true
default:
return false
+279
View File
@@ -0,0 +1,279 @@
package stacks
import (
"fmt"
"io"
"log"
"strings"
"sync"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
)
// R-51 (v0.156.0). The live defect these tests pin: on 2026-07-20 `immich-server` sat Exited for
// 18 hours behind three running helpers, the stack aggregated to StateRunning ("partial"), and
// because StateRunning is not a down state NOTHING fired — no dashboard banner, no
// `app_start_failed` hub event — while single-container Calibre-Web, down for the same reason,
// alerted in 90 s. Evidence: felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md
// finding F4.
//
// RED-PROOF (recorded in REPORT.md): with the mix branch reverted to its pre-v0.156.0 body
//
// if running > 0 { return StateRunning }
//
// TestAggregateState_DeadSupervisedMemberIsDegraded fails with
// "aggregateState = running, want degraded", which is exactly the shape the audit observed.
// immichLike is the F4 fixture: the primary Exited, the helpers up.
func immichLike() []ContainerInfo {
return []ContainerInfo{
{Name: "immich-server", State: StateExited, Status: "Exited (137) 18 hours ago"},
{Name: "immich-machine-learning", State: StateRunning, Status: "Up 18 hours"},
{Name: "immich-redis", State: StateRunning, Status: "Up 18 hours (healthy)"},
{Name: "immich-postgres", State: StateRunning, Status: "Up 18 hours (healthy)"},
}
}
// policyMap builds a lookup over a name→policy table; an unlisted name reads as UNKNOWN ("").
func policyMap(t *testing.T, m map[string]string) restartPolicyLookup {
t.Helper()
return func(name string) string { return m[name] }
}
func TestAggregateState_DeadSupervisedMemberIsDegraded(t *testing.T) {
got := aggregateState(immichLike(), policyMap(t, map[string]string{
"immich-server": "unless-stopped",
"immich-machine-learning": "unless-stopped",
"immich-redis": "unless-stopped",
"immich-postgres": "unless-stopped",
}))
if got != StateDegraded {
t.Fatalf("aggregateState = %q, want %q (a dead supervised primary must not read as running)", got, StateDegraded)
}
if !IsDownState(got) {
t.Fatalf("IsDownState(%q) = false — the whole point of R-51 is that this state alerts", got)
}
}
// Scenario B: a one-shot init/migrate container that has legitimately finished must NOT alarm.
func TestAggregateState_OneShotExitedMemberIsBenign(t *testing.T) {
for _, policy := range []string{"no", "on-failure", ""} {
name := policy
if name == "" {
name = "(absent)"
}
t.Run(name, func(t *testing.T) {
containers := []ContainerInfo{
{Name: "app-migrate", State: StateExited, Status: "Exited (0) 2 minutes ago"},
{Name: "app-web", State: StateRunning, Status: "Up 2 minutes"},
}
got := aggregateState(containers, policyMap(t, map[string]string{
"app-migrate": policy,
"app-web": "unless-stopped",
}))
want := StateRunning
if policy == "" {
// UNKNOWN is deliberately fail-CLOSED — see supervisedPolicy. An absent policy in
// the compose file resolves to Docker's "no" at inspect time, so the "" case here
// is the INSPECT-FAILED case, not the no-restart-policy case.
want = StateDegraded
}
if got != want {
t.Fatalf("policy %q: aggregateState = %q, want %q", policy, got, want)
}
})
}
}
// The unchanged branches: R-51 must not move any state the pre-existing aggregation produced.
func TestAggregateState_UnchangedBranches(t *testing.T) {
all := policyMap(t, map[string]string{"a": "unless-stopped", "b": "unless-stopped"})
cases := []struct {
name string
containers []ContainerInfo
want ContainerState
}{
{"no containers", nil, StateNotDeployed},
{"all running", []ContainerInfo{{Name: "a", State: StateRunning}, {Name: "b", State: StateRunning}}, StateRunning},
{"all stopped", []ContainerInfo{{Name: "a", State: StateExited}, {Name: "b", State: StateStopped}}, StateStopped},
{"any unhealthy wins", []ContainerInfo{{Name: "a", State: StateUnhealthy}, {Name: "b", State: StateExited}}, StateUnhealthy},
{"any starting wins over exited", []ContainerInfo{{Name: "a", State: StateStarting}, {Name: "b", State: StateExited}}, StateStarting},
{"any restarting wins over exited", []ContainerInfo{{Name: "a", State: StateRestarting}, {Name: "b", State: StateExited}}, StateRestarting},
{"single container exited", []ContainerInfo{{Name: "a", State: StateExited}}, StateStopped},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := aggregateState(tc.containers, all); got != tc.want {
t.Fatalf("aggregateState = %q, want %q", got, tc.want)
}
})
}
}
// The unhealthy/restarting/paused/unknown exclusions are the fix-3 contract (downstate_test.go owns
// them). This asserts the ONE addition, so a future reader can see R-51 widened the set by exactly
// one state and by nothing else.
func TestIsDownState_DegradedIsTheOnlyAddition(t *testing.T) {
if !IsDownState(StateDegraded) {
t.Fatalf("IsDownState(degraded) = false, want true")
}
for _, s := range []ContainerState{StateUnhealthy, StateRestarting, StatePaused, StateUnknown} {
if IsDownState(s) {
t.Fatalf("IsDownState(%q) = true — R-51 must not touch the fix-3 exclusions", s)
}
}
}
// --- production-path wiring test (§9 rule 6) ---------------------------------------------------
//
// Proves the chain the box actually runs: RefreshStatus → docker ps → aggregateState → docker
// inspect. An aggregateState-only test proves the function, not the caller.
type scriptedDocker struct {
mu sync.Mutex
ps string
policies map[string]string
inspects []string // every container name inspected, in order
}
func (s *scriptedDocker) exec(name string, args ...string) (string, error) {
s.mu.Lock()
defer s.mu.Unlock()
if name != "docker" {
return "", fmt.Errorf("unexpected command %q", name)
}
switch {
case len(args) > 0 && args[0] == "ps":
return s.ps, nil
case len(args) > 0 && args[0] == "inspect":
target := args[len(args)-1]
s.inspects = append(s.inspects, target)
p, ok := s.policies[target]
if !ok {
return "", fmt.Errorf("no such container: %s", target)
}
return p + "\n", nil
}
return "", fmt.Errorf("unexpected docker args %v", args)
}
func (s *scriptedDocker) inspectCount() int {
s.mu.Lock()
defer s.mu.Unlock()
return len(s.inspects)
}
func psLine(name, state, status, project string) string {
return strings.Join([]string{name, "img:1", state, status, project}, "\t")
}
func TestRefreshStatus_WiresDegradedThroughTheRealPath(t *testing.T) {
dock := &scriptedDocker{
ps: strings.Join([]string{
psLine("immich-server", "exited", "Exited (137) 18 hours ago", "immich"),
psLine("immich-redis", "running", "Up 18 hours (healthy)", "immich"),
psLine("calibre-web", "running", "Up 18 hours", "calibre-web"),
}, "\n"),
policies: map[string]string{"immich-server": "unless-stopped"},
}
m := &Manager{
cfg: &config.Config{},
logger: log.New(io.Discard, "", 0),
execFn: dock.exec,
stacks: map[string]*Stack{
"immich": {Name: "immich", Deployed: true},
"calibre-web": {Name: "calibre-web", Deployed: true},
},
}
if err := m.RefreshStatus(); err != nil {
t.Fatalf("RefreshStatus: %v", err)
}
if got := m.stacks["immich"].State; got != StateDegraded {
t.Fatalf("immich state = %q, want %q (the F4 shape must reach the stack map)", got, StateDegraded)
}
if got := m.stacks["calibre-web"].State; got != StateRunning {
t.Fatalf("calibre-web state = %q, want %q — a healthy app must be untouched", got, StateRunning)
}
// Only the DOWN member of the MIXED stack is inspected: never the running members, never the
// healthy stack. An inspect per container per 10 s refresh would be a real docker load.
if n := dock.inspectCount(); n != 1 {
t.Fatalf("docker inspect called %d times, want exactly 1 (%v)", n, dock.inspects)
}
// Second refresh: the answer comes from the cache, so the inspect count must NOT move.
if err := m.RefreshStatus(); err != nil {
t.Fatalf("RefreshStatus (2nd): %v", err)
}
if n := dock.inspectCount(); n != 1 {
t.Fatalf("docker inspect called %d times after a second refresh, want 1 — the cache is not being used", n)
}
if got := m.stacks["immich"].State; got != StateDegraded {
t.Fatalf("immich state after 2nd refresh = %q, want %q", got, StateDegraded)
}
}
// A container that vanishes must not leave its policy behind — an unbounded cache in a process that
// runs for months is a slow leak, and a stale entry would answer for a recreated container.
func TestRestartPolicyCache_PrunesVanishedContainers(t *testing.T) {
dock := &scriptedDocker{
ps: strings.Join([]string{
psLine("app-init", "exited", "Exited (0) 1 minute ago", "app"),
psLine("app-web", "running", "Up 1 minute", "app"),
}, "\n"),
policies: map[string]string{"app-init": "no"},
}
m := &Manager{
cfg: &config.Config{},
logger: log.New(io.Discard, "", 0),
execFn: dock.exec,
stacks: map[string]*Stack{"app": {Name: "app", Deployed: true}},
}
if err := m.RefreshStatus(); err != nil {
t.Fatalf("RefreshStatus: %v", err)
}
if got := m.stacks["app"].State; got != StateRunning {
t.Fatalf("state = %q, want running (a finished one-shot must not alarm)", got)
}
if len(m.restartPolicyCache) != 1 {
t.Fatalf("cache size = %d, want 1", len(m.restartPolicyCache))
}
// The one-shot container is reaped; only the web container remains.
dock.ps = psLine("app-web", "running", "Up 5 minutes", "app")
if err := m.RefreshStatus(); err != nil {
t.Fatalf("RefreshStatus (2nd): %v", err)
}
if len(m.restartPolicyCache) != 0 {
t.Fatalf("cache size = %d after the container vanished, want 0: %v", len(m.restartPolicyCache), m.restartPolicyCache)
}
}
// An inspect failure must not silence the alarm — see supervisedPolicy's fail-closed rationale.
func TestRefreshStatus_InspectFailureStillDegrades(t *testing.T) {
dock := &scriptedDocker{
ps: strings.Join([]string{
psLine("immich-server", "exited", "Exited (137) 1 hour ago", "immich"),
psLine("immich-redis", "running", "Up 1 hour", "immich"),
}, "\n"),
policies: map[string]string{}, // every inspect fails
}
m := &Manager{
cfg: &config.Config{},
logger: log.New(io.Discard, "", 0),
execFn: dock.exec,
stacks: map[string]*Stack{"immich": {Name: "immich", Deployed: true}},
}
if err := m.RefreshStatus(); err != nil {
t.Fatalf("RefreshStatus: %v", err)
}
if got := m.stacks["immich"].State; got != StateDegraded {
t.Fatalf("state = %q, want %q — an unreadable policy must not lose the alarm", got, StateDegraded)
}
// A failed inspect is deliberately NOT cached, so the next cycle retries.
if len(m.restartPolicyCache) != 0 {
t.Fatalf("failed inspect was cached: %v", m.restartPolicyCache)
}
}
+6 -2
View File
@@ -110,7 +110,9 @@ func (m *Manager) DeleteStack(name string, removeHDDData bool) (*DeleteResponse,
}
// Must be stopped (not running)
if stack.State == StateRunning || stack.State == StateStarting || stack.State == StateRestarting {
// StateDegraded (R-51) counts as running here: a degraded stack still has LIVE containers, and
// deleting its directory out from under them would leave orphans behind.
if stack.State == StateRunning || stack.State == StateStarting || stack.State == StateRestarting || stack.State == StateDegraded {
return nil, fmt.Errorf("stack %q is still running — stop it first before deleting", name)
}
@@ -313,7 +315,9 @@ func (m *Manager) RemoveStack(name string, removeHDDData bool, backupPathsToRemo
}
// Must be stopped (not running)
if stack.State == StateRunning || stack.State == StateStarting || stack.State == StateRestarting {
// StateDegraded (R-51) counts as running here: a degraded stack still has LIVE containers, and
// deleting its directory out from under them would leave orphans behind.
if stack.State == StateRunning || stack.State == StateStarting || stack.State == StateRestarting || stack.State == StateDegraded {
return nil, fmt.Errorf("stack %q is still running — stop it first before removing", name)
}
+3 -1
View File
@@ -27,7 +27,9 @@ func (m *Manager) RunHealthProbes() error {
skippedNotDue := 0
skippedNoContainer := 0
for name, stack := range m.stacks {
if stack.State != StateRunning && stack.State != StateUnhealthy {
// StateDegraded (R-51) is probed too: its live members still answer, and the probe result
// only ever overrides StateRunning below, so a degraded stack can never be masked as unhealthy.
if stack.State != StateRunning && stack.State != StateUnhealthy && stack.State != StateDegraded {
continue
}
hc := stack.Meta.HealthCheck
+139 -12
View File
@@ -28,6 +28,7 @@ const (
StateStarting ContainerState = "starting" // running but health: starting
StateUnhealthy ContainerState = "unhealthy" // running but health: unhealthy
StateStopped ContainerState = "stopped"
StateDegraded ContainerState = "degraded" // multi-container stack: a SUPERVISED member is dead (R-51)
StateRestarting ContainerState = "restarting"
StateExited ContainerState = "exited"
StatePaused ContainerState = "paused"
@@ -38,13 +39,20 @@ const (
)
// IsDownState reports whether a container state means a DEPLOYED app is not running and won't recover
// on its own (fix-3, CAMPAIGN-3). Only `stopped` and `exited` qualify — a Docker "created"/"dead"
// container (a failed-at-boot app, the F11 case) resolves to `stopped`. Deliberately NOT `starting`
// / `unhealthy` (running, with their own health handling), `restarting` (self-recovering),
// `deploying` (mid-deploy), `paused` (a deliberate user action), or `unknown` (ambiguous — fail-open,
// never manufacture a dead-app alert from an inconclusive read).
// on its own (fix-3, CAMPAIGN-3). Only `stopped`, `exited` and `degraded` qualify — a Docker
// "created"/"dead" container (a failed-at-boot app, the F11 case) resolves to `stopped`. Deliberately
// NOT `starting` / `unhealthy` (running, with their own health handling), `restarting`
// (self-recovering), `deploying` (mid-deploy), `paused` (a deliberate user action), or `unknown`
// (ambiguous — fail-open, never manufacture a dead-app alert from an inconclusive read).
//
// R-51 (v0.156.0) added `degraded`: a multi-container stack whose SUPERVISED member is dead is as
// unreachable as a single-container app that exited (immich-server sat Exited for 18 h with the app
// 100 % dead and no alert, while single-container Calibre-Web alerted in 90 s). This is deliberately
// NOT the same as folding `unhealthy` into down — that exclusion stays byte-identical, because
// `unhealthy` is a *running* container whose healthcheck is failing and folding it in reintroduces
// the flapping fix-3 was added to stop.
func IsDownState(s ContainerState) bool {
return s == StateStopped || s == StateExited
return s == StateStopped || s == StateExited || s == StateDegraded
}
// ContainerInfo holds status info about a single container within a stack.
@@ -108,6 +116,17 @@ type Manager struct {
backupRunning func() bool // mutual exclusion with the backup orchestrator (Change 3)
migDoneHook func(*MigrationJob) // fired on successful completion (decommission policy lives in caller)
testSeams *migSeams // nil in production; tests inject fakes
// R-51: docker restart policies for DOWN members of mixed stacks. Keyed by
// containerName+"|"+state so a transitioned or recreated container re-reads rather than
// answering from a stale entry; pruned every refresh to the live container set. Guarded by mu
// (every read/write happens under refreshStatusLocked's write lock).
restartPolicyCache map[string]string
// execFn replaces execCommand's process boundary in tests; nil in production.
execFn func(name string, args ...string) (string, error)
// inspectRestartPolicyFn is the docker-inspect seam for the above; nil in production
// (dockerRestartPolicy). Tests inject a scripted lookup and never touch docker.
inspectRestartPolicyFn func(containerName string) (string, error)
// isMountPoint reports whether a path is a live mountpoint; defaults to system.IsMountPoint.
// Injectable so the userdata-belt drive-absent gate is testable (a t.TempDir is never a real mount).
isMountPoint func(string) bool
@@ -306,7 +325,7 @@ func (m *Manager) DeployedStackNames() []string {
}
// RunningAppStacks returns the names of deployed, NON-protected stacks that currently have
// containers up (running/starting/unhealthy/restarting) — the set the quiesce loop (slice 8B)
// containers up (running/starting/unhealthy/restarting/degraded) — the set the quiesce loop (slice 8B)
// stops before an app-consistent backup and restarts after. Protected infra (traefik, cloudflared,
// felhom-controller) is excluded so the controller never stops its own tunnel/proxy or itself.
// Sorted for deterministic stop/start order.
@@ -319,7 +338,9 @@ func (m *Manager) RunningAppStacks() []string {
continue
}
switch stack.State {
case StateRunning, StateStarting, StateUnhealthy, StateRestarting:
// StateDegraded (R-51) belongs here: a degraded stack still has LIVE members, and the
// quiesce loop must stop them before an app-consistent backup and start them after.
case StateRunning, StateStarting, StateUnhealthy, StateRestarting, StateDegraded:
names = append(names, name)
}
}
@@ -451,6 +472,7 @@ func (m *Manager) refreshStatusLocked() error {
}
projectContainers := make(map[string][]ContainerInfo)
liveContainers := make(map[string]bool)
totalContainers := 0
for _, line := range strings.Split(strings.TrimSpace(output), "\n") {
@@ -469,8 +491,11 @@ func (m *Manager) refreshStatusLocked() error {
Status: parts[3],
}
projectContainers[parts[4]] = append(projectContainers[parts[4]], ci)
liveContainers[ci.Name] = true
totalContainers++
}
m.pruneRestartPolicyCacheLocked(liveContainers)
policyOf := m.restartPolicyLookupLocked()
// fix-6: refreshStatusLocked runs every 10s (the status-refresh job) — its per-cycle enumeration
// lines are TRACE (dropped from the debug ring) so they don't eat the post-incident window. A real
@@ -492,7 +517,7 @@ func (m *Manager) refreshStatusLocked() error {
}
} else {
stack.Containers = containers
stack.State = aggregateState(containers)
stack.State = aggregateState(containers, policyOf)
}
// Re-apply controller-side health probe results: if the last probe
@@ -511,6 +536,59 @@ func (m *Manager) refreshStatusLocked() error {
return nil
}
// dockerRestartPolicy reads one container's configured restart policy via docker inspect.
// Returns ("", err) when the container is gone or the inspect fails — the caller treats that as
// UNKNOWN (see supervisedPolicy).
func (m *Manager) dockerRestartPolicy(containerName string) (string, error) {
if m.inspectRestartPolicyFn != nil {
return m.inspectRestartPolicyFn(containerName)
}
out, err := m.execCommand("docker", "inspect", "-f", "{{.HostConfig.RestartPolicy.Name}}", containerName)
if err != nil {
return "", err
}
return strings.TrimSpace(out), nil
}
// restartPolicyLookupLocked returns the cached-and-memoizing lookup handed to aggregateState.
// MUST be called with m.mu held for writing (it populates the cache).
func (m *Manager) restartPolicyLookupLocked() restartPolicyLookup {
return func(name string) string {
key := name + "|policy"
if m.restartPolicyCache != nil {
if p, ok := m.restartPolicyCache[key]; ok {
return p
}
}
p, err := m.dockerRestartPolicy(name)
if err != nil {
// UNKNOWN — deliberately not cached, so a transient docker hiccup does not pin a
// container to "unknown" for the rest of the process lifetime.
m.logger.Printf("[WARN] [stacks] restart-policy inspect failed for container %q: %v (treating as supervised)", name, err)
return ""
}
if m.restartPolicyCache == nil {
m.restartPolicyCache = map[string]string{}
}
m.restartPolicyCache[key] = p
if m.isDebug() {
m.logger.Printf("[DEBUG] [stacks] restart-policy of down member %q = %q", name, p)
}
return p
}
}
// pruneRestartPolicyCacheLocked drops cache entries for containers that no longer exist, so a
// long-lived controller cannot accumulate entries for deleted apps. MUST hold mu for writing.
func (m *Manager) pruneRestartPolicyCacheLocked(live map[string]bool) {
for key := range m.restartPolicyCache {
name := strings.TrimSuffix(key, "|policy")
if !live[name] {
delete(m.restartPolicyCache, key)
}
}
}
// resolveContainerState determines the effective state by combining Docker's
// State field (running/exited/etc.) with the Status field that contains health info.
//
@@ -545,9 +623,35 @@ func resolveContainerState(dockerState, dockerStatus string) ContainerState {
}
}
// restartPolicyLookup returns a container's docker restart policy name ("always", "unless-stopped",
// "on-failure", "no"). An empty string means UNKNOWN — the inspect failed or no lookup was supplied.
type restartPolicyLookup func(containerName string) string
// supervisedPolicy reports whether a restart policy means "docker is supposed to keep this container
// running" — i.e. its being Exited is a fault, not a design.
//
// UNKNOWN ("") counts as supervised, deliberately fail-CLOSED, which is the opposite of the
// IsDownState fail-open rule and for a different reason: there the input is an *ambiguous state*,
// here we already KNOW a member is dead and only the excuse is missing. The P2 census (2026-07-21,
// 53 catalog templates / 78 services) found **every** catalog service on `unless-stopped` and zero
// one-shot init/migrate containers, so "unknown" in the field is an inspect failure on a container
// that is almost certainly supervised. Missing a real dead-primary alarm is the failure that cost
// 18 h; a false alarm is a banner.
func supervisedPolicy(policy string) bool {
switch policy {
case "no", "on-failure":
return false
default: // "always", "unless-stopped", "" (unknown)
return true
}
}
// aggregateState determines the overall stack state from its containers.
// Priority: unhealthy/starting > restarting > all-running > stopped
func aggregateState(containers []ContainerInfo) ContainerState {
// Priority: unhealthy/starting > restarting > all-running > degraded > stopped
//
// policyOf is consulted ONLY for the mixed case (some members up, some down) and ONLY for the down
// members — see the mix branch. nil is allowed (every exited member then reads as supervised).
func aggregateState(containers []ContainerInfo, policyOf restartPolicyLookup) ContainerState {
if len(containers) == 0 {
return StateNotDeployed
}
@@ -557,6 +661,7 @@ func aggregateState(containers []ContainerInfo) ContainerState {
unhealthy := 0
restarting := 0
stopped := 0
var down []ContainerInfo
for _, c := range containers {
switch c.State {
@@ -570,6 +675,7 @@ func aggregateState(containers []ContainerInfo) ContainerState {
restarting++
case StateStopped, StateExited:
stopped++
down = append(down, c)
}
}
@@ -595,8 +701,22 @@ func aggregateState(containers []ContainerInfo) ContainerState {
if stopped == total {
return StateStopped
}
// Mix (some running, some stopped) — report as running (partial)
// Mix (some members up, some down) — R-51. Until v0.156.0 this reported StateRunning
// unconditionally ("partial"), which is why a dead immich-server behind three live helpers was
// invisible to fix-3 for 18 hours. A down member whose restart policy says docker should be
// keeping it up is a FAULT → the whole stack is degraded (and degraded is a down state). A down
// member with policy `no`/`on-failure` is a one-shot init/migrate container that has legitimately
// finished → benign, the stack stays running.
if running > 0 {
for _, c := range down {
policy := ""
if policyOf != nil {
policy = policyOf(c.Name)
}
if supervisedPolicy(policy) {
return StateDegraded
}
}
return StateRunning
}
@@ -1043,6 +1163,13 @@ func (m *Manager) composeExecCustomEnv(dir string, env []string, args ...string)
}
func (m *Manager) execCommand(name string, args ...string) (string, error) {
// execFn is the process-boundary seam (nil in production). It exists so R-51 can be proven
// through the REAL refreshStatusLocked path — docker ps → aggregateState → docker inspect —
// rather than only through an injected aggregation helper, which would prove the helper and
// not the caller (the v0.154.0 / v0.91.0 inert-seam class).
if m.execFn != nil {
return m.execFn(name, args...)
}
cmd := exec.Command(name, args...)
var stdout, stderr bytes.Buffer
+13 -4
View File
@@ -37,7 +37,9 @@ func getTimezone() *time.Location {
// so an unhealthy app with a dead URL isn't mistaken for a merely-degraded-but-reachable one.
func routeUnpublished(state stacks.ContainerState) bool {
switch state {
case stacks.StateUnhealthy, stacks.StateRestarting:
// StateDegraded (R-51): the dead member is typically the one Traefik routes to, so the public
// URL 404s exactly as it does for an unhealthy container.
case stacks.StateUnhealthy, stacks.StateRestarting, stacks.StateDegraded:
return true
default:
return false
@@ -64,6 +66,9 @@ func (s *Server) templateFuncMap() template.FuncMap {
case stacks.StateRestarting:
// a restart loop is a problem, not progress
return "warn"
case stacks.StateDegraded:
// R-51: a supervised member is dead — a genuine failure, not a user action
return "warn"
case stacks.StateStopped, stacks.StateExited:
return "neutral"
case stacks.StatePaused:
@@ -84,6 +89,8 @@ func (s *Server) templateFuncMap() template.FuncMap {
return "Telepítés..."
case stacks.StateUnhealthy:
return "Nem egészséges"
case stacks.StateDegraded:
return "Részlegesen leállt"
case stacks.StateStopped, stacks.StateExited:
return "Leállítva"
case stacks.StateRestarting:
@@ -102,7 +109,7 @@ func (s *Server) templateFuncMap() template.FuncMap {
return "●"
case stacks.StateStarting, stacks.StateDeploying:
return "◐"
case stacks.StateUnhealthy:
case stacks.StateUnhealthy, stacks.StateDegraded:
return "◑"
case stacks.StateStopped, stacks.StateExited:
return "○"
@@ -119,7 +126,7 @@ func (s *Server) templateFuncMap() template.FuncMap {
// and is not stopped/exited — used by templates for showing action buttons
"isOperational": func(state stacks.ContainerState) bool {
switch state {
case stacks.StateRunning, stacks.StateStarting, stacks.StateUnhealthy, stacks.StateRestarting:
case stacks.StateRunning, stacks.StateStarting, stacks.StateUnhealthy, stacks.StateRestarting, stacks.StateDegraded:
return true
default:
return false
@@ -201,7 +208,9 @@ func (s *Server) templateFuncMap() template.FuncMap {
switch state {
case stacks.StateRunning, stacks.StateStarting, stacks.StateUnhealthy, stacks.StateRestarting:
return "running"
case stacks.StateStopped, stacks.StateExited, stacks.StatePaused:
case stacks.StateStopped, stacks.StateExited, stacks.StatePaused, stacks.StateDegraded:
// R-51: degraded filters with the stopped set — the customer's question is
// "is it working", and a stack with a dead supervised member is not.
return "stopped"
default:
if deployed {
+3 -1
View File
@@ -148,7 +148,9 @@ func (s *Server) dashboardHandler(w http.ResponseWriter, r *http.Request) {
switch st.State {
case stacks.StateRunning, stacks.StateStarting, stacks.StateUnhealthy, stacks.StateRestarting:
running++
case stacks.StateStopped, stacks.StateExited:
// R-51: degraded counts with stopped — the dashboard counter answers "how many of my apps
// work", and a stack with a dead supervised member does not.
case stacks.StateStopped, stacks.StateExited, stacks.StateDegraded:
stopped++
}
}