feat(v0.156.0): dead-primary alerting (R-51) + boot desired-state reconciliation (R-52)
R-51: aggregateState's mixed branch returned StateRunning ("partial"), so a stack whose
MAIN container was dead behind live helpers alerted on nothing — immich-server sat Exited
for 18 h, 100 % unreachable, no banner and no app_start_failed (audit F4). New
StateDegraded: a DOWN member whose docker restart policy is always/unless-stopped is a
fault (degraded, a down state); no/on-failure is a finished one-shot and stays benign; an
unreadable policy fails CLOSED. The unhealthy/restarting/paused/unknown exclusions are
byte-identical — folding unhealthy into down is the flapping fix-3 avoided.
R-52: new internal/bootrecon — one bounded start-once sweep at startup (2 attempts, 30 s
apart) for apps an interrupted boot left behind, inside the 90 s boot grace so a success
is silent and a failure still alerts. A zero-container stack is NEVER touched: the UI's
Stop is compose down, so a deliberate stop survives a reboot.
Both features carry a production-path wiring test (the v0.154.0 / v0.91.0 inert-seam
class). The main() assertion is an AST walk, not strings.Contains — the substring version
passed its own red-proof, because a commented-out call still contains the string.
Red-proofs run and restored: mix branch reverted -> "running" on the immich fixture;
boot hook commented out -> wiring test fails; zero-container gate dropped -> the
user-stopped app gets started.
NOTE: controller/cmd/controller/ is matched by .gitignore's `controller` entry, so new
files there need `git add -f` (and ripgrep silently skips main.go without --no-ignore).
This commit is contained in:
@@ -25,6 +25,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/appexport"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/assets"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/bootrecon"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/bootstrap"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/channelhealth"
|
||||
cf "gitea.dooplex.hu/admin/felhom-controller/internal/cloudflare"
|
||||
@@ -225,6 +226,14 @@ func main() {
|
||||
// Recover FIRST (restart any stacks left stopped by a crash mid-quiesce), then start the loop.
|
||||
quiesceLoop := startQuiesceLoop(ctx, cfg, stackMgr, logger)
|
||||
|
||||
// --- R-52: boot desired-state reconciliation ---
|
||||
// A deployed app that missed its boot start used to stay down until a human noticed (F5: immich
|
||||
// and calibre-web sat Exited for ~18 h while ten siblings came back). One bounded start-once
|
||||
// sweep, deliberately AFTER the quiesce recovery above so the two never race for the same stack,
|
||||
// and entirely inside deadAppBootGrace so a successful recovery is silent and a failed one still
|
||||
// alerts honestly. Never touches an app the customer stopped — see internal/bootrecon.
|
||||
go runBootReconcile(ctx, stackMgr, logger)
|
||||
|
||||
// --- Start CPU collector ---
|
||||
cpuCollector := system.NewCPUCollector(5 * time.Second)
|
||||
cpuCollector.Start(ctx)
|
||||
@@ -1109,6 +1118,29 @@ func main() {
|
||||
// own boot. After the grace, an app that still isn't running alerts (the F11 dead-at-boot case).
|
||||
const deadAppBootGrace = 90 * time.Second
|
||||
|
||||
// bootReconcileSettle lets the initial scan, the first status refresh and the quiesce recovery
|
||||
// settle before the R-52 sweep decides what "down" means. 5 s + at most one 30 s retry gap keeps
|
||||
// the whole sweep inside deadAppBootGrace (90 s), which is what makes a successful recovery silent.
|
||||
var bootReconcileSettle = 5 * time.Second
|
||||
|
||||
// bootReconcileFn is the R-52 sweep, a package var purely so the wiring below is testable from
|
||||
// package main (the v0.154.0 / v0.91.0 lesson: a seam proven only through injection proves the
|
||||
// component and not the caller).
|
||||
var bootReconcileFn = func(ctx context.Context, mgr bootrecon.StackProvider, logger *log.Logger) bootrecon.Result {
|
||||
return bootrecon.New(mgr, logger).Run(ctx)
|
||||
}
|
||||
|
||||
// runBootReconcile waits out the settle window, then performs exactly one bounded recovery sweep.
|
||||
// Called from main() in a goroutine; returns after the single sweep — there is no loop by design.
|
||||
func runBootReconcile(ctx context.Context, mgr bootrecon.StackProvider, logger *log.Logger) {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(bootReconcileSettle):
|
||||
}
|
||||
bootReconcileFn(ctx, mgr, logger)
|
||||
}
|
||||
|
||||
// scanDeployedAppRunStates returns the fix-3 view of the deployed apps: the DEAD ones (for the
|
||||
// state-based dashboard banner) and EVERY deployed app's run state (for the notifier's one-event-per-
|
||||
// transition tracking). Deploying apps are skipped (mid-deploy is not a fault). Pure over GetStacks().
|
||||
@@ -1472,7 +1504,9 @@ func (a *exportAdapter) GetStackHDDPath(name string) string {
|
||||
|
||||
func (a *exportAdapter) IsStackRunning(name string) bool {
|
||||
s, ok := a.mgr.GetStack(name)
|
||||
return ok && s.State == stacks.StateRunning
|
||||
// StateDegraded (R-51) counts as running: the export must stop the still-live members before
|
||||
// reading their volumes, exactly as it would for a fully running stack.
|
||||
return ok && (s.State == stacks.StateRunning || s.State == stacks.StateDegraded)
|
||||
}
|
||||
|
||||
func (a *exportAdapter) StopStack(name string) error {
|
||||
|
||||
Reference in New Issue
Block a user