v0.190.0 — the boot settle window, both gates on intent, and R-171
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
R-171 (a regression v0.189.0 introduced, CONFIRMED on hardware before any fix was written). Replacing isBootOrphan's container-count term with recorded intent made a drive-gate-stopped app read as a boot orphan: the gate stops apps with `compose down` (zero containers) and never touches desired_state, because it is not the customer. Observed on 9201 with the drive held unmounted — the sweep found and started it, burned both attempts, and handed it to the dead-app alarm. The write hazard did not materialise (the unbound mountpoint is host-root-owned and the guest is unprivileged) but that protection is accidental and untested. New consumer-side seam bootrecon.StartGate, fail-safe (cannot determine ⇒ do not start), wired in main.go. The rule is not new: the API's startGatedByMissingDrive already refuses this; the sweep bypassed it. R-157 mechanism A. The sweep looked once at T+5s, deriving candidates from a fleet docker was still restoring — three of six hard resets. Now a settle-then- sweep window: sample every 5s, settled after 3 identical samples, sweep ONCE at the end; ends on settled or a 50s budget, and the log says which. The budget is 50s because settle+budget+one retry must stay under the 90s dead-app grace — a test rejected 60s at 95s. A window that overruns emits a LATE RECOVERY warn rather than the grace being widened to hide it. Widening the window made two more holders reachable, so the one gate covers all three: an absent drive, a quiesce, and an in-flight app-data operation — reusing quiesce.SuppressedStacks() and a new read-only AppStopGuard.HeldStacks(). R-170. shouldRecreateOnBoot now reads desired_state with the identical three-way table; absent keeps the old hasContainers behaviour exactly. Its comment argued for the container count and was rewritten. presentStable is untouched. The two gates' agreement is pinned from both sides against one fixture table. 27/27 packages green; 6 red-proofs observed FAIL then restored.
This commit is contained in:
@@ -11,6 +11,7 @@ import (
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/agentapi"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||
)
|
||||
|
||||
// bootBindWait / bootBindPoll bound the readiness gate in processGuestBootChange. On a guest reboot
|
||||
@@ -106,31 +107,54 @@ func agentWhere(registeredPath string) string {
|
||||
// container-uptime sample and a `State != stopped` filter MISS exactly that case. Do not reintroduce
|
||||
// a state filter here.
|
||||
//
|
||||
// R-55: it DOES filter on whether the app still HAS containers, which is a different question and the
|
||||
// one that tells the truth about intent. This is R-52's `existing-Exited vs absent` distinction
|
||||
// (bootrecon.isBootOrphan), translated to this gate:
|
||||
// ── R-170: intent replaced the container count (v0.190.0) ────────────────────────────────────────
|
||||
//
|
||||
// - containers EXIST but are down → the guest went down under the app; docker's own records survive
|
||||
// the reboot, so this is a boot orphan → recreate.
|
||||
// - ZERO containers → a UI Stop is `compose down`, which REMOVES the containers.
|
||||
// Nothing else in the controller leaves a deployed app at zero containers. → the customer stopped
|
||||
// this on purpose → LEAVE IT ALONE.
|
||||
// The question this gate has to answer is "did the customer want this app running?", and until
|
||||
// v0.190.0 it answered by counting containers, exactly as bootrecon.isBootOrphan did:
|
||||
//
|
||||
// `deployed` cannot answer this: it is a deploy-lifecycle flag and stays true across a Stop. Before
|
||||
// R-55 the gate had no other signal and therefore silently undid a customer's Stop on every guest
|
||||
// reboot — including when apps were stopped deliberately to free resources for others, which is
|
||||
// precisely when resurrecting them is most harmful.
|
||||
// ZERO containers → a UI Stop is `compose down`, which REMOVES them → the customer stopped it.
|
||||
//
|
||||
// R-55 added that term for a real defect and it was the right fix FOR THE SIGNAL AVAILABLE THEN:
|
||||
// before it, the gate silently undid a customer's Stop on every guest reboot — including apps
|
||||
// stopped deliberately to free resources for others, which is precisely when resurrecting them is
|
||||
// most harmful. `deployed` could not answer it: it is a deploy-lifecycle flag and stays true across
|
||||
// a Stop.
|
||||
//
|
||||
// But the count has at least three causes and cannot separate them — a deliberate Stop, a power cut
|
||||
// mid-compose, an interrupted backup — so R-166 replaced it in `bootrecon` with the customer's
|
||||
// RECORDED intent (`desired_state` in app.yaml). This gate was left on the old signal for one
|
||||
// release, which left the two boot gates disagreeing about the same question. They now agree:
|
||||
//
|
||||
// stopped → NEVER recreate. The customer said so; no observation overrides it.
|
||||
// running → recreate, whatever the container count. This is the case the count could not see.
|
||||
// absent → fall back to `hasContainers`, i.e. EXACTLY the pre-v0.190.0 behaviour.
|
||||
//
|
||||
// The absent branch is not a leftover. Every app.yaml written before v0.189.0 lacks the field, so
|
||||
// absent is what an upgraded box reads for every app nobody has pressed a button on since; treating
|
||||
// it as `running` would recreate — and therefore start — apps their owners had deliberately stopped,
|
||||
// fleet-wide, on the first reboot after the upgrade.
|
||||
//
|
||||
// The evidence is read from the snapshot taken BEFORE any recreate runs, because `recreate` itself
|
||||
// calls StopStack (`compose down`) and so destroys it.
|
||||
// calls StopStack (`compose down`) and so destroys it (R-55).
|
||||
//
|
||||
// NOTE on the drive-absent gate: apps it stopped are also at zero containers, so they are skipped
|
||||
// here too. That is correct — they are recorded in StoragePath.StoppedStacks and restarted by
|
||||
// ReconcileDriveGates' `Return` branch, which runs on the same loop tick. Their recovery is that
|
||||
// path's job, not this one's.
|
||||
func shouldRecreateOnBoot(deployed bool, hdd string, presentStable map[string]bool, hasContainers bool) bool {
|
||||
return deployed && hdd != "" && strings.HasPrefix(hdd, StableParentDir+"/") &&
|
||||
presentStable[hdd] && hasContainers
|
||||
// NOTE on the drive-absent gate: `presentStable[hdd]` is still required and is still load-bearing.
|
||||
// An app whose drive is absent is never recreated here no matter what its intent says — it is
|
||||
// recorded in StoragePath.StoppedStacks and restarted by ReconcileDriveGates' `Return` branch, which
|
||||
// runs on the same loop tick. That term is what keeps this gate safe; it is the term the BOOT SWEEP
|
||||
// was missing until R-171 (audits/DIAG-bootrecon-drive-absent-2026-08-02.md), and it must not be
|
||||
// dropped in sympathy with the count.
|
||||
func shouldRecreateOnBoot(deployed bool, hdd string, presentStable map[string]bool, hasContainers bool, desired string) bool {
|
||||
if !deployed || hdd == "" || !strings.HasPrefix(hdd, StableParentDir+"/") || !presentStable[hdd] {
|
||||
return false
|
||||
}
|
||||
switch desired {
|
||||
case stacks.DesiredStateStopped:
|
||||
return false
|
||||
case stacks.DesiredStateRunning:
|
||||
return true
|
||||
default:
|
||||
return hasContainers // absent/legacy — byte-identical to the pre-v0.190.0 rule
|
||||
}
|
||||
}
|
||||
|
||||
// defaultPromotionTarget decides M1 (never leave zero default). If the path being decommissioned is NOT
|
||||
@@ -423,6 +447,9 @@ func (s *Server) processGuestBootChange() {
|
||||
bootStacks = append(bootStacks, bootStack{
|
||||
name: st.Name, deployed: cfg.Deployed, hdd: cfg.Env["HDD_PATH"], state: string(st.State),
|
||||
hasContainers: len(st.Containers) > 0,
|
||||
// R-170: read intent from the app.yaml just loaded, not from st.AppConfig — cfg is the
|
||||
// fresh on-disk read this loop already performs, so the two cannot disagree.
|
||||
desired: cfg.DesiredState,
|
||||
})
|
||||
}
|
||||
recreate := func(bs bootStack) {
|
||||
@@ -443,7 +470,7 @@ func (s *Server) processGuestBootChange() {
|
||||
if leftStopped > 0 {
|
||||
// INFO, not WARN: this is the gate working as intended (R-55). Make the honoured path
|
||||
// observable — a silent correct path is how an inert seam hides.
|
||||
s.logger.Printf("[INFO] [gate] boot %s: %d drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose", resp.GuestBootID, leftStopped)
|
||||
s.logger.Printf("[INFO] [gate] boot %s: %d drive-backed app(s) left stopped on purpose (recorded Stop, or a legacy app.yaml at zero containers)", resp.GuestBootID, leftStopped)
|
||||
}
|
||||
if serr := s.settings.SetLastGuestBootID(resp.GuestBootID); serr != nil {
|
||||
s.logger.Printf("[WARN] [gate] persist boot-id: %v", serr)
|
||||
@@ -458,8 +485,13 @@ type bootStack struct {
|
||||
state string
|
||||
// hasContainers is len(Stack.Containers) > 0, from `docker ps -a` — so Exited containers COUNT.
|
||||
// R-55's running-at-shutdown signal: a UI Stop is `compose down` and leaves zero. MUST be sampled
|
||||
// before any recreate runs, since recreate's StopStack erases it.
|
||||
// before any recreate runs, since recreate's StopStack erases it. Since R-170 it is only consulted
|
||||
// for apps with NO recorded intent (legacy app.yaml), but it is still sampled for all of them —
|
||||
// the legacy branch needs it and the snapshot has exactly one chance to take it.
|
||||
hasContainers bool
|
||||
// desired is the customer's recorded intent (stacks.DesiredState*), "" when the app.yaml predates
|
||||
// v0.189.0. R-170: this is what the gate decides on now, with hasContainers as the legacy fallback.
|
||||
desired string
|
||||
}
|
||||
|
||||
// recreateDriveBackedApps recreates every deployed drive-backed app whose drive bind is live, then
|
||||
@@ -474,11 +506,14 @@ type bootStack struct {
|
||||
// intended behaviour.
|
||||
func recreateDriveBackedApps(stacks []bootStack, presentStable map[string]bool, recreate func(bootStack), syncFB func()) (recreated, skipped, leftStopped int) {
|
||||
for _, bs := range stacks {
|
||||
if !shouldRecreateOnBoot(bs.deployed, bs.hdd, presentStable, bs.hasContainers) {
|
||||
if !shouldRecreateOnBoot(bs.deployed, bs.hdd, presentStable, bs.hasContainers, bs.desired) {
|
||||
if bs.deployed && strings.HasPrefix(bs.hdd, StableParentDir+"/") {
|
||||
switch {
|
||||
case presentStable[bs.hdd] && !bs.hasContainers:
|
||||
leftStopped++ // drive IS live; the app is at zero containers → stopped on purpose
|
||||
case presentStable[bs.hdd]:
|
||||
// The drive IS live and we still declined: the customer's recorded Stop, or a legacy
|
||||
// app.yaml at zero containers. Both are "left stopped on purpose", which is what this
|
||||
// counter has always meant — only the signal behind it changed (R-170).
|
||||
leftStopped++
|
||||
default:
|
||||
skipped++ // a deployed drive-backed app whose bind never went live → gate's job
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user