gate: the boot bind gate honours a customer's Stop (R-55, v0.157.0)

shouldRecreateOnBoot keyed on Deployed+drive-present alone. Deployed stays
true across a Stop, so a drive-backed app the customer switched off was
silently restarted on every guest reboot (proven live: immich).

Requires len(Containers)>0 as well - R-52's existing-Exited vs absent
distinction. A UI Stop is compose down and removes the containers; a guest
that went down under a running app leaves them. Container STATE is still
deliberately NOT a filter: that would miss a not-yet-restarted or stuck-Exited
app, which is the bug the boot-id path exists to fix.

Evidence sampled before any recreate - recreate's own StopStack erases it.
Honoured Stops counted and logged separately from no-live-bind skips.
This commit is contained in:
2026-07-21 14:53:11 +02:00
parent 83f20c8293
commit ac3790a11b
4 changed files with 200 additions and 41 deletions
+60 -14
View File
@@ -98,15 +98,39 @@ func agentWhere(registeredPath string) string {
}
// shouldRecreateOnBoot is the PURE decision for the boot-id recreate: on a fresh guest boot, recreate
// EVERY deployed drive-backed app whose drive is present (BoundUnderParent) onto its (re-propagated)
// drive. It is DETERMINISTIC — it depends ONLY on `app.yaml says should run` (deployed) + drive-present,
// NOT on the app's current container state. The current state must NOT be a filter: a momentarily-stopped
// app on a fresh reboot (docker hasn't auto-restarted it yet) would otherwise be MISSED — the exact bug
// the boot-id path replaces (the old container-uptime sample, and a State!=stopped filter, both miss it).
// (Tradeoff: a UI-stopped drive-backed app is brought back on a guest reboot — `deployed` is the only
// "should run" signal app.yaml carries; the gate manages drive-backed app lifecycle otherwise.)
func shouldRecreateOnBoot(deployed bool, hdd string, presentStable map[string]bool) bool {
return deployed && hdd != "" && strings.HasPrefix(hdd, StableParentDir+"/") && presentStable[hdd]
// a deployed drive-backed app whose drive is present (BoundUnderParent) onto its (re-propagated) drive.
//
// It is DETERMINISTIC and it still does NOT filter on the app's current container STATE. That remains
// load-bearing: a momentarily-stopped app on a fresh reboot (docker hasn't auto-restarted it yet, or
// its create-time bind failed with RestartCount=0) must be recreated, and both the old
// container-uptime sample and a `State != stopped` filter MISS exactly that case. Do not reintroduce
// a state filter here.
//
// R-55: it DOES filter on whether the app still HAS containers, which is a different question and the
// one that tells the truth about intent. This is R-52's `existing-Exited vs absent` distinction
// (bootrecon.isBootOrphan), translated to this gate:
//
// - containers EXIST but are down → the guest went down under the app; docker's own records survive
// the reboot, so this is a boot orphan → recreate.
// - ZERO containers → a UI Stop is `compose down`, which REMOVES the containers.
// Nothing else in the controller leaves a deployed app at zero containers. → the customer stopped
// this on purpose → LEAVE IT ALONE.
//
// `deployed` cannot answer this: it is a deploy-lifecycle flag and stays true across a Stop. Before
// R-55 the gate had no other signal and therefore silently undid a customer's Stop on every guest
// reboot — including when apps were stopped deliberately to free resources for others, which is
// precisely when resurrecting them is most harmful.
//
// The evidence is read from the snapshot taken BEFORE any recreate runs, because `recreate` itself
// calls StopStack (`compose down`) and so destroys it.
//
// NOTE on the drive-absent gate: apps it stopped are also at zero containers, so they are skipped
// here too. That is correct — they are recorded in StoragePath.StoppedStacks and restarted by
// ReconcileDriveGates' `Return` branch, which runs on the same loop tick. Their recovery is that
// path's job, not this one's.
func shouldRecreateOnBoot(deployed bool, hdd string, presentStable map[string]bool, hasContainers bool) bool {
return deployed && hdd != "" && strings.HasPrefix(hdd, StableParentDir+"/") &&
presentStable[hdd] && hasContainers
}
// defaultPromotionTarget decides M1 (never leave zero default). If the path being decommissioned is NOT
@@ -383,7 +407,11 @@ func (s *Server) processGuestBootChange() {
if cfg == nil {
continue
}
bootStacks = append(bootStacks, bootStack{name: st.Name, deployed: cfg.Deployed, hdd: cfg.Env["HDD_PATH"], state: string(st.State)})
// hasContainers is sampled HERE, before any recreate — recreate's StopStack destroys it (R-55).
bootStacks = append(bootStacks, bootStack{
name: st.Name, deployed: cfg.Deployed, hdd: cfg.Env["HDD_PATH"], state: string(st.State),
hasContainers: len(st.Containers) > 0,
})
}
recreate := func(bs bootStack) {
s.logger.Printf("[INFO] [gate] boot %s: live bind confirmed — recreating drive-backed app %s (state=%s) onto %s", resp.GuestBootID, bs.name, bs.state, bs.hdd)
@@ -396,10 +424,15 @@ func (s *Server) processGuestBootChange() {
s.logger.Printf("[INFO] [gate] boot %s: re-syncing FileBrowser mounts against the live binds", resp.GuestBootID)
go s.SyncFileBrowserMounts()
}
_, skipped := recreateDriveBackedApps(bootStacks, presentStable, recreate, syncFB)
_, skipped, leftStopped := recreateDriveBackedApps(bootStacks, presentStable, recreate, syncFB)
if skipped > 0 {
s.logger.Printf("[WARN] [gate] boot %s: %d drive-backed app(s) had no live bind within %s — leaving to the drive gate", resp.GuestBootID, skipped, bootBindWait)
}
if leftStopped > 0 {
// INFO, not WARN: this is the gate working as intended (R-55). Make the honoured path
// observable — a silent correct path is how an inert seam hides.
s.logger.Printf("[INFO] [gate] boot %s: %d drive-backed app(s) left stopped — zero containers means the customer stopped them on purpose", resp.GuestBootID, leftStopped)
}
if serr := s.settings.SetLastGuestBootID(resp.GuestBootID); serr != nil {
s.logger.Printf("[WARN] [gate] persist boot-id: %v", serr)
}
@@ -411,6 +444,10 @@ type bootStack struct {
deployed bool
hdd string
state string
// hasContainers is len(Stack.Containers) > 0, from `docker ps -a` — so Exited containers COUNT.
// R-55's running-at-shutdown signal: a UI Stop is `compose down` and leaves zero. MUST be sampled
// before any recreate runs, since recreate's StopStack erases it.
hasContainers bool
}
// recreateDriveBackedApps recreates every deployed drive-backed app whose drive bind is live, then
@@ -419,11 +456,20 @@ type bootStack struct {
// ran once pollLiveBinds confirmed the live binds), so FileBrowser's mounts reflect the now-live drives
// instead of going stale (the gap a host/guest reboot left before this fix). syncFB is always called so
// FileBrowser reflects the current bind state even if no app needed recreating. Pure (ops injected).
func recreateDriveBackedApps(stacks []bootStack, presentStable map[string]bool, recreate func(bootStack), syncFB func()) (recreated, skipped int) {
// R-55: `leftStopped` counts drive-backed apps deliberately NOT touched because they have zero
// containers (a customer Stop). It is reported separately from `skipped` — conflating the two would
// make an honoured Stop look like the "bind never went live" failure and fire a WARN for healthy,
// intended behaviour.
func recreateDriveBackedApps(stacks []bootStack, presentStable map[string]bool, recreate func(bootStack), syncFB func()) (recreated, skipped, leftStopped int) {
for _, bs := range stacks {
if !shouldRecreateOnBoot(bs.deployed, bs.hdd, presentStable) {
if !shouldRecreateOnBoot(bs.deployed, bs.hdd, presentStable, bs.hasContainers) {
if bs.deployed && strings.HasPrefix(bs.hdd, StableParentDir+"/") {
skipped++ // a deployed drive-backed app whose bind never went live → gate's job
switch {
case presentStable[bs.hdd] && !bs.hasContainers:
leftStopped++ // drive IS live; the app is at zero containers → stopped on purpose
default:
skipped++ // a deployed drive-backed app whose bind never went live → gate's job
}
}
continue
}