v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)

R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills.
It emits the PRE-EXISTING disk_warning/disk_critical pair, which was
allowlisted, copy'd, default-enabled and checkbox'd with no producer in any
repo — the sixth "built but never wired" instance here. Two threshold terms
(85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at
both ends of this fleet's size range. Edge-triggered on escalation only,
state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil
usage read is never a warning and never clears one. Per filesystem, never
per app. Daily 03:30, before the nightly app-data legs.

R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture
fails, loop continuing, carrying the target filesystem's used/free bytes.
Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed,
which is customer-enabled and would email the customer about a failure they
cannot act on. D-c overrides R-158's own proposal here.

R-174: the app-stop guard no longer starts apps onto MISSING drives — a
regression in v0.189.0 code, found by review and closed the same session.
SetStarter got the raw stack manager, whose StartStack has no drive gate,
and Recover runs at startup. R-171 one path over. bootDriveGate could not be
reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read
vars assigned after Recover runs), so holder #3 is extracted into a shared
driveStartGate with a test pinning the delegation. ErrStartRefused splits a
refusal from a failure: both keep the marker, only Failed alarms, because
routing a deliberate hold into NotifyBackupFailed is the same false alarm.

Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.

controller/internal/backup/appstop_marker.go

@@ -2,6 +2,7 @@ package backup
 
 import (
 	"encoding/json"
+	"errors"
 	"fmt"
 	"log"
 	"os"
@@ -72,10 +73,27 @@ type AppStopMarker struct {
 
 // AppStopStarter is the one thing recovery needs: the ability to start a stack. StartStack must be
 // idempotent (it is — `compose up -d` on a running stack is a no-op).
+//
+// R-174: production MUST pass a GATED starter, never the raw stack manager. Recover runs at STARTUP —
+// exactly when an external drive may not have come back — and `Manager.StartStack` has no drive gate
+// of its own. See `gatedAppStopStarter` in cmd/controller/main.go.
 type AppStopStarter interface {
 	StartStack(name string) error
 }
 
+// ErrStartRefused is what a gated starter returns when a DELIBERATE HOLDER — today the drive gate —
+// says an app must not be started. Wrap it (`fmt.Errorf("%w: …", ErrStartRefused)`) so the reason
+// survives; Recover matches with errors.Is.
+//
+// IT IS NOT A FAILURE, AND THE DISTINCTION IS THE WHOLE POINT OF THE TYPE. A refusal means the
+// holder is doing its job and owns the restart; a failure means the restart was attempted and broke.
+// Collapsing the two would put a deliberately-held app into `Failed`, which main.go reports through
+// `NotifyBackupFailed` — a type that is customer-enabled by default (`settings.DefaultEnabledEvents`)
+// and carries the Hungarian "A biztonsági mentés sikertelen!". That is R-171's defect one path over:
+// a false alarm about an app the drive gate is deliberately holding. Both buckets keep the marker;
+// only `Failed` alarms.
+var ErrStartRefused = errors.New("start refused by a deliberate holder")
+
 // AppStopGuard owns one marker file. Construct with NewAppStopGuard; the zero value is inert (every
 // method is a no-op on a nil guard), so a caller that was never wired degrades to pre-v0.189.0
 // behaviour instead of panicking.
@@ -98,7 +116,21 @@ type AppStopRecovery struct {
 	OpID      string
 	StartedAt time.Time
 	Restarted []string // apps started again by this recovery
-	Failed    []string // apps that could NOT be restarted (the marker was kept for these)
+	Failed    []string // apps whose restart was ATTEMPTED and broke (the marker was kept for these)
+	// Refused are apps a deliberate holder said must not start — today, an absent data drive
+	// (R-174). The marker is kept for these too, but they are NOT a fault and MUST NOT alarm: the
+	// holder owns the restart. Separate from Failed for the reason recorded on ErrStartRefused.
+	Refused []string
+}
+
+// Alarming reports whether this recovery is worth paging an operator about. A recovery that only
+// REFUSED starts is the drive gate working as designed, and reporting it through the customer-enabled
+// `backup_failed` type would be the R-171 false alarm one path over.
+func (r *AppStopRecovery) Alarming() bool {
+	if r == nil {
+		return false
+	}
+	return len(r.Failed) > 0 || len(r.Restarted) > 0
 }
 
 // Message is the operator-facing headline for an interrupted operation.
@@ -107,11 +139,23 @@ func (r *AppStopRecovery) Message() string {
 		return ""
 	}
 	if len(r.Failed) > 0 {
-		return fmt.Sprintf("%s was interrupted by a controller restart and %d of %d app(s) could NOT be restarted",
+		m := fmt.Sprintf("%s was interrupted by a controller restart and %d of %d app(s) could NOT be restarted",
 			r.Reason.humanReason(), len(r.Failed), len(r.Restarted)+len(r.Failed))
+		if len(r.Refused) > 0 {
+			m += fmt.Sprintf(" (a further %d are held by an absent drive and are not counted as failures)", len(r.Refused))
+		}
+		return m
 	}
-	return fmt.Sprintf("%s was interrupted by a controller restart — %d app(s) were left stopped and have been restarted",
+	if len(r.Refused) > 0 && len(r.Restarted) == 0 {
+		return fmt.Sprintf("%s was interrupted by a controller restart — %d app(s) are left stopped and HELD: their data drive is not available, so the drive gate restarts them when it returns",
+			r.Reason.humanReason(), len(r.Refused))
+	}
+	m := fmt.Sprintf("%s was interrupted by a controller restart — %d app(s) were left stopped and have been restarted",
 		r.Reason.humanReason(), len(r.Restarted))
+	if len(r.Refused) > 0 {
+		m += fmt.Sprintf("; %d more are held by an absent drive", len(r.Refused))
+	}
+	return m
 }
 
 // Detail is the machine-readable tail. App/stack NAMES only — never env values (§9.5).
@@ -124,6 +168,9 @@ func (r *AppStopRecovery) Detail() string {
 	if len(r.Failed) > 0 {
 		d += fmt.Sprintf(" restart_failed=%v", r.Failed)
 	}
+	if len(r.Refused) > 0 {
+		d += fmt.Sprintf(" held_by_drive=%v", r.Refused)
+	}
 	return d
 }
 
@@ -210,6 +257,15 @@ func (g *AppStopGuard) Recover() *AppStopRecovery {
 	res := &AppStopRecovery{Reason: m.Reason, OpID: m.OpID, StartedAt: m.StartedAt}
 	for _, name := range m.Stacks {
 		if err := g.starter.StartStack(name); err != nil {
+			// R-174: a REFUSAL is not a failure. The starter's gate has said this app must not be
+			// started (an absent data drive), so the app is left down deliberately and the holder
+			// owns the restart. Logged at WARN with the reason, and kept out of Failed so it never
+			// reaches the customer-enabled backup_failed alarm — see ErrStartRefused.
+			if errors.Is(err, ErrStartRefused) {
+				g.logger.Printf("[WARN] [appstop] crash recovery: NOT restarting %s — %v; the marker is KEPT and the holder owns the restart", name, err)
+				res.Refused = append(res.Refused, name)
+				continue
+			}
 			g.logger.Printf("[ERROR] [appstop] crash recovery: restart %s failed: %v", name, err)
 			res.Failed = append(res.Failed, name)
 			continue
@@ -218,13 +274,23 @@ func (g *AppStopGuard) Recover() *AppStopRecovery {
 		res.Restarted = append(res.Restarted, name)
 	}
 	sort.Strings(res.Failed)
+	sort.Strings(res.Refused)
 	sort.Strings(res.Restarted)
 
+	// The marker is kept for BOTH unfinished outcomes, for the same reason and with different
+	// urgency: a failed restart is retried next startup, and a refused one is genuinely unfinished
+	// until its drive returns. Clearing it in either case would erase the only durable record that
+	// an app is owed a restart.
 	if len(res.Failed) > 0 {
 		g.logger.Printf("[ERROR] [appstop] crash recovery: %d app(s) could not be restarted — KEEPING the marker so the next startup retries; the dead-app alarm owns them meanwhile: %v",
 			len(res.Failed), res.Failed)
 		return res
 	}
+	if len(res.Refused) > 0 {
+		g.logger.Printf("[WARN] [appstop] crash recovery: %d app(s) were deliberately NOT restarted (drive absent) — KEEPING the marker; this is the gate working, not a fault: %v",
+			len(res.Refused), res.Refused)
+		return res
+	}
 	g.End()
 	return res
 }