v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)

R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills.
It emits the PRE-EXISTING disk_warning/disk_critical pair, which was
allowlisted, copy'd, default-enabled and checkbox'd with no producer in any
repo — the sixth "built but never wired" instance here. Two threshold terms
(85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at
both ends of this fleet's size range. Edge-triggered on escalation only,
state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil
usage read is never a warning and never clears one. Per filesystem, never
per app. Daily 03:30, before the nightly app-data legs.

R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture
fails, loop continuing, carrying the target filesystem's used/free bytes.
Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed,
which is customer-enabled and would email the customer about a failure they
cannot act on. D-c overrides R-158's own proposal here.

R-174: the app-stop guard no longer starts apps onto MISSING drives — a
regression in v0.189.0 code, found by review and closed the same session.
SetStarter got the raw stack manager, whose StartStack has no drive gate,
and Recover runs at startup. R-171 one path over. bootDriveGate could not be
reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read
vars assigned after Recover runs), so holder #3 is extracted into a shared
driveStartGate with a test pinning the delegation. ErrStartRefused splits a
refusal from a failure: both keep the marker, only Failed alarms, because
routing a deliberate hold into NotifyBackupFailed is the same false alarm.

Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.

controller/cmd/controller/main.go

@@ -33,6 +33,7 @@ import (
 	cf "gitea.dooplex.hu/admin/felhom-controller/internal/cloudflare"
 	"gitea.dooplex.hu/admin/felhom-controller/internal/config"
 	"gitea.dooplex.hu/admin/felhom-controller/internal/crypto"
+	"gitea.dooplex.hu/admin/felhom-controller/internal/fillwatch"
 	"gitea.dooplex.hu/admin/felhom-controller/internal/infra"
 	"gitea.dooplex.hu/admin/felhom-controller/internal/integrations"
 	"gitea.dooplex.hu/admin/felhom-controller/internal/mailrelay"
@@ -242,8 +243,17 @@ func main() {
 	// constructed until ~40 lines below — and moving its construction up to suit this would be a far
 	// wider change than moving one object down. It is handed to the manager (SetAppStopGuard) and to
 	// the exporter later, so all three share ONE guard over ONE file.
+	//
+	// R-174: the starter is GATED, never the raw manager. Recover runs here, at startup — exactly
+	// when an external drive may not have come back — and `Manager.StartStack` has no drive gate of
+	// its own, so the un-gated version started apps onto missing drives. The gate is the SAME
+	// `driveStartGate` the boot sweep uses (holder #3 of bootDriveGate), so the two cannot disagree.
 	appStopGuard := backup.NewAppStopGuard(filepath.Join(cfg.Paths.DataDir, "appstop-state.json"), logger)
-	appStopGuard.SetStarter(stackMgr)
+	appStopGuard.SetStarter(gatedAppStopStarter{
+		inner:  stackMgr,
+		gate:   driveStartGate{mgr: stackMgr, sett: sett},
+		logger: logger,
+	})
 	appStopRecovery := appStopGuard.Recover()
 
 	// --- R-166: desired-state backfill (running-only) ---
@@ -358,8 +368,14 @@ func main() {
 	// hub's allowedEventTypes + customerMessages pair changed, which is a wire change, and this
 	// release ships no hub change. A controller emitting an unlisted type gets a flat 400 from
 	// POST /event. Reachability is covered by TestAppStopRecoveryIsWired.
-	if appStopRecovery != nil {
+	// R-174: `Alarming()` and not merely `!= nil`. A recovery that ONLY refused starts is the drive
+	// gate working as designed, and `backup_failed` is customer-enabled by default — reporting a
+	// deliberate hold through it would email the customer "A biztonsági mentés sikertelen!" about an
+	// app nothing is wrong with. The refusal is already logged at WARN with its reason.
+	if appStopRecovery != nil && appStopRecovery.Alarming() {
 		notifier.NotifyBackupFailed(appStopRecovery.Message(), appStopRecovery.Detail())
+	} else if appStopRecovery != nil {
+		logger.Printf("[WARN] [appstop] %s — not alarming: %s", appStopRecovery.Message(), appStopRecovery.Detail())
 	}
 
 	// --- Initialize the app-email SMTP shim (mailrelay) ---
@@ -708,6 +724,26 @@ func main() {
 					backupMgr.OffsiteFailureMessage(err, dur))
 			}
 		})
+		// R-158 / R-167 (D-c, operator half): a per-app Tier-1 recovery-unit capture failed. Until
+		// v0.191.0 this was a `[WARN]` line and nothing else — /backups/apps is the page you open to
+		// ask whether ONE app is backed up, and it was the one page that never said. Fires per app;
+		// the capture loop continues, so three failing apps produce three events and one failing app
+		// does not silence its siblings.
+		//
+		// OPERATOR-TIER, not backup_failed — a customer can take no action on a capture failure, and
+		// backup_failed is customer-enabled by default. The space figures ride along because the
+		// overwhelmingly likely cause is a full filesystem and they answer "why" without a login.
+		// No controller-side cooldown: the hub owns it.
+		backupMgr.SetUnitNotify(func(stackName string, err error, usage *backup.UnitSpace) {
+			d := notify.RecoveryUnitFailureDetails{App: stackName, Error: err.Error()}
+			if usage != nil {
+				d.TargetPath, d.UsedGB, d.AvailGB = usage.Path, usage.UsedGB, usage.AvailGB
+				d.TotalGB, d.UsedPercent, d.SpaceKnown = usage.TotalGB, usage.UsedPercent, true
+			}
+			notifier.NotifyRecoveryUnitCaptureFailed(fmt.Sprintf(
+				"Recovery unit capture FAILED for %q — the app has no fresh local (Tier-1) backup, and Tier-2/Tier-3 have nothing to copy. %s. Error: %v",
+				stackName, usage.String(), err), d)
+		})
 		// 3a: the pre-push enlargement gate blocked an app's userdata push (config+DB still saved). Edge-
 		// triggered by the engine (only NEW blocks notify), so the hub's per-event-type cooldown suffices —
 		// no controller-side timer (the hub owns cooldown).
@@ -749,6 +785,49 @@ func main() {
 		})
 	}
 
+	// --- R-167 (decision D-c, customer half): warn BEFORE a filesystem fills ---
+	//
+	// Nothing warned before v0.191.0. The first sign of a full filesystem was a backup that did not
+	// happen, and the only related signal — the healthcheck's generic `health_degraded` at 90% — looked
+	// at REGISTERED STORAGE PATHS ONLY, so the docker area and the system-data area (the one holding
+	// every driveless app's recovery unit) were invisible, and it never reported a free-byte figure or
+	// named a drive.
+	//
+	// CADENCE: DAILY, at 03:30. A fill is a slow-moving quantity — the thing that fills a disk is a
+	// customer's photo library or a nightly backup, not a spike — so a shorter interval buys no
+	// earlier warning and only costs statfs calls. 03:30 is deliberately BEFORE the nightly app-data
+	// legs (db-dump / tier2 / offbox), so a customer who is about to lose a backup to lack of space
+	// hears about it while there is still a night's margin, rather than after the failure.
+	// The interval is NOT a cooldown: repeats are impossible because the check is edge-triggered per
+	// filesystem, and the hub owns cooldown regardless.
+	fillWatcher := fillwatch.New(
+		filepath.Join(cfg.Paths.DataDir, "fillwatch-state.json"), logger,
+		func() []fillwatch.Target { return fillTargets(cfg, sett) },
+		func(p string) *fillwatch.Usage {
+			di := system.GetDiskUsage(p)
+			if di == nil {
+				return nil // §8.4 — unreadable is NOT full; never fabricate a zero here
+			}
+			return &fillwatch.Usage{
+				UsedPercent: di.UsedPercent, AvailGB: di.AvailGB,
+				UsedGB: di.UsedGB, TotalGB: di.TotalGB,
+			}
+		})
+	if notifier != nil {
+		fillWatcher.SetNotify(func(e fillwatch.Event) {
+			// The CUSTOMER's event — deliberately not operator-only. A customer can free space,
+			// delete files or add a drive, so this alert is theirs (D-c). The dynamic Hungarian
+			// message carries the label and the free space; the hub has NO customerMessages entry for
+			// these two types precisely so the template cannot discard them.
+			notifier.PushEvent(e.Band.EventType(), e.Band.Severity(), e.Message, map[string]any{
+				"path": e.Target.Path, "label": e.Target.Label,
+				"used_percent": e.Usage.UsedPercent, "avail_gb": e.Usage.AvailGB,
+				"total_gb": e.Usage.TotalGB, "band": e.Band.String(),
+			})
+		})
+	}
+	sched.Daily("fill-watch", "03:30", func(ctx context.Context) error { return fillWatcher.Check() })
+
 	// --- Central hub reporting schedule ---
 	if hubPusher != nil {
 		if cfg.Hub.Enabled {
@@ -1312,7 +1391,7 @@ var bootReconcileFn = func(ctx context.Context, mgr bootrecon.StackProvider, log
 	// R-171: the sweep must not start an app whose data drive is absent. Wired HERE, at the one
 	// place the sweep is constructed, so there is no path that builds an ungated reconciler.
 	if sm, ok := mgr.(*stacks.Manager); ok {
-		r.SetDriveGate(bootDriveGate{mgr: sm, sett: bootDriveSettings})
+		r.SetDriveGate(bootDriveGate{drive: driveStartGate{mgr: sm, sett: bootDriveSettings}})
 	}
 	return r.Run(ctx)
 }
@@ -1352,8 +1431,7 @@ var (
 //
 // Fail-safe per bootrecon.StartGate's contract: anything that cannot be determined returns false.
 type bootDriveGate struct {
-	mgr  *stacks.Manager
-	sett *settings.Settings
+	drive driveStartGate
 }
 
 func (g bootDriveGate) MayStart(stackName string) (bool, string) {
@@ -1367,7 +1445,35 @@ func (g bootDriveGate) MayStart(stackName string) (bool, string) {
 			return false, "an app-data operation is holding it — the app-stop guard restarts it when the operation ends"
 		}
 	}
-	// 3. the drive
+	// 3. the drive — the SHARED predicate, so this gate and the app-stop guard's crash recovery
+	//    (R-174) cannot disagree about whether an app's drive is available.
+	return g.drive.MayStart(stackName)
+}
+
+// driveStartGate is holder #3 of bootDriveGate, EXTRACTED so it has two callers and one
+// implementation (R-174).
+//
+// WHY IT IS SEPARATE FROM bootDriveGate RATHER THAN REUSED WHOLE. The app-stop guard's Recover needs
+// exactly this question and NOT the other two holders:
+//
+//   - Holder #2 reads `bootAppStopGuard.HeldStacks()`, which during Recover is the guard's OWN
+//     marker — the very stacks being recovered. Reusing bootDriveGate there would refuse every
+//     recovery it was meant to perform, self-referentially.
+//   - Holders #1 and #2 read package-level vars assigned in main() at the `bootDriveSettings` block,
+//     which runs AFTER `appStopGuard.Recover()`. They are nil at recovery time, so a whole-gate
+//     reuse would be correct only by accident of nil-safety — and would silently invert the moment
+//     anyone moved either line. This project has shipped that class of accident before.
+//
+// Fail-safe per bootrecon.StartGate's contract: anything that cannot be determined returns false.
+type driveStartGate struct {
+	mgr  *stacks.Manager
+	sett *settings.Settings
+}
+
+func (g driveStartGate) MayStart(stackName string) (bool, string) {
+	if g.mgr == nil {
+		return false, "no stack manager wired — the drive cannot be determined"
+	}
 	cfg := g.mgr.LoadAppConfigByName(stackName)
 	if cfg == nil {
 		// CANNOT DETERMINE. A deployed app whose app.yaml will not load cannot have its drive
@@ -1397,6 +1503,76 @@ func (g bootDriveGate) MayStart(stackName string) (bool, string) {
 	return true, ""
 }
 
+// gatedAppStopStarter is the app-stop guard's starter, wrapped in the drive gate (R-174).
+//
+// THE DEFECT IT CLOSES, found by review on 2026-08-02 in code shipped 2026-08-01 (v0.189.0):
+// `appStopGuard.SetStarter(stackMgr)` handed Recover the RAW stack manager, whose `StartStack` has
+// no drive gate. Recover runs at startup — precisely when an external drive may not have come back —
+// so a backup that stopped an app, followed by a power cut and a drive that did not remount, ended
+// with the app started onto a missing drive. That is R-171 one path over, and the rule is not new:
+// the API's own `startGatedByMissingDrive` already refuses this to the customer.
+//
+// The check stays HERE, per-caller, and is deliberately NOT pushed into `Manager.StartStack` — it has
+// fourteen callers and most of them legitimately start apps outside this concern.
+type gatedAppStopStarter struct {
+	inner  backup.AppStopStarter
+	gate   driveStartGate
+	logger *log.Logger
+}
+
+func (s gatedAppStopStarter) StartStack(name string) error {
+	if ok, why := s.gate.MayStart(name); !ok {
+		s.logger.Printf("[WARN] [appstop] refusing to restart %q after an interrupted operation: %s", name, why)
+		return fmt.Errorf("%w: %s", backup.ErrStartRefused, why)
+	}
+	return s.inner.StartStack(name)
+}
+
+// fillTargets is §8.1's watch list: the app-data volume, the system-data volume, and every
+// registered drive. Resolved at CHECK time, not at startup, so a drive added or decommissioned
+// between checks is picked up without a controller restart.
+//
+// WHY THESE THREE AND NOT ONLY THE REGISTERED DRIVES — which is all the healthcheck ever looked at:
+//
+//   - the APP-DATA volume (`mp0`, `/var/lib/docker`) holds every app's live named volumes;
+//   - the SYSTEM-DATA volume (`mp1`, `/mnt/sys_drive`) holds the retained recovery unit of every
+//     DRIVELESS app (architecture/07-backup-architecture.md §7.5) and is the smaller of the two at
+//     20 G against 50 G — the very mismatch R-165 exists to remove. It is also the filesystem whose
+//     silent exhaustion R-158 measured;
+//   - the registered DRIVES hold the customer's own data.
+//
+// De-duplicated by path: on a box where a drive is not a separate mount these collapse, and warning
+// twice about one filesystem is exactly the noise the per-filesystem rule exists to prevent. A
+// DECOMMISSIONED drive is dropped — it is deliberately out of service, and its fill is not news. A
+// DISCONNECTED one is kept in the list but will read as unreadable and be skipped by §8.4, which is
+// the correct outcome: its absence already has its own alert.
+func fillTargets(cfg *config.Config, sett *settings.Settings) []fillwatch.Target {
+	seen := make(map[string]bool)
+	var out []fillwatch.Target
+	add := func(path, label string) {
+		if path == "" || seen[path] {
+			return
+		}
+		seen[path] = true
+		out = append(out, fillwatch.Target{Path: path, Label: label})
+	}
+
+	// The docker data-root as seen from inside the guest, and the system-data area.
+	add(system.DockerVolumePath, "Alkalmazások területe")
+	if cfg != nil {
+		add(cfg.Paths.SystemDataPath, "Rendszer- és mentési terület")
+	}
+	if sett != nil {
+		for _, sp := range sett.GetStoragePaths() {
+			if sp.Decommissioned {
+				continue
+			}
+			add(sp.Path, sp.Label)
+		}
+	}
+	return fillwatch.SortTargets(out)
+}
+
 // bootFleetSample is a comparable snapshot of one deployed app — name, state and container count,
 // per §8.1. Container count is in it deliberately: a stack can go from 3 containers to 0 without its
 // aggregate state changing, and that IS the boot still moving.