v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s

R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills.
It emits the PRE-EXISTING disk_warning/disk_critical pair, which was
allowlisted, copy'd, default-enabled and checkbox'd with no producer in any
repo — the sixth "built but never wired" instance here. Two threshold terms
(85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at
both ends of this fleet's size range. Edge-triggered on escalation only,
state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil
usage read is never a warning and never clears one. Per filesystem, never
per app. Daily 03:30, before the nightly app-data legs.

R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture
fails, loop continuing, carrying the target filesystem's used/free bytes.
Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed,
which is customer-enabled and would email the customer about a failure they
cannot act on. D-c overrides R-158's own proposal here.

R-174: the app-stop guard no longer starts apps onto MISSING drives — a
regression in v0.189.0 code, found by review and closed the same session.
SetStarter got the raw stack manager, whose StartStack has no drive gate,
and Recover runs at startup. R-171 one path over. bootDriveGate could not be
reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read
vars assigned after Recover runs), so holder #3 is extracted into a shared
driveStartGate with a test pinning the delegation. ErrStartRefused splits a
refusal from a failure: both keep the marker, only Failed alarms, because
routing a deliberate hold into NotifyBackupFailed is the same false alarm.

Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
This commit is contained in:
2026-08-02 23:18:51 +02:00
parent 95eb5c2c1a
commit cf48214f6c
12 changed files with 1785 additions and 22 deletions
+32
View File
@@ -298,6 +298,38 @@ func (n *Notifier) NotifyBackupFailed(message, errMsg string) {
n.PushEvent("backup_failed", "error", message, BackupDetails{Error: errMsg})
}
// RecoveryUnitFailureDetails is the machine-readable tail of a Tier-1 capture failure. App NAMES and
// byte figures only — never an env value (§9.5).
type RecoveryUnitFailureDetails struct {
App string `json:"app"`
Error string `json:"error"`
TargetPath string `json:"target_path,omitempty"`
UsedGB float64 `json:"used_gb,omitempty"`
AvailGB float64 `json:"avail_gb,omitempty"`
TotalGB float64 `json:"total_gb,omitempty"`
UsedPercent float64 `json:"used_percent,omitempty"`
// SpaceKnown distinguishes "we read the filesystem and it says these numbers" from "we could not
// read it". Without it, an unreadable target is indistinguishable from an empty one — the
// presence-is-not-success trap, in the other direction.
SpaceKnown bool `json:"space_known"`
}
// NotifyRecoveryUnitCaptureFailed sends the OPERATOR-TIER alert for a per-app Tier-1 recovery-unit
// capture failure (R-158, D-c's operator half).
//
// DELIBERATELY NOT `backup_failed`. That type carries a `customerMessages` entry AND sits in
// `settings.DefaultEnabledEvents`, so reusing it would email the customer, in Hungarian, that their
// backup failed — an event they can take no action on. It is exactly the mistake R-97a avoided by
// minting `whole_guest_backup_failed`, and the reasoning is written into the hub's handler.go.
// R-158's original proposal named `backup_failed`; decision D-c routes this to the operator, and
// where the two disagree D-c wins.
//
// Operator-only is enforced by the hub's `notify.operatorOnlyEvents` register, NOT by the absence of
// a customerMessages entry — v0.78.0 claimed the latter and was wrong.
func (n *Notifier) NotifyRecoveryUnitCaptureFailed(message string, d RecoveryUnitFailureDetails) {
n.PushEvent("recovery_unit_capture_failed", "error", message, d)
}
// NotifyOffboxEnlargeBlocked sends a WARNING (not a failure) when an app's enlarged offsite push was
// refused by the pre-push quota gate — its config+DB were still saved. Customer-facing (Hungarian
// body). NOTE: the event type "offbox_enlarge_blocked" must be added to the hub's allowedEventTypes +