v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s
gates / gates (push) Successful in 9s
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills. It emits the PRE-EXISTING disk_warning/disk_critical pair, which was allowlisted, copy'd, default-enabled and checkbox'd with no producer in any repo — the sixth "built but never wired" instance here. Two threshold terms (85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at both ends of this fleet's size range. Edge-triggered on escalation only, state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil usage read is never a warning and never clears one. Per filesystem, never per app. Daily 03:30, before the nightly app-data legs. R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture fails, loop continuing, carrying the target filesystem's used/free bytes. Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed, which is customer-enabled and would email the customer about a failure they cannot act on. D-c overrides R-158's own proposal here. R-174: the app-stop guard no longer starts apps onto MISSING drives — a regression in v0.189.0 code, found by review and closed the same session. SetStarter got the raw stack manager, whose StartStack has no drive gate, and Recover runs at startup. R-171 one path over. bootDriveGate could not be reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read vars assigned after Recover runs), so holder #3 is extracted into a shared driveStartGate with a test pinning the delegation. ErrStartRefused splits a refusal from a failure: both keep the marker, only Failed alarms, because routing a deliberate hold into NotifyBackupFailed is the same false alarm. Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
This commit is contained in:
@@ -298,6 +298,38 @@ func (n *Notifier) NotifyBackupFailed(message, errMsg string) {
|
||||
n.PushEvent("backup_failed", "error", message, BackupDetails{Error: errMsg})
|
||||
}
|
||||
|
||||
// RecoveryUnitFailureDetails is the machine-readable tail of a Tier-1 capture failure. App NAMES and
|
||||
// byte figures only — never an env value (§9.5).
|
||||
type RecoveryUnitFailureDetails struct {
|
||||
App string `json:"app"`
|
||||
Error string `json:"error"`
|
||||
TargetPath string `json:"target_path,omitempty"`
|
||||
UsedGB float64 `json:"used_gb,omitempty"`
|
||||
AvailGB float64 `json:"avail_gb,omitempty"`
|
||||
TotalGB float64 `json:"total_gb,omitempty"`
|
||||
UsedPercent float64 `json:"used_percent,omitempty"`
|
||||
// SpaceKnown distinguishes "we read the filesystem and it says these numbers" from "we could not
|
||||
// read it". Without it, an unreadable target is indistinguishable from an empty one — the
|
||||
// presence-is-not-success trap, in the other direction.
|
||||
SpaceKnown bool `json:"space_known"`
|
||||
}
|
||||
|
||||
// NotifyRecoveryUnitCaptureFailed sends the OPERATOR-TIER alert for a per-app Tier-1 recovery-unit
|
||||
// capture failure (R-158, D-c's operator half).
|
||||
//
|
||||
// DELIBERATELY NOT `backup_failed`. That type carries a `customerMessages` entry AND sits in
|
||||
// `settings.DefaultEnabledEvents`, so reusing it would email the customer, in Hungarian, that their
|
||||
// backup failed — an event they can take no action on. It is exactly the mistake R-97a avoided by
|
||||
// minting `whole_guest_backup_failed`, and the reasoning is written into the hub's handler.go.
|
||||
// R-158's original proposal named `backup_failed`; decision D-c routes this to the operator, and
|
||||
// where the two disagree D-c wins.
|
||||
//
|
||||
// Operator-only is enforced by the hub's `notify.operatorOnlyEvents` register, NOT by the absence of
|
||||
// a customerMessages entry — v0.78.0 claimed the latter and was wrong.
|
||||
func (n *Notifier) NotifyRecoveryUnitCaptureFailed(message string, d RecoveryUnitFailureDetails) {
|
||||
n.PushEvent("recovery_unit_capture_failed", "error", message, d)
|
||||
}
|
||||
|
||||
// NotifyOffboxEnlargeBlocked sends a WARNING (not a failure) when an app's enlarged offsite push was
|
||||
// refused by the pre-push quota gate — its config+DB were still saved. Customer-facing (Hungarian
|
||||
// body). NOTE: the event type "offbox_enlarge_blocked" must be added to the hub's allowedEventTypes +
|
||||
|
||||
Reference in New Issue
Block a user