v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s
gates / gates (push) Successful in 9s
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills. It emits the PRE-EXISTING disk_warning/disk_critical pair, which was allowlisted, copy'd, default-enabled and checkbox'd with no producer in any repo — the sixth "built but never wired" instance here. Two threshold terms (85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at both ends of this fleet's size range. Edge-triggered on escalation only, state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil usage read is never a warning and never clears one. Per filesystem, never per app. Daily 03:30, before the nightly app-data legs. R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture fails, loop continuing, carrying the target filesystem's used/free bytes. Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed, which is customer-enabled and would email the customer about a failure they cannot act on. D-c overrides R-158's own proposal here. R-174: the app-stop guard no longer starts apps onto MISSING drives — a regression in v0.189.0 code, found by review and closed the same session. SetStarter got the raw stack manager, whose StartStack has no drive gate, and Recover runs at startup. R-171 one path over. bootDriveGate could not be reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read vars assigned after Recover runs), so holder #3 is extracted into a shared driveStartGate with a test pinning the delegation. ErrStartRefused splits a refusal from a failure: both keep the marker, only Failed alarms, because routing a deliberate hold into NotifyBackupFailed is the same false alarm. Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
This commit is contained in:
@@ -32,6 +32,27 @@ type Manager struct {
|
||||
// tier2Notify, if set, is called after each Tier 2 copy (success: err==nil) for notifications.
|
||||
tier2Notify func(stackName, destLabel string, dur time.Duration, err error)
|
||||
|
||||
// unitNotify (R-158 / R-167), if set, is called ONCE PER APP whose Tier-1 recovery-unit capture
|
||||
// FAILED, and the capture loop continues to the next app. Wired in cmd/controller/main.go.
|
||||
//
|
||||
// WHY IT EXISTS. `/backups/apps` is the page a person opens to ask whether ONE app is backed up,
|
||||
// and until now it was the one page that never said: a per-app capture failure was a `[WARN]`
|
||||
// line and went no further. The manager had three notify seams and none for the unit capture —
|
||||
// the FIFTH instance in this project of a mechanism built and left disconnected.
|
||||
//
|
||||
// IT CARRIES THE SPACE FIGURES DELIBERATELY. The overwhelmingly likely cause is a full
|
||||
// filesystem, and an operator who has the used/free bytes at the moment of failure can act
|
||||
// without logging in. It is the same pair of numbers the customer-facing fill warning reports,
|
||||
// which is why the two ship together.
|
||||
//
|
||||
// OPERATOR-TIER. Routed to a hub event type that is in `notify.operatorOnlyEvents` — a customer
|
||||
// can take no action on a capture failure. Deliberately NOT `backup_failed`, which is
|
||||
// customer-enabled by default and would email them in Hungarian about it (D-c).
|
||||
//
|
||||
// NO CONTROLLER-SIDE COOLDOWN — the hub owns cooldown, per the offboxEnlargeBlockedNotify
|
||||
// precedent.
|
||||
unitNotify func(stackName string, err error, usage *UnitSpace)
|
||||
|
||||
// appStop (R-166) is the crash marker for operations that stop an app, work on its data, and
|
||||
// start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power
|
||||
// cut in that window leaves a durable record that Recover honours at the next startup. Built in
|
||||
|
||||
Reference in New Issue
Block a user