v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s
gates / gates (push) Successful in 9s
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills. It emits the PRE-EXISTING disk_warning/disk_critical pair, which was allowlisted, copy'd, default-enabled and checkbox'd with no producer in any repo — the sixth "built but never wired" instance here. Two threshold terms (85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at both ends of this fleet's size range. Edge-triggered on escalation only, state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil usage read is never a warning and never clears one. Per filesystem, never per app. Daily 03:30, before the nightly app-data legs. R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture fails, loop continuing, carrying the target filesystem's used/free bytes. Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed, which is customer-enabled and would email the customer about a failure they cannot act on. D-c overrides R-158's own proposal here. R-174: the app-stop guard no longer starts apps onto MISSING drives — a regression in v0.189.0 code, found by review and closed the same session. SetStarter got the raw stack manager, whose StartStack has no drive gate, and Recover runs at startup. R-171 one path over. bootDriveGate could not be reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read vars assigned after Recover runs), so holder #3 is extracted into a shared driveStartGate with a test pinning the delegation. ErrStartRefused splits a refusal from a failure: both keep the marker, only Failed alarms, because routing a deliberate hold into NotifyBackupFailed is the same false alarm. Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
This commit is contained in:
@@ -1,5 +1,93 @@
|
||||
## Changelog
|
||||
|
||||
### v0.191.0 — warn before the wall comes down (2026-08-02, R-167 · R-158 · R-174) — MinAgent: none
|
||||
|
||||
**Storage monitoring and backup alerts, decision D-c, landing BEFORE the `mp1`→`mp0` merge (D-a /
|
||||
R-165) rather than with it.** D-a's own condition (2) says the monitoring ships in the same step and
|
||||
never after, because the merge removes a wall that currently fails safely. Landing it first is
|
||||
strictly better and costs nothing: the warnings go in and get proven on hardware while the wall is
|
||||
still standing. **No disk layout is touched in this release.**
|
||||
|
||||
**R-167 — the customer is warned BEFORE a filesystem fills, and the warning is the pair that already
|
||||
existed.** New `internal/fillwatch`. Nothing warned before this: the first sign of a full filesystem
|
||||
was a backup that did not happen, and the only related signal — the healthcheck's generic
|
||||
`health_degraded` at 90% — looked at REGISTERED STORAGE PATHS ONLY, so the docker area (`mp0`) and
|
||||
the system-data area (`mp1`, which holds every driveless app's retained recovery unit) were invisible,
|
||||
and it never reported free bytes or named a drive.
|
||||
|
||||
**`disk_warning` / `disk_critical` WERE ALREADY A COMPLETE PIPELINE WITH NO PRODUCER** — allowlisted
|
||||
in the hub, carrying Hungarian copy, sitting in `settings.DefaultEnabledEvents`, with a UI checkbox
|
||||
(`event_disk_alerts`) — and `grep` across all four repos found **zero emitters**. The sixth "built but
|
||||
never wired" instance in this project. This release is their producer; minting a new near-duplicate
|
||||
type would have left the pair inert forever.
|
||||
|
||||
- **Two threshold terms, whichever trips first** — used ≥ **85%** OR free < **5 GiB** (critical: 95% /
|
||||
2 GiB). A percentage alone lies at both ends of this fleet's size range: 85% of a 20 G backup area
|
||||
leaves 3 G, less than one DB-backed app's recovery unit (up to ~2× its data — measured 21.1 GB →
|
||||
40.2 GB, `07-backup-architecture.md` §7.5), while 85% of a 4 TB media drive leaves 600 G.
|
||||
- **Edge-triggered on ESCALATION ONLY**, state persisted across restarts. De-escalation is silent and
|
||||
re-arms. **Hysteresis dead zone** between clear (75% / 7 GiB) and warn holds the previous band, so a
|
||||
filesystem on the line does not flap; the gap is pinned by a test, because a warn and clear
|
||||
threshold that can be edited into equality is a flapping bug waiting to be introduced.
|
||||
- **The hub owns cooldown — no controller-side timer** (the `offboxEnlargeBlockedNotify` precedent).
|
||||
- **A nil usage read is NEVER a warning.** An absent, unmounted or unreadable filesystem is the drive
|
||||
gate's business and already has its own alert; calling it "full" would be a false alarm with a
|
||||
misleading cause. It also does not CLEAR an existing warning — a blipping drive must not silently
|
||||
retract a true alarm.
|
||||
- **Per FILESYSTEM, never per app** — one full disk holding ten apps would fire ten times, nine of them
|
||||
noise. Watched: the app-data volume, the system-data volume, and every registered non-decommissioned
|
||||
drive, de-duplicated by path and resolved at check time (a drive added between checks needs no
|
||||
restart). Daily at **03:30**, deliberately before the nightly app-data legs so a customer about to
|
||||
lose a backup to lack of space hears about it with a night's margin.
|
||||
|
||||
**R-158 — the operator hears about a failed per-app backup.** `captureAllRecoveryUnits` logged
|
||||
`[WARN] Recovery unit capture failed for %s` and stopped there; the manager carried three notify seams
|
||||
and none for the unit capture. `/backups/apps` is the page a person opens to ask whether ONE app is
|
||||
backed up, and it was the one page that never said. New `unitNotify` seam + `SetUnitNotify`, fired
|
||||
**per app with the loop continuing** (one app's failure neither aborts nor silences its siblings), and
|
||||
carrying the target filesystem's used/free bytes at the moment of failure — the overwhelmingly likely
|
||||
cause is a full filesystem, and those numbers answer "why" without an operator logging in. `UnitSpace`
|
||||
is **nil when the filesystem is unreadable** and renders as *"unavailable"*, never as zeros: "0 GB
|
||||
free" and "we could not look" are opposite diagnoses.
|
||||
|
||||
**It is OPERATOR-TIER (`recovery_unit_capture_failed`), deliberately NOT `backup_failed`.** That type
|
||||
carries a `customerMessages` entry AND sits in `DefaultEnabledEvents`, so reusing it — which is what
|
||||
R-158's own proposal said — would email the customer, in Hungarian, that their backup failed, about
|
||||
something they cannot act on. D-c routes it to the operator and overrides the proposal. Operator-only
|
||||
is enforced by the hub's `notify.operatorOnlyEvents` register, **not** by the absence of a
|
||||
`customerMessages` entry; v0.78.0 claimed the latter and was wrong, and there is a red-proof here
|
||||
demonstrating the customer receiving it when the register entry is removed.
|
||||
|
||||
**R-174 — the app-stop guard stopped starting apps onto missing drives. A regression in v0.189.0 code,
|
||||
found by review on 2026-08-02 and closed the same session.** `appStopGuard.SetStarter(stackMgr)`
|
||||
handed `Recover` the RAW stack manager, whose `StartStack` has no drive gate. The guard runs **at
|
||||
startup** — exactly when an external drive may not have come back — so a backup that stopped an app,
|
||||
followed by a power cut and a drive that did not remount, ended with the app started onto a missing
|
||||
drive. **This is R-171 one path over**, and the rule is not new: the API's own
|
||||
`startGatedByMissingDrive` already refused this to the customer.
|
||||
|
||||
- The starter is now wrapped in `gatedAppStopStarter`, using the **same** drive predicate the boot
|
||||
sweep uses. `bootDriveGate` could NOT be reused whole and the reason is recorded in the code: its
|
||||
holder #2 reads `bootAppStopGuard.HeldStacks()`, which during `Recover` is **the guard's own marker**
|
||||
— it would refuse every recovery it was meant to perform — and holders #1/#2 read package-level vars
|
||||
assigned *after* `Recover()` runs, so a whole-gate reuse would be correct only by accident of
|
||||
nil-safety. Holder #3 (the drive) is extracted into `driveStartGate`, which now has two callers and
|
||||
one implementation; a test pins that `bootDriveGate` keeps delegating to it.
|
||||
- **A refusal is not a failure.** New `ErrStartRefused` + an `AppStopRecovery.Refused` bucket. Both
|
||||
keep the marker — the operation is genuinely unfinished — but only `Failed` alarms. Collapsing them
|
||||
would push a deliberately-held app into `NotifyBackupFailed`, a customer-enabled type, producing
|
||||
exactly the R-171 false alarm this fix exists to prevent. `main.go` now guards the notify with
|
||||
`Alarming()` rather than `!= nil`, and the pre-existing seam test was **tightened** to require it.
|
||||
- Fail-safe per R-171's contract: cannot determine ⇒ do not start. The check stays per-caller and was
|
||||
deliberately NOT pushed into `Manager.StartStack` (fourteen callers, most legitimate).
|
||||
|
||||
**Tests:** 1157 → **1184** (+27). Every red-proof demonstrated failing and restored — the gate removed
|
||||
from the guard's starter; the `unitNotify` call removed; the edge trigger removed (fires twice); the
|
||||
clear thresholds edited into equality; and `recovery_unit_capture_failed` removed from
|
||||
`operatorOnlyEvents`, which showed the customer receiving an operator event. Seam wirings are pinned by
|
||||
walking `main.go`'s **AST**, not `strings.Contains` — a red-proof that comments out `SetNotify` fails
|
||||
the test while the string is still in the file.
|
||||
|
||||
### v0.190.0 — the boot-recovery story finished, and a regression v0.189.0 opened (2026-08-02, R-157 A · R-170 · R-171)
|
||||
|
||||
**R-171 — a regression introduced by v0.189.0, found by reading the diff and CONFIRMED on hardware
|
||||
|
||||
Reference in New Issue
Block a user