v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s

R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills.
It emits the PRE-EXISTING disk_warning/disk_critical pair, which was
allowlisted, copy'd, default-enabled and checkbox'd with no producer in any
repo — the sixth "built but never wired" instance here. Two threshold terms
(85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at
both ends of this fleet's size range. Edge-triggered on escalation only,
state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil
usage read is never a warning and never clears one. Per filesystem, never
per app. Daily 03:30, before the nightly app-data legs.

R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture
fails, loop continuing, carrying the target filesystem's used/free bytes.
Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed,
which is customer-enabled and would email the customer about a failure they
cannot act on. D-c overrides R-158's own proposal here.

R-174: the app-stop guard no longer starts apps onto MISSING drives — a
regression in v0.189.0 code, found by review and closed the same session.
SetStarter got the raw stack manager, whose StartStack has no drive gate,
and Recover runs at startup. R-171 one path over. bootDriveGate could not be
reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read
vars assigned after Recover runs), so holder #3 is extracted into a shared
driveStartGate with a test pinning the delegation. ErrStartRefused splits a
refusal from a failure: both keep the marker, only Failed alarms, because
routing a deliberate hold into NotifyBackupFailed is the same false alarm.

Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
This commit is contained in:
2026-08-02 23:18:51 +02:00
parent 95eb5c2c1a
commit cf48214f6c
12 changed files with 1785 additions and 22 deletions
+88
View File
@@ -1,5 +1,93 @@
## Changelog
### v0.191.0 — warn before the wall comes down (2026-08-02, R-167 · R-158 · R-174) — MinAgent: none
**Storage monitoring and backup alerts, decision D-c, landing BEFORE the `mp1``mp0` merge (D-a /
R-165) rather than with it.** D-a's own condition (2) says the monitoring ships in the same step and
never after, because the merge removes a wall that currently fails safely. Landing it first is
strictly better and costs nothing: the warnings go in and get proven on hardware while the wall is
still standing. **No disk layout is touched in this release.**
**R-167 — the customer is warned BEFORE a filesystem fills, and the warning is the pair that already
existed.** New `internal/fillwatch`. Nothing warned before this: the first sign of a full filesystem
was a backup that did not happen, and the only related signal — the healthcheck's generic
`health_degraded` at 90% — looked at REGISTERED STORAGE PATHS ONLY, so the docker area (`mp0`) and
the system-data area (`mp1`, which holds every driveless app's retained recovery unit) were invisible,
and it never reported free bytes or named a drive.
**`disk_warning` / `disk_critical` WERE ALREADY A COMPLETE PIPELINE WITH NO PRODUCER** — allowlisted
in the hub, carrying Hungarian copy, sitting in `settings.DefaultEnabledEvents`, with a UI checkbox
(`event_disk_alerts`) — and `grep` across all four repos found **zero emitters**. The sixth "built but
never wired" instance in this project. This release is their producer; minting a new near-duplicate
type would have left the pair inert forever.
- **Two threshold terms, whichever trips first** — used ≥ **85%** OR free < **5 GiB** (critical: 95% /
2 GiB). A percentage alone lies at both ends of this fleet's size range: 85% of a 20 G backup area
leaves 3 G, less than one DB-backed app's recovery unit (up to ~2× its data — measured 21.1 GB →
40.2 GB, `07-backup-architecture.md` §7.5), while 85% of a 4 TB media drive leaves 600 G.
- **Edge-triggered on ESCALATION ONLY**, state persisted across restarts. De-escalation is silent and
re-arms. **Hysteresis dead zone** between clear (75% / 7 GiB) and warn holds the previous band, so a
filesystem on the line does not flap; the gap is pinned by a test, because a warn and clear
threshold that can be edited into equality is a flapping bug waiting to be introduced.
- **The hub owns cooldown — no controller-side timer** (the `offboxEnlargeBlockedNotify` precedent).
- **A nil usage read is NEVER a warning.** An absent, unmounted or unreadable filesystem is the drive
gate's business and already has its own alert; calling it "full" would be a false alarm with a
misleading cause. It also does not CLEAR an existing warning — a blipping drive must not silently
retract a true alarm.
- **Per FILESYSTEM, never per app** — one full disk holding ten apps would fire ten times, nine of them
noise. Watched: the app-data volume, the system-data volume, and every registered non-decommissioned
drive, de-duplicated by path and resolved at check time (a drive added between checks needs no
restart). Daily at **03:30**, deliberately before the nightly app-data legs so a customer about to
lose a backup to lack of space hears about it with a night's margin.
**R-158 — the operator hears about a failed per-app backup.** `captureAllRecoveryUnits` logged
`[WARN] Recovery unit capture failed for %s` and stopped there; the manager carried three notify seams
and none for the unit capture. `/backups/apps` is the page a person opens to ask whether ONE app is
backed up, and it was the one page that never said. New `unitNotify` seam + `SetUnitNotify`, fired
**per app with the loop continuing** (one app's failure neither aborts nor silences its siblings), and
carrying the target filesystem's used/free bytes at the moment of failure — the overwhelmingly likely
cause is a full filesystem, and those numbers answer "why" without an operator logging in. `UnitSpace`
is **nil when the filesystem is unreadable** and renders as *"unavailable"*, never as zeros: "0 GB
free" and "we could not look" are opposite diagnoses.
**It is OPERATOR-TIER (`recovery_unit_capture_failed`), deliberately NOT `backup_failed`.** That type
carries a `customerMessages` entry AND sits in `DefaultEnabledEvents`, so reusing it — which is what
R-158's own proposal said — would email the customer, in Hungarian, that their backup failed, about
something they cannot act on. D-c routes it to the operator and overrides the proposal. Operator-only
is enforced by the hub's `notify.operatorOnlyEvents` register, **not** by the absence of a
`customerMessages` entry; v0.78.0 claimed the latter and was wrong, and there is a red-proof here
demonstrating the customer receiving it when the register entry is removed.
**R-174 — the app-stop guard stopped starting apps onto missing drives. A regression in v0.189.0 code,
found by review on 2026-08-02 and closed the same session.** `appStopGuard.SetStarter(stackMgr)`
handed `Recover` the RAW stack manager, whose `StartStack` has no drive gate. The guard runs **at
startup** — exactly when an external drive may not have come back — so a backup that stopped an app,
followed by a power cut and a drive that did not remount, ended with the app started onto a missing
drive. **This is R-171 one path over**, and the rule is not new: the API's own
`startGatedByMissingDrive` already refused this to the customer.
- The starter is now wrapped in `gatedAppStopStarter`, using the **same** drive predicate the boot
sweep uses. `bootDriveGate` could NOT be reused whole and the reason is recorded in the code: its
holder #2 reads `bootAppStopGuard.HeldStacks()`, which during `Recover` is **the guard's own marker**
— it would refuse every recovery it was meant to perform — and holders #1/#2 read package-level vars
assigned *after* `Recover()` runs, so a whole-gate reuse would be correct only by accident of
nil-safety. Holder #3 (the drive) is extracted into `driveStartGate`, which now has two callers and
one implementation; a test pins that `bootDriveGate` keeps delegating to it.
- **A refusal is not a failure.** New `ErrStartRefused` + an `AppStopRecovery.Refused` bucket. Both
keep the marker — the operation is genuinely unfinished — but only `Failed` alarms. Collapsing them
would push a deliberately-held app into `NotifyBackupFailed`, a customer-enabled type, producing
exactly the R-171 false alarm this fix exists to prevent. `main.go` now guards the notify with
`Alarming()` rather than `!= nil`, and the pre-existing seam test was **tightened** to require it.
- Fail-safe per R-171's contract: cannot determine ⇒ do not start. The check stays per-caller and was
deliberately NOT pushed into `Manager.StartStack` (fourteen callers, most legitimate).
**Tests:** 1157 → **1184** (+27). Every red-proof demonstrated failing and restored — the gate removed
from the guard's starter; the `unitNotify` call removed; the edge trigger removed (fires twice); the
clear thresholds edited into equality; and `recovery_unit_capture_failed` removed from
`operatorOnlyEvents`, which showed the customer receiving an operator event. Seam wirings are pinned by
walking `main.go`'s **AST**, not `strings.Contains` — a red-proof that comments out `SetNotify` fails
the test while the string is still in the file.
### v0.190.0 — the boot-recovery story finished, and a regression v0.189.0 opened (2026-08-02, R-157 A · R-170 · R-171)
**R-171 — a regression introduced by v0.189.0, found by reading the diff and CONFIRMED on hardware