v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s

R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills.
It emits the PRE-EXISTING disk_warning/disk_critical pair, which was
allowlisted, copy'd, default-enabled and checkbox'd with no producer in any
repo — the sixth "built but never wired" instance here. Two threshold terms
(85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at
both ends of this fleet's size range. Edge-triggered on escalation only,
state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil
usage read is never a warning and never clears one. Per filesystem, never
per app. Daily 03:30, before the nightly app-data legs.

R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture
fails, loop continuing, carrying the target filesystem's used/free bytes.
Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed,
which is customer-enabled and would email the customer about a failure they
cannot act on. D-c overrides R-158's own proposal here.

R-174: the app-stop guard no longer starts apps onto MISSING drives — a
regression in v0.189.0 code, found by review and closed the same session.
SetStarter got the raw stack manager, whose StartStack has no drive gate,
and Recover runs at startup. R-171 one path over. bootDriveGate could not be
reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read
vars assigned after Recover runs), so holder #3 is extracted into a shared
driveStartGate with a test pinning the delegation. ErrStartRefused splits a
refusal from a failure: both keep the marker, only Failed alarms, because
routing a deliberate hold into NotifyBackupFailed is the same false alarm.

Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
This commit is contained in:
2026-08-02 23:18:51 +02:00
parent 95eb5c2c1a
commit cf48214f6c
12 changed files with 1785 additions and 22 deletions
+54 -1
View File
@@ -12,6 +12,7 @@ import (
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/system"
"gopkg.in/yaml.v3"
)
@@ -196,8 +197,54 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
return nil
}
// UnitSpace is the target filesystem's occupancy at the moment a capture failed — the numbers that
// answer "why" without an operator logging in. Nil when the filesystem could not be read at all
// (system.GetDiskUsage returns nil on error), which is reported as unknown rather than as full.
type UnitSpace struct {
Path string
UsedGB float64
AvailGB float64
TotalGB float64
UsedPercent float64
}
// String renders the space figures for an operator, or says plainly that they are unknown. An absent
// reading must never render as zeros — "0 GB free" and "we could not look" are opposite diagnoses.
func (u *UnitSpace) String() string {
if u == nil {
return "target filesystem usage unavailable"
}
return fmt.Sprintf("%s: %.1f/%.1f GB used (%.0f%%), %.1f GB free",
u.Path, u.UsedGB, u.TotalGB, u.UsedPercent, u.AvailGB)
}
// SetUnitNotify wires the per-app recovery-unit capture failure alert (R-158 / R-167). INIT-ONLY —
// call once at startup, in main.go, alongside SetOffboxNotify. Nil-safe: an unwired seam is silently
// the pre-v0.191.0 behaviour, which is a `[WARN]` line and nothing else.
func (m *Manager) SetUnitNotify(fn func(stackName string, err error, usage *UnitSpace)) {
m.unitNotify = fn
}
// unitTargetSpace reads the occupancy of the filesystem a unit for `stackName` would be written to.
// Nil on an unreadable path — never a fabricated zero (§8.4: an unreadable filesystem is not a full
// one, and the drive gate already owns the absent-drive case).
func (m *Manager) unitTargetSpace(stackName string) *UnitSpace {
path := m.GetAppDrivePath(stackName)
if path == "" {
return nil
}
di := system.GetDiskUsage(path)
if di == nil {
return nil
}
return &UnitSpace{
Path: path, UsedGB: di.UsedGB, AvailGB: di.AvailGB,
TotalGB: di.TotalGB, UsedPercent: di.UsedPercent,
}
}
// captureAllRecoveryUnits refreshes the recovery unit for every deployed stack. Best-effort:
// a per-app failure is logged and does not abort the others.
// a per-app failure is logged, NOTIFIED (R-158), and does not abort the others.
func (m *Manager) captureAllRecoveryUnits() {
if m.stackProvider == nil {
return
@@ -209,6 +256,12 @@ func (m *Manager) captureAllRecoveryUnits() {
}
if err := m.CaptureRecoveryUnit(stack.Name); err != nil {
m.logger.Printf("[WARN] [backup] Recovery unit capture failed for %s: %v", stack.Name, err)
// R-158: per app, and the loop CONTINUES — one app's failure must not silence the
// others, and it must not abort their captures either. The space figures are read at
// the moment of failure, because the point is to answer "why" (usually: no room).
if m.unitNotify != nil {
m.unitNotify(stack.Name, err, m.unitTargetSpace(stack.Name))
}
}
}
}