v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s
gates / gates (push) Successful in 9s
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills. It emits the PRE-EXISTING disk_warning/disk_critical pair, which was allowlisted, copy'd, default-enabled and checkbox'd with no producer in any repo — the sixth "built but never wired" instance here. Two threshold terms (85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at both ends of this fleet's size range. Edge-triggered on escalation only, state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil usage read is never a warning and never clears one. Per filesystem, never per app. Daily 03:30, before the nightly app-data legs. R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture fails, loop continuing, carrying the target filesystem's used/free bytes. Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed, which is customer-enabled and would email the customer about a failure they cannot act on. D-c overrides R-158's own proposal here. R-174: the app-stop guard no longer starts apps onto MISSING drives — a regression in v0.189.0 code, found by review and closed the same session. SetStarter got the raw stack manager, whose StartStack has no drive gate, and Recover runs at startup. R-171 one path over. bootDriveGate could not be reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read vars assigned after Recover runs), so holder #3 is extracted into a shared driveStartGate with a test pinning the delegation. ErrStartRefused splits a refusal from a failure: both keep the marker, only Failed alarms, because routing a deliberate hold into NotifyBackupFailed is the same false alarm. Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
This commit is contained in:
@@ -12,6 +12,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/system"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
@@ -196,8 +197,54 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// UnitSpace is the target filesystem's occupancy at the moment a capture failed — the numbers that
|
||||
// answer "why" without an operator logging in. Nil when the filesystem could not be read at all
|
||||
// (system.GetDiskUsage returns nil on error), which is reported as unknown rather than as full.
|
||||
type UnitSpace struct {
|
||||
Path string
|
||||
UsedGB float64
|
||||
AvailGB float64
|
||||
TotalGB float64
|
||||
UsedPercent float64
|
||||
}
|
||||
|
||||
// String renders the space figures for an operator, or says plainly that they are unknown. An absent
|
||||
// reading must never render as zeros — "0 GB free" and "we could not look" are opposite diagnoses.
|
||||
func (u *UnitSpace) String() string {
|
||||
if u == nil {
|
||||
return "target filesystem usage unavailable"
|
||||
}
|
||||
return fmt.Sprintf("%s: %.1f/%.1f GB used (%.0f%%), %.1f GB free",
|
||||
u.Path, u.UsedGB, u.TotalGB, u.UsedPercent, u.AvailGB)
|
||||
}
|
||||
|
||||
// SetUnitNotify wires the per-app recovery-unit capture failure alert (R-158 / R-167). INIT-ONLY —
|
||||
// call once at startup, in main.go, alongside SetOffboxNotify. Nil-safe: an unwired seam is silently
|
||||
// the pre-v0.191.0 behaviour, which is a `[WARN]` line and nothing else.
|
||||
func (m *Manager) SetUnitNotify(fn func(stackName string, err error, usage *UnitSpace)) {
|
||||
m.unitNotify = fn
|
||||
}
|
||||
|
||||
// unitTargetSpace reads the occupancy of the filesystem a unit for `stackName` would be written to.
|
||||
// Nil on an unreadable path — never a fabricated zero (§8.4: an unreadable filesystem is not a full
|
||||
// one, and the drive gate already owns the absent-drive case).
|
||||
func (m *Manager) unitTargetSpace(stackName string) *UnitSpace {
|
||||
path := m.GetAppDrivePath(stackName)
|
||||
if path == "" {
|
||||
return nil
|
||||
}
|
||||
di := system.GetDiskUsage(path)
|
||||
if di == nil {
|
||||
return nil
|
||||
}
|
||||
return &UnitSpace{
|
||||
Path: path, UsedGB: di.UsedGB, AvailGB: di.AvailGB,
|
||||
TotalGB: di.TotalGB, UsedPercent: di.UsedPercent,
|
||||
}
|
||||
}
|
||||
|
||||
// captureAllRecoveryUnits refreshes the recovery unit for every deployed stack. Best-effort:
|
||||
// a per-app failure is logged and does not abort the others.
|
||||
// a per-app failure is logged, NOTIFIED (R-158), and does not abort the others.
|
||||
func (m *Manager) captureAllRecoveryUnits() {
|
||||
if m.stackProvider == nil {
|
||||
return
|
||||
@@ -209,6 +256,12 @@ func (m *Manager) captureAllRecoveryUnits() {
|
||||
}
|
||||
if err := m.CaptureRecoveryUnit(stack.Name); err != nil {
|
||||
m.logger.Printf("[WARN] [backup] Recovery unit capture failed for %s: %v", stack.Name, err)
|
||||
// R-158: per app, and the loop CONTINUES — one app's failure must not silence the
|
||||
// others, and it must not abort their captures either. The space figures are read at
|
||||
// the moment of failure, because the point is to answer "why" (usually: no room).
|
||||
if m.unitNotify != nil {
|
||||
m.unitNotify(stack.Name, err, m.unitTargetSpace(stack.Name))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user