v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s
gates / gates (push) Successful in 9s
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills. It emits the PRE-EXISTING disk_warning/disk_critical pair, which was allowlisted, copy'd, default-enabled and checkbox'd with no producer in any repo — the sixth "built but never wired" instance here. Two threshold terms (85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at both ends of this fleet's size range. Edge-triggered on escalation only, state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil usage read is never a warning and never clears one. Per filesystem, never per app. Daily 03:30, before the nightly app-data legs. R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture fails, loop continuing, carrying the target filesystem's used/free bytes. Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed, which is customer-enabled and would email the customer about a failure they cannot act on. D-c overrides R-158's own proposal here. R-174: the app-stop guard no longer starts apps onto MISSING drives — a regression in v0.189.0 code, found by review and closed the same session. SetStarter got the raw stack manager, whose StartStack has no drive gate, and Recover runs at startup. R-171 one path over. bootDriveGate could not be reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read vars assigned after Recover runs), so holder #3 is extracted into a shared driveStartGate with a test pinning the delegation. ErrStartRefused splits a refusal from a failure: both keep the marker, only Failed alarms, because routing a deliberate hold into NotifyBackupFailed is the same false alarm. Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
This commit is contained in:
@@ -0,0 +1,229 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-174 — the app-stop guard's crash recovery must not start an app onto a MISSING drive.
|
||||
//
|
||||
// The defect these pin, found by review on 2026-08-02 in code shipped 2026-08-01 (v0.189.0):
|
||||
// `appStopGuard.SetStarter(stackMgr)` handed Recover the raw stack manager, whose `StartStack` has
|
||||
// no drive gate. Recover runs at STARTUP — exactly when an external drive may not have come back —
|
||||
// so a backup that stopped an app, followed by a power cut and a drive that did not remount, ended
|
||||
// with the app started onto a missing drive. R-171 one path over.
|
||||
//
|
||||
// THE SEAM UNDER TEST IS THE STARTER, not the gate: `internal/backup` must not import `stacks` or
|
||||
// `settings`, so the production gate lives in `cmd/controller`. What is pinned here is the contract
|
||||
// between them — that a starter returning ErrStartRefused produces a REFUSAL (marker kept, no alarm)
|
||||
// and not a FAILURE. The production wiring itself is pinned by TestMainWiresGatedAppStopStarter.
|
||||
|
||||
// gatingStarter is a starter whose gate refuses a named set of apps, in the shape the production
|
||||
// `gatedAppStopStarter` uses: refuse BEFORE calling through, and wrap ErrStartRefused with a reason.
|
||||
type gatingStarter struct {
|
||||
inner *fakeStarter
|
||||
refuse map[string]string // app → reason
|
||||
refused []string
|
||||
}
|
||||
|
||||
func (s *gatingStarter) StartStack(name string) error {
|
||||
if why, ok := s.refuse[name]; ok {
|
||||
s.refused = append(s.refused, name)
|
||||
return fmt.Errorf("%w: %s", ErrStartRefused, why)
|
||||
}
|
||||
return s.inner.StartStack(name)
|
||||
}
|
||||
|
||||
func newGatedGuard(t *testing.T, dir string, refuse map[string]string) (*AppStopGuard, *gatingStarter) {
|
||||
t.Helper()
|
||||
s := &gatingStarter{inner: &fakeStarter{}, refuse: refuse}
|
||||
g := NewAppStopGuard(filepath.Join(dir, "appstop-state.json"), log.New(io.Discard, "", 0))
|
||||
g.SetStarter(s)
|
||||
return g, s
|
||||
}
|
||||
|
||||
// --- Scenario A — the guard does not start an app onto a missing drive ---------------------------
|
||||
|
||||
func TestRecover_DriveAbsent_RefusesTheStartAndKEEPSTheMarker(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
// process 1: a volume dump stops immich, then the box loses power. No End(), no defer.
|
||||
g1, _ := newGatedGuard(t, dir, nil)
|
||||
if err := g1.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich"}); err != nil {
|
||||
t.Fatalf("Begin: %v", err)
|
||||
}
|
||||
// <power cut> — and immich's drive does NOT come back.
|
||||
|
||||
// process 2: a fresh controller starts. The drive is absent.
|
||||
g2, starter := newGatedGuard(t, dir, map[string]string{
|
||||
"immich": "drive /mnt/felhom-drives/hdd_1 is not a live mountpoint",
|
||||
})
|
||||
res := g2.Recover()
|
||||
|
||||
if len(starter.inner.starts) != 0 {
|
||||
t.Fatalf("started %v — the app was started onto a MISSING drive, which is the whole defect",
|
||||
starter.inner.starts)
|
||||
}
|
||||
if res == nil {
|
||||
t.Fatal("Recover returned nil — the refusal is invisible to the caller, so nothing can report it")
|
||||
}
|
||||
if len(res.Refused) != 1 || res.Refused[0] != "immich" {
|
||||
t.Fatalf("refused=%v, want [immich]", res.Refused)
|
||||
}
|
||||
if len(res.Failed) != 0 {
|
||||
t.Fatalf("failed=%v — a deliberate hold was recorded as a FAILURE. That bucket reaches "+
|
||||
"NotifyBackupFailed, which is customer-enabled by default, so the customer would be "+
|
||||
"emailed \"A biztonsági mentés sikertelen!\" about an app nothing is wrong with (R-171's "+
|
||||
"false-alarm shape one path over)", res.Failed)
|
||||
}
|
||||
if !markerExists(t, dir) {
|
||||
t.Fatal("the marker was CLEARED after a refused start — the operation is genuinely " +
|
||||
"unfinished, and clearing it erases the only durable record that immich is owed a restart")
|
||||
}
|
||||
// The refusal must name the app AND the reason, or an operator cannot act on it.
|
||||
if d := res.Detail(); !strings.Contains(d, "held_by_drive") || !strings.Contains(d, "immich") {
|
||||
t.Fatalf("detail %q does not name the held app", d)
|
||||
}
|
||||
if msg := res.Message(); !strings.Contains(msg, "HELD") || !strings.Contains(msg, "drive") {
|
||||
t.Fatalf("operator message %q does not say the app is held by an absent drive", msg)
|
||||
}
|
||||
}
|
||||
|
||||
// A refusal-only recovery MUST NOT alarm. This is the assertion that keeps the fix from being the
|
||||
// bug it fixes: the drive gate doing its job is not a backup failure.
|
||||
func TestRecover_RefusalOnly_IsNotAlarming(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
g1, _ := newGatedGuard(t, dir, nil)
|
||||
if err := g1.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
g2, _ := newGatedGuard(t, dir, map[string]string{"immich": "drive /mnt/felhom-drives/hdd_1 is not a live mountpoint"})
|
||||
res := g2.Recover()
|
||||
|
||||
if res.Alarming() {
|
||||
t.Fatal("a recovery that only REFUSED starts reports as alarming — main.go would push it " +
|
||||
"through NotifyBackupFailed and email the customer about a working drive gate")
|
||||
}
|
||||
}
|
||||
|
||||
// A genuine failure alongside a refusal still alarms, and the two stay in different buckets.
|
||||
func TestRecover_FailureAlongsideRefusal_StillAlarmsAndKeepsThemApart(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
g1, _ := newGatedGuard(t, dir, nil)
|
||||
if err := g1.Begin("volume-dump:batch", ReasonVolumeDump, []string{"immich", "nextcloud", "homebox"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
g2, starter := newGatedGuard(t, dir, map[string]string{"immich": "drive /mnt/felhom-drives/hdd_1 is not a live mountpoint"})
|
||||
starter.inner.failWith = map[string]error{"nextcloud": errors.New("compose up: no such image")}
|
||||
res := g2.Recover()
|
||||
|
||||
if len(res.Refused) != 1 || res.Refused[0] != "immich" {
|
||||
t.Fatalf("refused=%v, want [immich]", res.Refused)
|
||||
}
|
||||
if len(res.Failed) != 1 || res.Failed[0] != "nextcloud" {
|
||||
t.Fatalf("failed=%v, want [nextcloud]", res.Failed)
|
||||
}
|
||||
if len(res.Restarted) != 1 || res.Restarted[0] != "homebox" {
|
||||
t.Fatalf("restarted=%v, want [homebox] — neither a refusal nor a failure may abort the loop",
|
||||
res.Restarted)
|
||||
}
|
||||
if !res.Alarming() {
|
||||
t.Fatal("a genuine restart FAILURE alongside a refusal no longer alarms — the refusal " +
|
||||
"swallowed a real fault")
|
||||
}
|
||||
if !markerExists(t, dir) {
|
||||
t.Fatal("the marker was cleared with work still owed")
|
||||
}
|
||||
// The message must not let the held app inflate the failure count.
|
||||
msg := res.Message()
|
||||
if !strings.Contains(msg, "1 of 2 app(s) could NOT be restarted") {
|
||||
t.Fatalf("operator message %q miscounts: the held app must not be counted as a failure", msg)
|
||||
}
|
||||
if !strings.Contains(msg, "not counted as failures") {
|
||||
t.Fatalf("operator message %q does not disclose the held app at all", msg)
|
||||
}
|
||||
}
|
||||
|
||||
// --- Scenario B — a live drive still recovers normally, byte-identical to before -----------------
|
||||
|
||||
func TestRecover_DriveLive_RecoversExactlyAsBefore(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
g1, _ := newGatedGuard(t, dir, nil)
|
||||
if err := g1.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich", "nextcloud"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Nothing refused — the gate says yes for both.
|
||||
g2, starter := newGatedGuard(t, dir, nil)
|
||||
res := g2.Recover()
|
||||
|
||||
if len(starter.inner.starts) != 2 {
|
||||
t.Fatalf("started %v, want both apps — the new gate refused a LEGITIMATE recovery",
|
||||
starter.inner.starts)
|
||||
}
|
||||
if len(res.Refused) != 0 || len(res.Failed) != 0 {
|
||||
t.Fatalf("refused=%v failed=%v, want neither on a live drive", res.Refused, res.Failed)
|
||||
}
|
||||
if len(res.Restarted) != 2 {
|
||||
t.Fatalf("restarted=%v, want both", res.Restarted)
|
||||
}
|
||||
if markerExists(t, dir) {
|
||||
t.Fatal("the marker survived a fully successful recovery — the next boot would restart the apps again")
|
||||
}
|
||||
if !res.Alarming() {
|
||||
t.Fatal("a successful recovery no longer reports to the operator — the interrupted operation " +
|
||||
"itself is what §2.4 wants reported, and it went silent")
|
||||
}
|
||||
}
|
||||
|
||||
// The next startup, with the drive back, completes the recovery and clears the marker. This is what
|
||||
// makes "keep the marker" a recovery rather than a leak.
|
||||
func TestRecover_HeldAppIsRestartedOnceTheDriveReturns(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
g1, _ := newGatedGuard(t, dir, nil)
|
||||
if err := g1.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Boot 1 — drive absent: refused, marker kept.
|
||||
g2, _ := newGatedGuard(t, dir, map[string]string{"immich": "drive /mnt/felhom-drives/hdd_1 is not a live mountpoint"})
|
||||
if res := g2.Recover(); len(res.Refused) != 1 {
|
||||
t.Fatalf("boot 1 refused=%v, want [immich]", res.Refused)
|
||||
}
|
||||
if !markerExists(t, dir) {
|
||||
t.Fatal("boot 1 cleared the marker — boot 2 has nothing to act on and immich stays down forever")
|
||||
}
|
||||
|
||||
// Boot 2 — the drive is back.
|
||||
g3, starter := newGatedGuard(t, dir, nil)
|
||||
res := g3.Recover()
|
||||
if len(starter.inner.starts) != 1 || starter.inner.starts[0] != "immich" {
|
||||
t.Fatalf("boot 2 started %v, want [immich] — the held app was never picked up again",
|
||||
starter.inner.starts)
|
||||
}
|
||||
if len(res.Restarted) != 1 {
|
||||
t.Fatalf("boot 2 restarted=%v, want [immich]", res.Restarted)
|
||||
}
|
||||
if markerExists(t, dir) {
|
||||
t.Fatal("boot 2 kept the marker after a fully successful recovery")
|
||||
}
|
||||
}
|
||||
|
||||
// ErrStartRefused must be matched with errors.Is, i.e. it survives wrapping. A starter that returns
|
||||
// a bare string reason would land in Failed and alarm — the exact collapse this type prevents.
|
||||
func TestErrStartRefused_SurvivesWrapping(t *testing.T) {
|
||||
err := fmt.Errorf("%w: drive /mnt/felhom-drives/hdd_1 is not a live mountpoint", ErrStartRefused)
|
||||
if !errors.Is(err, ErrStartRefused) {
|
||||
t.Fatal("a wrapped ErrStartRefused is no longer matched by errors.Is — every refusal would " +
|
||||
"be recorded as a restart failure and alarm the customer")
|
||||
}
|
||||
if errors.Is(errors.New("compose up: no such image"), ErrStartRefused) {
|
||||
t.Fatal("an ordinary restart failure matches ErrStartRefused — real faults would go silent")
|
||||
}
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package backup
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
@@ -72,10 +73,27 @@ type AppStopMarker struct {
|
||||
|
||||
// AppStopStarter is the one thing recovery needs: the ability to start a stack. StartStack must be
|
||||
// idempotent (it is — `compose up -d` on a running stack is a no-op).
|
||||
//
|
||||
// R-174: production MUST pass a GATED starter, never the raw stack manager. Recover runs at STARTUP —
|
||||
// exactly when an external drive may not have come back — and `Manager.StartStack` has no drive gate
|
||||
// of its own. See `gatedAppStopStarter` in cmd/controller/main.go.
|
||||
type AppStopStarter interface {
|
||||
StartStack(name string) error
|
||||
}
|
||||
|
||||
// ErrStartRefused is what a gated starter returns when a DELIBERATE HOLDER — today the drive gate —
|
||||
// says an app must not be started. Wrap it (`fmt.Errorf("%w: …", ErrStartRefused)`) so the reason
|
||||
// survives; Recover matches with errors.Is.
|
||||
//
|
||||
// IT IS NOT A FAILURE, AND THE DISTINCTION IS THE WHOLE POINT OF THE TYPE. A refusal means the
|
||||
// holder is doing its job and owns the restart; a failure means the restart was attempted and broke.
|
||||
// Collapsing the two would put a deliberately-held app into `Failed`, which main.go reports through
|
||||
// `NotifyBackupFailed` — a type that is customer-enabled by default (`settings.DefaultEnabledEvents`)
|
||||
// and carries the Hungarian "A biztonsági mentés sikertelen!". That is R-171's defect one path over:
|
||||
// a false alarm about an app the drive gate is deliberately holding. Both buckets keep the marker;
|
||||
// only `Failed` alarms.
|
||||
var ErrStartRefused = errors.New("start refused by a deliberate holder")
|
||||
|
||||
// AppStopGuard owns one marker file. Construct with NewAppStopGuard; the zero value is inert (every
|
||||
// method is a no-op on a nil guard), so a caller that was never wired degrades to pre-v0.189.0
|
||||
// behaviour instead of panicking.
|
||||
@@ -98,7 +116,21 @@ type AppStopRecovery struct {
|
||||
OpID string
|
||||
StartedAt time.Time
|
||||
Restarted []string // apps started again by this recovery
|
||||
Failed []string // apps that could NOT be restarted (the marker was kept for these)
|
||||
Failed []string // apps whose restart was ATTEMPTED and broke (the marker was kept for these)
|
||||
// Refused are apps a deliberate holder said must not start — today, an absent data drive
|
||||
// (R-174). The marker is kept for these too, but they are NOT a fault and MUST NOT alarm: the
|
||||
// holder owns the restart. Separate from Failed for the reason recorded on ErrStartRefused.
|
||||
Refused []string
|
||||
}
|
||||
|
||||
// Alarming reports whether this recovery is worth paging an operator about. A recovery that only
|
||||
// REFUSED starts is the drive gate working as designed, and reporting it through the customer-enabled
|
||||
// `backup_failed` type would be the R-171 false alarm one path over.
|
||||
func (r *AppStopRecovery) Alarming() bool {
|
||||
if r == nil {
|
||||
return false
|
||||
}
|
||||
return len(r.Failed) > 0 || len(r.Restarted) > 0
|
||||
}
|
||||
|
||||
// Message is the operator-facing headline for an interrupted operation.
|
||||
@@ -107,11 +139,23 @@ func (r *AppStopRecovery) Message() string {
|
||||
return ""
|
||||
}
|
||||
if len(r.Failed) > 0 {
|
||||
return fmt.Sprintf("%s was interrupted by a controller restart and %d of %d app(s) could NOT be restarted",
|
||||
m := fmt.Sprintf("%s was interrupted by a controller restart and %d of %d app(s) could NOT be restarted",
|
||||
r.Reason.humanReason(), len(r.Failed), len(r.Restarted)+len(r.Failed))
|
||||
if len(r.Refused) > 0 {
|
||||
m += fmt.Sprintf(" (a further %d are held by an absent drive and are not counted as failures)", len(r.Refused))
|
||||
}
|
||||
return m
|
||||
}
|
||||
return fmt.Sprintf("%s was interrupted by a controller restart — %d app(s) were left stopped and have been restarted",
|
||||
if len(r.Refused) > 0 && len(r.Restarted) == 0 {
|
||||
return fmt.Sprintf("%s was interrupted by a controller restart — %d app(s) are left stopped and HELD: their data drive is not available, so the drive gate restarts them when it returns",
|
||||
r.Reason.humanReason(), len(r.Refused))
|
||||
}
|
||||
m := fmt.Sprintf("%s was interrupted by a controller restart — %d app(s) were left stopped and have been restarted",
|
||||
r.Reason.humanReason(), len(r.Restarted))
|
||||
if len(r.Refused) > 0 {
|
||||
m += fmt.Sprintf("; %d more are held by an absent drive", len(r.Refused))
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Detail is the machine-readable tail. App/stack NAMES only — never env values (§9.5).
|
||||
@@ -124,6 +168,9 @@ func (r *AppStopRecovery) Detail() string {
|
||||
if len(r.Failed) > 0 {
|
||||
d += fmt.Sprintf(" restart_failed=%v", r.Failed)
|
||||
}
|
||||
if len(r.Refused) > 0 {
|
||||
d += fmt.Sprintf(" held_by_drive=%v", r.Refused)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
@@ -210,6 +257,15 @@ func (g *AppStopGuard) Recover() *AppStopRecovery {
|
||||
res := &AppStopRecovery{Reason: m.Reason, OpID: m.OpID, StartedAt: m.StartedAt}
|
||||
for _, name := range m.Stacks {
|
||||
if err := g.starter.StartStack(name); err != nil {
|
||||
// R-174: a REFUSAL is not a failure. The starter's gate has said this app must not be
|
||||
// started (an absent data drive), so the app is left down deliberately and the holder
|
||||
// owns the restart. Logged at WARN with the reason, and kept out of Failed so it never
|
||||
// reaches the customer-enabled backup_failed alarm — see ErrStartRefused.
|
||||
if errors.Is(err, ErrStartRefused) {
|
||||
g.logger.Printf("[WARN] [appstop] crash recovery: NOT restarting %s — %v; the marker is KEPT and the holder owns the restart", name, err)
|
||||
res.Refused = append(res.Refused, name)
|
||||
continue
|
||||
}
|
||||
g.logger.Printf("[ERROR] [appstop] crash recovery: restart %s failed: %v", name, err)
|
||||
res.Failed = append(res.Failed, name)
|
||||
continue
|
||||
@@ -218,13 +274,23 @@ func (g *AppStopGuard) Recover() *AppStopRecovery {
|
||||
res.Restarted = append(res.Restarted, name)
|
||||
}
|
||||
sort.Strings(res.Failed)
|
||||
sort.Strings(res.Refused)
|
||||
sort.Strings(res.Restarted)
|
||||
|
||||
// The marker is kept for BOTH unfinished outcomes, for the same reason and with different
|
||||
// urgency: a failed restart is retried next startup, and a refused one is genuinely unfinished
|
||||
// until its drive returns. Clearing it in either case would erase the only durable record that
|
||||
// an app is owed a restart.
|
||||
if len(res.Failed) > 0 {
|
||||
g.logger.Printf("[ERROR] [appstop] crash recovery: %d app(s) could not be restarted — KEEPING the marker so the next startup retries; the dead-app alarm owns them meanwhile: %v",
|
||||
len(res.Failed), res.Failed)
|
||||
return res
|
||||
}
|
||||
if len(res.Refused) > 0 {
|
||||
g.logger.Printf("[WARN] [appstop] crash recovery: %d app(s) were deliberately NOT restarted (drive absent) — KEEPING the marker; this is the gate working, not a fault: %v",
|
||||
len(res.Refused), res.Refused)
|
||||
return res
|
||||
}
|
||||
g.End()
|
||||
return res
|
||||
}
|
||||
|
||||
@@ -32,6 +32,27 @@ type Manager struct {
|
||||
// tier2Notify, if set, is called after each Tier 2 copy (success: err==nil) for notifications.
|
||||
tier2Notify func(stackName, destLabel string, dur time.Duration, err error)
|
||||
|
||||
// unitNotify (R-158 / R-167), if set, is called ONCE PER APP whose Tier-1 recovery-unit capture
|
||||
// FAILED, and the capture loop continues to the next app. Wired in cmd/controller/main.go.
|
||||
//
|
||||
// WHY IT EXISTS. `/backups/apps` is the page a person opens to ask whether ONE app is backed up,
|
||||
// and until now it was the one page that never said: a per-app capture failure was a `[WARN]`
|
||||
// line and went no further. The manager had three notify seams and none for the unit capture —
|
||||
// the FIFTH instance in this project of a mechanism built and left disconnected.
|
||||
//
|
||||
// IT CARRIES THE SPACE FIGURES DELIBERATELY. The overwhelmingly likely cause is a full
|
||||
// filesystem, and an operator who has the used/free bytes at the moment of failure can act
|
||||
// without logging in. It is the same pair of numbers the customer-facing fill warning reports,
|
||||
// which is why the two ship together.
|
||||
//
|
||||
// OPERATOR-TIER. Routed to a hub event type that is in `notify.operatorOnlyEvents` — a customer
|
||||
// can take no action on a capture failure. Deliberately NOT `backup_failed`, which is
|
||||
// customer-enabled by default and would email them in Hungarian about it (D-c).
|
||||
//
|
||||
// NO CONTROLLER-SIDE COOLDOWN — the hub owns cooldown, per the offboxEnlargeBlockedNotify
|
||||
// precedent.
|
||||
unitNotify func(stackName string, err error, usage *UnitSpace)
|
||||
|
||||
// appStop (R-166) is the crash marker for operations that stop an app, work on its data, and
|
||||
// start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power
|
||||
// cut in that window leaves a durable record that Recover honours at the next startup. Built in
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/system"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
@@ -196,8 +197,54 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// UnitSpace is the target filesystem's occupancy at the moment a capture failed — the numbers that
|
||||
// answer "why" without an operator logging in. Nil when the filesystem could not be read at all
|
||||
// (system.GetDiskUsage returns nil on error), which is reported as unknown rather than as full.
|
||||
type UnitSpace struct {
|
||||
Path string
|
||||
UsedGB float64
|
||||
AvailGB float64
|
||||
TotalGB float64
|
||||
UsedPercent float64
|
||||
}
|
||||
|
||||
// String renders the space figures for an operator, or says plainly that they are unknown. An absent
|
||||
// reading must never render as zeros — "0 GB free" and "we could not look" are opposite diagnoses.
|
||||
func (u *UnitSpace) String() string {
|
||||
if u == nil {
|
||||
return "target filesystem usage unavailable"
|
||||
}
|
||||
return fmt.Sprintf("%s: %.1f/%.1f GB used (%.0f%%), %.1f GB free",
|
||||
u.Path, u.UsedGB, u.TotalGB, u.UsedPercent, u.AvailGB)
|
||||
}
|
||||
|
||||
// SetUnitNotify wires the per-app recovery-unit capture failure alert (R-158 / R-167). INIT-ONLY —
|
||||
// call once at startup, in main.go, alongside SetOffboxNotify. Nil-safe: an unwired seam is silently
|
||||
// the pre-v0.191.0 behaviour, which is a `[WARN]` line and nothing else.
|
||||
func (m *Manager) SetUnitNotify(fn func(stackName string, err error, usage *UnitSpace)) {
|
||||
m.unitNotify = fn
|
||||
}
|
||||
|
||||
// unitTargetSpace reads the occupancy of the filesystem a unit for `stackName` would be written to.
|
||||
// Nil on an unreadable path — never a fabricated zero (§8.4: an unreadable filesystem is not a full
|
||||
// one, and the drive gate already owns the absent-drive case).
|
||||
func (m *Manager) unitTargetSpace(stackName string) *UnitSpace {
|
||||
path := m.GetAppDrivePath(stackName)
|
||||
if path == "" {
|
||||
return nil
|
||||
}
|
||||
di := system.GetDiskUsage(path)
|
||||
if di == nil {
|
||||
return nil
|
||||
}
|
||||
return &UnitSpace{
|
||||
Path: path, UsedGB: di.UsedGB, AvailGB: di.AvailGB,
|
||||
TotalGB: di.TotalGB, UsedPercent: di.UsedPercent,
|
||||
}
|
||||
}
|
||||
|
||||
// captureAllRecoveryUnits refreshes the recovery unit for every deployed stack. Best-effort:
|
||||
// a per-app failure is logged and does not abort the others.
|
||||
// a per-app failure is logged, NOTIFIED (R-158), and does not abort the others.
|
||||
func (m *Manager) captureAllRecoveryUnits() {
|
||||
if m.stackProvider == nil {
|
||||
return
|
||||
@@ -209,6 +256,12 @@ func (m *Manager) captureAllRecoveryUnits() {
|
||||
}
|
||||
if err := m.CaptureRecoveryUnit(stack.Name); err != nil {
|
||||
m.logger.Printf("[WARN] [backup] Recovery unit capture failed for %s: %v", stack.Name, err)
|
||||
// R-158: per app, and the loop CONTINUES — one app's failure must not silence the
|
||||
// others, and it must not abort their captures either. The space figures are read at
|
||||
// the moment of failure, because the point is to answer "why" (usually: no room).
|
||||
if m.unitNotify != nil {
|
||||
m.unitNotify(stack.Name, err, m.unitTargetSpace(stack.Name))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/appbackup"
|
||||
)
|
||||
|
||||
// R-158 / R-167 (D-c, operator half) — a per-app Tier-1 recovery-unit capture failure must reach a
|
||||
// hub channel.
|
||||
//
|
||||
// THE GAP THESE CLOSE. `captureAllRecoveryUnits` logged `[WARN] Recovery unit capture failed for %s`
|
||||
// and stopped there. The manager carried three notify seams — tier2Notify, offboxNotify,
|
||||
// offboxEnlargeBlockedNotify — and none for the unit capture, so the one page a person opens to ask
|
||||
// whether ONE app is backed up (`/backups/apps`) was the one page that never said. Fifth instance in
|
||||
// this project of a mechanism built and left disconnected.
|
||||
|
||||
// unitFailProvider lists a fixed set of stacks and refuses GetStackRecoveryInfo for the named ones,
|
||||
// which is the earliest real failure inside CaptureRecoveryUnit ("stack %q not found").
|
||||
type unitFailProvider struct {
|
||||
stacks []string
|
||||
fail map[string]bool
|
||||
dir string
|
||||
}
|
||||
|
||||
func (p *unitFailProvider) GetStackComposePath(string) (string, bool) { return "", false }
|
||||
func (p *unitFailProvider) ListDeployedStacks() []StackSummary {
|
||||
out := make([]StackSummary, 0, len(p.stacks))
|
||||
for _, s := range p.stacks {
|
||||
out = append(out, StackSummary{Name: s})
|
||||
}
|
||||
return out
|
||||
}
|
||||
func (p *unitFailProvider) GetStackHDDMounts(string) []string { return nil }
|
||||
func (p *unitFailProvider) GetStackHDDPath(string) string { return "" }
|
||||
func (p *unitFailProvider) GetImportRoot() string { return "" }
|
||||
func (p *unitFailProvider) GetDockerVolumes(string) []string { return nil }
|
||||
func (p *unitFailProvider) StopStack(string) error { return nil }
|
||||
func (p *unitFailProvider) StartStack(string) error { return nil }
|
||||
func (p *unitFailProvider) RefreshAndIsRunning(string) bool { return true }
|
||||
func (p *unitFailProvider) GetStackRecoveryInfo(name string) (RecoveryInfo, bool) {
|
||||
if p.fail[name] {
|
||||
return RecoveryInfo{}, false
|
||||
}
|
||||
return RecoveryInfo{StackDir: filepath.Join(p.dir, "stacks", name)}, true
|
||||
}
|
||||
func (p *unitFailProvider) RecoverStackSecrets(string, []string) map[string]string { return nil }
|
||||
func (p *unitFailProvider) RecreateStackDefinitionFromUnit(string, string, map[string]string) error {
|
||||
return nil
|
||||
}
|
||||
func (p *unitFailProvider) StartStackServices(string, []string) error { return nil }
|
||||
func (p *unitFailProvider) GetStackClassifiedBinds(string) ([]appbackup.ClassifiedBind, bool) {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
var _ appbackup.StackDataProvider = (*unitFailProvider)(nil)
|
||||
|
||||
type unitEvent struct {
|
||||
app string
|
||||
err string
|
||||
usage *UnitSpace
|
||||
}
|
||||
|
||||
func newUnitNotifyManager(t *testing.T, stacks []string, fail map[string]bool) (*Manager, *[]unitEvent) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
m := &Manager{
|
||||
logger: log.New(io.Discard, "", 0),
|
||||
systemDataPath: dir,
|
||||
stackProvider: &unitFailProvider{stacks: stacks, fail: fail, dir: dir},
|
||||
}
|
||||
var got []unitEvent
|
||||
m.SetUnitNotify(func(name string, err error, usage *UnitSpace) {
|
||||
got = append(got, unitEvent{app: name, err: err.Error(), usage: usage})
|
||||
})
|
||||
return m, &got
|
||||
}
|
||||
|
||||
// --- Scenario C — a local unit capture failure reaches the operator ------------------------------
|
||||
|
||||
func TestCaptureAll_FailureNotifiesOnceWithTheSpaceFigures(t *testing.T) {
|
||||
m, got := newUnitNotifyManager(t, []string{"immich"}, map[string]bool{"immich": true})
|
||||
|
||||
m.captureAllRecoveryUnits()
|
||||
|
||||
if len(*got) != 1 {
|
||||
t.Fatalf("got %d unit-failure events, want exactly 1 — a per-app Tier-1 capture failure "+
|
||||
"reached no hub channel, which is the R-158 gap un-fixed", len(*got))
|
||||
}
|
||||
e := (*got)[0]
|
||||
if e.app != "immich" {
|
||||
t.Fatalf("event names app %q, want immich — an operator cannot act on an unnamed app", e.app)
|
||||
}
|
||||
if e.err == "" {
|
||||
t.Fatal("the event carries no error — the operator is told a capture failed but not why")
|
||||
}
|
||||
// The space figures are the point: the overwhelmingly likely cause is a full filesystem, and
|
||||
// these answer "why" without an operator logging in.
|
||||
if e.usage == nil {
|
||||
t.Fatal("the event carries no space figures for a readable target filesystem — this is the " +
|
||||
"pair of numbers that makes the alert actionable, and the same pair the customer fill " +
|
||||
"warning reports (which is why the two ship together)")
|
||||
}
|
||||
if e.usage.Path == "" || e.usage.TotalGB <= 0 {
|
||||
t.Fatalf("space figures are not populated: %+v", e.usage)
|
||||
}
|
||||
}
|
||||
|
||||
// --- Scenario D — one failing app does not silence the others ------------------------------------
|
||||
|
||||
func TestCaptureAll_OneFailureDoesNotAbortOrDuplicate(t *testing.T) {
|
||||
m, got := newUnitNotifyManager(t,
|
||||
[]string{"homebox", "immich", "nextcloud"},
|
||||
map[string]bool{"immich": true})
|
||||
|
||||
m.captureAllRecoveryUnits()
|
||||
|
||||
if len(*got) != 1 {
|
||||
t.Fatalf("got %d events, want exactly 1 — either the loop ABORTED on the middle app "+
|
||||
"(and its siblings were never captured), or one failure produced several events: %+v",
|
||||
len(*got), *got)
|
||||
}
|
||||
if (*got)[0].app != "immich" {
|
||||
t.Fatalf("event names %q, want immich", (*got)[0].app)
|
||||
}
|
||||
|
||||
// The siblings must have been ATTEMPTED after the failure — a positive observable, not the
|
||||
// absence of an event. The provider records nothing, so assert via the failure set instead:
|
||||
// flip the LAST app to failing and require both events.
|
||||
m2, got2 := newUnitNotifyManager(t,
|
||||
[]string{"homebox", "immich", "nextcloud"},
|
||||
map[string]bool{"immich": true, "nextcloud": true})
|
||||
m2.captureAllRecoveryUnits()
|
||||
if len(*got2) != 2 {
|
||||
t.Fatalf("got %d events, want 2 — the app AFTER the first failure was never reached, so the "+
|
||||
"loop is aborting rather than continuing: %+v", len(*got2), *got2)
|
||||
}
|
||||
if (*got2)[0].app != "immich" || (*got2)[1].app != "nextcloud" {
|
||||
t.Fatalf("events %+v, want immich then nextcloud in loop order", *got2)
|
||||
}
|
||||
}
|
||||
|
||||
// A successful capture must be SILENT. An alert that fires on success is an alert an operator learns
|
||||
// to ignore.
|
||||
func TestCaptureAll_SuccessIsSilent(t *testing.T) {
|
||||
m, got := newUnitNotifyManager(t, []string{"homebox"}, nil)
|
||||
m.captureAllRecoveryUnits()
|
||||
if len(*got) != 0 {
|
||||
t.Fatalf("a successful capture fired %d event(s): %+v", len(*got), *got)
|
||||
}
|
||||
}
|
||||
|
||||
// The seam must be nil-safe: an unwired notify is the pre-v0.191.0 behaviour (a WARN line), never a
|
||||
// panic that takes the whole nightly backup down with it.
|
||||
func TestCaptureAll_UnwiredNotifyDoesNotPanic(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
m := &Manager{
|
||||
logger: log.New(io.Discard, "", 0),
|
||||
systemDataPath: dir,
|
||||
stackProvider: &unitFailProvider{stacks: []string{"immich"}, fail: map[string]bool{"immich": true}, dir: dir},
|
||||
}
|
||||
m.captureAllRecoveryUnits() // no SetUnitNotify — must not panic
|
||||
}
|
||||
|
||||
// §8.4 in the failure direction: an unreadable target filesystem is reported as UNKNOWN, never as
|
||||
// zeros. "0 GB free" and "we could not look" are opposite diagnoses, and rendering the second as the
|
||||
// first is the presence-is-not-success trap pointing the other way.
|
||||
func TestUnitSpace_NilRendersAsUnavailableNotZero(t *testing.T) {
|
||||
var u *UnitSpace
|
||||
s := u.String()
|
||||
if !strings.Contains(s, "unavailable") {
|
||||
t.Fatalf("nil UnitSpace renders as %q — it must say the reading is unavailable", s)
|
||||
}
|
||||
if strings.Contains(s, "0.0") {
|
||||
t.Fatalf("nil UnitSpace renders zeros (%q) — an operator would read \"the disk is full\" "+
|
||||
"from a filesystem nobody could read", s)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,373 @@
|
||||
// Package fillwatch warns the CUSTOMER that a filesystem is filling, BEFORE anything fails.
|
||||
//
|
||||
// R-167, operator decision D-c (2026-08-02, felhom.eu CONTEXT.md S-5). Nothing warned before this:
|
||||
// the first sign of a full filesystem was a backup that did not happen. The healthcheck folded disk
|
||||
// pressure into a generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never
|
||||
// looked at the docker area or the system-data area, never reported free bytes, and never named the
|
||||
// drive.
|
||||
//
|
||||
// WHY IT SHIPS BEFORE THE mp1→mp0 MERGE (D-a / R-165), not with it. Today an app that outgrows the
|
||||
// 20 G backup area is refused per app with its last good recovery unit preserved byte-identical —
|
||||
// a wall, but one that fails safely, one app at a time. The merge removes that wall so backups share
|
||||
// the large area. D-a's own condition (2) says the monitoring lands in the same step and never after;
|
||||
// landing it FIRST is strictly better and costs nothing, so the warnings go in and get proven on real
|
||||
// hardware while the wall is still standing.
|
||||
//
|
||||
// WHY THE FILESYSTEM IS THE UNIT AND NOT THE APP. One full disk holding ten apps would produce ten
|
||||
// identical warnings, nine of them noise. The customer's action — free space, delete files, add a
|
||||
// drive — is per filesystem too.
|
||||
package fillwatch
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// ── Thresholds ───────────────────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// TWO TERMS, AND WHICHEVER TRIPS FIRST WINS, because a percentage alone lies at both ends of the
|
||||
// size range this fleet actually has: 85% of a 20 G backup area leaves 3 G — not enough for one
|
||||
// DB-backed app's recovery unit, which is up to ~2× its data (measured: 21.1 GB → 40.2 GB,
|
||||
// architecture/07-backup-architecture.md §7.5) — while 85% of a 4 TB media drive leaves 600 G and is
|
||||
// nothing to write home about. A free-bytes floor catches the first; a percentage catches the second.
|
||||
const (
|
||||
// WarnUsedPercent / WarnFreeGiB — enter the warning band.
|
||||
WarnUsedPercent = 85.0
|
||||
WarnFreeGiB = 5.0
|
||||
|
||||
// CritUsedPercent / CritFreeGiB — enter the critical band. Below 2 GiB a volume tar of almost any
|
||||
// real app fails, so this is "the next backup will not complete", not "it is getting tight".
|
||||
CritUsedPercent = 95.0
|
||||
CritFreeGiB = 2.0
|
||||
|
||||
// ClearUsedPercent / ClearFreeGiB — the RETURN threshold, deliberately well below the warn pair.
|
||||
// Both must hold. Between clear and warn is a dead zone in which the previous band is HELD, so a
|
||||
// filesystem hovering on the line does not flap between warned and cleared. The gap is pinned by
|
||||
// TestThresholdsKeepTheirHysteresisGap — a warn and clear threshold that can be edited into
|
||||
// equality is a flapping bug waiting to be introduced.
|
||||
ClearUsedPercent = 75.0
|
||||
ClearFreeGiB = 7.0
|
||||
)
|
||||
|
||||
// Band is a filesystem's fill state. Ordered: escalation is an increase.
|
||||
type Band int
|
||||
|
||||
const (
|
||||
BandOK Band = iota
|
||||
BandWarning
|
||||
BandCritical
|
||||
)
|
||||
|
||||
func (b Band) String() string {
|
||||
switch b {
|
||||
case BandWarning:
|
||||
return "warning"
|
||||
case BandCritical:
|
||||
return "critical"
|
||||
default:
|
||||
return "ok"
|
||||
}
|
||||
}
|
||||
|
||||
// EventType maps a band to the hub event type it emits.
|
||||
//
|
||||
// These two types already existed in the hub's allowedEventTypes, already carried Hungarian copy,
|
||||
// already sat in the controller's DefaultEnabledEvents and already had a UI checkbox — and NOTHING
|
||||
// IN ANY REPO EMITTED THEM. A complete customer pipeline with no producer, the sixth "built but
|
||||
// never wired" instance in this project. This package is that producer; a new near-duplicate event
|
||||
// type would have left the pair inert forever.
|
||||
func (b Band) EventType() string {
|
||||
switch b {
|
||||
case BandCritical:
|
||||
return "disk_critical"
|
||||
case BandWarning:
|
||||
return "disk_warning"
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// Severity is the hub severity for a band. `disk_warning` must be "warning" and `disk_critical`
|
||||
// "critical" — the dispatcher drops "info" entirely (severityNotifies), so getting this wrong stores
|
||||
// the event and mails nobody.
|
||||
func (b Band) Severity() string {
|
||||
switch b {
|
||||
case BandCritical:
|
||||
return "critical"
|
||||
case BandWarning:
|
||||
return "warning"
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// Usage is one filesystem's occupancy. Deliberately not `system.DiskUsageInfo` — this package must be
|
||||
// testable without a real filesystem, and the seam is the reason it is.
|
||||
type Usage struct {
|
||||
UsedPercent float64
|
||||
AvailGB float64
|
||||
UsedGB float64
|
||||
TotalGB float64
|
||||
}
|
||||
|
||||
// Target is a watched filesystem: the path to stat, and the name a CUSTOMER will recognise.
|
||||
type Target struct {
|
||||
Path string
|
||||
Label string
|
||||
}
|
||||
|
||||
// Event is one crossing, handed to the notify seam.
|
||||
type Event struct {
|
||||
Target Target
|
||||
Band Band
|
||||
Usage Usage
|
||||
Message string // the Hungarian customer text, already rendered
|
||||
}
|
||||
|
||||
// Watcher holds the persisted per-filesystem band and emits on escalation only.
|
||||
type Watcher struct {
|
||||
statePath string
|
||||
logger *log.Logger
|
||||
|
||||
// targets returns the filesystems to watch, resolved at CHECK time — a drive added or removed
|
||||
// between checks must be picked up without a restart.
|
||||
targets func() []Target
|
||||
// usage reads one filesystem. NIL RESULT MEANS UNREADABLE, NEVER FULL (§8.4).
|
||||
usage func(path string) *Usage
|
||||
// notify pushes the customer event. Nil-safe.
|
||||
notify func(Event)
|
||||
|
||||
mu sync.Mutex
|
||||
bands map[string]Band
|
||||
}
|
||||
|
||||
// New builds a Watcher over a state file. Nothing is read until Check runs.
|
||||
func New(statePath string, logger *log.Logger, targets func() []Target, usage func(string) *Usage) *Watcher {
|
||||
if logger == nil {
|
||||
logger = log.Default()
|
||||
}
|
||||
return &Watcher{
|
||||
statePath: statePath,
|
||||
logger: logger,
|
||||
targets: targets,
|
||||
usage: usage,
|
||||
bands: make(map[string]Band),
|
||||
}
|
||||
}
|
||||
|
||||
// SetNotify wires the customer event push. INIT-ONLY — call once at startup.
|
||||
func (w *Watcher) SetNotify(fn func(Event)) { w.notify = fn }
|
||||
|
||||
// classify decides the band from a reading AND the previous band, which is what makes the hysteresis
|
||||
// work: between the clear and warn thresholds the previous band is HELD rather than recomputed.
|
||||
//
|
||||
// Pure, so it is unit-testable without a filesystem, a clock or a notifier.
|
||||
func classify(u Usage, prev Band) Band {
|
||||
switch {
|
||||
case u.UsedPercent >= CritUsedPercent || u.AvailGB < CritFreeGiB:
|
||||
return BandCritical
|
||||
case u.UsedPercent >= WarnUsedPercent || u.AvailGB < WarnFreeGiB:
|
||||
return BandWarning
|
||||
case u.UsedPercent <= ClearUsedPercent && u.AvailGB >= ClearFreeGiB:
|
||||
return BandOK
|
||||
default:
|
||||
// The dead zone. Holding `prev` is the whole hysteresis: a filesystem sitting at 80% neither
|
||||
// warns (it is below the warn line) nor clears (it is above the clear line).
|
||||
return prev
|
||||
}
|
||||
}
|
||||
|
||||
// Check reads every target once and emits for each ESCALATION. Safe to call from the scheduler.
|
||||
func (w *Watcher) Check() error {
|
||||
if w == nil {
|
||||
return nil
|
||||
}
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
w.loadLocked()
|
||||
|
||||
targets := w.targets()
|
||||
seen := make(map[string]bool, len(targets))
|
||||
changed := false
|
||||
|
||||
for _, t := range targets {
|
||||
if t.Path == "" {
|
||||
continue
|
||||
}
|
||||
seen[t.Path] = true
|
||||
u := w.usage(t.Path)
|
||||
if u == nil {
|
||||
// §8.4 — NOT-YET-MOUNTED IS NOT FULL. An absent, unmounted or unreadable filesystem is
|
||||
// the drive gate's business and already has its own alert (storage_disconnected /
|
||||
// backup_target_absent). Reporting it as "full" would be a false alarm with a misleading
|
||||
// cause, and would tell the customer to delete files that are not the problem.
|
||||
w.logger.Printf("[DEBUG] [fillwatch] %s: usage unreadable — skipped (an unreadable filesystem is not a full one)", t.Path)
|
||||
continue
|
||||
}
|
||||
prev := w.bands[t.Path]
|
||||
next := classify(*u, prev)
|
||||
if next == prev {
|
||||
continue
|
||||
}
|
||||
w.bands[t.Path] = next
|
||||
changed = true
|
||||
|
||||
if next <= prev {
|
||||
// De-escalation, including the return to OK, is SILENT. The customer already acted, or
|
||||
// the app deleted its own temp files; telling them a resolved problem is resolved is
|
||||
// noise. The state change re-arms the warning, which is what Scenario F asks for.
|
||||
w.logger.Printf("[INFO] [fillwatch] %s: %s → %s (%.0f%% used, %.1f GB free) — cleared silently, re-armed",
|
||||
t.Path, prev, next, u.UsedPercent, u.AvailGB)
|
||||
continue
|
||||
}
|
||||
|
||||
msg := Message(t, *u, next)
|
||||
w.logger.Printf("[WARN] [fillwatch] %s (%q): %s → %s — %.0f%% used, %.1f GB free of %.1f GB; notifying the customer",
|
||||
t.Path, t.Label, prev, next, u.UsedPercent, u.AvailGB, u.TotalGB)
|
||||
if w.notify != nil {
|
||||
w.notify(Event{Target: t, Band: next, Usage: *u, Message: msg})
|
||||
}
|
||||
}
|
||||
|
||||
// Forget filesystems that no longer exist (a decommissioned drive), so the state file cannot grow
|
||||
// without bound and a re-added drive starts from OK rather than from a stale band.
|
||||
for path := range w.bands {
|
||||
if !seen[path] {
|
||||
delete(w.bands, path)
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
|
||||
if !changed {
|
||||
return nil
|
||||
}
|
||||
return w.saveLocked()
|
||||
}
|
||||
|
||||
// Bands returns a copy of the current per-path state (diagnostics + tests).
|
||||
func (w *Watcher) Bands() map[string]Band {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
out := make(map[string]Band, len(w.bands))
|
||||
for k, v := range w.bands {
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ── State persistence ────────────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Persisted so a controller restart does not re-warn about a filesystem the customer has already
|
||||
// been told about — the offboxEnlargeBlockedNotify discipline. LOSING IT IS THE ACCEPTABLE
|
||||
// DIRECTION: at most one extra warning, never a missed one. THE HUB OWNS COOLDOWN; this package must
|
||||
// never add a timer of its own.
|
||||
|
||||
type persisted struct {
|
||||
Bands map[string]string `json:"bands"`
|
||||
}
|
||||
|
||||
func (w *Watcher) loadLocked() {
|
||||
if w.statePath == "" {
|
||||
return
|
||||
}
|
||||
data, err := os.ReadFile(w.statePath)
|
||||
if err != nil {
|
||||
if !os.IsNotExist(err) {
|
||||
w.logger.Printf("[WARN] [fillwatch] could not read %s: %v — starting from OK (at most one extra warning)", w.statePath, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
var p persisted
|
||||
if err := json.Unmarshal(data, &p); err != nil {
|
||||
w.logger.Printf("[WARN] [fillwatch] %s is corrupt: %v — starting from OK (at most one extra warning)", w.statePath, err)
|
||||
return
|
||||
}
|
||||
for path, band := range p.Bands {
|
||||
switch band {
|
||||
case "critical":
|
||||
w.bands[path] = BandCritical
|
||||
case "warning":
|
||||
w.bands[path] = BandWarning
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Watcher) saveLocked() error {
|
||||
if w.statePath == "" {
|
||||
return nil
|
||||
}
|
||||
p := persisted{Bands: make(map[string]string, len(w.bands))}
|
||||
for path, band := range w.bands {
|
||||
if band == BandOK {
|
||||
continue // OK is the default — do not persist it
|
||||
}
|
||||
p.Bands[path] = band.String()
|
||||
}
|
||||
data, err := json.MarshalIndent(p, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(w.statePath), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
tmp := w.statePath + ".tmp"
|
||||
if err := os.WriteFile(tmp, data, 0o600); err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Rename(tmp, w.statePath)
|
||||
}
|
||||
|
||||
// ── Customer copy (Hungarian) ────────────────────────────────────────────────────────────────────
|
||||
|
||||
// Message renders the customer-facing Hungarian warning.
|
||||
//
|
||||
// IT MUST SAY WHAT TO DO, not merely that something is happening — a warning a customer cannot act
|
||||
// on is an operator alert wearing the wrong clothes. It names the storage by its LABEL (the customer
|
||||
// named it; a path means nothing to them) and gives the free space in GB.
|
||||
//
|
||||
// It is sent as the event's MESSAGE, and the hub deliberately has no `customerMessages` entry for
|
||||
// these two types: `FormatCustomerEmail` PREFERS the entry over the message, so a static template
|
||||
// would discard the label and the figures — exactly why `offbox_enlarge_blocked` and
|
||||
// `disk_health_degraded` have none either.
|
||||
func Message(t Target, u Usage, band Band) string {
|
||||
name := t.Label
|
||||
if name == "" {
|
||||
name = t.Path
|
||||
}
|
||||
if band == BandCritical {
|
||||
return fmt.Sprintf(
|
||||
"A(z) „%s” tároló kritikusan megtelt: %s szabad hely maradt (%s foglalt). "+
|
||||
"A biztonsági mentések és az alkalmazások írásai bármikor meghiúsulhatnak. "+
|
||||
"Kérjük, mielőbb szabadíts fel helyet: törölj felesleges fájlokat, vagy csatlakoztass új meghajtót.",
|
||||
name, hunGB(u.AvailGB), hunPercent(u.UsedPercent))
|
||||
}
|
||||
return fmt.Sprintf(
|
||||
"A(z) „%s” tároló %s foglalt — %s szabad hely maradt. "+
|
||||
"Kérjük, szabadíts fel helyet, mielőtt megtelik: törölj felesleges fájlokat, vagy csatlakoztass új meghajtót. "+
|
||||
"Ha megtelik, a biztonsági mentések meghiúsulnak.",
|
||||
name, hunPercent(u.UsedPercent), hunGB(u.AvailGB))
|
||||
}
|
||||
|
||||
// hunGB formats a GB figure the Hungarian way — decimal COMMA, one decimal place. A "4.2 GB" in a
|
||||
// Hungarian sentence reads as a typo to the customer.
|
||||
func hunGB(gb float64) string {
|
||||
if gb >= 1024 {
|
||||
return strings.Replace(fmt.Sprintf("%.1f TB", gb/1024), ".", ",", 1)
|
||||
}
|
||||
return strings.Replace(fmt.Sprintf("%.1f GB", gb), ".", ",", 1)
|
||||
}
|
||||
|
||||
func hunPercent(p float64) string { return fmt.Sprintf("%.0f%%", p) }
|
||||
|
||||
// SortTargets gives Check a deterministic order, so a multi-filesystem crossing produces its events
|
||||
// in a stable sequence (tests, and a readable log).
|
||||
func SortTargets(ts []Target) []Target {
|
||||
sort.Slice(ts, func(i, j int) bool { return ts[i].Path < ts[j].Path })
|
||||
return ts
|
||||
}
|
||||
@@ -0,0 +1,320 @@
|
||||
package fillwatch
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// R-167 / decision D-c, customer half — the customer is warned BEFORE a fill, once.
|
||||
|
||||
type harness struct {
|
||||
w *Watcher
|
||||
events []Event
|
||||
usage map[string]*Usage
|
||||
target []Target
|
||||
}
|
||||
|
||||
func newHarness(t *testing.T, targets ...Target) *harness {
|
||||
t.Helper()
|
||||
h := &harness{usage: map[string]*Usage{}, target: targets}
|
||||
h.w = New(filepath.Join(t.TempDir(), "fillwatch.json"), log.New(io.Discard, "", 0),
|
||||
func() []Target { return h.target },
|
||||
func(p string) *Usage { return h.usage[p] })
|
||||
h.w.SetNotify(func(e Event) { h.events = append(h.events, e) })
|
||||
return h
|
||||
}
|
||||
|
||||
func (h *harness) set(path string, usedPct, availGB float64) {
|
||||
h.usage[path] = &Usage{UsedPercent: usedPct, AvailGB: availGB, TotalGB: 100, UsedGB: usedPct}
|
||||
}
|
||||
|
||||
func (h *harness) check(t *testing.T) {
|
||||
t.Helper()
|
||||
if err := h.w.Check(); err != nil {
|
||||
t.Fatalf("Check: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
var photos = Target{Path: "/mnt/felhom-drives/hdd_1", Label: "Fotók"}
|
||||
|
||||
// --- Scenario E — warned once, and the second pass is silent -------------------------------------
|
||||
|
||||
func TestWarnsOnceThenIsSilent(t *testing.T) {
|
||||
h := newHarness(t, photos)
|
||||
h.set(photos.Path, 87, 4.2)
|
||||
|
||||
h.check(t)
|
||||
if len(h.events) != 1 {
|
||||
t.Fatalf("got %d events on the first crossing, want 1 — the customer was not warned before "+
|
||||
"the fill, which is the whole customer half of D-c", len(h.events))
|
||||
}
|
||||
e := h.events[0]
|
||||
if e.Band != BandWarning {
|
||||
t.Fatalf("band = %v, want warning", e.Band)
|
||||
}
|
||||
if e.Band.EventType() != "disk_warning" {
|
||||
t.Fatalf("event type = %q, want disk_warning", e.Band.EventType())
|
||||
}
|
||||
if e.Band.Severity() != "warning" {
|
||||
t.Fatalf("severity = %q, want warning — the hub's severityNotifies DROPS \"info\", so a "+
|
||||
"wrong severity stores the event and mails nobody", e.Band.Severity())
|
||||
}
|
||||
|
||||
// SECOND PASS, nothing changed. Edge-triggered means exactly nothing fires.
|
||||
h.check(t)
|
||||
if len(h.events) != 1 {
|
||||
t.Fatalf("got %d events after an unchanged second pass, want still 1 — the warning is not "+
|
||||
"edge-triggered, so a daily schedule would re-warn the customer every single night",
|
||||
len(h.events))
|
||||
}
|
||||
}
|
||||
|
||||
// The state must survive a restart — a fresh Watcher over the SAME file must not re-warn.
|
||||
func TestEdgeStateSurvivesARestart(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
state := filepath.Join(dir, "fillwatch.json")
|
||||
usage := map[string]*Usage{photos.Path: {UsedPercent: 87, AvailGB: 4.2, TotalGB: 100}}
|
||||
targets := func() []Target { return []Target{photos} }
|
||||
read := func(p string) *Usage { return usage[p] }
|
||||
|
||||
var first []Event
|
||||
w1 := New(state, log.New(io.Discard, "", 0), targets, read)
|
||||
w1.SetNotify(func(e Event) { first = append(first, e) })
|
||||
if err := w1.Check(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(first) != 1 {
|
||||
t.Fatalf("first controller: %d events, want 1", len(first))
|
||||
}
|
||||
|
||||
// <restart> — a brand-new Watcher, same file.
|
||||
var second []Event
|
||||
w2 := New(state, log.New(io.Discard, "", 0), targets, read)
|
||||
w2.SetNotify(func(e Event) { second = append(second, e) })
|
||||
if err := w2.Check(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(second) != 0 {
|
||||
t.Fatalf("a restarted controller re-warned about an already-warned filesystem (%d events) — "+
|
||||
"the state is not persisted, so every restart nags the customer", len(second))
|
||||
}
|
||||
}
|
||||
|
||||
// --- Scenario F — it clears, and it can fire again ------------------------------------------------
|
||||
|
||||
func TestClearsSilentlyThenReArms(t *testing.T) {
|
||||
h := newHarness(t, photos)
|
||||
|
||||
h.set(photos.Path, 87, 4.2)
|
||||
h.check(t)
|
||||
if len(h.events) != 1 {
|
||||
t.Fatalf("warn: got %d events, want 1", len(h.events))
|
||||
}
|
||||
|
||||
// Drop below the CLEAR thresholds (both must hold): 70% used, 12 GB free.
|
||||
h.set(photos.Path, 70, 12)
|
||||
h.check(t)
|
||||
if len(h.events) != 1 {
|
||||
t.Fatalf("clearing fired an event (%d total) — a resolved problem must clear SILENTLY", len(h.events))
|
||||
}
|
||||
if b := h.w.Bands()[photos.Path]; b != BandOK {
|
||||
t.Fatalf("band after clearing = %v, want ok — it is latched, and the customer can never be "+
|
||||
"warned about this filesystem again", b)
|
||||
}
|
||||
|
||||
// Cross again — a NEW crossing must warn again.
|
||||
h.set(photos.Path, 88, 3.9)
|
||||
h.check(t)
|
||||
if len(h.events) != 2 {
|
||||
t.Fatalf("a NEW crossing after a clear produced %d events total, want 2 — the warning is "+
|
||||
"latched forever after one firing", len(h.events))
|
||||
}
|
||||
}
|
||||
|
||||
// The dead zone is the hysteresis. A filesystem that falls back to 80% — below warn, above clear —
|
||||
// must NOT clear, or it flaps warned/cleared/warned as it wobbles across one line.
|
||||
func TestDeadZoneHoldsThePreviousBand(t *testing.T) {
|
||||
h := newHarness(t, photos)
|
||||
h.set(photos.Path, 87, 4.2)
|
||||
h.check(t)
|
||||
|
||||
h.set(photos.Path, 80, 6) // between clear (75 / 7 GB) and warn (85 / 5 GB)
|
||||
h.check(t)
|
||||
if b := h.w.Bands()[photos.Path]; b != BandWarning {
|
||||
t.Fatalf("band in the dead zone = %v, want warning held — without hysteresis a filesystem "+
|
||||
"hovering on the line flaps between warned and cleared", b)
|
||||
}
|
||||
if len(h.events) != 1 {
|
||||
t.Fatalf("the dead zone produced an extra event (%d total)", len(h.events))
|
||||
}
|
||||
}
|
||||
|
||||
// Escalation warning → critical MUST fire: it is a different message and a different urgency.
|
||||
func TestEscalationToCriticalFires(t *testing.T) {
|
||||
h := newHarness(t, photos)
|
||||
h.set(photos.Path, 87, 4.2)
|
||||
h.check(t)
|
||||
|
||||
h.set(photos.Path, 96, 1.4)
|
||||
h.check(t)
|
||||
if len(h.events) != 2 {
|
||||
t.Fatalf("got %d events, want 2 — an escalation from warning to critical went unreported", len(h.events))
|
||||
}
|
||||
e := h.events[1]
|
||||
if e.Band != BandCritical || e.Band.EventType() != "disk_critical" {
|
||||
t.Fatalf("second event = %v/%s, want critical/disk_critical", e.Band, e.Band.EventType())
|
||||
}
|
||||
// De-escalating critical → warning must be silent (it is still bad; do not celebrate).
|
||||
h.set(photos.Path, 87, 4.2)
|
||||
h.check(t)
|
||||
if len(h.events) != 2 {
|
||||
t.Fatalf("a critical→warning de-escalation fired (%d total) — only escalation notifies", len(h.events))
|
||||
}
|
||||
}
|
||||
|
||||
// --- Scenario I / §8.4 — a nil usage read is never a warning -------------------------------------
|
||||
|
||||
func TestUnreadableFilesystemNeverWarns(t *testing.T) {
|
||||
h := newHarness(t, photos)
|
||||
// No entry in h.usage → the seam returns nil, which is what system.GetDiskUsage does on error.
|
||||
h.check(t)
|
||||
|
||||
if len(h.events) != 0 {
|
||||
t.Fatalf("an UNREADABLE filesystem produced %d warning(s) — an absent, unmounted or "+
|
||||
"unreadable drive is the drive gate's business and has its own alert; reporting it as "+
|
||||
"\"full\" is a false alarm with a misleading cause, and tells the customer to delete "+
|
||||
"files that are not the problem (§8.4)", len(h.events))
|
||||
}
|
||||
if b, ok := h.w.Bands()[photos.Path]; ok && b != BandOK {
|
||||
t.Fatalf("an unreadable filesystem was recorded as %v", b)
|
||||
}
|
||||
}
|
||||
|
||||
// An unreadable filesystem must not CLEAR an existing warning either — that would silently retract a
|
||||
// true alarm the moment a drive blipped.
|
||||
func TestUnreadableDoesNotClearAnExistingWarning(t *testing.T) {
|
||||
h := newHarness(t, photos)
|
||||
h.set(photos.Path, 87, 4.2)
|
||||
h.check(t)
|
||||
|
||||
delete(h.usage, photos.Path) // now unreadable
|
||||
h.check(t)
|
||||
if b := h.w.Bands()[photos.Path]; b != BandWarning {
|
||||
t.Fatalf("band after an unreadable read = %v, want the warning HELD — a blipping drive "+
|
||||
"would otherwise silently retract a true alarm", b)
|
||||
}
|
||||
}
|
||||
|
||||
// --- Group H — the thresholds keep their gap ------------------------------------------------------
|
||||
|
||||
// A warn and a clear threshold that can be edited into equality is a flapping bug waiting to be
|
||||
// introduced. This pins the ORDERING and a real margin, not the literal numbers.
|
||||
func TestThresholdsKeepTheirHysteresisGap(t *testing.T) {
|
||||
if ClearUsedPercent >= WarnUsedPercent {
|
||||
t.Fatalf("ClearUsedPercent (%.1f) must be strictly BELOW WarnUsedPercent (%.1f) — equal "+
|
||||
"thresholds make a filesystem sitting on the line warn, clear, warn, clear every check",
|
||||
ClearUsedPercent, WarnUsedPercent)
|
||||
}
|
||||
if ClearFreeGiB <= WarnFreeGiB {
|
||||
t.Fatalf("ClearFreeGiB (%.1f) must be strictly ABOVE WarnFreeGiB (%.1f) — the free-byte term "+
|
||||
"needs the same hysteresis as the percentage term, or it flaps on its own",
|
||||
ClearFreeGiB, WarnFreeGiB)
|
||||
}
|
||||
// A margin, not merely an inequality: a 0.1-point gap is arithmetically a gap and practically none.
|
||||
if WarnUsedPercent-ClearUsedPercent < 5 {
|
||||
t.Fatalf("the used-percent hysteresis gap is %.1f points — too narrow to damp real wobble",
|
||||
WarnUsedPercent-ClearUsedPercent)
|
||||
}
|
||||
if ClearFreeGiB-WarnFreeGiB < 1 {
|
||||
t.Fatalf("the free-space hysteresis gap is %.1f GiB — too narrow", ClearFreeGiB-WarnFreeGiB)
|
||||
}
|
||||
if CritUsedPercent <= WarnUsedPercent || CritFreeGiB >= WarnFreeGiB {
|
||||
t.Fatal("the critical band must be strictly tighter than the warning band on BOTH terms, " +
|
||||
"or a filesystem can be critical without ever having been warned")
|
||||
}
|
||||
}
|
||||
|
||||
// Both terms must be able to trip INDEPENDENTLY — that is the entire reason there are two.
|
||||
func TestEitherTermCanTripTheWarning(t *testing.T) {
|
||||
// A big drive: only 60% used, but under the free-space floor. 85% of a 4 TB drive leaves 600 GB,
|
||||
// so the percentage alone would never fire here.
|
||||
if got := classify(Usage{UsedPercent: 60, AvailGB: 3}, BandOK); got != BandWarning {
|
||||
t.Fatalf("60%% used with 3 GB free classified as %v — the free-byte term did not trip, so a "+
|
||||
"large drive can run out of space without ever warning", got)
|
||||
}
|
||||
// A small volume: plenty of GB free in absolute terms is impossible here, so the percentage is
|
||||
// what must fire. 88% of a 100 GB volume leaves 12 GB — above the 5 GiB floor.
|
||||
if got := classify(Usage{UsedPercent: 88, AvailGB: 12}, BandOK); got != BandWarning {
|
||||
t.Fatalf("88%% used with 12 GB free classified as %v — the percentage term did not trip", got)
|
||||
}
|
||||
if got := classify(Usage{UsedPercent: 50, AvailGB: 50}, BandOK); got != BandOK {
|
||||
t.Fatalf("a healthy filesystem classified as %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// --- The copy ------------------------------------------------------------------------------------
|
||||
|
||||
// The customer message must be ACTIONABLE and specific. A warning that says only "something is
|
||||
// filling" is an operator alert wearing the wrong clothes.
|
||||
func TestCustomerMessageNamesTheDriveTheSpaceAndTheAction(t *testing.T) {
|
||||
msg := Message(photos, Usage{UsedPercent: 87, AvailGB: 4.2, TotalGB: 100}, BandWarning)
|
||||
|
||||
if !strings.Contains(msg, "Fotók") {
|
||||
t.Fatalf("the message does not name the storage by its LABEL — a path means nothing to a "+
|
||||
"customer. Got: %s", msg)
|
||||
}
|
||||
if !strings.Contains(msg, "4,2 GB") {
|
||||
t.Fatalf("the message does not give the free space in Hungarian number format (decimal "+
|
||||
"COMMA). Got: %s", msg)
|
||||
}
|
||||
if !strings.Contains(msg, "87%") {
|
||||
t.Fatalf("the message does not give the fill percentage. Got: %s", msg)
|
||||
}
|
||||
if !strings.Contains(msg, "szabadíts fel helyet") && !strings.Contains(msg, "szabadíts fel helyet:") {
|
||||
t.Fatalf("the message does not tell the customer WHAT TO DO. Got: %s", msg)
|
||||
}
|
||||
// Design-system rule: no emoji in customer copy.
|
||||
for _, r := range msg {
|
||||
if r > 0x2100 {
|
||||
t.Fatalf("the message contains an emoji/symbol %q — the design system forbids it in "+
|
||||
"customer copy. Got: %s", string(r), msg)
|
||||
}
|
||||
}
|
||||
|
||||
crit := Message(photos, Usage{UsedPercent: 96, AvailGB: 1.4, TotalGB: 100}, BandCritical)
|
||||
if crit == msg {
|
||||
t.Fatal("the critical message is identical to the warning — the urgency must differ")
|
||||
}
|
||||
if !strings.Contains(crit, "1,4 GB") || !strings.Contains(crit, "Fotók") {
|
||||
t.Fatalf("the critical message lost its specifics. Got: %s", crit)
|
||||
}
|
||||
}
|
||||
|
||||
// A label-less target must still produce a usable message rather than an empty quote.
|
||||
func TestMessageFallsBackToThePathWhenUnlabelled(t *testing.T) {
|
||||
msg := Message(Target{Path: "/mnt/sys_drive"}, Usage{UsedPercent: 90, AvailGB: 1.5}, BandWarning)
|
||||
if !strings.Contains(msg, "/mnt/sys_drive") {
|
||||
t.Fatalf("an unlabelled target rendered without any identifier: %s", msg)
|
||||
}
|
||||
}
|
||||
|
||||
// A decommissioned drive must fall out of the state file, so it cannot grow without bound and a
|
||||
// re-added drive starts fresh.
|
||||
func TestVanishedTargetIsForgotten(t *testing.T) {
|
||||
h := newHarness(t, photos)
|
||||
h.set(photos.Path, 87, 4.2)
|
||||
h.check(t)
|
||||
if _, ok := h.w.Bands()[photos.Path]; !ok {
|
||||
t.Fatal("the warned filesystem was not recorded")
|
||||
}
|
||||
|
||||
h.target = nil // the drive is decommissioned
|
||||
h.check(t)
|
||||
if _, ok := h.w.Bands()[photos.Path]; ok {
|
||||
t.Fatal("a removed filesystem kept its band — the state file grows without bound and a " +
|
||||
"re-added drive would resume from a stale band instead of warning afresh")
|
||||
}
|
||||
}
|
||||
@@ -298,6 +298,38 @@ func (n *Notifier) NotifyBackupFailed(message, errMsg string) {
|
||||
n.PushEvent("backup_failed", "error", message, BackupDetails{Error: errMsg})
|
||||
}
|
||||
|
||||
// RecoveryUnitFailureDetails is the machine-readable tail of a Tier-1 capture failure. App NAMES and
|
||||
// byte figures only — never an env value (§9.5).
|
||||
type RecoveryUnitFailureDetails struct {
|
||||
App string `json:"app"`
|
||||
Error string `json:"error"`
|
||||
TargetPath string `json:"target_path,omitempty"`
|
||||
UsedGB float64 `json:"used_gb,omitempty"`
|
||||
AvailGB float64 `json:"avail_gb,omitempty"`
|
||||
TotalGB float64 `json:"total_gb,omitempty"`
|
||||
UsedPercent float64 `json:"used_percent,omitempty"`
|
||||
// SpaceKnown distinguishes "we read the filesystem and it says these numbers" from "we could not
|
||||
// read it". Without it, an unreadable target is indistinguishable from an empty one — the
|
||||
// presence-is-not-success trap, in the other direction.
|
||||
SpaceKnown bool `json:"space_known"`
|
||||
}
|
||||
|
||||
// NotifyRecoveryUnitCaptureFailed sends the OPERATOR-TIER alert for a per-app Tier-1 recovery-unit
|
||||
// capture failure (R-158, D-c's operator half).
|
||||
//
|
||||
// DELIBERATELY NOT `backup_failed`. That type carries a `customerMessages` entry AND sits in
|
||||
// `settings.DefaultEnabledEvents`, so reusing it would email the customer, in Hungarian, that their
|
||||
// backup failed — an event they can take no action on. It is exactly the mistake R-97a avoided by
|
||||
// minting `whole_guest_backup_failed`, and the reasoning is written into the hub's handler.go.
|
||||
// R-158's original proposal named `backup_failed`; decision D-c routes this to the operator, and
|
||||
// where the two disagree D-c wins.
|
||||
//
|
||||
// Operator-only is enforced by the hub's `notify.operatorOnlyEvents` register, NOT by the absence of
|
||||
// a customerMessages entry — v0.78.0 claimed the latter and was wrong.
|
||||
func (n *Notifier) NotifyRecoveryUnitCaptureFailed(message string, d RecoveryUnitFailureDetails) {
|
||||
n.PushEvent("recovery_unit_capture_failed", "error", message, d)
|
||||
}
|
||||
|
||||
// NotifyOffboxEnlargeBlocked sends a WARNING (not a failure) when an app's enlarged offsite push was
|
||||
// refused by the pre-push quota gate — its config+DB were still saved. Customer-facing (Hungarian
|
||||
// body). NOTE: the event type "offbox_enlarge_blocked" must be added to the hub's allowedEventTypes +
|
||||
|
||||
Reference in New Issue
Block a user