From 88897a224eaceb15b9b97f4dadc32a010fb64d75 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 3 Aug 2026 13:46:14 +0200 Subject: [PATCH] =?UTF-8?q?v0.194.0=20=E2=80=94=20one=20operator=20email?= =?UTF-8?q?=20per=20backup=20run,=20and=20nothing=20dropped=20without=20a?= =?UTF-8?q?=20trace=20(R-182)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed events reached the hub and TWO operator emails went out. The hub's operator cooldown key is customerID:eventType(+tier) and that event carries `app` but no `tier`, so the key held no app identifier: the first refused app took the hour's slot and every other app's failure was discarded BEFORE anything was written down, leaving no row on any channel. The obvious fix — put `app` in the key — was ruled against: on a full disk it produces one email per app, the volume problem wearing the correctness problem's clothes. internal/backup/runsummary.go: a per-run collector with exactly admissionSet's lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the end and only when something failed. A clean run emits nothing. The per-app event stays and becomes the RECORD — the hub routes it record-only, stored and logged every time, never competing for an email slot. The record and the notification are now different things. Deliberate skips (disconnected, decommissioned) are excluded: they have their own alert, and a nightly email about an unplugged drive is one the operator learns to ignore. A manual run always reports: the digest carries a unique run_id the cooldown cannot collapse. Someone pressing the button is actively trying to get a backup. THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a capture failure found between runs would be recorded and never notified — a new silence introduced while closing one. That path emits a digest with NO run_id, so the ordinary 1-hour cooldown caps it exactly as before while the mail now lists every failing app instead of whichever was first. A refusal is recorded ONCE, where the verdict is taken, not at the three legs that consult it — R-181's contract is one verdict per app per run. Noting it per leg listed one refused app three times and produced "2 of 1 apps failed". Found by the digest's own test, not in review. Silence is safe because the hub's deadline check raises expected_backup_missed from report freshness, independently of any mail this box sends (monitor/deadline.go:396,417). Confirmed, not assumed. 7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first attempt — the AST test walked the backup package and not main.go; the test was fixed and the mutation re-run rather than the pass recorded. --- CHANGELOG.md | 50 ++++ REUSE.md | 1 + controller/cmd/controller/main.go | 19 ++ controller/internal/api/router.go | 1 + controller/internal/backup/admission.go | 6 + controller/internal/backup/admission_test.go | 46 ++- controller/internal/backup/backup.go | 37 ++- .../internal/backup/capture_floor_test.go | 2 +- controller/internal/backup/recovery_unit.go | 2 + controller/internal/backup/runsummary.go | 270 ++++++++++++++++++ controller/internal/backup/runsummary_test.go | 270 ++++++++++++++++++ controller/internal/notify/notifier.go | 38 +++ controller/internal/web/handler_debug.go | 1 + 13 files changed, 739 insertions(+), 4 deletions(-) create mode 100644 controller/internal/backup/runsummary.go create mode 100644 controller/internal/backup/runsummary_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 9218e0a..33e9908 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,55 @@ ## Changelog +### v0.194.0 — one operator e-mail per backup run, and nothing dropped without a trace (2026-08-03, R-182) — MinAgent: none + +**The defect, measured rather than supposed.** On 2026-08-03 nine per-app +`recovery_unit_capture_failed` events reached the hub and **two operator e-mails went out**. The hub's +operator cooldown key is `customerID + ":" + eventType + tier-suffix`, and that event carries `app` +but **no `tier`** — so the key held no app identifier. The first refused app took the hour's slot and +**every other app's failure was discarded before anything was written down**, leaving no row on any +channel. A machine deciding not to tell you and nothing happening at all looked identical. + +**The obvious fix was ruled against, and the reason is worth keeping.** Putting `app` into the key +fixes the swallowing by producing **one e-mail per failing app**, which on a full disk is a dozen — +the volume problem wearing the correctness problem's clothes. + +**What ships instead: ONE digest per run, and every failure recorded when it happens.** + +- **`internal/backup/runsummary.go`** — a per-run collector with exactly `admissionSet`'s lifetime + (created where the run begins, cleared when it ends), fed by all three write legs. It emits + `backup_run_failures` once at the end, **only when something failed**. A clean run emits nothing — + not an empty digest. +- **The per-app event stays and becomes the RECORD.** The hub now routes it *record-only*: stored and + written to the notification log every time, never competing for an e-mail slot. The record and the + notification are now different things, which is the durable half of this change. +- **Deliberate skips are not failures.** A disconnected or decommissioned drive has its own alert and + is excluded, because a nightly e-mail about an unplugged drive is one the operator learns to ignore. +- **A manual run always reports**, even if the nightly one already wrote that hour: the digest carries + a unique `run_id` that the hub's cooldown cannot collapse. Someone pressing the button is actively + trying to get a backup. + +**THE PERIODIC SWEEP GETS A DIGEST TOO, and that is not symmetry for its own sake.** `GetFullStatus` +captures units outside any run. With the per-app event now record-only, a capture failure found +between runs would have been recorded and **never notified** — a new silence introduced while closing +one. So that path emits a digest as well, deliberately with **no `run_id`**, so the ordinary 1-hour +cooldown caps it exactly as before while the mail now lists *every* failing app instead of whichever +one happened to be first. + +**A refusal is recorded ONCE, where the verdict is taken**, not at each of the three legs that consult +it — R-181's contract is that one verdict covers all three. Noting it per leg listed a single refused +app three times and produced counts like *"2 of 1 apps failed"*. **Found by the digest's own test, not +in review.** + +**Why silence is safe, checked rather than assumed.** A digest is only safe if the absence of a mail +cannot mean "the run never finished". It cannot: the hub's daily deadline check raises +`expected_backup_missed` / `expected_dbdump_missed` (`hub/internal/monitor/deadline.go:396,417`) from +the box's **report freshness and stored events**, independently of any mail this box chooses to send. + +**Tests: 7 new, plus 4 red-proofs demonstrated failing then restored.** One of them — +the `main.go` seam walk — **did not fail on its first attempt**, because the AST test walked the +backup package and not `main.go`; the test was fixed and the mutation re-run rather than the pass +being recorded. + ### v0.193.1 — the refusal's size estimate is rendered in bytes, not `0.00 GiB` (2026-08-03, R-181 follow-on) **Found by the live proof run for v0.193.0, not by review.** The refusal message printed the estimate diff --git a/REUSE.md b/REUSE.md index b8d40ca..11f7194 100644 --- a/REUSE.md +++ b/REUSE.md @@ -174,6 +174,7 @@ | `api.GracefulSelfRestart` | controller/internal/api/selfrestart.go | `(logger)` | Controller self-restart | Detached exit; bootstrap unit re-runs the image | | `Settings.AddPendingEvent/DrainPendingEvents` | controller/internal/settings/settings.go | offline event queue | Events while hub unreachable | — | | `Manager.SetUnitNotify` + `UnitSpace` (R-158/R-167, v0.191.0) | controller/internal/backup/recovery_unit.go | `(func(stack string, err error, *UnitSpace))` | THE per-app Tier-1 recovery-unit capture failure alert — fires PER APP from `captureAllRecoveryUnits`, loop continues | **OPERATOR-TIER** (`recovery_unit_capture_failed`, in the hub's `operatorOnlyEvents`). **NEVER route it to `backup_failed`** — that type is in `DefaultEnabledEvents` and carries Hungarian copy, so it emails the CUSTOMER about a failure they cannot act on (D-c; R-158's own proposal said `backup_failed` and D-c overrides it). `UnitSpace` is **nil when the target filesystem is unreadable** and renders as *"unavailable"*, never as zeros — "0 GB free" and "we could not look" are opposite diagnoses. No controller-side cooldown: the hub owns it | +| `Manager.beginRunSummary` / `noteFailure` / `noteAttempted` / `emitRunSummary` / `SetRunSummaryNotify` (R-182, v0.194.0) | controller/internal/backup/runsummary.go | `(kind, runID) func()` / `(app, leg, reason)` / `(RunSummary)` | **THE per-run operator digest.** One `backup_run_failures` event at the end of a run listing every failed app, its leg and its reason — emitted ONLY when something failed | **The RECORD and the NOTIFICATION are different things and must stay so.** The per-app `recovery_unit_capture_failed` event is the record (hub routes it *record-only*, stored + logged every time); this digest is the notification. Before R-182 one event was both, and did neither: nine arrived, two were mailed, seven vanished before `LogNotification`. **Lifetime is `admissionSet`'s exactly** — absent collector means "no run in flight", never a stale answer. **A refusal is noted ONCE, inside `admitApp` where the verdict is taken**, not at the three legs that consult it: R-181's one-verdict-covers-all-three contract makes per-leg noting produce "2 of 1 apps failed". **Deliberate skips (disconnected / decommissioned) must NEVER be noted** — they have their own alert and a nightly digest about an unplugged drive is an ignored digest. **A clean run emits NOTHING**; silence is safe only because the hub's deadline check (`monitor/deadline.go:396,417`) raises a missed backup from report freshness independently — if that is ever weakened this design loses its footing. **`run_id` is unique per real run** (so the hub's 1-h cooldown cannot collapse a manual run into the nightly one) and **deliberately EMPTY on the periodic refresh sweep**, which must stay under that cooldown or a polled status page becomes a mail flood | | `Manager.admitApp` / `beginAdmissionRun` / `decideAdmission` / `estimatedWriteBytes` (R-181, v0.193.0) | controller/internal/backup/admission.go | `(stackName) bool` / `() func()` | **THE reserve gate. Call it before ANY per-app backup write** — one verdict per app per run, covering the DB dump, the volume dump and the unit capture (all three write under one per-app root) | **Decided LAZILY at the app's first write, never once at run start** — app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. **Never re-decided between an app's own legs**: that is exactly the split R-181 closed (bulk written, capture refused). **Reset per run** via the closer `beginAdmissionRun` returns. **Must sit ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided inside it has already bounced the app. Fires **exactly one** `unitNotify` per refused app per run. Nil admission set (periodic status refresh) → decides fresh, which is still once per app per sweep. Wiring pinned by an **AST walk** in `TestAdmission_IsWiredIntoEveryProductionWriteLeg`, not `strings.Contains` | | `Manager.floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` / `floorReason` (R-165 B2 v0.192.0, size term R-181 v0.193.0) | controller/internal/backup/recovery_unit.go | `(*UnitSpace, estGiB float64) (*UnitSpace, floorReason)` | The pure two-question predicate behind `admitApp`: is the filesystem already below the reserve (`floorHeadroom`), and would THIS app's write take it below (`floorSize`)? | **Headroom is about the FILESYSTEM, never a per-unit cap** — a size cap is R-163 rebuilt inside one volume; the size term bounds the *delta*, not the unit. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. **`estGiB == 0` degrades to headroom-only on purpose** — refusing an app with no history makes the FIRST backup the one that can never happen. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk | | `fillwatch.Watcher` (`New`/`SetNotify`/`Check`) (R-167, v0.191.0) | controller/internal/fillwatch/fillwatch.go | `(statePath, logger, targetsFn, usageFn)` → `Check() error` | THE customer fill warning — warns BEFORE a filesystem fills, per FILESYSTEM (never per app: one full disk holding ten apps would fire ten times) | Emits the **pre-existing** `disk_warning`/`disk_critical` pair, which was allowlisted + copy'd + default-enabled with **no producer in any repo** until now — do NOT mint a new type beside it. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range. **Edge-triggered on ESCALATION ONLY**, state persisted; de-escalation is silent and re-arms. Hysteresis dead zone between clear (75% / 7 GiB) and warn — pinned by `TestThresholdsKeepTheirHysteresisGap`. **A nil usage read is NEVER a warning** (§8.4). The hub has **no `customerMessages` entry** for either type on purpose — an entry would override the dynamic message and discard the drive label + free space | diff --git a/controller/cmd/controller/main.go b/controller/cmd/controller/main.go index a6e17ed..fdfd94a 100644 --- a/controller/cmd/controller/main.go +++ b/controller/cmd/controller/main.go @@ -744,6 +744,25 @@ func main() { "Recovery unit capture FAILED for %q — the app has no fresh local (Tier-1) backup, and Tier-2/Tier-3 have nothing to copy. %s. Error: %v", stackName, usage.String(), err), d) }) + // R-182: the ONE operator digest per backup run. The per-app events above are the RECORD + // (the hub routes them record-only); this is the NOTIFICATION, sent once at the end of a run + // and only when something actually failed. A clean run sends nothing — and that silence is + // safe because the hub's own daily deadline check raises expected_backup_missed from report + // freshness, independently of any mail this box chooses to send. + backupMgr.SetRunSummaryNotify(func(rs backup.RunSummary) { + d := notify.BackupRunFailuresDetails{ + RunID: rs.RunID, RunKind: rs.RunKind, + Failed: rs.Failed, Attempted: rs.Attempted, + } + for _, a := range rs.Apps { + d.Apps = append(d.Apps, notify.RunFailureDetail{App: a.App, Leg: a.Leg, Reason: a.Reason}) + } + if rs.Usage != nil { + d.TargetPath, d.UsedGB, d.AvailGB = rs.Usage.Path, rs.Usage.UsedGB, rs.Usage.AvailGB + d.TotalGB, d.UsedPercent, d.SpaceKnown = rs.Usage.TotalGB, rs.Usage.UsedPercent, true + } + notifier.NotifyBackupRunFailures(rs.Message, d) + }) // 3a: the pre-push enlargement gate blocked an app's userdata push (config+DB still saved). Edge- // triggered by the engine (only NEW blocks notify), so the hub's per-event-type cooldown suffices — // no controller-side timer (the hub owns cooldown). diff --git a/controller/internal/api/router.go b/controller/internal/api/router.go index d5e9ed0..f8a9393 100644 --- a/controller/internal/api/router.go +++ b/controller/internal/api/router.go @@ -1010,6 +1010,7 @@ func (r *Router) triggerBackup(w http.ResponseWriter, _ *http.Request) { } r.logger.Println("[INFO] [api] Manual app-data backup (DB dump) triggered") + r.backupMgr.MarkManualRun() // R-182: operator-triggered — its digest must not be collapsed into the nightly one go r.backupMgr.RunDBDumps(context.Background()) writeJSON(w, http.StatusOK, apiResponse{OK: true, Message: "Mentés elindítva"}) diff --git a/controller/internal/backup/admission.go b/controller/internal/backup/admission.go index b81b2ad..2443fe2 100644 --- a/controller/internal/backup/admission.go +++ b/controller/internal/backup/admission.go @@ -132,6 +132,12 @@ func (m *Manager) admitApp(stackName string) bool { if m.unitNotify != nil { m.unitNotify(stackName, v.err, v.usage) } + // R-182: the digest entry is recorded HERE, where the verdict is taken — once per app per run. + // Not at the three call sites that consult the memo: R-181's whole contract is that ONE verdict + // covers all three legs, so noting it per leg listed a single refused app three times and + // produced counts like "2 of 1 apps failed". The leg name says what actually happened, which is + // that nothing was attempted at all. + m.noteFailure(stackName, "whole app (refused before any write)", v.err.Error()) return false } diff --git a/controller/internal/backup/admission_test.go b/controller/internal/backup/admission_test.go index dd26b98..6b0a0d6 100644 --- a/controller/internal/backup/admission_test.go +++ b/controller/internal/backup/admission_test.go @@ -15,6 +15,8 @@ import ( "sort" "strings" "testing" + + "gitea.dooplex.hu/admin/felhom-controller/internal/settings" ) // R-181 — the reserve guards the write that fills the disk, and its promise is true. @@ -33,6 +35,7 @@ import ( type admissionProvider struct { stacks []string volumes map[string][]string + hdd map[string]string // per-app drive path, for the drive-state skip tests dir string infoHits []string stopped []string @@ -47,7 +50,7 @@ func (p *admissionProvider) ListDeployedStacks() []StackSummary { return out } func (p *admissionProvider) GetStackHDDMounts(string) []string { return nil } -func (p *admissionProvider) GetStackHDDPath(string) string { return "" } +func (p *admissionProvider) GetStackHDDPath(n string) string { return p.hdd[n] } func (p *admissionProvider) GetImportRoot() string { return "" } func (p *admissionProvider) GetDockerVolumes(name string) []string { if p.volumes == nil { @@ -88,7 +91,7 @@ func newAdmissionHarness(t *testing.T, stacks ...string) *admissionHarness { t.Helper() dir := t.TempDir() h := &admissionHarness{ - prov: &admissionProvider{stacks: stacks, dir: dir}, + prov: &admissionProvider{stacks: stacks, dir: dir, hdd: map[string]string{}}, usage: map[string]*UnitSpace{}, dir: dir, logs: &bytes.Buffer{}, @@ -116,6 +119,45 @@ func newAdmissionHarness(t *testing.T, stacks ...string) *admissionHarness { return h } +// markDisconnected / markDecommissioned put a real settings row behind the drive-state skips, so +// Scenario F exercises the production guards rather than a stub of them. +func (h *admissionHarness) markDisconnected(app string) { + h.driveState(app, true, false) +} + +func (h *admissionHarness) markDecommissioned(app string) { + h.driveState(app, false, true) +} + +func (h *admissionHarness) driveState(app string, disconnected, decommissioned bool) { + if h.m.settings == nil { + sett, err := settings.Load(filepath.Join(h.dir, "settings.json"), log.New(io.Discard, "", 0)) + if err != nil { + panic(err) + } + h.m.settings = sett + } + // Each such app gets its OWN drive path, or marking one would skip them all. + p := filepath.Join(h.dir, "drives", app) + if err := os.MkdirAll(p, 0o755); err != nil { + panic(err) + } + h.prov.hdd[app] = p + if err := h.m.settings.AddStoragePath(settings.StoragePath{Path: p, Label: app}); err != nil { + panic(err) + } + if disconnected { + if err := h.m.settings.SetDisconnected(p, true, nil); err != nil { + panic(err) + } + } + if decommissioned { + if err := h.m.settings.SetDecommissioned(p, ""); err != nil { + panic(err) + } + } +} + func (h *admissionHarness) nsRoot() string { return filepath.Join(h.dir, "felhom-data") } // setSpace states the filesystem's occupancy as a test INPUT — the whole point of the unitSpaceFn diff --git a/controller/internal/backup/backup.go b/controller/internal/backup/backup.go index 0c0d699..cd6ae3d 100644 --- a/controller/internal/backup/backup.go +++ b/controller/internal/backup/backup.go @@ -9,6 +9,7 @@ import ( "path/filepath" "strings" "sync" + "sync/atomic" "time" "gitea.dooplex.hu/admin/felhom-controller/internal/config" @@ -64,6 +65,16 @@ type Manager struct { admissionMu sync.Mutex admission *admissionSet + // summary (R-182) is the per-RUN digest collector, guarded by summaryMu. Same lifetime as + // `admission` and for the same reason: an absent collector means "no run in flight", never a + // stale answer from last night. runSummaryNotify is the operator digest seam, wired in main.go. + summaryMu sync.Mutex + summary *runSummary + runSummaryNotify func(RunSummary) + // manualRun tags the NEXT run as operator-triggered (cleared as the run starts), so the digest + // can say which kind it was and the hub can decline to collapse a manual run into a nightly one. + manualRun atomic.Bool + // appStop (R-166) is the crash marker for operations that stop an app, work on its data, and // start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power // cut in that window leaves a durable record that Recover honours at the next startup. Built in @@ -426,6 +437,14 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error { // question with last night's disk. defer m.beginAdmissionRun()() + // R-182: the digest scope has the same lifetime. `emitRunSummary` runs BEFORE the closer (defers + // unwind last-in-first-out), so the summary is still populated when it is sent, and it sends + // nothing at all when the run was clean. + kind := m.runKindFor() + m.manualRun.Store(false) // tags exactly ONE run; a stale flag would mislabel every later nightly + defer m.beginRunSummary(kind, newRunID())() + defer m.emitRunSummary() + dbs, err := DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames()) if err != nil { m.logger.Printf("[ERROR] [backup] Database discovery failed: %v", err) @@ -465,6 +484,7 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error { // where an app's verdict is taken, because the DB leg runs first; the volume leg and the // capture then read the same memo. SKIP, not FAIL — a deliberate hold is not a broken dump, // and the operator alert (fired once, inside admitApp) is the signal that it happened. + m.noteAttempted(db.StackName) if !m.admitApp(db.StackName) { summary = append(summary, fmt.Sprintf("SKIP %s (reserve — app backup refused)", db.ContainerName)) continue @@ -478,6 +498,7 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error { if result.Error != nil { allOK = false summary = append(summary, fmt.Sprintf("FAIL %s: %v", result.DB.ContainerName, result.Error)) + m.noteFailure(db.StackName, "database dump", result.Error.Error()) m.logger.Printf("[ERROR] [backup] DB dump failed for %s: %v", result.DB.ContainerName, result.Error) } else { totalSize += result.Size @@ -608,6 +629,7 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) { // R-181: the reserve, ahead of DumpAppVolumesSafe so a refused app is NOT stopped. For an app // that already has a DB this is a memo lookup taken before its DB dump; for a volume-only app // this is where its verdict is taken, still before its first byte. + m.noteAttempted(stack.Name) if !m.admitApp(stack.Name) { summary = append(summary, fmt.Sprintf("SKIP %s volumes (reserve — app backup refused)", stack.Name)) continue @@ -616,6 +638,7 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) { if err := dump(stack.Name); err != nil { allOK = false summary = append(summary, fmt.Sprintf("FAIL %s volumes: %v", stack.Name, err)) + m.noteFailure(stack.Name, "volume dump", err.Error()) m.logger.Printf("[ERROR] [backup] Volume dump failed for %s: %v", stack.Name, err) continue } @@ -1001,7 +1024,19 @@ func (m *Manager) RefreshCache(nextDBDump time.Time) { // Phase 2: keep each app's recovery unit current with its definition. Idempotent // (checksum-skip), so this periodic refresh only writes when the config actually changed, // and ensures units exist shortly after startup without waiting for the daily DB dump. - m.captureAllRecoveryUnits() + // + // R-182: this sweep gets its OWN digest scope. It has to, and the reason is the whole + // balance of this change. The per-app event is now record-only, so without a digest here a + // capture failure detected between runs would be recorded and NEVER notified — a new + // silence introduced while closing one. But this path can fire on every status poll, so its + // digest deliberately carries NO run id: the hub's ordinary 1-hour operator cooldown then + // applies, which caps it at one mail an hour exactly as before, while the mail now lists + // EVERY failing app instead of whichever one happened to be first. + func() { + defer m.beginRunSummary(runKindRefresh, "")() + defer m.emitRunSummary() + m.captureAllRecoveryUnits() + }() } // Fill in dynamic fields under lock. diff --git a/controller/internal/backup/capture_floor_test.go b/controller/internal/backup/capture_floor_test.go index 2a96ceb..c0e35b8 100644 --- a/controller/internal/backup/capture_floor_test.go +++ b/controller/internal/backup/capture_floor_test.go @@ -42,7 +42,7 @@ func (p *floorProvider) GetImportRoot() string { return "" } func (p *floorProvider) GetDockerVolumes(string) []string { return nil } func (p *floorProvider) StopStack(string) error { return nil } func (p *floorProvider) StartStack(string) error { return nil } -func (p *floorProvider) RefreshAndIsRunning(string) bool { return true } +func (p *floorProvider) RefreshAndIsRunning(string) bool { return true } func (p *floorProvider) GetStackRecoveryInfo(name string) (RecoveryInfo, bool) { p.infoHits = append(p.infoHits, name) return RecoveryInfo{StackDir: filepath.Join(p.dir, "stacks", name)}, true diff --git a/controller/internal/backup/recovery_unit.go b/controller/internal/backup/recovery_unit.go index de29196..1876d84 100644 --- a/controller/internal/backup/recovery_unit.go +++ b/controller/internal/backup/recovery_unit.go @@ -351,11 +351,13 @@ func (m *Manager) captureAllRecoveryUnits() { if m.settings != nil && (m.settings.IsDisconnected(drivePath) || m.settings.IsDecommissioned(drivePath)) { continue // drive not writable — skip, the existing unit stays as-is } + m.noteAttempted(stack.Name) // The reserve, checked BEFORE anything is written. Per app, and the loop continues. if !m.admitApp(stack.Name) { continue } if err := m.CaptureRecoveryUnit(stack.Name); err != nil { + m.noteFailure(stack.Name, "recovery-unit capture", err.Error()) m.logger.Printf("[WARN] [backup] Recovery unit capture failed for %s: %v", stack.Name, err) // R-158: per app, and the loop CONTINUES — one app's failure must not silence the // others, and it must not abort their captures either. The space figures are read at diff --git a/controller/internal/backup/runsummary.go b/controller/internal/backup/runsummary.go new file mode 100644 index 0000000..b1ef38a --- /dev/null +++ b/controller/internal/backup/runsummary.go @@ -0,0 +1,270 @@ +package backup + +import ( + "fmt" + "sort" + "strconv" + "strings" + "sync" + "time" +) + +// ── The backup run digest (R-182) ──────────────────────────────────────────────────────────────── +// +// WHAT WAS WRONG, MEASURED. On 2026-08-03 nine per-app `recovery_unit_capture_failed` events reached +// the hub and TWO operator e-mails went out. The hub's operator cooldown key is +// `customerID + ":" + eventType + tier-suffix`, and that event carries `app` but no `tier`, so the +// key held **no app identifier**: the first refused app took the hour's slot and every other app's +// failure was discarded — *before* anything was written down, so it left no row on any channel and +// could not be found afterwards. +// +// WHY NOT JUST PUT THE APP IN THE KEY. That was the obvious fix and the operator ruled against it: +// on a full disk it produces one e-mail per app, which is the volume problem wearing the correctness +// problem's clothes. **One digest per run instead**, listing every failure — and separately, every +// failure recorded when it happens. +// +// THE RECORD AND THE NOTIFICATION ARE DIFFERENT THINGS, and that separation is the durable part: +// +// - the RECORD is the per-app `recovery_unit_capture_failed` event, emitted unconditionally, now +// routed record-only by the hub so it never competes for an e-mail slot; +// - the NOTIFICATION is this digest, emitted once per run and only when something failed. +// +// WHY A DIGEST IS SAFE HERE — the one thing that could have made it dangerous. A digest introduces a +// silent-failure path if the ABSENCE of an e-mail could mean "the run never finished". It cannot: +// the hub's daily deadline check raises `expected_backup_missed` / `expected_dbdump_missed` +// (`hub/internal/monitor/deadline.go:396,417`) from the box's REPORT freshness and its stored +// events, entirely independently of any mail the controller chooses to send. Silence therefore still +// means "the run finished and found nothing wrong". If that check is ever weakened, this design +// loses its footing — which is why it is named here and not only in a report. + +// runKind distinguishes the paths that can produce a digest. It is carried in the details and shown +// in the subject, because "the nightly run failed" and "the run I just triggered failed" are read +// differently at 07:00. +const ( + runKindNightly = "nightly" + runKindManual = "manual" + // runKindRefresh is the PERIODIC status sweep, which captures units outside any backup run. + // + // IT DELIBERATELY CARRIES NO run_id. A run digest gets a unique run id so the hub's 1-hour + // cooldown can never collapse two real runs (the operator ruled on that explicitly). The sweep + // is the opposite case: it can fire every time the status page is polled, so it MUST fall under + // the ordinary cooldown or a full disk becomes a mail flood — the exact failure this whole + // change exists to avoid, arriving from the other direction. + // + // Without this path the sweep's failures would be recorded and never notified, because the + // per-app event is now record-only — a NEW silence introduced while fixing a silence. This is + // what stops that. + runKindRefresh = "refresh" +) + +// runFailure is one app's failed or refused leg within a run. +type runFailure struct { + App string + Leg string + Reason string +} + +// runSummary is the per-RUN collector. Same lifetime as `admissionSet` and for the same reason: an +// absent collector must mean "no run in flight", never "a stale answer from last night". +type runSummary struct { + mu sync.Mutex + kind string + runID string + attempted map[string]bool // apps this run actually tried to back up + failures []runFailure +} + +// beginRunSummary opens the per-run digest scope and returns the closer, mirroring +// beginAdmissionRun. A second call while one is live REPLACES it and the closer restores the +// previous, so nesting cannot silently drop a caller's scope. +func (m *Manager) beginRunSummary(kind, runID string) func() { + m.summaryMu.Lock() + prev := m.summary + m.summary = &runSummary{kind: kind, runID: runID, attempted: map[string]bool{}} + m.summaryMu.Unlock() + return func() { + m.summaryMu.Lock() + m.summary = prev + m.summaryMu.Unlock() + } +} + +// noteAttempted records that this run tried to back up an app. +// +// THE DENOMINATOR IS NOT DECORATION. "3 of 4 apps failed" is a catastrophe and "3 of 40" is a bad +// night, and a list of names cannot tell them apart — the operator's first decision, get up now or +// look after coffee, is made from exactly this ratio. +func (m *Manager) noteAttempted(app string) { + m.summaryMu.Lock() + defer m.summaryMu.Unlock() + if m.summary != nil { + m.summary.mu.Lock() + m.summary.attempted[app] = true + m.summary.mu.Unlock() + } +} + +// noteFailure records one failed or refused leg. Deliberate skips must NOT come through here — +// §8.1: a drive that is unplugged or decommissioned has its own alert, and putting it in the digest +// turns a nightly e-mail into one the operator learns to ignore. +func (m *Manager) noteFailure(app, leg, reason string) { + m.summaryMu.Lock() + defer m.summaryMu.Unlock() + if m.summary == nil { + return // no run in flight — nothing to summarise + } + m.summary.mu.Lock() + m.summary.attempted[app] = true + m.summary.failures = append(m.summary.failures, runFailure{App: app, Leg: leg, Reason: reason}) + m.summary.mu.Unlock() +} + +// emitRunSummary sends the digest, if and only if something failed. +// +// A CLEAN RUN EMITS NOTHING — not an empty digest. A nightly "0 failures" mail is an unread mail +// within a week, and it would also destroy the property the whole design rests on: that silence +// means the run finished and found nothing wrong. +func (m *Manager) emitRunSummary() { + m.summaryMu.Lock() + s := m.summary + m.summaryMu.Unlock() + if s == nil { + return + } + s.mu.Lock() + failures := append([]runFailure(nil), s.failures...) + attempted := len(s.attempted) + kind, runID := s.kind, s.runID + s.mu.Unlock() + + if len(failures) == 0 { + return + } + // Stable order so two identical runs render identically and a diff of two mails is meaningful. + sort.Slice(failures, func(i, j int) bool { + if failures[i].App != failures[j].App { + return failures[i].App < failures[j].App + } + return failures[i].Leg < failures[j].Leg + }) + + if m.runSummaryNotify == nil { + // Nil-safe, but say so: an unwired seam here means the digest exists and reaches nobody, + // which is the built-but-never-wired failure this project has shipped four times. + m.logger.Printf("[WARN] [backup] %d app(s) failed in this %s run but no run-summary notifier is wired — "+ + "the failures are recorded per app and NOT summarised to the operator", len(failures), kind) + return + } + + apps := make([]RunFailureDetail, 0, len(failures)) + names := make([]string, 0, len(failures)) + for _, f := range failures { + apps = append(apps, RunFailureDetail{App: f.App, Leg: f.Leg, Reason: f.Reason}) + names = append(names, f.App) + } + msg := fmt.Sprintf("%d of %d apps failed to back up in this %s run: %s", + len(failures), attempted, kind, strings.Join(dedupeStable(names), ", ")) + + m.logger.Printf("[INFO] [backup] Run summary: %d of %d apps failed (%s run) — notifying the operator once", + len(failures), attempted, kind) + m.runSummaryNotify(RunSummary{ + RunID: runID, + RunKind: kind, + Failed: len(failures), + Attempted: attempted, + Apps: apps, + Usage: m.summaryUsage(), + Message: msg, + }) +} + +// summaryUsage reads the target filesystem once for the digest. It uses the same seam the reserve +// does, so a test states occupancy as an input rather than manufacturing it. Nil when unreadable — +// which the hub renders as "unavailable", never as zeros. +func (m *Manager) summaryUsage() *UnitSpace { + if m.stackProvider == nil { + return nil + } + for _, st := range m.stackProvider.ListDeployedStacks() { + if u := m.readUnitSpace(st.Name); u != nil { + return u + } + } + return nil +} + +func dedupeStable(in []string) []string { + seen := map[string]bool{} + out := in[:0:0] + for _, s := range in { + if !seen[s] { + seen[s] = true + out = append(out, s) + } + } + return out +} + +// RunFailureDetail is one app's failure as the hub receives it. +type RunFailureDetail struct { + App string `json:"app"` + Leg string `json:"leg"` + Reason string `json:"reason"` +} + +// RunSummary is the digest payload handed to the notifier seam. +type RunSummary struct { + RunID string + RunKind string + Failed int + Attempted int + Apps []RunFailureDetail + Usage *UnitSpace + Message string +} + +// SetRunSummaryNotify wires the per-run digest. INIT-ONLY — call once at startup in main.go, +// alongside SetUnitNotify. Nil-safe, but an unwired seam is logged loudly rather than being silently +// the old behaviour. +func (m *Manager) SetRunSummaryNotify(fn func(RunSummary)) { + m.runSummaryNotify = fn +} + +// admissionReason returns the human reason this run refused an app, taken from the verdict the +// reserve already recorded. Reused rather than re-derived: the verdict carries the bound term, the +// estimate and the disk figures, and re-deriving them here would be a second source of truth for a +// sentence the operator reads. +func (m *Manager) admissionReason(stackName string) string { + m.admissionMu.Lock() + defer m.admissionMu.Unlock() + if set := m.admission; set != nil { + if v, ok := set.v[stackName]; ok && v.err != nil { + return v.err.Error() + } + } + return "refused by the capture reserve" +} + +// runKindFor reports whether this run was the scheduled one or one a person triggered. +// +// The distinction is the operator's ruling: someone pressing the button is actively trying to get a +// backup, so a manual run must report even if the nightly one already wrote this hour. It is carried +// into the subject line because "the nightly run failed" and "the run I just asked for failed" are +// acted on differently. +func (m *Manager) runKindFor() string { + if m.manualRun.Load() { + return runKindManual + } + return runKindNightly +} + +// MarkManualRun tags the NEXT backup run as operator-triggered. Called by the API/debug handlers +// that expose a "run backup now" control; the scheduled path leaves it alone. +func (m *Manager) MarkManualRun() { m.manualRun.Store(true) } + +// newRunID returns a per-run identifier. It only has to be distinct between two runs on one box +// within the hub's 1-hour cooldown window, which a nanosecond clock reading satisfies; it is never +// persisted, compared across boxes, or used as a security value. +func newRunID() string { + return "run-" + strconv.FormatInt(time.Now().UnixNano(), 36) +} diff --git a/controller/internal/backup/runsummary_test.go b/controller/internal/backup/runsummary_test.go new file mode 100644 index 0000000..b769f4c --- /dev/null +++ b/controller/internal/backup/runsummary_test.go @@ -0,0 +1,270 @@ +package backup + +import ( + "go/ast" + "go/parser" + "go/token" + "strings" + "testing" +) + +// R-182 — one digest per run, listing every failure, and nothing in it that is not a failure. +// +// The assertions here are the CONTENT of the digest, not that a function was called: the defect this +// replaces was one where the machinery ran correctly and the operator was told about one app out of +// nine. + +// digestOf runs the two legs that can run without Docker inside one digest scope and returns the +// summary that would have been sent (nil when none). It mirrors what runDBDumpsInternal does; the +// DB leg's wiring is pinned structurally by TestRunSummary_IsWiredIntoTheProductionPath, because +// DiscoverDatabases shells out to `docker`. +func (h *admissionHarness) digestOf(kind string) *RunSummary { + var got *RunSummary + h.m.SetRunSummaryNotify(func(rs RunSummary) { got = &rs }) + doneAdm := h.m.beginAdmissionRun() + doneSum := h.m.beginRunSummary(kind, "run-test") + h.m.runVolumeDumps() + h.m.captureAllRecoveryUnits() + h.m.emitRunSummary() + doneSum() + doneAdm() + return got +} + +// ── Scenario A — several failures, ONE digest, all of them in it ───────────────────────────────── + +func TestRunSummary_ListsEveryFailedApp(t *testing.T) { + apps := []string{"opengist", "privatebin", "immich", "homebox", "nextcloud"} + h := newAdmissionHarness(t, apps...) + for _, a := range apps { + h.setSpace(a, 99, 0.1, 70) // every app refused by the reserve + } + + rs := h.digestOf(runKindNightly) + if rs == nil { + t.Fatal("no digest was emitted although five apps failed — this is the 2026-08-03 defect: " + + "nine failures arrived and two e-mails went out") + } + if rs.Failed != len(apps) { + t.Fatalf("digest reports %d failures, want %d — the others were silently discarded, which is "+ + "exactly what the per-app path did", rs.Failed, len(apps)) + } + seen := map[string]bool{} + for _, a := range rs.Apps { + seen[a.App] = true + if a.Leg == "" { + t.Errorf("%s has no leg name — the operator cannot tell a failed database dump from a "+ + "failed volume dump, and they are different problems", a.App) + } + if !strings.Contains(a.Reason, "reserve") { + t.Errorf("%s reason %q does not say why", a.App, a.Reason) + } + } + for _, a := range apps { + if !seen[a] { + t.Errorf("%s is missing from the digest", a) + } + } + // The denominator: "5 of 5" and "5 of 40" are different nights. + if rs.Attempted < len(apps) { + t.Fatalf("attempted=%d, want >= %d — without the denominator a catastrophe and a bad night "+ + "read identically", rs.Attempted, len(apps)) + } + if rs.Usage == nil { + t.Fatal("the digest carries no filesystem figures — 'one broken app' and 'a full disk' must " + + "be distinguishable before the reasons are read") + } + if !strings.Contains(rs.Message, "of") || !strings.Contains(rs.Message, "nightly") { + t.Fatalf("summary message is not readable as a sentence: %q", rs.Message) + } +} + +// ── Scenario B — a clean run is silent ─────────────────────────────────────────────────────────── + +func TestRunSummary_CleanRunEmitsNothing(t *testing.T) { + h := newAdmissionHarness(t, "opengist", "privatebin") + h.setSpace("opengist", 20, 55, 70) + h.setSpace("privatebin", 20, 55, 70) + + if rs := h.digestOf(runKindNightly); rs != nil { + t.Fatalf("a clean run emitted a digest (%+v) — an empty nightly mail is an unread mail within "+ + "a week, and it would destroy the property this design rests on: that silence means the "+ + "run finished and found nothing wrong", rs) + } +} + +// ── Scenario F — deliberate skips are NOT failures ─────────────────────────────────────────────── + +// A drive that is unplugged or decommissioned has its own alert. Putting it in the digest produces a +// nightly e-mail on a box with an unplugged drive, which trains the operator to ignore digests — and +// an ignored digest is the same outcome as no digest. +func TestRunSummary_DeliberateSkipsAreNotFailures(t *testing.T) { + h := newAdmissionHarness(t, "gone", "retired", "healthy") + for _, a := range []string{"gone", "retired", "healthy"} { + h.setSpace(a, 20, 55, 70) // ample room — nothing may be refused for headroom + } + h.markDisconnected("gone") + h.markDecommissioned("retired") + + rs := h.digestOf(runKindNightly) + if rs != nil { + t.Fatalf("a run whose only 'issues' were a disconnected drive and a decommissioned one "+ + "emitted a digest: %+v — those have their own alerts", rs) + } +} + +// ── The collector's lifetime — the admission.go rule, restated ─────────────────────────────────── + +// An absent collector must mean "no run in flight", never a stale answer from last night. +func TestRunSummary_ResetsBetweenRuns(t *testing.T) { + h := newAdmissionHarness(t, "opengist") + h.setSpace("opengist", 99, 0.1, 70) + if rs := h.digestOf(runKindNightly); rs == nil || rs.Failed != 1 { + t.Fatalf("run 1: want 1 failure, got %+v", rs) + } + h.setSpace("opengist", 20, 55, 70) // space freed between runs + if rs := h.digestOf(runKindNightly); rs != nil { + t.Fatalf("run 2 reported %+v — the previous run's failures were carried over, so freeing "+ + "space could never take effect", rs) + } +} + +// A failure noted with NO run in flight must not panic and must not accumulate anywhere. +func TestRunSummary_NoteOutsideARunIsInert(t *testing.T) { + h := newAdmissionHarness(t, "opengist") + h.m.noteFailure("opengist", "volume dump", "boom") // no scope open + h.m.noteAttempted("opengist") + h.m.emitRunSummary() // must be a no-op, not a nil dereference +} + +// ── The refresh sweep carries NO run id, deliberately ──────────────────────────────────────────── + +// The run digest gets a unique id so the hub's 1-hour cooldown can never collapse two real runs. The +// periodic sweep is the opposite case — it can fire on every status poll — so it must fall UNDER the +// cooldown. Getting this backwards turns a full disk into a mail flood, which is the same failure as +// the one being fixed, arriving from the other side. +func TestRunSummary_RefreshSweepHasNoRunID(t *testing.T) { + h := newAdmissionHarness(t, "opengist") + h.setSpace("opengist", 99, 0.1, 70) + + var got *RunSummary + h.m.SetRunSummaryNotify(func(rs RunSummary) { got = &rs }) + func() { + defer h.m.beginAdmissionRun()() + defer h.m.beginRunSummary(runKindRefresh, "")() + defer h.m.emitRunSummary() + h.m.captureAllRecoveryUnits() + }() + if got == nil { + t.Fatal("the periodic sweep emitted no digest — with the per-app event now record-only, a " + + "capture failure found between runs would be recorded and NEVER notified, which is a " + + "new silence introduced while closing one") + } + if got.RunID != "" { + t.Fatalf("the refresh sweep carries run_id=%q — it must be EMPTY so the hub's ordinary "+ + "1-hour cooldown caps it, or a polled status page becomes a mail flood", got.RunID) + } + if got.RunKind != runKindRefresh { + t.Fatalf("run kind = %q, want %q", got.RunKind, runKindRefresh) + } +} + +// ── The seam is WIRED — walked as an AST, not grepped ──────────────────────────────────────────── + +// Four mechanisms in this project have been built and left disconnected. `strings.Contains` cannot +// tell a live call from a commented-out one, so this parses. +func TestRunSummary_IsWiredIntoTheProductionPath(t *testing.T) { + calls := callsByFunc(t, "backup.go") + + if !hasStr(calls["runDBDumpsInternal"], "beginRunSummary") { + t.Fatal("runDBDumpsInternal does not open a run-summary scope — every failure would be " + + "recorded per app and none of them summarised, which is the pre-R-182 behaviour") + } + if !hasStr(calls["runDBDumpsInternal"], "emitRunSummary") { + t.Fatal("runDBDumpsInternal never emits the summary — the collector fills and is discarded") + } + // Both dump legs and the capture leg must feed it, or a whole class of failure is invisible. + for fn, where := range map[string]string{ + "runDBDumpsInternal": "the database leg", + "runVolumeDumps": "the volume leg", + } { + if !hasStr(calls[fn], "noteFailure") { + t.Fatalf("%s (%s) never calls noteFailure — its failures cannot reach the digest", fn, where) + } + } + capCalls := callsByFunc(t, "recovery_unit.go") + if !hasStr(capCalls["captureAllRecoveryUnits"], "noteFailure") { + t.Fatal("the capture leg never calls noteFailure") + } + + // AND THE SEAM ITSELF must be wired in main.go. Without this assertion the collector fills, the + // digest is built, and `runSummaryNotify` is nil — so nothing is ever sent and every test above + // still passes. That is the built-but-never-wired shape exactly, and the first version of this + // test missed it: commenting the wiring out left the suite green. + mainCalls := callsByFunc(t, "../../cmd/controller/main.go") + wired := false + for _, calls := range mainCalls { + if hasStr(calls, "SetRunSummaryNotify") { + wired = true + } + } + if !wired { + t.Fatal("main.go never calls SetRunSummaryNotify — the digest is assembled and handed to a " + + "nil seam, so no operator mail is ever sent. Note this is an AST walk: a commented-out " + + "call still CONTAINS the string, which is why strings.Contains cannot be used here") + } + + // The manual paths must tag the run, or Scenario E cannot hold. + for _, f := range []string{"../web/handler_debug.go", "../api/router.go"} { + src := parseFile(t, f) + found := false + ast.Inspect(src, func(n ast.Node) bool { + if ce, ok := n.(*ast.CallExpr); ok { + if sel, ok := ce.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "MarkManualRun" { + found = true + } + } + return true + }) + if !found { + t.Fatalf("%s never calls MarkManualRun — an operator-triggered run would be labelled "+ + "nightly and could be collapsed into it", f) + } + } +} + +func parseFile(t *testing.T, path string) *ast.File { + t.Helper() + f, err := parser.ParseFile(token.NewFileSet(), path, nil, 0) // comments dropped + if err != nil { + t.Fatal(err) + } + return f +} + +// callsByFunc maps each top-level func to the names it calls, comments excluded. +func callsByFunc(t *testing.T, path string) map[string][]string { + t.Helper() + file := parseFile(t, path) + out := map[string][]string{} + var cur string + ast.Inspect(file, func(n ast.Node) bool { + switch v := n.(type) { + case *ast.FuncDecl: + cur = v.Name.Name + case *ast.CallExpr: + name := "" + switch fn := v.Fun.(type) { + case *ast.Ident: + name = fn.Name + case *ast.SelectorExpr: + name = fn.Sel.Name + } + if name != "" && cur != "" { + out[cur] = append(out[cur], name) + } + } + return true + }) + return out +} diff --git a/controller/internal/notify/notifier.go b/controller/internal/notify/notifier.go index e815878..f05eb43 100644 --- a/controller/internal/notify/notifier.go +++ b/controller/internal/notify/notifier.go @@ -330,6 +330,44 @@ func (n *Notifier) NotifyRecoveryUnitCaptureFailed(message string, d RecoveryUni n.PushEvent("recovery_unit_capture_failed", "error", message, d) } +// RunFailureDetail is one app's failed leg inside a backup run digest. +type RunFailureDetail struct { + App string `json:"app"` + Leg string `json:"leg"` + Reason string `json:"reason"` +} + +// BackupRunFailuresDetails is the per-RUN digest payload (R-182). App NAMES, leg names, reasons and +// byte figures only — never an env value (§9.4). +type BackupRunFailuresDetails struct { + // RunID makes the hub's 1-hour operator cooldown unable to collapse two real runs into one + // e-mail. EMPTY on the periodic refresh sweep, deliberately: that path can fire on every status + // poll, so it must fall under the ordinary cooldown instead. + RunID string `json:"run_id,omitempty"` + RunKind string `json:"run_kind"` + Failed int `json:"failed"` + Attempted int `json:"attempted"` + TargetPath string `json:"target_path,omitempty"` + UsedGB float64 `json:"used_gb,omitempty"` + AvailGB float64 `json:"avail_gb,omitempty"` + TotalGB float64 `json:"total_gb,omitempty"` + UsedPercent float64 `json:"used_percent,omitempty"` + SpaceKnown bool `json:"space_known"` + Apps []RunFailureDetail `json:"apps"` +} + +// NotifyBackupRunFailures sends the ONE operator digest for a backup run in which something failed +// (R-182). It is the NOTIFICATION; the per-app `recovery_unit_capture_failed` events are the RECORD, +// and the hub routes those record-only so they never compete for an e-mail slot. +// +// OPERATOR-TIER, and for the same reason as its per-app sibling: a customer can act on a full disk +// (that is the fill warning, which fires first and IS customer-facing) but not on a list of which +// apps' backups failed and why. `notify.operatorOnlyEvents` in the hub enforces that — NOT the +// absence of a customerMessages entry, which is a fallback rather than a block (v0.78.0). +func (n *Notifier) NotifyBackupRunFailures(message string, d BackupRunFailuresDetails) { + n.PushEvent("backup_run_failures", "error", message, d) +} + // NotifyOffboxEnlargeBlocked sends a WARNING (not a failure) when an app's enlarged offsite push was // refused by the pre-push quota gate — its config+DB were still saved. Customer-facing (Hungarian // body). NOTE: the event type "offbox_enlarge_blocked" must be added to the hub's allowedEventTypes + diff --git a/controller/internal/web/handler_debug.go b/controller/internal/web/handler_debug.go index 41746f8..926df49 100644 --- a/controller/internal/web/handler_debug.go +++ b/controller/internal/web/handler_debug.go @@ -379,6 +379,7 @@ func (s *Server) debugTriggerDBDump(w http.ResponseWriter, r *http.Request) { writeDebugJSON(w, http.StatusBadRequest, false, "Backup manager nincs konfigurálva", nil) return } + s.backupMgr.MarkManualRun() // R-182: a person pressed this, so its digest must not be collapsed go func() { if err := s.backupMgr.RunDBDumps(context.Background()); err != nil { s.logger.Printf("[WARN] Debug DB dump failed: %v", err)