v0.194.0 — one operator email per backup run, and nothing dropped without a trace (R-182)
gates / gates (push) Successful in 8s

MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed
events reached the hub and TWO operator emails went out. The hub's operator
cooldown key is customerID:eventType(+tier) and that event carries `app` but no
`tier`, so the key held no app identifier: the first refused app took the hour's
slot and every other app's failure was discarded BEFORE anything was written
down, leaving no row on any channel.

The obvious fix — put `app` in the key — was ruled against: on a full disk it
produces one email per app, the volume problem wearing the correctness problem's
clothes.

internal/backup/runsummary.go: a per-run collector with exactly admissionSet's
lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the
end and only when something failed. A clean run emits nothing.

The per-app event stays and becomes the RECORD — the hub routes it record-only,
stored and logged every time, never competing for an email slot. The record and
the notification are now different things.

Deliberate skips (disconnected, decommissioned) are excluded: they have their
own alert, and a nightly email about an unplugged drive is one the operator
learns to ignore.

A manual run always reports: the digest carries a unique run_id the cooldown
cannot collapse. Someone pressing the button is actively trying to get a backup.

THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a
capture failure found between runs would be recorded and never notified — a new
silence introduced while closing one. That path emits a digest with NO run_id,
so the ordinary 1-hour cooldown caps it exactly as before while the mail now
lists every failing app instead of whichever was first.

A refusal is recorded ONCE, where the verdict is taken, not at the three legs
that consult it — R-181's contract is one verdict per app per run. Noting it per
leg listed one refused app three times and produced "2 of 1 apps failed". Found
by the digest's own test, not in review.

Silence is safe because the hub's deadline check raises expected_backup_missed
from report freshness, independently of any mail this box sends
(monitor/deadline.go:396,417). Confirmed, not assumed.

7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first
attempt — the AST test walked the backup package and not main.go; the test was
fixed and the mutation re-run rather than the pass recorded.
This commit is contained in:
2026-08-03 13:46:14 +02:00
parent db0d4b129d
commit 88897a224e
13 changed files with 739 additions and 4 deletions
+50
View File
@@ -1,5 +1,55 @@
## Changelog
### v0.194.0 — one operator e-mail per backup run, and nothing dropped without a trace (2026-08-03, R-182) — MinAgent: none
**The defect, measured rather than supposed.** On 2026-08-03 nine per-app
`recovery_unit_capture_failed` events reached the hub and **two operator e-mails went out**. The hub's
operator cooldown key is `customerID + ":" + eventType + tier-suffix`, and that event carries `app`
but **no `tier`** — so the key held no app identifier. The first refused app took the hour's slot and
**every other app's failure was discarded before anything was written down**, leaving no row on any
channel. A machine deciding not to tell you and nothing happening at all looked identical.
**The obvious fix was ruled against, and the reason is worth keeping.** Putting `app` into the key
fixes the swallowing by producing **one e-mail per failing app**, which on a full disk is a dozen —
the volume problem wearing the correctness problem's clothes.
**What ships instead: ONE digest per run, and every failure recorded when it happens.**
- **`internal/backup/runsummary.go`** — a per-run collector with exactly `admissionSet`'s lifetime
(created where the run begins, cleared when it ends), fed by all three write legs. It emits
`backup_run_failures` once at the end, **only when something failed**. A clean run emits nothing —
not an empty digest.
- **The per-app event stays and becomes the RECORD.** The hub now routes it *record-only*: stored and
written to the notification log every time, never competing for an e-mail slot. The record and the
notification are now different things, which is the durable half of this change.
- **Deliberate skips are not failures.** A disconnected or decommissioned drive has its own alert and
is excluded, because a nightly e-mail about an unplugged drive is one the operator learns to ignore.
- **A manual run always reports**, even if the nightly one already wrote that hour: the digest carries
a unique `run_id` that the hub's cooldown cannot collapse. Someone pressing the button is actively
trying to get a backup.
**THE PERIODIC SWEEP GETS A DIGEST TOO, and that is not symmetry for its own sake.** `GetFullStatus`
captures units outside any run. With the per-app event now record-only, a capture failure found
between runs would have been recorded and **never notified** — a new silence introduced while closing
one. So that path emits a digest as well, deliberately with **no `run_id`**, so the ordinary 1-hour
cooldown caps it exactly as before while the mail now lists *every* failing app instead of whichever
one happened to be first.
**A refusal is recorded ONCE, where the verdict is taken**, not at each of the three legs that consult
it — R-181's contract is that one verdict covers all three. Noting it per leg listed a single refused
app three times and produced counts like *"2 of 1 apps failed"*. **Found by the digest's own test, not
in review.**
**Why silence is safe, checked rather than assumed.** A digest is only safe if the absence of a mail
cannot mean "the run never finished". It cannot: the hub's daily deadline check raises
`expected_backup_missed` / `expected_dbdump_missed` (`hub/internal/monitor/deadline.go:396,417`) from
the box's **report freshness and stored events**, independently of any mail this box chooses to send.
**Tests: 7 new, plus 4 red-proofs demonstrated failing then restored.** One of them —
the `main.go` seam walk — **did not fail on its first attempt**, because the AST test walked the
backup package and not `main.go`; the test was fixed and the mutation re-run rather than the pass
being recorded.
### v0.193.1 — the refusal's size estimate is rendered in bytes, not `0.00 GiB` (2026-08-03, R-181 follow-on)
**Found by the live proof run for v0.193.0, not by review.** The refusal message printed the estimate
+1
View File
@@ -174,6 +174,7 @@
| `api.GracefulSelfRestart` | controller/internal/api/selfrestart.go | `(logger)` | Controller self-restart | Detached exit; bootstrap unit re-runs the image |
| `Settings.AddPendingEvent/DrainPendingEvents` | controller/internal/settings/settings.go | offline event queue | Events while hub unreachable | — |
| `Manager.SetUnitNotify` + `UnitSpace` (R-158/R-167, v0.191.0) | controller/internal/backup/recovery_unit.go | `(func(stack string, err error, *UnitSpace))` | THE per-app Tier-1 recovery-unit capture failure alert — fires PER APP from `captureAllRecoveryUnits`, loop continues | **OPERATOR-TIER** (`recovery_unit_capture_failed`, in the hub's `operatorOnlyEvents`). **NEVER route it to `backup_failed`** — that type is in `DefaultEnabledEvents` and carries Hungarian copy, so it emails the CUSTOMER about a failure they cannot act on (D-c; R-158's own proposal said `backup_failed` and D-c overrides it). `UnitSpace` is **nil when the target filesystem is unreadable** and renders as *"unavailable"*, never as zeros — "0 GB free" and "we could not look" are opposite diagnoses. No controller-side cooldown: the hub owns it |
| `Manager.beginRunSummary` / `noteFailure` / `noteAttempted` / `emitRunSummary` / `SetRunSummaryNotify` (R-182, v0.194.0) | controller/internal/backup/runsummary.go | `(kind, runID) func()` / `(app, leg, reason)` / `(RunSummary)` | **THE per-run operator digest.** One `backup_run_failures` event at the end of a run listing every failed app, its leg and its reason — emitted ONLY when something failed | **The RECORD and the NOTIFICATION are different things and must stay so.** The per-app `recovery_unit_capture_failed` event is the record (hub routes it *record-only*, stored + logged every time); this digest is the notification. Before R-182 one event was both, and did neither: nine arrived, two were mailed, seven vanished before `LogNotification`. **Lifetime is `admissionSet`'s exactly** — absent collector means "no run in flight", never a stale answer. **A refusal is noted ONCE, inside `admitApp` where the verdict is taken**, not at the three legs that consult it: R-181's one-verdict-covers-all-three contract makes per-leg noting produce "2 of 1 apps failed". **Deliberate skips (disconnected / decommissioned) must NEVER be noted** — they have their own alert and a nightly digest about an unplugged drive is an ignored digest. **A clean run emits NOTHING**; silence is safe only because the hub's deadline check (`monitor/deadline.go:396,417`) raises a missed backup from report freshness independently — if that is ever weakened this design loses its footing. **`run_id` is unique per real run** (so the hub's 1-h cooldown cannot collapse a manual run into the nightly one) and **deliberately EMPTY on the periodic refresh sweep**, which must stay under that cooldown or a polled status page becomes a mail flood |
| `Manager.admitApp` / `beginAdmissionRun` / `decideAdmission` / `estimatedWriteBytes` (R-181, v0.193.0) | controller/internal/backup/admission.go | `(stackName) bool` / `() func()` | **THE reserve gate. Call it before ANY per-app backup write** — one verdict per app per run, covering the DB dump, the volume dump and the unit capture (all three write under one per-app root) | **Decided LAZILY at the app's first write, never once at run start** — app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. **Never re-decided between an app's own legs**: that is exactly the split R-181 closed (bulk written, capture refused). **Reset per run** via the closer `beginAdmissionRun` returns. **Must sit ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided inside it has already bounced the app. Fires **exactly one** `unitNotify` per refused app per run. Nil admission set (periodic status refresh) → decides fresh, which is still once per app per sweep. Wiring pinned by an **AST walk** in `TestAdmission_IsWiredIntoEveryProductionWriteLeg`, not `strings.Contains` |
| `Manager.floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` / `floorReason` (R-165 B2 v0.192.0, size term R-181 v0.193.0) | controller/internal/backup/recovery_unit.go | `(*UnitSpace, estGiB float64) (*UnitSpace, floorReason)` | The pure two-question predicate behind `admitApp`: is the filesystem already below the reserve (`floorHeadroom`), and would THIS app's write take it below (`floorSize`)? | **Headroom is about the FILESYSTEM, never a per-unit cap** — a size cap is R-163 rebuilt inside one volume; the size term bounds the *delta*, not the unit. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. **`estGiB == 0` degrades to headroom-only on purpose** — refusing an app with no history makes the FIRST backup the one that can never happen. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk |
| `fillwatch.Watcher` (`New`/`SetNotify`/`Check`) (R-167, v0.191.0) | controller/internal/fillwatch/fillwatch.go | `(statePath, logger, targetsFn, usageFn)``Check() error` | THE customer fill warning — warns BEFORE a filesystem fills, per FILESYSTEM (never per app: one full disk holding ten apps would fire ten times) | Emits the **pre-existing** `disk_warning`/`disk_critical` pair, which was allowlisted + copy'd + default-enabled with **no producer in any repo** until now — do NOT mint a new type beside it. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range. **Edge-triggered on ESCALATION ONLY**, state persisted; de-escalation is silent and re-arms. Hysteresis dead zone between clear (75% / 7 GiB) and warn — pinned by `TestThresholdsKeepTheirHysteresisGap`. **A nil usage read is NEVER a warning** (§8.4). The hub has **no `customerMessages` entry** for either type on purpose — an entry would override the dynamic message and discard the drive label + free space |
+19
View File
@@ -744,6 +744,25 @@ func main() {
"Recovery unit capture FAILED for %q — the app has no fresh local (Tier-1) backup, and Tier-2/Tier-3 have nothing to copy. %s. Error: %v",
stackName, usage.String(), err), d)
})
// R-182: the ONE operator digest per backup run. The per-app events above are the RECORD
// (the hub routes them record-only); this is the NOTIFICATION, sent once at the end of a run
// and only when something actually failed. A clean run sends nothing — and that silence is
// safe because the hub's own daily deadline check raises expected_backup_missed from report
// freshness, independently of any mail this box chooses to send.
backupMgr.SetRunSummaryNotify(func(rs backup.RunSummary) {
d := notify.BackupRunFailuresDetails{
RunID: rs.RunID, RunKind: rs.RunKind,
Failed: rs.Failed, Attempted: rs.Attempted,
}
for _, a := range rs.Apps {
d.Apps = append(d.Apps, notify.RunFailureDetail{App: a.App, Leg: a.Leg, Reason: a.Reason})
}
if rs.Usage != nil {
d.TargetPath, d.UsedGB, d.AvailGB = rs.Usage.Path, rs.Usage.UsedGB, rs.Usage.AvailGB
d.TotalGB, d.UsedPercent, d.SpaceKnown = rs.Usage.TotalGB, rs.Usage.UsedPercent, true
}
notifier.NotifyBackupRunFailures(rs.Message, d)
})
// 3a: the pre-push enlargement gate blocked an app's userdata push (config+DB still saved). Edge-
// triggered by the engine (only NEW blocks notify), so the hub's per-event-type cooldown suffices —
// no controller-side timer (the hub owns cooldown).
+1
View File
@@ -1010,6 +1010,7 @@ func (r *Router) triggerBackup(w http.ResponseWriter, _ *http.Request) {
}
r.logger.Println("[INFO] [api] Manual app-data backup (DB dump) triggered")
r.backupMgr.MarkManualRun() // R-182: operator-triggered — its digest must not be collapsed into the nightly one
go r.backupMgr.RunDBDumps(context.Background())
writeJSON(w, http.StatusOK, apiResponse{OK: true, Message: "Mentés elindítva"})
+6
View File
@@ -132,6 +132,12 @@ func (m *Manager) admitApp(stackName string) bool {
if m.unitNotify != nil {
m.unitNotify(stackName, v.err, v.usage)
}
// R-182: the digest entry is recorded HERE, where the verdict is taken — once per app per run.
// Not at the three call sites that consult the memo: R-181's whole contract is that ONE verdict
// covers all three legs, so noting it per leg listed a single refused app three times and
// produced counts like "2 of 1 apps failed". The leg name says what actually happened, which is
// that nothing was attempted at all.
m.noteFailure(stackName, "whole app (refused before any write)", v.err.Error())
return false
}
+44 -2
View File
@@ -15,6 +15,8 @@ import (
"sort"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-181 — the reserve guards the write that fills the disk, and its promise is true.
@@ -33,6 +35,7 @@ import (
type admissionProvider struct {
stacks []string
volumes map[string][]string
hdd map[string]string // per-app drive path, for the drive-state skip tests
dir string
infoHits []string
stopped []string
@@ -47,7 +50,7 @@ func (p *admissionProvider) ListDeployedStacks() []StackSummary {
return out
}
func (p *admissionProvider) GetStackHDDMounts(string) []string { return nil }
func (p *admissionProvider) GetStackHDDPath(string) string { return "" }
func (p *admissionProvider) GetStackHDDPath(n string) string { return p.hdd[n] }
func (p *admissionProvider) GetImportRoot() string { return "" }
func (p *admissionProvider) GetDockerVolumes(name string) []string {
if p.volumes == nil {
@@ -88,7 +91,7 @@ func newAdmissionHarness(t *testing.T, stacks ...string) *admissionHarness {
t.Helper()
dir := t.TempDir()
h := &admissionHarness{
prov: &admissionProvider{stacks: stacks, dir: dir},
prov: &admissionProvider{stacks: stacks, dir: dir, hdd: map[string]string{}},
usage: map[string]*UnitSpace{},
dir: dir,
logs: &bytes.Buffer{},
@@ -116,6 +119,45 @@ func newAdmissionHarness(t *testing.T, stacks ...string) *admissionHarness {
return h
}
// markDisconnected / markDecommissioned put a real settings row behind the drive-state skips, so
// Scenario F exercises the production guards rather than a stub of them.
func (h *admissionHarness) markDisconnected(app string) {
h.driveState(app, true, false)
}
func (h *admissionHarness) markDecommissioned(app string) {
h.driveState(app, false, true)
}
func (h *admissionHarness) driveState(app string, disconnected, decommissioned bool) {
if h.m.settings == nil {
sett, err := settings.Load(filepath.Join(h.dir, "settings.json"), log.New(io.Discard, "", 0))
if err != nil {
panic(err)
}
h.m.settings = sett
}
// Each such app gets its OWN drive path, or marking one would skip them all.
p := filepath.Join(h.dir, "drives", app)
if err := os.MkdirAll(p, 0o755); err != nil {
panic(err)
}
h.prov.hdd[app] = p
if err := h.m.settings.AddStoragePath(settings.StoragePath{Path: p, Label: app}); err != nil {
panic(err)
}
if disconnected {
if err := h.m.settings.SetDisconnected(p, true, nil); err != nil {
panic(err)
}
}
if decommissioned {
if err := h.m.settings.SetDecommissioned(p, ""); err != nil {
panic(err)
}
}
}
func (h *admissionHarness) nsRoot() string { return filepath.Join(h.dir, "felhom-data") }
// setSpace states the filesystem's occupancy as a test INPUT — the whole point of the unitSpaceFn
+36 -1
View File
@@ -9,6 +9,7 @@ import (
"path/filepath"
"strings"
"sync"
"sync/atomic"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
@@ -64,6 +65,16 @@ type Manager struct {
admissionMu sync.Mutex
admission *admissionSet
// summary (R-182) is the per-RUN digest collector, guarded by summaryMu. Same lifetime as
// `admission` and for the same reason: an absent collector means "no run in flight", never a
// stale answer from last night. runSummaryNotify is the operator digest seam, wired in main.go.
summaryMu sync.Mutex
summary *runSummary
runSummaryNotify func(RunSummary)
// manualRun tags the NEXT run as operator-triggered (cleared as the run starts), so the digest
// can say which kind it was and the hub can decline to collapse a manual run into a nightly one.
manualRun atomic.Bool
// appStop (R-166) is the crash marker for operations that stop an app, work on its data, and
// start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power
// cut in that window leaves a durable record that Recover honours at the next startup. Built in
@@ -426,6 +437,14 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
// question with last night's disk.
defer m.beginAdmissionRun()()
// R-182: the digest scope has the same lifetime. `emitRunSummary` runs BEFORE the closer (defers
// unwind last-in-first-out), so the summary is still populated when it is sent, and it sends
// nothing at all when the run was clean.
kind := m.runKindFor()
m.manualRun.Store(false) // tags exactly ONE run; a stale flag would mislabel every later nightly
defer m.beginRunSummary(kind, newRunID())()
defer m.emitRunSummary()
dbs, err := DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames())
if err != nil {
m.logger.Printf("[ERROR] [backup] Database discovery failed: %v", err)
@@ -465,6 +484,7 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
// where an app's verdict is taken, because the DB leg runs first; the volume leg and the
// capture then read the same memo. SKIP, not FAIL — a deliberate hold is not a broken dump,
// and the operator alert (fired once, inside admitApp) is the signal that it happened.
m.noteAttempted(db.StackName)
if !m.admitApp(db.StackName) {
summary = append(summary, fmt.Sprintf("SKIP %s (reserve — app backup refused)", db.ContainerName))
continue
@@ -478,6 +498,7 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
if result.Error != nil {
allOK = false
summary = append(summary, fmt.Sprintf("FAIL %s: %v", result.DB.ContainerName, result.Error))
m.noteFailure(db.StackName, "database dump", result.Error.Error())
m.logger.Printf("[ERROR] [backup] DB dump failed for %s: %v", result.DB.ContainerName, result.Error)
} else {
totalSize += result.Size
@@ -608,6 +629,7 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
// R-181: the reserve, ahead of DumpAppVolumesSafe so a refused app is NOT stopped. For an app
// that already has a DB this is a memo lookup taken before its DB dump; for a volume-only app
// this is where its verdict is taken, still before its first byte.
m.noteAttempted(stack.Name)
if !m.admitApp(stack.Name) {
summary = append(summary, fmt.Sprintf("SKIP %s volumes (reserve — app backup refused)", stack.Name))
continue
@@ -616,6 +638,7 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
if err := dump(stack.Name); err != nil {
allOK = false
summary = append(summary, fmt.Sprintf("FAIL %s volumes: %v", stack.Name, err))
m.noteFailure(stack.Name, "volume dump", err.Error())
m.logger.Printf("[ERROR] [backup] Volume dump failed for %s: %v", stack.Name, err)
continue
}
@@ -1001,7 +1024,19 @@ func (m *Manager) RefreshCache(nextDBDump time.Time) {
// Phase 2: keep each app's recovery unit current with its definition. Idempotent
// (checksum-skip), so this periodic refresh only writes when the config actually changed,
// and ensures units exist shortly after startup without waiting for the daily DB dump.
m.captureAllRecoveryUnits()
//
// R-182: this sweep gets its OWN digest scope. It has to, and the reason is the whole
// balance of this change. The per-app event is now record-only, so without a digest here a
// capture failure detected between runs would be recorded and NEVER notified — a new
// silence introduced while closing one. But this path can fire on every status poll, so its
// digest deliberately carries NO run id: the hub's ordinary 1-hour operator cooldown then
// applies, which caps it at one mail an hour exactly as before, while the mail now lists
// EVERY failing app instead of whichever one happened to be first.
func() {
defer m.beginRunSummary(runKindRefresh, "")()
defer m.emitRunSummary()
m.captureAllRecoveryUnits()
}()
}
// Fill in dynamic fields under lock.
@@ -42,7 +42,7 @@ func (p *floorProvider) GetImportRoot() string { return "" }
func (p *floorProvider) GetDockerVolumes(string) []string { return nil }
func (p *floorProvider) StopStack(string) error { return nil }
func (p *floorProvider) StartStack(string) error { return nil }
func (p *floorProvider) RefreshAndIsRunning(string) bool { return true }
func (p *floorProvider) RefreshAndIsRunning(string) bool { return true }
func (p *floorProvider) GetStackRecoveryInfo(name string) (RecoveryInfo, bool) {
p.infoHits = append(p.infoHits, name)
return RecoveryInfo{StackDir: filepath.Join(p.dir, "stacks", name)}, true
@@ -351,11 +351,13 @@ func (m *Manager) captureAllRecoveryUnits() {
if m.settings != nil && (m.settings.IsDisconnected(drivePath) || m.settings.IsDecommissioned(drivePath)) {
continue // drive not writable — skip, the existing unit stays as-is
}
m.noteAttempted(stack.Name)
// The reserve, checked BEFORE anything is written. Per app, and the loop continues.
if !m.admitApp(stack.Name) {
continue
}
if err := m.CaptureRecoveryUnit(stack.Name); err != nil {
m.noteFailure(stack.Name, "recovery-unit capture", err.Error())
m.logger.Printf("[WARN] [backup] Recovery unit capture failed for %s: %v", stack.Name, err)
// R-158: per app, and the loop CONTINUES — one app's failure must not silence the
// others, and it must not abort their captures either. The space figures are read at
+270
View File
@@ -0,0 +1,270 @@
package backup
import (
"fmt"
"sort"
"strconv"
"strings"
"sync"
"time"
)
// ── The backup run digest (R-182) ────────────────────────────────────────────────────────────────
//
// WHAT WAS WRONG, MEASURED. On 2026-08-03 nine per-app `recovery_unit_capture_failed` events reached
// the hub and TWO operator e-mails went out. The hub's operator cooldown key is
// `customerID + ":" + eventType + tier-suffix`, and that event carries `app` but no `tier`, so the
// key held **no app identifier**: the first refused app took the hour's slot and every other app's
// failure was discarded — *before* anything was written down, so it left no row on any channel and
// could not be found afterwards.
//
// WHY NOT JUST PUT THE APP IN THE KEY. That was the obvious fix and the operator ruled against it:
// on a full disk it produces one e-mail per app, which is the volume problem wearing the correctness
// problem's clothes. **One digest per run instead**, listing every failure — and separately, every
// failure recorded when it happens.
//
// THE RECORD AND THE NOTIFICATION ARE DIFFERENT THINGS, and that separation is the durable part:
//
// - the RECORD is the per-app `recovery_unit_capture_failed` event, emitted unconditionally, now
// routed record-only by the hub so it never competes for an e-mail slot;
// - the NOTIFICATION is this digest, emitted once per run and only when something failed.
//
// WHY A DIGEST IS SAFE HERE — the one thing that could have made it dangerous. A digest introduces a
// silent-failure path if the ABSENCE of an e-mail could mean "the run never finished". It cannot:
// the hub's daily deadline check raises `expected_backup_missed` / `expected_dbdump_missed`
// (`hub/internal/monitor/deadline.go:396,417`) from the box's REPORT freshness and its stored
// events, entirely independently of any mail the controller chooses to send. Silence therefore still
// means "the run finished and found nothing wrong". If that check is ever weakened, this design
// loses its footing — which is why it is named here and not only in a report.
// runKind distinguishes the paths that can produce a digest. It is carried in the details and shown
// in the subject, because "the nightly run failed" and "the run I just triggered failed" are read
// differently at 07:00.
const (
runKindNightly = "nightly"
runKindManual = "manual"
// runKindRefresh is the PERIODIC status sweep, which captures units outside any backup run.
//
// IT DELIBERATELY CARRIES NO run_id. A run digest gets a unique run id so the hub's 1-hour
// cooldown can never collapse two real runs (the operator ruled on that explicitly). The sweep
// is the opposite case: it can fire every time the status page is polled, so it MUST fall under
// the ordinary cooldown or a full disk becomes a mail flood — the exact failure this whole
// change exists to avoid, arriving from the other direction.
//
// Without this path the sweep's failures would be recorded and never notified, because the
// per-app event is now record-only — a NEW silence introduced while fixing a silence. This is
// what stops that.
runKindRefresh = "refresh"
)
// runFailure is one app's failed or refused leg within a run.
type runFailure struct {
App string
Leg string
Reason string
}
// runSummary is the per-RUN collector. Same lifetime as `admissionSet` and for the same reason: an
// absent collector must mean "no run in flight", never "a stale answer from last night".
type runSummary struct {
mu sync.Mutex
kind string
runID string
attempted map[string]bool // apps this run actually tried to back up
failures []runFailure
}
// beginRunSummary opens the per-run digest scope and returns the closer, mirroring
// beginAdmissionRun. A second call while one is live REPLACES it and the closer restores the
// previous, so nesting cannot silently drop a caller's scope.
func (m *Manager) beginRunSummary(kind, runID string) func() {
m.summaryMu.Lock()
prev := m.summary
m.summary = &runSummary{kind: kind, runID: runID, attempted: map[string]bool{}}
m.summaryMu.Unlock()
return func() {
m.summaryMu.Lock()
m.summary = prev
m.summaryMu.Unlock()
}
}
// noteAttempted records that this run tried to back up an app.
//
// THE DENOMINATOR IS NOT DECORATION. "3 of 4 apps failed" is a catastrophe and "3 of 40" is a bad
// night, and a list of names cannot tell them apart — the operator's first decision, get up now or
// look after coffee, is made from exactly this ratio.
func (m *Manager) noteAttempted(app string) {
m.summaryMu.Lock()
defer m.summaryMu.Unlock()
if m.summary != nil {
m.summary.mu.Lock()
m.summary.attempted[app] = true
m.summary.mu.Unlock()
}
}
// noteFailure records one failed or refused leg. Deliberate skips must NOT come through here —
// §8.1: a drive that is unplugged or decommissioned has its own alert, and putting it in the digest
// turns a nightly e-mail into one the operator learns to ignore.
func (m *Manager) noteFailure(app, leg, reason string) {
m.summaryMu.Lock()
defer m.summaryMu.Unlock()
if m.summary == nil {
return // no run in flight — nothing to summarise
}
m.summary.mu.Lock()
m.summary.attempted[app] = true
m.summary.failures = append(m.summary.failures, runFailure{App: app, Leg: leg, Reason: reason})
m.summary.mu.Unlock()
}
// emitRunSummary sends the digest, if and only if something failed.
//
// A CLEAN RUN EMITS NOTHING — not an empty digest. A nightly "0 failures" mail is an unread mail
// within a week, and it would also destroy the property the whole design rests on: that silence
// means the run finished and found nothing wrong.
func (m *Manager) emitRunSummary() {
m.summaryMu.Lock()
s := m.summary
m.summaryMu.Unlock()
if s == nil {
return
}
s.mu.Lock()
failures := append([]runFailure(nil), s.failures...)
attempted := len(s.attempted)
kind, runID := s.kind, s.runID
s.mu.Unlock()
if len(failures) == 0 {
return
}
// Stable order so two identical runs render identically and a diff of two mails is meaningful.
sort.Slice(failures, func(i, j int) bool {
if failures[i].App != failures[j].App {
return failures[i].App < failures[j].App
}
return failures[i].Leg < failures[j].Leg
})
if m.runSummaryNotify == nil {
// Nil-safe, but say so: an unwired seam here means the digest exists and reaches nobody,
// which is the built-but-never-wired failure this project has shipped four times.
m.logger.Printf("[WARN] [backup] %d app(s) failed in this %s run but no run-summary notifier is wired — "+
"the failures are recorded per app and NOT summarised to the operator", len(failures), kind)
return
}
apps := make([]RunFailureDetail, 0, len(failures))
names := make([]string, 0, len(failures))
for _, f := range failures {
apps = append(apps, RunFailureDetail{App: f.App, Leg: f.Leg, Reason: f.Reason})
names = append(names, f.App)
}
msg := fmt.Sprintf("%d of %d apps failed to back up in this %s run: %s",
len(failures), attempted, kind, strings.Join(dedupeStable(names), ", "))
m.logger.Printf("[INFO] [backup] Run summary: %d of %d apps failed (%s run) — notifying the operator once",
len(failures), attempted, kind)
m.runSummaryNotify(RunSummary{
RunID: runID,
RunKind: kind,
Failed: len(failures),
Attempted: attempted,
Apps: apps,
Usage: m.summaryUsage(),
Message: msg,
})
}
// summaryUsage reads the target filesystem once for the digest. It uses the same seam the reserve
// does, so a test states occupancy as an input rather than manufacturing it. Nil when unreadable —
// which the hub renders as "unavailable", never as zeros.
func (m *Manager) summaryUsage() *UnitSpace {
if m.stackProvider == nil {
return nil
}
for _, st := range m.stackProvider.ListDeployedStacks() {
if u := m.readUnitSpace(st.Name); u != nil {
return u
}
}
return nil
}
func dedupeStable(in []string) []string {
seen := map[string]bool{}
out := in[:0:0]
for _, s := range in {
if !seen[s] {
seen[s] = true
out = append(out, s)
}
}
return out
}
// RunFailureDetail is one app's failure as the hub receives it.
type RunFailureDetail struct {
App string `json:"app"`
Leg string `json:"leg"`
Reason string `json:"reason"`
}
// RunSummary is the digest payload handed to the notifier seam.
type RunSummary struct {
RunID string
RunKind string
Failed int
Attempted int
Apps []RunFailureDetail
Usage *UnitSpace
Message string
}
// SetRunSummaryNotify wires the per-run digest. INIT-ONLY — call once at startup in main.go,
// alongside SetUnitNotify. Nil-safe, but an unwired seam is logged loudly rather than being silently
// the old behaviour.
func (m *Manager) SetRunSummaryNotify(fn func(RunSummary)) {
m.runSummaryNotify = fn
}
// admissionReason returns the human reason this run refused an app, taken from the verdict the
// reserve already recorded. Reused rather than re-derived: the verdict carries the bound term, the
// estimate and the disk figures, and re-deriving them here would be a second source of truth for a
// sentence the operator reads.
func (m *Manager) admissionReason(stackName string) string {
m.admissionMu.Lock()
defer m.admissionMu.Unlock()
if set := m.admission; set != nil {
if v, ok := set.v[stackName]; ok && v.err != nil {
return v.err.Error()
}
}
return "refused by the capture reserve"
}
// runKindFor reports whether this run was the scheduled one or one a person triggered.
//
// The distinction is the operator's ruling: someone pressing the button is actively trying to get a
// backup, so a manual run must report even if the nightly one already wrote this hour. It is carried
// into the subject line because "the nightly run failed" and "the run I just asked for failed" are
// acted on differently.
func (m *Manager) runKindFor() string {
if m.manualRun.Load() {
return runKindManual
}
return runKindNightly
}
// MarkManualRun tags the NEXT backup run as operator-triggered. Called by the API/debug handlers
// that expose a "run backup now" control; the scheduled path leaves it alone.
func (m *Manager) MarkManualRun() { m.manualRun.Store(true) }
// newRunID returns a per-run identifier. It only has to be distinct between two runs on one box
// within the hub's 1-hour cooldown window, which a nanosecond clock reading satisfies; it is never
// persisted, compared across boxes, or used as a security value.
func newRunID() string {
return "run-" + strconv.FormatInt(time.Now().UnixNano(), 36)
}
@@ -0,0 +1,270 @@
package backup
import (
"go/ast"
"go/parser"
"go/token"
"strings"
"testing"
)
// R-182 — one digest per run, listing every failure, and nothing in it that is not a failure.
//
// The assertions here are the CONTENT of the digest, not that a function was called: the defect this
// replaces was one where the machinery ran correctly and the operator was told about one app out of
// nine.
// digestOf runs the two legs that can run without Docker inside one digest scope and returns the
// summary that would have been sent (nil when none). It mirrors what runDBDumpsInternal does; the
// DB leg's wiring is pinned structurally by TestRunSummary_IsWiredIntoTheProductionPath, because
// DiscoverDatabases shells out to `docker`.
func (h *admissionHarness) digestOf(kind string) *RunSummary {
var got *RunSummary
h.m.SetRunSummaryNotify(func(rs RunSummary) { got = &rs })
doneAdm := h.m.beginAdmissionRun()
doneSum := h.m.beginRunSummary(kind, "run-test")
h.m.runVolumeDumps()
h.m.captureAllRecoveryUnits()
h.m.emitRunSummary()
doneSum()
doneAdm()
return got
}
// ── Scenario A — several failures, ONE digest, all of them in it ─────────────────────────────────
func TestRunSummary_ListsEveryFailedApp(t *testing.T) {
apps := []string{"opengist", "privatebin", "immich", "homebox", "nextcloud"}
h := newAdmissionHarness(t, apps...)
for _, a := range apps {
h.setSpace(a, 99, 0.1, 70) // every app refused by the reserve
}
rs := h.digestOf(runKindNightly)
if rs == nil {
t.Fatal("no digest was emitted although five apps failed — this is the 2026-08-03 defect: " +
"nine failures arrived and two e-mails went out")
}
if rs.Failed != len(apps) {
t.Fatalf("digest reports %d failures, want %d — the others were silently discarded, which is "+
"exactly what the per-app path did", rs.Failed, len(apps))
}
seen := map[string]bool{}
for _, a := range rs.Apps {
seen[a.App] = true
if a.Leg == "" {
t.Errorf("%s has no leg name — the operator cannot tell a failed database dump from a "+
"failed volume dump, and they are different problems", a.App)
}
if !strings.Contains(a.Reason, "reserve") {
t.Errorf("%s reason %q does not say why", a.App, a.Reason)
}
}
for _, a := range apps {
if !seen[a] {
t.Errorf("%s is missing from the digest", a)
}
}
// The denominator: "5 of 5" and "5 of 40" are different nights.
if rs.Attempted < len(apps) {
t.Fatalf("attempted=%d, want >= %d — without the denominator a catastrophe and a bad night "+
"read identically", rs.Attempted, len(apps))
}
if rs.Usage == nil {
t.Fatal("the digest carries no filesystem figures — 'one broken app' and 'a full disk' must " +
"be distinguishable before the reasons are read")
}
if !strings.Contains(rs.Message, "of") || !strings.Contains(rs.Message, "nightly") {
t.Fatalf("summary message is not readable as a sentence: %q", rs.Message)
}
}
// ── Scenario B — a clean run is silent ───────────────────────────────────────────────────────────
func TestRunSummary_CleanRunEmitsNothing(t *testing.T) {
h := newAdmissionHarness(t, "opengist", "privatebin")
h.setSpace("opengist", 20, 55, 70)
h.setSpace("privatebin", 20, 55, 70)
if rs := h.digestOf(runKindNightly); rs != nil {
t.Fatalf("a clean run emitted a digest (%+v) — an empty nightly mail is an unread mail within "+
"a week, and it would destroy the property this design rests on: that silence means the "+
"run finished and found nothing wrong", rs)
}
}
// ── Scenario F — deliberate skips are NOT failures ───────────────────────────────────────────────
// A drive that is unplugged or decommissioned has its own alert. Putting it in the digest produces a
// nightly e-mail on a box with an unplugged drive, which trains the operator to ignore digests — and
// an ignored digest is the same outcome as no digest.
func TestRunSummary_DeliberateSkipsAreNotFailures(t *testing.T) {
h := newAdmissionHarness(t, "gone", "retired", "healthy")
for _, a := range []string{"gone", "retired", "healthy"} {
h.setSpace(a, 20, 55, 70) // ample room — nothing may be refused for headroom
}
h.markDisconnected("gone")
h.markDecommissioned("retired")
rs := h.digestOf(runKindNightly)
if rs != nil {
t.Fatalf("a run whose only 'issues' were a disconnected drive and a decommissioned one "+
"emitted a digest: %+v — those have their own alerts", rs)
}
}
// ── The collector's lifetime — the admission.go rule, restated ───────────────────────────────────
// An absent collector must mean "no run in flight", never a stale answer from last night.
func TestRunSummary_ResetsBetweenRuns(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
h.setSpace("opengist", 99, 0.1, 70)
if rs := h.digestOf(runKindNightly); rs == nil || rs.Failed != 1 {
t.Fatalf("run 1: want 1 failure, got %+v", rs)
}
h.setSpace("opengist", 20, 55, 70) // space freed between runs
if rs := h.digestOf(runKindNightly); rs != nil {
t.Fatalf("run 2 reported %+v — the previous run's failures were carried over, so freeing "+
"space could never take effect", rs)
}
}
// A failure noted with NO run in flight must not panic and must not accumulate anywhere.
func TestRunSummary_NoteOutsideARunIsInert(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
h.m.noteFailure("opengist", "volume dump", "boom") // no scope open
h.m.noteAttempted("opengist")
h.m.emitRunSummary() // must be a no-op, not a nil dereference
}
// ── The refresh sweep carries NO run id, deliberately ────────────────────────────────────────────
// The run digest gets a unique id so the hub's 1-hour cooldown can never collapse two real runs. The
// periodic sweep is the opposite case — it can fire on every status poll — so it must fall UNDER the
// cooldown. Getting this backwards turns a full disk into a mail flood, which is the same failure as
// the one being fixed, arriving from the other side.
func TestRunSummary_RefreshSweepHasNoRunID(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
h.setSpace("opengist", 99, 0.1, 70)
var got *RunSummary
h.m.SetRunSummaryNotify(func(rs RunSummary) { got = &rs })
func() {
defer h.m.beginAdmissionRun()()
defer h.m.beginRunSummary(runKindRefresh, "")()
defer h.m.emitRunSummary()
h.m.captureAllRecoveryUnits()
}()
if got == nil {
t.Fatal("the periodic sweep emitted no digest — with the per-app event now record-only, a " +
"capture failure found between runs would be recorded and NEVER notified, which is a " +
"new silence introduced while closing one")
}
if got.RunID != "" {
t.Fatalf("the refresh sweep carries run_id=%q — it must be EMPTY so the hub's ordinary "+
"1-hour cooldown caps it, or a polled status page becomes a mail flood", got.RunID)
}
if got.RunKind != runKindRefresh {
t.Fatalf("run kind = %q, want %q", got.RunKind, runKindRefresh)
}
}
// ── The seam is WIRED — walked as an AST, not grepped ────────────────────────────────────────────
// Four mechanisms in this project have been built and left disconnected. `strings.Contains` cannot
// tell a live call from a commented-out one, so this parses.
func TestRunSummary_IsWiredIntoTheProductionPath(t *testing.T) {
calls := callsByFunc(t, "backup.go")
if !hasStr(calls["runDBDumpsInternal"], "beginRunSummary") {
t.Fatal("runDBDumpsInternal does not open a run-summary scope — every failure would be " +
"recorded per app and none of them summarised, which is the pre-R-182 behaviour")
}
if !hasStr(calls["runDBDumpsInternal"], "emitRunSummary") {
t.Fatal("runDBDumpsInternal never emits the summary — the collector fills and is discarded")
}
// Both dump legs and the capture leg must feed it, or a whole class of failure is invisible.
for fn, where := range map[string]string{
"runDBDumpsInternal": "the database leg",
"runVolumeDumps": "the volume leg",
} {
if !hasStr(calls[fn], "noteFailure") {
t.Fatalf("%s (%s) never calls noteFailure — its failures cannot reach the digest", fn, where)
}
}
capCalls := callsByFunc(t, "recovery_unit.go")
if !hasStr(capCalls["captureAllRecoveryUnits"], "noteFailure") {
t.Fatal("the capture leg never calls noteFailure")
}
// AND THE SEAM ITSELF must be wired in main.go. Without this assertion the collector fills, the
// digest is built, and `runSummaryNotify` is nil — so nothing is ever sent and every test above
// still passes. That is the built-but-never-wired shape exactly, and the first version of this
// test missed it: commenting the wiring out left the suite green.
mainCalls := callsByFunc(t, "../../cmd/controller/main.go")
wired := false
for _, calls := range mainCalls {
if hasStr(calls, "SetRunSummaryNotify") {
wired = true
}
}
if !wired {
t.Fatal("main.go never calls SetRunSummaryNotify — the digest is assembled and handed to a " +
"nil seam, so no operator mail is ever sent. Note this is an AST walk: a commented-out " +
"call still CONTAINS the string, which is why strings.Contains cannot be used here")
}
// The manual paths must tag the run, or Scenario E cannot hold.
for _, f := range []string{"../web/handler_debug.go", "../api/router.go"} {
src := parseFile(t, f)
found := false
ast.Inspect(src, func(n ast.Node) bool {
if ce, ok := n.(*ast.CallExpr); ok {
if sel, ok := ce.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "MarkManualRun" {
found = true
}
}
return true
})
if !found {
t.Fatalf("%s never calls MarkManualRun — an operator-triggered run would be labelled "+
"nightly and could be collapsed into it", f)
}
}
}
func parseFile(t *testing.T, path string) *ast.File {
t.Helper()
f, err := parser.ParseFile(token.NewFileSet(), path, nil, 0) // comments dropped
if err != nil {
t.Fatal(err)
}
return f
}
// callsByFunc maps each top-level func to the names it calls, comments excluded.
func callsByFunc(t *testing.T, path string) map[string][]string {
t.Helper()
file := parseFile(t, path)
out := map[string][]string{}
var cur string
ast.Inspect(file, func(n ast.Node) bool {
switch v := n.(type) {
case *ast.FuncDecl:
cur = v.Name.Name
case *ast.CallExpr:
name := ""
switch fn := v.Fun.(type) {
case *ast.Ident:
name = fn.Name
case *ast.SelectorExpr:
name = fn.Sel.Name
}
if name != "" && cur != "" {
out[cur] = append(out[cur], name)
}
}
return true
})
return out
}
+38
View File
@@ -330,6 +330,44 @@ func (n *Notifier) NotifyRecoveryUnitCaptureFailed(message string, d RecoveryUni
n.PushEvent("recovery_unit_capture_failed", "error", message, d)
}
// RunFailureDetail is one app's failed leg inside a backup run digest.
type RunFailureDetail struct {
App string `json:"app"`
Leg string `json:"leg"`
Reason string `json:"reason"`
}
// BackupRunFailuresDetails is the per-RUN digest payload (R-182). App NAMES, leg names, reasons and
// byte figures only — never an env value (§9.4).
type BackupRunFailuresDetails struct {
// RunID makes the hub's 1-hour operator cooldown unable to collapse two real runs into one
// e-mail. EMPTY on the periodic refresh sweep, deliberately: that path can fire on every status
// poll, so it must fall under the ordinary cooldown instead.
RunID string `json:"run_id,omitempty"`
RunKind string `json:"run_kind"`
Failed int `json:"failed"`
Attempted int `json:"attempted"`
TargetPath string `json:"target_path,omitempty"`
UsedGB float64 `json:"used_gb,omitempty"`
AvailGB float64 `json:"avail_gb,omitempty"`
TotalGB float64 `json:"total_gb,omitempty"`
UsedPercent float64 `json:"used_percent,omitempty"`
SpaceKnown bool `json:"space_known"`
Apps []RunFailureDetail `json:"apps"`
}
// NotifyBackupRunFailures sends the ONE operator digest for a backup run in which something failed
// (R-182). It is the NOTIFICATION; the per-app `recovery_unit_capture_failed` events are the RECORD,
// and the hub routes those record-only so they never compete for an e-mail slot.
//
// OPERATOR-TIER, and for the same reason as its per-app sibling: a customer can act on a full disk
// (that is the fill warning, which fires first and IS customer-facing) but not on a list of which
// apps' backups failed and why. `notify.operatorOnlyEvents` in the hub enforces that — NOT the
// absence of a customerMessages entry, which is a fallback rather than a block (v0.78.0).
func (n *Notifier) NotifyBackupRunFailures(message string, d BackupRunFailuresDetails) {
n.PushEvent("backup_run_failures", "error", message, d)
}
// NotifyOffboxEnlargeBlocked sends a WARNING (not a failure) when an app's enlarged offsite push was
// refused by the pre-push quota gate — its config+DB were still saved. Customer-facing (Hungarian
// body). NOTE: the event type "offbox_enlarge_blocked" must be added to the hub's allowedEventTypes +
+1
View File
@@ -379,6 +379,7 @@ func (s *Server) debugTriggerDBDump(w http.ResponseWriter, r *http.Request) {
writeDebugJSON(w, http.StatusBadRequest, false, "Backup manager nincs konfigurálva", nil)
return
}
s.backupMgr.MarkManualRun() // R-182: a person pressed this, so its digest must not be collapsed
go func() {
if err := s.backupMgr.RunDBDumps(context.Background()); err != nil {
s.logger.Printf("[WARN] Debug DB dump failed: %v", err)