v0.194.0 — one operator email per backup run, and nothing dropped without a trace (R-182)

MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed
events reached the hub and TWO operator emails went out. The hub's operator
cooldown key is customerID:eventType(+tier) and that event carries `app` but no
`tier`, so the key held no app identifier: the first refused app took the hour's
slot and every other app's failure was discarded BEFORE anything was written
down, leaving no row on any channel.

The obvious fix — put `app` in the key — was ruled against: on a full disk it
produces one email per app, the volume problem wearing the correctness problem's
clothes.

internal/backup/runsummary.go: a per-run collector with exactly admissionSet's
lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the
end and only when something failed. A clean run emits nothing.

The per-app event stays and becomes the RECORD — the hub routes it record-only,
stored and logged every time, never competing for an email slot. The record and
the notification are now different things.

Deliberate skips (disconnected, decommissioned) are excluded: they have their
own alert, and a nightly email about an unplugged drive is one the operator
learns to ignore.

A manual run always reports: the digest carries a unique run_id the cooldown
cannot collapse. Someone pressing the button is actively trying to get a backup.

THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a
capture failure found between runs would be recorded and never notified — a new
silence introduced while closing one. That path emits a digest with NO run_id,
so the ordinary 1-hour cooldown caps it exactly as before while the mail now
lists every failing app instead of whichever was first.

A refusal is recorded ONCE, where the verdict is taken, not at the three legs
that consult it — R-181's contract is one verdict per app per run. Noting it per
leg listed one refused app three times and produced "2 of 1 apps failed". Found
by the digest's own test, not in review.

Silence is safe because the hub's deadline check raises expected_backup_missed
from report freshness, independently of any mail this box sends
(monitor/deadline.go:396,417). Confirmed, not assumed.

7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first
attempt — the AST test walked the backup package and not main.go; the test was
fixed and the mutation re-run rather than the pass recorded.

controller/internal/backup/backup.go

@@ -9,6 +9,7 @@ import (
 	"path/filepath"
 	"strings"
 	"sync"
+	"sync/atomic"
 	"time"
 
 	"gitea.dooplex.hu/admin/felhom-controller/internal/config"
@@ -64,6 +65,16 @@ type Manager struct {
 	admissionMu sync.Mutex
 	admission   *admissionSet
 
+	// summary (R-182) is the per-RUN digest collector, guarded by summaryMu. Same lifetime as
+	// `admission` and for the same reason: an absent collector means "no run in flight", never a
+	// stale answer from last night. runSummaryNotify is the operator digest seam, wired in main.go.
+	summaryMu        sync.Mutex
+	summary          *runSummary
+	runSummaryNotify func(RunSummary)
+	// manualRun tags the NEXT run as operator-triggered (cleared as the run starts), so the digest
+	// can say which kind it was and the hub can decline to collapse a manual run into a nightly one.
+	manualRun atomic.Bool
+
 	// appStop (R-166) is the crash marker for operations that stop an app, work on its data, and
 	// start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power
 	// cut in that window leaves a durable record that Recover honours at the next startup. Built in
@@ -426,6 +437,14 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
 	// question with last night's disk.
 	defer m.beginAdmissionRun()()
 
+	// R-182: the digest scope has the same lifetime. `emitRunSummary` runs BEFORE the closer (defers
+	// unwind last-in-first-out), so the summary is still populated when it is sent, and it sends
+	// nothing at all when the run was clean.
+	kind := m.runKindFor()
+	m.manualRun.Store(false) // tags exactly ONE run; a stale flag would mislabel every later nightly
+	defer m.beginRunSummary(kind, newRunID())()
+	defer m.emitRunSummary()
+
 	dbs, err := DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames())
 	if err != nil {
 		m.logger.Printf("[ERROR] [backup] Database discovery failed: %v", err)
@@ -465,6 +484,7 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
 		// where an app's verdict is taken, because the DB leg runs first; the volume leg and the
 		// capture then read the same memo. SKIP, not FAIL — a deliberate hold is not a broken dump,
 		// and the operator alert (fired once, inside admitApp) is the signal that it happened.
+		m.noteAttempted(db.StackName)
 		if !m.admitApp(db.StackName) {
 			summary = append(summary, fmt.Sprintf("SKIP %s (reserve — app backup refused)", db.ContainerName))
 			continue
@@ -478,6 +498,7 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
 		if result.Error != nil {
 			allOK = false
 			summary = append(summary, fmt.Sprintf("FAIL %s: %v", result.DB.ContainerName, result.Error))
+			m.noteFailure(db.StackName, "database dump", result.Error.Error())
 			m.logger.Printf("[ERROR] [backup] DB dump failed for %s: %v", result.DB.ContainerName, result.Error)
 		} else {
 			totalSize += result.Size
@@ -608,6 +629,7 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
 		// R-181: the reserve, ahead of DumpAppVolumesSafe so a refused app is NOT stopped. For an app
 		// that already has a DB this is a memo lookup taken before its DB dump; for a volume-only app
 		// this is where its verdict is taken, still before its first byte.
+		m.noteAttempted(stack.Name)
 		if !m.admitApp(stack.Name) {
 			summary = append(summary, fmt.Sprintf("SKIP %s volumes (reserve — app backup refused)", stack.Name))
 			continue
@@ -616,6 +638,7 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
 		if err := dump(stack.Name); err != nil {
 			allOK = false
 			summary = append(summary, fmt.Sprintf("FAIL %s volumes: %v", stack.Name, err))
+			m.noteFailure(stack.Name, "volume dump", err.Error())
 			m.logger.Printf("[ERROR] [backup] Volume dump failed for %s: %v", stack.Name, err)
 			continue
 		}
@@ -1001,7 +1024,19 @@ func (m *Manager) RefreshCache(nextDBDump time.Time) {
 		// Phase 2: keep each app's recovery unit current with its definition. Idempotent
 		// (checksum-skip), so this periodic refresh only writes when the config actually changed,
 		// and ensures units exist shortly after startup without waiting for the daily DB dump.
-		m.captureAllRecoveryUnits()
+		//
+		// R-182: this sweep gets its OWN digest scope. It has to, and the reason is the whole
+		// balance of this change. The per-app event is now record-only, so without a digest here a
+		// capture failure detected between runs would be recorded and NEVER notified — a new
+		// silence introduced while closing one. But this path can fire on every status poll, so its
+		// digest deliberately carries NO run id: the hub's ordinary 1-hour operator cooldown then
+		// applies, which caps it at one mail an hour exactly as before, while the mail now lists
+		// EVERY failing app instead of whichever one happened to be first.
+		func() {
+			defer m.beginRunSummary(runKindRefresh, "")()
+			defer m.emitRunSummary()
+			m.captureAllRecoveryUnits()
+		}()
 	}
 
 	// Fill in dynamic fields under lock.