hub v0.90.0 — a dropped notification leaves a trace, and the backup digest arrives (R-182)
gates / gates (push) Successful in 7s

processOperator's cooldown no longer returns bare. It dropped the event BEFORE
LogNotification, so a suppressed operator alert and an event that never happened
were indistinguishable — from the operator's side and from the hub's own records.
Measured 2026-08-03: nine recovery_unit_capture_failed events arrived, two were
mailed, seven left no row anywhere. That is why the defect took a day to get the
right way round: there was nothing to read.

A suppressed operator event now writes a `suppressed` row carrying the message
and the key that suppressed it. This applies to EVERY operator event, not only
the one that exposed it. It does NOT change the cooldown's duration or semantics.

backup_run_failures: the per-run digest. In allowedEventTypes AND in
operatorOnlyEvents — allowlisting alone does not make an event operator-only,
and FormatCustomerEmail falls back to the raw English message rather than
blocking. A test demonstrates a customer with the type enabled receiving nothing.

recordOnlyEvents: a third routing class — stored and recorded, never mailed.
recovery_unit_capture_failed moves here: it is the record, the digest is the
notification. A register rather than downgrading severity to info, which would
relabel a genuine failure as informational everywhere it is queried.

cooldownRunSuffix: a sibling of cooldownTierSuffix, not a branch inside it, so
tier keeps byte-identical semantics and R-97a's tests are untouched. It makes
the cooldown effectively inert for the digest, which is the intent — a digest is
already rate-limited by construction; the refresh sweep sends no run_id and so
stays under the ordinary hourly cooldown.

The email renders as a list, not a JSON blob. An absent space reading renders as
unavailable, never as zeros.
This commit is contained in:
2026-08-03 13:46:48 +02:00
parent 7dc1744eec
commit dd40f85bb8
8 changed files with 551 additions and 3 deletions
+100
View File
@@ -1,6 +1,7 @@
package notify
import (
"encoding/json"
"fmt"
"strings"
"time"
@@ -41,6 +42,17 @@ Severity: %s
Time: %s
Message: %s`, customerID, eventType, severity, now, message)
// R-182: the backup run digest gets a rendered list instead of a raw JSON blob. It is the one
// operator mail that carries a VARIABLE-LENGTH payload, and a dozen apps as one line of JSON is
// unreadable on a phone at 07:00, which is the only time it matters.
if eventType == "backup_run_failures" {
if rendered, sub, ok := renderBackupRunFailures(customerID, detailsJSON); ok {
return sub, body + rendered + fmt.Sprintf("\n\nDashboard: https://hub.felhom.eu/customers/%s", customerID)
}
// Unparseable details fall through to the raw form below rather than losing the mail. A
// digest that renders badly still tells the operator something; a swallowed one does not.
}
if detailsJSON != "" && detailsJSON != "{}" {
body += fmt.Sprintf("\nDetails: %s", detailsJSON)
}
@@ -282,3 +294,91 @@ Ha nem te kérted ezt, hagyd figyelmen kívül ezt az e-mailt.
Felhom.eu`, link)
return subject, body
}
// ──────────────────────────────────────────────────────────────────────
// R-182 — the backup run digest
// ──────────────────────────────────────────────────────────────────────
// backupRunFailure is one app's failed leg within a run.
type backupRunFailure struct {
App string `json:"app"`
Leg string `json:"leg"`
Reason string `json:"reason"`
}
// backupRunDetails is the digest payload the controller sends.
type backupRunDetails struct {
RunID string `json:"run_id"`
RunKind string `json:"run_kind"`
Failed int `json:"failed"`
Attempted int `json:"attempted"`
TargetPath string `json:"target_path"`
UsedGB float64 `json:"used_gb"`
AvailGB float64 `json:"avail_gb"`
TotalGB float64 `json:"total_gb"`
UsedPercent float64 `json:"used_percent"`
SpaceKnown bool `json:"space_known"`
Apps []backupRunFailure `json:"apps"`
}
// renderBackupRunFailures turns the digest details into an operator-readable block and a subject
// that says the count without being opened. Returns ok=false when the payload cannot be parsed or
// names no apps, so the caller can fall back to the raw rendering rather than mail an empty list.
//
// THE SUCCESS COUNT IS NOT DECORATION. "3 of 4 apps failed" is a catastrophe and "3 of 40" is a bad
// night; the list alone cannot tell them apart, and the operator's first decision — get up now, or
// look after coffee — depends entirely on which it is.
func renderBackupRunFailures(customerID, detailsJSON string) (string, string, bool) {
if detailsJSON == "" {
return "", "", false
}
var d backupRunDetails
if err := json.Unmarshal([]byte(detailsJSON), &d); err != nil || len(d.Apps) == 0 {
return "", "", false
}
kind := d.RunKind
if kind == "" {
kind = "backup"
}
subject := fmt.Sprintf("[Felhom] 🔴 %s: %d of %d apps failed to back up (%s run)",
customerID, d.Failed, d.Attempted, kind)
// Column-align the app names so the leg and reason line up and the block scans vertically.
width := 0
for _, a := range d.Apps {
if len(a.App) > width {
width = len(a.App)
}
}
legWidth := 0
for _, a := range d.Apps {
if len(a.Leg) > legWidth {
legWidth = len(a.Leg)
}
}
var b strings.Builder
fmt.Fprintf(&b, "\n\nFAILED: %d of %d apps attempted in this %s run.\n\n", d.Failed, d.Attempted, kind)
for _, a := range d.Apps {
reason := a.Reason
if reason == "" {
reason = "(no reason recorded)"
}
fmt.Fprintf(&b, " %-*s %-*s %s\n", width, a.App, legWidth, a.Leg, reason)
}
// The space figures answer "is this one broken app or a full disk" before the reasons are read.
// An absent reading renders as unavailable, never as zeros — "0 GB free" and "we could not look"
// are opposite diagnoses (the UnitSpace rule, same reasoning, other side of the wire).
if d.SpaceKnown {
fmt.Fprintf(&b, "\nFilesystem: %s — %.1f/%.1f GB used (%.0f%%), %.1f GB free\n",
d.TargetPath, d.UsedGB, d.TotalGB, d.UsedPercent, d.AvailGB)
} else {
fmt.Fprintf(&b, "\nFilesystem: %s — usage unavailable (the filesystem could not be read)\n", d.TargetPath)
}
b.WriteString("\nEvery failure above is also recorded individually in the notification log,\n")
b.WriteString("whether or not this mail was sent.")
return b.String(), subject, true
}