v0.194.0 — one operator email per backup run, and nothing dropped without a trace (R-182)
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed events reached the hub and TWO operator emails went out. The hub's operator cooldown key is customerID:eventType(+tier) and that event carries `app` but no `tier`, so the key held no app identifier: the first refused app took the hour's slot and every other app's failure was discarded BEFORE anything was written down, leaving no row on any channel. The obvious fix — put `app` in the key — was ruled against: on a full disk it produces one email per app, the volume problem wearing the correctness problem's clothes. internal/backup/runsummary.go: a per-run collector with exactly admissionSet's lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the end and only when something failed. A clean run emits nothing. The per-app event stays and becomes the RECORD — the hub routes it record-only, stored and logged every time, never competing for an email slot. The record and the notification are now different things. Deliberate skips (disconnected, decommissioned) are excluded: they have their own alert, and a nightly email about an unplugged drive is one the operator learns to ignore. A manual run always reports: the digest carries a unique run_id the cooldown cannot collapse. Someone pressing the button is actively trying to get a backup. THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a capture failure found between runs would be recorded and never notified — a new silence introduced while closing one. That path emits a digest with NO run_id, so the ordinary 1-hour cooldown caps it exactly as before while the mail now lists every failing app instead of whichever was first. A refusal is recorded ONCE, where the verdict is taken, not at the three legs that consult it — R-181's contract is one verdict per app per run. Noting it per leg listed one refused app three times and produced "2 of 1 apps failed". Found by the digest's own test, not in review. Silence is safe because the hub's deadline check raises expected_backup_missed from report freshness, independently of any mail this box sends (monitor/deadline.go:396,417). Confirmed, not assumed. 7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first attempt — the AST test walked the backup package and not main.go; the test was fixed and the mutation re-run rather than the pass recorded.
This commit is contained in:
@@ -1,5 +1,55 @@
|
|||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
|
### v0.194.0 — one operator e-mail per backup run, and nothing dropped without a trace (2026-08-03, R-182) — MinAgent: none
|
||||||
|
|
||||||
|
**The defect, measured rather than supposed.** On 2026-08-03 nine per-app
|
||||||
|
`recovery_unit_capture_failed` events reached the hub and **two operator e-mails went out**. The hub's
|
||||||
|
operator cooldown key is `customerID + ":" + eventType + tier-suffix`, and that event carries `app`
|
||||||
|
but **no `tier`** — so the key held no app identifier. The first refused app took the hour's slot and
|
||||||
|
**every other app's failure was discarded before anything was written down**, leaving no row on any
|
||||||
|
channel. A machine deciding not to tell you and nothing happening at all looked identical.
|
||||||
|
|
||||||
|
**The obvious fix was ruled against, and the reason is worth keeping.** Putting `app` into the key
|
||||||
|
fixes the swallowing by producing **one e-mail per failing app**, which on a full disk is a dozen —
|
||||||
|
the volume problem wearing the correctness problem's clothes.
|
||||||
|
|
||||||
|
**What ships instead: ONE digest per run, and every failure recorded when it happens.**
|
||||||
|
|
||||||
|
- **`internal/backup/runsummary.go`** — a per-run collector with exactly `admissionSet`'s lifetime
|
||||||
|
(created where the run begins, cleared when it ends), fed by all three write legs. It emits
|
||||||
|
`backup_run_failures` once at the end, **only when something failed**. A clean run emits nothing —
|
||||||
|
not an empty digest.
|
||||||
|
- **The per-app event stays and becomes the RECORD.** The hub now routes it *record-only*: stored and
|
||||||
|
written to the notification log every time, never competing for an e-mail slot. The record and the
|
||||||
|
notification are now different things, which is the durable half of this change.
|
||||||
|
- **Deliberate skips are not failures.** A disconnected or decommissioned drive has its own alert and
|
||||||
|
is excluded, because a nightly e-mail about an unplugged drive is one the operator learns to ignore.
|
||||||
|
- **A manual run always reports**, even if the nightly one already wrote that hour: the digest carries
|
||||||
|
a unique `run_id` that the hub's cooldown cannot collapse. Someone pressing the button is actively
|
||||||
|
trying to get a backup.
|
||||||
|
|
||||||
|
**THE PERIODIC SWEEP GETS A DIGEST TOO, and that is not symmetry for its own sake.** `GetFullStatus`
|
||||||
|
captures units outside any run. With the per-app event now record-only, a capture failure found
|
||||||
|
between runs would have been recorded and **never notified** — a new silence introduced while closing
|
||||||
|
one. So that path emits a digest as well, deliberately with **no `run_id`**, so the ordinary 1-hour
|
||||||
|
cooldown caps it exactly as before while the mail now lists *every* failing app instead of whichever
|
||||||
|
one happened to be first.
|
||||||
|
|
||||||
|
**A refusal is recorded ONCE, where the verdict is taken**, not at each of the three legs that consult
|
||||||
|
it — R-181's contract is that one verdict covers all three. Noting it per leg listed a single refused
|
||||||
|
app three times and produced counts like *"2 of 1 apps failed"*. **Found by the digest's own test, not
|
||||||
|
in review.**
|
||||||
|
|
||||||
|
**Why silence is safe, checked rather than assumed.** A digest is only safe if the absence of a mail
|
||||||
|
cannot mean "the run never finished". It cannot: the hub's daily deadline check raises
|
||||||
|
`expected_backup_missed` / `expected_dbdump_missed` (`hub/internal/monitor/deadline.go:396,417`) from
|
||||||
|
the box's **report freshness and stored events**, independently of any mail this box chooses to send.
|
||||||
|
|
||||||
|
**Tests: 7 new, plus 4 red-proofs demonstrated failing then restored.** One of them —
|
||||||
|
the `main.go` seam walk — **did not fail on its first attempt**, because the AST test walked the
|
||||||
|
backup package and not `main.go`; the test was fixed and the mutation re-run rather than the pass
|
||||||
|
being recorded.
|
||||||
|
|
||||||
### v0.193.1 — the refusal's size estimate is rendered in bytes, not `0.00 GiB` (2026-08-03, R-181 follow-on)
|
### v0.193.1 — the refusal's size estimate is rendered in bytes, not `0.00 GiB` (2026-08-03, R-181 follow-on)
|
||||||
|
|
||||||
**Found by the live proof run for v0.193.0, not by review.** The refusal message printed the estimate
|
**Found by the live proof run for v0.193.0, not by review.** The refusal message printed the estimate
|
||||||
|
|||||||
@@ -174,6 +174,7 @@
|
|||||||
| `api.GracefulSelfRestart` | controller/internal/api/selfrestart.go | `(logger)` | Controller self-restart | Detached exit; bootstrap unit re-runs the image |
|
| `api.GracefulSelfRestart` | controller/internal/api/selfrestart.go | `(logger)` | Controller self-restart | Detached exit; bootstrap unit re-runs the image |
|
||||||
| `Settings.AddPendingEvent/DrainPendingEvents` | controller/internal/settings/settings.go | offline event queue | Events while hub unreachable | — |
|
| `Settings.AddPendingEvent/DrainPendingEvents` | controller/internal/settings/settings.go | offline event queue | Events while hub unreachable | — |
|
||||||
| `Manager.SetUnitNotify` + `UnitSpace` (R-158/R-167, v0.191.0) | controller/internal/backup/recovery_unit.go | `(func(stack string, err error, *UnitSpace))` | THE per-app Tier-1 recovery-unit capture failure alert — fires PER APP from `captureAllRecoveryUnits`, loop continues | **OPERATOR-TIER** (`recovery_unit_capture_failed`, in the hub's `operatorOnlyEvents`). **NEVER route it to `backup_failed`** — that type is in `DefaultEnabledEvents` and carries Hungarian copy, so it emails the CUSTOMER about a failure they cannot act on (D-c; R-158's own proposal said `backup_failed` and D-c overrides it). `UnitSpace` is **nil when the target filesystem is unreadable** and renders as *"unavailable"*, never as zeros — "0 GB free" and "we could not look" are opposite diagnoses. No controller-side cooldown: the hub owns it |
|
| `Manager.SetUnitNotify` + `UnitSpace` (R-158/R-167, v0.191.0) | controller/internal/backup/recovery_unit.go | `(func(stack string, err error, *UnitSpace))` | THE per-app Tier-1 recovery-unit capture failure alert — fires PER APP from `captureAllRecoveryUnits`, loop continues | **OPERATOR-TIER** (`recovery_unit_capture_failed`, in the hub's `operatorOnlyEvents`). **NEVER route it to `backup_failed`** — that type is in `DefaultEnabledEvents` and carries Hungarian copy, so it emails the CUSTOMER about a failure they cannot act on (D-c; R-158's own proposal said `backup_failed` and D-c overrides it). `UnitSpace` is **nil when the target filesystem is unreadable** and renders as *"unavailable"*, never as zeros — "0 GB free" and "we could not look" are opposite diagnoses. No controller-side cooldown: the hub owns it |
|
||||||
|
| `Manager.beginRunSummary` / `noteFailure` / `noteAttempted` / `emitRunSummary` / `SetRunSummaryNotify` (R-182, v0.194.0) | controller/internal/backup/runsummary.go | `(kind, runID) func()` / `(app, leg, reason)` / `(RunSummary)` | **THE per-run operator digest.** One `backup_run_failures` event at the end of a run listing every failed app, its leg and its reason — emitted ONLY when something failed | **The RECORD and the NOTIFICATION are different things and must stay so.** The per-app `recovery_unit_capture_failed` event is the record (hub routes it *record-only*, stored + logged every time); this digest is the notification. Before R-182 one event was both, and did neither: nine arrived, two were mailed, seven vanished before `LogNotification`. **Lifetime is `admissionSet`'s exactly** — absent collector means "no run in flight", never a stale answer. **A refusal is noted ONCE, inside `admitApp` where the verdict is taken**, not at the three legs that consult it: R-181's one-verdict-covers-all-three contract makes per-leg noting produce "2 of 1 apps failed". **Deliberate skips (disconnected / decommissioned) must NEVER be noted** — they have their own alert and a nightly digest about an unplugged drive is an ignored digest. **A clean run emits NOTHING**; silence is safe only because the hub's deadline check (`monitor/deadline.go:396,417`) raises a missed backup from report freshness independently — if that is ever weakened this design loses its footing. **`run_id` is unique per real run** (so the hub's 1-h cooldown cannot collapse a manual run into the nightly one) and **deliberately EMPTY on the periodic refresh sweep**, which must stay under that cooldown or a polled status page becomes a mail flood |
|
||||||
| `Manager.admitApp` / `beginAdmissionRun` / `decideAdmission` / `estimatedWriteBytes` (R-181, v0.193.0) | controller/internal/backup/admission.go | `(stackName) bool` / `() func()` | **THE reserve gate. Call it before ANY per-app backup write** — one verdict per app per run, covering the DB dump, the volume dump and the unit capture (all three write under one per-app root) | **Decided LAZILY at the app's first write, never once at run start** — app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. **Never re-decided between an app's own legs**: that is exactly the split R-181 closed (bulk written, capture refused). **Reset per run** via the closer `beginAdmissionRun` returns. **Must sit ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided inside it has already bounced the app. Fires **exactly one** `unitNotify` per refused app per run. Nil admission set (periodic status refresh) → decides fresh, which is still once per app per sweep. Wiring pinned by an **AST walk** in `TestAdmission_IsWiredIntoEveryProductionWriteLeg`, not `strings.Contains` |
|
| `Manager.admitApp` / `beginAdmissionRun` / `decideAdmission` / `estimatedWriteBytes` (R-181, v0.193.0) | controller/internal/backup/admission.go | `(stackName) bool` / `() func()` | **THE reserve gate. Call it before ANY per-app backup write** — one verdict per app per run, covering the DB dump, the volume dump and the unit capture (all three write under one per-app root) | **Decided LAZILY at the app's first write, never once at run start** — app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. **Never re-decided between an app's own legs**: that is exactly the split R-181 closed (bulk written, capture refused). **Reset per run** via the closer `beginAdmissionRun` returns. **Must sit ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided inside it has already bounced the app. Fires **exactly one** `unitNotify` per refused app per run. Nil admission set (periodic status refresh) → decides fresh, which is still once per app per sweep. Wiring pinned by an **AST walk** in `TestAdmission_IsWiredIntoEveryProductionWriteLeg`, not `strings.Contains` |
|
||||||
| `Manager.floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` / `floorReason` (R-165 B2 v0.192.0, size term R-181 v0.193.0) | controller/internal/backup/recovery_unit.go | `(*UnitSpace, estGiB float64) (*UnitSpace, floorReason)` | The pure two-question predicate behind `admitApp`: is the filesystem already below the reserve (`floorHeadroom`), and would THIS app's write take it below (`floorSize`)? | **Headroom is about the FILESYSTEM, never a per-unit cap** — a size cap is R-163 rebuilt inside one volume; the size term bounds the *delta*, not the unit. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. **`estGiB == 0` degrades to headroom-only on purpose** — refusing an app with no history makes the FIRST backup the one that can never happen. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk |
|
| `Manager.floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` / `floorReason` (R-165 B2 v0.192.0, size term R-181 v0.193.0) | controller/internal/backup/recovery_unit.go | `(*UnitSpace, estGiB float64) (*UnitSpace, floorReason)` | The pure two-question predicate behind `admitApp`: is the filesystem already below the reserve (`floorHeadroom`), and would THIS app's write take it below (`floorSize`)? | **Headroom is about the FILESYSTEM, never a per-unit cap** — a size cap is R-163 rebuilt inside one volume; the size term bounds the *delta*, not the unit. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. **`estGiB == 0` degrades to headroom-only on purpose** — refusing an app with no history makes the FIRST backup the one that can never happen. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk |
|
||||||
| `fillwatch.Watcher` (`New`/`SetNotify`/`Check`) (R-167, v0.191.0) | controller/internal/fillwatch/fillwatch.go | `(statePath, logger, targetsFn, usageFn)` → `Check() error` | THE customer fill warning — warns BEFORE a filesystem fills, per FILESYSTEM (never per app: one full disk holding ten apps would fire ten times) | Emits the **pre-existing** `disk_warning`/`disk_critical` pair, which was allowlisted + copy'd + default-enabled with **no producer in any repo** until now — do NOT mint a new type beside it. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range. **Edge-triggered on ESCALATION ONLY**, state persisted; de-escalation is silent and re-arms. Hysteresis dead zone between clear (75% / 7 GiB) and warn — pinned by `TestThresholdsKeepTheirHysteresisGap`. **A nil usage read is NEVER a warning** (§8.4). The hub has **no `customerMessages` entry** for either type on purpose — an entry would override the dynamic message and discard the drive label + free space |
|
| `fillwatch.Watcher` (`New`/`SetNotify`/`Check`) (R-167, v0.191.0) | controller/internal/fillwatch/fillwatch.go | `(statePath, logger, targetsFn, usageFn)` → `Check() error` | THE customer fill warning — warns BEFORE a filesystem fills, per FILESYSTEM (never per app: one full disk holding ten apps would fire ten times) | Emits the **pre-existing** `disk_warning`/`disk_critical` pair, which was allowlisted + copy'd + default-enabled with **no producer in any repo** until now — do NOT mint a new type beside it. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range. **Edge-triggered on ESCALATION ONLY**, state persisted; de-escalation is silent and re-arms. Hysteresis dead zone between clear (75% / 7 GiB) and warn — pinned by `TestThresholdsKeepTheirHysteresisGap`. **A nil usage read is NEVER a warning** (§8.4). The hub has **no `customerMessages` entry** for either type on purpose — an entry would override the dynamic message and discard the drive label + free space |
|
||||||
|
|||||||
@@ -744,6 +744,25 @@ func main() {
|
|||||||
"Recovery unit capture FAILED for %q — the app has no fresh local (Tier-1) backup, and Tier-2/Tier-3 have nothing to copy. %s. Error: %v",
|
"Recovery unit capture FAILED for %q — the app has no fresh local (Tier-1) backup, and Tier-2/Tier-3 have nothing to copy. %s. Error: %v",
|
||||||
stackName, usage.String(), err), d)
|
stackName, usage.String(), err), d)
|
||||||
})
|
})
|
||||||
|
// R-182: the ONE operator digest per backup run. The per-app events above are the RECORD
|
||||||
|
// (the hub routes them record-only); this is the NOTIFICATION, sent once at the end of a run
|
||||||
|
// and only when something actually failed. A clean run sends nothing — and that silence is
|
||||||
|
// safe because the hub's own daily deadline check raises expected_backup_missed from report
|
||||||
|
// freshness, independently of any mail this box chooses to send.
|
||||||
|
backupMgr.SetRunSummaryNotify(func(rs backup.RunSummary) {
|
||||||
|
d := notify.BackupRunFailuresDetails{
|
||||||
|
RunID: rs.RunID, RunKind: rs.RunKind,
|
||||||
|
Failed: rs.Failed, Attempted: rs.Attempted,
|
||||||
|
}
|
||||||
|
for _, a := range rs.Apps {
|
||||||
|
d.Apps = append(d.Apps, notify.RunFailureDetail{App: a.App, Leg: a.Leg, Reason: a.Reason})
|
||||||
|
}
|
||||||
|
if rs.Usage != nil {
|
||||||
|
d.TargetPath, d.UsedGB, d.AvailGB = rs.Usage.Path, rs.Usage.UsedGB, rs.Usage.AvailGB
|
||||||
|
d.TotalGB, d.UsedPercent, d.SpaceKnown = rs.Usage.TotalGB, rs.Usage.UsedPercent, true
|
||||||
|
}
|
||||||
|
notifier.NotifyBackupRunFailures(rs.Message, d)
|
||||||
|
})
|
||||||
// 3a: the pre-push enlargement gate blocked an app's userdata push (config+DB still saved). Edge-
|
// 3a: the pre-push enlargement gate blocked an app's userdata push (config+DB still saved). Edge-
|
||||||
// triggered by the engine (only NEW blocks notify), so the hub's per-event-type cooldown suffices —
|
// triggered by the engine (only NEW blocks notify), so the hub's per-event-type cooldown suffices —
|
||||||
// no controller-side timer (the hub owns cooldown).
|
// no controller-side timer (the hub owns cooldown).
|
||||||
|
|||||||
@@ -1010,6 +1010,7 @@ func (r *Router) triggerBackup(w http.ResponseWriter, _ *http.Request) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
r.logger.Println("[INFO] [api] Manual app-data backup (DB dump) triggered")
|
r.logger.Println("[INFO] [api] Manual app-data backup (DB dump) triggered")
|
||||||
|
r.backupMgr.MarkManualRun() // R-182: operator-triggered — its digest must not be collapsed into the nightly one
|
||||||
go r.backupMgr.RunDBDumps(context.Background())
|
go r.backupMgr.RunDBDumps(context.Background())
|
||||||
|
|
||||||
writeJSON(w, http.StatusOK, apiResponse{OK: true, Message: "Mentés elindítva"})
|
writeJSON(w, http.StatusOK, apiResponse{OK: true, Message: "Mentés elindítva"})
|
||||||
|
|||||||
@@ -132,6 +132,12 @@ func (m *Manager) admitApp(stackName string) bool {
|
|||||||
if m.unitNotify != nil {
|
if m.unitNotify != nil {
|
||||||
m.unitNotify(stackName, v.err, v.usage)
|
m.unitNotify(stackName, v.err, v.usage)
|
||||||
}
|
}
|
||||||
|
// R-182: the digest entry is recorded HERE, where the verdict is taken — once per app per run.
|
||||||
|
// Not at the three call sites that consult the memo: R-181's whole contract is that ONE verdict
|
||||||
|
// covers all three legs, so noting it per leg listed a single refused app three times and
|
||||||
|
// produced counts like "2 of 1 apps failed". The leg name says what actually happened, which is
|
||||||
|
// that nothing was attempted at all.
|
||||||
|
m.noteFailure(stackName, "whole app (refused before any write)", v.err.Error())
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -15,6 +15,8 @@ import (
|
|||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||||
)
|
)
|
||||||
|
|
||||||
// R-181 — the reserve guards the write that fills the disk, and its promise is true.
|
// R-181 — the reserve guards the write that fills the disk, and its promise is true.
|
||||||
@@ -33,6 +35,7 @@ import (
|
|||||||
type admissionProvider struct {
|
type admissionProvider struct {
|
||||||
stacks []string
|
stacks []string
|
||||||
volumes map[string][]string
|
volumes map[string][]string
|
||||||
|
hdd map[string]string // per-app drive path, for the drive-state skip tests
|
||||||
dir string
|
dir string
|
||||||
infoHits []string
|
infoHits []string
|
||||||
stopped []string
|
stopped []string
|
||||||
@@ -47,7 +50,7 @@ func (p *admissionProvider) ListDeployedStacks() []StackSummary {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
func (p *admissionProvider) GetStackHDDMounts(string) []string { return nil }
|
func (p *admissionProvider) GetStackHDDMounts(string) []string { return nil }
|
||||||
func (p *admissionProvider) GetStackHDDPath(string) string { return "" }
|
func (p *admissionProvider) GetStackHDDPath(n string) string { return p.hdd[n] }
|
||||||
func (p *admissionProvider) GetImportRoot() string { return "" }
|
func (p *admissionProvider) GetImportRoot() string { return "" }
|
||||||
func (p *admissionProvider) GetDockerVolumes(name string) []string {
|
func (p *admissionProvider) GetDockerVolumes(name string) []string {
|
||||||
if p.volumes == nil {
|
if p.volumes == nil {
|
||||||
@@ -88,7 +91,7 @@ func newAdmissionHarness(t *testing.T, stacks ...string) *admissionHarness {
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
dir := t.TempDir()
|
dir := t.TempDir()
|
||||||
h := &admissionHarness{
|
h := &admissionHarness{
|
||||||
prov: &admissionProvider{stacks: stacks, dir: dir},
|
prov: &admissionProvider{stacks: stacks, dir: dir, hdd: map[string]string{}},
|
||||||
usage: map[string]*UnitSpace{},
|
usage: map[string]*UnitSpace{},
|
||||||
dir: dir,
|
dir: dir,
|
||||||
logs: &bytes.Buffer{},
|
logs: &bytes.Buffer{},
|
||||||
@@ -116,6 +119,45 @@ func newAdmissionHarness(t *testing.T, stacks ...string) *admissionHarness {
|
|||||||
return h
|
return h
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// markDisconnected / markDecommissioned put a real settings row behind the drive-state skips, so
|
||||||
|
// Scenario F exercises the production guards rather than a stub of them.
|
||||||
|
func (h *admissionHarness) markDisconnected(app string) {
|
||||||
|
h.driveState(app, true, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *admissionHarness) markDecommissioned(app string) {
|
||||||
|
h.driveState(app, false, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *admissionHarness) driveState(app string, disconnected, decommissioned bool) {
|
||||||
|
if h.m.settings == nil {
|
||||||
|
sett, err := settings.Load(filepath.Join(h.dir, "settings.json"), log.New(io.Discard, "", 0))
|
||||||
|
if err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
h.m.settings = sett
|
||||||
|
}
|
||||||
|
// Each such app gets its OWN drive path, or marking one would skip them all.
|
||||||
|
p := filepath.Join(h.dir, "drives", app)
|
||||||
|
if err := os.MkdirAll(p, 0o755); err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
h.prov.hdd[app] = p
|
||||||
|
if err := h.m.settings.AddStoragePath(settings.StoragePath{Path: p, Label: app}); err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
if disconnected {
|
||||||
|
if err := h.m.settings.SetDisconnected(p, true, nil); err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if decommissioned {
|
||||||
|
if err := h.m.settings.SetDecommissioned(p, ""); err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func (h *admissionHarness) nsRoot() string { return filepath.Join(h.dir, "felhom-data") }
|
func (h *admissionHarness) nsRoot() string { return filepath.Join(h.dir, "felhom-data") }
|
||||||
|
|
||||||
// setSpace states the filesystem's occupancy as a test INPUT — the whole point of the unitSpaceFn
|
// setSpace states the filesystem's occupancy as a test INPUT — the whole point of the unitSpaceFn
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||||
@@ -64,6 +65,16 @@ type Manager struct {
|
|||||||
admissionMu sync.Mutex
|
admissionMu sync.Mutex
|
||||||
admission *admissionSet
|
admission *admissionSet
|
||||||
|
|
||||||
|
// summary (R-182) is the per-RUN digest collector, guarded by summaryMu. Same lifetime as
|
||||||
|
// `admission` and for the same reason: an absent collector means "no run in flight", never a
|
||||||
|
// stale answer from last night. runSummaryNotify is the operator digest seam, wired in main.go.
|
||||||
|
summaryMu sync.Mutex
|
||||||
|
summary *runSummary
|
||||||
|
runSummaryNotify func(RunSummary)
|
||||||
|
// manualRun tags the NEXT run as operator-triggered (cleared as the run starts), so the digest
|
||||||
|
// can say which kind it was and the hub can decline to collapse a manual run into a nightly one.
|
||||||
|
manualRun atomic.Bool
|
||||||
|
|
||||||
// appStop (R-166) is the crash marker for operations that stop an app, work on its data, and
|
// appStop (R-166) is the crash marker for operations that stop an app, work on its data, and
|
||||||
// start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power
|
// start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power
|
||||||
// cut in that window leaves a durable record that Recover honours at the next startup. Built in
|
// cut in that window leaves a durable record that Recover honours at the next startup. Built in
|
||||||
@@ -426,6 +437,14 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
|
|||||||
// question with last night's disk.
|
// question with last night's disk.
|
||||||
defer m.beginAdmissionRun()()
|
defer m.beginAdmissionRun()()
|
||||||
|
|
||||||
|
// R-182: the digest scope has the same lifetime. `emitRunSummary` runs BEFORE the closer (defers
|
||||||
|
// unwind last-in-first-out), so the summary is still populated when it is sent, and it sends
|
||||||
|
// nothing at all when the run was clean.
|
||||||
|
kind := m.runKindFor()
|
||||||
|
m.manualRun.Store(false) // tags exactly ONE run; a stale flag would mislabel every later nightly
|
||||||
|
defer m.beginRunSummary(kind, newRunID())()
|
||||||
|
defer m.emitRunSummary()
|
||||||
|
|
||||||
dbs, err := DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames())
|
dbs, err := DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
m.logger.Printf("[ERROR] [backup] Database discovery failed: %v", err)
|
m.logger.Printf("[ERROR] [backup] Database discovery failed: %v", err)
|
||||||
@@ -465,6 +484,7 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
|
|||||||
// where an app's verdict is taken, because the DB leg runs first; the volume leg and the
|
// where an app's verdict is taken, because the DB leg runs first; the volume leg and the
|
||||||
// capture then read the same memo. SKIP, not FAIL — a deliberate hold is not a broken dump,
|
// capture then read the same memo. SKIP, not FAIL — a deliberate hold is not a broken dump,
|
||||||
// and the operator alert (fired once, inside admitApp) is the signal that it happened.
|
// and the operator alert (fired once, inside admitApp) is the signal that it happened.
|
||||||
|
m.noteAttempted(db.StackName)
|
||||||
if !m.admitApp(db.StackName) {
|
if !m.admitApp(db.StackName) {
|
||||||
summary = append(summary, fmt.Sprintf("SKIP %s (reserve — app backup refused)", db.ContainerName))
|
summary = append(summary, fmt.Sprintf("SKIP %s (reserve — app backup refused)", db.ContainerName))
|
||||||
continue
|
continue
|
||||||
@@ -478,6 +498,7 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
|
|||||||
if result.Error != nil {
|
if result.Error != nil {
|
||||||
allOK = false
|
allOK = false
|
||||||
summary = append(summary, fmt.Sprintf("FAIL %s: %v", result.DB.ContainerName, result.Error))
|
summary = append(summary, fmt.Sprintf("FAIL %s: %v", result.DB.ContainerName, result.Error))
|
||||||
|
m.noteFailure(db.StackName, "database dump", result.Error.Error())
|
||||||
m.logger.Printf("[ERROR] [backup] DB dump failed for %s: %v", result.DB.ContainerName, result.Error)
|
m.logger.Printf("[ERROR] [backup] DB dump failed for %s: %v", result.DB.ContainerName, result.Error)
|
||||||
} else {
|
} else {
|
||||||
totalSize += result.Size
|
totalSize += result.Size
|
||||||
@@ -608,6 +629,7 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
|
|||||||
// R-181: the reserve, ahead of DumpAppVolumesSafe so a refused app is NOT stopped. For an app
|
// R-181: the reserve, ahead of DumpAppVolumesSafe so a refused app is NOT stopped. For an app
|
||||||
// that already has a DB this is a memo lookup taken before its DB dump; for a volume-only app
|
// that already has a DB this is a memo lookup taken before its DB dump; for a volume-only app
|
||||||
// this is where its verdict is taken, still before its first byte.
|
// this is where its verdict is taken, still before its first byte.
|
||||||
|
m.noteAttempted(stack.Name)
|
||||||
if !m.admitApp(stack.Name) {
|
if !m.admitApp(stack.Name) {
|
||||||
summary = append(summary, fmt.Sprintf("SKIP %s volumes (reserve — app backup refused)", stack.Name))
|
summary = append(summary, fmt.Sprintf("SKIP %s volumes (reserve — app backup refused)", stack.Name))
|
||||||
continue
|
continue
|
||||||
@@ -616,6 +638,7 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
|
|||||||
if err := dump(stack.Name); err != nil {
|
if err := dump(stack.Name); err != nil {
|
||||||
allOK = false
|
allOK = false
|
||||||
summary = append(summary, fmt.Sprintf("FAIL %s volumes: %v", stack.Name, err))
|
summary = append(summary, fmt.Sprintf("FAIL %s volumes: %v", stack.Name, err))
|
||||||
|
m.noteFailure(stack.Name, "volume dump", err.Error())
|
||||||
m.logger.Printf("[ERROR] [backup] Volume dump failed for %s: %v", stack.Name, err)
|
m.logger.Printf("[ERROR] [backup] Volume dump failed for %s: %v", stack.Name, err)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -1001,7 +1024,19 @@ func (m *Manager) RefreshCache(nextDBDump time.Time) {
|
|||||||
// Phase 2: keep each app's recovery unit current with its definition. Idempotent
|
// Phase 2: keep each app's recovery unit current with its definition. Idempotent
|
||||||
// (checksum-skip), so this periodic refresh only writes when the config actually changed,
|
// (checksum-skip), so this periodic refresh only writes when the config actually changed,
|
||||||
// and ensures units exist shortly after startup without waiting for the daily DB dump.
|
// and ensures units exist shortly after startup without waiting for the daily DB dump.
|
||||||
|
//
|
||||||
|
// R-182: this sweep gets its OWN digest scope. It has to, and the reason is the whole
|
||||||
|
// balance of this change. The per-app event is now record-only, so without a digest here a
|
||||||
|
// capture failure detected between runs would be recorded and NEVER notified — a new
|
||||||
|
// silence introduced while closing one. But this path can fire on every status poll, so its
|
||||||
|
// digest deliberately carries NO run id: the hub's ordinary 1-hour operator cooldown then
|
||||||
|
// applies, which caps it at one mail an hour exactly as before, while the mail now lists
|
||||||
|
// EVERY failing app instead of whichever one happened to be first.
|
||||||
|
func() {
|
||||||
|
defer m.beginRunSummary(runKindRefresh, "")()
|
||||||
|
defer m.emitRunSummary()
|
||||||
m.captureAllRecoveryUnits()
|
m.captureAllRecoveryUnits()
|
||||||
|
}()
|
||||||
}
|
}
|
||||||
|
|
||||||
// Fill in dynamic fields under lock.
|
// Fill in dynamic fields under lock.
|
||||||
|
|||||||
@@ -351,11 +351,13 @@ func (m *Manager) captureAllRecoveryUnits() {
|
|||||||
if m.settings != nil && (m.settings.IsDisconnected(drivePath) || m.settings.IsDecommissioned(drivePath)) {
|
if m.settings != nil && (m.settings.IsDisconnected(drivePath) || m.settings.IsDecommissioned(drivePath)) {
|
||||||
continue // drive not writable — skip, the existing unit stays as-is
|
continue // drive not writable — skip, the existing unit stays as-is
|
||||||
}
|
}
|
||||||
|
m.noteAttempted(stack.Name)
|
||||||
// The reserve, checked BEFORE anything is written. Per app, and the loop continues.
|
// The reserve, checked BEFORE anything is written. Per app, and the loop continues.
|
||||||
if !m.admitApp(stack.Name) {
|
if !m.admitApp(stack.Name) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if err := m.CaptureRecoveryUnit(stack.Name); err != nil {
|
if err := m.CaptureRecoveryUnit(stack.Name); err != nil {
|
||||||
|
m.noteFailure(stack.Name, "recovery-unit capture", err.Error())
|
||||||
m.logger.Printf("[WARN] [backup] Recovery unit capture failed for %s: %v", stack.Name, err)
|
m.logger.Printf("[WARN] [backup] Recovery unit capture failed for %s: %v", stack.Name, err)
|
||||||
// R-158: per app, and the loop CONTINUES — one app's failure must not silence the
|
// R-158: per app, and the loop CONTINUES — one app's failure must not silence the
|
||||||
// others, and it must not abort their captures either. The space figures are read at
|
// others, and it must not abort their captures either. The space figures are read at
|
||||||
|
|||||||
@@ -0,0 +1,270 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ── The backup run digest (R-182) ────────────────────────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// WHAT WAS WRONG, MEASURED. On 2026-08-03 nine per-app `recovery_unit_capture_failed` events reached
|
||||||
|
// the hub and TWO operator e-mails went out. The hub's operator cooldown key is
|
||||||
|
// `customerID + ":" + eventType + tier-suffix`, and that event carries `app` but no `tier`, so the
|
||||||
|
// key held **no app identifier**: the first refused app took the hour's slot and every other app's
|
||||||
|
// failure was discarded — *before* anything was written down, so it left no row on any channel and
|
||||||
|
// could not be found afterwards.
|
||||||
|
//
|
||||||
|
// WHY NOT JUST PUT THE APP IN THE KEY. That was the obvious fix and the operator ruled against it:
|
||||||
|
// on a full disk it produces one e-mail per app, which is the volume problem wearing the correctness
|
||||||
|
// problem's clothes. **One digest per run instead**, listing every failure — and separately, every
|
||||||
|
// failure recorded when it happens.
|
||||||
|
//
|
||||||
|
// THE RECORD AND THE NOTIFICATION ARE DIFFERENT THINGS, and that separation is the durable part:
|
||||||
|
//
|
||||||
|
// - the RECORD is the per-app `recovery_unit_capture_failed` event, emitted unconditionally, now
|
||||||
|
// routed record-only by the hub so it never competes for an e-mail slot;
|
||||||
|
// - the NOTIFICATION is this digest, emitted once per run and only when something failed.
|
||||||
|
//
|
||||||
|
// WHY A DIGEST IS SAFE HERE — the one thing that could have made it dangerous. A digest introduces a
|
||||||
|
// silent-failure path if the ABSENCE of an e-mail could mean "the run never finished". It cannot:
|
||||||
|
// the hub's daily deadline check raises `expected_backup_missed` / `expected_dbdump_missed`
|
||||||
|
// (`hub/internal/monitor/deadline.go:396,417`) from the box's REPORT freshness and its stored
|
||||||
|
// events, entirely independently of any mail the controller chooses to send. Silence therefore still
|
||||||
|
// means "the run finished and found nothing wrong". If that check is ever weakened, this design
|
||||||
|
// loses its footing — which is why it is named here and not only in a report.
|
||||||
|
|
||||||
|
// runKind distinguishes the paths that can produce a digest. It is carried in the details and shown
|
||||||
|
// in the subject, because "the nightly run failed" and "the run I just triggered failed" are read
|
||||||
|
// differently at 07:00.
|
||||||
|
const (
|
||||||
|
runKindNightly = "nightly"
|
||||||
|
runKindManual = "manual"
|
||||||
|
// runKindRefresh is the PERIODIC status sweep, which captures units outside any backup run.
|
||||||
|
//
|
||||||
|
// IT DELIBERATELY CARRIES NO run_id. A run digest gets a unique run id so the hub's 1-hour
|
||||||
|
// cooldown can never collapse two real runs (the operator ruled on that explicitly). The sweep
|
||||||
|
// is the opposite case: it can fire every time the status page is polled, so it MUST fall under
|
||||||
|
// the ordinary cooldown or a full disk becomes a mail flood — the exact failure this whole
|
||||||
|
// change exists to avoid, arriving from the other direction.
|
||||||
|
//
|
||||||
|
// Without this path the sweep's failures would be recorded and never notified, because the
|
||||||
|
// per-app event is now record-only — a NEW silence introduced while fixing a silence. This is
|
||||||
|
// what stops that.
|
||||||
|
runKindRefresh = "refresh"
|
||||||
|
)
|
||||||
|
|
||||||
|
// runFailure is one app's failed or refused leg within a run.
|
||||||
|
type runFailure struct {
|
||||||
|
App string
|
||||||
|
Leg string
|
||||||
|
Reason string
|
||||||
|
}
|
||||||
|
|
||||||
|
// runSummary is the per-RUN collector. Same lifetime as `admissionSet` and for the same reason: an
|
||||||
|
// absent collector must mean "no run in flight", never "a stale answer from last night".
|
||||||
|
type runSummary struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
kind string
|
||||||
|
runID string
|
||||||
|
attempted map[string]bool // apps this run actually tried to back up
|
||||||
|
failures []runFailure
|
||||||
|
}
|
||||||
|
|
||||||
|
// beginRunSummary opens the per-run digest scope and returns the closer, mirroring
|
||||||
|
// beginAdmissionRun. A second call while one is live REPLACES it and the closer restores the
|
||||||
|
// previous, so nesting cannot silently drop a caller's scope.
|
||||||
|
func (m *Manager) beginRunSummary(kind, runID string) func() {
|
||||||
|
m.summaryMu.Lock()
|
||||||
|
prev := m.summary
|
||||||
|
m.summary = &runSummary{kind: kind, runID: runID, attempted: map[string]bool{}}
|
||||||
|
m.summaryMu.Unlock()
|
||||||
|
return func() {
|
||||||
|
m.summaryMu.Lock()
|
||||||
|
m.summary = prev
|
||||||
|
m.summaryMu.Unlock()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// noteAttempted records that this run tried to back up an app.
|
||||||
|
//
|
||||||
|
// THE DENOMINATOR IS NOT DECORATION. "3 of 4 apps failed" is a catastrophe and "3 of 40" is a bad
|
||||||
|
// night, and a list of names cannot tell them apart — the operator's first decision, get up now or
|
||||||
|
// look after coffee, is made from exactly this ratio.
|
||||||
|
func (m *Manager) noteAttempted(app string) {
|
||||||
|
m.summaryMu.Lock()
|
||||||
|
defer m.summaryMu.Unlock()
|
||||||
|
if m.summary != nil {
|
||||||
|
m.summary.mu.Lock()
|
||||||
|
m.summary.attempted[app] = true
|
||||||
|
m.summary.mu.Unlock()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// noteFailure records one failed or refused leg. Deliberate skips must NOT come through here —
|
||||||
|
// §8.1: a drive that is unplugged or decommissioned has its own alert, and putting it in the digest
|
||||||
|
// turns a nightly e-mail into one the operator learns to ignore.
|
||||||
|
func (m *Manager) noteFailure(app, leg, reason string) {
|
||||||
|
m.summaryMu.Lock()
|
||||||
|
defer m.summaryMu.Unlock()
|
||||||
|
if m.summary == nil {
|
||||||
|
return // no run in flight — nothing to summarise
|
||||||
|
}
|
||||||
|
m.summary.mu.Lock()
|
||||||
|
m.summary.attempted[app] = true
|
||||||
|
m.summary.failures = append(m.summary.failures, runFailure{App: app, Leg: leg, Reason: reason})
|
||||||
|
m.summary.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// emitRunSummary sends the digest, if and only if something failed.
|
||||||
|
//
|
||||||
|
// A CLEAN RUN EMITS NOTHING — not an empty digest. A nightly "0 failures" mail is an unread mail
|
||||||
|
// within a week, and it would also destroy the property the whole design rests on: that silence
|
||||||
|
// means the run finished and found nothing wrong.
|
||||||
|
func (m *Manager) emitRunSummary() {
|
||||||
|
m.summaryMu.Lock()
|
||||||
|
s := m.summary
|
||||||
|
m.summaryMu.Unlock()
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.mu.Lock()
|
||||||
|
failures := append([]runFailure(nil), s.failures...)
|
||||||
|
attempted := len(s.attempted)
|
||||||
|
kind, runID := s.kind, s.runID
|
||||||
|
s.mu.Unlock()
|
||||||
|
|
||||||
|
if len(failures) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Stable order so two identical runs render identically and a diff of two mails is meaningful.
|
||||||
|
sort.Slice(failures, func(i, j int) bool {
|
||||||
|
if failures[i].App != failures[j].App {
|
||||||
|
return failures[i].App < failures[j].App
|
||||||
|
}
|
||||||
|
return failures[i].Leg < failures[j].Leg
|
||||||
|
})
|
||||||
|
|
||||||
|
if m.runSummaryNotify == nil {
|
||||||
|
// Nil-safe, but say so: an unwired seam here means the digest exists and reaches nobody,
|
||||||
|
// which is the built-but-never-wired failure this project has shipped four times.
|
||||||
|
m.logger.Printf("[WARN] [backup] %d app(s) failed in this %s run but no run-summary notifier is wired — "+
|
||||||
|
"the failures are recorded per app and NOT summarised to the operator", len(failures), kind)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
apps := make([]RunFailureDetail, 0, len(failures))
|
||||||
|
names := make([]string, 0, len(failures))
|
||||||
|
for _, f := range failures {
|
||||||
|
apps = append(apps, RunFailureDetail{App: f.App, Leg: f.Leg, Reason: f.Reason})
|
||||||
|
names = append(names, f.App)
|
||||||
|
}
|
||||||
|
msg := fmt.Sprintf("%d of %d apps failed to back up in this %s run: %s",
|
||||||
|
len(failures), attempted, kind, strings.Join(dedupeStable(names), ", "))
|
||||||
|
|
||||||
|
m.logger.Printf("[INFO] [backup] Run summary: %d of %d apps failed (%s run) — notifying the operator once",
|
||||||
|
len(failures), attempted, kind)
|
||||||
|
m.runSummaryNotify(RunSummary{
|
||||||
|
RunID: runID,
|
||||||
|
RunKind: kind,
|
||||||
|
Failed: len(failures),
|
||||||
|
Attempted: attempted,
|
||||||
|
Apps: apps,
|
||||||
|
Usage: m.summaryUsage(),
|
||||||
|
Message: msg,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// summaryUsage reads the target filesystem once for the digest. It uses the same seam the reserve
|
||||||
|
// does, so a test states occupancy as an input rather than manufacturing it. Nil when unreadable —
|
||||||
|
// which the hub renders as "unavailable", never as zeros.
|
||||||
|
func (m *Manager) summaryUsage() *UnitSpace {
|
||||||
|
if m.stackProvider == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
for _, st := range m.stackProvider.ListDeployedStacks() {
|
||||||
|
if u := m.readUnitSpace(st.Name); u != nil {
|
||||||
|
return u
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func dedupeStable(in []string) []string {
|
||||||
|
seen := map[string]bool{}
|
||||||
|
out := in[:0:0]
|
||||||
|
for _, s := range in {
|
||||||
|
if !seen[s] {
|
||||||
|
seen[s] = true
|
||||||
|
out = append(out, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// RunFailureDetail is one app's failure as the hub receives it.
|
||||||
|
type RunFailureDetail struct {
|
||||||
|
App string `json:"app"`
|
||||||
|
Leg string `json:"leg"`
|
||||||
|
Reason string `json:"reason"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// RunSummary is the digest payload handed to the notifier seam.
|
||||||
|
type RunSummary struct {
|
||||||
|
RunID string
|
||||||
|
RunKind string
|
||||||
|
Failed int
|
||||||
|
Attempted int
|
||||||
|
Apps []RunFailureDetail
|
||||||
|
Usage *UnitSpace
|
||||||
|
Message string
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetRunSummaryNotify wires the per-run digest. INIT-ONLY — call once at startup in main.go,
|
||||||
|
// alongside SetUnitNotify. Nil-safe, but an unwired seam is logged loudly rather than being silently
|
||||||
|
// the old behaviour.
|
||||||
|
func (m *Manager) SetRunSummaryNotify(fn func(RunSummary)) {
|
||||||
|
m.runSummaryNotify = fn
|
||||||
|
}
|
||||||
|
|
||||||
|
// admissionReason returns the human reason this run refused an app, taken from the verdict the
|
||||||
|
// reserve already recorded. Reused rather than re-derived: the verdict carries the bound term, the
|
||||||
|
// estimate and the disk figures, and re-deriving them here would be a second source of truth for a
|
||||||
|
// sentence the operator reads.
|
||||||
|
func (m *Manager) admissionReason(stackName string) string {
|
||||||
|
m.admissionMu.Lock()
|
||||||
|
defer m.admissionMu.Unlock()
|
||||||
|
if set := m.admission; set != nil {
|
||||||
|
if v, ok := set.v[stackName]; ok && v.err != nil {
|
||||||
|
return v.err.Error()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "refused by the capture reserve"
|
||||||
|
}
|
||||||
|
|
||||||
|
// runKindFor reports whether this run was the scheduled one or one a person triggered.
|
||||||
|
//
|
||||||
|
// The distinction is the operator's ruling: someone pressing the button is actively trying to get a
|
||||||
|
// backup, so a manual run must report even if the nightly one already wrote this hour. It is carried
|
||||||
|
// into the subject line because "the nightly run failed" and "the run I just asked for failed" are
|
||||||
|
// acted on differently.
|
||||||
|
func (m *Manager) runKindFor() string {
|
||||||
|
if m.manualRun.Load() {
|
||||||
|
return runKindManual
|
||||||
|
}
|
||||||
|
return runKindNightly
|
||||||
|
}
|
||||||
|
|
||||||
|
// MarkManualRun tags the NEXT backup run as operator-triggered. Called by the API/debug handlers
|
||||||
|
// that expose a "run backup now" control; the scheduled path leaves it alone.
|
||||||
|
func (m *Manager) MarkManualRun() { m.manualRun.Store(true) }
|
||||||
|
|
||||||
|
// newRunID returns a per-run identifier. It only has to be distinct between two runs on one box
|
||||||
|
// within the hub's 1-hour cooldown window, which a nanosecond clock reading satisfies; it is never
|
||||||
|
// persisted, compared across boxes, or used as a security value.
|
||||||
|
func newRunID() string {
|
||||||
|
return "run-" + strconv.FormatInt(time.Now().UnixNano(), 36)
|
||||||
|
}
|
||||||
@@ -0,0 +1,270 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"go/ast"
|
||||||
|
"go/parser"
|
||||||
|
"go/token"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-182 — one digest per run, listing every failure, and nothing in it that is not a failure.
|
||||||
|
//
|
||||||
|
// The assertions here are the CONTENT of the digest, not that a function was called: the defect this
|
||||||
|
// replaces was one where the machinery ran correctly and the operator was told about one app out of
|
||||||
|
// nine.
|
||||||
|
|
||||||
|
// digestOf runs the two legs that can run without Docker inside one digest scope and returns the
|
||||||
|
// summary that would have been sent (nil when none). It mirrors what runDBDumpsInternal does; the
|
||||||
|
// DB leg's wiring is pinned structurally by TestRunSummary_IsWiredIntoTheProductionPath, because
|
||||||
|
// DiscoverDatabases shells out to `docker`.
|
||||||
|
func (h *admissionHarness) digestOf(kind string) *RunSummary {
|
||||||
|
var got *RunSummary
|
||||||
|
h.m.SetRunSummaryNotify(func(rs RunSummary) { got = &rs })
|
||||||
|
doneAdm := h.m.beginAdmissionRun()
|
||||||
|
doneSum := h.m.beginRunSummary(kind, "run-test")
|
||||||
|
h.m.runVolumeDumps()
|
||||||
|
h.m.captureAllRecoveryUnits()
|
||||||
|
h.m.emitRunSummary()
|
||||||
|
doneSum()
|
||||||
|
doneAdm()
|
||||||
|
return got
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Scenario A — several failures, ONE digest, all of them in it ─────────────────────────────────
|
||||||
|
|
||||||
|
func TestRunSummary_ListsEveryFailedApp(t *testing.T) {
|
||||||
|
apps := []string{"opengist", "privatebin", "immich", "homebox", "nextcloud"}
|
||||||
|
h := newAdmissionHarness(t, apps...)
|
||||||
|
for _, a := range apps {
|
||||||
|
h.setSpace(a, 99, 0.1, 70) // every app refused by the reserve
|
||||||
|
}
|
||||||
|
|
||||||
|
rs := h.digestOf(runKindNightly)
|
||||||
|
if rs == nil {
|
||||||
|
t.Fatal("no digest was emitted although five apps failed — this is the 2026-08-03 defect: " +
|
||||||
|
"nine failures arrived and two e-mails went out")
|
||||||
|
}
|
||||||
|
if rs.Failed != len(apps) {
|
||||||
|
t.Fatalf("digest reports %d failures, want %d — the others were silently discarded, which is "+
|
||||||
|
"exactly what the per-app path did", rs.Failed, len(apps))
|
||||||
|
}
|
||||||
|
seen := map[string]bool{}
|
||||||
|
for _, a := range rs.Apps {
|
||||||
|
seen[a.App] = true
|
||||||
|
if a.Leg == "" {
|
||||||
|
t.Errorf("%s has no leg name — the operator cannot tell a failed database dump from a "+
|
||||||
|
"failed volume dump, and they are different problems", a.App)
|
||||||
|
}
|
||||||
|
if !strings.Contains(a.Reason, "reserve") {
|
||||||
|
t.Errorf("%s reason %q does not say why", a.App, a.Reason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, a := range apps {
|
||||||
|
if !seen[a] {
|
||||||
|
t.Errorf("%s is missing from the digest", a)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// The denominator: "5 of 5" and "5 of 40" are different nights.
|
||||||
|
if rs.Attempted < len(apps) {
|
||||||
|
t.Fatalf("attempted=%d, want >= %d — without the denominator a catastrophe and a bad night "+
|
||||||
|
"read identically", rs.Attempted, len(apps))
|
||||||
|
}
|
||||||
|
if rs.Usage == nil {
|
||||||
|
t.Fatal("the digest carries no filesystem figures — 'one broken app' and 'a full disk' must " +
|
||||||
|
"be distinguishable before the reasons are read")
|
||||||
|
}
|
||||||
|
if !strings.Contains(rs.Message, "of") || !strings.Contains(rs.Message, "nightly") {
|
||||||
|
t.Fatalf("summary message is not readable as a sentence: %q", rs.Message)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Scenario B — a clean run is silent ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
func TestRunSummary_CleanRunEmitsNothing(t *testing.T) {
|
||||||
|
h := newAdmissionHarness(t, "opengist", "privatebin")
|
||||||
|
h.setSpace("opengist", 20, 55, 70)
|
||||||
|
h.setSpace("privatebin", 20, 55, 70)
|
||||||
|
|
||||||
|
if rs := h.digestOf(runKindNightly); rs != nil {
|
||||||
|
t.Fatalf("a clean run emitted a digest (%+v) — an empty nightly mail is an unread mail within "+
|
||||||
|
"a week, and it would destroy the property this design rests on: that silence means the "+
|
||||||
|
"run finished and found nothing wrong", rs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Scenario F — deliberate skips are NOT failures ───────────────────────────────────────────────
|
||||||
|
|
||||||
|
// A drive that is unplugged or decommissioned has its own alert. Putting it in the digest produces a
|
||||||
|
// nightly e-mail on a box with an unplugged drive, which trains the operator to ignore digests — and
|
||||||
|
// an ignored digest is the same outcome as no digest.
|
||||||
|
func TestRunSummary_DeliberateSkipsAreNotFailures(t *testing.T) {
|
||||||
|
h := newAdmissionHarness(t, "gone", "retired", "healthy")
|
||||||
|
for _, a := range []string{"gone", "retired", "healthy"} {
|
||||||
|
h.setSpace(a, 20, 55, 70) // ample room — nothing may be refused for headroom
|
||||||
|
}
|
||||||
|
h.markDisconnected("gone")
|
||||||
|
h.markDecommissioned("retired")
|
||||||
|
|
||||||
|
rs := h.digestOf(runKindNightly)
|
||||||
|
if rs != nil {
|
||||||
|
t.Fatalf("a run whose only 'issues' were a disconnected drive and a decommissioned one "+
|
||||||
|
"emitted a digest: %+v — those have their own alerts", rs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── The collector's lifetime — the admission.go rule, restated ───────────────────────────────────
|
||||||
|
|
||||||
|
// An absent collector must mean "no run in flight", never a stale answer from last night.
|
||||||
|
func TestRunSummary_ResetsBetweenRuns(t *testing.T) {
|
||||||
|
h := newAdmissionHarness(t, "opengist")
|
||||||
|
h.setSpace("opengist", 99, 0.1, 70)
|
||||||
|
if rs := h.digestOf(runKindNightly); rs == nil || rs.Failed != 1 {
|
||||||
|
t.Fatalf("run 1: want 1 failure, got %+v", rs)
|
||||||
|
}
|
||||||
|
h.setSpace("opengist", 20, 55, 70) // space freed between runs
|
||||||
|
if rs := h.digestOf(runKindNightly); rs != nil {
|
||||||
|
t.Fatalf("run 2 reported %+v — the previous run's failures were carried over, so freeing "+
|
||||||
|
"space could never take effect", rs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A failure noted with NO run in flight must not panic and must not accumulate anywhere.
|
||||||
|
func TestRunSummary_NoteOutsideARunIsInert(t *testing.T) {
|
||||||
|
h := newAdmissionHarness(t, "opengist")
|
||||||
|
h.m.noteFailure("opengist", "volume dump", "boom") // no scope open
|
||||||
|
h.m.noteAttempted("opengist")
|
||||||
|
h.m.emitRunSummary() // must be a no-op, not a nil dereference
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── The refresh sweep carries NO run id, deliberately ────────────────────────────────────────────
|
||||||
|
|
||||||
|
// The run digest gets a unique id so the hub's 1-hour cooldown can never collapse two real runs. The
|
||||||
|
// periodic sweep is the opposite case — it can fire on every status poll — so it must fall UNDER the
|
||||||
|
// cooldown. Getting this backwards turns a full disk into a mail flood, which is the same failure as
|
||||||
|
// the one being fixed, arriving from the other side.
|
||||||
|
func TestRunSummary_RefreshSweepHasNoRunID(t *testing.T) {
|
||||||
|
h := newAdmissionHarness(t, "opengist")
|
||||||
|
h.setSpace("opengist", 99, 0.1, 70)
|
||||||
|
|
||||||
|
var got *RunSummary
|
||||||
|
h.m.SetRunSummaryNotify(func(rs RunSummary) { got = &rs })
|
||||||
|
func() {
|
||||||
|
defer h.m.beginAdmissionRun()()
|
||||||
|
defer h.m.beginRunSummary(runKindRefresh, "")()
|
||||||
|
defer h.m.emitRunSummary()
|
||||||
|
h.m.captureAllRecoveryUnits()
|
||||||
|
}()
|
||||||
|
if got == nil {
|
||||||
|
t.Fatal("the periodic sweep emitted no digest — with the per-app event now record-only, a " +
|
||||||
|
"capture failure found between runs would be recorded and NEVER notified, which is a " +
|
||||||
|
"new silence introduced while closing one")
|
||||||
|
}
|
||||||
|
if got.RunID != "" {
|
||||||
|
t.Fatalf("the refresh sweep carries run_id=%q — it must be EMPTY so the hub's ordinary "+
|
||||||
|
"1-hour cooldown caps it, or a polled status page becomes a mail flood", got.RunID)
|
||||||
|
}
|
||||||
|
if got.RunKind != runKindRefresh {
|
||||||
|
t.Fatalf("run kind = %q, want %q", got.RunKind, runKindRefresh)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── The seam is WIRED — walked as an AST, not grepped ────────────────────────────────────────────
|
||||||
|
|
||||||
|
// Four mechanisms in this project have been built and left disconnected. `strings.Contains` cannot
|
||||||
|
// tell a live call from a commented-out one, so this parses.
|
||||||
|
func TestRunSummary_IsWiredIntoTheProductionPath(t *testing.T) {
|
||||||
|
calls := callsByFunc(t, "backup.go")
|
||||||
|
|
||||||
|
if !hasStr(calls["runDBDumpsInternal"], "beginRunSummary") {
|
||||||
|
t.Fatal("runDBDumpsInternal does not open a run-summary scope — every failure would be " +
|
||||||
|
"recorded per app and none of them summarised, which is the pre-R-182 behaviour")
|
||||||
|
}
|
||||||
|
if !hasStr(calls["runDBDumpsInternal"], "emitRunSummary") {
|
||||||
|
t.Fatal("runDBDumpsInternal never emits the summary — the collector fills and is discarded")
|
||||||
|
}
|
||||||
|
// Both dump legs and the capture leg must feed it, or a whole class of failure is invisible.
|
||||||
|
for fn, where := range map[string]string{
|
||||||
|
"runDBDumpsInternal": "the database leg",
|
||||||
|
"runVolumeDumps": "the volume leg",
|
||||||
|
} {
|
||||||
|
if !hasStr(calls[fn], "noteFailure") {
|
||||||
|
t.Fatalf("%s (%s) never calls noteFailure — its failures cannot reach the digest", fn, where)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
capCalls := callsByFunc(t, "recovery_unit.go")
|
||||||
|
if !hasStr(capCalls["captureAllRecoveryUnits"], "noteFailure") {
|
||||||
|
t.Fatal("the capture leg never calls noteFailure")
|
||||||
|
}
|
||||||
|
|
||||||
|
// AND THE SEAM ITSELF must be wired in main.go. Without this assertion the collector fills, the
|
||||||
|
// digest is built, and `runSummaryNotify` is nil — so nothing is ever sent and every test above
|
||||||
|
// still passes. That is the built-but-never-wired shape exactly, and the first version of this
|
||||||
|
// test missed it: commenting the wiring out left the suite green.
|
||||||
|
mainCalls := callsByFunc(t, "../../cmd/controller/main.go")
|
||||||
|
wired := false
|
||||||
|
for _, calls := range mainCalls {
|
||||||
|
if hasStr(calls, "SetRunSummaryNotify") {
|
||||||
|
wired = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !wired {
|
||||||
|
t.Fatal("main.go never calls SetRunSummaryNotify — the digest is assembled and handed to a " +
|
||||||
|
"nil seam, so no operator mail is ever sent. Note this is an AST walk: a commented-out " +
|
||||||
|
"call still CONTAINS the string, which is why strings.Contains cannot be used here")
|
||||||
|
}
|
||||||
|
|
||||||
|
// The manual paths must tag the run, or Scenario E cannot hold.
|
||||||
|
for _, f := range []string{"../web/handler_debug.go", "../api/router.go"} {
|
||||||
|
src := parseFile(t, f)
|
||||||
|
found := false
|
||||||
|
ast.Inspect(src, func(n ast.Node) bool {
|
||||||
|
if ce, ok := n.(*ast.CallExpr); ok {
|
||||||
|
if sel, ok := ce.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "MarkManualRun" {
|
||||||
|
found = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
})
|
||||||
|
if !found {
|
||||||
|
t.Fatalf("%s never calls MarkManualRun — an operator-triggered run would be labelled "+
|
||||||
|
"nightly and could be collapsed into it", f)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseFile(t *testing.T, path string) *ast.File {
|
||||||
|
t.Helper()
|
||||||
|
f, err := parser.ParseFile(token.NewFileSet(), path, nil, 0) // comments dropped
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
|
||||||
|
// callsByFunc maps each top-level func to the names it calls, comments excluded.
|
||||||
|
func callsByFunc(t *testing.T, path string) map[string][]string {
|
||||||
|
t.Helper()
|
||||||
|
file := parseFile(t, path)
|
||||||
|
out := map[string][]string{}
|
||||||
|
var cur string
|
||||||
|
ast.Inspect(file, func(n ast.Node) bool {
|
||||||
|
switch v := n.(type) {
|
||||||
|
case *ast.FuncDecl:
|
||||||
|
cur = v.Name.Name
|
||||||
|
case *ast.CallExpr:
|
||||||
|
name := ""
|
||||||
|
switch fn := v.Fun.(type) {
|
||||||
|
case *ast.Ident:
|
||||||
|
name = fn.Name
|
||||||
|
case *ast.SelectorExpr:
|
||||||
|
name = fn.Sel.Name
|
||||||
|
}
|
||||||
|
if name != "" && cur != "" {
|
||||||
|
out[cur] = append(out[cur], name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
})
|
||||||
|
return out
|
||||||
|
}
|
||||||
@@ -330,6 +330,44 @@ func (n *Notifier) NotifyRecoveryUnitCaptureFailed(message string, d RecoveryUni
|
|||||||
n.PushEvent("recovery_unit_capture_failed", "error", message, d)
|
n.PushEvent("recovery_unit_capture_failed", "error", message, d)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RunFailureDetail is one app's failed leg inside a backup run digest.
|
||||||
|
type RunFailureDetail struct {
|
||||||
|
App string `json:"app"`
|
||||||
|
Leg string `json:"leg"`
|
||||||
|
Reason string `json:"reason"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// BackupRunFailuresDetails is the per-RUN digest payload (R-182). App NAMES, leg names, reasons and
|
||||||
|
// byte figures only — never an env value (§9.4).
|
||||||
|
type BackupRunFailuresDetails struct {
|
||||||
|
// RunID makes the hub's 1-hour operator cooldown unable to collapse two real runs into one
|
||||||
|
// e-mail. EMPTY on the periodic refresh sweep, deliberately: that path can fire on every status
|
||||||
|
// poll, so it must fall under the ordinary cooldown instead.
|
||||||
|
RunID string `json:"run_id,omitempty"`
|
||||||
|
RunKind string `json:"run_kind"`
|
||||||
|
Failed int `json:"failed"`
|
||||||
|
Attempted int `json:"attempted"`
|
||||||
|
TargetPath string `json:"target_path,omitempty"`
|
||||||
|
UsedGB float64 `json:"used_gb,omitempty"`
|
||||||
|
AvailGB float64 `json:"avail_gb,omitempty"`
|
||||||
|
TotalGB float64 `json:"total_gb,omitempty"`
|
||||||
|
UsedPercent float64 `json:"used_percent,omitempty"`
|
||||||
|
SpaceKnown bool `json:"space_known"`
|
||||||
|
Apps []RunFailureDetail `json:"apps"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// NotifyBackupRunFailures sends the ONE operator digest for a backup run in which something failed
|
||||||
|
// (R-182). It is the NOTIFICATION; the per-app `recovery_unit_capture_failed` events are the RECORD,
|
||||||
|
// and the hub routes those record-only so they never compete for an e-mail slot.
|
||||||
|
//
|
||||||
|
// OPERATOR-TIER, and for the same reason as its per-app sibling: a customer can act on a full disk
|
||||||
|
// (that is the fill warning, which fires first and IS customer-facing) but not on a list of which
|
||||||
|
// apps' backups failed and why. `notify.operatorOnlyEvents` in the hub enforces that — NOT the
|
||||||
|
// absence of a customerMessages entry, which is a fallback rather than a block (v0.78.0).
|
||||||
|
func (n *Notifier) NotifyBackupRunFailures(message string, d BackupRunFailuresDetails) {
|
||||||
|
n.PushEvent("backup_run_failures", "error", message, d)
|
||||||
|
}
|
||||||
|
|
||||||
// NotifyOffboxEnlargeBlocked sends a WARNING (not a failure) when an app's enlarged offsite push was
|
// NotifyOffboxEnlargeBlocked sends a WARNING (not a failure) when an app's enlarged offsite push was
|
||||||
// refused by the pre-push quota gate — its config+DB were still saved. Customer-facing (Hungarian
|
// refused by the pre-push quota gate — its config+DB were still saved. Customer-facing (Hungarian
|
||||||
// body). NOTE: the event type "offbox_enlarge_blocked" must be added to the hub's allowedEventTypes +
|
// body). NOTE: the event type "offbox_enlarge_blocked" must be added to the hub's allowedEventTypes +
|
||||||
|
|||||||
@@ -379,6 +379,7 @@ func (s *Server) debugTriggerDBDump(w http.ResponseWriter, r *http.Request) {
|
|||||||
writeDebugJSON(w, http.StatusBadRequest, false, "Backup manager nincs konfigurálva", nil)
|
writeDebugJSON(w, http.StatusBadRequest, false, "Backup manager nincs konfigurálva", nil)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
s.backupMgr.MarkManualRun() // R-182: a person pressed this, so its digest must not be collapsed
|
||||||
go func() {
|
go func() {
|
||||||
if err := s.backupMgr.RunDBDumps(context.Background()); err != nil {
|
if err := s.backupMgr.RunDBDumps(context.Background()); err != nil {
|
||||||
s.logger.Printf("[WARN] Debug DB dump failed: %v", err)
|
s.logger.Printf("[WARN] Debug DB dump failed: %v", err)
|
||||||
|
|||||||
Reference in New Issue
Block a user