From 4be6467b501b7868a312d49357bc51e80ad8e93b Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 3 Aug 2026 06:30:19 +0200 Subject: [PATCH] =?UTF-8?q?v0.192.0=20=E2=80=94=20the=20capture=20floor=20?= =?UTF-8?q?replaces=20the=20bulkhead=20(R-165,=20decision=20B2)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ships BEFORE the disk-layout merge it exists for, and is harmless on a box that never gets it. The mp1 partition was a BULKHEAD as well as a ceiling: it kept a runaway capture from filling the space the container runtime needs, because /var/lib/docker was a different filesystem. After the merge it is the same one, and a full Docker data-root is a stopped box. The floor sits in captureAllRecoveryUnits, checked BEFORE anything is written: below the reserve, that ONE app's capture is refused, its previous unit is left byte-identical, the R-158 alert fires with the space figures, and the loop continues. Two terms whichever binds first (97% used / 1 GiB free) in fillwatch's shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned before a refusal can happen — a floor that fires before its own warning is a silent failure wearing a threshold. Headroom, never unit size: a per-unit cap would be R-163 rebuilt inside one volume. Refuses, never deletes: nothing here is generational, so pruning could only destroy a different app's only local copy; pruneStalePrimaryDirs is an orphan sweep, not retention, and must not be repurposed. Tests 1184 -> 1191. One fixture strengthened mid-red-proof: the "old 20 G ceiling is gone" test sat at exactly 20 GB and survived a literal UsedGB > 20 cap — hollow. Now 120 GB, and the mutation fails it. --- CHANGELOG.md | 45 +++ REUSE.md | 1 + controller/internal/backup/backup.go | 4 + .../internal/backup/capture_floor_test.go | 304 ++++++++++++++++++ controller/internal/backup/recovery_unit.go | 83 ++++- 5 files changed, 436 insertions(+), 1 deletion(-) create mode 100644 controller/internal/backup/capture_floor_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 21ebf48..70eef7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,50 @@ ## Changelog +### v0.192.0 — the capture floor replaces the bulkhead (2026-08-03, R-165 · decision B2) — MinAgent: none + +**Ships BEFORE the disk-layout merge it exists for, and is harmless on a box that never gets it.** +The `mp1`→`mp0` merge (R-165 / decision D-a) removes a wall that was quietly doing a second job: the +20 G backup partition kept a runaway recovery-unit capture from filling the space the container +runtime itself needs, because `/var/lib/docker` was a **different filesystem**. After the merge it is +the same one, and a full Docker data-root is a stopped box, not a slow one. Decision **B2** is that +bulkhead, done deliberately instead of by accident. + +**The floor, in `captureAllRecoveryUnits`, checked BEFORE anything is written.** If the target +filesystem is below the reserve, that **one app's** capture is refused, its previous unit is left +**byte-identical**, the operator alert wired in v0.191.0 fires with the used/free figures, and the +loop continues to the next app. + +**Two terms, whichever binds first — 97% used or 1 GiB free** — the same shape as `internal/fillwatch`, +which proved live on 2026-08-02 that a percentage alone is not enough (its critical alert fired on the +free-byte term at 91% used, where a percent-only rule stayed silent). + +**They sit deliberately BEYOND fillwatch's critical band (95% / 2 GiB), so the customer is ALWAYS +warned before a refusal can happen.** A floor that fires before its own warning is a silent failure +wearing a threshold; `TestFloorSitsBelowTheCriticalWarningBand` pins the whole ordering +(warn → critical → refuse) on both terms, and a red-proof setting the floor equal to the critical band +fails it. + +**IT IS ABOUT THE FILESYSTEM'S HEADROOM, NEVER THE UNIT'S SIZE.** A per-unit cap would be R-163 rebuilt +inside one volume — the wall moved rather than removed — so a 120 GB app on a filesystem with 180 GB +free is captured. A red-proof substituting `UsedGB > 20` for the headroom predicate fails two tests. + +**IT REFUSES; IT NEVER DELETES, and the reason is recorded because the question will be asked again.** +Nothing on this filesystem is generational: a unit is ONE fixed path per app +(`backups/primary/`) refreshed in place, and a DB dump is `-.sql`, also fixed. So +"prune the oldest" could only mean deleting a **different** app's only local recovery unit to make room +for this one. `pruneStalePrimaryDirs` is **not** a retention policy — it removes ORPHANED directories +left when an app moves drives and has no notion of age — and must never be repurposed here. + +**A nil usage read neither refuses nor warns** (§8.4): an unreadable filesystem is the drive gate's +business and already has its own alert, and refusing on it would block every capture on a box whose +drive merely blipped. + +**Tests:** 1184 → **1191** (+7). New `unitSpaceFn` seam so a filesystem's occupancy is a test input +rather than something a test must manufacture on a real disk. One fixture was **strengthened** during +the red-proofs: `TestFloor_TheOld20GCeilingIsGone` originally sat at exactly 20 GB and therefore +survived a literal `UsedGB > 20` cap — a hollow test that passed the very shape it forbids. Its +figure is now 120 GB and the mutation fails it. + ### v0.191.2 — a quiet fill check now says so (2026-08-02, R-167) — MinAgent: none **Earned during v0.191.1's own live validation, which is the strongest evidence it was needed.** After diff --git a/REUSE.md b/REUSE.md index b8678b2..18c4553 100644 --- a/REUSE.md +++ b/REUSE.md @@ -174,6 +174,7 @@ | `api.GracefulSelfRestart` | controller/internal/api/selfrestart.go | `(logger)` | Controller self-restart | Detached exit; bootstrap unit re-runs the image | | `Settings.AddPendingEvent/DrainPendingEvents` | controller/internal/settings/settings.go | offline event queue | Events while hub unreachable | — | | `Manager.SetUnitNotify` + `UnitSpace` (R-158/R-167, v0.191.0) | controller/internal/backup/recovery_unit.go | `(func(stack string, err error, *UnitSpace))` | THE per-app Tier-1 recovery-unit capture failure alert — fires PER APP from `captureAllRecoveryUnits`, loop continues | **OPERATOR-TIER** (`recovery_unit_capture_failed`, in the hub's `operatorOnlyEvents`). **NEVER route it to `backup_failed`** — that type is in `DefaultEnabledEvents` and carries Hungarian copy, so it emails the CUSTOMER about a failure they cannot act on (D-c; R-158's own proposal said `backup_failed` and D-c overrides it). `UnitSpace` is **nil when the target filesystem is unreadable** and renders as *"unavailable"*, never as zeros — "0 GB free" and "we could not look" are opposite diagnoses. No controller-side cooldown: the hub owns it | +| `Manager.unitFloorBlocked` / `floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` (R-165 B2, v0.192.0) | controller/internal/backup/recovery_unit.go | `(stackName) (*UnitSpace, bool)` / `(*UnitSpace) (*UnitSpace, bool)` | THE capture floor that replaces the `mp1` bulkhead — refuses ONE app's recovery-unit capture before anything is written when the target filesystem is below the reserve | **Headroom, NEVER unit size** — a per-unit cap is R-163 rebuilt inside one volume. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk | | `fillwatch.Watcher` (`New`/`SetNotify`/`Check`) (R-167, v0.191.0) | controller/internal/fillwatch/fillwatch.go | `(statePath, logger, targetsFn, usageFn)` → `Check() error` | THE customer fill warning — warns BEFORE a filesystem fills, per FILESYSTEM (never per app: one full disk holding ten apps would fire ten times) | Emits the **pre-existing** `disk_warning`/`disk_critical` pair, which was allowlisted + copy'd + default-enabled with **no producer in any repo** until now — do NOT mint a new type beside it. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range. **Edge-triggered on ESCALATION ONLY**, state persisted; de-escalation is silent and re-arms. Hysteresis dead zone between clear (75% / 7 GiB) and warn — pinned by `TestThresholdsKeepTheirHysteresisGap`. **A nil usage read is NEVER a warning** (§8.4). The hub has **no `customerMessages` entry** for either type on purpose — an entry would override the dynamic message and discard the drive label + free space | ### Scheduler / time / UI diff --git a/controller/internal/backup/backup.go b/controller/internal/backup/backup.go index 6d1fc5a..9633028 100644 --- a/controller/internal/backup/backup.go +++ b/controller/internal/backup/backup.go @@ -53,6 +53,10 @@ type Manager struct { // precedent. unitNotify func(stackName string, err error, usage *UnitSpace) + // unitSpaceFn (R-165 / B2), if set, replaces the real statfs behind the capture floor so a test + // can state a filesystem's occupancy as an input. Nil in production → `unitTargetSpace`. + unitSpaceFn func(stackName string) *UnitSpace + // appStop (R-166) is the crash marker for operations that stop an app, work on its data, and // start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power // cut in that window leaves a durable record that Recover honours at the next startup. Built in diff --git a/controller/internal/backup/capture_floor_test.go b/controller/internal/backup/capture_floor_test.go new file mode 100644 index 0000000..558572d --- /dev/null +++ b/controller/internal/backup/capture_floor_test.go @@ -0,0 +1,304 @@ +package backup + +import ( + "crypto/sha256" + "encoding/hex" + "errors" + "io" + "log" + "os" + "path/filepath" + "strings" + "testing" + + "gitea.dooplex.hu/admin/felhom-controller/internal/fillwatch" +) + +// R-165 / decision B2 — the capture floor that replaces the `mp1` bulkhead. +// +// Before the merge, the 20 G backup partition kept a runaway capture from reaching +// `/var/lib/docker`, because it was a different filesystem. After the merge it is the same one, and a +// full Docker data-root is a stopped box. These pin the replacement. + +// floorProvider lists stacks and always resolves recovery info — the floor must refuse BEFORE any of +// that is consulted, so a capture that gets as far as GetStackRecoveryInfo has already lost. +type floorProvider struct { + stacks []string + dir string + infoHits []string // records every app whose recovery info was read = a capture that was ATTEMPTED +} + +func (p *floorProvider) GetStackComposePath(string) (string, bool) { return "", false } +func (p *floorProvider) ListDeployedStacks() []StackSummary { + out := make([]StackSummary, 0, len(p.stacks)) + for _, s := range p.stacks { + out = append(out, StackSummary{Name: s}) + } + return out +} +func (p *floorProvider) GetStackHDDMounts(string) []string { return nil } +func (p *floorProvider) GetStackHDDPath(string) string { return "" } +func (p *floorProvider) GetImportRoot() string { return "" } +func (p *floorProvider) GetDockerVolumes(string) []string { return nil } +func (p *floorProvider) StopStack(string) error { return nil } +func (p *floorProvider) StartStack(string) error { return nil } +func (p *floorProvider) RefreshAndIsRunning(string) bool { return true } +func (p *floorProvider) GetStackRecoveryInfo(name string) (RecoveryInfo, bool) { + p.infoHits = append(p.infoHits, name) + return RecoveryInfo{StackDir: filepath.Join(p.dir, "stacks", name)}, true +} +func (p *floorProvider) RecoverStackSecrets(string, []string) map[string]string { return nil } +func (p *floorProvider) RecreateStackDefinitionFromUnit(string, string, map[string]string) error { + return nil +} +func (p *floorProvider) StartStackServices(string, []string) error { return nil } +func (p *floorProvider) GetStackClassifiedBinds(string) ([]ClassifiedBind, bool) { + return nil, false +} + +type floorHarness struct { + m *Manager + prov *floorProvider + events []unitEvent + usage map[string]*UnitSpace + dir string +} + +// newFloorHarness injects the usage read, so the filesystem's occupancy is a test input rather than +// something the test has to manufacture on a real disk. +func newFloorHarness(t *testing.T, stacks ...string) *floorHarness { + t.Helper() + dir := t.TempDir() + h := &floorHarness{ + prov: &floorProvider{stacks: stacks, dir: dir}, + usage: map[string]*UnitSpace{}, + dir: dir, + } + h.m = &Manager{ + logger: log.New(io.Discard, "", 0), + systemDataPath: dir, + stackProvider: h.prov, + unitSpaceFn: func(name string) *UnitSpace { return h.usage[name] }, + } + h.m.SetUnitNotify(func(name string, err error, u *UnitSpace) { + h.events = append(h.events, unitEvent{app: name, err: err.Error(), usage: u}) + }) + return h +} + +func (h *floorHarness) setSpace(app string, usedPct, availGB float64) { + h.usage[app] = &UnitSpace{ + Path: h.dir, UsedPercent: usedPct, AvailGB: availGB, + TotalGB: 100, UsedGB: usedPct, + } +} + +// --- Scenario D — the floor refuses, per app, and says so ---------------------------------------- + +func TestFloor_RefusesTheAppAndLeavesItsPreviousUnitByteIdentical(t *testing.T) { + h := newFloorHarness(t, "homebox", "immich", "nextcloud") + h.setSpace("homebox", 40, 60) + h.setSpace("immich", 98, 0.4) // below the floor on BOTH terms + h.setSpace("nextcloud", 40, 60) + + // A previous unit exists for the app about to be refused. Checksum it before and after. + unitDir := filepath.Join(h.dir, "felhom-data", "backups", "primary", "immich", "compose") + if err := os.MkdirAll(unitDir, 0o755); err != nil { + t.Fatal(err) + } + prev := filepath.Join(unitDir, "app.yaml") + if err := os.WriteFile(prev, []byte("deployed: true\nenv:\n A: previous-good-value\n"), 0o600); err != nil { + t.Fatal(err) + } + before := checksumFile(t, prev) + + h.m.captureAllRecoveryUnits() + + // The refused app must NOT have been attempted at all — the floor is checked BEFORE any write. + for _, hit := range h.prov.infoHits { + if hit == "immich" { + t.Fatal("the refused app's recovery info was read — the capture was ATTEMPTED rather than " + + "refused up front, so a write could have started and failed partway") + } + } + + if after := checksumFile(t, prev); after != before { + t.Fatalf("the previous unit changed (%s → %s) — a refused capture must leave the last good "+ + "copy byte-identical", before, after) + } + if _, err := os.Stat(prev); err != nil { + t.Fatalf("the previous unit is gone: %v — the floor REFUSES, it never deletes", err) + } + + // Exactly one alert, for the refused app, carrying the space figures. + if len(h.events) != 1 { + t.Fatalf("got %d alerts, want exactly 1: %+v", len(h.events), h.events) + } + e := h.events[0] + if e.app != "immich" { + t.Fatalf("alert names %q, want immich", e.app) + } + if e.usage == nil || e.usage.AvailGB != 0.4 { + t.Fatalf("the alert carries no/incorrect space figures: %+v", e.usage) + } + if !strings.Contains(e.err, "reserve") { + t.Fatalf("the alert message %q does not say it was a reserve refusal — an operator would read "+ + "it as a broken capture rather than a deliberate hold", e.err) + } + + // The other two must have been captured normally — one app's refusal must not silence its siblings. + got := strings.Join(h.prov.infoHits, ",") + if !strings.Contains(got, "homebox") || !strings.Contains(got, "nextcloud") { + t.Fatalf("attempted=%v — the loop did not continue past the refusal", h.prov.infoHits) + } +} + +// Nothing may be deleted to make room, under any threshold. Nothing on this filesystem is +// generational, so "the oldest" is always a DIFFERENT app's only local copy. +func TestFloor_NeverDeletesAnotherAppsUnit(t *testing.T) { + h := newFloorHarness(t, "immich", "nextcloud") + h.setSpace("immich", 99, 0.1) + h.setSpace("nextcloud", 99, 0.1) + + other := filepath.Join(h.dir, "felhom-data", "backups", "primary", "nextcloud") + if err := os.MkdirAll(other, 0o755); err != nil { + t.Fatal(err) + } + keep := filepath.Join(other, "manifest.json") + if err := os.WriteFile(keep, []byte(`{"app_name":"nextcloud"}`), 0o644); err != nil { + t.Fatal(err) + } + before := checksumFile(t, keep) + + h.m.captureAllRecoveryUnits() + + if _, err := os.Stat(keep); err != nil { + t.Fatalf("another app's unit was DELETED to make room: %v — nothing here is generational, so "+ + "pruning could only destroy an app's only local copy", err) + } + if after := checksumFile(t, keep); after != before { + t.Fatal("another app's unit was modified while the filesystem was under the floor") + } +} + +// --- Scenario E — the floor is not a wall by another name ---------------------------------------- + +// The floor is about the FILESYSTEM's remaining headroom, never the unit's size. A per-unit cap would +// be R-163 rebuilt inside one volume. +func TestFloor_LargeUnitWithAmpleSpaceIsCaptured(t *testing.T) { + h := newFloorHarness(t, "immich") + // A huge app on a huge, mostly-empty filesystem: 40% used, 600 GB free. + h.usage["immich"] = &UnitSpace{Path: h.dir, UsedPercent: 40, AvailGB: 600, TotalGB: 1000, UsedGB: 400} + + h.m.captureAllRecoveryUnits() + + if len(h.events) != 0 { + t.Fatalf("a capture was refused on a filesystem with 600 GB free (%+v) — the floor has become "+ + "a per-unit size cap, which is exactly the ceiling R-165 removed", h.events) + } + if len(h.prov.infoHits) != 1 || h.prov.infoHits[0] != "immich" { + t.Fatalf("attempted=%v, want [immich] — the capture was not even tried", h.prov.infoHits) + } +} + +// The old 20 G ceiling must not survive anywhere: a unit far larger than the retired partition is +// captured when the filesystem has room. +func TestFloor_TheOld20GCeilingIsGone(t *testing.T) { + h := newFloorHarness(t, "immich") + // 180 GB free, and the app's own data is 120 GB — SIX TIMES the retired 20 G area. The figure is + // deliberately far above 20 so that a literal `UsedGB > 20` cap cannot survive this test: a + // fixture sitting exactly on the old boundary would pass under the very shape it forbids. + h.usage["immich"] = &UnitSpace{Path: h.dir, UsedPercent: 40, AvailGB: 180, TotalGB: 300, UsedGB: 120} + h.m.captureAllRecoveryUnits() + if len(h.events) != 0 { + t.Fatalf("refused with 180 GB free: %+v — a fixed per-area limit survives somewhere", h.events) + } +} + +// --- Group E — the floor sits BELOW the critical warning band ------------------------------------- + +// A floor that fires before its own warning is a silent failure wearing a threshold: the customer +// would get a refusal with no prior notice that anything was wrong. The customer's `disk_critical` +// must always come first. +func TestFloorSitsBelowTheCriticalWarningBand(t *testing.T) { + if FloorUsedPercent <= fillwatch.CritUsedPercent { + t.Fatalf("FloorUsedPercent (%.1f) must be strictly ABOVE fillwatch.CritUsedPercent (%.1f) — "+ + "otherwise a capture can be refused before the customer was ever warned that the disk was "+ + "filling, which is a silent failure wearing a threshold", + FloorUsedPercent, fillwatch.CritUsedPercent) + } + if FloorFreeGiB >= fillwatch.CritFreeGiB { + t.Fatalf("FloorFreeGiB (%.1f) must be strictly BELOW fillwatch.CritFreeGiB (%.1f) — the "+ + "free-byte term needs the same ordering as the percentage term, or the free-byte path "+ + "refuses before it warns", FloorFreeGiB, fillwatch.CritFreeGiB) + } + // And below the WARNING band too, transitively — stated explicitly so the chain is readable. + if FloorUsedPercent <= fillwatch.WarnUsedPercent || FloorFreeGiB >= fillwatch.WarnFreeGiB { + t.Fatal("the floor is not beyond the warning band — the customer must be warned, then warned " + + "critically, and only then can a capture be refused") + } + + // Both terms must be able to refuse INDEPENDENTLY — that is why there are two. + if _, blocked := (&Manager{}).floorVerdict(&UnitSpace{UsedPercent: 50, AvailGB: 0.5}); !blocked { + t.Fatal("a filesystem with 0.5 GiB free at only 50% used was NOT refused — the free-byte term " + + "does not trip on its own, so a very large volume can run out without the floor engaging") + } + if _, blocked := (&Manager{}).floorVerdict(&UnitSpace{UsedPercent: 98, AvailGB: 40}); !blocked { + t.Fatal("a filesystem 98% used was NOT refused — the percentage term does not trip on its own") + } + if _, blocked := (&Manager{}).floorVerdict(&UnitSpace{UsedPercent: 50, AvailGB: 50}); blocked { + t.Fatal("a healthy filesystem was refused") + } +} + +// --- §8.4 — a nil usage read neither refuses nor warns -------------------------------------------- + +func TestFloor_UnreadableFilesystemNeitherRefusesNorWarns(t *testing.T) { + h := newFloorHarness(t, "immich") + // No entry → the injected reader returns nil, which is what system.GetDiskUsage does on error. + + h.m.captureAllRecoveryUnits() + + if len(h.events) != 0 { + t.Fatalf("an UNREADABLE filesystem produced %d alert(s): %+v — an absent, unmounted or "+ + "unreadable filesystem is the drive gate's business and already has its own alert; "+ + "refusing here would block every capture on a box whose drive merely blipped", len(h.events), h.events) + } + if len(h.prov.infoHits) != 1 { + t.Fatalf("the capture was not attempted on an unreadable read (attempted=%v) — a nil reading "+ + "must not refuse", h.prov.infoHits) + } +} + +// ErrCaptureFloor must be matchable, so a caller can tell a deliberate refusal from a broken capture. +func TestErrCaptureFloor_IsMatchable(t *testing.T) { + h := newFloorHarness(t, "immich") + h.setSpace("immich", 99, 0.2) + h.m.captureAllRecoveryUnits() + if len(h.events) != 1 { + t.Fatalf("want 1 event, got %d", len(h.events)) + } + // The seam hands a string, so assert on the sentinel's own text being present and distinct. + if !errors.Is(errWrapForTest(), ErrCaptureFloor) { + t.Fatal("ErrCaptureFloor does not survive wrapping") + } + if !strings.Contains(h.events[0].err, "Refused") && !strings.Contains(h.events[0].err, "refused") { + t.Fatalf("the alert %q does not identify itself as a refusal", h.events[0].err) + } +} + +func errWrapForTest() error { return errors.Join(ErrCaptureFloor, errors.New("ctx")) } + +func checksumFile(t *testing.T, path string) string { + t.Helper() + f, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + defer f.Close() + h := sha256.New() + if _, err := io.Copy(h, f); err != nil { + t.Fatal(err) + } + return hex.EncodeToString(h.Sum(nil)) +} diff --git a/controller/internal/backup/recovery_unit.go b/controller/internal/backup/recovery_unit.go index 2d8a3ec..605b051 100644 --- a/controller/internal/backup/recovery_unit.go +++ b/controller/internal/backup/recovery_unit.go @@ -4,6 +4,7 @@ import ( "crypto/sha256" "encoding/hex" "encoding/json" + "errors" "fmt" "io" "os" @@ -243,8 +244,76 @@ func (m *Manager) unitTargetSpace(stackName string) *UnitSpace { } } +// ── The capture floor (R-165 / decision B2) ────────────────────────────────────────────────────── +// +// WHAT IT REPLACES. Until the `mp1`→`mp0` merge, the 20 G backup partition was a BULKHEAD as well as +// a ceiling: an app whose unit outgrew it was refused per app, its last good unit preserved +// byte-identical, and the overflow **could not reach `/var/lib/docker`** because that was a different +// filesystem. After the merge it can, and a full Docker data-root is a stopped box, not a slow one. +// This floor is that bulkhead, done deliberately instead of by accident. +// +// IT IS ABOUT THE FILESYSTEM'S HEADROOM, NEVER THE UNIT'S SIZE. A per-unit size cap would be R-163 +// rebuilt inside one volume — the wall moved rather than removed — so a large unit on a filesystem +// with ample room is captured, whatever its size. +// +// IT REFUSES; IT NEVER DELETES. Nothing on this filesystem is generational: a unit is ONE fixed path +// per app (`backups/primary/`) refreshed in place, and a DB dump is `-.sql`, also +// fixed. So "prune the oldest" could only mean deleting a DIFFERENT app's only local recovery unit to +// make room for this one, and that is not a trade this system makes. `pruneStalePrimaryDirs` is NOT a +// retention policy — it removes ORPHANED directories left when an app moves drives, and has no notion +// of age — so it must never be repurposed here. +const ( + // FloorUsedPercent / FloorFreeGiB — the reserve. Two terms, whichever binds first, the same shape + // as `internal/fillwatch` (proven live on 2026-08-02: the critical alert fired on the free-byte + // term at 91% used, where a percent-only rule stayed silent). + // + // THEY SIT DELIBERATELY BEYOND fillwatch's CRITICAL BAND (95% / 2 GiB), so the customer is ALWAYS + // warned before a refusal can happen. A floor that fires before its own warning is a silent + // failure wearing a threshold; `TestFloorSitsBelowTheCriticalWarningBand` pins the ordering. + // + // 1 GiB is the reserve, not a working budget: §7.5 measures a DB-backed app's unit at up to ~2× its + // data, so no fixed number can guarantee a capture fits. What this guarantees is different and is + // the bulkhead's actual job — that a capture cannot consume the last of the space the container + // runtime needs to keep running. + FloorUsedPercent = 97.0 + FloorFreeGiB = 1.0 +) + +// ErrCaptureFloor marks a capture refused for headroom. It is a REFUSAL, not a failure of the capture +// machinery — the distinction matters to a reader of the alert, which is why the message names the +// reserve rather than reporting an I/O error. +var ErrCaptureFloor = errors.New("refused: capturing would leave the filesystem below the reserve") + +// floorVerdict is the PURE predicate: given a reading, does the floor refuse? Separated so the +// thresholds are unit-testable without a filesystem, a stack provider or a clock. +// +// §8.4 — A NIL READING NEITHER REFUSES NOR WARNS. An unreadable filesystem is the drive gate's +// business and has its own alert; refusing on it would block every capture on a box whose drive +// merely blipped, and warning on it would be a false alarm with a misleading cause. +func (m *Manager) floorVerdict(u *UnitSpace) (*UnitSpace, bool) { + if u == nil { + return nil, false + } + return u, u.UsedPercent >= FloorUsedPercent || u.AvailGB < FloorFreeGiB +} + +// unitFloorBlocked reads the target filesystem and applies the floor. +func (m *Manager) unitFloorBlocked(stackName string) (*UnitSpace, bool) { + return m.floorVerdict(m.readUnitSpace(stackName)) +} + +// readUnitSpace goes through the seam when one is injected, so a test can state the filesystem's +// occupancy as an input instead of manufacturing it on a real disk. Nil seam → the real statfs. +func (m *Manager) readUnitSpace(stackName string) *UnitSpace { + if m.unitSpaceFn != nil { + return m.unitSpaceFn(stackName) + } + return m.unitTargetSpace(stackName) +} + // captureAllRecoveryUnits refreshes the recovery unit for every deployed stack. Best-effort: -// a per-app failure is logged, NOTIFIED (R-158), and does not abort the others. +// a per-app failure is logged, NOTIFIED (R-158), and does not abort the others. Since R-165 a capture +// is also REFUSED per app when the target filesystem is below the reserve (B2). func (m *Manager) captureAllRecoveryUnits() { if m.stackProvider == nil { return @@ -254,6 +323,18 @@ func (m *Manager) captureAllRecoveryUnits() { if m.settings != nil && (m.settings.IsDisconnected(drivePath) || m.settings.IsDecommissioned(drivePath)) { continue // drive not writable — skip, the existing unit stays as-is } + // B2: the floor, checked BEFORE anything is written, so a refused app's previous unit is left + // byte-identical rather than half-overwritten. Per app, and the loop continues. + if usage, blocked := m.unitFloorBlocked(stack.Name); blocked { + err := fmt.Errorf("%w (reserve: %.0f%% used or %.1f GiB free) — %s", + ErrCaptureFloor, FloorUsedPercent, FloorFreeGiB, usage) + m.logger.Printf("[WARN] [backup] Recovery unit capture REFUSED for %s — %v; the previous unit is untouched and NOTHING was deleted", + stack.Name, err) + if m.unitNotify != nil { + m.unitNotify(stack.Name, err, usage) + } + continue + } if err := m.CaptureRecoveryUnit(stack.Name); err != nil { m.logger.Printf("[WARN] [backup] Recovery unit capture failed for %s: %v", stack.Name, err) // R-158: per app, and the loop CONTINUES — one app's failure must not silence the