v0.192.0 — the capture floor replaces the bulkhead (R-165, decision B2)
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
Ships BEFORE the disk-layout merge it exists for, and is harmless on a box that never gets it. The mp1 partition was a BULKHEAD as well as a ceiling: it kept a runaway capture from filling the space the container runtime needs, because /var/lib/docker was a different filesystem. After the merge it is the same one, and a full Docker data-root is a stopped box. The floor sits in captureAllRecoveryUnits, checked BEFORE anything is written: below the reserve, that ONE app's capture is refused, its previous unit is left byte-identical, the R-158 alert fires with the space figures, and the loop continues. Two terms whichever binds first (97% used / 1 GiB free) in fillwatch's shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned before a refusal can happen — a floor that fires before its own warning is a silent failure wearing a threshold. Headroom, never unit size: a per-unit cap would be R-163 rebuilt inside one volume. Refuses, never deletes: nothing here is generational, so pruning could only destroy a different app's only local copy; pruneStalePrimaryDirs is an orphan sweep, not retention, and must not be repurposed. Tests 1184 -> 1191. One fixture strengthened mid-red-proof: the "old 20 G ceiling is gone" test sat at exactly 20 GB and survived a literal UsedGB > 20 cap — hollow. Now 120 GB, and the mutation fails it.
This commit is contained in:
@@ -4,6 +4,7 @@ import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
@@ -243,8 +244,76 @@ func (m *Manager) unitTargetSpace(stackName string) *UnitSpace {
|
||||
}
|
||||
}
|
||||
|
||||
// ── The capture floor (R-165 / decision B2) ──────────────────────────────────────────────────────
|
||||
//
|
||||
// WHAT IT REPLACES. Until the `mp1`→`mp0` merge, the 20 G backup partition was a BULKHEAD as well as
|
||||
// a ceiling: an app whose unit outgrew it was refused per app, its last good unit preserved
|
||||
// byte-identical, and the overflow **could not reach `/var/lib/docker`** because that was a different
|
||||
// filesystem. After the merge it can, and a full Docker data-root is a stopped box, not a slow one.
|
||||
// This floor is that bulkhead, done deliberately instead of by accident.
|
||||
//
|
||||
// IT IS ABOUT THE FILESYSTEM'S HEADROOM, NEVER THE UNIT'S SIZE. A per-unit size cap would be R-163
|
||||
// rebuilt inside one volume — the wall moved rather than removed — so a large unit on a filesystem
|
||||
// with ample room is captured, whatever its size.
|
||||
//
|
||||
// IT REFUSES; IT NEVER DELETES. Nothing on this filesystem is generational: a unit is ONE fixed path
|
||||
// per app (`backups/primary/<app>`) refreshed in place, and a DB dump is `<stack>-<dbtype>.sql`, also
|
||||
// fixed. So "prune the oldest" could only mean deleting a DIFFERENT app's only local recovery unit to
|
||||
// make room for this one, and that is not a trade this system makes. `pruneStalePrimaryDirs` is NOT a
|
||||
// retention policy — it removes ORPHANED directories left when an app moves drives, and has no notion
|
||||
// of age — so it must never be repurposed here.
|
||||
const (
|
||||
// FloorUsedPercent / FloorFreeGiB — the reserve. Two terms, whichever binds first, the same shape
|
||||
// as `internal/fillwatch` (proven live on 2026-08-02: the critical alert fired on the free-byte
|
||||
// term at 91% used, where a percent-only rule stayed silent).
|
||||
//
|
||||
// THEY SIT DELIBERATELY BEYOND fillwatch's CRITICAL BAND (95% / 2 GiB), so the customer is ALWAYS
|
||||
// warned before a refusal can happen. A floor that fires before its own warning is a silent
|
||||
// failure wearing a threshold; `TestFloorSitsBelowTheCriticalWarningBand` pins the ordering.
|
||||
//
|
||||
// 1 GiB is the reserve, not a working budget: §7.5 measures a DB-backed app's unit at up to ~2× its
|
||||
// data, so no fixed number can guarantee a capture fits. What this guarantees is different and is
|
||||
// the bulkhead's actual job — that a capture cannot consume the last of the space the container
|
||||
// runtime needs to keep running.
|
||||
FloorUsedPercent = 97.0
|
||||
FloorFreeGiB = 1.0
|
||||
)
|
||||
|
||||
// ErrCaptureFloor marks a capture refused for headroom. It is a REFUSAL, not a failure of the capture
|
||||
// machinery — the distinction matters to a reader of the alert, which is why the message names the
|
||||
// reserve rather than reporting an I/O error.
|
||||
var ErrCaptureFloor = errors.New("refused: capturing would leave the filesystem below the reserve")
|
||||
|
||||
// floorVerdict is the PURE predicate: given a reading, does the floor refuse? Separated so the
|
||||
// thresholds are unit-testable without a filesystem, a stack provider or a clock.
|
||||
//
|
||||
// §8.4 — A NIL READING NEITHER REFUSES NOR WARNS. An unreadable filesystem is the drive gate's
|
||||
// business and has its own alert; refusing on it would block every capture on a box whose drive
|
||||
// merely blipped, and warning on it would be a false alarm with a misleading cause.
|
||||
func (m *Manager) floorVerdict(u *UnitSpace) (*UnitSpace, bool) {
|
||||
if u == nil {
|
||||
return nil, false
|
||||
}
|
||||
return u, u.UsedPercent >= FloorUsedPercent || u.AvailGB < FloorFreeGiB
|
||||
}
|
||||
|
||||
// unitFloorBlocked reads the target filesystem and applies the floor.
|
||||
func (m *Manager) unitFloorBlocked(stackName string) (*UnitSpace, bool) {
|
||||
return m.floorVerdict(m.readUnitSpace(stackName))
|
||||
}
|
||||
|
||||
// readUnitSpace goes through the seam when one is injected, so a test can state the filesystem's
|
||||
// occupancy as an input instead of manufacturing it on a real disk. Nil seam → the real statfs.
|
||||
func (m *Manager) readUnitSpace(stackName string) *UnitSpace {
|
||||
if m.unitSpaceFn != nil {
|
||||
return m.unitSpaceFn(stackName)
|
||||
}
|
||||
return m.unitTargetSpace(stackName)
|
||||
}
|
||||
|
||||
// captureAllRecoveryUnits refreshes the recovery unit for every deployed stack. Best-effort:
|
||||
// a per-app failure is logged, NOTIFIED (R-158), and does not abort the others.
|
||||
// a per-app failure is logged, NOTIFIED (R-158), and does not abort the others. Since R-165 a capture
|
||||
// is also REFUSED per app when the target filesystem is below the reserve (B2).
|
||||
func (m *Manager) captureAllRecoveryUnits() {
|
||||
if m.stackProvider == nil {
|
||||
return
|
||||
@@ -254,6 +323,18 @@ func (m *Manager) captureAllRecoveryUnits() {
|
||||
if m.settings != nil && (m.settings.IsDisconnected(drivePath) || m.settings.IsDecommissioned(drivePath)) {
|
||||
continue // drive not writable — skip, the existing unit stays as-is
|
||||
}
|
||||
// B2: the floor, checked BEFORE anything is written, so a refused app's previous unit is left
|
||||
// byte-identical rather than half-overwritten. Per app, and the loop continues.
|
||||
if usage, blocked := m.unitFloorBlocked(stack.Name); blocked {
|
||||
err := fmt.Errorf("%w (reserve: %.0f%% used or %.1f GiB free) — %s",
|
||||
ErrCaptureFloor, FloorUsedPercent, FloorFreeGiB, usage)
|
||||
m.logger.Printf("[WARN] [backup] Recovery unit capture REFUSED for %s — %v; the previous unit is untouched and NOTHING was deleted",
|
||||
stack.Name, err)
|
||||
if m.unitNotify != nil {
|
||||
m.unitNotify(stack.Name, err, usage)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err := m.CaptureRecoveryUnit(stack.Name); err != nil {
|
||||
m.logger.Printf("[WARN] [backup] Recovery unit capture failed for %s: %v", stack.Name, err)
|
||||
// R-158: per app, and the loop CONTINUES — one app's failure must not silence the
|
||||
|
||||
Reference in New Issue
Block a user