v0.193.0 — the reserve guards the write that fills the disk, and its promise is true (R-181)
gates / gates (push) Successful in 9s
gates / gates (push) Successful in 9s
B2's capture floor (v0.192.0) was consulted in exactly ONE place — captureAllRecoveryUnits, which writes a few KB. The two legs that write the BULK into the same backups/primary/<app> tree, the DB dump and the volume dump, ran FIRST and unguarded. Measured live on demo-hp 2026-08-03 06:40:03: opengist's volume dump wrote 2.0 GB with no check, free fell to 1.0 GB, and the floor then refused the cheap write it had already lost the argument to. Its refusal message claimed "the previous unit is untouched" — measured false: that app's tar had gone 182,272 B -> 2,147,666,432 B under a stale manifest. Sixth entry in CLAUDE.md's table of shipped guarantees the code did not provide. Fix: ONE admission verdict per app per run (internal/backup/admission.go), taken before that app's FIRST write and covering all three legs — they write under one per-app root, which is why one verdict can honestly cover them. - Lazy, at the app's first write, NOT once at run start: app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. - Remembered for the run, never re-decided between an app's own legs — that is the split this closes. Reset per run. - Placed ahead of DumpAppVolumesSafe, which stops the stack as its first act, so a refused app is never bounced. After the volume-less check, which has no write. - Exactly one operator alert per refused app per run. - Leg order unchanged: volume dumps still precede the capture. The floor is now SIZE-AWARE: it asks whether THIS app's write would cross the reserve, not only whether the filesystem is already below it — which is how an app was admitted at 96% and then allowed to write 2 GB. Estimate = the app's previous .sql + .tar on disk. No history -> headroom-only, deliberately, and the alert says so. A container-based du per volume was MEASURED and rejected: 66 timed runs on demo-hp guest 9201, median ~355 ms/volume (341-404) on volumes holding tens of KB — container start-up, not the walk. Decisive on top: docker run needs the writable layer, so it can fail under exactly the pressure the reserve handles. The message was NOT weakened; the behaviour was moved so the wording became true. It now also names which term bound. Every claim is checked against a sha256 fingerprint of the tree it describes, never against the log line. Still refuses and never deletes: nothing here is generational. 11 new tests through the production functions. The DB leg cannot run without Docker, so its gate is pinned by an AST walk of backup.go asserting admitApp precedes DumpOne (strings.Contains is insufficient — a commented-out call still contains the string). 4 red-proofs demonstrated failing then restored.
This commit is contained in:
@@ -279,27 +279,51 @@ const (
|
||||
FloorFreeGiB = 1.0
|
||||
)
|
||||
|
||||
// ErrCaptureFloor marks a capture refused for headroom. It is a REFUSAL, not a failure of the capture
|
||||
// machinery — the distinction matters to a reader of the alert, which is why the message names the
|
||||
// reserve rather than reporting an I/O error.
|
||||
var ErrCaptureFloor = errors.New("refused: capturing would leave the filesystem below the reserve")
|
||||
// ErrCaptureFloor marks an app's backup refused for headroom. It is a REFUSAL, not a failure of the
|
||||
// backup machinery — the distinction matters to a reader of the alert, which is why the message names
|
||||
// the reserve rather than reporting an I/O error.
|
||||
//
|
||||
// R-181 widened what it covers: it now refuses the app's DB dump, volume dump and capture together
|
||||
// (see admission.go), not the capture alone. The sentinel keeps its name because callers match it and
|
||||
// "capture" still reads correctly for "capturing this app's backup"; the MESSAGE is what changed, and
|
||||
// the message is what an operator sees.
|
||||
var ErrCaptureFloor = errors.New("refused: backing up this app would leave the filesystem below the reserve")
|
||||
|
||||
// floorVerdict is the PURE predicate: given a reading, does the floor refuse? Separated so the
|
||||
// thresholds are unit-testable without a filesystem, a stack provider or a clock.
|
||||
// floorVerdict is the PURE predicate: given a reading and this app's estimated write, does the floor
|
||||
// refuse, and on which term? Separated so the thresholds are unit-testable without a filesystem, a
|
||||
// stack provider or a clock.
|
||||
//
|
||||
// TWO QUESTIONS, NOT ONE (R-181). "Is the filesystem already below the reserve?" is the headroom term
|
||||
// and was all B2 asked. "Would THIS app's write take it below?" is the size term, and its absence is
|
||||
// how an app was admitted at 96% used and then allowed to write 2 GB. Both terms are evaluated
|
||||
// against BOTH thresholds — a large write can cross the percentage bound on a small volume and the
|
||||
// free-byte bound on a large one, which is the same reason the reserve has two terms at all.
|
||||
//
|
||||
// §8.4 — A NIL READING NEITHER REFUSES NOR WARNS. An unreadable filesystem is the drive gate's
|
||||
// business and has its own alert; refusing on it would block every capture on a box whose drive
|
||||
// merely blipped, and warning on it would be a false alarm with a misleading cause.
|
||||
func (m *Manager) floorVerdict(u *UnitSpace) (*UnitSpace, bool) {
|
||||
// business and has its own alert; refusing on it would block every backup on a box whose drive merely
|
||||
// blipped, and warning on it would be a false alarm with a misleading cause.
|
||||
//
|
||||
// estGiB == 0 (no previous dump to estimate from) degrades to the headroom term alone, deliberately:
|
||||
// refusing an app that has never been backed up would make the FIRST backup the one that can never
|
||||
// happen (Scenario E).
|
||||
func (m *Manager) floorVerdict(u *UnitSpace, estGiB float64) (*UnitSpace, floorReason) {
|
||||
if u == nil {
|
||||
return nil, false
|
||||
return nil, floorAdmit
|
||||
}
|
||||
return u, u.UsedPercent >= FloorUsedPercent || u.AvailGB < FloorFreeGiB
|
||||
}
|
||||
|
||||
// unitFloorBlocked reads the target filesystem and applies the floor.
|
||||
func (m *Manager) unitFloorBlocked(stackName string) (*UnitSpace, bool) {
|
||||
return m.floorVerdict(m.readUnitSpace(stackName))
|
||||
if u.UsedPercent >= FloorUsedPercent || u.AvailGB < FloorFreeGiB {
|
||||
return u, floorHeadroom
|
||||
}
|
||||
if estGiB > 0 {
|
||||
availAfter := u.AvailGB - estGiB
|
||||
usedAfter := u.UsedPercent
|
||||
if u.TotalGB > 0 {
|
||||
usedAfter = (u.UsedGB + estGiB) / u.TotalGB * 100
|
||||
}
|
||||
if availAfter < FloorFreeGiB || usedAfter >= FloorUsedPercent {
|
||||
return u, floorSize
|
||||
}
|
||||
}
|
||||
return u, floorAdmit
|
||||
}
|
||||
|
||||
// readUnitSpace goes through the seam when one is injected, so a test can state the filesystem's
|
||||
@@ -312,8 +336,12 @@ func (m *Manager) readUnitSpace(stackName string) *UnitSpace {
|
||||
}
|
||||
|
||||
// captureAllRecoveryUnits refreshes the recovery unit for every deployed stack. Best-effort:
|
||||
// a per-app failure is logged, NOTIFIED (R-158), and does not abort the others. Since R-165 a capture
|
||||
// is also REFUSED per app when the target filesystem is below the reserve (B2).
|
||||
// a per-app failure is logged, NOTIFIED (R-158), and does not abort the others.
|
||||
//
|
||||
// R-181: the reserve is consulted through `admitApp`, which is the SAME verdict the DB-dump and
|
||||
// volume-dump legs of this run already consulted for this app. When a run is in flight the answer
|
||||
// here is a memo lookup — an app refused before its first write is refused here too, silently,
|
||||
// because it was already alerted once. Outside a run (the periodic status refresh) it decides fresh.
|
||||
func (m *Manager) captureAllRecoveryUnits() {
|
||||
if m.stackProvider == nil {
|
||||
return
|
||||
@@ -323,16 +351,8 @@ func (m *Manager) captureAllRecoveryUnits() {
|
||||
if m.settings != nil && (m.settings.IsDisconnected(drivePath) || m.settings.IsDecommissioned(drivePath)) {
|
||||
continue // drive not writable — skip, the existing unit stays as-is
|
||||
}
|
||||
// B2: the floor, checked BEFORE anything is written, so a refused app's previous unit is left
|
||||
// byte-identical rather than half-overwritten. Per app, and the loop continues.
|
||||
if usage, blocked := m.unitFloorBlocked(stack.Name); blocked {
|
||||
err := fmt.Errorf("%w (reserve: %.0f%% used or %.1f GiB free) — %s",
|
||||
ErrCaptureFloor, FloorUsedPercent, FloorFreeGiB, usage)
|
||||
m.logger.Printf("[WARN] [backup] Recovery unit capture REFUSED for %s — %v; the previous unit is untouched and NOTHING was deleted",
|
||||
stack.Name, err)
|
||||
if m.unitNotify != nil {
|
||||
m.unitNotify(stack.Name, err, usage)
|
||||
}
|
||||
// The reserve, checked BEFORE anything is written. Per app, and the loop continues.
|
||||
if !m.admitApp(stack.Name) {
|
||||
continue
|
||||
}
|
||||
if err := m.CaptureRecoveryUnit(stack.Name); err != nil {
|
||||
|
||||
Reference in New Issue
Block a user