v0.193.0 — the reserve guards the write that fills the disk, and its promise is true (R-181)
gates / gates (push) Successful in 9s
gates / gates (push) Successful in 9s
B2's capture floor (v0.192.0) was consulted in exactly ONE place — captureAllRecoveryUnits, which writes a few KB. The two legs that write the BULK into the same backups/primary/<app> tree, the DB dump and the volume dump, ran FIRST and unguarded. Measured live on demo-hp 2026-08-03 06:40:03: opengist's volume dump wrote 2.0 GB with no check, free fell to 1.0 GB, and the floor then refused the cheap write it had already lost the argument to. Its refusal message claimed "the previous unit is untouched" — measured false: that app's tar had gone 182,272 B -> 2,147,666,432 B under a stale manifest. Sixth entry in CLAUDE.md's table of shipped guarantees the code did not provide. Fix: ONE admission verdict per app per run (internal/backup/admission.go), taken before that app's FIRST write and covering all three legs — they write under one per-app root, which is why one verdict can honestly cover them. - Lazy, at the app's first write, NOT once at run start: app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. - Remembered for the run, never re-decided between an app's own legs — that is the split this closes. Reset per run. - Placed ahead of DumpAppVolumesSafe, which stops the stack as its first act, so a refused app is never bounced. After the volume-less check, which has no write. - Exactly one operator alert per refused app per run. - Leg order unchanged: volume dumps still precede the capture. The floor is now SIZE-AWARE: it asks whether THIS app's write would cross the reserve, not only whether the filesystem is already below it — which is how an app was admitted at 96% and then allowed to write 2 GB. Estimate = the app's previous .sql + .tar on disk. No history -> headroom-only, deliberately, and the alert says so. A container-based du per volume was MEASURED and rejected: 66 timed runs on demo-hp guest 9201, median ~355 ms/volume (341-404) on volumes holding tens of KB — container start-up, not the walk. Decisive on top: docker run needs the writable layer, so it can fail under exactly the pressure the reserve handles. The message was NOT weakened; the behaviour was moved so the wording became true. It now also names which term bound. Every claim is checked against a sha256 fingerprint of the tree it describes, never against the log line. Still refuses and never deletes: nothing here is generational. 11 new tests through the production functions. The DB leg cannot run without Docker, so its gate is pinned by an AST walk of backup.go asserting admitApp precedes DumpOne (strings.Contains is insufficient — a commented-out call still contains the string). 4 red-proofs demonstrated failing then restored.
This commit is contained in:
@@ -57,6 +57,13 @@ type Manager struct {
|
||||
// can state a filesystem's occupancy as an input. Nil in production → `unitTargetSpace`.
|
||||
unitSpaceFn func(stackName string) *UnitSpace
|
||||
|
||||
// admission (R-181) is the per-RUN memo of the reserve's per-app verdict, guarded by admissionMu.
|
||||
// Non-nil only for the duration of a backup run (beginAdmissionRun → its closer). One verdict per
|
||||
// app covers all THREE write legs — DB dump, volume dump, unit capture — because all three write
|
||||
// under one per-app root; see admission.go for why it is decided lazily and never re-decided.
|
||||
admissionMu sync.Mutex
|
||||
admission *admissionSet
|
||||
|
||||
// appStop (R-166) is the crash marker for operations that stop an app, work on its data, and
|
||||
// start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power
|
||||
// cut in that window leaves a durable record that Recover honours at the next startup. Built in
|
||||
@@ -412,6 +419,13 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
|
||||
start := time.Now()
|
||||
m.logger.Printf("[INFO] [backup] Starting database dump run")
|
||||
|
||||
// R-181: open the per-run admission scope HERE, because this function is the single orchestrator
|
||||
// of all three write legs. Each app's reserve verdict is taken at its first write of this run and
|
||||
// then reused by the other two legs, so a refused app writes nothing at all and is alerted once.
|
||||
// The scope is closed on every exit path — a set that outlived its run would answer tonight's
|
||||
// question with last night's disk.
|
||||
defer m.beginAdmissionRun()()
|
||||
|
||||
dbs, err := DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames())
|
||||
if err != nil {
|
||||
m.logger.Printf("[ERROR] [backup] Database discovery failed: %v", err)
|
||||
@@ -447,6 +461,15 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
|
||||
continue
|
||||
}
|
||||
|
||||
// R-181: the reserve, BEFORE the first byte of this app's backup is written. This is usually
|
||||
// where an app's verdict is taken, because the DB leg runs first; the volume leg and the
|
||||
// capture then read the same memo. SKIP, not FAIL — a deliberate hold is not a broken dump,
|
||||
// and the operator alert (fired once, inside admitApp) is the signal that it happened.
|
||||
if !m.admitApp(db.StackName) {
|
||||
summary = append(summary, fmt.Sprintf("SKIP %s (reserve — app backup refused)", db.ContainerName))
|
||||
continue
|
||||
}
|
||||
|
||||
dumpDir := AppDBDumpPath(m.namespaceRoot(drivePath), db.StackName)
|
||||
|
||||
result := DumpOne(ctx, db, dumpDir, m.logger, m.isDebug())
|
||||
@@ -541,6 +564,12 @@ func failedSummaryLines(summary []string) []string {
|
||||
// variant stops the stack before its own volume check — calling it unconditionally would bounce
|
||||
// every volume-less app on every nightly run. Per-stack isolation mirrors the DB loop: one app's
|
||||
// failure is recorded and does not abort the others.
|
||||
//
|
||||
// R-181 adds the reserve to that order, and for the SAME reason: it sits ahead of DumpAppVolumesSafe,
|
||||
// so a refused app is never stopped. A refusal decided inside the Safe variant would already have
|
||||
// bounced the app it was refusing to back up. It sits AFTER the volume-less check because an app with
|
||||
// no named volumes writes nothing in this leg — there is no first write here to gate, and consulting
|
||||
// the reserve for it would only decide a verdict early on a stale reading.
|
||||
func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
|
||||
allOK = true
|
||||
if m.stackProvider == nil {
|
||||
@@ -576,6 +605,14 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
|
||||
continue
|
||||
}
|
||||
|
||||
// R-181: the reserve, ahead of DumpAppVolumesSafe so a refused app is NOT stopped. For an app
|
||||
// that already has a DB this is a memo lookup taken before its DB dump; for a volume-only app
|
||||
// this is where its verdict is taken, still before its first byte.
|
||||
if !m.admitApp(stack.Name) {
|
||||
summary = append(summary, fmt.Sprintf("SKIP %s volumes (reserve — app backup refused)", stack.Name))
|
||||
continue
|
||||
}
|
||||
|
||||
if err := dump(stack.Name); err != nil {
|
||||
allOK = false
|
||||
summary = append(summary, fmt.Sprintf("FAIL %s volumes: %v", stack.Name, err))
|
||||
|
||||
Reference in New Issue
Block a user