v0.193.0 — the reserve guards the write that fills the disk, and its promise is true (R-181)
gates / gates (push) Successful in 9s

B2's capture floor (v0.192.0) was consulted in exactly ONE place —
captureAllRecoveryUnits, which writes a few KB. The two legs that write the BULK
into the same backups/primary/<app> tree, the DB dump and the volume dump, ran
FIRST and unguarded. Measured live on demo-hp 2026-08-03 06:40:03: opengist's
volume dump wrote 2.0 GB with no check, free fell to 1.0 GB, and the floor then
refused the cheap write it had already lost the argument to. Its refusal message
claimed "the previous unit is untouched" — measured false: that app's tar had
gone 182,272 B -> 2,147,666,432 B under a stale manifest. Sixth entry in
CLAUDE.md's table of shipped guarantees the code did not provide.

Fix: ONE admission verdict per app per run (internal/backup/admission.go), taken
before that app's FIRST write and covering all three legs — they write under one
per-app root, which is why one verdict can honestly cover them.

- Lazy, at the app's first write, NOT once at run start: app A's dump can put app
  B under the reserve, so a run-start verdict reads a disk that no longer exists.
- Remembered for the run, never re-decided between an app's own legs — that is
  the split this closes. Reset per run.
- Placed ahead of DumpAppVolumesSafe, which stops the stack as its first act, so
  a refused app is never bounced. After the volume-less check, which has no write.
- Exactly one operator alert per refused app per run.
- Leg order unchanged: volume dumps still precede the capture.

The floor is now SIZE-AWARE: it asks whether THIS app's write would cross the
reserve, not only whether the filesystem is already below it — which is how an
app was admitted at 96% and then allowed to write 2 GB. Estimate = the app's
previous .sql + .tar on disk. No history -> headroom-only, deliberately, and the
alert says so.

A container-based du per volume was MEASURED and rejected: 66 timed runs on
demo-hp guest 9201, median ~355 ms/volume (341-404) on volumes holding tens of
KB — container start-up, not the walk. Decisive on top: docker run needs the
writable layer, so it can fail under exactly the pressure the reserve handles.

The message was NOT weakened; the behaviour was moved so the wording became true.
It now also names which term bound. Every claim is checked against a sha256
fingerprint of the tree it describes, never against the log line.

Still refuses and never deletes: nothing here is generational.

11 new tests through the production functions. The DB leg cannot run without
Docker, so its gate is pinned by an AST walk of backup.go asserting admitApp
precedes DumpOne (strings.Contains is insufficient — a commented-out call still
contains the string). 4 red-proofs demonstrated failing then restored.
This commit is contained in:
2026-08-03 10:53:48 +02:00
parent 4be6467b50
commit fef07c3923
8 changed files with 1119 additions and 33 deletions
+70
View File
@@ -1,5 +1,75 @@
## Changelog
### v0.193.0 — the reserve guards the write that fills the disk, and its promise is true (2026-08-03, R-181) — MinAgent: none
**The defect, found on live hardware and not by review.** v0.192.0's capture floor (B2) shipped as the
deliberate replacement for the bulkhead the `mp1` partition used to give, and it was consulted in
**exactly one place** — `captureAllRecoveryUnits`, which writes a manifest and three compose files: a
few KB. The two legs that write the **bulk** into the same `backups/primary/<app>` tree — the database
dump and the volume dump — ran **first** and **unguarded**. Measured on demo-hp 2026-08-03 06:40:03:
opengist's volume dump wrote **2.0 GB with no check**, free fell to 1.0 GB, and the floor then refused
the cheap write it had already lost the argument to.
**Second limb: the refusal message asserted something the code did not provide.** It printed *"the
previous unit is untouched and NOTHING was deleted"*. *Nothing was deleted* held. *Untouched* was
**measured false** — that app's tar had gone 182,272 B → 2,147,666,432 B under a `manifest.json` whose
`created_at` and `checksums` had not moved. **This is the sixth entry in `CLAUDE.md`'s table of
shipped guarantees the code did not provide**, and the fourth of those found on live hardware.
**The fix: ONE admission verdict per app per run, taken before that app's FIRST write, covering all
three legs** (`internal/backup/admission.go`). The three write under one per-app root, which is
exactly why one verdict can honestly cover them — and why the message may now claim what it claims.
- **Decided lazily, at the app's first write — NOT once at the start of the run.** Space changes
during a run: app A's 2 GB dump can put app B under the reserve, and a run-start verdict would wave
B through on a reading that was true before the disk filled. That is the same class of mistake,
moved one level up.
- **Remembered for the run, never re-decided between an app's own legs.** Re-deciding reintroduces
the split this closes (DB admitted → volume admitted → capture refused, with the bulk written).
Reset per run: a set carried between runs answers tonight's question with last night's disk.
- **Placed ahead of `DumpAppVolumesSafe`, which stops the stack as its first act** — a refusal decided
inside it would already have bounced the app it is refusing to back up. It sits *after* the
volume-less check, because an app with no named volumes has no first write in that leg to gate.
- **Exactly one operator alert per refused app per run.** Three legs must not mean three emails.
- **The leg order is unchanged** — volume dumps still precede the capture so the manifests enumerate
the fresh tars (`backup.go`'s load-bearing comment).
**The floor is now SIZE-AWARE, not merely headroom-aware.** It asks *would this app's write leave the
filesystem below the reserve?*, not only *is it below the reserve now* — which is how an app was
admitted at 96% used and then allowed to write 2 GB. The estimate is the app's **previous** `.sql` and
`.tar` already on disk: free to read, and the next write is usually close. **No history →
headroom-only**, deliberately, or the first backup would be the one that can never happen; the alert
says so when that applies. Both post-write terms are evaluated, because a large write crosses the
percentage bound on a small volume and the free-byte bound on a large one.
**A container-based `du` per volume was measured and REJECTED, not assumed.** 66 timed runs on
demo-hp's guest 9201: **median ~355 ms per volume** (341404 ms) on volumes holding tens of KB — the
cost is container start-up, not the walk, so it does not shrink for small apps and only grows for real
ones. Decisive on top of that: `docker run` needs the writable layer, so the measurement mechanism can
fail under exactly the disk pressure the reserve exists to handle. The previous-dump estimate also
measures the **artifact** that will be written rather than the live volume, which is the truer
predictor. The figure and the decision are recorded rather than left as a "we could do better".
**THE MESSAGE WAS NOT WEAKENED — the behaviour was moved so the wording became true.** It still says
the previous unit is untouched and nothing was deleted, and now adds *which* term bound (headroom or
size) and the estimate that produced a size refusal.
`TestAdmission_EveryClaimInTheRefusalMessageHoldsAgainstTheTree` checks **every** claim against a
sha256 fingerprint of the tree it describes — not against the log line, because a log line is exactly
what lied here.
**IT STILL REFUSES AND NEVER DELETES.** Unchanged and load-bearing: nothing on this filesystem is
generational, so "prune the oldest" could only mean destroying a **different** app's only local
recovery unit. `pruneStalePrimaryDirs` is not a retention policy and must never be repurposed for
headroom.
**Tests (11 new, all through the production functions; 4 red-proofs demonstrated failing then
restored).** The refusal assertions are **tree fingerprints before and after**, never log lines. The
DB leg cannot run without Docker, so its gate is pinned by an **AST walk** of `backup.go` asserting
`admitApp` precedes `DumpOne``strings.Contains` is insufficient, a commented-out call still
contains the string. Red-proofs: both dump-leg gates removed (= v0.192.0) → Scenario A red, tree shown
changing; the size term removed → Scenario D red; a prune injected into the refusal path → Scenario F
red; the floor moved above the warning band → Scenario G red.
### v0.192.0 — the capture floor replaces the bulkhead (2026-08-03, R-165 · decision B2) — MinAgent: none
**Ships BEFORE the disk-layout merge it exists for, and is harmless on a box that never gets it.**
+2 -1
View File
@@ -174,7 +174,8 @@
| `api.GracefulSelfRestart` | controller/internal/api/selfrestart.go | `(logger)` | Controller self-restart | Detached exit; bootstrap unit re-runs the image |
| `Settings.AddPendingEvent/DrainPendingEvents` | controller/internal/settings/settings.go | offline event queue | Events while hub unreachable | — |
| `Manager.SetUnitNotify` + `UnitSpace` (R-158/R-167, v0.191.0) | controller/internal/backup/recovery_unit.go | `(func(stack string, err error, *UnitSpace))` | THE per-app Tier-1 recovery-unit capture failure alert — fires PER APP from `captureAllRecoveryUnits`, loop continues | **OPERATOR-TIER** (`recovery_unit_capture_failed`, in the hub's `operatorOnlyEvents`). **NEVER route it to `backup_failed`** — that type is in `DefaultEnabledEvents` and carries Hungarian copy, so it emails the CUSTOMER about a failure they cannot act on (D-c; R-158's own proposal said `backup_failed` and D-c overrides it). `UnitSpace` is **nil when the target filesystem is unreadable** and renders as *"unavailable"*, never as zeros — "0 GB free" and "we could not look" are opposite diagnoses. No controller-side cooldown: the hub owns it |
| `Manager.unitFloorBlocked` / `floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` (R-165 B2, v0.192.0) | controller/internal/backup/recovery_unit.go | `(stackName) (*UnitSpace, bool)` / `(*UnitSpace) (*UnitSpace, bool)` | THE capture floor that replaces the `mp1` bulkhead — refuses ONE app's recovery-unit capture before anything is written when the target filesystem is below the reserve | **Headroom, NEVER unit size** — a per-unit cap is R-163 rebuilt inside one volume. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk |
| `Manager.admitApp` / `beginAdmissionRun` / `decideAdmission` / `estimatedWriteGiB` (R-181, v0.193.0) | controller/internal/backup/admission.go | `(stackName) bool` / `() func()` | **THE reserve gate. Call it before ANY per-app backup write** — one verdict per app per run, covering the DB dump, the volume dump and the unit capture (all three write under one per-app root) | **Decided LAZILY at the app's first write, never once at run start** — app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. **Never re-decided between an app's own legs**: that is exactly the split R-181 closed (bulk written, capture refused). **Reset per run** via the closer `beginAdmissionRun` returns. **Must sit ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided inside it has already bounced the app. Fires **exactly one** `unitNotify` per refused app per run. Nil admission set (periodic status refresh) → decides fresh, which is still once per app per sweep. Wiring pinned by an **AST walk** in `TestAdmission_IsWiredIntoEveryProductionWriteLeg`, not `strings.Contains` |
| `Manager.floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` / `floorReason` (R-165 B2 v0.192.0, size term R-181 v0.193.0) | controller/internal/backup/recovery_unit.go | `(*UnitSpace, estGiB float64) (*UnitSpace, floorReason)` | The pure two-question predicate behind `admitApp`: is the filesystem already below the reserve (`floorHeadroom`), and would THIS app's write take it below (`floorSize`)? | **Headroom is about the FILESYSTEM, never a per-unit cap** — a size cap is R-163 rebuilt inside one volume; the size term bounds the *delta*, not the unit. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. **`estGiB == 0` degrades to headroom-only on purpose** — refusing an app with no history makes the FIRST backup the one that can never happen. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk |
| `fillwatch.Watcher` (`New`/`SetNotify`/`Check`) (R-167, v0.191.0) | controller/internal/fillwatch/fillwatch.go | `(statePath, logger, targetsFn, usageFn)``Check() error` | THE customer fill warning — warns BEFORE a filesystem fills, per FILESYSTEM (never per app: one full disk holding ten apps would fire ten times) | Emits the **pre-existing** `disk_warning`/`disk_critical` pair, which was allowlisted + copy'd + default-enabled with **no producer in any repo** until now — do NOT mint a new type beside it. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range. **Edge-triggered on ESCALATION ONLY**, state persisted; de-escalation is silent and re-arms. Hysteresis dead zone between clear (75% / 7 GiB) and warn — pinned by `TestThresholdsKeepTheirHysteresisGap`. **A nil usage read is NEVER a warning** (§8.4). The hub has **no `customerMessages` entry** for either type on purpose — an entry would override the dynamic message and discard the drive label + free space |
### Scheduler / time / UI
+21
View File
@@ -623,6 +623,27 @@ Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypte
The backup system implements a **3-2-1 backup architecture**. Each tier is a **complete,
self-sufficient backup** — any single tier can fully restore an app.
**The reserve — per-app backup admission (v0.192.0 decision B2, widened by v0.193.0 / R-181).**
`internal/backup/admission.go`. Since the `mp1``mp0` merge (R-165) local backups and Docker's
data-root share one filesystem, so an unbounded backup write is a stopped box rather than a slow one.
Before **any** of an app's three local write legs runs — DB dump, volume dump, recovery-unit capture —
`admitApp` takes **one verdict for that app for that run** and the other two legs reuse it. A refused
app writes nothing at all, is **not stopped**, keeps its previous unit byte-identical, and produces
**exactly one** operator alert (`recovery_unit_capture_failed`, operator-tier).
- **The verdict is lazy, not run-wide.** It is taken at the app's first write, because app A's dump
can put app B under the reserve; a verdict taken at run start would read a disk that no longer
exists by the time B writes.
- **It is never re-decided between an app's own legs**, and the memo is reset per run.
- **Two questions, both against two thresholds (97% used / 1 GiB free).** *Headroom*: is the
filesystem already below the reserve? *Size*: would this app's own write take it below? The size
estimate is the app's **previous** `.sql` + `.tar` already on disk. **No history → headroom-only**,
deliberately — otherwise the first backup is the one that can never happen — and the alert says so.
- **The thresholds sit beyond `fillwatch`'s critical band (95% / 2 GiB)**, so the customer is always
warned before a refusal is possible.
- **It refuses; it never deletes.** Nothing here is generational — one unit per app at one fixed path
— so "prune the oldest" could only destroy a different app's only local copy.
**Sidebar behaviour (v0.146.0).** Groups that own sub-pages — Tárhely, Biztonsági mentés,
Megosztás — render as **accordions**: the header is a real `<button class="nav-group-toggle">`
(keyboard- and AT-reachable for free) carrying a chevron, and **exactly one group is open at a
+237
View File
@@ -0,0 +1,237 @@
package backup
import (
"fmt"
"os"
"path/filepath"
"strings"
)
// ── Backup admission (R-181) ─────────────────────────────────────────────────────────────────────
//
// WHAT WAS WRONG. B2's capture floor (v0.192.0, R-165) shipped as the deliberate replacement for the
// bulkhead the `mp1` partition used to give, and it was consulted in exactly ONE place —
// `captureAllRecoveryUnits`, which writes a manifest and three compose files: a few KB. The two legs
// that write the BULK into the same `backups/primary/<app>` tree — the database dump and the volume
// dump — ran FIRST and unguarded. Measured on demo-hp 2026-08-03 06:40:03: opengist's volume dump
// wrote 2.0 GB with no check, free fell to 1.0 GB, and the floor then refused the cheap write it had
// already lost the argument to. Its refusal message said *"the previous unit is untouched"*, which
// was false by then — that app's tar had gone 182,272 B → 2,147,666,432 B under a stale manifest.
//
// WHAT THIS IS. ONE verdict per app per run, taken before that app's FIRST write of the run, covering
// all three legs. The three write under one per-app root (`appbackup.RecoveryUnitPath`), which is
// exactly why one verdict can honestly cover them — and why the message may now claim what it claims.
//
// WHY IT IS DECIDED LAZILY AND NOT ONCE AT THE START OF THE RUN. Space changes during a run: app A's
// 2 GB dump can put app B under the reserve. A verdict taken at run start would wave B through on a
// reading that was true before the disk filled — the same class of mistake as the one being fixed,
// moved one level up.
//
// WHY IT IS REMEMBERED AND NOT RE-DECIDED PER LEG. Re-deciding between an app's own legs reintroduces
// the split this closes: the DB leg admitted, the volume leg admitted, the capture refused — with the
// bulk already written. Decide once, remember, reuse; reset per run, because a set carried between
// runs is a wrong answer with a confident face.
//
// IT REFUSES; IT NEVER DELETES. Unchanged from B2 and load-bearing: nothing on this filesystem is
// generational (one unit per app at one fixed path, refreshed in place), so "prune the oldest" could
// only mean destroying a DIFFERENT app's only local copy. `pruneStalePrimaryDirs` removes ORPHANED
// dirs an app left on a drive it moved off — it has no notion of age or of the current app — and must
// never be repurposed for headroom.
// floorReason records WHICH term bound, so the operator can tell "the disk is full" from "this app's
// backup is too big for what is left". An alert that says only "refused" sends them to read code.
type floorReason int
const (
floorAdmit floorReason = iota // admitted — no term binds
floorHeadroom // the filesystem is ALREADY at/below the reserve
floorSize // there is room now, but this app's own write would cross the reserve
)
func (r floorReason) String() string {
switch r {
case floorHeadroom:
return "headroom"
case floorSize:
return "size"
default:
return "admitted"
}
}
// admissionVerdict is one app's decision for one run. It carries everything the alert needs, so the
// alert is rendered once from the same value every leg consults.
type admissionVerdict struct {
admitted bool
reason floorReason
usage *UnitSpace
estGiB float64 // the estimated write, 0 when the app has no previous dump on disk
hasEst bool // whether an estimate was available at all (§8.2: distinct from "estimated 0")
err error // the refusal, nil when admitted
}
// admissionSet is the per-RUN memo. Deliberately not a field with a lifetime of its own: it is
// created by beginAdmissionRun and cleared by the returned func, so an absent set means "no run is in
// flight" rather than "a stale answer from last night".
type admissionSet struct {
v map[string]admissionVerdict
}
// beginAdmissionRun opens the per-run admission scope and returns the closer. Called once at the top
// of runDBDumpsInternal — which is the single orchestrator of all three legs — so the DB dump, the
// volume dump and the capture of one app all consult the SAME verdict.
//
// A second call while a set is live REPLACES it and the returned closer restores the previous one, so
// nesting cannot silently drop a caller's scope.
func (m *Manager) beginAdmissionRun() func() {
m.admissionMu.Lock()
prev := m.admission
m.admission = &admissionSet{v: map[string]admissionVerdict{}}
m.admissionMu.Unlock()
return func() {
m.admissionMu.Lock()
m.admission = prev
m.admissionMu.Unlock()
}
}
// admitApp is THE gate. It returns true when this app may write, false when the reserve refuses it.
//
// On the first refusal for an app it logs and fires EXACTLY ONE operator alert; every later leg in
// the same run reads the memo and stays silent, so a refused app produces one email and not three.
//
// With no run scope open (the periodic status refresh calls captureAllRecoveryUnits directly) it
// decides fresh. That is not a gap: each app appears once in that sweep, so "once per app" still
// holds — there is simply nothing to remember it across.
func (m *Manager) admitApp(stackName string) bool {
m.admissionMu.Lock()
defer m.admissionMu.Unlock()
if set := m.admission; set != nil {
if v, ok := set.v[stackName]; ok {
return v.admitted // already decided this run — do NOT re-decide, do NOT re-alert
}
}
v := m.decideAdmission(stackName)
if set := m.admission; set != nil {
set.v[stackName] = v
}
if v.admitted {
return true
}
// The claim below is now literally true, and that is the whole point of R-181: the verdict is
// taken before the FIRST of the three writes, so at this moment nothing under
// backups/primary/<app> has been touched by this run. TestAdmission_RefusedAppsTreeIsByteIdentical
// pins the consequence by checksumming the tree, not by reading this line.
m.logger.Printf("[WARN] [backup] App backup REFUSED for %s (%s) — %v; NO database dump, NO volume "+
"dump and NO recovery-unit capture was written for it, the previous unit is untouched and "+
"NOTHING was deleted", stackName, v.reason, v.err)
if m.unitNotify != nil {
m.unitNotify(stackName, v.err, v.usage)
}
return false
}
// decideAdmission applies the floor to a fresh reading plus this app's estimated write.
func (m *Manager) decideAdmission(stackName string) admissionVerdict {
estGiB, hasEst := m.estimatedWriteGiB(stackName)
usage, reason := m.floorVerdict(m.readUnitSpace(stackName), estGiB)
v := admissionVerdict{
admitted: reason == floorAdmit,
reason: reason,
usage: usage,
estGiB: estGiB,
hasEst: hasEst,
}
if !v.admitted {
v.err = floorRefusal(reason, usage, estGiB, hasEst)
}
return v
}
// floorRefusal renders the refusal an operator reads. It names the reserve (not an I/O error — this
// is a deliberate hold, not broken machinery), says WHICH term bound, and states plainly when the
// decision was headroom-only because the app has no previous backup to estimate from (§8.2).
func floorRefusal(reason floorReason, usage *UnitSpace, estGiB float64, hasEst bool) error {
var b strings.Builder
fmt.Fprintf(&b, "%%w (reserve: %.0f%%%% used or %.1f GiB free", FloorUsedPercent, FloorFreeGiB)
switch {
case reason == floorSize:
fmt.Fprintf(&b, "; this app's last backup was %.2f GiB and writing it again would cross the reserve", estGiB)
case hasEst:
fmt.Fprintf(&b, "; the filesystem is already below it, before this app's estimated %.2f GiB write", estGiB)
default:
b.WriteString("; this app has no previous backup on disk, so only current headroom was considered")
}
b.WriteString(") — %s")
return fmt.Errorf(b.String(), ErrCaptureFloor, usage)
}
// estimatedWriteGiB estimates what this app's three legs are about to write, from what the PREVIOUS
// run left in its unit: the `.sql` dumps and the `.tar` volume archives already on disk for this app.
//
// WHY THIS ESTIMATOR. It is free — two ReadDirs of a directory the caller is about to write into — and
// the next write is usually close to the last one. The alternative, a container-based `du` of every
// named volume, was measured on the demo box before being rejected; the figure is in REPORT.md §6.
//
// NO HISTORY → (0, false), and the caller falls back to headroom-only. Refusing an app because it has
// never been backed up would make the first backup the one that can never happen (Scenario E).
//
// It reads the app's CURRENT unit root, so an app that moved drives estimates from its new (probably
// empty) location and is treated as history-less — conservative in the admitting direction, which is
// the right way round for an estimate that only ever tightens a threshold.
func (m *Manager) estimatedWriteGiB(stackName string) (float64, bool) {
drivePath := m.GetAppDrivePath(stackName)
if drivePath == "" {
return 0, false
}
nsRoot := m.namespaceRoot(drivePath)
var total int64
var found bool
for _, d := range []struct {
dir string
ext string
}{
{AppDBDumpPath(nsRoot, stackName), ".sql"},
{AppVolumeDumpPath(nsRoot, stackName), ".tar"},
} {
n, ok := sumFileSizes(d.dir, d.ext)
total += n
found = found || ok
}
if !found {
return 0, false
}
return float64(total) / (1024 * 1024 * 1024), true
}
// sumFileSizes totals the sizes of files with the given suffix in dir. The bool reports whether ANY
// such file was seen — distinct from a zero total, because a 0-byte dump is history (a real, if
// alarming, previous result) while an absent directory is not.
//
// A stat error on one entry is skipped rather than aborting the sum: an estimate built from the
// readable files is worth more than no estimate, and the entry that could not be read is logged
// nowhere because this is a hint, not a measurement — it can only tighten a threshold, never relax
// one below what the headroom term already enforces.
func sumFileSizes(dir, suffix string) (int64, bool) {
entries, err := os.ReadDir(dir)
if err != nil {
return 0, false
}
var total int64
var found bool
for _, e := range entries {
if e.IsDir() || !strings.HasSuffix(e.Name(), suffix) {
continue
}
fi, err := os.Stat(filepath.Join(dir, e.Name()))
if err != nil {
continue
}
found = true
total += fi.Size()
}
return total, found
}
@@ -0,0 +1,699 @@
package backup
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"fmt"
"go/ast"
"go/parser"
"go/token"
"io"
"log"
"os"
"path/filepath"
"sort"
"strings"
"testing"
)
// R-181 — the reserve guards the write that fills the disk, and its promise is true.
//
// WHAT THESE ASSERT, AND WHY IT IS THE TREE AND NOT THE LOG. The defect being closed is precisely a
// log line that claimed something the filesystem contradicted: B2 printed *"the previous unit is
// untouched"* while the volume leg had already rewritten that unit's tar 182,272 B → 2,147,666,432 B.
// So a test that reads the message and believes it would have passed against the broken code. Every
// refusal test here checksums the whole `backups/primary` tree before and after and compares.
// ── Harness ──────────────────────────────────────────────────────────────────────────────────────
// admissionProvider records the two acts a refused app must never suffer: its recovery info being
// read (a capture that was ATTEMPTED) and its stack being stopped (which DumpAppVolumesSafe does as
// its first act, before any check of its own).
type admissionProvider struct {
stacks []string
volumes map[string][]string
dir string
infoHits []string
stopped []string
}
func (p *admissionProvider) GetStackComposePath(string) (string, bool) { return "", false }
func (p *admissionProvider) ListDeployedStacks() []StackSummary {
out := make([]StackSummary, 0, len(p.stacks))
for _, s := range p.stacks {
out = append(out, StackSummary{Name: s})
}
return out
}
func (p *admissionProvider) GetStackHDDMounts(string) []string { return nil }
func (p *admissionProvider) GetStackHDDPath(string) string { return "" }
func (p *admissionProvider) GetImportRoot() string { return "" }
func (p *admissionProvider) GetDockerVolumes(name string) []string {
if p.volumes == nil {
return []string{name + "_data"} // every app is volume-bearing unless told otherwise
}
return p.volumes[name]
}
func (p *admissionProvider) StopStack(name string) error {
p.stopped = append(p.stopped, name)
return nil
}
func (p *admissionProvider) StartStack(string) error { return nil }
func (p *admissionProvider) RefreshAndIsRunning(string) bool { return true }
func (p *admissionProvider) GetStackRecoveryInfo(name string) (RecoveryInfo, bool) {
p.infoHits = append(p.infoHits, name)
return RecoveryInfo{StackDir: filepath.Join(p.dir, "stacks", name)}, true
}
func (p *admissionProvider) RecoverStackSecrets(string, []string) map[string]string { return nil }
func (p *admissionProvider) RecreateStackDefinitionFromUnit(string, string, map[string]string) error {
return nil
}
func (p *admissionProvider) StartStackServices(string, []string) error { return nil }
func (p *admissionProvider) GetStackClassifiedBinds(string) ([]ClassifiedBind, bool) {
return nil, false
}
type admissionHarness struct {
m *Manager
prov *admissionProvider
events []unitEvent
usage map[string]*UnitSpace
dir string
logs *bytes.Buffer
volDumped []string
}
func newAdmissionHarness(t *testing.T, stacks ...string) *admissionHarness {
t.Helper()
dir := t.TempDir()
h := &admissionHarness{
prov: &admissionProvider{stacks: stacks, dir: dir},
usage: map[string]*UnitSpace{},
dir: dir,
logs: &bytes.Buffer{},
}
h.m = &Manager{
logger: log.New(h.logs, "", 0),
systemDataPath: dir,
stackProvider: h.prov,
unitSpaceFn: func(name string) *UnitSpace { return h.usage[name] },
}
// The volume-dump seam records the leg that writes the BULK — the one B2 never gated. A refused
// app must not reach it.
h.m.dumpVolumesSafe = func(name string) error {
h.volDumped = append(h.volDumped, name)
// Write what the real leg writes, so an ungated call is visible in the tree checksum too.
dumpDir := AppVolumeDumpPath(h.nsRoot(), name)
if err := os.MkdirAll(dumpDir, 0o755); err != nil {
return err
}
return os.WriteFile(filepath.Join(dumpDir, name+"_data.tar"), []byte("FRESH TAR FROM THIS RUN"), 0o644)
}
h.m.SetUnitNotify(func(name string, err error, u *UnitSpace) {
h.events = append(h.events, unitEvent{app: name, err: err.Error(), usage: u})
})
return h
}
func (h *admissionHarness) nsRoot() string { return filepath.Join(h.dir, "felhom-data") }
// setSpace states the filesystem's occupancy as a test INPUT — the whole point of the unitSpaceFn
// seam, so no test has to manufacture disk pressure on a real disk.
func (h *admissionHarness) setSpace(app string, usedPct, availGB, totalGB float64) {
h.usage[app] = &UnitSpace{
Path: h.dir, UsedPercent: usedPct, AvailGB: availGB,
TotalGB: totalGB, UsedGB: totalGB * usedPct / 100,
}
}
// seedUnit writes a previous recovery unit for an app: a manifest, a captured app.yaml, a DB dump and
// a volume tar of the given size. The tar is SPARSE (Truncate), so a 2 GiB "previous backup" costs no
// disk — the estimator reads st_size, which is what the next write will actually cost.
func (h *admissionHarness) seedUnit(t *testing.T, app string, tarBytes int64) {
t.Helper()
ns := h.nsRoot()
for _, d := range []string{
RecoveryUnitComposePath(ns, app),
AppDBDumpPath(ns, app),
AppVolumeDumpPath(ns, app),
} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
write := func(p string, b []byte, mode os.FileMode) {
if err := os.WriteFile(p, b, mode); err != nil {
t.Fatal(err)
}
}
write(RecoveryUnitManifestPath(ns, app), []byte(`{"app_name":"`+app+`","created_at":"2026-08-02T00:00:00Z"}`), 0o644)
write(filepath.Join(RecoveryUnitComposePath(ns, app), "app.yaml"), []byte("deployed: true\nenv:\n A: previous-good-value\n"), 0o600)
write(filepath.Join(AppDBDumpPath(ns, app), app+"-postgres.sql"), []byte("-- previous good dump\n"), 0o644)
tar := filepath.Join(AppVolumeDumpPath(ns, app), app+"_data.tar")
f, err := os.Create(tar)
if err != nil {
t.Fatal(err)
}
if _, err := f.WriteString("PREVIOUS GOOD TAR"); err != nil {
t.Fatal(err)
}
if tarBytes > 0 {
if err := f.Truncate(tarBytes); err != nil { // sparse — st_size is the estimate, blocks are not spent
t.Fatal(err)
}
}
if err := f.Close(); err != nil {
t.Fatal(err)
}
}
// runOneBackupRun performs exactly the sequence runDBDumpsInternal performs for the two legs that can
// be driven without Docker: the admission scope is opened, the volume leg runs, then the capture leg.
// The DB leg's wiring is pinned structurally by TestAdmission_IsWiredIntoEveryProductionWriteLeg,
// because DiscoverDatabases shells out to `docker` and cannot honestly run here.
func (h *admissionHarness) runOneBackupRun() {
done := h.m.beginAdmissionRun()
defer done()
h.m.runVolumeDumps()
h.m.captureAllRecoveryUnits()
}
// ── The instrument: a checksum of the whole backup tree ──────────────────────────────────────────
// treeFingerprint walks every file under backups/primary and returns "relpath mode sha256" lines,
// sorted. It is the ONLY honest way to check the refusal's claim: it detects a rewritten payload, an
// added file and a deleted one alike, which a log line and an exit code both fail to do.
func treeFingerprint(t *testing.T, root string) string {
t.Helper()
var lines []string
err := filepath.Walk(root, func(p string, fi os.FileInfo, err error) error {
if err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}
if fi.IsDir() {
return nil
}
f, err := os.Open(p)
if err != nil {
return err
}
defer f.Close()
sum := sha256.New()
if _, err := io.Copy(sum, f); err != nil {
return err
}
rel, _ := filepath.Rel(root, p)
lines = append(lines, fmt.Sprintf("%s %o %d %s", rel, fi.Mode().Perm(), fi.Size(), hex.EncodeToString(sum.Sum(nil))))
return nil
})
if err != nil {
t.Fatalf("fingerprinting %s: %v", root, err)
}
sort.Strings(lines)
return strings.Join(lines, "\n")
}
// treeStatFingerprint is the instrument for trees holding a multi-GiB fixture, where hashing every
// byte costs more than it proves: name + mode + SIZE. It still catches the act being tested — the
// volume leg replacing a 2 GiB tar with a freshly written one — because that changes the size, and it
// catches an added or deleted file by name. Content-identical-but-different-bytes is the one thing it
// cannot see, which is why the small-tree tests use treeFingerprint instead.
func treeStatFingerprint(t *testing.T, root string) string {
t.Helper()
var lines []string
_ = filepath.Walk(root, func(p string, fi os.FileInfo, err error) error {
if err != nil || fi.IsDir() {
return nil
}
rel, _ := filepath.Rel(root, p)
lines = append(lines, fmt.Sprintf("%s %o %d", rel, fi.Mode().Perm(), fi.Size()))
return nil
})
sort.Strings(lines)
return strings.Join(lines, "\n")
}
// treeFileList is the weaker instrument used for Scenario F: names only, so the assertion is
// specifically about DELETION and cannot be satisfied or broken by a content change.
func treeFileList(t *testing.T, root string) []string {
t.Helper()
var names []string
_ = filepath.Walk(root, func(p string, fi os.FileInfo, err error) error {
if err != nil || fi.IsDir() {
return nil
}
rel, _ := filepath.Rel(root, p)
names = append(names, rel)
return nil
})
sort.Strings(names)
return names
}
func (h *admissionHarness) primaryRoot() string {
return PrimaryBackupPath(h.nsRoot())
}
// ── Scenario A — one decision, taken before the first byte ───────────────────────────────────────
func TestAdmission_RefusedAppWritesNothingAndIsNotStopped(t *testing.T) {
h := newAdmissionHarness(t, "privatebin", "opengist", "homebox")
h.setSpace("privatebin", 40, 60, 100)
h.setSpace("opengist", 98, 0.4, 70) // below the reserve on BOTH terms
h.setSpace("homebox", 40, 60, 100)
h.seedUnit(t, "opengist", 0)
// Scoped to the REFUSED app's own unit: its two siblings are admitted and legitimately write
// theirs, so a whole-tree fingerprint would change for the right reason and prove nothing here.
// Scenario F below takes the whole-tree view, where every app is refused.
refusedUnit := RecoveryUnitPath(h.nsRoot(), "opengist")
before := treeFingerprint(t, refusedUnit)
if before == "" {
t.Fatal("the fixture seeded no previous unit, so 'byte-identical' would be vacuously true")
}
h.runOneBackupRun()
after := treeFingerprint(t, refusedUnit)
// 1. NOT ONE of the three legs ran for the refused app.
for _, got := range h.volDumped {
if got == "opengist" {
t.Fatal("the VOLUME leg ran for a refused app — this is the R-181 defect exactly: the leg " +
"that writes the bulk was never gated, so the reserve it protects was consumed by the " +
"very step it exists to bound")
}
}
for _, got := range h.prov.infoHits {
if got == "opengist" {
t.Fatal("the CAPTURE leg was attempted for a refused app — the verdict must be taken before " +
"any write is prepared, not partway through one")
}
}
// 2. The tree is byte-identical. This is the assertion the broken code could not pass.
if after != before {
t.Fatalf("the backup tree CHANGED across a refusal.\n--- before ---\n%s\n--- after ---\n%s\n"+
"A refusal that has already rewritten the payload is the defect, not the fix", before, after)
}
// 3. The app was never stopped. DumpAppVolumesSafe stops the stack as its FIRST act, so a gate
// placed inside it would bounce the app it is refusing to back up.
for _, got := range h.prov.stopped {
if got == "opengist" {
t.Fatal("the refused app was STOPPED — the reserve check has drifted behind the stop")
}
}
// 4. Exactly ONE alert, for that app, carrying the space figures. Three legs must not mean three
// emails about one disk.
if len(h.events) != 1 {
t.Fatalf("got %d alerts, want exactly 1 (one app refused, three legs): %+v", len(h.events), h.events)
}
if h.events[0].app != "opengist" {
t.Fatalf("alert names %q, want opengist", h.events[0].app)
}
if h.events[0].usage == nil || h.events[0].usage.AvailGB != 0.4 {
t.Fatalf("the alert carries no/incorrect space figures: %+v", h.events[0].usage)
}
}
// ── Scenario B — the other apps are unaffected ───────────────────────────────────────────────────
func TestAdmission_SiblingAppsProceedAndOnlyTheRefusedOneAlerts(t *testing.T) {
h := newAdmissionHarness(t, "privatebin", "opengist", "homebox")
h.setSpace("privatebin", 40, 60, 100)
h.setSpace("opengist", 99, 0.2, 70)
h.setSpace("homebox", 40, 60, 100)
h.runOneBackupRun()
for _, app := range []string{"privatebin", "homebox"} {
if !hasStr(h.volDumped, app) {
t.Errorf("%s was not volume-dumped (dumped=%v) — one app's refusal silenced its siblings", app, h.volDumped)
}
if !hasStr(h.prov.infoHits, app) {
t.Errorf("%s was not captured (attempted=%v) — the loop did not continue past the refusal", app, h.prov.infoHits)
}
if _, err := os.Stat(RecoveryUnitManifestPath(h.nsRoot(), app)); err != nil {
t.Errorf("%s has no manifest after the run: %v — an admitted app must be backed up normally", app, err)
}
}
if len(h.events) != 1 {
t.Fatalf("got %d alerts, want exactly 1: %+v", len(h.events), h.events)
}
}
// ── Scenario C — the promise is true ─────────────────────────────────────────────────────────────
// Every claim the shipped message makes is checked against the tree it describes. The wording is NOT
// weakened to fit the behaviour; the behaviour was moved so the wording became true (§8.3).
func TestAdmission_EveryClaimInTheRefusalMessageHoldsAgainstTheTree(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
h.setSpace("opengist", 98, 0.5, 70)
h.seedUnit(t, "opengist", 0)
beforeFP := treeFingerprint(t, h.primaryRoot())
beforeList := treeFileList(t, h.primaryRoot())
h.runOneBackupRun()
msg := h.logs.String()
if !strings.Contains(msg, "REFUSED for opengist") {
t.Fatalf("no refusal was logged for opengist; log was:\n%s", msg)
}
// Claim 1: "NO database dump, NO volume dump and NO recovery-unit capture was written for it".
for _, claim := range []string{"NO database dump", "NO volume dump", "NO recovery-unit capture"} {
if !strings.Contains(msg, claim) {
t.Fatalf("the message no longer claims %q — if a leg cannot be brought under the verdict the "+
"wording must be narrowed deliberately and the gap named, not dropped silently.\n%s", claim, msg)
}
}
if len(h.volDumped) != 0 || len(h.prov.infoHits) != 0 {
t.Fatalf("the message claims no leg ran, but volume=%v capture=%v", h.volDumped, h.prov.infoHits)
}
// Claim 2: "the previous unit is untouched" — the claim that was MEASURED FALSE in R-181.
if !strings.Contains(msg, "the previous unit is untouched") {
t.Fatalf("the message dropped the untouched claim: %s", msg)
}
if got := treeFingerprint(t, h.primaryRoot()); got != beforeFP {
t.Fatalf("the message says the previous unit is untouched; the tree says otherwise.\n"+
"--- before ---\n%s\n--- after ---\n%s", beforeFP, got)
}
// Claim 3: "NOTHING was deleted".
if !strings.Contains(msg, "NOTHING was deleted") {
t.Fatalf("the message dropped the no-deletion claim: %s", msg)
}
if got := treeFileList(t, h.primaryRoot()); !equalStrs(got, beforeList) {
t.Fatalf("files disappeared across a refusal: before=%v after=%v", beforeList, got)
}
// Claim 4: the reason is named, so the operator can tell which term bound.
if !strings.Contains(msg, "headroom") {
t.Fatalf("the message does not name WHICH term bound — an operator cannot tell 'the disk is "+
"full' from 'this app's backup is too big for what is left':\n%s", msg)
}
}
// ── Scenario D — size-aware, not just headroom-aware ─────────────────────────────────────────────
// The live R-181 sequence, reproduced as a unit: the filesystem is ABOVE the reserve on both terms
// when the run reaches the app, and the app's own write is what crosses it. Under B2 this app was
// admitted at 96% and then allowed to write 2 GB.
func TestAdmission_SizeTermRefusesAnAppWhoseOwnWriteWouldCrossTheReserve(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
// 96% used of 70 GiB, 3.0 GiB free — BOTH reserve terms deliberately still clear (97% / 1.0 GiB),
// exactly as on demo-hp at 06:40:03, so a headroom-only rule starts the run.
h.setSpace("opengist", 96, 3.0, 70)
if _, r := h.m.floorVerdict(h.usage["opengist"], 0); r != floorAdmit {
t.Fatalf("fixture is wrong: the headroom term already refuses (%v), so this test would pass "+
"without a size term and prove nothing", r)
}
h.seedUnit(t, "opengist", 2<<30) // its last backup was 2 GiB — the figure measured live
before := treeStatFingerprint(t, h.primaryRoot())
h.runOneBackupRun()
if len(h.events) != 1 {
t.Fatalf("got %d alerts, want 1 — the app was admitted at 96%% and would have been allowed to "+
"write 2 GiB, which is the R-181 sequence: %+v", len(h.events), h.events)
}
if !strings.Contains(h.logs.String(), "(size)") {
t.Fatalf("the refusal was not attributed to the SIZE term:\n%s", h.logs.String())
}
if !strings.Contains(h.events[0].err, "last backup was 2.00 GiB") {
t.Fatalf("the alert does not carry the estimate that produced the refusal: %q", h.events[0].err)
}
if len(h.volDumped) != 0 {
t.Fatalf("the volume leg ran anyway: %v", h.volDumped)
}
if got := treeStatFingerprint(t, h.primaryRoot()); got != before {
t.Fatalf("the tree changed despite the size-term refusal.\nbefore=%s\nafter =%s", before, got)
}
}
// ── Scenario E — a first-ever backup is not blocked by having no history ─────────────────────────
func TestAdmission_FirstEverBackupIsAdmitted(t *testing.T) {
h := newAdmissionHarness(t, "brandnew")
h.setSpace("brandnew", 40, 600, 1000) // ample room, and NO previous unit on disk
if est, ok := h.m.estimatedWriteGiB("brandnew"); ok || est != 0 {
t.Fatalf("estimatedWriteGiB = (%v, %v) for an app with no history, want (0, false)", est, ok)
}
h.runOneBackupRun()
if len(h.events) != 0 {
t.Fatalf("a brand-new app was refused: %+v — refusing every app that has no size to estimate "+
"from would make the FIRST backup the one that can never happen", h.events)
}
if !hasStr(h.volDumped, "brandnew") || !hasStr(h.prov.infoHits, "brandnew") {
t.Fatalf("the app was not backed up (volume=%v capture=%v)", h.volDumped, h.prov.infoHits)
}
}
// ── Scenario F — the reserve still never deletes ─────────────────────────────────────────────────
func TestAdmission_NothingUnderBackupsIsEverRemoved(t *testing.T) {
h := newAdmissionHarness(t, "privatebin", "opengist", "homebox")
for _, app := range []string{"privatebin", "opengist", "homebox"} {
h.setSpace(app, 99, 0.1, 70) // every app refused — maximum pressure to "make room"
h.seedUnit(t, app, 0)
}
before := treeFileList(t, h.primaryRoot())
h.runOneBackupRun()
after := treeFileList(t, h.primaryRoot())
if !equalStrs(before, after) {
t.Fatalf("the file list changed under the reserve.\nbefore=%v\nafter =%v\n"+
"Nothing here is generational — a unit is ONE fixed path per app — so 'prune the oldest' "+
"could only mean destroying a DIFFERENT app's only local recovery unit", before, after)
}
if len(before) == 0 {
t.Fatal("the fixture seeded no files, so this test would pass against code that deleted everything")
}
}
// ── §8.1 — one verdict per app per run, and it resets between runs ───────────────────────────────
// The verdict must not be re-taken between an app's own legs. Re-deciding is how the split this fixes
// came about: DB leg admitted, volume leg admitted, capture refused — with the bulk already written.
func TestAdmission_VerdictIsTakenOncePerAppPerRunAndNotRedecidedBetweenLegs(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
reads := 0
h.m.unitSpaceFn = func(string) *UnitSpace {
reads++
if reads == 1 {
return &UnitSpace{Path: h.dir, UsedPercent: 99, AvailGB: 0.1, TotalGB: 70, UsedGB: 69.3}
}
// The disk "recovers" mid-run. A re-decided verdict would admit the capture leg here — which
// is precisely the split R-181 closes, arriving from the other direction.
return &UnitSpace{Path: h.dir, UsedPercent: 10, AvailGB: 60, TotalGB: 70, UsedGB: 7}
}
h.runOneBackupRun()
if reads != 1 {
t.Fatalf("the filesystem was read %d times for ONE app in ONE run — the verdict is being "+
"re-decided between legs, which reintroduces the split (bulk written, capture refused)", reads)
}
if len(h.prov.infoHits) != 0 {
t.Fatal("the capture leg ran after the app was refused earlier in the same run")
}
if len(h.events) != 1 {
t.Fatalf("got %d alerts, want exactly 1 per app per run: %+v", len(h.events), h.events)
}
}
// A set carried between runs is a wrong answer with a confident face: tonight's question answered
// with last night's disk.
func TestAdmission_TheRememberedSetResetsBetweenRuns(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
h.setSpace("opengist", 99, 0.1, 70)
h.runOneBackupRun()
if len(h.events) != 1 {
t.Fatalf("run 1: want 1 alert, got %+v", h.events)
}
h.setSpace("opengist", 20, 55, 70) // space freed between runs
h.runOneBackupRun()
if !hasStr(h.volDumped, "opengist") {
t.Fatal("the second run still refused the app — the previous run's verdict was carried over, " +
"so freeing space could never take effect")
}
if len(h.events) != 1 {
t.Fatalf("the second (admitted) run alerted again: %+v", h.events)
}
}
// ── §8.4 — a nil reading neither refuses nor warns, across ALL THREE legs ────────────────────────
// Unchanged behaviour, re-pinned because the decision now governs three legs instead of one: an
// unreadable filesystem must not silently stop an app being backed up at all.
func TestAdmission_UnreadableFilesystemAdmitsEveryLegAndDoesNotWarn(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
// No usage entry → the reader returns nil, which is what system.GetDiskUsage does on error.
h.runOneBackupRun()
if len(h.events) != 0 {
t.Fatalf("an unreadable filesystem produced %d alert(s): %+v — that is the drive gate's "+
"business and has its own alert", len(h.events), h.events)
}
if !hasStr(h.volDumped, "opengist") {
t.Fatal("the VOLUME leg was refused on an unreadable read — a drive that merely blipped would " +
"now stop the bulk of the backup, not just the capture")
}
if !hasStr(h.prov.infoHits, "opengist") {
t.Fatal("the CAPTURE leg was refused on an unreadable read")
}
}
// ── The estimator, through the production path (no seam) ─────────────────────────────────────────
func TestEstimatedWriteGiB_SumsTheAppsPreviousDumpsFromRealFiles(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
h.seedUnit(t, "opengist", 3<<30) // 3 GiB sparse tar + a small .sql
est, ok := h.m.estimatedWriteGiB("opengist")
if !ok {
t.Fatal("history on disk was not recognised as history")
}
if est < 3.0 || est > 3.001 {
t.Fatalf("estimate = %.4f GiB, want ~3.0 (the .tar plus the small .sql)", est)
}
// An app whose unit exists but holds no dumps yet is history-LESS, not a zero-byte estimate.
other := AppVolumeDumpPath(h.nsRoot(), "empty")
if err := os.MkdirAll(other, 0o755); err != nil {
t.Fatal(err)
}
if est, ok := h.m.estimatedWriteGiB("empty"); ok || est != 0 {
t.Fatalf("an empty unit reported history (%v, %v) — an absent dump is not a 0-byte one", est, ok)
}
}
// ── The seam is WIRED — walked as an AST, not grepped ────────────────────────────────────────────
// FOUR mechanisms in this project have been built and left disconnected (REUSE.md's seam register).
// The behavioural tests above drive the two legs that can run without Docker; the DB leg cannot, so
// its gate is pinned HERE, structurally. `strings.Contains` is deliberately not used: a commented-out
// call still contains the string, and so does a call inside dead code.
func TestAdmission_IsWiredIntoEveryProductionWriteLeg(t *testing.T) {
fset := token.NewFileSet()
file, err := parser.ParseFile(fset, "backup.go", nil, 0) // comments dropped — only real calls survive
if err != nil {
t.Fatal(err)
}
calls := map[string][]string{} // enclosing func → called names, in source order
var current string
ast.Inspect(file, func(n ast.Node) bool {
switch v := n.(type) {
case *ast.FuncDecl:
current = v.Name.Name
case *ast.CallExpr:
name := ""
switch fn := v.Fun.(type) {
case *ast.Ident:
name = fn.Name
case *ast.SelectorExpr:
name = fn.Sel.Name
}
if name != "" && current != "" {
calls[current] = append(calls[current], name)
}
}
return true
})
// 1. The run scope is opened by the orchestrator of all three legs.
if !hasStr(calls["runDBDumpsInternal"], "beginAdmissionRun") {
t.Fatal("runDBDumpsInternal does not open the admission scope — without it every leg decides " +
"independently and the per-run memo never exists, which is the pre-R-181 behaviour")
}
// 2. The DB leg consults it BEFORE the dump. Order is the whole point: a gate after the write is
// the defect, relocated.
assertGateBefore(t, calls["runDBDumpsInternal"], "admitApp", "DumpOne",
"the DATABASE leg dumps before consulting the reserve")
// 3. The volume leg consults it BEFORE the dump seam — which stops the stack as its first act.
assertGateBefore(t, calls["runVolumeDumps"], "admitApp", "dump",
"the VOLUME leg — the one that writes the bulk, and the one B2 never gated — dumps before "+
"consulting the reserve")
// 4. The capture leg, in its own file.
rfset := token.NewFileSet()
rfile, err := parser.ParseFile(rfset, "recovery_unit.go", nil, 0)
if err != nil {
t.Fatal(err)
}
capCalls := map[string][]string{}
current = ""
ast.Inspect(rfile, func(n ast.Node) bool {
switch v := n.(type) {
case *ast.FuncDecl:
current = v.Name.Name
case *ast.CallExpr:
if sel, ok := v.Fun.(*ast.SelectorExpr); ok && current != "" {
capCalls[current] = append(capCalls[current], sel.Sel.Name)
}
}
return true
})
assertGateBefore(t, capCalls["captureAllRecoveryUnits"], "admitApp", "CaptureRecoveryUnit",
"the CAPTURE leg captures before consulting the reserve")
}
// assertGateBefore checks that `gate` appears in the call list before `act`.
func assertGateBefore(t *testing.T, calls []string, gate, act, why string) {
t.Helper()
gi, ai := -1, -1
for i, c := range calls {
if c == gate && gi < 0 {
gi = i
}
if c == act && ai < 0 {
ai = i
}
}
if gi < 0 {
t.Fatalf("%s: %q is never called there at all (calls=%v)", why, gate, calls)
}
if ai < 0 {
t.Fatalf("fixture drift: %q is no longer called in that function (calls=%v) — this test can no "+
"longer see the act it is ordering the gate against", act, calls)
}
if gi > ai {
t.Fatalf("%s: %q first appears at %d, after %q at %d", why, gate, gi, act, ai)
}
}
func hasStr(hay []string, needle string) bool {
for _, s := range hay {
if s == needle {
return true
}
}
return false
}
func equalStrs(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
+37
View File
@@ -57,6 +57,13 @@ type Manager struct {
// can state a filesystem's occupancy as an input. Nil in production → `unitTargetSpace`.
unitSpaceFn func(stackName string) *UnitSpace
// admission (R-181) is the per-RUN memo of the reserve's per-app verdict, guarded by admissionMu.
// Non-nil only for the duration of a backup run (beginAdmissionRun → its closer). One verdict per
// app covers all THREE write legs — DB dump, volume dump, unit capture — because all three write
// under one per-app root; see admission.go for why it is decided lazily and never re-decided.
admissionMu sync.Mutex
admission *admissionSet
// appStop (R-166) is the crash marker for operations that stop an app, work on its data, and
// start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power
// cut in that window leaves a durable record that Recover honours at the next startup. Built in
@@ -412,6 +419,13 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
start := time.Now()
m.logger.Printf("[INFO] [backup] Starting database dump run")
// R-181: open the per-run admission scope HERE, because this function is the single orchestrator
// of all three write legs. Each app's reserve verdict is taken at its first write of this run and
// then reused by the other two legs, so a refused app writes nothing at all and is alerted once.
// The scope is closed on every exit path — a set that outlived its run would answer tonight's
// question with last night's disk.
defer m.beginAdmissionRun()()
dbs, err := DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames())
if err != nil {
m.logger.Printf("[ERROR] [backup] Database discovery failed: %v", err)
@@ -447,6 +461,15 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
continue
}
// R-181: the reserve, BEFORE the first byte of this app's backup is written. This is usually
// where an app's verdict is taken, because the DB leg runs first; the volume leg and the
// capture then read the same memo. SKIP, not FAIL — a deliberate hold is not a broken dump,
// and the operator alert (fired once, inside admitApp) is the signal that it happened.
if !m.admitApp(db.StackName) {
summary = append(summary, fmt.Sprintf("SKIP %s (reserve — app backup refused)", db.ContainerName))
continue
}
dumpDir := AppDBDumpPath(m.namespaceRoot(drivePath), db.StackName)
result := DumpOne(ctx, db, dumpDir, m.logger, m.isDebug())
@@ -541,6 +564,12 @@ func failedSummaryLines(summary []string) []string {
// variant stops the stack before its own volume check — calling it unconditionally would bounce
// every volume-less app on every nightly run. Per-stack isolation mirrors the DB loop: one app's
// failure is recorded and does not abort the others.
//
// R-181 adds the reserve to that order, and for the SAME reason: it sits ahead of DumpAppVolumesSafe,
// so a refused app is never stopped. A refusal decided inside the Safe variant would already have
// bounced the app it was refusing to back up. It sits AFTER the volume-less check because an app with
// no named volumes writes nothing in this leg — there is no first write here to gate, and consulting
// the reserve for it would only decide a verdict early on a stale reading.
func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
allOK = true
if m.stackProvider == nil {
@@ -576,6 +605,14 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
continue
}
// R-181: the reserve, ahead of DumpAppVolumesSafe so a refused app is NOT stopped. For an app
// that already has a DB this is a memo lookup taken before its DB dump; for a volume-only app
// this is where its verdict is taken, still before its first byte.
if !m.admitApp(stack.Name) {
summary = append(summary, fmt.Sprintf("SKIP %s volumes (reserve — app backup refused)", stack.Name))
continue
}
if err := dump(stack.Name); err != nil {
allOK = false
summary = append(summary, fmt.Sprintf("FAIL %s volumes: %v", stack.Name, err))
@@ -238,15 +238,16 @@ func TestFloorSitsBelowTheCriticalWarningBand(t *testing.T) {
"critically, and only then can a capture be refused")
}
// Both terms must be able to refuse INDEPENDENTLY — that is why there are two.
if _, blocked := (&Manager{}).floorVerdict(&UnitSpace{UsedPercent: 50, AvailGB: 0.5}); !blocked {
// Both terms must be able to refuse INDEPENDENTLY — that is why there are two. estGiB=0 is the
// history-less case, which exercises the headroom term alone.
if _, r := (&Manager{}).floorVerdict(&UnitSpace{UsedPercent: 50, AvailGB: 0.5}, 0); r != floorHeadroom {
t.Fatal("a filesystem with 0.5 GiB free at only 50% used was NOT refused — the free-byte term " +
"does not trip on its own, so a very large volume can run out without the floor engaging")
}
if _, blocked := (&Manager{}).floorVerdict(&UnitSpace{UsedPercent: 98, AvailGB: 40}); !blocked {
if _, r := (&Manager{}).floorVerdict(&UnitSpace{UsedPercent: 98, AvailGB: 40}, 0); r != floorHeadroom {
t.Fatal("a filesystem 98% used was NOT refused — the percentage term does not trip on its own")
}
if _, blocked := (&Manager{}).floorVerdict(&UnitSpace{UsedPercent: 50, AvailGB: 50}); blocked {
if _, r := (&Manager{}).floorVerdict(&UnitSpace{UsedPercent: 50, AvailGB: 50}, 0); r != floorAdmit {
t.Fatal("a healthy filesystem was refused")
}
}
+48 -28
View File
@@ -279,27 +279,51 @@ const (
FloorFreeGiB = 1.0
)
// ErrCaptureFloor marks a capture refused for headroom. It is a REFUSAL, not a failure of the capture
// machinery — the distinction matters to a reader of the alert, which is why the message names the
// reserve rather than reporting an I/O error.
var ErrCaptureFloor = errors.New("refused: capturing would leave the filesystem below the reserve")
// ErrCaptureFloor marks an app's backup refused for headroom. It is a REFUSAL, not a failure of the
// backup machinery — the distinction matters to a reader of the alert, which is why the message names
// the reserve rather than reporting an I/O error.
//
// R-181 widened what it covers: it now refuses the app's DB dump, volume dump and capture together
// (see admission.go), not the capture alone. The sentinel keeps its name because callers match it and
// "capture" still reads correctly for "capturing this app's backup"; the MESSAGE is what changed, and
// the message is what an operator sees.
var ErrCaptureFloor = errors.New("refused: backing up this app would leave the filesystem below the reserve")
// floorVerdict is the PURE predicate: given a reading, does the floor refuse? Separated so the
// thresholds are unit-testable without a filesystem, a stack provider or a clock.
// floorVerdict is the PURE predicate: given a reading and this app's estimated write, does the floor
// refuse, and on which term? Separated so the thresholds are unit-testable without a filesystem, a
// stack provider or a clock.
//
// TWO QUESTIONS, NOT ONE (R-181). "Is the filesystem already below the reserve?" is the headroom term
// and was all B2 asked. "Would THIS app's write take it below?" is the size term, and its absence is
// how an app was admitted at 96% used and then allowed to write 2 GB. Both terms are evaluated
// against BOTH thresholds — a large write can cross the percentage bound on a small volume and the
// free-byte bound on a large one, which is the same reason the reserve has two terms at all.
//
// §8.4 — A NIL READING NEITHER REFUSES NOR WARNS. An unreadable filesystem is the drive gate's
// business and has its own alert; refusing on it would block every capture on a box whose drive
// merely blipped, and warning on it would be a false alarm with a misleading cause.
func (m *Manager) floorVerdict(u *UnitSpace) (*UnitSpace, bool) {
// business and has its own alert; refusing on it would block every backup on a box whose drive merely
// blipped, and warning on it would be a false alarm with a misleading cause.
//
// estGiB == 0 (no previous dump to estimate from) degrades to the headroom term alone, deliberately:
// refusing an app that has never been backed up would make the FIRST backup the one that can never
// happen (Scenario E).
func (m *Manager) floorVerdict(u *UnitSpace, estGiB float64) (*UnitSpace, floorReason) {
if u == nil {
return nil, false
return nil, floorAdmit
}
return u, u.UsedPercent >= FloorUsedPercent || u.AvailGB < FloorFreeGiB
}
// unitFloorBlocked reads the target filesystem and applies the floor.
func (m *Manager) unitFloorBlocked(stackName string) (*UnitSpace, bool) {
return m.floorVerdict(m.readUnitSpace(stackName))
if u.UsedPercent >= FloorUsedPercent || u.AvailGB < FloorFreeGiB {
return u, floorHeadroom
}
if estGiB > 0 {
availAfter := u.AvailGB - estGiB
usedAfter := u.UsedPercent
if u.TotalGB > 0 {
usedAfter = (u.UsedGB + estGiB) / u.TotalGB * 100
}
if availAfter < FloorFreeGiB || usedAfter >= FloorUsedPercent {
return u, floorSize
}
}
return u, floorAdmit
}
// readUnitSpace goes through the seam when one is injected, so a test can state the filesystem's
@@ -312,8 +336,12 @@ func (m *Manager) readUnitSpace(stackName string) *UnitSpace {
}
// captureAllRecoveryUnits refreshes the recovery unit for every deployed stack. Best-effort:
// a per-app failure is logged, NOTIFIED (R-158), and does not abort the others. Since R-165 a capture
// is also REFUSED per app when the target filesystem is below the reserve (B2).
// a per-app failure is logged, NOTIFIED (R-158), and does not abort the others.
//
// R-181: the reserve is consulted through `admitApp`, which is the SAME verdict the DB-dump and
// volume-dump legs of this run already consulted for this app. When a run is in flight the answer
// here is a memo lookup — an app refused before its first write is refused here too, silently,
// because it was already alerted once. Outside a run (the periodic status refresh) it decides fresh.
func (m *Manager) captureAllRecoveryUnits() {
if m.stackProvider == nil {
return
@@ -323,16 +351,8 @@ func (m *Manager) captureAllRecoveryUnits() {
if m.settings != nil && (m.settings.IsDisconnected(drivePath) || m.settings.IsDecommissioned(drivePath)) {
continue // drive not writable — skip, the existing unit stays as-is
}
// B2: the floor, checked BEFORE anything is written, so a refused app's previous unit is left
// byte-identical rather than half-overwritten. Per app, and the loop continues.
if usage, blocked := m.unitFloorBlocked(stack.Name); blocked {
err := fmt.Errorf("%w (reserve: %.0f%% used or %.1f GiB free) — %s",
ErrCaptureFloor, FloorUsedPercent, FloorFreeGiB, usage)
m.logger.Printf("[WARN] [backup] Recovery unit capture REFUSED for %s — %v; the previous unit is untouched and NOTHING was deleted",
stack.Name, err)
if m.unitNotify != nil {
m.unitNotify(stack.Name, err, usage)
}
// The reserve, checked BEFORE anything is written. Per app, and the loop continues.
if !m.admitApp(stack.Name) {
continue
}
if err := m.CaptureRecoveryUnit(stack.Name); err != nil {