diff --git a/CHANGELOG.md b/CHANGELOG.md index 70eef7b..ff77657 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,75 @@ ## Changelog +### v0.193.0 — the reserve guards the write that fills the disk, and its promise is true (2026-08-03, R-181) — MinAgent: none + +**The defect, found on live hardware and not by review.** v0.192.0's capture floor (B2) shipped as the +deliberate replacement for the bulkhead the `mp1` partition used to give, and it was consulted in +**exactly one place** — `captureAllRecoveryUnits`, which writes a manifest and three compose files: a +few KB. The two legs that write the **bulk** into the same `backups/primary/` tree — the database +dump and the volume dump — ran **first** and **unguarded**. Measured on demo-hp 2026-08-03 06:40:03: +opengist's volume dump wrote **2.0 GB with no check**, free fell to 1.0 GB, and the floor then refused +the cheap write it had already lost the argument to. + +**Second limb: the refusal message asserted something the code did not provide.** It printed *"the +previous unit is untouched and NOTHING was deleted"*. *Nothing was deleted* held. *Untouched* was +**measured false** — that app's tar had gone 182,272 B → 2,147,666,432 B under a `manifest.json` whose +`created_at` and `checksums` had not moved. **This is the sixth entry in `CLAUDE.md`'s table of +shipped guarantees the code did not provide**, and the fourth of those found on live hardware. + +**The fix: ONE admission verdict per app per run, taken before that app's FIRST write, covering all +three legs** (`internal/backup/admission.go`). The three write under one per-app root, which is +exactly why one verdict can honestly cover them — and why the message may now claim what it claims. + +- **Decided lazily, at the app's first write — NOT once at the start of the run.** Space changes + during a run: app A's 2 GB dump can put app B under the reserve, and a run-start verdict would wave + B through on a reading that was true before the disk filled. That is the same class of mistake, + moved one level up. +- **Remembered for the run, never re-decided between an app's own legs.** Re-deciding reintroduces + the split this closes (DB admitted → volume admitted → capture refused, with the bulk written). + Reset per run: a set carried between runs answers tonight's question with last night's disk. +- **Placed ahead of `DumpAppVolumesSafe`, which stops the stack as its first act** — a refusal decided + inside it would already have bounced the app it is refusing to back up. It sits *after* the + volume-less check, because an app with no named volumes has no first write in that leg to gate. +- **Exactly one operator alert per refused app per run.** Three legs must not mean three emails. +- **The leg order is unchanged** — volume dumps still precede the capture so the manifests enumerate + the fresh tars (`backup.go`'s load-bearing comment). + +**The floor is now SIZE-AWARE, not merely headroom-aware.** It asks *would this app's write leave the +filesystem below the reserve?*, not only *is it below the reserve now* — which is how an app was +admitted at 96% used and then allowed to write 2 GB. The estimate is the app's **previous** `.sql` and +`.tar` already on disk: free to read, and the next write is usually close. **No history → +headroom-only**, deliberately, or the first backup would be the one that can never happen; the alert +says so when that applies. Both post-write terms are evaluated, because a large write crosses the +percentage bound on a small volume and the free-byte bound on a large one. + +**A container-based `du` per volume was measured and REJECTED, not assumed.** 66 timed runs on +demo-hp's guest 9201: **median ~355 ms per volume** (341–404 ms) on volumes holding tens of KB — the +cost is container start-up, not the walk, so it does not shrink for small apps and only grows for real +ones. Decisive on top of that: `docker run` needs the writable layer, so the measurement mechanism can +fail under exactly the disk pressure the reserve exists to handle. The previous-dump estimate also +measures the **artifact** that will be written rather than the live volume, which is the truer +predictor. The figure and the decision are recorded rather than left as a "we could do better". + +**THE MESSAGE WAS NOT WEAKENED — the behaviour was moved so the wording became true.** It still says +the previous unit is untouched and nothing was deleted, and now adds *which* term bound (headroom or +size) and the estimate that produced a size refusal. +`TestAdmission_EveryClaimInTheRefusalMessageHoldsAgainstTheTree` checks **every** claim against a +sha256 fingerprint of the tree it describes — not against the log line, because a log line is exactly +what lied here. + +**IT STILL REFUSES AND NEVER DELETES.** Unchanged and load-bearing: nothing on this filesystem is +generational, so "prune the oldest" could only mean destroying a **different** app's only local +recovery unit. `pruneStalePrimaryDirs` is not a retention policy and must never be repurposed for +headroom. + +**Tests (11 new, all through the production functions; 4 red-proofs demonstrated failing then +restored).** The refusal assertions are **tree fingerprints before and after**, never log lines. The +DB leg cannot run without Docker, so its gate is pinned by an **AST walk** of `backup.go` asserting +`admitApp` precedes `DumpOne` — `strings.Contains` is insufficient, a commented-out call still +contains the string. Red-proofs: both dump-leg gates removed (= v0.192.0) → Scenario A red, tree shown +changing; the size term removed → Scenario D red; a prune injected into the refusal path → Scenario F +red; the floor moved above the warning band → Scenario G red. + ### v0.192.0 — the capture floor replaces the bulkhead (2026-08-03, R-165 · decision B2) — MinAgent: none **Ships BEFORE the disk-layout merge it exists for, and is harmless on a box that never gets it.** diff --git a/REUSE.md b/REUSE.md index 18c4553..6535380 100644 --- a/REUSE.md +++ b/REUSE.md @@ -174,7 +174,8 @@ | `api.GracefulSelfRestart` | controller/internal/api/selfrestart.go | `(logger)` | Controller self-restart | Detached exit; bootstrap unit re-runs the image | | `Settings.AddPendingEvent/DrainPendingEvents` | controller/internal/settings/settings.go | offline event queue | Events while hub unreachable | — | | `Manager.SetUnitNotify` + `UnitSpace` (R-158/R-167, v0.191.0) | controller/internal/backup/recovery_unit.go | `(func(stack string, err error, *UnitSpace))` | THE per-app Tier-1 recovery-unit capture failure alert — fires PER APP from `captureAllRecoveryUnits`, loop continues | **OPERATOR-TIER** (`recovery_unit_capture_failed`, in the hub's `operatorOnlyEvents`). **NEVER route it to `backup_failed`** — that type is in `DefaultEnabledEvents` and carries Hungarian copy, so it emails the CUSTOMER about a failure they cannot act on (D-c; R-158's own proposal said `backup_failed` and D-c overrides it). `UnitSpace` is **nil when the target filesystem is unreadable** and renders as *"unavailable"*, never as zeros — "0 GB free" and "we could not look" are opposite diagnoses. No controller-side cooldown: the hub owns it | -| `Manager.unitFloorBlocked` / `floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` (R-165 B2, v0.192.0) | controller/internal/backup/recovery_unit.go | `(stackName) (*UnitSpace, bool)` / `(*UnitSpace) (*UnitSpace, bool)` | THE capture floor that replaces the `mp1` bulkhead — refuses ONE app's recovery-unit capture before anything is written when the target filesystem is below the reserve | **Headroom, NEVER unit size** — a per-unit cap is R-163 rebuilt inside one volume. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk | +| `Manager.admitApp` / `beginAdmissionRun` / `decideAdmission` / `estimatedWriteGiB` (R-181, v0.193.0) | controller/internal/backup/admission.go | `(stackName) bool` / `() func()` | **THE reserve gate. Call it before ANY per-app backup write** — one verdict per app per run, covering the DB dump, the volume dump and the unit capture (all three write under one per-app root) | **Decided LAZILY at the app's first write, never once at run start** — app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. **Never re-decided between an app's own legs**: that is exactly the split R-181 closed (bulk written, capture refused). **Reset per run** via the closer `beginAdmissionRun` returns. **Must sit ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided inside it has already bounced the app. Fires **exactly one** `unitNotify` per refused app per run. Nil admission set (periodic status refresh) → decides fresh, which is still once per app per sweep. Wiring pinned by an **AST walk** in `TestAdmission_IsWiredIntoEveryProductionWriteLeg`, not `strings.Contains` | +| `Manager.floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` / `floorReason` (R-165 B2 v0.192.0, size term R-181 v0.193.0) | controller/internal/backup/recovery_unit.go | `(*UnitSpace, estGiB float64) (*UnitSpace, floorReason)` | The pure two-question predicate behind `admitApp`: is the filesystem already below the reserve (`floorHeadroom`), and would THIS app's write take it below (`floorSize`)? | **Headroom is about the FILESYSTEM, never a per-unit cap** — a size cap is R-163 rebuilt inside one volume; the size term bounds the *delta*, not the unit. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. **`estGiB == 0` degrades to headroom-only on purpose** — refusing an app with no history makes the FIRST backup the one that can never happen. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk | | `fillwatch.Watcher` (`New`/`SetNotify`/`Check`) (R-167, v0.191.0) | controller/internal/fillwatch/fillwatch.go | `(statePath, logger, targetsFn, usageFn)` → `Check() error` | THE customer fill warning — warns BEFORE a filesystem fills, per FILESYSTEM (never per app: one full disk holding ten apps would fire ten times) | Emits the **pre-existing** `disk_warning`/`disk_critical` pair, which was allowlisted + copy'd + default-enabled with **no producer in any repo** until now — do NOT mint a new type beside it. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range. **Edge-triggered on ESCALATION ONLY**, state persisted; de-escalation is silent and re-arms. Hysteresis dead zone between clear (75% / 7 GiB) and warn — pinned by `TestThresholdsKeepTheirHysteresisGap`. **A nil usage read is NEVER a warning** (§8.4). The hub has **no `customerMessages` entry** for either type on purpose — an entry would override the dynamic message and discard the drive label + free space | ### Scheduler / time / UI diff --git a/controller/README.md b/controller/README.md index a0093c3..4bc36b7 100644 --- a/controller/README.md +++ b/controller/README.md @@ -623,6 +623,27 @@ Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypte The backup system implements a **3-2-1 backup architecture**. Each tier is a **complete, self-sufficient backup** — any single tier can fully restore an app. +**The reserve — per-app backup admission (v0.192.0 decision B2, widened by v0.193.0 / R-181).** +`internal/backup/admission.go`. Since the `mp1`→`mp0` merge (R-165) local backups and Docker's +data-root share one filesystem, so an unbounded backup write is a stopped box rather than a slow one. +Before **any** of an app's three local write legs runs — DB dump, volume dump, recovery-unit capture — +`admitApp` takes **one verdict for that app for that run** and the other two legs reuse it. A refused +app writes nothing at all, is **not stopped**, keeps its previous unit byte-identical, and produces +**exactly one** operator alert (`recovery_unit_capture_failed`, operator-tier). + +- **The verdict is lazy, not run-wide.** It is taken at the app's first write, because app A's dump + can put app B under the reserve; a verdict taken at run start would read a disk that no longer + exists by the time B writes. +- **It is never re-decided between an app's own legs**, and the memo is reset per run. +- **Two questions, both against two thresholds (97% used / 1 GiB free).** *Headroom*: is the + filesystem already below the reserve? *Size*: would this app's own write take it below? The size + estimate is the app's **previous** `.sql` + `.tar` already on disk. **No history → headroom-only**, + deliberately — otherwise the first backup is the one that can never happen — and the alert says so. +- **The thresholds sit beyond `fillwatch`'s critical band (95% / 2 GiB)**, so the customer is always + warned before a refusal is possible. +- **It refuses; it never deletes.** Nothing here is generational — one unit per app at one fixed path + — so "prune the oldest" could only destroy a different app's only local copy. + **Sidebar behaviour (v0.146.0).** Groups that own sub-pages — Tárhely, Biztonsági mentés, Megosztás — render as **accordions**: the header is a real `