From fef07c3923824609046922215d6c3ce0d1a9cec6 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 3 Aug 2026 10:53:48 +0200 Subject: [PATCH] =?UTF-8?q?v0.193.0=20=E2=80=94=20the=20reserve=20guards?= =?UTF-8?q?=20the=20write=20that=20fills=20the=20disk,=20and=20its=20promi?= =?UTF-8?q?se=20is=20true=20(R-181)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit B2's capture floor (v0.192.0) was consulted in exactly ONE place — captureAllRecoveryUnits, which writes a few KB. The two legs that write the BULK into the same backups/primary/ tree, the DB dump and the volume dump, ran FIRST and unguarded. Measured live on demo-hp 2026-08-03 06:40:03: opengist's volume dump wrote 2.0 GB with no check, free fell to 1.0 GB, and the floor then refused the cheap write it had already lost the argument to. Its refusal message claimed "the previous unit is untouched" — measured false: that app's tar had gone 182,272 B -> 2,147,666,432 B under a stale manifest. Sixth entry in CLAUDE.md's table of shipped guarantees the code did not provide. Fix: ONE admission verdict per app per run (internal/backup/admission.go), taken before that app's FIRST write and covering all three legs — they write under one per-app root, which is why one verdict can honestly cover them. - Lazy, at the app's first write, NOT once at run start: app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. - Remembered for the run, never re-decided between an app's own legs — that is the split this closes. Reset per run. - Placed ahead of DumpAppVolumesSafe, which stops the stack as its first act, so a refused app is never bounced. After the volume-less check, which has no write. - Exactly one operator alert per refused app per run. - Leg order unchanged: volume dumps still precede the capture. The floor is now SIZE-AWARE: it asks whether THIS app's write would cross the reserve, not only whether the filesystem is already below it — which is how an app was admitted at 96% and then allowed to write 2 GB. Estimate = the app's previous .sql + .tar on disk. No history -> headroom-only, deliberately, and the alert says so. A container-based du per volume was MEASURED and rejected: 66 timed runs on demo-hp guest 9201, median ~355 ms/volume (341-404) on volumes holding tens of KB — container start-up, not the walk. Decisive on top: docker run needs the writable layer, so it can fail under exactly the pressure the reserve handles. The message was NOT weakened; the behaviour was moved so the wording became true. It now also names which term bound. Every claim is checked against a sha256 fingerprint of the tree it describes, never against the log line. Still refuses and never deletes: nothing here is generational. 11 new tests through the production functions. The DB leg cannot run without Docker, so its gate is pinned by an AST walk of backup.go asserting admitApp precedes DumpOne (strings.Contains is insufficient — a commented-out call still contains the string). 4 red-proofs demonstrated failing then restored. --- CHANGELOG.md | 70 ++ REUSE.md | 3 +- controller/README.md | 21 + controller/internal/backup/admission.go | 237 ++++++ controller/internal/backup/admission_test.go | 699 ++++++++++++++++++ controller/internal/backup/backup.go | 37 + .../internal/backup/capture_floor_test.go | 9 +- controller/internal/backup/recovery_unit.go | 76 +- 8 files changed, 1119 insertions(+), 33 deletions(-) create mode 100644 controller/internal/backup/admission.go create mode 100644 controller/internal/backup/admission_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 70eef7b..ff77657 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,75 @@ ## Changelog +### v0.193.0 — the reserve guards the write that fills the disk, and its promise is true (2026-08-03, R-181) — MinAgent: none + +**The defect, found on live hardware and not by review.** v0.192.0's capture floor (B2) shipped as the +deliberate replacement for the bulkhead the `mp1` partition used to give, and it was consulted in +**exactly one place** — `captureAllRecoveryUnits`, which writes a manifest and three compose files: a +few KB. The two legs that write the **bulk** into the same `backups/primary/` tree — the database +dump and the volume dump — ran **first** and **unguarded**. Measured on demo-hp 2026-08-03 06:40:03: +opengist's volume dump wrote **2.0 GB with no check**, free fell to 1.0 GB, and the floor then refused +the cheap write it had already lost the argument to. + +**Second limb: the refusal message asserted something the code did not provide.** It printed *"the +previous unit is untouched and NOTHING was deleted"*. *Nothing was deleted* held. *Untouched* was +**measured false** — that app's tar had gone 182,272 B → 2,147,666,432 B under a `manifest.json` whose +`created_at` and `checksums` had not moved. **This is the sixth entry in `CLAUDE.md`'s table of +shipped guarantees the code did not provide**, and the fourth of those found on live hardware. + +**The fix: ONE admission verdict per app per run, taken before that app's FIRST write, covering all +three legs** (`internal/backup/admission.go`). The three write under one per-app root, which is +exactly why one verdict can honestly cover them — and why the message may now claim what it claims. + +- **Decided lazily, at the app's first write — NOT once at the start of the run.** Space changes + during a run: app A's 2 GB dump can put app B under the reserve, and a run-start verdict would wave + B through on a reading that was true before the disk filled. That is the same class of mistake, + moved one level up. +- **Remembered for the run, never re-decided between an app's own legs.** Re-deciding reintroduces + the split this closes (DB admitted → volume admitted → capture refused, with the bulk written). + Reset per run: a set carried between runs answers tonight's question with last night's disk. +- **Placed ahead of `DumpAppVolumesSafe`, which stops the stack as its first act** — a refusal decided + inside it would already have bounced the app it is refusing to back up. It sits *after* the + volume-less check, because an app with no named volumes has no first write in that leg to gate. +- **Exactly one operator alert per refused app per run.** Three legs must not mean three emails. +- **The leg order is unchanged** — volume dumps still precede the capture so the manifests enumerate + the fresh tars (`backup.go`'s load-bearing comment). + +**The floor is now SIZE-AWARE, not merely headroom-aware.** It asks *would this app's write leave the +filesystem below the reserve?*, not only *is it below the reserve now* — which is how an app was +admitted at 96% used and then allowed to write 2 GB. The estimate is the app's **previous** `.sql` and +`.tar` already on disk: free to read, and the next write is usually close. **No history → +headroom-only**, deliberately, or the first backup would be the one that can never happen; the alert +says so when that applies. Both post-write terms are evaluated, because a large write crosses the +percentage bound on a small volume and the free-byte bound on a large one. + +**A container-based `du` per volume was measured and REJECTED, not assumed.** 66 timed runs on +demo-hp's guest 9201: **median ~355 ms per volume** (341–404 ms) on volumes holding tens of KB — the +cost is container start-up, not the walk, so it does not shrink for small apps and only grows for real +ones. Decisive on top of that: `docker run` needs the writable layer, so the measurement mechanism can +fail under exactly the disk pressure the reserve exists to handle. The previous-dump estimate also +measures the **artifact** that will be written rather than the live volume, which is the truer +predictor. The figure and the decision are recorded rather than left as a "we could do better". + +**THE MESSAGE WAS NOT WEAKENED — the behaviour was moved so the wording became true.** It still says +the previous unit is untouched and nothing was deleted, and now adds *which* term bound (headroom or +size) and the estimate that produced a size refusal. +`TestAdmission_EveryClaimInTheRefusalMessageHoldsAgainstTheTree` checks **every** claim against a +sha256 fingerprint of the tree it describes — not against the log line, because a log line is exactly +what lied here. + +**IT STILL REFUSES AND NEVER DELETES.** Unchanged and load-bearing: nothing on this filesystem is +generational, so "prune the oldest" could only mean destroying a **different** app's only local +recovery unit. `pruneStalePrimaryDirs` is not a retention policy and must never be repurposed for +headroom. + +**Tests (11 new, all through the production functions; 4 red-proofs demonstrated failing then +restored).** The refusal assertions are **tree fingerprints before and after**, never log lines. The +DB leg cannot run without Docker, so its gate is pinned by an **AST walk** of `backup.go` asserting +`admitApp` precedes `DumpOne` — `strings.Contains` is insufficient, a commented-out call still +contains the string. Red-proofs: both dump-leg gates removed (= v0.192.0) → Scenario A red, tree shown +changing; the size term removed → Scenario D red; a prune injected into the refusal path → Scenario F +red; the floor moved above the warning band → Scenario G red. + ### v0.192.0 — the capture floor replaces the bulkhead (2026-08-03, R-165 · decision B2) — MinAgent: none **Ships BEFORE the disk-layout merge it exists for, and is harmless on a box that never gets it.** diff --git a/REUSE.md b/REUSE.md index 18c4553..6535380 100644 --- a/REUSE.md +++ b/REUSE.md @@ -174,7 +174,8 @@ | `api.GracefulSelfRestart` | controller/internal/api/selfrestart.go | `(logger)` | Controller self-restart | Detached exit; bootstrap unit re-runs the image | | `Settings.AddPendingEvent/DrainPendingEvents` | controller/internal/settings/settings.go | offline event queue | Events while hub unreachable | — | | `Manager.SetUnitNotify` + `UnitSpace` (R-158/R-167, v0.191.0) | controller/internal/backup/recovery_unit.go | `(func(stack string, err error, *UnitSpace))` | THE per-app Tier-1 recovery-unit capture failure alert — fires PER APP from `captureAllRecoveryUnits`, loop continues | **OPERATOR-TIER** (`recovery_unit_capture_failed`, in the hub's `operatorOnlyEvents`). **NEVER route it to `backup_failed`** — that type is in `DefaultEnabledEvents` and carries Hungarian copy, so it emails the CUSTOMER about a failure they cannot act on (D-c; R-158's own proposal said `backup_failed` and D-c overrides it). `UnitSpace` is **nil when the target filesystem is unreadable** and renders as *"unavailable"*, never as zeros — "0 GB free" and "we could not look" are opposite diagnoses. No controller-side cooldown: the hub owns it | -| `Manager.unitFloorBlocked` / `floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` (R-165 B2, v0.192.0) | controller/internal/backup/recovery_unit.go | `(stackName) (*UnitSpace, bool)` / `(*UnitSpace) (*UnitSpace, bool)` | THE capture floor that replaces the `mp1` bulkhead — refuses ONE app's recovery-unit capture before anything is written when the target filesystem is below the reserve | **Headroom, NEVER unit size** — a per-unit cap is R-163 rebuilt inside one volume. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk | +| `Manager.admitApp` / `beginAdmissionRun` / `decideAdmission` / `estimatedWriteGiB` (R-181, v0.193.0) | controller/internal/backup/admission.go | `(stackName) bool` / `() func()` | **THE reserve gate. Call it before ANY per-app backup write** — one verdict per app per run, covering the DB dump, the volume dump and the unit capture (all three write under one per-app root) | **Decided LAZILY at the app's first write, never once at run start** — app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. **Never re-decided between an app's own legs**: that is exactly the split R-181 closed (bulk written, capture refused). **Reset per run** via the closer `beginAdmissionRun` returns. **Must sit ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided inside it has already bounced the app. Fires **exactly one** `unitNotify` per refused app per run. Nil admission set (periodic status refresh) → decides fresh, which is still once per app per sweep. Wiring pinned by an **AST walk** in `TestAdmission_IsWiredIntoEveryProductionWriteLeg`, not `strings.Contains` | +| `Manager.floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` / `floorReason` (R-165 B2 v0.192.0, size term R-181 v0.193.0) | controller/internal/backup/recovery_unit.go | `(*UnitSpace, estGiB float64) (*UnitSpace, floorReason)` | The pure two-question predicate behind `admitApp`: is the filesystem already below the reserve (`floorHeadroom`), and would THIS app's write take it below (`floorSize`)? | **Headroom is about the FILESYSTEM, never a per-unit cap** — a size cap is R-163 rebuilt inside one volume; the size term bounds the *delta*, not the unit. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. **`estGiB == 0` degrades to headroom-only on purpose** — refusing an app with no history makes the FIRST backup the one that can never happen. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk | | `fillwatch.Watcher` (`New`/`SetNotify`/`Check`) (R-167, v0.191.0) | controller/internal/fillwatch/fillwatch.go | `(statePath, logger, targetsFn, usageFn)` → `Check() error` | THE customer fill warning — warns BEFORE a filesystem fills, per FILESYSTEM (never per app: one full disk holding ten apps would fire ten times) | Emits the **pre-existing** `disk_warning`/`disk_critical` pair, which was allowlisted + copy'd + default-enabled with **no producer in any repo** until now — do NOT mint a new type beside it. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range. **Edge-triggered on ESCALATION ONLY**, state persisted; de-escalation is silent and re-arms. Hysteresis dead zone between clear (75% / 7 GiB) and warn — pinned by `TestThresholdsKeepTheirHysteresisGap`. **A nil usage read is NEVER a warning** (§8.4). The hub has **no `customerMessages` entry** for either type on purpose — an entry would override the dynamic message and discard the drive label + free space | ### Scheduler / time / UI diff --git a/controller/README.md b/controller/README.md index a0093c3..4bc36b7 100644 --- a/controller/README.md +++ b/controller/README.md @@ -623,6 +623,27 @@ Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypte The backup system implements a **3-2-1 backup architecture**. Each tier is a **complete, self-sufficient backup** — any single tier can fully restore an app. +**The reserve — per-app backup admission (v0.192.0 decision B2, widened by v0.193.0 / R-181).** +`internal/backup/admission.go`. Since the `mp1`→`mp0` merge (R-165) local backups and Docker's +data-root share one filesystem, so an unbounded backup write is a stopped box rather than a slow one. +Before **any** of an app's three local write legs runs — DB dump, volume dump, recovery-unit capture — +`admitApp` takes **one verdict for that app for that run** and the other two legs reuse it. A refused +app writes nothing at all, is **not stopped**, keeps its previous unit byte-identical, and produces +**exactly one** operator alert (`recovery_unit_capture_failed`, operator-tier). + +- **The verdict is lazy, not run-wide.** It is taken at the app's first write, because app A's dump + can put app B under the reserve; a verdict taken at run start would read a disk that no longer + exists by the time B writes. +- **It is never re-decided between an app's own legs**, and the memo is reset per run. +- **Two questions, both against two thresholds (97% used / 1 GiB free).** *Headroom*: is the + filesystem already below the reserve? *Size*: would this app's own write take it below? The size + estimate is the app's **previous** `.sql` + `.tar` already on disk. **No history → headroom-only**, + deliberately — otherwise the first backup is the one that can never happen — and the alert says so. +- **The thresholds sit beyond `fillwatch`'s critical band (95% / 2 GiB)**, so the customer is always + warned before a refusal is possible. +- **It refuses; it never deletes.** Nothing here is generational — one unit per app at one fixed path + — so "prune the oldest" could only destroy a different app's only local copy. + **Sidebar behaviour (v0.146.0).** Groups that own sub-pages — Tárhely, Biztonsági mentés, Megosztás — render as **accordions**: the header is a real `