felhom-controller/REPORT.md

# REPORT — R-181: the reserve guards the write that fills the disk, and its promise is true

**Date:** 2026-08-03 · **Repo:** `felhom-controller` · **v0.192.0 → v0.193.0 → v0.193.1**
**Commits on `main`:** `fef07c3` (v0.193.0), `6c43bf6` (v0.193.1) · **Baseline on arrival:** `4be6467b501b`, v0.192.0 (matched)

## The defect

B2's capture floor (v0.192.0, R-165) shipped as the deliberate replacement for the bulkhead the `mp1`
partition used to give, and it was consulted in **exactly one place** — `captureAllRecoveryUnits`,
which writes a manifest and three compose files: a few KB. The two legs that write the **bulk** into
the same `backups/primary/<app>` tree — the database dump and the volume dump — ran **first** and
**unguarded**.

Measured live on demo-hp 2026-08-03 06:40:03: opengist's volume dump wrote **2.0 GB with no check**,
free fell to 1.0 GB, and the floor then refused the cheap write it had already lost the argument to.

**Second limb:** the refusal printed *"the previous unit is untouched and NOTHING was deleted"*.
*Nothing was deleted* held. *Untouched* was **measured false** — that app's tar had gone
182,272 B → 2,147,666,432 B under a `manifest.json` whose `created_at` and `checksums` had not moved.
Sixth entry in `CLAUDE.md`'s table of shipped guarantees the code did not provide.

## The fix

**One admission verdict per app per run** (`internal/backup/admission.go`), taken before that app's
**first** write and consulted by all three legs. They write under one per-app root
(`appbackup.RecoveryUnitPath`), which is what makes one verdict able to cover them honestly.

- **Lazy, at the app's first write — not once at run start.** App A's dump can put app B under the
  reserve; a run-start verdict reads a disk that no longer exists.
- **Never re-decided between an app's own legs** (that is the split being closed); **reset per run**.
- **Ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act. **After** the volume-less
  check, which has no write to gate.
- **Exactly one operator alert per refused app per run.** Leg order unchanged.
- **Size term:** *would this app's write cross the reserve?*, estimated from its previous `.sql` +
  `.tar`. **No history → headroom-only**, or the first backup is the one that can never happen.

## Files

`internal/backup/admission.go` (new), `admission_test.go` (new, 11 tests), `backup.go` (run scope +
DB-leg and volume-leg gates), `recovery_unit.go` (`floorVerdict` size-aware; capture leg via
`admitApp`), `capture_floor_test.go` (3 call sites), `controller/README.md`, `REUSE.md`, `CHANGELOG.md`.

## Tests — `go build ./... && go vet ./... && go test ./...` → **28 packages ok, rc=0**

Read separately from any commit (standing rule 1). Refusal assertions are **sha256 tree fingerprints
before and after**, never log lines — the defect *is* a log line the tree contradicted.

The DB leg cannot run without Docker (`DiscoverDatabases` shells out), so its gate is pinned by an
**AST walk** of `backup.go` asserting `admitApp` precedes `DumpOne`; `strings.Contains` is
insufficient, a commented-out call still contains the string.

### Red-proofs — demonstrated failing, then restored

| Mutation | Result |
|---|---|
| **Both** dump-leg gates removed (= exactly v0.192.0) | Scenario A RED (*"the VOLUME leg ran for a refused app"*), and with the leg assertions temporarily non-fatal the **tree fingerprint changed** too. Also red: C, D, and the AST wiring test |
| The whole size term removed from `floorVerdict` | Scenario D RED — 0 alerts where 1 required |
| The reserve removed entirely | Scenario F **PASSED — recorded honestly.** It does not exercise the assertion: every app then writes, which overwrites and adds but **deletes nothing**, so a deletion-watching test correctly stays green |
| A prune injected into the refusal path | Scenario F RED — the mutation that actually proves it watches deletion |
| Floor moved above the warning band (90% / 6 GiB) | `TestFloorSitsBelowTheCriticalWarningBand` RED |

`python3 controller/scripts/controller_gates.py` → all 8 gates OK.

## Live validation — demo-hp guest 9201 (Tier 0)

**Method:** endpoint-level — `POST /api/debug/backup/dbdump`, the exact endpoint the debug UI button
calls, running the production `RunDBDumps`. No browser on DooPlex.

**Instrument re-proven first:** demo-hp's thin pool is 53.93 GiB, so a real fill of the 70 G volume
would exhaust it. A 5 GiB `fallocate` moved guest `df` 1.2G → 6.2G while `data_percent` held
**36.83 → 36.83** — zero blocks allocated. Re-checked at every fill step.

**Headroom term, 08:59:46 (906 MB free / 99%):** `TREE_SHA=111d1760c18d3440f700634ab325f8b8`
**identical before and after** (opengist's tar still 182,272 B); **0** volume dumps; **no
`Stopping <app> for safe volume dump` line at all** — evidence rather than absence, because that line
*is* present in the 08:58 baseline run; one alert per app, HTTP 200. Freed and re-run 09:01:33 → both
captured normally.

**Size term, 09:03:00, proven separately:** a real 2 GiB file in opengist's volume made its previous
tar **2,147,666,432 B** (the exact live figure); filesystem set to **91% used / 2.9 GB free — both
headroom terms deliberately clear**. opengist refused **`(size)`**; **privatebin was ADMITTED and
dumped normally**, so the term is per-app and not a global halt.

**One honest correction:** both apps' `StartedAt` did move, 26 s *after* the refusal — the **quiesce
loop** for the whole-guest PBS backup, which the fill had broken, not the app-data path. The
app-data claim rests on the absent `Stopping … for safe volume dump` line.

## The `du` measurement — measured, then rejected

**66 timed runs**, `docker run --rm -v <vol>:/v alpine du -sb /v`: **median ~355 ms/volume
(341–404 ms)** on volumes holding tens of KB — container start-up, not the walk. Rejected on two
further grounds: `docker run` needs the writable layer, so the instrument can fail under exactly the
pressure the reserve handles; and the previous-dump estimate measures the **artifact to be written**
rather than the live volume. The estimate stands.

## v0.193.1 — found by the proof run itself

The estimate was rendered fixed to two-decimal GiB, so opengist's real **178 KB** printed as
`estimated 0.00 GiB write` — which reads as *no estimate was available*, the opposite of what
happened. Shipped the same session because it is the same defect class. Arithmetic unchanged (still
GiB, the reserve's own unit); rendering moved to `humanizeBytes`. Re-verified live after redeploy:
`estimated 178.0 KB write`.

## Deployed

`gitea.dooplex.hu/admin/felhom-controller:0.193.1` on demo-hp guest 9201 — `Up (healthy)`.
Not deployed to demo-felhom (which carries the PBS-DR/offsite tier and was out of scope).

## §3's correction — confirmed, not chased

`restore_points.go:57-59` takes the manifest mtime then `newestArtifact` over `.sql` and `.tar`, so
the newest of the three wins. The restore point does **not** show a stale timestamp.

## Teardown

Fill removed; planted 2 GiB file removed; a final backup regenerated a correct 178 KB tar;
`pct fstrim 9201` returned 67.5 GiB and the pool settled at **29.43%**, below its 36.83% baseline; the
tree is byte-identical to the pre-test fingerprint. Guest helper scripts and the credential file
`shred`-ed. No `--no-verify` on either push; the pre-push hook ran and reported `gates OK` both times.

## Observations — NOT acted on (filed in `felhom.eu/documentation/backlog/OPEN-ITEMS.md`)

1. **R-182 (filed).** `GetFullStatus` → `captureAllRecoveryUnits` runs with no admission scope, so a
   refused app re-alerts on every status poll — measured: a second identical alert pair 13 s after the
   run's. **Pre-existing in v0.192.0**; this change touched neither caller.
2. **A reserve refusal does not fail the run.** Both dump legs record `SKIP`, not `FAIL`, so
   `lastDBDump.Success` stays true. Deliberate and consistent with v0.192.0, but "backup succeeded"
   and "every app was backed up" are then not the same statement.
3. **`UnitSpace.UsedPercent` and `df` disagree** (94% vs 99%) — `df` accounts for ext4 reserved blocks
   and the floor's figure does not. Harmless here, but the operator cannot reproduce the percent term
   with `df`.