felhom.eu/REPORT.md

# REPORT — R-181 (the reserve guards the write that fills the disk) + R-156 (papra) + two operator rulings

**Date:** 2026-08-03 · **Repos:** `felhom-controller` (v0.192.0 → **v0.193.1**), `app-catalog-felhom.eu`, `felhom.eu` (docs only — **no hub change, no hub version bump**)

## 1. Baselines — re-read on arrival, all matched §1

| Repo | `main` @ arrival | Version | Shipped |
|---|---|---|---|
| `felhom-controller` | `4be6467b501b` | v0.192.0 | **v0.193.0 `fef07c3`** → **v0.193.1 `6c43bf6`** |
| `app-catalog-felhom.eu` | `7cb58ecdf8e7` | n/a | `122bbee` |
| `felhom.eu` | `6b5d64c1fa73` | hub v0.89.0 | docs only, **no bump** |

All three clean (`git status --porcelain` empty, `HEAD == origin/main`) before every build.

## 2. The fix

**One admission verdict per app per run** (`controller/internal/backup/admission.go`), taken before
that app's **first** write and consulted by all three legs — DB dump, volume dump, unit capture. The
three write under one per-app root (`appbackup.RecoveryUnitPath`), which is what makes one verdict
able to cover them honestly.

- **Lazy, not run-wide.** App A's dump can put app B under the reserve; a run-start verdict reads a
  disk that no longer exists. **Never re-decided between an app's own legs** — that is the split being
  closed. **Reset per run.**
- **Ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act, so a refused app is never
  bounced. **After** the volume-less check, which has no write to gate.
- **Exactly one operator alert per refused app per run.** Leg order unchanged.
- **Size term added:** *would this app's write cross the reserve?* — estimated from its previous
  `.sql` + `.tar`. **No history → headroom-only**, or the first backup becomes the one that can never
  happen; the alert says so when that applies.

## 3. Files

| File | |
|---|---|
| `controller/internal/backup/admission.go` | **new** — the gate, the memo, the estimator |
| `controller/internal/backup/admission_test.go` | **new** — 11 tests |
| `controller/internal/backup/backup.go` | run scope + gates in the DB and volume legs |
| `controller/internal/backup/recovery_unit.go` | `floorVerdict` size-aware; capture leg via `admitApp` |
| `controller/internal/backup/capture_floor_test.go` | 3 call sites updated for the new signature |
| `controller/README.md`, `REUSE.md`, `CHANGELOG.md` | |
| `app-catalog-felhom.eu/templates/papra/docker-compose.yml` | mount moved to `/app/app-data` |

## 4. Tests — 28 packages `ok`, `rc=0` (read separately from any commit)

All 11 new tests pass, plus the pre-existing floor suite. Refusal assertions are **sha256 tree
fingerprints before and after**, never log lines — the defect being fixed *is* a log line the tree
contradicted.

The DB leg cannot run without Docker (`DiscoverDatabases` shells out), so its gate is pinned by an
**AST walk** of `backup.go` asserting `admitApp` precedes `DumpOne`. `strings.Contains` is
insufficient: a commented-out call still contains the string.

### Red-proofs — each demonstrated failing, then restored

| # | Mutation | Result |
|---|---|---|
| 1 | **Both** dump-leg `admitApp` gates removed (= exactly v0.192.0) | Scenario A **RED** — *"the VOLUME leg ran for a refused app"*; with the leg assertions temporarily made non-fatal, the **tree fingerprint changed** too. Also red: Scenario C, Scenario D, and the AST wiring test (which named the DB leg specifically) |
| 2 | The entire size term removed from `floorVerdict` (both its thresholds) | Scenario D **RED** — 0 alerts where 1 was required |
| 3a | The reserve removed entirely | Scenario F **PASSED — recorded honestly.** The specified mutation does not exercise the assertion: removing the reserve makes every app write, which overwrites and adds but **deletes nothing**, so a deletion-watching test correctly stays green |
| 3b | A prune injected into the refusal path | Scenario F **RED** — this is the mutation that proves the test watches deletion |
| 4 | Floor moved above the warning band (90% / 6 GiB) | `TestFloorSitsBelowTheCriticalWarningBand` **RED** |

Every mutation removed **every** guard its test covers (#1 removed both dump-leg gates, not one).

## 5. Live validation — demo-hp guest 9201 (Tier 0), the method that found the defect

**Method:** endpoint-level — `POST /api/debug/backup/dbdump`, the exact endpoint the debug UI button
calls, which runs the production `RunDBDumps`. No browser on DooPlex.

**The instrument was re-proven before use.** demo-hp's thin pool is 53.93 GiB, so a real fill of a
70 G volume would exhaust it and corrupt every guest. A 5 GiB `fallocate` step moved guest `df`
1.2G → 6.2G while thin-pool `data_percent` held **36.83 → 36.83** — zero blocks allocated. Re-checked
at every step of the fill.

### Headroom term — 08:59:46, 906 MB free / 99% used

| Observable | Result |
|---|---|
| Tree fingerprint before | `TREE_SHA=111d1760c18d3440f700634ab325f8b8` (10 files; opengist's tar **182,272 B** — R-181's own "before" figure) |
| Tree fingerprint after | **`111d1760c18d3440f700634ab325f8b8` — identical** |
| Volume dumps written | **0** (baseline run at 08:58 wrote 2) |
| `Stopping <app> for safe volume dump` | **absent** — and this is evidence, not an absence, because that line **is** present in the 08:58 baseline |
| Operator alerts | one `recovery_unit_capture_failed` per app, severity `error`, HTTP 200 |

Free space restored → re-run at **09:01:33**: both apps captured normally.

### Size term — 09:03:00, proven separately

Reproducing the original sequence: a real 2 GiB file planted in opengist's volume, backed up so its
**previous** tar became **2,147,666,432 B** (the exact live figure), then the filesystem set to
**91% used / 2.9 GB free — both headroom terms deliberately clear**.

- **opengist refused `(size)`** — *"this app's last backup was 2.0 GB and writing it again would cross the reserve"*
- **privatebin ADMITTED and dumped normally** — the term is per-app, not a global halt
- Tree unchanged; 1 volume dump instead of 2

### One honest correction to the "app not stopped" claim

`StartedAt` on both apps *did* move, 26 s **after** the refusal. It was the **quiesce loop** for the
whole-guest PBS backup, which my fill had broken — not the app-data path. Its own backoff logic then
behaved correctly (*"deferring its next quiesce by 15m so the apps are not stopped again for a backup
that cannot succeed"*). The app-data claim rests on the **absence of the `Stopping … for safe volume
dump` line**, which is the line that appears when that leg bounces an app.

## 6. The `du` measurement (§Part 1.3) — measured, then rejected

**66 timed runs** on demo-hp guest 9201, `docker run --rm -v <vol>:/v alpine du -sb /v`:
**median ~355 ms per volume, range 341–404 ms** — on volumes holding **tens of KB**. The cost is
container start-up, not the walk, so it does not shrink for small apps and only grows for real ones.

**Rejected**, on two grounds beyond the number: `docker run` needs the writable layer, so the
measurement mechanism can fail under exactly the disk pressure the reserve exists to handle; and the
previous-dump estimate measures the **artifact that will be written** rather than the live volume,
which is the truer predictor. The previous-dump estimate stands.

## 7. The refusal message as shipped, and what it guarantees

```
[WARN] [backup] App backup REFUSED for opengist (headroom) — refused: backing up this app would
leave the filesystem below the reserve (reserve: 97% used or 1.0 GiB free; the filesystem is already
below it, before this app's estimated 178.0 KB write) — /mnt/sys_drive: 64.3/68.7 GB used (94%),
0.9 GB free; NO database dump, NO volume dump and NO recovery-unit capture was written for it, the
previous unit is untouched and NOTHING was deleted
```

**It guarantees, for that app in that run:** no DB dump, no volume dump and no capture were written;
every file under `backups/primary/<app>` is byte-identical; the app was not stopped; nothing anywhere
was deleted; exactly one operator alert was sent. All five verified by fingerprint above.

**The wording was not weakened to fit the behaviour** — the behaviour moved so the wording became
true. What was *added* is the bound term (`headroom` / `size`) and the estimate.

**v0.193.1 — found by this very proof run.** The estimate was rendered fixed to two-decimal GiB, so
opengist's real **178 KB** printed as `estimated 0.00 GiB write`, which reads as *no estimate was
available* — the opposite of what happened. Shipped the same session because it is the same defect
class the whole task is about. Re-verified live after redeploy: `estimated 178.0 KB write`.

## 8. papra (R-156, last leg)

**Precondition checked, not inherited** — both boxes were wiped and rebuilt today, so the 2 August
evidence was re-measured: `docker ps -a` (**including stopped**) on **both** demo guests → no papra;
hub `/hosts` → exactly two enrolled hosts (`demo-felhom-8363b5`, `demo-hp-bb76ea`), **zero** papra.

**Decided from the image, not the README:** `WORKDIR=/app`, `DATABASE_URL=file:./app-data/db/db.sqlite`,
`DOCUMENT_STORAGE_FILESYSTEM_ROOT=./app-data/documents`, `PAPRA_CONFIG_DIR=./app-data` — and
**`/app/data` does not exist in the image at all**.

**Departure from the task's stated preference order, stated because it was deliberate.** Option (1)
(reconfigure the app to write to `/app/data`) *was* available — all three paths are env-settable. Not
taken: it enumerates data paths, so a fourth added upstream would silently escape to the writable
layer again — this defect re-armed and invisible. Mounting the app's own data **root** captures every
current and future path by construction.

**Gate output — the arbiter, run in both directions:**

- fixed → `papra CLEAN`, with the self-test passing on that run: *"prober flags the R-156 signature and clears a correct template — trustworthy"*
- reverted to `/app/data` (red-proof on the **real template**, not just the canary) → `BROKEN`: *"mount /app/data is NOT writable by the app's own uid=999"*, *"DATA in the writable layer at /app/app-data/db (db_signature=True, e.g. ['db.sqlite'])"*, *"declared volume /app/data is EMPTY"*
- `catalog_gates.py papra` (full, not `--fast`) → **rc=0**, all three gates OK

**Two operational findings about the gate:** it needs **root** (it reads `/var/lib/docker/volumes`,
mode `drwx--x---`; as a normal user its own canary fails UNDETERMINED and it correctly refuses a
verdict — fail-closed working as designed), and it hardcodes scratch path `/srv/felhom-gate`, created
on DooPlex. Unscoped it deploys all 53 templates; that run was aborted after 10 minutes and its
`volgate-*` scratch projects were cleaned up.

## 9. §3's correction — confirmed in passing, not chased

`restore_points.go:57-59` takes the manifest's mtime and then `newestArtifact` over the `.sql` and
`.tar` files, so **the newest of the three wins**. The restore point does **not** show a stale
timestamp. Confirmed and dropped, as instructed.

## 10. Register

| ID | Change |
|---|---|
| **R-181** | **CLOSED — SHIPPED** (v0.193.0 + v0.193.1), with the live evidence above |
| **R-156** | **CLOSED** — all three apps fixed |
| **R-110** | WAITING-ON-OPERATOR → **READY**, ruling attached: **option (b), tag-tracked**, and it must cover **both** channels (the `/scripts/` git-sync *and* the nine files fetched from `raw/branch/main`) or it only half-works |
| **R-115** | WAITING-ON-OPERATOR → **READY**, ruling attached: **mechanism (b)**, a build-side gate refusing to deploy or vouch an unpublished version; the third instance (agent v0.120.0) would have silently downgraded both demo boxes while reporting success |
| **R-182** | **NEW.** ID established free: `grep -ro "R-182\b"` over `documentation/` and `*.md` → 2 hits, both prose in `REPORT.md` recording it as *"checked and left unused"*; `R-183` → 0 hits and remains free |

**R-165** is collapsed to CLOSED/PROVEN-LIVE in `ROADMAP.md`; the capability map's local-backup row
moves to **PROVEN-LIVE, both halves**, because the live fill proved the fixed behaviour for **both**
reserve terms.

## 11. Observations — noticed, documented, NOT acted on

1. **R-182 (filed).** The periodic status refresh (`GetFullStatus` → `captureAllRecoveryUnits`) runs
   with no admission scope, so a refused app re-alerts on every poll — measured live: a second
   identical alert pair 13 s after the run's. **Pre-existing in v0.192.0**; R-181 changed neither
   caller. Its mitigation is a *comment* claiming the hub owns cooldown — which is exactly the
   "invariant asserted in a comment with no test pinning it" shape, so verify at the hub before
   scoping.
2. **A reserve refusal does not make the run fail.** The DB and volume legs record `SKIP`, not `FAIL`,
   so `lastDBDump.Success` stays true and the customer-facing status does not turn red. Deliberate and
   consistent with v0.192.0 (the capture refusal never set it either), and the operator alert is the
   signal — but it means "backup succeeded" and "every app was backed up" are not the same statement.
3. **`UnitSpace.UsedPercent` and `df` disagree** — `df` reported 99% where the alert said 94%, because
   `df`'s figure accounts for ext4 reserved blocks and the floor's does not. Harmless here (the
   free-byte term bound), but a percent-term threshold is being compared against a number the operator
   cannot reproduce with `df`.
4. **The whole-guest PBS backup fails when the volume is near-full**, pushing
   `whole_guest_backup_failed` (severity `error`). Expected under a deliberate fill, and its backoff
   behaved correctly; noted because it is collateral any future fill test will also produce.

## 12. Teardown

Fill file removed; the planted 2 GiB file removed; a final backup regenerated a correct 178 KB tar;
`pct fstrim 9201` returned 67.5 GiB and the thin pool settled at **29.43%**, *below* its 36.83%
baseline. The backups tree is byte-identical to the pre-test fingerprint. Guest helper scripts and the
credential file `shred`-ed. `volgate-*` scratch compose projects removed; the unrelated 9-day-old
`jarr-*` containers on DooPlex were left untouched. papra is **not** left deployed.

No `--no-verify` was used on any push; the `felhom-controller` pre-push hook ran and reported
`gates OK` on both pushes.