felhom.eu/REPORT.md

REPORT — R-181 (the reserve guards the write that fills the disk) + R-156 (papra) + two operator rulings

Date: 2026-08-03 · Repos: felhom-controller (v0.192.0 → v0.193.1), app-catalog-felhom.eu, felhom.eu (docs only — no hub change, no hub version bump)

1. Baselines — re-read on arrival, all matched §1

Repo	main @ arrival	Version	Shipped
felhom-controller	4be6467b501b	v0.192.0	v0.193.0 fef07c3 → v0.193.1 6c43bf6
app-catalog-felhom.eu	7cb58ecdf8e7	n/a	122bbee
felhom.eu	6b5d64c1fa73	hub v0.89.0	docs only, no bump

All three clean (git status --porcelain empty, HEAD == origin/main) before every build.

2. The fix

One admission verdict per app per run (controller/internal/backup/admission.go), taken before that app's first write and consulted by all three legs — DB dump, volume dump, unit capture. The three write under one per-app root (appbackup.RecoveryUnitPath), which is what makes one verdict able to cover them honestly.

• Lazy, not run-wide. App A's dump can put app B under the reserve; a run-start verdict reads a disk that no longer exists. Never re-decided between an app's own legs — that is the split being closed. Reset per run.
• Ahead of DumpAppVolumesSafe, which stops the stack as its first act, so a refused app is never bounced. After the volume-less check, which has no write to gate.
• Exactly one operator alert per refused app per run. Leg order unchanged.
• Size term added: would this app's write cross the reserve? — estimated from its previous .sql + .tar. No history → headroom-only, or the first backup becomes the one that can never happen; the alert says so when that applies.

3. Files

File
controller/internal/backup/admission.go	new — the gate, the memo, the estimator
controller/internal/backup/admission_test.go	new — 11 tests
controller/internal/backup/backup.go	run scope + gates in the DB and volume legs
controller/internal/backup/recovery_unit.go	floorVerdict size-aware; capture leg via admitApp
controller/internal/backup/capture_floor_test.go	3 call sites updated for the new signature
controller/README.md, REUSE.md, CHANGELOG.md
app-catalog-felhom.eu/templates/papra/docker-compose.yml	mount moved to /app/app-data

4. Tests — 28 packages ok, rc=0 (read separately from any commit)

All 11 new tests pass, plus the pre-existing floor suite. Refusal assertions are sha256 tree fingerprints before and after, never log lines — the defect being fixed is a log line the tree contradicted.

The DB leg cannot run without Docker (DiscoverDatabases shells out), so its gate is pinned by an AST walk of backup.go asserting admitApp precedes DumpOne. strings.Contains is insufficient: a commented-out call still contains the string.

Red-proofs — each demonstrated failing, then restored

#	Mutation	Result
1	Both dump-leg admitApp gates removed (= exactly v0.192.0)	Scenario A RED — "the VOLUME leg ran for a refused app"; with the leg assertions temporarily made non-fatal, the tree fingerprint changed too. Also red: Scenario C, Scenario D, and the AST wiring test (which named the DB leg specifically)
2	The entire size term removed from floorVerdict (both its thresholds)	Scenario D RED — 0 alerts where 1 was required
3a	The reserve removed entirely	Scenario F PASSED — recorded honestly. The specified mutation does not exercise the assertion: removing the reserve makes every app write, which overwrites and adds but deletes nothing, so a deletion-watching test correctly stays green
3b	A prune injected into the refusal path	Scenario F RED — this is the mutation that proves the test watches deletion
4	Floor moved above the warning band (90% / 6 GiB)	TestFloorSitsBelowTheCriticalWarningBand RED

Every mutation removed every guard its test covers (#1 removed both dump-leg gates, not one).

5. Live validation — demo-hp guest 9201 (Tier 0), the method that found the defect

Method: endpoint-level — POST /api/debug/backup/dbdump, the exact endpoint the debug UI button calls, which runs the production RunDBDumps. No browser on DooPlex.

The instrument was re-proven before use. demo-hp's thin pool is 53.93 GiB, so a real fill of a 70 G volume would exhaust it and corrupt every guest. A 5 GiB fallocate step moved guest df 1.2G → 6.2G while thin-pool data_percent held 36.83 → 36.83 — zero blocks allocated. Re-checked at every step of the fill.

Headroom term — 08:59:46, 906 MB free / 99% used

Observable	Result
Tree fingerprint before	TREE_SHA=111d1760c18d3440f700634ab325f8b8 (10 files; opengist's tar 182,272 B — R-181's own "before" figure)
Tree fingerprint after	111d1760c18d3440f700634ab325f8b8 — identical
Volume dumps written	0 (baseline run at 08:58 wrote 2)
Stopping <app> for safe volume dump	absent — and this is evidence, not an absence, because that line is present in the 08:58 baseline
Operator alerts	one recovery_unit_capture_failed per app, severity error, HTTP 200

Free space restored → re-run at 09:01:33: both apps captured normally.

Size term — 09:03:00, proven separately

Reproducing the original sequence: a real 2 GiB file planted in opengist's volume, backed up so its previous tar became 2,147,666,432 B (the exact live figure), then the filesystem set to 91% used / 2.9 GB free — both headroom terms deliberately clear.

• opengist refused (size) — "this app's last backup was 2.0 GB and writing it again would cross the reserve"
• privatebin ADMITTED and dumped normally — the term is per-app, not a global halt
• Tree unchanged; 1 volume dump instead of 2

One honest correction to the "app not stopped" claim

StartedAt on both apps did move, 26 s after the refusal. It was the quiesce loop for the whole-guest PBS backup, which my fill had broken — not the app-data path. Its own backoff logic then behaved correctly ("deferring its next quiesce by 15m so the apps are not stopped again for a backup that cannot succeed"). The app-data claim rests on the absence of the Stopping … for safe volume dump line, which is the line that appears when that leg bounces an app.

6. The du measurement (§Part 1.3) — measured, then rejected

66 timed runs on demo-hp guest 9201, docker run --rm -v <vol>:/v alpine du -sb /v: median ~355 ms per volume, range 341–404 ms — on volumes holding tens of KB. The cost is container start-up, not the walk, so it does not shrink for small apps and only grows for real ones.

Rejected, on two grounds beyond the number: docker run needs the writable layer, so the measurement mechanism can fail under exactly the disk pressure the reserve exists to handle; and the previous-dump estimate measures the artifact that will be written rather than the live volume, which is the truer predictor. The previous-dump estimate stands.

7. The refusal message as shipped, and what it guarantees

[WARN] [backup] App backup REFUSED for opengist (headroom) — refused: backing up this app would
leave the filesystem below the reserve (reserve: 97% used or 1.0 GiB free; the filesystem is already
below it, before this app's estimated 178.0 KB write) — /mnt/sys_drive: 64.3/68.7 GB used (94%),
0.9 GB free; NO database dump, NO volume dump and NO recovery-unit capture was written for it, the
previous unit is untouched and NOTHING was deleted

It guarantees, for that app in that run: no DB dump, no volume dump and no capture were written; every file under backups/primary/<app> is byte-identical; the app was not stopped; nothing anywhere was deleted; exactly one operator alert was sent. All five verified by fingerprint above.

The wording was not weakened to fit the behaviour — the behaviour moved so the wording became true. What was added is the bound term (headroom / size) and the estimate.

v0.193.1 — found by this very proof run. The estimate was rendered fixed to two-decimal GiB, so opengist's real 178 KB printed as estimated 0.00 GiB write, which reads as no estimate was available — the opposite of what happened. Shipped the same session because it is the same defect class the whole task is about. Re-verified live after redeploy: estimated 178.0 KB write.

8. papra (R-156, last leg)

Precondition checked, not inherited — both boxes were wiped and rebuilt today, so the 2 August evidence was re-measured: docker ps -a (including stopped) on both demo guests → no papra; hub /hosts → exactly two enrolled hosts (demo-felhom-8363b5, demo-hp-bb76ea), zero papra.

Decided from the image, not the README: WORKDIR=/app, DATABASE_URL=file:./app-data/db/db.sqlite, DOCUMENT_STORAGE_FILESYSTEM_ROOT=./app-data/documents, PAPRA_CONFIG_DIR=./app-data — and /app/data does not exist in the image at all.

Departure from the task's stated preference order, stated because it was deliberate. Option (1) (reconfigure the app to write to /app/data) was available — all three paths are env-settable. Not taken: it enumerates data paths, so a fourth added upstream would silently escape to the writable layer again — this defect re-armed and invisible. Mounting the app's own data root captures every current and future path by construction.

Gate output — the arbiter, run in both directions:

• fixed → papra CLEAN, with the self-test passing on that run: "prober flags the R-156 signature and clears a correct template — trustworthy"
• reverted to /app/data (red-proof on the real template, not just the canary) → BROKEN: "mount /app/data is NOT writable by the app's own uid=999", "DATA in the writable layer at /app/app-data/db (db_signature=True, e.g. ['db.sqlite'])", "declared volume /app/data is EMPTY"
• catalog_gates.py papra (full, not --fast) → rc=0, all three gates OK

Two operational findings about the gate: it needs root (it reads /var/lib/docker/volumes, mode drwx--x---; as a normal user its own canary fails UNDETERMINED and it correctly refuses a verdict — fail-closed working as designed), and it hardcodes scratch path /srv/felhom-gate, created on DooPlex. Unscoped it deploys all 53 templates; that run was aborted after 10 minutes and its volgate-* scratch projects were cleaned up.

9. §3's correction — confirmed in passing, not chased

restore_points.go:57-59 takes the manifest's mtime and then newestArtifact over the .sql and .tar files, so the newest of the three wins. The restore point does not show a stale timestamp. Confirmed and dropped, as instructed.

10. Register

ID	Change
R-181	CLOSED — SHIPPED (v0.193.0 + v0.193.1), with the live evidence above
R-156	CLOSED — all three apps fixed
R-110	WAITING-ON-OPERATOR → READY, ruling attached: option (b), tag-tracked, and it must cover both channels (the /scripts/ git-sync and the nine files fetched from raw/branch/main) or it only half-works
R-115	WAITING-ON-OPERATOR → READY, ruling attached: mechanism (b), a build-side gate refusing to deploy or vouch an unpublished version; the third instance (agent v0.120.0) would have silently downgraded both demo boxes while reporting success
R-182	NEW. ID established free: grep -ro "R-182\b" over documentation/ and *.md → 2 hits, both prose in REPORT.md recording it as "checked and left unused"; R-183 → 0 hits and remains free

R-165 is collapsed to CLOSED/PROVEN-LIVE in ROADMAP.md; the capability map's local-backup row moves to PROVEN-LIVE, both halves, because the live fill proved the fixed behaviour for both reserve terms.

11. Observations — noticed, documented, NOT acted on

1. R-182 (filed). The periodic status refresh (GetFullStatus → captureAllRecoveryUnits) runs with no admission scope, so a refused app re-alerts on every poll — measured live: a second identical alert pair 13 s after the run's. Pre-existing in v0.192.0; R-181 changed neither caller. Its mitigation is a comment claiming the hub owns cooldown — which is exactly the "invariant asserted in a comment with no test pinning it" shape, so verify at the hub before scoping.
2. A reserve refusal does not make the run fail. The DB and volume legs record SKIP, not FAIL, so lastDBDump.Success stays true and the customer-facing status does not turn red. Deliberate and consistent with v0.192.0 (the capture refusal never set it either), and the operator alert is the signal — but it means "backup succeeded" and "every app was backed up" are not the same statement.
3. UnitSpace.UsedPercent and df disagree — df reported 99% where the alert said 94%, because df's figure accounts for ext4 reserved blocks and the floor's does not. Harmless here (the free-byte term bound), but a percent-term threshold is being compared against a number the operator cannot reproduce with df.
4. The whole-guest PBS backup fails when the volume is near-full, pushing whole_guest_backup_failed (severity error). Expected under a deliberate fill, and its backoff behaved correctly; noted because it is collateral any future fill test will also produce.

12. Teardown

Fill file removed; the planted 2 GiB file removed; a final backup regenerated a correct 178 KB tar; pct fstrim 9201 returned 67.5 GiB and the thin pool settled at 29.43%, below its 36.83% baseline. The backups tree is byte-identical to the pre-test fingerprint. Guest helper scripts and the credential file shred-ed. volgate-* scratch compose projects removed; the unrelated 9-day-old jarr-* containers on DooPlex were left untouched. papra is not left deployed.

No --no-verify was used on any push; the felhom-controller pre-push hook ran and reported gates OK on both pushes.