felhom.eu/documentation/audits/SPIKE-recovery-unit-space-2026-08-02.md

SPIKE — where does the space actually go, and what happens when it runs out? (2026-08-02)

Class: spike. Measurements and source reading only. No production code, no template change, no design implementation. Continues CAMPAIGN-10-two-storage-soak-2026-07-31.md §6.

The three headline answers:

1. The ceiling is REAL and it is on mp1 (/mnt/sys_drive) — but its shape is a MISMATCH, not a single number. Live app data lives on a different 50 G volume. A box therefore permits 50 GB of app data while capping local backup at 20 GB.
2. Overflow behaves WELL. Backup refuses per-app, preserves the last good unit byte-for-byte, and a later restore of it returns correct data. Not the R-156 family.
3. But the failure is SILENT. No event on any customer- or operator-visible channel. R-158, and it is R-97's defect exactly one tier over.

---

1. Venue and baselines

Venue	the Campaign 10 rig — VM 311 on demo-hp, apps/canaries/runner intact. Not rebuilt, not torn down.
controller (running)	0.188.0 (/etc/felhom-controller-image) · golden bakes 0.188.0 · main 0.188.0 (4115e88)
agent	0.119.0 (felhom-agent/CHANGELOG.md top; 4663df7)
hub	0.86.0
felhom.eu HEAD	5f35aa0, clean
rig sys_drive	68.8 G — Campaign 10 grew it 20 G → 70 G with pct resize. The golden ships 20 G.
Untouched	demo-felhom, app-catalog-felhom.eu, local-lvm, drill-r50, guest 9201 on demo-hp (read-only)

---

2. Part A — the three unverified premises

A1 — Docker's data-root is a SEPARATE volume. App DBs are NOT on sys_drive.

build-golden.sh:68's "like the Docker-data" reading is correct. From the daemon and the rig, not inferred:

docker info → data-root=/var/lib/docker
every app volume → /var/lib/docker/volumes/<v>/_data → findmnt target /var/lib/docker

Mount table, each verified with mountpoint -q before reading sizes (Campaign 10's own sampler trap):

mount	device	size	used	avail	holds
/	disk-0	31.2 G	944.7 M	28.7 G	rootfs
/var/lib/docker (mp0)	disk-1	49 G	27 G	19.5 G	live app data + DBs
/mnt/sys_drive (mp1)	disk-2	68.8 G*	40.2 G	25.6 G	recovery units
/mnt/felhom-drives	pve-root bind	58.6 G	5.8 G	49.8 G	enrolled-drive binds

* rig value; the golden ships this at 20 G.

A2 — The recovery unit IS on sys_drive

/mnt/sys_drive/felhom-data/backups/primary/{rallly,homebox,grafana,papra}
findmnt --target … → /mnt/sys_drive  /dev/mapper/pve-vm--9201--disk--2

The 40.2 GB measured in Campaign 10 sat here. E-2 does not save it: the unit is written to the guest's own sys_drive, not to the backup-target drive.

A3 — SysDataGrowGB is a flag defaulting to 0, and real boxes do not pass it

felhom-agent/cmd/felhom-agent/main.go:178
  flag.IntVar(&sysDataGrow, "sysdata-grow", 0,
    "for --selftest=bring-up|provision: grow the golden's SSD user-data volume (mp1, /mnt/sys_drive)
     by this many GiB (0 = keep golden size)")

Not computed from the physical drive. And it is not merely a lab default — what a real provisioned box runs, read from demo-hp itself:

demo-hp:  ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json   ← no flag
demo-hp:  pct config 9201 → mp0 … size=50G   mp1 … size=20G   rootfs size=32G
agent.json: keys = [authz backup deployment_mode escrow hub lan_resolver local_api log_level
                    oob privileged proxmox storage wg_tunnel]   — grep 'grow|sysdata|datavol|size' = 0 hits

Tried and negative: the flag, the config file, and the live unit on a production host. Nothing supplies a non-zero grow.

A4 — Restore stages IN PLACE, on the docker volume, not on sys_drive

Measured, not read: a homebox restore run while sys_drive was constrained to 799 MB:

sys_drive avail  799.2M  →  799.2M     (untouched)
docker    avail   37.4G  →   37.4G     (extracted in place)
result: ok=true in 22s, 9 blobs / 2305 MB intact, canary unchanged, healthy

The operator's proposed optimisation is already true for restore. The constrained mount is written only during backup.

The verdict on the ceiling

Real, on mp1. But the useful statement is the mismatch:

A default box allows an app to grow to 50 GB of live data (mp0) while capping its local recovery unit at 20 GB (mp1). Nothing prevents an app from crossing the line, and crossing it is invisible until a backup fails.

---

3. Part C — the sizing rule, corrected

C1 — a file-only app is 1.00×, measured

app	volume	unit	ratio	shape
homebox	2305 MB	2305 MB	1.00×	file-only, sqlite, no db-dumps/ directory at all
grafana	50 MB	48 MB	0.96×	file-only
rallly (empty DB)	962 MB	963 MB	1.00×	DB app — tar 1 008 820 736 B + dump 102 766 B
rallly (21 GB DB)	21 616 MB	41 149 MB	1.90×	Campaign 10 §6

The rule is not a constant — it is a sum

unit ≈ volume-tar bytes + logical DB-dump bytes

1.90× was the DB worst case, where the volume is almost entirely live rows. The same app with an empty DB measured 1.00×, which is the proof the ratio tracks dump size and not "DB-ness".

app shape	ratio	what a 20 G default sys_drive holds
file-only	1.00×	≈ 19 GB of app data
DB app, volume mostly live rows	→ 2.0×	≈ 10 GB of app data

Two numbers, not one — and the difference is material for guidance.

C2 — both representations are used, and the reason is explicit

Not redundancy. internal/backup/restore_unit.go:262-266, verbatim:

// F17: the captured .sql dump is the authoritative logical DB state — replay it AFTER the volume
// restore, so the dump WINS over any volume-tar copy of the database.
// R-47: the replay happens with ONLY the database service up. This used to run after
// RecreateStackFromUnit had already brought the WHOLE stack up, letting the application rebuild
// schema objects underneath the replay (H4, DIAG-immich-restore-round2-2026-07-19).

Order is: stop → restore volume tars → recreate definition → DB-only start → replay dump → start stack.

Could the DB volume's tar be dropped? The dump is pg_dump -U <user> -d <db> --clean --if-exists --no-owner --no-privileges (internal/appbackup/dbdump.go:225) — a single database, no globals/roles. Because of --no-owner, a fresh initdb (which creates the role from POSTGRES_USER/POSTGRES_PASSWORD) plus this dump is logically sufficient for a single-DB app. So the tar is a physical fallback, not a correctness requirement.

And dropping it would fix the D5/R-127(b) trap rather than risk it. The trap exists because the tar is restored: PGDATA present ⇒ postgres skips initdb ⇒ POSTGRES_PASSWORD is ignored ⇒ a regenerated password authenticates over the local trust socket while failing over TCP. A fresh initdb applies the password correctly.

The reason to keep it anyway — and this is the answer worth more than the optimisation: the tar is the only copy if the dump is bad, and ValidateDump currently WARNS rather than gates. Observed live, twice, in this very session:

[WARN] [backup] ValidateDump: …/rallly-postgres.sql is structurally valid (32 tables) but its
       accounts table has NO rows — the dump may predate the customer's data

Dropping the DB volume tar would halve DB-app units and close R-127(b) — but only after ValidateDump is promoted from a warning to a gate. In its current form the tar is load-bearing.

---

4. Part B — what happens when it does not fit

Method: ballast /mnt/sys_drive to 799 MB free (homebox's unit needs 2.3 GB), then run the real backup. Pre-state recorded so truncation would be detectable.

On backup: outcome 1 — refuses cleanly, per app

[WARN]  Volume dump failed for homebox/homebox_homebox_data (last good dump preserved):
        tar: write error: No space left on device — exit status 1
[ERROR] Volume dump failed for homebox: volume dump failed for: homebox_homebox_data
…
[INFO]  Volume dump: papra/papra_papra_data → 2.5 KB      ← other apps CONTINUE
/api/backup/status → db_dump.success: FALSE

The "last good dump preserved" claim is TRUE — verified, not taken on trust:

check	pre	post
size	2 416 226 304 B	2 416 226 304 B
md5	4151146ed4af56cd19976cb5e97f137e	4151146ed4af56cd19976cb5e97f137e
tar -tf end-to-end	—	VALID
.tmp/.part residue	—	none

Write-to-temp-then-swap held. No partial artifact can later read as a completed backup — explicitly NOT the R-156 family, and the mount was not filled to the point of breaking the guest: apps stayed up and healthy throughout.

On restore: correct data, honest success

Restoring the preserved unit while still constrained returned the last good state intact — 9 blobs, 2305 MB, canary C10-M02-B-081947, healthy, ok: true in 22 s. It did not return partial data, and it did not claim success over a partial restore.

The gap: it is SILENT

Zero events reached the hub during the overflow window (checked in the hub's own log, which has 0 restarts over 43 h). The only trace is the controller's [ERROR] line and a success:false field in an API nobody polls.

---

5. Findings

R-158 (new) — the app-data (Tier-1 recovery unit) backup has NO failure signal to the hub

Register grepped: no item covers the app-data path. R-97 is the same defect on the whole-guest tier and is SHIPPED (ROADMAP.md:157). This is R-97 exactly one tier over.

• NotifyBackupFailed exists (internal/notify/notifier.go:297), and the hub allowlists backup_failed.
• Its only production caller is the off-box/NAS leg: cmd/controller/main.go:659 ("Off-box (NAS) mentés sikertelen").
• The backup manager's notify seams are tier2Notify, offboxNotify, offboxEnlargeBlockedNotify (internal/backup/backup.go:33,37,58) — none for the local recovery-unit capture.
• Measured consequence: 2 of 5 volume dumps failed with ENOSPC; nothing was emitted.

The codebase already names this pattern, in a test written for R-97 (cmd/controller/quiesce_notify_wiring_test.go:14):

"…which is exactly the state R-97 found internal/quiesce in (NotifyBackupFailed existed, the hub allowlisted backup_failed, and no code connected them)."

This is the fifth instance of seam built but never wired. Proposed shape (not implemented): a unitNotify seam on the recovery-unit capture, wired in main() like SetOffboxNotify, emitting the existing backup_failed; severity error; per-app detail. A test should pin the consequence (a failed unit capture produces an event) rather than the mechanism.

Not filed

• The 20 G / 50 G mismatch — a tier-sizing decision (§6), not a defect. SysDataGrowGB is the intended lever and it works; nothing sets it.
• ValidateDump warns rather than gates — pre-existing, and it is the reason the DB volume tar must stay for now (§C2). Named here as a dependency of any future unit-halving work, not re-filed.

---

6. The sizing rule as it stands — for the architecture doc, NOT written into it here

Recovery unit ≈ volume-tar bytes + logical DB-dump bytes.

• file-only app → 1.00× → a 20 G sys_drive holds ≈ 19 GB of app data
• DB app whose volume is mostly live rows → → 2.0× → a 20 G sys_drive holds ≈ 10 GB

mp0 (live data) ships at 50 G and mp1 (recovery units) at 20 G. SysDataGrowGB exists to raise mp1 per customer and is 0 on every box observed, including a production one.

What this bounds for D5. D5's promise is that restoring an app needs the drive and nothing else — Lane 1's independence from the guest. That independence is bounded by app size, and the bound is:

≈ 19 GB for a file-only app and ≈ 10 GB for a DB-backed app, on a default box.

Past that the unit cannot be captured at all, and the app silently falls back to the operator-driven whole-guest route — silently because of R-158.

---

7. What did not run, and why

• The ~5 GB linearity point — the brief said skip it; skipped.
• A third overflow mode (filling the mount to breakage) was not forced. The per-app refusal preserved space and other apps continued, so the mount never actually filled; provoking a true full-mount state would have required disabling the guard that turned out to work.
• MariaDB shape not measured — mariadb-dump --single-transaction --routines --triggers (dbdump.go:243) is also single-database, so the same reasoning should apply, but that is inference, not measurement.
• Teardown remains OWED and was NOT performed — VM 311, c10-scratch, PBS datastore felhom-c10
  • user/token, restic subaccount u629488-sub4 (id 281530), hub customer c10-soak (disposition: DELETE), and the credential dirs. Commands in CAMPAIGN-10-…md §9.
• Rig left as found, plus: the probe ballast was removed; cc_bulk (the M-band table) was dropped after its numbers were captured; homebox carries ~2.3 GB of probe blobs in /data/c10bulk. Apps, canaries and runner intact.