# SPIKE — where does the space actually go, and what happens when it runs out? (2026-08-02) **Class: spike.** Measurements and source reading only. No production code, no template change, no design implementation. Continues `CAMPAIGN-10-two-storage-soak-2026-07-31.md` §6. **The three headline answers:** 1. **The ceiling is REAL and it is on `mp1` (`/mnt/sys_drive`) — but its shape is a MISMATCH, not a single number.** Live app data lives on a *different* 50 G volume. A box therefore permits **50 GB of app data while capping local backup at 20 GB**. 2. **Overflow behaves WELL.** Backup refuses per-app, preserves the last good unit byte-for-byte, and a later restore of it returns correct data. **Not the R-156 family.** 3. **But the failure is SILENT.** No event on any customer- or operator-visible channel. **R-158**, and it is R-97's defect exactly one tier over. --- ## 1. Venue and baselines | | | |---|---| | Venue | the Campaign 10 rig — **VM 311 on demo-hp**, apps/canaries/runner intact. Not rebuilt, not torn down. | | controller (running) | **0.188.0** (`/etc/felhom-controller-image`) · golden bakes 0.188.0 · `main` 0.188.0 (`4115e88`) | | agent | **0.119.0** (`felhom-agent/CHANGELOG.md` top; `4663df7`) | | hub | **0.86.0** | | `felhom.eu` HEAD | `5f35aa0`, clean | | rig `sys_drive` | **68.8 G** — Campaign 10 grew it 20 G → 70 G with `pct resize`. **The golden ships 20 G.** | | Untouched | demo-felhom, `app-catalog-felhom.eu`, `local-lvm`, `drill-r50`, guest 9201 on demo-hp (read-only) | --- ## 2. Part A — the three unverified premises ### A1 — Docker's data-root is a SEPARATE volume. App DBs are NOT on sys_drive. `build-golden.sh:68`'s "like the Docker-data" reading is **correct**. From the daemon and the rig, not inferred: docker info → data-root=/var/lib/docker every app volume → /var/lib/docker/volumes//_data → findmnt target /var/lib/docker Mount table, each verified with `mountpoint -q` **before** reading sizes (Campaign 10's own sampler trap): | mount | device | size | used | avail | holds | |---|---|---|---|---|---| | `/` | disk-0 | 31.2 G | 944.7 M | 28.7 G | rootfs | | **`/var/lib/docker`** (mp0) | disk-1 | **49 G** | 27 G | 19.5 G | **live app data + DBs** | | **`/mnt/sys_drive`** (mp1) | disk-2 | 68.8 G* | 40.2 G | 25.6 G | **recovery units** | | `/mnt/felhom-drives` | pve-root bind | 58.6 G | 5.8 G | 49.8 G | enrolled-drive binds | \* rig value; the golden ships this at **20 G**. ### A2 — The recovery unit IS on sys_drive /mnt/sys_drive/felhom-data/backups/primary/{rallly,homebox,grafana,papra} findmnt --target … → /mnt/sys_drive /dev/mapper/pve-vm--9201--disk--2 The 40.2 GB measured in Campaign 10 sat here. **E-2 does not save it**: the unit is written to the guest's own `sys_drive`, not to the backup-target drive. ### A3 — `SysDataGrowGB` is a flag defaulting to 0, and real boxes do not pass it felhom-agent/cmd/felhom-agent/main.go:178 flag.IntVar(&sysDataGrow, "sysdata-grow", 0, "for --selftest=bring-up|provision: grow the golden's SSD user-data volume (mp1, /mnt/sys_drive) by this many GiB (0 = keep golden size)") **Not computed from the physical drive.** And it is not merely a lab default — what a *real* provisioned box runs, read from demo-hp itself: demo-hp: ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json ← no flag demo-hp: pct config 9201 → mp0 … size=50G mp1 … size=20G rootfs size=32G agent.json: keys = [authz backup deployment_mode escrow hub lan_resolver local_api log_level oob privileged proxmox storage wg_tunnel] — grep 'grow|sysdata|datavol|size' = 0 hits **Tried and negative:** the flag, the config file, and the live unit on a production host. Nothing supplies a non-zero grow. ### A4 — Restore stages IN PLACE, on the docker volume, not on sys_drive Measured, not read: a homebox restore run **while sys_drive was constrained to 799 MB**: sys_drive avail 799.2M → 799.2M (untouched) docker avail 37.4G → 37.4G (extracted in place) result: ok=true in 22s, 9 blobs / 2305 MB intact, canary unchanged, healthy **The operator's proposed optimisation is already true for restore.** The constrained mount is written only during **backup**. ### The verdict on the ceiling **Real, on `mp1`.** But the useful statement is the mismatch: > A default box allows an app to grow to **50 GB of live data** (mp0) while capping its **local > recovery unit at 20 GB** (mp1). Nothing prevents an app from crossing the line, and crossing it is > invisible until a backup fails. --- ## 3. Part C — the sizing rule, corrected ### C1 — a file-only app is 1.00×, measured | app | volume | unit | ratio | shape | |---|---|---|---|---| | **homebox** | 2305 MB | **2305 MB** | **1.00×** | file-only, sqlite, **no `db-dumps/` directory at all** | | grafana | 50 MB | 48 MB | 0.96× | file-only | | rallly (empty DB) | 962 MB | 963 MB | **1.00×** | DB app — tar 1 008 820 736 B + dump **102 766 B** | | rallly (21 GB DB) | 21 616 MB | 41 149 MB | 1.90× | Campaign 10 §6 | ### The rule is not a constant — it is a sum > **unit ≈ volume-tar bytes + logical DB-dump bytes** `1.90×` was the DB worst case, where the volume is almost entirely live rows. The same app with an empty DB measured **1.00×**, which is the proof the ratio tracks dump size and not "DB-ness". | app shape | ratio | what a **20 G** default sys_drive holds | |---|---|---| | file-only | **1.00×** | **≈ 19 GB of app data** | | DB app, volume mostly live rows | **→ 2.0×** | **≈ 10 GB of app data** | Two numbers, not one — and the difference is material for guidance. ### C2 — both representations are used, and the reason is explicit Not redundancy. `internal/backup/restore_unit.go:262-266`, verbatim: // F17: the captured .sql dump is the authoritative logical DB state — replay it AFTER the volume // restore, so the dump WINS over any volume-tar copy of the database. // R-47: the replay happens with ONLY the database service up. This used to run after // RecreateStackFromUnit had already brought the WHOLE stack up, letting the application rebuild // schema objects underneath the replay (H4, DIAG-immich-restore-round2-2026-07-19). Order is: stop → restore volume tars → recreate definition → **DB-only start → replay dump** → start stack. **Could the DB volume's tar be dropped?** The dump is `pg_dump -U -d --clean --if-exists --no-owner --no-privileges` (`internal/appbackup/dbdump.go:225`) — a **single database, no globals/roles**. Because of `--no-owner`, a fresh `initdb` (which creates the role from `POSTGRES_USER`/`POSTGRES_PASSWORD`) plus this dump is **logically sufficient for a single-DB app**. So the tar is a **physical fallback**, not a correctness requirement. **And dropping it would fix the D5/R-127(b) trap rather than risk it.** The trap exists *because* the tar is restored: PGDATA present ⇒ postgres skips `initdb` ⇒ `POSTGRES_PASSWORD` is ignored ⇒ a regenerated password authenticates over the local trust socket while failing over TCP. A fresh `initdb` applies the password correctly. **The reason to keep it anyway — and this is the answer worth more than the optimisation:** the tar is the only copy if the dump is bad, and **`ValidateDump` currently WARNS rather than gates**. Observed live, twice, in this very session: [WARN] [backup] ValidateDump: …/rallly-postgres.sql is structurally valid (32 tables) but its accounts table has NO rows — the dump may predate the customer's data Dropping the DB volume tar would halve DB-app units *and* close R-127(b) — **but only after `ValidateDump` is promoted from a warning to a gate.** In its current form the tar is load-bearing. --- ## 4. Part B — what happens when it does not fit Method: ballast `/mnt/sys_drive` to **799 MB free** (homebox's unit needs 2.3 GB), then run the real backup. Pre-state recorded so truncation would be detectable. ### On backup: **outcome 1 — refuses cleanly, per app** [WARN] Volume dump failed for homebox/homebox_homebox_data (last good dump preserved): tar: write error: No space left on device — exit status 1 [ERROR] Volume dump failed for homebox: volume dump failed for: homebox_homebox_data … [INFO] Volume dump: papra/papra_papra_data → 2.5 KB ← other apps CONTINUE /api/backup/status → db_dump.success: FALSE **The "last good dump preserved" claim is TRUE — verified, not taken on trust:** | check | pre | post | |---|---|---| | size | 2 416 226 304 B | **2 416 226 304 B** | | md5 | `4151146ed4af56cd19976cb5e97f137e` | **`4151146ed4af56cd19976cb5e97f137e`** | | `tar -tf` end-to-end | — | **VALID** | | `.tmp`/`.part` residue | — | **none** | Write-to-temp-then-swap held. **No partial artifact can later read as a completed backup — explicitly NOT the R-156 family**, and the mount was not filled to the point of breaking the guest: apps stayed up and healthy throughout. ### On restore: correct data, honest success Restoring the preserved unit **while still constrained** returned the last good state intact — 9 blobs, 2305 MB, canary `C10-M02-B-081947`, healthy, `ok: true` in 22 s. It did **not** return partial data, and it did **not** claim success over a partial restore. ### The gap: it is SILENT **Zero events** reached the hub during the overflow window (checked in the hub's own log, which has 0 restarts over 43 h). The only trace is the controller's `[ERROR]` line and a `success:false` field in an API nobody polls. --- ## 5. Findings ### R-158 (new) — the app-data (Tier-1 recovery unit) backup has NO failure signal to the hub **Register grepped:** no item covers the app-data path. **R-97** is the same defect on the **whole-guest** tier and is SHIPPED (`ROADMAP.md:157`). This is R-97 exactly one tier over. - `NotifyBackupFailed` **exists** (`internal/notify/notifier.go:297`), and the hub **allowlists** `backup_failed`. - Its **only production caller** is the off-box/NAS leg: `cmd/controller/main.go:659` (*"Off-box (NAS) mentés sikertelen"*). - The backup manager's notify seams are `tier2Notify`, `offboxNotify`, `offboxEnlargeBlockedNotify` (`internal/backup/backup.go:33,37,58`) — **none for the local recovery-unit capture**. - Measured consequence: 2 of 5 volume dumps failed with `ENOSPC`; nothing was emitted. The codebase already names this pattern, in a test written for R-97 (`cmd/controller/quiesce_notify_wiring_test.go:14`): > *"…which is exactly the state R-97 found `internal/quiesce` in (NotifyBackupFailed existed, the hub > allowlisted backup_failed, and no code connected them)."* This is the **fifth** instance of *seam built but never wired*. **Proposed shape** (not implemented): a `unitNotify` seam on the recovery-unit capture, wired in `main()` like `SetOffboxNotify`, emitting the existing `backup_failed`; severity error; per-app detail. A test should pin the **consequence** (a failed unit capture produces an event) rather than the mechanism. ### Not filed - **The 20 G / 50 G mismatch** — a tier-sizing decision (§6), not a defect. `SysDataGrowGB` is the intended lever and it works; nothing sets it. - **`ValidateDump` warns rather than gates** — pre-existing, and it is the reason the DB volume tar must stay for now (§C2). Named here as a dependency of any future unit-halving work, not re-filed. --- ## 6. The sizing rule as it stands — for the architecture doc, NOT written into it here > **Recovery unit ≈ volume-tar bytes + logical DB-dump bytes.** > - file-only app → **1.00×** → a 20 G `sys_drive` holds **≈ 19 GB** of app data > - DB app whose volume is mostly live rows → **→ 2.0×** → a 20 G `sys_drive` holds **≈ 10 GB** > > `mp0` (live data) ships at **50 G** and `mp1` (recovery units) at **20 G**. `SysDataGrowGB` exists to > raise mp1 per customer and **is 0 on every box observed**, including a production one. **What this bounds for D5.** D5's promise is that restoring an app needs *the drive and nothing else* — Lane 1's independence from the guest. **That independence is bounded by app size**, and the bound is: > **≈ 19 GB for a file-only app and ≈ 10 GB for a DB-backed app, on a default box.** Past that the unit cannot be captured at all, and the app silently falls back to the operator-driven whole-guest route — silently because of **R-158**. --- ## 7. What did not run, and why - **The ~5 GB linearity point** — the brief said skip it; skipped. - **A third overflow mode (filling the mount to breakage)** was not forced. The per-app refusal preserved space and other apps continued, so the mount never actually filled; provoking a true full-mount state would have required disabling the guard that turned out to work. - **MariaDB shape** not measured — `mariadb-dump --single-transaction --routines --triggers` (`dbdump.go:243`) is also single-database, so the same reasoning should apply, but that is **inference, not measurement**. - **Teardown remains OWED and was NOT performed** — VM 311, `c10-scratch`, PBS datastore `felhom-c10` + user/token, restic subaccount `u629488-sub4` (id 281530), hub customer **`c10-soak` (disposition: DELETE)**, and the credential dirs. Commands in `CAMPAIGN-10-…md` §9. - **Rig left as found**, plus: the probe ballast was removed; `cc_bulk` (the M-band table) was dropped after its numbers were captured; homebox carries ~2.3 GB of probe blobs in `/data/c10bulk`. Apps, canaries and runner intact.