# SPIKE — where does the space actually go, and what happens when it runs out? (2026-08-02)

**Class: spike.** Measurements and source reading only. No production code, no template change, no
design implementation. Continues `CAMPAIGN-10-two-storage-soak-2026-07-31.md` §6.

**The three headline answers:**

1. **The ceiling is REAL and it is on `mp1` (`/mnt/sys_drive`) — but its shape is a MISMATCH, not a
   single number.** Live app data lives on a *different* 50 G volume. A box therefore permits **50 GB
   of app data while capping local backup at 20 GB**.
2. **Overflow behaves WELL.** Backup refuses per-app, preserves the last good unit byte-for-byte, and a
   later restore of it returns correct data. **Not the R-156 family.**
3. **But the failure is SILENT.** No event on any customer- or operator-visible channel. **R-158**, and
   it is R-97's defect exactly one tier over.

---

## 1. Venue and baselines

| | |
|---|---|
| Venue | the Campaign 10 rig — **VM 311 on demo-hp**, apps/canaries/runner intact. Not rebuilt, not torn down. |
| controller (running) | **0.188.0** (`/etc/felhom-controller-image`) · golden bakes 0.188.0 · `main` 0.188.0 (`4115e88`) |
| agent | **0.119.0** (`felhom-agent/CHANGELOG.md` top; `4663df7`) |
| hub | **0.86.0** |
| `felhom.eu` HEAD | `5f35aa0`, clean |
| rig `sys_drive` | **68.8 G** — Campaign 10 grew it 20 G → 70 G with `pct resize`. **The golden ships 20 G.** |
| Untouched | demo-felhom, `app-catalog-felhom.eu`, `local-lvm`, `drill-r50`, guest 9201 on demo-hp (read-only) |

---

## 2. Part A — the three unverified premises

### A1 — Docker's data-root is a SEPARATE volume. App DBs are NOT on sys_drive.

`build-golden.sh:68`'s "like the Docker-data" reading is **correct**. From the daemon and the rig, not
inferred:

    docker info → data-root=/var/lib/docker
    every app volume → /var/lib/docker/volumes/<v>/_data → findmnt target /var/lib/docker

Mount table, each verified with `mountpoint -q` **before** reading sizes (Campaign 10's own sampler trap):

| mount | device | size | used | avail | holds |
|---|---|---|---|---|---|
| `/` | disk-0 | 31.2 G | 944.7 M | 28.7 G | rootfs |
| **`/var/lib/docker`** (mp0) | disk-1 | **49 G** | 27 G | 19.5 G | **live app data + DBs** |
| **`/mnt/sys_drive`** (mp1) | disk-2 | 68.8 G* | 40.2 G | 25.6 G | **recovery units** |
| `/mnt/felhom-drives` | pve-root bind | 58.6 G | 5.8 G | 49.8 G | enrolled-drive binds |

\* rig value; the golden ships this at **20 G**.

### A2 — The recovery unit IS on sys_drive

    /mnt/sys_drive/felhom-data/backups/primary/{rallly,homebox,grafana,papra}
    findmnt --target … → /mnt/sys_drive  /dev/mapper/pve-vm--9201--disk--2

The 40.2 GB measured in Campaign 10 sat here. **E-2 does not save it**: the unit is written to the
guest's own `sys_drive`, not to the backup-target drive.

### A3 — `SysDataGrowGB` is a flag defaulting to 0, and real boxes do not pass it

    felhom-agent/cmd/felhom-agent/main.go:178
      flag.IntVar(&sysDataGrow, "sysdata-grow", 0,
        "for --selftest=bring-up|provision: grow the golden's SSD user-data volume (mp1, /mnt/sys_drive)
         by this many GiB (0 = keep golden size)")

**Not computed from the physical drive.** And it is not merely a lab default — what a *real*
provisioned box runs, read from demo-hp itself:

    demo-hp:  ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json   ← no flag
    demo-hp:  pct config 9201 → mp0 … size=50G   mp1 … size=20G   rootfs size=32G
    agent.json: keys = [authz backup deployment_mode escrow hub lan_resolver local_api log_level
                        oob privileged proxmox storage wg_tunnel]   — grep 'grow|sysdata|datavol|size' = 0 hits

**Tried and negative:** the flag, the config file, and the live unit on a production host. Nothing
supplies a non-zero grow.

### A4 — Restore stages IN PLACE, on the docker volume, not on sys_drive

Measured, not read: a homebox restore run **while sys_drive was constrained to 799 MB**:

    sys_drive avail  799.2M  →  799.2M     (untouched)
    docker    avail   37.4G  →   37.4G     (extracted in place)
    result: ok=true in 22s, 9 blobs / 2305 MB intact, canary unchanged, healthy

**The operator's proposed optimisation is already true for restore.** The constrained mount is written
only during **backup**.

### The verdict on the ceiling

**Real, on `mp1`.** But the useful statement is the mismatch:

> A default box allows an app to grow to **50 GB of live data** (mp0) while capping its **local
> recovery unit at 20 GB** (mp1). Nothing prevents an app from crossing the line, and crossing it is
> invisible until a backup fails.

---

## 3. Part C — the sizing rule, corrected

### C1 — a file-only app is 1.00×, measured

| app | volume | unit | ratio | shape |
|---|---|---|---|---|
| **homebox** | 2305 MB | **2305 MB** | **1.00×** | file-only, sqlite, **no `db-dumps/` directory at all** |
| grafana | 50 MB | 48 MB | 0.96× | file-only |
| rallly (empty DB) | 962 MB | 963 MB | **1.00×** | DB app — tar 1 008 820 736 B + dump **102 766 B** |
| rallly (21 GB DB) | 21 616 MB | 41 149 MB | 1.90× | Campaign 10 §6 |

### The rule is not a constant — it is a sum

> **unit ≈ volume-tar bytes + logical DB-dump bytes**

`1.90×` was the DB worst case, where the volume is almost entirely live rows. The same app with an
empty DB measured **1.00×**, which is the proof the ratio tracks dump size and not "DB-ness".

| app shape | ratio | what a **20 G** default sys_drive holds |
|---|---|---|
| file-only | **1.00×** | **≈ 19 GB of app data** |
| DB app, volume mostly live rows | **→ 2.0×** | **≈ 10 GB of app data** |

Two numbers, not one — and the difference is material for guidance.

### C2 — both representations are used, and the reason is explicit

Not redundancy. `internal/backup/restore_unit.go:262-266`, verbatim:

    // F17: the captured .sql dump is the authoritative logical DB state — replay it AFTER the volume
    // restore, so the dump WINS over any volume-tar copy of the database.
    // R-47: the replay happens with ONLY the database service up. This used to run after
    // RecreateStackFromUnit had already brought the WHOLE stack up, letting the application rebuild
    // schema objects underneath the replay (H4, DIAG-immich-restore-round2-2026-07-19).

Order is: stop → restore volume tars → recreate definition → **DB-only start → replay dump** → start stack.

**Could the DB volume's tar be dropped?** The dump is
`pg_dump -U <user> -d <db> --clean --if-exists --no-owner --no-privileges` (`internal/appbackup/dbdump.go:225`)
— a **single database, no globals/roles**. Because of `--no-owner`, a fresh `initdb` (which creates the
role from `POSTGRES_USER`/`POSTGRES_PASSWORD`) plus this dump is **logically sufficient for a
single-DB app**. So the tar is a **physical fallback**, not a correctness requirement.

**And dropping it would fix the D5/R-127(b) trap rather than risk it.** The trap exists *because* the
tar is restored: PGDATA present ⇒ postgres skips `initdb` ⇒ `POSTGRES_PASSWORD` is ignored ⇒ a
regenerated password authenticates over the local trust socket while failing over TCP. A fresh
`initdb` applies the password correctly.

**The reason to keep it anyway — and this is the answer worth more than the optimisation:** the tar is
the only copy if the dump is bad, and **`ValidateDump` currently WARNS rather than gates**. Observed
live, twice, in this very session:

    [WARN] [backup] ValidateDump: …/rallly-postgres.sql is structurally valid (32 tables) but its
           accounts table has NO rows — the dump may predate the customer's data

Dropping the DB volume tar would halve DB-app units *and* close R-127(b) — **but only after
`ValidateDump` is promoted from a warning to a gate.** In its current form the tar is load-bearing.

---

## 4. Part B — what happens when it does not fit

Method: ballast `/mnt/sys_drive` to **799 MB free** (homebox's unit needs 2.3 GB), then run the real
backup. Pre-state recorded so truncation would be detectable.

### On backup: **outcome 1 — refuses cleanly, per app**

    [WARN]  Volume dump failed for homebox/homebox_homebox_data (last good dump preserved):
            tar: write error: No space left on device — exit status 1
    [ERROR] Volume dump failed for homebox: volume dump failed for: homebox_homebox_data
    …
    [INFO]  Volume dump: papra/papra_papra_data → 2.5 KB      ← other apps CONTINUE
    /api/backup/status → db_dump.success: FALSE

**The "last good dump preserved" claim is TRUE — verified, not taken on trust:**

| check | pre | post |
|---|---|---|
| size | 2 416 226 304 B | **2 416 226 304 B** |
| md5 | `4151146ed4af56cd19976cb5e97f137e` | **`4151146ed4af56cd19976cb5e97f137e`** |
| `tar -tf` end-to-end | — | **VALID** |
| `.tmp`/`.part` residue | — | **none** |

Write-to-temp-then-swap held. **No partial artifact can later read as a completed backup — explicitly
NOT the R-156 family**, and the mount was not filled to the point of breaking the guest: apps stayed
up and healthy throughout.

### On restore: correct data, honest success

Restoring the preserved unit **while still constrained** returned the last good state intact — 9 blobs,
2305 MB, canary `C10-M02-B-081947`, healthy, `ok: true` in 22 s. It did **not** return partial data,
and it did **not** claim success over a partial restore.

### The gap: it is SILENT

**Zero events** reached the hub during the overflow window (checked in the hub's own log, which has
0 restarts over 43 h). The only trace is the controller's `[ERROR]` line and a `success:false` field
in an API nobody polls.

---

## 5. Findings

### R-158 (new) — the app-data (Tier-1 recovery unit) backup has NO failure signal to the hub

**Register grepped:** no item covers the app-data path. **R-97** is the same defect on the
**whole-guest** tier and is SHIPPED (`ROADMAP.md:157`). This is R-97 exactly one tier over.

- `NotifyBackupFailed` **exists** (`internal/notify/notifier.go:297`), and the hub **allowlists**
  `backup_failed`.
- Its **only production caller** is the off-box/NAS leg: `cmd/controller/main.go:659`
  (*"Off-box (NAS) mentés sikertelen"*).
- The backup manager's notify seams are `tier2Notify`, `offboxNotify`,
  `offboxEnlargeBlockedNotify` (`internal/backup/backup.go:33,37,58`) — **none for the local
  recovery-unit capture**.
- Measured consequence: 2 of 5 volume dumps failed with `ENOSPC`; nothing was emitted.

The codebase already names this pattern, in a test written for R-97
(`cmd/controller/quiesce_notify_wiring_test.go:14`):

> *"…which is exactly the state R-97 found `internal/quiesce` in (NotifyBackupFailed existed, the hub
> allowlisted backup_failed, and no code connected them)."*

This is the **fifth** instance of *seam built but never wired*. **Proposed shape** (not implemented):
a `unitNotify` seam on the recovery-unit capture, wired in `main()` like `SetOffboxNotify`, emitting
the existing `backup_failed`; severity error; per-app detail. A test should pin the **consequence**
(a failed unit capture produces an event) rather than the mechanism.

### Not filed

- **The 20 G / 50 G mismatch** — a tier-sizing decision (§6), not a defect. `SysDataGrowGB` is the
  intended lever and it works; nothing sets it.
- **`ValidateDump` warns rather than gates** — pre-existing, and it is the reason the DB volume tar
  must stay for now (§C2). Named here as a dependency of any future unit-halving work, not re-filed.

---

## 6. The sizing rule as it stands — for the architecture doc, NOT written into it here

> **Recovery unit ≈ volume-tar bytes + logical DB-dump bytes.**
> - file-only app → **1.00×** → a 20 G `sys_drive` holds **≈ 19 GB** of app data
> - DB app whose volume is mostly live rows → **→ 2.0×** → a 20 G `sys_drive` holds **≈ 10 GB**
>
> `mp0` (live data) ships at **50 G** and `mp1` (recovery units) at **20 G**. `SysDataGrowGB` exists to
> raise mp1 per customer and **is 0 on every box observed**, including a production one.

**What this bounds for D5.** D5's promise is that restoring an app needs *the drive and nothing else* —
Lane 1's independence from the guest. **That independence is bounded by app size**, and the bound is:

> **≈ 19 GB for a file-only app and ≈ 10 GB for a DB-backed app, on a default box.**

Past that the unit cannot be captured at all, and the app silently falls back to the operator-driven
whole-guest route — silently because of **R-158**.

---

## 7. What did not run, and why

- **The ~5 GB linearity point** — the brief said skip it; skipped.
- **A third overflow mode (filling the mount to breakage)** was not forced. The per-app refusal
  preserved space and other apps continued, so the mount never actually filled; provoking a true
  full-mount state would have required disabling the guard that turned out to work.
- **MariaDB shape** not measured — `mariadb-dump --single-transaction --routines --triggers`
  (`dbdump.go:243`) is also single-database, so the same reasoning should apply, but that is
  **inference, not measurement**.
- **Teardown remains OWED and was NOT performed** — VM 311, `c10-scratch`, PBS datastore `felhom-c10`
  + user/token, restic subaccount `u629488-sub4` (id 281530), hub customer **`c10-soak` (disposition:
  DELETE)**, and the credential dirs. Commands in `CAMPAIGN-10-…md` §9.
- **Rig left as found**, plus: the probe ballast was removed; `cc_bulk` (the M-band table) was dropped
  after its numbers were captured; homebox carries ~2.3 GB of probe blobs in `/data/c10bulk`. Apps,
  canaries and runner intact.
