diff --git a/documentation/audits/SPIKE-recovery-unit-space-2026-08-02.md b/documentation/audits/SPIKE-recovery-unit-space-2026-08-02.md new file mode 100644 index 0000000..ace935c --- /dev/null +++ b/documentation/audits/SPIKE-recovery-unit-space-2026-08-02.md @@ -0,0 +1,270 @@ +# SPIKE — where does the space actually go, and what happens when it runs out? (2026-08-02) + +**Class: spike.** Measurements and source reading only. No production code, no template change, no +design implementation. Continues `CAMPAIGN-10-two-storage-soak-2026-07-31.md` §6. + +**The three headline answers:** + +1. **The ceiling is REAL and it is on `mp1` (`/mnt/sys_drive`) — but its shape is a MISMATCH, not a + single number.** Live app data lives on a *different* 50 G volume. A box therefore permits **50 GB + of app data while capping local backup at 20 GB**. +2. **Overflow behaves WELL.** Backup refuses per-app, preserves the last good unit byte-for-byte, and a + later restore of it returns correct data. **Not the R-156 family.** +3. **But the failure is SILENT.** No event on any customer- or operator-visible channel. **R-158**, and + it is R-97's defect exactly one tier over. + +--- + +## 1. Venue and baselines + +| | | +|---|---| +| Venue | the Campaign 10 rig — **VM 311 on demo-hp**, apps/canaries/runner intact. Not rebuilt, not torn down. | +| controller (running) | **0.188.0** (`/etc/felhom-controller-image`) · golden bakes 0.188.0 · `main` 0.188.0 (`4115e88`) | +| agent | **0.119.0** (`felhom-agent/CHANGELOG.md` top; `4663df7`) | +| hub | **0.86.0** | +| `felhom.eu` HEAD | `5f35aa0`, clean | +| rig `sys_drive` | **68.8 G** — Campaign 10 grew it 20 G → 70 G with `pct resize`. **The golden ships 20 G.** | +| Untouched | demo-felhom, `app-catalog-felhom.eu`, `local-lvm`, `drill-r50`, guest 9201 on demo-hp (read-only) | + +--- + +## 2. Part A — the three unverified premises + +### A1 — Docker's data-root is a SEPARATE volume. App DBs are NOT on sys_drive. + +`build-golden.sh:68`'s "like the Docker-data" reading is **correct**. From the daemon and the rig, not +inferred: + + docker info → data-root=/var/lib/docker + every app volume → /var/lib/docker/volumes//_data → findmnt target /var/lib/docker + +Mount table, each verified with `mountpoint -q` **before** reading sizes (Campaign 10's own sampler trap): + +| mount | device | size | used | avail | holds | +|---|---|---|---|---|---| +| `/` | disk-0 | 31.2 G | 944.7 M | 28.7 G | rootfs | +| **`/var/lib/docker`** (mp0) | disk-1 | **49 G** | 27 G | 19.5 G | **live app data + DBs** | +| **`/mnt/sys_drive`** (mp1) | disk-2 | 68.8 G* | 40.2 G | 25.6 G | **recovery units** | +| `/mnt/felhom-drives` | pve-root bind | 58.6 G | 5.8 G | 49.8 G | enrolled-drive binds | + +\* rig value; the golden ships this at **20 G**. + +### A2 — The recovery unit IS on sys_drive + + /mnt/sys_drive/felhom-data/backups/primary/{rallly,homebox,grafana,papra} + findmnt --target … → /mnt/sys_drive /dev/mapper/pve-vm--9201--disk--2 + +The 40.2 GB measured in Campaign 10 sat here. **E-2 does not save it**: the unit is written to the +guest's own `sys_drive`, not to the backup-target drive. + +### A3 — `SysDataGrowGB` is a flag defaulting to 0, and real boxes do not pass it + + felhom-agent/cmd/felhom-agent/main.go:178 + flag.IntVar(&sysDataGrow, "sysdata-grow", 0, + "for --selftest=bring-up|provision: grow the golden's SSD user-data volume (mp1, /mnt/sys_drive) + by this many GiB (0 = keep golden size)") + +**Not computed from the physical drive.** And it is not merely a lab default — what a *real* +provisioned box runs, read from demo-hp itself: + + demo-hp: ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json ← no flag + demo-hp: pct config 9201 → mp0 … size=50G mp1 … size=20G rootfs size=32G + agent.json: keys = [authz backup deployment_mode escrow hub lan_resolver local_api log_level + oob privileged proxmox storage wg_tunnel] — grep 'grow|sysdata|datavol|size' = 0 hits + +**Tried and negative:** the flag, the config file, and the live unit on a production host. Nothing +supplies a non-zero grow. + +### A4 — Restore stages IN PLACE, on the docker volume, not on sys_drive + +Measured, not read: a homebox restore run **while sys_drive was constrained to 799 MB**: + + sys_drive avail 799.2M → 799.2M (untouched) + docker avail 37.4G → 37.4G (extracted in place) + result: ok=true in 22s, 9 blobs / 2305 MB intact, canary unchanged, healthy + +**The operator's proposed optimisation is already true for restore.** The constrained mount is written +only during **backup**. + +### The verdict on the ceiling + +**Real, on `mp1`.** But the useful statement is the mismatch: + +> A default box allows an app to grow to **50 GB of live data** (mp0) while capping its **local +> recovery unit at 20 GB** (mp1). Nothing prevents an app from crossing the line, and crossing it is +> invisible until a backup fails. + +--- + +## 3. Part C — the sizing rule, corrected + +### C1 — a file-only app is 1.00×, measured + +| app | volume | unit | ratio | shape | +|---|---|---|---|---| +| **homebox** | 2305 MB | **2305 MB** | **1.00×** | file-only, sqlite, **no `db-dumps/` directory at all** | +| grafana | 50 MB | 48 MB | 0.96× | file-only | +| rallly (empty DB) | 962 MB | 963 MB | **1.00×** | DB app — tar 1 008 820 736 B + dump **102 766 B** | +| rallly (21 GB DB) | 21 616 MB | 41 149 MB | 1.90× | Campaign 10 §6 | + +### The rule is not a constant — it is a sum + +> **unit ≈ volume-tar bytes + logical DB-dump bytes** + +`1.90×` was the DB worst case, where the volume is almost entirely live rows. The same app with an +empty DB measured **1.00×**, which is the proof the ratio tracks dump size and not "DB-ness". + +| app shape | ratio | what a **20 G** default sys_drive holds | +|---|---|---| +| file-only | **1.00×** | **≈ 19 GB of app data** | +| DB app, volume mostly live rows | **→ 2.0×** | **≈ 10 GB of app data** | + +Two numbers, not one — and the difference is material for guidance. + +### C2 — both representations are used, and the reason is explicit + +Not redundancy. `internal/backup/restore_unit.go:262-266`, verbatim: + + // F17: the captured .sql dump is the authoritative logical DB state — replay it AFTER the volume + // restore, so the dump WINS over any volume-tar copy of the database. + // R-47: the replay happens with ONLY the database service up. This used to run after + // RecreateStackFromUnit had already brought the WHOLE stack up, letting the application rebuild + // schema objects underneath the replay (H4, DIAG-immich-restore-round2-2026-07-19). + +Order is: stop → restore volume tars → recreate definition → **DB-only start → replay dump** → start stack. + +**Could the DB volume's tar be dropped?** The dump is +`pg_dump -U -d --clean --if-exists --no-owner --no-privileges` (`internal/appbackup/dbdump.go:225`) +— a **single database, no globals/roles**. Because of `--no-owner`, a fresh `initdb` (which creates the +role from `POSTGRES_USER`/`POSTGRES_PASSWORD`) plus this dump is **logically sufficient for a +single-DB app**. So the tar is a **physical fallback**, not a correctness requirement. + +**And dropping it would fix the D5/R-127(b) trap rather than risk it.** The trap exists *because* the +tar is restored: PGDATA present ⇒ postgres skips `initdb` ⇒ `POSTGRES_PASSWORD` is ignored ⇒ a +regenerated password authenticates over the local trust socket while failing over TCP. A fresh +`initdb` applies the password correctly. + +**The reason to keep it anyway — and this is the answer worth more than the optimisation:** the tar is +the only copy if the dump is bad, and **`ValidateDump` currently WARNS rather than gates**. Observed +live, twice, in this very session: + + [WARN] [backup] ValidateDump: …/rallly-postgres.sql is structurally valid (32 tables) but its + accounts table has NO rows — the dump may predate the customer's data + +Dropping the DB volume tar would halve DB-app units *and* close R-127(b) — **but only after +`ValidateDump` is promoted from a warning to a gate.** In its current form the tar is load-bearing. + +--- + +## 4. Part B — what happens when it does not fit + +Method: ballast `/mnt/sys_drive` to **799 MB free** (homebox's unit needs 2.3 GB), then run the real +backup. Pre-state recorded so truncation would be detectable. + +### On backup: **outcome 1 — refuses cleanly, per app** + + [WARN] Volume dump failed for homebox/homebox_homebox_data (last good dump preserved): + tar: write error: No space left on device — exit status 1 + [ERROR] Volume dump failed for homebox: volume dump failed for: homebox_homebox_data + … + [INFO] Volume dump: papra/papra_papra_data → 2.5 KB ← other apps CONTINUE + /api/backup/status → db_dump.success: FALSE + +**The "last good dump preserved" claim is TRUE — verified, not taken on trust:** + +| check | pre | post | +|---|---|---| +| size | 2 416 226 304 B | **2 416 226 304 B** | +| md5 | `4151146ed4af56cd19976cb5e97f137e` | **`4151146ed4af56cd19976cb5e97f137e`** | +| `tar -tf` end-to-end | — | **VALID** | +| `.tmp`/`.part` residue | — | **none** | + +Write-to-temp-then-swap held. **No partial artifact can later read as a completed backup — explicitly +NOT the R-156 family**, and the mount was not filled to the point of breaking the guest: apps stayed +up and healthy throughout. + +### On restore: correct data, honest success + +Restoring the preserved unit **while still constrained** returned the last good state intact — 9 blobs, +2305 MB, canary `C10-M02-B-081947`, healthy, `ok: true` in 22 s. It did **not** return partial data, +and it did **not** claim success over a partial restore. + +### The gap: it is SILENT + +**Zero events** reached the hub during the overflow window (checked in the hub's own log, which has +0 restarts over 43 h). The only trace is the controller's `[ERROR]` line and a `success:false` field +in an API nobody polls. + +--- + +## 5. Findings + +### R-158 (new) — the app-data (Tier-1 recovery unit) backup has NO failure signal to the hub + +**Register grepped:** no item covers the app-data path. **R-97** is the same defect on the +**whole-guest** tier and is SHIPPED (`ROADMAP.md:157`). This is R-97 exactly one tier over. + +- `NotifyBackupFailed` **exists** (`internal/notify/notifier.go:297`), and the hub **allowlists** + `backup_failed`. +- Its **only production caller** is the off-box/NAS leg: `cmd/controller/main.go:659` + (*"Off-box (NAS) mentés sikertelen"*). +- The backup manager's notify seams are `tier2Notify`, `offboxNotify`, + `offboxEnlargeBlockedNotify` (`internal/backup/backup.go:33,37,58`) — **none for the local + recovery-unit capture**. +- Measured consequence: 2 of 5 volume dumps failed with `ENOSPC`; nothing was emitted. + +The codebase already names this pattern, in a test written for R-97 +(`cmd/controller/quiesce_notify_wiring_test.go:14`): + +> *"…which is exactly the state R-97 found `internal/quiesce` in (NotifyBackupFailed existed, the hub +> allowlisted backup_failed, and no code connected them)."* + +This is the **fifth** instance of *seam built but never wired*. **Proposed shape** (not implemented): +a `unitNotify` seam on the recovery-unit capture, wired in `main()` like `SetOffboxNotify`, emitting +the existing `backup_failed`; severity error; per-app detail. A test should pin the **consequence** +(a failed unit capture produces an event) rather than the mechanism. + +### Not filed + +- **The 20 G / 50 G mismatch** — a tier-sizing decision (§6), not a defect. `SysDataGrowGB` is the + intended lever and it works; nothing sets it. +- **`ValidateDump` warns rather than gates** — pre-existing, and it is the reason the DB volume tar + must stay for now (§C2). Named here as a dependency of any future unit-halving work, not re-filed. + +--- + +## 6. The sizing rule as it stands — for the architecture doc, NOT written into it here + +> **Recovery unit ≈ volume-tar bytes + logical DB-dump bytes.** +> - file-only app → **1.00×** → a 20 G `sys_drive` holds **≈ 19 GB** of app data +> - DB app whose volume is mostly live rows → **→ 2.0×** → a 20 G `sys_drive` holds **≈ 10 GB** +> +> `mp0` (live data) ships at **50 G** and `mp1` (recovery units) at **20 G**. `SysDataGrowGB` exists to +> raise mp1 per customer and **is 0 on every box observed**, including a production one. + +**What this bounds for D5.** D5's promise is that restoring an app needs *the drive and nothing else* — +Lane 1's independence from the guest. **That independence is bounded by app size**, and the bound is: + +> **≈ 19 GB for a file-only app and ≈ 10 GB for a DB-backed app, on a default box.** + +Past that the unit cannot be captured at all, and the app silently falls back to the operator-driven +whole-guest route — silently because of **R-158**. + +--- + +## 7. What did not run, and why + +- **The ~5 GB linearity point** — the brief said skip it; skipped. +- **A third overflow mode (filling the mount to breakage)** was not forced. The per-app refusal + preserved space and other apps continued, so the mount never actually filled; provoking a true + full-mount state would have required disabling the guard that turned out to work. +- **MariaDB shape** not measured — `mariadb-dump --single-transaction --routines --triggers` + (`dbdump.go:243`) is also single-database, so the same reasoning should apply, but that is + **inference, not measurement**. +- **Teardown remains OWED and was NOT performed** — VM 311, `c10-scratch`, PBS datastore `felhom-c10` + + user/token, restic subaccount `u629488-sub4` (id 281530), hub customer **`c10-soak` (disposition: + DELETE)**, and the credential dirs. Commands in `CAMPAIGN-10-…md` §9. +- **Rig left as found**, plus: the probe ballast was removed; `cc_bulk` (the M-band table) was dropped + after its numbers were captured; homebox carries ~2.3 GB of probe blobs in `/data/c10bulk`. Apps, + canaries and runner intact.