admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SPIKE: recovery-unit space — the ceiling is real on mp1, overflow is clean, but silent (R-158)

Browse Source 
Three headline answers.

1. The ceiling is REAL and on mp1 (/mnt/sys_drive), but its shape is a MISMATCH
   rather than a single number. A1: docker's data-root is a SEPARATE 50G volume
   (mp0) and every app volume resolves there, so app DBs are NOT on sys_drive -
   build-golden.sh:68's "like the Docker-data" reading is correct. A2: the
   recovery units ARE on sys_drive, which the golden ships at 20G. So a box
   permits 50 GB of live app data while capping local backup at 20 GB, and
   crossing that line is invisible until a backup fails.

   A3 rules out the lab-default explanation: --sysdata-grow defaults to 0
   (main.go:178) and is not computed from the drive. demo-hp's REAL guest 9201
   runs a bare --config ExecStart and shows mp0 50G / mp1 20G; agent.json has no
   sizing keys at all. A4, measured not read: restore extracts IN PLACE on the
   docker volume - sys_drive avail was 799.2M before and after a restore run under
   constraint - so the constrained mount is written only during backup.

2. Overflow behaves WELL. With sys_drive ballasted to 799 MB, backup refused
   per-app ("No space left on device"), other apps continued, status reported
   success=false, and the "last good dump preserved" claim VERIFIED byte-for-byte:
   size and md5 unchanged, tar valid end-to-end, no .tmp residue. Restoring that
   preserved unit under the same constraint returned correct data and claimed
   success honestly. Explicitly NOT the R-156 family.

3. But it is SILENT - R-158, filed. Zero events reached the hub.
   NotifyBackupFailed exists and the hub allowlists backup_failed, but the only
   production caller is the off-box/NAS leg (main.go:659); the backup manager has
   tier2/offbox/offbox-enlarge notify seams and none for the local recovery-unit
   capture. This is R-97's shipped defect exactly one tier over, and the fifth
   instance of "seam built but never wired" - a pattern the codebase names in its
   own R-97 wiring test.

Sizing rule corrected: unit ~= volume-tar bytes + logical dump bytes, not a
constant 1.90x. Measured C1: file-only apps are 1.00x (homebox 2305->2305 MB, no
db-dumps dir at all), and the SAME DB app with an empty DB is also 1.00x. So a 20G
sys_drive holds ~19 GB file-only or ~10 GB DB-backed. That is the bound on D5's
Lane-1 independence.

C2: both representations are used for a reason stated in code (F17 - the dump is
authoritative and WINS over the tar; R-47 - replayed with only the DB service up).
The dump is single-database pg_dump --no-owner, so a fresh initdb plus the dump is
logically sufficient and the tar is a PHYSICAL FALLBACK. Dropping it would halve
DB-app units and also close the D5/R-127(b) password trap (restored PGDATA makes
postgres skip initdb and ignore POSTGRES_PASSWORD) - but only after ValidateDump
is promoted from a warning to a gate, since it currently WARNS on a dump whose
accounts table has no rows. In its present form the tar is load-bearing.

No production code, no template change. Teardown still owed and itemised.
This commit is contained in:
authentik Default Admin
2026-08-02 09:07:09 +02:00
parent 5f35aa0346
commit 0afadbdeff
1 changed files with 270 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/SPIKE-recovery-unit-space-2026-08-02.md
+270
View File
@@ -0,0 +1,270 @@
# SPIKE — where does the space actually go, and what happens when it runs out? (2026-08-02)
**Class: spike.** Measurements and source reading only. No production code, no template change, no
design implementation. Continues `CAMPAIGN-10-two-storage-soak-2026-07-31.md` §6.
**The three headline answers:**
1. **The ceiling is REAL and it is on `mp1` (`/mnt/sys_drive`) — but its shape is a MISMATCH, not a
single number.** Live app data lives on a *different* 50 G volume. A box therefore permits **50 GB
of app data while capping local backup at 20 GB**.
2. **Overflow behaves WELL.** Backup refuses per-app, preserves the last good unit byte-for-byte, and a
later restore of it returns correct data. **Not the R-156 family.**
3. **But the failure is SILENT.** No event on any customer- or operator-visible channel. **R-158**, and
it is R-97's defect exactly one tier over.
---
## 1. Venue and baselines
| | |
|---|---|
| Venue | the Campaign 10 rig — **VM 311 on demo-hp**, apps/canaries/runner intact. Not rebuilt, not torn down. |
| controller (running) | **0.188.0** (`/etc/felhom-controller-image`) · golden bakes 0.188.0 · `main` 0.188.0 (`4115e88`) |
| agent | **0.119.0** (`felhom-agent/CHANGELOG.md` top; `4663df7`) |
| hub | **0.86.0** |
| `felhom.eu` HEAD | `5f35aa0`, clean |
| rig `sys_drive` | **68.8 G** — Campaign 10 grew it 20 G → 70 G with `pct resize`. **The golden ships 20 G.** |
| Untouched | demo-felhom, `app-catalog-felhom.eu`, `local-lvm`, `drill-r50`, guest 9201 on demo-hp (read-only) |
---
## 2. Part A — the three unverified premises
### A1 — Docker's data-root is a SEPARATE volume. App DBs are NOT on sys_drive.
`build-golden.sh:68`'s "like the Docker-data" reading is **correct**. From the daemon and the rig, not
inferred:
docker info → data-root=/var/lib/docker
every app volume → /var/lib/docker/volumes/<v>/_data → findmnt target /var/lib/docker
Mount table, each verified with `mountpoint -q` **before** reading sizes (Campaign 10's own sampler trap):
| mount | device | size | used | avail | holds |
|---|---|---|---|---|---|
| `/` | disk-0 | 31.2 G | 944.7 M | 28.7 G | rootfs |
| **`/var/lib/docker`** (mp0) | disk-1 | **49 G** | 27 G | 19.5 G | **live app data + DBs** |
| **`/mnt/sys_drive`** (mp1) | disk-2 | 68.8 G* | 40.2 G | 25.6 G | **recovery units** |
| `/mnt/felhom-drives` | pve-root bind | 58.6 G | 5.8 G | 49.8 G | enrolled-drive binds |
\* rig value; the golden ships this at **20 G**.
### A2 — The recovery unit IS on sys_drive
/mnt/sys_drive/felhom-data/backups/primary/{rallly,homebox,grafana,papra}
findmnt --target … → /mnt/sys_drive /dev/mapper/pve-vm--9201--disk--2
The 40.2 GB measured in Campaign 10 sat here. **E-2 does not save it**: the unit is written to the
guest's own `sys_drive`, not to the backup-target drive.
### A3 — `SysDataGrowGB` is a flag defaulting to 0, and real boxes do not pass it
felhom-agent/cmd/felhom-agent/main.go:178
flag.IntVar(&sysDataGrow, "sysdata-grow", 0,
"for --selftest=bring-up|provision: grow the golden's SSD user-data volume (mp1, /mnt/sys_drive)
by this many GiB (0 = keep golden size)")
**Not computed from the physical drive.** And it is not merely a lab default — what a *real*
provisioned box runs, read from demo-hp itself:
demo-hp: ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json ← no flag
demo-hp: pct config 9201 → mp0 … size=50G mp1 … size=20G rootfs size=32G
agent.json: keys = [authz backup deployment_mode escrow hub lan_resolver local_api log_level
oob privileged proxmox storage wg_tunnel] — grep 'grow|sysdata|datavol|size' = 0 hits
**Tried and negative:** the flag, the config file, and the live unit on a production host. Nothing
supplies a non-zero grow.
### A4 — Restore stages IN PLACE, on the docker volume, not on sys_drive
Measured, not read: a homebox restore run **while sys_drive was constrained to 799 MB**:
sys_drive avail 799.2M → 799.2M (untouched)
docker avail 37.4G → 37.4G (extracted in place)
result: ok=true in 22s, 9 blobs / 2305 MB intact, canary unchanged, healthy
**The operator's proposed optimisation is already true for restore.** The constrained mount is written
only during **backup**.
### The verdict on the ceiling
**Real, on `mp1`.** But the useful statement is the mismatch:
> A default box allows an app to grow to **50 GB of live data** (mp0) while capping its **local
> recovery unit at 20 GB** (mp1). Nothing prevents an app from crossing the line, and crossing it is
> invisible until a backup fails.
---
## 3. Part C — the sizing rule, corrected
### C1 — a file-only app is 1.00×, measured
| app | volume | unit | ratio | shape |
|---|---|---|---|---|
| **homebox** | 2305 MB | **2305 MB** | **1.00×** | file-only, sqlite, **no `db-dumps/` directory at all** |
| grafana | 50 MB | 48 MB | 0.96× | file-only |
| rallly (empty DB) | 962 MB | 963 MB | **1.00×** | DB app — tar 1 008 820 736 B + dump **102 766 B** |
| rallly (21 GB DB) | 21 616 MB | 41 149 MB | 1.90× | Campaign 10 §6 |
### The rule is not a constant — it is a sum
> **unit ≈ volume-tar bytes + logical DB-dump bytes**
`1.90×` was the DB worst case, where the volume is almost entirely live rows. The same app with an
empty DB measured **1.00×**, which is the proof the ratio tracks dump size and not "DB-ness".
| app shape | ratio | what a **20 G** default sys_drive holds |
|---|---|---|
| file-only | **1.00×** | **≈ 19 GB of app data** |
| DB app, volume mostly live rows | **→ 2.0×** | **≈ 10 GB of app data** |
Two numbers, not one — and the difference is material for guidance.
### C2 — both representations are used, and the reason is explicit
Not redundancy. `internal/backup/restore_unit.go:262-266`, verbatim:
// F17: the captured .sql dump is the authoritative logical DB state — replay it AFTER the volume
// restore, so the dump WINS over any volume-tar copy of the database.
// R-47: the replay happens with ONLY the database service up. This used to run after
// RecreateStackFromUnit had already brought the WHOLE stack up, letting the application rebuild
// schema objects underneath the replay (H4, DIAG-immich-restore-round2-2026-07-19).
Order is: stop → restore volume tars → recreate definition → **DB-only start → replay dump** → start stack.
**Could the DB volume's tar be dropped?** The dump is
`pg_dump -U <user> -d <db> --clean --if-exists --no-owner --no-privileges` (`internal/appbackup/dbdump.go:225`)
— a **single database, no globals/roles**. Because of `--no-owner`, a fresh `initdb` (which creates the
role from `POSTGRES_USER`/`POSTGRES_PASSWORD`) plus this dump is **logically sufficient for a
single-DB app**. So the tar is a **physical fallback**, not a correctness requirement.
**And dropping it would fix the D5/R-127(b) trap rather than risk it.** The trap exists *because* the
tar is restored: PGDATA present ⇒ postgres skips `initdb` ⇒ `POSTGRES_PASSWORD` is ignored ⇒ a
regenerated password authenticates over the local trust socket while failing over TCP. A fresh
`initdb` applies the password correctly.
**The reason to keep it anyway — and this is the answer worth more than the optimisation:** the tar is
the only copy if the dump is bad, and **`ValidateDump` currently WARNS rather than gates**. Observed
live, twice, in this very session:
[WARN] [backup] ValidateDump: …/rallly-postgres.sql is structurally valid (32 tables) but its
accounts table has NO rows — the dump may predate the customer's data
Dropping the DB volume tar would halve DB-app units *and* close R-127(b) — **but only after
`ValidateDump` is promoted from a warning to a gate.** In its current form the tar is load-bearing.
---
## 4. Part B — what happens when it does not fit
Method: ballast `/mnt/sys_drive` to **799 MB free** (homebox's unit needs 2.3 GB), then run the real
backup. Pre-state recorded so truncation would be detectable.
### On backup: **outcome 1 — refuses cleanly, per app**
[WARN] Volume dump failed for homebox/homebox_homebox_data (last good dump preserved):
tar: write error: No space left on device — exit status 1
[ERROR] Volume dump failed for homebox: volume dump failed for: homebox_homebox_data
…
[INFO] Volume dump: papra/papra_papra_data → 2.5 KB ← other apps CONTINUE
/api/backup/status → db_dump.success: FALSE
**The "last good dump preserved" claim is TRUE — verified, not taken on trust:**
| check | pre | post |
|---|---|---|
| size | 2 416 226 304 B | **2 416 226 304 B** |
| md5 | `4151146ed4af56cd19976cb5e97f137e` | **`4151146ed4af56cd19976cb5e97f137e`** |
| `tar -tf` end-to-end | — | **VALID** |
| `.tmp`/`.part` residue | — | **none** |
Write-to-temp-then-swap held. **No partial artifact can later read as a completed backup — explicitly
NOT the R-156 family**, and the mount was not filled to the point of breaking the guest: apps stayed
up and healthy throughout.
### On restore: correct data, honest success
Restoring the preserved unit **while still constrained** returned the last good state intact — 9 blobs,
2305 MB, canary `C10-M02-B-081947`, healthy, `ok: true` in 22 s. It did **not** return partial data,
and it did **not** claim success over a partial restore.
### The gap: it is SILENT
**Zero events** reached the hub during the overflow window (checked in the hub's own log, which has
0 restarts over 43 h). The only trace is the controller's `[ERROR]` line and a `success:false` field
in an API nobody polls.
---
## 5. Findings
### R-158 (new) — the app-data (Tier-1 recovery unit) backup has NO failure signal to the hub
**Register grepped:** no item covers the app-data path. **R-97** is the same defect on the
**whole-guest** tier and is SHIPPED (`ROADMAP.md:157`). This is R-97 exactly one tier over.
- `NotifyBackupFailed` **exists** (`internal/notify/notifier.go:297`), and the hub **allowlists**
`backup_failed`.
- Its **only production caller** is the off-box/NAS leg: `cmd/controller/main.go:659`
(*"Off-box (NAS) mentés sikertelen"*).
- The backup manager's notify seams are `tier2Notify`, `offboxNotify`,
`offboxEnlargeBlockedNotify` (`internal/backup/backup.go:33,37,58`) — **none for the local
recovery-unit capture**.
- Measured consequence: 2 of 5 volume dumps failed with `ENOSPC`; nothing was emitted.
The codebase already names this pattern, in a test written for R-97
(`cmd/controller/quiesce_notify_wiring_test.go:14`):
> *"…which is exactly the state R-97 found `internal/quiesce` in (NotifyBackupFailed existed, the hub
> allowlisted backup_failed, and no code connected them)."*
This is the **fifth** instance of *seam built but never wired*. **Proposed shape** (not implemented):
a `unitNotify` seam on the recovery-unit capture, wired in `main()` like `SetOffboxNotify`, emitting
the existing `backup_failed`; severity error; per-app detail. A test should pin the **consequence**
(a failed unit capture produces an event) rather than the mechanism.
### Not filed
- **The 20 G / 50 G mismatch** — a tier-sizing decision (§6), not a defect. `SysDataGrowGB` is the
intended lever and it works; nothing sets it.
- **`ValidateDump` warns rather than gates** — pre-existing, and it is the reason the DB volume tar
must stay for now (§C2). Named here as a dependency of any future unit-halving work, not re-filed.
---
## 6. The sizing rule as it stands — for the architecture doc, NOT written into it here
> **Recovery unit ≈ volume-tar bytes + logical DB-dump bytes.**
> - file-only app → **1.00×** → a 20 G `sys_drive` holds **≈ 19 GB** of app data
> - DB app whose volume is mostly live rows → **→ 2.0×** → a 20 G `sys_drive` holds **≈ 10 GB**
>
> `mp0` (live data) ships at **50 G** and `mp1` (recovery units) at **20 G**. `SysDataGrowGB` exists to
> raise mp1 per customer and **is 0 on every box observed**, including a production one.
**What this bounds for D5.** D5's promise is that restoring an app needs *the drive and nothing else* —
Lane 1's independence from the guest. **That independence is bounded by app size**, and the bound is:
> **≈ 19 GB for a file-only app and ≈ 10 GB for a DB-backed app, on a default box.**
Past that the unit cannot be captured at all, and the app silently falls back to the operator-driven
whole-guest route — silently because of **R-158**.
---
## 7. What did not run, and why
- **The ~5 GB linearity point** — the brief said skip it; skipped.
- **A third overflow mode (filling the mount to breakage)** was not forced. The per-app refusal
preserved space and other apps continued, so the mount never actually filled; provoking a true
full-mount state would have required disabling the guard that turned out to work.
- **MariaDB shape** not measured — `mariadb-dump --single-transaction --routines --triggers`
(`dbdump.go:243`) is also single-database, so the same reasoning should apply, but that is
**inference, not measurement**.
- **Teardown remains OWED and was NOT performed** — VM 311, `c10-scratch`, PBS datastore `felhom-c10`
+ user/token, restic subaccount `u629488-sub4` (id 281530), hub customer **`c10-soak` (disposition:
DELETE)**, and the credential dirs. Commands in `CAMPAIGN-10-…md` §9.
- **Rig left as found**, plus: the probe ballast was removed; `cc_bulk` (the M-band table) was dropped
after its numbers were captured; homebox carries ~2.3 GB of probe blobs in `/data/c10bulk`. Apps,
canaries and runner intact.