Campaign 10: M-band RTO measured — RTO ~= 40s + 26.9s/GB, and a capacity ceiling that matters more

The S figures (66 MB -> 42.0s, two passes agreeing to 0.6s) had a spread tight
enough to prove fixed work dominates, which is exactly why they said nothing about
M. Second point taken 327x larger, same app, same method: clock from restore
request to the app serving the correct discriminator.

rallly's postgres volume grown 66 MB -> 21.1 GB (200k rows, STORAGE EXTERNAL so
TOAST cannot compress it into a fake number). Two reps:

  rep 1  backup 406.4s  unit 41149 MB  RTO 624.5s  discriminator correct
  rep 2  backup 387.2s  unit 41133 MB  RTO 591.8s  discriminator correct

327x the data cost 14.5x the time - strongly sub-linear:
  RTO ~= 40s + 26.9 s/GB      backup ~= 29s + 17.4 s/GB
  10 GB -> 5.2 min   20 GB -> 9.6 min (measured 10.1)   100 GB -> 46 min
The fixed ~40s dominates below ~1.5 GB, which IS the S band and explains its tight
clustering.

The more consequential result is capacity. A DB-backed app's recovery unit is
1.90x its data (volume tar PLUS SQL dump): 21.1 GB produced a 40.2 GB unit. The
default appliance ships /mnt/sys_drive at 20 GB, so the largest app that can hold
a local Tier-1/2 recovery unit on a default box is about 10 GB - and that fills the
volume. The M band does not fit on a default box at all; this test only reached
21 GB because sys_drive was first grown 20G -> 70G with the same operation the
product performs via SysDataGrowGB. A tier-sizing decision, not a defect, but it
is invisible until an app crosses it.

Caveats stated in the doc: two points define a line but do not test linearity; the
1.90x is DB-app-specific and a file-only app should be nearer 1.0x (inferred, not
measured); synthetic incompressible data; one app, one box.
This commit is contained in:
2026-08-02 08:31:51 +02:00
parent 7ba7c2a271
commit 5f35aa0346
4 changed files with 125 additions and 8 deletions
+8 -3
View File
@@ -50,9 +50,14 @@ deliberately** — two violations were harness defects), run 2b (**39 cycles, 12
- **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via
`bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names
it. That is the case R-117's spike called "the worse half".
- **RTO (Tier 1, rallly, 66 MB):** run 1 median **42.0 s**, run 2b median **41.4 s** over 38 restores
— two independent passes agreeing to 0.6 s. **S band's lower end only**; nothing extrapolates to M
or L. **RPO not measured.**
- **RTO, both bands measured.** S: 66 MB → **42.0 s** / **41.4 s** across two passes (66 restores).
M: 21.1 GB → **608 s** mean over 2 reps, both returning the correct discriminator. 327× the data
cost 14.5× the time, giving **RTO ≈ 40 s + 26.9 s/GB** (backup ≈ 29 s + 17.4 s/GB). The fixed ~40 s
dominates below ~1.5 GB — that is the S band, and why its numbers clustered so tightly.
- **Capacity ceiling, and the more consequential result:** a DB-backed app's recovery unit is **1.90×**
its data (volume tar + SQL dump). The default `/mnt/sys_drive` is **20 GB**, so on a default box the
largest locally-backupable app is **≈ 10 GB** — **the M band does not fit at all** without a
per-customer `SysDataGrowGB`. **RPO still not measured.**
- **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds
flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots
and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)**