Campaign 10: M-band RTO measured — RTO ~= 40s + 26.9s/GB, and a capacity ceiling that matters more

The S figures (66 MB -> 42.0s, two passes agreeing to 0.6s) had a spread tight
enough to prove fixed work dominates, which is exactly why they said nothing about
M. Second point taken 327x larger, same app, same method: clock from restore
request to the app serving the correct discriminator.

rallly's postgres volume grown 66 MB -> 21.1 GB (200k rows, STORAGE EXTERNAL so
TOAST cannot compress it into a fake number). Two reps:

  rep 1  backup 406.4s  unit 41149 MB  RTO 624.5s  discriminator correct
  rep 2  backup 387.2s  unit 41133 MB  RTO 591.8s  discriminator correct

327x the data cost 14.5x the time - strongly sub-linear:
  RTO ~= 40s + 26.9 s/GB      backup ~= 29s + 17.4 s/GB
  10 GB -> 5.2 min   20 GB -> 9.6 min (measured 10.1)   100 GB -> 46 min
The fixed ~40s dominates below ~1.5 GB, which IS the S band and explains its tight
clustering.

The more consequential result is capacity. A DB-backed app's recovery unit is
1.90x its data (volume tar PLUS SQL dump): 21.1 GB produced a 40.2 GB unit. The
default appliance ships /mnt/sys_drive at 20 GB, so the largest app that can hold
a local Tier-1/2 recovery unit on a default box is about 10 GB - and that fills the
volume. The M band does not fit on a default box at all; this test only reached
21 GB because sys_drive was first grown 20G -> 70G with the same operation the
product performs via SysDataGrowGB. A tier-sizing decision, not a defect, but it
is invisible until an app crosses it.

Caveats stated in the doc: two points define a line but do not test linearity; the
1.90x is DB-app-specific and a file-only app should be nearer 1.0x (inferred, not
measured); synthetic incompressible data; one app, one box.
This commit is contained in:
2026-08-02 08:31:51 +02:00
parent 7ba7c2a271
commit 5f35aa0346
4 changed files with 125 additions and 8 deletions
@@ -404,12 +404,45 @@ reads like data loss and is not.
The two independent passes agree to **0.6 s on the median**, and run 2b's wider spread (29.149.6 s
against 38.844.3 s) is the cost of measuring under hard resets, agent kills and a near-full drive.
**Band: the S band's lower end only.** 66 MB is small, and the tight distribution says fixed work —
stack stop, volume restore, start, health wait — dominates. **The byte-proportional term is exactly
what 66 MB fails to exercise, so nothing here extrapolates to M or L**, and no such claim is made.
### M band — measured 2026-08-02
**RPO is not measured.** Backups were driven on demand, not on the schedule, so the observed
intervals say nothing about the product's RPO.
The S figures could not speak for M, so a second point was taken **327× larger, same app, same
method**. Full evidence: `../tests/campaign10-evidence-2026-07-31/rto-m-band-2026-08-02.md`.
| rep | volume | backup | recovery unit | **RTO to correct data** | discriminator |
|---|---|---|---|---|---|
| 1 | 21 616 MB | 406.4 s | 41 149 MB | **624.5 s** | correct |
| 2 | 21 615 MB | 387.2 s | 41 133 MB | **591.8 s** | correct |
| **mean** | **21.1 GB** | **396.8 s** | **40.2 GB (1.90×)** | **608.1 s (10.1 min)** | **2/2** |
**327× the data cost 14.5× the time** — strongly sub-linear, so:
> **RTO ≈ 40 s + 26.9 s/GB** · **backup ≈ 29 s + 17.4 s/GB**
>
> 1 GB → 67 s · 10 GB → 5.2 min · **20 GB → 9.6 min** (measured 10.1) · 50 GB → 23 min · 100 GB → 46 min
**The fixed ~40 s dominates below ~1.5 GB** — that is the S band, and it is why the S numbers were so
tightly clustered. Above it, RTO is essentially linear in data.
### The capacity ceiling — the more consequential result
**A DB-backed app's recovery unit is 1.90× its data** (volume tar **plus** SQL dump): 21.1 GB produced
a **40.2 GB** unit. The default appliance ships `/mnt/sys_drive` at **20 GB** (`mp1 … size=20G`;
`SysDataGrowGB` is a per-customer grow, 0 for this customer). Therefore:
> **on a default box the largest app that can hold a local Tier-1/2 recovery unit is ≈ 10 GB, and that
> fills the volume completely — realistically ≈ 8 GB. The M band does not fit on a default box at
> all.** This test only reached 21 GB because `/mnt/sys_drive` was first grown to 70 G.
A sizing decision for the S/M/L tiers rather than a defect — but it is the constraint that actually
bites, and it is invisible until an app crosses it.
**Caveats, stated not buried:** two points define a line but do not test linearity (no ~5 GB point was
taken); the 1.90× is DB-app-specific and a file-only app should be nearer 1.0× — **inferred, not
measured**; the data is deliberately incompressible synthetic, so well-compressing real data moves
both numbers; one app, one shape, one box.
**RPO is still not measured.** Backups were driven on demand, not on the schedule.
---