From 5f35aa0346562557efe2280c3c60d3613ed94606 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 2 Aug 2026 08:31:51 +0200 Subject: [PATCH] =?UTF-8?q?Campaign=2010:=20M-band=20RTO=20measured=20?= =?UTF-8?q?=E2=80=94=20RTO=20~=3D=2040s=20+=2026.9s/GB,=20and=20a=20capaci?= =?UTF-8?q?ty=20ceiling=20that=20matters=20more?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The S figures (66 MB -> 42.0s, two passes agreeing to 0.6s) had a spread tight enough to prove fixed work dominates, which is exactly why they said nothing about M. Second point taken 327x larger, same app, same method: clock from restore request to the app serving the correct discriminator. rallly's postgres volume grown 66 MB -> 21.1 GB (200k rows, STORAGE EXTERNAL so TOAST cannot compress it into a fake number). Two reps: rep 1 backup 406.4s unit 41149 MB RTO 624.5s discriminator correct rep 2 backup 387.2s unit 41133 MB RTO 591.8s discriminator correct 327x the data cost 14.5x the time - strongly sub-linear: RTO ~= 40s + 26.9 s/GB backup ~= 29s + 17.4 s/GB 10 GB -> 5.2 min 20 GB -> 9.6 min (measured 10.1) 100 GB -> 46 min The fixed ~40s dominates below ~1.5 GB, which IS the S band and explains its tight clustering. The more consequential result is capacity. A DB-backed app's recovery unit is 1.90x its data (volume tar PLUS SQL dump): 21.1 GB produced a 40.2 GB unit. The default appliance ships /mnt/sys_drive at 20 GB, so the largest app that can hold a local Tier-1/2 recovery unit on a default box is about 10 GB - and that fills the volume. The M band does not fit on a default box at all; this test only reached 21 GB because sys_drive was first grown 20G -> 70G with the same operation the product performs via SysDataGrowGB. A tier-sizing decision, not a defect, but it is invisible until an app crosses it. Caveats stated in the doc: two points define a line but do not test linearity; the 1.90x is DB-app-specific and a file-only app should be nearer 1.0x (inferred, not measured); synthetic incompressible data; one app, one box. --- REPORT-campaign10.md | 11 ++- ...CAMPAIGN-10-two-storage-soak-2026-07-31.md | 43 +++++++++-- .../rto-m-band-2026-08-02.md | 76 +++++++++++++++++++ .../state/rto-mband.tsv | 3 + 4 files changed, 125 insertions(+), 8 deletions(-) create mode 100644 documentation/tests/campaign10-evidence-2026-07-31/rto-m-band-2026-08-02.md create mode 100644 documentation/tests/campaign10-evidence-2026-07-31/state/rto-mband.tsv diff --git a/REPORT-campaign10.md b/REPORT-campaign10.md index 5478ba3..0f29432 100644 --- a/REPORT-campaign10.md +++ b/REPORT-campaign10.md @@ -50,9 +50,14 @@ deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 - **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via `bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names it. That is the case R-117's spike called "the worse half". -- **RTO (Tier 1, rallly, 66 MB):** run 1 median **42.0 s**, run 2b median **41.4 s** over 38 restores - — two independent passes agreeing to 0.6 s. **S band's lower end only**; nothing extrapolates to M - or L. **RPO not measured.** +- **RTO, both bands measured.** S: 66 MB → **42.0 s** / **41.4 s** across two passes (66 restores). + M: 21.1 GB → **608 s** mean over 2 reps, both returning the correct discriminator. 327× the data + cost 14.5× the time, giving **RTO ≈ 40 s + 26.9 s/GB** (backup ≈ 29 s + 17.4 s/GB). The fixed ~40 s + dominates below ~1.5 GB — that is the S band, and why its numbers clustered so tightly. +- **Capacity ceiling, and the more consequential result:** a DB-backed app's recovery unit is **1.90×** + its data (volume tar + SQL dump). The default `/mnt/sys_drive` is **20 GB**, so on a default box the + largest locally-backupable app is **≈ 10 GB** — **the M band does not fit at all** without a + per-customer `SysDataGrowGB`. **RPO still not measured.** - **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)** — diff --git a/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md b/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md index 16bf074..6740d56 100644 --- a/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md +++ b/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md @@ -404,12 +404,45 @@ reads like data loss and is not. The two independent passes agree to **0.6 s on the median**, and run 2b's wider spread (29.1–49.6 s against 38.8–44.3 s) is the cost of measuring under hard resets, agent kills and a near-full drive. -**Band: the S band's lower end only.** 66 MB is small, and the tight distribution says fixed work — -stack stop, volume restore, start, health wait — dominates. **The byte-proportional term is exactly -what 66 MB fails to exercise, so nothing here extrapolates to M or L**, and no such claim is made. +### M band — measured 2026-08-02 -**RPO is not measured.** Backups were driven on demand, not on the schedule, so the observed -intervals say nothing about the product's RPO. +The S figures could not speak for M, so a second point was taken **327× larger, same app, same +method**. Full evidence: `../tests/campaign10-evidence-2026-07-31/rto-m-band-2026-08-02.md`. + +| rep | volume | backup | recovery unit | **RTO to correct data** | discriminator | +|---|---|---|---|---|---| +| 1 | 21 616 MB | 406.4 s | 41 149 MB | **624.5 s** | correct | +| 2 | 21 615 MB | 387.2 s | 41 133 MB | **591.8 s** | correct | +| **mean** | **21.1 GB** | **396.8 s** | **40.2 GB (1.90×)** | **608.1 s (10.1 min)** | **2/2** | + +**327× the data cost 14.5× the time** — strongly sub-linear, so: + +> **RTO ≈ 40 s + 26.9 s/GB** · **backup ≈ 29 s + 17.4 s/GB** +> +> 1 GB → 67 s · 10 GB → 5.2 min · **20 GB → 9.6 min** (measured 10.1) · 50 GB → 23 min · 100 GB → 46 min + +**The fixed ~40 s dominates below ~1.5 GB** — that is the S band, and it is why the S numbers were so +tightly clustered. Above it, RTO is essentially linear in data. + +### The capacity ceiling — the more consequential result + +**A DB-backed app's recovery unit is 1.90× its data** (volume tar **plus** SQL dump): 21.1 GB produced +a **40.2 GB** unit. The default appliance ships `/mnt/sys_drive` at **20 GB** (`mp1 … size=20G`; +`SysDataGrowGB` is a per-customer grow, 0 for this customer). Therefore: + +> **on a default box the largest app that can hold a local Tier-1/2 recovery unit is ≈ 10 GB, and that +> fills the volume completely — realistically ≈ 8 GB. The M band does not fit on a default box at +> all.** This test only reached 21 GB because `/mnt/sys_drive` was first grown to 70 G. + +A sizing decision for the S/M/L tiers rather than a defect — but it is the constraint that actually +bites, and it is invisible until an app crosses it. + +**Caveats, stated not buried:** two points define a line but do not test linearity (no ~5 GB point was +taken); the 1.90× is DB-app-specific and a file-only app should be nearer 1.0× — **inferred, not +measured**; the data is deliberately incompressible synthetic, so well-compressing real data moves +both numbers; one app, one shape, one box. + +**RPO is still not measured.** Backups were driven on demand, not on the schedule. --- diff --git a/documentation/tests/campaign10-evidence-2026-07-31/rto-m-band-2026-08-02.md b/documentation/tests/campaign10-evidence-2026-07-31/rto-m-band-2026-08-02.md new file mode 100644 index 0000000..b5a5791 --- /dev/null +++ b/documentation/tests/campaign10-evidence-2026-07-31/rto-m-band-2026-08-02.md @@ -0,0 +1,76 @@ +# RTO — the M band, measured (2026-08-02) + +The S-band figure (66 MB → 42.0 s, two independent passes agreeing to 0.6 s) had a spread tight +enough to prove **fixed work dominates**, which is exactly why it said nothing about M. This adds a +second point 327× larger, measured on the **same app, the same way**: clock from the restore request +to the app **serving the correct discriminator**, read over the path `DATABASE_URL` actually names. + +## Method + +`rallly`'s postgres volume grown 66 MB → **21.1 GB** by inserting 200 000 rows into a `cc_bulk` table. +`ALTER TABLE … SET STORAGE EXTERNAL` is load-bearing: without it TOAST compresses `repeat(...)` to +almost nothing and the volume never really grows, which would have produced a fake M-band number. +The `cc_proof` canary row is untouched, so the discriminator is unchanged. + +`/mnt/sys_drive` was first grown 20 G → 70 G with `pct resize` — the same operation the product +performs via `SysDataGrowGB` (`felhom-agent/internal/reconcile/bringup.go:426`). **It had to be**: +see the capacity ceiling below. + +## Results — 2 reps, both returning the correct discriminator + +| rep | volume | backup | recovery unit | **RTO to correct data** | discriminator | +|---|---|---|---|---|---| +| 1 | 21 616 MB | 406.4 s | 41 149 MB | **624.5 s** | correct | +| 2 | 21 615 MB | 387.2 s | 41 133 MB | **591.8 s** | correct | +| **mean** | **21.1 GB** | **396.8 s** | **40.2 GB (1.90×)** | **608.1 s (10.1 min)** | **2/2** | + +## The two-point model + +| | 66 MB (S) | 21.1 GB (M) | ratio | +|---|---|---|---| +| data | ×1 | **×327** | | +| RTO | 42.0 s | **608.1 s** | **×14.5** | + +Time grew 14.5× for 327× the data — strongly sub-linear, i.e. a large fixed cost plus a +byte-proportional term: + +> **RTO ≈ 40 s + 26.9 s/GB** · **backup ≈ 29 s + 17.4 s/GB** + +| size | projected RTO | +|---|---| +| 1 GB | 67 s | +| 5 GB | 2.9 min | +| 10 GB | 5.2 min | +| **20 GB** | **9.6 min** (measured **10.1 min** — the model is fit through the mean) | +| 50 GB | 23.1 min | +| 100 GB | 45.5 min | + +**Where the bands actually break.** The fixed ~40 s dominates below roughly **1.5 GB** — that is the S +band, and it is why the S numbers were so tightly clustered. Above it the per-GB term takes over and +RTO becomes essentially linear in data. + +## The capacity ceiling — the more consequential result + +**The recovery unit is 1.90× the app's data**, because for a DB-backed app it carries *both* the +volume tar and the SQL dump: 21.1 GB of data produced a **40.2 GB** unit. + +The default appliance ships `/mnt/sys_drive` at **20 GB** (`mp1 … size=20G`, the golden's baked size; +`SysDataGrowGB` is a per-customer grow that was 0 for this customer). So on a default box: + +> **the largest app that can hold a local Tier-1/Tier-2 recovery unit is ≈ 10 GB — and that fills the +> volume completely.** A realistic ceiling is nearer **8 GB**. + +**The M band does not fit on a default box at all.** This test only reached 21 GB because +`/mnt/sys_drive` was grown to 70 G first. That is a sizing decision for the S/M/L tiers, not a defect +— but it is the constraint that actually bites, and it is invisible until an app crosses it. + +## Caveats — stated, not buried + +- **Two points define a line; they do not test linearity.** A third point (~5 GB) would confirm the + model's shape. Not taken. +- **The 1.90× ratio is DB-app-specific.** It is volume-tar + SQL-dump. A file-only app (sqlite, no DB + container) should be nearer 1.0×, but that was **not measured** and is inference. +- **Synthetic data.** `repeat(md5(...))` stored uncompressed is deliberately incompressible; real app + data that compresses well would move both the unit size and the per-GB rate. +- One app, one shape, one box. The fixed term in particular is a property of this stack's stop/start + and health-wait, which differs per app. diff --git a/documentation/tests/campaign10-evidence-2026-07-31/state/rto-mband.tsv b/documentation/tests/campaign10-evidence-2026-07-31/state/rto-mband.tsv new file mode 100644 index 0000000..af588aa --- /dev/null +++ b/documentation/tests/campaign10-evidence-2026-07-31/state/rto-mband.tsv @@ -0,0 +1,3 @@ +rep utc app volume_mb unit_mb backup_s restore_s correct detail +1 2026-08-02T06:12:50Z rallly 21615 41149 406.4 624.5 True {"op": "restore", "stack": "rallly", "ok": true, "message": "rallly vissza\u00e1ll\u00edtva (helyi).", "finished_at": "2026-08-02T06:12:29.923625994Z"} +2 2026-08-02T06:30:02Z rallly 21615 41133 387.2 591.8 True {"op": "restore", "stack": "rallly", "ok": true, "message": "rallly vissza\u00e1ll\u00edtva (helyi).", "finished_at": "2026-08-02T06:29:40.985423678Z"}