Files

T

admin 5f35aa0346 Campaign 10: M-band RTO measured — RTO ~= 40s + 26.9s/GB, and a capacity ceiling that matters more

The S figures (66 MB -> 42.0s, two passes agreeing to 0.6s) had a spread tight
enough to prove fixed work dominates, which is exactly why they said nothing about
M. Second point taken 327x larger, same app, same method: clock from restore
request to the app serving the correct discriminator.

rallly's postgres volume grown 66 MB -> 21.1 GB (200k rows, STORAGE EXTERNAL so
TOAST cannot compress it into a fake number). Two reps:

  rep 1  backup 406.4s  unit 41149 MB  RTO 624.5s  discriminator correct
  rep 2  backup 387.2s  unit 41133 MB  RTO 591.8s  discriminator correct

327x the data cost 14.5x the time - strongly sub-linear:
  RTO ~= 40s + 26.9 s/GB      backup ~= 29s + 17.4 s/GB
  10 GB -> 5.2 min   20 GB -> 9.6 min (measured 10.1)   100 GB -> 46 min
The fixed ~40s dominates below ~1.5 GB, which IS the S band and explains its tight
clustering.

The more consequential result is capacity. A DB-backed app's recovery unit is
1.90x its data (volume tar PLUS SQL dump): 21.1 GB produced a 40.2 GB unit. The
default appliance ships /mnt/sys_drive at 20 GB, so the largest app that can hold
a local Tier-1/2 recovery unit on a default box is about 10 GB - and that fills the
volume. The M band does not fit on a default box at all; this test only reached
21 GB because sys_drive was first grown 20G -> 70G with the same operation the
product performs via SysDataGrowGB. A tier-sizing decision, not a defect, but it
is invisible until an app crosses it.

Caveats stated in the doc: two points define a line but do not test linearity; the
1.90x is DB-app-specific and a file-only app should be nearer 1.0x (inferred, not
measured); synthetic incompressible data; one app, one box.

2026-08-02 08:31:51 +02:00

8.0 KiB

Raw Blame History

REPORT — Campaign 10, two-storage adversarial soak (2026-07-31)

Follows REPORT-campaign7/8/9.md. Root REPORT.md is another session's (hub v0.85.0) and was not clobbered — same shared-clone reasoning as REPORT-iso-release.md.

Full audit + evidence: documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md, documentation/tests/campaign10-evidence-2026-07-31/.

The sentence that matters

Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the "drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No resource leak over 13.5 h. Two findings: R-156 and R-157.

What was established

Venue — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the /mnt/nvme-1tb mount root (a subdirectory would have emitted storage_disconnected for demo-hp all night — the exact signal I1/I2 discriminate).
Baselines, all read fresh. controller main 0.188.0, golden 0.188.0 (not behind), agent 0.119.0 published+vouched, hub 0.86.0, ISO 1.26.1 (f3cc86d5…, round-trip verified live). The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129.
Isolation gate — both denials captured, each with a positive control. The PBS control failed first: four clean-looking 403s were worthless because the token was denied on its own datastore too (PBS token privilege separation). Fixed, re-run, denials stand.
A1 fresh install from the published ISO. 1.26.1 is a public release image — verified against its bytes that it has no auto-install path — so it was driven blind via screendump + sendkey through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which would otherwise have been mangled and locked the box out.
A2 claimed for real; discriminator flipped dashboard not yet claimed → authentication required.
A3 both drives enrolled through the real endpoint; mentes accepted as backup target via the offer flow, ending degraded:false / target:felhom-backup — the I5/I6 healthy baseline. Four apps healthy spanning both sides of D5's split (4 × type: secret, 1 × type: password).
A4 discriminators seed and read back across all four apps; rallly's over the path DATABASE_URL actually names, not the trusted socket that produced D5's false pass.

Phase B — the soak

Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, stopped deliberately — two violations were harness defects), run 2b (39 cycles, 12 atom families). 1 461 invariant checks in total.

I7 is the headline: 66 restores across both passes, 66 correct discriminators. Never stale, never empty. Run 2b added an I7-SKIP verdict so a check with unmet preconditions is recorded as skipped rather than silently green — it fired once.
I2, I3, I4, I5, I6, I10, I11: zero violations in either pass, including the abort-in-place variants and 330 secret-class assertions covering both sides of D5's split.
I1-under-load 5/5: the backup target pulled while a backup was running still produced backup_target_absent and a clean recovery.
R-117's Q7 case holds — a filesystem aborted in place (device still present) surfaces via bound_under_parent=false, the gate stops the app on the dead namespace, and the storage page names it. That is the case R-117's spike called "the worse half".
RTO, both bands measured. S: 66 MB → 42.0 s / 41.4 s across two passes (66 restores). M: 21.1 GB → 608 s mean over 2 reps, both returning the correct discriminator. 327× the data cost 14.5× the time, giving RTO ≈ 40 s + 26.9 s/GB (backup ≈ 29 s + 17.4 s/GB). The fixed ~40 s dominates below ~1.5 GB — that is the S band, and why its numbers clustered so tightly.
Capacity ceiling, and the more consequential result: a DB-backed app's recovery unit is 1.90× its data (volume tar + SQL dump). The default /mnt/sys_drive is 20 GB, so on a default box the largest locally-backupable app is ≈ 10 GB — the M band does not fit at all without a per-customer SysDataGrowGB. RPO still not measured.
No resource leak. 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds flat, and no orphaned volumes, images or containers despite dozens of redeploys, kills, reboots and hard resets. The only curve with real slope is the agent journal, 194 → 463 MB (~20 MB/h) — bounded by journald, but a lot of logging.
Every atom and invariant was proven by hand before automation; the runner asserts nothing that was not first observed live.
A Phase A gap was caught before the run: no app had HDD_PATH, so all data sat on the system disk and I3 could never have fired. calibre-web was deployed onto adatok first — otherwise the soak would have produced green cycles that tested nothing cross-drive.
Two violations were my harness, not the product, and run 2a was stopped for them: a seed that never landed became a fake "stale restore", and a real one would have looked identical. Fixed and red-proofed before restarting.

Findings

R-156 (new) — papra's data is neither persisted nor backed up, and it reports healthy. The template mounts papra_data:/app/data; the app writes /app/app-data/db/db.sqlite. The volume is empty and root-owned (the image is -rootless, so the app cannot even write there), the real DB sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
R-157 (new) — bootrecon's start-once sweep misses the boot orphan it exists to recover. Two mechanisms. A: the container is left Exited, the sweep runs ~5 s after controller start while docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences, intermittent ~50%). B: the interruption leaves the stack with zero containers, which is exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp check reported 0 currently down while a deployed: true app was not running, i.e. silent on every channel. A settle-condition fix closes A and leaves B open. Not fixed.
Tier 3 could not be isolated, so it was not run. Offsite hard-requires the DR tier (configs.go:1300), and the DR tier only provisions on ep0 (per-endpoint allocation deferred, hub/README.md:260). Both are recorded deliberate positions, so no R-n minted. The campaign therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows.

What did not run

12 of the brief's ~13 atom families ran (run 1 covered 6; run 2b added abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive). Previously reported as 6 of 12 — that was run 1 only.

Superseded detail: Still not run: Tier-3 backup/restore (§3, structurally un-isolatable) and I8. I9 was not automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed. kill_controller is still not literally "mid-backup"; the dedicated concurrent backup+detach atom covers that case properly. The run-1 flaw where reboot never interleaved with a detach was fixed.

Depth reached: 39 consecutive cycles, past the brief's "thirty-eighth", with c34–c39 clean on every invariant. Beyond 39 is untested, not proven clean.

Teardown — OWED, nothing removed

Still intact: the rig is reusable for the atoms that did not run. VM 311, c10-scratch, PBS datastore felhom-c10 + user/token, restic subaccount u629488-sub4, and hub customer c10-soak (disposition: DELETE) are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four orphaned scratch customers left by exactly this omission.

8.0 KiB Raw Blame History Unescape Escape