admin
5f35aa0346
The S figures (66 MB -> 42.0s, two passes agreeing to 0.6s) had a spread tight enough to prove fixed work dominates, which is exactly why they said nothing about M. Second point taken 327x larger, same app, same method: clock from restore request to the app serving the correct discriminator. rallly's postgres volume grown 66 MB -> 21.1 GB (200k rows, STORAGE EXTERNAL so TOAST cannot compress it into a fake number). Two reps: rep 1 backup 406.4s unit 41149 MB RTO 624.5s discriminator correct rep 2 backup 387.2s unit 41133 MB RTO 591.8s discriminator correct 327x the data cost 14.5x the time - strongly sub-linear: RTO ~= 40s + 26.9 s/GB backup ~= 29s + 17.4 s/GB 10 GB -> 5.2 min 20 GB -> 9.6 min (measured 10.1) 100 GB -> 46 min The fixed ~40s dominates below ~1.5 GB, which IS the S band and explains its tight clustering. The more consequential result is capacity. A DB-backed app's recovery unit is 1.90x its data (volume tar PLUS SQL dump): 21.1 GB produced a 40.2 GB unit. The default appliance ships /mnt/sys_drive at 20 GB, so the largest app that can hold a local Tier-1/2 recovery unit on a default box is about 10 GB - and that fills the volume. The M band does not fit on a default box at all; this test only reached 21 GB because sys_drive was first grown 20G -> 70G with the same operation the product performs via SysDataGrowGB. A tier-sizing decision, not a defect, but it is invisible until an app crosses it. Caveats stated in the doc: two points define a line but do not test linearity; the 1.90x is DB-app-specific and a file-only app should be nearer 1.0x (inferred, not measured); synthetic incompressible data; one app, one box.
114 lines
8.0 KiB
Markdown
114 lines
8.0 KiB
Markdown
# REPORT — Campaign 10, two-storage adversarial soak (2026-07-31)
|
||
|
||
Follows `REPORT-campaign7/8/9.md`. Root `REPORT.md` is another session's (hub v0.85.0) and was not
|
||
clobbered — same shared-clone reasoning as `REPORT-iso-release.md`.
|
||
|
||
**Full audit + evidence:** `documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`,
|
||
`documentation/tests/campaign10-evidence-2026-07-31/`.
|
||
|
||
## The sentence that matters
|
||
|
||
**Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the
|
||
"drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No
|
||
resource leak over 13.5 h. Two findings: R-156 and R-157.**
|
||
|
||
## What was established
|
||
|
||
- **Venue** — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the
|
||
`/mnt/nvme-1tb` **mount root** (a subdirectory would have emitted `storage_disconnected` for demo-hp
|
||
all night — the exact signal I1/I2 discriminate).
|
||
- **Baselines, all read fresh.** controller `main` 0.188.0, **golden 0.188.0 (not behind)**, agent
|
||
0.119.0 published+vouched, hub 0.86.0, ISO **1.26.1** (`f3cc86d5…`, round-trip verified live).
|
||
The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129.
|
||
- **Isolation gate — both denials captured, each with a positive control.** The PBS control **failed
|
||
first**: four clean-looking 403s were worthless because the token was denied on its own datastore
|
||
too (PBS token privilege separation). Fixed, re-run, denials stand.
|
||
- **A1** fresh install from the **published** ISO. 1.26.1 is a public release image — verified against
|
||
its bytes that it has no auto-install path — so it was driven blind via screendump + `sendkey`
|
||
through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which
|
||
would otherwise have been mangled and locked the box out.
|
||
- **A2** claimed for real; discriminator flipped `dashboard not yet claimed` → `authentication required`.
|
||
- **A3** both drives enrolled through the **real** endpoint; `mentes` accepted as backup target via
|
||
the offer flow, ending `degraded:false / target:felhom-backup` — the I5/I6 healthy baseline.
|
||
Four apps healthy spanning both sides of D5's split (4 × `type: secret`, 1 × `type: password`).
|
||
- **A4** discriminators seed and read back across all four apps; rallly's over the path
|
||
`DATABASE_URL` actually names, not the trusted socket that produced D5's false pass.
|
||
|
||
## Phase B — the soak
|
||
|
||
Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, **stopped
|
||
deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 atom families**).
|
||
1 461 invariant checks in total.
|
||
|
||
- **I7 is the headline: 66 restores across both passes, 66 correct discriminators.** Never stale,
|
||
never empty. Run 2b added an `I7-SKIP` verdict so a check with unmet preconditions is recorded as
|
||
skipped rather than silently green — it fired once.
|
||
- **I2, I3, I4, I5, I6, I10, I11: zero violations in either pass**, including the abort-in-place
|
||
variants and 330 secret-class assertions covering both sides of D5's split.
|
||
- **`I1-under-load` 5/5**: the backup target pulled **while a backup was running** still produced
|
||
`backup_target_absent` and a clean recovery.
|
||
- **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via
|
||
`bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names
|
||
it. That is the case R-117's spike called "the worse half".
|
||
- **RTO, both bands measured.** S: 66 MB → **42.0 s** / **41.4 s** across two passes (66 restores).
|
||
M: 21.1 GB → **608 s** mean over 2 reps, both returning the correct discriminator. 327× the data
|
||
cost 14.5× the time, giving **RTO ≈ 40 s + 26.9 s/GB** (backup ≈ 29 s + 17.4 s/GB). The fixed ~40 s
|
||
dominates below ~1.5 GB — that is the S band, and why its numbers clustered so tightly.
|
||
- **Capacity ceiling, and the more consequential result:** a DB-backed app's recovery unit is **1.90×**
|
||
its data (volume tar + SQL dump). The default `/mnt/sys_drive` is **20 GB**, so on a default box the
|
||
largest locally-backupable app is **≈ 10 GB** — **the M band does not fit at all** without a
|
||
per-customer `SysDataGrowGB`. **RPO still not measured.**
|
||
- **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds
|
||
flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots
|
||
and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)** —
|
||
bounded by journald, but a lot of logging.
|
||
- Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that
|
||
was not first observed live.
|
||
- **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system
|
||
disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the
|
||
soak would have produced green cycles that tested nothing cross-drive.
|
||
- **Two violations were my harness, not the product**, and run 2a was stopped for them: a seed that
|
||
never landed became a fake "stale restore", and a real one would have looked identical. Fixed and
|
||
red-proofed before restarting.
|
||
|
||
## Findings
|
||
|
||
- **R-156 (new)** — **papra's data is neither persisted nor backed up, and it reports healthy.** The
|
||
template mounts `papra_data:/app/data`; the app writes `/app/app-data/db/db.sqlite`. The volume is
|
||
empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB
|
||
sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its
|
||
Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
|
||
- **R-157 (new)** — **bootrecon's start-once sweep misses the boot orphan it exists to recover.** Two
|
||
mechanisms. **A:** the container is left `Exited`, the sweep runs ~5 s after controller start while
|
||
docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences,
|
||
intermittent ~50%). **B:** the interruption leaves the stack with **zero containers**, which is
|
||
exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp
|
||
check reported **`0 currently down`** while a `deployed: true` app was not running, i.e. silent on
|
||
every channel. A settle-condition fix closes A and leaves B open. Not fixed.
|
||
- **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier
|
||
(`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred,
|
||
`hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign
|
||
therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by
|
||
not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows.
|
||
|
||
## What did not run
|
||
|
||
**12 of the brief's ~13 atom families ran** (run 1 covered 6; run 2b added abort-fs-in-place,
|
||
kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive).
|
||
Previously reported as 6 of 12 — that was run 1 only.
|
||
|
||
**Superseded detail:** Still not run: **Tier-3 backup/restore** (§3, structurally un-isolatable) and **I8**. **I9** was not
|
||
automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed.
|
||
`kill_controller` is still not literally "mid-backup"; the dedicated concurrent backup+detach atom
|
||
covers that case properly. The run-1 flaw where `reboot` never interleaved with a detach was fixed.
|
||
|
||
**Depth reached: 39 consecutive cycles**, past the brief's "thirty-eighth", with c34–c39 clean on
|
||
every invariant. Beyond 39 is untested, not proven clean.
|
||
|
||
## Teardown — OWED, nothing removed
|
||
|
||
Still intact: the rig is reusable for the atoms that did not run. VM 311, `c10-scratch`, PBS datastore
|
||
`felhom-c10` + user/token, restic subaccount `u629488-sub4`, and **hub customer `c10-soak` (disposition:
|
||
DELETE)** are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four
|
||
orphaned scratch customers left by exactly this omission.
|