felhom.eu/REPORT-campaign10.md

REPORT — Campaign 10, two-storage adversarial soak (2026-07-31)

Follows REPORT-campaign7/8/9.md. Root REPORT.md is another session's (hub v0.85.0) and was not clobbered — same shared-clone reasoning as REPORT-iso-release.md.

Full audit + evidence: documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md, documentation/tests/campaign10-evidence-2026-07-31/.

The sentence that matters

Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the "drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No resource leak over 13.5 h. Two findings: R-156 and R-157.

What was established

• Venue — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the /mnt/nvme-1tb mount root (a subdirectory would have emitted storage_disconnected for demo-hp all night — the exact signal I1/I2 discriminate).
• Baselines, all read fresh. controller main 0.188.0, golden 0.188.0 (not behind), agent 0.119.0 published+vouched, hub 0.86.0, ISO 1.26.1 (f3cc86d5…, round-trip verified live). The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129.
• Isolation gate — both denials captured, each with a positive control. The PBS control failed first: four clean-looking 403s were worthless because the token was denied on its own datastore too (PBS token privilege separation). Fixed, re-run, denials stand.
• A1 fresh install from the published ISO. 1.26.1 is a public release image — verified against its bytes that it has no auto-install path — so it was driven blind via screendump + sendkey through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which would otherwise have been mangled and locked the box out.
• A2 claimed for real; discriminator flipped dashboard not yet claimed → authentication required.
• A3 both drives enrolled through the real endpoint; mentes accepted as backup target via the offer flow, ending degraded:false / target:felhom-backup — the I5/I6 healthy baseline. Four apps healthy spanning both sides of D5's split (4 × type: secret, 1 × type: password).
• A4 discriminators seed and read back across all four apps; rallly's over the path DATABASE_URL actually names, not the trusted socket that produced D5's false pass.

Phase B — the soak

Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, stopped deliberately — two violations were harness defects), run 2b (39 cycles, 12 atom families). 1 461 invariant checks in total.

• I7 is the headline: 66 restores across both passes, 66 correct discriminators. Never stale, never empty. Run 2b added an I7-SKIP verdict so a check with unmet preconditions is recorded as skipped rather than silently green — it fired once.
• I2, I3, I4, I5, I6, I10, I11: zero violations in either pass, including the abort-in-place variants and 330 secret-class assertions covering both sides of D5's split.
• I1-under-load 5/5: the backup target pulled while a backup was running still produced backup_target_absent and a clean recovery.
• R-117's Q7 case holds — a filesystem aborted in place (device still present) surfaces via bound_under_parent=false, the gate stops the app on the dead namespace, and the storage page names it. That is the case R-117's spike called "the worse half".
• RTO (Tier 1, rallly, 66 MB): run 1 median 42.0 s, run 2b median 41.4 s over 38 restores — two independent passes agreeing to 0.6 s. S band's lower end only; nothing extrapolates to M or L. RPO not measured.
• No resource leak. 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds flat, and no orphaned volumes, images or containers despite dozens of redeploys, kills, reboots and hard resets. The only curve with real slope is the agent journal, 194 → 463 MB (~20 MB/h) — bounded by journald, but a lot of logging.
• Every atom and invariant was proven by hand before automation; the runner asserts nothing that was not first observed live.
• A Phase A gap was caught before the run: no app had HDD_PATH, so all data sat on the system disk and I3 could never have fired. calibre-web was deployed onto adatok first — otherwise the soak would have produced green cycles that tested nothing cross-drive.
• Two violations were my harness, not the product, and run 2a was stopped for them: a seed that never landed became a fake "stale restore", and a real one would have looked identical. Fixed and red-proofed before restarting.

Findings

• R-156 (new) — papra's data is neither persisted nor backed up, and it reports healthy. The template mounts papra_data:/app/data; the app writes /app/app-data/db/db.sqlite. The volume is empty and root-owned (the image is -rootless, so the app cannot even write there), the real DB sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
• R-157 (new) — bootrecon's start-once sweep misses the boot orphan it exists to recover. Two mechanisms. A: the container is left Exited, the sweep runs ~5 s after controller start while docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences, intermittent ~50%). B: the interruption leaves the stack with zero containers, which is exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp check reported 0 currently down while a deployed: true app was not running, i.e. silent on every channel. A settle-condition fix closes A and leaves B open. Not fixed.
• Tier 3 could not be isolated, so it was not run. Offsite hard-requires the DR tier (configs.go:1300), and the DR tier only provisions on ep0 (per-endpoint allocation deferred, hub/README.md:260). Both are recorded deliberate positions, so no R-n minted. The campaign therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows.

What did not run

12 of the brief's ~13 atom families ran (run 1 covered 6; run 2b added abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive). Previously reported as 6 of 12 — that was run 1 only.

Superseded detail: Still not run: Tier-3 backup/restore (§3, structurally un-isolatable) and I8. I9 was not automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed. kill_controller is still not literally "mid-backup"; the dedicated concurrent backup+detach atom covers that case properly. The run-1 flaw where reboot never interleaved with a detach was fixed.

Depth reached: 39 consecutive cycles, past the brief's "thirty-eighth", with c34–c39 clean on every invariant. Beyond 39 is untested, not proven clean.

Teardown — OWED, nothing removed

Still intact: the rig is reusable for the atoms that did not run. VM 311, c10-scratch, PBS datastore felhom-c10 + user/token, restic subaccount u629488-sub4, and hub customer c10-soak (disposition: DELETE) are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four orphaned scratch customers left by exactly this omission.