Files

T

admin 69f896d3cd Campaign 10 Phase B: 27 cycles, 586 invariant checks, 0 violations

Ran the soak on the Phase A rig. Ended on its own deadline — no watchdog halt,
no atom exception, no I11 breach.

I1 28+28 pairs, I2 28+28 pairs, I3 56, I4 56, I5/I6 28 each, I7 28, I10 135,
I11 28. Zero violations. The row counts are themselves the no-silent-skip check:
I3/I4 twice per cycle (both drives), I10 = 5 secret-class fields x 27, REBOOT on
cycles 7/14/21 only.

I7 is the headline: 28 restores, 28 correct discriminators — never stale, never
empty. RTO (Tier 1, rallly, 66 MB): min 38.8s, median 42.0s, p90 42.5s, max
44.3s. That is the S band's lower end ONLY; the 5.5s spread over 28 runs says
fixed work dominates, so nothing extrapolates to M or L. RPO not measured.

Every atom and invariant was proven BY HAND before automation — the runner
asserts nothing that was not first observed live.

Caught a Phase A gap before starting: no app had HDD_PATH, so all data sat on the
system disk and I3 could never have fired. Deployed calibre-web onto adatok
first; otherwise the run would have produced 27 green cycles that tested nothing
cross-drive.

Investigated and DISPROVED a suspected defect (audit 5.2): /api/disks reports
state=attached for a physically absent drive, and intermediary.go:230 really does
compute presence from State=="attached". It is inert — planDriveGates only gates
paths under /mnt/felhom-drives/ and uses BoundUnderParent there, which was
correctly false. The gate fired; the storage page showed "Meghajtó leválasztva".
No R-n minted.

Honest gaps: 6 of ~12 atom families ran. Not run — Tier 3 (structurally
un-isolatable), abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write,
reboot-VM, both concurrency atoms, fill-drive-near-full. I8 not checked, I9 not
automated (cited from the tester-gate run, not re-claimed). kill_controller is
NOT mid-backup and reboot_guest never interleaved with a detach. 27 cycles does
not answer the brief's question about drift at the thirty-eighth.

Teardown still OWED, including hub customer c10-soak (disposition: DELETE).

2026-08-01 15:45:13 +02:00

5.7 KiB

Raw Blame History

REPORT — Campaign 10, two-storage adversarial soak (2026-07-31)

Follows REPORT-campaign7/8/9.md. Root REPORT.md is another session's (hub v0.85.0) and was not clobbered — same shared-clone reasoning as REPORT-iso-release.md.

Full audit + evidence: documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md, documentation/tests/campaign10-evidence-2026-07-31/.

The sentence that matters

Phase A passed every gate (07-31) and Phase B ran (08-01): 27 cycles, 586 invariant checks, zero violations, ended on its own deadline with no watchdog halt and no atom exception.

What was established

Venue — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the /mnt/nvme-1tb mount root (a subdirectory would have emitted storage_disconnected for demo-hp all night — the exact signal I1/I2 discriminate).
Baselines, all read fresh. controller main 0.188.0, golden 0.188.0 (not behind), agent 0.119.0 published+vouched, hub 0.86.0, ISO 1.26.1 (f3cc86d5…, round-trip verified live). The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129.
Isolation gate — both denials captured, each with a positive control. The PBS control failed first: four clean-looking 403s were worthless because the token was denied on its own datastore too (PBS token privilege separation). Fixed, re-run, denials stand.
A1 fresh install from the published ISO. 1.26.1 is a public release image — verified against its bytes that it has no auto-install path — so it was driven blind via screendump + sendkey through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which would otherwise have been mangled and locked the box out.
A2 claimed for real; discriminator flipped dashboard not yet claimed → authentication required.
A3 both drives enrolled through the real endpoint; mentes accepted as backup target via the offer flow, ending degraded:false / target:felhom-backup — the I5/I6 healthy baseline. Four apps healthy spanning both sides of D5's split (4 × type: secret, 1 × type: password).
A4 discriminators seed and read back across all four apps; rallly's over the path DATABASE_URL actually names, not the trusted socket that produced D5's false pass.

Phase B — the soak

27 cycles, 09:33–15:40 CEST, atoms permuted per cycle, every invariant checked every cycle.

I1 28 + 28 pairs, I2 28 + 28 pairs, I3 56, I4 56, I5/I6 28 each, I7 28, I10 135, I11 28 — 0 violations. Row counts are themselves the no-silent-skip check (I3/I4 twice per cycle = both drives; I10 = 5 secret-class fields × 27; REBOOT = cycles 7/14/21).
I7 is the headline: 28 restores, 28 correct discriminators. Never stale, never empty.
RTO (Tier 1, rallly, 66 MB): min 38.8 s, median 42.0 s, p90 42.5 s, max 44.3 s. Covers the S band only, and only its lower end — the 5.5 s spread over 28 runs says fixed work dominates, so nothing here extrapolates to M or L. RPO not measured (backups were driven on demand).
Every atom and invariant was proven by hand before automation; the runner asserts nothing that was not first observed live.
A Phase A gap was caught before the run: no app had HDD_PATH, so all data sat on the system disk and I3 could never have fired. calibre-web was deployed onto adatok first — otherwise the soak would have produced 27 green cycles that tested nothing cross-drive.

Findings

R-156 (new) — papra's data is neither persisted nor backed up, and it reports healthy. The template mounts papra_data:/app/data; the app writes /app/app-data/db/db.sqlite. The volume is empty and root-owned (the image is -rootless, so the app cannot even write there), the real DB sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
Tier 3 could not be isolated, so it was not run. Offsite hard-requires the DR tier (configs.go:1300), and the DR tier only provisions on ep0 (per-endpoint allocation deferred, hub/README.md:260). Both are recorded deliberate positions, so no R-n minted. The campaign therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows.

What did not run

6 of the brief's ~12 atom families ran. Not run: Tier-3 backup/restore (§3), abort-a-filesystem- in-place (R-117 Q7), kill-the-agent-mid-backup, hard-reset-the-VM-mid-write, reboot-the-VM, both concurrency atoms, and fill-a-drive-near-full. I8 not checked (no Tier 3); I9 not automated — it was proven on this same controller 0.188.0 by the tester-gate run hours earlier, which is cited, not re-claimed.

Two precision notes so atom names don't overclaim: kill_controller is not "mid-backup" (it lands at a permuted point, exercising recovery rather than the mid-write race), and reboot_guest was appended after the shuffle so it never interleaved with a detach.

The tail is untested, not clean. The brief's interest in "the state that quietly drifts on the thirty-eighth" is not answered by 27 cycles.

Teardown — OWED, nothing removed

Still intact: the rig is reusable for the atoms that did not run. VM 311, c10-scratch, PBS datastore felhom-c10 + user/token, restic subaccount u629488-sub4, and hub customer c10-soak (disposition: DELETE) are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four orphaned scratch customers left by exactly this omission.

5.7 KiB Raw Blame History Unescape Escape