admin
69f896d3cd
Ran the soak on the Phase A rig. Ended on its own deadline — no watchdog halt, no atom exception, no I11 breach. I1 28+28 pairs, I2 28+28 pairs, I3 56, I4 56, I5/I6 28 each, I7 28, I10 135, I11 28. Zero violations. The row counts are themselves the no-silent-skip check: I3/I4 twice per cycle (both drives), I10 = 5 secret-class fields x 27, REBOOT on cycles 7/14/21 only. I7 is the headline: 28 restores, 28 correct discriminators — never stale, never empty. RTO (Tier 1, rallly, 66 MB): min 38.8s, median 42.0s, p90 42.5s, max 44.3s. That is the S band's lower end ONLY; the 5.5s spread over 28 runs says fixed work dominates, so nothing extrapolates to M or L. RPO not measured. Every atom and invariant was proven BY HAND before automation — the runner asserts nothing that was not first observed live. Caught a Phase A gap before starting: no app had HDD_PATH, so all data sat on the system disk and I3 could never have fired. Deployed calibre-web onto adatok first; otherwise the run would have produced 27 green cycles that tested nothing cross-drive. Investigated and DISPROVED a suspected defect (audit 5.2): /api/disks reports state=attached for a physically absent drive, and intermediary.go:230 really does compute presence from State=="attached". It is inert — planDriveGates only gates paths under /mnt/felhom-drives/ and uses BoundUnderParent there, which was correctly false. The gate fired; the storage page showed "Meghajtó leválasztva". No R-n minted. Honest gaps: 6 of ~12 atom families ran. Not run — Tier 3 (structurally un-isolatable), abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms, fill-drive-near-full. I8 not checked, I9 not automated (cited from the tester-gate run, not re-claimed). kill_controller is NOT mid-backup and reboot_guest never interleaved with a detach. 27 cycles does not answer the brief's question about drift at the thirty-eighth. Teardown still OWED, including hub customer c10-soak (disposition: DELETE).
87 lines
5.7 KiB
Markdown
87 lines
5.7 KiB
Markdown
# REPORT — Campaign 10, two-storage adversarial soak (2026-07-31)
|
||
|
||
Follows `REPORT-campaign7/8/9.md`. Root `REPORT.md` is another session's (hub v0.85.0) and was not
|
||
clobbered — same shared-clone reasoning as `REPORT-iso-release.md`.
|
||
|
||
**Full audit + evidence:** `documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`,
|
||
`documentation/tests/campaign10-evidence-2026-07-31/`.
|
||
|
||
## The sentence that matters
|
||
|
||
**Phase A passed every gate (07-31) and Phase B ran (08-01): 27 cycles, 586 invariant checks, zero
|
||
violations, ended on its own deadline with no watchdog halt and no atom exception.**
|
||
|
||
## What was established
|
||
|
||
- **Venue** — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the
|
||
`/mnt/nvme-1tb` **mount root** (a subdirectory would have emitted `storage_disconnected` for demo-hp
|
||
all night — the exact signal I1/I2 discriminate).
|
||
- **Baselines, all read fresh.** controller `main` 0.188.0, **golden 0.188.0 (not behind)**, agent
|
||
0.119.0 published+vouched, hub 0.86.0, ISO **1.26.1** (`f3cc86d5…`, round-trip verified live).
|
||
The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129.
|
||
- **Isolation gate — both denials captured, each with a positive control.** The PBS control **failed
|
||
first**: four clean-looking 403s were worthless because the token was denied on its own datastore
|
||
too (PBS token privilege separation). Fixed, re-run, denials stand.
|
||
- **A1** fresh install from the **published** ISO. 1.26.1 is a public release image — verified against
|
||
its bytes that it has no auto-install path — so it was driven blind via screendump + `sendkey`
|
||
through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which
|
||
would otherwise have been mangled and locked the box out.
|
||
- **A2** claimed for real; discriminator flipped `dashboard not yet claimed` → `authentication required`.
|
||
- **A3** both drives enrolled through the **real** endpoint; `mentes` accepted as backup target via
|
||
the offer flow, ending `degraded:false / target:felhom-backup` — the I5/I6 healthy baseline.
|
||
Four apps healthy spanning both sides of D5's split (4 × `type: secret`, 1 × `type: password`).
|
||
- **A4** discriminators seed and read back across all four apps; rallly's over the path
|
||
`DATABASE_URL` actually names, not the trusted socket that produced D5's false pass.
|
||
|
||
## Phase B — the soak
|
||
|
||
27 cycles, 09:33–15:40 CEST, atoms permuted per cycle, every invariant checked every cycle.
|
||
|
||
- **I1** 28 + 28 pairs, **I2** 28 + 28 pairs, **I3** 56, **I4** 56, **I5/I6** 28 each, **I7** 28,
|
||
**I10** 135, **I11** 28 — **0 violations**. Row counts are themselves the no-silent-skip check
|
||
(I3/I4 twice per cycle = both drives; I10 = 5 secret-class fields × 27; REBOOT = cycles 7/14/21).
|
||
- **I7 is the headline: 28 restores, 28 correct discriminators.** Never stale, never empty.
|
||
- **RTO (Tier 1, rallly, 66 MB):** min 38.8 s, **median 42.0 s**, p90 42.5 s, max 44.3 s. **Covers the
|
||
S band only, and only its lower end** — the 5.5 s spread over 28 runs says fixed work dominates, so
|
||
nothing here extrapolates to M or L. **RPO not measured** (backups were driven on demand).
|
||
- Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that
|
||
was not first observed live.
|
||
- **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system
|
||
disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the
|
||
soak would have produced 27 green cycles that tested nothing cross-drive.
|
||
|
||
## Findings
|
||
|
||
- **R-156 (new)** — **papra's data is neither persisted nor backed up, and it reports healthy.** The
|
||
template mounts `papra_data:/app/data`; the app writes `/app/app-data/db/db.sqlite`. The volume is
|
||
empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB
|
||
sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its
|
||
Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
|
||
- **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier
|
||
(`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred,
|
||
`hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign
|
||
therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by
|
||
not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows.
|
||
|
||
## What did not run
|
||
|
||
**6 of the brief's ~12 atom families ran.** Not run: Tier-3 backup/restore (§3), abort-a-filesystem-
|
||
in-place (R-117 Q7), kill-the-agent-mid-backup, hard-reset-the-VM-mid-write, reboot-the-VM, both
|
||
concurrency atoms, and fill-a-drive-near-full. **I8** not checked (no Tier 3); **I9** not automated —
|
||
it was proven on this same controller 0.188.0 by the tester-gate run hours earlier, which is cited,
|
||
not re-claimed.
|
||
|
||
Two precision notes so atom names don't overclaim: `kill_controller` is **not** "mid-backup" (it lands
|
||
at a permuted point, exercising recovery rather than the mid-write race), and `reboot_guest` was
|
||
appended after the shuffle so it **never interleaved with a detach**.
|
||
|
||
**The tail is untested, not clean.** The brief's interest in "the state that quietly drifts on the
|
||
thirty-eighth" is not answered by 27 cycles.
|
||
|
||
## Teardown — OWED, nothing removed
|
||
|
||
Still intact: the rig is reusable for the atoms that did not run. VM 311, `c10-scratch`, PBS datastore
|
||
`felhom-c10` + user/token, restic subaccount `u629488-sub4`, and **hub customer `c10-soak` (disposition:
|
||
DELETE)** are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four
|
||
orphaned scratch customers left by exactly this omission.
|