From 7ba7c2a27186a2fdb8c559dfa048865748964fd8 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 2 Aug 2026 07:13:04 +0200 Subject: [PATCH] =?UTF-8?q?Campaign=2010:=20final=20results=20=E2=80=94=20?= =?UTF-8?q?39=20cycles,=20full=20atom=20set,=20R-156=20+=20R-157,=20no=20l?= =?UTF-8?q?eaks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase B completed in three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, stopped deliberately - two violations were harness defects), run 2b (39 cycles, 12 of the brief's ~13 atom families). 1461 invariant checks. Depth reached 39 consecutive cycles, past the brief's "drift at the thirty-eighth", with c34-c39 clean on every invariant. I7 headline: 66 restores across both passes, 66 correct discriminators - never stale, never empty. I2/I3/I4/I5/I6/I10/I11 zero violations in either pass. I1-under-load 5/5: the target pulled WHILE a backup ran still produced backup_target_absent and a clean recovery. R-117's Q7 case holds - a filesystem aborted in place surfaces and the gate stops the app on the dead namespace. RTO Tier-1 rallly 66MB: run 1 median 42.0s, run 2b median 41.4s over 38 restores - two independent passes agreeing to 0.6s. S band's lower end only; nothing extrapolates to M or L. RPO not measured. Monotonic growth, 9457 samples of 19 metrics over 13.5h: NO leak. Controller and agent RSS flat, fds flat, no orphaned volumes/images/containers despite dozens of redeploys, kills, reboots and hard resets. Only curve with real slope is the agent journal at ~20MB/h, bounded by journald. Findings: R-156 (papra's data neither persisted nor backed up, reports healthy) and R-157 (bootrecon's start-once sweep, two mechanisms - the zero-container one is silent on every channel). Four suspicions investigated and DISPROVED, each recorded with what settled it. --- REPORT-campaign10.md | 64 +++-- ...CAMPAIGN-10-two-storage-soak-2026-07-31.md | 253 +++++++++++------- 2 files changed, 203 insertions(+), 114 deletions(-) diff --git a/REPORT-campaign10.md b/REPORT-campaign10.md index a5ab2e5..5478ba3 100644 --- a/REPORT-campaign10.md +++ b/REPORT-campaign10.md @@ -8,8 +8,9 @@ clobbered — same shared-clone reasoning as `REPORT-iso-release.md`. ## The sentence that matters -**Phase A passed every gate (07-31) and Phase B ran (08-01): 27 cycles, 586 invariant checks, zero -violations, ended on its own deadline with no watchdog halt and no atom exception.** +**Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the +"drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No +resource leak over 13.5 h. Two findings: R-156 and R-157.** ## What was established @@ -35,20 +36,35 @@ violations, ended on its own deadline with no watchdog halt and no atom exceptio ## Phase B — the soak -27 cycles, 09:33–15:40 CEST, atoms permuted per cycle, every invariant checked every cycle. +Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, **stopped +deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 atom families**). +1 461 invariant checks in total. -- **I1** 28 + 28 pairs, **I2** 28 + 28 pairs, **I3** 56, **I4** 56, **I5/I6** 28 each, **I7** 28, - **I10** 135, **I11** 28 — **0 violations**. Row counts are themselves the no-silent-skip check - (I3/I4 twice per cycle = both drives; I10 = 5 secret-class fields × 27; REBOOT = cycles 7/14/21). -- **I7 is the headline: 28 restores, 28 correct discriminators.** Never stale, never empty. -- **RTO (Tier 1, rallly, 66 MB):** min 38.8 s, **median 42.0 s**, p90 42.5 s, max 44.3 s. **Covers the - S band only, and only its lower end** — the 5.5 s spread over 28 runs says fixed work dominates, so - nothing here extrapolates to M or L. **RPO not measured** (backups were driven on demand). +- **I7 is the headline: 66 restores across both passes, 66 correct discriminators.** Never stale, + never empty. Run 2b added an `I7-SKIP` verdict so a check with unmet preconditions is recorded as + skipped rather than silently green — it fired once. +- **I2, I3, I4, I5, I6, I10, I11: zero violations in either pass**, including the abort-in-place + variants and 330 secret-class assertions covering both sides of D5's split. +- **`I1-under-load` 5/5**: the backup target pulled **while a backup was running** still produced + `backup_target_absent` and a clean recovery. +- **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via + `bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names + it. That is the case R-117's spike called "the worse half". +- **RTO (Tier 1, rallly, 66 MB):** run 1 median **42.0 s**, run 2b median **41.4 s** over 38 restores + — two independent passes agreeing to 0.6 s. **S band's lower end only**; nothing extrapolates to M + or L. **RPO not measured.** +- **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds + flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots + and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)** — + bounded by journald, but a lot of logging. - Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that was not first observed live. - **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the - soak would have produced 27 green cycles that tested nothing cross-drive. + soak would have produced green cycles that tested nothing cross-drive. +- **Two violations were my harness, not the product**, and run 2a was stopped for them: a seed that + never landed became a fake "stale restore", and a real one would have looked identical. Fixed and + red-proofed before restarting. ## Findings @@ -57,6 +73,13 @@ violations, ended on its own deadline with no watchdog halt and no atom exceptio empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed. +- **R-157 (new)** — **bootrecon's start-once sweep misses the boot orphan it exists to recover.** Two + mechanisms. **A:** the container is left `Exited`, the sweep runs ~5 s after controller start while + docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences, + intermittent ~50%). **B:** the interruption leaves the stack with **zero containers**, which is + exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp + check reported **`0 currently down`** while a `deployed: true` app was not running, i.e. silent on + every channel. A settle-condition fix closes A and leaves B open. Not fixed. - **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier (`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred, `hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign @@ -65,18 +88,17 @@ violations, ended on its own deadline with no watchdog halt and no atom exceptio ## What did not run -**6 of the brief's ~12 atom families ran.** Not run: Tier-3 backup/restore (§3), abort-a-filesystem- -in-place (R-117 Q7), kill-the-agent-mid-backup, hard-reset-the-VM-mid-write, reboot-the-VM, both -concurrency atoms, and fill-a-drive-near-full. **I8** not checked (no Tier 3); **I9** not automated — -it was proven on this same controller 0.188.0 by the tester-gate run hours earlier, which is cited, -not re-claimed. +**12 of the brief's ~13 atom families ran** (run 1 covered 6; run 2b added abort-fs-in-place, +kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive). +Previously reported as 6 of 12 — that was run 1 only. -Two precision notes so atom names don't overclaim: `kill_controller` is **not** "mid-backup" (it lands -at a permuted point, exercising recovery rather than the mid-write race), and `reboot_guest` was -appended after the shuffle so it **never interleaved with a detach**. +**Superseded detail:** Still not run: **Tier-3 backup/restore** (§3, structurally un-isolatable) and **I8**. **I9** was not +automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed. +`kill_controller` is still not literally "mid-backup"; the dedicated concurrent backup+detach atom +covers that case properly. The run-1 flaw where `reboot` never interleaved with a detach was fixed. -**The tail is untested, not clean.** The brief's interest in "the state that quietly drifts on the -thirty-eighth" is not answered by 27 cycles. +**Depth reached: 39 consecutive cycles**, past the brief's "thirty-eighth", with c34–c39 clean on +every invariant. Beyond 39 is untested, not proven clean. ## Teardown — OWED, nothing removed diff --git a/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md b/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md index d87c128..16bf074 100644 --- a/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md +++ b/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md @@ -1,8 +1,13 @@ # CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01) -**Status: Phase A COMPLETE and gated (2026-07-31). Phase B RUN 2026-08-01 — 27 cycles, 586 invariant -checks, ZERO violations.** **One new finding: R-156. One structural scope constraint: Tier 3 cannot be -isolated (§3). One suspected defect investigated and DISPROVED (§5.2).** +**Phase A COMPLETE and gated (2026-07-31). Phase B run in three passes (2026-08-01/02): run 1 +27 cycles, run 2a 10 cycles (stopped — harness defect), run 2b **39 cycles**. Together **66 clean +cycles + 39 with the full atom set**, 1 461 invariant checks.** + +**Findings: R-156 (papra), R-157 (bootrecon, two mechanisms).** One structural scope constraint — +Tier 3 cannot be isolated (§3). **Three suspicions investigated and DISPROVED** (§5.2, §5.3), and +**two violations traced to the harness, not the product** (§4d). Monotonic-growth sampling over +13.5 h found **no leak** (§6b). Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step), `isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`. @@ -189,59 +194,65 @@ false pass. --- -## 4b. Phase B — the soak (2026-08-01) +## 4b. Phase B — the soak -**27 cycles, 09:33–15:40 CEST, ended on its own deadline. No watchdog halt, no atom exception, no -I11 breach. 586 invariant rows, 0 violations.** +Three passes. Run 2a was stopped deliberately at cycle 10 when two violations turned out to be +**harness** defects (§4d); the harness was fixed and red-proofed, and run 2b restarted from cycle 1 +with the full atom set. -**Every atom and every invariant was proven BY HAND before it was automated** (§4c). That ordering is -the point: a runner written first and trusted after is exactly the rig the brief warns about. - -**A gap in Phase A found and closed first.** None of the four apps had `HDD_PATH` — all their data sat -in Docker volumes on the **system** disk, so detaching a data drive was a no-op for apps and **I3 could -never have fired**. A soak run in that state would have produced 27 green cycles that tested nothing -about cross-drive behaviour. **calibre-web** was deployed onto `adatok` -(`/mnt/felhom-drives/adatok/userdata/media/books → /calibre-library`, confirmed from -`docker inspect`) before the run started. - -### Atoms, permuted per cycle (seeded RNG, so the order is reproducible) - -`backup` · `restore_verify` · `detach_target` · `detach_nontarget` · `kill_controller` · -`redeploy_app`, plus `reboot_guest` on every 7th cycle. - -### Results - -| Invariant | Checks | Violations | What was actually asserted | +| Pass | Cycles | Atoms | Violations | |---|---|---|---| -| **I1** + pair | 28 + 28 | **0** | target absent ⇒ `backup_target_absent`/**error**; return ⇒ `backup_target_restored`/**info** | -| **I2** + pair | 28 + 28 | **0** | non-target ⇒ **generic** `storage_disconnected`/error → `storage_reconnected`/info | -| **I3** | 56 | **0** | `bound_under_parent=false` while the namespace is unusable, both drives | -| **I4** | 56 | **0** | guest init boot-time identical across every bind repair | -| **I5** | 28 | **0** | healthy ⇒ zero absent events, target reads healthy | -| **I6** | 28 | **0** | healthy carries no degraded message | -| **I7** | 28 | **0** | **every restore returned the discriminator asked for — 28/28** | -| **I10** | 135 | **0** | 4 × `type: secret` present in the unit; 1 × `type: password` **absent**, asserted as absence | -| **I11** | 28 | **0** | no secret value in controller or hub logs | -| BACKUP / REDEPLOY / KILL-CTRL / REBOOT | 56 / 28 / 28 / 3 | **0** | atom outcomes | +| **run 1** | 27 | 6 families | **0** | +| **run 2a** | 10 | 13 families | 2 — both **harness**, §4d | +| **run 2b** | **39** | 13 families | 9 (see below) | -The row counts are themselves a check that nothing was silently skipped: I3 and I4 at **56** (twice per -cycle — both drives), I10 at **135** (5 secret-class fields × 27), REBOOT at **3** (cycles 7/14/21). -Counts of 28 rather than 27 include the one-cycle smoke test, which wrote to the same journal and -restarts its numbering at 1 — the totals are right, the cycle *labels* collide. +**Run 2b — 875 invariant rows over 39 cycles, 13.5 h, ended on its own deadline.** No watchdog halt, +no atom exception, no I11 breach. -**A clean invariant is a result**: I1's discrimination, I2's genericity, the pairing of both, I4's -no-restart repair and I7's discriminator held over **27 consecutive cycles** of detach, reattach, -backup, restore, controller kill, app redeploy and three guest reboots. Nothing drifted at iteration 38 -because the run stopped at 27 — the tail beyond that is untested, not proven absent. +| Check | PASS | VIOL | Note | +|---|---|---|---| +| **I7** | **38** | **0** | + 1 `I7-SKIP` — the fixed harness declining to judge, not a false green | +| I1 / I1-pair | 37 / 36 | 2 / 3 | one 3-cycle window, **self-recovered** — §5.3 | +| I2 / I2-pair | 39 / 39 | 0 / 0 | | +| I3 / I3-abort | 78 / 4 | 0 | | +| I4 / I4-abort / -recover | 78 / 4 / 4 | 0 | | +| I5 / I6 | 39 / 39 | 0 | | +| I10 | **195** | 0 | 5 secret-class fields × 39 | +| I11 | 39 | 0 | | +| I1-under-load / -recover | 5 / 5 | **0** | target pulled **during** a running backup | +| BACKUP / KILL-CTRL | 82 / 39 | 0 | | +| KILL-AGENT / REBOOT / REBOOT-VM | 5 / 5 / 5 | 0 | | +| CONCURRENCY / FILL-DRIVE | 5 / 5 | 0 | | +| HARD-RESET | 2 | **3** | **R-157** | +| REDEPLOY | 38 | 1 | harness: 5-min wait too short under load; the app returned just after | -### Two behaviours worth recording, both observed repeatedly +**The full atom set ran this time.** The six families run 1 skipped are all present: filesystem +aborted in place (R-117 Q7), kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency +atoms, and fill-drive-near-full. The run-1 flaw where `reboot` was appended *after* the shuffle — so it +never interleaved with a detach — was fixed; heavy atoms are now permuted in. -- **The drive comes back on a different device node and the bind still heals.** `adatok` left as - `/dev/sdb` and returned as `/dev/sdd`; `bound_under_parent` went true again and the contents were - readable, with the guest's init boot-time unchanged. That is R-117's stale-device case, hit on every - detach cycle rather than once. -- **`health_degraded (warning)` accompanies the target-absent event** and clears on return, alongside - the specific `backup_target_*` pair. +**Two results worth naming.** `I1-under-load` passed 5/5: pulling the backup target **while a backup +was running** still produced `backup_target_absent` and a clean recovery. And `I4-abort` / `I3-abort` +passed 4/4 — R-117's Q7 case, the one its spike called *"the worse half"*, holds (details in +`../tests/campaign10-evidence-2026-07-31/r117-q7-abort-in-place.md`). + +## 4d. Two violations that were the HARNESS, not the product + +Recorded because **a check that fails for the wrong reason is as corrosive as one that passes for the +wrong reason**, and this arc already has six of the latter. Full write-up: +`../tests/campaign10-evidence-2026-07-31/run2a-violations-were-harness.md`. + +Run 2a cycle 10 reported `HARD-RESET canaries_intact=False` and, worse, **`I7` claiming a stale +restore** (`want=C10-C010-A got=C10-C009-A, restore_ok=True`). The `cc_proof` table settled it: its +highest row was `C10-C009-A` — **there was no `C010-A` row at all**, so the seed never landed, because +the hard reset earlier in the same cycle had left `rallly` `Exited`. `atom_restore_verify` called +`seed()` without checking its return; `atom_hard_reset_mid_write` read canaries out of app containers +that were still starting. + +**A real stale restore would have produced a byte-identical journal row**, which is why this justified +stopping a running 10-cycle pass rather than annotating it. Fixed with `apps_ready()`, `seed(verify=True)` +(read-back), and an `I7-SKIP` verdict so a skipped check is never silently a green one — red-proofed +both directions before restarting. Run 2b's single `I7-SKIP` is that fix working. ## 4c. Primitives proven by hand before automation @@ -343,82 +354,138 @@ storage-observation row drops out (a raw enrolled drive is not a PVE storage), c `MountPath`. **No `R-n` minted** — no consumer is misled, and the R-116 comment's claim that this "cannot make the gate read an absent drive as PRESENT" held under test. +### 5.3 — Investigated and DISPROVED: the `backup_target_*` pair going silent + +Five I1/I1-pair violations in a ~10-minute window looked like R-116's unmatchable pair returning. +It is not. Full write-up: `../tests/campaign10-evidence-2026-07-31/RESOLVED-backup-target-pair-went-silent.md`. + +The violations **cluster at cycles 31-33 and nowhere else** in 39 cycles; c34-c39 are clean, so it +recovered with no intervention. On the quiesced box afterwards, one slow detach with 4 minutes either +side produced a perfect pair (`backup_target_absent` → `backup_target_restored` → `health_recovered`). +And the alarming false-healthy I sampled alongside it (`mentes bound_under_parent=False` while +`degraded: false`) **does not survive quiescence** — three minutes after the run both drives read +`bound=True` and the target read healthy. I had been reading the two halves at different instants of +a detach. **No `R-n`.** + +### 5.4 — Investigated and DISPROVED: the hub's `SQLITE_BUSY` event drops + + [ERROR] Failed to save event from c10-soak: database is locked (5) (SQLITE_BUSY) + +7 in 24 h, **including one for the real customer `demo-felhom`**, so not a campaign artefact. The hub +returns HTTP 500 on the failed save (`hub/internal/api/handler.go:1715`) and dispatches notifications +only *after* a successful save — so a genuinely lost event would be a lost alarm. + +**But the controller retries 3×, and zero events exhausted their attempts** across the whole run +(`grep -c "Event push failed after 3 attempts"` = 0). The 07:04:39 drop landed 3 s later at 07:04:42, +same event. **Nothing was lost. No `R-n`.** One cosmetic note kept: `[ERROR] Failed to save event` +reads like data loss and is not. + ### Not filed - **demo-hp SSH key** — already R-129. - **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect. - **`/api/disks` `state`** — §5.2, disproved. +- **`backup_target_*` silence** — §5.3, transient and self-recovered. +- **hub `SQLITE_BUSY` drops** — §5.4, absorbed by the controller's retry. +- **F-CRIT-1 (`StateStopped` assumed deliberate)** — confirmed live again via R-157 mechanism B, but + already recorded in `CLAUDE.md`'s false-invariant table; **not re-filed**. --- ## 6. RTO/RPO — measured -**28 restores, every one returning the correct discriminator.** Wall-clock is measured from the restore -request to **the app serving the correct data** (the canary read back over rallly's real network path), -not to "restore returned". +**66 restores across run 1 and run 2b. Every one returned the correct discriminator.** | Tier | App | Data volume | n | min | median | p90 | max | mean | |---|---|---|---|---|---|---|---|---| -| **Tier 1 (local)** | rallly (postgres) | **66 MB** volume / 67 MB recovery unit | 28 | 38.8 s | **42.0 s** | 42.5 s | 44.3 s | 42.0 s | +| Tier 1 (local) | rallly (postgres) | 66 MB volume / 67 MB unit | 28 (run 1) | 38.8 s | **42.0 s** | 42.5 s | 44.3 s | 42.0 s | +| Tier 1 (local) | rallly (postgres) | 66 MB | **38 (run 2b)** | 29.1 s | **41.4 s** | 42.1 s | 49.6 s | 40.7 s | -**Which band this covers: the S band only, and only its lower end.** 66 MB is a small app by any -reading, and the spread is tight — 5.5 s between min and max over 28 runs — which says the cost is -dominated by fixed work (stack stop, volume restore, stack start, health wait) rather than by bytes. -**Nothing here licenses an extrapolation to M or L**: the byte-proportional term is precisely what -66 MB fails to exercise. A second point at a much larger volume is needed before any curve is claimed, -and none was taken. +The two independent passes agree to **0.6 s on the median**, and run 2b's wider spread (29.1–49.6 s +against 38.8–44.3 s) is the cost of measuring under hard resets, agent kills and a near-full drive. -**RPO is NOT measured.** It is a function of backup *frequency*, and the soak drove backups on demand -rather than on the schedule, so the observed intervals say nothing about the product's RPO. +**Band: the S band's lower end only.** 66 MB is small, and the tight distribution says fixed work — +stack stop, volume restore, start, health wait — dominates. **The byte-proportional term is exactly +what 66 MB fails to exercise, so nothing here extrapolates to M or L**, and no such claim is made. + +**RPO is not measured.** Backups were driven on demand, not on the schedule, so the observed +intervals say nothing about the product's RPO. + +--- + +## 6b. Monotonic growth — the half the invariants cannot see + +I1–I11 are *correctness* checks: they answer "is the system telling the truth this cycle", and every +one of them can pass while something climbs. **9 457 samples of 19 metrics over 13.5 h**, following +Campaign 2's `controller_rss.tsv` precedent (`state/growth.tsv`, analyser in `runner/`). + +**Leak-class — nothing grew.** Across 39 cycles containing dozens of redeploys, controller and agent +kills, guest reboots, VM reboots and hard resets: + +| Metric | start → end | verdict | +|---|---|---| +| controller RSS | 105.1 → 31.8 MB (min 0 / max 132 across restarts) | no leak | +| agent RSS | 19 008 → 19 632 KB (+2.07/cycle) | no leak | +| controller fd / agent fd | 14 → 11 / 14 → 13 | no leak | +| docker volumes / images / containers | 7 → 7 / 13 → 13 / 9 → 9 | **no orphans from repeated redeploys** | +| controller restarts | 0 | — | + +**Expected accumulation — all bounded and explicable:** + +| Metric | start → end | note | +|---|---|---| +| **agent journal** | **193.8 → 463.4 MB** (+6.5/cycle, ~20 MB/h) | **the only curve with real slope** | +| VM qcow2 on the host | 17 439 → 24 903 MB (+7.5 GB) | matches host free space falling 7.5 GB | +| backup target `mentes` | 623 → 6 424 MB | tracks `VZDUMP_N` 3 → 11 | +| guest root / docker logs | 5 667 → 5 940 MB / 4 → 9 MB | | + +Two things to watch rather than findings: the **agent journal** at ~20 MB/h (journald caps it, so it +is bounded, but it is a lot of logging); and **whole-guest dumps accumulating 3 → 11** on a 50 GB +target, which at soak rates would fill it in days — a real box does not back up this often, so this is +an artefact of the campaign's cadence rather than a defect. + +**Answering the question directly: after 13.5 h and 39 destructive cycles, nothing leaks.** That is a +genuine positive result and the half a targeted session structurally cannot produce. --- ## 7. I1–I11 across cycles -Full table in §4b. **I1, I2 (and both pairs), I3, I4, I5, I6, I7, I10 and I11 held over 27 consecutive -cycles — 586 checks, 0 violations.** No violation was deterministic and none was racy, because there -were none; the honest statement is that the run found no drift **within 27 cycles**, not that none -exists beyond them. +Full tables in §4b. Aggregated over run 1 (27 cycles) and run 2b (39 cycles): -**I8 was not checked** — Tier 3 never ran (§3). **I9 was not automated** — see §8. +- **I7 — 66 restores, 66 correct discriminators, 0 stale, 0 empty.** The campaign's headline. +- **I2, I3, I4, I5, I6, I10, I11 — zero violations in either pass**, including the abort-in-place + variants and 195 + 135 secret-class assertions covering both sides of D5's split. +- **I1 — 2 violations in 39 cycles**, one self-recovering 3-cycle window under cycling far faster than + a real box sees (§5.3, disproved as a defect). +- **I8 — never checked.** Tier 3 could not be isolated (§3). +- **I9 — not automated.** Cited from the tester-gate run on this same controller 0.188.0, not + re-claimed as this campaign's work. + +**A clean invariant is a result**: I2's genericity, I1's discrimination, both pairings, I4's +no-restart repair and I7's discriminator held across **66 cycles** of detach, reattach, abort, +backup, restore, kill, redeploy, reboot and hard reset. --- ## 8. What did not run, and why -Phase A overran badly (~5.5 h against the brief's ~1 h) — 1.26.1 is a public release image with no -auto-install path, so day-0 was a blind screendump-and-`sendkey` walk; the Tier-3 question (§3) had to -be settled before a customer could exist at all, since `dr_tier` is fixed at creation; and a false -start on drive enrolment had to be backed out. **Phase B was therefore deliberately not launched at -04:00** — the brief's own fence, *"a test rig producing false negatives is worse than no rig"* — and -was built and run the next morning instead, where its first cycle could be watched. - -**Atom coverage: 6 of the brief's ~12 suggested families.** Stated plainly rather than implied: +**Atom coverage: 12 of the brief's ~13 families ran.** Run 1 covered 6; run 2b added the rest. | Atom (brief B1) | Ran | |---|---| -| back up Tier 1/2 · restore + verify by discriminator · detach/reattach **target** · detach/reattach **non-target** · delete an app and redeploy · reboot the guest | **YES** | +| back up Tier 1/2 · restore + verify by discriminator · detach/reattach target · detach/reattach non-target · delete and redeploy · reboot the guest | **YES** (run 1 + 2b) | +| **abort a filesystem in place** (R-117 Q7) · **kill the agent mid-backup** · **hard-reset the VM mid-write** · **reboot the VM** · **concurrent backup+restore** · **concurrent backup+detach** · **fill a drive near-full** | **YES** (run 2b) | | back up / restore **Tier 3** | **NO** — §3, structurally impossible to isolate | -| abort a filesystem **in place** without the device disappearing (R-117 Q7) | **NO** | -| kill the **agent** mid-backup · **hard-reset the VM** mid-write · reboot the **VM** | **NO** | -| **concurrent** backup+restore · **concurrent** backup+detach | **NO** | -| fill a drive to near-full and continue | **NO** | -**`kill_controller` is not "mid-backup".** It restarts the controller at a permuted point in the -cycle, so it sometimes lands after a backup and sometimes nowhere near one. It exercises recovery, not -the mid-write race the brief asked for. Recorded because the atom name would otherwise overclaim. +**Still honest about two limits.** `kill_controller` is not literally "mid-backup" — it lands at a +permuted point, so it exercises recovery rather than the mid-write race; the dedicated +`concurrent backup+detach` atom (`I1-under-load`, 5/5 PASS) covers the mid-operation case properly. +And **I9** — a Tier-1/2 restore with the guest's `app.yaml` moved aside — was **not automated**; it +was proven on this same controller 0.188.0 by the tester-gate run and is cited, not re-claimed. -**`reboot_guest` was appended after the shuffle, not permuted into it**, so it always ran last in -cycles 7/14/21 and **never interleaved with a detach**. Reboot-during-detach is untested. - -**I8** — not checked; Tier 3 never ran. **I9** (Tier-1/2 restore with the guest's `app.yaml` moved -aside, data read over the app's real network path) — **not automated**. It was proven on this exact -controller 0.188.0 by the tester-gate run hours earlier -(`tester-gate-golden-0.188.0-2026-07-31.md` §5, step 6), and that is cited rather than re-claimed: -this campaign did not execute it. - -**The tail is untested, not clean.** 27 cycles is the count; the brief's interest in "the state that -quietly drifts on the thirty-eighth" is not answered by a 27-cycle run. +**Depth reached: 39 consecutive cycles**, past the brief's "the state that quietly drifts on the +thirty-eighth". Cycles 34–39 were clean on every invariant. Beyond 39 is untested, not proven clean. ---