Campaign 10: final results — 39 cycles, full atom set, R-156 + R-157, no leaks

Phase B completed in three passes: run 1 (27 cycles, 6 atom families, 0
violations), run 2a (10 cycles, stopped deliberately - two violations were harness
defects), run 2b (39 cycles, 12 of the brief's ~13 atom families). 1461 invariant
checks. Depth reached 39 consecutive cycles, past the brief's "drift at the
thirty-eighth", with c34-c39 clean on every invariant.

I7 headline: 66 restores across both passes, 66 correct discriminators - never
stale, never empty. I2/I3/I4/I5/I6/I10/I11 zero violations in either pass.
I1-under-load 5/5: the target pulled WHILE a backup ran still produced
backup_target_absent and a clean recovery. R-117's Q7 case holds - a filesystem
aborted in place surfaces and the gate stops the app on the dead namespace.

RTO Tier-1 rallly 66MB: run 1 median 42.0s, run 2b median 41.4s over 38 restores -
two independent passes agreeing to 0.6s. S band's lower end only; nothing
extrapolates to M or L. RPO not measured.

Monotonic growth, 9457 samples of 19 metrics over 13.5h: NO leak. Controller and
agent RSS flat, fds flat, no orphaned volumes/images/containers despite dozens of
redeploys, kills, reboots and hard resets. Only curve with real slope is the agent
journal at ~20MB/h, bounded by journald.

Findings: R-156 (papra's data neither persisted nor backed up, reports healthy)
and R-157 (bootrecon's start-once sweep, two mechanisms - the zero-container one
is silent on every channel). Four suspicions investigated and DISPROVED, each
recorded with what settled it.

REPORT-campaign10.md

@@ -8,8 +8,9 @@ clobbered — same shared-clone reasoning as `REPORT-iso-release.md`.
 
 ## The sentence that matters
 
-**Phase A passed every gate (07-31) and Phase B ran (08-01): 27 cycles, 586 invariant checks, zero
-violations, ended on its own deadline with no watchdog halt and no atom exception.**
+**Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the
+"drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No
+resource leak over 13.5 h. Two findings: R-156 and R-157.**
 
 ## What was established
 
@@ -35,20 +36,35 @@ violations, ended on its own deadline with no watchdog halt and no atom exceptio
 
 ## Phase B — the soak
 
-27 cycles, 09:33–15:40 CEST, atoms permuted per cycle, every invariant checked every cycle.
+Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, **stopped
+deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 atom families**).
+1 461 invariant checks in total.
 
-- **I1** 28 + 28 pairs, **I2** 28 + 28 pairs, **I3** 56, **I4** 56, **I5/I6** 28 each, **I7** 28,
-  **I10** 135, **I11** 28 — **0 violations**. Row counts are themselves the no-silent-skip check
-  (I3/I4 twice per cycle = both drives; I10 = 5 secret-class fields × 27; REBOOT = cycles 7/14/21).
-- **I7 is the headline: 28 restores, 28 correct discriminators.** Never stale, never empty.
-- **RTO (Tier 1, rallly, 66 MB):** min 38.8 s, **median 42.0 s**, p90 42.5 s, max 44.3 s. **Covers the
-  S band only, and only its lower end** — the 5.5 s spread over 28 runs says fixed work dominates, so
-  nothing here extrapolates to M or L. **RPO not measured** (backups were driven on demand).
+- **I7 is the headline: 66 restores across both passes, 66 correct discriminators.** Never stale,
+  never empty. Run 2b added an `I7-SKIP` verdict so a check with unmet preconditions is recorded as
+  skipped rather than silently green — it fired once.
+- **I2, I3, I4, I5, I6, I10, I11: zero violations in either pass**, including the abort-in-place
+  variants and 330 secret-class assertions covering both sides of D5's split.
+- **`I1-under-load` 5/5**: the backup target pulled **while a backup was running** still produced
+  `backup_target_absent` and a clean recovery.
+- **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via
+  `bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names
+  it. That is the case R-117's spike called "the worse half".
+- **RTO (Tier 1, rallly, 66 MB):** run 1 median **42.0 s**, run 2b median **41.4 s** over 38 restores
+  — two independent passes agreeing to 0.6 s. **S band's lower end only**; nothing extrapolates to M
+  or L. **RPO not measured.**
+- **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds
+  flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots
+  and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)** —
+  bounded by journald, but a lot of logging.
 - Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that
   was not first observed live.
 - **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system
   disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the
-  soak would have produced 27 green cycles that tested nothing cross-drive.
+  soak would have produced green cycles that tested nothing cross-drive.
+- **Two violations were my harness, not the product**, and run 2a was stopped for them: a seed that
+  never landed became a fake "stale restore", and a real one would have looked identical. Fixed and
+  red-proofed before restarting.
 
 ## Findings
 
@@ -57,6 +73,13 @@ violations, ended on its own deadline with no watchdog halt and no atom exceptio
   empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB
   sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its
   Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
+- **R-157 (new)** — **bootrecon's start-once sweep misses the boot orphan it exists to recover.** Two
+  mechanisms. **A:** the container is left `Exited`, the sweep runs ~5 s after controller start while
+  docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences,
+  intermittent ~50%). **B:** the interruption leaves the stack with **zero containers**, which is
+  exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp
+  check reported **`0 currently down`** while a `deployed: true` app was not running, i.e. silent on
+  every channel. A settle-condition fix closes A and leaves B open. Not fixed.
 - **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier
   (`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred,
   `hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign
@@ -65,18 +88,17 @@ violations, ended on its own deadline with no watchdog halt and no atom exceptio
 
 ## What did not run
 
-**6 of the brief's ~12 atom families ran.** Not run: Tier-3 backup/restore (§3), abort-a-filesystem-
-in-place (R-117 Q7), kill-the-agent-mid-backup, hard-reset-the-VM-mid-write, reboot-the-VM, both
-concurrency atoms, and fill-a-drive-near-full. **I8** not checked (no Tier 3); **I9** not automated —
-it was proven on this same controller 0.188.0 by the tester-gate run hours earlier, which is cited,
-not re-claimed.
+**12 of the brief's ~13 atom families ran** (run 1 covered 6; run 2b added abort-fs-in-place,
+kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive).
+Previously reported as 6 of 12 — that was run 1 only.
 
-Two precision notes so atom names don't overclaim: `kill_controller` is **not** "mid-backup" (it lands
-at a permuted point, exercising recovery rather than the mid-write race), and `reboot_guest` was
-appended after the shuffle so it **never interleaved with a detach**.
+**Superseded detail:** Still not run: **Tier-3 backup/restore** (§3, structurally un-isolatable) and **I8**. **I9** was not
+automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed.
+`kill_controller` is still not literally "mid-backup"; the dedicated concurrent backup+detach atom
+covers that case properly. The run-1 flaw where `reboot` never interleaved with a detach was fixed.
 
-**The tail is untested, not clean.** The brief's interest in "the state that quietly drifts on the
-thirty-eighth" is not answered by 27 cycles.
+**Depth reached: 39 consecutive cycles**, past the brief's "thirty-eighth", with c34–c39 clean on
+every invariant. Beyond 39 is untested, not proven clean.
 
 ## Teardown — OWED, nothing removed