Campaign 10: two run-2a violations were HARNESS defects, not product defects — fixed

Run 2a hit its first two violations at cycle 10 and BOTH trace to my harness, not
the product. Recorded in full because a check that fails for the wrong reason is
as corrosive as one that passes for the wrong reason.

  HARD-RESET  VM returned=True canaries_intact=False
  I7          want=C10-C010-A-194530 got=C10-C009-A-192929 restore_ok=True

Root cause, evidenced: the cc_proof table's highest row is C10-C009-A — there is
NO C010-A row at all, so the seed never landed. The hard-reset atom ran earlier in
the same cycle and left rallly Exited(255); atom_restore_verify called seed() and
never checked its return value, so an unwritten generation became a fake stale

documentation/tests/campaign10-evidence-2026-07-31/runner/c10lib.py

@@ -96,8 +96,11 @@ def _psql(sql, timeout=180):
     return rc, out.strip(), err.strip()
 
 
-def seed(gen):
-    """Write the generation marker into every app. Returns {app: ok}."""
+def seed(gen, verify=True):
+    """Write the generation marker into every app. Returns {app: ok}.
+
+    With verify=True the value is READ BACK — a write that reported success but did not
+    land is the failure mode that turned a dead app into a fake 'stale restore' in run 2a."""
     res = {}
     _psql("CREATE TABLE IF NOT EXISTS cc_proof (id serial primary key, gen text, at timestamptz default now())")
     rc, out, err = _psql(f"INSERT INTO cc_proof (gen) VALUES ('{gen}') RETURNING gen")
@@ -109,6 +112,10 @@ def seed(gen):
         rc, out, err = guest(
             f"docker exec -u 0 {cont} sh -c {shq(f'echo {gen} > {d}/cc_proof.txt && cat {d}/cc_proof.txt')}", 180)
         res[app] = (gen in out)
+    if verify:
+        back = read_canaries()
+        for a in list(res):
+            res[a] = res[a] and (back.get(a) == gen)
     return res
 
 
@@ -124,6 +131,36 @@ def read_canaries():
     return res
 
 
+def apps_ready(names=None, timeout=600):
+    """Wait until the named app containers are RUNNING (healthy where they report health).
+
+    Load-bearing: a canary lives inside an app container, so reading one before the app is
+    back does not measure the product — it measures the harness. Two false violations in
+    run 2a came from exactly that (a seed that never landed, then a 'stale' restore)."""
+    names = names or (ALL_APPS + [DRIVE_APP])
+    t0 = time.time()
+    while time.time() - t0 < timeout:
+        c = containers()
+        ok = True
+        for n in names:
+            probe = "rallly-postgres" if n == "rallly" else n
+            st = c.get(n) or ""
+            st2 = c.get(probe) or ""
+            if not st or "Up" not in st or "starting" in st:
+                ok = False
+                break
+            if n == "rallly" and ("Up" not in st2 or "starting" in st2):
+                ok = False
+                break
+        if ok:
+            return True
+        time.sleep(10)
+    return False
+
+
+DRIVE_APP = "calibre-web"
+
+
 def containers():
     rc, out, _ = guest("docker ps --format '{{.Names}}|{{.Status}}'", 120)
     d = {}