diff --git a/documentation/tests/campaign10-evidence-2026-07-31/run2a-violations-were-harness.md b/documentation/tests/campaign10-evidence-2026-07-31/run2a-violations-were-harness.md new file mode 100644 index 0000000..95df5f9 --- /dev/null +++ b/documentation/tests/campaign10-evidence-2026-07-31/run2a-violations-were-harness.md @@ -0,0 +1,78 @@ +# Run 2a — the first two violations were HARNESS defects, not product defects (2026-08-01) + +Recorded in full, because **a check that fails for the wrong reason is as corrosive as one that +passes for the wrong reason** — and this arc already has six of the latter. + +## What fired + + cycle 10 HARD-RESET VIOLATION VM returned=True canaries_intact=False + cycle 10 I7 VIOLATION want=C10-C010-A-194530 got=C10-C009-A-192929 + snap=helyi restore_ok=True rto=41.0s + +I7 reporting a **stale restore** is the campaign's headline invariant failing. It was not real. + +## Root cause — evidenced, not reasoned + +Cycle 10's permutation was +`detach_target → detach_nontarget → hard_reset_mid_write → redeploy_app → restore_verify → …`, +so the hard reset ran **before** the restore in the same cycle. It left the app container +`rallly | Exited (255)` while `rallly-postgres` stayed healthy. + +The decisive evidence is the canary table itself — every generation the campaign ever wrote, in order: + + … + 39|C10-C008-A-185905 + 40|C10-C009-A-192929 <-- highest row + +**There is no `C10-C010-A` row at all.** The seed never landed, because `_psql` runs +`docker run --network container:rallly …` and there was no `rallly` container to attach to. The +restore then correctly returned the newest generation that *had* been written — C009-A — and the +harness called that "stale". + +Two harness defects, both mine: + +1. `atom_restore_verify` called `L.seed(genA)` and **never checked the return value**, so an unwritten + generation was indistinguishable from a stale restore. +2. `atom_hard_reset_mid_write` waited only for `felhom-controller` and then read canaries out of app + containers that were still starting — it measured startup latency, not corruption. + +## Why this justified stopping a 10-cycle run + +A **real** stale restore would have produced a byte-identical journal row. Leaving the flaw in place +would have made I7 — the one invariant the whole campaign is built around — unable to tell a genuine +data-loss event from a slow container start. Depth is worthless if the verdict is ambiguous. + +## The fixes, red-proofed in both directions before restarting + +- `L.apps_ready()` waits for the **app** containers, health-aware, not the controller. +- `L.seed(verify=True)` **reads the value back**; a write that reported success but did not land now + returns False. +- `atom_restore_verify` gates on both and records **`I7-SKIP`** — a PASS row carrying the reason — + when preconditions fail. A skipped check is never silently a green one. +- `atom_hard_reset_mid_write` judges canaries only once `apps_ready`; **apps failing to return within + 10 minutes is itself the violation**, which is the thing actually worth alarming on. + +| Control | Result | +|---|---| +| positive — all apps up | `apps_ready=True` in 2 s; all four seeds land | +| negative — grafana stopped | `seed → {grafana: False, …}`; `apps_ready=False` | + +## Product observations from the same window — NOT filed as findings + +Both are recorded so a later session does not rediscover them as novel, and neither is claimed as a +defect: the app recovered on its own and the behaviour was transient. + +- **The health probe named the wrong container.** During recovery the controller logged + `Health probe rallly: HTTP GET :3000/ → Get "http://rallly-postgres:3000/"` — the **database** + container's hostname on the **app's** port. **5 occurrences inside a ~70 s window**, then correct. + rallly's own log showed `▲ Next.js 16.2.6 ✓ Ready` throughout. An operator reading only this log + would chase a DNS fault on postgres. Transient mis-resolution while the app container was absent. +- **`ValidateDump` WARN:** *"structurally valid (32 tables) but its accounts table has NO rows — the + dump may predate the customer's data"* — the guard firing as designed, on a dump taken while the + app was down. + +## Disposition + +Run 2a stopped at cycle 10. Its evidence is preserved as `journal-run2a.tsv`, `rto-run2a.tsv`, +`runner-run2a.log`. **The 10 cycles' other invariant rows remain valid** — I1, I2, I3, I4, I5, I6, +I10 and I11 are independent of the seed path. Run 2b restarted from cycle 1 with the fixed harness.