admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Campaign 10: final results — 39 cycles, full atom set, R-156 + R-157, no leaks

Browse Source 
Phase B completed in three passes: run 1 (27 cycles, 6 atom families, 0
violations), run 2a (10 cycles, stopped deliberately - two violations were harness
defects), run 2b (39 cycles, 12 of the brief's ~13 atom families). 1461 invariant
checks. Depth reached 39 consecutive cycles, past the brief's "drift at the
thirty-eighth", with c34-c39 clean on every invariant.

I7 headline: 66 restores across both passes, 66 correct discriminators - never
stale, never empty. I2/I3/I4/I5/I6/I10/I11 zero violations in either pass.
I1-under-load 5/5: the target pulled WHILE a backup ran still produced
backup_target_absent and a clean recovery. R-117's Q7 case holds - a filesystem
aborted in place surfaces and the gate stops the app on the dead namespace.

RTO Tier-1 rallly 66MB: run 1 median 42.0s, run 2b median 41.4s over 38 restores -
two independent passes agreeing to 0.6s. S band's lower end only; nothing
extrapolates to M or L. RPO not measured.

Monotonic growth, 9457 samples of 19 metrics over 13.5h: NO leak. Controller and
agent RSS flat, fds flat, no orphaned volumes/images/containers despite dozens of
redeploys, kills, reboots and hard resets. Only curve with real slope is the agent
journal at ~20MB/h, bounded by journald.

Findings: R-156 (papra's data neither persisted nor backed up, reports healthy)
and R-157 (bootrecon's start-once sweep, two mechanisms - the zero-container one
is silent on every channel). Four suspicions investigated and DISPROVED, each
recorded with what settled it.
This commit is contained in:
authentik Default Admin
2026-08-02 07:13:04 +02:00
parent 405a795e32
commit 7ba7c2a271
2 changed files with 203 additions and 114 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT-campaign10.md
+43 -21
View File
@@ -8,8 +8,9 @@ clobbered — same shared-clone reasoning as `REPORT-iso-release.md`.
## The sentence that matters ## The sentence that matters
**Phase A passed every gate (07-31) and Phase B ran (08-01): 27 cycles, 586 invariant checks, zero **Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the
violations, ended on its own deadline with no watchdog halt and no atom exception.** "drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No
resource leak over 13.5 h. Two findings: R-156 and R-157.**
## What was established ## What was established
@@ -35,20 +36,35 @@ violations, ended on its own deadline with no watchdog halt and no atom exceptio
## Phase B — the soak ## Phase B — the soak
27 cycles, 09:33–15:40 CEST, atoms permuted per cycle, every invariant checked every cycle. Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, **stopped
deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 atom families**).
1 461 invariant checks in total.
- **I1** 28 + 28 pairs, **I2** 28 + 28 pairs, **I3** 56, **I4** 56, **I5/I6** 28 each, **I7** 28, - **I7 is the headline: 66 restores across both passes, 66 correct discriminators.** Never stale,
**I10** 135, **I11** 28 — **0 violations**. Row counts are themselves the no-silent-skip check never empty. Run 2b added an `I7-SKIP` verdict so a check with unmet preconditions is recorded as
(I3/I4 twice per cycle = both drives; I10 = 5 secret-class fields × 27; REBOOT = cycles 7/14/21). skipped rather than silently green — it fired once.
- **I7 is the headline: 28 restores, 28 correct discriminators.** Never stale, never empty. - **I2, I3, I4, I5, I6, I10, I11: zero violations in either pass**, including the abort-in-place
- **RTO (Tier 1, rallly, 66 MB):** min 38.8 s, **median 42.0 s**, p90 42.5 s, max 44.3 s. **Covers the variants and 330 secret-class assertions covering both sides of D5's split.
S band only, and only its lower end** — the 5.5 s spread over 28 runs says fixed work dominates, so - **`I1-under-load` 5/5**: the backup target pulled **while a backup was running** still produced
nothing here extrapolates to M or L. **RPO not measured** (backups were driven on demand). `backup_target_absent` and a clean recovery.
- **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via
`bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names
it. That is the case R-117's spike called "the worse half".
- **RTO (Tier 1, rallly, 66 MB):** run 1 median **42.0 s**, run 2b median **41.4 s** over 38 restores
— two independent passes agreeing to 0.6 s. **S band's lower end only**; nothing extrapolates to M
or L. **RPO not measured.**
- **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds
flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots
and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)** —
bounded by journald, but a lot of logging.
- Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that - Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that
was not first observed live. was not first observed live.
- **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system - **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system
disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the
soak would have produced 27 green cycles that tested nothing cross-drive. soak would have produced green cycles that tested nothing cross-drive.
- **Two violations were my harness, not the product**, and run 2a was stopped for them: a seed that
never landed became a fake "stale restore", and a real one would have looked identical. Fixed and
red-proofed before restarting.
## Findings ## Findings
@@ -57,6 +73,13 @@ violations, ended on its own deadline with no watchdog halt and no atom exceptio
empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB
sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its
Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed. Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
- **R-157 (new)** — **bootrecon's start-once sweep misses the boot orphan it exists to recover.** Two
mechanisms. **A:** the container is left `Exited`, the sweep runs ~5 s after controller start while
docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences,
intermittent ~50%). **B:** the interruption leaves the stack with **zero containers**, which is
exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp
check reported **`0 currently down`** while a `deployed: true` app was not running, i.e. silent on
every channel. A settle-condition fix closes A and leaves B open. Not fixed.
- **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier - **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier
(`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred, (`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred,
`hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign `hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign
@@ -65,18 +88,17 @@ violations, ended on its own deadline with no watchdog halt and no atom exceptio
## What did not run ## What did not run
**6 of the brief's ~12 atom families ran.** Not run: Tier-3 backup/restore (§3), abort-a-filesystem- **12 of the brief's ~13 atom families ran** (run 1 covered 6; run 2b added abort-fs-in-place,
in-place (R-117 Q7), kill-the-agent-mid-backup, hard-reset-the-VM-mid-write, reboot-the-VM, both kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive).
concurrency atoms, and fill-a-drive-near-full. **I8** not checked (no Tier 3); **I9** not automated — Previously reported as 6 of 12 — that was run 1 only.
it was proven on this same controller 0.188.0 by the tester-gate run hours earlier, which is cited,
not re-claimed.
Two precision notes so atom names don't overclaim: `kill_controller` is **not** "mid-backup" (it lands **Superseded detail:** Still not run: **Tier-3 backup/restore** (§3, structurally un-isolatable) and **I8**. **I9** was not
at a permuted point, exercising recovery rather than the mid-write race), and `reboot_guest` was automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed.
appended after the shuffle so it **never interleaved with a detach**. `kill_controller` is still not literally "mid-backup"; the dedicated concurrent backup+detach atom
covers that case properly. The run-1 flaw where `reboot` never interleaved with a detach was fixed.
**The tail is untested, not clean.** The brief's interest in "the state that quietly drifts on the **Depth reached: 39 consecutive cycles**, past the brief's "thirty-eighth", with c34–c39 clean on
thirty-eighth" is not answered by 27 cycles. every invariant. Beyond 39 is untested, not proven clean.
## Teardown — OWED, nothing removed ## Teardown — OWED, nothing removed
documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md
+160 -93
View File
@@ -1,8 +1,13 @@
# CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01) # CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01)
**Status: Phase A COMPLETE and gated (2026-07-31). Phase B RUN 2026-08-01 — 27 cycles, 586 invariant **Phase A COMPLETE and gated (2026-07-31). Phase B run in three passes (2026-08-01/02): run 1
checks, ZERO violations.** **One new finding: R-156. One structural scope constraint: Tier 3 cannot be 27 cycles, run 2a 10 cycles (stopped — harness defect), run 2b **39 cycles**. Together **66 clean
isolated (§3). One suspected defect investigated and DISPROVED (§5.2).** cycles + 39 with the full atom set**, 1 461 invariant checks.**
**Findings: R-156 (papra), R-157 (bootrecon, two mechanisms).** One structural scope constraint —
Tier 3 cannot be isolated (§3). **Three suspicions investigated and DISPROVED** (§5.2, §5.3), and
**two violations traced to the harness, not the product** (§4d). Monotonic-growth sampling over
13.5 h found **no leak** (§6b).
Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step), Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step),
`isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`. `isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`.
@@ -189,59 +194,65 @@ false pass.
--- ---
## 4b. Phase B — the soak (2026-08-01) ## 4b. Phase B — the soak
**27 cycles, 09:33–15:40 CEST, ended on its own deadline. No watchdog halt, no atom exception, no Three passes. Run 2a was stopped deliberately at cycle 10 when two violations turned out to be
I11 breach. 586 invariant rows, 0 violations.** **harness** defects (§4d); the harness was fixed and red-proofed, and run 2b restarted from cycle 1
with the full atom set.
**Every atom and every invariant was proven BY HAND before it was automated** (§4c). That ordering is | Pass | Cycles | Atoms | Violations |
the point: a runner written first and trusted after is exactly the rig the brief warns about.
**A gap in Phase A found and closed first.** None of the four apps had `HDD_PATH` — all their data sat
in Docker volumes on the **system** disk, so detaching a data drive was a no-op for apps and **I3 could
never have fired**. A soak run in that state would have produced 27 green cycles that tested nothing
about cross-drive behaviour. **calibre-web** was deployed onto `adatok`
(`/mnt/felhom-drives/adatok/userdata/media/books → /calibre-library`, confirmed from
`docker inspect`) before the run started.
### Atoms, permuted per cycle (seeded RNG, so the order is reproducible)
`backup` · `restore_verify` · `detach_target` · `detach_nontarget` · `kill_controller` ·
`redeploy_app`, plus `reboot_guest` on every 7th cycle.
### Results
| Invariant | Checks | Violations | What was actually asserted |
|---|---|---|---| |---|---|---|---|
| **I1** + pair | 28 + 28 | **0** | target absent ⇒ `backup_target_absent`/**error**; return ⇒ `backup_target_restored`/**info** | | **run 1** | 27 | 6 families | **0** |
| **I2** + pair | 28 + 28 | **0** | non-target ⇒ **generic** `storage_disconnected`/error → `storage_reconnected`/info | | **run 2a** | 10 | 13 families | 2 — both **harness**, §4d |
| **I3** | 56 | **0** | `bound_under_parent=false` while the namespace is unusable, both drives | | **run 2b** | **39** | 13 families | 9 (see below) |
| **I4** | 56 | **0** | guest init boot-time identical across every bind repair |
| **I5** | 28 | **0** | healthy ⇒ zero absent events, target reads healthy |
| **I6** | 28 | **0** | healthy carries no degraded message |
| **I7** | 28 | **0** | **every restore returned the discriminator asked for — 28/28** |
| **I10** | 135 | **0** | 4 × `type: secret` present in the unit; 1 × `type: password` **absent**, asserted as absence |
| **I11** | 28 | **0** | no secret value in controller or hub logs |
| BACKUP / REDEPLOY / KILL-CTRL / REBOOT | 56 / 28 / 28 / 3 | **0** | atom outcomes |
The row counts are themselves a check that nothing was silently skipped: I3 and I4 at **56** (twice per **Run 2b — 875 invariant rows over 39 cycles, 13.5 h, ended on its own deadline.** No watchdog halt,
cycle — both drives), I10 at **135** (5 secret-class fields × 27), REBOOT at **3** (cycles 7/14/21). no atom exception, no I11 breach.
Counts of 28 rather than 27 include the one-cycle smoke test, which wrote to the same journal and
restarts its numbering at 1 — the totals are right, the cycle *labels* collide.
**A clean invariant is a result**: I1's discrimination, I2's genericity, the pairing of both, I4's | Check | PASS | VIOL | Note |
no-restart repair and I7's discriminator held over **27 consecutive cycles** of detach, reattach, |---|---|---|---|
backup, restore, controller kill, app redeploy and three guest reboots. Nothing drifted at iteration 38 | **I7** | **38** | **0** | + 1 `I7-SKIP` — the fixed harness declining to judge, not a false green |
because the run stopped at 27 — the tail beyond that is untested, not proven absent. | I1 / I1-pair | 37 / 36 | 2 / 3 | one 3-cycle window, **self-recovered** — §5.3 |
| I2 / I2-pair | 39 / 39 | 0 / 0 | |
| I3 / I3-abort | 78 / 4 | 0 | |
| I4 / I4-abort / -recover | 78 / 4 / 4 | 0 | |
| I5 / I6 | 39 / 39 | 0 | |
| I10 | **195** | 0 | 5 secret-class fields × 39 |
| I11 | 39 | 0 | |
| I1-under-load / -recover | 5 / 5 | **0** | target pulled **during** a running backup |
| BACKUP / KILL-CTRL | 82 / 39 | 0 | |
| KILL-AGENT / REBOOT / REBOOT-VM | 5 / 5 / 5 | 0 | |
| CONCURRENCY / FILL-DRIVE | 5 / 5 | 0 | |
| HARD-RESET | 2 | **3** | **R-157** |
| REDEPLOY | 38 | 1 | harness: 5-min wait too short under load; the app returned just after |
### Two behaviours worth recording, both observed repeatedly **The full atom set ran this time.** The six families run 1 skipped are all present: filesystem
aborted in place (R-117 Q7), kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency
atoms, and fill-drive-near-full. The run-1 flaw where `reboot` was appended *after* the shuffle — so it
never interleaved with a detach — was fixed; heavy atoms are now permuted in.
- **The drive comes back on a different device node and the bind still heals.** `adatok` left as **Two results worth naming.** `I1-under-load` passed 5/5: pulling the backup target **while a backup
`/dev/sdb` and returned as `/dev/sdd`; `bound_under_parent` went true again and the contents were was running** still produced `backup_target_absent` and a clean recovery. And `I4-abort` / `I3-abort`
readable, with the guest's init boot-time unchanged. That is R-117's stale-device case, hit on every passed 4/4 — R-117's Q7 case, the one its spike called *"the worse half"*, holds (details in
detach cycle rather than once. `../tests/campaign10-evidence-2026-07-31/r117-q7-abort-in-place.md`).
- **`health_degraded (warning)` accompanies the target-absent event** and clears on return, alongside
the specific `backup_target_*` pair. ## 4d. Two violations that were the HARNESS, not the product
Recorded because **a check that fails for the wrong reason is as corrosive as one that passes for the
wrong reason**, and this arc already has six of the latter. Full write-up:
`../tests/campaign10-evidence-2026-07-31/run2a-violations-were-harness.md`.
Run 2a cycle 10 reported `HARD-RESET canaries_intact=False` and, worse, **`I7` claiming a stale
restore** (`want=C10-C010-A got=C10-C009-A, restore_ok=True`). The `cc_proof` table settled it: its
highest row was `C10-C009-A` — **there was no `C010-A` row at all**, so the seed never landed, because
the hard reset earlier in the same cycle had left `rallly` `Exited`. `atom_restore_verify` called
`seed()` without checking its return; `atom_hard_reset_mid_write` read canaries out of app containers
that were still starting.
**A real stale restore would have produced a byte-identical journal row**, which is why this justified
stopping a running 10-cycle pass rather than annotating it. Fixed with `apps_ready()`, `seed(verify=True)`
(read-back), and an `I7-SKIP` verdict so a skipped check is never silently a green one — red-proofed
both directions before restarting. Run 2b's single `I7-SKIP` is that fix working.
## 4c. Primitives proven by hand before automation ## 4c. Primitives proven by hand before automation
@@ -343,82 +354,138 @@ storage-observation row drops out (a raw enrolled drive is not a PVE storage), c
`MountPath`. **No `R-n` minted** — no consumer is misled, and the R-116 comment's claim that this `MountPath`. **No `R-n` minted** — no consumer is misled, and the R-116 comment's claim that this
"cannot make the gate read an absent drive as PRESENT" held under test. "cannot make the gate read an absent drive as PRESENT" held under test.
### 5.3 — Investigated and DISPROVED: the `backup_target_*` pair going silent
Five I1/I1-pair violations in a ~10-minute window looked like R-116's unmatchable pair returning.
It is not. Full write-up: `../tests/campaign10-evidence-2026-07-31/RESOLVED-backup-target-pair-went-silent.md`.
The violations **cluster at cycles 31-33 and nowhere else** in 39 cycles; c34-c39 are clean, so it
recovered with no intervention. On the quiesced box afterwards, one slow detach with 4 minutes either
side produced a perfect pair (`backup_target_absent` → `backup_target_restored` → `health_recovered`).
And the alarming false-healthy I sampled alongside it (`mentes bound_under_parent=False` while
`degraded: false`) **does not survive quiescence** — three minutes after the run both drives read
`bound=True` and the target read healthy. I had been reading the two halves at different instants of
a detach. **No `R-n`.**
### 5.4 — Investigated and DISPROVED: the hub's `SQLITE_BUSY` event drops
[ERROR] Failed to save event from c10-soak: database is locked (5) (SQLITE_BUSY)
7 in 24 h, **including one for the real customer `demo-felhom`**, so not a campaign artefact. The hub
returns HTTP 500 on the failed save (`hub/internal/api/handler.go:1715`) and dispatches notifications
only *after* a successful save — so a genuinely lost event would be a lost alarm.
**But the controller retries 3×, and zero events exhausted their attempts** across the whole run
(`grep -c "Event push failed after 3 attempts"` = 0). The 07:04:39 drop landed 3 s later at 07:04:42,
same event. **Nothing was lost. No `R-n`.** One cosmetic note kept: `[ERROR] Failed to save event`
reads like data loss and is not.
### Not filed ### Not filed
- **demo-hp SSH key** — already R-129. - **demo-hp SSH key** — already R-129.
- **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect. - **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect.
- **`/api/disks` `state`** — §5.2, disproved. - **`/api/disks` `state`** — §5.2, disproved.
- **`backup_target_*` silence** — §5.3, transient and self-recovered.
- **hub `SQLITE_BUSY` drops** — §5.4, absorbed by the controller's retry.
- **F-CRIT-1 (`StateStopped` assumed deliberate)** — confirmed live again via R-157 mechanism B, but
already recorded in `CLAUDE.md`'s false-invariant table; **not re-filed**.
--- ---
## 6. RTO/RPO — measured ## 6. RTO/RPO — measured
**28 restores, every one returning the correct discriminator.** Wall-clock is measured from the restore **66 restores across run 1 and run 2b. Every one returned the correct discriminator.**
request to **the app serving the correct data** (the canary read back over rallly's real network path),
not to "restore returned".
| Tier | App | Data volume | n | min | median | p90 | max | mean | | Tier | App | Data volume | n | min | median | p90 | max | mean |
|---|---|---|---|---|---|---|---|---| |---|---|---|---|---|---|---|---|---|
| **Tier 1 (local)** | rallly (postgres) | **66 MB** volume / 67 MB recovery unit | 28 | 38.8 s | **42.0 s** | 42.5 s | 44.3 s | 42.0 s | | Tier 1 (local) | rallly (postgres) | 66 MB volume / 67 MB unit | 28 (run 1) | 38.8 s | **42.0 s** | 42.5 s | 44.3 s | 42.0 s |
| Tier 1 (local) | rallly (postgres) | 66 MB | **38 (run 2b)** | 29.1 s | **41.4 s** | 42.1 s | 49.6 s | 40.7 s |
**Which band this covers: the S band only, and only its lower end.** 66 MB is a small app by any The two independent passes agree to **0.6 s on the median**, and run 2b's wider spread (29.1–49.6 s
reading, and the spread is tight — 5.5 s between min and max over 28 runs — which says the cost is against 38.8–44.3 s) is the cost of measuring under hard resets, agent kills and a near-full drive.
dominated by fixed work (stack stop, volume restore, stack start, health wait) rather than by bytes.
**Nothing here licenses an extrapolation to M or L**: the byte-proportional term is precisely what
66 MB fails to exercise. A second point at a much larger volume is needed before any curve is claimed,
and none was taken.
**RPO is NOT measured.** It is a function of backup *frequency*, and the soak drove backups on demand **Band: the S band's lower end only.** 66 MB is small, and the tight distribution says fixed work —
rather than on the schedule, so the observed intervals say nothing about the product's RPO. stack stop, volume restore, start, health wait — dominates. **The byte-proportional term is exactly
what 66 MB fails to exercise, so nothing here extrapolates to M or L**, and no such claim is made.
**RPO is not measured.** Backups were driven on demand, not on the schedule, so the observed
intervals say nothing about the product's RPO.
---
## 6b. Monotonic growth — the half the invariants cannot see
I1–I11 are *correctness* checks: they answer "is the system telling the truth this cycle", and every
one of them can pass while something climbs. **9 457 samples of 19 metrics over 13.5 h**, following
Campaign 2's `controller_rss.tsv` precedent (`state/growth.tsv`, analyser in `runner/`).
**Leak-class — nothing grew.** Across 39 cycles containing dozens of redeploys, controller and agent
kills, guest reboots, VM reboots and hard resets:
| Metric | start → end | verdict |
|---|---|---|
| controller RSS | 105.1 → 31.8 MB (min 0 / max 132 across restarts) | no leak |
| agent RSS | 19 008 → 19 632 KB (+2.07/cycle) | no leak |
| controller fd / agent fd | 14 → 11 / 14 → 13 | no leak |
| docker volumes / images / containers | 7 → 7 / 13 → 13 / 9 → 9 | **no orphans from repeated redeploys** |
| controller restarts | 0 | — |
**Expected accumulation — all bounded and explicable:**
| Metric | start → end | note |
|---|---|---|
| **agent journal** | **193.8 → 463.4 MB** (+6.5/cycle, ~20 MB/h) | **the only curve with real slope** |
| VM qcow2 on the host | 17 439 → 24 903 MB (+7.5 GB) | matches host free space falling 7.5 GB |
| backup target `mentes` | 623 → 6 424 MB | tracks `VZDUMP_N` 3 → 11 |
| guest root / docker logs | 5 667 → 5 940 MB / 4 → 9 MB | |
Two things to watch rather than findings: the **agent journal** at ~20 MB/h (journald caps it, so it
is bounded, but it is a lot of logging); and **whole-guest dumps accumulating 3 → 11** on a 50 GB
target, which at soak rates would fill it in days — a real box does not back up this often, so this is
an artefact of the campaign's cadence rather than a defect.
**Answering the question directly: after 13.5 h and 39 destructive cycles, nothing leaks.** That is a
genuine positive result and the half a targeted session structurally cannot produce.
--- ---
## 7. I1–I11 across cycles ## 7. I1–I11 across cycles
Full table in §4b. **I1, I2 (and both pairs), I3, I4, I5, I6, I7, I10 and I11 held over 27 consecutive Full tables in §4b. Aggregated over run 1 (27 cycles) and run 2b (39 cycles):
cycles — 586 checks, 0 violations.** No violation was deterministic and none was racy, because there
were none; the honest statement is that the run found no drift **within 27 cycles**, not that none
exists beyond them.
**I8 was not checked** — Tier 3 never ran (§3). **I9 was not automated** — see §8. - **I7 — 66 restores, 66 correct discriminators, 0 stale, 0 empty.** The campaign's headline.
- **I2, I3, I4, I5, I6, I10, I11 — zero violations in either pass**, including the abort-in-place
variants and 195 + 135 secret-class assertions covering both sides of D5's split.
- **I1 — 2 violations in 39 cycles**, one self-recovering 3-cycle window under cycling far faster than
a real box sees (§5.3, disproved as a defect).
- **I8 — never checked.** Tier 3 could not be isolated (§3).
- **I9 — not automated.** Cited from the tester-gate run on this same controller 0.188.0, not
re-claimed as this campaign's work.
**A clean invariant is a result**: I2's genericity, I1's discrimination, both pairings, I4's
no-restart repair and I7's discriminator held across **66 cycles** of detach, reattach, abort,
backup, restore, kill, redeploy, reboot and hard reset.
--- ---
## 8. What did not run, and why ## 8. What did not run, and why
Phase A overran badly (~5.5 h against the brief's ~1 h) — 1.26.1 is a public release image with no **Atom coverage: 12 of the brief's ~13 families ran.** Run 1 covered 6; run 2b added the rest.
auto-install path, so day-0 was a blind screendump-and-`sendkey` walk; the Tier-3 question (§3) had to
be settled before a customer could exist at all, since `dr_tier` is fixed at creation; and a false
start on drive enrolment had to be backed out. **Phase B was therefore deliberately not launched at
04:00** — the brief's own fence, *"a test rig producing false negatives is worse than no rig"* — and
was built and run the next morning instead, where its first cycle could be watched.
**Atom coverage: 6 of the brief's ~12 suggested families.** Stated plainly rather than implied:
| Atom (brief B1) | Ran | | Atom (brief B1) | Ran |
|---|---| |---|---|
| back up Tier 1/2 · restore + verify by discriminator · detach/reattach **target** · detach/reattach **non-target** · delete an app and redeploy · reboot the guest | **YES** | | back up Tier 1/2 · restore + verify by discriminator · detach/reattach target · detach/reattach non-target · delete and redeploy · reboot the guest | **YES** (run 1 + 2b) |
| **abort a filesystem in place** (R-117 Q7) · **kill the agent mid-backup** · **hard-reset the VM mid-write** · **reboot the VM** · **concurrent backup+restore** · **concurrent backup+detach** · **fill a drive near-full** | **YES** (run 2b) |
| back up / restore **Tier 3** | **NO** — §3, structurally impossible to isolate | | back up / restore **Tier 3** | **NO** — §3, structurally impossible to isolate |
| abort a filesystem **in place** without the device disappearing (R-117 Q7) | **NO** |
| kill the **agent** mid-backup · **hard-reset the VM** mid-write · reboot the **VM** | **NO** |
| **concurrent** backup+restore · **concurrent** backup+detach | **NO** |
| fill a drive to near-full and continue | **NO** |
**`kill_controller` is not "mid-backup".** It restarts the controller at a permuted point in the **Still honest about two limits.** `kill_controller` is not literally "mid-backup" — it lands at a
cycle, so it sometimes lands after a backup and sometimes nowhere near one. It exercises recovery, not permuted point, so it exercises recovery rather than the mid-write race; the dedicated
the mid-write race the brief asked for. Recorded because the atom name would otherwise overclaim. `concurrent backup+detach` atom (`I1-under-load`, 5/5 PASS) covers the mid-operation case properly.
And **I9** — a Tier-1/2 restore with the guest's `app.yaml` moved aside — was **not automated**; it
was proven on this same controller 0.188.0 by the tester-gate run and is cited, not re-claimed.
**`reboot_guest` was appended after the shuffle, not permuted into it**, so it always ran last in **Depth reached: 39 consecutive cycles**, past the brief's "the state that quietly drifts on the
cycles 7/14/21 and **never interleaved with a detach**. Reboot-during-detach is untested. thirty-eighth". Cycles 34–39 were clean on every invariant. Beyond 39 is untested, not proven clean.
**I8** — not checked; Tier 3 never ran. **I9** (Tier-1/2 restore with the guest's `app.yaml` moved
aside, data read over the app's real network path) — **not automated**. It was proven on this exact
controller 0.188.0 by the tester-gate run hours earlier
(`tester-gate-golden-0.188.0-2026-07-31.md` §5, step 6), and that is cited rather than re-claimed:
this campaign did not execute it.
**The tail is untested, not clean.** 27 cycles is the count; the brief's interest in "the state that
quietly drifts on the thirty-eighth" is not answered by a 27-cycle run.
--- ---