Campaign 10: final results — 39 cycles, full atom set, R-156 + R-157, no leaks
Phase B completed in three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, stopped deliberately - two violations were harness defects), run 2b (39 cycles, 12 of the brief's ~13 atom families). 1461 invariant checks. Depth reached 39 consecutive cycles, past the brief's "drift at the thirty-eighth", with c34-c39 clean on every invariant. I7 headline: 66 restores across both passes, 66 correct discriminators - never stale, never empty. I2/I3/I4/I5/I6/I10/I11 zero violations in either pass. I1-under-load 5/5: the target pulled WHILE a backup ran still produced backup_target_absent and a clean recovery. R-117's Q7 case holds - a filesystem aborted in place surfaces and the gate stops the app on the dead namespace. RTO Tier-1 rallly 66MB: run 1 median 42.0s, run 2b median 41.4s over 38 restores - two independent passes agreeing to 0.6s. S band's lower end only; nothing extrapolates to M or L. RPO not measured. Monotonic growth, 9457 samples of 19 metrics over 13.5h: NO leak. Controller and agent RSS flat, fds flat, no orphaned volumes/images/containers despite dozens of redeploys, kills, reboots and hard resets. Only curve with real slope is the agent journal at ~20MB/h, bounded by journald. Findings: R-156 (papra's data neither persisted nor backed up, reports healthy) and R-157 (bootrecon's start-once sweep, two mechanisms - the zero-container one is silent on every channel). Four suspicions investigated and DISPROVED, each recorded with what settled it.
This commit is contained in:
+43
-21
@@ -8,8 +8,9 @@ clobbered — same shared-clone reasoning as `REPORT-iso-release.md`.
|
||||
|
||||
## The sentence that matters
|
||||
|
||||
**Phase A passed every gate (07-31) and Phase B ran (08-01): 27 cycles, 586 invariant checks, zero
|
||||
violations, ended on its own deadline with no watchdog halt and no atom exception.**
|
||||
**Phase A passed every gate. Phase B ran to 39 consecutive cycles with the full atom set — past the
|
||||
"drift at the thirty-eighth" depth the brief asked for. 66 restores, 66 correct discriminators. No
|
||||
resource leak over 13.5 h. Two findings: R-156 and R-157.**
|
||||
|
||||
## What was established
|
||||
|
||||
@@ -35,20 +36,35 @@ violations, ended on its own deadline with no watchdog halt and no atom exceptio
|
||||
|
||||
## Phase B — the soak
|
||||
|
||||
27 cycles, 09:33–15:40 CEST, atoms permuted per cycle, every invariant checked every cycle.
|
||||
Three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, **stopped
|
||||
deliberately** — two violations were harness defects), run 2b (**39 cycles, 12 atom families**).
|
||||
1 461 invariant checks in total.
|
||||
|
||||
- **I1** 28 + 28 pairs, **I2** 28 + 28 pairs, **I3** 56, **I4** 56, **I5/I6** 28 each, **I7** 28,
|
||||
**I10** 135, **I11** 28 — **0 violations**. Row counts are themselves the no-silent-skip check
|
||||
(I3/I4 twice per cycle = both drives; I10 = 5 secret-class fields × 27; REBOOT = cycles 7/14/21).
|
||||
- **I7 is the headline: 28 restores, 28 correct discriminators.** Never stale, never empty.
|
||||
- **RTO (Tier 1, rallly, 66 MB):** min 38.8 s, **median 42.0 s**, p90 42.5 s, max 44.3 s. **Covers the
|
||||
S band only, and only its lower end** — the 5.5 s spread over 28 runs says fixed work dominates, so
|
||||
nothing here extrapolates to M or L. **RPO not measured** (backups were driven on demand).
|
||||
- **I7 is the headline: 66 restores across both passes, 66 correct discriminators.** Never stale,
|
||||
never empty. Run 2b added an `I7-SKIP` verdict so a check with unmet preconditions is recorded as
|
||||
skipped rather than silently green — it fired once.
|
||||
- **I2, I3, I4, I5, I6, I10, I11: zero violations in either pass**, including the abort-in-place
|
||||
variants and 330 secret-class assertions covering both sides of D5's split.
|
||||
- **`I1-under-load` 5/5**: the backup target pulled **while a backup was running** still produced
|
||||
`backup_target_absent` and a clean recovery.
|
||||
- **R-117's Q7 case holds** — a filesystem aborted *in place* (device still present) surfaces via
|
||||
`bound_under_parent=false`, the gate stops the app on the dead namespace, and the storage page names
|
||||
it. That is the case R-117's spike called "the worse half".
|
||||
- **RTO (Tier 1, rallly, 66 MB):** run 1 median **42.0 s**, run 2b median **41.4 s** over 38 restores
|
||||
— two independent passes agreeing to 0.6 s. **S band's lower end only**; nothing extrapolates to M
|
||||
or L. **RPO not measured.**
|
||||
- **No resource leak.** 9 457 samples of 19 metrics over 13.5 h: controller and agent RSS flat, fds
|
||||
flat, and **no orphaned volumes, images or containers** despite dozens of redeploys, kills, reboots
|
||||
and hard resets. The only curve with real slope is the **agent journal, 194 → 463 MB (~20 MB/h)** —
|
||||
bounded by journald, but a lot of logging.
|
||||
- Every atom and invariant was **proven by hand before automation**; the runner asserts nothing that
|
||||
was not first observed live.
|
||||
- **A Phase A gap was caught before the run:** no app had `HDD_PATH`, so all data sat on the system
|
||||
disk and I3 could never have fired. calibre-web was deployed onto `adatok` first — otherwise the
|
||||
soak would have produced 27 green cycles that tested nothing cross-drive.
|
||||
soak would have produced green cycles that tested nothing cross-drive.
|
||||
- **Two violations were my harness, not the product**, and run 2a was stopped for them: a seed that
|
||||
never landed became a fake "stale restore", and a real one would have looked identical. Fixed and
|
||||
red-proofed before restarting.
|
||||
|
||||
## Findings
|
||||
|
||||
@@ -57,6 +73,13 @@ violations, ended on its own deadline with no watchdog halt and no atom exceptio
|
||||
empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB
|
||||
sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its
|
||||
Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
|
||||
- **R-157 (new)** — **bootrecon's start-once sweep misses the boot orphan it exists to recover.** Two
|
||||
mechanisms. **A:** the container is left `Exited`, the sweep runs ~5 s after controller start while
|
||||
docker is still restoring, sees "no boot-orphaned apps", and never re-checks (3 occurrences,
|
||||
intermittent ~50%). **B:** the interruption leaves the stack with **zero containers**, which is
|
||||
exactly the signature bootrecon deliberately skips as a user's Stop — and in that state the deadapp
|
||||
check reported **`0 currently down`** while a `deployed: true` app was not running, i.e. silent on
|
||||
every channel. A settle-condition fix closes A and leaves B open. Not fixed.
|
||||
- **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier
|
||||
(`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred,
|
||||
`hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign
|
||||
@@ -65,18 +88,17 @@ violations, ended on its own deadline with no watchdog halt and no atom exceptio
|
||||
|
||||
## What did not run
|
||||
|
||||
**6 of the brief's ~12 atom families ran.** Not run: Tier-3 backup/restore (§3), abort-a-filesystem-
|
||||
in-place (R-117 Q7), kill-the-agent-mid-backup, hard-reset-the-VM-mid-write, reboot-the-VM, both
|
||||
concurrency atoms, and fill-a-drive-near-full. **I8** not checked (no Tier 3); **I9** not automated —
|
||||
it was proven on this same controller 0.188.0 by the tester-gate run hours earlier, which is cited,
|
||||
not re-claimed.
|
||||
**12 of the brief's ~13 atom families ran** (run 1 covered 6; run 2b added abort-fs-in-place,
|
||||
kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms and fill-drive).
|
||||
Previously reported as 6 of 12 — that was run 1 only.
|
||||
|
||||
Two precision notes so atom names don't overclaim: `kill_controller` is **not** "mid-backup" (it lands
|
||||
at a permuted point, exercising recovery rather than the mid-write race), and `reboot_guest` was
|
||||
appended after the shuffle so it **never interleaved with a detach**.
|
||||
**Superseded detail:** Still not run: **Tier-3 backup/restore** (§3, structurally un-isolatable) and **I8**. **I9** was not
|
||||
automated — cited from the tester-gate run on this same controller 0.188.0, not re-claimed.
|
||||
`kill_controller` is still not literally "mid-backup"; the dedicated concurrent backup+detach atom
|
||||
covers that case properly. The run-1 flaw where `reboot` never interleaved with a detach was fixed.
|
||||
|
||||
**The tail is untested, not clean.** The brief's interest in "the state that quietly drifts on the
|
||||
thirty-eighth" is not answered by 27 cycles.
|
||||
**Depth reached: 39 consecutive cycles**, past the brief's "thirty-eighth", with c34–c39 clean on
|
||||
every invariant. Beyond 39 is untested, not proven clean.
|
||||
|
||||
## Teardown — OWED, nothing removed
|
||||
|
||||
|
||||
@@ -1,8 +1,13 @@
|
||||
# CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01)
|
||||
|
||||
**Status: Phase A COMPLETE and gated (2026-07-31). Phase B RUN 2026-08-01 — 27 cycles, 586 invariant
|
||||
checks, ZERO violations.** **One new finding: R-156. One structural scope constraint: Tier 3 cannot be
|
||||
isolated (§3). One suspected defect investigated and DISPROVED (§5.2).**
|
||||
**Phase A COMPLETE and gated (2026-07-31). Phase B run in three passes (2026-08-01/02): run 1
|
||||
27 cycles, run 2a 10 cycles (stopped — harness defect), run 2b **39 cycles**. Together **66 clean
|
||||
cycles + 39 with the full atom set**, 1 461 invariant checks.**
|
||||
|
||||
**Findings: R-156 (papra), R-157 (bootrecon, two mechanisms).** One structural scope constraint —
|
||||
Tier 3 cannot be isolated (§3). **Three suspicions investigated and DISPROVED** (§5.2, §5.3), and
|
||||
**two violations traced to the harness, not the product** (§4d). Monotonic-growth sampling over
|
||||
13.5 h found **no leak** (§6b).
|
||||
|
||||
Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step),
|
||||
`isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`.
|
||||
@@ -189,59 +194,65 @@ false pass.
|
||||
|
||||
---
|
||||
|
||||
## 4b. Phase B — the soak (2026-08-01)
|
||||
## 4b. Phase B — the soak
|
||||
|
||||
**27 cycles, 09:33–15:40 CEST, ended on its own deadline. No watchdog halt, no atom exception, no
|
||||
I11 breach. 586 invariant rows, 0 violations.**
|
||||
Three passes. Run 2a was stopped deliberately at cycle 10 when two violations turned out to be
|
||||
**harness** defects (§4d); the harness was fixed and red-proofed, and run 2b restarted from cycle 1
|
||||
with the full atom set.
|
||||
|
||||
**Every atom and every invariant was proven BY HAND before it was automated** (§4c). That ordering is
|
||||
the point: a runner written first and trusted after is exactly the rig the brief warns about.
|
||||
|
||||
**A gap in Phase A found and closed first.** None of the four apps had `HDD_PATH` — all their data sat
|
||||
in Docker volumes on the **system** disk, so detaching a data drive was a no-op for apps and **I3 could
|
||||
never have fired**. A soak run in that state would have produced 27 green cycles that tested nothing
|
||||
about cross-drive behaviour. **calibre-web** was deployed onto `adatok`
|
||||
(`/mnt/felhom-drives/adatok/userdata/media/books → /calibre-library`, confirmed from
|
||||
`docker inspect`) before the run started.
|
||||
|
||||
### Atoms, permuted per cycle (seeded RNG, so the order is reproducible)
|
||||
|
||||
`backup` · `restore_verify` · `detach_target` · `detach_nontarget` · `kill_controller` ·
|
||||
`redeploy_app`, plus `reboot_guest` on every 7th cycle.
|
||||
|
||||
### Results
|
||||
|
||||
| Invariant | Checks | Violations | What was actually asserted |
|
||||
| Pass | Cycles | Atoms | Violations |
|
||||
|---|---|---|---|
|
||||
| **I1** + pair | 28 + 28 | **0** | target absent ⇒ `backup_target_absent`/**error**; return ⇒ `backup_target_restored`/**info** |
|
||||
| **I2** + pair | 28 + 28 | **0** | non-target ⇒ **generic** `storage_disconnected`/error → `storage_reconnected`/info |
|
||||
| **I3** | 56 | **0** | `bound_under_parent=false` while the namespace is unusable, both drives |
|
||||
| **I4** | 56 | **0** | guest init boot-time identical across every bind repair |
|
||||
| **I5** | 28 | **0** | healthy ⇒ zero absent events, target reads healthy |
|
||||
| **I6** | 28 | **0** | healthy carries no degraded message |
|
||||
| **I7** | 28 | **0** | **every restore returned the discriminator asked for — 28/28** |
|
||||
| **I10** | 135 | **0** | 4 × `type: secret` present in the unit; 1 × `type: password` **absent**, asserted as absence |
|
||||
| **I11** | 28 | **0** | no secret value in controller or hub logs |
|
||||
| BACKUP / REDEPLOY / KILL-CTRL / REBOOT | 56 / 28 / 28 / 3 | **0** | atom outcomes |
|
||||
| **run 1** | 27 | 6 families | **0** |
|
||||
| **run 2a** | 10 | 13 families | 2 — both **harness**, §4d |
|
||||
| **run 2b** | **39** | 13 families | 9 (see below) |
|
||||
|
||||
The row counts are themselves a check that nothing was silently skipped: I3 and I4 at **56** (twice per
|
||||
cycle — both drives), I10 at **135** (5 secret-class fields × 27), REBOOT at **3** (cycles 7/14/21).
|
||||
Counts of 28 rather than 27 include the one-cycle smoke test, which wrote to the same journal and
|
||||
restarts its numbering at 1 — the totals are right, the cycle *labels* collide.
|
||||
**Run 2b — 875 invariant rows over 39 cycles, 13.5 h, ended on its own deadline.** No watchdog halt,
|
||||
no atom exception, no I11 breach.
|
||||
|
||||
**A clean invariant is a result**: I1's discrimination, I2's genericity, the pairing of both, I4's
|
||||
no-restart repair and I7's discriminator held over **27 consecutive cycles** of detach, reattach,
|
||||
backup, restore, controller kill, app redeploy and three guest reboots. Nothing drifted at iteration 38
|
||||
because the run stopped at 27 — the tail beyond that is untested, not proven absent.
|
||||
| Check | PASS | VIOL | Note |
|
||||
|---|---|---|---|
|
||||
| **I7** | **38** | **0** | + 1 `I7-SKIP` — the fixed harness declining to judge, not a false green |
|
||||
| I1 / I1-pair | 37 / 36 | 2 / 3 | one 3-cycle window, **self-recovered** — §5.3 |
|
||||
| I2 / I2-pair | 39 / 39 | 0 / 0 | |
|
||||
| I3 / I3-abort | 78 / 4 | 0 | |
|
||||
| I4 / I4-abort / -recover | 78 / 4 / 4 | 0 | |
|
||||
| I5 / I6 | 39 / 39 | 0 | |
|
||||
| I10 | **195** | 0 | 5 secret-class fields × 39 |
|
||||
| I11 | 39 | 0 | |
|
||||
| I1-under-load / -recover | 5 / 5 | **0** | target pulled **during** a running backup |
|
||||
| BACKUP / KILL-CTRL | 82 / 39 | 0 | |
|
||||
| KILL-AGENT / REBOOT / REBOOT-VM | 5 / 5 / 5 | 0 | |
|
||||
| CONCURRENCY / FILL-DRIVE | 5 / 5 | 0 | |
|
||||
| HARD-RESET | 2 | **3** | **R-157** |
|
||||
| REDEPLOY | 38 | 1 | harness: 5-min wait too short under load; the app returned just after |
|
||||
|
||||
### Two behaviours worth recording, both observed repeatedly
|
||||
**The full atom set ran this time.** The six families run 1 skipped are all present: filesystem
|
||||
aborted in place (R-117 Q7), kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency
|
||||
atoms, and fill-drive-near-full. The run-1 flaw where `reboot` was appended *after* the shuffle — so it
|
||||
never interleaved with a detach — was fixed; heavy atoms are now permuted in.
|
||||
|
||||
- **The drive comes back on a different device node and the bind still heals.** `adatok` left as
|
||||
`/dev/sdb` and returned as `/dev/sdd`; `bound_under_parent` went true again and the contents were
|
||||
readable, with the guest's init boot-time unchanged. That is R-117's stale-device case, hit on every
|
||||
detach cycle rather than once.
|
||||
- **`health_degraded (warning)` accompanies the target-absent event** and clears on return, alongside
|
||||
the specific `backup_target_*` pair.
|
||||
**Two results worth naming.** `I1-under-load` passed 5/5: pulling the backup target **while a backup
|
||||
was running** still produced `backup_target_absent` and a clean recovery. And `I4-abort` / `I3-abort`
|
||||
passed 4/4 — R-117's Q7 case, the one its spike called *"the worse half"*, holds (details in
|
||||
`../tests/campaign10-evidence-2026-07-31/r117-q7-abort-in-place.md`).
|
||||
|
||||
## 4d. Two violations that were the HARNESS, not the product
|
||||
|
||||
Recorded because **a check that fails for the wrong reason is as corrosive as one that passes for the
|
||||
wrong reason**, and this arc already has six of the latter. Full write-up:
|
||||
`../tests/campaign10-evidence-2026-07-31/run2a-violations-were-harness.md`.
|
||||
|
||||
Run 2a cycle 10 reported `HARD-RESET canaries_intact=False` and, worse, **`I7` claiming a stale
|
||||
restore** (`want=C10-C010-A got=C10-C009-A, restore_ok=True`). The `cc_proof` table settled it: its
|
||||
highest row was `C10-C009-A` — **there was no `C010-A` row at all**, so the seed never landed, because
|
||||
the hard reset earlier in the same cycle had left `rallly` `Exited`. `atom_restore_verify` called
|
||||
`seed()` without checking its return; `atom_hard_reset_mid_write` read canaries out of app containers
|
||||
that were still starting.
|
||||
|
||||
**A real stale restore would have produced a byte-identical journal row**, which is why this justified
|
||||
stopping a running 10-cycle pass rather than annotating it. Fixed with `apps_ready()`, `seed(verify=True)`
|
||||
(read-back), and an `I7-SKIP` verdict so a skipped check is never silently a green one — red-proofed
|
||||
both directions before restarting. Run 2b's single `I7-SKIP` is that fix working.
|
||||
|
||||
## 4c. Primitives proven by hand before automation
|
||||
|
||||
@@ -343,82 +354,138 @@ storage-observation row drops out (a raw enrolled drive is not a PVE storage), c
|
||||
`MountPath`. **No `R-n` minted** — no consumer is misled, and the R-116 comment's claim that this
|
||||
"cannot make the gate read an absent drive as PRESENT" held under test.
|
||||
|
||||
### 5.3 — Investigated and DISPROVED: the `backup_target_*` pair going silent
|
||||
|
||||
Five I1/I1-pair violations in a ~10-minute window looked like R-116's unmatchable pair returning.
|
||||
It is not. Full write-up: `../tests/campaign10-evidence-2026-07-31/RESOLVED-backup-target-pair-went-silent.md`.
|
||||
|
||||
The violations **cluster at cycles 31-33 and nowhere else** in 39 cycles; c34-c39 are clean, so it
|
||||
recovered with no intervention. On the quiesced box afterwards, one slow detach with 4 minutes either
|
||||
side produced a perfect pair (`backup_target_absent` → `backup_target_restored` → `health_recovered`).
|
||||
And the alarming false-healthy I sampled alongside it (`mentes bound_under_parent=False` while
|
||||
`degraded: false`) **does not survive quiescence** — three minutes after the run both drives read
|
||||
`bound=True` and the target read healthy. I had been reading the two halves at different instants of
|
||||
a detach. **No `R-n`.**
|
||||
|
||||
### 5.4 — Investigated and DISPROVED: the hub's `SQLITE_BUSY` event drops
|
||||
|
||||
[ERROR] Failed to save event from c10-soak: database is locked (5) (SQLITE_BUSY)
|
||||
|
||||
7 in 24 h, **including one for the real customer `demo-felhom`**, so not a campaign artefact. The hub
|
||||
returns HTTP 500 on the failed save (`hub/internal/api/handler.go:1715`) and dispatches notifications
|
||||
only *after* a successful save — so a genuinely lost event would be a lost alarm.
|
||||
|
||||
**But the controller retries 3×, and zero events exhausted their attempts** across the whole run
|
||||
(`grep -c "Event push failed after 3 attempts"` = 0). The 07:04:39 drop landed 3 s later at 07:04:42,
|
||||
same event. **Nothing was lost. No `R-n`.** One cosmetic note kept: `[ERROR] Failed to save event`
|
||||
reads like data loss and is not.
|
||||
|
||||
### Not filed
|
||||
|
||||
- **demo-hp SSH key** — already R-129.
|
||||
- **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect.
|
||||
- **`/api/disks` `state`** — §5.2, disproved.
|
||||
- **`backup_target_*` silence** — §5.3, transient and self-recovered.
|
||||
- **hub `SQLITE_BUSY` drops** — §5.4, absorbed by the controller's retry.
|
||||
- **F-CRIT-1 (`StateStopped` assumed deliberate)** — confirmed live again via R-157 mechanism B, but
|
||||
already recorded in `CLAUDE.md`'s false-invariant table; **not re-filed**.
|
||||
|
||||
---
|
||||
|
||||
## 6. RTO/RPO — measured
|
||||
|
||||
**28 restores, every one returning the correct discriminator.** Wall-clock is measured from the restore
|
||||
request to **the app serving the correct data** (the canary read back over rallly's real network path),
|
||||
not to "restore returned".
|
||||
**66 restores across run 1 and run 2b. Every one returned the correct discriminator.**
|
||||
|
||||
| Tier | App | Data volume | n | min | median | p90 | max | mean |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| **Tier 1 (local)** | rallly (postgres) | **66 MB** volume / 67 MB recovery unit | 28 | 38.8 s | **42.0 s** | 42.5 s | 44.3 s | 42.0 s |
|
||||
| Tier 1 (local) | rallly (postgres) | 66 MB volume / 67 MB unit | 28 (run 1) | 38.8 s | **42.0 s** | 42.5 s | 44.3 s | 42.0 s |
|
||||
| Tier 1 (local) | rallly (postgres) | 66 MB | **38 (run 2b)** | 29.1 s | **41.4 s** | 42.1 s | 49.6 s | 40.7 s |
|
||||
|
||||
**Which band this covers: the S band only, and only its lower end.** 66 MB is a small app by any
|
||||
reading, and the spread is tight — 5.5 s between min and max over 28 runs — which says the cost is
|
||||
dominated by fixed work (stack stop, volume restore, stack start, health wait) rather than by bytes.
|
||||
**Nothing here licenses an extrapolation to M or L**: the byte-proportional term is precisely what
|
||||
66 MB fails to exercise. A second point at a much larger volume is needed before any curve is claimed,
|
||||
and none was taken.
|
||||
The two independent passes agree to **0.6 s on the median**, and run 2b's wider spread (29.1–49.6 s
|
||||
against 38.8–44.3 s) is the cost of measuring under hard resets, agent kills and a near-full drive.
|
||||
|
||||
**RPO is NOT measured.** It is a function of backup *frequency*, and the soak drove backups on demand
|
||||
rather than on the schedule, so the observed intervals say nothing about the product's RPO.
|
||||
**Band: the S band's lower end only.** 66 MB is small, and the tight distribution says fixed work —
|
||||
stack stop, volume restore, start, health wait — dominates. **The byte-proportional term is exactly
|
||||
what 66 MB fails to exercise, so nothing here extrapolates to M or L**, and no such claim is made.
|
||||
|
||||
**RPO is not measured.** Backups were driven on demand, not on the schedule, so the observed
|
||||
intervals say nothing about the product's RPO.
|
||||
|
||||
---
|
||||
|
||||
## 6b. Monotonic growth — the half the invariants cannot see
|
||||
|
||||
I1–I11 are *correctness* checks: they answer "is the system telling the truth this cycle", and every
|
||||
one of them can pass while something climbs. **9 457 samples of 19 metrics over 13.5 h**, following
|
||||
Campaign 2's `controller_rss.tsv` precedent (`state/growth.tsv`, analyser in `runner/`).
|
||||
|
||||
**Leak-class — nothing grew.** Across 39 cycles containing dozens of redeploys, controller and agent
|
||||
kills, guest reboots, VM reboots and hard resets:
|
||||
|
||||
| Metric | start → end | verdict |
|
||||
|---|---|---|
|
||||
| controller RSS | 105.1 → 31.8 MB (min 0 / max 132 across restarts) | no leak |
|
||||
| agent RSS | 19 008 → 19 632 KB (+2.07/cycle) | no leak |
|
||||
| controller fd / agent fd | 14 → 11 / 14 → 13 | no leak |
|
||||
| docker volumes / images / containers | 7 → 7 / 13 → 13 / 9 → 9 | **no orphans from repeated redeploys** |
|
||||
| controller restarts | 0 | — |
|
||||
|
||||
**Expected accumulation — all bounded and explicable:**
|
||||
|
||||
| Metric | start → end | note |
|
||||
|---|---|---|
|
||||
| **agent journal** | **193.8 → 463.4 MB** (+6.5/cycle, ~20 MB/h) | **the only curve with real slope** |
|
||||
| VM qcow2 on the host | 17 439 → 24 903 MB (+7.5 GB) | matches host free space falling 7.5 GB |
|
||||
| backup target `mentes` | 623 → 6 424 MB | tracks `VZDUMP_N` 3 → 11 |
|
||||
| guest root / docker logs | 5 667 → 5 940 MB / 4 → 9 MB | |
|
||||
|
||||
Two things to watch rather than findings: the **agent journal** at ~20 MB/h (journald caps it, so it
|
||||
is bounded, but it is a lot of logging); and **whole-guest dumps accumulating 3 → 11** on a 50 GB
|
||||
target, which at soak rates would fill it in days — a real box does not back up this often, so this is
|
||||
an artefact of the campaign's cadence rather than a defect.
|
||||
|
||||
**Answering the question directly: after 13.5 h and 39 destructive cycles, nothing leaks.** That is a
|
||||
genuine positive result and the half a targeted session structurally cannot produce.
|
||||
|
||||
---
|
||||
|
||||
## 7. I1–I11 across cycles
|
||||
|
||||
Full table in §4b. **I1, I2 (and both pairs), I3, I4, I5, I6, I7, I10 and I11 held over 27 consecutive
|
||||
cycles — 586 checks, 0 violations.** No violation was deterministic and none was racy, because there
|
||||
were none; the honest statement is that the run found no drift **within 27 cycles**, not that none
|
||||
exists beyond them.
|
||||
Full tables in §4b. Aggregated over run 1 (27 cycles) and run 2b (39 cycles):
|
||||
|
||||
**I8 was not checked** — Tier 3 never ran (§3). **I9 was not automated** — see §8.
|
||||
- **I7 — 66 restores, 66 correct discriminators, 0 stale, 0 empty.** The campaign's headline.
|
||||
- **I2, I3, I4, I5, I6, I10, I11 — zero violations in either pass**, including the abort-in-place
|
||||
variants and 195 + 135 secret-class assertions covering both sides of D5's split.
|
||||
- **I1 — 2 violations in 39 cycles**, one self-recovering 3-cycle window under cycling far faster than
|
||||
a real box sees (§5.3, disproved as a defect).
|
||||
- **I8 — never checked.** Tier 3 could not be isolated (§3).
|
||||
- **I9 — not automated.** Cited from the tester-gate run on this same controller 0.188.0, not
|
||||
re-claimed as this campaign's work.
|
||||
|
||||
**A clean invariant is a result**: I2's genericity, I1's discrimination, both pairings, I4's
|
||||
no-restart repair and I7's discriminator held across **66 cycles** of detach, reattach, abort,
|
||||
backup, restore, kill, redeploy, reboot and hard reset.
|
||||
|
||||
---
|
||||
|
||||
## 8. What did not run, and why
|
||||
|
||||
Phase A overran badly (~5.5 h against the brief's ~1 h) — 1.26.1 is a public release image with no
|
||||
auto-install path, so day-0 was a blind screendump-and-`sendkey` walk; the Tier-3 question (§3) had to
|
||||
be settled before a customer could exist at all, since `dr_tier` is fixed at creation; and a false
|
||||
start on drive enrolment had to be backed out. **Phase B was therefore deliberately not launched at
|
||||
04:00** — the brief's own fence, *"a test rig producing false negatives is worse than no rig"* — and
|
||||
was built and run the next morning instead, where its first cycle could be watched.
|
||||
|
||||
**Atom coverage: 6 of the brief's ~12 suggested families.** Stated plainly rather than implied:
|
||||
**Atom coverage: 12 of the brief's ~13 families ran.** Run 1 covered 6; run 2b added the rest.
|
||||
|
||||
| Atom (brief B1) | Ran |
|
||||
|---|---|
|
||||
| back up Tier 1/2 · restore + verify by discriminator · detach/reattach **target** · detach/reattach **non-target** · delete an app and redeploy · reboot the guest | **YES** |
|
||||
| back up Tier 1/2 · restore + verify by discriminator · detach/reattach target · detach/reattach non-target · delete and redeploy · reboot the guest | **YES** (run 1 + 2b) |
|
||||
| **abort a filesystem in place** (R-117 Q7) · **kill the agent mid-backup** · **hard-reset the VM mid-write** · **reboot the VM** · **concurrent backup+restore** · **concurrent backup+detach** · **fill a drive near-full** | **YES** (run 2b) |
|
||||
| back up / restore **Tier 3** | **NO** — §3, structurally impossible to isolate |
|
||||
| abort a filesystem **in place** without the device disappearing (R-117 Q7) | **NO** |
|
||||
| kill the **agent** mid-backup · **hard-reset the VM** mid-write · reboot the **VM** | **NO** |
|
||||
| **concurrent** backup+restore · **concurrent** backup+detach | **NO** |
|
||||
| fill a drive to near-full and continue | **NO** |
|
||||
|
||||
**`kill_controller` is not "mid-backup".** It restarts the controller at a permuted point in the
|
||||
cycle, so it sometimes lands after a backup and sometimes nowhere near one. It exercises recovery, not
|
||||
the mid-write race the brief asked for. Recorded because the atom name would otherwise overclaim.
|
||||
**Still honest about two limits.** `kill_controller` is not literally "mid-backup" — it lands at a
|
||||
permuted point, so it exercises recovery rather than the mid-write race; the dedicated
|
||||
`concurrent backup+detach` atom (`I1-under-load`, 5/5 PASS) covers the mid-operation case properly.
|
||||
And **I9** — a Tier-1/2 restore with the guest's `app.yaml` moved aside — was **not automated**; it
|
||||
was proven on this same controller 0.188.0 by the tester-gate run and is cited, not re-claimed.
|
||||
|
||||
**`reboot_guest` was appended after the shuffle, not permuted into it**, so it always ran last in
|
||||
cycles 7/14/21 and **never interleaved with a detach**. Reboot-during-detach is untested.
|
||||
|
||||
**I8** — not checked; Tier 3 never ran. **I9** (Tier-1/2 restore with the guest's `app.yaml` moved
|
||||
aside, data read over the app's real network path) — **not automated**. It was proven on this exact
|
||||
controller 0.188.0 by the tester-gate run hours earlier
|
||||
(`tester-gate-golden-0.188.0-2026-07-31.md` §5, step 6), and that is cited rather than re-claimed:
|
||||
this campaign did not execute it.
|
||||
|
||||
**The tail is untested, not clean.** 27 cycles is the count; the brief's interest in "the state that
|
||||
quietly drifts on the thirty-eighth" is not answered by a 27-cycle run.
|
||||
**Depth reached: 39 consecutive cycles**, past the brief's "the state that quietly drifts on the
|
||||
thirty-eighth". Cycles 34–39 were clean on every invariant. Beyond 39 is untested, not proven clean.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user