admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Campaign 10: final results — 39 cycles, full atom set, R-156 + R-157, no leaks

Browse Source 
Phase B completed in three passes: run 1 (27 cycles, 6 atom families, 0
violations), run 2a (10 cycles, stopped deliberately - two violations were harness
defects), run 2b (39 cycles, 12 of the brief's ~13 atom families). 1461 invariant
checks. Depth reached 39 consecutive cycles, past the brief's "drift at the
thirty-eighth", with c34-c39 clean on every invariant.

I7 headline: 66 restores across both passes, 66 correct discriminators - never
stale, never empty. I2/I3/I4/I5/I6/I10/I11 zero violations in either pass.
I1-under-load 5/5: the target pulled WHILE a backup ran still produced
backup_target_absent and a clean recovery. R-117's Q7 case holds - a filesystem
aborted in place surfaces and the gate stops the app on the dead namespace.

RTO Tier-1 rallly 66MB: run 1 median 42.0s, run 2b median 41.4s over 38 restores -
two independent passes agreeing to 0.6s. S band's lower end only; nothing
extrapolates to M or L. RPO not measured.

Monotonic growth, 9457 samples of 19 metrics over 13.5h: NO leak. Controller and
agent RSS flat, fds flat, no orphaned volumes/images/containers despite dozens of
redeploys, kills, reboots and hard resets. Only curve with real slope is the agent
journal at ~20MB/h, bounded by journald.

Findings: R-156 (papra's data neither persisted nor backed up, reports healthy)
and R-157 (bootrecon's start-once sweep, two mechanisms - the zero-container one
is silent on every channel). Four suspicions investigated and DISPROVED, each
recorded with what settled it.
This commit is contained in:
authentik Default Admin
2026-08-02 07:13:04 +02:00
parent 405a795e32
commit 7ba7c2a271
2 changed files with 203 additions and 114 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md
+160 -93
View File
@@ -1,8 +1,13 @@
# CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01)
**Status: Phase A COMPLETE and gated (2026-07-31). Phase B RUN 2026-08-01 — 27 cycles, 586 invariant
checks, ZERO violations.** **One new finding: R-156. One structural scope constraint: Tier 3 cannot be
isolated (§3). One suspected defect investigated and DISPROVED (§5.2).**
**Phase A COMPLETE and gated (2026-07-31). Phase B run in three passes (2026-08-01/02): run 1
27 cycles, run 2a 10 cycles (stopped — harness defect), run 2b **39 cycles**. Together **66 clean
cycles + 39 with the full atom set**, 1 461 invariant checks.**
**Findings: R-156 (papra), R-157 (bootrecon, two mechanisms).** One structural scope constraint —
Tier 3 cannot be isolated (§3). **Three suspicions investigated and DISPROVED** (§5.2, §5.3), and
**two violations traced to the harness, not the product** (§4d). Monotonic-growth sampling over
13.5 h found **no leak** (§6b).
Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step),
`isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`.
@@ -189,59 +194,65 @@ false pass.
---
## 4b. Phase B — the soak (2026-08-01)
## 4b. Phase B — the soak
**27 cycles, 09:33–15:40 CEST, ended on its own deadline. No watchdog halt, no atom exception, no
I11 breach. 586 invariant rows, 0 violations.**
Three passes. Run 2a was stopped deliberately at cycle 10 when two violations turned out to be
**harness** defects (§4d); the harness was fixed and red-proofed, and run 2b restarted from cycle 1
with the full atom set.
**Every atom and every invariant was proven BY HAND before it was automated** (§4c). That ordering is
the point: a runner written first and trusted after is exactly the rig the brief warns about.
**A gap in Phase A found and closed first.** None of the four apps had `HDD_PATH` — all their data sat
in Docker volumes on the **system** disk, so detaching a data drive was a no-op for apps and **I3 could
never have fired**. A soak run in that state would have produced 27 green cycles that tested nothing
about cross-drive behaviour. **calibre-web** was deployed onto `adatok`
(`/mnt/felhom-drives/adatok/userdata/media/books → /calibre-library`, confirmed from
`docker inspect`) before the run started.
### Atoms, permuted per cycle (seeded RNG, so the order is reproducible)
`backup` · `restore_verify` · `detach_target` · `detach_nontarget` · `kill_controller` ·
`redeploy_app`, plus `reboot_guest` on every 7th cycle.
### Results
| Invariant | Checks | Violations | What was actually asserted |
| Pass | Cycles | Atoms | Violations |
|---|---|---|---|
| **I1** + pair | 28 + 28 | **0** | target absent ⇒ `backup_target_absent`/**error**; return ⇒ `backup_target_restored`/**info** |
| **I2** + pair | 28 + 28 | **0** | non-target ⇒ **generic** `storage_disconnected`/error → `storage_reconnected`/info |
| **I3** | 56 | **0** | `bound_under_parent=false` while the namespace is unusable, both drives |
| **I4** | 56 | **0** | guest init boot-time identical across every bind repair |
| **I5** | 28 | **0** | healthy ⇒ zero absent events, target reads healthy |
| **I6** | 28 | **0** | healthy carries no degraded message |
| **I7** | 28 | **0** | **every restore returned the discriminator asked for — 28/28** |
| **I10** | 135 | **0** | 4 × `type: secret` present in the unit; 1 × `type: password` **absent**, asserted as absence |
| **I11** | 28 | **0** | no secret value in controller or hub logs |
| BACKUP / REDEPLOY / KILL-CTRL / REBOOT | 56 / 28 / 28 / 3 | **0** | atom outcomes |
| **run 1** | 27 | 6 families | **0** |
| **run 2a** | 10 | 13 families | 2 — both **harness**, §4d |
| **run 2b** | **39** | 13 families | 9 (see below) |
The row counts are themselves a check that nothing was silently skipped: I3 and I4 at **56** (twice per
cycle — both drives), I10 at **135** (5 secret-class fields × 27), REBOOT at **3** (cycles 7/14/21).
Counts of 28 rather than 27 include the one-cycle smoke test, which wrote to the same journal and
restarts its numbering at 1 — the totals are right, the cycle *labels* collide.
**Run 2b — 875 invariant rows over 39 cycles, 13.5 h, ended on its own deadline.** No watchdog halt,
no atom exception, no I11 breach.
**A clean invariant is a result**: I1's discrimination, I2's genericity, the pairing of both, I4's
no-restart repair and I7's discriminator held over **27 consecutive cycles** of detach, reattach,
backup, restore, controller kill, app redeploy and three guest reboots. Nothing drifted at iteration 38
because the run stopped at 27 — the tail beyond that is untested, not proven absent.
| Check | PASS | VIOL | Note |
|---|---|---|---|
| **I7** | **38** | **0** | + 1 `I7-SKIP` — the fixed harness declining to judge, not a false green |
| I1 / I1-pair | 37 / 36 | 2 / 3 | one 3-cycle window, **self-recovered** — §5.3 |
| I2 / I2-pair | 39 / 39 | 0 / 0 | |
| I3 / I3-abort | 78 / 4 | 0 | |
| I4 / I4-abort / -recover | 78 / 4 / 4 | 0 | |
| I5 / I6 | 39 / 39 | 0 | |
| I10 | **195** | 0 | 5 secret-class fields × 39 |
| I11 | 39 | 0 | |
| I1-under-load / -recover | 5 / 5 | **0** | target pulled **during** a running backup |
| BACKUP / KILL-CTRL | 82 / 39 | 0 | |
| KILL-AGENT / REBOOT / REBOOT-VM | 5 / 5 / 5 | 0 | |
| CONCURRENCY / FILL-DRIVE | 5 / 5 | 0 | |
| HARD-RESET | 2 | **3** | **R-157** |
| REDEPLOY | 38 | 1 | harness: 5-min wait too short under load; the app returned just after |
### Two behaviours worth recording, both observed repeatedly
**The full atom set ran this time.** The six families run 1 skipped are all present: filesystem
aborted in place (R-117 Q7), kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency
atoms, and fill-drive-near-full. The run-1 flaw where `reboot` was appended *after* the shuffle — so it
never interleaved with a detach — was fixed; heavy atoms are now permuted in.
- **The drive comes back on a different device node and the bind still heals.** `adatok` left as
`/dev/sdb` and returned as `/dev/sdd`; `bound_under_parent` went true again and the contents were
readable, with the guest's init boot-time unchanged. That is R-117's stale-device case, hit on every
detach cycle rather than once.
- **`health_degraded (warning)` accompanies the target-absent event** and clears on return, alongside
the specific `backup_target_*` pair.
**Two results worth naming.** `I1-under-load` passed 5/5: pulling the backup target **while a backup
was running** still produced `backup_target_absent` and a clean recovery. And `I4-abort` / `I3-abort`
passed 4/4 — R-117's Q7 case, the one its spike called *"the worse half"*, holds (details in
`../tests/campaign10-evidence-2026-07-31/r117-q7-abort-in-place.md`).
## 4d. Two violations that were the HARNESS, not the product
Recorded because **a check that fails for the wrong reason is as corrosive as one that passes for the
wrong reason**, and this arc already has six of the latter. Full write-up:
`../tests/campaign10-evidence-2026-07-31/run2a-violations-were-harness.md`.
Run 2a cycle 10 reported `HARD-RESET canaries_intact=False` and, worse, **`I7` claiming a stale
restore** (`want=C10-C010-A got=C10-C009-A, restore_ok=True`). The `cc_proof` table settled it: its
highest row was `C10-C009-A` — **there was no `C010-A` row at all**, so the seed never landed, because
the hard reset earlier in the same cycle had left `rallly` `Exited`. `atom_restore_verify` called
`seed()` without checking its return; `atom_hard_reset_mid_write` read canaries out of app containers
that were still starting.
**A real stale restore would have produced a byte-identical journal row**, which is why this justified
stopping a running 10-cycle pass rather than annotating it. Fixed with `apps_ready()`, `seed(verify=True)`
(read-back), and an `I7-SKIP` verdict so a skipped check is never silently a green one — red-proofed
both directions before restarting. Run 2b's single `I7-SKIP` is that fix working.
## 4c. Primitives proven by hand before automation
@@ -343,82 +354,138 @@ storage-observation row drops out (a raw enrolled drive is not a PVE storage), c
`MountPath`. **No `R-n` minted** — no consumer is misled, and the R-116 comment's claim that this
"cannot make the gate read an absent drive as PRESENT" held under test.
### 5.3 — Investigated and DISPROVED: the `backup_target_*` pair going silent
Five I1/I1-pair violations in a ~10-minute window looked like R-116's unmatchable pair returning.
It is not. Full write-up: `../tests/campaign10-evidence-2026-07-31/RESOLVED-backup-target-pair-went-silent.md`.
The violations **cluster at cycles 31-33 and nowhere else** in 39 cycles; c34-c39 are clean, so it
recovered with no intervention. On the quiesced box afterwards, one slow detach with 4 minutes either
side produced a perfect pair (`backup_target_absent` → `backup_target_restored` → `health_recovered`).
And the alarming false-healthy I sampled alongside it (`mentes bound_under_parent=False` while
`degraded: false`) **does not survive quiescence** — three minutes after the run both drives read
`bound=True` and the target read healthy. I had been reading the two halves at different instants of
a detach. **No `R-n`.**
### 5.4 — Investigated and DISPROVED: the hub's `SQLITE_BUSY` event drops
[ERROR] Failed to save event from c10-soak: database is locked (5) (SQLITE_BUSY)
7 in 24 h, **including one for the real customer `demo-felhom`**, so not a campaign artefact. The hub
returns HTTP 500 on the failed save (`hub/internal/api/handler.go:1715`) and dispatches notifications
only *after* a successful save — so a genuinely lost event would be a lost alarm.
**But the controller retries 3×, and zero events exhausted their attempts** across the whole run
(`grep -c "Event push failed after 3 attempts"` = 0). The 07:04:39 drop landed 3 s later at 07:04:42,
same event. **Nothing was lost. No `R-n`.** One cosmetic note kept: `[ERROR] Failed to save event`
reads like data loss and is not.
### Not filed
- **demo-hp SSH key** — already R-129.
- **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect.
- **`/api/disks` `state`** — §5.2, disproved.
- **`backup_target_*` silence** — §5.3, transient and self-recovered.
- **hub `SQLITE_BUSY` drops** — §5.4, absorbed by the controller's retry.
- **F-CRIT-1 (`StateStopped` assumed deliberate)** — confirmed live again via R-157 mechanism B, but
already recorded in `CLAUDE.md`'s false-invariant table; **not re-filed**.
---
## 6. RTO/RPO — measured
**28 restores, every one returning the correct discriminator.** Wall-clock is measured from the restore
request to **the app serving the correct data** (the canary read back over rallly's real network path),
not to "restore returned".
**66 restores across run 1 and run 2b. Every one returned the correct discriminator.**
| Tier | App | Data volume | n | min | median | p90 | max | mean |
|---|---|---|---|---|---|---|---|---|
| **Tier 1 (local)** | rallly (postgres) | **66 MB** volume / 67 MB recovery unit | 28 | 38.8 s | **42.0 s** | 42.5 s | 44.3 s | 42.0 s |
| Tier 1 (local) | rallly (postgres) | 66 MB volume / 67 MB unit | 28 (run 1) | 38.8 s | **42.0 s** | 42.5 s | 44.3 s | 42.0 s |
| Tier 1 (local) | rallly (postgres) | 66 MB | **38 (run 2b)** | 29.1 s | **41.4 s** | 42.1 s | 49.6 s | 40.7 s |
**Which band this covers: the S band only, and only its lower end.** 66 MB is a small app by any
reading, and the spread is tight — 5.5 s between min and max over 28 runs — which says the cost is
dominated by fixed work (stack stop, volume restore, stack start, health wait) rather than by bytes.
**Nothing here licenses an extrapolation to M or L**: the byte-proportional term is precisely what
66 MB fails to exercise. A second point at a much larger volume is needed before any curve is claimed,
and none was taken.
The two independent passes agree to **0.6 s on the median**, and run 2b's wider spread (29.1–49.6 s
against 38.8–44.3 s) is the cost of measuring under hard resets, agent kills and a near-full drive.
**RPO is NOT measured.** It is a function of backup *frequency*, and the soak drove backups on demand
rather than on the schedule, so the observed intervals say nothing about the product's RPO.
**Band: the S band's lower end only.** 66 MB is small, and the tight distribution says fixed work —
stack stop, volume restore, start, health wait — dominates. **The byte-proportional term is exactly
what 66 MB fails to exercise, so nothing here extrapolates to M or L**, and no such claim is made.
**RPO is not measured.** Backups were driven on demand, not on the schedule, so the observed
intervals say nothing about the product's RPO.
---
## 6b. Monotonic growth — the half the invariants cannot see
I1–I11 are *correctness* checks: they answer "is the system telling the truth this cycle", and every
one of them can pass while something climbs. **9 457 samples of 19 metrics over 13.5 h**, following
Campaign 2's `controller_rss.tsv` precedent (`state/growth.tsv`, analyser in `runner/`).
**Leak-class — nothing grew.** Across 39 cycles containing dozens of redeploys, controller and agent
kills, guest reboots, VM reboots and hard resets:
| Metric | start → end | verdict |
|---|---|---|
| controller RSS | 105.1 → 31.8 MB (min 0 / max 132 across restarts) | no leak |
| agent RSS | 19 008 → 19 632 KB (+2.07/cycle) | no leak |
| controller fd / agent fd | 14 → 11 / 14 → 13 | no leak |
| docker volumes / images / containers | 7 → 7 / 13 → 13 / 9 → 9 | **no orphans from repeated redeploys** |
| controller restarts | 0 | — |
**Expected accumulation — all bounded and explicable:**
| Metric | start → end | note |
|---|---|---|
| **agent journal** | **193.8 → 463.4 MB** (+6.5/cycle, ~20 MB/h) | **the only curve with real slope** |
| VM qcow2 on the host | 17 439 → 24 903 MB (+7.5 GB) | matches host free space falling 7.5 GB |
| backup target `mentes` | 623 → 6 424 MB | tracks `VZDUMP_N` 3 → 11 |
| guest root / docker logs | 5 667 → 5 940 MB / 4 → 9 MB | |
Two things to watch rather than findings: the **agent journal** at ~20 MB/h (journald caps it, so it
is bounded, but it is a lot of logging); and **whole-guest dumps accumulating 3 → 11** on a 50 GB
target, which at soak rates would fill it in days — a real box does not back up this often, so this is
an artefact of the campaign's cadence rather than a defect.
**Answering the question directly: after 13.5 h and 39 destructive cycles, nothing leaks.** That is a
genuine positive result and the half a targeted session structurally cannot produce.
---
## 7. I1–I11 across cycles
Full table in §4b. **I1, I2 (and both pairs), I3, I4, I5, I6, I7, I10 and I11 held over 27 consecutive
cycles — 586 checks, 0 violations.** No violation was deterministic and none was racy, because there
were none; the honest statement is that the run found no drift **within 27 cycles**, not that none
exists beyond them.
Full tables in §4b. Aggregated over run 1 (27 cycles) and run 2b (39 cycles):
**I8 was not checked** — Tier 3 never ran (§3). **I9 was not automated** — see §8.
- **I7 — 66 restores, 66 correct discriminators, 0 stale, 0 empty.** The campaign's headline.
- **I2, I3, I4, I5, I6, I10, I11 — zero violations in either pass**, including the abort-in-place
variants and 195 + 135 secret-class assertions covering both sides of D5's split.
- **I1 — 2 violations in 39 cycles**, one self-recovering 3-cycle window under cycling far faster than
a real box sees (§5.3, disproved as a defect).
- **I8 — never checked.** Tier 3 could not be isolated (§3).
- **I9 — not automated.** Cited from the tester-gate run on this same controller 0.188.0, not
re-claimed as this campaign's work.
**A clean invariant is a result**: I2's genericity, I1's discrimination, both pairings, I4's
no-restart repair and I7's discriminator held across **66 cycles** of detach, reattach, abort,
backup, restore, kill, redeploy, reboot and hard reset.
---
## 8. What did not run, and why
Phase A overran badly (~5.5 h against the brief's ~1 h) — 1.26.1 is a public release image with no
auto-install path, so day-0 was a blind screendump-and-`sendkey` walk; the Tier-3 question (§3) had to
be settled before a customer could exist at all, since `dr_tier` is fixed at creation; and a false
start on drive enrolment had to be backed out. **Phase B was therefore deliberately not launched at
04:00** — the brief's own fence, *"a test rig producing false negatives is worse than no rig"* — and
was built and run the next morning instead, where its first cycle could be watched.
**Atom coverage: 6 of the brief's ~12 suggested families.** Stated plainly rather than implied:
**Atom coverage: 12 of the brief's ~13 families ran.** Run 1 covered 6; run 2b added the rest.
| Atom (brief B1) | Ran |
|---|---|
| back up Tier 1/2 · restore + verify by discriminator · detach/reattach **target** · detach/reattach **non-target** · delete an app and redeploy · reboot the guest | **YES** |
| back up Tier 1/2 · restore + verify by discriminator · detach/reattach target · detach/reattach non-target · delete and redeploy · reboot the guest | **YES** (run 1 + 2b) |
| **abort a filesystem in place** (R-117 Q7) · **kill the agent mid-backup** · **hard-reset the VM mid-write** · **reboot the VM** · **concurrent backup+restore** · **concurrent backup+detach** · **fill a drive near-full** | **YES** (run 2b) |
| back up / restore **Tier 3** | **NO** — §3, structurally impossible to isolate |
| abort a filesystem **in place** without the device disappearing (R-117 Q7) | **NO** |
| kill the **agent** mid-backup · **hard-reset the VM** mid-write · reboot the **VM** | **NO** |
| **concurrent** backup+restore · **concurrent** backup+detach | **NO** |
| fill a drive to near-full and continue | **NO** |
**`kill_controller` is not "mid-backup".** It restarts the controller at a permuted point in the
cycle, so it sometimes lands after a backup and sometimes nowhere near one. It exercises recovery, not
the mid-write race the brief asked for. Recorded because the atom name would otherwise overclaim.
**Still honest about two limits.** `kill_controller` is not literally "mid-backup" — it lands at a
permuted point, so it exercises recovery rather than the mid-write race; the dedicated
`concurrent backup+detach` atom (`I1-under-load`, 5/5 PASS) covers the mid-operation case properly.
And **I9** — a Tier-1/2 restore with the guest's `app.yaml` moved aside — was **not automated**; it
was proven on this same controller 0.188.0 by the tester-gate run and is cited, not re-claimed.
**`reboot_guest` was appended after the shuffle, not permuted into it**, so it always ran last in
cycles 7/14/21 and **never interleaved with a detach**. Reboot-during-detach is untested.
**I8** — not checked; Tier 3 never ran. **I9** (Tier-1/2 restore with the guest's `app.yaml` moved
aside, data read over the app's real network path) — **not automated**. It was proven on this exact
controller 0.188.0 by the tester-gate run hours earlier
(`tester-gate-golden-0.188.0-2026-07-31.md` §5, step 6), and that is cited rather than re-claimed:
this campaign did not execute it.
**The tail is untested, not clean.** 27 cycles is the count; the brief's interest in "the state that
quietly drifts on the thirty-eighth" is not answered by a 27-cycle run.
**Depth reached: 39 consecutive cycles**, past the brief's "the state that quietly drifts on the
thirty-eighth". Cycles 34–39 were clean on every invariant. Beyond 39 is untested, not proven clean.
---