felhom.eu/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md

CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01)

Phase A COMPLETE and gated (2026-07-31). Phase B run in three passes (2026-08-01/02): run 1 27 cycles, run 2a 10 cycles (stopped — harness defect), run 2b 39 cycles. Together 66 clean cycles + 39 with the full atom set, 1 461 invariant checks.

Findings: R-156 (papra), R-157 (bootrecon, two mechanisms). One structural scope constraint — Tier 3 cannot be isolated (§3). Three suspicions investigated and DISPROVED (§5.2, §5.3), and two violations traced to the harness, not the product (§4d). Monotonic-growth sampling over 13.5 h found no leak (§6b).

Evidence: ../tests/campaign10-evidence-2026-07-31/ — state/phaseA-journal.md (step-by-step), isolation/pbs-denial.txt, isolation/restic-denial.txt, r156-papra-volume.txt.

---

1. Venue and baselines

Host	demo-hp (HP t740), Tier 0, the designated drill/build host
VM	311 c10-appliance — q35/OVMF (pre-enrolled-keys=0), 4 cores, 8 GB, cpu=host
Disks	scsi0 200 G system · scsi1 50 G · scsi2 50 G, qcow2 on c10-scratch
Storage	c10-scratch, dir at /mnt/nvme-1tb — the mount ROOT, content=images
Box	c10.felhom.eu @ 192.168.0.105 (demo-hp's vmbr0, 192.168.0.87/24)
Console	no browser on DooPlex → qm monitor screendump → PNG, read visually; sendkey for input
Untouched	drill-r50 (VM 300), guest 9201 on both demo boxes, local-lvm, both backup targets, ep0, the Storage Box

The exactMount choice, made deliberately (the brief asks which). c10-scratch sits at the mount root, not a subdirectory. A dir storage at a subdirectory sets exactMount=false → reachable=false → StorageStateDisconnected (felhom-agent/internal/storage/observe.go:336), which would have emitted storage_disconnected for demo-hp for the entire run — the exact signal I1/I2 exist to discriminate. It coexists with felhom-backup on the same path, which was not modified.

Baselines — every value read fresh

What	Value	How
controller main	v0.188.0	CHANGELOG top; HEAD=origin/main=4115e88, clean
golden's baked controller	0.188.0 (7353d8be…)	hub /configuration, selected option
agent published + vouched	0.119.0 (e37aca82…)	hub /configuration, selected option
hub	0.86.0	live deploy image tag == CHANGELOG top
ISO	1.26.1, f3cc86d5…, 1 705 322 496 B	live round trip from iso.felhom.eu
felhom.eu HEAD	e9a74a0, clean	git rev-parse
managed floor	v0.156.0 (DB override)	hub /configuration

Golden vs main: not behind — both 0.188.0. The deliberate decision the brief asks for is therefore vacuous this run; the campaign tests what is shipping and what is current.

Two of the brief's assumptions were stale; reading fresh caught both.

1. The brief was written against ISO v1.25.0. iso.felhom.eu began serving 1.26.1 at 18:37 Z the same day, ~90 minutes before this run. Operator chose 1.26.1 — the current published artifact.
2. The brief and three docs say demo-hp has no baked SSH key; ssh -o BatchMode=yes demo-hp authenticated by key, first try. Already R-129; not re-filed.

---

2. The isolation gate — §2

Target	Isolation	Proof
PBS	dedicated datastore felhom-c10 on DooPlex + auth-id c10@pbs!box scoped to it alone	CAPTURED — isolation/pbs-denial.txt
restic	scratch subaccount u629488-sub4, own chroot home felhom-campaign10	CAPTURED — isolation/restic-denial.txt
hub	the campaign's own record c10-soak; no other customer touched	CAPTURED — §2.3

Both denials carry a POSITIVE CONTROL, and the first attempt failed it. The initial PBS capture returned four tidy 403s — including on the campaign's own datastore. All four were worthless: PBS API tokens carry privilege separation, so a token's rights are the intersection of the user's ACL and the token's, and user c10@pbs had none. Granting the scoped role to the user as well turned the control green and left the denials standing. Without the control this would have shipped as a clean isolation proof that proved only that the credential was broken.

## POSITIVE CONTROL — same token, its OWN datastore
GET /api2/json/admin/datastore/felhom-c10/snapshots     {"data":[]}   HTTP 200

## DENIAL 1 — production datastore felhom-spike
permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-spike   403
## DENIAL 3 — DELETE namespace demo-felhom (destructive)
permission check failed - missing Datastore.Modify on /datastore/felhom-spike                   403

restic, same shape — the credential writes and reads back in its own home, and every production path is outside the chroot:

POSITIVE CONTROL  put/get/rm c10-canary.txt in own home        → succeeds, content verified
DENIAL  cd /home/felhom-peti-felhom      → stat remote: No such file or directory
DENIAL  ls ..                            → remote readdir("/home/../"): Permission denied

2.3 — hub. The run created exactly one record, c10-soak, and bound exactly one appliance (uuid 55ab3776-…) to it. Existing customers (demo-felhom, demo-hp, peti-felhom, drill-r50, sess-f) were read but never written. Verified from the hub's own log: every c10-soak line is the campaign's, and no line names another customer.

---

3. Scope constraint — Tier 3 cannot be isolated, so it was not run

The brief requires the campaign's PBS to be a dedicated datastore and namespace on DooPlex. That is unreachable, for two reasons that are each an already-recorded deliberate position:

1. Offsite hard-requires the DR tier — hub/internal/web/configs.go:1300 refuses outright: "Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS key)". Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4).
2. The DR tier only ever provisions on ep0 — peer allocation and endpoint sync use the lowest endpoint_id only; per-endpoint allocation is an explicitly deferred arc (hub/README.md:260).

So the only two configurations are DR tier on → campaign PBS traffic lands on ep0 (Tier 2, the offsite of last resort, RAM-constrained per R-90, fenced by §3), or DR tier off → no Tier 3 at all. Chosen: DR tier OFF, offsite OFF — the only option §3 permits.

Consequence, stated up front rather than discovered in the results: the campaign touches neither ep0 nor the Storage Box at any point — stronger isolation than asked for, obtained by not running the tier. Lost: every Tier-3 atom, I8 entirely, and the Tier-3 RTO/RPO rows. Unaffected: the two-drive core (R-112/113/114/116/117), D5, and I1–I7, I9–I11.

No R-n minted — the register was grepped first and both halves are already recorded. This is a campaign constraint, not a defect.

---

4. Phase A gates

Gate	Result	Positive observable
A1 install from the published ISO	PASS	ii felhom-bootstrap 1.26.1; unit enabled, fired at first boot; registered unclaimed with pairing code FD6-A6S, MACs, hardware and 3 SSH host keys; 4.2 GB actually written, system booted on its own
A2 claim	PASS	discriminator flipped dashboard not yet claimed → authentication required; hub: [claim] customer c10-soak CLAIMED its dashboard
A3 enrol both drives, apps across them	PASS	both registered via the real endpoint; backup target healthy; 4 apps healthy
A4 discriminators	PASS	all four apps seed and read back the same generation
A5 budget + watchdog	PASS (built for Phase B)	disk floor 120 G on /mnt/nvme-1tb + VM-liveness watchdog; heartbeat status.txt per phase; journal.tsv per check. Neither fired: the run ended on its deadline with 871 G free

A1 — 1.26.1 cannot auto-install, verified against the bytes (not inferred from its manifest): auto-installer-mode.toml 0 hits, answer.toml 0, proxmox-auto-install 0, exactly 2 interactive menuentries. Driven blind through the Terminal UI (the graphical entry is undrivable — monitor mouse events do not move the guest cursor). Three traps hit and handled:

• GRUB's default is the graphical entry; the first down landed after the 15 s countdown. Reset and sent it inside the window in one round trip.
• The guest keymap was Hungarian while sendkey emits US scancodes. Every symbol afterwards — notably @, which is AltGr+V on a Hungarian layout — would have been silently mangled, including the 24-char root password, locking the box out with no way back in. Switched to U.S. English before typing anything; confirmed positively when c10@felhom.eu rendered a correct @ and ..
• The recorded boot-order trap: --boot set in its own qm set after the disks existed and verified from qm config; flipped to order=scsi0 post-install and verified again.

Only scsi0 was attached during the install, so the installer offered exactly one target and could not reach a data drive. The 50 G drives were attached afterwards.

A2 — the installer's root password stops working at day-0; it is replaced by a hub-vaulted managed credential (vaulted break-glass recovery credential … secret 32 chars), retrieved via POST /hosts/c10-soak-8a405e/reveal-recovery-credential. R-119 confirmed live again: the claim code is emailed-only and the operator relayed it — the single human step.

A3 — both drives enrolled through the real endpoint (POST /api/storage/attach), which does format → assign → guest-attach → register-stable-path in order:

Drive	Device	Stable path	Role
adatok	/dev/sdb	/mnt/felhom-drives/adatok	app data, set_default
mentes	/dev/sdc	/mnt/felhom-drives/mentes	backup target (PVE storage felhom-backup at /mnt/mentes)

A false start worth recording: hand-driving format + assign first left the drives mounted but unregistered — invisible to both the raw-candidate scan and /disks, so the real enroll endpoint could not resolve a UUID for them (resolveEnrollUUID sees neither list). Backed out (eject leaves the raw /mnt/<name> mount by design, disks.go:606, so the mounts also had to be undone) and re-ran the real flow clean. The lesson is the brief's own: use the real endpoint, or manufacture a state the product never produces.

Backup-target offer flow (R-112/R-114/E-2) exercised as the customer would: assign returned restart_required: true, and the state only flipped after the agent restart it asked for —

before: {"degraded":true, "target":"local", "offer_path":"/mnt/felhom-drives/adatok", "message":"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after:  {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}

That healthy read is the I5/I6 baseline: everything healthy ⇒ no degraded banner, target reads healthy, zero absent events.

Apps — four, healthy, covering both sides of D5's split:

App	Data	Secret classes	Why chosen
rallly	postgres (rallly-postgres)	2 × type: secret (SECRET_PASSWORD hex:32 labelled Titkosítási kulcs, DB_PASSWORD)	the DB app; travelling class
homebox	sqlite volume	1 × secret, data_key: true	data-encrypting key
papra	see R-156	1 × secret, data_key: true	data-encrypting key
grafana	volume	1 × type: password	the withheld class — I10's negative half

4 travelling secret fields and 1 password field that must never travel, so I10 can assert both directions rather than only the positive one.

A4 — discriminators work. Every app carries a generation marker; a full seed/read cycle returns the seeded generation from all four. rallly's is a row in a cc_proof table read over the path DATABASE_URL actually names (docker run --network container:rallly … psql -h rallly-postgres), never 127.0.0.1 inside the postgres container — the trusted-socket path that produced D5's §1.2 false pass.

---

4b. Phase B — the soak

Three passes. Run 2a was stopped deliberately at cycle 10 when two violations turned out to be harness defects (§4d); the harness was fixed and red-proofed, and run 2b restarted from cycle 1 with the full atom set.

Pass	Cycles	Atoms	Violations
run 1	27	6 families	0
run 2a	10	13 families	2 — both harness, §4d
run 2b	39	13 families	9 (see below)

Run 2b — 875 invariant rows over 39 cycles, 13.5 h, ended on its own deadline. No watchdog halt, no atom exception, no I11 breach.

Check	PASS	VIOL	Note
I7	38	0	+ 1 I7-SKIP — the fixed harness declining to judge, not a false green
I1 / I1-pair	37 / 36	2 / 3	one 3-cycle window, self-recovered — §5.3
I2 / I2-pair	39 / 39	0 / 0
I3 / I3-abort	78 / 4	0
I4 / I4-abort / -recover	78 / 4 / 4	0
I5 / I6	39 / 39	0
I10	195	0	5 secret-class fields × 39
I11	39	0
I1-under-load / -recover	5 / 5	0	target pulled during a running backup
BACKUP / KILL-CTRL	82 / 39	0
KILL-AGENT / REBOOT / REBOOT-VM	5 / 5 / 5	0
CONCURRENCY / FILL-DRIVE	5 / 5	0
HARD-RESET	2	3	R-157
REDEPLOY	38	1	harness: 5-min wait too short under load; the app returned just after

The full atom set ran this time. The six families run 1 skipped are all present: filesystem aborted in place (R-117 Q7), kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms, and fill-drive-near-full. The run-1 flaw where reboot was appended after the shuffle — so it never interleaved with a detach — was fixed; heavy atoms are now permuted in.

Two results worth naming. I1-under-load passed 5/5: pulling the backup target while a backup was running still produced backup_target_absent and a clean recovery. And I4-abort / I3-abort passed 4/4 — R-117's Q7 case, the one its spike called "the worse half", holds (details in ../tests/campaign10-evidence-2026-07-31/r117-q7-abort-in-place.md).

4d. Two violations that were the HARNESS, not the product

Recorded because a check that fails for the wrong reason is as corrosive as one that passes for the wrong reason, and this arc already has six of the latter. Full write-up: ../tests/campaign10-evidence-2026-07-31/run2a-violations-were-harness.md.

Run 2a cycle 10 reported HARD-RESET canaries_intact=False and, worse, I7 claiming a stale restore (want=C10-C010-A got=C10-C009-A, restore_ok=True). The cc_proof table settled it: its highest row was C10-C009-A — there was no C010-A row at all, so the seed never landed, because the hard reset earlier in the same cycle had left rallly Exited. atom_restore_verify called seed() without checking its return; atom_hard_reset_mid_write read canaries out of app containers that were still starting.

A real stale restore would have produced a byte-identical journal row, which is why this justified stopping a running 10-cycle pass rather than annotating it. Fixed with apps_ready(), seed(verify=True) (read-back), and an I7-SKIP verdict so a skipped check is never silently a green one — red-proofed both directions before restarting. Run 2b's single I7-SKIP is that fix working.

4c. Primitives proven by hand before automation

Step	Observed
backup	POST /api/backup/run → Mentés elindítva, settles in ~23–34 s
restore	POST /backup/restore (form) is async; polled on /api/backup/restore-status, never IsRunning()
I7, first proof	seeded A → backup → seeded B → restored A; rallly returned A while the other three still read B
I1	backup_target_absent (error) — A rendszermentés meghajtója nem érhető el: Mentes
I2	storage_disconnected (error) — Meghajtó váratlanul leválasztva: Adatok
I6 degraded copy	"A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod, a teljes rendszermentés nem készül el."
I10	rallly DB_PASSWORD+SECRET_PASSWORD, homebox HBOX_AUTH_API_KEY_PEPPER, papra AUTH_SECRET present; grafana GF_SECURITY_ADMIN_PASSWORD absent; unit mode 0600

5. Findings

R-156 — papra's data is neither persisted nor backed up, and it reports healthy

New. Register grepped first: papra appears in R-41 (never-deployable / healthcheck) and R-127 (data_key misclassification); neither covers this. (Also corrected en route: a first grep suggested R-164 was taken — it is a mis-citation of controller v0.164.0, already flagged in REPORT-record-correction-2026-07-29.md:141. Highest real number is R-155, so this is R-156.)

The template mounts papra_data:/app/data, but the app persists to /app/app-data/db/db.sqlite. The mounted volume is therefore empty, and the real database lives in the container's writable layer.

Evidence (r156-papra-volume.txt), all four legs measured:

1. volume papra_papra_data -> /app/data
2. uid=999(nonroot);  drwxr-xr-x 2 root root /app/data
   touch: cannot touch '/app/data/.w': Permission denied
3. /app/data contains only the canary this campaign wrote AS ROOT
4. /app/app-data/db/db.sqlite  475136 bytes  owned nonroot   <-- the real DB, in no volume
5. health status: "healthy"

Consequences. (a) papra's documents do not survive docker compose down/redeploy/host restore — the writable layer is not persisted. (b) DumpAppVolumes dumps the volume, so papra is backed up as an empty directory: its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. (c) The healthcheck only probes the HTTP port, so none of this surfaces — the app is green throughout.

This is the presence is not success family from CLAUDE.md: a backup exists, a healthcheck passes, and neither means the data is there. It is also a second instance of the pattern R-41 names — an app that deploys and looks fine while being fundamentally broken.

Two-repo shape, if fixed: the volume target belongs in app-catalog-felhom.eu, and the general defence is a catalog gate asserting each template's mounted volume is a path the app actually writes. Not fixed here — the fences forbid it, and a fix mid-run would prove a version that did not exist when the run started.

R-157 — bootrecon's start-ONCE sweep acts on an unsettled snapshot; the boot orphan survives

New. Register grepped: R-52 is the feature, nothing covers its timing. INTERMITTENT — 3 of 6 hard resets (~50%) across two runs, with an identical signature each time. (An earlier draft called this deterministic because the failing cycle numbers matched; they matched only because the runner's RNG is seeded. Intermittency is what a race predicts and a wrong predicate does not.) Full evidence: ../tests/campaign10-evidence-2026-07-31/r157-bootrecon-start-once-race.md.

A qm reset mid-backup brought everything back except the app half of the DB-backed stack: rallly left Exited (255) (oom=false, restarts=0, its own log ending ✓ Ready — it died healthy) while rallly-postgres returned healthy.

20:28:13 Status refresh: 8 containers across 55 stacks      <-- docker ps -a shows NINE
20:28:18 [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start)
20:28:25 Status refresh: 7 containers … 8 containers        <-- still churning AFTER the sweep
20:39:14 [deadapp] 20 scans since boot, 5 deployed app(s) evaluated, 1 currently down

The predicate is not at fault — once settled the controller reports rallly state=degraded containers=2, and IsDownState includes StateDegraded (manager.go:55), so len(Containers)>0 && IsDownState(State) holds. The snapshot was: bootrecon fires as go runBootReconcile(...) (cmd/controller/main.go:236) ~5 s after start, while docker is still restoring containers, and is start-once by design so it never re-checks.

Consequence: the app stays down indefinitely. Detection is perfect and recovery never happens — exactly R-52's original shape, an alarm with no recovery (F5: two apps Exited for ~18 h). In run 2a the app only returned because a later campaign atom redeployed it. Not fixed — the fences forbid it, and the settle-condition fix belongs with a test that pins the consequence.

5.2 — Investigated and DISPROVED: /api/disks reports state=attached for an absent drive

Recorded because the chase is the useful artefact, not because anything is broken.

With adatok physically removed, /api/disks kept reporting state: "attached" for it across three polls (~90 s) — while /dev/sdb was gone, /mnt/adatok was absent from /proc/mounts, and the guest bind returned Input/output error. The other fields were right (backing_device: "", guest_attached: false, bound_under_parent: false).

It looked like the R-116/R-117 family, and there is a real consumer: internal/web/intermediary.go:230 — present[d.MountPath] |= d.State == "attached". A presence gate reading an absent drive as present is precisely I3's failure mode.

It is inert, and the gate is correct. planDriveGates only gates paths under StableParentDir+"/" (/mnt/felhom-drives/…). The registered path is the stable one, whose presence is computed from d.BoundUnderParent — observed false — not from State. The State-derived entry is keyed on the raw /mnt/adatok, which is not a registered stable path, so the loop skips it. The observable end-to-end behaviour agrees: the storage page showed "Meghajtó leválasztva: Adatok" with a disconnect timestamp and a reconnect action, i.e. the gate fired.

The stale state field comes from the registry row surviving in the union after the storage-observation row drops out (a raw enrolled drive is not a PVE storage), carrying its configured MountPath. No R-n minted — no consumer is misled, and the R-116 comment's claim that this "cannot make the gate read an absent drive as PRESENT" held under test.

5.3 — Investigated and DISPROVED: the backup_target_* pair going silent

Five I1/I1-pair violations in a ~10-minute window looked like R-116's unmatchable pair returning. It is not. Full write-up: ../tests/campaign10-evidence-2026-07-31/RESOLVED-backup-target-pair-went-silent.md.

The violations cluster at cycles 31-33 and nowhere else in 39 cycles; c34-c39 are clean, so it recovered with no intervention. On the quiesced box afterwards, one slow detach with 4 minutes either side produced a perfect pair (backup_target_absent → backup_target_restored → health_recovered). And the alarming false-healthy I sampled alongside it (mentes bound_under_parent=False while degraded: false) does not survive quiescence — three minutes after the run both drives read bound=True and the target read healthy. I had been reading the two halves at different instants of a detach. No R-n.

5.4 — Investigated and DISPROVED: the hub's SQLITE_BUSY event drops

[ERROR] Failed to save event from c10-soak: database is locked (5) (SQLITE_BUSY)

7 in 24 h, including one for the real customer demo-felhom, so not a campaign artefact. The hub returns HTTP 500 on the failed save (hub/internal/api/handler.go:1715) and dispatches notifications only after a successful save — so a genuinely lost event would be a lost alarm.

But the controller retries 3×, and zero events exhausted their attempts across the whole run (grep -c "Event push failed after 3 attempts" = 0). The 07:04:39 drop landed 3 s later at 07:04:42, same event. Nothing was lost. No R-n. One cosmetic note kept: [ERROR] Failed to save event reads like data loss and is not.

Not filed

• demo-hp SSH key — already R-129.
• Tier 3 isolation — §3; two recorded deliberate positions, not a defect.
• /api/disks state — §5.2, disproved.
• backup_target_* silence — §5.3, transient and self-recovered.
• hub SQLITE_BUSY drops — §5.4, absorbed by the controller's retry.
• F-CRIT-1 (StateStopped assumed deliberate) — confirmed live again via R-157 mechanism B, but already recorded in CLAUDE.md's false-invariant table; not re-filed.

---

6. RTO/RPO — measured

66 restores across run 1 and run 2b. Every one returned the correct discriminator.

Tier	App	Data volume	n	min	median	p90	max	mean
Tier 1 (local)	rallly (postgres)	66 MB volume / 67 MB unit	28 (run 1)	38.8 s	42.0 s	42.5 s	44.3 s	42.0 s
Tier 1 (local)	rallly (postgres)	66 MB	38 (run 2b)	29.1 s	41.4 s	42.1 s	49.6 s	40.7 s

The two independent passes agree to 0.6 s on the median, and run 2b's wider spread (29.1–49.6 s against 38.8–44.3 s) is the cost of measuring under hard resets, agent kills and a near-full drive.

M band — measured 2026-08-02

The S figures could not speak for M, so a second point was taken 327× larger, same app, same method. Full evidence: ../tests/campaign10-evidence-2026-07-31/rto-m-band-2026-08-02.md.

rep	volume	backup	recovery unit	RTO to correct data	discriminator
1	21 616 MB	406.4 s	41 149 MB	624.5 s	correct
2	21 615 MB	387.2 s	41 133 MB	591.8 s	correct
mean	21.1 GB	396.8 s	40.2 GB (1.90×)	608.1 s (10.1 min)	2/2

327× the data cost 14.5× the time — strongly sub-linear, so:

RTO ≈ 40 s + 26.9 s/GB · backup ≈ 29 s + 17.4 s/GB

1 GB → 67 s · 10 GB → 5.2 min · 20 GB → 9.6 min (measured 10.1) · 50 GB → 23 min · 100 GB → 46 min

The fixed ~40 s dominates below ~1.5 GB — that is the S band, and it is why the S numbers were so tightly clustered. Above it, RTO is essentially linear in data.

The capacity ceiling — the more consequential result

A DB-backed app's recovery unit is 1.90× its data (volume tar plus SQL dump): 21.1 GB produced a 40.2 GB unit. The default appliance ships /mnt/sys_drive at 20 GB (mp1 … size=20G; SysDataGrowGB is a per-customer grow, 0 for this customer). Therefore:

on a default box the largest app that can hold a local Tier-1/2 recovery unit is ≈ 10 GB, and that fills the volume completely — realistically ≈ 8 GB. The M band does not fit on a default box at all. This test only reached 21 GB because /mnt/sys_drive was first grown to 70 G.

A sizing decision for the S/M/L tiers rather than a defect — but it is the constraint that actually bites, and it is invisible until an app crosses it.

Caveats, stated not buried: two points define a line but do not test linearity (no ~5 GB point was taken); the 1.90× is DB-app-specific and a file-only app should be nearer 1.0× — inferred, not measured; the data is deliberately incompressible synthetic, so well-compressing real data moves both numbers; one app, one shape, one box.

RPO is still not measured. Backups were driven on demand, not on the schedule.

---

6b. Monotonic growth — the half the invariants cannot see

I1–I11 are correctness checks: they answer "is the system telling the truth this cycle", and every one of them can pass while something climbs. 9 457 samples of 19 metrics over 13.5 h, following Campaign 2's controller_rss.tsv precedent (state/growth.tsv, analyser in runner/).

Leak-class — nothing grew. Across 39 cycles containing dozens of redeploys, controller and agent kills, guest reboots, VM reboots and hard resets:

Metric	start → end	verdict
controller RSS	105.1 → 31.8 MB (min 0 / max 132 across restarts)	no leak
agent RSS	19 008 → 19 632 KB (+2.07/cycle)	no leak
controller fd / agent fd	14 → 11 / 14 → 13	no leak
docker volumes / images / containers	7 → 7 / 13 → 13 / 9 → 9	no orphans from repeated redeploys
controller restarts	0	—

Expected accumulation — all bounded and explicable:

Metric	start → end	note
agent journal	193.8 → 463.4 MB (+6.5/cycle, ~20 MB/h)	the only curve with real slope
VM qcow2 on the host	17 439 → 24 903 MB (+7.5 GB)	matches host free space falling 7.5 GB
backup target mentes	623 → 6 424 MB	tracks VZDUMP_N 3 → 11
guest root / docker logs	5 667 → 5 940 MB / 4 → 9 MB

Two things to watch rather than findings: the agent journal at ~20 MB/h (journald caps it, so it is bounded, but it is a lot of logging); and whole-guest dumps accumulating 3 → 11 on a 50 GB target, which at soak rates would fill it in days — a real box does not back up this often, so this is an artefact of the campaign's cadence rather than a defect.

Answering the question directly: after 13.5 h and 39 destructive cycles, nothing leaks. That is a genuine positive result and the half a targeted session structurally cannot produce.

---

7. I1–I11 across cycles

Full tables in §4b. Aggregated over run 1 (27 cycles) and run 2b (39 cycles):

• I7 — 66 restores, 66 correct discriminators, 0 stale, 0 empty. The campaign's headline.
• I2, I3, I4, I5, I6, I10, I11 — zero violations in either pass, including the abort-in-place variants and 195 + 135 secret-class assertions covering both sides of D5's split.
• I1 — 2 violations in 39 cycles, one self-recovering 3-cycle window under cycling far faster than a real box sees (§5.3, disproved as a defect).
• I8 — never checked. Tier 3 could not be isolated (§3).
• I9 — not automated. Cited from the tester-gate run on this same controller 0.188.0, not re-claimed as this campaign's work.

A clean invariant is a result: I2's genericity, I1's discrimination, both pairings, I4's no-restart repair and I7's discriminator held across 66 cycles of detach, reattach, abort, backup, restore, kill, redeploy, reboot and hard reset.

---

8. What did not run, and why

Atom coverage: 12 of the brief's ~13 families ran. Run 1 covered 6; run 2b added the rest.

Atom (brief B1)	Ran
back up Tier 1/2 · restore + verify by discriminator · detach/reattach target · detach/reattach non-target · delete and redeploy · reboot the guest	YES (run 1 + 2b)
abort a filesystem in place (R-117 Q7) · kill the agent mid-backup · hard-reset the VM mid-write · reboot the VM · concurrent backup+restore · concurrent backup+detach · fill a drive near-full	YES (run 2b)
back up / restore Tier 3	NO — §3, structurally impossible to isolate

Still honest about two limits. kill_controller is not literally "mid-backup" — it lands at a permuted point, so it exercises recovery rather than the mid-write race; the dedicated concurrent backup+detach atom (I1-under-load, 5/5 PASS) covers the mid-operation case properly. And I9 — a Tier-1/2 restore with the guest's app.yaml moved aside — was not automated; it was proven on this same controller 0.188.0 by the tester-gate run and is cited, not re-claimed.

Depth reached: 39 consecutive cycles, past the brief's "the state that quietly drifts on the thirty-eighth". Cycles 34–39 were clean on every invariant. Beyond 39 is untested, not proven clean.

---

9. Teardown — OWED, nothing removed yet

The rig is intact on purpose, so Phase B can run without repeating Phase A. Every item below is outstanding and must be removed when the campaign ends:

Layer	Item	Command
VM	311 c10-appliance on demo-hp	qm stop 311 && qm destroy 311 --purge
storage	c10-scratch dir storage	pvesm remove c10-scratch
PBS	datastore felhom-c10, user c10@pbs, token !box, both ACLs	proxmox-backup-manager datastore remove felhom-c10 · user delete c10@pbs · remove /mnt/5_hdd/backup/campaign10
restic	subaccount u629488-sub4 (felhom-campaign10) on box 611714 — unused, since Tier 3 never ran	DELETE /v1/storage_boxes/611714/subaccounts/281530
hub	customer c10-soak — disposition: DELETE. Named explicitly per R-131, which is four orphaned scratch customers left by exactly this omission	POST /configs/c10-soak/delete with ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=c10-soak
secrets	~/.config/campaign10/ on DooPlex (host + dashboard passwords, API key, app secrets, PBS token, restic password); /root/c10/ and /root/c10api.sh in guest 9201	shred -u both

sess-f ("R-120 golden 0.186.0 proof") is still present and is not this run's record — R-131 stands, untouched.

---

10. Hygiene note

The customer API key for c10-soak was printed into this session's transcript while being read from the hub page. It belongs to a scratch customer scheduled for deletion (§9) and grants nothing once that record is gone, but the write-out was avoidable and is recorded here rather than left implicit — the same class as R-132, where a correctly-made request was undone by how it was reported.