felhom.eu/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md

CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01)

Status: Phase A COMPLETE and gated (2026-07-31). Phase B RUN 2026-08-01 — 27 cycles, 586 invariant checks, ZERO violations. One new finding: R-156. One structural scope constraint: Tier 3 cannot be isolated (§3). One suspected defect investigated and DISPROVED (§5.2).

Evidence: ../tests/campaign10-evidence-2026-07-31/ — state/phaseA-journal.md (step-by-step), isolation/pbs-denial.txt, isolation/restic-denial.txt, r156-papra-volume.txt.

---

1. Venue and baselines

Host	demo-hp (HP t740), Tier 0, the designated drill/build host
VM	311 c10-appliance — q35/OVMF (pre-enrolled-keys=0), 4 cores, 8 GB, cpu=host
Disks	scsi0 200 G system · scsi1 50 G · scsi2 50 G, qcow2 on c10-scratch
Storage	c10-scratch, dir at /mnt/nvme-1tb — the mount ROOT, content=images
Box	c10.felhom.eu @ 192.168.0.105 (demo-hp's vmbr0, 192.168.0.87/24)
Console	no browser on DooPlex → qm monitor screendump → PNG, read visually; sendkey for input
Untouched	drill-r50 (VM 300), guest 9201 on both demo boxes, local-lvm, both backup targets, ep0, the Storage Box

The exactMount choice, made deliberately (the brief asks which). c10-scratch sits at the mount root, not a subdirectory. A dir storage at a subdirectory sets exactMount=false → reachable=false → StorageStateDisconnected (felhom-agent/internal/storage/observe.go:336), which would have emitted storage_disconnected for demo-hp for the entire run — the exact signal I1/I2 exist to discriminate. It coexists with felhom-backup on the same path, which was not modified.

Baselines — every value read fresh

What	Value	How
controller main	v0.188.0	CHANGELOG top; HEAD=origin/main=4115e88, clean
golden's baked controller	0.188.0 (7353d8be…)	hub /configuration, selected option
agent published + vouched	0.119.0 (e37aca82…)	hub /configuration, selected option
hub	0.86.0	live deploy image tag == CHANGELOG top
ISO	1.26.1, f3cc86d5…, 1 705 322 496 B	live round trip from iso.felhom.eu
felhom.eu HEAD	e9a74a0, clean	git rev-parse
managed floor	v0.156.0 (DB override)	hub /configuration

Golden vs main: not behind — both 0.188.0. The deliberate decision the brief asks for is therefore vacuous this run; the campaign tests what is shipping and what is current.

Two of the brief's assumptions were stale; reading fresh caught both.

1. The brief was written against ISO v1.25.0. iso.felhom.eu began serving 1.26.1 at 18:37 Z the same day, ~90 minutes before this run. Operator chose 1.26.1 — the current published artifact.
2. The brief and three docs say demo-hp has no baked SSH key; ssh -o BatchMode=yes demo-hp authenticated by key, first try. Already R-129; not re-filed.

---

2. The isolation gate — §2

Target	Isolation	Proof
PBS	dedicated datastore felhom-c10 on DooPlex + auth-id c10@pbs!box scoped to it alone	CAPTURED — isolation/pbs-denial.txt
restic	scratch subaccount u629488-sub4, own chroot home felhom-campaign10	CAPTURED — isolation/restic-denial.txt
hub	the campaign's own record c10-soak; no other customer touched	CAPTURED — §2.3

Both denials carry a POSITIVE CONTROL, and the first attempt failed it. The initial PBS capture returned four tidy 403s — including on the campaign's own datastore. All four were worthless: PBS API tokens carry privilege separation, so a token's rights are the intersection of the user's ACL and the token's, and user c10@pbs had none. Granting the scoped role to the user as well turned the control green and left the denials standing. Without the control this would have shipped as a clean isolation proof that proved only that the credential was broken.

## POSITIVE CONTROL — same token, its OWN datastore
GET /api2/json/admin/datastore/felhom-c10/snapshots     {"data":[]}   HTTP 200

## DENIAL 1 — production datastore felhom-spike
permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-spike   403
## DENIAL 3 — DELETE namespace demo-felhom (destructive)
permission check failed - missing Datastore.Modify on /datastore/felhom-spike                   403

restic, same shape — the credential writes and reads back in its own home, and every production path is outside the chroot:

POSITIVE CONTROL  put/get/rm c10-canary.txt in own home        → succeeds, content verified
DENIAL  cd /home/felhom-peti-felhom      → stat remote: No such file or directory
DENIAL  ls ..                            → remote readdir("/home/../"): Permission denied

2.3 — hub. The run created exactly one record, c10-soak, and bound exactly one appliance (uuid 55ab3776-…) to it. Existing customers (demo-felhom, demo-hp, peti-felhom, drill-r50, sess-f) were read but never written. Verified from the hub's own log: every c10-soak line is the campaign's, and no line names another customer.

---

3. Scope constraint — Tier 3 cannot be isolated, so it was not run

The brief requires the campaign's PBS to be a dedicated datastore and namespace on DooPlex. That is unreachable, for two reasons that are each an already-recorded deliberate position:

1. Offsite hard-requires the DR tier — hub/internal/web/configs.go:1300 refuses outright: "Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS key)". Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4).
2. The DR tier only ever provisions on ep0 — peer allocation and endpoint sync use the lowest endpoint_id only; per-endpoint allocation is an explicitly deferred arc (hub/README.md:260).

So the only two configurations are DR tier on → campaign PBS traffic lands on ep0 (Tier 2, the offsite of last resort, RAM-constrained per R-90, fenced by §3), or DR tier off → no Tier 3 at all. Chosen: DR tier OFF, offsite OFF — the only option §3 permits.

Consequence, stated up front rather than discovered in the results: the campaign touches neither ep0 nor the Storage Box at any point — stronger isolation than asked for, obtained by not running the tier. Lost: every Tier-3 atom, I8 entirely, and the Tier-3 RTO/RPO rows. Unaffected: the two-drive core (R-112/113/114/116/117), D5, and I1–I7, I9–I11.

No R-n minted — the register was grepped first and both halves are already recorded. This is a campaign constraint, not a defect.

---

4. Phase A gates

Gate	Result	Positive observable
A1 install from the published ISO	PASS	ii felhom-bootstrap 1.26.1; unit enabled, fired at first boot; registered unclaimed with pairing code FD6-A6S, MACs, hardware and 3 SSH host keys; 4.2 GB actually written, system booted on its own
A2 claim	PASS	discriminator flipped dashboard not yet claimed → authentication required; hub: [claim] customer c10-soak CLAIMED its dashboard
A3 enrol both drives, apps across them	PASS	both registered via the real endpoint; backup target healthy; 4 apps healthy
A4 discriminators	PASS	all four apps seed and read back the same generation
A5 budget + watchdog	PASS (built for Phase B)	disk floor 120 G on /mnt/nvme-1tb + VM-liveness watchdog; heartbeat status.txt per phase; journal.tsv per check. Neither fired: the run ended on its deadline with 871 G free

A1 — 1.26.1 cannot auto-install, verified against the bytes (not inferred from its manifest): auto-installer-mode.toml 0 hits, answer.toml 0, proxmox-auto-install 0, exactly 2 interactive menuentries. Driven blind through the Terminal UI (the graphical entry is undrivable — monitor mouse events do not move the guest cursor). Three traps hit and handled:

• GRUB's default is the graphical entry; the first down landed after the 15 s countdown. Reset and sent it inside the window in one round trip.
• The guest keymap was Hungarian while sendkey emits US scancodes. Every symbol afterwards — notably @, which is AltGr+V on a Hungarian layout — would have been silently mangled, including the 24-char root password, locking the box out with no way back in. Switched to U.S. English before typing anything; confirmed positively when c10@felhom.eu rendered a correct @ and ..
• The recorded boot-order trap: --boot set in its own qm set after the disks existed and verified from qm config; flipped to order=scsi0 post-install and verified again.

Only scsi0 was attached during the install, so the installer offered exactly one target and could not reach a data drive. The 50 G drives were attached afterwards.

A2 — the installer's root password stops working at day-0; it is replaced by a hub-vaulted managed credential (vaulted break-glass recovery credential … secret 32 chars), retrieved via POST /hosts/c10-soak-8a405e/reveal-recovery-credential. R-119 confirmed live again: the claim code is emailed-only and the operator relayed it — the single human step.

A3 — both drives enrolled through the real endpoint (POST /api/storage/attach), which does format → assign → guest-attach → register-stable-path in order:

Drive	Device	Stable path	Role
adatok	/dev/sdb	/mnt/felhom-drives/adatok	app data, set_default
mentes	/dev/sdc	/mnt/felhom-drives/mentes	backup target (PVE storage felhom-backup at /mnt/mentes)

A false start worth recording: hand-driving format + assign first left the drives mounted but unregistered — invisible to both the raw-candidate scan and /disks, so the real enroll endpoint could not resolve a UUID for them (resolveEnrollUUID sees neither list). Backed out (eject leaves the raw /mnt/<name> mount by design, disks.go:606, so the mounts also had to be undone) and re-ran the real flow clean. The lesson is the brief's own: use the real endpoint, or manufacture a state the product never produces.

Backup-target offer flow (R-112/R-114/E-2) exercised as the customer would: assign returned restart_required: true, and the state only flipped after the agent restart it asked for —

before: {"degraded":true, "target":"local", "offer_path":"/mnt/felhom-drives/adatok", "message":"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after:  {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}

That healthy read is the I5/I6 baseline: everything healthy ⇒ no degraded banner, target reads healthy, zero absent events.

Apps — four, healthy, covering both sides of D5's split:

App	Data	Secret classes	Why chosen
rallly	postgres (rallly-postgres)	2 × type: secret (SECRET_PASSWORD hex:32 labelled Titkosítási kulcs, DB_PASSWORD)	the DB app; travelling class
homebox	sqlite volume	1 × secret, data_key: true	data-encrypting key
papra	see R-156	1 × secret, data_key: true	data-encrypting key
grafana	volume	1 × type: password	the withheld class — I10's negative half

4 travelling secret fields and 1 password field that must never travel, so I10 can assert both directions rather than only the positive one.

A4 — discriminators work. Every app carries a generation marker; a full seed/read cycle returns the seeded generation from all four. rallly's is a row in a cc_proof table read over the path DATABASE_URL actually names (docker run --network container:rallly … psql -h rallly-postgres), never 127.0.0.1 inside the postgres container — the trusted-socket path that produced D5's §1.2 false pass.

---

4b. Phase B — the soak (2026-08-01)

27 cycles, 09:33–15:40 CEST, ended on its own deadline. No watchdog halt, no atom exception, no I11 breach. 586 invariant rows, 0 violations.

Every atom and every invariant was proven BY HAND before it was automated (§4c). That ordering is the point: a runner written first and trusted after is exactly the rig the brief warns about.

A gap in Phase A found and closed first. None of the four apps had HDD_PATH — all their data sat in Docker volumes on the system disk, so detaching a data drive was a no-op for apps and I3 could never have fired. A soak run in that state would have produced 27 green cycles that tested nothing about cross-drive behaviour. calibre-web was deployed onto adatok (/mnt/felhom-drives/adatok/userdata/media/books → /calibre-library, confirmed from docker inspect) before the run started.

Atoms, permuted per cycle (seeded RNG, so the order is reproducible)

backup · restore_verify · detach_target · detach_nontarget · kill_controller · redeploy_app, plus reboot_guest on every 7th cycle.

Results

Invariant	Checks	Violations	What was actually asserted
I1 + pair	28 + 28	0	target absent ⇒ backup_target_absent/error; return ⇒ backup_target_restored/info
I2 + pair	28 + 28	0	non-target ⇒ generic storage_disconnected/error → storage_reconnected/info
I3	56	0	bound_under_parent=false while the namespace is unusable, both drives
I4	56	0	guest init boot-time identical across every bind repair
I5	28	0	healthy ⇒ zero absent events, target reads healthy
I6	28	0	healthy carries no degraded message
I7	28	0	every restore returned the discriminator asked for — 28/28
I10	135	0	4 × type: secret present in the unit; 1 × type: password absent, asserted as absence
I11	28	0	no secret value in controller or hub logs
BACKUP / REDEPLOY / KILL-CTRL / REBOOT	56 / 28 / 28 / 3	0	atom outcomes

The row counts are themselves a check that nothing was silently skipped: I3 and I4 at 56 (twice per cycle — both drives), I10 at 135 (5 secret-class fields × 27), REBOOT at 3 (cycles 7/14/21). Counts of 28 rather than 27 include the one-cycle smoke test, which wrote to the same journal and restarts its numbering at 1 — the totals are right, the cycle labels collide.

A clean invariant is a result: I1's discrimination, I2's genericity, the pairing of both, I4's no-restart repair and I7's discriminator held over 27 consecutive cycles of detach, reattach, backup, restore, controller kill, app redeploy and three guest reboots. Nothing drifted at iteration 38 because the run stopped at 27 — the tail beyond that is untested, not proven absent.

Two behaviours worth recording, both observed repeatedly

• The drive comes back on a different device node and the bind still heals. adatok left as /dev/sdb and returned as /dev/sdd; bound_under_parent went true again and the contents were readable, with the guest's init boot-time unchanged. That is R-117's stale-device case, hit on every detach cycle rather than once.
• health_degraded (warning) accompanies the target-absent event and clears on return, alongside the specific backup_target_* pair.

4c. Primitives proven by hand before automation

Step	Observed
backup	POST /api/backup/run → Mentés elindítva, settles in ~23–34 s
restore	POST /backup/restore (form) is async; polled on /api/backup/restore-status, never IsRunning()
I7, first proof	seeded A → backup → seeded B → restored A; rallly returned A while the other three still read B
I1	backup_target_absent (error) — A rendszermentés meghajtója nem érhető el: Mentes
I2	storage_disconnected (error) — Meghajtó váratlanul leválasztva: Adatok
I6 degraded copy	"A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod, a teljes rendszermentés nem készül el."
I10	rallly DB_PASSWORD+SECRET_PASSWORD, homebox HBOX_AUTH_API_KEY_PEPPER, papra AUTH_SECRET present; grafana GF_SECURITY_ADMIN_PASSWORD absent; unit mode 0600

5. Findings

R-156 — papra's data is neither persisted nor backed up, and it reports healthy

New. Register grepped first: papra appears in R-41 (never-deployable / healthcheck) and R-127 (data_key misclassification); neither covers this. (Also corrected en route: a first grep suggested R-164 was taken — it is a mis-citation of controller v0.164.0, already flagged in REPORT-record-correction-2026-07-29.md:141. Highest real number is R-155, so this is R-156.)

The template mounts papra_data:/app/data, but the app persists to /app/app-data/db/db.sqlite. The mounted volume is therefore empty, and the real database lives in the container's writable layer.

Evidence (r156-papra-volume.txt), all four legs measured:

1. volume papra_papra_data -> /app/data
2. uid=999(nonroot);  drwxr-xr-x 2 root root /app/data
   touch: cannot touch '/app/data/.w': Permission denied
3. /app/data contains only the canary this campaign wrote AS ROOT
4. /app/app-data/db/db.sqlite  475136 bytes  owned nonroot   <-- the real DB, in no volume
5. health status: "healthy"

Consequences. (a) papra's documents do not survive docker compose down/redeploy/host restore — the writable layer is not persisted. (b) DumpAppVolumes dumps the volume, so papra is backed up as an empty directory: its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. (c) The healthcheck only probes the HTTP port, so none of this surfaces — the app is green throughout.

This is the presence is not success family from CLAUDE.md: a backup exists, a healthcheck passes, and neither means the data is there. It is also a second instance of the pattern R-41 names — an app that deploys and looks fine while being fundamentally broken.

Two-repo shape, if fixed: the volume target belongs in app-catalog-felhom.eu, and the general defence is a catalog gate asserting each template's mounted volume is a path the app actually writes. Not fixed here — the fences forbid it, and a fix mid-run would prove a version that did not exist when the run started.

5.2 — Investigated and DISPROVED: /api/disks reports state=attached for an absent drive

Recorded because the chase is the useful artefact, not because anything is broken.

With adatok physically removed, /api/disks kept reporting state: "attached" for it across three polls (~90 s) — while /dev/sdb was gone, /mnt/adatok was absent from /proc/mounts, and the guest bind returned Input/output error. The other fields were right (backing_device: "", guest_attached: false, bound_under_parent: false).

It looked like the R-116/R-117 family, and there is a real consumer: internal/web/intermediary.go:230 — present[d.MountPath] |= d.State == "attached". A presence gate reading an absent drive as present is precisely I3's failure mode.

It is inert, and the gate is correct. planDriveGates only gates paths under StableParentDir+"/" (/mnt/felhom-drives/…). The registered path is the stable one, whose presence is computed from d.BoundUnderParent — observed false — not from State. The State-derived entry is keyed on the raw /mnt/adatok, which is not a registered stable path, so the loop skips it. The observable end-to-end behaviour agrees: the storage page showed "Meghajtó leválasztva: Adatok" with a disconnect timestamp and a reconnect action, i.e. the gate fired.

The stale state field comes from the registry row surviving in the union after the storage-observation row drops out (a raw enrolled drive is not a PVE storage), carrying its configured MountPath. No R-n minted — no consumer is misled, and the R-116 comment's claim that this "cannot make the gate read an absent drive as PRESENT" held under test.

Not filed

• demo-hp SSH key — already R-129.
• Tier 3 isolation — §3; two recorded deliberate positions, not a defect.
• /api/disks state — §5.2, disproved.

---

6. RTO/RPO — measured

28 restores, every one returning the correct discriminator. Wall-clock is measured from the restore request to the app serving the correct data (the canary read back over rallly's real network path), not to "restore returned".

Tier	App	Data volume	n	min	median	p90	max	mean
Tier 1 (local)	rallly (postgres)	66 MB volume / 67 MB recovery unit	28	38.8 s	42.0 s	42.5 s	44.3 s	42.0 s

Which band this covers: the S band only, and only its lower end. 66 MB is a small app by any reading, and the spread is tight — 5.5 s between min and max over 28 runs — which says the cost is dominated by fixed work (stack stop, volume restore, stack start, health wait) rather than by bytes. Nothing here licenses an extrapolation to M or L: the byte-proportional term is precisely what 66 MB fails to exercise. A second point at a much larger volume is needed before any curve is claimed, and none was taken.

RPO is NOT measured. It is a function of backup frequency, and the soak drove backups on demand rather than on the schedule, so the observed intervals say nothing about the product's RPO.

---

7. I1–I11 across cycles

Full table in §4b. I1, I2 (and both pairs), I3, I4, I5, I6, I7, I10 and I11 held over 27 consecutive cycles — 586 checks, 0 violations. No violation was deterministic and none was racy, because there were none; the honest statement is that the run found no drift within 27 cycles, not that none exists beyond them.

I8 was not checked — Tier 3 never ran (§3). I9 was not automated — see §8.

---

8. What did not run, and why

Phase A overran badly (~5.5 h against the brief's ~1 h) — 1.26.1 is a public release image with no auto-install path, so day-0 was a blind screendump-and-sendkey walk; the Tier-3 question (§3) had to be settled before a customer could exist at all, since dr_tier is fixed at creation; and a false start on drive enrolment had to be backed out. Phase B was therefore deliberately not launched at 04:00 — the brief's own fence, "a test rig producing false negatives is worse than no rig" — and was built and run the next morning instead, where its first cycle could be watched.

Atom coverage: 6 of the brief's ~12 suggested families. Stated plainly rather than implied:

Atom (brief B1)	Ran
back up Tier 1/2 · restore + verify by discriminator · detach/reattach target · detach/reattach non-target · delete an app and redeploy · reboot the guest	YES
back up / restore Tier 3	NO — §3, structurally impossible to isolate
abort a filesystem in place without the device disappearing (R-117 Q7)	NO
kill the agent mid-backup · hard-reset the VM mid-write · reboot the VM	NO
concurrent backup+restore · concurrent backup+detach	NO
fill a drive to near-full and continue	NO

kill_controller is not "mid-backup". It restarts the controller at a permuted point in the cycle, so it sometimes lands after a backup and sometimes nowhere near one. It exercises recovery, not the mid-write race the brief asked for. Recorded because the atom name would otherwise overclaim.

reboot_guest was appended after the shuffle, not permuted into it, so it always ran last in cycles 7/14/21 and never interleaved with a detach. Reboot-during-detach is untested.

I8 — not checked; Tier 3 never ran. I9 (Tier-1/2 restore with the guest's app.yaml moved aside, data read over the app's real network path) — not automated. It was proven on this exact controller 0.188.0 by the tester-gate run hours earlier (tester-gate-golden-0.188.0-2026-07-31.md §5, step 6), and that is cited rather than re-claimed: this campaign did not execute it.

The tail is untested, not clean. 27 cycles is the count; the brief's interest in "the state that quietly drifts on the thirty-eighth" is not answered by a 27-cycle run.

---

9. Teardown — OWED, nothing removed yet

The rig is intact on purpose, so Phase B can run without repeating Phase A. Every item below is outstanding and must be removed when the campaign ends:

Layer	Item	Command
VM	311 c10-appliance on demo-hp	qm stop 311 && qm destroy 311 --purge
storage	c10-scratch dir storage	pvesm remove c10-scratch
PBS	datastore felhom-c10, user c10@pbs, token !box, both ACLs	proxmox-backup-manager datastore remove felhom-c10 · user delete c10@pbs · remove /mnt/5_hdd/backup/campaign10
restic	subaccount u629488-sub4 (felhom-campaign10) on box 611714 — unused, since Tier 3 never ran	DELETE /v1/storage_boxes/611714/subaccounts/281530
hub	customer c10-soak — disposition: DELETE. Named explicitly per R-131, which is four orphaned scratch customers left by exactly this omission	POST /configs/c10-soak/delete with ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=c10-soak
secrets	~/.config/campaign10/ on DooPlex (host + dashboard passwords, API key, app secrets, PBS token, restic password); /root/c10/ and /root/c10api.sh in guest 9201	shred -u both

sess-f ("R-120 golden 0.186.0 proof") is still present and is not this run's record — R-131 stands, untouched.

---

10. Hygiene note

The customer API key for c10-soak was printed into this session's transcript while being read from the hub page. It belongs to a scratch customer scheduled for deletion (§9) and grants nothing once that record is gone, but the write-out was avoidable and is recorded here rather than left implicit — the same class as R-132, where a correctly-made request was undone by how it was reported.