felhom.eu/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md

CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01)

Status: Phase A COMPLETE and gated. Phase B NOT STARTED — deliberately, see §8. One new finding: R-156. One structural scope constraint: Tier 3 cannot be isolated (§3).

Evidence: ../tests/campaign10-evidence-2026-07-31/ — state/phaseA-journal.md (step-by-step), isolation/pbs-denial.txt, isolation/restic-denial.txt, r156-papra-volume.txt.

---

1. Venue and baselines

Host	demo-hp (HP t740), Tier 0, the designated drill/build host
VM	311 c10-appliance — q35/OVMF (pre-enrolled-keys=0), 4 cores, 8 GB, cpu=host
Disks	scsi0 200 G system · scsi1 50 G · scsi2 50 G, qcow2 on c10-scratch
Storage	c10-scratch, dir at /mnt/nvme-1tb — the mount ROOT, content=images
Box	c10.felhom.eu @ 192.168.0.105 (demo-hp's vmbr0, 192.168.0.87/24)
Console	no browser on DooPlex → qm monitor screendump → PNG, read visually; sendkey for input
Untouched	drill-r50 (VM 300), guest 9201 on both demo boxes, local-lvm, both backup targets, ep0, the Storage Box

The exactMount choice, made deliberately (the brief asks which). c10-scratch sits at the mount root, not a subdirectory. A dir storage at a subdirectory sets exactMount=false → reachable=false → StorageStateDisconnected (felhom-agent/internal/storage/observe.go:336), which would have emitted storage_disconnected for demo-hp for the entire run — the exact signal I1/I2 exist to discriminate. It coexists with felhom-backup on the same path, which was not modified.

Baselines — every value read fresh

What	Value	How
controller main	v0.188.0	CHANGELOG top; HEAD=origin/main=4115e88, clean
golden's baked controller	0.188.0 (7353d8be…)	hub /configuration, selected option
agent published + vouched	0.119.0 (e37aca82…)	hub /configuration, selected option
hub	0.86.0	live deploy image tag == CHANGELOG top
ISO	1.26.1, f3cc86d5…, 1 705 322 496 B	live round trip from iso.felhom.eu
felhom.eu HEAD	e9a74a0, clean	git rev-parse
managed floor	v0.156.0 (DB override)	hub /configuration

Golden vs main: not behind — both 0.188.0. The deliberate decision the brief asks for is therefore vacuous this run; the campaign tests what is shipping and what is current.

Two of the brief's assumptions were stale; reading fresh caught both.

1. The brief was written against ISO v1.25.0. iso.felhom.eu began serving 1.26.1 at 18:37 Z the same day, ~90 minutes before this run. Operator chose 1.26.1 — the current published artifact.
2. The brief and three docs say demo-hp has no baked SSH key; ssh -o BatchMode=yes demo-hp authenticated by key, first try. Already R-129; not re-filed.

---

2. The isolation gate — §2

Target	Isolation	Proof
PBS	dedicated datastore felhom-c10 on DooPlex + auth-id c10@pbs!box scoped to it alone	CAPTURED — isolation/pbs-denial.txt
restic	scratch subaccount u629488-sub4, own chroot home felhom-campaign10	CAPTURED — isolation/restic-denial.txt
hub	the campaign's own record c10-soak; no other customer touched	CAPTURED — §2.3

Both denials carry a POSITIVE CONTROL, and the first attempt failed it. The initial PBS capture returned four tidy 403s — including on the campaign's own datastore. All four were worthless: PBS API tokens carry privilege separation, so a token's rights are the intersection of the user's ACL and the token's, and user c10@pbs had none. Granting the scoped role to the user as well turned the control green and left the denials standing. Without the control this would have shipped as a clean isolation proof that proved only that the credential was broken.

## POSITIVE CONTROL — same token, its OWN datastore
GET /api2/json/admin/datastore/felhom-c10/snapshots     {"data":[]}   HTTP 200

## DENIAL 1 — production datastore felhom-spike
permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-spike   403
## DENIAL 3 — DELETE namespace demo-felhom (destructive)
permission check failed - missing Datastore.Modify on /datastore/felhom-spike                   403

restic, same shape — the credential writes and reads back in its own home, and every production path is outside the chroot:

POSITIVE CONTROL  put/get/rm c10-canary.txt in own home        → succeeds, content verified
DENIAL  cd /home/felhom-peti-felhom      → stat remote: No such file or directory
DENIAL  ls ..                            → remote readdir("/home/../"): Permission denied

2.3 — hub. The run created exactly one record, c10-soak, and bound exactly one appliance (uuid 55ab3776-…) to it. Existing customers (demo-felhom, demo-hp, peti-felhom, drill-r50, sess-f) were read but never written. Verified from the hub's own log: every c10-soak line is the campaign's, and no line names another customer.

---

3. Scope constraint — Tier 3 cannot be isolated, so it was not run

The brief requires the campaign's PBS to be a dedicated datastore and namespace on DooPlex. That is unreachable, for two reasons that are each an already-recorded deliberate position:

1. Offsite hard-requires the DR tier — hub/internal/web/configs.go:1300 refuses outright: "Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS key)". Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4).
2. The DR tier only ever provisions on ep0 — peer allocation and endpoint sync use the lowest endpoint_id only; per-endpoint allocation is an explicitly deferred arc (hub/README.md:260).

So the only two configurations are DR tier on → campaign PBS traffic lands on ep0 (Tier 2, the offsite of last resort, RAM-constrained per R-90, fenced by §3), or DR tier off → no Tier 3 at all. Chosen: DR tier OFF, offsite OFF — the only option §3 permits.

Consequence, stated up front rather than discovered in the results: the campaign touches neither ep0 nor the Storage Box at any point — stronger isolation than asked for, obtained by not running the tier. Lost: every Tier-3 atom, I8 entirely, and the Tier-3 RTO/RPO rows. Unaffected: the two-drive core (R-112/113/114/116/117), D5, and I1–I7, I9–I11.

No R-n minted — the register was grepped first and both halves are already recorded. This is a campaign constraint, not a defect.

---

4. Phase A gates

Gate	Result	Positive observable
A1 install from the published ISO	PASS	ii felhom-bootstrap 1.26.1; unit enabled, fired at first boot; registered unclaimed with pairing code FD6-A6S, MACs, hardware and 3 SSH host keys; 4.2 GB actually written, system booted on its own
A2 claim	PASS	discriminator flipped dashboard not yet claimed → authentication required; hub: [claim] customer c10-soak CLAIMED its dashboard
A3 enrol both drives, apps across them	PASS	both registered via the real endpoint; backup target healthy; 4 apps healthy
A4 discriminators	PASS	all four apps seed and read back the same generation
A5 budget + watchdog	NOT DONE	§8

A1 — 1.26.1 cannot auto-install, verified against the bytes (not inferred from its manifest): auto-installer-mode.toml 0 hits, answer.toml 0, proxmox-auto-install 0, exactly 2 interactive menuentries. Driven blind through the Terminal UI (the graphical entry is undrivable — monitor mouse events do not move the guest cursor). Three traps hit and handled:

• GRUB's default is the graphical entry; the first down landed after the 15 s countdown. Reset and sent it inside the window in one round trip.
• The guest keymap was Hungarian while sendkey emits US scancodes. Every symbol afterwards — notably @, which is AltGr+V on a Hungarian layout — would have been silently mangled, including the 24-char root password, locking the box out with no way back in. Switched to U.S. English before typing anything; confirmed positively when c10@felhom.eu rendered a correct @ and ..
• The recorded boot-order trap: --boot set in its own qm set after the disks existed and verified from qm config; flipped to order=scsi0 post-install and verified again.

Only scsi0 was attached during the install, so the installer offered exactly one target and could not reach a data drive. The 50 G drives were attached afterwards.

A2 — the installer's root password stops working at day-0; it is replaced by a hub-vaulted managed credential (vaulted break-glass recovery credential … secret 32 chars), retrieved via POST /hosts/c10-soak-8a405e/reveal-recovery-credential. R-119 confirmed live again: the claim code is emailed-only and the operator relayed it — the single human step.

A3 — both drives enrolled through the real endpoint (POST /api/storage/attach), which does format → assign → guest-attach → register-stable-path in order:

Drive	Device	Stable path	Role
adatok	/dev/sdb	/mnt/felhom-drives/adatok	app data, set_default
mentes	/dev/sdc	/mnt/felhom-drives/mentes	backup target (PVE storage felhom-backup at /mnt/mentes)

A false start worth recording: hand-driving format + assign first left the drives mounted but unregistered — invisible to both the raw-candidate scan and /disks, so the real enroll endpoint could not resolve a UUID for them (resolveEnrollUUID sees neither list). Backed out (eject leaves the raw /mnt/<name> mount by design, disks.go:606, so the mounts also had to be undone) and re-ran the real flow clean. The lesson is the brief's own: use the real endpoint, or manufacture a state the product never produces.

Backup-target offer flow (R-112/R-114/E-2) exercised as the customer would: assign returned restart_required: true, and the state only flipped after the agent restart it asked for —

before: {"degraded":true, "target":"local", "offer_path":"/mnt/felhom-drives/adatok", "message":"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after:  {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}

That healthy read is the I5/I6 baseline: everything healthy ⇒ no degraded banner, target reads healthy, zero absent events.

Apps — four, healthy, covering both sides of D5's split:

App	Data	Secret classes	Why chosen
rallly	postgres (rallly-postgres)	2 × type: secret (SECRET_PASSWORD hex:32 labelled Titkosítási kulcs, DB_PASSWORD)	the DB app; travelling class
homebox	sqlite volume	1 × secret, data_key: true	data-encrypting key
papra	see R-156	1 × secret, data_key: true	data-encrypting key
grafana	volume	1 × type: password	the withheld class — I10's negative half

4 travelling secret fields and 1 password field that must never travel, so I10 can assert both directions rather than only the positive one.

A4 — discriminators work. Every app carries a generation marker; a full seed/read cycle returns the seeded generation from all four. rallly's is a row in a cc_proof table read over the path DATABASE_URL actually names (docker run --network container:rallly … psql -h rallly-postgres), never 127.0.0.1 inside the postgres container — the trusted-socket path that produced D5's §1.2 false pass.

---

5. Findings

R-156 — papra's data is neither persisted nor backed up, and it reports healthy

New. Register grepped first: papra appears in R-41 (never-deployable / healthcheck) and R-127 (data_key misclassification); neither covers this. (Also corrected en route: a first grep suggested R-164 was taken — it is a mis-citation of controller v0.164.0, already flagged in REPORT-record-correction-2026-07-29.md:141. Highest real number is R-155, so this is R-156.)

The template mounts papra_data:/app/data, but the app persists to /app/app-data/db/db.sqlite. The mounted volume is therefore empty, and the real database lives in the container's writable layer.

Evidence (r156-papra-volume.txt), all four legs measured:

1. volume papra_papra_data -> /app/data
2. uid=999(nonroot);  drwxr-xr-x 2 root root /app/data
   touch: cannot touch '/app/data/.w': Permission denied
3. /app/data contains only the canary this campaign wrote AS ROOT
4. /app/app-data/db/db.sqlite  475136 bytes  owned nonroot   <-- the real DB, in no volume
5. health status: "healthy"

Consequences. (a) papra's documents do not survive docker compose down/redeploy/host restore — the writable layer is not persisted. (b) DumpAppVolumes dumps the volume, so papra is backed up as an empty directory: its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. (c) The healthcheck only probes the HTTP port, so none of this surfaces — the app is green throughout.

This is the presence is not success family from CLAUDE.md: a backup exists, a healthcheck passes, and neither means the data is there. It is also a second instance of the pattern R-41 names — an app that deploys and looks fine while being fundamentally broken.

Two-repo shape, if fixed: the volume target belongs in app-catalog-felhom.eu, and the general defence is a catalog gate asserting each template's mounted volume is a path the app actually writes. Not fixed here — the fences forbid it, and a fix mid-run would prove a version that did not exist when the run started.

Not filed

• demo-hp SSH key — already R-129.
• Tier 3 isolation — §3; two recorded deliberate positions, not a defect.

---

6. RTO/RPO

No measurements. They are a byproduct of Phase B restores, which did not run. Nothing is extrapolated; no band is claimed.

---

7. I1–I11

Not measured across cycles. Only the healthy baseline (I5/I6) and the discriminator mechanism (I7) were established, as Phase A gates. A clean invariant is only a result with a cycle count behind it, and there is no count to report.

---

8. What did not run, and why

Phase B did not start. Phase A was budgeted at ~1 h by the brief and took ~5.5 h, almost entirely in places the brief could not have predicted:

• ISO 1.26.1 turned out to be a public release image with no auto-install path, so the install was a blind screendump-and-sendkey walk (~1.5 h) rather than an unattended boot.
• The Tier-3 investigation (§3) had to be settled before a customer could be created at all, because dr_tier is decided at customer-creation time.
• The enrol false start, and the CSRF/session/Bearer probing needed to drive the real endpoints.

That put the runner — the most correctness-sensitive component, the thing that decides destructive atoms unattended and judges eleven invariants — at ~04:00 with ~3 h of usable night. I stopped rather than write it there. The brief's own fence is the reason: "A test rig producing false negatives is worse than no rig." A soak whose judgements cannot be trusted would not just waste the night, it would contaminate the register with findings nobody could rely on.

Not run: every B1 atom; I1–I11 across cycles; §B3 RTO/RPO; A5's disk budget, watchdog and heartbeat (A5 exists to bound Phase B and was not written in isolation from it).

What is ready. The rig is built, gated and idle: VM 311 installed and claimed, two drives enrolled with the backup target healthy, four apps healthy spanning both D5 secret classes, a working discriminator across all four, an authenticated API path onto the controller, and both isolation denials captured. The driver library (c10lib.py) already implements the parts that took the longest — the jump-host command path, the authenticated API helper, canary seed/read, and the red-proofed psql read path. Phase B needs the atom set, the invariant checks and the watchdog on top of that, then it can run in daylight where its first cycles can be watched.

---

9. Teardown — OWED, nothing removed yet

The rig is intact on purpose, so Phase B can run without repeating Phase A. Every item below is outstanding and must be removed when the campaign ends:

Layer	Item	Command
VM	311 c10-appliance on demo-hp	qm stop 311 && qm destroy 311 --purge
storage	c10-scratch dir storage	pvesm remove c10-scratch
PBS	datastore felhom-c10, user c10@pbs, token !box, both ACLs	proxmox-backup-manager datastore remove felhom-c10 · user delete c10@pbs · remove /mnt/5_hdd/backup/campaign10
restic	subaccount u629488-sub4 (felhom-campaign10) on box 611714 — unused, since Tier 3 never ran	DELETE /v1/storage_boxes/611714/subaccounts/281530
hub	customer c10-soak — disposition: DELETE. Named explicitly per R-131, which is four orphaned scratch customers left by exactly this omission	POST /configs/c10-soak/delete with ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=c10-soak
secrets	~/.config/campaign10/ on DooPlex (host + dashboard passwords, API key, app secrets, PBS token, restic password); /root/c10/ and /root/c10api.sh in guest 9201	shred -u both

sess-f ("R-120 golden 0.186.0 proof") is still present and is not this run's record — R-131 stands, untouched.

---

10. Hygiene note

The customer API key for c10-soak was printed into this session's transcript while being read from the hub page. It belongs to a scratch customer scheduled for deletion (§9) and grants nothing once that record is gone, but the write-out was avoidable and is recorded here rather than left implicit — the same class as R-132, where a correctly-made request was undone by how it was reported.