# CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01) **Status: Phase A COMPLETE and gated. Phase B NOT STARTED — deliberately, see §8.** **One new finding: R-156. One structural scope constraint: Tier 3 cannot be isolated (§3).** Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step), `isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`. --- ## 1. Venue and baselines | | | |---|---| | Host | `demo-hp` (HP t740), **Tier 0**, the designated drill/build host | | VM | **311 `c10-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` | | Disks | `scsi0` 200 G system · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c10-scratch` | | Storage | **`c10-scratch`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images` | | Box | `c10.felhom.eu` @ `192.168.0.105` (demo-hp's vmbr0, `192.168.0.87/24`) | | Console | no browser on DooPlex → `qm monitor screendump` → PNG, read visually; `sendkey` for input | | Untouched | `drill-r50` (VM 300), guest 9201 on both demo boxes, `local-lvm`, both backup targets, ep0, the Storage Box | **The `exactMount` choice, made deliberately (the brief asks which).** `c10-scratch` sits at the mount **root**, not a subdirectory. A dir storage at a subdirectory sets `exactMount=false` → `reachable=false` → `StorageStateDisconnected` (`felhom-agent/internal/storage/observe.go:336`), which would have emitted `storage_disconnected` for demo-hp for the entire run — **the exact signal I1/I2 exist to discriminate**. It coexists with `felhom-backup` on the same path, which was not modified. ### Baselines — every value read fresh | What | Value | How | |---|---|---| | controller `main` | **v0.188.0** | CHANGELOG top; `HEAD`=`origin/main`=`4115e88`, clean | | **golden's baked controller** | **0.188.0** (`7353d8be…`) | hub `/configuration`, selected option | | agent published + vouched | **0.119.0** (`e37aca82…`) | hub `/configuration`, selected option | | hub | **0.86.0** | live deploy image tag == CHANGELOG top | | ISO | **1.26.1**, `f3cc86d5…`, 1 705 322 496 B | live round trip from `iso.felhom.eu` | | `felhom.eu` HEAD | `e9a74a0`, clean | `git rev-parse` | | managed floor | v0.156.0 (DB override) | hub `/configuration` | **Golden vs `main`: not behind — both 0.188.0.** The deliberate decision the brief asks for is therefore vacuous this run; the campaign tests what is shipping *and* what is current. **Two of the brief's assumptions were stale; reading fresh caught both.** 1. The brief was written against ISO **v1.25.0**. `iso.felhom.eu` began serving **1.26.1** at 18:37 Z the same day, ~90 minutes before this run. Operator chose 1.26.1 — the current published artifact. 2. The brief and three docs say demo-hp has **no baked SSH key**; `ssh -o BatchMode=yes demo-hp` authenticated **by key, first try**. Already **R-129**; **not re-filed**. --- ## 2. The isolation gate — §2 | Target | Isolation | Proof | |---|---|---| | **PBS** | dedicated datastore `felhom-c10` on DooPlex + auth-id `c10@pbs!box` scoped to it alone | **CAPTURED** — `isolation/pbs-denial.txt` | | **restic** | scratch subaccount `u629488-sub4`, own chroot home `felhom-campaign10` | **CAPTURED** — `isolation/restic-denial.txt` | | **hub** | the campaign's own record `c10-soak`; no other customer touched | **CAPTURED** — §2.3 | **Both denials carry a POSITIVE CONTROL, and the first attempt failed it.** The initial PBS capture returned four tidy 403s — including on the campaign's *own* datastore. All four were worthless: PBS API tokens carry privilege separation, so a token's rights are the intersection of the user's ACL and the token's, and user `c10@pbs` had none. Granting the scoped role to the user as well turned the control green and left the denials standing. Without the control this would have shipped as a clean isolation proof that proved only that the credential was broken. ## POSITIVE CONTROL — same token, its OWN datastore GET /api2/json/admin/datastore/felhom-c10/snapshots {"data":[]} HTTP 200 ## DENIAL 1 — production datastore felhom-spike permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-spike 403 ## DENIAL 3 — DELETE namespace demo-felhom (destructive) permission check failed - missing Datastore.Modify on /datastore/felhom-spike 403 restic, same shape — the credential writes and reads back in its own home, and every production path is outside the chroot: POSITIVE CONTROL put/get/rm c10-canary.txt in own home → succeeds, content verified DENIAL cd /home/felhom-peti-felhom → stat remote: No such file or directory DENIAL ls .. → remote readdir("/home/../"): Permission denied **2.3 — hub.** The run created exactly one record, `c10-soak`, and bound exactly one appliance (uuid `55ab3776-…`) to it. Existing customers (`demo-felhom`, `demo-hp`, `peti-felhom`, `drill-r50`, `sess-f`) were read but never written. Verified from the hub's own log: every `c10-soak` line is the campaign's, and no line names another customer. --- ## 3. Scope constraint — Tier 3 cannot be isolated, so it was not run The brief requires the campaign's PBS to be *a dedicated datastore and namespace on DooPlex*. That is unreachable, for two reasons that are each an **already-recorded deliberate position**: 1. **Offsite hard-requires the DR tier** — `hub/internal/web/configs.go:1300` refuses outright: *"Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS key)"*. Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4). 2. **The DR tier only ever provisions on ep0** — peer allocation and endpoint sync use the lowest `endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`). So the only two configurations are *DR tier on* → campaign PBS traffic lands on **ep0** (Tier 2, the offsite of last resort, RAM-constrained per R-90, fenced by §3), or *DR tier off* → **no Tier 3 at all**. Chosen: **DR tier OFF, offsite OFF** — the only option §3 permits. Consequence, stated up front rather than discovered in the results: the campaign touches **neither ep0 nor the Storage Box at any point** — stronger isolation than asked for, obtained by not running the tier. **Lost: every Tier-3 atom, I8 entirely, and the Tier-3 RTO/RPO rows.** Unaffected: the two-drive core (R-112/113/114/116/117), D5, and I1–I7, I9–I11. **No `R-n` minted** — the register was grepped first and both halves are already recorded. This is a campaign constraint, not a defect. --- ## 4. Phase A gates | Gate | Result | Positive observable | |---|---|---| | **A1** install from the published ISO | **PASS** | `ii felhom-bootstrap 1.26.1`; unit `enabled`, fired at first boot; registered unclaimed with pairing code `FD6-A6S`, MACs, hardware and 3 SSH host keys; **4.2 GB actually written**, system booted on its own | | **A2** claim | **PASS** | discriminator flipped `dashboard not yet claimed` → `authentication required`; hub: `[claim] customer c10-soak CLAIMED its dashboard` | | **A3** enrol both drives, apps across them | **PASS** | both registered via the real endpoint; backup target healthy; 4 apps healthy | | **A4** discriminators | **PASS** | all four apps seed and read back the same generation | | **A5** budget + watchdog | **NOT DONE** | §8 | **A1 — 1.26.1 cannot auto-install, verified against the bytes** (not inferred from its manifest): `auto-installer-mode.toml` 0 hits, `answer.toml` 0, `proxmox-auto-install` 0, exactly 2 interactive menuentries. Driven blind through the **Terminal UI** (the graphical entry is undrivable — monitor mouse events do not move the guest cursor). Three traps hit and handled: - GRUB's default is the graphical entry; the first `down` landed after the 15 s countdown. Reset and sent it inside the window in one round trip. - **The guest keymap was Hungarian while `sendkey` emits US scancodes.** Every symbol afterwards — notably `@`, which is AltGr+V on a Hungarian layout — would have been silently mangled, *including the 24-char root password*, locking the box out with no way back in. Switched to U.S. English before typing anything; confirmed positively when `c10@felhom.eu` rendered a correct `@` and `.`. - The recorded boot-order trap: `--boot` set in its **own** `qm set` after the disks existed and verified from `qm config`; flipped to `order=scsi0` post-install and verified again. Only `scsi0` was attached during the install, so the installer offered exactly one target and could not reach a data drive. The 50 G drives were attached afterwards. **A2** — the installer's root password stops working at day-0; it is replaced by a hub-vaulted managed credential (`vaulted break-glass recovery credential … secret 32 chars`), retrieved via `POST /hosts/c10-soak-8a405e/reveal-recovery-credential`. R-119 confirmed live again: the claim code is emailed-only and the operator relayed it — the single human step. **A3** — both drives enrolled through **the real endpoint** (`POST /api/storage/attach`), which does format → assign → guest-attach → register-stable-path in order: | Drive | Device | Stable path | Role | |---|---|---|---| | `adatok` | `/dev/sdb` | `/mnt/felhom-drives/adatok` | app data, `set_default` | | `mentes` | `/dev/sdc` | `/mnt/felhom-drives/mentes` | **backup target** (PVE storage `felhom-backup` at `/mnt/mentes`) | A false start worth recording: hand-driving `format` + `assign` first left the drives **mounted but unregistered** — invisible to both the raw-candidate scan and `/disks`, so the real enroll endpoint could not resolve a UUID for them (`resolveEnrollUUID` sees neither list). Backed out (eject leaves the raw `/mnt/` mount by design, `disks.go:606`, so the mounts also had to be undone) and re-ran the real flow clean. **The lesson is the brief's own:** use the real endpoint, or manufacture a state the product never produces. Backup-target offer flow (R-112/R-114/E-2) exercised as the customer would: assign returned `restart_required: true`, and the state only flipped after the agent restart it asked for — before: {"degraded":true, "target":"local", "offer_path":"/mnt/felhom-drives/adatok", "message":"A rendszermentés jelenleg ugyanazon a lemezen van…"} after: {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"} That healthy read is the **I5/I6 baseline**: everything healthy ⇒ no degraded banner, target reads healthy, zero absent events. **Apps — four, healthy, covering both sides of D5's split:** | App | Data | Secret classes | Why chosen | |---|---|---|---| | **rallly** | postgres (`rallly-postgres`) | 2 × `type: secret` (`SECRET_PASSWORD` hex:32 labelled *Titkosítási kulcs*, `DB_PASSWORD`) | the DB app; travelling class | | **homebox** | sqlite volume | 1 × `secret`, **`data_key: true`** | data-encrypting key | | **papra** | see **R-156** | 1 × `secret`, **`data_key: true`** | data-encrypting key | | **grafana** | volume | 1 × **`type: password`** | **the withheld class — I10's negative half** | 4 travelling `secret` fields and 1 `password` field that must never travel, so I10 can assert both directions rather than only the positive one. **A4 — discriminators work.** Every app carries a generation marker; a full seed/read cycle returns the seeded generation from all four. rallly's is a row in a `cc_proof` table read over **the path `DATABASE_URL` actually names** (`docker run --network container:rallly … psql -h rallly-postgres`), never `127.0.0.1` inside the postgres container — the trusted-socket path that produced D5's §1.2 false pass. --- ## 5. Findings ### R-156 — papra's data is neither persisted nor backed up, and it reports healthy **New. Register grepped first:** papra appears in R-41 (never-deployable / healthcheck) and R-127 (`data_key` misclassification); neither covers this. *(Also corrected en route: a first grep suggested R-164 was taken — it is a mis-citation of controller **v0.164.0**, already flagged in `REPORT-record-correction-2026-07-29.md:141`. Highest real number is R-155, so this is R-156.)* The template mounts `papra_data:/app/data`, but the app persists to **`/app/app-data/db/db.sqlite`**. The mounted volume is therefore empty, and the real database lives in the container's writable layer. Evidence (`r156-papra-volume.txt`), all four legs measured: 1. volume papra_papra_data -> /app/data 2. uid=999(nonroot); drwxr-xr-x 2 root root /app/data touch: cannot touch '/app/data/.w': Permission denied 3. /app/data contains only the canary this campaign wrote AS ROOT 4. /app/app-data/db/db.sqlite 475136 bytes owned nonroot <-- the real DB, in no volume 5. health status: "healthy" **Consequences.** (a) papra's documents do not survive `docker compose down`/redeploy/host restore — the writable layer is not persisted. (b) `DumpAppVolumes` dumps the volume, so papra is **backed up as an empty directory**: its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. (c) The healthcheck only probes the HTTP port, so none of this surfaces — the app is green throughout. This is the *presence is not success* family from `CLAUDE.md`: a backup exists, a healthcheck passes, and neither means the data is there. It is also a second instance of the pattern R-41 names — an app that deploys and looks fine while being fundamentally broken. **Two-repo shape**, if fixed: the volume target belongs in `app-catalog-felhom.eu`, and the general defence is a catalog gate asserting each template's mounted volume is a path the app actually writes. **Not fixed here** — the fences forbid it, and a fix mid-run would prove a version that did not exist when the run started. ### Not filed - **demo-hp SSH key** — already R-129. - **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect. --- ## 6. RTO/RPO **No measurements.** They are a byproduct of Phase B restores, which did not run. Nothing is extrapolated; no band is claimed. --- ## 7. I1–I11 **Not measured across cycles.** Only the healthy baseline (I5/I6) and the discriminator mechanism (I7) were established, as Phase A gates. A clean invariant is only a result with a cycle count behind it, and there is no count to report. --- ## 8. What did not run, and why **Phase B did not start.** Phase A was budgeted at ~1 h by the brief and took ~5.5 h, almost entirely in places the brief could not have predicted: - ISO **1.26.1** turned out to be a public release image with no auto-install path, so the install was a blind screendump-and-`sendkey` walk (~1.5 h) rather than an unattended boot. - The Tier-3 investigation (§3) had to be settled before a customer could be created at all, because `dr_tier` is decided at customer-creation time. - The enrol false start, and the CSRF/session/Bearer probing needed to drive the real endpoints. That put the runner — the most correctness-sensitive component, the thing that decides destructive atoms unattended and judges eleven invariants — at ~04:00 with ~3 h of usable night. **I stopped rather than write it there.** The brief's own fence is the reason: *"A test rig producing false negatives is worse than no rig."* A soak whose judgements cannot be trusted would not just waste the night, it would contaminate the register with findings nobody could rely on. **Not run:** every B1 atom; I1–I11 across cycles; §B3 RTO/RPO; A5's disk budget, watchdog and heartbeat (A5 exists to bound Phase B and was not written in isolation from it). **What is ready.** The rig is built, gated and idle: VM 311 installed and claimed, two drives enrolled with the backup target healthy, four apps healthy spanning both D5 secret classes, a working discriminator across all four, an authenticated API path onto the controller, and both isolation denials captured. The driver library (`c10lib.py`) already implements the parts that took the longest — the jump-host command path, the authenticated API helper, canary seed/read, and the red-proofed psql read path. **Phase B needs the atom set, the invariant checks and the watchdog on top of that, then it can run in daylight where its first cycles can be watched.** --- ## 9. Teardown — OWED, nothing removed yet The rig is intact **on purpose**, so Phase B can run without repeating Phase A. Every item below is outstanding and must be removed when the campaign ends: | Layer | Item | Command | |---|---|---| | VM | `311 c10-appliance` on demo-hp | `qm stop 311 && qm destroy 311 --purge` | | storage | `c10-scratch` dir storage | `pvesm remove c10-scratch` | | PBS | datastore `felhom-c10`, user `c10@pbs`, token `!box`, both ACLs | `proxmox-backup-manager datastore remove felhom-c10` · `user delete c10@pbs` · remove `/mnt/5_hdd/backup/campaign10` | | restic | subaccount **`u629488-sub4`** (`felhom-campaign10`) on box 611714 — **unused, since Tier 3 never ran** | `DELETE /v1/storage_boxes/611714/subaccounts/281530` | | **hub** | **customer `c10-soak`** — **disposition: DELETE.** Named explicitly per R-131, which is four orphaned scratch customers left by exactly this omission | `POST /configs/c10-soak/delete` with `ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=c10-soak` | | secrets | `~/.config/campaign10/` on DooPlex (host + dashboard passwords, API key, app secrets, PBS token, restic password); `/root/c10/` and `/root/c10api.sh` in guest 9201 | `shred -u` both | `sess-f` ("R-120 golden 0.186.0 proof") is still present and is **not** this run's record — R-131 stands, untouched. --- ## 10. Hygiene note The customer API key for `c10-soak` was printed into this session's transcript while being read from the hub page. It belongs to a scratch customer scheduled for deletion (§9) and grants nothing once that record is gone, but the write-out was avoidable and is recorded here rather than left implicit — the same class as R-132, where a correctly-made request was undone by how it was reported.