felhom.eu/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md

# CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01)

**Status: Phase A COMPLETE and gated. Phase B NOT STARTED — deliberately, see §8.**
**One new finding: R-156. One structural scope constraint: Tier 3 cannot be isolated (§3).**

Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step),
`isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`.

---

## 1. Venue and baselines

| | |
|---|---|
| Host | `demo-hp` (HP t740), **Tier 0**, the designated drill/build host |
| VM | **311 `c10-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` |
| Disks | `scsi0` 200 G system · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c10-scratch` |
| Storage | **`c10-scratch`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images` |
| Box | `c10.felhom.eu` @ `192.168.0.105` (demo-hp's vmbr0, `192.168.0.87/24`) |
| Console | no browser on DooPlex → `qm monitor screendump` → PNG, read visually; `sendkey` for input |
| Untouched | `drill-r50` (VM 300), guest 9201 on both demo boxes, `local-lvm`, both backup targets, ep0, the Storage Box |

**The `exactMount` choice, made deliberately (the brief asks which).** `c10-scratch` sits at the mount
**root**, not a subdirectory. A dir storage at a subdirectory sets `exactMount=false` →
`reachable=false` → `StorageStateDisconnected` (`felhom-agent/internal/storage/observe.go:336`), which
would have emitted `storage_disconnected` for demo-hp for the entire run — **the exact signal I1/I2
exist to discriminate**. It coexists with `felhom-backup` on the same path, which was not modified.

### Baselines — every value read fresh

| What | Value | How |
|---|---|---|
| controller `main` | **v0.188.0** | CHANGELOG top; `HEAD`=`origin/main`=`4115e88`, clean |
| **golden's baked controller** | **0.188.0** (`7353d8be…`) | hub `/configuration`, selected option |
| agent published + vouched | **0.119.0** (`e37aca82…`) | hub `/configuration`, selected option |
| hub | **0.86.0** | live deploy image tag == CHANGELOG top |
| ISO | **1.26.1**, `f3cc86d5…`, 1 705 322 496 B | live round trip from `iso.felhom.eu` |
| `felhom.eu` HEAD | `e9a74a0`, clean | `git rev-parse` |
| managed floor | v0.156.0 (DB override) | hub `/configuration` |

**Golden vs `main`: not behind — both 0.188.0.** The deliberate decision the brief asks for is
therefore vacuous this run; the campaign tests what is shipping *and* what is current.

**Two of the brief's assumptions were stale; reading fresh caught both.**

1. The brief was written against ISO **v1.25.0**. `iso.felhom.eu` began serving **1.26.1** at 18:37 Z
   the same day, ~90 minutes before this run. Operator chose 1.26.1 — the current published artifact.
2. The brief and three docs say demo-hp has **no baked SSH key**; `ssh -o BatchMode=yes demo-hp`
   authenticated **by key, first try**. Already **R-129**; **not re-filed**.

---

## 2. The isolation gate — §2

| Target | Isolation | Proof |
|---|---|---|
| **PBS** | dedicated datastore `felhom-c10` on DooPlex + auth-id `c10@pbs!box` scoped to it alone | **CAPTURED** — `isolation/pbs-denial.txt` |
| **restic** | scratch subaccount `u629488-sub4`, own chroot home `felhom-campaign10` | **CAPTURED** — `isolation/restic-denial.txt` |
| **hub** | the campaign's own record `c10-soak`; no other customer touched | **CAPTURED** — §2.3 |

**Both denials carry a POSITIVE CONTROL, and the first attempt failed it.** The initial PBS capture
returned four tidy 403s — including on the campaign's *own* datastore. All four were worthless: PBS
API tokens carry privilege separation, so a token's rights are the intersection of the user's ACL and
the token's, and user `c10@pbs` had none. Granting the scoped role to the user as well turned the
control green and left the denials standing. Without the control this would have shipped as a clean
isolation proof that proved only that the credential was broken.

    ## POSITIVE CONTROL — same token, its OWN datastore
    GET /api2/json/admin/datastore/felhom-c10/snapshots     {"data":[]}   HTTP 200

    ## DENIAL 1 — production datastore felhom-spike
    permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-spike   403
    ## DENIAL 3 — DELETE namespace demo-felhom (destructive)
    permission check failed - missing Datastore.Modify on /datastore/felhom-spike                   403

restic, same shape — the credential writes and reads back in its own home, and every production path
is outside the chroot:

    POSITIVE CONTROL  put/get/rm c10-canary.txt in own home        → succeeds, content verified
    DENIAL  cd /home/felhom-peti-felhom      → stat remote: No such file or directory
    DENIAL  ls ..                            → remote readdir("/home/../"): Permission denied

**2.3 — hub.** The run created exactly one record, `c10-soak`, and bound exactly one appliance
(uuid `55ab3776-…`) to it. Existing customers (`demo-felhom`, `demo-hp`, `peti-felhom`, `drill-r50`,
`sess-f`) were read but never written. Verified from the hub's own log: every `c10-soak` line is the
campaign's, and no line names another customer.

---

## 3. Scope constraint — Tier 3 cannot be isolated, so it was not run

The brief requires the campaign's PBS to be *a dedicated datastore and namespace on DooPlex*. That is
unreachable, for two reasons that are each an **already-recorded deliberate position**:

1. **Offsite hard-requires the DR tier** — `hub/internal/web/configs.go:1300` refuses outright:
   *"Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS
   key)"*. Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4).
2. **The DR tier only ever provisions on ep0** — peer allocation and endpoint sync use the lowest
   `endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).

So the only two configurations are *DR tier on* → campaign PBS traffic lands on **ep0** (Tier 2, the
offsite of last resort, RAM-constrained per R-90, fenced by §3), or *DR tier off* → **no Tier 3 at
all**. Chosen: **DR tier OFF, offsite OFF** — the only option §3 permits.

Consequence, stated up front rather than discovered in the results: the campaign touches **neither ep0
nor the Storage Box at any point** — stronger isolation than asked for, obtained by not running the
tier. **Lost: every Tier-3 atom, I8 entirely, and the Tier-3 RTO/RPO rows.** Unaffected: the
two-drive core (R-112/113/114/116/117), D5, and I1–I7, I9–I11.

**No `R-n` minted** — the register was grepped first and both halves are already recorded. This is a
campaign constraint, not a defect.

---

## 4. Phase A gates

| Gate | Result | Positive observable |
|---|---|---|
| **A1** install from the published ISO | **PASS** | `ii felhom-bootstrap 1.26.1`; unit `enabled`, fired at first boot; registered unclaimed with pairing code `FD6-A6S`, MACs, hardware and 3 SSH host keys; **4.2 GB actually written**, system booted on its own |
| **A2** claim | **PASS** | discriminator flipped `dashboard not yet claimed` → `authentication required`; hub: `[claim] customer c10-soak CLAIMED its dashboard` |
| **A3** enrol both drives, apps across them | **PASS** | both registered via the real endpoint; backup target healthy; 4 apps healthy |
| **A4** discriminators | **PASS** | all four apps seed and read back the same generation |
| **A5** budget + watchdog | **NOT DONE** | §8 |

**A1 — 1.26.1 cannot auto-install, verified against the bytes** (not inferred from its manifest):
`auto-installer-mode.toml` 0 hits, `answer.toml` 0, `proxmox-auto-install` 0, exactly 2 interactive
menuentries. Driven blind through the **Terminal UI** (the graphical entry is undrivable — monitor
mouse events do not move the guest cursor). Three traps hit and handled:

- GRUB's default is the graphical entry; the first `down` landed after the 15 s countdown. Reset and
  sent it inside the window in one round trip.
- **The guest keymap was Hungarian while `sendkey` emits US scancodes.** Every symbol afterwards —
  notably `@`, which is AltGr+V on a Hungarian layout — would have been silently mangled, *including
  the 24-char root password*, locking the box out with no way back in. Switched to U.S. English
  before typing anything; confirmed positively when `c10@felhom.eu` rendered a correct `@` and `.`.
- The recorded boot-order trap: `--boot` set in its **own** `qm set` after the disks existed and
  verified from `qm config`; flipped to `order=scsi0` post-install and verified again.

Only `scsi0` was attached during the install, so the installer offered exactly one target and could
not reach a data drive. The 50 G drives were attached afterwards.

**A2** — the installer's root password stops working at day-0; it is replaced by a hub-vaulted
managed credential (`vaulted break-glass recovery credential … secret 32 chars`), retrieved via
`POST /hosts/c10-soak-8a405e/reveal-recovery-credential`. R-119 confirmed live again: the claim code
is emailed-only and the operator relayed it — the single human step.

**A3** — both drives enrolled through **the real endpoint** (`POST /api/storage/attach`), which does
format → assign → guest-attach → register-stable-path in order:

| Drive | Device | Stable path | Role |
|---|---|---|---|
| `adatok` | `/dev/sdb` | `/mnt/felhom-drives/adatok` | app data, `set_default` |
| `mentes` | `/dev/sdc` | `/mnt/felhom-drives/mentes` | **backup target** (PVE storage `felhom-backup` at `/mnt/mentes`) |

A false start worth recording: hand-driving `format` + `assign` first left the drives **mounted but
unregistered** — invisible to both the raw-candidate scan and `/disks`, so the real enroll endpoint
could not resolve a UUID for them (`resolveEnrollUUID` sees neither list). Backed out (eject leaves
the raw `/mnt/<name>` mount by design, `disks.go:606`, so the mounts also had to be undone) and re-ran
the real flow clean. **The lesson is the brief's own:** use the real endpoint, or manufacture a state
the product never produces.

Backup-target offer flow (R-112/R-114/E-2) exercised as the customer would: assign returned
`restart_required: true`, and the state only flipped after the agent restart it asked for —

    before: {"degraded":true, "target":"local", "offer_path":"/mnt/felhom-drives/adatok", "message":"A rendszermentés jelenleg ugyanazon a lemezen van…"}
    after:  {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}

That healthy read is the **I5/I6 baseline**: everything healthy ⇒ no degraded banner, target reads
healthy, zero absent events.

**Apps — four, healthy, covering both sides of D5's split:**

| App | Data | Secret classes | Why chosen |
|---|---|---|---|
| **rallly** | postgres (`rallly-postgres`) | 2 × `type: secret` (`SECRET_PASSWORD` hex:32 labelled *Titkosítási kulcs*, `DB_PASSWORD`) | the DB app; travelling class |
| **homebox** | sqlite volume | 1 × `secret`, **`data_key: true`** | data-encrypting key |
| **papra** | see **R-156** | 1 × `secret`, **`data_key: true`** | data-encrypting key |
| **grafana** | volume | 1 × **`type: password`** | **the withheld class — I10's negative half** |

4 travelling `secret` fields and 1 `password` field that must never travel, so I10 can assert both
directions rather than only the positive one.

**A4 — discriminators work.** Every app carries a generation marker; a full seed/read cycle returns
the seeded generation from all four. rallly's is a row in a `cc_proof` table read over **the path
`DATABASE_URL` actually names** (`docker run --network container:rallly … psql -h rallly-postgres`),
never `127.0.0.1` inside the postgres container — the trusted-socket path that produced D5's §1.2
false pass.

---

## 5. Findings

### R-156 — papra's data is neither persisted nor backed up, and it reports healthy

**New. Register grepped first:** papra appears in R-41 (never-deployable / healthcheck) and R-127
(`data_key` misclassification); neither covers this. *(Also corrected en route: a first grep suggested
R-164 was taken — it is a mis-citation of controller **v0.164.0**, already flagged in
`REPORT-record-correction-2026-07-29.md:141`. Highest real number is R-155, so this is R-156.)*

The template mounts `papra_data:/app/data`, but the app persists to **`/app/app-data/db/db.sqlite`**.
The mounted volume is therefore empty, and the real database lives in the container's writable layer.

Evidence (`r156-papra-volume.txt`), all four legs measured:

    1. volume papra_papra_data -> /app/data
    2. uid=999(nonroot);  drwxr-xr-x 2 root root /app/data
       touch: cannot touch '/app/data/.w': Permission denied
    3. /app/data contains only the canary this campaign wrote AS ROOT
    4. /app/app-data/db/db.sqlite  475136 bytes  owned nonroot   <-- the real DB, in no volume
    5. health status: "healthy"

**Consequences.** (a) papra's documents do not survive `docker compose down`/redeploy/host restore —
the writable layer is not persisted. (b) `DumpAppVolumes` dumps the volume, so papra is **backed up
as an empty directory**: its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. (c) The
healthcheck only probes the HTTP port, so none of this surfaces — the app is green throughout.

This is the *presence is not success* family from `CLAUDE.md`: a backup exists, a healthcheck passes,
and neither means the data is there. It is also a second instance of the pattern R-41 names — an app
that deploys and looks fine while being fundamentally broken.

**Two-repo shape**, if fixed: the volume target belongs in `app-catalog-felhom.eu`, and the general
defence is a catalog gate asserting each template's mounted volume is a path the app actually writes.
**Not fixed here** — the fences forbid it, and a fix mid-run would prove a version that did not exist
when the run started.

### Not filed

- **demo-hp SSH key** — already R-129.
- **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect.

---

## 6. RTO/RPO

**No measurements.** They are a byproduct of Phase B restores, which did not run. Nothing is
extrapolated; no band is claimed.

---

## 7. I1–I11

**Not measured across cycles.** Only the healthy baseline (I5/I6) and the discriminator mechanism (I7)
were established, as Phase A gates. A clean invariant is only a result with a cycle count behind it,
and there is no count to report.

---

## 8. What did not run, and why

**Phase B did not start.** Phase A was budgeted at ~1 h by the brief and took ~5.5 h, almost entirely
in places the brief could not have predicted:

- ISO **1.26.1** turned out to be a public release image with no auto-install path, so the install was
  a blind screendump-and-`sendkey` walk (~1.5 h) rather than an unattended boot.
- The Tier-3 investigation (§3) had to be settled before a customer could be created at all, because
  `dr_tier` is decided at customer-creation time.
- The enrol false start, and the CSRF/session/Bearer probing needed to drive the real endpoints.

That put the runner — the most correctness-sensitive component, the thing that decides destructive
atoms unattended and judges eleven invariants — at ~04:00 with ~3 h of usable night. **I stopped
rather than write it there.** The brief's own fence is the reason: *"A test rig producing false
negatives is worse than no rig."* A soak whose judgements cannot be trusted would not just waste the
night, it would contaminate the register with findings nobody could rely on.

**Not run:** every B1 atom; I1–I11 across cycles; §B3 RTO/RPO; A5's disk budget, watchdog and
heartbeat (A5 exists to bound Phase B and was not written in isolation from it).

**What is ready.** The rig is built, gated and idle: VM 311 installed and claimed, two drives enrolled
with the backup target healthy, four apps healthy spanning both D5 secret classes, a working
discriminator across all four, an authenticated API path onto the controller, and both isolation
denials captured. The driver library (`c10lib.py`) already implements the parts that took the
longest — the jump-host command path, the authenticated API helper, canary seed/read, and the
red-proofed psql read path. **Phase B needs the atom set, the invariant checks and the watchdog on
top of that, then it can run in daylight where its first cycles can be watched.**

---

## 9. Teardown — OWED, nothing removed yet

The rig is intact **on purpose**, so Phase B can run without repeating Phase A. Every item below is
outstanding and must be removed when the campaign ends:

| Layer | Item | Command |
|---|---|---|
| VM | `311 c10-appliance` on demo-hp | `qm stop 311 && qm destroy 311 --purge` |
| storage | `c10-scratch` dir storage | `pvesm remove c10-scratch` |
| PBS | datastore `felhom-c10`, user `c10@pbs`, token `!box`, both ACLs | `proxmox-backup-manager datastore remove felhom-c10` · `user delete c10@pbs` · remove `/mnt/5_hdd/backup/campaign10` |
| restic | subaccount **`u629488-sub4`** (`felhom-campaign10`) on box 611714 — **unused, since Tier 3 never ran** | `DELETE /v1/storage_boxes/611714/subaccounts/281530` |
| **hub** | **customer `c10-soak`** — **disposition: DELETE.** Named explicitly per R-131, which is four orphaned scratch customers left by exactly this omission | `POST /configs/c10-soak/delete` with `ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=c10-soak` |
| secrets | `~/.config/campaign10/` on DooPlex (host + dashboard passwords, API key, app secrets, PBS token, restic password); `/root/c10/` and `/root/c10api.sh` in guest 9201 | `shred -u` both |

`sess-f` ("R-120 golden 0.186.0 proof") is still present and is **not** this run's record — R-131
stands, untouched.

---

## 10. Hygiene note

The customer API key for `c10-soak` was printed into this session's transcript while being read from
the hub page. It belongs to a scratch customer scheduled for deletion (§9) and grants nothing once
that record is gone, but the write-out was avoidable and is recorded here rather than left implicit —
the same class as R-132, where a correctly-made request was undone by how it was reported.