felhom.eu/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md

# CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01)

**Status: Phase A COMPLETE and gated (2026-07-31). Phase B RUN 2026-08-01 — 27 cycles, 586 invariant
checks, ZERO violations.** **One new finding: R-156. One structural scope constraint: Tier 3 cannot be
isolated (§3). One suspected defect investigated and DISPROVED (§5.2).**

Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step),
`isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`.

---

## 1. Venue and baselines

| | |
|---|---|
| Host | `demo-hp` (HP t740), **Tier 0**, the designated drill/build host |
| VM | **311 `c10-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` |
| Disks | `scsi0` 200 G system · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c10-scratch` |
| Storage | **`c10-scratch`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images` |
| Box | `c10.felhom.eu` @ `192.168.0.105` (demo-hp's vmbr0, `192.168.0.87/24`) |
| Console | no browser on DooPlex → `qm monitor screendump` → PNG, read visually; `sendkey` for input |
| Untouched | `drill-r50` (VM 300), guest 9201 on both demo boxes, `local-lvm`, both backup targets, ep0, the Storage Box |

**The `exactMount` choice, made deliberately (the brief asks which).** `c10-scratch` sits at the mount
**root**, not a subdirectory. A dir storage at a subdirectory sets `exactMount=false` →
`reachable=false` → `StorageStateDisconnected` (`felhom-agent/internal/storage/observe.go:336`), which
would have emitted `storage_disconnected` for demo-hp for the entire run — **the exact signal I1/I2
exist to discriminate**. It coexists with `felhom-backup` on the same path, which was not modified.

### Baselines — every value read fresh

| What | Value | How |
|---|---|---|
| controller `main` | **v0.188.0** | CHANGELOG top; `HEAD`=`origin/main`=`4115e88`, clean |
| **golden's baked controller** | **0.188.0** (`7353d8be…`) | hub `/configuration`, selected option |
| agent published + vouched | **0.119.0** (`e37aca82…`) | hub `/configuration`, selected option |
| hub | **0.86.0** | live deploy image tag == CHANGELOG top |
| ISO | **1.26.1**, `f3cc86d5…`, 1 705 322 496 B | live round trip from `iso.felhom.eu` |
| `felhom.eu` HEAD | `e9a74a0`, clean | `git rev-parse` |
| managed floor | v0.156.0 (DB override) | hub `/configuration` |

**Golden vs `main`: not behind — both 0.188.0.** The deliberate decision the brief asks for is
therefore vacuous this run; the campaign tests what is shipping *and* what is current.

**Two of the brief's assumptions were stale; reading fresh caught both.**

1. The brief was written against ISO **v1.25.0**. `iso.felhom.eu` began serving **1.26.1** at 18:37 Z
   the same day, ~90 minutes before this run. Operator chose 1.26.1 — the current published artifact.
2. The brief and three docs say demo-hp has **no baked SSH key**; `ssh -o BatchMode=yes demo-hp`
   authenticated **by key, first try**. Already **R-129**; **not re-filed**.

---

## 2. The isolation gate — §2

| Target | Isolation | Proof |
|---|---|---|
| **PBS** | dedicated datastore `felhom-c10` on DooPlex + auth-id `c10@pbs!box` scoped to it alone | **CAPTURED** — `isolation/pbs-denial.txt` |
| **restic** | scratch subaccount `u629488-sub4`, own chroot home `felhom-campaign10` | **CAPTURED** — `isolation/restic-denial.txt` |
| **hub** | the campaign's own record `c10-soak`; no other customer touched | **CAPTURED** — §2.3 |

**Both denials carry a POSITIVE CONTROL, and the first attempt failed it.** The initial PBS capture
returned four tidy 403s — including on the campaign's *own* datastore. All four were worthless: PBS
API tokens carry privilege separation, so a token's rights are the intersection of the user's ACL and
the token's, and user `c10@pbs` had none. Granting the scoped role to the user as well turned the
control green and left the denials standing. Without the control this would have shipped as a clean
isolation proof that proved only that the credential was broken.

    ## POSITIVE CONTROL — same token, its OWN datastore
    GET /api2/json/admin/datastore/felhom-c10/snapshots     {"data":[]}   HTTP 200

    ## DENIAL 1 — production datastore felhom-spike
    permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-spike   403
    ## DENIAL 3 — DELETE namespace demo-felhom (destructive)
    permission check failed - missing Datastore.Modify on /datastore/felhom-spike                   403

restic, same shape — the credential writes and reads back in its own home, and every production path
is outside the chroot:

    POSITIVE CONTROL  put/get/rm c10-canary.txt in own home        → succeeds, content verified
    DENIAL  cd /home/felhom-peti-felhom      → stat remote: No such file or directory
    DENIAL  ls ..                            → remote readdir("/home/../"): Permission denied

**2.3 — hub.** The run created exactly one record, `c10-soak`, and bound exactly one appliance
(uuid `55ab3776-…`) to it. Existing customers (`demo-felhom`, `demo-hp`, `peti-felhom`, `drill-r50`,
`sess-f`) were read but never written. Verified from the hub's own log: every `c10-soak` line is the
campaign's, and no line names another customer.

---

## 3. Scope constraint — Tier 3 cannot be isolated, so it was not run

The brief requires the campaign's PBS to be *a dedicated datastore and namespace on DooPlex*. That is
unreachable, for two reasons that are each an **already-recorded deliberate position**:

1. **Offsite hard-requires the DR tier** — `hub/internal/web/configs.go:1300` refuses outright:
   *"Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS
   key)"*. Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4).
2. **The DR tier only ever provisions on ep0** — peer allocation and endpoint sync use the lowest
   `endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).

So the only two configurations are *DR tier on* → campaign PBS traffic lands on **ep0** (Tier 2, the
offsite of last resort, RAM-constrained per R-90, fenced by §3), or *DR tier off* → **no Tier 3 at
all**. Chosen: **DR tier OFF, offsite OFF** — the only option §3 permits.

Consequence, stated up front rather than discovered in the results: the campaign touches **neither ep0
nor the Storage Box at any point** — stronger isolation than asked for, obtained by not running the
tier. **Lost: every Tier-3 atom, I8 entirely, and the Tier-3 RTO/RPO rows.** Unaffected: the
two-drive core (R-112/113/114/116/117), D5, and I1–I7, I9–I11.

**No `R-n` minted** — the register was grepped first and both halves are already recorded. This is a
campaign constraint, not a defect.

---

## 4. Phase A gates

| Gate | Result | Positive observable |
|---|---|---|
| **A1** install from the published ISO | **PASS** | `ii felhom-bootstrap 1.26.1`; unit `enabled`, fired at first boot; registered unclaimed with pairing code `FD6-A6S`, MACs, hardware and 3 SSH host keys; **4.2 GB actually written**, system booted on its own |
| **A2** claim | **PASS** | discriminator flipped `dashboard not yet claimed` → `authentication required`; hub: `[claim] customer c10-soak CLAIMED its dashboard` |
| **A3** enrol both drives, apps across them | **PASS** | both registered via the real endpoint; backup target healthy; 4 apps healthy |
| **A4** discriminators | **PASS** | all four apps seed and read back the same generation |
| **A5** budget + watchdog | **PASS** (built for Phase B) | disk floor 120 G on `/mnt/nvme-1tb` + VM-liveness watchdog; heartbeat `status.txt` per phase; `journal.tsv` per check. Neither fired: the run ended on its deadline with 871 G free |

**A1 — 1.26.1 cannot auto-install, verified against the bytes** (not inferred from its manifest):
`auto-installer-mode.toml` 0 hits, `answer.toml` 0, `proxmox-auto-install` 0, exactly 2 interactive
menuentries. Driven blind through the **Terminal UI** (the graphical entry is undrivable — monitor
mouse events do not move the guest cursor). Three traps hit and handled:

- GRUB's default is the graphical entry; the first `down` landed after the 15 s countdown. Reset and
  sent it inside the window in one round trip.
- **The guest keymap was Hungarian while `sendkey` emits US scancodes.** Every symbol afterwards —
  notably `@`, which is AltGr+V on a Hungarian layout — would have been silently mangled, *including
  the 24-char root password*, locking the box out with no way back in. Switched to U.S. English
  before typing anything; confirmed positively when `c10@felhom.eu` rendered a correct `@` and `.`.
- The recorded boot-order trap: `--boot` set in its **own** `qm set` after the disks existed and
  verified from `qm config`; flipped to `order=scsi0` post-install and verified again.

Only `scsi0` was attached during the install, so the installer offered exactly one target and could
not reach a data drive. The 50 G drives were attached afterwards.

**A2** — the installer's root password stops working at day-0; it is replaced by a hub-vaulted
managed credential (`vaulted break-glass recovery credential … secret 32 chars`), retrieved via
`POST /hosts/c10-soak-8a405e/reveal-recovery-credential`. R-119 confirmed live again: the claim code
is emailed-only and the operator relayed it — the single human step.

**A3** — both drives enrolled through **the real endpoint** (`POST /api/storage/attach`), which does
format → assign → guest-attach → register-stable-path in order:

| Drive | Device | Stable path | Role |
|---|---|---|---|
| `adatok` | `/dev/sdb` | `/mnt/felhom-drives/adatok` | app data, `set_default` |
| `mentes` | `/dev/sdc` | `/mnt/felhom-drives/mentes` | **backup target** (PVE storage `felhom-backup` at `/mnt/mentes`) |

A false start worth recording: hand-driving `format` + `assign` first left the drives **mounted but
unregistered** — invisible to both the raw-candidate scan and `/disks`, so the real enroll endpoint
could not resolve a UUID for them (`resolveEnrollUUID` sees neither list). Backed out (eject leaves
the raw `/mnt/<name>` mount by design, `disks.go:606`, so the mounts also had to be undone) and re-ran
the real flow clean. **The lesson is the brief's own:** use the real endpoint, or manufacture a state
the product never produces.

Backup-target offer flow (R-112/R-114/E-2) exercised as the customer would: assign returned
`restart_required: true`, and the state only flipped after the agent restart it asked for —

    before: {"degraded":true, "target":"local", "offer_path":"/mnt/felhom-drives/adatok", "message":"A rendszermentés jelenleg ugyanazon a lemezen van…"}
    after:  {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}

That healthy read is the **I5/I6 baseline**: everything healthy ⇒ no degraded banner, target reads
healthy, zero absent events.

**Apps — four, healthy, covering both sides of D5's split:**

| App | Data | Secret classes | Why chosen |
|---|---|---|---|
| **rallly** | postgres (`rallly-postgres`) | 2 × `type: secret` (`SECRET_PASSWORD` hex:32 labelled *Titkosítási kulcs*, `DB_PASSWORD`) | the DB app; travelling class |
| **homebox** | sqlite volume | 1 × `secret`, **`data_key: true`** | data-encrypting key |
| **papra** | see **R-156** | 1 × `secret`, **`data_key: true`** | data-encrypting key |
| **grafana** | volume | 1 × **`type: password`** | **the withheld class — I10's negative half** |

4 travelling `secret` fields and 1 `password` field that must never travel, so I10 can assert both
directions rather than only the positive one.

**A4 — discriminators work.** Every app carries a generation marker; a full seed/read cycle returns
the seeded generation from all four. rallly's is a row in a `cc_proof` table read over **the path
`DATABASE_URL` actually names** (`docker run --network container:rallly … psql -h rallly-postgres`),
never `127.0.0.1` inside the postgres container — the trusted-socket path that produced D5's §1.2
false pass.

---

## 4b. Phase B — the soak (2026-08-01)

**27 cycles, 09:33–15:40 CEST, ended on its own deadline. No watchdog halt, no atom exception, no
I11 breach. 586 invariant rows, 0 violations.**

**Every atom and every invariant was proven BY HAND before it was automated** (§4c). That ordering is
the point: a runner written first and trusted after is exactly the rig the brief warns about.

**A gap in Phase A found and closed first.** None of the four apps had `HDD_PATH` — all their data sat
in Docker volumes on the **system** disk, so detaching a data drive was a no-op for apps and **I3 could
never have fired**. A soak run in that state would have produced 27 green cycles that tested nothing
about cross-drive behaviour. **calibre-web** was deployed onto `adatok`
(`/mnt/felhom-drives/adatok/userdata/media/books → /calibre-library`, confirmed from
`docker inspect`) before the run started.

### Atoms, permuted per cycle (seeded RNG, so the order is reproducible)

`backup` · `restore_verify` · `detach_target` · `detach_nontarget` · `kill_controller` ·
`redeploy_app`, plus `reboot_guest` on every 7th cycle.

### Results

| Invariant | Checks | Violations | What was actually asserted |
|---|---|---|---|
| **I1** + pair | 28 + 28 | **0** | target absent ⇒ `backup_target_absent`/**error**; return ⇒ `backup_target_restored`/**info** |
| **I2** + pair | 28 + 28 | **0** | non-target ⇒ **generic** `storage_disconnected`/error → `storage_reconnected`/info |
| **I3** | 56 | **0** | `bound_under_parent=false` while the namespace is unusable, both drives |
| **I4** | 56 | **0** | guest init boot-time identical across every bind repair |
| **I5** | 28 | **0** | healthy ⇒ zero absent events, target reads healthy |
| **I6** | 28 | **0** | healthy carries no degraded message |
| **I7** | 28 | **0** | **every restore returned the discriminator asked for — 28/28** |
| **I10** | 135 | **0** | 4 × `type: secret` present in the unit; 1 × `type: password` **absent**, asserted as absence |
| **I11** | 28 | **0** | no secret value in controller or hub logs |
| BACKUP / REDEPLOY / KILL-CTRL / REBOOT | 56 / 28 / 28 / 3 | **0** | atom outcomes |

The row counts are themselves a check that nothing was silently skipped: I3 and I4 at **56** (twice per
cycle — both drives), I10 at **135** (5 secret-class fields × 27), REBOOT at **3** (cycles 7/14/21).
Counts of 28 rather than 27 include the one-cycle smoke test, which wrote to the same journal and
restarts its numbering at 1 — the totals are right, the cycle *labels* collide.

**A clean invariant is a result**: I1's discrimination, I2's genericity, the pairing of both, I4's
no-restart repair and I7's discriminator held over **27 consecutive cycles** of detach, reattach,
backup, restore, controller kill, app redeploy and three guest reboots. Nothing drifted at iteration 38
because the run stopped at 27 — the tail beyond that is untested, not proven absent.

### Two behaviours worth recording, both observed repeatedly

- **The drive comes back on a different device node and the bind still heals.** `adatok` left as
  `/dev/sdb` and returned as `/dev/sdd`; `bound_under_parent` went true again and the contents were
  readable, with the guest's init boot-time unchanged. That is R-117's stale-device case, hit on every
  detach cycle rather than once.
- **`health_degraded (warning)` accompanies the target-absent event** and clears on return, alongside
  the specific `backup_target_*` pair.

## 4c. Primitives proven by hand before automation

| Step | Observed |
|---|---|
| backup | `POST /api/backup/run` → `Mentés elindítva`, settles in ~23–34 s |
| restore | `POST /backup/restore` (form) is **async**; polled on `/api/backup/restore-status`, never `IsRunning()` |
| I7, first proof | seeded A → backup → seeded B → restored A; rallly returned **A** while the other three still read **B** |
| I1 | `backup_target_absent (error) — A rendszermentés meghajtója nem érhető el: Mentes` |
| I2 | `storage_disconnected (error) — Meghajtó váratlanul leválasztva: Adatok` |
| I6 degraded copy | *"A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod, a teljes rendszermentés nem készül el."* |
| I10 | rallly `DB_PASSWORD`+`SECRET_PASSWORD`, homebox `HBOX_AUTH_API_KEY_PEPPER`, papra `AUTH_SECRET` present; grafana `GF_SECURITY_ADMIN_PASSWORD` **absent**; unit mode `0600` |

## 5. Findings

### R-156 — papra's data is neither persisted nor backed up, and it reports healthy

**New. Register grepped first:** papra appears in R-41 (never-deployable / healthcheck) and R-127
(`data_key` misclassification); neither covers this. *(Also corrected en route: a first grep suggested
R-164 was taken — it is a mis-citation of controller **v0.164.0**, already flagged in
`REPORT-record-correction-2026-07-29.md:141`. Highest real number is R-155, so this is R-156.)*

The template mounts `papra_data:/app/data`, but the app persists to **`/app/app-data/db/db.sqlite`**.
The mounted volume is therefore empty, and the real database lives in the container's writable layer.

Evidence (`r156-papra-volume.txt`), all four legs measured:

    1. volume papra_papra_data -> /app/data
    2. uid=999(nonroot);  drwxr-xr-x 2 root root /app/data
       touch: cannot touch '/app/data/.w': Permission denied
    3. /app/data contains only the canary this campaign wrote AS ROOT
    4. /app/app-data/db/db.sqlite  475136 bytes  owned nonroot   <-- the real DB, in no volume
    5. health status: "healthy"

**Consequences.** (a) papra's documents do not survive `docker compose down`/redeploy/host restore —
the writable layer is not persisted. (b) `DumpAppVolumes` dumps the volume, so papra is **backed up
as an empty directory**: its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. (c) The
healthcheck only probes the HTTP port, so none of this surfaces — the app is green throughout.

This is the *presence is not success* family from `CLAUDE.md`: a backup exists, a healthcheck passes,
and neither means the data is there. It is also a second instance of the pattern R-41 names — an app
that deploys and looks fine while being fundamentally broken.

**Two-repo shape**, if fixed: the volume target belongs in `app-catalog-felhom.eu`, and the general
defence is a catalog gate asserting each template's mounted volume is a path the app actually writes.
**Not fixed here** — the fences forbid it, and a fix mid-run would prove a version that did not exist
when the run started.

### 5.2 — Investigated and DISPROVED: `/api/disks` reports `state=attached` for an absent drive

Recorded because the *chase* is the useful artefact, not because anything is broken.

With `adatok` physically removed, `/api/disks` kept reporting `state: "attached"` for it across three
polls (~90 s) — while `/dev/sdb` was gone, `/mnt/adatok` was absent from `/proc/mounts`, and the guest
bind returned `Input/output error`. The other fields were right (`backing_device: ""`,
`guest_attached: false`, `bound_under_parent: false`).

It looked like the R-116/R-117 family, and there is a real consumer:
`internal/web/intermediary.go:230` — `present[d.MountPath] |= d.State == "attached"`. A presence gate
reading an absent drive as present is precisely I3's failure mode.

**It is inert, and the gate is correct.** `planDriveGates` only gates paths under
`StableParentDir+"/"` (`/mnt/felhom-drives/…`). The registered path is the **stable** one, whose
presence is computed from `d.BoundUnderParent` — observed **false** — not from `State`. The
`State`-derived entry is keyed on the **raw** `/mnt/adatok`, which is not a registered stable path, so
the loop skips it. The observable end-to-end behaviour agrees: the storage page showed
*"Meghajtó leválasztva: Adatok"* with a disconnect timestamp and a reconnect action, i.e. the gate fired.

The stale `state` field comes from the **registry** row surviving in the union after the
storage-observation row drops out (a raw enrolled drive is not a PVE storage), carrying its configured
`MountPath`. **No `R-n` minted** — no consumer is misled, and the R-116 comment's claim that this
"cannot make the gate read an absent drive as PRESENT" held under test.

### Not filed

- **demo-hp SSH key** — already R-129.
- **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect.
- **`/api/disks` `state`** — §5.2, disproved.

---

## 6. RTO/RPO — measured

**28 restores, every one returning the correct discriminator.** Wall-clock is measured from the restore
request to **the app serving the correct data** (the canary read back over rallly's real network path),
not to "restore returned".

| Tier | App | Data volume | n | min | median | p90 | max | mean |
|---|---|---|---|---|---|---|---|---|
| **Tier 1 (local)** | rallly (postgres) | **66 MB** volume / 67 MB recovery unit | 28 | 38.8 s | **42.0 s** | 42.5 s | 44.3 s | 42.0 s |

**Which band this covers: the S band only, and only its lower end.** 66 MB is a small app by any
reading, and the spread is tight — 5.5 s between min and max over 28 runs — which says the cost is
dominated by fixed work (stack stop, volume restore, stack start, health wait) rather than by bytes.
**Nothing here licenses an extrapolation to M or L**: the byte-proportional term is precisely what
66 MB fails to exercise. A second point at a much larger volume is needed before any curve is claimed,
and none was taken.

**RPO is NOT measured.** It is a function of backup *frequency*, and the soak drove backups on demand
rather than on the schedule, so the observed intervals say nothing about the product's RPO.

---

## 7. I1–I11 across cycles

Full table in §4b. **I1, I2 (and both pairs), I3, I4, I5, I6, I7, I10 and I11 held over 27 consecutive
cycles — 586 checks, 0 violations.** No violation was deterministic and none was racy, because there
were none; the honest statement is that the run found no drift **within 27 cycles**, not that none
exists beyond them.

**I8 was not checked** — Tier 3 never ran (§3). **I9 was not automated** — see §8.

---

## 8. What did not run, and why

Phase A overran badly (~5.5 h against the brief's ~1 h) — 1.26.1 is a public release image with no
auto-install path, so day-0 was a blind screendump-and-`sendkey` walk; the Tier-3 question (§3) had to
be settled before a customer could exist at all, since `dr_tier` is fixed at creation; and a false
start on drive enrolment had to be backed out. **Phase B was therefore deliberately not launched at
04:00** — the brief's own fence, *"a test rig producing false negatives is worse than no rig"* — and
was built and run the next morning instead, where its first cycle could be watched.

**Atom coverage: 6 of the brief's ~12 suggested families.** Stated plainly rather than implied:

| Atom (brief B1) | Ran |
|---|---|
| back up Tier 1/2 · restore + verify by discriminator · detach/reattach **target** · detach/reattach **non-target** · delete an app and redeploy · reboot the guest | **YES** |
| back up / restore **Tier 3** | **NO** — §3, structurally impossible to isolate |
| abort a filesystem **in place** without the device disappearing (R-117 Q7) | **NO** |
| kill the **agent** mid-backup · **hard-reset the VM** mid-write · reboot the **VM** | **NO** |
| **concurrent** backup+restore · **concurrent** backup+detach | **NO** |
| fill a drive to near-full and continue | **NO** |

**`kill_controller` is not "mid-backup".** It restarts the controller at a permuted point in the
cycle, so it sometimes lands after a backup and sometimes nowhere near one. It exercises recovery, not
the mid-write race the brief asked for. Recorded because the atom name would otherwise overclaim.

**`reboot_guest` was appended after the shuffle, not permuted into it**, so it always ran last in
cycles 7/14/21 and **never interleaved with a detach**. Reboot-during-detach is untested.

**I8** — not checked; Tier 3 never ran. **I9** (Tier-1/2 restore with the guest's `app.yaml` moved
aside, data read over the app's real network path) — **not automated**. It was proven on this exact
controller 0.188.0 by the tester-gate run hours earlier
(`tester-gate-golden-0.188.0-2026-07-31.md` §5, step 6), and that is cited rather than re-claimed:
this campaign did not execute it.

**The tail is untested, not clean.** 27 cycles is the count; the brief's interest in "the state that
quietly drifts on the thirty-eighth" is not answered by a 27-cycle run.

---

## 9. Teardown — OWED, nothing removed yet

The rig is intact **on purpose**, so Phase B can run without repeating Phase A. Every item below is
outstanding and must be removed when the campaign ends:

| Layer | Item | Command |
|---|---|---|
| VM | `311 c10-appliance` on demo-hp | `qm stop 311 && qm destroy 311 --purge` |
| storage | `c10-scratch` dir storage | `pvesm remove c10-scratch` |
| PBS | datastore `felhom-c10`, user `c10@pbs`, token `!box`, both ACLs | `proxmox-backup-manager datastore remove felhom-c10` · `user delete c10@pbs` · remove `/mnt/5_hdd/backup/campaign10` |
| restic | subaccount **`u629488-sub4`** (`felhom-campaign10`) on box 611714 — **unused, since Tier 3 never ran** | `DELETE /v1/storage_boxes/611714/subaccounts/281530` |
| **hub** | **customer `c10-soak`** — **disposition: DELETE.** Named explicitly per R-131, which is four orphaned scratch customers left by exactly this omission | `POST /configs/c10-soak/delete` with `ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=c10-soak` |
| secrets | `~/.config/campaign10/` on DooPlex (host + dashboard passwords, API key, app secrets, PBS token, restic password); `/root/c10/` and `/root/c10api.sh` in guest 9201 | `shred -u` both |

`sess-f` ("R-120 golden 0.186.0 proof") is still present and is **not** this run's record — R-131
stands, untouched.

---

## 10. Hygiene note

The customer API key for `c10-soak` was printed into this session's transcript while being read from
the hub page. It belongs to a scratch customer scheduled for deletion (§9) and grants nothing once
that record is gone, but the write-out was avoidable and is recorded here rather than left implicit —
the same class as R-132, where a correctly-made request was undone by how it was reported.