Campaign 10: Phase A complete + gated; Phase B not run; R-156 filed
Phase A passed every gate on a fresh box built from the PUBLISHED ISO 1.26.1: install, claim, two drives enrolled through the real endpoints with the backup target healthy, four apps spanning both sides of D5's secret split, and a working discriminator across all four. Isolation gate: both denials captured, each with a positive control. The PBS control FAILED first — four clean-looking 403s were worthless because the token was denied on its own datastore too (PBS token privilege separation). Fixed and re-run; the denials stand. R-156 (new, register grepped): papra's data is neither persisted nor backed up, and it reports healthy. The template mounts papra_data:/app/data; the app writes /app/app-data/db/db.sqlite. Volume empty and root-owned against a -rootless image, real DB in the container writable layer, healthcheck only probes the HTTP port. Its Tier-1/2 backup is real, verifiable and contains nothing. Not fixed. Tier 3 could not be isolated so it was not run: offsite hard-requires the DR tier (configs.go:1300) and the DR tier only provisions on ep0 (per-endpoint allocation deferred, hub/README.md:260). Both are recorded deliberate positions, so no R-n minted. The campaign touched neither ep0 nor the Storage Box. Phase B did not start. Phase A was budgeted at ~1h and took ~5.5h (1.26.1 is a public release image with no auto-install path, so the install was a blind screendump+sendkey walk). That left the runner — which judges eleven invariants and fires destructive atoms unattended — to be written at 04:00 with ~3h of night left. Stopped on the brief's own fence: a rig producing false negatives is worse than no rig. The rig is built and idle; teardown is OWED and itemised, including hub customer c10-soak (disposition: DELETE).
This commit is contained in:
@@ -0,0 +1,302 @@
|
||||
# CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01)
|
||||
|
||||
**Status: Phase A COMPLETE and gated. Phase B NOT STARTED — deliberately, see §8.**
|
||||
**One new finding: R-156. One structural scope constraint: Tier 3 cannot be isolated (§3).**
|
||||
|
||||
Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step),
|
||||
`isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Venue and baselines
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Host | `demo-hp` (HP t740), **Tier 0**, the designated drill/build host |
|
||||
| VM | **311 `c10-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` |
|
||||
| Disks | `scsi0` 200 G system · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c10-scratch` |
|
||||
| Storage | **`c10-scratch`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images` |
|
||||
| Box | `c10.felhom.eu` @ `192.168.0.105` (demo-hp's vmbr0, `192.168.0.87/24`) |
|
||||
| Console | no browser on DooPlex → `qm monitor screendump` → PNG, read visually; `sendkey` for input |
|
||||
| Untouched | `drill-r50` (VM 300), guest 9201 on both demo boxes, `local-lvm`, both backup targets, ep0, the Storage Box |
|
||||
|
||||
**The `exactMount` choice, made deliberately (the brief asks which).** `c10-scratch` sits at the mount
|
||||
**root**, not a subdirectory. A dir storage at a subdirectory sets `exactMount=false` →
|
||||
`reachable=false` → `StorageStateDisconnected` (`felhom-agent/internal/storage/observe.go:336`), which
|
||||
would have emitted `storage_disconnected` for demo-hp for the entire run — **the exact signal I1/I2
|
||||
exist to discriminate**. It coexists with `felhom-backup` on the same path, which was not modified.
|
||||
|
||||
### Baselines — every value read fresh
|
||||
|
||||
| What | Value | How |
|
||||
|---|---|---|
|
||||
| controller `main` | **v0.188.0** | CHANGELOG top; `HEAD`=`origin/main`=`4115e88`, clean |
|
||||
| **golden's baked controller** | **0.188.0** (`7353d8be…`) | hub `/configuration`, selected option |
|
||||
| agent published + vouched | **0.119.0** (`e37aca82…`) | hub `/configuration`, selected option |
|
||||
| hub | **0.86.0** | live deploy image tag == CHANGELOG top |
|
||||
| ISO | **1.26.1**, `f3cc86d5…`, 1 705 322 496 B | live round trip from `iso.felhom.eu` |
|
||||
| `felhom.eu` HEAD | `e9a74a0`, clean | `git rev-parse` |
|
||||
| managed floor | v0.156.0 (DB override) | hub `/configuration` |
|
||||
|
||||
**Golden vs `main`: not behind — both 0.188.0.** The deliberate decision the brief asks for is
|
||||
therefore vacuous this run; the campaign tests what is shipping *and* what is current.
|
||||
|
||||
**Two of the brief's assumptions were stale; reading fresh caught both.**
|
||||
|
||||
1. The brief was written against ISO **v1.25.0**. `iso.felhom.eu` began serving **1.26.1** at 18:37 Z
|
||||
the same day, ~90 minutes before this run. Operator chose 1.26.1 — the current published artifact.
|
||||
2. The brief and three docs say demo-hp has **no baked SSH key**; `ssh -o BatchMode=yes demo-hp`
|
||||
authenticated **by key, first try**. Already **R-129**; **not re-filed**.
|
||||
|
||||
---
|
||||
|
||||
## 2. The isolation gate — §2
|
||||
|
||||
| Target | Isolation | Proof |
|
||||
|---|---|---|
|
||||
| **PBS** | dedicated datastore `felhom-c10` on DooPlex + auth-id `c10@pbs!box` scoped to it alone | **CAPTURED** — `isolation/pbs-denial.txt` |
|
||||
| **restic** | scratch subaccount `u629488-sub4`, own chroot home `felhom-campaign10` | **CAPTURED** — `isolation/restic-denial.txt` |
|
||||
| **hub** | the campaign's own record `c10-soak`; no other customer touched | **CAPTURED** — §2.3 |
|
||||
|
||||
**Both denials carry a POSITIVE CONTROL, and the first attempt failed it.** The initial PBS capture
|
||||
returned four tidy 403s — including on the campaign's *own* datastore. All four were worthless: PBS
|
||||
API tokens carry privilege separation, so a token's rights are the intersection of the user's ACL and
|
||||
the token's, and user `c10@pbs` had none. Granting the scoped role to the user as well turned the
|
||||
control green and left the denials standing. Without the control this would have shipped as a clean
|
||||
isolation proof that proved only that the credential was broken.
|
||||
|
||||
## POSITIVE CONTROL — same token, its OWN datastore
|
||||
GET /api2/json/admin/datastore/felhom-c10/snapshots {"data":[]} HTTP 200
|
||||
|
||||
## DENIAL 1 — production datastore felhom-spike
|
||||
permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-spike 403
|
||||
## DENIAL 3 — DELETE namespace demo-felhom (destructive)
|
||||
permission check failed - missing Datastore.Modify on /datastore/felhom-spike 403
|
||||
|
||||
restic, same shape — the credential writes and reads back in its own home, and every production path
|
||||
is outside the chroot:
|
||||
|
||||
POSITIVE CONTROL put/get/rm c10-canary.txt in own home → succeeds, content verified
|
||||
DENIAL cd /home/felhom-peti-felhom → stat remote: No such file or directory
|
||||
DENIAL ls .. → remote readdir("/home/../"): Permission denied
|
||||
|
||||
**2.3 — hub.** The run created exactly one record, `c10-soak`, and bound exactly one appliance
|
||||
(uuid `55ab3776-…`) to it. Existing customers (`demo-felhom`, `demo-hp`, `peti-felhom`, `drill-r50`,
|
||||
`sess-f`) were read but never written. Verified from the hub's own log: every `c10-soak` line is the
|
||||
campaign's, and no line names another customer.
|
||||
|
||||
---
|
||||
|
||||
## 3. Scope constraint — Tier 3 cannot be isolated, so it was not run
|
||||
|
||||
The brief requires the campaign's PBS to be *a dedicated datastore and namespace on DooPlex*. That is
|
||||
unreachable, for two reasons that are each an **already-recorded deliberate position**:
|
||||
|
||||
1. **Offsite hard-requires the DR tier** — `hub/internal/web/configs.go:1300` refuses outright:
|
||||
*"Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS
|
||||
key)"*. Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4).
|
||||
2. **The DR tier only ever provisions on ep0** — peer allocation and endpoint sync use the lowest
|
||||
`endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).
|
||||
|
||||
So the only two configurations are *DR tier on* → campaign PBS traffic lands on **ep0** (Tier 2, the
|
||||
offsite of last resort, RAM-constrained per R-90, fenced by §3), or *DR tier off* → **no Tier 3 at
|
||||
all**. Chosen: **DR tier OFF, offsite OFF** — the only option §3 permits.
|
||||
|
||||
Consequence, stated up front rather than discovered in the results: the campaign touches **neither ep0
|
||||
nor the Storage Box at any point** — stronger isolation than asked for, obtained by not running the
|
||||
tier. **Lost: every Tier-3 atom, I8 entirely, and the Tier-3 RTO/RPO rows.** Unaffected: the
|
||||
two-drive core (R-112/113/114/116/117), D5, and I1–I7, I9–I11.
|
||||
|
||||
**No `R-n` minted** — the register was grepped first and both halves are already recorded. This is a
|
||||
campaign constraint, not a defect.
|
||||
|
||||
---
|
||||
|
||||
## 4. Phase A gates
|
||||
|
||||
| Gate | Result | Positive observable |
|
||||
|---|---|---|
|
||||
| **A1** install from the published ISO | **PASS** | `ii felhom-bootstrap 1.26.1`; unit `enabled`, fired at first boot; registered unclaimed with pairing code `FD6-A6S`, MACs, hardware and 3 SSH host keys; **4.2 GB actually written**, system booted on its own |
|
||||
| **A2** claim | **PASS** | discriminator flipped `dashboard not yet claimed` → `authentication required`; hub: `[claim] customer c10-soak CLAIMED its dashboard` |
|
||||
| **A3** enrol both drives, apps across them | **PASS** | both registered via the real endpoint; backup target healthy; 4 apps healthy |
|
||||
| **A4** discriminators | **PASS** | all four apps seed and read back the same generation |
|
||||
| **A5** budget + watchdog | **NOT DONE** | §8 |
|
||||
|
||||
**A1 — 1.26.1 cannot auto-install, verified against the bytes** (not inferred from its manifest):
|
||||
`auto-installer-mode.toml` 0 hits, `answer.toml` 0, `proxmox-auto-install` 0, exactly 2 interactive
|
||||
menuentries. Driven blind through the **Terminal UI** (the graphical entry is undrivable — monitor
|
||||
mouse events do not move the guest cursor). Three traps hit and handled:
|
||||
|
||||
- GRUB's default is the graphical entry; the first `down` landed after the 15 s countdown. Reset and
|
||||
sent it inside the window in one round trip.
|
||||
- **The guest keymap was Hungarian while `sendkey` emits US scancodes.** Every symbol afterwards —
|
||||
notably `@`, which is AltGr+V on a Hungarian layout — would have been silently mangled, *including
|
||||
the 24-char root password*, locking the box out with no way back in. Switched to U.S. English
|
||||
before typing anything; confirmed positively when `c10@felhom.eu` rendered a correct `@` and `.`.
|
||||
- The recorded boot-order trap: `--boot` set in its **own** `qm set` after the disks existed and
|
||||
verified from `qm config`; flipped to `order=scsi0` post-install and verified again.
|
||||
|
||||
Only `scsi0` was attached during the install, so the installer offered exactly one target and could
|
||||
not reach a data drive. The 50 G drives were attached afterwards.
|
||||
|
||||
**A2** — the installer's root password stops working at day-0; it is replaced by a hub-vaulted
|
||||
managed credential (`vaulted break-glass recovery credential … secret 32 chars`), retrieved via
|
||||
`POST /hosts/c10-soak-8a405e/reveal-recovery-credential`. R-119 confirmed live again: the claim code
|
||||
is emailed-only and the operator relayed it — the single human step.
|
||||
|
||||
**A3** — both drives enrolled through **the real endpoint** (`POST /api/storage/attach`), which does
|
||||
format → assign → guest-attach → register-stable-path in order:
|
||||
|
||||
| Drive | Device | Stable path | Role |
|
||||
|---|---|---|---|
|
||||
| `adatok` | `/dev/sdb` | `/mnt/felhom-drives/adatok` | app data, `set_default` |
|
||||
| `mentes` | `/dev/sdc` | `/mnt/felhom-drives/mentes` | **backup target** (PVE storage `felhom-backup` at `/mnt/mentes`) |
|
||||
|
||||
A false start worth recording: hand-driving `format` + `assign` first left the drives **mounted but
|
||||
unregistered** — invisible to both the raw-candidate scan and `/disks`, so the real enroll endpoint
|
||||
could not resolve a UUID for them (`resolveEnrollUUID` sees neither list). Backed out (eject leaves
|
||||
the raw `/mnt/<name>` mount by design, `disks.go:606`, so the mounts also had to be undone) and re-ran
|
||||
the real flow clean. **The lesson is the brief's own:** use the real endpoint, or manufacture a state
|
||||
the product never produces.
|
||||
|
||||
Backup-target offer flow (R-112/R-114/E-2) exercised as the customer would: assign returned
|
||||
`restart_required: true`, and the state only flipped after the agent restart it asked for —
|
||||
|
||||
before: {"degraded":true, "target":"local", "offer_path":"/mnt/felhom-drives/adatok", "message":"A rendszermentés jelenleg ugyanazon a lemezen van…"}
|
||||
after: {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}
|
||||
|
||||
That healthy read is the **I5/I6 baseline**: everything healthy ⇒ no degraded banner, target reads
|
||||
healthy, zero absent events.
|
||||
|
||||
**Apps — four, healthy, covering both sides of D5's split:**
|
||||
|
||||
| App | Data | Secret classes | Why chosen |
|
||||
|---|---|---|---|
|
||||
| **rallly** | postgres (`rallly-postgres`) | 2 × `type: secret` (`SECRET_PASSWORD` hex:32 labelled *Titkosítási kulcs*, `DB_PASSWORD`) | the DB app; travelling class |
|
||||
| **homebox** | sqlite volume | 1 × `secret`, **`data_key: true`** | data-encrypting key |
|
||||
| **papra** | see **R-156** | 1 × `secret`, **`data_key: true`** | data-encrypting key |
|
||||
| **grafana** | volume | 1 × **`type: password`** | **the withheld class — I10's negative half** |
|
||||
|
||||
4 travelling `secret` fields and 1 `password` field that must never travel, so I10 can assert both
|
||||
directions rather than only the positive one.
|
||||
|
||||
**A4 — discriminators work.** Every app carries a generation marker; a full seed/read cycle returns
|
||||
the seeded generation from all four. rallly's is a row in a `cc_proof` table read over **the path
|
||||
`DATABASE_URL` actually names** (`docker run --network container:rallly … psql -h rallly-postgres`),
|
||||
never `127.0.0.1` inside the postgres container — the trusted-socket path that produced D5's §1.2
|
||||
false pass.
|
||||
|
||||
---
|
||||
|
||||
## 5. Findings
|
||||
|
||||
### R-156 — papra's data is neither persisted nor backed up, and it reports healthy
|
||||
|
||||
**New. Register grepped first:** papra appears in R-41 (never-deployable / healthcheck) and R-127
|
||||
(`data_key` misclassification); neither covers this. *(Also corrected en route: a first grep suggested
|
||||
R-164 was taken — it is a mis-citation of controller **v0.164.0**, already flagged in
|
||||
`REPORT-record-correction-2026-07-29.md:141`. Highest real number is R-155, so this is R-156.)*
|
||||
|
||||
The template mounts `papra_data:/app/data`, but the app persists to **`/app/app-data/db/db.sqlite`**.
|
||||
The mounted volume is therefore empty, and the real database lives in the container's writable layer.
|
||||
|
||||
Evidence (`r156-papra-volume.txt`), all four legs measured:
|
||||
|
||||
1. volume papra_papra_data -> /app/data
|
||||
2. uid=999(nonroot); drwxr-xr-x 2 root root /app/data
|
||||
touch: cannot touch '/app/data/.w': Permission denied
|
||||
3. /app/data contains only the canary this campaign wrote AS ROOT
|
||||
4. /app/app-data/db/db.sqlite 475136 bytes owned nonroot <-- the real DB, in no volume
|
||||
5. health status: "healthy"
|
||||
|
||||
**Consequences.** (a) papra's documents do not survive `docker compose down`/redeploy/host restore —
|
||||
the writable layer is not persisted. (b) `DumpAppVolumes` dumps the volume, so papra is **backed up
|
||||
as an empty directory**: its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. (c) The
|
||||
healthcheck only probes the HTTP port, so none of this surfaces — the app is green throughout.
|
||||
|
||||
This is the *presence is not success* family from `CLAUDE.md`: a backup exists, a healthcheck passes,
|
||||
and neither means the data is there. It is also a second instance of the pattern R-41 names — an app
|
||||
that deploys and looks fine while being fundamentally broken.
|
||||
|
||||
**Two-repo shape**, if fixed: the volume target belongs in `app-catalog-felhom.eu`, and the general
|
||||
defence is a catalog gate asserting each template's mounted volume is a path the app actually writes.
|
||||
**Not fixed here** — the fences forbid it, and a fix mid-run would prove a version that did not exist
|
||||
when the run started.
|
||||
|
||||
### Not filed
|
||||
|
||||
- **demo-hp SSH key** — already R-129.
|
||||
- **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect.
|
||||
|
||||
---
|
||||
|
||||
## 6. RTO/RPO
|
||||
|
||||
**No measurements.** They are a byproduct of Phase B restores, which did not run. Nothing is
|
||||
extrapolated; no band is claimed.
|
||||
|
||||
---
|
||||
|
||||
## 7. I1–I11
|
||||
|
||||
**Not measured across cycles.** Only the healthy baseline (I5/I6) and the discriminator mechanism (I7)
|
||||
were established, as Phase A gates. A clean invariant is only a result with a cycle count behind it,
|
||||
and there is no count to report.
|
||||
|
||||
---
|
||||
|
||||
## 8. What did not run, and why
|
||||
|
||||
**Phase B did not start.** Phase A was budgeted at ~1 h by the brief and took ~5.5 h, almost entirely
|
||||
in places the brief could not have predicted:
|
||||
|
||||
- ISO **1.26.1** turned out to be a public release image with no auto-install path, so the install was
|
||||
a blind screendump-and-`sendkey` walk (~1.5 h) rather than an unattended boot.
|
||||
- The Tier-3 investigation (§3) had to be settled before a customer could be created at all, because
|
||||
`dr_tier` is decided at customer-creation time.
|
||||
- The enrol false start, and the CSRF/session/Bearer probing needed to drive the real endpoints.
|
||||
|
||||
That put the runner — the most correctness-sensitive component, the thing that decides destructive
|
||||
atoms unattended and judges eleven invariants — at ~04:00 with ~3 h of usable night. **I stopped
|
||||
rather than write it there.** The brief's own fence is the reason: *"A test rig producing false
|
||||
negatives is worse than no rig."* A soak whose judgements cannot be trusted would not just waste the
|
||||
night, it would contaminate the register with findings nobody could rely on.
|
||||
|
||||
**Not run:** every B1 atom; I1–I11 across cycles; §B3 RTO/RPO; A5's disk budget, watchdog and
|
||||
heartbeat (A5 exists to bound Phase B and was not written in isolation from it).
|
||||
|
||||
**What is ready.** The rig is built, gated and idle: VM 311 installed and claimed, two drives enrolled
|
||||
with the backup target healthy, four apps healthy spanning both D5 secret classes, a working
|
||||
discriminator across all four, an authenticated API path onto the controller, and both isolation
|
||||
denials captured. The driver library (`c10lib.py`) already implements the parts that took the
|
||||
longest — the jump-host command path, the authenticated API helper, canary seed/read, and the
|
||||
red-proofed psql read path. **Phase B needs the atom set, the invariant checks and the watchdog on
|
||||
top of that, then it can run in daylight where its first cycles can be watched.**
|
||||
|
||||
---
|
||||
|
||||
## 9. Teardown — OWED, nothing removed yet
|
||||
|
||||
The rig is intact **on purpose**, so Phase B can run without repeating Phase A. Every item below is
|
||||
outstanding and must be removed when the campaign ends:
|
||||
|
||||
| Layer | Item | Command |
|
||||
|---|---|---|
|
||||
| VM | `311 c10-appliance` on demo-hp | `qm stop 311 && qm destroy 311 --purge` |
|
||||
| storage | `c10-scratch` dir storage | `pvesm remove c10-scratch` |
|
||||
| PBS | datastore `felhom-c10`, user `c10@pbs`, token `!box`, both ACLs | `proxmox-backup-manager datastore remove felhom-c10` · `user delete c10@pbs` · remove `/mnt/5_hdd/backup/campaign10` |
|
||||
| restic | subaccount **`u629488-sub4`** (`felhom-campaign10`) on box 611714 — **unused, since Tier 3 never ran** | `DELETE /v1/storage_boxes/611714/subaccounts/281530` |
|
||||
| **hub** | **customer `c10-soak`** — **disposition: DELETE.** Named explicitly per R-131, which is four orphaned scratch customers left by exactly this omission | `POST /configs/c10-soak/delete` with `ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=c10-soak` |
|
||||
| secrets | `~/.config/campaign10/` on DooPlex (host + dashboard passwords, API key, app secrets, PBS token, restic password); `/root/c10/` and `/root/c10api.sh` in guest 9201 | `shred -u` both |
|
||||
|
||||
`sess-f` ("R-120 golden 0.186.0 proof") is still present and is **not** this run's record — R-131
|
||||
stands, untouched.
|
||||
|
||||
---
|
||||
|
||||
## 10. Hygiene note
|
||||
|
||||
The customer API key for `c10-soak` was printed into this session's transcript while being read from
|
||||
the hub page. It belongs to a scratch customer scheduled for deletion (§9) and grants nothing once
|
||||
that record is gone, but the write-out was avoidable and is recorded here rather than left implicit —
|
||||
the same class as R-132, where a correctly-made request was undone by how it was reported.
|
||||
Reference in New Issue
Block a user