diff --git a/REPORT-campaign10.md b/REPORT-campaign10.md new file mode 100644 index 0000000..c977cb6 --- /dev/null +++ b/REPORT-campaign10.md @@ -0,0 +1,62 @@ +# REPORT — Campaign 10, two-storage adversarial soak (2026-07-31) + +Follows `REPORT-campaign7/8/9.md`. Root `REPORT.md` is another session's (hub v0.85.0) and was not +clobbered — same shared-clone reasoning as `REPORT-iso-release.md`. + +**Full audit + evidence:** `documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`, +`documentation/tests/campaign10-evidence-2026-07-31/`. + +## The sentence that matters + +**Phase A is complete and every gate passed. Phase B did not run.** The rig is built, fenced and +idle, waiting for a runner that should not be written at 04:00. + +## What was established + +- **Venue** — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the + `/mnt/nvme-1tb` **mount root** (a subdirectory would have emitted `storage_disconnected` for demo-hp + all night — the exact signal I1/I2 discriminate). +- **Baselines, all read fresh.** controller `main` 0.188.0, **golden 0.188.0 (not behind)**, agent + 0.119.0 published+vouched, hub 0.86.0, ISO **1.26.1** (`f3cc86d5…`, round-trip verified live). + The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129. +- **Isolation gate — both denials captured, each with a positive control.** The PBS control **failed + first**: four clean-looking 403s were worthless because the token was denied on its own datastore + too (PBS token privilege separation). Fixed, re-run, denials stand. +- **A1** fresh install from the **published** ISO. 1.26.1 is a public release image — verified against + its bytes that it has no auto-install path — so it was driven blind via screendump + `sendkey` + through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which + would otherwise have been mangled and locked the box out. +- **A2** claimed for real; discriminator flipped `dashboard not yet claimed` → `authentication required`. +- **A3** both drives enrolled through the **real** endpoint; `mentes` accepted as backup target via + the offer flow, ending `degraded:false / target:felhom-backup` — the I5/I6 healthy baseline. + Four apps healthy spanning both sides of D5's split (4 × `type: secret`, 1 × `type: password`). +- **A4** discriminators seed and read back across all four apps; rallly's over the path + `DATABASE_URL` actually names, not the trusted socket that produced D5's false pass. + +## Findings + +- **R-156 (new)** — **papra's data is neither persisted nor backed up, and it reports healthy.** The + template mounts `papra_data:/app/data`; the app writes `/app/app-data/db/db.sqlite`. The volume is + empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB + sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its + Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed. +- **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier + (`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred, + `hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign + therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by + not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows. + +## What did not run + +Every B1 atom, I1–I11 across cycles, RTO/RPO, and A5's budget/watchdog. Phase A was budgeted at ~1 h +and took ~5.5 h — the blind interactive install alone was ~1.5 h. That left the runner, the component +that judges eleven invariants and fires destructive atoms unattended, to be written at 04:00 with ~3 h +of night left. Stopped instead, on the brief's own fence: *a test rig producing false negatives is +worse than no rig.* + +## Teardown — OWED, nothing removed + +Deliberately intact so Phase B need not repeat Phase A. VM 311, `c10-scratch`, PBS datastore +`felhom-c10` + user/token, restic subaccount `u629488-sub4`, and **hub customer `c10-soak` (disposition: +DELETE)** are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four +orphaned scratch customers left by exactly this omission. diff --git a/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md b/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md new file mode 100644 index 0000000..43a5324 --- /dev/null +++ b/documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md @@ -0,0 +1,302 @@ +# CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01) + +**Status: Phase A COMPLETE and gated. Phase B NOT STARTED — deliberately, see §8.** +**One new finding: R-156. One structural scope constraint: Tier 3 cannot be isolated (§3).** + +Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step), +`isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`. + +--- + +## 1. Venue and baselines + +| | | +|---|---| +| Host | `demo-hp` (HP t740), **Tier 0**, the designated drill/build host | +| VM | **311 `c10-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` | +| Disks | `scsi0` 200 G system · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c10-scratch` | +| Storage | **`c10-scratch`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images` | +| Box | `c10.felhom.eu` @ `192.168.0.105` (demo-hp's vmbr0, `192.168.0.87/24`) | +| Console | no browser on DooPlex → `qm monitor screendump` → PNG, read visually; `sendkey` for input | +| Untouched | `drill-r50` (VM 300), guest 9201 on both demo boxes, `local-lvm`, both backup targets, ep0, the Storage Box | + +**The `exactMount` choice, made deliberately (the brief asks which).** `c10-scratch` sits at the mount +**root**, not a subdirectory. A dir storage at a subdirectory sets `exactMount=false` → +`reachable=false` → `StorageStateDisconnected` (`felhom-agent/internal/storage/observe.go:336`), which +would have emitted `storage_disconnected` for demo-hp for the entire run — **the exact signal I1/I2 +exist to discriminate**. It coexists with `felhom-backup` on the same path, which was not modified. + +### Baselines — every value read fresh + +| What | Value | How | +|---|---|---| +| controller `main` | **v0.188.0** | CHANGELOG top; `HEAD`=`origin/main`=`4115e88`, clean | +| **golden's baked controller** | **0.188.0** (`7353d8be…`) | hub `/configuration`, selected option | +| agent published + vouched | **0.119.0** (`e37aca82…`) | hub `/configuration`, selected option | +| hub | **0.86.0** | live deploy image tag == CHANGELOG top | +| ISO | **1.26.1**, `f3cc86d5…`, 1 705 322 496 B | live round trip from `iso.felhom.eu` | +| `felhom.eu` HEAD | `e9a74a0`, clean | `git rev-parse` | +| managed floor | v0.156.0 (DB override) | hub `/configuration` | + +**Golden vs `main`: not behind — both 0.188.0.** The deliberate decision the brief asks for is +therefore vacuous this run; the campaign tests what is shipping *and* what is current. + +**Two of the brief's assumptions were stale; reading fresh caught both.** + +1. The brief was written against ISO **v1.25.0**. `iso.felhom.eu` began serving **1.26.1** at 18:37 Z + the same day, ~90 minutes before this run. Operator chose 1.26.1 — the current published artifact. +2. The brief and three docs say demo-hp has **no baked SSH key**; `ssh -o BatchMode=yes demo-hp` + authenticated **by key, first try**. Already **R-129**; **not re-filed**. + +--- + +## 2. The isolation gate — §2 + +| Target | Isolation | Proof | +|---|---|---| +| **PBS** | dedicated datastore `felhom-c10` on DooPlex + auth-id `c10@pbs!box` scoped to it alone | **CAPTURED** — `isolation/pbs-denial.txt` | +| **restic** | scratch subaccount `u629488-sub4`, own chroot home `felhom-campaign10` | **CAPTURED** — `isolation/restic-denial.txt` | +| **hub** | the campaign's own record `c10-soak`; no other customer touched | **CAPTURED** — §2.3 | + +**Both denials carry a POSITIVE CONTROL, and the first attempt failed it.** The initial PBS capture +returned four tidy 403s — including on the campaign's *own* datastore. All four were worthless: PBS +API tokens carry privilege separation, so a token's rights are the intersection of the user's ACL and +the token's, and user `c10@pbs` had none. Granting the scoped role to the user as well turned the +control green and left the denials standing. Without the control this would have shipped as a clean +isolation proof that proved only that the credential was broken. + + ## POSITIVE CONTROL — same token, its OWN datastore + GET /api2/json/admin/datastore/felhom-c10/snapshots {"data":[]} HTTP 200 + + ## DENIAL 1 — production datastore felhom-spike + permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-spike 403 + ## DENIAL 3 — DELETE namespace demo-felhom (destructive) + permission check failed - missing Datastore.Modify on /datastore/felhom-spike 403 + +restic, same shape — the credential writes and reads back in its own home, and every production path +is outside the chroot: + + POSITIVE CONTROL put/get/rm c10-canary.txt in own home → succeeds, content verified + DENIAL cd /home/felhom-peti-felhom → stat remote: No such file or directory + DENIAL ls .. → remote readdir("/home/../"): Permission denied + +**2.3 — hub.** The run created exactly one record, `c10-soak`, and bound exactly one appliance +(uuid `55ab3776-…`) to it. Existing customers (`demo-felhom`, `demo-hp`, `peti-felhom`, `drill-r50`, +`sess-f`) were read but never written. Verified from the hub's own log: every `c10-soak` line is the +campaign's, and no line names another customer. + +--- + +## 3. Scope constraint — Tier 3 cannot be isolated, so it was not run + +The brief requires the campaign's PBS to be *a dedicated datastore and namespace on DooPlex*. That is +unreachable, for two reasons that are each an **already-recorded deliberate position**: + +1. **Offsite hard-requires the DR tier** — `hub/internal/web/configs.go:1300` refuses outright: + *"Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS + key)"*. Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4). +2. **The DR tier only ever provisions on ep0** — peer allocation and endpoint sync use the lowest + `endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`). + +So the only two configurations are *DR tier on* → campaign PBS traffic lands on **ep0** (Tier 2, the +offsite of last resort, RAM-constrained per R-90, fenced by §3), or *DR tier off* → **no Tier 3 at +all**. Chosen: **DR tier OFF, offsite OFF** — the only option §3 permits. + +Consequence, stated up front rather than discovered in the results: the campaign touches **neither ep0 +nor the Storage Box at any point** — stronger isolation than asked for, obtained by not running the +tier. **Lost: every Tier-3 atom, I8 entirely, and the Tier-3 RTO/RPO rows.** Unaffected: the +two-drive core (R-112/113/114/116/117), D5, and I1–I7, I9–I11. + +**No `R-n` minted** — the register was grepped first and both halves are already recorded. This is a +campaign constraint, not a defect. + +--- + +## 4. Phase A gates + +| Gate | Result | Positive observable | +|---|---|---| +| **A1** install from the published ISO | **PASS** | `ii felhom-bootstrap 1.26.1`; unit `enabled`, fired at first boot; registered unclaimed with pairing code `FD6-A6S`, MACs, hardware and 3 SSH host keys; **4.2 GB actually written**, system booted on its own | +| **A2** claim | **PASS** | discriminator flipped `dashboard not yet claimed` → `authentication required`; hub: `[claim] customer c10-soak CLAIMED its dashboard` | +| **A3** enrol both drives, apps across them | **PASS** | both registered via the real endpoint; backup target healthy; 4 apps healthy | +| **A4** discriminators | **PASS** | all four apps seed and read back the same generation | +| **A5** budget + watchdog | **NOT DONE** | §8 | + +**A1 — 1.26.1 cannot auto-install, verified against the bytes** (not inferred from its manifest): +`auto-installer-mode.toml` 0 hits, `answer.toml` 0, `proxmox-auto-install` 0, exactly 2 interactive +menuentries. Driven blind through the **Terminal UI** (the graphical entry is undrivable — monitor +mouse events do not move the guest cursor). Three traps hit and handled: + +- GRUB's default is the graphical entry; the first `down` landed after the 15 s countdown. Reset and + sent it inside the window in one round trip. +- **The guest keymap was Hungarian while `sendkey` emits US scancodes.** Every symbol afterwards — + notably `@`, which is AltGr+V on a Hungarian layout — would have been silently mangled, *including + the 24-char root password*, locking the box out with no way back in. Switched to U.S. English + before typing anything; confirmed positively when `c10@felhom.eu` rendered a correct `@` and `.`. +- The recorded boot-order trap: `--boot` set in its **own** `qm set` after the disks existed and + verified from `qm config`; flipped to `order=scsi0` post-install and verified again. + +Only `scsi0` was attached during the install, so the installer offered exactly one target and could +not reach a data drive. The 50 G drives were attached afterwards. + +**A2** — the installer's root password stops working at day-0; it is replaced by a hub-vaulted +managed credential (`vaulted break-glass recovery credential … secret 32 chars`), retrieved via +`POST /hosts/c10-soak-8a405e/reveal-recovery-credential`. R-119 confirmed live again: the claim code +is emailed-only and the operator relayed it — the single human step. + +**A3** — both drives enrolled through **the real endpoint** (`POST /api/storage/attach`), which does +format → assign → guest-attach → register-stable-path in order: + +| Drive | Device | Stable path | Role | +|---|---|---|---| +| `adatok` | `/dev/sdb` | `/mnt/felhom-drives/adatok` | app data, `set_default` | +| `mentes` | `/dev/sdc` | `/mnt/felhom-drives/mentes` | **backup target** (PVE storage `felhom-backup` at `/mnt/mentes`) | + +A false start worth recording: hand-driving `format` + `assign` first left the drives **mounted but +unregistered** — invisible to both the raw-candidate scan and `/disks`, so the real enroll endpoint +could not resolve a UUID for them (`resolveEnrollUUID` sees neither list). Backed out (eject leaves +the raw `/mnt/` mount by design, `disks.go:606`, so the mounts also had to be undone) and re-ran +the real flow clean. **The lesson is the brief's own:** use the real endpoint, or manufacture a state +the product never produces. + +Backup-target offer flow (R-112/R-114/E-2) exercised as the customer would: assign returned +`restart_required: true`, and the state only flipped after the agent restart it asked for — + + before: {"degraded":true, "target":"local", "offer_path":"/mnt/felhom-drives/adatok", "message":"A rendszermentés jelenleg ugyanazon a lemezen van…"} + after: {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"} + +That healthy read is the **I5/I6 baseline**: everything healthy ⇒ no degraded banner, target reads +healthy, zero absent events. + +**Apps — four, healthy, covering both sides of D5's split:** + +| App | Data | Secret classes | Why chosen | +|---|---|---|---| +| **rallly** | postgres (`rallly-postgres`) | 2 × `type: secret` (`SECRET_PASSWORD` hex:32 labelled *Titkosítási kulcs*, `DB_PASSWORD`) | the DB app; travelling class | +| **homebox** | sqlite volume | 1 × `secret`, **`data_key: true`** | data-encrypting key | +| **papra** | see **R-156** | 1 × `secret`, **`data_key: true`** | data-encrypting key | +| **grafana** | volume | 1 × **`type: password`** | **the withheld class — I10's negative half** | + +4 travelling `secret` fields and 1 `password` field that must never travel, so I10 can assert both +directions rather than only the positive one. + +**A4 — discriminators work.** Every app carries a generation marker; a full seed/read cycle returns +the seeded generation from all four. rallly's is a row in a `cc_proof` table read over **the path +`DATABASE_URL` actually names** (`docker run --network container:rallly … psql -h rallly-postgres`), +never `127.0.0.1` inside the postgres container — the trusted-socket path that produced D5's §1.2 +false pass. + +--- + +## 5. Findings + +### R-156 — papra's data is neither persisted nor backed up, and it reports healthy + +**New. Register grepped first:** papra appears in R-41 (never-deployable / healthcheck) and R-127 +(`data_key` misclassification); neither covers this. *(Also corrected en route: a first grep suggested +R-164 was taken — it is a mis-citation of controller **v0.164.0**, already flagged in +`REPORT-record-correction-2026-07-29.md:141`. Highest real number is R-155, so this is R-156.)* + +The template mounts `papra_data:/app/data`, but the app persists to **`/app/app-data/db/db.sqlite`**. +The mounted volume is therefore empty, and the real database lives in the container's writable layer. + +Evidence (`r156-papra-volume.txt`), all four legs measured: + + 1. volume papra_papra_data -> /app/data + 2. uid=999(nonroot); drwxr-xr-x 2 root root /app/data + touch: cannot touch '/app/data/.w': Permission denied + 3. /app/data contains only the canary this campaign wrote AS ROOT + 4. /app/app-data/db/db.sqlite 475136 bytes owned nonroot <-- the real DB, in no volume + 5. health status: "healthy" + +**Consequences.** (a) papra's documents do not survive `docker compose down`/redeploy/host restore — +the writable layer is not persisted. (b) `DumpAppVolumes` dumps the volume, so papra is **backed up +as an empty directory**: its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. (c) The +healthcheck only probes the HTTP port, so none of this surfaces — the app is green throughout. + +This is the *presence is not success* family from `CLAUDE.md`: a backup exists, a healthcheck passes, +and neither means the data is there. It is also a second instance of the pattern R-41 names — an app +that deploys and looks fine while being fundamentally broken. + +**Two-repo shape**, if fixed: the volume target belongs in `app-catalog-felhom.eu`, and the general +defence is a catalog gate asserting each template's mounted volume is a path the app actually writes. +**Not fixed here** — the fences forbid it, and a fix mid-run would prove a version that did not exist +when the run started. + +### Not filed + +- **demo-hp SSH key** — already R-129. +- **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect. + +--- + +## 6. RTO/RPO + +**No measurements.** They are a byproduct of Phase B restores, which did not run. Nothing is +extrapolated; no band is claimed. + +--- + +## 7. I1–I11 + +**Not measured across cycles.** Only the healthy baseline (I5/I6) and the discriminator mechanism (I7) +were established, as Phase A gates. A clean invariant is only a result with a cycle count behind it, +and there is no count to report. + +--- + +## 8. What did not run, and why + +**Phase B did not start.** Phase A was budgeted at ~1 h by the brief and took ~5.5 h, almost entirely +in places the brief could not have predicted: + +- ISO **1.26.1** turned out to be a public release image with no auto-install path, so the install was + a blind screendump-and-`sendkey` walk (~1.5 h) rather than an unattended boot. +- The Tier-3 investigation (§3) had to be settled before a customer could be created at all, because + `dr_tier` is decided at customer-creation time. +- The enrol false start, and the CSRF/session/Bearer probing needed to drive the real endpoints. + +That put the runner — the most correctness-sensitive component, the thing that decides destructive +atoms unattended and judges eleven invariants — at ~04:00 with ~3 h of usable night. **I stopped +rather than write it there.** The brief's own fence is the reason: *"A test rig producing false +negatives is worse than no rig."* A soak whose judgements cannot be trusted would not just waste the +night, it would contaminate the register with findings nobody could rely on. + +**Not run:** every B1 atom; I1–I11 across cycles; §B3 RTO/RPO; A5's disk budget, watchdog and +heartbeat (A5 exists to bound Phase B and was not written in isolation from it). + +**What is ready.** The rig is built, gated and idle: VM 311 installed and claimed, two drives enrolled +with the backup target healthy, four apps healthy spanning both D5 secret classes, a working +discriminator across all four, an authenticated API path onto the controller, and both isolation +denials captured. The driver library (`c10lib.py`) already implements the parts that took the +longest — the jump-host command path, the authenticated API helper, canary seed/read, and the +red-proofed psql read path. **Phase B needs the atom set, the invariant checks and the watchdog on +top of that, then it can run in daylight where its first cycles can be watched.** + +--- + +## 9. Teardown — OWED, nothing removed yet + +The rig is intact **on purpose**, so Phase B can run without repeating Phase A. Every item below is +outstanding and must be removed when the campaign ends: + +| Layer | Item | Command | +|---|---|---| +| VM | `311 c10-appliance` on demo-hp | `qm stop 311 && qm destroy 311 --purge` | +| storage | `c10-scratch` dir storage | `pvesm remove c10-scratch` | +| PBS | datastore `felhom-c10`, user `c10@pbs`, token `!box`, both ACLs | `proxmox-backup-manager datastore remove felhom-c10` · `user delete c10@pbs` · remove `/mnt/5_hdd/backup/campaign10` | +| restic | subaccount **`u629488-sub4`** (`felhom-campaign10`) on box 611714 — **unused, since Tier 3 never ran** | `DELETE /v1/storage_boxes/611714/subaccounts/281530` | +| **hub** | **customer `c10-soak`** — **disposition: DELETE.** Named explicitly per R-131, which is four orphaned scratch customers left by exactly this omission | `POST /configs/c10-soak/delete` with `ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=c10-soak` | +| secrets | `~/.config/campaign10/` on DooPlex (host + dashboard passwords, API key, app secrets, PBS token, restic password); `/root/c10/` and `/root/c10api.sh` in guest 9201 | `shred -u` both | + +`sess-f` ("R-120 golden 0.186.0 proof") is still present and is **not** this run's record — R-131 +stands, untouched. + +--- + +## 10. Hygiene note + +The customer API key for `c10-soak` was printed into this session's transcript while being read from +the hub page. It belongs to a scratch customer scheduled for deletion (§9) and grants nothing once +that record is gone, but the write-out was avoidable and is recorded here rather than left implicit — +the same class as R-132, where a correctly-made request was undone by how it was reported. diff --git a/documentation/tests/campaign10-evidence-2026-07-31/isolation/pbs-denial.txt b/documentation/tests/campaign10-evidence-2026-07-31/isolation/pbs-denial.txt new file mode 100644 index 0000000..44d0d64 --- /dev/null +++ b/documentation/tests/campaign10-evidence-2026-07-31/isolation/pbs-denial.txt @@ -0,0 +1,25 @@ +# PBS isolation proof — token c10@pbs!box (secret redacted, never echoed) +# DooPlex PBS https://localhost:8007 2026-07-31T20:14:40Z +# ACL: c10@pbs and c10@pbs!box hold DatastoreBackup on /datastore/felhom-c10 ONLY. +# felhom-spike (production, demo-felhom's backups) carries NO grant for this auth-id. + +## POSITIVE CONTROL — same token, its OWN datastore (must SUCCEED, else the 403s below prove nothing) +$ GET /api2/json/admin/datastore/felhom-c10/snapshots +{"data":[]} +HTTP 200 + +## DENIAL 1 — list snapshots of PRODUCTION datastore felhom-spike +permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-spike +HTTP 403 + +## DENIAL 2 — list NAMESPACES of felhom-spike +permission check failed +HTTP 403 + +## DENIAL 3 — DELETE namespace demo-felhom in felhom-spike (destructive) +permission check failed - missing Datastore.Modify on /datastore/felhom-spike +HTTP 403 + +## DENIAL 4 — status of felhom-spike +permission check failed +HTTP 403 diff --git a/documentation/tests/campaign10-evidence-2026-07-31/isolation/restic-denial.txt b/documentation/tests/campaign10-evidence-2026-07-31/isolation/restic-denial.txt new file mode 100644 index 0000000..2f67f19 --- /dev/null +++ b/documentation/tests/campaign10-evidence-2026-07-31/isolation/restic-denial.txt @@ -0,0 +1,53 @@ +# restic/Storage-Box isolation proof — subaccount u629488-sub4 (home felhom-campaign10) +# box 611714 storage-box-pool-1 · server u629488.your-storagebox.de · 2026-07-31T20:15:55Z +# Production subaccounts on the SAME box: sub1=felhom-demo-felhom sub2=felhom-peti-felhom sub3=felhom-demo-hp +# Password never echoed; supplied to sshpass via env. + +## POSITIVE CONTROL — sub4 in its OWN home (must SUCCEED) +$ sftp> pwd ; ls -a ; put canary ; ls +Warning: Permanently added '[u629488.your-storagebox.de]:23' (ED25519) to the list of known hosts. +Connected to u629488.your-storagebox.de. +sftp> pwd +Remote working directory: /home +sftp> ls -a +. .. + +## DENIAL 1 — reach Peti's REAL customer repo path from sub4 +$ sftp> cd /home/felhom-peti-felhom / cd ../felhom-peti-felhom / ls /felhom-peti-felhom +Connected to u629488.your-storagebox.de. +sftp> cd /home/felhom-peti-felhom +stat remote: No such file or directory +sftp> cd ../felhom-peti-felhom +stat remote: No such file or directory +sftp> ls /felhom-peti-felhom +Can't ls: "/felhom-peti-felhom" not found +sftp> ls .. +remote readdir("/home/../"): Permission denied + +## DENIAL 2 — reach demo-felhom's and demo-hp's production repo paths +Connected to u629488.your-storagebox.de. +sftp> cd /home/felhom-demo-felhom +stat remote: No such file or directory +sftp> ls /felhom-demo-felhom +Can't ls: "/felhom-demo-felhom" not found +sftp> cd /home/felhom-demo-hp +stat remote: No such file or directory +sftp> ls /felhom-demo-hp +Can't ls: "/felhom-demo-hp" not found + +## POSITIVE CONTROL (STRENGTHENED) — sub4 must be able to WRITE and READ BACK in its own home, +## proving the credential is fully functional and the denials above are AUTHORIZATION, not a dud login. +Connected to u629488.your-storagebox.de. +sftp> put /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0d68ad3e-5c39-4852-8e38-9c029dff2562/scratchpad/c10-canary.txt c10-canary.txt +Uploading /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0d68ad3e-5c39-4852-8e38-9c029dff2562/scratchpad/c10-canary.txt to /home/c10-canary.txt +sftp> ls -l +-rw-rw-r-- ? u629488-sub4 1059 56 Jul 31 22:16 c10-canary.txt +sftp> get c10-canary.txt /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0d68ad3e-5c39-4852-8e38-9c029dff2562/scratchpad/c10-canary-readback.txt +Fetching /home/c10-canary.txt to /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0d68ad3e-5c39-4852-8e38-9c029dff2562/scratchpad/c10-canary-readback.txt +sftp> rm c10-canary.txt +Removing /home/c10-canary.txt +sftp> ls -l +readback content: campaign10 write-capability canary 2026-07-31T20:16:23Z + +## VERDICT: write+read succeed in own home; every production path is outside the chroot +## ('No such file or directory') and the parent directory is an explicit 'Permission denied'. diff --git a/documentation/tests/campaign10-evidence-2026-07-31/r156-papra-volume.txt b/documentation/tests/campaign10-evidence-2026-07-31/r156-papra-volume.txt new file mode 100644 index 0000000..cf8f790 --- /dev/null +++ b/documentation/tests/campaign10-evidence-2026-07-31/r156-papra-volume.txt @@ -0,0 +1,20 @@ +# R-156 evidence ��� papra persists OUTSIDE the volume the compose mounts +## 1. compose mounts papra_data at /app/data +volume papra_papra_data -> /app/data + +## 2. the container runs as nonroot and CANNOT write there +uid=999(nonroot) gid=999(nonroot) groups=999(nonroot) +drwxr-xr-x 2 root root 4096 Jul 31 23:18 /app/data +touch: cannot touch '/app/data/.w': Permission denied +## 3. the volume is empty apart from the canary this campaign wrote AS ROOT +total 12 +drwxr-xr-x 2 root root 4096 Jul 31 23:18 . +drwxr-xr-x 1 nonroot nonroot 4096 Jul 31 23:12 .. +-rw-r--r-- 1 root root 19 Jul 31 23:18 cc_proof.txt +## 4. the REAL database is in the container writable layer, not in any volume +total 472 +drwxr-xr-x 1 nonroot nonroot 4096 Jul 31 23:12 . +drwxr-xr-x 1 nonroot nonroot 4096 Jul 4 22:19 .. +-rw-r--r-- 1 nonroot nonroot 475136 Jul 31 23:12 db.sqlite +## 5. healthcheck only probes the HTTP port, so this reads healthy +"healthy" diff --git a/documentation/tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md b/documentation/tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md new file mode 100644 index 0000000..83e3653 --- /dev/null +++ b/documentation/tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md @@ -0,0 +1,156 @@ +# Campaign 10 — Phase A journal (2026-07-31) + +All times CEST unless marked Z. + +## Venue + +| | | +|---|---| +| Host | `demo-hp` (HP t740), Tier 0, the designated drill/build host | +| VM | **311 `c10-appliance`** — q35 / OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, cpu=host | +| Disks | `scsi0` 200 G (system) · `scsi1` 50 G · `scsi2` 50 G — all qcow2 on `c10-scratch` | +| Storage | **`c10-scratch`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images`. Root chosen deliberately: a storage at a *subdirectory* sets `exactMount=false` → `reachable=false` → `StorageStateDisconnected` (`felhom-agent/internal/storage/observe.go:336`), which would emit `storage_disconnected` for demo-hp for the whole run — the exact signal I1/I2 test. Coexists with `felhom-backup` on the same path; that storage was **not modified**. | +| Guest network | demo-hp's vmbr0 is `192.168.0.87/24`, a **different physical LAN** from DooPlex's own 192.168.0.0/24. Reached via `ssh -J demo-hp`. | +| Console | no browser on DooPlex → `qm monitor screendump` → PPM → PNG, read visually; keystrokes via `sendkey` | +| Untouched | `drill-r50` (VM 300), guest 9201 on both demo boxes, `local-lvm`, both backup targets | + +## Baselines — every value read fresh at the start of this run + +| What | Value | How read | +|---|---|---| +| controller `main` | **v0.188.0** | `CHANGELOG.md` top; `HEAD`=`origin/main`=`4115e88`, tree clean | +| **golden's baked controller** | **0.188.0** (sha `7353d8be…`) | hub `/configuration`, *selected* option | +| agent published + vouched | **0.119.0** (sha `e37aca82…`) | hub `/configuration`, *selected* option | +| agent on both demo boxes | 0.119.0 | `felhom-agent --version` | +| agent `main` | v0.119.0 | `felhom-agent/CHANGELOG.md` | +| hub | **0.86.0** | live `deploy/hub` image tag == `hub/CHANGELOG.md` top | +| ISO | **1.26.1**, sha256 `f3cc86d5…`, 1 705 322 496 B | live round trip from `https://iso.felhom.eu/` | +| `felhom.eu` HEAD | `e9a74a0`, clean | `git rev-parse` | +| managed floor | v0.156.0 (DB override); env fallback v0.120.0 | hub `/configuration` | + +**Golden vs `main`: NOT behind — both 0.188.0.** The deliberate decision the brief asks for is therefore +vacuous this run: the campaign tests what is shipping *and* what is current, with no divergence. + +**Two brief assumptions were stale and were corrected by reading fresh:** + +1. The brief was written against ISO **v1.25.0**; `iso.felhom.eu` began serving **1.26.1** at 18:37 Z + the same day — ~90 minutes before this run started. +2. The brief (and `CLAUDE.md`, `target-selection.md:41-42`) say demo-hp has **no baked SSH key**. + `ssh -o BatchMode=yes demo-hp` authenticated **by key, first try**. Already filed as **R-129** by the + 2026-07-31 tester-gate run — **not re-filed** (register grep per the fences). + +## A1 — build the appliance from the published ISO + +**1.26.1 is a PUBLIC RELEASE image and cannot auto-install.** Verified against the ISO bytes on +demo-hp, not inferred from the manifest: + +| Probe | Hits | +|---|---| +| `auto-installer-mode.toml` | **0** | +| `answer.toml` | **0** | +| `proxmox-auto-install` | **0** | +| `felhom-bootstrap_1.26.1_all.deb` | 1 | +| Felhom grub menuentries | exactly **2**, both interactive | + +Manifest agrees: `mode: release … NO answer.toml, NO baked credential, interactive disk selection`, +`root-password: NONE — the installer prompts`, `loader: shim (stock MS-signed chain)`. + +Driven blind through the **Terminal UI** entry (the graphical entry is not drivable — the QEMU monitor's +mouse events do not move the guest cursor, per `REPORT-iso-release.md`). Screendump before every +keystroke. + +**Traps hit and handled, each verified rather than assumed:** + +- **GRUB default is the graphical entry.** The first attempt auto-booted it because the `down` keystroke + landed after the 15 s countdown expired. Reset, and sent `down` inside the window in one round trip. +- **Guest keymap was Hungarian**, but `sendkey` emits US scancodes — every symbol afterwards (notably + `@`, which is AltGr+V on a Hungarian layout) would have been silently mangled, *including the root + password*, locking the box out. Switched to **U.S. English** before typing anything. Confirmed + positively: `c10@felhom.eu` rendered with a correct `@` and `.`. +- **Disk-selection safety:** only `scsi0` was attached during the install, so the installer offered + exactly one target (`/dev/sda`, 200 GiB) and could not reach a data drive. The two 50 G data drives + were attached afterwards. +- **Boot-order trap** (`REPORT-iso-release.md`, recorded twice there): `--boot` was set in its **own** + `qm set` after the disks existed, and verified from `qm config` (`boot: order=ide2;scsi0`). After the + install, the order was flipped to `order=scsi0` — again verified from `qm config`, not from the screen. + +**Real completion signal** (not "an artifact changed"): 4.2 GB actually written to +`vm-311-disk-1.qcow2`, the installed system booted on its own, and the console showed the PVE banner at +`https://192.168.0.105:8006/`. + +**Positive observables — all three present:** + +| Observable | Evidence | +|---|---| +| package delivered | `ii felhom-bootstrap 1.26.1 all` on the box | +| unit fired at first boot | `systemctl is-enabled felhom-bootstrap` = `enabled`; `is-active` = `activating` | +| registers as an unclaimed appliance | hub `/hosts`: uuid `55ab3776-…` (matches VM smbios), pairing code **FD6-A6S**, MAC `bc:24:11:46:25:fe`, 3 SSH host keys, hardware read | + +**A1 — PASS.** + +## A2 — claim + +Customer created: **`c10-soak`** — "Campaign 10 two-storage soak (SCRATCH - delete at teardown)", +domain `c10.felhom.eu`, email `nagyfenyvesi.viktor@gmail.com`. Deliberately **not** `drill-r50`, +`sess-*`, or any existing record. + +Appliance 19 bound to it. Hub log, verbatim: + + 22:50:03 appliance 19 BOUND to customer c10-soak (mode=appliance) — delivery staged for its next poll + 22:50:12 appliance credentials DELIVERED once to appliance 19 (passphrase withheld) + 22:50:19 [claim] claim code (gen 1) emailed to the registered address of c10-soak + 22:50:49 host enrolled: c10-soak-8a405e (customer c10-soak) + 22:50:49 vaulted break-glass recovery credential for host c10-soak-8a405e (user=root@pam, secret 32 chars) + 22:50:51 Artifact manifest served for customer c10-soak (agent=0.119.0 golden=0.188.0) + 22:51:13 wg registered: host=c10-soak-8a405e ip=10.77.0.6/32 gen=1 sync=ok + 22:51:14 host-report from c10-soak-8a405e (0 guests, 2 storage targets, ...) + 22:51:14 DR-recipe host-half stored for customer c10-soak (host c10-soak-8a405e, v1) + +**The installer's root password stops working at 22:50:49** — day-0 replaces it with a hub-vaulted +managed credential (G1 `host_recovery`). Retrieved via `POST /hosts/c10-soak-8a405e/reveal-recovery-credential` +→ `root@pam`, 32 chars, stored file→file at `~/.config/campaign10/host-recovery.json` (0600), never printed. + +Box state after day-0: + + felhom-agent 0.119.0 (== published+vouched) + /etc/felhom-controller-image : …/felhom-controller:0.188.0 (== golden == main) + guest 9201 c10-soak running + docker: controller 0.188.0 (healthy), traefik v3.6.7, filebrowser 1.3.3-stable (healthy) + block devices: sda 200G, sdb 50G, sdc 50G + +Claim gate, the positive discriminator, read over the path the UI uses (traefik on 443, `Host:` header +mandatory — the controller does **not** listen on 8080; 80 is a 301 to HTTPS): + + GET https://127.0.0.1/api/apps Host: felhom.c10.felhom.eu + → {"ok":false,"error":"dashboard not yet claimed"} + +**A2 — BLOCKED on the emailed claim code (R-119, confirmed live again).** The code exists only in the +customer's mailbox; there is no operator-side route. + +## Scope constraint found in Phase A — Tier 3 cannot be isolated, so it will not run + +The brief's §2 requires the campaign's PBS to be *a dedicated datastore and namespace on DooPlex*. That +is not reachable, for two reasons that are each **already-recorded deliberate positions**, not new defects: + +1. **Offsite hard-requires the DR tier.** `hub/internal/web/configs.go:1300` refuses offsite without it: + *"Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS key)"*. + Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4). +2. **The DR tier only ever provisions on ep0.** Peer allocation and endpoint sync use the lowest + `endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`). + +So the only two configurations are: *DR tier on* → the campaign's PBS traffic lands on **ep0**, which is +Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90) and fenced by §3 — or *DR tier off* → +no Tier 3 at all. **Chosen: DR tier OFF, offsite OFF**, which is the only option §3 permits. + +Consequence, stated plainly rather than discovered later: the campaign touches **neither ep0 nor the +Storage Box at any point** — a stronger isolation than the brief asked for, obtained by not running the +tier. What is lost: all Tier-3 operations in B1, **I8** entirely, and the Tier-3 rows of the RTO/RPO +table. Everything else — the two-drive core (R-112/R-113/R-114/R-116/R-117), D5, I1–I7 and I9–I11 — is +unaffected, because those live on the drives and the local/whole-guest tiers. + +**No `R-n` minted**: per the fences, the register was grepped first and both halves are already recorded +(F-6 policy close; the `hub/README.md:260` deferral). This is a campaign constraint, not a finding. + +## Isolation gate — §2 + +See `../isolation/pbs-denial.txt` and `../isolation/restic-denial.txt`. Summary in the main audit.