admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Campaign 10: Phase A complete + gated; Phase B not run; R-156 filed

Browse Source 
Phase A passed every gate on a fresh box built from the PUBLISHED ISO 1.26.1:
install, claim, two drives enrolled through the real endpoints with the backup
target healthy, four apps spanning both sides of D5's secret split, and a
working discriminator across all four.

Isolation gate: both denials captured, each with a positive control. The PBS
control FAILED first — four clean-looking 403s were worthless because the token
was denied on its own datastore too (PBS token privilege separation). Fixed and
re-run; the denials stand.

R-156 (new, register grepped): papra's data is neither persisted nor backed up,
and it reports healthy. The template mounts papra_data:/app/data; the app writes
/app/app-data/db/db.sqlite. Volume empty and root-owned against a -rootless
image, real DB in the container writable layer, healthcheck only probes the HTTP
port. Its Tier-1/2 backup is real, verifiable and contains nothing. Not fixed.

Tier 3 could not be isolated so it was not run: offsite hard-requires the DR
tier (configs.go:1300) and the DR tier only provisions on ep0 (per-endpoint
allocation deferred, hub/README.md:260). Both are recorded deliberate positions,
so no R-n minted. The campaign touched neither ep0 nor the Storage Box.

Phase B did not start. Phase A was budgeted at ~1h and took ~5.5h (1.26.1 is a
public release image with no auto-install path, so the install was a blind
screendump+sendkey walk). That left the runner — which judges eleven invariants
and fires destructive atoms unattended — to be written at 04:00 with ~3h of
night left. Stopped on the brief's own fence: a rig producing false negatives is
worse than no rig. The rig is built and idle; teardown is OWED and itemised,
including hub customer c10-soak (disposition: DELETE).
This commit is contained in:
authentik Default Admin
2026-07-31 23:22:25 +02:00
parent e9a74a0019
commit 4691aa1a35
6 changed files with 618 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT-campaign10.md
+62
View File
@@ -0,0 +1,62 @@
# REPORT — Campaign 10, two-storage adversarial soak (2026-07-31)
Follows `REPORT-campaign7/8/9.md`. Root `REPORT.md` is another session's (hub v0.85.0) and was not
clobbered — same shared-clone reasoning as `REPORT-iso-release.md`.
**Full audit + evidence:** `documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md`,
`documentation/tests/campaign10-evidence-2026-07-31/`.
## The sentence that matters
**Phase A is complete and every gate passed. Phase B did not run.** The rig is built, fenced and
idle, waiting for a runner that should not be written at 04:00.
## What was established
- **Venue** — VM 311 on demo-hp (Tier 0), 200 G system + 2 × 50 G data, scratch storage at the
`/mnt/nvme-1tb` **mount root** (a subdirectory would have emitted `storage_disconnected` for demo-hp
all night — the exact signal I1/I2 discriminate).
- **Baselines, all read fresh.** controller `main` 0.188.0, **golden 0.188.0 (not behind)**, agent
0.119.0 published+vouched, hub 0.86.0, ISO **1.26.1** (`f3cc86d5…`, round-trip verified live).
The brief's ISO assumption (v1.25.0) was ~90 minutes stale; its "no baked SSH key" claim is R-129.
- **Isolation gate — both denials captured, each with a positive control.** The PBS control **failed
first**: four clean-looking 403s were worthless because the token was denied on its own datastore
too (PBS token privilege separation). Fixed, re-run, denials stand.
- **A1** fresh install from the **published** ISO. 1.26.1 is a public release image — verified against
its bytes that it has no auto-install path — so it was driven blind via screendump + `sendkey`
through the Terminal UI. Caught the Hungarian-keymap trap before typing the root password, which
would otherwise have been mangled and locked the box out.
- **A2** claimed for real; discriminator flipped `dashboard not yet claimed` → `authentication required`.
- **A3** both drives enrolled through the **real** endpoint; `mentes` accepted as backup target via
the offer flow, ending `degraded:false / target:felhom-backup` — the I5/I6 healthy baseline.
Four apps healthy spanning both sides of D5's split (4 × `type: secret`, 1 × `type: password`).
- **A4** discriminators seed and read back across all four apps; rallly's over the path
`DATABASE_URL` actually names, not the trusted socket that produced D5's false pass.
## Findings
- **R-156 (new)** — **papra's data is neither persisted nor backed up, and it reports healthy.** The
template mounts `papra_data:/app/data`; the app writes `/app/app-data/db/db.sqlite`. The volume is
empty and root-owned (the image is `-rootless`, so the app cannot even write there), the real DB
sits in the container's writable layer, and the healthcheck only probes the HTTP port. Its
Tier-1/Tier-2 backup is real, verifiable, and contains nothing. Not fixed.
- **Tier 3 could not be isolated, so it was not run.** Offsite hard-requires the DR tier
(`configs.go:1300`), and the DR tier only provisions on ep0 (per-endpoint allocation deferred,
`hub/README.md:260`). Both are recorded deliberate positions, so **no R-n minted**. The campaign
therefore touched neither ep0 nor the Storage Box — stronger isolation than asked for, obtained by
not running the tier. Cost: all Tier-3 atoms, I8, and the Tier-3 RTO/RPO rows.
## What did not run
Every B1 atom, I1–I11 across cycles, RTO/RPO, and A5's budget/watchdog. Phase A was budgeted at ~1 h
and took ~5.5 h — the blind interactive install alone was ~1.5 h. That left the runner, the component
that judges eleven invariants and fires destructive atoms unattended, to be written at 04:00 with ~3 h
of night left. Stopped instead, on the brief's own fence: *a test rig producing false negatives is
worse than no rig.*
## Teardown — OWED, nothing removed
Deliberately intact so Phase B need not repeat Phase A. VM 311, `c10-scratch`, PBS datastore
`felhom-c10` + user/token, restic subaccount `u629488-sub4`, and **hub customer `c10-soak` (disposition:
DELETE)** are all outstanding, with commands in the audit §9. Named explicitly because R-131 is four
orphaned scratch customers left by exactly this omission.
documentation/audits/CAMPAIGN-10-two-storage-soak-2026-07-31.md
+302
View File
@@ -0,0 +1,302 @@
# CAMPAIGN 10 — two-storage adversarial soak (2026-07-31 → 08-01)
**Status: Phase A COMPLETE and gated. Phase B NOT STARTED — deliberately, see §8.**
**One new finding: R-156. One structural scope constraint: Tier 3 cannot be isolated (§3).**
Evidence: `../tests/campaign10-evidence-2026-07-31/` — `state/phaseA-journal.md` (step-by-step),
`isolation/pbs-denial.txt`, `isolation/restic-denial.txt`, `r156-papra-volume.txt`.
---
## 1. Venue and baselines
| | |
|---|---|
| Host | `demo-hp` (HP t740), **Tier 0**, the designated drill/build host |
| VM | **311 `c10-appliance`** — q35/OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, `cpu=host` |
| Disks | `scsi0` 200 G system · `scsi1` 50 G · `scsi2` 50 G, qcow2 on `c10-scratch` |
| Storage | **`c10-scratch`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images` |
| Box | `c10.felhom.eu` @ `192.168.0.105` (demo-hp's vmbr0, `192.168.0.87/24`) |
| Console | no browser on DooPlex → `qm monitor screendump` → PNG, read visually; `sendkey` for input |
| Untouched | `drill-r50` (VM 300), guest 9201 on both demo boxes, `local-lvm`, both backup targets, ep0, the Storage Box |
**The `exactMount` choice, made deliberately (the brief asks which).** `c10-scratch` sits at the mount
**root**, not a subdirectory. A dir storage at a subdirectory sets `exactMount=false` →
`reachable=false` → `StorageStateDisconnected` (`felhom-agent/internal/storage/observe.go:336`), which
would have emitted `storage_disconnected` for demo-hp for the entire run — **the exact signal I1/I2
exist to discriminate**. It coexists with `felhom-backup` on the same path, which was not modified.
### Baselines — every value read fresh
| What | Value | How |
|---|---|---|
| controller `main` | **v0.188.0** | CHANGELOG top; `HEAD`=`origin/main`=`4115e88`, clean |
| **golden's baked controller** | **0.188.0** (`7353d8be…`) | hub `/configuration`, selected option |
| agent published + vouched | **0.119.0** (`e37aca82…`) | hub `/configuration`, selected option |
| hub | **0.86.0** | live deploy image tag == CHANGELOG top |
| ISO | **1.26.1**, `f3cc86d5…`, 1 705 322 496 B | live round trip from `iso.felhom.eu` |
| `felhom.eu` HEAD | `e9a74a0`, clean | `git rev-parse` |
| managed floor | v0.156.0 (DB override) | hub `/configuration` |
**Golden vs `main`: not behind — both 0.188.0.** The deliberate decision the brief asks for is
therefore vacuous this run; the campaign tests what is shipping *and* what is current.
**Two of the brief's assumptions were stale; reading fresh caught both.**
1. The brief was written against ISO **v1.25.0**. `iso.felhom.eu` began serving **1.26.1** at 18:37 Z
the same day, ~90 minutes before this run. Operator chose 1.26.1 — the current published artifact.
2. The brief and three docs say demo-hp has **no baked SSH key**; `ssh -o BatchMode=yes demo-hp`
authenticated **by key, first try**. Already **R-129**; **not re-filed**.
---
## 2. The isolation gate — §2
| Target | Isolation | Proof |
|---|---|---|
| **PBS** | dedicated datastore `felhom-c10` on DooPlex + auth-id `c10@pbs!box` scoped to it alone | **CAPTURED** — `isolation/pbs-denial.txt` |
| **restic** | scratch subaccount `u629488-sub4`, own chroot home `felhom-campaign10` | **CAPTURED** — `isolation/restic-denial.txt` |
| **hub** | the campaign's own record `c10-soak`; no other customer touched | **CAPTURED** — §2.3 |
**Both denials carry a POSITIVE CONTROL, and the first attempt failed it.** The initial PBS capture
returned four tidy 403s — including on the campaign's *own* datastore. All four were worthless: PBS
API tokens carry privilege separation, so a token's rights are the intersection of the user's ACL and
the token's, and user `c10@pbs` had none. Granting the scoped role to the user as well turned the
control green and left the denials standing. Without the control this would have shipped as a clean
isolation proof that proved only that the credential was broken.
## POSITIVE CONTROL — same token, its OWN datastore
GET /api2/json/admin/datastore/felhom-c10/snapshots {"data":[]} HTTP 200
## DENIAL 1 — production datastore felhom-spike
permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-spike 403
## DENIAL 3 — DELETE namespace demo-felhom (destructive)
permission check failed - missing Datastore.Modify on /datastore/felhom-spike 403
restic, same shape — the credential writes and reads back in its own home, and every production path
is outside the chroot:
POSITIVE CONTROL put/get/rm c10-canary.txt in own home → succeeds, content verified
DENIAL cd /home/felhom-peti-felhom → stat remote: No such file or directory
DENIAL ls .. → remote readdir("/home/../"): Permission denied
**2.3 — hub.** The run created exactly one record, `c10-soak`, and bound exactly one appliance
(uuid `55ab3776-…`) to it. Existing customers (`demo-felhom`, `demo-hp`, `peti-felhom`, `drill-r50`,
`sess-f`) were read but never written. Verified from the hub's own log: every `c10-soak` line is the
campaign's, and no line names another customer.
---
## 3. Scope constraint — Tier 3 cannot be isolated, so it was not run
The brief requires the campaign's PBS to be *a dedicated datastore and namespace on DooPlex*. That is
unreachable, for two reasons that are each an **already-recorded deliberate position**:
1. **Offsite hard-requires the DR tier** — `hub/internal/web/configs.go:1300` refuses outright:
*"Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS
key)"*. Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4).
2. **The DR tier only ever provisions on ep0** — peer allocation and endpoint sync use the lowest
`endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).
So the only two configurations are *DR tier on* → campaign PBS traffic lands on **ep0** (Tier 2, the
offsite of last resort, RAM-constrained per R-90, fenced by §3), or *DR tier off* → **no Tier 3 at
all**. Chosen: **DR tier OFF, offsite OFF** — the only option §3 permits.
Consequence, stated up front rather than discovered in the results: the campaign touches **neither ep0
nor the Storage Box at any point** — stronger isolation than asked for, obtained by not running the
tier. **Lost: every Tier-3 atom, I8 entirely, and the Tier-3 RTO/RPO rows.** Unaffected: the
two-drive core (R-112/113/114/116/117), D5, and I1–I7, I9–I11.
**No `R-n` minted** — the register was grepped first and both halves are already recorded. This is a
campaign constraint, not a defect.
---
## 4. Phase A gates
| Gate | Result | Positive observable |
|---|---|---|
| **A1** install from the published ISO | **PASS** | `ii felhom-bootstrap 1.26.1`; unit `enabled`, fired at first boot; registered unclaimed with pairing code `FD6-A6S`, MACs, hardware and 3 SSH host keys; **4.2 GB actually written**, system booted on its own |
| **A2** claim | **PASS** | discriminator flipped `dashboard not yet claimed` → `authentication required`; hub: `[claim] customer c10-soak CLAIMED its dashboard` |
| **A3** enrol both drives, apps across them | **PASS** | both registered via the real endpoint; backup target healthy; 4 apps healthy |
| **A4** discriminators | **PASS** | all four apps seed and read back the same generation |
| **A5** budget + watchdog | **NOT DONE** | §8 |
**A1 — 1.26.1 cannot auto-install, verified against the bytes** (not inferred from its manifest):
`auto-installer-mode.toml` 0 hits, `answer.toml` 0, `proxmox-auto-install` 0, exactly 2 interactive
menuentries. Driven blind through the **Terminal UI** (the graphical entry is undrivable — monitor
mouse events do not move the guest cursor). Three traps hit and handled:
- GRUB's default is the graphical entry; the first `down` landed after the 15 s countdown. Reset and
sent it inside the window in one round trip.
- **The guest keymap was Hungarian while `sendkey` emits US scancodes.** Every symbol afterwards —
notably `@`, which is AltGr+V on a Hungarian layout — would have been silently mangled, *including
the 24-char root password*, locking the box out with no way back in. Switched to U.S. English
before typing anything; confirmed positively when `c10@felhom.eu` rendered a correct `@` and `.`.
- The recorded boot-order trap: `--boot` set in its **own** `qm set` after the disks existed and
verified from `qm config`; flipped to `order=scsi0` post-install and verified again.
Only `scsi0` was attached during the install, so the installer offered exactly one target and could
not reach a data drive. The 50 G drives were attached afterwards.
**A2** — the installer's root password stops working at day-0; it is replaced by a hub-vaulted
managed credential (`vaulted break-glass recovery credential … secret 32 chars`), retrieved via
`POST /hosts/c10-soak-8a405e/reveal-recovery-credential`. R-119 confirmed live again: the claim code
is emailed-only and the operator relayed it — the single human step.
**A3** — both drives enrolled through **the real endpoint** (`POST /api/storage/attach`), which does
format → assign → guest-attach → register-stable-path in order:
| Drive | Device | Stable path | Role |
|---|---|---|---|
| `adatok` | `/dev/sdb` | `/mnt/felhom-drives/adatok` | app data, `set_default` |
| `mentes` | `/dev/sdc` | `/mnt/felhom-drives/mentes` | **backup target** (PVE storage `felhom-backup` at `/mnt/mentes`) |
A false start worth recording: hand-driving `format` + `assign` first left the drives **mounted but
unregistered** — invisible to both the raw-candidate scan and `/disks`, so the real enroll endpoint
could not resolve a UUID for them (`resolveEnrollUUID` sees neither list). Backed out (eject leaves
the raw `/mnt/<name>` mount by design, `disks.go:606`, so the mounts also had to be undone) and re-ran
the real flow clean. **The lesson is the brief's own:** use the real endpoint, or manufacture a state
the product never produces.
Backup-target offer flow (R-112/R-114/E-2) exercised as the customer would: assign returned
`restart_required: true`, and the state only flipped after the agent restart it asked for —
before: {"degraded":true, "target":"local", "offer_path":"/mnt/felhom-drives/adatok", "message":"A rendszermentés jelenleg ugyanazon a lemezen van…"}
after: {"degraded":false,"known":true,"label":"Mentes","target":"felhom-backup"}
That healthy read is the **I5/I6 baseline**: everything healthy ⇒ no degraded banner, target reads
healthy, zero absent events.
**Apps — four, healthy, covering both sides of D5's split:**
| App | Data | Secret classes | Why chosen |
|---|---|---|---|
| **rallly** | postgres (`rallly-postgres`) | 2 × `type: secret` (`SECRET_PASSWORD` hex:32 labelled *Titkosítási kulcs*, `DB_PASSWORD`) | the DB app; travelling class |
| **homebox** | sqlite volume | 1 × `secret`, **`data_key: true`** | data-encrypting key |
| **papra** | see **R-156** | 1 × `secret`, **`data_key: true`** | data-encrypting key |
| **grafana** | volume | 1 × **`type: password`** | **the withheld class — I10's negative half** |
4 travelling `secret` fields and 1 `password` field that must never travel, so I10 can assert both
directions rather than only the positive one.
**A4 — discriminators work.** Every app carries a generation marker; a full seed/read cycle returns
the seeded generation from all four. rallly's is a row in a `cc_proof` table read over **the path
`DATABASE_URL` actually names** (`docker run --network container:rallly … psql -h rallly-postgres`),
never `127.0.0.1` inside the postgres container — the trusted-socket path that produced D5's §1.2
false pass.
---
## 5. Findings
### R-156 — papra's data is neither persisted nor backed up, and it reports healthy
**New. Register grepped first:** papra appears in R-41 (never-deployable / healthcheck) and R-127
(`data_key` misclassification); neither covers this. *(Also corrected en route: a first grep suggested
R-164 was taken — it is a mis-citation of controller **v0.164.0**, already flagged in
`REPORT-record-correction-2026-07-29.md:141`. Highest real number is R-155, so this is R-156.)*
The template mounts `papra_data:/app/data`, but the app persists to **`/app/app-data/db/db.sqlite`**.
The mounted volume is therefore empty, and the real database lives in the container's writable layer.
Evidence (`r156-papra-volume.txt`), all four legs measured:
1. volume papra_papra_data -> /app/data
2. uid=999(nonroot); drwxr-xr-x 2 root root /app/data
touch: cannot touch '/app/data/.w': Permission denied
3. /app/data contains only the canary this campaign wrote AS ROOT
4. /app/app-data/db/db.sqlite 475136 bytes owned nonroot <-- the real DB, in no volume
5. health status: "healthy"
**Consequences.** (a) papra's documents do not survive `docker compose down`/redeploy/host restore —
the writable layer is not persisted. (b) `DumpAppVolumes` dumps the volume, so papra is **backed up
as an empty directory**: its Tier-1/Tier-2 backup is real, verifiable, and contains nothing. (c) The
healthcheck only probes the HTTP port, so none of this surfaces — the app is green throughout.
This is the *presence is not success* family from `CLAUDE.md`: a backup exists, a healthcheck passes,
and neither means the data is there. It is also a second instance of the pattern R-41 names — an app
that deploys and looks fine while being fundamentally broken.
**Two-repo shape**, if fixed: the volume target belongs in `app-catalog-felhom.eu`, and the general
defence is a catalog gate asserting each template's mounted volume is a path the app actually writes.
**Not fixed here** — the fences forbid it, and a fix mid-run would prove a version that did not exist
when the run started.
### Not filed
- **demo-hp SSH key** — already R-129.
- **Tier 3 isolation** — §3; two recorded deliberate positions, not a defect.
---
## 6. RTO/RPO
**No measurements.** They are a byproduct of Phase B restores, which did not run. Nothing is
extrapolated; no band is claimed.
---
## 7. I1–I11
**Not measured across cycles.** Only the healthy baseline (I5/I6) and the discriminator mechanism (I7)
were established, as Phase A gates. A clean invariant is only a result with a cycle count behind it,
and there is no count to report.
---
## 8. What did not run, and why
**Phase B did not start.** Phase A was budgeted at ~1 h by the brief and took ~5.5 h, almost entirely
in places the brief could not have predicted:
- ISO **1.26.1** turned out to be a public release image with no auto-install path, so the install was
a blind screendump-and-`sendkey` walk (~1.5 h) rather than an unattended boot.
- The Tier-3 investigation (§3) had to be settled before a customer could be created at all, because
`dr_tier` is decided at customer-creation time.
- The enrol false start, and the CSRF/session/Bearer probing needed to drive the real endpoints.
That put the runner — the most correctness-sensitive component, the thing that decides destructive
atoms unattended and judges eleven invariants — at ~04:00 with ~3 h of usable night. **I stopped
rather than write it there.** The brief's own fence is the reason: *"A test rig producing false
negatives is worse than no rig."* A soak whose judgements cannot be trusted would not just waste the
night, it would contaminate the register with findings nobody could rely on.
**Not run:** every B1 atom; I1–I11 across cycles; §B3 RTO/RPO; A5's disk budget, watchdog and
heartbeat (A5 exists to bound Phase B and was not written in isolation from it).
**What is ready.** The rig is built, gated and idle: VM 311 installed and claimed, two drives enrolled
with the backup target healthy, four apps healthy spanning both D5 secret classes, a working
discriminator across all four, an authenticated API path onto the controller, and both isolation
denials captured. The driver library (`c10lib.py`) already implements the parts that took the
longest — the jump-host command path, the authenticated API helper, canary seed/read, and the
red-proofed psql read path. **Phase B needs the atom set, the invariant checks and the watchdog on
top of that, then it can run in daylight where its first cycles can be watched.**
---
## 9. Teardown — OWED, nothing removed yet
The rig is intact **on purpose**, so Phase B can run without repeating Phase A. Every item below is
outstanding and must be removed when the campaign ends:
| Layer | Item | Command |
|---|---|---|
| VM | `311 c10-appliance` on demo-hp | `qm stop 311 && qm destroy 311 --purge` |
| storage | `c10-scratch` dir storage | `pvesm remove c10-scratch` |
| PBS | datastore `felhom-c10`, user `c10@pbs`, token `!box`, both ACLs | `proxmox-backup-manager datastore remove felhom-c10` · `user delete c10@pbs` · remove `/mnt/5_hdd/backup/campaign10` |
| restic | subaccount **`u629488-sub4`** (`felhom-campaign10`) on box 611714 — **unused, since Tier 3 never ran** | `DELETE /v1/storage_boxes/611714/subaccounts/281530` |
| **hub** | **customer `c10-soak`** — **disposition: DELETE.** Named explicitly per R-131, which is four orphaned scratch customers left by exactly this omission | `POST /configs/c10-soak/delete` with `ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=c10-soak` |
| secrets | `~/.config/campaign10/` on DooPlex (host + dashboard passwords, API key, app secrets, PBS token, restic password); `/root/c10/` and `/root/c10api.sh` in guest 9201 | `shred -u` both |
`sess-f` ("R-120 golden 0.186.0 proof") is still present and is **not** this run's record — R-131
stands, untouched.
---
## 10. Hygiene note
The customer API key for `c10-soak` was printed into this session's transcript while being read from
the hub page. It belongs to a scratch customer scheduled for deletion (§9) and grants nothing once
that record is gone, but the write-out was avoidable and is recorded here rather than left implicit —
the same class as R-132, where a correctly-made request was undone by how it was reported.
documentation/tests/campaign10-evidence-2026-07-31/isolation/pbs-denial.txt
+25
View File
@@ -0,0 +1,25 @@
# PBS isolation proof — token c10@pbs!box (secret redacted, never echoed)
# DooPlex PBS https://localhost:8007 2026-07-31T20:14:40Z
# ACL: c10@pbs and c10@pbs!box hold DatastoreBackup on /datastore/felhom-c10 ONLY.
# felhom-spike (production, demo-felhom's backups) carries NO grant for this auth-id.
## POSITIVE CONTROL — same token, its OWN datastore (must SUCCEED, else the 403s below prove nothing)
$ GET /api2/json/admin/datastore/felhom-c10/snapshots
{"data":[]}
HTTP 200
## DENIAL 1 — list snapshots of PRODUCTION datastore felhom-spike
permission check failed - missing Datastore.Audit|Datastore.Backup on /datastore/felhom-spike
HTTP 403
## DENIAL 2 — list NAMESPACES of felhom-spike
permission check failed
HTTP 403
## DENIAL 3 — DELETE namespace demo-felhom in felhom-spike (destructive)
permission check failed - missing Datastore.Modify on /datastore/felhom-spike
HTTP 403
## DENIAL 4 — status of felhom-spike
permission check failed
HTTP 403
documentation/tests/campaign10-evidence-2026-07-31/isolation/restic-denial.txt
+53
View File
@@ -0,0 +1,53 @@
# restic/Storage-Box isolation proof — subaccount u629488-sub4 (home felhom-campaign10)
# box 611714 storage-box-pool-1 · server u629488.your-storagebox.de · 2026-07-31T20:15:55Z
# Production subaccounts on the SAME box: sub1=felhom-demo-felhom sub2=felhom-peti-felhom sub3=felhom-demo-hp
# Password never echoed; supplied to sshpass via env.
## POSITIVE CONTROL — sub4 in its OWN home (must SUCCEED)
$ sftp> pwd ; ls -a ; put canary ; ls
Warning: Permanently added '[u629488.your-storagebox.de]:23' (ED25519) to the list of known hosts.
Connected to u629488.your-storagebox.de.
sftp> pwd
Remote working directory: /home
sftp> ls -a
. ..
## DENIAL 1 — reach Peti's REAL customer repo path from sub4
$ sftp> cd /home/felhom-peti-felhom / cd ../felhom-peti-felhom / ls /felhom-peti-felhom
Connected to u629488.your-storagebox.de.
sftp> cd /home/felhom-peti-felhom
stat remote: No such file or directory
sftp> cd ../felhom-peti-felhom
stat remote: No such file or directory
sftp> ls /felhom-peti-felhom
Can't ls: "/felhom-peti-felhom" not found
sftp> ls ..
remote readdir("/home/../"): Permission denied
## DENIAL 2 — reach demo-felhom's and demo-hp's production repo paths
Connected to u629488.your-storagebox.de.
sftp> cd /home/felhom-demo-felhom
stat remote: No such file or directory
sftp> ls /felhom-demo-felhom
Can't ls: "/felhom-demo-felhom" not found
sftp> cd /home/felhom-demo-hp
stat remote: No such file or directory
sftp> ls /felhom-demo-hp
Can't ls: "/felhom-demo-hp" not found
## POSITIVE CONTROL (STRENGTHENED) — sub4 must be able to WRITE and READ BACK in its own home,
## proving the credential is fully functional and the denials above are AUTHORIZATION, not a dud login.
Connected to u629488.your-storagebox.de.
sftp> put /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0d68ad3e-5c39-4852-8e38-9c029dff2562/scratchpad/c10-canary.txt c10-canary.txt
Uploading /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0d68ad3e-5c39-4852-8e38-9c029dff2562/scratchpad/c10-canary.txt to /home/c10-canary.txt
sftp> ls -l
-rw-rw-r-- ? u629488-sub4 1059 56 Jul 31 22:16 c10-canary.txt
sftp> get c10-canary.txt /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0d68ad3e-5c39-4852-8e38-9c029dff2562/scratchpad/c10-canary-readback.txt
Fetching /home/c10-canary.txt to /tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0d68ad3e-5c39-4852-8e38-9c029dff2562/scratchpad/c10-canary-readback.txt
sftp> rm c10-canary.txt
Removing /home/c10-canary.txt
sftp> ls -l
readback content: campaign10 write-capability canary 2026-07-31T20:16:23Z
## VERDICT: write+read succeed in own home; every production path is outside the chroot
## ('No such file or directory') and the parent directory is an explicit 'Permission denied'.
documentation/tests/campaign10-evidence-2026-07-31/r156-papra-volume.txt
+20
View File
@@ -0,0 +1,20 @@
# R-156 evidence ��� papra persists OUTSIDE the volume the compose mounts
## 1. compose mounts papra_data at /app/data
volume papra_papra_data -> /app/data
## 2. the container runs as nonroot and CANNOT write there
uid=999(nonroot) gid=999(nonroot) groups=999(nonroot)
drwxr-xr-x 2 root root 4096 Jul 31 23:18 /app/data
touch: cannot touch '/app/data/.w': Permission denied
## 3. the volume is empty apart from the canary this campaign wrote AS ROOT
total 12
drwxr-xr-x 2 root root 4096 Jul 31 23:18 .
drwxr-xr-x 1 nonroot nonroot 4096 Jul 31 23:12 ..
-rw-r--r-- 1 root root 19 Jul 31 23:18 cc_proof.txt
## 4. the REAL database is in the container writable layer, not in any volume
total 472
drwxr-xr-x 1 nonroot nonroot 4096 Jul 31 23:12 .
drwxr-xr-x 1 nonroot nonroot 4096 Jul 4 22:19 ..
-rw-r--r-- 1 nonroot nonroot 475136 Jul 31 23:12 db.sqlite
## 5. healthcheck only probes the HTTP port, so this reads healthy
"healthy"
documentation/tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md
+156
View File
@@ -0,0 +1,156 @@
# Campaign 10 — Phase A journal (2026-07-31)
All times CEST unless marked Z.
## Venue
| | |
|---|---|
| Host | `demo-hp` (HP t740), Tier 0, the designated drill/build host |
| VM | **311 `c10-appliance`** — q35 / OVMF (`pre-enrolled-keys=0`), 4 cores, 8 GB, cpu=host |
| Disks | `scsi0` 200 G (system) · `scsi1` 50 G · `scsi2` 50 G — all qcow2 on `c10-scratch` |
| Storage | **`c10-scratch`**, `dir` at **`/mnt/nvme-1tb` — the mount ROOT**, `content=images`. Root chosen deliberately: a storage at a *subdirectory* sets `exactMount=false` → `reachable=false` → `StorageStateDisconnected` (`felhom-agent/internal/storage/observe.go:336`), which would emit `storage_disconnected` for demo-hp for the whole run — the exact signal I1/I2 test. Coexists with `felhom-backup` on the same path; that storage was **not modified**. |
| Guest network | demo-hp's vmbr0 is `192.168.0.87/24`, a **different physical LAN** from DooPlex's own 192.168.0.0/24. Reached via `ssh -J demo-hp`. |
| Console | no browser on DooPlex → `qm monitor screendump` → PPM → PNG, read visually; keystrokes via `sendkey` |
| Untouched | `drill-r50` (VM 300), guest 9201 on both demo boxes, `local-lvm`, both backup targets |
## Baselines — every value read fresh at the start of this run
| What | Value | How read |
|---|---|---|
| controller `main` | **v0.188.0** | `CHANGELOG.md` top; `HEAD`=`origin/main`=`4115e88`, tree clean |
| **golden's baked controller** | **0.188.0** (sha `7353d8be…`) | hub `/configuration`, *selected* option |
| agent published + vouched | **0.119.0** (sha `e37aca82…`) | hub `/configuration`, *selected* option |
| agent on both demo boxes | 0.119.0 | `felhom-agent --version` |
| agent `main` | v0.119.0 | `felhom-agent/CHANGELOG.md` |
| hub | **0.86.0** | live `deploy/hub` image tag == `hub/CHANGELOG.md` top |
| ISO | **1.26.1**, sha256 `f3cc86d5…`, 1 705 322 496 B | live round trip from `https://iso.felhom.eu/` |
| `felhom.eu` HEAD | `e9a74a0`, clean | `git rev-parse` |
| managed floor | v0.156.0 (DB override); env fallback v0.120.0 | hub `/configuration` |
**Golden vs `main`: NOT behind — both 0.188.0.** The deliberate decision the brief asks for is therefore
vacuous this run: the campaign tests what is shipping *and* what is current, with no divergence.
**Two brief assumptions were stale and were corrected by reading fresh:**
1. The brief was written against ISO **v1.25.0**; `iso.felhom.eu` began serving **1.26.1** at 18:37 Z
the same day — ~90 minutes before this run started.
2. The brief (and `CLAUDE.md`, `target-selection.md:41-42`) say demo-hp has **no baked SSH key**.
`ssh -o BatchMode=yes demo-hp` authenticated **by key, first try**. Already filed as **R-129** by the
2026-07-31 tester-gate run — **not re-filed** (register grep per the fences).
## A1 — build the appliance from the published ISO
**1.26.1 is a PUBLIC RELEASE image and cannot auto-install.** Verified against the ISO bytes on
demo-hp, not inferred from the manifest:
| Probe | Hits |
|---|---|
| `auto-installer-mode.toml` | **0** |
| `answer.toml` | **0** |
| `proxmox-auto-install` | **0** |
| `felhom-bootstrap_1.26.1_all.deb` | 1 |
| Felhom grub menuentries | exactly **2**, both interactive |
Manifest agrees: `mode: release … NO answer.toml, NO baked credential, interactive disk selection`,
`root-password: NONE — the installer prompts`, `loader: shim (stock MS-signed chain)`.
Driven blind through the **Terminal UI** entry (the graphical entry is not drivable — the QEMU monitor's
mouse events do not move the guest cursor, per `REPORT-iso-release.md`). Screendump before every
keystroke.
**Traps hit and handled, each verified rather than assumed:**
- **GRUB default is the graphical entry.** The first attempt auto-booted it because the `down` keystroke
landed after the 15 s countdown expired. Reset, and sent `down` inside the window in one round trip.
- **Guest keymap was Hungarian**, but `sendkey` emits US scancodes — every symbol afterwards (notably
`@`, which is AltGr+V on a Hungarian layout) would have been silently mangled, *including the root
password*, locking the box out. Switched to **U.S. English** before typing anything. Confirmed
positively: `c10@felhom.eu` rendered with a correct `@` and `.`.
- **Disk-selection safety:** only `scsi0` was attached during the install, so the installer offered
exactly one target (`/dev/sda`, 200 GiB) and could not reach a data drive. The two 50 G data drives
were attached afterwards.
- **Boot-order trap** (`REPORT-iso-release.md`, recorded twice there): `--boot` was set in its **own**
`qm set` after the disks existed, and verified from `qm config` (`boot: order=ide2;scsi0`). After the
install, the order was flipped to `order=scsi0` — again verified from `qm config`, not from the screen.
**Real completion signal** (not "an artifact changed"): 4.2 GB actually written to
`vm-311-disk-1.qcow2`, the installed system booted on its own, and the console showed the PVE banner at
`https://192.168.0.105:8006/`.
**Positive observables — all three present:**
| Observable | Evidence |
|---|---|
| package delivered | `ii felhom-bootstrap 1.26.1 all` on the box |
| unit fired at first boot | `systemctl is-enabled felhom-bootstrap` = `enabled`; `is-active` = `activating` |
| registers as an unclaimed appliance | hub `/hosts`: uuid `55ab3776-…` (matches VM smbios), pairing code **FD6-A6S**, MAC `bc:24:11:46:25:fe`, 3 SSH host keys, hardware read |
**A1 — PASS.**
## A2 — claim
Customer created: **`c10-soak`** — "Campaign 10 two-storage soak (SCRATCH - delete at teardown)",
domain `c10.felhom.eu`, email `nagyfenyvesi.viktor@gmail.com`. Deliberately **not** `drill-r50`,
`sess-*`, or any existing record.
Appliance 19 bound to it. Hub log, verbatim:
22:50:03 appliance 19 BOUND to customer c10-soak (mode=appliance) — delivery staged for its next poll
22:50:12 appliance credentials DELIVERED once to appliance 19 (passphrase withheld)
22:50:19 [claim] claim code (gen 1) emailed to the registered address of c10-soak
22:50:49 host enrolled: c10-soak-8a405e (customer c10-soak)
22:50:49 vaulted break-glass recovery credential for host c10-soak-8a405e (user=root@pam, secret 32 chars)
22:50:51 Artifact manifest served for customer c10-soak (agent=0.119.0 golden=0.188.0)
22:51:13 wg registered: host=c10-soak-8a405e ip=10.77.0.6/32 gen=1 sync=ok
22:51:14 host-report from c10-soak-8a405e (0 guests, 2 storage targets, ...)
22:51:14 DR-recipe host-half stored for customer c10-soak (host c10-soak-8a405e, v1)
**The installer's root password stops working at 22:50:49** — day-0 replaces it with a hub-vaulted
managed credential (G1 `host_recovery`). Retrieved via `POST /hosts/c10-soak-8a405e/reveal-recovery-credential`
→ `root@pam`, 32 chars, stored file→file at `~/.config/campaign10/host-recovery.json` (0600), never printed.
Box state after day-0:
felhom-agent 0.119.0 (== published+vouched)
/etc/felhom-controller-image : …/felhom-controller:0.188.0 (== golden == main)
guest 9201 c10-soak running
docker: controller 0.188.0 (healthy), traefik v3.6.7, filebrowser 1.3.3-stable (healthy)
block devices: sda 200G, sdb 50G, sdc 50G
Claim gate, the positive discriminator, read over the path the UI uses (traefik on 443, `Host:` header
mandatory — the controller does **not** listen on 8080; 80 is a 301 to HTTPS):
GET https://127.0.0.1/api/apps Host: felhom.c10.felhom.eu
→ {"ok":false,"error":"dashboard not yet claimed"}
**A2 — BLOCKED on the emailed claim code (R-119, confirmed live again).** The code exists only in the
customer's mailbox; there is no operator-side route.
## Scope constraint found in Phase A — Tier 3 cannot be isolated, so it will not run
The brief's §2 requires the campaign's PBS to be *a dedicated datastore and namespace on DooPlex*. That
is not reachable, for two reasons that are each **already-recorded deliberate positions**, not new defects:
1. **Offsite hard-requires the DR tier.** `hub/internal/web/configs.go:1300` refuses offsite without it:
*"Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS key)"*.
Closed by policy 2026-07-12 (drill F-6, DR-tier-by-default decisions 3/4).
2. **The DR tier only ever provisions on ep0.** Peer allocation and endpoint sync use the lowest
`endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).
So the only two configurations are: *DR tier on* → the campaign's PBS traffic lands on **ep0**, which is
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90) and fenced by §3 — or *DR tier off* →
no Tier 3 at all. **Chosen: DR tier OFF, offsite OFF**, which is the only option §3 permits.
Consequence, stated plainly rather than discovered later: the campaign touches **neither ep0 nor the
Storage Box at any point** — a stronger isolation than the brief asked for, obtained by not running the
tier. What is lost: all Tier-3 operations in B1, **I8** entirely, and the Tier-3 rows of the RTO/RPO
table. Everything else — the two-drive core (R-112/R-113/R-114/R-116/R-117), D5, I1–I7 and I9–I11 — is
unaffected, because those live on the drives and the local/whole-guest tiers.
**No `R-n` minted**: per the fences, the register was grepped first and both halves are already recorded
(F-6 policy close; the `hub/README.md:260` deferral). This is a campaign constraint, not a finding.
## Isolation gate — §2
See `../isolation/pbs-denial.txt` and `../isolation/restic-denial.txt`. Summary in the main audit.