diff --git a/REPORT.md b/REPORT.md index 1d47363..4695bf8 100644 --- a/REPORT.md +++ b/REPORT.md @@ -4,55 +4,25 @@ --- -# REPORT — Impl-1 Part B: install the guarded-mkfs wrapper (host-install v1.8.0) +# REPORT — "no-mercy" campaign: full effects of the `felhom` pool + scoped-token migration -**Date:** 2026-07-01 · **Repo:** `felhom.eu` (`scripts/` only) · **Class:** Risky/supervised (paired with -the destructive-path change in felhom-agent v0.54.0). +**Date:** 2026-07-02 · **Class:** Runbook-style validation (aggressive, throwaway env) · **No code change, no version bump.** +**Findings doc:** [`documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md`](documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md) -Companion to felhom-agent v0.54.0 (the format-safety foundation). Script-only change: host-install now -installs the guarded-mkfs wrapper during agent install so every fresh box gets the safe format path. +Executed the full pool-effects campaign on felhom-pve (agent v0.59.0 live, host-install v1.7.0 scoped ACL): source-traced every `ListLXC` consumer in felhom-agent (read-only), then ran the complete agent→PVE operation matrix under the scoped token on scratch pool guests 9310/9311, adversarial pool-blindness probes with non-pool decoys, blast-radius regression re-confirms, and pool-membership side-effect checks. -## 1. Baseline → target +## Headline results -script `v1.7.0` @ `9e5bbc1` → **v1.8.0**. Paired with agent v0.54.0 (separate repo, `5209830`). +1. **Track A — 16/16 ops PASS, zero surprise 403s.** The scoped token is functionally complete for the agent's real operation surface (incl. the previously untested Destroy+disk-reap, Snapshot/Rollback/Delete, SetConfig Network/CPU/Options, Resize, ListRunningTasks, TaskLogTail, and the full restore-test cycle). +2. **B1 (the campaign's most important result):** the provision vmid is **always externally supplied** (no `nextid` scan in agent OR hub — collision-avoidance is operator-side). Adversarially provisioning onto an existing **non-pool** vmid: the pool-blind duplicate guard misses it, PVE refuses the restore (no clobber), **but the compensating rollback then attempts to DESTROY the innocent guest** — stopped only by the pool ACL (403). Same shape live-proven for the restore-test scratch band (a squatter at 990000 additionally makes the restore-test **permanently FAIL** with false "backup unrestorable" alerts). Safe-but-noisy under the scoped ACL; a genuine destroy-risk under any broad token. +3. **Containment intact (Track C):** non-pool guest ops 403, non-granted-storage writes 403, drive audit readable (3b-fix holds). +4. **Hub inventory is pool-only (B3, by design — now recorded as a decision):** operators must pick vmids from the host's real guest list, never the hub view. No pool-name leak into any customer/controller-facing field (Track D); `--adopt-pool` idempotent; uninstall never deletes a non-empty pool. -## 2. What changed +## Fix list (full detail + severities in the findings doc) -- `scripts/felhom-host-install.sh` `step_agent_install`: new step (before the sudoers install) that - `fetch_raw`s `configs/felhom-mkfs-guarded.sh` from Gitea, `bash -n`-validates it, and - `install -m0755 -o root -g root` → `/usr/local/sbin/felhom-mkfs-guarded`. Wrapper-before-sudoers so - the sudoers (which allowlists only the wrapper) is never installed ahead of the file it names. -- The agent v0.54.0 sudoers (fetched by the same step) drops raw `mkfs.*` (allowlists only the wrapper) - and adds read-only `pvs`/`zpool status -P` for the agent's unclaimed-disk guard. -- `scripts/CHANGELOG.md`, version banner. Commit **`7ea36bc`** on `main`. -- `bash -n` + `shellcheck` clean (0 new warnings; the 2 pre-existing SC2015 in `step_verify` unchanged). +- **F1 (HIGH, blocks shared-box trial):** arm the bring-up/restore-test compensating destroy only after `RestoreLXC` returns a UPID — never destroy what the job didn't create. +- **F2 (MED, blocks trial):** restore-test should advance past an "already exists" band vmid (or pre-verify via fenced root CLI); document 990000–990009 as reserved fleet-wide. +- **F3 (MED, doc-only, blocks trial):** provisioning runbook must say vmid selection uses the host's full guest list, not the hub inventory. +- F4/F5/F6 (LOW/INFO): non-pool source-config read in restore-test, pool-blind eject dependent-scan, latent reconcile desired-state rule. -## 3. Served-copy version - -`curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh | grep SCRIPT_VERSION` → **`1.8.0`**. - -## 4. Live validation (felhom-pve, agent v0.54.0 deploy) - -The two-layer format guard (agent unclaimed-filter + this wrapper) validated on the demo — details in -`felhom-agent/REPORT.md` §5: -- wrapper REFUSES the OS disk `/dev/sda` and the LVM-PV partition `/dev/sda3`; -- raw `sudo mkfs.ext4 /dev/sda` is DENIED (Part B — wildcard removed); -- an unclaimed throwaway `/dev/sdd` FORMATS (the only real format); -- the agent guard's sudo reads (`pvs`/`lsblk`/`zpool`) all work as the felhom-agent user. - -**Deploy note:** installing the sudoers via a manual scp of the Windows working-copy failed `visudo` -(CRLF line endings); fixed live by CR-stripping. The committed repo file is LF, and the host-install -`fetch_raw`-from-Gitea path serves LF — so the shipped path is unaffected. (Lesson recorded.) - -## 5. NOT yet live-validated / deferred - -- A **fresh install** exercising the new `step_agent_install` wrapper-install step end-to-end (the demo - was repaired in place; a clean box will exercise the fetch+install path — the code is dry-run-safe and - mirrors the existing sudoers/unit install). -- **Impl-2** (raw-device discovery + registry enrollment + wizard rewiring) and **Impl-3** (shared-box - operator gate for self-serve format) are the separate follow-up specs. - -## 6. Observations - -- The guarded-mkfs wrapper is `configs/felhom-mkfs-guarded.sh` in the **felhom-agent** repo (co-located - with the sudoers it pairs with); host-install fetches it the same way it fetches the unit/sudoers. +Demo left exactly as found: pool `felhom` = {9201}, all scratch/decoy guests and campaign artifacts removed, agent active, controller healthy. diff --git a/documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md b/documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md new file mode 100644 index 0000000..0280b81 --- /dev/null +++ b/documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md @@ -0,0 +1,119 @@ +# CAMPAIGN — full effects of the `felhom` pool + scoped-token migration + +**Date run:** 2026-07-02 (runbook authored 2026-07-01) · **Host:** felhom-pve (`demo-felhom`, PVE 9.2.2) +**Agent live:** v0.59.0 (runbook table said v0.55.0 — v0.56–0.59 shipped between authoring and execution; the ListLXC/priv surface is unchanged) +**ACL:** host-install v1.7.0 3-role scoped layout, verified live: `FelhomAgentGuest` @`/pool/felhom` (12 privs incl. `Pool.Allocate`), `FelhomAgentStore` (`Datastore.Allocate,AllocateSpace`) @`local`/`local-lvm`/`felhom-pbs`, `FelhomAgentBase` (`Sys.Audit,SDN.Use,Datastore.Audit`) @`/`. +**Method:** the agent's own code paths where they exist (`--selftest=bring-up|task|restore-test|hub`, run as the `felhom-agent` user), raw PVE-API probes **as the scoped token** for the rest (identical paths/params to `internal/proxmox/mutate.go`). Mutations on scratch pool guests 9310/9311 + throwaway decoys; 9201 got read/non-destructive probes only. **No code changed; demo left as found** (pool = {9201}, agent active, controller healthy, all scratch/decoys destroyed, 9310's PBS archive pruned). + +--- + +## 1. Track A — operation-matrix sweep: **16/16 PASS, zero surprise 403s** + +The scoped token is **functionally COMPLETE** for the agent's real operation surface. + +| # | Op | How exercised | Result | +|---|----|---------------|--------| +| 1 | RestoreLXC → pool | bring-up selftest (golden → 9310, 52s boot+running); PBS archive → 9311 raw probe | **PASS** (both local + PBS archive paths) | +| 2 | Vzdump → felhom-pbs | raw probe, mode=snapshot, on 9310 | **PASS** (task OK, 100s) | +| 3 | DestroyLXC + `destroy-unreferenced-disks=1` | raw probe on 9310+9311 | **PASS** — all 6 LVM volumes verifiably reaped | +| 4 | Snapshot | `--selftest=task -vmid 9310` | **PASS** | +| 5 | Rollback | same | **PASS** | +| 6 | DeleteSnapshot | same | **PASS** | +| 7 | SetConfig Disk/Network/CPU/Memory/Options | bring-up (Disk/mounts) + raw probes: `net1` add+delete, `cores`, `memory`, `onboot`, description (selftest) | **PASS** all five key classes | +| 8 | ResizeLXC | rootfs +1G raw probe (32G→33G, task OK) | **PASS** | +| 9 | Start / Stop | bring-up start; stop probe pre-destroy | **PASS** | +| 10 | ListLXC | probe | **PASS** — but returns ONLY pool guests (Track B) | +| 11 | GuestStatus/Config/ListSnapshots | selftests + probes | **PASS** (pool guests) | +| 12 | Nodes/NodeStatus/Version | `--selftest=hub` collect | **PASS** | +| 13 | ListRunningTasks | probed DURING a live vzdump | **PASS** — task visible mid-flight | +| 14 | ListStorage/NodeStorage/StorageContent/LatestBackupVolID | probes incl. felhom-pbs content enumeration | **PASS** (3b-fix intact) | +| 15 | TaskStatusOnce/WaitTask | used throughout (every async op) | **PASS** | +| 16 | TaskLogTail | full log of completed vzdump read | **PASS** | + +Full restore-test cycle (restore→link-down→boot→verify→teardown) also **PASS** end-to-end under the scoped token (20s), including teardown of its own pool-created scratch. + +Transient (not a finding): concurrent SetConfig during a running resize task → HTTP 500 "can't lock file pve-config-9310.lock" — ordinary PVE per-guest config locking, retry succeeds. + +--- + +## 2. Track B — pool-blindness (the depth) + +Mechanism proven live first: the scoped token's `GET /nodes/demo-felhom/lxc` returns **only {9201}** while the host holds 9000/9001/9100/9999/9201. Every ListLXC consumer inherits this view. + +### B1 — vmid origin + collision verdict ⭐ the campaign's most important result + +**Origin (source-traced, definitive):** the provision vmid is **always externally supplied** — both `BringUpSpec` call sites (`--selftest=bring-up`, `--selftest=provision`, `cmd/felhom-agent/main.go:1233/1388`) take `-vmid` as an explicit CLI flag. No `nextid`/next-free scan exists anywhere in the agent **or the hub** (grep-verified both repos). The **only** agent-computed vmid is the restore-test scratch (`pickScratchVMID` over ListLXC, band 990000–990009, `restoretest.go:419`). + +**Collision-avoidance therefore sits with the operator/hub — and the operator's hub inventory is now pool-only (B3), so on a shared box the operator can pick a vmid occupied by a guest the hub can't see.** What happens then: + +**Adversarial run 1 — bring-up onto existing non-pool 9001:** +1. The bring-up duplicate-guard (`bringup.go:180`, pool-blind ListLXC) **MISSED** the existing guest and proceeded. +2. PVE refused the restore: HTTP 500 "CT 9001 already exists" (agent never sets `force`) → **no clobber**. +3. **The compensating rollback then attempted `DestroyLXC(9001)` — a guest the job never created** (gate classed it benign `guest_destroy`, SameTxnCreated provenance). Only the pool ACL stopped it: HTTP 403 "missing privilege VM.Allocate at /vms/9001". **Under the pre-3b broad token this would have DESTROYED the innocent guest.** +4. The dangling in-flight Rollback journal entry self-resolved on the next Recover: pool-blind ListLXC can't see 9001 → "guest already gone" → terminal-clean. (Coincidentally correct: containment guarantees the agent can't have created/kept anything outside the pool.) + +**Adversarial run 2 — restore-test with a non-pool decoy parked at 990000 (band start):** +1. `pickScratchVMID` (pool-blind) picked the **occupied** 990000. +2. Restore refused ("CT 990000 already exists") → no clobber. +3. **The deferred teardown attempted `DestroyLXC(990000)` on the innocent decoy** → 403, decoy survived. +4. Journal entry self-resolved clean on next Recover (same pool-blind coincidence). +5. Result surfaced as **FAIL** (`pass:false`, "already exists") — **not** Skipped — i.e. a false "backup not restorable" signal to the hub. +6. Because the picker always takes the **lowest** free-looking vmid and the squatter never becomes visible, it re-picks 990000 **every run**: one non-pool squatter at the band start = restore-test **permanently broken** (it never advances to 990001). + +**Verdict:** no collision can clobber or destroy a foreign guest **as long as the pool-scoped ACL is in place** — but that is a *single* containment layer over an agent-logic flaw (destroy-what-you-didn't-create), and the failure mode is noisy-but-safe (provision fails, ERROR logs, false restore-test alerts). See fixes F1/F2/F3. + +Bonus finding (same run): with a **non-pool source guest** (the golden 9100), the restore-test cannot read the source config for bind-mount neutralization (403 VM.Audit) → logs "restoring as-is". Harmless for the golden (no bind mounts), but a bind-mount-bearing non-pool source archive would fail the restore-test instead of being neutralized (F4). + +### B2 — eject dependent-guest scan (`disks.go:1026 dependentGuests`) + +**Pool-blind at TWO layers, proven live without a disruptive eject:** (a) the scan enumerates via ListLXC → non-pool guests never enter the loop (list = {9201} proven); (b) even if they did, `GuestConfig` on a non-pool guest 403s (observed live on 9100). **On a shared box, ejecting a drive a non-pool guest bind-mounts would be under-warned.** Severity: low today (single-tenant demo; Felhom drives are enrolled to Felhom guests), real for the shared-box trial. Fix note in F5. + +### B3 — hub inventory (`collect.go:204`) + +Confirmed live via `--selftest=hub`: the host report's `guests` = **only 9201**; the four non-pool guests are invisible to the hub/operator. **Decision recorded: intended post-containment** — the agent manages (and reports) only its pool. The consequence that must be documented (F3): the hub inventory is NOT a full-host view, so **operator vmid selection for a new provision must consult the host's real guest list (PVE UI/root), never the hub inventory**, on any box that has non-Felhom guests. + +### B4 — remaining ListLXC consumers + +| Consumer | Behavior with a pool-only list | Verdict | +|---|---|---| +| stale-lock guard (`stalelock.go`) | Guest enumeration pool-only → recovers only pool guests (intended: non-pool guests aren't the agent's to unlock). Its safety input `ListRunningTasks` is **NOT pool-blind** — proven live: a **root-initiated vzdump of non-pool 9001 was visible** to the token mid-flight (Sys.Audit@/ covers the task list). No misfire vector found. | ✅ correct | +| reconcile `engine.readActual` (`engine.go:229`) | Actual-state is pool-only. Today EmptyProvider → no effect. **Latent rule:** desired-state must never reference a non-pool guest, or reconcile would treat it as absent and re-provision-over (which then hits the B1 chain, contained). | ✅ today; rule recorded | +| `recover.go` (scratch + bring-up reapers) | Existence check pool-only → an out-of-pool collision vmid resolves "already gone" (terminal-clean). Coincidentally correct under containment (see B1.4); would be WRONG (silent leak of a half-built guest) only if the agent could create outside the pool — it can't (restore without `pool=felhom` 403s). | ✅ under scoped ACL | +| `--selftest=read` list check (`main.go:1753`) | Cosmetic (prints pool guests only). | ✅ | + +--- + +## 3. Track C — blast-radius regression: **containment intact** + +| Probe | Result | +|---|---| +| Stop non-pool 9001 as token | **403** "Permission check failed (/vms/9001, VM.PowerMgmt)" | +| Vzdump 9201 → non-granted `felhom-usb` | **403** "(/storage/felhom-usb, Datastore.AllocateSpace)" — refused at POST, nothing silently retargeted, no task started | +| Drive-storage audit (felhom-usb/felhom-flash) | **readable** (status + list, HTTP 200) — 3b-fix (`Datastore.Audit`@/ in Base) intact, no false detach | + +--- + +## 4. Track D — pool-membership non-ACL effects: **no surprises** + +- **No leak:** the pool name appears in NO customer/controller-facing surface — hub host-report guest entries carry `vmid/name/status/controller_version/spec` only (verified live); `pct config 9201` has no pool field (pool membership lives in PVE `user.cfg`); the guest-visible bootstrap mount is `bootstrap.json` only. +- **`--adopt-pool`** (v1.8.0, live run): idempotent — "pool felhom already exists" + "guest 9201 already in pool felhom", guest not otherwise touched. +- **`--uninstall --dry-run`** (live): removes the `/pool/felhom` ACL grants and correctly **refuses to delete a non-empty pool** ("pool felhom not empty (members: 9201) — leaving it"). In a real uninstall the destroyed guest auto-leaves the pool first; the guard protects foreign members. + +--- + +## 5. Prioritized fix list + +| # | Sev | Finding | Recommended fix | Blocks shared-box trial? | +|---|-----|---------|-----------------|--------------------------| +| **F1** | **HIGH** (logic; contained by ACL) | Bring-up rollback + restore-test teardown fire `DestroyLXC` on a vmid the job **never created** (create failed "already exists"). One ACL layer away from destroying a foreign guest; any future broad-token/`root` deployment re-arms it. | Arm the compensating destroy **only after `RestoreLXC` returns a UPID** (nothing created ⇒ nothing to roll back). A POST *network* error (outcome unknown) may keep today's behavior. Applies to `bringup.go` defer + `restoretest.go` defer (+ their Recover twins are already exists-checked). | **YES** — fix before the trial (defense-in-depth; the trial box has foreign guests by definition) | +| **F2** | **MED** | One non-pool squatter at the scratch-band start ⇒ restore-test **permanently FAILs** (always re-picks the lowest invisible-occupied vmid; reports FAIL not Skip → false "backup unrestorable" alerts). | On an "already exists" restore failure, advance to the next band vmid (or pre-verify emptiness via the fenced root `pct list`); band exhausted ⇒ Skip (existing semantics). Additionally: document 990000–990009 as **reserved fleet-wide** on any Felhom host. | YES (cheap; false alerts would drown the trial's signal) | +| **F3** | **MED** (process/doc) | Hub inventory is pool-only (by design) while vmid assignment is operator-side ⇒ on a shared box the operator can pick an occupied, hub-invisible vmid. Outcome is safe-but-noisy (B1 chain). | Document in the provisioning runbook: **pick vmids from the host's full guest list (PVE UI / root `pct list`+`qm list`), never from the hub inventory.** Longer-term: Day-0/hub could record a host-declared reserved-vmid range. | YES (doc-only) | +| **F4** | LOW | Restore-test can't read a **non-pool source guest's** config for bind-mount neutralization (403) → "restoring as-is"; a bind-mount-bearing non-pool source archive would false-fail the test. | Accept + log (current behavior is fail-safe), or route the source-config read through the fenced root CLI. Today's sources (9201, golden) are unaffected. | no | +| **F5** | LOW | Eject `dependentGuests` scan is double-pool-blind (list + config 403) → non-pool dependents of an ejected drive are not warned about. | Either declare pool-scoped-by-design in the eject UX copy ("Felhom guests using this drive: …"), or scan via the fenced root CLI for the shared-box case. | no (drives are Felhom-enrolled; revisit if drives are ever shared) | +| **F6** | INFO | Stale-lock guard, reconcile readActual, recover reapers: all correct under a pool-only view (details §2 B4); the reconcile desired-state must never reference a non-pool vmid (latent rule, currently unfed). | Record the rule next to `DesiredProvider` when slice-10 feeds it. | no | + +**Pre-existing, noted in passing (NOT pool effects):** non-root agent can't read `/etc/pve/priv/storage/felhom-pbs.pw` (known OPEN from the Day-0 bundle — PBS datastore detail skipped in reports); the drive-enrollment candidacy bug has its own pending fix spec (explicitly out of scope per the runbook). + +## 6. Not changed + +Validation only — **no code, no version bump**. felhom-agent untouched (read-only source trace). The demo host was left exactly as found: guests {9000, 9001, 9100, 9201, 9999}, pool `felhom` = {9201}, agent v0.59.0 active, controller healthy, campaign scratch guests/decoys/PBS archive/probe artifacts all removed.