felhom.eu/documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md

# CAMPAIGN — full effects of the `felhom` pool + scoped-token migration

> **Resolution update (2026-07-02, agent v0.60.0):** **F1 FIXED at root** — proof-of-launch gating: a
> compensating destroy (bring-up rollback, restore-test teardown, AND Recover) fires only when
> `RestoreLXC` returned a UPID; no UPID ⇒ the transaction created nothing ⇒ never destroy. Recover is
> now safe by DESIGN (a no-UPID Scratch/Rollback entry is abandoned fail-safe), no longer by the
> pool-blind "already gone" accident (§2 B1.4). **F2 FIXED** — the restore-test advances past an
> "already exists" band vmid (bounded by the band width); a fully-occupied band → Skipped, no false
> alert. **F3 DOCUMENTED** — `documentation/runbooks/provisioning.md`. Red-proof-tested + live-proven
> on felhom-pve (provision onto existing 9001 → no destroy armed; restore-test advanced past a 990000
> decoy and passed at 990001). F4/F5/F6 remain open as recorded. Operator follow-up: bump the hub
> Day-0 artifact manifest to agent **0.60.0** (binary published to Gitea, sha256
> `45d19b87…51725831`).

**Date run:** 2026-07-02 (runbook authored 2026-07-01) · **Host:** felhom-pve (`demo-felhom`, PVE 9.2.2)
**Agent live:** v0.59.0 (runbook table said v0.55.0 — v0.56–0.59 shipped between authoring and execution; the ListLXC/priv surface is unchanged)
**ACL:** host-install v1.7.0 3-role scoped layout, verified live: `FelhomAgentGuest` @`/pool/felhom` (12 privs incl. `Pool.Allocate`), `FelhomAgentStore` (`Datastore.Allocate,AllocateSpace`) @`local`/`local-lvm`/`felhom-pbs`, `FelhomAgentBase` (`Sys.Audit,SDN.Use,Datastore.Audit`) @`/`.
**Method:** the agent's own code paths where they exist (`--selftest=bring-up|task|restore-test|hub`, run as the `felhom-agent` user), raw PVE-API probes **as the scoped token** for the rest (identical paths/params to `internal/proxmox/mutate.go`). Mutations on scratch pool guests 9310/9311 + throwaway decoys; 9201 got read/non-destructive probes only. **No code changed; demo left as found** (pool = {9201}, agent active, controller healthy, all scratch/decoys destroyed, 9310's PBS archive pruned).

---

## 1. Track A — operation-matrix sweep: **16/16 PASS, zero surprise 403s**

The scoped token is **functionally COMPLETE** for the agent's real operation surface.

| # | Op | How exercised | Result |
|---|----|---------------|--------|
| 1 | RestoreLXC → pool | bring-up selftest (golden → 9310, 52s boot+running); PBS archive → 9311 raw probe | **PASS** (both local + PBS archive paths) |
| 2 | Vzdump → felhom-pbs | raw probe, mode=snapshot, on 9310 | **PASS** (task OK, 100s) |
| 3 | DestroyLXC + `destroy-unreferenced-disks=1` | raw probe on 9310+9311 | **PASS** — all 6 LVM volumes verifiably reaped |
| 4 | Snapshot | `--selftest=task -vmid 9310` | **PASS** |
| 5 | Rollback | same | **PASS** |
| 6 | DeleteSnapshot | same | **PASS** |
| 7 | SetConfig Disk/Network/CPU/Memory/Options | bring-up (Disk/mounts) + raw probes: `net1` add+delete, `cores`, `memory`, `onboot`, description (selftest) | **PASS** all five key classes |
| 8 | ResizeLXC | rootfs +1G raw probe (32G→33G, task OK) | **PASS** |
| 9 | Start / Stop | bring-up start; stop probe pre-destroy | **PASS** |
| 10 | ListLXC | probe | **PASS** — but returns ONLY pool guests (Track B) |
| 11 | GuestStatus/Config/ListSnapshots | selftests + probes | **PASS** (pool guests) |
| 12 | Nodes/NodeStatus/Version | `--selftest=hub` collect | **PASS** |
| 13 | ListRunningTasks | probed DURING a live vzdump | **PASS** — task visible mid-flight |
| 14 | ListStorage/NodeStorage/StorageContent/LatestBackupVolID | probes incl. felhom-pbs content enumeration | **PASS** (3b-fix intact) |
| 15 | TaskStatusOnce/WaitTask | used throughout (every async op) | **PASS** |
| 16 | TaskLogTail | full log of completed vzdump read | **PASS** |

Full restore-test cycle (restore→link-down→boot→verify→teardown) also **PASS** end-to-end under the scoped token (20s), including teardown of its own pool-created scratch.

Transient (not a finding): concurrent SetConfig during a running resize task → HTTP 500 "can't lock file pve-config-9310.lock" — ordinary PVE per-guest config locking, retry succeeds.

---

## 2. Track B — pool-blindness (the depth)

Mechanism proven live first: the scoped token's `GET /nodes/demo-felhom/lxc` returns **only {9201}** while the host holds 9000/9001/9100/9999/9201. Every ListLXC consumer inherits this view.

### B1 — vmid origin + collision verdict ⭐ the campaign's most important result

**Origin (source-traced, definitive):** the provision vmid is **always externally supplied** — both `BringUpSpec` call sites (`--selftest=bring-up`, `--selftest=provision`, `cmd/felhom-agent/main.go:1233/1388`) take `-vmid` as an explicit CLI flag. No `nextid`/next-free scan exists anywhere in the agent **or the hub** (grep-verified both repos). The **only** agent-computed vmid is the restore-test scratch (`pickScratchVMID` over ListLXC, band 990000–990009, `restoretest.go:419`).

**Collision-avoidance therefore sits with the operator/hub — and the operator's hub inventory is now pool-only (B3), so on a shared box the operator can pick a vmid occupied by a guest the hub can't see.** What happens then:

**Adversarial run 1 — bring-up onto existing non-pool 9001:**
1. The bring-up duplicate-guard (`bringup.go:180`, pool-blind ListLXC) **MISSED** the existing guest and proceeded.
2. PVE refused the restore: HTTP 500 "CT 9001 already exists" (agent never sets `force`) → **no clobber**.
3. **The compensating rollback then attempted `DestroyLXC(9001)` — a guest the job never created** (gate classed it benign `guest_destroy`, SameTxnCreated provenance). Only the pool ACL stopped it: HTTP 403 "missing privilege VM.Allocate at /vms/9001". **Under the pre-3b broad token this would have DESTROYED the innocent guest.**
4. The dangling in-flight Rollback journal entry self-resolved on the next Recover: pool-blind ListLXC can't see 9001 → "guest already gone" → terminal-clean. (Coincidentally correct: containment guarantees the agent can't have created/kept anything outside the pool.)

**Adversarial run 2 — restore-test with a non-pool decoy parked at 990000 (band start):**
1. `pickScratchVMID` (pool-blind) picked the **occupied** 990000.
2. Restore refused ("CT 990000 already exists") → no clobber.
3. **The deferred teardown attempted `DestroyLXC(990000)` on the innocent decoy** → 403, decoy survived.
4. Journal entry self-resolved clean on next Recover (same pool-blind coincidence).
5. Result surfaced as **FAIL** (`pass:false`, "already exists") — **not** Skipped — i.e. a false "backup not restorable" signal to the hub.
6. Because the picker always takes the **lowest** free-looking vmid and the squatter never becomes visible, it re-picks 990000 **every run**: one non-pool squatter at the band start = restore-test **permanently broken** (it never advances to 990001).

**Verdict:** no collision can clobber or destroy a foreign guest **as long as the pool-scoped ACL is in place** — but that is a *single* containment layer over an agent-logic flaw (destroy-what-you-didn't-create), and the failure mode is noisy-but-safe (provision fails, ERROR logs, false restore-test alerts). See fixes F1/F2/F3.

Bonus finding (same run): with a **non-pool source guest** (the golden 9100), the restore-test cannot read the source config for bind-mount neutralization (403 VM.Audit) → logs "restoring as-is". Harmless for the golden (no bind mounts), but a bind-mount-bearing non-pool source archive would fail the restore-test instead of being neutralized (F4).

### B2 — eject dependent-guest scan (`disks.go:1026 dependentGuests`)

**Pool-blind at TWO layers, proven live without a disruptive eject:** (a) the scan enumerates via ListLXC → non-pool guests never enter the loop (list = {9201} proven); (b) even if they did, `GuestConfig` on a non-pool guest 403s (observed live on 9100). **On a shared box, ejecting a drive a non-pool guest bind-mounts would be under-warned.** Severity: low today (single-tenant demo; Felhom drives are enrolled to Felhom guests), real for the shared-box trial. Fix note in F5.

### B3 — hub inventory (`collect.go:204`)

Confirmed live via `--selftest=hub`: the host report's `guests` = **only 9201**; the four non-pool guests are invisible to the hub/operator. **Decision recorded: intended post-containment** — the agent manages (and reports) only its pool. The consequence that must be documented (F3): the hub inventory is NOT a full-host view, so **operator vmid selection for a new provision must consult the host's real guest list (PVE UI/root), never the hub inventory**, on any box that has non-Felhom guests.

### B4 — remaining ListLXC consumers

| Consumer | Behavior with a pool-only list | Verdict |
|---|---|---|
| stale-lock guard (`stalelock.go`) | Guest enumeration pool-only → recovers only pool guests (intended: non-pool guests aren't the agent's to unlock). Its safety input `ListRunningTasks` is **NOT pool-blind** — proven live: a **root-initiated vzdump of non-pool 9001 was visible** to the token mid-flight (Sys.Audit@/ covers the task list). No misfire vector found. | ✅ correct |
| reconcile `engine.readActual` (`engine.go:229`) | Actual-state is pool-only. Today EmptyProvider → no effect. **Latent rule:** desired-state must never reference a non-pool guest, or reconcile would treat it as absent and re-provision-over (which then hits the B1 chain, contained). | ✅ today; rule recorded |
| `recover.go` (scratch + bring-up reapers) | Existence check pool-only → an out-of-pool collision vmid resolves "already gone" (terminal-clean). Coincidentally correct under containment (see B1.4); would be WRONG (silent leak of a half-built guest) only if the agent could create outside the pool — it can't (restore without `pool=felhom` 403s). | ✅ under scoped ACL |
| `--selftest=read` list check (`main.go:1753`) | Cosmetic (prints pool guests only). | ✅ |

---

## 3. Track C — blast-radius regression: **containment intact**

| Probe | Result |
|---|---|
| Stop non-pool 9001 as token | **403** "Permission check failed (/vms/9001, VM.PowerMgmt)" |
| Vzdump 9201 → non-granted `felhom-usb` | **403** "(/storage/felhom-usb, Datastore.AllocateSpace)" — refused at POST, nothing silently retargeted, no task started |
| Drive-storage audit (felhom-usb/felhom-flash) | **readable** (status + list, HTTP 200) — 3b-fix (`Datastore.Audit`@/ in Base) intact, no false detach |

---

## 4. Track D — pool-membership non-ACL effects: **no surprises**

- **No leak:** the pool name appears in NO customer/controller-facing surface — hub host-report guest entries carry `vmid/name/status/controller_version/spec` only (verified live); `pct config 9201` has no pool field (pool membership lives in PVE `user.cfg`); the guest-visible bootstrap mount is `bootstrap.json` only.
- **`--adopt-pool`** (v1.8.0, live run): idempotent — "pool felhom already exists" + "guest 9201 already in pool felhom", guest not otherwise touched.
- **`--uninstall --dry-run`** (live): removes the `/pool/felhom` ACL grants and correctly **refuses to delete a non-empty pool** ("pool felhom not empty (members: 9201) — leaving it"). In a real uninstall the destroyed guest auto-leaves the pool first; the guard protects foreign members.

---

## 5. Prioritized fix list

| # | Sev | Finding | Recommended fix | Blocks shared-box trial? |
|---|-----|---------|-----------------|--------------------------|
| **F1** | **HIGH** (logic; contained by ACL) | Bring-up rollback + restore-test teardown fire `DestroyLXC` on a vmid the job **never created** (create failed "already exists"). One ACL layer away from destroying a foreign guest; any future broad-token/`root` deployment re-arms it. | Arm the compensating destroy **only after `RestoreLXC` returns a UPID** (nothing created ⇒ nothing to roll back). A POST *network* error (outcome unknown) may keep today's behavior. Applies to `bringup.go` defer + `restoretest.go` defer (+ their Recover twins are already exists-checked). | **YES** — fix before the trial (defense-in-depth; the trial box has foreign guests by definition) |
| **F2** | **MED** | One non-pool squatter at the scratch-band start ⇒ restore-test **permanently FAILs** (always re-picks the lowest invisible-occupied vmid; reports FAIL not Skip → false "backup unrestorable" alerts). | On an "already exists" restore failure, advance to the next band vmid (or pre-verify emptiness via the fenced root `pct list`); band exhausted ⇒ Skip (existing semantics). Additionally: document 990000–990009 as **reserved fleet-wide** on any Felhom host. | YES (cheap; false alerts would drown the trial's signal) |
| **F3** | **MED** (process/doc) | Hub inventory is pool-only (by design) while vmid assignment is operator-side ⇒ on a shared box the operator can pick an occupied, hub-invisible vmid. Outcome is safe-but-noisy (B1 chain). | Document in the provisioning runbook: **pick vmids from the host's full guest list (PVE UI / root `pct list`+`qm list`), never from the hub inventory.** Longer-term: Day-0/hub could record a host-declared reserved-vmid range. | YES (doc-only) |
| **F4** | LOW | Restore-test can't read a **non-pool source guest's** config for bind-mount neutralization (403) → "restoring as-is"; a bind-mount-bearing non-pool source archive would false-fail the test. | Accept + log (current behavior is fail-safe), or route the source-config read through the fenced root CLI. Today's sources (9201, golden) are unaffected. | no |
| **F5** | LOW | Eject `dependentGuests` scan is double-pool-blind (list + config 403) → non-pool dependents of an ejected drive are not warned about. | Either declare pool-scoped-by-design in the eject UX copy ("Felhom guests using this drive: …"), or scan via the fenced root CLI for the shared-box case. | no (drives are Felhom-enrolled; revisit if drives are ever shared) |
| **F6** | INFO | Stale-lock guard, reconcile readActual, recover reapers: all correct under a pool-only view (details §2 B4); the reconcile desired-state must never reference a non-pool vmid (latent rule, currently unfed). | Record the rule next to `DesiredProvider` when slice-10 feeds it. | no |

**Pre-existing, noted in passing (NOT pool effects):** non-root agent can't read `/etc/pve/priv/storage/felhom-pbs.pw` (known OPEN from the Day-0 bundle — PBS datastore detail skipped in reports); the drive-enrollment candidacy bug has its own pending fix spec (explicitly out of scope per the runbook).

## 6. Not changed

Validation only — **no code, no version bump**. felhom-agent untouched (read-only source trace). The demo host was left exactly as found: guests {9000, 9001, 9100, 9201, 9999}, pool `felhom` = {9201}, agent v0.59.0 active, controller healthy, campaign scratch guests/decoys/PBS archive/probe artifacts all removed.