felhom.eu/documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md

CAMPAIGN — full effects of the felhom pool + scoped-token migration

Resolution update (2026-07-02, agent v0.60.0): F1 FIXED at root — proof-of-launch gating: a compensating destroy (bring-up rollback, restore-test teardown, AND Recover) fires only when RestoreLXC returned a UPID; no UPID ⇒ the transaction created nothing ⇒ never destroy. Recover is now safe by DESIGN (a no-UPID Scratch/Rollback entry is abandoned fail-safe), no longer by the pool-blind "already gone" accident (§2 B1.4). F2 FIXED — the restore-test advances past an "already exists" band vmid (bounded by the band width); a fully-occupied band → Skipped, no false alert. F3 DOCUMENTED — documentation/runbooks/provisioning.md. Red-proof-tested + live-proven on felhom-pve (provision onto existing 9001 → no destroy armed; restore-test advanced past a 990000 decoy and passed at 990001). F4/F5/F6 remain open as recorded. Operator follow-up: bump the hub Day-0 artifact manifest to agent 0.60.0 (binary published to Gitea, sha256 45d19b87…51725831).

Date run: 2026-07-02 (runbook authored 2026-07-01) · Host: felhom-pve (demo-felhom, PVE 9.2.2) Agent live: v0.59.0 (runbook table said v0.55.0 — v0.56–0.59 shipped between authoring and execution; the ListLXC/priv surface is unchanged) ACL: host-install v1.7.0 3-role scoped layout, verified live: FelhomAgentGuest @/pool/felhom (12 privs incl. Pool.Allocate), FelhomAgentStore (Datastore.Allocate,AllocateSpace) @local/local-lvm/felhom-pbs, FelhomAgentBase (Sys.Audit,SDN.Use,Datastore.Audit) @/. Method: the agent's own code paths where they exist (--selftest=bring-up|task|restore-test|hub, run as the felhom-agent user), raw PVE-API probes as the scoped token for the rest (identical paths/params to internal/proxmox/mutate.go). Mutations on scratch pool guests 9310/9311 + throwaway decoys; 9201 got read/non-destructive probes only. No code changed; demo left as found (pool = {9201}, agent active, controller healthy, all scratch/decoys destroyed, 9310's PBS archive pruned).

---

1. Track A — operation-matrix sweep: 16/16 PASS, zero surprise 403s

The scoped token is functionally COMPLETE for the agent's real operation surface.

#	Op	How exercised	Result
1	RestoreLXC → pool	bring-up selftest (golden → 9310, 52s boot+running); PBS archive → 9311 raw probe	PASS (both local + PBS archive paths)
2	Vzdump → felhom-pbs	raw probe, mode=snapshot, on 9310	PASS (task OK, 100s)
3	DestroyLXC + destroy-unreferenced-disks=1	raw probe on 9310+9311	PASS — all 6 LVM volumes verifiably reaped
4	Snapshot	--selftest=task -vmid 9310	PASS
5	Rollback	same	PASS
6	DeleteSnapshot	same	PASS
7	SetConfig Disk/Network/CPU/Memory/Options	bring-up (Disk/mounts) + raw probes: net1 add+delete, cores, memory, onboot, description (selftest)	PASS all five key classes
8	ResizeLXC	rootfs +1G raw probe (32G→33G, task OK)	PASS
9	Start / Stop	bring-up start; stop probe pre-destroy	PASS
10	ListLXC	probe	PASS — but returns ONLY pool guests (Track B)
11	GuestStatus/Config/ListSnapshots	selftests + probes	PASS (pool guests)
12	Nodes/NodeStatus/Version	--selftest=hub collect	PASS
13	ListRunningTasks	probed DURING a live vzdump	PASS — task visible mid-flight
14	ListStorage/NodeStorage/StorageContent/LatestBackupVolID	probes incl. felhom-pbs content enumeration	PASS (3b-fix intact)
15	TaskStatusOnce/WaitTask	used throughout (every async op)	PASS
16	TaskLogTail	full log of completed vzdump read	PASS

Full restore-test cycle (restore→link-down→boot→verify→teardown) also PASS end-to-end under the scoped token (20s), including teardown of its own pool-created scratch.

Transient (not a finding): concurrent SetConfig during a running resize task → HTTP 500 "can't lock file pve-config-9310.lock" — ordinary PVE per-guest config locking, retry succeeds.

---

2. Track B — pool-blindness (the depth)

Mechanism proven live first: the scoped token's GET /nodes/demo-felhom/lxc returns only {9201} while the host holds 9000/9001/9100/9999/9201. Every ListLXC consumer inherits this view.

B1 — vmid origin + collision verdict ⭐ the campaign's most important result

Origin (source-traced, definitive): the provision vmid is always externally supplied — both BringUpSpec call sites (--selftest=bring-up, --selftest=provision, cmd/felhom-agent/main.go:1233/1388) take -vmid as an explicit CLI flag. No nextid/next-free scan exists anywhere in the agent or the hub (grep-verified both repos). The only agent-computed vmid is the restore-test scratch (pickScratchVMID over ListLXC, band 990000–990009, restoretest.go:419).

Collision-avoidance therefore sits with the operator/hub — and the operator's hub inventory is now pool-only (B3), so on a shared box the operator can pick a vmid occupied by a guest the hub can't see. What happens then:

Adversarial run 1 — bring-up onto existing non-pool 9001:

1. The bring-up duplicate-guard (bringup.go:180, pool-blind ListLXC) MISSED the existing guest and proceeded.
2. PVE refused the restore: HTTP 500 "CT 9001 already exists" (agent never sets force) → no clobber.
3. The compensating rollback then attempted DestroyLXC(9001) — a guest the job never created (gate classed it benign guest_destroy, SameTxnCreated provenance). Only the pool ACL stopped it: HTTP 403 "missing privilege VM.Allocate at /vms/9001". Under the pre-3b broad token this would have DESTROYED the innocent guest.
4. The dangling in-flight Rollback journal entry self-resolved on the next Recover: pool-blind ListLXC can't see 9001 → "guest already gone" → terminal-clean. (Coincidentally correct: containment guarantees the agent can't have created/kept anything outside the pool.)

Adversarial run 2 — restore-test with a non-pool decoy parked at 990000 (band start):

1. pickScratchVMID (pool-blind) picked the occupied 990000.
2. Restore refused ("CT 990000 already exists") → no clobber.
3. The deferred teardown attempted DestroyLXC(990000) on the innocent decoy → 403, decoy survived.
4. Journal entry self-resolved clean on next Recover (same pool-blind coincidence).
5. Result surfaced as FAIL (pass:false, "already exists") — not Skipped — i.e. a false "backup not restorable" signal to the hub.
6. Because the picker always takes the lowest free-looking vmid and the squatter never becomes visible, it re-picks 990000 every run: one non-pool squatter at the band start = restore-test permanently broken (it never advances to 990001).

Verdict: no collision can clobber or destroy a foreign guest as long as the pool-scoped ACL is in place — but that is a single containment layer over an agent-logic flaw (destroy-what-you-didn't-create), and the failure mode is noisy-but-safe (provision fails, ERROR logs, false restore-test alerts). See fixes F1/F2/F3.

Bonus finding (same run): with a non-pool source guest (the golden 9100), the restore-test cannot read the source config for bind-mount neutralization (403 VM.Audit) → logs "restoring as-is". Harmless for the golden (no bind mounts), but a bind-mount-bearing non-pool source archive would fail the restore-test instead of being neutralized (F4).

B2 — eject dependent-guest scan (disks.go:1026 dependentGuests)

Pool-blind at TWO layers, proven live without a disruptive eject: (a) the scan enumerates via ListLXC → non-pool guests never enter the loop (list = {9201} proven); (b) even if they did, GuestConfig on a non-pool guest 403s (observed live on 9100). On a shared box, ejecting a drive a non-pool guest bind-mounts would be under-warned. Severity: low today (single-tenant demo; Felhom drives are enrolled to Felhom guests), real for the shared-box trial. Fix note in F5.

B3 — hub inventory (collect.go:204)

Confirmed live via --selftest=hub: the host report's guests = only 9201; the four non-pool guests are invisible to the hub/operator. Decision recorded: intended post-containment — the agent manages (and reports) only its pool. The consequence that must be documented (F3): the hub inventory is NOT a full-host view, so operator vmid selection for a new provision must consult the host's real guest list (PVE UI/root), never the hub inventory, on any box that has non-Felhom guests.

B4 — remaining ListLXC consumers

Consumer	Behavior with a pool-only list	Verdict
stale-lock guard (stalelock.go)	Guest enumeration pool-only → recovers only pool guests (intended: non-pool guests aren't the agent's to unlock). Its safety input ListRunningTasks is NOT pool-blind — proven live: a root-initiated vzdump of non-pool 9001 was visible to the token mid-flight (Sys.Audit@/ covers the task list). No misfire vector found.	✅ correct
reconcile engine.readActual (engine.go:229)	Actual-state is pool-only. Today EmptyProvider → no effect. Latent rule: desired-state must never reference a non-pool guest, or reconcile would treat it as absent and re-provision-over (which then hits the B1 chain, contained).	✅ today; rule recorded
recover.go (scratch + bring-up reapers)	Existence check pool-only → an out-of-pool collision vmid resolves "already gone" (terminal-clean). Coincidentally correct under containment (see B1.4); would be WRONG (silent leak of a half-built guest) only if the agent could create outside the pool — it can't (restore without pool=felhom 403s).	✅ under scoped ACL
--selftest=read list check (main.go:1753)	Cosmetic (prints pool guests only).	✅

---

3. Track C — blast-radius regression: containment intact

Probe	Result
Stop non-pool 9001 as token	403 "Permission check failed (/vms/9001, VM.PowerMgmt)"
Vzdump 9201 → non-granted felhom-usb	403 "(/storage/felhom-usb, Datastore.AllocateSpace)" — refused at POST, nothing silently retargeted, no task started
Drive-storage audit (felhom-usb/felhom-flash)	readable (status + list, HTTP 200) — 3b-fix (Datastore.Audit@/ in Base) intact, no false detach

---

4. Track D — pool-membership non-ACL effects: no surprises

• No leak: the pool name appears in NO customer/controller-facing surface — hub host-report guest entries carry vmid/name/status/controller_version/spec only (verified live); pct config 9201 has no pool field (pool membership lives in PVE user.cfg); the guest-visible bootstrap mount is bootstrap.json only.
• --adopt-pool (v1.8.0, live run): idempotent — "pool felhom already exists" + "guest 9201 already in pool felhom", guest not otherwise touched.
• --uninstall --dry-run (live): removes the /pool/felhom ACL grants and correctly refuses to delete a non-empty pool ("pool felhom not empty (members: 9201) — leaving it"). In a real uninstall the destroyed guest auto-leaves the pool first; the guard protects foreign members.

---

5. Prioritized fix list

#	Sev	Finding	Recommended fix	Blocks shared-box trial?
F1	HIGH (logic; contained by ACL)	Bring-up rollback + restore-test teardown fire DestroyLXC on a vmid the job never created (create failed "already exists"). One ACL layer away from destroying a foreign guest; any future broad-token/root deployment re-arms it.	Arm the compensating destroy only after RestoreLXC returns a UPID (nothing created ⇒ nothing to roll back). A POST network error (outcome unknown) may keep today's behavior. Applies to bringup.go defer + restoretest.go defer (+ their Recover twins are already exists-checked).	YES — fix before the trial (defense-in-depth; the trial box has foreign guests by definition)
F2	MED	One non-pool squatter at the scratch-band start ⇒ restore-test permanently FAILs (always re-picks the lowest invisible-occupied vmid; reports FAIL not Skip → false "backup unrestorable" alerts).	On an "already exists" restore failure, advance to the next band vmid (or pre-verify emptiness via the fenced root pct list); band exhausted ⇒ Skip (existing semantics). Additionally: document 990000–990009 as reserved fleet-wide on any Felhom host.	YES (cheap; false alerts would drown the trial's signal)
F3	MED (process/doc)	Hub inventory is pool-only (by design) while vmid assignment is operator-side ⇒ on a shared box the operator can pick an occupied, hub-invisible vmid. Outcome is safe-but-noisy (B1 chain).	Document in the provisioning runbook: pick vmids from the host's full guest list (PVE UI / root pct list+qm list), never from the hub inventory. Longer-term: Day-0/hub could record a host-declared reserved-vmid range.	YES (doc-only)
F4	LOW	Restore-test can't read a non-pool source guest's config for bind-mount neutralization (403) → "restoring as-is"; a bind-mount-bearing non-pool source archive would false-fail the test.	Accept + log (current behavior is fail-safe), or route the source-config read through the fenced root CLI. Today's sources (9201, golden) are unaffected.	no
F5	LOW	Eject dependentGuests scan is double-pool-blind (list + config 403) → non-pool dependents of an ejected drive are not warned about.	Either declare pool-scoped-by-design in the eject UX copy ("Felhom guests using this drive: …"), or scan via the fenced root CLI for the shared-box case.	no (drives are Felhom-enrolled; revisit if drives are ever shared)
F6	INFO	Stale-lock guard, reconcile readActual, recover reapers: all correct under a pool-only view (details §2 B4); the reconcile desired-state must never reference a non-pool vmid (latent rule, currently unfed).	Record the rule next to DesiredProvider when slice-10 feeds it.	no

Pre-existing, noted in passing (NOT pool effects): non-root agent can't read /etc/pve/priv/storage/felhom-pbs.pw (known OPEN from the Day-0 bundle — PBS datastore detail skipped in reports); the drive-enrollment candidacy bug has its own pending fix spec (explicitly out of scope per the runbook).

6. Not changed

Validation only — no code, no version bump. felhom-agent untouched (read-only source trace). The demo host was left exactly as found: guests {9000, 9001, 9100, 9201, 9999}, pool felhom = {9201}, agent v0.59.0 active, controller healthy, campaign scratch guests/decoys/PBS archive/probe artifacts all removed.