felhom.eu/documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md

SPIKE — A1 pool-membership read for the stale-lock reaper (2026-07-03)

STATUS: COMPLETE — verdict below. This spike unblocks the A1 implementation spec (audit AUDIT-blast-radius-hostroot-localapi-2026-07-02.md §A, finding A1). No production code shipped; all probes were reads except a fully-torn-down throwaway user/token/role (T3). The production FelhomAgent{Guest,Store,Base} roles, the felhom-agent@pve!agent token, and the live agent were never modified.

Class: SPIKE (empirical validation; no product code). Repos read: felhom-agent @ 3e760a559a (v0.61.0, read-only), felhom.eu @ 9282d60f96. Host: felhom-pve (192.168.0.162, node demo-felhom, PVE 9.2.2, single-node). Identities probed: production scoped token felhom-agent@pve!agent (secret read out-of-band from /etc/felhom-agent/agent.json, length 36, never persisted) vs root@pam (pvesh) as ground-truth oracle.

Verdict (one line): LXC enumeration under the production scoped token is already pool-filtered (T1) — so A1 is a latent, not live, defect today — but the containment lives entirely in the ACL; the recommended defense-in-depth fix is add Pool.Audit to the FelhomAgentGuest role (proven minimal, T2+T3) + a GET /pools/felhom ownership cross-check in staleLockController.Guests(), failing safe (skip recovery) when the pool read fails.

---

1. Confirmed baseline facts (live, 2026-07-03)

• Box population (root pct list): 9001 spike-lxc (stopped), 9100 felhom-golden (stopped), 9201 demo-felhom (running), 9999 felhom-selftest-scratch (stopped). (/cluster/resources additionally shows QEMU VM 9000 spike-vm, irrelevant to pct/LXC paths.)
• felhom pool membership (root pvesh get /pools/felhom): exactly one member, 9201. So the T1 precondition (≥1 non-pool guest) was satisfied by three real non-pool guests — no throwaway guest needed.
• Production ACL: 3 roles per felhom-host-install.sh — PVE_PRIVS_GUEST="VM.Allocate VM.Audit VM.Config.* VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback VM.Backup Pool.Allocate" @ /pool/felhom, PVE_PRIVS_STORE="Datastore.Allocate Datastore.AllocateSpace" @ each storage, PVE_PRIVS_BASE="Sys.Audit SDN.Use Datastore.Audit" @ / (script L183-185). Pool.Audit is absent from all three — confirmed in-script and by T2's live 403.
• Reaper under test: RecoverStaleLockedGuests (felhom-agent/internal/localapi/stalelock.go:58) → staleLockController.Guests() (:159) → ListLXC = GET /nodes/{node}/lxc (internal/proxmox/query.go:33). proxmox.Guest (types.go:65) has no pool field — and T7 proves the API response itself carries none, so this is a PVE-API limitation, not a decoding gap.

2. Probe matrix — results (verbatim statuses/bodies; token secret redacted throughout)

All token calls: curl -sk -H "Authorization: PVEAPIToken=felhom-agent@pve!agent=<REDACTED len=36>" https://127.0.0.1:8006/api2/json/... run on the host itself.

T1 — enumeration scope (THE crux, Q2) — pool-filtered: YES

Identity	GET /nodes/demo-felhom/lxc	vmid set returned
production scoped token	HTTP 200	[9201]
root (pvesh)	200	[9001, 9100, 9201, 9999]

The scoped token's enumeration returns only the pool member. The reaper's scan is therefore already ownership-scoped as deployed today — the three non-pool guests are invisible to it. PVE filters the LXC index by per-guest VM.Audit visibility (the /pool/felhom grant resolves only for pool members — same mechanism as the pool spike's T7 mutation 403s).

T2 — token pool read (Q1) — 403, missing priv named exactly

GET /pools/felhom under the production token →

HTTP 403
{"data":null,"message":"Permission check failed (/pool/felhom, Pool.Audit)\n"}

Companion: GET /pools (index) under the token → HTTP 200, {"data":[]} — the pool index does not error; it silently filters to empty. (PVE read-index convention: no visibility ⇒ empty list, not 403 — see also T3b.)

T3 — priv delta, empirically confirmed on a THROWAWAY identity — Pool.Audit @ /pool/felhom is sufficient AND minimal

Setup (all torn down, §6): user felhom-probe@pve + privsep token felhom-probe@pve!probe + role FelhomProbeRead with only Pool.Audit, granted at /pool/felhom to BOTH user and token (privsep intersection). Production roles/token untouched.

Probe	Result
T3a GET /pools/felhom under probe token	HTTP 200 — poolid: felhom, members: [9201]
T3b GET /nodes/demo-felhom/lxc under probe token (no VM.Audit)	HTTP 200, [] — filtered-empty, NOT 403

T3a proves the exact delta: one read privilege, at the already-granted path. T3b is a load-bearing nuance for every ListLXC consumer: an ACL mis-scope does not error — it silently returns an empty guest list. For the reaper that direction is fail-safe (nothing to reap); for monitoring consumers it is a blind spot (§3 Q6).

T4 — fenced-root alternative — works, but adds root-CLI surface

pvesh get /pools/felhom --output-format json as root → members [9201] (§1). The installer already uses this (felhom-host-install.sh pool_members(), L354). Wiring it into the agent's reaper, however, means a new root-CLI call in proxmox.Privileged — a 4th exception to the deliberately-3-exception fence (keyctl create / USB mount / SMART), for a value the API can supply with a one-priv grant. Rejected on that basis (§3 Q3).

T5 — local ownership records — all partial; none is a membership registry

Inventory of /var/lib/felhom-agent/ + /var/lib/felhom-install/ on the live host:

Record	Content found live	Why it can't gate the reaper
guest-binds.json (GuestBindStore.Guests())	{9201: [3 drive uuids]}	only guests with enrolled drives; a managed guest with no drive would be skipped
journal.log (reconcile op journal)	newest entries: bring-up-9310-1 (vmid 9310)	op history, not membership — 9310 no longer exists on the box; adopted (not agent-created) guests never appear
guests/ state dir	9201	only agent-provisioned guests; an installer---adopt-pool-ed guest is absent
/var/lib/felhom-install/state.json	{completed: [preflight … verify]}	step ledger only; no member list

A local allowlist would fail toward skipping owned guests (fail-safe direction, but a silent functional regression: their stale locks would never heal — the exact customer-down scenario the reaper exists for). Rejected as the primary mechanism (§3 Q3).

T6 — destructive negative confirmation — NOT RUN (as pre-agreed)

Reproducing the audit's exploit live (reaper unlocks a foreign stale-locked guest) requires the reaper to see the foreign guest — i.e. redeploying the agent with a broad token, which this spike forbids. Unprovable live under the scoped token — and T1 is the proof of why (the foreign guest is invisible). Covered by the named unit tests instead (§3 Q5).

T7 — (extra) /cluster/resources shortcut — no free membership; pool field is priv-gated

Identity	GET /cluster/resources?type=vm	9201 pool field
production scoped token	HTTP 200, 1 entry (9201 only)	null
root	200, 5 entries	"felhom"

Even for a guest the token CAN see, PVE withholds the pool attribute without Pool.Audit. So there is no zero-ACL-change API path to membership — any pool-aware refactor needs the same Pool.Audit grant. Also confirmed at root: GET /nodes/{node}/lxc entries carry no pool key at all (PVE 9.2.2), matching the Guest struct.

3. The six required answers (task §8)

Q1 — Is enumeration pool-filtered under the scoped token? Yes (T1: token sees [9201], root sees [9001,9100,9201,9999]). The reaper's scan is already ownership-scoped by the ACL. Corollary (T3b): under a token with no guest visibility the same call returns 200 [], silently.

Q2 — Can the token read /pools/felhom? No — 403, body names the exact missing priv: Permission check failed (/pool/felhom, Pool.Audit). T3a proves Pool.Audit alone at /pool/felhom flips it to 200-with-members.

Q3 — Recommended mechanism: add Pool.Audit to FelhomAgentGuest + a GET /pools/felhom cross-check inside staleLockController.Guests() (intersect ListLXC with pool members; any pool-read error propagates → the existing RecoverStaleLockedGuests error path already skips the whole recovery, fail-safe).

Mechanism	ACL surface added	root-CLI surface added	New failure modes	Blast-radius delta	Verdict
(a) test-only (trust T1's filtering)	none	none	none	containment lives ONLY in the ACL; the audit's precondition (broad-token redeploy) silently re-arms A1	rejected as sole fix — but T1 means the fix is pure defense-in-depth, zero behavior change today
(b) Pool.Audit @ /pool/felhom (Guest role) + API cross-check	one read priv, at an already-granted scoped path (NOT / — a / grant would expose all pools)	none	pool-read failure ⇒ reaper skips all (fail-safe; loud log); mixed-version window: new agent + old ACL ⇒ recovery disabled until --rescope-acl	forecloses the foreign-guest reap even under a broad token	WINNER
(c) fenced-root pvesh get /pools/felhom	none	4th exception to the 3-exception proxmox.Privileged fence	root output parsing; fence erosion precedent	same as (b)	rejected — fence cost > one scoped read priv
(d) local ownership allowlist (T5)	none	none	provably incomplete records ⇒ silently skips owned guests (adopted / drive-less) — the reaper's purpose defeated for them	worse than (b): a "protected" guest stays customer-down	rejected

Why (b) wins: it is the only option that is complete (covers adopted + drive-less guests), scoped (the priv lands at /pool/felhom, readable-membership only), fence-preserving, and whose failure mode degrades to the reaper's existing documented fail-safe ("can't confirm ⇒ don't act", mirroring reconcile/recover.go:38-52).

Q4 — Exact delta for the winner (for the A1 impl spec to lift):

• Installer (felhom.eu/scripts/felhom-host-install.sh L183): PVE_PRIVS_GUEST gains Pool.Audit (12 → 13 privs). _ensure_role already role modifys to the exact set, so --rescope-acl re-application upgrades an existing box idempotently; remove_scoped_acl needs no change (it deletes by role name). Script version bump.
• Agent (felhom-agent): new read Client.Pool(ctx, name) → GET /pools/{name} (decode members[].vmid; internal/proxmox/query.go — read-only, API-side, keeps the fence). staleLockAPI (stalelock.go:140) gains it; staleLockController.Guests() (:159) becomes: ListLXC ∩ pool members, returning an error when the pool read fails — the existing RecoverStaleLockedGuests guard (:62-66, "guest list unavailable — skipping recovery") then fail-safes the whole scan with the already-present loud log. The filter sits before the recoverOne loop, exactly mirroring recover.go's pre-mutation gate. Pool name should come from the shared default (reconcile.DefaultPool, bringup.go:50) threaded via config, not a second literal.
• Deploy coupling: ship agent + re-run --rescope-acl together (same discipline as sudoers-with-binary). Optional hardening: add a pool-read check to the capability probe (internal/capability/) so an old-ACL box reports the degradation instead of silently not-reaping.

Q5 — Test shapes (felhom-testing doctrine: negative + non-trivial positive):

• TestStaleLock_ForeignGuestNotReaped — fake staleLockAPI whose ListLXC returns a broad-token-shaped list {9201 (pool), 5000 (foreign)}, pool members {9201}; guest 5000 seeded with lock: snapshot-delete + a vzdump snapshot + no running vzdump task (the audit's exact exploit precondition). Assert Unlock/DeleteSnapshot/Start are never invoked for 5000.
• TestStaleLock_PoolGuestStillReaped — companion positive: 9201 seeded identically is unlocked + snapshot-deleted + started (proves the filter isn't trivially "reap nothing").
• TestStaleLock_PoolReadFails_SkipsAll — pool read errors ⇒ zero mutations on ANY guest (fail-safe path, red-proofable by breaking the intersect).

Q6 — Blast radius of the T1 answer on the other ListLXC consumers. T1/T3b generalize: every consumer receives a pool-filtered list today, and would receive an unfiltered one under a broad token — none of them checks ownership itself:

Consumer	Under broad token	Risk
hub/collect.go:204 (host report)	foreign guests reported to hub	inventory leak to operator plane; low, but silently assumes pool-only
localapi/disks.go:1055 (guest list for attach)	foreign guests become visible attach targets	mutation still gated per-guest; visibility only
reconcile/recover.go:112,172	destroy-by-existence sees more	already hardened by proof-of-launch (F1 fix) — the pattern the A1 fix mirrors
reconcile/restoretest.go:126	band scan sees squatters it today can't	safer under broad (band-advance was built for the blind case)
reconcile/bringup.go:180, engine.go:229	existence checks see more	safer (fewer "already exists" surprises)
cmd/felhom-agent/main.go:1753 (selftest read)	prints more	none
(all, via T3b)	a mis-scoped token yields 200 [], not an error	"no guests" and "no visibility" are indistinguishable — for hub collect this is a monitoring blind spot worth a future capability-probe check

Only the stale-lock reaper both acts destructively AND relies solely on enumeration scope — which is why A1 is the one that needs the cross-check; the reconcile paths already carry their own fail-safes.

4. NOT validated by this spike

• The winning mechanism end-to-end in the agent (the Pool() client call, the intersect, the capability-probe surfacing) — that is the A1 impl task, gated on this doc.
• The live exploit itself (T6): needs a broad-token agent redeploy; deliberately not performed. The unit tests in Q5 are the designated coverage.
• Multi-node cluster semantics of GET /pools/felhom under the scoped token (single-node box; pools are cluster-wide, expected identical — same caveat as the pool spike §8).
• Whether pveum role modify on a live box propagates Pool.Audit to already-issued tickets instantly (T3 used a fresh token; PVE ACL changes are normally immediate, and the reaper runs at agent startup, so staleness risk is nil in practice).

5. Appendix — committed-secrets rotation micro-runbook (findings only, NO execution)

Unrelated to A1; folded in per the task's idle-time clause. manifests/felhom.secret.yaml still commits live stringData for three Secrets in felhom-system (verified 2026-07-03; values not reproduced here):

Secret	Committed keys (values in git)	Deployed consumer
healthchecks-config	SECRET_KEY, SUPERUSER_{EMAIL,PASSWORD}, SMTP settings (EMAIL_HOST_PASSWORD already de-gitted → out-of-band resend-api)	none — healthchecks not deployed
umami-config	APP_SECRET, POSTGRES_PASSWORD	umami.yaml (3 secretKeyRefs) — live
gitea-creds	username (admin), password	hub.yaml (2 secretKeyRefs) — live

Operator action list, per secret (pattern = the Resend de-git, documentation/runbooks/secrets.md):

1. Rotate the upstream value first — git history retains every old value, so de-git alone fixes nothing; rotation is the real remediation. (gitea-creds: change the Gitea admin password; umami-config: new APP_SECRET + rotate the CNPG role password; healthchecks-config: nothing live to rotate, but the committed superuser password should be considered burned.)
2. Create the Secret out-of-band with the NEW values: kubectl create secret generic <name> -n felhom-system --from-literal=KEY=... [...].
3. Delete that Secret's block from manifests/felhom.secret.yaml, commit, ArgoCD sync (pruning off for Secrets — verify the out-of-band object survives the sync), then rollout-restart the consumer.
4. Order: consumers keep reading the mounted/env value until restart, so do 1-2 before 3.

Execution is out of scope for this spike — recorded as an operator follow-up.

6. Cleanup confirmation

Verified live after teardown: 0 ACL entries, 0 roles, 0 users matching the probe identity (pveum acl list / role list / user list filtered — all empty). The throwaway felhom-probe@pve user, its probe token, and the FelhomProbeRead role were removed in-session; no throwaway guest was ever created (real non-pool guests satisfied T1). The production token's secret was read on-host into a shell variable only, never echoed, never persisted anywhere (recorded as length=36). Production roles/ACL/token/agent untouched; live agent still running against guest 9201.