# SPIKE — A1 pool-membership read for the stale-lock reaper (2026-07-03)

> **STATUS: COMPLETE — verdict below.** This spike unblocks the A1 implementation spec (audit
> `AUDIT-blast-radius-hostroot-localapi-2026-07-02.md` §A, finding A1). No production code shipped;
> all probes were reads except a fully-torn-down throwaway user/token/role (T3). The production
> `FelhomAgent{Guest,Store,Base}` roles, the `felhom-agent@pve!agent` token, and the live agent were
> **never modified**.

**Class:** SPIKE (empirical validation; no product code). **Repos read:** felhom-agent @ `3e760a559a`
(v0.61.0, read-only), felhom.eu @ `9282d60f96`. **Host:** felhom-pve (192.168.0.162, node
`demo-felhom`, PVE 9.2.2, single-node). **Identities probed:** production scoped token
`felhom-agent@pve!agent` (secret read out-of-band from `/etc/felhom-agent/agent.json`, length 36,
never persisted) vs `root@pam` (`pvesh`) as ground-truth oracle.

**Verdict (one line):** LXC enumeration under the production scoped token **is already
pool-filtered** (T1) — so A1 is a latent, not live, defect today — but the containment lives
entirely in the ACL; the recommended defense-in-depth fix is **add `Pool.Audit` to the
`FelhomAgentGuest` role** (proven minimal, T2+T3) **+ a `GET /pools/felhom` ownership cross-check
in `staleLockController.Guests()`**, failing safe (skip recovery) when the pool read fails.

---

## 1. Confirmed baseline facts (live, 2026-07-03)

- Box population (root `pct list`): **9001** `spike-lxc` (stopped), **9100** `felhom-golden`
  (stopped), **9201** `demo-felhom` (running), **9999** `felhom-selftest-scratch` (stopped).
  (`/cluster/resources` additionally shows QEMU VM 9000 `spike-vm`, irrelevant to `pct`/LXC paths.)
- `felhom` pool membership (root `pvesh get /pools/felhom`): **exactly one member, 9201**. So the
  T1 precondition (≥1 non-pool guest) was satisfied by three real non-pool guests — no throwaway
  guest needed.
- Production ACL: 3 roles per `felhom-host-install.sh` —
  `PVE_PRIVS_GUEST="VM.Allocate VM.Audit VM.Config.* VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback
  VM.Backup Pool.Allocate"` @ `/pool/felhom`, `PVE_PRIVS_STORE="Datastore.Allocate
  Datastore.AllocateSpace"` @ each storage, `PVE_PRIVS_BASE="Sys.Audit SDN.Use Datastore.Audit"`
  @ `/` (script L183-185). **`Pool.Audit` is absent from all three** — confirmed in-script and by
  T2's live 403.
- Reaper under test: `RecoverStaleLockedGuests` (`felhom-agent/internal/localapi/stalelock.go:58`)
  → `staleLockController.Guests()` (`:159`) → `ListLXC` = `GET /nodes/{node}/lxc`
  (`internal/proxmox/query.go:33`). `proxmox.Guest` (`types.go:65`) has **no pool field** — and T7
  proves the API response itself carries none, so this is a PVE-API limitation, not a decoding gap.

## 2. Probe matrix — results (verbatim statuses/bodies; token secret redacted throughout)

All token calls: `curl -sk -H "Authorization: PVEAPIToken=felhom-agent@pve!agent=<REDACTED len=36>"
https://127.0.0.1:8006/api2/json/...` run on the host itself.

### T1 — enumeration scope (THE crux, Q2) — **pool-filtered: YES**

| Identity | `GET /nodes/demo-felhom/lxc` | vmid set returned |
|---|---|---|
| production scoped token | **HTTP 200** | **[9201]** |
| root (`pvesh`) | 200 | [9001, 9100, 9201, 9999] |

The scoped token's enumeration returns **only the pool member**. The reaper's scan is therefore
already ownership-scoped *as deployed today* — the three non-pool guests are invisible to it.
PVE filters the LXC index by per-guest `VM.Audit` visibility (the `/pool/felhom` grant resolves
only for pool members — same mechanism as the pool spike's T7 mutation 403s).

### T2 — token pool read (Q1) — **403, missing priv named exactly**

`GET /pools/felhom` under the production token →

```
HTTP 403
{"data":null,"message":"Permission check failed (/pool/felhom, Pool.Audit)\n"}
```

Companion: `GET /pools` (index) under the token → **HTTP 200, `{"data":[]}`** — the pool *index*
does not error; it silently filters to empty. (PVE read-index convention: no visibility ⇒ empty
list, not 403 — see also T3b.)

### T3 — priv delta, empirically confirmed on a THROWAWAY identity — **`Pool.Audit` @ `/pool/felhom` is sufficient AND minimal**

Setup (all torn down, §6): user `felhom-probe@pve` + privsep token `felhom-probe@pve!probe` + role
`FelhomProbeRead` with **only** `Pool.Audit`, granted at `/pool/felhom` to BOTH user and token
(privsep intersection). Production roles/token untouched.

| Probe | Result |
|---|---|
| T3a `GET /pools/felhom` under probe token | **HTTP 200** — `poolid: felhom, members: [9201]` |
| T3b `GET /nodes/demo-felhom/lxc` under probe token (no `VM.Audit`) | **HTTP 200, `[]`** — filtered-empty, NOT 403 |

T3a proves the exact delta: one read privilege, at the already-granted path. T3b is a load-bearing
nuance for every `ListLXC` consumer: **an ACL mis-scope does not error — it silently returns an
empty guest list.** For the reaper that direction is fail-safe (nothing to reap); for monitoring
consumers it is a blind spot (§3 Q6).

### T4 — fenced-root alternative — works, but adds root-CLI surface

`pvesh get /pools/felhom --output-format json` as root → members `[9201]` (§1). The installer
already uses this (`felhom-host-install.sh` `pool_members()`, L354). Wiring it into the *agent's*
reaper, however, means a new root-CLI call in `proxmox.Privileged` — a **4th exception** to the
deliberately-3-exception fence (keyctl create / USB mount / SMART), for a value the API can supply
with a one-priv grant. Rejected on that basis (§3 Q3).

### T5 — local ownership records — **all partial; none is a membership registry**

Inventory of `/var/lib/felhom-agent/` + `/var/lib/felhom-install/` on the live host:

| Record | Content found live | Why it can't gate the reaper |
|---|---|---|
| `guest-binds.json` (`GuestBindStore.Guests()`) | `{9201: [3 drive uuids]}` | only guests with **enrolled drives**; a managed guest with no drive would be skipped |
| `journal.log` (reconcile op journal) | newest entries: `bring-up-9310-1` (vmid **9310**) | op *history*, not membership — **9310 no longer exists on the box**; adopted (not agent-created) guests never appear |
| `guests/` state dir | `9201` | only agent-provisioned guests; an installer-`--adopt-pool`-ed guest is absent |
| `/var/lib/felhom-install/state.json` | `{completed: [preflight … verify]}` | step ledger only; **no member list** |

A local allowlist would fail toward *skipping owned guests* (fail-safe direction, but a silent
functional regression: their stale locks would never heal — the exact customer-down scenario the
reaper exists for). Rejected as the primary mechanism (§3 Q3).

### T6 — destructive negative confirmation — **NOT RUN (as pre-agreed)**

Reproducing the audit's exploit live (reaper unlocks a foreign stale-locked guest) requires the
reaper to *see* the foreign guest — i.e. redeploying the agent with a broad token, which this spike
forbids. **Unprovable live under the scoped token — and T1 is the proof of why** (the foreign guest
is invisible). Covered by the named unit tests instead (§3 Q5).

### T7 — (extra) `/cluster/resources` shortcut — **no free membership; `pool` field is priv-gated**

| Identity | `GET /cluster/resources?type=vm` | 9201 `pool` field |
|---|---|---|
| production scoped token | HTTP 200, 1 entry (9201 only) | **`null`** |
| root | 200, 5 entries | `"felhom"` |

Even for a guest the token CAN see, PVE **withholds the `pool` attribute without `Pool.Audit`**.
So there is no zero-ACL-change API path to membership — any pool-aware refactor needs the same
`Pool.Audit` grant. Also confirmed at root: `GET /nodes/{node}/lxc` entries carry **no pool key at
all** (PVE 9.2.2), matching the `Guest` struct.

## 3. The six required answers (task §8)

**Q1 — Is enumeration pool-filtered under the scoped token?** **Yes** (T1: token sees `[9201]`,
root sees `[9001,9100,9201,9999]`). The reaper's scan is already ownership-scoped *by the ACL*.
Corollary (T3b): under a token with no guest visibility the same call returns `200 []`, silently.

**Q2 — Can the token read `/pools/felhom`?** **No** — 403, body names the exact missing priv:
`Permission check failed (/pool/felhom, Pool.Audit)`. T3a proves `Pool.Audit` alone at
`/pool/felhom` flips it to 200-with-members.

**Q3 — Recommended mechanism:** **add `Pool.Audit` to `FelhomAgentGuest` + a `GET /pools/felhom`
cross-check inside `staleLockController.Guests()`** (intersect `ListLXC` with pool members; any
pool-read error propagates → the existing `RecoverStaleLockedGuests` error path already skips the
whole recovery, fail-safe).

| Mechanism | ACL surface added | root-CLI surface added | New failure modes | Blast-radius delta | Verdict |
|---|---|---|---|---|---|
| (a) test-only (trust T1's filtering) | none | none | none | containment lives ONLY in the ACL; the audit's precondition (broad-token redeploy) silently re-arms A1 | **rejected as sole fix** — but T1 means the fix is pure defense-in-depth, zero behavior change today |
| (b) `Pool.Audit` @ `/pool/felhom` (Guest role) + API cross-check | **one read priv, at an already-granted scoped path** (NOT `/` — a `/` grant would expose all pools) | none | pool-read failure ⇒ reaper skips all (fail-safe; loud log); mixed-version window: new agent + old ACL ⇒ recovery disabled until `--rescope-acl` | forecloses the foreign-guest reap even under a broad token | **WINNER** |
| (c) fenced-root `pvesh get /pools/felhom` | none | **4th exception** to the 3-exception `proxmox.Privileged` fence | root output parsing; fence erosion precedent | same as (b) | rejected — fence cost > one scoped read priv |
| (d) local ownership allowlist (T5) | none | none | provably incomplete records ⇒ silently skips owned guests (adopted / drive-less) — the reaper's purpose defeated for them | worse than (b): a "protected" guest stays customer-down | rejected |

Why (b) wins: it is the only option that is complete (covers adopted + drive-less guests), scoped
(the priv lands at `/pool/felhom`, readable-membership only), fence-preserving, and whose failure
mode degrades to the reaper's existing documented fail-safe ("can't confirm ⇒ don't act",
mirroring `reconcile/recover.go:38-52`).

**Q4 — Exact delta for the winner** (for the A1 impl spec to lift):

- *Installer* (`felhom.eu/scripts/felhom-host-install.sh` L183): `PVE_PRIVS_GUEST` gains
  `Pool.Audit` (12 → 13 privs). `_ensure_role` already `role modify`s to the exact set, so
  `--rescope-acl` re-application upgrades an existing box idempotently; `remove_scoped_acl` needs
  no change (it deletes by role name). Script version bump.
- *Agent* (`felhom-agent`): new read `Client.Pool(ctx, name)` → `GET /pools/{name}` (decode
  `members[].vmid`; `internal/proxmox/query.go` — read-only, API-side, keeps the fence).
  `staleLockAPI` (`stalelock.go:140`) gains it; `staleLockController.Guests()` (`:159`) becomes:
  `ListLXC` ∩ pool members, returning an error when the pool read fails — the existing
  `RecoverStaleLockedGuests` guard (`:62-66`, "guest list unavailable — skipping recovery")
  then fail-safes the whole scan with the already-present loud log. The filter sits **before** the
  `recoverOne` loop, exactly mirroring recover.go's pre-mutation gate. Pool name should come from
  the shared default (`reconcile.DefaultPool`, `bringup.go:50`) threaded via config, not a second
  literal.
- *Deploy coupling*: ship agent + re-run `--rescope-acl` together (same discipline as
  sudoers-with-binary). Optional hardening: add a pool-read check to the capability probe
  (`internal/capability/`) so an old-ACL box reports the degradation instead of silently
  not-reaping.

**Q5 — Test shapes** (felhom-testing doctrine: negative + non-trivial positive):

- `TestStaleLock_ForeignGuestNotReaped` — fake `staleLockAPI` whose `ListLXC` returns a
  broad-token-shaped list `{9201 (pool), 5000 (foreign)}`, pool members `{9201}`; guest 5000 seeded
  with `lock: snapshot-delete` + a `vzdump` snapshot + no running vzdump task (the audit's exact
  exploit precondition). Assert `Unlock`/`DeleteSnapshot`/`Start` are **never** invoked for 5000.
- `TestStaleLock_PoolGuestStillReaped` — companion positive: 9201 seeded identically **is**
  unlocked + snapshot-deleted + started (proves the filter isn't trivially "reap nothing").
- `TestStaleLock_PoolReadFails_SkipsAll` — pool read errors ⇒ zero mutations on ANY guest
  (fail-safe path, red-proofable by breaking the intersect).

**Q6 — Blast radius of the T1 answer on the other `ListLXC` consumers.** T1/T3b generalize: every
consumer receives a *pool-filtered* list today, and would receive an *unfiltered* one under a broad
token — none of them checks ownership itself:

| Consumer | Under broad token | Risk |
|---|---|---|
| `hub/collect.go:204` (host report) | foreign guests reported to hub | inventory leak to operator plane; low, but silently assumes pool-only |
| `localapi/disks.go:1055` (guest list for attach) | foreign guests become visible attach targets | mutation still gated per-guest; visibility only |
| `reconcile/recover.go:112,172` | destroy-by-existence sees more | already hardened by proof-of-launch (F1 fix) — the pattern the A1 fix mirrors |
| `reconcile/restoretest.go:126` | band scan sees squatters it today can't | *safer* under broad (band-advance was built for the blind case) |
| `reconcile/bringup.go:180`, `engine.go:229` | existence checks see more | safer (fewer "already exists" surprises) |
| `cmd/felhom-agent/main.go:1753` (selftest read) | prints more | none |
| *(all, via T3b)* | a mis-scoped token yields `200 []`, not an error | "no guests" and "no visibility" are indistinguishable — for hub collect this is a monitoring blind spot worth a future capability-probe check |

Only the **stale-lock reaper** both acts destructively AND relies solely on enumeration scope —
which is why A1 is the one that needs the cross-check; the reconcile paths already carry their own
fail-safes.

## 4. NOT validated by this spike

- The winning mechanism end-to-end **in the agent** (the `Pool()` client call, the intersect, the
  capability-probe surfacing) — that is the A1 impl task, gated on this doc.
- The live exploit itself (T6): needs a broad-token agent redeploy; deliberately not performed.
  The unit tests in Q5 are the designated coverage.
- Multi-node cluster semantics of `GET /pools/felhom` under the scoped token (single-node box;
  pools are cluster-wide, expected identical — same caveat as the pool spike §8).
- Whether `pveum role modify` on a live box propagates `Pool.Audit` to already-issued tickets
  instantly (T3 used a fresh token; PVE ACL changes are normally immediate, and the reaper runs at
  agent startup, so staleness risk is nil in practice).

## 5. Appendix — committed-secrets rotation micro-runbook (findings only, NO execution)

Unrelated to A1; folded in per the task's idle-time clause. `manifests/felhom.secret.yaml` still
commits live `stringData` for three Secrets in `felhom-system` (verified 2026-07-03; values not
reproduced here):

| Secret | Committed keys (values in git) | Deployed consumer |
|---|---|---|
| `healthchecks-config` | `SECRET_KEY`, `SUPERUSER_{EMAIL,PASSWORD}`, SMTP settings (`EMAIL_HOST_PASSWORD` already de-gitted → out-of-band `resend-api`) | **none** — healthchecks not deployed |
| `umami-config` | `APP_SECRET`, `POSTGRES_PASSWORD` | **`umami.yaml`** (3 `secretKeyRef`s) — live |
| `gitea-creds` | `username` (admin), `password` | **`hub.yaml`** (2 `secretKeyRef`s) — live |

Operator action list, per secret (pattern = the Resend de-git, `documentation/runbooks/secrets.md`):
1. **Rotate the upstream value first** — git history retains every old value, so de-git alone fixes
   nothing; rotation is the real remediation. (`gitea-creds`: change the Gitea admin password;
   `umami-config`: new `APP_SECRET` + rotate the CNPG role password; `healthchecks-config`: nothing
   live to rotate, but the committed superuser password should be considered burned.)
2. Create the Secret out-of-band with the NEW values:
   `kubectl create secret generic <name> -n felhom-system --from-literal=KEY=... [...]`.
3. Delete that Secret's block from `manifests/felhom.secret.yaml`, commit, ArgoCD sync (pruning off
   for Secrets — verify the out-of-band object survives the sync), then rollout-restart the consumer.
4. Order: consumers keep reading the mounted/env value until restart, so do 1-2 before 3.

Execution is out of scope for this spike — recorded as an operator follow-up.

## 6. Cleanup confirmation

Verified live after teardown: **0** ACL entries, **0** roles, **0** users matching the probe
identity (`pveum acl list` / `role list` / `user list` filtered — all empty). The throwaway
`felhom-probe@pve` user, its `probe` token, and the `FelhomProbeRead` role were removed in-session;
no throwaway guest was ever created (real non-pool guests satisfied T1). The production token's
secret was read on-host into a shell variable only, never echoed, never persisted anywhere
(recorded as length=36). Production roles/ACL/token/agent untouched; live agent still running
against guest 9201.