docs: SPIKE — A1 pool-membership read for the stale-lock reaper
Live-probed on felhom-pve under the PRODUCTION scoped token vs root: enumeration IS pool-filtered (T1: token sees [9201] of 4 guests); pool read 403s naming Pool.Audit (T2); Pool.Audit@/pool/felhom alone suffices (T3, throwaway identity, torn down); /cluster/resources withholds the pool field without Pool.Audit (T7); local ownership records all partial (T5). Recommendation for the A1 impl spec (now unblocked): Pool.Audit added to FelhomAgentGuest + GET /pools/felhom cross-check in staleLockController, fail-safe skip on read failure. Appendix: committed-secrets rotation micro-runbook (operator follow-up). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -0,0 +1,251 @@
|
||||
# SPIKE — A1 pool-membership read for the stale-lock reaper (2026-07-03)
|
||||
|
||||
> **STATUS: COMPLETE — verdict below.** This spike unblocks the A1 implementation spec (audit
|
||||
> `AUDIT-blast-radius-hostroot-localapi-2026-07-02.md` §A, finding A1). No production code shipped;
|
||||
> all probes were reads except a fully-torn-down throwaway user/token/role (T3). The production
|
||||
> `FelhomAgent{Guest,Store,Base}` roles, the `felhom-agent@pve!agent` token, and the live agent were
|
||||
> **never modified**.
|
||||
|
||||
**Class:** SPIKE (empirical validation; no product code). **Repos read:** felhom-agent @ `3e760a559a`
|
||||
(v0.61.0, read-only), felhom.eu @ `9282d60f96`. **Host:** felhom-pve (192.168.0.162, node
|
||||
`demo-felhom`, PVE 9.2.2, single-node). **Identities probed:** production scoped token
|
||||
`felhom-agent@pve!agent` (secret read out-of-band from `/etc/felhom-agent/agent.json`, length 36,
|
||||
never persisted) vs `root@pam` (`pvesh`) as ground-truth oracle.
|
||||
|
||||
**Verdict (one line):** LXC enumeration under the production scoped token **is already
|
||||
pool-filtered** (T1) — so A1 is a latent, not live, defect today — but the containment lives
|
||||
entirely in the ACL; the recommended defense-in-depth fix is **add `Pool.Audit` to the
|
||||
`FelhomAgentGuest` role** (proven minimal, T2+T3) **+ a `GET /pools/felhom` ownership cross-check
|
||||
in `staleLockController.Guests()`**, failing safe (skip recovery) when the pool read fails.
|
||||
|
||||
---
|
||||
|
||||
## 1. Confirmed baseline facts (live, 2026-07-03)
|
||||
|
||||
- Box population (root `pct list`): **9001** `spike-lxc` (stopped), **9100** `felhom-golden`
|
||||
(stopped), **9201** `demo-felhom` (running), **9999** `felhom-selftest-scratch` (stopped).
|
||||
(`/cluster/resources` additionally shows QEMU VM 9000 `spike-vm`, irrelevant to `pct`/LXC paths.)
|
||||
- `felhom` pool membership (root `pvesh get /pools/felhom`): **exactly one member, 9201**. So the
|
||||
T1 precondition (≥1 non-pool guest) was satisfied by three real non-pool guests — no throwaway
|
||||
guest needed.
|
||||
- Production ACL: 3 roles per `felhom-host-install.sh` —
|
||||
`PVE_PRIVS_GUEST="VM.Allocate VM.Audit VM.Config.* VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback
|
||||
VM.Backup Pool.Allocate"` @ `/pool/felhom`, `PVE_PRIVS_STORE="Datastore.Allocate
|
||||
Datastore.AllocateSpace"` @ each storage, `PVE_PRIVS_BASE="Sys.Audit SDN.Use Datastore.Audit"`
|
||||
@ `/` (script L183-185). **`Pool.Audit` is absent from all three** — confirmed in-script and by
|
||||
T2's live 403.
|
||||
- Reaper under test: `RecoverStaleLockedGuests` (`felhom-agent/internal/localapi/stalelock.go:58`)
|
||||
→ `staleLockController.Guests()` (`:159`) → `ListLXC` = `GET /nodes/{node}/lxc`
|
||||
(`internal/proxmox/query.go:33`). `proxmox.Guest` (`types.go:65`) has **no pool field** — and T7
|
||||
proves the API response itself carries none, so this is a PVE-API limitation, not a decoding gap.
|
||||
|
||||
## 2. Probe matrix — results (verbatim statuses/bodies; token secret redacted throughout)
|
||||
|
||||
All token calls: `curl -sk -H "Authorization: PVEAPIToken=felhom-agent@pve!agent=<REDACTED len=36>"
|
||||
https://127.0.0.1:8006/api2/json/...` run on the host itself.
|
||||
|
||||
### T1 — enumeration scope (THE crux, Q2) — **pool-filtered: YES**
|
||||
|
||||
| Identity | `GET /nodes/demo-felhom/lxc` | vmid set returned |
|
||||
|---|---|---|
|
||||
| production scoped token | **HTTP 200** | **[9201]** |
|
||||
| root (`pvesh`) | 200 | [9001, 9100, 9201, 9999] |
|
||||
|
||||
The scoped token's enumeration returns **only the pool member**. The reaper's scan is therefore
|
||||
already ownership-scoped *as deployed today* — the three non-pool guests are invisible to it.
|
||||
PVE filters the LXC index by per-guest `VM.Audit` visibility (the `/pool/felhom` grant resolves
|
||||
only for pool members — same mechanism as the pool spike's T7 mutation 403s).
|
||||
|
||||
### T2 — token pool read (Q1) — **403, missing priv named exactly**
|
||||
|
||||
`GET /pools/felhom` under the production token →
|
||||
|
||||
```
|
||||
HTTP 403
|
||||
{"data":null,"message":"Permission check failed (/pool/felhom, Pool.Audit)\n"}
|
||||
```
|
||||
|
||||
Companion: `GET /pools` (index) under the token → **HTTP 200, `{"data":[]}`** — the pool *index*
|
||||
does not error; it silently filters to empty. (PVE read-index convention: no visibility ⇒ empty
|
||||
list, not 403 — see also T3b.)
|
||||
|
||||
### T3 — priv delta, empirically confirmed on a THROWAWAY identity — **`Pool.Audit` @ `/pool/felhom` is sufficient AND minimal**
|
||||
|
||||
Setup (all torn down, §6): user `felhom-probe@pve` + privsep token `felhom-probe@pve!probe` + role
|
||||
`FelhomProbeRead` with **only** `Pool.Audit`, granted at `/pool/felhom` to BOTH user and token
|
||||
(privsep intersection). Production roles/token untouched.
|
||||
|
||||
| Probe | Result |
|
||||
|---|---|
|
||||
| T3a `GET /pools/felhom` under probe token | **HTTP 200** — `poolid: felhom, members: [9201]` |
|
||||
| T3b `GET /nodes/demo-felhom/lxc` under probe token (no `VM.Audit`) | **HTTP 200, `[]`** — filtered-empty, NOT 403 |
|
||||
|
||||
T3a proves the exact delta: one read privilege, at the already-granted path. T3b is a load-bearing
|
||||
nuance for every `ListLXC` consumer: **an ACL mis-scope does not error — it silently returns an
|
||||
empty guest list.** For the reaper that direction is fail-safe (nothing to reap); for monitoring
|
||||
consumers it is a blind spot (§3 Q6).
|
||||
|
||||
### T4 — fenced-root alternative — works, but adds root-CLI surface
|
||||
|
||||
`pvesh get /pools/felhom --output-format json` as root → members `[9201]` (§1). The installer
|
||||
already uses this (`felhom-host-install.sh` `pool_members()`, L354). Wiring it into the *agent's*
|
||||
reaper, however, means a new root-CLI call in `proxmox.Privileged` — a **4th exception** to the
|
||||
deliberately-3-exception fence (keyctl create / USB mount / SMART), for a value the API can supply
|
||||
with a one-priv grant. Rejected on that basis (§3 Q3).
|
||||
|
||||
### T5 — local ownership records — **all partial; none is a membership registry**
|
||||
|
||||
Inventory of `/var/lib/felhom-agent/` + `/var/lib/felhom-install/` on the live host:
|
||||
|
||||
| Record | Content found live | Why it can't gate the reaper |
|
||||
|---|---|---|
|
||||
| `guest-binds.json` (`GuestBindStore.Guests()`) | `{9201: [3 drive uuids]}` | only guests with **enrolled drives**; a managed guest with no drive would be skipped |
|
||||
| `journal.log` (reconcile op journal) | newest entries: `bring-up-9310-1` (vmid **9310**) | op *history*, not membership — **9310 no longer exists on the box**; adopted (not agent-created) guests never appear |
|
||||
| `guests/` state dir | `9201` | only agent-provisioned guests; an installer-`--adopt-pool`-ed guest is absent |
|
||||
| `/var/lib/felhom-install/state.json` | `{completed: [preflight … verify]}` | step ledger only; **no member list** |
|
||||
|
||||
A local allowlist would fail toward *skipping owned guests* (fail-safe direction, but a silent
|
||||
functional regression: their stale locks would never heal — the exact customer-down scenario the
|
||||
reaper exists for). Rejected as the primary mechanism (§3 Q3).
|
||||
|
||||
### T6 — destructive negative confirmation — **NOT RUN (as pre-agreed)**
|
||||
|
||||
Reproducing the audit's exploit live (reaper unlocks a foreign stale-locked guest) requires the
|
||||
reaper to *see* the foreign guest — i.e. redeploying the agent with a broad token, which this spike
|
||||
forbids. **Unprovable live under the scoped token — and T1 is the proof of why** (the foreign guest
|
||||
is invisible). Covered by the named unit tests instead (§3 Q5).
|
||||
|
||||
### T7 — (extra) `/cluster/resources` shortcut — **no free membership; `pool` field is priv-gated**
|
||||
|
||||
| Identity | `GET /cluster/resources?type=vm` | 9201 `pool` field |
|
||||
|---|---|---|
|
||||
| production scoped token | HTTP 200, 1 entry (9201 only) | **`null`** |
|
||||
| root | 200, 5 entries | `"felhom"` |
|
||||
|
||||
Even for a guest the token CAN see, PVE **withholds the `pool` attribute without `Pool.Audit`**.
|
||||
So there is no zero-ACL-change API path to membership — any pool-aware refactor needs the same
|
||||
`Pool.Audit` grant. Also confirmed at root: `GET /nodes/{node}/lxc` entries carry **no pool key at
|
||||
all** (PVE 9.2.2), matching the `Guest` struct.
|
||||
|
||||
## 3. The six required answers (task §8)
|
||||
|
||||
**Q1 — Is enumeration pool-filtered under the scoped token?** **Yes** (T1: token sees `[9201]`,
|
||||
root sees `[9001,9100,9201,9999]`). The reaper's scan is already ownership-scoped *by the ACL*.
|
||||
Corollary (T3b): under a token with no guest visibility the same call returns `200 []`, silently.
|
||||
|
||||
**Q2 — Can the token read `/pools/felhom`?** **No** — 403, body names the exact missing priv:
|
||||
`Permission check failed (/pool/felhom, Pool.Audit)`. T3a proves `Pool.Audit` alone at
|
||||
`/pool/felhom` flips it to 200-with-members.
|
||||
|
||||
**Q3 — Recommended mechanism:** **add `Pool.Audit` to `FelhomAgentGuest` + a `GET /pools/felhom`
|
||||
cross-check inside `staleLockController.Guests()`** (intersect `ListLXC` with pool members; any
|
||||
pool-read error propagates → the existing `RecoverStaleLockedGuests` error path already skips the
|
||||
whole recovery, fail-safe).
|
||||
|
||||
| Mechanism | ACL surface added | root-CLI surface added | New failure modes | Blast-radius delta | Verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| (a) test-only (trust T1's filtering) | none | none | none | containment lives ONLY in the ACL; the audit's precondition (broad-token redeploy) silently re-arms A1 | **rejected as sole fix** — but T1 means the fix is pure defense-in-depth, zero behavior change today |
|
||||
| (b) `Pool.Audit` @ `/pool/felhom` (Guest role) + API cross-check | **one read priv, at an already-granted scoped path** (NOT `/` — a `/` grant would expose all pools) | none | pool-read failure ⇒ reaper skips all (fail-safe; loud log); mixed-version window: new agent + old ACL ⇒ recovery disabled until `--rescope-acl` | forecloses the foreign-guest reap even under a broad token | **WINNER** |
|
||||
| (c) fenced-root `pvesh get /pools/felhom` | none | **4th exception** to the 3-exception `proxmox.Privileged` fence | root output parsing; fence erosion precedent | same as (b) | rejected — fence cost > one scoped read priv |
|
||||
| (d) local ownership allowlist (T5) | none | none | provably incomplete records ⇒ silently skips owned guests (adopted / drive-less) — the reaper's purpose defeated for them | worse than (b): a "protected" guest stays customer-down | rejected |
|
||||
|
||||
Why (b) wins: it is the only option that is complete (covers adopted + drive-less guests), scoped
|
||||
(the priv lands at `/pool/felhom`, readable-membership only), fence-preserving, and whose failure
|
||||
mode degrades to the reaper's existing documented fail-safe ("can't confirm ⇒ don't act",
|
||||
mirroring `reconcile/recover.go:38-52`).
|
||||
|
||||
**Q4 — Exact delta for the winner** (for the A1 impl spec to lift):
|
||||
|
||||
- *Installer* (`felhom.eu/scripts/felhom-host-install.sh` L183): `PVE_PRIVS_GUEST` gains
|
||||
`Pool.Audit` (12 → 13 privs). `_ensure_role` already `role modify`s to the exact set, so
|
||||
`--rescope-acl` re-application upgrades an existing box idempotently; `remove_scoped_acl` needs
|
||||
no change (it deletes by role name). Script version bump.
|
||||
- *Agent* (`felhom-agent`): new read `Client.Pool(ctx, name)` → `GET /pools/{name}` (decode
|
||||
`members[].vmid`; `internal/proxmox/query.go` — read-only, API-side, keeps the fence).
|
||||
`staleLockAPI` (`stalelock.go:140`) gains it; `staleLockController.Guests()` (`:159`) becomes:
|
||||
`ListLXC` ∩ pool members, returning an error when the pool read fails — the existing
|
||||
`RecoverStaleLockedGuests` guard (`:62-66`, "guest list unavailable — skipping recovery")
|
||||
then fail-safes the whole scan with the already-present loud log. The filter sits **before** the
|
||||
`recoverOne` loop, exactly mirroring recover.go's pre-mutation gate. Pool name should come from
|
||||
the shared default (`reconcile.DefaultPool`, `bringup.go:50`) threaded via config, not a second
|
||||
literal.
|
||||
- *Deploy coupling*: ship agent + re-run `--rescope-acl` together (same discipline as
|
||||
sudoers-with-binary). Optional hardening: add a pool-read check to the capability probe
|
||||
(`internal/capability/`) so an old-ACL box reports the degradation instead of silently
|
||||
not-reaping.
|
||||
|
||||
**Q5 — Test shapes** (felhom-testing doctrine: negative + non-trivial positive):
|
||||
|
||||
- `TestStaleLock_ForeignGuestNotReaped` — fake `staleLockAPI` whose `ListLXC` returns a
|
||||
broad-token-shaped list `{9201 (pool), 5000 (foreign)}`, pool members `{9201}`; guest 5000 seeded
|
||||
with `lock: snapshot-delete` + a `vzdump` snapshot + no running vzdump task (the audit's exact
|
||||
exploit precondition). Assert `Unlock`/`DeleteSnapshot`/`Start` are **never** invoked for 5000.
|
||||
- `TestStaleLock_PoolGuestStillReaped` — companion positive: 9201 seeded identically **is**
|
||||
unlocked + snapshot-deleted + started (proves the filter isn't trivially "reap nothing").
|
||||
- `TestStaleLock_PoolReadFails_SkipsAll` — pool read errors ⇒ zero mutations on ANY guest
|
||||
(fail-safe path, red-proofable by breaking the intersect).
|
||||
|
||||
**Q6 — Blast radius of the T1 answer on the other `ListLXC` consumers.** T1/T3b generalize: every
|
||||
consumer receives a *pool-filtered* list today, and would receive an *unfiltered* one under a broad
|
||||
token — none of them checks ownership itself:
|
||||
|
||||
| Consumer | Under broad token | Risk |
|
||||
|---|---|---|
|
||||
| `hub/collect.go:204` (host report) | foreign guests reported to hub | inventory leak to operator plane; low, but silently assumes pool-only |
|
||||
| `localapi/disks.go:1055` (guest list for attach) | foreign guests become visible attach targets | mutation still gated per-guest; visibility only |
|
||||
| `reconcile/recover.go:112,172` | destroy-by-existence sees more | already hardened by proof-of-launch (F1 fix) — the pattern the A1 fix mirrors |
|
||||
| `reconcile/restoretest.go:126` | band scan sees squatters it today can't | *safer* under broad (band-advance was built for the blind case) |
|
||||
| `reconcile/bringup.go:180`, `engine.go:229` | existence checks see more | safer (fewer "already exists" surprises) |
|
||||
| `cmd/felhom-agent/main.go:1753` (selftest read) | prints more | none |
|
||||
| *(all, via T3b)* | a mis-scoped token yields `200 []`, not an error | "no guests" and "no visibility" are indistinguishable — for hub collect this is a monitoring blind spot worth a future capability-probe check |
|
||||
|
||||
Only the **stale-lock reaper** both acts destructively AND relies solely on enumeration scope —
|
||||
which is why A1 is the one that needs the cross-check; the reconcile paths already carry their own
|
||||
fail-safes.
|
||||
|
||||
## 4. NOT validated by this spike
|
||||
|
||||
- The winning mechanism end-to-end **in the agent** (the `Pool()` client call, the intersect, the
|
||||
capability-probe surfacing) — that is the A1 impl task, gated on this doc.
|
||||
- The live exploit itself (T6): needs a broad-token agent redeploy; deliberately not performed.
|
||||
The unit tests in Q5 are the designated coverage.
|
||||
- Multi-node cluster semantics of `GET /pools/felhom` under the scoped token (single-node box;
|
||||
pools are cluster-wide, expected identical — same caveat as the pool spike §8).
|
||||
- Whether `pveum role modify` on a live box propagates `Pool.Audit` to already-issued tickets
|
||||
instantly (T3 used a fresh token; PVE ACL changes are normally immediate, and the reaper runs at
|
||||
agent startup, so staleness risk is nil in practice).
|
||||
|
||||
## 5. Appendix — committed-secrets rotation micro-runbook (findings only, NO execution)
|
||||
|
||||
Unrelated to A1; folded in per the task's idle-time clause. `manifests/felhom.secret.yaml` still
|
||||
commits live `stringData` for three Secrets in `felhom-system` (verified 2026-07-03; values not
|
||||
reproduced here):
|
||||
|
||||
| Secret | Committed keys (values in git) | Deployed consumer |
|
||||
|---|---|---|
|
||||
| `healthchecks-config` | `SECRET_KEY`, `SUPERUSER_{EMAIL,PASSWORD}`, SMTP settings (`EMAIL_HOST_PASSWORD` already de-gitted → out-of-band `resend-api`) | **none** — healthchecks not deployed |
|
||||
| `umami-config` | `APP_SECRET`, `POSTGRES_PASSWORD` | **`umami.yaml`** (3 `secretKeyRef`s) — live |
|
||||
| `gitea-creds` | `username` (admin), `password` | **`hub.yaml`** (2 `secretKeyRef`s) — live |
|
||||
|
||||
Operator action list, per secret (pattern = the Resend de-git, `documentation/runbooks/secrets.md`):
|
||||
1. **Rotate the upstream value first** — git history retains every old value, so de-git alone fixes
|
||||
nothing; rotation is the real remediation. (`gitea-creds`: change the Gitea admin password;
|
||||
`umami-config`: new `APP_SECRET` + rotate the CNPG role password; `healthchecks-config`: nothing
|
||||
live to rotate, but the committed superuser password should be considered burned.)
|
||||
2. Create the Secret out-of-band with the NEW values:
|
||||
`kubectl create secret generic <name> -n felhom-system --from-literal=KEY=... [...]`.
|
||||
3. Delete that Secret's block from `manifests/felhom.secret.yaml`, commit, ArgoCD sync (pruning off
|
||||
for Secrets — verify the out-of-band object survives the sync), then rollout-restart the consumer.
|
||||
4. Order: consumers keep reading the mounted/env value until restart, so do 1-2 before 3.
|
||||
|
||||
Execution is out of scope for this spike — recorded as an operator follow-up.
|
||||
|
||||
## 6. Cleanup confirmation
|
||||
|
||||
Verified live after teardown: **0** ACL entries, **0** roles, **0** users matching the probe
|
||||
identity (`pveum acl list` / `role list` / `user list` filtered — all empty). The throwaway
|
||||
`felhom-probe@pve` user, its `probe` token, and the `FelhomProbeRead` role were removed in-session;
|
||||
no throwaway guest was ever created (real non-pool guests satisfied T1). The production token's
|
||||
secret was read on-host into a shell variable only, never echoed, never persisted anywhere
|
||||
(recorded as length=36). Production roles/ACL/token/agent untouched; live agent still running
|
||||
against guest 9201.
|
||||
Reference in New Issue
Block a user