docs: SPIKE — A1 pool-membership read for the stale-lock reaper

Live-probed on felhom-pve under the PRODUCTION scoped token vs root:
enumeration IS pool-filtered (T1: token sees [9201] of 4 guests); pool read
403s naming Pool.Audit (T2); Pool.Audit@/pool/felhom alone suffices (T3,
throwaway identity, torn down); /cluster/resources withholds the pool field
without Pool.Audit (T7); local ownership records all partial (T5).

Recommendation for the A1 impl spec (now unblocked): Pool.Audit added to
FelhomAgentGuest + GET /pools/felhom cross-check in staleLockController,
fail-safe skip on read failure. Appendix: committed-secrets rotation
micro-runbook (operator follow-up).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
2026-07-03 13:00:00 +02:00
parent 9282d60f96
commit 5e3dfcb714
4 changed files with 305 additions and 97 deletions
+7
View File
@@ -3,6 +3,13 @@
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
- **2026-07-03 — SPIKE A1 (pool-membership read) COMPLETE** —
`documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. **The A1 impl spec is now
unblocked** and must cite it: winner = `Pool.Audit` added to `FelhomAgentGuest`
(`felhom-host-install.sh` L183) + `GET /pools/felhom` cross-check in
`staleLockController.Guests()` (fail-safe skip). Enumeration proven pool-filtered under the
production token; fix is defense-in-depth for the broad-token precondition. Appendix: operator
follow-up to rotate + de-git the 3 committed secrets in `manifests/felhom.secret.yaml`.
- **2026-07-03 — hub v0.31.0 LIVE** (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy;
live probe: critical event stored + operator email delivered).
- **2026-07-03 — Felhom skills exist**: `skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}`,
+34 -97
View File
@@ -4,106 +4,43 @@
---
# REPORT — hub v0.31.0 critical-severity fix + Felhom skills + CLAUDE.md refresh (consolidated)
## SPIKE — A1 pool-membership read for the stale-lock reaper (2026-07-03)
**Date:** 2026-07-03 · **Class:** implementation (hub v0.31.0, deployed) + docs/skills across all repos
**Class:** SPIKE (docs-only; no production code, no felhom-agent commit, no live ACL/agent change).
**Deliverable:** `documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`.
## 1. Baselines & commits
Empirically answered the three open A1 mechanism questions on live felhom-pve, probing under the
**production** scoped token `felhom-agent@pve!agent` (secret read on-host, length-only recorded)
vs root as oracle:
All four repos matched the task's expected baselines (pulled clean).
- **T1 (crux):** `GET /nodes/demo-felhom/lxc` under the scoped token returns **[9201] only**; root
sees [9001, 9100, 9201, 9999]. LXC enumeration IS pool-filtered — the reaper's scan is already
ownership-scoped *as deployed*, so A1 is latent, re-armed only by a broad-token redeploy.
- **T2:** `GET /pools/felhom` under the token → **403 "Permission check failed (/pool/felhom,
Pool.Audit)"**. `GET /pools` → 200 `[]` (silently filtered, not an error).
- **T3:** throwaway `felhom-probe@pve!probe` with ONLY `Pool.Audit`@`/pool/felhom` → **200 with
members** — minimal priv delta proven. Bonus (T3b): a token with no VM.Audit gets **200 + empty
list** from the LXC index (mis-scope is silent, not an error). Full teardown verified (0 residue).
- **T4:** root `pvesh get /pools/felhom` works but would be a 4th `proxmox.Privileged` fence
exception — rejected.
- **T5:** local ownership records (guest-binds.json, reconcile journal, guests/ dir, installer
state.json) are all **partial** — journal's newest vmid (9310) no longer exists; adopted or
drive-less guests are absent — rejected as reaper gate.
- **T6:** live foreign-guest reap deliberately NOT run (needs broad-token redeploy); deferred to
the named unit test. **T7 (extra):** `/cluster/resources` withholds the `pool` field without
`Pool.Audit` — no zero-ACL-change API path exists.
| Repo | Baseline | Commits pushed to `main` |
|---|---|---|
| felhom.eu | `d331eb26d1` | `b5f0050` (fix+tests+REUSE), `4a18306` (manifest→0.31.0), `ad61e96` (render test), + the docs/skills commit carrying this REPORT |
| felhom-controller | `eda33400da` | `a518023` (CLAUDE.md refresh) |
| felhom-agent | `2a024b6039` | `3e760a5` (CLAUDE.md refresh) |
| app-catalog-felhom.eu | `c5a3d1b15b` | `15cdf7b` (CLAUDE.md expansion) |
**Recommendation (for the now-unblocked A1 impl spec):** add `Pool.Audit` to `PVE_PRIVS_GUEST`
(`scripts/felhom-host-install.sh` L183, granted at `/pool/felhom`) + intersect `ListLXC` with
`GET /pools/felhom` members inside `staleLockController.Guests()`; a pool-read failure propagates
to the existing "guest list unavailable — skipping recovery" fail-safe. Ship agent + `--rescope-acl`
together; optionally surface pool-read in the capability probe. Named tests:
`TestStaleLock_ForeignGuestNotReaped`, `TestStaleLock_PoolGuestStillReaped`,
`TestStaleLock_PoolReadFails_SkipsAll`.
## 2. Part 1 — hub v0.31.0 (the critical-severity fix)
**Appendix folded in:** committed-secrets rotation micro-runbook for the 3 live `stringData` blocks
in `manifests/felhom.secret.yaml` (healthchecks-config / umami-config / gitea-creds) — rotate
upstream first, out-of-band `kubectl create secret`, then de-git; operator follow-up, no execution.
**Change:** `handleEvent` severity switch gains `"critical"` (unknown values still coerce to `"info"`,
exact-match lowercase); `severityLabels["critical"] = "Kritikus hiba"` added; dashboard consumer
gains `EventCriticals`; critical badge renders FIRST in dashboard + customer-page count chains;
`style.css` defines the previously-referenced-but-undefined `.severity-critical` (`--crit` tokens)
and `.severity-ok` (neutral). Frozen legacy `/notify` path and `allowedEventTypes` untouched.
**Tests (new `hub/internal/api/event_test.go` — first tests on /event):**
- A: critical preserved to store — **companion red-proof run:** with the one-line fix reverted the
test FAILED with `stored severity = "info", want "critical"`; fix restored → PASS.
- B: severity `"banana"` → stored `"info"`. C: unknown event_type → 400 + store empty. Auth: 401.
- D: `TestTemplates_DashboardCriticalBadge` (render_test.go) asserts the rendered HTML contains the
`severity-critical` count badge ordered before the error badge. **Method note:** UI verification
used the Go render test — Chrome verification stopped at the operator login (password entry is
out of CC's scope).
- Green gate: `go build ./... && go vet ./... && go test ./...` — all green.
**Deploy (GitOps):** image `felhom-hub:0.31.0` built+pushed on 180; `manifests/hub.yaml` bumped;
ArgoCD app `felhom` hard-refresh + sync → `Synced`/`Healthy`, rollout complete, pod image
`...felhom-hub:0.31.0`, clean startup logs.
**Live probe (real pipeline):** POSTed one `event_type:"test"`, `severity:"critical"` event from
guest 9201 using its own controller.yaml credentials (key never echoed — length-only). Result:
HTTP 200; hub log `Event from demo-felhom: test (critical)` (stored severity critical, not info);
dispatcher sent the operator email (`Test email sent to nagyfenyvesi.viktor@gmail.com`).
**REUSE.md** §1 gotcha + §3 trap row updated in the same commit as the fix;
`reuse_refs_check.py``OK [felhom.eu]: 31 cited paths, all exist`.
## 3. Part 2 — skills
Three skills authored in `felhom.eu/skills/` (each SKILL.md ≤160 lines, pushy trigger descriptions):
`felhom-build-deploy`, `felhom-ui-design`, `felhom-testing`. Content verified against live
source/infra before writing (see §5). New `scripts/install_skills.py`: idempotent, junction-first
(`mklink /J`) with copy fallback + re-run reminder.
**Install mode: JUNCTION** — created for all three; second run detected them and no-op'd.
**Discovery (Scenario E):** fresh `claude -p` session listed all three:
```
felhom-build-deploy
felhom-testing
felhom-ui-design
```
Bonus trigger probe: "which skill to deploy a new hub version?" → answered `felhom-build-deploy`
with the correct first command (`./build.sh <VER> --push` on 180).
## 4. Part 3 — CLAUDE.md refresh
Standing rule adopted everywhere: **CLAUDE.md carries no version-pinned current state** (that lives
in CONTEXT/CHANGELOG/REUSE); deep runbooks live in the skills, CLAUDE.md keeps a summary table + pointer.
| File | Diffstat | Headlines |
|---|---|---|
| felhom-controller/CLAUDE.md | +119/294 (338→~160 lines) | full 30-package map (was 7); stale `/opt/docker` demo deploy replaced with verified 9201 bootstrap flow; hub build section deleted; patterns pruned to 6 invariants |
| felhom-agent/CLAUDE.md | +86/71 | "Current: v0.31.0" narrative + slice history deleted (30 versions stale); layout completed (+8 packages, opsign); deploy facts re-verified on felhom-pve |
| felhom.eu/CLAUDE.md | +73/58 | "Hub — current state (v0.7.x)" (23 versions stale) → version-free architecture section; GitOps rules kept incl. live-image-lag note; skills section added |
| app-catalog CLAUDE.md | +26/5 | stub → ~30-line proper file (purpose, deploy contract, pointers) |
| E:\git\CLAUDE.md (local) | +8 | Skills section only |
## 5. Commands/facts re-verified against live infra (stale findings)
- Guest 9201 deploy mechanism confirmed live (`/etc/felhom-controller-image` = 0.98.3 tag,
`felhom-controller-bootstrap.service` enabled, container healthy) — the OLD controller CLAUDE.md
still documented the pre-re-platform `/opt/docker` compose deploy for node 1: **stale, replaced**.
- Agent service confirmed on felhom-pve: `User=felhom-agent`,
`ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json`, live 0.61.0 — the
OLD agent CLAUDE.md claimed v0.31.0 current: **stale, removed**.
- Build dirs on 180 confirmed: `~/build/{felhom-agent,felhom-controller,felhom-hub}` + `~/git`
clones; Go present.
- Design tokens confirmed byte-identical across website site.css / hub style.css / controller
style.css `:root` blocks (values inlined in the ui-design skill).
- Hub GitOps flow re-verified by executing it (this deploy). Red-proof citation
(`dispatcher_test.go` ~L2749) verified in source.
- Website displays no hub version anywhere (checked) — no website change needed.
## 6. Observations (noticed, NOT acted on)
- The hub severity badge classes predating this task (`.severity-error/-warning/-info`) use legacy
raw hexes, not the v2 `--warn/--crit` tokens — left alone per the no-restyle rule; a token
migration is a candidate D-series follow-up.
- The hub login page appears unstyled relative to the v2 design system (default-blue button, white
background) — pre-existing, out of scope.
- `manifests/felhom.secret.yaml` inline credentials (from the REUSE report) remain an open operator
follow-up.
- The controller currently emits no `critical`-severity events; the fix future-proofs the channel
(hub-internal `host_disk_critical`/`storage_fill_critical` already flow at critical severity —
they bypass ingest).
**Cleanup:** probe user/token/role removed and verified absent; no throwaway guest needed (three
real non-pool guests existed); production roles/token/agent untouched; secrets never persisted.
@@ -0,0 +1,251 @@
# SPIKE — A1 pool-membership read for the stale-lock reaper (2026-07-03)
> **STATUS: COMPLETE — verdict below.** This spike unblocks the A1 implementation spec (audit
> `AUDIT-blast-radius-hostroot-localapi-2026-07-02.md` §A, finding A1). No production code shipped;
> all probes were reads except a fully-torn-down throwaway user/token/role (T3). The production
> `FelhomAgent{Guest,Store,Base}` roles, the `felhom-agent@pve!agent` token, and the live agent were
> **never modified**.
**Class:** SPIKE (empirical validation; no product code). **Repos read:** felhom-agent @ `3e760a559a`
(v0.61.0, read-only), felhom.eu @ `9282d60f96`. **Host:** felhom-pve (192.168.0.162, node
`demo-felhom`, PVE 9.2.2, single-node). **Identities probed:** production scoped token
`felhom-agent@pve!agent` (secret read out-of-band from `/etc/felhom-agent/agent.json`, length 36,
never persisted) vs `root@pam` (`pvesh`) as ground-truth oracle.
**Verdict (one line):** LXC enumeration under the production scoped token **is already
pool-filtered** (T1) — so A1 is a latent, not live, defect today — but the containment lives
entirely in the ACL; the recommended defense-in-depth fix is **add `Pool.Audit` to the
`FelhomAgentGuest` role** (proven minimal, T2+T3) **+ a `GET /pools/felhom` ownership cross-check
in `staleLockController.Guests()`**, failing safe (skip recovery) when the pool read fails.
---
## 1. Confirmed baseline facts (live, 2026-07-03)
- Box population (root `pct list`): **9001** `spike-lxc` (stopped), **9100** `felhom-golden`
(stopped), **9201** `demo-felhom` (running), **9999** `felhom-selftest-scratch` (stopped).
(`/cluster/resources` additionally shows QEMU VM 9000 `spike-vm`, irrelevant to `pct`/LXC paths.)
- `felhom` pool membership (root `pvesh get /pools/felhom`): **exactly one member, 9201**. So the
T1 precondition (≥1 non-pool guest) was satisfied by three real non-pool guests — no throwaway
guest needed.
- Production ACL: 3 roles per `felhom-host-install.sh`
`PVE_PRIVS_GUEST="VM.Allocate VM.Audit VM.Config.* VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback
VM.Backup Pool.Allocate"` @ `/pool/felhom`, `PVE_PRIVS_STORE="Datastore.Allocate
Datastore.AllocateSpace"` @ each storage, `PVE_PRIVS_BASE="Sys.Audit SDN.Use Datastore.Audit"`
@ `/` (script L183-185). **`Pool.Audit` is absent from all three** — confirmed in-script and by
T2's live 403.
- Reaper under test: `RecoverStaleLockedGuests` (`felhom-agent/internal/localapi/stalelock.go:58`)
`staleLockController.Guests()` (`:159`) → `ListLXC` = `GET /nodes/{node}/lxc`
(`internal/proxmox/query.go:33`). `proxmox.Guest` (`types.go:65`) has **no pool field** — and T7
proves the API response itself carries none, so this is a PVE-API limitation, not a decoding gap.
## 2. Probe matrix — results (verbatim statuses/bodies; token secret redacted throughout)
All token calls: `curl -sk -H "Authorization: PVEAPIToken=felhom-agent@pve!agent=<REDACTED len=36>"
https://127.0.0.1:8006/api2/json/...` run on the host itself.
### T1 — enumeration scope (THE crux, Q2) — **pool-filtered: YES**
| Identity | `GET /nodes/demo-felhom/lxc` | vmid set returned |
|---|---|---|
| production scoped token | **HTTP 200** | **[9201]** |
| root (`pvesh`) | 200 | [9001, 9100, 9201, 9999] |
The scoped token's enumeration returns **only the pool member**. The reaper's scan is therefore
already ownership-scoped *as deployed today* — the three non-pool guests are invisible to it.
PVE filters the LXC index by per-guest `VM.Audit` visibility (the `/pool/felhom` grant resolves
only for pool members — same mechanism as the pool spike's T7 mutation 403s).
### T2 — token pool read (Q1) — **403, missing priv named exactly**
`GET /pools/felhom` under the production token →
```
HTTP 403
{"data":null,"message":"Permission check failed (/pool/felhom, Pool.Audit)\n"}
```
Companion: `GET /pools` (index) under the token → **HTTP 200, `{"data":[]}`** — the pool *index*
does not error; it silently filters to empty. (PVE read-index convention: no visibility ⇒ empty
list, not 403 — see also T3b.)
### T3 — priv delta, empirically confirmed on a THROWAWAY identity — **`Pool.Audit` @ `/pool/felhom` is sufficient AND minimal**
Setup (all torn down, §6): user `felhom-probe@pve` + privsep token `felhom-probe@pve!probe` + role
`FelhomProbeRead` with **only** `Pool.Audit`, granted at `/pool/felhom` to BOTH user and token
(privsep intersection). Production roles/token untouched.
| Probe | Result |
|---|---|
| T3a `GET /pools/felhom` under probe token | **HTTP 200**`poolid: felhom, members: [9201]` |
| T3b `GET /nodes/demo-felhom/lxc` under probe token (no `VM.Audit`) | **HTTP 200, `[]`** — filtered-empty, NOT 403 |
T3a proves the exact delta: one read privilege, at the already-granted path. T3b is a load-bearing
nuance for every `ListLXC` consumer: **an ACL mis-scope does not error — it silently returns an
empty guest list.** For the reaper that direction is fail-safe (nothing to reap); for monitoring
consumers it is a blind spot (§3 Q6).
### T4 — fenced-root alternative — works, but adds root-CLI surface
`pvesh get /pools/felhom --output-format json` as root → members `[9201]` (§1). The installer
already uses this (`felhom-host-install.sh` `pool_members()`, L354). Wiring it into the *agent's*
reaper, however, means a new root-CLI call in `proxmox.Privileged` — a **4th exception** to the
deliberately-3-exception fence (keyctl create / USB mount / SMART), for a value the API can supply
with a one-priv grant. Rejected on that basis (§3 Q3).
### T5 — local ownership records — **all partial; none is a membership registry**
Inventory of `/var/lib/felhom-agent/` + `/var/lib/felhom-install/` on the live host:
| Record | Content found live | Why it can't gate the reaper |
|---|---|---|
| `guest-binds.json` (`GuestBindStore.Guests()`) | `{9201: [3 drive uuids]}` | only guests with **enrolled drives**; a managed guest with no drive would be skipped |
| `journal.log` (reconcile op journal) | newest entries: `bring-up-9310-1` (vmid **9310**) | op *history*, not membership — **9310 no longer exists on the box**; adopted (not agent-created) guests never appear |
| `guests/` state dir | `9201` | only agent-provisioned guests; an installer-`--adopt-pool`-ed guest is absent |
| `/var/lib/felhom-install/state.json` | `{completed: [preflight … verify]}` | step ledger only; **no member list** |
A local allowlist would fail toward *skipping owned guests* (fail-safe direction, but a silent
functional regression: their stale locks would never heal — the exact customer-down scenario the
reaper exists for). Rejected as the primary mechanism (§3 Q3).
### T6 — destructive negative confirmation — **NOT RUN (as pre-agreed)**
Reproducing the audit's exploit live (reaper unlocks a foreign stale-locked guest) requires the
reaper to *see* the foreign guest — i.e. redeploying the agent with a broad token, which this spike
forbids. **Unprovable live under the scoped token — and T1 is the proof of why** (the foreign guest
is invisible). Covered by the named unit tests instead (§3 Q5).
### T7 — (extra) `/cluster/resources` shortcut — **no free membership; `pool` field is priv-gated**
| Identity | `GET /cluster/resources?type=vm` | 9201 `pool` field |
|---|---|---|
| production scoped token | HTTP 200, 1 entry (9201 only) | **`null`** |
| root | 200, 5 entries | `"felhom"` |
Even for a guest the token CAN see, PVE **withholds the `pool` attribute without `Pool.Audit`**.
So there is no zero-ACL-change API path to membership — any pool-aware refactor needs the same
`Pool.Audit` grant. Also confirmed at root: `GET /nodes/{node}/lxc` entries carry **no pool key at
all** (PVE 9.2.2), matching the `Guest` struct.
## 3. The six required answers (task §8)
**Q1 — Is enumeration pool-filtered under the scoped token?** **Yes** (T1: token sees `[9201]`,
root sees `[9001,9100,9201,9999]`). The reaper's scan is already ownership-scoped *by the ACL*.
Corollary (T3b): under a token with no guest visibility the same call returns `200 []`, silently.
**Q2 — Can the token read `/pools/felhom`?** **No** — 403, body names the exact missing priv:
`Permission check failed (/pool/felhom, Pool.Audit)`. T3a proves `Pool.Audit` alone at
`/pool/felhom` flips it to 200-with-members.
**Q3 — Recommended mechanism:** **add `Pool.Audit` to `FelhomAgentGuest` + a `GET /pools/felhom`
cross-check inside `staleLockController.Guests()`** (intersect `ListLXC` with pool members; any
pool-read error propagates → the existing `RecoverStaleLockedGuests` error path already skips the
whole recovery, fail-safe).
| Mechanism | ACL surface added | root-CLI surface added | New failure modes | Blast-radius delta | Verdict |
|---|---|---|---|---|---|
| (a) test-only (trust T1's filtering) | none | none | none | containment lives ONLY in the ACL; the audit's precondition (broad-token redeploy) silently re-arms A1 | **rejected as sole fix** — but T1 means the fix is pure defense-in-depth, zero behavior change today |
| (b) `Pool.Audit` @ `/pool/felhom` (Guest role) + API cross-check | **one read priv, at an already-granted scoped path** (NOT `/` — a `/` grant would expose all pools) | none | pool-read failure ⇒ reaper skips all (fail-safe; loud log); mixed-version window: new agent + old ACL ⇒ recovery disabled until `--rescope-acl` | forecloses the foreign-guest reap even under a broad token | **WINNER** |
| (c) fenced-root `pvesh get /pools/felhom` | none | **4th exception** to the 3-exception `proxmox.Privileged` fence | root output parsing; fence erosion precedent | same as (b) | rejected — fence cost > one scoped read priv |
| (d) local ownership allowlist (T5) | none | none | provably incomplete records ⇒ silently skips owned guests (adopted / drive-less) — the reaper's purpose defeated for them | worse than (b): a "protected" guest stays customer-down | rejected |
Why (b) wins: it is the only option that is complete (covers adopted + drive-less guests), scoped
(the priv lands at `/pool/felhom`, readable-membership only), fence-preserving, and whose failure
mode degrades to the reaper's existing documented fail-safe ("can't confirm ⇒ don't act",
mirroring `reconcile/recover.go:38-52`).
**Q4 — Exact delta for the winner** (for the A1 impl spec to lift):
- *Installer* (`felhom.eu/scripts/felhom-host-install.sh` L183): `PVE_PRIVS_GUEST` gains
`Pool.Audit` (12 → 13 privs). `_ensure_role` already `role modify`s to the exact set, so
`--rescope-acl` re-application upgrades an existing box idempotently; `remove_scoped_acl` needs
no change (it deletes by role name). Script version bump.
- *Agent* (`felhom-agent`): new read `Client.Pool(ctx, name)``GET /pools/{name}` (decode
`members[].vmid`; `internal/proxmox/query.go` — read-only, API-side, keeps the fence).
`staleLockAPI` (`stalelock.go:140`) gains it; `staleLockController.Guests()` (`:159`) becomes:
`ListLXC` ∩ pool members, returning an error when the pool read fails — the existing
`RecoverStaleLockedGuests` guard (`:62-66`, "guest list unavailable — skipping recovery")
then fail-safes the whole scan with the already-present loud log. The filter sits **before** the
`recoverOne` loop, exactly mirroring recover.go's pre-mutation gate. Pool name should come from
the shared default (`reconcile.DefaultPool`, `bringup.go:50`) threaded via config, not a second
literal.
- *Deploy coupling*: ship agent + re-run `--rescope-acl` together (same discipline as
sudoers-with-binary). Optional hardening: add a pool-read check to the capability probe
(`internal/capability/`) so an old-ACL box reports the degradation instead of silently
not-reaping.
**Q5 — Test shapes** (felhom-testing doctrine: negative + non-trivial positive):
- `TestStaleLock_ForeignGuestNotReaped` — fake `staleLockAPI` whose `ListLXC` returns a
broad-token-shaped list `{9201 (pool), 5000 (foreign)}`, pool members `{9201}`; guest 5000 seeded
with `lock: snapshot-delete` + a `vzdump` snapshot + no running vzdump task (the audit's exact
exploit precondition). Assert `Unlock`/`DeleteSnapshot`/`Start` are **never** invoked for 5000.
- `TestStaleLock_PoolGuestStillReaped` — companion positive: 9201 seeded identically **is**
unlocked + snapshot-deleted + started (proves the filter isn't trivially "reap nothing").
- `TestStaleLock_PoolReadFails_SkipsAll` — pool read errors ⇒ zero mutations on ANY guest
(fail-safe path, red-proofable by breaking the intersect).
**Q6 — Blast radius of the T1 answer on the other `ListLXC` consumers.** T1/T3b generalize: every
consumer receives a *pool-filtered* list today, and would receive an *unfiltered* one under a broad
token — none of them checks ownership itself:
| Consumer | Under broad token | Risk |
|---|---|---|
| `hub/collect.go:204` (host report) | foreign guests reported to hub | inventory leak to operator plane; low, but silently assumes pool-only |
| `localapi/disks.go:1055` (guest list for attach) | foreign guests become visible attach targets | mutation still gated per-guest; visibility only |
| `reconcile/recover.go:112,172` | destroy-by-existence sees more | already hardened by proof-of-launch (F1 fix) — the pattern the A1 fix mirrors |
| `reconcile/restoretest.go:126` | band scan sees squatters it today can't | *safer* under broad (band-advance was built for the blind case) |
| `reconcile/bringup.go:180`, `engine.go:229` | existence checks see more | safer (fewer "already exists" surprises) |
| `cmd/felhom-agent/main.go:1753` (selftest read) | prints more | none |
| *(all, via T3b)* | a mis-scoped token yields `200 []`, not an error | "no guests" and "no visibility" are indistinguishable — for hub collect this is a monitoring blind spot worth a future capability-probe check |
Only the **stale-lock reaper** both acts destructively AND relies solely on enumeration scope —
which is why A1 is the one that needs the cross-check; the reconcile paths already carry their own
fail-safes.
## 4. NOT validated by this spike
- The winning mechanism end-to-end **in the agent** (the `Pool()` client call, the intersect, the
capability-probe surfacing) — that is the A1 impl task, gated on this doc.
- The live exploit itself (T6): needs a broad-token agent redeploy; deliberately not performed.
The unit tests in Q5 are the designated coverage.
- Multi-node cluster semantics of `GET /pools/felhom` under the scoped token (single-node box;
pools are cluster-wide, expected identical — same caveat as the pool spike §8).
- Whether `pveum role modify` on a live box propagates `Pool.Audit` to already-issued tickets
instantly (T3 used a fresh token; PVE ACL changes are normally immediate, and the reaper runs at
agent startup, so staleness risk is nil in practice).
## 5. Appendix — committed-secrets rotation micro-runbook (findings only, NO execution)
Unrelated to A1; folded in per the task's idle-time clause. `manifests/felhom.secret.yaml` still
commits live `stringData` for three Secrets in `felhom-system` (verified 2026-07-03; values not
reproduced here):
| Secret | Committed keys (values in git) | Deployed consumer |
|---|---|---|
| `healthchecks-config` | `SECRET_KEY`, `SUPERUSER_{EMAIL,PASSWORD}`, SMTP settings (`EMAIL_HOST_PASSWORD` already de-gitted → out-of-band `resend-api`) | **none** — healthchecks not deployed |
| `umami-config` | `APP_SECRET`, `POSTGRES_PASSWORD` | **`umami.yaml`** (3 `secretKeyRef`s) — live |
| `gitea-creds` | `username` (admin), `password` | **`hub.yaml`** (2 `secretKeyRef`s) — live |
Operator action list, per secret (pattern = the Resend de-git, `documentation/runbooks/secrets.md`):
1. **Rotate the upstream value first** — git history retains every old value, so de-git alone fixes
nothing; rotation is the real remediation. (`gitea-creds`: change the Gitea admin password;
`umami-config`: new `APP_SECRET` + rotate the CNPG role password; `healthchecks-config`: nothing
live to rotate, but the committed superuser password should be considered burned.)
2. Create the Secret out-of-band with the NEW values:
`kubectl create secret generic <name> -n felhom-system --from-literal=KEY=... [...]`.
3. Delete that Secret's block from `manifests/felhom.secret.yaml`, commit, ArgoCD sync (pruning off
for Secrets — verify the out-of-band object survives the sync), then rollout-restart the consumer.
4. Order: consumers keep reading the mounted/env value until restart, so do 1-2 before 3.
Execution is out of scope for this spike — recorded as an operator follow-up.
## 6. Cleanup confirmation
Verified live after teardown: **0** ACL entries, **0** roles, **0** users matching the probe
identity (`pveum acl list` / `role list` / `user list` filtered — all empty). The throwaway
`felhom-probe@pve` user, its `probe` token, and the `FelhomProbeRead` role were removed in-session;
no throwaway guest was ever created (real non-pool guests satisfied T1). The production token's
secret was read on-host into a shell variable only, never echoed, never persisted anywhere
(recorded as length=36). Production roles/ACL/token/agent untouched; live agent still running
against guest 9201.
+13
View File
@@ -1,5 +1,18 @@
# Felhom scripts — Changelog
## docs — SPIKE: A1 pool-membership read for the stale-lock reaper (2026-07-03)
Findings doc `documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. Live-probed on
felhom-pve under the PRODUCTION scoped token vs root: LXC enumeration IS already pool-filtered
(token sees only 9201 of 4 guests); `GET /pools/felhom` 403s naming `Pool.Audit`; a throwaway
token with ONLY `Pool.Audit`@`/pool/felhom` reads members (minimal delta proven, fully torn down);
`/cluster/resources` withholds the `pool` field without `Pool.Audit`; local ownership records are
all partial. Recommendation for the A1 impl spec: add `Pool.Audit` to `PVE_PRIVS_GUEST` in
`felhom-host-install.sh` (L183) + a `GET /pools/felhom` cross-check in the agent's
`staleLockController.Guests()`, fail-safe skip on read failure. No script/agent change in this
commit — docs only. Appendix: committed-secrets (felhom.secret.yaml) rotation micro-runbook,
operator follow-up.
## install_skills.py — new: Claude Code skills installer (2026-07-03)
Installs `skills/*/SKILL.md` (felhom-build-deploy, felhom-ui-design, felhom-testing) into