docs: SPIKE — A1 pool-membership read for the stale-lock reaper
Live-probed on felhom-pve under the PRODUCTION scoped token vs root: enumeration IS pool-filtered (T1: token sees [9201] of 4 guests); pool read 403s naming Pool.Audit (T2); Pool.Audit@/pool/felhom alone suffices (T3, throwaway identity, torn down); /cluster/resources withholds the pool field without Pool.Audit (T7); local ownership records all partial (T5). Recommendation for the A1 impl spec (now unblocked): Pool.Audit added to FelhomAgentGuest + GET /pools/felhom cross-check in staleLockController, fail-safe skip on read failure. Appendix: committed-secrets rotation micro-runbook (operator follow-up). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -3,6 +3,13 @@
|
||||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||||
|
||||
- **2026-07-03 — SPIKE A1 (pool-membership read) COMPLETE** —
|
||||
`documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. **The A1 impl spec is now
|
||||
unblocked** and must cite it: winner = `Pool.Audit` added to `FelhomAgentGuest`
|
||||
(`felhom-host-install.sh` L183) + `GET /pools/felhom` cross-check in
|
||||
`staleLockController.Guests()` (fail-safe skip). Enumeration proven pool-filtered under the
|
||||
production token; fix is defense-in-depth for the broad-token precondition. Appendix: operator
|
||||
follow-up to rotate + de-git the 3 committed secrets in `manifests/felhom.secret.yaml`.
|
||||
- **2026-07-03 — hub v0.31.0 LIVE** (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy;
|
||||
live probe: critical event stored + operator email delivered).
|
||||
- **2026-07-03 — Felhom skills exist**: `skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}`,
|
||||
|
||||
@@ -4,106 +4,43 @@
|
||||
|
||||
---
|
||||
|
||||
# REPORT — hub v0.31.0 critical-severity fix + Felhom skills + CLAUDE.md refresh (consolidated)
|
||||
## SPIKE — A1 pool-membership read for the stale-lock reaper (2026-07-03)
|
||||
|
||||
**Date:** 2026-07-03 · **Class:** implementation (hub v0.31.0, deployed) + docs/skills across all repos
|
||||
**Class:** SPIKE (docs-only; no production code, no felhom-agent commit, no live ACL/agent change).
|
||||
**Deliverable:** `documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`.
|
||||
|
||||
## 1. Baselines & commits
|
||||
Empirically answered the three open A1 mechanism questions on live felhom-pve, probing under the
|
||||
**production** scoped token `felhom-agent@pve!agent` (secret read on-host, length-only recorded)
|
||||
vs root as oracle:
|
||||
|
||||
All four repos matched the task's expected baselines (pulled clean).
|
||||
- **T1 (crux):** `GET /nodes/demo-felhom/lxc` under the scoped token returns **[9201] only**; root
|
||||
sees [9001, 9100, 9201, 9999]. LXC enumeration IS pool-filtered — the reaper's scan is already
|
||||
ownership-scoped *as deployed*, so A1 is latent, re-armed only by a broad-token redeploy.
|
||||
- **T2:** `GET /pools/felhom` under the token → **403 "Permission check failed (/pool/felhom,
|
||||
Pool.Audit)"**. `GET /pools` → 200 `[]` (silently filtered, not an error).
|
||||
- **T3:** throwaway `felhom-probe@pve!probe` with ONLY `Pool.Audit`@`/pool/felhom` → **200 with
|
||||
members** — minimal priv delta proven. Bonus (T3b): a token with no VM.Audit gets **200 + empty
|
||||
list** from the LXC index (mis-scope is silent, not an error). Full teardown verified (0 residue).
|
||||
- **T4:** root `pvesh get /pools/felhom` works but would be a 4th `proxmox.Privileged` fence
|
||||
exception — rejected.
|
||||
- **T5:** local ownership records (guest-binds.json, reconcile journal, guests/ dir, installer
|
||||
state.json) are all **partial** — journal's newest vmid (9310) no longer exists; adopted or
|
||||
drive-less guests are absent — rejected as reaper gate.
|
||||
- **T6:** live foreign-guest reap deliberately NOT run (needs broad-token redeploy); deferred to
|
||||
the named unit test. **T7 (extra):** `/cluster/resources` withholds the `pool` field without
|
||||
`Pool.Audit` — no zero-ACL-change API path exists.
|
||||
|
||||
| Repo | Baseline | Commits pushed to `main` |
|
||||
|---|---|---|
|
||||
| felhom.eu | `d331eb26d1` | `b5f0050` (fix+tests+REUSE), `4a18306` (manifest→0.31.0), `ad61e96` (render test), + the docs/skills commit carrying this REPORT |
|
||||
| felhom-controller | `eda33400da` | `a518023` (CLAUDE.md refresh) |
|
||||
| felhom-agent | `2a024b6039` | `3e760a5` (CLAUDE.md refresh) |
|
||||
| app-catalog-felhom.eu | `c5a3d1b15b` | `15cdf7b` (CLAUDE.md expansion) |
|
||||
**Recommendation (for the now-unblocked A1 impl spec):** add `Pool.Audit` to `PVE_PRIVS_GUEST`
|
||||
(`scripts/felhom-host-install.sh` L183, granted at `/pool/felhom`) + intersect `ListLXC` with
|
||||
`GET /pools/felhom` members inside `staleLockController.Guests()`; a pool-read failure propagates
|
||||
to the existing "guest list unavailable — skipping recovery" fail-safe. Ship agent + `--rescope-acl`
|
||||
together; optionally surface pool-read in the capability probe. Named tests:
|
||||
`TestStaleLock_ForeignGuestNotReaped`, `TestStaleLock_PoolGuestStillReaped`,
|
||||
`TestStaleLock_PoolReadFails_SkipsAll`.
|
||||
|
||||
## 2. Part 1 — hub v0.31.0 (the critical-severity fix)
|
||||
**Appendix folded in:** committed-secrets rotation micro-runbook for the 3 live `stringData` blocks
|
||||
in `manifests/felhom.secret.yaml` (healthchecks-config / umami-config / gitea-creds) — rotate
|
||||
upstream first, out-of-band `kubectl create secret`, then de-git; operator follow-up, no execution.
|
||||
|
||||
**Change:** `handleEvent` severity switch gains `"critical"` (unknown values still coerce to `"info"`,
|
||||
exact-match lowercase); `severityLabels["critical"] = "Kritikus hiba"` added; dashboard consumer
|
||||
gains `EventCriticals`; critical badge renders FIRST in dashboard + customer-page count chains;
|
||||
`style.css` defines the previously-referenced-but-undefined `.severity-critical` (`--crit` tokens)
|
||||
and `.severity-ok` (neutral). Frozen legacy `/notify` path and `allowedEventTypes` untouched.
|
||||
|
||||
**Tests (new `hub/internal/api/event_test.go` — first tests on /event):**
|
||||
- A: critical preserved to store — **companion red-proof run:** with the one-line fix reverted the
|
||||
test FAILED with `stored severity = "info", want "critical"`; fix restored → PASS.
|
||||
- B: severity `"banana"` → stored `"info"`. C: unknown event_type → 400 + store empty. Auth: 401.
|
||||
- D: `TestTemplates_DashboardCriticalBadge` (render_test.go) asserts the rendered HTML contains the
|
||||
`severity-critical` count badge ordered before the error badge. **Method note:** UI verification
|
||||
used the Go render test — Chrome verification stopped at the operator login (password entry is
|
||||
out of CC's scope).
|
||||
- Green gate: `go build ./... && go vet ./... && go test ./...` — all green.
|
||||
|
||||
**Deploy (GitOps):** image `felhom-hub:0.31.0` built+pushed on 180; `manifests/hub.yaml` bumped;
|
||||
ArgoCD app `felhom` hard-refresh + sync → `Synced`/`Healthy`, rollout complete, pod image
|
||||
`...felhom-hub:0.31.0`, clean startup logs.
|
||||
|
||||
**Live probe (real pipeline):** POSTed one `event_type:"test"`, `severity:"critical"` event from
|
||||
guest 9201 using its own controller.yaml credentials (key never echoed — length-only). Result:
|
||||
HTTP 200; hub log `Event from demo-felhom: test (critical)` (stored severity critical, not info);
|
||||
dispatcher sent the operator email (`Test email sent to nagyfenyvesi.viktor@gmail.com`).
|
||||
|
||||
**REUSE.md** §1 gotcha + §3 trap row updated in the same commit as the fix;
|
||||
`reuse_refs_check.py` → `OK [felhom.eu]: 31 cited paths, all exist`.
|
||||
|
||||
## 3. Part 2 — skills
|
||||
|
||||
Three skills authored in `felhom.eu/skills/` (each SKILL.md ≤160 lines, pushy trigger descriptions):
|
||||
`felhom-build-deploy`, `felhom-ui-design`, `felhom-testing`. Content verified against live
|
||||
source/infra before writing (see §5). New `scripts/install_skills.py`: idempotent, junction-first
|
||||
(`mklink /J`) with copy fallback + re-run reminder.
|
||||
|
||||
**Install mode: JUNCTION** — created for all three; second run detected them and no-op'd.
|
||||
**Discovery (Scenario E):** fresh `claude -p` session listed all three:
|
||||
```
|
||||
felhom-build-deploy
|
||||
felhom-testing
|
||||
felhom-ui-design
|
||||
```
|
||||
Bonus trigger probe: "which skill to deploy a new hub version?" → answered `felhom-build-deploy`
|
||||
with the correct first command (`./build.sh <VER> --push` on 180).
|
||||
|
||||
## 4. Part 3 — CLAUDE.md refresh
|
||||
|
||||
Standing rule adopted everywhere: **CLAUDE.md carries no version-pinned current state** (that lives
|
||||
in CONTEXT/CHANGELOG/REUSE); deep runbooks live in the skills, CLAUDE.md keeps a summary table + pointer.
|
||||
|
||||
| File | Diffstat | Headlines |
|
||||
|---|---|---|
|
||||
| felhom-controller/CLAUDE.md | +119/−294 (338→~160 lines) | full 30-package map (was 7); stale `/opt/docker` demo deploy replaced with verified 9201 bootstrap flow; hub build section deleted; patterns pruned to 6 invariants |
|
||||
| felhom-agent/CLAUDE.md | +86/−71 | "Current: v0.31.0" narrative + slice history deleted (30 versions stale); layout completed (+8 packages, opsign); deploy facts re-verified on felhom-pve |
|
||||
| felhom.eu/CLAUDE.md | +73/−58 | "Hub — current state (v0.7.x)" (23 versions stale) → version-free architecture section; GitOps rules kept incl. live-image-lag note; skills section added |
|
||||
| app-catalog CLAUDE.md | +26/−5 | stub → ~30-line proper file (purpose, deploy contract, pointers) |
|
||||
| E:\git\CLAUDE.md (local) | +8 | Skills section only |
|
||||
|
||||
## 5. Commands/facts re-verified against live infra (stale findings)
|
||||
|
||||
- Guest 9201 deploy mechanism confirmed live (`/etc/felhom-controller-image` = 0.98.3 tag,
|
||||
`felhom-controller-bootstrap.service` enabled, container healthy) — the OLD controller CLAUDE.md
|
||||
still documented the pre-re-platform `/opt/docker` compose deploy for node 1: **stale, replaced**.
|
||||
- Agent service confirmed on felhom-pve: `User=felhom-agent`,
|
||||
`ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json`, live 0.61.0 — the
|
||||
OLD agent CLAUDE.md claimed v0.31.0 current: **stale, removed**.
|
||||
- Build dirs on 180 confirmed: `~/build/{felhom-agent,felhom-controller,felhom-hub}` + `~/git`
|
||||
clones; Go present.
|
||||
- Design tokens confirmed byte-identical across website site.css / hub style.css / controller
|
||||
style.css `:root` blocks (values inlined in the ui-design skill).
|
||||
- Hub GitOps flow re-verified by executing it (this deploy). Red-proof citation
|
||||
(`dispatcher_test.go` ~L27–49) verified in source.
|
||||
- Website displays no hub version anywhere (checked) — no website change needed.
|
||||
|
||||
## 6. Observations (noticed, NOT acted on)
|
||||
|
||||
- The hub severity badge classes predating this task (`.severity-error/-warning/-info`) use legacy
|
||||
raw hexes, not the v2 `--warn/--crit` tokens — left alone per the no-restyle rule; a token
|
||||
migration is a candidate D-series follow-up.
|
||||
- The hub login page appears unstyled relative to the v2 design system (default-blue button, white
|
||||
background) — pre-existing, out of scope.
|
||||
- `manifests/felhom.secret.yaml` inline credentials (from the REUSE report) remain an open operator
|
||||
follow-up.
|
||||
- The controller currently emits no `critical`-severity events; the fix future-proofs the channel
|
||||
(hub-internal `host_disk_critical`/`storage_fill_critical` already flow at critical severity —
|
||||
they bypass ingest).
|
||||
**Cleanup:** probe user/token/role removed and verified absent; no throwaway guest needed (three
|
||||
real non-pool guests existed); production roles/token/agent untouched; secrets never persisted.
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
# SPIKE — A1 pool-membership read for the stale-lock reaper (2026-07-03)
|
||||
|
||||
> **STATUS: COMPLETE — verdict below.** This spike unblocks the A1 implementation spec (audit
|
||||
> `AUDIT-blast-radius-hostroot-localapi-2026-07-02.md` §A, finding A1). No production code shipped;
|
||||
> all probes were reads except a fully-torn-down throwaway user/token/role (T3). The production
|
||||
> `FelhomAgent{Guest,Store,Base}` roles, the `felhom-agent@pve!agent` token, and the live agent were
|
||||
> **never modified**.
|
||||
|
||||
**Class:** SPIKE (empirical validation; no product code). **Repos read:** felhom-agent @ `3e760a559a`
|
||||
(v0.61.0, read-only), felhom.eu @ `9282d60f96`. **Host:** felhom-pve (192.168.0.162, node
|
||||
`demo-felhom`, PVE 9.2.2, single-node). **Identities probed:** production scoped token
|
||||
`felhom-agent@pve!agent` (secret read out-of-band from `/etc/felhom-agent/agent.json`, length 36,
|
||||
never persisted) vs `root@pam` (`pvesh`) as ground-truth oracle.
|
||||
|
||||
**Verdict (one line):** LXC enumeration under the production scoped token **is already
|
||||
pool-filtered** (T1) — so A1 is a latent, not live, defect today — but the containment lives
|
||||
entirely in the ACL; the recommended defense-in-depth fix is **add `Pool.Audit` to the
|
||||
`FelhomAgentGuest` role** (proven minimal, T2+T3) **+ a `GET /pools/felhom` ownership cross-check
|
||||
in `staleLockController.Guests()`**, failing safe (skip recovery) when the pool read fails.
|
||||
|
||||
---
|
||||
|
||||
## 1. Confirmed baseline facts (live, 2026-07-03)
|
||||
|
||||
- Box population (root `pct list`): **9001** `spike-lxc` (stopped), **9100** `felhom-golden`
|
||||
(stopped), **9201** `demo-felhom` (running), **9999** `felhom-selftest-scratch` (stopped).
|
||||
(`/cluster/resources` additionally shows QEMU VM 9000 `spike-vm`, irrelevant to `pct`/LXC paths.)
|
||||
- `felhom` pool membership (root `pvesh get /pools/felhom`): **exactly one member, 9201**. So the
|
||||
T1 precondition (≥1 non-pool guest) was satisfied by three real non-pool guests — no throwaway
|
||||
guest needed.
|
||||
- Production ACL: 3 roles per `felhom-host-install.sh` —
|
||||
`PVE_PRIVS_GUEST="VM.Allocate VM.Audit VM.Config.* VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback
|
||||
VM.Backup Pool.Allocate"` @ `/pool/felhom`, `PVE_PRIVS_STORE="Datastore.Allocate
|
||||
Datastore.AllocateSpace"` @ each storage, `PVE_PRIVS_BASE="Sys.Audit SDN.Use Datastore.Audit"`
|
||||
@ `/` (script L183-185). **`Pool.Audit` is absent from all three** — confirmed in-script and by
|
||||
T2's live 403.
|
||||
- Reaper under test: `RecoverStaleLockedGuests` (`felhom-agent/internal/localapi/stalelock.go:58`)
|
||||
→ `staleLockController.Guests()` (`:159`) → `ListLXC` = `GET /nodes/{node}/lxc`
|
||||
(`internal/proxmox/query.go:33`). `proxmox.Guest` (`types.go:65`) has **no pool field** — and T7
|
||||
proves the API response itself carries none, so this is a PVE-API limitation, not a decoding gap.
|
||||
|
||||
## 2. Probe matrix — results (verbatim statuses/bodies; token secret redacted throughout)
|
||||
|
||||
All token calls: `curl -sk -H "Authorization: PVEAPIToken=felhom-agent@pve!agent=<REDACTED len=36>"
|
||||
https://127.0.0.1:8006/api2/json/...` run on the host itself.
|
||||
|
||||
### T1 — enumeration scope (THE crux, Q2) — **pool-filtered: YES**
|
||||
|
||||
| Identity | `GET /nodes/demo-felhom/lxc` | vmid set returned |
|
||||
|---|---|---|
|
||||
| production scoped token | **HTTP 200** | **[9201]** |
|
||||
| root (`pvesh`) | 200 | [9001, 9100, 9201, 9999] |
|
||||
|
||||
The scoped token's enumeration returns **only the pool member**. The reaper's scan is therefore
|
||||
already ownership-scoped *as deployed today* — the three non-pool guests are invisible to it.
|
||||
PVE filters the LXC index by per-guest `VM.Audit` visibility (the `/pool/felhom` grant resolves
|
||||
only for pool members — same mechanism as the pool spike's T7 mutation 403s).
|
||||
|
||||
### T2 — token pool read (Q1) — **403, missing priv named exactly**
|
||||
|
||||
`GET /pools/felhom` under the production token →
|
||||
|
||||
```
|
||||
HTTP 403
|
||||
{"data":null,"message":"Permission check failed (/pool/felhom, Pool.Audit)\n"}
|
||||
```
|
||||
|
||||
Companion: `GET /pools` (index) under the token → **HTTP 200, `{"data":[]}`** — the pool *index*
|
||||
does not error; it silently filters to empty. (PVE read-index convention: no visibility ⇒ empty
|
||||
list, not 403 — see also T3b.)
|
||||
|
||||
### T3 — priv delta, empirically confirmed on a THROWAWAY identity — **`Pool.Audit` @ `/pool/felhom` is sufficient AND minimal**
|
||||
|
||||
Setup (all torn down, §6): user `felhom-probe@pve` + privsep token `felhom-probe@pve!probe` + role
|
||||
`FelhomProbeRead` with **only** `Pool.Audit`, granted at `/pool/felhom` to BOTH user and token
|
||||
(privsep intersection). Production roles/token untouched.
|
||||
|
||||
| Probe | Result |
|
||||
|---|---|
|
||||
| T3a `GET /pools/felhom` under probe token | **HTTP 200** — `poolid: felhom, members: [9201]` |
|
||||
| T3b `GET /nodes/demo-felhom/lxc` under probe token (no `VM.Audit`) | **HTTP 200, `[]`** — filtered-empty, NOT 403 |
|
||||
|
||||
T3a proves the exact delta: one read privilege, at the already-granted path. T3b is a load-bearing
|
||||
nuance for every `ListLXC` consumer: **an ACL mis-scope does not error — it silently returns an
|
||||
empty guest list.** For the reaper that direction is fail-safe (nothing to reap); for monitoring
|
||||
consumers it is a blind spot (§3 Q6).
|
||||
|
||||
### T4 — fenced-root alternative — works, but adds root-CLI surface
|
||||
|
||||
`pvesh get /pools/felhom --output-format json` as root → members `[9201]` (§1). The installer
|
||||
already uses this (`felhom-host-install.sh` `pool_members()`, L354). Wiring it into the *agent's*
|
||||
reaper, however, means a new root-CLI call in `proxmox.Privileged` — a **4th exception** to the
|
||||
deliberately-3-exception fence (keyctl create / USB mount / SMART), for a value the API can supply
|
||||
with a one-priv grant. Rejected on that basis (§3 Q3).
|
||||
|
||||
### T5 — local ownership records — **all partial; none is a membership registry**
|
||||
|
||||
Inventory of `/var/lib/felhom-agent/` + `/var/lib/felhom-install/` on the live host:
|
||||
|
||||
| Record | Content found live | Why it can't gate the reaper |
|
||||
|---|---|---|
|
||||
| `guest-binds.json` (`GuestBindStore.Guests()`) | `{9201: [3 drive uuids]}` | only guests with **enrolled drives**; a managed guest with no drive would be skipped |
|
||||
| `journal.log` (reconcile op journal) | newest entries: `bring-up-9310-1` (vmid **9310**) | op *history*, not membership — **9310 no longer exists on the box**; adopted (not agent-created) guests never appear |
|
||||
| `guests/` state dir | `9201` | only agent-provisioned guests; an installer-`--adopt-pool`-ed guest is absent |
|
||||
| `/var/lib/felhom-install/state.json` | `{completed: [preflight … verify]}` | step ledger only; **no member list** |
|
||||
|
||||
A local allowlist would fail toward *skipping owned guests* (fail-safe direction, but a silent
|
||||
functional regression: their stale locks would never heal — the exact customer-down scenario the
|
||||
reaper exists for). Rejected as the primary mechanism (§3 Q3).
|
||||
|
||||
### T6 — destructive negative confirmation — **NOT RUN (as pre-agreed)**
|
||||
|
||||
Reproducing the audit's exploit live (reaper unlocks a foreign stale-locked guest) requires the
|
||||
reaper to *see* the foreign guest — i.e. redeploying the agent with a broad token, which this spike
|
||||
forbids. **Unprovable live under the scoped token — and T1 is the proof of why** (the foreign guest
|
||||
is invisible). Covered by the named unit tests instead (§3 Q5).
|
||||
|
||||
### T7 — (extra) `/cluster/resources` shortcut — **no free membership; `pool` field is priv-gated**
|
||||
|
||||
| Identity | `GET /cluster/resources?type=vm` | 9201 `pool` field |
|
||||
|---|---|---|
|
||||
| production scoped token | HTTP 200, 1 entry (9201 only) | **`null`** |
|
||||
| root | 200, 5 entries | `"felhom"` |
|
||||
|
||||
Even for a guest the token CAN see, PVE **withholds the `pool` attribute without `Pool.Audit`**.
|
||||
So there is no zero-ACL-change API path to membership — any pool-aware refactor needs the same
|
||||
`Pool.Audit` grant. Also confirmed at root: `GET /nodes/{node}/lxc` entries carry **no pool key at
|
||||
all** (PVE 9.2.2), matching the `Guest` struct.
|
||||
|
||||
## 3. The six required answers (task §8)
|
||||
|
||||
**Q1 — Is enumeration pool-filtered under the scoped token?** **Yes** (T1: token sees `[9201]`,
|
||||
root sees `[9001,9100,9201,9999]`). The reaper's scan is already ownership-scoped *by the ACL*.
|
||||
Corollary (T3b): under a token with no guest visibility the same call returns `200 []`, silently.
|
||||
|
||||
**Q2 — Can the token read `/pools/felhom`?** **No** — 403, body names the exact missing priv:
|
||||
`Permission check failed (/pool/felhom, Pool.Audit)`. T3a proves `Pool.Audit` alone at
|
||||
`/pool/felhom` flips it to 200-with-members.
|
||||
|
||||
**Q3 — Recommended mechanism:** **add `Pool.Audit` to `FelhomAgentGuest` + a `GET /pools/felhom`
|
||||
cross-check inside `staleLockController.Guests()`** (intersect `ListLXC` with pool members; any
|
||||
pool-read error propagates → the existing `RecoverStaleLockedGuests` error path already skips the
|
||||
whole recovery, fail-safe).
|
||||
|
||||
| Mechanism | ACL surface added | root-CLI surface added | New failure modes | Blast-radius delta | Verdict |
|
||||
|---|---|---|---|---|---|
|
||||
| (a) test-only (trust T1's filtering) | none | none | none | containment lives ONLY in the ACL; the audit's precondition (broad-token redeploy) silently re-arms A1 | **rejected as sole fix** — but T1 means the fix is pure defense-in-depth, zero behavior change today |
|
||||
| (b) `Pool.Audit` @ `/pool/felhom` (Guest role) + API cross-check | **one read priv, at an already-granted scoped path** (NOT `/` — a `/` grant would expose all pools) | none | pool-read failure ⇒ reaper skips all (fail-safe; loud log); mixed-version window: new agent + old ACL ⇒ recovery disabled until `--rescope-acl` | forecloses the foreign-guest reap even under a broad token | **WINNER** |
|
||||
| (c) fenced-root `pvesh get /pools/felhom` | none | **4th exception** to the 3-exception `proxmox.Privileged` fence | root output parsing; fence erosion precedent | same as (b) | rejected — fence cost > one scoped read priv |
|
||||
| (d) local ownership allowlist (T5) | none | none | provably incomplete records ⇒ silently skips owned guests (adopted / drive-less) — the reaper's purpose defeated for them | worse than (b): a "protected" guest stays customer-down | rejected |
|
||||
|
||||
Why (b) wins: it is the only option that is complete (covers adopted + drive-less guests), scoped
|
||||
(the priv lands at `/pool/felhom`, readable-membership only), fence-preserving, and whose failure
|
||||
mode degrades to the reaper's existing documented fail-safe ("can't confirm ⇒ don't act",
|
||||
mirroring `reconcile/recover.go:38-52`).
|
||||
|
||||
**Q4 — Exact delta for the winner** (for the A1 impl spec to lift):
|
||||
|
||||
- *Installer* (`felhom.eu/scripts/felhom-host-install.sh` L183): `PVE_PRIVS_GUEST` gains
|
||||
`Pool.Audit` (12 → 13 privs). `_ensure_role` already `role modify`s to the exact set, so
|
||||
`--rescope-acl` re-application upgrades an existing box idempotently; `remove_scoped_acl` needs
|
||||
no change (it deletes by role name). Script version bump.
|
||||
- *Agent* (`felhom-agent`): new read `Client.Pool(ctx, name)` → `GET /pools/{name}` (decode
|
||||
`members[].vmid`; `internal/proxmox/query.go` — read-only, API-side, keeps the fence).
|
||||
`staleLockAPI` (`stalelock.go:140`) gains it; `staleLockController.Guests()` (`:159`) becomes:
|
||||
`ListLXC` ∩ pool members, returning an error when the pool read fails — the existing
|
||||
`RecoverStaleLockedGuests` guard (`:62-66`, "guest list unavailable — skipping recovery")
|
||||
then fail-safes the whole scan with the already-present loud log. The filter sits **before** the
|
||||
`recoverOne` loop, exactly mirroring recover.go's pre-mutation gate. Pool name should come from
|
||||
the shared default (`reconcile.DefaultPool`, `bringup.go:50`) threaded via config, not a second
|
||||
literal.
|
||||
- *Deploy coupling*: ship agent + re-run `--rescope-acl` together (same discipline as
|
||||
sudoers-with-binary). Optional hardening: add a pool-read check to the capability probe
|
||||
(`internal/capability/`) so an old-ACL box reports the degradation instead of silently
|
||||
not-reaping.
|
||||
|
||||
**Q5 — Test shapes** (felhom-testing doctrine: negative + non-trivial positive):
|
||||
|
||||
- `TestStaleLock_ForeignGuestNotReaped` — fake `staleLockAPI` whose `ListLXC` returns a
|
||||
broad-token-shaped list `{9201 (pool), 5000 (foreign)}`, pool members `{9201}`; guest 5000 seeded
|
||||
with `lock: snapshot-delete` + a `vzdump` snapshot + no running vzdump task (the audit's exact
|
||||
exploit precondition). Assert `Unlock`/`DeleteSnapshot`/`Start` are **never** invoked for 5000.
|
||||
- `TestStaleLock_PoolGuestStillReaped` — companion positive: 9201 seeded identically **is**
|
||||
unlocked + snapshot-deleted + started (proves the filter isn't trivially "reap nothing").
|
||||
- `TestStaleLock_PoolReadFails_SkipsAll` — pool read errors ⇒ zero mutations on ANY guest
|
||||
(fail-safe path, red-proofable by breaking the intersect).
|
||||
|
||||
**Q6 — Blast radius of the T1 answer on the other `ListLXC` consumers.** T1/T3b generalize: every
|
||||
consumer receives a *pool-filtered* list today, and would receive an *unfiltered* one under a broad
|
||||
token — none of them checks ownership itself:
|
||||
|
||||
| Consumer | Under broad token | Risk |
|
||||
|---|---|---|
|
||||
| `hub/collect.go:204` (host report) | foreign guests reported to hub | inventory leak to operator plane; low, but silently assumes pool-only |
|
||||
| `localapi/disks.go:1055` (guest list for attach) | foreign guests become visible attach targets | mutation still gated per-guest; visibility only |
|
||||
| `reconcile/recover.go:112,172` | destroy-by-existence sees more | already hardened by proof-of-launch (F1 fix) — the pattern the A1 fix mirrors |
|
||||
| `reconcile/restoretest.go:126` | band scan sees squatters it today can't | *safer* under broad (band-advance was built for the blind case) |
|
||||
| `reconcile/bringup.go:180`, `engine.go:229` | existence checks see more | safer (fewer "already exists" surprises) |
|
||||
| `cmd/felhom-agent/main.go:1753` (selftest read) | prints more | none |
|
||||
| *(all, via T3b)* | a mis-scoped token yields `200 []`, not an error | "no guests" and "no visibility" are indistinguishable — for hub collect this is a monitoring blind spot worth a future capability-probe check |
|
||||
|
||||
Only the **stale-lock reaper** both acts destructively AND relies solely on enumeration scope —
|
||||
which is why A1 is the one that needs the cross-check; the reconcile paths already carry their own
|
||||
fail-safes.
|
||||
|
||||
## 4. NOT validated by this spike
|
||||
|
||||
- The winning mechanism end-to-end **in the agent** (the `Pool()` client call, the intersect, the
|
||||
capability-probe surfacing) — that is the A1 impl task, gated on this doc.
|
||||
- The live exploit itself (T6): needs a broad-token agent redeploy; deliberately not performed.
|
||||
The unit tests in Q5 are the designated coverage.
|
||||
- Multi-node cluster semantics of `GET /pools/felhom` under the scoped token (single-node box;
|
||||
pools are cluster-wide, expected identical — same caveat as the pool spike §8).
|
||||
- Whether `pveum role modify` on a live box propagates `Pool.Audit` to already-issued tickets
|
||||
instantly (T3 used a fresh token; PVE ACL changes are normally immediate, and the reaper runs at
|
||||
agent startup, so staleness risk is nil in practice).
|
||||
|
||||
## 5. Appendix — committed-secrets rotation micro-runbook (findings only, NO execution)
|
||||
|
||||
Unrelated to A1; folded in per the task's idle-time clause. `manifests/felhom.secret.yaml` still
|
||||
commits live `stringData` for three Secrets in `felhom-system` (verified 2026-07-03; values not
|
||||
reproduced here):
|
||||
|
||||
| Secret | Committed keys (values in git) | Deployed consumer |
|
||||
|---|---|---|
|
||||
| `healthchecks-config` | `SECRET_KEY`, `SUPERUSER_{EMAIL,PASSWORD}`, SMTP settings (`EMAIL_HOST_PASSWORD` already de-gitted → out-of-band `resend-api`) | **none** — healthchecks not deployed |
|
||||
| `umami-config` | `APP_SECRET`, `POSTGRES_PASSWORD` | **`umami.yaml`** (3 `secretKeyRef`s) — live |
|
||||
| `gitea-creds` | `username` (admin), `password` | **`hub.yaml`** (2 `secretKeyRef`s) — live |
|
||||
|
||||
Operator action list, per secret (pattern = the Resend de-git, `documentation/runbooks/secrets.md`):
|
||||
1. **Rotate the upstream value first** — git history retains every old value, so de-git alone fixes
|
||||
nothing; rotation is the real remediation. (`gitea-creds`: change the Gitea admin password;
|
||||
`umami-config`: new `APP_SECRET` + rotate the CNPG role password; `healthchecks-config`: nothing
|
||||
live to rotate, but the committed superuser password should be considered burned.)
|
||||
2. Create the Secret out-of-band with the NEW values:
|
||||
`kubectl create secret generic <name> -n felhom-system --from-literal=KEY=... [...]`.
|
||||
3. Delete that Secret's block from `manifests/felhom.secret.yaml`, commit, ArgoCD sync (pruning off
|
||||
for Secrets — verify the out-of-band object survives the sync), then rollout-restart the consumer.
|
||||
4. Order: consumers keep reading the mounted/env value until restart, so do 1-2 before 3.
|
||||
|
||||
Execution is out of scope for this spike — recorded as an operator follow-up.
|
||||
|
||||
## 6. Cleanup confirmation
|
||||
|
||||
Verified live after teardown: **0** ACL entries, **0** roles, **0** users matching the probe
|
||||
identity (`pveum acl list` / `role list` / `user list` filtered — all empty). The throwaway
|
||||
`felhom-probe@pve` user, its `probe` token, and the `FelhomProbeRead` role were removed in-session;
|
||||
no throwaway guest was ever created (real non-pool guests satisfied T1). The production token's
|
||||
secret was read on-host into a shell variable only, never echoed, never persisted anywhere
|
||||
(recorded as length=36). Production roles/ACL/token/agent untouched; live agent still running
|
||||
against guest 9201.
|
||||
@@ -1,5 +1,18 @@
|
||||
# Felhom scripts — Changelog
|
||||
|
||||
## docs — SPIKE: A1 pool-membership read for the stale-lock reaper (2026-07-03)
|
||||
|
||||
Findings doc `documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. Live-probed on
|
||||
felhom-pve under the PRODUCTION scoped token vs root: LXC enumeration IS already pool-filtered
|
||||
(token sees only 9201 of 4 guests); `GET /pools/felhom` 403s naming `Pool.Audit`; a throwaway
|
||||
token with ONLY `Pool.Audit`@`/pool/felhom` reads members (minimal delta proven, fully torn down);
|
||||
`/cluster/resources` withholds the `pool` field without `Pool.Audit`; local ownership records are
|
||||
all partial. Recommendation for the A1 impl spec: add `Pool.Audit` to `PVE_PRIVS_GUEST` in
|
||||
`felhom-host-install.sh` (L183) + a `GET /pools/felhom` cross-check in the agent's
|
||||
`staleLockController.Guests()`, fail-safe skip on read failure. No script/agent change in this
|
||||
commit — docs only. Appendix: committed-secrets (felhom.secret.yaml) rotation micro-runbook,
|
||||
operator follow-up.
|
||||
|
||||
## install_skills.py — new: Claude Code skills installer (2026-07-03)
|
||||
|
||||
Installs `skills/*/SKILL.md` (felhom-build-deploy, felhom-ui-design, felhom-testing) into
|
||||
|
||||
Reference in New Issue
Block a user