From 5e3dfcb71418f785d6694a9ecc959b568ffcb9d7 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 3 Jul 2026 13:00:00 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20SPIKE=20=E2=80=94=20A1=20pool-membershi?= =?UTF-8?q?p=20read=20for=20the=20stale-lock=20reaper?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live-probed on felhom-pve under the PRODUCTION scoped token vs root: enumeration IS pool-filtered (T1: token sees [9201] of 4 guests); pool read 403s naming Pool.Audit (T2); Pool.Audit@/pool/felhom alone suffices (T3, throwaway identity, torn down); /cluster/resources withholds the pool field without Pool.Audit (T7); local ownership records all partial (T5). Recommendation for the A1 impl spec (now unblocked): Pool.Audit added to FelhomAgentGuest + GET /pools/felhom cross-check in staleLockController, fail-safe skip on read failure. Appendix: committed-secrets rotation micro-runbook (operator follow-up). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- CONTEXT.md | 7 + REPORT.md | 131 +++------ ...PIKE-a1-pool-membership-read-2026-07-03.md | 251 ++++++++++++++++++ scripts/CHANGELOG.md | 13 + 4 files changed, 305 insertions(+), 97 deletions(-) create mode 100644 documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md diff --git a/CONTEXT.md b/CONTEXT.md index 25f2657..884299b 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,13 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-03 — SPIKE A1 (pool-membership read) COMPLETE** — + `documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. **The A1 impl spec is now + unblocked** and must cite it: winner = `Pool.Audit` added to `FelhomAgentGuest` + (`felhom-host-install.sh` L183) + `GET /pools/felhom` cross-check in + `staleLockController.Guests()` (fail-safe skip). Enumeration proven pool-filtered under the + production token; fix is defense-in-depth for the broad-token precondition. Appendix: operator + follow-up to rotate + de-git the 3 committed secrets in `manifests/felhom.secret.yaml`. - **2026-07-03 — hub v0.31.0 LIVE** (critical-severity ingest fix + UI badges; ArgoCD Synced/Healthy; live probe: critical event stored + operator email delivered). - **2026-07-03 — Felhom skills exist**: `skills/{felhom-build-deploy,felhom-ui-design,felhom-testing}`, diff --git a/REPORT.md b/REPORT.md index 0731e2a..0b1e9bc 100644 --- a/REPORT.md +++ b/REPORT.md @@ -4,106 +4,43 @@ --- -# REPORT — hub v0.31.0 critical-severity fix + Felhom skills + CLAUDE.md refresh (consolidated) +## SPIKE — A1 pool-membership read for the stale-lock reaper (2026-07-03) -**Date:** 2026-07-03 · **Class:** implementation (hub v0.31.0, deployed) + docs/skills across all repos +**Class:** SPIKE (docs-only; no production code, no felhom-agent commit, no live ACL/agent change). +**Deliverable:** `documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. -## 1. Baselines & commits +Empirically answered the three open A1 mechanism questions on live felhom-pve, probing under the +**production** scoped token `felhom-agent@pve!agent` (secret read on-host, length-only recorded) +vs root as oracle: -All four repos matched the task's expected baselines (pulled clean). +- **T1 (crux):** `GET /nodes/demo-felhom/lxc` under the scoped token returns **[9201] only**; root + sees [9001, 9100, 9201, 9999]. LXC enumeration IS pool-filtered — the reaper's scan is already + ownership-scoped *as deployed*, so A1 is latent, re-armed only by a broad-token redeploy. +- **T2:** `GET /pools/felhom` under the token → **403 "Permission check failed (/pool/felhom, + Pool.Audit)"**. `GET /pools` → 200 `[]` (silently filtered, not an error). +- **T3:** throwaway `felhom-probe@pve!probe` with ONLY `Pool.Audit`@`/pool/felhom` → **200 with + members** — minimal priv delta proven. Bonus (T3b): a token with no VM.Audit gets **200 + empty + list** from the LXC index (mis-scope is silent, not an error). Full teardown verified (0 residue). +- **T4:** root `pvesh get /pools/felhom` works but would be a 4th `proxmox.Privileged` fence + exception — rejected. +- **T5:** local ownership records (guest-binds.json, reconcile journal, guests/ dir, installer + state.json) are all **partial** — journal's newest vmid (9310) no longer exists; adopted or + drive-less guests are absent — rejected as reaper gate. +- **T6:** live foreign-guest reap deliberately NOT run (needs broad-token redeploy); deferred to + the named unit test. **T7 (extra):** `/cluster/resources` withholds the `pool` field without + `Pool.Audit` — no zero-ACL-change API path exists. -| Repo | Baseline | Commits pushed to `main` | -|---|---|---| -| felhom.eu | `d331eb26d1` | `b5f0050` (fix+tests+REUSE), `4a18306` (manifest→0.31.0), `ad61e96` (render test), + the docs/skills commit carrying this REPORT | -| felhom-controller | `eda33400da` | `a518023` (CLAUDE.md refresh) | -| felhom-agent | `2a024b6039` | `3e760a5` (CLAUDE.md refresh) | -| app-catalog-felhom.eu | `c5a3d1b15b` | `15cdf7b` (CLAUDE.md expansion) | +**Recommendation (for the now-unblocked A1 impl spec):** add `Pool.Audit` to `PVE_PRIVS_GUEST` +(`scripts/felhom-host-install.sh` L183, granted at `/pool/felhom`) + intersect `ListLXC` with +`GET /pools/felhom` members inside `staleLockController.Guests()`; a pool-read failure propagates +to the existing "guest list unavailable — skipping recovery" fail-safe. Ship agent + `--rescope-acl` +together; optionally surface pool-read in the capability probe. Named tests: +`TestStaleLock_ForeignGuestNotReaped`, `TestStaleLock_PoolGuestStillReaped`, +`TestStaleLock_PoolReadFails_SkipsAll`. -## 2. Part 1 — hub v0.31.0 (the critical-severity fix) +**Appendix folded in:** committed-secrets rotation micro-runbook for the 3 live `stringData` blocks +in `manifests/felhom.secret.yaml` (healthchecks-config / umami-config / gitea-creds) — rotate +upstream first, out-of-band `kubectl create secret`, then de-git; operator follow-up, no execution. -**Change:** `handleEvent` severity switch gains `"critical"` (unknown values still coerce to `"info"`, -exact-match lowercase); `severityLabels["critical"] = "Kritikus hiba"` added; dashboard consumer -gains `EventCriticals`; critical badge renders FIRST in dashboard + customer-page count chains; -`style.css` defines the previously-referenced-but-undefined `.severity-critical` (`--crit` tokens) -and `.severity-ok` (neutral). Frozen legacy `/notify` path and `allowedEventTypes` untouched. - -**Tests (new `hub/internal/api/event_test.go` — first tests on /event):** -- A: critical preserved to store — **companion red-proof run:** with the one-line fix reverted the - test FAILED with `stored severity = "info", want "critical"`; fix restored → PASS. -- B: severity `"banana"` → stored `"info"`. C: unknown event_type → 400 + store empty. Auth: 401. -- D: `TestTemplates_DashboardCriticalBadge` (render_test.go) asserts the rendered HTML contains the - `severity-critical` count badge ordered before the error badge. **Method note:** UI verification - used the Go render test — Chrome verification stopped at the operator login (password entry is - out of CC's scope). -- Green gate: `go build ./... && go vet ./... && go test ./...` — all green. - -**Deploy (GitOps):** image `felhom-hub:0.31.0` built+pushed on 180; `manifests/hub.yaml` bumped; -ArgoCD app `felhom` hard-refresh + sync → `Synced`/`Healthy`, rollout complete, pod image -`...felhom-hub:0.31.0`, clean startup logs. - -**Live probe (real pipeline):** POSTed one `event_type:"test"`, `severity:"critical"` event from -guest 9201 using its own controller.yaml credentials (key never echoed — length-only). Result: -HTTP 200; hub log `Event from demo-felhom: test (critical)` (stored severity critical, not info); -dispatcher sent the operator email (`Test email sent to nagyfenyvesi.viktor@gmail.com`). - -**REUSE.md** §1 gotcha + §3 trap row updated in the same commit as the fix; -`reuse_refs_check.py` → `OK [felhom.eu]: 31 cited paths, all exist`. - -## 3. Part 2 — skills - -Three skills authored in `felhom.eu/skills/` (each SKILL.md ≤160 lines, pushy trigger descriptions): -`felhom-build-deploy`, `felhom-ui-design`, `felhom-testing`. Content verified against live -source/infra before writing (see §5). New `scripts/install_skills.py`: idempotent, junction-first -(`mklink /J`) with copy fallback + re-run reminder. - -**Install mode: JUNCTION** — created for all three; second run detected them and no-op'd. -**Discovery (Scenario E):** fresh `claude -p` session listed all three: -``` -felhom-build-deploy -felhom-testing -felhom-ui-design -``` -Bonus trigger probe: "which skill to deploy a new hub version?" → answered `felhom-build-deploy` -with the correct first command (`./build.sh --push` on 180). - -## 4. Part 3 — CLAUDE.md refresh - -Standing rule adopted everywhere: **CLAUDE.md carries no version-pinned current state** (that lives -in CONTEXT/CHANGELOG/REUSE); deep runbooks live in the skills, CLAUDE.md keeps a summary table + pointer. - -| File | Diffstat | Headlines | -|---|---|---| -| felhom-controller/CLAUDE.md | +119/−294 (338→~160 lines) | full 30-package map (was 7); stale `/opt/docker` demo deploy replaced with verified 9201 bootstrap flow; hub build section deleted; patterns pruned to 6 invariants | -| felhom-agent/CLAUDE.md | +86/−71 | "Current: v0.31.0" narrative + slice history deleted (30 versions stale); layout completed (+8 packages, opsign); deploy facts re-verified on felhom-pve | -| felhom.eu/CLAUDE.md | +73/−58 | "Hub — current state (v0.7.x)" (23 versions stale) → version-free architecture section; GitOps rules kept incl. live-image-lag note; skills section added | -| app-catalog CLAUDE.md | +26/−5 | stub → ~30-line proper file (purpose, deploy contract, pointers) | -| E:\git\CLAUDE.md (local) | +8 | Skills section only | - -## 5. Commands/facts re-verified against live infra (stale findings) - -- Guest 9201 deploy mechanism confirmed live (`/etc/felhom-controller-image` = 0.98.3 tag, - `felhom-controller-bootstrap.service` enabled, container healthy) — the OLD controller CLAUDE.md - still documented the pre-re-platform `/opt/docker` compose deploy for node 1: **stale, replaced**. -- Agent service confirmed on felhom-pve: `User=felhom-agent`, - `ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json`, live 0.61.0 — the - OLD agent CLAUDE.md claimed v0.31.0 current: **stale, removed**. -- Build dirs on 180 confirmed: `~/build/{felhom-agent,felhom-controller,felhom-hub}` + `~/git` - clones; Go present. -- Design tokens confirmed byte-identical across website site.css / hub style.css / controller - style.css `:root` blocks (values inlined in the ui-design skill). -- Hub GitOps flow re-verified by executing it (this deploy). Red-proof citation - (`dispatcher_test.go` ~L27–49) verified in source. -- Website displays no hub version anywhere (checked) — no website change needed. - -## 6. Observations (noticed, NOT acted on) - -- The hub severity badge classes predating this task (`.severity-error/-warning/-info`) use legacy - raw hexes, not the v2 `--warn/--crit` tokens — left alone per the no-restyle rule; a token - migration is a candidate D-series follow-up. -- The hub login page appears unstyled relative to the v2 design system (default-blue button, white - background) — pre-existing, out of scope. -- `manifests/felhom.secret.yaml` inline credentials (from the REUSE report) remain an open operator - follow-up. -- The controller currently emits no `critical`-severity events; the fix future-proofs the channel - (hub-internal `host_disk_critical`/`storage_fill_critical` already flow at critical severity — - they bypass ingest). +**Cleanup:** probe user/token/role removed and verified absent; no throwaway guest needed (three +real non-pool guests existed); production roles/token/agent untouched; secrets never persisted. diff --git a/documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md b/documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md new file mode 100644 index 0000000..afde60a --- /dev/null +++ b/documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md @@ -0,0 +1,251 @@ +# SPIKE — A1 pool-membership read for the stale-lock reaper (2026-07-03) + +> **STATUS: COMPLETE — verdict below.** This spike unblocks the A1 implementation spec (audit +> `AUDIT-blast-radius-hostroot-localapi-2026-07-02.md` §A, finding A1). No production code shipped; +> all probes were reads except a fully-torn-down throwaway user/token/role (T3). The production +> `FelhomAgent{Guest,Store,Base}` roles, the `felhom-agent@pve!agent` token, and the live agent were +> **never modified**. + +**Class:** SPIKE (empirical validation; no product code). **Repos read:** felhom-agent @ `3e760a559a` +(v0.61.0, read-only), felhom.eu @ `9282d60f96`. **Host:** felhom-pve (192.168.0.162, node +`demo-felhom`, PVE 9.2.2, single-node). **Identities probed:** production scoped token +`felhom-agent@pve!agent` (secret read out-of-band from `/etc/felhom-agent/agent.json`, length 36, +never persisted) vs `root@pam` (`pvesh`) as ground-truth oracle. + +**Verdict (one line):** LXC enumeration under the production scoped token **is already +pool-filtered** (T1) — so A1 is a latent, not live, defect today — but the containment lives +entirely in the ACL; the recommended defense-in-depth fix is **add `Pool.Audit` to the +`FelhomAgentGuest` role** (proven minimal, T2+T3) **+ a `GET /pools/felhom` ownership cross-check +in `staleLockController.Guests()`**, failing safe (skip recovery) when the pool read fails. + +--- + +## 1. Confirmed baseline facts (live, 2026-07-03) + +- Box population (root `pct list`): **9001** `spike-lxc` (stopped), **9100** `felhom-golden` + (stopped), **9201** `demo-felhom` (running), **9999** `felhom-selftest-scratch` (stopped). + (`/cluster/resources` additionally shows QEMU VM 9000 `spike-vm`, irrelevant to `pct`/LXC paths.) +- `felhom` pool membership (root `pvesh get /pools/felhom`): **exactly one member, 9201**. So the + T1 precondition (≥1 non-pool guest) was satisfied by three real non-pool guests — no throwaway + guest needed. +- Production ACL: 3 roles per `felhom-host-install.sh` — + `PVE_PRIVS_GUEST="VM.Allocate VM.Audit VM.Config.* VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback + VM.Backup Pool.Allocate"` @ `/pool/felhom`, `PVE_PRIVS_STORE="Datastore.Allocate + Datastore.AllocateSpace"` @ each storage, `PVE_PRIVS_BASE="Sys.Audit SDN.Use Datastore.Audit"` + @ `/` (script L183-185). **`Pool.Audit` is absent from all three** — confirmed in-script and by + T2's live 403. +- Reaper under test: `RecoverStaleLockedGuests` (`felhom-agent/internal/localapi/stalelock.go:58`) + → `staleLockController.Guests()` (`:159`) → `ListLXC` = `GET /nodes/{node}/lxc` + (`internal/proxmox/query.go:33`). `proxmox.Guest` (`types.go:65`) has **no pool field** — and T7 + proves the API response itself carries none, so this is a PVE-API limitation, not a decoding gap. + +## 2. Probe matrix — results (verbatim statuses/bodies; token secret redacted throughout) + +All token calls: `curl -sk -H "Authorization: PVEAPIToken=felhom-agent@pve!agent=" +https://127.0.0.1:8006/api2/json/...` run on the host itself. + +### T1 — enumeration scope (THE crux, Q2) — **pool-filtered: YES** + +| Identity | `GET /nodes/demo-felhom/lxc` | vmid set returned | +|---|---|---| +| production scoped token | **HTTP 200** | **[9201]** | +| root (`pvesh`) | 200 | [9001, 9100, 9201, 9999] | + +The scoped token's enumeration returns **only the pool member**. The reaper's scan is therefore +already ownership-scoped *as deployed today* — the three non-pool guests are invisible to it. +PVE filters the LXC index by per-guest `VM.Audit` visibility (the `/pool/felhom` grant resolves +only for pool members — same mechanism as the pool spike's T7 mutation 403s). + +### T2 — token pool read (Q1) — **403, missing priv named exactly** + +`GET /pools/felhom` under the production token → + +``` +HTTP 403 +{"data":null,"message":"Permission check failed (/pool/felhom, Pool.Audit)\n"} +``` + +Companion: `GET /pools` (index) under the token → **HTTP 200, `{"data":[]}`** — the pool *index* +does not error; it silently filters to empty. (PVE read-index convention: no visibility ⇒ empty +list, not 403 — see also T3b.) + +### T3 — priv delta, empirically confirmed on a THROWAWAY identity — **`Pool.Audit` @ `/pool/felhom` is sufficient AND minimal** + +Setup (all torn down, §6): user `felhom-probe@pve` + privsep token `felhom-probe@pve!probe` + role +`FelhomProbeRead` with **only** `Pool.Audit`, granted at `/pool/felhom` to BOTH user and token +(privsep intersection). Production roles/token untouched. + +| Probe | Result | +|---|---| +| T3a `GET /pools/felhom` under probe token | **HTTP 200** — `poolid: felhom, members: [9201]` | +| T3b `GET /nodes/demo-felhom/lxc` under probe token (no `VM.Audit`) | **HTTP 200, `[]`** — filtered-empty, NOT 403 | + +T3a proves the exact delta: one read privilege, at the already-granted path. T3b is a load-bearing +nuance for every `ListLXC` consumer: **an ACL mis-scope does not error — it silently returns an +empty guest list.** For the reaper that direction is fail-safe (nothing to reap); for monitoring +consumers it is a blind spot (§3 Q6). + +### T4 — fenced-root alternative — works, but adds root-CLI surface + +`pvesh get /pools/felhom --output-format json` as root → members `[9201]` (§1). The installer +already uses this (`felhom-host-install.sh` `pool_members()`, L354). Wiring it into the *agent's* +reaper, however, means a new root-CLI call in `proxmox.Privileged` — a **4th exception** to the +deliberately-3-exception fence (keyctl create / USB mount / SMART), for a value the API can supply +with a one-priv grant. Rejected on that basis (§3 Q3). + +### T5 — local ownership records — **all partial; none is a membership registry** + +Inventory of `/var/lib/felhom-agent/` + `/var/lib/felhom-install/` on the live host: + +| Record | Content found live | Why it can't gate the reaper | +|---|---|---| +| `guest-binds.json` (`GuestBindStore.Guests()`) | `{9201: [3 drive uuids]}` | only guests with **enrolled drives**; a managed guest with no drive would be skipped | +| `journal.log` (reconcile op journal) | newest entries: `bring-up-9310-1` (vmid **9310**) | op *history*, not membership — **9310 no longer exists on the box**; adopted (not agent-created) guests never appear | +| `guests/` state dir | `9201` | only agent-provisioned guests; an installer-`--adopt-pool`-ed guest is absent | +| `/var/lib/felhom-install/state.json` | `{completed: [preflight … verify]}` | step ledger only; **no member list** | + +A local allowlist would fail toward *skipping owned guests* (fail-safe direction, but a silent +functional regression: their stale locks would never heal — the exact customer-down scenario the +reaper exists for). Rejected as the primary mechanism (§3 Q3). + +### T6 — destructive negative confirmation — **NOT RUN (as pre-agreed)** + +Reproducing the audit's exploit live (reaper unlocks a foreign stale-locked guest) requires the +reaper to *see* the foreign guest — i.e. redeploying the agent with a broad token, which this spike +forbids. **Unprovable live under the scoped token — and T1 is the proof of why** (the foreign guest +is invisible). Covered by the named unit tests instead (§3 Q5). + +### T7 — (extra) `/cluster/resources` shortcut — **no free membership; `pool` field is priv-gated** + +| Identity | `GET /cluster/resources?type=vm` | 9201 `pool` field | +|---|---|---| +| production scoped token | HTTP 200, 1 entry (9201 only) | **`null`** | +| root | 200, 5 entries | `"felhom"` | + +Even for a guest the token CAN see, PVE **withholds the `pool` attribute without `Pool.Audit`**. +So there is no zero-ACL-change API path to membership — any pool-aware refactor needs the same +`Pool.Audit` grant. Also confirmed at root: `GET /nodes/{node}/lxc` entries carry **no pool key at +all** (PVE 9.2.2), matching the `Guest` struct. + +## 3. The six required answers (task §8) + +**Q1 — Is enumeration pool-filtered under the scoped token?** **Yes** (T1: token sees `[9201]`, +root sees `[9001,9100,9201,9999]`). The reaper's scan is already ownership-scoped *by the ACL*. +Corollary (T3b): under a token with no guest visibility the same call returns `200 []`, silently. + +**Q2 — Can the token read `/pools/felhom`?** **No** — 403, body names the exact missing priv: +`Permission check failed (/pool/felhom, Pool.Audit)`. T3a proves `Pool.Audit` alone at +`/pool/felhom` flips it to 200-with-members. + +**Q3 — Recommended mechanism:** **add `Pool.Audit` to `FelhomAgentGuest` + a `GET /pools/felhom` +cross-check inside `staleLockController.Guests()`** (intersect `ListLXC` with pool members; any +pool-read error propagates → the existing `RecoverStaleLockedGuests` error path already skips the +whole recovery, fail-safe). + +| Mechanism | ACL surface added | root-CLI surface added | New failure modes | Blast-radius delta | Verdict | +|---|---|---|---|---|---| +| (a) test-only (trust T1's filtering) | none | none | none | containment lives ONLY in the ACL; the audit's precondition (broad-token redeploy) silently re-arms A1 | **rejected as sole fix** — but T1 means the fix is pure defense-in-depth, zero behavior change today | +| (b) `Pool.Audit` @ `/pool/felhom` (Guest role) + API cross-check | **one read priv, at an already-granted scoped path** (NOT `/` — a `/` grant would expose all pools) | none | pool-read failure ⇒ reaper skips all (fail-safe; loud log); mixed-version window: new agent + old ACL ⇒ recovery disabled until `--rescope-acl` | forecloses the foreign-guest reap even under a broad token | **WINNER** | +| (c) fenced-root `pvesh get /pools/felhom` | none | **4th exception** to the 3-exception `proxmox.Privileged` fence | root output parsing; fence erosion precedent | same as (b) | rejected — fence cost > one scoped read priv | +| (d) local ownership allowlist (T5) | none | none | provably incomplete records ⇒ silently skips owned guests (adopted / drive-less) — the reaper's purpose defeated for them | worse than (b): a "protected" guest stays customer-down | rejected | + +Why (b) wins: it is the only option that is complete (covers adopted + drive-less guests), scoped +(the priv lands at `/pool/felhom`, readable-membership only), fence-preserving, and whose failure +mode degrades to the reaper's existing documented fail-safe ("can't confirm ⇒ don't act", +mirroring `reconcile/recover.go:38-52`). + +**Q4 — Exact delta for the winner** (for the A1 impl spec to lift): + +- *Installer* (`felhom.eu/scripts/felhom-host-install.sh` L183): `PVE_PRIVS_GUEST` gains + `Pool.Audit` (12 → 13 privs). `_ensure_role` already `role modify`s to the exact set, so + `--rescope-acl` re-application upgrades an existing box idempotently; `remove_scoped_acl` needs + no change (it deletes by role name). Script version bump. +- *Agent* (`felhom-agent`): new read `Client.Pool(ctx, name)` → `GET /pools/{name}` (decode + `members[].vmid`; `internal/proxmox/query.go` — read-only, API-side, keeps the fence). + `staleLockAPI` (`stalelock.go:140`) gains it; `staleLockController.Guests()` (`:159`) becomes: + `ListLXC` ∩ pool members, returning an error when the pool read fails — the existing + `RecoverStaleLockedGuests` guard (`:62-66`, "guest list unavailable — skipping recovery") + then fail-safes the whole scan with the already-present loud log. The filter sits **before** the + `recoverOne` loop, exactly mirroring recover.go's pre-mutation gate. Pool name should come from + the shared default (`reconcile.DefaultPool`, `bringup.go:50`) threaded via config, not a second + literal. +- *Deploy coupling*: ship agent + re-run `--rescope-acl` together (same discipline as + sudoers-with-binary). Optional hardening: add a pool-read check to the capability probe + (`internal/capability/`) so an old-ACL box reports the degradation instead of silently + not-reaping. + +**Q5 — Test shapes** (felhom-testing doctrine: negative + non-trivial positive): + +- `TestStaleLock_ForeignGuestNotReaped` — fake `staleLockAPI` whose `ListLXC` returns a + broad-token-shaped list `{9201 (pool), 5000 (foreign)}`, pool members `{9201}`; guest 5000 seeded + with `lock: snapshot-delete` + a `vzdump` snapshot + no running vzdump task (the audit's exact + exploit precondition). Assert `Unlock`/`DeleteSnapshot`/`Start` are **never** invoked for 5000. +- `TestStaleLock_PoolGuestStillReaped` — companion positive: 9201 seeded identically **is** + unlocked + snapshot-deleted + started (proves the filter isn't trivially "reap nothing"). +- `TestStaleLock_PoolReadFails_SkipsAll` — pool read errors ⇒ zero mutations on ANY guest + (fail-safe path, red-proofable by breaking the intersect). + +**Q6 — Blast radius of the T1 answer on the other `ListLXC` consumers.** T1/T3b generalize: every +consumer receives a *pool-filtered* list today, and would receive an *unfiltered* one under a broad +token — none of them checks ownership itself: + +| Consumer | Under broad token | Risk | +|---|---|---| +| `hub/collect.go:204` (host report) | foreign guests reported to hub | inventory leak to operator plane; low, but silently assumes pool-only | +| `localapi/disks.go:1055` (guest list for attach) | foreign guests become visible attach targets | mutation still gated per-guest; visibility only | +| `reconcile/recover.go:112,172` | destroy-by-existence sees more | already hardened by proof-of-launch (F1 fix) — the pattern the A1 fix mirrors | +| `reconcile/restoretest.go:126` | band scan sees squatters it today can't | *safer* under broad (band-advance was built for the blind case) | +| `reconcile/bringup.go:180`, `engine.go:229` | existence checks see more | safer (fewer "already exists" surprises) | +| `cmd/felhom-agent/main.go:1753` (selftest read) | prints more | none | +| *(all, via T3b)* | a mis-scoped token yields `200 []`, not an error | "no guests" and "no visibility" are indistinguishable — for hub collect this is a monitoring blind spot worth a future capability-probe check | + +Only the **stale-lock reaper** both acts destructively AND relies solely on enumeration scope — +which is why A1 is the one that needs the cross-check; the reconcile paths already carry their own +fail-safes. + +## 4. NOT validated by this spike + +- The winning mechanism end-to-end **in the agent** (the `Pool()` client call, the intersect, the + capability-probe surfacing) — that is the A1 impl task, gated on this doc. +- The live exploit itself (T6): needs a broad-token agent redeploy; deliberately not performed. + The unit tests in Q5 are the designated coverage. +- Multi-node cluster semantics of `GET /pools/felhom` under the scoped token (single-node box; + pools are cluster-wide, expected identical — same caveat as the pool spike §8). +- Whether `pveum role modify` on a live box propagates `Pool.Audit` to already-issued tickets + instantly (T3 used a fresh token; PVE ACL changes are normally immediate, and the reaper runs at + agent startup, so staleness risk is nil in practice). + +## 5. Appendix — committed-secrets rotation micro-runbook (findings only, NO execution) + +Unrelated to A1; folded in per the task's idle-time clause. `manifests/felhom.secret.yaml` still +commits live `stringData` for three Secrets in `felhom-system` (verified 2026-07-03; values not +reproduced here): + +| Secret | Committed keys (values in git) | Deployed consumer | +|---|---|---| +| `healthchecks-config` | `SECRET_KEY`, `SUPERUSER_{EMAIL,PASSWORD}`, SMTP settings (`EMAIL_HOST_PASSWORD` already de-gitted → out-of-band `resend-api`) | **none** — healthchecks not deployed | +| `umami-config` | `APP_SECRET`, `POSTGRES_PASSWORD` | **`umami.yaml`** (3 `secretKeyRef`s) — live | +| `gitea-creds` | `username` (admin), `password` | **`hub.yaml`** (2 `secretKeyRef`s) — live | + +Operator action list, per secret (pattern = the Resend de-git, `documentation/runbooks/secrets.md`): +1. **Rotate the upstream value first** — git history retains every old value, so de-git alone fixes + nothing; rotation is the real remediation. (`gitea-creds`: change the Gitea admin password; + `umami-config`: new `APP_SECRET` + rotate the CNPG role password; `healthchecks-config`: nothing + live to rotate, but the committed superuser password should be considered burned.) +2. Create the Secret out-of-band with the NEW values: + `kubectl create secret generic -n felhom-system --from-literal=KEY=... [...]`. +3. Delete that Secret's block from `manifests/felhom.secret.yaml`, commit, ArgoCD sync (pruning off + for Secrets — verify the out-of-band object survives the sync), then rollout-restart the consumer. +4. Order: consumers keep reading the mounted/env value until restart, so do 1-2 before 3. + +Execution is out of scope for this spike — recorded as an operator follow-up. + +## 6. Cleanup confirmation + +Verified live after teardown: **0** ACL entries, **0** roles, **0** users matching the probe +identity (`pveum acl list` / `role list` / `user list` filtered — all empty). The throwaway +`felhom-probe@pve` user, its `probe` token, and the `FelhomProbeRead` role were removed in-session; +no throwaway guest was ever created (real non-pool guests satisfied T1). The production token's +secret was read on-host into a shell variable only, never echoed, never persisted anywhere +(recorded as length=36). Production roles/ACL/token/agent untouched; live agent still running +against guest 9201. diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index de8b5a6..fdf30ca 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,5 +1,18 @@ # Felhom scripts — Changelog +## docs — SPIKE: A1 pool-membership read for the stale-lock reaper (2026-07-03) + +Findings doc `documentation/audits/SPIKE-a1-pool-membership-read-2026-07-03.md`. Live-probed on +felhom-pve under the PRODUCTION scoped token vs root: LXC enumeration IS already pool-filtered +(token sees only 9201 of 4 guests); `GET /pools/felhom` 403s naming `Pool.Audit`; a throwaway +token with ONLY `Pool.Audit`@`/pool/felhom` reads members (minimal delta proven, fully torn down); +`/cluster/resources` withholds the `pool` field without `Pool.Audit`; local ownership records are +all partial. Recommendation for the A1 impl spec: add `Pool.Audit` to `PVE_PRIVS_GUEST` in +`felhom-host-install.sh` (L183) + a `GET /pools/felhom` cross-check in the agent's +`staleLockController.Guests()`, fail-safe skip on read failure. No script/agent change in this +commit — docs only. Appendix: committed-secrets (felhom.secret.yaml) rotation micro-runbook, +operator follow-up. + ## install_skills.py — new: Claude Code skills installer (2026-07-03) Installs `skills/*/SKILL.md` (felhom-build-deploy, felhom-ui-design, felhom-testing) into