admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: pool-effects no-mercy campaign — op-matrix 16/16, B1 rollback-destroy finding, fix list (F1-F6)

Browse Source 
Full validation of the felhom pool + scoped-token migration on felhom-pve:
- Track A: complete agent op surface under the scoped token — 16/16 pass
- Track B: vmid origin traced (externally supplied); adversarial collision
  probes prove the compensating rollback/teardown attempts to destroy a
  guest the job never created (contained only by the pool ACL)
- Track C: containment + 3b-fix intact; Track D: no pool leaks, adopt/
  uninstall pool handling correct
- Findings doc: documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md
No code change, no version bump. Demo left as found.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
authentik Default Admin
2026-07-02 09:16:46 +02:00
parent d775fda326
commit e59e3c0ae2
2 changed files with 134 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+15 -45
View File
@@ -4,55 +4,25 @@
---
# REPORT — Impl-1 Part B: install the guarded-mkfs wrapper (host-install v1.8.0)
# REPORT — "no-mercy" campaign: full effects of the `felhom` pool + scoped-token migration
**Date:** 2026-07-01 · **Repo:** `felhom.eu` (`scripts/` only) · **Class:** Risky/supervised (paired with
the destructive-path change in felhom-agent v0.54.0).
**Date:** 2026-07-02 · **Class:** Runbook-style validation (aggressive, throwaway env) · **No code change, no version bump.**
**Findings doc:** [`documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md`](documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md)
Companion to felhom-agent v0.54.0 (the format-safety foundation). Script-only change: host-install now
installs the guarded-mkfs wrapper during agent install so every fresh box gets the safe format path.
Executed the full pool-effects campaign on felhom-pve (agent v0.59.0 live, host-install v1.7.0 scoped ACL): source-traced every `ListLXC` consumer in felhom-agent (read-only), then ran the complete agent→PVE operation matrix under the scoped token on scratch pool guests 9310/9311, adversarial pool-blindness probes with non-pool decoys, blast-radius regression re-confirms, and pool-membership side-effect checks.
## 1. Baseline → target
## Headline results
script `v1.7.0` @ `9e5bbc1` → **v1.8.0**. Paired with agent v0.54.0 (separate repo, `5209830`).
1. **Track A — 16/16 ops PASS, zero surprise 403s.** The scoped token is functionally complete for the agent's real operation surface (incl. the previously untested Destroy+disk-reap, Snapshot/Rollback/Delete, SetConfig Network/CPU/Options, Resize, ListRunningTasks, TaskLogTail, and the full restore-test cycle).
2. **B1 (the campaign's most important result):** the provision vmid is **always externally supplied** (no `nextid` scan in agent OR hub — collision-avoidance is operator-side). Adversarially provisioning onto an existing **non-pool** vmid: the pool-blind duplicate guard misses it, PVE refuses the restore (no clobber), **but the compensating rollback then attempts to DESTROY the innocent guest** — stopped only by the pool ACL (403). Same shape live-proven for the restore-test scratch band (a squatter at 990000 additionally makes the restore-test **permanently FAIL** with false "backup unrestorable" alerts). Safe-but-noisy under the scoped ACL; a genuine destroy-risk under any broad token.
3. **Containment intact (Track C):** non-pool guest ops 403, non-granted-storage writes 403, drive audit readable (3b-fix holds).
4. **Hub inventory is pool-only (B3, by design — now recorded as a decision):** operators must pick vmids from the host's real guest list, never the hub view. No pool-name leak into any customer/controller-facing field (Track D); `--adopt-pool` idempotent; uninstall never deletes a non-empty pool.
## 2. What changed
## Fix list (full detail + severities in the findings doc)
- `scripts/felhom-host-install.sh` `step_agent_install`: new step (before the sudoers install) that
`fetch_raw`s `configs/felhom-mkfs-guarded.sh` from Gitea, `bash -n`-validates it, and
`install -m0755 -o root -g root` → `/usr/local/sbin/felhom-mkfs-guarded`. Wrapper-before-sudoers so
the sudoers (which allowlists only the wrapper) is never installed ahead of the file it names.
- The agent v0.54.0 sudoers (fetched by the same step) drops raw `mkfs.*` (allowlists only the wrapper)
and adds read-only `pvs`/`zpool status -P` for the agent's unclaimed-disk guard.
- `scripts/CHANGELOG.md`, version banner. Commit **`7ea36bc`** on `main`.
- `bash -n` + `shellcheck` clean (0 new warnings; the 2 pre-existing SC2015 in `step_verify` unchanged).
- **F1 (HIGH, blocks shared-box trial):** arm the bring-up/restore-test compensating destroy only after `RestoreLXC` returns a UPID — never destroy what the job didn't create.
- **F2 (MED, blocks trial):** restore-test should advance past an "already exists" band vmid (or pre-verify via fenced root CLI); document 990000–990009 as reserved fleet-wide.
- **F3 (MED, doc-only, blocks trial):** provisioning runbook must say vmid selection uses the host's full guest list, not the hub inventory.
- F4/F5/F6 (LOW/INFO): non-pool source-config read in restore-test, pool-blind eject dependent-scan, latent reconcile desired-state rule.
## 3. Served-copy version
`curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh | grep SCRIPT_VERSION` → **`1.8.0`**.
## 4. Live validation (felhom-pve, agent v0.54.0 deploy)
The two-layer format guard (agent unclaimed-filter + this wrapper) validated on the demo — details in
`felhom-agent/REPORT.md` §5:
- wrapper REFUSES the OS disk `/dev/sda` and the LVM-PV partition `/dev/sda3`;
- raw `sudo mkfs.ext4 /dev/sda` is DENIED (Part B — wildcard removed);
- an unclaimed throwaway `/dev/sdd` FORMATS (the only real format);
- the agent guard's sudo reads (`pvs`/`lsblk`/`zpool`) all work as the felhom-agent user.
**Deploy note:** installing the sudoers via a manual scp of the Windows working-copy failed `visudo`
(CRLF line endings); fixed live by CR-stripping. The committed repo file is LF, and the host-install
`fetch_raw`-from-Gitea path serves LF — so the shipped path is unaffected. (Lesson recorded.)
## 5. NOT yet live-validated / deferred
- A **fresh install** exercising the new `step_agent_install` wrapper-install step end-to-end (the demo
was repaired in place; a clean box will exercise the fetch+install path — the code is dry-run-safe and
mirrors the existing sudoers/unit install).
- **Impl-2** (raw-device discovery + registry enrollment + wizard rewiring) and **Impl-3** (shared-box
operator gate for self-serve format) are the separate follow-up specs.
## 6. Observations
- The guarded-mkfs wrapper is `configs/felhom-mkfs-guarded.sh` in the **felhom-agent** repo (co-located
with the sudoers it pairs with); host-install fetches it the same way it fetches the unit/sudoers.
Demo left exactly as found: pool `felhom` = {9201}, all scratch/decoy guests and campaign artifacts removed, agent active, controller healthy.
documentation/audits/CAMPAIGN-pool-effects-2026-07-01.md
+119
View File
@@ -0,0 +1,119 @@
# CAMPAIGN — full effects of the `felhom` pool + scoped-token migration
**Date run:** 2026-07-02 (runbook authored 2026-07-01) · **Host:** felhom-pve (`demo-felhom`, PVE 9.2.2)
**Agent live:** v0.59.0 (runbook table said v0.55.0 — v0.56–0.59 shipped between authoring and execution; the ListLXC/priv surface is unchanged)
**ACL:** host-install v1.7.0 3-role scoped layout, verified live: `FelhomAgentGuest` @`/pool/felhom` (12 privs incl. `Pool.Allocate`), `FelhomAgentStore` (`Datastore.Allocate,AllocateSpace`) @`local`/`local-lvm`/`felhom-pbs`, `FelhomAgentBase` (`Sys.Audit,SDN.Use,Datastore.Audit`) @`/`.
**Method:** the agent's own code paths where they exist (`--selftest=bring-up|task|restore-test|hub`, run as the `felhom-agent` user), raw PVE-API probes **as the scoped token** for the rest (identical paths/params to `internal/proxmox/mutate.go`). Mutations on scratch pool guests 9310/9311 + throwaway decoys; 9201 got read/non-destructive probes only. **No code changed; demo left as found** (pool = {9201}, agent active, controller healthy, all scratch/decoys destroyed, 9310's PBS archive pruned).
---
## 1. Track A — operation-matrix sweep: **16/16 PASS, zero surprise 403s**
The scoped token is **functionally COMPLETE** for the agent's real operation surface.
| # | Op | How exercised | Result |
|---|----|---------------|--------|
| 1 | RestoreLXC → pool | bring-up selftest (golden → 9310, 52s boot+running); PBS archive → 9311 raw probe | **PASS** (both local + PBS archive paths) |
| 2 | Vzdump → felhom-pbs | raw probe, mode=snapshot, on 9310 | **PASS** (task OK, 100s) |
| 3 | DestroyLXC + `destroy-unreferenced-disks=1` | raw probe on 9310+9311 | **PASS** — all 6 LVM volumes verifiably reaped |
| 4 | Snapshot | `--selftest=task -vmid 9310` | **PASS** |
| 5 | Rollback | same | **PASS** |
| 6 | DeleteSnapshot | same | **PASS** |
| 7 | SetConfig Disk/Network/CPU/Memory/Options | bring-up (Disk/mounts) + raw probes: `net1` add+delete, `cores`, `memory`, `onboot`, description (selftest) | **PASS** all five key classes |
| 8 | ResizeLXC | rootfs +1G raw probe (32G→33G, task OK) | **PASS** |
| 9 | Start / Stop | bring-up start; stop probe pre-destroy | **PASS** |
| 10 | ListLXC | probe | **PASS** — but returns ONLY pool guests (Track B) |
| 11 | GuestStatus/Config/ListSnapshots | selftests + probes | **PASS** (pool guests) |
| 12 | Nodes/NodeStatus/Version | `--selftest=hub` collect | **PASS** |
| 13 | ListRunningTasks | probed DURING a live vzdump | **PASS** — task visible mid-flight |
| 14 | ListStorage/NodeStorage/StorageContent/LatestBackupVolID | probes incl. felhom-pbs content enumeration | **PASS** (3b-fix intact) |
| 15 | TaskStatusOnce/WaitTask | used throughout (every async op) | **PASS** |
| 16 | TaskLogTail | full log of completed vzdump read | **PASS** |
Full restore-test cycle (restore→link-down→boot→verify→teardown) also **PASS** end-to-end under the scoped token (20s), including teardown of its own pool-created scratch.
Transient (not a finding): concurrent SetConfig during a running resize task → HTTP 500 "can't lock file pve-config-9310.lock" — ordinary PVE per-guest config locking, retry succeeds.
---
## 2. Track B — pool-blindness (the depth)
Mechanism proven live first: the scoped token's `GET /nodes/demo-felhom/lxc` returns **only {9201}** while the host holds 9000/9001/9100/9999/9201. Every ListLXC consumer inherits this view.
### B1 — vmid origin + collision verdict ⭐ the campaign's most important result
**Origin (source-traced, definitive):** the provision vmid is **always externally supplied** — both `BringUpSpec` call sites (`--selftest=bring-up`, `--selftest=provision`, `cmd/felhom-agent/main.go:1233/1388`) take `-vmid` as an explicit CLI flag. No `nextid`/next-free scan exists anywhere in the agent **or the hub** (grep-verified both repos). The **only** agent-computed vmid is the restore-test scratch (`pickScratchVMID` over ListLXC, band 990000–990009, `restoretest.go:419`).
**Collision-avoidance therefore sits with the operator/hub — and the operator's hub inventory is now pool-only (B3), so on a shared box the operator can pick a vmid occupied by a guest the hub can't see.** What happens then:
**Adversarial run 1 — bring-up onto existing non-pool 9001:**
1. The bring-up duplicate-guard (`bringup.go:180`, pool-blind ListLXC) **MISSED** the existing guest and proceeded.
2. PVE refused the restore: HTTP 500 "CT 9001 already exists" (agent never sets `force`) → **no clobber**.
3. **The compensating rollback then attempted `DestroyLXC(9001)` — a guest the job never created** (gate classed it benign `guest_destroy`, SameTxnCreated provenance). Only the pool ACL stopped it: HTTP 403 "missing privilege VM.Allocate at /vms/9001". **Under the pre-3b broad token this would have DESTROYED the innocent guest.**
4. The dangling in-flight Rollback journal entry self-resolved on the next Recover: pool-blind ListLXC can't see 9001 → "guest already gone" → terminal-clean. (Coincidentally correct: containment guarantees the agent can't have created/kept anything outside the pool.)
**Adversarial run 2 — restore-test with a non-pool decoy parked at 990000 (band start):**
1. `pickScratchVMID` (pool-blind) picked the **occupied** 990000.
2. Restore refused ("CT 990000 already exists") → no clobber.
3. **The deferred teardown attempted `DestroyLXC(990000)` on the innocent decoy** → 403, decoy survived.
4. Journal entry self-resolved clean on next Recover (same pool-blind coincidence).
5. Result surfaced as **FAIL** (`pass:false`, "already exists") — **not** Skipped — i.e. a false "backup not restorable" signal to the hub.
6. Because the picker always takes the **lowest** free-looking vmid and the squatter never becomes visible, it re-picks 990000 **every run**: one non-pool squatter at the band start = restore-test **permanently broken** (it never advances to 990001).
**Verdict:** no collision can clobber or destroy a foreign guest **as long as the pool-scoped ACL is in place** — but that is a *single* containment layer over an agent-logic flaw (destroy-what-you-didn't-create), and the failure mode is noisy-but-safe (provision fails, ERROR logs, false restore-test alerts). See fixes F1/F2/F3.
Bonus finding (same run): with a **non-pool source guest** (the golden 9100), the restore-test cannot read the source config for bind-mount neutralization (403 VM.Audit) → logs "restoring as-is". Harmless for the golden (no bind mounts), but a bind-mount-bearing non-pool source archive would fail the restore-test instead of being neutralized (F4).
### B2 — eject dependent-guest scan (`disks.go:1026 dependentGuests`)
**Pool-blind at TWO layers, proven live without a disruptive eject:** (a) the scan enumerates via ListLXC → non-pool guests never enter the loop (list = {9201} proven); (b) even if they did, `GuestConfig` on a non-pool guest 403s (observed live on 9100). **On a shared box, ejecting a drive a non-pool guest bind-mounts would be under-warned.** Severity: low today (single-tenant demo; Felhom drives are enrolled to Felhom guests), real for the shared-box trial. Fix note in F5.
### B3 — hub inventory (`collect.go:204`)
Confirmed live via `--selftest=hub`: the host report's `guests` = **only 9201**; the four non-pool guests are invisible to the hub/operator. **Decision recorded: intended post-containment** — the agent manages (and reports) only its pool. The consequence that must be documented (F3): the hub inventory is NOT a full-host view, so **operator vmid selection for a new provision must consult the host's real guest list (PVE UI/root), never the hub inventory**, on any box that has non-Felhom guests.
### B4 — remaining ListLXC consumers
| Consumer | Behavior with a pool-only list | Verdict |
|---|---|---|
| stale-lock guard (`stalelock.go`) | Guest enumeration pool-only → recovers only pool guests (intended: non-pool guests aren't the agent's to unlock). Its safety input `ListRunningTasks` is **NOT pool-blind** — proven live: a **root-initiated vzdump of non-pool 9001 was visible** to the token mid-flight (Sys.Audit@/ covers the task list). No misfire vector found. | ✅ correct |
| reconcile `engine.readActual` (`engine.go:229`) | Actual-state is pool-only. Today EmptyProvider → no effect. **Latent rule:** desired-state must never reference a non-pool guest, or reconcile would treat it as absent and re-provision-over (which then hits the B1 chain, contained). | ✅ today; rule recorded |
| `recover.go` (scratch + bring-up reapers) | Existence check pool-only → an out-of-pool collision vmid resolves "already gone" (terminal-clean). Coincidentally correct under containment (see B1.4); would be WRONG (silent leak of a half-built guest) only if the agent could create outside the pool — it can't (restore without `pool=felhom` 403s). | ✅ under scoped ACL |
| `--selftest=read` list check (`main.go:1753`) | Cosmetic (prints pool guests only). | ✅ |
---
## 3. Track C — blast-radius regression: **containment intact**
| Probe | Result |
|---|---|
| Stop non-pool 9001 as token | **403** "Permission check failed (/vms/9001, VM.PowerMgmt)" |
| Vzdump 9201 → non-granted `felhom-usb` | **403** "(/storage/felhom-usb, Datastore.AllocateSpace)" — refused at POST, nothing silently retargeted, no task started |
| Drive-storage audit (felhom-usb/felhom-flash) | **readable** (status + list, HTTP 200) — 3b-fix (`Datastore.Audit`@/ in Base) intact, no false detach |
---
## 4. Track D — pool-membership non-ACL effects: **no surprises**
- **No leak:** the pool name appears in NO customer/controller-facing surface — hub host-report guest entries carry `vmid/name/status/controller_version/spec` only (verified live); `pct config 9201` has no pool field (pool membership lives in PVE `user.cfg`); the guest-visible bootstrap mount is `bootstrap.json` only.
- **`--adopt-pool`** (v1.8.0, live run): idempotent — "pool felhom already exists" + "guest 9201 already in pool felhom", guest not otherwise touched.
- **`--uninstall --dry-run`** (live): removes the `/pool/felhom` ACL grants and correctly **refuses to delete a non-empty pool** ("pool felhom not empty (members: 9201) — leaving it"). In a real uninstall the destroyed guest auto-leaves the pool first; the guard protects foreign members.
---
## 5. Prioritized fix list
| # | Sev | Finding | Recommended fix | Blocks shared-box trial? |
|---|-----|---------|-----------------|--------------------------|
| **F1** | **HIGH** (logic; contained by ACL) | Bring-up rollback + restore-test teardown fire `DestroyLXC` on a vmid the job **never created** (create failed "already exists"). One ACL layer away from destroying a foreign guest; any future broad-token/`root` deployment re-arms it. | Arm the compensating destroy **only after `RestoreLXC` returns a UPID** (nothing created ⇒ nothing to roll back). A POST *network* error (outcome unknown) may keep today's behavior. Applies to `bringup.go` defer + `restoretest.go` defer (+ their Recover twins are already exists-checked). | **YES** — fix before the trial (defense-in-depth; the trial box has foreign guests by definition) |
| **F2** | **MED** | One non-pool squatter at the scratch-band start ⇒ restore-test **permanently FAILs** (always re-picks the lowest invisible-occupied vmid; reports FAIL not Skip → false "backup unrestorable" alerts). | On an "already exists" restore failure, advance to the next band vmid (or pre-verify emptiness via the fenced root `pct list`); band exhausted ⇒ Skip (existing semantics). Additionally: document 990000–990009 as **reserved fleet-wide** on any Felhom host. | YES (cheap; false alerts would drown the trial's signal) |
| **F3** | **MED** (process/doc) | Hub inventory is pool-only (by design) while vmid assignment is operator-side ⇒ on a shared box the operator can pick an occupied, hub-invisible vmid. Outcome is safe-but-noisy (B1 chain). | Document in the provisioning runbook: **pick vmids from the host's full guest list (PVE UI / root `pct list`+`qm list`), never from the hub inventory.** Longer-term: Day-0/hub could record a host-declared reserved-vmid range. | YES (doc-only) |
| **F4** | LOW | Restore-test can't read a **non-pool source guest's** config for bind-mount neutralization (403) → "restoring as-is"; a bind-mount-bearing non-pool source archive would false-fail the test. | Accept + log (current behavior is fail-safe), or route the source-config read through the fenced root CLI. Today's sources (9201, golden) are unaffected. | no |
| **F5** | LOW | Eject `dependentGuests` scan is double-pool-blind (list + config 403) → non-pool dependents of an ejected drive are not warned about. | Either declare pool-scoped-by-design in the eject UX copy ("Felhom guests using this drive: …"), or scan via the fenced root CLI for the shared-box case. | no (drives are Felhom-enrolled; revisit if drives are ever shared) |
| **F6** | INFO | Stale-lock guard, reconcile readActual, recover reapers: all correct under a pool-only view (details §2 B4); the reconcile desired-state must never reference a non-pool vmid (latent rule, currently unfed). | Record the rule next to `DesiredProvider` when slice-10 feeds it. | no |
**Pre-existing, noted in passing (NOT pool effects):** non-root agent can't read `/etc/pve/priv/storage/felhom-pbs.pw` (known OPEN from the Day-0 bundle — PBS datastore detail skipped in reports); the drive-enrollment candidacy bug has its own pending fix spec (explicitly out of scope per the runbook).
## 6. Not changed
Validation only — **no code, no version bump**. felhom-agent untouched (read-only source trace). The demo host was left exactly as found: guests {9000, 9001, 9100, 9201, 9999}, pool `felhom` = {9201}, agent v0.59.0 active, controller healthy, campaign scratch guests/decoys/PBS archive/probe artifacts all removed.