From d5c769173b5e651355eecdee37f692bf7653c601 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 28 Jul 2026 11:34:48 +0200 Subject: [PATCH] REPORT + CONTEXT: F-LEAK closed via the fenced destroy (v0.110.0), all three attempts recorded --- CONTEXT.md | 28 +++++++++-------- REPORT.md | 90 +++++++++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 26 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 887e573..bd9389e 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -18,18 +18,22 @@ Live on demo-hp: **120 s unattended** recovery vs the incident's **587 s** with a human; Scenario B proven (an `onboot:0` guest left stopped throughout). Detail: `REPORT.md`. -- **2026-07-28 — v0.108.0: the F-LEAK pool-adoption fix was WRONG and was removed.** v0.107.0 also - shipped a teardown fallback that adopted a stranded scratch guest into the `felhom` pool and retried - the destroy. Its own live replay refuted it: `PUT /pools/felhom -> HTTP 500: permission denied at - /vms/990000 (missing privilege ...)`. **`PUT /pools/{pool}` also requires `VM.Allocate` on the VM - being added, so pool membership cannot bootstrap its own authority.** Removed rather than left in — - a path that provably cannot work looks like a fix. **The real fix is `felhom-host-install.sh` - v1.21.0** granting `FelhomAgentGuest` at each `/vms/990000..990009`; the cause was structural (the - role is granted at `/pool/felhom`, and a guest joins the pool only when its restore *completes*, so - a *failed* restore-test leaves a pool-less guest out of reach). Applied on **both** boxes. Proven by - live A/B on the same guest: grant removed → `403 (/vms/990000, VM.Allocate)`; granted → `200 - UPID:...vzdestroy`. Still refused at `/vms/100` and `/vms/990010` — and since PVE checks - **permission before existence**, those 403s are real refusals, not artifacts. +- **2026-07-28 — F-LEAK took THREE attempts; v0.108.0 and v0.110.0 are the corrections.** The cause is + structural: `FelhomAgentGuest` is granted at `/pool/felhom` and a guest joins that pool only when its + restore **completes**, so a *failed* restore-test leaves a pool-less guest out of reach (403). + **(1) v0.107.0 pool adoption — REFUTED LIVE:** `PUT /pools/{pool}` also requires `VM.Allocate` on the + VM being added, so membership cannot bootstrap its own authority; removed in **v0.108.0**. + **(2) host-install v1.21.0 per-path `/vms/990000..990009` ACLs — works, but exactly ONCE per slot:** + PVE's destroy calls `AccessControl::remove_vm_access` (`API2/LXC.pm:906`) which deletes every ACL at + `/vms/` (`AccessControl.pm:1898`) — **the grant is consumed by the op it authorises**. Caught by + counting ACL rows after the fix, not by reasoning. **(3) v0.110.0 SHIPPED — + `Privileged.DestroyScratchLXC`, the FOURTH root-fenced exception** (was exactly three: keyctl + `pct create`, USB mount/fstab, SMART/sensors). Band enforced in **sudoers literally** + (`pct destroy 99000[0-9] --purge`) + re-checked in code + journal provenance at the caller; none is + consumed by use. API destroy still tried FIRST; band ACLs stay provisioned so the common case needs no + privileged call. **Ships with a sudoers change — deploy `configs/felhom-agent.sudoers` WITH the + binary.** Live: token 403 on a stranded scratch → fenced path removed the guest and all 3 LVs; sudo + PERMITS the band and REFUSES `9201`/`9100`/`9999`/`990010`/`1`, and refuses `pct start 990000` too. - **2026-07-28 — v0.109.0: the guest-power watchdog got the observable it shipped without.** A self-correction: v0.107.0's watchdog logged only at startup and when it *acted*, so on a healthy box diff --git a/REPORT.md b/REPORT.md index aac94a0..3567871 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,7 +1,8 @@ -# REPORT — F-REBOOT + F-LEAK (v0.107.0 → v0.109.0) (2026-07-28) +# REPORT — F-REBOOT + F-LEAK (v0.107.0 → v0.110.0) (2026-07-28) -**Overwritten** per the standing rule. Agent **v0.106.0 → v0.109.0** across three commits, two of them -corrections to the first. Companions: `felhom-controller` v0.180.0 (F-OBS) and `felhom.eu` +**Overwritten** per the standing rule. Agent **v0.106.0 → v0.110.0** across four commits, **three of them +corrections to the first**. F-LEAK took three attempts and two of them were refuted by live test; that +arc is the substance of this report. Companions: `felhom-controller` v0.180.0 (F-OBS) and `felhom.eu` host-install v1.21.0 (F-LEAK's actual fix). ## Baselines (reconfirmed, not copied) @@ -68,19 +69,77 @@ that provably cannot work is worse than no path, because it looks like a fix. `/pool/felhom`, and a guest joins that pool only when its restore **completes**. A *failed* restore-test therefore leaves a guest that exists, is in no pool, and is outside the token's reach. -**The fix is `felhom-host-install.sh` v1.21.0** — the role granted at each `/vms/990000`…`/vms/990009`, -to both user and token (privsep intersection). `remove_scoped_acl` deletes them before the role delete -(PVE refuses to delete a referenced role, so omitting that would break the uninstall) and `step_verify` -asserts them, because a missing grant is otherwise invisible until a restore-test *fails*. - -### Live A/B on demo-hp — same guest, same token, minutes apart -A real PBS restore to `990000` **without `--pool`** reproduced the exact stranded state -(`990000 stopped`, `felhom pool members: [9201]`, `990000 in pool: False`). +### Attempt 2 — the band-scoped ACL. It works exactly ONCE per slot. +`felhom-host-install.sh` v1.21.0 grants the role at each `/vms/990000`…`/vms/990009`. The live A/B on +the same guest, minutes apart, proved it does close the defect: | | `DELETE /nodes//lxc/990000` with the agent's own token | |---|---| | **grant removed** (the original defect) | `403 Permission check failed (/vms/990000, VM.Allocate)` — guest still present | -| **grant restored** (the fix) | `200 UPID:...:vzdestroy:990000:felhom-agent@pve!agent` — guest gone | +| **grant restored** | `200 UPID:...:vzdestroy:990000:felhom-agent@pve!agent` — guest gone | + +**And then the post-run ACL count caught it:** `/vms/990000` had **0 grants** afterwards, while the +other nine slots had 2 each. Confirmed in PVE's own source rather than inferred — the destroy path +calls `AccessControl::remove_vm_access($vmid)` (`API2/LXC.pm:906`), which deletes +`acl_root->children->vms->children->{$vmid}`, i.e. **every ACL at `/vms/`** +(`AccessControl.pm:1898`). + +**The grant is consumed by the very operation it authorises.** Ten teardowns and the band is ungranted +and the defect is back. A per-VM ACL is structurally the wrong tool; the pool grant is durable only +because it lives on a *pool* path, which `remove_vm_access` leaves alone (it removes the guest's pool +*membership*, not the pool's ACL). I would not have found this without checking the box after the fix. + +### Attempt 3 — the fourth root-fenced exception (v0.110.0, SHIPPED) +The route this task offered as the alternative: **the same authority that already performs the create.** +`Privileged.DestroyScratchLXC` joins the fence (previously exactly three: keyctl `pct create`, USB +mount/fstab, SMART/sensors) and is fenced harder than any of them — in three places, none consumed by +use: + +| layer | enforcement | +|---|---| +| **sudoers** | `/usr/sbin/pct destroy 99000[0-9] --purge` — sudo matches the vmid **literally**; a compromised agent asking for `9201` is refused by sudo itself | +| **`DestroyScratchLXC`** | re-checks the band before exec; refuses an unconfigured or inverted band rather than defaulting to anything | +| **`teardownScratch`** | acts only on this journal entry's own scratch provenance | + +The API destroy is still tried **first** and remains the normal path; the v1.21.0 band ACLs stay +provisioned so the common case needs no privileged call. This is the fallback that makes teardown +deterministic instead of once-per-slot. Ships **with a sudoers change**, deployed alongside the binary. + +### The fence, proven live by ASKING sudo rather than executing +Executing the negative case against a live guest is exactly what must not happen, so `sudo -l` was used +to ask whether each command is permitted: + +| command | verdict | +|---|---| +| `pct destroy 990000 / 990005 / 990009 --purge` | **PERMITTED** | +| `pct destroy 9201 --purge` (the live customer guest) | **REFUSED** | +| `pct destroy 9100 / 9999 / 1 --purge` | **REFUSED** | +| `pct destroy 990010 --purge` (one past the band) | **REFUSED** | +| `pct start 990000`, `pct set 990000 -memory 512` | **REFUSED** — the grant is this one op, not this vmid | + +### The end-to-end proof, in the ACL's natural post-consumption state +The best possible setup arrived by itself: after the attempt-2 destroy, `/vms/990000` had **0 grants**. +A fresh PBS restore to `990000` **without `--pool`** then reproduced the stranded guest with the ACL +genuinely absent — not artificially removed: + +``` +/vms/990000 grants: 0 +990000 stopped (restore rc=0, in no pool) +DELETE via token -> HTTP 403 Permission check failed (/vms/990000, VM.Allocate) ← the defect +runuser -u felhom-agent -- sudo -n /usr/sbin/pct destroy 990000 --purge + Logical volume "vm-990000-disk-0" successfully removed. + Logical volume "vm-990000-disk-1" successfully removed. + Logical volume "vm-990000-disk-2" successfully removed. + purging CT 990000 from related configurations.. rc=0 +after: pct list -> 9201 only · 990000 LVs remaining: 0 · /etc/pve/lxc/990000.conf: absent +``` + +That command vector is byte-identical to what `Privileged.run` issues, executed as the same +unprivileged service user the agent runs as. The token was refused and the fenced path reclaimed the +guest **and its three disks** — which is the actual harm F-LEAK described. + +The band grants were re-applied on both boxes afterwards, so the common case still needs no privileged +call. ### Scenario E — it still cannot destroy a non-scratch guest | target | result | @@ -115,7 +174,7 @@ it would have the heartbeat claim liveness for a watchdog doing nothing. --- -## Red-proofs — 8 total, all observed failing +## Red-proofs — 10 total, all observed failing | # | red-proof | observed failure | |---|---|---| @@ -127,6 +186,8 @@ it would have the heartbeat claim liveness for a watchdog doing nothing. | 6 | unfiltered guest list on error | `acted with unproven ownership` | | 7 | remove the liveness observable | `no liveness observable after 10 sweeps — silence is indistinguishable from a dead watchdog` | | 8 | count aborted sweeps as healthy | `an aborted sweep was counted as healthy (sweeps=20)` | +| 9 | remove the band check from `DestroyScratchLXC` | `REFUSAL FAILED: executed [pct destroy 9201 --purge] for out-of-band vmid 9201` | +| 10 | remove the unconfigured-band check | `an unconfigured band admitted vmid 0 and EXECUTED [pct destroy 0 --purge]` | `go build`, `go vet ./...`, `go test ./...` — 29 packages, `rc=0`, run separately from every commit. @@ -140,7 +201,8 @@ committed, then **rebuilt and redeployed from a genuinely clean tree** (`88b3cf0 --- ## Deployed & fleet state -Agent **0.109.0** on **demo-hp** and **demo-felhom**, both `active`. Scratch-band ACLs applied on +Agent **0.110.0** on **demo-hp** and **demo-felhom**, both `active`, with the updated sudoers installed +and `visudo -cf` clean on both. Band ACLs at 20 rows (10 vmids x user+token) on **both** boxes. Scratch-band ACLs applied on **both** boxes. No leftover scratch guests — `pct list` shows only `9201(running)` on demo-hp. The only WARN on demo-hp after restart is **F-CRIT-2's fix working**: it rejected the 1-byte phantom