14 KiB
REPORT — F-REBOOT + F-LEAK (v0.107.0 → v0.110.0) (2026-07-28)
Overwritten per the standing rule. Agent v0.106.0 → v0.110.0 across four commits, three of them
corrections to the first. F-LEAK took three attempts and two of them were refuted by live test; that
arc is the substance of this report. Companions: felhom-controller v0.180.0 (F-OBS) and felhom.eu
host-install v1.21.0 (F-LEAK's actual fix).
Baselines (reconfirmed, not copied)
Agent 0.106.0 live on both boxes at start; controller 0.179.0; felhom-agent af1c21a,
felhom.eu d0cec9d, felhom-controller 8f46495, all clean.
F-REBOOT — a guest rebooted mid-backup never came back (v0.107.0, SHIPPED + PROVEN LIVE)
Fault 11: the backup succeeded, and the guest was found stopped with 0 containers, no lock, and
nothing retrying — 9m47s of total appliance outage with every alarm silent, because nothing was broken
except that the box was off.
Why the existing recovery missed it. RecoverStaleLockedGuests already does unlock → delete
dangling snapshot → start iff onboot, and it is correct. Two narrow gaps: its predicate acts only
on a guest holding a stale vzdump lock (fault 11's guest was unlocked), and it runs once at agent
startup on the load-bearing invariant that a lock present then is stale by definition — a guest
that goes down while the agent is already up is never re-examined.
internal/localapi/guestpower.go closes exactly those two and nothing more: periodic (60 s), acting on
"should be running, is not, is not locked".
The trap, which is F-CRIT-1's shape
A deliberately stopped guest must not be auto-started; fighting the operator is worse than the outage.
The discriminator is onboot, and it is deliberately not invented here — it is already what
stalelock.go uses for this same decision (so the two paths cannot disagree), it is 0 on
scratch/golden guests, and it is what pve-guests itself consults at host boot, so the agent agrees
with the platform instead of keeping a second private definition of "should be running". The hub's
desired-state Run is stronger but hub-dependent; onboot still works on a box that has lost hub
contact, which is when an unattended appliance most needs to come back.
Guards, in order: running → forget history · !onboot → return (Scenario B) · lock != "" → leave to
the stale-lock path · BackupRunning true or error → do not start (a stop-mode backup stops the
guest on purpose) · bounded 3 attempts at 1m/2m/4m → then ERROR once (Scenario C).
Live, on demo-hp
| guest stopped out from under the agent | recovered unattended in 120 s |
| the finding's original incident | 587 s, needing a human pct start |
| lock-deferral guard | fired live during the replay |
| Scenario B, same box | an onboot:0 guest left stopped across the whole window |
F-LEAK — and the first fix was WRONG, refuted by its own live replay
What I shipped in v0.107.0 and then removed in v0.108.0. A teardown fallback that, on the 403,
adopted the stranded scratch into the felhom pool and retried the destroy — reasoning that the token
holds Pool.Allocate on /pool/felhom. It fired exactly as designed and PVE refused it:
WARN restore-test: scratch teardown failed — adopting the stranded scratch ... vmid=990000 pool=felhom
ERROR restore-test: pool adoption failed; left for Recover vmid=990000
err="proxmox: PUT /pools/felhom -> HTTP 500: permission denied at /vms/990000 (missing privilege ...)"
PUT /pools/{pool} also requires VM.Allocate on the VM being added. Pool membership cannot
bootstrap its own authority. The code and its guard were removed rather than left in place — a path
that provably cannot work is worse than no path, because it looks like a fix.
The real cause is structural, not a missing privilege in the role: FelhomAgentGuest is granted at
/pool/felhom, and a guest joins that pool only when its restore completes. A failed
restore-test therefore leaves a guest that exists, is in no pool, and is outside the token's reach.
Attempt 2 — the band-scoped ACL. It works exactly ONCE per slot.
felhom-host-install.sh v1.21.0 grants the role at each /vms/990000…/vms/990009. The live A/B on
the same guest, minutes apart, proved it does close the defect:
DELETE /nodes/<node>/lxc/990000 with the agent's own token |
|
|---|---|
| grant removed (the original defect) | 403 Permission check failed (/vms/990000, VM.Allocate) — guest still present |
| grant restored | 200 UPID:...:vzdestroy:990000:felhom-agent@pve!agent — guest gone |
And then the post-run ACL count caught it: /vms/990000 had 0 grants afterwards, while the
other nine slots had 2 each. Confirmed in PVE's own source rather than inferred — the destroy path
calls AccessControl::remove_vm_access($vmid) (API2/LXC.pm:906), which deletes
acl_root->children->vms->children->{$vmid}, i.e. every ACL at /vms/<vmid>
(AccessControl.pm:1898).
The grant is consumed by the very operation it authorises. Ten teardowns and the band is ungranted
and the defect is back. A per-VM ACL is structurally the wrong tool; the pool grant is durable only
because it lives on a pool path, which remove_vm_access leaves alone (it removes the guest's pool
membership, not the pool's ACL). I would not have found this without checking the box after the fix.
Attempt 3 — the fourth root-fenced exception (v0.110.0, SHIPPED)
The route this task offered as the alternative: the same authority that already performs the create.
Privileged.DestroyScratchLXC joins the fence (previously exactly three: keyctl pct create, USB
mount/fstab, SMART/sensors) and is fenced harder than any of them — in three places, none consumed by
use:
| layer | enforcement |
|---|---|
| sudoers | /usr/sbin/pct destroy 99000[0-9] --purge — sudo matches the vmid literally; a compromised agent asking for 9201 is refused by sudo itself |
DestroyScratchLXC |
re-checks the band before exec; refuses an unconfigured or inverted band rather than defaulting to anything |
teardownScratch |
acts only on this journal entry's own scratch provenance |
The API destroy is still tried first and remains the normal path; the v1.21.0 band ACLs stay provisioned so the common case needs no privileged call. This is the fallback that makes teardown deterministic instead of once-per-slot. Ships with a sudoers change, deployed alongside the binary.
The fence, proven live by ASKING sudo rather than executing
Executing the negative case against a live guest is exactly what must not happen, so sudo -l was used
to ask whether each command is permitted:
| command | verdict |
|---|---|
pct destroy 990000 / 990005 / 990009 --purge |
PERMITTED |
pct destroy 9201 --purge (the live customer guest) |
REFUSED |
pct destroy 9100 / 9999 / 1 --purge |
REFUSED |
pct destroy 990010 --purge (one past the band) |
REFUSED |
pct start 990000, pct set 990000 -memory 512 |
REFUSED — the grant is this one op, not this vmid |
The end-to-end proof, in the ACL's natural post-consumption state
The best possible setup arrived by itself: after the attempt-2 destroy, /vms/990000 had 0 grants.
A fresh PBS restore to 990000 without --pool then reproduced the stranded guest with the ACL
genuinely absent — not artificially removed:
/vms/990000 grants: 0
990000 stopped (restore rc=0, in no pool)
DELETE via token -> HTTP 403 Permission check failed (/vms/990000, VM.Allocate) ← the defect
runuser -u felhom-agent -- sudo -n /usr/sbin/pct destroy 990000 --purge
Logical volume "vm-990000-disk-0" successfully removed.
Logical volume "vm-990000-disk-1" successfully removed.
Logical volume "vm-990000-disk-2" successfully removed.
purging CT 990000 from related configurations.. rc=0
after: pct list -> 9201 only · 990000 LVs remaining: 0 · /etc/pve/lxc/990000.conf: absent
That command vector is byte-identical to what Privileged.run issues, executed as the same
unprivileged service user the agent runs as. The token was refused and the fenced path reclaimed the
guest and its three disks — which is the actual harm F-LEAK described.
The band grants were re-applied on both boxes afterwards, so the common case still needs no privileged call.
Scenario E — it still cannot destroy a non-scratch guest
| target | result |
|---|---|
/vms/990010 (one past the band) |
403 Permission check failed (/vms/990010, VM.Allocate) |
/vms/100 |
403 same |
990010 does not exist, and PVE still answered 403 rather than "does not exist" — so PVE checks
permission before existence and these are genuine authorization refusals, not artifacts. Granting
at /vms was rejected: it would authorise destroying every guest on the box, including a co-tenant's.
A careless step, recorded rather than buried. That probe loop also issued a live DELETE against
running guest 9201. It was refused with 500 container is running — but the permission check
passed (9201 is a pool member by design), so had the guest been stopped I would have destroyed the
live demo guest. The scratch-band probes were the safe ones; 9201 had no business in that list.
v0.109.0 — the guest-power watchdog gets the observable it shipped without
Self-correction, caught by applying this session's own F-OBS check to my own new code. The v0.107.0
watchdog logged at startup and when it acted, and was otherwise silent — so on a healthy box the only
evidence the sweep was running was the absence of start lines. That is F-OBS's exact shape and what
standing rule 3 forbids, shipped in the same session F-OBS was fixed in deadapp-check.
INFO summary every 10th sweep (10 min) carrying sweeps_since_boot, guests_evaluated,
currently_stopped — what it saw, not merely that it ran, because "alive, all up" and "alive, one
guest deliberately left down" are different operator facts. Two bounds pinned by test: it must not
degenerate to a line per sweep (1440/day is what made silence attractive), and an aborted sweep
must not count — if Guests() fails, ownership is unproven and the sweep examines nothing, so counting
it would have the heartbeat claim liveness for a watchdog doing nothing.
Red-proofs — 10 total, all observed failing
| # | red-proof | observed failure |
|---|---|---|
| 1 | delete the Start call |
guest 9201 was NOT started — this is F-REBOOT |
| 2 | remove the !onboot guard |
a deliberately stopped guest (onboot:0) was started |
| 3 | remove the max-attempts branch | start attempted N times, want at most 3 — infinite retry loop |
| 4 | drop the backoff | retried inside the backoff window — the bound is not being honoured |
| 5 | make BackupRunning non-fatal |
started despite being unable to confirm no backup is running |
| 6 | unfiltered guest list on error | acted with unproven ownership |
| 7 | remove the liveness observable | no liveness observable after 10 sweeps — silence is indistinguishable from a dead watchdog |
| 8 | count aborted sweeps as healthy | an aborted sweep was counted as healthy (sweeps=20) |
| 9 | remove the band check from DestroyScratchLXC |
REFUSAL FAILED: executed [pct destroy 9201 --purge] for out-of-band vmid 9201 |
| 10 | remove the unconfigured-band check | an unconfigured band admitted vmid 0 and EXECUTED [pct destroy 0 --purge] |
go build, go vet ./..., go test ./... — 29 packages, rc=0, run separately from every commit.
A process failure of mine, corrected mid-task. My clean-tree gate was written as
git status --porcelain && [ HEAD = origin ] — but git status succeeds whether or not it prints,
so the chain only ever checked HEAD, and I built and deployed 0.109.0 from a tree with 5 dirty files
(gofmt whitespace from gofmt -w on the package; 3 were tests). Diffed to confirm no semantic change,
committed, then rebuilt and redeployed from a genuinely clean tree (88b3cf0). The gate is now
if [ -n "$(git status --porcelain)" ]; then abort; fi.
Deployed & fleet state
Agent 0.110.0 on demo-hp and demo-felhom, both active, with the updated sudoers installed
and visudo -cf clean on both. Band ACLs at 20 rows (10 vmids x user+token) on both boxes. Scratch-band ACLs applied on
both boxes. No leftover scratch guests — pct list shows only 9201(running) on demo-hp.
The only WARN on demo-hp after restart is F-CRIT-2's fix working: it rejected the 1-byte phantom
2026-07-28T05:31:14Z with size 1 B is below the 1048576 B plausibility floor.
Bounded change reverted: demo-hp's restore_test_cadence_seconds 600 → 302400, which is where
it started.
NOT yet live-validated (carried forward)
- Fault 4 — restic transport interruption; four injection approaches defeated by guest-bridged networking. Still the most valuable follow-up: F-CRIT-2 answered the phantom question for PBS and it remains open for restic.
- R-100 (new, filed today) — a restic offsite tier failing nightly never goes stale on the hub;
isStalereads onlyLastRun, which the controller writes unconditionally on failure. Investigated, deliberately not fixed. - R-99 — prune never removes phantom snapshots (no retention bug; they accumulate).
- Fault 12 (host reboot mid-backup); the three-way backup + restore-test + GC overlap; the agent's DR
bring-up path (
ModeDRGuestLoss), still read-only-verified. contentionAlarmAfter(3h) — unit-tested with an injected clock, not live.- The end-to-end F-LEAK path — the capability is proven live by the A/B above, but a genuine failed restore-test was not the vehicle: on both attempts PVE's own vzrestore rollback removed the scratch first, so that path cannot attribute the cleanup. The A/B at the seam was chosen precisely because it can.