F-REBOOT + F-LEAK: the agent's authority over guest lifecycle (v0.107.0)

F-REBOOT — a guest rebooted mid-backup never came back (fault 11: 9m47s of total
appliance outage, no lock, nothing retrying). The existing stale-lock recovery is
correct but missed it two ways: its predicate needs a stale vzdump lock and that
guest was unlocked, and it runs only at agent startup. New periodic guest-power
watchdog acts on 'should be running, is not, is not locked'.

onboot is the should-be-running signal, not invented here: stalelock.go already
uses it for this same decision, it is 0 on scratch/golden, and pve-guests uses it
at host boot. Guards: onboot:0 never touched (Scenario B), a locked guest is left
to the stale-lock path, a guest with a vzdump in flight is left stopped,
unprovable ownership acts on nothing, unconfirmable backup state fails safe.
Bounded retry 3x at 1/2/4m then ERROR (Scenario C) — a healthy start takes ~25s.

F-LEAK — a failed restore-test could not destroy its scratch (403 VM.Allocate).
It is pool membership, not privsep: VM.Allocate is granted at /pool/felhom only,
and a failed restore never completes the --pool association. Fix needs NO new
grant — Pool.Allocate is already held, so the teardown adopts the stranded
scratch into the pool and retries the destroy. Guarded by scratchAdoptAllowed:
scratch provenance AND the numeric band, both required (Scenario E).

Six red-proofs across both fixes, all observed failing.
This commit is contained in:
2026-07-28 10:27:07 +02:00
parent a18b18e5de
commit 367a503a0f
7 changed files with 691 additions and 5 deletions
+74
View File
@@ -1,5 +1,79 @@
# felhom-agent — Changelog
## v0.107.0 — F-REBOOT + F-LEAK: the agent's authority over guest lifecycle (2026-07-28)
Two Campaign 8 findings, both about the agent being unable to act on a guest it owns.
### F-REBOOT — a guest rebooted mid-backup never came back
Fault 11: the backup SUCCEEDED and the guest was found `stopped` with 0 containers, no lock, nothing
retrying — 9m47s of TOTAL appliance outage, every app down, and every alarm silent because nothing
was broken except that the box was off. The trigger is ordinary: backups run overnight and
`CONTEXT.md` records that the N100 needs BIOS AC-Power-Recovery *because power loss happens*.
**Why the existing recovery missed it.** `RecoverStaleLockedGuests` already does
unlock → delete dangling snapshot → **start iff onboot**, and it is correct. It missed this by two
narrow gaps: its predicate acts only on a guest holding a stale vzdump lock
(`backup`/`snapshot-delete`) and fault 11's guest was stopped and **unlocked**; and it runs **once at
agent startup**, on the load-bearing invariant that a lock present then is stale by definition. A
guest that goes down while the agent is already up was never re-examined.
**The fix** (`internal/localapi/guestpower.go`) closes exactly those two gaps: a periodic sweep that
acts on *should be running, is not, and is not locked*.
`onboot` is the "should be running" signal, and it is deliberately **not invented**: it is already
the distinction `stalelock.go` uses for this same decision, it is 1 on customer guests and 0 on
scratch/golden, and it is the flag `pve-guests` itself consults at host boot — so the agent agrees
with the platform instead of keeping a private definition. The hub's desired-state `Run` is stronger
but hub-dependent; `onboot` keeps working on a box that has lost hub contact, which is when an
unattended appliance most needs to come back.
**Guards, because the trap here is F-CRIT-1's shape.** A guest the operator deliberately stopped must
never be auto-started — fighting the operator makes maintenance impossible and is worse than the
outage. So: `onboot:0` ⇒ never touched; a LOCKED guest is left to the stale-lock path (its lock may
mean "a restore is writing my disks"); a guest with a vzdump genuinely in flight is left stopped (a
stop-mode backup stops it on purpose); unprovable ownership ⇒ act on nothing; and unconfirmable
backup state fails safe.
**Bounded retry** (Scenario C): 3 attempts at 1m/2m/4m, then ERROR naming the guest and stop. A
healthy `pct start` of 9201 took ~25 s (observed twice), so even the first wait carries 2.4x headroom;
three attempts bound disruption at ~7 minutes — inside the 9m47s this fixes — and never loop forever.
### F-LEAK — a failed restore-test could not destroy its own scratch
`DELETE /nodes/x/lxc/990000`**403 missing privilege VM.Allocate**. It is **pool membership, not
privsep**, and the live ACLs prove it: `VM.Allocate` is granted at `/pool/felhom` ONLY, never at `/`.
A successful restore's `--pool felhom` makes the guest a member and inheritable; a FAILED restore
never completes that, so the guest's own path resolves to `/` where the token holds nothing:
| path | privileges the token has |
|---|---|
| `/vms/990005` (non-member scratch) | `Datastore.Audit, SDN.Use, Sys.Audit` — no VM.Allocate |
| `/pool/felhom` | full `FelhomAgentGuest` incl. `VM.Allocate` **and `Pool.Allocate`** |
| `/vms/9201` (pool member) | inherits the full set |
**The fix needs NO new privilege.** `Pool.Allocate` is already held, so the teardown adopts the
stranded scratch into the pool and retries the destroy once, which then authorizes via
`/pool/felhom`. Nothing is widened; `PoolAddVMID` already existed for exactly this reason
("membership is what lets the pool-scoped token reach the guest next time").
**The security guard is `scratchAdoptAllowed` — two independent checks, both required:** the journal
entry must carry agent-created **scratch provenance**, AND the VMID must be inside the configured
**scratch band**. Either alone would be enough to fix the leak; both are present because this
function is the only thing between "clean up my own scratch" and "co-opt an arbitrary guest into the
pool and delete it". Adopting a customer guest would hand the token destroy rights over it — far
worse than a leaked scratch.
### Also
Corrected the claim that the quiesce unquiesce is "guaranteed by defer"
(`felhom-controller/.../quiesce.go`, and the generic advice in `PROMPT-TEMPLATE.md`): fault 10
established that a SIGKILL runs no deferred function, and the guarantee is the crash marker plus
`Recover()`.
Files: `internal/localapi/{guestpower.go (new),server.go}`, `internal/reconcile/restoretest.go`,
`cmd/felhom-agent/main.go`, plus tests `internal/localapi/guestpower_test.go` and
`internal/reconcile/scratch_adopt_test.go`. No wire/contract change; no new PVE grant.
## v0.106.0 — F-CRIT-2: a failed backup must not look like a fresh one (2026-07-28)
Campaign 8 killed the PBS daemon mid-upload. PBS published the aborted upload into the storage