F-REBOOT + F-LEAK: the agent's authority over guest lifecycle (v0.107.0)
F-REBOOT — a guest rebooted mid-backup never came back (fault 11: 9m47s of total appliance outage, no lock, nothing retrying). The existing stale-lock recovery is correct but missed it two ways: its predicate needs a stale vzdump lock and that guest was unlocked, and it runs only at agent startup. New periodic guest-power watchdog acts on 'should be running, is not, is not locked'. onboot is the should-be-running signal, not invented here: stalelock.go already uses it for this same decision, it is 0 on scratch/golden, and pve-guests uses it at host boot. Guards: onboot:0 never touched (Scenario B), a locked guest is left to the stale-lock path, a guest with a vzdump in flight is left stopped, unprovable ownership acts on nothing, unconfirmable backup state fails safe. Bounded retry 3x at 1/2/4m then ERROR (Scenario C) — a healthy start takes ~25s. F-LEAK — a failed restore-test could not destroy its scratch (403 VM.Allocate). It is pool membership, not privsep: VM.Allocate is granted at /pool/felhom only, and a failed restore never completes the --pool association. Fix needs NO new grant — Pool.Allocate is already held, so the teardown adopts the stranded scratch into the pool and retries the destroy. Guarded by scratchAdoptAllowed: scratch provenance AND the numeric band, both required (Scenario E). Six red-proofs across both fixes, all observed failing.
This commit is contained in:
@@ -1,5 +1,79 @@
|
||||
# felhom-agent — Changelog
|
||||
|
||||
## v0.107.0 — F-REBOOT + F-LEAK: the agent's authority over guest lifecycle (2026-07-28)
|
||||
|
||||
Two Campaign 8 findings, both about the agent being unable to act on a guest it owns.
|
||||
|
||||
### F-REBOOT — a guest rebooted mid-backup never came back
|
||||
|
||||
Fault 11: the backup SUCCEEDED and the guest was found `stopped` with 0 containers, no lock, nothing
|
||||
retrying — 9m47s of TOTAL appliance outage, every app down, and every alarm silent because nothing
|
||||
was broken except that the box was off. The trigger is ordinary: backups run overnight and
|
||||
`CONTEXT.md` records that the N100 needs BIOS AC-Power-Recovery *because power loss happens*.
|
||||
|
||||
**Why the existing recovery missed it.** `RecoverStaleLockedGuests` already does
|
||||
unlock → delete dangling snapshot → **start iff onboot**, and it is correct. It missed this by two
|
||||
narrow gaps: its predicate acts only on a guest holding a stale vzdump lock
|
||||
(`backup`/`snapshot-delete`) and fault 11's guest was stopped and **unlocked**; and it runs **once at
|
||||
agent startup**, on the load-bearing invariant that a lock present then is stale by definition. A
|
||||
guest that goes down while the agent is already up was never re-examined.
|
||||
|
||||
**The fix** (`internal/localapi/guestpower.go`) closes exactly those two gaps: a periodic sweep that
|
||||
acts on *should be running, is not, and is not locked*.
|
||||
|
||||
`onboot` is the "should be running" signal, and it is deliberately **not invented**: it is already
|
||||
the distinction `stalelock.go` uses for this same decision, it is 1 on customer guests and 0 on
|
||||
scratch/golden, and it is the flag `pve-guests` itself consults at host boot — so the agent agrees
|
||||
with the platform instead of keeping a private definition. The hub's desired-state `Run` is stronger
|
||||
but hub-dependent; `onboot` keeps working on a box that has lost hub contact, which is when an
|
||||
unattended appliance most needs to come back.
|
||||
|
||||
**Guards, because the trap here is F-CRIT-1's shape.** A guest the operator deliberately stopped must
|
||||
never be auto-started — fighting the operator makes maintenance impossible and is worse than the
|
||||
outage. So: `onboot:0` ⇒ never touched; a LOCKED guest is left to the stale-lock path (its lock may
|
||||
mean "a restore is writing my disks"); a guest with a vzdump genuinely in flight is left stopped (a
|
||||
stop-mode backup stops it on purpose); unprovable ownership ⇒ act on nothing; and unconfirmable
|
||||
backup state fails safe.
|
||||
|
||||
**Bounded retry** (Scenario C): 3 attempts at 1m/2m/4m, then ERROR naming the guest and stop. A
|
||||
healthy `pct start` of 9201 took ~25 s (observed twice), so even the first wait carries 2.4x headroom;
|
||||
three attempts bound disruption at ~7 minutes — inside the 9m47s this fixes — and never loop forever.
|
||||
|
||||
### F-LEAK — a failed restore-test could not destroy its own scratch
|
||||
|
||||
`DELETE /nodes/x/lxc/990000` → **403 missing privilege VM.Allocate**. It is **pool membership, not
|
||||
privsep**, and the live ACLs prove it: `VM.Allocate` is granted at `/pool/felhom` ONLY, never at `/`.
|
||||
A successful restore's `--pool felhom` makes the guest a member and inheritable; a FAILED restore
|
||||
never completes that, so the guest's own path resolves to `/` where the token holds nothing:
|
||||
|
||||
| path | privileges the token has |
|
||||
|---|---|
|
||||
| `/vms/990005` (non-member scratch) | `Datastore.Audit, SDN.Use, Sys.Audit` — no VM.Allocate |
|
||||
| `/pool/felhom` | full `FelhomAgentGuest` incl. `VM.Allocate` **and `Pool.Allocate`** |
|
||||
| `/vms/9201` (pool member) | inherits the full set |
|
||||
|
||||
**The fix needs NO new privilege.** `Pool.Allocate` is already held, so the teardown adopts the
|
||||
stranded scratch into the pool and retries the destroy once, which then authorizes via
|
||||
`/pool/felhom`. Nothing is widened; `PoolAddVMID` already existed for exactly this reason
|
||||
("membership is what lets the pool-scoped token reach the guest next time").
|
||||
|
||||
**The security guard is `scratchAdoptAllowed` — two independent checks, both required:** the journal
|
||||
entry must carry agent-created **scratch provenance**, AND the VMID must be inside the configured
|
||||
**scratch band**. Either alone would be enough to fix the leak; both are present because this
|
||||
function is the only thing between "clean up my own scratch" and "co-opt an arbitrary guest into the
|
||||
pool and delete it". Adopting a customer guest would hand the token destroy rights over it — far
|
||||
worse than a leaked scratch.
|
||||
|
||||
### Also
|
||||
Corrected the claim that the quiesce unquiesce is "guaranteed by defer"
|
||||
(`felhom-controller/.../quiesce.go`, and the generic advice in `PROMPT-TEMPLATE.md`): fault 10
|
||||
established that a SIGKILL runs no deferred function, and the guarantee is the crash marker plus
|
||||
`Recover()`.
|
||||
|
||||
Files: `internal/localapi/{guestpower.go (new),server.go}`, `internal/reconcile/restoretest.go`,
|
||||
`cmd/felhom-agent/main.go`, plus tests `internal/localapi/guestpower_test.go` and
|
||||
`internal/reconcile/scratch_adopt_test.go`. No wire/contract change; no new PVE grant.
|
||||
|
||||
## v0.106.0 — F-CRIT-2: a failed backup must not look like a fresh one (2026-07-28)
|
||||
|
||||
Campaign 8 killed the PBS daemon mid-upload. PBS published the aborted upload into the storage
|
||||
|
||||
Reference in New Issue
Block a user