v0.62.0 — A1: pool-membership ownership check for the stale-lock reaper
staleLockController.Guests() = ListLXC ∩ GET /pools/felhom members (ownership PROVEN via the pool registry, never assumed from enumeration scope); pool-read failure fail-safes the whole recovery through the existing guest-list guard. New Client.Pool read (needs Pool.Audit — host-install v1.9.0; Pool.Allocate does NOT satisfy it, spike T2). Composed pve:pool-read capability (non-critical) + --selftest pool-read line. Red-proofed negative tests drive the REAL controller over a broad-token-shaped fake. Per SPIKE-a1-pool-membership-read-2026-07-03.md; audit A1 (AUDIT-blast-radius-hostroot-localapi-2026-07-02). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
@@ -1,3 +1,34 @@
|
||||
## v0.62.0 — A1: pool-membership ownership check for the stale-lock reaper (2026-07-03)
|
||||
|
||||
Implements audit finding A1 (`AUDIT-blast-radius-hostroot-localapi-2026-07-02` §A) per the spike
|
||||
verdict (`SPIKE-a1-pool-membership-read-2026-07-03` — enumeration is pool-filtered under the scoped
|
||||
token, so this is defense-in-depth: a future broad-token deployment can no longer re-arm the reaper
|
||||
against co-tenant guests). Companion: host-install **v1.9.0** (`Pool.Audit` added to
|
||||
`FelhomAgentGuest`) — **rescope BEFORE deploying this agent**, else the reaper fail-safes (skips)
|
||||
until the ACL catches up.
|
||||
|
||||
- **`Client.Pool`** (proxmox/query.go): `GET /pools/{name}` → `PoolInfo{PoolID, Members[]{VMID,Type}}`.
|
||||
Requires `Pool.Audit` at `/pool/{name}`; `Pool.Allocate` does NOT satisfy the read (spike T2).
|
||||
- **`staleLockController.Guests()`** (localapi/stalelock.go): now returns `ListLXC ∩ pool members`
|
||||
(nonzero-vmid, non-storage entries only). Ownership is PROVEN via the pool registry, never assumed
|
||||
from enumeration scope. A pool-read failure returns a wrapped error ("pool membership read
|
||||
(pool=felhom): …") that rides the existing "guest list unavailable — skipping recovery" guard —
|
||||
fail-safe: NO unlock/snapshot-delete/start on ANY guest, never a fallback to the unfiltered list.
|
||||
One new INFO line per scan: `stale-lock: scanning pool guests` (pool, listed, scanned) — emitted
|
||||
by the controller (the unchanged `StaleLockController` seam can't carry the pre-intersect count).
|
||||
- **`NewStaleLockController`** gains `(pool string, logger *slog.Logger)`; main.go threads
|
||||
`reconcile.DefaultPool`.
|
||||
- **Capability surfacing**: the hub-report prober is now a composed closure — the sudo manifest
|
||||
probe + one `pve:pool-read` status (non-critical; degraded ⇒ reaper is fail-safed, visible on the
|
||||
report, no operator page). Composed in main.go; `internal/capability/` untouched.
|
||||
- **`--selftest`**: new "pool read" line (pool id + member count + guest members).
|
||||
- **Tests** (stalelock_pool_test.go, driving the REAL controller over a broad-token-shaped fake):
|
||||
`TestStaleLock_ForeignGuestNotReaped` (red-proved: intersect removed ⇒ FAILS with
|
||||
`pct unlock 5000` recorded), `TestStaleLock_PoolGuestStillReaped` (anti-over-filter),
|
||||
`TestStaleLock_PoolReadFails_SkipsAll` (red-proved: fallback-to-unfiltered ⇒ FAILS with mutations
|
||||
recorded), `TestStaleLockController_GuestsIntersect` (storage-member + empty-pool edges). The 9
|
||||
existing Server-level stalelock tests pass unmodified.
|
||||
|
||||
## docs — CLAUDE.md refresh: stable orientation, complete layout (2026-07-03)
|
||||
|
||||
No code change, no version bump. Deleted the version-pinned "Current: v0.31.0" narrative and the
|
||||
|
||||
+6
-5
@@ -5,6 +5,12 @@
|
||||
|
||||
## Current
|
||||
|
||||
- **v0.62.0** (2026-07-03) — **audit A1 RESOLVED**: the stale-lock reaper's scan is now
|
||||
pool-intersected (`staleLockController.Guests()` = `ListLXC` ∩ `Client.Pool("felhom")` members),
|
||||
fail-safe skip on pool-read failure; `pve:pool-read` capability (non-critical) + `--selftest`
|
||||
"pool read" line. Companion host-install **v1.9.0** adds `Pool.Audit` to `FelhomAgentGuest` —
|
||||
**deploy order on any box: rescope ACL first, then this agent.** Per
|
||||
`SPIKE-a1-pool-membership-read-2026-07-03.md`; red-proofed tests in stalelock_pool_test.go.
|
||||
- **2026-07-03 — CLAUDE.md refreshed**: version narrative removed (state lives HERE + CHANGELOG top), layout completed (all 17 internal packages + cmd/felhom-opsign); deploy runbook now in the `felhom-build-deploy` skill (`felhom.eu/skills/`).
|
||||
- **2026-07-03 — `REUSE.md` exists at the repo root** (canonical helpers / format-safety guards / traps / seams, code-verified); maintenance rule active: update it in the same commit that changes a shared helper.
|
||||
- **v0.61.0** (2026-07-03) — blast-radius audit fixes **B1 + D1 + D2 + D3** from
|
||||
@@ -17,11 +23,6 @@
|
||||
|
||||
## Open threads
|
||||
|
||||
- **A1 (LOW, pending SPIKE — not fixed in v0.61.0):** stale-lock recovery
|
||||
(`internal/localapi/stalelock.go`) reaps guests by a pool-blind `ListLXC` scan, not ownership.
|
||||
Contained by the pool-scoped token today; **must be fixed before any broad/root-token deployment.**
|
||||
It is a spike, not a patch: the role lacks `Pool.Audit`, so there is no cheap authorized
|
||||
pool-membership read — which read to use is the open design question.
|
||||
- Deferred audit items (housekeeping/design, all INFO): C1 (controller-swap version floor), C2 (NAS
|
||||
server allowlist), A2 (gate journal cross-check), B2–B5, E1/E2.
|
||||
- Drive-enrollment leftovers: (a) `runStorageInit` slow-device detached-format polling; (b) Impl-3
|
||||
|
||||
@@ -68,7 +68,7 @@
|
||||
| `Server.scopedFromBody` | internal/localapi/server.go | `scopedFromBody(w, bodyVMID, tokenVMID, path) bool` | POST-body vmid self-scope check | Call right after decode; false = already 403'd |
|
||||
| `decodeBody` / `writeOK` / `writeErr` / `writeStatus` | internal/localapi/server.go | `writeStatus(w, code, ok, data, errMsg)` | ALL local-API JSON I/O | Do not hand-roll response envelopes |
|
||||
| `EnsureLeaf` | internal/localapi/cert.go | `EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)` | pinned self-signed leaf | `generated=true` invalidates every issued bootstrap pin — log LOUD (B.1) |
|
||||
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 OPEN: pool-blind `ListLXC` scan |
|
||||
| `Server.RecoverStaleLockedGuests` | internal/localapi/stalelock.go | `RecoverStaleLockedGuests(ctx)` | startup stale vzdump-lock heal (F2-b) | Clears ONLY `backup`/`snapshot-delete`, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (`ListLXC` ∩ `Client.Pool`), fail-safe skip on pool-read failure |
|
||||
| `ControllerSwapper.Swap` + `ValidControllerImage` | internal/localapi/controllerswap.go | `Swap(ctx, vmid, target) *ControllerSwapState` | agent-owned controller image swap + rollback | Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need `verifyDwell` |
|
||||
|
||||
### Proxmox client / hub / PBS / provisioning
|
||||
@@ -76,6 +76,7 @@
|
||||
| Symbol | File | Short signature | Use for | Gotchas |
|
||||
|---|---|---|---|---|
|
||||
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
|
||||
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
|
||||
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
|
||||
| `TLSConfig.build` / `normalizeFingerprint` | internal/proxmox/tls.go | `build() (*tls.Config, error)` | PVE leaf-cert SHA-256 pinning | No insecure default |
|
||||
| `pinnedTLS` | internal/pbs/pin.go | `pinnedTLS(fingerprint) (*tls.Config, error)` | PBS leaf pinning | Same model as PVE; 64-hex fingerprint normalized |
|
||||
@@ -116,7 +117,7 @@
|
||||
| Calling `Client.DestroyLXC`/`Vzdump`/`SetConfig` outside the gate/queue/journal | skips classification, signature, per-guest serialization, crash recovery | `reconcile.Engine` paths / `RunSignedJob`; queue via `Queue.Submit` |
|
||||
| `GuestBinder.AttachBind`/`DetachBind` (per-drive `pct set -mpN`) | legacy model; a missing bind source can brick guest boot (C1) | `AttachDrive`/`DetachDrive` (intermediary model) |
|
||||
| `isHostMountpoint` to reconcile bind state | boolean can't converge stacked double-binds (the /mnt doubling bug) | `countHostMounts` normalization inside `AttachDrive` |
|
||||
| Copying `RecoverStaleLockedGuests`' pool-blind `ListLXC` scan | audit A1: acts on guests the agent doesn't own — contained ONLY by today's pool-scoped token | don't extend it; A1 spike must land before any broad-token feature builds on that scan |
|
||||
| Acting on a raw `ListLXC` list as if it were "guests the agent owns" | audit A1 (pre-v0.62.0 the stale-lock reaper did exactly this — contained only by the pool-scoped token) | ownership must be PROVEN: intersect with `Client.Pool` membership like `staleLockController.Guests()` (v0.62.0), fail-safe on read failure |
|
||||
|
||||
## 4. Seams & interfaces (testing + cross-repo)
|
||||
|
||||
@@ -127,7 +128,7 @@
|
||||
| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go |
|
||||
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
|
||||
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
|
||||
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner) | `fakeStaleLock` internal/localapi/stalelock_test.go |
|
||||
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
|
||||
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
|
||||
| `reconcile.OpVerifier` | internal/reconcile/gate.go | `*authz.Verifier` | fake verifier in internal/reconcile gate tests |
|
||||
| `signedjobs.WipeOps` / `Executor` (`ExecutorChain`) | internal/signedjobs/wipe.go + runner.go | `*storage.SudoHostOps`; `WipeExecutor`+`DecommissionExecutor` | internal/signedjobs wipe/runner/decommission tests |
|
||||
|
||||
+52
-14
@@ -45,7 +45,7 @@ import (
|
||||
|
||||
// version is the agent version. Overridable at build time with
|
||||
// -ldflags "-X main.version=<v>"; defaults to the in-repo CHANGELOG version.
|
||||
var version = "0.61.0"
|
||||
var version = "0.62.0"
|
||||
|
||||
// runGuestHook is the PVE pre-start hook body (`felhom-agent guest-hook <vmid> <phase>`). On the
|
||||
// pre-start phase it creates placeholder dirs for any absent bind-mount source so the guest always boots
|
||||
@@ -274,6 +274,28 @@ func (r *gateRemounter) Remount(ctx context.Context, t storage.KnownTarget) {
|
||||
r.logger.Info("storage: re-mounted returned target", "target", t.Name, "where", t.MountPath)
|
||||
}
|
||||
|
||||
// poolReadStatus probes the felhom-pool membership read (GET /pools/felhom) the stale-lock reaper's
|
||||
// ownership scoping depends on (v0.62.0, audit A1). Needs `Pool.Audit` at `/pool/felhom` — granted
|
||||
// by host-install v1.9.0; on an older ACL this reports degraded and the reaper fail-safes (skips).
|
||||
func poolReadStatus(ctx context.Context, px *proxmox.Client) capability.Status {
|
||||
s := capability.Status{
|
||||
Name: "pve:pool-read",
|
||||
Feature: "stale-lock recovery scoping (pool ownership check)",
|
||||
Critical: false,
|
||||
Status: capability.StatusOK,
|
||||
}
|
||||
if px == nil {
|
||||
s.Status, s.Reason = capability.StatusDegraded, "not configured"
|
||||
return s
|
||||
}
|
||||
pctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
if _, err := px.Pool(pctx, reconcile.DefaultPool); err != nil {
|
||||
s.Status, s.Reason = capability.StatusDegraded, err.Error()
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// logCapabilities logs the privileged-capability self-check at startup: one INFO summary, plus an
|
||||
// ERROR per degraded capability naming the gated feature (so a missing grant is loud at cutover,
|
||||
// not days later). It never exits — serve-degraded.
|
||||
@@ -336,8 +358,14 @@ func runDaemon(cfg config.Config, logger *slog.Logger) int {
|
||||
// startup (loud on any denial) and attach the snapshot to every hub report; the hub owns the
|
||||
// ok→degraded alert. Serve-degraded — a missing grant never blocks startup.
|
||||
capProber := capability.Prober{Runner: &proxmox.ExecRunner{Mode: proxmox.RunnerDirect}}
|
||||
logCapabilities(capProber.Probe(context.Background()), logger)
|
||||
collector.SetCapabilityProber(capProber.Probe)
|
||||
// A1 (v0.62.0): compose the PVE pool-read check AROUND the sudo prober (an API read does not
|
||||
// belong inside the sudo-policy probe). Non-critical: a degraded pool read means the stale-lock
|
||||
// reaper fail-safes (locks stay uncleared) — visible on the hub report, no operator page.
|
||||
probeAll := func(ctx context.Context) []capability.Status {
|
||||
return append(capProber.Probe(ctx), poolReadStatus(ctx, px))
|
||||
}
|
||||
logCapabilities(probeAll(context.Background()), logger)
|
||||
collector.SetCapabilityProber(probeAll)
|
||||
loop := hub.NewLoop(collector, client, time.Duration(hcfg.PollSeconds)*time.Second, logger)
|
||||
interval := time.Duration(hcfg.PollSeconds) * time.Second
|
||||
|
||||
@@ -766,25 +794,26 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
|
||||
Backups: runner,
|
||||
Store: store,
|
||||
Storage: observer,
|
||||
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
||||
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
|
||||
HostReader: storage.NewProcHostReader(), // Impl-2b: durableIDForMount raw-mount fallback + role gate
|
||||
Tokens: tokens,
|
||||
BackupCadence: cfg.Backup.BackupCadence(),
|
||||
// Disk management (slice 8C): the privileged host surface + the data-bearing wipe gate.
|
||||
Disks: hostOps,
|
||||
DiskGate: storageGateAdapter{gate: gate, hostID: cfg.Hub.HostID},
|
||||
Guests2: px,
|
||||
GuestAttach: guestBinder, // slice 10 P2: bind enrolled data drives into the guest
|
||||
Disks: hostOps,
|
||||
DiskGate: storageGateAdapter{gate: gate, hostID: cfg.Hub.HostID},
|
||||
Guests2: px,
|
||||
GuestAttach: guestBinder, // slice 10 P2: bind enrolled data drives into the guest
|
||||
// Network storage (NAS) — Part A1: the privileged host network-mount surface (NFS/SMB automount).
|
||||
NetStorage: hostOps,
|
||||
SmbCredsDir: cfg.Privileged.SmbCredsDir,
|
||||
NetStorage: hostOps,
|
||||
SmbCredsDir: cfg.Privileged.SmbCredsDir,
|
||||
ControllerSwap: guestBinder, // Phase 1: agentic controller update — in-guest image swap
|
||||
// F2-b: recover a guest left with a stale vzdump lock by a reboot-during-backup. Reads + start
|
||||
// go through the API client; the `pct unlock` is the one fenced root-CLI op (no API equivalent).
|
||||
StaleLock: localapi.NewStaleLockController(px, &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}),
|
||||
Intent: intent, // slice 10 P3: record enroll/eject intent for self-heal
|
||||
GuestBinds: guestBinds, // F9: per-guest bind record for the startup re-assert
|
||||
FormatJobs: formatJobs, // F20-BUG3: detached-format job record + restart recovery
|
||||
// A1 (v0.62.0): the scan is restricted to felhom-pool members (ownership proven, not assumed).
|
||||
StaleLock: localapi.NewStaleLockController(px, &proxmox.ExecRunner{Mode: gaMode, SudoPath: cfg.Privileged.SudoPath}, reconcile.DefaultPool, logger),
|
||||
Intent: intent, // slice 10 P3: record enroll/eject intent for self-heal
|
||||
GuestBinds: guestBinds, // F9: per-guest bind record for the startup re-assert
|
||||
FormatJobs: formatJobs, // F20-BUG3: detached-format job record + restart recovery
|
||||
|
||||
// Host metrics (slice 9): the shared collector serves GET /host/metrics — a fresh host +
|
||||
// per-storage view to the customer's monitoring page (reuses the slice-4 collector).
|
||||
@@ -1756,6 +1785,15 @@ func runSelftestRead(ctx context.Context, cfg config.Config, logger *slog.Logger
|
||||
fmt.Printf(" - %d %q status=%s\n", g.VMID, g.Name, g.Status)
|
||||
}
|
||||
}
|
||||
// A1 (v0.62.0): the stale-lock reaper's ownership registry — needs Pool.Audit (host-install v1.9.0+).
|
||||
if p, err := client.Pool(ctx, reconcile.DefaultPool); report("pool read", err) {
|
||||
fmt.Printf(" [ ok ] %-14s pool %q, %d member(s)\n", "pool read", p.PoolID, len(p.Members))
|
||||
for _, m := range p.Members {
|
||||
if m.VMID != 0 && m.Type != "storage" {
|
||||
fmt.Printf(" - %d type=%s\n", m.VMID, m.Type)
|
||||
}
|
||||
}
|
||||
}
|
||||
if ss, err := client.NodeStorage(ctx); report("storage", err) {
|
||||
fmt.Printf(" [ ok ] %-14s %d store(s)\n", "storage", len(ss))
|
||||
for _, s := range ss {
|
||||
|
||||
@@ -2,6 +2,8 @@ package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strconv"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
@@ -22,6 +24,10 @@ import (
|
||||
// Scope is deliberately narrow: only the two vzdump-related locks are ever touched. migrate/disk/create/
|
||||
// rollback/mounted/snapshot locks are left alone (they belong to a different operation, mid-flight or
|
||||
// needing different handling). The recovery is idempotent and never touches a guest without a stale lock.
|
||||
//
|
||||
// Ownership (audit A1, v0.62.0): the scan covers ONLY felhom-pool members — the production controller
|
||||
// intersects ListLXC with GET /pools/{pool} membership, so even under a broad token the reaper can
|
||||
// never unlock/start a co-tenant's guest. A failed pool read fails safe (whole recovery skipped).
|
||||
|
||||
// staleBackupLocks are the lock values an interrupted vzdump can leave. ONLY these are cleared.
|
||||
var staleBackupLocks = map[string]bool{
|
||||
@@ -134,11 +140,14 @@ func (s *Server) recoverOneStaleLock(ctx context.Context, g proxmox.Guest) {
|
||||
type staleLockController struct {
|
||||
px staleLockAPI
|
||||
runner proxmox.Runner
|
||||
pool string // ownership registry: only members of this PVE pool are ever scanned (A1)
|
||||
logger *slog.Logger // the one success-path scan-summary line; nil = silent
|
||||
}
|
||||
|
||||
// staleLockAPI is the subset of *proxmox.Client the controller uses (kept narrow for clarity/testing).
|
||||
type staleLockAPI interface {
|
||||
ListLXC(ctx context.Context) ([]proxmox.Guest, error)
|
||||
Pool(ctx context.Context, name string) (proxmox.PoolInfo, error)
|
||||
GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
|
||||
ListSnapshots(ctx context.Context, vmid int) ([]proxmox.Snapshot, error)
|
||||
ListRunningTasks(ctx context.Context) ([]proxmox.TaskStatus, error)
|
||||
@@ -148,16 +157,48 @@ type staleLockAPI interface {
|
||||
}
|
||||
|
||||
// NewStaleLockController builds the production controller. Returns nil if px or runner is nil (the
|
||||
// feature then stays unwired and RecoverStaleLockedGuests is a no-op).
|
||||
func NewStaleLockController(px staleLockAPI, runner proxmox.Runner) StaleLockController {
|
||||
// feature then stays unwired and RecoverStaleLockedGuests is a no-op). pool names the PVE pool the
|
||||
// scan is restricted to (reconcile.DefaultPool in production).
|
||||
func NewStaleLockController(px staleLockAPI, runner proxmox.Runner, pool string, logger *slog.Logger) StaleLockController {
|
||||
if px == nil || runner == nil {
|
||||
return nil
|
||||
}
|
||||
return &staleLockController{px: px, runner: runner}
|
||||
return &staleLockController{px: px, runner: runner, pool: pool, logger: logger}
|
||||
}
|
||||
|
||||
// Guests returns ListLXC ∩ the felhom pool's members (audit A1: ownership is PROVEN via the pool
|
||||
// registry, never assumed from enumeration scope). Under the pool-scoped token the intersect is a
|
||||
// no-op (ListLXC is already pool-filtered — spike T1); under a broad token it is the guard that
|
||||
// keeps the reaper off co-tenant guests. A pool-read failure returns an error — the caller's
|
||||
// existing "guest list unavailable — skipping recovery" guard then fail-safes the whole scan
|
||||
// (unknown ownership ⇒ don't act, mirroring reconcile/recover.go's proof-of-launch gate). NEVER
|
||||
// fall back to the unfiltered ListLXC list on error.
|
||||
func (c *staleLockController) Guests(ctx context.Context) ([]proxmox.Guest, error) {
|
||||
return c.px.ListLXC(ctx)
|
||||
lxc, err := c.px.ListLXC(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pool, err := c.px.Pool(ctx, c.pool)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pool membership read (pool=%s): %w", c.pool, err)
|
||||
}
|
||||
// A pool can hold storages too (type "storage", no vmid) — membership is nonzero-vmid guests only.
|
||||
members := make(map[int]bool, len(pool.Members))
|
||||
for _, m := range pool.Members {
|
||||
if m.VMID != 0 && m.Type != "storage" {
|
||||
members[m.VMID] = true
|
||||
}
|
||||
}
|
||||
owned := lxc[:0:0]
|
||||
for _, g := range lxc {
|
||||
if members[g.VMID] {
|
||||
owned = append(owned, g)
|
||||
}
|
||||
}
|
||||
if c.logger != nil {
|
||||
c.logger.Info("stale-lock: scanning pool guests", "pool", c.pool, "listed", len(lxc), "scanned", len(owned))
|
||||
}
|
||||
return owned, nil
|
||||
}
|
||||
|
||||
func (c *staleLockController) Lock(ctx context.Context, vmid int) (string, bool, error) {
|
||||
|
||||
@@ -0,0 +1,224 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
|
||||
)
|
||||
|
||||
// A1 (audit AUDIT-blast-radius-hostroot-localapi-2026-07-02, spike SPIKE-a1-pool-membership-read):
|
||||
// these tests drive the REAL staleLockController (not the Server-level fakeStaleLock) so the
|
||||
// pool-membership intersect in Guests() is the code under test. The staleLockAPI fake simulates a
|
||||
// BROAD token (ListLXC returns non-pool guests too — the exploit precondition); the recording
|
||||
// runner captures every `pct unlock` the controller would execute.
|
||||
|
||||
// fakeStaleLockAPI is a scripted staleLockAPI: broad-token-shaped ListLXC + a scripted pool read,
|
||||
// with the mutating calls recorded.
|
||||
type fakeStaleLockAPI struct {
|
||||
lxc []proxmox.Guest
|
||||
pool proxmox.PoolInfo
|
||||
poolErr error
|
||||
cfg map[int]proxmox.GuestConfig
|
||||
snaps map[int][]proxmox.Snapshot
|
||||
tasks []proxmox.TaskStatus
|
||||
|
||||
delsnap []int
|
||||
started []int
|
||||
}
|
||||
|
||||
func (f *fakeStaleLockAPI) ListLXC(context.Context) ([]proxmox.Guest, error) { return f.lxc, nil }
|
||||
|
||||
func (f *fakeStaleLockAPI) Pool(_ context.Context, name string) (proxmox.PoolInfo, error) {
|
||||
if f.poolErr != nil {
|
||||
return proxmox.PoolInfo{}, f.poolErr
|
||||
}
|
||||
return f.pool, nil
|
||||
}
|
||||
|
||||
func (f *fakeStaleLockAPI) GuestConfig(_ context.Context, vmid int) (proxmox.GuestConfig, error) {
|
||||
return f.cfg[vmid], nil
|
||||
}
|
||||
|
||||
func (f *fakeStaleLockAPI) ListSnapshots(_ context.Context, vmid int) ([]proxmox.Snapshot, error) {
|
||||
return f.snaps[vmid], nil
|
||||
}
|
||||
|
||||
func (f *fakeStaleLockAPI) ListRunningTasks(context.Context) ([]proxmox.TaskStatus, error) {
|
||||
return f.tasks, nil
|
||||
}
|
||||
|
||||
func (f *fakeStaleLockAPI) DeleteSnapshot(_ context.Context, vmid int, snapname string) (string, error) {
|
||||
f.delsnap = append(f.delsnap, vmid)
|
||||
return "", nil // synchronous — no WaitTask
|
||||
}
|
||||
|
||||
func (f *fakeStaleLockAPI) Start(_ context.Context, vmid int) (string, error) {
|
||||
f.started = append(f.started, vmid)
|
||||
return "", nil
|
||||
}
|
||||
|
||||
func (f *fakeStaleLockAPI) WaitTask(_ context.Context, upid string, _ proxmox.WaitOptions) (proxmox.TaskStatus, error) {
|
||||
return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, nil
|
||||
}
|
||||
|
||||
// recordingRunner captures every fenced-CLI invocation (the controller's `pct unlock`).
|
||||
type recordingRunner struct {
|
||||
calls [][]string
|
||||
}
|
||||
|
||||
func (r *recordingRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
|
||||
r.calls = append(r.calls, append([]string{name}, args...))
|
||||
return nil, nil, nil
|
||||
}
|
||||
|
||||
func (r *recordingRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
|
||||
return r.Run(ctx, name, args...)
|
||||
}
|
||||
|
||||
func (r *recordingRunner) unlocked(vmid string) bool {
|
||||
for _, c := range r.calls {
|
||||
if len(c) == 3 && c[0] == "pct" && c[1] == "unlock" && c[2] == vmid {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// staleCfg builds a GuestConfig whose Lock()/OnBoot() read the given values (both live in Extra).
|
||||
func staleCfg(lock string, onboot bool) proxmox.GuestConfig {
|
||||
extra := map[string]json.RawMessage{}
|
||||
if lock != "" {
|
||||
extra["lock"] = json.RawMessage(`"` + lock + `"`)
|
||||
}
|
||||
if onboot {
|
||||
extra["onboot"] = json.RawMessage(`1`)
|
||||
}
|
||||
return proxmox.GuestConfig{Extra: extra}
|
||||
}
|
||||
|
||||
// poolScanServer wires the REAL production controller (fake API + recording runner) into a Server.
|
||||
func poolScanServer(api *fakeStaleLockAPI, runner *recordingRunner) *Server {
|
||||
ctrl := NewStaleLockController(api, runner, "felhom", nil)
|
||||
return &Server{staleLock: ctrl, logger: slog.New(slog.NewTextHandler(io.Discard, nil))}
|
||||
}
|
||||
|
||||
// TestStaleLock_ForeignGuestNotReaped forecloses the A1 exploit: under a broad token, a co-tenant
|
||||
// (non-pool) guest with the exact stale-lock signature (snapshot-delete lock + dangling vzdump
|
||||
// snapshot + no vzdump task visible yet) must be EXCLUDED by the pool intersect — never unlocked,
|
||||
// never snapshot-deleted, never started.
|
||||
func TestStaleLock_ForeignGuestNotReaped(t *testing.T) {
|
||||
api := &fakeStaleLockAPI{
|
||||
lxc: []proxmox.Guest{ // broad-token-shaped: the foreign guest IS enumerated
|
||||
{VMID: 9201, Status: "stopped"},
|
||||
{VMID: 5000, Status: "stopped"},
|
||||
},
|
||||
pool: proxmox.PoolInfo{PoolID: "felhom", Members: []proxmox.PoolMember{{VMID: 9201, Type: "lxc"}}},
|
||||
cfg: map[int]proxmox.GuestConfig{
|
||||
9201: staleCfg("", false), // healthy pool member
|
||||
5000: staleCfg("snapshot-delete", true), // the co-tenant's interrupted backup
|
||||
},
|
||||
snaps: map[int][]proxmox.Snapshot{5000: {{Name: "vzdump"}}},
|
||||
}
|
||||
runner := &recordingRunner{}
|
||||
poolScanServer(api, runner).RecoverStaleLockedGuests(context.Background())
|
||||
|
||||
if runner.unlocked("5000") {
|
||||
t.Fatalf("A1 exploit: foreign guest 5000 was unlocked; runner calls=%v", runner.calls)
|
||||
}
|
||||
if contains(api.delsnap, 5000) {
|
||||
t.Fatalf("A1 exploit: foreign guest 5000's vzdump snapshot was deleted; delsnap=%v", api.delsnap)
|
||||
}
|
||||
if contains(api.started, 5000) {
|
||||
t.Fatalf("A1 exploit: foreign guest 5000 was force-started; started=%v", api.started)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStaleLock_PoolGuestStillReaped is the over-filtering companion: an owned (pool-member) guest
|
||||
// with the same stale signature IS fully recovered — the filter must not be "reap nothing".
|
||||
func TestStaleLock_PoolGuestStillReaped(t *testing.T) {
|
||||
api := &fakeStaleLockAPI{
|
||||
lxc: []proxmox.Guest{
|
||||
{VMID: 9201, Status: "stopped"},
|
||||
{VMID: 5000, Status: "stopped"},
|
||||
},
|
||||
pool: proxmox.PoolInfo{PoolID: "felhom", Members: []proxmox.PoolMember{{VMID: 9201, Type: "lxc"}}},
|
||||
cfg: map[int]proxmox.GuestConfig{
|
||||
9201: staleCfg("snapshot-delete", true), // the F2-b recovery case, on the OWNED guest
|
||||
5000: staleCfg("", false),
|
||||
},
|
||||
snaps: map[int][]proxmox.Snapshot{9201: {{Name: "vzdump"}}},
|
||||
}
|
||||
runner := &recordingRunner{}
|
||||
poolScanServer(api, runner).RecoverStaleLockedGuests(context.Background())
|
||||
|
||||
if !runner.unlocked("9201") {
|
||||
t.Fatalf("owned guest 9201 must still be unlocked; runner calls=%v", runner.calls)
|
||||
}
|
||||
if !contains(api.delsnap, 9201) {
|
||||
t.Fatalf("owned guest 9201's dangling snapshot must be deleted; delsnap=%v", api.delsnap)
|
||||
}
|
||||
if !contains(api.started, 9201) {
|
||||
t.Fatalf("owned guest 9201 (onboot, stopped) must be started; started=%v", api.started)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStaleLock_PoolReadFails_SkipsAll: when ownership can't be PROVEN (pool read errors — 403 on a
|
||||
// pre-v1.9.0 ACL, timeout, parse failure), the whole recovery fail-safes: ZERO mutations on ANY
|
||||
// guest, never a fallback to the unfiltered list.
|
||||
func TestStaleLock_PoolReadFails_SkipsAll(t *testing.T) {
|
||||
api := &fakeStaleLockAPI{
|
||||
lxc: []proxmox.Guest{{VMID: 9201, Status: "stopped"}},
|
||||
poolErr: errors.New(`proxmox: GET /pools/felhom: 403 Permission check failed (/pool/felhom, Pool.Audit)`),
|
||||
cfg: map[int]proxmox.GuestConfig{9201: staleCfg("snapshot-delete", true)},
|
||||
snaps: map[int][]proxmox.Snapshot{9201: {{Name: "vzdump"}}},
|
||||
}
|
||||
runner := &recordingRunner{}
|
||||
poolScanServer(api, runner).RecoverStaleLockedGuests(context.Background())
|
||||
|
||||
if len(runner.calls) != 0 || len(api.delsnap) != 0 || len(api.started) != 0 {
|
||||
t.Fatalf("pool-read failure must skip ALL recovery; runner=%v delsnap=%v started=%v",
|
||||
runner.calls, api.delsnap, api.started)
|
||||
}
|
||||
|
||||
// The wrapped error the :64 guard logs must name the pool read (operator diagnosability).
|
||||
_, err := NewStaleLockController(api, runner, "felhom", nil).Guests(context.Background())
|
||||
if err == nil || !strings.Contains(err.Error(), "pool membership read (pool=felhom)") {
|
||||
t.Fatalf("Guests() error must name the pool read; got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStaleLockController_GuestsIntersect covers the intersect edges (§8): storage-type pool members
|
||||
// and zero-vmid entries never grant membership; an EMPTY pool is a valid empty scan, not an error.
|
||||
func TestStaleLockController_GuestsIntersect(t *testing.T) {
|
||||
api := &fakeStaleLockAPI{
|
||||
lxc: []proxmox.Guest{{VMID: 9201}, {VMID: 5000}},
|
||||
pool: proxmox.PoolInfo{PoolID: "felhom", Members: []proxmox.PoolMember{
|
||||
{VMID: 9201, Type: "lxc"},
|
||||
{VMID: 0, Type: "storage"}, // a pool-attached storage — must not grant vmid-0 membership
|
||||
}},
|
||||
}
|
||||
ctrl := NewStaleLockController(api, &recordingRunner{}, "felhom", nil)
|
||||
|
||||
got, err := ctrl.Guests(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("Guests: %v", err)
|
||||
}
|
||||
if len(got) != 1 || got[0].VMID != 9201 {
|
||||
t.Fatalf("intersect must keep exactly the guest pool members; got %v", got)
|
||||
}
|
||||
|
||||
api.pool = proxmox.PoolInfo{PoolID: "felhom"} // empty pool (a box before first provision)
|
||||
got, err = ctrl.Guests(context.Background())
|
||||
if err != nil {
|
||||
t.Fatalf("empty pool must not error: %v", err)
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("empty pool ⇒ empty scan; got %v", got)
|
||||
}
|
||||
}
|
||||
@@ -35,6 +35,15 @@ func (c *Client) ListLXC(ctx context.Context) ([]Guest, error) {
|
||||
return gs, c.get(ctx, "/nodes/"+c.node+"/lxc", &gs)
|
||||
}
|
||||
|
||||
// Pool returns GET /pools/{name} (the pool's membership — the stale-lock reaper's ownership
|
||||
// registry, audit A1). Requires `Pool.Audit` at `/pool/{name}` — NOTE: `Pool.Allocate` does NOT
|
||||
// satisfy the read (spike SPIKE-a1-pool-membership-read T2: the live 403 named Pool.Audit with
|
||||
// Allocate already granted). Host-install v1.9.0+ grants it in the FelhomAgentGuest role.
|
||||
func (c *Client) Pool(ctx context.Context, name string) (PoolInfo, error) {
|
||||
var p PoolInfo
|
||||
return p, c.get(ctx, "/pools/"+url.PathEscape(name), &p)
|
||||
}
|
||||
|
||||
// GuestStatus returns GET /nodes/{node}/lxc/{vmid}/status/current. The API body
|
||||
// has no vmid field (it is in the path), so it is set from the argument.
|
||||
func (c *Client) GuestStatus(ctx context.Context, vmid int) (Guest, error) {
|
||||
|
||||
@@ -76,6 +76,21 @@ type Guest struct {
|
||||
Uptime int64 `json:"uptime"`
|
||||
}
|
||||
|
||||
// PoolInfo is GET /pools/{poolid} — the pool's identity + membership. The stale-lock recovery
|
||||
// uses it as the ownership registry: only pool members are ever scanned (audit A1).
|
||||
type PoolInfo struct {
|
||||
PoolID string `json:"poolid"`
|
||||
Members []PoolMember `json:"members"`
|
||||
}
|
||||
|
||||
// PoolMember is one entry of PoolInfo.Members. A pool can hold guests AND storages; storage
|
||||
// entries carry type "storage" and no vmid, so membership checks must filter on both (spike
|
||||
// SPIKE-a1-pool-membership-read §8).
|
||||
type PoolMember struct {
|
||||
VMID int `json:"vmid"`
|
||||
Type string `json:"type"` // "lxc" | "qemu" | "storage"
|
||||
}
|
||||
|
||||
// GuestConfig is GET /nodes/{node}/lxc/{vmid}/config. The config surface is
|
||||
// dynamic (net0..netN, mp0..mpN, unusedN), so known fields are typed and the full
|
||||
// raw map is preserved in Extra for the dynamic ones.
|
||||
|
||||
Reference in New Issue
Block a user