v0.32.0: self-serve decommission endpoint + intent-aware re-assert (B2a)

POST /disks/decommission mirrors eject (withGuest, user-data role gate) — no
operator signature, non-destructive (never formats): sets IntentDecommissioned,
prunes the GuestBindStore entry, unmounts. ReassertGuestBinds is now intent-aware
(skip non-enrolled) so a decommissioned-but-present drive never auto-rebinds on
agent restart — the load-bearing F9-reconnect fix. GuestBindStore.Remove added.
Operator-signed DecommissionExecutor + classify untouched. Non-hollow tests incl.
the intent-aware reassert companion (mutation-proven to fail on intent-blind code).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-14 19:55:17 +02:00
parent 8e6d00a57f
commit f43697c881
6 changed files with 397 additions and 49 deletions
+27
View File
@@ -3,6 +3,33 @@
All notable changes to **felhom-agent** are recorded here. Update on every code
change that gets pushed.
## v0.32.0 — self-serve decommission + intent-aware re-assert (B2a) (2026-06-14)
Customer-self-serve storage decommission (no operator signature; non-destructive — never formats),
plus the load-bearing fix that keeps a decommissioned drive from auto-rebinding into the guest.
- **`POST /disks/decommission`** (`internal/localapi/disks.go` `handleDiskDecommission`, route in
`server.go`) — mirrors `handleDiskEject` exactly: `withGuest` self-scoping, `scopedFromBody`, and the
same **user-data role gate** (`roleForMountPath` must be `RoleUserData`, else 403; fail-safe-to-
protected on ambiguity) so a compromised controller can't decommission system/backup storage. It
records a PERMANENT `IntentDecommissioned`, prunes the `GuestBindStore` entry (hygiene), and unmounts
(so the drive is physically removable). It **NEVER** calls any format/mkfs path — the data stays on
the drive. The operator-signed `DecommissionExecutor` + `reconcile.Classify` classification are
untouched (the absent-drive/DR route).
- **`ReassertGuestBinds` is now intent-aware** (THE correctness fix): the startup re-assert skips any
durable-id whose intent is not `enrolled`, so a decommissioned- (or ejected-) but-still-present drive
is never auto-rebound into the guest on agent restart. A nil intent store falls back to legacy
bind-all (matching the watchdog's nil-intent rule). Covers both the self-serve and the operator-
signed decommission paths (both land on `IntentDecommissioned`).
- **`GuestBindStore.Remove(vmid, durableID)`** (`internal/localapi/guestbindstore.go`) — idempotent
(absent = no-op), atomic tmp+rename like `Record`; drops the vmid key when its set empties. Re-enroll
re-`Record`s via the existing `recordGuestBind` on guest-attach, so Remove doesn't break re-commission.
- `IntentRecorder` extended with `SetDecommissioned` + `Get` (both already on `*storage.IntentStore`).
- Non-hollow tests (`internal/localapi/decommission_test.go`): role-gate refuses system/backup (403,
no unmount); decommission sets intent + removes the bind + unmounts + never formats; intent-aware
re-assert does NOT rebind a decommissioned-but-present drive (companion: enrolled DOES rebind; the
intent-blind pre-fix code fails this); re-commission re-records; `Remove` idempotency + persistence.
## v0.31.0 — live-drive F9 + F20-BUG2 + F20-BUG3 (disk bind/wipe) (2026-06-14)
The last live-drive findings, all disk/`localapi`-side, implemented + deployed on `felhom-pve` and
+2 -2
View File
@@ -35,15 +35,15 @@ import (
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
"gitea.dooplex.hu/admin/felhom-agent/internal/pbs"
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
// version is the agent version. Overridable at build time with
// -ldflags "-X main.version=<v>"; defaults to the in-repo CHANGELOG version.
var version = "0.31.0"
var version = "0.32.0"
func main() {
var (
+222
View File
@@ -0,0 +1,222 @@
package localapi
import (
"context"
"io"
"log/slog"
"net/http"
"path/filepath"
"sync"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
// fakeIntent implements the extended IntentRecorder (SetEnrolled/SetEjected/SetDecommissioned + Get).
type fakeIntent struct {
mu sync.Mutex
m map[string]storage.DriveIntent
}
func newFakeIntent() *fakeIntent { return &fakeIntent{m: map[string]storage.DriveIntent{}} }
func (f *fakeIntent) set(id string, v storage.DriveIntent) {
f.mu.Lock()
f.m[id] = v
f.mu.Unlock()
}
func (f *fakeIntent) SetEnrolled(id string) error { f.set(id, storage.IntentEnrolled); return nil }
func (f *fakeIntent) SetEjected(id string) error { f.set(id, storage.IntentEjected); return nil }
func (f *fakeIntent) SetDecommissioned(id string) error {
f.set(id, storage.IntentDecommissioned)
return nil
}
func (f *fakeIntent) Get(id string) storage.DriveIntent {
f.mu.Lock()
defer f.mu.Unlock()
return f.m[id]
}
// decommServer wires a Server with Disks + Intent + GuestBinds + GuestAttach so both the endpoint and
// the intent-aware re-assert can be exercised. Returns the *Server (use .Handler() for HTTP tests).
func decommServer(t *testing.T, d *fakeDiskOps, sv StorageView, gl GuestLister, intent IntentRecorder, gb *GuestBindStore, ga GuestAttacher, mounts map[int]map[string]string) *Server {
t.Helper()
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0",
Guests: &fakeGuestsCfg{mounts: mounts},
Backups: &fakeBackups{},
Store: &fakeStore{},
Storage: sv,
Tokens: staticTokens{"A": 8200, "B": 9300},
Disks: d,
DiskGate: &fakeGate{},
Guests2: gl,
Intent: intent,
GuestBinds: gb,
GuestAttach: ga,
HostReader: sysOnSDA(),
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatalf("new server: %v", err)
}
srv.baseCtx = context.Background()
return srv
}
// userDataAndProtected is a storage view with a user-data USB, a system dir, and a backup PBS mount.
func userDataAndProtected() fakeStorage {
return fakeStorage{targets: []hub.StorageTarget{
{Name: "bulk", Type: hub.StorageTypeUSB, BackingDevice: "/dev/sdb1", MountPath: "/mnt/bulk", DurableID: "uuid:usb-1"},
{Name: "local", Type: "local", MountPath: "/var/lib/vz"},
{Name: "felhom-pbs", Type: hub.StorageTypePBS, MountPath: "/mnt/pbs"},
}}
}
// TestDecommission_RoleGated: like eject, a system/backup mount is refused 403 with NO unmount; only a
// user-data mount decommissions. (Mirrors TestEject_RoleGated — the gate is the security control.)
func TestDecommission_RoleGated(t *testing.T) {
for _, where := range []string{"/var/lib/vz", "/mnt/pbs", "/mnt/unknown"} {
d := &fakeDiskOps{}
srv := decommServer(t, d, userDataAndProtected(), fakeGuestList{}, newFakeIntent(), tempBindStore(t), &fakeGuestAttacher{}, nil)
w := do(t, srv.Handler(), "POST", "/disks/decommission", "A", `{"where":"`+where+`"}`)
if w.Code != http.StatusForbidden {
t.Fatalf("decommission %s: got %d want 403 (%s)", where, w.Code, w.Body.String())
}
d.mu.Lock()
if len(d.unmountCalls) != 0 {
t.Fatalf("Unmount called on protected mount %s — role-gate bypassed", where)
}
d.mu.Unlock()
}
}
// TestDecommission_Effects: a user-data decommission sets IntentDecommissioned, removes the
// GuestBindStore entry, and unmounts — assert ALL THREE (not just no-error). NEVER formats.
func TestDecommission_Effects(t *testing.T) {
d := &fakeDiskOps{}
intent := newFakeIntent()
intent.SetEnrolled("uuid:usb-1") // currently enrolled
gb := tempBindStore(t)
_ = gb.Record(8200, "uuid:usb-1")
srv := decommServer(t, d, userDataAndProtected(), fakeGuestList{}, intent, gb, &fakeGuestAttacher{}, nil)
w := do(t, srv.Handler(), "POST", "/disks/decommission", "A", `{"where":"/mnt/bulk"}`)
if w.Code != http.StatusOK {
t.Fatalf("decommission user-data: got %d want 200 (%s)", w.Code, w.Body.String())
}
// 1) intent recorded as decommissioned
if got := intent.Get("uuid:usb-1"); got != storage.IntentDecommissioned {
t.Errorf("intent = %q, want decommissioned", got)
}
// 2) guest-bind record pruned
if ids := gb.Guests()[8200]; len(ids) != 0 {
t.Errorf("guest-bind not removed: %v", ids)
}
// 3) unmounted, never formatted
d.mu.Lock()
defer d.mu.Unlock()
if len(d.unmountCalls) != 1 || d.unmountCalls[0] != "/mnt/bulk" {
t.Errorf("Unmount calls = %v, want [/mnt/bulk]", d.unmountCalls)
}
if len(d.formatCalls) != 0 {
t.Errorf("decommission must NEVER format; formatCalls = %v", d.formatCalls)
}
}
// TestReassertGuestBinds_SkipsDecommissioned is the load-bearing F9-reconnect invariant: a
// decommissioned-but-present drive still recorded in the bind store must NOT auto-rebind on agent
// restart. Companion: with intent=enrolled the SAME setup DOES rebind — proving the intent gate is
// what blocks it (the pre-fix intent-blind code would rebind both → this test FAILS on it).
func TestReassertGuestBinds_SkipsDecommissioned(t *testing.T) {
// decommissioned → must NOT rebind even though present + recorded.
gbD := tempBindStore(t)
_ = gbD.Record(8200, "uuid:usb-1")
intentD := newFakeIntent()
intentD.SetDecommissioned("uuid:usb-1")
gaD := &fakeGuestAttacher{}
srvD := decommServer(t, &fakeDiskOps{}, usbPresent(), fakeGuestList{}, intentD, gbD, gaD, map[int]map[string]string{
8200: {"mp0": "local-lvm:8,mp=/var/lib/docker"}, // bind missing → would re-add if not gated
})
srvD.ReassertGuestBinds(context.Background())
if gaD.count() != 0 {
t.Fatalf("decommissioned drive was re-bound (%d AttachBind) — intent gate missing", gaD.count())
}
// companion: enrolled → DOES rebind (same present drive + missing bind).
gbE := tempBindStore(t)
_ = gbE.Record(8200, "uuid:usb-1")
intentE := newFakeIntent()
intentE.SetEnrolled("uuid:usb-1")
gaE := &fakeGuestAttacher{}
srvE := decommServer(t, &fakeDiskOps{}, usbPresent(), fakeGuestList{}, intentE, gbE, gaE, map[int]map[string]string{
8200: {"mp0": "local-lvm:8,mp=/var/lib/docker"},
})
srvE.ReassertGuestBinds(context.Background())
if gaE.count() != 1 {
t.Fatalf("enrolled drive should rebind (got %d AttachBind) — gate too aggressive", gaE.count())
}
}
// TestReCommission_ReRecords: after a decommission prunes the bind, re-enrolling (SetEnrolled +
// Record) restores it so the re-assert rebinds again. Proves Remove doesn't break re-enroll.
func TestReCommission_ReRecords(t *testing.T) {
gb := tempBindStore(t)
intent := newFakeIntent()
// decommission removed the record + set decommissioned intent.
intent.SetDecommissioned("uuid:usb-1")
_ = gb.Remove(8200, "uuid:usb-1") // no-op (absent) — idempotent
// re-enroll (what handleDiskGuestAttach does): set enrolled + record.
intent.SetEnrolled("uuid:usb-1")
if err := gb.Record(8200, "uuid:usb-1"); err != nil {
t.Fatal(err)
}
ga := &fakeGuestAttacher{}
srv := decommServer(t, &fakeDiskOps{}, usbPresent(), fakeGuestList{}, intent, gb, ga, map[int]map[string]string{
8200: {"mp0": "local-lvm:8,mp=/var/lib/docker"},
})
srv.ReassertGuestBinds(context.Background())
if ga.count() != 1 {
t.Fatalf("re-commissioned drive should rebind (got %d)", ga.count())
}
}
// TestGuestBindStore_Remove covers idempotency (absent = no-op), present removal, empty-key drop, and
// persistence across reopen.
func TestGuestBindStore_Remove(t *testing.T) {
path := filepath.Join(t.TempDir(), "guest-binds.json")
gb, err := OpenGuestBindStore(path)
if err != nil {
t.Fatal(err)
}
// absent vmid + absent id → no-op, no error
if err := gb.Remove(8200, "uuid:nope"); err != nil {
t.Fatalf("Remove absent: %v", err)
}
_ = gb.Record(8200, "uuid:a")
_ = gb.Record(8200, "uuid:b")
// remove a present id → keeps the other
if err := gb.Remove(8200, "uuid:a"); err != nil {
t.Fatal(err)
}
if ids := gb.Guests()[8200]; len(ids) != 1 || ids[0] != "uuid:b" {
t.Fatalf("after remove uuid:a, vmid 8200 = %v want [uuid:b]", ids)
}
// remove the last id → vmid key dropped
if err := gb.Remove(8200, "uuid:b"); err != nil {
t.Fatal(err)
}
if _, ok := gb.Guests()[8200]; ok {
t.Errorf("empty vmid key should be dropped")
}
// persistence: reopen and confirm empty
re, err := OpenGuestBindStore(path)
if err != nil {
t.Fatal(err)
}
if len(re.Guests()) != 0 {
t.Errorf("store should be empty after reopen, got %v", re.Guests())
}
}
+70 -3
View File
@@ -71,12 +71,16 @@ type GuestAttacher interface {
RebootGuest(ctx context.Context, vmid int) error
}
// IntentRecorder persists drive enroll/eject INTENT (slice 10 P3 self-heal), keyed by durable-id, so
// the watchdog reconciles only enrolled drives and respects an official eject. Satisfied by
// *storage.IntentStore. Optional — when nil, the local API records no intent (self-heal is ungated).
// IntentRecorder persists drive enroll/eject/decommission INTENT (slice 10 P3 self-heal), keyed by
// durable-id, so the watchdog reconciles only enrolled drives and respects an official eject /
// permanent decommission. Get lets the startup re-assert be intent-aware (B2 — skip non-enrolled).
// Satisfied by *storage.IntentStore. Optional — when nil, the local API records no intent (self-heal
// is ungated and the re-assert falls back to legacy bind-all behavior).
type IntentRecorder interface {
SetEnrolled(durableID string) error
SetEjected(durableID string) error
SetDecommissioned(durableID string) error
Get(durableID string) storage.DriveIntent
}
// ---- handlers ---------------------------------------------------------------------------
@@ -246,6 +250,58 @@ func (s *Server) handleDiskEject(w http.ResponseWriter, r *http.Request, vmid in
writeOK(w, map[string]any{"vmid": vmid, "ejected": req.Where, "dependent_guests": dependents})
}
// handleDiskDecommission is the SELF-SERVE, NON-DESTRUCTIVE permanent removal of a user-data drive
// (B2). It mirrors handleDiskEject EXACTLY — withGuest self-scoping, scopedFromBody, and the same
// user-data ROLE GATE (a system/backup mount is refused 403; fail-safe-to-protected on ambiguity) so
// a compromised controller can't decommission protected storage. Unlike the operator-signed
// DecommissionExecutor it needs no signature: it is the customer's own drive. It records a PERMANENT
// decommission intent (the self-heal watchdog + the intent-aware re-assert never auto-mount/re-bind it
// again), prunes the guest-bind record (hygiene), and unmounts so the drive is physically removable.
// It NEVER calls any format/mkfs path — the data stays on the drive.
func (s *Server) handleDiskDecommission(w http.ResponseWriter, r *http.Request, vmid int) {
if s.disks == nil {
writeErr(w, http.StatusServiceUnavailable, "disk management not configured on this host")
return
}
var req ejectRequest
if !decodeBody(w, r, &req) {
return
}
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
return
}
if strings.TrimSpace(req.Where) == "" {
writeErr(w, http.StatusBadRequest, "where (mountpoint) is required")
return
}
// ROLE GATE (same as eject): user-data only. The agent classifies from its OWN view, never the
// caller's claim; an unresolvable mount fails safe to protected → refused.
if role := s.roleForMountPath(r.Context(), req.Where); role != storage.RoleUserData {
s.logger.Warn("local-api: protected — decommission refused by role",
"vmid", vmid, "where", req.Where, "role", role)
writeErr(w, http.StatusForbidden, "mount is system/backup-protected — decommission refused (role: "+string(role)+")")
return
}
dependents := s.dependentGuests(r.Context(), req.Where)
// Resolve the durable-id BEFORE unmounting (it still resolves while mounted) for the bind prune.
id := s.durableIDForMount(r.Context(), req.Where)
// Record the PERMANENT decommission intent first (self-heal never re-mounts it again).
s.recordIntent(r.Context(), req.Where, "decommissioned")
// Hygiene: drop the guest-bind record so the startup re-assert carries no stale id.
if id != "" && s.guestBinds != nil {
if err := s.guestBinds.Remove(vmid, id); err != nil {
s.logger.Warn("local-api: guest-bind remove failed", "vmid", vmid, "durable_id", id, "err", err)
}
}
// Unmount (mirror eject) — benign, data preserved. NEVER format/mkfs here.
if err := s.disks.Unmount(r.Context(), req.Where); err != nil {
s.logger.Error("local-api: disk decommission", "vmid", vmid, "where", req.Where, "err", err)
writeErr(w, http.StatusBadRequest, "decommission failed: "+err.Error())
return
}
writeOK(w, map[string]any{"vmid": vmid, "decommissioned": req.Where, "dependent_guests": dependents})
}
type guestAttachRequest struct {
VMID int `json:"vmid"`
Where string `json:"where"` // the host mount path of the enrolled drive (e.g. /mnt/felhom-usb)
@@ -670,6 +726,15 @@ func (s *Server) ReassertGuestBinds(ctx context.Context) {
}
}
for _, id := range ids {
// Intent-aware (B2, the load-bearing correctness fix): NEVER re-bind a drive that is not
// currently `enrolled` — an ejected or decommissioned drive must not auto-rebind into the
// guest on agent restart, even if it is still host-mounted. Covers both the self-serve and
// the operator-signed decommission paths (both land on IntentDecommissioned). A nil intent
// store falls back to legacy bind-all (ungated), matching the watchdog's nil-intent rule.
if s.intent != nil && s.intent.Get(id) != storage.IntentEnrolled {
s.logger.Warn("F9 re-assert: skipping non-enrolled drive (intent-gated)", "vmid", vmid, "durable_id", id, "intent", string(s.intent.Get(id)))
continue
}
where, present := mountByDurable[id]
if !present {
s.logger.Warn("F9 re-assert: enrolled drive not present (durable-id absent) — skipping", "vmid", vmid, "durable_id", id)
@@ -727,6 +792,8 @@ func (s *Server) recordIntent(ctx context.Context, where, action string) {
err = s.intent.SetEnrolled(id)
case "ejected":
err = s.intent.SetEjected(id)
case "decommissioned":
err = s.intent.SetDecommissioned(id)
}
if err != nil {
s.logger.Warn("local-api: intent record failed", "where", where, "action", action, "durable_id", id, "err", err)
+31
View File
@@ -71,6 +71,37 @@ func (s *GuestBindStore) Record(vmid int, durableID string) error {
return s.saveLocked()
}
// Remove drops (vmid, durableID) from the enrolled set. Idempotent — absent (vmid or id) is a no-op
// returning nil. Atomic write (tmp+rename) like Record/saveLocked. Called by the self-serve
// decommission endpoint so a permanently-removed drive no longer lingers in the startup re-assert
// record (hygiene — the intent-aware ReassertGuestBinds is the load-bearing guard).
func (s *GuestBindStore) Remove(vmid int, durableID string) error {
s.mu.Lock()
defer s.mu.Unlock()
ids, ok := s.m[vmid]
if !ok {
return nil
}
kept := ids[:0:0]
found := false
for _, id := range ids {
if id == durableID {
found = true
continue
}
kept = append(kept, id)
}
if !found {
return nil // idempotent: nothing to remove
}
if len(kept) == 0 {
delete(s.m, vmid)
} else {
s.m[vmid] = kept
}
return s.saveLocked()
}
// Guests returns a copy of the vmid → enrolled-durable-ids map.
func (s *GuestBindStore) Guests() map[int][]string {
s.mu.Lock()
+1
View File
@@ -237,6 +237,7 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks))
mux.HandleFunc("POST /disks/assign", s.withGuest(s.handleDiskAssign))
mux.HandleFunc("POST /disks/eject", s.withGuest(s.handleDiskEject))
mux.HandleFunc("POST /disks/decommission", s.withGuest(s.handleDiskDecommission))
mux.HandleFunc("POST /disks/format", s.withGuest(s.handleDiskFormat))
mux.HandleFunc("GET /disks/format/status", s.withGuest(s.handleDiskFormatStatus))
// Guest data-drive passthrough (slice 10 P2): bind an enrolled drive's felhom-data namespace in.