felhom-agent/REUSE.md

REUSE.md — felhom-agent

Before writing new code, check here. Canonical helpers, patterns to copy, traps to avoid. Maintenance: update in the SAME commit that adds/changes/deprecates a shared helper. Entries cite file + symbol. Line numbers are landmarks only — reconfirm before editing.

1. Canonical helpers (MUST reuse — do not reinvent)

Allowlisted exec / privileged surface (sudoers)

Symbol	File	Short signature	Use for	Gotchas
Runner / ExecRunner.Run, RunStdin	internal/proxmox/privileged.go	Run(ctx, name, args...) (stdout, stderr []byte, err)	ALL host command exec (direct or sudo -n prefix)	Arg vectors only, never a shell string; capBuf caps output at 1 MiB
Privileged (CreateGoldenLXC/MountUSBByUUID/SMART/Sensors)	internal/proxmox/privileged.go	methods on *Privileged	the 3 fenced root-CLI exceptions ONLY	Do NOT add methods — fence is structural (routing_test.go asserts it)
SudoHostOps.run	internal/storage/hostops.go	run(ctx, name, args...) error	allowlisted exec with stderr-wrapped error	Every arg pre-validated via validate.go before this is called
Prober.Probe	internal/capability/probe.go	Probe(ctx) []Status	live sudo-policy capability check (sudo -n -l --)	Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (Capability.GatedBy + Prober.GateActive) report inactive/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from pbsdr.Manager.DRConfigured (marker-backed across restarts)
stageTemp	internal/localapi/intermediary.go	stageTemp(pattern, content) (path, err)	random-named temp before a root install (audit B1)	Fixed /tmp names are a TOCTOU — sudoers globs expect /tmp/felhom-*-*.ext
guesthook.InstallSnippet / Register	internal/guesthook/install.go	InstallSnippet(ctx, runner) error	pre-start self-heal hook install (C1 net)	Same random-temp+install pattern; snippet delegates to the agent binary (no shell logic). Issues mkdir -p /var/lib/vz/snippets FIRST (v0.63.0, B2 — fresh boxes lack the dir; sudoers grants exactly that argv)

Disk / format safety (role gates, durable IDs, format guards)

Symbol	File	Short signature	Use for	Gotchas
SudoHostOps.Format	internal/storage/hostops.go	Format(ctx, device, fstype) error	THE only mkfs path	Guards, in order: ValidateBlockDevice + ValidateFSType → mandatory deviceUnclaimed (claim.go) → exec felhom-mkfs-guarded (sudoers allowlists ONLY the wrapper, not raw mkfs)
SudoHostOps.InspectDevice + DeviceProbe.DataBearing	internal/storage/hostops.go	InspectDevice(ctx, device) (DeviceProbe, error)	data-bearing verdict from the AGENT's own read	Fail-safe: Probed=false ⇒ DataBearing=true; blkid output is evidence, lsblk is read-success authority
classifyClaim / SudoHostOps.deviceUnclaimed	internal/storage/claim.go	classifyClaim(claimFacts) (unclaimed bool, reason string)	"is this disk provably free to format"	Pure function of gatherClaimFacts; ANY read error/ambiguity/empty-lsblk ⇒ CLAIMED (audit D2); Felhom's own /mnt/felhom-drives mounts are not a foreign claim
SudoHostOps.ListCandidateDisks	internal/storage/candidates.go	ListCandidateDisks(ctx) ([]CandidateDisk, error)	enroll-candidate discovery	Fail-safe: omits anything not provably unclaimed
antiRetargetResolveExpect (+ antiRetargetResolve, antiRetargetResolveBlank)	internal/localapi/wipe_reresolve.go	(durableID, expectDataBearing, resolve, derive, inspect) (device, err)	pre-mkfs anti-retarget: resolve durable id → re-derive+match → re-inspect	AGENT-001 + audit D3; refuses path-only bindings; wired via Server.reresolveWipe/reresolveBlank (test-injectable)
signedjobs.WipeExecutor.Execute	internal/signedjobs/wipe.go	Execute(ctx, op, params) error	operator-signed data-bearing wipe	Durable-id bound; nonce burned by gate BEFORE execute; refuses no-longer-data-bearing targets
selfupdate.Executor / selfupdate.Manager	internal/selfupdate/{executor,commit}.go	NewExecutor(Config) / NewManager(ManagerConfig)	operator-signed agent self-update (D1): download+verify-vs-signed-sha → wrapper apply; startup dwell → commit	sha is the ONLY integrity root; wrapper (felhom-selfupdate-guarded) re-verifies as root + does the A/B flip; NEVER rolls back (systemd + wrapper do). WrapperRunner seam. Report seam SelfUpdatePending()
Gate.AuthorizeStorageWipe	internal/reconcile/gate.go	AuthorizeStorageWipe(StorageWipeAuthz, *SignedOp) Decision	tiered wipe authz	user-data ⇒ customer confirm bound to agent's DeviceDurableID; system/backup ⇒ operator signature only, Confirmed IGNORED by role
Gate.Authorize	internal/reconcile/gate.go	Authorize(Intent, *SignedOp) Decision	every destructive intent	role-scoping (roleAuthorizes) + op-to-action binding; benign passes unsigned; audits every decision
storage.DeviceDurableID / ResolveDurableDevice	internal/storage/durable_device.go	DeviceDurableID(device) (string, error)	WIPE-binding ids (byid:/byuuid:)	Single seam for /disks list AND gate (F20-BUG2); ResolveDurableDevice refuses bare paths
storage.ResolveStorageDevice	internal/storage/durable_device.go	ResolveStorageDevice("uuid:<fs-uuid>") (dev, err)	re-resolve enrolled STORAGE drives (remount)	uuid: scheme ONLY — distinct from the wipe schemes; never trust a remembered /dev node
deriveDurableID	internal/storage/durableid.go	deriveDurableID(typ, s, backingDevice, uuid) string	storage-target durable id (DR re-attach key)	Deterministic per type; uuid: for usb/local-dir; PBS id includes #<fingerprint>
SystemDisks / isSystemBacked / RoleForStorage / RoleForRawDevice	internal/storage/role.go	RoleForRawDevice(device, sysDisks, sysKnown) DeviceRole	protection-tier classification	Fails safe to system (most protected) on any ambiguity; role is AGENT-derived, never caller-supplied
ValidateUUID/MountPath/BlockDevice/FSType/SMARTDevice/LVMName, UnitNameForMount	internal/storage/validate.go	Validate*(v) error	EVERY arg that reaches a root shell-out	The security boundary; strict whitelists (no by-* symlinks, no dm, no traversal); systemdEscapePath computed in-process
ValidateNetworkMountSpec	internal/storage/netmount.go	ValidateNetworkMountSpec(spec) error	NAS mount input boundary	Same discipline as validate.go; SMB requires a creds ref; mountpoint confined under NetworkMountRoot

Mount lifecycle (host + guest binds)

Symbol	File	Short signature	Use for	Gotchas
SudoHostOps.EnsureMount	internal/storage/hostops.go	EnsureMount(ctx, MountSpec) error	persistent by-UUID systemd .mount	Validate→render→stage→install→enable --now; idempotent
SudoHostOps.Unmount	internal/storage/hostops.go	Unmount(ctx, where) error	detach a mount unit	DESTRUCTIVE — caller MUST have gated it; does not self-authorize
SudoHostOps.ReassertEnrolledMounts	internal/storage/hostops.go	ReassertEnrolledMounts(ctx)	reboot remount (re-resolve by UUID)	Re-asserts unless mounted AND enabled (shouldReassertMount); skips absent UUIDs
GuestBinder.AttachDrive / DetachDrive	internal/localapi/intermediary.go	AttachDrive(ctx, vmid, where) (guestPath, err)	live drive hot-swap under /mnt/felhom-drives	Normalizes to EXACTLY ONE bind via countHostMounts (converges double-binds); force re-bind when guest can't see it
GuestBinder.EnsureSharedParent	internal/localapi/intermediary.go	EnsureSharedParent(ctx) error	shared-parent bind + boot unit	make-private+make-shared ONLY on first bind — re-running orphans the guest's slave; F2-a: compares script AND unit for staleness
StablePathForRaw / DriveNameFromRaw	internal/localapi/intermediary.go	StablePathForRaw("/mnt/<n>") string	raw host mount → stable guest path	Single source of truth both repos derive from
GuestBinder.GuestSeesMount / GuestBootID	internal/localapi/intermediary.go	GuestSeesMount(ctx, vmid, path) bool	guest-visible (usable) signal; reboot detection	Host bind present ≠ guest sees it (non-recursive parent bind)
SudoHostOps.EnsureNetworkMount / RemoveNetworkMount / ListNetworkMounts	internal/storage/netmount.go	EnsureNetworkMount(ctx, spec) error	NAS automount pair	rm glob confined to mnt-felhom* units; NAS ≠ drive (no durable-id/SMART/wipe); RemoveNetworkMount doubles as the verify-fail rollback (idempotent)
NetworkMountedAt / NetworkEndpointReachable	internal/storage/netmount.go	NetworkMountedAt(where) bool	verify mount-truth + the 2 s add pre-probe	/proc/mounts is the ONLY mount-success judge (autofs trigger ≠ mounted; readability ≠ mounted — SPIKE-nas-verify §8)
SudoHostOps.ReassertNetworkAutomounts + Server.ReassertNetworkMounts + guesthook.PostStartNetworkReassert	internal/storage/netreassert.go, internal/localapi/netreassert.go, internal/guesthook/netreassert.go	ReassertNetworkAutomounts(ctx) []NetReassertResult	NAS guest-reboot heal (RCA fix 1): re-arm idle automount triggers (stop + enable --now) so the fresh mount event propagates into running guests	NEVER call from periodic health paths (an idle trigger is HEALTHY); active real mounts are never touched; hook leg runs as root (direct systemctl), daemon leg via sudo
ClassifyNetVerifyFailure	internal/storage/netverify.go	ClassifyNetVerifyFailure(journalTail, tcpReachable) (code, hint)	NAS verify failure categories	String-based BY DESIGN (every mount failure is rc=32); substrings verbatim from SPIKE-nas-verify Q4; nfs_export merges not-found/not-permitted (NFSv4 identical)

Durable stores (atomic state)

Symbol	File	Short signature	Use for	Gotchas
IntentStore (Get/SetEnrolled/SetEjected/SetDecommissioned/OnAbsent)	internal/storage/intent.go	OpenIntentStore(path)	drive intent (4-state self-heal)	Keyed by durable-id only; OnAbsent is the ONLY ejected→enrolled path; refuses empty ids
GuestBindStore (Record/Remove/Guests)	internal/localapi/guestbindstore.go	OpenGuestBindStore(path)	per-guest enrolled binds (F9 re-assert)	Same tmp+rename 0600 pattern as IntentStore
FormatJobStore + startFormatDetached + RecoverFormatJob	internal/localapi/formatjob.go	startFormatDetached(device, durableID, fstype, blank) <-chan error	detached, restart-surviving mkfs (F20-BUG3)	Runs off s.baseCtx (60-min bound) so a request deadline can't SIGKILL mkfs; recovery re-resolves by durable id; blank jobs re-check STILL-blank
TokenStore.Mint / Lookup	internal/localapi/tokenstore.go	Mint(vmid) (plaintext, error)	per-guest local-API tokens	Only the SHA-256 hash persists (fsync'd append log); constant-time compare on lookup; plaintext returned exactly once. Lookup RELOADS the file once on a miss (v0.63.0, B3): the one-shot provisioner mints into the same file the daemon indexes — cross-process coherence without a restart; append-only size check bounds the re-read
FileNonceStore.SeenOrRecord	internal/authz/noncestore.go	SeenOrRecord(nonce, exp) bool	durable anti-replay	fsync'd before returning false; prune only after exp
Journal (Append/Latest/InFlight/AlreadyApplied)	internal/reconcile/journal.go	OpenJournal(path)	op journal + idempotency + crash recovery	Recover consumes InFlight(); scratch entries special-cased

Local-API plumbing

Symbol	File	Short signature	Use for	Gotchas
Server.withGuest	internal/localapi/server.go	withGuest(fn(w, r, vmid)) http.HandlerFunc	bearer auth + self-scoping for EVERY route	Token→vmid is the authority; explicit ?vmid= only a consistency check (403 on mismatch)
Server.scopedFromBody	internal/localapi/server.go	scopedFromBody(w, bodyVMID, tokenVMID, path) bool	POST-body vmid self-scope check	Call right after decode; false = already 403'd
decodeBody / writeOK / writeErr / writeStatus	internal/localapi/server.go	writeStatus(w, code, ok, data, errMsg)	ALL local-API JSON I/O	Do not hand-roll response envelopes
EnsureLeaf	internal/localapi/cert.go	EnsureLeaf(certPath, keyPath, host) (cert, fingerprint, generated, err)	pinned self-signed leaf	generated=true invalidates every issued bootstrap pin — log LOUD (B.1)
Server.RecoverStaleLockedGuests	internal/localapi/stalelock.go	RecoverStaleLockedGuests(ctx)	startup stale vzdump-lock heal (F2-b)	Clears ONLY backup/snapshot-delete, only when no vzdump in-flight; A1 RESOLVED (v0.62.0): scan is pool-intersected (ListLXC ∩ Client.Pool), fail-safe skip on pool-read failure
ControllerSwapper.Swap + ValidControllerImage	internal/localapi/controllerswap.go	Swap(ctx, vmid, target) *ControllerSwapState	agent-owned controller image swap + rollback	Strict image regex (repo + 3-part semver); state file written BEFORE swap; no-healthcheck images need verifyDwell
MemoryOps + Server.readMemoryBounds	internal/localapi/guestmemory.go	readMemoryBounds(ctx, vmid) (memoryBounds, err)	guest RAM resize (v0.90.0, R-24): GET/POST /guest/memory	NEW narrow seam (never extend GuestAPI — it breaks every fake); the AGENT is the boundary — bounds recomputed FRESH per request (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)); §8 UNITS TRAP (config memory=MB, status/node=bytes); verify maxmem==target after SetConfig before claiming success; SetConfig NEVER called on a refusal path

Proxmox client / hub / PBS / provisioning

Symbol	File	Short signature	Use for	Gotchas
Client.WaitTask	internal/proxmox/task.go	WaitTask(ctx, upid, opts) (TaskStatus, error)	asserting EVERY mutating op	POST 200 ≠ success; authz can fail at task exec; AllowWarnings opt-in
Client.Pool	internal/proxmox/query.go	Pool(ctx, name) (PoolInfo, error)	felhom-pool membership (the ownership registry, A1)	Needs Pool.Audit at /pool/<name> (host-install v1.9.0+); Pool.Allocate does NOT satisfy the read; members can be storages (type storage, vmid 0) — filter them
Client mutate wrappers (RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop)	internal/proxmox/mutate.go	return (upid, error)	all API mutations	Async → always pair with WaitTask; route via gate/queue, not ad-hoc
Client.PoolAddVMID	internal/proxmox/mutate.go	PoolAddVMID(ctx, pool, vmid) error	re-assert pool membership after a restore-over-existing (campaign-2 R2)	SYNC (no UPID, don't WaitTask); PVE PUT /pools is additive (merge, not replace) — delete=1 removes; idempotent (already-member swallowed); needs Pool.Allocate at /pool/<pool>. pct restore --pool sets membership only at CREATE — a restore over an existing vmid drops it, so bring-up re-asserts post-restore
TLSConfig.build / normalizeFingerprint	internal/proxmox/tls.go	build() (*tls.Config, error)	PVE leaf-cert SHA-256 pinning	No insecure default
pinnedTLS	internal/pbs/pin.go	pinnedTLS(fingerprint) (*tls.Config, error)	PBS leaf pinning	Same model as PVE; 64-hex fingerprint normalized
hub.Client.Report	internal/hub/client.go	Report(ctx, *HostReport) (*ControlEnvelope, error)	the heartbeat	Typed TransportError/HTTPError, never contain the bearer token
hub.Loop + MultiObserver	internal/hub/loop.go	NewLoop(...); MultiObserver(obs...)	resilient report loop + envelope fan-out	Errors logged, loop continues; interval clamped 60–3600 s
provision.BackHalf.Provision	internal/provision/backhalf.go	Provision(ctx, Input) (Result, error)	guest bootstrap back-half	mint→render→0600 write→chown 100000:100000→pct set ro bind→onboot; token NEVER logged/returned. Bootstrap local_api.endpoint = the caller's cfg.LocalAPI.ListenAddr (main.go) — moving the agent bind to the island moves the guest dial for free (R-50, no template)
buildBringUpConfig island NIC	internal/reconcile/bringup.go	(pure) BringUpSpec{IslandBridge,IslandGuestAddr} → params["net1"]	R-50 island control plane	When BOTH island fields are set (from cfg.LocalAPI), attaches a static net1=name=eth1,bridge=<vmbr9>,ip=<.2/30> (no hwaddr → fresh MAC), so the controller reaches the agent over a fixed private address immune to LAN/DHCP/site moves. Empty = pre-R-50, no net1. All-or-nothing + CIDR enforced in LocalAPIConfig.Validate. The guestnet healer is eth0-only (parseMode is dev-scoped) so it never touches the static island NIC
reconcile.Queue.Submit	internal/reconcile/queue.go	Submit(vmid, fn) <-chan error	per-guest serialization of ALL mutations	Same vmid strictly FIFO; lanes parallel across guests
Engine.RunSignedJob	internal/reconcile/job.go	RunSignedJob(ctx, intent, signed, exec) JobResult	executing a gated destructive job	Idempotency by nonce; journaled
escrow.Create	internal/escrow/escrow.go	Create(ctx, CreateOptions) (CreateResult, R, error)	PBS-key escrow (zero-knowledge)	Recovery code returned SEPARATELY from the result (anti-log); self-verifies recoverability
escrow.GenerateRecoveryCode / joinSafe / RecoveryCodeSep	internal/escrow/wordlist.go	GenerateRecoveryCode() (string, error)	minting the customer recovery code R	Draws from the EFF large list filtered of every word containing RecoveryCodeSep (4 entries: drop-down, felt-tip, t-shirt, yo-yo) so a code always segments back into exactly 10 words — a hyphenated word made codes ambiguous to transcribe AND flaked the test ~1/5 (v0.93.0). Generation-only: already-issued codes stay valid, R is verified as a whole passphrase and never re-split. Never count words by splitting the joined string — count what the generator drew
escrow.CeremonyBinary / CeremonyArgs() / CeremonyOutput	internal/escrow/ceremony.go	the ONE fixed sudo self-invocation argv + the --output=json wire object (v1)	controller-driven ceremony (v0.88.0)	SINGLE SOURCE shared by the localapi exec, the capability manifest entry, and (byte-identically) the FELHOM_ESCROW sudoers line — TestEscrowCeremonyArgvPinned + TestManifestCoveredBySudoers lock all three. Never flag-helpers, never --→- (spike §2.2)
localapi escrow ceremony job	internal/localapi/escrow_ceremony.go	POST /escrow/ceremony + status + ONE-SHOT claim + preflight	the wizard's agent half	R lives ONLY in Server.escrowR (NEVER the job struct — snapshots must be structurally R-free); zeroed on claim/supersede/10-min TTL (unclaimed_void); in-memory BY DESIGN (restart loses R safely; re-run supersedes); subprocess stdout is SECRET-BEARING → parsed then zeroed, never logged
poke.Listener + poke.Port	internal/poke/poke.go	NewListener(resolve, trigger, port, logger); poke.Port = 51822	agent-plane immediate-sync (Direction-2a, v0.89.0)	Binds a contentless UDP socket EXCLUSIVELY to the box's WG /32 (wgtunnel.LoadAssignedAddr), fires the hub-loop out-of-band trigger. Port 51822 is a SHARED cross-repo contract — the hub poke sender + the ep0 felhom-poke forced-command target the SAME number; change one → change all three. Contentless (payload ignored), leading-edge debounced (DebounceWindow), WG-confined (kernel EKEYREJECTED refuses non-peer /32s). Wired only when wg_tunnel.enabled
wgtunnel.LoadAssignedAddr	internal/wgtunnel/manager.go	LoadAssignedAddr(stateDir) (netip.Addr, bool)	the box's own WG /32 without a Manager	Reads registered.json; ok=false until registered; strips the /32 → bare addr (the poke bind target)
fasttick.Loop + fasttick.SourceFunc	internal/fasttick/fasttick.go	New(out chan<- struct{}, interval, logger, sources...); Source.Unconverged() (bool, reason)	agent-plane immediacy SECONDARY (v0.90.0, R-28): pulse the SAME out-of-band trigger every 30 s while ANY source is unconverged, self-disarm on convergence	STATE-BASED (no timer, nothing to journal). Every source MUST be a CACHED read (no exec/network per tick) — desiredProvider.Generation(), reconcile.Engine.LastResult() (Planned−Pending>0), pbsdrLoop.PBSDRStatus().State=="waiting_secret" ONLY, wgtunnel.Manager.TunnelConvergence(). The LOUD pbsdr states + destructive pending_signature are DELIBERATELY EXCLUDED (a stuck-loud box must not hammer). Pulses the cap-1 channel non-blocking (coalesces with poke/watchdog); NEVER touch MinPollSeconds/clampInterval/the ticker
wgtunnel.Manager.TunnelConvergence / reconcile.Engine.LastResult	internal/wgtunnel/manager.go, internal/reconcile/engine.go	TunnelConvergence() (desired, operational bool); LastResult() (Result, bool)	cached convergence snapshots for the fast-tick	TunnelConvergence is refreshed at the END of every Apply (its own cadence) so the fast-tick never execs wg/systemctl; LastResult is mutex-recorded per reconcileOnce, ok=false until the first pass

2. Canonical patterns (copy structure from THE named file)

Pattern	Canonical file	Key traits
Validate-then-exec privileged op	internal/storage/hostops.go (EnsureMount)	validate EVERY arg (validate.go) → render → stage in agent dir → root install → systemctl; refuse before any command exists
Fail-safe pure classifier over gathered facts	internal/storage/claim.go (classifyClaim over claimFacts)	pure function ⇒ fixture-testable; any error/ambiguity refuses; gather separated from verdict
Anti-retarget durable-id binding	internal/localapi/wipe_reresolve.go	resolve id → re-derive + exact match → re-inspect expected state → act on RE-RESOLVED device only
Atomic single-file JSON store	internal/storage/intent.go	Open* loads (missing=empty, corrupt=fail-loud), mutex, tmp+rename 0600, idempotent set
Durable append-only log + index	internal/authz/noncestore.go (FileNonceStore)	fsync before returning "new"; replay into index on open; expiry-only compaction
Injectable seam funcs on Server	internal/localapi/server.go (reresolveWipe, deviceDurableID, boundCheck, deviceCheck, livenessCheck, net-verify: netTrigger/netMounted/netJournal/netReachable)	prod default wired in NewServer; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. For mount-table predicates prefer the DATA seams procSelfMountinfo / procGuestMountinfo (internal/localapi/intermediary.go) over boundCheck/livenessCheck: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did
Server.devicePresent (R-113, v0.114.0)	internal/localapi/disks.go	devicePresent(rawMountPath) bool; seam deviceCheck, default isHostMountpoint
bindLiveness + BindLiveness (R-117, v0.117.0)	internal/localapi/intermediary.go	bindLiveness(stable, raw) BindLiveness; seam livenessCheck; read verdicts ONLY via .Usable()
AttachDrive repair ruling (R-117, v0.117.0)	internal/localapi/intermediary.go	the switch bindLiveness(...) inside the n == 1 && GuestSeesMount arm
Detached IN-MEMORY verify job (single slot, deliberately unpersisted)	internal/localapi/netverifyjob.go	claim slot sync (single-flight 409) → detached pipeline off baseCtx → auto-rollback on fail; restart ⇒ slot empty ⇒ the CALLER rolls back (Scenario F) — contrast formatjob (persisted+recovered)
Optional dependency degradation	internal/localapi/server.go (Options)	nil dep ⇒ endpoint answers "not configured" (503), never a crash
Version channel (v0.82.0)	internal/localapi/server.go (Options.AgentVersion; Handler() mux wrap)	sets X-Felhom-Agent-Version on EVERY response (all routes/statuses, incl. auth-fail/404) — the controller's capability-comparison source; empty version ⇒ header omitted
Root-file install via random temp	internal/localapi/intermediary.go (installSharedParentUnit)	stageTemp (os.CreateTemp) → sudoers-globbed install -m → pinned destination
Detached destructive job + restart recovery	internal/localapi/formatjob.go	persist running → run off baseCtx → record outcome; recovery re-resolves durable id, never a path
Signed-op verify pipeline	internal/authz/verifier.go (Verify)	armor→namespace→key-material allowlist→crypto over RAW bytes→blob→target→window→nonce LAST
Resilient daemon loop	internal/hub/loop.go (Loop.Run)	ctx-cancel returns nil; errors logged and loop continues
Guarded-below-the-agent wrapper	configs/felhom-mkfs-guarded.sh	root re-checks catastrophic cases (system disk/LVM PV/foreign mount/RO/member FS) even against an agent bug

3. Dangerous lookalikes — do NOT reuse

Trap	Why it bites	Use instead
Acting on the caller's req.Device (or any remembered /dev path) after inspection	classify→mkfs TOCTOU (AGENT-001): /dev re-enumeration retargets the node to a different physical disk	Server.reresolveWipe/reresolveBlank → format the RE-RESOLVED device
Exec'ing raw mkfs.* (incl. Binaries.MkfsExt4/MkfsXfs)	sudoers no longer allowlists raw mkfs; bypasses the claim filter + wrapper re-checks	SudoHostOps.Format (→ felhom-mkfs-guarded)
DiskInfo.DurableID (uuid:) as a wipe-confirmation id	wipe gate resolves byid:/byuuid: — a uuid: id is a binding_mismatch (F20-BUG2)	DiskInfo.WipeDurableID / storage.DeviceDurableID
ResolveDurableDevice for enrolled-storage remounts (or vice versa)	schemes differ: wipe = byid:/byuuid:, storage = uuid: — each refuses the other	ResolveStorageDevice for mounts; ResolveDurableDevice for wipes
authz.MemoryNonceStore on a real host	replay protection dies on restart	authz.FileNonceStore
Adding methods to proxmox.Privileged	breaks the 3-exception root-CLI fence (routing_test.go)	proxmox.Runner + a new sudoers Cmnd_Alias + validate.go-style checks
Calling Client.DestroyLXC/Vzdump/SetConfig outside the gate/queue/journal	skips classification, signature, per-guest serialization, crash recovery	reconcile.Engine paths / RunSignedJob; queue via Queue.Submit
GuestBinder.AttachBind/DetachBind (per-drive pct set -mpN)	legacy model; a missing bind source can brick guest boot (C1)	AttachDrive/DetachDrive (intermediary model)
isHostMountpoint to reconcile bind state	boolean can't converge stacked double-binds (the /mnt doubling bug)	countHostMounts normalization inside AttachDrive
Acting on a raw ListLXC list as if it were "guests the agent owns"	audit A1 (pre-v0.62.0 the stale-lock reaper did exactly this — contained only by the pool-scoped token)	ownership must be PROVEN: intersect with Client.Pool membership like staleLockController.Guests() (v0.62.0), fail-safe on read failure

4. Seams & interfaces (testing + cross-repo)

Interface	Defined in	Implemented by	Fakes/tests at
proxmox.Runner	internal/proxmox/privileged.go	*ExecRunner (direct/sudo)	mockRunner internal/proxmox/mock_test.go; runner fakes in storage tests
storage.HostOps	internal/storage/hostops.go	*SudoHostOps (prod), NoopHostOps (degraded)	fakes in internal/storage/observe_test.go, watchdog_test.go
storage.HostReader	internal/storage/hostread.go	*ProcHostReader	fakeHostReader internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: BlockSlaves(name) lists /sys/block/<name>/slaves (root-free) — backs the SystemDisks dm/md walk (physicalDisksOf/walkSlaves, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test TestSystemDisks_WalkTopologies (root-backing disk always in the system set).
localapi.DiskOps / StorageGate / GuestAttacher / GuestLister	internal/localapi/disks.go	*storage.SudoHostOps; storageGateAdapter (cmd/felhom-agent/main.go); *GuestBinder; *proxmox.Client	fakeDiskOps/fakeGate/fakeGuestAttacher/fakeGuestList internal/localapi/disks_test.go
localapi.GuestAPI / BackupService / BackupStore / TokenAuthority	internal/localapi/server.go	*proxmox.Client, *backup.BackupRunner, *backup.Store, *TokenStore	fakeGuests/fakeBackups/fakeStore internal/localapi/server_test.go
backup.InFlight	internal/backup/inflight.go	TryAcquire(what) (release, busy, ok) / Busy()	THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85)
capability store-grant probe (storeGrantStatuses / storeGrantVerdict / Client.Permissions)	cmd/felhom-agent/main.go, internal/proxmox/query.go	"may the agent READ this backup tier?", one capability.Status per configured tier	R-185. Never infer permission from an empty content listing — {"data":[]} is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask /access/permissions as the agent's own token (root always says yes). The ungranted answer is not empty and not a 403 — it carries the privileges inherited from the box-wide / grant, so test for Datastore.AllocateSpace specifically; path-presence or Datastore.Audit reports a blinded storage healthy. Probed set comes from BackupTiers(), never a fixed list. Critical except the local fallback. Composes AROUND the sudo prober (the poolReadStatus precedent); Status's wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok.
backup.RestoreTestState	internal/backup/restoretest_state.go	RecordSuccess(target,archive,tier,verified,t) / ProvenArchive(target) / ProvenRestoreTests(ctx) / LastSuccess(target) / OldestFirst(targets)	Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — which archive was proven, and when (R-86)
hub.ProvenRestoreTestReporter + Collector.SetProvenRestoreTests	internal/hub/collect.go	the DURABLE restore-test source, merged with the in-memory one	R-189. Merge rule: one entry per tier, newest by TestedAt wins — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. The wiring is pinned by an AST test — the method this replaced (RestoreTestState.Snapshot) carried a doc comment naming a host-report gauge and had no caller for weeks.
backup.SpecBuilder / backup.TierPicker / (*BackupRunner).PickSettledRestoreCandidateOn	internal/backup/schedule.go, runner.go	func(ctx,archive) RestoreTestSpec; func(ctx,target,notAfter) (archive,landed,error)	The per-run restore-test spec + per-tier settled candidate lookup (R-85, widened by R-86)
localapi.BackupTier + normalizeBackupTiers / config.BackupConfig.BackupTiers	internal/localapi/backup_tiers.go, internal/config/config.go	normalizeBackupTiers(tiers, legacy, cadence) []BackupTier; BackupTiers() ([]BackupTier, []string)	THE R-82 multi-tier resolution — one runner per tier, primary first
localapi.StaleLockController	internal/localapi/stalelock.go	*staleLockController (Client + Runner + pool)	fakeStaleLock (Server-level) stalelock_test.go; fakeStaleLockAPI (controller-level, tests the A1 pool intersect) stalelock_pool_test.go
localapi.GuestExecutor	internal/localapi/controllerswap.go	*GuestBinder (pct exec)	fakeGuestExec internal/localapi/controllerswap_test.go
guestnet.Runner / guestnet.GuestSource (R-54, v0.92.0)	internal/guestnet/{probe,watchdog}.go	*proxmox.ExecRunner; the POOL-VERIFIED localapi.StaleLockController.Guests (ListLXC ∩ felhom pool, audit A1)	scriptedRunner + fakeGuests internal/guestnet/watchdog_test.go. Never wire a bare ListLXC here — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record zero heal calls
guestnet.Watchdog.SetDampers / now (clock seam)	internal/guestnet/watchdog.go	config guest_net.*; now defaults to time.Now	tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment
hub.GuestNetReporter (R-54)	internal/hub/collect.go	*guestnet.Watchdog (GuestNetStatus)	internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION Collect path AND that the guest_net key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind
hub.AddressEnumerator (v0.119.0)	internal/hub/hostaddr.go	defaults to the REAL systemInterfaces when Collector.addrEnum is nil — deliberately inverting the nil-reporter-means-off convention, because this stanza has no config gate and a forgotten wiring call would otherwise ship silently empty (the inert-seam shape, four instances on record)	internal/hub/hostaddr_test.go drives fixtures TRANSCRIBED from ip -o addr show on demo-felhom AND demo-hp, including the address-less veth/NIC rows — the "no denylist needed" claim rests on those rows really being empty, so omitting them would prove the claim by assuming it. filterHostAddresses keeps GLOBAL UNICAST only: one predicate that drops loopback, fe80::/10, and 169.254/16 — the last being the R-50 island literal, identical on every box and actively misleading if surfaced
reconcile.OpVerifier	internal/reconcile/gate.go	*authz.Verifier	fake verifier in internal/reconcile gate tests
signedjobs.WipeOps / Executor (ExecutorChain)	internal/signedjobs/wipe.go + runner.go	*storage.SudoHostOps; WipeExecutor+DecommissionExecutor	internal/signedjobs wipe/runner/decommission tests
hub.reporter / collectorIface / EnvelopeObserver	internal/hub/loop.go	*hub.Client, *hub.Collector; desired.Syncer + signedjobs.Runner	fakeReporter/fakeCollector internal/hub/loop_test.go; recordingReporter loop_logtail_test.go
applog.Ring (always-DEBUG capture ring) + fan-out applog.New → (logger, ring)	internal/log/log.go	wired in cmd/felhom-agent/main.go → localapi.Options.LogRing + Loop.SetLogTailSource(ring.Lines)	internal/log/log_test.go; localapi/debuglogs_test.go — v0.83.0; the byte-capped Lines is the heartbeat tail source
pbsdr.StorageReader / SecretConsumer / Manager.probeFP (func seam)	internal/pbsdr/manager.go	*proxmox.Client; *hub.Client; pbs.ProbeFingerprint	fakeStorage/fakeConsumer/fakeRunner internal/pbsdr/manager_test.go (argv+stdin recorder)
capability.Runner	internal/capability/probe.go	*proxmox.ExecRunner (RunnerDirect)	fakeRunner internal/capability/probe_test.go
Cross-repo: local API ↔ controller	internal/localapi/server.go routes; contract seeded by internal/provision/doc.go (bootstrap.json: endpoint + leaf fingerprint + token)	felhom-controller's agentapi client	pin = served leaf cert (memory gotcha)
Cross-repo: agent ↔ hub	internal/hub/report.go (HostReport), ControlEnvelope; POST /api/v1/host-report	hub mirrors structs field-for-field	new event/report fields need hub-side ingest changes
Cross-repo: shipped host artifacts	configs/felhom-agent.sudoers, configs/felhom-mkfs-guarded.sh, configs/felhom-pbs-apply, shared-parent script/unit (inline in internal/localapi/intermediary.go)	deployed WITH the binary	sudoers globs must match stageTemp patterns + staging dirs exactly
Operator signing	internal/authz (OpBlob, SSHSIG)	cmd/felhom-opsign (offline CLI)	blob/verify tests in internal/authz

5. Extension points (where new features plug in)

• Local-API route: add to Server.Handler (internal/localapi/server.go) wrapped in s.withGuest; new deps go into Options as OPTIONAL fields degrading to "not configured".
• New signed-job verb: implement signedjobs.Executor (return ErrNoExecutor for foreign ops) and append to the signedjobs.ExecutorChain in cmd/felhom-agent/main.go; add the op class + role scoping in internal/reconcile (classify.go, gate.go roleAuthorizes).
• New privileged host op: validate args (internal/storage/validate.go style) → exec via Runner → add a Cmnd_Alias to configs/felhom-agent.sudoers → add a probe vector to internal/capability/manifest.go (so degradation is visible) → ship sudoers with the binary.
• New reconcile action: ActionKind + classOfAction (internal/reconcile/classify.go), plan emission in internal/reconcile/plan.go; destructive ⇒ gate handles it automatically.
• Hub-report field: extend hub.HostReport (internal/hub/report.go) + Collector — hub side must mirror + allowlist it (cross-repo).
• DR-recipe section (host-half): add the field to DRRecipeHostHalf (internal/hub/dr_recipe.go) AND to the hub's hostHalfShape + AssembledRecipe (felhom.eu hub/internal/store/dr_recipe.go). Those two hub structs are ALLOW-LISTS: a section only the agent knows about is stored intact and silently dropped before any operator sees it — that is R-122, which cost offsite_restic its entire existence. Then update BOTH copies of testdata/host-report.golden.json (byte-identical, cross-repo) and extend TestAssembleDRRecipe_CarriesEveryEmittedSection. A recipe field that cannot be resolved records an explicit unknown with a reason — never a default, an empty string, or a placeholder (DRState* / DRReason*); a recipe read during a rebuild must not present a guess as a fact.
• Envelope-driven behavior: implement hub.EnvelopeObserver, add to the MultiObserver in cmd/felhom-agent/main.go.
• Selftest mode: selftestFlag + runSelftest* in cmd/felhom-agent/main.go.
• Config: internal/config/config.go (Load + applyEnv FELHOM_AGENT_* overlay; keep secrets out of Redacted() output).

6. Known duplication (observed — NOT fixed)

• Two lsblk -J parsers with near-identical structs: parseLsblkDevice/lsblkDevice (internal/storage/hostops.go) vs parseLsblkNodes/lsblkDev (internal/storage/claim.go).
• Two smartctl -a -j paths: SudoHostOps.SMART (internal/storage/hostops.go, parsed hub.SmartSummary) vs Privileged.SMART (internal/proxmox/privileged.go, raw map).
• SMART device resolution (v0.95.0): smartDeviceFor (internal/storage/observe.go) resolves partition→disk AND dm/LVM→disk (dmWholeDisk in internal/storage/smartdev.go, via /sys/block/<dm>/slaves, sysBlockRoot test seam). storage.SmartReader.SMARTForBacking is the shared read the localapi /disks union path uses (Fix B) — do NOT re-implement smartctl parsing. The builtin-local SMART device comes from containingMountDevice (SMART-only; never feeds backing/durable_id).
• Atomic tmp+rename JSON store implemented 3×: IntentStore.saveLocked (internal/storage/intent.go), FormatJobStore.save (internal/localapi/formatjob.go), GuestBindStore.saveLocked (internal/localapi/guestbindstore.go) — comments say "mirrors", no shared helper.
• run(ctx, name, args...) error stderr-wrapping helper duplicated 4×: SudoHostOps.run, Privileged.run, BackHalf.run (internal/provision/backhalf.go), GuestBinder.run (internal/localapi/guestbind.go).
• Several independent /proc mount-table readers: SudoHostOps.mountedSet (internal/storage/hostops.go), ProcHostReader.Mounts (internal/storage/hostread.go). In localapi they were unified in v0.117.0: isHostMountpoint and countHostMounts are now one-liners over hostMountEntries, the single parser that also yields devno/fstype/super-options for bindLiveness.
• Deliberate mirror: antiRetargetResolveExpect (internal/localapi/wipe_reresolve.go) duplicates WipeExecutor.Execute steps 1–3 (internal/signedjobs/wipe.go) across packages.
• stableParentDir literal duplicated in internal/provision/backhalf.go to avoid a provision→localapi import edge (commented as intentional); trim (internal/storage/hostops.go) vs trimBody (internal/proxmox/errors.go) output-truncation twins.