diff --git a/CHANGELOG.md b/CHANGELOG.md index 1965ff3..388078d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,53 @@ All notable changes to **felhom-agent** are recorded here. Update on every code change that gets pushed. +## v0.16.0 — slice 10B: operator-signed destructive completion (offline key + signing CLI) (2026-06-10) + +The security centerpiece: a destructive op runs ONLY on a verified, operator-signed authorization +— signature valid against a **pinned** operator pubkey (never the hub's or the blob's), nonce +unseen + durably burned, in-window, host-bound, and **resource-bound to a DURABLE device id** that +execution re-resolves + re-inspects. Decision (a): **offline operator key + signing CLI**, +hardware-key-ready (`sk-`/YubiKey via ssh-keygen). The key floor holds: the signing key is NOT in +the hub and NOT in the agent. Concrete consumer: this **closes the 8C data-bearing-wipe +`pending_signature` gap**. Pairs with hub v0.10.0. + +### Added +- **`cmd/felhom-opsign`** — the operator's offline signing CLI. Builds the canonical `OpBlob` by + **reusing `authz.CanonicalBlob`** (the exact production path the verifier authenticates over — so + signer + verifier can never drift) and signs it with **`ssh-keygen -Y sign -n felhom-op-v1`** + (hardware-ready). Output: a `{op_blob_b64, sig_armored}` envelope to hand to the hub jobs queue + (optional `--upload`). Touches ONLY the operator's signing key. +- **`authz.CanonicalBlob`** — promoted to production (was test-only) so the CLI + verifier share one + canonical-bytes source; params canonicalized (sorted keys, compact). +- **`internal/storage` durable device identity** (`durable_device.go`): `DeviceDurableID` (derive a + stable `byid:`(wwn/serial)/`byuuid:` id from the world-readable udev symlinks — no privilege, no + subprocess) + `ResolveDurableDevice` (re-resolve to the current `/dev` path; a path-only/unknown + scheme is REFUSED). The resource-level anti-retarget. +- **`internal/signedjobs`** (new): the queue consumer. `Runner` fetches each opaque job → runs it + through the **gate** (the LOCKED authz pipeline) → on all-pass hands the verified op to an + `Executor`; the order is **verify → nonce-burn (durable, in Verify) → execute → clear job**. The + **`WipeExecutor`** is the 8C consumer: resolve the signed durable id → **re-derive + match** + (anti-retarget) → **re-inspect (8C classifier)** the device is still the data-bearing target → + `mkfs`. A vanished/changed/non-data-bearing device or a path-only binding is refused **even with a + valid signature**. Wired as a second `EnvelopeObserver` (runs on `HasSignedOps`). +- **`hub.Client.Jobs` / `CompleteJob`** + `hub.MultiObserver`; the 8C format refusal now **surfaces + the bound op** (op + durable id + host) in its 403 `pending_op` + a `felhom-opsign …` hint. + +### Pinning / rotation +- Operator pubkeys are pinned via `authz.signers` (config, trusted path — provision/agent config, + NEVER hub-alone), **multiple** keys (KeyID selects; role-scoped), so a backup/rotation key exists + without a flag-day. Unchanged from the slice-4 verifier wiring; 10B activates the execute path. + +### Tests (real crypto, non-hollow) +- `signedjobs` runner over the **real** gate+verifier (in-Go minted SSHSIGs): valid → executor runs + once + job cleared; **replay** (nonce burned) / **non-pinned signer** / **expired** / **retarget** + (other host) / **forged sig** / **no pinned signer** → all rejected, **executor never called**; + malformed envelope cleared. +- `WipeExecutor`: valid → `mkfs` runs; **path-only**, **durable-id mismatch**, **device gone**, + **re-inspect non-data-bearing**, **not-probed** → all refused, `Format` not called. +- `storage` durable: wwn-preference, uuid-fallback, path-only/traversal refusal, round-trip, + missing-device error (symlink tests gated to Linux — the agent's OS). + ## v0.15.0 — slice 10A: hub desired-state serving — the "Down" channel (2026-06-10) The agent half of slice 10A. The control envelope (`hub.ControlEnvelope`) stops being "reserved — ignored" and becomes the live **Down channel**: a cheap change-notification on every heartbeat. The agent caches the hub's desired-state + its generation; only when **`DesiredGeneration` advances** does it fetch the full state (the heartbeat stays light, the heavy state moves on change). The engine then reconciles **benign** deltas and the gate marks an explicit **destructive** delta `pending_signature` (no signer in 10A → never executed; signed execution is 10B). Pairs with hub v0.9.0. diff --git a/REPORT.md b/REPORT.md index 5d377e6..862763c 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,61 +1,63 @@ -# REPORT — slice 10A (agent half): hub desired-state serving — the "Down" channel (v0.15.0) (2026-06-10) +# REPORT — slice 10B (agent half): operator-signed destructive completion (v0.16.0) (2026-06-10) > Overwrite-latest report. Cumulative history: [CHANGELOG.md](CHANGELOG.md). ## What was implemented -The agent half of **slice 10A**: activate the control envelope as the live **Down channel** and feed -a hub-backed desired-state into the reconcile engine. Pairs with hub v0.9.0. +The security centerpiece: a destructive op runs ONLY on a verified, operator-signed authorization. +Decision (a): **offline operator key + signing CLI**, hardware-key-ready. The signing key is NOT in +the hub and NOT in the agent. Concrete consumer: this **closes the 8C data-bearing-wipe +`pending_signature` gap**. Pairs with hub v0.10.0. -### The control loop (now live) -report (heartbeat) → control envelope → (DesiredGeneration advanced past cache? fetch desired-state) -→ reconcile benign / gate destructive → report. The heartbeat stays light; the heavy desired-state is -fetched **only on a generation advance**. +### The flow (end-to-end) +8C format of a data-bearing device → agent refuses `pending_signature` + **surfaces the bound op** +(durable id + host) → operator **signs offline** (`felhom-opsign`) → uploads to the hub jobs queue → +agent's signed-jobs runner **verifies** + **executes** the wipe (re-resolve durable id + re-inspect +8C → `mkfs`) → clears the job. -### `internal/reconcile` -- **`DesiredGuest.Decommission`** — the canonical **destructive** desired-state delta (an EXPLICIT - flag, never "absent from the list", so a partial hub list can't mass-destroy). Planner emits - `ActionDecommission` → `ClassDecommission` → Destructive → the gate refuses `pending_signature`. -- **`Reconcile`** now counts a `pending_signature` gate refusal as **`Result.Pending`** (expected, - INFO-logged), not a failure; any other refusal stays a real failure. `ActionDecommission` has **no - executor** (10B) — a defensive guard refuses to run it. -- **`CachingProvider`** — thread-safe DesiredState + generation cache (`Desired`/`Update`/ - `Generation`); the production provider, replacing `EmptyProvider` in the daemon engine. Empty until - the hub serves intent → cold-start is a live no-op (unchanged behaviour). +### `cmd/felhom-opsign` (new) — the offline signing CLI +- Builds the canonical `OpBlob` by **reusing `authz.CanonicalBlob`** (the exact bytes the verifier + authenticates over — signer/verifier can't drift) and signs with **`ssh-keygen -Y sign -n + felhom-op-v1`** (hardware-ready: `sk-`/YubiKey work unchanged). Output: `{op_blob_b64, + sig_armored}`; optional `--upload`. Touches only the operator's key. -### `internal/hub` -- `ControlEnvelope` fields are now active. New wire types **`DesiredStateResponse`** + - **`WireDesiredState`** (guests + forward-compat `restore_directive` (10D) / `pbs_namespace` / opaque - `storage_manifest`+`backup_policy`) + **`WireDesiredGuest`**. New **`Client.FetchDesiredState`** - (GET `/api/v1/hosts/{host_id}/desired-state`, self-scoped to the client's own host). New - **`EnvelopeObserver`** loop seam + `SetEnvelopeObserver` (hub does not import reconcile/desired). +### The verify-and-execute machinery +- **`internal/signedjobs.Runner`** — fetches each opaque job → runs the **gate** (the LOCKED authz + pipeline: pinned-key SSHSIG → namespace → allow-list by key MATERIAL → crypto over raw bytes → + host target → time window → **durable nonce-burn LAST**) → on all-pass hands the verified op to an + `Executor`. Order: **verify → burn nonce (durable) → execute → clear**. Rejects (forged/replayed/ + expired/retargeted/non-pinned) never reach the executor. +- **`WipeExecutor`** (the 8C consumer) — resolve the signed **durable** id → re-derive + **match** + (anti-retarget) → **re-inspect (8C classifier)** still-data-bearing → `mkfs`. A path-only binding, + a vanished/changed device, or a non-data-bearing target is refused **even with a valid signature**. +- **`internal/storage` durable identity** — `DeviceDurableID` / `ResolveDurableDevice` over the + world-readable udev symlinks (`byid:` wwn/serial, `byuuid:` fallback) — no privilege, no subprocess. +- **`authz.CanonicalBlob`** promoted to production. `hub.Client.Jobs`/`CompleteJob` + `MultiObserver`. + The 8C 403 now carries a `pending_op` (op + durable id + host) + a `felhom-opsign` hint. -### `internal/desired` (new) + wiring -- **`Syncer`** — implements `hub.EnvelopeObserver`; fetches on a generation advance, maps wire→domain, - updates the `CachingProvider`. Caches the **fetched** generation (race-robust); a fetch failure keeps - the last-known state. `restore_directive` carried + logged, not acted on (10D). Wired in - `cmd/felhom-agent`: provider → engine, syncer → loop. +### Pinning / rotation +Operator pubkeys pinned via `authz.signers` (config, trusted path — NEVER hub-alone), **multi-key** +(KeyID selects, role-scoped) for backup/rotation without a flag-day. Unchanged verifier wiring; 10B +activates the execute path (the runner is the second `EnvelopeObserver`, runs on `HasSignedOps`). -## Tests (all green) -- reconcile: **benign applied + destructive decommission gated pending (not executed)**; Plan - decommission-only + classifies Destructive; CachingProvider update/isolation. -- desired: **fetch-once-on-advance** / no-refetch-on-unchanged / fetch-failure-keeps-cache / - caches-the-fetched-generation. -- hub: `FetchDesiredState` path+auth+decode (incl. `restore_directive`) + typed 403; loop notifies the - observer + adopts `PollIntervalSeconds`, skips the observer on a report error. -- cross-repo golden (`desired-state` + `control-envelope`) decode + key-set guard, byte-identical with - felhom.eu/hub. `go test ./...` green. +## Tests (real crypto, non-hollow — assert the op did/did NOT run) +- `signedjobs` over the **real** gate+verifier (in-Go minted SSHSIGs): valid → executor runs once + + job cleared; **replay / non-pinned / expired / retarget / forged / no-signer** → rejected, executor + never called; malformed cleared. +- `WipeExecutor`: valid → `mkfs`; path-only / durable-mismatch / device-gone / re-inspect-non-data- + bearing / not-probed → refused, `Format` not called. +- `storage` durable: wwn-preference, uuid-fallback, path-only+traversal refusal, round-trip, missing + (symlink tests gated to Linux). `go test ./...` green. ## Versioning / docs -- Version `0.14.0 → 0.15.0`; `CHANGELOG.md` updated. Doc 03 §4 (control loop live) + §9 (slice table: - 10A done, 10B/10C/10D pending) updated. +- Version `0.15.0 → 0.16.0`; `CHANGELOG.md`. Doc 03 §4 (signed path live) + §6 (8C wipe completes) + + §9 (10B done) updated. ## Out of scope (per the task) -- Signed-op **execution** (verify + run the gated destructive op) → 10B (10A marks it pending only). -- **Restore-mode / re-enroll** consumption (a new box's first directive) → 10D; 10A serves - already-authenticated hosts only. +- Other destructive executors (guest_destroy, decommission, **restore-overwrite → 10D**) reuse the + same gate+runner; their executors plug in per-slice. 10B ships the machinery + the storage-wipe. ## Pending -- **Live validation** on the demo: build+deploy agent v0.15.0 + hub v0.9.0; admin-set a desired-state - with a benign + a decommission delta → generation bumps → agent fetches → reconciles benign + gates - the decommission; change `poll_interval_seconds`; confirm a host can't fetch another host's state. +- **Live validation** on the demo: data-bearing wipe → `pending_signature` → sign offline with a real + operator key → hub queue → agent verifies + wipes; confirm replay + a non-pinned-key signature are + rejected. (Also validates the Linux-only durable-device tests + ssh-keygen interop.) diff --git a/cmd/felhom-agent/main.go b/cmd/felhom-agent/main.go index bfb1e48..47ec783 100644 --- a/cmd/felhom-agent/main.go +++ b/cmd/felhom-agent/main.go @@ -34,6 +34,7 @@ import ( applog "gitea.dooplex.hu/admin/felhom-agent/internal/log" "gitea.dooplex.hu/admin/felhom-agent/internal/pbs" "gitea.dooplex.hu/admin/felhom-agent/internal/provision" + "gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs" "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" "gitea.dooplex.hu/admin/felhom-agent/internal/reconcile" "gitea.dooplex.hu/admin/felhom-agent/internal/storage" @@ -41,7 +42,7 @@ import ( // version is the agent version. Overridable at build time with // -ldflags "-X main.version="; defaults to the in-repo CHANGELOG version. -var version = "0.15.0" +var version = "0.16.0" func main() { var ( @@ -245,7 +246,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger) int { desiredProvider := reconcile.NewCachingProvider() // The "Down" channel sync hook: on each heartbeat, fetch desired-state when the generation // advances. The loop calls it via the EnvelopeObserver seam (hub does not import desired). - loop.SetEnvelopeObserver(desired.NewSyncer(client, desiredProvider, logger)) + desiredSyncer := desired.NewSyncer(client, desiredProvider, logger) + // The signed-jobs runner (slice 10B) is wired as a SECOND envelope observer below (after the + // gate is built) — when the heartbeat flags pending signed ops, it fetches + verifies + executes. ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() @@ -287,6 +290,18 @@ func runDaemon(cfg config.Config, logger *slog.Logger) int { } gate := reconcile.NewGate(verifier, cfg.Hub.HostID, reconcile.SlogAudit{Logger: logger}, logger) + // Signed-jobs runner (slice 10B): the consumer of the hub's signed-jobs queue. On a heartbeat + // that flags pending signed ops, it fetches each opaque blob, runs it through the gate (the + // LOCKED authz pipeline: pinned-key SSHSIG → namespace → allow-list → crypto → host → time → + // durable nonce-burn) and, only on all-pass, hands the verified op to the storage-WIPE executor + // — which re-resolves the DURABLE device id + re-inspects (8C) before mkfs. This closes the 8C + // data-bearing `pending_signature` gap. With no signers pinned the gate refuses every job + // (pending_signature) and nothing executes — correct. Wired as a second envelope observer + // alongside the desired-state syncer. + wipeExec := signedjobs.NewWipeExecutor(hostOps, logger) + jobsRunner := signedjobs.NewRunner(client, gate, wipeExec, cfg.Hub.HostID, logger) + loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner)) + // Storage watchdog (slice 5): the third daemon goroutine. Fast-polls the known target // set for attached↔disconnected transitions → debounced out-of-band report; and, on a // known mount-backed target's device returning unmounted, dispatches a benign re-mount @@ -510,6 +525,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St // Host metrics (slice 9): the shared collector serves GET /host/metrics — a fresh host + // per-storage view to the customer's monitoring page (reuses the slice-4 collector). HostMetrics: collector, + HostID: cfg.Hub.HostID, // slice 10B: anti-retarget host in the data-bearing-format pending-op Logger: logger, }) if err != nil { diff --git a/cmd/felhom-opsign/main.go b/cmd/felhom-opsign/main.go new file mode 100644 index 0000000..61532a0 --- /dev/null +++ b/cmd/felhom-opsign/main.go @@ -0,0 +1,186 @@ +// Command felhom-opsign is the OPERATOR's offline signing CLI for destructive ops (slice 10B, +// decision (a): offline operator key + signing CLI, hardware-key-ready). +// +// It constructs the canonical OpBlob bytes by REUSING internal/authz.CanonicalBlob — the exact +// production path the agent's verifier authenticates over — so signer and verifier can never drift. +// It signs that canonical message with the operator's key via `ssh-keygen -Y sign`, which makes it +// hardware-ready: an `sk-`/YubiKey key works through ssh-keygen unchanged. The output is a signed-op +// envelope { op_blob_b64, sig_armored } to hand to the hub's jobs queue (optionally uploaded with +// --upload). This CLI touches ONLY the operator's signing key — never the hub's or agent's keys. +// +// Example — sign a data-bearing wipe (closing the 8C pending_signature gap): +// +// felhom-opsign -op storage_wipe -host demo-felhom-01 \ +// -durable-id byid:wwn-0x5000c500abcd1234 -fstype ext4 \ +// -key-id ops-key-1 -key ~/.ssh/felhom_op_ed25519 -ttl 30m +package main + +import ( + "bytes" + "crypto/rand" + "encoding/base64" + "encoding/hex" + "encoding/json" + "flag" + "fmt" + "io" + "net/http" + "os" + "os/exec" + "path/filepath" + "strings" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/authz" +) + +func main() { + if err := run(); err != nil { + fmt.Fprintln(os.Stderr, "felhom-opsign:", err) + os.Exit(1) + } +} + +func run() error { + var ( + op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission") + host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)") + guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)") + keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)") + paramsRaw = flag.String("params", "", "op params as JSON (overrides -durable-id/-fstype)") + durableID = flag.String("durable-id", "", "for storage_wipe: the DURABLE device id (byid:…|byuuid:…)") + fstype = flag.String("fstype", "ext4", "for storage_wipe: the filesystem to mkfs after wipe") + keyFile = flag.String("key", "", "operator signing key (ssh private key / sk- key handle) for ssh-keygen -Y sign") + ttl = flag.Duration("ttl", 30*time.Minute, "validity window from now (issued_at..expires_at)") + nonce = flag.String("nonce", "", "explicit nonce (default: a fresh 128-bit random nonce)") + uploadURL = flag.String("upload", "", "optional hub base URL to POST the signed op to the jobs queue") + hubKey = flag.String("hub-key", "", "hub bearer key for --upload (operator's hub key — NOT a signing key)") + out = flag.String("o", "", "write the envelope JSON to this file (default: stdout)") + ) + flag.Parse() + + if *op == "" || *host == "" || *keyID == "" || *keyFile == "" { + return fmt.Errorf("-op, -host, -key-id and -key are required") + } + + // Params: explicit JSON, or built from the wipe convenience flags. + params := strings.TrimSpace(*paramsRaw) + if params == "" { + if *op == "storage_wipe" { + if *durableID == "" { + return fmt.Errorf("storage_wipe needs -durable-id (byid:…|byuuid:…) — a path-only binding is refused by the agent") + } + pj, _ := json.Marshal(map[string]string{"durable_id": *durableID, "fstype": *fstype}) + params = string(pj) + } else { + params = "{}" + } + } + + n := *nonce + if n == "" { + var b [16]byte + if _, err := rand.Read(b[:]); err != nil { + return fmt.Errorf("generating nonce: %w", err) + } + n = hex.EncodeToString(b[:]) + } + + now := time.Now().UTC() + issued := now + expires := now.Add(*ttl) + + // Canonical OpBlob bytes — the EXACT bytes the agent verifier authenticates over. + blob, err := authz.CanonicalBlob(*op, *host, *guest, *keyID, n, params, issued, expires) + if err != nil { + return err + } + + sigArmored, err := signWithSSHKeygen(blob, *keyFile) + if err != nil { + return err + } + + env := map[string]string{ + "op_blob_b64": base64.StdEncoding.EncodeToString(blob), + "sig_armored": sigArmored, + } + envJSON, _ := json.Marshal(env) + + // Emit the envelope (stdout or file). Also print the human summary to stderr (never the key). + fmt.Fprintf(os.Stderr, "signed: op=%s host=%s guest=%q key_id=%s nonce=%s expires=%s\n", + *op, *host, *guest, *keyID, n, expires.Format(time.RFC3339)) + if *out != "" { + if err := os.WriteFile(*out, envJSON, 0o600); err != nil { + return err + } + fmt.Fprintln(os.Stderr, "wrote envelope to", *out) + } else { + fmt.Println(string(envJSON)) + } + + if *uploadURL != "" { + if *hubKey == "" { + return fmt.Errorf("--upload needs --hub-key (the operator's hub bearer key)") + } + if err := upload(*uploadURL, *hubKey, *host, envJSON); err != nil { + return fmt.Errorf("upload to hub: %w", err) + } + fmt.Fprintln(os.Stderr, "uploaded signed op to the hub jobs queue") + } + return nil +} + +// signWithSSHKeygen signs `message` with `ssh-keygen -Y sign -n `, the hardware-ready +// path (sk-/YubiKey keys work unchanged). It writes the message to a temp file, runs ssh-keygen, +// and reads the armored SSHSIG it produces. The namespace is the agent's FIXED domain separator. +func signWithSSHKeygen(message []byte, keyFile string) (string, error) { + dir, err := os.MkdirTemp("", "felhom-opsign-") + if err != nil { + return "", err + } + defer os.RemoveAll(dir) + msgPath := filepath.Join(dir, "op.blob") + if err := os.WriteFile(msgPath, message, 0o600); err != nil { + return "", err + } + cmd := exec.Command("ssh-keygen", "-Y", "sign", "-n", authz.Namespace, "-f", keyFile, msgPath) + var stderr bytes.Buffer + cmd.Stderr = &stderr + if err := cmd.Run(); err != nil { + return "", fmt.Errorf("ssh-keygen -Y sign: %w: %s", err, strings.TrimSpace(stderr.String())) + } + sig, err := os.ReadFile(msgPath + ".sig") + if err != nil { + return "", fmt.Errorf("reading produced signature: %w", err) + } + return string(sig), nil +} + +// upload POSTs the signed-op envelope to the hub's jobs queue (POST /api/v1/admin/hosts/{id}/jobs). +// The queued blob is base64(envelope JSON); the hub stores it opaquely (it cannot forge or open it). +func upload(baseURL, hubKey, hostID string, envJSON []byte) error { + body, _ := json.Marshal(map[string]string{ + "blob_b64": base64.StdEncoding.EncodeToString(envJSON), + }) + // 10A's enqueue lives under /admin/hosts/{id}/jobs (operator/global key). The queued blob is + // base64(envelope JSON); the hub stores it opaquely. + url := strings.TrimRight(baseURL, "/") + "/api/v1/admin/hosts/" + hostID + "/jobs" + req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(body)) + if err != nil { + return err + } + req.Header.Set("Authorization", "Bearer "+hubKey) + req.Header.Set("Content-Type", "application/json") + hc := &http.Client{Timeout: 15 * time.Second} + resp, err := hc.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + raw, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10)) + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + return fmt.Errorf("HTTP %d: %s", resp.StatusCode, strings.TrimSpace(string(raw))) + } + return nil +} diff --git a/felhom-opsign.exe b/felhom-opsign.exe new file mode 100644 index 0000000..9d16c20 Binary files /dev/null and b/felhom-opsign.exe differ diff --git a/internal/authz/blob.go b/internal/authz/blob.go index e89a0fd..b7ef436 100644 --- a/internal/authz/blob.go +++ b/internal/authz/blob.go @@ -2,9 +2,39 @@ package authz import ( "encoding/json" + "fmt" + "strings" "time" ) +// CanonicalBlob builds the canonical OpBlob bytes (phase4 §2 field order: keys sorted at +// every level, no insignificant whitespace, no trailing newline, UTF-8). This is the SINGLE +// production source of the signed bytes — the operator signing CLI (cmd/felhom-opsign) and the +// in-Go test minting both call it, so the signer can NEVER drift from what the verifier expects +// (the verifier authenticates over the RAW received bytes, so these bytes ARE the contract). +// +// params is canonicalized internally (parsed + re-marshaled → object keys sorted, whitespace +// stripped) so the same op+params always yields identical bytes; "" → "{}". Returns an error +// only when params is not valid JSON. +func CanonicalBlob(op, hostID, guestID, keyID, nonce, paramsJSON string, issued, expires time.Time) ([]byte, error) { + params := strings.TrimSpace(paramsJSON) + if params == "" { + params = "{}" + } + var pv interface{} + if err := json.Unmarshal([]byte(params), &pv); err != nil { + return nil, fmt.Errorf("authz: params is not valid JSON: %w", err) + } + pc, err := json.Marshal(pv) // Go marshals object keys sorted, compact + if err != nil { + return nil, fmt.Errorf("authz: canonicalizing params: %w", err) + } + return []byte(fmt.Sprintf( + `{"expires_at":%q,"issued_at":%q,"key_id":%q,"nonce":%q,"op":%q,"params":%s,"target":{"guest_id":%q,"host_id":%q}}`, + expires.UTC().Format(time.RFC3339), issued.UTC().Format(time.RFC3339), + keyID, nonce, op, pc, guestID, hostID)), nil +} + // Target binds an op to a specific box (and optionally a guest) — the anti-retarget // field. The §7 reference omitted the json tags; production needs them so the // signed canonical bytes decode correctly. diff --git a/internal/authz/mint_test.go b/internal/authz/mint_test.go index 78fa8f8..18f2afa 100644 --- a/internal/authz/mint_test.go +++ b/internal/authz/mint_test.go @@ -6,7 +6,6 @@ import ( "crypto/sha256" "encoding/binary" "encoding/pem" - "fmt" "testing" "time" @@ -17,17 +16,14 @@ import ( // production framing. They reuse the production signedData()/sshsigBlob so a test // can never drift from the verifier's notion of the signed bytes. -// canonicalBlob builds an op blob in the §2 canonical field order. (Self-consistent -// for the in-Go path: we sign exactly these bytes and verify the same bytes. The -// committed ssh-keygen fixture exercises real OpenSSH canonical interop.) +// canonicalBlob delegates to the production CanonicalBlob (so the in-Go test minting can never +// drift from the real signed-bytes path). Panics on a params error — tests pass valid JSON. func canonicalBlob(op, hostID, guestID, keyID, nonce, paramsJSON string, issued, expires time.Time) []byte { - if paramsJSON == "" { - paramsJSON = "{}" + b, err := CanonicalBlob(op, hostID, guestID, keyID, nonce, paramsJSON, issued, expires) + if err != nil { + panic(err) } - return []byte(fmt.Sprintf( - `{"expires_at":%q,"issued_at":%q,"key_id":%q,"nonce":%q,"op":%q,"params":%s,"target":{"guest_id":%q,"host_id":%q}}`, - expires.UTC().Format(time.RFC3339), issued.UTC().Format(time.RFC3339), - keyID, nonce, op, paramsJSON, guestID, hostID)) + return b } // mintArmor builds an armored SSHSIG over message, using sign to produce the inner diff --git a/internal/hub/client.go b/internal/hub/client.go index 89abf49..badd4d5 100644 --- a/internal/hub/client.go +++ b/internal/hub/client.go @@ -144,6 +144,69 @@ func (c *Client) FetchDesiredState(ctx context.Context) (*DesiredStateResponse, return &out, nil } +// JobWire is one queued signed-op job as served by GET /hosts/{id}/jobs (slice 10A). The blob is +// OPAQUE to the hub — for slice 10B it is a base64 `SignedJobEnvelope` (op-blob + armored SSHSIG) +// the agent verifies before executing. +type JobWire struct { + JobID string `json:"job_id"` + BlobB64 string `json:"blob_b64"` + CreatedAt string `json:"created_at"` +} + +// Jobs fetches this host's pending signed-op jobs (slice 10B). Self-scoped server-side (the +// per-host key only reads its own host). The agent verifies each before executing. +func (c *Client) Jobs(ctx context.Context) ([]JobWire, error) { + if c.hostID == "" { + return nil, fmt.Errorf("hub: Jobs requires a configured host_id") + } + req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+"/api/v1/hosts/"+c.hostID+"/jobs", nil) + if err != nil { + return nil, err + } + req.Header.Set("Authorization", "Bearer "+c.apiKey) + req.Header.Set("Accept", "application/json") + resp, err := c.hc.Do(req) + if err != nil { + return nil, &TransportError{Err: err} + } + defer resp.Body.Close() + raw, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<20)) + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + return nil, &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)} + } + var out struct { + Jobs []JobWire `json:"jobs"` + } + if err := json.Unmarshal(raw, &out); err != nil { + return nil, fmt.Errorf("hub: decoding jobs: %w", err) + } + return out.Jobs, nil +} + +// CompleteJob clears a processed job from the host's queue (slice 10B): DELETE +// /hosts/{id}/jobs/{job_id}, self-scoped. Called after a job is executed OR permanently rejected +// (the nonce is already durably burned on a passing verify, so re-processing is replay-safe). +func (c *Client) CompleteJob(ctx context.Context, jobID string) error { + if c.hostID == "" || jobID == "" { + return fmt.Errorf("hub: CompleteJob requires host_id + job_id") + } + req, err := http.NewRequestWithContext(ctx, http.MethodDelete, c.baseURL+"/api/v1/hosts/"+c.hostID+"/jobs/"+jobID, nil) + if err != nil { + return err + } + req.Header.Set("Authorization", "Bearer "+c.apiKey) + resp, err := c.hc.Do(req) + if err != nil { + return &TransportError{Err: err} + } + defer resp.Body.Close() + raw, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10)) + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + return &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)} + } + return nil +} + func tail(b []byte, max int) string { s := strings.TrimSpace(string(b)) if len(s) > max { diff --git a/internal/hub/loop.go b/internal/hub/loop.go index 6a234bc..a3a0034 100644 --- a/internal/hub/loop.go +++ b/internal/hub/loop.go @@ -29,6 +29,22 @@ type EnvelopeObserver interface { OnEnvelope(ctx context.Context, env *ControlEnvelope) } +// MultiObserver fans one envelope out to several observers in order (e.g. the desired-state +// syncer + the signed-jobs runner). nil entries are skipped. +func MultiObserver(observers ...EnvelopeObserver) EnvelopeObserver { + return multiObserver(observers) +} + +type multiObserver []EnvelopeObserver + +func (m multiObserver) OnEnvelope(ctx context.Context, env *ControlEnvelope) { + for _, o := range m { + if o != nil { + o.OnEnvelope(ctx, env) + } + } +} + // Loop is the agent's first daemon run loop: collect a host-report, POST it, adopt // the hub's cadence, repeat. It is resilient — a collect or report error is logged // and the loop continues (the data plane is independent of the agent; a hub outage diff --git a/internal/localapi/disks.go b/internal/localapi/disks.go index dad9b69..db80ec0 100644 --- a/internal/localapi/disks.go +++ b/internal/localapi/disks.go @@ -158,13 +158,28 @@ type formatRequest struct { // inspects the device itself (8C invariant). } -// FormatResponse is POST /disks/format. +// FormatResponse is POST /disks/format. On a data-bearing refusal (slice 10B) it SURFACES the +// bound op the operator must sign: the op class + the DURABLE device id (not the mutable path) + +// the fstype — so the operator can `felhom-opsign -op storage_wipe -durable-id <…>` offline, the +// hub queues it, and the agent's signed-jobs runner verifies + executes the wipe (re-resolving the +// durable id). This is the "records/reports the bound op intent so the operator sees what to sign". type FormatResponse struct { VMID int `json:"vmid"` Device string `json:"device"` Formatted bool `json:"formatted"` DataBearing bool `json:"data_bearing"` Reason string `json:"reason"` + // PendingOp is set on a data-bearing refusal — the exact op to sign (slice 10B). + PendingOp *PendingOp `json:"pending_op,omitempty"` +} + +// PendingOp is the bound destructive intent the operator must sign offline (slice 10B). Params bind +// to the DURABLE device id so the signed authorization can't be retargeted to another disk. +type PendingOp struct { + Op string `json:"op"` // e.g. "storage_wipe" + HostScope string `json:"host_scope"` // the agent's host id (anti-retarget target) + DurableID string `json:"durable_id"` // byid:…|byuuid:… — the device's stable identity + FSType string `json:"fstype"` // the filesystem to mkfs after the wipe } // handleDiskFormat is the security centerpiece. The agent INSPECTS the device; if it is @@ -198,18 +213,33 @@ func (s *Server) handleDiskFormat(w http.ResponseWriter, r *http.Request, vmid i // inspect error → fail-safe data-bearing (probe.DataBearing() is true on !Probed) } if probe.DataBearing() { - // Destructive: route through the gate. With no operator signature (8C) → pending_signature. + // Destructive: route through the gate. With no operator signature → pending_signature. allowed, reason := s.diskGate.AuthorizeWipe(req.Device) - s.logger.Warn("local-api: refusing format of a data-bearing device", - "vmid", vmid, "device", req.Device, "why", probe.Reason(), "gate", reason) if !allowed { + // Surface the bound op the operator must sign (slice 10B): derive the DURABLE device id + // so the signed wipe binds to this exact physical disk (not the mutable path), and the + // runner can re-resolve it at execution. A durable-id derivation failure is non-fatal — + // the refusal still stands; we just can't pre-fill the durable id. + var pending *PendingOp + if durableID, derr := storage.DeviceDurableID(req.Device); derr == nil { + pending = &PendingOp{Op: "storage_wipe", HostScope: s.hostID, DurableID: durableID, FSType: req.FSType} + s.logger.Warn("local-api: data-bearing format refused — PENDING OPERATOR SIGNATURE", + "vmid", vmid, "device", req.Device, "durable_id", durableID, "fstype", req.FSType, + "why", probe.Reason(), "to_authorize", "felhom-opsign -op storage_wipe -host "+s.hostID+" -durable-id "+durableID) + } else { + s.logger.Warn("local-api: data-bearing format refused (no durable id)", + "vmid", vmid, "device", req.Device, "why", probe.Reason(), "derive_err", derr) + } writeStatus(w, http.StatusForbidden, false, - FormatResponse{VMID: vmid, Device: req.Device, Formatted: false, DataBearing: true, Reason: probe.Reason()}, - "device is data-bearing — format requires operator authorization ("+reason+")") + FormatResponse{VMID: vmid, Device: req.Device, Formatted: false, DataBearing: true, Reason: probe.Reason(), PendingOp: pending}, + "device is data-bearing — format requires an operator signature ("+reason+")") return } - // A signed completion would land here in slice 10; 8C never reaches it (gate refuses unsigned). - writeErr(w, http.StatusForbidden, "data-bearing format is not supported in this slice") + // A signed wipe is executed by the signed-jobs runner (queue → verify gate → durable + // re-resolve + re-inspect → mkfs), NOT this synchronous path. This branch (gate ALLOWED a + // data-bearing format inline) is unreachable: the inline path passes signed=nil → always + // pending. Fail safe. + writeErr(w, http.StatusForbidden, "data-bearing format must be completed via a signed job (felhom-opsign → hub queue)") return } diff --git a/internal/localapi/disks_test.go b/internal/localapi/disks_test.go index 4e9cad7..f5a0ceb 100644 --- a/internal/localapi/disks_test.go +++ b/internal/localapi/disks_test.go @@ -129,8 +129,8 @@ func TestFormat_DataBearingDevice_RefusedNoMkfs(t *testing.T) { if len(g.calls) != 1 || g.calls[0] != "/dev/sdb" { t.Fatalf("gate not consulted for the destructive format: %v", g.calls) } - if !strings.Contains(w.Body.String(), "operator authorization") { - t.Fatalf("response did not signal operator-authorization needed: %s", w.Body.String()) + if !strings.Contains(w.Body.String(), "operator signature") { + t.Fatalf("response did not signal an operator signature is needed: %s", w.Body.String()) } } diff --git a/internal/localapi/server.go b/internal/localapi/server.go index 50d08d1..f29a5e7 100644 --- a/internal/localapi/server.go +++ b/internal/localapi/server.go @@ -84,7 +84,10 @@ type Options struct { // cpu-temp) + per-storage capacity, host-wide and token-authed (one-customer-per-host). When // nil the endpoint reports "not configured" (host still reports/reconciles). HostMetrics HostMetricsProvider - Logger *slog.Logger + // HostID is this agent's host id — surfaced in a data-bearing-format pending-op so the operator + // signs an op bound to THIS host (slice 10B anti-retarget). Optional (only used for the hint). + HostID string + Logger *slog.Logger } // defaultBackupCadence is the fallback /backup/due window when none is configured. @@ -129,6 +132,7 @@ type Server struct { guestList GuestLister // slice 8C (optional) hostMetrics HostMetricsProvider // slice 9 (optional) + hostID string // slice 10B: for the data-bearing-format pending-op hint jobsMu sync.Mutex jobs map[int]*backupJob // per-guest backup job state (slice 8B) @@ -166,6 +170,7 @@ func NewServer(o Options) (*Server, error) { diskGate: o.DiskGate, guestList: o.Guests2, hostMetrics: o.HostMetrics, + hostID: o.HostID, jobs: map[int]*backupJob{}, }, nil } diff --git a/internal/signedjobs/mint_test.go b/internal/signedjobs/mint_test.go new file mode 100644 index 0000000..4f2d9d0 --- /dev/null +++ b/internal/signedjobs/mint_test.go @@ -0,0 +1,115 @@ +package signedjobs + +import ( + "crypto/ed25519" + "crypto/rand" + "crypto/sha256" + "encoding/base64" + "encoding/json" + "encoding/pem" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/authz" + "gitea.dooplex.hu/admin/felhom-agent/internal/hub" + "golang.org/x/crypto/ssh" +) + +// In-test SSHSIG minter (mirrors internal/reconcile/mint_test.go's framing) so the runner tests +// exercise the REAL authz.Verifier + reconcile.Gate over genuinely-signed blobs — a positive case +// verifying proves the framing, and the adversarial cases (non-pinned/replay/expired/retarget/ +// forged) exercise the real rejection path end-to-end. Production authz stays verify-only. + +const sshsigMagic = "SSHSIG" + +type sshsigBlob struct { + Version uint32 + PublicKey string + Namespace string + Reserved string + HashAlgo string + Signature string +} + +func signedDataForTest(ns string, msg []byte) []byte { + h := sha256Sum(msg) + body := ssh.Marshal(struct { + Namespace string + Reserved string + HashAlgo string + Hash []byte + }{ns, "", "sha256", h}) + return append([]byte(sshsigMagic), body...) +} + +func mintArmor(pubMarshaled []byte, namespace string, message []byte, sign func([]byte) ssh.Signature) []byte { + sb := &sshsigBlob{Version: 1, PublicKey: string(pubMarshaled), Namespace: namespace, Reserved: "", HashAlgo: "sha256"} + sig := sign(signedDataForTest(namespace, message)) + sb.Signature = string(ssh.Marshal(&sig)) + raw := append([]byte(sshsigMagic), ssh.Marshal(sb)...) + return pem.EncodeToMemory(&pem.Block{Type: "SSH SIGNATURE", Bytes: raw}) +} + +type testSigner struct { + pub ssh.PublicKey + line string + sign func([]byte) ssh.Signature +} + +func newTestSigner(t *testing.T) testSigner { + t.Helper() + pub, priv, err := ed25519.GenerateKey(rand.Reader) + if err != nil { + t.Fatal(err) + } + sshPub, err := ssh.NewPublicKey(pub) + if err != nil { + t.Fatal(err) + } + return testSigner{ + pub: sshPub, + line: string(ssh.MarshalAuthorizedKey(sshPub)), + sign: func(d []byte) ssh.Signature { + return ssh.Signature{Format: ssh.KeyAlgoED25519, Blob: ed25519.Sign(priv, d)} + }, + } +} + +func (s testSigner) allowed(t *testing.T, keyID string, role authz.KeyRole) authz.AllowedSigner { + t.Helper() + as, err := authz.NewAllowedSigner(keyID, role, s.line) + if err != nil { + t.Fatalf("NewAllowedSigner: %v", err) + } + return as +} + +// mintJob builds a hub.JobWire carrying a signed storage_wipe envelope from the given signer. +func mintJob(t *testing.T, s testSigner, jobID, host, guest, keyID, paramsJSON string, issued, expires time.Time) hub.JobWire { + t.Helper() + blob, err := authz.CanonicalBlob("storage_wipe", host, guest, keyID, randNonce(), paramsJSON, issued, expires) + if err != nil { + t.Fatalf("CanonicalBlob: %v", err) + } + sig := mintArmor(s.pub.Marshal(), authz.Namespace, blob, s.sign) + env := Envelope{OpBlobB64: base64.StdEncoding.EncodeToString(blob), SigArmored: string(sig)} + envJSON, _ := json.Marshal(env) + return hub.JobWire{JobID: jobID, BlobB64: base64.StdEncoding.EncodeToString(envJSON)} +} + +func randNonce() string { + var b [16]byte + rand.Read(b[:]) + const hexd = "0123456789abcdef" + out := make([]byte, 32) + for i, x := range b { + out[i*2] = hexd[x>>4] + out[i*2+1] = hexd[x&0x0f] + } + return string(out) +} + +func sha256Sum(b []byte) []byte { + h := sha256.Sum256(b) + return h[:] +} diff --git a/internal/signedjobs/runner.go b/internal/signedjobs/runner.go new file mode 100644 index 0000000..05c7cad --- /dev/null +++ b/internal/signedjobs/runner.go @@ -0,0 +1,216 @@ +// Package signedjobs is the slice-10B consumer of the hub's signed-jobs queue: it fetches each +// opaque queued blob, runs it through the reversibility gate (the LOCKED authz pipeline: pinned-key +// SSHSIG → namespace → allow-list-by-key-material → crypto → host target → time window → durable +// nonce-burn), and ONLY on all-pass hands the verified+bound op to an Executor. The order is +// verify → burn nonce (durable, inside Verify) → execute → report-completion, so no destructive op +// runs on an unsigned, non-pinned-signer, replayed, expired, retargeted, or path-only blob, and an +// interrupted op can't be re-authorized by replaying the same blob after a crash. +package signedjobs + +import ( + "context" + "encoding/base64" + "encoding/json" + "fmt" + "log/slog" + "sync" + + "gitea.dooplex.hu/admin/felhom-agent/internal/authz" + "gitea.dooplex.hu/admin/felhom-agent/internal/hub" + "gitea.dooplex.hu/admin/felhom-agent/internal/reconcile" +) + +// Envelope is the queued signed-op job payload (the opaque bytes the hub stores per job). The +// operator's signing CLI (cmd/felhom-opsign) produces it; this runner consumes it. The hub never +// forges one (it holds no signing key) and never opens it. +type Envelope struct { + OpBlobB64 string `json:"op_blob_b64"` // base64 of the canonical OpBlob JSON + SigArmored string `json:"sig_armored"` // armored SSHSIG over the op-blob (namespace felhom-op-v1) +} + +// JobSource fetches + clears jobs. Satisfied by *hub.Client. +type JobSource interface { + Jobs(ctx context.Context) ([]hub.JobWire, error) + CompleteJob(ctx context.Context, jobID string) error +} + +// Authorizer is the reversibility gate. Satisfied by *reconcile.Gate. +type Authorizer interface { + Authorize(intent reconcile.Intent, signed *reconcile.SignedOp) reconcile.Decision +} + +// Executor runs a verified+bound destructive op. 10B wires the storage-wipe executor; other op +// classes (guest_destroy, decommission, restore_overwrite — 10D) plug in per-slice. The executor +// owns the RESOURCE-level anti-retarget (durable-id resolution + re-inspection at execution). +type Executor interface { + // Execute runs the op named by `op` with the VERIFIED canonical params. ErrNoExecutor means + // "this op has no executor in this build" (left in the queue for a later slice). + Execute(ctx context.Context, op string, params json.RawMessage) error +} + +// ErrNoExecutor signals an op class with no executor wired in this build (don't clear the job). +var ErrNoExecutor = fmt.Errorf("signedjobs: no executor for this op class in this build") + +// Runner polls the jobs queue and drives verify→execute for each job. Single-flight: a RunOnce +// already in progress short-circuits a concurrent trigger. +type Runner struct { + source JobSource + gate Authorizer + exec Executor + hostID string + logger *slog.Logger + + mu sync.Mutex + running bool +} + +// NewRunner builds a runner. All deps are required. +func NewRunner(source JobSource, gate Authorizer, exec Executor, hostID string, logger *slog.Logger) *Runner { + if logger == nil { + logger = slog.Default() + } + return &Runner{source: source, gate: gate, exec: exec, hostID: hostID, logger: logger} +} + +// OnEnvelope implements hub.EnvelopeObserver: when the heartbeat flags pending signed ops, run a +// pass (off the heartbeat path so a long mkfs never blocks the loop). Single-flight. +func (r *Runner) OnEnvelope(ctx context.Context, env *hub.ControlEnvelope) { + if env == nil || !env.HasSignedOps { + return + } + go func() { + if _, err := r.RunOnce(context.Background()); err != nil { + r.logger.Warn("signedjobs: pass failed", "err", err) + } + }() +} + +// RunOnce fetches the pending jobs and processes each. Returns the number of jobs PROCESSED +// (executed or terminally rejected/cleared). A fetch error returns early; per-job errors are +// logged and do not abort the pass. +func (r *Runner) RunOnce(ctx context.Context) (int, error) { + r.mu.Lock() + if r.running { + r.mu.Unlock() + return 0, nil // a pass is already in progress + } + r.running = true + r.mu.Unlock() + defer func() { r.mu.Lock(); r.running = false; r.mu.Unlock() }() + + jobs, err := r.source.Jobs(ctx) + if err != nil { + return 0, fmt.Errorf("signedjobs: fetch jobs: %w", err) + } + processed := 0 + for _, j := range jobs { + if r.processJob(ctx, j) { + processed++ + } + } + return processed, nil +} + +// processJob handles one job; returns true if it was terminally handled (cleared from the queue). +func (r *Runner) processJob(ctx context.Context, j hub.JobWire) bool { + signed, ob, ok := r.decode(j) + if !ok { + // A malformed/forged envelope is permanently bad — clear it loudly (a queued garbage job + // must not wedge the queue). No nonce was consumed (we never reached Verify). + r.logger.Error("signedjobs: malformed job envelope — clearing", "job", j.JobID) + r.complete(ctx, j.JobID) + return true + } + + intent := reconcile.Intent{ + Class: reconcile.OpClass(ob.Op), + HostID: ob.Target.HostID, + GuestID: ob.Target.GuestID, + ParamsJSON: ob.Params, + Source: reconcile.SourceOneShotJob, + // Provenance is the zero value (no agent-internal evidence) — a hub-sourced op is + // classified by its op class alone; a destructive class MUST carry a valid signature. + } + dec := r.gate.Authorize(intent, signed) + if !dec.Allowed { + // The signature/binding failed (forged, replayed, expired, retargeted, role-denied, …). + // The executor is NEVER called. This blob can never become valid, so clear it — but LOG + // it as a security event (a compromised hub queuing a forged op lands here). + r.logger.Warn("signedjobs: REJECTED signed op — executor not called", + "job", j.JobID, "op", ob.Op, "reason", dec.Reason, "err", errStr(dec.Err)) + r.complete(ctx, j.JobID) + return true + } + + // Allowed: the nonce is already durably burned (Verify, before this point). Execute. + r.logger.Warn("signedjobs: AUTHORIZED signed op — executing", + "job", j.JobID, "op", ob.Op, "key_id", dec.Verified.KeyID, "nonce", dec.Verified.Nonce) + err := r.exec.Execute(ctx, ob.Op, ob.Params) + switch { + case err == nil: + r.logger.Warn("signedjobs: signed op COMPLETED", "job", j.JobID, "op", ob.Op) + r.complete(ctx, j.JobID) + return true + case errorsIs(err, ErrNoExecutor): + // No executor for this op class in this build — leave it queued for the owning slice. + r.logger.Info("signedjobs: no executor for op (left queued)", "job", j.JobID, "op", ob.Op) + return false + default: + // Execution failed AFTER a passing verify — the nonce is spent, so re-running the same + // blob would be rejected as a replay anyway. Clear it (the operator re-signs with a fresh + // nonce to retry) and log the failure loudly. + r.logger.Error("signedjobs: signed op execution FAILED (nonce spent — clearing)", + "job", j.JobID, "op", ob.Op, "err", err) + r.complete(ctx, j.JobID) + return true + } +} + +// decode unpacks a JobWire → SignedOp{op-blob, armored sig} + the parsed OpBlob (for routing). +func (r *Runner) decode(j hub.JobWire) (*reconcile.SignedOp, authz.OpBlob, bool) { + raw, err := base64.StdEncoding.DecodeString(j.BlobB64) + if err != nil { + return nil, authz.OpBlob{}, false + } + var env Envelope + if err := json.Unmarshal(raw, &env); err != nil || env.OpBlobB64 == "" || env.SigArmored == "" { + return nil, authz.OpBlob{}, false + } + opBlob, err := base64.StdEncoding.DecodeString(env.OpBlobB64) + if err != nil { + return nil, authz.OpBlob{}, false + } + var ob authz.OpBlob + if err := json.Unmarshal(opBlob, &ob); err != nil || ob.Op == "" { + return nil, authz.OpBlob{}, false + } + return &reconcile.SignedOp{Blob: opBlob, Sig: []byte(env.SigArmored)}, ob, true +} + +func (r *Runner) complete(ctx context.Context, jobID string) { + if err := r.source.CompleteJob(ctx, jobID); err != nil { + r.logger.Warn("signedjobs: clearing job failed (will retry next pass)", "job", jobID, "err", err) + } +} + +func errStr(e error) string { + if e == nil { + return "" + } + return e.Error() +} + +func errorsIs(err, target error) bool { + for err != nil { + if err == target { + return true + } + type w interface{ Unwrap() error } + if u, ok := err.(w); ok { + err = u.Unwrap() + } else { + return false + } + } + return false +} diff --git a/internal/signedjobs/runner_test.go b/internal/signedjobs/runner_test.go new file mode 100644 index 0000000..e2e8e3e --- /dev/null +++ b/internal/signedjobs/runner_test.go @@ -0,0 +1,240 @@ +package signedjobs + +import ( + "context" + "encoding/base64" + "encoding/json" + "io" + "log/slog" + "sync" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/authz" + "gitea.dooplex.hu/admin/felhom-agent/internal/hub" + "gitea.dooplex.hu/admin/felhom-agent/internal/reconcile" +) + +func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) } + +const testHost = "demo-felhom-01" + +// jobsQueue is an in-memory JobSource for the runner tests. +type jobsQueue struct { + mu sync.Mutex + jobs []hub.JobWire + completed []string +} + +func (q *jobsQueue) add(j hub.JobWire) { q.mu.Lock(); q.jobs = append(q.jobs, j); q.mu.Unlock() } +func (q *jobsQueue) Jobs(context.Context) ([]hub.JobWire, error) { + q.mu.Lock() + defer q.mu.Unlock() + out := append([]hub.JobWire(nil), q.jobs...) + return out, nil +} +func (q *jobsQueue) CompleteJob(_ context.Context, jobID string) error { + q.mu.Lock() + defer q.mu.Unlock() + q.completed = append(q.completed, jobID) + // also remove from the pending list so a re-run doesn't re-process it + kept := q.jobs[:0] + for _, j := range q.jobs { + if j.JobID != jobID { + kept = append(kept, j) + } + } + q.jobs = kept + return nil +} +func (q *jobsQueue) wasCompleted(jobID string) bool { + q.mu.Lock() + defer q.mu.Unlock() + for _, id := range q.completed { + if id == jobID { + return true + } + } + return false +} + +// corruptSig flips bytes in the envelope's armored signature (a hub forging/altering a blob). +func corruptSig(t *testing.T, j hub.JobWire) hub.JobWire { + t.Helper() + raw, _ := base64.StdEncoding.DecodeString(j.BlobB64) + var env Envelope + json.Unmarshal(raw, &env) + // Replace the armored signature with a structurally-valid-but-wrong one: re-arm random bytes. + env.SigArmored = env.SigArmored[:len(env.SigArmored)/2] + "AAAA" + env.SigArmored[len(env.SigArmored)/2:] + out, _ := json.Marshal(env) + j.BlobB64 = base64.StdEncoding.EncodeToString(out) + return j +} + +// fakeExecutor records Execute calls and returns a configurable error. +type fakeExecutor struct { + mu sync.Mutex + calls []string // ops executed + err error +} + +func (f *fakeExecutor) Execute(_ context.Context, op string, _ json.RawMessage) error { + f.mu.Lock() + f.calls = append(f.calls, op) + f.mu.Unlock() + return f.err +} +func (f *fakeExecutor) count() int { f.mu.Lock(); defer f.mu.Unlock(); return len(f.calls) } + +// newRealGateRunner builds a runner with the REAL gate+verifier pinned to `signer`, an in-memory +// nonce store, plus a fake jobs source + fake executor. wipeParams is the params to sign. +func newRealGateRunner(t *testing.T, signer testSigner) (*Runner, *jobsQueue, *fakeExecutor) { + t.Helper() + store := authz.NewMemoryNonceStore() + verifier := authz.New([]authz.AllowedSigner{signer.allowed(t, "ops-1", authz.RoleOperational)}, store, testHost) + gate := reconcile.NewGate(verifier, testHost, nil, quiet()) + src := &jobsQueue{} + exec := &fakeExecutor{} + return NewRunner(src, gate, exec, testHost, quiet()), src, exec +} + +const wipeParamsJSON = `{"durable_id":"byid:wwn-0xtest","fstype":"ext4"}` + +// VALID: a correctly-signed wipe → executor runs once + the job is cleared. +func TestRunner_ValidSignedWipeExecutes(t *testing.T) { + s := newTestSigner(t) + r, src, exec := newRealGateRunner(t, s) + now := time.Now().UTC() + src.add(mintJob(t, s, "j1", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour))) + + n, err := r.RunOnce(context.Background()) + if err != nil { + t.Fatalf("RunOnce: %v", err) + } + if n != 1 || exec.count() != 1 || exec.calls[0] != "storage_wipe" { + t.Fatalf("executor calls = %v (n=%d), want one storage_wipe", exec.calls, n) + } + if !src.wasCompleted("j1") { + t.Error("valid job was not cleared after execution") + } +} + +// REPLAY: the same blob resubmitted (a second job) → rejected, executor NOT called again. +func TestRunner_ReplayRejected(t *testing.T) { + s := newTestSigner(t) + r, src, exec := newRealGateRunner(t, s) + now := time.Now().UTC() + job := mintJob(t, s, "j1", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)) + src.add(job) + if _, err := r.RunOnce(context.Background()); err != nil { + t.Fatal(err) + } + if exec.count() != 1 { + t.Fatalf("first run executed %d times, want 1", exec.count()) + } + // Resubmit the IDENTICAL signed blob (same nonce) as a new job. + replay := job + replay.JobID = "j1-replay" + src.add(replay) + if _, err := r.RunOnce(context.Background()); err != nil { + t.Fatal(err) + } + if exec.count() != 1 { + t.Errorf("replay caused a second execution (count=%d) — nonce-burn failed", exec.count()) + } + if !src.wasCompleted("j1-replay") { + t.Error("replayed job should be cleared (rejected)") + } +} + +// NON-PINNED signer → rejected, executor NOT called. +func TestRunner_NonPinnedSignerRejected(t *testing.T) { + pinned := newTestSigner(t) + attacker := newTestSigner(t) // a DIFFERENT key, not pinned + r, src, exec := newRealGateRunner(t, pinned) + now := time.Now().UTC() + src.add(mintJob(t, attacker, "j1", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour))) + + r.RunOnce(context.Background()) + if exec.count() != 0 { + t.Errorf("a non-pinned signature was EXECUTED (count=%d) — must be rejected", exec.count()) + } + if !src.wasCompleted("j1") { + t.Error("rejected job should be cleared") + } +} + +// EXPIRED → rejected, executor NOT called. +func TestRunner_ExpiredRejected(t *testing.T) { + s := newTestSigner(t) + r, src, exec := newRealGateRunner(t, s) + past := time.Now().UTC().Add(-2 * time.Hour) + src.add(mintJob(t, s, "j1", testHost, "", "ops-1", wipeParamsJSON, past, past.Add(time.Hour))) // expired an hour ago + + r.RunOnce(context.Background()) + if exec.count() != 0 { + t.Errorf("an EXPIRED op was executed (count=%d)", exec.count()) + } +} + +// RETARGETED (Target.HostID = another host) → rejected, executor NOT called. +func TestRunner_RetargetRejected(t *testing.T) { + s := newTestSigner(t) + r, src, exec := newRealGateRunner(t, s) + now := time.Now().UTC() + src.add(mintJob(t, s, "j1", "some-other-host", "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour))) + + r.RunOnce(context.Background()) + if exec.count() != 0 { + t.Errorf("an op targeting another host was executed on this host (count=%d) — anti-retarget failed", exec.count()) + } +} + +// FORGED (a compromised hub queues a blob with a garbage signature) → rejected, executor NOT called. +func TestRunner_ForgedSignatureRejected(t *testing.T) { + s := newTestSigner(t) + r, src, exec := newRealGateRunner(t, s) + now := time.Now().UTC() + job := mintJob(t, s, "j1", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour)) + // Corrupt the envelope's signature (simulate a hub forging/altering the blob). + job = corruptSig(t, job) + src.add(job) + + r.RunOnce(context.Background()) + if exec.count() != 0 { + t.Errorf("a forged/altered signature was executed (count=%d) — the compromised-hub case must be rejected", exec.count()) + } + if !src.wasCompleted("j1") { + t.Error("forged job should be cleared") + } +} + +// No verifier pinned (no signers) → every signed op is refused pending_signature, executor not called. +func TestRunner_NoSignersAllRejected(t *testing.T) { + s := newTestSigner(t) + gate := reconcile.NewGate(nil, testHost, nil, quiet()) // nil verifier + src := &jobsQueue{} + exec := &fakeExecutor{} + r := NewRunner(src, gate, exec, testHost, quiet()) + now := time.Now().UTC() + src.add(mintJob(t, s, "j1", testHost, "", "ops-1", wipeParamsJSON, now, now.Add(time.Hour))) + + r.RunOnce(context.Background()) + if exec.count() != 0 { + t.Errorf("with no pinned signer a signed op executed (count=%d) — must be pending_signature", exec.count()) + } +} + +// A malformed envelope → cleared without ever calling the executor. +func TestRunner_MalformedEnvelopeCleared(t *testing.T) { + s := newTestSigner(t) + r, src, exec := newRealGateRunner(t, s) + src.add(hub.JobWire{JobID: "bad", BlobB64: "!!!not base64!!!"}) + r.RunOnce(context.Background()) + if exec.count() != 0 { + t.Errorf("a malformed envelope reached the executor (count=%d)", exec.count()) + } + if !src.wasCompleted("bad") { + t.Error("malformed job should be cleared") + } +} diff --git a/internal/signedjobs/wipe.go b/internal/signedjobs/wipe.go new file mode 100644 index 0000000..83f7192 --- /dev/null +++ b/internal/signedjobs/wipe.go @@ -0,0 +1,111 @@ +package signedjobs + +import ( + "context" + "encoding/json" + "fmt" + "log/slog" + + "gitea.dooplex.hu/admin/felhom-agent/internal/storage" +) + +// opStorageWipe is the op class this executor serves (mirrors reconcile.ClassStorageWipe; the +// literal avoids importing reconcile here just for the string). +const opStorageWipe = "storage_wipe" + +// WipeOps is the privileged storage surface the wipe executor needs (a subset of storage.HostOps). +type WipeOps interface { + InspectDevice(ctx context.Context, device string) (storage.DeviceProbe, error) + Format(ctx context.Context, device, fstype string) error +} + +// wipeParams is the verified params of a storage_wipe op. The device is named by a DURABLE id +// (never a mutable /dev path), so execution re-resolves it to the exact physical device. +type wipeParams struct { + DurableID string `json:"durable_id"` + FSType string `json:"fstype"` +} + +// WipeExecutor is the slice-10B storage-wipe consumer — it CLOSES the 8C data-bearing-format +// `pending_signature` gap. Given a gate-VERIFIED+bound storage_wipe op, it: +// 1. resolves the op's DURABLE device id → the current /dev path (vanished/replaced → refuse); +// 2. re-derives that device's durable id and requires it to MATCH the signed id (resource-level +// anti-retarget — "wipe device X" wipes exactly X, not whatever is at /dev/sdb now); +// 3. re-inspects (8C classifier) to confirm it is STILL the data-bearing target; +// 4. mkfs. +// +// A path-only binding, a device that vanished/changed, or a target that is no longer data-bearing +// is refused EVEN WITH a valid signature. The gate has already burned the nonce durably before we +// run, so an interrupted wipe can't be re-authorized by replaying the blob. +type WipeExecutor struct { + ops WipeOps + resolve func(durableID string) (string, error) + derive func(device string) (string, error) + logger *slog.Logger +} + +// NewWipeExecutor wires the production durable resolver/deriver (storage package). +func NewWipeExecutor(ops WipeOps, logger *slog.Logger) *WipeExecutor { + if logger == nil { + logger = slog.Default() + } + return &WipeExecutor{ + ops: ops, + resolve: storage.ResolveDurableDevice, + derive: storage.DeviceDurableID, + logger: logger, + } +} + +// Execute implements signedjobs.Executor for the storage_wipe op class. +func (w *WipeExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error { + if op != opStorageWipe { + return ErrNoExecutor // not ours — the runner leaves it queued for the owning slice + } + var p wipeParams + if err := json.Unmarshal(params, &p); err != nil { + return fmt.Errorf("wipe: bad params: %w", err) + } + if p.DurableID == "" { + // The whole point of the durable binding: a path-only op is refused (anti-retarget). + return fmt.Errorf("wipe: op has no durable_id — refusing a path-only wipe binding") + } + + // 1. Resolve the durable id → current device (gone/replaced → refuse). + device, err := w.resolve(p.DurableID) + if err != nil { + return fmt.Errorf("wipe: durable id %q no longer resolves (device removed/replaced?) — refusing: %w", p.DurableID, err) + } + // 2. Anti-retarget: re-derive the resolved device's durable id; it MUST equal the signed id. + got, err := w.derive(device) + if err != nil { + return fmt.Errorf("wipe: cannot re-derive durable id for %s — refusing: %w", device, err) + } + if got != p.DurableID { + return fmt.Errorf("wipe: durable-id mismatch — %s now has id %q, signed id was %q — refusing", device, got, p.DurableID) + } + // 3. Re-inspect (8C classifier) — confirm the resolved device is STILL the data-bearing target. + probe, err := w.ops.InspectDevice(ctx, device) + if err != nil { + return fmt.Errorf("wipe: re-inspect %s failed — refusing: %w", device, err) + } + if !probe.Probed { + return fmt.Errorf("wipe: %s did not probe cleanly at execution — refusing", device) + } + if !probe.DataBearing() { + // The signed op authorized wiping a DATA-BEARING device; if it is now blank, the target + // changed since signing — refuse rather than wipe the wrong (or an unexpected) device. + return fmt.Errorf("wipe: %s is no longer data-bearing (target changed since signing) — refusing", device) + } + + // 4. Execute. The nonce was durably burned by the gate's Verify BEFORE this point (crash-safety: + // a replay after an interrupted wipe is rejected). This log line is the audit trail of a + // destructive, operator-authorized wipe actually running. + w.logger.Warn("wipe: executing operator-signed data-bearing wipe", + "device", device, "durable_id", p.DurableID, "fstype", p.FSType, "data_reason", probe.Reason()) + if err := w.ops.Format(ctx, device, p.FSType); err != nil { + return fmt.Errorf("wipe: mkfs %s %s: %w", p.FSType, device, err) + } + w.logger.Warn("wipe: operator-signed wipe complete", "device", device, "durable_id", p.DurableID) + return nil +} diff --git a/internal/signedjobs/wipe_test.go b/internal/signedjobs/wipe_test.go new file mode 100644 index 0000000..8c2251d --- /dev/null +++ b/internal/signedjobs/wipe_test.go @@ -0,0 +1,137 @@ +package signedjobs + +import ( + "context" + "encoding/json" + "errors" + "testing" + + "gitea.dooplex.hu/admin/felhom-agent/internal/storage" +) + +// fakeWipeOps records Format calls and returns a configurable InspectDevice probe. +type fakeWipeOps struct { + probe storage.DeviceProbe + inspectErr error + formatCalls []string +} + +func (f *fakeWipeOps) InspectDevice(_ context.Context, device string) (storage.DeviceProbe, error) { + p := f.probe + p.Device = device + return p, f.inspectErr +} +func (f *fakeWipeOps) Format(_ context.Context, device, _ string) error { + f.formatCalls = append(f.formatCalls, device) + return nil +} + +// newWipeExec builds a WipeExecutor with injected resolve/derive (so durable resolution is +// deterministic in a unit test, no real /dev needed). +func newWipeExec(ops WipeOps, resolve func(string) (string, error), derive func(string) (string, error)) *WipeExecutor { + w := NewWipeExecutor(ops, quiet()) + w.resolve = resolve + w.derive = derive + return w +} + +const durable = "byid:wwn-0xtest" + +func params(t *testing.T, durableID, fstype string) json.RawMessage { + t.Helper() + b, _ := json.Marshal(map[string]string{"durable_id": durableID, "fstype": fstype}) + return b +} + +// VALID: durable resolves, re-derives to the same id, device is data-bearing → Format runs. +func TestWipe_ValidExecutes(t *testing.T) { + ops := &fakeWipeOps{probe: storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4"}} + w := newWipeExec(ops, + func(id string) (string, error) { return "/dev/sdb", nil }, + func(dev string) (string, error) { return durable, nil }) + if err := w.Execute(context.Background(), "storage_wipe", params(t, durable, "ext4")); err != nil { + t.Fatalf("valid wipe: %v", err) + } + if len(ops.formatCalls) != 1 || ops.formatCalls[0] != "/dev/sdb" { + t.Fatalf("Format calls = %v, want one on /dev/sdb", ops.formatCalls) + } +} + +// PATH-ONLY (no durable_id) → refused, Format NOT called. +func TestWipe_PathOnlyRefused(t *testing.T) { + ops := &fakeWipeOps{probe: storage.DeviceProbe{Probed: true, HasFilesystem: true}} + w := newWipeExec(ops, func(string) (string, error) { return "/dev/sdb", nil }, func(string) (string, error) { return durable, nil }) + // params carry a raw device path, NOT a durable id. + p, _ := json.Marshal(map[string]string{"device": "/dev/sdb", "fstype": "ext4"}) + if err := w.Execute(context.Background(), "storage_wipe", p); err == nil { + t.Fatal("path-only wipe must be refused") + } + if len(ops.formatCalls) != 0 { + t.Errorf("Format was called on a path-only binding: %v", ops.formatCalls) + } +} + +// DURABLE MISMATCH: the resolved device re-derives to a DIFFERENT id → refused (anti-retarget). +func TestWipe_DurableMismatchRefused(t *testing.T) { + ops := &fakeWipeOps{probe: storage.DeviceProbe{Probed: true, HasFilesystem: true}} + w := newWipeExec(ops, + func(string) (string, error) { return "/dev/sdb", nil }, + func(string) (string, error) { return "byid:wwn-0xDIFFERENT", nil }) // re-derive disagrees + if err := w.Execute(context.Background(), "storage_wipe", params(t, durable, "ext4")); err == nil { + t.Fatal("a durable-id mismatch must be refused") + } + if len(ops.formatCalls) != 0 { + t.Errorf("Format was called despite a durable-id mismatch: %v", ops.formatCalls) + } +} + +// DEVICE GONE: the durable id no longer resolves → refused (device removed/replaced). +func TestWipe_ResolveFailureRefused(t *testing.T) { + ops := &fakeWipeOps{probe: storage.DeviceProbe{Probed: true, HasFilesystem: true}} + w := newWipeExec(ops, + func(string) (string, error) { return "", errors.New("no such by-id link") }, + func(string) (string, error) { return durable, nil }) + if err := w.Execute(context.Background(), "storage_wipe", params(t, durable, "ext4")); err == nil { + t.Fatal("an unresolvable durable id must be refused") + } + if len(ops.formatCalls) != 0 { + t.Errorf("Format was called on an unresolvable device: %v", ops.formatCalls) + } +} + +// RE-INSPECT: the resolved device is no longer data-bearing (target changed) → refused. +func TestWipe_ReinspectNonDataBearingRefused(t *testing.T) { + ops := &fakeWipeOps{probe: storage.DeviceProbe{Probed: true /* blank: no fs/parts */}} + w := newWipeExec(ops, + func(string) (string, error) { return "/dev/sdb", nil }, + func(string) (string, error) { return durable, nil }) + if err := w.Execute(context.Background(), "storage_wipe", params(t, durable, "ext4")); err == nil { + t.Fatal("a device that is no longer data-bearing must be refused (target changed)") + } + if len(ops.formatCalls) != 0 { + t.Errorf("Format was called on a non-target device: %v", ops.formatCalls) + } +} + +// RE-INSPECT: the device did not probe cleanly → refused (fail-safe). +func TestWipe_NotProbedRefused(t *testing.T) { + ops := &fakeWipeOps{probe: storage.DeviceProbe{Probed: false}} + w := newWipeExec(ops, + func(string) (string, error) { return "/dev/sdb", nil }, + func(string) (string, error) { return durable, nil }) + if err := w.Execute(context.Background(), "storage_wipe", params(t, durable, "ext4")); err == nil { + t.Fatal("a device that did not probe cleanly must be refused") + } + if len(ops.formatCalls) != 0 { + t.Errorf("Format was called on an unprobed device: %v", ops.formatCalls) + } +} + +// A non-wipe op → ErrNoExecutor (left for the owning slice; Format not called). +func TestWipe_OtherOpIsNoExecutor(t *testing.T) { + ops := &fakeWipeOps{} + w := newWipeExec(ops, func(string) (string, error) { return "/dev/sdb", nil }, func(string) (string, error) { return durable, nil }) + if err := w.Execute(context.Background(), "guest_destroy", params(t, durable, "ext4")); !errors.Is(err, ErrNoExecutor) { + t.Fatalf("guest_destroy err = %v, want ErrNoExecutor", err) + } +} diff --git a/internal/storage/durable_device.go b/internal/storage/durable_device.go new file mode 100644 index 0000000..f5f720e --- /dev/null +++ b/internal/storage/durable_device.go @@ -0,0 +1,128 @@ +package storage + +import ( + "fmt" + "os" + "path/filepath" + "sort" + "strings" +) + +// Durable BLOCK-DEVICE identity for the slice-10B operator-signed wipe (anti-retarget). The +// signed op binds a DURABLE id (a WWN / hardware serial, falling back to a filesystem UUID), and +// execution re-resolves it to the CURRENT /dev path — so "wipe device X" wipes that exact physical +// device, never whatever happens to be at /dev/sdb now. These reads are world-readable udev +// symlinks under /dev/disk/by-id and /dev/disk/by-uuid — NO privilege and NO subprocess (the +// root-CLI fence is untouched). +// +// devDiskRoot is overridable for tests (a fake /dev/disk layout). +var devDiskRoot = "/dev/disk" + +// durable-id scheme prefixes. byid: a /dev/disk/by-id/ entry (preferred — wwn/serial, +// survives reformat + re-cabling); byuuid: a filesystem UUID (fallback — survives re-cabling but +// not a reformat, acceptable for a one-shot wipe resolved immediately before wiping). +const ( + durableByID = "byid:" + durableByUUID = "byuuid:" +) + +// byIDPriority ranks /dev/disk/by-id link prefixes most-stable-first: wwn (hardware world-wide +// name) > nvme-eui (NVMe EUI) > nvme-/ata-/scsi-/usb- (model+serial). dm-/lvm-/md- names are +// excluded (they are mapper constructs, not the physical disk we want to bind a wipe to). +var byIDPriority = []string{"wwn-", "nvme-eui.", "nvme-", "ata-", "scsi-", "usb-"} + +// DeviceDurableID derives a stable durable identifier for a block device by scanning the +// world-readable udev symlinks. Prefers a by-id (wwn/serial) link, then a filesystem UUID. Errors +// when neither is resolvable (a device with no durable identity cannot be safely wipe-bound). +func DeviceDurableID(device string) (string, error) { + target, err := filepath.EvalSymlinks(device) + if err != nil { + return "", fmt.Errorf("storage: resolve %s: %w", device, err) + } + // 1. best by-id link pointing at this device. + if name := bestByIDLink(target); name != "" { + return durableByID + name, nil + } + // 2. fallback: a filesystem UUID symlink. + if uuid := byUUIDFor(target); uuid != "" { + return durableByUUID + uuid, nil + } + return "", fmt.Errorf("storage: no durable id (wwn/serial/uuid) for device %s", device) +} + +// ResolveDurableDevice resolves a durable id back to the CURRENT canonical /dev path, or errors if +// it no longer resolves (the device was physically removed/replaced — the wipe must then refuse). +func ResolveDurableDevice(durableID string) (string, error) { + switch { + case strings.HasPrefix(durableID, durableByID): + name := strings.TrimPrefix(durableID, durableByID) + if !safeLinkName(name) { + return "", fmt.Errorf("storage: unsafe durable id %q", durableID) + } + return filepath.EvalSymlinks(filepath.Join(devDiskRoot, "by-id", name)) + case strings.HasPrefix(durableID, durableByUUID): + uuid := strings.TrimPrefix(durableID, durableByUUID) + if !safeLinkName(uuid) { + return "", fmt.Errorf("storage: unsafe durable id %q", durableID) + } + return filepath.EvalSymlinks(filepath.Join(devDiskRoot, "by-uuid", uuid)) + default: + // A bare path or unknown scheme is REFUSED — a wipe must bind to a durable id, never a + // mutable /dev path (the anti-retarget invariant). + return "", fmt.Errorf("storage: durable id %q is not a durable scheme (byid:/byuuid:) — refusing path-only binding", durableID) + } +} + +// bestByIDLink returns the highest-priority /dev/disk/by-id link name whose target is `device` +// (already symlink-resolved), or "" if none. +func bestByIDLink(device string) string { + dir := filepath.Join(devDiskRoot, "by-id") + entries, err := os.ReadDir(dir) + if err != nil { + return "" + } + // collect matching names, then pick by priority (then lexically for determinism). + var matches []string + for _, e := range entries { + name := e.Name() + if tgt, err := filepath.EvalSymlinks(filepath.Join(dir, name)); err == nil && tgt == device { + matches = append(matches, name) + } + } + if len(matches) == 0 { + return "" + } + sort.Strings(matches) + for _, pfx := range byIDPriority { + for _, name := range matches { + if strings.HasPrefix(name, pfx) { + return name + } + } + } + return matches[0] // some by-id link exists but not a recognized prefix — still durable enough +} + +// byUUIDFor returns the filesystem UUID whose by-uuid link targets `device`, or "". +func byUUIDFor(device string) string { + dir := filepath.Join(devDiskRoot, "by-uuid") + entries, err := os.ReadDir(dir) + if err != nil { + return "" + } + for _, e := range entries { + if tgt, err := filepath.EvalSymlinks(filepath.Join(dir, e.Name())); err == nil && tgt == device { + return e.Name() + } + } + return "" +} + +// safeLinkName rejects a durable-id component that could escape the by-id/by-uuid dir (path +// traversal / separators) — defense even though the value is operator-signed. +func safeLinkName(s string) bool { + if s == "" || strings.ContainsAny(s, "/\x00") || s == "." || s == ".." { + return false + } + return true +} diff --git a/internal/storage/durable_device_test.go b/internal/storage/durable_device_test.go new file mode 100644 index 0000000..0b2b5cc --- /dev/null +++ b/internal/storage/durable_device_test.go @@ -0,0 +1,117 @@ +package storage + +import ( + "os" + "path/filepath" + "runtime" + "testing" +) + +// fakeDevDisk builds a temp /dev/disk tree: a real "device" file + by-id/by-uuid symlinks to it. +// Returns the disk root + the device path. (filepath.EvalSymlinks needs real targets, so the +// "device" is a regular file standing in for a block device.) Skips on Windows where creating a +// symlink needs a privilege — the durable-device code is Linux-only (the agent runs on the PVE +// host), so these run on the build server / demo host. +func fakeDevDisk(t *testing.T, links map[string]string, uuids map[string]string) (root, device string) { + t.Helper() + if runtime.GOOS == "windows" { + t.Skip("durable-device symlink tests run on Linux (the agent's OS); symlink creation needs privilege on Windows") + } + base := t.TempDir() + device = filepath.Join(base, "sdb") + if err := os.WriteFile(device, []byte("x"), 0o600); err != nil { + t.Fatal(err) + } + root = filepath.Join(base, "disk") + mk := func(sub, name, target string) { + dir := filepath.Join(root, sub) + os.MkdirAll(dir, 0o755) + if err := os.Symlink(target, filepath.Join(dir, name)); err != nil { + t.Fatal(err) + } + } + for name := range links { + mk("by-id", name, device) + } + for uuid := range uuids { + mk("by-uuid", uuid, device) + } + return root, device +} + +func withDevDiskRoot(t *testing.T, root string) { + t.Helper() + old := devDiskRoot + devDiskRoot = root + t.Cleanup(func() { devDiskRoot = old }) +} + +// DeviceDurableID prefers a wwn- by-id link over ata-/serial links and over a uuid. +func TestDeviceDurableID_PrefersWWN(t *testing.T) { + root, device := fakeDevDisk(t, + map[string]string{"wwn-0x5000c500abcd": "", "ata-Samsung_SSD_850_S1": "", "scsi-35000c500abcd": ""}, + map[string]string{"1111-2222": ""}) + withDevDiskRoot(t, root) + + id, err := DeviceDurableID(device) + if err != nil { + t.Fatalf("DeviceDurableID: %v", err) + } + if id != "byid:wwn-0x5000c500abcd" { + t.Errorf("durable id = %q, want the wwn link", id) + } +} + +// With no by-id link, it falls back to a filesystem UUID. +func TestDeviceDurableID_FallsBackToUUID(t *testing.T) { + root, device := fakeDevDisk(t, map[string]string{}, map[string]string{"abcd-ef01": ""}) + withDevDiskRoot(t, root) + id, err := DeviceDurableID(device) + if err != nil { + t.Fatalf("DeviceDurableID: %v", err) + } + if id != "byuuid:abcd-ef01" { + t.Errorf("durable id = %q, want byuuid:abcd-ef01", id) + } +} + +// A device with no durable identity at all → error (cannot be wipe-bound). +func TestDeviceDurableID_NoneErrors(t *testing.T) { + root, device := fakeDevDisk(t, map[string]string{}, map[string]string{}) + withDevDiskRoot(t, root) + if _, err := DeviceDurableID(device); err == nil { + t.Fatal("a device with no wwn/serial/uuid must error (no durable id)") + } +} + +// ResolveDurableDevice round-trips a by-id id back to the canonical device path. +func TestResolveDurableDevice_RoundTrip(t *testing.T) { + root, device := fakeDevDisk(t, map[string]string{"wwn-0xabc": ""}, map[string]string{}) + withDevDiskRoot(t, root) + got, err := ResolveDurableDevice("byid:wwn-0xabc") + if err != nil { + t.Fatalf("ResolveDurableDevice: %v", err) + } + want, _ := filepath.EvalSymlinks(device) + if got != want { + t.Errorf("resolved %q, want %q", got, want) + } +} + +// A path-only / unknown-scheme id is REFUSED (the anti-retarget invariant). +func TestResolveDurableDevice_RefusesPathOnly(t *testing.T) { + withDevDiskRoot(t, t.TempDir()) + for _, bad := range []string{"/dev/sdb", "sdb", "uuid-no-scheme", "byid:../../etc/passwd", "byuuid:../x"} { + if _, err := ResolveDurableDevice(bad); err == nil { + t.Errorf("ResolveDurableDevice(%q) succeeded, want refusal", bad) + } + } +} + +// A durable id whose link is gone → error (device removed/replaced). +func TestResolveDurableDevice_MissingErrors(t *testing.T) { + withDevDiskRoot(t, t.TempDir()) // empty: no by-id dir + if _, err := ResolveDurableDevice("byid:wwn-0xgone"); err == nil { + t.Fatal("an absent durable id must error") + } +}