diff --git a/REUSE.md b/REUSE.md index 7a11cde..5a2bff9 100644 --- a/REUSE.md +++ b/REUSE.md @@ -121,7 +121,7 @@ | Anti-retarget durable-id binding | internal/localapi/wipe_reresolve.go | resolve id → re-derive + exact match → re-inspect expected state → act on RE-RESOLVED device only | | Atomic single-file JSON store | internal/storage/intent.go | `Open*` loads (missing=empty, corrupt=fail-loud), mutex, tmp+rename 0600, idempotent set | | Durable append-only log + index | internal/authz/noncestore.go (`FileNonceStore`) | fsync before returning "new"; replay into index on open; expiry-only compaction | -| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did | +| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`; R-856 `crashGuardStatePath` — GET /host/crash-guard's state file, internal/localapi/crashguard.go) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did | | `Server.devicePresent` (R-113, v0.114.0) | internal/localapi/disks.go | `devicePresent(rawMountPath) bool`; seam `deviceCheck`, default `isHostMountpoint` | the agent's DEVICE-presence signal — asks whether the drive's RAW mount is still mounted | **Use this, never the bind, to answer "is the drive there".** The raw mount is a device-bound systemd unit and dies with its device; the agent's own bind under the shared parent is NOT device-bound and outlives it as a stale shell. `BoundUnderParent` is now `boundUnderParent(...) && devicePresent(...)` at BOTH /disks construction sites — dropping either half is a regression with its own red-proof. Empty path ⇒ **true** (unknown is never absent: absent stops a customer's apps) | | `bindLiveness` + `BindLiveness` (R-117, v0.117.0) | internal/localapi/intermediary.go | `bindLiveness(stable, raw) BindLiveness`; seam `livenessCheck`; read verdicts ONLY via `.Usable()` | the agent's bind-LIVENESS signal — the third term of `BoundUnderParent` | **`devicePresent` and `boundUnderParent` are both PATH-PRESENCE tests and neither is liveness.** They compare only mountinfo field 5, so both stay true over a bind that names the drive that went away while the raw mount healed onto the returning one (measured: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb `shutdown`, EIO both ways, payload healthy). Two dead states, and a fix needs BOTH checks: devno mismatch (the detach/return case) AND the ext4 abort tokens `shutdown`/`emergency_ro` (the steady-state case, where the devnos AGREE because the device never left). **THREE states, never a bool** — `BindUnknown` must exist and `Usable()` treats it as PRESENT (absent stops a customer's apps). **Order matters:** compare devices first and read the abort flag off the RAW mount in the stale case — abort-first classifies the real return state as aborted and refuses the re-bind that repairs it. **NO BLOCK I/O, ever** (CLAUDE.md rule; a probe on a wedged device survives SIGKILL). 6 red-proofs | | `AttachDrive` repair ruling (R-117, v0.117.0) | internal/localapi/intermediary.go | the `switch bindLiveness(...)` inside the `n == 1 && GuestSeesMount` arm | decides whether the existing self-heal runs | `BindStaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, no guest restart). `BindAborted` ⇒ **quiet no-op** — a re-bind lands on the SAME dead superblock and this runs every 20 s, so re-binding is an infinite silent retry that also masks the state; it must surface via `BoundUnderParent=false`. `BindLive`/`BindUnknown` ⇒ no-op, unchanged. **Do not return an error for the aborted case** — the reconcile loop would log a failure every 20 s | diff --git a/internal/localapi/crashguard.go b/internal/localapi/crashguard.go new file mode 100644 index 0000000..47df138 --- /dev/null +++ b/internal/localapi/crashguard.go @@ -0,0 +1,102 @@ +package localapi + +import ( + "encoding/json" + "errors" + "io" + "io/fs" + "net/http" + "os" + "time" +) + +// GET /host/crash-guard (R-856, `09` §3 decision 143): what the host's crash guard +// (configs/felhom-crash-guard, `11` §5.9) recorded about the most recent HOST boot. The controller +// reads it once after it starts: when the host's last boot followed an UNCLEAN stop, its app mails +// wait ~15 minutes instead of the normal 90 s boot grace. +// +// Read-only and Proxmox-free: the agent reads the guard's state file (root-owned, 0644 — the +// non-root agent can read it) and passes four fields through. Host-wide, token-authed (any valid +// per-guest token sees the host's view, as GET /host/metrics does). +// +// NEVER an error page. A missing file (no guard installed, or no boot recorded yet), an unreadable +// one, or one that does not parse answers 200 with present:false — the controller reads that as +// UNKNOWN and keeps its normal boot grace. Pinned by TestR856_CrashGuard*. + +// defaultCrashGuardStatePath is where configs/felhom-crash-guard writes its state (STATE_DIR there). +const defaultCrashGuardStatePath = "/var/lib/felhom-crash-guard/state.json" + +// crashGuardStateMax bounds the read; the real file is well under 4 KiB. +const crashGuardStateMax = 1 << 20 + +// CrashGuardResponse is the data block of GET /host/crash-guard. Field names are the controller's +// agentapi.CrashGuardState (felhom-controller internal/agentapi/crashguard.go) — a wire contract, +// pinned by TestR856_CrashGuardWireMatchesControllerClient. +type CrashGuardResponse struct { + Present bool `json:"present"` + LastBootAt string `json:"last_boot_at,omitempty"` // RFC3339 UTC ("2006-01-02T15:04:05Z") + LastBootUnclean bool `json:"last_boot_unclean"` + Tripped bool `json:"tripped"` +} + +// crashGuardFile is the subset of the guard's state.json the route passes through. Every other key +// (armed, boot_id, config, unclean_boots, last_trip, ...) is ignored. +type crashGuardFile struct { + LastBootAt string `json:"last_boot_at"` + LastBootUnclean bool `json:"last_boot_unclean"` + Tripped bool `json:"tripped"` +} + +// readCrashGuardState reads and parses the guard's state file. ok=false on ANY failure (missing, +// unreadable, oversized, not a JSON object, a field of the wrong type); reason says which, for the log. +func readCrashGuardState(path string) (resp CrashGuardResponse, ok bool, reason string) { + f, err := os.Open(path) + if err != nil { + if errors.Is(err, fs.ErrNotExist) { + return resp, false, "no state file" + } + return resp, false, "unreadable: " + err.Error() + } + defer f.Close() + raw, err := io.ReadAll(io.LimitReader(f, crashGuardStateMax+1)) + if err != nil { + return resp, false, "read: " + err.Error() + } + if len(raw) > crashGuardStateMax { + return resp, false, "state file too large" + } + var st crashGuardFile + // Unmarshal into a struct fails on a non-object top level (null decodes, so reject it below). + if err := json.Unmarshal(raw, &st); err != nil { + return resp, false, "unparseable: " + err.Error() + } + var probe map[string]json.RawMessage + if err := json.Unmarshal(raw, &probe); err != nil || probe == nil { + return resp, false, "unparseable: not a JSON object" + } + resp = CrashGuardResponse{Present: true, LastBootUnclean: st.LastBootUnclean, Tripped: st.Tripped} + // Normalise to RFC3339 UTC; an unparseable time passes through as-is (the controller reads an + // unparseable boot time as "not this start's boot" → its normal grace). + if t, perr := time.Parse(time.RFC3339, st.LastBootAt); perr == nil { + resp.LastBootAt = t.UTC().Format(time.RFC3339) + } else { + resp.LastBootAt = st.LastBootAt + } + return resp, true, "" +} + +func (s *Server) handleCrashGuard(w http.ResponseWriter, r *http.Request, vmid int) { + path := s.crashGuardStatePath + if path == "" { + path = defaultCrashGuardStatePath + } + resp, ok, reason := readCrashGuardState(path) + if !ok { + s.logger.Debug("local-api: /host/crash-guard not present", "vmid", vmid, "reason", reason) + writeOK(w, CrashGuardResponse{Present: false}) + return + } + s.logger.Debug("local-api: /host/crash-guard served", "vmid", vmid, + "last_boot_at", resp.LastBootAt, "last_boot_unclean", resp.LastBootUnclean, "tripped", resp.Tripped) + writeOK(w, resp) +} diff --git a/internal/localapi/crashguard_test.go b/internal/localapi/crashguard_test.go new file mode 100644 index 0000000..f3b77f5 --- /dev/null +++ b/internal/localapi/crashguard_test.go @@ -0,0 +1,205 @@ +package localapi + +import ( + "encoding/json" + "io" + "log/slog" + "net/http" + "os" + "path/filepath" + "testing" +) + +// The shape of /var/lib/felhom-crash-guard/state.json as read on demo-hp on 2026-10-06 (values from +// that read where they matter; lists/objects kept to the same key set). +const crashGuardFixture = `{ + "armed": true, + "boot_id": "3f1c0f1e-6a0b-4d7e-9b7a-0c2d4e6f8a1b", + "config": {"LIMIT": 3, "WINDOW_MINUTES": 60, "PANIC_SECONDS": 10}, + "kernel_panic": 10, + "last_boot_at": "2026-10-05T07:56:41Z", + "last_boot_unclean": true, + "last_trip": {}, + "rearmed_at": "2026-10-04T14:02:11Z", + "rearmed_by": "operator", + "tripped": false, + "unclean_boots": ["2026-10-05T07:56:41Z"], + "unclean_boots_24h": 1, + "unclean_boots_in_window": 1, + "updated_at": "2026-10-05T07:57:02Z", + "version": 1 +}` + +// controllerCrashGuardState is a COPY of the controller's wire type, felhom-controller +// controller/internal/agentapi/crashguard.go `CrashGuardState` (commit 8b13a5e) — same field names, +// same tags. If either side renames a key, the contract test below fails. +type controllerCrashGuardState struct { + Present bool `json:"present"` + LastBootAt string `json:"last_boot_at,omitempty"` + LastBootUnclean bool `json:"last_boot_unclean"` + Tripped bool `json:"tripped"` +} + +func newCrashGuardServer(t *testing.T, statePath string) http.Handler { + t.Helper() + srv, err := NewServer(Options{ + ListenAddr: "127.0.0.1:0", + Guests: &fakeGuests{}, + Backups: &fakeBackups{}, + Store: &fakeStore{}, + Storage: fakeStorage{}, + Tokens: staticTokens{"A": 8200, "B": 9300}, + Logger: slog.New(slog.NewTextHandler(io.Discard, nil)), + }) + if err != nil { + t.Fatalf("new server: %v", err) + } + srv.crashGuardStatePath = statePath + return srv.Handler() +} + +func writeCrashGuardFixture(t *testing.T, body string) string { + t.Helper() + p := filepath.Join(t.TempDir(), "state.json") + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + return p +} + +// getCrashGuard calls the route and decodes the envelope with the CONTROLLER's type. +func getCrashGuard(t *testing.T, h http.Handler, token string) (int, controllerCrashGuardState, string) { + t.Helper() + w := do(t, h, "GET", "/host/crash-guard", token, "") + var env struct { + OK bool `json:"ok"` + Data controllerCrashGuardState `json:"data"` + } + if w.Code == http.StatusOK { + if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil { + t.Fatalf("decode %q: %v", w.Body.String(), err) + } + if !env.OK { + t.Fatalf("ok=false: %s", w.Body.String()) + } + } + return w.Code, env.Data, w.Body.String() +} + +// A present state file (demo-hp's shape) passes the three facts through. +func TestR856_CrashGuardPresentFile(t *testing.T) { + h := newCrashGuardServer(t, writeCrashGuardFixture(t, crashGuardFixture)) + code, st, body := getCrashGuard(t, h, "A") + if code != http.StatusOK { + t.Fatalf("got %d, want 200 (%s)", code, body) + } + want := controllerCrashGuardState{Present: true, LastBootAt: "2026-10-05T07:56:41Z", LastBootUnclean: true, Tripped: false} + if st != want { + t.Fatalf("state = %+v, want %+v", st, want) + } + + // A tripped, clean boot reads back as such (both bools are carried, not defaulted). + h = newCrashGuardServer(t, writeCrashGuardFixture(t, + `{"last_boot_at":"2026-10-05T09:56:41+02:00","last_boot_unclean":false,"tripped":true,"version":1}`)) + _, st, _ = getCrashGuard(t, h, "B") + want = controllerCrashGuardState{Present: true, LastBootAt: "2026-10-05T07:56:41Z", LastBootUnclean: false, Tripped: true} + if st != want { + t.Fatalf("offset time / tripped: state = %+v, want %+v (time normalised to UTC Z)", st, want) + } +} + +// No state file (no guard on this host, or no boot recorded yet) → 200 present:false. +func TestR856_CrashGuardMissingFile(t *testing.T) { + h := newCrashGuardServer(t, filepath.Join(t.TempDir(), "absent", "state.json")) + code, st, body := getCrashGuard(t, h, "A") + if code != http.StatusOK { + t.Fatalf("missing file: got %d, want 200 (%s)", code, body) + } + if st.Present || st.LastBootUnclean || st.Tripped || st.LastBootAt != "" { + t.Fatalf("missing file: state = %+v, want present:false and nothing else", st) + } +} + +// A garbled file → 200 present:false, never a 5xx — every shape of garbage. +func TestR856_CrashGuardGarbageFile(t *testing.T) { + for name, body := range map[string]string{ + "truncated": crashGuardFixture[:40], + "not json": "this is not json\n", + "empty": "", + "null": "null", + "array": `[{"last_boot_unclean":true}]`, + "wrong type": `{"last_boot_at":"2026-10-05T07:56:41Z","last_boot_unclean":"yes","tripped":false}`, + "lone brace": "{", + } { + t.Run(name, func(t *testing.T) { + h := newCrashGuardServer(t, writeCrashGuardFixture(t, body)) + code, st, raw := getCrashGuard(t, h, "A") + if code != http.StatusOK { + t.Fatalf("got %d, want 200 (%s)", code, raw) + } + if st.Present || st.LastBootUnclean { + t.Fatalf("garbage %q read as %+v, want present:false", name, st) + } + }) + } + // The path is a directory, not a file: unreadable → present:false, 200. + h := newCrashGuardServer(t, t.TempDir()) + if code, st, raw := getCrashGuard(t, h, "A"); code != http.StatusOK || st.Present { + t.Fatalf("directory path: got %d %+v (%s), want 200 present:false", code, st, raw) + } +} + +// No / unknown token → 401, like every sibling route; a cross-guest ?vmid= → 403. +func TestR856_CrashGuardRequiresGuestToken(t *testing.T) { + h := newCrashGuardServer(t, writeCrashGuardFixture(t, crashGuardFixture)) + for _, tok := range []string{"", "bogus"} { + w := do(t, h, "GET", "/host/crash-guard", tok, "") + if w.Code != http.StatusUnauthorized { + t.Fatalf("token %q: got %d, want 401", tok, w.Code) + } + if json.Valid(w.Body.Bytes()) { + var env struct { + Data controllerCrashGuardState `json:"data"` + } + _ = json.Unmarshal(w.Body.Bytes(), &env) + if env.Data.Present || env.Data.LastBootUnclean { + t.Fatalf("token %q: the refusal leaked the state: %s", tok, w.Body.String()) + } + } + } + if w := do(t, h, "GET", "/host/crash-guard?vmid=9300", "A", ""); w.Code != http.StatusForbidden { + t.Fatalf("cross-guest query: got %d, want 403", w.Code) + } +} + +// Wire contract: every key the controller's CrashGuardState decodes is emitted under exactly that +// name, and the agent emits no key the controller does not know. +func TestR856_CrashGuardWireMatchesControllerClient(t *testing.T) { + h := newCrashGuardServer(t, writeCrashGuardFixture(t, crashGuardFixture)) + w := do(t, h, "GET", "/host/crash-guard", "A", "") + var env struct { + OK bool `json:"ok"` + Data map[string]json.RawMessage `json:"data"` + } + if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil || !env.OK { + t.Fatalf("envelope: %v %s", err, w.Body.String()) + } + want := []string{"present", "last_boot_at", "last_boot_unclean", "tripped"} + for _, k := range want { + if _, ok := env.Data[k]; !ok { + t.Errorf("agent does not emit %q, which the controller decodes (%s)", k, w.Body.String()) + } + } + if len(env.Data) != len(want) { + t.Errorf("agent emits %d keys, controller knows %d: %s", len(env.Data), len(want), w.Body.String()) + } + // And the agent's own type agrees with the controller's copy, field for field. + var mine CrashGuardResponse + var theirs controllerCrashGuardState + raw, _ := json.Marshal(env.Data) + _ = json.Unmarshal(raw, &mine) + _ = json.Unmarshal(raw, &theirs) + if (controllerCrashGuardState{mine.Present, mine.LastBootAt, mine.LastBootUnclean, mine.Tripped}) != theirs { + t.Errorf("agent %+v vs controller %+v", mine, theirs) + } +} diff --git a/internal/localapi/server.go b/internal/localapi/server.go index 456dbc8..c6526be 100644 --- a/internal/localapi/server.go +++ b/internal/localapi/server.go @@ -294,6 +294,9 @@ type Server struct { netMountRoot string // the user-data namespace root for the network-mount role gate smbCredsDir string // where SMB creds files are written (out-of-band, 0600) escrowStagePath string // fork-4: 0600 staging file for the pushed restic repo password + // crashGuardStatePath (R-856) is the host crash guard's state file read by GET /host/crash-guard; + // empty = defaultCrashGuardStatePath. A seam: tests point it at a fixture. + crashGuardStatePath string // escrowRecovery (R-199, v0.125.0) assembles chain links 6-8: fetch this host's own sealed // identity blob from the hub, unseal it with the customer's recovery code, return ONLY the // offsite repository password. OPTIONAL — nil (no hub client configured) makes @@ -520,6 +523,9 @@ func (s *Server) Handler() http.Handler { // Host metrics (slice 9): host-wide health + per-storage capacity for the customer's monitoring // view. Host-wide, token-authed, fresh (a live collect, not the 15-min hub snapshot). mux.HandleFunc("GET /host/metrics", s.withGuest(s.handleHostMetrics)) + // R-856 (`09` §3 decision 143): the host crash guard's record of the last HOST boot — the controller + // waits ~15 min with app mails after an unclean one. Read-only; a missing/garbled file = present:false. + mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard)) // Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate. mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks)) mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates))