R-856: GET /host/crash-guard — the host crash guard's last-boot record for the controller (09 decision 143)

The controller waits ~15 minutes with app mails after a crash boot of the host; it learns of the
crash boot from this route. Reads /var/lib/felhom-crash-guard/state.json (read-only, no Proxmox call)
and passes present/last_boot_at/last_boot_unclean/tripped through; a missing, unreadable or garbled
file answers 200 present:false. Guest-token authed like every sibling route.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

REUSE.md

@@ -121,7 +121,7 @@
 | Anti-retarget durable-id binding | internal/localapi/wipe_reresolve.go | resolve id → re-derive + exact match → re-inspect expected state → act on RE-RESOLVED device only |
 | Atomic single-file JSON store | internal/storage/intent.go | `Open*` loads (missing=empty, corrupt=fail-loud), mutex, tmp+rename 0600, idempotent set |
 | Durable append-only log + index | internal/authz/noncestore.go (`FileNonceStore`) | fsync before returning "new"; replay into index on open; expiry-only compaction |
-| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
+| Injectable seam funcs on Server | internal/localapi/server.go (`reresolveWipe`, `deviceDurableID`, `boundCheck`, `deviceCheck`, `livenessCheck`, net-verify: `netTrigger`/`netMounted`/`netJournal`/`netReachable`; R-856 `crashGuardStatePath` — GET /host/crash-guard's state file, internal/localapi/crashguard.go) | prod default wired in `NewServer`; tests override — no real /dev, /proc/mounts, journalctl or TCP in tests. **For mount-table predicates prefer the DATA seams `procSelfMountinfo` / `procGuestMountinfo` (internal/localapi/intermediary.go) over `boundCheck`/`livenessCheck`**: pointing them at a captured fixture runs the real parser, the real predicate and the real handler, so the test cannot go hollow the way R-116's did |
 | `Server.devicePresent` (R-113, v0.114.0) | internal/localapi/disks.go | `devicePresent(rawMountPath) bool`; seam `deviceCheck`, default `isHostMountpoint` | the agent's DEVICE-presence signal — asks whether the drive's RAW mount is still mounted | **Use this, never the bind, to answer "is the drive there".** The raw mount is a device-bound systemd unit and dies with its device; the agent's own bind under the shared parent is NOT device-bound and outlives it as a stale shell. `BoundUnderParent` is now `boundUnderParent(...) && devicePresent(...)` at BOTH /disks construction sites — dropping either half is a regression with its own red-proof. Empty path ⇒ **true** (unknown is never absent: absent stops a customer's apps) |
 | `bindLiveness` + `BindLiveness` (R-117, v0.117.0) | internal/localapi/intermediary.go | `bindLiveness(stable, raw) BindLiveness`; seam `livenessCheck`; read verdicts ONLY via `.Usable()` | the agent's bind-LIVENESS signal — the third term of `BoundUnderParent` | **`devicePresent` and `boundUnderParent` are both PATH-PRESENCE tests and neither is liveness.** They compare only mountinfo field 5, so both stay true over a bind that names the drive that went away while the raw mount healed onto the returning one (measured: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb `shutdown`, EIO both ways, payload healthy). Two dead states, and a fix needs BOTH checks: devno mismatch (the detach/return case) AND the ext4 abort tokens `shutdown`/`emergency_ro` (the steady-state case, where the devnos AGREE because the device never left). **THREE states, never a bool** — `BindUnknown` must exist and `Usable()` treats it as PRESENT (absent stops a customer's apps). **Order matters:** compare devices first and read the abort flag off the RAW mount in the stale case — abort-first classifies the real return state as aborted and refuses the re-bind that repairs it. **NO BLOCK I/O, ever** (CLAUDE.md rule; a probe on a wedged device survives SIGKILL). 6 red-proofs |
 | `AttachDrive` repair ruling (R-117, v0.117.0) | internal/localapi/intermediary.go | the `switch bindLiveness(...)` inside the `n == 1 && GuestSeesMount` arm | decides whether the existing self-heal runs | `BindStaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, no guest restart). `BindAborted` ⇒ **quiet no-op** — a re-bind lands on the SAME dead superblock and this runs every 20 s, so re-binding is an infinite silent retry that also masks the state; it must surface via `BoundUnderParent=false`. `BindLive`/`BindUnknown` ⇒ no-op, unchanged. **Do not return an error for the aborted case** — the reconcile loop would log a failure every 20 s |

internal/localapi/crashguard.go

@@ -0,0 +1,102 @@
+package localapi
+
+import (
+	"encoding/json"
+	"errors"
+	"io"
+	"io/fs"
+	"net/http"
+	"os"
+	"time"
+)
+
+// GET /host/crash-guard (R-856, `09` §3 decision 143): what the host's crash guard
+// (configs/felhom-crash-guard, `11` §5.9) recorded about the most recent HOST boot. The controller
+// reads it once after it starts: when the host's last boot followed an UNCLEAN stop, its app mails
+// wait ~15 minutes instead of the normal 90 s boot grace.
+//
+// Read-only and Proxmox-free: the agent reads the guard's state file (root-owned, 0644 — the
+// non-root agent can read it) and passes four fields through. Host-wide, token-authed (any valid
+// per-guest token sees the host's view, as GET /host/metrics does).
+//
+// NEVER an error page. A missing file (no guard installed, or no boot recorded yet), an unreadable
+// one, or one that does not parse answers 200 with present:false — the controller reads that as
+// UNKNOWN and keeps its normal boot grace. Pinned by TestR856_CrashGuard*.
+
+// defaultCrashGuardStatePath is where configs/felhom-crash-guard writes its state (STATE_DIR there).
+const defaultCrashGuardStatePath = "/var/lib/felhom-crash-guard/state.json"
+
+// crashGuardStateMax bounds the read; the real file is well under 4 KiB.
+const crashGuardStateMax = 1 << 20
+
+// CrashGuardResponse is the data block of GET /host/crash-guard. Field names are the controller's
+// agentapi.CrashGuardState (felhom-controller internal/agentapi/crashguard.go) — a wire contract,
+// pinned by TestR856_CrashGuardWireMatchesControllerClient.
+type CrashGuardResponse struct {
+	Present         bool   `json:"present"`
+	LastBootAt      string `json:"last_boot_at,omitempty"` // RFC3339 UTC ("2006-01-02T15:04:05Z")
+	LastBootUnclean bool   `json:"last_boot_unclean"`
+	Tripped         bool   `json:"tripped"`
+}
+
+// crashGuardFile is the subset of the guard's state.json the route passes through. Every other key
+// (armed, boot_id, config, unclean_boots, last_trip, ...) is ignored.
+type crashGuardFile struct {
+	LastBootAt      string `json:"last_boot_at"`
+	LastBootUnclean bool   `json:"last_boot_unclean"`
+	Tripped         bool   `json:"tripped"`
+}
+
+// readCrashGuardState reads and parses the guard's state file. ok=false on ANY failure (missing,
+// unreadable, oversized, not a JSON object, a field of the wrong type); reason says which, for the log.
+func readCrashGuardState(path string) (resp CrashGuardResponse, ok bool, reason string) {
+	f, err := os.Open(path)
+	if err != nil {
+		if errors.Is(err, fs.ErrNotExist) {
+			return resp, false, "no state file"
+		}
+		return resp, false, "unreadable: " + err.Error()
+	}
+	defer f.Close()
+	raw, err := io.ReadAll(io.LimitReader(f, crashGuardStateMax+1))
+	if err != nil {
+		return resp, false, "read: " + err.Error()
+	}
+	if len(raw) > crashGuardStateMax {
+		return resp, false, "state file too large"
+	}
+	var st crashGuardFile
+	// Unmarshal into a struct fails on a non-object top level (null decodes, so reject it below).
+	if err := json.Unmarshal(raw, &st); err != nil {
+		return resp, false, "unparseable: " + err.Error()
+	}
+	var probe map[string]json.RawMessage
+	if err := json.Unmarshal(raw, &probe); err != nil || probe == nil {
+		return resp, false, "unparseable: not a JSON object"
+	}
+	resp = CrashGuardResponse{Present: true, LastBootUnclean: st.LastBootUnclean, Tripped: st.Tripped}
+	// Normalise to RFC3339 UTC; an unparseable time passes through as-is (the controller reads an
+	// unparseable boot time as "not this start's boot" → its normal grace).
+	if t, perr := time.Parse(time.RFC3339, st.LastBootAt); perr == nil {
+		resp.LastBootAt = t.UTC().Format(time.RFC3339)
+	} else {
+		resp.LastBootAt = st.LastBootAt
+	}
+	return resp, true, ""
+}
+
+func (s *Server) handleCrashGuard(w http.ResponseWriter, r *http.Request, vmid int) {
+	path := s.crashGuardStatePath
+	if path == "" {
+		path = defaultCrashGuardStatePath
+	}
+	resp, ok, reason := readCrashGuardState(path)
+	if !ok {
+		s.logger.Debug("local-api: /host/crash-guard not present", "vmid", vmid, "reason", reason)
+		writeOK(w, CrashGuardResponse{Present: false})
+		return
+	}
+	s.logger.Debug("local-api: /host/crash-guard served", "vmid", vmid,
+		"last_boot_at", resp.LastBootAt, "last_boot_unclean", resp.LastBootUnclean, "tripped", resp.Tripped)
+	writeOK(w, resp)
+}

internal/localapi/crashguard_test.go

@@ -0,0 +1,205 @@
+package localapi
+
+import (
+	"encoding/json"
+	"io"
+	"log/slog"
+	"net/http"
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+// The shape of /var/lib/felhom-crash-guard/state.json as read on demo-hp on 2026-10-06 (values from
+// that read where they matter; lists/objects kept to the same key set).
+const crashGuardFixture = `{
+  "armed": true,
+  "boot_id": "3f1c0f1e-6a0b-4d7e-9b7a-0c2d4e6f8a1b",
+  "config": {"LIMIT": 3, "WINDOW_MINUTES": 60, "PANIC_SECONDS": 10},
+  "kernel_panic": 10,
+  "last_boot_at": "2026-10-05T07:56:41Z",
+  "last_boot_unclean": true,
+  "last_trip": {},
+  "rearmed_at": "2026-10-04T14:02:11Z",
+  "rearmed_by": "operator",
+  "tripped": false,
+  "unclean_boots": ["2026-10-05T07:56:41Z"],
+  "unclean_boots_24h": 1,
+  "unclean_boots_in_window": 1,
+  "updated_at": "2026-10-05T07:57:02Z",
+  "version": 1
+}`
+
+// controllerCrashGuardState is a COPY of the controller's wire type, felhom-controller
+// controller/internal/agentapi/crashguard.go `CrashGuardState` (commit 8b13a5e) — same field names,
+// same tags. If either side renames a key, the contract test below fails.
+type controllerCrashGuardState struct {
+	Present         bool   `json:"present"`
+	LastBootAt      string `json:"last_boot_at,omitempty"`
+	LastBootUnclean bool   `json:"last_boot_unclean"`
+	Tripped         bool   `json:"tripped"`
+}
+
+func newCrashGuardServer(t *testing.T, statePath string) http.Handler {
+	t.Helper()
+	srv, err := NewServer(Options{
+		ListenAddr: "127.0.0.1:0",
+		Guests:     &fakeGuests{},
+		Backups:    &fakeBackups{},
+		Store:      &fakeStore{},
+		Storage:    fakeStorage{},
+		Tokens:     staticTokens{"A": 8200, "B": 9300},
+		Logger:     slog.New(slog.NewTextHandler(io.Discard, nil)),
+	})
+	if err != nil {
+		t.Fatalf("new server: %v", err)
+	}
+	srv.crashGuardStatePath = statePath
+	return srv.Handler()
+}
+
+func writeCrashGuardFixture(t *testing.T, body string) string {
+	t.Helper()
+	p := filepath.Join(t.TempDir(), "state.json")
+	if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	return p
+}
+
+// getCrashGuard calls the route and decodes the envelope with the CONTROLLER's type.
+func getCrashGuard(t *testing.T, h http.Handler, token string) (int, controllerCrashGuardState, string) {
+	t.Helper()
+	w := do(t, h, "GET", "/host/crash-guard", token, "")
+	var env struct {
+		OK   bool                      `json:"ok"`
+		Data controllerCrashGuardState `json:"data"`
+	}
+	if w.Code == http.StatusOK {
+		if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil {
+			t.Fatalf("decode %q: %v", w.Body.String(), err)
+		}
+		if !env.OK {
+			t.Fatalf("ok=false: %s", w.Body.String())
+		}
+	}
+	return w.Code, env.Data, w.Body.String()
+}
+
+// A present state file (demo-hp's shape) passes the three facts through.
+func TestR856_CrashGuardPresentFile(t *testing.T) {
+	h := newCrashGuardServer(t, writeCrashGuardFixture(t, crashGuardFixture))
+	code, st, body := getCrashGuard(t, h, "A")
+	if code != http.StatusOK {
+		t.Fatalf("got %d, want 200 (%s)", code, body)
+	}
+	want := controllerCrashGuardState{Present: true, LastBootAt: "2026-10-05T07:56:41Z", LastBootUnclean: true, Tripped: false}
+	if st != want {
+		t.Fatalf("state = %+v, want %+v", st, want)
+	}
+
+	// A tripped, clean boot reads back as such (both bools are carried, not defaulted).
+	h = newCrashGuardServer(t, writeCrashGuardFixture(t,
+		`{"last_boot_at":"2026-10-05T09:56:41+02:00","last_boot_unclean":false,"tripped":true,"version":1}`))
+	_, st, _ = getCrashGuard(t, h, "B")
+	want = controllerCrashGuardState{Present: true, LastBootAt: "2026-10-05T07:56:41Z", LastBootUnclean: false, Tripped: true}
+	if st != want {
+		t.Fatalf("offset time / tripped: state = %+v, want %+v (time normalised to UTC Z)", st, want)
+	}
+}
+
+// No state file (no guard on this host, or no boot recorded yet) → 200 present:false.
+func TestR856_CrashGuardMissingFile(t *testing.T) {
+	h := newCrashGuardServer(t, filepath.Join(t.TempDir(), "absent", "state.json"))
+	code, st, body := getCrashGuard(t, h, "A")
+	if code != http.StatusOK {
+		t.Fatalf("missing file: got %d, want 200 (%s)", code, body)
+	}
+	if st.Present || st.LastBootUnclean || st.Tripped || st.LastBootAt != "" {
+		t.Fatalf("missing file: state = %+v, want present:false and nothing else", st)
+	}
+}
+
+// A garbled file → 200 present:false, never a 5xx — every shape of garbage.
+func TestR856_CrashGuardGarbageFile(t *testing.T) {
+	for name, body := range map[string]string{
+		"truncated":  crashGuardFixture[:40],
+		"not json":   "this is not json\n",
+		"empty":      "",
+		"null":       "null",
+		"array":      `[{"last_boot_unclean":true}]`,
+		"wrong type": `{"last_boot_at":"2026-10-05T07:56:41Z","last_boot_unclean":"yes","tripped":false}`,
+		"lone brace": "{",
+	} {
+		t.Run(name, func(t *testing.T) {
+			h := newCrashGuardServer(t, writeCrashGuardFixture(t, body))
+			code, st, raw := getCrashGuard(t, h, "A")
+			if code != http.StatusOK {
+				t.Fatalf("got %d, want 200 (%s)", code, raw)
+			}
+			if st.Present || st.LastBootUnclean {
+				t.Fatalf("garbage %q read as %+v, want present:false", name, st)
+			}
+		})
+	}
+	// The path is a directory, not a file: unreadable → present:false, 200.
+	h := newCrashGuardServer(t, t.TempDir())
+	if code, st, raw := getCrashGuard(t, h, "A"); code != http.StatusOK || st.Present {
+		t.Fatalf("directory path: got %d %+v (%s), want 200 present:false", code, st, raw)
+	}
+}
+
+// No / unknown token → 401, like every sibling route; a cross-guest ?vmid= → 403.
+func TestR856_CrashGuardRequiresGuestToken(t *testing.T) {
+	h := newCrashGuardServer(t, writeCrashGuardFixture(t, crashGuardFixture))
+	for _, tok := range []string{"", "bogus"} {
+		w := do(t, h, "GET", "/host/crash-guard", tok, "")
+		if w.Code != http.StatusUnauthorized {
+			t.Fatalf("token %q: got %d, want 401", tok, w.Code)
+		}
+		if json.Valid(w.Body.Bytes()) {
+			var env struct {
+				Data controllerCrashGuardState `json:"data"`
+			}
+			_ = json.Unmarshal(w.Body.Bytes(), &env)
+			if env.Data.Present || env.Data.LastBootUnclean {
+				t.Fatalf("token %q: the refusal leaked the state: %s", tok, w.Body.String())
+			}
+		}
+	}
+	if w := do(t, h, "GET", "/host/crash-guard?vmid=9300", "A", ""); w.Code != http.StatusForbidden {
+		t.Fatalf("cross-guest query: got %d, want 403", w.Code)
+	}
+}
+
+// Wire contract: every key the controller's CrashGuardState decodes is emitted under exactly that
+// name, and the agent emits no key the controller does not know.
+func TestR856_CrashGuardWireMatchesControllerClient(t *testing.T) {
+	h := newCrashGuardServer(t, writeCrashGuardFixture(t, crashGuardFixture))
+	w := do(t, h, "GET", "/host/crash-guard", "A", "")
+	var env struct {
+		OK   bool                       `json:"ok"`
+		Data map[string]json.RawMessage `json:"data"`
+	}
+	if err := json.Unmarshal(w.Body.Bytes(), &env); err != nil || !env.OK {
+		t.Fatalf("envelope: %v %s", err, w.Body.String())
+	}
+	want := []string{"present", "last_boot_at", "last_boot_unclean", "tripped"}
+	for _, k := range want {
+		if _, ok := env.Data[k]; !ok {
+			t.Errorf("agent does not emit %q, which the controller decodes (%s)", k, w.Body.String())
+		}
+	}
+	if len(env.Data) != len(want) {
+		t.Errorf("agent emits %d keys, controller knows %d: %s", len(env.Data), len(want), w.Body.String())
+	}
+	// And the agent's own type agrees with the controller's copy, field for field.
+	var mine CrashGuardResponse
+	var theirs controllerCrashGuardState
+	raw, _ := json.Marshal(env.Data)
+	_ = json.Unmarshal(raw, &mine)
+	_ = json.Unmarshal(raw, &theirs)
+	if (controllerCrashGuardState{mine.Present, mine.LastBootAt, mine.LastBootUnclean, mine.Tripped}) != theirs {
+		t.Errorf("agent %+v vs controller %+v", mine, theirs)
+	}
+}

internal/localapi/server.go

@@ -294,6 +294,9 @@ type Server struct {
 	netMountRoot    string              // the user-data namespace root for the network-mount role gate
 	smbCredsDir     string              // where SMB creds files are written (out-of-band, 0600)
 	escrowStagePath string              // fork-4: 0600 staging file for the pushed restic repo password
+	// crashGuardStatePath (R-856) is the host crash guard's state file read by GET /host/crash-guard;
+	// empty = defaultCrashGuardStatePath. A seam: tests point it at a fixture.
+	crashGuardStatePath string
 	// escrowRecovery (R-199, v0.125.0) assembles chain links 6-8: fetch this host's own sealed
 	// identity blob from the hub, unseal it with the customer's recovery code, return ONLY the
 	// offsite repository password. OPTIONAL — nil (no hub client configured) makes
@@ -520,6 +523,9 @@ func (s *Server) Handler() http.Handler {
 	// Host metrics (slice 9): host-wide health + per-storage capacity for the customer's monitoring
 	// view. Host-wide, token-authed, fresh (a live collect, not the 15-min hub snapshot).
 	mux.HandleFunc("GET /host/metrics", s.withGuest(s.handleHostMetrics))
+	// R-856 (`09` §3 decision 143): the host crash guard's record of the last HOST boot — the controller
+	// waits ~15 min with app mails after an unclean one. Read-only; a missing/garbled file = present:false.
+	mux.HandleFunc("GET /host/crash-guard", s.withGuest(s.handleCrashGuard))
 	// Disk management (slice 8C) — self-scoped; format routes through the data-bearing classifier+gate.
 	mux.HandleFunc("GET /disks", s.withGuest(s.handleDisks))
 	mux.HandleFunc("GET /disks/candidates", s.withGuest(s.handleDiskCandidates))