diff --git a/documentation/architecture/05-hub-architecture.md b/documentation/architecture/05-hub-architecture.md index 16c84398..e8aad816 100644 --- a/documentation/architecture/05-hub-architecture.md +++ b/documentation/architecture/05-hub-architecture.md @@ -222,6 +222,25 @@ Today: `GET /config/{id}` with an `X-Retrieval-Password` (Hungarian passphrase) The existing `configgen` deep-merge + Hungarian-passphrase machinery is the base; it grows the agent-first + key-pinning + restore-mode steps. +**A reinstall over an existing host record re-issues when the old box is gone [DESIGN, 2026-10-19 — R-1154, brief +2026-10-19; hub v0.159.0].** `POST /host-enroll` is the installer's step 4, run on every pass (a `--resume` too), with +the customer's passphrase; an existing host record is REUSED (same id and key — the reinstalled box takes it over). +The clean-slate re-issues (the claim code, the off-site login — both live in the CONTROLLER) used to run only when a +record was minted, so a reinstall that kept the old record skipped both in silence. The reuse path now runs them +when, and only when, the old box is gone: **no controller wait connection open now**, and either the household +switched the box off on purpose (`deliberate_off_at` inside its hold) or **both** the agent's host report and the +controller's report are older than `alerting.stale_threshold` (or absent) — the threshold and comparison +`hostStatus` uses, no second definition. **Once per silence:** not again when it already ran since the host's last +report (an installer `--resume` before the new agent's first report). **Why this rule and not "re-issue on every +reuse":** a live box re-enrolls legitimately (a `--resume` after its agent started, a repair re-run), and re-issuing +there rotates the household's claim code and resets its off-site login under a running box; not re-issuing costs a +reinstall two operator presses, which the skip record now names. The controller report counts because the codes +live there: a box whose agent is broken but whose controller reports is live. **Known residual:** a reinstall done +less than the threshold after the old box's last report is still treated as live — it is recorded +(`host_reenroll_not_reissued`, operator-only, info) instead of silent, and the operator presses the two buttons or +deletes the old host first. The re-issue writes `host_reenroll_reissued` (also the once-per-silence record); both +types are hub-origin only. `hub/internal/api/reenroll.go`, pinned by `hub/internal/api/r1154_reenroll_test.go`. + ## 9. DR model The headline: the **old heavy infra-backup push retires** — not because the hub authors everything diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index c71b99de..7c1d380f 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,41 @@ +## v0.159.0 — a refused whole-guest backup has its own alarm text (R-1150); a reinstall re-issues the claim code and the off-site login when the old host is stale or off (R-1154) (2026-10-19) + +- **`whole_guest_backup_refused`** (R-1150). The agent's R-973 cadence cap reports a CRITICAL capability + `localapi:backup-cadence-cap` as degraded for 24 h after it refused a whole-guest backup a guest asked for beyond its + schedule; the hub mailed that as `agent_capability_degraded` „agent privileged capability degraded" — read as a + broken sudo grant. `HostCapabilityChecker` now keeps it as a SECOND, independent state per host: the cadence cap + alone raises `whole_guest_backup_refused` (warning) „Host : a whole-guest backup was refused — the guest asked + for more whole-guest backups than its schedule allows. Its controller may be misbehaving or compromised: check the + guest.", the agent's reason (guest, target, time) in the details; its end is `whole_guest_backup_refusal_cleared` + (info: recorded, never mailed). Every other critical capability keeps `agent_capability_degraded` / + `_recovered` with unchanged text and never names the cadence cap; both on one report → one event per state, never + two for one; a grant restored while the refusal stands → `agent_capability_recovered` alone. The agent is unchanged + (its status name stays). Both new types are operator-only and hub-origin only — **deliberately NOT in + `allowedEventTypes`** (the brief suggested it): the alarm suspects the guest's controller, which holds the + `/event` credential, and a type it can post it can fake or post first to spend the operator cooldown (the R-895 + reasoning, `TestR1150_RefusedBackupEventsAreOperatorOnlyAndHubOrigin`). **Also:** `agent_capability_degraded` / + `_recovered` enter `operatorOnlyEvents` — the checker has said „Customer is NEVER notified" since v0.44.0 and only + `enabled_events` configuration made it true (the R-97c shape). Tests `TestR1150_CadenceCapAloneRaisesWholeGuestBackupRefused`, + `TestR1150_OtherCapStillRaisesCapabilityDegraded`, `TestR1150_BornRefusedAlertsOnce` (red-proved: the pre-fix + shape emits `agent_capability_degraded` instead), `TestR1150_RefusedBackupAndCapabilityEventsNeverReachTheCustomer` + (red-proved: without the register lines the household is mailed). +- **A reinstall over the old host record re-issues** (R-1154; `05` §8 [DESIGN]). `handleHostEnroll` reused an + existing host record and returned at once, so `reissueOnReenroll` (claim code + off-site login) ran only on a mint — + a reinstall that kept the old record skipped both in silence (demo-hp 2026-08-21; Tester4's standing reinstalls). + The reuse path (same host id and key, unchanged) now re-issues when the old box is gone: no controller wait + connection open, and either `deliberate_off_at` inside its hold or BOTH the host report and the controller report + older than `alerting.stale_threshold` (or absent) — the `hostStatus` threshold, wired from main + (`SetStaleThreshold`; unwired = never re-issues). Once per silence (an installer `--resume` before the new agent + reports does not re-issue again): the record is the info event `host_reenroll_reissued`, written by the mint path + too. A live box re-enrolling gets nothing re-issued and the skip is recorded (`host_reenroll_not_reissued`), so a + reinstall within the threshold is visible instead of silent. Both types operator-only, hub-origin only. + `internal/api/reenroll.go`. Tests `TestR1154_StaleOldHostIsReissued`, `TestR1154_DeliberatelyOffOldHostIsReissued`, + `TestR1154_LiveHostIsNotReissued` (host report / controller report / wait connection / connected-while-off), + `TestR1154_ResumeDoesNotReissueTwice`, `TestR1154_NeverReportedAndUnwiredEdges`, `TestR1154_EventsAreOperatorOnlyAndHubOrigin`, + `TestR1154_MainWiresTheStaleThresholdIntoTheAPI` — each red-proved (the pre-fix early return: „claim generation 1→1, + claim mails 0, off-site re-issues 0"; an unconditional re-issue: „claim generation 1→2 … off-site re-issues 1" on a + live box; no once-per-silence check: „claim generation 1→3, off-site re-issues 2"). + ## v0.158.0 — agent-only vouch, the hub's own snapshot alarms, the hub's own access log (2026-10-17) — built that night; DEPLOYED 2026-10-18 on the operator's yes (12:16) - **An agent can be vouched mid-week** (`09` §3 decision 250 items 3–4). The Day-0 save's R-120 golden guard refused diff --git a/hub/README.md b/hub/README.md index 11cd3c17..d597cf21 100644 --- a/hub/README.md +++ b/hub/README.md @@ -148,7 +148,13 @@ Events are the primary monitoring mechanism. Each event has: customer_id, event_ - `node_stale` / `node_down` / `node_recovered` — dead man's switch from staleness checker (every 60s) - `host_stale` / `host_down` / `host_recovered` — host-domain dead man's switch (agent host reports) - `agent_capability_degraded` / `agent_capability_recovered` — `HostCapabilityChecker`: an agent's - required `sudo -n` grant went missing (the non-root cutover class) + required `sudo -n` grant went missing (the non-root cutover class). Operator-only since v0.159.0. +- `whole_guest_backup_refused` (warning) / `whole_guest_backup_refusal_cleared` (info) — the same checker, a + SEPARATE state (R-1150, v0.159.0): the agent's `localapi:backup-cadence-cap` reports that it refused a whole-guest + backup a guest asked for beyond its schedule (R-973). Operator-only; hub-origin only (not POST-able by a box). +- `host_reenroll_reissued` / `host_reenroll_not_reissued` (info) — `POST /host-enroll` reused an existing host + record and re-issued the claim code and the off-site login (the old box is gone), or did not (the box is live) + (R-1154, v0.159.0). Operator-only; hub-origin only. - `host_leaf_changed` — `HostLeafChecker` (`monitor/host_leaf.go`, every 60s): an agent's reported local-API **leaf fingerprint** changed (an agent re-key) — proactive, fleet-wide, independent of any controller's channel-health check. Trust-on-first-report baseline; empty fp = unknown (never alerts). @@ -181,7 +187,7 @@ Config retrieval uses a separate per-customer retrieval password (not the API ke |--------|------|-------------| | `POST` | `/api/v1/host-enroll` | Mint-or-reuse a host credential, body `{customer_id}` (auth: `X-Retrieval-Password` header) → `{host_id, api_key}` | -The Day-0 first-boot handshake (`documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md`) enrolls a Proxmox host's agent **using the customer's retrieval passphrase only** — the operator-tier global key never enters the field deploy path. `host-enroll` mints the per-host credential on the first call and **reuses it byte-for-byte** on every later call (idempotent — re-running the host-bootstrap never orphans a running agent's key), backed by `Store.GetHostByCustomer`. Auth is checked **before** any mint, so a wrong passphrase never creates a host row. Behaviour: first → `201`, reuse → `200`, wrong/missing passphrase → `401`, unknown customer → `404`, missing `customer_id` → `400`. +The Day-0 first-boot handshake (`documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md`) enrolls a Proxmox host's agent **using the customer's retrieval passphrase only** — the operator-tier global key never enters the field deploy path. `host-enroll` mints the per-host credential on the first call and **reuses it byte-for-byte** on every later call (idempotent — re-running the host-bootstrap never orphans a running agent's key), backed by `Store.GetHostByCustomer`. Auth is checked **before** any mint, so a wrong passphrase never creates a host row. Behaviour: first → `201`, reuse → `200`, wrong/missing passphrase → `401`, unknown customer → `404`, missing `customer_id` → `400`. **Reinstall (R-1154, v0.159.0):** on a reuse whose old box is gone — its host report AND its controller report both older than `alerting.stale_threshold` (or absent), or the household switched it off on purpose, and no controller wait connection open — the enroll also runs the clean-slate re-issues (claim code, off-site login), once per silence; a live box re-enrolling gets nothing re-issued (`internal/api/reenroll.go`, `05` §8). The global-key `POST /api/v1/admin/hosts` (operator/HQ pre-mint) remains as the escape hatch pending the enrollment-cutover lock-down; it and `GET /config/{id}` are unchanged by this endpoint. diff --git a/hub/cmd/hub/main.go b/hub/cmd/hub/main.go index 196845b3..1177e653 100644 --- a/hub/cmd/hub/main.go +++ b/hub/cmd/hub/main.go @@ -337,6 +337,9 @@ func main() { } apiHandler := api.New(dataStore, cfg.API.ReportAPIKey, cfg.Notifications.ResendAPIKey, cfg.Notifications.FromEmail, templateProvider, logger) apiHandler.SetAssetManager(assetsMgr) + // R-1154: the host-enroll reuse path judges "the old box is gone" by the same threshold as hostStatus and the + // staleness checkers (unwired = it never re-issues). Pinned by r1154_wiring_test.go. + apiHandler.SetStaleThreshold(staleThreshold) // Initialize notification dispatcher dispatcher := notify.NewDispatcher( diff --git a/hub/cmd/hub/r1154_wiring_test.go b/hub/cmd/hub/r1154_wiring_test.go new file mode 100644 index 00000000..31eb2aa0 --- /dev/null +++ b/hub/cmd/hub/r1154_wiring_test.go @@ -0,0 +1,41 @@ +package main + +import ( + "go/ast" + "go/parser" + "go/token" + "testing" +) + +// R-1154 seam wiring: main() must hand the api handler the SAME stale threshold the staleness checkers get +// (apiHandler.SetStaleThreshold(staleThreshold)). Unwired, the host-enroll reuse path never re-issues the claim +// code and the off-site login on a reinstall — the R-1154 defect, back in silence. +// +// COMPANION RED-PROOF (observed 2026-10-19): with the call deleted from main.go this fails with +// "cmd/hub/main.go never calls apiHandler.SetStaleThreshold(staleThreshold)". Restored. +func TestR1154_MainWiresTheStaleThresholdIntoTheAPI(t *testing.T) { + f, err := parser.ParseFile(token.NewFileSet(), "main.go", nil, 0) + if err != nil { + t.Fatal(err) + } + wired := false + ast.Inspect(f, func(n ast.Node) bool { + c, ok := n.(*ast.CallExpr) + if !ok { + return true + } + sel, ok := c.Fun.(*ast.SelectorExpr) + if !ok || sel.Sel.Name != "SetStaleThreshold" || len(c.Args) != 1 { + return true + } + recv, _ := sel.X.(*ast.Ident) + arg, _ := c.Args[0].(*ast.Ident) + if recv != nil && recv.Name == "apiHandler" && arg != nil && arg.Name == "staleThreshold" { + wired = true + } + return true + }) + if !wired { + t.Error("cmd/hub/main.go never calls apiHandler.SetStaleThreshold(staleThreshold)") + } +} diff --git a/hub/internal/api/handler.go b/hub/internal/api/handler.go index aeb52fc0..906f6ab6 100644 --- a/hub/internal/api/handler.go +++ b/hub/internal/api/handler.go @@ -111,6 +111,17 @@ type Handler struct { // box ticks in seconds instead of ≤15 min. Shared with the web server (same *poke.Notifier). nil // = poke disabled (a no-op; the report cycle still reconciles). poker Poker + + // staleThreshold (R-1154, v0.159.0) is the hub's report-liveness threshold (alerting.stale_threshold — the + // same value hostStatus and both staleness checkers read). The host-enroll REUSE path re-issues the + // clean-slate credentials only when the old box has been silent past it (reenroll.go). 0 = not wired: the + // reuse path then never re-issues (the pre-v0.159.0 behaviour — the fail-safe direction). main wires it. + staleThreshold time.Duration +} + +// SetStaleThreshold wires the report-liveness threshold the host-enroll reuse path judges by (R-1154). +func (h *Handler) SetStaleThreshold(d time.Duration) { + h.staleThreshold = d } // SetClaimEngine wires the customer-claim code engine (nil-safe everywhere it is used). @@ -1205,7 +1216,10 @@ func (h *Handler) handleHostEnroll(w http.ResponseWriter, r *http.Request) { return } - // Mint-once-reuse: an existing host for this customer is returned as-is (idempotent). + // Mint-once-reuse: an existing host for this customer is returned as-is (idempotent) — the SAME host id and + // key, so re-running the installer never orphans a live agent's key and a reinstalled box takes over the + // record. R-1154 (v0.159.0): when the old box has gone silent (or was switched off on purpose), this + // enroll is a reinstall and the clean-slate re-issues run here too — reenroll.go decides. existing, err := h.store.GetHostByCustomer(req.CustomerID) if err != nil { h.logger.Printf("[ERROR] host-enroll: host lookup failed for %s: %v", req.CustomerID, err) @@ -1219,6 +1233,7 @@ func (h *Handler) handleHostEnroll(w http.ResponseWriter, r *http.Request) { http.Error(w, "Internal error", http.StatusInternalServerError) return } + h.reissueOnReuseIfReinstall(cc, existing, time.Now()) w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusOK) json.NewEncoder(w).Encode(map[string]string{"host_id": existing.HostID, "api_key": existing.APIKey}) @@ -1245,13 +1260,17 @@ func (h *Handler) handleHostEnroll(w http.ResponseWriter, r *http.Request) { h.logger.Printf("[INFO] host enrolled: %s (customer %s)", hostID, req.CustomerID) // F2 (v0.57.0) — clean-slate reinstall of an existing customer. This mint path fires exactly - // once per fresh host record (the clean-slate flow deletes the stale host, so re-enroll mints), - // so it is the natural single-shot re-enroll hook. For a CLAIMED customer the fresh box has no + // once per fresh host record (a host deleted before the reinstall), so it is one of the two + // single-shot re-enroll hooks — the other is the reuse path above for a silent old host (R-1154). + // For a CLAIMED customer the fresh box has no // password; auto-issue a reset code so the customer isn't stranded at the claim page hunting for // the manual "request a new code" button (delivery rides the report ACK). No-op for an unclaimed // customer (first provision). Also re-issue offsite credentials to the fresh box (F3) — the // one-time offsite password only ever reached the OLD controller, so the fresh one has no target. h.reissueOnReenroll(cc) + // R-1154: the record that the re-issues ran for this host record — an installer --resume that + // enrolls again before the new agent's first report must not re-issue a second time. + h.recordReenrollReissued(cc.CustomerID, hostID, "a new host record") w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusCreated) diff --git a/hub/internal/api/r1150_events_test.go b/hub/internal/api/r1150_events_test.go new file mode 100644 index 00000000..0ccbdf1a --- /dev/null +++ b/hub/internal/api/r1150_events_test.go @@ -0,0 +1,23 @@ +package api + +import ( + "testing" + + "gitea.dooplex.hu/admin/felhom-hub/internal/monitor" + "gitea.dooplex.hu/admin/felhom-hub/internal/notify" +) + +// R-1150 (v0.159.0): the refused whole-guest backup pair is operator-only AND hub-origin only — not in +// allowedEventTypes, so no box can POST one to /api/v1/event. The alarm says the guest's controller may be +// compromised; that controller holds the /event credential, and a type it could post it could fake, or post +// first to spend the operator cooldown and drown the real one (the R-895 reasoning). Both registers in one test. +func TestR1150_RefusedBackupEventsAreOperatorOnlyAndHubOrigin(t *testing.T) { + for _, et := range []string{monitor.EventWholeGuestBackupRefused, monitor.EventWholeGuestBackupRefusalCleared} { + if !notify.IsOperatorOnly(et) { + t.Errorf("%s is not operator-only", et) + } + if allowedEventTypes[et] { + t.Errorf("%s is in allowedEventTypes — a box could POST it", et) + } + } +} diff --git a/hub/internal/api/r1154_reenroll_test.go b/hub/internal/api/r1154_reenroll_test.go new file mode 100644 index 00000000..66f66632 --- /dev/null +++ b/hub/internal/api/r1154_reenroll_test.go @@ -0,0 +1,331 @@ +package api + +import ( + "context" + "database/sql" + "encoding/json" + "fmt" + "net/http" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/intent" + "gitea.dooplex.hu/admin/felhom-hub/internal/notify" + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// R-1154 (v0.159.0) — a reinstall while the customer's OLD host record still exists re-issues the claim code and +// the off-site login when the old box is gone (silent past the stale threshold on BOTH report streams, or switched +// off on purpose), and NEVER while the box is live. reenroll.go carries the rule; `05` §8 the design. + +const r1154Host = "c1-abc123" + +type r1154Fixture struct { + h *Handler + st *store.Store + db *sql.DB + mail *nullMailer + offsite int // offsite re-issuer calls + gen0 int // the claim generation before the enroll +} + +// newR1154 builds a CLAIMED customer c1 with an existing host record that has reported, the hub threshold at 45 m +// (the production value), a claim engine and a counting off-site re-issuer. +func newR1154(t *testing.T) *r1154Fixture { + t.Helper() + h, st, path := newTestHandler(t) + f := &r1154Fixture{h: h, st: st} + cc := &store.CustomerConfig{CustomerID: "c1", APIKey: "ckey", RetrievalPassword: "pass-phrase", + Email: "owner@example.hu", Domain: "example.hu", ConfigJSON: "{}"} + if err := st.SaveCustomerConfig(cc); err != nil { + t.Fatal(err) + } + f.mail = withClaimEngine(t, h, st) + if _, err := h.claimEngine.EnsureIssued(cc); err != nil { + t.Fatalf("EnsureIssued: %v", err) + } + if _, err := st.MarkClaimed("c1"); err != nil { + t.Fatalf("MarkClaimed: %v", err) + } + h.SetOffsiteReissuer(func(_ context.Context, id string) error { + if id == "c1" { + f.offsite++ + } + return nil + }) + h.SetStaleThreshold(45 * time.Minute) + if err := st.UpsertHost(&store.Host{HostID: r1154Host, CustomerID: "c1", APIKey: "old-host-key-0123456789"}); err != nil { + t.Fatal(err) + } + if err := st.SaveHostReport(r1154Host, "c1", []byte(`{"host_id":"`+r1154Host+`"}`), store.HostReportDenorm{}); err != nil { + t.Fatal(err) + } + db, err := sql.Open("sqlite", path) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { db.Close() }) + f.db = db + f.exec(t, `UPDATE hosts SET created_at = datetime('now', '-30 days') WHERE host_id = ?`, r1154Host) + f.mail.sends = 0 + f.gen0 = f.generation(t) + return f +} + +func (f *r1154Fixture) exec(t *testing.T, q string, args ...any) { + t.Helper() + if _, err := f.db.Exec(q, args...); err != nil { + t.Fatalf("%s: %v", q, err) + } +} + +func (f *r1154Fixture) hostReportedAgo(t *testing.T, d time.Duration) { + f.exec(t, `UPDATE hosts SET last_report_at = datetime('now', ?) WHERE host_id = ?`, fmt.Sprintf("-%d seconds", int(d.Seconds())), r1154Host) +} + +func (f *r1154Fixture) controllerReportedAgo(t *testing.T, d time.Duration) { + t.Helper() + if err := f.st.SaveReport("c1", []byte(`{"customer_id":"c1"}`)); err != nil { + t.Fatal(err) + } + f.exec(t, `UPDATE reports SET received_at = datetime('now', ?) WHERE customer_id = 'c1'`, fmt.Sprintf("-%d seconds", int(d.Seconds()))) +} + +func (f *r1154Fixture) generation(t *testing.T) int { + t.Helper() + cs, err := f.st.GetClaim("c1") + if err != nil || cs == nil { + t.Fatalf("GetClaim: %+v %v", cs, err) + } + return cs.Generation +} + +// enroll runs the installer's call and checks the reuse contract that must hold on EVERY branch: 200, the same +// host id and key. +func (f *r1154Fixture) enroll(t *testing.T) { + t.Helper() + rr := doEnroll(f.h, "c1", "pass-phrase") + if rr.Code != http.StatusOK { + t.Fatalf("enroll = %d, want 200 (reuse), body=%s", rr.Code, rr.Body.String()) + } + var got enrollResp + if err := json.Unmarshal(rr.Body.Bytes(), &got); err != nil { + t.Fatal(err) + } + if got.HostID != r1154Host || got.APIKey != "old-host-key-0123456789" { + t.Fatalf("reuse returned another credential: %s", got.HostID) + } + if n := countHostsForCustomer(t, f.st, "c1"); n != 1 { + t.Fatalf("host rows = %d, want 1 (the record is reused, never duplicated)", n) + } +} + +// reissued reports what the enroll did: claim generation moved, mails sent, off-site calls. +func (f *r1154Fixture) assertReissued(t *testing.T, want bool) { + t.Helper() + gen := f.generation(t) + if want { + if gen != f.gen0+1 || f.mail.sends != 1 || f.offsite != 1 { + t.Fatalf("want re-issued once: claim generation %d→%d, claim mails %d, off-site re-issues %d", f.gen0, gen, f.mail.sends, f.offsite) + } + return + } + if gen != f.gen0 || f.mail.sends != 0 || f.offsite != 0 { + t.Fatalf("want NOT re-issued: claim generation %d→%d, claim mails %d, off-site re-issues %d", f.gen0, gen, f.mail.sends, f.offsite) + } +} + +func (f *r1154Fixture) eventAt(t *testing.T, et string) time.Time { + t.Helper() + at, err := f.st.LastEventAt("c1", et) + if err != nil { + t.Fatal(err) + } + return at +} + +// The old box silent on both streams past the threshold → the enroll re-issues both, keeps the record, records it. +// +// COMPANION RED-PROOF (observed 2026-10-19): with the pre-fix reuse path restored (the existing host returned at +// once, reissueOnReuseIfReinstall not called) this fails with +// +// r1154_reenroll_test.go: want re-issued once: claim generation 1→1, claim mails 0, off-site re-issues 0 +// +// Restored. +func TestR1154_StaleOldHostIsReissued(t *testing.T) { + f := newR1154(t) + f.hostReportedAgo(t, 2*time.Hour) + f.controllerReportedAgo(t, 2*time.Hour) + f.enroll(t) + f.assertReissued(t, true) + if f.eventAt(t, eventHostReenrollReissued).IsZero() { + t.Error("host_reenroll_reissued not recorded") + } + if !f.eventAt(t, "claim_reissued_reenroll").After(time.Time{}) { + t.Error("claim_reissued_reenroll (the claim leg's own event) not recorded") + } +} + +// Switched off on purpose minutes ago (both report clocks still fresh) → re-issued: the box said it was going down. +// +// COMPANION RED-PROOF (observed 2026-10-19): with the deliberate-off branch removed from decideReenroll this fails +// with +// +// r1154_reenroll_test.go: want re-issued once: claim generation 1→1, claim mails 0, off-site re-issues 0 +// +// Restored. +func TestR1154_DeliberatelyOffOldHostIsReissued(t *testing.T) { + f := newR1154(t) + f.hostReportedAgo(t, 3*time.Minute) + f.controllerReportedAgo(t, 3*time.Minute) + off := time.Now().UTC().Add(-5 * time.Minute).Truncate(time.Second) + if _, err := f.st.SetHostDeliberateOff(r1154Host, &off); err != nil { + t.Fatal(err) + } + f.enroll(t) + f.assertReissued(t, true) +} + +// A LIVE box enrolling again (an installer --resume after its agent started, a repair re-run) keeps every +// credential: nothing re-issued, and the skip is recorded so a too-early reinstall is visible. Three ways to be +// live, each alone: the host report, the controller report (the codes live in the controller), the wait connection. +// +// COMPANION RED-PROOF (observed 2026-10-19): with decideReenroll replaced by an unconditional re-issue on the reuse +// path this fails with +// +// r1154_reenroll_test.go: host report fresh: want NOT re-issued: claim generation 1→2, claim mails 1, off-site re-issues 1 +// +// (and the same for the other two cases). Restored. +func TestR1154_LiveHostIsNotReissued(t *testing.T) { + cases := []struct { + name string + setup func(t *testing.T, f *r1154Fixture) + }{ + {"host report fresh", func(t *testing.T, f *r1154Fixture) { + f.hostReportedAgo(t, 2*time.Minute) + f.controllerReportedAgo(t, 2*time.Hour) + }}, + {"controller report fresh, agent silent", func(t *testing.T, f *r1154Fixture) { + f.hostReportedAgo(t, 2*time.Hour) + f.controllerReportedAgo(t, 5*time.Minute) + }}, + {"controller connected, both reports old", func(t *testing.T, f *r1154Fixture) { + f.hostReportedAgo(t, 2*time.Hour) + f.controllerReportedAgo(t, 2*time.Hour) + hub := intent.New() + f.h.SetIntentHub(hub) + hub.MarkWaitStart("c1") + }}, + {"controller connected, deliberately off", func(t *testing.T, f *r1154Fixture) { + f.hostReportedAgo(t, 2*time.Minute) + f.controllerReportedAgo(t, 2*time.Minute) + off := time.Now().UTC().Add(-time.Minute).Truncate(time.Second) + f.st.SetHostDeliberateOff(r1154Host, &off) + hub := intent.New() + f.h.SetIntentHub(hub) + hub.MarkWaitStart("c1") + }}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + f := newR1154(t) + c.setup(t, f) + f.enroll(t) + if gen := f.generation(t); gen != f.gen0 || f.mail.sends != 0 || f.offsite != 0 { + t.Fatalf("%s: want NOT re-issued: claim generation %d→%d, claim mails %d, off-site re-issues %d", + c.name, f.gen0, gen, f.mail.sends, f.offsite) + } + if f.eventAt(t, eventHostReenrollNotReissued).IsZero() { + t.Errorf("%s: the live skip was not recorded (host_reenroll_not_reissued)", c.name) + } + if !f.eventAt(t, eventHostReenrollReissued).IsZero() { + t.Errorf("%s: a re-issue record was written for a live box", c.name) + } + }) + } +} + +// Once per silence: an installer --resume of the same reinstall enrolls again before the new agent's first report +// — the old box is still "silent", and the re-issues must not run twice (the claim engine's single-shot rule). +// Then the new agent reports, goes silent again past the threshold (the next reinstall): re-issued again. +// +// COMPANION RED-PROOF (observed 2026-10-19): with the once-per-silence check removed from decideReenroll this fails +// with +// +// r1154_reenroll_test.go: the --resume re-issued again: claim generation 1→3, off-site re-issues 2 +// +// Restored. +func TestR1154_ResumeDoesNotReissueTwice(t *testing.T) { + f := newR1154(t) + f.hostReportedAgo(t, 2*time.Hour) + f.controllerReportedAgo(t, 2*time.Hour) + f.enroll(t) + f.enroll(t) // the --resume + if gen := f.generation(t); gen != f.gen0+1 || f.offsite != 1 { + t.Fatalf("the --resume re-issued again: claim generation %d→%d, off-site re-issues %d", f.gen0, gen, f.offsite) + } + // The record is backdated so the next report is strictly after it (second-granularity clocks). + f.exec(t, `UPDATE events SET created_at = datetime('now', '-3 hours') WHERE event_type = ?`, eventHostReenrollReissued) + f.exec(t, `UPDATE hosts SET last_report_at = datetime('now', '-2 hours', '-30 minutes') WHERE host_id = ?`, r1154Host) + f.enroll(t) + if gen := f.generation(t); gen != f.gen0+2 || f.offsite != 2 { + t.Fatalf("a later reinstall (the box reported after the last re-issue, then went silent) must re-issue: claim generation %d→%d, off-site re-issues %d", + f.gen0, gen, f.offsite) + } +} + +// Edges the code shows. A host record that never reported: a provisional record (operator /admin/hosts, no +// re-issue ran for it) → re-issued, like the mint it stands in for; a record the enroll itself minted (the +// re-issues ran at mint) → not again on the --resume. Not wired (no threshold) → nothing re-issued, the +// pre-v0.159.0 behaviour. +// +// COMPANION RED-PROOF (observed 2026-10-19): with the pre-fix reuse path restored, „provisional never reported" +// fails with „want re-issued once: claim generation 1→1, claim mails 0, off-site re-issues 0"; with the +// once-per-silence check removed, „minted by the enroll" fails with „off-site re-issues = 2, want 1"; with an +// unconditional re-issue, „threshold not wired" fails with „want NOT re-issued: claim generation 1→2, claim mails +// 1, off-site re-issues 1". Restored. +func TestR1154_NeverReportedAndUnwiredEdges(t *testing.T) { + t.Run("provisional never reported", func(t *testing.T) { + f := newR1154(t) + f.exec(t, `UPDATE hosts SET last_report_at = NULL WHERE host_id = ?`, r1154Host) + f.enroll(t) + f.assertReissued(t, true) + }) + t.Run("minted by the enroll, never reported", func(t *testing.T) { + h, st, _ := newTestHandler(t) + st.SaveCustomerConfig(&store.CustomerConfig{CustomerID: "c1", APIKey: "ckey", RetrievalPassword: "pass-phrase"}) + h.SetStaleThreshold(45 * time.Minute) + calls := 0 + h.SetOffsiteReissuer(func(context.Context, string) error { calls++; return nil }) + if rr := doEnroll(h, "c1", "pass-phrase"); rr.Code != http.StatusCreated { + t.Fatalf("first enroll = %d", rr.Code) + } + if rr := doEnroll(h, "c1", "pass-phrase"); rr.Code != http.StatusOK { + t.Fatalf("resume enroll = %d", rr.Code) + } + if calls != 1 { + t.Fatalf("off-site re-issues = %d, want 1 (at the mint only, not again on the --resume)", calls) + } + }) + t.Run("threshold not wired", func(t *testing.T) { + f := newR1154(t) + f.h.SetStaleThreshold(0) + f.hostReportedAgo(t, 2*time.Hour) + f.controllerReportedAgo(t, 2*time.Hour) + f.enroll(t) + f.assertReissued(t, false) + }) +} + +// Both R-1154 event types: operator-only, and hub-origin only — a box able to POST host_reenroll_reissued could +// plant the once-per-silence record and block the re-issue of its own reinstall. +func TestR1154_EventsAreOperatorOnlyAndHubOrigin(t *testing.T) { + for _, et := range []string{eventHostReenrollReissued, eventHostReenrollNotReissued} { + if !notify.IsOperatorOnly(et) { + t.Errorf("%s is not operator-only", et) + } + if allowedEventTypes[et] { + t.Errorf("%s is in allowedEventTypes — a box could POST it", et) + } + } +} diff --git a/hub/internal/api/reenroll.go b/hub/internal/api/reenroll.go new file mode 100644 index 00000000..bb8b7832 --- /dev/null +++ b/hub/internal/api/reenroll.go @@ -0,0 +1,148 @@ +package api + +import ( + "encoding/json" + "fmt" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/intent" + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// R-1154 (hub v0.159.0, brief 2026-10-19) — a reinstall while the customer's OLD host record still exists. +// +// POST /host-enroll is called by the installer (felhom-host-install.sh step 4) on EVERY pass, a --resume +// included, with the customer's retrieval passphrase. An existing host record is reused (same host id and +// key — a reinstalled box takes over the record). Until v0.159.0 the reuse path returned at once, so the +// clean-slate re-issues (reissueOnReenroll: the claim code, the off-site login) ran ONLY when the old host +// had been deleted first, and a plain reinstall skipped both in silence: the household stuck at the claim +// page, the off-site tier stuck (proven on demo-hp 2026-08-21). +// +// THE RULE (`05` §8). The reuse path re-issues when the OLD BOX IS GONE, judged by the hub's own clocks: +// +// 1. NEVER while the box's controller holds a wait connection right now (intent presence "connected"). +// 2. The household switched the box off on purpose (deliberate_off_at inside its hold) — the report clocks +// are not consulted: the box said it was going down. +// 3. Otherwise BOTH the agent's host report AND the controller's report are older than staleThreshold (or +// absent). The claim code and the off-site login live in the CONTROLLER, so a box whose agent is +// silent but whose controller still reports is live for this purpose (an installer re-run to repair a +// broken agent must not cost the household its codes). Same threshold, same comparison as hostStatus: +// no second definition of "live". +// 4. Once per silence: not when the re-issues already ran for this host since its last report (or, for a +// never-reported host, since the record was created) — a --resume of the same reinstall enrolls again +// before the new agent's first report. The record is the hub-origin info event host_reenroll_reissued. +// +// A live box that enrolls again (a --resume after its agent started, a repair re-run) gets its credentials +// and NOTHING re-issued; that skip is recorded as the info event host_reenroll_not_reissued, so a reinstall +// done within the threshold of the old box's last report is visible on the event list instead of silent. +// Not wired (staleThreshold 0) or a failed lookup: nothing is re-issued — the pre-v0.159.0 behaviour. +// +// Both event types are hub-origin only (not in allowedEventTypes — a box must not be able to plant the +// once-per-silence record) and operator-only. Pinned by r1154_reenroll_test.go. + +const ( + eventHostReenrollReissued = "host_reenroll_reissued" + eventHostReenrollNotReissued = "host_reenroll_not_reissued" +) + +// reenrollDecision is the verdict on a reuse-path enroll. +type reenrollDecision struct { + reissue bool // run the clean-slate re-issues + live bool // skipped because the box is live (recorded as an event) + why string // operator English, for the log line and the event +} + +func (h *Handler) decideReenroll(host *store.Host, now time.Time) reenrollDecision { + if h.staleThreshold <= 0 { + return reenrollDecision{why: "no stale threshold wired"} + } + if h.intentHub != nil { + if p := h.intentHub.Presence(host.CustomerID, now); p.State == intent.PresenceConnected { + return reenrollDecision{live: true, why: "the box's controller is connected to the hub right now"} + } + } + var why string + if store.DeliberateOffHeld(host.DeliberateOffAt, now) { + why = "the household switched the box off at " + host.DeliberateOffAt.UTC().Format("2006-01-02 15:04 UTC") + } else { + hostAge := "never reported" + if host.LastReportAt != nil { + age := now.Sub(*host.LastReportAt) + if age <= h.staleThreshold { + return reenrollDecision{live: true, why: "the host reported " + roundAge(age) + " ago"} + } + hostAge = "last reported " + roundAge(age) + " ago" + } + found, ctrlAt, _, err := h.store.LatestReportOffsitePresence(host.CustomerID) + if err != nil { + h.logger.Printf("[WARN] host-enroll: controller report lookup failed for %s: %v — nothing re-issued", host.CustomerID, err) + return reenrollDecision{why: "the controller report lookup failed"} + } + ctrlAge := "the controller never reported" + if found { + age := now.Sub(ctrlAt) + if age <= h.staleThreshold { + return reenrollDecision{live: true, why: "the controller reported " + roundAge(age) + " ago"} + } + ctrlAge = "the controller last reported " + roundAge(age) + " ago" + } + why = "the host " + hostAge + ", " + ctrlAge + " (stale threshold " + h.staleThreshold.String() + ")" + } + // Once per silence. + since := host.CreatedAt + if host.LastReportAt != nil && host.LastReportAt.After(since) { + since = *host.LastReportAt + } + last, err := h.store.LastEventAt(host.CustomerID, eventHostReenrollReissued) + if err != nil { + h.logger.Printf("[WARN] host-enroll: re-issue record lookup failed for %s: %v — nothing re-issued", host.CustomerID, err) + return reenrollDecision{why: "the re-issue record lookup failed"} + } + if !last.IsZero() && !last.Before(since) { + return reenrollDecision{why: "already re-issued at " + last.UTC().Format("2006-01-02 15:04:05 UTC") + " for this install"} + } + return reenrollDecision{reissue: true, why: why} +} + +// reissueOnReuseIfReinstall runs the clean-slate re-issues on the reuse path when decideReenroll says the +// old box is gone, and records the outcome. Best-effort, like reissueOnReenroll: it never fails the enroll. +func (h *Handler) reissueOnReuseIfReinstall(cc *store.CustomerConfig, host *store.Host, now time.Time) { + d := h.decideReenroll(host, now) + switch { + case d.reissue: + h.logger.Printf("[INFO] host re-enrolled: %s (customer %s) — re-issuing the claim code and the off-site login: %s", + host.HostID, cc.CustomerID, d.why) + h.reissueOnReenroll(cc) + h.recordReenrollReissued(cc.CustomerID, host.HostID, d.why) + case d.live: + h.logger.Printf("[INFO] host re-enrolled: %s (customer %s) — nothing re-issued, the box is live: %s", + host.HostID, cc.CustomerID, d.why) + msg := "Host " + host.HostID + " enrolled again while the box is live (" + d.why + ") — the claim code and " + + "the off-site login were NOT re-issued. If this box was reinstalled, send a new claim code and re-issue " + + "the off-site login from the customer page." + h.saveReenrollEvent(cc.CustomerID, eventHostReenrollNotReissued, msg, host.HostID, d.why) + default: + h.logger.Printf("[INFO] host re-enrolled: %s (customer %s) — nothing re-issued: %s", host.HostID, cc.CustomerID, d.why) + } +} + +// recordReenrollReissued stores the once-per-silence record (and the operator's line) after the re-issues ran. +func (h *Handler) recordReenrollReissued(customerID, hostID, why string) { + msg := "Host " + hostID + " enrolled — " + why + " — the clean-slate re-issues ran (the claim code if the " + + "customer is claimed, the off-site login if it is provisioned)" + h.saveReenrollEvent(customerID, eventHostReenrollReissued, msg, hostID, why) +} + +func (h *Handler) saveReenrollEvent(customerID, eventType, msg, hostID, why string) { + details, _ := json.Marshal(map[string]string{"host_id": hostID, "reason": why}) + if _, err := h.store.SaveEvent(customerID, eventType, "info", msg, string(details), "hub"); err != nil { + h.logger.Printf("[WARN] save %s for %s: %v", eventType, customerID, err) + } +} + +func roundAge(d time.Duration) string { + if d < time.Minute { + return fmt.Sprintf("%ds", int(d.Seconds())) + } + return d.Round(time.Minute).String() +} diff --git a/hub/internal/claim/engine.go b/hub/internal/claim/engine.go index 211e6c2c..3047b6ee 100644 --- a/hub/internal/claim/engine.go +++ b/hub/internal/claim/engine.go @@ -191,8 +191,9 @@ func (e *Engine) RequestReset(cc *store.CustomerConfig) error { // // No-op for an UNCLAIMED customer — that is the first-provision path where EnsureIssued already // owns the first code; re-enrolling before the first claim must NOT rotate. The CALLER guarantees -// single-shot by invoking this only on a genuinely fresh host record (the host-enroll mint path, -// which fires exactly once per reinstall). Returns (generation, reissued, error). +// single-shot: the host-enroll mint path (a fresh host record), and since hub v0.159.0 (R-1154) the +// reuse path once per silence of the old box (api/reenroll.go). Pinned by api/r1154_reenroll_test.go +// TestR1154_ResumeDoesNotReissueTwice. Returns (generation, reissued, error). func (e *Engine) ReissueForReenroll(cc *store.CustomerConfig) (gen int, reissued bool, err error) { cs, err := e.Store.GetClaim(cc.CustomerID) if err != nil { diff --git a/hub/internal/monitor/host_capability.go b/hub/internal/monitor/host_capability.go index cff7db2f..60a79aba 100644 --- a/hub/internal/monitor/host_capability.go +++ b/hub/internal/monitor/host_capability.go @@ -20,17 +20,49 @@ import ( // (a missing user-facing grant — the multi-drive-flapping class). Non-critical degradations ride // the report + the agent's own logs but do NOT alert the operator (avoid noise). The event is // attributed to the host's customer, so the existing operator notification UX picks it up -// unchanged. Customer is NEVER notified — internal capability health is operator-only. +// unchanged. Customer is NEVER notified — internal capability health is operator-only: both event +// types sit in notify.operatorOnlyEvents (R-1150, hub v0.159.0; before that only configuration kept +// them from a household). Pinned by notify/r1150_operator_only_test.go. +// +// R-1150 (hub v0.159.0) — ONE capability is not a grant. The agent's `localapi:backup-cadence-cap` +// (felhom-agent cmd/felhom-agent/main.go backupCapStatusName, R-973) is CRITICAL and "degraded" for +// 24 h after the agent REFUSED a whole-guest backup a guest asked for beyond its schedule. Reported +// as „agent privileged capability degraded" it read as a broken sudo grant. It is therefore a SECOND, +// independent state per host with its own event pair: +// +// - the grants (every other critical capability): ok↔degraded → agent_capability_degraded / +// agent_capability_recovered, exactly as before, the cadence cap never named in them; +// - the backup refusal (the cadence cap alone): ok→degraded → whole_guest_backup_refused (warning), +// degraded→ok → whole_guest_backup_refusal_cleared (info: recorded, never mailed). +// +// Two states rather than one verdict with a special case, because the two can change on the SAME +// report independently (a refusal while a grant is missing, a grant restored while the refusal +// stands): each fires on its own edge, so one report flip is at most one event per state — never +// two events for one state, and never a "grants restored" line when only the refusal window ended. +// Pinned by host_capability_test.go TestR1150_*. type HostCapabilityChecker struct { store *store.Store logger *log.Logger onEvent EventNotifyFunc mu sync.Mutex - states map[string]string // hostID → "ok" | "degraded" + states map[string]string // hostID → "ok" | "degraded" (the grants) + refusals map[string]string // hostID → "ok" | "degraded" (the backup cadence cap, R-1150) customerOf map[string]string // hostID → customerID } +// backupCadenceCapName is the agent's R-973 cadence-cap status name. It must equal felhom-agent's +// backupCapStatusName (cmd/felhom-agent/main.go) — the agent's status name stays (R-1150). +const backupCadenceCapName = "localapi:backup-cadence-cap" + +// R-1150 event types — hub-origin ONLY (deliberately not in api.allowedEventTypes: the guest's +// controller is the party this alarm suspects, and a box able to POST the type could fake it or, +// by spending the operator cooldown first, drown the real one) and operator-only. +const ( + EventWholeGuestBackupRefused = "whole_guest_backup_refused" + EventWholeGuestBackupRefusalCleared = "whole_guest_backup_refusal_cleared" +) + // NewHostCapabilityChecker creates the checker and seeds state from the current snapshots. No // events are generated during initialization (mirrors the staleness checker). func NewHostCapabilityChecker(s *store.Store, onEvent EventNotifyFunc, logger *log.Logger) *HostCapabilityChecker { @@ -39,6 +71,7 @@ func NewHostCapabilityChecker(s *store.Store, onEvent EventNotifyFunc, logger *l logger: logger, onEvent: onEvent, states: make(map[string]string), + refusals: make(map[string]string), customerOf: make(map[string]string), } rows, err := s.GetHostCapabilities() @@ -46,7 +79,7 @@ func NewHostCapabilityChecker(s *store.Store, onEvent EventNotifyFunc, logger *l logger.Printf("[WARN] Host capability checker: failed to seed states: %v", err) return cc } - var okCount, degCount int + var okCount, degCount, refusedCount int for _, row := range rows { if s.IsCustomerBlocked(row.CustomerID) { continue @@ -56,14 +89,20 @@ func NewHostCapabilityChecker(s *store.Store, onEvent EventNotifyFunc, logger *l // F2: seed only HEALTHY hosts. A host already degraded at (re)start is left UNSEEDED so the // first Check() observes oldState=="" and emits once — otherwise a box already broken when the // hub restarts would stay silent forever. The dispatcher's 1h cooldown dedups the re-emit. + // The same rule, separately, for the backup refusal (R-1150). if st == "degraded" { degCount++ - continue + } else { + cc.states[row.HostID] = st + okCount++ + } + if rst, _ := backupRefusalState(row.Capabilities); rst == "degraded" { + refusedCount++ + } else { + cc.refusals[row.HostID] = rst } - cc.states[row.HostID] = st - okCount++ } - logger.Printf("[INFO] Host capability checker initialized: %d ok, %d degraded (degraded left unseeded → first Check emits)", okCount, degCount) + logger.Printf("[INFO] Host capability checker initialized: %d ok, %d degraded, %d with a refused whole-guest backup (degraded/refused left unseeded → first Check emits)", okCount, degCount, refusedCount) return cc } @@ -83,37 +122,44 @@ func (cc *HostCapabilityChecker) Check() { seen[row.HostID] = true if cc.store.IsCustomerBlocked(row.CustomerID) { delete(cc.states, row.HostID) + delete(cc.refusals, row.HostID) continue } cc.customerOf[row.HostID] = row.CustomerID newState, names, features := capabilityState(row.Capabilities) - oldState := cc.states[row.HostID] - if oldState == "" { - // F2: first observation (incl. a hub restart). A healthy first-obs seeds silently; a host - // already DEGRADED at first sight emits once (else a host that was already broken when the - // hub (re)started would never alert). The dispatcher's 1h operator cooldown dedups a re-emit - // across a hub restart. - cc.states[row.HostID] = newState - if newState == "degraded" { - cc.emitTransition(row.HostID, row.CustomerID, "unknown", newState, names, features) - } - continue + if old, changed := step(cc.states, row.HostID, newState); changed { + cc.emitTransition(row.HostID, row.CustomerID, old, newState, names, features) } - if oldState == newState { - continue + refState, reason := backupRefusalState(row.Capabilities) + if old, changed := step(cc.refusals, row.HostID, refState); changed { + cc.emitRefusal(row.HostID, row.CustomerID, old, refState, reason) } - cc.states[row.HostID] = newState - cc.emitTransition(row.HostID, row.CustomerID, oldState, newState, names, features) } - for id := range cc.states { - if !seen[id] { - delete(cc.states, id) + for _, m := range []map[string]string{cc.states, cc.refusals} { + for id := range m { + if !seen[id] { + delete(m, id) + } } } } +// step records newState for hostID in m and reports whether an event is due, with the state it left +// ("unknown" on a first observation). F2: a first observation (incl. a hub restart) seeds a healthy +// state silently, and a state already DEGRADED at first sight is due once (else a host that was already +// broken when the hub (re)started would never alert). The dispatcher's 1h operator cooldown dedups a +// re-emit across a hub restart. A steady state is never due. +func step(m map[string]string, hostID, newState string) (oldState string, due bool) { + oldState = m[hostID] + m[hostID] = newState + if oldState == "" { + return "unknown", newState == "degraded" + } + return oldState, oldState != newState +} + // GetState returns the current capability state for a host ("unknown" if unseen). func (cc *HostCapabilityChecker) GetState(hostID string) string { cc.mu.Lock() @@ -125,11 +171,27 @@ func (cc *HostCapabilityChecker) GetState(hostID string) string { return s } +// GetBackupRefusalState returns the host's whole-guest backup refusal state (R-1150): "ok", "degraded" +// (the agent refused one inside its window), or "unknown" if unseen. +func (cc *HostCapabilityChecker) GetBackupRefusalState(hostID string) string { + cc.mu.Lock() + defer cc.mu.Unlock() + s := cc.refusals[hostID] + if s == "" { + return "unknown" + } + return s +} + // capabilityState reduces a snapshot to "ok"/"degraded" plus the degraded CRITICAL capability names // + features (for the event detail). An empty/absent snapshot (pre-v0.44.0 agent) is "ok" — an old -// agent can't trip a false alert. +// agent can't trip a false alert. The backup cadence cap is NOT a grant and is left to +// backupRefusalState (R-1150). func capabilityState(caps []store.CapabilityStatus) (state string, names, features []string) { for _, c := range caps { + if c.Name == backupCadenceCapName { + continue + } if c.Critical && c.Status == "degraded" { names = append(names, c.Name) features = append(features, c.Feature) @@ -175,6 +237,50 @@ func (cc *HostCapabilityChecker) emitTransition(hostID, customerID, oldState, ne } } +// backupRefusalState reads the agent's cadence-cap status (R-1150): "degraded" with the agent's own +// reason while it reports a refused whole-guest backup, else "ok" (absent on an agent without the cap). +// The agent always reports the cap as critical; Critical is required here as it is for every alarm. +func backupRefusalState(caps []store.CapabilityStatus) (state, reason string) { + for _, c := range caps { + if c.Name == backupCadenceCapName && c.Critical && c.Status == "degraded" { + return "degraded", c.Reason + } + } + return "ok", "" +} + +// emitRefusal raises the operator's alarm for a refused whole-guest backup (R-1150) — the agent turned +// down a backup the guest asked for beyond its schedule — and records its end. +func (cc *HostCapabilityChecker) emitRefusal(hostID, customerID, oldState, newState, reason string) { + var eventType, severity, message string + switch { + case newState == "degraded": + eventType = EventWholeGuestBackupRefused + severity = "warning" + message = "Host " + hostID + ": a whole-guest backup was refused — the guest asked for more whole-guest " + + "backups than its schedule allows. Its controller may be misbehaving or compromised: check the guest." + case newState == "ok" && oldState == "degraded": + eventType = EventWholeGuestBackupRefusalCleared + severity = "info" + message = "Host " + hostID + ": the agent no longer reports a refused whole-guest backup (its refusal window has ended)" + default: + return + } + details, _ := json.Marshal(map[string]any{ + "host_id": hostID, + "capability": backupCadenceCapName, + "agent_reason": reason, + }) + cc.logger.Printf("[INFO] Host capability: %s backup refusal %s → %s (%s)", hostID, oldState, newState, eventType) + if _, err := cc.store.SaveEvent(customerID, eventType, severity, message, string(details), "hub"); err != nil { + cc.logger.Printf("[WARN] Failed to save backup refusal event for %s: %v", hostID, err) + return + } + if cc.onEvent != nil { + cc.onEvent(customerID, eventType, severity, message, string(details), "hub") + } +} + func dedup(in []string) []string { seen := make(map[string]bool, len(in)) var out []string diff --git a/hub/internal/monitor/host_capability_test.go b/hub/internal/monitor/host_capability_test.go index 6efd134a..da244cad 100644 --- a/hub/internal/monitor/host_capability_test.go +++ b/hub/internal/monitor/host_capability_test.go @@ -4,6 +4,7 @@ import ( "io" "log" "path/filepath" + "strings" "testing" "gitea.dooplex.hu/admin/felhom-hub/internal/store" @@ -130,3 +131,170 @@ func TestHostCapabilityChecker_F2_BornDegradedAlerts(t *testing.T) { t.Fatalf("steady degraded must not re-emit, got %v", events) } } + +// ── R-1150: a refused whole-guest backup has its own alarm ─────────────────────────────────────── + +const capCadenceRefused = `[{"name":"guest-init-pid","feature":"drive-gate","critical":true,"status":"ok"},` + + `{"name":"localapi:backup-cadence-cap","feature":"whole-guest restore points","critical":true,"status":"degraded",` + + `"reason":"refused a whole-guest backup of guest 9202 on target \"local\" at 2026-10-19T03:15:00Z — more than the schedule allows; the guest's controller may be misbehaving or compromised: check the guest"}]` +const capCadenceOK = `[{"name":"guest-init-pid","feature":"drive-gate","critical":true,"status":"ok"},` + + `{"name":"localapi:backup-cadence-cap","feature":"whole-guest restore points","critical":true,"status":"ok"}]` +const capGrantAndCadenceDegraded = `[{"name":"guest-init-pid","feature":"drive-gate guest-sees","critical":true,"status":"degraded","reason":"sudo policy denied"},` + + `{"name":"localapi:backup-cadence-cap","feature":"whole-guest restore points","critical":true,"status":"degraded","reason":"refused a whole-guest backup of guest 9202"}]` +const capGrantDegradedCadenceOK = `[{"name":"guest-init-pid","feature":"drive-gate guest-sees","critical":true,"status":"degraded","reason":"sudo policy denied"},` + + `{"name":"localapi:backup-cadence-cap","feature":"whole-guest restore points","critical":true,"status":"ok"}]` + +type capEvent struct{ eventType, severity, message, details string } + +func recordCapEvents(evs *[]capEvent) EventNotifyFunc { + return func(_, eventType, severity, message, details, _ string) { + *evs = append(*evs, capEvent{eventType, severity, message, details}) + } +} + +func capTypes(evs []capEvent) []string { + out := make([]string, len(evs)) + for i, e := range evs { + out[i] = e.eventType + } + return out +} + +// The cadence cap ALONE degraded → exactly one whole_guest_backup_refused (warning, plain text, the agent's +// reason in the details) and NOT agent_capability_degraded; steady → nothing; the window ending → one +// whole_guest_backup_refusal_cleared (info) and NOT agent_capability_recovered. +// +// COMPANION RED-PROOF (observed 2026-10-19): with the pre-fix shape restored (capabilityState counting the +// cadence cap like any grant, no separate refusal state) this test fails with +// +// host_capability_test.go: cadence cap alone → want [whole_guest_backup_refused], got [agent_capability_degraded] +// +// Restored. +func TestR1150_CadenceCapAloneRaisesWholeGuestBackupRefused(t *testing.T) { + st := newCapStore(t) + st.SaveHostReport("h1", "c1", reportWith(capCadenceOK), store.HostReportDenorm{}) + var evs []capEvent + cc := NewHostCapabilityChecker(st, recordCapEvents(&evs), log.New(io.Discard, "", 0)) + + st.SaveHostReport("h1", "c1", reportWith(capCadenceRefused), store.HostReportDenorm{}) + cc.Check() + if got := capTypes(evs); len(got) != 1 || got[0] != "whole_guest_backup_refused" { + t.Fatalf("cadence cap alone → want [whole_guest_backup_refused], got %v", got) + } + e := evs[0] + if e.severity != "warning" { + t.Errorf("severity = %q, want warning", e.severity) + } + if !strings.Contains(e.message, "a whole-guest backup was refused") || !strings.Contains(e.message, "check the guest") { + t.Errorf("message does not say what happened in plain words: %q", e.message) + } + if strings.Contains(e.message, "privileged capability") { + t.Errorf("message still reads as a broken grant: %q", e.message) + } + if !strings.Contains(e.details, "guest 9202") || !strings.Contains(e.details, "localapi:backup-cadence-cap") { + t.Errorf("details must carry the agent's reason and the capability name: %s", e.details) + } + if cc.GetState("h1") != "ok" { + t.Errorf("the grants state = %s, want ok (the cadence cap is not a grant)", cc.GetState("h1")) + } + if cc.GetBackupRefusalState("h1") != "degraded" { + t.Errorf("refusal state = %s, want degraded", cc.GetBackupRefusalState("h1")) + } + // The event is STORED too (the operator's event list), under the new type. + if at, err := st.LastEventAt("c1", "whole_guest_backup_refused"); err != nil || at.IsZero() { + t.Errorf("whole_guest_backup_refused not stored: at=%v err=%v", at, err) + } + + st.SaveHostReport("h1", "c1", reportWith(capCadenceRefused), store.HostReportDenorm{}) + cc.Check() + if len(evs) != 1 { + t.Fatalf("steady refusal must not re-emit, got %v", capTypes(evs)) + } + + st.SaveHostReport("h1", "c1", reportWith(capCadenceOK), store.HostReportDenorm{}) + cc.Check() + if got := capTypes(evs); len(got) != 2 || got[1] != "whole_guest_backup_refusal_cleared" { + t.Fatalf("refusal window ended → want [.. whole_guest_backup_refusal_cleared], got %v", got) + } + if evs[1].severity != "info" { + t.Errorf("cleared severity = %q, want info (recorded, never mailed)", evs[1].severity) + } +} + +// Another critical cap degraded → agent_capability_degraded unchanged (the cadence cap never named in it); +// both degraded on ONE report → exactly one event per state; the grant restored while the refusal stands → +// agent_capability_recovered alone; nothing for a steady state. +// +// COMPANION RED-PROOF (observed 2026-10-19): with the pre-fix shape restored the both-degraded report emits a +// single event naming both caps and the test fails with +// +// host_capability_test.go: grant + cadence on one report → want [agent_capability_degraded whole_guest_backup_refused], got [agent_capability_degraded] +// +// Restored. +func TestR1150_OtherCapStillRaisesCapabilityDegraded(t *testing.T) { + st := newCapStore(t) + st.SaveHostReport("h1", "c1", reportWith(capCadenceOK), store.HostReportDenorm{}) + var evs []capEvent + cc := NewHostCapabilityChecker(st, recordCapEvents(&evs), log.New(io.Discard, "", 0)) + + // A grant alone: the old event, unchanged text. + st.SaveHostReport("h1", "c1", reportWith(capGrantDegradedCadenceOK), store.HostReportDenorm{}) + cc.Check() + if got := capTypes(evs); len(got) != 1 || got[0] != "agent_capability_degraded" { + t.Fatalf("grant alone → want [agent_capability_degraded], got %v", got) + } + if !strings.Contains(evs[0].message, "agent privileged capability degraded — guest-init-pid") { + t.Errorf("agent_capability_degraded text changed: %q", evs[0].message) + } + // Back to clean, then both on one report. + st.SaveHostReport("h1", "c1", reportWith(capCadenceOK), store.HostReportDenorm{}) + cc.Check() + evs = nil + st.SaveHostReport("h1", "c1", reportWith(capGrantAndCadenceDegraded), store.HostReportDenorm{}) + cc.Check() + got := capTypes(evs) + if len(got) != 2 || got[0] != "agent_capability_degraded" || got[1] != "whole_guest_backup_refused" { + t.Fatalf("grant + cadence on one report → want [agent_capability_degraded whole_guest_backup_refused], got %v", got) + } + if strings.Contains(evs[0].message, "backup-cadence-cap") || strings.Contains(evs[0].details, "backup-cadence-cap") { + t.Errorf("agent_capability_degraded must not name the cadence cap: %q / %s", evs[0].message, evs[0].details) + } + // Steady: nothing. + st.SaveHostReport("h1", "c1", reportWith(capGrantAndCadenceDegraded), store.HostReportDenorm{}) + cc.Check() + if len(evs) != 2 { + t.Fatalf("steady both-degraded must not re-emit, got %v", capTypes(evs)) + } + // The grant restored, the refusal still standing: only the grants recover. + st.SaveHostReport("h1", "c1", reportWith(capCadenceRefused), store.HostReportDenorm{}) + cc.Check() + if got := capTypes(evs); len(got) != 3 || got[2] != "agent_capability_recovered" { + t.Fatalf("grant restored, refusal standing → want [.. agent_capability_recovered], got %v", got) + } + if cc.GetBackupRefusalState("h1") != "degraded" { + t.Errorf("refusal state = %s, want still degraded", cc.GetBackupRefusalState("h1")) + } +} + +// F2 for the refusal: a host already reporting a refused backup when the hub (re)starts alerts ONCE on the +// first Check (not at construction), and never again while steady. +// +// COMPANION RED-PROOF (observed 2026-10-19): with the pre-fix shape restored it fails with +// +// host_capability_test.go: born-refused → want exactly [whole_guest_backup_refused], got [agent_capability_degraded] +// +// Restored. +func TestR1150_BornRefusedAlertsOnce(t *testing.T) { + st := newCapStore(t) + st.SaveHostReport("h1", "c1", reportWith(capCadenceRefused), store.HostReportDenorm{}) + var evs []capEvent + cc := NewHostCapabilityChecker(st, recordCapEvents(&evs), log.New(io.Discard, "", 0)) + if len(evs) != 0 { + t.Fatalf("construction must not emit, got %v", capTypes(evs)) + } + cc.Check() + cc.Check() + if got := capTypes(evs); len(got) != 1 || got[0] != "whole_guest_backup_refused" { + t.Fatalf("born-refused → want exactly [whole_guest_backup_refused], got %v", got) + } +} diff --git a/hub/internal/notify/dispatcher.go b/hub/internal/notify/dispatcher.go index e9c4bf84..6c802671 100644 --- a/hub/internal/notify/dispatcher.go +++ b/hub/internal/notify/dispatcher.go @@ -880,6 +880,22 @@ var operatorOnlyEvents = map[string]bool{ // definition: someone pushed a catalog change asking for container rights the box does not give. The household can // do nothing; its app page says the app keeps its last good definition. Registered in the same commit that mints it. "app_definition_refused": true, + // R-1150 (v0.159.0). The agent refused a whole-guest backup a guest asked for beyond its schedule (the R-973 + // cadence cap): the guest's controller may be misbehaving or compromised — the operator's question, never the + // household's. Hub-origin only (monitor.HostCapabilityChecker; deliberately NOT in api.allowedEventTypes — the + // suspected controller must not be able to post or pre-empt it). Listed in the SAME commit that mints them. + "whole_guest_backup_refused": true, + "whole_guest_backup_refusal_cleared": true, + // R-1150 (v0.159.0): the agent's privileged-capability pair. Its checker has said „Customer is NEVER notified" + // since v0.44.0, and only configuration (enabled_events) made that true — the R-97c shape. Now structural. + "agent_capability_degraded": true, + "agent_capability_recovered": true, + // R-1154 (v0.159.0): a host-enroll that reused the old host record — re-issued the clean-slate credentials, or + // did NOT because the box is live. Info (recorded, never mailed), host ids and report ages: the operator's + // record. Hub-origin only (api/reenroll.go; not in api.allowedEventTypes — a box must not plant the + // once-per-silence record). Listed in the SAME commit that mints them. + "host_reenroll_reissued": true, + "host_reenroll_not_reissued": true, } // IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the diff --git a/hub/internal/notify/r1150_operator_only_test.go b/hub/internal/notify/r1150_operator_only_test.go new file mode 100644 index 00000000..6922b70e --- /dev/null +++ b/hub/internal/notify/r1150_operator_only_test.go @@ -0,0 +1,43 @@ +package notify + +import "testing" + +// R-1150 (v0.159.0) — the refused whole-guest backup pair and the agent's capability pair never reach a +// household, under the configuration that would break it: the customer HAS the type in enabled_events and +// HAS an e-mail. The operator still gets the warning. (The api package pins the other register: none of +// these is in allowedEventTypes, so no box can POST one.) +// +// COMPANION RED-PROOF (observed 2026-10-19): with the four lines removed from operatorOnlyEvents this fails +// with +// +// r1150_operator_only_test.go: whole_guest_backup_refused: the household was mailed (customer@example.com) — enabled_events opted it in +// r1150_operator_only_test.go: agent_capability_degraded: the household was mailed (customer@example.com) — enabled_events opted it in +// +// Restored. +func TestR1150_RefusedBackupAndCapabilityEventsNeverReachTheCustomer(t *testing.T) { + for _, et := range []string{"whole_guest_backup_refused", "agent_capability_degraded"} { + st := opOnlyStore(t) + if err := st.SaveNotificationPrefs("c1", "customer@example.com", []string{et}, 6); err != nil { + t.Fatalf("SaveNotificationPrefs: %v", err) + } + rec := &sentTo{} + d := opOnlyDispatcher(t, st, rec) + d.ProcessEvent("c1", et, "warning", "Host c1-abc123: test", `{"host_id":"c1-abc123"}`, "hub") + var customer, operator bool + for _, to := range rec.to { + customer = customer || to == "customer@example.com" + operator = operator || to == "operator@felhom.eu" + } + if customer { + t.Errorf("%s: the household was mailed (customer@example.com) — enabled_events opted it in", et) + } + if !operator { + t.Errorf("%s: the operator was not mailed — the guard must be customer-only", et) + } + } + for _, et := range []string{"whole_guest_backup_refusal_cleared", "agent_capability_recovered"} { + if !operatorOnlyEvents[et] { + t.Errorf("%s must be operator-only (the pair's other edge)", et) + } + } +}