diff --git a/REUSE.md b/REUSE.md index 299d70b5..224e570e 100644 --- a/REUSE.md +++ b/REUSE.md @@ -124,6 +124,7 @@ | `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. | | `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). | | `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. | +| `monitor.HeldAppsChecker` + `store.UpsertHeldApp`/`ClearHeldApp`/`ListHeldApps` + `HeldAppGrace`/`HeldAppWhoClears` (D53, unreleased) | hub/internal/monitor/held_apps.go · hub/internal/store/held_apps.go · hub/internal/web/held_apps.go | `NewHeldAppsChecker(st, graceOverride, onEvent, logger)`; `.Check()` on the 60 s sweep | `08` §10: the boxes' held apps — the mirror (fed by `app_held`/`app_hold_cleared` in `handleEvent`), the „Held apps" list on the host page and the Hosts page, one operator reminder after the grace per reason, then weekly | **The hub never clears a hold** — a row goes only on the box's `app_hold_cleared`. The reminder is recorded BEFORE it is sent (a restart cannot mail twice); a row the box stopped repeating (> 50 h) is listed, never reminded. `app_held_reminder` is operator-only and per-app cooldown. `graceOverride` is a TEST configuration only (`alerting.held_app_grace_override`). Pinned by `TestD53_*` (monitor, api, web, notify) | | `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. **R-86 (2026-08-03): the window is PER TIER, not one constant.** `restoreProvenWindow(tier, observed, ok)` = `clamp(4 × max(observed, declared), floor 7d, cap 12d)`, where `declared` is that tier's own backup-freshness threshold (`backupStaleAfter` 26 h / `offsiteBackupStaleAfter` 8 d — reuse those, never a second opinion) and `observed` comes from `observedArchiveIntervals` over the retained window. **Observation may only WIDEN**: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. `assessRestoreProven` takes the window as an argument and **every reason string names it** (R-100's corollary). | | `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. | | `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. | diff --git a/documentation/audits/d53-held-apps-2026-10-26/redproof-hub.txt b/documentation/audits/d53-held-apps-2026-10-26/redproof-hub.txt new file mode 100644 index 00000000..2be7fabc --- /dev/null +++ b/documentation/audits/d53-held-apps-2026-10-26/redproof-hub.txt @@ -0,0 +1,7 @@ +RED internal/notify/dispatcher.go | "app_held_reminder": true, } // IsOperatorOnly | ['--- FAIL: TestD53_TheHeldAppReminderMailsTheOperatorOnly (0.04s)'] +RED internal/api/handler.go | h.recordHeldApp(payload.CustomerID, payload.EventType, paylo | ['--- FAIL: TestD53_TheBoxsHeldAppsAreMirrored (0.05s)'] +RED internal/monitor/held_apps.go | if age < hc.grace(h.Reason) { return } | ['--- FAIL: TestD53_OneReminderAfterTheGraceThenWeekly (0.04s)', '--- FAIL: TestD53_TheGraceFollowsTheReason (0.17s)'] +RED internal/monitor/held_apps.go | if !h.RemindedAt.IsZero() && now.Sub(h.RemindedAt) < heldApp | ['--- FAIL: TestD53_OneReminderAfterTheGraceThenWeekly (0.03s)'] +RED internal/web/hosts.go | "HeldApps": heldApps, | [] +RED internal/web/templates/host_detail_body.html | {{.WhoClears}}{{if .Reminded}} | ['--- FAIL: TestD53_HeldAppsListOnTheHostPageAndTheHostsPage (0.05s)'] +RED internal/web/hosts.go | HeldApps emptied (compiling mutation) | TestD53_HeldAppsListOnTheHostPageAndTheHostsPage diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index fa6a4100..84c5882b 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,24 @@ +## Unreleased (next: v0.173.0) — D53: a held app reaches the operator (`08` §10) + +Operator ruling D53 (2026-10-26, `09` §3 decision 265): operator-only reminders first. Pairs with controller `5bd03ad` +(`internal/holdreport`); a box on an older controller sends nothing new, and its list stays empty. + +- `internal/api/handler.go`: `app_held` and `app_hold_cleared` in `allowedEventTypes`; `handleEvent` → `recordHeldApp` + mirrors them in the new `held_apps` table (`internal/store/held_apps.go`: one row per customer + stack; a repeat moves + `last_seen` only; a new reason or since starts the row over; the clear deletes it). +- `internal/monitor/held_apps.go` `HeldAppsChecker` (60 s sweep, one pass per 15 min): ONE operator reminder + `app_held_reminder` (warning) after the grace per reason — `putback_pending` 6 h, `restore_failed`/`restore_mixed` 1 day, + `update_failed`/`unhealthy_stop` 3 days — then weekly while it lasts; recorded before it is sent; a row the box stopped + repeating (> 50 h) is not reminded. `alerting.held_app_grace_override` (a duration) replaces every grace — a TEST + configuration only, logged at start. +- `internal/notify`: all three types operator-only; `app_held_reminder` per-app cooldown and a subject that names the app; + hu + en `mail.event.*` lines (the coupling rule) and their six mail goldens. +- Web: a „Held apps" list on the host page (app, reason, since, the box's sentence, who may clear it) and on the Hosts + page (every box, linked to its host); the empty sentence when none. The hub never clears a hold. +- Tests `TestD53_*`: the mirror (api), the reminder after the grace / weekly / per reason / none after a clear / none for + a stale row / a new hold starts over (monitor), operator only and per app (notify), both pages with and without a held + app (web). Red-proved by six mutations (`documentation/audits/d53-held-apps-2026-10-26/redproof-hub.txt`). + ## v0.172.0 (2026-10-26) — a fresh install's whole-guest off-site set-up never fails silently (R-1295) — RELEASED 2026-10-26 13:08 UTC on the operator's yes (image sha256:7c8d129a…, from `232f42d2`, CI 2366; rc.1 ran the live proof) - **Cause (measured, `documentation/audits/r1295-pbsdr-selfcheck-2026-10-26/`):** PBS's proxy caches permissions for up to diff --git a/hub/cmd/hub/main.go b/hub/cmd/hub/main.go index 65db6154..c6a9feba 100644 --- a/hub/cmd/hub/main.go +++ b/hub/cmd/hub/main.go @@ -72,6 +72,9 @@ type Config struct { // the host-root thresholds above so a single storage's policy can be tuned separately. StorageFillWarnPercent float64 `yaml:"storage_fill_warn_percent"` StorageFillCritPercent float64 `yaml:"storage_fill_crit_percent"` + // HeldAppGraceOverride (D53) replaces every reason's reminder grace when set (a Go duration, e.g. "3m"). A TEST + // configuration only — the live proof shortens the wait; production leaves it empty (`08` §10's per-reason grace). + HeldAppGraceOverride string `yaml:"held_app_grace_override"` // Offsite POOL-BOX aggregate thresholds (v0.64.0, R-5). Fill = box used vs capacity (percent); // oversub = Σ(shared soft quotas)/capacity (ratio). Empty/0/invalid → defaults 80/90/2.0 (the // checker normalizes). Threshold VALUES are Claude's encoding of the starter suggestion — Viktor's @@ -899,6 +902,16 @@ func main() { // durable (events table), so a hub restart neither floods nor silently re-heals. // R-523: the agent's in-guest controller supervisor → controller_restarted_by_agent / controller_crashloop. controllerSupervisorChecker := monitor.NewControllerSupervisorChecker(dataStore, dispatcher.ProcessEvent, logger) + // D53 (`08` §10): a held app reaches the OPERATOR — one reminder after the grace per reason, then weekly. + var heldGrace time.Duration + if v := cfg.Alerting.HeldAppGraceOverride; v != "" { + if d, err := time.ParseDuration(v); err == nil && d > 0 { + heldGrace = d + } else { + logger.Printf("[WARN] alerting.held_app_grace_override %q is not a positive duration — ignored", v) + } + } + heldAppsChecker := monitor.NewHeldAppsChecker(dataStore, heldGrace, dispatcher.ProcessEvent, logger) offsiteDeliveryChecker := monitor.NewOffsiteDeliveryChecker(dataStore, offsiteHealReissuer, dispatcher.ProcessEvent, logger) // R-242 second half (S4, decision 199): published golden vs vouched golden. Needs the registry creds // (the same Gitea client as the Day-0 dropdowns); without them the check is off and says so. @@ -931,6 +944,7 @@ func main() { restoreTestChecker.Check() // R-85: restore-test failure + per-tier staleness offsiteDeliveryChecker.Check() controllerSupervisorChecker.Check() + heldAppsChecker.Check() // D53: throttled to one pass per 15 min if goldenVouchChecker != nil { goldenVouchChecker.Check() // R-242: throttled to one Gitea listing per 15 min } diff --git a/hub/internal/api/d53_held_apps_test.go b/hub/internal/api/d53_held_apps_test.go new file mode 100644 index 00000000..56c99a69 --- /dev/null +++ b/hub/internal/api/d53_held_apps_test.go @@ -0,0 +1,45 @@ +package api + +import ( + "testing" + + "gitea.dooplex.hu/admin/felhom-hub/internal/notify" +) + +// D53 (`08` §10): a box's app_held is accepted and mirrored (stack, reason, since); its daily repeat keeps ONE row; its +// app_hold_cleared removes the row. Both types are allow-listed AND operator-only (the v0.78.0 pair, checked together). +func TestD53_TheBoxsHeldAppsAreMirrored(t *testing.T) { + h, st, _ := newTestHandler(t) + for _, et := range []string{"app_held", "app_hold_cleared"} { + if !allowedEventTypes[et] { + t.Fatalf("%s is not allow-listed — POST /event would 400", et) + } + if !notify.IsOperatorOnly(et) { + t.Fatalf("%s is not operator-only", et) + } + } + if !notify.IsOperatorOnly("app_held_reminder") || allowedEventTypes["app_held_reminder"] { + t.Fatal("app_held_reminder must be operator-only and hub-origin (a box must not post the reminder)") + } + held := `{"customer_id":"c1","event_type":"app_held","severity":"info","message":"App kimai is held", + "details":{"stack_name":"kimai","reason":"unhealthy_stop","since":"2026-10-14T08:00:00Z","why":"stopped by the box"}}` + for i := 0; i < 2; i++ { // the first report and the daily repeat + if rr := do(h, "POST", "/event", globalKey, held); rr.Code != 200 { + t.Fatalf("app_held: %d %s", rr.Code, rr.Body.String()) + } + } + rows, err := st.ListHeldApps("c1") + if err != nil || len(rows) != 1 { + t.Fatalf("want one held row, got %+v (%v)", rows, err) + } + if r := rows[0]; r.Stack != "kimai" || r.Reason != "unhealthy_stop" || r.Since.Format("2006-01-02T15:04:05Z07:00") != "2026-10-14T08:00:00Z" || r.Why == "" { + t.Fatalf("row: %+v", r) + } + cleared := `{"customer_id":"c1","event_type":"app_hold_cleared","severity":"info","message":"App kimai is no longer held","details":{"stack_name":"kimai"}}` + if rr := do(h, "POST", "/event", globalKey, cleared); rr.Code != 200 { + t.Fatalf("app_hold_cleared: %d", rr.Code) + } + if rows, _ := st.ListHeldApps("c1"); len(rows) != 0 { + t.Fatalf("the cleared hold is still listed: %+v", rows) + } +} diff --git a/hub/internal/api/handler.go b/hub/internal/api/handler.go index 35a90d02..307bc145 100644 --- a/hub/internal/api/handler.go +++ b/hub/internal/api/handler.go @@ -2262,6 +2262,12 @@ var allowedEventTypes = map[string]bool{ // v0.123.0 (decision 28, controller v0.269.0): the box stopped an app that kept crashing / ran out // of memory. Household + operator; hu AND en `mail.event.*`; per-app cooldown on both legs. "app_stopped_unhealthy": true, + // D53 (`08` §10, controller with internal/holdreport): the box's held apps — app_held for each hold in place (at once, + // then daily), app_hold_cleared when one is gone. Severity info: never mailed on receipt; the hub mirrors them in + // held_apps (handleEvent → recordHeldApp) and the reminder checker mails the OPERATOR after a grace. Operator-only + // (notify.operatorOnlyEvents); hu AND en `mail.event.*` lines exist only because the coupling rule needs them. + "app_held": true, + "app_hold_cleared": true, // R-871 (v0.134.0, controller v0.295.0, `09` decision 109): the box was off at its backup time and made the // missed backups when it came back. Info — recorded on the box's timeline, never mailed. @@ -2520,6 +2526,12 @@ func (h *Handler) handleEvent(w http.ResponseWriter, r *http.Request) { h.logger.Printf("[INFO] Event from %s: %s (%s) — %s", payload.CustomerID, payload.EventType, payload.Severity, payload.Message) + // D53: mirror the box's held apps. A failure is logged and the event still dispatched — the box repeats + // app_held daily, so the next one heals the mirror. + if payload.EventType == "app_held" || payload.EventType == "app_hold_cleared" { + h.recordHeldApp(payload.CustomerID, payload.EventType, payload.Details) + } + // Dispatch notifications (non-blocking) if h.dispatcher != nil { go h.dispatcher.ProcessBoxEvent(payload.CustomerID, payload.EventType, payload.Severity, @@ -3189,3 +3201,40 @@ func (h *Handler) checkTunnel(hostID, custID, status, detail string) { go h.dispatcher.ProcessEvent(custID, typ, sev, msg, string(details), "hub") } } + +// recordHeldApp applies one app_held / app_hold_cleared to the held_apps mirror (D53, `08` §10). The hub never +// clears a hold on a box; it only forgets one the box says is gone. +func (h *Handler) recordHeldApp(customerID, eventType string, raw json.RawMessage) { + var d struct { + StackName string `json:"stack_name"` + Reason string `json:"reason"` + Since string `json:"since"` + Why string `json:"why"` + } + if err := json.Unmarshal(raw, &d); err != nil || d.StackName == "" { + h.logger.Printf("[WARN] [held-apps] %s from %s has no readable stack_name — not recorded", eventType, customerID) + return + } + if eventType == "app_hold_cleared" { + had, err := h.store.ClearHeldApp(customerID, d.StackName) + if err != nil { + h.logger.Printf("[WARN] [held-apps] clear %s/%s: %v", customerID, d.StackName, err) + return + } + h.logger.Printf("[INFO] [held-apps] %s/%s no longer held (a row existed: %v)", customerID, d.StackName, had) + return + } + var since time.Time + if d.Since != "" { + if t, err := time.Parse(time.RFC3339, d.Since); err == nil { + since = t + } else { + h.logger.Printf("[WARN] [held-apps] %s/%s: since %q is not RFC3339 — the hub's first sighting is the clock", customerID, d.StackName, d.Since) + } + } + if err := h.store.UpsertHeldApp(customerID, d.StackName, d.Reason, since, d.Why, time.Now()); err != nil { + h.logger.Printf("[WARN] [held-apps] record %s/%s: %v", customerID, d.StackName, err) + return + } + h.logger.Printf("[DEBUG] [held-apps] %s/%s held (%s since %s)", customerID, d.StackName, d.Reason, d.Since) +} diff --git a/hub/internal/i18n/locales/en.json b/hub/internal/i18n/locales/en.json index 859c28c2..0a1942bf 100644 --- a/hub/internal/i18n/locales/en.json +++ b/hub/internal/i18n/locales/en.json @@ -89,6 +89,9 @@ "mail.event.app_update_undone": "%s: the update did not work; the app runs on its previous version", "mail.event.app_update_held": "%s: the app is stopped and needs a restore", "mail.event.app_stopped_unhealthy": "%s: the app was stopped because it kept crashing", + "mail.event.app_held": "An app is stopped and waits for a person to act", + "mail.event.app_hold_cleared": "The app is no longer held", + "mail.event.app_held_reminder": "An app is still held", "bind.resend.hint": "If this was your link and your box is not linked yet, we send a fresh link to the e-mail address you gave Felhom.", "bind.resend.button": "Send me a new link", "bind.resend.newer": "If you have a newer mail from Felhom, use the link in that one.", diff --git a/hub/internal/i18n/locales/hu.json b/hub/internal/i18n/locales/hu.json index 68faf892..0a25a929 100644 --- a/hub/internal/i18n/locales/hu.json +++ b/hub/internal/i18n/locales/hu.json @@ -89,6 +89,9 @@ "mail.event.app_update_undone": "%s: a frissítés nem sikerült, az alkalmazás a korábbi változattal fut", "mail.event.app_update_held": "%s: az alkalmazás leállítva, visszaállítás szükséges", "mail.event.app_stopped_unhealthy": "%s: az alkalmazást leállítottuk, mert újra és újra összeomlott", + "mail.event.app_held": "Egy alkalmazás le van állítva, és emberi beavatkozásra vár", + "mail.event.app_hold_cleared": "Az alkalmazás leállítása megszűnt", + "mail.event.app_held_reminder": "Egy alkalmazás még mindig le van állítva", "bind.resend.hint": "Ha ez a te linked volt, és a dobozod még nincs összekötve, új linket küldünk arra az e-mail címre, amelyet a Felhomnál megadtál.", "bind.resend.button": "Új linket kérek", "bind.resend.newer": "Ha van újabb leveled a Felhomtól, abban lévő linket használd.", diff --git a/hub/internal/monitor/held_apps.go b/hub/internal/monitor/held_apps.go new file mode 100644 index 00000000..c8d0ba7f --- /dev/null +++ b/hub/internal/monitor/held_apps.go @@ -0,0 +1,174 @@ +package monitor + +import ( + "encoding/json" + "fmt" + "log" + "sync" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// ── D53 (`08` §10, R-1291) — A HELD APP REACHES THE OPERATOR ────────────────────────────────────── +// +// A held app is stopped on purpose and stays stopped until a person acts. demo-hp's Kimai sat held for 11 days +// (2026-10-14 → 10-26) and nobody was told after the first minute. The box now reports its holds (app_held / +// app_hold_cleared, severity info — nothing is mailed on receipt); this checker turns a hold that LASTS into ONE +// operator mail after a grace that follows the reason, then one a week while it lasts. +// +// THE GRACE (`08` §10): putback_pending 6 h — it should have cleared itself, so it is stuck; restore_failed and +// restore_mixed 1 day — only a person can act; update_failed and unhealthy_stop 3 days — the household can act first. +// An unknown reason gets 1 day. The clock is the hold's own time (the box's), else when the hub first heard of it. +// +// OPERATOR ONLY (D53): app_held_reminder is in the dispatcher's operatorOnlyEvents, so no household is ever mailed. +// THE HUB NEVER CLEARS A HOLD — a row goes only when the box sends app_hold_cleared. +// +// A ROW THE BOX STOPPED CONFIRMING is not reminded: the box repeats app_held daily, so a row older than heldAppStale +// belongs to a box that is down or gone, which its own staleness alarms report. The list still shows it. +// +// The reminder is recorded (MarkHeldAppReminded) BEFORE it is sent, so a hub restart cannot mail twice. +// Pinned by held_apps_test.go. +const ( + eventHeldAppReminder = "app_held_reminder" + heldAppRepeat = 7 * 24 * time.Hour + heldAppStale = 50 * time.Hour + heldAppCheckEvery = 15 * time.Minute +) + +// HeldAppGrace returns the wait before the first reminder for a reason (`08` §10). +func HeldAppGrace(reason string) time.Duration { + switch reason { + case "putback_pending": + return 6 * time.Hour + case "update_failed", "unhealthy_stop": + return 72 * time.Hour + default: // restore_failed, restore_mixed, and a reason this hub does not know yet + return 24 * time.Hour + } +} + +// HeldAppWhoClears says who may lift a hold, in the operator's words. +func HeldAppWhoClears(reason string) string { + switch reason { + case "update_failed": + return "the household, by restoring the app from its backup" + case "unhealthy_stop": + return "the household's Start (one more try)" + case "putback_pending": + return "nobody — it clears itself when the drive with the pre-restore copy comes back" + default: + return "a person only (support: the controller's -clear-restore-hold)" + } +} + +type heldAppsStore interface { + ListHeldApps(customerID string) ([]store.HeldApp, error) + MarkHeldAppReminded(customerID, stack, reason string, since, at time.Time) error + SaveEvent(customerID, eventType, severity, message, detailsJSON, source string) (int64, error) +} + +// HeldAppsChecker sends the operator's reminders for held apps. +type HeldAppsChecker struct { + store heldAppsStore + onEvent EventNotifyFunc + logger *log.Logger + now func() time.Time + every time.Duration + // graceOverride, when > 0, replaces every reason's grace. A TEST configuration only (the live proof shortens + // the wait); production leaves it 0. Logged at start when set. + graceOverride time.Duration + + mu sync.Mutex + lastRun time.Time +} + +// NewHeldAppsChecker builds the checker. graceOverride 0 = the per-reason grace. +func NewHeldAppsChecker(st heldAppsStore, graceOverride time.Duration, onEvent EventNotifyFunc, logger *log.Logger) *HeldAppsChecker { + if graceOverride > 0 && logger != nil { + logger.Printf("[WARN] Held apps: the reminder grace is OVERRIDDEN to %s for every reason — a test configuration", graceOverride) + } + return &HeldAppsChecker{store: st, onEvent: onEvent, logger: logger, now: time.Now, every: heldAppCheckEvery, graceOverride: graceOverride} +} + +func (hc *HeldAppsChecker) grace(reason string) time.Duration { + if hc.graceOverride > 0 { + return hc.graceOverride + } + return HeldAppGrace(reason) +} + +// Check runs on the hub's 60 s sweep and throttles itself to one pass per heldAppCheckEvery. +func (hc *HeldAppsChecker) Check() { + hc.mu.Lock() + defer hc.mu.Unlock() + now := hc.now() + every := hc.every + if hc.graceOverride > 0 && hc.graceOverride < every { + every = hc.graceOverride // a test grace of minutes must not wait a 15-minute pass + } + if !hc.lastRun.IsZero() && now.Sub(hc.lastRun) < every { + return + } + hc.lastRun = now + held, err := hc.store.ListHeldApps("") + if err != nil { + hc.logger.Printf("[WARN] Held apps: list failed: %v — no reminder this pass", err) + return + } + for _, h := range held { + hc.checkOne(h, now) + } +} + +func (hc *HeldAppsChecker) checkOne(h store.HeldApp, now time.Time) { + if now.Sub(h.LastSeen) > heldAppStale { + hc.logger.Printf("[DEBUG] Held apps: %s/%s not confirmed since %s — no reminder (the box is not reporting)", + h.CustomerID, h.Stack, h.LastSeen.Format(time.RFC3339)) + return + } + anchor := h.Since + if anchor.IsZero() { + anchor = h.FirstSeen + } + age := now.Sub(anchor) + if age < hc.grace(h.Reason) { + return + } + if !h.RemindedAt.IsZero() && now.Sub(h.RemindedAt) < heldAppRepeat { + return + } + if err := hc.store.MarkHeldAppReminded(h.CustomerID, h.Stack, h.Reason, h.Since, now); err != nil { + hc.logger.Printf("[WARN] Held apps: could not record the reminder for %s/%s (%v) — not sending, so a restart cannot mail twice", + h.CustomerID, h.Stack, err) + return + } + repeat := !h.RemindedAt.IsZero() + msg := fmt.Sprintf("Customer %s: the app %s has been held for %s (%s): %s. Who may clear it: %s. The hub never clears a hold.", + h.CustomerID, h.Stack, roundAge(age), h.Reason, h.Why, HeldAppWhoClears(h.Reason)) + if repeat { + msg += " Weekly reminder: it is still held." + } + details, _ := json.Marshal(map[string]any{ + "customer_id": h.CustomerID, "stack_name": h.Stack, "reason": h.Reason, + "since": anchor.UTC().Format(time.RFC3339), "why": h.Why, "repeat": repeat, + }) + hc.logger.Printf("[INFO] Held apps: reminder for %s/%s (%s, held %s, repeat=%v)", h.CustomerID, h.Stack, h.Reason, roundAge(age), repeat) + if _, err := hc.store.SaveEvent(h.CustomerID, eventHeldAppReminder, "warning", msg, string(details), "hub"); err != nil { + hc.logger.Printf("[WARN] Failed to save %s for %s: %v", eventHeldAppReminder, h.CustomerID, err) + return + } + if hc.onEvent != nil { + hc.onEvent(h.CustomerID, eventHeldAppReminder, "warning", msg, string(details), "hub") + } +} + +func roundAge(d time.Duration) string { + if d >= 48*time.Hour { + return fmt.Sprintf("%d days", int(d/(24*time.Hour))) + } + if d >= time.Hour { + return d.Round(time.Hour).String() + } + return d.Round(time.Minute).String() +} diff --git a/hub/internal/monitor/held_apps_test.go b/hub/internal/monitor/held_apps_test.go new file mode 100644 index 00000000..b94bf943 --- /dev/null +++ b/hub/internal/monitor/held_apps_test.go @@ -0,0 +1,123 @@ +package monitor + +import ( + "io" + "log" + "path/filepath" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +type heldEv struct{ customer, typ, sev, details string } + +func heldSetup(t *testing.T) (*store.Store, *HeldAppsChecker, *[]heldEv, *time.Time) { + t.Helper() + st, err := store.New(filepath.Join(t.TempDir(), "h.db"), log.New(io.Discard, "", 0)) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { st.Close() }) + var evs []heldEv + now := time.Date(2026, 10, 26, 12, 0, 0, 0, time.UTC) + hc := NewHeldAppsChecker(st, 0, func(c, typ, sev, msg, details, src string) { + evs = append(evs, heldEv{c, typ, sev, details}) + }, log.New(io.Discard, "", 0)) + hc.now = func() time.Time { return now } + hc.every = 0 + return st, hc, &evs, &now +} + +// The reminder after the grace (D53): a crash-loop hold waits 3 days, then ONE warning reminder to the operator, then +// nothing until a week has passed, then one more. +func TestD53_OneReminderAfterTheGraceThenWeekly(t *testing.T) { + st, hc, evs, now := heldSetup(t) + since := now.Add(-time.Hour) + if err := st.UpsertHeldApp("c1", "kimai", "unhealthy_stop", since, "stopped by the box", *now); err != nil { + t.Fatal(err) + } + hc.Check() + if len(*evs) != 0 { + t.Fatalf("a reminder inside the 3-day grace: %+v", *evs) + } + for _, step := range []time.Duration{72 * time.Hour, time.Hour, 24 * time.Hour} { + *now = now.Add(step) + _ = st.UpsertHeldApp("c1", "kimai", "unhealthy_stop", since, "stopped by the box", *now) // the box's daily repeat + hc.Check() + } + if len(*evs) != 1 || (*evs)[0].typ != "app_held_reminder" || (*evs)[0].sev != "warning" || (*evs)[0].customer != "c1" { + t.Fatalf("want exactly one warning app_held_reminder after the grace, got %+v", *evs) + } + *now = now.Add(7 * 24 * time.Hour) + _ = st.UpsertHeldApp("c1", "kimai", "unhealthy_stop", since, "stopped by the box", *now) + hc.Check() + if len(*evs) != 2 { + t.Fatalf("want the weekly repeat, got %d events", len(*evs)) + } +} + +// No reminder after a clear: the box's app_hold_cleared removes the row, and nothing is mailed for it. +func TestD53_NoReminderAfterAClear(t *testing.T) { + st, hc, evs, now := heldSetup(t) + _ = st.UpsertHeldApp("c1", "db", "restore_failed", now.Add(-2*time.Hour), "held after a failed restore", *now) + if ok, err := st.ClearHeldApp("c1", "db"); err != nil || !ok { + t.Fatalf("clear: %v %v", ok, err) + } + *now = now.Add(30 * 24 * time.Hour) + hc.Check() + if len(*evs) != 0 { + t.Fatalf("a cleared hold was reminded: %+v", *evs) + } + if rows, _ := st.ListHeldApps(""); len(rows) != 0 { + t.Fatalf("the list still holds a cleared app: %+v", rows) + } +} + +// The grace follows the reason (`08` §10): put-back 6 h, restore 1 day, update / crash loop 3 days. +func TestD53_TheGraceFollowsTheReason(t *testing.T) { + cases := map[string]time.Duration{"putback_pending": 6 * time.Hour, "restore_failed": 24 * time.Hour, + "restore_mixed": 24 * time.Hour, "update_failed": 72 * time.Hour, "unhealthy_stop": 72 * time.Hour} + for reason, grace := range cases { + st, hc, evs, now := heldSetup(t) + _ = st.UpsertHeldApp("c1", "a", reason, *now, "x", *now) + *now = now.Add(grace - time.Minute) + _ = st.UpsertHeldApp("c1", "a", reason, now.Add(-(grace - time.Minute)), "x", *now) + hc.Check() + if len(*evs) != 0 { + t.Errorf("%s: reminded before its grace %s", reason, grace) + } + *now = now.Add(2 * time.Minute) + hc.Check() + if len(*evs) != 1 { + t.Errorf("%s: no reminder just after its grace %s (%d events)", reason, grace, len(*evs)) + } + } +} + +// A box that stopped confirming its hold (down or gone) is not reminded about it; its own staleness alarms speak. +func TestD53_AStaleRowIsNotReminded(t *testing.T) { + st, hc, evs, now := heldSetup(t) + _ = st.UpsertHeldApp("c1", "a", "restore_failed", now.Add(-10*24*time.Hour), "x", now.Add(-3*24*time.Hour)) + hc.Check() + if len(*evs) != 0 { + t.Fatalf("a row the box stopped confirming was reminded: %+v", *evs) + } +} + +// A new hold on the same app (another reason) starts its own grace: the old reminder does not carry over. +func TestD53_ANewHoldOnTheSameAppStartsOver(t *testing.T) { + st, hc, evs, now := heldSetup(t) + _ = st.UpsertHeldApp("c1", "a", "putback_pending", now.Add(-7*time.Hour), "x", *now) + hc.Check() + if len(*evs) != 1 { + t.Fatalf("want the put-back reminder, got %d", len(*evs)) + } + _ = st.UpsertHeldApp("c1", "a", "restore_mixed", *now, "y", *now) + *now = now.Add(25 * time.Hour) + _ = st.UpsertHeldApp("c1", "a", "restore_mixed", now.Add(-25*time.Hour), "y", *now) + hc.Check() + if len(*evs) != 2 { + t.Fatalf("the new hold's reminder was swallowed by the old one's week: %d events", len(*evs)) + } +} diff --git a/hub/internal/notify/d53_held_reminder_test.go b/hub/internal/notify/d53_held_reminder_test.go new file mode 100644 index 00000000..1d1288e6 --- /dev/null +++ b/hub/internal/notify/d53_held_reminder_test.go @@ -0,0 +1,43 @@ +package notify + +import ( + "strings" + "testing" +) + +// D53 (operator ruling 2026-10-26: operator-only reminders first): the held-app reminder reaches the OPERATOR — one mail +// whose subject names the app — and never the household, even with every event switched on (the R-97c configuration). +// The box's own app_held / app_hold_cleared are info: they mail nobody on either leg. +func TestD53_TheHeldAppReminderMailsTheOperatorOnly(t *testing.T) { + st := opOnlyStore(t) + if err := st.SaveNotificationPrefs("c1", "customer@example.com", []string{"app_held_reminder", "app_held", "app_hold_cleared", "backup_failed"}, 6); err != nil { + t.Fatal(err) + } + var to, subjects []string + d := opOnlyDispatcher(t, st, &sentTo{}) + d.sendEmailFn = func(addr, subject, body string, headers map[string]string) error { + to = append(to, addr) + subjects = append(subjects, subject) + return nil + } + d.ProcessBoxEvent("c1", "app_held", "info", "App kimai is held: …", "", `{"stack_name":"kimai","reason":"unhealthy_stop"}`, "controller") + d.ProcessBoxEvent("c1", "app_hold_cleared", "info", "App kimai is no longer held", "", `{"stack_name":"kimai"}`, "controller") + if len(to) != 0 { + t.Fatalf("the box's held-app events mailed %v — they must be recorded, never mailed", to) + } + d.ProcessEvent("c1", "app_held_reminder", "warning", "Customer c1: the app kimai has been held for 3 days …", + `{"stack_name":"kimai","reason":"unhealthy_stop"}`, "hub") + d.ProcessEvent("c1", "app_held_reminder", "warning", "Customer c1: the app paperless has been held for 3 days …", + `{"stack_name":"paperless","reason":"update_failed"}`, "hub") + for _, a := range to { + if a == "customer@example.com" { + t.Fatal("the household was mailed an operator-only held-app reminder") + } + } + if len(to) != 2 { + t.Fatalf("want 2 operator mails (two apps, per-app cooldown), got %d: %v", len(to), to) + } + if !strings.Contains(subjects[0], "kimai is still held") || !strings.Contains(subjects[1], "paperless is still held") { + t.Fatalf("each subject must name its app: %q", subjects) + } +} diff --git a/hub/internal/notify/dispatcher.go b/hub/internal/notify/dispatcher.go index 64a21766..6ec7cdb8 100644 --- a/hub/internal/notify/dispatcher.go +++ b/hub/internal/notify/dispatcher.go @@ -501,6 +501,8 @@ var perAppCooldownEvents = map[string]bool{ // v0.169.0 (the upload watch): one app's unexplained upload or CPU, no digest — two hacked apps are two alarms. "app_unexplained_upload": true, "app_unexplained_cpu": true, + // D53: one held app's reminder, no digest — two apps held on one box are two reminders. + "app_held_reminder": true, } // perAppCustomerCooldownEvents is the CUSTOMER-leg sibling of perAppCooldownEvents (v0.120.0). The @@ -938,6 +940,11 @@ var operatorOnlyEvents = map[string]bool{ // R-1295 (v0.172.0): the WG-registration hook could not set up a box's whole-guest off-site tier (web/pbsdr.go). // Host ids and an instruction only the operator can carry out. Hub-origin only. "pbsdr_provision_failed": true, + // D53 (`08` §10, operator ruling 2026-10-26: operator-only reminders first). The box's held-app events are info + // and never routed; the hub's reminder after the grace is the operator's mail. Listed in the SAME commit. + "app_held": true, + "app_hold_cleared": true, + "app_held_reminder": true, } // IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the diff --git a/hub/internal/notify/r389_cooldown_grain_test.go b/hub/internal/notify/r389_cooldown_grain_test.go index d90c6275..eacd7d45 100644 --- a/hub/internal/notify/r389_cooldown_grain_test.go +++ b/hub/internal/notify/r389_cooldown_grain_test.go @@ -73,7 +73,8 @@ func TestR389_TheAllowListHasExactlyOneMember(t *testing.T) { // v0.123.0 once more (decision 28): an unhealthy stop is one app's event with no digest. // v0.150.0 once more (R-1007): an app's repair after an unclean boot is one app's event with no digest. // v0.169.0 once more (the upload watch): an app's unexplained upload or CPU is one app's event with no digest. - want := []string{"app_hold_no_whole_copy", "app_oom_storm", "app_repaired_after_crash", "app_start_failed", "app_stopped_unhealthy", "app_unexplained_cpu", "app_unexplained_upload", "app_update_held", "app_update_undone"} + // D53 once more: a held app's reminder is one app's event with no digest — two held apps are two reminders. + want := []string{"app_held_reminder", "app_hold_no_whole_copy", "app_oom_storm", "app_repaired_after_crash", "app_start_failed", "app_stopped_unhealthy", "app_unexplained_cpu", "app_unexplained_upload", "app_update_held", "app_update_undone"} ok := len(perAppCooldownEvents) == len(want) for _, w := range want { ok = ok && perAppCooldownEvents[w] @@ -83,7 +84,7 @@ func TestR389_TheAllowListHasExactlyOneMember(t *testing.T) { for k := range perAppCooldownEvents { got = append(got, k) } - t.Fatalf("perAppCooldownEvents = %v, want exactly [app_hold_no_whole_copy app_oom_storm app_repaired_after_crash app_start_failed app_stopped_unhealthy app_unexplained_cpu app_unexplained_upload app_update_held app_update_undone]. Adding a member is the "+ + t.Fatalf("perAppCooldownEvents = %v, want exactly [app_held_reminder app_hold_no_whole_copy app_oom_storm app_repaired_after_crash app_start_failed app_stopped_unhealthy app_unexplained_cpu app_unexplained_upload app_update_held app_update_undone]. Adding a member is the "+ "fenced act: the backup family's cooldown is coarse ON PURPOSE (R-97a, R-182) so one full "+ "disk sends one digest, not one mail per app. Read the fence before widening this.", got) } diff --git a/hub/internal/notify/templates.go b/hub/internal/notify/templates.go index e73a2ab7..14d04439 100644 --- a/hub/internal/notify/templates.go +++ b/hub/internal/notify/templates.go @@ -48,6 +48,12 @@ func FormatOperatorEmail(customerID, eventType, severity, message, detailsJSON s if eventType == "pbsdr_provision_failed" { subject = fmt.Sprintf("[Felhom] %s %s: the whole-box off-site copy (PBS DR) could not be set up — save the customer config to retry", icon, customerID) } + // D53: the subject names the held app, so two reminders are told apart in the inbox. + if eventType == "app_held_reminder" { + if st := stackNameOf(detailsJSON); st != "" { + subject = fmt.Sprintf("[Felhom] %s %s: the app %s is still held — a person must act", icon, customerID, st) + } + } now := nowFn().In(budapest).Format("2006-01-02 15:04 MST") body := fmt.Sprintf(`Customer: %s diff --git a/hub/internal/notify/testdata/mail_goldens/en/customer_event_app_held.txt b/hub/internal/notify/testdata/mail_goldens/en/customer_event_app_held.txt new file mode 100644 index 00000000..01c48321 --- /dev/null +++ b/hub/internal/notify/testdata/mail_goldens/en/customer_event_app_held.txt @@ -0,0 +1,18 @@ +SUBJECT: [Felhom] Warning: An app is stopped and waits for a person to act +--- +Dear Customer, + +Your Felhom system sent the following notification: + +An app is stopped and waits for a person to act + +Details: +- Server: demo-fixture +- Time: 2026-01-15 10:30 +- Level: Warning +- Type: app_held + +If you have any questions, contact your operator. + +Best regards, +Felhom.eu monitoring diff --git a/hub/internal/notify/testdata/mail_goldens/en/customer_event_app_held_reminder.txt b/hub/internal/notify/testdata/mail_goldens/en/customer_event_app_held_reminder.txt new file mode 100644 index 00000000..df2a5c53 --- /dev/null +++ b/hub/internal/notify/testdata/mail_goldens/en/customer_event_app_held_reminder.txt @@ -0,0 +1,18 @@ +SUBJECT: [Felhom] Warning: An app is still held +--- +Dear Customer, + +Your Felhom system sent the following notification: + +An app is still held + +Details: +- Server: demo-fixture +- Time: 2026-01-15 10:30 +- Level: Warning +- Type: app_held_reminder + +If you have any questions, contact your operator. + +Best regards, +Felhom.eu monitoring diff --git a/hub/internal/notify/testdata/mail_goldens/en/customer_event_app_hold_cleared.txt b/hub/internal/notify/testdata/mail_goldens/en/customer_event_app_hold_cleared.txt new file mode 100644 index 00000000..a1df068f --- /dev/null +++ b/hub/internal/notify/testdata/mail_goldens/en/customer_event_app_hold_cleared.txt @@ -0,0 +1,18 @@ +SUBJECT: [Felhom] Warning: The app is no longer held +--- +Dear Customer, + +Your Felhom system sent the following notification: + +The app is no longer held + +Details: +- Server: demo-fixture +- Time: 2026-01-15 10:30 +- Level: Warning +- Type: app_hold_cleared + +If you have any questions, contact your operator. + +Best regards, +Felhom.eu monitoring diff --git a/hub/internal/notify/testdata/mail_goldens/hu/customer_event_app_held.txt b/hub/internal/notify/testdata/mail_goldens/hu/customer_event_app_held.txt new file mode 100644 index 00000000..4f2524c3 --- /dev/null +++ b/hub/internal/notify/testdata/mail_goldens/hu/customer_event_app_held.txt @@ -0,0 +1,18 @@ +SUBJECT: [Felhom] Figyelmeztetés: Egy alkalmazás le van állítva, és emberi beavatkozásra vár +--- +Kedves Ügyfél! + +A Felhom rendszered a következő értesítést küldte: + +Egy alkalmazás le van állítva, és emberi beavatkozásra vár + +Részletek: +- Szerver: demo-fixture +- Időpont: 2026-01-15 10:30 +- Szint: Figyelmeztetés +- Típus: app_held + +Ha kérdésed van, vedd fel a kapcsolatot az üzemeltetővel. + +Üdvözlettel, +Felhom.eu monitoring diff --git a/hub/internal/notify/testdata/mail_goldens/hu/customer_event_app_held_reminder.txt b/hub/internal/notify/testdata/mail_goldens/hu/customer_event_app_held_reminder.txt new file mode 100644 index 00000000..9e8b9151 --- /dev/null +++ b/hub/internal/notify/testdata/mail_goldens/hu/customer_event_app_held_reminder.txt @@ -0,0 +1,18 @@ +SUBJECT: [Felhom] Figyelmeztetés: Egy alkalmazás még mindig le van állítva +--- +Kedves Ügyfél! + +A Felhom rendszered a következő értesítést küldte: + +Egy alkalmazás még mindig le van állítva + +Részletek: +- Szerver: demo-fixture +- Időpont: 2026-01-15 10:30 +- Szint: Figyelmeztetés +- Típus: app_held_reminder + +Ha kérdésed van, vedd fel a kapcsolatot az üzemeltetővel. + +Üdvözlettel, +Felhom.eu monitoring diff --git a/hub/internal/notify/testdata/mail_goldens/hu/customer_event_app_hold_cleared.txt b/hub/internal/notify/testdata/mail_goldens/hu/customer_event_app_hold_cleared.txt new file mode 100644 index 00000000..397a56b7 --- /dev/null +++ b/hub/internal/notify/testdata/mail_goldens/hu/customer_event_app_hold_cleared.txt @@ -0,0 +1,18 @@ +SUBJECT: [Felhom] Figyelmeztetés: Az alkalmazás leállítása megszűnt +--- +Kedves Ügyfél! + +A Felhom rendszered a következő értesítést küldte: + +Az alkalmazás leállítása megszűnt + +Részletek: +- Szerver: demo-fixture +- Időpont: 2026-01-15 10:30 +- Szint: Figyelmeztetés +- Típus: app_hold_cleared + +Ha kérdésed van, vedd fel a kapcsolatot az üzemeltetővel. + +Üdvözlettel, +Felhom.eu monitoring diff --git a/hub/internal/store/held_apps.go b/hub/internal/store/held_apps.go new file mode 100644 index 00000000..6ba7f104 --- /dev/null +++ b/hub/internal/store/held_apps.go @@ -0,0 +1,124 @@ +package store + +import ( + "fmt" + "time" +) + +// ── D53 (`08` §10, R-1291): the held apps a box reports ─────────────────────────────────────────── +// +// A box (controller ≥ the D53 release) sends app_held for every hold in place — at once, then once a day — and +// app_hold_cleared when one is gone. This table is the hub's mirror of that: one row per (customer, stack). The hub +// NEVER clears a hold on a box; a row goes only when the box says the hold is gone. The reminder checker +// (monitor.HeldAppsChecker) reads it and records here when it mailed, so a hub restart neither re-mails nor forgets. +// All times are stored as RFC3339 UTC text. + +// HeldApp is one held app as the hub knows it. +type HeldApp struct { + CustomerID string + Stack string + Reason string // restore_failed, restore_mixed, update_failed, unhealthy_stop, putback_pending + Since time.Time // the hold's own time (the box's) + Why string // the box's one-line sentence: what put it there and what lifts it + FirstSeen time.Time // when the hub first heard of this hold + LastSeen time.Time // the newest app_held for it — the box repeats it daily + RemindedAt time.Time // the last operator reminder; zero = none yet +} + +func (s *Store) migrateHeldApps() error { + _, err := s.db.Exec(` + CREATE TABLE IF NOT EXISTS held_apps ( + customer_id TEXT NOT NULL, + stack TEXT NOT NULL, + reason TEXT NOT NULL DEFAULT '', + since TEXT NOT NULL DEFAULT '', + why TEXT NOT NULL DEFAULT '', + first_seen TEXT NOT NULL, + last_seen TEXT NOT NULL, + reminded_at TEXT NOT NULL DEFAULT '', + PRIMARY KEY (customer_id, stack) + );`) + return err +} + +func fmtT(t time.Time) string { + if t.IsZero() { + return "" + } + return t.UTC().Format(time.RFC3339) +} + +func parseT(s string) time.Time { + if s == "" { + return time.Time{} + } + return parseSQLiteTime(s) +} + +// UpsertHeldApp records an app_held. A repeat of the same hold (same reason and since) only moves last_seen; a NEW +// hold on the stack (another reason or another since) starts over — first_seen now and no reminder yet — so its own +// grace runs from the start. +func (s *Store) UpsertHeldApp(customerID, stack, reason string, since time.Time, why string, now time.Time) error { + if customerID == "" || stack == "" { + return fmt.Errorf("held app: empty customer or stack") + } + res, err := s.db.Exec(`UPDATE held_apps SET last_seen = ?, why = ? + WHERE customer_id = ? AND stack = ? AND reason = ? AND since = ?`, + fmtT(now), why, customerID, stack, reason, fmtT(since)) + if err != nil { + return err + } + if n, _ := res.RowsAffected(); n > 0 { + return nil + } + _, err = s.db.Exec(`INSERT INTO held_apps (customer_id, stack, reason, since, why, first_seen, last_seen, reminded_at) + VALUES (?, ?, ?, ?, ?, ?, ?, '') + ON CONFLICT(customer_id, stack) DO UPDATE SET reason = excluded.reason, since = excluded.since, + why = excluded.why, first_seen = excluded.first_seen, last_seen = excluded.last_seen, reminded_at = ''`, + customerID, stack, reason, fmtT(since), why, fmtT(now), fmtT(now)) + return err +} + +// ClearHeldApp removes a hold the box says is gone. Returns whether a row existed. +func (s *Store) ClearHeldApp(customerID, stack string) (bool, error) { + res, err := s.db.Exec(`DELETE FROM held_apps WHERE customer_id = ? AND stack = ?`, customerID, stack) + if err != nil { + return false, err + } + n, _ := res.RowsAffected() + return n > 0, nil +} + +// MarkHeldAppReminded records the operator reminder for the hold as it is now (same reason and since). +func (s *Store) MarkHeldAppReminded(customerID, stack, reason string, since, at time.Time) error { + _, err := s.db.Exec(`UPDATE held_apps SET reminded_at = ? WHERE customer_id = ? AND stack = ? AND reason = ? AND since = ?`, + fmtT(at), customerID, stack, reason, fmtT(since)) + return err +} + +// ListHeldApps returns every held app, or one customer's when customerID is not empty, oldest hold first. +func (s *Store) ListHeldApps(customerID string) ([]HeldApp, error) { + q := `SELECT customer_id, stack, reason, since, why, first_seen, last_seen, reminded_at FROM held_apps` + args := []interface{}{} + if customerID != "" { + q += ` WHERE customer_id = ?` + args = append(args, customerID) + } + q += ` ORDER BY since, customer_id, stack` + rows, err := s.db.Query(q, args...) + if err != nil { + return nil, err + } + defer rows.Close() + var out []HeldApp + for rows.Next() { + var h HeldApp + var since, first, last, rem string + if err := rows.Scan(&h.CustomerID, &h.Stack, &h.Reason, &since, &h.Why, &first, &last, &rem); err != nil { + return nil, err + } + h.Since, h.FirstSeen, h.LastSeen, h.RemindedAt = parseT(since), parseT(first), parseT(last), parseT(rem) + out = append(out, h) + } + return out, rows.Err() +} diff --git a/hub/internal/store/store.go b/hub/internal/store/store.go index fe189503..7185902f 100644 --- a/hub/internal/store/store.go +++ b/hub/internal/store/store.go @@ -889,6 +889,10 @@ func (s *Store) migrate() error { if err := s.migrateOperatorActions(); err != nil { return fmt.Errorf("operator_actions: %w", err) } + // D53 (`08` §10): the held apps the boxes report (held_apps.go). + if err := s.migrateHeldApps(); err != nil { + return fmt.Errorf("held_apps: %w", err) + } // Operator access tokens (hub v0.152.0, `05` §17, api_tokens.go). if err := s.migrateAPITokens(); err != nil { return fmt.Errorf("api_tokens: %w", err) diff --git a/hub/internal/web/d53_held_apps_test.go b/hub/internal/web/d53_held_apps_test.go new file mode 100644 index 00000000..65d14a48 --- /dev/null +++ b/hub/internal/web/d53_held_apps_test.go @@ -0,0 +1,66 @@ +package web + +import ( + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// D53 (`08` §10): the „Held apps" list on the host page and on the Hosts page — both branches of each {{if}} rendered +// (the seam-wiring rule): a held app shows its app, reason and who may clear it; after the clear, the empty sentence. +func TestD53_HeldAppsListOnTheHostPageAndTheHostsPage(t *testing.T) { + s, st := newTestServer(t) + if err := st.UpsertHost(&store.Host{HostID: "demo-hp-01", CustomerID: "c1", APIKey: "k1"}); err != nil { + t.Fatal(err) + } + now := time.Now() + if err := st.UpsertHeldApp("c1", "kimai", "unhealthy_stop", now.Add(-3*24*time.Hour), "stopped by the box for a crash loop", now); err != nil { + t.Fatal(err) + } + render := func() (string, string) { + rr := httptest.NewRecorder() + s.handleHostsList(rr, httptest.NewRequest(http.MethodGet, "/hosts", nil)) + hd := httptest.NewRecorder() + s.handleHostDetail(hd, httptest.NewRequest(http.MethodGet, "/hosts/demo-hp-01", nil), "demo-hp-01") + if rr.Code != 200 || hd.Code != 200 { + t.Fatalf("status: hosts %d, host %d", rr.Code, hd.Code) + } + return rr.Body.String(), hd.Body.String() + } + fleet, host := render() + for name, body := range map[string]string{"hosts": fleet, "host": host} { + if !strings.Contains(body, "Held apps") || !strings.Contains(body, `data-held-app=`) || !strings.Contains(body, "kimai") || + !strings.Contains(body, "unhealthy_stop") || !strings.Contains(body, "Start (one more try)") { + t.Errorf("%s page: the held app is not listed with its reason and who may clear it", name) + } + } + if !strings.Contains(fleet, `href="/hosts/demo-hp-01#held-apps"`) { + t.Error("hosts page: the held app does not link to its host") + } + if !strings.Contains(host, "stopped by the box for a crash loop") { + t.Error("host page: the box's sentence is missing") + } + if _, err := st.ClearHeldApp("c1", "kimai"); err != nil { + t.Fatal(err) + } + fleet, host = render() + if strings.Contains(fleet, `data-held-app=`) || !strings.Contains(fleet, "No app is held on any box.") { + t.Error("hosts page: after the clear the list is not empty") + } + if strings.Contains(host, `data-held-app=`) || !strings.Contains(host, "No app is held on this box.") { + t.Error("host page: after the clear the list is not empty") + } +} + +// A host with no customer shows no held apps — never the whole fleet's. +func TestD53_AHostWithNoCustomerShowsNoHeldApps(t *testing.T) { + s, st := newTestServer(t) + _ = st.UpsertHeldApp("c9", "x", "restore_failed", time.Now(), "y", time.Now()) + if rows := s.hostHeldAppRows("", time.Now()); len(rows) != 0 { + t.Fatalf("a customer-less host lists %d held app(s)", len(rows)) + } +} diff --git a/hub/internal/web/held_apps.go b/hub/internal/web/held_apps.go new file mode 100644 index 00000000..ec76039a --- /dev/null +++ b/hub/internal/web/held_apps.go @@ -0,0 +1,57 @@ +package web + +import ( + "time" + + "gitea.dooplex.hu/admin/felhom-hub/internal/monitor" + "gitea.dooplex.hu/admin/felhom-hub/internal/store" +) + +// heldAppRow is one line of the „Held apps" list (D53, `08` §10) on the host page and the Hosts page. +type heldAppRow struct { + CustomerID string + HostID string // the customer's host, for the link on the Hosts page; "" when unknown + Stack string + Reason string + Since time.Time + Why string // the box's own sentence + WhoClears string + Reminded bool + Stale bool // the box has not repeated it for heldAppStale — it may be down +} + +// heldAppStaleAfter mirrors monitor's heldAppStale: a box repeats app_held daily. +const heldAppStaleAfter = 50 * time.Hour + +// heldAppRows reads the held-app mirror, one customer's or every one's ("" = all). A read failure is logged and +// shown as an empty list — the page must still render. +func (s *Server) heldAppRows(customerID string, hostOf map[string]string, now time.Time) []heldAppRow { + held, err := s.store.ListHeldApps(customerID) + if err != nil { + s.logger.Printf("[WARN] held apps: list for %q: %v", customerID, err) + return nil + } + return toHeldAppRows(held, hostOf, now) +} + +// hostHeldAppRows is one host's list. A host with no customer has none — never the fleet's („" lists every one). +func (s *Server) hostHeldAppRows(customerID string, now time.Time) []heldAppRow { + if customerID == "" { + return nil + } + return s.heldAppRows(customerID, nil, now) +} + +func toHeldAppRows(held []store.HeldApp, hostOf map[string]string, now time.Time) []heldAppRow { + out := make([]heldAppRow, 0, len(held)) + for _, h := range held { + since := h.Since + if since.IsZero() { + since = h.FirstSeen + } + out = append(out, heldAppRow{CustomerID: h.CustomerID, HostID: hostOf[h.CustomerID], Stack: h.Stack, Reason: h.Reason, + Since: since, Why: h.Why, WhoClears: monitor.HeldAppWhoClears(h.Reason), Reminded: !h.RemindedAt.IsZero(), + Stale: now.Sub(h.LastSeen) > heldAppStaleAfter}) + } + return out +} diff --git a/hub/internal/web/hosts.go b/hub/internal/web/hosts.go index a1d51482..1f6d7f67 100644 --- a/hub/internal/web/hosts.go +++ b/hub/internal/web/hosts.go @@ -658,9 +658,19 @@ func (s *Server) handleHostsList(w http.ResponseWriter, r *http.Request) { // non-fatal: still render the host list } + // D53 (`08` §10): every held app across the fleet, each linked to its customer's host. + hostOf := map[string]string{} + for _, row := range rows { + if row.CustomerID != "" { + hostOf[row.CustomerID] = row.HostID + } + } + heldApps := s.heldAppRows("", hostOf, time.Now()) + data := map[string]interface{}{ "Hosts": rows, "Unclaimed": unclaimed, + "HeldApps": heldApps, "CustomerPicker": picker, "Flash": r.URL.Query().Get("flash"), "CSRFToken": s.getCSRFToken(r), @@ -873,6 +883,7 @@ func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]in "DesiredGeneration": host.DesiredGeneration, "Vitals": vitals, "Guests": guests, + "HeldApps": s.hostHeldAppRows(host.CustomerID, now), // D53, `08` §10 "GuestRunning": guestRunning, "GuestTotal": len(guests), "StorageTargets": storageTargets, diff --git a/hub/internal/web/templates/host_detail_body.html b/hub/internal/web/templates/host_detail_body.html index 16eb6c73..2092d8b9 100644 --- a/hub/internal/web/templates/host_detail_body.html +++ b/hub/internal/web/templates/host_detail_body.html @@ -146,6 +146,39 @@ {{end}} + +
+

Held apps

+ {{if .HeldApps}} + + + + + + + + + + + + {{range .HeldApps}} + + + + + + + + {{end}} + +
AppReasonSinceWhyWho may clear it
{{.Stack}}{{.Reason}}{{if .Stale}} (not confirmed by the box for 2 days){{end}}{{timeAgo .Since}}{{.Why}}{{.WhoClears}}{{if .Reminded}} (operator reminded){{end}}
+ {{else}} +
+

No app is held on this box.

+
+ {{end}} +
+

Storage Targets

diff --git a/hub/internal/web/templates/hosts.html b/hub/internal/web/templates/hosts.html index 85313179..e77dc93e 100644 --- a/hub/internal/web/templates/hosts.html +++ b/hub/internal/web/templates/hosts.html @@ -34,6 +34,39 @@ {{end}} + +
+

Held apps

+ {{if .HeldApps}} + + + + + + + + + + + + {{range .HeldApps}} + + + + + + + + {{end}} + +
CustomerAppReasonSinceWho may clear it
{{if .HostID}}{{.CustomerID}}{{else}}{{.CustomerID}}{{end}}{{.Stack}}{{.Reason}}{{if .Stale}} (not confirmed for 2 days){{end}}{{timeAgo .Since}}{{.WhoClears}}
+ {{else}} +
+

No app is held on any box.

+
+ {{end}} +
+ {{if .Unclaimed}}

Unclaimed appliances (booted from the generic ISO, awaiting a bind)