hub(D53): held apps reach the operator — mirror, Held apps list (host + Hosts page), reminder after the grace
gates / gates (push) Successful in 1m58s
gates / gates (push) Successful in 1m58s
app_held / app_hold_cleared allow-listed and operator-only; handleEvent mirrors them in held_apps; HeldAppsChecker sends ONE app_held_reminder (warning, operator-only, per-app cooldown) after the grace per reason (6 h / 1 d / 3 d), then weekly; recorded before sending; stale rows not reminded; the hub never clears a hold. Test-only alerting.held_app_grace_override. TestD53_* in api/monitor/notify/web, red-proved by six mutations. Commit only.
This commit is contained in:
admin committed
1 parent
ecd82abab8
commit
7e57e824a2
27 files changed
+935
-2
No files matched your search
@@ -124,6 +124,7 @@
|
||||
| `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
|
||||
| `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). |
|
||||
| `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
|
||||
| `monitor.HeldAppsChecker` + `store.UpsertHeldApp`/`ClearHeldApp`/`ListHeldApps` + `HeldAppGrace`/`HeldAppWhoClears` (D53, unreleased) | hub/internal/monitor/held_apps.go · hub/internal/store/held_apps.go · hub/internal/web/held_apps.go | `NewHeldAppsChecker(st, graceOverride, onEvent, logger)`; `.Check()` on the 60 s sweep | `08` §10: the boxes' held apps — the mirror (fed by `app_held`/`app_hold_cleared` in `handleEvent`), the „Held apps" list on the host page and the Hosts page, one operator reminder after the grace per reason, then weekly | **The hub never clears a hold** — a row goes only on the box's `app_hold_cleared`. The reminder is recorded BEFORE it is sent (a restart cannot mail twice); a row the box stopped repeating (> 50 h) is listed, never reminded. `app_held_reminder` is operator-only and per-app cooldown. `graceOverride` is a TEST configuration only (`alerting.held_app_grace_override`). Pinned by `TestD53_*` (monitor, api, web, notify) |
|
||||
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. **R-86 (2026-08-03): the window is PER TIER, not one constant.** `restoreProvenWindow(tier, observed, ok)` = `clamp(4 × max(observed, declared), floor 7d, cap 12d)`, where `declared` is that tier's own backup-freshness threshold (`backupStaleAfter` 26 h / `offsiteBackupStaleAfter` 8 d — reuse those, never a second opinion) and `observed` comes from `observedArchiveIntervals` over the retained window. **Observation may only WIDEN**: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. `assessRestoreProven` takes the window as an argument and **every reason string names it** (R-100's corollary). |
|
||||
| `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
|
||||
| `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
RED internal/notify/dispatcher.go | "app_held_reminder": true, } // IsOperatorOnly | ['--- FAIL: TestD53_TheHeldAppReminderMailsTheOperatorOnly (0.04s)']
|
||||
RED internal/api/handler.go | h.recordHeldApp(payload.CustomerID, payload.EventType, paylo | ['--- FAIL: TestD53_TheBoxsHeldAppsAreMirrored (0.05s)']
|
||||
RED internal/monitor/held_apps.go | if age < hc.grace(h.Reason) { return } | ['--- FAIL: TestD53_OneReminderAfterTheGraceThenWeekly (0.04s)', '--- FAIL: TestD53_TheGraceFollowsTheReason (0.17s)']
|
||||
RED internal/monitor/held_apps.go | if !h.RemindedAt.IsZero() && now.Sub(h.RemindedAt) < heldApp | ['--- FAIL: TestD53_OneReminderAfterTheGraceThenWeekly (0.03s)']
|
||||
RED internal/web/hosts.go | "HeldApps": heldApps, | []
|
||||
RED internal/web/templates/host_detail_body.html | <td>{{.WhoClears}}{{if .Reminded}} | ['--- FAIL: TestD53_HeldAppsListOnTheHostPageAndTheHostsPage (0.05s)']
|
||||
RED internal/web/hosts.go | HeldApps emptied (compiling mutation) | TestD53_HeldAppsListOnTheHostPageAndTheHostsPage
|
||||
@@ -1,3 +1,24 @@
|
||||
## Unreleased (next: v0.173.0) — D53: a held app reaches the operator (`08` §10)
|
||||
|
||||
Operator ruling D53 (2026-10-26, `09` §3 decision 265): operator-only reminders first. Pairs with controller `5bd03ad`
|
||||
(`internal/holdreport`); a box on an older controller sends nothing new, and its list stays empty.
|
||||
|
||||
- `internal/api/handler.go`: `app_held` and `app_hold_cleared` in `allowedEventTypes`; `handleEvent` → `recordHeldApp`
|
||||
mirrors them in the new `held_apps` table (`internal/store/held_apps.go`: one row per customer + stack; a repeat moves
|
||||
`last_seen` only; a new reason or since starts the row over; the clear deletes it).
|
||||
- `internal/monitor/held_apps.go` `HeldAppsChecker` (60 s sweep, one pass per 15 min): ONE operator reminder
|
||||
`app_held_reminder` (warning) after the grace per reason — `putback_pending` 6 h, `restore_failed`/`restore_mixed` 1 day,
|
||||
`update_failed`/`unhealthy_stop` 3 days — then weekly while it lasts; recorded before it is sent; a row the box stopped
|
||||
repeating (> 50 h) is not reminded. `alerting.held_app_grace_override` (a duration) replaces every grace — a TEST
|
||||
configuration only, logged at start.
|
||||
- `internal/notify`: all three types operator-only; `app_held_reminder` per-app cooldown and a subject that names the app;
|
||||
hu + en `mail.event.*` lines (the coupling rule) and their six mail goldens.
|
||||
- Web: a „Held apps" list on the host page (app, reason, since, the box's sentence, who may clear it) and on the Hosts
|
||||
page (every box, linked to its host); the empty sentence when none. The hub never clears a hold.
|
||||
- Tests `TestD53_*`: the mirror (api), the reminder after the grace / weekly / per reason / none after a clear / none for
|
||||
a stale row / a new hold starts over (monitor), operator only and per app (notify), both pages with and without a held
|
||||
app (web). Red-proved by six mutations (`documentation/audits/d53-held-apps-2026-10-26/redproof-hub.txt`).
|
||||
|
||||
## v0.172.0 (2026-10-26) — a fresh install's whole-guest off-site set-up never fails silently (R-1295) — RELEASED 2026-10-26 13:08 UTC on the operator's yes (image sha256:7c8d129a…, from `232f42d2`, CI 2366; rc.1 ran the live proof)
|
||||
|
||||
- **Cause (measured, `documentation/audits/r1295-pbsdr-selfcheck-2026-10-26/`):** PBS's proxy caches permissions for up to
|
||||
|
||||
@@ -72,6 +72,9 @@ type Config struct {
|
||||
// the host-root thresholds above so a single storage's policy can be tuned separately.
|
||||
StorageFillWarnPercent float64 `yaml:"storage_fill_warn_percent"`
|
||||
StorageFillCritPercent float64 `yaml:"storage_fill_crit_percent"`
|
||||
// HeldAppGraceOverride (D53) replaces every reason's reminder grace when set (a Go duration, e.g. "3m"). A TEST
|
||||
// configuration only — the live proof shortens the wait; production leaves it empty (`08` §10's per-reason grace).
|
||||
HeldAppGraceOverride string `yaml:"held_app_grace_override"`
|
||||
// Offsite POOL-BOX aggregate thresholds (v0.64.0, R-5). Fill = box used vs capacity (percent);
|
||||
// oversub = Σ(shared soft quotas)/capacity (ratio). Empty/0/invalid → defaults 80/90/2.0 (the
|
||||
// checker normalizes). Threshold VALUES are Claude's encoding of the starter suggestion — Viktor's
|
||||
@@ -899,6 +902,16 @@ func main() {
|
||||
// durable (events table), so a hub restart neither floods nor silently re-heals.
|
||||
// R-523: the agent's in-guest controller supervisor → controller_restarted_by_agent / controller_crashloop.
|
||||
controllerSupervisorChecker := monitor.NewControllerSupervisorChecker(dataStore, dispatcher.ProcessEvent, logger)
|
||||
// D53 (`08` §10): a held app reaches the OPERATOR — one reminder after the grace per reason, then weekly.
|
||||
var heldGrace time.Duration
|
||||
if v := cfg.Alerting.HeldAppGraceOverride; v != "" {
|
||||
if d, err := time.ParseDuration(v); err == nil && d > 0 {
|
||||
heldGrace = d
|
||||
} else {
|
||||
logger.Printf("[WARN] alerting.held_app_grace_override %q is not a positive duration — ignored", v)
|
||||
}
|
||||
}
|
||||
heldAppsChecker := monitor.NewHeldAppsChecker(dataStore, heldGrace, dispatcher.ProcessEvent, logger)
|
||||
offsiteDeliveryChecker := monitor.NewOffsiteDeliveryChecker(dataStore, offsiteHealReissuer, dispatcher.ProcessEvent, logger)
|
||||
// R-242 second half (S4, decision 199): published golden vs vouched golden. Needs the registry creds
|
||||
// (the same Gitea client as the Day-0 dropdowns); without them the check is off and says so.
|
||||
@@ -931,6 +944,7 @@ func main() {
|
||||
restoreTestChecker.Check() // R-85: restore-test failure + per-tier staleness
|
||||
offsiteDeliveryChecker.Check()
|
||||
controllerSupervisorChecker.Check()
|
||||
heldAppsChecker.Check() // D53: throttled to one pass per 15 min
|
||||
if goldenVouchChecker != nil {
|
||||
goldenVouchChecker.Check() // R-242: throttled to one Gitea listing per 15 min
|
||||
}
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
|
||||
)
|
||||
|
||||
// D53 (`08` §10): a box's app_held is accepted and mirrored (stack, reason, since); its daily repeat keeps ONE row; its
|
||||
// app_hold_cleared removes the row. Both types are allow-listed AND operator-only (the v0.78.0 pair, checked together).
|
||||
func TestD53_TheBoxsHeldAppsAreMirrored(t *testing.T) {
|
||||
h, st, _ := newTestHandler(t)
|
||||
for _, et := range []string{"app_held", "app_hold_cleared"} {
|
||||
if !allowedEventTypes[et] {
|
||||
t.Fatalf("%s is not allow-listed — POST /event would 400", et)
|
||||
}
|
||||
if !notify.IsOperatorOnly(et) {
|
||||
t.Fatalf("%s is not operator-only", et)
|
||||
}
|
||||
}
|
||||
if !notify.IsOperatorOnly("app_held_reminder") || allowedEventTypes["app_held_reminder"] {
|
||||
t.Fatal("app_held_reminder must be operator-only and hub-origin (a box must not post the reminder)")
|
||||
}
|
||||
held := `{"customer_id":"c1","event_type":"app_held","severity":"info","message":"App kimai is held",
|
||||
"details":{"stack_name":"kimai","reason":"unhealthy_stop","since":"2026-10-14T08:00:00Z","why":"stopped by the box"}}`
|
||||
for i := 0; i < 2; i++ { // the first report and the daily repeat
|
||||
if rr := do(h, "POST", "/event", globalKey, held); rr.Code != 200 {
|
||||
t.Fatalf("app_held: %d %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
}
|
||||
rows, err := st.ListHeldApps("c1")
|
||||
if err != nil || len(rows) != 1 {
|
||||
t.Fatalf("want one held row, got %+v (%v)", rows, err)
|
||||
}
|
||||
if r := rows[0]; r.Stack != "kimai" || r.Reason != "unhealthy_stop" || r.Since.Format("2006-01-02T15:04:05Z07:00") != "2026-10-14T08:00:00Z" || r.Why == "" {
|
||||
t.Fatalf("row: %+v", r)
|
||||
}
|
||||
cleared := `{"customer_id":"c1","event_type":"app_hold_cleared","severity":"info","message":"App kimai is no longer held","details":{"stack_name":"kimai"}}`
|
||||
if rr := do(h, "POST", "/event", globalKey, cleared); rr.Code != 200 {
|
||||
t.Fatalf("app_hold_cleared: %d", rr.Code)
|
||||
}
|
||||
if rows, _ := st.ListHeldApps("c1"); len(rows) != 0 {
|
||||
t.Fatalf("the cleared hold is still listed: %+v", rows)
|
||||
}
|
||||
}
|
||||
@@ -2262,6 +2262,12 @@ var allowedEventTypes = map[string]bool{
|
||||
// v0.123.0 (decision 28, controller v0.269.0): the box stopped an app that kept crashing / ran out
|
||||
// of memory. Household + operator; hu AND en `mail.event.*`; per-app cooldown on both legs.
|
||||
"app_stopped_unhealthy": true,
|
||||
// D53 (`08` §10, controller with internal/holdreport): the box's held apps — app_held for each hold in place (at once,
|
||||
// then daily), app_hold_cleared when one is gone. Severity info: never mailed on receipt; the hub mirrors them in
|
||||
// held_apps (handleEvent → recordHeldApp) and the reminder checker mails the OPERATOR after a grace. Operator-only
|
||||
// (notify.operatorOnlyEvents); hu AND en `mail.event.*` lines exist only because the coupling rule needs them.
|
||||
"app_held": true,
|
||||
"app_hold_cleared": true,
|
||||
|
||||
// R-871 (v0.134.0, controller v0.295.0, `09` decision 109): the box was off at its backup time and made the
|
||||
// missed backups when it came back. Info — recorded on the box's timeline, never mailed.
|
||||
@@ -2520,6 +2526,12 @@ func (h *Handler) handleEvent(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
h.logger.Printf("[INFO] Event from %s: %s (%s) — %s", payload.CustomerID, payload.EventType, payload.Severity, payload.Message)
|
||||
|
||||
// D53: mirror the box's held apps. A failure is logged and the event still dispatched — the box repeats
|
||||
// app_held daily, so the next one heals the mirror.
|
||||
if payload.EventType == "app_held" || payload.EventType == "app_hold_cleared" {
|
||||
h.recordHeldApp(payload.CustomerID, payload.EventType, payload.Details)
|
||||
}
|
||||
|
||||
// Dispatch notifications (non-blocking)
|
||||
if h.dispatcher != nil {
|
||||
go h.dispatcher.ProcessBoxEvent(payload.CustomerID, payload.EventType, payload.Severity,
|
||||
@@ -3189,3 +3201,40 @@ func (h *Handler) checkTunnel(hostID, custID, status, detail string) {
|
||||
go h.dispatcher.ProcessEvent(custID, typ, sev, msg, string(details), "hub")
|
||||
}
|
||||
}
|
||||
|
||||
// recordHeldApp applies one app_held / app_hold_cleared to the held_apps mirror (D53, `08` §10). The hub never
|
||||
// clears a hold on a box; it only forgets one the box says is gone.
|
||||
func (h *Handler) recordHeldApp(customerID, eventType string, raw json.RawMessage) {
|
||||
var d struct {
|
||||
StackName string `json:"stack_name"`
|
||||
Reason string `json:"reason"`
|
||||
Since string `json:"since"`
|
||||
Why string `json:"why"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &d); err != nil || d.StackName == "" {
|
||||
h.logger.Printf("[WARN] [held-apps] %s from %s has no readable stack_name — not recorded", eventType, customerID)
|
||||
return
|
||||
}
|
||||
if eventType == "app_hold_cleared" {
|
||||
had, err := h.store.ClearHeldApp(customerID, d.StackName)
|
||||
if err != nil {
|
||||
h.logger.Printf("[WARN] [held-apps] clear %s/%s: %v", customerID, d.StackName, err)
|
||||
return
|
||||
}
|
||||
h.logger.Printf("[INFO] [held-apps] %s/%s no longer held (a row existed: %v)", customerID, d.StackName, had)
|
||||
return
|
||||
}
|
||||
var since time.Time
|
||||
if d.Since != "" {
|
||||
if t, err := time.Parse(time.RFC3339, d.Since); err == nil {
|
||||
since = t
|
||||
} else {
|
||||
h.logger.Printf("[WARN] [held-apps] %s/%s: since %q is not RFC3339 — the hub's first sighting is the clock", customerID, d.StackName, d.Since)
|
||||
}
|
||||
}
|
||||
if err := h.store.UpsertHeldApp(customerID, d.StackName, d.Reason, since, d.Why, time.Now()); err != nil {
|
||||
h.logger.Printf("[WARN] [held-apps] record %s/%s: %v", customerID, d.StackName, err)
|
||||
return
|
||||
}
|
||||
h.logger.Printf("[DEBUG] [held-apps] %s/%s held (%s since %s)", customerID, d.StackName, d.Reason, d.Since)
|
||||
}
|
||||
@@ -89,6 +89,9 @@
|
||||
"mail.event.app_update_undone": "%s: the update did not work; the app runs on its previous version",
|
||||
"mail.event.app_update_held": "%s: the app is stopped and needs a restore",
|
||||
"mail.event.app_stopped_unhealthy": "%s: the app was stopped because it kept crashing",
|
||||
"mail.event.app_held": "An app is stopped and waits for a person to act",
|
||||
"mail.event.app_hold_cleared": "The app is no longer held",
|
||||
"mail.event.app_held_reminder": "An app is still held",
|
||||
"bind.resend.hint": "If this was your link and your box is not linked yet, we send a fresh link to the e-mail address you gave Felhom.",
|
||||
"bind.resend.button": "Send me a new link",
|
||||
"bind.resend.newer": "If you have a newer mail from Felhom, use the link in that one.",
|
||||
|
||||
@@ -89,6 +89,9 @@
|
||||
"mail.event.app_update_undone": "%s: a frissítés nem sikerült, az alkalmazás a korábbi változattal fut",
|
||||
"mail.event.app_update_held": "%s: az alkalmazás leállítva, visszaállítás szükséges",
|
||||
"mail.event.app_stopped_unhealthy": "%s: az alkalmazást leállítottuk, mert újra és újra összeomlott",
|
||||
"mail.event.app_held": "Egy alkalmazás le van állítva, és emberi beavatkozásra vár",
|
||||
"mail.event.app_hold_cleared": "Az alkalmazás leállítása megszűnt",
|
||||
"mail.event.app_held_reminder": "Egy alkalmazás még mindig le van állítva",
|
||||
"bind.resend.hint": "Ha ez a te linked volt, és a dobozod még nincs összekötve, új linket küldünk arra az e-mail címre, amelyet a Felhomnál megadtál.",
|
||||
"bind.resend.button": "Új linket kérek",
|
||||
"bind.resend.newer": "Ha van újabb leveled a Felhomtól, abban lévő linket használd.",
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// ── D53 (`08` §10, R-1291) — A HELD APP REACHES THE OPERATOR ──────────────────────────────────────
|
||||
//
|
||||
// A held app is stopped on purpose and stays stopped until a person acts. demo-hp's Kimai sat held for 11 days
|
||||
// (2026-10-14 → 10-26) and nobody was told after the first minute. The box now reports its holds (app_held /
|
||||
// app_hold_cleared, severity info — nothing is mailed on receipt); this checker turns a hold that LASTS into ONE
|
||||
// operator mail after a grace that follows the reason, then one a week while it lasts.
|
||||
//
|
||||
// THE GRACE (`08` §10): putback_pending 6 h — it should have cleared itself, so it is stuck; restore_failed and
|
||||
// restore_mixed 1 day — only a person can act; update_failed and unhealthy_stop 3 days — the household can act first.
|
||||
// An unknown reason gets 1 day. The clock is the hold's own time (the box's), else when the hub first heard of it.
|
||||
//
|
||||
// OPERATOR ONLY (D53): app_held_reminder is in the dispatcher's operatorOnlyEvents, so no household is ever mailed.
|
||||
// THE HUB NEVER CLEARS A HOLD — a row goes only when the box sends app_hold_cleared.
|
||||
//
|
||||
// A ROW THE BOX STOPPED CONFIRMING is not reminded: the box repeats app_held daily, so a row older than heldAppStale
|
||||
// belongs to a box that is down or gone, which its own staleness alarms report. The list still shows it.
|
||||
//
|
||||
// The reminder is recorded (MarkHeldAppReminded) BEFORE it is sent, so a hub restart cannot mail twice.
|
||||
// Pinned by held_apps_test.go.
|
||||
const (
|
||||
eventHeldAppReminder = "app_held_reminder"
|
||||
heldAppRepeat = 7 * 24 * time.Hour
|
||||
heldAppStale = 50 * time.Hour
|
||||
heldAppCheckEvery = 15 * time.Minute
|
||||
)
|
||||
|
||||
// HeldAppGrace returns the wait before the first reminder for a reason (`08` §10).
|
||||
func HeldAppGrace(reason string) time.Duration {
|
||||
switch reason {
|
||||
case "putback_pending":
|
||||
return 6 * time.Hour
|
||||
case "update_failed", "unhealthy_stop":
|
||||
return 72 * time.Hour
|
||||
default: // restore_failed, restore_mixed, and a reason this hub does not know yet
|
||||
return 24 * time.Hour
|
||||
}
|
||||
}
|
||||
|
||||
// HeldAppWhoClears says who may lift a hold, in the operator's words.
|
||||
func HeldAppWhoClears(reason string) string {
|
||||
switch reason {
|
||||
case "update_failed":
|
||||
return "the household, by restoring the app from its backup"
|
||||
case "unhealthy_stop":
|
||||
return "the household's Start (one more try)"
|
||||
case "putback_pending":
|
||||
return "nobody — it clears itself when the drive with the pre-restore copy comes back"
|
||||
default:
|
||||
return "a person only (support: the controller's -clear-restore-hold)"
|
||||
}
|
||||
}
|
||||
|
||||
type heldAppsStore interface {
|
||||
ListHeldApps(customerID string) ([]store.HeldApp, error)
|
||||
MarkHeldAppReminded(customerID, stack, reason string, since, at time.Time) error
|
||||
SaveEvent(customerID, eventType, severity, message, detailsJSON, source string) (int64, error)
|
||||
}
|
||||
|
||||
// HeldAppsChecker sends the operator's reminders for held apps.
|
||||
type HeldAppsChecker struct {
|
||||
store heldAppsStore
|
||||
onEvent EventNotifyFunc
|
||||
logger *log.Logger
|
||||
now func() time.Time
|
||||
every time.Duration
|
||||
// graceOverride, when > 0, replaces every reason's grace. A TEST configuration only (the live proof shortens
|
||||
// the wait); production leaves it 0. Logged at start when set.
|
||||
graceOverride time.Duration
|
||||
|
||||
mu sync.Mutex
|
||||
lastRun time.Time
|
||||
}
|
||||
|
||||
// NewHeldAppsChecker builds the checker. graceOverride 0 = the per-reason grace.
|
||||
func NewHeldAppsChecker(st heldAppsStore, graceOverride time.Duration, onEvent EventNotifyFunc, logger *log.Logger) *HeldAppsChecker {
|
||||
if graceOverride > 0 && logger != nil {
|
||||
logger.Printf("[WARN] Held apps: the reminder grace is OVERRIDDEN to %s for every reason — a test configuration", graceOverride)
|
||||
}
|
||||
return &HeldAppsChecker{store: st, onEvent: onEvent, logger: logger, now: time.Now, every: heldAppCheckEvery, graceOverride: graceOverride}
|
||||
}
|
||||
|
||||
func (hc *HeldAppsChecker) grace(reason string) time.Duration {
|
||||
if hc.graceOverride > 0 {
|
||||
return hc.graceOverride
|
||||
}
|
||||
return HeldAppGrace(reason)
|
||||
}
|
||||
|
||||
// Check runs on the hub's 60 s sweep and throttles itself to one pass per heldAppCheckEvery.
|
||||
func (hc *HeldAppsChecker) Check() {
|
||||
hc.mu.Lock()
|
||||
defer hc.mu.Unlock()
|
||||
now := hc.now()
|
||||
every := hc.every
|
||||
if hc.graceOverride > 0 && hc.graceOverride < every {
|
||||
every = hc.graceOverride // a test grace of minutes must not wait a 15-minute pass
|
||||
}
|
||||
if !hc.lastRun.IsZero() && now.Sub(hc.lastRun) < every {
|
||||
return
|
||||
}
|
||||
hc.lastRun = now
|
||||
held, err := hc.store.ListHeldApps("")
|
||||
if err != nil {
|
||||
hc.logger.Printf("[WARN] Held apps: list failed: %v — no reminder this pass", err)
|
||||
return
|
||||
}
|
||||
for _, h := range held {
|
||||
hc.checkOne(h, now)
|
||||
}
|
||||
}
|
||||
|
||||
func (hc *HeldAppsChecker) checkOne(h store.HeldApp, now time.Time) {
|
||||
if now.Sub(h.LastSeen) > heldAppStale {
|
||||
hc.logger.Printf("[DEBUG] Held apps: %s/%s not confirmed since %s — no reminder (the box is not reporting)",
|
||||
h.CustomerID, h.Stack, h.LastSeen.Format(time.RFC3339))
|
||||
return
|
||||
}
|
||||
anchor := h.Since
|
||||
if anchor.IsZero() {
|
||||
anchor = h.FirstSeen
|
||||
}
|
||||
age := now.Sub(anchor)
|
||||
if age < hc.grace(h.Reason) {
|
||||
return
|
||||
}
|
||||
if !h.RemindedAt.IsZero() && now.Sub(h.RemindedAt) < heldAppRepeat {
|
||||
return
|
||||
}
|
||||
if err := hc.store.MarkHeldAppReminded(h.CustomerID, h.Stack, h.Reason, h.Since, now); err != nil {
|
||||
hc.logger.Printf("[WARN] Held apps: could not record the reminder for %s/%s (%v) — not sending, so a restart cannot mail twice",
|
||||
h.CustomerID, h.Stack, err)
|
||||
return
|
||||
}
|
||||
repeat := !h.RemindedAt.IsZero()
|
||||
msg := fmt.Sprintf("Customer %s: the app %s has been held for %s (%s): %s. Who may clear it: %s. The hub never clears a hold.",
|
||||
h.CustomerID, h.Stack, roundAge(age), h.Reason, h.Why, HeldAppWhoClears(h.Reason))
|
||||
if repeat {
|
||||
msg += " Weekly reminder: it is still held."
|
||||
}
|
||||
details, _ := json.Marshal(map[string]any{
|
||||
"customer_id": h.CustomerID, "stack_name": h.Stack, "reason": h.Reason,
|
||||
"since": anchor.UTC().Format(time.RFC3339), "why": h.Why, "repeat": repeat,
|
||||
})
|
||||
hc.logger.Printf("[INFO] Held apps: reminder for %s/%s (%s, held %s, repeat=%v)", h.CustomerID, h.Stack, h.Reason, roundAge(age), repeat)
|
||||
if _, err := hc.store.SaveEvent(h.CustomerID, eventHeldAppReminder, "warning", msg, string(details), "hub"); err != nil {
|
||||
hc.logger.Printf("[WARN] Failed to save %s for %s: %v", eventHeldAppReminder, h.CustomerID, err)
|
||||
return
|
||||
}
|
||||
if hc.onEvent != nil {
|
||||
hc.onEvent(h.CustomerID, eventHeldAppReminder, "warning", msg, string(details), "hub")
|
||||
}
|
||||
}
|
||||
|
||||
func roundAge(d time.Duration) string {
|
||||
if d >= 48*time.Hour {
|
||||
return fmt.Sprintf("%d days", int(d/(24*time.Hour)))
|
||||
}
|
||||
if d >= time.Hour {
|
||||
return d.Round(time.Hour).String()
|
||||
}
|
||||
return d.Round(time.Minute).String()
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package monitor
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
type heldEv struct{ customer, typ, sev, details string }
|
||||
|
||||
func heldSetup(t *testing.T) (*store.Store, *HeldAppsChecker, *[]heldEv, *time.Time) {
|
||||
t.Helper()
|
||||
st, err := store.New(filepath.Join(t.TempDir(), "h.db"), log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { st.Close() })
|
||||
var evs []heldEv
|
||||
now := time.Date(2026, 10, 26, 12, 0, 0, 0, time.UTC)
|
||||
hc := NewHeldAppsChecker(st, 0, func(c, typ, sev, msg, details, src string) {
|
||||
evs = append(evs, heldEv{c, typ, sev, details})
|
||||
}, log.New(io.Discard, "", 0))
|
||||
hc.now = func() time.Time { return now }
|
||||
hc.every = 0
|
||||
return st, hc, &evs, &now
|
||||
}
|
||||
|
||||
// The reminder after the grace (D53): a crash-loop hold waits 3 days, then ONE warning reminder to the operator, then
|
||||
// nothing until a week has passed, then one more.
|
||||
func TestD53_OneReminderAfterTheGraceThenWeekly(t *testing.T) {
|
||||
st, hc, evs, now := heldSetup(t)
|
||||
since := now.Add(-time.Hour)
|
||||
if err := st.UpsertHeldApp("c1", "kimai", "unhealthy_stop", since, "stopped by the box", *now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
hc.Check()
|
||||
if len(*evs) != 0 {
|
||||
t.Fatalf("a reminder inside the 3-day grace: %+v", *evs)
|
||||
}
|
||||
for _, step := range []time.Duration{72 * time.Hour, time.Hour, 24 * time.Hour} {
|
||||
*now = now.Add(step)
|
||||
_ = st.UpsertHeldApp("c1", "kimai", "unhealthy_stop", since, "stopped by the box", *now) // the box's daily repeat
|
||||
hc.Check()
|
||||
}
|
||||
if len(*evs) != 1 || (*evs)[0].typ != "app_held_reminder" || (*evs)[0].sev != "warning" || (*evs)[0].customer != "c1" {
|
||||
t.Fatalf("want exactly one warning app_held_reminder after the grace, got %+v", *evs)
|
||||
}
|
||||
*now = now.Add(7 * 24 * time.Hour)
|
||||
_ = st.UpsertHeldApp("c1", "kimai", "unhealthy_stop", since, "stopped by the box", *now)
|
||||
hc.Check()
|
||||
if len(*evs) != 2 {
|
||||
t.Fatalf("want the weekly repeat, got %d events", len(*evs))
|
||||
}
|
||||
}
|
||||
|
||||
// No reminder after a clear: the box's app_hold_cleared removes the row, and nothing is mailed for it.
|
||||
func TestD53_NoReminderAfterAClear(t *testing.T) {
|
||||
st, hc, evs, now := heldSetup(t)
|
||||
_ = st.UpsertHeldApp("c1", "db", "restore_failed", now.Add(-2*time.Hour), "held after a failed restore", *now)
|
||||
if ok, err := st.ClearHeldApp("c1", "db"); err != nil || !ok {
|
||||
t.Fatalf("clear: %v %v", ok, err)
|
||||
}
|
||||
*now = now.Add(30 * 24 * time.Hour)
|
||||
hc.Check()
|
||||
if len(*evs) != 0 {
|
||||
t.Fatalf("a cleared hold was reminded: %+v", *evs)
|
||||
}
|
||||
if rows, _ := st.ListHeldApps(""); len(rows) != 0 {
|
||||
t.Fatalf("the list still holds a cleared app: %+v", rows)
|
||||
}
|
||||
}
|
||||
|
||||
// The grace follows the reason (`08` §10): put-back 6 h, restore 1 day, update / crash loop 3 days.
|
||||
func TestD53_TheGraceFollowsTheReason(t *testing.T) {
|
||||
cases := map[string]time.Duration{"putback_pending": 6 * time.Hour, "restore_failed": 24 * time.Hour,
|
||||
"restore_mixed": 24 * time.Hour, "update_failed": 72 * time.Hour, "unhealthy_stop": 72 * time.Hour}
|
||||
for reason, grace := range cases {
|
||||
st, hc, evs, now := heldSetup(t)
|
||||
_ = st.UpsertHeldApp("c1", "a", reason, *now, "x", *now)
|
||||
*now = now.Add(grace - time.Minute)
|
||||
_ = st.UpsertHeldApp("c1", "a", reason, now.Add(-(grace - time.Minute)), "x", *now)
|
||||
hc.Check()
|
||||
if len(*evs) != 0 {
|
||||
t.Errorf("%s: reminded before its grace %s", reason, grace)
|
||||
}
|
||||
*now = now.Add(2 * time.Minute)
|
||||
hc.Check()
|
||||
if len(*evs) != 1 {
|
||||
t.Errorf("%s: no reminder just after its grace %s (%d events)", reason, grace, len(*evs))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A box that stopped confirming its hold (down or gone) is not reminded about it; its own staleness alarms speak.
|
||||
func TestD53_AStaleRowIsNotReminded(t *testing.T) {
|
||||
st, hc, evs, now := heldSetup(t)
|
||||
_ = st.UpsertHeldApp("c1", "a", "restore_failed", now.Add(-10*24*time.Hour), "x", now.Add(-3*24*time.Hour))
|
||||
hc.Check()
|
||||
if len(*evs) != 0 {
|
||||
t.Fatalf("a row the box stopped confirming was reminded: %+v", *evs)
|
||||
}
|
||||
}
|
||||
|
||||
// A new hold on the same app (another reason) starts its own grace: the old reminder does not carry over.
|
||||
func TestD53_ANewHoldOnTheSameAppStartsOver(t *testing.T) {
|
||||
st, hc, evs, now := heldSetup(t)
|
||||
_ = st.UpsertHeldApp("c1", "a", "putback_pending", now.Add(-7*time.Hour), "x", *now)
|
||||
hc.Check()
|
||||
if len(*evs) != 1 {
|
||||
t.Fatalf("want the put-back reminder, got %d", len(*evs))
|
||||
}
|
||||
_ = st.UpsertHeldApp("c1", "a", "restore_mixed", *now, "y", *now)
|
||||
*now = now.Add(25 * time.Hour)
|
||||
_ = st.UpsertHeldApp("c1", "a", "restore_mixed", now.Add(-25*time.Hour), "y", *now)
|
||||
hc.Check()
|
||||
if len(*evs) != 2 {
|
||||
t.Fatalf("the new hold's reminder was swallowed by the old one's week: %d events", len(*evs))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// D53 (operator ruling 2026-10-26: operator-only reminders first): the held-app reminder reaches the OPERATOR — one mail
|
||||
// whose subject names the app — and never the household, even with every event switched on (the R-97c configuration).
|
||||
// The box's own app_held / app_hold_cleared are info: they mail nobody on either leg.
|
||||
func TestD53_TheHeldAppReminderMailsTheOperatorOnly(t *testing.T) {
|
||||
st := opOnlyStore(t)
|
||||
if err := st.SaveNotificationPrefs("c1", "customer@example.com", []string{"app_held_reminder", "app_held", "app_hold_cleared", "backup_failed"}, 6); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var to, subjects []string
|
||||
d := opOnlyDispatcher(t, st, &sentTo{})
|
||||
d.sendEmailFn = func(addr, subject, body string, headers map[string]string) error {
|
||||
to = append(to, addr)
|
||||
subjects = append(subjects, subject)
|
||||
return nil
|
||||
}
|
||||
d.ProcessBoxEvent("c1", "app_held", "info", "App kimai is held: …", "", `{"stack_name":"kimai","reason":"unhealthy_stop"}`, "controller")
|
||||
d.ProcessBoxEvent("c1", "app_hold_cleared", "info", "App kimai is no longer held", "", `{"stack_name":"kimai"}`, "controller")
|
||||
if len(to) != 0 {
|
||||
t.Fatalf("the box's held-app events mailed %v — they must be recorded, never mailed", to)
|
||||
}
|
||||
d.ProcessEvent("c1", "app_held_reminder", "warning", "Customer c1: the app kimai has been held for 3 days …",
|
||||
`{"stack_name":"kimai","reason":"unhealthy_stop"}`, "hub")
|
||||
d.ProcessEvent("c1", "app_held_reminder", "warning", "Customer c1: the app paperless has been held for 3 days …",
|
||||
`{"stack_name":"paperless","reason":"update_failed"}`, "hub")
|
||||
for _, a := range to {
|
||||
if a == "customer@example.com" {
|
||||
t.Fatal("the household was mailed an operator-only held-app reminder")
|
||||
}
|
||||
}
|
||||
if len(to) != 2 {
|
||||
t.Fatalf("want 2 operator mails (two apps, per-app cooldown), got %d: %v", len(to), to)
|
||||
}
|
||||
if !strings.Contains(subjects[0], "kimai is still held") || !strings.Contains(subjects[1], "paperless is still held") {
|
||||
t.Fatalf("each subject must name its app: %q", subjects)
|
||||
}
|
||||
}
|
||||
@@ -501,6 +501,8 @@ var perAppCooldownEvents = map[string]bool{
|
||||
// v0.169.0 (the upload watch): one app's unexplained upload or CPU, no digest — two hacked apps are two alarms.
|
||||
"app_unexplained_upload": true,
|
||||
"app_unexplained_cpu": true,
|
||||
// D53: one held app's reminder, no digest — two apps held on one box are two reminders.
|
||||
"app_held_reminder": true,
|
||||
}
|
||||
|
||||
// perAppCustomerCooldownEvents is the CUSTOMER-leg sibling of perAppCooldownEvents (v0.120.0). The
|
||||
@@ -938,6 +940,11 @@ var operatorOnlyEvents = map[string]bool{
|
||||
// R-1295 (v0.172.0): the WG-registration hook could not set up a box's whole-guest off-site tier (web/pbsdr.go).
|
||||
// Host ids and an instruction only the operator can carry out. Hub-origin only.
|
||||
"pbsdr_provision_failed": true,
|
||||
// D53 (`08` §10, operator ruling 2026-10-26: operator-only reminders first). The box's held-app events are info
|
||||
// and never routed; the hub's reminder after the grace is the operator's mail. Listed in the SAME commit.
|
||||
"app_held": true,
|
||||
"app_hold_cleared": true,
|
||||
"app_held_reminder": true,
|
||||
}
|
||||
|
||||
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
|
||||
|
||||
@@ -73,7 +73,8 @@ func TestR389_TheAllowListHasExactlyOneMember(t *testing.T) {
|
||||
// v0.123.0 once more (decision 28): an unhealthy stop is one app's event with no digest.
|
||||
// v0.150.0 once more (R-1007): an app's repair after an unclean boot is one app's event with no digest.
|
||||
// v0.169.0 once more (the upload watch): an app's unexplained upload or CPU is one app's event with no digest.
|
||||
want := []string{"app_hold_no_whole_copy", "app_oom_storm", "app_repaired_after_crash", "app_start_failed", "app_stopped_unhealthy", "app_unexplained_cpu", "app_unexplained_upload", "app_update_held", "app_update_undone"}
|
||||
// D53 once more: a held app's reminder is one app's event with no digest — two held apps are two reminders.
|
||||
want := []string{"app_held_reminder", "app_hold_no_whole_copy", "app_oom_storm", "app_repaired_after_crash", "app_start_failed", "app_stopped_unhealthy", "app_unexplained_cpu", "app_unexplained_upload", "app_update_held", "app_update_undone"}
|
||||
ok := len(perAppCooldownEvents) == len(want)
|
||||
for _, w := range want {
|
||||
ok = ok && perAppCooldownEvents[w]
|
||||
@@ -83,7 +84,7 @@ func TestR389_TheAllowListHasExactlyOneMember(t *testing.T) {
|
||||
for k := range perAppCooldownEvents {
|
||||
got = append(got, k)
|
||||
}
|
||||
t.Fatalf("perAppCooldownEvents = %v, want exactly [app_hold_no_whole_copy app_oom_storm app_repaired_after_crash app_start_failed app_stopped_unhealthy app_unexplained_cpu app_unexplained_upload app_update_held app_update_undone]. Adding a member is the "+
|
||||
t.Fatalf("perAppCooldownEvents = %v, want exactly [app_held_reminder app_hold_no_whole_copy app_oom_storm app_repaired_after_crash app_start_failed app_stopped_unhealthy app_unexplained_cpu app_unexplained_upload app_update_held app_update_undone]. Adding a member is the "+
|
||||
"fenced act: the backup family's cooldown is coarse ON PURPOSE (R-97a, R-182) so one full "+
|
||||
"disk sends one digest, not one mail per app. Read the fence before widening this.", got)
|
||||
}
|
||||
|
||||
@@ -48,6 +48,12 @@ func FormatOperatorEmail(customerID, eventType, severity, message, detailsJSON s
|
||||
if eventType == "pbsdr_provision_failed" {
|
||||
subject = fmt.Sprintf("[Felhom] %s %s: the whole-box off-site copy (PBS DR) could not be set up — save the customer config to retry", icon, customerID)
|
||||
}
|
||||
// D53: the subject names the held app, so two reminders are told apart in the inbox.
|
||||
if eventType == "app_held_reminder" {
|
||||
if st := stackNameOf(detailsJSON); st != "" {
|
||||
subject = fmt.Sprintf("[Felhom] %s %s: the app %s is still held — a person must act", icon, customerID, st)
|
||||
}
|
||||
}
|
||||
|
||||
now := nowFn().In(budapest).Format("2006-01-02 15:04 MST")
|
||||
body := fmt.Sprintf(`Customer: %s
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
SUBJECT: [Felhom] Warning: An app is stopped and waits for a person to act
|
||||
---
|
||||
Dear Customer,
|
||||
|
||||
Your Felhom system sent the following notification:
|
||||
|
||||
An app is stopped and waits for a person to act
|
||||
|
||||
Details:
|
||||
- Server: demo-fixture
|
||||
- Time: 2026-01-15 10:30
|
||||
- Level: Warning
|
||||
- Type: app_held
|
||||
|
||||
If you have any questions, contact your operator.
|
||||
|
||||
Best regards,
|
||||
Felhom.eu monitoring
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
SUBJECT: [Felhom] Warning: An app is still held
|
||||
---
|
||||
Dear Customer,
|
||||
|
||||
Your Felhom system sent the following notification:
|
||||
|
||||
An app is still held
|
||||
|
||||
Details:
|
||||
- Server: demo-fixture
|
||||
- Time: 2026-01-15 10:30
|
||||
- Level: Warning
|
||||
- Type: app_held_reminder
|
||||
|
||||
If you have any questions, contact your operator.
|
||||
|
||||
Best regards,
|
||||
Felhom.eu monitoring
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
SUBJECT: [Felhom] Warning: The app is no longer held
|
||||
---
|
||||
Dear Customer,
|
||||
|
||||
Your Felhom system sent the following notification:
|
||||
|
||||
The app is no longer held
|
||||
|
||||
Details:
|
||||
- Server: demo-fixture
|
||||
- Time: 2026-01-15 10:30
|
||||
- Level: Warning
|
||||
- Type: app_hold_cleared
|
||||
|
||||
If you have any questions, contact your operator.
|
||||
|
||||
Best regards,
|
||||
Felhom.eu monitoring
|
||||
@@ -0,0 +1,18 @@
|
||||
SUBJECT: [Felhom] Figyelmeztetés: Egy alkalmazás le van állítva, és emberi beavatkozásra vár
|
||||
---
|
||||
Kedves Ügyfél!
|
||||
|
||||
A Felhom rendszered a következő értesítést küldte:
|
||||
|
||||
Egy alkalmazás le van állítva, és emberi beavatkozásra vár
|
||||
|
||||
Részletek:
|
||||
- Szerver: demo-fixture
|
||||
- Időpont: 2026-01-15 10:30
|
||||
- Szint: Figyelmeztetés
|
||||
- Típus: app_held
|
||||
|
||||
Ha kérdésed van, vedd fel a kapcsolatot az üzemeltetővel.
|
||||
|
||||
Üdvözlettel,
|
||||
Felhom.eu monitoring
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
SUBJECT: [Felhom] Figyelmeztetés: Egy alkalmazás még mindig le van állítva
|
||||
---
|
||||
Kedves Ügyfél!
|
||||
|
||||
A Felhom rendszered a következő értesítést küldte:
|
||||
|
||||
Egy alkalmazás még mindig le van állítva
|
||||
|
||||
Részletek:
|
||||
- Szerver: demo-fixture
|
||||
- Időpont: 2026-01-15 10:30
|
||||
- Szint: Figyelmeztetés
|
||||
- Típus: app_held_reminder
|
||||
|
||||
Ha kérdésed van, vedd fel a kapcsolatot az üzemeltetővel.
|
||||
|
||||
Üdvözlettel,
|
||||
Felhom.eu monitoring
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
SUBJECT: [Felhom] Figyelmeztetés: Az alkalmazás leállítása megszűnt
|
||||
---
|
||||
Kedves Ügyfél!
|
||||
|
||||
A Felhom rendszered a következő értesítést küldte:
|
||||
|
||||
Az alkalmazás leállítása megszűnt
|
||||
|
||||
Részletek:
|
||||
- Szerver: demo-fixture
|
||||
- Időpont: 2026-01-15 10:30
|
||||
- Szint: Figyelmeztetés
|
||||
- Típus: app_hold_cleared
|
||||
|
||||
Ha kérdésed van, vedd fel a kapcsolatot az üzemeltetővel.
|
||||
|
||||
Üdvözlettel,
|
||||
Felhom.eu monitoring
|
||||
@@ -0,0 +1,124 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ── D53 (`08` §10, R-1291): the held apps a box reports ───────────────────────────────────────────
|
||||
//
|
||||
// A box (controller ≥ the D53 release) sends app_held for every hold in place — at once, then once a day — and
|
||||
// app_hold_cleared when one is gone. This table is the hub's mirror of that: one row per (customer, stack). The hub
|
||||
// NEVER clears a hold on a box; a row goes only when the box says the hold is gone. The reminder checker
|
||||
// (monitor.HeldAppsChecker) reads it and records here when it mailed, so a hub restart neither re-mails nor forgets.
|
||||
// All times are stored as RFC3339 UTC text.
|
||||
|
||||
// HeldApp is one held app as the hub knows it.
|
||||
type HeldApp struct {
|
||||
CustomerID string
|
||||
Stack string
|
||||
Reason string // restore_failed, restore_mixed, update_failed, unhealthy_stop, putback_pending
|
||||
Since time.Time // the hold's own time (the box's)
|
||||
Why string // the box's one-line sentence: what put it there and what lifts it
|
||||
FirstSeen time.Time // when the hub first heard of this hold
|
||||
LastSeen time.Time // the newest app_held for it — the box repeats it daily
|
||||
RemindedAt time.Time // the last operator reminder; zero = none yet
|
||||
}
|
||||
|
||||
func (s *Store) migrateHeldApps() error {
|
||||
_, err := s.db.Exec(`
|
||||
CREATE TABLE IF NOT EXISTS held_apps (
|
||||
customer_id TEXT NOT NULL,
|
||||
stack TEXT NOT NULL,
|
||||
reason TEXT NOT NULL DEFAULT '',
|
||||
since TEXT NOT NULL DEFAULT '',
|
||||
why TEXT NOT NULL DEFAULT '',
|
||||
first_seen TEXT NOT NULL,
|
||||
last_seen TEXT NOT NULL,
|
||||
reminded_at TEXT NOT NULL DEFAULT '',
|
||||
PRIMARY KEY (customer_id, stack)
|
||||
);`)
|
||||
return err
|
||||
}
|
||||
|
||||
func fmtT(t time.Time) string {
|
||||
if t.IsZero() {
|
||||
return ""
|
||||
}
|
||||
return t.UTC().Format(time.RFC3339)
|
||||
}
|
||||
|
||||
func parseT(s string) time.Time {
|
||||
if s == "" {
|
||||
return time.Time{}
|
||||
}
|
||||
return parseSQLiteTime(s)
|
||||
}
|
||||
|
||||
// UpsertHeldApp records an app_held. A repeat of the same hold (same reason and since) only moves last_seen; a NEW
|
||||
// hold on the stack (another reason or another since) starts over — first_seen now and no reminder yet — so its own
|
||||
// grace runs from the start.
|
||||
func (s *Store) UpsertHeldApp(customerID, stack, reason string, since time.Time, why string, now time.Time) error {
|
||||
if customerID == "" || stack == "" {
|
||||
return fmt.Errorf("held app: empty customer or stack")
|
||||
}
|
||||
res, err := s.db.Exec(`UPDATE held_apps SET last_seen = ?, why = ?
|
||||
WHERE customer_id = ? AND stack = ? AND reason = ? AND since = ?`,
|
||||
fmtT(now), why, customerID, stack, reason, fmtT(since))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if n, _ := res.RowsAffected(); n > 0 {
|
||||
return nil
|
||||
}
|
||||
_, err = s.db.Exec(`INSERT INTO held_apps (customer_id, stack, reason, since, why, first_seen, last_seen, reminded_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, '')
|
||||
ON CONFLICT(customer_id, stack) DO UPDATE SET reason = excluded.reason, since = excluded.since,
|
||||
why = excluded.why, first_seen = excluded.first_seen, last_seen = excluded.last_seen, reminded_at = ''`,
|
||||
customerID, stack, reason, fmtT(since), why, fmtT(now), fmtT(now))
|
||||
return err
|
||||
}
|
||||
|
||||
// ClearHeldApp removes a hold the box says is gone. Returns whether a row existed.
|
||||
func (s *Store) ClearHeldApp(customerID, stack string) (bool, error) {
|
||||
res, err := s.db.Exec(`DELETE FROM held_apps WHERE customer_id = ? AND stack = ?`, customerID, stack)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
n, _ := res.RowsAffected()
|
||||
return n > 0, nil
|
||||
}
|
||||
|
||||
// MarkHeldAppReminded records the operator reminder for the hold as it is now (same reason and since).
|
||||
func (s *Store) MarkHeldAppReminded(customerID, stack, reason string, since, at time.Time) error {
|
||||
_, err := s.db.Exec(`UPDATE held_apps SET reminded_at = ? WHERE customer_id = ? AND stack = ? AND reason = ? AND since = ?`,
|
||||
fmtT(at), customerID, stack, reason, fmtT(since))
|
||||
return err
|
||||
}
|
||||
|
||||
// ListHeldApps returns every held app, or one customer's when customerID is not empty, oldest hold first.
|
||||
func (s *Store) ListHeldApps(customerID string) ([]HeldApp, error) {
|
||||
q := `SELECT customer_id, stack, reason, since, why, first_seen, last_seen, reminded_at FROM held_apps`
|
||||
args := []interface{}{}
|
||||
if customerID != "" {
|
||||
q += ` WHERE customer_id = ?`
|
||||
args = append(args, customerID)
|
||||
}
|
||||
q += ` ORDER BY since, customer_id, stack`
|
||||
rows, err := s.db.Query(q, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []HeldApp
|
||||
for rows.Next() {
|
||||
var h HeldApp
|
||||
var since, first, last, rem string
|
||||
if err := rows.Scan(&h.CustomerID, &h.Stack, &h.Reason, &since, &h.Why, &first, &last, &rem); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
h.Since, h.FirstSeen, h.LastSeen, h.RemindedAt = parseT(since), parseT(first), parseT(last), parseT(rem)
|
||||
out = append(out, h)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
@@ -889,6 +889,10 @@ func (s *Store) migrate() error {
|
||||
if err := s.migrateOperatorActions(); err != nil {
|
||||
return fmt.Errorf("operator_actions: %w", err)
|
||||
}
|
||||
// D53 (`08` §10): the held apps the boxes report (held_apps.go).
|
||||
if err := s.migrateHeldApps(); err != nil {
|
||||
return fmt.Errorf("held_apps: %w", err)
|
||||
}
|
||||
// Operator access tokens (hub v0.152.0, `05` §17, api_tokens.go).
|
||||
if err := s.migrateAPITokens(); err != nil {
|
||||
return fmt.Errorf("api_tokens: %w", err)
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// D53 (`08` §10): the „Held apps" list on the host page and on the Hosts page — both branches of each {{if}} rendered
|
||||
// (the seam-wiring rule): a held app shows its app, reason and who may clear it; after the clear, the empty sentence.
|
||||
func TestD53_HeldAppsListOnTheHostPageAndTheHostsPage(t *testing.T) {
|
||||
s, st := newTestServer(t)
|
||||
if err := st.UpsertHost(&store.Host{HostID: "demo-hp-01", CustomerID: "c1", APIKey: "k1"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
now := time.Now()
|
||||
if err := st.UpsertHeldApp("c1", "kimai", "unhealthy_stop", now.Add(-3*24*time.Hour), "stopped by the box for a crash loop", now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
render := func() (string, string) {
|
||||
rr := httptest.NewRecorder()
|
||||
s.handleHostsList(rr, httptest.NewRequest(http.MethodGet, "/hosts", nil))
|
||||
hd := httptest.NewRecorder()
|
||||
s.handleHostDetail(hd, httptest.NewRequest(http.MethodGet, "/hosts/demo-hp-01", nil), "demo-hp-01")
|
||||
if rr.Code != 200 || hd.Code != 200 {
|
||||
t.Fatalf("status: hosts %d, host %d", rr.Code, hd.Code)
|
||||
}
|
||||
return rr.Body.String(), hd.Body.String()
|
||||
}
|
||||
fleet, host := render()
|
||||
for name, body := range map[string]string{"hosts": fleet, "host": host} {
|
||||
if !strings.Contains(body, "Held apps") || !strings.Contains(body, `data-held-app=`) || !strings.Contains(body, "kimai") ||
|
||||
!strings.Contains(body, "unhealthy_stop") || !strings.Contains(body, "Start (one more try)") {
|
||||
t.Errorf("%s page: the held app is not listed with its reason and who may clear it", name)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(fleet, `href="/hosts/demo-hp-01#held-apps"`) {
|
||||
t.Error("hosts page: the held app does not link to its host")
|
||||
}
|
||||
if !strings.Contains(host, "stopped by the box for a crash loop") {
|
||||
t.Error("host page: the box's sentence is missing")
|
||||
}
|
||||
if _, err := st.ClearHeldApp("c1", "kimai"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fleet, host = render()
|
||||
if strings.Contains(fleet, `data-held-app=`) || !strings.Contains(fleet, "No app is held on any box.") {
|
||||
t.Error("hosts page: after the clear the list is not empty")
|
||||
}
|
||||
if strings.Contains(host, `data-held-app=`) || !strings.Contains(host, "No app is held on this box.") {
|
||||
t.Error("host page: after the clear the list is not empty")
|
||||
}
|
||||
}
|
||||
|
||||
// A host with no customer shows no held apps — never the whole fleet's.
|
||||
func TestD53_AHostWithNoCustomerShowsNoHeldApps(t *testing.T) {
|
||||
s, st := newTestServer(t)
|
||||
_ = st.UpsertHeldApp("c9", "x", "restore_failed", time.Now(), "y", time.Now())
|
||||
if rows := s.hostHeldAppRows("", time.Now()); len(rows) != 0 {
|
||||
t.Fatalf("a customer-less host lists %d held app(s)", len(rows))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// heldAppRow is one line of the „Held apps" list (D53, `08` §10) on the host page and the Hosts page.
|
||||
type heldAppRow struct {
|
||||
CustomerID string
|
||||
HostID string // the customer's host, for the link on the Hosts page; "" when unknown
|
||||
Stack string
|
||||
Reason string
|
||||
Since time.Time
|
||||
Why string // the box's own sentence
|
||||
WhoClears string
|
||||
Reminded bool
|
||||
Stale bool // the box has not repeated it for heldAppStale — it may be down
|
||||
}
|
||||
|
||||
// heldAppStaleAfter mirrors monitor's heldAppStale: a box repeats app_held daily.
|
||||
const heldAppStaleAfter = 50 * time.Hour
|
||||
|
||||
// heldAppRows reads the held-app mirror, one customer's or every one's ("" = all). A read failure is logged and
|
||||
// shown as an empty list — the page must still render.
|
||||
func (s *Server) heldAppRows(customerID string, hostOf map[string]string, now time.Time) []heldAppRow {
|
||||
held, err := s.store.ListHeldApps(customerID)
|
||||
if err != nil {
|
||||
s.logger.Printf("[WARN] held apps: list for %q: %v", customerID, err)
|
||||
return nil
|
||||
}
|
||||
return toHeldAppRows(held, hostOf, now)
|
||||
}
|
||||
|
||||
// hostHeldAppRows is one host's list. A host with no customer has none — never the fleet's („" lists every one).
|
||||
func (s *Server) hostHeldAppRows(customerID string, now time.Time) []heldAppRow {
|
||||
if customerID == "" {
|
||||
return nil
|
||||
}
|
||||
return s.heldAppRows(customerID, nil, now)
|
||||
}
|
||||
|
||||
func toHeldAppRows(held []store.HeldApp, hostOf map[string]string, now time.Time) []heldAppRow {
|
||||
out := make([]heldAppRow, 0, len(held))
|
||||
for _, h := range held {
|
||||
since := h.Since
|
||||
if since.IsZero() {
|
||||
since = h.FirstSeen
|
||||
}
|
||||
out = append(out, heldAppRow{CustomerID: h.CustomerID, HostID: hostOf[h.CustomerID], Stack: h.Stack, Reason: h.Reason,
|
||||
Since: since, Why: h.Why, WhoClears: monitor.HeldAppWhoClears(h.Reason), Reminded: !h.RemindedAt.IsZero(),
|
||||
Stale: now.Sub(h.LastSeen) > heldAppStaleAfter})
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -658,9 +658,19 @@ func (s *Server) handleHostsList(w http.ResponseWriter, r *http.Request) {
|
||||
// non-fatal: still render the host list
|
||||
}
|
||||
|
||||
// D53 (`08` §10): every held app across the fleet, each linked to its customer's host.
|
||||
hostOf := map[string]string{}
|
||||
for _, row := range rows {
|
||||
if row.CustomerID != "" {
|
||||
hostOf[row.CustomerID] = row.HostID
|
||||
}
|
||||
}
|
||||
heldApps := s.heldAppRows("", hostOf, time.Now())
|
||||
|
||||
data := map[string]interface{}{
|
||||
"Hosts": rows,
|
||||
"Unclaimed": unclaimed,
|
||||
"HeldApps": heldApps,
|
||||
"CustomerPicker": picker,
|
||||
"Flash": r.URL.Query().Get("flash"),
|
||||
"CSRFToken": s.getCSRFToken(r),
|
||||
@@ -873,6 +883,7 @@ func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]in
|
||||
"DesiredGeneration": host.DesiredGeneration,
|
||||
"Vitals": vitals,
|
||||
"Guests": guests,
|
||||
"HeldApps": s.hostHeldAppRows(host.CustomerID, now), // D53, `08` §10
|
||||
"GuestRunning": guestRunning,
|
||||
"GuestTotal": len(guests),
|
||||
"StorageTargets": storageTargets,
|
||||
|
||||
@@ -146,6 +146,39 @@
|
||||
{{end}}
|
||||
</section>
|
||||
|
||||
<!-- Held apps (D53, `08` §10): apps the box keeps stopped until a person acts. The hub never clears one. -->
|
||||
<section class="card" id="held-apps" style="padding: 0; overflow: hidden;">
|
||||
<h2 style="padding: 1.25rem 1.25rem 0.5rem;">Held apps</h2>
|
||||
{{if .HeldApps}}
|
||||
<table class="data-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>App</th>
|
||||
<th>Reason</th>
|
||||
<th>Since</th>
|
||||
<th>Why</th>
|
||||
<th>Who may clear it</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{range .HeldApps}}
|
||||
<tr data-held-app="{{.Stack}}">
|
||||
<td><code>{{.Stack}}</code></td>
|
||||
<td><span class="severity-badge severity-warning">{{.Reason}}</span>{{if .Stale}} <span class="text-muted">(not confirmed by the box for 2 days)</span>{{end}}</td>
|
||||
<td>{{timeAgo .Since}}</td>
|
||||
<td>{{.Why}}</td>
|
||||
<td>{{.WhoClears}}{{if .Reminded}} <span class="text-muted">(operator reminded)</span>{{end}}</td>
|
||||
</tr>
|
||||
{{end}}
|
||||
</tbody>
|
||||
</table>
|
||||
{{else}}
|
||||
<div class="empty-state" style="border: none;">
|
||||
<p>No app is held on this box.</p>
|
||||
</div>
|
||||
{{end}}
|
||||
</section>
|
||||
|
||||
<!-- Storage Targets -->
|
||||
<section class="card" style="padding: 0; overflow: hidden;">
|
||||
<h2 style="padding: 1.25rem 1.25rem 0.5rem;">Storage Targets</h2>
|
||||
|
||||
@@ -34,6 +34,39 @@
|
||||
</div>
|
||||
{{end}}
|
||||
|
||||
<!-- Held apps across the fleet (D53, `08` §10). The hub never clears a hold; each row links to its host. -->
|
||||
<section class="card" id="held-apps" style="margin-bottom: 1.5rem; padding: 0; overflow: hidden;">
|
||||
<h2 style="padding: 1.25rem 1.25rem 0.5rem; margin-top: 0;">Held apps</h2>
|
||||
{{if .HeldApps}}
|
||||
<table class="data-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Customer</th>
|
||||
<th>App</th>
|
||||
<th>Reason</th>
|
||||
<th>Since</th>
|
||||
<th>Who may clear it</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{{range .HeldApps}}
|
||||
<tr data-held-app="{{.CustomerID}}/{{.Stack}}">
|
||||
<td>{{if .HostID}}<a href="/hosts/{{.HostID}}#held-apps">{{.CustomerID}}</a>{{else}}{{.CustomerID}}{{end}}</td>
|
||||
<td><code>{{.Stack}}</code></td>
|
||||
<td><span class="severity-badge severity-warning">{{.Reason}}</span>{{if .Stale}} <span class="text-muted">(not confirmed for 2 days)</span>{{end}}</td>
|
||||
<td>{{timeAgo .Since}}</td>
|
||||
<td>{{.WhoClears}}</td>
|
||||
</tr>
|
||||
{{end}}
|
||||
</tbody>
|
||||
</table>
|
||||
{{else}}
|
||||
<div class="empty-state" style="border: none;">
|
||||
<p>No app is held on any box.</p>
|
||||
</div>
|
||||
{{end}}
|
||||
</section>
|
||||
|
||||
{{if .Unclaimed}}
|
||||
<section class="card" style="margin-bottom: 1.5rem; border-color: var(--warn);">
|
||||
<h2 style="margin-top: 0;">Unclaimed appliances <span class="text-muted" style="font-size: 0.8em; font-weight: normal;">(booted from the generic ISO, awaiting a bind)</span></h2>
|
||||
|
||||
Reference in new issue
Block a user