hub(D53): held apps reach the operator — mirror, Held apps list (host + Hosts page), reminder after the grace
gates / gates (push) Successful in 1m58s

app_held / app_hold_cleared allow-listed and operator-only; handleEvent mirrors them in held_apps; HeldAppsChecker
sends ONE app_held_reminder (warning, operator-only, per-app cooldown) after the grace per reason (6 h / 1 d / 3 d),
then weekly; recorded before sending; stale rows not reminded; the hub never clears a hold. Test-only
alerting.held_app_grace_override. TestD53_* in api/monitor/notify/web, red-proved by six mutations. Commit only.
This commit is contained in:
admin committed 2026-10-26 18:25:00 +01:00
1 parent ecd82abab8
commit 7e57e824a2
27 files changed
+935 -2

No files matched your search

+1
View File
@@ -124,6 +124,7 @@
| `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. | | `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
| `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). | | `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). |
| `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. | | `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
| `monitor.HeldAppsChecker` + `store.UpsertHeldApp`/`ClearHeldApp`/`ListHeldApps` + `HeldAppGrace`/`HeldAppWhoClears` (D53, unreleased) | hub/internal/monitor/held_apps.go · hub/internal/store/held_apps.go · hub/internal/web/held_apps.go | `NewHeldAppsChecker(st, graceOverride, onEvent, logger)`; `.Check()` on the 60 s sweep | `08` §10: the boxes' held apps — the mirror (fed by `app_held`/`app_hold_cleared` in `handleEvent`), the „Held apps" list on the host page and the Hosts page, one operator reminder after the grace per reason, then weekly | **The hub never clears a hold** — a row goes only on the box's `app_hold_cleared`. The reminder is recorded BEFORE it is sent (a restart cannot mail twice); a row the box stopped repeating (> 50 h) is listed, never reminded. `app_held_reminder` is operator-only and per-app cooldown. `graceOverride` is a TEST configuration only (`alerting.held_app_grace_override`). Pinned by `TestD53_*` (monitor, api, web, notify) |
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. **R-86 (2026-08-03): the window is PER TIER, not one constant.** `restoreProvenWindow(tier, observed, ok)` = `clamp(4 × max(observed, declared), floor 7d, cap 12d)`, where `declared` is that tier's own backup-freshness threshold (`backupStaleAfter` 26 h / `offsiteBackupStaleAfter` 8 d — reuse those, never a second opinion) and `observed` comes from `observedArchiveIntervals` over the retained window. **Observation may only WIDEN**: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. `assessRestoreProven` takes the window as an argument and **every reason string names it** (R-100's corollary). | | `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. **R-86 (2026-08-03): the window is PER TIER, not one constant.** `restoreProvenWindow(tier, observed, ok)` = `clamp(4 × max(observed, declared), floor 7d, cap 12d)`, where `declared` is that tier's own backup-freshness threshold (`backupStaleAfter` 26 h / `offsiteBackupStaleAfter` 8 d — reuse those, never a second opinion) and `observed` comes from `observedArchiveIntervals` over the retained window. **Observation may only WIDEN**: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. `assessRestoreProven` takes the window as an argument and **every reason string names it** (R-100's corollary). |
| `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. | | `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
| `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. | | `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
@@ -0,0 +1,7 @@
RED internal/notify/dispatcher.go | "app_held_reminder": true, } // IsOperatorOnly | ['--- FAIL: TestD53_TheHeldAppReminderMailsTheOperatorOnly (0.04s)']
RED internal/api/handler.go | h.recordHeldApp(payload.CustomerID, payload.EventType, paylo | ['--- FAIL: TestD53_TheBoxsHeldAppsAreMirrored (0.05s)']
RED internal/monitor/held_apps.go | if age < hc.grace(h.Reason) { return } | ['--- FAIL: TestD53_OneReminderAfterTheGraceThenWeekly (0.04s)', '--- FAIL: TestD53_TheGraceFollowsTheReason (0.17s)']
RED internal/monitor/held_apps.go | if !h.RemindedAt.IsZero() && now.Sub(h.RemindedAt) < heldApp | ['--- FAIL: TestD53_OneReminderAfterTheGraceThenWeekly (0.03s)']
RED internal/web/hosts.go | "HeldApps": heldApps, | []
RED internal/web/templates/host_detail_body.html | <td>{{.WhoClears}}{{if .Reminded}} | ['--- FAIL: TestD53_HeldAppsListOnTheHostPageAndTheHostsPage (0.05s)']
RED internal/web/hosts.go | HeldApps emptied (compiling mutation) | TestD53_HeldAppsListOnTheHostPageAndTheHostsPage
+21
View File
@@ -1,3 +1,24 @@
## Unreleased (next: v0.173.0) — D53: a held app reaches the operator (`08` §10)
Operator ruling D53 (2026-10-26, `09` §3 decision 265): operator-only reminders first. Pairs with controller `5bd03ad`
(`internal/holdreport`); a box on an older controller sends nothing new, and its list stays empty.
- `internal/api/handler.go`: `app_held` and `app_hold_cleared` in `allowedEventTypes`; `handleEvent` → `recordHeldApp`
mirrors them in the new `held_apps` table (`internal/store/held_apps.go`: one row per customer + stack; a repeat moves
`last_seen` only; a new reason or since starts the row over; the clear deletes it).
- `internal/monitor/held_apps.go` `HeldAppsChecker` (60 s sweep, one pass per 15 min): ONE operator reminder
`app_held_reminder` (warning) after the grace per reason — `putback_pending` 6 h, `restore_failed`/`restore_mixed` 1 day,
`update_failed`/`unhealthy_stop` 3 days — then weekly while it lasts; recorded before it is sent; a row the box stopped
repeating (> 50 h) is not reminded. `alerting.held_app_grace_override` (a duration) replaces every grace — a TEST
configuration only, logged at start.
- `internal/notify`: all three types operator-only; `app_held_reminder` per-app cooldown and a subject that names the app;
hu + en `mail.event.*` lines (the coupling rule) and their six mail goldens.
- Web: a „Held apps" list on the host page (app, reason, since, the box's sentence, who may clear it) and on the Hosts
page (every box, linked to its host); the empty sentence when none. The hub never clears a hold.
- Tests `TestD53_*`: the mirror (api), the reminder after the grace / weekly / per reason / none after a clear / none for
a stale row / a new hold starts over (monitor), operator only and per app (notify), both pages with and without a held
app (web). Red-proved by six mutations (`documentation/audits/d53-held-apps-2026-10-26/redproof-hub.txt`).
## v0.172.0 (2026-10-26) — a fresh install's whole-guest off-site set-up never fails silently (R-1295) — RELEASED 2026-10-26 13:08 UTC on the operator's yes (image sha256:7c8d129a…, from `232f42d2`, CI 2366; rc.1 ran the live proof) ## v0.172.0 (2026-10-26) — a fresh install's whole-guest off-site set-up never fails silently (R-1295) — RELEASED 2026-10-26 13:08 UTC on the operator's yes (image sha256:7c8d129a…, from `232f42d2`, CI 2366; rc.1 ran the live proof)
- **Cause (measured, `documentation/audits/r1295-pbsdr-selfcheck-2026-10-26/`):** PBS's proxy caches permissions for up to - **Cause (measured, `documentation/audits/r1295-pbsdr-selfcheck-2026-10-26/`):** PBS's proxy caches permissions for up to
+14
View File
@@ -72,6 +72,9 @@ type Config struct {
// the host-root thresholds above so a single storage's policy can be tuned separately. // the host-root thresholds above so a single storage's policy can be tuned separately.
StorageFillWarnPercent float64 `yaml:"storage_fill_warn_percent"` StorageFillWarnPercent float64 `yaml:"storage_fill_warn_percent"`
StorageFillCritPercent float64 `yaml:"storage_fill_crit_percent"` StorageFillCritPercent float64 `yaml:"storage_fill_crit_percent"`
// HeldAppGraceOverride (D53) replaces every reason's reminder grace when set (a Go duration, e.g. "3m"). A TEST
// configuration only — the live proof shortens the wait; production leaves it empty (`08` §10's per-reason grace).
HeldAppGraceOverride string `yaml:"held_app_grace_override"`
// Offsite POOL-BOX aggregate thresholds (v0.64.0, R-5). Fill = box used vs capacity (percent); // Offsite POOL-BOX aggregate thresholds (v0.64.0, R-5). Fill = box used vs capacity (percent);
// oversub = Σ(shared soft quotas)/capacity (ratio). Empty/0/invalid → defaults 80/90/2.0 (the // oversub = Σ(shared soft quotas)/capacity (ratio). Empty/0/invalid → defaults 80/90/2.0 (the
// checker normalizes). Threshold VALUES are Claude's encoding of the starter suggestion — Viktor's // checker normalizes). Threshold VALUES are Claude's encoding of the starter suggestion — Viktor's
@@ -899,6 +902,16 @@ func main() {
// durable (events table), so a hub restart neither floods nor silently re-heals. // durable (events table), so a hub restart neither floods nor silently re-heals.
// R-523: the agent's in-guest controller supervisor → controller_restarted_by_agent / controller_crashloop. // R-523: the agent's in-guest controller supervisor → controller_restarted_by_agent / controller_crashloop.
controllerSupervisorChecker := monitor.NewControllerSupervisorChecker(dataStore, dispatcher.ProcessEvent, logger) controllerSupervisorChecker := monitor.NewControllerSupervisorChecker(dataStore, dispatcher.ProcessEvent, logger)
// D53 (`08` §10): a held app reaches the OPERATOR — one reminder after the grace per reason, then weekly.
var heldGrace time.Duration
if v := cfg.Alerting.HeldAppGraceOverride; v != "" {
if d, err := time.ParseDuration(v); err == nil && d > 0 {
heldGrace = d
} else {
logger.Printf("[WARN] alerting.held_app_grace_override %q is not a positive duration — ignored", v)
}
}
heldAppsChecker := monitor.NewHeldAppsChecker(dataStore, heldGrace, dispatcher.ProcessEvent, logger)
offsiteDeliveryChecker := monitor.NewOffsiteDeliveryChecker(dataStore, offsiteHealReissuer, dispatcher.ProcessEvent, logger) offsiteDeliveryChecker := monitor.NewOffsiteDeliveryChecker(dataStore, offsiteHealReissuer, dispatcher.ProcessEvent, logger)
// R-242 second half (S4, decision 199): published golden vs vouched golden. Needs the registry creds // R-242 second half (S4, decision 199): published golden vs vouched golden. Needs the registry creds
// (the same Gitea client as the Day-0 dropdowns); without them the check is off and says so. // (the same Gitea client as the Day-0 dropdowns); without them the check is off and says so.
@@ -931,6 +944,7 @@ func main() {
restoreTestChecker.Check() // R-85: restore-test failure + per-tier staleness restoreTestChecker.Check() // R-85: restore-test failure + per-tier staleness
offsiteDeliveryChecker.Check() offsiteDeliveryChecker.Check()
controllerSupervisorChecker.Check() controllerSupervisorChecker.Check()
heldAppsChecker.Check() // D53: throttled to one pass per 15 min
if goldenVouchChecker != nil { if goldenVouchChecker != nil {
goldenVouchChecker.Check() // R-242: throttled to one Gitea listing per 15 min goldenVouchChecker.Check() // R-242: throttled to one Gitea listing per 15 min
} }
+45
View File
@@ -0,0 +1,45 @@
package api
import (
"testing"
"gitea.dooplex.hu/admin/felhom-hub/internal/notify"
)
// D53 (`08` §10): a box's app_held is accepted and mirrored (stack, reason, since); its daily repeat keeps ONE row; its
// app_hold_cleared removes the row. Both types are allow-listed AND operator-only (the v0.78.0 pair, checked together).
func TestD53_TheBoxsHeldAppsAreMirrored(t *testing.T) {
h, st, _ := newTestHandler(t)
for _, et := range []string{"app_held", "app_hold_cleared"} {
if !allowedEventTypes[et] {
t.Fatalf("%s is not allow-listed — POST /event would 400", et)
}
if !notify.IsOperatorOnly(et) {
t.Fatalf("%s is not operator-only", et)
}
}
if !notify.IsOperatorOnly("app_held_reminder") || allowedEventTypes["app_held_reminder"] {
t.Fatal("app_held_reminder must be operator-only and hub-origin (a box must not post the reminder)")
}
held := `{"customer_id":"c1","event_type":"app_held","severity":"info","message":"App kimai is held",
"details":{"stack_name":"kimai","reason":"unhealthy_stop","since":"2026-10-14T08:00:00Z","why":"stopped by the box"}}`
for i := 0; i < 2; i++ { // the first report and the daily repeat
if rr := do(h, "POST", "/event", globalKey, held); rr.Code != 200 {
t.Fatalf("app_held: %d %s", rr.Code, rr.Body.String())
}
}
rows, err := st.ListHeldApps("c1")
if err != nil || len(rows) != 1 {
t.Fatalf("want one held row, got %+v (%v)", rows, err)
}
if r := rows[0]; r.Stack != "kimai" || r.Reason != "unhealthy_stop" || r.Since.Format("2006-01-02T15:04:05Z07:00") != "2026-10-14T08:00:00Z" || r.Why == "" {
t.Fatalf("row: %+v", r)
}
cleared := `{"customer_id":"c1","event_type":"app_hold_cleared","severity":"info","message":"App kimai is no longer held","details":{"stack_name":"kimai"}}`
if rr := do(h, "POST", "/event", globalKey, cleared); rr.Code != 200 {
t.Fatalf("app_hold_cleared: %d", rr.Code)
}
if rows, _ := st.ListHeldApps("c1"); len(rows) != 0 {
t.Fatalf("the cleared hold is still listed: %+v", rows)
}
}
+49
View File
@@ -2262,6 +2262,12 @@ var allowedEventTypes = map[string]bool{
// v0.123.0 (decision 28, controller v0.269.0): the box stopped an app that kept crashing / ran out // v0.123.0 (decision 28, controller v0.269.0): the box stopped an app that kept crashing / ran out
// of memory. Household + operator; hu AND en `mail.event.*`; per-app cooldown on both legs. // of memory. Household + operator; hu AND en `mail.event.*`; per-app cooldown on both legs.
"app_stopped_unhealthy": true, "app_stopped_unhealthy": true,
// D53 (`08` §10, controller with internal/holdreport): the box's held apps — app_held for each hold in place (at once,
// then daily), app_hold_cleared when one is gone. Severity info: never mailed on receipt; the hub mirrors them in
// held_apps (handleEvent → recordHeldApp) and the reminder checker mails the OPERATOR after a grace. Operator-only
// (notify.operatorOnlyEvents); hu AND en `mail.event.*` lines exist only because the coupling rule needs them.
"app_held": true,
"app_hold_cleared": true,
// R-871 (v0.134.0, controller v0.295.0, `09` decision 109): the box was off at its backup time and made the // R-871 (v0.134.0, controller v0.295.0, `09` decision 109): the box was off at its backup time and made the
// missed backups when it came back. Info — recorded on the box's timeline, never mailed. // missed backups when it came back. Info — recorded on the box's timeline, never mailed.
@@ -2520,6 +2526,12 @@ func (h *Handler) handleEvent(w http.ResponseWriter, r *http.Request) {
h.logger.Printf("[INFO] Event from %s: %s (%s) — %s", payload.CustomerID, payload.EventType, payload.Severity, payload.Message) h.logger.Printf("[INFO] Event from %s: %s (%s) — %s", payload.CustomerID, payload.EventType, payload.Severity, payload.Message)
// D53: mirror the box's held apps. A failure is logged and the event still dispatched — the box repeats
// app_held daily, so the next one heals the mirror.
if payload.EventType == "app_held" || payload.EventType == "app_hold_cleared" {
h.recordHeldApp(payload.CustomerID, payload.EventType, payload.Details)
}
// Dispatch notifications (non-blocking) // Dispatch notifications (non-blocking)
if h.dispatcher != nil { if h.dispatcher != nil {
go h.dispatcher.ProcessBoxEvent(payload.CustomerID, payload.EventType, payload.Severity, go h.dispatcher.ProcessBoxEvent(payload.CustomerID, payload.EventType, payload.Severity,
@@ -3189,3 +3201,40 @@ func (h *Handler) checkTunnel(hostID, custID, status, detail string) {
go h.dispatcher.ProcessEvent(custID, typ, sev, msg, string(details), "hub") go h.dispatcher.ProcessEvent(custID, typ, sev, msg, string(details), "hub")
} }
} }
// recordHeldApp applies one app_held / app_hold_cleared to the held_apps mirror (D53, `08` §10). The hub never
// clears a hold on a box; it only forgets one the box says is gone.
func (h *Handler) recordHeldApp(customerID, eventType string, raw json.RawMessage) {
var d struct {
StackName string `json:"stack_name"`
Reason string `json:"reason"`
Since string `json:"since"`
Why string `json:"why"`
}
if err := json.Unmarshal(raw, &d); err != nil || d.StackName == "" {
h.logger.Printf("[WARN] [held-apps] %s from %s has no readable stack_name — not recorded", eventType, customerID)
return
}
if eventType == "app_hold_cleared" {
had, err := h.store.ClearHeldApp(customerID, d.StackName)
if err != nil {
h.logger.Printf("[WARN] [held-apps] clear %s/%s: %v", customerID, d.StackName, err)
return
}
h.logger.Printf("[INFO] [held-apps] %s/%s no longer held (a row existed: %v)", customerID, d.StackName, had)
return
}
var since time.Time
if d.Since != "" {
if t, err := time.Parse(time.RFC3339, d.Since); err == nil {
since = t
} else {
h.logger.Printf("[WARN] [held-apps] %s/%s: since %q is not RFC3339 — the hub's first sighting is the clock", customerID, d.StackName, d.Since)
}
}
if err := h.store.UpsertHeldApp(customerID, d.StackName, d.Reason, since, d.Why, time.Now()); err != nil {
h.logger.Printf("[WARN] [held-apps] record %s/%s: %v", customerID, d.StackName, err)
return
}
h.logger.Printf("[DEBUG] [held-apps] %s/%s held (%s since %s)", customerID, d.StackName, d.Reason, d.Since)
}
+3
View File
@@ -89,6 +89,9 @@
"mail.event.app_update_undone": "%s: the update did not work; the app runs on its previous version", "mail.event.app_update_undone": "%s: the update did not work; the app runs on its previous version",
"mail.event.app_update_held": "%s: the app is stopped and needs a restore", "mail.event.app_update_held": "%s: the app is stopped and needs a restore",
"mail.event.app_stopped_unhealthy": "%s: the app was stopped because it kept crashing", "mail.event.app_stopped_unhealthy": "%s: the app was stopped because it kept crashing",
"mail.event.app_held": "An app is stopped and waits for a person to act",
"mail.event.app_hold_cleared": "The app is no longer held",
"mail.event.app_held_reminder": "An app is still held",
"bind.resend.hint": "If this was your link and your box is not linked yet, we send a fresh link to the e-mail address you gave Felhom.", "bind.resend.hint": "If this was your link and your box is not linked yet, we send a fresh link to the e-mail address you gave Felhom.",
"bind.resend.button": "Send me a new link", "bind.resend.button": "Send me a new link",
"bind.resend.newer": "If you have a newer mail from Felhom, use the link in that one.", "bind.resend.newer": "If you have a newer mail from Felhom, use the link in that one.",
+3
View File
@@ -89,6 +89,9 @@
"mail.event.app_update_undone": "%s: a frissítés nem sikerült, az alkalmazás a korábbi változattal fut", "mail.event.app_update_undone": "%s: a frissítés nem sikerült, az alkalmazás a korábbi változattal fut",
"mail.event.app_update_held": "%s: az alkalmazás leállítva, visszaállítás szükséges", "mail.event.app_update_held": "%s: az alkalmazás leállítva, visszaállítás szükséges",
"mail.event.app_stopped_unhealthy": "%s: az alkalmazást leállítottuk, mert újra és újra összeomlott", "mail.event.app_stopped_unhealthy": "%s: az alkalmazást leállítottuk, mert újra és újra összeomlott",
"mail.event.app_held": "Egy alkalmazás le van állítva, és emberi beavatkozásra vár",
"mail.event.app_hold_cleared": "Az alkalmazás leállítása megszűnt",
"mail.event.app_held_reminder": "Egy alkalmazás még mindig le van állítva",
"bind.resend.hint": "Ha ez a te linked volt, és a dobozod még nincs összekötve, új linket küldünk arra az e-mail címre, amelyet a Felhomnál megadtál.", "bind.resend.hint": "Ha ez a te linked volt, és a dobozod még nincs összekötve, új linket küldünk arra az e-mail címre, amelyet a Felhomnál megadtál.",
"bind.resend.button": "Új linket kérek", "bind.resend.button": "Új linket kérek",
"bind.resend.newer": "Ha van újabb leveled a Felhomtól, abban lévő linket használd.", "bind.resend.newer": "Ha van újabb leveled a Felhomtól, abban lévő linket használd.",
+174
View File
@@ -0,0 +1,174 @@
package monitor
import (
"encoding/json"
"fmt"
"log"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// ── D53 (`08` §10, R-1291) — A HELD APP REACHES THE OPERATOR ──────────────────────────────────────
//
// A held app is stopped on purpose and stays stopped until a person acts. demo-hp's Kimai sat held for 11 days
// (2026-10-14 → 10-26) and nobody was told after the first minute. The box now reports its holds (app_held /
// app_hold_cleared, severity info — nothing is mailed on receipt); this checker turns a hold that LASTS into ONE
// operator mail after a grace that follows the reason, then one a week while it lasts.
//
// THE GRACE (`08` §10): putback_pending 6 h — it should have cleared itself, so it is stuck; restore_failed and
// restore_mixed 1 day — only a person can act; update_failed and unhealthy_stop 3 days — the household can act first.
// An unknown reason gets 1 day. The clock is the hold's own time (the box's), else when the hub first heard of it.
//
// OPERATOR ONLY (D53): app_held_reminder is in the dispatcher's operatorOnlyEvents, so no household is ever mailed.
// THE HUB NEVER CLEARS A HOLD — a row goes only when the box sends app_hold_cleared.
//
// A ROW THE BOX STOPPED CONFIRMING is not reminded: the box repeats app_held daily, so a row older than heldAppStale
// belongs to a box that is down or gone, which its own staleness alarms report. The list still shows it.
//
// The reminder is recorded (MarkHeldAppReminded) BEFORE it is sent, so a hub restart cannot mail twice.
// Pinned by held_apps_test.go.
const (
eventHeldAppReminder = "app_held_reminder"
heldAppRepeat = 7 * 24 * time.Hour
heldAppStale = 50 * time.Hour
heldAppCheckEvery = 15 * time.Minute
)
// HeldAppGrace returns the wait before the first reminder for a reason (`08` §10).
func HeldAppGrace(reason string) time.Duration {
switch reason {
case "putback_pending":
return 6 * time.Hour
case "update_failed", "unhealthy_stop":
return 72 * time.Hour
default: // restore_failed, restore_mixed, and a reason this hub does not know yet
return 24 * time.Hour
}
}
// HeldAppWhoClears says who may lift a hold, in the operator's words.
func HeldAppWhoClears(reason string) string {
switch reason {
case "update_failed":
return "the household, by restoring the app from its backup"
case "unhealthy_stop":
return "the household's Start (one more try)"
case "putback_pending":
return "nobody — it clears itself when the drive with the pre-restore copy comes back"
default:
return "a person only (support: the controller's -clear-restore-hold)"
}
}
type heldAppsStore interface {
ListHeldApps(customerID string) ([]store.HeldApp, error)
MarkHeldAppReminded(customerID, stack, reason string, since, at time.Time) error
SaveEvent(customerID, eventType, severity, message, detailsJSON, source string) (int64, error)
}
// HeldAppsChecker sends the operator's reminders for held apps.
type HeldAppsChecker struct {
store heldAppsStore
onEvent EventNotifyFunc
logger *log.Logger
now func() time.Time
every time.Duration
// graceOverride, when > 0, replaces every reason's grace. A TEST configuration only (the live proof shortens
// the wait); production leaves it 0. Logged at start when set.
graceOverride time.Duration
mu sync.Mutex
lastRun time.Time
}
// NewHeldAppsChecker builds the checker. graceOverride 0 = the per-reason grace.
func NewHeldAppsChecker(st heldAppsStore, graceOverride time.Duration, onEvent EventNotifyFunc, logger *log.Logger) *HeldAppsChecker {
if graceOverride > 0 && logger != nil {
logger.Printf("[WARN] Held apps: the reminder grace is OVERRIDDEN to %s for every reason — a test configuration", graceOverride)
}
return &HeldAppsChecker{store: st, onEvent: onEvent, logger: logger, now: time.Now, every: heldAppCheckEvery, graceOverride: graceOverride}
}
func (hc *HeldAppsChecker) grace(reason string) time.Duration {
if hc.graceOverride > 0 {
return hc.graceOverride
}
return HeldAppGrace(reason)
}
// Check runs on the hub's 60 s sweep and throttles itself to one pass per heldAppCheckEvery.
func (hc *HeldAppsChecker) Check() {
hc.mu.Lock()
defer hc.mu.Unlock()
now := hc.now()
every := hc.every
if hc.graceOverride > 0 && hc.graceOverride < every {
every = hc.graceOverride // a test grace of minutes must not wait a 15-minute pass
}
if !hc.lastRun.IsZero() && now.Sub(hc.lastRun) < every {
return
}
hc.lastRun = now
held, err := hc.store.ListHeldApps("")
if err != nil {
hc.logger.Printf("[WARN] Held apps: list failed: %v — no reminder this pass", err)
return
}
for _, h := range held {
hc.checkOne(h, now)
}
}
func (hc *HeldAppsChecker) checkOne(h store.HeldApp, now time.Time) {
if now.Sub(h.LastSeen) > heldAppStale {
hc.logger.Printf("[DEBUG] Held apps: %s/%s not confirmed since %s — no reminder (the box is not reporting)",
h.CustomerID, h.Stack, h.LastSeen.Format(time.RFC3339))
return
}
anchor := h.Since
if anchor.IsZero() {
anchor = h.FirstSeen
}
age := now.Sub(anchor)
if age < hc.grace(h.Reason) {
return
}
if !h.RemindedAt.IsZero() && now.Sub(h.RemindedAt) < heldAppRepeat {
return
}
if err := hc.store.MarkHeldAppReminded(h.CustomerID, h.Stack, h.Reason, h.Since, now); err != nil {
hc.logger.Printf("[WARN] Held apps: could not record the reminder for %s/%s (%v) — not sending, so a restart cannot mail twice",
h.CustomerID, h.Stack, err)
return
}
repeat := !h.RemindedAt.IsZero()
msg := fmt.Sprintf("Customer %s: the app %s has been held for %s (%s): %s. Who may clear it: %s. The hub never clears a hold.",
h.CustomerID, h.Stack, roundAge(age), h.Reason, h.Why, HeldAppWhoClears(h.Reason))
if repeat {
msg += " Weekly reminder: it is still held."
}
details, _ := json.Marshal(map[string]any{
"customer_id": h.CustomerID, "stack_name": h.Stack, "reason": h.Reason,
"since": anchor.UTC().Format(time.RFC3339), "why": h.Why, "repeat": repeat,
})
hc.logger.Printf("[INFO] Held apps: reminder for %s/%s (%s, held %s, repeat=%v)", h.CustomerID, h.Stack, h.Reason, roundAge(age), repeat)
if _, err := hc.store.SaveEvent(h.CustomerID, eventHeldAppReminder, "warning", msg, string(details), "hub"); err != nil {
hc.logger.Printf("[WARN] Failed to save %s for %s: %v", eventHeldAppReminder, h.CustomerID, err)
return
}
if hc.onEvent != nil {
hc.onEvent(h.CustomerID, eventHeldAppReminder, "warning", msg, string(details), "hub")
}
}
func roundAge(d time.Duration) string {
if d >= 48*time.Hour {
return fmt.Sprintf("%d days", int(d/(24*time.Hour)))
}
if d >= time.Hour {
return d.Round(time.Hour).String()
}
return d.Round(time.Minute).String()
}
+123
View File
@@ -0,0 +1,123 @@
package monitor
import (
"io"
"log"
"path/filepath"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
type heldEv struct{ customer, typ, sev, details string }
func heldSetup(t *testing.T) (*store.Store, *HeldAppsChecker, *[]heldEv, *time.Time) {
t.Helper()
st, err := store.New(filepath.Join(t.TempDir(), "h.db"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { st.Close() })
var evs []heldEv
now := time.Date(2026, 10, 26, 12, 0, 0, 0, time.UTC)
hc := NewHeldAppsChecker(st, 0, func(c, typ, sev, msg, details, src string) {
evs = append(evs, heldEv{c, typ, sev, details})
}, log.New(io.Discard, "", 0))
hc.now = func() time.Time { return now }
hc.every = 0
return st, hc, &evs, &now
}
// The reminder after the grace (D53): a crash-loop hold waits 3 days, then ONE warning reminder to the operator, then
// nothing until a week has passed, then one more.
func TestD53_OneReminderAfterTheGraceThenWeekly(t *testing.T) {
st, hc, evs, now := heldSetup(t)
since := now.Add(-time.Hour)
if err := st.UpsertHeldApp("c1", "kimai", "unhealthy_stop", since, "stopped by the box", *now); err != nil {
t.Fatal(err)
}
hc.Check()
if len(*evs) != 0 {
t.Fatalf("a reminder inside the 3-day grace: %+v", *evs)
}
for _, step := range []time.Duration{72 * time.Hour, time.Hour, 24 * time.Hour} {
*now = now.Add(step)
_ = st.UpsertHeldApp("c1", "kimai", "unhealthy_stop", since, "stopped by the box", *now) // the box's daily repeat
hc.Check()
}
if len(*evs) != 1 || (*evs)[0].typ != "app_held_reminder" || (*evs)[0].sev != "warning" || (*evs)[0].customer != "c1" {
t.Fatalf("want exactly one warning app_held_reminder after the grace, got %+v", *evs)
}
*now = now.Add(7 * 24 * time.Hour)
_ = st.UpsertHeldApp("c1", "kimai", "unhealthy_stop", since, "stopped by the box", *now)
hc.Check()
if len(*evs) != 2 {
t.Fatalf("want the weekly repeat, got %d events", len(*evs))
}
}
// No reminder after a clear: the box's app_hold_cleared removes the row, and nothing is mailed for it.
func TestD53_NoReminderAfterAClear(t *testing.T) {
st, hc, evs, now := heldSetup(t)
_ = st.UpsertHeldApp("c1", "db", "restore_failed", now.Add(-2*time.Hour), "held after a failed restore", *now)
if ok, err := st.ClearHeldApp("c1", "db"); err != nil || !ok {
t.Fatalf("clear: %v %v", ok, err)
}
*now = now.Add(30 * 24 * time.Hour)
hc.Check()
if len(*evs) != 0 {
t.Fatalf("a cleared hold was reminded: %+v", *evs)
}
if rows, _ := st.ListHeldApps(""); len(rows) != 0 {
t.Fatalf("the list still holds a cleared app: %+v", rows)
}
}
// The grace follows the reason (`08` §10): put-back 6 h, restore 1 day, update / crash loop 3 days.
func TestD53_TheGraceFollowsTheReason(t *testing.T) {
cases := map[string]time.Duration{"putback_pending": 6 * time.Hour, "restore_failed": 24 * time.Hour,
"restore_mixed": 24 * time.Hour, "update_failed": 72 * time.Hour, "unhealthy_stop": 72 * time.Hour}
for reason, grace := range cases {
st, hc, evs, now := heldSetup(t)
_ = st.UpsertHeldApp("c1", "a", reason, *now, "x", *now)
*now = now.Add(grace - time.Minute)
_ = st.UpsertHeldApp("c1", "a", reason, now.Add(-(grace - time.Minute)), "x", *now)
hc.Check()
if len(*evs) != 0 {
t.Errorf("%s: reminded before its grace %s", reason, grace)
}
*now = now.Add(2 * time.Minute)
hc.Check()
if len(*evs) != 1 {
t.Errorf("%s: no reminder just after its grace %s (%d events)", reason, grace, len(*evs))
}
}
}
// A box that stopped confirming its hold (down or gone) is not reminded about it; its own staleness alarms speak.
func TestD53_AStaleRowIsNotReminded(t *testing.T) {
st, hc, evs, now := heldSetup(t)
_ = st.UpsertHeldApp("c1", "a", "restore_failed", now.Add(-10*24*time.Hour), "x", now.Add(-3*24*time.Hour))
hc.Check()
if len(*evs) != 0 {
t.Fatalf("a row the box stopped confirming was reminded: %+v", *evs)
}
}
// A new hold on the same app (another reason) starts its own grace: the old reminder does not carry over.
func TestD53_ANewHoldOnTheSameAppStartsOver(t *testing.T) {
st, hc, evs, now := heldSetup(t)
_ = st.UpsertHeldApp("c1", "a", "putback_pending", now.Add(-7*time.Hour), "x", *now)
hc.Check()
if len(*evs) != 1 {
t.Fatalf("want the put-back reminder, got %d", len(*evs))
}
_ = st.UpsertHeldApp("c1", "a", "restore_mixed", *now, "y", *now)
*now = now.Add(25 * time.Hour)
_ = st.UpsertHeldApp("c1", "a", "restore_mixed", now.Add(-25*time.Hour), "y", *now)
hc.Check()
if len(*evs) != 2 {
t.Fatalf("the new hold's reminder was swallowed by the old one's week: %d events", len(*evs))
}
}
@@ -0,0 +1,43 @@
package notify
import (
"strings"
"testing"
)
// D53 (operator ruling 2026-10-26: operator-only reminders first): the held-app reminder reaches the OPERATOR — one mail
// whose subject names the app — and never the household, even with every event switched on (the R-97c configuration).
// The box's own app_held / app_hold_cleared are info: they mail nobody on either leg.
func TestD53_TheHeldAppReminderMailsTheOperatorOnly(t *testing.T) {
st := opOnlyStore(t)
if err := st.SaveNotificationPrefs("c1", "customer@example.com", []string{"app_held_reminder", "app_held", "app_hold_cleared", "backup_failed"}, 6); err != nil {
t.Fatal(err)
}
var to, subjects []string
d := opOnlyDispatcher(t, st, &sentTo{})
d.sendEmailFn = func(addr, subject, body string, headers map[string]string) error {
to = append(to, addr)
subjects = append(subjects, subject)
return nil
}
d.ProcessBoxEvent("c1", "app_held", "info", "App kimai is held: …", "", `{"stack_name":"kimai","reason":"unhealthy_stop"}`, "controller")
d.ProcessBoxEvent("c1", "app_hold_cleared", "info", "App kimai is no longer held", "", `{"stack_name":"kimai"}`, "controller")
if len(to) != 0 {
t.Fatalf("the box's held-app events mailed %v — they must be recorded, never mailed", to)
}
d.ProcessEvent("c1", "app_held_reminder", "warning", "Customer c1: the app kimai has been held for 3 days …",
`{"stack_name":"kimai","reason":"unhealthy_stop"}`, "hub")
d.ProcessEvent("c1", "app_held_reminder", "warning", "Customer c1: the app paperless has been held for 3 days …",
`{"stack_name":"paperless","reason":"update_failed"}`, "hub")
for _, a := range to {
if a == "customer@example.com" {
t.Fatal("the household was mailed an operator-only held-app reminder")
}
}
if len(to) != 2 {
t.Fatalf("want 2 operator mails (two apps, per-app cooldown), got %d: %v", len(to), to)
}
if !strings.Contains(subjects[0], "kimai is still held") || !strings.Contains(subjects[1], "paperless is still held") {
t.Fatalf("each subject must name its app: %q", subjects)
}
}
+7
View File
@@ -501,6 +501,8 @@ var perAppCooldownEvents = map[string]bool{
// v0.169.0 (the upload watch): one app's unexplained upload or CPU, no digest — two hacked apps are two alarms. // v0.169.0 (the upload watch): one app's unexplained upload or CPU, no digest — two hacked apps are two alarms.
"app_unexplained_upload": true, "app_unexplained_upload": true,
"app_unexplained_cpu": true, "app_unexplained_cpu": true,
// D53: one held app's reminder, no digest — two apps held on one box are two reminders.
"app_held_reminder": true,
} }
// perAppCustomerCooldownEvents is the CUSTOMER-leg sibling of perAppCooldownEvents (v0.120.0). The // perAppCustomerCooldownEvents is the CUSTOMER-leg sibling of perAppCooldownEvents (v0.120.0). The
@@ -938,6 +940,11 @@ var operatorOnlyEvents = map[string]bool{
// R-1295 (v0.172.0): the WG-registration hook could not set up a box's whole-guest off-site tier (web/pbsdr.go). // R-1295 (v0.172.0): the WG-registration hook could not set up a box's whole-guest off-site tier (web/pbsdr.go).
// Host ids and an instruction only the operator can carry out. Hub-origin only. // Host ids and an instruction only the operator can carry out. Hub-origin only.
"pbsdr_provision_failed": true, "pbsdr_provision_failed": true,
// D53 (`08` §10, operator ruling 2026-10-26: operator-only reminders first). The box's held-app events are info
// and never routed; the hub's reminder after the grace is the operator's mail. Listed in the SAME commit.
"app_held": true,
"app_hold_cleared": true,
"app_held_reminder": true,
} }
// IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the // IsOperatorOnly reports whether an event type is barred from customer dispatch. Exported so the
@@ -73,7 +73,8 @@ func TestR389_TheAllowListHasExactlyOneMember(t *testing.T) {
// v0.123.0 once more (decision 28): an unhealthy stop is one app's event with no digest. // v0.123.0 once more (decision 28): an unhealthy stop is one app's event with no digest.
// v0.150.0 once more (R-1007): an app's repair after an unclean boot is one app's event with no digest. // v0.150.0 once more (R-1007): an app's repair after an unclean boot is one app's event with no digest.
// v0.169.0 once more (the upload watch): an app's unexplained upload or CPU is one app's event with no digest. // v0.169.0 once more (the upload watch): an app's unexplained upload or CPU is one app's event with no digest.
want := []string{"app_hold_no_whole_copy", "app_oom_storm", "app_repaired_after_crash", "app_start_failed", "app_stopped_unhealthy", "app_unexplained_cpu", "app_unexplained_upload", "app_update_held", "app_update_undone"} // D53 once more: a held app's reminder is one app's event with no digest — two held apps are two reminders.
want := []string{"app_held_reminder", "app_hold_no_whole_copy", "app_oom_storm", "app_repaired_after_crash", "app_start_failed", "app_stopped_unhealthy", "app_unexplained_cpu", "app_unexplained_upload", "app_update_held", "app_update_undone"}
ok := len(perAppCooldownEvents) == len(want) ok := len(perAppCooldownEvents) == len(want)
for _, w := range want { for _, w := range want {
ok = ok && perAppCooldownEvents[w] ok = ok && perAppCooldownEvents[w]
@@ -83,7 +84,7 @@ func TestR389_TheAllowListHasExactlyOneMember(t *testing.T) {
for k := range perAppCooldownEvents { for k := range perAppCooldownEvents {
got = append(got, k) got = append(got, k)
} }
t.Fatalf("perAppCooldownEvents = %v, want exactly [app_hold_no_whole_copy app_oom_storm app_repaired_after_crash app_start_failed app_stopped_unhealthy app_unexplained_cpu app_unexplained_upload app_update_held app_update_undone]. Adding a member is the "+ t.Fatalf("perAppCooldownEvents = %v, want exactly [app_held_reminder app_hold_no_whole_copy app_oom_storm app_repaired_after_crash app_start_failed app_stopped_unhealthy app_unexplained_cpu app_unexplained_upload app_update_held app_update_undone]. Adding a member is the "+
"fenced act: the backup family's cooldown is coarse ON PURPOSE (R-97a, R-182) so one full "+ "fenced act: the backup family's cooldown is coarse ON PURPOSE (R-97a, R-182) so one full "+
"disk sends one digest, not one mail per app. Read the fence before widening this.", got) "disk sends one digest, not one mail per app. Read the fence before widening this.", got)
} }
+6
View File
@@ -48,6 +48,12 @@ func FormatOperatorEmail(customerID, eventType, severity, message, detailsJSON s
if eventType == "pbsdr_provision_failed" { if eventType == "pbsdr_provision_failed" {
subject = fmt.Sprintf("[Felhom] %s %s: the whole-box off-site copy (PBS DR) could not be set up — save the customer config to retry", icon, customerID) subject = fmt.Sprintf("[Felhom] %s %s: the whole-box off-site copy (PBS DR) could not be set up — save the customer config to retry", icon, customerID)
} }
// D53: the subject names the held app, so two reminders are told apart in the inbox.
if eventType == "app_held_reminder" {
if st := stackNameOf(detailsJSON); st != "" {
subject = fmt.Sprintf("[Felhom] %s %s: the app %s is still held — a person must act", icon, customerID, st)
}
}
now := nowFn().In(budapest).Format("2006-01-02 15:04 MST") now := nowFn().In(budapest).Format("2006-01-02 15:04 MST")
body := fmt.Sprintf(`Customer: %s body := fmt.Sprintf(`Customer: %s
@@ -0,0 +1,18 @@
SUBJECT: [Felhom] Warning: An app is stopped and waits for a person to act
---
Dear Customer,
Your Felhom system sent the following notification:
An app is stopped and waits for a person to act
Details:
- Server: demo-fixture
- Time: 2026-01-15 10:30
- Level: Warning
- Type: app_held
If you have any questions, contact your operator.
Best regards,
Felhom.eu monitoring
@@ -0,0 +1,18 @@
SUBJECT: [Felhom] Warning: An app is still held
---
Dear Customer,
Your Felhom system sent the following notification:
An app is still held
Details:
- Server: demo-fixture
- Time: 2026-01-15 10:30
- Level: Warning
- Type: app_held_reminder
If you have any questions, contact your operator.
Best regards,
Felhom.eu monitoring
@@ -0,0 +1,18 @@
SUBJECT: [Felhom] Warning: The app is no longer held
---
Dear Customer,
Your Felhom system sent the following notification:
The app is no longer held
Details:
- Server: demo-fixture
- Time: 2026-01-15 10:30
- Level: Warning
- Type: app_hold_cleared
If you have any questions, contact your operator.
Best regards,
Felhom.eu monitoring
@@ -0,0 +1,18 @@
SUBJECT: [Felhom] Figyelmeztetés: Egy alkalmazás le van állítva, és emberi beavatkozásra vár
---
Kedves Ügyfél!
A Felhom rendszered a következő értesítést küldte:
Egy alkalmazás le van állítva, és emberi beavatkozásra vár
Részletek:
- Szerver: demo-fixture
- Időpont: 2026-01-15 10:30
- Szint: Figyelmeztetés
- Típus: app_held
Ha kérdésed van, vedd fel a kapcsolatot az üzemeltetővel.
Üdvözlettel,
Felhom.eu monitoring
@@ -0,0 +1,18 @@
SUBJECT: [Felhom] Figyelmeztetés: Egy alkalmazás még mindig le van állítva
---
Kedves Ügyfél!
A Felhom rendszered a következő értesítést küldte:
Egy alkalmazás még mindig le van állítva
Részletek:
- Szerver: demo-fixture
- Időpont: 2026-01-15 10:30
- Szint: Figyelmeztetés
- Típus: app_held_reminder
Ha kérdésed van, vedd fel a kapcsolatot az üzemeltetővel.
Üdvözlettel,
Felhom.eu monitoring
@@ -0,0 +1,18 @@
SUBJECT: [Felhom] Figyelmeztetés: Az alkalmazás leállítása megszűnt
---
Kedves Ügyfél!
A Felhom rendszered a következő értesítést küldte:
Az alkalmazás leállítása megszűnt
Részletek:
- Szerver: demo-fixture
- Időpont: 2026-01-15 10:30
- Szint: Figyelmeztetés
- Típus: app_hold_cleared
Ha kérdésed van, vedd fel a kapcsolatot az üzemeltetővel.
Üdvözlettel,
Felhom.eu monitoring
+124
View File
@@ -0,0 +1,124 @@
package store
import (
"fmt"
"time"
)
// ── D53 (`08` §10, R-1291): the held apps a box reports ───────────────────────────────────────────
//
// A box (controller ≥ the D53 release) sends app_held for every hold in place — at once, then once a day — and
// app_hold_cleared when one is gone. This table is the hub's mirror of that: one row per (customer, stack). The hub
// NEVER clears a hold on a box; a row goes only when the box says the hold is gone. The reminder checker
// (monitor.HeldAppsChecker) reads it and records here when it mailed, so a hub restart neither re-mails nor forgets.
// All times are stored as RFC3339 UTC text.
// HeldApp is one held app as the hub knows it.
type HeldApp struct {
CustomerID string
Stack string
Reason string // restore_failed, restore_mixed, update_failed, unhealthy_stop, putback_pending
Since time.Time // the hold's own time (the box's)
Why string // the box's one-line sentence: what put it there and what lifts it
FirstSeen time.Time // when the hub first heard of this hold
LastSeen time.Time // the newest app_held for it — the box repeats it daily
RemindedAt time.Time // the last operator reminder; zero = none yet
}
func (s *Store) migrateHeldApps() error {
_, err := s.db.Exec(`
CREATE TABLE IF NOT EXISTS held_apps (
customer_id TEXT NOT NULL,
stack TEXT NOT NULL,
reason TEXT NOT NULL DEFAULT '',
since TEXT NOT NULL DEFAULT '',
why TEXT NOT NULL DEFAULT '',
first_seen TEXT NOT NULL,
last_seen TEXT NOT NULL,
reminded_at TEXT NOT NULL DEFAULT '',
PRIMARY KEY (customer_id, stack)
);`)
return err
}
func fmtT(t time.Time) string {
if t.IsZero() {
return ""
}
return t.UTC().Format(time.RFC3339)
}
func parseT(s string) time.Time {
if s == "" {
return time.Time{}
}
return parseSQLiteTime(s)
}
// UpsertHeldApp records an app_held. A repeat of the same hold (same reason and since) only moves last_seen; a NEW
// hold on the stack (another reason or another since) starts over — first_seen now and no reminder yet — so its own
// grace runs from the start.
func (s *Store) UpsertHeldApp(customerID, stack, reason string, since time.Time, why string, now time.Time) error {
if customerID == "" || stack == "" {
return fmt.Errorf("held app: empty customer or stack")
}
res, err := s.db.Exec(`UPDATE held_apps SET last_seen = ?, why = ?
WHERE customer_id = ? AND stack = ? AND reason = ? AND since = ?`,
fmtT(now), why, customerID, stack, reason, fmtT(since))
if err != nil {
return err
}
if n, _ := res.RowsAffected(); n > 0 {
return nil
}
_, err = s.db.Exec(`INSERT INTO held_apps (customer_id, stack, reason, since, why, first_seen, last_seen, reminded_at)
VALUES (?, ?, ?, ?, ?, ?, ?, '')
ON CONFLICT(customer_id, stack) DO UPDATE SET reason = excluded.reason, since = excluded.since,
why = excluded.why, first_seen = excluded.first_seen, last_seen = excluded.last_seen, reminded_at = ''`,
customerID, stack, reason, fmtT(since), why, fmtT(now), fmtT(now))
return err
}
// ClearHeldApp removes a hold the box says is gone. Returns whether a row existed.
func (s *Store) ClearHeldApp(customerID, stack string) (bool, error) {
res, err := s.db.Exec(`DELETE FROM held_apps WHERE customer_id = ? AND stack = ?`, customerID, stack)
if err != nil {
return false, err
}
n, _ := res.RowsAffected()
return n > 0, nil
}
// MarkHeldAppReminded records the operator reminder for the hold as it is now (same reason and since).
func (s *Store) MarkHeldAppReminded(customerID, stack, reason string, since, at time.Time) error {
_, err := s.db.Exec(`UPDATE held_apps SET reminded_at = ? WHERE customer_id = ? AND stack = ? AND reason = ? AND since = ?`,
fmtT(at), customerID, stack, reason, fmtT(since))
return err
}
// ListHeldApps returns every held app, or one customer's when customerID is not empty, oldest hold first.
func (s *Store) ListHeldApps(customerID string) ([]HeldApp, error) {
q := `SELECT customer_id, stack, reason, since, why, first_seen, last_seen, reminded_at FROM held_apps`
args := []interface{}{}
if customerID != "" {
q += ` WHERE customer_id = ?`
args = append(args, customerID)
}
q += ` ORDER BY since, customer_id, stack`
rows, err := s.db.Query(q, args...)
if err != nil {
return nil, err
}
defer rows.Close()
var out []HeldApp
for rows.Next() {
var h HeldApp
var since, first, last, rem string
if err := rows.Scan(&h.CustomerID, &h.Stack, &h.Reason, &since, &h.Why, &first, &last, &rem); err != nil {
return nil, err
}
h.Since, h.FirstSeen, h.LastSeen, h.RemindedAt = parseT(since), parseT(first), parseT(last), parseT(rem)
out = append(out, h)
}
return out, rows.Err()
}
+4
View File
@@ -889,6 +889,10 @@ func (s *Store) migrate() error {
if err := s.migrateOperatorActions(); err != nil { if err := s.migrateOperatorActions(); err != nil {
return fmt.Errorf("operator_actions: %w", err) return fmt.Errorf("operator_actions: %w", err)
} }
// D53 (`08` §10): the held apps the boxes report (held_apps.go).
if err := s.migrateHeldApps(); err != nil {
return fmt.Errorf("held_apps: %w", err)
}
// Operator access tokens (hub v0.152.0, `05` §17, api_tokens.go). // Operator access tokens (hub v0.152.0, `05` §17, api_tokens.go).
if err := s.migrateAPITokens(); err != nil { if err := s.migrateAPITokens(); err != nil {
return fmt.Errorf("api_tokens: %w", err) return fmt.Errorf("api_tokens: %w", err)
+66
View File
@@ -0,0 +1,66 @@
package web
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// D53 (`08` §10): the „Held apps" list on the host page and on the Hosts page — both branches of each {{if}} rendered
// (the seam-wiring rule): a held app shows its app, reason and who may clear it; after the clear, the empty sentence.
func TestD53_HeldAppsListOnTheHostPageAndTheHostsPage(t *testing.T) {
s, st := newTestServer(t)
if err := st.UpsertHost(&store.Host{HostID: "demo-hp-01", CustomerID: "c1", APIKey: "k1"}); err != nil {
t.Fatal(err)
}
now := time.Now()
if err := st.UpsertHeldApp("c1", "kimai", "unhealthy_stop", now.Add(-3*24*time.Hour), "stopped by the box for a crash loop", now); err != nil {
t.Fatal(err)
}
render := func() (string, string) {
rr := httptest.NewRecorder()
s.handleHostsList(rr, httptest.NewRequest(http.MethodGet, "/hosts", nil))
hd := httptest.NewRecorder()
s.handleHostDetail(hd, httptest.NewRequest(http.MethodGet, "/hosts/demo-hp-01", nil), "demo-hp-01")
if rr.Code != 200 || hd.Code != 200 {
t.Fatalf("status: hosts %d, host %d", rr.Code, hd.Code)
}
return rr.Body.String(), hd.Body.String()
}
fleet, host := render()
for name, body := range map[string]string{"hosts": fleet, "host": host} {
if !strings.Contains(body, "Held apps") || !strings.Contains(body, `data-held-app=`) || !strings.Contains(body, "kimai") ||
!strings.Contains(body, "unhealthy_stop") || !strings.Contains(body, "Start (one more try)") {
t.Errorf("%s page: the held app is not listed with its reason and who may clear it", name)
}
}
if !strings.Contains(fleet, `href="/hosts/demo-hp-01#held-apps"`) {
t.Error("hosts page: the held app does not link to its host")
}
if !strings.Contains(host, "stopped by the box for a crash loop") {
t.Error("host page: the box's sentence is missing")
}
if _, err := st.ClearHeldApp("c1", "kimai"); err != nil {
t.Fatal(err)
}
fleet, host = render()
if strings.Contains(fleet, `data-held-app=`) || !strings.Contains(fleet, "No app is held on any box.") {
t.Error("hosts page: after the clear the list is not empty")
}
if strings.Contains(host, `data-held-app=`) || !strings.Contains(host, "No app is held on this box.") {
t.Error("host page: after the clear the list is not empty")
}
}
// A host with no customer shows no held apps — never the whole fleet's.
func TestD53_AHostWithNoCustomerShowsNoHeldApps(t *testing.T) {
s, st := newTestServer(t)
_ = st.UpsertHeldApp("c9", "x", "restore_failed", time.Now(), "y", time.Now())
if rows := s.hostHeldAppRows("", time.Now()); len(rows) != 0 {
t.Fatalf("a customer-less host lists %d held app(s)", len(rows))
}
}
+57
View File
@@ -0,0 +1,57 @@
package web
import (
"time"
"gitea.dooplex.hu/admin/felhom-hub/internal/monitor"
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
)
// heldAppRow is one line of the „Held apps" list (D53, `08` §10) on the host page and the Hosts page.
type heldAppRow struct {
CustomerID string
HostID string // the customer's host, for the link on the Hosts page; "" when unknown
Stack string
Reason string
Since time.Time
Why string // the box's own sentence
WhoClears string
Reminded bool
Stale bool // the box has not repeated it for heldAppStale — it may be down
}
// heldAppStaleAfter mirrors monitor's heldAppStale: a box repeats app_held daily.
const heldAppStaleAfter = 50 * time.Hour
// heldAppRows reads the held-app mirror, one customer's or every one's ("" = all). A read failure is logged and
// shown as an empty list — the page must still render.
func (s *Server) heldAppRows(customerID string, hostOf map[string]string, now time.Time) []heldAppRow {
held, err := s.store.ListHeldApps(customerID)
if err != nil {
s.logger.Printf("[WARN] held apps: list for %q: %v", customerID, err)
return nil
}
return toHeldAppRows(held, hostOf, now)
}
// hostHeldAppRows is one host's list. A host with no customer has none — never the fleet's („" lists every one).
func (s *Server) hostHeldAppRows(customerID string, now time.Time) []heldAppRow {
if customerID == "" {
return nil
}
return s.heldAppRows(customerID, nil, now)
}
func toHeldAppRows(held []store.HeldApp, hostOf map[string]string, now time.Time) []heldAppRow {
out := make([]heldAppRow, 0, len(held))
for _, h := range held {
since := h.Since
if since.IsZero() {
since = h.FirstSeen
}
out = append(out, heldAppRow{CustomerID: h.CustomerID, HostID: hostOf[h.CustomerID], Stack: h.Stack, Reason: h.Reason,
Since: since, Why: h.Why, WhoClears: monitor.HeldAppWhoClears(h.Reason), Reminded: !h.RemindedAt.IsZero(),
Stale: now.Sub(h.LastSeen) > heldAppStaleAfter})
}
return out
}
+11
View File
@@ -658,9 +658,19 @@ func (s *Server) handleHostsList(w http.ResponseWriter, r *http.Request) {
// non-fatal: still render the host list // non-fatal: still render the host list
} }
// D53 (`08` §10): every held app across the fleet, each linked to its customer's host.
hostOf := map[string]string{}
for _, row := range rows {
if row.CustomerID != "" {
hostOf[row.CustomerID] = row.HostID
}
}
heldApps := s.heldAppRows("", hostOf, time.Now())
data := map[string]interface{}{ data := map[string]interface{}{
"Hosts": rows, "Hosts": rows,
"Unclaimed": unclaimed, "Unclaimed": unclaimed,
"HeldApps": heldApps,
"CustomerPicker": picker, "CustomerPicker": picker,
"Flash": r.URL.Query().Get("flash"), "Flash": r.URL.Query().Get("flash"),
"CSRFToken": s.getCSRFToken(r), "CSRFToken": s.getCSRFToken(r),
@@ -873,6 +883,7 @@ func (s *Server) hostDetailData(host *store.Host, r *http.Request) map[string]in
"DesiredGeneration": host.DesiredGeneration, "DesiredGeneration": host.DesiredGeneration,
"Vitals": vitals, "Vitals": vitals,
"Guests": guests, "Guests": guests,
"HeldApps": s.hostHeldAppRows(host.CustomerID, now), // D53, `08` §10
"GuestRunning": guestRunning, "GuestRunning": guestRunning,
"GuestTotal": len(guests), "GuestTotal": len(guests),
"StorageTargets": storageTargets, "StorageTargets": storageTargets,
@@ -146,6 +146,39 @@
{{end}} {{end}}
</section> </section>
<!-- Held apps (D53, `08` §10): apps the box keeps stopped until a person acts. The hub never clears one. -->
<section class="card" id="held-apps" style="padding: 0; overflow: hidden;">
<h2 style="padding: 1.25rem 1.25rem 0.5rem;">Held apps</h2>
{{if .HeldApps}}
<table class="data-table">
<thead>
<tr>
<th>App</th>
<th>Reason</th>
<th>Since</th>
<th>Why</th>
<th>Who may clear it</th>
</tr>
</thead>
<tbody>
{{range .HeldApps}}
<tr data-held-app="{{.Stack}}">
<td><code>{{.Stack}}</code></td>
<td><span class="severity-badge severity-warning">{{.Reason}}</span>{{if .Stale}} <span class="text-muted">(not confirmed by the box for 2 days)</span>{{end}}</td>
<td>{{timeAgo .Since}}</td>
<td>{{.Why}}</td>
<td>{{.WhoClears}}{{if .Reminded}} <span class="text-muted">(operator reminded)</span>{{end}}</td>
</tr>
{{end}}
</tbody>
</table>
{{else}}
<div class="empty-state" style="border: none;">
<p>No app is held on this box.</p>
</div>
{{end}}
</section>
<!-- Storage Targets --> <!-- Storage Targets -->
<section class="card" style="padding: 0; overflow: hidden;"> <section class="card" style="padding: 0; overflow: hidden;">
<h2 style="padding: 1.25rem 1.25rem 0.5rem;">Storage Targets</h2> <h2 style="padding: 1.25rem 1.25rem 0.5rem;">Storage Targets</h2>
+33
View File
@@ -34,6 +34,39 @@
</div> </div>
{{end}} {{end}}
<!-- Held apps across the fleet (D53, `08` §10). The hub never clears a hold; each row links to its host. -->
<section class="card" id="held-apps" style="margin-bottom: 1.5rem; padding: 0; overflow: hidden;">
<h2 style="padding: 1.25rem 1.25rem 0.5rem; margin-top: 0;">Held apps</h2>
{{if .HeldApps}}
<table class="data-table">
<thead>
<tr>
<th>Customer</th>
<th>App</th>
<th>Reason</th>
<th>Since</th>
<th>Who may clear it</th>
</tr>
</thead>
<tbody>
{{range .HeldApps}}
<tr data-held-app="{{.CustomerID}}/{{.Stack}}">
<td>{{if .HostID}}<a href="/hosts/{{.HostID}}#held-apps">{{.CustomerID}}</a>{{else}}{{.CustomerID}}{{end}}</td>
<td><code>{{.Stack}}</code></td>
<td><span class="severity-badge severity-warning">{{.Reason}}</span>{{if .Stale}} <span class="text-muted">(not confirmed for 2 days)</span>{{end}}</td>
<td>{{timeAgo .Since}}</td>
<td>{{.WhoClears}}</td>
</tr>
{{end}}
</tbody>
</table>
{{else}}
<div class="empty-state" style="border: none;">
<p>No app is held on any box.</p>
</div>
{{end}}
</section>
{{if .Unclaimed}} {{if .Unclaimed}}
<section class="card" style="margin-bottom: 1.5rem; border-color: var(--warn);"> <section class="card" style="margin-bottom: 1.5rem; border-color: var(--warn);">
<h2 style="margin-top: 0;">Unclaimed appliances <span class="text-muted" style="font-size: 0.8em; font-weight: normal;">(booted from the generic ISO, awaiting a bind)</span></h2> <h2 style="margin-top: 0;">Unclaimed appliances <span class="text-muted" style="font-size: 0.8em; font-weight: normal;">(booted from the generic ISO, awaiting a bind)</span></h2>