diff --git a/CHANGELOG.md b/CHANGELOG.md index 70c8c7b..83c66c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,29 @@ +## v0.265.0 — the cause of "runs but not installed", a held app that says so, a louder OOM storm (2026-09-23, R-634, R-625, R-636, R-647) + +**MinAgent: 0.131.0** (unchanged). **Needs hub v0.121.0** (deployed first). New strings: yes (hu + en). + +- **R-634 — the mechanism, diagnosed and fixed.** Reproduced on 9202: deploy `outline`, press the + whole-box backup 20 s later. The backup's list read the in-memory `Deployed` flag, which is true from + the moment a deploy is ACCEPTED, so the volume leg stopped the deploying app (`compose down`), dumped its + half-made volumes and ran a SECOND `compose up -d` beside the deploy's own; both failed and the deploy + recorded „not deployed" (on 2026-09-22 the backup's `up` won and the containers ran under it). Fix: + `ListDeployedStacks` leaves out a deploying app; the volume leg asks again right before the stop (SKIP, + never a failure); `StopStack` / `StartStack` refuse a deploying stack for every caller + (`ErrStackDeploying`). `sparkyfitness` did not reproduce alone (deployed in 47 s). +- **R-625 — a held app shows the truth.** The update badge of a held app reads „Megállítva — + visszaállítás szükséges" / "Stopped — restore needed" (`tag-error`, title = the hold's first sentence + in the reader's language). No Update button (the list already hid it; now pinned). Parity fixtures + `app_info_deployed` and `stacks_full` regenerated — measured diff: exactly the new badge, one line each. +- **R-636 — a repeating OOM problem gets louder.** The scan reads the kernel's `oom_kill` counter, + `memory.max` and `memory.peak` for flagged containers; 20+ kills in 30 min of the same container run → + ONE `app_oom_storm` (error, operator-only). `app_oom` unchanged. RomM's measured rate was ≈375 kills / + 30 min; one hiccup is 1–3. +- **R-647 — the three leftovers.** A held update's error is the key `update.error.held`, rendered per + reader (page funcs now apply to Hungarian too; same bytes on a Hungarian box); the `app_update_held` + details carry the `copy_holds` KEY, not the Hungarian phrase; the undone event no longer logs + `hold recorded`, and the disabled-notifier line names the real severity of a health change. +- Red-proofs: eight in this repo, each seen failing (REPORT). + ## v0.264.0 — the undo reaches the fleet, and the household is TOLD, in its own language (2026-09-23, `09` §6.4 parts 2–3) **MinAgent: 0.131.0** (unchanged). **Needs hub v0.120.0** (deployed first). New strings: yes (hu + en). diff --git a/controller/README.md b/controller/README.md index 227e59d..7a3b43c 100644 --- a/controller/README.md +++ b/controller/README.md @@ -650,6 +650,18 @@ the old version back; a power cut while undoing resumes the undo. Reasoning and `felhom.eu/documentation/architecture/09-update-architecture.md` §6.1a, `felhom.eu/documentation/audits/undo-bakeoff-2026-09-23/`. +**Held apps say so (v0.265.0, R-625).** While a hold stands, the update badge reads „Megállítva — +visszaállítás szükséges" / "Stopped — restore needed" (`tag-error`, title = the hold sentence's first +sentence, in the reader's language) and no Update button is rendered; the API still answers 409 `held`. A +held update's error is stored as the key `update.error.held` and rendered per reader, so a reader in the +other language than the box reads the hold in theirs (R-647). + +**Deploys are not interrupted (v0.265.0, R-634).** A deploy still running is not in any backup leg's app +list, the volume leg asks again right before it stops an app, and `StopStack` / `StartStack` refuse a +deploying stack (`ErrStackDeploying`) for every caller. Measured cause: the whole-box backup's `compose down` +and second `compose up -d` in the middle of the deploy's own `up` — the deploy then recorded „not deployed" +over running containers. + **The household is told (v0.264.0).** An undone update sends `app_update_undone` (warning) ONCE; an update that ends held sends `app_update_held` (error) ONCE — also when the hold itself could not be saved. Details `{app, stack_name, from, to, at, copy_tier, copy_date, copy_holds}`. Both are in @@ -1132,6 +1144,7 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri > **Absent-storage tier skip (v0.243.0, R-518).** A tier whose storage the agent (≥ 0.131.0) reports `absent` is dropped from the manual and the scheduled run before anything is stopped (`quiesce.skipAbsentTiers`), logged, and reported once as `backup_tier_skipped`. `unknown` or a legacy agent is never skipped. > **OOM visibility (v0.243.0, R-514).** The 30 s dead-app check also reads `State.OOMKilled` for running app containers (`Manager.ScanOOMKilled`); the dashboard shows „Memória elfogyott" and the hub gets `app_oom` once per container run. The controller does not restart the app. +> **OOM storm (v0.265.0, R-636).** For a flagged container the scan also reads the kernel's own kill counter (`memory.events` `oom_kill`, plus `memory.max` / `memory.peak`) with one `docker exec … cat` — `OOMKilled` is sticky, so it cannot count. When the SAME container run's counter rises by **20 or more within 30 minutes**, the notifier sends **`app_oom_storm`** (severity `error`, operator-only, details `{app, container, kills, window_min, mem_limit, peak}`) — ONCE per container run. `app_oom` itself is unchanged. An unreadable counter (an image without `cat`) never escalates. > **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest > backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0, @@ -2432,6 +2445,7 @@ The controller pushes structured events to the Hub's `/api/v1/event` endpoint. T | `app_start_failed` | **warning** | A DEPLOYED app is not running (fix-3) — fired ONCE per running→down transition. **Customer-switchable („Alkalmazás nem fut"), OFF by default; the OPERATOR is e-mailed regardless.** Was `warn` until v0.223.0 — see the severity note below | | `app_update_undone` | warning | v0.264.0 — a guarded update failed and the box put the previous version and its data back. Once per app per failed step. Household ON by default | | `app_update_held` | **error** | v0.264.0 — a guarded update (or its undo) failed and the app is held until a restore. The mail's line is the hold sentence in the household's language. Household ON by default | +| `app_oom_storm` | **error** | v0.265.0 (R-636) — the same container run OOM-killed 20+ times in 30 min (the kernel's `oom_kill` counter). Once per container run. Operator-only | | `disaster_recovery_started` | warning | DR restore begins | | `disaster_recovery_completed` | info/error | DR restore finishes (success/partial) | diff --git a/controller/cmd/controller/main.go b/controller/cmd/controller/main.go index 176dad2..f44c4af 100644 --- a/controller/cmd/controller/main.go +++ b/controller/cmd/controller/main.go @@ -620,7 +620,7 @@ func main() { d.CopyDate = ev.CopyDate.UTC().Format(time.RFC3339) } if backupMgr != nil && ev.CopyTier > 0 { - d.CopyHolds = backupMgr.UpdateCopyHolds(ev.App, ev.CopyTier) + d.CopyHolds = backupMgr.UpdateCopyHoldsKey(ev.App, ev.CopyTier) // R-647: the key, never the Hungarian phrase } notifier.NotifyAppUpdateHeld(d, func(lang string) string { if ev.HoldRecorded && backupMgr != nil { @@ -851,8 +851,8 @@ func main() { logger.Printf("[WARN] [deadapp] OOM scan failed: %v", oerr) } else { for _, o := range ooms { - logger.Printf("[WARN] [deadapp] %s: container %s was OOM-killed (started %s)", o.Stack, o.Container, o.StartedAt) - notifier.NotifyAppOOM(o.Stack, o.Container, o.StartedAt) + logger.Printf("[WARN] [deadapp] %s: container %s was OOM-killed (started %s; kills %d, limit %s, peak %s)", o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak) + notifier.NotifyAppOOM(o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak) } } deadAppScans++ @@ -2556,7 +2556,10 @@ func (a *stackAdapter) GetStackClassifiedBinds(name string) ([]backup.Classified func (a *stackAdapter) ListDeployedStacks() []backup.StackSummary { var result []backup.StackSummary for _, s := range a.mgr.GetStacks() { - if !s.Deployed { + // R-634 (v0.265.0): `Deployed` is set true in memory the moment a deploy is ACCEPTED (the + // no-stale-button UX), so without the second test a deploy still pulling was in every backup + // leg's list — and the volume leg stopped and restarted it in the middle of its own `up`. + if !s.Deployed || s.Deploying { continue } result = append(result, backup.StackSummary{ @@ -2574,6 +2577,13 @@ func (a *stackAdapter) StopStack(name string) error { return a.mgr.StopStack(name) } +// IsDeploying answers backup's optional deployingReporter (R-634): the volume leg asks it again +// immediately before it stops an app, because its list is taken once at the start of the run. +func (a *stackAdapter) IsDeploying(name string) bool { + s, ok := a.mgr.GetStack(name) + return ok && s.Deploying +} + func (a *stackAdapter) StartStack(name string) error { return a.mgr.StartStack(name) } @@ -3471,6 +3481,14 @@ func (a *updateGuardsAdapter) HoldFor(name string) (bool, string) { return a.b.RestoreHoldFor(name) } +// HoldForLang renders the hold sentence for one reader (R-647, v0.265.0). +func (a *updateGuardsAdapter) HoldForLang(name, lang string) (bool, string) { + if a.b == nil { + return false, "" + } + return a.b.RestoreHoldForLang(name, lang) +} + func (a *updateGuardsAdapter) Busy(name string) (bool, string) { if a.q.SuppressedStacks()[name] { return true, "a whole-guest backup (quiesce) is holding it" diff --git a/controller/cmd/controller/update_events_wiring_test.go b/controller/cmd/controller/update_events_wiring_test.go index 51a3cd5..784bdc5 100644 --- a/controller/cmd/controller/update_events_wiring_test.go +++ b/controller/cmd/controller/update_events_wiring_test.go @@ -1,6 +1,10 @@ package main -import "testing" +import ( + "os" + "strings" + "testing" +) // v0.264.0. The update-event sink and the R-646 backfill are both CALLED from main.go (an AST walk — // a comment naming them would not count). A sink built and never wired would fail silently: no event, @@ -21,3 +25,16 @@ func TestUpdateEventSinkIsWiredAtStartup(t *testing.T) { t.Error("BackfillAppliedMeta (R-646) must be called, after AdoptPins") } } + +// R-647 (2) — the held event carries the copy_holds KEY, never the Hungarian phrase: the hub prints the +// raw details as the household mail's `Note:` line. COMPANION RED-PROOF (REPORT.md): wire +// UpdateCopyHolds back — this fails. +func TestR647_HeldEventCarriesTheCopyHoldsKey(t *testing.T) { + src, err := os.ReadFile("main.go") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(src), "d.CopyHolds = backupMgr.UpdateCopyHoldsKey(") { + t.Fatal("main.go must fill AppUpdateDetails.CopyHolds with UpdateCopyHoldsKey (the key), not the Hungarian phrase") + } +} diff --git a/controller/internal/api/router.go b/controller/internal/api/router.go index 4414c77..e5a6593 100644 --- a/controller/internal/api/router.go +++ b/controller/internal/api/router.go @@ -19,7 +19,6 @@ import ( "gitea.dooplex.hu/admin/felhom-controller/internal/backup" cf "gitea.dooplex.hu/admin/felhom-controller/internal/cloudflare" "gitea.dooplex.hu/admin/felhom-controller/internal/config" - "gitea.dooplex.hu/admin/felhom-controller/internal/i18n" "gitea.dooplex.hu/admin/felhom-controller/internal/integrations" "gitea.dooplex.hu/admin/felhom-controller/internal/metrics" "gitea.dooplex.hu/admin/felhom-controller/internal/notify" @@ -383,18 +382,12 @@ func (r *Router) getStack(w http.ResponseWriter, req *http.Request, name string) // v0.264.0 (R-606): the page polls this for the Update button's label and the outcome. Rendered in // the reader's language; a Hungarian reader gets exactly the bytes it always got. GetStack returns a // COPY, so nothing here touches the manager's own state. - if lang := r.langFor(req); lang != i18n.Default { - stack.UpdatePhaseLabel = stacks.UpdatePhaseLabelIn(lang, stack.UpdatePhase) - stack.UpdateError = stack.UpdateErrorIn(lang) - if stack.HoldReason != "" && r.backupMgr != nil { - if held, why := r.backupMgr.RestoreHoldForLang(name, lang); held && why != "" { - stack.HoldReason = why - if stack.UpdateErrorKey == "" && stack.UpdatePhase == stacks.UpdatePhaseFailed { - stack.UpdateError = why // a held update's UpdateError IS the hold sentence - } - } - } - } + // v0.265.0 (R-647): EVERY reader, Hungarian included — a Hungarian reader of an English box read + // the hold in English before. On a Hungarian box a Hungarian reader still gets the same bytes. + lang := r.langFor(req) + stack.UpdatePhaseLabel = stacks.UpdatePhaseLabelIn(lang, stack.UpdatePhase) + stack.UpdateError = r.stackMgr.UpdateErrorFor(*stack, lang) + stack.HoldReason = r.stackMgr.HoldReasonFor(*stack, lang) writeJSON(w, http.StatusOK, apiResponse{OK: true, Data: stack}) } diff --git a/controller/internal/backup/backup.go b/controller/internal/backup/backup.go index 4b9fd7d..e51e4f3 100644 --- a/controller/internal/backup/backup.go +++ b/controller/internal/backup/backup.go @@ -724,6 +724,16 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) { continue } + // R-634 (v0.265.0): an app whose deploy is still running is NOT an installation to back up, + // and stopping it is what broke the deploy. The list is taken once at the start of the run, so + // the question is asked again here, immediately before the stop. A SKIP, not a failure: there + // is nothing of the household's in it yet, and a FAIL would page the operator for a deploy. + if m.stackIsDeploying(stack.Name) { + m.logger.Printf("[INFO] [backup] Skipping volume dump for %s — the app is still being deployed (R-634)", stack.Name) + summary = append(summary, fmt.Sprintf("SKIP %s volumes (deploying)", stack.Name)) + continue + } + // R-181: the reserve, ahead of DumpAppVolumesSafe so a refused app is NOT stopped. For an app // that already has a DB this is a memo lookup taken before its DB dump; for a volume-only app // this is where its verdict is taken, still before its first byte. @@ -1398,3 +1408,17 @@ func (m *Manager) note(key string, args ...interface{}) string { // noteErr renders an error into a saved note in the box's language: its bundle message when it // carries one (release B), its own text otherwise. func (m *Manager) noteErr(err error) string { return util.ErrText(m.boxLang(), err) } + +// deployingReporter is the OPTIONAL half of the stack provider that can say a deploy is in flight +// (R-634). Optional on purpose: the production adapter implements it, and the many test fakes that +// predate it need not — a provider that cannot answer is read as "not deploying", today's behaviour. +type deployingReporter interface { + IsDeploying(name string) bool +} + +func (m *Manager) stackIsDeploying(name string) bool { + if d, ok := m.stackProvider.(deployingReporter); ok { + return d.IsDeploying(name) + } + return false +} diff --git a/controller/internal/backup/r634_deploying_test.go b/controller/internal/backup/r634_deploying_test.go new file mode 100644 index 0000000..e359b57 --- /dev/null +++ b/controller/internal/backup/r634_deploying_test.go @@ -0,0 +1,61 @@ +package backup + +import ( + "io" + "log" + "path/filepath" + "testing" + + "gitea.dooplex.hu/admin/felhom-controller/internal/config" +) + +// R-634 (v0.265.0). Measured 2026-09-23 on 9202: a whole-box backup pressed 20 s into outline's +// deploy stopped it (`compose down`), dumped its half-made volumes and ran a SECOND `compose up -d` +// beside the deploy's own — both failed, and the deploy recorded „not deployed". + +type deployingFake struct { + volDumpFakeProvider + deploying map[string]bool +} + +func (f *deployingFake) IsDeploying(name string) bool { return f.deploying[name] } + +// Through the dumpVolumesSafe SEAM, never the real dump: the real one runs `docker run … tar` against a +// named volume, which on a developer machine CREATES that volume (it did, once, on DooPlex while this +// test was written). The seam is where the stop happens, so a call to it IS the leak. +// +// COMPANION RED-PROOF (REPORT.md): delete the stackIsDeploying block in runVolumeDumps — the dump +// (and with it the stop) is called for outline and this fails with dumped=[outline]. +func TestR634_VolumeLegNeverStopsADeployingApp(t *testing.T) { + cfg := &config.Config{} + cfg.Paths.SystemDataPath = filepath.Join(t.TempDir(), "sys") + fake := &deployingFake{ + volDumpFakeProvider: volDumpFakeProvider{ + stacks: []StackSummary{{Name: "outline"}}, + volumes: map[string][]string{"outline": {"outline_outline_data"}}, + }, + deploying: map[string]bool{"outline": true}, + } + m := &Manager{cfg: cfg, logger: log.New(io.Discard, "", 0), systemDataPath: cfg.Paths.SystemDataPath, + stackProvider: fake} + var calls []string + m.dumpVolumesSafe = func(name string) error { calls = append(calls, name); return nil } + + summary, dumped, ok := m.runVolumeDumps() + if len(calls) != 0 { + t.Fatalf("a DEPLOYING app was stopped and dumped by the backup: dumped=%v (R-634's race)", calls) + } + if !ok || dumped != 0 { + t.Errorf("a deploy in flight is a SKIP, never a failure that pages the operator: ok=%v dumped=%d %v", ok, dumped, summary) + } + if !containsSummary(summary, "SKIP outline volumes (deploying)") { + t.Errorf("the skip must be visible in the run summary, got %v", summary) + } + + // Control: the same app once its deploy has finished IS backed up (the skip is conditional). + fake.deploying["outline"] = false + m.runVolumeDumps() + if len(calls) != 1 || calls[0] != "outline" { + t.Errorf("control: a finished app must be dumped as before, dumped=%v", calls) + } +} diff --git a/controller/internal/backup/undo_hold_test.go b/controller/internal/backup/undo_hold_test.go index cc20830..48f027a 100644 --- a/controller/internal/backup/undo_hold_test.go +++ b/controller/internal/backup/undo_hold_test.go @@ -7,6 +7,10 @@ import ( "strings" "testing" "time" + + "gitea.dooplex.hu/admin/felhom-controller/internal/config" + "gitea.dooplex.hu/admin/felhom-controller/internal/i18n" + "gitea.dooplex.hu/admin/felhom-controller/internal/util" ) // v0.263.0 — a hold after a FAILED UNDO opens with what was tried and what state the data is in, in the @@ -65,3 +69,15 @@ func TestUndo_HoldSentenceSaysTheUndoWasTriedAndTheDataState(t *testing.T) { t.Errorf("no Hungarian may remain on an English box, got %q", why) } } + +// R-647 (2) — the key form of the copy verdict is a bundle KEY, the same verdict as the phrase. +func TestR647_UpdateCopyHoldsKeyIsTheKeyOfThePhrase(t *testing.T) { + m := NewManager(&config.Config{}, nil, log.New(io.Discard, "", 0)) + k := m.UpdateCopyHoldsKey("x", UpdateTierLocal) + if !strings.HasPrefix(k, "hold.copy_holds.") { + t.Fatalf("want a hold.copy_holds.* key, got %q", k) + } + if util.Text(i18n.Default, k) != m.UpdateCopyHolds("x", UpdateTierLocal) { + t.Fatal("the key must render to exactly the phrase UpdateCopyHolds gives") + } +} diff --git a/controller/internal/backup/update_guard.go b/controller/internal/backup/update_guard.go index 82d3b85..5e965ac 100644 --- a/controller/internal/backup/update_guard.go +++ b/controller/internal/backup/update_guard.go @@ -152,6 +152,13 @@ func (m *Manager) UpdateCopyHolds(stackName string, tier int) string { return util.Text(i18n.Default, updateCopyHoldsKey(m.DataOutsideUnit(stackName), tier)) } +// UpdateCopyHoldsKey is the same verdict as UpdateCopyHolds, as its bundle KEY (R-647, v0.265.0). The +// app_update_held event carries the key, not the Hungarian phrase: the hub prints the raw details as +// the mail's `Note:` line, and an English household read „a beállításokat, …" there. +func (m *Manager) UpdateCopyHoldsKey(stackName string, tier int) string { + return updateCopyHoldsKey(m.DataOutsideUnit(stackName), tier) +} + func updateCopyHoldsKey(outside bool, tier int) string { switch tier { case UpdateTierLocal: diff --git a/controller/internal/i18n/locales/en.json b/controller/internal/i18n/locales/en.json index 3dc2b40..461f8a4 100644 --- a/controller/internal/i18n/locales/en.json +++ b/controller/internal/i18n/locales/en.json @@ -571,6 +571,7 @@ "badge.update.behind.today": " — today", "badge.update.current": "Up to date", "badge.update.current.title": "This app is running the newest version available.", + "badge.update.held": "Stopped — restore needed", "catchall.alkalmazas_kezelese": "Manage app", "catchall.alkalmazasok": "Apps", "catchall.vezerlopult": "Dashboard", diff --git a/controller/internal/i18n/locales/hu.json b/controller/internal/i18n/locales/hu.json index f50549f..8e3cd62 100644 --- a/controller/internal/i18n/locales/hu.json +++ b/controller/internal/i18n/locales/hu.json @@ -566,6 +566,7 @@ "badge.update.behind.today": " — ma", "badge.update.current": "Naprakész", "badge.update.current.title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.", + "badge.update.held": "Megállítva — visszaállítás szükséges", "catchall.alkalmazas_kezelese": "Alkalmazás kezelése", "catchall.alkalmazasok": "Alkalmazások", "catchall.vezerlopult": "Vezérlőpult", diff --git a/controller/internal/notify/notifier.go b/controller/internal/notify/notifier.go index a1ee98c..68034a2 100644 --- a/controller/internal/notify/notifier.go +++ b/controller/internal/notify/notifier.go @@ -46,6 +46,11 @@ type Notifier struct { // oomSeen (R-514) remembers container runs already reported as OOM-killed. oomSeen map[string]bool + // oomTrack (R-636) remembers, per container run, the kill counter over the last 30 minutes and + // whether the ONE storm alarm of that run has gone. + oomTrack map[string]*oomRun + // nowFn is a test clock (nil → time.Now). + nowFn func() time.Time // appDown tracks which deployed apps are currently in the DOWN state so app_start_failed fires // ONCE per running→down transition, not every health cycle (fix-3 anti-spam). In-memory: a // controller restart re-notifies once (acceptable — better than missing). The hub owns the real @@ -359,7 +364,7 @@ func (n *Notifier) pushEventBoth(eventType, severity, message, messageCustomer s // Detects both degradation (ok→warn, ok→fail, warn→fail) and recovery (fail→ok, warn→ok, fail→warn). func (n *Notifier) NotifyHealthChange(status string, issues, warnings []string) { if !n.enabled { - n.dropped("health_change", status) + n.dropped("health_change", healthSeverity(status)) // R-647: the severity, not the health status return } @@ -714,21 +719,108 @@ func (n *Notifier) NotifyAppStartFailures(apps []AppRunState) { // limit (Docker State.OOMKilled). Fires ONCE per container run (keyed by StartedAt), so a container // that stays OOM-marked does not repeat. "warning" — the hub vocabulary (R-329). Operator-only // hub-side (hub >= v0.114.0 registers it): the household sees the dashboard tag. -func (n *Notifier) NotifyAppOOM(stack, container, startedAt string) { +func (n *Notifier) NotifyAppOOM(stack, container, startedAt string, kills int64, memLimit, peak string) { key := container + "|" + startedAt n.mu.Lock() if n.oomSeen == nil { n.oomSeen = map[string]bool{} } - if n.oomSeen[key] { - n.mu.Unlock() - return - } + first := !n.oomSeen[key] n.oomSeen[key] = true + storm, inWindow := n.oomStormLocked(key, startedAt, kills) n.mu.Unlock() - n.emit("app_oom", "warning", - fmt.Sprintf("Alkalmazás memóriája elfogyott: %s (%s) — egy folyamatát a memóriakorlát leállította", stack, container), - AppDetails{StackName: stack, DisplayName: container}) + if first { + n.emit("app_oom", "warning", + fmt.Sprintf("Alkalmazás memóriája elfogyott: %s (%s) — egy folyamatát a memóriakorlát leállította", stack, container), + AppDetails{StackName: stack, DisplayName: container}) + } + if storm { + n.logger.Printf("[ERROR] [notify] %s: container %s OOM STORM — %d kills in %d min (limit %s, peak %s)", + stack, container, inWindow, oomStormWindowMin, memLimit, peak) + n.emit("app_oom_storm", "error", + fmt.Sprintf("Alkalmazás memóriája ismételten elfogy: %s (%s) — %d leállítás %d percen belül (korlát %s, csúcs %s)", + stack, container, inWindow, oomStormWindowMin, memLimit, peak), + OOMStormDetails{App: stack, StackName: stack, Container: container, Kills: inWindow, + WindowMin: oomStormWindowMin, MemLimit: memLimit, Peak: peak}) + } +} + +// R-636 (v0.265.0) — ONE louder alarm per container run when OOM kills keep coming. +// +// The once-per-run app_oom stays exactly as it was (it is what stops a crash loop from mailing 4,530 +// times, R-629). Beside it: when the kernel's own kill counter for the SAME run rises by +// oomStormKills or more within oomStormWindowMin minutes, app_oom_storm goes once, at `error`, +// operator-only. RomM's real rate on 2026-09-22 was 4,530 kills in 6 h ≈ 375 per 30 min; one hiccup +// is 1–3. Twenty in thirty minutes sits far from both. Pinned by TestR636_*. +const ( + oomStormKills = 20 + oomStormWindowMin = 30 +) + +type oomSample struct { + at time.Time + kills int64 +} + +type oomRun struct { + samples []oomSample + stormSent bool +} + +// OOMStormDetails is the app_oom_storm payload. stack_name lets the hub's per-app cooldown tell two +// storming apps apart. +type OOMStormDetails struct { + App string `json:"app"` + StackName string `json:"stack_name"` + Container string `json:"container"` + Kills int64 `json:"kills"` + WindowMin int `json:"window_min"` + MemLimit string `json:"mem_limit,omitempty"` + Peak string `json:"peak,omitempty"` +} + +func (n *Notifier) now() time.Time { + if n.nowFn != nil { + return n.nowFn() + } + return time.Now() +} + +// oomStormLocked records one reading and says whether the storm alarm fires now. Caller holds n.mu. +// A run seen for the first time within the window of its own start gets a zero reading AT its start: +// the counter began at zero there, so a run that is already storming when the controller first looks +// is not made to wait another thirty minutes. +func (n *Notifier) oomStormLocked(key, startedAt string, kills int64) (bool, int64) { + if kills < 0 { + return false, 0 // unreadable: never escalate on a guess + } + if n.oomTrack == nil { + n.oomTrack = map[string]*oomRun{} + } + now := n.now() + window := time.Duration(oomStormWindowMin) * time.Minute + r := n.oomTrack[key] + if r == nil { + r = &oomRun{} + if t, err := time.Parse(time.RFC3339Nano, startedAt); err == nil && now.Sub(t) <= window { + r.samples = append(r.samples, oomSample{at: t, kills: 0}) + } + n.oomTrack[key] = r + } + r.samples = append(r.samples, oomSample{at: now, kills: kills}) + keep := r.samples[:0] + for _, s := range r.samples { + if now.Sub(s.at) <= window { + keep = append(keep, s) + } + } + r.samples = keep + inWindow := kills - r.samples[0].kills + if r.stormSent || inWindow < oomStormKills { + return false, inWindow + } + r.stormSent = true + return true, inWindow } // DiskHealthDetails is the event-detail payload for disk_health_degraded. @@ -1211,3 +1303,15 @@ func (n *Notifier) NotifyAppUpdateHeld(d AppUpdateDetails, sentence func(lang st } n.pushEventBoth("app_update_held", "error", hu, household, d) } + +// healthSeverity is the event severity a health status would be sent at (R-647): the disabled path +// names what it drops, and "warn" is a health STATUS, not a severity the hub knows. +func healthSeverity(status string) string { + switch status { + case "fail": + return "error" + case "warn": + return "warning" + } + return "info" +} diff --git a/controller/internal/notify/r636_oom_storm_test.go b/controller/internal/notify/r636_oom_storm_test.go new file mode 100644 index 0000000..bd83277 --- /dev/null +++ b/controller/internal/notify/r636_oom_storm_test.go @@ -0,0 +1,95 @@ +package notify + +import ( + "io" + "log" + "testing" + "time" +) + +// R-636 — a repeating OOM problem gets ONE louder alarm; one hiccup stays one quiet warning. +func stormRecorder(t *testing.T) (*Notifier, *time.Time, map[string]int) { + t.Helper() + n := New("", "", "c1", nil, log.New(io.Discard, "", 0), false) + got := map[string]int{} + n.pushFn = func(eventType, severity, msg, msgCustomer string, details interface{}) { + got[eventType+"/"+severity]++ + } + clock := time.Date(2026, 9, 22, 9, 0, 0, 0, time.UTC) + n.nowFn = func() time.Time { return clock } + return n, &clock, got +} + +// COMPANION RED-PROOF (REPORT.md): make oomStormLocked always return false — the 20-kill case sends no +// storm and this fails. +func TestR636_TwentyKillsInThirtyMinutesIsOneStorm(t *testing.T) { + for _, c := range []struct { + kills int64 + storm int + }{{19, 0}, {20, 1}, {200, 1}} { + n, clock, got := stormRecorder(t) + start := clock.Add(-time.Hour).Format(time.RFC3339Nano) // started long before: no zero baseline + for i := int64(0); i <= 40; i++ { // 20 minutes of 30 s scans, counter 5 → 5+kills + n.NotifyAppOOM("romm", "romm", start, 5+c.kills*i/40, "1280M", "1279M") + *clock = clock.Add(30 * time.Second) + } + if got["app_oom_storm/error"] != c.storm { + t.Errorf("kills=%d in 20 min: storms=%d, want %d", c.kills, got["app_oom_storm/error"], c.storm) + } + if c.kills == 200 { // RomM's shape: it keeps going for six hours — still ONE storm for the run + for i := int64(1); i <= 720; i++ { + n.NotifyAppOOM("romm", "romm", start, 205+i*12, "1280M", "1279M") + *clock = clock.Add(30 * time.Second) + } + if got["app_oom_storm/error"] != 1 { + t.Errorf("six hours of storm must stay ONE storm per container run, got %d", got["app_oom_storm/error"]) + } + } + if got["app_oom/warning"] != 1 { + t.Errorf("kills=%d: the quiet app_oom must still go exactly once per run, got %d", c.kills, got["app_oom/warning"]) + } + } +} + +// One kill, and then the flag just stays set for hours (the sticky OOMKilled): never a storm. +func TestR636_OneHiccupNeverStorms(t *testing.T) { + n, clock, got := stormRecorder(t) + start := clock.Format(time.RFC3339Nano) + for i := 0; i < 6*120; i++ { // six hours of 30 s scans, counter stuck at 1 + n.NotifyAppOOM("paperless-ngx", "paperless-webserver", start, 1, "768M", "700M") + *clock = clock.Add(30 * time.Second) + } + if got["app_oom_storm/error"] != 0 || got["app_oom/warning"] != 1 { + t.Fatalf("one hiccup: want 1 warning and 0 storms, got %v", got) + } +} + +// Slow and steady — 20 kills spread over three hours is not a storm. +func TestR636_SlowKillsAreNotAStorm(t *testing.T) { + n, clock, got := stormRecorder(t) + start := clock.Add(-time.Hour).Format(time.RFC3339Nano) + for i := int64(0); i <= 360; i++ { // 3 h, counter +20 in total + n.NotifyAppOOM("x", "x", start, i*20/360, "", "") + *clock = clock.Add(30 * time.Second) + } + if got["app_oom_storm/error"] != 0 { + t.Fatalf("20 kills over 3 h must not storm, got %v", got) + } +} + +// A run first seen already storming (the controller restarted mid-storm) fires at once: its counter +// started at zero at its own start, inside the window. An unreadable counter never escalates. +func TestR636_AlreadyStormingAtFirstSightAndUnreadable(t *testing.T) { + n, clock, got := stormRecorder(t) + n.NotifyAppOOM("romm", "romm", clock.Add(-10*time.Minute).Format(time.RFC3339Nano), 150, "1280M", "1279M") + if got["app_oom_storm/error"] != 1 { + t.Fatalf("150 kills 10 min after the start must storm at first sight, got %v", got) + } + n2, _, got2 := stormRecorder(t) + for i := 0; i < 100; i++ { + n2.NotifyAppOOM("romm", "romm", "", -1, "", "") + } + if got2["app_oom_storm/error"] != 0 { + t.Fatalf("an unreadable counter must never escalate, got %v", got2) + } +} diff --git a/controller/internal/notify/update_events_test.go b/controller/internal/notify/update_events_test.go index 4f6a095..06c9d19 100644 --- a/controller/internal/notify/update_events_test.go +++ b/controller/internal/notify/update_events_test.go @@ -102,3 +102,15 @@ func TestR620_DisabledNotifierSaysWhatItDrops(t *testing.T) { } } } + +// R-647 (3) — the disabled path names the SEVERITY a health change would be sent at, never the health +// status. COMPANION RED-PROOF (REPORT.md): pass `status` again — the line reads "severity warn". +func TestR647_DisabledHealthChangeNamesTheSeverity(t *testing.T) { + var buf bytes.Buffer + n := New("", "", "c1", nil, log.New(&buf, "", 0), false) + buf.Reset() + n.NotifyHealthChange("warn", nil, nil) + if !strings.Contains(buf.String(), "DROPPED event health_change (severity warning)") { + t.Fatalf("want the severity 'warning' named, got:\n%s", buf.String()) + } +} diff --git a/controller/internal/stacks/deploy_errors.go b/controller/internal/stacks/deploy_errors.go index 1ead34f..fe7a8a1 100644 --- a/controller/internal/stacks/deploy_errors.go +++ b/controller/internal/stacks/deploy_errors.go @@ -22,6 +22,11 @@ var ( ErrPathMissing = errors.New("path field does not exist") // ErrNotEnoughMemory — the memory verdict refused the deploy (API: 400). ErrNotEnoughMemory = errors.New("not enough memory") + // ErrStackDeploying — R-634 (v0.265.0): a stop or start was asked of a stack whose deploy is still + // running. Refused, because a `compose down` or a second `compose up -d` in the middle of the + // deploy's own `up` makes BOTH fail, and the deploy then records „not deployed" over whatever + // containers the race left behind. Pinned by TestR634_StopAndStartRefuseADeployingStack. + ErrStackDeploying = errors.New("stack is being deployed") ) // msgHU renders a bundle message in Hungarian. diff --git a/controller/internal/stacks/manager.go b/controller/internal/stacks/manager.go index 2fd53f7..c94c905 100644 --- a/controller/internal/stacks/manager.go +++ b/controller/internal/stacks/manager.go @@ -1147,6 +1147,12 @@ func (m *Manager) StartStack(name string) error { if !ok { return fmt.Errorf("stack %q not found", name) } + if stack.Deploying { + // R-634: a second `compose up -d` beside the deploy's own is the race that left apps running + // under „not deployed". The deploy brings the app up itself; nothing else may. + m.logger.Printf("[WARN] [stacks] StartStack %s refused: the app's deploy is still running (R-634)", name) + return fmt.Errorf("starting stack %s: %w", name, ErrStackDeploying) + } if m.isDebug() { m.logger.Printf("[DEBUG] [stacks] StartStack %s: current state=%s deployed=%v", name, stack.State, stack.Deployed) @@ -1229,6 +1235,13 @@ func (m *Manager) StopStack(name string) error { if !ok { return fmt.Errorf("stack %q not found", name) } + if stack.Deploying { + // R-634, the backstop for EVERY caller (backup, quiesce, restore, export, storage, the Stop + // button): `compose down` in the middle of a deploy was measured to make the deploy fail and + // record „not deployed" while the caller's own restart brought the containers back. + m.logger.Printf("[WARN] [stacks] StopStack %s refused: the app's deploy is still running (R-634)", name) + return fmt.Errorf("stopping stack %s: %w", name, ErrStackDeploying) + } if m.isDebug() { m.logger.Printf("[DEBUG] [stacks] StopStack %s: current state=%s deployed=%v containers=%d", name, stack.State, stack.Deployed, len(stack.Containers)) diff --git a/controller/internal/stacks/oom.go b/controller/internal/stacks/oom.go index 59910be..d56a6c5 100644 --- a/controller/internal/stacks/oom.go +++ b/controller/internal/stacks/oom.go @@ -1,7 +1,9 @@ package stacks import ( + "fmt" "sort" + "strconv" "strings" ) @@ -18,6 +20,13 @@ type OOMContainer struct { Stack string Container string StartedAt string // identifies the container run — one event per run + // Kills is the kernel's own count of OOM kills in this container's cgroup (`memory.events` + // oom_kill) — R-636, v0.265.0. OOMKilled is a STICKY flag: it stays true for the container's whole + // life after ONE kill, so "the key re-fired" means nothing; this counter is what separates one + // hiccup from RomM's 4,530 kills in six hours. -1 when it could not be read. + Kills int64 + // MemLimit / Peak are the cgroup's memory.max and memory.peak, as "M" (or "max"); "" unread. + MemLimit, Peak string } // ScanOOMKilled inspects the containers of every deployed, running-ish stack in ONE docker call and @@ -56,13 +65,53 @@ func (m *Manager) ScanOOMKilled() ([]OOMContainer, error) { } cname := strings.TrimPrefix(f[0], "/") if st, ok := owner[cname]; ok { - found = append(found, OOMContainer{Stack: st, Container: cname, StartedAt: f[2]}) + found = append(found, OOMContainer{Stack: st, Container: cname, StartedAt: f[2], Kills: -1}) } } + // R-636: only for the containers already flagged — one `docker exec` each, and the flagged set is + // empty on a healthy box. Read from INSIDE the container (its cgroup namespace makes + // /sys/fs/cgroup its own cgroup); the controller's own namespace cannot see the others. + for i := range found { + out, _ := m.execCommand("docker", "exec", found[i].Container, "cat", + "/sys/fs/cgroup/memory.events", "/sys/fs/cgroup/memory.max", "/sys/fs/cgroup/memory.peak") + found[i].Kills, found[i].MemLimit, found[i].Peak = parseCgroupMemory(out) + } m.setOOMCache(found) return found, nil } +// parseCgroupMemory reads `cat memory.events memory.max memory.peak`: the key/value lines give +// oom_kill; the first bare line is memory.max, the second memory.peak (absent on older kernels — cat +// then fails on that file and still prints the others). Unreadable → Kills -1. +func parseCgroupMemory(out string) (kills int64, limit, peak string) { + kills = -1 + var bare []string + for _, line := range strings.Split(strings.TrimSpace(out), "\n") { + f := strings.Fields(line) + switch { + case len(f) == 2 && f[0] == "oom_kill": + if n, err := strconv.ParseInt(f[1], 10, 64); err == nil { + kills = n + } + case len(f) == 1: + bare = append(bare, f[0]) + } + } + mb := func(v string) string { + if n, err := strconv.ParseInt(v, 10, 64); err == nil { + return fmt.Sprintf("%dM", n/(1024*1024)) + } + return v // "max" + } + if len(bare) > 0 { + limit = mb(bare[0]) + } + if len(bare) > 1 { + peak = mb(bare[1]) + } + return kills, limit, peak +} + func (m *Manager) setOOMCache(found []OOMContainer) { cache := map[string][]string{} for _, o := range found { diff --git a/controller/internal/stacks/oom_test.go b/controller/internal/stacks/oom_test.go index d6e2770..adab645 100644 --- a/controller/internal/stacks/oom_test.go +++ b/controller/internal/stacks/oom_test.go @@ -53,3 +53,46 @@ func TestScanOOMKilled_SeesKilledWorkerInRunningContainer(t *testing.T) { t.Fatalf("cache not cleared after a clean scan: %v", got) } } + +// R-636 — the kernel's kill counter, the limit and the peak are read from `cat memory.events +// memory.max memory.peak` inside the container; an unreadable one is -1, never 0. +func TestR636_ParseCgroupMemory(t *testing.T) { + out := "low 0\nhigh 0\nmax 4521\noom 4530\noom_kill 4530\noom_group_kill 0\n1342177280\n1341128704\n" + k, lim, peak := parseCgroupMemory(out) + if k != 4530 || lim != "1280M" || peak != "1279M" { + t.Fatalf("got kills=%d limit=%q peak=%q", k, lim, peak) + } + if k, lim, _ := parseCgroupMemory("oom_kill 3\nmax\n"); k != 3 || lim != "max" { + t.Fatalf("no memory.peak (older kernel), unlimited: got %d %q", k, lim) + } + if k, _, _ := parseCgroupMemory("OCI runtime exec failed: exec: \"cat\": executable file not found"); k != -1 { + t.Fatalf("an image without cat must read -1 (unknown), got %d", k) + } +} + +// R-636 — the scan asks the kill counter of the FLAGGED containers only (one exec each; none on a +// healthy box). COMPANION RED-PROOF (REPORT.md): drop the exec loop — Kills stays -1 and this fails. +func TestR636_ScanReadsTheCounterOfFlaggedContainersOnly(t *testing.T) { + var execd []string + m := &Manager{logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{ + "romm": {Name: "romm", Deployed: true, Containers: []ContainerInfo{{Name: "romm", State: StateRunning}}}, + "bookstack": {Name: "bookstack", Deployed: true, Containers: []ContainerInfo{{Name: "bookstack", State: StateRunning}}}, + }} + m.execFn = func(name string, args ...string) (string, error) { + if args[0] == "exec" { + execd = append(execd, args[1]) + return "oom_kill 377\n1342177280\n1341128704\n", nil + } + return "/bookstack|false|t0\n/romm|true|2026-09-22T09:08:00Z\n", nil + } + ooms, err := m.ScanOOMKilled() + if err != nil || len(ooms) != 1 { + t.Fatalf("ooms=%+v err=%v", ooms, err) + } + if ooms[0].Kills != 377 || ooms[0].MemLimit != "1280M" || ooms[0].Peak != "1279M" { + t.Errorf("the counter must be read for the flagged container: %+v", ooms[0]) + } + if len(execd) != 1 || execd[0] != "romm" { + t.Errorf("exec must reach the flagged container only, got %v", execd) + } +} diff --git a/controller/internal/stacks/r634_deploying_test.go b/controller/internal/stacks/r634_deploying_test.go new file mode 100644 index 0000000..85b5c38 --- /dev/null +++ b/controller/internal/stacks/r634_deploying_test.go @@ -0,0 +1,36 @@ +package stacks + +import ( + "errors" + "testing" +) + +// R-634 (v0.265.0) — the backstop for every caller: a stop or a start while the deploy's own +// `compose up` is running is refused BEFORE any compose command. +// +// COMPANION RED-PROOF (REPORT.md): delete the Deploying check in StopStack — the stop falls through to +// `compose down` and the error is no longer ErrStackDeploying. +func TestR634_StopAndStartRefuseADeployingStack(t *testing.T) { + m, _, _, _ := newSlice4Manager(t) + // NEVER a real compose: the control below falls through to `compose down`, and the stack dir is + // named like a real app. An empty PATH + the v1 binary name guarantees the exec cannot find + // anything to run, on any machine this test runs on. + m.composeCmd = "docker-compose" + t.Setenv("PATH", t.TempDir()) + m.mu.Lock() + m.stacks["nextcloud"].Deploying = true + m.mu.Unlock() + if err := m.StopStack("nextcloud"); !errors.Is(err, ErrStackDeploying) { + t.Fatalf("StopStack on a deploying stack = %v, want ErrStackDeploying", err) + } + if err := m.StartStack("nextcloud"); !errors.Is(err, ErrStackDeploying) { + t.Fatalf("StartStack on a deploying stack = %v, want ErrStackDeploying", err) + } + // Control: once the deploy is over, the same calls are not refused for THIS reason. + m.mu.Lock() + m.stacks["nextcloud"].Deploying = false + m.mu.Unlock() + if err := m.StopStack("nextcloud"); errors.Is(err, ErrStackDeploying) { + t.Fatal("control: a stack that is not deploying must not be refused as deploying") + } +} diff --git a/controller/internal/stacks/undo.go b/controller/internal/stacks/undo.go index 35be22e..b74a67c 100644 --- a/controller/internal/stacks/undo.go +++ b/controller/internal/stacks/undo.go @@ -471,7 +471,11 @@ func (m *Manager) emitUpdateEvent(kind, name string, entry *updateJournalEntry, if entry != nil { ev.From, ev.To = entry.PrevPin, entry.NewPin } - m.logger.Printf("[INFO] [stacks] update %s: event %s (hold recorded: %v)", name, kind, holdRecorded) + if kind == UpdateEventHeld { + m.logger.Printf("[INFO] [stacks] update %s: event %s (hold recorded: %v)", name, kind, holdRecorded) + } else { // R-647: "hold recorded" means nothing for an undone update + m.logger.Printf("[INFO] [stacks] update %s: event %s", name, kind) + } sink(ev) } diff --git a/controller/internal/stacks/update.go b/controller/internal/stacks/update.go index 9514cbd..8e45992 100644 --- a/controller/internal/stacks/update.go +++ b/controller/internal/stacks/update.go @@ -564,7 +564,7 @@ func (m *Manager) finishUpdate(name, phase, msg string) { // beside it for the page. key "" = `plain` is a finished sentence and is stored as it is. func (m *Manager) finishUpdateKey(name, phase, key, plain string, args ...interface{}) { msg := plain - if key != "" { + if key != "" && key != UpdateErrorKeyHeld { // the held key names no bundle entry: plain IS the sentence msg = util.Text(i18n.Default, key, args...) } m.mu.Lock() @@ -590,12 +590,57 @@ func UpdatePhaseLabelIn(lang, phase string) string { // UpdateErrorIn is the stack's update sentence in lang (v0.264.0, R-606). func (s Stack) UpdateErrorIn(lang string) string { - if s.UpdateErrorKey == "" || lang == i18n.Default { + if s.UpdateErrorKey == "" || s.UpdateErrorKey == UpdateErrorKeyHeld || lang == i18n.Default { return s.UpdateError } return util.Text(lang, s.UpdateErrorKey, s.UpdateErrorArgs...) } +// UpdateErrorKeyHeld marks an UpdateError that IS the hold sentence (R-647, v0.265.0). It names no +// bundle entry: the sentence is composed by the backup side, so UpdateErrorFor asks it for the +// reader's language, and UpdateErrorIn (no manager to ask) returns the stored Hungarian. +const UpdateErrorKeyHeld = "update.error.held" + +// holdLanguage is the OPTIONAL half of UpdateGuards that renders the hold sentence in a given +// language (the production adapter implements it; the test fakes need not). +type holdLanguage interface { + HoldForLang(name, lang string) (bool, string) +} + +func holdForLang(g UpdateGuards, name, lang string) (bool, string) { + if g == nil { + return false, "" + } + if hl, ok := g.(holdLanguage); ok { + return hl.HoldForLang(name, lang) + } + return g.HoldFor(name) +} + +// HoldReasonFor is a stack's hold sentence in the READER's language (R-647): "" when nothing holds it. +// A Hungarian reader on a Hungarian box gets exactly Stack.HoldReason. +func (m *Manager) HoldReasonFor(st Stack, lang string) string { + if st.HoldReason == "" { + return "" + } + if held, why := holdForLang(m.guards(), st.Name, lang); held && why != "" { + return why + } + return st.HoldReason +} + +// UpdateErrorFor is a stack's update error in the READER's language (R-606 + R-647). A held update's +// error is the hold sentence, rendered by the backup side for this reader. +func (m *Manager) UpdateErrorFor(st Stack, lang string) string { + if st.UpdateErrorKey == UpdateErrorKeyHeld { + if held, why := holdForLang(m.guards(), st.Name, lang); held && why != "" { + return why + } + return st.UpdateError + } + return st.UpdateErrorIn(lang) +} + func (m *Manager) updateCompose(dir string, env []string, args ...string) (string, error) { if m.updateComposeFn != nil { return m.updateComposeFn(dir, env, args...) @@ -880,7 +925,7 @@ func (m *Manager) failAndHold(ctx context.Context, name, dir string, env []strin m.logger.Printf("[ERROR] [stacks] update %s: no UpdateGuards — the hold CANNOT be recorded", name) } else if err := g.HoldAfterFailedUpdate(name, m.now(), rp, undoState); err != nil { m.logger.Printf("[ERROR] [stacks] update %s: %v", name, err) - } else if _, w := g.HoldFor(name); w != "" { + } else if _, w := holdForLang(g, name, i18n.Default); w != "" { holdWhy = w m.markUpdateHeld(name) } @@ -890,8 +935,10 @@ func (m *Manager) failAndHold(ctx context.Context, name, dir string, env []strin if holdWhy == "" { m.finishUpdateKey(name, UpdatePhaseFailed, "update.error.hold_unsaved", "") } else { - // The hold's own sentence (the page renders it per reader through RestoreHoldForLang). - m.finishUpdate(name, UpdatePhaseFailed, holdWhy) + // R-647 (v0.265.0): stored as the HELD key + the Hungarian sentence, so every reader — the + // `?lang=` switch, an operator reading a box in the other language — gets the hold sentence in + // THEIR language (UpdateErrorFor), and the stored text is Hungarian as for every other key. + m.finishUpdateKey(name, UpdatePhaseFailed, UpdateErrorKeyHeld, holdWhy) } m.emitUpdateEvent(UpdateEventHeld, name, entry, rp, holdWhy != "") } diff --git a/controller/internal/web/i18n_web.go b/controller/internal/web/i18n_web.go index 59f0d41..d523d41 100644 --- a/controller/internal/web/i18n_web.go +++ b/controller/internal/web/i18n_web.go @@ -322,17 +322,6 @@ func (s *Server) localeFuncs(lang string) template.FuncMap { } return stk.UpdatePhaseLabel // no translation for this phase: the stored label, never an empty line }, - "updateErrorText": func(st interface{}) string { return stackOf(st).UpdateErrorIn(lang) }, - "holdText": func(st interface{}) string { - stk := stackOf(st) - if stk.HoldReason == "" || s.backupMgr == nil { - return stk.HoldReason - } - if held, why := s.backupMgr.RestoreHoldForLang(stk.Name, lang); held && why != "" { - return why - } - return stk.HoldReason - }, "timeAgo": func(t time.Time) string { if t.IsZero() { return "–" @@ -387,6 +376,13 @@ func (s *Server) localeFuncs(lang string) template.FuncMap { // Found live on 2026-09-20: on an English app page the only Felhom-authored Hungarian left // was this badge, „Naprakész", with a Hungarian tooltip under it. "updateBadge": func(st stacks.Stack) *MetaBadge { + if st.HoldReason != "" { // R-625's twin — see updatebadge.go + return &MetaBadge{ + Label: b.Msg(lang, "badge.update.held"), + Class: "tag-error", + Title: firstSentence(st.HoldReason), + } + } switch compareInstalledToTemplate(st) { case updateCurrent: return &MetaBadge{ @@ -554,3 +550,43 @@ func (s *Server) note(key string, args ...interface{}) string { // noteErr renders an error into a saved note in the box's language: its bundle message when it // carries one (release B), its own text otherwise. func (s *Server) noteErr(err error) string { return util.ErrText(s.boxLang(), err) } + +// readerFuncs are the funcs that need the SERVER to answer in the reader's language, and — unlike +// localeFuncs — they apply to EVERY language, Hungarian included (R-647, v0.265.0). A hold sentence is +// composed by the backup side in one language at a time; before this a Hungarian reader of an English +// box (the `?lang=` switch, an operator) read it in English. On a Hungarian box a Hungarian reader gets +// exactly the bytes templateFuncMap gave (the parity fixtures are rendered through here). +// `prev` is the func set they wrap (templateFuncMap + localeFuncs for this language). +func (s *Server) readerFuncs(lang string, prev template.FuncMap) template.FuncMap { + holdFor := func(st stacks.Stack) string { + if st.HoldReason == "" || s.backupMgr == nil { + return st.HoldReason + } + if held, why := s.backupMgr.RestoreHoldForLang(st.Name, lang); held && why != "" { + return why + } + return st.HoldReason + } + out := template.FuncMap{ + "holdText": func(st interface{}) string { return holdFor(stackOf(st)) }, + "updateErrorText": func(st interface{}) string { + stk := stackOf(st) + if stk.UpdateErrorKey == stacks.UpdateErrorKeyHeld && s.backupMgr != nil { + if held, why := s.backupMgr.RestoreHoldForLang(stk.Name, lang); held && why != "" { + return why + } + } + return stk.UpdateErrorIn(lang) + }, + } + if base, ok := prev["updateBadge"].(func(stacks.Stack) *MetaBadge); ok { + out["updateBadge"] = func(st stacks.Stack) *MetaBadge { + bd := base(st) + if bd != nil && st.HoldReason != "" { // R-625: the title is the hold's opening, for THIS reader + bd.Title = firstSentence(holdFor(st)) + } + return bd + } + } + return out +} diff --git a/controller/internal/web/r625_r647_held_test.go b/controller/internal/web/r625_r647_held_test.go new file mode 100644 index 0000000..56e1b7f --- /dev/null +++ b/controller/internal/web/r625_r647_held_test.go @@ -0,0 +1,136 @@ +package web + +import ( + "html" + "io" + "log" + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-controller/internal/backup" + "gitea.dooplex.hu/admin/felhom-controller/internal/i18n" + "gitea.dooplex.hu/admin/felhom-controller/internal/stacks" +) + +// heldBehindStack is R-625's measured shape: HELD by a failed update, while the catalog has already +// published a newer version — the moment the page used to say „Frissítés elérhető". +func heldBehindStack(t *testing.T, s *Server, boxLang string) (stacks.Stack, *backup.Manager) { + t.Helper() + b := backup.NewManager(s.cfg, s.settings, log.New(io.Discard, "", 0)) + s.backupMgr = b + if boxLang != "" { + if err := s.settings.SetLanguage(boxLang); err != nil { + t.Fatal(err) + } + } + at := time.Date(2026, 9, 23, 8, 0, 0, 0, time.UTC) + if err := b.HoldAfterFailedUpdateHolding("kimai", at, at.Add(-6*time.Hour), backup.UpdateTierLocal, b.UpdateCopyHolds("kimai", backup.UpdateTierLocal), ""); err != nil { + t.Fatal(err) + } + _, why := b.RestoreHoldFor("kimai") // in the BOX's language, as fillHoldReason stores it + st := i18nStack("kimai", "Kimai Time", stacks.StateStopped, true) + st.AppConfig = &stacks.AppConfig{Deployed: true, InstalledImages: map[string]stacks.InstalledImage{"web": {Ref: "kimai/kimai2:2.30.0"}}} + st.CatalogImages = map[string]string{"web": "kimai/kimai2:2.31.0"} + if compareInstalledToTemplate(st) != updateBehind { + t.Fatal("fixture: the held app must also be BEHIND the catalog (R-625's shape)") + } + st.HoldReason = why + st.UpdatePhase, st.UpdateErrorKey, st.UpdateError = stacks.UpdatePhaseFailed, stacks.UpdateErrorKeyHeld, why + return st, b +} + +func renderHeld(t *testing.T, s *Server, st stacks.Stack, lang, tmpl string) string { + t.Helper() + c := i18nCase{"r625_" + tmpl, tmpl, func() map[string]interface{} { + if tmpl == "stacks" { + d := i18nLayoutData("stacks", "Alkalmazások") + d["Stacks"] = []stacks.Stack{st} + for _, k := range []string{"Subdomains", "MissingStorage", "NetworkStubs", "NetworkWarnings", "StorageLabels"} { + d[k] = map[string]string{} + } + return d + } + d := i18nLayoutData("stacks", "Kimai") + cp := st + d["Stack"], d["Meta"], d["AppInfo"] = &cp, cp.Meta, cp.Meta.AppInfo + return d + }} + return html.UnescapeString(renderI18nCase(t, s, lang, c)) +} + +// R-625 — a held app says it is stopped and needs a restore, in both languages, on the app page; and +// NEITHER page offers the Update button. +// +// COMPANION RED-PROOF (REPORT.md): remove the HoldReason branch from updateBadgeAt (v0.264.0's shape) +// — the Hungarian page says „Frissítés elérhető" and this fails. +func TestR625_AHeldAppSaysStoppedRestoreNeeded(t *testing.T) { + s := i18nTestServer(t) + st, _ := heldBehindStack(t, s, "") + for _, c := range []struct{ lang, badge, invite, title string }{ + {"hu", "Megállítva — visszaállítás szükséges", "Frissítés elérhető", "A(z) kimai frissítése"}, + {"en", "Stopped — restore needed", "Update available", "The update of kimai"}, + } { + page := renderHeld(t, s, st, c.lang, "app_info") + if !strings.Contains(page, ` 0 && !strings.Contains(page[max(0, i-400):i], c.title) { + t.Errorf("%s: the badge title must open with the hold sentence (%q)", c.lang, c.title) + } + for _, tmpl := range []string{"app_info", "stacks"} { + if strings.Contains(renderHeld(t, s, st, c.lang, tmpl), `'kimai', 'update')`) { + t.Errorf("%s %s: the Update button is rendered for a HELD app", c.lang, tmpl) + } + } + } + // CONTROL: the same app, not held and running, DOES get the button on the list — so the check above + // can see a button when there is one. + free := st + free.HoldReason, free.UpdateError, free.UpdateErrorKey, free.State = "", "", "", stacks.StateRunning + if !strings.Contains(renderHeld(t, s, free, "hu", "stacks"), `'kimai', 'update')`) { + t.Fatal("control: a running, not-held app must show the Update button on the list") + } +} + +// R-647 (1) — on an ENGLISH box, a Hungarian reader (`?lang=hu`, an operator) reads the hold and the +// held update's error in HUNGARIAN; the English reader reads English. Both ways, both pages. +// +// COMPANION RED-PROOF (REPORT.md): make readerFuncs return nil (v0.264.0: no Hungarian override) — the +// Hungarian page carries the English hold and this fails. +func TestR647_AReaderInTheOtherLanguageReadsTheHoldInTheirs(t *testing.T) { + s := i18nTestServer(t) + st, _ := heldBehindStack(t, s, "en") + if !strings.Contains(st.HoldReason, "The update of kimai") { + t.Fatalf("fixture: an English box must store the English hold, got %q", st.HoldReason) + } + const huWant, enWant = "A(z) kimai frissítése 2026-09-23 10:00-kor nem sikerült", "The update of kimai at 2026-09-23 10:00 did not succeed" + for _, tmpl := range []string{"app_info", "stacks"} { + hu := renderHeld(t, s, st, "hu", tmpl) + if !strings.Contains(hu, huWant) || strings.Contains(hu, enWant) { + t.Errorf("%s: the Hungarian reader of an English box must read the hold in Hungarian only", tmpl) + } + en := renderHeld(t, s, st, "en", tmpl) + if !strings.Contains(en, enWant) || strings.Contains(en, huWant) { + t.Errorf("%s: the English reader must read the hold in English only", tmpl) + } + } +} + +// The held badge's Hungarian literal and the bundle say the same thing (the parity rule for funcmap). +func TestR625_HeldBadgeHungarianMatchesTheBundle(t *testing.T) { + b, err := i18n.Shared() + if err != nil { + t.Fatal(err) + } + st := stacks.Stack{Name: "x", HoldReason: "Egy. Kettő."} + if got, want := b.Msg("hu", "badge.update.held"), updateBadgeAt(st, time.Now()).Label; got != want { + t.Fatalf("badge.update.held: hu.json %q, funcmap %q", got, want) + } + if got := updateBadgeAt(st, time.Now()).Title; got != "Egy." { + t.Fatalf("the title is the first sentence, got %q", got) + } +} diff --git a/controller/internal/web/server.go b/controller/internal/web/server.go index b78c9bf..512f6e8 100644 --- a/controller/internal/web/server.go +++ b/controller/internal/web/server.go @@ -310,6 +310,9 @@ func (s *Server) loadTemplates() { for name, fn := range s.localeFuncs(lang) { funcs[name] = fn } + for name, fn := range s.readerFuncs(lang, funcs) { + funcs[name] = fn + } start := time.Now() t, st, err := parseTemplateSet(b, lang, funcs) took := time.Since(start) diff --git a/controller/internal/web/testdata/i18n_parity/app_info_deployed.html b/controller/internal/web/testdata/i18n_parity/app_info_deployed.html index 31d4514..4e24bac 100644 --- a/controller/internal/web/testdata/i18n_parity/app_info_deployed.html +++ b/controller/internal/web/testdata/i18n_parity/app_info_deployed.html @@ -177,7 +177,7 @@ Fut - + Megállítva — visszaállítás szükséges Megnyitás ↗ Napló diff --git a/controller/internal/web/testdata/i18n_parity/stacks_full.html b/controller/internal/web/testdata/i18n_parity/stacks_full.html index b48c20a..648a50d 100644 --- a/controller/internal/web/testdata/i18n_parity/stacks_full.html +++ b/controller/internal/web/testdata/i18n_parity/stacks_full.html @@ -252,7 +252,7 @@ Leállítva - + Megállítva — visszaállítás szükséges diff --git a/controller/internal/web/updatebadge.go b/controller/internal/web/updatebadge.go index 2e0d357..96dda11 100644 --- a/controller/internal/web/updatebadge.go +++ b/controller/internal/web/updatebadge.go @@ -2,6 +2,7 @@ package web import ( "fmt" + "strings" "time" "gitea.dooplex.hu/admin/felhom-controller/internal/stacks" @@ -73,6 +74,18 @@ func compareInstalledToTemplate(s stacks.Stack) updateState { // never the decision. A new branch here needs its twin there, and // TestLocaleFuncsHungarianBundleMatchesFuncMap fails if hu.json and these literals disagree. func updateBadgeAt(s stacks.Stack, now time.Time) *MetaBadge { + // R-625 (v0.265.0). A HELD app is stopped, and the only way back is a restore — a newer catalog + // version does not lift the hold (`09` §6.1), so „Frissítés elérhető" beside a button that answers + // 409 was two verdicts disagreeing. One verdict, read by the badge and by the button (stacks.html + // renders no Update button while HoldReason is set), exactly as R-524 did it for Ahead. True for + // both hold kinds: a failed restore holds the app stopped too, and the way back is also a restore. + if s.HoldReason != "" { + return &MetaBadge{ + Label: "Megállítva — visszaállítás szükséges", + Class: "tag-error", + Title: firstSentence(s.HoldReason), + } + } switch compareInstalledToTemplate(s) { case updateCurrent: return &MetaBadge{ @@ -120,3 +133,12 @@ func updateBadgeAt(s stacks.Stack, now time.Time) *MetaBadge { // // It is INFORMATION ONLY. It is wired to no action, and the Frissítés button is untouched. func updateBadge(s stacks.Stack) *MetaBadge { return updateBadgeAt(s, time.Now().UTC()) } + +// firstSentence is the hold sentence's opening clause, for a badge title (R-625): up to and including +// the first full stop that ends a sentence; the whole text when it has none. +func firstSentence(s string) string { + if i := strings.Index(s, ". "); i >= 0 { + return s[:i+1] + } + return s +} diff --git a/controller/scripts/i18n_go_keys.json b/controller/scripts/i18n_go_keys.json index 2340da3..e20cd5d 100644 --- a/controller/scripts/i18n_go_keys.json +++ b/controller/scripts/i18n_go_keys.json @@ -61,7 +61,8 @@ "page.title.storage_network": "slice 1 -- pinned by TestHandlerTitleKeysMatchHungarianTitle", "event.app_update_undone": "BORN AS A KEY, v0.264.0 (R-606 / the update events) -- a NEW sentence, never a Go literal.", "settings_notifications.app_update_undone": "BORN AS A KEY, v0.264.0 (R-606 / the update events) -- a NEW sentence, never a Go literal.", - "settings_notifications.app_update_held": "BORN AS A KEY, v0.264.0 (R-606 / the update events) -- a NEW sentence, never a Go literal." + "settings_notifications.app_update_held": "BORN AS A KEY, v0.264.0 (R-606 / the update events) -- a NEW sentence, never a Go literal.", + "badge.update.held": "BORN AS A KEY, v0.265.0 (R-625) -- a NEW badge for a held app; its Hungarian twin in updatebadge.go is pinned by TestR625_HeldBadgeHungarianMatchesTheBundle." }, "flash.share.already_on": "A megosztás már be van kapcsolva.", "flash.share.enable_failed": "A megosztás bekapcsolása nem sikerült.",