controller v0.265.0: R-634 cause fixed, held apps say so, OOM storm alarm, R-647 leftovers
gates / gates (push) Successful in 27s
gates / gates (push) Successful in 27s
R-634: a whole-box backup no longer stops/restarts a DEPLOYING app (the measured cause of containers running under 'not deployed'); StopStack and StartStack refuse a deploying stack for every caller. R-625: held badge 'Stopped - restore needed', no Update button. R-636: kernel oom_kill counter; 20+ in 30 min -> one app_oom_storm. R-647: held error per reader, copy_holds key, two log wordings. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -1,3 +1,29 @@
|
|||||||
|
## v0.265.0 — the cause of "runs but not installed", a held app that says so, a louder OOM storm (2026-09-23, R-634, R-625, R-636, R-647)
|
||||||
|
|
||||||
|
**MinAgent: 0.131.0** (unchanged). **Needs hub v0.121.0** (deployed first). New strings: yes (hu + en).
|
||||||
|
|
||||||
|
- **R-634 — the mechanism, diagnosed and fixed.** Reproduced on 9202: deploy `outline`, press the
|
||||||
|
whole-box backup 20 s later. The backup's list read the in-memory `Deployed` flag, which is true from
|
||||||
|
the moment a deploy is ACCEPTED, so the volume leg stopped the deploying app (`compose down`), dumped its
|
||||||
|
half-made volumes and ran a SECOND `compose up -d` beside the deploy's own; both failed and the deploy
|
||||||
|
recorded „not deployed" (on 2026-09-22 the backup's `up` won and the containers ran under it). Fix:
|
||||||
|
`ListDeployedStacks` leaves out a deploying app; the volume leg asks again right before the stop (SKIP,
|
||||||
|
never a failure); `StopStack` / `StartStack` refuse a deploying stack for every caller
|
||||||
|
(`ErrStackDeploying`). `sparkyfitness` did not reproduce alone (deployed in 47 s).
|
||||||
|
- **R-625 — a held app shows the truth.** The update badge of a held app reads „Megállítva —
|
||||||
|
visszaállítás szükséges" / "Stopped — restore needed" (`tag-error`, title = the hold's first sentence
|
||||||
|
in the reader's language). No Update button (the list already hid it; now pinned). Parity fixtures
|
||||||
|
`app_info_deployed` and `stacks_full` regenerated — measured diff: exactly the new badge, one line each.
|
||||||
|
- **R-636 — a repeating OOM problem gets louder.** The scan reads the kernel's `oom_kill` counter,
|
||||||
|
`memory.max` and `memory.peak` for flagged containers; 20+ kills in 30 min of the same container run →
|
||||||
|
ONE `app_oom_storm` (error, operator-only). `app_oom` unchanged. RomM's measured rate was ≈375 kills /
|
||||||
|
30 min; one hiccup is 1–3.
|
||||||
|
- **R-647 — the three leftovers.** A held update's error is the key `update.error.held`, rendered per
|
||||||
|
reader (page funcs now apply to Hungarian too; same bytes on a Hungarian box); the `app_update_held`
|
||||||
|
details carry the `copy_holds` KEY, not the Hungarian phrase; the undone event no longer logs
|
||||||
|
`hold recorded`, and the disabled-notifier line names the real severity of a health change.
|
||||||
|
- Red-proofs: eight in this repo, each seen failing (REPORT).
|
||||||
|
|
||||||
## v0.264.0 — the undo reaches the fleet, and the household is TOLD, in its own language (2026-09-23, `09` §6.4 parts 2–3)
|
## v0.264.0 — the undo reaches the fleet, and the household is TOLD, in its own language (2026-09-23, `09` §6.4 parts 2–3)
|
||||||
|
|
||||||
**MinAgent: 0.131.0** (unchanged). **Needs hub v0.120.0** (deployed first). New strings: yes (hu + en).
|
**MinAgent: 0.131.0** (unchanged). **Needs hub v0.120.0** (deployed first). New strings: yes (hu + en).
|
||||||
|
|||||||
@@ -650,6 +650,18 @@ the old version back; a power cut while undoing resumes the undo. Reasoning and
|
|||||||
`felhom.eu/documentation/architecture/09-update-architecture.md` §6.1a,
|
`felhom.eu/documentation/architecture/09-update-architecture.md` §6.1a,
|
||||||
`felhom.eu/documentation/audits/undo-bakeoff-2026-09-23/`.
|
`felhom.eu/documentation/audits/undo-bakeoff-2026-09-23/`.
|
||||||
|
|
||||||
|
**Held apps say so (v0.265.0, R-625).** While a hold stands, the update badge reads „Megállítva —
|
||||||
|
visszaállítás szükséges" / "Stopped — restore needed" (`tag-error`, title = the hold sentence's first
|
||||||
|
sentence, in the reader's language) and no Update button is rendered; the API still answers 409 `held`. A
|
||||||
|
held update's error is stored as the key `update.error.held` and rendered per reader, so a reader in the
|
||||||
|
other language than the box reads the hold in theirs (R-647).
|
||||||
|
|
||||||
|
**Deploys are not interrupted (v0.265.0, R-634).** A deploy still running is not in any backup leg's app
|
||||||
|
list, the volume leg asks again right before it stops an app, and `StopStack` / `StartStack` refuse a
|
||||||
|
deploying stack (`ErrStackDeploying`) for every caller. Measured cause: the whole-box backup's `compose down`
|
||||||
|
and second `compose up -d` in the middle of the deploy's own `up` — the deploy then recorded „not deployed"
|
||||||
|
over running containers.
|
||||||
|
|
||||||
**The household is told (v0.264.0).** An undone update sends `app_update_undone` (warning) ONCE; an
|
**The household is told (v0.264.0).** An undone update sends `app_update_undone` (warning) ONCE; an
|
||||||
update that ends held sends `app_update_held` (error) ONCE — also when the hold itself could not be
|
update that ends held sends `app_update_held` (error) ONCE — also when the hold itself could not be
|
||||||
saved. Details `{app, stack_name, from, to, at, copy_tier, copy_date, copy_holds}`. Both are in
|
saved. Details `{app, stack_name, from, to, at, copy_tier, copy_date, copy_holds}`. Both are in
|
||||||
@@ -1132,6 +1144,7 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri
|
|||||||
> **Absent-storage tier skip (v0.243.0, R-518).** A tier whose storage the agent (≥ 0.131.0) reports `absent` is dropped from the manual and the scheduled run before anything is stopped (`quiesce.skipAbsentTiers`), logged, and reported once as `backup_tier_skipped`. `unknown` or a legacy agent is never skipped.
|
> **Absent-storage tier skip (v0.243.0, R-518).** A tier whose storage the agent (≥ 0.131.0) reports `absent` is dropped from the manual and the scheduled run before anything is stopped (`quiesce.skipAbsentTiers`), logged, and reported once as `backup_tier_skipped`. `unknown` or a legacy agent is never skipped.
|
||||||
|
|
||||||
> **OOM visibility (v0.243.0, R-514).** The 30 s dead-app check also reads `State.OOMKilled` for running app containers (`Manager.ScanOOMKilled`); the dashboard shows „Memória elfogyott" and the hub gets `app_oom` once per container run. The controller does not restart the app.
|
> **OOM visibility (v0.243.0, R-514).** The 30 s dead-app check also reads `State.OOMKilled` for running app containers (`Manager.ScanOOMKilled`); the dashboard shows „Memória elfogyott" and the hub gets `app_oom` once per container run. The controller does not restart the app.
|
||||||
|
> **OOM storm (v0.265.0, R-636).** For a flagged container the scan also reads the kernel's own kill counter (`memory.events` `oom_kill`, plus `memory.max` / `memory.peak`) with one `docker exec … cat` — `OOMKilled` is sticky, so it cannot count. When the SAME container run's counter rises by **20 or more within 30 minutes**, the notifier sends **`app_oom_storm`** (severity `error`, operator-only, details `{app, container, kills, window_min, mem_limit, peak}`) — ONCE per container run. `app_oom` itself is unchanged. An unreadable counter (an image without `cat`) never escalates.
|
||||||
|
|
||||||
> **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest
|
> **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest
|
||||||
> backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0,
|
> backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0,
|
||||||
@@ -2432,6 +2445,7 @@ The controller pushes structured events to the Hub's `/api/v1/event` endpoint. T
|
|||||||
| `app_start_failed` | **warning** | A DEPLOYED app is not running (fix-3) — fired ONCE per running→down transition. **Customer-switchable („Alkalmazás nem fut"), OFF by default; the OPERATOR is e-mailed regardless.** Was `warn` until v0.223.0 — see the severity note below |
|
| `app_start_failed` | **warning** | A DEPLOYED app is not running (fix-3) — fired ONCE per running→down transition. **Customer-switchable („Alkalmazás nem fut"), OFF by default; the OPERATOR is e-mailed regardless.** Was `warn` until v0.223.0 — see the severity note below |
|
||||||
| `app_update_undone` | warning | v0.264.0 — a guarded update failed and the box put the previous version and its data back. Once per app per failed step. Household ON by default |
|
| `app_update_undone` | warning | v0.264.0 — a guarded update failed and the box put the previous version and its data back. Once per app per failed step. Household ON by default |
|
||||||
| `app_update_held` | **error** | v0.264.0 — a guarded update (or its undo) failed and the app is held until a restore. The mail's line is the hold sentence in the household's language. Household ON by default |
|
| `app_update_held` | **error** | v0.264.0 — a guarded update (or its undo) failed and the app is held until a restore. The mail's line is the hold sentence in the household's language. Household ON by default |
|
||||||
|
| `app_oom_storm` | **error** | v0.265.0 (R-636) — the same container run OOM-killed 20+ times in 30 min (the kernel's `oom_kill` counter). Once per container run. Operator-only |
|
||||||
| `disaster_recovery_started` | warning | DR restore begins |
|
| `disaster_recovery_started` | warning | DR restore begins |
|
||||||
| `disaster_recovery_completed` | info/error | DR restore finishes (success/partial) |
|
| `disaster_recovery_completed` | info/error | DR restore finishes (success/partial) |
|
||||||
|
|
||||||
|
|||||||
@@ -620,7 +620,7 @@ func main() {
|
|||||||
d.CopyDate = ev.CopyDate.UTC().Format(time.RFC3339)
|
d.CopyDate = ev.CopyDate.UTC().Format(time.RFC3339)
|
||||||
}
|
}
|
||||||
if backupMgr != nil && ev.CopyTier > 0 {
|
if backupMgr != nil && ev.CopyTier > 0 {
|
||||||
d.CopyHolds = backupMgr.UpdateCopyHolds(ev.App, ev.CopyTier)
|
d.CopyHolds = backupMgr.UpdateCopyHoldsKey(ev.App, ev.CopyTier) // R-647: the key, never the Hungarian phrase
|
||||||
}
|
}
|
||||||
notifier.NotifyAppUpdateHeld(d, func(lang string) string {
|
notifier.NotifyAppUpdateHeld(d, func(lang string) string {
|
||||||
if ev.HoldRecorded && backupMgr != nil {
|
if ev.HoldRecorded && backupMgr != nil {
|
||||||
@@ -851,8 +851,8 @@ func main() {
|
|||||||
logger.Printf("[WARN] [deadapp] OOM scan failed: %v", oerr)
|
logger.Printf("[WARN] [deadapp] OOM scan failed: %v", oerr)
|
||||||
} else {
|
} else {
|
||||||
for _, o := range ooms {
|
for _, o := range ooms {
|
||||||
logger.Printf("[WARN] [deadapp] %s: container %s was OOM-killed (started %s)", o.Stack, o.Container, o.StartedAt)
|
logger.Printf("[WARN] [deadapp] %s: container %s was OOM-killed (started %s; kills %d, limit %s, peak %s)", o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak)
|
||||||
notifier.NotifyAppOOM(o.Stack, o.Container, o.StartedAt)
|
notifier.NotifyAppOOM(o.Stack, o.Container, o.StartedAt, o.Kills, o.MemLimit, o.Peak)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
deadAppScans++
|
deadAppScans++
|
||||||
@@ -2556,7 +2556,10 @@ func (a *stackAdapter) GetStackClassifiedBinds(name string) ([]backup.Classified
|
|||||||
func (a *stackAdapter) ListDeployedStacks() []backup.StackSummary {
|
func (a *stackAdapter) ListDeployedStacks() []backup.StackSummary {
|
||||||
var result []backup.StackSummary
|
var result []backup.StackSummary
|
||||||
for _, s := range a.mgr.GetStacks() {
|
for _, s := range a.mgr.GetStacks() {
|
||||||
if !s.Deployed {
|
// R-634 (v0.265.0): `Deployed` is set true in memory the moment a deploy is ACCEPTED (the
|
||||||
|
// no-stale-button UX), so without the second test a deploy still pulling was in every backup
|
||||||
|
// leg's list — and the volume leg stopped and restarted it in the middle of its own `up`.
|
||||||
|
if !s.Deployed || s.Deploying {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
result = append(result, backup.StackSummary{
|
result = append(result, backup.StackSummary{
|
||||||
@@ -2574,6 +2577,13 @@ func (a *stackAdapter) StopStack(name string) error {
|
|||||||
return a.mgr.StopStack(name)
|
return a.mgr.StopStack(name)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// IsDeploying answers backup's optional deployingReporter (R-634): the volume leg asks it again
|
||||||
|
// immediately before it stops an app, because its list is taken once at the start of the run.
|
||||||
|
func (a *stackAdapter) IsDeploying(name string) bool {
|
||||||
|
s, ok := a.mgr.GetStack(name)
|
||||||
|
return ok && s.Deploying
|
||||||
|
}
|
||||||
|
|
||||||
func (a *stackAdapter) StartStack(name string) error {
|
func (a *stackAdapter) StartStack(name string) error {
|
||||||
return a.mgr.StartStack(name)
|
return a.mgr.StartStack(name)
|
||||||
}
|
}
|
||||||
@@ -3471,6 +3481,14 @@ func (a *updateGuardsAdapter) HoldFor(name string) (bool, string) {
|
|||||||
return a.b.RestoreHoldFor(name)
|
return a.b.RestoreHoldFor(name)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// HoldForLang renders the hold sentence for one reader (R-647, v0.265.0).
|
||||||
|
func (a *updateGuardsAdapter) HoldForLang(name, lang string) (bool, string) {
|
||||||
|
if a.b == nil {
|
||||||
|
return false, ""
|
||||||
|
}
|
||||||
|
return a.b.RestoreHoldForLang(name, lang)
|
||||||
|
}
|
||||||
|
|
||||||
func (a *updateGuardsAdapter) Busy(name string) (bool, string) {
|
func (a *updateGuardsAdapter) Busy(name string) (bool, string) {
|
||||||
if a.q.SuppressedStacks()[name] {
|
if a.q.SuppressedStacks()[name] {
|
||||||
return true, "a whole-guest backup (quiesce) is holding it"
|
return true, "a whole-guest backup (quiesce) is holding it"
|
||||||
|
|||||||
@@ -1,6 +1,10 @@
|
|||||||
package main
|
package main
|
||||||
|
|
||||||
import "testing"
|
import (
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
// v0.264.0. The update-event sink and the R-646 backfill are both CALLED from main.go (an AST walk —
|
// v0.264.0. The update-event sink and the R-646 backfill are both CALLED from main.go (an AST walk —
|
||||||
// a comment naming them would not count). A sink built and never wired would fail silently: no event,
|
// a comment naming them would not count). A sink built and never wired would fail silently: no event,
|
||||||
@@ -21,3 +25,16 @@ func TestUpdateEventSinkIsWiredAtStartup(t *testing.T) {
|
|||||||
t.Error("BackfillAppliedMeta (R-646) must be called, after AdoptPins")
|
t.Error("BackfillAppliedMeta (R-646) must be called, after AdoptPins")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// R-647 (2) — the held event carries the copy_holds KEY, never the Hungarian phrase: the hub prints the
|
||||||
|
// raw details as the household mail's `Note:` line. COMPANION RED-PROOF (REPORT.md): wire
|
||||||
|
// UpdateCopyHolds back — this fails.
|
||||||
|
func TestR647_HeldEventCarriesTheCopyHoldsKey(t *testing.T) {
|
||||||
|
src, err := os.ReadFile("main.go")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(src), "d.CopyHolds = backupMgr.UpdateCopyHoldsKey(") {
|
||||||
|
t.Fatal("main.go must fill AppUpdateDetails.CopyHolds with UpdateCopyHoldsKey (the key), not the Hungarian phrase")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -19,7 +19,6 @@ import (
|
|||||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||||
cf "gitea.dooplex.hu/admin/felhom-controller/internal/cloudflare"
|
cf "gitea.dooplex.hu/admin/felhom-controller/internal/cloudflare"
|
||||||
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||||
"gitea.dooplex.hu/admin/felhom-controller/internal/i18n"
|
|
||||||
"gitea.dooplex.hu/admin/felhom-controller/internal/integrations"
|
"gitea.dooplex.hu/admin/felhom-controller/internal/integrations"
|
||||||
"gitea.dooplex.hu/admin/felhom-controller/internal/metrics"
|
"gitea.dooplex.hu/admin/felhom-controller/internal/metrics"
|
||||||
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
|
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
|
||||||
@@ -383,18 +382,12 @@ func (r *Router) getStack(w http.ResponseWriter, req *http.Request, name string)
|
|||||||
// v0.264.0 (R-606): the page polls this for the Update button's label and the outcome. Rendered in
|
// v0.264.0 (R-606): the page polls this for the Update button's label and the outcome. Rendered in
|
||||||
// the reader's language; a Hungarian reader gets exactly the bytes it always got. GetStack returns a
|
// the reader's language; a Hungarian reader gets exactly the bytes it always got. GetStack returns a
|
||||||
// COPY, so nothing here touches the manager's own state.
|
// COPY, so nothing here touches the manager's own state.
|
||||||
if lang := r.langFor(req); lang != i18n.Default {
|
// v0.265.0 (R-647): EVERY reader, Hungarian included — a Hungarian reader of an English box read
|
||||||
|
// the hold in English before. On a Hungarian box a Hungarian reader still gets the same bytes.
|
||||||
|
lang := r.langFor(req)
|
||||||
stack.UpdatePhaseLabel = stacks.UpdatePhaseLabelIn(lang, stack.UpdatePhase)
|
stack.UpdatePhaseLabel = stacks.UpdatePhaseLabelIn(lang, stack.UpdatePhase)
|
||||||
stack.UpdateError = stack.UpdateErrorIn(lang)
|
stack.UpdateError = r.stackMgr.UpdateErrorFor(*stack, lang)
|
||||||
if stack.HoldReason != "" && r.backupMgr != nil {
|
stack.HoldReason = r.stackMgr.HoldReasonFor(*stack, lang)
|
||||||
if held, why := r.backupMgr.RestoreHoldForLang(name, lang); held && why != "" {
|
|
||||||
stack.HoldReason = why
|
|
||||||
if stack.UpdateErrorKey == "" && stack.UpdatePhase == stacks.UpdatePhaseFailed {
|
|
||||||
stack.UpdateError = why // a held update's UpdateError IS the hold sentence
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
writeJSON(w, http.StatusOK, apiResponse{OK: true, Data: stack})
|
writeJSON(w, http.StatusOK, apiResponse{OK: true, Data: stack})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -724,6 +724,16 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// R-634 (v0.265.0): an app whose deploy is still running is NOT an installation to back up,
|
||||||
|
// and stopping it is what broke the deploy. The list is taken once at the start of the run, so
|
||||||
|
// the question is asked again here, immediately before the stop. A SKIP, not a failure: there
|
||||||
|
// is nothing of the household's in it yet, and a FAIL would page the operator for a deploy.
|
||||||
|
if m.stackIsDeploying(stack.Name) {
|
||||||
|
m.logger.Printf("[INFO] [backup] Skipping volume dump for %s — the app is still being deployed (R-634)", stack.Name)
|
||||||
|
summary = append(summary, fmt.Sprintf("SKIP %s volumes (deploying)", stack.Name))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
// R-181: the reserve, ahead of DumpAppVolumesSafe so a refused app is NOT stopped. For an app
|
// R-181: the reserve, ahead of DumpAppVolumesSafe so a refused app is NOT stopped. For an app
|
||||||
// that already has a DB this is a memo lookup taken before its DB dump; for a volume-only app
|
// that already has a DB this is a memo lookup taken before its DB dump; for a volume-only app
|
||||||
// this is where its verdict is taken, still before its first byte.
|
// this is where its verdict is taken, still before its first byte.
|
||||||
@@ -1398,3 +1408,17 @@ func (m *Manager) note(key string, args ...interface{}) string {
|
|||||||
// noteErr renders an error into a saved note in the box's language: its bundle message when it
|
// noteErr renders an error into a saved note in the box's language: its bundle message when it
|
||||||
// carries one (release B), its own text otherwise.
|
// carries one (release B), its own text otherwise.
|
||||||
func (m *Manager) noteErr(err error) string { return util.ErrText(m.boxLang(), err) }
|
func (m *Manager) noteErr(err error) string { return util.ErrText(m.boxLang(), err) }
|
||||||
|
|
||||||
|
// deployingReporter is the OPTIONAL half of the stack provider that can say a deploy is in flight
|
||||||
|
// (R-634). Optional on purpose: the production adapter implements it, and the many test fakes that
|
||||||
|
// predate it need not — a provider that cannot answer is read as "not deploying", today's behaviour.
|
||||||
|
type deployingReporter interface {
|
||||||
|
IsDeploying(name string) bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Manager) stackIsDeploying(name string) bool {
|
||||||
|
if d, ok := m.stackProvider.(deployingReporter); ok {
|
||||||
|
return d.IsDeploying(name)
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,61 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-634 (v0.265.0). Measured 2026-09-23 on 9202: a whole-box backup pressed 20 s into outline's
|
||||||
|
// deploy stopped it (`compose down`), dumped its half-made volumes and ran a SECOND `compose up -d`
|
||||||
|
// beside the deploy's own — both failed, and the deploy recorded „not deployed".
|
||||||
|
|
||||||
|
type deployingFake struct {
|
||||||
|
volDumpFakeProvider
|
||||||
|
deploying map[string]bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *deployingFake) IsDeploying(name string) bool { return f.deploying[name] }
|
||||||
|
|
||||||
|
// Through the dumpVolumesSafe SEAM, never the real dump: the real one runs `docker run … tar` against a
|
||||||
|
// named volume, which on a developer machine CREATES that volume (it did, once, on DooPlex while this
|
||||||
|
// test was written). The seam is where the stop happens, so a call to it IS the leak.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (REPORT.md): delete the stackIsDeploying block in runVolumeDumps — the dump
|
||||||
|
// (and with it the stop) is called for outline and this fails with dumped=[outline].
|
||||||
|
func TestR634_VolumeLegNeverStopsADeployingApp(t *testing.T) {
|
||||||
|
cfg := &config.Config{}
|
||||||
|
cfg.Paths.SystemDataPath = filepath.Join(t.TempDir(), "sys")
|
||||||
|
fake := &deployingFake{
|
||||||
|
volDumpFakeProvider: volDumpFakeProvider{
|
||||||
|
stacks: []StackSummary{{Name: "outline"}},
|
||||||
|
volumes: map[string][]string{"outline": {"outline_outline_data"}},
|
||||||
|
},
|
||||||
|
deploying: map[string]bool{"outline": true},
|
||||||
|
}
|
||||||
|
m := &Manager{cfg: cfg, logger: log.New(io.Discard, "", 0), systemDataPath: cfg.Paths.SystemDataPath,
|
||||||
|
stackProvider: fake}
|
||||||
|
var calls []string
|
||||||
|
m.dumpVolumesSafe = func(name string) error { calls = append(calls, name); return nil }
|
||||||
|
|
||||||
|
summary, dumped, ok := m.runVolumeDumps()
|
||||||
|
if len(calls) != 0 {
|
||||||
|
t.Fatalf("a DEPLOYING app was stopped and dumped by the backup: dumped=%v (R-634's race)", calls)
|
||||||
|
}
|
||||||
|
if !ok || dumped != 0 {
|
||||||
|
t.Errorf("a deploy in flight is a SKIP, never a failure that pages the operator: ok=%v dumped=%d %v", ok, dumped, summary)
|
||||||
|
}
|
||||||
|
if !containsSummary(summary, "SKIP outline volumes (deploying)") {
|
||||||
|
t.Errorf("the skip must be visible in the run summary, got %v", summary)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Control: the same app once its deploy has finished IS backed up (the skip is conditional).
|
||||||
|
fake.deploying["outline"] = false
|
||||||
|
m.runVolumeDumps()
|
||||||
|
if len(calls) != 1 || calls[0] != "outline" {
|
||||||
|
t.Errorf("control: a finished app must be dumped as before, dumped=%v", calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -7,6 +7,10 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/i18n"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/util"
|
||||||
)
|
)
|
||||||
|
|
||||||
// v0.263.0 — a hold after a FAILED UNDO opens with what was tried and what state the data is in, in the
|
// v0.263.0 — a hold after a FAILED UNDO opens with what was tried and what state the data is in, in the
|
||||||
@@ -65,3 +69,15 @@ func TestUndo_HoldSentenceSaysTheUndoWasTriedAndTheDataState(t *testing.T) {
|
|||||||
t.Errorf("no Hungarian may remain on an English box, got %q", why)
|
t.Errorf("no Hungarian may remain on an English box, got %q", why)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// R-647 (2) — the key form of the copy verdict is a bundle KEY, the same verdict as the phrase.
|
||||||
|
func TestR647_UpdateCopyHoldsKeyIsTheKeyOfThePhrase(t *testing.T) {
|
||||||
|
m := NewManager(&config.Config{}, nil, log.New(io.Discard, "", 0))
|
||||||
|
k := m.UpdateCopyHoldsKey("x", UpdateTierLocal)
|
||||||
|
if !strings.HasPrefix(k, "hold.copy_holds.") {
|
||||||
|
t.Fatalf("want a hold.copy_holds.* key, got %q", k)
|
||||||
|
}
|
||||||
|
if util.Text(i18n.Default, k) != m.UpdateCopyHolds("x", UpdateTierLocal) {
|
||||||
|
t.Fatal("the key must render to exactly the phrase UpdateCopyHolds gives")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -152,6 +152,13 @@ func (m *Manager) UpdateCopyHolds(stackName string, tier int) string {
|
|||||||
return util.Text(i18n.Default, updateCopyHoldsKey(m.DataOutsideUnit(stackName), tier))
|
return util.Text(i18n.Default, updateCopyHoldsKey(m.DataOutsideUnit(stackName), tier))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// UpdateCopyHoldsKey is the same verdict as UpdateCopyHolds, as its bundle KEY (R-647, v0.265.0). The
|
||||||
|
// app_update_held event carries the key, not the Hungarian phrase: the hub prints the raw details as
|
||||||
|
// the mail's `Note:` line, and an English household read „a beállításokat, …" there.
|
||||||
|
func (m *Manager) UpdateCopyHoldsKey(stackName string, tier int) string {
|
||||||
|
return updateCopyHoldsKey(m.DataOutsideUnit(stackName), tier)
|
||||||
|
}
|
||||||
|
|
||||||
func updateCopyHoldsKey(outside bool, tier int) string {
|
func updateCopyHoldsKey(outside bool, tier int) string {
|
||||||
switch tier {
|
switch tier {
|
||||||
case UpdateTierLocal:
|
case UpdateTierLocal:
|
||||||
|
|||||||
@@ -571,6 +571,7 @@
|
|||||||
"badge.update.behind.today": " — today",
|
"badge.update.behind.today": " — today",
|
||||||
"badge.update.current": "Up to date",
|
"badge.update.current": "Up to date",
|
||||||
"badge.update.current.title": "This app is running the newest version available.",
|
"badge.update.current.title": "This app is running the newest version available.",
|
||||||
|
"badge.update.held": "Stopped — restore needed",
|
||||||
"catchall.alkalmazas_kezelese": "Manage app",
|
"catchall.alkalmazas_kezelese": "Manage app",
|
||||||
"catchall.alkalmazasok": "Apps",
|
"catchall.alkalmazasok": "Apps",
|
||||||
"catchall.vezerlopult": "Dashboard",
|
"catchall.vezerlopult": "Dashboard",
|
||||||
|
|||||||
@@ -566,6 +566,7 @@
|
|||||||
"badge.update.behind.today": " — ma",
|
"badge.update.behind.today": " — ma",
|
||||||
"badge.update.current": "Naprakész",
|
"badge.update.current": "Naprakész",
|
||||||
"badge.update.current.title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.",
|
"badge.update.current.title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.",
|
||||||
|
"badge.update.held": "Megállítva — visszaállítás szükséges",
|
||||||
"catchall.alkalmazas_kezelese": "Alkalmazás kezelése",
|
"catchall.alkalmazas_kezelese": "Alkalmazás kezelése",
|
||||||
"catchall.alkalmazasok": "Alkalmazások",
|
"catchall.alkalmazasok": "Alkalmazások",
|
||||||
"catchall.vezerlopult": "Vezérlőpult",
|
"catchall.vezerlopult": "Vezérlőpult",
|
||||||
|
|||||||
@@ -46,6 +46,11 @@ type Notifier struct {
|
|||||||
|
|
||||||
// oomSeen (R-514) remembers container runs already reported as OOM-killed.
|
// oomSeen (R-514) remembers container runs already reported as OOM-killed.
|
||||||
oomSeen map[string]bool
|
oomSeen map[string]bool
|
||||||
|
// oomTrack (R-636) remembers, per container run, the kill counter over the last 30 minutes and
|
||||||
|
// whether the ONE storm alarm of that run has gone.
|
||||||
|
oomTrack map[string]*oomRun
|
||||||
|
// nowFn is a test clock (nil → time.Now).
|
||||||
|
nowFn func() time.Time
|
||||||
// appDown tracks which deployed apps are currently in the DOWN state so app_start_failed fires
|
// appDown tracks which deployed apps are currently in the DOWN state so app_start_failed fires
|
||||||
// ONCE per running→down transition, not every health cycle (fix-3 anti-spam). In-memory: a
|
// ONCE per running→down transition, not every health cycle (fix-3 anti-spam). In-memory: a
|
||||||
// controller restart re-notifies once (acceptable — better than missing). The hub owns the real
|
// controller restart re-notifies once (acceptable — better than missing). The hub owns the real
|
||||||
@@ -359,7 +364,7 @@ func (n *Notifier) pushEventBoth(eventType, severity, message, messageCustomer s
|
|||||||
// Detects both degradation (ok→warn, ok→fail, warn→fail) and recovery (fail→ok, warn→ok, fail→warn).
|
// Detects both degradation (ok→warn, ok→fail, warn→fail) and recovery (fail→ok, warn→ok, fail→warn).
|
||||||
func (n *Notifier) NotifyHealthChange(status string, issues, warnings []string) {
|
func (n *Notifier) NotifyHealthChange(status string, issues, warnings []string) {
|
||||||
if !n.enabled {
|
if !n.enabled {
|
||||||
n.dropped("health_change", status)
|
n.dropped("health_change", healthSeverity(status)) // R-647: the severity, not the health status
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -714,21 +719,108 @@ func (n *Notifier) NotifyAppStartFailures(apps []AppRunState) {
|
|||||||
// limit (Docker State.OOMKilled). Fires ONCE per container run (keyed by StartedAt), so a container
|
// limit (Docker State.OOMKilled). Fires ONCE per container run (keyed by StartedAt), so a container
|
||||||
// that stays OOM-marked does not repeat. "warning" — the hub vocabulary (R-329). Operator-only
|
// that stays OOM-marked does not repeat. "warning" — the hub vocabulary (R-329). Operator-only
|
||||||
// hub-side (hub >= v0.114.0 registers it): the household sees the dashboard tag.
|
// hub-side (hub >= v0.114.0 registers it): the household sees the dashboard tag.
|
||||||
func (n *Notifier) NotifyAppOOM(stack, container, startedAt string) {
|
func (n *Notifier) NotifyAppOOM(stack, container, startedAt string, kills int64, memLimit, peak string) {
|
||||||
key := container + "|" + startedAt
|
key := container + "|" + startedAt
|
||||||
n.mu.Lock()
|
n.mu.Lock()
|
||||||
if n.oomSeen == nil {
|
if n.oomSeen == nil {
|
||||||
n.oomSeen = map[string]bool{}
|
n.oomSeen = map[string]bool{}
|
||||||
}
|
}
|
||||||
if n.oomSeen[key] {
|
first := !n.oomSeen[key]
|
||||||
n.mu.Unlock()
|
|
||||||
return
|
|
||||||
}
|
|
||||||
n.oomSeen[key] = true
|
n.oomSeen[key] = true
|
||||||
|
storm, inWindow := n.oomStormLocked(key, startedAt, kills)
|
||||||
n.mu.Unlock()
|
n.mu.Unlock()
|
||||||
|
if first {
|
||||||
n.emit("app_oom", "warning",
|
n.emit("app_oom", "warning",
|
||||||
fmt.Sprintf("Alkalmazás memóriája elfogyott: %s (%s) — egy folyamatát a memóriakorlát leállította", stack, container),
|
fmt.Sprintf("Alkalmazás memóriája elfogyott: %s (%s) — egy folyamatát a memóriakorlát leállította", stack, container),
|
||||||
AppDetails{StackName: stack, DisplayName: container})
|
AppDetails{StackName: stack, DisplayName: container})
|
||||||
|
}
|
||||||
|
if storm {
|
||||||
|
n.logger.Printf("[ERROR] [notify] %s: container %s OOM STORM — %d kills in %d min (limit %s, peak %s)",
|
||||||
|
stack, container, inWindow, oomStormWindowMin, memLimit, peak)
|
||||||
|
n.emit("app_oom_storm", "error",
|
||||||
|
fmt.Sprintf("Alkalmazás memóriája ismételten elfogy: %s (%s) — %d leállítás %d percen belül (korlát %s, csúcs %s)",
|
||||||
|
stack, container, inWindow, oomStormWindowMin, memLimit, peak),
|
||||||
|
OOMStormDetails{App: stack, StackName: stack, Container: container, Kills: inWindow,
|
||||||
|
WindowMin: oomStormWindowMin, MemLimit: memLimit, Peak: peak})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// R-636 (v0.265.0) — ONE louder alarm per container run when OOM kills keep coming.
|
||||||
|
//
|
||||||
|
// The once-per-run app_oom stays exactly as it was (it is what stops a crash loop from mailing 4,530
|
||||||
|
// times, R-629). Beside it: when the kernel's own kill counter for the SAME run rises by
|
||||||
|
// oomStormKills or more within oomStormWindowMin minutes, app_oom_storm goes once, at `error`,
|
||||||
|
// operator-only. RomM's real rate on 2026-09-22 was 4,530 kills in 6 h ≈ 375 per 30 min; one hiccup
|
||||||
|
// is 1–3. Twenty in thirty minutes sits far from both. Pinned by TestR636_*.
|
||||||
|
const (
|
||||||
|
oomStormKills = 20
|
||||||
|
oomStormWindowMin = 30
|
||||||
|
)
|
||||||
|
|
||||||
|
type oomSample struct {
|
||||||
|
at time.Time
|
||||||
|
kills int64
|
||||||
|
}
|
||||||
|
|
||||||
|
type oomRun struct {
|
||||||
|
samples []oomSample
|
||||||
|
stormSent bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// OOMStormDetails is the app_oom_storm payload. stack_name lets the hub's per-app cooldown tell two
|
||||||
|
// storming apps apart.
|
||||||
|
type OOMStormDetails struct {
|
||||||
|
App string `json:"app"`
|
||||||
|
StackName string `json:"stack_name"`
|
||||||
|
Container string `json:"container"`
|
||||||
|
Kills int64 `json:"kills"`
|
||||||
|
WindowMin int `json:"window_min"`
|
||||||
|
MemLimit string `json:"mem_limit,omitempty"`
|
||||||
|
Peak string `json:"peak,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (n *Notifier) now() time.Time {
|
||||||
|
if n.nowFn != nil {
|
||||||
|
return n.nowFn()
|
||||||
|
}
|
||||||
|
return time.Now()
|
||||||
|
}
|
||||||
|
|
||||||
|
// oomStormLocked records one reading and says whether the storm alarm fires now. Caller holds n.mu.
|
||||||
|
// A run seen for the first time within the window of its own start gets a zero reading AT its start:
|
||||||
|
// the counter began at zero there, so a run that is already storming when the controller first looks
|
||||||
|
// is not made to wait another thirty minutes.
|
||||||
|
func (n *Notifier) oomStormLocked(key, startedAt string, kills int64) (bool, int64) {
|
||||||
|
if kills < 0 {
|
||||||
|
return false, 0 // unreadable: never escalate on a guess
|
||||||
|
}
|
||||||
|
if n.oomTrack == nil {
|
||||||
|
n.oomTrack = map[string]*oomRun{}
|
||||||
|
}
|
||||||
|
now := n.now()
|
||||||
|
window := time.Duration(oomStormWindowMin) * time.Minute
|
||||||
|
r := n.oomTrack[key]
|
||||||
|
if r == nil {
|
||||||
|
r = &oomRun{}
|
||||||
|
if t, err := time.Parse(time.RFC3339Nano, startedAt); err == nil && now.Sub(t) <= window {
|
||||||
|
r.samples = append(r.samples, oomSample{at: t, kills: 0})
|
||||||
|
}
|
||||||
|
n.oomTrack[key] = r
|
||||||
|
}
|
||||||
|
r.samples = append(r.samples, oomSample{at: now, kills: kills})
|
||||||
|
keep := r.samples[:0]
|
||||||
|
for _, s := range r.samples {
|
||||||
|
if now.Sub(s.at) <= window {
|
||||||
|
keep = append(keep, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
r.samples = keep
|
||||||
|
inWindow := kills - r.samples[0].kills
|
||||||
|
if r.stormSent || inWindow < oomStormKills {
|
||||||
|
return false, inWindow
|
||||||
|
}
|
||||||
|
r.stormSent = true
|
||||||
|
return true, inWindow
|
||||||
}
|
}
|
||||||
|
|
||||||
// DiskHealthDetails is the event-detail payload for disk_health_degraded.
|
// DiskHealthDetails is the event-detail payload for disk_health_degraded.
|
||||||
@@ -1211,3 +1303,15 @@ func (n *Notifier) NotifyAppUpdateHeld(d AppUpdateDetails, sentence func(lang st
|
|||||||
}
|
}
|
||||||
n.pushEventBoth("app_update_held", "error", hu, household, d)
|
n.pushEventBoth("app_update_held", "error", hu, household, d)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// healthSeverity is the event severity a health status would be sent at (R-647): the disabled path
|
||||||
|
// names what it drops, and "warn" is a health STATUS, not a severity the hub knows.
|
||||||
|
func healthSeverity(status string) string {
|
||||||
|
switch status {
|
||||||
|
case "fail":
|
||||||
|
return "error"
|
||||||
|
case "warn":
|
||||||
|
return "warning"
|
||||||
|
}
|
||||||
|
return "info"
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
package notify
|
||||||
|
|
||||||
|
import (
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-636 — a repeating OOM problem gets ONE louder alarm; one hiccup stays one quiet warning.
|
||||||
|
func stormRecorder(t *testing.T) (*Notifier, *time.Time, map[string]int) {
|
||||||
|
t.Helper()
|
||||||
|
n := New("", "", "c1", nil, log.New(io.Discard, "", 0), false)
|
||||||
|
got := map[string]int{}
|
||||||
|
n.pushFn = func(eventType, severity, msg, msgCustomer string, details interface{}) {
|
||||||
|
got[eventType+"/"+severity]++
|
||||||
|
}
|
||||||
|
clock := time.Date(2026, 9, 22, 9, 0, 0, 0, time.UTC)
|
||||||
|
n.nowFn = func() time.Time { return clock }
|
||||||
|
return n, &clock, got
|
||||||
|
}
|
||||||
|
|
||||||
|
// COMPANION RED-PROOF (REPORT.md): make oomStormLocked always return false — the 20-kill case sends no
|
||||||
|
// storm and this fails.
|
||||||
|
func TestR636_TwentyKillsInThirtyMinutesIsOneStorm(t *testing.T) {
|
||||||
|
for _, c := range []struct {
|
||||||
|
kills int64
|
||||||
|
storm int
|
||||||
|
}{{19, 0}, {20, 1}, {200, 1}} {
|
||||||
|
n, clock, got := stormRecorder(t)
|
||||||
|
start := clock.Add(-time.Hour).Format(time.RFC3339Nano) // started long before: no zero baseline
|
||||||
|
for i := int64(0); i <= 40; i++ { // 20 minutes of 30 s scans, counter 5 → 5+kills
|
||||||
|
n.NotifyAppOOM("romm", "romm", start, 5+c.kills*i/40, "1280M", "1279M")
|
||||||
|
*clock = clock.Add(30 * time.Second)
|
||||||
|
}
|
||||||
|
if got["app_oom_storm/error"] != c.storm {
|
||||||
|
t.Errorf("kills=%d in 20 min: storms=%d, want %d", c.kills, got["app_oom_storm/error"], c.storm)
|
||||||
|
}
|
||||||
|
if c.kills == 200 { // RomM's shape: it keeps going for six hours — still ONE storm for the run
|
||||||
|
for i := int64(1); i <= 720; i++ {
|
||||||
|
n.NotifyAppOOM("romm", "romm", start, 205+i*12, "1280M", "1279M")
|
||||||
|
*clock = clock.Add(30 * time.Second)
|
||||||
|
}
|
||||||
|
if got["app_oom_storm/error"] != 1 {
|
||||||
|
t.Errorf("six hours of storm must stay ONE storm per container run, got %d", got["app_oom_storm/error"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if got["app_oom/warning"] != 1 {
|
||||||
|
t.Errorf("kills=%d: the quiet app_oom must still go exactly once per run, got %d", c.kills, got["app_oom/warning"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// One kill, and then the flag just stays set for hours (the sticky OOMKilled): never a storm.
|
||||||
|
func TestR636_OneHiccupNeverStorms(t *testing.T) {
|
||||||
|
n, clock, got := stormRecorder(t)
|
||||||
|
start := clock.Format(time.RFC3339Nano)
|
||||||
|
for i := 0; i < 6*120; i++ { // six hours of 30 s scans, counter stuck at 1
|
||||||
|
n.NotifyAppOOM("paperless-ngx", "paperless-webserver", start, 1, "768M", "700M")
|
||||||
|
*clock = clock.Add(30 * time.Second)
|
||||||
|
}
|
||||||
|
if got["app_oom_storm/error"] != 0 || got["app_oom/warning"] != 1 {
|
||||||
|
t.Fatalf("one hiccup: want 1 warning and 0 storms, got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Slow and steady — 20 kills spread over three hours is not a storm.
|
||||||
|
func TestR636_SlowKillsAreNotAStorm(t *testing.T) {
|
||||||
|
n, clock, got := stormRecorder(t)
|
||||||
|
start := clock.Add(-time.Hour).Format(time.RFC3339Nano)
|
||||||
|
for i := int64(0); i <= 360; i++ { // 3 h, counter +20 in total
|
||||||
|
n.NotifyAppOOM("x", "x", start, i*20/360, "", "")
|
||||||
|
*clock = clock.Add(30 * time.Second)
|
||||||
|
}
|
||||||
|
if got["app_oom_storm/error"] != 0 {
|
||||||
|
t.Fatalf("20 kills over 3 h must not storm, got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A run first seen already storming (the controller restarted mid-storm) fires at once: its counter
|
||||||
|
// started at zero at its own start, inside the window. An unreadable counter never escalates.
|
||||||
|
func TestR636_AlreadyStormingAtFirstSightAndUnreadable(t *testing.T) {
|
||||||
|
n, clock, got := stormRecorder(t)
|
||||||
|
n.NotifyAppOOM("romm", "romm", clock.Add(-10*time.Minute).Format(time.RFC3339Nano), 150, "1280M", "1279M")
|
||||||
|
if got["app_oom_storm/error"] != 1 {
|
||||||
|
t.Fatalf("150 kills 10 min after the start must storm at first sight, got %v", got)
|
||||||
|
}
|
||||||
|
n2, _, got2 := stormRecorder(t)
|
||||||
|
for i := 0; i < 100; i++ {
|
||||||
|
n2.NotifyAppOOM("romm", "romm", "", -1, "", "")
|
||||||
|
}
|
||||||
|
if got2["app_oom_storm/error"] != 0 {
|
||||||
|
t.Fatalf("an unreadable counter must never escalate, got %v", got2)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -102,3 +102,15 @@ func TestR620_DisabledNotifierSaysWhatItDrops(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// R-647 (3) — the disabled path names the SEVERITY a health change would be sent at, never the health
|
||||||
|
// status. COMPANION RED-PROOF (REPORT.md): pass `status` again — the line reads "severity warn".
|
||||||
|
func TestR647_DisabledHealthChangeNamesTheSeverity(t *testing.T) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
n := New("", "", "c1", nil, log.New(&buf, "", 0), false)
|
||||||
|
buf.Reset()
|
||||||
|
n.NotifyHealthChange("warn", nil, nil)
|
||||||
|
if !strings.Contains(buf.String(), "DROPPED event health_change (severity warning)") {
|
||||||
|
t.Fatalf("want the severity 'warning' named, got:\n%s", buf.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -22,6 +22,11 @@ var (
|
|||||||
ErrPathMissing = errors.New("path field does not exist")
|
ErrPathMissing = errors.New("path field does not exist")
|
||||||
// ErrNotEnoughMemory — the memory verdict refused the deploy (API: 400).
|
// ErrNotEnoughMemory — the memory verdict refused the deploy (API: 400).
|
||||||
ErrNotEnoughMemory = errors.New("not enough memory")
|
ErrNotEnoughMemory = errors.New("not enough memory")
|
||||||
|
// ErrStackDeploying — R-634 (v0.265.0): a stop or start was asked of a stack whose deploy is still
|
||||||
|
// running. Refused, because a `compose down` or a second `compose up -d` in the middle of the
|
||||||
|
// deploy's own `up` makes BOTH fail, and the deploy then records „not deployed" over whatever
|
||||||
|
// containers the race left behind. Pinned by TestR634_StopAndStartRefuseADeployingStack.
|
||||||
|
ErrStackDeploying = errors.New("stack is being deployed")
|
||||||
)
|
)
|
||||||
|
|
||||||
// msgHU renders a bundle message in Hungarian.
|
// msgHU renders a bundle message in Hungarian.
|
||||||
|
|||||||
@@ -1147,6 +1147,12 @@ func (m *Manager) StartStack(name string) error {
|
|||||||
if !ok {
|
if !ok {
|
||||||
return fmt.Errorf("stack %q not found", name)
|
return fmt.Errorf("stack %q not found", name)
|
||||||
}
|
}
|
||||||
|
if stack.Deploying {
|
||||||
|
// R-634: a second `compose up -d` beside the deploy's own is the race that left apps running
|
||||||
|
// under „not deployed". The deploy brings the app up itself; nothing else may.
|
||||||
|
m.logger.Printf("[WARN] [stacks] StartStack %s refused: the app's deploy is still running (R-634)", name)
|
||||||
|
return fmt.Errorf("starting stack %s: %w", name, ErrStackDeploying)
|
||||||
|
}
|
||||||
|
|
||||||
if m.isDebug() {
|
if m.isDebug() {
|
||||||
m.logger.Printf("[DEBUG] [stacks] StartStack %s: current state=%s deployed=%v", name, stack.State, stack.Deployed)
|
m.logger.Printf("[DEBUG] [stacks] StartStack %s: current state=%s deployed=%v", name, stack.State, stack.Deployed)
|
||||||
@@ -1229,6 +1235,13 @@ func (m *Manager) StopStack(name string) error {
|
|||||||
if !ok {
|
if !ok {
|
||||||
return fmt.Errorf("stack %q not found", name)
|
return fmt.Errorf("stack %q not found", name)
|
||||||
}
|
}
|
||||||
|
if stack.Deploying {
|
||||||
|
// R-634, the backstop for EVERY caller (backup, quiesce, restore, export, storage, the Stop
|
||||||
|
// button): `compose down` in the middle of a deploy was measured to make the deploy fail and
|
||||||
|
// record „not deployed" while the caller's own restart brought the containers back.
|
||||||
|
m.logger.Printf("[WARN] [stacks] StopStack %s refused: the app's deploy is still running (R-634)", name)
|
||||||
|
return fmt.Errorf("stopping stack %s: %w", name, ErrStackDeploying)
|
||||||
|
}
|
||||||
|
|
||||||
if m.isDebug() {
|
if m.isDebug() {
|
||||||
m.logger.Printf("[DEBUG] [stacks] StopStack %s: current state=%s deployed=%v containers=%d", name, stack.State, stack.Deployed, len(stack.Containers))
|
m.logger.Printf("[DEBUG] [stacks] StopStack %s: current state=%s deployed=%v containers=%d", name, stack.State, stack.Deployed, len(stack.Containers))
|
||||||
|
|||||||
@@ -1,7 +1,9 @@
|
|||||||
package stacks
|
package stacks
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
"sort"
|
"sort"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -18,6 +20,13 @@ type OOMContainer struct {
|
|||||||
Stack string
|
Stack string
|
||||||
Container string
|
Container string
|
||||||
StartedAt string // identifies the container run — one event per run
|
StartedAt string // identifies the container run — one event per run
|
||||||
|
// Kills is the kernel's own count of OOM kills in this container's cgroup (`memory.events`
|
||||||
|
// oom_kill) — R-636, v0.265.0. OOMKilled is a STICKY flag: it stays true for the container's whole
|
||||||
|
// life after ONE kill, so "the key re-fired" means nothing; this counter is what separates one
|
||||||
|
// hiccup from RomM's 4,530 kills in six hours. -1 when it could not be read.
|
||||||
|
Kills int64
|
||||||
|
// MemLimit / Peak are the cgroup's memory.max and memory.peak, as "<n>M" (or "max"); "" unread.
|
||||||
|
MemLimit, Peak string
|
||||||
}
|
}
|
||||||
|
|
||||||
// ScanOOMKilled inspects the containers of every deployed, running-ish stack in ONE docker call and
|
// ScanOOMKilled inspects the containers of every deployed, running-ish stack in ONE docker call and
|
||||||
@@ -56,13 +65,53 @@ func (m *Manager) ScanOOMKilled() ([]OOMContainer, error) {
|
|||||||
}
|
}
|
||||||
cname := strings.TrimPrefix(f[0], "/")
|
cname := strings.TrimPrefix(f[0], "/")
|
||||||
if st, ok := owner[cname]; ok {
|
if st, ok := owner[cname]; ok {
|
||||||
found = append(found, OOMContainer{Stack: st, Container: cname, StartedAt: f[2]})
|
found = append(found, OOMContainer{Stack: st, Container: cname, StartedAt: f[2], Kills: -1})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// R-636: only for the containers already flagged — one `docker exec` each, and the flagged set is
|
||||||
|
// empty on a healthy box. Read from INSIDE the container (its cgroup namespace makes
|
||||||
|
// /sys/fs/cgroup its own cgroup); the controller's own namespace cannot see the others.
|
||||||
|
for i := range found {
|
||||||
|
out, _ := m.execCommand("docker", "exec", found[i].Container, "cat",
|
||||||
|
"/sys/fs/cgroup/memory.events", "/sys/fs/cgroup/memory.max", "/sys/fs/cgroup/memory.peak")
|
||||||
|
found[i].Kills, found[i].MemLimit, found[i].Peak = parseCgroupMemory(out)
|
||||||
|
}
|
||||||
m.setOOMCache(found)
|
m.setOOMCache(found)
|
||||||
return found, nil
|
return found, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// parseCgroupMemory reads `cat memory.events memory.max memory.peak`: the key/value lines give
|
||||||
|
// oom_kill; the first bare line is memory.max, the second memory.peak (absent on older kernels — cat
|
||||||
|
// then fails on that file and still prints the others). Unreadable → Kills -1.
|
||||||
|
func parseCgroupMemory(out string) (kills int64, limit, peak string) {
|
||||||
|
kills = -1
|
||||||
|
var bare []string
|
||||||
|
for _, line := range strings.Split(strings.TrimSpace(out), "\n") {
|
||||||
|
f := strings.Fields(line)
|
||||||
|
switch {
|
||||||
|
case len(f) == 2 && f[0] == "oom_kill":
|
||||||
|
if n, err := strconv.ParseInt(f[1], 10, 64); err == nil {
|
||||||
|
kills = n
|
||||||
|
}
|
||||||
|
case len(f) == 1:
|
||||||
|
bare = append(bare, f[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mb := func(v string) string {
|
||||||
|
if n, err := strconv.ParseInt(v, 10, 64); err == nil {
|
||||||
|
return fmt.Sprintf("%dM", n/(1024*1024))
|
||||||
|
}
|
||||||
|
return v // "max"
|
||||||
|
}
|
||||||
|
if len(bare) > 0 {
|
||||||
|
limit = mb(bare[0])
|
||||||
|
}
|
||||||
|
if len(bare) > 1 {
|
||||||
|
peak = mb(bare[1])
|
||||||
|
}
|
||||||
|
return kills, limit, peak
|
||||||
|
}
|
||||||
|
|
||||||
func (m *Manager) setOOMCache(found []OOMContainer) {
|
func (m *Manager) setOOMCache(found []OOMContainer) {
|
||||||
cache := map[string][]string{}
|
cache := map[string][]string{}
|
||||||
for _, o := range found {
|
for _, o := range found {
|
||||||
|
|||||||
@@ -53,3 +53,46 @@ func TestScanOOMKilled_SeesKilledWorkerInRunningContainer(t *testing.T) {
|
|||||||
t.Fatalf("cache not cleared after a clean scan: %v", got)
|
t.Fatalf("cache not cleared after a clean scan: %v", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// R-636 — the kernel's kill counter, the limit and the peak are read from `cat memory.events
|
||||||
|
// memory.max memory.peak` inside the container; an unreadable one is -1, never 0.
|
||||||
|
func TestR636_ParseCgroupMemory(t *testing.T) {
|
||||||
|
out := "low 0\nhigh 0\nmax 4521\noom 4530\noom_kill 4530\noom_group_kill 0\n1342177280\n1341128704\n"
|
||||||
|
k, lim, peak := parseCgroupMemory(out)
|
||||||
|
if k != 4530 || lim != "1280M" || peak != "1279M" {
|
||||||
|
t.Fatalf("got kills=%d limit=%q peak=%q", k, lim, peak)
|
||||||
|
}
|
||||||
|
if k, lim, _ := parseCgroupMemory("oom_kill 3\nmax\n"); k != 3 || lim != "max" {
|
||||||
|
t.Fatalf("no memory.peak (older kernel), unlimited: got %d %q", k, lim)
|
||||||
|
}
|
||||||
|
if k, _, _ := parseCgroupMemory("OCI runtime exec failed: exec: \"cat\": executable file not found"); k != -1 {
|
||||||
|
t.Fatalf("an image without cat must read -1 (unknown), got %d", k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// R-636 — the scan asks the kill counter of the FLAGGED containers only (one exec each; none on a
|
||||||
|
// healthy box). COMPANION RED-PROOF (REPORT.md): drop the exec loop — Kills stays -1 and this fails.
|
||||||
|
func TestR636_ScanReadsTheCounterOfFlaggedContainersOnly(t *testing.T) {
|
||||||
|
var execd []string
|
||||||
|
m := &Manager{logger: log.New(io.Discard, "", 0), stacks: map[string]*Stack{
|
||||||
|
"romm": {Name: "romm", Deployed: true, Containers: []ContainerInfo{{Name: "romm", State: StateRunning}}},
|
||||||
|
"bookstack": {Name: "bookstack", Deployed: true, Containers: []ContainerInfo{{Name: "bookstack", State: StateRunning}}},
|
||||||
|
}}
|
||||||
|
m.execFn = func(name string, args ...string) (string, error) {
|
||||||
|
if args[0] == "exec" {
|
||||||
|
execd = append(execd, args[1])
|
||||||
|
return "oom_kill 377\n1342177280\n1341128704\n", nil
|
||||||
|
}
|
||||||
|
return "/bookstack|false|t0\n/romm|true|2026-09-22T09:08:00Z\n", nil
|
||||||
|
}
|
||||||
|
ooms, err := m.ScanOOMKilled()
|
||||||
|
if err != nil || len(ooms) != 1 {
|
||||||
|
t.Fatalf("ooms=%+v err=%v", ooms, err)
|
||||||
|
}
|
||||||
|
if ooms[0].Kills != 377 || ooms[0].MemLimit != "1280M" || ooms[0].Peak != "1279M" {
|
||||||
|
t.Errorf("the counter must be read for the flagged container: %+v", ooms[0])
|
||||||
|
}
|
||||||
|
if len(execd) != 1 || execd[0] != "romm" {
|
||||||
|
t.Errorf("exec must reach the flagged container only, got %v", execd)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,36 @@
|
|||||||
|
package stacks
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// R-634 (v0.265.0) — the backstop for every caller: a stop or a start while the deploy's own
|
||||||
|
// `compose up` is running is refused BEFORE any compose command.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (REPORT.md): delete the Deploying check in StopStack — the stop falls through to
|
||||||
|
// `compose down` and the error is no longer ErrStackDeploying.
|
||||||
|
func TestR634_StopAndStartRefuseADeployingStack(t *testing.T) {
|
||||||
|
m, _, _, _ := newSlice4Manager(t)
|
||||||
|
// NEVER a real compose: the control below falls through to `compose down`, and the stack dir is
|
||||||
|
// named like a real app. An empty PATH + the v1 binary name guarantees the exec cannot find
|
||||||
|
// anything to run, on any machine this test runs on.
|
||||||
|
m.composeCmd = "docker-compose"
|
||||||
|
t.Setenv("PATH", t.TempDir())
|
||||||
|
m.mu.Lock()
|
||||||
|
m.stacks["nextcloud"].Deploying = true
|
||||||
|
m.mu.Unlock()
|
||||||
|
if err := m.StopStack("nextcloud"); !errors.Is(err, ErrStackDeploying) {
|
||||||
|
t.Fatalf("StopStack on a deploying stack = %v, want ErrStackDeploying", err)
|
||||||
|
}
|
||||||
|
if err := m.StartStack("nextcloud"); !errors.Is(err, ErrStackDeploying) {
|
||||||
|
t.Fatalf("StartStack on a deploying stack = %v, want ErrStackDeploying", err)
|
||||||
|
}
|
||||||
|
// Control: once the deploy is over, the same calls are not refused for THIS reason.
|
||||||
|
m.mu.Lock()
|
||||||
|
m.stacks["nextcloud"].Deploying = false
|
||||||
|
m.mu.Unlock()
|
||||||
|
if err := m.StopStack("nextcloud"); errors.Is(err, ErrStackDeploying) {
|
||||||
|
t.Fatal("control: a stack that is not deploying must not be refused as deploying")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -471,7 +471,11 @@ func (m *Manager) emitUpdateEvent(kind, name string, entry *updateJournalEntry,
|
|||||||
if entry != nil {
|
if entry != nil {
|
||||||
ev.From, ev.To = entry.PrevPin, entry.NewPin
|
ev.From, ev.To = entry.PrevPin, entry.NewPin
|
||||||
}
|
}
|
||||||
|
if kind == UpdateEventHeld {
|
||||||
m.logger.Printf("[INFO] [stacks] update %s: event %s (hold recorded: %v)", name, kind, holdRecorded)
|
m.logger.Printf("[INFO] [stacks] update %s: event %s (hold recorded: %v)", name, kind, holdRecorded)
|
||||||
|
} else { // R-647: "hold recorded" means nothing for an undone update
|
||||||
|
m.logger.Printf("[INFO] [stacks] update %s: event %s", name, kind)
|
||||||
|
}
|
||||||
sink(ev)
|
sink(ev)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -564,7 +564,7 @@ func (m *Manager) finishUpdate(name, phase, msg string) {
|
|||||||
// beside it for the page. key "" = `plain` is a finished sentence and is stored as it is.
|
// beside it for the page. key "" = `plain` is a finished sentence and is stored as it is.
|
||||||
func (m *Manager) finishUpdateKey(name, phase, key, plain string, args ...interface{}) {
|
func (m *Manager) finishUpdateKey(name, phase, key, plain string, args ...interface{}) {
|
||||||
msg := plain
|
msg := plain
|
||||||
if key != "" {
|
if key != "" && key != UpdateErrorKeyHeld { // the held key names no bundle entry: plain IS the sentence
|
||||||
msg = util.Text(i18n.Default, key, args...)
|
msg = util.Text(i18n.Default, key, args...)
|
||||||
}
|
}
|
||||||
m.mu.Lock()
|
m.mu.Lock()
|
||||||
@@ -590,12 +590,57 @@ func UpdatePhaseLabelIn(lang, phase string) string {
|
|||||||
|
|
||||||
// UpdateErrorIn is the stack's update sentence in lang (v0.264.0, R-606).
|
// UpdateErrorIn is the stack's update sentence in lang (v0.264.0, R-606).
|
||||||
func (s Stack) UpdateErrorIn(lang string) string {
|
func (s Stack) UpdateErrorIn(lang string) string {
|
||||||
if s.UpdateErrorKey == "" || lang == i18n.Default {
|
if s.UpdateErrorKey == "" || s.UpdateErrorKey == UpdateErrorKeyHeld || lang == i18n.Default {
|
||||||
return s.UpdateError
|
return s.UpdateError
|
||||||
}
|
}
|
||||||
return util.Text(lang, s.UpdateErrorKey, s.UpdateErrorArgs...)
|
return util.Text(lang, s.UpdateErrorKey, s.UpdateErrorArgs...)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// UpdateErrorKeyHeld marks an UpdateError that IS the hold sentence (R-647, v0.265.0). It names no
|
||||||
|
// bundle entry: the sentence is composed by the backup side, so UpdateErrorFor asks it for the
|
||||||
|
// reader's language, and UpdateErrorIn (no manager to ask) returns the stored Hungarian.
|
||||||
|
const UpdateErrorKeyHeld = "update.error.held"
|
||||||
|
|
||||||
|
// holdLanguage is the OPTIONAL half of UpdateGuards that renders the hold sentence in a given
|
||||||
|
// language (the production adapter implements it; the test fakes need not).
|
||||||
|
type holdLanguage interface {
|
||||||
|
HoldForLang(name, lang string) (bool, string)
|
||||||
|
}
|
||||||
|
|
||||||
|
func holdForLang(g UpdateGuards, name, lang string) (bool, string) {
|
||||||
|
if g == nil {
|
||||||
|
return false, ""
|
||||||
|
}
|
||||||
|
if hl, ok := g.(holdLanguage); ok {
|
||||||
|
return hl.HoldForLang(name, lang)
|
||||||
|
}
|
||||||
|
return g.HoldFor(name)
|
||||||
|
}
|
||||||
|
|
||||||
|
// HoldReasonFor is a stack's hold sentence in the READER's language (R-647): "" when nothing holds it.
|
||||||
|
// A Hungarian reader on a Hungarian box gets exactly Stack.HoldReason.
|
||||||
|
func (m *Manager) HoldReasonFor(st Stack, lang string) string {
|
||||||
|
if st.HoldReason == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
if held, why := holdForLang(m.guards(), st.Name, lang); held && why != "" {
|
||||||
|
return why
|
||||||
|
}
|
||||||
|
return st.HoldReason
|
||||||
|
}
|
||||||
|
|
||||||
|
// UpdateErrorFor is a stack's update error in the READER's language (R-606 + R-647). A held update's
|
||||||
|
// error is the hold sentence, rendered by the backup side for this reader.
|
||||||
|
func (m *Manager) UpdateErrorFor(st Stack, lang string) string {
|
||||||
|
if st.UpdateErrorKey == UpdateErrorKeyHeld {
|
||||||
|
if held, why := holdForLang(m.guards(), st.Name, lang); held && why != "" {
|
||||||
|
return why
|
||||||
|
}
|
||||||
|
return st.UpdateError
|
||||||
|
}
|
||||||
|
return st.UpdateErrorIn(lang)
|
||||||
|
}
|
||||||
|
|
||||||
func (m *Manager) updateCompose(dir string, env []string, args ...string) (string, error) {
|
func (m *Manager) updateCompose(dir string, env []string, args ...string) (string, error) {
|
||||||
if m.updateComposeFn != nil {
|
if m.updateComposeFn != nil {
|
||||||
return m.updateComposeFn(dir, env, args...)
|
return m.updateComposeFn(dir, env, args...)
|
||||||
@@ -880,7 +925,7 @@ func (m *Manager) failAndHold(ctx context.Context, name, dir string, env []strin
|
|||||||
m.logger.Printf("[ERROR] [stacks] update %s: no UpdateGuards — the hold CANNOT be recorded", name)
|
m.logger.Printf("[ERROR] [stacks] update %s: no UpdateGuards — the hold CANNOT be recorded", name)
|
||||||
} else if err := g.HoldAfterFailedUpdate(name, m.now(), rp, undoState); err != nil {
|
} else if err := g.HoldAfterFailedUpdate(name, m.now(), rp, undoState); err != nil {
|
||||||
m.logger.Printf("[ERROR] [stacks] update %s: %v", name, err)
|
m.logger.Printf("[ERROR] [stacks] update %s: %v", name, err)
|
||||||
} else if _, w := g.HoldFor(name); w != "" {
|
} else if _, w := holdForLang(g, name, i18n.Default); w != "" {
|
||||||
holdWhy = w
|
holdWhy = w
|
||||||
m.markUpdateHeld(name)
|
m.markUpdateHeld(name)
|
||||||
}
|
}
|
||||||
@@ -890,8 +935,10 @@ func (m *Manager) failAndHold(ctx context.Context, name, dir string, env []strin
|
|||||||
if holdWhy == "" {
|
if holdWhy == "" {
|
||||||
m.finishUpdateKey(name, UpdatePhaseFailed, "update.error.hold_unsaved", "")
|
m.finishUpdateKey(name, UpdatePhaseFailed, "update.error.hold_unsaved", "")
|
||||||
} else {
|
} else {
|
||||||
// The hold's own sentence (the page renders it per reader through RestoreHoldForLang).
|
// R-647 (v0.265.0): stored as the HELD key + the Hungarian sentence, so every reader — the
|
||||||
m.finishUpdate(name, UpdatePhaseFailed, holdWhy)
|
// `?lang=` switch, an operator reading a box in the other language — gets the hold sentence in
|
||||||
|
// THEIR language (UpdateErrorFor), and the stored text is Hungarian as for every other key.
|
||||||
|
m.finishUpdateKey(name, UpdatePhaseFailed, UpdateErrorKeyHeld, holdWhy)
|
||||||
}
|
}
|
||||||
m.emitUpdateEvent(UpdateEventHeld, name, entry, rp, holdWhy != "")
|
m.emitUpdateEvent(UpdateEventHeld, name, entry, rp, holdWhy != "")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -322,17 +322,6 @@ func (s *Server) localeFuncs(lang string) template.FuncMap {
|
|||||||
}
|
}
|
||||||
return stk.UpdatePhaseLabel // no translation for this phase: the stored label, never an empty line
|
return stk.UpdatePhaseLabel // no translation for this phase: the stored label, never an empty line
|
||||||
},
|
},
|
||||||
"updateErrorText": func(st interface{}) string { return stackOf(st).UpdateErrorIn(lang) },
|
|
||||||
"holdText": func(st interface{}) string {
|
|
||||||
stk := stackOf(st)
|
|
||||||
if stk.HoldReason == "" || s.backupMgr == nil {
|
|
||||||
return stk.HoldReason
|
|
||||||
}
|
|
||||||
if held, why := s.backupMgr.RestoreHoldForLang(stk.Name, lang); held && why != "" {
|
|
||||||
return why
|
|
||||||
}
|
|
||||||
return stk.HoldReason
|
|
||||||
},
|
|
||||||
"timeAgo": func(t time.Time) string {
|
"timeAgo": func(t time.Time) string {
|
||||||
if t.IsZero() {
|
if t.IsZero() {
|
||||||
return "–"
|
return "–"
|
||||||
@@ -387,6 +376,13 @@ func (s *Server) localeFuncs(lang string) template.FuncMap {
|
|||||||
// Found live on 2026-09-20: on an English app page the only Felhom-authored Hungarian left
|
// Found live on 2026-09-20: on an English app page the only Felhom-authored Hungarian left
|
||||||
// was this badge, „Naprakész", with a Hungarian tooltip under it.
|
// was this badge, „Naprakész", with a Hungarian tooltip under it.
|
||||||
"updateBadge": func(st stacks.Stack) *MetaBadge {
|
"updateBadge": func(st stacks.Stack) *MetaBadge {
|
||||||
|
if st.HoldReason != "" { // R-625's twin — see updatebadge.go
|
||||||
|
return &MetaBadge{
|
||||||
|
Label: b.Msg(lang, "badge.update.held"),
|
||||||
|
Class: "tag-error",
|
||||||
|
Title: firstSentence(st.HoldReason),
|
||||||
|
}
|
||||||
|
}
|
||||||
switch compareInstalledToTemplate(st) {
|
switch compareInstalledToTemplate(st) {
|
||||||
case updateCurrent:
|
case updateCurrent:
|
||||||
return &MetaBadge{
|
return &MetaBadge{
|
||||||
@@ -554,3 +550,43 @@ func (s *Server) note(key string, args ...interface{}) string {
|
|||||||
// noteErr renders an error into a saved note in the box's language: its bundle message when it
|
// noteErr renders an error into a saved note in the box's language: its bundle message when it
|
||||||
// carries one (release B), its own text otherwise.
|
// carries one (release B), its own text otherwise.
|
||||||
func (s *Server) noteErr(err error) string { return util.ErrText(s.boxLang(), err) }
|
func (s *Server) noteErr(err error) string { return util.ErrText(s.boxLang(), err) }
|
||||||
|
|
||||||
|
// readerFuncs are the funcs that need the SERVER to answer in the reader's language, and — unlike
|
||||||
|
// localeFuncs — they apply to EVERY language, Hungarian included (R-647, v0.265.0). A hold sentence is
|
||||||
|
// composed by the backup side in one language at a time; before this a Hungarian reader of an English
|
||||||
|
// box (the `?lang=` switch, an operator) read it in English. On a Hungarian box a Hungarian reader gets
|
||||||
|
// exactly the bytes templateFuncMap gave (the parity fixtures are rendered through here).
|
||||||
|
// `prev` is the func set they wrap (templateFuncMap + localeFuncs for this language).
|
||||||
|
func (s *Server) readerFuncs(lang string, prev template.FuncMap) template.FuncMap {
|
||||||
|
holdFor := func(st stacks.Stack) string {
|
||||||
|
if st.HoldReason == "" || s.backupMgr == nil {
|
||||||
|
return st.HoldReason
|
||||||
|
}
|
||||||
|
if held, why := s.backupMgr.RestoreHoldForLang(st.Name, lang); held && why != "" {
|
||||||
|
return why
|
||||||
|
}
|
||||||
|
return st.HoldReason
|
||||||
|
}
|
||||||
|
out := template.FuncMap{
|
||||||
|
"holdText": func(st interface{}) string { return holdFor(stackOf(st)) },
|
||||||
|
"updateErrorText": func(st interface{}) string {
|
||||||
|
stk := stackOf(st)
|
||||||
|
if stk.UpdateErrorKey == stacks.UpdateErrorKeyHeld && s.backupMgr != nil {
|
||||||
|
if held, why := s.backupMgr.RestoreHoldForLang(stk.Name, lang); held && why != "" {
|
||||||
|
return why
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return stk.UpdateErrorIn(lang)
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if base, ok := prev["updateBadge"].(func(stacks.Stack) *MetaBadge); ok {
|
||||||
|
out["updateBadge"] = func(st stacks.Stack) *MetaBadge {
|
||||||
|
bd := base(st)
|
||||||
|
if bd != nil && st.HoldReason != "" { // R-625: the title is the hold's opening, for THIS reader
|
||||||
|
bd.Title = firstSentence(holdFor(st))
|
||||||
|
}
|
||||||
|
return bd
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,136 @@
|
|||||||
|
package web
|
||||||
|
|
||||||
|
import (
|
||||||
|
"html"
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/i18n"
|
||||||
|
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||||
|
)
|
||||||
|
|
||||||
|
// heldBehindStack is R-625's measured shape: HELD by a failed update, while the catalog has already
|
||||||
|
// published a newer version — the moment the page used to say „Frissítés elérhető".
|
||||||
|
func heldBehindStack(t *testing.T, s *Server, boxLang string) (stacks.Stack, *backup.Manager) {
|
||||||
|
t.Helper()
|
||||||
|
b := backup.NewManager(s.cfg, s.settings, log.New(io.Discard, "", 0))
|
||||||
|
s.backupMgr = b
|
||||||
|
if boxLang != "" {
|
||||||
|
if err := s.settings.SetLanguage(boxLang); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
at := time.Date(2026, 9, 23, 8, 0, 0, 0, time.UTC)
|
||||||
|
if err := b.HoldAfterFailedUpdateHolding("kimai", at, at.Add(-6*time.Hour), backup.UpdateTierLocal, b.UpdateCopyHolds("kimai", backup.UpdateTierLocal), ""); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_, why := b.RestoreHoldFor("kimai") // in the BOX's language, as fillHoldReason stores it
|
||||||
|
st := i18nStack("kimai", "Kimai Time", stacks.StateStopped, true)
|
||||||
|
st.AppConfig = &stacks.AppConfig{Deployed: true, InstalledImages: map[string]stacks.InstalledImage{"web": {Ref: "kimai/kimai2:2.30.0"}}}
|
||||||
|
st.CatalogImages = map[string]string{"web": "kimai/kimai2:2.31.0"}
|
||||||
|
if compareInstalledToTemplate(st) != updateBehind {
|
||||||
|
t.Fatal("fixture: the held app must also be BEHIND the catalog (R-625's shape)")
|
||||||
|
}
|
||||||
|
st.HoldReason = why
|
||||||
|
st.UpdatePhase, st.UpdateErrorKey, st.UpdateError = stacks.UpdatePhaseFailed, stacks.UpdateErrorKeyHeld, why
|
||||||
|
return st, b
|
||||||
|
}
|
||||||
|
|
||||||
|
func renderHeld(t *testing.T, s *Server, st stacks.Stack, lang, tmpl string) string {
|
||||||
|
t.Helper()
|
||||||
|
c := i18nCase{"r625_" + tmpl, tmpl, func() map[string]interface{} {
|
||||||
|
if tmpl == "stacks" {
|
||||||
|
d := i18nLayoutData("stacks", "Alkalmazások")
|
||||||
|
d["Stacks"] = []stacks.Stack{st}
|
||||||
|
for _, k := range []string{"Subdomains", "MissingStorage", "NetworkStubs", "NetworkWarnings", "StorageLabels"} {
|
||||||
|
d[k] = map[string]string{}
|
||||||
|
}
|
||||||
|
return d
|
||||||
|
}
|
||||||
|
d := i18nLayoutData("stacks", "Kimai")
|
||||||
|
cp := st
|
||||||
|
d["Stack"], d["Meta"], d["AppInfo"] = &cp, cp.Meta, cp.Meta.AppInfo
|
||||||
|
return d
|
||||||
|
}}
|
||||||
|
return html.UnescapeString(renderI18nCase(t, s, lang, c))
|
||||||
|
}
|
||||||
|
|
||||||
|
// R-625 — a held app says it is stopped and needs a restore, in both languages, on the app page; and
|
||||||
|
// NEITHER page offers the Update button.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (REPORT.md): remove the HoldReason branch from updateBadgeAt (v0.264.0's shape)
|
||||||
|
// — the Hungarian page says „Frissítés elérhető" and this fails.
|
||||||
|
func TestR625_AHeldAppSaysStoppedRestoreNeeded(t *testing.T) {
|
||||||
|
s := i18nTestServer(t)
|
||||||
|
st, _ := heldBehindStack(t, s, "")
|
||||||
|
for _, c := range []struct{ lang, badge, invite, title string }{
|
||||||
|
{"hu", "Megállítva — visszaállítás szükséges", "Frissítés elérhető", "A(z) kimai frissítése"},
|
||||||
|
{"en", "Stopped — restore needed", "Update available", "The update of kimai"},
|
||||||
|
} {
|
||||||
|
page := renderHeld(t, s, st, c.lang, "app_info")
|
||||||
|
if !strings.Contains(page, `<span class="tag tag-error" title="`) || !strings.Contains(page, c.badge) {
|
||||||
|
t.Errorf("%s app page: the held badge %q (tag-error) is missing", c.lang, c.badge)
|
||||||
|
}
|
||||||
|
if strings.Contains(page, c.invite) {
|
||||||
|
t.Errorf("%s app page still invites an update (%q) on a HELD app — R-625", c.lang, c.invite)
|
||||||
|
}
|
||||||
|
if i := strings.Index(page, c.badge); i > 0 && !strings.Contains(page[max(0, i-400):i], c.title) {
|
||||||
|
t.Errorf("%s: the badge title must open with the hold sentence (%q)", c.lang, c.title)
|
||||||
|
}
|
||||||
|
for _, tmpl := range []string{"app_info", "stacks"} {
|
||||||
|
if strings.Contains(renderHeld(t, s, st, c.lang, tmpl), `'kimai', 'update')`) {
|
||||||
|
t.Errorf("%s %s: the Update button is rendered for a HELD app", c.lang, tmpl)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// CONTROL: the same app, not held and running, DOES get the button on the list — so the check above
|
||||||
|
// can see a button when there is one.
|
||||||
|
free := st
|
||||||
|
free.HoldReason, free.UpdateError, free.UpdateErrorKey, free.State = "", "", "", stacks.StateRunning
|
||||||
|
if !strings.Contains(renderHeld(t, s, free, "hu", "stacks"), `'kimai', 'update')`) {
|
||||||
|
t.Fatal("control: a running, not-held app must show the Update button on the list")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// R-647 (1) — on an ENGLISH box, a Hungarian reader (`?lang=hu`, an operator) reads the hold and the
|
||||||
|
// held update's error in HUNGARIAN; the English reader reads English. Both ways, both pages.
|
||||||
|
//
|
||||||
|
// COMPANION RED-PROOF (REPORT.md): make readerFuncs return nil (v0.264.0: no Hungarian override) — the
|
||||||
|
// Hungarian page carries the English hold and this fails.
|
||||||
|
func TestR647_AReaderInTheOtherLanguageReadsTheHoldInTheirs(t *testing.T) {
|
||||||
|
s := i18nTestServer(t)
|
||||||
|
st, _ := heldBehindStack(t, s, "en")
|
||||||
|
if !strings.Contains(st.HoldReason, "The update of kimai") {
|
||||||
|
t.Fatalf("fixture: an English box must store the English hold, got %q", st.HoldReason)
|
||||||
|
}
|
||||||
|
const huWant, enWant = "A(z) kimai frissítése 2026-09-23 10:00-kor nem sikerült", "The update of kimai at 2026-09-23 10:00 did not succeed"
|
||||||
|
for _, tmpl := range []string{"app_info", "stacks"} {
|
||||||
|
hu := renderHeld(t, s, st, "hu", tmpl)
|
||||||
|
if !strings.Contains(hu, huWant) || strings.Contains(hu, enWant) {
|
||||||
|
t.Errorf("%s: the Hungarian reader of an English box must read the hold in Hungarian only", tmpl)
|
||||||
|
}
|
||||||
|
en := renderHeld(t, s, st, "en", tmpl)
|
||||||
|
if !strings.Contains(en, enWant) || strings.Contains(en, huWant) {
|
||||||
|
t.Errorf("%s: the English reader must read the hold in English only", tmpl)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The held badge's Hungarian literal and the bundle say the same thing (the parity rule for funcmap).
|
||||||
|
func TestR625_HeldBadgeHungarianMatchesTheBundle(t *testing.T) {
|
||||||
|
b, err := i18n.Shared()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
st := stacks.Stack{Name: "x", HoldReason: "Egy. Kettő."}
|
||||||
|
if got, want := b.Msg("hu", "badge.update.held"), updateBadgeAt(st, time.Now()).Label; got != want {
|
||||||
|
t.Fatalf("badge.update.held: hu.json %q, funcmap %q", got, want)
|
||||||
|
}
|
||||||
|
if got := updateBadgeAt(st, time.Now()).Title; got != "Egy." {
|
||||||
|
t.Fatalf("the title is the first sentence, got %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -310,6 +310,9 @@ func (s *Server) loadTemplates() {
|
|||||||
for name, fn := range s.localeFuncs(lang) {
|
for name, fn := range s.localeFuncs(lang) {
|
||||||
funcs[name] = fn
|
funcs[name] = fn
|
||||||
}
|
}
|
||||||
|
for name, fn := range s.readerFuncs(lang, funcs) {
|
||||||
|
funcs[name] = fn
|
||||||
|
}
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
t, st, err := parseTemplateSet(b, lang, funcs)
|
t, st, err := parseTemplateSet(b, lang, funcs)
|
||||||
took := time.Since(start)
|
took := time.Since(start)
|
||||||
|
|||||||
@@ -177,7 +177,7 @@
|
|||||||
<span class="stack-state-badge state-run">Fut</span>
|
<span class="stack-state-badge state-run">Fut</span>
|
||||||
|
|
||||||
|
|
||||||
|
<span class="tag tag-error" title="A frissítés megállt: nincs friss mentés.">Megállítva — visszaállítás szükséges</span>
|
||||||
<a href="https://paste.example.hu" target="_blank" class="btn btn-sm btn-outline">Megnyitás ↗</a>
|
<a href="https://paste.example.hu" target="_blank" class="btn btn-sm btn-outline">Megnyitás ↗</a>
|
||||||
<a href="/stacks/privatebin/logs" class="btn btn-sm btn-outline">Napló</a>
|
<a href="/stacks/privatebin/logs" class="btn btn-sm btn-outline">Napló</a>
|
||||||
|
|
||||||
|
|||||||
@@ -252,7 +252,7 @@
|
|||||||
<span class="tag tag-neutral"><span class="dot"></span>Leállítva</span>
|
<span class="tag tag-neutral"><span class="dot"></span>Leállítva</span>
|
||||||
|
|
||||||
|
|
||||||
|
<span class="tag tag-error" title="A frissítés megállt, a mentés hiányzik.">Megállítva — visszaállítás szükséges</span>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package web
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||||
@@ -73,6 +74,18 @@ func compareInstalledToTemplate(s stacks.Stack) updateState {
|
|||||||
// never the decision. A new branch here needs its twin there, and
|
// never the decision. A new branch here needs its twin there, and
|
||||||
// TestLocaleFuncsHungarianBundleMatchesFuncMap fails if hu.json and these literals disagree.
|
// TestLocaleFuncsHungarianBundleMatchesFuncMap fails if hu.json and these literals disagree.
|
||||||
func updateBadgeAt(s stacks.Stack, now time.Time) *MetaBadge {
|
func updateBadgeAt(s stacks.Stack, now time.Time) *MetaBadge {
|
||||||
|
// R-625 (v0.265.0). A HELD app is stopped, and the only way back is a restore — a newer catalog
|
||||||
|
// version does not lift the hold (`09` §6.1), so „Frissítés elérhető" beside a button that answers
|
||||||
|
// 409 was two verdicts disagreeing. One verdict, read by the badge and by the button (stacks.html
|
||||||
|
// renders no Update button while HoldReason is set), exactly as R-524 did it for Ahead. True for
|
||||||
|
// both hold kinds: a failed restore holds the app stopped too, and the way back is also a restore.
|
||||||
|
if s.HoldReason != "" {
|
||||||
|
return &MetaBadge{
|
||||||
|
Label: "Megállítva — visszaállítás szükséges",
|
||||||
|
Class: "tag-error",
|
||||||
|
Title: firstSentence(s.HoldReason),
|
||||||
|
}
|
||||||
|
}
|
||||||
switch compareInstalledToTemplate(s) {
|
switch compareInstalledToTemplate(s) {
|
||||||
case updateCurrent:
|
case updateCurrent:
|
||||||
return &MetaBadge{
|
return &MetaBadge{
|
||||||
@@ -120,3 +133,12 @@ func updateBadgeAt(s stacks.Stack, now time.Time) *MetaBadge {
|
|||||||
//
|
//
|
||||||
// It is INFORMATION ONLY. It is wired to no action, and the Frissítés button is untouched.
|
// It is INFORMATION ONLY. It is wired to no action, and the Frissítés button is untouched.
|
||||||
func updateBadge(s stacks.Stack) *MetaBadge { return updateBadgeAt(s, time.Now().UTC()) }
|
func updateBadge(s stacks.Stack) *MetaBadge { return updateBadgeAt(s, time.Now().UTC()) }
|
||||||
|
|
||||||
|
// firstSentence is the hold sentence's opening clause, for a badge title (R-625): up to and including
|
||||||
|
// the first full stop that ends a sentence; the whole text when it has none.
|
||||||
|
func firstSentence(s string) string {
|
||||||
|
if i := strings.Index(s, ". "); i >= 0 {
|
||||||
|
return s[:i+1]
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|||||||
@@ -61,7 +61,8 @@
|
|||||||
"page.title.storage_network": "slice 1 -- pinned by TestHandlerTitleKeysMatchHungarianTitle",
|
"page.title.storage_network": "slice 1 -- pinned by TestHandlerTitleKeysMatchHungarianTitle",
|
||||||
"event.app_update_undone": "BORN AS A KEY, v0.264.0 (R-606 / the update events) -- a NEW sentence, never a Go literal.",
|
"event.app_update_undone": "BORN AS A KEY, v0.264.0 (R-606 / the update events) -- a NEW sentence, never a Go literal.",
|
||||||
"settings_notifications.app_update_undone": "BORN AS A KEY, v0.264.0 (R-606 / the update events) -- a NEW sentence, never a Go literal.",
|
"settings_notifications.app_update_undone": "BORN AS A KEY, v0.264.0 (R-606 / the update events) -- a NEW sentence, never a Go literal.",
|
||||||
"settings_notifications.app_update_held": "BORN AS A KEY, v0.264.0 (R-606 / the update events) -- a NEW sentence, never a Go literal."
|
"settings_notifications.app_update_held": "BORN AS A KEY, v0.264.0 (R-606 / the update events) -- a NEW sentence, never a Go literal.",
|
||||||
|
"badge.update.held": "BORN AS A KEY, v0.265.0 (R-625) -- a NEW badge for a held app; its Hungarian twin in updatebadge.go is pinned by TestR625_HeldBadgeHungarianMatchesTheBundle."
|
||||||
},
|
},
|
||||||
"flash.share.already_on": "A megosztás már be van kapcsolva.",
|
"flash.share.already_on": "A megosztás már be van kapcsolva.",
|
||||||
"flash.share.enable_failed": "A megosztás bekapcsolása nem sikerült.",
|
"flash.share.enable_failed": "A megosztás bekapcsolása nem sikerült.",
|
||||||
|
|||||||
Reference in New Issue
Block a user