diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 688de7ca..8d8cac41 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,56 @@ +## v0.108.0 — only the first broken app per hour reached the operator (2026-08-23, R-389) + +**The hour is unchanged. The GRAIN is what was wrong.** + +`processOperator` keys its 1-hour cooldown on `customerID:eventType`, plus the tier and run suffixes. +**None of them names an app.** So every app that went down inside the same hour collapsed onto one +key and only the first was mailed. Measured live on `demo-hp` 2026-08-23: `bookstack` alarmed at +09:27:51 and was `sent`; `privatebin` alarmed four minutes later and was recorded +`suppressed — operator cooldown 1h, key=demo-hp:app_start_failed`. **Three apps dying together +produced one mail.** + +The app's identity was already on the wire — `AppDetails{StackName, DisplayName}` serialises as +`stack_name` — and the hub already made exactly this distinction twice, for backup tiers and for +backup runs. + +### The fix: a third sibling, allow-listed to ONE event type + +`cooldownStackSuffix` follows `cooldownTierSuffix` and `cooldownRunSuffix` exactly: guard on the +substring, unmarshal a one-field struct, return `""` on any doubt. A **separate function**, for the +reason `cooldownRunSuffix`'s docstring already gives — the existing two keep byte-identical semantics +for every type that uses them, so R-97a's and R-182's behaviour and their tests are untouched. + +**It takes the event type as well as the details, unlike its siblings, and that asymmetry is the whole +safety property.** `tier` and `run_id` appear only on types that want that grain; `stack_name` does +not have that property, so a payload-shape rule would have been wrong. + +**`perAppCooldownEvents` is a named register with `app_start_failed` and nothing else.** The backup +family's cooldown is coarse **on purpose** — R-97a and R-182 exist so one full disk sends one digest +rather than one mail per app. **This is not hypothetical: `crossdrive_failed` is severity `error`, +reaches the operator leg, and carries `stack_name`** through a different struct (`CrossDriveDetails`, +not `AppDetails`). A global suffix would have split it into one mail per app, silently. + +**`app_start_failed` gets per-app because it has no digest.** There is no `apps_down_run` summarising +a scan the way `backup_run_failures` summarises a run, so per-app is the only grain available that +does not lose alarms. + +**Fail-soft:** absent, empty or malformed details return `""`, the key degrades to v0.107.0's, and the +mail still goes. Losing an alarm is worse than mis-routing one. + +### Tests + +`internal/notify/r389_cooldown_grain_test.go`. Test count **709 → 716**. + +Scenario A asserts **two `sent` operator rows in the stored notification log**, not a return value, +and runs a positive control first — the operator leg must be shown delivering before any count means +anything. Scenario C's "no other key changed" is an absence claim and carries its own control: the +test proves it can *see* a key change before reporting that none occurred. + +**Red-proofs, both seen failing.** Dropping the suffix from the key reproduces the live symptom +verbatim (`1 operator mail(s), want 2`, and a suppression row reading `key=c1:app_start_failed`). +Removing the allow-list splits `crossdrive_failed`, `app_deployed`, `app_removed` and `backup_failed` +per app — the fence convicting exactly the types it was written for. + ## v0.107.0 — the hub rewrote a severity and said nothing, and the guard for that sat downstream of the rewrite (2026-08-23, R-387) **One handler, two fields, opposite discipline.** An unknown **event type** is rejected with a loud diff --git a/manifests/hub.yaml b/manifests/hub.yaml index a6ebac0f..d27bbd58 100644 --- a/manifests/hub.yaml +++ b/manifests/hub.yaml @@ -125,7 +125,7 @@ spec: spec: containers: - name: hub - image: gitea.dooplex.hu/admin/felhom-hub:0.107.0 + image: gitea.dooplex.hu/admin/felhom-hub:0.108.0 ports: - containerPort: 8080 name: http