hub v0.108.0: deploy the per-app cooldown grain (R-389)
gates / gates (push) Successful in 17s

Image built and pushed to the registry BEFORE this manifest bump lands, so a
sync can never point at a missing tag. Auto-sync is off; the sync that follows
is deliberate. Never kubectl set image.
This commit is contained in:
2026-08-23 13:54:48 +02:00
parent 2fc4a15fa3
commit 45659bdc5a
2 changed files with 54 additions and 1 deletions
+53
View File
@@ -1,3 +1,56 @@
## v0.108.0 — only the first broken app per hour reached the operator (2026-08-23, R-389)
**The hour is unchanged. The GRAIN is what was wrong.**
`processOperator` keys its 1-hour cooldown on `customerID:eventType`, plus the tier and run suffixes.
**None of them names an app.** So every app that went down inside the same hour collapsed onto one
key and only the first was mailed. Measured live on `demo-hp` 2026-08-23: `bookstack` alarmed at
09:27:51 and was `sent`; `privatebin` alarmed four minutes later and was recorded
`suppressed — operator cooldown 1h, key=demo-hp:app_start_failed`. **Three apps dying together
produced one mail.**
The app's identity was already on the wire — `AppDetails{StackName, DisplayName}` serialises as
`stack_name` — and the hub already made exactly this distinction twice, for backup tiers and for
backup runs.
### The fix: a third sibling, allow-listed to ONE event type
`cooldownStackSuffix` follows `cooldownTierSuffix` and `cooldownRunSuffix` exactly: guard on the
substring, unmarshal a one-field struct, return `""` on any doubt. A **separate function**, for the
reason `cooldownRunSuffix`'s docstring already gives — the existing two keep byte-identical semantics
for every type that uses them, so R-97a's and R-182's behaviour and their tests are untouched.
**It takes the event type as well as the details, unlike its siblings, and that asymmetry is the whole
safety property.** `tier` and `run_id` appear only on types that want that grain; `stack_name` does
not have that property, so a payload-shape rule would have been wrong.
**`perAppCooldownEvents` is a named register with `app_start_failed` and nothing else.** The backup
family's cooldown is coarse **on purpose** — R-97a and R-182 exist so one full disk sends one digest
rather than one mail per app. **This is not hypothetical: `crossdrive_failed` is severity `error`,
reaches the operator leg, and carries `stack_name`** through a different struct (`CrossDriveDetails`,
not `AppDetails`). A global suffix would have split it into one mail per app, silently.
**`app_start_failed` gets per-app because it has no digest.** There is no `apps_down_run` summarising
a scan the way `backup_run_failures` summarises a run, so per-app is the only grain available that
does not lose alarms.
**Fail-soft:** absent, empty or malformed details return `""`, the key degrades to v0.107.0's, and the
mail still goes. Losing an alarm is worse than mis-routing one.
### Tests
`internal/notify/r389_cooldown_grain_test.go`. Test count **709 → 716**.
Scenario A asserts **two `sent` operator rows in the stored notification log**, not a return value,
and runs a positive control first — the operator leg must be shown delivering before any count means
anything. Scenario C's "no other key changed" is an absence claim and carries its own control: the
test proves it can *see* a key change before reporting that none occurred.
**Red-proofs, both seen failing.** Dropping the suffix from the key reproduces the live symptom
verbatim (`1 operator mail(s), want 2`, and a suppression row reading `key=c1:app_start_failed`).
Removing the allow-list splits `crossdrive_failed`, `app_deployed`, `app_removed` and `backup_failed`
per app — the fence convicting exactly the types it was written for.
## v0.107.0 — the hub rewrote a severity and said nothing, and the guard for that sat downstream of the rewrite (2026-08-23, R-387)
**One handler, two fields, opposite discipline.** An unknown **event type** is rejected with a loud
+1 -1
View File
@@ -125,7 +125,7 @@ spec:
spec:
containers:
- name: hub
image: gitea.dooplex.hu/admin/felhom-hub:0.107.0
image: gitea.dooplex.hu/admin/felhom-hub:0.108.0
ports:
- containerPort: 8080
name: http