admin
2fc4a15fa3
The cooldown key was customerID:eventType plus the tier and run suffixes, and none of them names an app, so every app going down inside the same hour collapsed onto one key and only the first was mailed. Measured on demo-hp: bookstack sent 09:27:51, privatebin suppressed 09:31:51 under key=demo-hp:app_start_failed. cooldownStackSuffix is the third sibling of cooldownTierSuffix and cooldownRunSuffix, and separate for the reason the second one's docstring already gives: the existing two keep byte-identical semantics for every type that uses them. It is ALLOW-LISTED to app_start_failed and takes the event type as well as the details, unlike its siblings, and that asymmetry is the safety property. The backup family's cooldown is coarse ON PURPOSE (R-97a, R-182) so one full disk sends one digest rather than one mail per app - and crossdrive_failed is severity error, reaches the operator leg, and carries stack_name through a DIFFERENT struct, so a payload-shape rule would have split it silently. The hour itself does not change. Gate 11 refuses a push whose REPORT.md carries an observation with neither `FILED: R-NNN` nor `NOT-A-FINDING: <reason>`. It deliberately does NOT accept a passing mention of some other R-number: the lost item cited R-182 as an analogy, so "cites a register row" would have passed the very item the gate exists to catch. That discrepancy with the spec is recorded in the gate's docstring. Registered here and in the controller and agent runners. NOT in the catalog runner - it has no shared-gate mechanism and appends --all to every gate; filed as R-391 rather than left as a sentence, which is this session's lesson. PROMPT-TEMPLATE.md §15.9 corrected: "documented, NOT acted on" was the wording that invited the gap, and it now names the markers and points at the gate. R-390 filed for the golden-bake runbook's missing `pveam update`. Hub tests 709 -> 716.
6.8 KiB
REPORT — felhom.eu: hub v0.107.0 (R-387), the alarm ladder, and golden 0.223.0
Session 2026-08-23. Companion to felhom-controller v0.223.0 (R-329, R-386) — see that repo's
REPORT.md for the controller work and the full live walk.
1. Baselines, and the hub's four numbers as read
| Repo | at start | at end |
|---|---|---|
| felhom.eu | 55274d5e |
hub v0.107.0 deployed |
| felhom-controller | 14137efa (v0.222.0) |
v0.223.0 deployed |
| felhom-agent | 40d857b5 |
untouched |
Hub's four numbers, read live from GET /configuration before starting:
golden_version 0.222.0 · agent_version 0.130.0 · min_agent 0.129.0 ·
controller floor 0.222.0. All four as the task predicted; the operator's 0.222.0 vouch had landed.
2. The hub's deployment path — §6's premise was wrong, and here it is
felhom.eu/manifests/hub.yaml, line 128. ArgoCD Application/felhom tracks
https://gitea.dooplex.hu/admin/felhom.eu.git, path manifests, with syncPolicy.automated.enabled = false. Bumped 0.106.0 → 0.107.0 in commit 68a9f54. There is no out-of-git deployment
path — the finding §6 braced for does not exist. The image was built and pushed to the registry
before the manifest landed, so a sync could never have pointed at a missing tag, and the sync was
then requested deliberately (refresh=hard, then a patched operation). Never kubectl set image.
3. R-387 — what was wrong
One handler, two fields, opposite discipline: an unknown event_type is rejected with a loud 400;
an unknown severity was rewritten to info without a word, and severityNotifies drops info
before both legs. The guard built to catch exactly this sat downstream of the rewrite — the
dispatcher's unrecognized severity line can never execute for an API event, because the coercion one
line earlier guarantees the value it looks for cannot arrive.
Measured on the live hub DB: 91 app_start_failed events stored all-time, 0
notification_log rows before this session — not one, on any channel, while every POST returned 200.
The coercion stays. A rejected event is a lost event, and losing an alarm is worse than mis-routing one. Only the silence is fixed.
4. The dead-branch decision, and the reason
KEPT. Not caution — evidence. cmd/hub/main.go wires dispatcher.ProcessEvent directly as
the monitor.EventNotifyFunc for the staleness, host-staleness and offsite-box checkers, and those
hub-generated events never pass through the ingest handler at all. For every one of them that line is
the only severity guard there is. Deleting it as "dead" would have removed the live half while the
dead half supplied the justification.
Verified while deciding: all 90 severity literals in internal/monitor are already valid, so the
guard is silent because the producers are correct. ("warn" in internal/web is UI badge vocabulary,
not a severity.)
5. Files changed, commits, CI
| Commit | Contents |
|---|---|
68a9f54 |
hub v0.107.0 (ingest WARN + kept-branch note + tests), manifest bump, golden 0.223.0 evidence |
<docs> |
alarm ladder §6.1/§7/§8, register, STATUS.md, REPORT.md, drill record |
Files: hub/internal/api/handler.go, hub/internal/notify/dispatcher.go, hub/CHANGELOG.md,
manifests/hub.yaml, documentation/architecture/08-alarm-ladder.md,
documentation/backlog/{OPEN,CLOSED}-ITEMS.md, documentation/tests/golden-0.223.0-2026-08-23/,
documentation/audits/DRILL-r329-r386-2026-08-23/, plus two new test files.
CI runs confirmed BY ID (id and run_number diverge — both printed): see §5 of the controller
REPORT for its runs; felhom.eu's are listed at the end of this file.
6. Tests and red-proofs
internal/api/r387_severity_visibility_test.go — the event is not lost, the stored severity is
still info, and the WARN names customer + type + value; plus a guard that a valid severity stays
silent, because an alarm on the normal path is one people learn to ignore.
internal/notify/r329_app_start_failed_test.go — the routing consequence: operator emailed, customer
not, unless opted in, in which case both legs deliver and the customer's copy carries the Hungarian
template. Scenario B is also the positive control for Scenario A's absence claim.
Test count 702 → 709.
Red-proof (seen failing): delete the ingest WARN → the hub rewrote a severity and said nothing, with the log showing only the ordinary [INFO] Event from c1: backup_failed (info). The
guard sits at ingest, because that is the last point at which the offending value still exists.
7. Golden
Baked and PUBLISHED: 0.223.0. GOLDEN_SHA256 = 9eaf39ac39219b42ec9e6cbf890275febcdcc6f53325fe0c0f591d3431044f17; upload OK (HTTP 201); round-trip
HTTP 206; all five acceptance markers counted (docker OK (overlay2 1, including mount point 2,
upload OK 1, excluding 0, FATAL 0). VOUCHING IS THE OPERATOR'S ACT AND WAS NOT DONE HERE.
Runbook deviation, second session running: §4.1 omits pveam update, so the virgin snapshot's
stale template index fails as 400 … no such template.
8. Scenario H, live against v0.107.0
[WARN] [api] Event from demo-hp: severity "warn" is not in {info,warning,error,critical} — coercing
to "info", which severityNotifies DROPS, so this backup_failed alert will reach NOBODY. Fix the
emitting controller; this event is stored but not routed.
Both the bad-severity POST and the error control returned 200 (nothing lost), and the control
produced no warning — the guard does not fire on the normal path.
9. Register size
| File | Before | After |
|---|---|---|
OPEN-ITEMS.md |
328,325 B | 328,132 B |
CLOSED-ITEMS.md |
71,441 B | 74,642 B |
R-329 and R-386 compressed into CLOSED with their rules kept; R-387 (closed) and R-388 (the notification-model product decision, open, operator's call) filed.
10. Observations
Markers added 2026-08-24 (gate 11, R-389). Item 1 is the finding that had no row; adding its marker is the first thing the gate ever asked for. The observations' text is unchanged.
- The operator cooldown key carries no app identifier. PrivateBin's alarm four minutes after
BookStack's was logged
suppressed — operator cooldown 1h, key=demo-hp:app_start_failed, so only the first app-down per hour reaches the operator by e-mail. R-182's known shape; harmless while the event was undeliverable, and no longer. Not fixed here. FILED: R-389 - Two probe events remain as rows for
demo-hpfrom Scenario H — inert, and named rather than left. NOT-A-FINDING: two inert event rows on a Tier 0 demo box, created deliberately as a live control and named in that session's teardown; they carry no state and nothing reads them.