felhom.eu/REPORT-dooplex-alarms-2026-10-24.md

REPORT — two DooPlex backup alarms, every alarm the mail route delivers, and R-1250 (2026-10-24, attended)

Brief: „two DooPlex backup alarms — find why, fix, and check every other alarm the new mail route now delivers". Read first: runbooks/total-loss-of-dooplex.md, runbooks/RUNBOOK-hub-db-offsite-backup.md, audits/dooplex-survival-2026-10-22/PLAN.md, R-232, R-1227, R-1241, the two homelab rules and the Alertmanager route, homelab HM-040…HM-049 and the window record; design 08 §9. Evidence: audits/dooplex-alarms-2026-10-24/.

Three lines

1. Both alarms had one cause, and both are fixed. Yesterday's reboot restarted the database, and it now takes the password of the other of its two secrets; the dump job could not log in, and the Gitea off-site push then refused the old dump. The job now reads the right secret. Cleared: the dump alarm 10:44Z, the Gitea alarm 10:51Z.
2. Nothing else fires now. In 7 days only the pulled disk's two alarms fired long (silenced by you since 10-12, ended by the pull); the rest were minutes long or are known.
3. Fewer mails: critical every 4 h (was 1 h), warning every 12 h (was 4 h), all backup alarms in one mail.

What I asked you, and what you said

• The dump fix → the CronJob only (the database's own definition untouched).
• The mail change needs an Alertmanager restart, not a reload → restart.
• The R-1250 proof needed a downloader on another machine → use DooPlex's k3s qBittorrent.

The alarm table (7 days, Prometheus ALERTS; nothing firing at 10:51Z)

Alarm	Severity	When (UTC)	True?	Cause	Fix / row
DooplexGiteaOffsiteStale	critical	10-24 00:57 → 10:51	true	the push refused a 12 h old dump	the dump fix; R-232 note
PostgresqlDumpStale	warning	10-23 22:17 → 10-24 10:44	true	dump job login failed (28P01)	homelab 531023d
DiskOfflineUncorrectable, DiskUncorrectableSectors (Z6A07P2G)	critical	whole window (≥ 145 h) → 10-23 12:02	true	the dying HDD4	silenced by you 10-12; pulled 10-23 — R-934 closed
DiskDisappeared (Z6A07P2G)	critical	10-23 12:17 → 18:02	true	the pull	expected; cleared
DiskInventoryChanged	warning	10-23 12:17 → 20:17	true	the pull	expected; cleared
LonghornVolumeDegraded (two Plex volumes)	warning	10-23 08:22 → 11:22	true	the HM-040 rebuild	cleared
PodRestartingFrequently (felhom-webpage)	warning	10-19 09:02 → 10:07	true	git-syncs stopped on errors	R-1163 (closed 10-19)
WanUploadDegraded, WanDownloadDegraded, WanUpstreamPacketLoss, WanBufferbloat	warning/info	minutes, 10-17 → 10-20	true	short WAN dips (the flood days)	none needed
AuthentikHighLatency	warning	10-21 08:12, 5 min	true	a short spike	none needed

The mail change (homelab 22d7697, applied by a restart, read back from the running Alertmanager)

	before	after
critical	every 1 h	every 4 h
warning	every 4 h	every 12 h
other (info)	every 4 h	every 12 h
backup alarms	one mail stream per alarm	one mail for the group, every 4 h
a disk	one stream per alarm	one mail per drive, daily (unchanged)
resolved mail	send_resolved: true	unchanged; the mail counter rose once after each clear (10:47, and by 10:57) — that they read „RESOLVED" is not seen: the mailbox is not mine

Part 4 — R-1250, committed and proven

Agent 29bcf44 + 1e087bb, controller 046b808 (commit only; release in the next train). A seeding box: the agent says „waiting for quiet"; the controller records the torrent app's limit, holds it at 1 KiB/s, asks for one test, puts the exact limit back. Proven on demo-hp with a real seed (10 Mbit/s from 9202 to DooPlex's qBittorrent): refused without the hold (11.1 Mbit/s); with it the uplink fell to 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, the limit back to 1 249 280 B/s; a kill in the middle → the start-up check put it back. The brief said „a nested VM": NOT done — a nested VM would need its own tunnel to ep0 to run the real test; demo-hp's host with probes did the same with the real tunnel. The pinned agent client is unit-tested only.

Commits, CI, undo

Repo	Commit	What	CI
homelab-manifests	531023d	the dump CronJob's secret	no CI in this repo
homelab-manifests	22d7697	the mail routes, the dump alarm's label	no CI
homelab-manifests	9f399fe	REPORT	no CI
felhom-agent	29bcf44, 1e087bb	R-1250	2286, 2288 success
felhom-controller	046b808	R-1250	2287 success
app-catalog-felhom.eu	7de554e, 5e196e1	the box walk: HDD_PATH, the leftover check	2289 success, (final push)
felhom-agent / felhom-controller	5e05627 / 321df1f	REPORT, README	(final push)
felhom.eu	this commit	register, STATUS, CONTEXT, 08 §9, evidence	(final push)

Undo: audits/dooplex-alarms-2026-10-24/README.md § Undo. Backups of the three homelab files: *.before there.

Rows

Before 191, after 190; opened 0, closed 1 (R-934 — the disk was already pulled). R-232: a dated note and a dated check (2026-10-25, tonight's sweep). R-1250: NARROWED (fixed on main, waits for the release train).

Observations

• The database's two secrets for one role stay (only the CronJob moved) — homelab list item 1 (not a Felhom row).
• A stale dump stops the Gitea + secrets off-site push for as long as it lasts — NOT-A-FINDING: the designed fail-closed order (PLAN.md: the dump first, for consistency); the alarm caught it inside its 26 h, and it now comes in one mail with its cause.
• Alertmanager cannot reload a ConfigMap change (the brief assumed it could) — NOT-A-FINDING: the homelab manifest's design (envsubst init container); you chose the restart; the fact is in CC's memory and the homelab list.
• The agent said „waiting for quiet" before it had watched a minute — fixed in the session (1e087bb, found live).
• The box walk sent a folder inside the drive as HDD_PATH (refused since R-839) — fixed in the session (7de554e).
• The box walk called the template folder a leftover after every remove — fixed in the session (5e196e1).
• R-934 said the failed disk was still in DooPlex; STATUS still listed R-1214's done check — fixed (row closed, line removed).
• CC's memory said Prometheus has no config-reloader; it has one since 2026-10-12 — fixed (memory note).
• govulncheck warns GO-2026-6809 (crypto/x509, stdlib) in the agent and the controller — NOT-A-FINDING here: the Go toolchain row R-972 owns it.
• My Gmail connector's mailbox gets no Alertmanager mail — NOT-A-FINDING: it is another mailbox than SMTP_TO; 60 sent, 0 failed in 7 days. The Alertmanager config did not change yesterday (last 10-12).
• kubectl exec -i never closed stdin for pg_restore — NOT-A-FINDING: my method; the host's own pg_restore read the dumps.
• The 9202 ISO torrent never uploaded (behind NAT, 18 leechers / 1080 seeds) — NOT-A-FINDING: why the LAN peer was used.
• I overwrote homelab-manifests' REPORT.md, which held another session's HM-050 report from today — NOT-A-FINDING: the repo's overwrite rule; HM-050's entry stays in its CHANGELOG.

Teardown, three layers

• Machine: 9202 — qBittorrent removed through the product (no container, no volume; the template folder stays, as for every app); the test files, the forwarder and the probe removed. DooPlex's qBittorrent — the test torrent and its files deleted (by hash), nothing else touched.
• Host: demo-hp — the agent probe stopped and deleted, no iperf3 left; the installed agent untouched. DooPlex — the 9202 password copy and the helper's session files shredded; the clones removed.
• Hub: nothing provisioned.

Instruction files, poster

No instruction file edited. No poster fact changed (the mail intervals are not on it).