felhom.eu/REPORT-dooplex-alarms-2026-10-24.md

# REPORT — two DooPlex backup alarms, every alarm the mail route delivers, and R-1250 (2026-10-24, attended)

Brief: „two DooPlex backup alarms — find why, fix, and check every other alarm the new mail route now delivers".
Read first: `runbooks/total-loss-of-dooplex.md`, `runbooks/RUNBOOK-hub-db-offsite-backup.md`,
`audits/dooplex-survival-2026-10-22/PLAN.md`, R-232, R-1227, R-1241, the two homelab rules and the Alertmanager
route, homelab HM-040…HM-049 and the window record; design `08` §9. Evidence: `audits/dooplex-alarms-2026-10-24/`.

## Three lines

1. **Both alarms had one cause, and both are fixed.** Yesterday's reboot restarted the database, and it now takes the
   password of the other of its two secrets; the dump job could not log in, and the Gitea off-site push then refused
   the old dump. The job now reads the right secret. Cleared: the dump alarm 10:44Z, the Gitea alarm 10:51Z.
2. **Nothing else fires now.** In 7 days only the pulled disk's two alarms fired long (silenced by you since 10-12,
   ended by the pull); the rest were minutes long or are known.
3. **Fewer mails:** critical every 4 h (was 1 h), warning every 12 h (was 4 h), all backup alarms in one mail.

## What I asked you, and what you said

- The dump fix → **the CronJob only** (the database's own definition untouched).
- The mail change needs an Alertmanager restart, not a reload → **restart**.
- The R-1250 proof needed a downloader on another machine → **use DooPlex's k3s qBittorrent**.

## The alarm table (7 days, Prometheus `ALERTS`; nothing firing at 10:51Z)

| Alarm | Severity | When (UTC) | True? | Cause | Fix / row |
|---|---|---|---|---|---|
| `DooplexGiteaOffsiteStale` | critical | 10-24 00:57 → 10:51 | true | the push refused a 12 h old dump | the dump fix; R-232 note |
| `PostgresqlDumpStale` | warning | 10-23 22:17 → 10-24 10:44 | true | dump job login failed (28P01) | homelab `531023d` |
| `DiskOfflineUncorrectable`, `DiskUncorrectableSectors` (Z6A07P2G) | critical | whole window (≥ 145 h) → 10-23 12:02 | true | the dying HDD4 | silenced by you 10-12; pulled 10-23 — R-934 closed |
| `DiskDisappeared` (Z6A07P2G) | critical | 10-23 12:17 → 18:02 | true | the pull | expected; cleared |
| `DiskInventoryChanged` | warning | 10-23 12:17 → 20:17 | true | the pull | expected; cleared |
| `LonghornVolumeDegraded` (two Plex volumes) | warning | 10-23 08:22 → 11:22 | true | the HM-040 rebuild | cleared |
| `PodRestartingFrequently` (felhom-webpage) | warning | 10-19 09:02 → 10:07 | true | git-syncs stopped on errors | R-1163 (closed 10-19) |
| `WanUploadDegraded`, `WanDownloadDegraded`, `WanUpstreamPacketLoss`, `WanBufferbloat` | warning/info | minutes, 10-17 → 10-20 | true | short WAN dips (the flood days) | none needed |
| `AuthentikHighLatency` | warning | 10-21 08:12, 5 min | true | a short spike | none needed |

## The mail change (homelab `22d7697`, applied by a restart, read back from the running Alertmanager)

| | before | after |
|---|---|---|
| critical | every 1 h | every 4 h |
| warning | every 4 h | every 12 h |
| other (info) | every 4 h | every 12 h |
| backup alarms | one mail stream per alarm | one mail for the group, every 4 h |
| a disk | one stream per alarm | one mail per drive, daily (unchanged) |
| resolved mail | `send_resolved: true` | unchanged; the mail counter rose once after each clear (10:47, and by 10:57) — that they read „RESOLVED" is not seen: the mailbox is not mine |

## Part 4 — R-1250, committed and proven

Agent `29bcf44` + `1e087bb`, controller `046b808` (commit only; release in the next train). A seeding box: the agent
says „waiting for quiet"; the controller records the torrent app's limit, holds it at 1 KiB/s, asks for one test,
puts the exact limit back. **Proven on demo-hp with a real seed** (10 Mbit/s from 9202 to DooPlex's qBittorrent):
refused without the hold (11.1 Mbit/s); with it the uplink fell to 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, the limit
back to 1 249 280 B/s; a kill in the middle → the start-up check put it back. The brief said „a nested VM": NOT done —
a nested VM would need its own tunnel to ep0 to run the real test; demo-hp's host with probes did the same with the
real tunnel. The pinned agent client is unit-tested only.

## Commits, CI, undo

| Repo | Commit | What | CI |
|---|---|---|---|
| homelab-manifests | `531023d` | the dump CronJob's secret | no CI in this repo |
| homelab-manifests | `22d7697` | the mail routes, the dump alarm's label | no CI |
| homelab-manifests | `9f399fe` | REPORT | no CI |
| felhom-agent | `29bcf44`, `1e087bb` | R-1250 | 2286, 2288 success |
| felhom-controller | `046b808` | R-1250 | 2287 success |
| app-catalog-felhom.eu | `7de554e`, `5e196e1` | the box walk: HDD_PATH, the leftover check | 2289 success, (final push) |
| felhom-agent / felhom-controller | `5e05627` / `321df1f` | REPORT, README | (final push) |
| felhom.eu | this commit | register, STATUS, CONTEXT, `08` §9, evidence | (final push) |

Undo: `audits/dooplex-alarms-2026-10-24/README.md` § Undo. Backups of the three homelab files: `*.before` there.

## Rows

**Before 191, after 190; opened 0, closed 1 (R-934 — the disk was already pulled).** R-232: a dated note and a dated
check (2026-10-25, tonight's sweep). R-1250: NARROWED (fixed on main, waits for the release train).

## Observations

- The database's two secrets for one role stay (only the CronJob moved) — homelab list item 1 (not a Felhom row).
- A stale dump stops the Gitea + secrets off-site push for as long as it lasts — NOT-A-FINDING: the designed fail-closed
  order (`PLAN.md`: the dump first, for consistency); the alarm caught it inside its 26 h, and it now comes in one mail
  with its cause.
- Alertmanager cannot reload a ConfigMap change (the brief assumed it could) — NOT-A-FINDING: the homelab manifest's
  design (`envsubst` init container); you chose the restart; the fact is in CC's memory and the homelab list.
- The agent said „waiting for quiet" before it had watched a minute — fixed in the session (`1e087bb`, found live).
- The box walk sent a folder inside the drive as HDD_PATH (refused since R-839) — fixed in the session (`7de554e`).
- The box walk called the template folder a leftover after every remove — fixed in the session (`5e196e1`).
- R-934 said the failed disk was still in DooPlex; STATUS still listed R-1214's done check — fixed (row closed, line removed).
- CC's memory said Prometheus has no config-reloader; it has one since 2026-10-12 — fixed (memory note).
- `govulncheck` warns GO-2026-6809 (crypto/x509, stdlib) in the agent and the controller — NOT-A-FINDING here: the Go
  toolchain row R-972 owns it.
- My Gmail connector's mailbox gets no Alertmanager mail — NOT-A-FINDING: it is another mailbox than `SMTP_TO`; 60 sent,
  0 failed in 7 days. The Alertmanager config did not change yesterday (last 10-12).
- `kubectl exec -i` never closed stdin for `pg_restore` — NOT-A-FINDING: my method; the host's own `pg_restore` read the dumps.
- The 9202 ISO torrent never uploaded (behind NAT, 18 leechers / 1080 seeds) — NOT-A-FINDING: why the LAN peer was used.
- I overwrote homelab-manifests' REPORT.md, which held another session's HM-050 report from today — NOT-A-FINDING: the
  repo's overwrite rule; HM-050's entry stays in its CHANGELOG.

## Teardown, three layers

- **Machine:** 9202 — qBittorrent removed through the product (no container, no volume; the template folder stays, as for
  every app); the test files, the forwarder and the probe removed. DooPlex's qBittorrent — the test torrent and its
  files deleted (by hash), nothing else touched.
- **Host:** demo-hp — the agent probe stopped and deleted, no iperf3 left; the installed agent untouched. DooPlex — the
  9202 password copy and the helper's session files shredded; the clones removed.
- **Hub:** nothing provisioned.

## Instruction files, poster

No instruction file edited. No poster fact changed (the mail intervals are not on it).