admin
44d1904286
gates / gates (push) Successful in 12m24s
R-232 dated note + a dated check for tonight's sweep; R-934 closed (the disk was pulled 2026-10-23); R-1250 fixed on main, live proof on demo-hp with a real seed; 08 §9 item 5. Evidence audits/dooplex-alarms-2026-10-24/.
52 lines
4.1 KiB
Markdown
52 lines
4.1 KiB
Markdown
# Two DooPlex backup alarms, and every alarm the mail route delivers — 2026-10-24 (attended)
|
||
|
||
Brief: „two DooPlex backup alarms — find why, fix, and check every other alarm the new mail route now delivers".
|
||
Read first: `runbooks/total-loss-of-dooplex.md`, `runbooks/RUNBOOK-hub-db-offsite-backup.md`,
|
||
`audits/dooplex-survival-2026-10-22/PLAN.md`, R-232, the two rules in homelab `mon-system/prometheus-rules.yaml`, the
|
||
Alertmanager route, homelab HM-040…HM-049 and the 2026-10-23 window record.
|
||
|
||
| File | What it shows |
|
||
|---|---|
|
||
| `part1-2-diagnosis.txt` | The off-site job's `FAILED: newest complete dump … 12 h old`; the dump jobs (3 failed), the failed dump folders (only an empty `globals.sql`); the database log's `28P01` (48 failed logins in 24 h, all from the backup pods, first 2026-10-23 16:00:01Z); which secret the role accepts (ok/refused only); the ruled-out causes (tunnel, block list, disk, overlap) |
|
||
| `part1-2-fix.txt` | After homelab `531023d`: the by-hand dump (13 files, `pg_restore --list` 1114 / 1660 / 753 entries), the CronJob's success time, the by-hand off-site push, the restore test (26 456 files, 12 repos `git fsck`), the snapshot on ep0, the clear times |
|
||
| `part3-alarms.txt`, `alerts-7d.json` | No alarm firing at 10:51Z; 7 days of `ALERTS` (Prometheus' whole window); the mail counter (60 sent, 0 failed); the silences; the routing read back from the running Alertmanager |
|
||
| `*.before` | The three homelab files as they were before this session's edits |
|
||
| `part4-*.txt` | R-1250's live proof (below) |
|
||
|
||
## Why the two alarms fired — one cause
|
||
|
||
1. The 2026-10-23 window reboot restarted `postgresql-2` (13:50Z).
|
||
2. CNPG 1.29 sets the role `postgres` twice at each instance start (`refreshCredentialsFromSecret`): first from the
|
||
superuser secret `postgresql-superuser`, then from the `bootstrap.initdb` owner secret `postgres-superuser`
|
||
(`owner: postgres`). The second wins.
|
||
3. The dump CronJob read `postgresql-superuser` → `password authentication failed` from 16:00Z → `PostgresqlDumpStale`
|
||
(22:15Z).
|
||
4. The Gitea off-site push at 00:20 refused the 12 h old Gitea dump, as designed → `DooplexGiteaOffsiteStale` (00:57Z).
|
||
|
||
Fix (operator's pick): the CronJob reads `postgres-superuser` (homelab `531023d`, ArgoCD sync of that one CronJob).
|
||
`PostgresqlDumpStale` last firing sample 10:44:00Z; `DooplexGiteaOffsiteStale` 10:51:00Z.
|
||
|
||
## Undo
|
||
|
||
- CronJob: `git -C homelab-manifests revert 531023d`, then sync `CronJob/postgresql-backup` in app `database-system`.
|
||
- Routing: `git -C homelab-manifests revert 22d7697`, sync ConfigMaps `alertmanager-config-template` and
|
||
`prometheus-rules` in app `monitoring`, `curl -XPOST http://10.43.75.209:9090/-/reload`,
|
||
`sudo kubectl -n mon-system rollout restart deploy/alertmanager`.
|
||
|
||
## For the homelab session (homelab findings — no HM numbers minted here)
|
||
|
||
1. **The two-writer race on the role `postgres` stays.** The Cluster still names two secrets for one role; only the
|
||
CronJob was moved (operator's pick). If `postgresql-superuser` is rewritten while the instance runs, the role flips
|
||
back and the dump fails again (`PostgresqlDumpStale` catches it in ~12 h). The lasting fix is
|
||
`spec.superuserSecret: {name: postgres-superuser}` on the Cluster (both writers then write one value; the password
|
||
in force does not change).
|
||
2. **The routing change made:** critical 1 h → 4 h, warning 4 h → 12 h, root 4 h → 12 h, one backup group mail
|
||
(`component: backup`, `PostgresqlDumpStale` relabelled), one mail per drive (homelab `22d7697`). Applied by a
|
||
restart: Alertmanager cannot reload a ConfigMap change (`envsubst` init container).
|
||
3. **The silence for the pulled disk Z6A07P2G** (`device=~.*_Z6A07P2G`, until 2026-11-11) matches nothing now; expire it
|
||
when convenient (Felhom R-934 closed).
|
||
4. **Long-firing alarms in the 7 days:** `DiskOfflineUncorrectable` + `DiskUncorrectableSectors` on Z6A07P2G fired the
|
||
whole window (≥ 145 h, critical) — silenced by the operator since 2026-10-12, so seen, and ended by the pull. Nothing
|
||
else fired longer than 12.4 h.
|
||
5. **The window's catch-all silence** (`alertname=~.+`, 2026-10-23 12:06–14:05Z) expired as planned; nothing to do.
|