felhom.eu/documentation/audits/dooplex-alarms-2026-10-24

Two DooPlex backup alarms, and every alarm the mail route delivers — 2026-10-24 (attended)

Brief: „two DooPlex backup alarms — find why, fix, and check every other alarm the new mail route now delivers". Read first: runbooks/total-loss-of-dooplex.md, runbooks/RUNBOOK-hub-db-offsite-backup.md, audits/dooplex-survival-2026-10-22/PLAN.md, R-232, the two rules in homelab mon-system/prometheus-rules.yaml, the Alertmanager route, homelab HM-040…HM-049 and the 2026-10-23 window record.

File	What it shows
part1-2-diagnosis.txt	The off-site job's FAILED: newest complete dump … 12 h old; the dump jobs (3 failed), the failed dump folders (only an empty globals.sql); the database log's 28P01 (48 failed logins in 24 h, all from the backup pods, first 2026-10-23 16:00:01Z); which secret the role accepts (ok/refused only); the ruled-out causes (tunnel, block list, disk, overlap)
part1-2-fix.txt	After homelab 531023d: the by-hand dump (13 files, pg_restore --list 1114 / 1660 / 753 entries), the CronJob's success time, the by-hand off-site push, the restore test (26 456 files, 12 repos git fsck), the snapshot on ep0, the clear times
part3-alarms.txt, alerts-7d.json	No alarm firing at 10:51Z; 7 days of ALERTS (Prometheus' whole window); the mail counter (60 sent, 0 failed); the silences; the routing read back from the running Alertmanager
*.before	The three homelab files as they were before this session's edits
part4-*.txt	R-1250's live proof (below)

Why the two alarms fired — one cause

1. The 2026-10-23 window reboot restarted postgresql-2 (13:50Z).
2. CNPG 1.29 sets the role postgres twice at each instance start (refreshCredentialsFromSecret): first from the superuser secret postgresql-superuser, then from the bootstrap.initdb owner secret postgres-superuser (owner: postgres). The second wins.
3. The dump CronJob read postgresql-superuser → password authentication failed from 16:00Z → PostgresqlDumpStale (22:15Z).
4. The Gitea off-site push at 00:20 refused the 12 h old Gitea dump, as designed → DooplexGiteaOffsiteStale (00:57Z).

Fix (operator's pick): the CronJob reads postgres-superuser (homelab 531023d, ArgoCD sync of that one CronJob). PostgresqlDumpStale last firing sample 10:44:00Z; DooplexGiteaOffsiteStale 10:51:00Z.

Undo

• CronJob: git -C homelab-manifests revert 531023d, then sync CronJob/postgresql-backup in app database-system.
• Routing: git -C homelab-manifests revert 22d7697, sync ConfigMaps alertmanager-config-template and prometheus-rules in app monitoring, curl -XPOST http://10.43.75.209:9090/-/reload, sudo kubectl -n mon-system rollout restart deploy/alertmanager.

For the homelab session (homelab findings — no HM numbers minted here)

1. The two-writer race on the role postgres stays. The Cluster still names two secrets for one role; only the CronJob was moved (operator's pick). If postgresql-superuser is rewritten while the instance runs, the role flips back and the dump fails again (PostgresqlDumpStale catches it in ~12 h). The lasting fix is spec.superuserSecret: {name: postgres-superuser} on the Cluster (both writers then write one value; the password in force does not change).
2. The routing change made: critical 1 h → 4 h, warning 4 h → 12 h, root 4 h → 12 h, one backup group mail (component: backup, PostgresqlDumpStale relabelled), one mail per drive (homelab 22d7697). Applied by a restart: Alertmanager cannot reload a ConfigMap change (envsubst init container).
3. The silence for the pulled disk Z6A07P2G (device=~.*_Z6A07P2G, until 2026-11-11) matches nothing now; expire it when convenient (Felhom R-934 closed).
4. Long-firing alarms in the 7 days: DiskOfflineUncorrectable + DiskUncorrectableSectors on Z6A07P2G fired the whole window (≥ 145 h, critical) — silenced by the operator since 2026-10-12, so seen, and ended by the pull. Nothing else fired longer than 12.4 h.
5. The window's catch-all silence (alertname=~.+, 2026-10-23 12:06–14:05Z) expired as planned; nothing to do.