DooPlex backup alarms: one cause (the dump job's password after the reboot), fixed; alarm mail quieter; R-1250 proven
gates / gates (push) Successful in 12m24s
gates / gates (push) Successful in 12m24s
R-232 dated note + a dated check for tonight's sweep; R-934 closed (the disk was pulled 2026-10-23); R-1250 fixed on main, live proof on demo-hp with a real seed; 08 §9 item 5. Evidence audits/dooplex-alarms-2026-10-24/.
This commit is contained in:
+12
@@ -16,6 +16,18 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-24 (afternoon, attended) — two DooPlex backup alarms — DONE.** One cause: the 2026-10-23 window reboot
|
||||
> restarted `postgresql-2`; CNPG 1.29 sets the role `postgres` from the superuser secret, then from the
|
||||
> `bootstrap.initdb` owner secret (`owner: postgres`), so `postgres-superuser` is in force at every start; the dump
|
||||
> CronJob read `postgresql-superuser` → 28P01 from 16:00Z → `PostgresqlDumpStale`; the 00:20 Gitea push refused the
|
||||
> stale dump → `DooplexGiteaOffsiteStale`. **Operator rulings 2026-10-24:** (1) fix the CronJob only (homelab
|
||||
> `531023d`; the Cluster keeps two writers — homelab list); (2) apply the routing change by restarting Alertmanager
|
||||
> (its `envsubst` init container makes `/-/reload` useless; homelab `22d7697`: critical 4 h, warning 12 h, one mail per
|
||||
> backup group and per drive); (3) the R-1250 live proof may use DooPlex's k3s qBittorrent as the downloading peer
|
||||
> (one private LAN torrent in its container `/tmp`, deleted with its files). R-1250 committed (agent `29bcf44`,
|
||||
> `1e087bb`; controller `046b808`), not released. R-934 closed (the disk was pulled 2026-10-23). Catalog `7de554e`:
|
||||
> the box walk sends the drive itself as HDD_PATH (R-839). Report: `REPORT-dooplex-alarms-2026-10-24.md`.
|
||||
|
||||
> **2026-10-24 (attended release run) — DONE.** Hub 0.168.0 (05:39Z) and 0.169.0 (07:51Z) deployed; agent 0.164.0
|
||||
> released, delivered to the three test boxes and vouched by the operator; controller 0.342.0 the global floor (08:29Z,
|
||||
> MinAgent 0.158.1). ep0 runs `felhom-iperf3` (tunnel only). First line tests: Tester 1 94.3, demo-hp 94.4, demo-felhom
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
# REPORT — two DooPlex backup alarms, every alarm the mail route delivers, and R-1250 (2026-10-24, attended)
|
||||
|
||||
Brief: „two DooPlex backup alarms — find why, fix, and check every other alarm the new mail route now delivers".
|
||||
Read first: `runbooks/total-loss-of-dooplex.md`, `runbooks/RUNBOOK-hub-db-offsite-backup.md`,
|
||||
`audits/dooplex-survival-2026-10-22/PLAN.md`, R-232, R-1227, R-1241, the two homelab rules and the Alertmanager
|
||||
route, homelab HM-040…HM-049 and the window record; design `08` §9. Evidence: `audits/dooplex-alarms-2026-10-24/`.
|
||||
|
||||
## Three lines
|
||||
|
||||
1. **Both alarms had one cause, and both are fixed.** Yesterday's reboot restarted the database, and it now takes the
|
||||
password of the other of its two secrets; the dump job could not log in, and the Gitea off-site push then refused
|
||||
the old dump. The job now reads the right secret. Cleared: the dump alarm 10:44Z, the Gitea alarm 10:51Z.
|
||||
2. **Nothing else fires now.** In 7 days only the pulled disk's two alarms fired long (silenced by you since 10-12,
|
||||
ended by the pull); the rest were minutes long or are known.
|
||||
3. **Fewer mails:** critical every 4 h (was 1 h), warning every 12 h (was 4 h), all backup alarms in one mail.
|
||||
|
||||
## What I asked you, and what you said
|
||||
|
||||
- The dump fix → **the CronJob only** (the database's own definition untouched).
|
||||
- The mail change needs an Alertmanager restart, not a reload → **restart**.
|
||||
- The R-1250 proof needed a downloader on another machine → **use DooPlex's k3s qBittorrent**.
|
||||
|
||||
## The alarm table (7 days, Prometheus `ALERTS`; nothing firing at 10:51Z)
|
||||
|
||||
| Alarm | Severity | When (UTC) | True? | Cause | Fix / row |
|
||||
|---|---|---|---|---|---|
|
||||
| `DooplexGiteaOffsiteStale` | critical | 10-24 00:57 → 10:51 | true | the push refused a 12 h old dump | the dump fix; R-232 note |
|
||||
| `PostgresqlDumpStale` | warning | 10-23 22:17 → 10-24 10:44 | true | dump job login failed (28P01) | homelab `531023d` |
|
||||
| `DiskOfflineUncorrectable`, `DiskUncorrectableSectors` (Z6A07P2G) | critical | whole window (≥ 145 h) → 10-23 12:02 | true | the dying HDD4 | silenced by you 10-12; pulled 10-23 — R-934 closed |
|
||||
| `DiskDisappeared` (Z6A07P2G) | critical | 10-23 12:17 → 18:02 | true | the pull | expected; cleared |
|
||||
| `DiskInventoryChanged` | warning | 10-23 12:17 → 20:17 | true | the pull | expected; cleared |
|
||||
| `LonghornVolumeDegraded` (two Plex volumes) | warning | 10-23 08:22 → 11:22 | true | the HM-040 rebuild | cleared |
|
||||
| `PodRestartingFrequently` (felhom-webpage) | warning | 10-19 09:02 → 10:07 | true | git-syncs stopped on errors | R-1163 (closed 10-19) |
|
||||
| `WanUploadDegraded`, `WanDownloadDegraded`, `WanUpstreamPacketLoss`, `WanBufferbloat` | warning/info | minutes, 10-17 → 10-20 | true | short WAN dips (the flood days) | none needed |
|
||||
| `AuthentikHighLatency` | warning | 10-21 08:12, 5 min | true | a short spike | none needed |
|
||||
|
||||
## The mail change (homelab `22d7697`, applied by a restart, read back from the running Alertmanager)
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| critical | every 1 h | every 4 h |
|
||||
| warning | every 4 h | every 12 h |
|
||||
| other (info) | every 4 h | every 12 h |
|
||||
| backup alarms | one mail stream per alarm | one mail for the group, every 4 h |
|
||||
| a disk | one stream per alarm | one mail per drive, daily (unchanged) |
|
||||
| resolved mail | `send_resolved: true` | unchanged; the mail counter rose once after each clear (10:47, and by 10:57) — that they read „RESOLVED" is not seen: the mailbox is not mine |
|
||||
|
||||
## Part 4 — R-1250, committed and proven
|
||||
|
||||
Agent `29bcf44` + `1e087bb`, controller `046b808` (commit only; release in the next train). A seeding box: the agent
|
||||
says „waiting for quiet"; the controller records the torrent app's limit, holds it at 1 KiB/s, asks for one test,
|
||||
puts the exact limit back. **Proven on demo-hp with a real seed** (10 Mbit/s from 9202 to DooPlex's qBittorrent):
|
||||
refused without the hold (11.1 Mbit/s); with it the uplink fell to 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, the limit
|
||||
back to 1 249 280 B/s; a kill in the middle → the start-up check put it back. The brief said „a nested VM": NOT done —
|
||||
a nested VM would need its own tunnel to ep0 to run the real test; demo-hp's host with probes did the same with the
|
||||
real tunnel. The pinned agent client is unit-tested only.
|
||||
|
||||
## Commits, CI, undo
|
||||
|
||||
| Repo | Commit | What | CI |
|
||||
|---|---|---|---|
|
||||
| homelab-manifests | `531023d` | the dump CronJob's secret | no CI in this repo |
|
||||
| homelab-manifests | `22d7697` | the mail routes, the dump alarm's label | no CI |
|
||||
| homelab-manifests | `9f399fe` | REPORT | no CI |
|
||||
| felhom-agent | `29bcf44`, `1e087bb` | R-1250 | 2286, 2288 success |
|
||||
| felhom-controller | `046b808` | R-1250 | 2287 success |
|
||||
| app-catalog-felhom.eu | `7de554e`, `5e196e1` | the box walk: HDD_PATH, the leftover check | 2289 success, (final push) |
|
||||
| felhom-agent / felhom-controller | `5e05627` / `321df1f` | REPORT, README | (final push) |
|
||||
| felhom.eu | this commit | register, STATUS, CONTEXT, `08` §9, evidence | (final push) |
|
||||
|
||||
Undo: `audits/dooplex-alarms-2026-10-24/README.md` § Undo. Backups of the three homelab files: `*.before` there.
|
||||
|
||||
## Rows
|
||||
|
||||
**Before 191, after 190; opened 0, closed 1 (R-934 — the disk was already pulled).** R-232: a dated note and a dated
|
||||
check (2026-10-25, tonight's sweep). R-1250: NARROWED (fixed on main, waits for the release train).
|
||||
|
||||
## Observations
|
||||
|
||||
- The database's two secrets for one role stay (only the CronJob moved) — homelab list item 1 (not a Felhom row).
|
||||
- A stale dump stops the Gitea + secrets off-site push for as long as it lasts — NOT-A-FINDING: the designed fail-closed
|
||||
order (`PLAN.md`: the dump first, for consistency); the alarm caught it inside its 26 h, and it now comes in one mail
|
||||
with its cause.
|
||||
- Alertmanager cannot reload a ConfigMap change (the brief assumed it could) — NOT-A-FINDING: the homelab manifest's
|
||||
design (`envsubst` init container); you chose the restart; the fact is in CC's memory and the homelab list.
|
||||
- The agent said „waiting for quiet" before it had watched a minute — fixed in the session (`1e087bb`, found live).
|
||||
- The box walk sent a folder inside the drive as HDD_PATH (refused since R-839) — fixed in the session (`7de554e`).
|
||||
- The box walk called the template folder a leftover after every remove — fixed in the session (`5e196e1`).
|
||||
- R-934 said the failed disk was still in DooPlex; STATUS still listed R-1214's done check — fixed (row closed, line removed).
|
||||
- CC's memory said Prometheus has no config-reloader; it has one since 2026-10-12 — fixed (memory note).
|
||||
- `govulncheck` warns GO-2026-6809 (crypto/x509, stdlib) in the agent and the controller — NOT-A-FINDING here: the Go
|
||||
toolchain row R-972 owns it.
|
||||
- My Gmail connector's mailbox gets no Alertmanager mail — NOT-A-FINDING: it is another mailbox than `SMTP_TO`; 60 sent,
|
||||
0 failed in 7 days. The Alertmanager config did not change yesterday (last 10-12).
|
||||
- `kubectl exec -i` never closed stdin for `pg_restore` — NOT-A-FINDING: my method; the host's own `pg_restore` read the dumps.
|
||||
- The 9202 ISO torrent never uploaded (behind NAT, 18 leechers / 1080 seeds) — NOT-A-FINDING: why the LAN peer was used.
|
||||
- I overwrote homelab-manifests' REPORT.md, which held another session's HM-050 report from today — NOT-A-FINDING: the
|
||||
repo's overwrite rule; HM-050's entry stays in its CHANGELOG.
|
||||
|
||||
## Teardown, three layers
|
||||
|
||||
- **Machine:** 9202 — qBittorrent removed through the product (no container, no volume; the template folder stays, as for
|
||||
every app); the test files, the forwarder and the probe removed. DooPlex's qBittorrent — the test torrent and its
|
||||
files deleted (by hash), nothing else touched.
|
||||
- **Host:** demo-hp — the agent probe stopped and deleted, no iperf3 left; the installed agent untouched. DooPlex — the
|
||||
9202 password copy and the helper's session files shredded; the clones removed.
|
||||
- **Hub:** nothing provisioned.
|
||||
|
||||
## Instruction files, poster
|
||||
|
||||
No instruction file edited. No poster fact changed (the mail intervals are not on it).
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
|
||||
|
||||
**Updated 2026-10-24 (day, attended, done).** Released on your yeses: hub 0.168.0 (tokens locked) and 0.169.0, agent 0.164.0 (vouched), controller 0.342.0 (the floor). Each box tests its upload weekly (94, 94, 93 Mbit/s); a hacked app that uploads or computes more than it explains alarms only you, and qBittorrent gets half the tested line. The hub is out of DooPlex's open backup folder.
|
||||
**Updated 2026-10-24 (afternoon, attended).** The two DooPlex backup alarms had one cause: yesterday's reboot changed which password the database takes, so the database copy job could not log in, and the Gitea off-site copy then refused its old copy. Fixed and cleared at 12:51 local; critical alarms now repeat every 4 hours (all backup alarms in one mail), warnings every 12. A box whose torrent app seeds all day can now get its weekly line test (proven on demo-hp, not yet released).
|
||||
|
||||
## What works
|
||||
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
|
||||
@@ -20,10 +20,10 @@
|
||||
## What is broken
|
||||
- **P1:** any hacked app on DooPlex can read the hub's nightly backups, and maybe the signing keys (R-1241, homelab session); about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
|
||||
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). The scratch box cannot use agent features (R-1083).
|
||||
- DooPlex: the failed disk is still in the box, its alarms silenced (R-934); something sets the shared checkout to „bare" (R-935).
|
||||
- DooPlex: something sets the shared checkout to „bare" (R-935).
|
||||
|
||||
## What we work on
|
||||
- Dated checks, run by CC. 2026-10-24: the 8 GB box's first night copy on 0.340.0 (R-1214). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes).
|
||||
- Dated checks, run by CC. 2026-10-25: tonight's database copy reaches the restic sweep (R-232). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes).
|
||||
|
||||
## Waiting on you — decide
|
||||
|
||||
@@ -51,10 +51,9 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
|
||||
2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef. If not: two dead entries alarm.
|
||||
3. D37 and D38 in the table (two small yes/no). If not: Gitea may restart in release hours; no golden in the release copy.
|
||||
4. Give the homelab session R-1241, R-1242, the 7-item list at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md`, and an upload alarm on DooPlex's `eno1` (the line is ~100 Mbit/s; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups.
|
||||
4. Give the homelab session R-1241, R-1242, the 7-item list at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md`, the 5-item list at the end of `documentation/audits/dooplex-alarms-2026-10-24/README.md` (the database's two passwords), and an upload alarm on DooPlex's `eno1` (the line is ~100 Mbit/s; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups.
|
||||
**The rest**
|
||||
- System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy.
|
||||
- Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12.
|
||||
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
|
||||
- Two DooPlex leftovers: `leftovers/2026-10-14/first-post/` (keep or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). Since 2026-10-14.
|
||||
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784).
|
||||
|
||||
@@ -564,9 +564,17 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me
|
||||
household is told only about the qBittorrent limit, because they can change it (R-1246 asks the operator).
|
||||
5. **qBittorrent's default upload limit:** 50 % of the TESTED line (D41 = 50 %, operator 2026-10-24; never of the
|
||||
push floor), every 15 minutes, re-set when the line moves > 20 %; never over a limit the household set (one there
|
||||
first, or any change from the value qBittorrent stored for the box — it keeps KiB/s). No test → none set. **Known
|
||||
gap (R-1250):** a qBittorrent that seeds all day keeps the uplink from ever being quiet for 10 minutes, so that box is
|
||||
never tested and gets no limit — the case the limit exists for. While the box owns it, qBittorrent's page says: „A qBittorrent
|
||||
first, or any change from the value qBittorrent stored for the box — it keeps KiB/s). No test → none set. **A box
|
||||
that seeds all day (R-1250, brief 2026-10-24; committed agent `29bcf44`+`1e087bb`, controller `046b808`, not yet
|
||||
released):** the 10 quiet minutes never come, so when the agent reports `waiting_for_quiet` (due, in the window,
|
||||
tunnel up, the full 10-minute window watched and busy) and qBittorrent or Transmission runs, the controller writes
|
||||
`upwatch.line_test_hold` (the app's limit exactly as stored) FIRST, holds the app at 1 KiB/s (Transmission 1 kB/s),
|
||||
calls `POST /host/line-test/run` (5 s settle, 10 s quiet check < 2 Mbit/s, three runs 5 s apart — about 55 s), then
|
||||
puts the old limit back, the household's own included; a change made during the test stays; a killed controller's
|
||||
start-up check puts it back; the 50 % rule waits while a record exists. Measured on demo-hp with a real seed (10
|
||||
Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; with it the uplink fell to 0.02–0.16 Mbit/s,
|
||||
94.2 Mbit/s measured, the app held 56.8 s and back to 1 249 280 B/s exactly (`audits/dooplex-alarms-2026-10-24/`).
|
||||
While the box owns it, qBittorrent's page says: „A qBittorrent
|
||||
feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
|
||||
6. **Limits, stated:** an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
|
||||
a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
# Two DooPlex backup alarms, and every alarm the mail route delivers — 2026-10-24 (attended)
|
||||
|
||||
Brief: „two DooPlex backup alarms — find why, fix, and check every other alarm the new mail route now delivers".
|
||||
Read first: `runbooks/total-loss-of-dooplex.md`, `runbooks/RUNBOOK-hub-db-offsite-backup.md`,
|
||||
`audits/dooplex-survival-2026-10-22/PLAN.md`, R-232, the two rules in homelab `mon-system/prometheus-rules.yaml`, the
|
||||
Alertmanager route, homelab HM-040…HM-049 and the 2026-10-23 window record.
|
||||
|
||||
| File | What it shows |
|
||||
|---|---|
|
||||
| `part1-2-diagnosis.txt` | The off-site job's `FAILED: newest complete dump … 12 h old`; the dump jobs (3 failed), the failed dump folders (only an empty `globals.sql`); the database log's `28P01` (48 failed logins in 24 h, all from the backup pods, first 2026-10-23 16:00:01Z); which secret the role accepts (ok/refused only); the ruled-out causes (tunnel, block list, disk, overlap) |
|
||||
| `part1-2-fix.txt` | After homelab `531023d`: the by-hand dump (13 files, `pg_restore --list` 1114 / 1660 / 753 entries), the CronJob's success time, the by-hand off-site push, the restore test (26 456 files, 12 repos `git fsck`), the snapshot on ep0, the clear times |
|
||||
| `part3-alarms.txt`, `alerts-7d.json` | No alarm firing at 10:51Z; 7 days of `ALERTS` (Prometheus' whole window); the mail counter (60 sent, 0 failed); the silences; the routing read back from the running Alertmanager |
|
||||
| `*.before` | The three homelab files as they were before this session's edits |
|
||||
| `part4-*.txt` | R-1250's live proof (below) |
|
||||
|
||||
## Why the two alarms fired — one cause
|
||||
|
||||
1. The 2026-10-23 window reboot restarted `postgresql-2` (13:50Z).
|
||||
2. CNPG 1.29 sets the role `postgres` twice at each instance start (`refreshCredentialsFromSecret`): first from the
|
||||
superuser secret `postgresql-superuser`, then from the `bootstrap.initdb` owner secret `postgres-superuser`
|
||||
(`owner: postgres`). The second wins.
|
||||
3. The dump CronJob read `postgresql-superuser` → `password authentication failed` from 16:00Z → `PostgresqlDumpStale`
|
||||
(22:15Z).
|
||||
4. The Gitea off-site push at 00:20 refused the 12 h old Gitea dump, as designed → `DooplexGiteaOffsiteStale` (00:57Z).
|
||||
|
||||
Fix (operator's pick): the CronJob reads `postgres-superuser` (homelab `531023d`, ArgoCD sync of that one CronJob).
|
||||
`PostgresqlDumpStale` last firing sample 10:44:00Z; `DooplexGiteaOffsiteStale` 10:51:00Z.
|
||||
|
||||
## Undo
|
||||
|
||||
- CronJob: `git -C homelab-manifests revert 531023d`, then sync `CronJob/postgresql-backup` in app `database-system`.
|
||||
- Routing: `git -C homelab-manifests revert 22d7697`, sync ConfigMaps `alertmanager-config-template` and
|
||||
`prometheus-rules` in app `monitoring`, `curl -XPOST http://10.43.75.209:9090/-/reload`,
|
||||
`sudo kubectl -n mon-system rollout restart deploy/alertmanager`.
|
||||
|
||||
## For the homelab session (homelab findings — no HM numbers minted here)
|
||||
|
||||
1. **The two-writer race on the role `postgres` stays.** The Cluster still names two secrets for one role; only the
|
||||
CronJob was moved (operator's pick). If `postgresql-superuser` is rewritten while the instance runs, the role flips
|
||||
back and the dump fails again (`PostgresqlDumpStale` catches it in ~12 h). The lasting fix is
|
||||
`spec.superuserSecret: {name: postgres-superuser}` on the Cluster (both writers then write one value; the password
|
||||
in force does not change).
|
||||
2. **The routing change made:** critical 1 h → 4 h, warning 4 h → 12 h, root 4 h → 12 h, one backup group mail
|
||||
(`component: backup`, `PostgresqlDumpStale` relabelled), one mail per drive (homelab `22d7697`). Applied by a
|
||||
restart: Alertmanager cannot reload a ConfigMap change (`envsubst` init container).
|
||||
3. **The silence for the pulled disk Z6A07P2G** (`device=~.*_Z6A07P2G`, until 2026-11-11) matches nothing now; expire it
|
||||
when convenient (Felhom R-934 closed).
|
||||
4. **Long-firing alarms in the 7 days:** `DiskOfflineUncorrectable` + `DiskUncorrectableSectors` on Z6A07P2G fired the
|
||||
whole window (≥ 145 h, critical) — silenced by the operator since 2026-10-12, so seen, and ended by the pull. Nothing
|
||||
else fired longer than 12.4 h.
|
||||
5. **The window's catch-all silence** (`alertname=~.+`, 2026-10-23 12:06–14:05Z) expired as planned; nothing to do.
|
||||
@@ -0,0 +1,293 @@
|
||||
---
|
||||
# ============================================================================
|
||||
# alertmanager — alert routing and email delivery
|
||||
#
|
||||
# Brought into this repo on 2026-08-14. Until then it ran live with NO manifest
|
||||
# anywhere: CLAUDE.md listed it as one of three such workloads, which meant any
|
||||
# change to its routing survived only until something recreated it, and nobody
|
||||
# could tell what the running config was without reading the cluster.
|
||||
#
|
||||
# Credentials are NOT here and must never be. smtp_from / auth username /
|
||||
# password / recipient are ${VAR} placeholders resolved at pod start by the
|
||||
# envsubst init container from the alertmanager-smtp Secret, which is created
|
||||
# imperatively. Only the smarthost (smtp.gmail.com:587) and require_tls are
|
||||
# literal, and neither is sensitive.
|
||||
#
|
||||
# Two deliberate fixes applied while adopting, both flagged rather than silent:
|
||||
# * envsubst was running :latest. Pinned to the digest that was actually
|
||||
# running, so a restart can no longer silently change the code.
|
||||
# * the init container had no resource requests/limits, against house style.
|
||||
# ============================================================================
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: alertmanager-config-template
|
||||
namespace: mon-system
|
||||
labels:
|
||||
app: alertmanager
|
||||
data:
|
||||
alertmanager.yml.tmpl: |
|
||||
global:
|
||||
smtp_smarthost: 'smtp.gmail.com:587'
|
||||
smtp_from: '${SMTP_AUTH_USERNAME}'
|
||||
smtp_auth_username: '${SMTP_AUTH_USERNAME}'
|
||||
smtp_auth_password: '${SMTP_AUTH_PASSWORD}'
|
||||
smtp_require_tls: true
|
||||
resolve_timeout: 5m
|
||||
|
||||
route:
|
||||
receiver: 'email-notifications'
|
||||
group_by: ['alertname', 'severity']
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 4h
|
||||
|
||||
routes:
|
||||
# --- disk health -------------------------------------------------------
|
||||
# A failing drive does not improve between emails. The generic
|
||||
# severity: critical route below repeats hourly, which produced ~24 mails
|
||||
# a day for a single dying disk on 2026-08-13/14. Once you have seen it,
|
||||
# a daily reminder is enough.
|
||||
#
|
||||
# The two *Growing alerts are deliberately kept frequent: they only fire
|
||||
# while the SMART counters are actively climbing, i.e. something is
|
||||
# degrading RIGHT NOW rather than sitting in a known-broken state.
|
||||
- match:
|
||||
alertname: DiskReportedUncorrectGrowing
|
||||
receiver: 'email-notifications'
|
||||
repeat_interval: 4h
|
||||
- match:
|
||||
alertname: DiskReallocatedSectorsGrowing
|
||||
receiver: 'email-notifications'
|
||||
repeat_interval: 4h
|
||||
- match:
|
||||
component: disk
|
||||
receiver: 'email-notifications'
|
||||
repeat_interval: 24h
|
||||
# --- workload health ---------------------------------------------------
|
||||
# Same reasoning as the disk route: a crashlooping pod does not fix
|
||||
# itself, so the generic critical route's hourly repeat would have sent
|
||||
# 27 mails for the single external-dns-pihole failure that motivated
|
||||
# these alerts. 6h still reaches you four times a day, which is far more
|
||||
# than the zero notifications that outage actually produced.
|
||||
#
|
||||
# Must stay ABOVE the severity: critical route -- Alertmanager takes the
|
||||
# first matching route, so ordering is what makes this override apply.
|
||||
- match:
|
||||
component: workload
|
||||
receiver: 'email-notifications'
|
||||
repeat_interval: 6h
|
||||
- match:
|
||||
severity: critical
|
||||
receiver: 'email-notifications'
|
||||
repeat_interval: 1h
|
||||
- match:
|
||||
severity: warning
|
||||
receiver: 'email-notifications'
|
||||
repeat_interval: 4h
|
||||
|
||||
receivers:
|
||||
- name: 'email-notifications'
|
||||
email_configs:
|
||||
- to: '${SMTP_TO}'
|
||||
send_resolved: true
|
||||
headers:
|
||||
Subject: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}'
|
||||
html: |
|
||||
{{ range .Alerts }}
|
||||
<h2>{{ .Status | toUpper }}: {{ .Labels.alertname }}</h2>
|
||||
<p><strong>Severity:</strong> {{ .Labels.severity }}</p>
|
||||
<p><strong>Summary:</strong> {{ .Annotations.summary }}</p>
|
||||
<p><strong>Description:</strong> {{ .Annotations.description }}</p>
|
||||
<p><strong>Started:</strong> {{ .StartsAt.Format "2006-01-02 15:04:05" }}</p>
|
||||
{{ if .EndsAt }}
|
||||
<p><strong>Ended:</strong> {{ .EndsAt.Format "2006-01-02 15:04:05" }}</p>
|
||||
{{ end }}
|
||||
<hr>
|
||||
{{ end }}
|
||||
|
||||
inhibit_rules:
|
||||
- source_match:
|
||||
severity: 'critical'
|
||||
target_match:
|
||||
severity: 'warning'
|
||||
equal: ['alertname', 'volume']
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: alertmanager-data
|
||||
namespace: mon-system
|
||||
labels:
|
||||
app: alertmanager
|
||||
# Explicit membership rather than relying on implicit default-group
|
||||
# inclusion, which any stray recurring-job label would silently break.
|
||||
recurring-job-group.longhorn.io/default: enabled
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
# Explicit, not the cluster default: longhorn-ssd2 is the default but is the
|
||||
# smallest disk on the box.
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: alertmanager
|
||||
namespace: mon-system
|
||||
labels:
|
||||
app: alertmanager
|
||||
spec:
|
||||
replicas: 1
|
||||
# Recreate, not RollingUpdate: the data volume is ReadWriteOnce, so a rolling
|
||||
# update would deadlock with the new pod waiting for a volume the old pod
|
||||
# still holds.
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app: alertmanager
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: alertmanager
|
||||
spec:
|
||||
# R-886 (operator yes 2026-10-12, felhom 09 §3 decision 217). The image runs as nobody (65534)
|
||||
# and the Longhorn volume's root is root:root 0755, so since the 2026-10-05 Longhorn restart
|
||||
# every nflog/silences write failed with "permission denied" and silences died at each restart.
|
||||
# fsGroup makes the kubelet give the volume to group 65534 (group-writable) at mount.
|
||||
securityContext:
|
||||
fsGroup: 65534
|
||||
initContainers:
|
||||
# Renders the template into the real config, substituting the SMTP
|
||||
# credentials from the Secret. The processed file lands in an emptyDir,
|
||||
# which is why a ConfigMap edit needs a POD RESTART to take effect --
|
||||
# POST /-/reload would just re-read the already-rendered file.
|
||||
- name: envsubst
|
||||
image: bhgedigital/envsubst@sha256:36a92e9344ef965e9318c8d2f70dd35cad654a4615b889c6a4dd483ca1ddec7c
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- envsubst < /template/alertmanager.yml.tmpl > /config/alertmanager.yml
|
||||
env:
|
||||
- name: SMTP_AUTH_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: alertmanager-smtp
|
||||
key: SMTP_AUTH_USERNAME
|
||||
- name: SMTP_AUTH_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: alertmanager-smtp
|
||||
key: SMTP_AUTH_PASSWORD
|
||||
- name: SMTP_TO
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: alertmanager-smtp
|
||||
key: SMTP_TO
|
||||
volumeMounts:
|
||||
- name: config-template
|
||||
mountPath: /template
|
||||
- name: config-processed
|
||||
mountPath: /config
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
containers:
|
||||
- name: alertmanager
|
||||
image: prom/alertmanager:v0.34.1
|
||||
args:
|
||||
- --config.file=/etc/alertmanager/alertmanager.yml
|
||||
- --storage.path=/alertmanager
|
||||
- --web.external-url=http://alertmanager.home
|
||||
- --cluster.listen-address=
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 9093
|
||||
protocol: TCP
|
||||
volumeMounts:
|
||||
- name: config-processed
|
||||
mountPath: /etc/alertmanager
|
||||
- name: storage
|
||||
mountPath: /alertmanager
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /-/healthy
|
||||
port: 9093
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /-/ready
|
||||
port: 9093
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
volumes:
|
||||
- name: config-template
|
||||
configMap:
|
||||
name: alertmanager-config-template
|
||||
- name: config-processed
|
||||
emptyDir: {}
|
||||
# Real storage, not emptyDir. This was an emptyDir until 2026-08-14,
|
||||
# which meant SILENCES DID NOT SURVIVE A RESTART -- they live in this
|
||||
# volume and nothing rebuilds them. Restarting alertmanager to pick up
|
||||
# a config change therefore silently un-silenced everything and
|
||||
# re-notified every firing alert, which is exactly what happened when
|
||||
# the disk-alert routing was deployed.
|
||||
- name: storage
|
||||
persistentVolumeClaim:
|
||||
claimName: alertmanager-data
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: alertmanager
|
||||
namespace: mon-system
|
||||
labels:
|
||||
app: alertmanager
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: alertmanager
|
||||
ports:
|
||||
- name: http
|
||||
port: 9093
|
||||
targetPort: 9093
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: alertmanager
|
||||
namespace: mon-system
|
||||
labels:
|
||||
app: alertmanager
|
||||
annotations:
|
||||
external-dns.alpha.kubernetes.io/hostname: alertmanager.home
|
||||
spec:
|
||||
ingressClassName: nginx-internal
|
||||
rules:
|
||||
- host: alertmanager.home
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: alertmanager
|
||||
port:
|
||||
number: 9093
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,51 @@
|
||||
# Part 1+2 diagnosis, read 2026-10-24T10:39:09Z
|
||||
|
||||
## felhom-dooplex-offsite journal 2026-10-24
|
||||
Oct 24 00:20:48 dooplex systemd[1]: Starting felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232)...
|
||||
Oct 24 00:20:48 dooplex felhom-dooplex-offsite[1455158]: felhom-dooplex-offsite: FAILED: newest complete dump 20261023-100001 is 12 h old (limit 7 h) — the dump CronJob stopped
|
||||
Oct 24 00:20:48 dooplex systemd[1]: felhom-dooplex-offsite.service: Main process exited, code=exited, status=1/FAILURE
|
||||
Oct 24 00:20:48 dooplex systemd[1]: felhom-dooplex-offsite.service: Failed with result 'exit-code'.
|
||||
Oct 24 00:20:48 dooplex systemd[1]: Failed to start felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232).
|
||||
Oct 24 00:20:48 dooplex systemd[1]: felhom-dooplex-offsite.service: Triggering OnFailure= dependencies.
|
||||
|
||||
## postgresql-backup jobs
|
||||
NAME STATUS COMPLETIONS DURATION AGE
|
||||
postgresql-backup-29878440 Complete 1/1 17s 36h
|
||||
postgresql-backup-29878800 Complete 1/1 17s 30h
|
||||
postgresql-backup-29879160 Complete 1/1 13s 24h
|
||||
postgresql-backup-29879880 Failed 0/1 12h 12h
|
||||
postgresql-backup-29880240 Failed 0/1 6h39m 6h39m
|
||||
postgresql-backup-29880600 Failed 0/1 39m 39m
|
||||
|
||||
## CronJob status
|
||||
lastSchedule=2026-10-24T10:00:00Z lastSuccess=2026-10-23T10:00:13Z
|
||||
|
||||
## failed dump folders (globals.sql only, 0 bytes)
|
||||
total 8
|
||||
drwxr-xr-x 2 26 tape 4096 Oct 23 18:00 .
|
||||
drwxr-xr-x 29 26 tape 4096 Oct 24 12:00 ..
|
||||
-rw-r--r-- 1 26 tape 0 Oct 23 18:00 globals.sql
|
||||
total 8
|
||||
drwxr-xr-x 2 26 tape 4096 Oct 24 12:00 .
|
||||
drwxr-xr-x 29 26 tape 4096 Oct 24 12:00 ..
|
||||
-rw-r--r-- 1 26 tape 0 Oct 24 12:00 globals.sql
|
||||
|
||||
## database log: failed logins in 24 h (user, client, count) — first 2026-10-23 16:00:01 UTC; all four clients are backup job pods
|
||||
message: password authentication failed for user "postgres" (sql_state 28P01)
|
||||
|
||||
## which secret's password does role postgres accept (scratchpad pgpw-check.sh; prints ok/refused only)
|
||||
postgresql-superuser: refused (the CNPG superuser secret; the CronJob reads this)
|
||||
postgres-superuser: ok (the bootstrap.initdb secret; owner=postgres)
|
||||
|
||||
## cluster status.secretsResourceVersion
|
||||
{"applicationSecretVersion":"4427397","clientCaSecretVersion":"80646921","replicationSecretVersion":"80646924","serverCaSecretVersion":"80646921","serverSecretVersion":"80646922","superuserSecretVersion":"59321453"}
|
||||
|
||||
## instance start: postgresql-2 instance manager up 2026-10-23T13:50:14Z (window reboot); last good dump 10:00Z, first bad 16:00Z
|
||||
|
||||
## ruled out
|
||||
tunnel: active since 2026-10-23 13:41:29Z (one 'No route to host' at boot, restart after 31 s); PBS via 127.0.0.1:18007 -> 200
|
||||
nft inet hm040_blocklist: ioc_ipv4 = { 5.255.109.74, 5.255.115.190 } only; ep0 167.233.158.164 not in it
|
||||
disk: Filesystem Size Used Avail Use% Mounted on
|
||||
disk: /dev/sda1 9.1T 3.2T 5.4T 38% /mnt/5_hdd
|
||||
disk: /dev/sdb1 445G 239G 184G 57% /
|
||||
overlap: the job died at 00:20:48 in 0 s on the dump-age check, before any network use; release copy 01:40, hub copy 02:30 (UTC+2) do not overlap
|
||||
@@ -0,0 +1,52 @@
|
||||
# Fix and proof, 2026-10-24T10:58:25Z
|
||||
|
||||
## homelab-manifests 531023d: CronJob reads postgres-superuser; ArgoCD sync of CronJob/postgresql-backup only
|
||||
env after sync: PGUSER=postgres-superuser PGPASSWORD=postgres-superuser
|
||||
|
||||
## by-hand job postgresql-backup-manual-1043 (kubectl create job --from=cronjob/postgresql-backup)
|
||||
total 22064
|
||||
drwxr-xr-x 2 26 tape 4096 Oct 24 12:43 .
|
||||
drwxr-xr-x 25 26 tape 4096 Oct 24 12:43 ..
|
||||
-rw-r--r-- 1 26 tape 5297319 Oct 24 12:43 authentik.dump
|
||||
-rw-r--r-- 1 26 tape 711473 Oct 24 12:43 calcom.dump
|
||||
-rw-r--r-- 1 26 tape 10258 Oct 24 12:43 calcom_saml.dump
|
||||
-rw-r--r-- 1 26 tape 8598648 Oct 24 12:43 gitea.dump
|
||||
-rw-r--r-- 1 26 tape 4016 Oct 24 12:43 globals.sql
|
||||
-rw-r--r-- 1 26 tape 90743 Oct 24 12:43 guacamole_db.dump
|
||||
-rw-r--r-- 1 26 tape 1367243 Oct 24 12:43 homarr.dump
|
||||
-rw-r--r-- 1 26 tape 3624770 Oct 24 12:43 outline.dump
|
||||
-rw-r--r-- 1 26 tape 1224943 Oct 24 12:43 paperless.dump
|
||||
-rw-r--r-- 1 26 tape 2754 Oct 24 12:43 postgres.dump
|
||||
-rw-r--r-- 1 26 tape 0 Oct 24 12:43 SUCCESS
|
||||
-rw-r--r-- 1 26 tape 594915 Oct 24 12:43 tandoor.dump
|
||||
-rw-r--r-- 1 26 tape 648505 Oct 24 12:43 wger.dump
|
||||
-rw-r--r-- 1 26 tape 382029 Oct 24 12:43 zipline.dump
|
||||
gitea.dump pg_restore --list rc=0 entries=1114
|
||||
authentik.dump pg_restore --list rc=0 entries=1660
|
||||
paperless.dump pg_restore --list rc=0 entries=753
|
||||
CronJob lastSuccessfulTime: 2026-10-24T10:43:55Z
|
||||
|
||||
## felhom-dooplex-offsite by hand + restore test
|
||||
Oct 24 12:48:40 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: database: 20261024-104341, 8598648 bytes, 4 min old
|
||||
Oct 24 12:49:30 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: files: 12 repositories, 813 MB
|
||||
Oct 24 12:49:30 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: secrets: 3 file(s) of 20261024_031421
|
||||
Oct 24 12:49:32 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: vaultwarden: integrity ok, 1 user(s), files: db.sqlite3 rsa_key.pem
|
||||
Oct 24 12:49:32 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: signing: 3 key(s) with their public halves
|
||||
Oct 24 12:50:20 dooplex felhom-dooplex-offsite[509023]: dooplex.pxar: had to backup 422.032 MiB of 750.918 MiB (compressed 413.468 MiB) in 43.59 s (average 9.681 MiB/s)
|
||||
Oct 24 12:50:20 dooplex felhom-dooplex-offsite[509023]: dooplex.pxar: backup was done incrementally, reused 328.886 MiB (43.8%)
|
||||
Oct 24 12:50:22 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: pushed to ep0 (ns operator, host/dooplex-gitea) in 46 s
|
||||
Oct 24 12:50:22 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: success signal written
|
||||
Oct 24 12:50:23 dooplex systemd[1]: Finished felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232).
|
||||
Oct 24 12:50:40 dooplex felhom-dooplex-offsite-restore-test[521926]: felhom-dooplex-offsite-restore-test: restoring host/dooplex-gitea/2026-10-24T10:49:36Z
|
||||
Oct 24 12:51:07 dooplex felhom-dooplex-offsite-restore-test[522066]: restore complete (750.918 MiB processed in 27.1s, average 27.745 MiB/s)
|
||||
Oct 24 12:51:33 dooplex felhom-dooplex-offsite-restore-test[521926]: felhom-dooplex-offsite-restore-test: checked: 26456 files match the manifest, 12 repositories pass git fsck, gitea.dump readable, 3 secrets file(s), Vaultwarden 1 user(s) / 811 item(s), 3 signing key(s) match
|
||||
Oct 24 12:51:33 dooplex felhom-dooplex-offsite-restore-test[521926]: felhom-dooplex-offsite-restore-test: success signal written
|
||||
Oct 24 12:51:34 dooplex systemd[1]: Finished felhom-dooplex-offsite-restore-test.service - Felhom: restore-test the Gitea + secrets copy on ep0 (R-232).
|
||||
|
||||
## ep0 (read only): host/dooplex-gitea snapshot 2026-10-24T10:49:36Z present (catalog.pcat1.didx, dooplex.pxar.didx, index.json.blob, 10:50Z)
|
||||
|
||||
## alarms cleared (Prometheus ALERTS, 30 s step)
|
||||
PostgresqlDumpStale last firing sample 10:44:00Z; DooplexGiteaOffsiteStale last firing sample 10:51:00Z
|
||||
metric ages at 10:51Z: offsite 83 s, dump 470 s; Alertmanager /api/v2/alerts = [] at 10:51:52Z
|
||||
|
||||
## the restic sweep: dooplex-backup.timer 03:14 local; 2026-10-24 03:15:48 snapshot a801930d of /mnt/5_hdd/backup/postgresql/dumps ok; not run by hand (apply_retention would forget today's snapshot); the next run is the proof (dated check)
|
||||
@@ -0,0 +1,21 @@
|
||||
# Part 3: alarms, 2026-10-24T10:58:26Z
|
||||
firing now: Prometheus /api/v1/alerts none; Alertmanager /api/v2/alerts []
|
||||
|
||||
## ALERTS{alertstate=firing} 7 days (Prometheus range start = retention window), episodes
|
||||
|
||||
## Alertmanager mail counter (old pod): 60 sent in 7 d, 20 in 24 h, 0 failed (all reasons)
|
||||
## Gmail connector mailbox (felhom.eu catch-all): 0 alarm mails in 3 d — SMTP_TO is another mailbox; delivery seen by the operator only
|
||||
## silences
|
||||
active 2026-10-12T14:32 -> 2026-11-11T14:32 cc (operator yes 2026-10-12) | disk Z6A07P2G evacuated 2026-10-10, waiting for physical removal
|
||||
expired 2026-10-23T12:06 -> 2026-10-23T14:05 cc (operator OK 2026-10-23) | DooPlex maintenance window 2026-10-23 (HM-043)
|
||||
|
||||
## routing after homelab 22d7697 + pod restart (read from /api/v2/status)
|
||||
root repeat 12h group_by ['alertname', 'severity']
|
||||
{'component': 'backup'} ['component'] 4h
|
||||
{'alertname': 'DiskReportedUncorrectGrowing'} 4h
|
||||
{'alertname': 'DiskReallocatedSectorsGrowing'} 4h
|
||||
{'component': 'disk'} ['component', 'device'] 1d
|
||||
{'component': 'workload'} 6h
|
||||
{'severity': 'critical'} 4h
|
||||
{'severity': 'warning'} 12h
|
||||
send_resolved True uptime 2026-10-24T10:56:44.705Z
|
||||
@@ -0,0 +1,59 @@
|
||||
# R-1250 live proof, demo-hp, 2026-10-24 — probes built from agent 1e087bb and controller 046b808 (clean clones; not committed probes: the agent's linetest.Tester behind the two routes as plain HTTP on the host; the controller's upwatch.Service hold inside 9202's controller container against the real qBittorrent 5.2.4)
|
||||
# Seed: a private LAN-only 2 GiB torrent made by 9202's qBittorrent, downloaded by DooPlex's k3s qBittorrent (operator's offer) through a TCP+UDP forwarder on 9202's :6881; 9202's own limit 1250000 B/s (stored 1249280) stands in for the household's
|
||||
|
||||
== 11:40:14 0. before: uplink "mbps_5s":11.4927312; agent: {"data":{"present":false,"waiting_for_quiet":true},"ok":true}
|
||||
qbittorrent limit {Bytes:1249280 Enabled:false KBps:0} err=<nil>
|
||||
store:
|
||||
== 11:40:21 1. CONTROL — a test on request WITHOUT the hold (the seed keeps uploading):
|
||||
{"data":{"error":"the uplink sent 11.1 Mbit/s while held","ok":false,"refused":"not-quiet"},"ok":true}
|
||||
|
||||
== 11:40:36 2. a controller killed in the middle (kill -9 while the agent call runs):
|
||||
2026/10/24 11:40:37 probe: agent waiting_for_quiet=true
|
||||
2026/10/24 11:40:37 [INFO] [upwatch] line test: qbittorrent held at {Bytes:1024 Enabled:false KBps:0} (was {Bytes:1249280 Enabled:false KBps:0}) — asking the agent for a test
|
||||
2026/10/24 11:40:37 probe: killmid — sleeping inside the agent call; kill -9 me now
|
||||
after the kill: uplink "mbps_5s":0.0409568
|
||||
qbittorrent limit {Bytes:1024 Enabled:false KBps:0} err=<nil>
|
||||
store: {"line_test_hold":{"app":"qbittorrent","prev":{"bytes":1249280},"held":{"bytes":1024},"at":"2026-10-24T11:40:37Z"}}
|
||||
== 11:40:52 3. the restarted controller's start-up check:
|
||||
2026/10/24 11:40:53 [INFO] [upwatch] qbittorrent was left at the line-test value (since 2026-10-24T11:40:37Z) — putting its limit back
|
||||
2026/10/24 11:40:53 [INFO] [upwatch] qbittorrent's upload limit is back as it was ({Bytes:1249280 Enabled:false KBps:0})
|
||||
RecoverHold -> true
|
||||
store: {}
|
||||
qbittorrent limit {Bytes:1249280 Enabled:false KBps:0} err=<nil>
|
||||
store: {}
|
||||
uplink "mbps_5s":10.6444544
|
||||
== 11:41:00 4. the real hold: SeedLineTestNow (sampling the uplink every 5 s alongside):
|
||||
2026/10/24 11:41:01 probe: agent waiting_for_quiet=true
|
||||
2026/10/24 11:41:01 [INFO] [upwatch] line test: qbittorrent held at {Bytes:1024 Enabled:false KBps:0} (was {Bytes:1249280 Enabled:false KBps:0}) — asking the agent for a test
|
||||
11:41:00 uplink "mbps_5s":2.6076112
|
||||
11:41:05 uplink "mbps_5s":0.0209008
|
||||
11:41:10 uplink "mbps_5s":0.164744
|
||||
11:41:15 uplink "mbps_5s":69.2326144
|
||||
11:41:20 uplink "mbps_5s":104.54756
|
||||
11:41:25 uplink "mbps_5s":34.787547200000006
|
||||
11:41:30 uplink "mbps_5s":58.9638304
|
||||
11:41:35 uplink "mbps_5s":100.62775359999999
|
||||
11:41:40 uplink "mbps_5s":40.3203488
|
||||
11:41:45 uplink "mbps_5s":57.652910399999996
|
||||
11:41:50 uplink "mbps_5s":104.8901584
|
||||
2026/10/24 11:41:57 [INFO] [upwatch] qbittorrent's upload limit is back as it was ({Bytes:1249280 Enabled:false KBps:0})
|
||||
2026/10/24 11:41:57 [INFO] [upwatch] line test on request: 94.2 Mbit/s
|
||||
SeedLineTestNow returned after 56.8 s
|
||||
store: {}
|
||||
11:41:55 uplink "mbps_5s":50.785411200000006
|
||||
11:42:00 uplink "mbps_5s":10.329262400000001
|
||||
11:42:05 uplink "mbps_5s":11.0489872
|
||||
== 11:42:10 5. after: agent: {"data":{"mbps":94.16292344340873,"measured_at":"2026-10-24T11:41:16Z","present":true,"waiting_for_quiet":false},"ok":true}
|
||||
qbittorrent limit {Bytes:1249280 Enabled:false KBps:0} err=<nil>
|
||||
store: {}
|
||||
uplink "mbps_5s":10.612328
|
||||
|
||||
## agent probe log (the requested test)
|
||||
time=2026-10-24T13:40:36.092+02:00 level=INFO msg="linetest: test on request refused — the uplink is not quiet" mbps=11.1
|
||||
time=2026-10-24T13:40:36.092+02:00 level=INFO msg="probe: POST /host/line-test/run" seconds=15.014445708 mbps=0 err="linetest: not-quiet: the uplink sent 11.1 Mbit/s while held"
|
||||
time=2026-10-24T13:41:16.501+02:00 level=INFO msg="linetest: test on request (a held torrent app, R-1250)" server=10.77.0.1:5201
|
||||
time=2026-10-24T13:41:26.830+02:00 level=INFO msg="linetest: run 1: 93.6 Mbit/s" server=10.77.0.1:5201 server_bytes=117571584 server_seconds=10.049774
|
||||
time=2026-10-24T13:41:42.187+02:00 level=INFO msg="linetest: run 2: 89.6 Mbit/s" server=10.77.0.1:5201 server_bytes=112590848 server_seconds=10.056309
|
||||
time=2026-10-24T13:41:57.557+02:00 level=INFO msg="linetest: run 3: 94.2 Mbit/s" server=10.77.0.1:5201 server_bytes=118358016 server_seconds=10.055594
|
||||
time=2026-10-24T13:41:57.557+02:00 level=INFO msg="linetest: upload line measured: 94.2 Mbit/s (best of 3)" server=10.77.0.1:5201 runs=3 duration_s=41.05557078
|
||||
time=2026-10-24T13:41:57.557+02:00 level=INFO msg="probe: POST /host/line-test/run" seconds=56.058478583 mbps=94.16292344340873 err=<nil>
|
||||
@@ -0,0 +1,151 @@
|
||||
# ============================================================================
|
||||
# postgresql-1 logical backup — 6-hourly pg_dump to HDD5 (rides Restic)
|
||||
# ----------------------------------------------------------------------------
|
||||
# The shared CNPG cluster `postgresql` (database-system, single instance) had no
|
||||
# backup. This CronJob does a LOGICAL backup every 6h:
|
||||
# - pg_dumpall --globals-only -> globals.sql (roles/grants; needed for restore)
|
||||
# - pg_dump -Fc per database -> <db>.dump (custom format; selective restore)
|
||||
# written to hostPath /mnt/5_hdd/backup/postgresql/dumps/<timestamp>/ (HDD5, OFF
|
||||
# the DB's own HDD4 disk). The host backup flow (/opt/backup, backup-postgresql.sh)
|
||||
# then restic-backs-up that dir in its daily run (7d/4w/6m retention). Local
|
||||
# retention here = 3 days. RPO <=6h, no PITR (accepted).
|
||||
#
|
||||
# Auth: connects to postgresql-rw as the CNPG-managed superuser (secret
|
||||
# `postgres-superuser`, keys username/password). Requires enableSuperuserAccess:
|
||||
# true on the Cluster (set in postgresql-cluster.yaml). The superuser is the only
|
||||
# role that can read every DB + dump globals; per-app roles cannot.
|
||||
#
|
||||
# Freshness alert: kube_cronjob_status_last_successful_time (KSM) — see
|
||||
# mon-system/prometheus-rules.yaml (PostgresqlDumpStale, >12h).
|
||||
# ============================================================================
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: postgresql-backup-script
|
||||
namespace: database-system
|
||||
labels:
|
||||
app.kubernetes.io/name: postgresql-backup
|
||||
app.kubernetes.io/instance: postgresql-backup
|
||||
data:
|
||||
pg-backup.sh: |
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
export PGHOST="${PGHOST:-postgresql-rw}" PGPORT="${PGPORT:-5432}"
|
||||
TS="$(date +%Y%m%d-%H%M%S)"
|
||||
OUT="/backup/dumps/${TS}"
|
||||
mkdir -p "${OUT}"
|
||||
echo "[pgbackup] ${TS} -> ${OUT}"
|
||||
|
||||
echo "[pgbackup] globals (roles/grants)"
|
||||
pg_dumpall --globals-only > "${OUT}/globals.sql"
|
||||
|
||||
mapfile -t DBS < <(psql -tAc \
|
||||
"SELECT datname FROM pg_database WHERE NOT datistemplate AND datallowconn ORDER BY 1;")
|
||||
echo "[pgbackup] databases: ${DBS[*]}"
|
||||
for db in "${DBS[@]}"; do
|
||||
echo "[pgbackup] pg_dump -Fc ${db}"
|
||||
pg_dump -Fc -d "${db}" -f "${OUT}/${db}.dump"
|
||||
done
|
||||
|
||||
touch "${OUT}/SUCCESS"
|
||||
echo "[pgbackup] pruning local dumps older than 3 days"
|
||||
find /backup/dumps -mindepth 1 -maxdepth 1 -type d -mtime +3 -exec rm -rf {} + || true
|
||||
echo "[pgbackup] done"
|
||||
ls -lh "${OUT}"
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: postgresql-backup
|
||||
namespace: database-system
|
||||
labels:
|
||||
app.kubernetes.io/name: postgresql-backup
|
||||
app.kubernetes.io/instance: postgresql-backup
|
||||
spec:
|
||||
schedule: "0 */6 * * *"
|
||||
timeZone: "Europe/Budapest"
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 3
|
||||
startingDeadlineSeconds: 600
|
||||
jobTemplate:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: postgresql-backup
|
||||
app.kubernetes.io/instance: postgresql-backup
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
activeDeadlineSeconds: 1800
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: postgresql-backup
|
||||
app.kubernetes.io/instance: postgresql-backup
|
||||
annotations:
|
||||
# postgres image is pinned to the CNPG cluster's version (X.Y)
|
||||
match-regex.version-checker.io/pgbackup: '^\d+\.\d+$'
|
||||
spec:
|
||||
enableServiceLinks: false
|
||||
restartPolicy: OnFailure
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 26
|
||||
runAsGroup: 26
|
||||
fsGroup: 26
|
||||
containers:
|
||||
- name: pgbackup
|
||||
# MUST match the CNPG cluster's Postgres version (pg_dump >= server).
|
||||
image: ghcr.io/cloudnative-pg/postgresql:17.2
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["/bin/bash", "/scripts/pg-backup.sh"]
|
||||
env:
|
||||
- name: PGHOST
|
||||
value: postgresql-rw
|
||||
- name: PGPORT
|
||||
value: "5432"
|
||||
- name: PGUSER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: postgresql-superuser
|
||||
key: username
|
||||
- name: PGPASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: postgresql-superuser
|
||||
key: password
|
||||
- name: HOME
|
||||
value: /tmp
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 512Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
volumeMounts:
|
||||
- name: script
|
||||
mountPath: /scripts
|
||||
readOnly: true
|
||||
- name: backup
|
||||
mountPath: /backup
|
||||
- name: tmp
|
||||
mountPath: /tmp
|
||||
volumes:
|
||||
- name: script
|
||||
configMap:
|
||||
name: postgresql-backup-script
|
||||
defaultMode: 0755
|
||||
- name: backup
|
||||
hostPath:
|
||||
path: /mnt/5_hdd/backup/postgresql
|
||||
type: Directory
|
||||
- name: tmp
|
||||
emptyDir:
|
||||
sizeLimit: 256Mi
|
||||
@@ -0,0 +1,879 @@
|
||||
# =============================================================================
|
||||
# Prometheus Alerting Rules for Longhorn
|
||||
# =============================================================================
|
||||
# =============================================================================
|
||||
# Prometheus Alerting Rules for Longhorn
|
||||
# Excludes prometheus-data PVC since it's designed to run at ~95% capacity
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: prometheus-rules
|
||||
namespace: mon-system
|
||||
labels:
|
||||
app: prometheus
|
||||
data:
|
||||
|
||||
authentik-alerts.yml: |
|
||||
groups:
|
||||
- name: authentik-availability
|
||||
rules:
|
||||
# Fires if Prometheus cannot scrape the server metrics endpoint.
|
||||
# Equivalent to "authentik-server pod is unreachable or crashed."
|
||||
- alert: AuthentikServerDown
|
||||
expr: up{job="authentik-server"} == 0
|
||||
for: 3m
|
||||
labels:
|
||||
severity: critical
|
||||
component: authentik
|
||||
annotations:
|
||||
summary: "Authentik server is unreachable"
|
||||
description: "authentik-server scrape has been failing for 3+ minutes. Logins will fail."
|
||||
|
||||
- alert: AuthentikWorkerDown
|
||||
expr: up{job="authentik-worker"} == 0
|
||||
for: 3m
|
||||
labels:
|
||||
severity: critical
|
||||
component: authentik
|
||||
annotations:
|
||||
summary: "Authentik worker is unreachable"
|
||||
description: "authentik-worker scrape has been failing for 3+ minutes. Background tasks are not running."
|
||||
|
||||
# Per-outpost disconnect. Downgraded to warning because a single
|
||||
# outpost failure only breaks a subset of apps (e.g. just the Arr
|
||||
# stack if arr-outpost is down), not the entire IDP.
|
||||
- alert: AuthentikOutpostDown
|
||||
expr: up{job="authentik-outposts"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
component: authentik
|
||||
annotations:
|
||||
summary: "Authentik outpost {{ $labels.outpost }} is unreachable"
|
||||
description: "Outpost {{ $labels.outpost }} has been down for 5+ minutes. Apps behind this outpost cannot authenticate."
|
||||
|
||||
- name: authentik-functional
|
||||
rules:
|
||||
# *** The 13-days-ago alert. ***
|
||||
# Fires when the worker has tasks in progress but throughput
|
||||
# has collapsed. Probes cannot catch this because the pod is
|
||||
# technically alive — only the queue dynamics give it away.
|
||||
- alert: AuthentikTaskQueueStuck
|
||||
expr: |
|
||||
sum(authentik_tasks_in_progress{job="authentik-worker"}) > 5
|
||||
and
|
||||
sum(rate(authentik_tasks_total{job="authentik-worker"}[5m])) < 0.01
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
component: authentik
|
||||
annotations:
|
||||
summary: "Authentik worker queue is stuck"
|
||||
description: "{{ $value }} tasks in progress with near-zero throughput for 10+ min. Worker is alive but not draining the queue — typically resolved by restarting the deployment."
|
||||
|
||||
# Softer version: queue is growing but still moving somewhat.
|
||||
# Could indicate an expensive task blocking the workers, or
|
||||
# a task that keeps retrying, or steady overload.
|
||||
- alert: AuthentikTaskBacklog
|
||||
expr: sum(authentik_tasks_in_progress{job="authentik-worker"}) > 20
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
component: authentik
|
||||
annotations:
|
||||
summary: "Authentik task backlog >20 for 15 min"
|
||||
description: "{{ $value }} tasks in progress for 15+ min. Likely overload or a single hanging task."
|
||||
|
||||
# User-visible error signal. Threshold is conservative —
|
||||
# on a low-traffic homelab, 0.1 err/sec = ~6 errors/min which
|
||||
# is already noticeable to users.
|
||||
- alert: AuthentikHighErrorRate
|
||||
expr: |
|
||||
sum(rate(django_http_responses_total_by_status_total{job="authentik-server",status=~"5.."}[5m])) > 0.1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
component: authentik
|
||||
annotations:
|
||||
summary: "Authentik is serving 5xx errors"
|
||||
description: "{{ $value | printf \"%.2f\" }} 5xx responses/sec for 5+ min."
|
||||
|
||||
# p95 latency guard. 2s is high for modern auth flows — at this
|
||||
# point users are visibly waiting on the login page.
|
||||
- alert: AuthentikHighLatency
|
||||
expr: |
|
||||
histogram_quantile(0.95,
|
||||
sum by (le) (rate(authentik_main_request_duration_seconds_bucket{job="authentik-server",dest="core"}[5m]))
|
||||
) > 2
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
component: authentik
|
||||
annotations:
|
||||
summary: "Authentik p95 request latency > 2s"
|
||||
description: "p95 latency {{ $value | printf \"%.2f\" }}s for 10+ min. Logins are slow."
|
||||
|
||||
postgresql-alerts.yml: |
|
||||
groups:
|
||||
- name: postgresql-availability
|
||||
rules:
|
||||
- alert: PostgresExporterDown
|
||||
expr: up{job="cloudnativepg"} == 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
component: postgresql
|
||||
annotations:
|
||||
summary: "CloudNativePG metrics endpoint unreachable"
|
||||
description: "CNPG metrics exporter on {{ $labels.pod }} has been down for 2+ min. Postgres may be down or the sidecar may have crashed."
|
||||
|
||||
- name: postgresql-capacity
|
||||
rules:
|
||||
# Threshold of 80% of your 200 max_connections = 160.
|
||||
# If you raise max_connections later, update the number.
|
||||
- alert: PostgresHighConnections
|
||||
expr: sum by (cluster) (cnpg_backends_total) > 160
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
component: postgresql
|
||||
annotations:
|
||||
summary: "Postgres cluster {{ $labels.cluster }} nearing connection limit"
|
||||
description: "{{ $value }} active connections (>80% of max_connections=200). Check for connection leaks."
|
||||
|
||||
# Locks held long enough that other queries are waiting.
|
||||
# Usually cleared fast; a sustained non-zero value is abnormal.
|
||||
- alert: PostgresBackendsWaiting
|
||||
expr: cnpg_backends_waiting_total > 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
component: postgresql
|
||||
annotations:
|
||||
summary: "Postgres has queries blocked on locks"
|
||||
description: "{{ $value }} backends waiting on locks for 5+ min. Investigate long-running transactions."
|
||||
|
||||
longhorn-alerts.yml: |
|
||||
groups:
|
||||
- name: longhorn-volume-alerts
|
||||
rules:
|
||||
# Critical: volume filesystem 95% full
|
||||
- alert: LonghornVolumeSpaceCritical
|
||||
# Filesystem fill as the pod sees it (kubelet), NOT longhorn_volume_actual_size_bytes:
|
||||
# actual size also counts blocks held only by Longhorn snapshots, so a volume that
|
||||
# rewrites files (felhom hub-data: 31% full) read 101% and alarmed all night (2026-10-13).
|
||||
# label_join keeps a `volume` label (namespace/pvc) for the alertmanager inhibit rule.
|
||||
expr: |
|
||||
label_join(
|
||||
max by (namespace, persistentvolumeclaim) (kubelet_volume_stats_used_bytes)
|
||||
/
|
||||
max by (namespace, persistentvolumeclaim) (kubelet_volume_stats_capacity_bytes),
|
||||
"volume", "/", "namespace", "persistentvolumeclaim"
|
||||
) * 100 > 95
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Longhorn volume {{ $labels.volume }} is critically full"
|
||||
description: "Volume {{ $labels.volume }} is at {{ printf \"%.1f\" $value }}% capacity. Immediate action required."
|
||||
# Warning: volume filesystem 85% full
|
||||
- alert: LonghornVolumeSpaceWarning
|
||||
# Filesystem fill as the pod sees it (kubelet), NOT longhorn_volume_actual_size_bytes:
|
||||
# actual size also counts blocks held only by Longhorn snapshots, so a volume that
|
||||
# rewrites files (felhom hub-data: 31% full) read 101% and alarmed all night (2026-10-13).
|
||||
# label_join keeps a `volume` label (namespace/pvc) for the alertmanager inhibit rule.
|
||||
expr: |
|
||||
label_join(
|
||||
max by (namespace, persistentvolumeclaim) (kubelet_volume_stats_used_bytes)
|
||||
/
|
||||
max by (namespace, persistentvolumeclaim) (kubelet_volume_stats_capacity_bytes),
|
||||
"volume", "/", "namespace", "persistentvolumeclaim"
|
||||
) * 100 > 85
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Longhorn volume {{ $labels.volume }} is running low on space"
|
||||
description: "Volume {{ $labels.volume }} is at {{ printf \"%.1f\" $value }}% capacity. Consider expanding or cleaning up."
|
||||
# Volume degraded
|
||||
- alert: LonghornVolumeDegraded
|
||||
expr: longhorn_volume_robustness != 1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Longhorn volume {{ $labels.volume }} is degraded"
|
||||
description: "Volume {{ $labels.volume }} robustness is not healthy. Check replica status."
|
||||
|
||||
# Node storage pressure
|
||||
- alert: LonghornNodeStoragePressure
|
||||
expr: |
|
||||
(
|
||||
longhorn_node_storage_usage_bytes
|
||||
/
|
||||
longhorn_node_storage_capacity_bytes
|
||||
) * 100 > 90
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Longhorn node {{ $labels.node }} storage pressure"
|
||||
description: "Node {{ $labels.node }} disk usage is at {{ printf \"%.1f\" $value }}%."
|
||||
|
||||
# Add this as a new data key (wan-alerts.yml) in the existing
|
||||
# prometheus-rules ConfigMap (mon-system). Thresholds anchored to One.hu's
|
||||
# "normal conditions" figures: 700 Mbit/s down / 28 Mbit/s up.
|
||||
# Throughput is sampled every ~15 min, so `for:` spans >=2 samples to avoid
|
||||
# firing on a single fluke. Recalibrate floors after a week of baseline data.
|
||||
# NOTE: uses Prometheus template funcs (humanize/humanizePercentage/humanizeDuration);
|
||||
# mul/div are NOT valid Prometheus template functions.
|
||||
wan-alerts.yml: |
|
||||
groups:
|
||||
- name: wan-quality-alerts
|
||||
rules:
|
||||
# --- upstream loss: the prime suspect for dropped calls / WireGuard ---
|
||||
# Retuned 2026-08-14: 1% for 2m caught ordinary transient loss.
|
||||
# Measured over 24h, upstream peaks at 5.8% and downstream at 0.64%
|
||||
# on a link that is otherwise fine. 3% held for 15m means loss that
|
||||
# actually persists, which is what breaks calls and WireGuard.
|
||||
- alert: WanUpstreamPacketLoss
|
||||
expr: wan_irtt_loss_ratio{direction="upstream",condition="idle"} > 0.03
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "WAN upstream packet loss to {{ $labels.target }}"
|
||||
description: "irtt upstream loss {{ $value | humanizePercentage }} (>3%) sustained 15m. Brief spikes to ~6% are normal on this link; sustained loss is not. Cable-upstream symptom; capture for ISP."
|
||||
|
||||
- alert: WanDownstreamPacketLoss
|
||||
expr: wan_irtt_loss_ratio{direction="downstream",condition="idle"} > 0.03
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "WAN downstream packet loss to {{ $labels.target }}"
|
||||
description: "irtt downstream loss {{ $value | humanizePercentage }} (>3%) sustained 15m. Measured baseline peak here is 0.64%."
|
||||
|
||||
# --- latency / jitter ---
|
||||
# Retuned 2026-08-14. This watched stat="max" -- the single worst
|
||||
# packet in a probe run -- which is inherently spiky: measured over
|
||||
# 24h its p99 was 327 ms and its absolute peak 1786 ms, on a link
|
||||
# whose mean RTT p99 is a perfectly healthy 41.4 ms. The rule was
|
||||
# therefore true 35.4% of the day. Now keyed on the mean, at a
|
||||
# threshold ~3.5x the observed p99, held for 15m: sustained real
|
||||
# degradation, not one late packet.
|
||||
- alert: WanLatencyHigh
|
||||
expr: wan_irtt_rtt_seconds{stat="mean",condition="idle"} > 0.15
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "WAN latency sustained high to {{ $labels.target }}"
|
||||
description: "irtt MEAN RTT {{ $value | humanizeDuration }} (>150 ms) sustained 15m while idle. Normal here is ~27 ms median with a 41 ms p99, so this is a real change, not a spike. Real-time apps will feel it."
|
||||
|
||||
# 30 ms tripped on ordinary variance; measured round-trip jitter sits
|
||||
# near 3 ms with brief 11 ms peaks. 60 ms held for 15m means the line
|
||||
# is genuinely unstable.
|
||||
- alert: WanJitterHigh
|
||||
expr: wan_irtt_jitter_seconds{direction="round_trip",condition="idle"} > 0.06
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "WAN jitter high to {{ $labels.target }}"
|
||||
description: "Round-trip jitter {{ $value | humanizeDuration }} (>60 ms) sustained 15m. Baseline is ~3 ms. Degrades VoIP/video."
|
||||
|
||||
# --- bufferbloat: latency added while the line is saturated ---
|
||||
- alert: WanBufferbloat
|
||||
expr: |
|
||||
(
|
||||
wan_irtt_rtt_seconds{stat="mean",condition="under_load"}
|
||||
- on(target) wan_irtt_rtt_seconds{stat="mean",condition="idle"}
|
||||
) > 0.1
|
||||
for: 0m
|
||||
labels:
|
||||
severity: info
|
||||
annotations:
|
||||
summary: "WAN bufferbloat on {{ $labels.target }}"
|
||||
description: "RTT rises {{ $value | humanizeDuration }} under load (>100 ms). Line buckles when saturated."
|
||||
|
||||
# --- throughput vs One.hu "normal" 700/28 (alert below 50%) ---
|
||||
# Retuned 2026-08-14. iperf3 runs vary a lot by themselves -- measured
|
||||
# download ranged 88 Mb/s to 869 Mb/s over 24h on a healthy link, so
|
||||
# "below half of plan" fired on ordinary variance. 100 Mb/s is ~1/7th
|
||||
# of normal: that is a broken line, not a slow run. 30m spans at
|
||||
# least two probe cycles (cadence peaks at 16.4 min), so a single bad
|
||||
# measurement cannot trigger it.
|
||||
- alert: WanDownloadDegraded
|
||||
expr: wan_throughput_bits_per_second{direction="download"} < 100e6
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "WAN download severely degraded"
|
||||
description: "Download {{ $value | humanize }}bit/s (<100M) for 30m, against a normal range of roughly 200-870 Mb/s. Two consecutive probe cycles agree, so this is the line rather than a noisy iperf3 run."
|
||||
|
||||
# The old rule assumed a 28 Mb/s plan and alerted under 14M. Measured
|
||||
# upload is ~99 Mb/s with a 24h floor of 37 Mb/s, so the plan comment
|
||||
# was stale and the threshold unreachable -- it could never fire.
|
||||
- alert: WanUploadDegraded
|
||||
expr: wan_throughput_bits_per_second{direction="upload"} < 30e6
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "WAN upload severely degraded"
|
||||
description: "Upload {{ $value | humanize }}bit/s (<30M) for 30m. Measured normal is ~99 Mb/s with a 24h low of 37 Mb/s."
|
||||
|
||||
# --- the monitor itself stopped producing data ---
|
||||
# Retuned 2026-08-14. The 300s threshold was SHORTER THAN THE PROBE
|
||||
# INTERVAL, so this could not not fire: measured over 24h, normal
|
||||
# staleness peaks at 16.4 min for both the irtt and throughput
|
||||
# probes. It flapped continuously (firing 15:02-19:32, 21:47-03:02,
|
||||
# 05:32-07:32 on 2026-08-13/14) and trained the eye to ignore WAN
|
||||
# mail. 45 min is ~3x the observed cadence, so it now means the
|
||||
# monitor is genuinely dead rather than merely between runs.
|
||||
- alert: WanProbeStalled
|
||||
expr: time() - max by(probe) (wan_probe_last_run_timestamp_seconds) > 2700
|
||||
for: 0m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "WAN probe '{{ $labels.probe }}' stalled"
|
||||
description: "No fresh samples for >45 min, which is about 3 probe cycles (normal cadence peaks at 16.4 min). The wan-monitor pod or its Hetzner endpoint is stuck, so every other WAN alert is now blind. Check kubectl -n admin-system logs deploy/wan-monitor."
|
||||
|
||||
# A single external target failing is that site's problem, not yours.
|
||||
# jarrs.eu alone was unreachable 1.25% of 24h while every other
|
||||
# target stayed at 0.00%. Raised 3m -> 30m so this only fires for a
|
||||
# sustained single-target outage; the real link failure is WanOutage.
|
||||
- alert: WanBlackboxTargetDown
|
||||
expr: probe_success{job=~"wan-.*"} == 0
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "WAN probe to {{ $labels.instance }} failing"
|
||||
description: "blackbox {{ $labels.job }} to {{ $labels.instance }} unreachable for 30m. One target only -- if the rest are fine this is the remote site, not your connection. If several are failing, WanOutage will fire instead."
|
||||
|
||||
# THE actual "the internet is gone" alert, and the only WAN rule at
|
||||
# critical. Fires when most EXTERNAL targets fail at once; LAN
|
||||
# targets (192.168.x) are excluded so a router/DNS problem does not
|
||||
# masquerade as a WAN outage. Over the measured 24h this never came
|
||||
# close: 13 targets, worst single one down 1.25%, everything else
|
||||
# 0.00%.
|
||||
- alert: WanOutage
|
||||
expr: |
|
||||
(
|
||||
count(probe_success{job=~"wan-.*", instance!~"192\\.168\\..*"} == 0)
|
||||
/
|
||||
count(probe_success{job=~"wan-.*", instance!~"192\\.168\\..*"})
|
||||
) > 0.6
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "WAN outage: {{ $value | humanizePercentage }} of external targets unreachable"
|
||||
description: "More than 60% of external probe targets (ICMP, DNS and HTTP across independent destinations) have been failing for 5m. That is a real connectivity loss rather than one flaky endpoint. Check the ONE.hu link and the router before looking at anything in the cluster."
|
||||
node-housekeeping-alerts.yml: |
|
||||
groups:
|
||||
- name: node-housekeeping
|
||||
rules:
|
||||
# The weekly housekeeping timer has not SUCCEEDED in over 14 days.
|
||||
# Weekly schedule => this means ~2 consecutive missed/failed runs.
|
||||
- alert: HousekeepingStale
|
||||
expr: time() - node_housekeeping_last_success_timestamp_seconds > 1209600
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
component: node-housekeeping
|
||||
annotations:
|
||||
summary: "Node housekeeping is stale on {{ $labels.instance }}"
|
||||
description: "No successful node-housekeeping run for >14 days (last success {{ printf \"%.0f\" $value }}s ago). The weekly timer may be disabled, failing, or the host has been off. Check `systemctl status node-housekeeping.timer` and the last journal for node-housekeeping.service."
|
||||
|
||||
# The last run exited non-zero. last_success is preserved across
|
||||
# failures, so HousekeepingStale stays quiet until it really lapses;
|
||||
# this catches the failure immediately.
|
||||
- alert: HousekeepingFailing
|
||||
expr: node_housekeeping_exit_code != 0
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
component: node-housekeeping
|
||||
annotations:
|
||||
summary: "Node housekeeping last run failed on {{ $labels.instance }}"
|
||||
description: "node-housekeeping.service exited with code {{ $value }}. Inspect `journalctl -u node-housekeeping.service`. Image prune or journal vacuum likely errored."
|
||||
|
||||
# Judgment alert: housekeeping IS running (success within 14d) but `/`
|
||||
# is still tight (<25% avail). That means the pressure is NOT stale
|
||||
# images/journals — it is data growth. Investigate rather than clean.
|
||||
# NB: <25% avail on `/` is also roughly where Longhorn's default disk
|
||||
# crosses its storage reservation and stops scheduling new replicas.
|
||||
#
|
||||
# `and on(instance)` is LOAD-BEARING, not style. A bare `and` requires
|
||||
# IDENTICAL label sets on both sides. The left vector carries
|
||||
# {device,fstype,instance,job,mountpoint} from node-exporter; the right
|
||||
# is a textfile metric carrying only {instance,job}. No series ever
|
||||
# matched, so this rule was dead from the day it was written and never
|
||||
# fired through the 2026-07 build-cache incident that took `/` to 86%.
|
||||
# Red-proof (2026-08-05, both sides individually true, threshold
|
||||
# substituted to 0.5 because the incident had already been cleared):
|
||||
# bare `and` -> result_count 0
|
||||
# `and on(instance)` -> result_count 1
|
||||
# Evidence: felhom.eu documentation/audits/SPIKE-dooplex-buildcache-2026-08-05.md
|
||||
- alert: RootFsPressureDespiteHousekeeping
|
||||
expr: |
|
||||
node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.25
|
||||
and on(instance)
|
||||
time() - node_housekeeping_last_success_timestamp_seconds < 1209600
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
component: node-housekeeping
|
||||
annotations:
|
||||
summary: "Root fs <25% free on {{ $labels.instance }} despite current housekeeping"
|
||||
description: "`/` is below 25% available even though node-housekeeping succeeded within the last 14 days. Image/journal cleanup is not the cause. Check UNBOUNDED DOCKER BUILD CACHE FIRST — `docker buildx du` — which caused the 2026-07 incident (157 GB, ~95% of that growth) and is capped at 30 GB via /etc/docker/daemon.json; a cap that has been reverted or a daemon that failed to apply it looks exactly like this. `docker builder prune -af` reclaims it synchronously. Only if the cache is small, investigate Longhorn replica/snapshot growth on the default disk (~10% of that incident; also near where the default Longhorn disk hits its scheduling floor)."
|
||||
|
||||
postgresql-backup-alerts.yml: |
|
||||
groups:
|
||||
- name: postgresql-backup
|
||||
rules:
|
||||
- alert: PostgresqlDumpStale
|
||||
expr: time() - kube_cronjob_status_last_successful_time{cronjob="postgresql-backup",namespace="database-system"} > 43200
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
component: database-backup
|
||||
annotations:
|
||||
summary: "postgresql-1 logical backup is stale on {{ $labels.cronjob }}"
|
||||
description: "No successful postgresql-backup CronJob run in >12h. pg_dump dumps land on /mnt/5_hdd/backup/postgresql/dumps and are swept by restic daily. Check kubectl -n database-system get cronjob postgresql-backup and the last job logs."
|
||||
|
||||
backup-freshness-alerts.yml: |
|
||||
groups:
|
||||
- name: backup-freshness
|
||||
rules:
|
||||
# Gauges come from homelab-ansible role backup_freshness (every 15 min, textfile
|
||||
# collector). The gauge is 0 — not absent — when no world-*.tar.gz exists, so an empty
|
||||
# folder or an unmounted HDD5 fires this too. mc-backup runs every 6h relative to pod
|
||||
# start; 13h = two missed runs.
|
||||
- alert: MinecraftBackupStale
|
||||
expr: time() - minecraft_backup_last_success_timestamp_seconds > 13 * 3600
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
component: backup
|
||||
annotations:
|
||||
summary: "minecraft-tlauncher world backup is stale"
|
||||
description: "Newest world-*.tar.gz in /mnt/5_hdd/backup/minecraft-tlauncher is {{ $value | humanizeDuration }} old (6h interval). Check `kubectl -n crafty-system logs deploy/minecraft-tlauncher -c backup --tail=50` — an RCON login failure means the two containers disagree on the password (Secret minecraft-tlauncher-rcon) — and that /mnt/5_hdd is mounted."
|
||||
|
||||
# Guards the guard: if the freshness timer stops, the gauge above freezes at its last
|
||||
# value and MinecraftBackupStale can never fire. node_textfile_mtime_seconds carries the
|
||||
# FULL in-container path (measured: file="/host/textfile/node_housekeeping.prom"), hence
|
||||
# the regex rather than file="backup_freshness.prom", which would match nothing.
|
||||
- alert: BackupFreshnessExporterDead
|
||||
expr: |
|
||||
absent(minecraft_backup_last_success_timestamp_seconds)
|
||||
or
|
||||
time() - node_textfile_mtime_seconds{file=~".*/backup_freshness\\.prom"} > 3600
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
component: backup
|
||||
annotations:
|
||||
summary: "backup-freshness textfile exporter is not updating"
|
||||
description: "backup_freshness.prom is missing or has not been rewritten for >1h (15-min timer). Backup staleness alerts are blind until it is fixed. Check `systemctl status backup-freshness.timer` and `journalctl -u backup-freshness.service`."
|
||||
|
||||
# Felhom hub database off-site copy (R-173, felhom.eu runbooks/RUNBOOK-hub-db-offsite-backup.md).
|
||||
# Written ONLY on success by felhom-hub-db-backup (every 6 h since D34, 2026-10-23: 02:30 08:30 14:30 20:30)
|
||||
# and felhom-hub-db-restore-test (Sun 04:30),
|
||||
# felhom.eu scripts/hub-db-backup/. absent() makes "the script never ran / the file is gone" an alarm too.
|
||||
- alert: HubDBBackupStale
|
||||
expr: |
|
||||
time() - felhom_hub_db_backup_last_success_timestamp_seconds > 13 * 3600
|
||||
or
|
||||
absent(felhom_hub_db_backup_last_success_timestamp_seconds)
|
||||
for: 30m
|
||||
labels:
|
||||
severity: critical
|
||||
component: backup
|
||||
annotations:
|
||||
summary: "The Felhom hub database has not reached ep0 for 13 h (R-173, D34)"
|
||||
description: "No successful push of the hub DB snapshot to ep0 for >13h — two 6-hourly pushes missed (02:30 08:30 14:30 20:30). Check `journalctl -u felhom-hub-db-backup.service`, the tunnel `systemctl status felhom-ep0-pbs-tunnel`, and that the hub logs `db snapshot written` at 02:00/08:00/14:00/20:00."
|
||||
# Felhom release copy to ep0 (D33, R-1227; felhom.eu scripts/release-copy/, runbooks/total-loss-of-dooplex.md).
|
||||
# Written ONLY after the push AND a checksum compare showed no difference (nightly 01:40).
|
||||
- alert: FelhomReleaseCopyStale
|
||||
expr: |
|
||||
time() - felhom_release_copy_last_success_timestamp_seconds > 26 * 3600
|
||||
or
|
||||
absent(felhom_release_copy_last_success_timestamp_seconds)
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
component: backup
|
||||
annotations:
|
||||
summary: "The Felhom release images and agent packages have not reached ep0 for 26 h (D33)"
|
||||
description: "No verified copy of the released images and agent packages to ep0 for >26h (nightly 01:40). Check `journalctl -u felhom-release-copy.service`."
|
||||
- alert: HubDBRestoreTestStale
|
||||
expr: |
|
||||
time() - felhom_hub_db_restore_test_last_success_timestamp_seconds > 8 * 24 * 3600
|
||||
or
|
||||
absent(felhom_hub_db_restore_test_last_success_timestamp_seconds)
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
component: backup
|
||||
annotations:
|
||||
summary: "The hub database copy on ep0 has not passed a restore test for 8 days (R-173)"
|
||||
description: "The weekly restore test (Sun 04:30) has not succeeded for >8 days. Check `journalctl -u felhom-hub-db-restore-test.service`."
|
||||
# Gitea + DooPlex secrets off-site (R-232 b/h). Written ONLY on success by felhom-dooplex-offsite (00:20 daily) and
|
||||
# felhom-dooplex-offsite-restore-test (Sun 05:30), felhom.eu scripts/dooplex-offsite/. absent() included.
|
||||
- alert: DooplexGiteaOffsiteStale
|
||||
expr: |
|
||||
time() - felhom_dooplex_offsite_last_success_timestamp_seconds > 26 * 3600
|
||||
or
|
||||
absent(felhom_dooplex_offsite_last_success_timestamp_seconds)
|
||||
for: 30m
|
||||
labels:
|
||||
severity: critical
|
||||
component: backup
|
||||
annotations:
|
||||
summary: "Gitea and DooPlex's secrets have not reached ep0 for 26 h (R-232)"
|
||||
description: "No successful off-site push of Gitea (repositories, database dump, config) and the secrets export for >26h (daily at 00:20). Check `journalctl -u felhom-dooplex-offsite.service` and the tunnel `systemctl status felhom-ep0-pbs-tunnel`."
|
||||
- alert: DooplexGiteaRestoreTestStale
|
||||
expr: |
|
||||
time() - felhom_dooplex_offsite_restore_test_last_success_timestamp_seconds > 8 * 24 * 3600
|
||||
or
|
||||
absent(felhom_dooplex_offsite_restore_test_last_success_timestamp_seconds)
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
component: backup
|
||||
annotations:
|
||||
summary: "The Gitea copy on ep0 has not passed a restore test for 8 days (R-232)"
|
||||
description: "The weekly restore test (Sun 05:30) has not succeeded for >8 days. Check `journalctl -u felhom-dooplex-offsite-restore-test.service`."
|
||||
|
||||
smart-alerts.yml: |
|
||||
# =========================================================================
|
||||
# Physical disk health, from smartctl-exporter (admin-system).
|
||||
#
|
||||
# Written after 2026-08-13, when /dev/sdg (ST3000VX010, S/N Z6A07P2G)
|
||||
# developed 120 uncorrectable sectors, corrupted the radarr-kids and
|
||||
# opengist Longhorn volumes, and nothing alerted. Every volume in this
|
||||
# cluster is numberOfReplicas: 1, so one bad sector is an outage plus data
|
||||
# loss -- these alerts are the only early warning that exists.
|
||||
#
|
||||
# NOTHING HERE KEYS ON smartctl_device_smart_status. Measured on the day of
|
||||
# the failure, that flag read PASSED (1) on all seven drives INCLUDING the
|
||||
# one with 120 pending sectors and 665 reported uncorrectables. It is kept
|
||||
# only as a last-resort critical signal, never as the primary one.
|
||||
#
|
||||
# The SSDs do not expose every attribute (the Apacer reports no pending /
|
||||
# reallocated / uncorrectable at all), so the severity recording rules are
|
||||
# split per check and combined with max by (device). A device that lacks an
|
||||
# attribute simply contributes fewer series instead of dropping out of the
|
||||
# score entirely, which is what a plain sum would do.
|
||||
# =========================================================================
|
||||
groups:
|
||||
- name: disk-smart-severity
|
||||
rules:
|
||||
# --- per-check severity: 0 = healthy, 1 = warning, 2 = critical ---
|
||||
|
||||
# Baseline so every drive has a score even when perfectly healthy.
|
||||
# Keyed on temperature because it is the one metric all 7 report.
|
||||
- record: disk:smart_severity
|
||||
expr: 0 * max by (device) (smartctl_device_temperature)
|
||||
labels:
|
||||
check: baseline
|
||||
|
||||
# Sectors the drive could not read and has not yet remapped. This is
|
||||
# the attribute that actually caught the 2026-08-13 failure.
|
||||
- record: disk:smart_severity
|
||||
expr: 2 * (max by (device) (smartctl_device_attribute{attribute_name="Current_Pending_Sector",attribute_value_type="raw"}) > bool 0)
|
||||
labels:
|
||||
check: pending_sectors
|
||||
|
||||
- record: disk:smart_severity
|
||||
expr: 2 * (max by (device) (smartctl_device_attribute{attribute_name="Offline_Uncorrectable",attribute_value_type="raw"}) > bool 0)
|
||||
labels:
|
||||
check: offline_uncorrectable
|
||||
|
||||
# Warn on any remap, critical past 10. A handful of reallocations is
|
||||
# normal ageing; a climbing count is not.
|
||||
- record: disk:smart_severity
|
||||
expr: (max by (device) (smartctl_device_attribute{attribute_name="Reallocated_Sector_Ct",attribute_value_type="raw"}) > bool 0) + (max by (device) (smartctl_device_attribute{attribute_name="Reallocated_Sector_Ct",attribute_value_type="raw"}) > bool 10)
|
||||
labels:
|
||||
check: reallocated_sectors
|
||||
|
||||
- record: disk:smart_severity
|
||||
expr: (max by (device) (smartctl_device_attribute{attribute_name="Reported_Uncorrect",attribute_value_type="raw"}) > bool 0) + (max by (device) (smartctl_device_attribute{attribute_name="Reported_Uncorrect",attribute_value_type="raw"}) > bool 10)
|
||||
labels:
|
||||
check: reported_uncorrect
|
||||
|
||||
# WD Red Pro spec max is 65C; warn well before that. The 10TB WDs
|
||||
# idle at 47-50C in this chassis, so 55 is the first real signal.
|
||||
- record: disk:smart_severity
|
||||
expr: (max by (device) (smartctl_device_temperature) >= bool 55) + (max by (device) (smartctl_device_temperature) >= bool 60)
|
||||
labels:
|
||||
check: temperature
|
||||
|
||||
# Last resort only -- see the header note on why this is not trusted.
|
||||
- record: disk:smart_severity
|
||||
expr: 2 * (max by (device) (smartctl_device_smart_status) < bool 1)
|
||||
labels:
|
||||
check: smart_status
|
||||
|
||||
# Single number per drive for the Grafana health indicator.
|
||||
- record: disk:smart_health_score
|
||||
expr: max by (device) (disk:smart_severity)
|
||||
|
||||
- name: disk-smart-alerts
|
||||
rules:
|
||||
- alert: DiskUncorrectableSectors
|
||||
expr: max by (device) (smartctl_device_attribute{attribute_name="Current_Pending_Sector",attribute_value_type="raw"}) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
component: disk
|
||||
annotations:
|
||||
summary: "{{ $labels.device }} has {{ $value }} unreadable sectors"
|
||||
description: "Current_Pending_Sector is above zero, meaning the drive cannot read those sectors and has not remapped them. THIS IS THE SIGNAL THAT WAS MISSING ON 2026-08-13 -- the same condition on /dev/sdg corrupted two Longhorn volumes. Every volume here is single-replica, so treat this as data loss in progress. Evacuate the Longhorn disk (set allowScheduling false + evictionRequested true on nodes.longhorn.io/dooplex), verify backups, and plan replacement. SMART overall-health will still say PASSED; ignore it."
|
||||
|
||||
- alert: DiskOfflineUncorrectable
|
||||
expr: max by (device) (smartctl_device_attribute{attribute_name="Offline_Uncorrectable",attribute_value_type="raw"}) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
component: disk
|
||||
annotations:
|
||||
summary: "{{ $labels.device }} reports {{ $value }} offline-uncorrectable sectors"
|
||||
description: "The drive's offline scan found sectors it cannot recover. Usually tracks Current_Pending_Sector. Same response: evacuate and replace."
|
||||
|
||||
- alert: DiskReportedUncorrectGrowing
|
||||
expr: increase(smartctl_device_attribute{attribute_name="Reported_Uncorrect",attribute_value_type="raw"}[6h]) > 0
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
component: disk
|
||||
annotations:
|
||||
summary: "{{ $labels.device }} uncorrectable error count is climbing"
|
||||
description: "Reported_Uncorrect grew in the last 6h, so the drive is actively failing reads rather than sitting on old damage. On 2026-08-13 this counter moved 633 -> 665 within a single session while the disk was being evacuated. A rising count means act now, not at the next maintenance window."
|
||||
|
||||
- alert: DiskReallocatedSectorsGrowing
|
||||
expr: increase(smartctl_device_attribute{attribute_name="Reallocated_Sector_Ct",attribute_value_type="raw"}[24h]) > 0
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
component: disk
|
||||
annotations:
|
||||
summary: "{{ $labels.device }} is remapping sectors"
|
||||
description: "Reallocated_Sector_Ct increased in the last 24h. The drive is still covering for itself, but a count that moves is a drive on the way out. Note the 2026-08-13 failure had Reallocated_Sector_Ct stuck at 0 while 120 sectors sat pending -- that model cannot remap, so a flat zero here is not reassurance on Seagate ST3000VX010 units."
|
||||
|
||||
- alert: DiskOverheating
|
||||
expr: max by (device) (smartctl_device_temperature) >= 60
|
||||
for: 20m
|
||||
labels:
|
||||
severity: critical
|
||||
component: disk
|
||||
annotations:
|
||||
summary: "{{ $labels.device }} at {{ $value }}C"
|
||||
description: "Sustained 60C or above. Check chassis fans (Fan RPM panel on the Grafana overview dashboard) and airflow before the drive throttles or fails."
|
||||
|
||||
- alert: DiskOverheatingWarning
|
||||
expr: max by (device) (smartctl_device_temperature) >= 55
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
component: disk
|
||||
annotations:
|
||||
summary: "{{ $labels.device }} running warm at {{ $value }}C"
|
||||
description: "Above 55C for an hour. Baseline in this chassis is 33-50C, so this is a real change. Not urgent, but check airflow."
|
||||
|
||||
- alert: DiskSmartStatusFailed
|
||||
expr: max by (device) (smartctl_device_smart_status) < 1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
component: disk
|
||||
annotations:
|
||||
summary: "{{ $labels.device }} SMART self-assessment FAILED"
|
||||
description: "The drive itself has given up -- this flag stays PASSED through severe damage, so a FAILED reading means the drive is at the very end. Replace immediately; do not schedule around it."
|
||||
|
||||
# Change detector, deliberately NOT a fixed count. An earlier version
|
||||
# of this alert compared against a hardcoded 7, which had to be
|
||||
# hand-edited on every disk change, went stale silently if anyone
|
||||
# forgot, and could not notice a disk being ADDED. This form needs no
|
||||
# maintenance and names the specific drive that went away.
|
||||
#
|
||||
# max_over_time rather than `offset`: a scrape gap (exporter restart,
|
||||
# node reboot) must not erase the fact that a disk was there. Fires
|
||||
# for 6h after the disappearance, then self-resolves -- it is a
|
||||
# notification that the inventory changed, not a standing condition.
|
||||
- alert: DiskDisappeared
|
||||
expr: max by (device) (max_over_time(smartctl_device_smart_status[6h])) unless max by (device) (smartctl_device_smart_status)
|
||||
for: 10m
|
||||
labels:
|
||||
severity: critical
|
||||
component: disk
|
||||
annotations:
|
||||
summary: "{{ $labels.device }} vanished from smartctl-exporter"
|
||||
description: "This drive reported SMART data within the last 6h and now reports none. Either it stopped responding (dying controller, dropped SATA link, powered-off drive) or it was physically removed. If the removal was deliberate, also drop its --smartctl.device entry from admin-system/smartctl-exporter.yaml, otherwise the exporter keeps retrying a device that is not there. If it was NOT deliberate, treat it as a failed drive: check dmesg for ATA link resets and confirm which Longhorn volumes had replicas on it."
|
||||
|
||||
# Catch-all for the other direction -- a disk being added, or any
|
||||
# count movement the per-device rule above does not express.
|
||||
- alert: DiskInventoryChanged
|
||||
expr: changes(smartctl_devices[6h]) > 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
component: disk
|
||||
annotations:
|
||||
summary: "Number of disks visible to smartctl-exporter changed ({{ $value }} change(s) in 6h)"
|
||||
description: "The count of monitored physical disks moved. Expected after adding or replacing a drive -- in that case update the --smartctl.device list in admin-system/smartctl-exporter.yaml so the new disk is actually monitored, since the list is explicit and a new drive is NOT picked up automatically. Unexpected movement means a drive is dropping in and out, which usually precedes failure."
|
||||
|
||||
- alert: SmartctlExporterDown
|
||||
expr: up{service="smartctl-exporter"} == 0
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
component: disk
|
||||
annotations:
|
||||
summary: "smartctl-exporter is not being scraped"
|
||||
description: "No disk health data is being collected, so every alert in this group is silently inert. The exporter is a DaemonSet in admin-system discovered via the prometheus.io/scrape annotation on its Service (the 'kubernetes-services' job). Check kubectl -n admin-system get pods -l app=smartctl-exporter."
|
||||
|
||||
# ===========================================================================
|
||||
# Workload health -- the gap that let external-dns-pihole crashloop for 27h
|
||||
# with 326 restarts and nobody notice (2026-08-23).
|
||||
#
|
||||
# Until now every alert here watched a *subsystem* (authentik, postgres,
|
||||
# longhorn, WAN, disks) via that subsystem's own exporter. Nothing watched
|
||||
# Kubernetes itself, so a workload with no dedicated exporter could die
|
||||
# permanently in silence. These rules close that by keying on
|
||||
# kube-state-metrics, which reports on every pod in the cluster with no
|
||||
# per-app wiring.
|
||||
#
|
||||
# DEPENDENCY, and it is a real one: kube-state-metrics runs live in mon-system
|
||||
# but exists in NO manifest in this repo (see CLAUDE.md "Known traps"). It
|
||||
# survives only because `monitoring` is manual-sync and does not prune. If it
|
||||
# is ever removed, these three alerts go silently inert -- which is exactly
|
||||
# the failure mode they exist to prevent. KubeStateMetricsDown below is the
|
||||
# guard: it fires from `up`, not from kube-state-metrics' own output, so it
|
||||
# still works when the exporter is the thing that is gone.
|
||||
# ===========================================================================
|
||||
workload-alerts.yml: |
|
||||
groups:
|
||||
- name: workload-health
|
||||
rules:
|
||||
# The direct answer to "a pod is crashlooping". kubelet sets this
|
||||
# waiting reason once it starts backing off restarts, so it means
|
||||
# "has already failed repeatedly", not "failed once".
|
||||
#
|
||||
# for: 15m rather than something tight: a container that crashes two
|
||||
# or three times while waiting for its database is normal here and
|
||||
# recovers on its own. 15m is still ~100x faster than the 27h this
|
||||
# alert was written for.
|
||||
- alert: PodCrashLooping
|
||||
expr: max by (namespace, pod, container) (kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff",namespace!="jarr-nightly-system"}) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
component: workload
|
||||
annotations:
|
||||
summary: "{{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} is crashlooping"
|
||||
description: "The container has been in CrashLoopBackOff for over 15m, so it is failing on startup and kubelet has given up restarting it quickly. Get the reason from the PREVIOUS container, not the current one, which is usually still waiting: kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c {{ $labels.container }} --previous. A frequent cause here is Renovate bumping an image to a version that removed a flag the manifest still passes -- the log line is then a one-liner like \"unknown long flag\". Check whether an auto-syncing Application deployed a new tag just before the restarts began."
|
||||
|
||||
# Catches the same class of silent breakage that never reaches
|
||||
# CrashLoopBackOff, because the container never starts at all: a bad
|
||||
# image tag, a missing secret key, an unparseable config. Renovate can
|
||||
# produce every one of these, and none of them show up above.
|
||||
- alert: ContainerStuckWaiting
|
||||
expr: max by (namespace, pod, container, reason) (kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull|InvalidImageName|CreateContainerConfigError|CreateContainerError",namespace!="jarr-nightly-system"}) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
component: workload
|
||||
annotations:
|
||||
summary: "{{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} stuck: {{ $labels.reason }}"
|
||||
description: "The container cannot start at all, so it never reaches CrashLoopBackOff and PodCrashLooping will not fire for it. ImagePullBackOff/ErrImagePull/InvalidImageName mean the tag does not exist -- suspect a bad Renovate bump or a typo in the manifest. CreateContainerConfigError almost always means a referenced secret or configMap key is missing, which in this repo means a secret that was never created imperatively (secrets are not in git). kubectl -n {{ $labels.namespace }} describe pod {{ $labels.pod }} names the exact object."
|
||||
|
||||
# Crash-and-recover loops that never sit in CrashLoopBackOff long
|
||||
# enough for the first alert: OOMKills under bursty load, a liveness
|
||||
# probe that is slightly too aggressive. increase() is reset-aware, so
|
||||
# a pod being recreated does not produce a false spike.
|
||||
- alert: PodRestartingFrequently
|
||||
expr: increase(kube_pod_container_status_restarts_total{namespace!="jarr-nightly-system"}[1h]) > 4
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
component: workload
|
||||
annotations:
|
||||
summary: "{{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} restarted {{ $value | printf \"%.0f\" }}x in 1h"
|
||||
description: "The container keeps dying and coming back without staying down long enough to be flagged as crashlooping. Check the exit code: kubectl -n {{ $labels.namespace }} describe pod {{ $labels.pod }} | grep -A3 'Last State'. Exit 137 with Reason OOMKilled means the memory limit is too low for this workload's peaks -- raise limits.memory in the manifest rather than removing the limit. Otherwise suspect a liveness probe whose timeout is shorter than the app's worst-case response."
|
||||
|
||||
# Guard for the three rules above. Deliberately keyed on `up`, so it
|
||||
# reports the exporter being gone even though every other rule in this
|
||||
# group would be evaluating to "no data" and therefore silent.
|
||||
- alert: KubeStateMetricsDown
|
||||
expr: absent(up{job=~".*kube-state-metrics.*"} == 1) or max(up{job=~".*kube-state-metrics.*"}) == 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
component: workload
|
||||
annotations:
|
||||
summary: "kube-state-metrics is not being scraped -- all pod-health alerting is inert"
|
||||
description: "Nothing is watching pod state cluster-wide: PodCrashLooping, ContainerStuckWaiting and PodRestartingFrequently cannot fire while this is true. kube-state-metrics runs in mon-system but is NOT defined in any manifest in this repo, so it cannot be resynced from git -- check kubectl -n mon-system get pods -l app.kubernetes.io/name=kube-state-metrics and, if it is gone, it has to be recreated by hand. Bringing it into the repo is HM-007."
|
||||
|
||||
# =======================================================================
|
||||
# HM-040: a cryptominer ran in the Plex pod (CPU above 4 cores from
|
||||
# 2026-10-17 15:10Z to 10-23 08:15Z, ~20 of 40 cores) and nothing fired:
|
||||
# Plex had no CPU limit and no rule looked at container CPU at all. Both
|
||||
# CPU rules use cAdvisor series (job kubernetes-cadvisor), NOT
|
||||
# kube-state-metrics, so they keep working when KSM is gone (HM-007).
|
||||
#
|
||||
# Measured 2026-10-23 over the 14 d of retention: the ONLY container whose
|
||||
# 10-minute average ever exceeded 4 cores was plex (the miner); the next
|
||||
# highest peaks were ~1.5 cores (longhorn instance-manager, gitea). Plex's
|
||||
# own pre-incident peak (09-23 -> 10-17) was 3.02 cores.
|
||||
#
|
||||
# Two tiers (operator decision 2026-10-23):
|
||||
# 10 cores, critical, 15m, no allowlist -- nothing legitimate is near it.
|
||||
# A container capped below 10 cores (Plex is now 6, HM-040/HM-041)
|
||||
# cannot reach it; the 4-core warning covers those.
|
||||
# 4 cores, warning, 30m, immich ML excepted (container `main`, matched on
|
||||
# the pod name) -- measured never above 4 cores, kept for model warm-ups.
|
||||
# No Plex evening window: it would have hidden this miner up to 7 h a day.
|
||||
#
|
||||
# No process-count rule, on purpose. KSM has no process metric; cAdvisor's
|
||||
# container_processes did not separate this incident: Plex ran 7 (p50) /
|
||||
# 8 (max) processes before 10-17 and 9 (p50, also the minimum) during it.
|
||||
# Replayed over 14 d, a 2x-own-baseline rule was true for 9 five-minute
|
||||
# steps; container_threads overlapped the baseline (86/187 vs 90/193).
|
||||
# The miner is ONE process with 46 threads. CPU is the signal.
|
||||
# =======================================================================
|
||||
- name: workload-abuse
|
||||
rules:
|
||||
- alert: ContainerCPUAbove10Cores
|
||||
expr: |
|
||||
sum by (namespace, pod, container) (
|
||||
rate(container_cpu_usage_seconds_total{container!="",container!="POD"}[5m])
|
||||
) > 10
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
component: workload
|
||||
annotations:
|
||||
summary: "{{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} is using {{ $value | printf \"%.1f\" }} cores"
|
||||
description: "A container has averaged more than 10 of the node's 40 cores for 15 minutes. Nothing on this cluster has ever done that legitimately (measured 2026-10-23: the 14-day max outside the HM-040 miner was ~1.5 cores). Treat it as a compromise until shown otherwise: list the container's processes and where they run from -- for p in $(sudo grep -l <pod-uid-with-underscores> /proc/[0-9]*/cgroup | cut -d/ -f3); do sudo readlink /proc/$p/exe; done -- anything under /tmp, /var/tmp, /dev/shm or '(deleted)' is the HM-040 shape. Capture before you kill (INCIDENT.md of HM-040)."
|
||||
|
||||
- alert: ContainerCPUAbove4Cores
|
||||
expr: |
|
||||
sum by (namespace, pod, container) (
|
||||
rate(container_cpu_usage_seconds_total{container!="",container!="POD",pod!~"immich-machine-learning-.*"}[5m])
|
||||
) > 4
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
component: workload
|
||||
annotations:
|
||||
summary: "{{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} has used {{ $value | printf \"%.1f\" }} cores for 30m"
|
||||
description: "A container has averaged more than 4 cores for 30 minutes. Measured 2026-10-23, no container except the HM-040 miner had ever done that (next highest ~1.5 cores; Plex's own peak 3.02). Check where its processes run from before calling it load: anything under /tmp, /var/tmp, /dev/shm or '(deleted)' is the HM-040 shape. If it is genuine load, raise the threshold for that workload here rather than ignoring the alert."
|
||||
|
||||
@@ -34,6 +34,7 @@
|
||||
| **R-1245** | **DECIDE: how large is qBittorrent's default upload limit?** (P3) | CLOSED 2026-10-24 — **D41 = 50 %** (operator ruling 07:22), of the host's TESTED line (the weekly iperf3 test, R-1247 — never the push floor); built in controller 0.342.0 (`ed1011b`); released with R-1244 | `08` §9.5 |
|
||||
| **R-1246** | **DECIDE: who is told when the box limits or flags something?** (P3) | CLOSED 2026-10-24 — **D42** (operator ruling 07:22): the family only about the qBittorrent limit (one line on its page); `app_unexplained_upload` / `app_unexplained_cpu` are operator-only (hub 0.169.0); released with R-1244 | `08` §9.4 |
|
||||
| **R-1244** | **The upload and CPU watch was built and not released.** (P2) | CLOSED 2026-10-24 — released on the operator's yes: hub **0.169.0** (the four operator-only events + the System page's „Upload line"; sha256:fb9b2489…, deployed 07:51Z, a 30-minute watch with no alarm) and controller **0.342.0** (sha256:d002d3b7…, `42cda07`; per-box floors → Tester 1, demo-hp, demo-felhom healthy on it, box checks + https-only green; then the global floor 0.342.0, MinAgent 0.158.1, 08:29:19Z, read back on the hub log and page). demo-felhom's qBittorrent got 50 % of its tested line (5 789 696 B/s ≈ 46 Mbit/s; the log, qBittorrent's own config and the box's record agree). Open beside it: R-1250 (a seeding qBittorrent is never tested), R-1248 (an app's API can lie) | `audits/line-test-2026-10-24/controller-0342-boxes.txt` |
|
||||
| **R-934** | **Pull the failed disk ST3000VX010 S/N Z6A07P2G out of DooPlex.** (P3) | CLOSED 2026-10-24 — pulled by the operator in the 2026-10-23 maintenance window (homelab HM-043, `1bb4c4d`): `DiskDisappeared` and `DiskInventoryChanged` fired at the pull and cleared (18:02Z, 20:17Z); no `smartctl_*` series names the serial any more and no manifest lists it (comments only). The silence `device=~.*_Z6A07P2G` (until 2026-11-11) now matches nothing and lapses by itself — on the homelab list to expire early. The twin Z6A070XZ (`/mnt/3_hdd`) stays. Full text: `git show 6018486f:documentation/backlog/OPEN-ITEMS.md`. | `audits/dooplex-alarms-2026-10-24/part3-alarms.txt` (the alarm history and the silence) |
|
||||
| **R-1247** | **The box's upload line was only a lower bound, learnt from its own off-site push.** (P4) | CLOSED 2026-10-24 — the weekly iperf3 line test (operator ruling 07:26): ep0's tunnel-only `felhom-iperf3` (no answer on the internet, `:22` control open), agent **0.164.0** (`internal/linetest`, its own iperf3 client; vouched by the operator) and the controller reading it. The three boxes' first tests: Tester 1 **94.3**, demo-hp **94.4**, demo-felhom **92.6** Mbit/s (best of 3, 10.05 s per run; demo-felhom met „server busy" twice and tried again the same day); the push floor said „at least 10" | `audits/line-test-2026-10-24/first-tests-three-boxes.txt`, `ep0-install.txt` |
|
||||
|
||||
## 2026-10-23 day (later) — D39 = B: the security-fix lane
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-10-24 (afternoon)
|
||||
|
||||
**Updated 2026-10-24 (day, attended, done).** Released on your yeses: hub 0.168.0 (tokens locked) and 0.169.0, agent 0.164.0 (vouched), controller 0.342.0 (the floor). Each box tests its upload weekly (94, 94, 93 Mbit/s); a hacked app that uploads or computes more than it explains alarms only you, and qBittorrent gets half the tested line. The hub is out of DooPlex's open backup folder.
|
||||
|
||||
## Moved from STATUS on 2026-10-24 (day, done)
|
||||
|
||||
**Updated 2026-10-24 (day, attended).** Hub 0.168.0 is live: the last plain tokens are locked, and every copy of the hub database that leaves DooPlex is clean. The hub is out of DooPlex's open backup folder. Each box can now test its upload once a week (demo-hp: 90.7 Mbit/s, not 10); the releases follow on your yes.
|
||||
|
||||
Reference in New Issue
Block a user