From 44d1904286c1daea475a05f7275cb565bcb698a7 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 24 Oct 2026 13:53:57 +0200 Subject: [PATCH] DooPlex backup alarms: one cause (the dump job's password after the reboot), fixed; alarm mail quieter; R-1250 proven MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit R-232 dated note + a dated check for tonight's sweep; R-934 closed (the disk was pulled 2026-10-23); R-1250 fixed on main, live proof on demo-hp with a real seed; 08 §9 item 5. Evidence audits/dooplex-alarms-2026-10-24/. --- CONTEXT.md | 12 + REPORT-dooplex-alarms-2026-10-24.md | 111 +++ STATUS.md | 9 +- documentation/architecture/08-alarm-ladder.md | 14 +- .../dooplex-alarms-2026-10-24/README.md | 51 + .../alertmanager.yaml.before | 293 ++++++ .../dooplex-alarms-2026-10-24/alerts-7d.json | 1 + .../part1-2-diagnosis.txt | 51 + .../dooplex-alarms-2026-10-24/part1-2-fix.txt | 52 ++ .../part3-alarms.txt | 21 + .../part4-r1250-live-proof.txt | 59 ++ .../postgresql-backup.yaml.before | 151 +++ .../prometheus-rules.yaml.before | 879 ++++++++++++++++++ documentation/backlog/CLOSED-ITEMS.md | 1 + documentation/backlog/OPEN-ITEMS.md | 6 +- documentation/backlog/STATUS-HISTORY.md | 4 + 16 files changed, 1704 insertions(+), 11 deletions(-) create mode 100644 REPORT-dooplex-alarms-2026-10-24.md create mode 100644 documentation/audits/dooplex-alarms-2026-10-24/README.md create mode 100644 documentation/audits/dooplex-alarms-2026-10-24/alertmanager.yaml.before create mode 100644 documentation/audits/dooplex-alarms-2026-10-24/alerts-7d.json create mode 100644 documentation/audits/dooplex-alarms-2026-10-24/part1-2-diagnosis.txt create mode 100644 documentation/audits/dooplex-alarms-2026-10-24/part1-2-fix.txt create mode 100644 documentation/audits/dooplex-alarms-2026-10-24/part3-alarms.txt create mode 100644 documentation/audits/dooplex-alarms-2026-10-24/part4-r1250-live-proof.txt create mode 100644 documentation/audits/dooplex-alarms-2026-10-24/postgresql-backup.yaml.before create mode 100644 documentation/audits/dooplex-alarms-2026-10-24/prometheus-rules.yaml.before diff --git a/CONTEXT.md b/CONTEXT.md index 4109ec4a..1aef5d52 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,18 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-10-24 (afternoon, attended) — two DooPlex backup alarms — DONE.** One cause: the 2026-10-23 window reboot +> restarted `postgresql-2`; CNPG 1.29 sets the role `postgres` from the superuser secret, then from the +> `bootstrap.initdb` owner secret (`owner: postgres`), so `postgres-superuser` is in force at every start; the dump +> CronJob read `postgresql-superuser` → 28P01 from 16:00Z → `PostgresqlDumpStale`; the 00:20 Gitea push refused the +> stale dump → `DooplexGiteaOffsiteStale`. **Operator rulings 2026-10-24:** (1) fix the CronJob only (homelab +> `531023d`; the Cluster keeps two writers — homelab list); (2) apply the routing change by restarting Alertmanager +> (its `envsubst` init container makes `/-/reload` useless; homelab `22d7697`: critical 4 h, warning 12 h, one mail per +> backup group and per drive); (3) the R-1250 live proof may use DooPlex's k3s qBittorrent as the downloading peer +> (one private LAN torrent in its container `/tmp`, deleted with its files). R-1250 committed (agent `29bcf44`, +> `1e087bb`; controller `046b808`), not released. R-934 closed (the disk was pulled 2026-10-23). Catalog `7de554e`: +> the box walk sends the drive itself as HDD_PATH (R-839). Report: `REPORT-dooplex-alarms-2026-10-24.md`. + > **2026-10-24 (attended release run) — DONE.** Hub 0.168.0 (05:39Z) and 0.169.0 (07:51Z) deployed; agent 0.164.0 > released, delivered to the three test boxes and vouched by the operator; controller 0.342.0 the global floor (08:29Z, > MinAgent 0.158.1). ep0 runs `felhom-iperf3` (tunnel only). First line tests: Tester 1 94.3, demo-hp 94.4, demo-felhom diff --git a/REPORT-dooplex-alarms-2026-10-24.md b/REPORT-dooplex-alarms-2026-10-24.md new file mode 100644 index 00000000..3d85e701 --- /dev/null +++ b/REPORT-dooplex-alarms-2026-10-24.md @@ -0,0 +1,111 @@ +# REPORT — two DooPlex backup alarms, every alarm the mail route delivers, and R-1250 (2026-10-24, attended) + +Brief: „two DooPlex backup alarms — find why, fix, and check every other alarm the new mail route now delivers". +Read first: `runbooks/total-loss-of-dooplex.md`, `runbooks/RUNBOOK-hub-db-offsite-backup.md`, +`audits/dooplex-survival-2026-10-22/PLAN.md`, R-232, R-1227, R-1241, the two homelab rules and the Alertmanager +route, homelab HM-040…HM-049 and the window record; design `08` §9. Evidence: `audits/dooplex-alarms-2026-10-24/`. + +## Three lines + +1. **Both alarms had one cause, and both are fixed.** Yesterday's reboot restarted the database, and it now takes the + password of the other of its two secrets; the dump job could not log in, and the Gitea off-site push then refused + the old dump. The job now reads the right secret. Cleared: the dump alarm 10:44Z, the Gitea alarm 10:51Z. +2. **Nothing else fires now.** In 7 days only the pulled disk's two alarms fired long (silenced by you since 10-12, + ended by the pull); the rest were minutes long or are known. +3. **Fewer mails:** critical every 4 h (was 1 h), warning every 12 h (was 4 h), all backup alarms in one mail. + +## What I asked you, and what you said + +- The dump fix → **the CronJob only** (the database's own definition untouched). +- The mail change needs an Alertmanager restart, not a reload → **restart**. +- The R-1250 proof needed a downloader on another machine → **use DooPlex's k3s qBittorrent**. + +## The alarm table (7 days, Prometheus `ALERTS`; nothing firing at 10:51Z) + +| Alarm | Severity | When (UTC) | True? | Cause | Fix / row | +|---|---|---|---|---|---| +| `DooplexGiteaOffsiteStale` | critical | 10-24 00:57 → 10:51 | true | the push refused a 12 h old dump | the dump fix; R-232 note | +| `PostgresqlDumpStale` | warning | 10-23 22:17 → 10-24 10:44 | true | dump job login failed (28P01) | homelab `531023d` | +| `DiskOfflineUncorrectable`, `DiskUncorrectableSectors` (Z6A07P2G) | critical | whole window (≥ 145 h) → 10-23 12:02 | true | the dying HDD4 | silenced by you 10-12; pulled 10-23 — R-934 closed | +| `DiskDisappeared` (Z6A07P2G) | critical | 10-23 12:17 → 18:02 | true | the pull | expected; cleared | +| `DiskInventoryChanged` | warning | 10-23 12:17 → 20:17 | true | the pull | expected; cleared | +| `LonghornVolumeDegraded` (two Plex volumes) | warning | 10-23 08:22 → 11:22 | true | the HM-040 rebuild | cleared | +| `PodRestartingFrequently` (felhom-webpage) | warning | 10-19 09:02 → 10:07 | true | git-syncs stopped on errors | R-1163 (closed 10-19) | +| `WanUploadDegraded`, `WanDownloadDegraded`, `WanUpstreamPacketLoss`, `WanBufferbloat` | warning/info | minutes, 10-17 → 10-20 | true | short WAN dips (the flood days) | none needed | +| `AuthentikHighLatency` | warning | 10-21 08:12, 5 min | true | a short spike | none needed | + +## The mail change (homelab `22d7697`, applied by a restart, read back from the running Alertmanager) + +| | before | after | +|---|---|---| +| critical | every 1 h | every 4 h | +| warning | every 4 h | every 12 h | +| other (info) | every 4 h | every 12 h | +| backup alarms | one mail stream per alarm | one mail for the group, every 4 h | +| a disk | one stream per alarm | one mail per drive, daily (unchanged) | +| resolved mail | `send_resolved: true` | unchanged; the mail counter rose once after each clear (10:47, and by 10:57) — that they read „RESOLVED" is not seen: the mailbox is not mine | + +## Part 4 — R-1250, committed and proven + +Agent `29bcf44` + `1e087bb`, controller `046b808` (commit only; release in the next train). A seeding box: the agent +says „waiting for quiet"; the controller records the torrent app's limit, holds it at 1 KiB/s, asks for one test, +puts the exact limit back. **Proven on demo-hp with a real seed** (10 Mbit/s from 9202 to DooPlex's qBittorrent): +refused without the hold (11.1 Mbit/s); with it the uplink fell to 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, the limit +back to 1 249 280 B/s; a kill in the middle → the start-up check put it back. The brief said „a nested VM": NOT done — +a nested VM would need its own tunnel to ep0 to run the real test; demo-hp's host with probes did the same with the +real tunnel. The pinned agent client is unit-tested only. + +## Commits, CI, undo + +| Repo | Commit | What | CI | +|---|---|---|---| +| homelab-manifests | `531023d` | the dump CronJob's secret | no CI in this repo | +| homelab-manifests | `22d7697` | the mail routes, the dump alarm's label | no CI | +| homelab-manifests | `9f399fe` | REPORT | no CI | +| felhom-agent | `29bcf44`, `1e087bb` | R-1250 | 2286, 2288 success | +| felhom-controller | `046b808` | R-1250 | 2287 success | +| app-catalog-felhom.eu | `7de554e`, `5e196e1` | the box walk: HDD_PATH, the leftover check | 2289 success, (final push) | +| felhom-agent / felhom-controller | `5e05627` / `321df1f` | REPORT, README | (final push) | +| felhom.eu | this commit | register, STATUS, CONTEXT, `08` §9, evidence | (final push) | + +Undo: `audits/dooplex-alarms-2026-10-24/README.md` § Undo. Backups of the three homelab files: `*.before` there. + +## Rows + +**Before 191, after 190; opened 0, closed 1 (R-934 — the disk was already pulled).** R-232: a dated note and a dated +check (2026-10-25, tonight's sweep). R-1250: NARROWED (fixed on main, waits for the release train). + +## Observations + +- The database's two secrets for one role stay (only the CronJob moved) — homelab list item 1 (not a Felhom row). +- A stale dump stops the Gitea + secrets off-site push for as long as it lasts — NOT-A-FINDING: the designed fail-closed + order (`PLAN.md`: the dump first, for consistency); the alarm caught it inside its 26 h, and it now comes in one mail + with its cause. +- Alertmanager cannot reload a ConfigMap change (the brief assumed it could) — NOT-A-FINDING: the homelab manifest's + design (`envsubst` init container); you chose the restart; the fact is in CC's memory and the homelab list. +- The agent said „waiting for quiet" before it had watched a minute — fixed in the session (`1e087bb`, found live). +- The box walk sent a folder inside the drive as HDD_PATH (refused since R-839) — fixed in the session (`7de554e`). +- The box walk called the template folder a leftover after every remove — fixed in the session (`5e196e1`). +- R-934 said the failed disk was still in DooPlex; STATUS still listed R-1214's done check — fixed (row closed, line removed). +- CC's memory said Prometheus has no config-reloader; it has one since 2026-10-12 — fixed (memory note). +- `govulncheck` warns GO-2026-6809 (crypto/x509, stdlib) in the agent and the controller — NOT-A-FINDING here: the Go + toolchain row R-972 owns it. +- My Gmail connector's mailbox gets no Alertmanager mail — NOT-A-FINDING: it is another mailbox than `SMTP_TO`; 60 sent, + 0 failed in 7 days. The Alertmanager config did not change yesterday (last 10-12). +- `kubectl exec -i` never closed stdin for `pg_restore` — NOT-A-FINDING: my method; the host's own `pg_restore` read the dumps. +- The 9202 ISO torrent never uploaded (behind NAT, 18 leechers / 1080 seeds) — NOT-A-FINDING: why the LAN peer was used. +- I overwrote homelab-manifests' REPORT.md, which held another session's HM-050 report from today — NOT-A-FINDING: the + repo's overwrite rule; HM-050's entry stays in its CHANGELOG. + +## Teardown, three layers + +- **Machine:** 9202 — qBittorrent removed through the product (no container, no volume; the template folder stays, as for + every app); the test files, the forwarder and the probe removed. DooPlex's qBittorrent — the test torrent and its + files deleted (by hash), nothing else touched. +- **Host:** demo-hp — the agent probe stopped and deleted, no iperf3 left; the installed agent untouched. DooPlex — the + 9202 password copy and the helper's session files shredded; the clones removed. +- **Hub:** nothing provisioned. + +## Instruction files, poster + +No instruction file edited. No poster fact changed (the mail intervals are not on it). diff --git a/STATUS.md b/STATUS.md index 5b93a27b..d7d0e956 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-10-24 (day, attended, done).** Released on your yeses: hub 0.168.0 (tokens locked) and 0.169.0, agent 0.164.0 (vouched), controller 0.342.0 (the floor). Each box tests its upload weekly (94, 94, 93 Mbit/s); a hacked app that uploads or computes more than it explains alarms only you, and qBittorrent gets half the tested line. The hub is out of DooPlex's open backup folder. +**Updated 2026-10-24 (afternoon, attended).** The two DooPlex backup alarms had one cause: yesterday's reboot changed which password the database takes, so the database copy job could not log in, and the Gitea off-site copy then refused its old copy. Fixed and cleared at 12:51 local; critical alarms now repeat every 4 hours (all backup alarms in one mail), warnings every 12. A box whose torrent app seeds all day can now get its weekly line test (proven on demo-hp, not yet released). ## What works - **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241). @@ -20,10 +20,10 @@ ## What is broken - **P1:** any hacked app on DooPlex can read the hub's nightly backups, and maybe the signing keys (R-1241, homelab session); about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours). - Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). The scratch box cannot use agent features (R-1083). -- DooPlex: the failed disk is still in the box, its alarms silenced (R-934); something sets the shared checkout to „bare" (R-935). +- DooPlex: something sets the shared checkout to „bare" (R-935). ## What we work on -- Dated checks, run by CC. 2026-10-24: the 8 GB box's first night copy on 0.340.0 (R-1214). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes). +- Dated checks, run by CC. 2026-10-25: tonight's database copy reaches the restic sweep (R-232). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes). ## Waiting on you — decide @@ -51,10 +51,9 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te 1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens. 2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef. If not: two dead entries alarm. 3. D37 and D38 in the table (two small yes/no). If not: Gitea may restart in release hours; no golden in the release copy. -4. Give the homelab session R-1241, R-1242, the 7-item list at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md`, and an upload alarm on DooPlex's `eno1` (the line is ~100 Mbit/s; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups. +4. Give the homelab session R-1241, R-1242, the 7-item list at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md`, the 5-item list at the end of `documentation/audits/dooplex-alarms-2026-10-24/README.md` (the database's two passwords), and an upload alarm on DooPlex's `eno1` (the line is ~100 Mbit/s; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups. **The rest** - System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy. -- Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12. - Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09. - Two DooPlex leftovers: `leftovers/2026-10-14/first-post/` (keep or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). Since 2026-10-14. - Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784). diff --git a/documentation/architecture/08-alarm-ladder.md b/documentation/architecture/08-alarm-ladder.md index fc90d93b..66650610 100644 --- a/documentation/architecture/08-alarm-ladder.md +++ b/documentation/architecture/08-alarm-ladder.md @@ -564,9 +564,17 @@ normal use; work it does not explain is the sign of a hacked app. Numbers and me household is told only about the qBittorrent limit, because they can change it (R-1246 asks the operator). 5. **qBittorrent's default upload limit:** 50 % of the TESTED line (D41 = 50 %, operator 2026-10-24; never of the push floor), every 15 minutes, re-set when the line moves > 20 %; never over a limit the household set (one there - first, or any change from the value qBittorrent stored for the box — it keeps KiB/s). No test → none set. **Known - gap (R-1250):** a qBittorrent that seeds all day keeps the uplink from ever being quiet for 10 minutes, so that box is - never tested and gets no limit — the case the limit exists for. While the box owns it, qBittorrent's page says: „A qBittorrent + first, or any change from the value qBittorrent stored for the box — it keeps KiB/s). No test → none set. **A box + that seeds all day (R-1250, brief 2026-10-24; committed agent `29bcf44`+`1e087bb`, controller `046b808`, not yet + released):** the 10 quiet minutes never come, so when the agent reports `waiting_for_quiet` (due, in the window, + tunnel up, the full 10-minute window watched and busy) and qBittorrent or Transmission runs, the controller writes + `upwatch.line_test_hold` (the app's limit exactly as stored) FIRST, holds the app at 1 KiB/s (Transmission 1 kB/s), + calls `POST /host/line-test/run` (5 s settle, 10 s quiet check < 2 Mbit/s, three runs 5 s apart — about 55 s), then + puts the old limit back, the household's own included; a change made during the test stays; a killed controller's + start-up check puts it back; the 50 % rule waits while a record exists. Measured on demo-hp with a real seed (10 + Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; with it the uplink fell to 0.02–0.16 Mbit/s, + 94.2 Mbit/s measured, the app held 56.8 s and back to 1 249 280 B/s exactly (`audits/dooplex-alarms-2026-10-24/`). + While the box owns it, qBittorrent's page says: „A qBittorrent feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod." 6. **Limits, stated:** an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood diff --git a/documentation/audits/dooplex-alarms-2026-10-24/README.md b/documentation/audits/dooplex-alarms-2026-10-24/README.md new file mode 100644 index 00000000..45aaed6f --- /dev/null +++ b/documentation/audits/dooplex-alarms-2026-10-24/README.md @@ -0,0 +1,51 @@ +# Two DooPlex backup alarms, and every alarm the mail route delivers — 2026-10-24 (attended) + +Brief: „two DooPlex backup alarms — find why, fix, and check every other alarm the new mail route now delivers". +Read first: `runbooks/total-loss-of-dooplex.md`, `runbooks/RUNBOOK-hub-db-offsite-backup.md`, +`audits/dooplex-survival-2026-10-22/PLAN.md`, R-232, the two rules in homelab `mon-system/prometheus-rules.yaml`, the +Alertmanager route, homelab HM-040…HM-049 and the 2026-10-23 window record. + +| File | What it shows | +|---|---| +| `part1-2-diagnosis.txt` | The off-site job's `FAILED: newest complete dump … 12 h old`; the dump jobs (3 failed), the failed dump folders (only an empty `globals.sql`); the database log's `28P01` (48 failed logins in 24 h, all from the backup pods, first 2026-10-23 16:00:01Z); which secret the role accepts (ok/refused only); the ruled-out causes (tunnel, block list, disk, overlap) | +| `part1-2-fix.txt` | After homelab `531023d`: the by-hand dump (13 files, `pg_restore --list` 1114 / 1660 / 753 entries), the CronJob's success time, the by-hand off-site push, the restore test (26 456 files, 12 repos `git fsck`), the snapshot on ep0, the clear times | +| `part3-alarms.txt`, `alerts-7d.json` | No alarm firing at 10:51Z; 7 days of `ALERTS` (Prometheus' whole window); the mail counter (60 sent, 0 failed); the silences; the routing read back from the running Alertmanager | +| `*.before` | The three homelab files as they were before this session's edits | +| `part4-*.txt` | R-1250's live proof (below) | + +## Why the two alarms fired — one cause + +1. The 2026-10-23 window reboot restarted `postgresql-2` (13:50Z). +2. CNPG 1.29 sets the role `postgres` twice at each instance start (`refreshCredentialsFromSecret`): first from the + superuser secret `postgresql-superuser`, then from the `bootstrap.initdb` owner secret `postgres-superuser` + (`owner: postgres`). The second wins. +3. The dump CronJob read `postgresql-superuser` → `password authentication failed` from 16:00Z → `PostgresqlDumpStale` + (22:15Z). +4. The Gitea off-site push at 00:20 refused the 12 h old Gitea dump, as designed → `DooplexGiteaOffsiteStale` (00:57Z). + +Fix (operator's pick): the CronJob reads `postgres-superuser` (homelab `531023d`, ArgoCD sync of that one CronJob). +`PostgresqlDumpStale` last firing sample 10:44:00Z; `DooplexGiteaOffsiteStale` 10:51:00Z. + +## Undo + +- CronJob: `git -C homelab-manifests revert 531023d`, then sync `CronJob/postgresql-backup` in app `database-system`. +- Routing: `git -C homelab-manifests revert 22d7697`, sync ConfigMaps `alertmanager-config-template` and + `prometheus-rules` in app `monitoring`, `curl -XPOST http://10.43.75.209:9090/-/reload`, + `sudo kubectl -n mon-system rollout restart deploy/alertmanager`. + +## For the homelab session (homelab findings — no HM numbers minted here) + +1. **The two-writer race on the role `postgres` stays.** The Cluster still names two secrets for one role; only the + CronJob was moved (operator's pick). If `postgresql-superuser` is rewritten while the instance runs, the role flips + back and the dump fails again (`PostgresqlDumpStale` catches it in ~12 h). The lasting fix is + `spec.superuserSecret: {name: postgres-superuser}` on the Cluster (both writers then write one value; the password + in force does not change). +2. **The routing change made:** critical 1 h → 4 h, warning 4 h → 12 h, root 4 h → 12 h, one backup group mail + (`component: backup`, `PostgresqlDumpStale` relabelled), one mail per drive (homelab `22d7697`). Applied by a + restart: Alertmanager cannot reload a ConfigMap change (`envsubst` init container). +3. **The silence for the pulled disk Z6A07P2G** (`device=~.*_Z6A07P2G`, until 2026-11-11) matches nothing now; expire it + when convenient (Felhom R-934 closed). +4. **Long-firing alarms in the 7 days:** `DiskOfflineUncorrectable` + `DiskUncorrectableSectors` on Z6A07P2G fired the + whole window (≥ 145 h, critical) — silenced by the operator since 2026-10-12, so seen, and ended by the pull. Nothing + else fired longer than 12.4 h. +5. **The window's catch-all silence** (`alertname=~.+`, 2026-10-23 12:06–14:05Z) expired as planned; nothing to do. diff --git a/documentation/audits/dooplex-alarms-2026-10-24/alertmanager.yaml.before b/documentation/audits/dooplex-alarms-2026-10-24/alertmanager.yaml.before new file mode 100644 index 00000000..13d7c0d4 --- /dev/null +++ b/documentation/audits/dooplex-alarms-2026-10-24/alertmanager.yaml.before @@ -0,0 +1,293 @@ +--- +# ============================================================================ +# alertmanager — alert routing and email delivery +# +# Brought into this repo on 2026-08-14. Until then it ran live with NO manifest +# anywhere: CLAUDE.md listed it as one of three such workloads, which meant any +# change to its routing survived only until something recreated it, and nobody +# could tell what the running config was without reading the cluster. +# +# Credentials are NOT here and must never be. smtp_from / auth username / +# password / recipient are ${VAR} placeholders resolved at pod start by the +# envsubst init container from the alertmanager-smtp Secret, which is created +# imperatively. Only the smarthost (smtp.gmail.com:587) and require_tls are +# literal, and neither is sensitive. +# +# Two deliberate fixes applied while adopting, both flagged rather than silent: +# * envsubst was running :latest. Pinned to the digest that was actually +# running, so a restart can no longer silently change the code. +# * the init container had no resource requests/limits, against house style. +# ============================================================================ +apiVersion: v1 +kind: ConfigMap +metadata: + name: alertmanager-config-template + namespace: mon-system + labels: + app: alertmanager +data: + alertmanager.yml.tmpl: | + global: + smtp_smarthost: 'smtp.gmail.com:587' + smtp_from: '${SMTP_AUTH_USERNAME}' + smtp_auth_username: '${SMTP_AUTH_USERNAME}' + smtp_auth_password: '${SMTP_AUTH_PASSWORD}' + smtp_require_tls: true + resolve_timeout: 5m + + route: + receiver: 'email-notifications' + group_by: ['alertname', 'severity'] + group_wait: 30s + group_interval: 5m + repeat_interval: 4h + + routes: + # --- disk health ------------------------------------------------------- + # A failing drive does not improve between emails. The generic + # severity: critical route below repeats hourly, which produced ~24 mails + # a day for a single dying disk on 2026-08-13/14. Once you have seen it, + # a daily reminder is enough. + # + # The two *Growing alerts are deliberately kept frequent: they only fire + # while the SMART counters are actively climbing, i.e. something is + # degrading RIGHT NOW rather than sitting in a known-broken state. + - match: + alertname: DiskReportedUncorrectGrowing + receiver: 'email-notifications' + repeat_interval: 4h + - match: + alertname: DiskReallocatedSectorsGrowing + receiver: 'email-notifications' + repeat_interval: 4h + - match: + component: disk + receiver: 'email-notifications' + repeat_interval: 24h + # --- workload health --------------------------------------------------- + # Same reasoning as the disk route: a crashlooping pod does not fix + # itself, so the generic critical route's hourly repeat would have sent + # 27 mails for the single external-dns-pihole failure that motivated + # these alerts. 6h still reaches you four times a day, which is far more + # than the zero notifications that outage actually produced. + # + # Must stay ABOVE the severity: critical route -- Alertmanager takes the + # first matching route, so ordering is what makes this override apply. + - match: + component: workload + receiver: 'email-notifications' + repeat_interval: 6h + - match: + severity: critical + receiver: 'email-notifications' + repeat_interval: 1h + - match: + severity: warning + receiver: 'email-notifications' + repeat_interval: 4h + + receivers: + - name: 'email-notifications' + email_configs: + - to: '${SMTP_TO}' + send_resolved: true + headers: + Subject: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}' + html: | + {{ range .Alerts }} +

{{ .Status | toUpper }}: {{ .Labels.alertname }}

+

Severity: {{ .Labels.severity }}

+

Summary: {{ .Annotations.summary }}

+

Description: {{ .Annotations.description }}

+

Started: {{ .StartsAt.Format "2006-01-02 15:04:05" }}

+ {{ if .EndsAt }} +

Ended: {{ .EndsAt.Format "2006-01-02 15:04:05" }}

+ {{ end }} +
+ {{ end }} + + inhibit_rules: + - source_match: + severity: 'critical' + target_match: + severity: 'warning' + equal: ['alertname', 'volume'] +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: alertmanager-data + namespace: mon-system + labels: + app: alertmanager + # Explicit membership rather than relying on implicit default-group + # inclusion, which any stray recurring-job label would silently break. + recurring-job-group.longhorn.io/default: enabled +spec: + accessModes: + - ReadWriteOnce + # Explicit, not the cluster default: longhorn-ssd2 is the default but is the + # smallest disk on the box. + storageClassName: longhorn + resources: + requests: + storage: 1Gi +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: alertmanager + namespace: mon-system + labels: + app: alertmanager +spec: + replicas: 1 + # Recreate, not RollingUpdate: the data volume is ReadWriteOnce, so a rolling + # update would deadlock with the new pod waiting for a volume the old pod + # still holds. + strategy: + type: Recreate + selector: + matchLabels: + app: alertmanager + template: + metadata: + labels: + app: alertmanager + spec: + # R-886 (operator yes 2026-10-12, felhom 09 §3 decision 217). The image runs as nobody (65534) + # and the Longhorn volume's root is root:root 0755, so since the 2026-10-05 Longhorn restart + # every nflog/silences write failed with "permission denied" and silences died at each restart. + # fsGroup makes the kubelet give the volume to group 65534 (group-writable) at mount. + securityContext: + fsGroup: 65534 + initContainers: + # Renders the template into the real config, substituting the SMTP + # credentials from the Secret. The processed file lands in an emptyDir, + # which is why a ConfigMap edit needs a POD RESTART to take effect -- + # POST /-/reload would just re-read the already-rendered file. + - name: envsubst + image: bhgedigital/envsubst@sha256:36a92e9344ef965e9318c8d2f70dd35cad654a4615b889c6a4dd483ca1ddec7c + command: + - sh + - -c + - envsubst < /template/alertmanager.yml.tmpl > /config/alertmanager.yml + env: + - name: SMTP_AUTH_USERNAME + valueFrom: + secretKeyRef: + name: alertmanager-smtp + key: SMTP_AUTH_USERNAME + - name: SMTP_AUTH_PASSWORD + valueFrom: + secretKeyRef: + name: alertmanager-smtp + key: SMTP_AUTH_PASSWORD + - name: SMTP_TO + valueFrom: + secretKeyRef: + name: alertmanager-smtp + key: SMTP_TO + volumeMounts: + - name: config-template + mountPath: /template + - name: config-processed + mountPath: /config + resources: + requests: + cpu: 10m + memory: 16Mi + limits: + cpu: 100m + memory: 64Mi + containers: + - name: alertmanager + image: prom/alertmanager:v0.34.1 + args: + - --config.file=/etc/alertmanager/alertmanager.yml + - --storage.path=/alertmanager + - --web.external-url=http://alertmanager.home + - --cluster.listen-address= + ports: + - name: http + containerPort: 9093 + protocol: TCP + volumeMounts: + - name: config-processed + mountPath: /etc/alertmanager + - name: storage + mountPath: /alertmanager + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + cpu: 100m + memory: 128Mi + livenessProbe: + httpGet: + path: /-/healthy + port: 9093 + initialDelaySeconds: 10 + periodSeconds: 10 + readinessProbe: + httpGet: + path: /-/ready + port: 9093 + initialDelaySeconds: 5 + periodSeconds: 10 + volumes: + - name: config-template + configMap: + name: alertmanager-config-template + - name: config-processed + emptyDir: {} + # Real storage, not emptyDir. This was an emptyDir until 2026-08-14, + # which meant SILENCES DID NOT SURVIVE A RESTART -- they live in this + # volume and nothing rebuilds them. Restarting alertmanager to pick up + # a config change therefore silently un-silenced everything and + # re-notified every firing alert, which is exactly what happened when + # the disk-alert routing was deployed. + - name: storage + persistentVolumeClaim: + claimName: alertmanager-data +--- +apiVersion: v1 +kind: Service +metadata: + name: alertmanager + namespace: mon-system + labels: + app: alertmanager +spec: + type: ClusterIP + selector: + app: alertmanager + ports: + - name: http + port: 9093 + targetPort: 9093 + protocol: TCP +--- +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: alertmanager + namespace: mon-system + labels: + app: alertmanager + annotations: + external-dns.alpha.kubernetes.io/hostname: alertmanager.home +spec: + ingressClassName: nginx-internal + rules: + - host: alertmanager.home + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: alertmanager + port: + number: 9093 diff --git a/documentation/audits/dooplex-alarms-2026-10-24/alerts-7d.json b/documentation/audits/dooplex-alarms-2026-10-24/alerts-7d.json new file mode 100644 index 00000000..2cc6e9e6 --- /dev/null +++ b/documentation/audits/dooplex-alarms-2026-10-24/alerts-7d.json @@ -0,0 +1 @@ +{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"__name__":"ALERTS","alertname":"AuthentikHighLatency","alertstate":"firing","component":"authentik","severity":"warning"},"values":[[1792570323,"1"],[1792570623,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"DiskDisappeared","alertstate":"firing","component":"disk","device":"ata-ST3000VX010-2E3166_Z6A07P2G","severity":"critical"},"values":[[1792757823,"1"],[1792758123,"1"],[1792758423,"1"],[1792758723,"1"],[1792759023,"1"],[1792759323,"1"],[1792759623,"1"],[1792759923,"1"],[1792760223,"1"],[1792760523,"1"],[1792760823,"1"],[1792763823,"1"],[1792764123,"1"],[1792764423,"1"],[1792764723,"1"],[1792765023,"1"],[1792765323,"1"],[1792765623,"1"],[1792765923,"1"],[1792766223,"1"],[1792766523,"1"],[1792766823,"1"],[1792767123,"1"],[1792767423,"1"],[1792767723,"1"],[1792768023,"1"],[1792768323,"1"],[1792768623,"1"],[1792768923,"1"],[1792769223,"1"],[1792769523,"1"],[1792769823,"1"],[1792770123,"1"],[1792770423,"1"],[1792770723,"1"],[1792771023,"1"],[1792771323,"1"],[1792771623,"1"],[1792771923,"1"],[1792772223,"1"],[1792772523,"1"],[1792772823,"1"],[1792773123,"1"],[1792773423,"1"],[1792773723,"1"],[1792774023,"1"],[1792774323,"1"],[1792774623,"1"],[1792774923,"1"],[1792775223,"1"],[1792775523,"1"],[1792775823,"1"],[1792776123,"1"],[1792776423,"1"],[1792776723,"1"],[1792777023,"1"],[1792777323,"1"],[1792777623,"1"],[1792777923,"1"],[1792778223,"1"],[1792778523,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"DiskInventoryChanged","alertstate":"firing","app":"smartctl-exporter","app_kubernetes_io_name":"smartctl-exporter","component":"disk","instance":"smartctl-exporter.admin-system.svc:9633","job":"kubernetes-services","namespace":"admin-system","service":"smartctl-exporter","severity":"warning"},"values":[[1792757823,"1"],[1792758123,"1"],[1792758423,"1"],[1792758723,"1"],[1792759023,"1"],[1792759323,"1"],[1792759623,"1"],[1792759923,"1"],[1792760223,"1"],[1792760523,"1"],[1792760823,"1"],[1792763823,"1"],[1792764123,"1"],[1792764423,"1"],[1792764723,"1"],[1792765023,"1"],[1792765323,"1"],[1792765623,"1"],[1792765923,"1"],[1792766223,"1"],[1792766523,"1"],[1792766823,"1"],[1792767123,"1"],[1792767423,"1"],[1792767723,"1"],[1792768023,"1"],[1792768323,"1"],[1792768623,"1"],[1792768923,"1"],[1792769223,"1"],[1792769523,"1"],[1792769823,"1"],[1792770123,"1"],[1792770423,"1"],[1792770723,"1"],[1792771023,"1"],[1792771323,"1"],[1792771623,"1"],[1792771923,"1"],[1792772223,"1"],[1792772523,"1"],[1792772823,"1"],[1792773123,"1"],[1792773423,"1"],[1792773723,"1"],[1792774023,"1"],[1792774323,"1"],[1792774623,"1"],[1792774923,"1"],[1792775223,"1"],[1792775523,"1"],[1792775823,"1"],[1792776123,"1"],[1792776423,"1"],[1792776723,"1"],[1792777023,"1"],[1792777323,"1"],[1792777623,"1"],[1792777923,"1"],[1792778223,"1"],[1792778523,"1"],[1792778823,"1"],[1792779123,"1"],[1792779423,"1"],[1792779723,"1"],[1792780023,"1"],[1792780323,"1"],[1792780623,"1"],[1792780923,"1"],[1792781223,"1"],[1792781523,"1"],[1792781823,"1"],[1792782123,"1"],[1792782423,"1"],[1792782723,"1"],[1792783023,"1"],[1792783323,"1"],[1792783623,"1"],[1792783923,"1"],[1792784223,"1"],[1792784523,"1"],[1792784823,"1"],[1792785123,"1"],[1792785423,"1"],[1792785723,"1"],[1792786023,"1"],[1792786323,"1"],[1792786623,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"DiskOfflineUncorrectable","alertstate":"firing","component":"disk","device":"ata-ST3000VX010-2E3166_Z6A07P2G","severity":"critical"},"values":[[1792234323,"1"],[1792234623,"1"],[1792234923,"1"],[1792235223,"1"],[1792235523,"1"],[1792235823,"1"],[1792236123,"1"],[1792236423,"1"],[1792236723,"1"],[1792237023,"1"],[1792237323,"1"],[1792237623,"1"],[1792237923,"1"],[1792238223,"1"],[1792238523,"1"],[1792238823,"1"],[1792239123,"1"],[1792239423,"1"],[1792239723,"1"],[1792240023,"1"],[1792240323,"1"],[1792240623,"1"],[1792240923,"1"],[1792241223,"1"],[1792241523,"1"],[1792241823,"1"],[1792242123,"1"],[1792242423,"1"],[1792242723,"1"],[1792243023,"1"],[1792243323,"1"],[1792243623,"1"],[1792243923,"1"],[1792244223,"1"],[1792244523,"1"],[1792244823,"1"],[1792245123,"1"],[1792245423,"1"],[1792245723,"1"],[1792246023,"1"],[1792246323,"1"],[1792246623,"1"],[1792246923,"1"],[1792247223,"1"],[1792247523,"1"],[1792247823,"1"],[1792248123,"1"],[1792248423,"1"],[1792248723,"1"],[1792249023,"1"],[1792249323,"1"],[1792249623,"1"],[1792249923,"1"],[1792250223,"1"],[1792250523,"1"],[1792250823,"1"],[1792251123,"1"],[1792251423,"1"],[1792251723,"1"],[1792252023,"1"],[1792252323,"1"],[1792252623,"1"],[1792252923,"1"],[1792253223,"1"],[1792253523,"1"],[1792253823,"1"],[1792254123,"1"],[1792254423,"1"],[1792254723,"1"],[1792255023,"1"],[1792255323,"1"],[1792255623,"1"],[1792255923,"1"],[1792256223,"1"],[1792256523,"1"],[1792256823,"1"],[1792257123,"1"],[1792257423,"1"],[1792257723,"1"],[1792258023,"1"],[1792258323,"1"],[1792258623,"1"],[1792258923,"1"],[1792259223,"1"],[1792259523,"1"],[1792259823,"1"],[1792260123,"1"],[1792260423,"1"],[1792260723,"1"],[1792261023,"1"],[1792261323,"1"],[1792261623,"1"],[1792261923,"1"],[1792262223,"1"],[1792262523,"1"],[1792262823,"1"],[1792263123,"1"],[1792263423,"1"],[1792263723,"1"],[1792264023,"1"],[1792264323,"1"],[1792264623,"1"],[1792264923,"1"],[1792265223,"1"],[1792265523,"1"],[1792265823,"1"],[1792266123,"1"],[1792266423,"1"],[1792266723,"1"],[1792267023,"1"],[1792267323,"1"],[1792267623,"1"],[1792267923,"1"],[1792268223,"1"],[1792268523,"1"],[1792268823,"1"],[1792269123,"1"],[1792269423,"1"],[1792269723,"1"],[1792270023,"1"],[1792270323,"1"],[1792270623,"1"],[1792270923,"1"],[1792271223,"1"],[1792271523,"1"],[1792271823,"1"],[1792272123,"1"],[1792272423,"1"],[1792272723,"1"],[1792273023,"1"],[1792273323,"1"],[1792273623,"1"],[1792273923,"1"],[1792274223,"1"],[1792274523,"1"],[1792274823,"1"],[1792275123,"1"],[1792275423,"1"],[1792275723,"1"],[1792276023,"1"],[1792276323,"1"],[1792276623,"1"],[1792276923,"1"],[1792277223,"1"],[1792277523,"1"],[1792277823,"1"],[1792278123,"1"],[1792278423,"1"],[1792278723,"1"],[1792279023,"1"],[1792279323,"1"],[1792279623,"1"],[1792279923,"1"],[1792280223,"1"],[1792280523,"1"],[1792280823,"1"],[1792281123,"1"],[1792281423,"1"],[1792281723,"1"],[1792282023,"1"],[1792282323,"1"],[1792282623,"1"],[1792282923,"1"],[1792283223,"1"],[1792283523,"1"],[1792283823,"1"],[1792284123,"1"],[1792284423,"1"],[1792284723,"1"],[1792285023,"1"],[1792285323,"1"],[1792285623,"1"],[1792285923,"1"],[1792286223,"1"],[1792286523,"1"],[1792286823,"1"],[1792287123,"1"],[1792287423,"1"],[1792287723,"1"],[1792288023,"1"],[1792288323,"1"],[1792288623,"1"],[1792288923,"1"],[1792289223,"1"],[1792289523,"1"],[1792289823,"1"],[1792290123,"1"],[1792290423,"1"],[1792290723,"1"],[1792291023,"1"],[1792291323,"1"],[1792291623,"1"],[1792291923,"1"],[1792292223,"1"],[1792292523,"1"],[1792292823,"1"],[1792293123,"1"],[1792293423,"1"],[1792293723,"1"],[1792294023,"1"],[1792294323,"1"],[1792294623,"1"],[1792294923,"1"],[1792295223,"1"],[1792295523,"1"],[1792295823,"1"],[1792296123,"1"],[1792296423,"1"],[1792296723,"1"],[1792297023,"1"],[1792297323,"1"],[1792297623,"1"],[1792297923,"1"],[1792298223,"1"],[1792298523,"1"],[1792298823,"1"],[1792299123,"1"],[1792299423,"1"],[1792299723,"1"],[1792300023,"1"],[1792300323,"1"],[1792300623,"1"],[1792300923,"1"],[1792301223,"1"],[1792301523,"1"],[1792301823,"1"],[1792302123,"1"],[1792302423,"1"],[1792302723,"1"],[1792303023,"1"],[1792303323,"1"],[1792303623,"1"],[1792303923,"1"],[1792304223,"1"],[1792304523,"1"],[1792304823,"1"],[1792305123,"1"],[1792305423,"1"],[1792305723,"1"],[1792306023,"1"],[1792306323,"1"],[1792306623,"1"],[1792306923,"1"],[1792307223,"1"],[1792307523,"1"],[1792307823,"1"],[1792308123,"1"],[1792308423,"1"],[1792308723,"1"],[1792309023,"1"],[1792309323,"1"],[1792309623,"1"],[1792309923,"1"],[1792310223,"1"],[1792310523,"1"],[1792310823,"1"],[1792311123,"1"],[1792311423,"1"],[1792311723,"1"],[1792312023,"1"],[1792312323,"1"],[1792312623,"1"],[1792312923,"1"],[1792313223,"1"],[1792313523,"1"],[1792313823,"1"],[1792314123,"1"],[1792314423,"1"],[1792314723,"1"],[1792315023,"1"],[1792315323,"1"],[1792315623,"1"],[1792315923,"1"],[1792316223,"1"],[1792316523,"1"],[1792316823,"1"],[1792317123,"1"],[1792317423,"1"],[1792317723,"1"],[1792318023,"1"],[1792318323,"1"],[1792318623,"1"],[1792318923,"1"],[1792319223,"1"],[1792319523,"1"],[1792319823,"1"],[1792320123,"1"],[1792320423,"1"],[1792320723,"1"],[1792321023,"1"],[1792321323,"1"],[1792321623,"1"],[1792321923,"1"],[1792322223,"1"],[1792322523,"1"],[1792322823,"1"],[1792323123,"1"],[1792323423,"1"],[1792323723,"1"],[1792324023,"1"],[1792324323,"1"],[1792324623,"1"],[1792324923,"1"],[1792325223,"1"],[1792325523,"1"],[1792325823,"1"],[1792326123,"1"],[1792326423,"1"],[1792326723,"1"],[1792327023,"1"],[1792327323,"1"],[1792327623,"1"],[1792327923,"1"],[1792328223,"1"],[1792328523,"1"],[1792328823,"1"],[1792329123,"1"],[1792329423,"1"],[1792329723,"1"],[1792330023,"1"],[1792330323,"1"],[1792330623,"1"],[1792330923,"1"],[1792331223,"1"],[1792331523,"1"],[1792331823,"1"],[1792332123,"1"],[1792332423,"1"],[1792332723,"1"],[1792333023,"1"],[1792333323,"1"],[1792333623,"1"],[1792333923,"1"],[1792334223,"1"],[1792334523,"1"],[1792334823,"1"],[1792335123,"1"],[1792335423,"1"],[1792335723,"1"],[1792336023,"1"],[1792336323,"1"],[1792336623,"1"],[1792336923,"1"],[1792337223,"1"],[1792337523,"1"],[1792337823,"1"],[1792338123,"1"],[1792338423,"1"],[1792338723,"1"],[1792339023,"1"],[1792339323,"1"],[1792339623,"1"],[1792339923,"1"],[1792340223,"1"],[1792340523,"1"],[1792340823,"1"],[1792341123,"1"],[1792341423,"1"],[1792341723,"1"],[1792342023,"1"],[1792342323,"1"],[1792342623,"1"],[1792342923,"1"],[1792343223,"1"],[1792343523,"1"],[1792343823,"1"],[1792344123,"1"],[1792344423,"1"],[1792344723,"1"],[1792345023,"1"],[1792345323,"1"],[1792345623,"1"],[1792345923,"1"],[1792346223,"1"],[1792346523,"1"],[1792346823,"1"],[1792347123,"1"],[1792347423,"1"],[1792347723,"1"],[1792348023,"1"],[1792348323,"1"],[1792348623,"1"],[1792348923,"1"],[1792349223,"1"],[1792349523,"1"],[1792349823,"1"],[1792350123,"1"],[1792350423,"1"],[1792350723,"1"],[1792351023,"1"],[1792351323,"1"],[1792351623,"1"],[1792351923,"1"],[1792352223,"1"],[1792352523,"1"],[1792352823,"1"],[1792353123,"1"],[1792353423,"1"],[1792353723,"1"],[1792354023,"1"],[1792354323,"1"],[1792354623,"1"],[1792354923,"1"],[1792355223,"1"],[1792355523,"1"],[1792355823,"1"],[1792356123,"1"],[1792356423,"1"],[1792356723,"1"],[1792357023,"1"],[1792357323,"1"],[1792357623,"1"],[1792357923,"1"],[1792358223,"1"],[1792358523,"1"],[1792358823,"1"],[1792359123,"1"],[1792359423,"1"],[1792359723,"1"],[1792360023,"1"],[1792360323,"1"],[1792360623,"1"],[1792360923,"1"],[1792361223,"1"],[1792361523,"1"],[1792361823,"1"],[1792362123,"1"],[1792362423,"1"],[1792362723,"1"],[1792363023,"1"],[1792363323,"1"],[1792363623,"1"],[1792363923,"1"],[1792364223,"1"],[1792364523,"1"],[1792364823,"1"],[1792365123,"1"],[1792365423,"1"],[1792365723,"1"],[1792366023,"1"],[1792366323,"1"],[1792366623,"1"],[1792366923,"1"],[1792367223,"1"],[1792367523,"1"],[1792367823,"1"],[1792368123,"1"],[1792368423,"1"],[1792368723,"1"],[1792369023,"1"],[1792369323,"1"],[1792369623,"1"],[1792369923,"1"],[1792370223,"1"],[1792370523,"1"],[1792370823,"1"],[1792371123,"1"],[1792371423,"1"],[1792371723,"1"],[1792372023,"1"],[1792372323,"1"],[1792372623,"1"],[1792372923,"1"],[1792373223,"1"],[1792373523,"1"],[1792373823,"1"],[1792374123,"1"],[1792374423,"1"],[1792374723,"1"],[1792375023,"1"],[1792375323,"1"],[1792375623,"1"],[1792375923,"1"],[1792376223,"1"],[1792376523,"1"],[1792376823,"1"],[1792377123,"1"],[1792377423,"1"],[1792377723,"1"],[1792378023,"1"],[1792378323,"1"],[1792378623,"1"],[1792378923,"1"],[1792379223,"1"],[1792379523,"1"],[1792379823,"1"],[1792380123,"1"],[1792380423,"1"],[1792380723,"1"],[1792381023,"1"],[1792381323,"1"],[1792381623,"1"],[1792381923,"1"],[1792382223,"1"],[1792382523,"1"],[1792382823,"1"],[1792383123,"1"],[1792383423,"1"],[1792383723,"1"],[1792384023,"1"],[1792384323,"1"],[1792384623,"1"],[1792384923,"1"],[1792385223,"1"],[1792385523,"1"],[1792385823,"1"],[1792386123,"1"],[1792386423,"1"],[1792386723,"1"],[1792387023,"1"],[1792387323,"1"],[1792387623,"1"],[1792387923,"1"],[1792388223,"1"],[1792388523,"1"],[1792388823,"1"],[1792389123,"1"],[1792389423,"1"],[1792389723,"1"],[1792390023,"1"],[1792390323,"1"],[1792390623,"1"],[1792390923,"1"],[1792391223,"1"],[1792391523,"1"],[1792391823,"1"],[1792392123,"1"],[1792392423,"1"],[1792392723,"1"],[1792393023,"1"],[1792393323,"1"],[1792393623,"1"],[1792393923,"1"],[1792394223,"1"],[1792394523,"1"],[1792394823,"1"],[1792395123,"1"],[1792395423,"1"],[1792395723,"1"],[1792396023,"1"],[1792396323,"1"],[1792396623,"1"],[1792396923,"1"],[1792397223,"1"],[1792397523,"1"],[1792397823,"1"],[1792398123,"1"],[1792398423,"1"],[1792398723,"1"],[1792399023,"1"],[1792399323,"1"],[1792399623,"1"],[1792399923,"1"],[1792400223,"1"],[1792400523,"1"],[1792400823,"1"],[1792401123,"1"],[1792401423,"1"],[1792401723,"1"],[1792402023,"1"],[1792402323,"1"],[1792402623,"1"],[1792402923,"1"],[1792403223,"1"],[1792403523,"1"],[1792403823,"1"],[1792404123,"1"],[1792404423,"1"],[1792404723,"1"],[1792405023,"1"],[1792405323,"1"],[1792405623,"1"],[1792405923,"1"],[1792406223,"1"],[1792406523,"1"],[1792406823,"1"],[1792407123,"1"],[1792407423,"1"],[1792407723,"1"],[1792408023,"1"],[1792408323,"1"],[1792408623,"1"],[1792408923,"1"],[1792409223,"1"],[1792409523,"1"],[1792409823,"1"],[1792410123,"1"],[1792410423,"1"],[1792410723,"1"],[1792411023,"1"],[1792411323,"1"],[1792411623,"1"],[1792411923,"1"],[1792412223,"1"],[1792412523,"1"],[1792412823,"1"],[1792413123,"1"],[1792413423,"1"],[1792413723,"1"],[1792414023,"1"],[1792414323,"1"],[1792414623,"1"],[1792414923,"1"],[1792415223,"1"],[1792415523,"1"],[1792415823,"1"],[1792416123,"1"],[1792416423,"1"],[1792416723,"1"],[1792417023,"1"],[1792417323,"1"],[1792417623,"1"],[1792417923,"1"],[1792418223,"1"],[1792418523,"1"],[1792418823,"1"],[1792419123,"1"],[1792419423,"1"],[1792419723,"1"],[1792420023,"1"],[1792420323,"1"],[1792420623,"1"],[1792420923,"1"],[1792421223,"1"],[1792421523,"1"],[1792421823,"1"],[1792422123,"1"],[1792422423,"1"],[1792422723,"1"],[1792423023,"1"],[1792423323,"1"],[1792423623,"1"],[1792423923,"1"],[1792424223,"1"],[1792424523,"1"],[1792424823,"1"],[1792425123,"1"],[1792425423,"1"],[1792425723,"1"],[1792426023,"1"],[1792426323,"1"],[1792426623,"1"],[1792426923,"1"],[1792427223,"1"],[1792427523,"1"],[1792427823,"1"],[1792428123,"1"],[1792428423,"1"],[1792428723,"1"],[1792429023,"1"],[1792429323,"1"],[1792429623,"1"],[1792429923,"1"],[1792430223,"1"],[1792430523,"1"],[1792430823,"1"],[1792431123,"1"],[1792431423,"1"],[1792431723,"1"],[1792432023,"1"],[1792432323,"1"],[1792432623,"1"],[1792432923,"1"],[1792433223,"1"],[1792433523,"1"],[1792433823,"1"],[1792434123,"1"],[1792434423,"1"],[1792434723,"1"],[1792435023,"1"],[1792435323,"1"],[1792435623,"1"],[1792435923,"1"],[1792436223,"1"],[1792436523,"1"],[1792436823,"1"],[1792437123,"1"],[1792437423,"1"],[1792437723,"1"],[1792438023,"1"],[1792438323,"1"],[1792438623,"1"],[1792438923,"1"],[1792439223,"1"],[1792439523,"1"],[1792439823,"1"],[1792440123,"1"],[1792440423,"1"],[1792440723,"1"],[1792441023,"1"],[1792441323,"1"],[1792441623,"1"],[1792441923,"1"],[1792442223,"1"],[1792442523,"1"],[1792442823,"1"],[1792443123,"1"],[1792443423,"1"],[1792443723,"1"],[1792444023,"1"],[1792444323,"1"],[1792444623,"1"],[1792444923,"1"],[1792445223,"1"],[1792445523,"1"],[1792445823,"1"],[1792446123,"1"],[1792446423,"1"],[1792446723,"1"],[1792447023,"1"],[1792447323,"1"],[1792447623,"1"],[1792447923,"1"],[1792448223,"1"],[1792448523,"1"],[1792448823,"1"],[1792449123,"1"],[1792449423,"1"],[1792449723,"1"],[1792450023,"1"],[1792450323,"1"],[1792450623,"1"],[1792450923,"1"],[1792451223,"1"],[1792451523,"1"],[1792451823,"1"],[1792452123,"1"],[1792452423,"1"],[1792452723,"1"],[1792453023,"1"],[1792453323,"1"],[1792453623,"1"],[1792453923,"1"],[1792454223,"1"],[1792454523,"1"],[1792454823,"1"],[1792455123,"1"],[1792455423,"1"],[1792455723,"1"],[1792456023,"1"],[1792456323,"1"],[1792456623,"1"],[1792456923,"1"],[1792457223,"1"],[1792457523,"1"],[1792457823,"1"],[1792458123,"1"],[1792458423,"1"],[1792458723,"1"],[1792459023,"1"],[1792459323,"1"],[1792459623,"1"],[1792459923,"1"],[1792460223,"1"],[1792460523,"1"],[1792460823,"1"],[1792461123,"1"],[1792461423,"1"],[1792461723,"1"],[1792462023,"1"],[1792462323,"1"],[1792462623,"1"],[1792462923,"1"],[1792463223,"1"],[1792463523,"1"],[1792463823,"1"],[1792464123,"1"],[1792464423,"1"],[1792464723,"1"],[1792465023,"1"],[1792465323,"1"],[1792465623,"1"],[1792465923,"1"],[1792466223,"1"],[1792466523,"1"],[1792466823,"1"],[1792467123,"1"],[1792467423,"1"],[1792467723,"1"],[1792468023,"1"],[1792468323,"1"],[1792468623,"1"],[1792468923,"1"],[1792469223,"1"],[1792469523,"1"],[1792469823,"1"],[1792470123,"1"],[1792470423,"1"],[1792470723,"1"],[1792471023,"1"],[1792471323,"1"],[1792471623,"1"],[1792471923,"1"],[1792472223,"1"],[1792472523,"1"],[1792472823,"1"],[1792473123,"1"],[1792473423,"1"],[1792473723,"1"],[1792474023,"1"],[1792474323,"1"],[1792474623,"1"],[1792474923,"1"],[1792475223,"1"],[1792475523,"1"],[1792475823,"1"],[1792476123,"1"],[1792476423,"1"],[1792476723,"1"],[1792477023,"1"],[1792477323,"1"],[1792477623,"1"],[1792477923,"1"],[1792478223,"1"],[1792478523,"1"],[1792478823,"1"],[1792479123,"1"],[1792479423,"1"],[1792479723,"1"],[1792480023,"1"],[1792480323,"1"],[1792480623,"1"],[1792480923,"1"],[1792481223,"1"],[1792481523,"1"],[1792481823,"1"],[1792482123,"1"],[1792482423,"1"],[1792482723,"1"],[1792483023,"1"],[1792483323,"1"],[1792483623,"1"],[1792483923,"1"],[1792484223,"1"],[1792484523,"1"],[1792484823,"1"],[1792485123,"1"],[1792485423,"1"],[1792485723,"1"],[1792486023,"1"],[1792486323,"1"],[1792486623,"1"],[1792486923,"1"],[1792487223,"1"],[1792487523,"1"],[1792487823,"1"],[1792488123,"1"],[1792488423,"1"],[1792488723,"1"],[1792489023,"1"],[1792489323,"1"],[1792489623,"1"],[1792489923,"1"],[1792490223,"1"],[1792490523,"1"],[1792490823,"1"],[1792491123,"1"],[1792491423,"1"],[1792491723,"1"],[1792492023,"1"],[1792492323,"1"],[1792492623,"1"],[1792492923,"1"],[1792493223,"1"],[1792493523,"1"],[1792493823,"1"],[1792494123,"1"],[1792494423,"1"],[1792494723,"1"],[1792495023,"1"],[1792495323,"1"],[1792495623,"1"],[1792495923,"1"],[1792496223,"1"],[1792496523,"1"],[1792496823,"1"],[1792497123,"1"],[1792497423,"1"],[1792497723,"1"],[1792498023,"1"],[1792498323,"1"],[1792498623,"1"],[1792498923,"1"],[1792499223,"1"],[1792499523,"1"],[1792499823,"1"],[1792500123,"1"],[1792500423,"1"],[1792500723,"1"],[1792501023,"1"],[1792501323,"1"],[1792501623,"1"],[1792501923,"1"],[1792502223,"1"],[1792502523,"1"],[1792502823,"1"],[1792503123,"1"],[1792503423,"1"],[1792503723,"1"],[1792504023,"1"],[1792504323,"1"],[1792504623,"1"],[1792504923,"1"],[1792505223,"1"],[1792505523,"1"],[1792505823,"1"],[1792506123,"1"],[1792506423,"1"],[1792506723,"1"],[1792507023,"1"],[1792507323,"1"],[1792507623,"1"],[1792507923,"1"],[1792508223,"1"],[1792508523,"1"],[1792508823,"1"],[1792509123,"1"],[1792509423,"1"],[1792509723,"1"],[1792510023,"1"],[1792510323,"1"],[1792510623,"1"],[1792510923,"1"],[1792511223,"1"],[1792511523,"1"],[1792511823,"1"],[1792512123,"1"],[1792512423,"1"],[1792512723,"1"],[1792513023,"1"],[1792513323,"1"],[1792513623,"1"],[1792513923,"1"],[1792514223,"1"],[1792514523,"1"],[1792514823,"1"],[1792515123,"1"],[1792515423,"1"],[1792515723,"1"],[1792516023,"1"],[1792516323,"1"],[1792516623,"1"],[1792516923,"1"],[1792517223,"1"],[1792517523,"1"],[1792517823,"1"],[1792518123,"1"],[1792518423,"1"],[1792518723,"1"],[1792519023,"1"],[1792519323,"1"],[1792519623,"1"],[1792519923,"1"],[1792520223,"1"],[1792520523,"1"],[1792520823,"1"],[1792521123,"1"],[1792521423,"1"],[1792521723,"1"],[1792522023,"1"],[1792522323,"1"],[1792522623,"1"],[1792522923,"1"],[1792523223,"1"],[1792523523,"1"],[1792523823,"1"],[1792524123,"1"],[1792524423,"1"],[1792524723,"1"],[1792525023,"1"],[1792525323,"1"],[1792525623,"1"],[1792525923,"1"],[1792526223,"1"],[1792526523,"1"],[1792526823,"1"],[1792527123,"1"],[1792527423,"1"],[1792527723,"1"],[1792528023,"1"],[1792528323,"1"],[1792528623,"1"],[1792528923,"1"],[1792529223,"1"],[1792529523,"1"],[1792529823,"1"],[1792530123,"1"],[1792530423,"1"],[1792530723,"1"],[1792531023,"1"],[1792531323,"1"],[1792531623,"1"],[1792531923,"1"],[1792532223,"1"],[1792532523,"1"],[1792532823,"1"],[1792533123,"1"],[1792533423,"1"],[1792533723,"1"],[1792534023,"1"],[1792534323,"1"],[1792534623,"1"],[1792534923,"1"],[1792535223,"1"],[1792535523,"1"],[1792535823,"1"],[1792536123,"1"],[1792536423,"1"],[1792536723,"1"],[1792537023,"1"],[1792537323,"1"],[1792537623,"1"],[1792537923,"1"],[1792538223,"1"],[1792538523,"1"],[1792538823,"1"],[1792539123,"1"],[1792539423,"1"],[1792539723,"1"],[1792540023,"1"],[1792540323,"1"],[1792540623,"1"],[1792540923,"1"],[1792541223,"1"],[1792541523,"1"],[1792541823,"1"],[1792542123,"1"],[1792542423,"1"],[1792542723,"1"],[1792543023,"1"],[1792543323,"1"],[1792543623,"1"],[1792543923,"1"],[1792544223,"1"],[1792544523,"1"],[1792544823,"1"],[1792545123,"1"],[1792545423,"1"],[1792545723,"1"],[1792546023,"1"],[1792546323,"1"],[1792546623,"1"],[1792546923,"1"],[1792547223,"1"],[1792547523,"1"],[1792547823,"1"],[1792548123,"1"],[1792548423,"1"],[1792548723,"1"],[1792549023,"1"],[1792549323,"1"],[1792549623,"1"],[1792549923,"1"],[1792550223,"1"],[1792550523,"1"],[1792550823,"1"],[1792551123,"1"],[1792551423,"1"],[1792551723,"1"],[1792552023,"1"],[1792552323,"1"],[1792552623,"1"],[1792552923,"1"],[1792553223,"1"],[1792553523,"1"],[1792553823,"1"],[1792554123,"1"],[1792554423,"1"],[1792554723,"1"],[1792555023,"1"],[1792555323,"1"],[1792555623,"1"],[1792555923,"1"],[1792556223,"1"],[1792556523,"1"],[1792556823,"1"],[1792557123,"1"],[1792557423,"1"],[1792557723,"1"],[1792558023,"1"],[1792558323,"1"],[1792558623,"1"],[1792558923,"1"],[1792559223,"1"],[1792559523,"1"],[1792559823,"1"],[1792560123,"1"],[1792560423,"1"],[1792560723,"1"],[1792561023,"1"],[1792561323,"1"],[1792561623,"1"],[1792561923,"1"],[1792562223,"1"],[1792562523,"1"],[1792562823,"1"],[1792563123,"1"],[1792563423,"1"],[1792563723,"1"],[1792564023,"1"],[1792564323,"1"],[1792564623,"1"],[1792564923,"1"],[1792565223,"1"],[1792565523,"1"],[1792565823,"1"],[1792566123,"1"],[1792566423,"1"],[1792566723,"1"],[1792567023,"1"],[1792567323,"1"],[1792567623,"1"],[1792567923,"1"],[1792568223,"1"],[1792568523,"1"],[1792568823,"1"],[1792569123,"1"],[1792569423,"1"],[1792569723,"1"],[1792570023,"1"],[1792570323,"1"],[1792570623,"1"],[1792570923,"1"],[1792571223,"1"],[1792571523,"1"],[1792571823,"1"],[1792572123,"1"],[1792572423,"1"],[1792572723,"1"],[1792573023,"1"],[1792573323,"1"],[1792573623,"1"],[1792573923,"1"],[1792574223,"1"],[1792574523,"1"],[1792574823,"1"],[1792575123,"1"],[1792575423,"1"],[1792575723,"1"],[1792576023,"1"],[1792576323,"1"],[1792576623,"1"],[1792576923,"1"],[1792577223,"1"],[1792577523,"1"],[1792577823,"1"],[1792578123,"1"],[1792578423,"1"],[1792578723,"1"],[1792579023,"1"],[1792579323,"1"],[1792579623,"1"],[1792579923,"1"],[1792580223,"1"],[1792580523,"1"],[1792580823,"1"],[1792581123,"1"],[1792581423,"1"],[1792581723,"1"],[1792582023,"1"],[1792582323,"1"],[1792582623,"1"],[1792582923,"1"],[1792583223,"1"],[1792583523,"1"],[1792583823,"1"],[1792584123,"1"],[1792584423,"1"],[1792584723,"1"],[1792585023,"1"],[1792585323,"1"],[1792585623,"1"],[1792585923,"1"],[1792586223,"1"],[1792586523,"1"],[1792586823,"1"],[1792587123,"1"],[1792587423,"1"],[1792587723,"1"],[1792588023,"1"],[1792588323,"1"],[1792588623,"1"],[1792588923,"1"],[1792589223,"1"],[1792589523,"1"],[1792589823,"1"],[1792590123,"1"],[1792590423,"1"],[1792590723,"1"],[1792591023,"1"],[1792591323,"1"],[1792591623,"1"],[1792591923,"1"],[1792592223,"1"],[1792592523,"1"],[1792592823,"1"],[1792593123,"1"],[1792593423,"1"],[1792593723,"1"],[1792594023,"1"],[1792594323,"1"],[1792594623,"1"],[1792594923,"1"],[1792595223,"1"],[1792595523,"1"],[1792595823,"1"],[1792596123,"1"],[1792596423,"1"],[1792596723,"1"],[1792597023,"1"],[1792597323,"1"],[1792597623,"1"],[1792597923,"1"],[1792598223,"1"],[1792598523,"1"],[1792598823,"1"],[1792599123,"1"],[1792599423,"1"],[1792599723,"1"],[1792600023,"1"],[1792600323,"1"],[1792600623,"1"],[1792600923,"1"],[1792601223,"1"],[1792601523,"1"],[1792601823,"1"],[1792602123,"1"],[1792602423,"1"],[1792602723,"1"],[1792603023,"1"],[1792603323,"1"],[1792603623,"1"],[1792603923,"1"],[1792604223,"1"],[1792604523,"1"],[1792604823,"1"],[1792605123,"1"],[1792605423,"1"],[1792605723,"1"],[1792606023,"1"],[1792606323,"1"],[1792606623,"1"],[1792606923,"1"],[1792607223,"1"],[1792607523,"1"],[1792607823,"1"],[1792608123,"1"],[1792608423,"1"],[1792608723,"1"],[1792609023,"1"],[1792609323,"1"],[1792609623,"1"],[1792609923,"1"],[1792610223,"1"],[1792610523,"1"],[1792610823,"1"],[1792611123,"1"],[1792611423,"1"],[1792611723,"1"],[1792612023,"1"],[1792612323,"1"],[1792612623,"1"],[1792612923,"1"],[1792613223,"1"],[1792613523,"1"],[1792613823,"1"],[1792614123,"1"],[1792614423,"1"],[1792614723,"1"],[1792615023,"1"],[1792615323,"1"],[1792615623,"1"],[1792615923,"1"],[1792616223,"1"],[1792616523,"1"],[1792616823,"1"],[1792617123,"1"],[1792617423,"1"],[1792617723,"1"],[1792618023,"1"],[1792618323,"1"],[1792618623,"1"],[1792618923,"1"],[1792619223,"1"],[1792619523,"1"],[1792619823,"1"],[1792620123,"1"],[1792620423,"1"],[1792620723,"1"],[1792621023,"1"],[1792621323,"1"],[1792621623,"1"],[1792621923,"1"],[1792622223,"1"],[1792622523,"1"],[1792622823,"1"],[1792623123,"1"],[1792623423,"1"],[1792623723,"1"],[1792624023,"1"],[1792624323,"1"],[1792624623,"1"],[1792624923,"1"],[1792625223,"1"],[1792625523,"1"],[1792625823,"1"],[1792626123,"1"],[1792626423,"1"],[1792626723,"1"],[1792627023,"1"],[1792627323,"1"],[1792627623,"1"],[1792627923,"1"],[1792628223,"1"],[1792628523,"1"],[1792628823,"1"],[1792629123,"1"],[1792629423,"1"],[1792629723,"1"],[1792630023,"1"],[1792630323,"1"],[1792630623,"1"],[1792630923,"1"],[1792631223,"1"],[1792631523,"1"],[1792631823,"1"],[1792632123,"1"],[1792632423,"1"],[1792632723,"1"],[1792633023,"1"],[1792633323,"1"],[1792633623,"1"],[1792633923,"1"],[1792634223,"1"],[1792634523,"1"],[1792634823,"1"],[1792635123,"1"],[1792635423,"1"],[1792635723,"1"],[1792636023,"1"],[1792636323,"1"],[1792636623,"1"],[1792636923,"1"],[1792637223,"1"],[1792637523,"1"],[1792637823,"1"],[1792638123,"1"],[1792638423,"1"],[1792638723,"1"],[1792639023,"1"],[1792639323,"1"],[1792639623,"1"],[1792639923,"1"],[1792640223,"1"],[1792640523,"1"],[1792640823,"1"],[1792641123,"1"],[1792641423,"1"],[1792641723,"1"],[1792642023,"1"],[1792642323,"1"],[1792642623,"1"],[1792642923,"1"],[1792643223,"1"],[1792643523,"1"],[1792643823,"1"],[1792644123,"1"],[1792644423,"1"],[1792644723,"1"],[1792645023,"1"],[1792645323,"1"],[1792645623,"1"],[1792645923,"1"],[1792646223,"1"],[1792646523,"1"],[1792646823,"1"],[1792647123,"1"],[1792647423,"1"],[1792647723,"1"],[1792648023,"1"],[1792648323,"1"],[1792648623,"1"],[1792648923,"1"],[1792649223,"1"],[1792649523,"1"],[1792649823,"1"],[1792650123,"1"],[1792650423,"1"],[1792650723,"1"],[1792651023,"1"],[1792651323,"1"],[1792651623,"1"],[1792651923,"1"],[1792652223,"1"],[1792652523,"1"],[1792652823,"1"],[1792653123,"1"],[1792653423,"1"],[1792653723,"1"],[1792654023,"1"],[1792654323,"1"],[1792654623,"1"],[1792654923,"1"],[1792655223,"1"],[1792655523,"1"],[1792655823,"1"],[1792656123,"1"],[1792656423,"1"],[1792656723,"1"],[1792657023,"1"],[1792657323,"1"],[1792657623,"1"],[1792657923,"1"],[1792658223,"1"],[1792658523,"1"],[1792658823,"1"],[1792659123,"1"],[1792659423,"1"],[1792659723,"1"],[1792660023,"1"],[1792660323,"1"],[1792660623,"1"],[1792660923,"1"],[1792661223,"1"],[1792661523,"1"],[1792661823,"1"],[1792662123,"1"],[1792662423,"1"],[1792662723,"1"],[1792663023,"1"],[1792663323,"1"],[1792663623,"1"],[1792663923,"1"],[1792664223,"1"],[1792664523,"1"],[1792664823,"1"],[1792665123,"1"],[1792665423,"1"],[1792665723,"1"],[1792666023,"1"],[1792666323,"1"],[1792666623,"1"],[1792666923,"1"],[1792667223,"1"],[1792667523,"1"],[1792667823,"1"],[1792668123,"1"],[1792668423,"1"],[1792668723,"1"],[1792669023,"1"],[1792669323,"1"],[1792669623,"1"],[1792669923,"1"],[1792670223,"1"],[1792670523,"1"],[1792670823,"1"],[1792671123,"1"],[1792671423,"1"],[1792671723,"1"],[1792672023,"1"],[1792672323,"1"],[1792672623,"1"],[1792672923,"1"],[1792673223,"1"],[1792673523,"1"],[1792673823,"1"],[1792674123,"1"],[1792674423,"1"],[1792674723,"1"],[1792675023,"1"],[1792675323,"1"],[1792675623,"1"],[1792675923,"1"],[1792676223,"1"],[1792676523,"1"],[1792676823,"1"],[1792677123,"1"],[1792677423,"1"],[1792677723,"1"],[1792678023,"1"],[1792678323,"1"],[1792678623,"1"],[1792678923,"1"],[1792679223,"1"],[1792679523,"1"],[1792679823,"1"],[1792680123,"1"],[1792680423,"1"],[1792680723,"1"],[1792681023,"1"],[1792681323,"1"],[1792681623,"1"],[1792681923,"1"],[1792682223,"1"],[1792682523,"1"],[1792682823,"1"],[1792683123,"1"],[1792683423,"1"],[1792683723,"1"],[1792684023,"1"],[1792684323,"1"],[1792684623,"1"],[1792684923,"1"],[1792685223,"1"],[1792685523,"1"],[1792685823,"1"],[1792686123,"1"],[1792686423,"1"],[1792686723,"1"],[1792687023,"1"],[1792687323,"1"],[1792687623,"1"],[1792687923,"1"],[1792688223,"1"],[1792688523,"1"],[1792688823,"1"],[1792689123,"1"],[1792689423,"1"],[1792689723,"1"],[1792690023,"1"],[1792690323,"1"],[1792690623,"1"],[1792690923,"1"],[1792691223,"1"],[1792691523,"1"],[1792691823,"1"],[1792692123,"1"],[1792692423,"1"],[1792692723,"1"],[1792693023,"1"],[1792693323,"1"],[1792693623,"1"],[1792693923,"1"],[1792694223,"1"],[1792694523,"1"],[1792694823,"1"],[1792695123,"1"],[1792695423,"1"],[1792695723,"1"],[1792696023,"1"],[1792696323,"1"],[1792696623,"1"],[1792696923,"1"],[1792697223,"1"],[1792697523,"1"],[1792697823,"1"],[1792698123,"1"],[1792698423,"1"],[1792698723,"1"],[1792699023,"1"],[1792699323,"1"],[1792699623,"1"],[1792699923,"1"],[1792700223,"1"],[1792700523,"1"],[1792700823,"1"],[1792701123,"1"],[1792701423,"1"],[1792701723,"1"],[1792702023,"1"],[1792702323,"1"],[1792702623,"1"],[1792702923,"1"],[1792703223,"1"],[1792703523,"1"],[1792703823,"1"],[1792704123,"1"],[1792704423,"1"],[1792704723,"1"],[1792705023,"1"],[1792705323,"1"],[1792705623,"1"],[1792705923,"1"],[1792706223,"1"],[1792706523,"1"],[1792706823,"1"],[1792707123,"1"],[1792707423,"1"],[1792707723,"1"],[1792708023,"1"],[1792708323,"1"],[1792708623,"1"],[1792708923,"1"],[1792709223,"1"],[1792709523,"1"],[1792709823,"1"],[1792710123,"1"],[1792710423,"1"],[1792710723,"1"],[1792711023,"1"],[1792711323,"1"],[1792711623,"1"],[1792711923,"1"],[1792712223,"1"],[1792712523,"1"],[1792712823,"1"],[1792713123,"1"],[1792713423,"1"],[1792713723,"1"],[1792714023,"1"],[1792714323,"1"],[1792714623,"1"],[1792714923,"1"],[1792715223,"1"],[1792715523,"1"],[1792715823,"1"],[1792716123,"1"],[1792716423,"1"],[1792716723,"1"],[1792717023,"1"],[1792717323,"1"],[1792717623,"1"],[1792717923,"1"],[1792718223,"1"],[1792718523,"1"],[1792718823,"1"],[1792719123,"1"],[1792719423,"1"],[1792719723,"1"],[1792720023,"1"],[1792720323,"1"],[1792720623,"1"],[1792720923,"1"],[1792721223,"1"],[1792721523,"1"],[1792721823,"1"],[1792722123,"1"],[1792722423,"1"],[1792722723,"1"],[1792723023,"1"],[1792723323,"1"],[1792723623,"1"],[1792723923,"1"],[1792724223,"1"],[1792724523,"1"],[1792724823,"1"],[1792725123,"1"],[1792725423,"1"],[1792725723,"1"],[1792726023,"1"],[1792726323,"1"],[1792726623,"1"],[1792726923,"1"],[1792727223,"1"],[1792727523,"1"],[1792727823,"1"],[1792728123,"1"],[1792728423,"1"],[1792728723,"1"],[1792729023,"1"],[1792729323,"1"],[1792729623,"1"],[1792729923,"1"],[1792730223,"1"],[1792730523,"1"],[1792730823,"1"],[1792731123,"1"],[1792731423,"1"],[1792731723,"1"],[1792732023,"1"],[1792732323,"1"],[1792732623,"1"],[1792732923,"1"],[1792733223,"1"],[1792733523,"1"],[1792733823,"1"],[1792734123,"1"],[1792734423,"1"],[1792734723,"1"],[1792735023,"1"],[1792735323,"1"],[1792735623,"1"],[1792735923,"1"],[1792736223,"1"],[1792736523,"1"],[1792736823,"1"],[1792737123,"1"],[1792737423,"1"],[1792737723,"1"],[1792738023,"1"],[1792738323,"1"],[1792738623,"1"],[1792738923,"1"],[1792739223,"1"],[1792739523,"1"],[1792739823,"1"],[1792740123,"1"],[1792740423,"1"],[1792740723,"1"],[1792741023,"1"],[1792741323,"1"],[1792741623,"1"],[1792741923,"1"],[1792742223,"1"],[1792742523,"1"],[1792742823,"1"],[1792743123,"1"],[1792743423,"1"],[1792743723,"1"],[1792744023,"1"],[1792744323,"1"],[1792744623,"1"],[1792744923,"1"],[1792745223,"1"],[1792745523,"1"],[1792745823,"1"],[1792746123,"1"],[1792746423,"1"],[1792746723,"1"],[1792747023,"1"],[1792747323,"1"],[1792747623,"1"],[1792747923,"1"],[1792748223,"1"],[1792748523,"1"],[1792748823,"1"],[1792749123,"1"],[1792749423,"1"],[1792749723,"1"],[1792750023,"1"],[1792750323,"1"],[1792750623,"1"],[1792750923,"1"],[1792751223,"1"],[1792751523,"1"],[1792751823,"1"],[1792752123,"1"],[1792752423,"1"],[1792752723,"1"],[1792753023,"1"],[1792753323,"1"],[1792753623,"1"],[1792753923,"1"],[1792754223,"1"],[1792754523,"1"],[1792754823,"1"],[1792755123,"1"],[1792755423,"1"],[1792755723,"1"],[1792756023,"1"],[1792756323,"1"],[1792756623,"1"],[1792756923,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"DiskUncorrectableSectors","alertstate":"firing","component":"disk","device":"ata-ST3000VX010-2E3166_Z6A07P2G","severity":"critical"},"values":[[1792234323,"1"],[1792234623,"1"],[1792234923,"1"],[1792235223,"1"],[1792235523,"1"],[1792235823,"1"],[1792236123,"1"],[1792236423,"1"],[1792236723,"1"],[1792237023,"1"],[1792237323,"1"],[1792237623,"1"],[1792237923,"1"],[1792238223,"1"],[1792238523,"1"],[1792238823,"1"],[1792239123,"1"],[1792239423,"1"],[1792239723,"1"],[1792240023,"1"],[1792240323,"1"],[1792240623,"1"],[1792240923,"1"],[1792241223,"1"],[1792241523,"1"],[1792241823,"1"],[1792242123,"1"],[1792242423,"1"],[1792242723,"1"],[1792243023,"1"],[1792243323,"1"],[1792243623,"1"],[1792243923,"1"],[1792244223,"1"],[1792244523,"1"],[1792244823,"1"],[1792245123,"1"],[1792245423,"1"],[1792245723,"1"],[1792246023,"1"],[1792246323,"1"],[1792246623,"1"],[1792246923,"1"],[1792247223,"1"],[1792247523,"1"],[1792247823,"1"],[1792248123,"1"],[1792248423,"1"],[1792248723,"1"],[1792249023,"1"],[1792249323,"1"],[1792249623,"1"],[1792249923,"1"],[1792250223,"1"],[1792250523,"1"],[1792250823,"1"],[1792251123,"1"],[1792251423,"1"],[1792251723,"1"],[1792252023,"1"],[1792252323,"1"],[1792252623,"1"],[1792252923,"1"],[1792253223,"1"],[1792253523,"1"],[1792253823,"1"],[1792254123,"1"],[1792254423,"1"],[1792254723,"1"],[1792255023,"1"],[1792255323,"1"],[1792255623,"1"],[1792255923,"1"],[1792256223,"1"],[1792256523,"1"],[1792256823,"1"],[1792257123,"1"],[1792257423,"1"],[1792257723,"1"],[1792258023,"1"],[1792258323,"1"],[1792258623,"1"],[1792258923,"1"],[1792259223,"1"],[1792259523,"1"],[1792259823,"1"],[1792260123,"1"],[1792260423,"1"],[1792260723,"1"],[1792261023,"1"],[1792261323,"1"],[1792261623,"1"],[1792261923,"1"],[1792262223,"1"],[1792262523,"1"],[1792262823,"1"],[1792263123,"1"],[1792263423,"1"],[1792263723,"1"],[1792264023,"1"],[1792264323,"1"],[1792264623,"1"],[1792264923,"1"],[1792265223,"1"],[1792265523,"1"],[1792265823,"1"],[1792266123,"1"],[1792266423,"1"],[1792266723,"1"],[1792267023,"1"],[1792267323,"1"],[1792267623,"1"],[1792267923,"1"],[1792268223,"1"],[1792268523,"1"],[1792268823,"1"],[1792269123,"1"],[1792269423,"1"],[1792269723,"1"],[1792270023,"1"],[1792270323,"1"],[1792270623,"1"],[1792270923,"1"],[1792271223,"1"],[1792271523,"1"],[1792271823,"1"],[1792272123,"1"],[1792272423,"1"],[1792272723,"1"],[1792273023,"1"],[1792273323,"1"],[1792273623,"1"],[1792273923,"1"],[1792274223,"1"],[1792274523,"1"],[1792274823,"1"],[1792275123,"1"],[1792275423,"1"],[1792275723,"1"],[1792276023,"1"],[1792276323,"1"],[1792276623,"1"],[1792276923,"1"],[1792277223,"1"],[1792277523,"1"],[1792277823,"1"],[1792278123,"1"],[1792278423,"1"],[1792278723,"1"],[1792279023,"1"],[1792279323,"1"],[1792279623,"1"],[1792279923,"1"],[1792280223,"1"],[1792280523,"1"],[1792280823,"1"],[1792281123,"1"],[1792281423,"1"],[1792281723,"1"],[1792282023,"1"],[1792282323,"1"],[1792282623,"1"],[1792282923,"1"],[1792283223,"1"],[1792283523,"1"],[1792283823,"1"],[1792284123,"1"],[1792284423,"1"],[1792284723,"1"],[1792285023,"1"],[1792285323,"1"],[1792285623,"1"],[1792285923,"1"],[1792286223,"1"],[1792286523,"1"],[1792286823,"1"],[1792287123,"1"],[1792287423,"1"],[1792287723,"1"],[1792288023,"1"],[1792288323,"1"],[1792288623,"1"],[1792288923,"1"],[1792289223,"1"],[1792289523,"1"],[1792289823,"1"],[1792290123,"1"],[1792290423,"1"],[1792290723,"1"],[1792291023,"1"],[1792291323,"1"],[1792291623,"1"],[1792291923,"1"],[1792292223,"1"],[1792292523,"1"],[1792292823,"1"],[1792293123,"1"],[1792293423,"1"],[1792293723,"1"],[1792294023,"1"],[1792294323,"1"],[1792294623,"1"],[1792294923,"1"],[1792295223,"1"],[1792295523,"1"],[1792295823,"1"],[1792296123,"1"],[1792296423,"1"],[1792296723,"1"],[1792297023,"1"],[1792297323,"1"],[1792297623,"1"],[1792297923,"1"],[1792298223,"1"],[1792298523,"1"],[1792298823,"1"],[1792299123,"1"],[1792299423,"1"],[1792299723,"1"],[1792300023,"1"],[1792300323,"1"],[1792300623,"1"],[1792300923,"1"],[1792301223,"1"],[1792301523,"1"],[1792301823,"1"],[1792302123,"1"],[1792302423,"1"],[1792302723,"1"],[1792303023,"1"],[1792303323,"1"],[1792303623,"1"],[1792303923,"1"],[1792304223,"1"],[1792304523,"1"],[1792304823,"1"],[1792305123,"1"],[1792305423,"1"],[1792305723,"1"],[1792306023,"1"],[1792306323,"1"],[1792306623,"1"],[1792306923,"1"],[1792307223,"1"],[1792307523,"1"],[1792307823,"1"],[1792308123,"1"],[1792308423,"1"],[1792308723,"1"],[1792309023,"1"],[1792309323,"1"],[1792309623,"1"],[1792309923,"1"],[1792310223,"1"],[1792310523,"1"],[1792310823,"1"],[1792311123,"1"],[1792311423,"1"],[1792311723,"1"],[1792312023,"1"],[1792312323,"1"],[1792312623,"1"],[1792312923,"1"],[1792313223,"1"],[1792313523,"1"],[1792313823,"1"],[1792314123,"1"],[1792314423,"1"],[1792314723,"1"],[1792315023,"1"],[1792315323,"1"],[1792315623,"1"],[1792315923,"1"],[1792316223,"1"],[1792316523,"1"],[1792316823,"1"],[1792317123,"1"],[1792317423,"1"],[1792317723,"1"],[1792318023,"1"],[1792318323,"1"],[1792318623,"1"],[1792318923,"1"],[1792319223,"1"],[1792319523,"1"],[1792319823,"1"],[1792320123,"1"],[1792320423,"1"],[1792320723,"1"],[1792321023,"1"],[1792321323,"1"],[1792321623,"1"],[1792321923,"1"],[1792322223,"1"],[1792322523,"1"],[1792322823,"1"],[1792323123,"1"],[1792323423,"1"],[1792323723,"1"],[1792324023,"1"],[1792324323,"1"],[1792324623,"1"],[1792324923,"1"],[1792325223,"1"],[1792325523,"1"],[1792325823,"1"],[1792326123,"1"],[1792326423,"1"],[1792326723,"1"],[1792327023,"1"],[1792327323,"1"],[1792327623,"1"],[1792327923,"1"],[1792328223,"1"],[1792328523,"1"],[1792328823,"1"],[1792329123,"1"],[1792329423,"1"],[1792329723,"1"],[1792330023,"1"],[1792330323,"1"],[1792330623,"1"],[1792330923,"1"],[1792331223,"1"],[1792331523,"1"],[1792331823,"1"],[1792332123,"1"],[1792332423,"1"],[1792332723,"1"],[1792333023,"1"],[1792333323,"1"],[1792333623,"1"],[1792333923,"1"],[1792334223,"1"],[1792334523,"1"],[1792334823,"1"],[1792335123,"1"],[1792335423,"1"],[1792335723,"1"],[1792336023,"1"],[1792336323,"1"],[1792336623,"1"],[1792336923,"1"],[1792337223,"1"],[1792337523,"1"],[1792337823,"1"],[1792338123,"1"],[1792338423,"1"],[1792338723,"1"],[1792339023,"1"],[1792339323,"1"],[1792339623,"1"],[1792339923,"1"],[1792340223,"1"],[1792340523,"1"],[1792340823,"1"],[1792341123,"1"],[1792341423,"1"],[1792341723,"1"],[1792342023,"1"],[1792342323,"1"],[1792342623,"1"],[1792342923,"1"],[1792343223,"1"],[1792343523,"1"],[1792343823,"1"],[1792344123,"1"],[1792344423,"1"],[1792344723,"1"],[1792345023,"1"],[1792345323,"1"],[1792345623,"1"],[1792345923,"1"],[1792346223,"1"],[1792346523,"1"],[1792346823,"1"],[1792347123,"1"],[1792347423,"1"],[1792347723,"1"],[1792348023,"1"],[1792348323,"1"],[1792348623,"1"],[1792348923,"1"],[1792349223,"1"],[1792349523,"1"],[1792349823,"1"],[1792350123,"1"],[1792350423,"1"],[1792350723,"1"],[1792351023,"1"],[1792351323,"1"],[1792351623,"1"],[1792351923,"1"],[1792352223,"1"],[1792352523,"1"],[1792352823,"1"],[1792353123,"1"],[1792353423,"1"],[1792353723,"1"],[1792354023,"1"],[1792354323,"1"],[1792354623,"1"],[1792354923,"1"],[1792355223,"1"],[1792355523,"1"],[1792355823,"1"],[1792356123,"1"],[1792356423,"1"],[1792356723,"1"],[1792357023,"1"],[1792357323,"1"],[1792357623,"1"],[1792357923,"1"],[1792358223,"1"],[1792358523,"1"],[1792358823,"1"],[1792359123,"1"],[1792359423,"1"],[1792359723,"1"],[1792360023,"1"],[1792360323,"1"],[1792360623,"1"],[1792360923,"1"],[1792361223,"1"],[1792361523,"1"],[1792361823,"1"],[1792362123,"1"],[1792362423,"1"],[1792362723,"1"],[1792363023,"1"],[1792363323,"1"],[1792363623,"1"],[1792363923,"1"],[1792364223,"1"],[1792364523,"1"],[1792364823,"1"],[1792365123,"1"],[1792365423,"1"],[1792365723,"1"],[1792366023,"1"],[1792366323,"1"],[1792366623,"1"],[1792366923,"1"],[1792367223,"1"],[1792367523,"1"],[1792367823,"1"],[1792368123,"1"],[1792368423,"1"],[1792368723,"1"],[1792369023,"1"],[1792369323,"1"],[1792369623,"1"],[1792369923,"1"],[1792370223,"1"],[1792370523,"1"],[1792370823,"1"],[1792371123,"1"],[1792371423,"1"],[1792371723,"1"],[1792372023,"1"],[1792372323,"1"],[1792372623,"1"],[1792372923,"1"],[1792373223,"1"],[1792373523,"1"],[1792373823,"1"],[1792374123,"1"],[1792374423,"1"],[1792374723,"1"],[1792375023,"1"],[1792375323,"1"],[1792375623,"1"],[1792375923,"1"],[1792376223,"1"],[1792376523,"1"],[1792376823,"1"],[1792377123,"1"],[1792377423,"1"],[1792377723,"1"],[1792378023,"1"],[1792378323,"1"],[1792378623,"1"],[1792378923,"1"],[1792379223,"1"],[1792379523,"1"],[1792379823,"1"],[1792380123,"1"],[1792380423,"1"],[1792380723,"1"],[1792381023,"1"],[1792381323,"1"],[1792381623,"1"],[1792381923,"1"],[1792382223,"1"],[1792382523,"1"],[1792382823,"1"],[1792383123,"1"],[1792383423,"1"],[1792383723,"1"],[1792384023,"1"],[1792384323,"1"],[1792384623,"1"],[1792384923,"1"],[1792385223,"1"],[1792385523,"1"],[1792385823,"1"],[1792386123,"1"],[1792386423,"1"],[1792386723,"1"],[1792387023,"1"],[1792387323,"1"],[1792387623,"1"],[1792387923,"1"],[1792388223,"1"],[1792388523,"1"],[1792388823,"1"],[1792389123,"1"],[1792389423,"1"],[1792389723,"1"],[1792390023,"1"],[1792390323,"1"],[1792390623,"1"],[1792390923,"1"],[1792391223,"1"],[1792391523,"1"],[1792391823,"1"],[1792392123,"1"],[1792392423,"1"],[1792392723,"1"],[1792393023,"1"],[1792393323,"1"],[1792393623,"1"],[1792393923,"1"],[1792394223,"1"],[1792394523,"1"],[1792394823,"1"],[1792395123,"1"],[1792395423,"1"],[1792395723,"1"],[1792396023,"1"],[1792396323,"1"],[1792396623,"1"],[1792396923,"1"],[1792397223,"1"],[1792397523,"1"],[1792397823,"1"],[1792398123,"1"],[1792398423,"1"],[1792398723,"1"],[1792399023,"1"],[1792399323,"1"],[1792399623,"1"],[1792399923,"1"],[1792400223,"1"],[1792400523,"1"],[1792400823,"1"],[1792401123,"1"],[1792401423,"1"],[1792401723,"1"],[1792402023,"1"],[1792402323,"1"],[1792402623,"1"],[1792402923,"1"],[1792403223,"1"],[1792403523,"1"],[1792403823,"1"],[1792404123,"1"],[1792404423,"1"],[1792404723,"1"],[1792405023,"1"],[1792405323,"1"],[1792405623,"1"],[1792405923,"1"],[1792406223,"1"],[1792406523,"1"],[1792406823,"1"],[1792407123,"1"],[1792407423,"1"],[1792407723,"1"],[1792408023,"1"],[1792408323,"1"],[1792408623,"1"],[1792408923,"1"],[1792409223,"1"],[1792409523,"1"],[1792409823,"1"],[1792410123,"1"],[1792410423,"1"],[1792410723,"1"],[1792411023,"1"],[1792411323,"1"],[1792411623,"1"],[1792411923,"1"],[1792412223,"1"],[1792412523,"1"],[1792412823,"1"],[1792413123,"1"],[1792413423,"1"],[1792413723,"1"],[1792414023,"1"],[1792414323,"1"],[1792414623,"1"],[1792414923,"1"],[1792415223,"1"],[1792415523,"1"],[1792415823,"1"],[1792416123,"1"],[1792416423,"1"],[1792416723,"1"],[1792417023,"1"],[1792417323,"1"],[1792417623,"1"],[1792417923,"1"],[1792418223,"1"],[1792418523,"1"],[1792418823,"1"],[1792419123,"1"],[1792419423,"1"],[1792419723,"1"],[1792420023,"1"],[1792420323,"1"],[1792420623,"1"],[1792420923,"1"],[1792421223,"1"],[1792421523,"1"],[1792421823,"1"],[1792422123,"1"],[1792422423,"1"],[1792422723,"1"],[1792423023,"1"],[1792423323,"1"],[1792423623,"1"],[1792423923,"1"],[1792424223,"1"],[1792424523,"1"],[1792424823,"1"],[1792425123,"1"],[1792425423,"1"],[1792425723,"1"],[1792426023,"1"],[1792426323,"1"],[1792426623,"1"],[1792426923,"1"],[1792427223,"1"],[1792427523,"1"],[1792427823,"1"],[1792428123,"1"],[1792428423,"1"],[1792428723,"1"],[1792429023,"1"],[1792429323,"1"],[1792429623,"1"],[1792429923,"1"],[1792430223,"1"],[1792430523,"1"],[1792430823,"1"],[1792431123,"1"],[1792431423,"1"],[1792431723,"1"],[1792432023,"1"],[1792432323,"1"],[1792432623,"1"],[1792432923,"1"],[1792433223,"1"],[1792433523,"1"],[1792433823,"1"],[1792434123,"1"],[1792434423,"1"],[1792434723,"1"],[1792435023,"1"],[1792435323,"1"],[1792435623,"1"],[1792435923,"1"],[1792436223,"1"],[1792436523,"1"],[1792436823,"1"],[1792437123,"1"],[1792437423,"1"],[1792437723,"1"],[1792438023,"1"],[1792438323,"1"],[1792438623,"1"],[1792438923,"1"],[1792439223,"1"],[1792439523,"1"],[1792439823,"1"],[1792440123,"1"],[1792440423,"1"],[1792440723,"1"],[1792441023,"1"],[1792441323,"1"],[1792441623,"1"],[1792441923,"1"],[1792442223,"1"],[1792442523,"1"],[1792442823,"1"],[1792443123,"1"],[1792443423,"1"],[1792443723,"1"],[1792444023,"1"],[1792444323,"1"],[1792444623,"1"],[1792444923,"1"],[1792445223,"1"],[1792445523,"1"],[1792445823,"1"],[1792446123,"1"],[1792446423,"1"],[1792446723,"1"],[1792447023,"1"],[1792447323,"1"],[1792447623,"1"],[1792447923,"1"],[1792448223,"1"],[1792448523,"1"],[1792448823,"1"],[1792449123,"1"],[1792449423,"1"],[1792449723,"1"],[1792450023,"1"],[1792450323,"1"],[1792450623,"1"],[1792450923,"1"],[1792451223,"1"],[1792451523,"1"],[1792451823,"1"],[1792452123,"1"],[1792452423,"1"],[1792452723,"1"],[1792453023,"1"],[1792453323,"1"],[1792453623,"1"],[1792453923,"1"],[1792454223,"1"],[1792454523,"1"],[1792454823,"1"],[1792455123,"1"],[1792455423,"1"],[1792455723,"1"],[1792456023,"1"],[1792456323,"1"],[1792456623,"1"],[1792456923,"1"],[1792457223,"1"],[1792457523,"1"],[1792457823,"1"],[1792458123,"1"],[1792458423,"1"],[1792458723,"1"],[1792459023,"1"],[1792459323,"1"],[1792459623,"1"],[1792459923,"1"],[1792460223,"1"],[1792460523,"1"],[1792460823,"1"],[1792461123,"1"],[1792461423,"1"],[1792461723,"1"],[1792462023,"1"],[1792462323,"1"],[1792462623,"1"],[1792462923,"1"],[1792463223,"1"],[1792463523,"1"],[1792463823,"1"],[1792464123,"1"],[1792464423,"1"],[1792464723,"1"],[1792465023,"1"],[1792465323,"1"],[1792465623,"1"],[1792465923,"1"],[1792466223,"1"],[1792466523,"1"],[1792466823,"1"],[1792467123,"1"],[1792467423,"1"],[1792467723,"1"],[1792468023,"1"],[1792468323,"1"],[1792468623,"1"],[1792468923,"1"],[1792469223,"1"],[1792469523,"1"],[1792469823,"1"],[1792470123,"1"],[1792470423,"1"],[1792470723,"1"],[1792471023,"1"],[1792471323,"1"],[1792471623,"1"],[1792471923,"1"],[1792472223,"1"],[1792472523,"1"],[1792472823,"1"],[1792473123,"1"],[1792473423,"1"],[1792473723,"1"],[1792474023,"1"],[1792474323,"1"],[1792474623,"1"],[1792474923,"1"],[1792475223,"1"],[1792475523,"1"],[1792475823,"1"],[1792476123,"1"],[1792476423,"1"],[1792476723,"1"],[1792477023,"1"],[1792477323,"1"],[1792477623,"1"],[1792477923,"1"],[1792478223,"1"],[1792478523,"1"],[1792478823,"1"],[1792479123,"1"],[1792479423,"1"],[1792479723,"1"],[1792480023,"1"],[1792480323,"1"],[1792480623,"1"],[1792480923,"1"],[1792481223,"1"],[1792481523,"1"],[1792481823,"1"],[1792482123,"1"],[1792482423,"1"],[1792482723,"1"],[1792483023,"1"],[1792483323,"1"],[1792483623,"1"],[1792483923,"1"],[1792484223,"1"],[1792484523,"1"],[1792484823,"1"],[1792485123,"1"],[1792485423,"1"],[1792485723,"1"],[1792486023,"1"],[1792486323,"1"],[1792486623,"1"],[1792486923,"1"],[1792487223,"1"],[1792487523,"1"],[1792487823,"1"],[1792488123,"1"],[1792488423,"1"],[1792488723,"1"],[1792489023,"1"],[1792489323,"1"],[1792489623,"1"],[1792489923,"1"],[1792490223,"1"],[1792490523,"1"],[1792490823,"1"],[1792491123,"1"],[1792491423,"1"],[1792491723,"1"],[1792492023,"1"],[1792492323,"1"],[1792492623,"1"],[1792492923,"1"],[1792493223,"1"],[1792493523,"1"],[1792493823,"1"],[1792494123,"1"],[1792494423,"1"],[1792494723,"1"],[1792495023,"1"],[1792495323,"1"],[1792495623,"1"],[1792495923,"1"],[1792496223,"1"],[1792496523,"1"],[1792496823,"1"],[1792497123,"1"],[1792497423,"1"],[1792497723,"1"],[1792498023,"1"],[1792498323,"1"],[1792498623,"1"],[1792498923,"1"],[1792499223,"1"],[1792499523,"1"],[1792499823,"1"],[1792500123,"1"],[1792500423,"1"],[1792500723,"1"],[1792501023,"1"],[1792501323,"1"],[1792501623,"1"],[1792501923,"1"],[1792502223,"1"],[1792502523,"1"],[1792502823,"1"],[1792503123,"1"],[1792503423,"1"],[1792503723,"1"],[1792504023,"1"],[1792504323,"1"],[1792504623,"1"],[1792504923,"1"],[1792505223,"1"],[1792505523,"1"],[1792505823,"1"],[1792506123,"1"],[1792506423,"1"],[1792506723,"1"],[1792507023,"1"],[1792507323,"1"],[1792507623,"1"],[1792507923,"1"],[1792508223,"1"],[1792508523,"1"],[1792508823,"1"],[1792509123,"1"],[1792509423,"1"],[1792509723,"1"],[1792510023,"1"],[1792510323,"1"],[1792510623,"1"],[1792510923,"1"],[1792511223,"1"],[1792511523,"1"],[1792511823,"1"],[1792512123,"1"],[1792512423,"1"],[1792512723,"1"],[1792513023,"1"],[1792513323,"1"],[1792513623,"1"],[1792513923,"1"],[1792514223,"1"],[1792514523,"1"],[1792514823,"1"],[1792515123,"1"],[1792515423,"1"],[1792515723,"1"],[1792516023,"1"],[1792516323,"1"],[1792516623,"1"],[1792516923,"1"],[1792517223,"1"],[1792517523,"1"],[1792517823,"1"],[1792518123,"1"],[1792518423,"1"],[1792518723,"1"],[1792519023,"1"],[1792519323,"1"],[1792519623,"1"],[1792519923,"1"],[1792520223,"1"],[1792520523,"1"],[1792520823,"1"],[1792521123,"1"],[1792521423,"1"],[1792521723,"1"],[1792522023,"1"],[1792522323,"1"],[1792522623,"1"],[1792522923,"1"],[1792523223,"1"],[1792523523,"1"],[1792523823,"1"],[1792524123,"1"],[1792524423,"1"],[1792524723,"1"],[1792525023,"1"],[1792525323,"1"],[1792525623,"1"],[1792525923,"1"],[1792526223,"1"],[1792526523,"1"],[1792526823,"1"],[1792527123,"1"],[1792527423,"1"],[1792527723,"1"],[1792528023,"1"],[1792528323,"1"],[1792528623,"1"],[1792528923,"1"],[1792529223,"1"],[1792529523,"1"],[1792529823,"1"],[1792530123,"1"],[1792530423,"1"],[1792530723,"1"],[1792531023,"1"],[1792531323,"1"],[1792531623,"1"],[1792531923,"1"],[1792532223,"1"],[1792532523,"1"],[1792532823,"1"],[1792533123,"1"],[1792533423,"1"],[1792533723,"1"],[1792534023,"1"],[1792534323,"1"],[1792534623,"1"],[1792534923,"1"],[1792535223,"1"],[1792535523,"1"],[1792535823,"1"],[1792536123,"1"],[1792536423,"1"],[1792536723,"1"],[1792537023,"1"],[1792537323,"1"],[1792537623,"1"],[1792537923,"1"],[1792538223,"1"],[1792538523,"1"],[1792538823,"1"],[1792539123,"1"],[1792539423,"1"],[1792539723,"1"],[1792540023,"1"],[1792540323,"1"],[1792540623,"1"],[1792540923,"1"],[1792541223,"1"],[1792541523,"1"],[1792541823,"1"],[1792542123,"1"],[1792542423,"1"],[1792542723,"1"],[1792543023,"1"],[1792543323,"1"],[1792543623,"1"],[1792543923,"1"],[1792544223,"1"],[1792544523,"1"],[1792544823,"1"],[1792545123,"1"],[1792545423,"1"],[1792545723,"1"],[1792546023,"1"],[1792546323,"1"],[1792546623,"1"],[1792546923,"1"],[1792547223,"1"],[1792547523,"1"],[1792547823,"1"],[1792548123,"1"],[1792548423,"1"],[1792548723,"1"],[1792549023,"1"],[1792549323,"1"],[1792549623,"1"],[1792549923,"1"],[1792550223,"1"],[1792550523,"1"],[1792550823,"1"],[1792551123,"1"],[1792551423,"1"],[1792551723,"1"],[1792552023,"1"],[1792552323,"1"],[1792552623,"1"],[1792552923,"1"],[1792553223,"1"],[1792553523,"1"],[1792553823,"1"],[1792554123,"1"],[1792554423,"1"],[1792554723,"1"],[1792555023,"1"],[1792555323,"1"],[1792555623,"1"],[1792555923,"1"],[1792556223,"1"],[1792556523,"1"],[1792556823,"1"],[1792557123,"1"],[1792557423,"1"],[1792557723,"1"],[1792558023,"1"],[1792558323,"1"],[1792558623,"1"],[1792558923,"1"],[1792559223,"1"],[1792559523,"1"],[1792559823,"1"],[1792560123,"1"],[1792560423,"1"],[1792560723,"1"],[1792561023,"1"],[1792561323,"1"],[1792561623,"1"],[1792561923,"1"],[1792562223,"1"],[1792562523,"1"],[1792562823,"1"],[1792563123,"1"],[1792563423,"1"],[1792563723,"1"],[1792564023,"1"],[1792564323,"1"],[1792564623,"1"],[1792564923,"1"],[1792565223,"1"],[1792565523,"1"],[1792565823,"1"],[1792566123,"1"],[1792566423,"1"],[1792566723,"1"],[1792567023,"1"],[1792567323,"1"],[1792567623,"1"],[1792567923,"1"],[1792568223,"1"],[1792568523,"1"],[1792568823,"1"],[1792569123,"1"],[1792569423,"1"],[1792569723,"1"],[1792570023,"1"],[1792570323,"1"],[1792570623,"1"],[1792570923,"1"],[1792571223,"1"],[1792571523,"1"],[1792571823,"1"],[1792572123,"1"],[1792572423,"1"],[1792572723,"1"],[1792573023,"1"],[1792573323,"1"],[1792573623,"1"],[1792573923,"1"],[1792574223,"1"],[1792574523,"1"],[1792574823,"1"],[1792575123,"1"],[1792575423,"1"],[1792575723,"1"],[1792576023,"1"],[1792576323,"1"],[1792576623,"1"],[1792576923,"1"],[1792577223,"1"],[1792577523,"1"],[1792577823,"1"],[1792578123,"1"],[1792578423,"1"],[1792578723,"1"],[1792579023,"1"],[1792579323,"1"],[1792579623,"1"],[1792579923,"1"],[1792580223,"1"],[1792580523,"1"],[1792580823,"1"],[1792581123,"1"],[1792581423,"1"],[1792581723,"1"],[1792582023,"1"],[1792582323,"1"],[1792582623,"1"],[1792582923,"1"],[1792583223,"1"],[1792583523,"1"],[1792583823,"1"],[1792584123,"1"],[1792584423,"1"],[1792584723,"1"],[1792585023,"1"],[1792585323,"1"],[1792585623,"1"],[1792585923,"1"],[1792586223,"1"],[1792586523,"1"],[1792586823,"1"],[1792587123,"1"],[1792587423,"1"],[1792587723,"1"],[1792588023,"1"],[1792588323,"1"],[1792588623,"1"],[1792588923,"1"],[1792589223,"1"],[1792589523,"1"],[1792589823,"1"],[1792590123,"1"],[1792590423,"1"],[1792590723,"1"],[1792591023,"1"],[1792591323,"1"],[1792591623,"1"],[1792591923,"1"],[1792592223,"1"],[1792592523,"1"],[1792592823,"1"],[1792593123,"1"],[1792593423,"1"],[1792593723,"1"],[1792594023,"1"],[1792594323,"1"],[1792594623,"1"],[1792594923,"1"],[1792595223,"1"],[1792595523,"1"],[1792595823,"1"],[1792596123,"1"],[1792596423,"1"],[1792596723,"1"],[1792597023,"1"],[1792597323,"1"],[1792597623,"1"],[1792597923,"1"],[1792598223,"1"],[1792598523,"1"],[1792598823,"1"],[1792599123,"1"],[1792599423,"1"],[1792599723,"1"],[1792600023,"1"],[1792600323,"1"],[1792600623,"1"],[1792600923,"1"],[1792601223,"1"],[1792601523,"1"],[1792601823,"1"],[1792602123,"1"],[1792602423,"1"],[1792602723,"1"],[1792603023,"1"],[1792603323,"1"],[1792603623,"1"],[1792603923,"1"],[1792604223,"1"],[1792604523,"1"],[1792604823,"1"],[1792605123,"1"],[1792605423,"1"],[1792605723,"1"],[1792606023,"1"],[1792606323,"1"],[1792606623,"1"],[1792606923,"1"],[1792607223,"1"],[1792607523,"1"],[1792607823,"1"],[1792608123,"1"],[1792608423,"1"],[1792608723,"1"],[1792609023,"1"],[1792609323,"1"],[1792609623,"1"],[1792609923,"1"],[1792610223,"1"],[1792610523,"1"],[1792610823,"1"],[1792611123,"1"],[1792611423,"1"],[1792611723,"1"],[1792612023,"1"],[1792612323,"1"],[1792612623,"1"],[1792612923,"1"],[1792613223,"1"],[1792613523,"1"],[1792613823,"1"],[1792614123,"1"],[1792614423,"1"],[1792614723,"1"],[1792615023,"1"],[1792615323,"1"],[1792615623,"1"],[1792615923,"1"],[1792616223,"1"],[1792616523,"1"],[1792616823,"1"],[1792617123,"1"],[1792617423,"1"],[1792617723,"1"],[1792618023,"1"],[1792618323,"1"],[1792618623,"1"],[1792618923,"1"],[1792619223,"1"],[1792619523,"1"],[1792619823,"1"],[1792620123,"1"],[1792620423,"1"],[1792620723,"1"],[1792621023,"1"],[1792621323,"1"],[1792621623,"1"],[1792621923,"1"],[1792622223,"1"],[1792622523,"1"],[1792622823,"1"],[1792623123,"1"],[1792623423,"1"],[1792623723,"1"],[1792624023,"1"],[1792624323,"1"],[1792624623,"1"],[1792624923,"1"],[1792625223,"1"],[1792625523,"1"],[1792625823,"1"],[1792626123,"1"],[1792626423,"1"],[1792626723,"1"],[1792627023,"1"],[1792627323,"1"],[1792627623,"1"],[1792627923,"1"],[1792628223,"1"],[1792628523,"1"],[1792628823,"1"],[1792629123,"1"],[1792629423,"1"],[1792629723,"1"],[1792630023,"1"],[1792630323,"1"],[1792630623,"1"],[1792630923,"1"],[1792631223,"1"],[1792631523,"1"],[1792631823,"1"],[1792632123,"1"],[1792632423,"1"],[1792632723,"1"],[1792633023,"1"],[1792633323,"1"],[1792633623,"1"],[1792633923,"1"],[1792634223,"1"],[1792634523,"1"],[1792634823,"1"],[1792635123,"1"],[1792635423,"1"],[1792635723,"1"],[1792636023,"1"],[1792636323,"1"],[1792636623,"1"],[1792636923,"1"],[1792637223,"1"],[1792637523,"1"],[1792637823,"1"],[1792638123,"1"],[1792638423,"1"],[1792638723,"1"],[1792639023,"1"],[1792639323,"1"],[1792639623,"1"],[1792639923,"1"],[1792640223,"1"],[1792640523,"1"],[1792640823,"1"],[1792641123,"1"],[1792641423,"1"],[1792641723,"1"],[1792642023,"1"],[1792642323,"1"],[1792642623,"1"],[1792642923,"1"],[1792643223,"1"],[1792643523,"1"],[1792643823,"1"],[1792644123,"1"],[1792644423,"1"],[1792644723,"1"],[1792645023,"1"],[1792645323,"1"],[1792645623,"1"],[1792645923,"1"],[1792646223,"1"],[1792646523,"1"],[1792646823,"1"],[1792647123,"1"],[1792647423,"1"],[1792647723,"1"],[1792648023,"1"],[1792648323,"1"],[1792648623,"1"],[1792648923,"1"],[1792649223,"1"],[1792649523,"1"],[1792649823,"1"],[1792650123,"1"],[1792650423,"1"],[1792650723,"1"],[1792651023,"1"],[1792651323,"1"],[1792651623,"1"],[1792651923,"1"],[1792652223,"1"],[1792652523,"1"],[1792652823,"1"],[1792653123,"1"],[1792653423,"1"],[1792653723,"1"],[1792654023,"1"],[1792654323,"1"],[1792654623,"1"],[1792654923,"1"],[1792655223,"1"],[1792655523,"1"],[1792655823,"1"],[1792656123,"1"],[1792656423,"1"],[1792656723,"1"],[1792657023,"1"],[1792657323,"1"],[1792657623,"1"],[1792657923,"1"],[1792658223,"1"],[1792658523,"1"],[1792658823,"1"],[1792659123,"1"],[1792659423,"1"],[1792659723,"1"],[1792660023,"1"],[1792660323,"1"],[1792660623,"1"],[1792660923,"1"],[1792661223,"1"],[1792661523,"1"],[1792661823,"1"],[1792662123,"1"],[1792662423,"1"],[1792662723,"1"],[1792663023,"1"],[1792663323,"1"],[1792663623,"1"],[1792663923,"1"],[1792664223,"1"],[1792664523,"1"],[1792664823,"1"],[1792665123,"1"],[1792665423,"1"],[1792665723,"1"],[1792666023,"1"],[1792666323,"1"],[1792666623,"1"],[1792666923,"1"],[1792667223,"1"],[1792667523,"1"],[1792667823,"1"],[1792668123,"1"],[1792668423,"1"],[1792668723,"1"],[1792669023,"1"],[1792669323,"1"],[1792669623,"1"],[1792669923,"1"],[1792670223,"1"],[1792670523,"1"],[1792670823,"1"],[1792671123,"1"],[1792671423,"1"],[1792671723,"1"],[1792672023,"1"],[1792672323,"1"],[1792672623,"1"],[1792672923,"1"],[1792673223,"1"],[1792673523,"1"],[1792673823,"1"],[1792674123,"1"],[1792674423,"1"],[1792674723,"1"],[1792675023,"1"],[1792675323,"1"],[1792675623,"1"],[1792675923,"1"],[1792676223,"1"],[1792676523,"1"],[1792676823,"1"],[1792677123,"1"],[1792677423,"1"],[1792677723,"1"],[1792678023,"1"],[1792678323,"1"],[1792678623,"1"],[1792678923,"1"],[1792679223,"1"],[1792679523,"1"],[1792679823,"1"],[1792680123,"1"],[1792680423,"1"],[1792680723,"1"],[1792681023,"1"],[1792681323,"1"],[1792681623,"1"],[1792681923,"1"],[1792682223,"1"],[1792682523,"1"],[1792682823,"1"],[1792683123,"1"],[1792683423,"1"],[1792683723,"1"],[1792684023,"1"],[1792684323,"1"],[1792684623,"1"],[1792684923,"1"],[1792685223,"1"],[1792685523,"1"],[1792685823,"1"],[1792686123,"1"],[1792686423,"1"],[1792686723,"1"],[1792687023,"1"],[1792687323,"1"],[1792687623,"1"],[1792687923,"1"],[1792688223,"1"],[1792688523,"1"],[1792688823,"1"],[1792689123,"1"],[1792689423,"1"],[1792689723,"1"],[1792690023,"1"],[1792690323,"1"],[1792690623,"1"],[1792690923,"1"],[1792691223,"1"],[1792691523,"1"],[1792691823,"1"],[1792692123,"1"],[1792692423,"1"],[1792692723,"1"],[1792693023,"1"],[1792693323,"1"],[1792693623,"1"],[1792693923,"1"],[1792694223,"1"],[1792694523,"1"],[1792694823,"1"],[1792695123,"1"],[1792695423,"1"],[1792695723,"1"],[1792696023,"1"],[1792696323,"1"],[1792696623,"1"],[1792696923,"1"],[1792697223,"1"],[1792697523,"1"],[1792697823,"1"],[1792698123,"1"],[1792698423,"1"],[1792698723,"1"],[1792699023,"1"],[1792699323,"1"],[1792699623,"1"],[1792699923,"1"],[1792700223,"1"],[1792700523,"1"],[1792700823,"1"],[1792701123,"1"],[1792701423,"1"],[1792701723,"1"],[1792702023,"1"],[1792702323,"1"],[1792702623,"1"],[1792702923,"1"],[1792703223,"1"],[1792703523,"1"],[1792703823,"1"],[1792704123,"1"],[1792704423,"1"],[1792704723,"1"],[1792705023,"1"],[1792705323,"1"],[1792705623,"1"],[1792705923,"1"],[1792706223,"1"],[1792706523,"1"],[1792706823,"1"],[1792707123,"1"],[1792707423,"1"],[1792707723,"1"],[1792708023,"1"],[1792708323,"1"],[1792708623,"1"],[1792708923,"1"],[1792709223,"1"],[1792709523,"1"],[1792709823,"1"],[1792710123,"1"],[1792710423,"1"],[1792710723,"1"],[1792711023,"1"],[1792711323,"1"],[1792711623,"1"],[1792711923,"1"],[1792712223,"1"],[1792712523,"1"],[1792712823,"1"],[1792713123,"1"],[1792713423,"1"],[1792713723,"1"],[1792714023,"1"],[1792714323,"1"],[1792714623,"1"],[1792714923,"1"],[1792715223,"1"],[1792715523,"1"],[1792715823,"1"],[1792716123,"1"],[1792716423,"1"],[1792716723,"1"],[1792717023,"1"],[1792717323,"1"],[1792717623,"1"],[1792717923,"1"],[1792718223,"1"],[1792718523,"1"],[1792718823,"1"],[1792719123,"1"],[1792719423,"1"],[1792719723,"1"],[1792720023,"1"],[1792720323,"1"],[1792720623,"1"],[1792720923,"1"],[1792721223,"1"],[1792721523,"1"],[1792721823,"1"],[1792722123,"1"],[1792722423,"1"],[1792722723,"1"],[1792723023,"1"],[1792723323,"1"],[1792723623,"1"],[1792723923,"1"],[1792724223,"1"],[1792724523,"1"],[1792724823,"1"],[1792725123,"1"],[1792725423,"1"],[1792725723,"1"],[1792726023,"1"],[1792726323,"1"],[1792726623,"1"],[1792726923,"1"],[1792727223,"1"],[1792727523,"1"],[1792727823,"1"],[1792728123,"1"],[1792728423,"1"],[1792728723,"1"],[1792729023,"1"],[1792729323,"1"],[1792729623,"1"],[1792729923,"1"],[1792730223,"1"],[1792730523,"1"],[1792730823,"1"],[1792731123,"1"],[1792731423,"1"],[1792731723,"1"],[1792732023,"1"],[1792732323,"1"],[1792732623,"1"],[1792732923,"1"],[1792733223,"1"],[1792733523,"1"],[1792733823,"1"],[1792734123,"1"],[1792734423,"1"],[1792734723,"1"],[1792735023,"1"],[1792735323,"1"],[1792735623,"1"],[1792735923,"1"],[1792736223,"1"],[1792736523,"1"],[1792736823,"1"],[1792737123,"1"],[1792737423,"1"],[1792737723,"1"],[1792738023,"1"],[1792738323,"1"],[1792738623,"1"],[1792738923,"1"],[1792739223,"1"],[1792739523,"1"],[1792739823,"1"],[1792740123,"1"],[1792740423,"1"],[1792740723,"1"],[1792741023,"1"],[1792741323,"1"],[1792741623,"1"],[1792741923,"1"],[1792742223,"1"],[1792742523,"1"],[1792742823,"1"],[1792743123,"1"],[1792743423,"1"],[1792743723,"1"],[1792744023,"1"],[1792744323,"1"],[1792744623,"1"],[1792744923,"1"],[1792745223,"1"],[1792745523,"1"],[1792745823,"1"],[1792746123,"1"],[1792746423,"1"],[1792746723,"1"],[1792747023,"1"],[1792747323,"1"],[1792747623,"1"],[1792747923,"1"],[1792748223,"1"],[1792748523,"1"],[1792748823,"1"],[1792749123,"1"],[1792749423,"1"],[1792749723,"1"],[1792750023,"1"],[1792750323,"1"],[1792750623,"1"],[1792750923,"1"],[1792751223,"1"],[1792751523,"1"],[1792751823,"1"],[1792752123,"1"],[1792752423,"1"],[1792752723,"1"],[1792753023,"1"],[1792753323,"1"],[1792753623,"1"],[1792753923,"1"],[1792754223,"1"],[1792754523,"1"],[1792754823,"1"],[1792755123,"1"],[1792755423,"1"],[1792755723,"1"],[1792756023,"1"],[1792756323,"1"],[1792756623,"1"],[1792756923,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"DooplexGiteaOffsiteStale","alertstate":"firing","component":"backup","instance":"dooplex","job":"node-exporter","severity":"critical"},"values":[[1792803423,"1"],[1792803723,"1"],[1792804023,"1"],[1792804323,"1"],[1792804623,"1"],[1792804923,"1"],[1792805223,"1"],[1792805523,"1"],[1792805823,"1"],[1792806123,"1"],[1792806423,"1"],[1792806723,"1"],[1792807023,"1"],[1792807323,"1"],[1792807623,"1"],[1792807923,"1"],[1792808223,"1"],[1792808523,"1"],[1792808823,"1"],[1792809123,"1"],[1792809423,"1"],[1792809723,"1"],[1792810023,"1"],[1792810323,"1"],[1792810623,"1"],[1792810923,"1"],[1792811223,"1"],[1792811523,"1"],[1792811823,"1"],[1792812123,"1"],[1792812423,"1"],[1792812723,"1"],[1792813023,"1"],[1792813323,"1"],[1792813623,"1"],[1792813923,"1"],[1792814223,"1"],[1792814523,"1"],[1792814823,"1"],[1792815123,"1"],[1792815423,"1"],[1792815723,"1"],[1792816023,"1"],[1792816323,"1"],[1792816623,"1"],[1792816923,"1"],[1792817223,"1"],[1792817523,"1"],[1792817823,"1"],[1792818123,"1"],[1792818423,"1"],[1792818723,"1"],[1792819023,"1"],[1792819323,"1"],[1792819623,"1"],[1792819923,"1"],[1792820223,"1"],[1792820523,"1"],[1792820823,"1"],[1792821123,"1"],[1792821423,"1"],[1792821723,"1"],[1792822023,"1"],[1792822323,"1"],[1792822623,"1"],[1792822923,"1"],[1792823223,"1"],[1792823523,"1"],[1792823823,"1"],[1792824123,"1"],[1792824423,"1"],[1792824723,"1"],[1792825023,"1"],[1792825323,"1"],[1792825623,"1"],[1792825923,"1"],[1792826223,"1"],[1792826523,"1"],[1792826823,"1"],[1792827123,"1"],[1792827423,"1"],[1792827723,"1"],[1792828023,"1"],[1792828323,"1"],[1792828623,"1"],[1792828923,"1"],[1792829223,"1"],[1792829523,"1"],[1792829823,"1"],[1792830123,"1"],[1792830423,"1"],[1792830723,"1"],[1792831023,"1"],[1792831323,"1"],[1792831623,"1"],[1792831923,"1"],[1792832223,"1"],[1792832523,"1"],[1792832823,"1"],[1792833123,"1"],[1792833423,"1"],[1792833723,"1"],[1792834023,"1"],[1792834323,"1"],[1792834623,"1"],[1792834923,"1"],[1792835223,"1"],[1792835523,"1"],[1792835823,"1"],[1792836123,"1"],[1792836423,"1"],[1792836723,"1"],[1792837023,"1"],[1792837323,"1"],[1792837623,"1"],[1792837923,"1"],[1792838223,"1"],[1792838523,"1"],[1792838823,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"LonghornVolumeDegraded","alertstate":"firing","instance":"longhorn-backend.longhorn-system:9500","job":"longhorn","node":"dooplex","pvc":"pms-config-plex-plex-media-server-0","pvc_namespace":"mediaserver-system","severity":"warning","volume":"pvc-34a1540e-8b89-4c1d-8465-796bdbfb172f"},"values":[[1792743723,"1"],[1792744023,"1"],[1792744323,"1"],[1792744623,"1"],[1792744923,"1"],[1792745823,"1"],[1792746123,"1"],[1792746423,"1"],[1792746723,"1"],[1792747023,"1"],[1792747323,"1"],[1792747623,"1"],[1792747923,"1"],[1792748223,"1"],[1792748523,"1"],[1792748823,"1"],[1792749123,"1"],[1792749423,"1"],[1792749723,"1"],[1792750023,"1"],[1792750323,"1"],[1792750623,"1"],[1792751523,"1"],[1792751823,"1"],[1792752123,"1"],[1792752423,"1"],[1792752723,"1"],[1792753023,"1"],[1792753323,"1"],[1792753623,"1"],[1792753923,"1"],[1792754223,"1"],[1792754523,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"LonghornVolumeDegraded","alertstate":"firing","instance":"longhorn-backend.longhorn-system:9500","job":"longhorn","node":"dooplex","pvc":"pms-config-v2","pvc_namespace":"mediaserver-system","severity":"warning","volume":"pvc-7fc45c5f-1567-4dfa-ad59-1e55d59f0043"},"values":[[1792748523,"1"],[1792748823,"1"],[1792749123,"1"],[1792749423,"1"],[1792749723,"1"],[1792750023,"1"],[1792750323,"1"],[1792750623,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"PodRestartingFrequently","alertstate":"firing","component":"workload","container":"git-sync","instance":"kube-state-metrics:8080","job":"kube-state-metrics","namespace":"felhom-system","pod":"felhom-webpage-68484d6485-6cw7k","severity":"warning","uid":"fc74f75f-d0e4-4ff0-933d-8033e1bab8be"},"values":[[1792401723,"1"],[1792402023,"1"],[1792402323,"1"],[1792402623,"1"],[1792402923,"1"],[1792403223,"1"],[1792403523,"1"],[1792403823,"1"],[1792404123,"1"],[1792404423,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"PodRestartingFrequently","alertstate":"firing","component":"workload","container":"git-sync","instance":"kube-state-metrics:8080","job":"kube-state-metrics","namespace":"felhom-system","pod":"felhom-webpage-68484d6485-zhwsk","severity":"warning","uid":"e424c07d-561a-4d1b-9f12-c866725dc30e"},"values":[[1792400523,"1"],[1792400823,"1"],[1792401123,"1"],[1792401423,"1"],[1792401723,"1"],[1792402023,"1"],[1792402323,"1"],[1792402623,"1"],[1792402923,"1"],[1792403223,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"PodRestartingFrequently","alertstate":"firing","component":"workload","container":"git-sync-scripts","instance":"kube-state-metrics:8080","job":"kube-state-metrics","namespace":"felhom-system","pod":"felhom-webpage-68484d6485-6cw7k","severity":"warning","uid":"fc74f75f-d0e4-4ff0-933d-8033e1bab8be"},"values":[[1792402023,"1"],[1792402323,"1"],[1792402623,"1"],[1792402923,"1"],[1792403223,"1"],[1792403523,"1"],[1792403823,"1"],[1792404123,"1"],[1792404423,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"PodRestartingFrequently","alertstate":"firing","component":"workload","container":"git-sync-scripts","instance":"kube-state-metrics:8080","job":"kube-state-metrics","namespace":"felhom-system","pod":"felhom-webpage-68484d6485-zhwsk","severity":"warning","uid":"e424c07d-561a-4d1b-9f12-c866725dc30e"},"values":[[1792400523,"1"],[1792400823,"1"],[1792401123,"1"],[1792401423,"1"],[1792401723,"1"],[1792402023,"1"],[1792402323,"1"],[1792402623,"1"],[1792402923,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"PostgresqlDumpStale","alertstate":"firing","component":"database-backup","cronjob":"postgresql-backup","instance":"kube-state-metrics:8080","job":"kube-state-metrics","namespace":"database-system","severity":"warning"},"values":[[1792793823,"1"],[1792794123,"1"],[1792794423,"1"],[1792794723,"1"],[1792795023,"1"],[1792795323,"1"],[1792795623,"1"],[1792795923,"1"],[1792796223,"1"],[1792796523,"1"],[1792796823,"1"],[1792797123,"1"],[1792797423,"1"],[1792797723,"1"],[1792798023,"1"],[1792798323,"1"],[1792798623,"1"],[1792798923,"1"],[1792799223,"1"],[1792799523,"1"],[1792799823,"1"],[1792800123,"1"],[1792800423,"1"],[1792800723,"1"],[1792801023,"1"],[1792801323,"1"],[1792801623,"1"],[1792801923,"1"],[1792802223,"1"],[1792802523,"1"],[1792802823,"1"],[1792803123,"1"],[1792803423,"1"],[1792803723,"1"],[1792804023,"1"],[1792804323,"1"],[1792804623,"1"],[1792804923,"1"],[1792805223,"1"],[1792805523,"1"],[1792805823,"1"],[1792806123,"1"],[1792806423,"1"],[1792806723,"1"],[1792807023,"1"],[1792807323,"1"],[1792807623,"1"],[1792807923,"1"],[1792808223,"1"],[1792808523,"1"],[1792808823,"1"],[1792809123,"1"],[1792809423,"1"],[1792809723,"1"],[1792810023,"1"],[1792810323,"1"],[1792810623,"1"],[1792810923,"1"],[1792811223,"1"],[1792811523,"1"],[1792811823,"1"],[1792812123,"1"],[1792812423,"1"],[1792812723,"1"],[1792813023,"1"],[1792813323,"1"],[1792813623,"1"],[1792813923,"1"],[1792814223,"1"],[1792814523,"1"],[1792814823,"1"],[1792815123,"1"],[1792815423,"1"],[1792815723,"1"],[1792816023,"1"],[1792816323,"1"],[1792816623,"1"],[1792816923,"1"],[1792817223,"1"],[1792817523,"1"],[1792817823,"1"],[1792818123,"1"],[1792818423,"1"],[1792818723,"1"],[1792819023,"1"],[1792819323,"1"],[1792819623,"1"],[1792819923,"1"],[1792820223,"1"],[1792820523,"1"],[1792820823,"1"],[1792821123,"1"],[1792821423,"1"],[1792821723,"1"],[1792822023,"1"],[1792822323,"1"],[1792822623,"1"],[1792822923,"1"],[1792823223,"1"],[1792823523,"1"],[1792823823,"1"],[1792824123,"1"],[1792824423,"1"],[1792824723,"1"],[1792825023,"1"],[1792825323,"1"],[1792825623,"1"],[1792825923,"1"],[1792826223,"1"],[1792826523,"1"],[1792826823,"1"],[1792827123,"1"],[1792827423,"1"],[1792827723,"1"],[1792828023,"1"],[1792828323,"1"],[1792828623,"1"],[1792828923,"1"],[1792829223,"1"],[1792829523,"1"],[1792829823,"1"],[1792830123,"1"],[1792830423,"1"],[1792830723,"1"],[1792831023,"1"],[1792831323,"1"],[1792831623,"1"],[1792831923,"1"],[1792832223,"1"],[1792832523,"1"],[1792832823,"1"],[1792833123,"1"],[1792833423,"1"],[1792833723,"1"],[1792834023,"1"],[1792834323,"1"],[1792834623,"1"],[1792834923,"1"],[1792835223,"1"],[1792835523,"1"],[1792835823,"1"],[1792836123,"1"],[1792836423,"1"],[1792836723,"1"],[1792837023,"1"],[1792837323,"1"],[1792837623,"1"],[1792837923,"1"],[1792838223,"1"],[1792838523,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"WanBufferbloat","alertstate":"firing","severity":"info","target":"hetzner"},"values":[[1792336923,"1"],[1792342023,"1"],[1792342323,"1"],[1792403223,"1"],[1792529223,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"WanDownloadDegraded","alertstate":"firing","direction":"download","instance":"wan-monitor.admin-system:9116","job":"wan-probe","severity":"warning","target":"hetzner"},"values":[[1792531923,"1"],[1792532223,"1"],[1792532523,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"WanUploadDegraded","alertstate":"firing","direction":"upload","instance":"wan-monitor.admin-system:9116","job":"wan-probe","severity":"warning","target":"hetzner"},"values":[[1792259823,"1"],[1792260123,"1"],[1792260423,"1"],[1792397523,"1"],[1792530723,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"WanUpstreamPacketLoss","alertstate":"firing","condition":"idle","direction":"upstream","instance":"wan-monitor.admin-system:9116","job":"wan-probe","severity":"warning","target":"hetzner"},"values":[[1792530423,"1"],[1792530723,"1"]]}]}} \ No newline at end of file diff --git a/documentation/audits/dooplex-alarms-2026-10-24/part1-2-diagnosis.txt b/documentation/audits/dooplex-alarms-2026-10-24/part1-2-diagnosis.txt new file mode 100644 index 00000000..abe1d818 --- /dev/null +++ b/documentation/audits/dooplex-alarms-2026-10-24/part1-2-diagnosis.txt @@ -0,0 +1,51 @@ +# Part 1+2 diagnosis, read 2026-10-24T10:39:09Z + +## felhom-dooplex-offsite journal 2026-10-24 +Oct 24 00:20:48 dooplex systemd[1]: Starting felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232)... +Oct 24 00:20:48 dooplex felhom-dooplex-offsite[1455158]: felhom-dooplex-offsite: FAILED: newest complete dump 20261023-100001 is 12 h old (limit 7 h) — the dump CronJob stopped +Oct 24 00:20:48 dooplex systemd[1]: felhom-dooplex-offsite.service: Main process exited, code=exited, status=1/FAILURE +Oct 24 00:20:48 dooplex systemd[1]: felhom-dooplex-offsite.service: Failed with result 'exit-code'. +Oct 24 00:20:48 dooplex systemd[1]: Failed to start felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232). +Oct 24 00:20:48 dooplex systemd[1]: felhom-dooplex-offsite.service: Triggering OnFailure= dependencies. + +## postgresql-backup jobs +NAME STATUS COMPLETIONS DURATION AGE +postgresql-backup-29878440 Complete 1/1 17s 36h +postgresql-backup-29878800 Complete 1/1 17s 30h +postgresql-backup-29879160 Complete 1/1 13s 24h +postgresql-backup-29879880 Failed 0/1 12h 12h +postgresql-backup-29880240 Failed 0/1 6h39m 6h39m +postgresql-backup-29880600 Failed 0/1 39m 39m + +## CronJob status +lastSchedule=2026-10-24T10:00:00Z lastSuccess=2026-10-23T10:00:13Z + +## failed dump folders (globals.sql only, 0 bytes) +total 8 +drwxr-xr-x 2 26 tape 4096 Oct 23 18:00 . +drwxr-xr-x 29 26 tape 4096 Oct 24 12:00 .. +-rw-r--r-- 1 26 tape 0 Oct 23 18:00 globals.sql +total 8 +drwxr-xr-x 2 26 tape 4096 Oct 24 12:00 . +drwxr-xr-x 29 26 tape 4096 Oct 24 12:00 .. +-rw-r--r-- 1 26 tape 0 Oct 24 12:00 globals.sql + +## database log: failed logins in 24 h (user, client, count) — first 2026-10-23 16:00:01 UTC; all four clients are backup job pods +message: password authentication failed for user "postgres" (sql_state 28P01) + +## which secret's password does role postgres accept (scratchpad pgpw-check.sh; prints ok/refused only) +postgresql-superuser: refused (the CNPG superuser secret; the CronJob reads this) +postgres-superuser: ok (the bootstrap.initdb secret; owner=postgres) + +## cluster status.secretsResourceVersion +{"applicationSecretVersion":"4427397","clientCaSecretVersion":"80646921","replicationSecretVersion":"80646924","serverCaSecretVersion":"80646921","serverSecretVersion":"80646922","superuserSecretVersion":"59321453"} + +## instance start: postgresql-2 instance manager up 2026-10-23T13:50:14Z (window reboot); last good dump 10:00Z, first bad 16:00Z + +## ruled out +tunnel: active since 2026-10-23 13:41:29Z (one 'No route to host' at boot, restart after 31 s); PBS via 127.0.0.1:18007 -> 200 +nft inet hm040_blocklist: ioc_ipv4 = { 5.255.109.74, 5.255.115.190 } only; ep0 167.233.158.164 not in it +disk: Filesystem Size Used Avail Use% Mounted on +disk: /dev/sda1 9.1T 3.2T 5.4T 38% /mnt/5_hdd +disk: /dev/sdb1 445G 239G 184G 57% / +overlap: the job died at 00:20:48 in 0 s on the dump-age check, before any network use; release copy 01:40, hub copy 02:30 (UTC+2) do not overlap diff --git a/documentation/audits/dooplex-alarms-2026-10-24/part1-2-fix.txt b/documentation/audits/dooplex-alarms-2026-10-24/part1-2-fix.txt new file mode 100644 index 00000000..c9d913df --- /dev/null +++ b/documentation/audits/dooplex-alarms-2026-10-24/part1-2-fix.txt @@ -0,0 +1,52 @@ +# Fix and proof, 2026-10-24T10:58:25Z + +## homelab-manifests 531023d: CronJob reads postgres-superuser; ArgoCD sync of CronJob/postgresql-backup only +env after sync: PGUSER=postgres-superuser PGPASSWORD=postgres-superuser + +## by-hand job postgresql-backup-manual-1043 (kubectl create job --from=cronjob/postgresql-backup) +total 22064 +drwxr-xr-x 2 26 tape 4096 Oct 24 12:43 . +drwxr-xr-x 25 26 tape 4096 Oct 24 12:43 .. +-rw-r--r-- 1 26 tape 5297319 Oct 24 12:43 authentik.dump +-rw-r--r-- 1 26 tape 711473 Oct 24 12:43 calcom.dump +-rw-r--r-- 1 26 tape 10258 Oct 24 12:43 calcom_saml.dump +-rw-r--r-- 1 26 tape 8598648 Oct 24 12:43 gitea.dump +-rw-r--r-- 1 26 tape 4016 Oct 24 12:43 globals.sql +-rw-r--r-- 1 26 tape 90743 Oct 24 12:43 guacamole_db.dump +-rw-r--r-- 1 26 tape 1367243 Oct 24 12:43 homarr.dump +-rw-r--r-- 1 26 tape 3624770 Oct 24 12:43 outline.dump +-rw-r--r-- 1 26 tape 1224943 Oct 24 12:43 paperless.dump +-rw-r--r-- 1 26 tape 2754 Oct 24 12:43 postgres.dump +-rw-r--r-- 1 26 tape 0 Oct 24 12:43 SUCCESS +-rw-r--r-- 1 26 tape 594915 Oct 24 12:43 tandoor.dump +-rw-r--r-- 1 26 tape 648505 Oct 24 12:43 wger.dump +-rw-r--r-- 1 26 tape 382029 Oct 24 12:43 zipline.dump +gitea.dump pg_restore --list rc=0 entries=1114 +authentik.dump pg_restore --list rc=0 entries=1660 +paperless.dump pg_restore --list rc=0 entries=753 +CronJob lastSuccessfulTime: 2026-10-24T10:43:55Z + +## felhom-dooplex-offsite by hand + restore test +Oct 24 12:48:40 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: database: 20261024-104341, 8598648 bytes, 4 min old +Oct 24 12:49:30 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: files: 12 repositories, 813 MB +Oct 24 12:49:30 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: secrets: 3 file(s) of 20261024_031421 +Oct 24 12:49:32 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: vaultwarden: integrity ok, 1 user(s), files: db.sqlite3 rsa_key.pem +Oct 24 12:49:32 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: signing: 3 key(s) with their public halves +Oct 24 12:50:20 dooplex felhom-dooplex-offsite[509023]: dooplex.pxar: had to backup 422.032 MiB of 750.918 MiB (compressed 413.468 MiB) in 43.59 s (average 9.681 MiB/s) +Oct 24 12:50:20 dooplex felhom-dooplex-offsite[509023]: dooplex.pxar: backup was done incrementally, reused 328.886 MiB (43.8%) +Oct 24 12:50:22 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: pushed to ep0 (ns operator, host/dooplex-gitea) in 46 s +Oct 24 12:50:22 dooplex felhom-dooplex-offsite[501204]: felhom-dooplex-offsite: success signal written +Oct 24 12:50:23 dooplex systemd[1]: Finished felhom-dooplex-offsite.service - Felhom: push Gitea + DooPlex secrets to ep0, encrypted (R-232). +Oct 24 12:50:40 dooplex felhom-dooplex-offsite-restore-test[521926]: felhom-dooplex-offsite-restore-test: restoring host/dooplex-gitea/2026-10-24T10:49:36Z +Oct 24 12:51:07 dooplex felhom-dooplex-offsite-restore-test[522066]: restore complete (750.918 MiB processed in 27.1s, average 27.745 MiB/s) +Oct 24 12:51:33 dooplex felhom-dooplex-offsite-restore-test[521926]: felhom-dooplex-offsite-restore-test: checked: 26456 files match the manifest, 12 repositories pass git fsck, gitea.dump readable, 3 secrets file(s), Vaultwarden 1 user(s) / 811 item(s), 3 signing key(s) match +Oct 24 12:51:33 dooplex felhom-dooplex-offsite-restore-test[521926]: felhom-dooplex-offsite-restore-test: success signal written +Oct 24 12:51:34 dooplex systemd[1]: Finished felhom-dooplex-offsite-restore-test.service - Felhom: restore-test the Gitea + secrets copy on ep0 (R-232). + +## ep0 (read only): host/dooplex-gitea snapshot 2026-10-24T10:49:36Z present (catalog.pcat1.didx, dooplex.pxar.didx, index.json.blob, 10:50Z) + +## alarms cleared (Prometheus ALERTS, 30 s step) +PostgresqlDumpStale last firing sample 10:44:00Z; DooplexGiteaOffsiteStale last firing sample 10:51:00Z +metric ages at 10:51Z: offsite 83 s, dump 470 s; Alertmanager /api/v2/alerts = [] at 10:51:52Z + +## the restic sweep: dooplex-backup.timer 03:14 local; 2026-10-24 03:15:48 snapshot a801930d of /mnt/5_hdd/backup/postgresql/dumps ok; not run by hand (apply_retention would forget today's snapshot); the next run is the proof (dated check) diff --git a/documentation/audits/dooplex-alarms-2026-10-24/part3-alarms.txt b/documentation/audits/dooplex-alarms-2026-10-24/part3-alarms.txt new file mode 100644 index 00000000..5ed82c43 --- /dev/null +++ b/documentation/audits/dooplex-alarms-2026-10-24/part3-alarms.txt @@ -0,0 +1,21 @@ +# Part 3: alarms, 2026-10-24T10:58:26Z +firing now: Prometheus /api/v1/alerts none; Alertmanager /api/v2/alerts [] + +## ALERTS{alertstate=firing} 7 days (Prometheus range start = retention window), episodes + +## Alertmanager mail counter (old pod): 60 sent in 7 d, 20 in 24 h, 0 failed (all reasons) +## Gmail connector mailbox (felhom.eu catch-all): 0 alarm mails in 3 d — SMTP_TO is another mailbox; delivery seen by the operator only +## silences +active 2026-10-12T14:32 -> 2026-11-11T14:32 cc (operator yes 2026-10-12) | disk Z6A07P2G evacuated 2026-10-10, waiting for physical removal +expired 2026-10-23T12:06 -> 2026-10-23T14:05 cc (operator OK 2026-10-23) | DooPlex maintenance window 2026-10-23 (HM-043) + +## routing after homelab 22d7697 + pod restart (read from /api/v2/status) +root repeat 12h group_by ['alertname', 'severity'] + {'component': 'backup'} ['component'] 4h + {'alertname': 'DiskReportedUncorrectGrowing'} 4h + {'alertname': 'DiskReallocatedSectorsGrowing'} 4h + {'component': 'disk'} ['component', 'device'] 1d + {'component': 'workload'} 6h + {'severity': 'critical'} 4h + {'severity': 'warning'} 12h +send_resolved True uptime 2026-10-24T10:56:44.705Z diff --git a/documentation/audits/dooplex-alarms-2026-10-24/part4-r1250-live-proof.txt b/documentation/audits/dooplex-alarms-2026-10-24/part4-r1250-live-proof.txt new file mode 100644 index 00000000..ea988f15 --- /dev/null +++ b/documentation/audits/dooplex-alarms-2026-10-24/part4-r1250-live-proof.txt @@ -0,0 +1,59 @@ +# R-1250 live proof, demo-hp, 2026-10-24 — probes built from agent 1e087bb and controller 046b808 (clean clones; not committed probes: the agent's linetest.Tester behind the two routes as plain HTTP on the host; the controller's upwatch.Service hold inside 9202's controller container against the real qBittorrent 5.2.4) +# Seed: a private LAN-only 2 GiB torrent made by 9202's qBittorrent, downloaded by DooPlex's k3s qBittorrent (operator's offer) through a TCP+UDP forwarder on 9202's :6881; 9202's own limit 1250000 B/s (stored 1249280) stands in for the household's + +== 11:40:14 0. before: uplink "mbps_5s":11.4927312; agent: {"data":{"present":false,"waiting_for_quiet":true},"ok":true} +qbittorrent limit {Bytes:1249280 Enabled:false KBps:0} err= +store: +== 11:40:21 1. CONTROL — a test on request WITHOUT the hold (the seed keeps uploading): +{"data":{"error":"the uplink sent 11.1 Mbit/s while held","ok":false,"refused":"not-quiet"},"ok":true} + +== 11:40:36 2. a controller killed in the middle (kill -9 while the agent call runs): +2026/10/24 11:40:37 probe: agent waiting_for_quiet=true +2026/10/24 11:40:37 [INFO] [upwatch] line test: qbittorrent held at {Bytes:1024 Enabled:false KBps:0} (was {Bytes:1249280 Enabled:false KBps:0}) — asking the agent for a test +2026/10/24 11:40:37 probe: killmid — sleeping inside the agent call; kill -9 me now + after the kill: uplink "mbps_5s":0.0409568 +qbittorrent limit {Bytes:1024 Enabled:false KBps:0} err= +store: {"line_test_hold":{"app":"qbittorrent","prev":{"bytes":1249280},"held":{"bytes":1024},"at":"2026-10-24T11:40:37Z"}} +== 11:40:52 3. the restarted controller's start-up check: +2026/10/24 11:40:53 [INFO] [upwatch] qbittorrent was left at the line-test value (since 2026-10-24T11:40:37Z) — putting its limit back +2026/10/24 11:40:53 [INFO] [upwatch] qbittorrent's upload limit is back as it was ({Bytes:1249280 Enabled:false KBps:0}) +RecoverHold -> true +store: {} +qbittorrent limit {Bytes:1249280 Enabled:false KBps:0} err= +store: {} + uplink "mbps_5s":10.6444544 +== 11:41:00 4. the real hold: SeedLineTestNow (sampling the uplink every 5 s alongside): +2026/10/24 11:41:01 probe: agent waiting_for_quiet=true +2026/10/24 11:41:01 [INFO] [upwatch] line test: qbittorrent held at {Bytes:1024 Enabled:false KBps:0} (was {Bytes:1249280 Enabled:false KBps:0}) — asking the agent for a test + 11:41:00 uplink "mbps_5s":2.6076112 + 11:41:05 uplink "mbps_5s":0.0209008 + 11:41:10 uplink "mbps_5s":0.164744 + 11:41:15 uplink "mbps_5s":69.2326144 + 11:41:20 uplink "mbps_5s":104.54756 + 11:41:25 uplink "mbps_5s":34.787547200000006 + 11:41:30 uplink "mbps_5s":58.9638304 + 11:41:35 uplink "mbps_5s":100.62775359999999 + 11:41:40 uplink "mbps_5s":40.3203488 + 11:41:45 uplink "mbps_5s":57.652910399999996 + 11:41:50 uplink "mbps_5s":104.8901584 +2026/10/24 11:41:57 [INFO] [upwatch] qbittorrent's upload limit is back as it was ({Bytes:1249280 Enabled:false KBps:0}) +2026/10/24 11:41:57 [INFO] [upwatch] line test on request: 94.2 Mbit/s +SeedLineTestNow returned after 56.8 s +store: {} + 11:41:55 uplink "mbps_5s":50.785411200000006 + 11:42:00 uplink "mbps_5s":10.329262400000001 + 11:42:05 uplink "mbps_5s":11.0489872 +== 11:42:10 5. after: agent: {"data":{"mbps":94.16292344340873,"measured_at":"2026-10-24T11:41:16Z","present":true,"waiting_for_quiet":false},"ok":true} +qbittorrent limit {Bytes:1249280 Enabled:false KBps:0} err= +store: {} + uplink "mbps_5s":10.612328 + +## agent probe log (the requested test) +time=2026-10-24T13:40:36.092+02:00 level=INFO msg="linetest: test on request refused — the uplink is not quiet" mbps=11.1 +time=2026-10-24T13:40:36.092+02:00 level=INFO msg="probe: POST /host/line-test/run" seconds=15.014445708 mbps=0 err="linetest: not-quiet: the uplink sent 11.1 Mbit/s while held" +time=2026-10-24T13:41:16.501+02:00 level=INFO msg="linetest: test on request (a held torrent app, R-1250)" server=10.77.0.1:5201 +time=2026-10-24T13:41:26.830+02:00 level=INFO msg="linetest: run 1: 93.6 Mbit/s" server=10.77.0.1:5201 server_bytes=117571584 server_seconds=10.049774 +time=2026-10-24T13:41:42.187+02:00 level=INFO msg="linetest: run 2: 89.6 Mbit/s" server=10.77.0.1:5201 server_bytes=112590848 server_seconds=10.056309 +time=2026-10-24T13:41:57.557+02:00 level=INFO msg="linetest: run 3: 94.2 Mbit/s" server=10.77.0.1:5201 server_bytes=118358016 server_seconds=10.055594 +time=2026-10-24T13:41:57.557+02:00 level=INFO msg="linetest: upload line measured: 94.2 Mbit/s (best of 3)" server=10.77.0.1:5201 runs=3 duration_s=41.05557078 +time=2026-10-24T13:41:57.557+02:00 level=INFO msg="probe: POST /host/line-test/run" seconds=56.058478583 mbps=94.16292344340873 err= diff --git a/documentation/audits/dooplex-alarms-2026-10-24/postgresql-backup.yaml.before b/documentation/audits/dooplex-alarms-2026-10-24/postgresql-backup.yaml.before new file mode 100644 index 00000000..9561fb9c --- /dev/null +++ b/documentation/audits/dooplex-alarms-2026-10-24/postgresql-backup.yaml.before @@ -0,0 +1,151 @@ +# ============================================================================ +# postgresql-1 logical backup — 6-hourly pg_dump to HDD5 (rides Restic) +# ---------------------------------------------------------------------------- +# The shared CNPG cluster `postgresql` (database-system, single instance) had no +# backup. This CronJob does a LOGICAL backup every 6h: +# - pg_dumpall --globals-only -> globals.sql (roles/grants; needed for restore) +# - pg_dump -Fc per database -> .dump (custom format; selective restore) +# written to hostPath /mnt/5_hdd/backup/postgresql/dumps// (HDD5, OFF +# the DB's own HDD4 disk). The host backup flow (/opt/backup, backup-postgresql.sh) +# then restic-backs-up that dir in its daily run (7d/4w/6m retention). Local +# retention here = 3 days. RPO <=6h, no PITR (accepted). +# +# Auth: connects to postgresql-rw as the CNPG-managed superuser (secret +# `postgres-superuser`, keys username/password). Requires enableSuperuserAccess: +# true on the Cluster (set in postgresql-cluster.yaml). The superuser is the only +# role that can read every DB + dump globals; per-app roles cannot. +# +# Freshness alert: kube_cronjob_status_last_successful_time (KSM) — see +# mon-system/prometheus-rules.yaml (PostgresqlDumpStale, >12h). +# ============================================================================ +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: postgresql-backup-script + namespace: database-system + labels: + app.kubernetes.io/name: postgresql-backup + app.kubernetes.io/instance: postgresql-backup +data: + pg-backup.sh: | + #!/usr/bin/env bash + set -euo pipefail + export PGHOST="${PGHOST:-postgresql-rw}" PGPORT="${PGPORT:-5432}" + TS="$(date +%Y%m%d-%H%M%S)" + OUT="/backup/dumps/${TS}" + mkdir -p "${OUT}" + echo "[pgbackup] ${TS} -> ${OUT}" + + echo "[pgbackup] globals (roles/grants)" + pg_dumpall --globals-only > "${OUT}/globals.sql" + + mapfile -t DBS < <(psql -tAc \ + "SELECT datname FROM pg_database WHERE NOT datistemplate AND datallowconn ORDER BY 1;") + echo "[pgbackup] databases: ${DBS[*]}" + for db in "${DBS[@]}"; do + echo "[pgbackup] pg_dump -Fc ${db}" + pg_dump -Fc -d "${db}" -f "${OUT}/${db}.dump" + done + + touch "${OUT}/SUCCESS" + echo "[pgbackup] pruning local dumps older than 3 days" + find /backup/dumps -mindepth 1 -maxdepth 1 -type d -mtime +3 -exec rm -rf {} + || true + echo "[pgbackup] done" + ls -lh "${OUT}" +--- +apiVersion: batch/v1 +kind: CronJob +metadata: + name: postgresql-backup + namespace: database-system + labels: + app.kubernetes.io/name: postgresql-backup + app.kubernetes.io/instance: postgresql-backup +spec: + schedule: "0 */6 * * *" + timeZone: "Europe/Budapest" + concurrencyPolicy: Forbid + successfulJobsHistoryLimit: 3 + failedJobsHistoryLimit: 3 + startingDeadlineSeconds: 600 + jobTemplate: + metadata: + labels: + app.kubernetes.io/name: postgresql-backup + app.kubernetes.io/instance: postgresql-backup + spec: + backoffLimit: 2 + activeDeadlineSeconds: 1800 + template: + metadata: + labels: + app.kubernetes.io/name: postgresql-backup + app.kubernetes.io/instance: postgresql-backup + annotations: + # postgres image is pinned to the CNPG cluster's version (X.Y) + match-regex.version-checker.io/pgbackup: '^\d+\.\d+$' + spec: + enableServiceLinks: false + restartPolicy: OnFailure + securityContext: + runAsNonRoot: true + runAsUser: 26 + runAsGroup: 26 + fsGroup: 26 + containers: + - name: pgbackup + # MUST match the CNPG cluster's Postgres version (pg_dump >= server). + image: ghcr.io/cloudnative-pg/postgresql:17.2 + imagePullPolicy: IfNotPresent + command: ["/bin/bash", "/scripts/pg-backup.sh"] + env: + - name: PGHOST + value: postgresql-rw + - name: PGPORT + value: "5432" + - name: PGUSER + valueFrom: + secretKeyRef: + name: postgresql-superuser + key: username + - name: PGPASSWORD + valueFrom: + secretKeyRef: + name: postgresql-superuser + key: password + - name: HOME + value: /tmp + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: "1" + memory: 512Mi + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + volumeMounts: + - name: script + mountPath: /scripts + readOnly: true + - name: backup + mountPath: /backup + - name: tmp + mountPath: /tmp + volumes: + - name: script + configMap: + name: postgresql-backup-script + defaultMode: 0755 + - name: backup + hostPath: + path: /mnt/5_hdd/backup/postgresql + type: Directory + - name: tmp + emptyDir: + sizeLimit: 256Mi diff --git a/documentation/audits/dooplex-alarms-2026-10-24/prometheus-rules.yaml.before b/documentation/audits/dooplex-alarms-2026-10-24/prometheus-rules.yaml.before new file mode 100644 index 00000000..01100a9b --- /dev/null +++ b/documentation/audits/dooplex-alarms-2026-10-24/prometheus-rules.yaml.before @@ -0,0 +1,879 @@ +# ============================================================================= +# Prometheus Alerting Rules for Longhorn +# ============================================================================= +# ============================================================================= +# Prometheus Alerting Rules for Longhorn +# Excludes prometheus-data PVC since it's designed to run at ~95% capacity +# ============================================================================= +apiVersion: v1 +kind: ConfigMap +metadata: + name: prometheus-rules + namespace: mon-system + labels: + app: prometheus +data: + + authentik-alerts.yml: | + groups: + - name: authentik-availability + rules: + # Fires if Prometheus cannot scrape the server metrics endpoint. + # Equivalent to "authentik-server pod is unreachable or crashed." + - alert: AuthentikServerDown + expr: up{job="authentik-server"} == 0 + for: 3m + labels: + severity: critical + component: authentik + annotations: + summary: "Authentik server is unreachable" + description: "authentik-server scrape has been failing for 3+ minutes. Logins will fail." + + - alert: AuthentikWorkerDown + expr: up{job="authentik-worker"} == 0 + for: 3m + labels: + severity: critical + component: authentik + annotations: + summary: "Authentik worker is unreachable" + description: "authentik-worker scrape has been failing for 3+ minutes. Background tasks are not running." + + # Per-outpost disconnect. Downgraded to warning because a single + # outpost failure only breaks a subset of apps (e.g. just the Arr + # stack if arr-outpost is down), not the entire IDP. + - alert: AuthentikOutpostDown + expr: up{job="authentik-outposts"} == 0 + for: 5m + labels: + severity: warning + component: authentik + annotations: + summary: "Authentik outpost {{ $labels.outpost }} is unreachable" + description: "Outpost {{ $labels.outpost }} has been down for 5+ minutes. Apps behind this outpost cannot authenticate." + + - name: authentik-functional + rules: + # *** The 13-days-ago alert. *** + # Fires when the worker has tasks in progress but throughput + # has collapsed. Probes cannot catch this because the pod is + # technically alive — only the queue dynamics give it away. + - alert: AuthentikTaskQueueStuck + expr: | + sum(authentik_tasks_in_progress{job="authentik-worker"}) > 5 + and + sum(rate(authentik_tasks_total{job="authentik-worker"}[5m])) < 0.01 + for: 10m + labels: + severity: critical + component: authentik + annotations: + summary: "Authentik worker queue is stuck" + description: "{{ $value }} tasks in progress with near-zero throughput for 10+ min. Worker is alive but not draining the queue — typically resolved by restarting the deployment." + + # Softer version: queue is growing but still moving somewhat. + # Could indicate an expensive task blocking the workers, or + # a task that keeps retrying, or steady overload. + - alert: AuthentikTaskBacklog + expr: sum(authentik_tasks_in_progress{job="authentik-worker"}) > 20 + for: 15m + labels: + severity: warning + component: authentik + annotations: + summary: "Authentik task backlog >20 for 15 min" + description: "{{ $value }} tasks in progress for 15+ min. Likely overload or a single hanging task." + + # User-visible error signal. Threshold is conservative — + # on a low-traffic homelab, 0.1 err/sec = ~6 errors/min which + # is already noticeable to users. + - alert: AuthentikHighErrorRate + expr: | + sum(rate(django_http_responses_total_by_status_total{job="authentik-server",status=~"5.."}[5m])) > 0.1 + for: 5m + labels: + severity: warning + component: authentik + annotations: + summary: "Authentik is serving 5xx errors" + description: "{{ $value | printf \"%.2f\" }} 5xx responses/sec for 5+ min." + + # p95 latency guard. 2s is high for modern auth flows — at this + # point users are visibly waiting on the login page. + - alert: AuthentikHighLatency + expr: | + histogram_quantile(0.95, + sum by (le) (rate(authentik_main_request_duration_seconds_bucket{job="authentik-server",dest="core"}[5m])) + ) > 2 + for: 10m + labels: + severity: warning + component: authentik + annotations: + summary: "Authentik p95 request latency > 2s" + description: "p95 latency {{ $value | printf \"%.2f\" }}s for 10+ min. Logins are slow." + + postgresql-alerts.yml: | + groups: + - name: postgresql-availability + rules: + - alert: PostgresExporterDown + expr: up{job="cloudnativepg"} == 0 + for: 2m + labels: + severity: critical + component: postgresql + annotations: + summary: "CloudNativePG metrics endpoint unreachable" + description: "CNPG metrics exporter on {{ $labels.pod }} has been down for 2+ min. Postgres may be down or the sidecar may have crashed." + + - name: postgresql-capacity + rules: + # Threshold of 80% of your 200 max_connections = 160. + # If you raise max_connections later, update the number. + - alert: PostgresHighConnections + expr: sum by (cluster) (cnpg_backends_total) > 160 + for: 5m + labels: + severity: warning + component: postgresql + annotations: + summary: "Postgres cluster {{ $labels.cluster }} nearing connection limit" + description: "{{ $value }} active connections (>80% of max_connections=200). Check for connection leaks." + + # Locks held long enough that other queries are waiting. + # Usually cleared fast; a sustained non-zero value is abnormal. + - alert: PostgresBackendsWaiting + expr: cnpg_backends_waiting_total > 0 + for: 5m + labels: + severity: warning + component: postgresql + annotations: + summary: "Postgres has queries blocked on locks" + description: "{{ $value }} backends waiting on locks for 5+ min. Investigate long-running transactions." + + longhorn-alerts.yml: | + groups: + - name: longhorn-volume-alerts + rules: + # Critical: volume filesystem 95% full + - alert: LonghornVolumeSpaceCritical + # Filesystem fill as the pod sees it (kubelet), NOT longhorn_volume_actual_size_bytes: + # actual size also counts blocks held only by Longhorn snapshots, so a volume that + # rewrites files (felhom hub-data: 31% full) read 101% and alarmed all night (2026-10-13). + # label_join keeps a `volume` label (namespace/pvc) for the alertmanager inhibit rule. + expr: | + label_join( + max by (namespace, persistentvolumeclaim) (kubelet_volume_stats_used_bytes) + / + max by (namespace, persistentvolumeclaim) (kubelet_volume_stats_capacity_bytes), + "volume", "/", "namespace", "persistentvolumeclaim" + ) * 100 > 95 + for: 5m + labels: + severity: critical + annotations: + summary: "Longhorn volume {{ $labels.volume }} is critically full" + description: "Volume {{ $labels.volume }} is at {{ printf \"%.1f\" $value }}% capacity. Immediate action required." + # Warning: volume filesystem 85% full + - alert: LonghornVolumeSpaceWarning + # Filesystem fill as the pod sees it (kubelet), NOT longhorn_volume_actual_size_bytes: + # actual size also counts blocks held only by Longhorn snapshots, so a volume that + # rewrites files (felhom hub-data: 31% full) read 101% and alarmed all night (2026-10-13). + # label_join keeps a `volume` label (namespace/pvc) for the alertmanager inhibit rule. + expr: | + label_join( + max by (namespace, persistentvolumeclaim) (kubelet_volume_stats_used_bytes) + / + max by (namespace, persistentvolumeclaim) (kubelet_volume_stats_capacity_bytes), + "volume", "/", "namespace", "persistentvolumeclaim" + ) * 100 > 85 + for: 10m + labels: + severity: warning + annotations: + summary: "Longhorn volume {{ $labels.volume }} is running low on space" + description: "Volume {{ $labels.volume }} is at {{ printf \"%.1f\" $value }}% capacity. Consider expanding or cleaning up." + # Volume degraded + - alert: LonghornVolumeDegraded + expr: longhorn_volume_robustness != 1 + for: 5m + labels: + severity: warning + annotations: + summary: "Longhorn volume {{ $labels.volume }} is degraded" + description: "Volume {{ $labels.volume }} robustness is not healthy. Check replica status." + + # Node storage pressure + - alert: LonghornNodeStoragePressure + expr: | + ( + longhorn_node_storage_usage_bytes + / + longhorn_node_storage_capacity_bytes + ) * 100 > 90 + for: 10m + labels: + severity: warning + annotations: + summary: "Longhorn node {{ $labels.node }} storage pressure" + description: "Node {{ $labels.node }} disk usage is at {{ printf \"%.1f\" $value }}%." + + # Add this as a new data key (wan-alerts.yml) in the existing + # prometheus-rules ConfigMap (mon-system). Thresholds anchored to One.hu's + # "normal conditions" figures: 700 Mbit/s down / 28 Mbit/s up. + # Throughput is sampled every ~15 min, so `for:` spans >=2 samples to avoid + # firing on a single fluke. Recalibrate floors after a week of baseline data. + # NOTE: uses Prometheus template funcs (humanize/humanizePercentage/humanizeDuration); + # mul/div are NOT valid Prometheus template functions. + wan-alerts.yml: | + groups: + - name: wan-quality-alerts + rules: + # --- upstream loss: the prime suspect for dropped calls / WireGuard --- + # Retuned 2026-08-14: 1% for 2m caught ordinary transient loss. + # Measured over 24h, upstream peaks at 5.8% and downstream at 0.64% + # on a link that is otherwise fine. 3% held for 15m means loss that + # actually persists, which is what breaks calls and WireGuard. + - alert: WanUpstreamPacketLoss + expr: wan_irtt_loss_ratio{direction="upstream",condition="idle"} > 0.03 + for: 15m + labels: + severity: warning + annotations: + summary: "WAN upstream packet loss to {{ $labels.target }}" + description: "irtt upstream loss {{ $value | humanizePercentage }} (>3%) sustained 15m. Brief spikes to ~6% are normal on this link; sustained loss is not. Cable-upstream symptom; capture for ISP." + + - alert: WanDownstreamPacketLoss + expr: wan_irtt_loss_ratio{direction="downstream",condition="idle"} > 0.03 + for: 15m + labels: + severity: warning + annotations: + summary: "WAN downstream packet loss to {{ $labels.target }}" + description: "irtt downstream loss {{ $value | humanizePercentage }} (>3%) sustained 15m. Measured baseline peak here is 0.64%." + + # --- latency / jitter --- + # Retuned 2026-08-14. This watched stat="max" -- the single worst + # packet in a probe run -- which is inherently spiky: measured over + # 24h its p99 was 327 ms and its absolute peak 1786 ms, on a link + # whose mean RTT p99 is a perfectly healthy 41.4 ms. The rule was + # therefore true 35.4% of the day. Now keyed on the mean, at a + # threshold ~3.5x the observed p99, held for 15m: sustained real + # degradation, not one late packet. + - alert: WanLatencyHigh + expr: wan_irtt_rtt_seconds{stat="mean",condition="idle"} > 0.15 + for: 15m + labels: + severity: warning + annotations: + summary: "WAN latency sustained high to {{ $labels.target }}" + description: "irtt MEAN RTT {{ $value | humanizeDuration }} (>150 ms) sustained 15m while idle. Normal here is ~27 ms median with a 41 ms p99, so this is a real change, not a spike. Real-time apps will feel it." + + # 30 ms tripped on ordinary variance; measured round-trip jitter sits + # near 3 ms with brief 11 ms peaks. 60 ms held for 15m means the line + # is genuinely unstable. + - alert: WanJitterHigh + expr: wan_irtt_jitter_seconds{direction="round_trip",condition="idle"} > 0.06 + for: 15m + labels: + severity: warning + annotations: + summary: "WAN jitter high to {{ $labels.target }}" + description: "Round-trip jitter {{ $value | humanizeDuration }} (>60 ms) sustained 15m. Baseline is ~3 ms. Degrades VoIP/video." + + # --- bufferbloat: latency added while the line is saturated --- + - alert: WanBufferbloat + expr: | + ( + wan_irtt_rtt_seconds{stat="mean",condition="under_load"} + - on(target) wan_irtt_rtt_seconds{stat="mean",condition="idle"} + ) > 0.1 + for: 0m + labels: + severity: info + annotations: + summary: "WAN bufferbloat on {{ $labels.target }}" + description: "RTT rises {{ $value | humanizeDuration }} under load (>100 ms). Line buckles when saturated." + + # --- throughput vs One.hu "normal" 700/28 (alert below 50%) --- + # Retuned 2026-08-14. iperf3 runs vary a lot by themselves -- measured + # download ranged 88 Mb/s to 869 Mb/s over 24h on a healthy link, so + # "below half of plan" fired on ordinary variance. 100 Mb/s is ~1/7th + # of normal: that is a broken line, not a slow run. 30m spans at + # least two probe cycles (cadence peaks at 16.4 min), so a single bad + # measurement cannot trigger it. + - alert: WanDownloadDegraded + expr: wan_throughput_bits_per_second{direction="download"} < 100e6 + for: 30m + labels: + severity: warning + annotations: + summary: "WAN download severely degraded" + description: "Download {{ $value | humanize }}bit/s (<100M) for 30m, against a normal range of roughly 200-870 Mb/s. Two consecutive probe cycles agree, so this is the line rather than a noisy iperf3 run." + + # The old rule assumed a 28 Mb/s plan and alerted under 14M. Measured + # upload is ~99 Mb/s with a 24h floor of 37 Mb/s, so the plan comment + # was stale and the threshold unreachable -- it could never fire. + - alert: WanUploadDegraded + expr: wan_throughput_bits_per_second{direction="upload"} < 30e6 + for: 30m + labels: + severity: warning + annotations: + summary: "WAN upload severely degraded" + description: "Upload {{ $value | humanize }}bit/s (<30M) for 30m. Measured normal is ~99 Mb/s with a 24h low of 37 Mb/s." + + # --- the monitor itself stopped producing data --- + # Retuned 2026-08-14. The 300s threshold was SHORTER THAN THE PROBE + # INTERVAL, so this could not not fire: measured over 24h, normal + # staleness peaks at 16.4 min for both the irtt and throughput + # probes. It flapped continuously (firing 15:02-19:32, 21:47-03:02, + # 05:32-07:32 on 2026-08-13/14) and trained the eye to ignore WAN + # mail. 45 min is ~3x the observed cadence, so it now means the + # monitor is genuinely dead rather than merely between runs. + - alert: WanProbeStalled + expr: time() - max by(probe) (wan_probe_last_run_timestamp_seconds) > 2700 + for: 0m + labels: + severity: warning + annotations: + summary: "WAN probe '{{ $labels.probe }}' stalled" + description: "No fresh samples for >45 min, which is about 3 probe cycles (normal cadence peaks at 16.4 min). The wan-monitor pod or its Hetzner endpoint is stuck, so every other WAN alert is now blind. Check kubectl -n admin-system logs deploy/wan-monitor." + + # A single external target failing is that site's problem, not yours. + # jarrs.eu alone was unreachable 1.25% of 24h while every other + # target stayed at 0.00%. Raised 3m -> 30m so this only fires for a + # sustained single-target outage; the real link failure is WanOutage. + - alert: WanBlackboxTargetDown + expr: probe_success{job=~"wan-.*"} == 0 + for: 30m + labels: + severity: warning + annotations: + summary: "WAN probe to {{ $labels.instance }} failing" + description: "blackbox {{ $labels.job }} to {{ $labels.instance }} unreachable for 30m. One target only -- if the rest are fine this is the remote site, not your connection. If several are failing, WanOutage will fire instead." + + # THE actual "the internet is gone" alert, and the only WAN rule at + # critical. Fires when most EXTERNAL targets fail at once; LAN + # targets (192.168.x) are excluded so a router/DNS problem does not + # masquerade as a WAN outage. Over the measured 24h this never came + # close: 13 targets, worst single one down 1.25%, everything else + # 0.00%. + - alert: WanOutage + expr: | + ( + count(probe_success{job=~"wan-.*", instance!~"192\\.168\\..*"} == 0) + / + count(probe_success{job=~"wan-.*", instance!~"192\\.168\\..*"}) + ) > 0.6 + for: 5m + labels: + severity: critical + annotations: + summary: "WAN outage: {{ $value | humanizePercentage }} of external targets unreachable" + description: "More than 60% of external probe targets (ICMP, DNS and HTTP across independent destinations) have been failing for 5m. That is a real connectivity loss rather than one flaky endpoint. Check the ONE.hu link and the router before looking at anything in the cluster." + node-housekeeping-alerts.yml: | + groups: + - name: node-housekeeping + rules: + # The weekly housekeeping timer has not SUCCEEDED in over 14 days. + # Weekly schedule => this means ~2 consecutive missed/failed runs. + - alert: HousekeepingStale + expr: time() - node_housekeeping_last_success_timestamp_seconds > 1209600 + for: 1h + labels: + severity: warning + component: node-housekeeping + annotations: + summary: "Node housekeeping is stale on {{ $labels.instance }}" + description: "No successful node-housekeeping run for >14 days (last success {{ printf \"%.0f\" $value }}s ago). The weekly timer may be disabled, failing, or the host has been off. Check `systemctl status node-housekeeping.timer` and the last journal for node-housekeeping.service." + + # The last run exited non-zero. last_success is preserved across + # failures, so HousekeepingStale stays quiet until it really lapses; + # this catches the failure immediately. + - alert: HousekeepingFailing + expr: node_housekeeping_exit_code != 0 + for: 1h + labels: + severity: warning + component: node-housekeeping + annotations: + summary: "Node housekeeping last run failed on {{ $labels.instance }}" + description: "node-housekeeping.service exited with code {{ $value }}. Inspect `journalctl -u node-housekeeping.service`. Image prune or journal vacuum likely errored." + + # Judgment alert: housekeeping IS running (success within 14d) but `/` + # is still tight (<25% avail). That means the pressure is NOT stale + # images/journals — it is data growth. Investigate rather than clean. + # NB: <25% avail on `/` is also roughly where Longhorn's default disk + # crosses its storage reservation and stops scheduling new replicas. + # + # `and on(instance)` is LOAD-BEARING, not style. A bare `and` requires + # IDENTICAL label sets on both sides. The left vector carries + # {device,fstype,instance,job,mountpoint} from node-exporter; the right + # is a textfile metric carrying only {instance,job}. No series ever + # matched, so this rule was dead from the day it was written and never + # fired through the 2026-07 build-cache incident that took `/` to 86%. + # Red-proof (2026-08-05, both sides individually true, threshold + # substituted to 0.5 because the incident had already been cleared): + # bare `and` -> result_count 0 + # `and on(instance)` -> result_count 1 + # Evidence: felhom.eu documentation/audits/SPIKE-dooplex-buildcache-2026-08-05.md + - alert: RootFsPressureDespiteHousekeeping + expr: | + node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.25 + and on(instance) + time() - node_housekeeping_last_success_timestamp_seconds < 1209600 + for: 30m + labels: + severity: warning + component: node-housekeeping + annotations: + summary: "Root fs <25% free on {{ $labels.instance }} despite current housekeeping" + description: "`/` is below 25% available even though node-housekeeping succeeded within the last 14 days. Image/journal cleanup is not the cause. Check UNBOUNDED DOCKER BUILD CACHE FIRST — `docker buildx du` — which caused the 2026-07 incident (157 GB, ~95% of that growth) and is capped at 30 GB via /etc/docker/daemon.json; a cap that has been reverted or a daemon that failed to apply it looks exactly like this. `docker builder prune -af` reclaims it synchronously. Only if the cache is small, investigate Longhorn replica/snapshot growth on the default disk (~10% of that incident; also near where the default Longhorn disk hits its scheduling floor)." + + postgresql-backup-alerts.yml: | + groups: + - name: postgresql-backup + rules: + - alert: PostgresqlDumpStale + expr: time() - kube_cronjob_status_last_successful_time{cronjob="postgresql-backup",namespace="database-system"} > 43200 + for: 15m + labels: + severity: warning + component: database-backup + annotations: + summary: "postgresql-1 logical backup is stale on {{ $labels.cronjob }}" + description: "No successful postgresql-backup CronJob run in >12h. pg_dump dumps land on /mnt/5_hdd/backup/postgresql/dumps and are swept by restic daily. Check kubectl -n database-system get cronjob postgresql-backup and the last job logs." + + backup-freshness-alerts.yml: | + groups: + - name: backup-freshness + rules: + # Gauges come from homelab-ansible role backup_freshness (every 15 min, textfile + # collector). The gauge is 0 — not absent — when no world-*.tar.gz exists, so an empty + # folder or an unmounted HDD5 fires this too. mc-backup runs every 6h relative to pod + # start; 13h = two missed runs. + - alert: MinecraftBackupStale + expr: time() - minecraft_backup_last_success_timestamp_seconds > 13 * 3600 + for: 30m + labels: + severity: warning + component: backup + annotations: + summary: "minecraft-tlauncher world backup is stale" + description: "Newest world-*.tar.gz in /mnt/5_hdd/backup/minecraft-tlauncher is {{ $value | humanizeDuration }} old (6h interval). Check `kubectl -n crafty-system logs deploy/minecraft-tlauncher -c backup --tail=50` — an RCON login failure means the two containers disagree on the password (Secret minecraft-tlauncher-rcon) — and that /mnt/5_hdd is mounted." + + # Guards the guard: if the freshness timer stops, the gauge above freezes at its last + # value and MinecraftBackupStale can never fire. node_textfile_mtime_seconds carries the + # FULL in-container path (measured: file="/host/textfile/node_housekeeping.prom"), hence + # the regex rather than file="backup_freshness.prom", which would match nothing. + - alert: BackupFreshnessExporterDead + expr: | + absent(minecraft_backup_last_success_timestamp_seconds) + or + time() - node_textfile_mtime_seconds{file=~".*/backup_freshness\\.prom"} > 3600 + for: 15m + labels: + severity: warning + component: backup + annotations: + summary: "backup-freshness textfile exporter is not updating" + description: "backup_freshness.prom is missing or has not been rewritten for >1h (15-min timer). Backup staleness alerts are blind until it is fixed. Check `systemctl status backup-freshness.timer` and `journalctl -u backup-freshness.service`." + + # Felhom hub database off-site copy (R-173, felhom.eu runbooks/RUNBOOK-hub-db-offsite-backup.md). + # Written ONLY on success by felhom-hub-db-backup (every 6 h since D34, 2026-10-23: 02:30 08:30 14:30 20:30) + # and felhom-hub-db-restore-test (Sun 04:30), + # felhom.eu scripts/hub-db-backup/. absent() makes "the script never ran / the file is gone" an alarm too. + - alert: HubDBBackupStale + expr: | + time() - felhom_hub_db_backup_last_success_timestamp_seconds > 13 * 3600 + or + absent(felhom_hub_db_backup_last_success_timestamp_seconds) + for: 30m + labels: + severity: critical + component: backup + annotations: + summary: "The Felhom hub database has not reached ep0 for 13 h (R-173, D34)" + description: "No successful push of the hub DB snapshot to ep0 for >13h — two 6-hourly pushes missed (02:30 08:30 14:30 20:30). Check `journalctl -u felhom-hub-db-backup.service`, the tunnel `systemctl status felhom-ep0-pbs-tunnel`, and that the hub logs `db snapshot written` at 02:00/08:00/14:00/20:00." + # Felhom release copy to ep0 (D33, R-1227; felhom.eu scripts/release-copy/, runbooks/total-loss-of-dooplex.md). + # Written ONLY after the push AND a checksum compare showed no difference (nightly 01:40). + - alert: FelhomReleaseCopyStale + expr: | + time() - felhom_release_copy_last_success_timestamp_seconds > 26 * 3600 + or + absent(felhom_release_copy_last_success_timestamp_seconds) + for: 30m + labels: + severity: warning + component: backup + annotations: + summary: "The Felhom release images and agent packages have not reached ep0 for 26 h (D33)" + description: "No verified copy of the released images and agent packages to ep0 for >26h (nightly 01:40). Check `journalctl -u felhom-release-copy.service`." + - alert: HubDBRestoreTestStale + expr: | + time() - felhom_hub_db_restore_test_last_success_timestamp_seconds > 8 * 24 * 3600 + or + absent(felhom_hub_db_restore_test_last_success_timestamp_seconds) + for: 1h + labels: + severity: warning + component: backup + annotations: + summary: "The hub database copy on ep0 has not passed a restore test for 8 days (R-173)" + description: "The weekly restore test (Sun 04:30) has not succeeded for >8 days. Check `journalctl -u felhom-hub-db-restore-test.service`." + # Gitea + DooPlex secrets off-site (R-232 b/h). Written ONLY on success by felhom-dooplex-offsite (00:20 daily) and + # felhom-dooplex-offsite-restore-test (Sun 05:30), felhom.eu scripts/dooplex-offsite/. absent() included. + - alert: DooplexGiteaOffsiteStale + expr: | + time() - felhom_dooplex_offsite_last_success_timestamp_seconds > 26 * 3600 + or + absent(felhom_dooplex_offsite_last_success_timestamp_seconds) + for: 30m + labels: + severity: critical + component: backup + annotations: + summary: "Gitea and DooPlex's secrets have not reached ep0 for 26 h (R-232)" + description: "No successful off-site push of Gitea (repositories, database dump, config) and the secrets export for >26h (daily at 00:20). Check `journalctl -u felhom-dooplex-offsite.service` and the tunnel `systemctl status felhom-ep0-pbs-tunnel`." + - alert: DooplexGiteaRestoreTestStale + expr: | + time() - felhom_dooplex_offsite_restore_test_last_success_timestamp_seconds > 8 * 24 * 3600 + or + absent(felhom_dooplex_offsite_restore_test_last_success_timestamp_seconds) + for: 1h + labels: + severity: warning + component: backup + annotations: + summary: "The Gitea copy on ep0 has not passed a restore test for 8 days (R-232)" + description: "The weekly restore test (Sun 05:30) has not succeeded for >8 days. Check `journalctl -u felhom-dooplex-offsite-restore-test.service`." + + smart-alerts.yml: | + # ========================================================================= + # Physical disk health, from smartctl-exporter (admin-system). + # + # Written after 2026-08-13, when /dev/sdg (ST3000VX010, S/N Z6A07P2G) + # developed 120 uncorrectable sectors, corrupted the radarr-kids and + # opengist Longhorn volumes, and nothing alerted. Every volume in this + # cluster is numberOfReplicas: 1, so one bad sector is an outage plus data + # loss -- these alerts are the only early warning that exists. + # + # NOTHING HERE KEYS ON smartctl_device_smart_status. Measured on the day of + # the failure, that flag read PASSED (1) on all seven drives INCLUDING the + # one with 120 pending sectors and 665 reported uncorrectables. It is kept + # only as a last-resort critical signal, never as the primary one. + # + # The SSDs do not expose every attribute (the Apacer reports no pending / + # reallocated / uncorrectable at all), so the severity recording rules are + # split per check and combined with max by (device). A device that lacks an + # attribute simply contributes fewer series instead of dropping out of the + # score entirely, which is what a plain sum would do. + # ========================================================================= + groups: + - name: disk-smart-severity + rules: + # --- per-check severity: 0 = healthy, 1 = warning, 2 = critical --- + + # Baseline so every drive has a score even when perfectly healthy. + # Keyed on temperature because it is the one metric all 7 report. + - record: disk:smart_severity + expr: 0 * max by (device) (smartctl_device_temperature) + labels: + check: baseline + + # Sectors the drive could not read and has not yet remapped. This is + # the attribute that actually caught the 2026-08-13 failure. + - record: disk:smart_severity + expr: 2 * (max by (device) (smartctl_device_attribute{attribute_name="Current_Pending_Sector",attribute_value_type="raw"}) > bool 0) + labels: + check: pending_sectors + + - record: disk:smart_severity + expr: 2 * (max by (device) (smartctl_device_attribute{attribute_name="Offline_Uncorrectable",attribute_value_type="raw"}) > bool 0) + labels: + check: offline_uncorrectable + + # Warn on any remap, critical past 10. A handful of reallocations is + # normal ageing; a climbing count is not. + - record: disk:smart_severity + expr: (max by (device) (smartctl_device_attribute{attribute_name="Reallocated_Sector_Ct",attribute_value_type="raw"}) > bool 0) + (max by (device) (smartctl_device_attribute{attribute_name="Reallocated_Sector_Ct",attribute_value_type="raw"}) > bool 10) + labels: + check: reallocated_sectors + + - record: disk:smart_severity + expr: (max by (device) (smartctl_device_attribute{attribute_name="Reported_Uncorrect",attribute_value_type="raw"}) > bool 0) + (max by (device) (smartctl_device_attribute{attribute_name="Reported_Uncorrect",attribute_value_type="raw"}) > bool 10) + labels: + check: reported_uncorrect + + # WD Red Pro spec max is 65C; warn well before that. The 10TB WDs + # idle at 47-50C in this chassis, so 55 is the first real signal. + - record: disk:smart_severity + expr: (max by (device) (smartctl_device_temperature) >= bool 55) + (max by (device) (smartctl_device_temperature) >= bool 60) + labels: + check: temperature + + # Last resort only -- see the header note on why this is not trusted. + - record: disk:smart_severity + expr: 2 * (max by (device) (smartctl_device_smart_status) < bool 1) + labels: + check: smart_status + + # Single number per drive for the Grafana health indicator. + - record: disk:smart_health_score + expr: max by (device) (disk:smart_severity) + + - name: disk-smart-alerts + rules: + - alert: DiskUncorrectableSectors + expr: max by (device) (smartctl_device_attribute{attribute_name="Current_Pending_Sector",attribute_value_type="raw"}) > 0 + for: 15m + labels: + severity: critical + component: disk + annotations: + summary: "{{ $labels.device }} has {{ $value }} unreadable sectors" + description: "Current_Pending_Sector is above zero, meaning the drive cannot read those sectors and has not remapped them. THIS IS THE SIGNAL THAT WAS MISSING ON 2026-08-13 -- the same condition on /dev/sdg corrupted two Longhorn volumes. Every volume here is single-replica, so treat this as data loss in progress. Evacuate the Longhorn disk (set allowScheduling false + evictionRequested true on nodes.longhorn.io/dooplex), verify backups, and plan replacement. SMART overall-health will still say PASSED; ignore it." + + - alert: DiskOfflineUncorrectable + expr: max by (device) (smartctl_device_attribute{attribute_name="Offline_Uncorrectable",attribute_value_type="raw"}) > 0 + for: 15m + labels: + severity: critical + component: disk + annotations: + summary: "{{ $labels.device }} reports {{ $value }} offline-uncorrectable sectors" + description: "The drive's offline scan found sectors it cannot recover. Usually tracks Current_Pending_Sector. Same response: evacuate and replace." + + - alert: DiskReportedUncorrectGrowing + expr: increase(smartctl_device_attribute{attribute_name="Reported_Uncorrect",attribute_value_type="raw"}[6h]) > 0 + for: 30m + labels: + severity: warning + component: disk + annotations: + summary: "{{ $labels.device }} uncorrectable error count is climbing" + description: "Reported_Uncorrect grew in the last 6h, so the drive is actively failing reads rather than sitting on old damage. On 2026-08-13 this counter moved 633 -> 665 within a single session while the disk was being evacuated. A rising count means act now, not at the next maintenance window." + + - alert: DiskReallocatedSectorsGrowing + expr: increase(smartctl_device_attribute{attribute_name="Reallocated_Sector_Ct",attribute_value_type="raw"}[24h]) > 0 + for: 1h + labels: + severity: warning + component: disk + annotations: + summary: "{{ $labels.device }} is remapping sectors" + description: "Reallocated_Sector_Ct increased in the last 24h. The drive is still covering for itself, but a count that moves is a drive on the way out. Note the 2026-08-13 failure had Reallocated_Sector_Ct stuck at 0 while 120 sectors sat pending -- that model cannot remap, so a flat zero here is not reassurance on Seagate ST3000VX010 units." + + - alert: DiskOverheating + expr: max by (device) (smartctl_device_temperature) >= 60 + for: 20m + labels: + severity: critical + component: disk + annotations: + summary: "{{ $labels.device }} at {{ $value }}C" + description: "Sustained 60C or above. Check chassis fans (Fan RPM panel on the Grafana overview dashboard) and airflow before the drive throttles or fails." + + - alert: DiskOverheatingWarning + expr: max by (device) (smartctl_device_temperature) >= 55 + for: 1h + labels: + severity: warning + component: disk + annotations: + summary: "{{ $labels.device }} running warm at {{ $value }}C" + description: "Above 55C for an hour. Baseline in this chassis is 33-50C, so this is a real change. Not urgent, but check airflow." + + - alert: DiskSmartStatusFailed + expr: max by (device) (smartctl_device_smart_status) < 1 + for: 5m + labels: + severity: critical + component: disk + annotations: + summary: "{{ $labels.device }} SMART self-assessment FAILED" + description: "The drive itself has given up -- this flag stays PASSED through severe damage, so a FAILED reading means the drive is at the very end. Replace immediately; do not schedule around it." + + # Change detector, deliberately NOT a fixed count. An earlier version + # of this alert compared against a hardcoded 7, which had to be + # hand-edited on every disk change, went stale silently if anyone + # forgot, and could not notice a disk being ADDED. This form needs no + # maintenance and names the specific drive that went away. + # + # max_over_time rather than `offset`: a scrape gap (exporter restart, + # node reboot) must not erase the fact that a disk was there. Fires + # for 6h after the disappearance, then self-resolves -- it is a + # notification that the inventory changed, not a standing condition. + - alert: DiskDisappeared + expr: max by (device) (max_over_time(smartctl_device_smart_status[6h])) unless max by (device) (smartctl_device_smart_status) + for: 10m + labels: + severity: critical + component: disk + annotations: + summary: "{{ $labels.device }} vanished from smartctl-exporter" + description: "This drive reported SMART data within the last 6h and now reports none. Either it stopped responding (dying controller, dropped SATA link, powered-off drive) or it was physically removed. If the removal was deliberate, also drop its --smartctl.device entry from admin-system/smartctl-exporter.yaml, otherwise the exporter keeps retrying a device that is not there. If it was NOT deliberate, treat it as a failed drive: check dmesg for ATA link resets and confirm which Longhorn volumes had replicas on it." + + # Catch-all for the other direction -- a disk being added, or any + # count movement the per-device rule above does not express. + - alert: DiskInventoryChanged + expr: changes(smartctl_devices[6h]) > 0 + for: 10m + labels: + severity: warning + component: disk + annotations: + summary: "Number of disks visible to smartctl-exporter changed ({{ $value }} change(s) in 6h)" + description: "The count of monitored physical disks moved. Expected after adding or replacing a drive -- in that case update the --smartctl.device list in admin-system/smartctl-exporter.yaml so the new disk is actually monitored, since the list is explicit and a new drive is NOT picked up automatically. Unexpected movement means a drive is dropping in and out, which usually precedes failure." + + - alert: SmartctlExporterDown + expr: up{service="smartctl-exporter"} == 0 + for: 10m + labels: + severity: warning + component: disk + annotations: + summary: "smartctl-exporter is not being scraped" + description: "No disk health data is being collected, so every alert in this group is silently inert. The exporter is a DaemonSet in admin-system discovered via the prometheus.io/scrape annotation on its Service (the 'kubernetes-services' job). Check kubectl -n admin-system get pods -l app=smartctl-exporter." + + # =========================================================================== + # Workload health -- the gap that let external-dns-pihole crashloop for 27h + # with 326 restarts and nobody notice (2026-08-23). + # + # Until now every alert here watched a *subsystem* (authentik, postgres, + # longhorn, WAN, disks) via that subsystem's own exporter. Nothing watched + # Kubernetes itself, so a workload with no dedicated exporter could die + # permanently in silence. These rules close that by keying on + # kube-state-metrics, which reports on every pod in the cluster with no + # per-app wiring. + # + # DEPENDENCY, and it is a real one: kube-state-metrics runs live in mon-system + # but exists in NO manifest in this repo (see CLAUDE.md "Known traps"). It + # survives only because `monitoring` is manual-sync and does not prune. If it + # is ever removed, these three alerts go silently inert -- which is exactly + # the failure mode they exist to prevent. KubeStateMetricsDown below is the + # guard: it fires from `up`, not from kube-state-metrics' own output, so it + # still works when the exporter is the thing that is gone. + # =========================================================================== + workload-alerts.yml: | + groups: + - name: workload-health + rules: + # The direct answer to "a pod is crashlooping". kubelet sets this + # waiting reason once it starts backing off restarts, so it means + # "has already failed repeatedly", not "failed once". + # + # for: 15m rather than something tight: a container that crashes two + # or three times while waiting for its database is normal here and + # recovers on its own. 15m is still ~100x faster than the 27h this + # alert was written for. + - alert: PodCrashLooping + expr: max by (namespace, pod, container) (kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff",namespace!="jarr-nightly-system"}) > 0 + for: 15m + labels: + severity: critical + component: workload + annotations: + summary: "{{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} is crashlooping" + description: "The container has been in CrashLoopBackOff for over 15m, so it is failing on startup and kubelet has given up restarting it quickly. Get the reason from the PREVIOUS container, not the current one, which is usually still waiting: kubectl -n {{ $labels.namespace }} logs {{ $labels.pod }} -c {{ $labels.container }} --previous. A frequent cause here is Renovate bumping an image to a version that removed a flag the manifest still passes -- the log line is then a one-liner like \"unknown long flag\". Check whether an auto-syncing Application deployed a new tag just before the restarts began." + + # Catches the same class of silent breakage that never reaches + # CrashLoopBackOff, because the container never starts at all: a bad + # image tag, a missing secret key, an unparseable config. Renovate can + # produce every one of these, and none of them show up above. + - alert: ContainerStuckWaiting + expr: max by (namespace, pod, container, reason) (kube_pod_container_status_waiting_reason{reason=~"ImagePullBackOff|ErrImagePull|InvalidImageName|CreateContainerConfigError|CreateContainerError",namespace!="jarr-nightly-system"}) > 0 + for: 15m + labels: + severity: critical + component: workload + annotations: + summary: "{{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} stuck: {{ $labels.reason }}" + description: "The container cannot start at all, so it never reaches CrashLoopBackOff and PodCrashLooping will not fire for it. ImagePullBackOff/ErrImagePull/InvalidImageName mean the tag does not exist -- suspect a bad Renovate bump or a typo in the manifest. CreateContainerConfigError almost always means a referenced secret or configMap key is missing, which in this repo means a secret that was never created imperatively (secrets are not in git). kubectl -n {{ $labels.namespace }} describe pod {{ $labels.pod }} names the exact object." + + # Crash-and-recover loops that never sit in CrashLoopBackOff long + # enough for the first alert: OOMKills under bursty load, a liveness + # probe that is slightly too aggressive. increase() is reset-aware, so + # a pod being recreated does not produce a false spike. + - alert: PodRestartingFrequently + expr: increase(kube_pod_container_status_restarts_total{namespace!="jarr-nightly-system"}[1h]) > 4 + for: 10m + labels: + severity: warning + component: workload + annotations: + summary: "{{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} restarted {{ $value | printf \"%.0f\" }}x in 1h" + description: "The container keeps dying and coming back without staying down long enough to be flagged as crashlooping. Check the exit code: kubectl -n {{ $labels.namespace }} describe pod {{ $labels.pod }} | grep -A3 'Last State'. Exit 137 with Reason OOMKilled means the memory limit is too low for this workload's peaks -- raise limits.memory in the manifest rather than removing the limit. Otherwise suspect a liveness probe whose timeout is shorter than the app's worst-case response." + + # Guard for the three rules above. Deliberately keyed on `up`, so it + # reports the exporter being gone even though every other rule in this + # group would be evaluating to "no data" and therefore silent. + - alert: KubeStateMetricsDown + expr: absent(up{job=~".*kube-state-metrics.*"} == 1) or max(up{job=~".*kube-state-metrics.*"}) == 0 + for: 15m + labels: + severity: critical + component: workload + annotations: + summary: "kube-state-metrics is not being scraped -- all pod-health alerting is inert" + description: "Nothing is watching pod state cluster-wide: PodCrashLooping, ContainerStuckWaiting and PodRestartingFrequently cannot fire while this is true. kube-state-metrics runs in mon-system but is NOT defined in any manifest in this repo, so it cannot be resynced from git -- check kubectl -n mon-system get pods -l app.kubernetes.io/name=kube-state-metrics and, if it is gone, it has to be recreated by hand. Bringing it into the repo is HM-007." + + # ======================================================================= + # HM-040: a cryptominer ran in the Plex pod (CPU above 4 cores from + # 2026-10-17 15:10Z to 10-23 08:15Z, ~20 of 40 cores) and nothing fired: + # Plex had no CPU limit and no rule looked at container CPU at all. Both + # CPU rules use cAdvisor series (job kubernetes-cadvisor), NOT + # kube-state-metrics, so they keep working when KSM is gone (HM-007). + # + # Measured 2026-10-23 over the 14 d of retention: the ONLY container whose + # 10-minute average ever exceeded 4 cores was plex (the miner); the next + # highest peaks were ~1.5 cores (longhorn instance-manager, gitea). Plex's + # own pre-incident peak (09-23 -> 10-17) was 3.02 cores. + # + # Two tiers (operator decision 2026-10-23): + # 10 cores, critical, 15m, no allowlist -- nothing legitimate is near it. + # A container capped below 10 cores (Plex is now 6, HM-040/HM-041) + # cannot reach it; the 4-core warning covers those. + # 4 cores, warning, 30m, immich ML excepted (container `main`, matched on + # the pod name) -- measured never above 4 cores, kept for model warm-ups. + # No Plex evening window: it would have hidden this miner up to 7 h a day. + # + # No process-count rule, on purpose. KSM has no process metric; cAdvisor's + # container_processes did not separate this incident: Plex ran 7 (p50) / + # 8 (max) processes before 10-17 and 9 (p50, also the minimum) during it. + # Replayed over 14 d, a 2x-own-baseline rule was true for 9 five-minute + # steps; container_threads overlapped the baseline (86/187 vs 90/193). + # The miner is ONE process with 46 threads. CPU is the signal. + # ======================================================================= + - name: workload-abuse + rules: + - alert: ContainerCPUAbove10Cores + expr: | + sum by (namespace, pod, container) ( + rate(container_cpu_usage_seconds_total{container!="",container!="POD"}[5m]) + ) > 10 + for: 15m + labels: + severity: critical + component: workload + annotations: + summary: "{{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} is using {{ $value | printf \"%.1f\" }} cores" + description: "A container has averaged more than 10 of the node's 40 cores for 15 minutes. Nothing on this cluster has ever done that legitimately (measured 2026-10-23: the 14-day max outside the HM-040 miner was ~1.5 cores). Treat it as a compromise until shown otherwise: list the container's processes and where they run from -- for p in $(sudo grep -l /proc/[0-9]*/cgroup | cut -d/ -f3); do sudo readlink /proc/$p/exe; done -- anything under /tmp, /var/tmp, /dev/shm or '(deleted)' is the HM-040 shape. Capture before you kill (INCIDENT.md of HM-040)." + + - alert: ContainerCPUAbove4Cores + expr: | + sum by (namespace, pod, container) ( + rate(container_cpu_usage_seconds_total{container!="",container!="POD",pod!~"immich-machine-learning-.*"}[5m]) + ) > 4 + for: 30m + labels: + severity: warning + component: workload + annotations: + summary: "{{ $labels.namespace }}/{{ $labels.pod }} container {{ $labels.container }} has used {{ $value | printf \"%.1f\" }} cores for 30m" + description: "A container has averaged more than 4 cores for 30 minutes. Measured 2026-10-23, no container except the HM-040 miner had ever done that (next highest ~1.5 cores; Plex's own peak 3.02). Check where its processes run from before calling it load: anything under /tmp, /var/tmp, /dev/shm or '(deleted)' is the HM-040 shape. If it is genuine load, raise the threshold for that workload here rather than ignoring the alert." + diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 4d589279..c515ffb0 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -34,6 +34,7 @@ | **R-1245** | **DECIDE: how large is qBittorrent's default upload limit?** (P3) | CLOSED 2026-10-24 — **D41 = 50 %** (operator ruling 07:22), of the host's TESTED line (the weekly iperf3 test, R-1247 — never the push floor); built in controller 0.342.0 (`ed1011b`); released with R-1244 | `08` §9.5 | | **R-1246** | **DECIDE: who is told when the box limits or flags something?** (P3) | CLOSED 2026-10-24 — **D42** (operator ruling 07:22): the family only about the qBittorrent limit (one line on its page); `app_unexplained_upload` / `app_unexplained_cpu` are operator-only (hub 0.169.0); released with R-1244 | `08` §9.4 | | **R-1244** | **The upload and CPU watch was built and not released.** (P2) | CLOSED 2026-10-24 — released on the operator's yes: hub **0.169.0** (the four operator-only events + the System page's „Upload line"; sha256:fb9b2489…, deployed 07:51Z, a 30-minute watch with no alarm) and controller **0.342.0** (sha256:d002d3b7…, `42cda07`; per-box floors → Tester 1, demo-hp, demo-felhom healthy on it, box checks + https-only green; then the global floor 0.342.0, MinAgent 0.158.1, 08:29:19Z, read back on the hub log and page). demo-felhom's qBittorrent got 50 % of its tested line (5 789 696 B/s ≈ 46 Mbit/s; the log, qBittorrent's own config and the box's record agree). Open beside it: R-1250 (a seeding qBittorrent is never tested), R-1248 (an app's API can lie) | `audits/line-test-2026-10-24/controller-0342-boxes.txt` | +| **R-934** | **Pull the failed disk ST3000VX010 S/N Z6A07P2G out of DooPlex.** (P3) | CLOSED 2026-10-24 — pulled by the operator in the 2026-10-23 maintenance window (homelab HM-043, `1bb4c4d`): `DiskDisappeared` and `DiskInventoryChanged` fired at the pull and cleared (18:02Z, 20:17Z); no `smartctl_*` series names the serial any more and no manifest lists it (comments only). The silence `device=~.*_Z6A07P2G` (until 2026-11-11) now matches nothing and lapses by itself — on the homelab list to expire early. The twin Z6A070XZ (`/mnt/3_hdd`) stays. Full text: `git show 6018486f:documentation/backlog/OPEN-ITEMS.md`. | `audits/dooplex-alarms-2026-10-24/part3-alarms.txt` (the alarm history and the silence) | | **R-1247** | **The box's upload line was only a lower bound, learnt from its own off-site push.** (P4) | CLOSED 2026-10-24 — the weekly iperf3 line test (operator ruling 07:26): ep0's tunnel-only `felhom-iperf3` (no answer on the internet, `:22` control open), agent **0.164.0** (`internal/linetest`, its own iperf3 client; vouched by the operator) and the controller reading it. The three boxes' first tests: Tester 1 **94.3**, demo-hp **94.4**, demo-felhom **92.6** Mbit/s (best of 3, 10.05 s per run; demo-felhom met „server busy" twice and tried again the same day); the push floor said „at least 10" | `audits/line-test-2026-10-24/first-tests-three-boxes.txt`, `ep0-install.txt` | ## 2026-10-23 day (later) — D39 = B: the security-fix lane diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index bca33a4b..4d3ca7d6 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -157,7 +157,7 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | NARROWED — **2026-10-23 (operator yes D33, D34):** the hub DB now leaves every 6 hours (hub 0.167.0 + `scripts/hub-db-backup/`, alarm at 13 h; restored back byte-identical) and the released images and agent packages leave nightly (`scripts/release-copy/`, R-1227 closed). (c)–(g) unchanged. **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest **2026-10-23 night — survey again, read only (`audits/dooplex-survival-2026-10-22/PLAN.md`):** both nightly off-site jobs and both weekly restore tests green (copies 2026-10-22, tests 2026-10-18); the local tree is still the same box (2026-10-22 03:16 ok). New in the survey: the hub DB copy is up to 24 h old (G5: every 6 h — the operator's yes; a DooPlex change). | — | — | operator | +| **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | NARROWED — **2026-10-23 (operator yes D33, D34):** the hub DB now leaves every 6 hours (hub 0.167.0 + `scripts/hub-db-backup/`, alarm at 13 h; restored back byte-identical) and the released images and agent packages leave nightly (`scripts/release-copy/`, R-1227 closed). (c)–(g) unchanged. **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest **2026-10-24 (attended): `DooplexGiteaOffsiteStale` fired from 00:57Z — not this path's fault.** The 00:20 push refused, as designed, a 12 h old Gitea dump: the PostgreSQL dump CronJob had failed every run since 2026-10-23 16:00Z (`28P01`) because the window reboot restarted the database and CNPG put the bootstrap secret's password in force (homelab `531023d`, operator's pick: the CronJob only). Fixed; dump, push and restore test run by hand (26 456 files, 12 repos `git fsck`, `gitea.dump` 1114 entries); cleared 10:51Z (`audits/dooplex-alarms-2026-10-24/`). Backup alarms now arrive as ONE mail every 4 h (homelab `22d7697`). **2026-10-23 night — survey again, read only (`audits/dooplex-survival-2026-10-22/PLAN.md`):** both nightly off-site jobs and both weekly restore tests green (copies 2026-10-22, tests 2026-10-18); the local tree is still the same box (2026-10-22 03:16 ok). New in the survey: the hub DB copy is up to 24 h old (G5: every 6 h — the operator's yes; a DooPlex change). | — | — | operator | | **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):** `_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). **FIRST VERDICT READ 2026-10-21 (rehearsal session, read-only):** the 07:10 CEST run judged three tiers against yesterday — demo-felhom 5, demo-hp 14, tester-1 4 new snapshot files, „each inside a reported box run"; nothing vanished, nothing unexplained; Tester4 stored its first baseline (44 files); Tester-2's own listing failed (no `snapshots/` dir — the box has never pushed). No `offsite_snapshots_vanished` / `_unexplained` mail in the operator mailbox (second channel). Evidence `audits/due-checks-2026-10-21/R-895-hub-0710.txt`. The positive alarm path is still proven only by the unit red-proofs. | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC | | **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | NARROWED — **2026-10-23 (D32):** the sheet is current — names and places only, one A4, HU + EN (`runbooks/break-glass-sheet.md`); every place checked to exist. LEFT: the operator prints it, fills it by hand, keeps it outside the home. **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G1/G3/G10):** unchanged; the sheet is still the one act that breaks the circle, and S5's offline form is unknown to CC — on the sheet too (decide list in STATUS). | operator: print the sheet | Operator: print + store the sheet; then close | operator | | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC | @@ -269,7 +269,7 @@ stopping line that lies. |---|---|---|---|---|---|---|---| | **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC | | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC | -| **R-1250** | Monitoring & notifications | P3 | **A qBittorrent that seeds all day is never line-tested, so it never gets the limit that exists for it.** The weekly test (R-1247) starts only after 10 minutes in which the host's uplink sent < 2 Mbit/s each minute; a seed at the line's speed keeps it busy through every window, so no test, no tested line, and no qBittorrent limit (D41 uses the tested line only). Found 2026-10-24 by reading the rule against the case it serves; not yet seen on a box (no test box seeds). | OPEN | — | Design: on a box with qBittorrent and no tested line, the controller asks qBittorrent to pause uploads (its alternative speed limit) for the test window the agent announces, then restores it; or the watch reports „line unknown, a seeding qBittorrent" to the operator. Pick, red-proof, prove on 9202 | CC | +| **R-1250** | Monitoring & notifications | P3 | **A qBittorrent that seeds all day is never line-tested, so it never gets the limit that exists for it.** The weekly test (R-1247) starts only after 10 minutes in which the host's uplink sent < 2 Mbit/s each minute; a seed at the line's speed keeps it busy through every window, so no test, no tested line, and no qBittorrent limit (D41 uses the tested line only). Found 2026-10-24 by reading the rule against the case it serves; not yet seen on a box (no test box seeds). | **NARROWED 2026-10-24 — FIXED on main, not released** (brief 2026-10-24 Part 4, commit only): agent `29bcf44`+`1e087bb` (`waiting_for_quiet`, `POST /host/line-test/run`), controller `046b808` (`internal/upwatch/linehold.go`: record first, hold at 1 KiB/s, test, the exact old limit back; the start-up check after a kill; the 50 % rule waits). Red-proofs in both CHANGELOGs. **Live on demo-hp with a real seed** (10 Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; held: uplink 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, back to 1 249 280 B/s exactly; a kill -9 mid-test left the record and the start-up check put the limit back (`audits/dooplex-alarms-2026-10-24/part4-r1250-live-proof.txt`). The live run used probes over plain HTTP (the pinned agent client is unit-tested, not live). | the next release train (agent ≥ 0.165.0 needs the operator's vouch) | Release agent + controller in the next normal train; on a seeding box read one Tuesday's `line test on request` line; then close | CC | | **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Design note (2026-10-21, after R-1192's diagnosis): the freeze's signature is cheap to see from the HOST side — the guest cgroup's memory PSI „full" above ~50 % for a minute, or its `memory.events max` climbing, while the host still reports; the agent could raise `guest_memory_thrash` (operator mail in its own words, not „no report for 45 m") and the household banner „A dobozod most nagyon lassú…". With the swap fix (R-1192) the measured thrash ends within a minute, so this alarm should be built only if a freeze is seen again after the agent release; until then it stays a design. Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC | | **R-1202** | Monitoring & notifications | P3 | **A box whose clock is behind blames the hub and stops an app without telling anyone.** Rehearsal 2026-10-20 B6 #8: RTC set 2 years back (the OS raised it to 2026-04-13): the dashboard shows only „Hub nem elérhető" while the log says „x509: certificate … not yet valid: current time 2026-04-13…"; Paperless crash-looped and the box stopped it with a restore hold; the `app_stopped_unhealthy` event failed (hub unreachable), so neither the household nor the operator heard. After the real time came back the app stayed stopped until a Start (clear sentence, 1 min). | **OPEN** | — | say „the box's clock is wrong (it says )" when TLS fails with not-yet-valid/expired against a hub that answers; keep the stop event queued until the hub is reachable | CC | | **R-340** | Monitoring & notifications | P3 | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice** **REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc//fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC | @@ -337,7 +337,6 @@ stopping line that lies. | **R-987** | Process & tooling | P4 | **The 53 catalog app images have never been scanned for known vulnerabilities.** Fix: a monthly trivy pass over the images the catalog pins, reported, not gating. | OPEN | — | Add to the monthly re-test | CC | | **R-733** | Process & tooling | P3 | **[P3-LOW] The test bench has NO swap and the boxes have 512 MiB — so a box proof can pass on swap where the bench fails, and nobody records whether a customer guest has swap.** MEASURED 2026-09-30 (R-732): immich's first start was OOM-killed 61–104 times on the bench (swap 0) and passed on 9202 by swapping ~108 MB; the bench given 512 MiB swap passed too. demo-hp 9201, 9202 and demo-felhom 9201 all read `swap: 512`; the golden's guest config is not recorded in its bake evidence, so a customer guest's swap is NOT measured. The harness's memory watch judges `anon` against the limit and never reads `memory.swap.current`. **Needs:** the golden's `swap` read and recorded; the box walk and the harness report `memory.swap.peak` beside `anon`; a decision whether proofs run with swap off (the stricter venue, as R-732's fix was proven). | **READY — rank P3-LOW; owner: CC (harness + golden evidence)** **2026-10-06 night: NARROWED** — the harness records each container's `swap_peak` and the venue's swap (catalog branch `night-held-2026-10-06`, `3f4611c`; SwapRecorded red-proved; bench SwapTotal 0 kB, 9202 524288 kB). LEFT: the golden's swap in its bake evidence, the box walk's record, and whether proofs run with swap off. | — | — | CC | | **R-902** | Process & tooling | P3 | **The contact form's mailer has no source in any repository: the running image cannot be rebuilt or changed.** FOUND 2026-10-08 (website refresh, brief Part C5). `manifests/contact-mailer.yaml` names `contact-mailer:latest` with `imagePullPolicy: Never` (imported into k3s by hand); the binary in the pod is `/app/contact-mailer` dated 2026-02-05, Go module `github.com/felhom/contact-mailer` (devel). Searched: `git log --all` of felhom.eu (only the manifest), every repo under `/mnt/5_hdd/felhom.eu/git`, `find /mnt/5_hdd/felhom.eu ~ -maxdepth 5 -iname '*contact-mailer*'`. READ from the binary (strings, read-only `kubectl exec … cat`): ONE send per form (`sendViaResend`) to `TO_EMAIL`, `reply_to` = the visitor; nothing is sent back to the visitor; the operator's notification is Hungarian (`buildEmailHTML`, „Csatolmány … fájl"). The English form needs no mailer change (its subject labels carry „(EN)"). `audits/website-refresh-2026-10-08/mailer.md` | **OPEN — NARROWED 2026-10-08 (search done, NOT FOUND): owner: operator.** A read-only search found no source on DooPlex (55,548 `*.go`/`go.mod`/`Dockerfile*` files, every disk, media/Longhorn/containerd/PBS excluded by name), in the web editor's storage and file history, in all 10 Gitea repositories' history, or in Docker's build cache (oldest record 2026-09-28). The binary's own clues: one file `/build/main.go`, go1.23.12, no `vcs.*` stamp; image built 2026-02-05 10:40 by BuildKit. The binary is now KEPT in git (`audits/mailer-source-2026-10-08/contact-mailer.bin`), so a DooPlex rebuild no longer loses the program. A replacement plan is written (`audits/mailer-source-2026-10-08/PLAN-replacement.md`); one question left: is the February folder on the operator's Windows workstation? | — | Operator: look on the workstation; if absent, build the replacement from the plan (attended deploy) | operator | -| **R-934** | Process & tooling | P3 | **Pull the failed disk ST3000VX010 S/N Z6A07P2G out of DooPlex; follow the checklist.** The data was moved off at the first alarm (2026-10-10); the disk is unmounted, its fstab line commented out since 2026-08-14, k3s no longer waits for it, and its empty Longhorn disk entry was removed 2026-10-12. Its two SMART alarms (352 unreadable / offline-uncorrectable sectors, flat since 2026-10-09) are silenced by serial until **2026-11-11 14:32 UTC** — if the disk is still in the machine then, they come back on purpose. **A TWIN of the same model (S/N Z6A070XZ, `/mnt/3_hdd`) is in use: read the serial before unplugging.** Checklist: `runbooks/dooplex-remove-disk-Z6A07P2G.md`. | **OPEN** — NEW 2026-10-12 | the operator (hands, a power-off window) | Follow the checklist; after the first boot remove the exporter line and the silence; tell CC | operator | | **R-935** | Process & tooling | P3 | **Something sets `core.bare = true` on the shared `felhom.eu` checkout, and then every git command there fails („this operation must be run in a work tree").** SEEN 2026-10-12 evening twice: `.git/config` changed at 16:29:37 and 16:37:33 CEST, each time with `core.bare=true`; restored by hand both times (`git config core.bare false`, a flag only, no file touched). **Not caused by the gates:** `repo_gates.py --fast` run plainly and with `GIT_DIR=.git` (as a hook gets it) left the flag `false`; a scratch `git init --bare ` under `GIT_DIR` does not flip the parent either. The first flip came while this session did not touch the repo; another session pushed to main at 16:44 from elsewhere. | **OPEN** — 2026-10-12 later: across this session's next push the flag stayed `false` (read before and after); not this session's pushes. **OPEN** — NEW 2026-10-12 **Note 2026-10-12 (security review):** at least some of these flips match the security review's pushes from a git worktree of this clone (about 16:29 and 16:37): inside the pre-push hook, `scripts/test_gate_decoys.py`'s poster-facts case runs `git init`/`git config`/`git commit` with the hook's inherited `GIT_DIR`, so it wrote into the real repository — four test commits in the worktree and `user.name d` / `user.email d@d` in the shared `.git/config` (the review removed that `[user]` section at about 16:50; commits 6c2c3dd5 and 186f3aa3 carry that author). The review stopped pushing from the worktree. Fix: the test helper must drop `GIT_DIR`, `GIT_WORK_TREE` and `GIT_INDEX_FILE` from its environment. | finding the writer | Watch `.git/config` (an inotify or audit rule on the file) during a day with the parallel sessions; name the command; fix it there | CC | | **R-928** | Process & tooling | P4 | **Two sessions running `repo_gates.py` at the same time in the shared `felhom.eu` clone see each other's DECOY plants.** SEEN 2026-10-10 ~15:40: while another session's gate run was in its decoy suite, this session's working tree showed `website/assets/crafty-controller-logo.png` renamed to `crafty-logo.png`, `website/en/apps.html` with a leftover text language switch, then `website/en/contact.html` and `scripts/felhom-host-install.sh` modified — each a plant of `test_gate_decoys.py`, each gone when that run ended. This session's own `site_gates.py` failed three times in that window on plants, not on its change, and a `git add` of a broad path at that moment would have committed a plant. A killed run (`timeout`) leaves its plant in place (its cleanup never runs). Fix direction (a design): decoys plant into a temporary COPY of the tree (as the catalog's volume-persistence decoys already do), or a lock file that a second run waits on. | **OPEN** — filed 2026-10-10 (register-shrink session) | — | Design the copy-or-lock; meanwhile stage explicit paths only (already the rule) and re-run a red site gate once another session's run has ended | CC | | **R-394** | Process & tooling | P4 | **`felhom-build-deploy/SKILL.md` is 179 lines, over the 150-line limit its own repo now enforces.** Found 2026-08-25 by `scripts/check_skills.py` on its first run — the over-length was discovered BY the new checker, on the day the limit was written down, which is the checker working as intended. **It is not edited and not trimmed here:** the task that introduced the limit explicitly scoped the four pre-existing skills out, and trimming a build-and-deploy skill without exercising its commands is how a wrong command ships to a live host. **It is a named single-entry exception in `GRANDFATHERED` in `scripts/check_skills.py`, printed as a WARN on every run**, so it cannot fade; a NEW skill over the limit is convicted normally, and growing the set requires editing that file in a commit with a row to name. **The rationale for the limit** — attention thins across the excess, so the lines that matter are not the ones that survive — is in `skills/felhom-doc-authoring/SKILL.md` §5. | **OPEN — LOW** | — | Trim `felhom-build-deploy/SKILL.md` under 150 lines in a session that can VERIFY the commands it keeps, then delete its `GRANDFATHERED` entry in the same commit. The likely trim is the per-artifact command blocks moving behind a pointer to the runbooks, keeping the gotchas inline — but that is a judgement for a session with a build to run, not a line-count exercise. | CC | @@ -370,6 +369,7 @@ stopping line that lies. the R-row. Duplicating them here would create the second source this design avoids. --> | item | due (UTC) | what to measure | |---|---|---| +| R-232 | 2026-10-25 | the 03:14 local restic sweep (`dooplex-backup`, `pgdump` repo) took a snapshot of `/mnt/5_hdd/backup/postgresql/dumps` that holds a folder with `SUCCESS` from 2026-10-24 (the CronJob's 00:00/06:00 local runs on `postgres-superuser`), and `PostgresqlDumpStale` / `DooplexGiteaOffsiteStale` did not fire overnight | | R-1243 | 2026-10-25 | no new Longhorn backup of the hub volume pvc-486c9809 after the 02:00Z daily job while the other volumes got theirs; detail in the R-1243 row | | R-435 | 2026-10-27 | Tester 1's weekly off-site clean-up ran inside a hub window with no drop alarm (box log + mailbox); detail in the R-435 row | | R-1183 | 2026-10-26 | every hub daily job ran ONCE on 2026-10-25 (hub log + one snapshot file per slot — four slots since hub 0.167.0, D34, the 02:00 one not twice); detail in the R-1183 row | diff --git a/documentation/backlog/STATUS-HISTORY.md b/documentation/backlog/STATUS-HISTORY.md index be6e378f..d46acab6 100644 --- a/documentation/backlog/STATUS-HISTORY.md +++ b/documentation/backlog/STATUS-HISTORY.md @@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.** +## Moved from STATUS on 2026-10-24 (afternoon) + +**Updated 2026-10-24 (day, attended, done).** Released on your yeses: hub 0.168.0 (tokens locked) and 0.169.0, agent 0.164.0 (vouched), controller 0.342.0 (the floor). Each box tests its upload weekly (94, 94, 93 Mbit/s); a hacked app that uploads or computes more than it explains alarms only you, and qBittorrent gets half the tested line. The hub is out of DooPlex's open backup folder. + ## Moved from STATUS on 2026-10-24 (day, done) **Updated 2026-10-24 (day, attended).** Hub 0.168.0 is live: the last plain tokens are locked, and every copy of the hub database that leaves DooPlex is clean. The hub is out of DooPlex's open backup folder. Each box can now test its upload once a week (demo-hp: 90.7 Mbit/s, not 10); the releases follow on your yes.