answers 2026-10-12 Part B: the four DooPlex changes live and read back; R-886 + R-211 closed (112 -> 110)

- S21 / R-887: runner fetch_timeout 30s, Gitea Ingress 2 rps per address (homelab-manifests ad8618f); WATCHING to 2026-10-19 - S22 / R-886: Alertmanager fsGroup 65534; a test silence survived a pod restart (4168451) - closed - S23 / R-211: Prometheus config-reloader; two automatic reloads, no manual POST (ab40c7b, test e70e0f1/78d4be4) - closed - S20 / R-231: backup scripts in homelab-ansible/roles/dooplex_backup, installed from git (fdd324b); VERIFY 2026-10-13 - evidence: documentation/audits/answers-2026-10-12/s2{0,1,2,3}-*.txt Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-12 14:33:07 +02:00
parent ee43929040
commit e2b09f25ea
7 changed files with 58 additions and 9 deletions
@@ -0,0 +1,10 @@
+# S20 / R-231 — DooPlex change 4, installed 2026-10-12 ~12:32Z with the operator in the session (yes in chat), homelab-manifests fdd324b
+live before: /opt/backup/scripts = 11 scripts + 4 .bak-<date> copies; units /etc/systemd/system/dooplex-backup.{service,timer} mode 744.
+copied: the 11 scripts (cmp: byte-identical) + the 2 units (cmp: byte-identical) into homelab-ansible/roles/dooplex_backup/files/; .bak copies left out.
+secret check: every file grepped for password/token/key/BEGIN/AKIA/credential URLs; only PATHS found (/etc/backup/restic-password,
+  /etc/backup/gitea-token, k3s token paths). No value in any file.
+dry run (--check --diff): scripts ok (identical), units "changed" (mode 744 -> 0644 only).
+install (install.sh --apply): ok=6 changed=1 failed=0; units now 644; timer enabled, next 2026-10-13 03:10:41 CEST; scripts cmp-identical to the repo.
+baseline: dooplex-backup.service last run Result=success, ExecMainStatus=0, exit 2026-10-12 03:16:17 CEST.
+to read 2026-10-13: the same three fields after the 03:10 run (DUE-CHECK R-231).
+undo: the role only rewrote files with identical content; git revert fdd324b removes it from the repo.
@@ -0,0 +1,9 @@
+# S21 / R-887 — DooPlex change 1, applied 2026-10-12 12:17Z (operator yes in chat), homelab-manifests ad8618f
+before (24 h Ingress log to 12:10Z): 21 505 gitea requests; 216.73.216.6 (crawler UA "compatible ...") 20 546, max 475/min;
+  37.191.56.193 (boxes' git + registry) 811, max 6/s, 25/min. Crawler paths: /src/commit 11 299, /commits/commit 4 420, /raw/commit 2 807.
+ArgoCD app gitea: OutOfSync on exactly ConfigMap/act-runner-config, Deployment/act-runner, Ingress/gitea -> sync Succeeded Synced Healthy.
+runner read-back: process "act_runner daemon --config /config/config.yaml"; /config/config.yaml fetch_timeout: 30s;
+  /data/.runner labels ['felhom-gates:host'] (unchanged); log "declare successfully".
+nginx read-back: limit_req_zone ... rate=2r/s; limit_req ... burst=10 nodelay (2 locations: gitea.dooplex.hu, gitea.home)
+first minute after: 216.73.216.6 14x200 + 7x503; 37.191.56.193 12x200, 0x503.
+undo: git revert ad8618f + sync app gitea.
@@ -0,0 +1,10 @@
+# S22 / R-886 — DooPlex change 2, applied 2026-10-12 ~12:19Z (operator yes in chat), homelab-manifests 4168451
+before: /alertmanager root:root 0755; log every 15 min "Running maintenance failed ... open /alertmanager/silences.*: permission denied" (and nflog); 0 silences.
+ArgoCD app monitoring: OutOfSync on exactly Deployment/alertmanager -> resource-only sync Succeeded.
+after: /alertmanager 0:65534 drwxrwsr-x (fsGroup applied).
+test: POST silence 52fb455b (matcher alertname=CC_R886_TEST_NO_SUCH_ALERT, matches nothing, 30 min);
+  delete pod alertmanager-9b557d68-98z2l -> new pod alertmanager-9b557d68-28rrq;
+  /alertmanager now holds nflog + silences (65534, 12:20); GET /api/v2/silences returns 52fb455b state active. Survived.
+cleanup: amtool silence expire 52fb455b -> state expired. No other silence existed.
+note: the restart re-sends firing alerts; two were active (LonghornVolumeSpaceWarning pvc-486c9809 91.2 %; SMART pending sectors on ata-ST3000VX010-2E3166_Z6A07P2G, 352, firing since 2026-10-10 08:21Z).
+undo: git revert 4168451 + sync Deployment/alertmanager.
@@ -0,0 +1,13 @@
+# S23 / R-211 — DooPlex change 3, applied 2026-10-12 12:21Z (operator yes in chat), homelab-manifests ab40c7b
+ArgoCD app monitoring: OutOfSync on exactly Deployment/prometheus -> resource-only sync Succeeded; pod prometheus-6f548d5cdd-l926j, containers config-reloader=ready prometheus=ready.
+reloader log: Watching directory "/etc/prometheus" and "/etc/prometheus/rules".
+test (no manual POST /-/reload anywhere):
+  e70e0f1 one comment changed in authentik-alerts.yml; ConfigMap-only sync 12:21:48Z ->
+    12:22:28 reloader "config map updated ... successfully triggered reload"; prometheus 12:22:28.506 "Loading configuration file", 12:22:28.556 "Completed loading"; file in pod holds the test text (grep -c = 1).
+  78d4be4 the comment removed (the rules file equals ab40c7b again); ConfigMap-only sync 12:28:11Z ->
+    12:28:42 reloader "successfully triggered reload"; grep -c in pod = 0.
+end state: app monitoring Synced Healthy at 78d4be4; /api/v1/rules 12 groups, 56 rules, health {ok}.
+incident, mine: a local `git revert` + `--amend` produced a wrong local commit; its push was REFUSED (non-fast-forward) and the
+  ArgoCD sync to that unknown revision ended Error with nothing applied. Fixed with `git reset --soft origin/main` (other people's
+  uncommitted files untouched) and a hand revert (78d4be4).
+undo: git revert ab40c7b + sync Deployment/prometheus.
@@ -32,6 +32,8 @@ The full text of the rows below: `git show 25ab47eeaa:documentation/backlog/OPEN

 | Row | What | Closed | Evidence |
 |---|---|---|---|
+| **R-886** | **DooPlex's Alertmanager cannot write its own state since the Longhorn restart of 2026-10-05 13:20Z** (P3) | CLOSED 2026-10-12 — **fixed on DooPlex** (operator yes in chat; S22, `09` §3 decision 217): Alertmanager pod `fsGroup: 65534` (homelab-manifests `4168451`, resource-only sync). Volume now `0:65534 drwxrwsr-x`; a test silence survived a pod delete (old pod `…98z2l` → new `…28rrq`, `GET /api/v2/silences` = active), then expired by amtool; `nflog` and `silences` files now written. | `s22-alertmanager-silence.txt` |
+| **R-211** | **Prometheus has no config-reloader — a rules change reaches the pod and is never read** (P3) | CLOSED 2026-10-12 — **fixed on DooPlex** (operator yes in chat; S23, `09` §3 decision 218): `config-reloader` sidecar (configmap-reload v0.15.0, by digest; homelab-manifests `ab40c7b`). Proven twice with no manual reload: a rule-comment change (`e70e0f1`) and its removal (`78d4be4`) each reached Prometheus ~35–40 s after a ConfigMap-only sync („successfully triggered reload" + „Completed loading of configuration file"); 56 rules `ok`. | `s23-prometheus-reloader.txt` |
 | **R-243** | **A box in the R-241 state silently stops backing up off-site, and NO ALARM OF ANY KIND FIRES.** (P2) | CLOSED 2026-10-12 — **the alarm fired on time.** The `offsite_escrow_pending` mail for Tester-2 reached the operator mailbox 2026-10-11 16:13:51Z (the first sweep after the 7-day line, ~16:06Z); the running hub logged „Operator email sent for Tester-2/offsite_escrow_pending" at the same second. Two channels. | hub v0.144.0+ (`b119301c`); `r243-escrow-pending-mail.txt` |
 | **R-831** | **The Hetzner storage API token (`HETZNER_TOKEN`, the storage project's token in `Secret/storagebox`) was printed into the 2026-10-03 session transcript** (P3) | CLOSED 2026-10-12 — operator ruling 2026-10-12 (S2, `09` §3 decision 197): **A — accept.** The Hetzner storage token stays valid. | the ruling |
 | **R-870** | **Tester 1's two Cloudflare credentials — the zone API token (`infrastructure.cf_api_token`) and the tunnel token (`infrastructure.cf_tunnel_token`) of the hub's `customer_configs` row `tester-1` — were printed into the 2026-10-04 night session's transcript** (P3) | CLOSED 2026-10-12 — operator ruling 2026-10-12 (S3, `09` §3 decision 198): **accepted**; rotate when Tester 1 is retired. | the ruling |