answers 2026-10-12 Part B: the four DooPlex changes live and read back; R-886 + R-211 closed (112 -> 110)

- S21 / R-887: runner fetch_timeout 30s, Gitea Ingress 2 rps per address (homelab-manifests ad8618f); WATCHING to 2026-10-19
- S22 / R-886: Alertmanager fsGroup 65534; a test silence survived a pod restart (4168451) - closed
- S23 / R-211: Prometheus config-reloader; two automatic reloads, no manual POST (ab40c7b, test e70e0f1/78d4be4) - closed
- S20 / R-231: backup scripts in homelab-ansible/roles/dooplex_backup, installed from git (fdd324b); VERIFY 2026-10-13
- evidence: documentation/audits/answers-2026-10-12/s2{0,1,2,3}-*.txt

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

STATUS.md

@@ -3,16 +3,22 @@
 **Ready for the first real tester (Tester-2): yes. Tester 2 (a laptop) is off; nothing was sent to it.**
 
 **Updated 2026-10-11 (M3 live): hub 0.146.0; demo-hp, demo-felhom and Tester 1 run agent 0.154.0 and controller
-0.307.0. The open-items list is at 112 (2026-10-12: your 38 answers, 17 closed). Reports: `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
+0.307.0. The open-items list is at 110 (2026-10-12: your 38 answers, 19 closed). Reports: `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
 
-## Monday 2026-10-12 (afternoon): your 38 answers — written down, and 16 items closed
+## Monday 2026-10-12 (afternoon): your 38 answers — written down, the four DooPlex fixes done, 19 items closed
 
 - **All 38 answers are written down** in the decisions list (numbers 196–233). Where your words differed from my pick (S1, S2, S8, S32, S37), your words are the rule.
-- **The list went from 128 to 112.** 17 items closed with no code: the ones you accepted, the ones that are by design, and the apps you said no to. One new item: mirror the images of every app we support into our own registry, before we grow (your S8 note).
+- **The list went from 128 to 110.** 19 items closed: 16 with no code (accepted, by design, or apps you said no to), the Tester 2 mail check, and two DooPlex fixes. One new item: mirror the images of every app we support into our own registry, before we grow (your S8 note).
 - **The hub mailed you on time that Tester 2 never made an off-site copy** (Sunday 18:13). That check is closed.
 - **Prometheus on DooPlex is in sync** (S37). I read it once, read only: ArgoCD says Synced, and the live settings equal git.
 - **The Cloudflare token recipe is in the install runbook** (S31): two permissions, one zone, nothing else.
-- **What remains of the sheet, in order:** the four DooPlex changes (I ask you in chat before each), one hub release (you present for the deploy), one box release, Glance behind the family gate, the capability map, the notification-settings design, and last the six hub readers.
+- **The four DooPlex changes are done**, each after your yes, each checked:
+  - **CI:** the runner now waits 30 s for an answer (was 5 s), and the crawler is limited to 2 requests a second. In the first minute it got 7 refusals; our own traffic got none. I check for lost jobs again on 19 October.
+  - **Alertmanager** can save its state again. A test silence survived a restart; then I removed it.
+  - **Prometheus** now reloads its alarm rules by itself, about 40 s after a change. Tested twice, no manual step.
+  - **DooPlex's backup scripts are in git** (homelab-manifests) and were installed from there with you in the session. Nothing in them changed. Tonight's 03:10 backup is the test; I read it tomorrow.
+  - **One thing for you to know:** two DooPlex alarms are firing and were re-sent when Alertmanager restarted — a Longhorn volume at 91 %, and a disk (`ST3000VX010 …Z6A07P2G`) with 352 unreadable sectors since 10 October.
+- **What remains of the sheet, in order:** one hub release (you present for the deploy), one box release, Glance behind the family gate, the capability map, the notification-settings design, and last the six hub readers.
 
 ## Monday 2026-10-12 (review round): a page-by-page review, and its findings fixed
 

documentation/audits/answers-2026-10-12/s20-backup-scripts.txt

@@ -0,0 +1,10 @@
+# S20 / R-231 — DooPlex change 4, installed 2026-10-12 ~12:32Z with the operator in the session (yes in chat), homelab-manifests fdd324b
+live before: /opt/backup/scripts = 11 scripts + 4 .bak-<date> copies; units /etc/systemd/system/dooplex-backup.{service,timer} mode 744.
+copied: the 11 scripts (cmp: byte-identical) + the 2 units (cmp: byte-identical) into homelab-ansible/roles/dooplex_backup/files/; .bak copies left out.
+secret check: every file grepped for password/token/key/BEGIN/AKIA/credential URLs; only PATHS found (/etc/backup/restic-password,
+  /etc/backup/gitea-token, k3s token paths). No value in any file.
+dry run (--check --diff): scripts ok (identical), units "changed" (mode 744 -> 0644 only).
+install (install.sh --apply): ok=6 changed=1 failed=0; units now 644; timer enabled, next 2026-10-13 03:10:41 CEST; scripts cmp-identical to the repo.
+baseline: dooplex-backup.service last run Result=success, ExecMainStatus=0, exit 2026-10-12 03:16:17 CEST.
+to read 2026-10-13: the same three fields after the 03:10 run (DUE-CHECK R-231).
+undo: the role only rewrote files with identical content; git revert fdd324b removes it from the repo.

documentation/audits/answers-2026-10-12/s21-runner-and-ratelimit.txt

@@ -0,0 +1,9 @@
+# S21 / R-887 — DooPlex change 1, applied 2026-10-12 12:17Z (operator yes in chat), homelab-manifests ad8618f
+before (24 h Ingress log to 12:10Z): 21 505 gitea requests; 216.73.216.6 (crawler UA "compatible ...") 20 546, max 475/min;
+  37.191.56.193 (boxes' git + registry) 811, max 6/s, 25/min. Crawler paths: /src/commit 11 299, /commits/commit 4 420, /raw/commit 2 807.
+ArgoCD app gitea: OutOfSync on exactly ConfigMap/act-runner-config, Deployment/act-runner, Ingress/gitea -> sync Succeeded Synced Healthy.
+runner read-back: process "act_runner daemon --config /config/config.yaml"; /config/config.yaml fetch_timeout: 30s;
+  /data/.runner labels ['felhom-gates:host'] (unchanged); log "declare successfully".
+nginx read-back: limit_req_zone ... rate=2r/s; limit_req ... burst=10 nodelay (2 locations: gitea.dooplex.hu, gitea.home)
+first minute after: 216.73.216.6 14x200 + 7x503; 37.191.56.193 12x200, 0x503.
+undo: git revert ad8618f + sync app gitea.

documentation/audits/answers-2026-10-12/s22-alertmanager-silence.txt

@@ -0,0 +1,10 @@
+# S22 / R-886 — DooPlex change 2, applied 2026-10-12 ~12:19Z (operator yes in chat), homelab-manifests 4168451
+before: /alertmanager root:root 0755; log every 15 min "Running maintenance failed ... open /alertmanager/silences.*: permission denied" (and nflog); 0 silences.
+ArgoCD app monitoring: OutOfSync on exactly Deployment/alertmanager -> resource-only sync Succeeded.
+after: /alertmanager 0:65534 drwxrwsr-x (fsGroup applied).
+test: POST silence 52fb455b (matcher alertname=CC_R886_TEST_NO_SUCH_ALERT, matches nothing, 30 min);
+  delete pod alertmanager-9b557d68-98z2l -> new pod alertmanager-9b557d68-28rrq;
+  /alertmanager now holds nflog + silences (65534, 12:20); GET /api/v2/silences returns 52fb455b state active. Survived.
+cleanup: amtool silence expire 52fb455b -> state expired. No other silence existed.
+note: the restart re-sends firing alerts; two were active (LonghornVolumeSpaceWarning pvc-486c9809 91.2 %; SMART pending sectors on ata-ST3000VX010-2E3166_Z6A07P2G, 352, firing since 2026-10-10 08:21Z).
+undo: git revert 4168451 + sync Deployment/alertmanager.

documentation/audits/answers-2026-10-12/s23-prometheus-reloader.txt

@@ -0,0 +1,13 @@
+# S23 / R-211 — DooPlex change 3, applied 2026-10-12 12:21Z (operator yes in chat), homelab-manifests ab40c7b
+ArgoCD app monitoring: OutOfSync on exactly Deployment/prometheus -> resource-only sync Succeeded; pod prometheus-6f548d5cdd-l926j, containers config-reloader=ready prometheus=ready.
+reloader log: Watching directory "/etc/prometheus" and "/etc/prometheus/rules".
+test (no manual POST /-/reload anywhere):
+  e70e0f1 one comment changed in authentik-alerts.yml; ConfigMap-only sync 12:21:48Z ->
+    12:22:28 reloader "config map updated ... successfully triggered reload"; prometheus 12:22:28.506 "Loading configuration file", 12:22:28.556 "Completed loading"; file in pod holds the test text (grep -c = 1).
+  78d4be4 the comment removed (the rules file equals ab40c7b again); ConfigMap-only sync 12:28:11Z ->
+    12:28:42 reloader "successfully triggered reload"; grep -c in pod = 0.
+end state: app monitoring Synced Healthy at 78d4be4; /api/v1/rules 12 groups, 56 rules, health {ok}.
+incident, mine: a local `git revert` + `--amend` produced a wrong local commit; its push was REFUSED (non-fast-forward) and the
+  ArgoCD sync to that unknown revision ended Error with nothing applied. Fixed with `git reset --soft origin/main` (other people's
+  uncommitted files untouched) and a hand revert (78d4be4).
+undo: git revert ab40c7b + sync Deployment/prometheus.

documentation/backlog/CLOSED-ITEMS.md

@@ -32,6 +32,8 @@ The full text of the rows below: `git show 25ab47eeaa:documentation/backlog/OPEN
 
 | Row | What | Closed | Evidence |
 |---|---|---|---|
+| **R-886** | **DooPlex's Alertmanager cannot write its own state since the Longhorn restart of 2026-10-05 13:20Z** (P3) | CLOSED 2026-10-12 — **fixed on DooPlex** (operator yes in chat; S22, `09` §3 decision 217): Alertmanager pod `fsGroup: 65534` (homelab-manifests `4168451`, resource-only sync). Volume now `0:65534 drwxrwsr-x`; a test silence survived a pod delete (old pod `…98z2l` → new `…28rrq`, `GET /api/v2/silences` = active), then expired by amtool; `nflog` and `silences` files now written. | `s22-alertmanager-silence.txt` |
+| **R-211** | **Prometheus has no config-reloader — a rules change reaches the pod and is never read** (P3) | CLOSED 2026-10-12 — **fixed on DooPlex** (operator yes in chat; S23, `09` §3 decision 218): `config-reloader` sidecar (configmap-reload v0.15.0, by digest; homelab-manifests `ab40c7b`). Proven twice with no manual reload: a rule-comment change (`e70e0f1`) and its removal (`78d4be4`) each reached Prometheus ~35–40 s after a ConfigMap-only sync („successfully triggered reload" + „Completed loading of configuration file"); 56 rules `ok`. | `s23-prometheus-reloader.txt` |
 | **R-243** | **A box in the R-241 state silently stops backing up off-site, and NO ALARM OF ANY KIND FIRES.** (P2) | CLOSED 2026-10-12 — **the alarm fired on time.** The `offsite_escrow_pending` mail for Tester-2 reached the operator mailbox 2026-10-11 16:13:51Z (the first sweep after the 7-day line, ~16:06Z); the running hub logged „Operator email sent for Tester-2/offsite_escrow_pending" at the same second. Two channels. | hub v0.144.0+ (`b119301c`); `r243-escrow-pending-mail.txt` |
 | **R-831** | **The Hetzner storage API token (`HETZNER_TOKEN`, the storage project's token in `Secret/storagebox`) was printed into the 2026-10-03 session transcript** (P3) | CLOSED 2026-10-12 — operator ruling 2026-10-12 (S2, `09` §3 decision 197): **A — accept.** The Hetzner storage token stays valid. | the ruling |
 | **R-870** | **Tester 1's two Cloudflare credentials — the zone API token (`infrastructure.cf_api_token`) and the tunnel token (`infrastructure.cf_tunnel_token`) of the hub's `customer_configs` row `tester-1` — were printed into the 2026-10-04 night session's transcript** (P3) | CLOSED 2026-10-12 — operator ruling 2026-10-12 (S3, `09` §3 decision 198): **accepted**; rotate when Tester 1 is retired. | the ruling |