From afc939bb3c47e383dfd90e821bddb5a62297c945 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 19 Oct 2026 03:42:11 +0200 Subject: [PATCH] R-887 closed: 381 CI jobs since the S21 change, 0 lost (the 3 re-runs' first attempts ran 47-49 s; control: the known lost attempt ran 11 m 48 s); register 168 -> 167 --- .../day-2026-10-19/R-887/r887-attempts.txt | 11 ++++++++ .../day-2026-10-19/R-887/r887-control.txt | 4 +++ .../day-2026-10-19/R-887/r887-measure.txt | 26 +++++++++++++++++++ .../day-2026-10-19/R-887/r887-tasks.txt | 16 ++++++++++++ documentation/backlog/CLOSED-ITEMS.md | 6 +++++ documentation/backlog/OPEN-ITEMS.md | 2 -- 6 files changed, 63 insertions(+), 2 deletions(-) create mode 100644 documentation/audits/day-2026-10-19/R-887/r887-attempts.txt create mode 100644 documentation/audits/day-2026-10-19/R-887/r887-control.txt create mode 100644 documentation/audits/day-2026-10-19/R-887/r887-measure.txt create mode 100644 documentation/audits/day-2026-10-19/R-887/r887-tasks.txt diff --git a/documentation/audits/day-2026-10-19/R-887/r887-attempts.txt b/documentation/audits/day-2026-10-19/R-887/r887-attempts.txt new file mode 100644 index 00000000..7e61911a --- /dev/null +++ b/documentation/audits/day-2026-10-19/R-887/r887-attempts.txt @@ -0,0 +1,11 @@ +control (before the window): jobs with run_attempt>1: 17 + ('felhom-controller', 1401, 2, '2026-10-06T00:44:57Z', 'success') + ('felhom-controller', 1357, 2, '2026-10-05T15:05:37Z', 'success') + ('felhom-controller', 1549, 2, '2026-10-08T08:26:11Z', 'success') + ('felhom-controller', 1639, 2, '2026-10-10T13:09:06Z', 'success') + ('felhom-agent', 1579, 2, '2026-10-09T04:58:15Z', 'success') + ('app-catalog-felhom.eu', 1368, 2, '2026-10-05T17:15:12Z', 'success') +window since 2026-10-12T12:00:00Z : jobs with run_attempt>1: 3 + ('felhom-controller', 1944, 2, '2026-10-16T13:50:35Z', 'success') + ('felhom-controller', 1959, 2, '2026-10-16T17:07:53Z', 'success') + ('felhom-controller', 1991, 2, '2026-10-17T09:59:02Z', 'success') diff --git a/documentation/audits/day-2026-10-19/R-887/r887-control.txt b/documentation/audits/day-2026-10-19/R-887/r887-control.txt new file mode 100644 index 00000000..10d489d1 --- /dev/null +++ b/documentation/audits/day-2026-10-19/R-887/r887-control.txt @@ -0,0 +1,4 @@ +felhom.eu 1645 log bytes 38590 +felhom.eu 1646 log bytes 38503 +felhom-controller 1639 log bytes 18965 +felhom.eu 1738 log bytes 46399 diff --git a/documentation/audits/day-2026-10-19/R-887/r887-measure.txt b/documentation/audits/day-2026-10-19/R-887/r887-measure.txt new file mode 100644 index 00000000..472a06e1 --- /dev/null +++ b/documentation/audits/day-2026-10-19/R-887/r887-measure.txt @@ -0,0 +1,26 @@ +window since 2026-10-12T12:00:00Z: 381 jobs in 4 repos; not-success 25; with no log (lost) 0 + not-success ('felhom.eu', 1738, 'c2a6cf7a', 'completed', 'failure', '2026-10-13T04:47:12Z', 1) has log + not-success ('felhom.eu', 1747, 'ed59bf3c', 'completed', 'failure', '2026-10-13T17:28:57Z', 1) has log + not-success ('felhom.eu', 1796, 'a2ca6eb4', 'completed', 'failure', '2026-10-14T16:51:48Z', 1) has log + not-success ('felhom.eu', 1889, '6749118c', 'completed', 'failure', '2026-10-15T19:21:34Z', 1) has log + not-success ('felhom.eu', 1993, 'b9c4f73b', 'completed', 'failure', '2026-10-17T10:02:08Z', 1) has log + not-success ('felhom.eu', 1997, '498b9139', 'completed', 'failure', '2026-10-17T10:42:11Z', 1) has log + not-success ('felhom-controller', 1865, 'e66d4669', 'completed', 'failure', '2026-10-15T16:37:58Z', 1) has log + not-success ('felhom-controller', 1867, 'fe0228c8', 'completed', 'failure', '2026-10-15T17:05:00Z', 1) has log + not-success ('felhom-controller', 1869, 'db52f5a4', 'completed', 'failure', '2026-10-15T17:27:25Z', 1) has log + not-success ('felhom-controller', 1870, '83392064', 'completed', 'failure', '2026-10-15T17:32:48Z', 1) has log + not-success ('felhom-controller', 1871, '9cc7ffcf', 'completed', 'failure', '2026-10-15T18:08:11Z', 1) has log + not-success ('felhom-controller', 1873, '7f3e58ff', 'completed', 'failure', '2026-10-15T18:30:00Z', 1) has log + not-success ('felhom-controller', 1875, '48d86748', 'completed', 'failure', '2026-10-15T18:33:27Z', 1) has log + not-success ('felhom-controller', 1876, '006ceab5', 'completed', 'failure', '2026-10-15T18:41:12Z', 1) has log + not-success ('felhom-controller', 1877, 'f908eee2', 'completed', 'failure', '2026-10-15T18:45:59Z', 1) has log + not-success ('felhom-controller', 1878, '02252556', 'completed', 'failure', '2026-10-15T18:46:27Z', 1) has log + not-success ('felhom-controller', 1879, '9e68e526', 'completed', 'failure', '2026-10-15T18:51:46Z', 1) has log + not-success ('felhom-controller', 1880, '8c7aa10a', 'completed', 'failure', '2026-10-15T18:52:08Z', 1) has log + not-success ('felhom-controller', 1882, 'ec438479', 'completed', 'failure', '2026-10-15T18:57:00Z', 1) has log + not-success ('felhom-controller', 1883, 'f9151f08', 'completed', 'failure', '2026-10-15T19:04:54Z', 1) has log + not-success ('felhom-controller', 1884, '4261a38b', 'completed', 'failure', '2026-10-15T19:05:47Z', 1) has log + not-success ('felhom-controller', 1887, 'c6a56712', 'completed', 'failure', '2026-10-15T19:09:40Z', 1) has log + not-success ('felhom-controller', 1990, '334df7db', 'completed', 'failure', '2026-10-17T09:53:24Z', 1) has log + not-success ('felhom-controller', 2036, 'e30091bf', 'completed', 'failure', '2026-10-17T15:30:09Z', 1) has log + not-success ('app-catalog-felhom.eu', 2067, 'c97eb1c9', 'completed', 'failure', '2026-10-18T18:45:53Z', 1) has log diff --git a/documentation/audits/day-2026-10-19/R-887/r887-tasks.txt b/documentation/audits/day-2026-10-19/R-887/r887-tasks.txt new file mode 100644 index 00000000..788250a6 --- /dev/null +++ b/documentation/audits/day-2026-10-19/R-887/r887-tasks.txt @@ -0,0 +1,16 @@ +task fields: ['created_at', 'display_title', 'event', 'head_branch', 'head_sha', 'id', 'name', 'run_number', 'run_started_at', 'status', 'updated_at', 'url', 'workflow_id'] +{'id': 1991, 'name': 'gates', 'head_sha': 'b69585789a9fa34a7c33faf24da44926f166a9da', 'run_number': 393, 'status': 'success', 'event': 'push', 'created_at': '2026-10-16T21:50:09Z', 'run_started_at': '2026-10-16T21:50:09Z', 'updated_at': '2026-10-16T21:50:58Z'} +{'id': 1959, 'name': 'gates', 'head_sha': 'f30a80841e372abf9d7992dc9eb3eec607078e90', 'run_number': 379, 'status': 'success', 'event': 'push', 'created_at': '2026-10-16T13:19:24Z', 'run_started_at': '2026-10-16T13:19:24Z', 'updated_at': '2026-10-16T13:20:14Z'} +{'id': 1639, 'name': 'gates', 'head_sha': '1cb43a21a2dc5cd44fbcc0749b460dba1ffa8088', 'run_number': 326, 'status': 'success', 'event': 'push', 'created_at': '2026-10-10T07:55:47Z', 'run_started_at': '2026-10-10T07:55:47Z', 'updated_at': '2026-10-10T07:56:52Z'} +job 1944 sha c02c9d83 attempt 2 created 2026-10-16T13:50:35Z started 2026-10-16T14:02:29Z completed 2026-10-16T14:03:21Z +job 1959 sha 29a3ce8c attempt 2 created 2026-10-16T17:07:53Z started 2026-10-16T17:19:08Z completed 2026-10-16T17:20:00Z +job 1991 sha 3e8d0276 attempt 2 created 2026-10-17T09:59:02Z started 2026-10-17T10:15:35Z completed 2026-10-17T10:16:23Z + task 2015 for job 1991 success 2026-10-17T10:15:35Z -> 2026-10-17T10:16:24Z + task 2011 for job 1991 failure 2026-10-17T09:59:02Z -> 2026-10-17T09:59:49Z + task 1980 for job 1959 success 2026-10-16T17:19:08Z -> 2026-10-16T17:20:00Z + task 1978 for job 1959 failure 2026-10-16T17:07:54Z -> 2026-10-16T17:08:42Z + task 1964 for job 1944 success 2026-10-16T14:02:29Z -> 2026-10-16T14:03:21Z + task 1962 for job 1944 failure 2026-10-16T13:50:35Z -> 2026-10-16T13:51:24Z +CONTROL job 1639 (a known lost attempt, 2026-10-10) sha 9e5b8814 + task 1655 success 2026-10-10T13:22:35Z -> 2026-10-10T13:23:51Z + task 1653 failure 2026-10-10T13:09:10Z -> 2026-10-10T13:20:58Z diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index b15fcdd3..c685a1c9 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,12 @@ --- +## 2026-10-19 (day) — agent 0.160.0, one sign-in for the file manager, hub small fixes (full text of each row: `git show e4f022a0:documentation/backlog/OPEN-ITEMS.md`) + +| Row | What | Closed | Evidence | +|---|---|---|---| +| **R-887** | **Some CI jobs are never run, and Gitea fails them ~10–13 minutes later with no log** (Gitea's zombie-task stop after the runner abandoned its fetch; S21 on 2026-10-12: runner `fetch_timeout` 5 s → 30 s, crawler rate limit). (P3) | CLOSED 2026-10-19 — **the dated check is met:** since 2026-10-12 12:00Z the four repos ran **381 jobs, 0 lost**. 25 ended `failure`, every one with a readable log (the runner ran them); 3 jobs had a second attempt (controller 1944, 1959, 1991), and each first attempt (tasks 1962, 1978, 2011) failed in 47–49 s — a run that did work, not the lost shape (one cause on record: the register not pushed yet, `REPORT-burndown2-2026-10-16.md`). **Control:** the known lost attempt of controller job 1639 (task 1653, 2026-10-10) ran 11 min 48 s, and the attempt count finds it. **Reasoning kept:** the jobs endpoint shows only the LAST attempt (R-1137), so „a failed job with no log" alone cannot find a lost attempt that was re-run — count `run_attempt > 1` and read the first attempt in `actions/tasks`. | `audits/day-2026-10-19/R-887/` | + ## 2026-10-18 (evening) — a fresh install from scratch, OnlyOffice saves, Redis without a volume, CI mail link (controller v0.331.0, catalog `30eea97`) | Row | What | Closed | Evidence | diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 08c357fb..e995608f 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -314,7 +314,6 @@ stopping line that lies. | **R-987** | Process & tooling | P4 | **The 53 catalog app images have never been scanned for known vulnerabilities.** Fix: a monthly trivy pass over the images the catalog pins, reported, not gating. | OPEN | — | Add to the monthly re-test | CC | | **R-986** | Process & tooling | P3 | **No dependency or secret scanner runs on any push, so a known-vulnerable library or a committed key is found only by a review like this one.** Proposed gate: govulncheck (called vulns only), gitleaks on the diff, trivy on our own images; start as WARN with an accepted-hits file, then FAIL. Tools and pinned versions: `audits/security-review-2026-10-12/SCANNERS.md`. Note: gitleaks' default rules did not find R-908's key. | OPEN | — | Build the gate, warn-first | CC | | **R-733** | Process & tooling | P3 | **[P3-LOW] The test bench has NO swap and the boxes have 512 MiB — so a box proof can pass on swap where the bench fails, and nobody records whether a customer guest has swap.** MEASURED 2026-09-30 (R-732): immich's first start was OOM-killed 61–104 times on the bench (swap 0) and passed on 9202 by swapping ~108 MB; the bench given 512 MiB swap passed too. demo-hp 9201, 9202 and demo-felhom 9201 all read `swap: 512`; the golden's guest config is not recorded in its bake evidence, so a customer guest's swap is NOT measured. The harness's memory watch judges `anon` against the limit and never reads `memory.swap.current`. **Needs:** the golden's `swap` read and recorded; the box walk and the harness report `memory.swap.peak` beside `anon`; a decision whether proofs run with swap off (the stricter venue, as R-732's fix was proven). | **READY — rank P3-LOW; owner: CC (harness + golden evidence)** **2026-10-06 night: NARROWED** — the harness records each container's `swap_peak` and the venue's swap (catalog branch `night-held-2026-10-06`, `3f4611c`; SwapRecorded red-proved; bench SwapTotal 0 kB, 9202 524288 kB). LEFT: the golden's swap in its bake evidence, the box walk's record, and whether proofs run with swap off. | — | — | CC | -| **R-887** | Process & tooling | P3 | **Some CI jobs are never run, and Gitea fails them ~10–13 minutes later with no log.** Seen 2026-10-05: felhom.eu job 1361 (commit `1122b5c`) and felhom-controller job 1357 (`114ff27`): every step reads `failure`, including the first fetch, the log API answers `file does not exist`, and the runner pod's log has no `task` line for them (its task ids are job id + 1). A re-run through the API ran the controller job normally (success in 32 s) but the felhom.eu job was again never picked up and failed after ~12 min. The runner pod (`gitea-system/act-runner`, image `felhom-act-runner:0.1.0`) had restarted 5 times ~142 min earlier, around the Longhorn instance-manager restart (R-882). Suspected, NOT measured: a stale runner registration claims jobs it never runs — the session's Gitea token cannot list runners (`read:admin` scope). Consequence: a red CI verdict that is not about the code, and **no failure mail** (the alarm step never runs either), so only the pull check sees it. **CORRECTED 2026-10-05 18:21 (operator's screenshot of Gitea → Site Administration → Runners): ONE runner only — ID 2, `felhom-gates-runner`, v0.6.1, label `felhom-gates`, Idle, last online „now". There is no old registration; the stale-registration guess (this row's first text and the reviewer's) was WRONG.** **RE-DIAGNOSED 2026-10-05 (round 2), from the logs that survive:** (1) **„lost in a runner restart" does NOT fit** — the runner pod last restarted 13:24:42Z (`restartCount 5`, all around the 13:20Z Longhorn restart), the lost attempts started 1.5–2.5 h later. (2) **FOUR attempts were lost, not two:** controller job 1357 (start 15:05:41Z → failed 15:18:38Z), felhom.eu job 1359 (15:15:37 → 15:28:38 — the previous session blamed that one on the BusyBox fault; the runner never ran it), job 1361 (15:33:21 → 15:43:38) and its API re-run (15:46:48 → 15:58:38). None has a `task` line in the runner log; every one was failed at a :38-second mark on a 5-minute step, 10–13 min after it was handed out — **the shape of Gitea's periodic „zombie task" stop** (a task assigned to a runner that never reports is failed after ~10 min; no log exists because none was written). (3) The runner's task ids are NOT job id + 1 (the controller re-run was task 1363). (4) **Gitea's own log for the window is gone** — the pod log starts 16:16:32Z (rotated), so the assignment side cannot be read. **Likely mechanism, NOT proven:** the runner's fetch-task request timed out on its side after Gitea had already assigned the task, so the task was orphaned. In that same hour this session polled Gitea's jobs API hard (15 pages every 15 s per wait loop) and Gitea logged „slow" requests — a plausible load cause, and the session's own. Mitigation taken: the session's CI waiter now polls once a minute. **Nothing changed on DooPlex.** **MECHANISM SEEN 2026-10-05 17:15–17:28Z, with Gitea's own log (round 2):** catalog run 1368 (`4828dc7`) — 17:15:14 the job is marked started; 17:15:16 `router: slow POST /api/actions/runner.v1.RunnerService/FetchTask for 10.42.0.42 (the runner), elapsed 3192ms`, then `UpdateRepoRunsNumbers … context canceled` and `GetActionWorkflow: EOF` — **the runner abandoned its fetch after Gitea had assigned the task**; the runner log has no line for task 1371; 17:28:39 `actions/clear_tasks.go:174 stopTasks() [W] Cannot transfer logs of task 1371` — Gitea's zombie-task stop. **The load at that minute:** an outside crawler (216.73.216.78) walking commit pages and `archive/*.tar.gz`, and THIS session's CI waiter, whose 15-page job listings took 13–31 s each. An API re-run passed in 7 s. **Done in-session:** the waiter now asks `GET …/actions/runs?head_sha=` once a minute (1 s). **Not done (DooPlex, the operator's):** the runner's fetch timeout and Gitea's exposure to the crawler. **-- 2026-10-08 (seen):** felhom-controller job 1549 (`05e1292`, a REPORT/CONTEXT-only commit) started 08:29:46Z and ended 08:43:17Z with all three steps `failure` at the same second and no log (logs endpoint HTTP 500); the same tree passed `controller_gates.py --fast` locally; a re-run (`POST …/actions/runs/1549/rerun`) completed `success`. A lost job inside the watch window. **SEEN AGAIN 2026-10-10, and this time TWO JOBS IN TWO REPOS DIED AT THE SAME INSTANT — which is new information about the cause:** app-catalog-felhom.eu job **1631** (`b7f0f7c`, started 10:31:54Z, three steps) and felhom.eu job **1632** (`970616b7`, started 10:34:17Z, seven steps) both have EVERY step `failure` at **10:45:58Z with 0 s each** — the same wall-clock second — after ~14 and ~11.5 minutes in set-up with nothing run, and both logs answer `file does not exist`. A per-job problem cannot do that; one runner event took both. The SAME TREES passed on the next push (catalog job **1633**, `8ea72aa`, success in the normal time), and a shallow clone with no sibling — the CI shape — runs all nine applicable catalog gates green by hand. So this is the flake, not the code. **That makes the DUE-CHECK's premise false: jobs HAVE been lost since 2026-10-05 16:00Z.** **THREE jobs were lost on 2026-10-10 — 1631, 1632 and 1634 — and ALL THREE ended at :45:58 past the hour** (10:45:58Z, 10:45:58Z, 12:45:58Z) after sitting 10.5–14 minutes in set-up. A job that is simply never picked up would end at an arbitrary second; ending at the same second-of-the-hour twice two hours apart points at a periodic sweep or a fixed-deadline timer, not at the job. Between the lost ones the runner DID work: catalog job **1633** started 12:35:06Z and succeeded normally. Worth knowing for the diagnosis: the runner pod was restarted earlier the same day (`audits/release-2026-10-10/ci-runner-restart.txt`). **Next step that would settle it:** read the runner pod's own log around an HH:45:58 and see what it says about the task id (job id + 1 — 1645, 1646, 1648 here), and look for a 10-minute or hourly timer in its config. | **WATCHING** — **2026-10-12 12:50Z, my own change broke pushes, fixed at once:** the 2 r/s limit refused a hub image push with 503 (DooPlex and the home boxes reach `gitea.dooplex.hu` via the home public address 37.191.56.193). homelab-manifests `d0767fe`: that address added to the whitelist; a new Ingress `gitea-registry` serves `/v2` without any limit (boxes anywhere pull in parallel). The rebuild then pushed. The security review noted that `/v2` login attempts are not rate-limited — as before 2026-10-12, when nothing was. **WATCHING** — **2026-10-12 12:17Z: the S21 change is LIVE on DooPlex** (operator yes in chat; homelab-manifests `ad8618f`): runner `fetch_timeout` 5s → 30s (`/config/config.yaml`, read in the running process); Gitea Ingress `limit-rps 2` (burst 10), LAN not limited. Measured before: the crawler 20 546 requests in 24 h, up to 475/min; first minute after: crawler 7×503, our traffic 0×503 (`audits/answers-2026-10-12/s21-runner-and-ratelimit.txt`). Closes if no job is lost by 2026-10-19 (DUE-CHECKS). **READY** — operator ruling 2026-10-12 (S21, `09` §3 decision 216): yes — raise the act-runner fetch timeout and rate-limit the public Gitea pages the crawler walks (DooPlex, each change after the operator's yes in chat). **2026-10-12 dated check read: NOT met** — 4 jobs lost 2026-10-10/11 (controller 1639, felhom.eu 1645, 1646, 1672; each re-run once, each passed); re-dated 2026-10-19, measured over the 7 days after the S21 change. **OPEN** **DATED CHECK 2026-10-12 (DUE-CHECKS):** if no job was lost since 2026-10-05 16:00Z (no completed job whose runner log has no `task` line / whose log API answers `file does not exist`), close. **NIGHT WATCH 2026-10-05/06 (burn-down night): 2 jobs lost of ~30 runs** — felhom.eu run 1384 (`4aa4d837`, 21:23→21:33Z, no log) and felhom-controller run 1401 (`c67b26be`, 00:45→00:58Z, no log); each re-run once through the API and each passed (2 m 05 s, 57 s). The night's waiter made one filtered call a minute. So the 2026-10-12 close condition („no job lost since 2026-10-05 16:00Z") is already NOT met. `audits/night-burndown-2026-10-05/r887-lost-jobs.txt`. **NIGHT WATCH 2026-10-06/07 (second burn-down night): 0 jobs lost** — every push checked by its commit (felhom.eu, controller, agent, catalog main and the held branch), each completed `success`, one filtered call per check. | — | 2026-10-19: count lost jobs since 2026-10-12 12:17Z; none → close | CC | | **R-902** | Process & tooling | P3 | **The contact form's mailer has no source in any repository: the running image cannot be rebuilt or changed.** FOUND 2026-10-08 (website refresh, brief Part C5). `manifests/contact-mailer.yaml` names `contact-mailer:latest` with `imagePullPolicy: Never` (imported into k3s by hand); the binary in the pod is `/app/contact-mailer` dated 2026-02-05, Go module `github.com/felhom/contact-mailer` (devel). Searched: `git log --all` of felhom.eu (only the manifest), every repo under `/mnt/5_hdd/felhom.eu/git`, `find /mnt/5_hdd/felhom.eu ~ -maxdepth 5 -iname '*contact-mailer*'`. READ from the binary (strings, read-only `kubectl exec … cat`): ONE send per form (`sendViaResend`) to `TO_EMAIL`, `reply_to` = the visitor; nothing is sent back to the visitor; the operator's notification is Hungarian (`buildEmailHTML`, „Csatolmány … fájl"). The English form needs no mailer change (its subject labels carry „(EN)"). `audits/website-refresh-2026-10-08/mailer.md` | **OPEN — NARROWED 2026-10-08 (search done, NOT FOUND): owner: operator.** A read-only search found no source on DooPlex (55,548 `*.go`/`go.mod`/`Dockerfile*` files, every disk, media/Longhorn/containerd/PBS excluded by name), in the web editor's storage and file history, in all 10 Gitea repositories' history, or in Docker's build cache (oldest record 2026-09-28). The binary's own clues: one file `/build/main.go`, go1.23.12, no `vcs.*` stamp; image built 2026-02-05 10:40 by BuildKit. The binary is now KEPT in git (`audits/mailer-source-2026-10-08/contact-mailer.bin`), so a DooPlex rebuild no longer loses the program. A replacement plan is written (`audits/mailer-source-2026-10-08/PLAN-replacement.md`); one question left: is the February folder on the operator's Windows workstation? | — | Operator: look on the workstation; if absent, build the replacement from the plan (attended deploy) | operator | | **R-934** | Process & tooling | P3 | **Pull the failed disk ST3000VX010 S/N Z6A07P2G out of DooPlex; follow the checklist.** The data was moved off at the first alarm (2026-10-10); the disk is unmounted, its fstab line commented out since 2026-08-14, k3s no longer waits for it, and its empty Longhorn disk entry was removed 2026-10-12. Its two SMART alarms (352 unreadable / offline-uncorrectable sectors, flat since 2026-10-09) are silenced by serial until **2026-11-11 14:32 UTC** — if the disk is still in the machine then, they come back on purpose. **A TWIN of the same model (S/N Z6A070XZ, `/mnt/3_hdd`) is in use: read the serial before unplugging.** Checklist: `runbooks/dooplex-remove-disk-Z6A07P2G.md`. | **OPEN** — NEW 2026-10-12 | the operator (hands, a power-off window) | Follow the checklist; after the first boot remove the exporter line and the silence; tell CC | operator | | **R-935** | Process & tooling | P3 | **Something sets `core.bare = true` on the shared `felhom.eu` checkout, and then every git command there fails („this operation must be run in a work tree").** SEEN 2026-10-12 evening twice: `.git/config` changed at 16:29:37 and 16:37:33 CEST, each time with `core.bare=true`; restored by hand both times (`git config core.bare false`, a flag only, no file touched). **Not caused by the gates:** `repo_gates.py --fast` run plainly and with `GIT_DIR=.git` (as a hook gets it) left the flag `false`; a scratch `git init --bare ` under `GIT_DIR` does not flip the parent either. The first flip came while this session did not touch the repo; another session pushed to main at 16:44 from elsewhere. | **OPEN** — 2026-10-12 later: across this session's next push the flag stayed `false` (read before and after); not this session's pushes. **OPEN** — NEW 2026-10-12 **Note 2026-10-12 (security review):** at least some of these flips match the security review's pushes from a git worktree of this clone (about 16:29 and 16:37): inside the pre-push hook, `scripts/test_gate_decoys.py`'s poster-facts case runs `git init`/`git config`/`git commit` with the hook's inherited `GIT_DIR`, so it wrote into the real repository — four test commits in the worktree and `user.name d` / `user.email d@d` in the shared `.git/config` (the review removed that `[user]` section at about 16:50; commits 6c2c3dd5 and 186f3aa3 carry that author). The review stopped pushing from the worktree. Fix: the test helper must drop `GIT_DIR`, `GIT_WORK_TREE` and `GIT_INDEX_FILE` from its environment. | finding the writer | Watch `.git/config` (an inotify or audit rule on the file) during a day with the parallel sessions; name the command; fix it there | CC | @@ -347,7 +346,6 @@ stopping line that lies. the R-row. Duplicating them here would create the second source this design avoids. --> | item | due (UTC) | what to measure | |---|---|---| -| R-887 | 2026-10-19 | no CI job lost in the 7 days after the S21 DooPlex change (act-runner fetch timeout + crawler rate limit); detail in the R-887 row | | R-895 | 2026-10-20 | the first verdict of the hub's own off-site snapshot check (hub 0.158.0; 2026-10-19 07:10 is the baseline): read the 07:10 log lines (count, newest upload vs the box's last run) and whether either alarm fired; detail in the R-895 row | | R-435 | 2026-10-20 | Tester 1's weekly off-site clean-up ran inside a hub window with no drop alarm (box log + mailbox); detail in the R-435 row | | R-1050 | 2026-11-15 | HSTS one year if no https trouble since 2026-10-15; detail in the R-1050 row |