53 lines
4.3 KiB
Markdown
53 lines
4.3 KiB
Markdown
# Why the watch's memory showed two sources down for a week (R-1334, R-1303) — numbers first, 2026-11-02
|
||
|
||
Read from the CI logs of every night since 10-28 (`night-<job>.log` here; older: `audits/dockerhub-bigsteps-2026-10-28/part1/`).
|
||
**The 03:20 run is `gates` (repo_gates on the night's HEAD), not the watch. The watch is only the 03:40 run.**
|
||
|
||
| Night | Job | Runner | TAGS fetched / cache / retried-429 | hub.docker.com requests | `registry:docker.io` failures | `runtime` failures |
|
||
|---|---|---|---|---|---|---|
|
||
| 10-29 | 2483 | runner-2 | 0 / 116 / 0 | 1 | minio (401) | minio (401) + 6 private |
|
||
| 10-30 | 2531 | runner | 117 / 0 / 0 | 206 | minio + `alpine/k8s` (read timeout) | same 7 |
|
||
| 10-31 | 2543 | runner-2 | 117 / 0 / 0 | 207 | minio + `linuxserver/code-server` (timeout) | same 7 |
|
||
| 11-01 | 2545 | runner-2 | 1 / 115 / 0 | 4 | minio | same 7 |
|
||
| 11-02 | 2547 | runner-2 | 116 / 0 / 0 | 207 | minio + `mongodb-community-server`, `wanderer-db`, `linuxserver/grocy` (timeouts) | same 7 |
|
||
|
||
The 7: `minio/minio:latest` (Docker Hub answers 401 on tags and manifest: the image is no longer published there) and six
|
||
of our own images in the private registry (`gitea.dooplex.hu/admin/{felhom-act-runner,jarr,recipe-importer,
|
||
revfulop-calendar,sparkyfitness-export,wan-probe}`, „no anonymous token route"). All seven are DooPlex-group pins.
|
||
|
||
## What this means
|
||
|
||
1. **R-1334's fix DID hold.** Five real nights, two different runners, **0 HTTP 429** from Docker Hub, 0 retries. The brief's
|
||
reading („did not hold") came from the streak numbers; the logs say the streak has another cause.
|
||
2. **The watch was not blind to Docker Hub.** About 115 of 116 tag lists and every readable runtime were read every night
|
||
(`RUNTIME … from-cache=181–185`, read-path failures 0). The zstd wheel is fine (`zstandard 0.25.0`, sha256 checked).
|
||
3. **What kept the streak open — the mechanism:** `plan()` counts a streak per SOURCE (`scripts/security_watch.py`, „for s in
|
||
sorted(set(fails) | set(failed_now))"): one pin failing keeps the whole source's streak running. Seven pins that can
|
||
NEVER be read fail every night, so `registry:docker.io` and `runtime` could never reset, `mailed` stayed true, and **a
|
||
real outage of either source could not have mailed**. That is the real hole, and it is why the operator heard nothing:
|
||
nothing new was happening, but nothing new could have been heard either.
|
||
4. **Real one-night misses: read timeouts.** 0–3 pins a night, a different one each time (30 s, `Net.get`, no retry), plus
|
||
one GitHub feed (nginx, 10-29). Each was read fine the next night. On 2026-11-02 by hand from DooPlex: also a ghcr.io 429
|
||
on one tag list, and timeouts on karakeep, Radarr and wanderer feeds. So it is the line, not one host.
|
||
5. **Three instruments lied, so the diagnosis took the state file, not the log:**
|
||
- `RUNTIME … not-checked=0` for six nights while `runtime` failed: the resolve path returned before it counted.
|
||
- The table shows one class per pin: an ADVISORY pin hid its failed source (night 10-27: streak 1 with `NOT_CHECKED` absent
|
||
from COUNTS).
|
||
- Dry runs looked clean because the same 7 were there too and read as „expected".
|
||
|
||
## What was read in the blind week that is reported for the first time
|
||
|
||
**Nothing because of the blindness:** every pin that timed out was read the next night, and the seven stated pins are still
|
||
unread (now a stated GAP, in every weekly mail). The fixed run's new lines come from R-1310's better reading (`dry-local-after-fix.log`,
|
||
13 new): each GHSA a release names (Vaultwarden's two, Uptime Kuma's GHSA-v2p3, **Traefik's GHSA-xv5x**), and notices newer than
|
||
the last night (Meilisearch's release lines, Actual's GHSA-p7p2). They go to Part 2.
|
||
|
||
## The fix (felhom.eu only)
|
||
|
||
- The seven are a GAP with their reason, listed in every weekly mail: no failed source, so the streak resets. `STATED_UNREADABLE`
|
||
names minio with the ONE answer it gives; an unlisted 401, or minio failing another way, stays NOT CHECKED. Test W15, W12.
|
||
- A read timeout is tried again on every source (a budget of 30 a night, so an outage cannot pass the job's limit); a ghcr.io
|
||
429 too. The resolve gets one more try. Test W15.
|
||
- The log: `FAILED-SOURCE …` for every failed source of every pin, `FAILED-SOURCES n`, a truthful RUNTIME line, `retried-timeout`.
|
||
- No Docker Hub login and no runner image change are needed.
|