felhom.eu/documentation/audits/security-2026-11-02/part1/DIAGNOSIS.md

Why the watch's memory showed two sources down for a week (R-1334, R-1303) — numbers first, 2026-11-02

Read from the CI logs of every night since 10-28 (night-<job>.log here; older: audits/dockerhub-bigsteps-2026-10-28/part1/). The 03:20 run is gates (repo_gates on the night's HEAD), not the watch. The watch is only the 03:40 run.

Night	Job	Runner	TAGS fetched / cache / retried-429	hub.docker.com requests	registry:docker.io failures	runtime failures
10-29	2483	runner-2	0 / 116 / 0	1	minio (401)	minio (401) + 6 private
10-30	2531	runner	117 / 0 / 0	206	minio + alpine/k8s (read timeout)	same 7
10-31	2543	runner-2	117 / 0 / 0	207	minio + linuxserver/code-server (timeout)	same 7
11-01	2545	runner-2	1 / 115 / 0	4	minio	same 7
11-02	2547	runner-2	116 / 0 / 0	207	minio + mongodb-community-server, wanderer-db, linuxserver/grocy (timeouts)	same 7

The 7: minio/minio:latest (Docker Hub answers 401 on tags and manifest: the image is no longer published there) and six of our own images in the private registry (gitea.dooplex.hu/admin/{felhom-act-runner,jarr,recipe-importer, revfulop-calendar,sparkyfitness-export,wan-probe}, „no anonymous token route"). All seven are DooPlex-group pins.

What this means

1. R-1334's fix DID hold. Five real nights, two different runners, 0 HTTP 429 from Docker Hub, 0 retries. The brief's reading („did not hold") came from the streak numbers; the logs say the streak has another cause.
2. The watch was not blind to Docker Hub. About 115 of 116 tag lists and every readable runtime were read every night (RUNTIME … from-cache=181–185, read-path failures 0). The zstd wheel is fine (zstandard 0.25.0, sha256 checked).
3. What kept the streak open — the mechanism: plan() counts a streak per SOURCE (scripts/security_watch.py, „for s in sorted(set(fails) | set(failed_now))"): one pin failing keeps the whole source's streak running. Seven pins that can NEVER be read fail every night, so registry:docker.io and runtime could never reset, mailed stayed true, and a real outage of either source could not have mailed. That is the real hole, and it is why the operator heard nothing: nothing new was happening, but nothing new could have been heard either.
4. Real one-night misses: read timeouts. 0–3 pins a night, a different one each time (30 s, Net.get, no retry), plus one GitHub feed (nginx, 10-29). Each was read fine the next night. On 2026-11-02 by hand from DooPlex: also a ghcr.io 429 on one tag list, and timeouts on karakeep, Radarr and wanderer feeds. So it is the line, not one host.
5. Three instruments lied, so the diagnosis took the state file, not the log:
   • RUNTIME … not-checked=0 for six nights while runtime failed: the resolve path returned before it counted.
   • The table shows one class per pin: an ADVISORY pin hid its failed source (night 10-27: streak 1 with NOT_CHECKED absent from COUNTS).
   • Dry runs looked clean because the same 7 were there too and read as „expected".

What was read in the blind week that is reported for the first time

Nothing because of the blindness: every pin that timed out was read the next night, and the seven stated pins are still unread (now a stated GAP, in every weekly mail). The fixed run's new lines come from R-1310's better reading (dry-local-after-fix.log, 13 new): each GHSA a release names (Vaultwarden's two, Uptime Kuma's GHSA-v2p3, Traefik's GHSA-xv5x), and notices newer than the last night (Meilisearch's release lines, Actual's GHSA-p7p2). They go to Part 2.

The fix (felhom.eu only)

• The seven are a GAP with their reason, listed in every weekly mail: no failed source, so the streak resets. STATED_UNREADABLE names minio with the ONE answer it gives; an unlisted 401, or minio failing another way, stays NOT CHECKED. Test W15, W12.
• A read timeout is tried again on every source (a budget of 30 a night, so an outage cannot pass the job's limit); a ghcr.io 429 too. The resolve gets one more try. Test W15.
• The log: FAILED-SOURCE … for every failed source of every pin, FAILED-SOURCES n, a truthful RUNTIME line, retried-timeout.
• No Docker Hub login and no runner image change are needed.