felhom.eu/REPORT-dockerhub-bigsteps-2026-10-28.md

REPORT — 2026-10-28 (from midday, attended at the STOPs): the watch and Docker Hub; the four big security updates

Architecture read: 08-alarm-ladder.md §11 (the watch); 09-update-architecture.md §3 (decisions 255, 274, 275) and §6.4 (the ladder); the catalog's CLAUDE.md (the test record) and its move gates.

In plain words

1. Why Docker Hub refused the watch: speed, not volume. The night run asked 211 times in about 2 minutes from six workers and got 15 refusals. Now the requests are spaced, and a refusal is retried. No Docker Hub login was needed.
2. One CI night with 0 refusals: not yet. Two daytime test runs had 0 refusals. Tonight's 03:40 run is the check (R-1334 stays open until then).
3. The four apps — all live, each a normal ladder step (bench + box + undo proven):
   • Wanderer v0.21.2
   • Termix 26.10.5 (two steps: 2.8.0 → 2.9.2 → 26.10.5)
   • Homepage v2.7.0
   • Seerr v3.5.1 (a definition step: new image name, new start command)
4. Wanderer is offered again: yes (your word; D59 ended). Federation from other servers stays off (your word).
5. What you do: nothing today. Tomorrow morning I check that the night run had 0 Docker Hub refusals and then close R-1334.

Decisions (yours, at the STOPs)

• Pushed, one STOP each: Wanderer, Termix, Homepage, Seerr.
• Wanderer offered again, and its inbound federation stays off (09 §3 decision 279).
• The Docker Hub key was not asked for. The fix without a login measured clean.

Part 1 — R-1334 (evidence: documentation/audits/dockerhub-bigsteps-2026-10-28/part1/)

The cause, measured. Docker Hub's tag-list API (hub.docker.com) refused requests that came in a burst:

Run	Requests	Time	Refused
Night 2446	211	~2 min	15
Dry 2450	224	~3.5 min	0

No scheduled job on DooPlex calls Docker Hub at that hour (k3s cron jobs and CI schedules, read only: Renovate runs on Saturdays; the others are Longhorn backups and Postgres dumps). Other traffic from the same address was not measured.

The fix.

• fe677bae:
  • Docker Hub requests are spaced: one per second across the workers.
  • A 429 is retried after its Retry-After. The image is NOT CHECKED only after the 4th try.
  • Tag lists are kept for 24 hours per registry path, on the CI runner. Two names of one image fetch once.
  • Test W14: 7 checks, red first.
• 8644450d: the image reader also retries a registry 429. ghcr.io had refused two blob reads when two dry runs came 15 minutes apart. Its retry checks were red first.

Measured after. Dry runs 2458 and 2459:

• 0 Docker Hub 429s and 0 retries. 210 requests over about 7 minutes.
• The only NOT CHECKED images are 6 private DooPlex ones (by design) and minio/minio:latest, which Docker Hub no longer serves (noted on R-1328 for the homelab session).
• The two CI runners each keep their own tag cache. So the cache helps a re-run on the same runner. The spacing is what fixes the night.

Part 2 — the four big steps (evidence: documentation/audits/dockerhub-bigsteps-2026-10-28/<app>/)

The path, by the rules. No box runs any of the four apps (checked on all three boxes and on 9202). There is still no pin move without a step: the catalog's test record and its move gates require it. 09 did not say so, so it now has a [DESIGN] note. So each app is a normal ladder step. In every case below the undo was proven on 9202: the new version, with its checks broken on purpose, ended undone, and the data read back on the old version.

Wanderer v0.20.0 → v0.21.2 — catalog 1825247

• Template, changed at v0.20.0 first:
  • PocketBase is internal (no public address, no database-subdomain question at install). Upstream says the db must not be reachable from outside.
  • The sign-up lock now also covers PUT /api/v1/user. v0.20.0 already had that route, and the old lock missed it.
  • No POCKETBASE_PROXY_SECRET, so inbound federation is rejected. There is no federation switch, and this is the only way to close it.
• Fixture: a user and a private list.
• Proof:
  • Bench: proven.
  • Box 9202: update in 78 s.
  • Undo: back to v0.20.0 in 28 s.
• Memory: Meilisearch fills whatever limit it gets: 256 → 384 MB, still 88.8 %. That is filed as R-1338.

Termix 2.8.0 → 2.9.2 → 26.10.5 — catalog e0b9594, 589c957

• Two pushes, because 2.9 must start once before 26.x.
• Fixture: the existing one.
• Proof, per hop: bench, box and undo all proven.
• D58, measured on 9202. Seven wrong logins, each with a different forged X-Forwarded-For:
  • From the LAN: Traefik replaces the header. The 6th try got HTTP 429 on 2.8.0, 2.9.2 and 26.10.5.
  • Through the tunnel's own address (a test container at cloudflared's address, with the chain Cloudflare delivers): the 6th try got 429 on all three.
  • So the bypass was not reachable on our boxes, before the update either. 26.10.5 adds the app's own guard (trust proxy: loopback).
• A loopback-only login route in 26.10.5: off in our image. NODE_ENV=production, and nginx sets X-Real-IP on every request.

Homepage v1.13.2 → v2.7.0 — catalog 1a254d4

• Fixture: a new one. It reads the dashboard's own services and bookmarks (written at first start) and plants nothing.
• Proof: bench, box and undo all proven. The family gate still answers 302.

Seerr: Jellyseerr 2.7.3 → Seerr v3.5.1 — catalog 9ef06a7

• A definition step:
  • The new image, plus init: true.
  • The container starts as root only to hand /app/config to UID 1000. Then it runs Seerr as node (smoke-tested first).
• Fixture: a new one. It brings a throwaway Jellyfin for Seerr's first sign-in and stores an application title.
• Proof:
  • Bench: proven. Its bare abort loses the seed on the converted data, as Jellyfin did yesterday.
  • Box 9202: proven.
  • Undo: Seerr ran its migration, failed its check, and the box's undo copy brought the data back.

Tools added in the catalog.

• box_walk.py --to-definition, with 8 checks in test_box_walk_drill_move.py.
• The Wanderer, Homepage and Seerr fixtures.
• The bench's Wanderer override was removed, because the template does it now.
• The undo tool: audits/dockerhub-bigsteps-2026-10-28/undo_proof2.py. Its broken probe now uses port 1, because a single-page app answers 200 on any path.

Commits and CI (all green by hash)

Repo	Commit	What	CI job
felhom.eu	fe677bae	Docker Hub spacing, retry, tag cache	gates 2457; dry runs 2458 and 2459
felhom.eu	8644450d, 3c0a33c1	image-reader retry; evidence	2460
catalog	1825247	Wanderer	2461
catalog	e0b9594	Termix hop 1	2462
catalog	589c957	Termix hop 2	2463
catalog	1a254d4	Homepage	2464
catalog	9ef06a7	Seerr	2467
felhom.eu	this close-out	docs, register, evidence	in the final check

Register — 229 before, 226 after

Opened (1)	Closed (4)
R-1338 — Meilisearch memory	R-1335 — Wanderer
	R-1311 — Termix
	R-1320 — Homepage
	R-1317 — Seerr

R-1334 is VERIFY; it closes after tonight. R-1328 has a note added (minio).

Observations

Observation	Result
Meilisearch fills its memory limit	FILED: R-1338
minio/minio:latest gives 401 on Docker Hub	FILED: noted on R-1328 (the homelab list)
Three security-review notices: „image downgrade" in the drill	NOT-A-FINDING: the drill is the test copy and must stand at FROM before a walk; the live catalog is at the new versions
The update accepts a container whose Docker check is still „starting"	NOT-A-FINDING: by design the declared probe decides (09 §3); the undo tool now breaks the probe on port 1
test-record-move judges a whole push as one move, so two hops refused together	NOT-A-FINDING: the rule works as built; Termix went out as two pushes
The memory_tight mark needs a raised limit and a new measurement	NOT-A-FINDING: the gate did its job; done for Wanderer
The Termix header test, first try: no gate cookie, and the shell expanded $x too early	NOT-A-FINDING: my test tool; fixed and re-run, and every try is kept
The Termix header test with a forged-only chain skipped the limit on every version	NOT-A-FINDING: Cloudflare always appends the real address; with the real chain every version locks
Termix 2.9.2's route dropped for a while right after the first install (502, then 404)	NOT-A-FINDING: the test now waits for health; 2.9.2 is only a stop on the way to 26.10.5
Termix 26.10.5's loopback auto-login route	NOT-A-FINDING: off in production, and nginx sets X-Real-IP
Secret scan of the evidence found "password": "wrong-"	NOT-A-FINDING: the deliberately wrong test password, a code literal

Teardown

• Machine: 9202 is on the LIVE catalog (repoint-9202-live.txt, cache 9ef06a7). It runs only its standing apps. Every test app was removed through the product. The throwaway Jellyfin containers were removed on the bench and on 9202.
• Bench: 9401 is stopped, with 0 containers.
• Host and hub: nothing provisioned, no change.