felhom.eu/REPORT-dockerhub-bigsteps-2026-10-28.md

# REPORT — 2026-10-28 (from midday, attended at the STOPs): the watch and Docker Hub; the four big security updates

Architecture read: `08-alarm-ladder.md` §11 (the watch); `09-update-architecture.md` §3 (decisions 255, 274, 275) and
§6.4 (the ladder); the catalog's `CLAUDE.md` (the test record) and its move gates.

## In plain words

1. **Why Docker Hub refused the watch:** speed, not volume. The night run asked 211 times in about 2 minutes from six
   workers and got 15 refusals. Now the requests are spaced, and a refusal is retried. **No Docker Hub login was needed.**
2. **One CI night with 0 refusals:** not yet. Two daytime test runs had 0 refusals. Tonight's 03:40 run is the check
   (R-1334 stays open until then).
3. **The four apps — all live, each a normal ladder step** (bench + box + undo proven):
   - Wanderer v0.21.2
   - Termix 26.10.5 (two steps: 2.8.0 → 2.9.2 → 26.10.5)
   - Homepage v2.7.0
   - Seerr v3.5.1 (a definition step: new image name, new start command)
4. **Wanderer is offered again: yes** (your word; D59 ended). Federation from other servers stays off (your word).
5. **What you do:** nothing today. Tomorrow morning I check that the night run had 0 Docker Hub refusals and then
   close R-1334.

## Decisions (yours, at the STOPs)

- **Pushed, one STOP each:** Wanderer, Termix, Homepage, Seerr.
- **Wanderer offered again**, and **its inbound federation stays off** (`09` §3 decision 279).
- **The Docker Hub key was not asked for.** The fix without a login measured clean.

## Part 1 — R-1334 (evidence: `documentation/audits/dockerhub-bigsteps-2026-10-28/part1/`)

**The cause, measured.** Docker Hub's tag-list API (`hub.docker.com`) refused requests that came in a burst:

| Run | Requests | Time | Refused |
|---|---|---|---|
| Night 2446 | 211 | ~2 min | 15 |
| Dry 2450 | 224 | ~3.5 min | 0 |

No scheduled job on DooPlex calls Docker Hub at that hour (k3s cron jobs and CI schedules, read only: Renovate runs
on Saturdays; the others are Longhorn backups and Postgres dumps). Other traffic from the same address was not measured.

**The fix.**
- `fe677bae`:
  - Docker Hub requests are spaced: one per second across the workers.
  - A 429 is retried after its `Retry-After`. The image is NOT CHECKED only after the 4th try.
  - Tag lists are kept for 24 hours per registry path, on the CI runner. Two names of one image fetch once.
  - Test W14: 7 checks, red first.
- `8644450d`: the image reader also retries a registry 429. ghcr.io had refused two blob reads when two dry runs came
  15 minutes apart. Its retry checks were red first.

**Measured after.** Dry runs 2458 and 2459:
- 0 Docker Hub 429s and 0 retries. 210 requests over about 7 minutes.
- The only NOT CHECKED images are 6 private DooPlex ones (by design) and `minio/minio:latest`, which Docker Hub no
  longer serves (noted on R-1328 for the homelab session).
- The two CI runners each keep their own tag cache. So the cache helps a re-run on the same runner. The spacing is
  what fixes the night.

## Part 2 — the four big steps (evidence: `documentation/audits/dockerhub-bigsteps-2026-10-28/<app>/`)

**The path, by the rules.** No box runs any of the four apps (checked on all three boxes and on 9202). There is still
no pin move without a step: the catalog's test record and its move gates require it. `09` did not say so, so it now
has a `[DESIGN]` note. So each app is a normal ladder step. In every case below the undo was proven on 9202: the new
version, with its checks broken on purpose, ended `undone`, and the data read back on the old version.

### Wanderer v0.20.0 → v0.21.2 — catalog `1825247`

- **Template, changed at v0.20.0 first:**
  - PocketBase is internal (no public address, no database-subdomain question at install). Upstream says the db must
    not be reachable from outside.
  - The sign-up lock now also covers `PUT /api/v1/user`. v0.20.0 already had that route, and the old lock missed it.
  - No `POCKETBASE_PROXY_SECRET`, so inbound federation is rejected. There is no federation switch, and this is the
    only way to close it.
- **Fixture:** a user and a private list.
- **Proof:**
  - Bench: `proven`.
  - Box 9202: update in 78 s.
  - Undo: back to v0.20.0 in 28 s.
- **Memory:** Meilisearch fills whatever limit it gets: 256 → 384 MB, still 88.8 %. That is filed as R-1338.

### Termix 2.8.0 → 2.9.2 → 26.10.5 — catalog `e0b9594`, `589c957`

- **Two pushes**, because 2.9 must start once before 26.x.
- **Fixture:** the existing one.
- **Proof, per hop:** bench, box and undo all proven.
- **D58, measured on 9202.** Seven wrong logins, each with a different forged `X-Forwarded-For`:
  - **From the LAN:** Traefik replaces the header. The 6th try got HTTP 429 on 2.8.0, 2.9.2 and 26.10.5.
  - **Through the tunnel's own address** (a test container at cloudflared's address, with the chain Cloudflare
    delivers): the 6th try got 429 on all three.
  - So the bypass was not reachable on our boxes, before the update either. 26.10.5 adds the app's own guard
    (`trust proxy: loopback`).
- **A loopback-only login route in 26.10.5:** off in our image. `NODE_ENV=production`, and nginx sets `X-Real-IP` on
  every request.

### Homepage v1.13.2 → v2.7.0 — catalog `1a254d4`

- **Fixture:** a new one. It reads the dashboard's own services and bookmarks (written at first start) and plants
  nothing.
- **Proof:** bench, box and undo all proven. The family gate still answers 302.

### Seerr: Jellyseerr 2.7.3 → Seerr v3.5.1 — catalog `9ef06a7`

- **A definition step:**
  - The new image, plus `init: true`.
  - The container starts as root only to hand `/app/config` to UID 1000. Then it runs Seerr as `node` (smoke-tested
    first).
- **Fixture:** a new one. It brings a throwaway Jellyfin for Seerr's first sign-in and stores an application title.
- **Proof:**
  - Bench: `proven`. Its bare abort loses the seed on the converted data, as Jellyfin did yesterday.
  - Box 9202: `proven`.
  - Undo: Seerr ran its migration, failed its check, and the box's undo copy brought the data back.

**Tools added in the catalog.**
- `box_walk.py --to-definition`, with 8 checks in `test_box_walk_drill_move.py`.
- The Wanderer, Homepage and Seerr fixtures.
- The bench's Wanderer override was removed, because the template does it now.
- The undo tool: `audits/dockerhub-bigsteps-2026-10-28/undo_proof2.py`. Its broken probe now uses port 1, because a
  single-page app answers 200 on any path.

## Commits and CI (all green by hash)

| Repo | Commit | What | CI job |
|---|---|---|---|
| felhom.eu | `fe677bae` | Docker Hub spacing, retry, tag cache | gates 2457; dry runs 2458 and 2459 |
| felhom.eu | `8644450d`, `3c0a33c1` | image-reader retry; evidence | 2460 |
| catalog | `1825247` | Wanderer | 2461 |
| catalog | `e0b9594` | Termix hop 1 | 2462 |
| catalog | `589c957` | Termix hop 2 | 2463 |
| catalog | `1a254d4` | Homepage | 2464 |
| catalog | `9ef06a7` | Seerr | 2467 |
| felhom.eu | this close-out | docs, register, evidence | in the final check |

## Register — 229 before, 226 after

| Opened (1) | Closed (4) |
|---|---|
| R-1338 — Meilisearch memory | R-1335 — Wanderer |
| | R-1311 — Termix |
| | R-1320 — Homepage |
| | R-1317 — Seerr |

R-1334 is VERIFY; it closes after tonight. R-1328 has a note added (minio).

## Observations

| Observation | Result |
|---|---|
| Meilisearch fills its memory limit | FILED: R-1338 |
| `minio/minio:latest` gives 401 on Docker Hub | FILED: noted on R-1328 (the homelab list) |
| Three security-review notices: „image downgrade" in the drill | NOT-A-FINDING: the drill is the test copy and must stand at FROM before a walk; the live catalog is at the new versions |
| The update accepts a container whose Docker check is still „starting" | NOT-A-FINDING: by design the declared probe decides (`09` §3); the undo tool now breaks the probe on port 1 |
| `test-record-move` judges a whole push as one move, so two hops refused together | NOT-A-FINDING: the rule works as built; Termix went out as two pushes |
| The `memory_tight` mark needs a raised limit and a new measurement | NOT-A-FINDING: the gate did its job; done for Wanderer |
| The Termix header test, first try: no gate cookie, and the shell expanded `$x` too early | NOT-A-FINDING: my test tool; fixed and re-run, and every try is kept |
| The Termix header test with a forged-only chain skipped the limit on every version | NOT-A-FINDING: Cloudflare always appends the real address; with the real chain every version locks |
| Termix 2.9.2's route dropped for a while right after the first install (502, then 404) | NOT-A-FINDING: the test now waits for health; 2.9.2 is only a stop on the way to 26.10.5 |
| Termix 26.10.5's loopback auto-login route | NOT-A-FINDING: off in production, and nginx sets `X-Real-IP` |
| Secret scan of the evidence found `"password": "wrong-"` | NOT-A-FINDING: the deliberately wrong test password, a code literal |

## Teardown

- **Machine:** 9202 is on the LIVE catalog (`repoint-9202-live.txt`, cache `9ef06a7`). It runs only its standing apps.
  Every test app was removed through the product. The throwaway Jellyfin containers were removed on the bench and
  on 9202.
- **Bench:** 9401 is stopped, with 0 containers.
- **Host and hub:** nothing provisioned, no change.