admin
761dc3856e
New operator CLI (curl+jq, dry-run default) to list, prune (keep-N or older-than), and reclaim old container images in the self-hosted Gitea registry. Reclaim implements the three-step mechanism proven live on Gitea 1.26.2: delete tag (frees only the index pointer) -> delete the orphaned sha256 manifest versions (default cleanup_packages does NOT remove untagged manifests) -> cleanup_packages cron GCs the now unreferenced blobs. Orders by upload date, protects ^latest$, fail-closed orphan detection, audit log, never logs the token. Live-verified: single-version spike freed 5.1 MiB; cleaning felhom-hub's 16 orphan manifests freed 86 MiB; surviving tags still docker-pull. felhom-controller and other packages left untouched for the operator. Adds README section (usage, minimal token scopes, reclaim caveat, native cleanup-rule recommendation), CHANGELOG, REPORT, and .gitattributes (LF). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
109 lines
5.9 KiB
Markdown
109 lines
5.9 KiB
Markdown
# REPORT — `gitea-image-prune.sh` (2026-06-17)
|
||
|
||
New operator CLI to inspect/prune old container images in the Gitea registry and
|
||
reclaim disk, with a load-bearing live spike to **prove** the reclaim mechanism.
|
||
|
||
## Confirmed baselines (live)
|
||
|
||
| Thing | Value |
|
||
|---|---|
|
||
| Gitea version | **1.26.2** (`GET /api/v1/version`) |
|
||
| Owner namespace | `admin` (standard per-owner packages API; "admin" in the path is the owner) |
|
||
| Container packages | `felhom-controller` **96 tags / 247 digests**, `felhom-hub` **42 tags / 96 digests**, plus `recipe-importer` (42), `revfulop-calendar` (12), `jarr` (2), `wan-probe` (1) |
|
||
| Packages cron | `cleanup_packages`, default `@midnight`, `OLDER_THAN = 24h` |
|
||
| Packages dir | `/data/gitea/packages` (gitea-system pod, container `gitea`), **baseline 5,122,143,723 B ≈ 4.77 GiB** |
|
||
| Tooling | `jq` 1.7 present on build server; `shellcheck` **absent** (not run — script written carefully, `bash -n` clean) |
|
||
|
||
## Minimal token scope set (empirically confirmed via 403 bodies)
|
||
|
||
| Operation | Required scope |
|
||
|---|---|
|
||
| list packages / versions / files | `read:package` |
|
||
| delete a tag / manifest version | `write:package` |
|
||
| list cron tasks | `read:admin` |
|
||
| trigger `cleanup_packages` cron | **`write:admin`** |
|
||
|
||
The build server token (`~/.gitea-token`) has `read:admin` + `write:package` but
|
||
**not** `write:admin` (its 403 body named exactly `required=[write:admin]`), so it
|
||
can list/prune/delete-orphans but cannot trigger the GC cron on demand. Token must
|
||
belong to a site-admin user. (Scopes were *not* minimized by minting reduced
|
||
tokens — that needs `write:user`, which this token also lacks — but each required
|
||
scope was confirmed by a successful call and the cron requirement by its 403.)
|
||
|
||
## §3 spike — the reclaim mechanism (PROVEN, not assumed)
|
||
|
||
Gitea stores a tag as a tiny OCI **index** pointer; the real bytes are in untagged
|
||
`sha256:` **manifest versions** (config + layer blobs), whose layers are shared
|
||
across tags. The mechanism turned out to be **three steps**, not two:
|
||
|
||
| Step (single-version spike, `felhom-hub`) | `du` (bytes) | freed |
|
||
|---|---|---|
|
||
| baseline | 5,122,143,723 | — |
|
||
| DELETE tag `0.1.1` (via the script, HTTP 204) | 5,122,143,723 | **0** |
|
||
| run `cleanup_packages` (tag-only) | 5,122,138,771 | ~5 KB (index pointer only) |
|
||
| DELETE tag `0.1.2` + its **2 orphaned manifests** (204×3) | 5,122,138,771 | **0** |
|
||
| run `cleanup_packages` GC | 5,116,799,814 | **5,338,957 B ≈ 5.1 MiB** |
|
||
|
||
**Conclusions:**
|
||
1. Deleting a tag frees ~nothing (only the index pointer).
|
||
2. **Default `cleanup_packages` does NOT remove untagged manifest versions** —
|
||
only unreferenced *blobs*. So the orphaned `sha256:` manifests must be deleted
|
||
explicitly (the script's reclaim does this); otherwise their blobs stay
|
||
referenced forever. (Confirmed: a tag-only delete + cron left `felhom-hub`
|
||
digests at 96.)
|
||
3. Once the orphaned manifests are deleted, `cleanup_packages` GCs their *unique*
|
||
blobs (created > `OLDER_THAN`); shared base layers stay. 5.1 MiB freed for one
|
||
9.4 MB-apparent image — the difference is shared layers, correctly retained.
|
||
|
||
Because the token lacks `write:admin`, the GC cron was triggered by adding
|
||
`[cron.cleanup_packages] RUN_AT_START = true` to `app.ini` (on the data PVC,
|
||
backup `app.ini.bak.prune-spike`) and rolling-restarting Gitea — left in place per
|
||
operator request (the daily `@midnight` run also performs the GC).
|
||
|
||
## §9 verification results
|
||
|
||
1. **List (read-only):** `--repo felhom-hub list` → 42 tags, newest-first, per-tag
|
||
sizes resolved via OCI (24 MB recent, ~9 MB older), `latest` flagged PROTECTED,
|
||
shared-layer caveat printed. `--all` lists all 6 packages; `--no-sizes` fast path
|
||
works. No mutation.
|
||
2. **Dry-run prune:** `--repo felhom-hub --keep 5 --dry-run` → would delete 36,
|
||
keep 5 + 1 protected, oldest first, totals shown, **nothing changed**.
|
||
3. **One-version live proof (spike):** see table above — delete-alone = 0 bytes;
|
||
delete + orphan-manifest delete + GC = 5.1 MiB.
|
||
4. **Full reclaim path validated** on `felhom-hub` only: `reclaim --apply` deleted
|
||
the **16 accumulated orphan manifests** (untagged, referenced by no tag — dead
|
||
weight from re-pointed `latest` + buildx attestations), then GC freed
|
||
**5,116,799,814 → 5,026,431,222 = 90,368,592 B ≈ 86 MiB**.
|
||
5. **Safety:** after reclaim, surviving tags still resolve and **`docker pull`
|
||
cleanly** (`latest`, `0.1.3` — the immediate neighbor of the deleted tags).
|
||
Orphan detection is fail-closed (skips a package if any surviving tag won't
|
||
resolve).
|
||
6. **Audit log** captured every RUN / DRY-RUN / APPLIED / RECLAIM line; **token
|
||
scan of the log = 0 hits**. Edge cases: `--keep`+`--older-than` → error;
|
||
`--keep 999` → "Nothing to prune (40 tags: 39 kept, 1 protected)".
|
||
|
||
## State left on the registry
|
||
|
||
- `felhom-hub`: tags `0.1.1` and `0.1.2` deleted (spike); 16 orphan manifests
|
||
cleaned; now **40 tags / 78 digests**; ~91 MiB reclaimed total. All remaining
|
||
tags pull cleanly.
|
||
- **`felhom-controller` (96 tags) and all other packages: UNTOUCHED.**
|
||
- `app.ini`: `RUN_AT_START = true` added for `cleanup_packages` (kept).
|
||
|
||
## NOT yet run
|
||
|
||
**The real bulk cleanup — left to the operator (interactive).** This run proved
|
||
the mechanism on one disposable version and validated the full reclaim path on
|
||
`felhom-hub`'s dead orphans only. Pruning the ~90 `felhom-controller` tags (and
|
||
the bulk of `felhom-hub`/`recipe-importer` history) is the operator's call via
|
||
`gitea-image-prune.sh --repo … --keep N --apply --reclaim`.
|
||
|
||
## Backlog / notes
|
||
|
||
- A `write:admin` token (or the native cleanup rule in the UI) would let `reclaim`
|
||
trigger the GC immediately instead of relying on the `@midnight`/restart run.
|
||
- Per-tag "apparent" sizes overlap (shared base layers counted once per tag);
|
||
`--measure` (`du`) is the honest real-reclaim signal. Documented in the tool.
|
||
- `shellcheck` was unavailable on the build server, so the script was not
|
||
statically linted (only `bash -n` syntax-checked + extensively run live).
|