misc-scripts/REPORT.md

REPORT — gitea-image-prune.sh (2026-06-17)

New operator CLI to inspect/prune old container images in the Gitea registry and reclaim disk, with a load-bearing live spike to prove the reclaim mechanism.

Confirmed baselines (live)

Thing	Value
Gitea version	1.26.2 (GET /api/v1/version)
Owner namespace	admin (standard per-owner packages API; "admin" in the path is the owner)
Container packages	felhom-controller 96 tags / 247 digests, felhom-hub 42 tags / 96 digests, plus recipe-importer (42), revfulop-calendar (12), jarr (2), wan-probe (1)
Packages cron	cleanup_packages, default @midnight, OLDER_THAN = 24h
Packages dir	/data/gitea/packages (gitea-system pod, container gitea), baseline 5,122,143,723 B ≈ 4.77 GiB
Tooling	jq 1.7 present on build server; shellcheck absent (not run — script written carefully, bash -n clean)

Minimal token scope set (empirically confirmed via 403 bodies)

Operation	Required scope
list packages / versions / files	read:package
delete a tag / manifest version	write:package
list cron tasks	read:admin
trigger cleanup_packages cron	write:admin

The build server token (~/.gitea-token) has read:admin + write:package but not write:admin (its 403 body named exactly required=[write:admin]), so it can list/prune/delete-orphans but cannot trigger the GC cron on demand. Token must belong to a site-admin user. (Scopes were not minimized by minting reduced tokens — that needs write:user, which this token also lacks — but each required scope was confirmed by a successful call and the cron requirement by its 403.)

§3 spike — the reclaim mechanism (PROVEN, not assumed)

Gitea stores a tag as a tiny OCI index pointer; the real bytes are in untagged sha256: manifest versions (config + layer blobs), whose layers are shared across tags. The mechanism turned out to be three steps, not two:

Step (single-version spike, felhom-hub)	du (bytes)	freed
baseline	5,122,143,723	—
DELETE tag 0.1.1 (via the script, HTTP 204)	5,122,143,723	0
run cleanup_packages (tag-only)	5,122,138,771	~5 KB (index pointer only)
DELETE tag 0.1.2 + its 2 orphaned manifests (204×3)	5,122,138,771	0
run cleanup_packages GC	5,116,799,814	5,338,957 B ≈ 5.1 MiB

Conclusions:

1. Deleting a tag frees ~nothing (only the index pointer).
2. Default cleanup_packages does NOT remove untagged manifest versions — only unreferenced blobs. So the orphaned sha256: manifests must be deleted explicitly (the script's reclaim does this); otherwise their blobs stay referenced forever. (Confirmed: a tag-only delete + cron left felhom-hub digests at 96.)
3. Once the orphaned manifests are deleted, cleanup_packages GCs their unique blobs (created > OLDER_THAN); shared base layers stay. 5.1 MiB freed for one 9.4 MB-apparent image — the difference is shared layers, correctly retained.

Because the token lacks write:admin, the GC cron was triggered by adding [cron.cleanup_packages] RUN_AT_START = true to app.ini (on the data PVC, backup app.ini.bak.prune-spike) and rolling-restarting Gitea — left in place per operator request (the daily @midnight run also performs the GC).

§9 verification results

1. List (read-only): --repo felhom-hub list → 42 tags, newest-first, per-tag sizes resolved via OCI (24 MB recent, ~9 MB older), latest flagged PROTECTED, shared-layer caveat printed. --all lists all 6 packages; --no-sizes fast path works. No mutation.
2. Dry-run prune: --repo felhom-hub --keep 5 --dry-run → would delete 36, keep 5 + 1 protected, oldest first, totals shown, nothing changed.
3. One-version live proof (spike): see table above — delete-alone = 0 bytes; delete + orphan-manifest delete + GC = 5.1 MiB.
4. Full reclaim path validated on felhom-hub only: reclaim --apply deleted the 16 accumulated orphan manifests (untagged, referenced by no tag — dead weight from re-pointed latest + buildx attestations), then GC freed 5,116,799,814 → 5,026,431,222 = 90,368,592 B ≈ 86 MiB.
5. Safety: after reclaim, surviving tags still resolve and docker pull cleanly (latest, 0.1.3 — the immediate neighbor of the deleted tags). Orphan detection is fail-closed (skips a package if any surviving tag won't resolve).
6. Audit log captured every RUN / DRY-RUN / APPLIED / RECLAIM line; token scan of the log = 0 hits. Edge cases: --keep+--older-than → error; --keep 999 → "Nothing to prune (40 tags: 39 kept, 1 protected)".

State left on the registry

• felhom-hub: tags 0.1.1 and 0.1.2 deleted (spike); 16 orphan manifests cleaned; now 40 tags / 78 digests; ~91 MiB reclaimed total. All remaining tags pull cleanly.
• felhom-controller (96 tags) and all other packages: UNTOUCHED.
• app.ini: RUN_AT_START = true added for cleanup_packages (kept).

NOT yet run

The real bulk cleanup — left to the operator (interactive). This run proved the mechanism on one disposable version and validated the full reclaim path on felhom-hub's dead orphans only. Pruning the ~90 felhom-controller tags (and the bulk of felhom-hub/recipe-importer history) is the operator's call via gitea-image-prune.sh --repo … --keep N --apply --reclaim.

Backlog / notes

• A write:admin token (or the native cleanup rule in the UI) would let reclaim trigger the GC immediately instead of relying on the @midnight/restart run.
• Per-tag "apparent" sizes overlap (shared base layers counted once per tag); --measure (du) is the honest real-reclaim signal. Documented in the tool.
• shellcheck was unavailable on the build server, so the script was not statically linted (only bash -n syntax-checked + extensively run live).