admin
c9d5ed575c
Protected whatever --keep says: the controller floor, the vouched golden, the vouched agent and min_agent (the hub's Configuration page), every image of ours the vouched golden baked (its bake.log), the hub manifest's image. The dry-run prints each kept version and why. tests/test-prune-plan.sh pins it without network (red-proofed). No --apply was run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
158 lines
8.0 KiB
Markdown
158 lines
8.0 KiB
Markdown
# misc-scripts
|
||
|
||
Operator helper scripts for the Felhom / DooPlex infrastructure. These are
|
||
stand-alone CLI utilities (English output — operator-facing, not customer UI).
|
||
|
||
| Script | Purpose |
|
||
|---|---|
|
||
| `build-felhom-hub.sh` | Build & push the `felhom-hub` Docker image to the Gitea registry. |
|
||
| `build-felhom-controller.sh` | Build & push the `felhom-controller` image. |
|
||
| `collect-repos.sh` | Concatenate repo sources into text files (for review/archival). |
|
||
| `gitea-image-prune.sh` | Inspect & prune old container images in the Gitea registry, then reclaim disk. |
|
||
|
||
---
|
||
|
||
## `gitea-image-prune.sh`
|
||
|
||
`build-felhom-{hub,controller}.sh` push `:<version>` **and** `:latest` on every
|
||
build, so the container packages accumulate one image per build and the Gitea
|
||
Longhorn PVC fills up. This tool lists, prunes, and reclaims that space, with a
|
||
safe **dry-run default**.
|
||
|
||
Best run on the **build server (192.168.0.180)** — it has `kubectl` for the
|
||
optional disk measurement and network to Gitea. The core (list / prune /
|
||
reclaim) is pure `curl` + `jq` and runs from any host with a token.
|
||
|
||
### How Gitea stores container images (why reclaim takes three steps)
|
||
|
||
A pushed **tag** is an OCI image *index* — a ~850 B pointer. The real bytes live
|
||
in untagged **`sha256:` manifest versions** (config + layer blobs, ~9–25 MB
|
||
each), whose layer blobs are content-addressed and **shared across tags**.
|
||
|
||
Confirmed live on Gitea 1.26.2 (2026-06-17), the reclaim path is **three steps**:
|
||
|
||
1. **Delete the tag** → frees ~nothing (only the tiny index pointer).
|
||
2. **Delete the now-orphaned `sha256:` manifest versions** (referenced by no
|
||
surviving tag). Default `cleanup_packages` does **not** remove untagged
|
||
manifests — only a configured *cleanup rule* would — so the tool deletes them
|
||
itself. This makes their *unique* blobs unreferenced.
|
||
3. **`cleanup_packages` cron runs** → garbage-collects unreferenced blobs created
|
||
more than `OLDER_THAN` (24 h default) ago → **this is what frees disk**. Layer
|
||
blobs still shared with surviving tags are correctly retained.
|
||
|
||
`prune` does step 1; `reclaim` does steps 2 + 3.
|
||
|
||
> **Live proof:** deleting one 9.4 MB-apparent tag + its 2 manifests + GC freed
|
||
> 5.1 MiB (the rest was shared base layers, correctly kept). Cleaning 16
|
||
> accumulated orphan manifests on `felhom-hub` then freed **86 MiB**. After
|
||
> reclaim, surviving tags (`latest`, `0.1.3`, …) still `docker pull` cleanly.
|
||
|
||
### Credentials
|
||
|
||
The script resolves credentials in this order:
|
||
|
||
1. `GITEA_TOKEN` env var
|
||
2. `--token-file <path>`
|
||
3. **git's stored credential for the Gitea host** — auto-discovered when you run
|
||
the script inside a clone: first the token embedded in the remote URL
|
||
(`https://user:token@host/…`), else a configured **credential helper**
|
||
(`git credential fill`). No prompting.
|
||
|
||
So from a configured clone you can just run `./gitea-image-prune.sh --repo … list`
|
||
with no token at all. The startup banner prints which source was used. Caveat: a
|
||
*git* credential may only carry repo scope — if it lacks package/admin scope, the
|
||
call returns a 403 naming the missing scope (see below).
|
||
|
||
### Required scopes
|
||
|
||
Pass an admin-user token via `GITEA_TOKEN` (env) or `--token-file <path>`. The
|
||
token must belong to a Gitea **site-admin** user. Minimal fine-grained scopes
|
||
(empirically confirmed against 1.26.2 via the API's 403 bodies):
|
||
|
||
| Operation | Scope |
|
||
|---|---|
|
||
| list packages / versions / files | `read:package` |
|
||
| delete a tag / manifest version | `write:package` |
|
||
| list cron tasks | `read:admin` |
|
||
| **trigger** the `cleanup_packages` GC cron | `write:admin` |
|
||
|
||
The token is **never** echoed, logged, or committed. If the token lacks
|
||
`write:admin`, the tool still deletes orphaned manifests and reports that their
|
||
blobs will be freed by the daily `@midnight` `cleanup_packages` run (or on the
|
||
next Gitea restart — see "native retention" below).
|
||
|
||
> The project's read-only token (and the build server's `~/.gitea-token`, which
|
||
> has `read:admin` + `write:package` but **not** `write:admin`) can list, prune,
|
||
> and delete orphans, but cannot trigger the GC cron on demand.
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
# Interactive menu (pick packages, then an action):
|
||
GITEA_TOKEN=… ./gitea-image-prune.sh
|
||
|
||
# List one / all packages (with apparent per-tag sizes):
|
||
./gitea-image-prune.sh --repo felhom-hub list
|
||
./gitea-image-prune.sh --all list
|
||
./gitea-image-prune.sh --all --no-sizes list # fast, skip size resolution
|
||
|
||
# Dry-run prune (DEFAULT — shows what would go, mutates nothing). --keep defaults to 20:
|
||
./gitea-image-prune.sh --all prune # containers
|
||
./gitea-image-prune.sh --type generic --all prune # agent + golden
|
||
./gitea-image-prune.sh --repo felhom-controller --older-than 90
|
||
|
||
# Apply for real — a PERSON's act, after reading the dry-run (typed confirmation unless --yes):
|
||
./gitea-image-prune.sh --all prune --apply --reclaim --measure
|
||
|
||
# Reclaim only (delete orphaned manifests + trigger/await GC):
|
||
./gitea-image-prune.sh --repo felhom-hub reclaim --apply
|
||
```
|
||
|
||
### The retention rule (Felhom `09` §3 decision 62, operator ruling 2026-10-01)
|
||
|
||
A prune keeps the **newest 20** versions of every package **plus every version in use**, whatever `--keep` or
|
||
`--older-than` says: the controller floor, the vouched golden, the vouched agent and its `min_agent` (read from the
|
||
hub's operator Configuration page with `HUB_PW` — env, or `~/.config/credentials` through
|
||
`felhom.eu/scripts/read_credential.py`), every `gitea.dooplex.hu/admin/<pkg>:<tag>` the vouched golden baked (its
|
||
`bake.log` under `felhom.eu/documentation/tests/`), and the hub image `felhom.eu/manifests/hub.yaml` runs. The dry-run
|
||
prints each kept version with its reason. **If any of these cannot be read, the prune refuses (exit 3).** Nothing runs
|
||
this script on a schedule; `--apply` is a person's act. `tests/test-prune-plan.sh` pins the rule without network.
|
||
|
||
### Flags
|
||
|
||
| Flag | Meaning |
|
||
|---|---|
|
||
| `--repo NAME` (repeatable) / `--all` | Which packages to act on. |
|
||
| `list` / `prune` / `reclaim` | Action (positional; inferred as `prune` if `--keep`/`--older-than` given, else `list`). |
|
||
| `--keep N` | Keep the N most-recent tags; delete older. |
|
||
| `--older-than DAYS` | Keep tags newer than DAYS; delete older. (Mutually exclusive with `--keep`.) |
|
||
| `--dry-run` (default) / `--apply` | Mutate only with `--apply`. |
|
||
| `--yes` | Skip confirmation prompts (for cron). |
|
||
| `--reclaim` | After an `--apply` prune, also run the reclaim stage. |
|
||
| `--measure` | Best-effort before/after `du` of the packages dir (needs `kubectl`; skipped cleanly otherwise). |
|
||
| `--protect REGEX` (repeatable) | Never delete matching tags. `^latest$` is always protected. |
|
||
| `--no-sizes` | Skip per-tag size resolution (much faster lists). |
|
||
| `--token-file PATH` | Read the token from a file instead of `GITEA_TOKEN`. |
|
||
| `--owner NAME` | Package owner (default `admin`). |
|
||
| `--log PATH` | Audit log (default `misc-scripts/logs/gitea-prune-<date>.log`). |
|
||
|
||
Safety: dry-run is the default; `latest` (and `--protect` matches) are never
|
||
deleted even if "old"; tags are ordered by **upload date**, never by parsing the
|
||
tag (the registry mixes `v`-prefixed and bare tags); every HTTP status is checked
|
||
(no silent failures); orphan detection is **fail-closed** (if any surviving tag
|
||
can't be resolved, that package's orphan cleanup is skipped rather than risk
|
||
deleting a referenced manifest).
|
||
|
||
### Native retention (set-and-forget complement)
|
||
|
||
This script is the **on-demand** tool. The **standing** complement is a Gitea
|
||
per-owner **cleanup rule** (web UI: *Packages → owner → Settings → Cleanup
|
||
Rules*): set *Remove all versions except the most recent N*, and add a tag-match
|
||
exclusion for `latest`. The daily `cleanup_packages` cron then enforces it
|
||
automatically. This tool does **not** auto-create rules (deliberately) — configure
|
||
them once in the UI if you want hands-off retention.
|
||
|
||
> Note: on this instance, `[cron.cleanup_packages] RUN_AT_START = true` was added
|
||
> to `app.ini` (2026-06-17) so the GC also runs on every Gitea restart — useful
|
||
> when reclaiming without a `write:admin` token.
|