admin
5b4d8ec6cf
When GITEA_TOKEN/--token-file aren't set and the script runs inside a Gitea-host clone, reuse git's stored credential: the token embedded in the remote URL, else a configured credential helper (git credential fill, no prompting). Switch to HTTP Basic auth (user:token) when a username is known so both API tokens and the embedded-URL/helper credential work; keep the token header for a bare GITEA_TOKEN. Banner reports the source. Live-verified: env token + git credential helper (as kisfenyo) both list and resolve OCI sizes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
743 lines
34 KiB
Bash
Executable File
743 lines
34 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# gitea-image-prune.sh — inspect & prune old container images in the
|
|
# self-hosted Gitea registry, then reclaim disk.
|
|
# =============================================================================
|
|
# Target server : gitea.dooplex.hu (Gitea 1.26.2, owner namespace "admin")
|
|
# Best run on : build server 192.168.0.180 (has kubectl for --measure),
|
|
# but the core (list/prune/reclaim) is pure curl + jq and runs
|
|
# from any host that can reach Gitea and holds an admin token.
|
|
#
|
|
# -----------------------------------------------------------------------------
|
|
# WHY THIS EXISTS
|
|
# build-felhom-{hub,controller}.sh push :<version> AND :latest on every build,
|
|
# so felhom-controller (~96 tags) / felhom-hub (~42 tags) accumulate one image
|
|
# per build and the Gitea Longhorn PVC keeps filling.
|
|
#
|
|
# HOW GITEA STORES CONTAINER IMAGES (load-bearing — drives the reclaim design)
|
|
# A pushed tag is an OCI image *index* (a ~850 B pointer). The real bytes live
|
|
# in untagged "sha256:..." manifest *versions* (config + layer blobs, ~9-25 MB
|
|
# each), whose layer blobs are content-addressed and SHARED across tags.
|
|
# Deleting a tag removes only the tiny index pointer — the manifest versions it
|
|
# referenced linger as untagged versions, and their blobs stay referenced.
|
|
#
|
|
# PROVEN RECLAIM MECHANISM (verified live on this instance, 2026-06-17 — see §3
|
|
# of the task / REPORT.md). It is THREE steps, not two:
|
|
# 1. DELETE the tag(s) -> frees ~nothing (only the index ptr)
|
|
# 2. DELETE the now-ORPHANED "sha256:" manifest versions (referenced by no
|
|
# surviving tag) -> still frees nothing on its own, BUT
|
|
# makes their unique blobs unreferenced. (Default cleanup_packages does
|
|
# NOT remove untagged manifests — only a cleanup *rule* would — so the
|
|
# script must delete the orphaned manifests itself. write:package scope.)
|
|
# 3. cleanup_packages cron runs -> GCs unreferenced blobs created
|
|
# >OLDER_THAN (24h default) ago -> THIS frees disk. Shared base layers
|
|
# still referenced by surviving tags are correctly retained.
|
|
# Live proof: deleting one 9.4 MB-apparent tag + its 2 manifests + GC freed
|
|
# 5.1 MiB (the rest was shared base layers, correctly kept).
|
|
# => "prune" deletes tags; "reclaim" deletes the orphaned manifests and then
|
|
# triggers (or defers to) the cleanup_packages cron.
|
|
#
|
|
# TRIGGERING THE GC CRON: POST /api/v1/admin/cron/cleanup_packages needs
|
|
# write:admin. If the token lacks it, the orphaned manifests are still deleted
|
|
# and their blobs are freed by the daily "@midnight" run (or on the next Gitea
|
|
# restart — RUN_AT_START was enabled in app.ini on 2026-06-17).
|
|
#
|
|
# CREDENTIALS (in order: GITEA_TOKEN env -> --token-file -> git's stored
|
|
# credential for the Gitea host: the remote URL's embedded token, else a
|
|
# configured credential helper). Auto-discovery lets you just run the script
|
|
# inside a clone with no token fuss — but a *git* credential may only have repo
|
|
# scope; if so, a 403 will name the missing package/admin scope.
|
|
#
|
|
# REQUIRED TOKEN (env GITEA_TOKEN, or --token-file). Must belong to a Gitea
|
|
# site-admin user. Minimal fine-grained scopes (Gitea 1.26):
|
|
# read:package — list packages / versions / files (list, prune planning)
|
|
# write:package — delete a version (prune --apply)
|
|
# read:admin — list cron tasks
|
|
# write:admin — run the cleanup_packages cron (reclaim)
|
|
# The project read-only token is insufficient (no package scope).
|
|
# The token is NEVER echoed, logged, or committed.
|
|
#
|
|
# USAGE
|
|
# GITEA_TOKEN=... ./gitea-image-prune.sh # interactive menu
|
|
# ./gitea-image-prune.sh --repo felhom-hub list # list one package
|
|
# ./gitea-image-prune.sh --all list # list all packages
|
|
# ./gitea-image-prune.sh --repo felhom-hub --keep 10 # dry-run prune (default)
|
|
# ./gitea-image-prune.sh --repo felhom-hub --keep 10 --apply # really delete
|
|
# ./gitea-image-prune.sh --repo felhom-controller --older-than 90 --apply --reclaim
|
|
# ./gitea-image-prune.sh --all --keep 15 --apply --yes --reclaim --measure # cron-friendly
|
|
# ./gitea-image-prune.sh --repo felhom-hub reclaim # run cleanup cron only
|
|
#
|
|
# SET-AND-FORGET COMPLEMENT: configure a native Gitea cleanup rule per owner
|
|
# (package settings -> Cleanup Rules: keep most-recent N, exclude ^latest$);
|
|
# the daily cleanup_packages cron then enforces it. See README. This script
|
|
# does NOT auto-create rules — it is the on-demand tool.
|
|
# =============================================================================
|
|
set -euo pipefail
|
|
|
|
# --- Configuration --------------------------------------------------------
|
|
GITEA_URL="${GITEA_URL:-https://gitea.dooplex.hu}"
|
|
OWNER="${GITEA_OWNER:-admin}"
|
|
CLEANUP_CRON="cleanup_packages"
|
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
|
|
# --- Colors / log helpers (match build-felhom-hub.sh) ---------------------
|
|
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
|
|
info() { echo -e "${GREEN}[INFO]${NC} $*"; }
|
|
warn() { echo -e "${YELLOW}[WARN]${NC} $*" >&2; }
|
|
error() { echo -e "${RED}[ERROR]${NC} $*" >&2; }
|
|
step() { echo -e "${CYAN}[STEP]${NC} $*"; }
|
|
note() { echo -e "${CYAN} $*${NC}"; }
|
|
|
|
# --- Defaults / arg state -------------------------------------------------
|
|
declare -a REPOS=()
|
|
declare -a PROTECT=('^latest$') # always-protected tags; --protect appends
|
|
ALL=false
|
|
ACTION="" # list | prune | reclaim (positional or inferred)
|
|
KEEP=""
|
|
OLDER_THAN=""
|
|
APPLY=false # dry-run is the default
|
|
ASSUME_YES=false
|
|
DO_RECLAIM=false # run cleanup cron after an --apply prune
|
|
MEASURE=false
|
|
NO_SIZES=false
|
|
TOKEN_FILE=""
|
|
LOG_FILE=""
|
|
GITEA_TOKEN="${GITEA_TOKEN:-}"
|
|
|
|
# --- Temp workspace -------------------------------------------------------
|
|
TMP="$(mktemp -d)"
|
|
cleanup() { rm -rf "$TMP"; }
|
|
trap cleanup EXIT
|
|
|
|
usage() {
|
|
sed -n '2,/^set -euo/p' "$0" | sed 's/^# \{0,1\}//; s/^#//' | sed '$d'
|
|
exit "${1:-0}"
|
|
}
|
|
|
|
# =============================================================================
|
|
# Argument parsing
|
|
# =============================================================================
|
|
while [[ $# -gt 0 ]]; do
|
|
case "$1" in
|
|
list|prune|reclaim) ACTION="$1"; shift ;;
|
|
--repo) REPOS+=("$2"); shift 2 ;;
|
|
--all) ALL=true; shift ;;
|
|
--keep) KEEP="$2"; shift 2 ;;
|
|
--older-than) OLDER_THAN="$2"; shift 2 ;;
|
|
--dry-run) APPLY=false; shift ;;
|
|
--apply) APPLY=true; shift ;;
|
|
--yes|-y) ASSUME_YES=true; shift ;;
|
|
--reclaim) DO_RECLAIM=true; shift ;;
|
|
--measure) MEASURE=true; shift ;;
|
|
--no-sizes) NO_SIZES=true; shift ;;
|
|
--protect) PROTECT+=("$2"); shift 2 ;;
|
|
--token-file) TOKEN_FILE="$2"; shift 2 ;;
|
|
--owner) OWNER="$2"; shift 2 ;;
|
|
--log) LOG_FILE="$2"; shift 2 ;;
|
|
-h|--help) usage 0 ;;
|
|
*) error "Unknown argument: $1"; usage 1 ;;
|
|
esac
|
|
done
|
|
|
|
# --- Validate flag combinations -------------------------------------------
|
|
if [[ -n "$KEEP" && -n "$OLDER_THAN" ]]; then
|
|
error "--keep and --older-than are mutually exclusive (pick one prune mode)."
|
|
exit 2
|
|
fi
|
|
if [[ -n "$KEEP" && ! "$KEEP" =~ ^[0-9]+$ ]]; then error "--keep must be a non-negative integer"; exit 2; fi
|
|
if [[ -n "$OLDER_THAN" && ! "$OLDER_THAN" =~ ^[0-9]+$ ]]; then error "--older-than must be a non-negative integer (days)"; exit 2; fi
|
|
|
|
# =============================================================================
|
|
# Preflight
|
|
# =============================================================================
|
|
for bin in curl jq; do
|
|
command -v "$bin" &>/dev/null || { error "Required tool not found: $bin"; exit 1; }
|
|
done
|
|
|
|
# Token: --token-file beats env. Never printed.
|
|
GITEA_USER="${GITEA_USER:-}" # set when credentials come with a username (-> Basic auth)
|
|
CRED_SRC="GITEA_TOKEN env"
|
|
|
|
if [[ -n "$TOKEN_FILE" ]]; then
|
|
[[ -r "$TOKEN_FILE" ]] || { error "--token-file not readable: $TOKEN_FILE"; exit 1; }
|
|
GITEA_TOKEN="$(tr -d ' \t\r\n' < "$TOKEN_FILE")"
|
|
CRED_SRC="--token-file"
|
|
fi
|
|
|
|
# Auto-discover from git when no explicit token: reuse the credential git already
|
|
# has for the Gitea host (embedded remote URL, or a configured credential helper).
|
|
# Convenient, but a *git* token may lack package/admin scopes — a 403 will say so.
|
|
discover_git_credential() {
|
|
command -v git &>/dev/null || return 1
|
|
git rev-parse --is-inside-work-tree &>/dev/null || return 1
|
|
local host="${GITEA_URL#*://}"; host="${host%%/*}"
|
|
# (1) credentials embedded in the remote URL (https://user:token@host/...)
|
|
local url; url="$(git remote get-url origin 2>/dev/null || true)"
|
|
if [[ "$url" == *"$host"* && "$url" =~ ^https?://([^:/@]+):([^@/]+)@ ]]; then
|
|
GITEA_USER="${BASH_REMATCH[1]}"; GITEA_TOKEN="${BASH_REMATCH[2]}"; CRED_SRC="git remote URL"; return 0
|
|
fi
|
|
# (2) a configured credential helper (store / cache / manager). Never prompt.
|
|
if git config --get credential.helper &>/dev/null; then
|
|
local out user pass
|
|
out="$(printf 'protocol=https\nhost=%s\n\n' "$host" | GIT_TERMINAL_PROMPT=0 git credential fill 2>/dev/null || true)"
|
|
pass="$(printf '%s\n' "$out" | sed -n 's/^password=//p' | head -1)"
|
|
user="$(printf '%s\n' "$out" | sed -n 's/^username=//p' | head -1)"
|
|
if [[ -n "$pass" ]]; then GITEA_TOKEN="$pass"; GITEA_USER="${user:-$OWNER}"; CRED_SRC="git credential helper"; return 0; fi
|
|
fi
|
|
return 1
|
|
}
|
|
|
|
if [[ -z "$GITEA_TOKEN" ]]; then
|
|
discover_git_credential || true
|
|
fi
|
|
|
|
if [[ -z "$GITEA_TOKEN" ]]; then
|
|
error "No credentials. Set GITEA_TOKEN env, pass --token-file <path>, or run inside"
|
|
error "a clone of a ${GITEA_URL#*://} repo whose git credentials are configured."
|
|
error "Needs a site-admin token with read/write:package + read/write:admin scope."
|
|
exit 1
|
|
fi
|
|
|
|
# Auth method: Basic (user:token) when a username is known — works for both API
|
|
# tokens and passwords; bare 'token' header otherwise. OCI /v2/ always uses Basic.
|
|
declare -a AUTH
|
|
if [[ -n "$GITEA_USER" ]]; then AUTH=(-u "${GITEA_USER}:${GITEA_TOKEN}"); else AUTH=(-H "Authorization: token ${GITEA_TOKEN}"); fi
|
|
OCI_USER="${GITEA_USER:-$OWNER}"
|
|
|
|
# Default audit log
|
|
if [[ -z "$LOG_FILE" ]]; then
|
|
LOG_FILE="${SCRIPT_DIR}/logs/gitea-prune-$(date +%Y-%m-%d).log"
|
|
fi
|
|
mkdir -p "$(dirname "$LOG_FILE")"
|
|
|
|
# Redact the token from anything we print (belt-and-suspenders vs set -x etc.)
|
|
redact() { sed "s|${GITEA_TOKEN}|***TOKEN***|g"; }
|
|
|
|
audit() { # free-form line -> audit log (token-free by construction)
|
|
printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >> "$LOG_FILE"
|
|
}
|
|
|
|
# =============================================================================
|
|
# HTTP helpers — every call checks status; non-2xx is never silent.
|
|
# =============================================================================
|
|
# api_get <path> -> body on stdout; dies on non-2xx
|
|
# api_delete <path> -> echoes HTTP code; returns non-zero on non-204
|
|
# api_post <path> -> echoes HTTP code; dies on non-2xx
|
|
# oci_get <name> <ref> -> OCI manifest JSON (Basic auth); empty + rc1 on fail
|
|
api_get() {
|
|
local path="$1" out code
|
|
out="$(mktemp)"
|
|
code="$(curl -sS "${AUTH[@]}" \
|
|
-o "$out" -w '%{http_code}' "${GITEA_URL}${path}" || true)"
|
|
if [[ ! "$code" =~ ^2[0-9][0-9]$ ]]; then
|
|
error "GET ${path} -> HTTP ${code}"
|
|
head -c 400 "$out" | redact >&2; echo >&2
|
|
rm -f "$out"; return 1
|
|
fi
|
|
cat "$out"; rm -f "$out"
|
|
}
|
|
|
|
api_delete() {
|
|
local path="$1" code
|
|
code="$(curl -sS -X DELETE "${AUTH[@]}" \
|
|
-o "$TMP/del.body" -w '%{http_code}' "${GITEA_URL}${path}" || true)"
|
|
echo "$code"
|
|
[[ "$code" == "204" ]]
|
|
}
|
|
|
|
api_post() {
|
|
local path="$1" code
|
|
code="$(curl -sS -X POST "${AUTH[@]}" \
|
|
-o "$TMP/post.body" -w '%{http_code}' "${GITEA_URL}${path}" || true)"
|
|
if [[ ! "$code" =~ ^2[0-9][0-9]$ ]]; then
|
|
error "POST ${path} -> HTTP ${code}"
|
|
head -c 400 "$TMP/post.body" | redact >&2; echo >&2
|
|
return 1
|
|
fi
|
|
echo "$code"
|
|
}
|
|
|
|
oci_get() {
|
|
local name="$1" ref="$2" out code
|
|
out="$(mktemp)"
|
|
code="$(curl -sS -u "${OCI_USER}:${GITEA_TOKEN}" \
|
|
-H 'Accept: application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json' \
|
|
-o "$out" -w '%{http_code}' "${GITEA_URL}/v2/${OWNER}/${name}/manifests/${ref}" || true)"
|
|
if [[ ! "$code" =~ ^2[0-9][0-9]$ ]]; then rm -f "$out"; return 1; fi
|
|
cat "$out"; rm -f "$out"
|
|
}
|
|
|
|
# =============================================================================
|
|
# Data load — fetch ALL container versions for OWNER once (paginated), cache it.
|
|
# =============================================================================
|
|
VERSIONS_JSON="$TMP/versions.json" # JSON-lines, one version object per line
|
|
|
|
load_versions() {
|
|
step "Fetching container packages for owner '${OWNER}' from ${GITEA_URL} ..."
|
|
: > "$VERSIONS_JSON"
|
|
local page=1 limit=50 body n total=0
|
|
while :; do
|
|
body="$(api_get "/api/v1/packages/${OWNER}?type=container&page=${page}&limit=${limit}")" || exit 1
|
|
n="$(echo "$body" | jq 'length')"
|
|
echo "$body" | jq -c '.[]' >> "$VERSIONS_JSON"
|
|
total=$((total + n))
|
|
[[ "$n" -lt "$limit" ]] && break
|
|
page=$((page + 1))
|
|
done
|
|
info "Loaded ${total} version records across $(jq -rs '[.[].name]|unique|length' "$VERSIONS_JSON") package(s)."
|
|
}
|
|
|
|
# Distinct package names (sorted)
|
|
package_names() { jq -rs '[.[].name]|unique|.[]' "$VERSIONS_JSON"; }
|
|
|
|
# Tagged versions of a package, newest first: "tag<TAB>created_at"
|
|
tagged_versions() {
|
|
local name="$1"
|
|
jq -rs --arg n "$name" '
|
|
[ .[] | select(.name==$n and (.version|startswith("sha256:")|not)) ]
|
|
| sort_by(.created_at) | reverse
|
|
| .[] | "\(.version)\t\(.created_at)"' "$VERSIONS_JSON"
|
|
}
|
|
|
|
# Count of digest (sha256:) versions of a package
|
|
digest_count() {
|
|
local name="$1"
|
|
jq -rs --arg n "$name" '[ .[] | select(.name==$n and (.version|startswith("sha256:"))) ] | length' "$VERSIONS_JSON"
|
|
}
|
|
|
|
# =============================================================================
|
|
# Size resolution (best-effort, OCI). Per-tag "apparent" image size in bytes.
|
|
# tag -> OCI index -> referenced manifest digests -> files-API byte sums.
|
|
# Caches digest->bytes so repeated/shared digests aren't refetched.
|
|
# =============================================================================
|
|
declare -A DIGEST_BYTES_CACHE=()
|
|
|
|
digest_files_bytes() { # <name> <sha256:...> -> bytes (files-API sum)
|
|
local name="$1" digest="$2"
|
|
local key="${name}@${digest}"
|
|
if [[ -n "${DIGEST_BYTES_CACHE[$key]:-}" ]]; then echo "${DIGEST_BYTES_CACHE[$key]}"; return; fi
|
|
local body sum
|
|
body="$(api_get "/api/v1/packages/${OWNER}/container/${name}/${digest}/files" 2>/dev/null || echo '[]')"
|
|
sum="$(echo "$body" | jq '[.[].size] | add // 0' 2>/dev/null || echo 0)"
|
|
DIGEST_BYTES_CACHE[$key]="$sum"
|
|
echo "$sum"
|
|
}
|
|
|
|
resolve_tag_bytes() { # <name> <tag> -> apparent image bytes (0 if unresolved)
|
|
local name="$1" tag="$2" man total=0 d
|
|
man="$(oci_get "$name" "$tag")" || { echo 0; return; }
|
|
if echo "$man" | jq -e '.manifests' >/dev/null 2>&1; then
|
|
# OCI index / manifest list: sum referenced manifest digest versions
|
|
while IFS= read -r d; do
|
|
[[ -z "$d" ]] && continue
|
|
total=$(( total + $(digest_files_bytes "$name" "$d") ))
|
|
done < <(echo "$man" | jq -r '.manifests[].digest')
|
|
else
|
|
# Single image manifest: config + layers from the manifest itself
|
|
total="$(echo "$man" | jq '((.config.size // 0) + ([.layers[].size] | add // 0))')"
|
|
fi
|
|
echo "$total"
|
|
}
|
|
|
|
human() { # bytes -> human readable
|
|
local b="${1:-0}"
|
|
if command -v numfmt &>/dev/null; then numfmt --to=iec --suffix=B "$b" 2>/dev/null || echo "${b}B";
|
|
else echo "${b}B"; fi
|
|
}
|
|
|
|
is_protected() { # <tag> -> 0 if protected
|
|
local tag="$1" rx
|
|
for rx in "${PROTECT[@]}"; do [[ "$tag" =~ $rx ]] && return 0; done
|
|
return 1
|
|
}
|
|
|
|
# =============================================================================
|
|
# Orphan-manifest detection (drives reclaim).
|
|
# A "sha256:" manifest version is an ORPHAN if no SURVIVING tag's index
|
|
# references it. Deleting orphans makes their unique blobs GC-able.
|
|
# FAIL-CLOSED: if any tag fails to resolve (HTTP/format), we refuse to compute
|
|
# orphans for that package — never risk deleting a still-referenced manifest.
|
|
# =============================================================================
|
|
# referenced_digests <name> <outfile> -> rc0 (outfile = digests referenced by
|
|
# surviving tags), rc1 if unsafe to proceed
|
|
referenced_digests() {
|
|
local name="$1" out="$2" tag created man
|
|
: > "$out"
|
|
while IFS=$'\t' read -r tag created; do
|
|
[[ -z "$tag" ]] && continue
|
|
man="$(oci_get "$name" "$tag")" || {
|
|
warn " cannot resolve tag '${tag}' of ${name} (HTTP/format) — skipping orphan cleanup (safety)"; return 1; }
|
|
if echo "$man" | jq -e 'has("manifests")' >/dev/null 2>&1; then
|
|
echo "$man" | jq -r '.manifests[].digest' >> "$out"
|
|
else
|
|
# Single-arch image manifest (no index). Our registry uses indexes;
|
|
# if we hit this, bail rather than guess which sha256 version is safe.
|
|
warn " tag '${tag}' of ${name} is not a manifest index — skipping orphan cleanup (safety)"; return 1
|
|
fi
|
|
done < <(tagged_versions "$name")
|
|
return 0
|
|
}
|
|
|
|
# find_orphans <name> -> prints orphan sha256: versions (one per line); rc1 if unsafe
|
|
find_orphans() {
|
|
local name="$1" ref="$TMP/ref.${name//\//_}" sha="$TMP/sha.${name//\//_}"
|
|
referenced_digests "$name" "$ref" || return 1
|
|
LC_ALL=C sort -u "$ref" -o "$ref"
|
|
jq -rs --arg n "$name" \
|
|
'.[] | select(.name==$n and (.version|startswith("sha256:"))) | .version' \
|
|
"$VERSIONS_JSON" | LC_ALL=C sort -u > "$sha"
|
|
LC_ALL=C comm -23 "$sha" "$ref"
|
|
}
|
|
|
|
# Quiet cron trigger (separate from api_post so we can message a 403 nicely).
|
|
# echoes "ok <code>" on 2xx, "denied" on 403, "err <code>" otherwise; rc reflects.
|
|
trigger_cleanup_cron() {
|
|
local code
|
|
code="$(curl -sS -X POST "${AUTH[@]}" \
|
|
-o "$TMP/cron.body" -w '%{http_code}' \
|
|
"${GITEA_URL}/api/v1/admin/cron/${CLEANUP_CRON}" || true)"
|
|
if [[ "$code" =~ ^2[0-9][0-9]$ ]]; then echo "ok ${code}"; return 0; fi
|
|
if [[ "$code" == "403" ]]; then echo "denied"; return 1; fi
|
|
echo "err ${code}"; return 1
|
|
}
|
|
|
|
# =============================================================================
|
|
# Disk measurement (best-effort; kubectl optional)
|
|
# =============================================================================
|
|
GITEA_NS="${GITEA_NS:-gitea-system}"
|
|
GITEA_PKG_PATH="/data/gitea/packages"
|
|
KUBECTL=""
|
|
detect_kubectl() {
|
|
if command -v kubectl &>/dev/null; then KUBECTL="kubectl";
|
|
elif sudo -n kubectl version --client &>/dev/null 2>&1; then KUBECTL="sudo kubectl";
|
|
elif command -v sudo &>/dev/null; then KUBECTL="sudo kubectl"; fi
|
|
}
|
|
gitea_pod() {
|
|
[[ -n "$KUBECTL" ]] || return 1
|
|
$KUBECTL get pods -n "$GITEA_NS" -o name 2>/dev/null | grep -i gitea | head -1 | sed 's|pod/||'
|
|
}
|
|
measure_bytes() { # echoes byte count of packages dir, or empty if unavailable
|
|
local pod
|
|
pod="$(gitea_pod)" || return 1
|
|
[[ -n "$pod" ]] || return 1
|
|
$KUBECTL exec -n "$GITEA_NS" "$pod" -c gitea -- du -sb "$GITEA_PKG_PATH" 2>/dev/null | awk '{print $1}'
|
|
}
|
|
|
|
# =============================================================================
|
|
# Actions
|
|
# =============================================================================
|
|
|
|
# ---- LIST ----------------------------------------------------------------
|
|
do_list() {
|
|
local name="$1" line tag created bytes total_bytes=0 count=0 prot
|
|
echo ""
|
|
echo -e "${BOLD}== ${name} ==${NC} (owner ${OWNER}; $(digest_count "$name") digest manifests)"
|
|
printf ' %-28s %-22s %-12s %s\n' "TAG" "UPLOADED" "APPARENT" "FLAG"
|
|
printf ' %-28s %-22s %-12s %s\n' "---" "--------" "--------" "----"
|
|
while IFS=$'\t' read -r tag created; do
|
|
[[ -z "$tag" ]] && continue
|
|
count=$((count + 1))
|
|
if $NO_SIZES; then bytes=0; else bytes="$(resolve_tag_bytes "$name" "$tag")"; fi
|
|
total_bytes=$((total_bytes + bytes))
|
|
prot=""; is_protected "$tag" && prot="PROTECTED"
|
|
printf ' %-28s %-22s %-12s %s\n' "$tag" "${created:0:19}" "$( $NO_SIZES && echo '-' || human "$bytes")" "$prot"
|
|
done < <(tagged_versions "$name")
|
|
echo " ----"
|
|
if $NO_SIZES; then
|
|
info " ${count} tagged version(s). (sizes skipped: --no-sizes)"
|
|
else
|
|
info " ${count} tagged version(s); apparent total ≈ $(human "$total_bytes")"
|
|
note " CAVEAT: apparent sizes count shared base layers once PER TAG, so they"
|
|
note " overlap heavily. Real reclaimed space is much less — measure with --measure."
|
|
fi
|
|
}
|
|
|
|
# ---- Compute prune plan: prints "DELETE\t<tag>\t<created>" / "KEEP..." ----
|
|
# Sets globals: PLAN_DELETE (array of tags), PLAN_KEEP_N, PLAN_PROT_N
|
|
declare -a PLAN_DELETE=()
|
|
PLAN_KEEP_N=0; PLAN_PROT_N=0; PLAN_TOTAL=0
|
|
compute_plan() {
|
|
local name="$1" tag created epoch now cutoff idx=0
|
|
PLAN_DELETE=(); PLAN_KEEP_N=0; PLAN_PROT_N=0; PLAN_TOTAL=0
|
|
now="$(date +%s)"
|
|
[[ -n "$OLDER_THAN" ]] && cutoff=$(( now - OLDER_THAN * 86400 ))
|
|
while IFS=$'\t' read -r tag created; do
|
|
[[ -z "$tag" ]] && continue
|
|
PLAN_TOTAL=$((PLAN_TOTAL + 1))
|
|
# Protected always wins.
|
|
if is_protected "$tag"; then PLAN_PROT_N=$((PLAN_PROT_N + 1)); continue; fi
|
|
if [[ -n "$KEEP" ]]; then
|
|
# tags arrive newest-first; keep the first KEEP non-protected... but
|
|
# protected tags don't consume a keep slot — count index over all tags.
|
|
idx=$((idx + 1))
|
|
if [[ "$idx" -le "$KEEP" ]]; then PLAN_KEEP_N=$((PLAN_KEEP_N + 1)); else PLAN_DELETE+=("$tag"); fi
|
|
elif [[ -n "$OLDER_THAN" ]]; then
|
|
epoch="$(date -d "$created" +%s 2>/dev/null || echo "$now")"
|
|
if [[ "$epoch" -lt "$cutoff" ]]; then PLAN_DELETE+=("$tag"); else PLAN_KEEP_N=$((PLAN_KEEP_N + 1)); fi
|
|
fi
|
|
done < <(tagged_versions "$name")
|
|
}
|
|
|
|
# ---- PRUNE ---------------------------------------------------------------
|
|
PRUNE_DELETED=0; PRUNE_FAILED=0
|
|
declare -a PRUNE_TOUCHED_REPOS=()
|
|
do_prune_repo() {
|
|
local name="$1" tag bytes code
|
|
compute_plan "$name"
|
|
|
|
echo ""
|
|
echo -e "${BOLD}== prune ${name} ==${NC} mode: $( [[ -n "$KEEP" ]] && echo "keep-last ${KEEP}" || echo "older-than ${OLDER_THAN}d" )"
|
|
if [[ "${#PLAN_DELETE[@]}" -eq 0 ]]; then
|
|
info " Nothing to prune (${PLAN_TOTAL} tags: ${PLAN_KEEP_N} kept, ${PLAN_PROT_N} protected)."
|
|
return
|
|
fi
|
|
echo " Would delete ${#PLAN_DELETE[@]} tag(s); keep ${PLAN_KEEP_N}; protect ${PLAN_PROT_N}:"
|
|
local plan_bytes=0 szlabel
|
|
for tag in "${PLAN_DELETE[@]}"; do
|
|
if $NO_SIZES; then bytes=0; szlabel=""; else bytes="$(resolve_tag_bytes "$name" "$tag")"; szlabel="≈ $(human "$bytes")"; fi
|
|
plan_bytes=$((plan_bytes + bytes))
|
|
printf ' %-28s %s\n' "$tag" "$szlabel"
|
|
done
|
|
$NO_SIZES || note " apparent total to delete ≈ $(human "$plan_bytes") (overlaps shared layers — real reclaim less)"
|
|
|
|
if ! $APPLY; then
|
|
warn " DRY-RUN — nothing deleted. Re-run with --apply to delete."
|
|
for tag in "${PLAN_DELETE[@]}"; do audit "DRY-RUN would-delete ${OWNER}/${name}:${tag}"; done
|
|
return
|
|
fi
|
|
|
|
# --- confirmation gate ---
|
|
if ! $ASSUME_YES; then
|
|
echo ""
|
|
warn " About to DELETE ${#PLAN_DELETE[@]} tag(s) from ${name}. This is destructive."
|
|
read -r -p " Type the package name '${name}' to confirm: " reply
|
|
if [[ "$reply" != "$name" ]]; then warn " Skipped ${name} (confirmation mismatch)."; return; fi
|
|
fi
|
|
|
|
for tag in "${PLAN_DELETE[@]}"; do
|
|
if $NO_SIZES; then bytes=0; else bytes="$(resolve_tag_bytes "$name" "$tag")"; fi
|
|
code="$(api_delete "/api/v1/packages/${OWNER}/container/${name}/${tag}")" && {
|
|
info " deleted ${name}:${tag} (HTTP ${code})"
|
|
audit "APPLIED deleted ${OWNER}/${name}:${tag} apparent=${bytes}B http=${code}"
|
|
PRUNE_DELETED=$((PRUNE_DELETED + 1))
|
|
} || {
|
|
error " FAILED ${name}:${tag} (HTTP ${code}) — continuing"
|
|
audit "FAILED delete ${OWNER}/${name}:${tag} http=${code}"
|
|
PRUNE_FAILED=$((PRUNE_FAILED + 1))
|
|
}
|
|
done
|
|
PRUNE_TOUCHED_REPOS+=("$name")
|
|
}
|
|
|
|
# ---- RECLAIM -------------------------------------------------------------
|
|
# Step 2 + 3 of the proven mechanism: delete orphaned manifest versions, then
|
|
# trigger (or defer to) the cleanup_packages GC cron. Honors --dry-run/--apply.
|
|
do_reclaim() {
|
|
step "Reclaim: find & delete orphaned (untagged, unreferenced) manifests, then GC blobs."
|
|
# Reload state — a preceding prune deleted tags, so the cached listing is stale.
|
|
load_versions
|
|
|
|
local before; before=""
|
|
if $MEASURE; then before="$(measure_bytes || true)"; [[ -n "$before" ]] && info " packages dir now: $(human "$before")"; fi
|
|
|
|
local name orphans norph d code total_orphans=0 deleted=0 failed=0 unsafe=0
|
|
for name in "${TARGETS[@]}"; do
|
|
if ! orphans="$(find_orphans "$name")"; then
|
|
unsafe=$((unsafe + 1)); continue # warning already emitted; skip this package
|
|
fi
|
|
norph="$(printf '%s\n' "$orphans" | grep -c . || true)"
|
|
if [[ "$norph" -eq 0 ]]; then info " ${name}: no orphaned manifests."; continue; fi
|
|
total_orphans=$((total_orphans + norph))
|
|
if ! $APPLY; then
|
|
info " ${name}: ${norph} orphaned manifest version(s) WOULD be deleted (dry-run):"
|
|
printf '%s\n' "$orphans" | sed 's/^/ /'
|
|
continue
|
|
fi
|
|
info " ${name}: deleting ${norph} orphaned manifest version(s)..."
|
|
while IFS= read -r d; do
|
|
[[ -z "$d" ]] && continue
|
|
code="$(api_delete "/api/v1/packages/${OWNER}/container/${name}/${d}")" && {
|
|
deleted=$((deleted + 1)); audit "RECLAIM deleted-orphan ${OWNER}/${name}/${d} http=${code}"
|
|
} || {
|
|
failed=$((failed + 1)); error " FAILED orphan ${name}/${d} (HTTP ${code})"
|
|
audit "RECLAIM FAILED-orphan ${OWNER}/${name}/${d} http=${code}"
|
|
}
|
|
done <<< "$orphans"
|
|
done
|
|
|
|
if ! $APPLY; then
|
|
echo ""
|
|
warn " DRY-RUN — ${total_orphans} orphaned manifest(s) shown, none deleted, cron not triggered."
|
|
warn " Re-run reclaim with --apply to delete them and free disk."
|
|
return 0
|
|
fi
|
|
|
|
info " Orphaned manifests: deleted=${deleted} failed=${failed}$( [[ $unsafe -gt 0 ]] && echo " (skipped ${unsafe} pkg for safety)")"
|
|
|
|
# --- Step 3: GC the now-unreferenced blobs ---
|
|
local cron_ran=false res
|
|
step " Triggering '${CLEANUP_CRON}' cron to GC now-unreferenced blobs..."
|
|
res="$(trigger_cleanup_cron)" && cron_ran=true || true
|
|
case "$res" in
|
|
ok\ *) info " cron '${CLEANUP_CRON}' triggered (HTTP ${res#ok }) — GC running."
|
|
audit "RECLAIM triggered cron ${CLEANUP_CRON} ${res}" ;;
|
|
denied) warn " Token lacks write:admin — cannot trigger the GC cron directly."
|
|
note " The orphaned manifests are deleted; their unique blobs are now unreferenced and"
|
|
note " WILL be freed by the daily '@midnight' ${CLEANUP_CRON} run, or immediately on the"
|
|
note " next Gitea restart (RUN_AT_START is enabled). No data is lost by waiting."
|
|
audit "RECLAIM cron-trigger-denied (no write:admin); GC deferred to schedule/restart" ;;
|
|
*) warn " cron trigger returned: ${res}"; audit "RECLAIM cron-trigger ${res}" ;;
|
|
esac
|
|
|
|
# Only a real GC run changes disk; measuring otherwise is misleading.
|
|
if $MEASURE && $cron_ran && [[ -n "$before" ]]; then
|
|
sleep 6
|
|
local after; after="$(measure_bytes || true)"
|
|
if [[ -n "$after" ]]; then
|
|
info " packages dir after: $(human "$after")"
|
|
info " real reclaimed ≈ $(human "$(( before - after ))") (du; Longhorn 'actual size' lags)"
|
|
audit "RECLAIM measured before=${before}B after=${after}B freed=$(( before - after ))B"
|
|
else
|
|
note " (could not re-measure — re-run reclaim --measure later)"
|
|
fi
|
|
elif $MEASURE && ! $cron_ran; then
|
|
note " (skipping after-measurement: GC cron did not run; measure after the scheduled run/restart)"
|
|
fi
|
|
}
|
|
|
|
# =============================================================================
|
|
# Interactive menu (when no --repo/--all and a TTY is present)
|
|
# =============================================================================
|
|
interactive_menu() {
|
|
load_versions
|
|
local -a names=(); local n
|
|
while IFS= read -r n; do names+=("$n"); done < <(package_names)
|
|
if [[ "${#names[@]}" -eq 0 ]]; then warn "No container packages found for owner '${OWNER}'."; exit 0; fi
|
|
|
|
echo ""
|
|
echo -e "${BOLD}Container packages under '${OWNER}':${NC}"
|
|
local i
|
|
for i in "${!names[@]}"; do
|
|
printf ' %2d) %-22s %s tags, %s digest manifests\n' \
|
|
"$((i+1))" "${names[$i]}" \
|
|
"$(tagged_versions "${names[$i]}" | grep -c . || true)" \
|
|
"$(digest_count "${names[$i]}")"
|
|
done
|
|
echo " a) all packages"
|
|
echo ""
|
|
read -r -p "Select package(s) [numbers/space-sep, or 'a' for all]: " sel
|
|
local -a chosen=()
|
|
if [[ "$sel" == "a" || "$sel" == "all" ]]; then chosen=("${names[@]}");
|
|
else
|
|
for tok in $sel; do
|
|
[[ "$tok" =~ ^[0-9]+$ ]] && [[ "$tok" -ge 1 ]] && [[ "$tok" -le "${#names[@]}" ]] && chosen+=("${names[$((tok-1))]}")
|
|
done
|
|
fi
|
|
[[ "${#chosen[@]}" -eq 0 ]] && { warn "Nothing selected."; exit 0; }
|
|
REPOS=("${chosen[@]}")
|
|
|
|
echo ""
|
|
echo "Action: 1) list 2) prune 3) reclaim (cleanup cron)"
|
|
read -r -p "Choose [1-3]: " act
|
|
case "$act" in
|
|
1) ACTION="list" ;;
|
|
2) ACTION="prune"
|
|
read -r -p "Keep how many most-recent tags? [10]: " KEEP; KEEP="${KEEP:-10}"
|
|
[[ "$KEEP" =~ ^[0-9]+$ ]] || { error "invalid number"; exit 2; }
|
|
read -r -p "Apply for real now? (dry-run otherwise) [y/N]: " ap
|
|
[[ "$ap" =~ ^[Yy]$ ]] && APPLY=true
|
|
read -r -p "Also reclaim disk (run cleanup cron) after? [y/N]: " rc
|
|
[[ "$rc" =~ ^[Yy]$ ]] && DO_RECLAIM=true
|
|
;;
|
|
3) ACTION="reclaim" ;;
|
|
*) error "invalid action"; exit 2 ;;
|
|
esac
|
|
}
|
|
|
|
# =============================================================================
|
|
# Main
|
|
# =============================================================================
|
|
echo ""
|
|
info "╔══════════════════════════════════════════╗"
|
|
info "║ Gitea container-image prune ║"
|
|
info "╚══════════════════════════════════════════╝"
|
|
info "Server: ${GITEA_URL} Owner: ${OWNER} Log: ${LOG_FILE}"
|
|
info "Auth: ${CRED_SRC}$( [[ -n "$GITEA_USER" ]] && echo " (user: ${GITEA_USER})")"
|
|
|
|
# Sanity: Gitea version (public, no auth)
|
|
GV="$(curl -fsS "${GITEA_URL}/api/v1/version" 2>/dev/null | jq -r '.version' 2>/dev/null || echo '?')"
|
|
info "Gitea version: ${GV}"
|
|
[[ "$GV" == 1.26.* ]] || warn "Tested against Gitea 1.26.2 — server reports '${GV}'. Verify API shapes."
|
|
|
|
if $MEASURE; then detect_kubectl; [[ -n "$KUBECTL" ]] || warn "kubectl not usable — --measure will be skipped."; fi
|
|
|
|
# --- Selection / mode -----------------------------------------------------
|
|
if ! $ALL && [[ "${#REPOS[@]}" -eq 0 ]]; then
|
|
if [[ -t 0 ]]; then
|
|
interactive_menu
|
|
else
|
|
error "No --repo/--all given and not a TTY (cannot show interactive menu)."
|
|
usage 1
|
|
fi
|
|
else
|
|
load_versions
|
|
fi
|
|
|
|
# Resolve repo set
|
|
declare -a TARGETS=()
|
|
if $ALL; then
|
|
while IFS= read -r n; do TARGETS+=("$n"); done < <(package_names)
|
|
else
|
|
for r in "${REPOS[@]}"; do
|
|
if jq -rs --arg n "$r" 'any(.[]; .name==$n)' "$VERSIONS_JSON" | grep -q true; then
|
|
TARGETS+=("$r")
|
|
else
|
|
warn "Package not found for owner '${OWNER}': ${r} (skipping)"
|
|
fi
|
|
done
|
|
fi
|
|
[[ "${#TARGETS[@]}" -eq 0 ]] && { error "No valid target packages."; exit 1; }
|
|
|
|
# Infer action: keep/older-than => prune; else list. reclaim is always explicit.
|
|
if [[ -z "$ACTION" ]]; then
|
|
if [[ -n "$KEEP" || -n "$OLDER_THAN" ]]; then ACTION="prune"; else ACTION="list"; fi
|
|
fi
|
|
# prune needs a mode
|
|
if [[ "$ACTION" == "prune" && -z "$KEEP" && -z "$OLDER_THAN" ]]; then
|
|
error "prune needs --keep N or --older-than DAYS."; exit 2
|
|
fi
|
|
|
|
info "Action: ${ACTION} Targets: ${TARGETS[*]}"
|
|
audit "RUN action=${ACTION} owner=${OWNER} targets='${TARGETS[*]}' apply=${APPLY} keep='${KEEP}' older_than='${OLDER_THAN}' protect='${PROTECT[*]}'"
|
|
|
|
case "$ACTION" in
|
|
list)
|
|
for t in "${TARGETS[@]}"; do do_list "$t"; done
|
|
;;
|
|
prune)
|
|
# Strict confirmation for --all --apply
|
|
if $ALL && $APPLY && ! $ASSUME_YES; then
|
|
echo ""
|
|
warn "You are about to PRUNE ALL ${#TARGETS[@]} packages: ${TARGETS[*]}"
|
|
read -r -p "Type 'DELETE ALL' to confirm: " reply
|
|
[[ "$reply" == "DELETE ALL" ]] || { error "Aborted."; exit 1; }
|
|
ASSUME_YES=true # per-repo prompts already covered by this gate
|
|
fi
|
|
for t in "${TARGETS[@]}"; do do_prune_repo "$t"; done
|
|
echo ""
|
|
info "Prune summary: deleted=${PRUNE_DELETED} failed=${PRUNE_FAILED} (dry-run=$( $APPLY && echo no || echo yes ))"
|
|
audit "PRUNE-SUMMARY deleted=${PRUNE_DELETED} failed=${PRUNE_FAILED} apply=${APPLY}"
|
|
if $APPLY && $DO_RECLAIM && [[ "$PRUNE_DELETED" -gt 0 ]]; then
|
|
echo ""; do_reclaim || true
|
|
elif $APPLY && [[ "$PRUNE_DELETED" -gt 0 ]]; then
|
|
note "Tags deleted, but bytes remain until reclaim. Run: $0 reclaim --measure (or pass --reclaim)"
|
|
fi
|
|
;;
|
|
reclaim)
|
|
do_reclaim || exit 1
|
|
;;
|
|
esac
|
|
|
|
echo ""
|
|
info "Done. Audit log: ${LOG_FILE}"
|