admin
c9d5ed575c
Protected whatever --keep says: the controller floor, the vouched golden, the vouched agent and min_agent (the hub's Configuration page), every image of ours the vouched golden baked (its bake.log), the hub manifest's image. The dry-run prints each kept version and why. tests/test-prune-plan.sh pins it without network (red-proofed). No --apply was run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
938 lines
46 KiB
Bash
Executable File
938 lines
46 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# =============================================================================
|
|
# gitea-image-prune.sh — inspect & prune old container images and generic
|
|
# artifacts in the self-hosted Gitea registry, then
|
|
# reclaim disk.
|
|
# =============================================================================
|
|
# Target server : gitea.dooplex.hu (Gitea 1.26.2, owner namespace "admin")
|
|
# Best run on : build server 192.168.0.180 (has kubectl for --measure),
|
|
# but the core (list/prune/reclaim) is pure curl + jq and runs
|
|
# from any host that can reach Gitea and holds an admin token.
|
|
#
|
|
# -----------------------------------------------------------------------------
|
|
# WHY THIS EXISTS
|
|
# build-felhom-{hub,controller}.sh push :<version> AND :latest on every build,
|
|
# so felhom-controller (~96 tags) / felhom-hub (~42 tags) accumulate one image
|
|
# per build and the Gitea Longhorn PVC keeps filling.
|
|
#
|
|
# RETENTION RULE — Felhom `09-update-architecture.md` §3 decision 62 (operator ruling 2026-10-01, R-750)
|
|
# A prune keeps the newest 20 versions of each package (--keep defaults to 20 when
|
|
# pruning) PLUS every version in use, whatever --keep or --older-than says:
|
|
# - the vouched golden, the controller floor, the vouched agent and its min_agent
|
|
# (read from the hub's operator Configuration page — Basic auth, HUB_PW);
|
|
# - every gitea.dooplex.hu/admin/<pkg>:<tag> the vouched golden BAKED (its bake.log
|
|
# in felhom.eu/documentation/tests/golden-<version>-*/);
|
|
# - the hub image the GitOps manifest runs (felhom.eu/manifests/hub.yaml).
|
|
# If any of those cannot be read, a prune REFUSES (exit 3) — never "protect nothing".
|
|
# --apply is a person's act: nothing schedules this script (no cron, no timer).
|
|
# The August 2026 run (`--all --keep 7 --apply`, 2026-08-22, HM-024) predates this rule.
|
|
#
|
|
# PACKAGE TYPES — --type, default "container" (added 2026-08-23)
|
|
# Gitea namespaces packages by TYPE, and every API path embeds it. This script
|
|
# handled only type=container, so --all meant "all *container* packages" and
|
|
# type=generic artifacts were invisible to every mode. That is how
|
|
# admin/felhom-golden (22 versions x ~627 MB = 13 GB) grew to 88% of the Gitea
|
|
# PVC while --all --keep 7 --apply reported success: it pruned ~1.3 GB of
|
|
# container images and never saw the 13 GB sitting next to them.
|
|
# One type per run. Sweep both (dry-run first; --apply is a person's act):
|
|
# ./gitea-image-prune.sh --all prune # containers, keep 20 + in use
|
|
# ./gitea-image-prune.sh --type generic --all prune # generic, keep 20 + in use
|
|
#
|
|
# HOW GITEA STORES GENERIC PACKAGES (contrast with containers, below)
|
|
# No index, no manifest indirection, no shared layers: a version IS its files,
|
|
# and its blobs are unique to it. So reclaim has only TWO steps — delete the
|
|
# version, then let cleanup_packages GC the blobs. There are never orphaned
|
|
# manifests to hunt, and "apparent" size equals real reclaimed size (the
|
|
# shared-layer caveat that applies to containers does not apply here).
|
|
#
|
|
# HOW GITEA STORES CONTAINER IMAGES (load-bearing — drives the reclaim design)
|
|
# A pushed tag is an OCI image *index* (a ~850 B pointer). The real bytes live
|
|
# in untagged "sha256:..." manifest *versions* (config + layer blobs, ~9-25 MB
|
|
# each), whose layer blobs are content-addressed and SHARED across tags.
|
|
# Deleting a tag removes only the tiny index pointer — the manifest versions it
|
|
# referenced linger as untagged versions, and their blobs stay referenced.
|
|
#
|
|
# PROVEN RECLAIM MECHANISM (verified live on this instance, 2026-06-17 — see §3
|
|
# of the task / REPORT.md). It is THREE steps, not two:
|
|
# 1. DELETE the tag(s) -> frees ~nothing (only the index ptr)
|
|
# 2. DELETE the now-ORPHANED "sha256:" manifest versions (referenced by no
|
|
# surviving tag) -> still frees nothing on its own, BUT
|
|
# makes their unique blobs unreferenced. (Default cleanup_packages does
|
|
# NOT remove untagged manifests — only a cleanup *rule* would — so the
|
|
# script must delete the orphaned manifests itself. write:package scope.)
|
|
# 3. cleanup_packages cron runs -> GCs unreferenced blobs created
|
|
# >OLDER_THAN (24h default) ago -> THIS frees disk. Shared base layers
|
|
# still referenced by surviving tags are correctly retained.
|
|
# Live proof: deleting one 9.4 MB-apparent tag + its 2 manifests + GC freed
|
|
# 5.1 MiB (the rest was shared base layers, correctly kept).
|
|
# => "prune" deletes tags; "reclaim" deletes the orphaned manifests and then
|
|
# triggers (or defers to) the cleanup_packages cron.
|
|
#
|
|
# TRIGGERING THE GC CRON: POST /api/v1/admin/cron/cleanup_packages needs
|
|
# write:admin. If the token lacks it, the orphaned manifests are still deleted
|
|
# and their blobs are freed by the daily "@midnight" run (or on the next Gitea
|
|
# restart — RUN_AT_START was enabled in app.ini on 2026-06-17).
|
|
#
|
|
# CREDENTIALS (in order: GITEA_TOKEN env -> --token-file -> git's stored
|
|
# credential for the Gitea host: the remote URL's embedded token, else a
|
|
# configured credential helper). Auto-discovery lets you just run the script
|
|
# inside a clone with no token fuss — but a *git* credential may only have repo
|
|
# scope; if so, a 403 will name the missing package/admin scope.
|
|
#
|
|
# REQUIRED TOKEN (env GITEA_TOKEN, or --token-file). Must belong to a Gitea
|
|
# site-admin user. Minimal fine-grained scopes (Gitea 1.26):
|
|
# read:package — list packages / versions / files (list, prune planning)
|
|
# write:package — delete a version (prune --apply)
|
|
# read:admin — list cron tasks
|
|
# write:admin — run the cleanup_packages cron (reclaim)
|
|
# The project read-only token is insufficient (no package scope).
|
|
# The token is NEVER echoed, logged, or committed.
|
|
#
|
|
# USAGE
|
|
# GITEA_TOKEN=... ./gitea-image-prune.sh # interactive menu
|
|
# ./gitea-image-prune.sh --repo felhom-hub list # list one package
|
|
# ./gitea-image-prune.sh --all list # list all packages
|
|
# ./gitea-image-prune.sh --repo felhom-hub --keep 10 # dry-run prune (default)
|
|
# ./gitea-image-prune.sh --repo felhom-hub --keep 10 --apply # really delete
|
|
# ./gitea-image-prune.sh --repo felhom-controller --older-than 90 --apply --reclaim
|
|
# ./gitea-image-prune.sh --all prune # dry-run: keep 20 + in use
|
|
# ./gitea-image-prune.sh --repo felhom-hub reclaim # run cleanup cron only
|
|
# ./gitea-image-prune.sh --type generic --all list # the OTHER half
|
|
# ./gitea-image-prune.sh --type generic --repo felhom-golden --keep 3 --apply --reclaim
|
|
#
|
|
# NO SET-AND-FORGET: a native Gitea cleanup rule cannot know which versions are IN USE (the
|
|
# vouched golden, the floor, the vouched agent), so decision 62 rules it out — this on-demand
|
|
# script, run by a person, is the only pruner. (None exists today: package_cleanup_rule is empty.)
|
|
# =============================================================================
|
|
set -euo pipefail
|
|
|
|
# --- Configuration --------------------------------------------------------
|
|
GITEA_URL="${GITEA_URL:-https://gitea.dooplex.hu}"
|
|
OWNER="${GITEA_OWNER:-admin}"
|
|
CLEANUP_CRON="cleanup_packages"
|
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
|
|
|
# --- Colors / log helpers (match build-felhom-hub.sh) ---------------------
|
|
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; BOLD='\033[1m'; NC='\033[0m'
|
|
info() { echo -e "${GREEN}[INFO]${NC} $*"; }
|
|
warn() { echo -e "${YELLOW}[WARN]${NC} $*" >&2; }
|
|
error() { echo -e "${RED}[ERROR]${NC} $*" >&2; }
|
|
step() { echo -e "${CYAN}[STEP]${NC} $*"; }
|
|
note() { echo -e "${CYAN} $*${NC}"; }
|
|
|
|
# --- Defaults / arg state -------------------------------------------------
|
|
declare -a REPOS=()
|
|
declare -a PROTECT=('^latest$') # always-protected tags; --protect appends
|
|
ALL=false
|
|
PKG_TYPE="container" # Gitea package type; --type. Every API path embeds it.
|
|
ACTION="" # list | prune | reclaim (positional or inferred)
|
|
KEEP=""
|
|
OLDER_THAN=""
|
|
APPLY=false # dry-run is the default
|
|
ASSUME_YES=false
|
|
DO_RECLAIM=false # run cleanup cron after an --apply prune
|
|
MEASURE=false
|
|
NO_SIZES=false
|
|
TOKEN_FILE=""
|
|
LOG_FILE=""
|
|
GITEA_TOKEN="${GITEA_TOKEN:-}"
|
|
DEFAULT_KEEP=20 # decision 62: the newest 20 of each package
|
|
FELHOM_EU="${FELHOM_EU:-/mnt/5_hdd/felhom.eu/git/felhom.eu}" # the vouch records' checkout
|
|
HUB_URL="${HUB_URL:-}" # the hub (default: its ClusterIP via kubectl)
|
|
declare -A VOUCH_PROTECT=() # "<package>:<tag>" -> why it is in use
|
|
# Test seams (tests/test-prune-plan.sh): a versions fixture instead of the API, a protect
|
|
# list instead of the hub. Never set in normal use.
|
|
PRUNE_TEST_VERSIONS="${PRUNE_TEST_VERSIONS:-}"
|
|
PRUNE_TEST_PROTECT="${PRUNE_TEST_PROTECT:-}"
|
|
|
|
# --- Temp workspace -------------------------------------------------------
|
|
TMP="$(mktemp -d)"
|
|
cleanup() { rm -rf "$TMP"; }
|
|
trap cleanup EXIT
|
|
|
|
usage() {
|
|
sed -n '2,/^set -euo/p' "$0" | sed 's/^# \{0,1\}//; s/^#//' | sed '$d'
|
|
exit "${1:-0}"
|
|
}
|
|
|
|
# =============================================================================
|
|
# Argument parsing
|
|
# =============================================================================
|
|
while [[ $# -gt 0 ]]; do
|
|
case "$1" in
|
|
list|prune|reclaim) ACTION="$1"; shift ;;
|
|
--repo) REPOS+=("$2"); shift 2 ;;
|
|
--all) ALL=true; shift ;;
|
|
--type) PKG_TYPE="$2"; shift 2 ;;
|
|
--keep) KEEP="$2"; shift 2 ;;
|
|
--older-than) OLDER_THAN="$2"; shift 2 ;;
|
|
--dry-run) APPLY=false; shift ;;
|
|
--apply) APPLY=true; shift ;;
|
|
--yes|-y) ASSUME_YES=true; shift ;;
|
|
--reclaim) DO_RECLAIM=true; shift ;;
|
|
--measure) MEASURE=true; shift ;;
|
|
--no-sizes) NO_SIZES=true; shift ;;
|
|
--protect) PROTECT+=("$2"); shift 2 ;;
|
|
--token-file) TOKEN_FILE="$2"; shift 2 ;;
|
|
--owner) OWNER="$2"; shift 2 ;;
|
|
--log) LOG_FILE="$2"; shift 2 ;;
|
|
-h|--help) usage 0 ;;
|
|
*) error "Unknown argument: $1"; usage 1 ;;
|
|
esac
|
|
done
|
|
|
|
# --- Validate flag combinations -------------------------------------------
|
|
if [[ -n "$KEEP" && -n "$OLDER_THAN" ]]; then
|
|
error "--keep and --older-than are mutually exclusive (pick one prune mode)."
|
|
exit 2
|
|
fi
|
|
if [[ -n "$KEEP" && ! "$KEEP" =~ ^[0-9]+$ ]]; then error "--keep must be a non-negative integer"; exit 2; fi
|
|
if [[ -n "$OLDER_THAN" && ! "$OLDER_THAN" =~ ^[0-9]+$ ]]; then error "--older-than must be a non-negative integer (days)"; exit 2; fi
|
|
if [[ ! "$PKG_TYPE" =~ ^[a-z_]+$ ]]; then error "--type must be a Gitea package type slug (e.g. container, generic)"; exit 2; fi
|
|
|
|
# Only container carries the OCI index -> manifest -> blob indirection. Every
|
|
# other Gitea type stores a version's files directly, so it takes the simple
|
|
# path: delete the version, GC the blobs. Both are implemented; anything beyond
|
|
# container/generic is untested here but structurally identical to generic.
|
|
IS_CONTAINER=false
|
|
[[ "$PKG_TYPE" == "container" ]] && IS_CONTAINER=true
|
|
case "$PKG_TYPE" in
|
|
container|generic) ;;
|
|
*) warn "--type '${PKG_TYPE}' is untested; treating it like 'generic' (no manifest indirection)." ;;
|
|
esac
|
|
|
|
# =============================================================================
|
|
# Preflight
|
|
# =============================================================================
|
|
for bin in curl jq; do
|
|
command -v "$bin" &>/dev/null || { error "Required tool not found: $bin"; exit 1; }
|
|
done
|
|
|
|
# Token: --token-file beats env. Never printed.
|
|
GITEA_USER="${GITEA_USER:-}" # set when credentials come with a username (-> Basic auth)
|
|
CRED_SRC="GITEA_TOKEN env"
|
|
|
|
if [[ -n "$TOKEN_FILE" ]]; then
|
|
[[ -r "$TOKEN_FILE" ]] || { error "--token-file not readable: $TOKEN_FILE"; exit 1; }
|
|
GITEA_TOKEN="$(tr -d ' \t\r\n' < "$TOKEN_FILE")"
|
|
CRED_SRC="--token-file"
|
|
fi
|
|
|
|
# Auto-discover from git when no explicit token: reuse the credential git already
|
|
# has for the Gitea host (embedded remote URL, or a configured credential helper).
|
|
# Convenient, but a *git* token may lack package/admin scopes — a 403 will say so.
|
|
discover_git_credential() {
|
|
command -v git &>/dev/null || return 1
|
|
git rev-parse --is-inside-work-tree &>/dev/null || return 1
|
|
local host="${GITEA_URL#*://}"; host="${host%%/*}"
|
|
# (1) credentials embedded in the remote URL (https://user:token@host/...)
|
|
local url; url="$(git remote get-url origin 2>/dev/null || true)"
|
|
if [[ "$url" == *"$host"* && "$url" =~ ^https?://([^:/@]+):([^@/]+)@ ]]; then
|
|
GITEA_USER="${BASH_REMATCH[1]}"; GITEA_TOKEN="${BASH_REMATCH[2]}"; CRED_SRC="git remote URL"; return 0
|
|
fi
|
|
# (2) a configured credential helper (store / cache / manager). Never prompt.
|
|
if git config --get credential.helper &>/dev/null; then
|
|
local out user pass
|
|
out="$(printf 'protocol=https\nhost=%s\n\n' "$host" | GIT_TERMINAL_PROMPT=0 git credential fill 2>/dev/null || true)"
|
|
pass="$(printf '%s\n' "$out" | sed -n 's/^password=//p' | head -1)"
|
|
user="$(printf '%s\n' "$out" | sed -n 's/^username=//p' | head -1)"
|
|
if [[ -n "$pass" ]]; then GITEA_TOKEN="$pass"; GITEA_USER="${user:-$OWNER}"; CRED_SRC="git credential helper"; return 0; fi
|
|
fi
|
|
return 1
|
|
}
|
|
|
|
if [[ -n "$PRUNE_TEST_VERSIONS" ]]; then
|
|
GITEA_TOKEN="test-no-network"; CRED_SRC="test fixture (no network)"
|
|
fi
|
|
if [[ -z "$GITEA_TOKEN" ]]; then
|
|
discover_git_credential || true
|
|
fi
|
|
|
|
if [[ -z "$GITEA_TOKEN" ]]; then
|
|
error "No credentials. Set GITEA_TOKEN env, pass --token-file <path>, or run inside"
|
|
error "a clone of a ${GITEA_URL#*://} repo whose git credentials are configured."
|
|
error "Needs a site-admin token with read/write:package + read/write:admin scope."
|
|
exit 1
|
|
fi
|
|
|
|
# Auth method: Basic (user:token) when a username is known — works for both API
|
|
# tokens and passwords; bare 'token' header otherwise. OCI /v2/ always uses Basic.
|
|
declare -a AUTH
|
|
if [[ -n "$GITEA_USER" ]]; then AUTH=(-u "${GITEA_USER}:${GITEA_TOKEN}"); else AUTH=(-H "Authorization: token ${GITEA_TOKEN}"); fi
|
|
OCI_USER="${GITEA_USER:-$OWNER}"
|
|
|
|
# Default audit log
|
|
if [[ -z "$LOG_FILE" ]]; then
|
|
LOG_FILE="${SCRIPT_DIR}/logs/gitea-prune-$(date +%Y-%m-%d).log"
|
|
fi
|
|
mkdir -p "$(dirname "$LOG_FILE")"
|
|
|
|
# Redact the token from anything we print (belt-and-suspenders vs set -x etc.)
|
|
redact() { sed "s|${GITEA_TOKEN}|***TOKEN***|g"; }
|
|
|
|
audit() { # free-form line -> audit log (token-free by construction)
|
|
printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >> "$LOG_FILE"
|
|
}
|
|
|
|
# =============================================================================
|
|
# HTTP helpers — every call checks status; non-2xx is never silent.
|
|
# =============================================================================
|
|
# api_get <path> -> body on stdout; dies on non-2xx
|
|
# api_delete <path> -> echoes HTTP code; returns non-zero on non-204
|
|
# api_post <path> -> echoes HTTP code; dies on non-2xx
|
|
# oci_get <name> <ref> -> OCI manifest JSON (Basic auth); empty + rc1 on fail
|
|
api_get() {
|
|
local path="$1" out code
|
|
out="$(mktemp)"
|
|
code="$(curl -sS "${AUTH[@]}" \
|
|
-o "$out" -w '%{http_code}' "${GITEA_URL}${path}" || true)"
|
|
if [[ ! "$code" =~ ^2[0-9][0-9]$ ]]; then
|
|
error "GET ${path} -> HTTP ${code}"
|
|
head -c 400 "$out" | redact >&2; echo >&2
|
|
rm -f "$out"; return 1
|
|
fi
|
|
cat "$out"; rm -f "$out"
|
|
}
|
|
|
|
api_delete() {
|
|
local path="$1" code
|
|
code="$(curl -sS -X DELETE "${AUTH[@]}" \
|
|
-o "$TMP/del.body" -w '%{http_code}' "${GITEA_URL}${path}" || true)"
|
|
echo "$code"
|
|
[[ "$code" == "204" ]]
|
|
}
|
|
|
|
api_post() {
|
|
local path="$1" code
|
|
code="$(curl -sS -X POST "${AUTH[@]}" \
|
|
-o "$TMP/post.body" -w '%{http_code}' "${GITEA_URL}${path}" || true)"
|
|
if [[ ! "$code" =~ ^2[0-9][0-9]$ ]]; then
|
|
error "POST ${path} -> HTTP ${code}"
|
|
head -c 400 "$TMP/post.body" | redact >&2; echo >&2
|
|
return 1
|
|
fi
|
|
echo "$code"
|
|
}
|
|
|
|
oci_get() {
|
|
local name="$1" ref="$2" out code
|
|
out="$(mktemp)"
|
|
code="$(curl -sS -u "${OCI_USER}:${GITEA_TOKEN}" \
|
|
-H 'Accept: application/vnd.oci.image.index.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json' \
|
|
-o "$out" -w '%{http_code}' "${GITEA_URL}/v2/${OWNER}/${name}/manifests/${ref}" || true)"
|
|
if [[ ! "$code" =~ ^2[0-9][0-9]$ ]]; then rm -f "$out"; return 1; fi
|
|
cat "$out"; rm -f "$out"
|
|
}
|
|
|
|
# =============================================================================
|
|
# Data load — fetch ALL versions of PKG_TYPE for OWNER once (paginated), cache it.
|
|
# NOTE: the type filter is the whole reason --all used to miss 13 GB. --all
|
|
# means "every package OF THIS TYPE", never "every package".
|
|
# =============================================================================
|
|
VERSIONS_JSON="$TMP/versions.json" # JSON-lines, one version object per line
|
|
|
|
load_versions() {
|
|
if [[ -n "$PRUNE_TEST_VERSIONS" ]]; then
|
|
jq -c '.[]' "$PRUNE_TEST_VERSIONS" > "$VERSIONS_JSON"
|
|
info "Loaded $(grep -c . "$VERSIONS_JSON") version records from the test fixture."
|
|
return
|
|
fi
|
|
step "Fetching ${PKG_TYPE} packages for owner '${OWNER}' from ${GITEA_URL} ..."
|
|
: > "$VERSIONS_JSON"
|
|
local page=1 limit=50 body n total=0
|
|
while :; do
|
|
body="$(api_get "/api/v1/packages/${OWNER}?type=${PKG_TYPE}&page=${page}&limit=${limit}")" || exit 1
|
|
n="$(echo "$body" | jq 'length')"
|
|
echo "$body" | jq -c '.[]' >> "$VERSIONS_JSON"
|
|
total=$((total + n))
|
|
[[ "$n" -lt "$limit" ]] && break
|
|
page=$((page + 1))
|
|
done
|
|
info "Loaded ${total} version records across $(jq -rs '[.[].name]|unique|length' "$VERSIONS_JSON") package(s)."
|
|
}
|
|
|
|
# Distinct package names (sorted)
|
|
package_names() { jq -rs '[.[].name]|unique|.[]' "$VERSIONS_JSON"; }
|
|
|
|
# Tagged versions of a package, newest first: "tag<TAB>created_at"
|
|
tagged_versions() {
|
|
local name="$1"
|
|
jq -rs --arg n "$name" '
|
|
[ .[] | select(.name==$n and (.version|startswith("sha256:")|not)) ]
|
|
| sort_by(.created_at) | reverse
|
|
| .[] | "\(.version)\t\(.created_at)"' "$VERSIONS_JSON"
|
|
}
|
|
|
|
# Count of digest (sha256:) versions of a package
|
|
digest_count() {
|
|
local name="$1"
|
|
jq -rs --arg n "$name" '[ .[] | select(.name==$n and (.version|startswith("sha256:"))) ] | length' "$VERSIONS_JSON"
|
|
}
|
|
|
|
# =============================================================================
|
|
# Size resolution (best-effort). Per-version "apparent" size in bytes.
|
|
# container: tag -> OCI index -> referenced manifest digests -> files-API sums.
|
|
# Caches digest->bytes so shared digests aren't refetched. Apparent
|
|
# size OVERSTATES reclaim, because base layers are shared.
|
|
# other: version -> files-API sum directly. Blobs are unique to the
|
|
# version, so apparent size IS the reclaimable size.
|
|
# =============================================================================
|
|
declare -A DIGEST_BYTES_CACHE=()
|
|
|
|
digest_files_bytes() { # <name> <sha256:...> -> bytes (files-API sum)
|
|
local name="$1" digest="$2"
|
|
local key="${name}@${digest}"
|
|
if [[ -n "${DIGEST_BYTES_CACHE[$key]:-}" ]]; then echo "${DIGEST_BYTES_CACHE[$key]}"; return; fi
|
|
local body sum
|
|
body="$(api_get "/api/v1/packages/${OWNER}/${PKG_TYPE}/${name}/${digest}/files" 2>/dev/null || echo '[]')"
|
|
sum="$(echo "$body" | jq '[.[].size] | add // 0' 2>/dev/null || echo 0)"
|
|
DIGEST_BYTES_CACHE[$key]="$sum"
|
|
echo "$sum"
|
|
}
|
|
|
|
resolve_tag_bytes() { # <name> <version> -> apparent bytes (0 if unresolved)
|
|
local name="$1" tag="$2" man total=0 d
|
|
# Non-container types have no manifest indirection: the version's files ARE
|
|
# its bytes. URL-encode nothing — Gitea version strings here are path-safe.
|
|
if ! $IS_CONTAINER; then digest_files_bytes "$name" "$tag"; return; fi
|
|
man="$(oci_get "$name" "$tag")" || { echo 0; return; }
|
|
if echo "$man" | jq -e '.manifests' >/dev/null 2>&1; then
|
|
# OCI index / manifest list: sum referenced manifest digest versions
|
|
while IFS= read -r d; do
|
|
[[ -z "$d" ]] && continue
|
|
total=$(( total + $(digest_files_bytes "$name" "$d") ))
|
|
done < <(echo "$man" | jq -r '.manifests[].digest')
|
|
else
|
|
# Single image manifest: config + layers from the manifest itself
|
|
total="$(echo "$man" | jq '((.config.size // 0) + ([.layers[].size] | add // 0))')"
|
|
fi
|
|
echo "$total"
|
|
}
|
|
|
|
human() { # bytes -> human readable
|
|
local b="${1:-0}"
|
|
if command -v numfmt &>/dev/null; then numfmt --to=iec --suffix=B "$b" 2>/dev/null || echo "${b}B";
|
|
else echo "${b}B"; fi
|
|
}
|
|
|
|
is_protected() { # <package> <tag> -> 0 if protected (a --protect regex, or a version in use)
|
|
local name="$1" tag="$2" rx
|
|
for rx in "${PROTECT[@]}"; do [[ "$tag" =~ $rx ]] && return 0; done
|
|
[[ -n "${VOUCH_PROTECT[${name}:${tag}]:-}" ]] && return 0
|
|
return 1
|
|
}
|
|
|
|
# protect_reason <package> <tag> -> why it is kept ("" if not protected)
|
|
protect_reason() {
|
|
local name="$1" tag="$2" rx
|
|
for rx in "${PROTECT[@]}"; do [[ "$tag" =~ $rx ]] && { echo "matches --protect ${rx}"; return; }; done
|
|
echo "${VOUCH_PROTECT[${name}:${tag}]:-}"
|
|
}
|
|
|
|
# =============================================================================
|
|
# The versions IN USE (decision 62). Fills VOUCH_PROTECT or returns non-zero — the caller
|
|
# then refuses to prune. Every value is checked to look like a version before it counts.
|
|
# =============================================================================
|
|
vouch_add() { # <package> <tag> <why>
|
|
VOUCH_PROTECT["$1:$2"]="${VOUCH_PROTECT["$1:$2"]:+${VOUCH_PROTECT["$1:$2"]}; }$3"
|
|
}
|
|
load_vouch_protect() {
|
|
if [[ -n "$PRUNE_TEST_PROTECT" ]]; then
|
|
local pkg tag why
|
|
while read -r pkg tag why; do [[ -n "$pkg" ]] && vouch_add "$pkg" "$tag" "$why"; done < "$PRUNE_TEST_PROTECT"
|
|
info "In-use list from the test fixture: ${#VOUCH_PROTECT[@]} version(s)."
|
|
return 0
|
|
fi
|
|
local hubpw="${HUB_PW:-}" page floor golden agent minagent bake hubtag
|
|
if [[ -z "$hubpw" ]]; then
|
|
local cred="${FELHOM_EU}/scripts/read_credential.py" f
|
|
f="$(mktemp)"
|
|
if [[ -r "$cred" ]] && python3 "$cred" HUB_PW "$f" >/dev/null 2>&1; then hubpw="$(cat "$f")"; fi
|
|
rm -f "$f"
|
|
fi
|
|
[[ -n "$hubpw" ]] || { error "In-use list: no HUB_PW (env or ~/.config/credentials) — cannot read the vouch."; return 1; }
|
|
if [[ -z "$HUB_URL" ]]; then
|
|
local ip; ip="$(sudo -n kubectl -n felhom-system get svc hub -o jsonpath='{.spec.clusterIP}' 2>/dev/null || true)"
|
|
[[ -n "$ip" ]] && HUB_URL="http://${ip}:8080"
|
|
fi
|
|
[[ -n "$HUB_URL" ]] || { error "In-use list: the hub's address is unknown (set HUB_URL)."; return 1; }
|
|
page="$(mktemp)"
|
|
# Basic auth through -u; NEVER -w '%{redirect_url}' (it prints the password, R-580).
|
|
curl -fsS --max-time 20 -u ":${hubpw}" -o "$page" "${HUB_URL}/configuration" 2>/dev/null \
|
|
|| { rm -f "$page"; error "In-use list: GET ${HUB_URL}/configuration failed."; return 1; }
|
|
hubpw=""
|
|
floor="$(grep -o 'name="min_controller_version" value="[^"]*"' "$page" | head -1 | sed 's/.*value="//; s/"$//')"
|
|
golden="$(tr '\n' ' ' < "$page" | grep -o '<select name="golden_version".*</select>' | grep -o '<option value="[^"]*"[^>]*selected' | head -1 | sed 's/<option value="//; s/".*//')"
|
|
agent="$(tr '\n' ' ' < "$page" | grep -o '<select name="agent_version".*' | grep -o '<option value="[^"]*"[^>]*selected' | head -1 | sed 's/<option value="//; s/".*//')"
|
|
minagent="$(grep -o 'name="min_agent" value="[^"]*"' "$page" | sed 's/.*value="//; s/"$//' | sort -u)"
|
|
rm -f "$page"
|
|
local vre='^[0-9]+\.[0-9]+\.[0-9]+$' v
|
|
for v in "$floor" "$golden" "$agent"; do
|
|
[[ "$v" =~ $vre ]] || { error "In-use list: the hub page did not give a version (floor='${floor}' golden='${golden}' agent='${agent}')."; return 1; }
|
|
done
|
|
vouch_add felhom-controller "$floor" "the controller floor"
|
|
vouch_add felhom-controller "$golden" "the vouched golden's controller"
|
|
vouch_add felhom-golden "$golden" "the vouched golden"
|
|
vouch_add felhom-agent "$agent" "the vouched agent"
|
|
for v in $minagent; do
|
|
[[ "$v" =~ $vre ]] || { error "In-use list: min_agent '${v}' is not a version."; return 1; }
|
|
vouch_add felhom-agent "$v" "min_agent"
|
|
done
|
|
bake="$(ls -d "${FELHOM_EU}"/documentation/tests/golden-"${golden}"-*/bake.log 2>/dev/null | head -1)"
|
|
[[ -r "$bake" ]] || { error "In-use list: no bake.log for golden ${golden} under ${FELHOM_EU}/documentation/tests/."; return 1; }
|
|
local ref pkg tag n=0
|
|
while read -r ref; do
|
|
pkg="${ref#gitea.dooplex.hu/${OWNER}/}"; tag="${pkg##*:}"; pkg="${pkg%%:*}"
|
|
vouch_add "$pkg" "$tag" "baked into golden ${golden}"; n=$((n + 1))
|
|
done < <(grep -o "gitea\.dooplex\.hu/${OWNER}/[a-z0-9-]*:[A-Za-z0-9._-]*" "$bake" | sort -u)
|
|
[[ "$n" -gt 0 ]] || { error "In-use list: golden ${golden}'s bake.log names no image of ours — refusing."; return 1; }
|
|
hubtag="$(grep -o "gitea\.dooplex\.hu/${OWNER}/felhom-hub:[A-Za-z0-9._-]*" "${FELHOM_EU}/manifests/hub.yaml" 2>/dev/null | head -1)"
|
|
[[ -n "$hubtag" ]] || { error "In-use list: no hub image in ${FELHOM_EU}/manifests/hub.yaml."; return 1; }
|
|
vouch_add felhom-hub "${hubtag##*:}" "the hub the GitOps manifest runs"
|
|
info "In-use list (decision 62): ${#VOUCH_PROTECT[@]} version(s) from the hub's vouch, golden ${golden}'s bake.log and the hub manifest."
|
|
return 0
|
|
}
|
|
|
|
# =============================================================================
|
|
# Orphan-manifest detection (drives reclaim).
|
|
# A "sha256:" manifest version is an ORPHAN if no SURVIVING tag's index
|
|
# references it. Deleting orphans makes their unique blobs GC-able.
|
|
# FAIL-CLOSED: if any tag fails to resolve (HTTP/format), we refuse to compute
|
|
# orphans for that package — never risk deleting a still-referenced manifest.
|
|
# =============================================================================
|
|
# referenced_digests <name> <outfile> -> rc0 (outfile = digests referenced by
|
|
# surviving tags), rc1 if unsafe to proceed
|
|
referenced_digests() {
|
|
local name="$1" out="$2" tag created man
|
|
: > "$out"
|
|
while IFS=$'\t' read -r tag created; do
|
|
[[ -z "$tag" ]] && continue
|
|
man="$(oci_get "$name" "$tag")" || {
|
|
warn " cannot resolve tag '${tag}' of ${name} (HTTP/format) — skipping orphan cleanup (safety)"; return 1; }
|
|
if echo "$man" | jq -e 'has("manifests")' >/dev/null 2>&1; then
|
|
echo "$man" | jq -r '.manifests[].digest' >> "$out"
|
|
else
|
|
# Single-arch image manifest (no index). Our registry uses indexes;
|
|
# if we hit this, bail rather than guess which sha256 version is safe.
|
|
warn " tag '${tag}' of ${name} is not a manifest index — skipping orphan cleanup (safety)"; return 1
|
|
fi
|
|
done < <(tagged_versions "$name")
|
|
return 0
|
|
}
|
|
|
|
# find_orphans <name> -> prints orphan sha256: versions (one per line); rc1 if unsafe
|
|
find_orphans() {
|
|
local name="$1" ref="$TMP/ref.${name//\//_}" sha="$TMP/sha.${name//\//_}"
|
|
# Only container has untagged manifest versions to orphan. Everything else
|
|
# frees its blobs the moment the version is deleted — nothing to hunt.
|
|
$IS_CONTAINER || return 0
|
|
referenced_digests "$name" "$ref" || return 1
|
|
LC_ALL=C sort -u "$ref" -o "$ref"
|
|
jq -rs --arg n "$name" \
|
|
'.[] | select(.name==$n and (.version|startswith("sha256:"))) | .version' \
|
|
"$VERSIONS_JSON" | LC_ALL=C sort -u > "$sha"
|
|
LC_ALL=C comm -23 "$sha" "$ref"
|
|
}
|
|
|
|
# Quiet cron trigger (separate from api_post so we can message a 403 nicely).
|
|
# echoes "ok <code>" on 2xx, "denied" on 403, "err <code>" otherwise; rc reflects.
|
|
trigger_cleanup_cron() {
|
|
local code
|
|
code="$(curl -sS -X POST "${AUTH[@]}" \
|
|
-o "$TMP/cron.body" -w '%{http_code}' \
|
|
"${GITEA_URL}/api/v1/admin/cron/${CLEANUP_CRON}" || true)"
|
|
if [[ "$code" =~ ^2[0-9][0-9]$ ]]; then echo "ok ${code}"; return 0; fi
|
|
if [[ "$code" == "403" ]]; then echo "denied"; return 1; fi
|
|
echo "err ${code}"; return 1
|
|
}
|
|
|
|
# =============================================================================
|
|
# Disk measurement (best-effort; kubectl optional)
|
|
# =============================================================================
|
|
GITEA_NS="${GITEA_NS:-gitea-system}"
|
|
GITEA_PKG_PATH="/data/gitea/packages"
|
|
KUBECTL=""
|
|
detect_kubectl() {
|
|
if command -v kubectl &>/dev/null; then KUBECTL="kubectl";
|
|
elif sudo -n kubectl version --client &>/dev/null 2>&1; then KUBECTL="sudo kubectl";
|
|
elif command -v sudo &>/dev/null; then KUBECTL="sudo kubectl"; fi
|
|
}
|
|
gitea_pod() {
|
|
[[ -n "$KUBECTL" ]] || return 1
|
|
$KUBECTL get pods -n "$GITEA_NS" -o name 2>/dev/null | grep -i gitea | head -1 | sed 's|pod/||'
|
|
}
|
|
measure_bytes() { # echoes byte count of packages dir, or empty if unavailable
|
|
local pod
|
|
pod="$(gitea_pod)" || return 1
|
|
[[ -n "$pod" ]] || return 1
|
|
$KUBECTL exec -n "$GITEA_NS" "$pod" -c gitea -- du -sb "$GITEA_PKG_PATH" 2>/dev/null | awk '{print $1}'
|
|
}
|
|
|
|
# =============================================================================
|
|
# Actions
|
|
# =============================================================================
|
|
|
|
# ---- LIST ----------------------------------------------------------------
|
|
do_list() {
|
|
local name="$1" line tag created bytes total_bytes=0 count=0 prot hdr
|
|
echo ""
|
|
if $IS_CONTAINER; then
|
|
echo -e "${BOLD}== ${name} ==${NC} (owner ${OWNER}; $(digest_count "$name") digest manifests)"
|
|
hdr="TAG"
|
|
else
|
|
echo -e "${BOLD}== ${name} ==${NC} (owner ${OWNER}; type ${PKG_TYPE})"
|
|
hdr="VERSION"
|
|
fi
|
|
printf ' %-28s %-22s %-12s %s\n' "$hdr" "UPLOADED" "APPARENT" "FLAG"
|
|
printf ' %-28s %-22s %-12s %s\n' "---" "--------" "--------" "----"
|
|
while IFS=$'\t' read -r tag created; do
|
|
[[ -z "$tag" ]] && continue
|
|
count=$((count + 1))
|
|
if $NO_SIZES; then bytes=0; else bytes="$(resolve_tag_bytes "$name" "$tag")"; fi
|
|
total_bytes=$((total_bytes + bytes))
|
|
prot=""; is_protected "$name" "$tag" && prot="PROTECTED ($(protect_reason "$name" "$tag"))"
|
|
printf ' %-28s %-22s %-12s %s\n' "$tag" "${created:0:19}" "$( $NO_SIZES && echo '-' || human "$bytes")" "$prot"
|
|
done < <(tagged_versions "$name")
|
|
echo " ----"
|
|
if $NO_SIZES; then
|
|
info " ${count} tagged version(s). (sizes skipped: --no-sizes)"
|
|
else
|
|
info " ${count} tagged version(s); apparent total ≈ $(human "$total_bytes")"
|
|
if $IS_CONTAINER; then
|
|
note " CAVEAT: apparent sizes count shared base layers once PER TAG, so they"
|
|
note " overlap heavily. Real reclaimed space is much less — measure with --measure."
|
|
else
|
|
note " ${PKG_TYPE} blobs are unique per version (no shared layers), so these"
|
|
note " sizes are the real reclaimable bytes."
|
|
fi
|
|
fi
|
|
}
|
|
|
|
# ---- Compute prune plan: prints "DELETE\t<tag>\t<created>" / "KEEP..." ----
|
|
# Sets globals: PLAN_DELETE (array of tags), PLAN_KEEP_N, PLAN_PROT_N
|
|
declare -a PLAN_DELETE=() PLAN_PROTECTED=()
|
|
PLAN_KEEP_N=0; PLAN_PROT_N=0; PLAN_TOTAL=0
|
|
compute_plan() {
|
|
local name="$1" tag created epoch now cutoff idx=0
|
|
PLAN_DELETE=(); PLAN_KEEP_N=0; PLAN_PROT_N=0; PLAN_TOTAL=0; PLAN_PROTECTED=()
|
|
now="$(date +%s)"
|
|
[[ -n "$OLDER_THAN" ]] && cutoff=$(( now - OLDER_THAN * 86400 ))
|
|
while IFS=$'\t' read -r tag created; do
|
|
[[ -z "$tag" ]] && continue
|
|
PLAN_TOTAL=$((PLAN_TOTAL + 1))
|
|
# Protected always wins.
|
|
if is_protected "$name" "$tag"; then
|
|
PLAN_PROT_N=$((PLAN_PROT_N + 1)); PLAN_PROTECTED+=("${tag} — $(protect_reason "$name" "$tag")"); continue
|
|
fi
|
|
if [[ -n "$KEEP" ]]; then
|
|
# tags arrive newest-first; keep the first KEEP non-protected... but
|
|
# protected tags don't consume a keep slot — count index over all tags.
|
|
idx=$((idx + 1))
|
|
if [[ "$idx" -le "$KEEP" ]]; then PLAN_KEEP_N=$((PLAN_KEEP_N + 1)); else PLAN_DELETE+=("$tag"); fi
|
|
elif [[ -n "$OLDER_THAN" ]]; then
|
|
epoch="$(date -d "$created" +%s 2>/dev/null || echo "$now")"
|
|
if [[ "$epoch" -lt "$cutoff" ]]; then PLAN_DELETE+=("$tag"); else PLAN_KEEP_N=$((PLAN_KEEP_N + 1)); fi
|
|
fi
|
|
done < <(tagged_versions "$name")
|
|
}
|
|
|
|
# ---- PRUNE ---------------------------------------------------------------
|
|
PRUNE_DELETED=0; PRUNE_FAILED=0
|
|
declare -a PRUNE_TOUCHED_REPOS=()
|
|
do_prune_repo() {
|
|
local name="$1" tag bytes code
|
|
compute_plan "$name"
|
|
|
|
echo ""
|
|
echo -e "${BOLD}== prune ${name} ==${NC} mode: $( [[ -n "$KEEP" ]] && echo "keep-last ${KEEP}" || echo "older-than ${OLDER_THAN}d" )"
|
|
local p
|
|
for p in "${PLAN_PROTECTED[@]}"; do printf ' PROTECTED %s\n' "$p"; done
|
|
if [[ "${#PLAN_DELETE[@]}" -eq 0 ]]; then
|
|
info " Nothing to prune (${PLAN_TOTAL} tags: ${PLAN_KEEP_N} kept, ${PLAN_PROT_N} protected)."
|
|
return
|
|
fi
|
|
echo " Would delete ${#PLAN_DELETE[@]} tag(s); keep ${PLAN_KEEP_N}; protect ${PLAN_PROT_N}:"
|
|
local plan_bytes=0 szlabel
|
|
for tag in "${PLAN_DELETE[@]}"; do
|
|
if $NO_SIZES; then bytes=0; szlabel=""; else bytes="$(resolve_tag_bytes "$name" "$tag")"; szlabel="≈ $(human "$bytes")"; fi
|
|
plan_bytes=$((plan_bytes + bytes))
|
|
printf ' %-28s %s\n' "$tag" "$szlabel"
|
|
done
|
|
if ! $NO_SIZES; then
|
|
if $IS_CONTAINER; then
|
|
note " apparent total to delete ≈ $(human "$plan_bytes") (overlaps shared layers — real reclaim less)"
|
|
else
|
|
note " total to delete ≈ $(human "$plan_bytes") (blobs unique per version — reclaim is real)"
|
|
fi
|
|
fi
|
|
|
|
if ! $APPLY; then
|
|
warn " DRY-RUN — nothing deleted. --apply is a person's act (decision 62); nothing schedules it."
|
|
for tag in "${PLAN_DELETE[@]}"; do audit "DRY-RUN would-delete ${OWNER}/${name}:${tag}"; done
|
|
return
|
|
fi
|
|
|
|
# --- confirmation gate ---
|
|
if ! $ASSUME_YES; then
|
|
echo ""
|
|
warn " About to DELETE ${#PLAN_DELETE[@]} tag(s) from ${name}. This is destructive."
|
|
read -r -p " Type the package name '${name}' to confirm: " reply
|
|
if [[ "$reply" != "$name" ]]; then warn " Skipped ${name} (confirmation mismatch)."; return; fi
|
|
fi
|
|
|
|
for tag in "${PLAN_DELETE[@]}"; do
|
|
if $NO_SIZES; then bytes=0; else bytes="$(resolve_tag_bytes "$name" "$tag")"; fi
|
|
code="$(api_delete "/api/v1/packages/${OWNER}/${PKG_TYPE}/${name}/${tag}")" && {
|
|
info " deleted ${name}:${tag} (HTTP ${code})"
|
|
audit "APPLIED deleted ${OWNER}/${name}:${tag} apparent=${bytes}B http=${code}"
|
|
PRUNE_DELETED=$((PRUNE_DELETED + 1))
|
|
} || {
|
|
error " FAILED ${name}:${tag} (HTTP ${code}) — continuing"
|
|
audit "FAILED delete ${OWNER}/${name}:${tag} http=${code}"
|
|
PRUNE_FAILED=$((PRUNE_FAILED + 1))
|
|
}
|
|
done
|
|
PRUNE_TOUCHED_REPOS+=("$name")
|
|
}
|
|
|
|
# ---- RECLAIM -------------------------------------------------------------
|
|
# Step 2 + 3 of the proven mechanism: delete orphaned manifest versions, then
|
|
# trigger (or defer to) the cleanup_packages GC cron. Honors --dry-run/--apply.
|
|
do_reclaim() {
|
|
if $IS_CONTAINER; then
|
|
step "Reclaim: find & delete orphaned (untagged, unreferenced) manifests, then GC blobs."
|
|
else
|
|
step "Reclaim: ${PKG_TYPE} has no manifest indirection — GC the deleted versions' blobs."
|
|
fi
|
|
# Reload state — a preceding prune deleted tags, so the cached listing is stale.
|
|
load_versions
|
|
|
|
local before; before=""
|
|
if $MEASURE; then before="$(measure_bytes || true)"; [[ -n "$before" ]] && info " packages dir now: $(human "$before")"; fi
|
|
|
|
local name orphans norph d code total_orphans=0 deleted=0 failed=0 unsafe=0
|
|
for name in "${TARGETS[@]}"; do
|
|
if ! orphans="$(find_orphans "$name")"; then
|
|
unsafe=$((unsafe + 1)); continue # warning already emitted; skip this package
|
|
fi
|
|
norph="$(printf '%s\n' "$orphans" | grep -c . || true)"
|
|
if [[ "$norph" -eq 0 ]]; then
|
|
if $IS_CONTAINER; then info " ${name}: no orphaned manifests."; fi
|
|
continue
|
|
fi
|
|
total_orphans=$((total_orphans + norph))
|
|
if ! $APPLY; then
|
|
info " ${name}: ${norph} orphaned manifest version(s) WOULD be deleted (dry-run):"
|
|
printf '%s\n' "$orphans" | sed 's/^/ /'
|
|
continue
|
|
fi
|
|
info " ${name}: deleting ${norph} orphaned manifest version(s)..."
|
|
while IFS= read -r d; do
|
|
[[ -z "$d" ]] && continue
|
|
code="$(api_delete "/api/v1/packages/${OWNER}/${PKG_TYPE}/${name}/${d}")" && {
|
|
deleted=$((deleted + 1)); audit "RECLAIM deleted-orphan ${OWNER}/${name}/${d} http=${code}"
|
|
} || {
|
|
failed=$((failed + 1)); error " FAILED orphan ${name}/${d} (HTTP ${code})"
|
|
audit "RECLAIM FAILED-orphan ${OWNER}/${name}/${d} http=${code}"
|
|
}
|
|
done <<< "$orphans"
|
|
done
|
|
|
|
if ! $APPLY; then
|
|
echo ""
|
|
if $IS_CONTAINER; then
|
|
warn " DRY-RUN — ${total_orphans} orphaned manifest(s) shown, none deleted, cron not triggered."
|
|
else
|
|
warn " DRY-RUN — GC cron not triggered."
|
|
fi
|
|
warn " Re-run reclaim with --apply to free disk."
|
|
return 0
|
|
fi
|
|
|
|
if $IS_CONTAINER; then
|
|
info " Orphaned manifests: deleted=${deleted} failed=${failed}$( [[ $unsafe -gt 0 ]] && echo " (skipped ${unsafe} pkg for safety)")"
|
|
fi
|
|
|
|
# --- Step 3: GC the now-unreferenced blobs ---
|
|
local cron_ran=false res
|
|
step " Triggering '${CLEANUP_CRON}' cron to GC now-unreferenced blobs..."
|
|
res="$(trigger_cleanup_cron)" && cron_ran=true || true
|
|
case "$res" in
|
|
ok\ *) info " cron '${CLEANUP_CRON}' triggered (HTTP ${res#ok }) — GC running."
|
|
note " Blobs UPLOADED within [cron.cleanup_packages] OLDER_THAN (24h default) are"
|
|
note " NOT collected by this pass — their space returns on a later run."
|
|
audit "RECLAIM triggered cron ${CLEANUP_CRON} ${res}" ;;
|
|
denied) warn " Token lacks write:admin — cannot trigger the GC cron directly."
|
|
note " The orphaned manifests are deleted; their unique blobs are now unreferenced and"
|
|
note " WILL be freed by the daily '@midnight' ${CLEANUP_CRON} run, or immediately on the"
|
|
note " next Gitea restart (RUN_AT_START is enabled). No data is lost by waiting."
|
|
audit "RECLAIM cron-trigger-denied (no write:admin); GC deferred to schedule/restart" ;;
|
|
*) warn " cron trigger returned: ${res}"; audit "RECLAIM cron-trigger ${res}" ;;
|
|
esac
|
|
|
|
# Only a real GC run changes disk; measuring otherwise is misleading.
|
|
if $MEASURE && $cron_ran && [[ -n "$before" ]]; then
|
|
sleep 6
|
|
local after; after="$(measure_bytes || true)"
|
|
if [[ -n "$after" ]]; then
|
|
info " packages dir after: $(human "$after")"
|
|
info " real reclaimed ≈ $(human "$(( before - after ))") (du; Longhorn 'actual size' lags)"
|
|
audit "RECLAIM measured before=${before}B after=${after}B freed=$(( before - after ))B"
|
|
else
|
|
note " (could not re-measure — re-run reclaim --measure later)"
|
|
fi
|
|
elif $MEASURE && ! $cron_ran; then
|
|
note " (skipping after-measurement: GC cron did not run; measure after the scheduled run/restart)"
|
|
fi
|
|
}
|
|
|
|
# =============================================================================
|
|
# Interactive menu (when no --repo/--all and a TTY is present)
|
|
# =============================================================================
|
|
interactive_menu() {
|
|
load_versions
|
|
local -a names=(); local n
|
|
while IFS= read -r n; do names+=("$n"); done < <(package_names)
|
|
if [[ "${#names[@]}" -eq 0 ]]; then warn "No ${PKG_TYPE} packages found for owner '${OWNER}'."; exit 0; fi
|
|
|
|
echo ""
|
|
echo -e "${BOLD}Packages of type '${PKG_TYPE}' under '${OWNER}':${NC}"
|
|
local i
|
|
for i in "${!names[@]}"; do
|
|
if $IS_CONTAINER; then
|
|
printf ' %2d) %-22s %s tags, %s digest manifests\n' \
|
|
"$((i+1))" "${names[$i]}" \
|
|
"$(tagged_versions "${names[$i]}" | grep -c . || true)" \
|
|
"$(digest_count "${names[$i]}")"
|
|
else
|
|
printf ' %2d) %-22s %s versions\n' \
|
|
"$((i+1))" "${names[$i]}" \
|
|
"$(tagged_versions "${names[$i]}" | grep -c . || true)"
|
|
fi
|
|
done
|
|
echo " a) all packages of this type"
|
|
note " (other types are NOT listed — re-run with --type <t> to see them)"
|
|
echo ""
|
|
read -r -p "Select package(s) [numbers/space-sep, or 'a' for all]: " sel
|
|
local -a chosen=()
|
|
if [[ "$sel" == "a" || "$sel" == "all" ]]; then chosen=("${names[@]}");
|
|
else
|
|
for tok in $sel; do
|
|
[[ "$tok" =~ ^[0-9]+$ ]] && [[ "$tok" -ge 1 ]] && [[ "$tok" -le "${#names[@]}" ]] && chosen+=("${names[$((tok-1))]}")
|
|
done
|
|
fi
|
|
[[ "${#chosen[@]}" -eq 0 ]] && { warn "Nothing selected."; exit 0; }
|
|
REPOS=("${chosen[@]}")
|
|
|
|
echo ""
|
|
echo "Action: 1) list 2) prune 3) reclaim (cleanup cron)"
|
|
read -r -p "Choose [1-3]: " act
|
|
case "$act" in
|
|
1) ACTION="list" ;;
|
|
2) ACTION="prune"
|
|
read -r -p "Keep how many most-recent tags? [${DEFAULT_KEEP}]: " KEEP; KEEP="${KEEP:-$DEFAULT_KEEP}"
|
|
[[ "$KEEP" =~ ^[0-9]+$ ]] || { error "invalid number"; exit 2; }
|
|
read -r -p "Apply for real now? (dry-run otherwise) [y/N]: " ap
|
|
[[ "$ap" =~ ^[Yy]$ ]] && APPLY=true
|
|
read -r -p "Also reclaim disk (run cleanup cron) after? [y/N]: " rc
|
|
[[ "$rc" =~ ^[Yy]$ ]] && DO_RECLAIM=true
|
|
;;
|
|
3) ACTION="reclaim" ;;
|
|
*) error "invalid action"; exit 2 ;;
|
|
esac
|
|
}
|
|
|
|
# =============================================================================
|
|
# Main
|
|
# =============================================================================
|
|
echo ""
|
|
info "╔══════════════════════════════════════════╗"
|
|
info "║ Gitea package prune ║"
|
|
info "╚══════════════════════════════════════════╝"
|
|
info "Server: ${GITEA_URL} Owner: ${OWNER} Type: ${PKG_TYPE} Log: ${LOG_FILE}"
|
|
info "Auth: ${CRED_SRC}$( [[ -n "$GITEA_USER" ]] && echo " (user: ${GITEA_USER})")"
|
|
|
|
# Sanity: Gitea version (public, no auth)
|
|
if [[ -n "$PRUNE_TEST_VERSIONS" ]]; then GV="1.26.test"; else
|
|
GV="$(curl -fsS "${GITEA_URL}/api/v1/version" 2>/dev/null | jq -r '.version' 2>/dev/null || echo '?')"; fi
|
|
info "Gitea version: ${GV}"
|
|
[[ "$GV" == 1.26.* ]] || warn "Tested against Gitea 1.26.2 — server reports '${GV}'. Verify API shapes."
|
|
|
|
if $MEASURE; then detect_kubectl; [[ -n "$KUBECTL" ]] || warn "kubectl not usable — --measure will be skipped."; fi
|
|
|
|
# --- Selection / mode -----------------------------------------------------
|
|
if ! $ALL && [[ "${#REPOS[@]}" -eq 0 ]]; then
|
|
if [[ -t 0 ]]; then
|
|
interactive_menu
|
|
else
|
|
error "No --repo/--all given and not a TTY (cannot show interactive menu)."
|
|
usage 1
|
|
fi
|
|
else
|
|
load_versions
|
|
fi
|
|
|
|
# Resolve repo set
|
|
declare -a TARGETS=()
|
|
if $ALL; then
|
|
while IFS= read -r n; do TARGETS+=("$n"); done < <(package_names)
|
|
warn "--all covers type '${PKG_TYPE}' ONLY. Other types are untouched by this run."
|
|
else
|
|
for r in "${REPOS[@]}"; do
|
|
if jq -rs --arg n "$r" 'any(.[]; .name==$n)' "$VERSIONS_JSON" | grep -q true; then
|
|
TARGETS+=("$r")
|
|
else
|
|
warn "Package not found for owner '${OWNER}': ${r} (skipping)"
|
|
fi
|
|
done
|
|
fi
|
|
[[ "${#TARGETS[@]}" -eq 0 ]] && { error "No valid target packages."; exit 1; }
|
|
|
|
# Infer action: keep/older-than => prune; else list. reclaim is always explicit.
|
|
if [[ -z "$ACTION" ]]; then
|
|
if [[ -n "$KEEP" || -n "$OLDER_THAN" ]]; then ACTION="prune"; else ACTION="list"; fi
|
|
fi
|
|
# prune: --keep defaults to 20 (decision 62)
|
|
if [[ "$ACTION" == "prune" && -z "$KEEP" && -z "$OLDER_THAN" ]]; then
|
|
KEEP="$DEFAULT_KEEP"; info "prune: --keep defaults to ${DEFAULT_KEEP} (decision 62)"
|
|
fi
|
|
# prune: the versions in use are read FIRST, or nothing is pruned (dry-run included — its plan would lie)
|
|
if [[ "$ACTION" == "prune" ]]; then
|
|
load_vouch_protect || { error "REFUSING to prune: the versions in use could not be read (decision 62 — never 'protect nothing')."; audit "REFUSED prune: in-use list unreadable"; exit 3; }
|
|
for k in $(printf '%s\n' "${!VOUCH_PROTECT[@]}" | sort); do note " in use: ${k} — ${VOUCH_PROTECT[$k]}"; done
|
|
fi
|
|
|
|
info "Action: ${ACTION} Type: ${PKG_TYPE} Targets: ${TARGETS[*]}"
|
|
audit "RUN action=${ACTION} owner=${OWNER} type=${PKG_TYPE} targets='${TARGETS[*]}' apply=${APPLY} keep='${KEEP}' older_than='${OLDER_THAN}' protect='${PROTECT[*]}'"
|
|
|
|
case "$ACTION" in
|
|
list)
|
|
for t in "${TARGETS[@]}"; do do_list "$t"; done
|
|
;;
|
|
prune)
|
|
# Strict confirmation for --all --apply
|
|
if $ALL && $APPLY && ! $ASSUME_YES; then
|
|
echo ""
|
|
warn "You are about to PRUNE ALL ${#TARGETS[@]} packages: ${TARGETS[*]}"
|
|
read -r -p "Type 'DELETE ALL' to confirm: " reply
|
|
[[ "$reply" == "DELETE ALL" ]] || { error "Aborted."; exit 1; }
|
|
ASSUME_YES=true # per-repo prompts already covered by this gate
|
|
fi
|
|
for t in "${TARGETS[@]}"; do do_prune_repo "$t"; done
|
|
echo ""
|
|
info "Prune summary: deleted=${PRUNE_DELETED} failed=${PRUNE_FAILED} (dry-run=$( $APPLY && echo no || echo yes ))"
|
|
audit "PRUNE-SUMMARY deleted=${PRUNE_DELETED} failed=${PRUNE_FAILED} apply=${APPLY}"
|
|
if $APPLY && $DO_RECLAIM && [[ "$PRUNE_DELETED" -gt 0 ]]; then
|
|
echo ""; do_reclaim || true
|
|
elif $APPLY && [[ "$PRUNE_DELETED" -gt 0 ]]; then
|
|
note "Tags deleted, but bytes remain until reclaim. Run: $0 reclaim --measure (or pass --reclaim)"
|
|
fi
|
|
;;
|
|
reclaim)
|
|
do_reclaim || exit 1
|
|
;;
|
|
esac
|
|
|
|
echo ""
|
|
info "Done. Audit log: ${LOG_FILE}"
|