felhom.eu/documentation/runbooks/monthly-floating-retest.md

RUNBOOK — the monthly re-test of same-name security fixes (09 §3 decisions 52, 54, 55; R-740, R-743)

What it does. An image such as postgres:18-alpine or redis:7-alpine gets security fixes under the SAME name. A box takes such a fix at night only when the catalog has re-tested the tag at the new digest on both venues and written it as a ladder step. This runbook is that re-test, once a month. One command does the work; the steps around it set up the two venues and tear them down.

Scope (decision 55): every app with a proven ladder — not only the database and redis lines. The web apps face the internet; the databases do not. --engines-only is the narrow switch, not the default.

Who runs it (decision 54): a CC session the operator starts once a month with the standing brief claude/MONTHLY-security-retest.md (in the planning project), from DooPlex. STATUS carries "Monthly security re-test: last run , next due " — update it at the end of every run. Why not a cron job (measured 2026-09-30): it needs a fresh bench LXC on demo-hp, scratch guest 9202 pointed at the drill catalog, a drill reset (a force-push — the permission check refused it once and the operator allowed it), and pushes to the LIVE catalog. None of that should happen with nobody watching.

1. Look first (read-only, 1 minute)

cd /mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu && git pull -q
python3 scripts/retest-floating.py --dry-run                  # every app with a proven ladder (decision 55)

nothing to re-test today ends the month. Otherwise go on.

2. The bench (LXC 9401 on demo-hp)

The recipe in audits/rulings-2026-10-01/A/A1-bench-create.txt (60 GB disk on nvme-scratch — 40 GB filled up on 2026-09-30), swap 0 (the stricter venue, R-733); pveam download the Debian 13 template first if it is gone. retest-floating.py syncs the catalog's scripts and templates to it itself (it checks only docker + python3 first, R-749).

3. The box (scratch guest 9202)

1. Reset the drill to the live catalog: git -C /mnt/5_hdd/felhom.eu/drill/app-catalog-drill fetch live && git reset --hard live/main && git push -f origin main (force-push: ask if the permission check refuses).
2. Point 9202 at the drill (09 §6.5): the repoint.py drill of the latest audit's tools/ (saves controller.yaml, sets the drill URL + credentials, removes the catalog cache, restarts). Quote repo_url read back.
3. export SC=<a 0600 scratch dir> holding .ctlpw (9202's dashboard password — never committed).

4. The run

python3 scripts/retest-floating.py --push \
    --evidence $SC/evidence --evidence-rel felhom.eu/documentation/audits/retest-<YYYY-MM>

Per app: bench (the full method, 10-minute memory watch), box (fresh install at the OLD tested digest, seed, the re-test entry in the drill, the guarded Update, read-back, the running digest must be the NEW one), then the writer, the catalog gates and one commit (pushed with --push; the pre-push gates run). A failure stops that app and never the list; the summary names each app DONE or STOPPED with its reason. Order: database/redis lines first, then the rest alphabetically (nextcloud — internet-facing, holds data — comes before the linuxserver apps). Copy $SC/evidence/<date>/* to felhom.eu/documentation/audits/retest-<YYYY-MM>/ (the ladder entries cite retest-<YYYY-MM>/<app>/{bench,box}).

Monthly cost (measured 2026-10-01): see "What it cost" below. linuxserver images (bookstack, radarr, sonarr, code-server) are rebuilt upstream weekly under the same tag, so most months they come up.

4a. Infrastructure pins (R-838, 2026-10-04)

The box's three built-in containers — traefik, cloudflared, filebrowser — are pinned in felhom-controller/controller/internal/infra/infra.go (TraefikImage, CloudflaredImage, FileBrowserImage). They are not catalog templates, so retest-floating.py never sees them. Each month:

cd felhom-controller/controller && python3 scripts/check-infra-pins.py   # exit 1 = at least one BEHIND (report only)

For each BEHIND pin: read the upstream release notes between the pinned and the newest version (same channel: traefik v3.x, cloudflared YYYY.M.P, filebrowser N.N.N-stable — never a beta), name any breaking change against what we configure, raise the constant, release the controller, prove it on 9202 then on both demo boxes (the containers are recreated within ~20 s of the new controller starting: traefik/cloudflared by the base-infra bring-up, filebrowser by the start-up mount sync), and time the public gap through the tunnel. Measured 2026-10-04: ≤ 19.6 s on demo-hp, ≤ 14.7 s on demo-felhom, counting the controller's own restart (audits/os-guest-lane-2026-10-04/partF/).

5. Teardown (three layers, stated)

Machine: 9202 back on the live catalog (repoint.py restore), apps the run installed are removed by it. Host: pct destroy 9401 --purge. Hub: nothing touched. Reset the drill again (§3.1).

What proves it works (2026-09-30)

audits/night-rulings-2026-09-30/A/e2e/: docmost at an OLDER redis:7-alpine digest on 9202, the re-test on the bench, "run tonight's chain now" → the leg's docmost: step pressed … step ended done after 95.0 s, the new digest running, the data read back, the badge back to "Naprakész".