- upgrade-test.py --restep <verdict> --definition <dir> --catalog <c> --evidence <rel>: the only way a
superseded step's own files (steps/<key>.yml + .felhom.yml) change after the fact (Part D: immich's
step 0b8272068aab36bf still pins 512M). Refuses a non-proven or OOM-killing re-proof, the head entry,
and a definition whose images are not the step's; leaves the ladder entry untouched (digests, box
evidence, the box's failed-step fingerprint). Two tests; the gate accepts the result.
- zipline's verify waited on `/` for 200/302/307; on 9202 it answers 301 and the readback returned
False with no line — it now waits on /api/healthcheck like its seed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
A new front-door fixture: the app's own socket.io messages over plain HTTP polling (setup, login,
addStatusPage), read back through its public /api/status-page/<slug>, with an absent-slug control.
Bench 9401 (harness v4, swap 0): proven, 10-min watch 0 kills (anon peak 43.3 %); the abort starts
and serves. Box 9202: done in 33.9 s through the setup gate, the status page read back.
Written by upgrade-test.py --write-ladder.
Evidence: felhom.eu/documentation/audits/more-night-apps-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
A new front-door fixture: crafty's own API v2 (login as admin, a role created and read back, a
wrong-token control), reached over its own HTTPS port on the bench as traefik reaches it on a box.
Bench 9401 (harness v4, swap 0): proven, 10-min watch 0 kills (anon peak 2.7 % of 2048M); the abort
starts and serves. Box 9202: done in 51.3 s, the role read back. Written by upgrade-test.py --write-ladder.
Evidence: felhom.eu/documentation/audits/more-night-apps-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
A new front-door fixture: the web login, then a weight entry through the app's own API, read back
with a no-session and a never-entered-weight control. Both venues ran on the R-738 template (7a4ff48,
migrations at start): bench 9401 proven, its migrations ran, 10-min watch 0 kills (anon peak 49.8 %);
box 9202 done in 46.2 s, the migrations ran, the entry read back. The first box walk WITHOUT the fix
failed (login 500, 12 migrations unapplied) and wrote nothing. Putting 2.6 back after the migration
loses the entries (recorded, as always; the box's undo restores the pre-update copy instead).
Written by upgrade-test.py --write-ladder.
Evidence: felhom.eu/documentation/audits/more-night-apps-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
- fixtures (upgrade_fixtures_box.py): calibre-web (Upload form -> OPDS readback + the served EPUB),
wger (web login -> weight entry API), crafty-controller (API v2 roles), uptime-kuma (socket.io
polling: setup, login, addStatusPage -> public /api/status-page/<slug>); gitea posts its own
first-run installer form (R-624's fixable case) and keeps the admin CLI for an installed one.
calibre-web and wger run the template's own after_install on the bench, which has none.
- R-735: the bench's `password:N:special` now has the controller's shape (randomWithSpecial);
test seen failing first (length 32, no special), then 45/45.
- upgrade_boxport / the memory watch: a backend traefik reaches over https (loadbalancer.server.scheme)
is reached over https on the bench too (crafty-controller).
- upgrade-test: files-before/after-detail.json and `files_changed_detail` NAME the files behind a
files_may_change mark (R-734's method, now in the harness); test ChangedFiles.
- test_catalog_gates: the gate count was stale (9, the runner has 10) and red on main; now 10.
Evidence: felhom.eu/documentation/audits/more-night-apps-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
A new front-door fixture: a book uploaded through the app's own Upload form and read back through
its OPDS feed (listed, and the served EPUB carries the marker), with a wrong-password and an
absent-title control. Bench 9401 (harness v4, swap 0): proven, 10-min watch 0 kills (anon peak
19.3 %); the harness saw the household books folder's tree hash change, so the step carries
files_may_change (a night press needs a fresh whole copy). Box 9202: done in 64.6 s, the book read
back, and not one file in the books folder changed (per-file hashes before/after).
Written by upgrade-test.py --write-ladder.
Evidence: felhom.eu/documentation/audits/more-night-apps-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
wger's image runs `manage.py migrate` only when DJANGO_PERFORM_MIGRATIONS=True (entrypoint.sh).
Without it, 2.6 -> 2.7 through the guarded Update on 9202 ended `done` and left 12 migrations
unapplied: the web login answered 500 (no such column: core_userprofile.time_zone). With the switch,
the same step on 9202 ran the migrations and the seeded weight entry read back. No image moves here;
no box reporting to the hub runs wger.
Evidence: felhom.eu/documentation/audits/more-night-apps-2026-09-30/box/wger/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Bench 9401 (harness v4, swap 0): the onboarding owner seeded and logged in before and after,
10-min memory watch 0 kills (anon peak 32.0 %); the abort starts and serves. Box 9202: the guarded
Update done in 113.8 s through the setup gate, the owner logs in. The old head was a backfilled
harness-v1 entry; this step carries the memory watch. Written by upgrade-test.py --write-ladder.
Evidence: felhom.eu/documentation/audits/more-night-apps-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Bench 9401 (harness v4, swap 0): the site set up through /ghost/api/admin/authentication/setup/,
its title read back before and after, 10-min memory watch 0 kills (anon peak 26.7 %); the abort
starts and serves. Box 9202: the guarded Update done in 107.7 s through the setup gate, read back.
Written by upgrade-test.py --write-ladder.
Evidence: felhom.eu/documentation/audits/more-night-apps-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Bench 9401 (harness v4, swap 0): the startup-wizard user seeded and read back before and after,
10-min memory watch 0 kills (anon peak 4.7 %); the abort starts and serves. Box 9202: the guarded
Update done in 23.6 s through the setup gate, read back. Written by upgrade-test.py --write-ladder;
the superseded step keeps its definition in steps/.
Evidence: felhom.eu/documentation/audits/more-night-apps-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Bench 9401 (harness v4, swap 0): the first-run installer form posted, a repo seeded and read back
before and after, 10-min memory watch 0 kills (anon peak 26.6 %, cgroup 52.9 %). Box 9202: the
guarded Update done in 21.6 s through the setup gate, the repo read back. Written by
upgrade-test.py --write-ladder. 28.0.0 is a major (2026-09-02 ruling) and is not touched.
Evidence: felhom.eu/documentation/audits/more-night-apps-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The ONLY image move in this commit; the limit rides it (09 decision 39's precedent), because the step
itself re-runs immich's geodata import (228 294 -> 228 571 places) — the load that killed the database.
Cause, measured on the bench (audits/immich-first-start-2026-09-30/A-cause.md): the first-start import
runs up to 9 concurrent 5000-row INSERTs; the database then needs ~400 MB anon + ~170 MB touched
shared_buffers (the image's own postgresql.conf fixes 512MB). At 512M with no swap: 61 kills; with 512 MB
swap: 0 (swapped ~70 MB) — why 9202 passed; shared_buffers 128MB alone: still killed; 1024M: 0; 768M: 0.
Proof at the new definition, fresh install from birth, no swap: bench x2 (anon 409/412 MB = 53 %, 0 kills,
import 8.1 s) and box 9202 (anon 368 MB = 48 %, 0 kills, swap.peak 0).
Step, written by upgrade-test.py --write-ladder: bench (harness v4) proven, 10-min watch 0 kills 0
restarts, anon peak 51.1 %; box 9202 through the guarded Update: done 58.5 s, album read back, the running
database limit 805306368 after. The step carries `files_may_change` (the harness saw only immich's six
13-byte `.immich` folder markers rewritten at start) — the night leg takes it only with a whole copy.
Per-box cost: +256 MB on immich-postgres; `mem_limit` 4096M -> 4480M (the old figure was already 128 MB
under the sum of the four limits). Header comment corrected (it said postgres 256M).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
zipline 4's first-run route is POST /api/setup (measured on the bench 2026-09-30, v4.6.1: GET ->
{"firstSetup":true}, POST {username,password} -> 200 SUPERADMIN, the login works). The two paths the
fixture tried stay as fallbacks. No image moves in this commit.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
{'komga': 'gotson/komga:1.27.1'} -> {'komga': 'gotson/komga:1.28.0'}
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder (catalog gates rc=0):
- bench LXC 9401 (harness v4): the seed read back before and after; 10-minute memory watch:
komga anon 64.5 % (cgroup 67.5 %); 0 kills, 0 restarts; the abort starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 44.2 s, the seed
read back through the app's own front door.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/komga/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
{'grafana': 'grafana/grafana:13.2.2'} -> {'grafana': 'grafana/grafana:13.2.3'}
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder (catalog gates rc=0):
- bench LXC 9401 (harness v4): the seed read back before and after; 10-minute memory watch:
grafana anon 47.1 % (cgroup 100.0 %); 0 kills, 0 restarts; the abort starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 23.6 s, the seed
read back through the app's own front door.
The box walk was run TWICE: the first read back 401 because the instrument lost the deploy's
admin password between two processes (box/grafana/step-attempt1-instrument-401.txt); fixed in the
walk tools, grafana reinstalled at 13.2.2 from the drill catalog, seeded, stepped: read back.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/grafana/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
{'navidrome': 'deluan/navidrome:0.64.1'} -> {'navidrome': 'deluan/navidrome:0.64.2'}
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder (catalog gates rc=0):
- bench LXC 9401 (harness v4): the seed read back before and after; 10-minute memory watch:
navidrome anon 6.9 % (cgroup 9.4 %); 0 kills, 0 restarts; the abort starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 14.4 s, the seed
read back through the app's own front door.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/navidrome/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
{'n8n': 'n8nio/n8n:2.41.2'} -> {'n8n': 'n8nio/n8n:2.42.1'}
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder (gates rc=0):
- bench LXC 9401 (harness v4): the seed read back before and after; 10-minute memory watch: n8n anon 23.0 % | migration: [2Kn8n | Migrations in progress, please do NOT stop the process.;
0 kills, 0 restarts; the abort starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 122.1 s, the seed
read back through the app's own front door.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/n8n/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
{'audiobookshelf': 'ghcr.io/advplyr/audiobookshelf:2.36.1'} -> {'audiobookshelf': 'ghcr.io/advplyr/audiobookshelf:2.37.1'}
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder (gates rc=0):
- bench LXC 9401 (harness v4): the seed read back before and after; 10-minute memory watch: audiobookshelf anon 10.5 % | migration: [2Kaudiobookshelf | [2026-09-30 13:52:16.057] INFO: [MigrationManager] No migr;
0 kills, 0 restarts; the abort starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 24.6 s, the seed
read back through the app's own front door.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/audiobookshelf/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The ONLY image move in this commit. Upstream stopped the `apache-` tag prefix after 2.57.0;
MEASURED 2026-09-30 on Docker Hub: `2.67.0`, `2`, `stable` and `apache` all resolve to ONE digest
(sha256:3084f1e5…), while `fpm`/`latest` are another — so the plain tag IS the Apache variant.
Written by upgrade-test.py --write-ladder:
- bench LXC 9401 (harness v4, box fixture Kimai via boxport): the seeded user read back before
and after, Doctrine migrations ran (to Version20260804090000), 10-minute memory watch: kimai
anon 6.0 % (cgroup 91.7 % = file cache, decision 22), kimai-db anon 45.0 %, 0 kills, 0 restarts;
the abort starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 82.1 s, the
seeded user read back through the app's own CLI.
The superseded step keeps its definition at steps/def18c0c25cf49f7.{yml,felhom.yml}.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/kimai/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder:
- bench LXC 9401 (harness v4, the bench's own BookStack fixture): FROM settled, the seeded admin
read back, migrations ran, read back after, 10-minute memory watch peak 43.7 % (anon), 0 kills,
0 restarts; the abort (old image back) starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 64.6 s, the
seeded admin read back through the app's own CLI.
The superseded step keeps its definition at steps/fdb7afbec5ea4dbc.{yml,felhom.yml} (09 §6.4 part 5).
The database half only, as before (R-460). MariaDB 12.3 does not move.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/bookstack/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder from both venues:
- bench LXC 9401 (harness v4): converted 15 -> 18 in 13.3 s, check equal over 94 tables, the seeded
check-in weight read back, 10-minute memory watch peak 31.3 % (anon), 0 kills, 0 restarts;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): CONVERTED 15 -> 18 in
9.8 s (2 databases, 94 tables, 261 rows equal), done in 58.5 s, PG_VERSION 18, the weight read
back through the front door. Undo case on 9202 (after a reinstall at 15): the load failing
(adminpack) -> undone in 74.9 s, back on 15, the weight read back.
Target 18: SparkyFitness's own upstream compose pairs v0.17.3 with postgres:18.3-alpine at
/var/lib/postgresql (09 decision 42's rule). PGDATA is left as it was (/var/lib/postgresql/data/pgdata),
now a folder inside the moved volume — the definition both venues tested. Only the header comment
changes beside the image and its mount.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder from both venues:
- bench LXC 9401 (harness v4): converted 16 -> 18 in 26.7 s, check equal over 42 tables, the seeded
document read back, 10-minute memory watch peak 48.9 % (anon), 0 kills, 0 restarts;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): converted, done in
41.1 s, PG_VERSION 18, the document read back through the front door. Undo case on 9202 first:
the load failing (adminpack) -> undone in 57.5 s, back on 16, the document read back.
Target 18: Outline's official self-host compose (docs.getoutline.com, Docker) runs postgres:18 at
/var/lib/postgresql (09 decision 42's rule); Outline documents PostgreSQL 14+.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder from both venues:
- bench LXC 9401 (harness v4): converted 16 -> 18 in 21.4 s, check equal over 31 tables, the seeded
poll read back, 10-minute memory watch peak 61.6 % (anon), 0 kills, 0 restarts;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): converted, done in
32.8 s, PG_VERSION 18, the poll read back through the front door. Undo case on 9202 first: the
load failing (adminpack, which 18 lacks) -> undone in 57.5 s, back on 16, the poll read back.
Target 18: rallly's own upstream compose runs postgres:18-alpine at /var/lib/postgresql at
v4.11.1 (09 decision 42's rule). The data mount moves with the image (pg_mounts_for).
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
- Sparkyfitness: better-auth sign-up/sign-in, a check-in weight stored and read back; a wrong
password and an empty date must read as absent. Waits out the app's own 429 (one client
address behind traefik).
- Rallly: sign-up, the six-digit e-mail code READ (select only) from the app's own
verifications row in place of a mailbox, verify-email, sign-in, polls.make, readback by the
public polls.get; an unknown id must be not found.
- Outline: the self-hosted first-run route installation.create (workspace + admin, refused once a
team exists), an API key with Outline's own CSRF pair, a document, readback by documents.info;
an unknown id must 404 and a wrong key 401.
- outline and rallly leave the NoRoute list: both had a front-door route after all.
Measured on the bench (LXC 9401) and on 9202 2026-09-30:
felhom.eu/documentation/audits/pg-last-six-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Proven on 9202 with controller 0.279.0 (drill 5ac5daf): the default no longer logs in, the generated first
password from the app page does, a wrong one does not. Copy freeze: bookstack's new/changed strings signed off.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Proven on 9202 with controller 0.279.0 (drill catalog c4cfb83): the default no longer authenticates, the
generated first password from the app page does, a wrong one does not; a restore keeps it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Written by upgrade-test.py --write-ladder: bench (harness v4) healthy in 217 s, the world-data file present and reused ("Latest data already imported"), 6 migration lines, seed read back, memory watch anon peak 73.0 %, 0 restarts; box 9202 through the guarded Update in 204 s with the product's default 5-min wait (a first try under the drill's 90 s wait was UNDONE correctly). Same commit: (1) the frontend healthcheck override is dropped — v0.13.0 brings its own (node at /usr/bin/node); (2) a world-data file that does not parse to its end is renamed aside before start so download-countries fetches it again — with an import pending, the image's own start crash-looped 10 times on IncompleteJSONError, with the step it was set aside, downloaded and healthy. Evidence felhom.eu/documentation/audits/version-travel-2026-09-26/C/.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Written by upgrade-test.py --write-ladder: bench converted in 23.6 s, check equal over 100 tables, seed read back, memory watch anon peak 78.7 % (the app, not the engine), 0 kills; box 9202 converted through the guarded Update in 52.3 s, seed read back. The superseded 16 step keeps its definition in steps/. Evidence felhom.eu/documentation/audits/version-travel-2026-09-26/B/.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
admin