Bench 9401 (harness v4, swap 0): the startup-wizard user seeded and read back before and after,
10-min memory watch 0 kills (anon peak 4.7 %); the abort starts and serves. Box 9202: the guarded
Update done in 23.6 s through the setup gate, read back. Written by upgrade-test.py --write-ladder;
the superseded step keeps its definition in steps/.
Evidence: felhom.eu/documentation/audits/more-night-apps-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Bench 9401 (harness v4, swap 0): the first-run installer form posted, a repo seeded and read back
before and after, 10-min memory watch 0 kills (anon peak 26.6 %, cgroup 52.9 %). Box 9202: the
guarded Update done in 21.6 s through the setup gate, the repo read back. Written by
upgrade-test.py --write-ladder. 28.0.0 is a major (2026-09-02 ruling) and is not touched.
Evidence: felhom.eu/documentation/audits/more-night-apps-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The ONLY image move in this commit; the limit rides it (09 decision 39's precedent), because the step
itself re-runs immich's geodata import (228 294 -> 228 571 places) — the load that killed the database.
Cause, measured on the bench (audits/immich-first-start-2026-09-30/A-cause.md): the first-start import
runs up to 9 concurrent 5000-row INSERTs; the database then needs ~400 MB anon + ~170 MB touched
shared_buffers (the image's own postgresql.conf fixes 512MB). At 512M with no swap: 61 kills; with 512 MB
swap: 0 (swapped ~70 MB) — why 9202 passed; shared_buffers 128MB alone: still killed; 1024M: 0; 768M: 0.
Proof at the new definition, fresh install from birth, no swap: bench x2 (anon 409/412 MB = 53 %, 0 kills,
import 8.1 s) and box 9202 (anon 368 MB = 48 %, 0 kills, swap.peak 0).
Step, written by upgrade-test.py --write-ladder: bench (harness v4) proven, 10-min watch 0 kills 0
restarts, anon peak 51.1 %; box 9202 through the guarded Update: done 58.5 s, album read back, the running
database limit 805306368 after. The step carries `files_may_change` (the harness saw only immich's six
13-byte `.immich` folder markers rewritten at start) — the night leg takes it only with a whole copy.
Per-box cost: +256 MB on immich-postgres; `mem_limit` 4096M -> 4480M (the old figure was already 128 MB
under the sum of the four limits). Header comment corrected (it said postgres 256M).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
{'komga': 'gotson/komga:1.27.1'} -> {'komga': 'gotson/komga:1.28.0'}
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder (catalog gates rc=0):
- bench LXC 9401 (harness v4): the seed read back before and after; 10-minute memory watch:
komga anon 64.5 % (cgroup 67.5 %); 0 kills, 0 restarts; the abort starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 44.2 s, the seed
read back through the app's own front door.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/komga/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
{'grafana': 'grafana/grafana:13.2.2'} -> {'grafana': 'grafana/grafana:13.2.3'}
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder (catalog gates rc=0):
- bench LXC 9401 (harness v4): the seed read back before and after; 10-minute memory watch:
grafana anon 47.1 % (cgroup 100.0 %); 0 kills, 0 restarts; the abort starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 23.6 s, the seed
read back through the app's own front door.
The box walk was run TWICE: the first read back 401 because the instrument lost the deploy's
admin password between two processes (box/grafana/step-attempt1-instrument-401.txt); fixed in the
walk tools, grafana reinstalled at 13.2.2 from the drill catalog, seeded, stepped: read back.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/grafana/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
{'navidrome': 'deluan/navidrome:0.64.1'} -> {'navidrome': 'deluan/navidrome:0.64.2'}
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder (catalog gates rc=0):
- bench LXC 9401 (harness v4): the seed read back before and after; 10-minute memory watch:
navidrome anon 6.9 % (cgroup 9.4 %); 0 kills, 0 restarts; the abort starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 14.4 s, the seed
read back through the app's own front door.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/navidrome/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
{'n8n': 'n8nio/n8n:2.41.2'} -> {'n8n': 'n8nio/n8n:2.42.1'}
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder (gates rc=0):
- bench LXC 9401 (harness v4): the seed read back before and after; 10-minute memory watch: n8n anon 23.0 % | migration: [2Kn8n | Migrations in progress, please do NOT stop the process.;
0 kills, 0 restarts; the abort starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 122.1 s, the seed
read back through the app's own front door.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/n8n/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
{'audiobookshelf': 'ghcr.io/advplyr/audiobookshelf:2.36.1'} -> {'audiobookshelf': 'ghcr.io/advplyr/audiobookshelf:2.37.1'}
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder (gates rc=0):
- bench LXC 9401 (harness v4): the seed read back before and after; 10-minute memory watch: audiobookshelf anon 10.5 % | migration: [2Kaudiobookshelf | [2026-09-30 13:52:16.057] INFO: [MigrationManager] No migr;
0 kills, 0 restarts; the abort starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 24.6 s, the seed
read back through the app's own front door.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/audiobookshelf/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The ONLY image move in this commit. Upstream stopped the `apache-` tag prefix after 2.57.0;
MEASURED 2026-09-30 on Docker Hub: `2.67.0`, `2`, `stable` and `apache` all resolve to ONE digest
(sha256:3084f1e5…), while `fpm`/`latest` are another — so the plain tag IS the Apache variant.
Written by upgrade-test.py --write-ladder:
- bench LXC 9401 (harness v4, box fixture Kimai via boxport): the seeded user read back before
and after, Doctrine migrations ran (to Version20260804090000), 10-minute memory watch: kimai
anon 6.0 % (cgroup 91.7 % = file cache, decision 22), kimai-db anon 45.0 %, 0 kills, 0 restarts;
the abort starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 82.1 s, the
seeded user read back through the app's own CLI.
The superseded step keeps its definition at steps/def18c0c25cf49f7.{yml,felhom.yml}.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/kimai/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder:
- bench LXC 9401 (harness v4, the bench's own BookStack fixture): FROM settled, the seeded admin
read back, migrations ran, read back after, 10-minute memory watch peak 43.7 % (anon), 0 kills,
0 restarts; the abort (old image back) starts and serves the data;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): done in 64.6 s, the
seeded admin read back through the app's own CLI.
The superseded step keeps its definition at steps/fdb7afbec5ea4dbc.{yml,felhom.yml} (09 §6.4 part 5).
The database half only, as before (R-460). MariaDB 12.3 does not move.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/F/ and .../box/bookstack/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder from both venues:
- bench LXC 9401 (harness v4): converted 15 -> 18 in 13.3 s, check equal over 94 tables, the seeded
check-in weight read back, 10-minute memory watch peak 31.3 % (anon), 0 kills, 0 restarts;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): CONVERTED 15 -> 18 in
9.8 s (2 databases, 94 tables, 261 rows equal), done in 58.5 s, PG_VERSION 18, the weight read
back through the front door. Undo case on 9202 (after a reinstall at 15): the load failing
(adminpack) -> undone in 74.9 s, back on 15, the weight read back.
Target 18: SparkyFitness's own upstream compose pairs v0.17.3 with postgres:18.3-alpine at
/var/lib/postgresql (09 decision 42's rule). PGDATA is left as it was (/var/lib/postgresql/data/pgdata),
now a folder inside the moved volume — the definition both venues tested. Only the header comment
changes beside the image and its mount.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder from both venues:
- bench LXC 9401 (harness v4): converted 16 -> 18 in 26.7 s, check equal over 42 tables, the seeded
document read back, 10-minute memory watch peak 48.9 % (anon), 0 kills, 0 restarts;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): converted, done in
41.1 s, PG_VERSION 18, the document read back through the front door. Undo case on 9202 first:
the load failing (adminpack) -> undone in 57.5 s, back on 16, the document read back.
Target 18: Outline's official self-host compose (docs.getoutline.com, Docker) runs postgres:18 at
/var/lib/postgresql (09 decision 42's rule); Outline documents PostgreSQL 14+.
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The ONLY image move in this commit. Written by upgrade-test.py --write-ladder from both venues:
- bench LXC 9401 (harness v4): converted 16 -> 18 in 21.4 s, check equal over 31 tables, the seeded
poll read back, 10-minute memory watch peak 61.6 % (anon), 0 kills, 0 restarts;
- box 9202 (controller 0.283.1, the product's guarded Update, drill catalog): converted, done in
32.8 s, PG_VERSION 18, the poll read back through the front door. Undo case on 9202 first: the
load failing (adminpack, which 18 lacks) -> undone in 57.5 s, back on 16, the poll read back.
Target 18: rallly's own upstream compose runs postgres:18-alpine at /var/lib/postgresql at
v4.11.1 (09 decision 42's rule). The data mount moves with the image (pg_mounts_for).
Evidence: felhom.eu/documentation/audits/pg-last-six-2026-09-30/
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Proven on 9202 with controller 0.279.0 (drill 5ac5daf): the default no longer logs in, the generated first
password from the app page does, a wrong one does not. Copy freeze: bookstack's new/changed strings signed off.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Proven on 9202 with controller 0.279.0 (drill catalog c4cfb83): the default no longer authenticates, the
generated first password from the app page does, a wrong one does not; a restore keeps it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Written by upgrade-test.py --write-ladder: bench (harness v4) healthy in 217 s, the world-data file present and reused ("Latest data already imported"), 6 migration lines, seed read back, memory watch anon peak 73.0 %, 0 restarts; box 9202 through the guarded Update in 204 s with the product's default 5-min wait (a first try under the drill's 90 s wait was UNDONE correctly). Same commit: (1) the frontend healthcheck override is dropped — v0.13.0 brings its own (node at /usr/bin/node); (2) a world-data file that does not parse to its end is renamed aside before start so download-countries fetches it again — with an import pending, the image's own start crash-looped 10 times on IncompleteJSONError, with the step it was set aside, downloaded and healthy. Evidence felhom.eu/documentation/audits/version-travel-2026-09-26/C/.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Written by upgrade-test.py --write-ladder: bench converted in 23.6 s, check equal over 100 tables, seed read back, memory watch anon peak 78.7 % (the app, not the engine), 0 kills; box 9202 converted through the guarded Update in 52.3 s, seed read back. The superseded 16 step keeps its definition in steps/. Evidence felhom.eu/documentation/audits/version-travel-2026-09-26/B/.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
"Upgrading nextcloud from 34.0.1.2 ..." then the seeded user read back
through occ (a never-created uid reads absent on the same call). The
bench's drive tree changed under the app (its appdata), so the entry is
marked files_may_change: an automatic update needs a fresh copy of the
files (09 decision 13). Memory: nextcloud's own 7.9 % (cgroup 100 % =
file cache), db 24.2 %; 0 kills. Abort refuses ("the version of the data
is higher") — the undo, not the old image, is the way back. Box (9202):
done, read back, R-626 clean. 09 decision 21.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-22's move took immich-server to v3.2.2 and left machine-learning at
v3.0.3; this step aligns them. Bench: the admin + an album (its own API)
read back; memory watch 12 053 requests all 200: server 51.4 %, ML 10.2 %,
postgres 35.1 % own memory (its cgroup peak 100 % is file cache, decision
22); 0 kills; abort starts-and-serves. Box (9202): done, read back, R-626
clean. 09 decision 21.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
The engine alone moves (R-450: an engine change gets its own edge). Asked of
the engine, not the log: "already upgraded to 11.8.9-MariaDB" after
MARIADB_AUTO_UPGRADE's mariadb-upgrade ran (Phase 1/8 on the box). Bench:
the seeded user read back; memory: kimai 6 % own (cgroup 100 %, cache),
kimai-db 37.9 %; 0 kills; abort starts-and-serves. Box (9202): done, read
back, R-626 clean. 09 decision 21.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Bench at 512M: proven but memory_tight (its own memory 94.6 %); per the
bar the limit is raised in the same commit and the watch re-run once:
at 768M 60 %, 0 kills, 11 940 requests all 200; abort starts-and-serves.
Box (9202, guarded Update, at 512M): done, read back, R-626 clean.
The komga fixture was fixed tonight (users/me lives at /api/v2).
09 decision 21.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
1.15 (v1.15.2 today; its digest is in the entry) moved every page under /-/
(/login now 404, /-/login serves) and marks its session cookie Secure —
the account read back through its own page, a never-created user 404.
Memory: app 77.8 % of 128M, cgroup peak 100 % (file cache), 0 kills.
Box (9202): done, read back, R-626 clean. R-654 for the /login bookmark.
09 decision 21.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
update_ladder: in .felhom.yml, one JSON entry per line (spiked live on
controller v0.266.0 and v0.267.0 first). Two gates: check-test-record.py
(static, CI too) and check-test-record-move.py (history + registry for
moved refs only). 16 decoys, 3 red-proofs. The ONLY writer is
upgrade-test.py --write-ladder (bench AND box proven, digests resolved).
Harness v3: box fixtures on the bench, files_may_change.
Backfill: the 21 moves of 2026-09-22, 21 proven from their records.
No image: line moved.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
No image: line moved.
paperless-ngx has no container named after its stack, so its probe had NEVER run on any box.
immich has four immich-* containers and no exact match, so the old first-prefix rule picked
whichever came first - possibly the database.
The gate now resolves the target by the same four rules as findProbeContainerMeta: exact name,
explicit container, a UNIQUE prefix, else refuse - and refusing is right, because verifying waits
on this probe and a successful update of such an app gets stopped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Walked through the product's own guarded Update on scratch guest 9202 during the
twenty-eight drill, seeded and read back through the app's own front door.
Evidence: felhom.eu/documentation/audits/the-28-2026-09-22/apps/termix/verdict.json
catalog_since -> 2026-09-22 (R-452).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Walked through the product's own guarded Update on scratch guest 9202 during the
twenty-eight drill, seeded and read back through the app's own front door.
Evidence: felhom.eu/documentation/audits/the-28-2026-09-22/apps/sonarr/verdict.json
catalog_since -> 2026-09-22 (R-452).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Walked through the product's own guarded Update on scratch guest 9202 during the
twenty-eight drill, seeded and read back through the app's own front door.
Evidence: felhom.eu/documentation/audits/the-28-2026-09-22/apps/radarr/verdict.json
catalog_since -> 2026-09-22 (R-452).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
Walked through the product's own guarded Update on scratch guest 9202 during the
twenty-eight drill, seeded and read back through the app's own front door.
Evidence: felhom.eu/documentation/audits/the-28-2026-09-22/apps/immich/verdict.json
catalog_since -> 2026-09-22 (R-452).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
admin