felhom.eu/documentation/audits/PROBE-FIX-2026-09-22.md

THE MORNING AFTER — probe fix, gate, and the promotion train, 2026-09-22

Evidence: probe-fix-2026-09-22/. The night this follows: DRILL-update-night-2026-09-21.md.

---

Not done, or changed from the brief

Read this first. Everything else in this document is a claim; this section is where the claims are bounded.

1. The brief said "the fourteen proven versions". FIFTEEN moved. The night's fourteen included nextcloud's MariaDB engine move; tandoor was re-walked today and became the fifteenth. Both are named below with why.

2. I nearly dropped nextcloud's engine move on a wrong assumption, and the gate corrected me. I assumed check-engine-major.py would refuse mariadb:11.6 -> 12.3 and had written it up as "dropped, named". Running the gate against that exact commit instead of assuming it ALLOWED the move by name, citing R-469 (Slice 4 shipped; MARIADB_AUTO_UPGRADE=1 is already in that template, verified by reading it). The brief forbade PostgreSQL engine moves; this is MariaDB and it was proven end to end in 217.4 s. It moved. This is the second time in two days that running the instrument beat reasoning about it — see R-628.

3. The first push of the fifteen moves FAILED CI, and I found it by checking rather than by being told. Job 877 on 15d7c2b read conclusion: failure: the CI runner has no PyYAML and the new gate answered INCONCLUSIVE, which the runner rightly refuses to call a pass. Fixed with a degraded line-reader mode and five more decoy cases; job 878 on 1ad1f34 is success. The moves themselves were never in doubt — the gate that shipped beside them was.

4. bookstack's phase trace on demo-hp is INCOMPLETE and the reason is my own instrument. A 115-second probe run, meant only to list which apps were installed, pressed the Update as designed and then hit its timeout mid-update. Its phases are recorded to +21.6 s starting; the rest was read off the box afterwards (done, pin and installed both at 26.05.5). The second, full run then pressed Update on an already-current app and completed in 3.1 s, moving nothing. That second trace is real and is reported, but it is not a 26.05.2 -> 26.05.5 trace. Said here rather than presented as one.

5. The brief asked me to "list templates the night's sweep could not judge". The honest answer needed three categories, not one — 20 with a verdict record, 4 deployed as props with no edge walked, 1 deployed only to measure its probe, and 28 never deployed at all. Filed as R-632 with the machine-readable list.

6. Two things the brief could not have known, both found by the new gate and both left OPEN: paperless-ngx's health probe has never run on any box (R-630), and five templates cannot be judged by any static rule (R-631). Neither is fixed here: R-630's fix is product code or a container rename on live boxes, and the brief forbade product code.

One brief claim that turned out wrong: none. All three probe faults the brief named were verified against the files before any edit and all three were exactly as stated — tandoor's compose line 42 dials 127.0.0.1:80/accounts/login/, zipline's line 38 dials /api/healthcheck, wger's line 42 dials 127.0.0.1:8000.

---

Part 1 — the probes

1.1 The fix

One commit, app-catalog-felhom.eu@793c4fb. No image: line moved, so no catalog_since moved.

app	was	is	the oracle that was already in the file
tandoor	port 8080	port 80	wget --spider -q http://127.0.0.1:80/accounts/login/
zipline	path /api/health	path /api/healthcheck	http.get('http://127.0.0.1:3000/api/healthcheck', …)
wger	port 80	port 8000	wget --spider -q http://127.0.0.1:8000

1.2 Red-proofed live on 9202, through the product, in both directions

Deployed at the live (unfixed) pin first. Evidence probe-fix-2026-09-22/probe-before-observe.json, front-door-before.json, probe-after.json.

	tandoor	zipline	wger
BEFORE — controller state	unhealthy	unhealthy	unhealthy
BEFORE — the app page, HU	Nem egészséges	Nem egészséges	Nem egészséges
BEFORE — the app page, EN	Not healthy	Not healthy	Not healthy
BEFORE — docker's own healthcheck	healthy, healthy	healthy, healthy	healthy
BEFORE — the front door, followed	200 Login / Sign In	200 Zipline	200 wger Workout Manager
AFTER — controller state	running	running	running
AFTER — the app page, HU / EN	Fut / Running	Fut / Running	Fut / Running

The fix arrived through the real sync: POST /api/sync answered „Sablonok frissítve — frissítve: tandoor, wger, zipline", and all three read running at the next poll — no redeploy, no container restart.

The badge I quote is the HEALTH word, not the version badge. The version badge reads Naprakész / Up to date in both states and would have proved nothing; a first pass captured it and it was corrected. The scan strips <script> and <style> first (the same page carries those words inside a script block).

1.2b tandoor's edge, re-walked

Via the drill catalog (09 §6.5 — git.repo_url and rm -rf <data>/catalog-cache, R-615). Evidence probe-fix-2026-09-22/tandoor-rerun/.

	2026-09-21, before the fix	2026-09-22, after
entered verifying	+58.4 s	+5.2 s
outcome	failed at +361.9 s, app STOPPED	done at +41.1 s
seed read back after	UNUSABLE — No such container: tandoor	found = True
installed_images	rolled back to 2.6.13	2.6.15
containers	none	both running, restarts=0

Same app, same versions, same button. The only change is one port number. tandoor's verdict moves failed -> proven and it is on the live catalog.

Teardown, stated: controller.yaml restored, git.repo_url read back as the live catalog, <data>/data/catalog-cache re-cloned from the live remote and quoted (probe-fix-2026-09-22/teardown-catalog-pointer.txt), drill repo reset to the live main.

1.3 The gate

app-catalog-felhom.eu/scripts/check-probe-matches-compose.py, registered in catalog_gates.py as a --fast row, so it runs in the pre-push hook and in CI.

The oracle was already in every template. The probed service's own compose healthcheck.test dials the app on loopback. The gate resolves the probed service exactly as findProbeContainer does (healthprobe.go:297) and compares.

Two verdicts, and the narrower one is a measurement. A wrong PORT refuses for every check type — nothing listens, every dial is refused. A wrong PATH refuses only where the probe can fail on it: probeHTTP calls any response healthy for type: http, and for type: api with no expect block (healthprobe.go:253-262). So the path rule convicts zipline and merely warns about home-assistant, whose badge is right today and breaks the day someone adds an expect.

Red-proofs, four: on the unfixed tree (d4392e2a10f4 — the hash the brief names, and the repo's HEAD when I started) it refuses all three; on the fixed tree it passes; a clean app given a wrong port is refused; an api+expect probe given a wrong path is refused. Each of the three real faults is re-introduced by the test rather than read off the tree, because reading them off the tree would have gone green for the wrong reason the moment R-618 was fixed.

Decoys, both ways, ten in total (suite now 56 cases): the port moved in a YAML comment; Traefik's loadbalancer.server.port; a published ports: mapping; a NON-probed sidecar's own healthcheck; a path mismatch on a probe that cannot fail on it; and five more with PyYAML shadowed out — the mode CI actually runs — where the three faults must still be refused.

--root=<dir> exists so the suite judges its own clone. Without it every case would read identical bytes and pass for nothing: the constant-for-measurement shape this repo's suite already warns about.

Run over all 53 — six WARNINGS, none a conviction and none a pass:

app	why no verdict
paperless-ngx	no container name matches the stack name — the probe has never run (R-630)
vikunja	the probed service has no compose healthcheck — no oracle
crafty-controller, mealie, uptime-kuma	the healthcheck runs through a python/script helper — no loopback URL to read
home-assistant	path mismatch on a probe that cannot fail on it — right today, fragile (R-631)

1.4 What the night's sweep could not judge

28 of 53 templates have never been deployed by any drill (R-632). Full list in the register and in probe-fix-2026-09-22/not-judged.json. Also: 4 were deployed as props in the bad-days legs with no edge walked (bentopdf, glance, uptime-kuma, wishlist), and 1 only to measure its probe (wger).

---

Part 2 — the promotion train

The moves

Fifteen commits, one per app, catalog_since: "2026-09-22" on each. Every gate run before each commit; nothing forced, nothing dropped. check-image-resolvable.py confirms all 18 unique pins still exist upstream.

app	move	commit
actualbudget	26.7.0 -> 26.9.0	2060032
audiobookshelf	2.35.1 -> 2.36.1	6525b8e
bookstack	26.05.2 -> 26.05.5	ac4828f
docmost	0.95.0 -> 0.96.0	6d8cd87
grafana	13.1.0 -> 13.2.2	068f445
home-assistant	2026.7.2 -> 2026.9.3	4405f12
mealie	v3.20.1 -> v3.27.0	008348b
n8n	2.31.3 -> 2.40.5	4132e35
navidrome	0.63.2 -> 0.64.0	7708a04
papra	26.6.1 -> 26.6.2	e96887e
privatebin	2.0.5 -> 2.0.6	f547f16
romm	5.0.0 -> 5.3.0	15f9ebf
tandoor	2.6.13 -> 2.6.15	c3807c7
vikunja	2.3.0 -> 2.6.0	22f598b
nextcloud	ENGINE mariadb:11.6 -> mariadb:12.3	39374d5

The guarded Update, pressed on both demo boxes

demo-hp — https://192.168.0.138

app	badge before (HU / EN)	phases	pin after	installed after	front-door state
bookstack	Naprakész / Up to date	safety-dump +0.0s → pulling +1.1s → starting +2.1s → done +3.1s	lscr.io/linuxserver/bookstack:26.05.5	lscr.io/linuxserver/bookstack:26.05.5	running
docmost	Frissítés elérhető — ma / Update available — today	safety-dump +0.0s → pulling +1.0s → starting +79.0s → verifying +91.3s → done +106.6s	docmost/docmost:0.96.0	docmost/docmost:0.96.0	running
privatebin	Frissítés elérhető — ma / Update available — today	safety-dump +0.0s → pulling +1.1s → starting +4.1s → verifying +5.2s → done +15.4s	privatebin/pdo:2.0.6	privatebin/pdo:2.0.6	running
romm	Frissítés elérhető — ma / Update available — today	safety-dump +0.0s → pulling +1.1s → starting +13.3s → verifying +40.0s → done +74.8s	rommapp/romm:5.3.0	rommapp/romm:5.3.0	running

Page after the update, as the household reads it:

• bookstack — HU: ← Alkalmazások BookStack Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Egyszerű wiki platform - polcok, könyvek, fejezetek és oldalak ~150M RAM productivity Pi k EN: ← Apps BookStack Running Up to date Open ↗ Log Export Settings A simple wiki platform - shelves, books, chapters and pages ~150M RAM productivity Runs on Pi Move to anoth
• docmost — HU: *← Alkalmazások Docmost Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Modern wiki és dokumentáció platform Notion-szerű szerkesztővel ~200M RAM productivity Csak * EN: ← Apps Docmost Running Up to date Open ↗ Log Export Settings A modern wiki and documentation platform with a Notion-like editor ~200M RAM productivity x86 only Move to an
• privatebin — HU: ← Alkalmazások PrivateBin Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Titkosított szöveg megosztás - a szerver nem látja a tartalmat ~30M RAM security Pi kompa EN: ← Apps PrivateBin Running Up to date Open ↗ Log Export Settings Encrypted text sharing - the server never sees the content ~30M RAM security Runs on Pi Move to another st
• romm — HU: *← Alkalmazások RomM Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Retró játékgyűjtemény kezelő, böngésző és lejátszó ~300M RAM media HDD szükséges Csak x86 Hova * EN: *← Apps RomM Running Up to date Open ↗ Log Export Settings Retro game collection manager, browser and player ~300M RAM media Needs a hard drive x86 only Where do I put my *

demo-felhom — https://192.168.0.149

Not installed on this box, so not pressed: bookstack, docmost, privatebin, romm.

app	badge before (HU / EN)	phases	pin after	installed after	front-door state

Page after the update, as the household reads it:

The second box, honestly: the four moved apps were pressed on one demo box, not two, because they exist on only one. Both boxes are on catalog 1ad1f34.

---

What this leaves open

row	what
R-630 (new, P2)	paperless-ngx's probe has never run on any box. Not fixed: both candidate fixes are product code or rename a running container on live boxes. Not measured, and said so: what verifying does for a stack with no probe target.
R-631 (new, P3)	five templates no static rule can judge; home-assistant is right only because its check type cannot fail.
R-632 (new, P3)	28 of 53 templates have never been deployed by any drill. This is the rotation's queue.
R-618	CLOSED — three probes fixed, red-proofed live both ways, gate shipped with ten decoys, tandoor re-walked failed -> proven.
R-462	widened: the night's 14 proven / 3 failed / 4 inconclusive becomes 15 proven / 2 failed / 4 inconclusive, and all fifteen are now on the live catalog.

unproven.py --summary did not move: 55 claims, 20 walked, 35 not walked — the same as 2026-08-22. Nothing here changed a claim's status; it changed the evidence behind one row of the capability map.

Fences

Peti's box is parked and was not touched. It receives a badge if it ever syncs and nothing else. Nothing ran on DooPlex beyond ordinary pushes, nothing on ep0. felhom-controller, felhom-agent and the hub were read only — no product code was written. No golden, no bake, no vouch, no --no-verify, no branch. local-lvm untouched, no prune, tester-1 never reset.

Repository state

repo	head	CI
app-catalog-felhom.eu	1ad1f34	job 878 success (job 877 on 15d7c2b was failure — see "Not done" §3)
felhom.eu	this commit	all 16 gates green locally