admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3208cb208366120288bbff98273fba76718288a4
felhom.eu/documentation/audits/PROBE-FIX-2026-09-22.md
T
admin a975cfde5b
gates / gates (push) Successful in 28s
Details
probe fix, the gate, and the promotion train (R-618 closed, R-630..632 opened) 
Part 1: tandoor/zipline/wger probes corrected in the catalog and red-proofed live on 9202 in both
directions - "Nem egeszseges" with the front door serving 200, then "Fut" after the real sync with
no redeploy. tandoor's failed edge re-walked: done at +41.1s where it was failed at +361.9s.

Part 2: fifteen proven versions on the live catalog, one commit per app; the guarded Update pressed
on four apps on demo-hp, all four done.

Opened: R-630 (paperless-ngx's probe has never run on any box - a silent absence, worse than the
wrong probe that was found in one night), R-631 (five templates no static rule can judge),
R-632 (28 of 53 templates never deployed by any drill). Closed: R-618.

Register 318 -> 321. No product code.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-22 11:10:58 +02:00

248 lines
15 KiB
Markdown
Raw Blame History

# THE MORNING AFTER — probe fix, gate, and the promotion train, 2026-09-22
Evidence: `probe-fix-2026-09-22/`. The night this follows: `DRILL-update-night-2026-09-21.md`.
---
## Not done, or changed from the brief
**Read this first. Everything else in this document is a claim; this section is where the claims
are bounded.**
1. **The brief said "the fourteen proven versions". FIFTEEN moved.** The night's fourteen included
`nextcloud`'s MariaDB engine move; tandoor was re-walked today and became the fifteenth. Both are
named below with why.
2. **I nearly dropped `nextcloud`'s engine move on a wrong assumption, and the gate corrected me.**
I assumed `check-engine-major.py` would refuse `mariadb:11.6 -> 12.3` and had written it up as
"dropped, named". Running the gate against that exact commit instead of assuming it ALLOWED the
move by name, citing **R-469** (Slice 4 shipped; `MARIADB_AUTO_UPGRADE=1` is already in that
template, verified by reading it). The brief forbade **PostgreSQL** engine moves; this is MariaDB
and it was proven end to end in 217.4 s. It moved. **This is the second time in two days that
running the instrument beat reasoning about it** — see R-628.
3. **The first push of the fifteen moves FAILED CI, and I found it by checking rather than by being
told.** Job **877** on `15d7c2b` read `conclusion: failure`: the CI runner has no PyYAML and the
new gate answered INCONCLUSIVE, which the runner rightly refuses to call a pass. Fixed with a
degraded line-reader mode and five more decoy cases; job **878** on `1ad1f34` is `success`.
**The moves themselves were never in doubt — the gate that shipped beside them was.**
4. **bookstack's phase trace on demo-hp is INCOMPLETE and the reason is my own instrument.** A
115-second probe run, meant only to list which apps were installed, pressed the Update as
designed and then hit its timeout mid-update. Its phases are recorded to `+21.6 s starting`; the
rest was read off the box afterwards (`done`, pin and installed both at `26.05.5`). The second,
full run then pressed Update on an already-current app and completed in **3.1 s**, moving
nothing. That second trace is real and is reported, but it is **not** a `26.05.2 -> 26.05.5`
trace. Said here rather than presented as one.
5. **The brief asked me to "list templates the night's sweep could not judge". The honest answer
needed three categories, not one** — 20 with a verdict record, 4 deployed as props with no edge
walked, 1 deployed only to measure its probe, and **28 never deployed at all**. Filed as R-632
with the machine-readable list.
6. **Two things the brief could not have known, both found by the new gate and both left OPEN:**
`paperless-ngx`'s health probe has never run on any box (R-630), and five templates cannot be
judged by any static rule (R-631). Neither is fixed here: R-630's fix is product code or a
container rename on live boxes, and the brief forbade product code.
**One brief claim that turned out wrong:** none. All three probe faults the brief named were
verified against the files before any edit and all three were exactly as stated — tandoor's compose
line 42 dials `127.0.0.1:80/accounts/login/`, zipline's line 38 dials `/api/healthcheck`, wger's
line 42 dials `127.0.0.1:8000`.
---
## Part 1 — the probes
### 1.1 The fix
One commit, `app-catalog-felhom.eu@793c4fb`. No `image:` line moved, so no `catalog_since` moved.
| app | was | is | the oracle that was already in the file |
|---|---|---|---|
| tandoor | port `8080` | port `80` | `wget --spider -q http://127.0.0.1:80/accounts/login/` |
| zipline | path `/api/health` | path `/api/healthcheck` | `http.get('http://127.0.0.1:3000/api/healthcheck', …)` |
| wger | port `80` | port `8000` | `wget --spider -q http://127.0.0.1:8000` |
### 1.2 Red-proofed live on 9202, through the product, in both directions
Deployed at the **live (unfixed) pin** first. Evidence `probe-fix-2026-09-22/probe-before-observe.json`,
`front-door-before.json`, `probe-after.json`.
| | tandoor | zipline | wger |
|---|---|---|---|
| **BEFORE — controller state** | `unhealthy` | `unhealthy` | `unhealthy` |
| **BEFORE — the app page, HU** | `Nem egészséges` | `Nem egészséges` | `Nem egészséges` |
| **BEFORE — the app page, EN** | `Not healthy` | `Not healthy` | `Not healthy` |
| **BEFORE — docker's own healthcheck** | `healthy, healthy` | `healthy, healthy` | `healthy` |
| **BEFORE — the front door, followed** | **200** `Login / Sign In` | **200** `Zipline` | **200** `wger Workout Manager` |
| **AFTER — controller state** | `running` | `running` | `running` |
| **AFTER — the app page, HU / EN** | `Fut` / `Running` | `Fut` / `Running` | `Fut` / `Running` |
The fix arrived through the **real sync**: `POST /api/sync` answered
*„Sablonok frissítve — frissítve: tandoor, wger, zipline"*, and all three read `running` at the next
poll — **no redeploy, no container restart**.
**The badge I quote is the HEALTH word, not the version badge.** The version badge reads
`Naprakész` / `Up to date` in both states and would have proved nothing; a first pass captured it
and it was corrected. The scan strips `<script>` and `<style>` first (the same page carries those
words inside a script block).
### 1.2b tandoor's edge, re-walked
Via the drill catalog (`09` §6.5 — `git.repo_url` **and** `rm -rf <data>/catalog-cache`, R-615).
Evidence `probe-fix-2026-09-22/tandoor-rerun/`.
| | 2026-09-21, before the fix | 2026-09-22, after |
|---|---|---|
| entered `verifying` | +58.4 s | **+5.2 s** |
| outcome | **`failed` at +361.9 s**, app STOPPED | **`done` at +41.1 s** |
| seed read back after | **UNUSABLE** — `No such container: tandoor` | **found = True** |
| `installed_images` | rolled back to `2.6.13` | `2.6.15` |
| containers | none | both `running`, `restarts=0` |
**Same app, same versions, same button. The only change is one port number.** tandoor's verdict
moves `failed -> proven` and it is on the live catalog.
**Teardown, stated:** `controller.yaml` restored, `git.repo_url` read back as the live catalog,
`<data>/data/catalog-cache` re-cloned from the live remote and quoted
(`probe-fix-2026-09-22/teardown-catalog-pointer.txt`), drill repo reset to the live `main`.
### 1.3 The gate
`app-catalog-felhom.eu/scripts/check-probe-matches-compose.py`, registered in `catalog_gates.py` as
a **`--fast`** row, so it runs in the pre-push hook and in CI.
**The oracle was already in every template.** The probed service's own compose `healthcheck.test`
dials the app on loopback. The gate resolves the probed service exactly as `findProbeContainer`
does (`healthprobe.go:297`) and compares.
**Two verdicts, and the narrower one is a measurement.** A wrong PORT refuses for every check type —
nothing listens, every dial is refused. A wrong PATH refuses **only** where the probe can fail on
it: `probeHTTP` calls any response healthy for `type: http`, and for `type: api` with no `expect`
block (`healthprobe.go:253-262`). So the path rule convicts `zipline` and merely warns about
`home-assistant`, whose badge is right today and breaks the day someone adds an `expect`.
**Red-proofs, four:** on the unfixed tree (`d4392e2a10f4` — the hash the brief names, and the repo's
HEAD when I started) it refuses all three; on the fixed tree it passes; a clean app given a wrong
port is refused; an `api`+`expect` probe given a wrong path is refused. Each of the three real
faults is **re-introduced by the test** rather than read off the tree, because reading them off the
tree would have gone green for the wrong reason the moment R-618 was fixed.
**Decoys, both ways, ten in total** (suite now **56 cases**): the port moved in a YAML comment;
Traefik's `loadbalancer.server.port`; a published `ports:` mapping; a NON-probed sidecar's own
healthcheck; a path mismatch on a probe that cannot fail on it; and five more with **PyYAML shadowed
out** — the mode CI actually runs — where the three faults must still be refused.
**`--root=<dir>`** exists so the suite judges its own clone. Without it every case would read
identical bytes and pass for nothing: the constant-for-measurement shape this repo's suite already
warns about.
**Run over all 53 — six WARNINGS, none a conviction and none a pass:**
| app | why no verdict |
|---|---|
| `paperless-ngx` | no container name matches the stack name — **the probe has never run** (R-630) |
| `vikunja` | the probed service has no compose healthcheck — no oracle |
| `crafty-controller`, `mealie`, `uptime-kuma` | the healthcheck runs through a python/script helper — no loopback URL to read |
| `home-assistant` | path mismatch on a probe that cannot fail on it — right today, fragile (R-631) |
### 1.4 What the night's sweep could not judge
**28 of 53 templates have never been deployed by any drill** (R-632). Full list in the register and
in `probe-fix-2026-09-22/not-judged.json`. Also: 4 were deployed as props in the bad-days legs with
no edge walked (`bentopdf`, `glance`, `uptime-kuma`, `wishlist`), and 1 only to measure its probe
(`wger`).
---
## Part 2 — the promotion train
### The moves
Fifteen commits, one per app, `catalog_since: "2026-09-22"` on each. Every gate run before each commit; nothing forced, nothing dropped. `check-image-resolvable.py` confirms all 18 unique pins still exist upstream.
| app | move | commit |
|---|---|---|
| `actualbudget` | 26.7.0 -> 26.9.0 | `2060032` |
| `audiobookshelf` | 2.35.1 -> 2.36.1 | `6525b8e` |
| `bookstack` | 26.05.2 -> 26.05.5 | `ac4828f` |
| `docmost` | 0.95.0 -> 0.96.0 | `6d8cd87` |
| `grafana` | 13.1.0 -> 13.2.2 | `068f445` |
| `home-assistant` | 2026.7.2 -> 2026.9.3 | `4405f12` |
| `mealie` | v3.20.1 -> v3.27.0 | `008348b` |
| `n8n` | 2.31.3 -> 2.40.5 | `4132e35` |
| `navidrome` | 0.63.2 -> 0.64.0 | `7708a04` |
| `papra` | 26.6.1 -> 26.6.2 | `e96887e` |
| `privatebin` | 2.0.5 -> 2.0.6 | `f547f16` |
| `romm` | 5.0.0 -> 5.3.0 | `15f9ebf` |
| `tandoor` | 2.6.13 -> 2.6.15 | `c3807c7` |
| `vikunja` | 2.3.0 -> 2.6.0 | `22f598b` |
| `nextcloud` | **ENGINE** `mariadb:11.6` -> `mariadb:12.3` | `39374d5` |
### The guarded Update, pressed on both demo boxes
#### `demo-hp` — https://192.168.0.138
| app | badge before (HU / EN) | phases | pin after | installed after | front-door state |
|---|---|---|---|---|---|
| `bookstack` | `Naprakész` / `Up to date` | `safety-dump` +0.0s → `pulling` +1.1s → `starting` +2.1s → `done` +3.1s | `lscr.io/linuxserver/bookstack:26.05.5` | `lscr.io/linuxserver/bookstack:26.05.5` | `running` |
| `docmost` | `Frissítés elérhető — ma` / `Update available — today` | `safety-dump` +0.0s → `pulling` +1.0s → `starting` +79.0s → `verifying` +91.3s → `done` +106.6s | `docmost/docmost:0.96.0` | `docmost/docmost:0.96.0` | `running` |
| `privatebin` | `Frissítés elérhető — ma` / `Update available — today` | `safety-dump` +0.0s → `pulling` +1.1s → `starting` +4.1s → `verifying` +5.2s → `done` +15.4s | `privatebin/pdo:2.0.6` | `privatebin/pdo:2.0.6` | `running` |
| `romm` | `Frissítés elérhető — ma` / `Update available — today` | `safety-dump` +0.0s → `pulling` +1.1s → `starting` +13.3s → `verifying` +40.0s → `done` +74.8s | `rommapp/romm:5.3.0` | `rommapp/romm:5.3.0` | `running` |
Page after the update, as the household reads it:
- **`bookstack`** — HU: *← Alkalmazások BookStack Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Egyszerű wiki platform - polcok, könyvek, fejezetek és oldalak ~150M RAM productivity Pi k*
EN: *← Apps BookStack Running Up to date Open ↗ Log Export Settings A simple wiki platform - shelves, books, chapters and pages ~150M RAM productivity Runs on Pi Move to anoth*
- **`docmost`** — HU: *← Alkalmazások Docmost Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Modern wiki és dokumentáció platform Notion-szerű szerkesztővel ~200M RAM productivity Csak *
EN: *← Apps Docmost Running Up to date Open ↗ Log Export Settings A modern wiki and documentation platform with a Notion-like editor ~200M RAM productivity x86 only Move to an*
- **`privatebin`** — HU: *← Alkalmazások PrivateBin Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Titkosított szöveg megosztás - a szerver nem látja a tartalmat ~30M RAM security Pi kompa*
EN: *← Apps PrivateBin Running Up to date Open ↗ Log Export Settings Encrypted text sharing - the server never sees the content ~30M RAM security Runs on Pi Move to another st*
- **`romm`** — HU: *← Alkalmazások RomM Fut Naprakész Megnyitás ↗ Napló Exportálás Beállítások Retró játékgyűjtemény kezelő, böngésző és lejátszó ~300M RAM media HDD szükséges Csak x86 Hova *
EN: *← Apps RomM Running Up to date Open ↗ Log Export Settings Retro game collection manager, browser and player ~300M RAM media Needs a hard drive x86 only Where do I put my *
#### `demo-felhom` — https://192.168.0.149
**Not installed on this box, so not pressed:** `bookstack`, `docmost`, `privatebin`, `romm`.
| app | badge before (HU / EN) | phases | pin after | installed after | front-door state |
|---|---|---|---|---|---|
Page after the update, as the household reads it:
**The second box, honestly:** the four moved apps were pressed on **one** demo box, not two, because
they exist on only one. Both boxes are on catalog `1ad1f34`.
---
## What this leaves open
| row | what |
|---|---|
| **R-630** (new, P2) | `paperless-ngx`'s probe has never run on any box. Not fixed: both candidate fixes are product code or rename a running container on live boxes. **Not measured, and said so:** what `verifying` does for a stack with no probe target. |
| **R-631** (new, P3) | five templates no static rule can judge; `home-assistant` is right only because its check type cannot fail. |
| **R-632** (new, P3) | 28 of 53 templates have never been deployed by any drill. This is the rotation's queue. |
| **R-618** | **CLOSED** — three probes fixed, red-proofed live both ways, gate shipped with ten decoys, tandoor re-walked `failed -> proven`. |
| **R-462** | widened: the night's `14 proven / 3 failed / 4 inconclusive` becomes **15 proven / 2 failed / 4 inconclusive**, and all fifteen are now on the live catalog. |
**`unproven.py --summary` did not move:** 55 claims, 20 walked, 35 not walked — the same as
2026-08-22. Nothing here changed a claim's status; it changed the evidence behind one row of the
capability map.
## Fences
**`Peti's box` is parked and was not touched.** It receives a badge if it ever syncs and nothing
else. Nothing ran on DooPlex beyond ordinary pushes, nothing on `ep0`. `felhom-controller`,
`felhom-agent` and the hub were read only — **no product code was written.** No golden, no bake, no
vouch, no `--no-verify`, no branch. `local-lvm` untouched, no `prune`, `tester-1` never reset.
## Repository state
| repo | head | CI |
|---|---|---|
| `app-catalog-felhom.eu` | `1ad1f34` | job **878** `success` (job **877** on `15d7c2b` was `failure` — see "Not done" §3) |
| `felhom.eu` | this commit | all 16 gates green locally |
Reference in New Issue View Git Blame Copy Permalink