felhom.eu/documentation/audits/DRILL-the-28-2026-09-22.md

THE TWENTY-EIGHT — every app no update drill had ever touched, 2026-09-22

Evidence: the-28-2026-09-22/. What came before: DRILL-update-night-2026-09-21.md (21 edges, 19 apps) and PROBE-FIX-2026-09-22.md (the probe fix and the fifteen moves).

---

Not done, or changed from the brief

Read this first.

Claims in the brief that turned out wrong

1. There is no requires: key in .felhom.yml. The brief said "A .felhom.yml requires: (HDD, x86) that 9202 cannot meet → recorded, not forced." No template has such a key. The constraints live under resources: as needs_hdd and pi_compatible. And none of them excluded anything: 9202 is x86_64 with /mnt/felhom-drives/scratch_hdd mounted, so every one of the 28 was installable on those grounds. Nothing was skipped for a resource reason.

2. The brief's file-leg list is wrong, and it told me where to check. It grouped "immich, jellyfin, plex, emby, komga, calibre-web, gokapi, homebox, gramps-web" as file-leg apps. Read from 07-backup-architecture.md §6.2 — which the brief itself says to read rather than re-derive — only four of the 28 are class A (at least one readable file leg): calibre-web, immich, komga, paperless-ngx. jellyfin, plex and emby are class B precisely because their only bind is a :ro media mount, which ClassifyBinds excludes; gokapi, homebox and gramps-web keep everything in named volumes. The table below uses 07 §6.2's classes, not the brief's.

3. The brief's database list is incomplete. It named calcom, claper, outline, paperless-ngx, rallly and sparkyfitness for PostgreSQL and kimai for MariaDB. immich also carries PostgreSQL (a vectorchord variant) and redis, and wanderer carries meilisearch — two apps the brief put in the "file-leg" group actually run their own datastore.

4. One of the 28 cannot be installed at all, by design. plant-it is lifecycle: "abandoned", and the product's lifecycle gate refused the deploy with 409 „Ez az alkalmazás jelenleg nem telepíthető." That is correct behaviour, measured live for the first time. It is the only lifecycle-gated template in the whole catalog of 53.

Claims in the brief that were verified true, by looking

• The drill repo's Actions are off (R-629). Read from the API: has_actions: false, private: true. And measured rather than trusted: 47 CI jobs before the reset push, 47 after — the push produced no run and no mail.
• repoint_drill.py still works after the catalog moved. 9202's cache now reads origin …/app-catalog-drill.git at 1ad1f34.
• 9202 has the capacity. 25.9 GB RAM (23.5 free), 28 GB free on /, 842 GB on the scratch drive. The root disk is the binding constraint, so each app's images are removed by name after its verdict — never prune (rule 3). Disk held at 1.9 GB used throughout.

Changed method, named

5. A restore that is REFUSED is recorded as refused-with-a-sentence, not as a failed restore. The first version of the harness collapsed the two and mislabelled calibre-web. The product had in fact done the right thing — see the finding below — and a harness that calls a correct refusal a failure would have buried it.

6. The harness now waits for a restore to settle before removing. It did not at first, and that race produced R-633, a real defect. The race was left in the record for gokapi and fenced out afterwards, so the remaining apps measure the product rather than the harness.

7. Three bugs in tonight's own harness, each named with what it cost. A missing import re in the restore step killed the restore half for wanderer, claper, sparkyfitness and calcom. A variable named m shadowed the app's metadata and broke paperless-ngx's teardown. An empty phase list crashed on [-1] when the Update was refused before any phase existed, which cost rallly its whole walk. All four apps, and five more, were re-walked serially afterwards — and that re-walk is what corrected R-634 and produced ghost's proof. An instrument that can drop results silently is not a measurement; these dropped them loudly and were re-run.

8. Concurrency is part of the method and it changed two results. Three walks ran at once to fit 28 apps in one night. POST /api/backup/run is box-wide, so a second caller gets 409 „Mentés már folyamatban", and the Update refuses while a backup or restore is in flight (409 busy). Both refusals are the product being right and both are quoted below. But they cost ghost and rallly their edge on the first pass, and they are implicated in two of R-634's three instances. The nine re-walks were serial for exactly this reason.

Corrected 2026-09-22 (evening), from the records rather than by re-running

9. calibre-web's restore was refused-with-a-sentence, not failed. The refusal is in this app's own log.txt line 12 as a flash_error on the redirect, and its state stayed running throughout. It was recorded failed because that walk ran before the refusal-capture code was added later the same night — the document's own section "A restore that is REFUSED" already said so while the table and the record disagreed with it.

10. calcom's restore was inconclusive, not failed — and that was my harness, not the product. The restore was accepted (flash=flash.restore.started, no flash_error), the app read running at +45 s with no hold and no phase, and the classifier looked once more and saw starting — a settling state it did not list beside running/unhealthy, so it fell through to failed. The task brief supposed a different cause — "a read-back of data that was never seeded" — and that is wrong: the seed half is recorded separately and was already no route. The record's own restore_state_seen: "starting" is the evidence.

    Totals move with it: restores correctly REFUSED go from 2 to 3.

---

What this night is, in three lines

• Interventions: SEVEN — over the brief's limit of five, and six of the seven were my own harness, not the product. Three were bugs in tonight's driver that cost apps their walk and were fixed mid-run (a missing import re; a variable that shadowed the app's metadata; a crash when the Update was refused before any phase existed). Two were deliberate method changes (recording a REFUSED restore as its own verdict; making the harness wait for a restore to settle before removing). One was a waiter that deadlocked on its own command line. The seventh was the product's: three leftovers it could not clear, which a shell had to.
• 26 of 28 deployed; 6 proven; 5 inconclusive; 14 with no upstream edge; 1 failed honestly; 2 that could not be deployed — one of those by design.
• The one result that matters most: an app with no health probe at all has its working installation stopped by a successful update. paperless-ngx was healthy on all three containers; the Update ran the full five-minute health wait and then held the app, and the controller named the reason itself — no probe container. R-630 is raised to P1.

---

The two findings that are not about any single app

R-633 — a remove sent while a restore is still running reports success and leaves an orphan

gokapi was restored from its own local copy at 11:34:07 and removed at 11:34:22. POST /backup/restore answers 302 and does its work in the background; the remove tore down what existed, and the restore's own compose up then re-created the container at 11:34:24. Both calls returned success.

Twenty-five minutes later, GET /api/stacks/gokapi reads deployed: false while docker ps -a shows gokapi Restarting (1) with its full Traefik label set still attached — including traefik.http.routers.gokapi.rule: Host(.enkisfelhom.hu), a rule with an empty subdomain, because the deploy values that filled it were deleted with the app. Its own log loops „Salt for admin password invalid… password does not appear to be a SHA-1 hash" — the volume holding its config was removed correctly, so the binary can never start.

A household can press exactly those two buttons in that order. The product accepted both and the remove reported success while leaving the orphan; nothing in the alarm ladder can fire, because 08 §4 keys on stacks the controller still knows about. This is R-626's class with the mechanism finally visible — that row saw a removed navidrome come back and could not diagnose it, because the controller had restarted and its log no longer reached the moment. Here the window is seventeen seconds and both halves are in the evidence.

The harness was then fenced against its own race, so every app after gokapi measures the product. Evidence: apps/gokapi/came-back-evidence.txt.

A restore that is REFUSED is the product being right, and nearly went down as a failure

calibre-web is a class-A app (07 §6.2): it has a readable file leg, and the local Tier-1 copy does not hold it. The restore was refused, with this sentence:

„Ez a mentés nem tartalmazza az alkalmazás fájljait, ezért nem állítjuk vissza az adatbázist föléjük — a fájlok így a helyükön maradnak. A fájlok a távoli másolatból állíthatók vissza: Biztonsági mentés → Visszaállítás, „Teljes visszaállítás (fájlok + adatbázis)"."

That is exactly what 07 §6.2 predicts, it names the action that does work, and it refuses before touching anything. The first version of tonight's harness recorded it as a failed restore. A harness that calls a correct refusal a failure buries the best result of the night, so refusals are now recorded as their own verdict and the sentence is quoted.

One real upstream edge HELD honestly — outline 1.9.1 → 1.10.1

The most valuable single result after R-630, because it is the guarded update's own promise exercised on a real upstream version rather than a staged one.

phase	at
safety-dump	0.0 s
pulling	+1.1 s
starting	+64.8 s
verifying	+65.8 s
failed	+368.6 s

The app was stopped and held, and the sentence the household reads names the tier, the date and what the copy contains:

„A(z) outline frissítése 2026-09-22 14:50-kor nem sikerült, és az alkalmazás nem indult el az új verzióval. Az alkalmazás biztonsági okból leállítva marad, hogy az adatai ne sérüljenek. Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-22 14:44 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza."

The restore named in that sentence was then walked and the app came back. outline must not be promoted.

Two refusals that are the product guarding itself, and both name the blocker

• The Update refuses while a backup or restore runs: „A frissítés most nem indítható: mentés/visszaállítás folyamatban. Próbáld újra, ha befejeződött."
• A second restore refuses and NAMES the app that is blocking it: „Egy visszaállítási művelet (jellyfin) már fut, ezért most nem indítható újabb."

That second guard is exactly the fence R-633 is missing. The product already knows how to refuse a conflicting operation and how to say which one — for update and for restore. remove has no such guard, which is why a remove sent during a restore reports success and leaves an orphan.

---

The table — all twenty-eight

Classes are 07-backup-architecture.md §6.2's, not re-derived.

app	class	deployed	seeded	backup	edge	update	restore	removed clean	s	evidence
calcom	B volumes-only + postgres	yes	no route	1 copy	none upstream	—	inconclusive	yes	675.8	apps/calcom/
calibre-web	A file-leg	yes	no route	1 copy	none upstream	—	refused-with-a-sentence	yes	187.5	apps/calibre-web/
claper	B volumes-only + postgres	yes	no route	1 copy	none upstream	—	ok	yes	255.3	apps/claper/
code-server	B volumes-only	yes	route failed	1 copy	4.129.0 → 4.138.0	done	ok	yes	299.9	apps/code-server/
crafty-controller	B volumes-only	yes	no route	1 copy	4.10.7 → 4.11.0	done	ok	yes	288.5	apps/crafty-controller/
emby	B volumes-only	yes	yes	1 copy	4.10.0.20 → 4.11.0.1	done	ok	yes	198.7	apps/emby/
ghost	B volumes-only	yes	yes	1 copy	6.53.0-alpine → 6.64.0-alpine	done	ok	yes	289.6	apps/ghost/
gokapi	B volumes-only	yes	no route	1 copy	none upstream	—	ok	no	132.3	apps/gokapi/
gramps-web	B volumes-only	yes	route failed	1 copy	none upstream	—	ok	yes	334.1	apps/gramps-web/
homebox	B volumes-only	yes	route failed	1 copy	none upstream	—	ok	yes	127.9	apps/homebox/
homepage	B volumes-only	yes	no route	1 copy	none upstream	—	ok	yes	137.1	apps/homepage/
immich	A file-leg + postgres+redis	yes	yes	1 copy	v3.0.3 → v3.2.2	done	refused-with-a-sentence	yes	375.0	apps/immich/
jellyfin	B volumes-only	yes	route failed	1 copy	none upstream	—	ok	yes	220.7	apps/jellyfin/
kimai	B volumes-only + mariadb	yes	route failed	1 copy	none upstream	—	ok	yes	311.3	apps/kimai/
komga	A file-leg	yes	route failed	1 copy	1.25.0 → 1.27.1	done	ok	yes	251.9	apps/komga/
onlyoffice	B volumes-only	yes	route failed	1 copy	none upstream	—	ok	yes	227.3	apps/onlyoffice/
outline	B volumes-only + postgres+redis	yes	route failed	1 copy	1.9.1 → 1.10.1	failed	failed	yes	1389.6	apps/outline/
paperless-ngx	A file-leg + postgres+redis	yes	no route	1 copy	none upstream	—	refused-with-a-sentence	yes	253.2	apps/paperless-ngx/
plant-it	B volumes-only	no	no route	—	none upstream	—	not-attempted	yes	78.2	apps/plant-it/
plex	B volumes-only	yes	route failed	1 copy	1.41.4.9463-630c9f557 → 1.43.4.10903-e5521bd8c	done	ok	yes	262.1	apps/plex/
radarr	B volumes-only	yes	yes	1 copy	6.3.0 → 6.4.4	done	ok	yes	227.8	apps/radarr/
rallly	B volumes-only + postgres	yes	route failed	1 copy	4.11.1 → 4.15.2	done	ok	yes	230.1	apps/rallly/
recipe-importer	B volumes-only	yes	no route	1 copy	none upstream	—	ok	yes	123.0	apps/recipe-importer/
seerr	B volumes-only	yes	no route	1 copy	none upstream	—	ok	yes	218.8	apps/seerr/
sonarr	B volumes-only	yes	yes	1 copy	4.0.19 → 4.0.20	done	ok	yes	216.6	apps/sonarr/
sparkyfitness	B volumes-only + postgres	no	no route	—	none upstream	—	not-attempted	no	534.0	apps/sparkyfitness/
termix	B volumes-only	yes	yes	1 copy	2.5.0 → 2.8.0	done	ok	no	232.5	apps/termix/
wanderer	B volumes-only + meilisearch	yes	no route	1 copy	none upstream	—	ok	yes	400.5	apps/wanderer/

Totals: 14 no-edge · 6 proven · 5 inconclusive · 2 could-not-deploy · 1 failed — 28 of 28 recorded.

Read across the walk rather than down one column: 26 of 28 deployed, 6 had a non-browser route that seeded AND read back, 21 restored from their own copy, 3 were correctly REFUSED a restore, 6 proven / 1 failed on the apps that had an upstream edge, and 2 left a container behind (R-633).

---

5.2 — the templates the static gate cannot judge (R-631)

The probe gate's oracle is the probed service's own compose healthcheck. For five templates there is no such oracle, or the paths differ on a check that cannot fail. A static rule cannot settle any of them; asking the running container can. Each was deployed on 9202, its listening sockets read from inside the container, and the probe's own target dialled on the compose network — the same call the controller makes.

app	probe	what it listens on	the probe's own dial	verdict
mealie	tcp 9000	0.0.0.0:9000	200	correct
uptime-kuma	http 3001	*:3001	302	correct — http calls any response healthy, and 302 proves something answers
vikunja	api 3456 /api/v1/info expect 200	(busybox: no ss, no netstat)	200	correct — and this is the one that could have failed, because its expect block compares the code
home-assistant	api 8123 /api/ no expect	0.0.0.0:8123	401	correct today, and the 401 is the measurement that proves the warning
crafty-controller	tcp 8443	(not read — the app never reached deployed; see R-634)	ok (1 ms)	correct

home-assistant is the one to carry forward. Its probe dials /api/ and gets 401 — not 200. It reads healthy only because probeHTTP treats any response as healthy when the type is api with no expect block (healthprobe.go:253-262). Add expect: {status: 200} to that template — a change that looks like a tightening — and home-assistant goes permanently unhealthy, and every successful update of it starts stopping it. That is R-618's failure exactly, one edit away, and it is now a measured number rather than a caution.

All five are settled. crafty-controller's reading came from its own walk rather than the side job: the controller's log shows Health probe crafty-controller: TCP :8443 -> ok (1ms) twice, six minutes apart, while the app was running. The gate's WARN list is therefore not a backlog of suspects: it is four correct templates the gate honestly cannot prove, and one that is correct by accident.

---

5.1 — what the guarded Update does when NO probe exists (R-630)

paperless-ngx has no container whose name equals or begins with its stack name, so findProbeContainer returns "" and RunHealthProbes skips the stack silently. Its probe has never run on any box. The open question was what verifying — which waits on that same probe — does when there is nothing to wait on: pass at once, wait out the timeout, or hold.

It waits out the full timeout and then HOLDS, stopping a working app.

Deployed on 9202, all three containers reported healthy, the controller read running, the front door answered 302. No upstream edge exists for paperless-ngx tonight, so the Update was pressed on the same version — which is what a household does on an up-to-date app, and it still walks the whole phase machine. That difference is stated, not glossed.

phase	at
checking → safety-dump → pinning → pulling	0.0–1.1 s
starting	+2.1 s
verifying	+3.1 s
failed	+313.0 s — the app STOPPED

Afterwards: controller state stopped, front door 404.

The controller names the cause itself, so no inference was needed:

update paperless-ngx FAILED after the new version was started: not healthy: not healthy within 5m0s (last: no probe container) — stopping and HOLDING the app; the pin stays on the new version (its migration may have run)

no probe container. And the hold sentence is correct about the route back — for this class-A app it warns „csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem".

This is R-618's outcome reached by the opposite road. There a probe named a port the app does not answer; here no probe exists at all — and the static gate cannot see it, because there is nothing to compare. The gate does print it as a WARNING on every push, which is how it was found. R-630 is raised P2 → P1.

The second promotion list — for the operator, not for me

CC promotes nothing. These are proposals with the evidence beside them.

Proposed to move (6)

app	move	update took	its own migration line
emby	4.10.0.20 → 4.11.0.1	198.7 s	yes — emby | Info SqliteUserRepository: Sqlite compiler options: ATOMIC_INTRINSICS=1,COMPILER=g
ghost	6.53.0-alpine → 6.64.0-alpine	289.6 s	yes — ghost | [2026-09-22 15:13:37] INFO Stripe members-migrations skipped because it
immich	v3.0.3 → v3.2.2	375.0 s	none printed
radarr	6.3.0 → 6.4.4	227.8 s	yes — radarr | [migrations] started
sonarr	4.0.19 → 4.0.20	216.6 s	yes — sonarr | [migrations] started
termix	2.5.0 → 2.8.0	232.5 s	yes — termix | [1:37:29 PM] [INFO] [🗄️] Database layer pre-upgrade backup created [op:database_

Must NOT move, with why (6)

app	edge	why not
code-server	4.129.0 → 4.138.0	the update reached done, but this app has no non-browser data route, so nothing proves the household's data survived it
crafty-controller	4.10.7 → 4.11.0	the update reached done, but this app has no non-browser data route, so nothing proves the household's data survived it
komga	1.25.0 → 1.27.1	the update reached done, but this app has no non-browser data route, so nothing proves the household's data survived it
outline	1.9.1 → 1.10.1	the update ended failed — fixture ran and found no non-browser seed route: sign-in requires an external identity provider (OIDC/Slack/Google); no local sign-up route exists
plex	1.41.4.9463-630c9f557 → 1.43.4.10903-e5521bd8c	the update reached done, but this app has no non-browser data route, so nothing proves the household's data survived it
rallly	4.11.1 → 4.15.2	the update reached done, but this app has no non-browser data route, so nothing proves the household's data survived it

No upstream edge tonight, so nothing to propose (16): calcom, calibre-web, claper, gokapi, gramps-web, homebox, homepage, jellyfin, kimai, onlyoffice, paperless-ngx, plant-it, recipe-importer, seerr, sparkyfitness, wanderer.

---

Teardown — three layers plus Gitea, every claim READ BACK

The machine (9202). controller.yaml restored from controller.yaml.pre-28; git.repo_url reads back as the live catalog with an empty token; and — the one that actually decides which remote is followed (R-615) — the cache reads origin https://gitea.dooplex.hu/admin/app-catalog-felhom.eu.git at 1ad1f34. No drill images. Disk 1.9 GB used of 32 GB, unchanged from the start.

Three things the product could NOT clear, and a shell had to. This is not tidy-up, it is the finding: the termix and gokapi containers left by R-633, and sparkyfitness's app.yaml left by R-634. gokapi was still Restarting two hours later. They were removed by name (docker rm -f termix gokapi, rm .../sparkyfitness/app.yaml) — never a prune. Afterwards 9202 runs exactly felhom-controller, filebrowser, traefik, and no app.yaml exists anywhere. A household has no shell. Evidence: teardown/manual-cleanup.txt.

The host (demo-hp). pct list before and after: 9201 demo-hp and 9202 demo-hp-scratch, both running, unchanged. pvesm status unchanged but for expected scratch growth (nvme-scratch 5.19% → 6.95%). Guest 9201 was never touched.

The hub. Nothing provisioned, nothing changed. 9202 runs hub.enabled: false (R-620).

Gitea. The drill repo is reset to the live main (1ad1f34b6e51). git diff of the live catalog's templates/ against the night's baseline: 0 lines, and image: lines changed: NONE.

The fences, each read back rather than asserted:

fence	at the start	at the end
live catalog origin/main	1ad1f34b6e51	1ad1f34b6e51
demo-hp guest 9201 catalog cache	1ad1f34, live remote	1ad1f34, live remote
demo-felhom guest 9201 catalog cache	1ad1f34, live remote	1ad1f34, live remote
drill repo CI jobs	47	47 — no run, no mail, all night (R-629 holds)

Peti's box is parked and received nothing. Nothing ran on DooPlex beyond ordinary pushes, and nothing on ep0. felhom-controller, felhom-agent and the hub were read only — no product code was written. No golden, no bake, no vouch, no --no-verify, no branch. local-lvm untouched, no prune, tester-1 never reset, drill-r50 untouched.