felhom.eu/documentation/audits/UPDATE-ARC-STATE-2026-09-21.md

UPDATE ARC — the state, measured (2026-09-21)

Phase 0 of the resumed update arc. Nothing here is an estimate. Every number was read from a live box, a live registry or live source on 2026-09-21. Raw captures: audits/update-arc-2026-09-21/.

Baselines at the start: controller 19ef0329ab66 v0.259.0 · agent d9864a94bf62 v0.132.0 · felhom.eu bcdd5b205875 hub v0.119.0 · catalog 18a6d2d8243e. All four trees clean and level with origin/main.

---

1. Three claims in the brief that turned out wrong — named first

1.1 R-589 is NOT open. It shipped in v0.258.0. The brief said, reviewer-verified, that updatebadge.go L82–L99 builds the badge from four raw Hungarian literals and that R-589 is therefore open. The literals are real and the conclusion does not follow. Those literals are the Hungarian form, and they are deliberately frozen — that IS the localisation parity guarantee. The ENGLISH form has been rebuilt from the bundle in web.localeFuncs (internal/web/i18n_web.go, the "updateBadge" entry) since v0.258.0, with badge.update.* in both hu.json and en.json, pinned by TestUpdateBadgeFollowsTheLanguage, and proven live on a fresh box the same morning — DRILL-first-hour-en-0258-2026-09-20.md item 9 reads "PASS — 'Up to date' in English (R-589, fixed this morning, proven on a fresh box)", and R-561's own row records R-589 among the three defects Part 0 of v0.258.0 fixed.

The register row is stale, not the code. It is closed in this session with that citation. The lesson is the general one: a reviewer who reads one producer cannot see a second producer that overrides it. Reading updatebadge.go alone gives exactly the wrong answer, and the file now says so in its own comment.

1.2 The chaos-night canary is NOT a defect to fix. The brief asked why check-image-resolvable and check-volume-persistence returned INCONCLUSIVE on 2026-09-17 and whether the fix is bounded. They behaved exactly as designed. Both scripts' own headers state the rule: a detector that cannot prove itself must refuse to report rather than guess — check-image-resolvable.py cites the 2026-07-21 incident where treating a Docker Hub throttle as failure swept 24 of 65 pins as falsely dead, and the phrase in the drill doc, "its own canary failed", is check-volume-persistence.py's own literal error text (its self_test, which needs to docker build a canary image and therefore needs registry access). The most-supported reading is that registry access was constrained at that moment; that is inferred from the code plus the documented throttle precedent, not observed — no raw stdout of the gate run survives in either evidence directory. Round 7's response — substitute use for update, log the deviation, leave the tree clean — is what the scripts' own documentation asks for.

No register row named it, and there is one real gap, which is diagnostic rather than behavioural: catalog_gates.py collapses "the harness refused to run at all" and "a per-app result is genuinely undetermined" into one INCONCLUSIVE label, so a reader cannot tell which happened without re-running with output captured. Filed as R-605.

1.3 "The hub report carries no image field" — CONFIRMED, on both sides, with one useful nuance. The brief confirmed the controller half (internal/report/types.go L98–103: name, state, CPU, memory). The hub half was this session's to check. Store.SaveReport (hub/internal/store/store.go:965) stores the raw report JSON whole and denormalises only a fixed list out of it — controller version, URL, language, CPU/memory percent, container total and running counts, last snapshot, health status. Containers is two integers; there is no per-app structure anywhere, and no hub code reads an image tag.

The nuance matters for costing Slice 7: because the raw payload is kept whole, a new controller-side field would already be stored the day the controller sends it — what is missing is the denormalisation and the page, not the transport. That makes Slice 7 additive on both sides, as 09 §8.9 says, and slightly cheaper than "a hub-side change" suggests.

---

2. How far behind are the two demo boxes? Not at all.

Read on-disk on both boxes (app.yaml installed_images against the syncer's own catalog clone at <data>/catalog-cache/templates/<app>/docker-compose.yml — the exact two inputs stacks.CatalogOrder reads), controller v0.259.0 on both, catalog cache at 18a6d2d8 on both.

box	deployed apps	behind	unknown (no record)	oldest catalog_since
N100 demo-felhom-8363b5	1 (opengist)	0	0	2026-07-18
HP t740 demo-hp-bb76ea	9	0	0	2026-07-12

Every deployed app on the fleet reads „Naprakész" today, and every one of them has a record. The v0.234.0 backfill has done its job: the "unknown on a quiet box" residue of §8.3 is zero here.

This is the honest caveat, and it is the whole of §8.1: „Naprakész" on these boxes means the reference matches, not the image has not moved. Section 3 measures exactly how false that can be.

---

3. Upstream drift — the number that decides how urgent Slice 6 is

53 apps · 79 image: lines · 66 unique pins. Measured from DooPlex against the public registries, never from a box (§8.1's rule applies to the product, not to an audit). The dead-image sweep used the catalog's own check-image-resolvable.py --all, unmodified.

3.1 The exact pins

	count
exact pins evaluated	58
behind upstream today	46
…of those, a MAJOR step	7
…of those, a minor/patch step	39
already newest	10
Felhom's own internal image (not upstream)	1
inconclusive	1

39 of 46 are within a major — that is the population Slice 6's automatic update would serve. Seven cross a major and are human work by the 2026-09-02 ruling: nextcloud 34→35, paperless-ngx 2.20→3.2, claper 2.5→3.0, gokapi 1.9→2.2, homepage 1.13→2.4, sparkyfitness 0.17→1.7 (two images). All seven were pinned 2026-07-18 — 65 days ago.

3.2 By class

class	apps	pins	floating	behind (of exact)	major	minor/patch
database	14	23	7	14 of 16	5	9
file-leg	9	9	0	7 of 9	0	7
other	30	34	1	25 of 33	2	23

Classification caveat, stated rather than folded in: adventurelog runs a postgis/postgis sidecar, which is PostgreSQL, but the literal substring rule puts it in other. It is the one app that moves if postgis counts as a database (other 30→29, database 14→15). v0.240.0's R-484 already treats PostGIS/pgvector/TimescaleDB as Postgres for the logical dump, so the product is right and only this table's rule is literal.

3.3 The floating pins — §8.1 is now MEASURED, and its COUNT was stale

First, the count itself. 09 §8.1, REUSE.md, the badge's own comment and R-446 all said "23 of the catalog's 66 distinct pins float". That number matches no definition the catalog supports today, and it has been corrected in all four places. Recounted at catalog 18a6d2d8, with the definition written down so it can be rechecked: a pin FLOATS when its tag names a version LINE rather than an exact release.

shape	count	examples
full X.Y.Z	48	privatebin/pdo:2.0.5
two-part line	6	mariadb:11.4/11.6/12.3, claper:2.5, opengist:1.13, wger/server:2.6
major line	4	postgres:15-alpine, postgres:16-alpine, redis:7-alpine, postgis:16-3.5-alpine
exact version + variant suffix	8	ghost:6.53.0-alpine, nextcloud:34.0.1-apache, kimai/kimai2:apache-2.57.0

10 of 66 float. The last row does not, by this definition — those name an exact release and only wear a flavour.

Then, the movement. The sweep measured the 8 database and cache engine pins, of which 7 were measurable:

For each floating tag, the registry's own last-push date against the date the catalog set the pin:

pin	apps	pinned	last repushed upstream	moved?
postgres:16-alpine	8 apps	2026-07-18	2026-09-21	yes
postgres:15-alpine	sparkyfitness	2026-07-18	2026-09-21	yes
redis:7-alpine	6 apps	2026-07-18	2026-09-18	yes
mariadb:11.4	romm	2026-07-18	2026-09-18	yes
mariadb:12.3	bookstack	2026-07-18	2026-09-18	yes
postgis/postgis:16-3.5-alpine	adventurelog	2026-07-18	2026-08-31	yes
mariadb:11.6	kimai, nextcloud	2026-07-18	2025-02-04	no
ghcr.io/immich-app/postgres:16-vectorchord…	immich	2026-07-18	—	UNMEASURED

Six of the seven measurable floating tags have been repushed since the catalog pinned them. So on demo-hp today, adventurelog, bookstack, docmost and romm read „Naprakész" over a database engine image that has demonstrably moved. The badge is not lying — it answers the question it was built to answer — but the question a household hears is the other one. R-446 is no longer a theoretical blind spot: it is six pins out of seven. The eighth (immich's own ghcr build) could not be measured: ghcr's anonymous API exposes no last-modified timestamp and no authenticated token was used. Stated unmeasured rather than guessed.

3.4 One image is gone upstream

msdeluise/plant-it:0.10.0 — the gate says INCONCLUSIVE (a denied/unauthorized stderr, which it refuses to read as "dead"), and two independent signals say it really is gone: Docker Hub's catalog API answers object not found for the whole repository, and the upstream GitHub repo 404s. The app is already lifecycle: abandoned, so this is the expected end state, not an incident. A deployed plant-it would survive — nothing re-pulls a running image — but it can never be redeployed.

---

4. What this session then did

• controller v0.260.0 — R-524 closed: a box AHEAD of the catalog reads „Naprakész", and the guarded Update refuses to move a pin backwards (downgrade, 409). The comparison moved to stacks.CatalogOrder so the badge and the refusal cannot drift. Three red-proofs, each seen to fail. See the repo's CHANGELOG.
• catalog — R-469: the MariaDB half of the engine-major rule lifted, with R-450's own-edge clause enforced in its place; PostgreSQL and MySQL stay refused, now citing R-463 rather than the shipped R-448. Two new decoys, two red-proofs.
• R-589 closed as already-shipped (§1.1). R-605 filed (§1.2).

Live proof of v0.260.0 and the R-520 power cut: audits/update-arc-2026-09-21/.

---

4a. R-524 — PROVEN LIVE, both languages, both surfaces

The state was built the honest way, not by editing a file on the box: a throwaway uptime-kuma on the scratch guest, a real catalog move 2.4.0 → 2.5.0, the guarded Update pressed and run to completion, then the catalog reverted to 2.4.0 — exactly the BIGNIGHT Phase 6 shape that R-524 recorded. The box then runs 2.5.0 against a 2.4.0 catalog.

The badge — app page AND apps list, both languages:

[hu] <span class="tag tag-ok" title="Ez az alkalmazás a katalógusnál újabb változatot futtat,
                                     ezért nincs teendőd.">Naprakész</span>
[en] <span class="tag tag-ok" title="This app is running a version newer than the catalog offers,
                                     so there is nothing for you to do.">Up to date</span>

tag-warn is absent on all four captures (negative control), and the AHEAD title is a different sentence from the ordinary up-to-date title captured at baseline — which is the v0.260.0 behaviour, not a coincidence of wording.

The refusal — the endpoint the button invokes:

request	code	body
POST /api/stacks/uptime-kuma/update	409	„Ez a változat újabb a katalógusban lévőnél — visszalépés csak az üzemeltető kérésére."
the same with ?lang=en	409	"This version is newer than the one in the catalog — moving back needs the operator."
the same with ?lang=hu (control)	409	the Hungarian sentence

The English one is the half that was nearly missed. The new refusal was born as a bundle key, but api.Router rendered update refusals from ref.Message — the Hungarian fallback — so without the one-line change to errText in the same release the key would have been a seam built and never wired. The ?lang=en row above is the proof that it is wired.

The log names both maps, so the verdict can be audited without guessing:

[ERROR] [stacks] update uptime-kuma REFUSED (downgrade): installed is provably NEWER than the
catalog on every differing service (installed=map[uptime-kuma:{louislam/uptime-kuma:2.5.0 …}]
catalog=map[uptime-kuma:louislam/uptime-kuma:2.4.0])

And NOTHING MOVED — which is the property that makes the refusal safe to add. After three refused POSTs the app is still up and healthy on 2.5.0, the live compose line still reads 2.5.0, and no update journal exists (with a positive control proving the directory searched was the real one). That is §6.1's phase-0 contract intact: a refusal taken before the intent is recorded leaves no trace, so a household that presses a refused button has changed nothing.

---

4b. R-520 — a power cut during a REAL version change. CLOSED, and it behaved.

What R-520 asked for, and why the 2026-09-14 attempt could not answer it: that night the only Update the catalog allowed was a same-version one, so nothing could have gone wrong and nothing did. The row asked for the same cut on a real bump.

The measurement, on the scratch guest 9202 (demo-hp), controller v0.260.0, with a throwaway uptime-kuma and a real one-step catalog move 2.4.0 → 2.5.0 (reverted in the same session):

11:04:54.436Z	poller sees updating: true, update_phase: pulling
11:04:54.439Z	pct stop 9202 — the cut
11:04:58.207Z	stopped

Honest limit on the instrument, recorded rather than smoothed: pct stop took 3.8 s to return, so the phase at the moment of the decision is observed and the phase at the moment the kernel froze is inferred.

After pct start, the box said so itself — the POSITIVE observable, not an absence:

update recovery: uptime-kuma was interrupted in pulling (started 2026-09-21T11:04:52Z)
                 — nothing had run; putting the pin back
pin uptime-kuma: uptime-kuma=louislam/uptime-kuma:2.4.0
update uptime-kuma: pin and definition PUT BACK to the pre-update version

• pinned_images and installed_images both 2.4.0; the live compose line back to 2.4.0.
• No hold (correctly — nothing ran), update_phase: failed.
• The app is running and healthy on 2.4.0, 31 s after boot.
• The household is told, and the sentence is true: „A frissítés megszakadt, mert a vezérlő újraindult, mielőtt az új verzió elindult volna. Az alkalmazás a korábbi verzióval fut tovább."

R-520 closes. A power cut in pulling during a real version change is recovered honestly: the pin goes back, the old version runs, and the page says what happened.

The instrument trap this run found, and it is worth more than the result

The first post-crash read, taken from the host against the stopped guest, reported the update journal ABSENT — which would have made this a defect finding. It was a false negative.

pct mount maps the guest's ROOTFS ONLY and does not apply the guest's own internal mounts. Guest 9202 keeps /var/lib/docker on its own ext4 mount under the mp0 volume; from the host that path exists and is an empty stub. The journal was at /var/lib/lxc/9202/rootfs/var/lib/felhom/docker/volumes/felhom-controller-data/_data/data/, and the box's own recovery read it at the next boot — which is the proof that it was there all along.

It was caught by running find over the whole rootfs instead of trusting one constructed path, and by demanding a positive control for the directory being searched. Read from the right place, the journal held exactly what §6.1 promises: phase: "pulling", prev_pin: …2.4.0. An empty directory is not evidence of an absent file — it is equally consistent with "you are looking at an unmounted stub". That is R-96 rule 3 in a new place.

A second instrument trap in the same run — and it nearly bought a false proof

POST /api/sync answered „nincs változás" while the box's catalog cache file HAD moved to the new tag, and the API's catalog_images stayed at the old value until a separate POST /api/stacks/rescan. Since CatalogImages is the one input stacks.CatalogOrder compares against, the badge answers from a stale catalog for that window — and the session nearly recorded a tag-ok badge as proof of the R-524 ahead arm when the badge was merely stale. The honest reading came only after the rescan. An instrument that can report an old value as a current one is not a measurement. Filed as R-607; neither half of it was isolated, so the row records the observation, not a diagnosis.

---

4c. A defect the live run found — filed, not fixed today

On the English app page, the badge is English and the sentence under it is Hungarian. Captured verbatim at step 5, directly above one another:

[en] <span class="tag tag-warn" title="A newer version of this app is available…">Update available — today</span>
     A frissítés megszakadt, mert a vezérlő újraindult, mielőtt az új verzió elindult volna.
     Az alkalmazás a korábbi verzióval fut tovább.

Stack.UpdateError is a finished Hungarian STRING, not a key. Manager.finishUpdate stores it from eight raw literals in update.go, and both app_info.html and stacks.html render it verbatim. The same path carries UpdatePhaseLabel and the HOLD sentence — including backup.UpdateCopyHolds's „csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem", which is a promise about whether the customer's files come back.

v0.260.0 did not close this. That release routed the 409 REFUSAL through errText, and a refusal is the path where nothing happened; these are the sentences for when something did. Filed as R-606 (P2), not fixed — one release per repo per session, and today's had already shipped.

---

4d. Teardown — three layers, stated

layer	state
machine (guest 9202)	throwaway uptime-kuma removed through the product — the API refused the remove while it ran (409, "stop it first"), so it was stopped through the product and then removed, taking its volume and its 24K backup. Only the three protected infra containers remain, as at baseline. No update journal.
host (demo-hp)	guest 9202 left running, as it was found. Guest 9201 untouched — 24 containers before and after. felhom-pve not involved.
hub	nothing provisioned, nothing enrolled, nothing discarded. Guest 9202 reports to no hub by design.

The catalog is back where it started: louislam/uptime-kuma:2.4.0, tree clean and level with origin/main. catalog_since reads 2026-09-21 rather than 2026-07-18 — the gate requires an image move to carry the day's date, and the bump and its revert are two moves that net to zero. No box is behind on that app, so no age count is affected.

---

5. What the numbers say about the two open slices

Slice 6 (automatic within a major) is worth more than its P2 suggests, and the reason is §3.1: 39 of 58 exact pins are one minor step behind, every one of them a change the 2026-09-02 ruling already says may happen without a human. Nobody presses 39 buttons. The support window (§3 decision 2) runs on how far behind the catalog a box is, so today every box is inside it only because the catalog has not moved since 2026-09-15 — the moment the catalog catches up with upstream, every box on the fleet is 46 pins behind at once.

Slice 7 (the fleet view) is still correctly P3-LOW at two enrolled boxes — but §2 shows why it will not stay cheap: the only reason a person could answer "is the fleet current?" today is that a session read both boxes' files by hand.

R-446 (digests) is the cheapest real improvement on this list. §3.3 measured the blind spot at six pins of seven, and the catalog's check-image-resolvable.py already resolves a digest for every pin at push time. Recording it costs one field and closes the gap without the box ever reaching a registry.