admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update arc resumed: the state measured, R-524/R-520/R-589/R-469 closed, seven questions put to the operator
gates / gates (push) Successful in 24s
Details

Browse Source 
Phase 0 — measured, never estimated:
- both demo boxes: 10 apps, 0 behind, 0 unknown
- 46 of 58 exact catalog pins are behind upstream; 39 within a major, 7 across
- 6 of 7 measurable floating pins have been repushed since the catalog set them
  (R-446 is no longer theoretical)
- the "23 of 66 floating pins" figure repeated in four places was STALE; recounted
  to 10, with the definition written down beside it

Three claims in the brief corrected, named first:
- R-589 was NOT open — it shipped in v0.258.0; only the row was stale
- the chaos-night canary is NOT a defect — both gates refused to certify by design
- the hub half of the report confirmed, with the nuance that the raw payload is
  stored whole, so Slice 7 is cheaper than the row implies

Closed: R-524 (controller v0.260.0, proven live in both languages), R-520 (power cut
during a REAL version change — the pin goes back, the app runs, the page says so),
R-589, R-469 (MariaDB half). Filed: R-605, R-606. R-462's stale scope corrected.

09 gains §3 decision 10 (decided by CC unattended — operator may reverse), §3b with
the seven questions in the decision shape, §6.2/6.3 the two open slices, and §6.4 an
update night costed from R-462's real numbers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
authentik Default Admin
2026-09-21 13:13:23 +02:00
parent bcdd5b2058
commit 0c263c77f2
26 changed files with 3125 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/architecture/09-update-architecture.md
+264 -7
View File
@@ -186,6 +186,174 @@ These are rulings, not proposals. Anything specced against a different assumptio
own unit → off-site) stands. Either way the hold sentence ends with what the named copy holds, so a
customer is never sent to a copy that cannot bring the data back without being told so.
### 2026-09-21 — decided by CC unattended, operator may reverse
10. **A box AHEAD of the catalog reads „Naprakész", and the guarded Update refuses to move a pin
backwards** (R-524, controller v0.260.0). *One sentence:* when the catalog is reverted under a box
that already updated, is that a "Frissítés elérhető"? **Options:** (a) leave it — the label
compares for difference, as §5.4 says; (b) show „Naprakész" and let the button still run;
(c) show „Naprakész" and refuse the button. **Costs:** (a) is free and offers a household a
downgrade onto a datadir the newer version may have migrated, which §4 says cannot be undone;
(b) removes the invitation but leaves the loaded gun; (c) costs one comparison and can, wrongly
applied, block a legitimate update. **Why (c):** the direction was already settled — §3 decision 3
says a version change that cannot be undone needs a human, and this is one. The risk in (c) is
bounded by making the Ahead verdict NARROW: every differing service must be orderable AND newer,
or the answer falls back to today's behaviour. Reversible, no customer-data risk, and it only ever
withholds an act. **Implementation:** `stacks.CatalogOrder`, one verdict read by both the badge
and `UpdatePreflight`.
---
## 3b. OPEN — the seven questions Slices 6 and 7 need answered
**These are questions, not rulings. CC does not decide them.** Each is one answerable sentence, the
options, what each costs, the recommendation, and what happens if nothing is decided. The measurement
behind them is `audits/UPDATE-ARC-STATE-2026-09-21.md`; the short version is that **46 of the
catalog's 58 exact pins are behind upstream today and 39 of those are within a major** — the
population §3 decision 3 already says may move without a human, and nobody presses 39 buttons.
### Q1 — When may a box update itself?
*May the box run the guarded Update by itself between 02:30 and 05:00, nightly?*
| option | cost |
|---|---|
| **02:30–05:00 nightly, after the backup legs** | the update leans on a copy made hours earlier the same night, which is the freshest the box ever has. The app is down for the health wait in the middle of the night. |
| a weekly window | fewer interruptions; a box sits up to 7 days on a version the catalog already moved past, which widens the support window §3 decision 2 runs on |
| the household picks the window | one more setting on a page that already has several, for a choice almost nobody will change |
**Recommendation: 02:30–05:00 nightly.** The DB dump runs 02:30 and restic 03:00 on a demo box, so a
window that starts at 02:30 and ends at 05:00 sits on top of the freshest copy of the night without a
new mechanism. **If nothing is decided:** Slice 6 cannot be built at all — every other question below
is downstream of this one.
### Q2 — May an automatic update run on a bind-data app when no copy holds its FILES?
*The button's rule and the automatic rule can differ. Should they?*
**The mechanism, verified at source this session, because an earlier draft had it backwards:** the
guard does **not** refuse these apps. Since v0.239.0/v0.241.0 (§3 decisions 8–9) the Update is refused
only when no copy exists on ANY tier and none can be taken. For an app whose data is bind-mounted
files, `Manager.UpdateTierOrderFor` (`controller/internal/backup/update_guard.go:136-141`) walks
second drive → off-site → **own unit last**, and when the own unit is the copy chosen,
`UpdateCopyHolds` (`:145-165`) ends the hold sentence with *„csak a beállításokat és az adatbázist
tartalmazza, a fájlokat nem"* — it holds the settings and the database and **not the files**. So the
update **proceeds**, and the household is told what the copy holds.
**With a human pressing, that is an informed choice. With nobody pressing, nobody was informed.**
| option | cost |
|---|---|
| **automatic requires a fresh copy that HOLDS THE FILES; the button keeps today's rule** | the nine file-leg apps (and any other bind-data app) update automatically only on a box with a second drive or off-site; on a one-drive box they wait for a person. Two rules to hold in one's head. |
| one rule for both — automatic follows the button | simpler; a file-leg app can be updated unattended against a copy that cannot bring its files back, and the sentence saying so is read by nobody |
| automatic skips bind-data apps entirely | simplest; the seven file-leg apps that are behind never move by themselves even when a good copy exists |
**Recommendation: the first.** It is the smallest rule that keeps the promise the hold sentence makes.
**If nothing is decided:** Slice 6 must be built for the safe subset only, and the file-leg apps stay
manual — which is the third option by default, without anyone choosing it.
### Q3 — What counts as "within a major" when the tag is not a version number?
*§3 decision 3 says automatic within a major, never across. What about `postgres:16-alpine`,
`kimai/kimai2:apache-2.57.0`, a date stamp, a digest?*
**And the test is per compose SERVICE, with ALL of them having to pass.** An app bump that is minor
while its `mariadb:` sidecar moves a major is **ACROSS** — that sidecar now converts the customer's
datadir by itself (R-459), so the edge carries a migration whatever the app's own number says.
| option | cost |
|---|---|
| **an unorderable tag on ANY service makes the whole edge ACROSS → human** | the 8 floating pins and every suffix-versioned image stay manual. Conservative, and it is the same rule v0.260.0's `CompareImageRefs` already implements and tests. |
| teach the comparator each shape | every new shape is a new rule, and a wrong rule silently automates a major |
| compare digests instead | needs Q6 first, and a digest carries no order at all — it can say "different", never "newer" |
**Recommendation: the first**, reusing `stacks.CompareImageRefs` rather than writing a second rule.
*One small extension is needed and is named here so it is not discovered late:* v0.260.0's
`CompareImageRefs` answers *orderable?* and *newer?*, which is all R-524 needed. Slice 6 also needs
*same major?*, so the parsed major has to be exposed from the same normaliser — **an addition to the
one comparator, never a second one.**
**If nothing is decided:** Slice 6 would have to invent a rule under time pressure, which is how a
major gets automated by accident.
### Q4 — A held app: who is told, when, and does the box try again?
*An automatic update that ends HELD happened while everyone was asleep.*
| option | cost |
|---|---|
| **the household on the app page and by mail ONCE; the operator by event; NO retry until the catalog moves again or a person presses** | one mail per held app. The app stays down until someone acts — which is already true of a held update today. |
| retry the next night | a broken edge takes the app down every night and mails every morning; the hold exists precisely because the box cannot fix it |
| tell only the operator | the household finds their app down and has no sentence explaining it |
**Recommendation: the first.** It is what the manual hold already does (`settings.RestoreHold` with
`reason: update_failed`), plus one mail. **If nothing is decided:** the safe default is no automatic
update at all, because a hold nobody is told about is worse than a version nobody moved.
### Q5 — PostgreSQL: what has to exist before the catalog may move `postgres:16` to `17`?
*Eleven templates, and the image performs no conversion — it refuses to start on an older major's
datadir (R-463).*
| option | cost |
|---|---|
| **a scripted `pg_upgrade` edge in the harness, proven on all eleven, before the catalog may move** | real work: eleven fixtures, and `pg_upgrade` needs both major's binaries present. The engine-major gate keeps the rule until it exists. |
| move the pin and let the update HOLD honestly | every one of the eleven apps goes down on the same night and comes back only by a restore |
| never move PostgreSQL majors | the fleet sits on an engine that eventually loses upstream support |
**Recommendation: the first, and the gate stays until it lands.** As of 2026-09-21 the engine-major
rule's MariaDB half is LIFTED (R-469 — MariaDB has both a backup in front of it and
`MARIADB_AUTO_UPGRADE=1`); this half is exactly what stays. **If nothing is decided:** nothing breaks
— the gate refuses the move — but the eleven apps drift further from upstream every month.
### Q6 — Should the catalog record each pin's DIGEST at push time?
*So the box can tell a moved floating tag from an unmoved one without ever reaching a registry.*
**This is no longer theoretical. Measured 2026-09-21: six of the seven measurable floating pins have
been repushed upstream since the catalog set them** — `postgres:16-alpine` (8 apps),
`postgres:15-alpine`, `redis:7-alpine` (6 apps), `mariadb:11.4`, `mariadb:12.3`,
`postgis:16-3.5-alpine`. On demo-hp today, four apps read „Naprakész" over a database engine image
that has demonstrably moved.
| option | cost |
|---|---|
| **the catalog records the digest at push time; the box compares digests** | one field per pin. `check-image-resolvable.py` already resolves the digest, so the producer exists. §8.1's rule — the box never queries a registry — is untouched. |
| the box queries registries | breaks §8.1 outright: a page that cannot render without eight upstream registries |
| leave it | the badge stays right about the question it asks and wrong about the one a household hears |
**Recommendation: yes.** It is the cheapest real improvement on this list and it closes R-446.
**If nothing is decided:** „Naprakész" keeps meaning "the reference matches", which is measurably not
what it sounds like.
### Q7 — What does the hub's report need to carry for a fleet view?
*Slice 7 lets the operator SEE and MOVE how far behind every box is.*
**Verified both sides this session:** the controller's report payload carries name, state, CPU and
memory and no image (`controller/internal/report/types.go` L98–103), and the hub's
`Store.SaveReport` (`hub/internal/store/store.go:965`) denormalises only container **counts**. **But
the hub stores the raw report JSON whole**, so a new controller field lands there the day it is sent —
what is missing is the denormalisation and the page, not the transport.
| option | cost |
|---|---|
| **per app: installed reference + catalog reference + badge state; the hub lists boxes behind, with a "move" that is the same guarded Update, operator-triggered** | additive on both sides; the report grows by a few fields per app |
| badge state only | smaller payload; the operator cannot see WHAT is behind, only that something is |
| leave it to per-box pages | free today at two boxes; unusable at twenty |
**Recommendation: the first, and it stays P3-LOW until the fleet grows.** **If nothing is decided:**
the only way to answer "is the fleet current?" is what this session did — read both boxes' files by
hand.
---
### Not a question — already ruled
**R-462's scope was decided on 2026-09-13.** §3 decision 6: the upgrade test goes to **all** apps
through the nightly rotation, explicitly *not* "database apps first". The register row R-462 still
says *"VIKTOR rules on scope"* — **that row is stale and is corrected to cite decision 6.** The update
night below proposes an ORDER *inside* that ruling; it does not reopen it.
## 4. The vocabulary ruling — "rollback" is struck
**App data CANNOT be rolled back.** Measured on Nextcloud (spike §7): once a migration has actually
@@ -329,7 +497,7 @@ earlier feature is the failure mode to look for whenever a file changes meaning.
|---|---|---|
| **1** | **The box records what it actually installed** — `app.yaml.installed_images`, per compose service, ref + digest + first-seen. | **SHIPPED, controller v0.233.0 (2026-09-02)** |
| **1b** | **Seed the record for apps nobody touches** — a startup backfill, so the label is not restricted to apps that happen to get restarted. | **SHIPPED, controller v0.234.0 (2026-09-03)** |
| **2** | **One badge says whether the app is current** — „Naprakész" / „Frissítés elérhető — N napja", from `catalog_since`. No version number. | **SHIPPED, controller v0.233.0 + catalog `69761cf` (2026-09-02)** |
| **2** | **One badge says whether the app is current** — „Naprakész" / „Frissítés elérhető — N napja", from `catalog_since`. No version number. | **SHIPPED, controller v0.233.0 + catalog `69761cf` (2026-09-02); English since v0.258.0 (R-589); a FOURTH verdict — AHEAD — and the downgrade refusal in v0.260.0 (R-524, §3 decision 10)** |
| **3** | **The compose file becomes DERIVED** — the pin in `app.yaml` wins; the syncer renders instead of copying. | **SHIPPED, controller v0.235.0 (2026-09-06)** — operator ruling §3.4 |
| **4** | **A guarded update** — verified-backup precondition, abort-on-failure, and the truth at the moment of action rather than 5m16s later (R-443). | **SHIPPED + PROVEN LIVE, controller v0.237.0 (job) + v0.238.0 (page) + v0.238.1 (2026-09-13); any backup tier since v0.239.0 (§3 decision 8)** — §6.1 |
| **5** | **An upgrade test that runs again** — a harness that upgrades a real app with real data in it and asks the app for the data back. | **SHIPPED, `app-catalog/scripts/upgrade-test.py` (2026-09-06)** — 7 edges, 3 apps; see §4.1 and §10 |
@@ -484,6 +652,82 @@ breaks.
---
### 6.2 Slice 6, as it would be built (OPEN — R-450; needs Q1–Q4)
**Not a design yet; the shape the seven questions bound.** Written down so the answers have somewhere
to land.
**Nothing new happens to the app.** Inside the window, for an app that qualifies, the box runs
**exactly the guarded Update of §6.1** — same precondition, same safety dump, same pin journal, same
health wait, same hold. Slice 6 adds a *caller*, not a *path*. That is the whole reason it is
affordable: every failure mode was measured in slice 4 and every one of them already ends in a hold
the household can read.
**An app qualifies when ALL of these hold** (each clause is a question above, not a decision taken):
1. the window is open (Q1);
2. `stacks.CatalogOrder` says **Behind** — never Unknown, never Ahead (v0.260.0 gives all four);
3. the edge is **within a major for EVERY compose service**, the engine sidecar included (Q3),
judged by `stacks.CompareImageRefs` — one unorderable service makes the whole edge *across*;
4. a fresh copy exists on a tier that holds what this app's data actually is (Q2);
5. the app's own switch is on (Q2's default: on).
**What the household sees.** An event and a line on the app page's timeline, before and after, in both
languages: *„Automatikus frissítés 03:12-kor — sikeres"* / *„— megállítva, a másolat 2026-09-20-i"*.
A held app is not retried until the catalog moves again or a person presses (Q4).
**Where it would live.** A scheduler beside the existing nightly legs, reading `settings` for the
window and the per-app switch, and calling `Manager.StartGuardedUpdate`. **It must respect the same
`isHeld`/`SetUpdatingCheck` interlocks v0.238.1 added** — the nightly capture running *inside* an
update's health wait is the defect that release fixed, and a second unattended caller is exactly the
shape that finds it again.
**Ships behind `auto_update: off` with no UI until the operator answers Q1.**
### 6.3 Slice 7, as it would be built (OPEN — R-451; needs Q7)
Three additive pieces, and the transport already exists (§3b Q7):
1. **controller** — the report's per-app object gains installed reference, catalog reference and badge
state. Additive; an older hub ignores it.
2. **hub** — denormalise those out of the raw report it already stores whole, and list boxes by how
far behind they are.
3. **hub → box** — a "move" button that is the same guarded Update, operator-triggered, through the
existing command path. **Not a second update mechanism**, and not automatic.
Rank stays P3-LOW at two enrolled boxes. It rises with the fleet, and §2 of the state audit is what
that looks like today: the only way to answer *"is the fleet current?"* was to read both boxes' files
by hand.
### 6.4 The update night — a drill brief outline, costed from R-462's real numbers
**The ruling is decision 6: all 53 apps, through the nightly rotation.** This is an ORDER inside that
ruling, not a scope change. The database apps go first because they are the ones where a wrong answer
costs data rather than uptime.
**The real numbers this rests on** (R-462, measured 2026-09-06): a successful edge takes
**6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly 8×, because a negative is
only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**. **Machine
time is not the cost — fixtures are.** Two of the three apps needed a bespoke non-browser seed route,
one needed two attempts and a discarded approach, and one (bookstack) can only ever be half-proven
headlessly (R-460).
| leg | what | cost |
|---|---|---|
| A | the **15 database services** — 4 MariaDB + 11 PostgreSQL, across 14 apps by the substring rule plus `adventurelog`'s postgis — one edge each, fixture per app | **15–25 CC-hours**, dominated by seed routes; ~30 min machine time at the median; ~25 GB |
| B | one **power cut mid-update** on a real version change, in `pulling` and again in `starting` | 1–2 CC-hours (R-520 — the first half is measured in this session) |
| C | one **PostgreSQL `pg_upgrade` rehearsal**, the Q5 edge, on one app before any of the eleven | 3–4 CC-hours |
| D | one **downgrade refusal** | **already done** — v0.260.0, proven live 2026-09-21 |
| E | the **automatic night** on a throwaway: one app, one real catalog step, inside a simulated window, with the guard and the hold; then the same edge made to fail → HOLD, the event, no retry loop | 2–3 CC-hours |
| F | the remaining **38 apps**, through the nightly rotation as decision 6 directs | ~1 app/night; fixtures amortised |
**Total for legs A–E: roughly 21–34 CC-hours**, plus ~25–30 GB of images on a scratch host. Legs C
and E are the ones that unblock a decision; leg A is the one that takes the time.
**Venue:** a scratch host, never a customer box — `demo-hp`'s guest 9202 for the box-side legs, the
harness on DooPlex for the image-side ones.
## 7. What slices 1 and 2 actually built
### 7.1 The record (slice 1)
@@ -547,12 +791,25 @@ Version strings stay in the logs, the API and the hub.
## 8. Known limitations, stated plainly
1. **„Naprakész" can be FALSE for the 23 floating pins.** The comparison is reference-to-reference and
queries no registry — a customer's box must not depend on reaching eight upstream registries to
render a page. For `postgres:16-alpine`, `mariadb:11.6` and 21 others the reference can be
identical while the image behind it has moved. **Measured, not theorised:** spike §5 found
`mariadb:11.4` and `mariadb:12.3` had both already moved upstream, with two fully-pinned controls
holding. Digest-level comparison needs a registry query and is deferred — **R-446**.
1. **„Naprakész" can be FALSE for the floating pins, and 2026-09-21 measured HOW false.** The
comparison is reference-to-reference and queries no registry — a customer's box must not depend on
reaching eight upstream registries to render a page. For `postgres:16-alpine`, `mariadb:11.6` and
the others the reference can be identical while the image behind it has moved.
**NUMBERS, 2026-09-21** (`audits/UPDATE-ARC-STATE-2026-09-21.md` §3.3). **The count this document
carried — "23 of 66" — is STALE and matched no definition the catalog supports today.** Recounted
at catalog `18a6d2d8`, with the definition stated so it can be rechecked: a pin FLOATS when its
tag names a version LINE rather than an exact release. Of 66 unique pins, 48 are full `X.Y.Z`,
**6 are two-part lines** (`mariadb:11.4`/`11.6`/`12.3`, `claper:2.5`, `opengist:1.13`,
`wger/server:2.6`) and **4 are major lines** (`postgres:15-alpine`, `postgres:16-alpine`,
`redis:7-alpine`, `postgis:16-3.5-alpine`) — **10 float**. The remaining 8 are exact versions
wearing a variant suffix (`ghost:6.53.0-alpine`, `nextcloud:34.0.1-apache`, …), which do not float
by this definition. Of the 8 database and cache engine pins the sweep measured, 7 were measurable
and **6 have been repushed upstream since the catalog set them** — `postgres:16-alpine` (8 apps), `postgres:15-alpine`, `redis:7-alpine`
(6 apps), `mariadb:11.4`, `mariadb:12.3`, `postgis:16-3.5-alpine`. Only `mariadb:11.6` has not.
The 8th, immich's own ghcr build, is UNMEASURED — ghcr exposes no anonymous last-modified. So on
demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.
Digest-level comparison needs the catalog to record the digest at push time — **R-446**, put to the
operator as §3b **Q6**, recommended YES.
2. **~~Nothing enforces `catalog_since`.~~ Enforced by the pre-push hook since 2026-09-13 (R-452, `app-catalog-felhom.eu/scripts/check-catalog-since.py`); CI's shallow clone still skips it out loud.** A commit that moves an `image:` line and forgets the date
under-reports how far behind a box is. The gates runner fetches at `--depth 1` and has no parent
commit to diff against, so the gate needs a deeper fetch — **R-452**.