Review round 2 (batch 2 + close-out): 02/07/09/10/11/14/15/16/inventory facts corrected at file:line (EDITS.md), undocumented decisions as notes (R-1265), rulings left for the operator (R-1266); register 190 -> 207 (R-1251..R-1267 opened: R-1251 P1 cut drive-app restore put back over newer data, R-1252 P2 rejected backup block deletes second-drive copies - reproduced on 9202); STATUS D43-D45; CONTEXT; audit README
gates / gates (push) Successful in 13m10s
gates / gates (push) Successful in 13m10s
This commit is contained in:
admin committed
1 parent
77015997f8
commit
876d92367e
23 files changed
+551
-70
No files matched your search
@@ -65,6 +65,13 @@ build is added to both in the same commit.
|
||||
|
||||
### 1.1 The button
|
||||
|
||||
> *Dated note 2026-10-24 (review round 2): §1 is the 2026-09 MEASUREMENT before the arc. `UpdateStack` no longer exists
|
||||
> (deleted in v0.237.0, controller `REUSE.md`); the button calls `StartGuardedUpdate` — backup precondition, safety
|
||||
> dump, pin journal, undo copy, health wait, undo (`internal/api/router.go:794`; `internal/stacks/update.go`). The line
|
||||
> numbers below are the measurement's; e.g. `copyTemplates` is now `sync.go:436`, `copyIfChanged` `:700`, the `15m`
|
||||
> default `config.go:399`, `RestartStack` `manager.go:1454` (it now runs `up -d --force-recreate` and errors when no
|
||||
> container restarted, R-1026).*
|
||||
|
||||
`Manager.UpdateStack` (`felhom-controller/controller/internal/stacks/manager.go:1199`) is two compose
|
||||
commands and nothing else:
|
||||
|
||||
@@ -267,7 +274,8 @@ builds them (`audits/update-rulings-2026-09-23/`).
|
||||
next night. **Why:** the update then leans on the freshest copy the box ever has, and there is no
|
||||
second window for anyone to set or misread. **Replaces:** §3b Q1's recommendation of a fixed
|
||||
02:30–05:00. *For the builder:* `stacks.update_window` (`config.go` L195, default `03:00-05:00`)
|
||||
exists and **no Go code reads it** — it appears only in `config.go` (field + default),
|
||||
exists and **no Go code reads it** *(dated note 2026-10-24: the field is now removed — no controller source names
|
||||
it; `controller/README.md` „`stacks.update_window` is removed")* — it appears only in `config.go` (field + default),
|
||||
`setup/handlers.go` L482 (written into a new box's `controller.yaml`) and
|
||||
`configs/controller.yaml.example`. Remove it or fold it into the chain; do not build a second
|
||||
window.
|
||||
@@ -424,7 +432,8 @@ R-636's louder repeated alarm.
|
||||
(a) broke the second. Shipped as controller **v0.269.1** — a second release in the session, instead of
|
||||
shipping (a) to the fleet with the floor.
|
||||
*Note 2026-09-30 (R-740) — the cost line above is not what the code does.* „(or the automatic leg)": the
|
||||
automatic leg never presses a digest-only change (`unattended.go:433–435`, skip `no_test_record`). And the
|
||||
automatic leg never presses a digest-only change (`unattended.go:433–435` then, `:449-451` now — line corrected
|
||||
2026-10-24 — skip `no_test_record`). And the
|
||||
catalog never writes a re-test of a floating tag at a new digest (the writer refuses a step that moves no
|
||||
image), so a same-name upstream fix reaches no box by either route today. Measured, with the options, in
|
||||
R-740; the decision is the operator's (STATUS 2026-09-30). This decision's ruling — the sync never moves
|
||||
@@ -1981,7 +1990,7 @@ moves a stack dir.
|
||||
| writer | pin source |
|
||||
|---|---|
|
||||
| the deploy path (`runComposeDeploy`) | the template just deployed from |
|
||||
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull |
|
||||
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull — *as built: `UpdateStack` was deleted in v0.237.0; the guarded update writes the pin (`advancePinTo`, the `pinning` phase before `pulling`), and two more paths move a pin BACK — `pinBack` and the undo's `restoreDefinition` (`internal/stacks/update.go`, `undo.go`; noted 2026-10-24)* |
|
||||
| the restore (`stackAdapter.RecreateStackDefinitionFromUnit`) | the recovery unit's captured compose — **this closes R-441** |
|
||||
| `Manager.AdoptPins` | the observation, once, and only when complete AND matching |
|
||||
|
||||
@@ -2047,7 +2056,7 @@ earlier feature is the failure mode to look for whenever a file changes meaning.
|
||||
| **3** | **The compose file becomes DERIVED** — the pin in `app.yaml` wins; the syncer renders instead of copying. | **SHIPPED, controller v0.235.0 (2026-09-06)** — operator ruling §3.4 |
|
||||
| **4** | **A guarded update** — verified-backup precondition, abort-on-failure, and the truth at the moment of action rather than 5m16s later (R-443). | **SHIPPED + PROVEN LIVE, controller v0.237.0 (job) + v0.238.0 (page) + v0.238.1 (2026-09-13); any backup tier since v0.239.0 (§3 decision 8)** — §6.1 |
|
||||
| **5** | **An upgrade test that runs again** — a harness that upgrades a real app with real data in it and asks the app for the data back. | **SHIPPED, `app-catalog/scripts/upgrade-test.py` (2026-09-06)** — 7 edges, 3 apps; see §4.1 and §10 |
|
||||
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | OPEN — R-450; rulings 2026-09-23; build order §6.4 |
|
||||
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | ~~OPEN — R-450~~ **BUILT** — the leg in `internal/stacks/unattended.go`, wired into the off-site job; R-450 CLOSED 2026-10-05 (capability map B11) *(noted 2026-10-24, review round 2)* |
|
||||
| **7** | **A fleet sweep pipeline** — the operator can see, and move, how far behind every box is. | OPEN — R-451; ruled 2026-09-23 (decision 18), built later |
|
||||
|
||||
### 6.1 Slice 4 as SHIPPED (controller v0.237.0 + v0.238.0, 2026-09-13)
|
||||
@@ -2062,12 +2071,13 @@ closed by construction: nothing reports an update complete on the compose exit c
|
||||
| # | phase | what happens | on failure |
|
||||
|---|---|---|---|
|
||||
| 0 | refusals (409, before the intent is recorded) | held (R-439), busy (backup/restore/app-data op/quiesce), migration, already updating, deploying, memory (the deploy's `memoryVerdict`, releasing the app's own request), disk (**fixed 2 GB floor** — image size unknown without a registry), **no copy on ANY tier and no backup can be taken now** (since v0.239.0; before it, no restorable Tier-2 copy) | nothing moves, nothing is recorded |
|
||||
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) | nothing moves |
|
||||
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) — *an app whose data lives in bind-mounted files walks second drive → OFF-SITE → own unit (decision 9, R-479, v0.241.0; `internal/backup/update_guard.go` `updateTierOrderBindData`; noted 2026-10-24)* | nothing moves |
|
||||
| 2 | `backing-up` — only when no tier holds a fresh copy | `RunAppBackupNow`: this app's DB dump → volume dump → unit capture (marked proven current) → Tier-2 copy, whose failure is a WARN since v0.239.0 | refused with the backup's own error; nothing moves |
|
||||
| 3 | `safety-dump` | `WriteUpdateSafetyDump` (R-361's undo copy) — **before the pin moves** | refused; nothing moves |
|
||||
| 4 | `pinning` | the previous definition is copied aside and journaled, then the pin advances | pin put back |
|
||||
| 5 | `pulling` | `compose pull` | **pin and definition PUT BACK** — nothing ran (Scenario E) |
|
||||
| 5a | `copying` **(v0.263.0)** | `compose stop`, then each NAMED volume `cp -a` into `<volume>.pre-update-<stamp>` by a helper that writes a finished-marker last (bind folders never) | copies removed, **pin put back, the old version started again** — nothing new ran |
|
||||
| 5b | `converting` **(v0.273.0, row added 2026-10-24)** | only on a step whose ladder entry carries `engine_conversion`: the box converts the PostgreSQL major (dump from the old engine, load into the new, check) — `internal/stacks/pgconvert.go` | the undo puts the old engine's data back |
|
||||
| 6 | `starting` | `compose up -d --remove-orphans` | **UNDO** (below), HOLD only if the undo fails |
|
||||
| 7 | `verifying` | the `.felhom.yml` health check through the existing probe **when it resolves to a container**, else 60 s of every container running and none restarting; bounded by `update.health_timeout` | **UNDO** (below), HOLD only if the undo fails. Before v0.263.0: stop + HOLD, the pin stays (Scenario F) |
|
||||
| 7a | `undoing` **(v0.263.0)** | every copy validated (marker) BEFORE anything is poured back; volumes emptied and refilled; definition, pin and the pinned version's `.felhom.yml` record put back from the job's own copies; `up`; health with the OLD version's probe | HOLD, the sentence prefixed *„A frissítés nem sikerült, és az automatikus visszaállítás sem."* + the data state (`untouched` / `half` / `not_started`); copies kept |
|
||||
@@ -2116,7 +2126,7 @@ the harness has proven it, is slice 6's.
|
||||
> copy taken before the pin moved**, so the old version meets pre-migration data. Spiked by hand on
|
||||
> 9202 the same day — §6.1a.
|
||||
|
||||
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6.
|
||||
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6. *(Noted 2026-10-24: stepping is built — the ladder, one tested step per press, decision 14.)*
|
||||
|
||||
### 6.1a The undo (decision 15) — SPIKED BY HAND, then BUILT: controller v0.263.2 (2026-09-23)
|
||||
|
||||
@@ -2130,7 +2140,8 @@ only the live box could show, fixed the same day:** the undo's probe was never a
|
||||
current probe held the app `unhealthy` (v0.263.1), and the "old" `.felhom.yml` taken at update time
|
||||
was already the new one, because `.felhom.yml` flows in on every catalog sync (v0.263.2: the pinned
|
||||
version's file is now recorded in `applied-meta/` whenever a version is pinned). **Residual (R-646):**
|
||||
an app pinned before v0.263.2 has no such record until its next pin. **Two more, found by the chaos hour of
|
||||
an app pinned before v0.263.2 has no such record until its next pin. *(Closed since: `BackfillAppliedMeta` records it at
|
||||
start for every app CURRENT with the catalog, and names the rest — `internal/stacks/undo.go:646-652`; noted 2026-10-24.)* **Two more, found by the chaos hour of
|
||||
2026-09-23 night (`audits/DRILL-night-2026-09-23.md` Part D):** the undo finds an app's volumes by their compose
|
||||
label, and a restore recreates them WITHOUT it — so after any restore the undo copies nothing (R-658, P1); and a
|
||||
held file-leg app is pointed at a restore that refuses a database-only copy, with no route left on a box without
|
||||
@@ -2270,7 +2281,7 @@ breaks.
|
||||
|
||||
---
|
||||
|
||||
### 6.2 Slice 6, as it will be built (OPEN — R-450; ruled 2026-09-23, decisions 11–17)
|
||||
### 6.2 Slice 6, as it will be built (~~OPEN — R-450~~ BUILT, R-450 closed 2026-10-05 — noted 2026-10-24; ruled 2026-09-23, decisions 11–17)
|
||||
|
||||
**The shape the rulings fix. The build order and costs are §6.4.** Rewritten 2026-09-23; the
|
||||
earlier "as it would be built" draft keyed on the window of Q1's recommendation and on
|
||||
@@ -2341,7 +2352,7 @@ retried. A working caller in this exact shape exists as evidence, not product:
|
||||
**The per-box switch key is `app_update.unattended`**, default **true** (decision 12).
|
||||
|
||||
⚠ **NOT `auto_update` — that name is TAKEN, and by the very thing this must not collide with.**
|
||||
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281, default
|
||||
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281 then, L278-279 now; default
|
||||
**04:30** at L422) are the CONTROLLER's own update. Two settings with that name, one meaning the
|
||||
controller and one meaning apps, is the kind of collision that is only discovered by an operator who
|
||||
turned off the wrong one. The app-scoped key is `app_update.*`. **And `stacks.update_window` is
|
||||
@@ -2371,12 +2382,12 @@ what the part can do to a household's data if it is wrong, not how likely that i
|
||||
|
||||
| # | part | rulings / rows | cost | depends on | risk to customer data |
|
||||
|---|---|---|---|---|---|
|
||||
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold`: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
|
||||
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold` *(as SHIPPED — noted 2026-10-24: the copy is the FOLDER copy decision 19 chose: every named volume `cp -a` into a sibling volume with a finished-marker, put back on failure; no database load — `internal/stacks/undo.go:26-36`; the text that follows is the design before the bake-off)*: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
|
||||
| **2** | **SHIPPED — controller v0.264.0 + hub v0.120.0, proven live on 9202 and 9201 2026-09-23** (`audits/undo-fleet-2026-09-23/`). **The update sentences in the household's language** (R-606) and a mail when an automatic update is undone or held: events `app_update_undone` (warning) and `app_update_held` (error), one per app per outcome, on by default, mailed in the household's language with the app named in the subject; per-app cooldown on both legs. Leftovers: R-647. | R-606, 15 | **1** | — | none |
|
||||
| **3** | **SHIPPED — controller v0.264.0, proven live on 9202 2026-09-23.** **A disabled notifier says so** (R-620), so the mail of part 2 can be measured on a scratch box at all. | R-620 | **0.5** | — | none |
|
||||
| **4** | **SHIPPED — catalog `6db08a5`, night 2026-09-23** (`audits/DRILL-night-2026-09-23.md`): `update_ladder:` in `.felhom.yml`, one JSON entry per line (spiked on controller v0.266.0 and v0.267.0 first — the controller ignores the key); `check-test-record.py` (static, CI) + `check-test-record-move.py` (history + the registry for moved refs only, decision 23); the only writer `upgrade-test.py --write-ladder`; the 21 moves of 2026-09-22 backfilled from their records (21 proven). Not built: `steps/<to>.yml` (part 5 needs it once an app has two steps); `CompareImageRefs` did NOT move to the gate — the gate asks for a proven test instead, which is decision 13's own test. **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only |
|
||||
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
|
||||
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. **The box half (compare, render `name:tag@sha256`) is not built.** **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
|
||||
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). *(Closed since: a step's own `steps/<key>.felhom.yml` is read when present — `ladder.go:127-132`, v0.269.0; noted 2026-10-24.)* **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
|
||||
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. ~~**The box half (compare, render `name:tag@sha256`) is not built.**~~ *Built — the box renders the tested digest, carries it on sync, and the badge reads the tested digest (`internal/stacks/digest.go:15-24`, `updateorder.go`; noted 2026-10-24).* **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
|
||||
| **7** | **SHIPPED — controller v0.271.0 (`9cf13a3`), proven on 9202 over four simulated nights 2026-09-24/25 and watched through the demo boxes' first real automatic night (`audits/DRILL-night-2026-09-25.md`).** Decisions 31–33 taken unattended. Gaps a scratch box cannot close: R-687; no resume after a restart: R-686. *(Before:)* **SPIKED 2026-09-24 (night, Part C) — the measured build brief is §6.4.2 below.** **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has ENDED (on every exit path, skips included), one app at a time, one step per press, `app_update.unattended` default ON (decision 12), `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it, and the full-system backup's gate waits for it until W+5h (decision 20). | 11, 12, 14, 15, 20 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe |
|
||||
| **8** | **SHIPPED — controller v0.265.0 + hub v0.121.0, proven live on 9202 2026-09-23** (`audits/cleanup-2026-09-23/`; a kernel `oom_kill` counter, not the sticky flag — `08` §6.2). **R-636** — the same OOM key re-firing escalates instead of staying one `warning` for six hours. | R-636 | **1** | — | none |
|
||||
| **9** | **SHIPPED — controller v0.265.0, proven live on 9202 2026-09-23** (badge „Megállítva — visszaállítás szükséges" / "Stopped — restore needed", no Update button, 409 `held` unchanged). **R-625** — a held app stops offering an Update it will refuse. With the undo, holds become rarer; the lie on the page does not go away by itself. | R-625 | **0.5** | 1 | none |
|
||||
@@ -2459,7 +2470,7 @@ During the db-dump leg every press was refused `busy` (transient, retried next p
|
||||
update's own finish should refresh them.
|
||||
- **An Update pressed on an app already at the head runs the whole guarded update** — backup, pull,
|
||||
restart, `done` — with no refusal (R-679: navidrome restarted four times for nothing). The leg never
|
||||
presses an app whose pin equals the head; the preflight should refuse with reason `current`.
|
||||
presses an app whose pin equals the head; the preflight should refuse with reason `current` (as built the reason key is `already_current`, `update.go:432` — noted 2026-10-24).
|
||||
- **The product does not remember a failed step.** After vikunja's undo the badge still offers the
|
||||
same step; only the caller's memory kept it from a re-press. Decision 15 needs it on the box: the
|
||||
failed `to` is recorded per app and the leg skips it until the catalog's ladder changes.
|
||||
@@ -2541,6 +2552,24 @@ and E are the ones that unblock a decision; leg A is the one that takes the time
|
||||
harness on DooPlex for the image-side ones.
|
||||
|
||||
|
||||
### 6.4.x Decisions in code no section records [DESIGN — undocumented, found 2026-10-24, review round 2 — the operator confirms them through one register row]
|
||||
|
||||
Read at `file:line` on controller `main` (`321df1f`) by the main session:
|
||||
|
||||
- **`build.sh --push` also refuses a DIRTY tree, a HEAD that is not `origin/main`, and a directory that is not a git
|
||||
checkout** — `controller/build.sh:144-166`. §0 names only the tag-exists and unreachable-registry refusals.
|
||||
- **The leg retries a passing refusal ONCE in the same night** (`busy`, `updating`, `deploying`, `migrating`,
|
||||
`self_updating`), after `RetryWait` — not „the next pass/night" (`internal/stacks/unattended.go:65-66`, the pass loop).
|
||||
- **The leg skips an app the household stopped** (`stopped_by_household`, R-721): an update would start it
|
||||
(`unattended.go:416-418`).
|
||||
- **A manual run of the chain** (R-705, the debug action) gives the leg a deadline of its start + the leg's normal
|
||||
length (`backupwindow.UpdateLegLengthMin`), not W+5h (`unattended.go:232-238`).
|
||||
- **Two preflight refusals are plain skips to the leg:** `already_current` and `engine_no_test`
|
||||
(`internal/stacks/update.go:432,454`).
|
||||
- **The render table (§5.4) has one more row:** when the catalog template's images cannot be read, the syncer leaves
|
||||
the app's compose file alone for that cycle (`internal/sync/sync.go:622-627`); and the box's R-971 allow-list runs
|
||||
before ANY compose write (`cmd/controller/main.go:421`, `SetAdmitFn`).
|
||||
|
||||
## 6.5 The drill catalog and the image store — the standing method for update drills
|
||||
|
||||
**Why this section exists.** On 2026-09-21 an afternoon session put a deliberately broken image into
|
||||
@@ -2629,7 +2658,9 @@ customer box has credentials for it. The store listens on the guest's loopback o
|
||||
### 7.1 The record (slice 1)
|
||||
|
||||
`Manager.recordInstalledImages` (`felhom-controller/controller/internal/stacks/installed.go`) runs
|
||||
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`,
|
||||
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`
|
||||
*(as built — noted 2026-10-24: `UpdateStack` is gone; the callers are `StartStack`, `RestartStack`, the deploy path, the
|
||||
end of the guarded update and the end of the undo — `manager.go`, `deploy.go`, `update.go`, `undo.go`)*,
|
||||
and writes `app.yaml`:
|
||||
|
||||
```yaml
|
||||
@@ -2705,7 +2736,7 @@ Version strings stay in the logs, the API and the hub.
|
||||
The 8th, immich's own ghcr build, is UNMEASURED — ghcr exposes no anonymous last-modified. So on
|
||||
demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.
|
||||
Digest-level comparison needs the catalog to record the digest at push time — **R-446**, put to the
|
||||
operator as §3b **Q6**, recommended YES.
|
||||
operator as §3b **Q6**, recommended YES. *(Built since for every ref with a ladder entry — `updateorder.go`, `digest.go`; the limitation remains only for a ref no entry records; noted 2026-10-24.)*
|
||||
2. **~~Nothing enforces `catalog_since`.~~ Enforced by the pre-push hook since 2026-09-13 (R-452, `app-catalog-felhom.eu/scripts/check-catalog-since.py`); CI's shallow clone still skips it out loud.** A commit that moves an `image:` line and forgets the date
|
||||
under-reports how far behind a box is. The gates runner fetches at `--depth 1` and has no parent
|
||||
commit to diff against, so the gate needs a deeper fetch — **R-452**.
|
||||
@@ -2730,13 +2761,18 @@ Version strings stay in the logs, the API and the hub.
|
||||
refuses without a restorable, proven Tier-2 copy, backs up first when that copy is stale, takes a
|
||||
safety dump, and holds an app that does not come up. **What stays true:** it still has no automatic
|
||||
rollback (deliberately, §6.1) and can still attempt a multi-major jump the app will refuse (R-40) —
|
||||
that now ends HELD rather than crash-looping behind a green button.
|
||||
that now ends HELD rather than crash-looping behind a green button. *(Dated note 2026-10-24: a copy on ANY tier now
|
||||
satisfies the precondition (v0.239.0), the box undoes a failed update itself (decision 15, v0.263.2), and the
|
||||
ladder climbs one tested step per press (decision 14) — `internal/backup/update_guard.go`, `internal/stacks/undo.go`,
|
||||
`ladder.go`.)*
|
||||
7. ~~**An engine major can be applied without its datadir upgrade, and nothing notices.**~~ **CLOSED
|
||||
2026-09-13 for MariaDB (R-459):** every `mariadb:` sidecar carries `MARIADB_AUTO_UPGRADE=1`, and the
|
||||
harness shows the conversion running on the E3/E3b edges (§3 decision 5). **What stays true:** the
|
||||
PostgreSQL half (R-463) has no equivalent — the image performs no `pg_upgrade` — and the
|
||||
engine-major rule (§3 precaution 3, R-469) is what keeps both engines inside their major until
|
||||
Slice 4 gives the Update button a backup.
|
||||
Slice 4 gives the Update button a backup. *(Dated note 2026-10-24: MariaDB is lifted — its major is its own edge; a
|
||||
PostgreSQL major may pass with a two-venue proof and an `engine_conversion` mark, and the box converts it
|
||||
(v0.273.0) — `app-catalog-felhom.eu/scripts/check-engine-major.py`.)*
|
||||
8. ~~**Only three of 53 apps have ever had an upgrade measured.**~~ **WIDENED 2026-09-21 to 21
|
||||
EDGES ACROSS 19 APPS** (`audits/DRILL-update-night-2026-09-21.md`), on scratch guest 9202
|
||||
through the product's own guarded Update, against a PRIVATE DRILL CATALOG so the live catalog
|
||||
|
||||
Reference in new issue
Block a user