## Changelog ### v0.156.0 — a dead primary alerts (R-51); a boot orphan restarts itself (R-52) (2026-07-21) **No new agent coupling — MinAgent stays 0.90.0.** Two independent failures from the same live audit, both unattended-resilience holes: the box was broken and nobody was told, then the box could have fixed itself and did not. **R-51 — a multi-container app whose MAIN container is dead now counts as down.** On 2026-07-20 `immich-server` sat `Exited` for **18 hours** with the app 100 % unreachable, and the box produced no dead-app banner and no `app_start_failed` event — while single-container Calibre-Web, down for the same reason, alerted in 90 seconds (AUDIT-vacation-remote-ops-2026-07-20 F4). The defect was one branch in `aggregateState`: a stack with *some* members running and *some* stopped returned `StateRunning` — "partial" — and `IsDownState` (correctly) does not treat running as down. So the alarm never had anything to fire on. *(The ROADMAP row's diagnosis — "aggregation classifies such a stack `unhealthy`" — is wrong at the source; corrected in the row.)* - New `StateDegraded`. The mixed branch now asks each DOWN member for its restart policy: a member docker is supposed to keep running (`always` / `unless-stopped`) makes the stack **degraded**, a finished one-shot (`no` / `on-failure`) leaves it running. `IsDownState` gains `degraded` and **nothing else** — the `unhealthy` / `restarting` / `paused` / `unknown` exclusions are byte- identical, because folding `unhealthy` into down is what fix-3 removed the flapping by not doing. - An **unreadable** policy counts as supervised (fail-CLOSED), the opposite of the IsDownState fail-open rule and for a different reason: there the *state* is ambiguous, here a member is known dead and only the excuse is missing. The P2 census backs it — all 53 catalog templates / 78 services are `unless-stopped`, and zero one-shot containers exist today. - The policy read is one `docker inspect` per down member of a *mixed* stack, cached per container+state and pruned to the live container set, so the 10 s refresh does not grow a docker call per container. - Everything that asks "are there live containers here" learns the state too: quiesce (`RunningAppStacks`), delete's stop-first guard, the export stop-first guard, telemetry, health probes. Everything that asks "is this app working" counts it as down: the dashboard counter, the stopped filter, the dead-app banner and the alarm. UI: „Részlegesen leállt", warn colour, and the URL is flagged unpublished (Traefik 404s when the routed member is the dead one). **R-52 — an app the boot left behind now gets exactly one recovery.** The same shutdown left immich and calibre-web `Exited` while ten sibling containers came back; the controller *reported* them for 18 hours and never started them (F5). - New `internal/bootrecon`: one bounded sweep at startup — at most 2 attempts, 30 s apart, then it stops and the alarm owns the problem. **Never a restart loop.** - **A deliberate Stop survives a reboot.** The UI's Stop is `compose down`, which REMOVES the containers; an interrupted boot leaves them behind as `Exited`. So the boot-orphan signature is "deployed, has containers, and they are down", and a zero-container stack is never touched. - The whole sweep (5 s settle + one 30 s gap) fits inside the 90 s `deadAppBootGrace`, so a successful recovery never alerts and a failed one alerts honestly. A test asserts that arithmetic rather than leaving it to a comment. **Seam discipline (the reason both features have a wiring test).** Two inert-seam defects shipped in the two days before this: controller v0.154.0 and agent v0.91.0, both a correct component with green tests and no production caller. So the boot sweep is asserted from `package main` — including an AST walk proving `func main()` actually contains the `go runBootReconcile(...)`. That test was written first as a `strings.Contains` and **its own red-proof passed it**, because a commented-out call still contains the string. Comments are not callers; the AST version fails as it should. Red-proofs (all run, all failed on the pre-fix shape, all restored): the mix branch reverted to `return StateRunning` → the immich fixture and both production-path tests fail with `"running"`; the boot hook commented out → the wiring test fails; the zero-container gate dropped → the user-stopped app is started, which is the one thing R-52 must never do. ### v0.155.0 — the restore wizard read the wrong "is something running" flag (2026-07-21) **No new agent coupling — MinAgent stays 0.90.0.** Fixes a defect shipped in v0.154.0 and found by the operator on the first live click-through, plus the dead phase-strip label from the same release. **The bug.** `backup.Manager` carries two different booleans and v0.154.0 read the wrong one: | flag | read by | set by | covers the verification restore? | |---|---|---|---| | `running` | `IsRunning()` | `acquireRunning()`, **inside** the goroutine | **no — `RestoreOffboxScratch` never acquires it at all** | | `opRunning` | `RestoreStatus()` | `BeginRestoreOp()`, in the handler, synchronously | yes, all four offsite actions | The wizard sourced `OpRunning` from `IsRunning()`. For „Ellenőrzés" and the full-restore preparation — the wizard's two most-used actions, and the long ones, since they stream from restic — that flag is false for the *entire* operation. So the execution step was unreachable: the page kept offering all three intents with live buttons while a restore was downloading, and the progress banner (which polls the op status) contradicted the phase strip on the same screen. Any button pressed there would have been refused by the handler — which is exactly the "offering a control guaranteed to fail" dishonesty R-48 exists to remove. **The fix** is one line of behaviour behind a named seam: `restoreOpInFlight(st)` takes the `RestoreOpStatus` the handler already reads once, and its doc comment states which flag is which and why. The handler now takes a single `RestoreStatus()` read, so the strip, the suppression decision and the running-op name can no longer disagree with each other. **Why the v0.154.0 tests missed it.** The Scenario-E table proved `deriveWizardStep` behaves correctly *given* `OpRunning=true`; nothing proved the handler ever computes `true`. Hollow at exactly that seam. `TestRestoreOpInFlight_UsesDisplayFlagNotConcurrencyFlag` now drives a real `Manager` through `BeginRestoreOp` and asserts the wizard suppresses every form — red-proofed against the v0.154.0 shape. **„Eredmény" is now reachable.** The fourth phase label never lit up in v0.154.0. The strip's highlight is now its own derived value (`Phase`), separate from `Step`: a finished restore is back on the intent step — everything is available again — while the strip rightly reads „Eredmény" and an outcome card shows the result. Bounded by `restoreResultWindow` (10 min) so a stale result cannot claim to be fresh, and bound to the app, so a finished bookstack restore does not light up immich's page with bookstack's message. The card survives a reload, which the redirect flash does not. ### v0.154.0 — one restore entry per app, and the intent is a described choice (2026-07-21) Closes **R-48**. **No new agent coupling — MinAgent stays 0.90.0.** This is a UI-layer change: `internal/backup`, `internal/appbackup` and `internal/selfupdate` are untouched, and the release adds **no mutation endpoint** — every action still posts to the `/backup/offbox/*` handler it always did, with the same field names and the same gates. **The defect.** The „Ellenőrző visszaállítás a távoli tárolóból" list rendered up to five inline `
` blocks per app row: verify, prepare, the revealed size-gated commit, the missing-only merge, and the true reconstitution. Two of them sat next to each other as sibling buttons — - „Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" — additive; **cannot** bring deleted content back, and - „Teljes visszaállítás (fájlok + adatbázis)" — the real restore — and the difference between them is whether the customer's data comes back at all. This is not theoretical: it caused the round-2 incident. An operator who had *read the source* pressed the missing-only button, and the controller log shows `/backup/offbox/reconstitute` was never hit (`felhom.eu/documentation/audits/DIAG-immich-restore-round2-2026-07-19.md`, finding 1). The second half of the trap was that the decisive „Teljes visszaállítás indítása" appeared **only after** „…előkészítése" had been pressed, with nothing signposting that a second step existed or that the first one had done nothing to live data. **The rule this establishes,** worth stating once and applying past this page: *two adjacent controls whose difference is "your data comes back" vs "your data cannot come back" must never be distinguishable only by layout.* **The change.** Each app row on `/backups/restore` now carries exactly **one** control — „Visszaállítás…" — linking to a per-app wizard at `GET /backups/restore/app?name=`, built on the `backups_escrow.html` precedent: - **Three intent CARDS**, each with its own consequence sentence rather than a label alone: ellenőrzés külön mappába (live data untouched) · hiányzó fájlok visszahozása (additive, no database, deleted content does not reappear) · teljes visszaállítás (files + database, danger styling, the R-43 double-confirm carried over **verbatim** with its pair-honesty facts). - **A visible phase strip** — Előkészítés · Megerősítés · Végrehajtás · Eredmény — so the sequence is legible before the first click instead of after it. - **Server-derived steps.** `deriveWizardStep` is a pure function of (op running, size-gate flash, scratch ready); the step is never accepted from the request. Precedence is strict: a running op outranks a stale `?full_prep=` in the URL, so no commit button can reappear mid-restore. - **Mutation forms are suppressed server-side while any op runs** — the backup manager's single-flight is process-wide, so a restore for app X now suppresses app Y's controls instead of offering a button guaranteed to 409. - **No JavaScript requirement.** Every step is a real form POST and the server renders the next one. **Redirect retargeting.** The app-scoped `/backup/offbox/{restore,place,reconstitute}` outcomes now land back on the wizard the customer acted from rather than on the list. Fixing that surfaced a latent bug in `offboxRedirectTo`, which hardcoded `"?"` when appending the flash — against a target that already carries a query (`?name=`) that would have buried the flash inside the `name` value. The separator is now chosen. **Deliberately NOT in scope:** the shares (`_shares`) entry, the local restore panel and the .fab block are untouched; the R-45 job registry is still its own item — the wizard polls the two existing status surfaces as-is. ### v0.153.0 — the database replay no longer races the application, on BOTH restore paths (2026-07-20) Closes **R-47**. **No new agent coupling — MinAgent stays 0.90.0.** Nothing in this release talks to the host agent; the whole change is inside the controller's own compose orchestration. **The defect, measured to the second.** On 2026-07-19 the offsite reconstitution was run deliberately and correctly (`felhom.eu/documentation/audits/DIAG-immich-restore-round2-2026-07-19.md`, finding **H4**). It executed its designed sequence — safety dump, stop, start, replay — and the replay aborted: ``` 10:58:25 controller: replaying DB dump into immich-postgres 10:58:33 immich-server: "Reindexing clip_index" -> "Reindexed clip_index" <- the app recreates it 10:58:35 controller: ERROR relation "clip_index" already exists - exit status 3 ``` The replay needs a running database container, so the code started the WHOLE stack first. That gave immich-server an eight-second window in which to rebuild the very schema objects the dump was about to create, and under `ON_ERROR_STOP=1` the collision aborted the script. The photos came back anyway **by accident**: `pg_dump` emits COPY data before CREATE INDEX, so the abort landed after the rows. A collision earlier in the script would have left a genuinely half-restored database and reported it identically. The operation reported failure and immich then reported schema drift. **The fix: a DB-only window.** After the files are placed, only the stack's database service(s) come up; the dump is replayed into them with the application still stopped; the rest of the stack starts only once the replay has exited 0. Nothing about the replay itself changed — `--clean --if-exists` and `ON_ERROR_STOP=1` were always correct. The bug was the window, not the flags. **This was a class defect and both paths carried it.** The local `RestoreFromRecoveryUnit` had the same start-then-replay shape, hidden inside `RecreateStackFromUnit` (which ended in a full `compose up -d`). Fixing only the offsite path would have left the identical race one button away. Both are re-sequenced here. **What changed** - `appbackup.DBServiceNames(composePath)` names the compose SERVICE(s) whose `image:` identifies a database — `docker compose up -d` takes service names, not container names. It is a yaml.v3 `services:` map parse, deliberately not a line scan: immich's real template carries top-level `immich_ml_cache:` and `immich_postgres_data:` volume keys that sit at exactly the indentation a service name does. - The image heuristic that `DiscoverDatabases` had inline is extracted to `dbTypeForImage` and shared by both. That sharing is what makes the safety argument hold: a `.sql` dump can only exist because discovery matched the running container's image, and the compose `image:` value IS that image string — so "a dump exists" and "a service can be named" are answered by one predicate. - `stacks.Manager.StartStackServices(name, services)` runs the scoped `up -d`. It **refuses an empty service list**: an argument-less `up -d` is a full start, which is precisely the behaviour the window exists to avoid, and a silent fall-through would have reintroduced the race at the one call site that most needs it not to. - `RedeployFromEnv` is split. Its persist half is now `PersistUnitRedeployConfig` (app.yaml, locked fields, in-memory flags — starting nothing); `RedeployFromEnv` is that plus its unchanged up-and-report tail, so its public behaviour is byte-identical. The split is what lets the restore path put the DB-only window between persisting the definition and starting the app. - `StackDataProvider.RecreateStackFromUnit` becomes `RecreateStackDefinitionFromUnit` (files + persist, no start), and gains `StartStackServices`. The rename is deliberate: the old name promised less than the method did, and the hidden `up -d` inside it is what carried the defect on the local path. **Fail-closed, both paths.** If a `.sql` dump exists but no database service can be identified in the compose, the restore **refuses before the first mutation** — no stop, no file overwrite, no volume restore. The alternative would be to start everything and replay into the race. Given the shared predicate this should be structurally unreachable; it is the belt for template drift, not an expected path. **Every exit from the window still starts the app.** A failed replay, or a failed DB-only start, is surfaced as before — but a best-effort full `StartStack` runs first. The DB-only state is a deliberate half-started one, and leaving a customer with a running database and no application would turn a failed restore into an outage. **Tests.** 19 new (Groups A–G): ordering plus **state-at-replay-time** on both paths (a recording provider captures whether the full stack was up at the moment the import fired — asserting "no error" would have passed on the pre-fix shape, which is how this shipped), the no-DB negatives, the zero-mutation fail-closed effects, the replay-failure bring-up, the compose-parser decoys built from the catalog's real immich template, and the empty-list refusal. Three companion red-proofs run and reverted: the pre-fix full start on the offsite path, the pre-fix full start on the local path, and deletion of both fail-closed gates — each failing on the intended assertion. 23/23 packages green. **Live-validated on the demo box, 2026-07-20 (operator present).** Endpoint-level, against the SAME snapshot (`49e7cb46`) that aborted in round 2: ``` 15:39:42 [stacks] Stopping stack: immich 15:39:43 [stacks] Starting stack immich services only: [immich-postgres] 15:39:43 [backup] Restore immich: replaying DB dump into immich-postgres (postgres) 15:40:03 [backup] Restore immich: replayed 1 DB dump(s) <- rc-0, no "already exists" 15:40:03 [stacks] Starting stack: immich 15:40:27 [offbox] reconstituted immich: 6 file(s) placed, 1 DB dump(s) replayed, skewed=false ``` The operation reported **success** (round 2 reported failure); immich's own DatabaseService logged **`No schema drift detected`** twice, where round 2 left it reporting drift; 11 assets `active`, all four containers healthy, 231 `public` indexes. Details in `REPORT.md` §4b. **Golden 0.153.0 baked + published the same day** (`build-golden.sh` v2.1.0, from the vacation site after a registry-reachability probe). First golden carrying **all four** infra images — the list came from `--print-infra-images` on the 0.153.0 binary itself, so the historical 3-image fallback never fired and `felhom-samba:1.1.0` is baked. Upload 201, anonymous GET byte-matches, ranged 206. ``` GOLDEN_VERSION=0.153.0 GOLDEN_SHA256=15fdd191f3c660a60dc8651111053dd84281aeebc6c4c0f9ecdd3a87cb45a9d0 ``` **Still outstanding:** the two password-gated hub saves (Day-0 manifest Golden → 0.153.0, then floor → v0.153.0 **last**; Agent 0.90.1 / MinAgent 0.90.0 unchanged), Viktor's C6 customer-restore UI run, and the immich timeline screenshot — all of which need the operator UI or a browser. ### v0.152.0 — Megosztás on a Mac: mDNS in the image, and the page stops giving Mac users a dead form (2026-07-20) Closes **S-3** of `felhom.eu/documentation/audits/DIAG-sharing-2026-07-20.md`, and fixes a copy defect v0.151.0 shipped the same day. **Pairs with felhom-samba 1.1.0** — the pin in `infra.SambaImage` moves with it, so `Images()` and the golden bake follow automatically. **The finding that redirected the fix — macOS asks, gets a correct answer, and ignores it.** The first theory was that modern macOS no longer does NetBIOS. A packet capture on the box disproved that: on a bare `smb://FELHOM` the Mac broadcasts a well-formed NBNS query for `FELHOM<20>` (the File Server Service suffix — exactly right for SMB), and nmbd answers in 140 microseconds with a textbook positive response — flags `0x8580` (response, authoritative, RCODE=0), ANCOUNT 1, unique B-node, the correct address. **macOS never opens a TCP connection.** Sixteen seconds later the same Mac connected through `smb://FELHOM.local` on the first try. NetBIOS on macOS feeds legacy browsing, not `smb://` URL resolution — so no change on our side can ever make the bare name work there, and nmbd is not the thing that was broken. (nmbd answers twice per broadcast, because it holds `0.0.0.0:137`, `:137` and `:137` and a broadcast lands on two of them. Standard Samba; investigated and dismissed — a duplicated correct answer is still a correct answer.) **felhom-samba 1.1.0 — avahi + dbus, so the Mac has a mechanism at all.** The image's discovery set was Windows-only: nmbd for flat-name resolution, wsdd for Explorer's Network view, and nothing whatsoever for Bonjour. It now runs avahi, with `avahi-daemon.conf` and an `_smb._tcp` service file **templated from `FELHOM_SERVER_NAME` in the entrypoint** — renaming the server in the UI re-advertises under the new name, where a baked name would leave the box answering to something the customer can no longer see anywhere. A static service file rather than smbd's own `multicast dns register`: it needs no line in `smb.conf` (bind-mounted READ-ONLY, owned by the controller's renderer) and it lets us publish `_device-info._tcp` for a sensible Finder icon. Both new daemons are non-fatal on failure — sharing over an address must not become an outage because a discovery daemon did not come up. Proven live from the operator's Mac before the image was built, then the built image smoke-tested with all five daemons up and avahi registered as `.local`. **The page no longer tells Mac users to do the one thing that cannot work.** v0.151.0's connect card offered `smb://` for Mac. That is precisely the dead form. It is now `smb://.local`; the Windows line stays the flat `\\`, which nmbd serves correctly and which this release must not disturb. Red-proofed: reverting the template to the bare name turns `TestSharingConnectCard_MacLineIsDotLocalNotBareName` red on both the missing `.local` and the present bare form, for two different configured names — and the same test asserts the Windows line neither disappears nor wrongly gains `.local`. **NOT claimed: automatic Finder-sidebar discovery.** The `_smb._tcp` record is published and answers browse queries on the wire, but the test Mac's sidebar stayed empty — it had no Network/Bonjour section shown at all, which is a Finder Settings toggle rather than something the box controls. This is recorded as OPEN in the DIAG, deliberately not as a shipped feature. **Two test bugs surfaced and fixed, neither a production defect.** `TestRenderSambaCompose` asserted the literal tag `felhom-samba:1.0.0`, so a routine image bump read as a renderer regression; it now derives from `SambaImage` and separately asserts what actually matters — that the tag is explicit and never `:latest`. And `TestFabUpload_GCAndIdleTimeout` raced: `expireIdleUpload` nils the slot, releases the mutex, and only then closes and unlinks the `.part`, so "the slot is free" does not yet mean "the file is gone" — the test stat-ed immediately and passed only by luck. It failed in the full package while passing in isolation once this release's new render tests made the `web` package heavier. Now it waits for the outcome it asserts, on the same 3 s deadline; red-proofed by removing the unlink from production, which still fails it. ### v0.151.0 — the Megosztás page stops reloading, and says how to connect (2026-07-20) Closes **S-1**, **S-2**, **S-5** and the core of **S-4** from `felhom.eu/documentation/audits/DIAG-sharing-2026-07-20.md`. **S-1 — `/sharing` reload-looped about once a second, for every customer with sharing enabled.** `GET /sharing/status` carries two things that mean different things to the client: `phase` (the ensure JOB — the page answers a terminal `running` with a one-shot `location.reload()`, because the „Állapot" badge is server-rendered) and `running` (the service LEVEL, straight from the liveness probe). v0.147.0 coerced `idle`→`running` on the PHASE channel so that a missing job could never contradict a live container. That duty was real, but it belongs to — and was already discharged by — the `running` field beside it; on the phase channel the same value reads as a fresh success edge. The poll's `tick()` runs synchronously at script end, so the FIRST poll of every steady-state page load reported a terminal job that had never run, scheduled a reload 1.2s later, and the new page did it again. The coercion is gone: no job, no edge. The defensive intent it was written for is now pinned by its own named regression test on the `running` field. **S-4 (core) — a REAL bring-up is now reported exactly once.** Without this the loop would return after every future image update: the finished job outlives the reload it triggered, so the next page load found `phase:"running"` waiting for it. `consumeIfRunning` serves a terminal `running` once and clears it — and only while the single-flight slot is free, since the job goroutine sets the phase before its deferred `release()` and eating it in that window would lose the success the customer is waiting on. `failed` and `needs_password` stay sticky (their client path stops the timer and shows a card with NO reload, so stickiness is informative and cannot loop), and in-flight phases are never consumed. Accepted cost, stated rather than hidden: with two tabs open during a bring-up only the first gets the success banner — both still show the true state, which comes from the level channel. The unified async-job feedback layer remains the ROADMAP item; this is the minimal contract fix. **S-2 + S-5 — the page now names both ways in.** It had only ever shown the configured NetBIOS name, so a customer whose network fails to resolve it had no fallback but a guess — and the guess that produced the diagnosis was the Proxmox HOST's address, which never ran smbd. New „Csatlakozás a megosztáshoz" card: the Windows form, the Mac form, and the direct `smb://`. The address comes from `stacks.SambaLANAddress()`, which reads the guest's netns through the SAMBA container (`network_mode: host`) — the controller is on a docker bridge and would answer `172.x`, the same trap `setup.DetectLocalIPs` needs `HOST_IP` for. Reading it there also makes it the right kind of true: it is the address smbd is bound to, not merely one the box owns. **Derived per render and cached nowhere** — the guest holds it by DHCP, so a stored copy eventually misdirects people (S-5) — and an underivable address omits the line, because no address beats a wrong address. `sharing.html`'s `