R-51: aggregateState's mixed branch returned StateRunning ("partial"), so a stack whose
MAIN container was dead behind live helpers alerted on nothing — immich-server sat Exited
for 18 h, 100 % unreachable, no banner and no app_start_failed (audit F4). New
StateDegraded: a DOWN member whose docker restart policy is always/unless-stopped is a
fault (degraded, a down state); no/on-failure is a finished one-shot and stays benign; an
unreadable policy fails CLOSED. The unhealthy/restarting/paused/unknown exclusions are
byte-identical — folding unhealthy into down is the flapping fix-3 avoided.
R-52: new internal/bootrecon — one bounded start-once sweep at startup (2 attempts, 30 s
apart) for apps an interrupted boot left behind, inside the 90 s boot grace so a success
is silent and a failure still alerts. A zero-container stack is NEVER touched: the UI's
Stop is compose down, so a deliberate stop survives a reboot.
Both features carry a production-path wiring test (the v0.154.0 / v0.91.0 inert-seam
class). The main() assertion is an AST walk, not strings.Contains — the substring version
passed its own red-proof, because a commented-out call still contains the string.
Red-proofs run and restored: mix branch reverted -> "running" on the immich fixture;
boot hook commented out -> wiring test fails; zero-container gate dropped -> the
user-stopped app gets started.
NOTE: controller/cmd/controller/ is matched by .gitignore's `controller` entry, so new
files there need `git add -f` (and ripgrep silently skips main.go without --no-ignore).
Fixes a defect shipped in v0.154.0, found by the operator on the first live
click-through of the new wizard.
backup.Manager carries TWO running booleans. `running` (read by IsRunning) is the
concurrency single-flight, acquired inside the background goroutine — and
RestoreOffboxScratch never acquires it at all. `opRunning` (read by RestoreStatus) is the
display flag, set synchronously by BeginRestoreOp in the handler.
The wizard sourced OpRunning from IsRunning(), so for „Ellenőrzés" and the full-restore
preparation — its two most-used and longest actions, both streaming from restic — the
execution step was unreachable: the page offered all three intents with live buttons
while a restore was running, and the progress banner contradicted the phase strip on the
same screen. Pressing anything there would have been refused by the handler, which is the
exact "offering a control guaranteed to fail" dishonesty R-48 exists to remove.
Fix: restoreOpInFlight(st) behind a documented seam, fed by a SINGLE RestoreStatus() read
per render so the strip, the suppression decision and the running-op name cannot diverge.
Why the tests missed it: the Scenario-E table proved deriveWizardStep behaves correctly
GIVEN OpRunning=true, but nothing proved the handler ever computes true — hollow at
exactly that seam. TestRestoreOpInFlight_UsesDisplayFlagNotConcurrencyFlag now drives a
real Manager through BeginRestoreOp and asserts the render suppresses every form.
Red-proofed against the v0.154.0 shape.
Also: „Eredmény" was a dead label. The strip's highlight is now its own derived Phase,
separate from Step — a finished restore returns to the intent step (everything available
again) while the strip reads „Eredmény" and an outcome card shows the result. Bounded by
restoreResultWindow (10 min) so a stale result cannot look fresh, and bound to the app so
a finished bookstack restore does not light immich's page with bookstack's message. The
card survives a reload; the redirect flash does not.
No new agent coupling — MinAgent stays 0.90.0.
The offsite restore list rendered up to five inline forms per app row. Two of them —
„Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" and „Teljes visszaállítás
(fájlok + adatbázis)" — were sibling buttons whose difference is whether the customer's
data comes back at all. That mis-selection CAUSED the round-2 incident: an operator who
had read the source pressed the missing-only button and /backup/offbox/reconstitute was
never hit (DIAG-immich-restore-round2-2026-07-19, finding 1).
The rule this establishes: two adjacent controls whose difference is "your data comes
back" vs "your data cannot come back" must never be distinguishable only by layout.
Each row now carries ONE „Visszaállítás…" entry linking to a per-app wizard at
GET /backups/restore/app?name=<app>: three intent CARDS with consequence sentences, a
visible phase strip, danger styling plus the R-43 double-confirm carried over verbatim on
the destructive one, and the pair-honesty panel bound to real OffsiteScratchPair facts.
deriveWizardStep is pure over (op running, size-gate flash, scratch ready); the step is
never accepted from the request, and a running op outranks a stale ?full_prep= so no
commit button survives into a restore. While ANY op runs every mutation form is suppressed
server-side instead of being offered and refused.
No new mutation endpoint: every card posts to the pre-existing /backup/offbox/* handler
with the same field names and gates, and the page works with JavaScript disabled.
internal/{backup,appbackup,selfupdate} untouched. R-45's job registry stays its own item.
Fixes a latent bug found on the way: offboxRedirectTo hardcoded "?" when appending its
flash, which against the wizard's ?name=<app> target would have buried the flash inside
the name value.
No new agent coupling — MinAgent stays 0.90.0.
Closes R-47. No new agent coupling — MinAgent stays 0.90.0.
The replay needs a running DB container, so both restore paths started the
WHOLE stack first, giving the application a window to rebuild the very schema
objects the dump was about to create. Measured live on 2026-07-19 (H4,
DIAG-immich-restore-round2): immich-server rebuilt clip_index two seconds
before the dump's CREATE INDEX, the replay aborted "already exists" under
ON_ERROR_STOP=1, and immich reported schema drift. The data survived only
because pg_dump emits COPY before CREATE INDEX.
Both paths now open a DB-ONLY window: only the stack's database service(s)
come up, the dump is replayed with the app still down, and the full start
runs only after the replay exits 0. Fail-closed: a dump with no identifiable
DB service refuses BEFORE the first mutation. Every exit from the window
still does a best-effort full start, so a failed restore never leaves a box
with a database and no application.
New: appbackup.DBServiceNames (yaml.v3 services-map parse — never a line
scan; immich's top-level volume keys are the decoy) sharing dbTypeForImage
with DiscoverDatabases; stacks.Manager.StartStackServices (refuses an empty
list — argument-less `up -d` is a full start); RedeployFromEnv split into
PersistUnitRedeployConfig + its unchanged tail. StackDataProvider's
RecreateStackFromUnit becomes RecreateStackDefinitionFromUnit — the hidden
`up -d` inside the old name is what carried the defect on the local path.
19 new tests (ordering plus state-at-replay-time, zero-mutation fail-closed
effects, replay-failure bring-up, parser decoys, empty-list refusal); three
companion red-proofs run and reverted. 23/23 packages green.
Not yet live-validated: STOP-1 supervised reconstitute, golden 0.153.0.
S-1: /sharing/status coerced idle->running on the PHASE channel, so the first
poll of every steady-state page load reported a terminal job that never ran and
the client's repaint-reload fired ~1.2s apart, forever. The coercion's real duty
(liveness must never be contradicted) belongs to the 'running' LEVEL field
beside it, and is now pinned by its own regression test.
S-4 core: a terminal 'running' is served exactly once, so a REAL bring-up cannot
re-arm the reload on the page it just caused. failed/needs_password/in-flight
are never consumed. Unified async-job feedback stays the ROADMAP item.
S-2/S-5: new connect card with the Windows form, the Mac form and the direct
smb://<IP>, read from the SAMBA container's netns (the controller is on a docker
bridge and would answer 172.x). Derived per render, cached nowhere - the address
is a DHCP lease. Underivable => the line is omitted.
sharing.html's <script> block is byte-identical to v0.150.0. Red-proofed three
ways. 23/23 packages green.
Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.
R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.
New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
- nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
created after the snapshot survives as an extra;
- the undo exists before the act — the pre-restore- dump is verified ON DISK
before anything is stopped, overwritten or replayed; if it cannot be taken
the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.
R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.
Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.
11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.
NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.
Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
offsiteRestoreRootFor as THE home for backups/offsite-restore (it was open-coded
in three places). Two patterns: the detached-job+poll shape, now noting that FIVE
of them exist and agree on nothing so a sixth should extend rather than clone;
and streaming subprocess progress, which encodes the two traps tonight found —
a source reporting nothing is normal, and progress may only update on unit
completion, so degrade rather than fake a percentage. Seams: sambaImgFn and
offboxStreamRunner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The samba_classify.go SCOPE NOTE and the README KNOWN GAP both described a gap that
R-7b closes; both now describe the sibling-shares-source execution instead.
Records the R-7 slice 1 ship, the four red-proof outcomes, live-validation
evidence, the two bugs live validation caught, and the Part-4 Step-1 enumeration
finding + the reported design fork (share data classified but not yet in a live
backup run — needs a Viktor ruling, suggested R-7b).
After a completed escrow ceremony the Távoli mentés page showed the yellow
"Helyreállítási kód szükséges" card for ~15 min until the next hub-report ACK
flipped pending→escrowed. Phase-0 diagnosis (read-only) = verdict A (report-cycle
lag), already resolved on the demo box (escrow_state:"escrowed"); hub Hypothesis B
verified false (SaveHostEscrow ON CONFLICT already clears stale_at on upload) → no
hub change.
- settings.OffboxTarget.CeremonyCompletedAt: stamped on the recovery-code claim,
zeroed on the auto-confirmer Flip + the deprecated manual confirm; persisted.
- web/handlers.go: offboxCeremonyWaitState + escrowCeremonyGraceWindow (35m).
- backups_remote.html: info "megerősítésre vár, legfeljebb 15 perc" card → warn
"a megerősítés nem érkezett meg" past the window. Existing branches untouched.
- backups_escrow.html: "Mi történik ezután?" note on the wizard's final step.
- Test web/escrow_wait_state_test.go (truth table + red-proof recorded in REPORT).
No scheduler/agent/hub/endpoint changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp
Task 2 of the backup-classification-redesign arc. Ships the referential-
coupling classification as DATA + PARSER + PURE CLASSIFIER, deliberately
inert — no backup tier changes behavior. Task 3 (tier policy engine) and
Task 4 (manual .fab UI) consume it.
- appbackup/classify.go: BackupSpec/BindSpec/ComposeBind/ClassifiedBind;
ClassifyBinds (SQ5 two-level default — explicit beats :ro; unlisted
writable→mandatory, unlisted :ro→excluded; nil spec→legacy/false);
ValidateBackupSpec (whole-block-reject on any defect, first defect named).
- stacks/classify_binds.go: ParseComposeClassifiableBinds — ${VAR}-relative
binds + :ro flag (NOT ParseComposeHDDMounts/ExportDataMounts, the traps).
- Metadata.Backup + LoadMetadata as the single validation choke point (bad
catalog block → nil + one ERROR → legacy, within one sync cycle).
- Manager.ClassifiedBinds + StackDataProvider.GetStackClassifiedBinds seam
(delegated by stackAdapter, nil-stubbed in every fake) — wired + tested
now so Task 3 consumes a tested seam.
INERT: full pre-existing suite green with zero test-logic edits. +14 tests;
red-proofs RP-1..RP-4 confirmed. The 13 catalog backup: blocks ship in the
same app-catalog change (this controller deploys first).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
The controller assumed an app's HDD appdata dir is always appdata/<stackName>.
paperless-ngx writes appdata/paperless (stack paperless-ngx), so every consumer
keying by stack name silently missed it via a stat-and-skip. One canonical
resolver appbackup.AppDataDirNames derives the real dir name(s) from the app's
compose ${HDD_PATH} binds; all consumers use it.
- F-S2 (tier-2): RunTier2 mirrors the resolved appdata/<name> (paperless docs
got NO tier-2 copy before). Tier2Info size + RestoreTier2Files live dir use it.
WARN when a declared appdata dir is absent. New tier2Mirror seam.
- F-S3 (migrate, NEW): all six per-app appdata legs (collision/size/copy/verify/
cleanup/skip-set) now loop resolved names. scope="app" migration of paperless
previously copied nothing and left an empty media dir (scope="all" was saved by
the merge walk). WARN on missing declared dir in the copy leg.
- Multi-dir (N>1) refusal: tier-2 backup/info/restore refuse loudly (Hungarian);
migrate supports N. No catalog app hits it today; lifted by Task 3.
- Display: storage page sums resolved dirs.
- Truth repair: the v0.130.0 "tier-2 copies the namespace wholesale" claim is
false; corrected in CHANGELOG + main.go export-adapter comment.
+9 tests; red-proofs RP-1..RP-5 all confirmed. Controller-only, no agent/hub
coupling. Task 1 of the backup-classification-redesign arc.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A45Qop8YY8tS94bz63LFne
admin