R-237: /backups/restore listed apps that are CURRENTLY DEPLOYED and CURRENTLY
TOGGLED ON for future off-site backups. A rebuilt box has neither, so a household
that had just lost everything was shown nothing to restore while the repository
held their snapshots — measured live on the R-201 re-walk. To restore an app you
had to select it, to select it you had to have installed it, and to know what to
install you had to see the backup you could not see.
The store is now the source of the list (offsite_restore_list.go), built on the
existing R-193 OffsiteInventoryList. Installed-ness became a property OF a row,
never a filter on it. Every case is answered rather than hidden: a snapshot for an
app that is not installed is offered and says it will reinstall first; an installed
app with no snapshot is shown as having nothing; an unreadable store renders as
UNKNOWN (R-225's rule, one screen over) AND keeps the action, because "we could
not look" is not "there is nothing"; no-target is its own state. The felhom-offbox
and _shares marker tags are excluded from the app list.
R-238 classified as a HARNESS ARTIFACT: mode=full without confirm=1 is step 1 of a
deliberate two-step — it starts no job by design and redirects carrying
&full_prep=<app>, which deriveWizardStep requires to reveal the commit. A driver
that did not carry it forward landed back on the intent step. The operator's
browser run completed the same restore. The wizard's precedence rules were NOT
re-keyed: a stale ?full_prep= must never resurrect a commit button mid-restore.
The residue WAS real and is fixed: neither branch of that step wrote anything to
the log, so a refusal — including by the headroom gate — left no trace on the box.
Both branches now log, and so does the concurrent-op refusal.
resolveWizardApp is removed: it was dead once the gate moved, and its test pinned
the defect's behaviour (an untoggled app refused), which would have read as policy.
28 packages ok, 9/9 gates OK. Three red-proofs, each asserted to have applied.
R-225 — an unread store said '0 pillanatkép / 0 / 50 GB' above a card stating
it held backups under another key. An SFTP listing found snapshot f3d9cd67 and
12 535 KB really there; snapshot_count and repo_size_bytes were simply ABSENT
and the zero value spoke for them. StatsKnown is now NAMED, for the same reason
OffsiteInventory.Empty is: zero is what an unread store and an empty one both
look like, and on the wire 'absent' and '0' are the same bytes. The fill bar
renders only when the fill is known — a 0%-wide bar is a picture of emptiness,
and a picture is a claim. A measured zero still says zero.
R-227 — WHICH LAYER ANSWERS: traefik, and this repo generates its config. But
traefik v3 serves no static files, so a branded proxy page needs a new always-up
container for every 502 on the box — out of proportion, and scoped in the report
rather than built. Shipped instead: the unlock posts via fetch and answers a
gateway failure in Hungarian without leaving the page. Progressive enhancement —
with no JS the plain POST is unchanged and still shows the proxy's error, which
the report says plainly rather than implying otherwise.
R-228 — the set-aside history was recorded in orphaned_renamed_to and read by
nobody: a census found zero references in any template or handler, while 12 535
KB sat at that path. It is surfaced as two facts and stops. It does NOT promise
the history can be reopened, because it cannot be by anyone today (R-199's
inventory is unbuilt) — and the set-aside CONFIRMATION copy was corrected for
the same reason: 'a helyreállítási kód nélkül többé nem lesznek megnyithatók'
implied that WITH the code they could be. The field's own comment called it
'recovery-code-recoverable', which was the same over-promise in the code.
Tests: scenarios F, G, H as render tests per branch of each gate. Red-proofs,
each demonstrated failing then restored: remove the StatsKnown guards (F,
'R-225 RETURNED: an unread store reports a snapshot COUNT of zero'), delete the
set-aside block (H). The F assertion on the fill bar is scoped to the bar's own
container — a bare width:0% search matched unrelated elements and would have
passed for the wrong reason.
28 packages ok, vet clean, all controller gates OK (the emoji gate caught a
warning sign in a template comment).
mode=unit restores the recovery unit — the app's definition, configuration
and database dumps — and NOT the customer's own files: RestoreOffboxScratch
passes --include <unit path> and the userdata in the same snapshot is excluded
by it. The outcome was one sentence for both modes and named neither scope,
so on the last step of a disaster recovery the customer was told the app had
been restored after the thing they were looking for had not been.
restoreScratchOutcomeMsg states what came back, what did not, and the next
step that gets it. The wizard's intent card states its scope before the choice.
The full-restore size gate is untouched and pinned as unchanged; the default
stays unit, since all three wizard forms set mode explicitly.
LastRun records an attempt, not a result. New OffboxTarget.LastSuccess, set only on the
success branch via the pure offboxAnchorAfterRun rule, carried to the hub as last_success.
Closes two silent-wipe sites (settings save, hub re-apply).
The offsite restore list rendered up to five inline forms per app row. Two of them —
„Helyreállítás az élő adatok közé (csak a hiányzó fájlok)" and „Teljes visszaállítás
(fájlok + adatbázis)" — were sibling buttons whose difference is whether the customer's
data comes back at all. That mis-selection CAUSED the round-2 incident: an operator who
had read the source pressed the missing-only button and /backup/offbox/reconstitute was
never hit (DIAG-immich-restore-round2-2026-07-19, finding 1).
The rule this establishes: two adjacent controls whose difference is "your data comes
back" vs "your data cannot come back" must never be distinguishable only by layout.
Each row now carries ONE „Visszaállítás…" entry linking to a per-app wizard at
GET /backups/restore/app?name=<app>: three intent CARDS with consequence sentences, a
visible phase strip, danger styling plus the R-43 double-confirm carried over verbatim on
the destructive one, and the pair-honesty panel bound to real OffsiteScratchPair facts.
deriveWizardStep is pure over (op running, size-gate flash, scratch ready); the step is
never accepted from the request, and a running op outranks a stale ?full_prep= so no
commit button survives into a restore. While ANY op runs every mutation form is suppressed
server-side instead of being offered and refused.
No new mutation endpoint: every card posts to the pre-existing /backup/offbox/* handler
with the same field names and gates, and the page works with JavaScript disabled.
internal/{backup,appbackup,selfupdate} untouched. R-45's job registry stays its own item.
Fixes a latent bug found on the way: offboxRedirectTo hardcoded "?" when appending its
flash, which against the wizard's ?name=<app> target would have buried the flash inside
the name value.
No new agent coupling — MinAgent stays 0.90.0.
Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.
R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.
New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
- nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
created after the snapshot survives as an extra;
- the undo exists before the act — the pre-restore- dump is verified ON DISK
before anything is stopped, overwritten or replayed; if it cannot be taken
the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.
R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.
Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.
11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.
NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.
Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
The systemic complaint, twice in one evening: you press a button and nothing
happens. No progress, no ETA, no named result. Three worst offenders, fixed on
the two patterns already here (deploy 3-step panel, storage-init status poll).
No new framework — that is a ROADMAP item; three targeted cards ship tonight.
4a — a verification restore names its result. The flash said the app had been
restored "to a verification folder on the drive"; which folder, on which drive,
was invisible, so the customer could not go and look at what they had just asked
for. Full path now. The restore page gained a listing of existing verification
copies (app, size, date, path) — nothing anywhere showed these, so they piled up
and the only way to find them was SSH — each with a double-confirmed delete.
That delete is the only one this release adds, so it names a STACK, never a
path: the Manager resolves the name inside a backups/offsite-restore root it
computed itself and refuses anything landing outside. Red-proofed — neutralise
the name guard and stack:"" resolves to the offsite-restore ROOT and takes every
copy with it. Refusals are asserted as non-effects.
4b — Megosztás enable shows what it is waiting for. Enabling ran ReconcileSamba
synchronously inside the POST handler; on a golden without felhom-samba baked
that is compose pulling ~100MB, i.e. minutes of an apparently-hung form post
followed by "Beállítás mentve." whether or not anything came up. Detached +
polled now, distinguishing "képfájl letöltése" from "indítás" — decided BEFORE
the work starts, since afterwards the image is always present. Success is
probed, not inferred (compose up -d exits 0 on a crash-loop). The password form
starts the same job: with UserSet false reconcile deploys nothing, so on a fresh
box that is where the pull actually happens.
4c — "Távoli mentés most" streams real progress. restic was already reporting
bytes and percent; the runner seam used CombinedOutput() and discarded them. The
manual run now passes --json and scans stdout line-by-line: total bytes, percent,
current app. Manual only — the nightly stays silent, pinned by a test that fails
if it ever passes --json. The poll now arms unconditionally, closing a race the
manual trigger always ran: the redirect rendered before the goroutine wrote
LastStatus=running, so the poll never armed and the page sat static during the
very run just started. Red-proofed twice.
Also closes the golden/controller infra-image drift at the source: infra.Images()
derives from the existing pins and --print-infra-images exposes it, so the golden
bake can stop carrying its own copy. That copy had already drifted — felhom-samba
was never added, so the golden baked 3 of 4, which is why enabling Megosztás
pulled at runtime in the first place.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
Part 4 — restore: RestoreSharesScratch + PlaceSharesRestore as SIBLINGS of the
per-app scratch/place pair. Files merged missing-only (never overwriting), each
destination PREFIX-ASSERTED against registered LIVE storage roots; definitions
merged with existing-wins; ReconcileSamba via a seam (backup must not import
stacks); credential restored best-effort into the samba named volume.
New routes POST /backup/shares/{restore,place} + a restore-page entry that renders
'Megosztasok', never the raw reserved key.
Also adds scratchJoin: reconstructing an absolute captured path under a scratch
must strip the volume name rather than rely on filepath.Join.
Part 5 — liveness: EffectiveProtected gains a settings-backed dynamic extra so the
samba CONTAINER (not the stack name — they differ) is watched exactly while sharing
is on. FINDING: the issue -> health 'fail' -> existing health_critical event ->
alert -> Hungarian degradation e-mail path needs NO further change, and introduces
no new event type, so the allowlist gotcha does not apply.
Part 6 — UI: per-tier backup status lines on the Megosztas page (amber only on
deviation). Verified the two warning-prose sites (offbox_capture/tier2_capture)
only ever receive per-app stack names, so no mapping is needed there.
RED-PROOFS RUN AND REVERTED (both fired):
4. prefix-assert removed -> place-guard traversal test FAILS
5. dynamic samba extra removed -> Scenario E enabled-case FAILS
After a completed escrow ceremony the Távoli mentés page showed the yellow
"Helyreállítási kód szükséges" card for ~15 min until the next hub-report ACK
flipped pending→escrowed. Phase-0 diagnosis (read-only) = verdict A (report-cycle
lag), already resolved on the demo box (escrow_state:"escrowed"); hub Hypothesis B
verified false (SaveHostEscrow ON CONFLICT already clears stale_at on upload) → no
hub change.
- settings.OffboxTarget.CeremonyCompletedAt: stamped on the recovery-code claim,
zeroed on the auto-confirmer Flip + the deprecated manual confirm; persisted.
- web/handlers.go: offboxCeremonyWaitState + escrowCeremonyGraceWindow (35m).
- backups_remote.html: info "megerősítésre vár, legfeljebb 15 perc" card → warn
"a megerősítés nem érkezett meg" past the window. Existing branches untouched.
- backups_escrow.html: "Mi történik ezután?" note on the wizard's final step.
- Test web/escrow_wait_state_test.go (truth table + red-proof recorded in REPORT).
No scheduler/agent/hub/endpoint changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp
EscrowAutoConfirmer flips pending->escrowed ONLY when sha256(local repo
password) matches the ACK's restic_pw_sha256 (blob-presence alone never
confirms — red-proofed). Mismatch warns once per hash naming the ceremony;
never un-confirms; wipes the staged secret on flip. Pinned cross-repo hash
vector; manual confirm deprecated to a legacy-blob fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Key-auth-first: a KeyAuthProber seam lets the bridge skip consume+install
when the already-installed key still authenticates (pinned to the freshly
verified host key) — descriptor changes on provisioned guests no longer
loop on consume-404. Fingerprint verify still precedes everything.
Wipe-on-escrowed: confirm-escrow now calls the agent's new
DELETE /escrow/stage-secret (v0.78.0) best-effort, closing the hygiene gap
where a ceremony-less confirm left the staged password file behind.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Pairs with agent v0.77.0. StageEscrowSecret pushes the repo password to the
agent (POST /escrow/stage-secret) at offsite-enable → EscrowState="pending".
Atomicity gate: RunOffboxBackup (scheduler + handler) refuses until
EscrowState="escrowed" (operator POST /backup/offbox/confirm-escrow after the
escrow ceremony) — no un-recoverable offsite ciphertext can exist. DR:
POST /backup/offbox/inject-password pre-places a recovered 64-hex password 0600
(honored by WriteOffboxSecrets' IsNotExist guard; refuses clobber without
force). DR recipe gains non-secret offsite_restic coords (DRResticCoord); SFTP
key regenerated at DR, not escrowed. New settings.OffboxTarget.EscrowState.
Tests + atomicity & inject companion red-proofs green; UI gates pass. NOT yet
live-validated (supervised ceremony).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
offbox located each toggled app's recovery unit via AppNamespaceRoot→GetAppDrivePath,
which reads the app's LIVE app.yaml HDD_PATH and silently falls back to systemDataPath
when the app isn't deployed → looked on the wrong drive, backed up nothing, reported
ok/0 (DIAG root cause). Now:
- discoverOffboxUnit/offboxCandidateNSRoots scan the durable storage registry
(schedulable non-decommissioned paths ∪ systemDataPath) for backups/primary/<app>,
independent of deploy state; newest-by-manifest-CreatedAt wins on drive churn.
- RunOffboxBackup: runOffboxInternal returns (backedUp, missing, err); 0-of-N toggled →
hard error + operator alert; partial → ok + new OffboxTarget.LastWarning (shown on
/backups, preserved across config edit).
- AppNamespaceRoot + primary WRITE paths unchanged.
- 6 non-hollow tests (A-E + edge) + both companion red-proofs run (reverted).
- NOT yet live-validated against the Storage Box (spike creds torn down).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Re-adjudicates F4: /backup/restore, /backup/tier2/restore, /backup/offbox/restore
blocked the HTTP request until completion, so through cloudflared's 100s cap a
customer got an error page while the restore succeeded (offbox worse — bounded
on r.Context(), canceling the SFTP restore mid-flight). Convert all three to the
offboxRun async shape: fast-path IsRunning refuse, background goroutine
(offbox ctx off r.Context() -> Background+30m), instant redirect. Add mutex-
guarded op-status (opstatus.go) + GET /api/backup/restore-status + a 3s-polling
backups.html banner (neutral running, red on failure). Restore single-flight
unchanged. Tests + red-proof (sync handler blocks indefinitely vs <500ms async).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Encrypted restic repo over SFTP for the app-data tier (the off-site 3-2-1 leg). A dead
NAS fails fast via -oConnectTimeout (spike Q8), never hangs the runner; secrets are 0600
files (ride DR via PBS whole-CT); init-if-absent, retention forget --prune, restore,
single-flight, per-app toggle + UI. restic re-added to the image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
admin