R-225 — an unread store said '0 pillanatkép / 0 / 50 GB' above a card stating
it held backups under another key. An SFTP listing found snapshot f3d9cd67 and
12 535 KB really there; snapshot_count and repo_size_bytes were simply ABSENT
and the zero value spoke for them. StatsKnown is now NAMED, for the same reason
OffsiteInventory.Empty is: zero is what an unread store and an empty one both
look like, and on the wire 'absent' and '0' are the same bytes. The fill bar
renders only when the fill is known — a 0%-wide bar is a picture of emptiness,
and a picture is a claim. A measured zero still says zero.
R-227 — WHICH LAYER ANSWERS: traefik, and this repo generates its config. But
traefik v3 serves no static files, so a branded proxy page needs a new always-up
container for every 502 on the box — out of proportion, and scoped in the report
rather than built. Shipped instead: the unlock posts via fetch and answers a
gateway failure in Hungarian without leaving the page. Progressive enhancement —
with no JS the plain POST is unchanged and still shows the proxy's error, which
the report says plainly rather than implying otherwise.
R-228 — the set-aside history was recorded in orphaned_renamed_to and read by
nobody: a census found zero references in any template or handler, while 12 535
KB sat at that path. It is surfaced as two facts and stops. It does NOT promise
the history can be reopened, because it cannot be by anyone today (R-199's
inventory is unbuilt) — and the set-aside CONFIRMATION copy was corrected for
the same reason: 'a helyreállítási kód nélkül többé nem lesznek megnyithatók'
implied that WITH the code they could be. The field's own comment called it
'recovery-code-recoverable', which was the same over-promise in the code.
Tests: scenarios F, G, H as render tests per branch of each gate. Red-proofs,
each demonstrated failing then restored: remove the StatsKnown guards (F,
'R-225 RETURNED: an unread store reports a snapshot COUNT of zero'), delete the
set-aside block (H). The F assertion on the fill bar is scoped to the bar's own
container — a bare width:0% search matched unrelated elements and would have
passed for the wrong reason.
28 packages ok, vet clean, all controller gates OK (the emoji gate caught a
warning sign in a template comment).
R-216: the offsite key recovery is a coupled feature and now says so. featureProbes +
featureMinAgent 0.125.0 + a Supports gate at the unlock entry point, FAILING CLOSED — an
agent that cannot answer is named as such instead of the customer's code being blamed.
Measured live: a 404 from agent 0.120.0 came back as "we did not accept your recovery
code, check that all ten words", in 0.134 s, against a perfect code.
R-218: delete the repo-password short-circuit in needsOffsiteCredential. The declaration
stops when the TIER WORKS, not when a key exists — installing a key is the recovery
screen's whole job, so succeeding at recovery was switching off the mechanism that would
have delivered the coordinates to use it.
R-219: the unlock finishes the job — place the key, bring the tier up, then list. Without
it the promised listing could never render on the shape the screen exists for.
R-217: an unreadable store no longer claims to have opened with unattributable content
(the OffsiteInventory{} zero value). Opened / empty / unreadable are three states.
R-222: a code that is right about a RETAINED earlier package is named, not blamed. States
what the hub knows and promises nothing — no read path exists.
R-215: GET /recovery is gated on the same predicate as the interception.
Five red-proofs, each demonstrated failing and restored.
A customer whose machine was rebuilt had everything needed to get their data
back and no way to find out: the only route was a command line. This is the
screen that closes that.
IT UNLOCKS, AND ONLY UNLOCKS (operator ruling). It explains, takes the recovery
code, opens the repository and shows what is in there — apps, dates, sizes. It
restores nothing: restore is already per-app and lives in the backups area, and
a screen that unlocks and then offers to overwrite is two decisions wearing one
button.
ONE CORE, TWO CALLERS. RecoverInstallCore is split out of RecoverAndInstall; the
CLI wrapper keeps its exit codes and printed lines byte-identical, and the
handler drives the same function. Two implementations of the one operation that
can permanently lose a customer's data would drift, and only one would be
tested. Asserted from source on both sides by AST.
THREE WAYS OUT, none a dismiss button: recover; 'most nem' (the full page stops
interrupting, the backups-area entry point stays PERMANENTLY, bound to the offer
and never to the postpone flag); and 'I do not want the old data' — confirmed
TWICE and reaching the SHIPPED move-aside, which sets aside and never deletes.
THE CODE IS HANDLED NO MORE LOOSELY THAN ON THE COMMAND LINE: POST body only,
never logged, never persisted, never echoed, cleared on every path, no-store,
autocomplete off. No lockout — the code is a ten-word phrase, and locking a
customer out of their own data for a typo is worse than anything it prevents.
TWO DEFECTS THE TESTS CAUGHT, both fixed: an UNCLAIMED (legacy-open) box would
have been shown the page, because RequireAuth passes such a box through; and the
inventory nil-dereferenced when no off-site target was configured, which is
exactly the pristine rebuilt shape.
An absent off-site object has four meanings — never configured, mid-restart, a
transient config read failure, and rebuilt-and-stranded — and the hub cannot tell
them apart. The box can, from two local facts it holds with certainty, so it says
so instead of leaving the hub to deduce it from a silence (operator ruling).
The ACK's identity_blob_present is now recorded on EVERY ACK, before the gates
that used to discard it: on a box with no off-site target the auto-confirm returns
immediately, which is exactly a rebuilt box, so the one fact distinguishing it from
a box that never had off-site backups was thrown away every cycle.
The declaration needs BOTH halves — a fresh data area AND a hub-held recovery
package. Freshness alone is a box that never had off-site backups; dropping that
condition makes the whole fleet ask for credentials, which is what the Scenario B
test exists to catch.
The object carries enabled:false and zero sizes, which is what makes it inert to
the hub's existing fill and staleness checkers and to a pre-upgrade hub. A
configured box's JSON is byte-identical to v0.198.0's.
The gap was already detected and warned about, in Hungarian, naming the app and the folders --
that warning is what stopped the R-201 drill. The defect was that the run still reported `ok`
beside it, and a warning standing beside a success is read as a success.
last_status gains "incomplete": minted, because "ok" | "error" | "running" had nothing meaning
"it ran, and this app is not fully protected". NOT "error" -- the rest of the run worked and
what was captured is real, so SnapshotCount and the LastSuccess anchor still record it. Half a
backup is not no backup.
The gaps are now recorded STRUCTURALLY (offboxRunResult.mandatoryGaps), not only as prose, so
the verdict has something to act on. It reaches the operator through the EXISTING per-run digest
(backup_run_failures) rather than a new event type -- a new type is a two-repo change and the
hub drops anything outside allowedEventTypes.
The stat-filter gains the ClassMandatory check Tier 2 already had. It is a NO-OP today
(TierOffsite admits mandatory only), so no customer-visible warning disappears -- demonstrated
by widening the tier filter alone and watching the check hold the line.
ANTICIPATED: calibre-web on demo-hp has exactly this gap, so its off-site status becomes
incomplete the moment this ships. That is correct and is the point.
Red-proofs: my first Scenario-C proof PASSED because the test only reached offboxCaptureSet
while the mutation lives in runOffboxInternal -- a mutation the test cannot observe is not a
red-proof, and the fix was the test. The run-level test now fails under both mutations
(unreachable gap recording; unconditional ok).
Tier2LastRun is the attempt clock and was rendered as 'Legutóbbi másolat' in the restore
confirm dialog. New LastSuccess + SuccessTracked anchor; tier2Update makes the three
rebuild sites safe by construction. F-DIAG: six distinct causes, target-aware redaction.
LastRun records an attempt, not a result. New OffboxTarget.LastSuccess, set only on the
success branch via the pure offboxAnchorAfterRun rule, carried to the hub as last_success.
Closes two silent-wipe sites (settings save, hub re-apply).
Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.
R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.
New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
- nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
created after the snapshot survives as an extra;
- the undo exists before the act — the pre-restore- dump is verified ON DISK
before anything is stopped, overwritten or replayed; if it cannot be taken
the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.
R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.
Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.
11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.
NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.
Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
Third real run, third thing only a live run could show. The per-app legs
finished in ~15 seconds; the remaining 40 of the 57-second run was the shares leg
and forget --prune, during which the card sat frozen on "calibre-web — 8 / 8
fájl". The same frozen-looking silence 4c exists to remove, relocated to the end.
Progress now carries a phase. The post-app stages announce themselves and the
app-scoped counters are cleared when a phase starts, so the last app's finished
numbers are never shown against work that is no longer about that app. Starting
the next app clears the phase again. Pinned by a test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The systemic complaint, twice in one evening: you press a button and nothing
happens. No progress, no ETA, no named result. Three worst offenders, fixed on
the two patterns already here (deploy 3-step panel, storage-init status poll).
No new framework — that is a ROADMAP item; three targeted cards ship tonight.
4a — a verification restore names its result. The flash said the app had been
restored "to a verification folder on the drive"; which folder, on which drive,
was invisible, so the customer could not go and look at what they had just asked
for. Full path now. The restore page gained a listing of existing verification
copies (app, size, date, path) — nothing anywhere showed these, so they piled up
and the only way to find them was SSH — each with a double-confirmed delete.
That delete is the only one this release adds, so it names a STACK, never a
path: the Manager resolves the name inside a backups/offsite-restore root it
computed itself and refuses anything landing outside. Red-proofed — neutralise
the name guard and stack:"" resolves to the offsite-restore ROOT and takes every
copy with it. Refusals are asserted as non-effects.
4b — Megosztás enable shows what it is waiting for. Enabling ran ReconcileSamba
synchronously inside the POST handler; on a golden without felhom-samba baked
that is compose pulling ~100MB, i.e. minutes of an apparently-hung form post
followed by "Beállítás mentve." whether or not anything came up. Detached +
polled now, distinguishing "képfájl letöltése" from "indítás" — decided BEFORE
the work starts, since afterwards the image is always present. Success is
probed, not inferred (compose up -d exits 0 on a crash-loop). The password form
starts the same job: with UserSet false reconcile deploys nothing, so on a fresh
box that is where the pull actually happens.
4c — "Távoli mentés most" streams real progress. restic was already reporting
bytes and percent; the runner seam used CombinedOutput() and discarded them. The
manual run now passes --json and scans stdout line-by-line: total bytes, percent,
current app. Manual only — the nightly stays silent, pinned by a test that fails
if it ever passes --json. The poll now arms unconditionally, closing a race the
manual trigger always ran: the redirect rendered before the goroutine wrote
LastStatus=running, so the poll never armed and the page sat static during the
very run just started. Red-proofed twice.
Also closes the golden/controller infra-image drift at the source: infra.Images()
derives from the existing pins and --print-infra-images exposes it, so the golden
bake can stop carrying its own copy. That copy had already drifted — felhom-samba
was never added, so the golden baked 3 of 4, which is why enabling Megosztás
pulled at runtime in the first place.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
ONE additional restic call tagged [felhom-offbox, _shares] carrying the payload
staging dir + every mandatory share folder. Hooked into runOffboxInternal AFTER the
per-app loop and BEFORE retention, so forget --group-by host,tags covers the _shares
group with no flag change. Reuses resticStep, the caller's repo-ensure and
single-flight, and the SAME enlargement-gate arithmetic.
- quota gate degrades the push to MANIFEST-ONLY, never to nothing
- EnlargedBlocked keeps the RAW _shares key (templates index by it); the display
mapping applies only at the notification + Hungarian-prose boundaries
- OffboxTarget gains SharesLastRun/Status/Count for per-tier page truth
- zero-toggle notice suppressed when the shares leg provided coverage
- reserved-name defense: an app keyed _shares is excluded from the run loudly
RED-PROOFS RUN AND REVERTED (all fired):
1. shares leg appends into the app's argv -> isolation test FAILS
2. mandatory->offsite mapping inverted -> Scenario A + B FAIL
3. manifest-only degradation dropped -> Scenario C FAILS
resticStep escalates a restic lock error to `unlock --remove-all` + one
retry (safe: single-writer repo — sub-account isolation + single-flight
mutex); plain `unlock` is stale-only and can't clear a crash lock across a
container-hostname change. Pre-run stale unlock hygiene on run+restore.
C1: NewManager flips a persisted LastStatus=running to a truthful error.
Both red-proofed (A reproduces the exact campaign backup failure).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The QuotaGB hash change triggered a live re-apply that demoted the
escrowed demo to pending and wiped its runtime status. ApplyOffsiteTarget
now carries over EscrowState (custody tracks the preserved repo password,
not the coords) + status fields; fresh guests still land pending.
Red-proofed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
QuotaGB rides the descriptor into OffboxTarget; RepoSizeBytes persisted
from restic stats. Pre-run gate: >=100% refuses NEW backups (Hungarian
notice + operator alert) but prune STILL runs (red-proofed) and restore is
never gated; >=80% warns. /backups usage bar (quota>0 only). The hub
report gains the non-secret offsite status object for the OffsiteChecker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
EscrowAutoConfirmer flips pending->escrowed ONLY when sha256(local repo
password) matches the ACK's restic_pw_sha256 (blob-presence alone never
confirms — red-proofed). Mismatch warns once per hash naming the ceremony;
never un-confirms; wipes the staged secret on flip. Pinned cross-repo hash
vector; manual confirm deprecated to a legacy-blob fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
On startup reconcile the hub-served offsite: descriptor into a key-only offbox
target. internal/offsiteapply.Bridge: verify-pin box host key vs host_fingerprint
(NO blind TOFU) → consume the one-time password (single-use, never logged) →
sshpass ssh-copy-id -s -f install + verify → configure offbox → EscrowState=pending
(fork-4 via Manager.ApplyOffsiteTarget) → persist a descriptor-hash marker LAST.
Idempotent + fail-safe. Seams faked in tests; both red-proofs run+reverted.
Dockerfile + sshpass. NOT yet live-applied (supervised end-to-end next runbook).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Pairs with agent v0.77.0. StageEscrowSecret pushes the repo password to the
agent (POST /escrow/stage-secret) at offsite-enable → EscrowState="pending".
Atomicity gate: RunOffboxBackup (scheduler + handler) refuses until
EscrowState="escrowed" (operator POST /backup/offbox/confirm-escrow after the
escrow ceremony) — no un-recoverable offsite ciphertext can exist. DR:
POST /backup/offbox/inject-password pre-places a recovered 64-hex password 0600
(honored by WriteOffboxSecrets' IsNotExist guard; refuses clobber without
force). DR recipe gains non-secret offsite_restic coords (DRResticCoord); SFTP
key regenerated at DR, not escrowed. New settings.OffboxTarget.EscrowState.
Tests + atomicity & inject companion red-proofs green; UI gates pass. NOT yet
live-validated (supervised ceremony).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
offbox located each toggled app's recovery unit via AppNamespaceRoot→GetAppDrivePath,
which reads the app's LIVE app.yaml HDD_PATH and silently falls back to systemDataPath
when the app isn't deployed → looked on the wrong drive, backed up nothing, reported
ok/0 (DIAG root cause). Now:
- discoverOffboxUnit/offboxCandidateNSRoots scan the durable storage registry
(schedulable non-decommissioned paths ∪ systemDataPath) for backups/primary/<app>,
independent of deploy state; newest-by-manifest-CreatedAt wins on drive churn.
- RunOffboxBackup: runOffboxInternal returns (backedUp, missing, err); 0-of-N toggled →
hard error + operator alert; partial → ok + new OffboxTarget.LastWarning (shown on
/backups, preserved across config edit).
- AppNamespaceRoot + primary WRITE paths unchanged.
- 6 non-hollow tests (A-E + edge) + both companion red-proofs run (reverted).
- NOT yet live-validated against the Storage Box (spike creds torn down).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Live validation surfaced 'option sftp.args is not known' on restic 0.14.0; switch to the
portable sftp.command SSH invocation (ConnectTimeout/StrictHostKeyChecking preserved).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
Encrypted restic repo over SFTP for the app-data tier (the off-site 3-2-1 leg). A dead
NAS fails fast via -oConnectTimeout (spike Q8), never hangs the runner; secrets are 0600
files (ride DR via PBS whole-CT); init-if-absent, retention forget --prune, restore,
single-flight, per-app toggle + UI. restic re-added to the image.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
admin