MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed
events reached the hub and TWO operator emails went out. The hub's operator
cooldown key is customerID:eventType(+tier) and that event carries `app` but no
`tier`, so the key held no app identifier: the first refused app took the hour's
slot and every other app's failure was discarded BEFORE anything was written
down, leaving no row on any channel.
The obvious fix — put `app` in the key — was ruled against: on a full disk it
produces one email per app, the volume problem wearing the correctness problem's
clothes.
internal/backup/runsummary.go: a per-run collector with exactly admissionSet's
lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the
end and only when something failed. A clean run emits nothing.
The per-app event stays and becomes the RECORD — the hub routes it record-only,
stored and logged every time, never competing for an email slot. The record and
the notification are now different things.
Deliberate skips (disconnected, decommissioned) are excluded: they have their
own alert, and a nightly email about an unplugged drive is one the operator
learns to ignore.
A manual run always reports: the digest carries a unique run_id the cooldown
cannot collapse. Someone pressing the button is actively trying to get a backup.
THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a
capture failure found between runs would be recorded and never notified — a new
silence introduced while closing one. That path emits a digest with NO run_id,
so the ordinary 1-hour cooldown caps it exactly as before while the mail now
lists every failing app instead of whichever was first.
A refusal is recorded ONCE, where the verdict is taken, not at the three legs
that consult it — R-181's contract is one verdict per app per run. Noting it per
leg listed one refused app three times and produced "2 of 1 apps failed". Found
by the digest's own test, not in review.
Silence is safe because the hub's deadline check raises expected_backup_missed
from report freshness, independently of any mail this box sends
(monitor/deadline.go:396,417). Confirmed, not assumed.
7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first
attempt — the AST test walked the backup package and not main.go; the test was
fixed and the mutation re-run rather than the pass recorded.
B2's capture floor (v0.192.0) was consulted in exactly ONE place —
captureAllRecoveryUnits, which writes a few KB. The two legs that write the BULK
into the same backups/primary/<app> tree, the DB dump and the volume dump, ran
FIRST and unguarded. Measured live on demo-hp 2026-08-03 06:40:03: opengist's
volume dump wrote 2.0 GB with no check, free fell to 1.0 GB, and the floor then
refused the cheap write it had already lost the argument to. Its refusal message
claimed "the previous unit is untouched" — measured false: that app's tar had
gone 182,272 B -> 2,147,666,432 B under a stale manifest. Sixth entry in
CLAUDE.md's table of shipped guarantees the code did not provide.
Fix: ONE admission verdict per app per run (internal/backup/admission.go), taken
before that app's FIRST write and covering all three legs — they write under one
per-app root, which is why one verdict can honestly cover them.
- Lazy, at the app's first write, NOT once at run start: app A's dump can put app
B under the reserve, so a run-start verdict reads a disk that no longer exists.
- Remembered for the run, never re-decided between an app's own legs — that is
the split this closes. Reset per run.
- Placed ahead of DumpAppVolumesSafe, which stops the stack as its first act, so
a refused app is never bounced. After the volume-less check, which has no write.
- Exactly one operator alert per refused app per run.
- Leg order unchanged: volume dumps still precede the capture.
The floor is now SIZE-AWARE: it asks whether THIS app's write would cross the
reserve, not only whether the filesystem is already below it — which is how an
app was admitted at 96% and then allowed to write 2 GB. Estimate = the app's
previous .sql + .tar on disk. No history -> headroom-only, deliberately, and the
alert says so.
A container-based du per volume was MEASURED and rejected: 66 timed runs on
demo-hp guest 9201, median ~355 ms/volume (341-404) on volumes holding tens of
KB — container start-up, not the walk. Decisive on top: docker run needs the
writable layer, so it can fail under exactly the pressure the reserve handles.
The message was NOT weakened; the behaviour was moved so the wording became true.
It now also names which term bound. Every claim is checked against a sha256
fingerprint of the tree it describes, never against the log line.
Still refuses and never deletes: nothing here is generational.
11 new tests through the production functions. The DB leg cannot run without
Docker, so its gate is pinned by an AST walk of backup.go asserting admitApp
precedes DumpOne (strings.Contains is insufficient — a commented-out call still
contains the string). 4 red-proofs demonstrated failing then restored.
Ships BEFORE the disk-layout merge it exists for, and is harmless on a box
that never gets it. The mp1 partition was a BULKHEAD as well as a ceiling:
it kept a runaway capture from filling the space the container runtime
needs, because /var/lib/docker was a different filesystem. After the merge
it is the same one, and a full Docker data-root is a stopped box.
The floor sits in captureAllRecoveryUnits, checked BEFORE anything is
written: below the reserve, that ONE app's capture is refused, its previous
unit is left byte-identical, the R-158 alert fires with the space figures,
and the loop continues.
Two terms whichever binds first (97% used / 1 GiB free) in fillwatch's
shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer
is always warned before a refusal can happen — a floor that fires before
its own warning is a silent failure wearing a threshold.
Headroom, never unit size: a per-unit cap would be R-163 rebuilt inside one
volume. Refuses, never deletes: nothing here is generational, so pruning
could only destroy a different app's only local copy; pruneStalePrimaryDirs
is an orphan sweep, not retention, and must not be repurposed.
Tests 1184 -> 1191. One fixture strengthened mid-red-proof: the "old 20 G
ceiling is gone" test sat at exactly 20 GB and survived a literal
UsedGB > 20 cap — hollow. Now 120 GB, and the mutation fails it.
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills.
It emits the PRE-EXISTING disk_warning/disk_critical pair, which was
allowlisted, copy'd, default-enabled and checkbox'd with no producer in any
repo — the sixth "built but never wired" instance here. Two threshold terms
(85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at
both ends of this fleet's size range. Edge-triggered on escalation only,
state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil
usage read is never a warning and never clears one. Per filesystem, never
per app. Daily 03:30, before the nightly app-data legs.
R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture
fails, loop continuing, carrying the target filesystem's used/free bytes.
Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed,
which is customer-enabled and would email the customer about a failure they
cannot act on. D-c overrides R-158's own proposal here.
R-174: the app-stop guard no longer starts apps onto MISSING drives — a
regression in v0.189.0 code, found by review and closed the same session.
SetStarter got the raw stack manager, whose StartStack has no drive gate,
and Recover runs at startup. R-171 one path over. bootDriveGate could not be
reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read
vars assigned after Recover runs), so holder #3 is extracted into a shared
driveStartGate with a test pinning the delegation. ErrStartRefused splits a
refusal from a failure: both keep the marker, only Failed alarms, because
routing a deliberate hold into NotifyBackupFailed is the same false alarm.
Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
The recovery unit on the customer's drive now carries the PORTABLE secret
class, so Tier-1/Tier-2 restore no longer depends on the whole-guest tier.
A customer needs the drive and nothing else.
Part 0's rulings overturned the brief's recommendation, on evidence:
- the data_key flag is untrustworthy (4+ encryption keys the catalog itself
labels as such are unflagged) -> R-127
- a DB password is not resettable in practice: POSTGRES_PASSWORD is ignored
once PGDATA is non-empty, so a regenerated value leaves the app unable to
authenticate against its own restored rows while the dump replay still
reports success (proven on a throwaway postgres:16-alpine)
Ruling (operator): type:secret travels, type:password never does, minus the
nonPortableSecrets code register. Plaintext -- withholding the internet-
reachable class is what licenses that, and the two are coupled.
Precedence: the UNIT WINS over the guest -- the unit's secrets were captured
in the same run as the dumps beside them, so they match the data being
restored. The fail-closed data-key gate is unchanged.
Secret values are never logged; the manifest records NAMES only.
gofmt -w across the controller tree (46 files) so gofmt -l is empty — disarms the
formatting landmine where a targeted edit + accidental gofmt -w swept ~46 unrelated
files. Pure formatting: whitespace + gofmt's optional-semicolon removal in reflowed
inline closures. One doc comment reworded ('' -> 'the empty string') to avoid gofmt's
Go-1.19 doc-comment typographic substitition ('' -> curly quote) muddying its meaning.
No build/vet/test behavior change.
Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.
R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.
New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
- nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
created after the snapshot survives as an extra;
- the undo exists before the act — the pre-restore- dump is verified ON DISK
before anything is stopped, overwritten or replayed; if it cannot be taken
the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.
R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.
Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.
11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.
NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.
Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
Tier 2 rsync-mirrors each HDD app's recovery unit + appdata to a DIFFERENT physical
disk (the only off-drive protection bind-mounted userdata can get; PBS can't reach it).
Auto-enabled, auto-target: prefer another registered drive (different physical disk via
system.SamePhysicalDevice), else the internal SSD for SMALL units only — with a
size-aware headroom guard that REFUSES rather than fill the ~8G guest rootfs, recording
an honest "needs 2nd HDD" status. Status persisted via the surviving CrossDriveBackup;
"2. mentés" UI card now populated. Daily tier2-backup job + POST /api/backup/tier2.
- backup/tier2.go (engine+selection+headroom), tier2_test.go (headroom arithmetic)
- system.SamePhysicalDevice (linux Stat_t.Dev + stub)
- handlers.go Tier2 UI population + tier2DestLabel; backups.html honest no-target reason
- fixed stale TestBackupCopiesOnPath (old felhom-data layout -> in-guest layout)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CaptureRecoveryUnit now builds content in memory and skips writes when the unit
is already current (checksum + dump-set + version), so it can run from RefreshCache
(startup + every 5m) without thrashing the USB drive. Units now exist shortly after
startup and track config changes without waiting for the daily DB dump. +idempotency test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin