demo-felhom 9201 + demo-hp 9201 (G1 break-glass, secret shredded) both clean +
healthy on 0.162.0 with the settle-gate GO line. Honest B' accounting recorded:
the floor is in-memory (not persisted) so both above-floor boxes logged
"awaiting floor knowledge" then GOed ~10s later once the report ACK landed —
the report-ACK latency the 90s sub-bound is sized to, not a regression. The
zero-wait-when-floor-known invariant stays unit-proven (test E).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N7Drmtm2RzoqbkJZCNSFNQ
The day-0 race (DIAG-f10): a fresh box boots below the operator floor, the
apply-bridge consumes the single-use offsite password, then ~35s later the
managed auto-floor update replaces the container mid-install -> the new process
finds no installed key -> consume -> 404 -> offsite dead until an operator
Re-issue. Recurs on every onboarding whose ISO floor lags the managed floor.
Ordering-only fix (consume/install/persist internals + the 404-no-oracle
contract + the Consumer UNTOUCHED; R-71(b) rejected-by-design):
- New seam offsiteapply.SettleProvider.SettleState() + SettleFunc adapter over
the self-updater's own GetFloor()/IsUpdateRunning() (no second floor path).
- Bridge.AwaitSettle polls 10s BEFORE the 3-min Reconcile ctx: defers while an
update runs or the box is below the known floor; GOes at/above floor on the
first poll with zero added latency (B'). Bounds 90s floor sub-bound / 5min
overall, both GO+WARN (hub that can't serve a floor can't serve a consume ->
no burn risk; R-71c is the belt). ReconcileWhenSettled = gate then reconcile.
- main.go: bridge goroutine moved after the updater is built; wired only when an
updater exists (nil Settle = reconcile immediately, old behavior).
Finding: the floor is in-memory (report-ACK ~5-10s), NOT persisted -> unknown on
any restart until the first ACK; the 90s sub-bound is sized to that.
Tests (injectable clock, fake SettleState, recorded Consumer): A-E + nil-provider
+ cancelled-gate. Four red-proofs all observed FAIL then restored: gate removed /
updateRunning branch / floor sub-bound / overall bound. Deferral paths ship
unit-proven + red-proofed, NOT live-fired -- their precondition is now
structurally prevented by the v1.25.0 build gate. Layering: gate prevents, (a)
defers, (c) heals.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N7Drmtm2RzoqbkJZCNSFNQ
C6: operator deleted the photos, EMPTIED THE TRASH, then restored through
the customer UI. 40 files placed (vs 6 in the non-destructive run) — the
files were really gone and really came back. 11 assets active, no schema
drift, timeline confirmed. Full UI path, no endpoint shortcuts.
R-23(a): the STOP-2 floor save turned out to be exactly the trigger. Hub set
the floor at 18:56:27 CEST; the controller logged 'wait woke: generation=1'
at 16:56:27 UTC — the same second — with the out-of-cycle report 2s later.
The wake is logged at DEBUG only, so it is invisible in docker logs and
lives only in the debug ring.
R-23(a) is NOT complete: the floor was set to a version the box already ran,
so nothing restarted and the restart-single-fire assertion is unexercised.
STOP-2 applied by the operator in the right order: Day-0 Golden -> 0.153.0
(sha256 matches the bake), Agent/MinAgent 0.90.0 unchanged, global floor ->
v0.153.0 saved last. Build guest 9100 destroyed.
Agent 0.90.1's absence from the hub dropdown is CORRECT, not a defect: it
was never published (Gitea 404; 0.90.0 returns 206) and felhom-pve runs
0.90.0. The v0.90.1 source is committed at 9596d5a, so R-39 is a
build+publish away.
CORRECTION: my earlier claim that the stored controller password was stale
was WRONG. Values in ~/.config/credentials are single-quoted and my cut
extraction kept the quotes, sending a 15-char string instead of the 13-char
password. Re-tested stripped: 302 + session. The credential was always
correct.
Also records honestly that the reconstitute restored over an already-good
state, so C6 still needs the destructive empty-the-trash drill.
build-golden.sh v2.1.0 on felhom-pve from the vacation site. P3 (registry
reachability) passed before any mutation; the one deviation from the 0.146.0
recipe was an empty template cache, handled with pveam download (approved).
First golden carrying all FOUR infra images — the list came from
--print-infra-images on the 0.153.0 binary itself, so the historical
3-image fallback never fired and felhom-samba:1.1.0 is baked.
Upload HTTP 201; anonymous GET 200 with an exact sha256 byte-match
(15fdd191f3c660a6...); ranged request 206; the hub pod can fetch it.
GL-1 teardown: token via 0600 env file (never argv), shredded; leak grep 0;
build guest 9100 left stopped.
Bake log retained at 180:/mnt/5_hdd/felhom.eu/drill/bake-0.153.0.log.
Endpoint-level supervised run against snapshot 49e7cb46, the same one that
aborted in round 2. Controller log shows stop -> 'Starting stack immich
services only: [immich-postgres]' -> replay rc-0 in 20s -> full start.
No 'already exists'. Operation reported success (round 2: failure). immich's
own DatabaseService logged 'No schema drift detected' twice, where round 2
left it reporting drift. 11 assets active, 4/4 containers healthy, 231
public indexes.
Credentials were supplied file-to-file, never echoed, and shredded with both
cookie jars at the end of the run.
Closes R-47. No new agent coupling — MinAgent stays 0.90.0.
The replay needs a running DB container, so both restore paths started the
WHOLE stack first, giving the application a window to rebuild the very schema
objects the dump was about to create. Measured live on 2026-07-19 (H4,
DIAG-immich-restore-round2): immich-server rebuilt clip_index two seconds
before the dump's CREATE INDEX, the replay aborted "already exists" under
ON_ERROR_STOP=1, and immich reported schema drift. The data survived only
because pg_dump emits COPY before CREATE INDEX.
Both paths now open a DB-ONLY window: only the stack's database service(s)
come up, the dump is replayed with the app still down, and the full start
runs only after the replay exits 0. Fail-closed: a dump with no identifiable
DB service refuses BEFORE the first mutation. Every exit from the window
still does a best-effort full start, so a failed restore never leaves a box
with a database and no application.
New: appbackup.DBServiceNames (yaml.v3 services-map parse — never a line
scan; immich's top-level volume keys are the decoy) sharing dbTypeForImage
with DiscoverDatabases; stacks.Manager.StartStackServices (refuses an empty
list — argument-less `up -d` is a full start); RedeployFromEnv split into
PersistUnitRedeployConfig + its unchanged tail. StackDataProvider's
RecreateStackFromUnit becomes RecreateStackDefinitionFromUnit — the hidden
`up -d` inside the old name is what carried the defect on the local path.
19 new tests (ordering plus state-at-replay-time, zero-mutation fail-closed
effects, replay-failure bring-up, parser decoys, empty-list refusal); three
companion red-proofs run and reverted. 23/23 packages green.
Not yet live-validated: STOP-1 supervised reconstitute, golden 0.153.0.
The fab-upload GC test stat-ed the .part immediately after observing the slot
free, but expireIdleUpload unlinks AFTER releasing the mutex. Passed alone,
failed in the full package once this release's render tests made web heavier.
Not a production defect - a new upload mints a fresh random .part. The test now
waits for the outcome it asserts on the same deadline; red-proofed by removing
the unlink from production.
Closes the two findings from DIAG-immich-restore-2026-07-19. Viktor deleted 11
immich photos to test offsite restore; both runs flashed success and the photos
stayed gone. Two independent defects.
R-43 — no offsite path could restore a database. All three buttons were
file-only: the two "visszaállítás" actions staged to a scratch folder and never
touched postgres, and place-to-live merged only MISSING files. For a DB-indexed
app the bytes returned and the app still could not see them. The dump was
carried INTO every snapshot and could never be replayed OUT of one.
New ReconstituteFromOffsite (/backup/offbox/reconstitute): safety dump → stop →
files overwritten to the snapshot version → start → the snapshot's own dump
replayed → health wait. Two invariants:
- nothing is ever deleted (-a, no --ignore-existing, no --delete): a file
created after the snapshot survives as an extra;
- the undo exists before the act — the pre-restore- dump is verified ON DISK
before anything is stopped, overwritten or replayed; if it cannot be taken
the operation refuses with zero changes.
The replay reads the SCRATCH unit: the live unit is never overwritten, so
replaying from it would replay the current DB over itself and restore nothing.
R-44 — a manual push shipped an unrefreshed dump (up to ~24h old). That day's
predated the customer's account by four hours and probed to asset:0/user:0/
album:0 inside 52MB whose bulk was immich's shipped geodata. Every run, manual
AND nightly, now refreshes dumps + units BEFORE capturing. Order is the
mechanism: the gap can only ADD files the DB does not reference yet, never
remove one it does. Manifests carry offsite_run_id + dumps_at, so coherence is
verifiable at restore time rather than assumed; the periodic refresh carries a
prior stamp forward and never invents one.
Honesty surfaces, all warn-level and none a gate: unstamped (pre-v0.148) pairs
report their skew, ValidateDump gained an EXACT-match accounts-table sniff for
customer-empty dumps, the completion flash states an outcome instead of a
mechanism, and the missing-only button now says what it does NOT do.
11 tests; 5 red-proofs run and reverted. Two of those found real test weaknesses
rather than confirming strength — the first undo mutation was caught by a second
guard, and the first table-matching test did not discriminate between the two
matchers at all. Both tests were rewritten to the cases that separate them.
NOT in scope: R-41's catalog invariant check, nightly cadence, retention, quota
math, tier-2, and v0.147.x progress semantics beyond one added phase line.
Live acceptance (§9) has NOT run: no capability-map flip, customer-restore row
stays MISSING, R-3 stays DRAFT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9Nn14TWGzKoqAJAiVwC2s
Overwrites REPORT.md per convention. Records the two changes, the four design
decisions behind the accordion (real <button>, landing-pages-survive check,
server-side open state, grid-rows instead of max-height), and — importantly —
what could NOT be verified and why: the demo controller's password is
customer-owned since the claim flow, so the build-server credentials are stale
and a curl-login returns the Bejelentkezés page. The four red-proofed render
tests stand in for the server-side half; the visual leg needs Viktor's browser.
Also notes the fleet gap: v0.146.0 is live on the demo box but the golden still
bakes 0.143.0, deferred to the next session.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
The gate landed v0.125.0 (2026-07-13), realVolumeSize landed v0.129.0 (2026-07-14)
-> red for 16 releases unnoticed. Records that the flagged mount is a NAMED VOLUME
(the safe shape) so the fix is an allowlist entry, not a rewrite of correct code.
The prior revision claimed the demo box had no offsite target and that both legs
were unexercised. All three clauses were false on a box that had run a successful
offsite backup minutes earlier.
Evidence: offsite snapshots e0b9d723 (Viktor 12:18:16Z) and 4e2b15ec, tagged
felhom-offbox,_shares, each carrying the payload dir (manifest + passdb.tar, 0600)
plus both share folders. Restore round-trip through the real routes returned a
deleted probe byte-identical and a deleted DEFINITION with its original flags,
without overwriting the two pre-existing files. R-7b is fully PROVEN-LIVE.
Root cause in new REPORT section 7b: the check read a GUESSED settings key
('offbox_target') where the real tag is 'offbox' (settings.go:96), got None for
every field, and read that as 'not configured' instead of 'wrong key'.
Findings, all six red-proof outcomes, deployment, live-validation evidence, and an
explicit list of what was NOT live-exercised (offsite leg + restore round-trip: the
demo box has no offsite target).
Viktor confirmed on the demo box: Network -> FELHOM -> both shares open; an
INTERACTIVE Explorer save into dokumentumok succeeded and landed owned 1000:1000
(force user holds for a real Explorer write, not just a scripted one); a write
into the read-only filmek was refused by Windows with the folder left empty on
disk. Capability-map row flipped to PROVEN-LIVE in felhom.eu.
Records the R-7 slice 1 ship, the four red-proof outcomes, live-validation
evidence, the two bugs live validation caught, and the Part-4 Step-1 enumeration
finding + the reported design fork (share data classified but not yet in a live
backup run — needs a Viktor ruling, suggested R-7b).
Out-of-cycle pushes at 17:42:19/17:42:34Z (quiet-window + min-interval
pacing exactly as designed, trailing edge carried the final state);
scheduled 15-min cycle unaffected (17:49:00Z tick on both ends).
Red-proof outcomes and the NOT-yet-live-validated list recorded.
After a completed escrow ceremony the Távoli mentés page showed the yellow
"Helyreállítási kód szükséges" card for ~15 min until the next hub-report ACK
flipped pending→escrowed. Phase-0 diagnosis (read-only) = verdict A (report-cycle
lag), already resolved on the demo box (escrow_state:"escrowed"); hub Hypothesis B
verified false (SaveHostEscrow ON CONFLICT already clears stale_at on upload) → no
hub change.
- settings.OffboxTarget.CeremonyCompletedAt: stamped on the recovery-code claim,
zeroed on the auto-confirmer Flip + the deprecated manual confirm; persisted.
- web/handlers.go: offboxCeremonyWaitState + escrowCeremonyGraceWindow (35m).
- backups_remote.html: info "megerősítésre vár, legfeljebb 15 perc" card → warn
"a megerősítés nem érkezett meg" past the window. Existing branches untouched.
- backups_escrow.html: "Mi történik ezután?" note on the wizard's final step.
- Test web/escrow_wait_state_test.go (truth table + red-proof recorded in REPORT).
No scheduler/agent/hub/endpoint changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017qDiBqKKQ5vPB5fXBqu7Kp
SQ6 over-capture FIXED for classified apps: the userdata root tar is exclude-scoped (keeps only
ancestors/descendants of a SELECTED bind relpath — R1-C, the tier2Reconcile keep-rule); no selected
userdata bind → no root tar (radarr state-only). New appbackup.ComputeFabBuckets (shared
resolveGuardCollapse pipeline; class buckets; guards over ALL classes; no cross-bucket containment).
appexport/fabplan.go: computeFabPlan + tarDirectoryExcluding + fabEstimateSplit. ExportRequest gains
DeselectOptional/OptInExcluded (both start handlers — two-call-site); mandatory is a server-side
floor. Manifest v1 + import UNTOUCHED; legacy apps byte-identical to v0.130.0. Estimate additive
class split; export UI: locked-mandatory/optional-checkboxes/excluded-opt-in + two-number warning.
All 6 §10 red-proofs verified. 6D Accept #1 now runs against this shape.
tier2_capture.go: classified apps get TierSecondary per-bind legs (paperless copy shrinks — export
drops); legacy apps keep the byte-identical resolver set. v2 relpath-mirroring layout
(backups/secondary/<stack>/{marker LAST, recovery-unit/, hdd/<rel>/, userdata/<rel>/}); N>1 native
(errTier2MultiDir/tier2AppDataName deleted). Migration=delete-and-rebuild + reconcile; all RemoveAll
via tier2SafeRemove (refuses outside backups/secondary/). SSD=state-only tier. selectTier2Target
never picks network storage (pinned+auto, F-6C-1). Restore reads v2 behind a marker gate.
Part 0: offbox_enlarge_blocked is a persisted one-time Load seed (opt-out sticks), not a getter
append. Part 0.5: offsite restore scratch prefers a local (non-network) path.
Full v2 test suite + all 10 §10 red-proofs verified. Destructive writes bounded to backups/secondary/.
PlaceOffsiteRestore: live target via raw GetStackHDDPath not AppNamespaceRoot (F-3a-1a: no SSD
merge; undeployed refused), placement headroom gate (F-3a-1b), stat pre-pass over all placements
before any copy (F-3a-4: no partial writes), scratch removed on success/kept on failure (F-3a-2).
mapOffsiteRestorePaths refuses the namespace root itself (F-3a-3).
Delivery chain: DefaultEnabledEvents + GetNotificationPrefs append-if-absent migration + settings
checkbox + handler slice; paired with hub v0.55.0 allowlist (no customerMessages entry — raw
dynamic message survives). +8 tests; all 6 controller §10 red-proofs verified.
admin