Six red-proofs with observed failure text, the bounds chosen for contention
(15m retry / 3h alarm) justified against the agent's own 120m PBS restore-test
ceiling and the 12m01s longest observed restore-test, and the live replay of
both findings on demo-hp with the hub DB as arbiter:
- failed restart alarmed 9s after grace expiry; banner shows (stopped)
- a deliberate user stop stayed silent through 9 dead-app scans
- 409 contention produced 0 operator emails (8 -> 8)
- a real transport failure produced 1 (8 -> 9)
Same box, same event type; the only difference is 409 vs a genuine error.
Records the valve-licence table, why a missing field means legacy rather than
unknown, and the near-miss: TieredBackend is runtime-asserted, so the signature
change built and vetted clean while the adapter silently stopped satisfying it.
Includes the cooldown keying finding (per-event-type only; fixed narrowly with a
tier suffix), the 180s grace window derivation, and the end-to-end event evidence
with its control: the new type 200s, a bogus type 400s, and notification_log shows
one operator row and zero customer rows.
The quiesce loop is silent by design when nothing is due, so 'no error lines' is
not evidence. The observable that is: every /backup/due makes the agent read that
tier's storage content, which lands in pveproxy/access.log. Both tiers polled
every 5 min, both HTTP 200, tick phase consistent with the deploy restart.
Includes the Phase 0 findings: the nil is agent-side (case a, a type boundary
that cannot represent 'unknown'), the four restarts were deliberate rather than
self-update, and the loop produced zero backup_failed events because quiesce
never imports notify (filed R-97).
The section read 'NOT deployed'. True when written (17:56 CEST), stale 26 min
later when I deployed (18:22 CEST) and did not update it. REPORT.md is defined
as 'overwrite with the most recent state', so an artifact this project validates
against was left asserting the opposite of reality.
Precision matters on the apparent contradiction: a 17:01 quiesce log 'before' a
17:56 commit is a TIMEZONE artifact (controller logs UTC, git commits CEST), not
evidence of a false claim. In one clock: REPORT 17:56 -> deploy 18:22 -> quiesce
19:01. Consistent. That host=CEST/controller=UTC mismatch is a recorded project
trap and caught me once earlier the same day.
Also records the live multi-tier quiesce evidence and what remains unproven.
The multi-tier path only engages against an agent serving /backup/tiers, and the
agent rollout was still mid-flight, so both production controllers remain on
v0.173.0. Names the three things that are therefore unproven live.
Feature-detected navigator.share opens the OS share sheet with title+text+URL only
(no QR files: — narrow support drops the URL when given file+URL). Hidden unless
supported; Link másolása stays the universal fallback and catches the non-cancel
rejection; AbortError is silent. Template JS + tests only. 2 red-proofs verified red.
Mint a 160-bit capability URL (/s/<token>) serving a standalone read-only guest
launcher: same tiles, opens apps in new tabs, no account, no admin session.
Information only, zero control — every privilege stays behind each app's own auth.
- /s/ pre-auth pass-through (after the claim gate) + session-CSRF exemption; guest
password POST carries its own pre-auth HMAC CSRF.
- Constant-time token match; empty stored token = disabled = byte-identical mux 404.
- Optional per-share password: separate bcrypt hash + own attempt map; signed cookie
= HMAC(token|passwordHash) keyed with web.session_secret, so rotate/change invalidates.
- Guest labels ride the v0.164.0 ruling; never expose internal state vocabulary.
- Token redacted in logs (/s/<redacted>); never in CHANGELOG/REPORT/CONTEXT.
- Admin modal: copy-link, QR (go-qrcode), set/clear password, rotate, disable.
- Tests: Groups A-G (14) + 3 red-proofs verified red.
demo-felhom 9201 + demo-hp 9201 (G1 break-glass, secret shredded) both clean +
healthy on 0.162.0 with the settle-gate GO line. Honest B' accounting recorded:
the floor is in-memory (not persisted) so both above-floor boxes logged
"awaiting floor knowledge" then GOed ~10s later once the report ACK landed —
the report-ACK latency the 90s sub-bound is sized to, not a regression. The
zero-wait-when-floor-known invariant stays unit-proven (test E).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N7Drmtm2RzoqbkJZCNSFNQ
The day-0 race (DIAG-f10): a fresh box boots below the operator floor, the
apply-bridge consumes the single-use offsite password, then ~35s later the
managed auto-floor update replaces the container mid-install -> the new process
finds no installed key -> consume -> 404 -> offsite dead until an operator
Re-issue. Recurs on every onboarding whose ISO floor lags the managed floor.
Ordering-only fix (consume/install/persist internals + the 404-no-oracle
contract + the Consumer UNTOUCHED; R-71(b) rejected-by-design):
- New seam offsiteapply.SettleProvider.SettleState() + SettleFunc adapter over
the self-updater's own GetFloor()/IsUpdateRunning() (no second floor path).
- Bridge.AwaitSettle polls 10s BEFORE the 3-min Reconcile ctx: defers while an
update runs or the box is below the known floor; GOes at/above floor on the
first poll with zero added latency (B'). Bounds 90s floor sub-bound / 5min
overall, both GO+WARN (hub that can't serve a floor can't serve a consume ->
no burn risk; R-71c is the belt). ReconcileWhenSettled = gate then reconcile.
- main.go: bridge goroutine moved after the updater is built; wired only when an
updater exists (nil Settle = reconcile immediately, old behavior).
Finding: the floor is in-memory (report-ACK ~5-10s), NOT persisted -> unknown on
any restart until the first ACK; the 90s sub-bound is sized to that.
Tests (injectable clock, fake SettleState, recorded Consumer): A-E + nil-provider
+ cancelled-gate. Four red-proofs all observed FAIL then restored: gate removed /
updateRunning branch / floor sub-bound / overall bound. Deferral paths ship
unit-proven + red-proofed, NOT live-fired -- their precondition is now
structurally prevented by the v1.25.0 build gate. Layering: gate prevents, (a)
defers, (c) heals.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N7Drmtm2RzoqbkJZCNSFNQ
C6: operator deleted the photos, EMPTIED THE TRASH, then restored through
the customer UI. 40 files placed (vs 6 in the non-destructive run) — the
files were really gone and really came back. 11 assets active, no schema
drift, timeline confirmed. Full UI path, no endpoint shortcuts.
R-23(a): the STOP-2 floor save turned out to be exactly the trigger. Hub set
the floor at 18:56:27 CEST; the controller logged 'wait woke: generation=1'
at 16:56:27 UTC — the same second — with the out-of-cycle report 2s later.
The wake is logged at DEBUG only, so it is invisible in docker logs and
lives only in the debug ring.
R-23(a) is NOT complete: the floor was set to a version the box already ran,
so nothing restarted and the restart-single-fire assertion is unexercised.
STOP-2 applied by the operator in the right order: Day-0 Golden -> 0.153.0
(sha256 matches the bake), Agent/MinAgent 0.90.0 unchanged, global floor ->
v0.153.0 saved last. Build guest 9100 destroyed.
Agent 0.90.1's absence from the hub dropdown is CORRECT, not a defect: it
was never published (Gitea 404; 0.90.0 returns 206) and felhom-pve runs
0.90.0. The v0.90.1 source is committed at 9596d5a, so R-39 is a
build+publish away.
CORRECTION: my earlier claim that the stored controller password was stale
was WRONG. Values in ~/.config/credentials are single-quoted and my cut
extraction kept the quotes, sending a 15-char string instead of the 13-char
password. Re-tested stripped: 302 + session. The credential was always
correct.
Also records honestly that the reconstitute restored over an already-good
state, so C6 still needs the destructive empty-the-trash drill.
build-golden.sh v2.1.0 on felhom-pve from the vacation site. P3 (registry
reachability) passed before any mutation; the one deviation from the 0.146.0
recipe was an empty template cache, handled with pveam download (approved).
First golden carrying all FOUR infra images — the list came from
--print-infra-images on the 0.153.0 binary itself, so the historical
3-image fallback never fired and felhom-samba:1.1.0 is baked.
Upload HTTP 201; anonymous GET 200 with an exact sha256 byte-match
(15fdd191f3c660a6...); ranged request 206; the hub pod can fetch it.
GL-1 teardown: token via 0600 env file (never argv), shredded; leak grep 0;
build guest 9100 left stopped.
Bake log retained at 180:/mnt/5_hdd/felhom.eu/drill/bake-0.153.0.log.
Endpoint-level supervised run against snapshot 49e7cb46, the same one that
aborted in round 2. Controller log shows stop -> 'Starting stack immich
services only: [immich-postgres]' -> replay rc-0 in 20s -> full start.
No 'already exists'. Operation reported success (round 2: failure). immich's
own DatabaseService logged 'No schema drift detected' twice, where round 2
left it reporting drift. 11 assets active, 4/4 containers healthy, 231
public indexes.
Credentials were supplied file-to-file, never echoed, and shredded with both
cookie jars at the end of the run.
Closes R-47. No new agent coupling — MinAgent stays 0.90.0.
The replay needs a running DB container, so both restore paths started the
WHOLE stack first, giving the application a window to rebuild the very schema
objects the dump was about to create. Measured live on 2026-07-19 (H4,
DIAG-immich-restore-round2): immich-server rebuilt clip_index two seconds
before the dump's CREATE INDEX, the replay aborted "already exists" under
ON_ERROR_STOP=1, and immich reported schema drift. The data survived only
because pg_dump emits COPY before CREATE INDEX.
Both paths now open a DB-ONLY window: only the stack's database service(s)
come up, the dump is replayed with the app still down, and the full start
runs only after the replay exits 0. Fail-closed: a dump with no identifiable
DB service refuses BEFORE the first mutation. Every exit from the window
still does a best-effort full start, so a failed restore never leaves a box
with a database and no application.
New: appbackup.DBServiceNames (yaml.v3 services-map parse — never a line
scan; immich's top-level volume keys are the decoy) sharing dbTypeForImage
with DiscoverDatabases; stacks.Manager.StartStackServices (refuses an empty
list — argument-less `up -d` is a full start); RedeployFromEnv split into
PersistUnitRedeployConfig + its unchanged tail. StackDataProvider's
RecreateStackFromUnit becomes RecreateStackDefinitionFromUnit — the hidden
`up -d` inside the old name is what carried the defect on the local path.
19 new tests (ordering plus state-at-replay-time, zero-mutation fail-closed
effects, replay-failure bring-up, parser decoys, empty-list refusal); three
companion red-proofs run and reverted. 23/23 packages green.
Not yet live-validated: STOP-1 supervised reconstitute, golden 0.153.0.
The fab-upload GC test stat-ed the .part immediately after observing the slot
free, but expireIdleUpload unlinks AFTER releasing the mutex. Passed alone,
failed in the full package once this release's render tests made web heavier.
Not a production defect - a new upload mints a fresh random .part. The test now
waits for the outcome it asserts on the same deadline; red-proofed by removing
the unlink from production.
admin
Claude Code