docs: R-47 live-validated + golden 0.153.0 published

ROADMAP R-47 and the capability map's offsite-restore row now carry the live
evidence: a supervised reconstitute of immich from snapshot 49e7cb46 — the
same snapshot that aborted in round 2 — ran stop -> DB-service-only start ->
replay rc-0 -> full start, with no 'already exists', a SUCCESS outcome, and
immich's own 'No schema drift detected' twice where round 2 left it
reporting drift.

The offsite-restore row's residual for a PROVEN-LIVE flip is now only the
immich timeline screenshot (no browser on DooPlex).

Golden 0.153.0 baked and published the same day, sha256 15fdd191f3c660a6...,
first golden carrying all four infra images.
This commit is contained in:
2026-07-20 18:04:02 +02:00
parent db3378bbf7
commit 9c8505649a
2 changed files with 2 additions and 2 deletions
@@ -58,7 +58,7 @@
| Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 |
| Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | |
| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` |
| Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PARTIAL***scope corrected 2026-07-19* | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | **2026-07-19:** `audits/DIAG-immich-restore-2026-07-19.md` finds **no offsite path loads a DB dump** — all three buttons are file-only (R-43). The mechanics in this row's title are each proven; the phrase "**immich end-to-end from offsite alone**" is what is contested, since a DB-indexed app cannot be reconstituted by any offsite action. **RULED 2026-07-19 (Viktor): 6D's destruction hit the FILE TREE ONLY — the database survived in its named volume** (`immich_postgres_data` is a named volume in both the v2 and v3 template eras), so "immich end-to-end from offsite alone" **overclaimed scope**: the file half was proven, the DB half was never destroyed and therefore never restored. Row downgraded PROVEN-LIVE → **PARTIAL**, scope-corrected. Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (no offsite path could replay a DB at all) + the P-FAB destructive re-import (the proven-replay evidence, on the LOCAL path). **2026-07-19, controller v0.148.0:** the DB half now exists in code (R-43 + R-44) and its replay reached a live box — but **round 2 found it aborts against a running app** (`audits/DIAG-immich-restore-round2-2026-07-19.md`, H4: the replay races immich's own schema repair; `clip_index` recreated by the app 2 s before the dump's CREATE INDEX). **2026-07-20, controller v0.153.0: H4 IS CLOSED (R-47)** — both restore paths now replay into a DB-ONLY window (`StartStackServices` brings up the database service alone; the app starts only after the replay exits 0), with a fail-closed refusal when a dump has no identifiable DB service. *(The earlier note here said "closes in v0.149" — that was wrong: v0.149.0 was the F3 dashboard `BackupStatus` fix. R-47 shipped in v0.153.0.)* Row stays **PARTIAL** until one clean acceptance run exists — the code window is closed, the evidence is not yet taken |
| Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PARTIAL***scope corrected 2026-07-19* | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | **2026-07-19:** `audits/DIAG-immich-restore-2026-07-19.md` finds **no offsite path loads a DB dump** — all three buttons are file-only (R-43). The mechanics in this row's title are each proven; the phrase "**immich end-to-end from offsite alone**" is what is contested, since a DB-indexed app cannot be reconstituted by any offsite action. **RULED 2026-07-19 (Viktor): 6D's destruction hit the FILE TREE ONLY — the database survived in its named volume** (`immich_postgres_data` is a named volume in both the v2 and v3 template eras), so "immich end-to-end from offsite alone" **overclaimed scope**: the file half was proven, the DB half was never destroyed and therefore never restored. Row downgraded PROVEN-LIVE → **PARTIAL**, scope-corrected. Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (no offsite path could replay a DB at all) + the P-FAB destructive re-import (the proven-replay evidence, on the LOCAL path). **2026-07-19, controller v0.148.0:** the DB half now exists in code (R-43 + R-44) and its replay reached a live box — but **round 2 found it aborts against a running app** (`audits/DIAG-immich-restore-round2-2026-07-19.md`, H4: the replay races immich's own schema repair; `clip_index` recreated by the app 2 s before the dump's CREATE INDEX). **2026-07-20, controller v0.153.0: H4 IS CLOSED (R-47)** — both restore paths now replay into a DB-ONLY window (`StartStackServices` brings up the database service alone; the app starts only after the replay exits 0), with a fail-closed refusal when a dump has no identifiable DB service. *(The earlier note here said "closes in v0.149" — that was wrong: v0.149.0 was the F3 dashboard `BackupStatus` fix. R-47 shipped in v0.153.0.)* **2026-07-20: the clean run HAPPENED** — endpoint-level supervised reconstitute of immich from snapshot `49e7cb46` (the very snapshot that aborted in round 2): stop → DB-service-only start → replay rc-0 → full start, no `already exists`, operation reported SUCCESS, immich's own DatabaseService logged `No schema drift detected` twice, 11 assets `active`, 4/4 containers healthy, 231 `public` indexes. Evidence: `felhom-controller/REPORT.md` §4b. **Residual for a PROVEN-LIVE flip: the immich timeline SCREENSHOT** (no browser on DooPlex — operator's) |
| Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill |
| PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending |