R-403 CLOSED (controller v0.230.0), R-404 filed as a decision for Viktor
gates / gates (push) Failing after 17s
gates / gates (push) Failing after 17s
07-backup-architecture gains section 8.2, placed beside row 5 on purpose: the derived-copy rebuild rule is UNCHANGED and section 8.2 names the single exception, so a future reader who finds RunTier2 skipping a leg does not fix it back. It carries the measurement (120 082 104 B -> 7 036 B on the shipped v0.229.0), the four-case table, why hollow is a manifest question and not a size question, why the data legs are deliberately not guarded, and why the capture job is not guarded either. 00-capability-map: the Tier-2 row's status does NOT move, stated explicitly rather than left ambiguous. R-403 removes a way the route could be DESTROYED between uses; it does not change what the route can be relied on for. Register: R-403 CLOSED and compressed into CLOSED-ITEMS (594 -> 593 open lines). R-404 FILED as a DECISION and deliberately not acted on - should a documents-only push be subject to the golden-currency gate, now that it has been correctly bypassed six times? Both sides stated, plus what happens if Viktor does nothing. The gate was NOT changed. R-242: seventh conviction, and the FIRST where the day-0 ground does not apply - R-403 is a defect in the nightly Tier-2 copy, which a day-0 box starts running on its first night. This push uses git push --no-verify, declared here and in felhom-controller/REPORT.md. A golden carrying 0.230.0 is owed and is more urgent than the previous six. STATUS: the R-403 item moves out of 'Broken' into what works, in plain words; the delivery item now says a golden is owed and that the fleet carries the defect; R-404 goes into the decide section with its do-nothing outcome. Drill evidence: nine phase logs, including the two things that went wrong (a repair whose rsync was not installed in the guest and silently did nothing, and a session that expired mid-run so a POST did nothing).
This commit is contained in:
@@ -126,7 +126,7 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis
|
||||
|---|---|---|---|---|
|
||||
| Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded **DEGRADED** rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | `E2D-fresh-vm-2026-07-29` C1 (real 1.22.0 install, rc=0, `Day-0 provision SUCCESS`) + C2 (both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort) | Case A (a second drive already present at install) has never fired naturally — only Case B has |
|
||||
| Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 **DB replay route → `07-backup-architecture.md` §8 row 3** |
|
||||
| Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | **Route + RTO → `07-backup-architecture.md` §8 rows 1, 2, 3b, 4, 5.** **R-102 CLOSED 2026-08-31 (controller v0.229.0):** the copy's `recovery-unit/` mirror is now restorable — „Teljes visszaállítás a másolatból" / `POST /backup/tier2/unit-restore` — and was proven live on `demo-hp` with the primary unit moved aside (`audits/DRILL-r102-tier2-unit-2026-08-31/`, §8 row 3b, 28.65 s). R-103 closed with it: the refusal that used to name a button on another page now offers the action on the row itself |
|
||||
| Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | **Route + RTO → `07-backup-architecture.md` §8 rows 1, 2, 3b, 4, 5.** **R-403 (controller v0.230.0, 2026-08-31) changes NO status on this row and that is stated rather than left ambiguous:** the nightly copy now refuses to replace a complete unit package with an empty one (`07` §8.2), which removes a way the route could be DESTROYED between uses — it does not change what the route can be relied on for, and every leg it covers is the same one it covered yesterday. Proven live on demo-hp: the same state that deleted 120 082 104 B on v0.229.0 preserved all 7 files on v0.230.0. **R-102 CLOSED 2026-08-31 (controller v0.229.0):** the copy's `recovery-unit/` mirror is now restorable — „Teljes visszaállítás a másolatból" / `POST /backup/tier2/unit-restore` — and was proven live on `demo-hp` with the primary unit moved aside (`audits/DRILL-r102-tier2-unit-2026-08-31/`, §8 row 3b, 28.65 s). R-103 closed with it: the refusal that used to name a button on another page now offers the action on the row itself |
|
||||
| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` **Route + RTO → `07-backup-architecture.md` §8 rows 4, 10, 12, 15** (incl. the R-95 delete exposure and the R-104 stale-lock defect). **2026-08-04 (R-193/R-197, `audits/SPIKE-offsite-credential-recovery-2026-08-04.md`) — the row is NOT overclaiming and the guard is not the gap; the CADENCE is.** This row already recorded that a recreated data volume orphans the repo, and the spike confirms the mechanism at source: `WriteOffboxSecrets` (`offbox.go:392`) mints a fresh 256-bit repo password whenever `<DataDir>/offbox/repo_password` is absent, and **no automatic path ever consults the escrowed one** — `InjectOffboxPassword` has exactly one caller, a web form a human pastes into. What was NOT recorded is that this now fires on an **ordinary, planned, unattended guest rebuild**, on every box: measured on BOTH demo boxes 2026-08-03/04 by comparing `host_escrow.restic_pw_sha256` against `host_escrow_superseded.restic_pw_sha256` (demo-hp `8e03eddf…`→`8a9e33aa…`, demo-felhom `48741892…`→`c60c8bc7…`), orphaning **15 snapshots / 40.9 MB** and **36 snapshots / 1.14 GB** respectively. **demo-felhom is the important half:** it kept its DELIVERY (a stale staged secret restored the target in 76 s) and lost its REPOSITORY anyway, with **nothing marking the escrow stale for 13 h** — `escrow_stale` is wired to `ReissueCredentials`, the one path that does NOT change the repo password (**R-196**), and absent from the rebuild path that does. Status unchanged: the classify-and-move-aside guard remains PROVEN-LIVE and correct, and is predicted (not yet measured) to refuse the 2026-08-05 run on both boxes rather than start a silent fresh history |
|
||||
| Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** (2026-07-20) | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | **2026-07-19:** `audits/DIAG-immich-restore-2026-07-19.md` finds **no offsite path loads a DB dump** — all three buttons are file-only (R-43). The mechanics in this row's title are each proven; the phrase "**immich end-to-end from offsite alone**" is what is contested, since a DB-indexed app cannot be reconstituted by any offsite action. **RULED 2026-07-19 (Viktor): 6D's destruction hit the FILE TREE ONLY — the database survived in its named volume** (`immich_postgres_data` is a named volume in both the v2 and v3 template eras), so "immich end-to-end from offsite alone" **overclaimed scope**: the file half was proven, the DB half was never destroyed and therefore never restored. Row downgraded PROVEN-LIVE → **PARTIAL**, scope-corrected. Evidence: `audits/DIAG-immich-restore-2026-07-19.md` (no offsite path could replay a DB at all) + the P-FAB destructive re-import (the proven-replay evidence, on the LOCAL path). **2026-07-19, controller v0.148.0:** the DB half now exists in code (R-43 + R-44) and its replay reached a live box — but **round 2 found it aborts against a running app** (`audits/DIAG-immich-restore-round2-2026-07-19.md`, H4: the replay races immich's own schema repair; `clip_index` recreated by the app 2 s before the dump's CREATE INDEX). **2026-07-20, controller v0.153.0: H4 IS CLOSED (R-47)** — both restore paths now replay into a DB-ONLY window (`StartStackServices` brings up the database service alone; the app starts only after the replay exits 0), with a fail-closed refusal when a dump has no identifiable DB service. *(The earlier note here said "closes in v0.149" — that was wrong: v0.149.0 was the F3 dashboard `BackupStatus` fix. R-47 shipped in v0.153.0.)* **2026-07-20: the clean run HAPPENED** — endpoint-level supervised reconstitute of immich from snapshot `49e7cb46` (the very snapshot that aborted in round 2): stop → DB-service-only start → replay rc-0 → full start, no `already exists`, operation reported SUCCESS, immich's own DatabaseService logged `No schema drift detected` twice, 11 assets `active`, 4/4 containers healthy, 231 `public` indexes. **Operator confirmed the immich timeline renders correctly after the reconstitute** (screenshot held, 2026-07-20). Evidence: `felhom-controller/REPORT.md` §4b. **2026-07-20, LATER THE SAME DAY — the destructive drill RAN and the row now earns PROVEN-LIVE.** The operator deleted the photos in immich own UI **and emptied the trash** (the step whose absence makes a drill prove nothing — the round-1 lesson), then restored through the customer-facing UI. **`40 file(s) placed`** against the 6 of the earlier non-destructive run — the files were really gone and really came back — plus 1 DB dump replayed rc-0, 11 assets `active`, `No schema drift detected`, timeline confirmed by the operator. This is the destroy-then-recover proof the 6D downgrade asked for, and it was taken through the customer own buttons, not endpoint shortcuts. Evidence: `felhom-controller/REPORT.md` 4e **Route + RTO → `07-backup-architecture.md` §8 rows 3, 4** — the matrix also records that no offsite action unpacks the named-volume tars it captures (→ R-107) |
|
||||
| Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
|
||||
|
||||
Reference in New Issue
Block a user