docs(audits): remote-ops remediation — F1 mitigated, F2 closed, F3 fixed (v0.149.0), Immich resolved; R-50..R-53 minted

This commit is contained in:
2026-07-20 09:09:25 +02:00
parent 648ee67910
commit 30fd9d3ecd
3 changed files with 155 additions and 0 deletions
@@ -63,6 +63,7 @@
| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill |
| PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | hub v0.56.0 (`pbsdrheal/reconciler.go`, `RestageHostPBSSecret`, all §10 red-proofs); `SPIKE-pbsdr-selfheal-2026-07-15` (root cause) | Reconciler is **scoped to one host** (`PBSDRHEAL_ONLY_HOST`), not fleet-wide; already fired live hands-free on drill qm300 (07-15) — real-customer firing + fleet-wide widening pending |
| Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | `CAMPAIGN-6D` P5-REST (SIGKILL mid-offbox → auto-restart ~15s, run marked failed not false-success, no stale lock); `CAMPAIGN-6E` B1-B3 | (Cited `CAMPAIGN-2` T-RBT-* legs were empty / auth-hollow — corrected.) Live mid-**migration** crash→self-heal is the weakest sub-claim (P5-REST is mid-backup) |
| Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent, controller, bootstrap | **PARTIAL** | `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — a real relocation of the demo box: guest + hub telemetry + WG/PBS + Cloudflare tunnel all survived untouched, but the **controller↔agent control plane did not** (agent binds a LAN literal → `bind: cannot assign requested address` → storage/PBS-backup/quiesce/restore-test/DR down until fixed). Mitigated for the window by pinning `vmbr0` static | **R-50** (island-bridge control plane, spike-first) is the durable fix. Related: **R-51** (dead-primary alerting) and **R-52** (boot desired-state reconciliation) — the same event left two apps `Exited` with no alarm and no recovery |
| Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
| **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence**`audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. Blocked on H4 closing in v0.149, then one clean run. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count |