From 6c57658c1b09282227eb12e7bee183565f1791fb Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 28 Jul 2026 06:06:57 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20CAMPAIGN=208=20=E2=80=94=20faults=2011?= =?UTF-8?q?=20and=2018=20results,=20F-LEAK=20root=20cause,=20evidence=20co?= =?UTF-8?q?rrections?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tail-of-campaign additions after the Phase D revert (both re-injections declared): - fault 18 (delete a snapshot mid restore-test): detection PASS, and it ROOT-CAUSES F-LEAK — a failed restore-test cannot destroy its own scratch guest (403, missing VM.Allocate; the agent token is pool-scoped and a failed restore never joins the felhom pool) - fault 11 (guest reboot mid-backup): new finding F-REBOOT — the backup succeeds but the guest never comes back; ~9m47s outage until a manual pct start Two evidence corrections, both self-inflicted tooling errors: - pgrep -cf matches its own ssh command line, which invalidated fault 11's first two injections and put one unsound line in fault 9 (withdrawn; that finding stands on the controller's own job state) - ep0 runs Etc/UTC, so its 03:30 prune fires at 05:30 CEST — nearly misread as a broken prune job Nine findings now, still two HIGH. Fleet healthy. --- REPORT-campaign8.md | 12 ++++-- .../CAMPAIGN-8-backup-restore-2026-07-27.md | 38 +++++++++++++------ 2 files changed, 35 insertions(+), 15 deletions(-) diff --git a/REPORT-campaign8.md b/REPORT-campaign8.md index 76be9cb..07d1a1c 100644 --- a/REPORT-campaign8.md +++ b/REPORT-campaign8.md @@ -21,7 +21,8 @@ intact. | F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT | | F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) | | F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT | -| F-LEAK | A leaked restore-test scratch guest is never reclaimed; the 10-slot VMID band shrinks silently | MEDIUM | DEFECT | +| F-LEAK | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed | MEDIUM | DEFECT (root-caused by fault 18) | +| F-REBOOT | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT | | F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT | | F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT | | F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational | @@ -49,9 +50,12 @@ zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8. ## Not tested (with reasons) Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the -most valuable follow-up**, because F-CRIT-2 raises the same question for restic), faults 11/12 -(reboots — reasoned skip after F9/F10 covered hard-kill crash safety), fault 18, fault 13's GC leg, -and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for documented reasons. +most valuable follow-up**, because F-CRIT-2 raises the same question for restic), fault 12 (host +reboot — reasoned skip), and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for +documented reasons. Faults 11 and 18 WERE run in the campaign's tail and both produced findings. + +**Campaign-caused outage, stated plainly:** fault 11 took demo-hp guest 9201 down for ~9m47s +(the guest did not restart after a mid-backup reboot) until manually started. Fleet healthy after. ## Note on repo conventions This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an diff --git a/documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md b/documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md index fdb38ce..598f607 100644 --- a/documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md +++ b/documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md @@ -16,7 +16,7 @@ collectors throughout. Nothing in this report is reconstructed after the fact. ## Headline -Six findings, two of them HIGH severity, both in the same place: **the system's ability to tell you +Nine findings, two of them HIGH severity, both in the same place: **the system's ability to tell you that a backup did not happen.** - **F-CRIT-1** — an app that fails to restart after a quiesce **never alarms, on any channel**. @@ -181,16 +181,16 @@ young. | 6 | fill local disk mid-backup | **INCONCLUSIVE** — backup succeeded on ext4's 5% root reserve (2.07 GB) while `df` read 100% | `F6-fill-local-disk.md` | | 7 | fill the PBS volume to 100% | **PASS** — clean ENOSPC, no corruption, no leaked LVM snapshot, instant recovery | `F7-fill-pbs-volume.md` | | 8 | fill the guest disk during a quiesce | **PARTIAL** — vzdump correctly unaffected; the write-path volume was not the one filled | `F8-fill-guest-disk.md` | -| 9 | kill the agent mid-backup | **PASS** — apps restarted in 10 s; agent auto-restarted in 17 s; vzdump survived (it is a PVE task) | `F9-kill-agent-mid-backup.md` | +| 9 | kill the agent mid-backup | **PASS** — apps restarted in 10 s; agent auto-restarted in 17 s. (One evidence line withdrawn — see the correction note) | `F9-kill-agent-mid-backup.md` | | 10 | kill the controller mid-quiesce | **PASS** — crash recovery in **1 second**; and it corrects the premise (see below) | `F10-kill-controller-mid-quiesce.md` | -| 11 | reboot the guest mid-backup | **NOT TESTED** — see §9 | — | +| 11 | reboot the guest mid-backup | **FINDING (F-REBOOT)** — the backup succeeded but **the guest never came back**; found `stopped` with 0 containers, no self-heal, ~9m47s outage until manual `pct start` | `F11-reboot-guest-mid-backup.md` | | 12 | reboot the host mid-backup | **NOT TESTED** — see §9 | — | | 13 | backup + restore-test + GC concurrently | **HALF PASS** — backup vs restore-test single-flight HOLDS, proven live and unplanned. The +GC combination was not run | `F-A1-*.md` | | 14 | two quiesce triggers at once | **PASS** — `TryLock` + skip, exactly one cycle ran | `F14-two-triggers.md` | | 15 | clock skew (+6 h) | **PASS** — gate follows wall clock (correct); **breaker is skew-proof** via Go monotonic time | `F15-clock-skew.md` | | 16 | `age_state=absent` (never observed live) | **PASS** — valve fired, cycle ran OUTSIDE a closed window on a genuinely never-backed-up tier | `F16-age-state-absent.md` | | 17 | corrupt a chunk | **PASS on all four legs** — detect → quarantine `.0.bad` → `verify_state: failed` → restore fails cleanly | `F17-corrupt-chunk.md` | -| 18 | delete a snapshot mid restore-test | **NOT TESTED** — see §9 | — | +| 18 | delete a snapshot mid restore-test | **PASS on detection** — restore-test failed loudly, did NOT advance the proven-restorable timestamp; **and it root-caused F-LEAK** | `F18-delete-snapshot-mid-restore-test.md` | | 19 | R-97b both halves | **part 1 PASS** (suppression, with the detector proven alive); **part 2 FAIL → F-CRIT-1** | `F19-part1-*.md`, `F-CRIT-1-*.md` | **Fault 10 corrects its own premise.** The spec describes the mechanism as "the guaranteed @@ -238,7 +238,8 @@ Neither restored guest was booted — deliberately, see F-OPS below. | **F-CRIT-2** | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** | **HIGH** | **DEFECT** (worse on real cadences) | | **F-A1** | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker and pages the operator | MEDIUM | **DEFECT** in behaviour / ARTIFACT in frequency | | **F-HUB** | The hub **dropped an event** under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT (frequency compression-influenced) | -| **F-LEAK** | A leaked restore-test scratch guest is never reclaimed; the 10-slot VMID band shrinks silently | MEDIUM | DEFECT in the reclaim path | +| **F-LEAK** | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed and the 10-slot VMID band shrinks silently | MEDIUM | DEFECT (root-caused by fault 18) | +| **F-REBOOT** | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT | | **F-DIAG** | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT | | **F-OBS** | `deadapp-check` leaves NO positive observable on a default (info-level) box | LOW | DEFECT | | **F-OPS** | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational, not code | @@ -328,6 +329,11 @@ against a daily backup collides roughly once per 420 guest-days, i.e. **every ~4 - **R-87: the restic tier has a restore round-trip**, byte-verified. - **R-82 one-quiesce-two-tiers**, and per-tier isolation under a real one-tier-fails case. - **Single-flight**: backup vs restore-test, and scheduled vs manual trigger. +- **A restore-test whose source vanishes fails loudly and does NOT falsely advance the tier's + proven-restorable timestamp** (fault 18). +- **R-89's write-only box grant** — the box was correctly refused when it tried to delete its own + offsite snapshot (`missing Datastore.Modify|Datastore.Prune`). A compromised box cannot destroy + its own backups. **Still NOT validated:** - **R-97b's other half** — an app that genuinely fails to restart does **not** alarm (F-CRIT-1). @@ -369,12 +375,12 @@ F-CRIT-2's live effect. version, and a guest-side blackhole route did not cover the container's path. **This is the single most valuable follow-up**, because F-CRIT-2 raises the analogous question for restic: does an interrupted restic run leave a partial pack that later looks like success? -- **Faults 11 and 12 (guest / host reboot mid-backup)** — not run. Deliberate: F9 and F10 already - exercised hard-kill crash safety for both the agent and the controller, and a host reboot at - 02:00 on a remote box with no console carried recovery risk disproportionate to the marginal - information. Recorded as a reasoned skip, not an oversight. -- **Fault 18 (delete a snapshot out from under a restore-test)** — not reached before the campaign's - time budget ran out. +- **Fault 12 (host reboot mid-backup)** — not run. Deliberate: F9 and F10 already exercised + hard-kill crash safety for both the agent and the controller, and rebooting a remote Proxmox host + with no console access carried recovery risk disproportionate to the marginal information. +- ~~Fault 18~~ — **was run** in the campaign's tail (see the matrix). It required a single, declared + re-injection after Phase D (restore-test cadence on demo-hp only, 84h→600s), which was reverted + and Phase D re-verified immediately afterwards. - **Fault 13's GC leg** — the backup-vs-restore-test half was proven; adding GC was not run. - **Fault 6** — inconclusive: ext4's 5% root reserve (2.07 GB, confirmed via `tune2fs`) meant the filesystem was never truly full for a root-run vzdump. Driving a Proxmox host's root filesystem @@ -395,5 +401,15 @@ F-CRIT-2's live effect. 2. **A ~64 KB single-line ceiling through `ssh → pct exec`** silently returned 0 bytes for a debug-ring fetch at `limit=400`, and the parser swallowed the error — presenting as "this box has no DEBUG lines". Parser now fails loudly. +3. **`pgrep -cf ` matches its own shell** when the pattern appears in the `ssh` command + line that `pgrep -f` inspects. `pgrep -cf vzdump` → 1 while + `ps -eo args | grep -c '[/]usr/bin/vzdump'` → 0. This invalidated fault 11's injection outright + and put one unsound line into fault 9's evidence (withdrawn; that finding stands on the + controller's own job state instead). Drive such triggers off the controller's job log, which is + authoritative and cannot self-match. + +All three traps share a shape: **the instrument agreed with the hypothesis for the wrong reason.** +Two were caught only because a second, independent source disagreed — a container count, and a +job-state log. Where no second source existed, the campaign reports NOT TESTED rather than a pass. ---