admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: CAMPAIGN 8 — faults 11 and 18 results, F-LEAK root cause, evidence corrections

Browse Source 
Tail-of-campaign additions after the Phase D revert (both re-injections declared):
- fault 18 (delete a snapshot mid restore-test): detection PASS, and it ROOT-CAUSES
  F-LEAK — a failed restore-test cannot destroy its own scratch guest (403,
  missing VM.Allocate; the agent token is pool-scoped and a failed restore never
  joins the felhom pool)
- fault 11 (guest reboot mid-backup): new finding F-REBOOT — the backup succeeds
  but the guest never comes back; ~9m47s outage until a manual pct start

Two evidence corrections, both self-inflicted tooling errors:
- pgrep -cf <pattern> matches its own ssh command line, which invalidated fault
  11's first two injections and put one unsound line in fault 9 (withdrawn; that
  finding stands on the controller's own job state)
- ep0 runs Etc/UTC, so its 03:30 prune fires at 05:30 CEST — nearly misread as a
  broken prune job

Nine findings now, still two HIGH. Fleet healthy.
This commit is contained in:
authentik Default Admin
2026-07-28 06:06:57 +02:00
parent 14b0939060
commit 6c57658c1b
2 changed files with 35 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT-campaign8.md
+8 -4
View File
@@ -21,7 +21,8 @@ intact.
| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT |
| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
| F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT |
| F-LEAK | A leaked restore-test scratch guest is never reclaimed; the 10-slot VMID band shrinks silently | MEDIUM | DEFECT |
| F-LEAK | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed | MEDIUM | DEFECT (root-caused by fault 18) |
| F-REBOOT | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
| F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT |
| F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational |
@@ -49,9 +50,12 @@ zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
## Not tested (with reasons)
Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), faults 11/12
(reboots — reasoned skip after F9/F10 covered hard-kill crash safety), fault 18, fault 13's GC leg,
and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for documented reasons.
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), fault 12 (host
reboot — reasoned skip), and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for
documented reasons. Faults 11 and 18 WERE run in the campaign's tail and both produced findings.
**Campaign-caused outage, stated plainly:** fault 11 took demo-hp guest 9201 down for ~9m47s
(the guest did not restart after a mid-backup reboot) until manually started. Fleet healthy after.
## Note on repo conventions
This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an
documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md
+27 -11
View File
@@ -16,7 +16,7 @@ collectors throughout. Nothing in this report is reconstructed after the fact.
## Headline
Six findings, two of them HIGH severity, both in the same place: **the system's ability to tell you
Nine findings, two of them HIGH severity, both in the same place: **the system's ability to tell you
that a backup did not happen.**
- **F-CRIT-1** — an app that fails to restart after a quiesce **never alarms, on any channel**.
@@ -181,16 +181,16 @@ young.
| 6 | fill local disk mid-backup | **INCONCLUSIVE** — backup succeeded on ext4's 5% root reserve (2.07 GB) while `df` read 100% | `F6-fill-local-disk.md` |
| 7 | fill the PBS volume to 100% | **PASS** — clean ENOSPC, no corruption, no leaked LVM snapshot, instant recovery | `F7-fill-pbs-volume.md` |
| 8 | fill the guest disk during a quiesce | **PARTIAL** — vzdump correctly unaffected; the write-path volume was not the one filled | `F8-fill-guest-disk.md` |
| 9 | kill the agent mid-backup | **PASS** — apps restarted in 10 s; agent auto-restarted in 17 s; vzdump survived (it is a PVE task) | `F9-kill-agent-mid-backup.md` |
| 9 | kill the agent mid-backup | **PASS** — apps restarted in 10 s; agent auto-restarted in 17 s. (One evidence line withdrawn — see the correction note) | `F9-kill-agent-mid-backup.md` |
| 10 | kill the controller mid-quiesce | **PASS** — crash recovery in **1 second**; and it corrects the premise (see below) | `F10-kill-controller-mid-quiesce.md` |
| 11 | reboot the guest mid-backup | **NOT TESTED** — see §9 | — |
| 11 | reboot the guest mid-backup | **FINDING (F-REBOOT)** — the backup succeeded but **the guest never came back**; found `stopped` with 0 containers, no self-heal, ~9m47s outage until manual `pct start` | `F11-reboot-guest-mid-backup.md` |
| 12 | reboot the host mid-backup | **NOT TESTED** — see §9 | — |
| 13 | backup + restore-test + GC concurrently | **HALF PASS** — backup vs restore-test single-flight HOLDS, proven live and unplanned. The +GC combination was not run | `F-A1-*.md` |
| 14 | two quiesce triggers at once | **PASS** — `TryLock` + skip, exactly one cycle ran | `F14-two-triggers.md` |
| 15 | clock skew (+6 h) | **PASS** — gate follows wall clock (correct); **breaker is skew-proof** via Go monotonic time | `F15-clock-skew.md` |
| 16 | `age_state=absent` (never observed live) | **PASS** — valve fired, cycle ran OUTSIDE a closed window on a genuinely never-backed-up tier | `F16-age-state-absent.md` |
| 17 | corrupt a chunk | **PASS on all four legs** — detect → quarantine `.0.bad` → `verify_state: failed` → restore fails cleanly | `F17-corrupt-chunk.md` |
| 18 | delete a snapshot mid restore-test | **NOT TESTED** — see §9 | — |
| 18 | delete a snapshot mid restore-test | **PASS on detection** — restore-test failed loudly, did NOT advance the proven-restorable timestamp; **and it root-caused F-LEAK** | `F18-delete-snapshot-mid-restore-test.md` |
| 19 | R-97b both halves | **part 1 PASS** (suppression, with the detector proven alive); **part 2 FAIL → F-CRIT-1** | `F19-part1-*.md`, `F-CRIT-1-*.md` |
**Fault 10 corrects its own premise.** The spec describes the mechanism as "the guaranteed
@@ -238,7 +238,8 @@ Neither restored guest was booted — deliberately, see F-OPS below.
| **F-CRIT-2** | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** | **HIGH** | **DEFECT** (worse on real cadences) |
| **F-A1** | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker and pages the operator | MEDIUM | **DEFECT** in behaviour / ARTIFACT in frequency |
| **F-HUB** | The hub **dropped an event** under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT (frequency compression-influenced) |
| **F-LEAK** | A leaked restore-test scratch guest is never reclaimed; the 10-slot VMID band shrinks silently | MEDIUM | DEFECT in the reclaim path |
| **F-LEAK** | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed and the 10-slot VMID band shrinks silently | MEDIUM | DEFECT (root-caused by fault 18) |
| **F-REBOOT** | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
| **F-DIAG** | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
| **F-OBS** | `deadapp-check` leaves NO positive observable on a default (info-level) box | LOW | DEFECT |
| **F-OPS** | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational, not code |
@@ -328,6 +329,11 @@ against a daily backup collides roughly once per 420 guest-days, i.e. **every ~4
- **R-87: the restic tier has a restore round-trip**, byte-verified.
- **R-82 one-quiesce-two-tiers**, and per-tier isolation under a real one-tier-fails case.
- **Single-flight**: backup vs restore-test, and scheduled vs manual trigger.
- **A restore-test whose source vanishes fails loudly and does NOT falsely advance the tier's
proven-restorable timestamp** (fault 18).
- **R-89's write-only box grant** — the box was correctly refused when it tried to delete its own
offsite snapshot (`missing Datastore.Modify|Datastore.Prune`). A compromised box cannot destroy
its own backups.
**Still NOT validated:**
- **R-97b's other half** — an app that genuinely fails to restart does **not** alarm (F-CRIT-1).
@@ -369,12 +375,12 @@ F-CRIT-2's live effect.
version, and a guest-side blackhole route did not cover the container's path. **This is the single
most valuable follow-up**, because F-CRIT-2 raises the analogous question for restic: does an
interrupted restic run leave a partial pack that later looks like success?
- **Faults 11 and 12 (guest / host reboot mid-backup)** — not run. Deliberate: F9 and F10 already
exercised hard-kill crash safety for both the agent and the controller, and a host reboot at
02:00 on a remote box with no console carried recovery risk disproportionate to the marginal
information. Recorded as a reasoned skip, not an oversight.
- **Fault 18 (delete a snapshot out from under a restore-test)** — not reached before the campaign's
time budget ran out.
- **Fault 12 (host reboot mid-backup)** — not run. Deliberate: F9 and F10 already exercised
hard-kill crash safety for both the agent and the controller, and rebooting a remote Proxmox host
with no console access carried recovery risk disproportionate to the marginal information.
- ~~Fault 18~~ — **was run** in the campaign's tail (see the matrix). It required a single, declared
re-injection after Phase D (restore-test cadence on demo-hp only, 84h→600s), which was reverted
and Phase D re-verified immediately afterwards.
- **Fault 13's GC leg** — the backup-vs-restore-test half was proven; adding GC was not run.
- **Fault 6** — inconclusive: ext4's 5% root reserve (2.07 GB, confirmed via `tune2fs`) meant the
filesystem was never truly full for a root-run vzdump. Driving a Proxmox host's root filesystem
@@ -395,5 +401,15 @@ F-CRIT-2's live effect.
2. **A ~64 KB single-line ceiling through `ssh → pct exec`** silently returned 0 bytes for a
debug-ring fetch at `limit=400`, and the parser swallowed the error — presenting as "this box has
no DEBUG lines". Parser now fails loudly.
3. **`pgrep -cf <pattern>` matches its own shell** when the pattern appears in the `ssh` command
line that `pgrep -f` inspects. `pgrep -cf vzdump` → 1 while
`ps -eo args | grep -c '[/]usr/bin/vzdump'` → 0. This invalidated fault 11's injection outright
and put one unsound line into fault 9's evidence (withdrawn; that finding stands on the
controller's own job state instead). Drive such triggers off the controller's job log, which is
authoritative and cannot self-match.
All three traps share a shape: **the instrument agreed with the hypothesis for the wrong reason.**
Two were caught only because a second, independent source disagreed — a container count, and a
job-state log. Where no second source existed, the campaign reports NOT TESTED rather than a pass.
---