docs: CAMPAIGN 8 — faults 11 and 18 results, F-LEAK root cause, evidence corrections

Tail-of-campaign additions after the Phase D revert (both re-injections declared):
- fault 18 (delete a snapshot mid restore-test): detection PASS, and it ROOT-CAUSES
  F-LEAK — a failed restore-test cannot destroy its own scratch guest (403,
  missing VM.Allocate; the agent token is pool-scoped and a failed restore never
  joins the felhom pool)
- fault 11 (guest reboot mid-backup): new finding F-REBOOT — the backup succeeds
  but the guest never comes back; ~9m47s outage until a manual pct start

Two evidence corrections, both self-inflicted tooling errors:
- pgrep -cf <pattern> matches its own ssh command line, which invalidated fault
  11's first two injections and put one unsound line in fault 9 (withdrawn; that
  finding stands on the controller's own job state)
- ep0 runs Etc/UTC, so its 03:30 prune fires at 05:30 CEST — nearly misread as a
  broken prune job

Nine findings now, still two HIGH. Fleet healthy.
This commit is contained in:
2026-07-28 06:06:57 +02:00
parent 14b0939060
commit 6c57658c1b
2 changed files with 35 additions and 15 deletions
+8 -4
View File
@@ -21,7 +21,8 @@ intact.
| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT |
| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
| F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT |
| F-LEAK | A leaked restore-test scratch guest is never reclaimed; the 10-slot VMID band shrinks silently | MEDIUM | DEFECT |
| F-LEAK | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed | MEDIUM | DEFECT (root-caused by fault 18) |
| F-REBOOT | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOWMED | DEFECT |
| F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT |
| F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational |
@@ -49,9 +50,12 @@ zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
## Not tested (with reasons)
Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), faults 11/12
(reboots — reasoned skip after F9/F10 covered hard-kill crash safety), fault 18, fault 13's GC leg,
and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for documented reasons.
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), fault 12 (host
reboot — reasoned skip), and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for
documented reasons. Faults 11 and 18 WERE run in the campaign's tail and both produced findings.
**Campaign-caused outage, stated plainly:** fault 11 took demo-hp guest 9201 down for ~9m47s
(the guest did not restart after a mid-backup reboot) until manually started. Fleet healthy after.
## Note on repo conventions
This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an