docs: CAMPAIGN 8 — faults 11 and 18 results, F-LEAK root cause, evidence corrections
Tail-of-campaign additions after the Phase D revert (both re-injections declared): - fault 18 (delete a snapshot mid restore-test): detection PASS, and it ROOT-CAUSES F-LEAK — a failed restore-test cannot destroy its own scratch guest (403, missing VM.Allocate; the agent token is pool-scoped and a failed restore never joins the felhom pool) - fault 11 (guest reboot mid-backup): new finding F-REBOOT — the backup succeeds but the guest never comes back; ~9m47s outage until a manual pct start Two evidence corrections, both self-inflicted tooling errors: - pgrep -cf <pattern> matches its own ssh command line, which invalidated fault 11's first two injections and put one unsound line in fault 9 (withdrawn; that finding stands on the controller's own job state) - ep0 runs Etc/UTC, so its 03:30 prune fires at 05:30 CEST — nearly misread as a broken prune job Nine findings now, still two HIGH. Fleet healthy.
This commit is contained in:
+8
-4
@@ -21,7 +21,8 @@ intact.
|
||||
| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT |
|
||||
| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
|
||||
| F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT |
|
||||
| F-LEAK | A leaked restore-test scratch guest is never reclaimed; the 10-slot VMID band shrinks silently | MEDIUM | DEFECT |
|
||||
| F-LEAK | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed | MEDIUM | DEFECT (root-caused by fault 18) |
|
||||
| F-REBOOT | A guest rebooted during its backup **does not come back** — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
|
||||
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
|
||||
| F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT |
|
||||
| F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational |
|
||||
@@ -49,9 +50,12 @@ zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
|
||||
|
||||
## Not tested (with reasons)
|
||||
Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the
|
||||
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), faults 11/12
|
||||
(reboots — reasoned skip after F9/F10 covered hard-kill crash safety), fault 18, fault 13's GC leg,
|
||||
and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for documented reasons.
|
||||
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), fault 12 (host
|
||||
reboot — reasoned skip), and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for
|
||||
documented reasons. Faults 11 and 18 WERE run in the campaign's tail and both produced findings.
|
||||
|
||||
**Campaign-caused outage, stated plainly:** fault 11 took demo-hp guest 9201 down for ~9m47s
|
||||
(the guest did not restart after a mid-backup reboot) until manually started. Fleet healthy after.
|
||||
|
||||
## Note on repo conventions
|
||||
This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an
|
||||
|
||||
Reference in New Issue
Block a user