admin
6c57658c1b
Tail-of-campaign additions after the Phase D revert (both re-injections declared): - fault 18 (delete a snapshot mid restore-test): detection PASS, and it ROOT-CAUSES F-LEAK — a failed restore-test cannot destroy its own scratch guest (403, missing VM.Allocate; the agent token is pool-scoped and a failed restore never joins the felhom pool) - fault 11 (guest reboot mid-backup): new finding F-REBOOT — the backup succeeds but the guest never comes back; ~9m47s outage until a manual pct start Two evidence corrections, both self-inflicted tooling errors: - pgrep -cf <pattern> matches its own ssh command line, which invalidated fault 11's first two injections and put one unsound line in fault 9 (withdrawn; that finding stands on the controller's own job state) - ep0 runs Etc/UTC, so its 03:30 prune fires at 05:30 CEST — nearly misread as a broken prune job Nine findings now, still two HIGH. Fleet healthy.
4.2 KiB
REPORT — CAMPAIGN 8: the backup & restore subsystem (2026-07-27/28)
Adversarial, destructive, unattended run against demo-felhom, demo-hp and ep0.
Full report: documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md.
Evidence: DooPlex:~/campaign8/evidence/ (103 files, 35 MB, written continuously by 11 collectors).
No production code was changed. Findings are recorded and ranked, never fixed inline, per the campaign's own rules.
Scope safety
peti-felhom, its namespace and u629488-sub2 were never touched. Phase 0 established with five
documented probes that peti has no data at all in felhom-offsite, which is what made the
operator-approved 100% datastore-fill safe. The 13 GB rollback copy /srv/pbs-felhom on ep0 is
intact.
Findings
| # | Finding | Severity | Class |
|---|---|---|---|
| F-CRIT-1 | An app that fails to restart after a quiesce never alarms, on any channel | HIGH | DEFECT |
| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that resets tier freshness (up to 7 days silent on real cadences) | HIGH | DEFECT |
| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
| F-HUB | The hub dropped an event under concurrent load (SQLITE_BUSY), no retry, cause unnamed |
MEDIUM | DEFECT |
| F-LEAK | A failed restore-test cannot destroy its own scratch guest (403 VM.Allocate); leaks are never reclaimed |
MEDIUM | DEFECT (root-caused by fault 18) |
| F-REBOOT | A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal | MEDIUM | DEFECT |
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
| F-OBS | deadapp-check leaves no positive observable on a default (info-level) box |
LOW | DEFECT |
| F-OPS | A manual pct restore inherits the source guest's binds (live data drive + another guest's credentials) |
LOW | Operational |
Both HIGH findings are in the same place: the system's ability to tell you a backup did not happen. Both cite the code and the comment that asserts the property the code does not provide.
What is now proven that was not before
R-88 breaker arming and its full ladder (15m/30m/1h/2h/4h/4h-cap) · per-tier isolation under a
real one-tier-fails case · whole_guest_backup_failed end-to-end with correct tier attribution ·
R-97c operator-only routing verified against the hub DB (zero customer rows with status='sent') ·
whole_guest_backup_recovered + the R-68 pairing gate firing live · age_state=absent ·
R-97b's suppression half · the crash-recovery unquiesce by an actual SIGKILL (1 s) ·
R-87 — the first restic restore round-trip ever performed, byte-verified (6/7 sha256 identical,
the 7th explained) · R-82 one-quiesce-two-tiers · single-flight on two independent paths.
Restore round-trips
restic (R-87) · local vzdump → fresh CT · PBS offsite → fresh CT · corrupted snapshot → fails
cleanly. mount_parity exact on both whole-guest tiers, unprivileged: 1 preserved.
Fleet state
Healthy. Nothing left broken. All four compression knobs reverted and verified; every fault
unwound; no leaked scratch guests, nft rules, ballast files or clock skew; ep0 datastore clean with
zero .bad chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
Not tested (with reasons)
Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; the most valuable follow-up, because F-CRIT-2 raises the same question for restic), fault 12 (host reboot — reasoned skip), and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for documented reasons. Faults 11 and 18 WERE run in the campaign's tail and both produced findings.
Campaign-caused outage, stated plainly: fault 11 took demo-hp guest 9201 down for ~9m47s (the guest did not restart after a mid-backup reboot) until manually started. Fleet healthy after.
Note on repo conventions
This run touched no hub/, scripts/ or website/ code, so none of the per-area CHANGELOGs has an
entry — there is nothing shipped to log. The deliverable is the audit document plus this report.