admin
14b0939060
Unattended 10h run against demo-felhom, demo-hp and ep0. No production code changed; findings recorded and ranked, not fixed inline. 8 findings, 2 HIGH — both in the system's ability to report that a backup did NOT happen: - F-CRIT-1: an app failing to restart after a quiesce never alarms (invariant I1 in main.go:1213 is false for the failed-restart path) - F-CRIT-2: a failed offsite backup leaves a phantom snapshot that resets the tier's freshness clock (NewestArchiveTime has no completeness check) Retires several never-validated items, including R-87 (first restic restore round-trip, byte-verified), the full R-88 backoff ladder, age_state=absent, and the crash-recovery unquiesce under a real SIGKILL. peti-felhom untouched; ep0 rollback copy intact; fleet healthy at end.
3.8 KiB
REPORT — CAMPAIGN 8: the backup & restore subsystem (2026-07-27/28)
Adversarial, destructive, unattended run against demo-felhom, demo-hp and ep0.
Full report: documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md.
Evidence: DooPlex:~/campaign8/evidence/ (103 files, 35 MB, written continuously by 11 collectors).
No production code was changed. Findings are recorded and ranked, never fixed inline, per the campaign's own rules.
Scope safety
peti-felhom, its namespace and u629488-sub2 were never touched. Phase 0 established with five
documented probes that peti has no data at all in felhom-offsite, which is what made the
operator-approved 100% datastore-fill safe. The 13 GB rollback copy /srv/pbs-felhom on ep0 is
intact.
Findings
| # | Finding | Severity | Class |
|---|---|---|---|
| F-CRIT-1 | An app that fails to restart after a quiesce never alarms, on any channel | HIGH | DEFECT |
| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that resets tier freshness (up to 7 days silent on real cadences) | HIGH | DEFECT |
| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
| F-HUB | The hub dropped an event under concurrent load (SQLITE_BUSY), no retry, cause unnamed |
MEDIUM | DEFECT |
| F-LEAK | A leaked restore-test scratch guest is never reclaimed; the 10-slot VMID band shrinks silently | MEDIUM | DEFECT |
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
| F-OBS | deadapp-check leaves no positive observable on a default (info-level) box |
LOW | DEFECT |
| F-OPS | A manual pct restore inherits the source guest's binds (live data drive + another guest's credentials) |
LOW | Operational |
Both HIGH findings are in the same place: the system's ability to tell you a backup did not happen. Both cite the code and the comment that asserts the property the code does not provide.
What is now proven that was not before
R-88 breaker arming and its full ladder (15m/30m/1h/2h/4h/4h-cap) · per-tier isolation under a
real one-tier-fails case · whole_guest_backup_failed end-to-end with correct tier attribution ·
R-97c operator-only routing verified against the hub DB (zero customer rows with status='sent') ·
whole_guest_backup_recovered + the R-68 pairing gate firing live · age_state=absent ·
R-97b's suppression half · the crash-recovery unquiesce by an actual SIGKILL (1 s) ·
R-87 — the first restic restore round-trip ever performed, byte-verified (6/7 sha256 identical,
the 7th explained) · R-82 one-quiesce-two-tiers · single-flight on two independent paths.
Restore round-trips
restic (R-87) · local vzdump → fresh CT · PBS offsite → fresh CT · corrupted snapshot → fails
cleanly. mount_parity exact on both whole-guest tiers, unprivileged: 1 preserved.
Fleet state
Healthy. Nothing left broken. All four compression knobs reverted and verified; every fault
unwound; no leaked scratch guests, nft rules, ballast files or clock skew; ep0 datastore clean with
zero .bad chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
Not tested (with reasons)
Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; the most valuable follow-up, because F-CRIT-2 raises the same question for restic), faults 11/12 (reboots — reasoned skip after F9/F10 covered hard-kill crash safety), fault 18, fault 13's GC leg, and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for documented reasons.
Note on repo conventions
This run touched no hub/, scripts/ or website/ code, so none of the per-area CHANGELOGs has an
entry — there is nothing shipped to log. The deliverable is the audit document plus this report.