admin
14b0939060
Unattended 10h run against demo-felhom, demo-hp and ep0. No production code changed; findings recorded and ranked, not fixed inline. 8 findings, 2 HIGH — both in the system's ability to report that a backup did NOT happen: - F-CRIT-1: an app failing to restart after a quiesce never alarms (invariant I1 in main.go:1213 is false for the failed-restart path) - F-CRIT-2: a failed offsite backup leaves a phantom snapshot that resets the tier's freshness clock (NewestArchiveTime has no completeness check) Retires several never-validated items, including R-87 (first restic restore round-trip, byte-verified), the full R-88 backoff ladder, age_state=absent, and the crash-recovery unquiesce under a real SIGKILL. peti-felhom untouched; ep0 rollback copy intact; fleet healthy at end.
59 lines
3.8 KiB
Markdown
59 lines
3.8 KiB
Markdown
# REPORT — CAMPAIGN 8: the backup & restore subsystem (2026-07-27/28)
|
||
|
||
Adversarial, destructive, unattended run against `demo-felhom`, `demo-hp` and `ep0`.
|
||
Full report: `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||
Evidence: `DooPlex:~/campaign8/evidence/` (103 files, 35 MB, written continuously by 11 collectors).
|
||
|
||
**No production code was changed.** Findings are recorded and ranked, never fixed inline, per the
|
||
campaign's own rules.
|
||
|
||
## Scope safety
|
||
`peti-felhom`, its namespace and `u629488-sub2` were never touched. Phase 0 established with five
|
||
documented probes that peti has **no data at all** in `felhom-offsite`, which is what made the
|
||
operator-approved 100% datastore-fill safe. The 13 GB rollback copy `/srv/pbs-felhom` on ep0 is
|
||
intact.
|
||
|
||
## Findings
|
||
|
||
| # | Finding | Severity | Class |
|
||
|---|---|---|---|
|
||
| F-CRIT-1 | An app that fails to restart after a quiesce **never alarms**, on any channel | **HIGH** | DEFECT |
|
||
| F-CRIT-2 | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** (up to 7 days silent on real cadences) | **HIGH** | DEFECT |
|
||
| F-A1 | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker, pages the operator | MEDIUM | DEFECT (behaviour) / ARTIFACT (frequency) |
|
||
| F-HUB | The hub dropped an event under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT |
|
||
| F-LEAK | A leaked restore-test scratch guest is never reclaimed; the 10-slot VMID band shrinks silently | MEDIUM | DEFECT |
|
||
| F-DIAG | Four distinct offsite failure causes collapse into two operator-visible strings | LOW–MED | DEFECT |
|
||
| F-OBS | `deadapp-check` leaves no positive observable on a default (info-level) box | LOW | DEFECT |
|
||
| F-OPS | A manual `pct restore` inherits the source guest's binds (live data drive + another guest's credentials) | LOW | Operational |
|
||
|
||
Both HIGH findings are in the same place: **the system's ability to tell you a backup did not
|
||
happen.** Both cite the code and the comment that asserts the property the code does not provide.
|
||
|
||
## What is now proven that was not before
|
||
R-88 breaker arming **and its full ladder** (15m/30m/1h/2h/4h/4h-cap) · per-tier isolation under a
|
||
real one-tier-fails case · `whole_guest_backup_failed` end-to-end with correct tier attribution ·
|
||
R-97c operator-only routing verified against the hub DB (zero customer rows with `status='sent'`) ·
|
||
`whole_guest_backup_recovered` + the R-68 pairing gate firing live · **`age_state=absent`** ·
|
||
R-97b's suppression half · the crash-recovery unquiesce by an actual SIGKILL (1 s) ·
|
||
**R-87 — the first restic restore round-trip ever performed**, byte-verified (6/7 sha256 identical,
|
||
the 7th explained) · R-82 one-quiesce-two-tiers · single-flight on two independent paths.
|
||
|
||
## Restore round-trips
|
||
restic (R-87) · local vzdump → fresh CT · PBS offsite → fresh CT · corrupted snapshot → fails
|
||
cleanly. `mount_parity` exact on both whole-guest tiers, `unprivileged: 1` preserved.
|
||
|
||
## Fleet state
|
||
**Healthy. Nothing left broken.** All four compression knobs reverted and verified; every fault
|
||
unwound; no leaked scratch guests, nft rules, ballast files or clock skew; ep0 datastore clean with
|
||
zero `.bad` chunks. demo-felhom 15/15 containers healthy, demo-hp 8/8.
|
||
|
||
## Not tested (with reasons)
|
||
Fault 4 (restic transport — four injection approaches defeated by guest-bridged networking; **the
|
||
most valuable follow-up**, because F-CRIT-2 raises the same question for restic), faults 11/12
|
||
(reboots — reasoned skip after F9/F10 covered hard-kill crash safety), fault 18, fault 13's GC leg,
|
||
and the agent's own DR bring-up path. Faults 6 and 8 were inconclusive for documented reasons.
|
||
|
||
## Note on repo conventions
|
||
This run touched no `hub/`, `scripts/` or `website/` code, so none of the per-area CHANGELOGs has an
|
||
entry — there is nothing shipped to log. The deliverable is the audit document plus this report.
|