diff --git a/REPORT-fcrit2.md b/REPORT-fcrit2.md new file mode 100644 index 0000000..469a303 --- /dev/null +++ b/REPORT-fcrit2.md @@ -0,0 +1,40 @@ +# REPORT — F-CRIT-2 fixed: a failed backup no longer looks like a fresh one (2026-07-28) + +Scope: `felhom-agent` v0.105.0 → **v0.106.0**. Docs here. Implementation detail and the full live +re-test live in `felhom-agent/REPORT.md`; the campaign that found it is +`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`. + +## What changed +`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, and NEWEST) as a +successful backup, so the tier read fresh, went **not due**, and was never retried — seven days of +silence on the real 168h cadence, invisible to both the R-88 breaker (defers only *due* tiers) and +the hub deadline monitor (reads the same freshness). It now counts only *plausibly complete* +entries via a measured 1 MiB floor; undecidable ⇒ not counted. + +**Size is the only tier-agnostic discriminator.** `verification` and `encrypted` are absent on every +local (dir) archive AND on a good PBS snapshot until `verify-new` catches up — gating on either +would have rejected 100% of local backups and produced fleet-wide backup thrash. That inverse risk +is a first-class test, red-proofed by making the filter reject everything. + +## Verified live, not just in unit tests +Campaign fault 2 was replayed against the fixed agent on demo-hp — phantom created, rejected and +announced once; the tier correctly reported DUE and backed up (4,359,968,099 B landed); and the +inverse showed **no thrash**, with 91 scheduler ticks as the positive observable that the loop was +alive rather than dead. + +## Settled along the way — no retention bug +Server-side prune does **not** count phantoms toward `keep-last`: a dry-run against three real +snapshots plus a phantom retained two real ones plus the phantom. The feared "two phantoms ⇒ zero +real backups" does not occur. Prune never removes them either, so they accumulate one per aborted +upload — filed as **R-99** (LOW, hygiene), not as a retention bug. + +## Docs touched +- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-2 → SHIPPED+PROVEN-LIVE; **R-99** filed; + **F-CRIT-1** filed as READY-HIGHEST (Campaign 8's other HIGH finding, untouched here). +- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — F-CRIT-2 row struck through and + a closing section added. + +## Still open, highest first +**F-CRIT-1** — an app that fails to restart after a quiesce never alarms, for two independent +reasons. Then fault 4 (restic transport interruption), which this fix makes more pointed: the +phantom question is now answered for PBS and still open for restic. diff --git a/documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md b/documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md index 86e20a5..f97d3e3 100644 --- a/documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md +++ b/documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md @@ -266,7 +266,7 @@ Neither restored guest was booted — deliberately, see F-OPS below. | # | Finding | Severity | DEFECT / ARTIFACT | |---|---|---|---| | **F-CRIT-1** | An app that fails to restart after a quiesce **never alarms** | **HIGH** | **DEFECT** (compression-independent) | -| **F-CRIT-2** | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** | **HIGH** | **DEFECT** (worse on real cadences) | +| ~~**F-CRIT-2**~~ | ~~A failed offsite backup leaves a phantom snapshot that **resets tier freshness**~~ — **FIXED, agent v0.106.0 (2026-07-28), re-tested live by replaying fault 2** | ~~HIGH~~ | DEFECT — closed | | **F-A1** | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker and pages the operator | MEDIUM | **DEFECT** in behaviour / ARTIFACT in frequency | | **F-HUB** | The hub **dropped an event** under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT (frequency compression-influenced) | | **F-LEAK** | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed and the 10-slot VMID band shrinks silently | MEDIUM | DEFECT (root-caused by fault 18) | @@ -336,6 +336,23 @@ Frequency is an ARTIFACT; the behaviour is a DEFECT — at real cadences a ~12-m against a daily backup collides roughly once per 420 guest-days, i.e. **every ~4 days on a 100-guest fleet**, forever training R-97a's alarm into noise. +### F-CRIT-2 — FIXED (agent v0.106.0, 2026-07-28) +`NewestArchiveTime` now counts only *plausibly complete* entries, via a measured size floor +(`minPlausibleArchiveBytes` = 1 MiB; smallest real backup on the fleet is 612,397,450 B, so 584x +headroom, asserted by a test). Undecidable ⇒ not counted. Size is the only tier-agnostic +discriminator: `verification` and `encrypted` are absent on every local dir archive and on a good +PBS snapshot until verify-new catches up, so gating on either would have rejected 100% of local +backups and caused fleet-wide backup thrash. + +Verified by replaying fault 2 against the fixed agent on demo-hp: the phantom was created, rejected, +and announced once; the tier was correctly reported DUE and backed up; and the inverse case showed +no re-backup thrash (with 91 scheduler ticks proving the loop was alive). Detail: +`felhom-agent/REPORT.md`. + +**Also settled:** server-side prune does NOT count phantoms toward `keep-last` — a dry-run retained +two real snapshots PLUS the phantom — so the feared retention/data-loss scenario does not occur. It +never removes them either; filed as R-99. + *(Full write-ups for every finding, with complete evidence, are in `~/campaign8/evidence/phaseB/`.)* --- diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 5addd8b..d451cbb 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -22,6 +22,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | — | demo-felhom's next weekly PBS backup (newest is 2026-07-26) | WATCHING | schedule | ~2026-08-02; also releases R-91 | CC | | — | demo-felhom's next restore-test (84 h cadence, last 2026-07-27 06:38 UTC) | WATCHING | schedule | ~2026-07-30 18:38 UTC | CC | | **R-97** | ~~Whole-guest backup tier had no hub signal; quiesce blamed the apps~~ | **SHIPPED** (controller v0.177.0 + hub v0.78.0/v0.79.0, 2026-07-27) | — | v0.79.0 (R-97c) replaced a FALSE operator-only comment with a real `operatorOnlyEvents` register | — | +| **F-CRIT-2** | ~~A failed offsite backup left a phantom snapshot (1 B, manifest-less, NEWEST) that RESET the tier's freshness clock — 7 days silent on the real 168h cadence, invisible to both the R-88 breaker and the hub deadline monitor~~ | **SHIPPED + PROVEN-LIVE** (agent v0.106.0, 2026-07-28) | — | `NewestArchiveTime` now counts only plausibly-complete entries (measured 1 MiB floor; undecidable ⇒ not counted). Campaign fault 2 replayed on demo-hp: phantom rejected + logged once, tier correctly DUE and backed up, and **no thrash** on the inverse | — | +| **R-99** | Server-side prune **never removes** a phantom snapshot. Confirmed it does NOT count them toward `keep-last` (dry-run kept 2 real + the phantom) so there is **no retention/data-loss bug** — but one accumulates per aborted upload, forever | READY (S) | — | Decide a cleanup path. Deletion on a **customer** datastore is a separate ruling — detection shipped, removal deliberately not automated | CC | +| **F-CRIT-1** | An app that **fails to restart** after a quiesce never alarms on any channel. Two independent causes: `restartAll` discards the error (returns nothing), and `StateStopped` is whitelisted at `main.go:1237` on invariant I1, which is false for the failed-restart path | **READY — HIGHEST** | — | Campaign 8's other HIGH finding, untouched by the F-CRIT-2 fix. Evidence: `audits/CAMPAIGN-8-backup-restore-2026-07-27.md` | CC | | **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC | | **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC | | **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |