docs: F-CRIT-2 shipped (agent v0.106.0) — OPEN-ITEMS, campaign doc, report
F-CRIT-2 marked SHIPPED + PROVEN-LIVE: NewestArchiveTime now counts only plausibly-complete entries (measured 1 MiB floor), verified by replaying campaign fault 2 on demo-hp in both directions. Files two follow-ups: - R-99: prune never REMOVES phantoms. It does NOT count them toward keep-last (dry-run kept 2 real + the phantom), so there is no retention/data-loss bug — but they accumulate one per aborted upload. - F-CRIT-1: filed READY-HIGHEST. Campaign 8's other HIGH finding, untouched by this fix.
This commit is contained in:
@@ -0,0 +1,40 @@
|
||||
# REPORT — F-CRIT-2 fixed: a failed backup no longer looks like a fresh one (2026-07-28)
|
||||
|
||||
Scope: `felhom-agent` v0.105.0 → **v0.106.0**. Docs here. Implementation detail and the full live
|
||||
re-test live in `felhom-agent/REPORT.md`; the campaign that found it is
|
||||
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
|
||||
|
||||
## What changed
|
||||
`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, and NEWEST) as a
|
||||
successful backup, so the tier read fresh, went **not due**, and was never retried — seven days of
|
||||
silence on the real 168h cadence, invisible to both the R-88 breaker (defers only *due* tiers) and
|
||||
the hub deadline monitor (reads the same freshness). It now counts only *plausibly complete*
|
||||
entries via a measured 1 MiB floor; undecidable ⇒ not counted.
|
||||
|
||||
**Size is the only tier-agnostic discriminator.** `verification` and `encrypted` are absent on every
|
||||
local (dir) archive AND on a good PBS snapshot until `verify-new` catches up — gating on either
|
||||
would have rejected 100% of local backups and produced fleet-wide backup thrash. That inverse risk
|
||||
is a first-class test, red-proofed by making the filter reject everything.
|
||||
|
||||
## Verified live, not just in unit tests
|
||||
Campaign fault 2 was replayed against the fixed agent on demo-hp — phantom created, rejected and
|
||||
announced once; the tier correctly reported DUE and backed up (4,359,968,099 B landed); and the
|
||||
inverse showed **no thrash**, with 91 scheduler ticks as the positive observable that the loop was
|
||||
alive rather than dead.
|
||||
|
||||
## Settled along the way — no retention bug
|
||||
Server-side prune does **not** count phantoms toward `keep-last`: a dry-run against three real
|
||||
snapshots plus a phantom retained two real ones plus the phantom. The feared "two phantoms ⇒ zero
|
||||
real backups" does not occur. Prune never removes them either, so they accumulate one per aborted
|
||||
upload — filed as **R-99** (LOW, hygiene), not as a retention bug.
|
||||
|
||||
## Docs touched
|
||||
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-2 → SHIPPED+PROVEN-LIVE; **R-99** filed;
|
||||
**F-CRIT-1** filed as READY-HIGHEST (Campaign 8's other HIGH finding, untouched here).
|
||||
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — F-CRIT-2 row struck through and
|
||||
a closing section added.
|
||||
|
||||
## Still open, highest first
|
||||
**F-CRIT-1** — an app that fails to restart after a quiesce never alarms, for two independent
|
||||
reasons. Then fault 4 (restic transport interruption), which this fix makes more pointed: the
|
||||
phantom question is now answered for PBS and still open for restic.
|
||||
@@ -266,7 +266,7 @@ Neither restored guest was booted — deliberately, see F-OPS below.
|
||||
| # | Finding | Severity | DEFECT / ARTIFACT |
|
||||
|---|---|---|---|
|
||||
| **F-CRIT-1** | An app that fails to restart after a quiesce **never alarms** | **HIGH** | **DEFECT** (compression-independent) |
|
||||
| **F-CRIT-2** | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** | **HIGH** | **DEFECT** (worse on real cadences) |
|
||||
| ~~**F-CRIT-2**~~ | ~~A failed offsite backup leaves a phantom snapshot that **resets tier freshness**~~ — **FIXED, agent v0.106.0 (2026-07-28), re-tested live by replaying fault 2** | ~~HIGH~~ | DEFECT — closed |
|
||||
| **F-A1** | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker and pages the operator | MEDIUM | **DEFECT** in behaviour / ARTIFACT in frequency |
|
||||
| **F-HUB** | The hub **dropped an event** under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT (frequency compression-influenced) |
|
||||
| **F-LEAK** | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed and the 10-slot VMID band shrinks silently | MEDIUM | DEFECT (root-caused by fault 18) |
|
||||
@@ -336,6 +336,23 @@ Frequency is an ARTIFACT; the behaviour is a DEFECT — at real cadences a ~12-m
|
||||
against a daily backup collides roughly once per 420 guest-days, i.e. **every ~4 days on a
|
||||
100-guest fleet**, forever training R-97a's alarm into noise.
|
||||
|
||||
### F-CRIT-2 — FIXED (agent v0.106.0, 2026-07-28)
|
||||
`NewestArchiveTime` now counts only *plausibly complete* entries, via a measured size floor
|
||||
(`minPlausibleArchiveBytes` = 1 MiB; smallest real backup on the fleet is 612,397,450 B, so 584x
|
||||
headroom, asserted by a test). Undecidable ⇒ not counted. Size is the only tier-agnostic
|
||||
discriminator: `verification` and `encrypted` are absent on every local dir archive and on a good
|
||||
PBS snapshot until verify-new catches up, so gating on either would have rejected 100% of local
|
||||
backups and caused fleet-wide backup thrash.
|
||||
|
||||
Verified by replaying fault 2 against the fixed agent on demo-hp: the phantom was created, rejected,
|
||||
and announced once; the tier was correctly reported DUE and backed up; and the inverse case showed
|
||||
no re-backup thrash (with 91 scheduler ticks proving the loop was alive). Detail:
|
||||
`felhom-agent/REPORT.md`.
|
||||
|
||||
**Also settled:** server-side prune does NOT count phantoms toward `keep-last` — a dry-run retained
|
||||
two real snapshots PLUS the phantom — so the feared retention/data-loss scenario does not occur. It
|
||||
never removes them either; filed as R-99.
|
||||
|
||||
*(Full write-ups for every finding, with complete evidence, are in `~/campaign8/evidence/phaseB/`.)*
|
||||
|
||||
---
|
||||
|
||||
@@ -22,6 +22,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| — | demo-felhom's next weekly PBS backup (newest is 2026-07-26) | WATCHING | schedule | ~2026-08-02; also releases R-91 | CC |
|
||||
| — | demo-felhom's next restore-test (84 h cadence, last 2026-07-27 06:38 UTC) | WATCHING | schedule | ~2026-07-30 18:38 UTC | CC |
|
||||
| **R-97** | ~~Whole-guest backup tier had no hub signal; quiesce blamed the apps~~ | **SHIPPED** (controller v0.177.0 + hub v0.78.0/v0.79.0, 2026-07-27) | — | v0.79.0 (R-97c) replaced a FALSE operator-only comment with a real `operatorOnlyEvents` register | — |
|
||||
| **F-CRIT-2** | ~~A failed offsite backup left a phantom snapshot (1 B, manifest-less, NEWEST) that RESET the tier's freshness clock — 7 days silent on the real 168h cadence, invisible to both the R-88 breaker and the hub deadline monitor~~ | **SHIPPED + PROVEN-LIVE** (agent v0.106.0, 2026-07-28) | — | `NewestArchiveTime` now counts only plausibly-complete entries (measured 1 MiB floor; undecidable ⇒ not counted). Campaign fault 2 replayed on demo-hp: phantom rejected + logged once, tier correctly DUE and backed up, and **no thrash** on the inverse | — |
|
||||
| **R-99** | Server-side prune **never removes** a phantom snapshot. Confirmed it does NOT count them toward `keep-last` (dry-run kept 2 real + the phantom) so there is **no retention/data-loss bug** — but one accumulates per aborted upload, forever | READY (S) | — | Decide a cleanup path. Deletion on a **customer** datastore is a separate ruling — detection shipped, removal deliberately not automated | CC |
|
||||
| **F-CRIT-1** | An app that **fails to restart** after a quiesce never alarms on any channel. Two independent causes: `restartAll` discards the error (returns nothing), and `StateStopped` is whitelisted at `main.go:1237` on invariant I1, which is false for the failed-restart path | **READY — HIGHEST** | — | Campaign 8's other HIGH finding, untouched by the F-CRIT-2 fix. Evidence: `audits/CAMPAIGN-8-backup-restore-2026-07-27.md` | CC |
|
||||
| **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
|
||||
| **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
|
||||
| **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |
|
||||
|
||||
Reference in New Issue
Block a user