docs: F-CRIT-2 shipped (agent v0.106.0) — OPEN-ITEMS, campaign doc, report

F-CRIT-2 marked SHIPPED + PROVEN-LIVE: NewestArchiveTime now counts only
plausibly-complete entries (measured 1 MiB floor), verified by replaying
campaign fault 2 on demo-hp in both directions.

Files two follow-ups:
- R-99: prune never REMOVES phantoms. It does NOT count them toward keep-last
  (dry-run kept 2 real + the phantom), so there is no retention/data-loss bug —
  but they accumulate one per aborted upload.
- F-CRIT-1: filed READY-HIGHEST. Campaign 8's other HIGH finding, untouched by
  this fix.
This commit is contained in:
2026-07-28 08:08:56 +02:00
parent 33a7cc51d3
commit d0cec9d3f3
3 changed files with 61 additions and 1 deletions
+40
View File
@@ -0,0 +1,40 @@
# REPORT — F-CRIT-2 fixed: a failed backup no longer looks like a fresh one (2026-07-28)
Scope: `felhom-agent` v0.105.0 → **v0.106.0**. Docs here. Implementation detail and the full live
re-test live in `felhom-agent/REPORT.md`; the campaign that found it is
`documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
## What changed
`NewestArchiveTime` counted an aborted PBS upload (1 byte, manifest-less, and NEWEST) as a
successful backup, so the tier read fresh, went **not due**, and was never retried — seven days of
silence on the real 168h cadence, invisible to both the R-88 breaker (defers only *due* tiers) and
the hub deadline monitor (reads the same freshness). It now counts only *plausibly complete*
entries via a measured 1 MiB floor; undecidable ⇒ not counted.
**Size is the only tier-agnostic discriminator.** `verification` and `encrypted` are absent on every
local (dir) archive AND on a good PBS snapshot until `verify-new` catches up — gating on either
would have rejected 100% of local backups and produced fleet-wide backup thrash. That inverse risk
is a first-class test, red-proofed by making the filter reject everything.
## Verified live, not just in unit tests
Campaign fault 2 was replayed against the fixed agent on demo-hp — phantom created, rejected and
announced once; the tier correctly reported DUE and backed up (4,359,968,099 B landed); and the
inverse showed **no thrash**, with 91 scheduler ticks as the positive observable that the loop was
alive rather than dead.
## Settled along the way — no retention bug
Server-side prune does **not** count phantoms toward `keep-last`: a dry-run against three real
snapshots plus a phantom retained two real ones plus the phantom. The feared "two phantoms ⇒ zero
real backups" does not occur. Prune never removes them either, so they accumulate one per aborted
upload — filed as **R-99** (LOW, hygiene), not as a retention bug.
## Docs touched
- `documentation/backlog/OPEN-ITEMS.md` — F-CRIT-2 → SHIPPED+PROVEN-LIVE; **R-99** filed;
**F-CRIT-1** filed as READY-HIGHEST (Campaign 8's other HIGH finding, untouched here).
- `documentation/audits/CAMPAIGN-8-backup-restore-2026-07-27.md` — F-CRIT-2 row struck through and
a closing section added.
## Still open, highest first
**F-CRIT-1** — an app that fails to restart after a quiesce never alarms, for two independent
reasons. Then fault 4 (restic transport interruption), which this fix makes more pointed: the
phantom question is now answered for PBS and still open for restic.
@@ -266,7 +266,7 @@ Neither restored guest was booted — deliberately, see F-OPS below.
| # | Finding | Severity | DEFECT / ARTIFACT |
|---|---|---|---|
| **F-CRIT-1** | An app that fails to restart after a quiesce **never alarms** | **HIGH** | **DEFECT** (compression-independent) |
| **F-CRIT-2** | A failed offsite backup leaves a phantom snapshot that **resets tier freshness** | **HIGH** | **DEFECT** (worse on real cadences) |
| ~~**F-CRIT-2**~~ | ~~A failed offsite backup leaves a phantom snapshot that **resets tier freshness**~~**FIXED, agent v0.106.0 (2026-07-28), re-tested live by replaying fault 2** | ~~HIGH~~ | DEFECT — closed |
| **F-A1** | A restore-test in progress makes a healthy backup report as FAILED, arms the breaker and pages the operator | MEDIUM | **DEFECT** in behaviour / ARTIFACT in frequency |
| **F-HUB** | The hub **dropped an event** under concurrent load (`SQLITE_BUSY`), no retry, cause unnamed | MEDIUM | DEFECT (frequency compression-influenced) |
| **F-LEAK** | A **failed** restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); leaks are never reclaimed and the 10-slot VMID band shrinks silently | MEDIUM | DEFECT (root-caused by fault 18) |
@@ -336,6 +336,23 @@ Frequency is an ARTIFACT; the behaviour is a DEFECT — at real cadences a ~12-m
against a daily backup collides roughly once per 420 guest-days, i.e. **every ~4 days on a
100-guest fleet**, forever training R-97a's alarm into noise.
### F-CRIT-2 — FIXED (agent v0.106.0, 2026-07-28)
`NewestArchiveTime` now counts only *plausibly complete* entries, via a measured size floor
(`minPlausibleArchiveBytes` = 1 MiB; smallest real backup on the fleet is 612,397,450 B, so 584x
headroom, asserted by a test). Undecidable ⇒ not counted. Size is the only tier-agnostic
discriminator: `verification` and `encrypted` are absent on every local dir archive and on a good
PBS snapshot until verify-new catches up, so gating on either would have rejected 100% of local
backups and caused fleet-wide backup thrash.
Verified by replaying fault 2 against the fixed agent on demo-hp: the phantom was created, rejected,
and announced once; the tier was correctly reported DUE and backed up; and the inverse case showed
no re-backup thrash (with 91 scheduler ticks proving the loop was alive). Detail:
`felhom-agent/REPORT.md`.
**Also settled:** server-side prune does NOT count phantoms toward `keep-last` — a dry-run retained
two real snapshots PLUS the phantom — so the feared retention/data-loss scenario does not occur. It
never removes them either; filed as R-99.
*(Full write-ups for every finding, with complete evidence, are in `~/campaign8/evidence/phaseB/`.)*
---
+3
View File
@@ -22,6 +22,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
| — | demo-felhom's next weekly PBS backup (newest is 2026-07-26) | WATCHING | schedule | ~2026-08-02; also releases R-91 | CC |
| — | demo-felhom's next restore-test (84 h cadence, last 2026-07-27 06:38 UTC) | WATCHING | schedule | ~2026-07-30 18:38 UTC | CC |
| **R-97** | ~~Whole-guest backup tier had no hub signal; quiesce blamed the apps~~ | **SHIPPED** (controller v0.177.0 + hub v0.78.0/v0.79.0, 2026-07-27) | — | v0.79.0 (R-97c) replaced a FALSE operator-only comment with a real `operatorOnlyEvents` register | — |
| **F-CRIT-2** | ~~A failed offsite backup left a phantom snapshot (1 B, manifest-less, NEWEST) that RESET the tier's freshness clock — 7 days silent on the real 168h cadence, invisible to both the R-88 breaker and the hub deadline monitor~~ | **SHIPPED + PROVEN-LIVE** (agent v0.106.0, 2026-07-28) | — | `NewestArchiveTime` now counts only plausibly-complete entries (measured 1 MiB floor; undecidable ⇒ not counted). Campaign fault 2 replayed on demo-hp: phantom rejected + logged once, tier correctly DUE and backed up, and **no thrash** on the inverse | — |
| **R-99** | Server-side prune **never removes** a phantom snapshot. Confirmed it does NOT count them toward `keep-last` (dry-run kept 2 real + the phantom) so there is **no retention/data-loss bug** — but one accumulates per aborted upload, forever | READY (S) | — | Decide a cleanup path. Deletion on a **customer** datastore is a separate ruling — detection shipped, removal deliberately not automated | CC |
| **F-CRIT-1** | An app that **fails to restart** after a quiesce never alarms on any channel. Two independent causes: `restartAll` discards the error (returns nothing), and `StateStopped` is whitelisted at `main.go:1237` on invariant I1, which is false for the failed-restart path | **READY — HIGHEST** | — | Campaign 8's other HIGH finding, untouched by the F-CRIT-2 fix. Evidence: `audits/CAMPAIGN-8-backup-restore-2026-07-27.md` | CC |
| **R-89** | Retention as a per-customer **commercial** policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
| **R-92** | Hub PBS-DR gauge is 0.1 GB-granular — small deltas unverifiable | READY (XS) | — | Widen precision when retention becomes customer-visible | CC |
| **R-93** | `drill-r50` is both a blocked customer and the only drift fixture | READY (XS) | — | Retire it for a synthetic fixture, or unblock + silence per-customer | CC |