F-CRIT-2: a failed backup must not look like a fresh one (v0.106.0)
NewestArchiveTime counted an aborted PBS upload (1 byte, no manifest, NEWEST) as a successful backup, so the tier reported fresh, went not-due, and was never retried. On the real 168h offsite cadence that is 7 days of silence, and neither the R-88 breaker (defers only DUE tiers) nor the hub deadline monitor (reads the same freshness) can catch it. R-84's storage-as-ground-truth was right; the bug is that presence was taken for validity. Now only plausibly-complete entries count, via a measured size floor (minPlausibleArchiveBytes = 1 MiB). Undecidable => not counted. Size is the only tier-agnostic discriminator: verification and encrypted are absent on EVERY local dir archive (and on a good PBS snapshot until verify-new catches up), so gating on either would reject 100% of local backups and cause fleet-wide backup thrash. Floor measured against the fleet: smallest real backup is 612,397,450 B, so 1 MiB leaves 584x headroom — asserted by a test. Rejections are announced at WARN once per distinct volid, naming snapshot and reason; per-poll logging would emit ~288 lines/day and bury the signal. Four red-proofs, all observed failing.
This commit is contained in:
@@ -1,5 +1,53 @@
|
||||
# felhom-agent — Changelog
|
||||
|
||||
## v0.106.0 — F-CRIT-2: a failed backup must not look like a fresh one (2026-07-28)
|
||||
|
||||
Campaign 8 killed the PBS daemon mid-upload. PBS published the aborted upload into the storage
|
||||
listing anyway — **1 byte, no manifest, and the NEWEST entry**. `NewestArchiveTime` counted it, so
|
||||
the tier reported freshly backed up, went **not due**, and was never attempted again. On the real
|
||||
168h offsite cadence that is **seven days of silence**, and neither backstop helps: the R-88 breaker
|
||||
only defers tiers that are *due*, and the hub's deadline monitor reads the same freshness.
|
||||
|
||||
R-84 replaced remembered state with *"ask the storage, it is ground truth"*. That was right. The bug
|
||||
is that **presence was taken for validity** — and a phantom is a more convincing lie than an empty
|
||||
array, because absence at least reads as absence.
|
||||
|
||||
**The fix.** `NewestArchiveTime` now counts only entries that are *plausibly complete*, via a
|
||||
measured size floor (`minPlausibleArchiveBytes` = 1 MiB). Undecidable ⇒ **not** counted: erring
|
||||
toward "less fresh" costs one extra backup, whereas counting an undecidable entry is the defect.
|
||||
|
||||
**Why size and nothing else.** This runner is tier-agnostic — the same predicate runs against PBS
|
||||
and against a plain `dir` storage. Verified against the live PVE API (2026-07-28):
|
||||
|
||||
| candidate | phantom | good PBS | good **local** | verdict |
|
||||
|---|---|---|---|---|
|
||||
| `verification` | absent | present | **absent** | unusable — would reject every local backup |
|
||||
| `encrypted` | absent | present | **absent** | unusable — same |
|
||||
| `notes` | absent | present | present | too fragile (agent-set only) |
|
||||
| `size` | **1 B** | 4.35 GB | 1.59 GB | **robust, tier-agnostic** |
|
||||
|
||||
The floor is measured, not chosen: the smallest real backup anywhere on the fleet is **612,397,450 B**
|
||||
(~584 MiB); 1 MiB sits 584x below it and 1,048,576x above the phantom. A test asserts that headroom
|
||||
so nobody can quietly raise the floor into the thrash zone.
|
||||
|
||||
**The opposite risk is real and is a first-class test.** A filter that is too aggressive does not
|
||||
merely lose safety — the tier reports absent every poll, backs up every cycle, and **R-88 cannot
|
||||
save it because those backups succeed**. That is a continuous multi-GB write loop across the fleet.
|
||||
`TestNewestArchiveTime_ValidSnapshotsAreStillCounted` is that guard, red-proofed by making the
|
||||
filter reject everything.
|
||||
|
||||
**A rejected archive is never silent.** WARN, once per distinct volid (not per 5-minute poll —
|
||||
~288 lines/day would bury it), naming the snapshot and the reason.
|
||||
|
||||
Also established while investigating, and worth recording: **server-side prune does NOT count a
|
||||
phantom toward `keep-last`** — a dry-run with `keep-last 2` against three real snapshots plus a
|
||||
phantom retained *two real ones plus the phantom*. So there is **no retention/data-loss bug**. But
|
||||
prune never removes phantoms either, so they accumulate one per aborted upload, forever.
|
||||
|
||||
Files: `internal/backup/runner.go`, `internal/backup/archive_completeness_test.go` (new).
|
||||
No wire/contract change; the controller needs no change — it consumes `age_state` and now simply
|
||||
receives the truth.
|
||||
|
||||
## v0.105.0 — R-88 Part 2: the agent can finally say "unknown" (2026-07-27)
|
||||
|
||||
`newestArchiveOn` promised, in its own doc comment, that *"errors and unsupported services degrade to
|
||||
|
||||
Reference in New Issue
Block a user