F-CRIT-2: a failed backup must not look like a fresh one (v0.106.0)

NewestArchiveTime counted an aborted PBS upload (1 byte, no manifest, NEWEST)
as a successful backup, so the tier reported fresh, went not-due, and was never
retried. On the real 168h offsite cadence that is 7 days of silence, and neither
the R-88 breaker (defers only DUE tiers) nor the hub deadline monitor (reads the
same freshness) can catch it.

R-84's storage-as-ground-truth was right; the bug is that presence was taken for
validity. Now only plausibly-complete entries count, via a measured size floor
(minPlausibleArchiveBytes = 1 MiB). Undecidable => not counted.

Size is the only tier-agnostic discriminator: verification and encrypted are
absent on EVERY local dir archive (and on a good PBS snapshot until verify-new
catches up), so gating on either would reject 100% of local backups and cause
fleet-wide backup thrash. Floor measured against the fleet: smallest real backup
is 612,397,450 B, so 1 MiB leaves 584x headroom — asserted by a test.

Rejections are announced at WARN once per distinct volid, naming snapshot and
reason; per-poll logging would emit ~288 lines/day and bury the signal.

Four red-proofs, all observed failing.

internal/backup/runner.go

@@ -6,6 +6,7 @@ import (
 	"log/slog"
 	"sort"
 	"strings"
+	"sync"
 	"time"
 
 	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
@@ -53,7 +54,13 @@ type BackupRunner struct {
 	// there is a deliberate act.
 	allowPBSPrune bool
 	logger        *slog.Logger
-	now         func() time.Time
+	now           func() time.Time
+	// rejected remembers the volids already announced by warnRejectedArchiveOnce, so an incomplete
+	// archive is reported ONCE rather than on every 5-minute due-check. Bounded in practice: one
+	// entry per aborted upload, and a process restart clears it. Guarded by rejectedMu because the
+	// due-check is served from the local-API handler goroutines.
+	rejectedMu sync.Mutex
+	rejected   map[string]struct{}
 }
 
 // NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
@@ -297,6 +304,75 @@ func (r *BackupRunner) latestArchive(ctx context.Context, vmid int) (string, int
 // reported "no successful backup recorded yet" and the controller dutifully took another one. On the
 // local tier that is wasted minutes; on the OFFSITE tier it is a wasted multi-hour WAN upload after
 // every agent deploy — and agent deploys are routine. Three redundant local backups were observed on
+// minPlausibleArchiveBytes is the floor below which a storage entry cannot be a real whole-guest
+// backup and is therefore treated as an INCOMPLETE artefact rather than a successful one.
+//
+// MEASURED, not chosen by feel — fleet survey 2026-07-28 (Campaign 8, finding F-CRIT-2):
+//
+//	smallest REAL backup anywhere on the fleet ...  612,397,450 B  (~584 MiB, a guest-9100 vzdump)
+//	demo-hp      local / PBS ..................... 1.59 GB / 4.35-4.37 GB
+//	demo-felhom  local / PBS ..................... 5.82-5.84 GB / 14.47-14.51 GB
+//	the phantom left by a PBS daemon killed mid-upload ....... 1 B
+//
+// 1 MiB sits 584x below the smallest real backup and 1,048,576x above the phantom. The two
+// populations are nine orders of magnitude apart, so this floor cannot plausibly clip a real
+// archive — which is the property that matters, because a floor set too HIGH does not merely lose
+// safety margin, it causes fleet-wide backup THRASH (see archivePlausiblyComplete).
+const minPlausibleArchiveBytes int64 = 1 << 20
+
+// archivePlausiblyComplete reports whether a storage entry can be a COMPLETE backup, and if not,
+// why. Pure, so the contract is unit-testable without a storage.
+//
+// WHY SIZE, AND NOTHING ELSE. The richer PBS fields look like better discriminators and are all
+// traps, because this runner is TIER-AGNOSTIC — the same predicate runs against a PBS datastore and
+// against a plain `dir` storage (verified against the live PVE API, 2026-07-28):
+//
+//   - `verification` is absent on the phantom, but ALSO absent on every local (dir) archive — a dir
+//     storage has no verification concept — and absent on a good PBS snapshot until verify-new
+//     catches up. Gating on it would reject 100% of local backups and every freshly-taken offsite
+//     one: continuous re-backup across the fleet.
+//   - `encrypted` fails the same way, and for the same reason.
+//   - `notes` happens to be present on both good tiers today only because the agent sets it; an
+//     archive written by any other path lacks it. Too fragile to gate freshness on.
+//
+// Size is the only signal that means the same thing on every tier.
+//
+// THE FAIL-SAFE DIRECTION, stated explicitly: when completeness cannot be established the entry is
+// NOT counted as a successful backup. That errs toward the tier looking LESS fresh, and its worst
+// case is one extra backup. Counting an undecidable entry is precisely the F-CRIT-2 defect — a
+// failed upload that made its tier look freshly backed up and silenced it for a full cadence.
+func archivePlausiblyComplete(e proxmox.StorageContent) (bool, string) {
+	if e.Size < minPlausibleArchiveBytes {
+		return false, fmt.Sprintf("size %d B is below the %d B plausibility floor — an aborted/incomplete archive, not a successful backup",
+			e.Size, minPlausibleArchiveBytes)
+	}
+	return true, ""
+}
+
+// warnRejectedArchiveOnce announces a rejected archive at WARN exactly once per distinct volid.
+//
+// A rejected archive must never be silent: a tier that quietly ignores the newest entry on its
+// storage is a new quiet path, and quiet paths are what F-CRIT-2 was. But the due-check runs every
+// 5 minutes and a phantom persists indefinitely — server-side prune does NOT collect it (verified
+// by dry-run 2026-07-28: with keep-last 2 it retained two real snapshots PLUS the phantom) — so
+// logging per poll would emit ~288 identical lines a day and bury the one that matters.
+func (r *BackupRunner) warnRejectedArchiveOnce(e proxmox.StorageContent, why string) {
+	r.rejectedMu.Lock()
+	if r.rejected == nil {
+		r.rejected = map[string]struct{}{}
+	}
+	_, seen := r.rejected[e.VolID]
+	if !seen {
+		r.rejected[e.VolID] = struct{}{}
+	}
+	r.rejectedMu.Unlock()
+	if seen {
+		return
+	}
+	r.logger.Warn("backup: ignoring an INCOMPLETE archive when computing tier freshness — it is not a successful backup",
+		"target", r.target, "vmid", e.VMID, "volid", e.VolID, "size_bytes", e.Size, "reason", why)
+}
+
 // demo-felhom in a single afternoon of deploys (2026-07-26).
 //
 // Asking the STORAGE rather than persisting the store is deliberate:
@@ -305,7 +381,10 @@ func (r *BackupRunner) latestArchive(ctx context.Context, vmid int) (string, int
 //   - it needs no new on-disk state and no migration;
 //   - it is the same source `latestArchive` already trusts to build the post-backup record.
 //
-// It answers ONLY "when did a backup last land", which is exactly what the due-check needs. The
+// It answers ONLY "when did a plausibly COMPLETE backup last land", which is exactly what the
+// due-check needs. Completeness is not optional here: PBS publishes an aborted upload into the same
+// listing (manifest-less, 1 byte, and NEWEST), and counting it made the tier report fresh and go
+// silent for a whole cadence — F-CRIT-2. Presence is not validity. The
 // richer fields (size, duration, uncovered volumes, error) stay with the real in-memory records — a
 // synthesized record would put invented numbers into the host-report.
 func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Time, bool, error) {
@@ -315,7 +394,14 @@ func (r *BackupRunner) NewestArchiveTime(ctx context.Context, vmid int) (time.Ti
 	}
 	var best int64 = -1
 	for _, e := range contents {
-		if e.Content == "backup" && e.VMID == vmid && e.CTime > best {
+		if e.Content != "backup" || e.VMID != vmid {
+			continue
+		}
+		if ok, why := archivePlausiblyComplete(e); !ok {
+			r.warnRejectedArchiveOnce(e, why)
+			continue
+		}
+		if e.CTime > best {
 			best = e.CTime
 		}
 	}