NewestArchiveTime counted an aborted PBS upload (1 byte, no manifest, NEWEST)
as a successful backup, so the tier reported fresh, went not-due, and was never
retried. On the real 168h offsite cadence that is 7 days of silence, and neither
the R-88 breaker (defers only DUE tiers) nor the hub deadline monitor (reads the
same freshness) can catch it.
R-84's storage-as-ground-truth was right; the bug is that presence was taken for
validity. Now only plausibly-complete entries count, via a measured size floor
(minPlausibleArchiveBytes = 1 MiB). Undecidable => not counted.
Size is the only tier-agnostic discriminator: verification and encrypted are
absent on EVERY local dir archive (and on a good PBS snapshot until verify-new
catches up), so gating on either would reject 100% of local backups and cause
fleet-wide backup thrash. Floor measured against the fleet: smallest real backup
is 612,397,450 B, so 1 MiB leaves 584x headroom — asserted by a test.
Rejections are announced at WARN once per distinct volid, naming snapshot and
reason; per-poll logging would emit ~288 lines/day and bury the signal.
Four red-proofs, all observed failing.
The scheduler could only ever see cfg.Backup.BackupTarget(), so the offsite
tier's archives were never candidates — which is why demo-hp's DR tier reported
'applied' with zero snapshots for five days and nobody noticed.
Selection: oldest-first (operator ruling, Option 1). Never-proven sorts first,
which is where the offsite tier starts. Ties break on target id so ordering is
deterministic rather than following Go's randomised map order. Rotation credit
only on SUCCESS — a permanently failing tier must keep sorting first, not look
freshly proven and stop being retried.
- backup.RestoreTestState: persisted last-success per tier (atomic tmp+rename).
This genuinely needs persistence unlike R-84: R-84 had ground truth to consult
(the archive is still on the storage), whereas a restore-test destroys its
scratch and leaves no artifact. Corrupt/missing file -> 'nothing proven'.
- backup.InFlight: host-wide one-heavy-op gate shared with the local-API backup
path. A LINK concern, not a lock one — an offsite restore pulls multi-GB over
the same tunnel a backup pushes one, and at ~33 MB/min both drift toward
timeout, which is how a healthy tier gets recorded as failed. Callers DEFER,
never cancel.
- PickRestoreCandidateOn: newest archive on a named tier; '' is not an error, or
every fresh box looks broken for its first week.
- An empty tier is skipped and the next tried; it cannot starve, since it is
still least-recently-proven once it has an archive.
- POST /backup joins the gate (409 naming the holder).
Red-proofs A/E/F observed with the documented text. Full suite green (29
packages, rc=0).
SchedulerOptions.Spec was a VALUE produced by an immediately-invoked function
at daemon start, so storageTier() and restoreTaskTimeout() were evaluated once
and reused for every run for the process lifetime. Nothing tier-varying was
expressible (the offsite tier could never be scheduled), and it was a latent
staleness bug besides: a storage-type or config change did not take effect
until restart.
- backup.SpecBuilder: func(ctx, archive) RestoreTestSpec, called once per run.
The archive is passed because the tier MUST come from it (v0.100.0 rule) —
config-derived is what classified a PBS archive as 'local' and killed a
14.46 GB WAN restore at the 10-minute local bound.
- A nil spec builder SKIPS loudly instead of panicking: a wiring bug must cost a
restore-test, never the daemon goroutine.
Red-proof observed. Full suite green (29 packages, rc=0).
Observed live: three redundant local backups on demo-felhom in one afternoon of
deploys. The backup Store is in-memory ('lost on restart; the cadence
re-populates'), so after every restart /backup/due said 'no successful backup
recorded yet' and the controller took another one. On the offsite tier that is a
wasted multi-hour WAN upload after every agent deploy.
- BackupRunner.NewestArchiveTime: when a backup last LANDED on this tier's
storage, read from the storage.
- localapi.BackupArchiveLister (optional BackupService extension): the due-check
takes whichever is newer, the in-memory record or the storage.
Asking the storage rather than persisting the store is deliberate: it is ground
truth (a pruned archive correctly stops counting, where a persisted record would
keep claiming a backup that no longer exists), needs no new on-disk state, and
answers only 'when did a backup last land' — the richer fields stay with real
records so the host-report never carries invented numbers.
Fail-safes: read error -> fall back to memory (never fake freshness, never
suppress); genuinely empty -> due; old archive -> still due; service without the
lister -> unchanged.
Red-proof observed; full suite green (29 packages).
Ruling 1 (2 weeks of weekly offsite backups): localPruneSpec's blanket PBS
refusal is now scoped — an ADDITIONAL tier with an explicit keep_last may
prune its PBS target. The refusal still applies in full to the PRIMARY tier,
because BackupTarget() defaults to felhom-pbs and KeepLast() defaults to 3, so
a box with neither key set would silently prune its offsite DR to 3 restore
points. An additional tier cannot have that accident (keep_last defaults to 0).
Ruling 3 (first backup runs as long as needed; nothing else starts until done):
- additional-tier wait bound 6h -> 12h (measured ~33 MB/min => ~5h for a first
full 10 GB snapshot; 12h gives margin but stays bounded so a hung task still
surfaces)
- ONE BACKUP AT A TIME PER GUEST across all tiers: POST /backup returns 409
when a DIFFERENT tier is in flight, naming the busy tier, with NO data object
so nothing is parseable as the caller's own job. Same tier still returns that
job (202, unchanged).
- snapshotted now counts as in-flight, not just running — after the snapshot the
vzdump is still uploading and holding the lock. The old check left a window
where a second POST started a real second vzdump. Latent bug, closed.
Full suite green (29 packages); red-proof observed and restored.
Found by live validation on demo-felhom, not by review.
The first real PBS-targeted backup ran past the runner's hard-coded 30-minute
WaitTask bound. The agent stopped waiting and recorded success=false WHILE THE
VZDUMP KEPT RUNNING (still running 72 min later, 2.4 GB uploaded). Consequences:
the tier stays permanently due, the next attempt collides with the guest lock
the live vzdump holds, and the hub sees a DR tier that never succeeds — R-82's
'applied and empty' fault re-created by a timeout.
Measured: ~33 MB/min over wg to Hetzner, so a first FULL ~10 GB snapshot
projects to ~5h.
- BackupTargetConfig.WaitTimeoutSeconds: per-tier bound. Primary 30m UNCHANGED
(a local vzdump hanging 30m IS a real fault); additional tier 6h, sized from
the measurement.
- backup.NewBackupRunnerWithWait: per-instance (per-tier) bound.
NewBackupRunner keeps its signature, so restore-test/selftest are untouched.
- localapi.BackupTier.WaitTimeout: the fire-and-forget context is sized from the
tier, not a fixed 2h. BOTH bounds had to move — a 6h runner bound under a 2h
outer context reproduces the same false failure four hours later.
Same direction as restore_test_pbs_restore_timeout_seconds: when in doubt wait
LONGER. A slow backup is a slow backup; a false timeout is a corrupt status
plus lock contention.
Red-proof observed and restored; full suite green.
The restore-test had GL-5 finding #2's mirror image: its live-source-config
bind-override path tripped PVE's drop-unlisted-mountpoints rule, so scratch
guests boot-verified WITHOUT their storage mpN - weaker verification than
claimed. Params now derive from the ARCHIVE's own embedded config via
ExtractArchiveConfig + drRestoreOverrides (the object under test; full
layout, content genuinely extracted - the added runtime IS the
verification); unreadable/unknown-topology archives refuse up front. NEW
mount-parity assert (2b, pre-start): restored mpN set vs the archive's -
missing/mispathed/undersized/extra mpN fail the test naming the delta, so
constraint (b) can never regress into a green light. MountParity +
MountInventory ride the result + hub wire record (additive). Dead
bindMountOverrides/archiveVMID path deleted with its tests (no reachable
lookalike). DR bring-up untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The preventive counterpart to host_disk + storage_fill detectors: the periodic local
whole-guest vzdump now prunes its own old archives (keep-last=3, clamped >=1) so a box
can't refill its own root via its own backups. Local target only — PBS never pruned
(resolved via ListStorage; fail-safe skip on unknown). Seeded in host-install.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
BackupRunner.BackupWithSnapshotHook tails the task log for the 'create storage
snapshot' marker (snapshot mode only) and fires onSnapshot once; localapi flips
/backup/status to 'snapshotted' before 'done' so the controller resumes early.
Phase 0 validated on PVE 9.2.2: marker confirmed, downtime ~24s->~1s (934MB).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes the crying-wolf false-fail surfaced by the live hub-enrollment runbook:
PVE's guest-start task exits "WARNINGS: 1" for the benign systemd-nesting
advisory, and WaitTask treated any non-OK exitstatus as failure, so the verdict
was decided by an advisory exit code before the real boot check ran. Every
modern-distro restore-test reported pass:false.
- proxmox.WaitOptions.AllowWarnings (opt-in; default keeps all callers strict)
- restore-test start step accepts warnings, surfaces them, verdict stays waitRunning
- RestoreTestResult.StartWarnings/.WarningsRecognized + version-free "enable
nesting" recognizer (can't rot back at systemd 258+); GuestAPI.TaskLogTail
- hub.RestoreTest.warnings/.warnings_recognized wire fields (consumed by hub v0.7.5)
- scheduler logs clean / passed-with-recognized / passed-with-unrecognized warnings
- tests: WaitTask warnings matrix; restore-test pass/fail-on-liveness; version-free
regression guard (systemd 256-300)
Single agent bump 0.6.0 -> 0.7.0 covering the agent half of both task phases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spike-proven that backup/restore-to-PBS reuse Phase A unchanged; the only new code is
the verify capability, a small PBS-API client, and PBSSnapshot reporting.
- internal/pbs: fingerprint-pinned, token-authed PBS-API client (Verify/Snapshots/
TaskStatus, node-from-UPID; secret read from /etc/pve/priv/storage/<id>.pw at runtime,
never logged) + the verify maintenance loop (own cadence, default 6h, NOT gated/journaled,
like the watchdog) + SnapshotStore.
- hub: PBSSnapshot filled (namespace/type/id/time/size/owner/protected/encrypted/
verify_state/verify_upid); PBSReporter collector seam; cross-repo golden + bidirectional
key-set tests; hub handler parses pbs_snapshots + logs a failed-verify WARN.
- backup: report the ACTUAL vzdump mode (parsed from the task log; PVE may downgrade
snapshot->stop). proxmox.Storage.Username. config PBSVerifyCadence/secret-dir.
--selftest=pbs-verify. Backup/restore-to-PBS unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin
Claude Code