REPORT — R-82: per-target backup tiers, agent v0.97.0 → v0.102.0 (2026-07-26)

Overwritten per the standing rule. Live on demo-felhom and demo-hp. Full cross-repo arc + the Phase-0 gates: felhom.eu/REPORT.md.

Versions in this arc


v0.97.0	Slice A — per-target cadence/retention, `/backup/tiers`, per-tier due + runner + job keys
v0.98.0	the 30-minute false failure — per-tier vzdump wait bound
v0.99.0	operator rulings — 2-week offsite retention (scoped PBS prune opt-in) + one backup at a time per guest
v0.100.0	the restore tier comes from the ARCHIVE, not the configured target
v0.101.0	a leaked restore-test scratch can no longer auto-start (`onboot=0` at restore time)
v0.102.0	an unprovisioned tier DEFERS instead of firing at a missing storage
v0.103.0	R-84 — an agent restart no longer triggers a redundant backup

Four defects found by RUNNING it, not reviewing it

v0.98.0 — a 41-minute PBS backup was recorded success:false at 30 minutes while it was still running, and it later completed TASK OK. Worse than "the backup didn't happen": the tier stays permanently due and the retry collides with the guest lock.
v0.100.0 — a felhom-pbs: archive was classified local and got the 10-minute bound against a 14.46 GB WAN restore, failing at 600 s. A silent regression of the S4.1 fix — the mechanism was never removed; its input changed when local_backup_target was retargeted to local. A fix keyed on "the configured target" stops holding the moment more than one target exists.
v0.101.0 — a leaked scratch guest kept onboot: 1, so a host reboot would have started a clone of the live guest. onboot=0 is now set at restore time, because "after" is the path that leaks.
v0.102.0 — a tier fires at a storage that does not exist yet (a fresh box before DR provisioning). Now deferred, going live with no restart once the storage appears.
v0.103.0 (R-84) — three redundant local backups ran on demo-felhom in one afternoon of deploys, because the in-memory Store is empty after a restart and the due-check read that as "no backup ever". On the offsite tier that is a wasted multi-hour WAN upload after every agent deploy. The due-check now consults the STORAGE (ground truth: a pruned archive correctly stops counting, where a persisted record would keep claiming a backup that no longer exists).

The frozen contract

Untargeted /backup/due, POST /backup and /backup/status keep the primary tier and the pre-R-82 response bytes (Target is omitempty and stays empty); the primary's job-id format is unchanged. Verified live on demo-felhom:

/backup/due (untargeted)      {"vmid":9201,"due":true,"reason":"no successful backup recorded yet"}
/backup/due?target=felhom-pbs {...,"target":"felhom-pbs"}
/backup/due?target=bogus      400 "unknown backup target: bogus"

An unknown ?target= is a 400, never a silent fallback — a caller asking about a tier this agent does not serve must find out, not act on another tier's freshness.

Fail-safe directions, stated once

Unparseable timestamp → due (a spurious backup is cheap; a skipped one is not).
Storage-view error → present ("I could not check" is not "not there" — reading it that way would silently suppress backups, the rule this project has relearned three times).
Missing tier cadence → rejected, logged at ERROR (never defaulted: a weekly DR tier silently running daily would fill the datastore).
Unset keep_last → never prune (a DR tier must not start pruning itself because someone forgot a field). The primary tier keeps the absolute PBS-prune refusal, because its target and retention both default and could prune the DR by accident.

Live validation (demo-felhom)

Both tiers armed: local 24h keep_last=3 wait 30m + felhom-pbs 168h keep_last=2 wait 12h prune_pbs_allowed=true. First real PBS-targeted backup: TASK OK, 41 minutes, 14.46 GB snapshot, and it restored cleanly (vzrestore: stopped OK). Measured incremental cost: +2.7 GB on disk for a 14.46 GB logical snapshot (~81 % dedup).

demo-hp — the box that had ZERO offsite snapshots, ever: deployed via the documented break-glass path (binary + config backed up first), tier armed, and its first ever PBS backup landed — felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z, 4.25 GB, into a namespace that was verifiably empty beforehand. That is the R-82 finding closed on the box where it was worst.

R-84 proven live on both boxes — immediately after a restart, with the in-memory store cold, the due-check reports real ages read from the storage instead of "no successful backup recorded yet":

demo-felhom  local age=5298s   felhom-pbs age=14373s   both due:false
demo-hp      local age=498s    felhom-pbs age=2346s    both due:false

Before v0.103.0 every one of those would have said "no successful backup recorded yet" and triggered a redundant backup — on the offsite tier, a multi-hour upload.

The restore round-trip — PASSED (demo-hp, 2026-07-26)

source_archive : felhom-pbs:backup/ct/9201/2026-07-26T15:42:42Z
source_tier    : pbs        <- the v0.100.0 fix; the earlier attempt said "local" and died at 600s
pass           : true
verified       : boot+running
mount_parity   : ok         <- mp0=/var/lib/docker 50G, mp1=/mnt/sys_drive 20G, mp8/mp9 stand-ins
duration       : 4m5s       (restore + boot + verify + teardown)

mount_parity is the non-hollow half — a boot-only verify cannot see a missing data volume. The scratch tore down cleanly (no 403, no leak), confirming that the teardown 403 was a phantom (a short-timeout consequence, as 06-offsite-connectivity.md already recorded) and correcting my earlier framing of it as a standing privilege gap.

Tests

go build ./... && go vet ./... && go test ./... — rc=0, 29 packages, vet run unpiped.

Red-proofs observed and restored: old-controller compat ("target":"local" leaking into the untargeted body), the per-tier wait bound, and the onboot override. A per-tier single-flight defect was caught by its own test before it ever shipped — job ids were unique only by clock luck, so the PBS request was handed the local job's id.

Process failure recorded: I ran the suite and committed in the same command, read packages ok: 28 and pushed without checking rc=1; five of my own tests were failing (a harness artifact, fixed in the follow-up). That is the exact exit-code trap recorded twice earlier in this arc.

NOT done

The scheduled restore-test still only ever selects the primary tier, so the offsite tier is never automatically restore-tested — arguably the more important half of "is the DR tier real?".
The agent does not report per-tier cadences, so the hub must infer "PBS ⇒ weekly" from storage type.
The unattended offsite restore-test — the scheduled path still only selects the primary tier. (The manual/selftest round-trip is now PROVEN: pass:true, verified:"boot+running", mount_parity:"ok", source_tier:"pbs", 4m5s, clean teardown on demo-hp.)

7.2 KiB Raw Blame History