v0.97.0 — R-82 Slice A: per-target backup tiers (local daily + PBS weekly)

Mechanism only. No box changes behaviour until a backup_targets entry is
added to its config (Slice D); an untouched config resolves to exactly one
tier and behaves byte-identically to v0.96.0.

- config: BackupTargetConfig + ExtraTargets + BackupTiers(); each tier carries
  its OWN cadence and retention (keep-last=3 is three days on a daily tier and
  three weeks on a weekly one). A missing cadence is REJECTED, not defaulted —
  a weekly DR tier silently running daily would fill the 37.2 GB datastore.
  main.go logs every rejection at ERROR.
- /backup/due?target= judges a tier against its OWN newest successful backup.
  Without that filter a fresh local backup satisfies the weekly PBS cadence and
  the DR tier never runs — today's bug, re-created in code.
- GET /backup/tiers advertises the tiers; a 404 is the controller's pre-R-82
  capability probe (Slice B).
- Jobs keyed by (vmid,target): single-flight is per tier, which is what lets
  the weekly night run both backups in ONE quiesce window. Job ids are unique
  per tier by construction, not by clock luck.
- One runner per tier: the runner holds target+retention as immutable state,
  so parameterising one runner would risk pairing tier A's target with tier B's
  retention.

COMPATIBILITY (frozen): untargeted /backup/due, POST /backup and
/backup/status keep the primary tier and the pre-R-82 response BYTES —
Target is omitempty and stays empty. The primary's job-id format is unchanged.

NOT changed: the local tier; PBS is still never pruned by the per-run flag
(keep_last defaults to 0 = never prune — enabling DR pruning is irreversible
and needs an operator ruling).

Tests 748->768. Red-proof #1 observed and restored.
Phase 0: felhom.eu/documentation/audits/SPIKE-r82-phase0-2026-07-26.md
This commit is contained in:
Claude Code
2026-07-26 12:20:58 +02:00
parent dfd5d731ee
commit 739b3c3b58
8 changed files with 980 additions and 36 deletions
+73
View File
@@ -1,3 +1,76 @@
## v0.97.0 — R-82 Slice A: per-target backup tiers (local daily + PBS weekly) (2026-07-26)
Additive; **MinAgent floor rises** for the multi-tier contract (a controller that wants per-tier
backups needs this agent — but see the compatibility rule: an OLD controller is unaffected).
Slice A of R-82. `BackupTarget()` returned ONE string and `BackupCadence()` ONE 24h window, so
"local daily **and** PBS weekly" was not expressible at all — which is why the DR promise is
currently unbacked (demo-felhom holds one PBS snapshot from 2026-07-18, demo-hp zero, ever).
Phase-0 gate results: `felhom.eu/documentation/audits/SPIKE-r82-phase0-2026-07-26.md`.
**This slice ships the mechanism only. No box's behaviour changes until a `backup_targets` entry is
added to its config (Slice D).** An untouched config resolves to exactly one tier and behaves
byte-identically to v0.96.0.
### The compatibility rule (load-bearing)
The agent and controller deploy independently, so the untargeted local-API contract is **frozen**:
- `GET /backup/due` with **no** `?target=` → the PRIMARY tier, same cadence, same response **bytes**.
`BackupDueResponse.Target` is `omitempty` and left EMPTY for untargeted requests, so an old
controller cannot tell this agent from the old one. Pinned by a red-proofed test.
- Same for `POST /backup` and `GET /backup/status`.
- The primary tier's **job-id format is unchanged**; only additive tiers carry a target segment.
### Added
- **`config.BackupTargetConfig` + `BackupConfig.ExtraTargets`** (`backup_targets`) — each tier
carries its OWN cadence and its OWN retention. Those are semantically different per tier:
`keep-last=3` is three DAYS on a daily tier and three WEEKS on a weekly one, so sharing one knob
guarantees one of them is wrong.
- **`BackupConfig.BackupTiers() ([]BackupTier, []string)`** — resolves the tier list, primary first,
plus warnings the caller MUST log. A tier is **rejected, not defaulted**, when its cadence is
missing: silently defaulting a weekly DR tier to the 24h local default would fill the 37.2 GB
datastore. Empty/duplicate targets are rejected too. `main.go` logs every rejection at **ERROR**
a silently dropped backup tier is the "applied and empty" fault this task exists to fix.
- **`GET /backup/tiers`** — advertises the tiers, primary first. This is the controller's capability
probe: a **404 means a pre-R-82 agent**, and Slice B falls back to single-tier on it.
- **`localapi.BackupTier` + `normalizeBackupTiers`** — nil tiers synthesize the legacy single tier,
so every existing caller and test hits the pre-R-82 path untouched. A tier with a nil runner is
DROPPED rather than advertised (advertising one would be an applied-and-empty tier).
### Changed
- **`/backup/due?target=` judges that tier against ITS OWN newest successful backup**
(`latestSuccessfulBackupForTarget`). Without this filter a fresh daily local backup would satisfy
the weekly PBS cadence and the DR tier would never run — today's bug, re-created in code. The
store was already keyed by target, so this is a lookup change, not a data-model change.
- **Backup jobs are keyed by (vmid, target), not vmid.** Single-flight is now PER TIER, which is
what lets the weekly night run both backups inside ONE quiesce window (Slice B). Keying by vmid
alone handed the second caller the first tier's job id — **caught by its own test**, and it would
have made the controller believe a PBS backup ran when only the local one had.
- **Job ids are unique per tier by construction**, not by clock luck (two tiers can start in the
same nanosecond). The primary keeps the old format; additive tiers carry the target segment.
- **One runner per tier** (`main.go`). The runner holds target/mode/notes/retention as immutable
construction state and `localPruneSpec` reads that retention — parameterising a single runner by
target would risk pairing tier A's target with tier B's retention.
- An unknown `?target=` is a **400, never a silent fallback to the primary**. A controller asking
about a tier this agent does not serve must find out, not act on another tier's freshness.
### NOT changed (deliberate)
- **The local tier.** Same target, same 24h cadence, same keep-last=3 clamp. It is the only honest
tier today and this slice does not touch it.
- **PBS is still never pruned by the per-run `--prune-backups` flag** (`localPruneSpec`). Per-tier
retention is plumbed and a tier's `keep_last` defaults to 0 = never prune, but turning on
automatic pruning of the DR datastore is irreversible and needs an operator ruling — R-82 Phase 0
already flagged retention as needing one. Recorded, not decided here.
- The fail-safe-toward-due rule on an unparseable timestamp. A spurious backup is cheap; a skipped
one is not.
### Tests
768 (was 748), +20 across `internal/localapi/backup_tiers_test.go` and
`internal/config/backup_tiers_test.go`. Red-proof #1 (old controller ↔ new agent) observed:
removing the untargeted compat branch makes the untargeted body carry `"target":"local"` and the
test fails on it. Restored.
## v0.96.0 — R-50 island NIC: provision attaches the guest's island net1 (2026-07-25) ## v0.96.0 — R-50 island NIC: provision attaches the guest's island net1 (2026-07-25)
Additive; **MinAgent unchanged** (no controller coupling — the controller dials whatever `bootstrap.json` Additive; **MinAgent unchanged** (no controller coupling — the controller dials whatever `bootstrap.json`
+1
View File
@@ -144,6 +144,7 @@
| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). | | `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go | | `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go | | `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go | | `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go | | `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls | | `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls |
+31 -1
View File
@@ -1269,7 +1269,36 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
} }
// v0.48.0: ride the served leaf fp on every host report so the hub can detect a re-key fleet-wide. // v0.48.0: ride the served leaf fp on every host report so the hub can detect a re-key fleet-wide.
collector.SetLeafFingerprint(fp) collector.SetLeafFingerprint(fp)
runner := backup.NewBackupRunner(px, cfg.Backup.BackupTarget(), "", "felhom local-api", cfg.Backup.PruneBackupsSpec(), logger) // R-82: ONE RUNNER PER TIER. The runner holds its target, mode, notes and retention as
// immutable construction state, and localPruneSpec reads that retention — so parameterising a
// single runner by target would risk a call pairing tier A's target with tier B's retention.
// One runner per tier keeps each tier's policy structurally inseparable from its target.
backupTiers, tierWarnings := cfg.Backup.BackupTiers()
for _, wmsg := range tierWarnings {
// LOUD on purpose: a silently dropped backup tier is an "applied and empty" DR tier, which
// is the exact fault R-82 exists to fix. Never downgrade this to DEBUG.
logger.Error("backup tier REJECTED — this tier will never run", "detail", wmsg)
}
apiTiers := make([]localapi.BackupTier, 0, len(backupTiers))
var runner *backup.BackupRunner
for _, t := range backupTiers {
prune := ""
if t.KeepLast > 0 {
prune = fmt.Sprintf("keep-last=%d", t.KeepLast)
}
r := backup.NewBackupRunner(px, t.TargetID, "", "felhom local-api", prune, logger)
if t.Primary {
runner = r
}
apiTiers = append(apiTiers, localapi.BackupTier{
TargetID: t.TargetID,
Cadence: t.Cadence,
Primary: t.Primary,
Service: r,
})
logger.Info("backup tier armed", "target", t.TargetID, "cadence", t.Cadence.String(),
"keep_last", t.KeepLast, "primary", t.Primary)
}
// Guest data-drive passthrough (slice 10 P2): a root-CLI runner for the `pct set` bind + chown // Guest data-drive passthrough (slice 10 P2): a root-CLI runner for the `pct set` bind + chown
// (same fenced ExecRunner the host-storage + provision back-half use). // (same fenced ExecRunner the host-storage + provision back-half use).
gaMode := proxmox.RunnerMode(cfg.Privileged.Mode) gaMode := proxmox.RunnerMode(cfg.Privileged.Mode)
@@ -1283,6 +1312,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel AgentVersion: version, // v0.82.0: the X-Felhom-Agent-Version capability channel
Guests: px, Guests: px,
Backups: runner, Backups: runner,
BackupTiers: apiTiers, // R-82: primary first; untargeted endpoints act on the primary
Store: store, Store: store,
Storage: observer, Storage: observer,
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages) DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
+150
View File
@@ -0,0 +1,150 @@
package config
import (
"encoding/json"
"strings"
"testing"
"time"
)
// R-82 Slice A.1 — per-target cadence + retention resolution.
//
// The load-bearing property is ADDITIVITY: every config that exists on a live box today must
// resolve to exactly one tier that behaves as it does now. The second property is that a
// mis-configured tier is REJECTED LOUDLY rather than defaulted — a weekly DR tier silently running
// daily would fill the datastore, and a silently dropped tier is the "applied and empty" fault
// R-82 exists to fix.
func TestBackupTiers_LegacyConfigIsUnchanged(t *testing.T) {
// Exactly the shape live on demo-felhom today.
var b BackupConfig
raw := `{"local_backup_target":"local","local_backup_retention":3,"backup_cadence_seconds":0}`
if err := json.Unmarshal([]byte(raw), &b); err != nil {
t.Fatal(err)
}
tiers, warnings := b.BackupTiers()
if len(warnings) != 0 {
t.Fatalf("a legacy config must produce NO warnings; got %v", warnings)
}
if len(tiers) != 1 {
t.Fatalf("a config with no backup_targets must resolve to exactly ONE tier; got %+v", tiers)
}
got := tiers[0]
if got.TargetID != "local" || got.Cadence != 24*time.Hour || got.KeepLast != 3 || !got.Primary {
t.Fatalf("legacy tier changed: %+v", got)
}
}
// An empty BackupConfig still resolves — to the felhom-pbs default target, 24h, keep-last 3.
// (Unchanged pre-R-82 behaviour; pinned so the default target can't drift unnoticed.)
func TestBackupTiers_ZeroConfigKeepsDefaults(t *testing.T) {
tiers, warnings := BackupConfig{}.BackupTiers()
if len(warnings) != 0 || len(tiers) != 1 {
t.Fatalf("zero config: tiers=%+v warnings=%v", tiers, warnings)
}
if tiers[0].TargetID != defaultBackupTarget || tiers[0].Cadence != 24*time.Hour || tiers[0].KeepLast != 3 {
t.Fatalf("zero-config defaults changed: %+v", tiers[0])
}
}
// The whole point: local daily + PBS weekly, each with its OWN retention.
func TestBackupTiers_LocalDailyPlusPBSWeekly(t *testing.T) {
var b BackupConfig
raw := `{
"local_backup_target":"local",
"local_backup_retention":3,
"backup_cadence_seconds":86400,
"backup_targets":[{"target_id":"felhom-pbs","cadence_seconds":604800,"keep_last":2}]
}`
if err := json.Unmarshal([]byte(raw), &b); err != nil {
t.Fatal(err)
}
tiers, warnings := b.BackupTiers()
if len(warnings) != 0 {
t.Fatalf("unexpected warnings: %v", warnings)
}
if len(tiers) != 2 {
t.Fatalf("want 2 tiers, got %+v", tiers)
}
if !tiers[0].Primary || tiers[0].TargetID != "local" || tiers[0].Cadence != 24*time.Hour || tiers[0].KeepLast != 3 {
t.Fatalf("primary tier wrong: %+v", tiers[0])
}
if tiers[1].Primary || tiers[1].TargetID != "felhom-pbs" || tiers[1].Cadence != 7*24*time.Hour || tiers[1].KeepLast != 2 {
t.Fatalf("PBS tier wrong: %+v", tiers[1])
}
// THE knob-sharing check: the two retentions are independent values, not one shared number.
if tiers[0].KeepLast == tiers[1].KeepLast {
t.Fatalf("this fixture sets 3 and 2 deliberately — equal values mean the knob is shared: %+v", tiers)
}
}
// A tier with no cadence is REJECTED, not defaulted. Defaulting would turn a weekly DR tier into a
// daily one and fill the 37.2 GB datastore (R-82 Phase 0, P0.3).
func TestBackupTiers_MissingCadenceIsRejectedLoudly(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", KeepLast: 2}},
}
tiers, warnings := b.BackupTiers()
if len(tiers) != 1 {
t.Fatalf("a cadence-less tier must NOT be armed; got %+v", tiers)
}
if len(warnings) != 1 || !strings.Contains(warnings[0], "cadence_seconds must be > 0") {
t.Fatalf("rejection must be reported so the caller can log it loudly; got %v", warnings)
}
if !strings.Contains(warnings[0], "felhom-pbs") {
t.Fatalf("the warning must name the tier it dropped; got %q", warnings[0])
}
}
func TestBackupTiers_RejectsEmptyAndDuplicateTargets(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
ExtraTargets: []BackupTargetConfig{
{TargetID: "", CadenceSeconds: 3600},
{TargetID: "local", CadenceSeconds: 3600}, // repeats the primary
{TargetID: "felhom-pbs", CadenceSeconds: 604800}, // good
{TargetID: "felhom-pbs", CadenceSeconds: 99}, // duplicate
},
}
tiers, warnings := b.BackupTiers()
if len(tiers) != 2 || tiers[1].TargetID != "felhom-pbs" || tiers[1].Cadence != 7*24*time.Hour {
t.Fatalf("want primary + one PBS tier at the FIRST definition; got %+v", tiers)
}
if len(warnings) != 3 {
t.Fatalf("want 3 rejections (empty, duplicate-of-primary, duplicate); got %v", warnings)
}
}
// keep_last unset means DO NOT PRUNE. That is the fail-safe: a DR tier must never start pruning
// itself because someone forgot a field.
func TestBackupTiers_UnsetKeepLastMeansNoPrune(t *testing.T) {
b := BackupConfig{
LocalBackupTarget: "local",
ExtraTargets: []BackupTargetConfig{{TargetID: "felhom-pbs", CadenceSeconds: 604800}},
}
tiers, _ := b.BackupTiers()
if len(tiers) != 2 {
t.Fatalf("got %+v", tiers)
}
if tiers[1].KeepLast != 0 {
t.Fatalf("an unset keep_last must resolve to 0 = never prune; got %d", tiers[1].KeepLast)
}
// And a negative is clamped to the same fail-safe rather than becoming a prune spec.
b.ExtraTargets[0].KeepLast = -5
tiers, _ = b.BackupTiers()
if tiers[1].KeepLast != 0 {
t.Fatalf("a negative keep_last must clamp to 0 (never prune); got %d", tiers[1].KeepLast)
}
}
// The primary's retention still comes from the legacy knob with its legacy clamp — untouched.
func TestBackupTiers_PrimaryRetentionClampUnchanged(t *testing.T) {
for _, tc := range []struct{ in, want int }{{0, 3}, {-1, 3}, {1, 1}, {7, 7}} {
b := BackupConfig{LocalBackupTarget: "local", LocalBackupRetention: tc.in}
tiers, _ := b.BackupTiers()
if tiers[0].KeepLast != tc.want {
t.Fatalf("LocalBackupRetention=%d → KeepLast=%d, want %d", tc.in, tiers[0].KeepLast, tc.want)
}
}
}
+89
View File
@@ -366,6 +366,95 @@ type BackupConfig struct {
// default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup. // default 3; ALWAYS clamped to ≥1 by KeepLast() so a mis-config can never prune the fresh backup.
// NEVER applied to a PBS target (offsite retention is a separate lifecycle). // NEVER applied to a PBS target (offsite retention is a separate lifecycle).
LocalBackupRetention int `json:"local_backup_retention"` LocalBackupRetention int `json:"local_backup_retention"`
// ExtraTargets (R-82) are ADDITIONAL backup tiers beyond the primary one above — the shape that
// makes "local daily + PBS weekly" expressible at all. Each carries its OWN cadence and its OWN
// retention, because those are semantically different per tier: keep-last=3 on a daily tier is
// three DAYS of restore points; on a weekly tier it is three WEEKS. Sharing one knob between
// tiers silently means one of them is wrong.
//
// ADDITIVE BY CONSTRUCTION: an existing config with no `backup_targets` key resolves to exactly
// one tier — the primary — and behaves byte-identically to pre-R-82. Nothing here changes the
// local tier.
ExtraTargets []BackupTargetConfig `json:"backup_targets"`
}
// BackupTargetConfig is ONE additional backup tier: a vzdump storage plus its own cadence and
// retention. A tier with no cadence is not a tier — see BackupTiers for why that is rejected loudly
// rather than defaulted.
type BackupTargetConfig struct {
// TargetID is the Proxmox storage id (content=backup), e.g. "felhom-pbs".
TargetID string `json:"target_id"`
// CadenceSeconds is THIS tier's /backup/due window. REQUIRED (>0) — see BackupTiers.
CadenceSeconds int `json:"cadence_seconds"`
// KeepLast is THIS tier's per-run `--prune-backups` keep-last. 0/unset → NEVER prune this tier
// (the fail-safe default, and the current behaviour for every PBS target). A PBS tier is never
// pruned by the per-run flag regardless — see BackupRunner.localPruneSpec.
KeepLast int `json:"keep_last"`
}
// BackupTier is a RESOLVED backup tier: one target, its own cadence, its own retention. The agent
// builds one runner per tier from these.
type BackupTier struct {
TargetID string
Cadence time.Duration
// KeepLast is the per-run prune keep-last; 0 means DO NOT PRUNE this tier.
KeepLast int
// Primary marks the tier that the UNTARGETED local-API endpoints act on — the pre-R-82 tier.
// Exactly one tier is primary, and it is always first.
Primary bool
}
// BackupTiers resolves the effective tier list, primary first, plus any warnings the caller MUST
// log (they describe tiers that were REJECTED, and a silently-dropped backup tier is precisely the
// "applied and empty" fault R-82 exists to fix).
//
// Rules:
// - Tier 0 is always the primary, built from BackupTarget()/BackupCadence()/KeepLast() — so a
// config with no `backup_targets` is byte-identical to pre-R-82.
// - An extra with an empty target_id is rejected.
// - An extra with cadence_seconds <= 0 is REJECTED, not defaulted. Defaulting a PBS tier to the
// 24h local default would quietly turn a weekly tier into a daily one and fill the DR datastore;
// a tier whose cadence you did not state is not a tier.
// - An extra repeating the primary's target is rejected (one policy per target, or the two
// cadences race and neither is the truth).
// - Duplicate extras are rejected after the first.
func (b BackupConfig) BackupTiers() ([]BackupTier, []string) {
primary := BackupTier{
TargetID: b.BackupTarget(),
Cadence: b.BackupCadence(),
KeepLast: b.KeepLast(),
Primary: true,
}
tiers := []BackupTier{primary}
var warnings []string
seen := map[string]bool{primary.TargetID: true}
for i, t := range b.ExtraTargets {
id := strings.TrimSpace(t.TargetID)
switch {
case id == "":
warnings = append(warnings, fmt.Sprintf("backup_targets[%d]: empty target_id — tier ignored", i))
continue
case seen[id]:
warnings = append(warnings, fmt.Sprintf("backup_targets[%d]: target %q already configured — duplicate tier ignored", i, id))
continue
case t.CadenceSeconds <= 0:
warnings = append(warnings, fmt.Sprintf("backup_targets[%d] (%s): cadence_seconds must be > 0 — tier ignored (a cadence is NOT defaulted: a weekly tier silently running daily would fill the DR datastore)", i, id))
continue
}
seen[id] = true
keep := t.KeepLast
if keep < 0 {
keep = 0
}
tiers = append(tiers, BackupTier{
TargetID: id,
Cadence: time.Duration(t.CadenceSeconds) * time.Second,
KeepLast: keep,
})
}
return tiers, warnings
} }
// defaultLocalBackupKeepLast is the local vzdump retention default (newest N restore points kept). // defaultLocalBackupKeepLast is the local vzdump retention default (newest N restore points kept).
+56
View File
@@ -0,0 +1,56 @@
package localapi
import "time"
// normalizeBackupTiers resolves the tier list the Server serves.
//
// Contract (R-82), and the reason this is a named function rather than inline setup: the UNTARGETED
// local-API endpoints must keep behaving exactly as they did before multi-tier existed, forever.
// That property lives here.
//
// - tiers == nil → synthesize ONE tier from the legacy (Backups, BackupCadence) pair and mark it
// primary. This is the pre-R-82 shape; every existing caller and test hits this path.
// - tiers supplied → keep order but hoist the primary to the front; if none is marked primary,
// the FIRST becomes primary (a tier list with no primary would leave untargeted requests with
// nothing to act on, which would silently stop backups).
// - tiers with a nil Service are dropped: a tier with no runner cannot back anything up, and
// advertising it would be an "applied and empty" tier — the exact fault R-82 exists to fix.
func normalizeBackupTiers(tiers []BackupTier, legacy BackupService, cadence time.Duration) []BackupTier {
usable := make([]BackupTier, 0, len(tiers))
for _, t := range tiers {
if t.Service == nil || t.TargetID == "" {
continue
}
if t.Cadence <= 0 {
t.Cadence = cadence
}
usable = append(usable, t)
}
if len(usable) == 0 {
if legacy == nil {
return nil
}
return []BackupTier{{TargetID: "", Cadence: cadence, Primary: true, Service: legacy}}
}
primary := -1
for i, t := range usable {
if t.Primary {
primary = i
break
}
}
if primary < 0 {
primary = 0
}
out := make([]BackupTier, 0, len(usable))
usable[primary].Primary = true
out = append(out, usable[primary])
for i, t := range usable {
if i == primary {
continue
}
t.Primary = false
out = append(out, t)
}
return out
}
+390
View File
@@ -0,0 +1,390 @@
package localapi
import (
"context"
"encoding/json"
"io"
"log/slog"
"net/http"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// R-82 Slice A — per-target cadence, due and runner.
//
// The agent and the controller deploy INDEPENDENTLY. A new agent will serve old controllers for as
// long as it takes the fleet to catch up, so the untargeted contract is frozen, not merely
// "probably fine". These tests pin that freeze; the multi-tier behaviour is additive on top.
// tieredServer builds a two-tier server: primary "local" (24h) + "felhom-pbs" (7d), each with its
// own runner, exactly as main.go wires it.
func tieredServer(t *testing.T, st *fakeStore, localSvc, pbsSvc *fakeBackups) *Server {
t.Helper()
srv, err := NewServer(Options{
ListenAddr: "127.0.0.1:0",
Guests: &fakeGuests{},
Backups: localSvc,
Store: st,
Storage: fakeStorage{},
Tokens: staticTokens{"A": 8200, "B": 9300},
BackupTiers: []BackupTier{
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: localSvc},
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: pbsSvc},
},
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
})
if err != nil {
t.Fatalf("new server: %v", err)
}
srv.baseCtx = context.Background()
srv.now = func() time.Time { return testNow }
return srv
}
// seen returns the vmids this fake runner was invoked for (mutex-guarded — the backup runs on a
// goroutine).
func (f *fakeBackups) seen() []int {
f.mu.Lock()
defer f.mu.Unlock()
return append([]int(nil), f.vmids...)
}
// waitFor polls cond for up to 2s. POST /backup is fire-and-forget, so the assertion has to wait
// for the goroutine rather than assume it has run.
func waitFor(t *testing.T, cond func() bool) {
t.Helper()
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
if cond() {
return
}
time.Sleep(5 * time.Millisecond)
}
t.Fatalf("condition not met within 2s")
}
func backupAt(target string, vmid int, ago time.Duration, ok bool) hub.Backup {
return hub.Backup{
TargetID: target,
VMID: vmid,
Success: ok,
StartedAt: testNow.Add(-ago).Format(time.RFC3339),
}
}
// ── RED-PROOF 1 — old controller ↔ new agent ────────────────────────────────────────────────
//
// An old controller sends `GET /backup/due` with no query string and parses the pre-R-82 response.
// The response must be BYTE-IDENTICAL — not merely semantically similar. A stray `"target":"local"`
// key is harmless to a tolerant JSON decoder and fatal to a strict one, and we do not get to choose
// which the deployed fleet has.
//
// COMPANION RED-PROOF (observed): drop the `omitempty` from BackupDueResponse.Target and have
// tierFromRequest echo the primary's id for an untargeted request → this test fails with the
// observed body carrying `"target":"local"`. Restored.
func TestBackupDue_Untargeted_ResponseBytesUnchanged(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("local", 8200, 2*time.Hour, true))
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
rr := do(t, h, "GET", "/backup/due", "A", "")
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, want 200", rr.Code)
}
var got map[string]any
if err := json.Unmarshal(rr.Body.Bytes(), &got); err != nil {
t.Fatalf("unmarshal: %v (body %s)", err, rr.Body.String())
}
data, _ := got["data"].(map[string]any)
if data == nil {
t.Fatalf("no data object in %s", rr.Body.String())
}
if _, present := data["target"]; present {
t.Fatalf("UNTARGETED response MUST NOT carry a target key — an old controller sees a changed contract; body: %s", rr.Body.String())
}
if data["due"] != false {
t.Fatalf("2h-old local backup under a 24h cadence must not be due; body: %s", rr.Body.String())
}
if data["reason"] != "within cadence window" {
t.Fatalf("reason string changed: %v", data["reason"])
}
}
// The untargeted verdict must come from the PRIMARY tier's cadence, not from whichever tier
// happens to be freshest. With a stale local and a fresh PBS backup, untargeted must say DUE.
func TestBackupDue_Untargeted_UsesPrimaryTierOnly(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("local", 8200, 30*time.Hour, true)) // stale for 24h cadence
st.RecordBackup(backupAt("felhom-pbs", 8200, 1*time.Hour, true)) // fresh, different tier
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
var resp struct {
Data BackupDueResponse `json:"data"`
}
rr := do(t, h, "GET", "/backup/due", "A", "")
if err := json.Unmarshal(rr.Body.Bytes(), &resp); err != nil {
t.Fatal(err)
}
if !resp.Data.Due {
t.Fatalf("a fresh backup on ANOTHER tier must not satisfy the primary's cadence; got %+v", resp.Data)
}
}
// ── Per-tier due-ness ────────────────────────────────────────────────────────────────────────
// THE POINT OF THE WHOLE SLICE: a fresh daily local backup must NOT satisfy the weekly PBS tier.
// Without the per-target filter in latestSuccessfulBackupForTarget the DR tier would never run —
// which is exactly today's "applied and empty" state, re-created in code.
func TestBackupDue_PerTier_LocalFreshDoesNotSatisfyPBS(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("local", 8200, 2*time.Hour, true)) // fresh daily
st.RecordBackup(backupAt("felhom-pbs", 8200, 8*24*time.Hour, true)) // 8d — past the 7d weekly
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
var local, pbs struct {
Data BackupDueResponse `json:"data"`
}
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=local", "A", "").Body.Bytes(), &local); err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
t.Fatal(err)
}
if local.Data.Due {
t.Fatalf("local tier: 2h old under 24h cadence must NOT be due; got %+v", local.Data)
}
if !pbs.Data.Due {
t.Fatalf("PBS tier: 8d old under a 7d cadence MUST be due; got %+v", pbs.Data)
}
if pbs.Data.Target != "felhom-pbs" || local.Data.Target != "local" {
t.Fatalf("a targeted response must echo its tier; got local=%q pbs=%q", local.Data.Target, pbs.Data.Target)
}
}
// A 6-day-old PBS snapshot is INSIDE the weekly window — it must not be due. The mirror of the
// hub-side threshold test in Slice C, asserted here at the source of truth.
func TestBackupDue_PerTier_PBSWithinWeeklyWindow(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("felhom-pbs", 8200, 6*24*time.Hour, true))
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
var pbs struct {
Data BackupDueResponse `json:"data"`
}
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
t.Fatal(err)
}
if pbs.Data.Due {
t.Fatalf("6d old under a 7d cadence must NOT be due; got %+v", pbs.Data)
}
}
// A failed backup must not satisfy any tier's cadence (pre-existing rule, re-asserted per-tier).
func TestBackupDue_PerTier_FailedBackupDoesNotCount(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("felhom-pbs", 8200, time.Hour, false))
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
var pbs struct {
Data BackupDueResponse `json:"data"`
}
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
t.Fatal(err)
}
if !pbs.Data.Due {
t.Fatalf("a FAILED backup must not satisfy the cadence; got %+v", pbs.Data)
}
}
// The fail-safe-toward-due rule survives per-tier: an unparseable timestamp yields DUE.
// A spurious backup is cheap; a skipped one is not.
func TestBackupDue_PerTier_UnparseableTimestampIsDue(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(hub.Backup{TargetID: "felhom-pbs", VMID: 8200, Success: true, StartedAt: "not-a-time"})
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
var pbs struct {
Data BackupDueResponse `json:"data"`
}
if err := json.Unmarshal(do(t, h, "GET", "/backup/due?target=felhom-pbs", "A", "").Body.Bytes(), &pbs); err != nil {
t.Fatal(err)
}
if !pbs.Data.Due {
t.Fatalf("unparseable timestamp must fail SAFE toward due; got %+v", pbs.Data)
}
}
// An unknown target is a 400 — never a silent fallback to the primary. A controller asking about a
// tier this agent does not serve must find out, not be handed a different tier's freshness and act
// on it.
func TestBackupDue_UnknownTarget_IsAnErrorNotAFallback(t *testing.T) {
st := &fakeStore{}
st.RecordBackup(backupAt("local", 8200, 2*time.Hour, true))
h := tieredServer(t, st, &fakeBackups{}, &fakeBackups{}).Handler()
rr := do(t, h, "GET", "/backup/due?target=does-not-exist", "A", "")
if rr.Code != http.StatusBadRequest {
t.Fatalf("unknown target must be 400 (got %d, body %s)", rr.Code, rr.Body.String())
}
}
// ── Tier advertisement ───────────────────────────────────────────────────────────────────────
// GET /backup/tiers is the controller's capability probe. Primary must be first and flagged, so a
// controller can tell which tier the untargeted endpoints act on.
func TestBackupTiers_AdvertisesPrimaryFirst(t *testing.T) {
h := tieredServer(t, &fakeStore{}, &fakeBackups{}, &fakeBackups{}).Handler()
var resp struct {
Data BackupTiersResponse `json:"data"`
}
rr := do(t, h, "GET", "/backup/tiers", "A", "")
if rr.Code != http.StatusOK {
t.Fatalf("status = %d", rr.Code)
}
if err := json.Unmarshal(rr.Body.Bytes(), &resp); err != nil {
t.Fatal(err)
}
if len(resp.Data.Tiers) != 2 {
t.Fatalf("want 2 tiers, got %+v", resp.Data.Tiers)
}
if !resp.Data.Tiers[0].Primary || resp.Data.Tiers[0].Target != "local" {
t.Fatalf("primary must be first and flagged; got %+v", resp.Data.Tiers)
}
if resp.Data.Tiers[1].Target != "felhom-pbs" || resp.Data.Tiers[1].CadenceSeconds != int64((7 * 24 * time.Hour).Seconds()) {
t.Fatalf("PBS tier mis-advertised: %+v", resp.Data.Tiers[1])
}
}
// ── POST /backup routing + per-tier single-flight ────────────────────────────────────────────
// A targeted POST must run THAT tier's runner. Routing both tiers to one runner would silently
// write every "PBS" backup to local — a DR tier that reports success and stores nothing.
func TestBackupPost_RoutesToTheTargetsOwnRunner(t *testing.T) {
local, pbs := &fakeBackups{}, &fakeBackups{}
srv := tieredServer(t, &fakeStore{}, local, pbs)
h := srv.Handler()
if rr := do(t, h, "POST", "/backup?target=felhom-pbs", "A", ""); rr.Code != http.StatusAccepted {
t.Fatalf("status = %d, body %s", rr.Code, rr.Body.String())
}
waitFor(t, func() bool { return len(pbs.seen()) == 1 })
if got := len(local.seen()); got != 0 {
t.Fatalf("the LOCAL runner must not have run for a PBS-targeted request (ran %d times)", got)
}
}
// Untargeted POST routes to the primary — the old controller's path.
func TestBackupPost_UntargetedRoutesToPrimary(t *testing.T) {
local, pbs := &fakeBackups{}, &fakeBackups{}
h := tieredServer(t, &fakeStore{}, local, pbs).Handler()
if rr := do(t, h, "POST", "/backup", "A", ""); rr.Code != http.StatusAccepted {
t.Fatalf("status = %d", rr.Code)
}
waitFor(t, func() bool { return len(local.seen()) == 1 })
if got := len(pbs.seen()); got != 0 {
t.Fatalf("untargeted POST must not touch a non-primary tier (ran %d times)", got)
}
}
// Single-flight is PER TIER. A PBS backup starting while the local one is still running must get
// its OWN job id — this is what lets Slice B run both inside one quiesce window. Keying jobs by
// vmid alone would hand the second call the first job's id and the controller would believe the
// PBS backup finished when only the local one had.
func TestBackupPost_SingleFlightIsPerTierNotPerGuest(t *testing.T) {
localGate := make(chan struct{})
local := &fakeBackups{gate: localGate}
pbs := &fakeBackups{}
h := tieredServer(t, &fakeStore{}, local, pbs).Handler()
var first, second BackupResponse
rr1 := do(t, h, "POST", "/backup", "A", "") // local; blocks on the gate
if err := json.Unmarshal(rr1.Body.Bytes(), &struct {
Data *BackupResponse `json:"data"`
}{Data: &first}); err != nil {
t.Fatal(err)
}
rr2 := do(t, h, "POST", "/backup?target=felhom-pbs", "A", "") // different tier → new job
if err := json.Unmarshal(rr2.Body.Bytes(), &struct {
Data *BackupResponse `json:"data"`
}{Data: &second}); err != nil {
t.Fatal(err)
}
if first.JobID == "" || second.JobID == "" {
t.Fatalf("both tiers must get a job id; got %q / %q", first.JobID, second.JobID)
}
if first.JobID == second.JobID {
t.Fatalf("PER-TIER single-flight violated: the PBS request was handed the LOCAL job id %q — the controller would believe the PBS backup ran", first.JobID)
}
waitFor(t, func() bool { return len(pbs.seen()) == 1 })
close(localGate)
}
// Same tier, still single-flight: a second POST to a running tier returns the SAME job.
func TestBackupPost_SameTierStillSingleFlight(t *testing.T) {
gate := make(chan struct{})
local := &fakeBackups{gate: gate}
h := tieredServer(t, &fakeStore{}, local, &fakeBackups{}).Handler()
var a, b BackupResponse
if err := json.Unmarshal(do(t, h, "POST", "/backup", "A", "").Body.Bytes(), &struct {
Data *BackupResponse `json:"data"`
}{Data: &a}); err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(do(t, h, "POST", "/backup", "A", "").Body.Bytes(), &struct {
Data *BackupResponse `json:"data"`
}{Data: &b}); err != nil {
t.Fatal(err)
}
if a.JobID != b.JobID {
t.Fatalf("same tier must single-flight: %q vs %q", a.JobID, b.JobID)
}
close(gate)
}
// ── normalizeBackupTiers — the compatibility core ────────────────────────────────────────────
func TestNormalizeBackupTiers(t *testing.T) {
svc := &fakeBackups{}
t.Run("nil tiers synthesize the legacy single tier", func(t *testing.T) {
got := normalizeBackupTiers(nil, svc, 24*time.Hour)
if len(got) != 1 || !got[0].Primary || got[0].Cadence != 24*time.Hour {
t.Fatalf("legacy synthesis broken: %+v", got)
}
})
t.Run("primary is hoisted to the front", func(t *testing.T) {
got := normalizeBackupTiers([]BackupTier{
{TargetID: "felhom-pbs", Cadence: time.Hour, Service: svc},
{TargetID: "local", Cadence: time.Hour, Primary: true, Service: svc},
}, svc, 24*time.Hour)
if len(got) != 2 || got[0].TargetID != "local" || !got[0].Primary || got[1].Primary {
t.Fatalf("primary not hoisted / uniqueness broken: %+v", got)
}
})
t.Run("no primary marked → first becomes primary", func(t *testing.T) {
got := normalizeBackupTiers([]BackupTier{
{TargetID: "a", Cadence: time.Hour, Service: svc},
{TargetID: "b", Cadence: time.Hour, Service: svc},
}, svc, 24*time.Hour)
if len(got) != 2 || got[0].TargetID != "a" || !got[0].Primary {
t.Fatalf("want first-as-primary, got %+v", got)
}
})
t.Run("a tier with no runner is DROPPED, not advertised", func(t *testing.T) {
got := normalizeBackupTiers([]BackupTier{
{TargetID: "local", Cadence: time.Hour, Primary: true, Service: svc},
{TargetID: "felhom-pbs", Cadence: time.Hour, Service: nil},
}, svc, 24*time.Hour)
if len(got) != 1 || got[0].TargetID != "local" {
t.Fatalf("a serviceless tier must not be advertised (it could never run): %+v", got)
}
})
}
+190 -35
View File
@@ -41,6 +41,20 @@ type BackupService interface {
BackupWithSnapshotHook(ctx context.Context, vmid int, onSnapshot func()) (hub.Backup, error) BackupWithSnapshotHook(ctx context.Context, vmid int, onSnapshot func()) (hub.Backup, error)
} }
// BackupTier (R-82) binds ONE backup tier's runner to its own policy. The agent builds one per
// resolved config tier; the local API serves each independently so "local daily + PBS weekly" is
// expressible over the wire, not just in config.
//
// COMPATIBILITY CONTRACT (load-bearing — the agent and controller deploy independently):
// the tier whose Primary is true is what EVERY untargeted endpoint acts on. An old controller
// never sends `?target=`, so it sees exactly the pre-R-82 behaviour and response bytes.
type BackupTier struct {
TargetID string
Cadence time.Duration
Primary bool
Service BackupService
}
// BackupStore records + reads the latest backup/restore-test state. Satisfied by *backup.Store. // BackupStore records + reads the latest backup/restore-test state. Satisfied by *backup.Store.
type BackupStore interface { type BackupStore interface {
RecordBackup(hub.Backup) RecordBackup(hub.Backup)
@@ -92,7 +106,11 @@ type Options struct {
// BackupCadence is the per-guest backup interval driving GET /backup/due (slice 8B). A guest // BackupCadence is the per-guest backup interval driving GET /backup/due (slice 8B). A guest
// is "due" when no successful backup is recorded OR the newest one is older than this. 0 → a // is "due" when no successful backup is recorded OR the newest one is older than this. 0 → a
// safe default (24h). The hub-served policy is slice 10; this is the agent-local cadence. // safe default (24h). The hub-served policy is slice 10; this is the agent-local cadence.
// When BackupTiers is supplied this is IGNORED (the primary tier carries its own cadence).
BackupCadence time.Duration BackupCadence time.Duration
// BackupTiers (R-82) is the resolved multi-tier policy, primary first. OPTIONAL: nil → one tier
// synthesized from Backups + BackupCadence, i.e. exactly the pre-R-82 behaviour.
BackupTiers []BackupTier
// Disk management (slice 8C) — OPTIONAL. When Disks + DiskGate are set, the /disks endpoints // Disk management (slice 8C) — OPTIONAL. When Disks + DiskGate are set, the /disks endpoints
// are served; otherwise they report "not configured". DiskGate authorizes the destructive // are served; otherwise they report "not configured". DiskGate authorizes the destructive
// (data-bearing) format path; Guests lists guests for the eject dependent-warning. // (data-bearing) format path; Guests lists guests for the eject dependent-warning.
@@ -183,6 +201,12 @@ type backupJob struct {
Error string Error string
} }
// backupJobKey identifies one guest's job on ONE tier (R-82).
type backupJobKey struct {
vmid int
target string
}
// Server is the per-guest local API (doc 03 §6). It serves the agent's pinned self-signed leaf // Server is the per-guest local API (doc 03 §6). It serves the agent's pinned self-signed leaf
// and authorizes every request against the token's guest only. // and authorizes every request against the token's guest only.
type Server struct { type Server struct {
@@ -196,6 +220,10 @@ type Server struct {
smart SmartReader // v0.95.0 Fix B: SMART for the /disks union path (optional) smart SmartReader // v0.95.0 Fix B: SMART for the /disks union path (optional)
tokens TokenAuthority tokens TokenAuthority
cadence time.Duration cadence time.Duration
// tiers (R-82) is the resolved backup-tier list, PRIMARY FIRST. Always non-empty: when the
// caller supplies no tiers it holds exactly one, synthesized from Backups+BackupCadence, which
// is the pre-R-82 shape.
tiers []BackupTier
logger *slog.Logger logger *slog.Logger
now func() time.Time now func() time.Time
@@ -247,7 +275,10 @@ type Server struct {
boundCheck func(string) bool boundCheck func(string) bool
jobsMu sync.Mutex jobsMu sync.Mutex
jobs map[int]*backupJob // per-guest backup job state (slice 8B) // jobs is per-guest-PER-TARGET backup job state (slice 8B; keyed by target too since R-82).
// Keying by vmid alone would let a PBS backup started inside the same quiesce window collide
// with the local one's single-flight and hand the caller the WRONG job id.
jobs map[backupJobKey]*backupJob
// agentic controller update (Phase 1): the swapper + per-guest single-flight gate. // agentic controller update (Phase 1): the swapper + per-guest single-flight gate.
swap *ControllerSwapper swap *ControllerSwapper
@@ -336,9 +367,17 @@ func NewServer(o Options) (*Server, error) {
hostID: o.HostID, hostID: o.HostID,
agentVersion: o.AgentVersion, agentVersion: o.AgentVersion,
logRing: o.LogRing, logRing: o.LogRing,
jobs: map[int]*backupJob{}, jobs: map[backupJobKey]*backupJob{},
swapInFlight: map[int]bool{}, swapInFlight: map[int]bool{},
} }
// R-82 tier resolution. Options.BackupTiers is authoritative when supplied; otherwise ONE tier
// is synthesized from Backups + BackupCadence — the pre-R-82 shape, so every existing caller
// (and every existing test) keeps working untouched. Exactly one tier is marked primary, and
// the primary is always first, because that is what the untargeted endpoints act on.
s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence)
if s.backups == nil && len(s.tiers) > 0 {
s.backups = s.tiers[0].Service
}
if s.escrowStagePath == "" { if s.escrowStagePath == "" {
s.escrowStagePath = escrow.StagedResticPasswordPath() s.escrowStagePath = escrow.StagedResticPasswordPath()
} }
@@ -372,6 +411,7 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /rollback", s.withGuest(s.handleRollback)) mux.HandleFunc("POST /rollback", s.withGuest(s.handleRollback))
mux.HandleFunc("POST /backup", s.withGuest(s.handleBackup)) mux.HandleFunc("POST /backup", s.withGuest(s.handleBackup))
mux.HandleFunc("GET /backup/due", s.withGuest(s.handleBackupDue)) mux.HandleFunc("GET /backup/due", s.withGuest(s.handleBackupDue))
mux.HandleFunc("GET /backup/tiers", s.withGuest(s.handleBackupTiers))
mux.HandleFunc("GET /backup/status", s.withGuest(s.handleBackupStatus)) mux.HandleFunc("GET /backup/status", s.withGuest(s.handleBackupStatus))
mux.HandleFunc("GET /restore-test/status", s.withGuest(s.handleRestoreTestStatus)) mux.HandleFunc("GET /restore-test/status", s.withGuest(s.handleRestoreTestStatus))
// Host metrics (slice 9): host-wide health + per-storage capacity for the customer's monitoring // Host metrics (slice 9): host-wide health + per-storage capacity for the customer's monitoring
@@ -640,6 +680,8 @@ type BackupResponse struct {
VMID int `json:"vmid"` VMID int `json:"vmid"`
JobID string `json:"job_id"` JobID string `json:"job_id"`
Phase string `json:"phase"` Phase string `json:"phase"`
// Target (R-82) echoes the tier; empty + omitted for an untargeted request (pre-R-82 bytes).
Target string `json:"target,omitempty"`
} }
func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int) { func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int) {
@@ -653,17 +695,33 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
return return
} }
} }
// Single-flight per guest: if a backup is already running for this guest, return that job tier, echo, ok := s.tierFromRequest(w, r)
// (don't start a second concurrent vzdump). The controller polls /backup/status on it. if !ok {
s.jobsMu.Lock()
if cur := s.jobs[vmid]; cur != nil && cur.Phase == PhaseRunning {
job := *cur
s.jobsMu.Unlock()
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: job.JobID, Phase: job.Phase}, "")
return return
} }
key := backupJobKey{vmid: vmid, target: tier.TargetID}
// Single-flight per guest PER TIER: if a backup is already running for this guest ON THIS
// TIER, return that job (don't start a second concurrent vzdump to the same target). A
// DIFFERENT tier is a different job — that is what lets the weekly night run both backups
// inside one quiesce window without the second call being handed the first one's id.
s.jobsMu.Lock()
if cur := s.jobs[key]; cur != nil && cur.Phase == PhaseRunning {
job := *cur
s.jobsMu.Unlock()
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: job.JobID, Phase: job.Phase, Target: echo}, "")
return
}
// Job ids must be unique PER TIER, and by construction rather than by clock luck: two tiers
// started inside the same nanosecond (the weekly both-due night, or any injected clock) would
// otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the
// pre-R-82 format byte-for-byte — an old controller stores this string and polls with it — so
// only the additive tiers carry the target segment.
jobID := "backup-" + strconv.Itoa(vmid) + "-" + strconv.FormatInt(s.now().UnixNano(), 10) jobID := "backup-" + strconv.Itoa(vmid) + "-" + strconv.FormatInt(s.now().UnixNano(), 10)
s.jobs[vmid] = &backupJob{JobID: jobID, Phase: PhaseRunning, StartedAt: s.now()} if !tier.Primary && tier.TargetID != "" {
jobID = "backup-" + strconv.Itoa(vmid) + "-" + tier.TargetID + "-" + strconv.FormatInt(s.now().UnixNano(), 10)
}
s.jobs[key] = &backupJob{JobID: jobID, Phase: PhaseRunning, StartedAt: s.now()}
s.jobsMu.Unlock() s.jobsMu.Unlock()
// Enqueue: a vzdump runs for minutes, so we do not block the request. The backup runs on the // Enqueue: a vzdump runs for minutes, so we do not block the request. The backup runs on the
@@ -680,42 +738,47 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
defer cancel() defer cancel()
// 8B.2: flip the job to `snapshotted` when the storage snapshot is taken, so the // 8B.2: flip the job to `snapshotted` when the storage snapshot is taken, so the
// controller resumes its app early (snapshot mode only; in stop mode this never fires). // controller resumes its app early (snapshot mode only; in stop mode this never fires).
b, err := s.backups.BackupWithSnapshotHook(bctx, vmid, func() { s.markSnapshotted(vmid, jobID) }) b, err := tier.Service.BackupWithSnapshotHook(bctx, vmid, func() { s.markSnapshotted(key, jobID) })
if err != nil { if err != nil {
b.VMID = vmid b.VMID = vmid
b.Success = false b.Success = false
if b.Error == "" { if b.Error == "" {
b.Error = err.Error() b.Error = err.Error()
} }
s.logger.Error("local-api: backup job failed", "vmid", vmid, "job", jobID, "err", err) // TargetID is what the hub attributes the record to; a failed run must still say which
// tier failed, and the runner may not have set it on the error path.
if b.TargetID == "" {
b.TargetID = tier.TargetID
}
s.logger.Error("local-api: backup job failed", "vmid", vmid, "target", tier.TargetID, "job", jobID, "err", err)
} else { } else {
s.logger.Info("local-api: backup job complete", "vmid", vmid, "job", jobID, "archive", b.Archive) s.logger.Info("local-api: backup job complete", "vmid", vmid, "target", tier.TargetID, "job", jobID, "archive", b.Archive)
} }
s.store.RecordBackup(b) s.store.RecordBackup(b)
s.finishJob(vmid, jobID, b) s.finishJob(key, jobID, b)
}() }()
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "") writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "")
} }
// markSnapshotted flips the guest's running job to the `snapshotted` phase (8B.2) — only if it is // markSnapshotted flips the guest's running job to the `snapshotted` phase (8B.2) — only if it is
// still the current job and still running (don't regress done/failed, and don't touch a newer job). // still the current job and still running (don't regress done/failed, and don't touch a newer job).
func (s *Server) markSnapshotted(vmid int, jobID string) { func (s *Server) markSnapshotted(key backupJobKey, jobID string) {
s.jobsMu.Lock() s.jobsMu.Lock()
defer s.jobsMu.Unlock() defer s.jobsMu.Unlock()
cur := s.jobs[vmid] cur := s.jobs[key]
if cur == nil || cur.JobID != jobID || cur.Phase != PhaseRunning { if cur == nil || cur.JobID != jobID || cur.Phase != PhaseRunning {
return return
} }
cur.Phase = PhaseSnapshotted cur.Phase = PhaseSnapshotted
s.logger.Info("local-api: backup reached snapshotted (app may resume)", "vmid", vmid, "job", jobID) s.logger.Info("local-api: backup reached snapshotted (app may resume)", "vmid", key.vmid, "target", key.target, "job", jobID)
} }
// finishJob transitions the guest's job to done/failed (only if it is still the current job — a // finishJob transitions the guest's job to done/failed (only if it is still the current job — a
// later job started after a single-flight gap must not be overwritten by an older one's result). // later job started after a single-flight gap must not be overwritten by an older one's result).
func (s *Server) finishJob(vmid int, jobID string, b hub.Backup) { func (s *Server) finishJob(key backupJobKey, jobID string, b hub.Backup) {
s.jobsMu.Lock() s.jobsMu.Lock()
defer s.jobsMu.Unlock() defer s.jobsMu.Unlock()
cur := s.jobs[vmid] cur := s.jobs[key]
if cur == nil || cur.JobID != jobID { if cur == nil || cur.JobID != jobID {
return return
} }
@@ -730,10 +793,10 @@ func (s *Server) finishJob(vmid int, jobID string, b hub.Backup) {
} }
// jobSnapshot returns a copy of the guest's current job (ok=false if none). // jobSnapshot returns a copy of the guest's current job (ok=false if none).
func (s *Server) jobSnapshot(vmid int) (backupJob, bool) { func (s *Server) jobSnapshot(key backupJobKey) (backupJob, bool) {
s.jobsMu.Lock() s.jobsMu.Lock()
defer s.jobsMu.Unlock() defer s.jobsMu.Unlock()
if j := s.jobs[vmid]; j != nil { if j := s.jobs[key]; j != nil {
return *j, true return *j, true
} }
return backupJob{}, false return backupJob{}, false
@@ -748,26 +811,96 @@ type BackupDueResponse struct {
Due bool `json:"due"` Due bool `json:"due"`
Reason string `json:"reason"` Reason string `json:"reason"`
AgeSecs *int64 `json:"age_seconds,omitempty"` // age of the newest successful backup; null if none AgeSecs *int64 `json:"age_seconds,omitempty"` // age of the newest successful backup; null if none
// Target (R-82) echoes the tier this verdict is about. EMPTY (and omitted) for an untargeted
// request, which is what keeps the pre-R-82 response bytes identical for old controllers.
Target string `json:"target,omitempty"`
} }
func (s *Server) handleBackupDue(w http.ResponseWriter, r *http.Request, vmid int) { func (s *Server) handleBackupDue(w http.ResponseWriter, r *http.Request, vmid int) {
latest := s.latestSuccessfulBackupFor(r.Context(), vmid) tier, echo, ok := s.tierFromRequest(w, r)
if latest == nil { if !ok {
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "no successful backup recorded yet"})
return return
} }
age, ok := backupAge(latest.StartedAt, s.now()) latest := s.latestSuccessfulBackupForTarget(r.Context(), vmid, tier.TargetID)
if !ok { if latest == nil {
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "no successful backup recorded yet", Target: echo})
return
}
age, ok2 := backupAge(latest.StartedAt, s.now())
if !ok2 {
// Unparseable timestamp: fail safe toward "due" so a backup still happens. // Unparseable timestamp: fail safe toward "due" so a backup still happens.
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "last backup time unparseable — treating as due"}) writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "last backup time unparseable — treating as due", Target: echo})
return return
} }
ageSecs := int64(age.Seconds()) ageSecs := int64(age.Seconds())
if age >= s.cadence { if age >= tier.Cadence {
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "older than cadence", AgeSecs: &ageSecs}) writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "older than cadence", AgeSecs: &ageSecs, Target: echo})
return return
} }
writeOK(w, BackupDueResponse{VMID: vmid, Due: false, Reason: "within cadence window", AgeSecs: &ageSecs}) writeOK(w, BackupDueResponse{VMID: vmid, Due: false, Reason: "within cadence window", AgeSecs: &ageSecs, Target: echo})
}
// BackupTiersResponse is GET /backup/tiers (R-82): the tiers this agent serves, primary first.
// A controller that gets 404 here is talking to a PRE-R-82 agent and must fall back to the single
// untargeted tier — that 404 is the designed capability probe.
type BackupTiersResponse struct {
VMID int `json:"vmid"`
Tiers []BackupTierInfo `json:"tiers"`
}
// BackupTierInfo is one tier as advertised to the controller.
type BackupTierInfo struct {
Target string `json:"target"`
CadenceSeconds int64 `json:"cadence_seconds"`
Primary bool `json:"primary"`
}
func (s *Server) handleBackupTiers(w http.ResponseWriter, r *http.Request, vmid int) {
resp := BackupTiersResponse{VMID: vmid, Tiers: make([]BackupTierInfo, 0, len(s.tiers))}
for _, t := range s.tiers {
resp.Tiers = append(resp.Tiers, BackupTierInfo{
Target: t.TargetID,
CadenceSeconds: int64(t.Cadence.Seconds()),
Primary: t.Primary,
})
}
writeOK(w, resp)
}
// tierFromRequest resolves the `?target=` query parameter to a tier.
//
// THE COMPATIBILITY RULE (§4): NO target parameter → the PRIMARY tier, and the echoed target is
// EMPTY so the response marshals byte-identically to pre-R-82 (BackupDueResponse.Target is
// omitempty). An old controller cannot tell this agent from the old one.
//
// An UNKNOWN target is a 400, never a silent fallback to the primary: a controller asking about a
// tier this agent does not serve must find out, not be told about a different tier's freshness.
func (s *Server) tierFromRequest(w http.ResponseWriter, r *http.Request) (BackupTier, string, bool) {
want := strings.TrimSpace(r.URL.Query().Get("target"))
if want == "" {
return s.primaryTier(), "", true
}
for _, t := range s.tiers {
if t.TargetID == want {
return t, t.TargetID, true
}
}
writeStatus(w, http.StatusBadRequest, false, nil, "unknown backup target: "+want)
return BackupTier{}, "", false
}
// primaryTier returns the tier every untargeted endpoint acts on. tiers is never empty (New
// synthesizes one), but this stays defensive: a zero tier would silently disable backups.
func (s *Server) primaryTier() BackupTier {
for _, t := range s.tiers {
if t.Primary {
return t
}
}
if len(s.tiers) > 0 {
return s.tiers[0]
}
return BackupTier{TargetID: "", Cadence: defaultBackupCadence, Service: s.backups}
} }
// BackupStatusResponse is GET /backup/status (slice 8B): the current/last job phase + the latest // BackupStatusResponse is GET /backup/status (slice 8B): the current/last job phase + the latest
@@ -778,11 +911,20 @@ type BackupStatusResponse struct {
JobID string `json:"job_id,omitempty"` JobID string `json:"job_id,omitempty"`
Error string `json:"error,omitempty"` Error string `json:"error,omitempty"`
Backup *hub.Backup `json:"backup,omitempty"` // latest recorded backup for this guest Backup *hub.Backup `json:"backup,omitempty"` // latest recorded backup for this guest
// Target (R-82) echoes the tier; empty + omitted when untargeted (pre-R-82 bytes).
Target string `json:"target,omitempty"`
} }
func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) { func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) {
resp := BackupStatusResponse{VMID: vmid, Phase: PhaseIdle, Backup: s.latestBackupFor(r.Context(), vmid)} tier, echo, ok0 := s.tierFromRequest(w, r)
if job, ok := s.jobSnapshot(vmid); ok { if !ok0 {
return
}
// Untargeted keeps the pre-R-82 meaning EXACTLY: the primary tier's job, and the newest backup
// across ANY target (echo == "" → pickLatestBackup's match-any path).
resp := BackupStatusResponse{VMID: vmid, Phase: PhaseIdle, Target: echo,
Backup: s.pickLatestBackup(r.Context(), vmid, false, echo)}
if job, ok := s.jobSnapshot(backupJobKey{vmid: vmid, target: tier.TargetID}); ok {
resp.Phase = job.Phase resp.Phase = job.Phase
resp.JobID = job.JobID resp.JobID = job.JobID
resp.Error = job.Error resp.Error = job.Error
@@ -805,21 +947,34 @@ func (s *Server) handleRestoreTestStatus(w http.ResponseWriter, r *http.Request,
// latestBackupFor returns this guest's most recent backup from the store (nil if none). // latestBackupFor returns this guest's most recent backup from the store (nil if none).
func (s *Server) latestBackupFor(ctx context.Context, vmid int) *hub.Backup { func (s *Server) latestBackupFor(ctx context.Context, vmid int) *hub.Backup {
return s.pickLatestBackup(ctx, vmid, false) return s.pickLatestBackup(ctx, vmid, false, "")
} }
// latestSuccessfulBackupFor returns this guest's most recent SUCCESSFUL backup (nil if none) — // latestSuccessfulBackupFor returns this guest's most recent SUCCESSFUL backup (nil if none) —
// the basis for /backup/due (a failed backup must not satisfy the cadence). // the basis for /backup/due (a failed backup must not satisfy the cadence).
func (s *Server) latestSuccessfulBackupFor(ctx context.Context, vmid int) *hub.Backup { func (s *Server) latestSuccessfulBackupFor(ctx context.Context, vmid int) *hub.Backup {
return s.pickLatestBackup(ctx, vmid, true) return s.pickLatestBackup(ctx, vmid, true, "")
} }
func (s *Server) pickLatestBackup(ctx context.Context, vmid int, successOnly bool) *hub.Backup { // latestSuccessfulBackupForTarget is the R-82 per-tier twin: a tier's due-ness must be judged
// against ITS OWN newest successful backup. The store is already keyed by target, so this is a
// filter, not a data-model change — but WITHOUT it a fresh local backup would satisfy the PBS
// tier's cadence and the DR tier would never run.
func (s *Server) latestSuccessfulBackupForTarget(ctx context.Context, vmid int, target string) *hub.Backup {
return s.pickLatestBackup(ctx, vmid, true, target)
}
// pickLatestBackup returns the newest matching record. target "" matches ANY target (the pre-R-82
// behaviour, kept for the untargeted status endpoint).
func (s *Server) pickLatestBackup(ctx context.Context, vmid int, successOnly bool, target string) *hub.Backup {
var latest *hub.Backup var latest *hub.Backup
for _, b := range s.store.Backups(ctx) { for _, b := range s.store.Backups(ctx) {
if b.VMID != vmid || (successOnly && !b.Success) { if b.VMID != vmid || (successOnly && !b.Success) {
continue continue
} }
if target != "" && b.TargetID != target {
continue
}
bb := b bb := b
if latest == nil || bb.StartedAt > latest.StartedAt { // RFC3339 sorts lexically if latest == nil || bb.StartedAt > latest.StartedAt { // RFC3339 sorts lexically
latest = &bb latest = &bb