admin/felhom-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.97.0 — R-82 Slice A: per-target backup tiers (local daily + PBS weekly)

Browse Source 
Mechanism only. No box changes behaviour until a backup_targets entry is
added to its config (Slice D); an untouched config resolves to exactly one
tier and behaves byte-identically to v0.96.0.

- config: BackupTargetConfig + ExtraTargets + BackupTiers(); each tier carries
  its OWN cadence and retention (keep-last=3 is three days on a daily tier and
  three weeks on a weekly one). A missing cadence is REJECTED, not defaulted —
  a weekly DR tier silently running daily would fill the 37.2 GB datastore.
  main.go logs every rejection at ERROR.
- /backup/due?target= judges a tier against its OWN newest successful backup.
  Without that filter a fresh local backup satisfies the weekly PBS cadence and
  the DR tier never runs — today's bug, re-created in code.
- GET /backup/tiers advertises the tiers; a 404 is the controller's pre-R-82
  capability probe (Slice B).
- Jobs keyed by (vmid,target): single-flight is per tier, which is what lets
  the weekly night run both backups in ONE quiesce window. Job ids are unique
  per tier by construction, not by clock luck.
- One runner per tier: the runner holds target+retention as immutable state,
  so parameterising one runner would risk pairing tier A's target with tier B's
  retention.

COMPATIBILITY (frozen): untargeted /backup/due, POST /backup and
/backup/status keep the primary tier and the pre-R-82 response BYTES —
Target is omitempty and stays empty. The primary's job-id format is unchanged.

NOT changed: the local tier; PBS is still never pruned by the per-run flag
(keep_last defaults to 0 = never prune — enabling DR pruning is irreversible
and needs an operator ruling).

Tests 748->768. Red-proof #1 observed and restored.
Phase 0: felhom.eu/documentation/audits/SPIKE-r82-phase0-2026-07-26.md
This commit is contained in:
Claude Code
2026-07-26 12:20:58 +02:00
parent dfd5d731ee
commit 739b3c3b58
8 changed files with 980 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/localapi/server.go
+190 -35
View File
@@ -41,6 +41,20 @@ type BackupService interface {
BackupWithSnapshotHook(ctx context.Context, vmid int, onSnapshot func()) (hub.Backup, error)
}
// BackupTier (R-82) binds ONE backup tier's runner to its own policy. The agent builds one per
// resolved config tier; the local API serves each independently so "local daily + PBS weekly" is
// expressible over the wire, not just in config.
//
// COMPATIBILITY CONTRACT (load-bearing — the agent and controller deploy independently):
// the tier whose Primary is true is what EVERY untargeted endpoint acts on. An old controller
// never sends `?target=`, so it sees exactly the pre-R-82 behaviour and response bytes.
type BackupTier struct {
TargetID string
Cadence time.Duration
Primary bool
Service BackupService
}
// BackupStore records + reads the latest backup/restore-test state. Satisfied by *backup.Store.
type BackupStore interface {
RecordBackup(hub.Backup)
@@ -92,7 +106,11 @@ type Options struct {
// BackupCadence is the per-guest backup interval driving GET /backup/due (slice 8B). A guest
// is "due" when no successful backup is recorded OR the newest one is older than this. 0 → a
// safe default (24h). The hub-served policy is slice 10; this is the agent-local cadence.
// When BackupTiers is supplied this is IGNORED (the primary tier carries its own cadence).
BackupCadence time.Duration
// BackupTiers (R-82) is the resolved multi-tier policy, primary first. OPTIONAL: nil → one tier
// synthesized from Backups + BackupCadence, i.e. exactly the pre-R-82 behaviour.
BackupTiers []BackupTier
// Disk management (slice 8C) — OPTIONAL. When Disks + DiskGate are set, the /disks endpoints
// are served; otherwise they report "not configured". DiskGate authorizes the destructive
// (data-bearing) format path; Guests lists guests for the eject dependent-warning.
@@ -183,6 +201,12 @@ type backupJob struct {
Error string
}
// backupJobKey identifies one guest's job on ONE tier (R-82).
type backupJobKey struct {
vmid int
target string
}
// Server is the per-guest local API (doc 03 §6). It serves the agent's pinned self-signed leaf
// and authorizes every request against the token's guest only.
type Server struct {
@@ -196,6 +220,10 @@ type Server struct {
smart SmartReader // v0.95.0 Fix B: SMART for the /disks union path (optional)
tokens TokenAuthority
cadence time.Duration
// tiers (R-82) is the resolved backup-tier list, PRIMARY FIRST. Always non-empty: when the
// caller supplies no tiers it holds exactly one, synthesized from Backups+BackupCadence, which
// is the pre-R-82 shape.
tiers []BackupTier
logger *slog.Logger
now func() time.Time
@@ -247,7 +275,10 @@ type Server struct {
boundCheck func(string) bool
jobsMu sync.Mutex
jobs map[int]*backupJob // per-guest backup job state (slice 8B)
// jobs is per-guest-PER-TARGET backup job state (slice 8B; keyed by target too since R-82).
// Keying by vmid alone would let a PBS backup started inside the same quiesce window collide
// with the local one's single-flight and hand the caller the WRONG job id.
jobs map[backupJobKey]*backupJob
// agentic controller update (Phase 1): the swapper + per-guest single-flight gate.
swap *ControllerSwapper
@@ -336,9 +367,17 @@ func NewServer(o Options) (*Server, error) {
hostID: o.HostID,
agentVersion: o.AgentVersion,
logRing: o.LogRing,
jobs: map[int]*backupJob{},
jobs: map[backupJobKey]*backupJob{},
swapInFlight: map[int]bool{},
}
// R-82 tier resolution. Options.BackupTiers is authoritative when supplied; otherwise ONE tier
// is synthesized from Backups + BackupCadence — the pre-R-82 shape, so every existing caller
// (and every existing test) keeps working untouched. Exactly one tier is marked primary, and
// the primary is always first, because that is what the untargeted endpoints act on.
s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence)
if s.backups == nil && len(s.tiers) > 0 {
s.backups = s.tiers[0].Service
}
if s.escrowStagePath == "" {
s.escrowStagePath = escrow.StagedResticPasswordPath()
}
@@ -372,6 +411,7 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /rollback", s.withGuest(s.handleRollback))
mux.HandleFunc("POST /backup", s.withGuest(s.handleBackup))
mux.HandleFunc("GET /backup/due", s.withGuest(s.handleBackupDue))
mux.HandleFunc("GET /backup/tiers", s.withGuest(s.handleBackupTiers))
mux.HandleFunc("GET /backup/status", s.withGuest(s.handleBackupStatus))
mux.HandleFunc("GET /restore-test/status", s.withGuest(s.handleRestoreTestStatus))
// Host metrics (slice 9): host-wide health + per-storage capacity for the customer's monitoring
@@ -640,6 +680,8 @@ type BackupResponse struct {
VMID int `json:"vmid"`
JobID string `json:"job_id"`
Phase string `json:"phase"`
// Target (R-82) echoes the tier; empty + omitted for an untargeted request (pre-R-82 bytes).
Target string `json:"target,omitempty"`
}
func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int) {
@@ -653,17 +695,33 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
return
}
}
// Single-flight per guest: if a backup is already running for this guest, return that job
// (don't start a second concurrent vzdump). The controller polls /backup/status on it.
s.jobsMu.Lock()
if cur := s.jobs[vmid]; cur != nil && cur.Phase == PhaseRunning {
job := *cur
s.jobsMu.Unlock()
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: job.JobID, Phase: job.Phase}, "")
tier, echo, ok := s.tierFromRequest(w, r)
if !ok {
return
}
key := backupJobKey{vmid: vmid, target: tier.TargetID}
// Single-flight per guest PER TIER: if a backup is already running for this guest ON THIS
// TIER, return that job (don't start a second concurrent vzdump to the same target). A
// DIFFERENT tier is a different job — that is what lets the weekly night run both backups
// inside one quiesce window without the second call being handed the first one's id.
s.jobsMu.Lock()
if cur := s.jobs[key]; cur != nil && cur.Phase == PhaseRunning {
job := *cur
s.jobsMu.Unlock()
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: job.JobID, Phase: job.Phase, Target: echo}, "")
return
}
// Job ids must be unique PER TIER, and by construction rather than by clock luck: two tiers
// started inside the same nanosecond (the weekly both-due night, or any injected clock) would
// otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the
// pre-R-82 format byte-for-byte — an old controller stores this string and polls with it — so
// only the additive tiers carry the target segment.
jobID := "backup-" + strconv.Itoa(vmid) + "-" + strconv.FormatInt(s.now().UnixNano(), 10)
s.jobs[vmid] = &backupJob{JobID: jobID, Phase: PhaseRunning, StartedAt: s.now()}
if !tier.Primary && tier.TargetID != "" {
jobID = "backup-" + strconv.Itoa(vmid) + "-" + tier.TargetID + "-" + strconv.FormatInt(s.now().UnixNano(), 10)
}
s.jobs[key] = &backupJob{JobID: jobID, Phase: PhaseRunning, StartedAt: s.now()}
s.jobsMu.Unlock()
// Enqueue: a vzdump runs for minutes, so we do not block the request. The backup runs on the
@@ -680,42 +738,47 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
defer cancel()
// 8B.2: flip the job to `snapshotted` when the storage snapshot is taken, so the
// controller resumes its app early (snapshot mode only; in stop mode this never fires).
b, err := s.backups.BackupWithSnapshotHook(bctx, vmid, func() { s.markSnapshotted(vmid, jobID) })
b, err := tier.Service.BackupWithSnapshotHook(bctx, vmid, func() { s.markSnapshotted(key, jobID) })
if err != nil {
b.VMID = vmid
b.Success = false
if b.Error == "" {
b.Error = err.Error()
}
s.logger.Error("local-api: backup job failed", "vmid", vmid, "job", jobID, "err", err)
// TargetID is what the hub attributes the record to; a failed run must still say which
// tier failed, and the runner may not have set it on the error path.
if b.TargetID == "" {
b.TargetID = tier.TargetID
}
s.logger.Error("local-api: backup job failed", "vmid", vmid, "target", tier.TargetID, "job", jobID, "err", err)
} else {
s.logger.Info("local-api: backup job complete", "vmid", vmid, "job", jobID, "archive", b.Archive)
s.logger.Info("local-api: backup job complete", "vmid", vmid, "target", tier.TargetID, "job", jobID, "archive", b.Archive)
}
s.store.RecordBackup(b)
s.finishJob(vmid, jobID, b)
s.finishJob(key, jobID, b)
}()
writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "")
}
// markSnapshotted flips the guest's running job to the `snapshotted` phase (8B.2) — only if it is
// still the current job and still running (don't regress done/failed, and don't touch a newer job).
func (s *Server) markSnapshotted(vmid int, jobID string) {
func (s *Server) markSnapshotted(key backupJobKey, jobID string) {
s.jobsMu.Lock()
defer s.jobsMu.Unlock()
cur := s.jobs[vmid]
cur := s.jobs[key]
if cur == nil || cur.JobID != jobID || cur.Phase != PhaseRunning {
return
}
cur.Phase = PhaseSnapshotted
s.logger.Info("local-api: backup reached snapshotted (app may resume)", "vmid", vmid, "job", jobID)
s.logger.Info("local-api: backup reached snapshotted (app may resume)", "vmid", key.vmid, "target", key.target, "job", jobID)
}
// finishJob transitions the guest's job to done/failed (only if it is still the current job — a
// later job started after a single-flight gap must not be overwritten by an older one's result).
func (s *Server) finishJob(vmid int, jobID string, b hub.Backup) {
func (s *Server) finishJob(key backupJobKey, jobID string, b hub.Backup) {
s.jobsMu.Lock()
defer s.jobsMu.Unlock()
cur := s.jobs[vmid]
cur := s.jobs[key]
if cur == nil || cur.JobID != jobID {
return
}
@@ -730,10 +793,10 @@ func (s *Server) finishJob(vmid int, jobID string, b hub.Backup) {
}
// jobSnapshot returns a copy of the guest's current job (ok=false if none).
func (s *Server) jobSnapshot(vmid int) (backupJob, bool) {
func (s *Server) jobSnapshot(key backupJobKey) (backupJob, bool) {
s.jobsMu.Lock()
defer s.jobsMu.Unlock()
if j := s.jobs[vmid]; j != nil {
if j := s.jobs[key]; j != nil {
return *j, true
}
return backupJob{}, false
@@ -748,26 +811,96 @@ type BackupDueResponse struct {
Due bool `json:"due"`
Reason string `json:"reason"`
AgeSecs *int64 `json:"age_seconds,omitempty"` // age of the newest successful backup; null if none
// Target (R-82) echoes the tier this verdict is about. EMPTY (and omitted) for an untargeted
// request, which is what keeps the pre-R-82 response bytes identical for old controllers.
Target string `json:"target,omitempty"`
}
func (s *Server) handleBackupDue(w http.ResponseWriter, r *http.Request, vmid int) {
latest := s.latestSuccessfulBackupFor(r.Context(), vmid)
if latest == nil {
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "no successful backup recorded yet"})
tier, echo, ok := s.tierFromRequest(w, r)
if !ok {
return
}
age, ok := backupAge(latest.StartedAt, s.now())
if !ok {
latest := s.latestSuccessfulBackupForTarget(r.Context(), vmid, tier.TargetID)
if latest == nil {
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "no successful backup recorded yet", Target: echo})
return
}
age, ok2 := backupAge(latest.StartedAt, s.now())
if !ok2 {
// Unparseable timestamp: fail safe toward "due" so a backup still happens.
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "last backup time unparseable — treating as due"})
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "last backup time unparseable — treating as due", Target: echo})
return
}
ageSecs := int64(age.Seconds())
if age >= s.cadence {
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "older than cadence", AgeSecs: &ageSecs})
if age >= tier.Cadence {
writeOK(w, BackupDueResponse{VMID: vmid, Due: true, Reason: "older than cadence", AgeSecs: &ageSecs, Target: echo})
return
}
writeOK(w, BackupDueResponse{VMID: vmid, Due: false, Reason: "within cadence window", AgeSecs: &ageSecs})
writeOK(w, BackupDueResponse{VMID: vmid, Due: false, Reason: "within cadence window", AgeSecs: &ageSecs, Target: echo})
}
// BackupTiersResponse is GET /backup/tiers (R-82): the tiers this agent serves, primary first.
// A controller that gets 404 here is talking to a PRE-R-82 agent and must fall back to the single
// untargeted tier — that 404 is the designed capability probe.
type BackupTiersResponse struct {
VMID int `json:"vmid"`
Tiers []BackupTierInfo `json:"tiers"`
}
// BackupTierInfo is one tier as advertised to the controller.
type BackupTierInfo struct {
Target string `json:"target"`
CadenceSeconds int64 `json:"cadence_seconds"`
Primary bool `json:"primary"`
}
func (s *Server) handleBackupTiers(w http.ResponseWriter, r *http.Request, vmid int) {
resp := BackupTiersResponse{VMID: vmid, Tiers: make([]BackupTierInfo, 0, len(s.tiers))}
for _, t := range s.tiers {
resp.Tiers = append(resp.Tiers, BackupTierInfo{
Target: t.TargetID,
CadenceSeconds: int64(t.Cadence.Seconds()),
Primary: t.Primary,
})
}
writeOK(w, resp)
}
// tierFromRequest resolves the `?target=` query parameter to a tier.
//
// THE COMPATIBILITY RULE (§4): NO target parameter → the PRIMARY tier, and the echoed target is
// EMPTY so the response marshals byte-identically to pre-R-82 (BackupDueResponse.Target is
// omitempty). An old controller cannot tell this agent from the old one.
//
// An UNKNOWN target is a 400, never a silent fallback to the primary: a controller asking about a
// tier this agent does not serve must find out, not be told about a different tier's freshness.
func (s *Server) tierFromRequest(w http.ResponseWriter, r *http.Request) (BackupTier, string, bool) {
want := strings.TrimSpace(r.URL.Query().Get("target"))
if want == "" {
return s.primaryTier(), "", true
}
for _, t := range s.tiers {
if t.TargetID == want {
return t, t.TargetID, true
}
}
writeStatus(w, http.StatusBadRequest, false, nil, "unknown backup target: "+want)
return BackupTier{}, "", false
}
// primaryTier returns the tier every untargeted endpoint acts on. tiers is never empty (New
// synthesizes one), but this stays defensive: a zero tier would silently disable backups.
func (s *Server) primaryTier() BackupTier {
for _, t := range s.tiers {
if t.Primary {
return t
}
}
if len(s.tiers) > 0 {
return s.tiers[0]
}
return BackupTier{TargetID: "", Cadence: defaultBackupCadence, Service: s.backups}
}
// BackupStatusResponse is GET /backup/status (slice 8B): the current/last job phase + the latest
@@ -778,11 +911,20 @@ type BackupStatusResponse struct {
JobID string `json:"job_id,omitempty"`
Error string `json:"error,omitempty"`
Backup *hub.Backup `json:"backup,omitempty"` // latest recorded backup for this guest
// Target (R-82) echoes the tier; empty + omitted when untargeted (pre-R-82 bytes).
Target string `json:"target,omitempty"`
}
func (s *Server) handleBackupStatus(w http.ResponseWriter, r *http.Request, vmid int) {
resp := BackupStatusResponse{VMID: vmid, Phase: PhaseIdle, Backup: s.latestBackupFor(r.Context(), vmid)}
if job, ok := s.jobSnapshot(vmid); ok {
tier, echo, ok0 := s.tierFromRequest(w, r)
if !ok0 {
return
}
// Untargeted keeps the pre-R-82 meaning EXACTLY: the primary tier's job, and the newest backup
// across ANY target (echo == "" → pickLatestBackup's match-any path).
resp := BackupStatusResponse{VMID: vmid, Phase: PhaseIdle, Target: echo,
Backup: s.pickLatestBackup(r.Context(), vmid, false, echo)}
if job, ok := s.jobSnapshot(backupJobKey{vmid: vmid, target: tier.TargetID}); ok {
resp.Phase = job.Phase
resp.JobID = job.JobID
resp.Error = job.Error
@@ -805,21 +947,34 @@ func (s *Server) handleRestoreTestStatus(w http.ResponseWriter, r *http.Request,
// latestBackupFor returns this guest's most recent backup from the store (nil if none).
func (s *Server) latestBackupFor(ctx context.Context, vmid int) *hub.Backup {
return s.pickLatestBackup(ctx, vmid, false)
return s.pickLatestBackup(ctx, vmid, false, "")
}
// latestSuccessfulBackupFor returns this guest's most recent SUCCESSFUL backup (nil if none) —
// the basis for /backup/due (a failed backup must not satisfy the cadence).
func (s *Server) latestSuccessfulBackupFor(ctx context.Context, vmid int) *hub.Backup {
return s.pickLatestBackup(ctx, vmid, true)
return s.pickLatestBackup(ctx, vmid, true, "")
}
func (s *Server) pickLatestBackup(ctx context.Context, vmid int, successOnly bool) *hub.Backup {
// latestSuccessfulBackupForTarget is the R-82 per-tier twin: a tier's due-ness must be judged
// against ITS OWN newest successful backup. The store is already keyed by target, so this is a
// filter, not a data-model change — but WITHOUT it a fresh local backup would satisfy the PBS
// tier's cadence and the DR tier would never run.
func (s *Server) latestSuccessfulBackupForTarget(ctx context.Context, vmid int, target string) *hub.Backup {
return s.pickLatestBackup(ctx, vmid, true, target)
}
// pickLatestBackup returns the newest matching record. target "" matches ANY target (the pre-R-82
// behaviour, kept for the untargeted status endpoint).
func (s *Server) pickLatestBackup(ctx context.Context, vmid int, successOnly bool, target string) *hub.Backup {
var latest *hub.Backup
for _, b := range s.store.Backups(ctx) {
if b.VMID != vmid || (successOnly && !b.Success) {
continue
}
if target != "" && b.TargetID != target {
continue
}
bb := b
if latest == nil || bb.StartedAt > latest.StartedAt { // RFC3339 sorts lexically
latest = &bb