diff --git a/CHANGELOG.md b/CHANGELOG.md index 84f6768..d6b368d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,78 @@ ## Changelog +### v0.174.0 — R-82 Slice B: one quiesce window, two backup tiers (2026-07-26) + +**MinAgent UNCHANGED — deliberately.** This release degrades gracefully against ANY older agent; it +does not require v0.97.0. Against a pre-R-82 agent it uses the untargeted single-tier path exactly as +before, logs the degrade once, and **still takes the backup**. + +The agent gained per-target backup tiers in v0.97.0 ("local daily + PBS weekly"). The **controller** +owns quiescing, so the multi-tier schedule has to be reconciled here: on the weekly night both tiers +come due at once, and two quiesce cycles would mean **two app outages for one night's work** — +undoing the entire argument for weekly-over-daily. + +### The dedup rule (specified, not emergent) + +| local due | PBS due | result | +|---|---|---| +| yes | no | one quiesce, local backup | +| no | yes | one quiesce, PBS backup | +| **yes** | **yes** | **ONE quiesce window, BOTH backups inside it — never two cycles** | +| no | no | no quiesce | + +### Added +- **`quiesce.TieredBackend`** (optional extension to `Backend`) + `quiesce.BackupTier`, + `ErrTiersUnsupported`. A backend that does not implement it — or whose `Tiers` returns + `ErrTiersUnsupported` — drives the pre-R-82 single-tier path unchanged. +- **`agentapi` per-tier client**: `BackupTiers`, `BackupDueFor`, `StartBackupFor`, + `BackupStatusFor` (`internal/agentapi/backup_tiers.go`). `targetQuery("")` yields an EMPTY suffix, + so an untargeted call hits the untargeted route byte-for-byte. +- **`Loop.resolveDueTiers`** — the dedup rule in one place, returning due tiers in AGENT ORDER. +- **`Loop.quiesceAndPollTiers` + `pollTier`** — one marker, one stop, N sequential backups, one + resume, tail polled to completion. + +### Capability detection +`GET /backup/tiers` 404 ⇒ pre-R-82 agent. This is the project's documented ROUTE-PROBE mechanism +(`internal/agentapi/features.go`: "a route that shipped together with the coupled semantics either +answers (2xx ⇒ supported) or 404s"). It is **not** registered in the `featureProbes` table on +purpose: that table answers a yes/no at a UI entry point, whereas the loop needs the tier LIST +itself, so a table row would be a second probe of the same route for no gain. The degrade is logged +**exactly once per process** — once because it is a steady state during a rollout, never zero times +because a silent degrade is indistinguishable from multi-tier working. + +### Two decisions worth stating plainly + +**The app stays quiesced until the LAST tier snapshots.** Resuming after tier 1's snapshot would +leave the following tier capturing a RUNNING app — losing app-consistency on exactly the DR tier we +most want it on. **Consequence, user-visible:** on the both-due night downtime is +*(first tier's full backup)* + *(last tier's snapshot)*, not one snapshot. Tiers must therefore run +**fast-first**: vzdump holds a guest lock so they are necessarily sequential, and the agent +advertises primary (local) first — local-then-PBS makes downtime ≈ local backup + PBS snapshot, +whereas the reverse would be ≈ PBS backup + local snapshot, far worse. + +**A manual "Mentés most" covers EVERY tier**, in one window, due-ness ignored. A manual run that +silently skipped the DR tier would be the same applied-and-empty fault in a different costume. + +### Resilience (unchanged guarantees, extended per tier) +- Marker written BEFORE anything stops; unquiesce guaranteed by `defer` and fires **exactly once** + no matter which tier fails; a crash between two backups leaves the marker and `Recover()` restarts + the stacks at startup. +- One tier failing to START does not prevent the other tier's backup, and the app still resumes once. +- One tier's due-check erroring does not drop the other tier's backup. +- An agent advertising ZERO tiers falls back to the untargeted path — never "nothing to do". +- The window gate's safety valve now evaluates the OLDEST (most overdue) due tier, so a stale DR + tier cannot be starved by a fresher local one (`oldestAge`; a never-backed-up tier wins outright). + +### Tests ++11 in `internal/quiesce/tiers_test.go`; full suite green. Red-proofs observed and restored: +- **#3 both-due night** — a per-tier cycle instead of one window fails with + `want EXACTLY 1 stop and 1 start, got stops=2 starts=2`. The COUNT is the assertion; asserting + only "both backups ran" would pass against a double-quiesce implementation. +- **#2 new controller ↔ old agent** — treating `ErrTiersUnsupported` as "nothing due" fails with + `OLD AGENT: a backup MUST still be taken via the untargeted path; got started=[]`. The hollow + version of this test asserts only "no error", which passes while silently skipping the backup. + + ### v0.173.0 — R-77: endpoint-drift detection, samba protected-set gate, channel log honesty (2026-07-26) Source: `felhom.eu/documentation/audits/DIAG-agent-channel-2026-07-26.md`. diff --git a/REUSE.md b/REUSE.md index df0340e..0d3c45e 100644 --- a/REUSE.md +++ b/REUSE.md @@ -72,6 +72,8 @@ | `infra.SambaContainerName` / `SambaPassdbVolume` / `SambaPassdbMount` | controller/internal/infra/samba.go | consts | single source of truth for the samba container identity | the compose renderer interpolates them; stacks/backup/monitor read them. The CONTAINER name (`felhom-samba`) is NOT the stack name (`samba`) — `EffectiveProtected` needs the container one | | `sambaWriteAtomic` | controller/internal/stacks/samba.go | `(path, data, mode) error` | samba smb.conf/compose writes | tmp+**fsync**+rename (the only one of these that fsyncs). Fourth atomic-write helper in the tree — see §6 | | `Loop.writeMarker` / `Recover` | controller/internal/quiesce/quiesce.go | `(m Marker)` / `()` | Quiesce crash-safety | Marker written BEFORE stopping stacks; Recover restarts stranded stacks at boot | +| `quiesce.TieredBackend` + `Loop.resolveDueTiers` / `quiesceAndPollTiers` | controller/internal/quiesce/tiers.go, quiesce.go | `Tiers/DueFor/StartBackupFor/BackupStatusFor`; `resolveDueTiers(ctx) ([]dueTier,bool,error)` | THE R-82 multi-tier backup schedule — several whole-guest tiers (local daily + PBS weekly) reconciled into ONE quiesce window | **Both tiers due ⇒ ONE stop/start pair**, never two (two = two app outages for one night). Tiers run SEQUENTIALLY (vzdump holds a guest lock) and the app stays down until the LAST tier snapshots — resuming earlier loses app-consistency on the DR tier. Order is fast-first (agent advertises primary first) or downtime blows up. `ErrTiersUnsupported` (route 404) ⇒ pre-R-82 agent ⇒ degrade to the untargeted path and **STILL BACK UP** — never read it as "nothing due". | +| `agentapi.BackupTiers` / `BackupDueFor` / `StartBackupFor` / `BackupStatusFor` | controller/internal/agentapi/backup_tiers.go | `(ctx[, target]) (…, error)` | The per-tier agent surface (agent >= v0.97.0) | `targetQuery("")` returns an EMPTY suffix so an untargeted call hits the pre-R-82 route byte-for-byte. `BackupTiers` maps a 404 to `ErrTiersUnsupported` — the documented ROUTE-PROBE capability signal, NOT a `featureProbes` row (the loop needs the tier LIST, not a yes/no). | ### Compose ops / stack lifecycle diff --git a/controller/README.md b/controller/README.md index 6a1540b..3a2e46a 100644 --- a/controller/README.md +++ b/controller/README.md @@ -701,6 +701,20 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri > the three daily legs via `scheduler.UpdateDaily` and takes effect **without a restart**. Precedence: > settings > controller.yaml `db_dump_schedule` > "02:30". See `internal/backupwindow`. +> **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest +> backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0, +> `GET /backup/tiers`). The controller owns quiescing, so it reconciles them: it collects EVERY due +> tier up front and runs them inside **ONE quiesce window** — one stop, N sequential backups (vzdump +> holds a guest lock), one resume. Two cycles on the weekly night would mean two app outages for one +> night's work. The app stays quiesced until the **LAST** tier snapshots, so every tier is +> app-consistent; the consequence is that both-due-night downtime is *(first tier's full backup)* + +> *(last tier's snapshot)*, which is why tiers run fast-first (the agent advertises primary/local +> first). A manual **"Mentés most"** covers every tier, due-ness ignored. The window gate's safety +> valve evaluates the OLDEST due tier, so a stale DR tier cannot be starved by a fresher local one. +> Against a **pre-R-82 agent** (`/backup/tiers` 404s) the loop degrades to the single untargeted +> tier, logs it once, and still takes the backup — MinAgent is unchanged. See `internal/quiesce` +> (`tiers.go`) and `internal/agentapi/backup_tiers.go`. + > **Atomic dump writes (v0.118.0, CAMPAIGN-3 F7).** BOTH dump paths are crash-safe: the DB dump > (`dbdump.go` DumpOne) and the Docker-volume dump (`DumpAppVolumes`) write to a `.tmp` sibling, fsync, > then `os.Rename` over the restore point ONLY on success. A mid-write failure (a NFS cut mid-tar, an diff --git a/controller/cmd/controller/main.go b/controller/cmd/controller/main.go index 843dd28..cf449fc 100644 --- a/controller/cmd/controller/main.go +++ b/controller/cmd/controller/main.go @@ -1704,6 +1704,41 @@ func (b quiesceBackend) BackupStatus(ctx context.Context) (string, error) { return r.Phase, err } +// ---- R-82: the tiered surface (quiesce.TieredBackend) ------------------------------------ +// +// quiesceBackend satisfies quiesce.TieredBackend as well, so the loop schedules per tier when the +// agent supports it. Against a PRE-R-82 agent, Tiers returns quiesce.ErrTiersUnsupported and the +// loop degrades to the untargeted methods above — still taking a backup, never skipping one. + +func (b quiesceBackend) Tiers(ctx context.Context) ([]quiesce.BackupTier, error) { + r, err := b.c.BackupTiers(ctx) + if errors.Is(err, agentapi.ErrTiersUnsupported) { + // Translate the transport-layer probe into the loop's vocabulary; the loop keys on this. + return nil, quiesce.ErrTiersUnsupported + } + if err != nil { + return nil, err + } + out := make([]quiesce.BackupTier, 0, len(r.Tiers)) + for _, t := range r.Tiers { + out = append(out, quiesce.BackupTier{Target: t.Target, Primary: t.Primary}) + } + return out, nil +} + +func (b quiesceBackend) DueFor(ctx context.Context, target string) (bool, *int64, error) { + r, err := b.c.BackupDueFor(ctx, target) + return r.Due, r.AgeSecs, err +} +func (b quiesceBackend) StartBackupFor(ctx context.Context, target string) (string, error) { + r, err := b.c.StartBackupFor(ctx, target) + return r.JobID, err +} +func (b quiesceBackend) BackupStatusFor(ctx context.Context, target string) (string, error) { + r, err := b.c.BackupStatusFor(ctx, target) + return r.Phase, err +} + // startQuiesceLoop wires + starts the slice-8B quiesce loop when the local API is configured and // quiesce is enabled. It Recovers (restarts stacks left stopped by a mid-quiesce crash) before // starting the loop goroutine. Non-fatal: any misconfig disables the loop with a log line. diff --git a/controller/internal/agentapi/backup_tiers.go b/controller/internal/agentapi/backup_tiers.go new file mode 100644 index 0000000..2ab448b --- /dev/null +++ b/controller/internal/agentapi/backup_tiers.go @@ -0,0 +1,103 @@ +package agentapi + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "net/url" +) + +// R-82 Slice B — the per-tier backup surface (agent >= v0.97.0). +// +// Every method here is ADDITIVE. The untargeted BackupDue/StartBackup/BackupStatus keep their exact +// pre-R-82 meaning and are still the single-tier path used against an older agent. + +// ErrTiersUnsupported reports that this agent does not serve GET /backup/tiers — it predates R-82. +// It is the DESIGNED capability probe (the route 404s), not a fault. The caller MUST degrade to the +// untargeted single-tier path and still take a backup; concluding "nothing to do" from it would +// silently stop backups during a fleet rollout. +var ErrTiersUnsupported = errors.New("agentapi: agent does not serve /backup/tiers (pre-R-82)") + +// BackupTierInfo is one advertised tier. +type BackupTierInfo struct { + Target string `json:"target"` + CadenceSeconds int64 `json:"cadence_seconds"` + Primary bool `json:"primary"` +} + +// TiersResponse mirrors the agent's GET /backup/tiers payload. +type TiersResponse struct { + VMID int `json:"vmid"` + Tiers []BackupTierInfo `json:"tiers"` +} + +// BackupTiers lists the agent's backup tiers, primary first. +// Returns ErrTiersUnsupported (wrapped) on a pre-R-82 agent — key on it with errors.Is. +func (c *Client) BackupTiers(ctx context.Context) (TiersResponse, error) { + var out TiersResponse + body, err := c.get(ctx, "/backup/tiers") + if err != nil { + var se *StatusError + if errors.As(err, &se) && se.Code == http.StatusNotFound { + return out, ErrTiersUnsupported + } + return out, err + } + if err := json.Unmarshal(body, &out); err != nil { + return out, fmt.Errorf("agentapi: decode /backup/tiers: %w", err) + } + return out, nil +} + +// targetQuery renders the ?target= suffix. An EMPTY target yields an empty string, so the caller +// hits the untargeted route byte-for-byte — that is what keeps the pre-R-82 contract intact when +// this client talks to an older agent. +func targetQuery(target string) string { + if target == "" { + return "" + } + return "?target=" + url.QueryEscape(target) +} + +// BackupDueFor reports whether THIS TIER is due. A fresh backup on another tier must not satisfy it +// — that filtering happens agent-side (latestSuccessfulBackupForTarget); this just asks per tier. +func (c *Client) BackupDueFor(ctx context.Context, target string) (DueResponse, error) { + var out DueResponse + body, err := c.get(ctx, "/backup/due"+targetQuery(target)) + if err != nil { + return out, err + } + if err := json.Unmarshal(body, &out); err != nil { + return out, fmt.Errorf("agentapi: decode /backup/due (target %q): %w", target, err) + } + return out, nil +} + +// StartBackupFor enqueues a backup of this guest ON THE GIVEN TIER. +func (c *Client) StartBackupFor(ctx context.Context, target string) (BackupResponse, error) { + var out BackupResponse + body, err := c.post(ctx, "/backup"+targetQuery(target), struct{}{}) + if err != nil { + return out, err + } + if err := json.Unmarshal(body, &out); err != nil { + return out, fmt.Errorf("agentapi: decode POST /backup (target %q): %w", target, err) + } + return out, nil +} + +// BackupStatusFor reports THIS TIER's current/last job phase. Jobs are keyed per tier agent-side, +// so polling the wrong target would report a different tier's progress. +func (c *Client) BackupStatusFor(ctx context.Context, target string) (StatusResponse, error) { + var out StatusResponse + body, err := c.get(ctx, "/backup/status"+targetQuery(target)) + if err != nil { + return out, err + } + if err := json.Unmarshal(body, &out); err != nil { + return out, fmt.Errorf("agentapi: decode /backup/status (target %q): %w", target, err) + } + return out, nil +} diff --git a/controller/internal/quiesce/quiesce.go b/controller/internal/quiesce/quiesce.go index d764e43..9b520b3 100644 --- a/controller/internal/quiesce/quiesce.go +++ b/controller/internal/quiesce/quiesce.go @@ -98,6 +98,8 @@ type Loop struct { // two can never stop the same stacks concurrently (the persisted marker covers crash-safety across // restarts; this covers concurrency within the process — which a manual trigger introduces). mu sync.Mutex + // degradeOnce reports the pre-R-82 agent fallback exactly once per process (see tiers.go). + degradeOnce sync.Once } // New builds a Loop with sane defaults for any unset duration. @@ -177,26 +179,47 @@ func (l *Loop) runOnce(ctx context.Context) error { return nil } - due, ageSecs, err := l.backend.Due(ctx) + // R-82: resolve EVERY due tier up front. This is the dedup rule (tiers.go): both tiers due on + // the weekly night yields ONE window with two backups, never two stop/start cycles. + dueTiers, _, err := l.resolveDueTiers(ctx) if err != nil { return fmt.Errorf("check due: %w", err) } - if !due { + if len(dueTiers) == 0 { return nil } // Window gate (Part 3) — SCHEDULED path only. TriggerNow calls quiesceAndPoll directly and is // never gated. Disabled when no window fn is wired (pre-v0.168.0 behavior). + // + // With several tiers due, the gate is evaluated against the OLDEST (most overdue) tier's age, + // so the safety valve — "run regardless of the window once the last success is older than + // cadence+24h" — cannot be suppressed by a fresher sibling tier. if l.windowStartFn != nil { window := l.windowStartFn() - if !scheduledRunAllowed(l.now().In(budapestLocation()), window, ageSecs, l.cadence) { + if !scheduledRunAllowed(l.now().In(budapestLocation()), window, oldestAge(dueTiers), l.cadence) { from, to := gateBounds(window) l.logger.Printf("[DEBUG] [quiesce] scheduled backup due but outside the backup window [%s–%s) — deferring to the next poll inside it", from, to) return nil } } - return l.quiesceAndPoll(ctx) + return l.quiesceAndPollTiers(ctx, dueTiers) +} + +// oldestAge returns the largest (most overdue) age among the due tiers; nil when any tier has never +// backed up (nil age = "never", which is maximally overdue and must win). +func oldestAge(tiers []dueTier) *int64 { + var oldest *int64 + for _, t := range tiers { + if t.ageSecs == nil { + return nil // never backed up — the strongest claim on the safety valve + } + if oldest == nil || *t.ageSecs > *oldest { + oldest = t.ageSecs + } + } + return oldest } // TriggerNow forces an app-consistent backup NOW (the manual "Mentés most" action), bypassing the @@ -220,7 +243,10 @@ func (l *Loop) TriggerNow() error { ctx, cancel := context.WithTimeout(context.Background(), l.maxQuiesce+5*time.Minute) defer cancel() l.logger.Printf("[INFO] [quiesce] manual backup requested — quiescing now") - if err := l.quiesceAndPoll(ctx); err != nil { + // Manual runs bypass due-ness (that is the point) but must still cover EVERY tier, in one + // window. A manual "Mentés most" that silently skipped the DR tier would be the same + // applied-and-empty fault in a different costume. + if err := l.quiesceAndPollTiers(ctx, l.allTiersForManualRun(ctx)); err != nil { l.logger.Printf("[ERROR] [quiesce] manual backup cycle error: %v", err) } }() @@ -232,6 +258,49 @@ func (l *Loop) TriggerNow() error { // MUST hold l.mu. Unquiesce is guaranteed via the deferred closure (backup error, status-poll error, // the max-quiesce bound, or context cancellation all still restart the stacks and clear the marker). func (l *Loop) quiesceAndPoll(ctx context.Context) error { + return l.quiesceAndPollTiers(ctx, []dueTier{{target: ""}}) +} + +// allTiersForManualRun lists every tier a manual run should cover: all advertised tiers on an R-82 +// agent, or the single untargeted tier otherwise. Due-ness is deliberately NOT consulted. +func (l *Loop) allTiersForManualRun(ctx context.Context) []dueTier { + tb, ok := l.backend.(TieredBackend) + if !ok { + return []dueTier{{target: ""}} + } + tiers, err := tb.Tiers(ctx) + if err != nil || len(tiers) == 0 { + if errors.Is(err, ErrTiersUnsupported) { + l.logTierDegradeOnce() + } else if err != nil { + l.logger.Printf("[WARN] [quiesce] manual run: tier list unavailable (%v) — using the untargeted tier", err) + } + return []dueTier{{target: ""}} + } + out := make([]dueTier, 0, len(tiers)) + for _, t := range tiers { + out = append(out, dueTier{target: t.Target}) + } + return out +} + +// quiesceAndPollTiers is the R-82 multi-tier cycle: ONE marker, ONE stop, N backups run +// SEQUENTIALLY inside the window, ONE resume, then the tail polled to completion. +// +// Why sequential: vzdump takes a guest lock, so a second backup cannot start until the first +// finishes. Why the app stays down until the LAST tier snapshots: the whole point of quiescing is +// app-consistency, and resuming after tier 1's snapshot would leave tier 2 capturing a RUNNING app. +// Consequence, stated plainly because it is user-visible: on the both-due night downtime is +// (first tier's full backup) + (last tier's snapshot), not one snapshot. Tier ORDER therefore +// matters — see resolveDueTiers. +// +// Crash-safety is unchanged and non-negotiable: the marker is written BEFORE anything stops, +// unquiesce is guaranteed by defer, and it fires exactly once no matter which tier fails. A crash +// between two backups leaves the marker on disk and Recover() restarts the stacks at startup. +func (l *Loop) quiesceAndPollTiers(ctx context.Context, tiers []dueTier) error { + if len(tiers) == 0 { + return nil + } running := l.stacks.RunningAppStacks() marker := Marker{Active: true, StartedAt: l.now(), StoppedStacks: running} if err := l.writeMarker(marker); err != nil { @@ -253,63 +322,103 @@ func (l *Loop) quiesceAndPoll(ctx context.Context) error { } defer unquiesce("deferred") - l.logger.Printf("[INFO] [quiesce] backup due — quiescing %d stack(s): %v", len(running), running) - for _, s := range running { - if err := l.stacks.StopStack(s); err != nil { - l.logger.Printf("[ERROR] [quiesce] stop %s: %v (continuing)", s, err) + l.logger.Printf("[INFO] [quiesce] backup due on %d tier(s) — quiescing %d stack(s): %v", + len(tiers), len(running), running) + for _, st := range running { + if err := l.stacks.StopStack(st); err != nil { + l.logger.Printf("[ERROR] [quiesce] stop %s: %v (continuing)", st, err) } } - jobID, err := l.backend.StartBackup(ctx) - if err != nil { - unquiesce("backup start failed") - return fmt.Errorf("start backup: %w", err) - } - marker.JobID = jobID - _ = l.writeMarker(marker) // best-effort: record the job id for diagnosis - l.logger.Printf("[INFO] [quiesce] backup job %s started — polling to completion", jobID) - deadline := l.now().Add(l.maxQuiesce) + var firstErr error + + // ONE window, N tiers, sequential. The app resumes only after the LAST tier snapshots. + for i, t := range tiers { + label := tierLabel(t.target) + last := i == len(tiers)-1 + + jobID, err := l.startBackupOn(ctx, t.target) + if err != nil { + l.logger.Printf("[ERROR] [quiesce] start backup on tier %s: %v", label, err) + if firstErr == nil { + firstErr = fmt.Errorf("start backup on %s: %w", label, err) + } + // A tier that will not start must not hold the app down for the others. + if last { + unquiesce("last tier failed to start") + } + continue + } + marker.JobID = jobID + _ = l.writeMarker(marker) // best-effort: record the CURRENT tier's job id for diagnosis + l.logger.Printf("[INFO] [quiesce] tier %s: backup job %s started — polling", label, jobID) + + phase, perr := l.pollTier(ctx, t.target, jobID, label, deadline, last, &unquiesced, unquiesce) + if perr != nil && firstErr == nil { + firstErr = perr + } + if phase == phaseFailed { + l.logger.Printf("[WARN] [quiesce] tier %s: backup job %s failed", label, jobID) + } + // The max-quiesce guard already unquiesced; keep going so the remaining tiers still run + // (the app is up — the backups simply continue without the quiesce guarantee, which is + // strictly better than skipping the DR tier entirely). + } + + // Belt: if every tier failed to start, nothing above unquiesced. The deferred call covers it, + // but doing it here keeps the "resume as soon as possible" property explicit. + unquiesce("cycle complete") + return firstErr +} + +// pollTier polls ONE tier's job. It returns the terminal (or snapshotted-at-deadline) phase. +// +// The app is resumed ONLY when this is the LAST tier — that is what keeps every tier +// app-consistent while still costing exactly one stop/start pair. For a non-last tier the loop +// waits for a TERMINAL phase (done/failed), because vzdump holds the guest lock and the next tier +// cannot start until this one truly finishes. +func (l *Loop) pollTier(ctx context.Context, target, jobID, label string, deadline time.Time, + last bool, unquiesced *bool, unquiesce func(string)) (string, error) { for { if !l.now().Before(deadline) { - l.logger.Printf("[WARN] [quiesce] max-quiesce-duration (%s) exceeded for job %s — unquiescing while the backup continues on the agent", - l.maxQuiesce, jobID) + l.logger.Printf("[WARN] [quiesce] max-quiesce-duration (%s) exceeded on tier %s (job %s) — unquiescing while the backup continues on the agent", + l.maxQuiesce, label, jobID) unquiesce("max-quiesce guard") - return nil + return "", nil } - phase, err := l.backend.BackupStatus(ctx) + phase, err := l.backupStatusOn(ctx, target) if err != nil { unquiesce("status poll failed") - return fmt.Errorf("poll backup status: %w", err) + return "", fmt.Errorf("poll backup status on %s: %w", label, err) } switch phase { case phaseSnapshotted: - // 8B.2: the storage snapshot is taken — the app-stopped state is captured, so the app - // may resume NOW (downtime = until-snapshot, not until-backup-done) with no loss of - // app-consistency. unquiesce is idempotent (fires once); we then KEEP polling to - // done/failed so a new backup isn't started until this one truly finishes (and so a - // post-snapshot failure is observed). The marker is cleared on resume — a crash in this - // tail leaves the app already up, nothing to recover. - if !unquiesced { - l.logger.Printf("[INFO] [quiesce] backup job %s snapshotted — resuming app early (8B.2)", jobID) - unquiesce("snapshotted (early resume)") + // 8B.2 early resume — but ONLY on the last tier. Resuming here on a non-last tier would + // leave the following tier capturing a running app, losing app-consistency for exactly + // the DR tier we most want it on. + if last && !*unquiesced { + l.logger.Printf("[INFO] [quiesce] tier %s: job %s snapshotted — resuming app early (8B.2)", label, jobID) + unquiesce("snapshotted (early resume, last tier)") } case phaseDone: - // Fallback (stop/downgraded mode never emits snapshotted): resume at done, exactly 8B. - l.logger.Printf("[INFO] [quiesce] backup job %s done", jobID) - unquiesce("backup done") - return nil + if last { + l.logger.Printf("[INFO] [quiesce] tier %s: backup job %s done", label, jobID) + unquiesce("backup done") + } else { + l.logger.Printf("[INFO] [quiesce] tier %s: backup job %s done — next tier may start (app still quiesced)", label, jobID) + } + return phaseDone, nil case phaseFailed: - // If we already resumed at snapshotted, the app is up — just note the backup failed - // (recorded for the agent's due window when it stores the failed result). - l.logger.Printf("[WARN] [quiesce] backup job %s failed", jobID) - unquiesce("backup failed") - return nil + if last { + unquiesce("backup failed") + } + return phaseFailed, nil } select { case <-ctx.Done(): unquiesce("controller shutting down") - return ctx.Err() + return "", ctx.Err() case <-time.After(l.statusPoll): } } diff --git a/controller/internal/quiesce/tiers.go b/controller/internal/quiesce/tiers.go new file mode 100644 index 0000000..5b2295f --- /dev/null +++ b/controller/internal/quiesce/tiers.go @@ -0,0 +1,147 @@ +package quiesce + +import ( + "context" + "errors" +) + +// R-82 Slice B — one quiesce window, two tiers. +// +// The agent gained per-target backup tiers in v0.97.0 ("local daily + PBS weekly"). The controller +// owns quiescing, so the multi-tier schedule has to be reconciled HERE: on the weekly night both +// tiers come due at once, and running two quiesce cycles would mean **two app outages for one +// night's work** — which would undo the entire argument for weekly-over-daily. +// +// THE DEDUP RULE (specified, not emergent): +// +// local due | PBS due | result +// ----------+---------+--------------------------------------------------------------- +// yes | no | one quiesce, local backup +// no | yes | one quiesce, PBS backup +// yes | yes | ONE quiesce window, BOTH backups inside it — never two cycles +// no | no | no quiesce +// +// ErrTiersUnsupported is returned by TieredBackend.Tiers when the agent does not serve +// GET /backup/tiers — i.e. it predates R-82 (the endpoint 404s). It is the DESIGNED capability +// probe, not an error condition: the loop degrades to the single untargeted tier and logs it once. +// +// It must NEVER be treated as "nothing to do". A new controller meeting an old agent must still +// back up; concluding "not due" from an unrecognised response would silently stop backups +// fleet-wide during a rollout — the exact failure this project has hit before (controller v0.154.0, +// agent v0.91.0, the hub allowedEventTypes 400 in R-77). +var ErrTiersUnsupported = errors.New("quiesce: agent does not serve /backup/tiers (pre-R-82)") + +// BackupTier is one tier as advertised by the agent, primary first. +type BackupTier struct { + Target string + Primary bool +} + +// TieredBackend is the OPTIONAL R-82 extension to Backend. A backend that does not implement it +// (or whose Tiers returns ErrTiersUnsupported) drives the pre-R-82 single-tier path unchanged. +// +// The untargeted Backend methods are NOT redundant: they remain the single-tier path, and the agent +// guarantees they keep their exact pre-R-82 meaning and response bytes. +type TieredBackend interface { + Backend + // Tiers lists the agent's backup tiers, primary first. ErrTiersUnsupported ⇒ pre-R-82 agent. + Tiers(ctx context.Context) ([]BackupTier, error) + DueFor(ctx context.Context, target string) (due bool, ageSecs *int64, err error) + StartBackupFor(ctx context.Context, target string) (jobID string, err error) + BackupStatusFor(ctx context.Context, target string) (phase string, err error) +} + +// dueTier is a tier this cycle must back up. +type dueTier struct { + target string // "" = the untargeted single-tier path (pre-R-82 agent) + ageSecs *int64 +} + +// resolveDueTiers answers "what must this cycle back up?" — the dedup rule above, in one place. +// +// Returns the due tiers IN AGENT ORDER (primary first). That order is deliberate and it is a +// downtime decision, not cosmetics: tiers run SEQUENTIALLY because vzdump holds a guest lock, and +// the app stays stopped until the LAST tier has snapshotted. Running the fast local tier first and +// the slow WAN/PBS tier last makes downtime ≈ (local backup) + (PBS snapshot); the reverse order +// would make it ≈ (PBS backup) + (local snapshot), which is far worse. +// +// degraded is true when the agent is pre-R-82 and the caller must use the untargeted path. +func (l *Loop) resolveDueTiers(ctx context.Context) (due []dueTier, degraded bool, err error) { + tb, ok := l.backend.(TieredBackend) + if !ok { + // Backend built without the tiered surface — the pre-R-82 path, no probe needed. + return l.resolveUntargeted(ctx) + } + tiers, terr := tb.Tiers(ctx) + if errors.Is(terr, ErrTiersUnsupported) { + // A new controller meeting an OLD agent. Degrade — and SAY SO, once. + l.logTierDegradeOnce() + return l.resolveUntargeted(ctx) + } + if terr != nil { + return nil, false, terr + } + if len(tiers) == 0 { + // An agent that advertises no tiers cannot be backed up per-tier, but it can still be + // backed up untargeted. Fail toward DOING the backup, never toward skipping it. + l.logger.Printf("[WARN] [quiesce] agent advertised ZERO backup tiers — falling back to the untargeted path") + return l.resolveUntargeted(ctx) + } + for _, t := range tiers { + isDue, age, derr := tb.DueFor(ctx, t.Target) + if derr != nil { + // One tier's due-check failing must not silently drop the OTHER tier's backup. + l.logger.Printf("[ERROR] [quiesce] due-check failed for tier %q: %v (other tiers still evaluated)", t.Target, derr) + continue + } + if isDue { + due = append(due, dueTier{target: t.Target, ageSecs: age}) + } + } + return due, false, nil +} + +// resolveUntargeted is the pre-R-82 single-tier resolution. +func (l *Loop) resolveUntargeted(ctx context.Context) ([]dueTier, bool, error) { + isDue, age, err := l.backend.Due(ctx) + if err != nil { + return nil, true, err + } + if !isDue { + return nil, true, nil + } + return []dueTier{{target: "", ageSecs: age}}, true, nil +} + +// logTierDegradeOnce reports the pre-R-82 fallback exactly once per process. Once, because it is a +// steady state during a rollout and would otherwise log every poll; but never zero times, because a +// silent degrade is indistinguishable from multi-tier working. +func (l *Loop) logTierDegradeOnce() { + l.degradeOnce.Do(func() { + l.logger.Printf("[INFO] [quiesce] agent predates R-82 (no /backup/tiers) — using the single untargeted backup tier; per-tier scheduling is inactive until the agent is upgraded") + }) +} + +// startBackupOn starts a backup on one tier (untargeted when target is ""). +func (l *Loop) startBackupOn(ctx context.Context, target string) (string, error) { + if target == "" { + return l.backend.StartBackup(ctx) + } + return l.backend.(TieredBackend).StartBackupFor(ctx, target) +} + +// backupStatusOn reads one tier's job phase (untargeted when target is ""). +func (l *Loop) backupStatusOn(ctx context.Context, target string) (string, error) { + if target == "" { + return l.backend.BackupStatus(ctx) + } + return l.backend.(TieredBackend).BackupStatusFor(ctx, target) +} + +// tierLabel renders a tier for logs ("" → the untargeted tier). +func tierLabel(target string) string { + if target == "" { + return "(untargeted)" + } + return target +} diff --git a/controller/internal/quiesce/tiers_test.go b/controller/internal/quiesce/tiers_test.go new file mode 100644 index 0000000..ad07d65 --- /dev/null +++ b/controller/internal/quiesce/tiers_test.go @@ -0,0 +1,399 @@ +package quiesce + +import ( + "context" + "fmt" + "io" + "log" + "path/filepath" + "strings" + "sync" + "testing" + "time" +) + +// R-82 Slice B — one quiesce window, two tiers. +// +// Two properties are load-bearing and neither is provable by "no error was returned": +// 1. On the both-due night there is EXACTLY ONE stop/start pair. Two would mean two app outages +// for one night's work, undoing the whole argument for weekly-over-daily. +// 2. A new controller against an OLD agent still TAKES A BACKUP. The hollow version of that test +// asserts "no error" while silently skipping the backup — the exact failure it exists to catch. + +// ---- fakes ------------------------------------------------------------------------------- + +// NOTE: fakeStacks comes from quiesce_test.go (same package) — reused rather than duplicated. +// Counts come from len(stoppedNames()) / len(startedNames()). + +// tierBackend is a multi-tier fake agent. phases[target] is the phase sequence returned by +// successive BackupStatusFor calls for that tier. +type tierBackend struct { + mu sync.Mutex + tiers []BackupTier + tiersErr error + dueSet map[string]bool + phases map[string][]string + phaseIdx map[string]int + started []string // targets StartBackupFor/StartBackup was called with, in order + untargetedDue bool + startErrOn string + // stacks (optional) lets a start sample how many restarts have happened SO FAR — the direct + // way to assert "the app had not resumed when this tier started". + stacks *fakeStacks + startsAtStart []int +} + +func newTierBackend() *tierBackend { + return &tierBackend{ + dueSet: map[string]bool{}, phases: map[string][]string{}, phaseIdx: map[string]int{}, + } +} + +func (b *tierBackend) Tiers(context.Context) ([]BackupTier, error) { + if b.tiersErr != nil { + return nil, b.tiersErr + } + return b.tiers, nil +} +func (b *tierBackend) DueFor(_ context.Context, target string) (bool, *int64, error) { + b.mu.Lock() + defer b.mu.Unlock() + return b.dueSet[target], nil, nil +} +func (b *tierBackend) StartBackupFor(_ context.Context, target string) (string, error) { + b.mu.Lock() + defer b.mu.Unlock() + if b.startErrOn == target { + return "", fmt.Errorf("simulated start failure on %s", target) + } + b.started = append(b.started, target) + if b.stacks != nil { + b.startsAtStart = append(b.startsAtStart, len(b.stacks.startedNames())) + } + return "job-" + target, nil +} +func (b *tierBackend) BackupStatusFor(_ context.Context, target string) (string, error) { + b.mu.Lock() + defer b.mu.Unlock() + seq := b.phases[target] + i := b.phaseIdx[target] + if i < len(seq) { + b.phaseIdx[target]++ + return seq[i], nil + } + if len(seq) > 0 { + return seq[len(seq)-1], nil + } + return phaseDone, nil +} + +// The untargeted (pre-R-82) surface. +func (b *tierBackend) Due(context.Context) (bool, *int64, error) { + return b.untargetedDue, nil, nil +} +func (b *tierBackend) StartBackup(context.Context) (string, error) { + b.mu.Lock() + defer b.mu.Unlock() + b.started = append(b.started, "(untargeted)") + return "job-untargeted", nil +} +func (b *tierBackend) BackupStatus(context.Context) (string, error) { return phaseDone, nil } + +func (b *tierBackend) restartsWhenEachTierStarted() []int { + b.mu.Lock() + defer b.mu.Unlock() + return append([]int(nil), b.startsAtStart...) +} + +func (b *tierBackend) startedTargets() []string { + b.mu.Lock() + defer b.mu.Unlock() + return append([]string(nil), b.started...) +} + +func newTierLoop(t *testing.T, be Backend, st *fakeStacks, logTo io.Writer) *Loop { + t.Helper() + if logTo == nil { + logTo = io.Discard + } + return New(Options{ + Backend: be, + Stacks: st, + MarkerPath: filepath.Join(t.TempDir(), "quiesce-state.json"), + StatusPoll: time.Millisecond, + MaxQuiesce: 30 * time.Second, + Logger: log.New(logTo, "", 0), + }) +} + +// ---- RED-PROOF 3 — the both-due night ---------------------------------------------------- + +// EXACTLY ONE stop/start pair, with BOTH backups inside it. Asserting only "both backups ran" +// would pass against an implementation that quiesces twice — the count is the assertion. +// +// COMPANION RED-PROOF (observed): make runOnce call quiesceAndPollTiers once per due tier +// (a per-tier cycle instead of one window) → stops/starts become 2/2 and this fails with +// "want EXACTLY 1 stop and 1 start ... got stops=2 starts=2". Restored. +func TestBothTiersDue_ExactlyOneQuiesceWindow(t *testing.T) { + be := newTierBackend() + be.tiers = []BackupTier{{Target: "local", Primary: true}, {Target: "felhom-pbs"}} + be.dueSet["local"] = true + be.dueSet["felhom-pbs"] = true + be.phases["local"] = []string{phaseDone} + be.phases["felhom-pbs"] = []string{phaseSnapshotted, phaseDone} + + st := &fakeStacks{running: []string{"immich"}} + l := newTierLoop(t, be, st, nil) + + if err := l.runOnce(context.Background()); err != nil { + t.Fatalf("runOnce: %v", err) + } + stops, starts := len(st.stoppedNames()), len(st.startedNames()) + if stops != 1 || starts != 1 { + t.Fatalf("both-due night must be ONE quiesce window: want EXACTLY 1 stop and 1 start, got stops=%d starts=%d (stopped=%v started=%v)", + stops, starts, st.stoppedNames(), st.startedNames()) + } + got := be.startedTargets() + if len(got) != 2 || got[0] != "local" || got[1] != "felhom-pbs" { + t.Fatalf("both tiers must back up, primary first: got %v", got) + } +} + +// One tier due → one quiesce, that tier only. +func TestOnlyPBSDue_OneQuiesceThatTierOnly(t *testing.T) { + be := newTierBackend() + be.tiers = []BackupTier{{Target: "local", Primary: true}, {Target: "felhom-pbs"}} + be.dueSet["felhom-pbs"] = true + be.phases["felhom-pbs"] = []string{phaseDone} + + st := &fakeStacks{running: []string{"immich"}} + l := newTierLoop(t, be, st, nil) + if err := l.runOnce(context.Background()); err != nil { + t.Fatal(err) + } + stops, starts := len(st.stoppedNames()), len(st.startedNames()) + if stops != 1 || starts != 1 { + t.Fatalf("want 1/1, got stops=%d starts=%d", stops, starts) + } + if got := be.startedTargets(); len(got) != 1 || got[0] != "felhom-pbs" { + t.Fatalf("only the due tier may back up; got %v", got) + } +} + +// Neither due → no quiesce at all. The app must not be touched. +func TestNoTierDue_NoQuiesce(t *testing.T) { + be := newTierBackend() + be.tiers = []BackupTier{{Target: "local", Primary: true}, {Target: "felhom-pbs"}} + st := &fakeStacks{running: []string{"immich"}} + l := newTierLoop(t, be, st, nil) + if err := l.runOnce(context.Background()); err != nil { + t.Fatal(err) + } + if stops, starts := len(st.stoppedNames()), len(st.startedNames()); stops != 0 || starts != 0 { + t.Fatalf("no tier due must not touch the app; got stops=%d starts=%d", stops, starts) + } + if got := be.startedTargets(); len(got) != 0 { + t.Fatalf("no backup may start; got %v", got) + } +} + +// The app must stay DOWN until the LAST tier snapshots. Resuming after tier 1 would leave the DR +// tier capturing a running app — losing app-consistency on exactly the tier we most want it on. +func TestNonLastTierSnapshot_DoesNotResumeApp(t *testing.T) { + st := &fakeStacks{running: []string{"immich"}} + be := newTierBackend() + be.stacks = st + be.tiers = []BackupTier{{Target: "local", Primary: true}, {Target: "felhom-pbs"}} + be.dueSet["local"] = true + be.dueSet["felhom-pbs"] = true + // The local tier snapshots first, then finishes. The app must NOT come back at its snapshot. + be.phases["local"] = []string{phaseSnapshotted, phaseSnapshotted, phaseDone} + be.phases["felhom-pbs"] = []string{phaseSnapshotted, phaseDone} + + l := newTierLoop(t, be, st, nil) + if err := l.runOnce(context.Background()); err != nil { + t.Fatal(err) + } + // THE assertion: when the SECOND (last) tier started, zero restarts had happened — i.e. the app + // was still quiesced. Resuming at tier 1's snapshot would leave the DR tier capturing a RUNNING + // app, losing app-consistency on exactly the tier we most want it on. + at := be.restartsWhenEachTierStarted() + if len(at) != 2 { + t.Fatalf("both tiers must start; sampled %v", at) + } + if at[1] != 0 { + t.Fatalf("the app had ALREADY resumed (%d restarts) when the last tier started — non-last snapshot must not resume", at[1]) + } + if stops, starts := len(st.stoppedNames()), len(st.startedNames()); stops != 1 || starts != 1 { + t.Fatalf("want exactly one stop/start pair; got %d/%d", stops, starts) + } +} + +// ---- RED-PROOF 2 — new controller ↔ OLD agent -------------------------------------------- + +// The agent 404s /backup/tiers. The controller MUST degrade to the untargeted tier, LOG it, and +// STILL TAKE A BACKUP. +// +// The hollow version of this test asserts only "runOnce returned nil" — which passes against an +// implementation that silently skips the backup entirely. The assertion that matters is that a +// backup actually started. +// +// COMPANION RED-PROOF (observed): make resolveDueTiers return (nil, false, nil) on +// ErrTiersUnsupported — i.e. treat "no tier support" as "nothing due" → this test fails with +// "OLD AGENT: a backup MUST still be taken; got started=[]". Restored. +func TestOldAgent_DegradesToUntargetedAndStillBacksUp(t *testing.T) { + be := newTierBackend() + be.tiersErr = ErrTiersUnsupported + be.untargetedDue = true + + st := &fakeStacks{running: []string{"immich"}} + var logbuf strings.Builder + l := newTierLoop(t, be, st, &logbuf) + + if err := l.runOnce(context.Background()); err != nil { + t.Fatalf("runOnce against an old agent must not error: %v", err) + } + got := be.startedTargets() + if len(got) != 1 || got[0] != "(untargeted)" { + t.Fatalf("OLD AGENT: a backup MUST still be taken via the untargeted path; got started=%v", got) + } + if stops, starts := len(st.stoppedNames()), len(st.startedNames()); stops != 1 || starts != 1 { + t.Fatalf("old-agent path must still be one quiesce window; got %d/%d", stops, starts) + } + if !strings.Contains(logbuf.String(), "predates R-82") { + t.Fatalf("the degrade must be LOGGED — a silent degrade is indistinguishable from multi-tier working; log:\n%s", logbuf.String()) + } +} + +// The degrade line is logged ONCE, not every poll (it is a steady state during a rollout). +func TestOldAgent_DegradeLoggedOnce(t *testing.T) { + be := newTierBackend() + be.tiersErr = ErrTiersUnsupported + be.untargetedDue = false // not due → cheap repeated polls + + var logbuf strings.Builder + l := newTierLoop(t, be, &fakeStacks{}, &logbuf) + for i := 0; i < 5; i++ { + if err := l.runOnce(context.Background()); err != nil { + t.Fatal(err) + } + } + if n := strings.Count(logbuf.String(), "predates R-82"); n != 1 { + t.Fatalf("degrade must log exactly once across polls, got %d:\n%s", n, logbuf.String()) + } +} + +// A backend that does not implement TieredBackend at all (an older controller build path) uses the +// untargeted route with no probe and no degrade log. +func TestPlainBackend_UsesUntargetedPath(t *testing.T) { + be := &plainBackend{due: true} + st := &fakeStacks{running: []string{"immich"}} + l := newTierLoop(t, be, st, nil) + if err := l.runOnce(context.Background()); err != nil { + t.Fatal(err) + } + if be.started != 1 { + t.Fatalf("a plain Backend must still back up; started=%d", be.started) + } + if stops, starts := len(st.stoppedNames()), len(st.startedNames()); stops != 1 || starts != 1 { + t.Fatalf("want 1/1, got %d/%d", stops, starts) + } +} + +type plainBackend struct { + due bool + started int +} + +func (p *plainBackend) Due(context.Context) (bool, *int64, error) { return p.due, nil, nil } +func (p *plainBackend) StartBackup(context.Context) (string, error) { + p.started++ + return "job", nil +} +func (p *plainBackend) BackupStatus(context.Context) (string, error) { return phaseDone, nil } + +// ---- resilience -------------------------------------------------------------------------- + +// One tier failing to START must not prevent the other tier's backup, and the app must still resume +// exactly once. +func TestOneTierFailsToStart_OtherStillRunsAndAppResumes(t *testing.T) { + be := newTierBackend() + be.tiers = []BackupTier{{Target: "local", Primary: true}, {Target: "felhom-pbs"}} + be.dueSet["local"] = true + be.dueSet["felhom-pbs"] = true + be.startErrOn = "local" + be.phases["felhom-pbs"] = []string{phaseDone} + + st := &fakeStacks{running: []string{"immich"}} + l := newTierLoop(t, be, st, nil) + err := l.runOnce(context.Background()) + if err == nil { + t.Fatal("a tier start failure must be reported, not swallowed") + } + if got := be.startedTargets(); len(got) != 1 || got[0] != "felhom-pbs" { + t.Fatalf("the surviving tier must still back up; got %v", got) + } + if stops, starts := len(st.stoppedNames()), len(st.startedNames()); stops != 1 || starts != 1 { + t.Fatalf("app must resume exactly once even when a tier fails; got %d/%d", stops, starts) + } +} + +// A tier whose due-check errors must not drop the OTHER tier's backup. +func TestDueCheckErrorOnOneTier_OtherTierStillEvaluated(t *testing.T) { + be := &dueErrBackend{tierBackend: newTierBackend(), errOn: "local"} + be.tiers = []BackupTier{{Target: "local", Primary: true}, {Target: "felhom-pbs"}} + be.dueSet["felhom-pbs"] = true + be.phases["felhom-pbs"] = []string{phaseDone} + + st := &fakeStacks{running: []string{"immich"}} + l := newTierLoop(t, be, st, nil) + if err := l.runOnce(context.Background()); err != nil { + t.Fatalf("a single tier's due-check failure must not fail the cycle: %v", err) + } + if got := be.startedTargets(); len(got) != 1 || got[0] != "felhom-pbs" { + t.Fatalf("the healthy tier must still back up; got %v", got) + } +} + +type dueErrBackend struct { + *tierBackend + errOn string +} + +func (d *dueErrBackend) DueFor(ctx context.Context, target string) (bool, *int64, error) { + if target == d.errOn { + return false, nil, fmt.Errorf("simulated due-check failure") + } + return d.tierBackend.DueFor(ctx, target) +} + +// An agent advertising ZERO tiers must fall back to the untargeted path, not do nothing. +func TestZeroTiersAdvertised_FallsBackNotSilent(t *testing.T) { + be := newTierBackend() + be.tiers = nil // advertised, but empty + be.untargetedDue = true + + st := &fakeStacks{running: []string{"immich"}} + l := newTierLoop(t, be, st, nil) + if err := l.runOnce(context.Background()); err != nil { + t.Fatal(err) + } + if got := be.startedTargets(); len(got) != 1 || got[0] != "(untargeted)" { + t.Fatalf("zero advertised tiers must fall back, never skip; got %v", got) + } +} + +// oldestAge drives the window gate's safety valve: a NEVER-backed-up tier (nil age) must win over +// a fresher sibling, or a stale DR tier could be starved by a healthy local one. +func TestOldestAge(t *testing.T) { + i := func(v int64) *int64 { return &v } + if got := oldestAge([]dueTier{{ageSecs: i(10)}, {ageSecs: i(99)}}); got == nil || *got != 99 { + t.Fatalf("want 99, got %v", got) + } + if got := oldestAge([]dueTier{{ageSecs: i(10)}, {ageSecs: nil}}); got != nil { + t.Fatalf("a never-backed-up tier (nil) must win; got %v", *got) + } + if got := oldestAge(nil); got != nil { + t.Fatalf("empty → nil, got %v", *got) + } +}