v0.174.0 — R-82 Slice B: one quiesce window, two backup tiers
MinAgent UNCHANGED — degrades gracefully against ANY older agent.
The agent gained per-target tiers in v0.97.0. The controller owns quiescing,
so the multi-tier schedule is reconciled here: every due tier is collected up
front and run inside ONE quiesce window (one stop, N sequential backups, one
resume). Two cycles on the weekly night would mean two app outages for one
night's work.
Dedup rule: local-only -> one quiesce; PBS-only -> one quiesce; BOTH due ->
ONE window with both backups inside; neither -> no quiesce.
- quiesce.TieredBackend + BackupTier + ErrTiersUnsupported (optional extension)
- agentapi: BackupTiers/BackupDueFor/StartBackupFor/BackupStatusFor;
targetQuery("") yields an EMPTY suffix so untargeted hits the pre-R-82 route
byte-for-byte
- Loop.resolveDueTiers = the dedup rule in one place, agent order preserved
- quiesceAndPollTiers + pollTier: app stays quiesced until the LAST tier
snapshots (resuming earlier loses app-consistency on the DR tier). Consequence
stated in the docs: both-due-night downtime = first tier's full backup + last
tier's snapshot, which is why tiers run fast-first.
- Manual 'Mentes most' covers EVERY tier, due-ness ignored.
- Window-gate safety valve now uses the OLDEST due tier, so a stale DR tier
cannot be starved by a fresher local one.
Capability detection: /backup/tiers 404 = pre-R-82 agent (the documented
route-probe mechanism). Not a featureProbes row on purpose — the loop needs the
tier LIST, not a yes/no. Degrade logged exactly once per process.
Tests +11, full suite green. Red-proofs #2 and #3 observed and restored.
This commit is contained in:
@@ -1,5 +1,78 @@
|
||||
## Changelog
|
||||
|
||||
### v0.174.0 — R-82 Slice B: one quiesce window, two backup tiers (2026-07-26)
|
||||
|
||||
**MinAgent UNCHANGED — deliberately.** This release degrades gracefully against ANY older agent; it
|
||||
does not require v0.97.0. Against a pre-R-82 agent it uses the untargeted single-tier path exactly as
|
||||
before, logs the degrade once, and **still takes the backup**.
|
||||
|
||||
The agent gained per-target backup tiers in v0.97.0 ("local daily + PBS weekly"). The **controller**
|
||||
owns quiescing, so the multi-tier schedule has to be reconciled here: on the weekly night both tiers
|
||||
come due at once, and two quiesce cycles would mean **two app outages for one night's work** —
|
||||
undoing the entire argument for weekly-over-daily.
|
||||
|
||||
### The dedup rule (specified, not emergent)
|
||||
|
||||
| local due | PBS due | result |
|
||||
|---|---|---|
|
||||
| yes | no | one quiesce, local backup |
|
||||
| no | yes | one quiesce, PBS backup |
|
||||
| **yes** | **yes** | **ONE quiesce window, BOTH backups inside it — never two cycles** |
|
||||
| no | no | no quiesce |
|
||||
|
||||
### Added
|
||||
- **`quiesce.TieredBackend`** (optional extension to `Backend`) + `quiesce.BackupTier`,
|
||||
`ErrTiersUnsupported`. A backend that does not implement it — or whose `Tiers` returns
|
||||
`ErrTiersUnsupported` — drives the pre-R-82 single-tier path unchanged.
|
||||
- **`agentapi` per-tier client**: `BackupTiers`, `BackupDueFor`, `StartBackupFor`,
|
||||
`BackupStatusFor` (`internal/agentapi/backup_tiers.go`). `targetQuery("")` yields an EMPTY suffix,
|
||||
so an untargeted call hits the untargeted route byte-for-byte.
|
||||
- **`Loop.resolveDueTiers`** — the dedup rule in one place, returning due tiers in AGENT ORDER.
|
||||
- **`Loop.quiesceAndPollTiers` + `pollTier`** — one marker, one stop, N sequential backups, one
|
||||
resume, tail polled to completion.
|
||||
|
||||
### Capability detection
|
||||
`GET /backup/tiers` 404 ⇒ pre-R-82 agent. This is the project's documented ROUTE-PROBE mechanism
|
||||
(`internal/agentapi/features.go`: "a route that shipped together with the coupled semantics either
|
||||
answers (2xx ⇒ supported) or 404s"). It is **not** registered in the `featureProbes` table on
|
||||
purpose: that table answers a yes/no at a UI entry point, whereas the loop needs the tier LIST
|
||||
itself, so a table row would be a second probe of the same route for no gain. The degrade is logged
|
||||
**exactly once per process** — once because it is a steady state during a rollout, never zero times
|
||||
because a silent degrade is indistinguishable from multi-tier working.
|
||||
|
||||
### Two decisions worth stating plainly
|
||||
|
||||
**The app stays quiesced until the LAST tier snapshots.** Resuming after tier 1's snapshot would
|
||||
leave the following tier capturing a RUNNING app — losing app-consistency on exactly the DR tier we
|
||||
most want it on. **Consequence, user-visible:** on the both-due night downtime is
|
||||
*(first tier's full backup)* + *(last tier's snapshot)*, not one snapshot. Tiers must therefore run
|
||||
**fast-first**: vzdump holds a guest lock so they are necessarily sequential, and the agent
|
||||
advertises primary (local) first — local-then-PBS makes downtime ≈ local backup + PBS snapshot,
|
||||
whereas the reverse would be ≈ PBS backup + local snapshot, far worse.
|
||||
|
||||
**A manual "Mentés most" covers EVERY tier**, in one window, due-ness ignored. A manual run that
|
||||
silently skipped the DR tier would be the same applied-and-empty fault in a different costume.
|
||||
|
||||
### Resilience (unchanged guarantees, extended per tier)
|
||||
- Marker written BEFORE anything stops; unquiesce guaranteed by `defer` and fires **exactly once**
|
||||
no matter which tier fails; a crash between two backups leaves the marker and `Recover()` restarts
|
||||
the stacks at startup.
|
||||
- One tier failing to START does not prevent the other tier's backup, and the app still resumes once.
|
||||
- One tier's due-check erroring does not drop the other tier's backup.
|
||||
- An agent advertising ZERO tiers falls back to the untargeted path — never "nothing to do".
|
||||
- The window gate's safety valve now evaluates the OLDEST (most overdue) due tier, so a stale DR
|
||||
tier cannot be starved by a fresher local one (`oldestAge`; a never-backed-up tier wins outright).
|
||||
|
||||
### Tests
|
||||
+11 in `internal/quiesce/tiers_test.go`; full suite green. Red-proofs observed and restored:
|
||||
- **#3 both-due night** — a per-tier cycle instead of one window fails with
|
||||
`want EXACTLY 1 stop and 1 start, got stops=2 starts=2`. The COUNT is the assertion; asserting
|
||||
only "both backups ran" would pass against a double-quiesce implementation.
|
||||
- **#2 new controller ↔ old agent** — treating `ErrTiersUnsupported` as "nothing due" fails with
|
||||
`OLD AGENT: a backup MUST still be taken via the untargeted path; got started=[]`. The hollow
|
||||
version of this test asserts only "no error", which passes while silently skipping the backup.
|
||||
|
||||
|
||||
### v0.173.0 — R-77: endpoint-drift detection, samba protected-set gate, channel log honesty (2026-07-26)
|
||||
|
||||
Source: `felhom.eu/documentation/audits/DIAG-agent-channel-2026-07-26.md`.
|
||||
|
||||
@@ -72,6 +72,8 @@
|
||||
| `infra.SambaContainerName` / `SambaPassdbVolume` / `SambaPassdbMount` | controller/internal/infra/samba.go | consts | single source of truth for the samba container identity | the compose renderer interpolates them; stacks/backup/monitor read them. The CONTAINER name (`felhom-samba`) is NOT the stack name (`samba`) — `EffectiveProtected` needs the container one |
|
||||
| `sambaWriteAtomic` | controller/internal/stacks/samba.go | `(path, data, mode) error` | samba smb.conf/compose writes | tmp+**fsync**+rename (the only one of these that fsyncs). Fourth atomic-write helper in the tree — see §6 |
|
||||
| `Loop.writeMarker` / `Recover` | controller/internal/quiesce/quiesce.go | `(m Marker)` / `()` | Quiesce crash-safety | Marker written BEFORE stopping stacks; Recover restarts stranded stacks at boot |
|
||||
| `quiesce.TieredBackend` + `Loop.resolveDueTiers` / `quiesceAndPollTiers` | controller/internal/quiesce/tiers.go, quiesce.go | `Tiers/DueFor/StartBackupFor/BackupStatusFor`; `resolveDueTiers(ctx) ([]dueTier,bool,error)` | THE R-82 multi-tier backup schedule — several whole-guest tiers (local daily + PBS weekly) reconciled into ONE quiesce window | **Both tiers due ⇒ ONE stop/start pair**, never two (two = two app outages for one night). Tiers run SEQUENTIALLY (vzdump holds a guest lock) and the app stays down until the LAST tier snapshots — resuming earlier loses app-consistency on the DR tier. Order is fast-first (agent advertises primary first) or downtime blows up. `ErrTiersUnsupported` (route 404) ⇒ pre-R-82 agent ⇒ degrade to the untargeted path and **STILL BACK UP** — never read it as "nothing due". |
|
||||
| `agentapi.BackupTiers` / `BackupDueFor` / `StartBackupFor` / `BackupStatusFor` | controller/internal/agentapi/backup_tiers.go | `(ctx[, target]) (…, error)` | The per-tier agent surface (agent >= v0.97.0) | `targetQuery("")` returns an EMPTY suffix so an untargeted call hits the pre-R-82 route byte-for-byte. `BackupTiers` maps a 404 to `ErrTiersUnsupported` — the documented ROUTE-PROBE capability signal, NOT a `featureProbes` row (the loop needs the tier LIST, not a yes/no). |
|
||||
|
||||
### Compose ops / stack lifecycle
|
||||
|
||||
|
||||
@@ -701,6 +701,20 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri
|
||||
> the three daily legs via `scheduler.UpdateDaily` and takes effect **without a restart**. Precedence:
|
||||
> settings > controller.yaml `db_dump_schedule` > "02:30". See `internal/backupwindow`.
|
||||
|
||||
> **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest
|
||||
> backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0,
|
||||
> `GET /backup/tiers`). The controller owns quiescing, so it reconciles them: it collects EVERY due
|
||||
> tier up front and runs them inside **ONE quiesce window** — one stop, N sequential backups (vzdump
|
||||
> holds a guest lock), one resume. Two cycles on the weekly night would mean two app outages for one
|
||||
> night's work. The app stays quiesced until the **LAST** tier snapshots, so every tier is
|
||||
> app-consistent; the consequence is that both-due-night downtime is *(first tier's full backup)* +
|
||||
> *(last tier's snapshot)*, which is why tiers run fast-first (the agent advertises primary/local
|
||||
> first). A manual **"Mentés most"** covers every tier, due-ness ignored. The window gate's safety
|
||||
> valve evaluates the OLDEST due tier, so a stale DR tier cannot be starved by a fresher local one.
|
||||
> Against a **pre-R-82 agent** (`/backup/tiers` 404s) the loop degrades to the single untargeted
|
||||
> tier, logs it once, and still takes the backup — MinAgent is unchanged. See `internal/quiesce`
|
||||
> (`tiers.go`) and `internal/agentapi/backup_tiers.go`.
|
||||
|
||||
> **Atomic dump writes (v0.118.0, CAMPAIGN-3 F7).** BOTH dump paths are crash-safe: the DB dump
|
||||
> (`dbdump.go` DumpOne) and the Docker-volume dump (`DumpAppVolumes`) write to a `.tmp` sibling, fsync,
|
||||
> then `os.Rename` over the restore point ONLY on success. A mid-write failure (a NFS cut mid-tar, an
|
||||
|
||||
@@ -1704,6 +1704,41 @@ func (b quiesceBackend) BackupStatus(ctx context.Context) (string, error) {
|
||||
return r.Phase, err
|
||||
}
|
||||
|
||||
// ---- R-82: the tiered surface (quiesce.TieredBackend) ------------------------------------
|
||||
//
|
||||
// quiesceBackend satisfies quiesce.TieredBackend as well, so the loop schedules per tier when the
|
||||
// agent supports it. Against a PRE-R-82 agent, Tiers returns quiesce.ErrTiersUnsupported and the
|
||||
// loop degrades to the untargeted methods above — still taking a backup, never skipping one.
|
||||
|
||||
func (b quiesceBackend) Tiers(ctx context.Context) ([]quiesce.BackupTier, error) {
|
||||
r, err := b.c.BackupTiers(ctx)
|
||||
if errors.Is(err, agentapi.ErrTiersUnsupported) {
|
||||
// Translate the transport-layer probe into the loop's vocabulary; the loop keys on this.
|
||||
return nil, quiesce.ErrTiersUnsupported
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]quiesce.BackupTier, 0, len(r.Tiers))
|
||||
for _, t := range r.Tiers {
|
||||
out = append(out, quiesce.BackupTier{Target: t.Target, Primary: t.Primary})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (b quiesceBackend) DueFor(ctx context.Context, target string) (bool, *int64, error) {
|
||||
r, err := b.c.BackupDueFor(ctx, target)
|
||||
return r.Due, r.AgeSecs, err
|
||||
}
|
||||
func (b quiesceBackend) StartBackupFor(ctx context.Context, target string) (string, error) {
|
||||
r, err := b.c.StartBackupFor(ctx, target)
|
||||
return r.JobID, err
|
||||
}
|
||||
func (b quiesceBackend) BackupStatusFor(ctx context.Context, target string) (string, error) {
|
||||
r, err := b.c.BackupStatusFor(ctx, target)
|
||||
return r.Phase, err
|
||||
}
|
||||
|
||||
// startQuiesceLoop wires + starts the slice-8B quiesce loop when the local API is configured and
|
||||
// quiesce is enabled. It Recovers (restarts stacks left stopped by a mid-quiesce crash) before
|
||||
// starting the loop goroutine. Non-fatal: any misconfig disables the loop with a log line.
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
package agentapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
)
|
||||
|
||||
// R-82 Slice B — the per-tier backup surface (agent >= v0.97.0).
|
||||
//
|
||||
// Every method here is ADDITIVE. The untargeted BackupDue/StartBackup/BackupStatus keep their exact
|
||||
// pre-R-82 meaning and are still the single-tier path used against an older agent.
|
||||
|
||||
// ErrTiersUnsupported reports that this agent does not serve GET /backup/tiers — it predates R-82.
|
||||
// It is the DESIGNED capability probe (the route 404s), not a fault. The caller MUST degrade to the
|
||||
// untargeted single-tier path and still take a backup; concluding "nothing to do" from it would
|
||||
// silently stop backups during a fleet rollout.
|
||||
var ErrTiersUnsupported = errors.New("agentapi: agent does not serve /backup/tiers (pre-R-82)")
|
||||
|
||||
// BackupTierInfo is one advertised tier.
|
||||
type BackupTierInfo struct {
|
||||
Target string `json:"target"`
|
||||
CadenceSeconds int64 `json:"cadence_seconds"`
|
||||
Primary bool `json:"primary"`
|
||||
}
|
||||
|
||||
// TiersResponse mirrors the agent's GET /backup/tiers payload.
|
||||
type TiersResponse struct {
|
||||
VMID int `json:"vmid"`
|
||||
Tiers []BackupTierInfo `json:"tiers"`
|
||||
}
|
||||
|
||||
// BackupTiers lists the agent's backup tiers, primary first.
|
||||
// Returns ErrTiersUnsupported (wrapped) on a pre-R-82 agent — key on it with errors.Is.
|
||||
func (c *Client) BackupTiers(ctx context.Context) (TiersResponse, error) {
|
||||
var out TiersResponse
|
||||
body, err := c.get(ctx, "/backup/tiers")
|
||||
if err != nil {
|
||||
var se *StatusError
|
||||
if errors.As(err, &se) && se.Code == http.StatusNotFound {
|
||||
return out, ErrTiersUnsupported
|
||||
}
|
||||
return out, err
|
||||
}
|
||||
if err := json.Unmarshal(body, &out); err != nil {
|
||||
return out, fmt.Errorf("agentapi: decode /backup/tiers: %w", err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// targetQuery renders the ?target= suffix. An EMPTY target yields an empty string, so the caller
|
||||
// hits the untargeted route byte-for-byte — that is what keeps the pre-R-82 contract intact when
|
||||
// this client talks to an older agent.
|
||||
func targetQuery(target string) string {
|
||||
if target == "" {
|
||||
return ""
|
||||
}
|
||||
return "?target=" + url.QueryEscape(target)
|
||||
}
|
||||
|
||||
// BackupDueFor reports whether THIS TIER is due. A fresh backup on another tier must not satisfy it
|
||||
// — that filtering happens agent-side (latestSuccessfulBackupForTarget); this just asks per tier.
|
||||
func (c *Client) BackupDueFor(ctx context.Context, target string) (DueResponse, error) {
|
||||
var out DueResponse
|
||||
body, err := c.get(ctx, "/backup/due"+targetQuery(target))
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if err := json.Unmarshal(body, &out); err != nil {
|
||||
return out, fmt.Errorf("agentapi: decode /backup/due (target %q): %w", target, err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// StartBackupFor enqueues a backup of this guest ON THE GIVEN TIER.
|
||||
func (c *Client) StartBackupFor(ctx context.Context, target string) (BackupResponse, error) {
|
||||
var out BackupResponse
|
||||
body, err := c.post(ctx, "/backup"+targetQuery(target), struct{}{})
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if err := json.Unmarshal(body, &out); err != nil {
|
||||
return out, fmt.Errorf("agentapi: decode POST /backup (target %q): %w", target, err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// BackupStatusFor reports THIS TIER's current/last job phase. Jobs are keyed per tier agent-side,
|
||||
// so polling the wrong target would report a different tier's progress.
|
||||
func (c *Client) BackupStatusFor(ctx context.Context, target string) (StatusResponse, error) {
|
||||
var out StatusResponse
|
||||
body, err := c.get(ctx, "/backup/status"+targetQuery(target))
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if err := json.Unmarshal(body, &out); err != nil {
|
||||
return out, fmt.Errorf("agentapi: decode /backup/status (target %q): %w", target, err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -98,6 +98,8 @@ type Loop struct {
|
||||
// two can never stop the same stacks concurrently (the persisted marker covers crash-safety across
|
||||
// restarts; this covers concurrency within the process — which a manual trigger introduces).
|
||||
mu sync.Mutex
|
||||
// degradeOnce reports the pre-R-82 agent fallback exactly once per process (see tiers.go).
|
||||
degradeOnce sync.Once
|
||||
}
|
||||
|
||||
// New builds a Loop with sane defaults for any unset duration.
|
||||
@@ -177,26 +179,47 @@ func (l *Loop) runOnce(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
due, ageSecs, err := l.backend.Due(ctx)
|
||||
// R-82: resolve EVERY due tier up front. This is the dedup rule (tiers.go): both tiers due on
|
||||
// the weekly night yields ONE window with two backups, never two stop/start cycles.
|
||||
dueTiers, _, err := l.resolveDueTiers(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("check due: %w", err)
|
||||
}
|
||||
if !due {
|
||||
if len(dueTiers) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Window gate (Part 3) — SCHEDULED path only. TriggerNow calls quiesceAndPoll directly and is
|
||||
// never gated. Disabled when no window fn is wired (pre-v0.168.0 behavior).
|
||||
//
|
||||
// With several tiers due, the gate is evaluated against the OLDEST (most overdue) tier's age,
|
||||
// so the safety valve — "run regardless of the window once the last success is older than
|
||||
// cadence+24h" — cannot be suppressed by a fresher sibling tier.
|
||||
if l.windowStartFn != nil {
|
||||
window := l.windowStartFn()
|
||||
if !scheduledRunAllowed(l.now().In(budapestLocation()), window, ageSecs, l.cadence) {
|
||||
if !scheduledRunAllowed(l.now().In(budapestLocation()), window, oldestAge(dueTiers), l.cadence) {
|
||||
from, to := gateBounds(window)
|
||||
l.logger.Printf("[DEBUG] [quiesce] scheduled backup due but outside the backup window [%s–%s) — deferring to the next poll inside it", from, to)
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
return l.quiesceAndPoll(ctx)
|
||||
return l.quiesceAndPollTiers(ctx, dueTiers)
|
||||
}
|
||||
|
||||
// oldestAge returns the largest (most overdue) age among the due tiers; nil when any tier has never
|
||||
// backed up (nil age = "never", which is maximally overdue and must win).
|
||||
func oldestAge(tiers []dueTier) *int64 {
|
||||
var oldest *int64
|
||||
for _, t := range tiers {
|
||||
if t.ageSecs == nil {
|
||||
return nil // never backed up — the strongest claim on the safety valve
|
||||
}
|
||||
if oldest == nil || *t.ageSecs > *oldest {
|
||||
oldest = t.ageSecs
|
||||
}
|
||||
}
|
||||
return oldest
|
||||
}
|
||||
|
||||
// TriggerNow forces an app-consistent backup NOW (the manual "Mentés most" action), bypassing the
|
||||
@@ -220,7 +243,10 @@ func (l *Loop) TriggerNow() error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), l.maxQuiesce+5*time.Minute)
|
||||
defer cancel()
|
||||
l.logger.Printf("[INFO] [quiesce] manual backup requested — quiescing now")
|
||||
if err := l.quiesceAndPoll(ctx); err != nil {
|
||||
// Manual runs bypass due-ness (that is the point) but must still cover EVERY tier, in one
|
||||
// window. A manual "Mentés most" that silently skipped the DR tier would be the same
|
||||
// applied-and-empty fault in a different costume.
|
||||
if err := l.quiesceAndPollTiers(ctx, l.allTiersForManualRun(ctx)); err != nil {
|
||||
l.logger.Printf("[ERROR] [quiesce] manual backup cycle error: %v", err)
|
||||
}
|
||||
}()
|
||||
@@ -232,6 +258,49 @@ func (l *Loop) TriggerNow() error {
|
||||
// MUST hold l.mu. Unquiesce is guaranteed via the deferred closure (backup error, status-poll error,
|
||||
// the max-quiesce bound, or context cancellation all still restart the stacks and clear the marker).
|
||||
func (l *Loop) quiesceAndPoll(ctx context.Context) error {
|
||||
return l.quiesceAndPollTiers(ctx, []dueTier{{target: ""}})
|
||||
}
|
||||
|
||||
// allTiersForManualRun lists every tier a manual run should cover: all advertised tiers on an R-82
|
||||
// agent, or the single untargeted tier otherwise. Due-ness is deliberately NOT consulted.
|
||||
func (l *Loop) allTiersForManualRun(ctx context.Context) []dueTier {
|
||||
tb, ok := l.backend.(TieredBackend)
|
||||
if !ok {
|
||||
return []dueTier{{target: ""}}
|
||||
}
|
||||
tiers, err := tb.Tiers(ctx)
|
||||
if err != nil || len(tiers) == 0 {
|
||||
if errors.Is(err, ErrTiersUnsupported) {
|
||||
l.logTierDegradeOnce()
|
||||
} else if err != nil {
|
||||
l.logger.Printf("[WARN] [quiesce] manual run: tier list unavailable (%v) — using the untargeted tier", err)
|
||||
}
|
||||
return []dueTier{{target: ""}}
|
||||
}
|
||||
out := make([]dueTier, 0, len(tiers))
|
||||
for _, t := range tiers {
|
||||
out = append(out, dueTier{target: t.Target})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// quiesceAndPollTiers is the R-82 multi-tier cycle: ONE marker, ONE stop, N backups run
|
||||
// SEQUENTIALLY inside the window, ONE resume, then the tail polled to completion.
|
||||
//
|
||||
// Why sequential: vzdump takes a guest lock, so a second backup cannot start until the first
|
||||
// finishes. Why the app stays down until the LAST tier snapshots: the whole point of quiescing is
|
||||
// app-consistency, and resuming after tier 1's snapshot would leave tier 2 capturing a RUNNING app.
|
||||
// Consequence, stated plainly because it is user-visible: on the both-due night downtime is
|
||||
// (first tier's full backup) + (last tier's snapshot), not one snapshot. Tier ORDER therefore
|
||||
// matters — see resolveDueTiers.
|
||||
//
|
||||
// Crash-safety is unchanged and non-negotiable: the marker is written BEFORE anything stops,
|
||||
// unquiesce is guaranteed by defer, and it fires exactly once no matter which tier fails. A crash
|
||||
// between two backups leaves the marker on disk and Recover() restarts the stacks at startup.
|
||||
func (l *Loop) quiesceAndPollTiers(ctx context.Context, tiers []dueTier) error {
|
||||
if len(tiers) == 0 {
|
||||
return nil
|
||||
}
|
||||
running := l.stacks.RunningAppStacks()
|
||||
marker := Marker{Active: true, StartedAt: l.now(), StoppedStacks: running}
|
||||
if err := l.writeMarker(marker); err != nil {
|
||||
@@ -253,63 +322,103 @@ func (l *Loop) quiesceAndPoll(ctx context.Context) error {
|
||||
}
|
||||
defer unquiesce("deferred")
|
||||
|
||||
l.logger.Printf("[INFO] [quiesce] backup due — quiescing %d stack(s): %v", len(running), running)
|
||||
for _, s := range running {
|
||||
if err := l.stacks.StopStack(s); err != nil {
|
||||
l.logger.Printf("[ERROR] [quiesce] stop %s: %v (continuing)", s, err)
|
||||
l.logger.Printf("[INFO] [quiesce] backup due on %d tier(s) — quiescing %d stack(s): %v",
|
||||
len(tiers), len(running), running)
|
||||
for _, st := range running {
|
||||
if err := l.stacks.StopStack(st); err != nil {
|
||||
l.logger.Printf("[ERROR] [quiesce] stop %s: %v (continuing)", st, err)
|
||||
}
|
||||
}
|
||||
|
||||
jobID, err := l.backend.StartBackup(ctx)
|
||||
if err != nil {
|
||||
unquiesce("backup start failed")
|
||||
return fmt.Errorf("start backup: %w", err)
|
||||
}
|
||||
marker.JobID = jobID
|
||||
_ = l.writeMarker(marker) // best-effort: record the job id for diagnosis
|
||||
l.logger.Printf("[INFO] [quiesce] backup job %s started — polling to completion", jobID)
|
||||
|
||||
deadline := l.now().Add(l.maxQuiesce)
|
||||
var firstErr error
|
||||
|
||||
// ONE window, N tiers, sequential. The app resumes only after the LAST tier snapshots.
|
||||
for i, t := range tiers {
|
||||
label := tierLabel(t.target)
|
||||
last := i == len(tiers)-1
|
||||
|
||||
jobID, err := l.startBackupOn(ctx, t.target)
|
||||
if err != nil {
|
||||
l.logger.Printf("[ERROR] [quiesce] start backup on tier %s: %v", label, err)
|
||||
if firstErr == nil {
|
||||
firstErr = fmt.Errorf("start backup on %s: %w", label, err)
|
||||
}
|
||||
// A tier that will not start must not hold the app down for the others.
|
||||
if last {
|
||||
unquiesce("last tier failed to start")
|
||||
}
|
||||
continue
|
||||
}
|
||||
marker.JobID = jobID
|
||||
_ = l.writeMarker(marker) // best-effort: record the CURRENT tier's job id for diagnosis
|
||||
l.logger.Printf("[INFO] [quiesce] tier %s: backup job %s started — polling", label, jobID)
|
||||
|
||||
phase, perr := l.pollTier(ctx, t.target, jobID, label, deadline, last, &unquiesced, unquiesce)
|
||||
if perr != nil && firstErr == nil {
|
||||
firstErr = perr
|
||||
}
|
||||
if phase == phaseFailed {
|
||||
l.logger.Printf("[WARN] [quiesce] tier %s: backup job %s failed", label, jobID)
|
||||
}
|
||||
// The max-quiesce guard already unquiesced; keep going so the remaining tiers still run
|
||||
// (the app is up — the backups simply continue without the quiesce guarantee, which is
|
||||
// strictly better than skipping the DR tier entirely).
|
||||
}
|
||||
|
||||
// Belt: if every tier failed to start, nothing above unquiesced. The deferred call covers it,
|
||||
// but doing it here keeps the "resume as soon as possible" property explicit.
|
||||
unquiesce("cycle complete")
|
||||
return firstErr
|
||||
}
|
||||
|
||||
// pollTier polls ONE tier's job. It returns the terminal (or snapshotted-at-deadline) phase.
|
||||
//
|
||||
// The app is resumed ONLY when this is the LAST tier — that is what keeps every tier
|
||||
// app-consistent while still costing exactly one stop/start pair. For a non-last tier the loop
|
||||
// waits for a TERMINAL phase (done/failed), because vzdump holds the guest lock and the next tier
|
||||
// cannot start until this one truly finishes.
|
||||
func (l *Loop) pollTier(ctx context.Context, target, jobID, label string, deadline time.Time,
|
||||
last bool, unquiesced *bool, unquiesce func(string)) (string, error) {
|
||||
for {
|
||||
if !l.now().Before(deadline) {
|
||||
l.logger.Printf("[WARN] [quiesce] max-quiesce-duration (%s) exceeded for job %s — unquiescing while the backup continues on the agent",
|
||||
l.maxQuiesce, jobID)
|
||||
l.logger.Printf("[WARN] [quiesce] max-quiesce-duration (%s) exceeded on tier %s (job %s) — unquiescing while the backup continues on the agent",
|
||||
l.maxQuiesce, label, jobID)
|
||||
unquiesce("max-quiesce guard")
|
||||
return nil
|
||||
return "", nil
|
||||
}
|
||||
phase, err := l.backend.BackupStatus(ctx)
|
||||
phase, err := l.backupStatusOn(ctx, target)
|
||||
if err != nil {
|
||||
unquiesce("status poll failed")
|
||||
return fmt.Errorf("poll backup status: %w", err)
|
||||
return "", fmt.Errorf("poll backup status on %s: %w", label, err)
|
||||
}
|
||||
switch phase {
|
||||
case phaseSnapshotted:
|
||||
// 8B.2: the storage snapshot is taken — the app-stopped state is captured, so the app
|
||||
// may resume NOW (downtime = until-snapshot, not until-backup-done) with no loss of
|
||||
// app-consistency. unquiesce is idempotent (fires once); we then KEEP polling to
|
||||
// done/failed so a new backup isn't started until this one truly finishes (and so a
|
||||
// post-snapshot failure is observed). The marker is cleared on resume — a crash in this
|
||||
// tail leaves the app already up, nothing to recover.
|
||||
if !unquiesced {
|
||||
l.logger.Printf("[INFO] [quiesce] backup job %s snapshotted — resuming app early (8B.2)", jobID)
|
||||
unquiesce("snapshotted (early resume)")
|
||||
// 8B.2 early resume — but ONLY on the last tier. Resuming here on a non-last tier would
|
||||
// leave the following tier capturing a running app, losing app-consistency for exactly
|
||||
// the DR tier we most want it on.
|
||||
if last && !*unquiesced {
|
||||
l.logger.Printf("[INFO] [quiesce] tier %s: job %s snapshotted — resuming app early (8B.2)", label, jobID)
|
||||
unquiesce("snapshotted (early resume, last tier)")
|
||||
}
|
||||
case phaseDone:
|
||||
// Fallback (stop/downgraded mode never emits snapshotted): resume at done, exactly 8B.
|
||||
l.logger.Printf("[INFO] [quiesce] backup job %s done", jobID)
|
||||
unquiesce("backup done")
|
||||
return nil
|
||||
if last {
|
||||
l.logger.Printf("[INFO] [quiesce] tier %s: backup job %s done", label, jobID)
|
||||
unquiesce("backup done")
|
||||
} else {
|
||||
l.logger.Printf("[INFO] [quiesce] tier %s: backup job %s done — next tier may start (app still quiesced)", label, jobID)
|
||||
}
|
||||
return phaseDone, nil
|
||||
case phaseFailed:
|
||||
// If we already resumed at snapshotted, the app is up — just note the backup failed
|
||||
// (recorded for the agent's due window when it stores the failed result).
|
||||
l.logger.Printf("[WARN] [quiesce] backup job %s failed", jobID)
|
||||
unquiesce("backup failed")
|
||||
return nil
|
||||
if last {
|
||||
unquiesce("backup failed")
|
||||
}
|
||||
return phaseFailed, nil
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
unquiesce("controller shutting down")
|
||||
return ctx.Err()
|
||||
return "", ctx.Err()
|
||||
case <-time.After(l.statusPoll):
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
package quiesce
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
)
|
||||
|
||||
// R-82 Slice B — one quiesce window, two tiers.
|
||||
//
|
||||
// The agent gained per-target backup tiers in v0.97.0 ("local daily + PBS weekly"). The controller
|
||||
// owns quiescing, so the multi-tier schedule has to be reconciled HERE: on the weekly night both
|
||||
// tiers come due at once, and running two quiesce cycles would mean **two app outages for one
|
||||
// night's work** — which would undo the entire argument for weekly-over-daily.
|
||||
//
|
||||
// THE DEDUP RULE (specified, not emergent):
|
||||
//
|
||||
// local due | PBS due | result
|
||||
// ----------+---------+---------------------------------------------------------------
|
||||
// yes | no | one quiesce, local backup
|
||||
// no | yes | one quiesce, PBS backup
|
||||
// yes | yes | ONE quiesce window, BOTH backups inside it — never two cycles
|
||||
// no | no | no quiesce
|
||||
//
|
||||
// ErrTiersUnsupported is returned by TieredBackend.Tiers when the agent does not serve
|
||||
// GET /backup/tiers — i.e. it predates R-82 (the endpoint 404s). It is the DESIGNED capability
|
||||
// probe, not an error condition: the loop degrades to the single untargeted tier and logs it once.
|
||||
//
|
||||
// It must NEVER be treated as "nothing to do". A new controller meeting an old agent must still
|
||||
// back up; concluding "not due" from an unrecognised response would silently stop backups
|
||||
// fleet-wide during a rollout — the exact failure this project has hit before (controller v0.154.0,
|
||||
// agent v0.91.0, the hub allowedEventTypes 400 in R-77).
|
||||
var ErrTiersUnsupported = errors.New("quiesce: agent does not serve /backup/tiers (pre-R-82)")
|
||||
|
||||
// BackupTier is one tier as advertised by the agent, primary first.
|
||||
type BackupTier struct {
|
||||
Target string
|
||||
Primary bool
|
||||
}
|
||||
|
||||
// TieredBackend is the OPTIONAL R-82 extension to Backend. A backend that does not implement it
|
||||
// (or whose Tiers returns ErrTiersUnsupported) drives the pre-R-82 single-tier path unchanged.
|
||||
//
|
||||
// The untargeted Backend methods are NOT redundant: they remain the single-tier path, and the agent
|
||||
// guarantees they keep their exact pre-R-82 meaning and response bytes.
|
||||
type TieredBackend interface {
|
||||
Backend
|
||||
// Tiers lists the agent's backup tiers, primary first. ErrTiersUnsupported ⇒ pre-R-82 agent.
|
||||
Tiers(ctx context.Context) ([]BackupTier, error)
|
||||
DueFor(ctx context.Context, target string) (due bool, ageSecs *int64, err error)
|
||||
StartBackupFor(ctx context.Context, target string) (jobID string, err error)
|
||||
BackupStatusFor(ctx context.Context, target string) (phase string, err error)
|
||||
}
|
||||
|
||||
// dueTier is a tier this cycle must back up.
|
||||
type dueTier struct {
|
||||
target string // "" = the untargeted single-tier path (pre-R-82 agent)
|
||||
ageSecs *int64
|
||||
}
|
||||
|
||||
// resolveDueTiers answers "what must this cycle back up?" — the dedup rule above, in one place.
|
||||
//
|
||||
// Returns the due tiers IN AGENT ORDER (primary first). That order is deliberate and it is a
|
||||
// downtime decision, not cosmetics: tiers run SEQUENTIALLY because vzdump holds a guest lock, and
|
||||
// the app stays stopped until the LAST tier has snapshotted. Running the fast local tier first and
|
||||
// the slow WAN/PBS tier last makes downtime ≈ (local backup) + (PBS snapshot); the reverse order
|
||||
// would make it ≈ (PBS backup) + (local snapshot), which is far worse.
|
||||
//
|
||||
// degraded is true when the agent is pre-R-82 and the caller must use the untargeted path.
|
||||
func (l *Loop) resolveDueTiers(ctx context.Context) (due []dueTier, degraded bool, err error) {
|
||||
tb, ok := l.backend.(TieredBackend)
|
||||
if !ok {
|
||||
// Backend built without the tiered surface — the pre-R-82 path, no probe needed.
|
||||
return l.resolveUntargeted(ctx)
|
||||
}
|
||||
tiers, terr := tb.Tiers(ctx)
|
||||
if errors.Is(terr, ErrTiersUnsupported) {
|
||||
// A new controller meeting an OLD agent. Degrade — and SAY SO, once.
|
||||
l.logTierDegradeOnce()
|
||||
return l.resolveUntargeted(ctx)
|
||||
}
|
||||
if terr != nil {
|
||||
return nil, false, terr
|
||||
}
|
||||
if len(tiers) == 0 {
|
||||
// An agent that advertises no tiers cannot be backed up per-tier, but it can still be
|
||||
// backed up untargeted. Fail toward DOING the backup, never toward skipping it.
|
||||
l.logger.Printf("[WARN] [quiesce] agent advertised ZERO backup tiers — falling back to the untargeted path")
|
||||
return l.resolveUntargeted(ctx)
|
||||
}
|
||||
for _, t := range tiers {
|
||||
isDue, age, derr := tb.DueFor(ctx, t.Target)
|
||||
if derr != nil {
|
||||
// One tier's due-check failing must not silently drop the OTHER tier's backup.
|
||||
l.logger.Printf("[ERROR] [quiesce] due-check failed for tier %q: %v (other tiers still evaluated)", t.Target, derr)
|
||||
continue
|
||||
}
|
||||
if isDue {
|
||||
due = append(due, dueTier{target: t.Target, ageSecs: age})
|
||||
}
|
||||
}
|
||||
return due, false, nil
|
||||
}
|
||||
|
||||
// resolveUntargeted is the pre-R-82 single-tier resolution.
|
||||
func (l *Loop) resolveUntargeted(ctx context.Context) ([]dueTier, bool, error) {
|
||||
isDue, age, err := l.backend.Due(ctx)
|
||||
if err != nil {
|
||||
return nil, true, err
|
||||
}
|
||||
if !isDue {
|
||||
return nil, true, nil
|
||||
}
|
||||
return []dueTier{{target: "", ageSecs: age}}, true, nil
|
||||
}
|
||||
|
||||
// logTierDegradeOnce reports the pre-R-82 fallback exactly once per process. Once, because it is a
|
||||
// steady state during a rollout and would otherwise log every poll; but never zero times, because a
|
||||
// silent degrade is indistinguishable from multi-tier working.
|
||||
func (l *Loop) logTierDegradeOnce() {
|
||||
l.degradeOnce.Do(func() {
|
||||
l.logger.Printf("[INFO] [quiesce] agent predates R-82 (no /backup/tiers) — using the single untargeted backup tier; per-tier scheduling is inactive until the agent is upgraded")
|
||||
})
|
||||
}
|
||||
|
||||
// startBackupOn starts a backup on one tier (untargeted when target is "").
|
||||
func (l *Loop) startBackupOn(ctx context.Context, target string) (string, error) {
|
||||
if target == "" {
|
||||
return l.backend.StartBackup(ctx)
|
||||
}
|
||||
return l.backend.(TieredBackend).StartBackupFor(ctx, target)
|
||||
}
|
||||
|
||||
// backupStatusOn reads one tier's job phase (untargeted when target is "").
|
||||
func (l *Loop) backupStatusOn(ctx context.Context, target string) (string, error) {
|
||||
if target == "" {
|
||||
return l.backend.BackupStatus(ctx)
|
||||
}
|
||||
return l.backend.(TieredBackend).BackupStatusFor(ctx, target)
|
||||
}
|
||||
|
||||
// tierLabel renders a tier for logs ("" → the untargeted tier).
|
||||
func tierLabel(target string) string {
|
||||
if target == "" {
|
||||
return "(untargeted)"
|
||||
}
|
||||
return target
|
||||
}
|
||||
@@ -0,0 +1,399 @@
|
||||
package quiesce
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// R-82 Slice B — one quiesce window, two tiers.
|
||||
//
|
||||
// Two properties are load-bearing and neither is provable by "no error was returned":
|
||||
// 1. On the both-due night there is EXACTLY ONE stop/start pair. Two would mean two app outages
|
||||
// for one night's work, undoing the whole argument for weekly-over-daily.
|
||||
// 2. A new controller against an OLD agent still TAKES A BACKUP. The hollow version of that test
|
||||
// asserts "no error" while silently skipping the backup — the exact failure it exists to catch.
|
||||
|
||||
// ---- fakes -------------------------------------------------------------------------------
|
||||
|
||||
// NOTE: fakeStacks comes from quiesce_test.go (same package) — reused rather than duplicated.
|
||||
// Counts come from len(stoppedNames()) / len(startedNames()).
|
||||
|
||||
// tierBackend is a multi-tier fake agent. phases[target] is the phase sequence returned by
|
||||
// successive BackupStatusFor calls for that tier.
|
||||
type tierBackend struct {
|
||||
mu sync.Mutex
|
||||
tiers []BackupTier
|
||||
tiersErr error
|
||||
dueSet map[string]bool
|
||||
phases map[string][]string
|
||||
phaseIdx map[string]int
|
||||
started []string // targets StartBackupFor/StartBackup was called with, in order
|
||||
untargetedDue bool
|
||||
startErrOn string
|
||||
// stacks (optional) lets a start sample how many restarts have happened SO FAR — the direct
|
||||
// way to assert "the app had not resumed when this tier started".
|
||||
stacks *fakeStacks
|
||||
startsAtStart []int
|
||||
}
|
||||
|
||||
func newTierBackend() *tierBackend {
|
||||
return &tierBackend{
|
||||
dueSet: map[string]bool{}, phases: map[string][]string{}, phaseIdx: map[string]int{},
|
||||
}
|
||||
}
|
||||
|
||||
func (b *tierBackend) Tiers(context.Context) ([]BackupTier, error) {
|
||||
if b.tiersErr != nil {
|
||||
return nil, b.tiersErr
|
||||
}
|
||||
return b.tiers, nil
|
||||
}
|
||||
func (b *tierBackend) DueFor(_ context.Context, target string) (bool, *int64, error) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
return b.dueSet[target], nil, nil
|
||||
}
|
||||
func (b *tierBackend) StartBackupFor(_ context.Context, target string) (string, error) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
if b.startErrOn == target {
|
||||
return "", fmt.Errorf("simulated start failure on %s", target)
|
||||
}
|
||||
b.started = append(b.started, target)
|
||||
if b.stacks != nil {
|
||||
b.startsAtStart = append(b.startsAtStart, len(b.stacks.startedNames()))
|
||||
}
|
||||
return "job-" + target, nil
|
||||
}
|
||||
func (b *tierBackend) BackupStatusFor(_ context.Context, target string) (string, error) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
seq := b.phases[target]
|
||||
i := b.phaseIdx[target]
|
||||
if i < len(seq) {
|
||||
b.phaseIdx[target]++
|
||||
return seq[i], nil
|
||||
}
|
||||
if len(seq) > 0 {
|
||||
return seq[len(seq)-1], nil
|
||||
}
|
||||
return phaseDone, nil
|
||||
}
|
||||
|
||||
// The untargeted (pre-R-82) surface.
|
||||
func (b *tierBackend) Due(context.Context) (bool, *int64, error) {
|
||||
return b.untargetedDue, nil, nil
|
||||
}
|
||||
func (b *tierBackend) StartBackup(context.Context) (string, error) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
b.started = append(b.started, "(untargeted)")
|
||||
return "job-untargeted", nil
|
||||
}
|
||||
func (b *tierBackend) BackupStatus(context.Context) (string, error) { return phaseDone, nil }
|
||||
|
||||
func (b *tierBackend) restartsWhenEachTierStarted() []int {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
return append([]int(nil), b.startsAtStart...)
|
||||
}
|
||||
|
||||
func (b *tierBackend) startedTargets() []string {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
return append([]string(nil), b.started...)
|
||||
}
|
||||
|
||||
func newTierLoop(t *testing.T, be Backend, st *fakeStacks, logTo io.Writer) *Loop {
|
||||
t.Helper()
|
||||
if logTo == nil {
|
||||
logTo = io.Discard
|
||||
}
|
||||
return New(Options{
|
||||
Backend: be,
|
||||
Stacks: st,
|
||||
MarkerPath: filepath.Join(t.TempDir(), "quiesce-state.json"),
|
||||
StatusPoll: time.Millisecond,
|
||||
MaxQuiesce: 30 * time.Second,
|
||||
Logger: log.New(logTo, "", 0),
|
||||
})
|
||||
}
|
||||
|
||||
// ---- RED-PROOF 3 — the both-due night ----------------------------------------------------
|
||||
|
||||
// EXACTLY ONE stop/start pair, with BOTH backups inside it. Asserting only "both backups ran"
|
||||
// would pass against an implementation that quiesces twice — the count is the assertion.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): make runOnce call quiesceAndPollTiers once per due tier
|
||||
// (a per-tier cycle instead of one window) → stops/starts become 2/2 and this fails with
|
||||
// "want EXACTLY 1 stop and 1 start ... got stops=2 starts=2". Restored.
|
||||
func TestBothTiersDue_ExactlyOneQuiesceWindow(t *testing.T) {
|
||||
be := newTierBackend()
|
||||
be.tiers = []BackupTier{{Target: "local", Primary: true}, {Target: "felhom-pbs"}}
|
||||
be.dueSet["local"] = true
|
||||
be.dueSet["felhom-pbs"] = true
|
||||
be.phases["local"] = []string{phaseDone}
|
||||
be.phases["felhom-pbs"] = []string{phaseSnapshotted, phaseDone}
|
||||
|
||||
st := &fakeStacks{running: []string{"immich"}}
|
||||
l := newTierLoop(t, be, st, nil)
|
||||
|
||||
if err := l.runOnce(context.Background()); err != nil {
|
||||
t.Fatalf("runOnce: %v", err)
|
||||
}
|
||||
stops, starts := len(st.stoppedNames()), len(st.startedNames())
|
||||
if stops != 1 || starts != 1 {
|
||||
t.Fatalf("both-due night must be ONE quiesce window: want EXACTLY 1 stop and 1 start, got stops=%d starts=%d (stopped=%v started=%v)",
|
||||
stops, starts, st.stoppedNames(), st.startedNames())
|
||||
}
|
||||
got := be.startedTargets()
|
||||
if len(got) != 2 || got[0] != "local" || got[1] != "felhom-pbs" {
|
||||
t.Fatalf("both tiers must back up, primary first: got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// One tier due → one quiesce, that tier only.
|
||||
func TestOnlyPBSDue_OneQuiesceThatTierOnly(t *testing.T) {
|
||||
be := newTierBackend()
|
||||
be.tiers = []BackupTier{{Target: "local", Primary: true}, {Target: "felhom-pbs"}}
|
||||
be.dueSet["felhom-pbs"] = true
|
||||
be.phases["felhom-pbs"] = []string{phaseDone}
|
||||
|
||||
st := &fakeStacks{running: []string{"immich"}}
|
||||
l := newTierLoop(t, be, st, nil)
|
||||
if err := l.runOnce(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stops, starts := len(st.stoppedNames()), len(st.startedNames())
|
||||
if stops != 1 || starts != 1 {
|
||||
t.Fatalf("want 1/1, got stops=%d starts=%d", stops, starts)
|
||||
}
|
||||
if got := be.startedTargets(); len(got) != 1 || got[0] != "felhom-pbs" {
|
||||
t.Fatalf("only the due tier may back up; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Neither due → no quiesce at all. The app must not be touched.
|
||||
func TestNoTierDue_NoQuiesce(t *testing.T) {
|
||||
be := newTierBackend()
|
||||
be.tiers = []BackupTier{{Target: "local", Primary: true}, {Target: "felhom-pbs"}}
|
||||
st := &fakeStacks{running: []string{"immich"}}
|
||||
l := newTierLoop(t, be, st, nil)
|
||||
if err := l.runOnce(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if stops, starts := len(st.stoppedNames()), len(st.startedNames()); stops != 0 || starts != 0 {
|
||||
t.Fatalf("no tier due must not touch the app; got stops=%d starts=%d", stops, starts)
|
||||
}
|
||||
if got := be.startedTargets(); len(got) != 0 {
|
||||
t.Fatalf("no backup may start; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The app must stay DOWN until the LAST tier snapshots. Resuming after tier 1 would leave the DR
|
||||
// tier capturing a running app — losing app-consistency on exactly the tier we most want it on.
|
||||
func TestNonLastTierSnapshot_DoesNotResumeApp(t *testing.T) {
|
||||
st := &fakeStacks{running: []string{"immich"}}
|
||||
be := newTierBackend()
|
||||
be.stacks = st
|
||||
be.tiers = []BackupTier{{Target: "local", Primary: true}, {Target: "felhom-pbs"}}
|
||||
be.dueSet["local"] = true
|
||||
be.dueSet["felhom-pbs"] = true
|
||||
// The local tier snapshots first, then finishes. The app must NOT come back at its snapshot.
|
||||
be.phases["local"] = []string{phaseSnapshotted, phaseSnapshotted, phaseDone}
|
||||
be.phases["felhom-pbs"] = []string{phaseSnapshotted, phaseDone}
|
||||
|
||||
l := newTierLoop(t, be, st, nil)
|
||||
if err := l.runOnce(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// THE assertion: when the SECOND (last) tier started, zero restarts had happened — i.e. the app
|
||||
// was still quiesced. Resuming at tier 1's snapshot would leave the DR tier capturing a RUNNING
|
||||
// app, losing app-consistency on exactly the tier we most want it on.
|
||||
at := be.restartsWhenEachTierStarted()
|
||||
if len(at) != 2 {
|
||||
t.Fatalf("both tiers must start; sampled %v", at)
|
||||
}
|
||||
if at[1] != 0 {
|
||||
t.Fatalf("the app had ALREADY resumed (%d restarts) when the last tier started — non-last snapshot must not resume", at[1])
|
||||
}
|
||||
if stops, starts := len(st.stoppedNames()), len(st.startedNames()); stops != 1 || starts != 1 {
|
||||
t.Fatalf("want exactly one stop/start pair; got %d/%d", stops, starts)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- RED-PROOF 2 — new controller ↔ OLD agent --------------------------------------------
|
||||
|
||||
// The agent 404s /backup/tiers. The controller MUST degrade to the untargeted tier, LOG it, and
|
||||
// STILL TAKE A BACKUP.
|
||||
//
|
||||
// The hollow version of this test asserts only "runOnce returned nil" — which passes against an
|
||||
// implementation that silently skips the backup entirely. The assertion that matters is that a
|
||||
// backup actually started.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed): make resolveDueTiers return (nil, false, nil) on
|
||||
// ErrTiersUnsupported — i.e. treat "no tier support" as "nothing due" → this test fails with
|
||||
// "OLD AGENT: a backup MUST still be taken; got started=[]". Restored.
|
||||
func TestOldAgent_DegradesToUntargetedAndStillBacksUp(t *testing.T) {
|
||||
be := newTierBackend()
|
||||
be.tiersErr = ErrTiersUnsupported
|
||||
be.untargetedDue = true
|
||||
|
||||
st := &fakeStacks{running: []string{"immich"}}
|
||||
var logbuf strings.Builder
|
||||
l := newTierLoop(t, be, st, &logbuf)
|
||||
|
||||
if err := l.runOnce(context.Background()); err != nil {
|
||||
t.Fatalf("runOnce against an old agent must not error: %v", err)
|
||||
}
|
||||
got := be.startedTargets()
|
||||
if len(got) != 1 || got[0] != "(untargeted)" {
|
||||
t.Fatalf("OLD AGENT: a backup MUST still be taken via the untargeted path; got started=%v", got)
|
||||
}
|
||||
if stops, starts := len(st.stoppedNames()), len(st.startedNames()); stops != 1 || starts != 1 {
|
||||
t.Fatalf("old-agent path must still be one quiesce window; got %d/%d", stops, starts)
|
||||
}
|
||||
if !strings.Contains(logbuf.String(), "predates R-82") {
|
||||
t.Fatalf("the degrade must be LOGGED — a silent degrade is indistinguishable from multi-tier working; log:\n%s", logbuf.String())
|
||||
}
|
||||
}
|
||||
|
||||
// The degrade line is logged ONCE, not every poll (it is a steady state during a rollout).
|
||||
func TestOldAgent_DegradeLoggedOnce(t *testing.T) {
|
||||
be := newTierBackend()
|
||||
be.tiersErr = ErrTiersUnsupported
|
||||
be.untargetedDue = false // not due → cheap repeated polls
|
||||
|
||||
var logbuf strings.Builder
|
||||
l := newTierLoop(t, be, &fakeStacks{}, &logbuf)
|
||||
for i := 0; i < 5; i++ {
|
||||
if err := l.runOnce(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if n := strings.Count(logbuf.String(), "predates R-82"); n != 1 {
|
||||
t.Fatalf("degrade must log exactly once across polls, got %d:\n%s", n, logbuf.String())
|
||||
}
|
||||
}
|
||||
|
||||
// A backend that does not implement TieredBackend at all (an older controller build path) uses the
|
||||
// untargeted route with no probe and no degrade log.
|
||||
func TestPlainBackend_UsesUntargetedPath(t *testing.T) {
|
||||
be := &plainBackend{due: true}
|
||||
st := &fakeStacks{running: []string{"immich"}}
|
||||
l := newTierLoop(t, be, st, nil)
|
||||
if err := l.runOnce(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if be.started != 1 {
|
||||
t.Fatalf("a plain Backend must still back up; started=%d", be.started)
|
||||
}
|
||||
if stops, starts := len(st.stoppedNames()), len(st.startedNames()); stops != 1 || starts != 1 {
|
||||
t.Fatalf("want 1/1, got %d/%d", stops, starts)
|
||||
}
|
||||
}
|
||||
|
||||
type plainBackend struct {
|
||||
due bool
|
||||
started int
|
||||
}
|
||||
|
||||
func (p *plainBackend) Due(context.Context) (bool, *int64, error) { return p.due, nil, nil }
|
||||
func (p *plainBackend) StartBackup(context.Context) (string, error) {
|
||||
p.started++
|
||||
return "job", nil
|
||||
}
|
||||
func (p *plainBackend) BackupStatus(context.Context) (string, error) { return phaseDone, nil }
|
||||
|
||||
// ---- resilience --------------------------------------------------------------------------
|
||||
|
||||
// One tier failing to START must not prevent the other tier's backup, and the app must still resume
|
||||
// exactly once.
|
||||
func TestOneTierFailsToStart_OtherStillRunsAndAppResumes(t *testing.T) {
|
||||
be := newTierBackend()
|
||||
be.tiers = []BackupTier{{Target: "local", Primary: true}, {Target: "felhom-pbs"}}
|
||||
be.dueSet["local"] = true
|
||||
be.dueSet["felhom-pbs"] = true
|
||||
be.startErrOn = "local"
|
||||
be.phases["felhom-pbs"] = []string{phaseDone}
|
||||
|
||||
st := &fakeStacks{running: []string{"immich"}}
|
||||
l := newTierLoop(t, be, st, nil)
|
||||
err := l.runOnce(context.Background())
|
||||
if err == nil {
|
||||
t.Fatal("a tier start failure must be reported, not swallowed")
|
||||
}
|
||||
if got := be.startedTargets(); len(got) != 1 || got[0] != "felhom-pbs" {
|
||||
t.Fatalf("the surviving tier must still back up; got %v", got)
|
||||
}
|
||||
if stops, starts := len(st.stoppedNames()), len(st.startedNames()); stops != 1 || starts != 1 {
|
||||
t.Fatalf("app must resume exactly once even when a tier fails; got %d/%d", stops, starts)
|
||||
}
|
||||
}
|
||||
|
||||
// A tier whose due-check errors must not drop the OTHER tier's backup.
|
||||
func TestDueCheckErrorOnOneTier_OtherTierStillEvaluated(t *testing.T) {
|
||||
be := &dueErrBackend{tierBackend: newTierBackend(), errOn: "local"}
|
||||
be.tiers = []BackupTier{{Target: "local", Primary: true}, {Target: "felhom-pbs"}}
|
||||
be.dueSet["felhom-pbs"] = true
|
||||
be.phases["felhom-pbs"] = []string{phaseDone}
|
||||
|
||||
st := &fakeStacks{running: []string{"immich"}}
|
||||
l := newTierLoop(t, be, st, nil)
|
||||
if err := l.runOnce(context.Background()); err != nil {
|
||||
t.Fatalf("a single tier's due-check failure must not fail the cycle: %v", err)
|
||||
}
|
||||
if got := be.startedTargets(); len(got) != 1 || got[0] != "felhom-pbs" {
|
||||
t.Fatalf("the healthy tier must still back up; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
type dueErrBackend struct {
|
||||
*tierBackend
|
||||
errOn string
|
||||
}
|
||||
|
||||
func (d *dueErrBackend) DueFor(ctx context.Context, target string) (bool, *int64, error) {
|
||||
if target == d.errOn {
|
||||
return false, nil, fmt.Errorf("simulated due-check failure")
|
||||
}
|
||||
return d.tierBackend.DueFor(ctx, target)
|
||||
}
|
||||
|
||||
// An agent advertising ZERO tiers must fall back to the untargeted path, not do nothing.
|
||||
func TestZeroTiersAdvertised_FallsBackNotSilent(t *testing.T) {
|
||||
be := newTierBackend()
|
||||
be.tiers = nil // advertised, but empty
|
||||
be.untargetedDue = true
|
||||
|
||||
st := &fakeStacks{running: []string{"immich"}}
|
||||
l := newTierLoop(t, be, st, nil)
|
||||
if err := l.runOnce(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := be.startedTargets(); len(got) != 1 || got[0] != "(untargeted)" {
|
||||
t.Fatalf("zero advertised tiers must fall back, never skip; got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// oldestAge drives the window gate's safety valve: a NEVER-backed-up tier (nil age) must win over
|
||||
// a fresher sibling, or a stale DR tier could be starved by a healthy local one.
|
||||
func TestOldestAge(t *testing.T) {
|
||||
i := func(v int64) *int64 { return &v }
|
||||
if got := oldestAge([]dueTier{{ageSecs: i(10)}, {ageSecs: i(99)}}); got == nil || *got != 99 {
|
||||
t.Fatalf("want 99, got %v", got)
|
||||
}
|
||||
if got := oldestAge([]dueTier{{ageSecs: i(10)}, {ageSecs: nil}}); got != nil {
|
||||
t.Fatalf("a never-backed-up tier (nil) must win; got %v", *got)
|
||||
}
|
||||
if got := oldestAge(nil); got != nil {
|
||||
t.Fatalf("empty → nil, got %v", *got)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user