A cadence cap on the local-API whole-guest backup route (R-973, unreleased)
gates / gates (push) Successful in 49s
gates / gates (push) Successful in 49s
Each successful run prunes its tier, so a compromised controller calling POST /backup in a loop pushed out every pre-compromise restore point. A tier with a cadence of at most 24 h now allows 3 successful runs per rolling 24 h (night, a press, the next night); a longer tier 1 per max(cadence - 24 h, cadence/2), 6 days for the weekly off-site tier. Only successes count. A refusal is a 429, a WARN, saved, and for 24 h a critical host-report status that the hub turns into its existing operator-only agent_capability_degraded event. The success history and the refusal are kept in the R-894 state file, so a restart does not reset the cap. No sudoers change. Tests TestBackupCap_* and the state, status and wiring tests, red-proved.
This commit is contained in:
@@ -36,6 +36,25 @@ Committed, not released — the release and its vouch are the operator's.
|
||||
`deferUpload: false` → „json mode must defer the upload"; the `Upload:` line removed → „has no Upload". **Residual:**
|
||||
the customer's typed-back check after the claim is browser-only, so „confirmed" here means „claimed" — a customer
|
||||
who opens the code and closes the page without writing it down has replaced the package.
|
||||
- **R-973 — the local-API whole-guest backup route has a cadence cap.** Every successful run prunes its tier, so a
|
||||
compromised controller calling `POST /backup` in a loop pushed out every pre-compromise restore point (local
|
||||
keep-last 3; off-site via ep0's keep-last 2). Now (`internal/localapi/backup_cap.go`): a tier whose cadence is ≤ 24 h
|
||||
allows **3 successful runs per rolling 24 h** (the `07` §6 day: the night run, a „Mentés most" press, and the next
|
||||
night, which since R-899 still runs and can land inside the same 24 h — so a press never costs the night); a longer
|
||||
tier allows **1 per max(cadence − 24 h, cadence/2)** — 6 days for the 7-day off-site tier, which the controller only
|
||||
runs when due and the button never targets. Only successes count (a failed run prunes nothing). A refusal answers
|
||||
**429** with the next allowed time, is logged at WARN, is saved, and for 24 h raises the CRITICAL host-report status
|
||||
`localapi:backup-cadence-cap` — the hub's existing operator-only `agent_capability_degraded` event, **no hub change**.
|
||||
**The cap survives an agent restart:** `BackupSuccessState` (the R-894 file) now also keeps the newest 8 success
|
||||
times and the last refusal per tier and guest (additive JSON fields; a pre-R-973 file counts its one known
|
||||
success). Tests `TestBackupCap_*` (an off-site flood keeps a pre-compromise copy for 5+ days; a local flood runs 3
|
||||
a day; the scheduled day is never refused; restart; failures; the operator signal) + `TestBackupCadenceCap`,
|
||||
`TestBackupSuccessState_R973_*`, `TestBackupCapStatus_*`, `TestR973_CapRefusalIsWiredIntoTheReport`. Red-proved:
|
||||
the cap disabled → „the off-site tier lost every pre-compromise copy: 10 runs"; the disk history ignored → „after a
|
||||
restart the 4th run in a day was allowed"; the wiring line removed → „never binds backupCapRefusal". **Limits, stated:**
|
||||
with keep-last 3 the local tier's three points can still be replaced within one day (3 per day is the schedule's own
|
||||
pace plus one press); the hub's event text reads „agent privileged capability degraded" (the type is generic). No
|
||||
sudoers or bundle change.
|
||||
|
||||
## v0.159.2 — the orphan thin check runs only after a failed backup (R-1012 follow-up) (2026-10-18)
|
||||
|
||||
|
||||
+12
@@ -1,5 +1,17 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-10-18 (night) — UNRELEASED on main: R-1093, R-978, R-973** (release + vouch: the operator's). R-1093 an empty
|
||||
> partition is blank (lsblk's PTTYPE on a partition is the parent's; set aside only on a clean blkid exit 0 with a
|
||||
> matching `PART_ENTRY_SCHEME`; an unparsable lsblk is unprobed). R-978 the escrow package reaches the hub only at the
|
||||
> claim (ceremony output v2, `PendingUpload`; `EscrowCeremonyConfig.Upload`). R-973 the `POST /backup` cadence cap.
|
||||
> **Decision (decided by CC unattended — operator may reverse): the cap's numbers.** Q: how many whole-guest runs may
|
||||
> the guest ask for? Options: (a) 2 per 24 h on the daily tier — keeps one local point older than a day, but refuses
|
||||
> the NIGHT after a press (R-899's night can fall inside the same 24 h); (b) 3 per 24 h + 1 per 6 days on the weekly
|
||||
> tier — the schedule never refused, a second press in a day refused, the local tier still replaceable in a day;
|
||||
> (c) none — status quo. Chose (b): the off-site tier is the one `07` §8 row 10 relies on and it is fully protected
|
||||
> for ~6 days; the local half's real fix is a time-bucketed prune (keep-daily), which is a retention change for the
|
||||
> operator. No sudoers or bundle change in any of the three.
|
||||
|
||||
> **2026-10-17 — v0.159.1 RELEASED (0.159.0 superseded, do not vouch).** Go 1.27.2; R-1124 a decommissioned drive that is
|
||||
> gone is forgotten (`ForgetRetiredDrives`, root verb `unit-forget`); R-1012 the read-only `felhom-thin-check` names an
|
||||
> orphan thin device in a failed backup's text; the signed decommission prunes the guest-bind record. Bundle adds a path →
|
||||
|
||||
@@ -174,6 +174,7 @@
|
||||
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
|
||||
| `backup.RestoreTestState` | internal/backup/restoretest_state.go | `RecordSuccess(target,archive,tier,verified,t)` / `ProvenArchive(target)` / `ProvenRestoreTests(ctx)` / `LastSuccess(target)` / `OldestFirst(targets)` | Per-tier restore-test PROOF state, persisted (atomic tmp+rename) — **which archive** was proven, and when (R-86) | **Credit ONLY on success** — a permanently failing tier must keep sorting first, or it looks freshly proven and stops being retried. Ties break on target id: without it, two tiers proven in the same second rotate by Go's randomised map order. **This one NEEDS persistence unlike R-84** — R-84 had ground truth to consult (the archive is still on the storage); a restore-test destroys its scratch and leaves no artifact. **R-86: the ARCHIVE is the state, the time is metadata** — a time alone cannot answer "have we proven THIS archive", which is the due-check's whole question. A pre-R-86 file (bare RFC3339 per target) keeps its time and yields NO proven archive, so each tier is due once after the upgrade; reading a legacy time as proof of the current archive would invent a guarantee. **R-189: it is also the REPORTABLE half of the restore-test signal.** The in-memory `backup.Store` holds only this process's latest run, and under per-archive due-ness the agent will not re-test a proven archive — so a proof lost to a restart is not repeated for a whole archive generation (observed live: a passing 14.5 GB offsite restore reached no host-report). `ProvenRestoreTests` renders the stored proofs as `hub.RestoreTest` entries and the collector merges them; a record missing the archive or the tier is NOT emitted, because an unproven tier reading as proven is worse than the defect. **Only successes are stored, deliberately:** a success suppresses future work, a failure causes it. |
|
||||
| `backup.BackupSuccessState` | internal/backup/backup_state.go | `RecordBackupSuccess(target, b)` / `LastKnownSuccess(target, vmid)` | Newest SUCCESSFUL backup per tier+guest, persisted (atomic tmp+rename) — the due-check's fallback when the tier's storage cannot be read after a restart (R-894) | **Read ONLY when the storage cannot be read** — a storage that answers is the ground truth (R-84), and an archive absent there must make the tier due even when this file remembers one. A saved copy older than the cadence still reads due. Only successes are written. |
|
||||
| localapi backup cadence cap (R-973) | internal/localapi/backup_cap.go | `backupCadenceCap(cadence)`, `Server.capRefuses`, `Server.BackupCapRefusal()` | POST /backup refuses (429) a run the `07` §6 schedule would never ask for: ≤24 h tier → 3 successes per rolling 24 h; longer tier → 1 per max(cadence−24 h, cadence/2) | Counts SUCCESSES only, from this process AND `BackupSuccessState.RecentBackupSuccesses` (so a restart does not reset it); refusals saved (`RecordBackupRefusal`) and raised for 24 h as the CRITICAL host-report status `localapi:backup-cadence-cap` (`backupCapStatus` in main) → the hub's existing `agent_capability_degraded`. Tests `TestBackupCap_*`. |
|
||||
| `hub.ProvenRestoreTestReporter` + `Collector.SetProvenRestoreTests` | internal/hub/collect.go | the DURABLE restore-test source, merged with the in-memory one | R-189. Merge rule: **one entry per tier, newest by `TestedAt` wins** — a fresh failure beats a stored success (the failure is the news, and it lives nowhere else), a stored success beats a stale in-memory entry after a restart, and a tier never appears twice (the hub would read two tests). An unparseable timestamp counts as OLDER, so a malformed entry cannot displace a good one. **The wiring is pinned by an AST test** — the method this replaced (`RestoreTestState.Snapshot`) carried a doc comment naming a host-report gauge and had no caller for weeks. |
|
||||
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
|
||||
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"go/ast"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
|
||||
)
|
||||
|
||||
// R-973: a cadence-cap refusal becomes a CRITICAL degraded status — what the hub's existing
|
||||
// agent_capability_degraded event (operator-only) keys on — and clears when the refusal ages out.
|
||||
func TestBackupCapStatus_RefusalIsCriticalDegraded(t *testing.T) {
|
||||
at := time.Date(2026, 10, 18, 3, 0, 0, 0, time.UTC)
|
||||
s := backupCapStatus(func() (string, int, time.Time, bool) { return "felhom-pbs", 9201, at, true })
|
||||
if !s.Critical || s.Status != capability.StatusDegraded || !strings.Contains(s.Reason, "9201") || !strings.Contains(s.Reason, "felhom-pbs") {
|
||||
t.Fatalf("a refusal must raise a critical degraded status naming the guest and the tier: %+v", s)
|
||||
}
|
||||
if s := backupCapStatus(func() (string, int, time.Time, bool) { return "", 0, time.Time{}, false }); s.Status != capability.StatusOK || !s.Critical {
|
||||
t.Fatalf("no refusal → ok (and still critical, so a later refusal alerts): %+v", s)
|
||||
}
|
||||
if s := backupCapStatus(nil); s.Status != capability.StatusOK {
|
||||
t.Fatalf("no local API → ok: %+v", s)
|
||||
}
|
||||
}
|
||||
|
||||
// Wiring (the built-but-never-wired class): runDaemon's capability probe appends backupCapStatus, and the
|
||||
// late-bound source is set from the local API server.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-10-18): delete `backupCapRefusal = localSrv.BackupCapRefusal` →
|
||||
// "runDaemon never binds backupCapRefusal to the local API server". Restored.
|
||||
func TestR973_CapRefusalIsWiredIntoTheReport(t *testing.T) {
|
||||
_, f := parseMain(t)
|
||||
var appended, bound bool
|
||||
for _, d := range f.Decls {
|
||||
fd, ok := d.(*ast.FuncDecl)
|
||||
if !ok || fd.Name == nil || fd.Name.Name != "runDaemon" || fd.Body == nil {
|
||||
continue
|
||||
}
|
||||
ast.Inspect(fd.Body, func(n ast.Node) bool {
|
||||
switch x := n.(type) {
|
||||
case *ast.CallExpr:
|
||||
if id, ok := x.Fun.(*ast.Ident); ok && id.Name == "backupCapStatus" && len(x.Args) == 1 {
|
||||
if a, ok := x.Args[0].(*ast.Ident); ok && a.Name == "backupCapRefusal" {
|
||||
appended = true
|
||||
}
|
||||
}
|
||||
case *ast.AssignStmt:
|
||||
if len(x.Lhs) == 1 && len(x.Rhs) == 1 {
|
||||
if l, ok := x.Lhs[0].(*ast.Ident); ok && l.Name == "backupCapRefusal" {
|
||||
if sel, ok := x.Rhs[0].(*ast.SelectorExpr); ok && sel.Sel.Name == "BackupCapRefusal" {
|
||||
bound = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
if !appended {
|
||||
t.Fatal("runDaemon's capability probe does not call backupCapStatus(backupCapRefusal)")
|
||||
}
|
||||
if !bound {
|
||||
t.Fatal("runDaemon never binds backupCapRefusal to the local API server")
|
||||
}
|
||||
}
|
||||
@@ -404,6 +404,31 @@ func (r *gateRemounter) Remount(ctx context.Context, t storage.KnownTarget) {
|
||||
// poolReadStatus probes the felhom-pool membership read (GET /pools/felhom) the stale-lock reaper's
|
||||
// ownership scoping depends on (v0.62.0, audit A1). Needs `Pool.Audit` at `/pool/felhom` — granted
|
||||
// by host-install v1.9.0; on an older ACL this reports degraded and the reaper fail-safes (skips).
|
||||
// backupCapStatusName is the host-report status the R-973 cadence cap raises.
|
||||
const backupCapStatusName = "localapi:backup-cadence-cap"
|
||||
|
||||
// backupCapStatus turns the local-API backup cadence cap's newest refusal (R-973) into a CRITICAL
|
||||
// capability status: degraded for 24 h after a refusal, so the hub raises its operator-only
|
||||
// agent_capability_degraded event (no hub change). The Reason names the guest, the tier and the time, and
|
||||
// says what to do. nil source (local API off) → ok.
|
||||
func backupCapStatus(refusal func() (string, int, time.Time, bool)) capability.Status {
|
||||
s := capability.Status{
|
||||
Name: backupCapStatusName,
|
||||
Feature: "whole-guest restore points: a guest asked for more whole-guest backups than its schedule allows, and the extra ones were refused (R-973)",
|
||||
Critical: true,
|
||||
Status: capability.StatusOK,
|
||||
}
|
||||
if refusal == nil {
|
||||
return s
|
||||
}
|
||||
if target, vmid, at, ok := refusal(); ok {
|
||||
s.Status = capability.StatusDegraded
|
||||
s.Reason = fmt.Sprintf("refused a whole-guest backup of guest %d on target %q at %s — more than the schedule allows; the guest's controller may be misbehaving or compromised: check the guest",
|
||||
vmid, target, at.UTC().Format(time.RFC3339))
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func poolReadStatus(ctx context.Context, px *proxmox.Client) capability.Status {
|
||||
s := capability.Status{
|
||||
Name: "pve:pool-read",
|
||||
@@ -830,8 +855,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
run: (&proxmox.ExecRunner{Mode: proxmox.RunnerMode(cfg.Privileged.Mode)}).Run,
|
||||
log: logger,
|
||||
}
|
||||
// R-973: the local-API backup cadence cap's refusals ride the report as a CRITICAL status, so the hub's
|
||||
// existing agent_capability_degraded event (operator-only) tells the operator. Late-bound like
|
||||
// drConfigured: the local API server is built further down, before the loop starts.
|
||||
var backupCapRefusal func() (string, int, time.Time, bool)
|
||||
probeAll := func(ctx context.Context) []capability.Status {
|
||||
out := append(capProber.Probe(ctx), poolReadStatus(ctx, px))
|
||||
out = append(out, backupCapStatus(backupCapRefusal))
|
||||
return append(out, storeGrantStatuses(ctx, px, cfg, grantRepairer)...)
|
||||
}
|
||||
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
|
||||
@@ -1209,6 +1239,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
|
||||
if localSrv != nil {
|
||||
backupCapRefusal = localSrv.BackupCapRefusal // R-973
|
||||
internalPlaceExec.SetEnsure(localSrv.EnsureInternalPlace)
|
||||
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
|
||||
pwMode := proxmox.RunnerMode(cfg.Privileged.Mode)
|
||||
|
||||
@@ -39,13 +39,25 @@ type savedSuccess struct {
|
||||
target string
|
||||
vmid int
|
||||
at time.Time
|
||||
// recent (R-973) holds the start times of the newest successes (at most maxRecentSuccesses, newest
|
||||
// last) — what the local-API backup cadence cap counts, so an agent restart does not reset the cap.
|
||||
recent []time.Time
|
||||
// refusedAt (R-973) is the newest time the cap refused a backup of this guest on this tier.
|
||||
refusedAt time.Time
|
||||
}
|
||||
|
||||
// backupSuccessJSON is one entry on disk.
|
||||
// maxRecentSuccesses bounds the per-tier success history kept for the cadence cap (R-973). The cap
|
||||
// allows at most 3 runs in a window, so 8 is ample.
|
||||
const maxRecentSuccesses = 8
|
||||
|
||||
// backupSuccessJSON is one entry on disk. Recent and LastRefusedAt are additive (R-973): a file written
|
||||
// before them reads as "no history", and an older agent ignores them.
|
||||
type backupSuccessJSON struct {
|
||||
Target string `json:"target"`
|
||||
VMID int `json:"vmid"`
|
||||
StartedAt string `json:"started_at"`
|
||||
Target string `json:"target"`
|
||||
VMID int `json:"vmid"`
|
||||
StartedAt string `json:"started_at"`
|
||||
Recent []string `json:"recent,omitempty"`
|
||||
LastRefusedAt string `json:"last_refused_at,omitempty"`
|
||||
}
|
||||
|
||||
func backupStateKey(target string, vmid int) string { return target + "/" + strconv.Itoa(vmid) }
|
||||
@@ -67,7 +79,19 @@ func NewBackupSuccessState(path string) *BackupSuccessState {
|
||||
if perr != nil {
|
||||
continue // one unreadable entry must not lose the others
|
||||
}
|
||||
s.last[backupStateKey(e.Target, e.VMID)] = savedSuccess{target: e.Target, vmid: e.VMID, at: t.UTC()}
|
||||
ss := savedSuccess{target: e.Target, vmid: e.VMID, at: t.UTC()}
|
||||
for _, r := range e.Recent {
|
||||
if rt, rerr := time.Parse(time.RFC3339, r); rerr == nil {
|
||||
ss.recent = append(ss.recent, rt.UTC())
|
||||
}
|
||||
}
|
||||
if len(ss.recent) == 0 {
|
||||
ss.recent = []time.Time{ss.at} // a pre-R-973 file: its one known success counts
|
||||
}
|
||||
if rt, rerr := time.Parse(time.RFC3339, e.LastRefusedAt); rerr == nil {
|
||||
ss.refusedAt = rt.UTC()
|
||||
}
|
||||
s.last[backupStateKey(e.Target, e.VMID)] = ss
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -85,13 +109,72 @@ func (s *BackupSuccessState) RecordBackupSuccess(target string, b hub.Backup) er
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
k := backupStateKey(target, b.VMID)
|
||||
if old, ok := s.last[k]; ok && !t.After(old.at) {
|
||||
old, had := s.last[k]
|
||||
if had && !t.After(old.at) {
|
||||
return nil
|
||||
}
|
||||
s.last[k] = savedSuccess{target: target, vmid: b.VMID, at: t.UTC()}
|
||||
ss := savedSuccess{target: target, vmid: b.VMID, at: t.UTC(), refusedAt: old.refusedAt}
|
||||
ss.recent = append(append([]time.Time(nil), old.recent...), t.UTC())
|
||||
if len(ss.recent) > maxRecentSuccesses {
|
||||
ss.recent = ss.recent[len(ss.recent)-maxRecentSuccesses:]
|
||||
}
|
||||
s.last[k] = ss
|
||||
return s.saveLocked()
|
||||
}
|
||||
|
||||
// RecentBackupSuccesses returns the saved success start times for this tier and guest that are not
|
||||
// older than since (R-973 — the cadence cap's restart-proof count).
|
||||
func (s *BackupSuccessState) RecentBackupSuccesses(target string, vmid int, since time.Time) []time.Time {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
var out []time.Time
|
||||
for _, t := range s.last[backupStateKey(target, vmid)].recent {
|
||||
if !t.Before(since) {
|
||||
out = append(out, t)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// RecordBackupRefusal saves that the cadence cap refused a backup of this guest on this tier at at
|
||||
// (R-973), so the operator signal it raises survives an agent restart.
|
||||
func (s *BackupSuccessState) RecordBackupRefusal(target string, vmid int, at time.Time) error {
|
||||
if s == nil {
|
||||
return nil
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
k := backupStateKey(target, vmid)
|
||||
ss, ok := s.last[k]
|
||||
if !ok {
|
||||
return nil // nothing to cap against without a success; the cap never refuses then
|
||||
}
|
||||
if !at.After(ss.refusedAt) {
|
||||
return nil
|
||||
}
|
||||
ss.refusedAt = at.UTC()
|
||||
s.last[k] = ss
|
||||
return s.saveLocked()
|
||||
}
|
||||
|
||||
// LastBackupRefusal returns the newest saved cadence-cap refusal across every tier and guest.
|
||||
func (s *BackupSuccessState) LastBackupRefusal() (target string, vmid int, at time.Time, ok bool) {
|
||||
if s == nil {
|
||||
return "", 0, time.Time{}, false
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
for _, e := range s.last {
|
||||
if !e.refusedAt.IsZero() && e.refusedAt.After(at) {
|
||||
target, vmid, at, ok = e.target, e.vmid, e.refusedAt, true
|
||||
}
|
||||
}
|
||||
return target, vmid, at, ok
|
||||
}
|
||||
|
||||
// LastKnownSuccess returns the newest saved success for this tier and guest (ok=false = none on file).
|
||||
func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Time, bool) {
|
||||
if s == nil {
|
||||
@@ -129,7 +212,14 @@ func (s *BackupSuccessState) KnownBackupSuccesses() []hub.Backup {
|
||||
func (s *BackupSuccessState) saveLocked() error {
|
||||
entries := make([]backupSuccessJSON, 0, len(s.last))
|
||||
for _, e := range s.last {
|
||||
entries = append(entries, backupSuccessJSON{Target: e.target, VMID: e.vmid, StartedAt: e.at.Format(time.RFC3339)})
|
||||
j := backupSuccessJSON{Target: e.target, VMID: e.vmid, StartedAt: e.at.Format(time.RFC3339)}
|
||||
for _, r := range e.recent {
|
||||
j.Recent = append(j.Recent, r.Format(time.RFC3339))
|
||||
}
|
||||
if !e.refusedAt.IsZero() {
|
||||
j.LastRefusedAt = e.refusedAt.Format(time.RFC3339)
|
||||
}
|
||||
entries = append(entries, j)
|
||||
}
|
||||
// Deterministic file content (Go's map order is random).
|
||||
sort.Slice(entries, func(i, j int) bool {
|
||||
|
||||
@@ -74,3 +74,40 @@ func TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen(t *testing.T) {
|
||||
t.Fatal("uncovered_volumes must marshal as [], not null")
|
||||
}
|
||||
}
|
||||
|
||||
// R-973: the cadence cap's history and refusals survive a reopen; the history is bounded; a file
|
||||
// written before R-973 counts its one known success (never "no history" for a tier that has a copy).
|
||||
func TestBackupSuccessState_R973_RecentHistoryAndRefusalSurviveAReopen(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "backup-success-state.json")
|
||||
s := NewBackupSuccessState(path)
|
||||
base := time.Date(2026, 10, 18, 2, 30, 0, 0, time.UTC)
|
||||
for i := 0; i < 10; i++ {
|
||||
at := base.Add(time.Duration(i) * time.Hour)
|
||||
if err := s.RecordBackupSuccess("local", hub.Backup{VMID: 9201, Success: true, StartedAt: at.Format(time.RFC3339)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
refused := base.Add(11 * time.Hour)
|
||||
if err := s.RecordBackupRefusal("local", 9201, refused); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := NewBackupSuccessState(path)
|
||||
all := r.RecentBackupSuccesses("local", 9201, time.Time{})
|
||||
if len(all) != maxRecentSuccesses || !all[len(all)-1].Equal(base.Add(9*time.Hour)) {
|
||||
t.Fatalf("recent history after a reopen: %v (want the newest %d, ending at +9h)", all, maxRecentSuccesses)
|
||||
}
|
||||
if got := r.RecentBackupSuccesses("local", 9201, base.Add(7*time.Hour)); len(got) != 3 {
|
||||
t.Fatalf("successes since +7h: %d, want 3", len(got))
|
||||
}
|
||||
if tg, vm, at, ok := r.LastBackupRefusal(); !ok || tg != "local" || vm != 9201 || !at.Equal(refused) {
|
||||
t.Fatalf("refusal after a reopen: %q %d %v %v", tg, vm, at, ok)
|
||||
}
|
||||
|
||||
old := filepath.Join(t.TempDir(), "old.json")
|
||||
if err := os.WriteFile(old, []byte(`[{"target":"felhom-pbs","vmid":9201,"started_at":"2026-10-17T03:00:00Z"}]`), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := NewBackupSuccessState(old).RecentBackupSuccesses("felhom-pbs", 9201, time.Time{}); len(got) != 1 {
|
||||
t.Fatalf("a pre-R-973 file must count its one success: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// The whole-guest backup cadence cap (R-973, security review 2026-10-12).
|
||||
//
|
||||
// POST /backup is reachable from inside the guest, and every successful run prunes the tier
|
||||
// (`--prune-backups keep-last=N` on a local tier; ep0's server-side keep-last 2 on the off-site tier). So
|
||||
// a compromised controller calling it in a loop pushed every pre-compromise whole-guest restore point out
|
||||
// within minutes — the threat `07` §8 row 10 counts the off-site tier as the answer to. The agent now
|
||||
// refuses a run that the schedule in `07` §6 would never ask for:
|
||||
//
|
||||
// - a tier whose cadence is at most 24 h (the local tier): at most 3 successful runs in any rolling
|
||||
// 24 h. The schedule is one night run; a „Mentés most" press makes a LOCAL copy and, since R-899, the
|
||||
// night still takes its own after it, which can land inside the same 24 h as the previous night (the
|
||||
// window opens anywhere in [W+2h, W+6h)). So a normal day needs at most 3: two nights plus one press.
|
||||
// A second press inside 24 h is refused — the margin is spent on the night, which must never be refused
|
||||
// because of the household's presses.
|
||||
// - a longer tier (the 7-day off-site tier): at most 1 successful run per max(cadence − 24 h, cadence/2)
|
||||
// — 6 days for 7 days. The controller asks /backup/due first and runs only when the newest copy is older
|
||||
// than the cadence, and the button never targets this tier (`07` §6, controller v0.301.0), so a
|
||||
// legitimate run never comes sooner; 24 h of slack absorbs clock and archive-time differences.
|
||||
//
|
||||
// Only SUCCESSES count — a failed run prunes nothing and must stay retryable. The count survives an agent
|
||||
// restart: it reads the success history saved on disk (LastKnownBackupStore, the R-894 file) as well as
|
||||
// this process's own record. A refusal answers 429, is logged at WARN, is saved, and raises the operator
|
||||
// signal BackupCapRefusal feeds (a critical capability in the host report → the hub's existing
|
||||
// `agent_capability_degraded` event, operator-only). Pinned by TestBackupCap_* (backup_cap_test.go).
|
||||
|
||||
// backupCapRefusalHold is how long one refusal keeps the operator signal raised.
|
||||
const backupCapRefusalHold = 24 * time.Hour
|
||||
|
||||
// backupCadenceCap returns the rolling window and the most successful runs allowed in it for a tier.
|
||||
func backupCadenceCap(cadence time.Duration) (window time.Duration, max int) {
|
||||
if cadence <= 0 {
|
||||
cadence = defaultBackupCadence
|
||||
}
|
||||
if cadence <= 24*time.Hour {
|
||||
return 24 * time.Hour, 3
|
||||
}
|
||||
w := cadence - 24*time.Hour
|
||||
if w < cadence/2 {
|
||||
w = cadence / 2
|
||||
}
|
||||
return w, 1
|
||||
}
|
||||
|
||||
// recentCapRuns returns the start times of the successful runs on this tier for this guest inside the
|
||||
// window ending now — this process's record and the saved one, each counted once. Caller holds no lock.
|
||||
func (s *Server) recentCapRuns(key backupJobKey, since time.Time) []time.Time {
|
||||
seen := map[int64]bool{}
|
||||
var out []time.Time
|
||||
add := func(t time.Time) {
|
||||
if t.Before(since) || seen[t.Unix()] {
|
||||
return
|
||||
}
|
||||
seen[t.Unix()] = true
|
||||
out = append(out, t)
|
||||
}
|
||||
s.capMu.Lock()
|
||||
for _, t := range s.capRuns[key] {
|
||||
add(t)
|
||||
}
|
||||
s.capMu.Unlock()
|
||||
if s.lastKnown != nil {
|
||||
for _, t := range s.lastKnown.RecentBackupSuccesses(key.target, key.vmid, since) {
|
||||
add(t)
|
||||
}
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].Before(out[j]) })
|
||||
return out
|
||||
}
|
||||
|
||||
// recordCapRun notes a successful run for the cap (the saved half is written by RecordBackupSuccess).
|
||||
func (s *Server) recordCapRun(key backupJobKey, b hub.Backup) {
|
||||
t, err := time.Parse(time.RFC3339, b.StartedAt)
|
||||
if err != nil {
|
||||
t = s.now()
|
||||
}
|
||||
s.capMu.Lock()
|
||||
defer s.capMu.Unlock()
|
||||
if s.capRuns == nil {
|
||||
s.capRuns = map[backupJobKey][]time.Time{}
|
||||
}
|
||||
runs := append(s.capRuns[key], t.UTC())
|
||||
if len(runs) > 8 {
|
||||
runs = runs[len(runs)-8:]
|
||||
}
|
||||
s.capRuns[key] = runs
|
||||
}
|
||||
|
||||
// capRefuses applies the cap to a new run request. It writes the 429 and returns true when refused.
|
||||
func (s *Server) capRefuses(w http.ResponseWriter, key backupJobKey, tier BackupTier) bool {
|
||||
window, max := backupCadenceCap(tier.Cadence)
|
||||
now := s.now()
|
||||
runs := s.recentCapRuns(key, now.Add(-window))
|
||||
if len(runs) < max {
|
||||
return false
|
||||
}
|
||||
next := runs[len(runs)-max].Add(window)
|
||||
s.capMu.Lock()
|
||||
s.capRefusal = capRefusal{target: key.target, vmid: key.vmid, at: now.UTC()}
|
||||
s.capMu.Unlock()
|
||||
if s.lastKnown != nil {
|
||||
if err := s.lastKnown.RecordBackupRefusal(key.target, key.vmid, now); err != nil {
|
||||
s.logger.Warn("local-api: could not save the backup refusal on disk (R-973)", "vmid", key.vmid, "target", key.target, "err", err)
|
||||
}
|
||||
}
|
||||
s.logger.Warn("local-api: backup REFUSED — more whole-guest backups than the schedule allows (R-973 cadence cap; check the guest)",
|
||||
"vmid", key.vmid, "target", key.target, "runs_in_window", len(runs), "allowed", max,
|
||||
"window_h", int(window.Hours()), "next_allowed", next.UTC().Format(time.RFC3339))
|
||||
writeStatus(w, http.StatusTooManyRequests, false, nil, fmt.Sprintf(
|
||||
"too many whole-guest backups on target %q: %d successful in the last %d h, the schedule allows %d — refused (the next one is allowed after %s)",
|
||||
key.target, len(runs), int(window.Hours()), max, next.UTC().Format(time.RFC3339)))
|
||||
return true
|
||||
}
|
||||
|
||||
type capRefusal struct {
|
||||
target string
|
||||
vmid int
|
||||
at time.Time
|
||||
}
|
||||
|
||||
// BackupCapRefusal reports the newest cadence-cap refusal still inside the 24 h hold (this process's or
|
||||
// the saved one) — the source of the operator signal in the host report (R-973). ok=false = none.
|
||||
func (s *Server) BackupCapRefusal() (target string, vmid int, at time.Time, ok bool) {
|
||||
s.capMu.Lock()
|
||||
r := s.capRefusal
|
||||
s.capMu.Unlock()
|
||||
if s.lastKnown != nil {
|
||||
if t, v, a, found := s.lastKnown.LastBackupRefusal(); found && a.After(r.at) {
|
||||
r = capRefusal{target: t, vmid: v, at: a}
|
||||
}
|
||||
}
|
||||
if r.at.IsZero() || s.now().Sub(r.at) > backupCapRefusalHold {
|
||||
return "", 0, time.Time{}, false
|
||||
}
|
||||
return r.target, r.vmid, r.at, true
|
||||
}
|
||||
@@ -0,0 +1,293 @@
|
||||
package localapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
)
|
||||
|
||||
// R-973 — the whole-guest backup cadence cap. A compromised controller calling POST /backup in a loop
|
||||
// pushed out every pre-compromise restore point: each success prunes the tier. These tests model the tier
|
||||
// the way Proxmox and ep0 keep it (keep-last N, pruned after each success) and ask what is LEFT.
|
||||
|
||||
// keepLastTier is a whole-guest tier: every successful run adds an archive stamped with the clock and
|
||||
// prunes to keepLast, as vzdump's --prune-backups keep-last (local) and ep0's keep-last 2 (off-site) do.
|
||||
type keepLastTier struct {
|
||||
mu sync.Mutex
|
||||
clock *testClock
|
||||
keepLast int
|
||||
archives []time.Time
|
||||
fail bool
|
||||
runs int
|
||||
}
|
||||
|
||||
func (k *keepLastTier) BackupWithSnapshotHook(_ context.Context, vmid int, _ func()) (hub.Backup, error) {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
k.runs++
|
||||
now := k.clock.Now()
|
||||
if k.fail {
|
||||
return hub.Backup{VMID: vmid, StartedAt: now.Format(time.RFC3339)}, fmt.Errorf("vzdump failed")
|
||||
}
|
||||
k.archives = append(k.archives, now)
|
||||
if len(k.archives) > k.keepLast {
|
||||
k.archives = k.archives[len(k.archives)-k.keepLast:]
|
||||
}
|
||||
return hub.Backup{VMID: vmid, Success: true, Archive: "x", StartedAt: now.Format(time.RFC3339)}, nil
|
||||
}
|
||||
|
||||
// oldestBefore reports whether the tier still holds an archive made before t (a pre-compromise point).
|
||||
func (k *keepLastTier) oldestBefore(t time.Time) bool {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return len(k.archives) > 0 && k.archives[0].Before(t)
|
||||
}
|
||||
|
||||
func (k *keepLastTier) count() (runs, kept int) {
|
||||
k.mu.Lock()
|
||||
defer k.mu.Unlock()
|
||||
return k.runs, len(k.archives)
|
||||
}
|
||||
|
||||
type testClock struct {
|
||||
mu sync.Mutex
|
||||
t time.Time
|
||||
}
|
||||
|
||||
func (c *testClock) Now() time.Time { c.mu.Lock(); defer c.mu.Unlock(); return c.t }
|
||||
func (c *testClock) Advance(d time.Duration) { c.mu.Lock(); c.t = c.t.Add(d); c.mu.Unlock() }
|
||||
|
||||
// capServer builds the two-tier box of `07` §6: local 24 h keep-last 3, off-site 7 d keep-last 2, the
|
||||
// success history on disk at path (a second capServer on the same path is an agent restart).
|
||||
func capServer(t *testing.T, path string, clock *testClock, local, offsite *keepLastTier) *Server {
|
||||
t.Helper()
|
||||
srv, err := NewServer(Options{
|
||||
ListenAddr: "127.0.0.1:0", Guests: &fakeGuests{}, Backups: &fakeBackups{}, Store: &fakeStore{},
|
||||
Storage: fakeStorage{targets: []hub.StorageTarget{{Name: "local"}, {Name: "felhom-pbs"}}},
|
||||
Tokens: staticTokens{"A": 8200},
|
||||
BackupTiers: []BackupTier{
|
||||
{TargetID: "local", Cadence: 24 * time.Hour, Primary: true, Service: local},
|
||||
{TargetID: "felhom-pbs", Cadence: 7 * 24 * time.Hour, Service: offsite},
|
||||
},
|
||||
LastKnownBackups: backup.NewBackupSuccessState(path),
|
||||
Logger: slog.New(slog.NewTextHandler(io.Discard, nil)),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv.baseCtx = context.Background()
|
||||
srv.now = clock.Now
|
||||
return srv
|
||||
}
|
||||
|
||||
// runBackup POSTs one backup and waits for the job to end. Returns the HTTP status of the POST.
|
||||
func runBackup(t *testing.T, srv *Server, target string) int {
|
||||
t.Helper()
|
||||
rr := do(t, srv.Handler(), "POST", "/backup?target="+target, "A", "")
|
||||
if rr.Code == http.StatusAccepted {
|
||||
waitFor(t, func() bool {
|
||||
srv.jobsMu.Lock()
|
||||
defer srv.jobsMu.Unlock()
|
||||
j := srv.jobs[backupJobKey{vmid: 8200, target: target}]
|
||||
return j != nil && !backupInFlight(j.Phase)
|
||||
})
|
||||
}
|
||||
return rr.Code
|
||||
}
|
||||
|
||||
// The off-site tier — the one `07` §8 row 10 counts on against ransomware. Before the cap ten calls in a
|
||||
// row left ep0 holding only the attacker's two copies. Now ONE runs (the tier was due), the other nine
|
||||
// are refused, and the newest pre-compromise copy survives; five days of flooding later it still does.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-10-18): make capRefuses always return false → this fails with
|
||||
// "the off-site tier lost every pre-compromise copy: 10 runs" . Restored.
|
||||
func TestBackupCap_OffsiteFloodKeepsAPreCompromiseCopy(t *testing.T) {
|
||||
clock := &testClock{t: testNow}
|
||||
local := &keepLastTier{clock: clock, keepLast: 3}
|
||||
off := &keepLastTier{clock: clock, keepLast: 2}
|
||||
path := filepath.Join(t.TempDir(), "backup-success-state.json")
|
||||
srv := capServer(t, path, clock, local, off)
|
||||
|
||||
// The honest history: two weekly copies, the newest 7 days and 1 hour old (so the tier is due).
|
||||
for _, age := range []time.Duration{14*24*time.Hour + time.Hour, 7*24*time.Hour + time.Hour} {
|
||||
clock.t = testNow.Add(-age)
|
||||
if code := runBackup(t, srv, "felhom-pbs"); code != http.StatusAccepted {
|
||||
t.Fatalf("seeding the honest weekly copy: %d", code)
|
||||
}
|
||||
}
|
||||
clock.t = testNow
|
||||
compromise := testNow
|
||||
|
||||
refused := 0
|
||||
for i := 0; i < 10; i++ {
|
||||
if runBackup(t, srv, "felhom-pbs") == http.StatusTooManyRequests {
|
||||
refused++
|
||||
}
|
||||
clock.Advance(time.Minute)
|
||||
}
|
||||
runs, _ := off.count()
|
||||
if !off.oldestBefore(compromise) {
|
||||
t.Fatalf("the off-site tier lost every pre-compromise copy: %d runs", runs-2)
|
||||
}
|
||||
if runs-2 != 1 || refused != 9 {
|
||||
t.Fatalf("flood: %d runs, %d refused — want 1 run (the tier was due) and 9 refused", runs-2, refused)
|
||||
}
|
||||
for d := 0; d < 5; d++ { // five more days of flooding, every hour
|
||||
for h := 0; h < 24; h++ {
|
||||
clock.Advance(time.Hour)
|
||||
if code := runBackup(t, srv, "felhom-pbs"); code != http.StatusTooManyRequests {
|
||||
t.Fatalf("day %d hour %d: an off-site run was allowed %v after the last one", d, h, clock.Now().Sub(compromise))
|
||||
}
|
||||
}
|
||||
}
|
||||
if !off.oldestBefore(compromise) {
|
||||
t.Fatal("five days of flooding pushed out the pre-compromise off-site copy")
|
||||
}
|
||||
}
|
||||
|
||||
// The local tier: a flood runs exactly 3 in a day (the schedule's own day: two nights + one press) and
|
||||
// every further request is refused. Honest about the limit: with keep-last 3 those 3 replace the 3 kept
|
||||
// points — the cap slows a flush from minutes to a day; it cannot keep a point a 3-per-day pace prunes.
|
||||
func TestBackupCap_LocalFloodRunsAtMostThreeADay(t *testing.T) {
|
||||
clock := &testClock{t: testNow}
|
||||
local := &keepLastTier{clock: clock, keepLast: 3}
|
||||
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
|
||||
refused := 0
|
||||
for i := 0; i < 20; i++ {
|
||||
if runBackup(t, srv, "local") == http.StatusTooManyRequests {
|
||||
refused++
|
||||
}
|
||||
clock.Advance(10 * time.Minute)
|
||||
}
|
||||
if runs, _ := local.count(); runs != 3 || refused != 17 {
|
||||
t.Fatalf("local flood over ~3 h: %d runs, %d refused — want 3 and 17", runs, refused)
|
||||
}
|
||||
clock.Advance(24 * time.Hour) // a day later the window has moved on
|
||||
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
|
||||
t.Fatalf("a run a day later must be allowed: %d", code)
|
||||
}
|
||||
}
|
||||
|
||||
// The schedule itself is never refused: a night at W+6h, a press at 10:00, the next night at W+2h (all
|
||||
// three inside one rolling 24 h) run; a second press that day is refused; nights keep running after it.
|
||||
func TestBackupCap_TheScheduledDayIsNeverRefused(t *testing.T) {
|
||||
day := time.Date(2026, 6, 10, 0, 0, 0, 0, time.UTC)
|
||||
clock := &testClock{t: day.Add(8*time.Hour + 29*time.Minute)} // W=02:30 → W+6h ≈ 08:29
|
||||
local := &keepLastTier{clock: clock, keepLast: 3}
|
||||
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
|
||||
steps := []struct {
|
||||
at time.Time
|
||||
want int
|
||||
what string
|
||||
}{
|
||||
{day.Add(8*time.Hour + 29*time.Minute), http.StatusAccepted, "night 1 at W+6h"},
|
||||
{day.Add(10 * time.Hour), http.StatusAccepted, "a press at 10:00"},
|
||||
{day.Add(24*time.Hour + 4*time.Hour + 30*time.Minute), http.StatusAccepted, "night 2 at W+2h (20 h later)"},
|
||||
{day.Add(24*time.Hour + 7*time.Hour), http.StatusTooManyRequests, "a second press inside the same 24 h"},
|
||||
{day.Add(48*time.Hour + 4*time.Hour + 30*time.Minute), http.StatusAccepted, "night 3"},
|
||||
}
|
||||
for _, s := range steps {
|
||||
clock.t = s.at
|
||||
if got := runBackup(t, srv, "local"); got != s.want {
|
||||
t.Fatalf("%s: got %d, want %d", s.what, got, s.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A restart must not reset the cap: the history is read back from disk.
|
||||
//
|
||||
// COMPANION RED-PROOF (observed 2026-10-18): make recentCapRuns ignore s.lastKnown → this fails with
|
||||
// "after a restart the 4th run in a day was allowed (got 202)". Restored.
|
||||
func TestBackupCap_SurvivesARestart(t *testing.T) {
|
||||
clock := &testClock{t: testNow}
|
||||
path := filepath.Join(t.TempDir(), "s.json")
|
||||
local := &keepLastTier{clock: clock, keepLast: 3}
|
||||
srv := capServer(t, path, clock, local, &keepLastTier{clock: clock, keepLast: 2})
|
||||
for i := 0; i < 3; i++ {
|
||||
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
|
||||
t.Fatalf("run %d: %d", i+1, code)
|
||||
}
|
||||
clock.Advance(time.Minute)
|
||||
}
|
||||
srv2 := capServer(t, path, clock, local, &keepLastTier{clock: clock, keepLast: 2}) // the restart
|
||||
if code := runBackup(t, srv2, "local"); code != http.StatusTooManyRequests {
|
||||
t.Fatalf("after a restart the 4th run in a day was allowed (got %d)", code)
|
||||
}
|
||||
}
|
||||
|
||||
// Failed runs prune nothing and must stay retryable: they do not count.
|
||||
func TestBackupCap_FailuresDoNotCount(t *testing.T) {
|
||||
clock := &testClock{t: testNow}
|
||||
local := &keepLastTier{clock: clock, keepLast: 3, fail: true}
|
||||
srv := capServer(t, filepath.Join(t.TempDir(), "s.json"), clock, local, &keepLastTier{clock: clock, keepLast: 2})
|
||||
for i := 0; i < 6; i++ {
|
||||
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
|
||||
t.Fatalf("failed attempt %d was refused: %d", i+1, code)
|
||||
}
|
||||
clock.Advance(15 * time.Minute)
|
||||
}
|
||||
local.mu.Lock()
|
||||
local.fail = false
|
||||
local.mu.Unlock()
|
||||
if code := runBackup(t, srv, "local"); code != http.StatusAccepted {
|
||||
t.Fatalf("a success after six failures was refused: %d", code)
|
||||
}
|
||||
}
|
||||
|
||||
// A refusal raises the operator signal for 24 h, and the signal survives a restart.
|
||||
func TestBackupCap_RefusalRaisesTheOperatorSignal(t *testing.T) {
|
||||
clock := &testClock{t: testNow}
|
||||
path := filepath.Join(t.TempDir(), "s.json")
|
||||
off := &keepLastTier{clock: clock, keepLast: 2}
|
||||
srv := capServer(t, path, clock, &keepLastTier{clock: clock, keepLast: 3}, off)
|
||||
if _, _, _, ok := srv.BackupCapRefusal(); ok {
|
||||
t.Fatal("a fresh box must not raise the signal")
|
||||
}
|
||||
runBackup(t, srv, "felhom-pbs")
|
||||
clock.Advance(time.Hour)
|
||||
if code := runBackup(t, srv, "felhom-pbs"); code != http.StatusTooManyRequests {
|
||||
t.Fatalf("second off-site run an hour later: %d, want 429", code)
|
||||
}
|
||||
target, vmid, at, ok := srv.BackupCapRefusal()
|
||||
if !ok || target != "felhom-pbs" || vmid != 8200 || !at.Equal(clock.Now()) {
|
||||
t.Fatalf("signal after a refusal: ok=%v target=%q vmid=%d at=%v", ok, target, vmid, at)
|
||||
}
|
||||
clock.Advance(23 * time.Hour)
|
||||
srv2 := capServer(t, path, clock, &keepLastTier{clock: clock, keepLast: 3}, off) // a restart
|
||||
if _, _, _, ok := srv2.BackupCapRefusal(); !ok {
|
||||
t.Fatal("the signal did not survive a restart")
|
||||
}
|
||||
clock.Advance(2 * time.Hour)
|
||||
if _, _, _, ok := srv2.BackupCapRefusal(); ok {
|
||||
t.Fatal("the signal must clear 24 h after the last refusal")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupCadenceCap(t *testing.T) {
|
||||
cases := []struct {
|
||||
cadence time.Duration
|
||||
window time.Duration
|
||||
max int
|
||||
}{
|
||||
{0, 24 * time.Hour, 3},
|
||||
{24 * time.Hour, 24 * time.Hour, 3},
|
||||
{12 * time.Hour, 24 * time.Hour, 3},
|
||||
{7 * 24 * time.Hour, 6 * 24 * time.Hour, 1},
|
||||
{36 * time.Hour, 18 * time.Hour, 1},
|
||||
{30 * 24 * time.Hour, 29 * 24 * time.Hour, 1},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if w, m := backupCadenceCap(c.cadence); w != c.window || m != c.max {
|
||||
t.Errorf("cadence %v: got %v/%d, want %v/%d", c.cadence, w, m, c.window, c.max)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -90,6 +90,10 @@ type BackupStore interface {
|
||||
type LastKnownBackupStore interface {
|
||||
RecordBackupSuccess(target string, b hub.Backup) error
|
||||
LastKnownSuccess(target string, vmid int) (time.Time, bool)
|
||||
// R-973: the cadence cap's restart-proof history and its refusals.
|
||||
RecentBackupSuccesses(target string, vmid int, since time.Time) []time.Time
|
||||
RecordBackupRefusal(target string, vmid int, at time.Time) error
|
||||
LastBackupRefusal() (target string, vmid int, at time.Time, ok bool)
|
||||
}
|
||||
|
||||
// StorageView yields the host's observed storage targets (for mapping a mount's storage id →
|
||||
@@ -302,6 +306,11 @@ type Server struct {
|
||||
logger *slog.Logger
|
||||
now func() time.Time
|
||||
|
||||
// R-973 cadence cap: this process's successful runs per tier+guest, and the newest refusal.
|
||||
capMu sync.Mutex
|
||||
capRuns map[backupJobKey][]time.Time
|
||||
capRefusal capRefusal
|
||||
|
||||
disks DiskOps // slice 8C (optional)
|
||||
diskGate StorageGate // slice 8C (optional)
|
||||
guestList GuestLister // slice 8C (optional)
|
||||
@@ -880,6 +889,12 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
"a backup is already in flight on target "+busyTarget+" (job "+busyJob+") — only one backup runs at a time per guest")
|
||||
return
|
||||
}
|
||||
// R-973: the cadence cap (backup_cap.go). Checked under jobsMu, after the in-flight answers, so two
|
||||
// requests cannot both pass it (only one run per guest is ever in flight).
|
||||
if s.capRefuses(w, key, tier) {
|
||||
s.jobsMu.Unlock()
|
||||
return
|
||||
}
|
||||
// Job ids must be unique PER TIER, and by construction rather than by clock luck: two tiers
|
||||
// started inside the same nanosecond (the weekly both-due night, or any injected clock) would
|
||||
// otherwise collide and hand the second caller the first tier's id. The PRIMARY keeps the
|
||||
@@ -943,6 +958,9 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
|
||||
s.logger.Info("local-api: backup job complete", "vmid", vmid, "target", tier.TargetID, "job", jobID, "archive", b.Archive)
|
||||
}
|
||||
s.store.RecordBackup(b)
|
||||
if b.Success {
|
||||
s.recordCapRun(key, b) // R-973
|
||||
}
|
||||
if b.Success && s.lastKnown != nil {
|
||||
if err := s.lastKnown.RecordBackupSuccess(tier.TargetID, b); err != nil {
|
||||
// Not fatal: the backup exists. Only the fallback after a restart loses this copy.
|
||||
|
||||
Reference in New Issue
Block a user