felhom-controller/REPORT.md

REPORT — R-82 Slice B: one quiesce window, two backup tiers (v0.175.0) (2026-07-26)

Overwritten per the standing rule. Controller v0.174.0 → v0.175.0. Full cross-repo arc: felhom.eu/REPORT.md.

MinAgent UNCHANGED — deliberately. This release degrades gracefully against ANY older agent. It does not require agent v0.97.0.

---

1. Why the controller had to change

The agent gained per-target backup tiers (R-82 Slice A): local daily + offsite weekly. But the controller owns quiescing — it polls /backup/due, stops the app stacks, calls POST /backup, and restarts them. So the multi-tier schedule has to be reconciled here, and one night a week both tiers come due at once.

Two quiesce cycles that night would mean two app outages for one night's work — undoing the entire argument for weekly-over-daily, since the quiesce blip was the only real cost of the offsite tier.

2. The dedup rule — specified, not emergent

local due	PBS due	result
yes	no	one quiesce, local backup
no	yes	one quiesce, PBS backup
yes	yes	ONE quiesce window, BOTH backups inside it
no	no	no quiesce

3. What shipped

• quiesce.TieredBackend (optional extension to Backend) + BackupTier, ErrTiersUnsupported. A backend that does not implement it drives the pre-R-82 path unchanged.
• agentapi: BackupTiers, BackupDueFor, StartBackupFor, BackupStatusFor. targetQuery("") yields an empty suffix, so an untargeted call hits the untargeted route byte-for-byte.
• Loop.resolveDueTiers — the dedup rule in one place, tiers in agent order.
• quiesceAndPollTiers + pollTier — one marker, one stop, N sequential backups, one resume.

4. Two decisions worth stating plainly

The app stays quiesced until the LAST tier snapshots. Resuming after tier 1's snapshot would leave the following tier capturing a running app — losing app-consistency on exactly the DR tier we most want it on. The consequence is user-visible and documented: on the both-due night downtime is (first tier's full backup) + (last tier's snapshot), not one snapshot. Tiers therefore run fast-first — vzdump holds a guest lock so they are necessarily sequential, and the agent advertises primary (local) first. The reverse order would make downtime ≈ (offsite backup) + (local snapshot), far worse.

A manual "Mentés most" covers EVERY tier, in one window, due-ness ignored. A manual run that silently skipped the DR tier would be the same applied-and-empty fault in a different costume.

5. Capability detection

GET /backup/tiers 404 ⇒ pre-R-82 agent. This is the project's documented route-probe mechanism (internal/agentapi/features.go). It is deliberately not a featureProbes row: that table answers a yes/no at a UI entry point, whereas the loop needs the tier list, so a row would be a second probe of the same route for no gain. The degrade logs exactly once per process — once because it is a steady state during a rollout, never zero times because a silent degrade is indistinguishable from multi-tier working.

6. v0.175.0 — a tier that overruns the quiesce bound defers the rest

Operator ruling 2026-07-26: "let the first backup run as long as needed; other backups shouldn't start until finished."

A first FULL offsite snapshot legitimately runs for hours. When max_quiesce elapses the app resumes — correct, unchanged — but the loop then started the next tier while the first was still uploading. That is now a break.

Why it matters: vzdump still holds the guest lock, so the second start would be refused by the agent (409, v0.99.0) or fail on the lock — and a failed backup never satisfies a cadence, so the tier would stay permanently due and retry into the same wall every poll.

7. Tests

go build ./... && go vet ./... && go test ./... — rc=0, 27 packages (vet run unpiped; note grep FAIL returns rc=1 when it finds nothing, which is not a failure signal — that idiom caused a red commit elsewhere in this arc).

+12 tests in internal/quiesce/tiers_test.go. Red-proofs observed and restored:

• Both-due night — a per-tier cycle instead of one window fails with want EXACTLY 1 stop and 1 start, got stops=2 starts=2. The COUNT is the assertion; asserting only "both backups ran" would pass against a double-quiesce implementation.
• New controller ↔ old agent — treating ErrTiersUnsupported as "nothing due" fails with OLD AGENT: a backup MUST still be taken via the untargeted path; got started=[]. The hollow version of this test asserts only "no error", which passes while silently skipping the backup.
• Overrun defer — dropping the break fails with the second tier MUST NOT start while the first is still running; started=[local felhom-pbs].

8. NOT live-validated — and NOT deployed

The controller change is unit-proven only. Both production controllers still run v0.173.0; this release was deliberately not deployed, because the multi-tier path only engages against an agent that serves /backup/tiers, and the agent rollout was still mid-flight (demo-hp's first offsite backup in progress). Specifically unproven live:

1. A real both-due night with exactly one stop/start pair observed on a box.
2. The degrade path against a genuinely old agent in production.
3. The overrun defer firing on a real long-running offsite backup.

Deploying is the natural next step once demo-hp's first offsite backup lands and the agent rollout is settled.