felhom-controller/REPORT.md

REPORT — R-82 Slice B: one quiesce window, two backup tiers (v0.175.0) (2026-07-26)

Overwritten per the standing rule. Controller v0.174.0 → v0.175.0. Full cross-repo arc: felhom.eu/REPORT.md.

MinAgent UNCHANGED — deliberately. This release degrades gracefully against ANY older agent. It does not require agent v0.97.0.

---

1. Why the controller had to change

The agent gained per-target backup tiers (R-82 Slice A): local daily + offsite weekly. But the controller owns quiescing — it polls /backup/due, stops the app stacks, calls POST /backup, and restarts them. So the multi-tier schedule has to be reconciled here, and one night a week both tiers come due at once.

Two quiesce cycles that night would mean two app outages for one night's work — undoing the entire argument for weekly-over-daily, since the quiesce blip was the only real cost of the offsite tier.

2. The dedup rule — specified, not emergent

local due	PBS due	result
yes	no	one quiesce, local backup
no	yes	one quiesce, PBS backup
yes	yes	ONE quiesce window, BOTH backups inside it
no	no	no quiesce

3. What shipped

• quiesce.TieredBackend (optional extension to Backend) + BackupTier, ErrTiersUnsupported. A backend that does not implement it drives the pre-R-82 path unchanged.
• agentapi: BackupTiers, BackupDueFor, StartBackupFor, BackupStatusFor. targetQuery("") yields an empty suffix, so an untargeted call hits the untargeted route byte-for-byte.
• Loop.resolveDueTiers — the dedup rule in one place, tiers in agent order.
• quiesceAndPollTiers + pollTier — one marker, one stop, N sequential backups, one resume.

4. Two decisions worth stating plainly

The app stays quiesced until the LAST tier snapshots. Resuming after tier 1's snapshot would leave the following tier capturing a running app — losing app-consistency on exactly the DR tier we most want it on. The consequence is user-visible and documented: on the both-due night downtime is (first tier's full backup) + (last tier's snapshot), not one snapshot. Tiers therefore run fast-first — vzdump holds a guest lock so they are necessarily sequential, and the agent advertises primary (local) first. The reverse order would make downtime ≈ (offsite backup) + (local snapshot), far worse.

A manual "Mentés most" covers EVERY tier, in one window, due-ness ignored. A manual run that silently skipped the DR tier would be the same applied-and-empty fault in a different costume.

5. Capability detection

GET /backup/tiers 404 ⇒ pre-R-82 agent. This is the project's documented route-probe mechanism (internal/agentapi/features.go). It is deliberately not a featureProbes row: that table answers a yes/no at a UI entry point, whereas the loop needs the tier list, so a row would be a second probe of the same route for no gain. The degrade logs exactly once per process — once because it is a steady state during a rollout, never zero times because a silent degrade is indistinguishable from multi-tier working.

6. v0.175.0 — a tier that overruns the quiesce bound defers the rest

Operator ruling 2026-07-26: "let the first backup run as long as needed; other backups shouldn't start until finished."

A first FULL offsite snapshot legitimately runs for hours. When max_quiesce elapses the app resumes — correct, unchanged — but the loop then started the next tier while the first was still uploading. That is now a break.

Why it matters: vzdump still holds the guest lock, so the second start would be refused by the agent (409, v0.99.0) or fail on the lock — and a failed backup never satisfies a cadence, so the tier would stay permanently due and retry into the same wall every poll.

7. Tests

go build ./... && go vet ./... && go test ./... — rc=0, 27 packages (vet run unpiped; note grep FAIL returns rc=1 when it finds nothing, which is not a failure signal — that idiom caused a red commit elsewhere in this arc).

+12 tests in internal/quiesce/tiers_test.go. Red-proofs observed and restored:

• Both-due night — a per-tier cycle instead of one window fails with want EXACTLY 1 stop and 1 start, got stops=2 starts=2. The COUNT is the assertion; asserting only "both backups ran" would pass against a double-quiesce implementation.
• New controller ↔ old agent — treating ErrTiersUnsupported as "nothing due" fails with OLD AGENT: a backup MUST still be taken via the untargeted path; got started=[]. The hollow version of this test asserts only "no error", which passes while silently skipping the backup.
• Overrun defer — dropping the break fails with the second tier MUST NOT start while the first is still running; started=[local felhom-pbs].

8. Deployment — DEPLOYED 2026-07-26 18:22 CEST (this section was stale; corrected)

Live on both boxes: felhom-controller:0.175.0, healthy.

The correction, and why the record disagreed with itself

This section previously read "NOT live-validated — and NOT deployed". That was true when written and went stale 26 minutes later:

CEST	event
17:56:10	this REPORT committed (f5e1064) — "not deployed", accurate at that moment
18:22:56	controller 0.175.0 deployed to both boxes (container StartedAt 16:22:56 UTC)
19:01:39	the multi-tier quiesce cycle ran (controller log line 17:01:39 — UTC)

The defect is mine and it is real: REPORT.md is defined as "overwrite with the most recent state", and I deployed without updating it. An artifact this project validates against was left asserting the opposite of reality for the rest of the session.

One thing to be precise about, because it nearly became a second wrong conclusion: the apparent contradiction — a 17:01 quiesce log "before" a 17:56 commit — is a timezone artifact, not evidence of a false claim. The controller logs UTC; git commit timestamps here are CEST. Converted to one clock the ordering is consistent and unremarkable: REPORT 17:56 → deploy 18:22 → quiesce 19:01. That host=CEST / controller=UTC mismatch is already recorded as a project trap and it caught me once earlier the same day.

So: a stale artifact (real, mine) — not a claim contradicted by its own evidence.

9. Live validation

The multi-tier quiesce ran on demo-hp, driven through the real UI endpoint (POST /api/guest-backup/trigger, session auth + CSRF — the exact call "Mentés most" makes):

17:01:39Z  manual backup requested — quiescing now
17:01:39Z  backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx]      <- ONE stop
17:01:46Z  tier local: backup job ... started
17:02:56Z  tier local: ... done — next tier may start (app still quiesced)      <- app stays DOWN
17:02:56Z  tier felhom-pbs: backup job backup-9201-felhom-pbs-... started
17:03:06Z  tier felhom-pbs: ... snapshotted — resuming app early (8B.2)
17:03:06Z  unquiescing (snapshotted (early resume, last tier)): restarting 1 stack(s)   <- ONE start

Exactly one stop/start pair with both backups inside it — the assertion that matters, since "both backups ran" would also pass against an implementation that quiesces twice. Tier order local-first/PBS-last as designed; the app stayed quiesced through the non-last tier, preserving app-consistency on the DR tier; it resumed at the last tier's snapshot rather than its upload. Total app downtime 1m27s for both tiers, and paperless came back healthy.

10. Still NOT live-validated

1. A scheduled both-due night (this was the manual TriggerNow path; the scheduled path shares quiesceAndPollTiers but has not been observed firing on its own).
2. The degrade path against a genuinely pre-R-82 agent in production — both boxes now run an agent that serves /backup/tiers, so the fallback has not been exercised live.
3. The overrun defer firing on a real long-running offsite backup.