felhom-controller/REPORT.md

# REPORT — R-82 Slice B: one quiesce window, two backup tiers (v0.175.0) (2026-07-26)

**Overwritten** per the standing rule. Controller **v0.174.0 → v0.175.0**.
Full cross-repo arc: `felhom.eu/REPORT.md`.

**MinAgent UNCHANGED — deliberately.** This release degrades gracefully against ANY older agent. It
does not require agent v0.97.0.

---

## 1. Why the controller had to change

The agent gained per-target backup tiers (R-82 Slice A): local **daily** + offsite **weekly**. But
the **controller** owns quiescing — it polls `/backup/due`, stops the app stacks, calls `POST
/backup`, and restarts them. So the multi-tier schedule has to be reconciled here, and one night a
week both tiers come due at once.

Two quiesce cycles that night would mean **two app outages for one night's work** — undoing the
entire argument for weekly-over-daily, since the quiesce blip was the only real cost of the offsite
tier.

## 2. The dedup rule — specified, not emergent

| local due | PBS due | result |
|---|---|---|
| yes | no | one quiesce, local backup |
| no | yes | one quiesce, PBS backup |
| **yes** | **yes** | **ONE quiesce window, BOTH backups inside it** |
| no | no | no quiesce |

## 3. What shipped

- **`quiesce.TieredBackend`** (optional extension to `Backend`) + `BackupTier`,
  `ErrTiersUnsupported`. A backend that does not implement it drives the pre-R-82 path unchanged.
- **`agentapi`**: `BackupTiers`, `BackupDueFor`, `StartBackupFor`, `BackupStatusFor`.
  `targetQuery("")` yields an **empty** suffix, so an untargeted call hits the untargeted route
  byte-for-byte.
- **`Loop.resolveDueTiers`** — the dedup rule in one place, tiers in agent order.
- **`quiesceAndPollTiers` + `pollTier`** — one marker, one stop, N sequential backups, one resume.

## 4. Two decisions worth stating plainly

**The app stays quiesced until the LAST tier snapshots.** Resuming after tier 1's snapshot would
leave the following tier capturing a **running** app — losing app-consistency on exactly the DR tier
we most want it on. The consequence is user-visible and documented: on the both-due night downtime is
*(first tier's full backup)* + *(last tier's snapshot)*, not one snapshot. Tiers therefore run
**fast-first** — vzdump holds a guest lock so they are necessarily sequential, and the agent
advertises primary (local) first. The reverse order would make downtime ≈ *(offsite backup)* +
*(local snapshot)*, far worse.

**A manual "Mentés most" covers EVERY tier**, in one window, due-ness ignored. A manual run that
silently skipped the DR tier would be the same applied-and-empty fault in a different costume.

## 5. Capability detection

`GET /backup/tiers` 404 ⇒ pre-R-82 agent. This is the project's documented **route-probe** mechanism
(`internal/agentapi/features.go`). It is deliberately **not** a `featureProbes` row: that table
answers a yes/no at a UI entry point, whereas the loop needs the tier **list**, so a row would be a
second probe of the same route for no gain. The degrade logs **exactly once per process** — once
because it is a steady state during a rollout, never zero times because a silent degrade is
indistinguishable from multi-tier working.

## 6. v0.175.0 — a tier that overruns the quiesce bound defers the rest

Operator ruling 2026-07-26: *"let the first backup run as long as needed; other backups shouldn't
start until finished."*

A first FULL offsite snapshot legitimately runs for **hours**. When `max_quiesce` elapses the app
resumes — correct, unchanged — but the loop then started the **next** tier while the first was still
uploading. That is now a `break`.

Why it matters: vzdump still holds the guest lock, so the second start would be **refused by the
agent (409, v0.99.0)** or fail on the lock — and a failed backup never satisfies a cadence, so the
tier would stay permanently due and retry into the same wall every poll.

## 7. Tests

`go build ./... && go vet ./... && go test ./...` — **rc=0, 27 packages** (vet run unpiped; note
`grep FAIL` returns rc=1 when it finds nothing, which is not a failure signal — that idiom caused a
red commit elsewhere in this arc).

+12 tests in `internal/quiesce/tiers_test.go`. Red-proofs observed and restored:

- **Both-due night** — a per-tier cycle instead of one window fails with
  `want EXACTLY 1 stop and 1 start, got stops=2 starts=2`. **The COUNT is the assertion**; asserting
  only "both backups ran" would pass against a double-quiesce implementation.
- **New controller ↔ old agent** — treating `ErrTiersUnsupported` as "nothing due" fails with
  `OLD AGENT: a backup MUST still be taken via the untargeted path; got started=[]`. The hollow
  version of this test asserts only "no error", which passes while silently skipping the backup.
- **Overrun defer** — dropping the `break` fails with
  `the second tier MUST NOT start while the first is still running; started=[local felhom-pbs]`.

## 8. Deployment — DEPLOYED 2026-07-26 18:22 CEST (this section was stale; corrected)

**Live on both boxes:** `felhom-controller:0.175.0`, healthy.

### The correction, and why the record disagreed with itself

This section previously read *"NOT live-validated — and NOT deployed"*. That was **true when
written** and went **stale 26 minutes later**:

| CEST | event |
|---|---|
| 17:56:10 | this REPORT committed (`f5e1064`) — "not deployed", accurate at that moment |
| 18:22:56 | controller **0.175.0 deployed** to both boxes (container `StartedAt` 16:22:56 **UTC**) |
| 19:01:39 | the multi-tier quiesce cycle ran (controller log line `17:01:39` — **UTC**) |

**The defect is mine and it is real:** `REPORT.md` is defined as "overwrite with the most recent
state", and I deployed without updating it. An artifact this project validates against was left
asserting the opposite of reality for the rest of the session.

**One thing to be precise about, because it nearly became a second wrong conclusion:** the apparent
*contradiction* — a 17:01 quiesce log "before" a 17:56 commit — is a **timezone artifact, not
evidence of a false claim**. The controller logs **UTC**; git commit timestamps here are **CEST**.
Converted to one clock the ordering is consistent and unremarkable: REPORT 17:56 → deploy 18:22 →
quiesce 19:01. That host=CEST / controller=UTC mismatch is already recorded as a project trap and it
caught me once earlier the same day.

So: a stale artifact (real, mine) — not a claim contradicted by its own evidence.

## 9. Live validation

**The multi-tier quiesce ran on demo-hp**, driven through the real UI endpoint
(`POST /api/guest-backup/trigger`, session auth + CSRF — the exact call "Mentés most" makes):

```
17:01:39Z  manual backup requested — quiescing now
17:01:39Z  backup due on 2 tier(s) — quiescing 1 stack(s): [paperless-ngx]      <- ONE stop
17:01:46Z  tier local: backup job ... started
17:02:56Z  tier local: ... done — next tier may start (app still quiesced)      <- app stays DOWN
17:02:56Z  tier felhom-pbs: backup job backup-9201-felhom-pbs-... started
17:03:06Z  tier felhom-pbs: ... snapshotted — resuming app early (8B.2)
17:03:06Z  unquiescing (snapshotted (early resume, last tier)): restarting 1 stack(s)   <- ONE start
```

**Exactly one stop/start pair with both backups inside it** — the assertion that matters, since
"both backups ran" would also pass against an implementation that quiesces twice. Tier order
local-first/PBS-last as designed; the app stayed quiesced *through* the non-last tier, preserving
app-consistency on the DR tier; it resumed at the **last** tier's snapshot rather than its upload.
**Total app downtime 1m27s for both tiers**, and paperless came back healthy.

## 10. Still NOT live-validated

1. A **scheduled** both-due night (this was the manual `TriggerNow` path; the scheduled path shares
   `quiesceAndPollTiers` but has not been observed firing on its own).
2. The **degrade path** against a genuinely pre-R-82 agent in production — both boxes now run an
   agent that serves `/backup/tiers`, so the fallback has not been exercised live.
3. The **overrun defer** firing on a real long-running offsite backup.