felhom-controller/REPORT.md

REPORT — R-88 Part 1: the quiesce failure breaker (v0.176.0) (2026-07-27)

Overwritten per the standing rule. Controller v0.175.0 → v0.176.0. MinAgent UNCHANGED — this is controller-internal scheduling; no agent surface is touched.

What was broken

internal/quiesce had no consecutive-failure counter, no backoff and no circuit breaker of any kind, and the loop driver is a plain 5-minute time.Ticker. A tier that was due and kept failing was therefore re-quiesced every five minutes — and a quiesce cycle stops and restarts every customer app stack.

Observed live on demo-felhom 2026-07-27: three full cycles at 09:02:57, 09:07:58 and 09:12:57 Budapest, each stopping and restarting bookstack calibre-web docmost immich (~19 s down per cycle, ~50 s per full cycle) against a PBS tier that could not succeed. It stopped after three only because PBS came back, not because anything gave up. Had the outage lasted, so would the loop.

Phase 0 (read-only) — the three answers

P0.1 — case (a): agent-side. The storage read errored; it was never an empty success. The defect is a type boundary: newestArchiveOn (localapi/server.go:1095-1111) promises in its own comment that "errors … degrade to unknown, never to 'no backup'", but its (time.Time, bool) signature cannot represent unknown — an error and a genuinely-empty storage both become (zero, false). handleBackupDue (server.go:934-941) then emits a positive claim: Due: true, Reason: "no successful backup recorded yet", AgeSecs: nil. The fail-safe that does exist (targetStoragePresent — "a storage-view error must never read as 'not there'") answers a different question and behaved correctly.

Decisive for scoping: the errored path and the genuine-never path are byte-identical on the wire. Same Due, same Reason, same nil AgeSecs — the controller has nothing to discriminate on, so Part 2 cannot be done here. It needs an agent wire change plus a compat rule both ways → own task.

P0.2 — deliberate restarts, not self-update. Four on 2026-07-27 (07:36:39, 07:54:06, 08:50:16, 11:31:52 CEST), all clean Stopping → Stopped → Started, NRestarts=0, Result=success, zero self-update lines. No automatic mechanism arms the trigger — but ordinary operator/config work armed it four times in one day, making the trigger more frequent than "only when ep0 is down", not less.

P0.3 — the loop alerted nobody. Zero backup_failed events, and not because of the allowlist (the hub carries that type). internal/quiesce does not import internal/notify at all. The only trace was 07:13:27 info app_start_failed "Telepített alkalmazás nem fut: BookStack" — a customer-tier, Hungarian, info-severity symptom of the third cycle catching BookStack mid-restart. Filed as R-97; out of scope here.

What shipped

internal/quiesce/breaker.go — consecutive failures per target; a tier inside its backoff is dropped from the due set before any stack is stopped. The gate is on the QUIESCE, not the backup, because the harm was never the failing backup but the outage taken to attempt it.

Backoff 15m → 30m → 1h → 2h → 4h, then 4h forever. The cap is chosen against two real constants: 4 h sits well inside the shortest tier cadence (local 24 h), so a recovered tier still gets several attempts within its own cadence; and it equals the width of the window gate [W+2h, W+6h), so a tier at maximum backoff still gets at least one attempt inside any given night's window rather than stepping over it.

Bounded four ways, each with a test: never permanent (the cap bounds the interval, never stops retrying — a latched breaker is a silent backup outage, worse than the loop); never global; never gates TriggerNow (manual runs still record their outcome, so a manual success clears the backoff); stillRunning is not a failure (a first full offsite snapshot legitimately runs for hours). State is in-memory on purpose — a restart forgets the backoff and re-attempts, the cheap direction to fail.

Part 3 — the invariant is written at the head of scheduledRunAllowed: a missing value means UNKNOWN, and only a positive "never backed up" may fire the safety valve. Fourth instance of the class (hub v0.12.0, v0.73.0, R-81, R-88), all four named in the comment.

Tests — 27 packages ok, 0 failed; build / vet / test each rc=0, run separately

11 new: 7 breaker + 4 contract. Three red-proofs, each observed and restored:

Scenario	Mutation	Observed failure
A	remove the dropBackedOffTiers call	R-88: 3 failing ticks stopped the apps 3 time(s); want 1 — the breaker did not defer anything
D	flip the nil branch to return false	CONTRACT VIOLATED: a never-backed-up box outside its window did NOT back up (0 stack stop(s)) — the safety valve was removed; a box only ever powered on outside its window would starve
F	make the breaker global (any blocked tier halts all)	R-88 Scenario F: the healthy local tier was backed up 2 time(s) across 4 ticks; want >= 3 — a broken tier must not halt a working one

Scenario A asserts a COUNT, not a log line — the harm is the stop/start pairs, so the test counts them, at the exact incident cadence: 3 failing ticks → 3 pairs before the fix, 1 after. (Deleting the D branch outright panics on the nil deref two lines down; return false is the mutation a real over-correction would produce, so that is the one recorded.)

How Scenario D is preserved: the nil branch of scheduledRunAllowed is untouched and still fires the valve, so a genuinely never-backed-up box powered on only outside its window still backs up. TestContract_NeverBackedUp_RunsOutsideTheWindow pins it; TestContract_RecentBackup_DefersOutsideTheWindow is the control (without it the first test would also pass against a gate that was simply disabled); and TestContract_BreakerNeverStarvesAFirstBackup proves the breaker cannot starve a first backup either.

Deployed

gitea.dooplex.hu/admin/felhom-controller:0.176.0 — demo-hp first, then demo-felhom, both Up … (healthy), both image digests matching the DooPlex build byte-for-byte (sha256:98fcbc677bfa1ab8…), both logging [quiesce] loop started (poll 5m0s, max-quiesce 30m0s).

Positive confirmation the loop still works, not merely the absence of complaints. The quiesce loop is silent by design when nothing is due (len(dueTiers) == 0 → return nil), so "no error lines" proves nothing on its own. The observable that does prove it is the agent's per-tier due-check: every /backup/due makes the agent read that tier's storage content (R-84 ground truth), which lands in pveproxy/access.log. On demo-felhom after the deploy:

16:28:44 +0200 GET …/storage/local/content       200
16:28:45 +0200 GET …/storage/felhom-pbs/content  200
16:33:44 +0200 GET …/storage/local/content       200
16:33:45 +0200 GET …/storage/felhom-pbs/content  200

Both tiers polled every 5 minutes, both answering 200 — so the loop is ticking, both tiers are being evaluated, and the storage reads succeed (real ages, no nil, no valve firing). The tick phase shifted from :02:57/:07:58/:12:57 to :28:44/:33:44, which is the 14:23:43 UTC restart resetting the ticker — consistent with the deploy and nothing else. Zero backoff or deferral lines on either box, and no vzdump attempted since the incident's last failure at 07:13:06 UTC.

NOT yet live-validated

• The loop was reproduced in UNIT TESTS ONLY. Reproducing it live needs PBS unavailable, and ep0 must not be taken down for it — a live DR tier on a box with no swap that already OOM'd once today (R-90). No non-production controller was stood up.
• No backoff has armed on real hardware — both boxes are healthy, so what is verified live is the negative: it does not fire spuriously.
• Scenario C is not fixed and not testable here → R-88 Part 2 (agent).
• demo-felhom's local tier becomes due ~2026-07-27 14:53 UTC; with a real age (~24 h, under cadence+24 h) it should defer to the 02:30–06:30 UTC window rather than run — correct behaviour, and it means the first live in-window backup under v0.176.0 lands tomorrow.