felhom-controller/REPORT.md

REPORT — R-97: a failing backup is HEARD, and stops blaming the apps (v0.177.0) (2026-07-27)

Overwritten per the standing rule. Controller v0.176.0 → v0.177.0; hub v0.77.0 → v0.78.0 (required — the controller emits event types the hub must allowlist). MinAgent unchanged.

Part 0 — ops

Item	Result
0.1 swap on ep0	4 GiB live and persistent. swapon --show → /swapfile file 4G; free -h → Swap: 4.0Gi. Persistence verified without rebooting: systemd-fstab-generator produced swapfile.swap (RequiredBy=swap.target, is-enabled: generated, ActiveState=active), and findmnt --verify --fstab → 0 parse errors, 0 errors. vm.swappiness unchanged at the default 60 — no sysctl override added. Done ahead of Sunday's first-ever GC.
0.2 R-95 mitigation	Snapshot plan IS set on storage-box-pool-1 (611714): {"max_snapshots":7,"minute":0,"hour":0,"day_of_week":null,"day_of_month":null} → daily at 00:00, keep 7. But 0 snapshots exist and size_snapshots=0 — it is ARMED, not yet PROVEN; the first fires tonight. The root exposure remains open: subaccounts are still readonly=False and restic forget --prune still runs from the box.
0.3 R-91 gate	NOT satisfied — do not delete. demo-felhom's newest offsite snapshot is still 2026-07-26T12:21:48Z, i.e. before the volume migration (attached 2026-07-27 06:29:40 UTC). demo-hp has two post-migration (07-27 08:25:47Z, 09:37:29Z). /srv/pbs-felhom remains 13 G.
0.4 residue	Removed /root/.ssh/known_hosts.pre-storagebox.bak and stripped the 3 u629193 lines from /root/.bash_history (0 remaining). CONTEXT.md corrected: the datastore is /mnt/pbs-datastore, not /srv/pbs-felhom.
0.4 R-88 watcher	The first in-window scheduled backup has NOT landed, and could not have. The window is 02:30–06:30 UTC and the fix deployed at 14:23 UTC, so the next opportunity is 2026-07-28. Last vzdump of any kind: the incident's 07:13:06Z FAIL. This is timing, not health — reporting it as "quiet therefore fine" would be the absent-log-line fallacy the new rule 3 forbids.

The cooldown keying finding

Per-event-type only, and it would have masked one tier with the other. dispatcher.go:233 keyed the operator cooldown customerID + ":" + eventType with a hardcoded 1 h window. Correct for an event describing ONE thing; wrong for one describing ONE TIER when a box has two — felhom-pbs failing at 09:00 would swallow local failing at 09:20 for the rest of the hour.

Two clarifications that changed the fix's shape:

• it gates the email only — SaveEvent runs before dispatch, so both events are always stored;
• widening the key for everything would be a flood, not a fix (one hourly app_start_failed becomes one per app).

So the fix is narrow: cooldownTierSuffix appends :<tier> only when the event details carry a tier. No existing event type's behaviour changes. Pinned by TestCooldownTierSuffix_EmptyForEverythingElse.

The grace window (Part 2)

quiesceAlarmGrace = 180 s, derived rather than picked round:

• the deploy flow already allows 120 s for a stack to come up healthy — the project's own existing answer to "how long is too long";
• the slowest catalog healthcheck start_period is Mealie's 60 s, after which a couple of check intervals must still elapse before a verdict means anything.

180 s clears both with margin, and is deliberately not longer: the app-state scan runs on its own cadence, so a genuinely dead app alarms on the first scan after the window closes. The cost is a bounded delay in reporting a real failure, never its loss.

Why a cycle-keyed window and not a state test: v0.164.0's filter is IsDownState(st.State) && st.State != StateStopped and suppresses deliberately stopped apps. BookStack alarmed because the third cycle caught it mid-restart — starting/unhealthy, not StateStopped. No state classification can distinguish "restarting because a backup stopped me" from "restarting because I keep crashing"; the distinguishing fact is that we stopped it, and we know it. Consumed at the same single derivation point (classifyRunStates) that computes both the banner dead-list and the notifier Down-set — still one place.

Tests — controller 27 pkg ok / hub 17 pkg ok, 0 failed; build/vet/test each rc=0, run separately, vet unpiped

Controller +9 (8 quiesce + 1 wiring reachability); hub +8. Red-proofs observed and restored:

Scenario	Mutation	Observed failure
C	key the breaker on a constant (one global failure record)	both tiers must be reported; got local=1 felhom-pbs=0 — one broken tier must not mask another
E	SuppressedStacks returns nil (pre-fix shape)	R-97b: the apps we stopped for a backup must not be reported down; bookstack still alarmed
F	suppression never expires (permanent)	CONTRACT VIOLATED: suppression must EXPIRE — bookstack is still suppressed 3m1s after the cycle; permanent suppression is a silent real alarm

The end-to-end event evidence

Induced deliberately, and it travelled. A real whole_guest_backup_failed was POSTed from inside guest 9201, with the controller's own hub URL and API key, over the live /api/v1/event route — the exact path PushEvent uses. Message prefixed [R-97a VERIFICATION - induced by Claude Code, NOT a real fault], details {"tier":"felhom-pbs",...}.

1. the new type            → HTTP 200 {"ok":true}
2. CONTROL: bogus type     → HTTP 400 "Invalid event_type: whole_guest_backup_bogus"

The control matters: without it, a 200 is equally consistent with "my type is allowlisted" and "the gate is off". At the far end, from a freshly copied hub DB (mtime checked — a stale copy already produced one confident wrong answer today):

events:            2026-07-27 15:04:57  error  whole_guest_backup_failed  [R-97a VERIFICATION …]
notification_log:  2026-07-27 15:04:58  whole_guest_backup_failed  channel=operator  status=sent
                   OPERATOR rows: 1   CUSTOMER rows: 0

Operator notified, customer not — the operator-only property confirmed positively, not assumed. No storage target was altered and nothing was pointed at a dead endpoint, so nothing needed restoring.

Deployed

• hub v0.78.0 — GitOps: manifest bumped, ArgoCD hard-refresh + deliberate sync → Synced, rollout complete, pod image felhom-hub:0.78.0, startup clean. Deployed FIRST, so the controller never emits a type the hub would 400.
• controller v0.177.0 — demo-hp first, then demo-felhom; both Up (healthy).

Post-deploy on both boxes: 0 spurious suppression/backoff lines, 0 startup errors, quiesce loop started, all app stacks up (10 on demo-felhom, 3 on demo-hp). The positive observable still lands — both tiers polled every 5 min with 200s in pveproxy/access.log.

NOT yet live-validated

• The quiesce → notify link is UNIT-PROVEN ONLY. The hub half is proven live end-to-end with a control (above), and the adapter's reachability is pinned by TestQuiesceTierNotifierIsWired — but no real quiesce failure has fired the seam on hardware, because inducing one means an app outage on a demo box. The join is proven by test + inspection, not observation.
• No R-88 backoff has ever armed on real hardware. Both boxes are healthy, so the breaker's behaviour under a failing tier still rests on unit tests alone. Unchanged by this task.
• R-97b's suppression has never fired live — it needs a real quiesce cycle, i.e. tomorrow's in-window backup at the earliest.
• The 180 s grace has not been measured against a real restart on these boxes; it is derived from the deploy flow's 120 s and Mealie's 60 s, not observed.
• The snapshot plan (0.2) is armed but has taken 0 snapshots — first one due tonight at 00:00.
• demo-felhom's first in-window scheduled backup under v0.176.0/v0.177.0 lands 2026-07-28 at the earliest.

OPEN-ITEMS rows touched

Closed: R-97 (both halves shipped) · R-96 (three standing rules committed to CLAUDE.md) · R-90 (swap done; the CX33 rescale stays open as the durable fix). Re-ranked: R-95 split — mitigation ARMED (awaiting first snapshot), root exposure still READY. Unchanged: R-91 (gate not satisfied) · R-88b (agent wire change) · R-86 · R-87.