felhom.eu/REPORT.md

REPORT — R-182: one email per backup run, nothing dropped without a trace + ep0 rescaled (R-90, R-86)

Date: 2026-08-03 · Repos: felhom-controller v0.194.0 (88897a2), felhom.eu hub v0.90.0 → v0.90.1 (dd40f85, f21e7ca)

1. Baselines — re-read on arrival

Repo	@ arrival	Version	Matched §1?
felhom-controller	db0d4b129d3b	v0.193.1	yes
felhom.eu	7dc1744eece3	hub v0.89.0	yes — my local clone was one commit behind origin (another session had deleted 24 stale REPORT-*.md files); pulled before starting

2. §3's safety property — confirmed at file:line

hub/internal/monitor/deadline.go:396 raises expected_backup_missed and :417 raises expected_dbdump_missed, both from assessBackupFreshness(reportJSON,…) and GetEventsByType — the box's report freshness and stored events, never from an email. So the hub notices a run that never finished independently of any mail the controller chooses to send, and a digest's silence still means "the run finished and found nothing wrong". The whole design rests on this; weakening that check re-opens a silent-failure path.

3. Files

Hub: internal/notify/dispatcher.go (suppression row, cooldownRunSuffix, recordOnlyEvents, operatorOnlyEvents) · internal/notify/templates.go (digest renderer, trimRepeatedUsage) · internal/api/handler.go (allowlist) · internal/store/store.go (status doc) · two new test files · recovery_unit_dispatch_test.go (updated, deliberately) · hub/CHANGELOG.md · manifests/hub.yaml

Docs: CONTEXT.md (S-16) · STATUS.md · documentation/architecture/07-backup-architecture.md (the alerting contract, S-1) · 00-capability-map.md · backlog/{OPEN-ITEMS,ROADMAP}.md · five runbooks corrected · two audit/evidence documents annotated

4. Tests and red-proofs

go build && go vet && go test ./... in hub/ → 17 packages ok, rc=0. repo_gates.py --fast → all 5 gates OK.

#	Mutation	Result
D	the suppression row removed (back to the bare return)	RED — want 1 'suppressed' operator row, got 0
E	cooldownRunSuffix removed from the key	RED — operator mails = 1, want 2; the manual run's digest is swallowed
G	the digest removed from operatorOnlyEvents	RED — and it showed a customer actually receiving the operator digest, demonstrated rather than argued
A	(controller) all four noteFailure feeds removed	RED
I	(controller) the main.go seam commented out	DID NOT FAIL FIRST TIME — see below

Red-proof I is the one worth reading. The AST test walked the backup package but not main.go, so the seam could be disconnected with the whole suite green. Per §9.11 that is not a pass: the test was fixed and the mutation re-run, at which point it failed correctly. The string SetRunSummaryNotify is still present in the commented-out file, which is exactly why this is an AST walk.

One passing test was changed, deliberately. TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer required the OPERATOR to be emailed a per-app capture failure — correct when that event was the only signal, wrong now that it is the record and the digest is the notification. Its customer-safety claim is unchanged and is why the test still exists; the operator assertion is inverted with the reasoning written in place, and it now asserts the recorded row instead. R-158's guarantee moved rather than weakened.

5. The live run

Demo-hp guest 9201, filled for real to 241 MB free / 100% used (thin pool held 30.78 → 30.78). Triggered through POST /api/debug/backup/dbdump — the exact endpoint the debug UI button calls.

The hub's notification_log, quoted:

11:50:45 | recovery_unit_capture_failed | operator | recorded  | record-only: the per-run digest … | Recovery unit capture FAILED for "opengist" …
11:50:45 | recovery_unit_capture_failed | operator | recorded  | record-only: the per-run digest … | Recovery unit capture FAILED for "privatebin" …
11:50:45 | backup_run_failures          | operator | sent      |                                  | 2 of 2 apps failed to back up in this manual run: opengist, …
11:50:45 | backup_run_failures          | customer | skipped   | operator_only                    | 2 of 2 apps failed …
11:51:00 | disk_critical                | operator | suppressed| operator cooldown 1h, key=demo-hp:disk_critical | …
11:51:00 | disk_critical                | operator | sent      |                                  | …

Two failures → both recorded, one email naming both, customer refused with the reason logged. Against the measured previous behaviour: two failures → one email naming one app, one leaving no trace.

Scenario D proved itself on an event I had not planned. disk_critical alarmed on two filesystems; the second was collapsed by the cooldown, and that collapse is now visible with its key. Yesterday it would have left nothing at all — which is the entire finding, demonstrated by accident on a different event type.

Scenario E: a second run in the same hour (11:51:28) produced a second digest. Scenario B: after freeing the space, the run completed with 2 volume dumps and no digest line.

6. The email as shipped

SUBJECT: [Felhom] 🔴 demo-hp: 2 of 2 apps failed to back up (manual run)

Customer: demo-hp
Event:    backup_run_failures
Severity: error
Time:     2026-08-03 13:52 CEST
Message:  2 of 2 apps failed to back up in this manual run: opengist, privatebin

FAILED: 2 of 2 apps attempted in this manual run.

  opengist    whole app (refused before any write)  refused: backing up this app would leave the
                                                    filesystem below the reserve (reserve: 97% used
                                                    or 1.0 GiB free; the filesystem is already below
                                                    it, before this app's estimated 178.0 KB write)
  privatebin  whole app (refused before any write)  refused: … estimated 2.5 KB write

Filesystem: /mnt/sys_drive — 65.0/68.7 GB used (95%), 0.2 GB free

Every failure above is also recorded individually in the notification log,
whether or not this mail was sent.

Dashboard: https://hub.felhom.eu/customers/demo-hp

v0.90.1 came from reading exactly this. In the first rendering every app row also ended with — /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free — the same clause the Filesystem: line already carries. On two apps that is untidy; down a list of a dozen it is forty characters twelve times, pushing the part that differs off a phone screen at 07:00. The reserve's message is unchanged (it is right for a single-app alert); the digest trims the duplicate when rendering, and only an exact — <target path>: suffix.

7. §8.1's table, in practice

The run refused both deployed apps on the reserve → both in the digest. No deliberate skips occurred in the live run (no drive was disconnected or decommissioned on that box), so that row of the table was exercised by test only — TestRunSummary_DeliberateSkipsAreNotFailures, which marks one app disconnected and one decommissioned through the real settings store and asserts no digest at all.

8. ep0 — measured, then corrected

	measured on the box
hostname	felhom-hetzner
RAM	7757 MB (8 GB) — was 3.8
vCPU	4
swapfile	survived the resize — /swapfile, 4 GiB, Jul 27 14:40, active (swapon --show → /swapfile file 4G 0B -2)
local disk	/ 38 G, 58% used — UNCHANGED, as expected for a CPU/RAM resize
datastore volume	/mnt/pbs-datastore 98 G, 8% — unaffected

The swapfile was checked rather than assumed: a resize is a stop/start, so "the interim lever is still in place" was an assumption until measured.

Corrected (five places, three of which the spec's list did not name — found by searching): RUNBOOK-ep0-datastore-volume-2026-07-27.md:6 · RUNBOOK-pbs-prune-serverside-2026-07-27.md:232,:310 · runbooks/offsite-endpoint.md:8,:34 · runbooks/target-selection.md:101.

Annotated, body untouched (they record what was true when written, and that is their value): audits/SPIKE-connectivity-wireguard-2026-07-03.md:13 · tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md:142.

No disk figure was touched anywhere — the 40 GB local disk did not change.

9. Register

ID	Outcome
R-182	CLOSED — SHIPPED, with the live evidence above
R-90	CLOSED — ep0 rescaled to a CX33; measured on the box, swapfile confirmed surviving
R-86	READY, unblocked and re-ranked — endpoint headroom no longer sets the restore-test cadence. Stated honestly: 8 GB is comfortable, not unbounded (the original OOM was a 14.46 GB restore), so the cadence should still be paced — just not by fear of the endpoint

No new IDs were minted, so no grep was needed. R-184 remains the highest.

target-selection.md's ep0 protection question is STILL OPEN and deliberately untouched — "D-d did not name ep0 either way. Confirm it explicitly." That is the operator's to settle, not CC's.

10. Deployed

• Controller 0.194.0 on demo-hp guest 9201 — Up (healthy).
• Hub 0.90.1 — ArgoCD Synced / Healthy, image gitea.dooplex.hu/admin/felhom-hub:0.90.1, startup log clean (Listening on :8080, deadline-check scheduled for 05:00 CEST).

11. Teardown

Fill file removed; guest helper scripts and the credential file shred-ed; pct fstrim 9201 returned 63.3 GiB; thin pool 30.76; the guest is back to 1.9 G used / 64 G free. The scratch email renderer under hub/tmprender/ was deleted after use. The hub DB copy is scratch-only.

12. Observations — NOT acted on

1. Only two apps are deployed on the demo box, so "several apps" was 2 of 2 — exactly the measured case, but a five-app demonstration would be stronger.
2. FormatOperatorEmail puts an emoji in every operator subject. Part 3 asked for none; the icon comes from the shared formatter and changing it globally would alter every other operator mail and its tests. The digest's own copy has no emoji.
3. A recorded row is written for record-only events on the operator channel. It is not an operator notification, so the channel label is slightly overloaded — readable today, worth a thought if a third channel ever appears.
4. The customer-channel skipped/operator_only row no longer appears for the per-app event, because record-only returns before the customer leg. That is correct, but anyone grepping history for that pattern will see it stop on 2026-08-03.