# REPORT — R-182: one email per backup run, nothing dropped without a trace + ep0 rescaled (R-90, R-86) **Date:** 2026-08-03 · **Repos:** `felhom-controller` **v0.194.0** (`88897a2`), `felhom.eu` hub **v0.90.0 → v0.90.1** (`dd40f85`, `f21e7ca`) ## 1. Baselines — re-read on arrival | Repo | @ arrival | Version | Matched §1? | |---|---|---|---| | `felhom-controller` | `db0d4b129d3b` | v0.193.1 | **yes** | | `felhom.eu` | `7dc1744eece3` | hub v0.89.0 | **yes** — my local clone was one commit behind origin (another session had deleted 24 stale `REPORT-*.md` files); pulled before starting | ## 2. §3's safety property — confirmed at file:line `hub/internal/monitor/deadline.go:396` raises `expected_backup_missed` and `:417` raises `expected_dbdump_missed`, both from `assessBackupFreshness(reportJSON,…)` and `GetEventsByType` — the box's **report freshness and stored events**, never from an email. So the hub notices a run that never finished **independently of any mail the controller chooses to send**, and a digest's silence still means "the run finished and found nothing wrong". **The whole design rests on this; weakening that check re-opens a silent-failure path.** ## 3. Files **Hub:** `internal/notify/dispatcher.go` (suppression row, `cooldownRunSuffix`, `recordOnlyEvents`, `operatorOnlyEvents`) · `internal/notify/templates.go` (digest renderer, `trimRepeatedUsage`) · `internal/api/handler.go` (allowlist) · `internal/store/store.go` (status doc) · two new test files · `recovery_unit_dispatch_test.go` (updated, deliberately) · `hub/CHANGELOG.md` · `manifests/hub.yaml` **Docs:** `CONTEXT.md` (S-16) · `STATUS.md` · `documentation/architecture/07-backup-architecture.md` (the alerting contract, S-1) · `00-capability-map.md` · `backlog/{OPEN-ITEMS,ROADMAP}.md` · five runbooks corrected · two audit/evidence documents annotated ## 4. Tests and red-proofs `go build && go vet && go test ./...` in `hub/` → **17 packages ok, rc=0**. `repo_gates.py --fast` → all 5 gates OK. | # | Mutation | Result | |---|---|---| | D | the suppression row removed (back to the bare `return`) | **RED** — `want 1 'suppressed' operator row, got 0` | | E | `cooldownRunSuffix` removed from the key | **RED** — `operator mails = 1, want 2`; the manual run's digest is swallowed | | G | the digest removed from `operatorOnlyEvents` | **RED** — and it showed **a customer actually receiving the operator digest**, demonstrated rather than argued | | A | (controller) all four `noteFailure` feeds removed | **RED** | | I | (controller) the `main.go` seam commented out | **DID NOT FAIL FIRST TIME** — see below | **Red-proof I is the one worth reading.** The AST test walked the backup package but not `main.go`, so the seam could be disconnected with the whole suite green. Per §9.11 that is not a pass: **the test was fixed and the mutation re-run**, at which point it failed correctly. The string `SetRunSummaryNotify` is still present in the commented-out file, which is exactly why this is an AST walk. **One passing test was changed, deliberately.** `TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer` required the OPERATOR to be emailed a per-app capture failure — correct when that event was the only signal, wrong now that it is the record and the digest is the notification. Its **customer-safety claim is unchanged** and is why the test still exists; the operator assertion is inverted with the reasoning written in place, and it now asserts the `recorded` row instead. R-158's guarantee **moved rather than weakened**. ## 5. The live run Demo-hp guest 9201, filled for real to **241 MB free / 100% used** (thin pool held **30.78 → 30.78**). Triggered through `POST /api/debug/backup/dbdump` — the exact endpoint the debug UI button calls. **The hub's `notification_log`, quoted:** ``` 11:50:45 | recovery_unit_capture_failed | operator | recorded | record-only: the per-run digest … | Recovery unit capture FAILED for "opengist" … 11:50:45 | recovery_unit_capture_failed | operator | recorded | record-only: the per-run digest … | Recovery unit capture FAILED for "privatebin" … 11:50:45 | backup_run_failures | operator | sent | | 2 of 2 apps failed to back up in this manual run: opengist, … 11:50:45 | backup_run_failures | customer | skipped | operator_only | 2 of 2 apps failed … 11:51:00 | disk_critical | operator | suppressed| operator cooldown 1h, key=demo-hp:disk_critical | … 11:51:00 | disk_critical | operator | sent | | … ``` Two failures → **both recorded**, **one** email naming **both**, customer refused with the reason logged. Against the measured previous behaviour: two failures → one email naming one app, one leaving no trace. **Scenario D proved itself on an event I had not planned.** `disk_critical` alarmed on two filesystems; the second was collapsed by the cooldown, and that collapse is now **visible with its key**. Yesterday it would have left nothing at all — which is the entire finding, demonstrated by accident on a different event type. **Scenario E:** a second run in the same hour (11:51:28) produced a second digest. **Scenario B:** after freeing the space, the run completed with 2 volume dumps and **no** digest line. ## 6. The email as shipped ``` SUBJECT: [Felhom] 🔴 demo-hp: 2 of 2 apps failed to back up (manual run) Customer: demo-hp Event: backup_run_failures Severity: error Time: 2026-08-03 13:52 CEST Message: 2 of 2 apps failed to back up in this manual run: opengist, privatebin FAILED: 2 of 2 apps attempted in this manual run. opengist whole app (refused before any write) refused: backing up this app would leave the filesystem below the reserve (reserve: 97% used or 1.0 GiB free; the filesystem is already below it, before this app's estimated 178.0 KB write) privatebin whole app (refused before any write) refused: … estimated 2.5 KB write Filesystem: /mnt/sys_drive — 65.0/68.7 GB used (95%), 0.2 GB free Every failure above is also recorded individually in the notification log, whether or not this mail was sent. Dashboard: https://hub.felhom.eu/customers/demo-hp ``` **v0.90.1 came from reading exactly this.** In the first rendering every app row also ended with `— /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free` — the same clause the `Filesystem:` line already carries. On two apps that is untidy; down a list of a dozen it is forty characters twelve times, pushing the part that *differs* off a phone screen at 07:00. The reserve's message is unchanged (it is right for a single-app alert); the digest trims the duplicate when rendering, and only an exact `— :` suffix. ## 7. §8.1's table, in practice The run refused **both** deployed apps on the reserve → both in the digest. **No** deliberate skips occurred in the live run (no drive was disconnected or decommissioned on that box), so that row of the table was exercised by test only — `TestRunSummary_DeliberateSkipsAreNotFailures`, which marks one app disconnected and one decommissioned through the real `settings` store and asserts **no digest at all**. ## 8. ep0 — measured, then corrected | | measured on the box | |---|---| | hostname | `felhom-hetzner` | | RAM | **7757 MB (8 GB)** — was 3.8 | | vCPU | **4** | | swapfile | **survived the resize** — `/swapfile`, 4 GiB, `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`) | | local disk | `/` 38 G, 58% used — **UNCHANGED**, as expected for a CPU/RAM resize | | datastore volume | `/mnt/pbs-datastore` 98 G, 8% — unaffected | The swapfile was **checked rather than assumed**: a resize is a stop/start, so "the interim lever is still in place" was an assumption until measured. **Corrected** (five places, three of which the spec's list did not name — found by searching): `RUNBOOK-ep0-datastore-volume-2026-07-27.md:6` · `RUNBOOK-pbs-prune-serverside-2026-07-27.md:232,:310` · `runbooks/offsite-endpoint.md:8,:34` · **`runbooks/target-selection.md:101`**. **Annotated, body untouched** (they record what was true when written, and that is their value): `audits/SPIKE-connectivity-wireguard-2026-07-03.md:13` · `tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md:142`. **No disk figure was touched anywhere** — the 40 GB local disk did not change. ## 9. Register | ID | Outcome | |---|---| | **R-182** | **CLOSED — SHIPPED**, with the live evidence above | | **R-90** | **CLOSED** — ep0 rescaled to a CX33; measured on the box, swapfile confirmed surviving | | **R-86** | **READY, unblocked and re-ranked** — endpoint headroom no longer sets the restore-test cadence. Stated honestly: 8 GB is comfortable, not unbounded (the original OOM was a 14.46 GB restore), so the cadence should still be paced — just not by fear of the endpoint | **No new IDs were minted**, so no grep was needed. R-184 remains the highest. **`target-selection.md`'s ep0 protection question is STILL OPEN and deliberately untouched** — *"D-d did not name ep0 either way. Confirm it explicitly."* That is the operator's to settle, not CC's. ## 10. Deployed - Controller **0.194.0** on demo-hp guest 9201 — `Up (healthy)`. - Hub **0.90.1** — ArgoCD `Synced` / `Healthy`, image `gitea.dooplex.hu/admin/felhom-hub:0.90.1`, startup log clean (`Listening on :8080`, deadline-check scheduled for 05:00 CEST). ## 11. Teardown Fill file removed; guest helper scripts and the credential file `shred`-ed; `pct fstrim 9201` returned 63.3 GiB; thin pool **30.76**; the guest is back to 1.9 G used / 64 G free. The scratch email renderer under `hub/tmprender/` was deleted after use. The hub DB copy is scratch-only. ## 12. Observations — NOT acted on 1. **Only two apps are deployed on the demo box**, so "several apps" was 2 of 2 — exactly the measured case, but a five-app demonstration would be stronger. 2. **`FormatOperatorEmail` puts an emoji in every operator subject.** Part 3 asked for none; the icon comes from the shared formatter and changing it globally would alter every other operator mail and its tests. The digest's own copy has no emoji. 3. **A `recorded` row is written for record-only events on the `operator` channel.** It is not an operator *notification*, so the channel label is slightly overloaded — readable today, worth a thought if a third channel ever appears. 4. **The customer-channel `skipped/operator_only` row no longer appears for the per-app event**, because record-only returns before the customer leg. That is correct, but anyone grepping history for that pattern will see it stop on 2026-08-03.