felhom.eu/REPORT.md

# REPORT — R-182: one email per backup run, nothing dropped without a trace + ep0 rescaled (R-90, R-86)

**Date:** 2026-08-03 · **Repos:** `felhom-controller` **v0.194.0** (`88897a2`), `felhom.eu` hub
**v0.90.0 → v0.90.1** (`dd40f85`, `f21e7ca`)

## 1. Baselines — re-read on arrival

| Repo | @ arrival | Version | Matched §1? |
|---|---|---|---|
| `felhom-controller` | `db0d4b129d3b` | v0.193.1 | **yes** |
| `felhom.eu` | `7dc1744eece3` | hub v0.89.0 | **yes** — my local clone was one commit behind origin (another session had deleted 24 stale `REPORT-*.md` files); pulled before starting |

## 2. §3's safety property — confirmed at file:line

`hub/internal/monitor/deadline.go:396` raises `expected_backup_missed` and `:417` raises
`expected_dbdump_missed`, both from `assessBackupFreshness(reportJSON,…)` and `GetEventsByType` — the
box's **report freshness and stored events**, never from an email. So the hub notices a run that never
finished **independently of any mail the controller chooses to send**, and a digest's silence still
means "the run finished and found nothing wrong". **The whole design rests on this; weakening that
check re-opens a silent-failure path.**

## 3. Files

**Hub:** `internal/notify/dispatcher.go` (suppression row, `cooldownRunSuffix`, `recordOnlyEvents`,
`operatorOnlyEvents`) · `internal/notify/templates.go` (digest renderer, `trimRepeatedUsage`) ·
`internal/api/handler.go` (allowlist) · `internal/store/store.go` (status doc) · two new test files ·
`recovery_unit_dispatch_test.go` (updated, deliberately) · `hub/CHANGELOG.md` · `manifests/hub.yaml`

**Docs:** `CONTEXT.md` (S-16) · `STATUS.md` · `documentation/architecture/07-backup-architecture.md`
(the alerting contract, S-1) · `00-capability-map.md` · `backlog/{OPEN-ITEMS,ROADMAP}.md` · five
runbooks corrected · two audit/evidence documents annotated

## 4. Tests and red-proofs

`go build && go vet && go test ./...` in `hub/` → **17 packages ok, rc=0**. `repo_gates.py --fast` →
all 5 gates OK.

| # | Mutation | Result |
|---|---|---|
| D | the suppression row removed (back to the bare `return`) | **RED** — `want 1 'suppressed' operator row, got 0` |
| E | `cooldownRunSuffix` removed from the key | **RED** — `operator mails = 1, want 2`; the manual run's digest is swallowed |
| G | the digest removed from `operatorOnlyEvents` | **RED** — and it showed **a customer actually receiving the operator digest**, demonstrated rather than argued |
| A | (controller) all four `noteFailure` feeds removed | **RED** |
| I | (controller) the `main.go` seam commented out | **DID NOT FAIL FIRST TIME** — see below |

**Red-proof I is the one worth reading.** The AST test walked the backup package but not `main.go`, so
the seam could be disconnected with the whole suite green. Per §9.11 that is not a pass: **the test was
fixed and the mutation re-run**, at which point it failed correctly. The string `SetRunSummaryNotify`
is still present in the commented-out file, which is exactly why this is an AST walk.

**One passing test was changed, deliberately.**
`TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer` required the OPERATOR to be emailed a per-app
capture failure — correct when that event was the only signal, wrong now that it is the record and the
digest is the notification. Its **customer-safety claim is unchanged** and is why the test still
exists; the operator assertion is inverted with the reasoning written in place, and it now asserts the
`recorded` row instead. R-158's guarantee **moved rather than weakened**.

## 5. The live run

Demo-hp guest 9201, filled for real to **241 MB free / 100% used** (thin pool held **30.78 → 30.78**).
Triggered through `POST /api/debug/backup/dbdump` — the exact endpoint the debug UI button calls.

**The hub's `notification_log`, quoted:**

```
11:50:45 | recovery_unit_capture_failed | operator | recorded  | record-only: the per-run digest … | Recovery unit capture FAILED for "opengist" …
11:50:45 | recovery_unit_capture_failed | operator | recorded  | record-only: the per-run digest … | Recovery unit capture FAILED for "privatebin" …
11:50:45 | backup_run_failures          | operator | sent      |                                  | 2 of 2 apps failed to back up in this manual run: opengist, …
11:50:45 | backup_run_failures          | customer | skipped   | operator_only                    | 2 of 2 apps failed …
11:51:00 | disk_critical                | operator | suppressed| operator cooldown 1h, key=demo-hp:disk_critical | …
11:51:00 | disk_critical                | operator | sent      |                                  | …
```

Two failures → **both recorded**, **one** email naming **both**, customer refused with the reason
logged. Against the measured previous behaviour: two failures → one email naming one app, one leaving
no trace.

**Scenario D proved itself on an event I had not planned.** `disk_critical` alarmed on two
filesystems; the second was collapsed by the cooldown, and that collapse is now **visible with its
key**. Yesterday it would have left nothing at all — which is the entire finding, demonstrated by
accident on a different event type.

**Scenario E:** a second run in the same hour (11:51:28) produced a second digest.
**Scenario B:** after freeing the space, the run completed with 2 volume dumps and **no** digest line.

## 6. The email as shipped

```
SUBJECT: [Felhom] 🔴 demo-hp: 2 of 2 apps failed to back up (manual run)

Customer: demo-hp
Event:    backup_run_failures
Severity: error
Time:     2026-08-03 13:52 CEST
Message:  2 of 2 apps failed to back up in this manual run: opengist, privatebin

FAILED: 2 of 2 apps attempted in this manual run.

  opengist    whole app (refused before any write)  refused: backing up this app would leave the
                                                    filesystem below the reserve (reserve: 97% used
                                                    or 1.0 GiB free; the filesystem is already below
                                                    it, before this app's estimated 178.0 KB write)
  privatebin  whole app (refused before any write)  refused: … estimated 2.5 KB write

Filesystem: /mnt/sys_drive — 65.0/68.7 GB used (95%), 0.2 GB free

Every failure above is also recorded individually in the notification log,
whether or not this mail was sent.

Dashboard: https://hub.felhom.eu/customers/demo-hp
```

**v0.90.1 came from reading exactly this.** In the first rendering every app row also ended with
`— /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free` — the same clause the `Filesystem:` line
already carries. On two apps that is untidy; down a list of a dozen it is forty characters twelve
times, pushing the part that *differs* off a phone screen at 07:00. The reserve's message is unchanged
(it is right for a single-app alert); the digest trims the duplicate when rendering, and only an exact
`— <target path>:` suffix.

## 7. §8.1's table, in practice

The run refused **both** deployed apps on the reserve → both in the digest. **No** deliberate skips
occurred in the live run (no drive was disconnected or decommissioned on that box), so that row of the
table was exercised by test only — `TestRunSummary_DeliberateSkipsAreNotFailures`, which marks one app
disconnected and one decommissioned through the real `settings` store and asserts **no digest at all**.

## 8. ep0 — measured, then corrected

| | measured on the box |
|---|---|
| hostname | `felhom-hetzner` |
| RAM | **7757 MB (8 GB)** — was 3.8 |
| vCPU | **4** |
| swapfile | **survived the resize** — `/swapfile`, 4 GiB, `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`) |
| local disk | `/` 38 G, 58% used — **UNCHANGED**, as expected for a CPU/RAM resize |
| datastore volume | `/mnt/pbs-datastore` 98 G, 8% — unaffected |

The swapfile was **checked rather than assumed**: a resize is a stop/start, so "the interim lever is
still in place" was an assumption until measured.

**Corrected** (five places, three of which the spec's list did not name — found by searching):
`RUNBOOK-ep0-datastore-volume-2026-07-27.md:6` · `RUNBOOK-pbs-prune-serverside-2026-07-27.md:232,:310`
· `runbooks/offsite-endpoint.md:8,:34` · **`runbooks/target-selection.md:101`**.

**Annotated, body untouched** (they record what was true when written, and that is their value):
`audits/SPIKE-connectivity-wireguard-2026-07-03.md:13` ·
`tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md:142`.

**No disk figure was touched anywhere** — the 40 GB local disk did not change.

## 9. Register

| ID | Outcome |
|---|---|
| **R-182** | **CLOSED — SHIPPED**, with the live evidence above |
| **R-90** | **CLOSED** — ep0 rescaled to a CX33; measured on the box, swapfile confirmed surviving |
| **R-86** | **READY, unblocked and re-ranked** — endpoint headroom no longer sets the restore-test cadence. Stated honestly: 8 GB is comfortable, not unbounded (the original OOM was a 14.46 GB restore), so the cadence should still be paced — just not by fear of the endpoint |

**No new IDs were minted**, so no grep was needed. R-184 remains the highest.

**`target-selection.md`'s ep0 protection question is STILL OPEN and deliberately untouched** —
*"D-d did not name ep0 either way. Confirm it explicitly."* That is the operator's to settle, not CC's.

## 10. Deployed

- Controller **0.194.0** on demo-hp guest 9201 — `Up (healthy)`.
- Hub **0.90.1** — ArgoCD `Synced` / `Healthy`, image `gitea.dooplex.hu/admin/felhom-hub:0.90.1`,
  startup log clean (`Listening on :8080`, deadline-check scheduled for 05:00 CEST).

## 11. Teardown

Fill file removed; guest helper scripts and the credential file `shred`-ed; `pct fstrim 9201` returned
63.3 GiB; thin pool **30.76**; the guest is back to 1.9 G used / 64 G free. The scratch email renderer
under `hub/tmprender/` was deleted after use. The hub DB copy is scratch-only.

## 12. Observations — NOT acted on

1. **Only two apps are deployed on the demo box**, so "several apps" was 2 of 2 — exactly the measured
   case, but a five-app demonstration would be stronger.
2. **`FormatOperatorEmail` puts an emoji in every operator subject.** Part 3 asked for none; the icon
   comes from the shared formatter and changing it globally would alter every other operator mail and
   its tests. The digest's own copy has no emoji.
3. **A `recorded` row is written for record-only events on the `operator` channel.** It is not an
   operator *notification*, so the channel label is slightly overloaded — readable today, worth a
   thought if a third channel ever appears.
4. **The customer-channel `skipped/operator_only` row no longer appears for the per-app event**,
   because record-only returns before the customer leg. That is correct, but anyone grepping history
   for that pattern will see it stop on 2026-08-03.