admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected
gates / gates (push) Successful in 7s
Details

Browse Source 
R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The
hub's notification_log for the run reads: two per-app failures RECORDED, one
digest SENT naming both, and the customer channel SKIPPED with operator_only.
Against the measured previous behaviour — two failures, one email naming one
app, one leaving no trace anywhere.

Scenario D proved itself on an event I had not planned: disk_critical alarmed on
two filesystems, the second was collapsed by the cooldown, and that collapse is
now visible WITH ITS KEY. Yesterday it would have left nothing at all.

A gap the spec did not anticipate is recorded with its fix: the per-app event
also fires from the periodic sweep, outside any run, so making it record-only
would have created a NEW silence. The sweep emits a digest too, with no run_id,
so it stays under the ordinary hourly cooldown.

ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile
SURVIVED the resize and is active (checked, because a resize is a stop/start).
The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere.

Five documents corrected — three of which the task's list did not name, found by
searching. Two audit/evidence documents ANNOTATED, body untouched: they record
what was true when written and that is their value.

R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable,
not unbounded — the original OOM was a 14.46 GB restore — so the restore-test
cadence should still be paced, just not by fear of the endpoint.

target-selection.md's "D-d did not name ep0 either way" is deliberately left
standing. It is the operator's question, not CC's.

STATUS.md 127 -> 83 lines, items rather than sentences.
This commit is contained in:
authentik Default Admin
2026-08-03 14:00:54 +02:00
parent f21e7caed1
commit e34b614e5b
13 changed files with 279 additions and 279 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+162 -174
View File
@@ -1,200 +1,188 @@
# REPORT — publishing becomes an act, not a side-effect (R-110, R-115) + R-182 measured, R-183/R-184 filed
# REPORT — R-182: one email per backup run, nothing dropped without a trace + ep0 rescaled (R-90, R-86)
**Date:** 2026-08-03 · **Repos:** `felhom.eu` (installer **v1.22.0 → v1.23.0**), `felhom-agent` (**no bump**)
**Nothing was built** — no image, no binary, no golden. **Hub stays v0.89.0.**
**Date:** 2026-08-03 · **Repos:** `felhom-controller` **v0.194.0** (`88897a2`), `felhom.eu` hub
**v0.90.0 → v0.90.1** (`dd40f85`, `f21e7ca`)
## 1. Baselines — re-read on arrival, both matched §1
## 1. Baselines — re-read on arrival
| Repo | @ arrival | Version | Result |
| Repo | @ arrival | Version | Matched §1? |
|---|---|---|---|
| `felhom.eu` | `8360f940bfb2` | hub v0.89.0, `SCRIPT_VERSION="1.22.0"`, **0 tags** (confirmed) | installer **v1.23.0**, first tag `installer-v1.23.0` |
| `felhom-agent` | `9dfd89cb947e` | v0.120.0 | **unchanged** — scripts and gates only |
| `felhom-controller` | `db0d4b129d3b` | v0.193.1 | **yes** |
| `felhom.eu` | `7dc1744eece3` | hub v0.89.0 | **yes** — my local clone was one commit behind origin (another session had deleted 24 stale `REPORT-*.md` files); pulled before starting |
## 2. Part 0 — the R-182 measurement, and it REVERSED the row
## 2. §3's safety property — confirmed at file:line
Filed yesterday as *"the reserve re-alerts on every status refresh"* — **too many** alerts, observed
at the sending end. Measured at the **receiving end**, it is the opposite.
`hub/internal/monitor/deadline.go:396` raises `expected_backup_missed` and `:417` raises
`expected_dbdump_missed`, both from `assessBackupFreshness(reportJSON,…)` and `GetEventsByType` — the
box's **report freshness and stored events**, never from an email. So the hub notices a run that never
finished **independently of any mail the controller chooses to send**, and a digest's silence still
means "the run finished and found nothing wrong". **The whole design rests on this; weakening that
check re-opens a silent-failure path.**
Method: the hub's SQLite copied **with its `-wal`** (4 MB and newer than the db — copying `hub.db`
alone would have read stale data, the exact trap this project recorded before), freshness confirmed by
the newest `notification_log` row post-dating the session.
## 3. Files
**9 `recovery_unit_capture_failed` events received today → 2 operator emails sent.**
**Hub:** `internal/notify/dispatcher.go` (suppression row, `cooldownRunSuffix`, `recordOnlyEvents`,
`operatorOnlyEvents`) · `internal/notify/templates.go` (digest renderer, `trimRepeatedUsage`) ·
`internal/api/handler.go` (allowlist) · `internal/store/store.go` (status doc) · two new test files ·
`recovery_unit_dispatch_test.go` (updated, deliberately) · `hub/CHANGELOG.md` · `manifests/hub.yaml`
| time | apps refused (events in) | operator emails out |
|---|---|---|
| 06:40:03 | privatebin, opengist | **opengist only** |
| 08:59:46/47 | opengist, privatebin | **privatebin only** |
| 08:59:59 | privatebin, opengist | **none** |
| 09:03:00 | opengist | **none** |
| 09:07:06 | privatebin, opengist | **none** |
**Docs:** `CONTEXT.md` (S-16) · `STATUS.md` · `documentation/architecture/07-backup-architecture.md`
(the alerting contract, S-1) · `00-capability-map.md` · `backlog/{OPEN-ITEMS,ROADMAP}.md` · five
runbooks corrected · two audit/evidence documents annotated
**Cause, confirmed at source:** the operator cooldown key is
`customerID + ":" + eventType + cooldownTierSuffix(details)` (`dispatcher.go:268`, 1 hour hardcoded).
`RecoveryUnitFailureDetails` carries **`app`** and **no `tier`**, so the suffix is empty and the key
holds **no app identifier**. The first refused app takes the slot; every other app's refusal for the
next hour is dropped — and dropped **before `LogNotification`**, so it leaves **no row on any
channel** and cannot be audited afterwards.
## 4. Tests and red-proofs
This is **R-97a's failure mode in a second event type**; that row's own comment states it
(*"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20"*). `cooldownTierSuffix` was
written narrow on purpose; `recovery_unit_capture_failed` simply never opted in.
**A correction I owe on yesterday's report.** It said *"one `recovery_unit_capture_failed` per app,
HTTP 200"*. That was true of what the **controller pushed**, and a reader would take it as *the
operator was told about each app* — which is false. The gap between an accepted event and a sent
email is the whole of this row.
**Nothing was changed** (§8.5). R-182 is re-scoped with the evidence and the fix shape.
## 3. Probes
| | Question | Method | Verdict |
|---|---|---|---|
| **P-A** | does git-sync v4.4.0 follow a tag, and notice a **moved** one? | throwaway `docker run` git-sync against this repo, tag moved under it | **PASS both halves** — `update required … local:fb65202 remote:8360f94` → `updated successfully`, one period (~20 s) |
| **P-B** | does Gitea serve `raw/tag/<tag>/<path>`? | one fetch on a throwaway tag | **PASS** — HTTP 200, byte-identical to `raw/branch/main` |
| **P-C** | can CI read the package registry? | anonymous fetches | **PARTIAL, and it changed the gate's design** — package **download** 200 (and **404** for a fake version, so it discriminates), **tags** api 200; package **listing** api **401**, hub artifact manifest **401** |
**Publish model P-A implies:** publishing is **moving the tag**; rollback is **moving it back**, in
~30 s with no ArgoCD sync and no deploy. Probe teardown: container, sync tree and probe tag all gone
(`git ls-remote --tags` → 0 at the time).
## 4. §8.2's three channels — enumerated
| Channel | Before | After | |
|---|---|---|---|
| 1. the served script | `main`, 30 s | **`installer-v1.23.0`** | **MOVED** — `webpage.yaml` split into two syncs |
| 2. the run-time fetches | `raw/branch/main` | **`raw/tag/v$ART_AGENT_VER`** | **MOVED** — but see below |
| 3. the URL producers | `main` | unchanged | **NO CHANGE NEEDED** — and that is a finding, not an omission |
**Channel 2 was not what the spec described, and the spec's mechanism for it was unimplementable.**
There are **sixteen** fetches, not nine, and they come from **`felhom-agent`**, not `felhom.eu` — so
no tag on this repo could ever have covered them, and §8.1's *"derive the tag from `SCRIPT_VERSION`"*
was impossible for them. Raised before building; operator ruled to pin them to **the agent version
being installed**, which the installer already resolves from the hub manifest and already sha-verifies.
That is strictly better than any installer-derived tag: binary and configs now come from one ref.
**Channel 3 needed no change because the URL never carried a ref** —
`https://felhom.eu/scripts/felhom-host-install.sh` is path-based; the ref lives in the manifest. So
`felhom-bootstrap.sh` and the hub's day-0 command follow the tag automatically. **No hub template
change ⇒ no hub bump**, so §1's rule was never in tension and the STOP it anticipated never arose.
## 5. The tag convention
- **Shape:** `installer-v<SCRIPT_VERSION>` in `felhom.eu` (prefixed so it cannot be read as a hub,
agent, controller or golden version); `v<semver>` in `felhom-agent` (that repo versions one thing).
**No new constant in the installer** — channel 2 derives its ref from `$ART_AGENT_VER` at run time,
and channel 1's ref lives only in the manifest.
- **Publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml` (sidecar *and*
init container), commit, sync.
- **Roll back:** move the tag back to the previous commit — takes ~30 s, **no ArgoCD sync, no deploy**.
## 6. Scenario A — proven by HTTP
A real commit was pushed to `main` (a marker comment in the installer) **without moving the tag**, and
three sync periods were allowed to pass so "unchanged" means "had every chance to change":
```
website tree (main): .worktrees/6a82719… <- ADVANCED to the new commit
scripts tree (tag): .worktrees/bee6848… <- STAYED
sha256 before push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
sha256 after push: 2f859555382c4c69c18c48dccd8d8b132ffd49b4dbe4e03e5dbb192e8d883555
marker present at the served URL? 0
https://felhom.eu/ -> HTTP 200
```
Both halves of the split in one observation: the site still tracks `main`, the installer does not.
## 7. Scenario B — publish and rollback, both directions
| act | result |
|---|---|
| tag moved `bee6848 → 6a82719` | scripts tree moved in **~40 s**; served `sha256 ea2b4aa9…`; **marker present** |
| tag moved back `→ bee6848` | scripts tree back in **~40 s**; served `sha256 2f859555…` — **exactly** the pre-publish sha; **marker gone** |
`https://felhom.eu/` returned 200 throughout. The marker commit was then reverted, and the tag moved
to `main`'s head — a **byte no-op**, verified by the served sha not changing.
## 8. Files, commits, tags
**`felhom.eu`** — `bee6848` (installer + gate + manifest), `6a82719` (Scenario A marker), `e79a20b`
(marker removed), plus the docs commit below.
`scripts/felhom-host-install.sh` · `scripts/hostinstall_gates.py` · `scripts/CHANGELOG.md` ·
`manifests/webpage.yaml` · `CLAUDE.md` · `CONTEXT.md` · `STATUS.md` · `REPORT.md` ·
`documentation/backlog/{OPEN-ITEMS,ROADMAP}.md` · `documentation/architecture/00-capability-map.md`
**`felhom-agent`** — `dd2d1fe` (release path + gate + CI), `0db7766` (REPORT).
`scripts/release-agent.sh` **(new)** · `scripts/check-published-versions.py` **(new)** ·
`scripts/agent_gates.py` · `.gitea/workflows/gates.yml` · `CLAUDE.md` · `CHANGELOG.md` · `REPORT.md`
**Tags created:** `felhom.eu/installer-v1.23.0` (the first tag this repo has ever had) and
`felhom-agent/v0.120.0` (retroactive, at `cd6e267` — the commit the published binary was built from;
`configs/` is byte-identical there and at `main`, so nothing depended on the choice).
## 9. Tests and red-proofs
| Check | Result |
|---|---|
| `felhom.eu` `repo_gates.py --fast` | all 5 gates OK |
| `felhom-agent` `go build ./... && go vet ./...` | OK |
| `felhom-agent` `go test ./...` | **29 packages ok, rc=0** (read separately from any commit) |
| `agent_gates.py --fast` | `published` correctly **SKIPPED** (hook must not fail on a network blip) |
| `agent_gates.py` (full) | both OK |
**Red-proofs, each demonstrated failing then restored:**
`go build && go vet && go test ./...` in `hub/` → **17 packages ok, rc=0**. `repo_gates.py --fast` →
all 5 gates OK.
| # | Mutation | Result |
|---|---|---|
| C | one of the sixteen fetches reverted to `raw/branch/main` | **RED** — gate 6a *and* 6b both fired |
| D | assertions 6a **and** 6b removed (every guard the test covers), same bad installer | **zero** mentions of the regression — the guards are what catch it |
| 6c | the manifest before the split | **RED** on its own, before I fixed it — the gate was demonstrated red by the real pre-change state |
| F | `v9.9.9` tagged and not published | **RED**, `binary NOT downloadable (HTTP 404 …)`, rc=1 |
| F′ | the gate **deregistered** from `agent_gates.py`, same bad state | **rc=0, "all agent gates OK"** — restored → `CONVICTED: published`, rc=1 |
| D | the suppression row removed (back to the bare `return`) | **RED** — `want 1 'suppressed' operator row, got 0` |
| E | `cooldownRunSuffix` removed from the key | **RED** — `operator mails = 1, want 2`; the manual run's digest is swallowed |
| G | the digest removed from `operatorOnlyEvents` | **RED** — and it showed **a customer actually receiving the operator digest**, demonstrated rather than argued |
| A | (controller) all four `noteFailure` feeds removed | **RED** |
| I | (controller) the `main.go` seam commented out | **DID NOT FAIL FIRST TIME** — see below |
**Scenario F measured on real CI, not inferred.** Runs **69** and **70** are on the *same commit*
`0db7766`: **success** before `v9.9.9` existed, **failure** after pushing it. One variable. This also
retrospectively explains runs 67/68. **One deliberate CI failure email reached the operator — that was
this proof, not an incident.** I could not read CI's own step log: the jobs endpoint needs a Gitea API
token, and the only credential available (`~/.docker/config.json`) is a registry password that the API
rejects — so the controlled before/after replaced the log rather than an assumption standing in for it.
**Red-proof I is the one worth reading.** The AST test walked the backup package but not `main.go`, so
the seam could be disconnected with the whole suite green. Per §9.11 that is not a pass: **the test was
fixed and the mutation re-run**, at which point it failed correctly. The string `SetRunSummaryNotify`
is still present in the commented-out file, which is exactly why this is an AST walk.
## 10. No version bumps, nothing built
**One passing test was changed, deliberately.**
`TestRecoveryUnitCaptureFailed_NeverReachesTheCustomer` required the OPERATOR to be emailed a per-app
capture failure — correct when that event was the only signal, wrong now that it is the record and the
digest is the notification. Its **customer-safety claim is unchanged** and is why the test still
exists; the operator assertion is inverted with the reasoning written in place, and it now asserts the
`recorded` row instead. R-158's guarantee **moved rather than weakened**.
`felhom-agent` **v0.120.0** unchanged (no Go code changed). Hub **v0.89.0** unchanged (no hub file
touched). The installer's `SCRIPT_VERSION` **did** go 1.22.0 → 1.23.0 — the installer is not in §12's
no-bump list, its behaviour changed materially, and the tag derives from it. No image, binary or
golden was built.
## 5. The live run
## 11. Register
Demo-hp guest 9201, filled for real to **241 MB free / 100% used** (thin pool held **30.78 → 30.78**).
Triggered through `POST /api/debug/backup/dbdump` — the exact endpoint the debug UI button calls.
**The hub's `notification_log`, quoted:**
```
11:50:45 | recovery_unit_capture_failed | operator | recorded | record-only: the per-run digest … | Recovery unit capture FAILED for "opengist" …
11:50:45 | recovery_unit_capture_failed | operator | recorded | record-only: the per-run digest … | Recovery unit capture FAILED for "privatebin" …
11:50:45 | backup_run_failures | operator | sent | | 2 of 2 apps failed to back up in this manual run: opengist, …
11:50:45 | backup_run_failures | customer | skipped | operator_only | 2 of 2 apps failed …
11:51:00 | disk_critical | operator | suppressed| operator cooldown 1h, key=demo-hp:disk_critical | …
11:51:00 | disk_critical | operator | sent | | …
```
Two failures → **both recorded**, **one** email naming **both**, customer refused with the reason
logged. Against the measured previous behaviour: two failures → one email naming one app, one leaving
no trace.
**Scenario D proved itself on an event I had not planned.** `disk_critical` alarmed on two
filesystems; the second was collapsed by the cooldown, and that collapse is now **visible with its
key**. Yesterday it would have left nothing at all — which is the entire finding, demonstrated by
accident on a different event type.
**Scenario E:** a second run in the same hour (11:51:28) produced a second digest.
**Scenario B:** after freeing the space, the run completed with 2 volume dumps and **no** digest line.
## 6. The email as shipped
```
SUBJECT: [Felhom] 🔴 demo-hp: 2 of 2 apps failed to back up (manual run)
Customer: demo-hp
Event: backup_run_failures
Severity: error
Time: 2026-08-03 13:52 CEST
Message: 2 of 2 apps failed to back up in this manual run: opengist, privatebin
FAILED: 2 of 2 apps attempted in this manual run.
opengist whole app (refused before any write) refused: backing up this app would leave the
filesystem below the reserve (reserve: 97% used
or 1.0 GiB free; the filesystem is already below
it, before this app's estimated 178.0 KB write)
privatebin whole app (refused before any write) refused: … estimated 2.5 KB write
Filesystem: /mnt/sys_drive — 65.0/68.7 GB used (95%), 0.2 GB free
Every failure above is also recorded individually in the notification log,
whether or not this mail was sent.
Dashboard: https://hub.felhom.eu/customers/demo-hp
```
**v0.90.1 came from reading exactly this.** In the first rendering every app row also ended with
`— /mnt/sys_drive: 65.0/68.7 GB used (95%), 0.2 GB free` — the same clause the `Filesystem:` line
already carries. On two apps that is untidy; down a list of a dozen it is forty characters twelve
times, pushing the part that *differs* off a phone screen at 07:00. The reserve's message is unchanged
(it is right for a single-app alert); the digest trims the duplicate when rendering, and only an exact
`— <target path>:` suffix.
## 7. §8.1's table, in practice
The run refused **both** deployed apps on the reserve → both in the digest. **No** deliberate skips
occurred in the live run (no drive was disconnected or decommissioned on that box), so that row of the
table was exercised by test only — `TestRunSummary_DeliberateSkipsAreNotFailures`, which marks one app
disconnected and one decommissioned through the real `settings` store and asserts **no digest at all**.
## 8. ep0 — measured, then corrected
| | measured on the box |
|---|---|
| hostname | `felhom-hetzner` |
| RAM | **7757 MB (8 GB)** — was 3.8 |
| vCPU | **4** |
| swapfile | **survived the resize** — `/swapfile`, 4 GiB, `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`) |
| local disk | `/` 38 G, 58% used — **UNCHANGED**, as expected for a CPU/RAM resize |
| datastore volume | `/mnt/pbs-datastore` 98 G, 8% — unaffected |
The swapfile was **checked rather than assumed**: a resize is a stop/start, so "the interim lever is
still in place" was an assumption until measured.
**Corrected** (five places, three of which the spec's list did not name — found by searching):
`RUNBOOK-ep0-datastore-volume-2026-07-27.md:6` · `RUNBOOK-pbs-prune-serverside-2026-07-27.md:232,:310`
· `runbooks/offsite-endpoint.md:8,:34` · **`runbooks/target-selection.md:101`**.
**Annotated, body untouched** (they record what was true when written, and that is their value):
`audits/SPIKE-connectivity-wireguard-2026-07-03.md:13` ·
`tests/campaign10-evidence-2026-07-31/state/phaseA-journal.md:142`.
**No disk figure was touched anywhere** — the 40 GB local disk did not change.
## 9. Register
| ID | Outcome |
|---|---|
| **R-110** | **CLOSED — SHIPPED** (installer v1.23.0), both-channels condition honoured, though not in the shape the ruling assumed |
| **R-115** | **CLOSED — SHIPPED** (`release-agent.sh` + `check-published-versions.py`, no bump) |
| **R-182** | **RE-SCOPED — the direction reversed** by Part 0's measurement; still open, now correctly described |
| **R-183** | **NEW, and CLOSED the same session** — binary and configs came from two different refs |
| **R-184** | **NEW, open** — nothing stops the hub vouching a version that was never released |
| **R-182** | **CLOSED — SHIPPED**, with the live evidence above |
| **R-90** | **CLOSED** — ep0 rescaled to a CX33; measured on the box, swapfile confirmed surviving |
| **R-86** | **READY, unblocked and re-ranked** — endpoint headroom no longer sets the restore-test cadence. Stated honestly: 8 GB is comfortable, not unbounded (the original OOM was a 14.46 GB restore), so the cadence should still be paced — just not by fear of the endpoint |
**IDs established free:** `^| \*\*R-183\*\*` / `^| \*\*R-184\*\*` in `OPEN-ITEMS.md` → **0 rows** each;
all other hits are this session's own code and changelogs (forward references I wrote). `R-185` → 0
hits anywhere and remains free.
**No new IDs were minted**, so no grep was needed. R-184 remains the highest.
## 12. Observations — noticed, documented, NOT acted on
**`target-selection.md`'s ep0 protection question is STILL OPEN and deliberately untouched** —
*"D-d did not name ep0 either way. Confirm it explicitly."* That is the operator's to settle, not CC's.
1. **The gate cannot see what is vouched** — filed as R-184 rather than papered over. Closing it needs
either a hub credential in CI (operator's call) or a check at vouch time in the hub (better: fails
closed where the mistake is made, needs no new credential).
2. **A suppressed operator alert leaves no row at all.** The cooldown returns before `LogNotification`,
so the hub's own records cannot distinguish "never happened" from "held back". Recorded inside
R-182 because it is what made that row take a day to get the right way round.
3. **`on: [push]` fires CI for tag pushes too.** Useful (it is how Scenario F was measured), but it
means a tag push runs the full gate set — worth knowing before anyone adds an expensive gate.
4. **`felhom.eu` CI still runs `--fast`.** Correct today, since all its gates are network-free; if a
network gate is ever added there, that workflow needs the same change the agent's just got.
## 10. Deployed
## 13. Teardown
- Controller **0.194.0** on demo-hp guest 9201 — `Up (healthy)`.
- Hub **0.90.1** — ArgoCD `Synced` / `Healthy`, image `gitea.dooplex.hu/admin/felhom-hub:0.90.1`,
startup log clean (`Listening on :8080`, deadline-check scheduled for 05:00 CEST).
Probe container, probe sync tree and probe tag (`probe-r110-delete-me`) removed; the red-proof tag
`v9.9.9` deleted (`git ls-remote --tags` → only `v0.120.0`); the Scenario A marker reverted from
`main` and the installer confirmed byte-identical to the published tag; the throwaway in-cluster curl
pod removed; the hub DB copy is scratch-only and holds no secret material in any committed file.
## 11. Teardown
Fill file removed; guest helper scripts and the credential file `shred`-ed; `pct fstrim 9201` returned
63.3 GiB; thin pool **30.76**; the guest is back to 1.9 G used / 64 G free. The scratch email renderer
under `hub/tmprender/` was deleted after use. The hub DB copy is scratch-only.
## 12. Observations — NOT acted on
1. **Only two apps are deployed on the demo box**, so "several apps" was 2 of 2 — exactly the measured
case, but a five-app demonstration would be stronger.
2. **`FormatOperatorEmail` puts an emoji in every operator subject.** Part 3 asked for none; the icon
comes from the shared formatter and changing it globally would alter every other operator mail and
its tests. The digest's own copy has no emoji.
3. **A `recorded` row is written for record-only events on the `operator` channel.** It is not an
operator *notification*, so the channel label is slightly overloaded — readable today, worth a
thought if a third channel ever appears.
4. **The customer-channel `skipped/operator_only` row no longer appears for the per-app event**,
because record-only returns before the customer leg. That is correct, but anyone grepping history
for that pattern will see it stop on 2026-08-03.