From b14eb0fad48065f160ba14c766d5688527447f26 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Tue, 10 Nov 2026 14:25:41 +0100 Subject: [PATCH] =?UTF-8?q?hub:=20a=20full=20host=20disk=20is=20the=20oper?= =?UTF-8?q?ator's=20alarm=20(D68,=20decision=20287)=20=E2=80=94=20commit?= =?UTF-8?q?=20only,=20not=20built?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit host_disk_warning and host_disk_critical join the operator-only register, so a household that already has the type saved gets no mail. The type leaves the claim seed list. storage_fill_* is split by storage kind: a drive the household can free (usb, local-dir, nfs, cifs) still reaches the household; the thin pool, a PBS store, an unknown kind and a missing kind go to the operator only. Tests: TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt, TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree, TestD68_RegisterAndGate, TestD68_SeedHoldsNoOperatorOnlyType. Red seen three times (register lines out; gate reading the register only; type back in the seed). No hub build, no deploy. Also: dated lines on R-1368 and R-1371 for the agent (776025c8) and controller (844f19f1) commits, and the filesystem-identity measurement the controller commit cites. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- CONTEXT.md | 9 ++ REUSE.md | 1 + documentation/architecture/08-alarm-ladder.md | 17 +++ .../evidence/extra/fs-identity.txt | 26 ++++ documentation/backlog/OPEN-ITEMS.md | 4 +- hub/CHANGELOG.md | 25 ++++ hub/internal/claim/d68_seed_test.go | 30 ++++ hub/internal/claim/engine.go | 4 +- .../d68_host_disk_operator_only_test.go | 131 ++++++++++++++++++ hub/internal/notify/dispatcher.go | 47 ++++++- 10 files changed, 290 insertions(+), 4 deletions(-) create mode 100644 documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt create mode 100644 hub/internal/claim/d68_seed_test.go create mode 100644 hub/internal/notify/d68_host_disk_operator_only_test.go diff --git a/CONTEXT.md b/CONTEXT.md index e9dc716c..9434527b 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,6 +16,15 @@ > and holds nothing of its own; this file does hold its own content, namely the standing rulings below. +> **2026-11-10 (day, operator brief) — COMMITTED, NOT RELEASED: D68 in the hub, and two sibling fixes.** (1) Hub source: +> `host_disk_warning`/`host_disk_critical` are operator-only and the critical is not seeded at claim; `storage_fill_*` +> is split by `storage_type` in `notify.operatorOnlyFor` (household: `usb`, `local-dir`, `nfs`, `cifs`; operator only: +> `lvmthin`, `pbs`, unknown, missing). **Decided by CC unattended — operator may reverse:** which kinds count as the +> household's; and moving the WARNING with the critical. No hub build, no deploy. (2) Agent `776025c8`: the self-update +> folder prunes itself after a good update. (3) Controller `844f19f1`: one fill warning per filesystem. (4) Kernel +> clean-up proven by hand on demo-hp (`audits/kernel-cleanup-spike-2026-11-10/`); the rule is DESIGN in `11` §5.11.1. +> Report: `REPORT-endings-kernels-fixes-2026-11-10.md`. +> > **2026-11-03 (day, attended) — RELEASED: controller v0.350.0 (the global floor since 08:18 CET, MinAgent 0.158.1, `sha256:9ee53908…` from `5c336d82`), agent v0.172.0, golden 0.350.0 (both VOUCHED 10:18:10 CET), hub v0.179.0 (09:05 CET), catalog `f4c5c79`.** Traefik v3.7.17 on every box (R-1354): on 9202 v3.7.16 grew 18.7 → 79.0 MiB after 920 NTLM connections, v3.7.17 stayed near 25 MiB for 25,116. Golden 0.349.0 was never vouched. The fresh install ran the public installer with one added line naming the unvouched pair, because an install refuses any golden but the vouched one (R-297). A reinstall of a customer whose controller has reported is refused until the operator deletes the host, however long one waits (R-1159). R-1302's ERROR is a BYO box's: the appliance path pre-positions the grant. Report: `REPORT-release-2026-11-03.md`; evidence `documentation/audits/release-2026-11-03/`. > > **2026-11-02/03 (night, unattended) — the security watch and the new notices; no release.** (1) **R-1334's fix held:** 0 Docker Hub 429 on five nights. A streak is per SOURCE, so seven pins that can never be read (`gitea.dooplex.hu/admin/*` ×6, `minio/minio`) held `registry:docker.io` and `runtime` open: `security_watch.py` `STATED_UNREADABLE` + private registry = GAP with a reason; `Transient` retried on every source (budget 30); `FAILED-SOURCE` lines; R-1310's rules (`PACKAGE_NOT_OURS`, `WORD_ONLY`, one entry per GHSA, date tags). (2) **Traefik v3.7.16 has GHSA-xv5x (high):** pin `c4b6844`, unreleased (R-1354). (3) **Catalog live `c760fca`:** gitea 28.3.0, uptime-kuma 2.5.9 (crosses 2.5.8's TLS removal on TCP monitors — decided by CC unattended, operator may reverse), grafana 13.2.5, actualbudget 26.11.0, n8n 2.46.4, vaultwarden 1.37.5-alpine. (4) **Commit only:** controller `c4b6844` + `3764445` (R-1347, R-1348, R-1339, R-1351, R-1323), agent `50eeccc` (R-1302), hub `2f7e836f` (R-1352). (5) **Walks:** 8 claims moved to walked (31 → 23 not walked). R-1361: an event sent while the hub is away is lost after 9 s. R-1362, R-1364 filed. Report: `REPORT-security-2026-11-02.md`; evidence `documentation/audits/security-2026-11-02/`. diff --git a/REUSE.md b/REUSE.md index 0e667a03..63f1aa51 100644 --- a/REUSE.md +++ b/REUSE.md @@ -230,6 +230,7 @@ - **Operator-only** → add to `notify.operatorOnlyEvents` (hub/internal/notify/dispatcher.go) and give it **no** `customerMessages` entry. **Allowlisting alone does NOT make a type operator-only** — `FormatCustomerEmail` treats a missing `customerMessages` entry as a *fallback to the raw message*, not a block, and the only customer gate is configuration. v0.78.0 asserted the opposite in a comment and shipped the defect (R-97c). Examples: `whole_guest_backup_failed`, `recovery_unit_capture_failed`. - **Customer-facing with a STATIC message** → add a `customerMessages` entry (hub/internal/notify/templates.go) and the controller's `settings.DefaultEnabledEvents` if it should be on by default. - **Customer-facing with a DYNAMIC message** (the producer builds Hungarian text carrying names/numbers) → deliberately **no** `customerMessages` entry: `FormatCustomerEmail` PREFERS the entry over the message, so adding one silently discards the specifics. Examples: `offbox_enlarge_blocked`, `disk_health_degraded`, and since v0.89.0 `disk_warning`/`disk_critical`. + - **Operator-only for SOME payloads** → `notify.operatorOnlyFor(eventType, detailsJSON)` (hub/internal/notify/dispatcher.go; D68, 2026-11-10) is the gate `processCustomer` reads: the register, plus `storage_fill_*` split by the event's `storage_type` (fail closed on a missing or unknown kind). A new payload-split family goes THERE, with a per-kind table test; `IsOperatorOnly` stays the type-level answer and says false for a split family. - Pin BOTH registers in ONE test (hub/internal/api/recovery_unit_event_test.go is the model) — fixing one and not the other is the realistic mistake, and `notify.IsOperatorOnly` exists so the api package can assert it. - **A type in these registers with no PRODUCER is inert.** `disk_warning`/`disk_critical` were allowlisted, copy'd, default-enabled and checkbox'd from early on, and nothing in any repo emitted them until controller v0.191.0 — grep for an emitter before assuming a type works. - **New monitor checker**: copy hub/internal/monitor/staleness.go (§2 pattern); wire in hub/cmd/hub/main.go with an `EventNotifyFunc`; severity must be warning/error/critical to notify. diff --git a/documentation/architecture/08-alarm-ladder.md b/documentation/architecture/08-alarm-ladder.md index 255a3b19..f2055161 100644 --- a/documentation/architecture/08-alarm-ladder.md +++ b/documentation/architecture/08-alarm-ladder.md @@ -403,6 +403,23 @@ a handshake older than 6 h or stamped in the future, edge-triggered (D30, R-1204 (R-1203; `05` §13); `golden_not_vouched` — a published golden the hub does not vouch (decision 199, R-242); `recovery_older_package` — a household's code opens, or may open, an older sealed package (decision 183, R-304 C). +**A full disk the household cannot free is the operator's alarm (D68, `09` §3 decision 287, 2026-11-10; in source, NOT +released).** `host_disk_warning` / `host_disk_critical` (the host's system disk, `monitor/host_disk.go`) are in +`operatorOnlyEvents`, and the critical is no longer seeded at claim. `storage_fill_*` is the one family split by its +payload (`notify.operatorOnlyFor`): the household is mailed only when the event's `storage_type` is a kind it can free. + +| `storage_type` | Who can free it | Household mailed | +|---|---|---| +| `usb`, `local-dir` | the household (its drive) | yes | +| `nfs`, `cifs` | the household (its own network storage) | yes | +| `lvmthin` | the operator (the pool all guests live on) | no | +| `pbs` | the operator (the backup server) | no | +| unknown, or missing | not known | no (fail closed) | + +The operator is mailed in every row. *Which kinds count as the household's: decided by CC unattended, operator may +reverse.* Open design point: a backup drive reports as `usb` / `local-dir` too (`16`, „disk space"). Pinned by +`TestD68_*` in `hub/internal/notify/` and `hub/internal/claim/`. + ## 6.4 A box that is not always on: the missed-backup deadline and the household's outage mail [DESIGN, hub v0.134.0, 2026-10-05] Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *operator may reverse*). diff --git a/documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt b/documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt new file mode 100644 index 00000000..8717e11b --- /dev/null +++ b/documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt @@ -0,0 +1,26 @@ +# what the controller container sees: filesystem type, id, size, free, device number — read-only, 2026-11-10T13:11:39Z +# command per guest: pct exec -- docker exec felhom-controller sh -c '' +== demo-hp:9202 +/ type=overlayfs fsid=8db558c73fd96a61 blocks=17917726 free=14503898 dev=91 +/mnt/sys_drive type=ext2/ext3 fsid=74efa8cc5db98f9a blocks=17917726 free=14503898 dev=1793 +/etc/hostname type=ext2/ext3 fsid=74efa8cc5db98f9a blocks=17917726 free=14503898 dev=1793 +/etc/hosts type=ext2/ext3 fsid=74efa8cc5db98f9a blocks=17917726 free=14503898 dev=1793 +/etc/resolv.conf type=ext2/ext3 fsid=74efa8cc5db98f9a blocks=17917726 free=14503898 dev=1793 +/opt/docker/stacks type=ext2/ext3 fsid=fba7bd0515d186de blocks=8179140 free=7710195 dev=1792 +state: { "bands": {}} +== demo-hp:9201 +/ type=overlayfs fsid=b9f88a53a84831d6 blocks=18016108 free=10674453 dev=139 +/mnt/sys_drive type=ext2/ext3 fsid=e1508f4b7cb1b4b0 blocks=18016108 free=10674452 dev=64519 +/etc/hostname type=ext2/ext3 fsid=e1508f4b7cb1b4b0 blocks=18016108 free=10674452 dev=64519 +/etc/hosts type=ext2/ext3 fsid=e1508f4b7cb1b4b0 blocks=18016108 free=10674452 dev=64519 +/etc/resolv.conf type=ext2/ext3 fsid=e1508f4b7cb1b4b0 blocks=18016108 free=10674452 dev=64519 +/opt/docker/stacks type=ext2/ext3 fsid=85fbdcb4db8310a7 blocks=8179140 free=7847637 dev=64518 +state: { "bands": {}} +== felhom-pve:9201 +/ type=overlayfs fsid=791e1a6786a9d03f blocks=64230001 free=63626008 dev=88 +/mnt/sys_drive type=ext2/ext3 fsid=6968e12ca6ac967a blocks=64230001 free=63626008 dev=64519 +/etc/hostname type=ext2/ext3 fsid=6968e12ca6ac967a blocks=64230001 free=63626008 dev=64519 +/etc/hosts type=ext2/ext3 fsid=6968e12ca6ac967a blocks=64230001 free=63626008 dev=64519 +/etc/resolv.conf type=ext2/ext3 fsid=6968e12ca6ac967a blocks=64230001 free=63626008 dev=64519 +/opt/docker/stacks type=ext2/ext3 fsid=415453d44fdc474b blocks=8179140 free=7889061 dev=64518 +state: diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 7bdbf717..c6e1953d 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -281,7 +281,7 @@ stopping line that lies. | **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC | | **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC | | **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC | -| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | +| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | | **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC | | **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. | — | Docs: a short „disk space” section in an architecture document: each part of the system disk with its limit and its cleaner, pointing at the audit table. Then the installer can refuse a disk that cannot hold the sum. | CC | | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC | @@ -326,7 +326,7 @@ stopping line that lies. | **R-1361** | Monitoring & notifications | P2 | **An event the box sends while the hub cannot be reached is lost: 3 tries over 9 s, then dropped.** Walked 2026-11-03 on Tester 1 (hub blackholed 13 min): the box ran normally, an app install worked, a crash loop stopped the app and held it, and `app_stopped_unhealthy` and `app_held` failed after 3 attempts (`notifier.go`). Afterwards NOTHING was re-sent: the hub's log shows no event from the outage, its „Held apps" had no row (`a row existed: false`), and the hold reporter had already recorded the app as reported (`holdreport.go`, the push is asynchronous) so only its periodic re-send can repair it. Any DooPlex restart (2026-11-01 was one) can swallow a household's and the operator's mails this way. `audits/security-2026-11-02/part5/outage-hub.log` | OPEN — filed 2026-11-03 **2026-11-03, a second cost seen:** the walk's cut (01:17–01:39 UTC) covered Tester 1's 01:30 dump, its `db_dump_completed` was lost, and at 04:00 UTC the hub mailed `expected_dbdump_missed` „No DB dump completed or failed since midnight", while the box's own report read `last_db_dump` 03:15 UTC. A lost event becomes a false alarm. | — | a small on-disk outbox for events, sent in order when the hub answers (a design: which types, how long kept); the hold reporter marks „reported" only after a 2xx. Test red first | CC | | **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC | | **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) | -| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC | +| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC | | **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC | ## Hub & operator — 16 rows (P2 1, P3 4, P4 11) diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 2d97a28a..4923ba65 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,3 +1,28 @@ +## Unreleased (2026-11-10) — D68: a full host disk is the operator's alarm — COMMITTED, NOT BUILT, NOT DEPLOYED (it waits for a release session) + +- **The ruling (`09` §3 decision 287, R-1371).** A household is not mailed about a disk it cannot free. +- **`host_disk_warning` and `host_disk_critical` are operator-only** (`notify.operatorOnlyEvents`). The critical was in + the claim's seed list, so a household was mailed „free up some space" about the host's system disk. A household that + already holds the type in its saved list gets no mail: the register is read before the saved list. No saved list is + rewritten; the string stays there and does nothing. +- **`host_disk_critical` is not seeded at claim any more** (`claim.defaultSeedEvents`). +- **`storage_fill_*` is split by the storage's kind** (`notify.operatorOnlyFor`, the event's `storage_type`). Checked in + source first (`monitor/storage_fill.go`, the agent's `storage.reportType`): the check judges six kinds. A household + can free four — `usb`, `local-dir`, `nfs`, `cifs` — and is still mailed for those. `lvmthin` (the pool all guests live + on) and `pbs` (the operator's backup server) go to the operator only. A kind the list does not know, or an event + with no kind, goes to the operator only (fail closed). *Which kinds a household can free: decided by CC, operator may + reverse.* +- **No switch does nothing.** The controller's notification page never offered either type (grep of + `felhom-controller/controller`, positive control `disk_critical`); the claim's seed was the only way into a list. +- **The operator's mail is unchanged**: it does not pass through this gate. +- **Tests** (`internal/notify/d68_host_disk_operator_only_test.go`, `internal/claim/d68_seed_test.go`): + `TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt`, `TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree` + (ten kinds), `TestD68_RegisterAndGate`, `TestD68_SeedHoldsNoOperatorOnlyType`. **Red seen three times:** the register + lines removed (the household is mailed both types); the gate reading the register only (six kinds mailed); the type + back in the seed. +- **Open:** a backup drive reports as `usb` or `local-dir` like a data drive, and the household cannot free a backup + drive by hand. The hub cannot tell the two apart from the event. A design point in `16-storage-physical-identity.md`. + ## v0.179.1 (2026-11-08) — a host reboot ends „the host needs a restart" at once - **The defect (demo-felhom, 2026-11-08).** The night's host pass said reboot-needed (05:31 CET); the kernel step rebooted diff --git a/hub/internal/claim/d68_seed_test.go b/hub/internal/claim/d68_seed_test.go new file mode 100644 index 00000000..9d9085bc --- /dev/null +++ b/hub/internal/claim/d68_seed_test.go @@ -0,0 +1,30 @@ +package claim + +import ( + "testing" + + "gitea.dooplex.hu/admin/felhom-hub/internal/notify" +) + +// D68 (`09` §3 decision 287, 2026-11-10): the claim must not seed a type the dispatcher will never deliver — a saved +// string that does nothing. And the two types the ruling names are pinned by name. +// +// RED-PROOF (observed 2026-11-10): put "host_disk_critical" back into defaultSeedEvents → fails naming it. +func TestD68_SeedHoldsNoOperatorOnlyType(t *testing.T) { + seeded := map[string]bool{} + for _, et := range defaultSeedEvents { + seeded[et] = true + if notify.IsOperatorOnly(et) { + t.Errorf("defaultSeedEvents holds %q, which is operator-only: the household would get a saved type that never mails", et) + } + } + if seeded["host_disk_critical"] { + t.Error("host_disk_critical must not be seeded: the host's system disk is the operator's (D68)") + } + if !seeded["storage_fill_critical"] { + t.Error("storage_fill_critical must stay seeded: a household is told when a drive it can free is full (D68)") + } + if !seeded["disk_critical"] { + t.Error("disk_critical must stay seeded: it is the household's own fill alarm") + } +} diff --git a/hub/internal/claim/engine.go b/hub/internal/claim/engine.go index 211e6c2c..682bf2a0 100644 --- a/hub/internal/claim/engine.go +++ b/hub/internal/claim/engine.go @@ -230,7 +230,9 @@ var defaultSeedEvents = []string{ "node_down", "backup_failed", "disk_critical", - "host_disk_critical", + // D68 (`09` §3 decision 287, 2026-11-10): `host_disk_critical` is NOT seeded any more — the host's system disk is + // the operator's (notify.operatorOnlyEvents). `storage_fill_critical` stays: the dispatcher lets it through only + // for a storage kind the household can free (notify.operatorOnlyFor). Pinned by TestD68_SeedHoldsNoOperatorOnlyType. "storage_fill_critical", "offbox_repo_orphaned", // v0.120.0 (`09` §3 decision 15): the household is told ONCE when an update is undone, and when diff --git a/hub/internal/notify/d68_host_disk_operator_only_test.go b/hub/internal/notify/d68_host_disk_operator_only_test.go new file mode 100644 index 00000000..3725187c --- /dev/null +++ b/hub/internal/notify/d68_host_disk_operator_only_test.go @@ -0,0 +1,131 @@ +package notify + +import ( + "strings" + "testing" +) + +// D68 (`09` §3 decision 287, 2026-11-10; R-1371) — a full HOST system disk is the operator's alarm, and a full +// STORAGE is the household's only when the household can free it. +// +// These tests run under the breaking configuration: the household HAS the type in its saved list and HAS a mail +// address. That is the state every household claimed before this commit is in — `host_disk_critical` was in the +// claim's seed list. +// +// RED-PROOFS (all three observed 2026-11-10, then restored): +// 1. remove the two `host_disk_*` lines from operatorOnlyEvents → +// TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt fails: "the household was mailed host_disk_critical". +// 2. make processCustomer read operatorOnlyEvents[eventType] again (not operatorOnlyFor) → +// TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree fails on lvmthin, pbs, lvm, the missing kind, the +// empty details and the broken details: "the household was mailed". +// 3. put "host_disk_critical" back into claim.defaultSeedEvents → +// claim.TestD68_SeedHoldsNoOperatorOnlyType fails naming it. + +func d68Mailed(rec *sentTo, addr string) bool { + for _, to := range rec.to { + if to == addr { + return true + } + } + return false +} + +func TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt(t *testing.T) { + for _, tc := range []struct{ eventType, severity string }{ + {"host_disk_critical", "critical"}, + {"host_disk_warning", "warning"}, + } { + t.Run(tc.eventType, func(t *testing.T) { + st := opOnlyStore(t) + // The breaking configuration: the claim seeded this very string before D68. + if err := st.SaveNotificationPrefs("c1", "customer@example.com", + []string{"host_disk_critical", "host_disk_warning", "backup_failed"}, 6); err != nil { + t.Fatal(err) + } + rec := &sentTo{} + d := opOnlyDispatcher(t, st, rec) + d.ProcessEvent("c1", tc.eventType, tc.severity, + "Host demo: root filesystem CRITICALLY full at 96%", `{"host_id":"demo","percent":96}`, "hub") + + if d68Mailed(rec, "customer@example.com") { + t.Fatalf("D68: the household was mailed %s — it cannot free the host's system disk", tc.eventType) + } + if tc.severity == "critical" && !d68Mailed(rec, "operator@felhom.eu") { + t.Fatalf("D68: the operator was NOT mailed %s — the gate must be household-only (sent to %v)", tc.eventType, rec.to) + } + logs, err := st.GetRecentNotifications("c1", 20) + if err != nil { + t.Fatal(err) + } + seen := false + for _, l := range logs { + if l.Channel == "customer" && l.Status == "skipped" && strings.Contains(l.ErrorMessage, "operator_only") { + seen = true + } + } + if !seen { + t.Fatalf("D68: the household skip of %s must be logged as skipped/operator_only; got %d row(s)", tc.eventType, len(logs)) + } + }) + } +} + +// One storage_fill event per storage kind. The household is mailed for a kind it can free and for no other. +func TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree(t *testing.T) { + for _, tc := range []struct { + name, details string + household bool + }{ + {"usb", `{"host_id":"h1","storage":"usb-drive","storage_type":"usb","percent":96}`, true}, + {"local-dir", `{"host_id":"h1","storage":"data1","storage_type":"local-dir","percent":96}`, true}, + {"nfs", `{"host_id":"h1","storage":"nas","storage_type":"nfs","percent":96}`, true}, + {"cifs", `{"host_id":"h1","storage":"nas","storage_type":"cifs","percent":96}`, true}, + {"lvmthin", `{"host_id":"h1","storage":"local-lvm","storage_type":"lvmthin","percent":96}`, false}, + {"pbs", `{"host_id":"h1","storage":"felhom-pbs","storage_type":"pbs","percent":96}`, false}, + {"lvm (a kind the list does not know)", `{"host_id":"h1","storage":"vg0","storage_type":"lvm","percent":96}`, false}, + {"no storage_type", `{"host_id":"h1","storage":"x","percent":96}`, false}, + {"empty details", ``, false}, + {"details that do not parse", `{"storage_type":"usb"`, false}, + } { + t.Run(tc.name, func(t *testing.T) { + // A fresh store and dispatcher per case: the cooldowns are per dispatcher. + st := opOnlyStore(t) + if err := st.SaveNotificationPrefs("c1", "customer@example.com", + []string{"storage_fill_critical", "storage_fill_warning"}, 6); err != nil { + t.Fatal(err) + } + rec := &sentTo{} + d := opOnlyDispatcher(t, st, rec) + d.ProcessEvent("c1", "storage_fill_critical", "critical", + `Host h1: storage CRITICALLY full at 96%`, tc.details, "hub") + + got := d68Mailed(rec, "customer@example.com") + if got != tc.household { + if got { + t.Fatalf("D68: the household was mailed a full storage of kind %q — only the operator can free it", tc.name) + } + t.Fatalf("D68: the household was NOT mailed a full storage of kind %q — it is the household's own to free (sent to %v)", tc.name, rec.to) + } + if !d68Mailed(rec, "operator@felhom.eu") { + t.Fatalf("D68: the operator was NOT mailed (kind %q) — the gate must be household-only (sent to %v)", tc.name, rec.to) + } + }) + } +} + +// The register and the split, by name: the type-level answer other packages read (IsOperatorOnly) and the gate. +func TestD68_RegisterAndGate(t *testing.T) { + for _, et := range []string{"host_disk_warning", "host_disk_critical"} { + if !IsOperatorOnly(et) { + t.Errorf("%s must be operator-only (D68)", et) + } + } + for _, et := range []string{"storage_fill_warning", "storage_fill_critical", "disk_critical", "disk_warning"} { + if IsOperatorOnly(et) { + t.Errorf("%s must NOT be in the type register: storage_fill is split by kind, disk_* is the household's own fill warning", et) + } + } + if operatorOnlyFor("disk_critical", "") { + t.Error("the storage-kind rule must touch storage_fill_* only — disk_critical with empty details was blocked") + } +} diff --git a/hub/internal/notify/dispatcher.go b/hub/internal/notify/dispatcher.go index b261831d..3ab2add3 100644 --- a/hub/internal/notify/dispatcher.go +++ b/hub/internal/notify/dispatcher.go @@ -744,6 +744,15 @@ var recordOnlyEvents = map[string]bool{ // is customer-grade and would be DISCARDED by a template). Turning the fallback into a gate would // change behaviour well outside this concern. var operatorOnlyEvents = map[string]bool{ + // D68 (`09` §3 decision 287, 2026-11-10; R-1371): the HOST's system disk is the operator's. A household cannot + // free it — it has no login to the host, and what fills it (old kernels, the agent's files, backup dumps on the + // root disk) is not theirs — so „free up some space" was a mail nobody could act on. The critical was in the + // claim's seed list (claim.defaultSeedEvents) until this commit. The warning was never seeded and no page offers + // it, but a saved list can hold any string: belt, not inference. + // Pinned by TestD68_HostDiskIsOperatorOnly_EvenWhenTheHouseholdSavedIt. + // `storage_fill_*` is NOT here: it is split by the storage's KIND — see operatorOnlyFor. + "host_disk_warning": true, + "host_disk_critical": true, // R-1344 (2026-10-29): the controller runs but does not answer (agent stanza) — the operator's; and (2026-10-30) the // agent's restart of it. "controller_unresponsive": true, @@ -964,6 +973,41 @@ var operatorOnlyEvents = map[string]bool{ // Read-only: the register itself stays unexported so nothing can widen it at runtime. func IsOperatorOnly(eventType string) bool { return operatorOnlyEvents[eventType] } +// householdFreeableStorage is the storage KINDS (the agent's reported vocabulary, `storage_type` in the event's +// details) on which a household can free space by itself: a drive it plugged in or was given (`usb`, `local-dir`) and +// its own network storage (`nfs`, `cifs`). D68 (`09` §3 decision 287, 2026-11-10). +// +// Everything else is the operator's: `lvmthin` is the pool every guest's disk lives on, `pbs` is the operator's +// backup server, and a kind this list does not know is treated the same way. +var householdFreeableStorage = map[string]bool{ + "usb": true, + "local-dir": true, + "nfs": true, + "cifs": true, +} + +// operatorOnlyFor is the customer gate processCustomer uses: the type register, plus the one type family that is +// split by its PAYLOAD. A `storage_fill_*` event reaches a household only when its details name a storage kind the +// household can free (householdFreeableStorage). FAIL CLOSED: no details, details that do not parse, no +// `storage_type`, or an unknown kind → operator only. The operator's own mail does not pass through this gate, so +// a closed gate never loses the alarm. +// Pinned by TestD68_StorageFillReachesAHouseholdOnlyForADriveItCanFree. +func operatorOnlyFor(eventType, detailsJSON string) bool { + if operatorOnlyEvents[eventType] { + return true + } + if eventType != "storage_fill_warning" && eventType != "storage_fill_critical" { + return false + } + var d struct { + StorageType string `json:"storage_type"` + } + if detailsJSON == "" || json.Unmarshal([]byte(detailsJSON), &d) != nil { + return true + } + return !householdFreeableStorage[d.StorageType] +} + // householdLivenessTypes are the "your server cannot be reached" family; householdLivenessQuiet is how long after // one reached the household the next is held back (R-873). var ( @@ -977,7 +1021,8 @@ func (d *Dispatcher) processCustomer(customerID, eventType, severity, message, m // R-97c: operator-tier events stop here, BEFORE prefs are consulted — the point is that no // customer configuration can opt in. Logged rather than dropped, so the skip is visible in // notification_log instead of looking like a delivery that never happened. - if operatorOnlyEvents[eventType] { + // D68: operatorOnlyFor = the register, plus `storage_fill_*` by the storage's kind. + if operatorOnlyFor(eventType, detailsJSON) { d.store.LogNotification(customerID, eventType, severity, message, "skipped", "operator_only", "customer") return }