Upload watch night data: one-second reads of the real push on three boxes (spike 2.1); R-1245 gets the measured floor; R-1214 dated check read (stays WATCHING); weekend-night report

documentation/audits/SPIKE-upload-watch-2026-10-23.md

@@ -53,7 +53,21 @@ it is a **lower bound** (restic over SFTP to the Storage Box may not fill the li
 
 ### 2.1 One-second samples of the real push
 
-*(filled after 02:30Z — see the section at the end, „Night data")*
+Read 2026-10-24 02:08–02:30Z, once a second, the controller container's own transmit counter (the counter the watch
+reads: `eth0` in the controller's network namespace), on the three test boxes, read only
+(`SPIKE-upload-watch-2026-10-23/night-push-1s-<box>.txt`; the reader `tools/nightsample.sh`).
+
+| Box | Sent in the 22 min | Best 10 s | 1-second peak | Counts (≥ 10 MiB in the best 10 s)? |
+|---|---|---|---|---|
+| demo-hp | 19.3 MB | **12.6 MB in 10.1 s = 10.0 Mbit/s** (02:17:32Z) | 70.3 Mbit/s | **yes** → line „at least 10 Mbit/s" |
+| demo-felhom | 3.3 MB | 3.2 MB = 2.5 Mbit/s (02:15:22Z) | 13.7 Mbit/s | no |
+| Tester 1 | 3.6 MB | 3.5 MB = 2.7 Mbit/s (02:15:40Z) | 14.7 Mbit/s | no |
+
+**What it says:** one night in three counted. The push is a burst of a few seconds (3–9 seconds above 100 kB/s), so
+even the counting night gives a low floor: 10 Mbit/s, while one second of the same push reached 70 Mbit/s. The design
+keeps the best counting night of the last 14, so the line rises only when a bigger night comes. **Consequence for the
+qBittorrent limit (R-1245):** 50 % of tonight's floor is ~5 Mbit/s on demo-hp — likely far below its real line. This is
+the known lower-bound limit (R-1247), now measured; it is written into R-1245 so the operator decides with it.
 
 ## 3. The line with a test — NOT run tonight
 

documentation/backlog/OPEN-ITEMS.md

@@ -254,7 +254,7 @@ stopping line that lies.
 | **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC |
 | **R-974** | Box system & updates | P3 | **A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root.** Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix the wrapper with a test on a throwaway box | CC |
 | **R-972** | Box system & updates | P2 | **The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub.** Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: `audits/security-review-2026-10-12/scanners/`. | **NARROWED** 2026-10-16 (burndown 2): **controller v0.322.0 released** on Go 1.27.2 + x/crypto v0.58.0, runtime debian:12.13-slim + `apt-get upgrade` (Debian 13 deliberately not: restic 0.18, R-1105) — scanner: critical 16 → 4, fixable criticals and highs in the OS 0; govulncheck 62 → 1 (module-level, `openpgp` not linked). **Hub v0.156.0 deployed** 2026-10-16 19:26 on the operator's yes (Go 1.27.2, alpine 3.24.2: 0 critical, 0 high); Synced/Healthy, reports from all three boxes after the start, the floor reads 0.322.0. **Agent: Go 1.27.2 committed (`37b99d7`), tested, NOT released** — the next vouched agent release carries it. Open: the Samba image (48 fixable high, not rebuilt); docker-compose plugin's own Go (4 high, upstream). Evidence `audits/burndown2-2026-10-16/P4/` **2026-10-16 (evening): the Samba part is DONE in controller v0.323.0** — felhom-samba 1.2.0 on Alpine 3.24.2 (0 critical / 0 high, was 48 fixable high), proven on demo-hp with the same household password (smbclient, nmbd, mDNS, WS-Discovery); on the three test boxes now, on every box with the global floor (R-1105's night check). The controller's Debian 13 OS part: 1 critical / 70 high, none fixable. Left: docker-compose plugin's own Go (4 high, upstream); the agent release; restic's binary is R-1107. **2026-10-17 (evening): agent v0.159.1 (Go 1.27.2) RELEASED and running on demo-hp, Tester 1 and demo-felhom** (signed jobs on the operator's yes); waiting for the operator's vouch — then only the upstream compose plugin is left (close then, recorded). | the next vouched agent release | Close when the global floor carries v0.323.0 and the next vouched agent release carries Go 1.27.2 (compose plugin: upstream, recorded) | CC |
-| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC |
+| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **2026-10-24 02:4xZ dated check read (weekend-night session, read only):** Tester 1 (8 GB host, guest 4 GB, controller 0.341.0) ran its 03:30 copy at 01:30:00–01:30:13Z (3 apps, 157 MB + 86 MB + 16 KB; no „capped copy container did not start" WARN, no leftover `felhom-tier2-copy-*`); the guest cgroup's memory „full" stall counter on the host (`/sys/fs/cgroup/lxc/9201/memory.pressure`) read **total 444 µs** over the whole 11.1 h since the host started (15:32Z), so „full" stayed far under 20 % (20 % of 10 s = 2 s); 0 OOM kills. **Not shown:** that the copy ran in the capped container (the INFO log does not name it; Docker keeps only ~200 events, from 02:36Z). **Weak load:** 0.24 GB, not the soak's large copy — so the row stays WATCHING; close after a night whose copy moves ≥ 1 GB, read from the same counter. WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC |
 | **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
 | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
 | **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement | — | — | CC |
@@ -285,7 +285,7 @@ stopping line that lies.
 | **R-1211** | Monitoring & notifications | P3 | **The household's `backup_failed` mail says only „A biztonsági mentés sikertelen! Kérjük, ellenőrizd a rendszert." — no app named, no next step, and the formal „Kérjük".** Rehearsal 2026-10-20 (the one alarm the household got, `audits/rehearsal-2026-10-20/B9-words/words.md` W25); the box's own page now says which app and that the box retries the next night (controller `5776d54`, W25), but the mail is the hub's `customerMessages` entry. Found by the night helper 2026-10-21. | OPEN | — | Hub: build the household mail from the event's app names and the classifier's class, in the te-form, with „a doboz a következő éjjel újra próbálkozik"; mail golden | CC |
 | **R-1222** | Monitoring & notifications | P3 | **A household that sets an EVENING backup time gets a false „database dump missed” error every morning.** REAL in code (night 2026-10-22; split from R-1189 (e)): the hub's 05:00 check wants a dump since Budapest midnight (`hub/internal/monitor/deadline.go:416-419, 555-565`), so an evening dump (the banner suggests evening times) is always „yesterday”; on 2026-10-25 the check also runs at 04:00 (R-1183). Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN | — | Fix: the check wants a dump within the last 26 h (or since the box's own scheduled time); a hub test with an evening schedule; rides a hub deploy | CC |
 | **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC |
-| **R-1245** | Monitoring & notifications | P3 | **DECIDE: how large is qBittorrent's default upload limit?** The box sets it from the measured line and never over the household's own limit. CC's pick: **50 % of the measured upload** (the family keeps half the line; a seeding qBittorrent measured 131 Mbit/s on the bench — the whole line). Other: 30 % (gentler on the family, slower seeding) or none (only the alarm). `audits/SPIKE-upload-watch-2026-10-23.md` §4, §6.6. | WAITING-ON-OPERATOR | — | Operator: say a percentage (or none) | operator |
+| **R-1245** | Monitoring & notifications | P3 | **DECIDE: how large is qBittorrent's default upload limit?** The box sets it from the measured line and never over the household's own limit. CC's pick: **50 % of the measured upload** (the family keeps half the line; a seeding qBittorrent measured 131 Mbit/s on the bench — the whole line). Other: 30 % (gentler on the family, slower seeding) or none (only the alarm). `audits/SPIKE-upload-watch-2026-10-23.md` §4, §6.6. **Measured 2026-10-24 night (spike §2.1):** of three boxes one push counted (demo-hp: at least 10 Mbit/s from its best 10 s; one second reached 70) — so 50 % of the lower bound would hold qBittorrent at ~5 Mbit/s there until a bigger night comes (best of 14). | WAITING-ON-OPERATOR | — | Operator: say a percentage (or none) | operator |
 | **R-1246** | Monitoring & notifications | P3 | **DECIDE: who is told when the box limits or flags something?** CC's pick: the family only about the qBittorrent limit (one line on its page; they can change it); the unexplained-upload and unexplained-CPU alarms reach the operator only (R-388: the household can do nothing about a hacked app but wait for support). Other: tell the family too (that support was told). `audits/SPIKE-upload-watch-2026-10-23.md` §6.4. | WAITING-ON-OPERATOR | — | Operator: pick | operator |
 | **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
 | **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
@@ -375,7 +375,6 @@ stopping line that lies.
 |---|---|---|
 | R-435 | 2026-10-27 | Tester 1's weekly off-site clean-up ran inside a hub window with no drop alarm (box log + mailbox); detail in the R-435 row |
 | R-1183 | 2026-10-26 | every hub daily job ran ONCE on 2026-10-25 (hub log + one snapshot file per slot — four slots since hub 0.167.0, D34, the 02:00 one not twice); detail in the R-1183 row |
-| R-1214 | 2026-10-24 | Tester 1's first night copy on 0.340.0 ran in the capped container, memory „full” pressure under 20 %; detail in the R-1214 row |
 | R-1228 | 2026-10-30 | D35 (yes, after the release week): start moving the website + installer to Cloudflare; detail in the R-1228 row |
 | R-1174 | 2026-10-27 | the status gate refuses from today: one conviction seen exiting 1 (a hand run on a copy of the page is enough); detail in the R-1174 row |
 | R-1050 | 2026-11-15 | HSTS one year if no https trouble since 2026-10-15; detail in the R-1050 row |