Upload watch night data: one-second reads of the real push on three boxes (spike 2.1); R-1245 gets the measured floor; R-1214 dated check read (stays WATCHING); weekend-night report

REPORT-weekend-night-2026-10-23.md

@@ -0,0 +1,129 @@
+# REPORT — weekend night 2026-10-23: the hub's last plain tokens locked; the upload and CPU watch (spike, design, build)
+
+**Safest readings taken (unattended), stated first:**
+- **A test image is not a release.** The upload watch was proven on scratch 9202 with `felhom-controller:0.342.0-rc.1`,
+  set by hand in 9202's `/etc/felhom-controller-image` (the shape of today's security-lane rc.1). An `-rc` tag is never
+  offered to a household's update button. No floor was touched; only 9202 ran it, and 9202 is back on 0.341.0.
+- **9202's line was seeded by hand** (one 40 Mbit/s record): its off-site backup is off by design, so it can never learn a
+  line itself. The learning path is pinned by unit tests and by tonight's one-second read of the real pushes on the
+  three test boxes (read only).
+- **The hub 0.168.0 image was built** (the brief asked for it) **and not deployed.** The four new hub event types are in
+  hub 0.169.0, committed only.
+- **Per-app trigger instead of a whole-box trigger.** The brief said „the box's upload above X % of the line". Inside its
+  container the controller cannot tell LAN from internet bytes, nor see the box's total; it can see each app's exact
+  bytes, and the per-app view is what the explanation compares. So the trigger is per app (§6.2 of the spike).
+- **The first day:** apps the box can ask are judged at once on the 5 Mbit/s floor (a mismatch is evidence whatever the
+  line); apps it cannot ask are judged only after ~20 h of history. The brief's „first day: no alarm, only learning" holds
+  for those apps; for the others a mismatch on day one is still an alarm.
+
+## Five lines
+
+1. **Tokens locked:** the Cloudflare tunnel token, the Cloudflare API token and the git token are sealed in **hub 0.168.0**
+   — built, CI green, **not deployed** (you deploy it on Monday).
+2. **Backup copies:** two options — encrypt the hub's Longhorn disk, or take it out of the Longhorn backup because the
+   hub's own copy to ep0 is already encrypted at rest and in transit (measured). **My pick: take it out** (D40).
+3. **The line:** measured from the box's own nightly off-site upload (one-second samples, best 10 seconds) — a lower
+   bound; most nights the upload is too small to say anything, so a real test is your decision (R-1247).
+4. **Apps that explain their traffic:** qBittorrent and Transmission (their own byte counters, within 1–7 %), Plex (its
+   viewers and transcodes). Jellyfin needs a key the household makes; Nextcloud and Immich give no number.
+5. **The fake hacked app was caught: yes** — a flood from inside qBittorrent, hidden behind a real seed: one alarm after
+   15 minutes, one clear 15 minutes after it stopped. A CPU burner in Plex: the same. A real seed: no alarm.
+
+## The spike's numbers
+
+`documentation/audits/SPIKE-upload-watch-2026-10-23.md` (evidence beside it). Short form:
+
+- **Normal levels** (three boxes, 19–31 days of one-minute samples): no app's own upload p95 reached 6 Mbit/s; the
+  longest run at ≥ 5 Mbit/s was **3 minutes**, at ≥ 1 Mbit/s 9 minutes. CPU: the longest run at ≥ 150 % of a core was
+  **7 minutes**, at ≥ 80 % 13 minutes. DooPlex's hacked Plex: ~44 Mbit/s for hours, ~10 cores.
+- **Line speed:** at a one-minute sample the push exceeded 2 Mbit/s on 3 of 81 box-nights (the push moves a few MB);
+  the box keeps no byte count of it. Tonight's one-second read: §Night data below. No active test was run (ep0 changes
+  forbidden; a public test's terms not read).
+- **Apps:** qBittorrent counted 131.1 Mbit/s while its container sent 134.6–140.5 (×1.03–1.07); Transmission ×1.001–1.02;
+  the limit API works and persists (KiB/s rounding); Plex answers its session lists (with the token it already has when
+  claimed); Jellyfin 401 without a household-made key.
+- **A hard cap** (`tc` from a helper in the app's network namespace) works in an unprivileged guest: a flood held at
+  20 Mbit/s, the app kept answering. Not built (R-1248).
+
+## The household's line (final text)
+
+- HU: „A qBittorrent feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
+- EN: "qBittorrent uploads at most N Mbit/s, so the family internet stays fast. You can change this in qBittorrent."
+- The system settings page: „Feltöltési sebesség: legalább N Mbit/s (az éjszakai távoli mentésből mérve)" /
+  "Upload speed: at least N Mbit/s (measured from the nightly off-site backup)"; „még mérjük …" until known.
+
+## Morning list (Monday)
+
+1. Deploy hub 0.168.0 (the last plain tokens locked). Then read its start line `config tokens sealed at rest (8 …)`.
+2. D40: the hub's plain Longhorn copies — stop that backup (my pick) or encrypt the disk; and whether to delete the two
+   plain copies already there.
+3. D41: qBittorrent's default limit — 50 % of the measured line (my pick). D42: who is told — the family only about the
+   limit, the hacked-app alarms only you (my pick).
+4. Release the upload watch when you say yes (hub 0.169.0 build + deploy, then controller 0.342.0 through the normal path).
+5. For the homelab session (list only, no HM numbers): an upload-saturation alert on DooPlex's `eno1` (the line is
+   ~100 Mbit/s; the flood peaked at 905 Mbit/s).
+
+## Commits, images, CI (by commit)
+
+| Repo | Commit | What | CI |
+|---|---|---|---|
+| felhom.eu | `5277d859` | hub 0.168.0: the tokens sealed; R-1243; STATUS D40 | run 2269 **success** |
+| felhom.eu | `af5c005a` | hub 0.169.0: the four events (allowed, operator-only, mail line) | run 2271 **success** |
+| felhom.eu | `a91d0b24` | spike, design (`08` §9), live proofs, rows R-1244..R-1248, STATUS | run 2272 **success** |
+| felhom-controller | `a9c1997f` | v0.342.0 (unreleased): the upload and CPU watch | run 2270 **success** |
+| felhom-controller | `60e487d` | CONTEXT + REPORT | run 2273 **success** |
+
+Images: `felhom-hub:0.168.0` `sha256:3bfdecaae527d4762059600cdfaad83daf4e4b5b20661fc47acf128180d87cb3` (from `5277d859`,
+not deployed); `felhom-controller:0.342.0-rc.1` (from `a9c1997f`, scratch 9202 only, removed from 9202 after).
+
+**Teardown, three layers:** machine — 9202: the four throwaway apps removed through the product (containers and volumes
+gone), my test files, probes and the `alpine:3.20` helper image removed, back on 0.341.0, the seeded record removed;
+host — demo-hp: the receiver LXC 9471 destroyed with its disk; DooPlex: my private copy of the hub snapshot shredded,
+the 9202 password copy shredded; hub — none. ep0: read only (the hub-copy snapshot list). No household box touched.
+
+## Rows
+
+**Before 188, after 194; opened 6 (R-1243 … R-1248), closed 0.** R-975 stays open (WAITING-ON-OPERATOR: deploy 0.168.0).
+
+## Observations
+
+- The hub's Cloudflare/tunnel/git tokens unsealed in `config_json` — FIXED in hub 0.168.0 (R-975, not yet deployed).
+- The hub's Longhorn copies are plain and sit in the open folder — FILED: R-1243.
+- The upload watch is not released — FILED: R-1244; its limit size — FILED: R-1245; who is told — FILED: R-1246.
+- The line is only a lower bound — FILED: R-1247. An app's API can lie — FILED: R-1248.
+- `appliance_registrations.pairing_code` is plain — NOT-A-FINDING: it is shown on the box's screen and is one of two
+  factors; the other is hashed.
+- R-879's sealing (0.138.0) ran no VACUUM, so old plaintext of those columns could sit in freed blocks of the live
+  `hub.db` — NOT-A-FINDING: 0.168.0's one-time VACUUM rewrites the file on its first start.
+- The hub CHANGELOG's 0.167.0 header still said „NOT built" though it is deployed — fixed without a row (small).
+- The shared memory index went over its size ceiling twice in one day (another session's lines) and blocked the push
+  gates — fixed without a row (shortened, no link lost); it sits 14 bytes under the ceiling now.
+- The watch's very first minute after a start cannot reach apps by name (network joined seconds later) — NOT-A-FINDING:
+  that minute only sets baselines.
+- The media wiring's generated override file stays after an app's removal — NOT-A-FINDING: rewritten before every start.
+- `docker stats` text in the stored metrics has three significant digits — NOT-A-FINDING for the watch (it reads the
+  Engine API); the charts are display only.
+- `felhom-samba` runs on the host network and has no counter — NOT-A-FINDING: infrastructure, not judged.
+- Transmission's rate under its limit was not observed (no peer contact) — NOT-A-FINDING: the limit API was read back;
+  qBittorrent's was observed.
+- A dated check came due on the push (R-1214, Tester 1's night copy): read read-only — the guest's memory „full" stall
+  was 444 µs in total over 11 h, 0 OOM kills; the copy was small (0.24 GB) and the capped container is not shown — the
+  row stays WATCHING, the due line removed.
+- The night's line floor is low (10 Mbit/s on demo-hp, one second at 70) — FILED in R-1245's text (D41), no new row.
+
+## Instruction files, poster
+
+No instruction file edited. System poster: no fact on it changed (nothing deployed) — none of the three applied.
+
+## Night data
+
+Read 02:08–02:30Z, once a second, on the three test boxes (read only; spike §2.1):
+
+- **demo-hp: counts.** Best 10 seconds 12.6 MB = **10.0 Mbit/s**. One second reached 70 Mbit/s.
+- **demo-felhom: too small** (best 10 seconds 3.2 MB = 2.5 Mbit/s).
+- **Tester 1: too small** (3.5 MB = 2.7 Mbit/s).
+
+**What it means:** the nightly push is a burst of a few seconds. So the line it gives is a low floor. On demo-hp,
+50 % of that floor is about 5 Mbit/s for qBittorrent — probably much less than the real line. The floor rises only when
+a bigger night comes (the box keeps the best of 14 nights). Added to the D41 decision row (R-1245): if you want a
+looser limit before a real line test exists (R-1247), pick a higher share or „none".

documentation/audits/SPIKE-upload-watch-2026-10-23.md

@@ -53,7 +53,21 @@ it is a **lower bound** (restic over SFTP to the Storage Box may not fill the li
 
 ### 2.1 One-second samples of the real push
 
-*(filled after 02:30Z — see the section at the end, „Night data")*
+Read 2026-10-24 02:08–02:30Z, once a second, the controller container's own transmit counter (the counter the watch
+reads: `eth0` in the controller's network namespace), on the three test boxes, read only
+(`SPIKE-upload-watch-2026-10-23/night-push-1s-<box>.txt`; the reader `tools/nightsample.sh`).
+
+| Box | Sent in the 22 min | Best 10 s | 1-second peak | Counts (≥ 10 MiB in the best 10 s)? |
+|---|---|---|---|---|
+| demo-hp | 19.3 MB | **12.6 MB in 10.1 s = 10.0 Mbit/s** (02:17:32Z) | 70.3 Mbit/s | **yes** → line „at least 10 Mbit/s" |
+| demo-felhom | 3.3 MB | 3.2 MB = 2.5 Mbit/s (02:15:22Z) | 13.7 Mbit/s | no |
+| Tester 1 | 3.6 MB | 3.5 MB = 2.7 Mbit/s (02:15:40Z) | 14.7 Mbit/s | no |
+
+**What it says:** one night in three counted. The push is a burst of a few seconds (3–9 seconds above 100 kB/s), so
+even the counting night gives a low floor: 10 Mbit/s, while one second of the same push reached 70 Mbit/s. The design
+keeps the best counting night of the last 14, so the line rises only when a bigger night comes. **Consequence for the
+qBittorrent limit (R-1245):** 50 % of tonight's floor is ~5 Mbit/s on demo-hp — likely far below its real line. This is
+the known lower-bound limit (R-1247), now measured; it is written into R-1245 so the operator decides with it.
 
 ## 3. The line with a test — NOT run tonight
 

documentation/backlog/OPEN-ITEMS.md

@@ -254,7 +254,7 @@ stopping line that lies.
 | **R-979** | Box system & updates | P3 | **Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only.** Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN — design-proposal | — | Design note, then build | CC |
 | **R-974** | Box system & updates | P3 | **A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root.** Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Fix the wrapper with a test on a throwaway box | CC |
 | **R-972** | Box system & updates | P2 | **The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub.** Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: `audits/security-review-2026-10-12/scanners/`. | **NARROWED** 2026-10-16 (burndown 2): **controller v0.322.0 released** on Go 1.27.2 + x/crypto v0.58.0, runtime debian:12.13-slim + `apt-get upgrade` (Debian 13 deliberately not: restic 0.18, R-1105) — scanner: critical 16 → 4, fixable criticals and highs in the OS 0; govulncheck 62 → 1 (module-level, `openpgp` not linked). **Hub v0.156.0 deployed** 2026-10-16 19:26 on the operator's yes (Go 1.27.2, alpine 3.24.2: 0 critical, 0 high); Synced/Healthy, reports from all three boxes after the start, the floor reads 0.322.0. **Agent: Go 1.27.2 committed (`37b99d7`), tested, NOT released** — the next vouched agent release carries it. Open: the Samba image (48 fixable high, not rebuilt); docker-compose plugin's own Go (4 high, upstream). Evidence `audits/burndown2-2026-10-16/P4/` **2026-10-16 (evening): the Samba part is DONE in controller v0.323.0** — felhom-samba 1.2.0 on Alpine 3.24.2 (0 critical / 0 high, was 48 fixable high), proven on demo-hp with the same household password (smbclient, nmbd, mDNS, WS-Discovery); on the three test boxes now, on every box with the global floor (R-1105's night check). The controller's Debian 13 OS part: 1 critical / 70 high, none fixable. Left: docker-compose plugin's own Go (4 high, upstream); the agent release; restic's binary is R-1107. **2026-10-17 (evening): agent v0.159.1 (Go 1.27.2) RELEASED and running on demo-hp, Tester 1 and demo-felhom** (signed jobs on the operator's yes); waiting for the operator's vouch — then only the upstream compose plugin is left (close then, recorded). | the next vouched agent release | Close when the global floor carries v0.323.0 and the next vouched agent release carries Go 1.27.2 (compose plugin: upstream, recorded) | CC |
-| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC |
+| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **2026-10-24 02:4xZ dated check read (weekend-night session, read only):** Tester 1 (8 GB host, guest 4 GB, controller 0.341.0) ran its 03:30 copy at 01:30:00–01:30:13Z (3 apps, 157 MB + 86 MB + 16 KB; no „capped copy container did not start" WARN, no leftover `felhom-tier2-copy-*`); the guest cgroup's memory „full" stall counter on the host (`/sys/fs/cgroup/lxc/9201/memory.pressure`) read **total 444 µs** over the whole 11.1 h since the host started (15:32Z), so „full" stayed far under 20 % (20 % of 10 s = 2 s); 0 OOM kills. **Not shown:** that the copy ran in the capped container (the INFO log does not name it; Docker keeps only ~200 events, from 02:36Z). **Weak load:** 0.24 GB, not the soak's large copy — so the row stays WATCHING; close after a night whose copy moves ≥ 1 GB, read from the same counter. WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC |
 | **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
 | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
 | **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement | — | — | CC |
@@ -285,7 +285,7 @@ stopping line that lies.
 | **R-1211** | Monitoring & notifications | P3 | **The household's `backup_failed` mail says only „A biztonsági mentés sikertelen! Kérjük, ellenőrizd a rendszert." — no app named, no next step, and the formal „Kérjük".** Rehearsal 2026-10-20 (the one alarm the household got, `audits/rehearsal-2026-10-20/B9-words/words.md` W25); the box's own page now says which app and that the box retries the next night (controller `5776d54`, W25), but the mail is the hub's `customerMessages` entry. Found by the night helper 2026-10-21. | OPEN | — | Hub: build the household mail from the event's app names and the classifier's class, in the te-form, with „a doboz a következő éjjel újra próbálkozik"; mail golden | CC |
 | **R-1222** | Monitoring & notifications | P3 | **A household that sets an EVENING backup time gets a false „database dump missed” error every morning.** REAL in code (night 2026-10-22; split from R-1189 (e)): the hub's 05:00 check wants a dump since Budapest midnight (`hub/internal/monitor/deadline.go:416-419, 555-565`), so an evening dump (the banner suggests evening times) is always „yesterday”; on 2026-10-25 the check also runs at 04:00 (R-1183). Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN | — | Fix: the check wants a dump within the last 26 h (or since the box's own scheduled time); a hub test with an evening schedule; rides a hub deploy | CC |
 | **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC |
-| **R-1245** | Monitoring & notifications | P3 | **DECIDE: how large is qBittorrent's default upload limit?** The box sets it from the measured line and never over the household's own limit. CC's pick: **50 % of the measured upload** (the family keeps half the line; a seeding qBittorrent measured 131 Mbit/s on the bench — the whole line). Other: 30 % (gentler on the family, slower seeding) or none (only the alarm). `audits/SPIKE-upload-watch-2026-10-23.md` §4, §6.6. | WAITING-ON-OPERATOR | — | Operator: say a percentage (or none) | operator |
+| **R-1245** | Monitoring & notifications | P3 | **DECIDE: how large is qBittorrent's default upload limit?** The box sets it from the measured line and never over the household's own limit. CC's pick: **50 % of the measured upload** (the family keeps half the line; a seeding qBittorrent measured 131 Mbit/s on the bench — the whole line). Other: 30 % (gentler on the family, slower seeding) or none (only the alarm). `audits/SPIKE-upload-watch-2026-10-23.md` §4, §6.6. **Measured 2026-10-24 night (spike §2.1):** of three boxes one push counted (demo-hp: at least 10 Mbit/s from its best 10 s; one second reached 70) — so 50 % of the lower bound would hold qBittorrent at ~5 Mbit/s there until a bigger night comes (best of 14). | WAITING-ON-OPERATOR | — | Operator: say a percentage (or none) | operator |
 | **R-1246** | Monitoring & notifications | P3 | **DECIDE: who is told when the box limits or flags something?** CC's pick: the family only about the qBittorrent limit (one line on its page; they can change it); the unexplained-upload and unexplained-CPU alarms reach the operator only (R-388: the household can do nothing about a hacked app but wait for support). Other: tell the family too (that support was told). `audits/SPIKE-upload-watch-2026-10-23.md` §6.4. | WAITING-ON-OPERATOR | — | Operator: pick | operator |
 | **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
 | **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
@@ -375,7 +375,6 @@ stopping line that lies.
 |---|---|---|
 | R-435 | 2026-10-27 | Tester 1's weekly off-site clean-up ran inside a hub window with no drop alarm (box log + mailbox); detail in the R-435 row |
 | R-1183 | 2026-10-26 | every hub daily job ran ONCE on 2026-10-25 (hub log + one snapshot file per slot — four slots since hub 0.167.0, D34, the 02:00 one not twice); detail in the R-1183 row |
-| R-1214 | 2026-10-24 | Tester 1's first night copy on 0.340.0 ran in the capped container, memory „full” pressure under 20 %; detail in the R-1214 row |
 | R-1228 | 2026-10-30 | D35 (yes, after the release week): start moving the website + installer to Cloudflare; detail in the R-1228 row |
 | R-1174 | 2026-10-27 | the status gate refuses from today: one conviction seen exiting 1 (a hand run on a copy of the page is enough); detail in the R-1174 row |
 | R-1050 | 2026-11-15 | HSTS one year if no https trouble since 2026-10-15; detail in the R-1050 row |