felhom.eu/documentation/audits/SPIKE-upload-watch-2026-10-23.md

# SPIKE — who is using the upload line, and does the app explain it? (2026-10-23 night)

Operator ruling 2026-10-23 19:22 (`CONTEXT.md`): measure the line; on a sustained high upload attribute it to an app
and ask the app; a mismatch is the alarm; the same shape for CPU; qBittorrent gets a default upload limit from the
measured line, household changes respected. Brief Part 2.1. Design written from these numbers: §6 here and
`architecture/08-alarm-ladder.md` §9 [DESIGN — proposed]. Evidence and tools: `SPIKE-upload-watch-2026-10-23/`.

Read first: `08-alarm-ladder.md` (R-388, §8.1), `07-backup-architecture.md` (the night chain: W = 02:30 local, the
off-site push after it, the proof at 03:30 UTC), `15-media-library.md`, `13-threat-model.md` (attacker j),
`audits/plex2shell-felhom-impact-2026-10-23/README.md` (the 1.5 TB Plex upload). Controller: `internal/metrics/`
(`container_metrics` — one row per container per minute, 30 days), `internal/monitor/healthcheck.go` (the instant
whole-box CPU warning), `internal/mediawire/` (the app clients and their credentials, `09` §3 decision 248).

Machines: the three test boxes READ ONLY (their stored metrics; their off-site record; tonight a one-second read of
the controller's own counter during the real push); scratch 9202 on demo-hp (throwaway qBittorrent, Transmission, Plex,
Jellyfin through the product's deploy); a throwaway receiver LXC **9471 `cc-uw-recv`** on demo-hp's scratch storage
(never the thin pool). Every flood went 9202 → 9471 on demo-hp's own bridge; nothing went toward the internet.

## 1. Normal levels (stored metrics, three boxes)

History: **demo-hp** 2026-09-23 02:00 → 10-23 18:09 UTC (30.7 d, 998 216 rows); **demo-felhom** (the N100) same span
(30.7 d, 268 120 rows); **Tester 1** 2026-10-04 19:41 → 10-23 18:08 (18.9 d, 294 433 rows). Per app: upload from the
`net_tx_bytes` deltas (one-minute samples), CPU in percent of ONE core. Full tables: `s1-levels-per-app.txt`.

| box | the busiest uploaders (p50 / p95 / max Mbit/s) | the busiest CPU (p95 / max % of a core) |
|---|---|---|
| demo-hp | qbittorrent 0.02 / 1.2 / 4.0; immich-server 0.03 / 0.13 / 243 (one minute); felhom-controller 0 / 0.003 / 17.8 (the push) | grimmory 272 / 305 (a scan, 94 samples); immich-ML 32 / 548; romm 3 / 442; paperless 0.3 / 404 |
| demo-felhom | immich-server 0.67 / 5.95 / 12.9 (32 samples); jellyfin 0 / 0.85 / 2.9 | immich-ML 295 / 296 (an import); onlyoffice 131 / 148; paperless 137 / 140 |
| Tester 1 | qbittorrent 0.007 / 0.13 / 42.4; transmission 0.001 / 0.013 / 57.4; controller 0 / 0.003 / 37.5 | paperless 0.7 / 508; controller 46 / 423; plex 4.3 / 198 |

**How long a normal high lasts** (`s1-longest-runs.txt` — the longest run of consecutive minutes): upload ≥ 5 Mbit/s:
**3 min** at most (qBittorrent, Tester 1); ≥ 1 Mbit/s: 9 min. CPU ≥ 150 %: **7 min** at most (Immich machine learning);
≥ 80 %: 13 min (Kimai). Busiest upload hours (UTC): demo-hp 03h, 02h; demo-felhom 16h, 05h; Tester 1 07h, 18h — no fixed
pattern beyond the night push. For scale: DooPlex's hacked Plex sent ~474 GB in one day (≈ 44 Mbit/s on average, for
hours) and used ~10 cores.

`felhom-samba` runs with the host network: its counter is always 0 (no own network namespace) — the watch cannot see it.

## 2. The line without a test

**What the box already sees:** the off-site push (restic, inside the controller container) is the box's only big
upload of its own. At the stored one-minute sample (`s2-controller-push-per-night.txt`): the controller's best five
minutes per night exceeded 2 Mbit/s on **2 of 31 nights** on demo-hp (5.1 and 6.9 Mbit/s; best single minute 17.0),
**1 of 19** on Tester 1 (7.6; best minute 37.5) and **0 of 31** on demo-felhom. The off-site record keeps the push's
duration (33 s on demo-felhom, 1 m 8 s on Tester 1, 3 m 19 s on demo-hp on 2026-10-23) and **no byte count**.

**Conclusion:** at a one-minute sample the push shows the night's DATA SIZE (a few MB, incremental), not the line. Not
enough as it stands. Two changes make it usable: sample once a SECOND during the push (the controller knows when its
push runs and can read its own counter), and let a night count only when its best 10 seconds moved ≥ 10 MB. Even then
it is a **lower bound** (restic over SFTP to the Storage Box may not fill the line; the family uploads too).

**Tonight's one-second read of the real push** (2026-10-24 ~02:16Z, three boxes, read only): §2.1 below.

### 2.1 One-second samples of the real push

Read 2026-10-24 02:08–02:30Z, once a second, the controller container's own transmit counter (the counter the watch
reads: `eth0` in the controller's network namespace), on the three test boxes, read only
(`SPIKE-upload-watch-2026-10-23/night-push-1s-<box>.txt`; the reader `tools/nightsample.sh`).

| Box | Sent in the 22 min | Best 10 s | 1-second peak | Counts (≥ 10 MiB in the best 10 s)? |
|---|---|---|---|---|
| demo-hp | 19.3 MB | **12.6 MB in 10.1 s = 10.0 Mbit/s** (02:17:32Z) | 70.3 Mbit/s | **yes** → line „at least 10 Mbit/s" |
| demo-felhom | 3.3 MB | 3.2 MB = 2.5 Mbit/s (02:15:22Z) | 13.7 Mbit/s | no |
| Tester 1 | 3.6 MB | 3.5 MB = 2.7 Mbit/s (02:15:40Z) | 14.7 Mbit/s | no |

**What it says:** one night in three counted. The push is a burst of a few seconds (3–9 seconds above 100 kB/s), so
even the counting night gives a low floor: 10 Mbit/s, while one second of the same push reached 70 Mbit/s. The design
keeps the best counting night of the last 14, so the line rises only when a bigger night comes. **Consequence for the
qBittorrent limit (R-1245):** 50 % of tonight's floor is ~5 Mbit/s on demo-hp — likely far below its real line. This is
the known lower-bound limit (R-1247), now measured; it is written into R-1245 so the operator decides with it.

## 3. The line with a test — NOT run tonight

| Candidate | Ours? | Terms | Cost per test | Tonight |
|---|---|---|---|---|
| ep0 (a receiver on our Hetzner box) | yes | ours | 10 s × line ≈ 125 MB at 100 Mbit/s | **not built** — a change on ep0 is forbidden tonight |
| the box's own Storage Box sub-account, through the same transport as the push | yes (Hetzner, unlimited traffic) | Hetzner's — ours | same | not tried — a test file in the account that holds the restic repository needs its own design (the repo check, the quota) |
| Cloudflare R2 (where the ISO lives) | yes | ours | same + R2 class-A writes | not used — its write key is the operator's ISO-publish key; a test object in a public bucket is wrong |
| public speed tests (speed.cloudflare.com, LibreSpeed servers) | no | **not read** | — | not used (the brief: no third-party test without its terms) |

A quiet hour: never 02:30–05:30 local (dumps 02:30, second drive, off-site push, the 03:30 UTC proof). The family shares
the line, so take the highest of several runs. **Not needed for the first build** — §6 works with the push's lower
bound and an absolute floor; an active test is a decide row (R-1247).

## 4. Can the box ask each app? (measured on 9202 with throwaway installs; a private torrent 9202 → 9471)

Set-up: qBittorrent 5.2.4 and Transmission 4.1.3 installed through the product (`POST /api/stacks/<n>/deploy`; the
first try was refused BEFORE anything was made — the data place must be the drive itself; two empty helper folders
removed). The same 2 GiB file (an AES-CTR keystream of a fixed key) on both sides; a private torrent; a 40-line HTTP
tracker on 9471 (`tools/tracker.py`); `transmission-daemon` on 9471 as the leecher. The admin passwords were read from
the containers at use and never printed.

| App | Its own number | Measured | Limit | Household's limit visible? |
|---|---|---|---|---|
| **qBittorrent** | `/api/v2/transfer/info` → `up_info_data` (cumulative), `up_info_speed` | seeding: app **131.1 Mbit/s** vs container **134.6–140.5** → **×1.03–1.07** (`s4-qb-seed-nolimit.jsonl`) | `transfer/setUploadLimit` (bytes/s) = `preferences.up_limit`; stored in KiB/s (2 500 000 → **2 499 584**); survives a restart (`Session\GlobalUPSpeedLimit=2441`); measured: ~20 Mbit/s held (`s4-qb-seed-limit20.jsonl`) | yes: `transfer/uploadLimit` ≠ the value the box stored |
| **Transmission** | RPC `session-stats.current-stats.uploadedBytes`, `uploadSpeed` | seeding: app 487.1 / 390.3 / 115.8 vs container 487.6 / 390.7 / 117.9 Mbit/s → **×1.001–1.02** (`s4-tr-seed-nolimit.jsonl`) | `session-set speed-limit-up` (kB/s) + `-enabled` (read back); **its rate under the limit was NOT observed** — the re-added leecher got no connection in 2 minutes | yes: `speed-limit-up-enabled` + value |
| **Plex** | `/status/sessions`, `/transcode/sessions` (`MediaContainer.size`) | unclaimed 1.43.5: both 200 without a token from the docker network, `{"size":0}`; a claimed server needs its PlexOnlineToken (read at use from Preferences.xml, as mediawire does — no new secret) | — | — |
| **Jellyfin** | `/Sessions` | **401** without a key; the key named `felhom` exists only when the household made it (mediawire's „one household step") — the box cannot create one at install without the admin login it does not have | — | — |
| Nextcloud, Immich | **no upload-explaining number found** (not measured further); Immich's job queue needs an API key the box does not hold | — | — | → the plain rule (§6) |

**The fake hacked app** (code inside qBittorrent's own container, `docker exec … python3` → 9471:5001, Plex2Shell's
shape): the container sent **~10 100 Mbit/s** (LAN, one bridge) while qBittorrent reported **0** (`s4-qb-flood.jsonl`).

**The counter itself:** the Docker Engine API's one-shot stats (`networks.*.tx_bytes`) equal the kernel's counter in
the container's network namespace to the byte; `docker stats`' text (what `internal/metrics` stores) has three
significant digits — ~10 MB steps at a GB total, ~1.3 Mbit/s at a one-minute sample. The watch reads the Engine API.

**A limit the hacked app controls:** an attacker with code in the app's container can also answer its API. The check
catches an attacker who does not know about it; one who does could lie. §5 is the answer to that.

## 5. A universal upload ceiling — spike only

A helper container sharing the app's network namespace (`docker run --rm --network container:qbittorrent --cap-add
NET_ADMIN alpine:3.20` + `iproute2-tc`) added `tc qdisc … root tbf rate 20mbit burst 64kb latency 400ms` on the app's
`eth0` **inside the unprivileged 9202 guest**. The same flood then ran at **19.9–20.1 Mbit/s** (`s4-qb-flood-tbf.jsonl`),
qBittorrent's API kept answering, and `tc qdisc del` restored it. Costs: a helper image with `tc` (a new external image,
or `tc` shipped in the controller image), `NET_ADMIN` for that helper, re-applied after every app restart (the qdisc
dies with the namespace), upload only. **Not built** (brief). It would cap a hacked app's flood whatever its API says.

## 6. The design (numbers from §1–§5; full text `08-alarm-ladder.md` §9)

1. **Line:** one-second samples of the controller's own sent bytes during the off-site push; the best ≥ 10-second rate
   per night; a night counts when that window moved ≥ 10 MB; the line = the highest counting night of the last 14;
   settings.json `upwatch.line`; shown on the system settings page as „legalább N Mbit/s" („még mérjük" until known).
2. **Trigger:** an app (its containers summed, exact counters, once a minute) sends ≥ **max(5 Mbit/s, 25 % of the line)**
   for **15 minutes** — five times the longest normal run at ≥ 5 Mbit/s (3 min). Per app, not per box: the box cannot
   tell LAN from internet bytes inside its controller container, and the per-app view is what the check compares.
3. **Attribution:** the app's compose project; infrastructure (controller, traefik, cloudflared, Samba) never judged.
4. **Explanation:** qBittorrent / Transmission: explained when container bytes ≤ 1.25 × the app's own count + 1 Mbit/s
   (measured overhead 1–7 %); Plex: any viewer explains the upload (session bandwidth not measured — the conservative
   half; no viewer + a big upload = the Plex2Shell signature). Not explained → **`app_unexplained_upload`**,
   operator-only, one per episode, a clear after 15 quiet minutes. No explainer: alarm only at ≥ 10 × its own p95 for
   60 minutes, after ~20 h of history, worded „no app number to compare".
5. **CPU:** ≥ **max(150 %, 50 % × cores)** of one core for **30 minutes** (four times the longest normal run at ≥ 150 %),
   explained by a Plex transcode or the box's own work on the app (backup, update, deploy) → else
   **`app_unexplained_cpu`**. No explainer: ≥ max(200 %, 75 % × cores) for 60 minutes.
6. **qBittorrent limit:** 50 % of the line (CC's pick — R-1245), every 15 minutes, re-set when the line moves > 20 %;
   never over a household's limit (a limit there first, or any change from the stored value). No line → nothing set.
   The household's line (HU + EN in the report).
7. **Edge cases:** the night push is the controller's (never judged); the first day: apps with an explainer are judged
   on the floor (a mismatch is evidence whatever the line), apps without one are not judged before ~20 h of history; a
   slow or changing line (a Wi-Fi laptop): the line is a max over 14 nights — a falling line lowers nothing until it
   rolls out (R-1247 notes the active test); two apps at once: each judged on its own bytes; the household's own devices:
   invisible to the box — the trigger uses the box's own bytes only; an app reinstalled from scratch reads as „the
   household removed the limit" and is left alone.

## 7. Live proofs on 9202 (controller `0.342.0-rc.1`, commit `a9c1997`, CI run 2270 success)

Set by hand in 9202's `/etc/felhom-controller-image` (an `-rc` tag is never offered to a household); 9202's hub link
is off by design, so every event was handed to the notifier and DROPPED there — the hub side (allow-list, operator-only,
the mail line) is proven by hub 0.169.0's tests. 9202's off-site is off, so one line record (40 Mbit/s) was seeded by
hand into its settings.json. Logs: `SPIKE-upload-watch-2026-10-23/live/2-9202-watch-log-full.txt`.

| # | What | Result (UTC) |
|---|---|---|
| 1 | qBittorrent unlimited, line 40 Mbit/s → the box sets the limit | **19:47:42** „qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)" — read back from qBittorrent: 2 499 584 |
| 2 | **a real torrent seeding at the limit** (5 GB, private, 9202 → 9471) | **20:04:42** „qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s)"; no alarm (counter ratio 1.04) |
| 3 | **a CPU burner in Plex with no transcode** (4 busy loops, 400 % on a 7-core guest; trigger 350 %) | **20:07:42** „plex used 399% CPU … for 30 min with no work that explains it — operator alarm" → `app_unexplained_cpu`; ONE |
| 4 | **the fake hacked app**: a 50 Mbit/s flood from inside qBittorrent's container to 9471, hidden behind the real seed | **20:19:42** „qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s) — operator alarm" → `app_unexplained_upload`; ONE, none more while it ran |
| 5 | the burner stopped 20:11 | **20:25:41** „plex: the unexplained CPU stopped … — cleared" → `app_unexplained_cpu_cleared`; ONE |
| 6 | the household changes qBittorrent's limit by hand (1 500 000 → stored 1 499 136) at 20:21 | **20:32:42** „qBittorrent's upload limit is the household's — the box leaves it alone from now on"; read back at 20:43: still 1 499 136 |
| 7 | the flood stopped 20:26 (the seed went on, explained) | **20:41:41** „qbittorrent: the unexplained upload stopped … — cleared" → `app_unexplained_upload_cleared`; ONE |
| 8 | the system settings page (`GET /settings`) | „Feltöltési sebesség: legalább 40 Mbit/s (az éjszakai távoli mentésből mérve)"; qBittorrent's page carried no limit line after step 6 (the household's) — the box-owned branch is pinned by `TestUpwatch_QBLimitLineOnQBittorrentsPageOnly`, not captured live |

**Found live:** the very first minute after a start failed to reach the apps by name (the bootstrap joins the controller
to the app network a few seconds after the start); every later minute answered. That first minute only sets baselines —
**NOT-A-FINDING**. The media wiring's generated `docker-compose.override.yml` stays beside the template after a removal
(Jellyfin tonight; Plex's from the morning session) — rewritten before every start, harmless — **NOT-A-FINDING**.

**Teardown, three layers:** machine — the four throwaway apps removed through the product (containers and volumes gone,
verified), my test files and probes deleted, the `alpine:3.20` helper image removed, 9202 back on 0.341.0 with the seeded
record removed, the rc image removed; host — the receiver LXC 9471 destroyed (`--purge`, its disk gone); hub — none (9202
is not on the hub). The night sampler on the three test boxes is reads only.