17 KiB
SPIKE — who is using the upload line, and does the app explain it? (2026-10-23 night)
Operator ruling 2026-10-23 19:22 (CONTEXT.md): measure the line; on a sustained high upload attribute it to an app
and ask the app; a mismatch is the alarm; the same shape for CPU; qBittorrent gets a default upload limit from the
measured line, household changes respected. Brief Part 2.1. Design written from these numbers: §6 here and
architecture/08-alarm-ladder.md §9 [DESIGN — proposed]. Evidence and tools: SPIKE-upload-watch-2026-10-23/.
Read first: 08-alarm-ladder.md (R-388, §8.1), 07-backup-architecture.md (the night chain: W = 02:30 local, the
off-site push after it, the proof at 03:30 UTC), 15-media-library.md, 13-threat-model.md (attacker j),
audits/plex2shell-felhom-impact-2026-10-23/README.md (the 1.5 TB Plex upload). Controller: internal/metrics/
(container_metrics — one row per container per minute, 30 days), internal/monitor/healthcheck.go (the instant
whole-box CPU warning), internal/mediawire/ (the app clients and their credentials, 09 §3 decision 248).
Machines: the three test boxes READ ONLY (their stored metrics; their off-site record; tonight a one-second read of
the controller's own counter during the real push); scratch 9202 on demo-hp (throwaway qBittorrent, Transmission, Plex,
Jellyfin through the product's deploy); a throwaway receiver LXC 9471 cc-uw-recv on demo-hp's scratch storage
(never the thin pool). Every flood went 9202 → 9471 on demo-hp's own bridge; nothing went toward the internet.
1. Normal levels (stored metrics, three boxes)
History: demo-hp 2026-09-23 02:00 → 10-23 18:09 UTC (30.7 d, 998 216 rows); demo-felhom (the N100) same span
(30.7 d, 268 120 rows); Tester 1 2026-10-04 19:41 → 10-23 18:08 (18.9 d, 294 433 rows). Per app: upload from the
net_tx_bytes deltas (one-minute samples), CPU in percent of ONE core. Full tables: s1-levels-per-app.txt.
| box | the busiest uploaders (p50 / p95 / max Mbit/s) | the busiest CPU (p95 / max % of a core) |
|---|---|---|
| demo-hp | qbittorrent 0.02 / 1.2 / 4.0; immich-server 0.03 / 0.13 / 243 (one minute); felhom-controller 0 / 0.003 / 17.8 (the push) | grimmory 272 / 305 (a scan, 94 samples); immich-ML 32 / 548; romm 3 / 442; paperless 0.3 / 404 |
| demo-felhom | immich-server 0.67 / 5.95 / 12.9 (32 samples); jellyfin 0 / 0.85 / 2.9 | immich-ML 295 / 296 (an import); onlyoffice 131 / 148; paperless 137 / 140 |
| Tester 1 | qbittorrent 0.007 / 0.13 / 42.4; transmission 0.001 / 0.013 / 57.4; controller 0 / 0.003 / 37.5 | paperless 0.7 / 508; controller 46 / 423; plex 4.3 / 198 |
How long a normal high lasts (s1-longest-runs.txt — the longest run of consecutive minutes): upload ≥ 5 Mbit/s:
3 min at most (qBittorrent, Tester 1); ≥ 1 Mbit/s: 9 min. CPU ≥ 150 %: 7 min at most (Immich machine learning);
≥ 80 %: 13 min (Kimai). Busiest upload hours (UTC): demo-hp 03h, 02h; demo-felhom 16h, 05h; Tester 1 07h, 18h — no fixed
pattern beyond the night push. For scale: DooPlex's hacked Plex sent ~474 GB in one day (≈ 44 Mbit/s on average, for
hours) and used ~10 cores.
felhom-samba runs with the host network: its counter is always 0 (no own network namespace) — the watch cannot see it.
2. The line without a test
What the box already sees: the off-site push (restic, inside the controller container) is the box's only big
upload of its own. At the stored one-minute sample (s2-controller-push-per-night.txt): the controller's best five
minutes per night exceeded 2 Mbit/s on 2 of 31 nights on demo-hp (5.1 and 6.9 Mbit/s; best single minute 17.0),
1 of 19 on Tester 1 (7.6; best minute 37.5) and 0 of 31 on demo-felhom. The off-site record keeps the push's
duration (33 s on demo-felhom, 1 m 8 s on Tester 1, 3 m 19 s on demo-hp on 2026-10-23) and no byte count.
Conclusion: at a one-minute sample the push shows the night's DATA SIZE (a few MB, incremental), not the line. Not enough as it stands. Two changes make it usable: sample once a SECOND during the push (the controller knows when its push runs and can read its own counter), and let a night count only when its best 10 seconds moved ≥ 10 MB. Even then it is a lower bound (restic over SFTP to the Storage Box may not fill the line; the family uploads too).
Tonight's one-second read of the real push (2026-10-24 ~02:16Z, three boxes, read only): §2.1 below.
2.1 One-second samples of the real push
Read 2026-10-24 02:08–02:30Z, once a second, the controller container's own transmit counter (the counter the watch
reads: eth0 in the controller's network namespace), on the three test boxes, read only
(SPIKE-upload-watch-2026-10-23/night-push-1s-<box>.txt; the reader tools/nightsample.sh).
| Box | Sent in the 22 min | Best 10 s | 1-second peak | Counts (≥ 10 MiB in the best 10 s)? |
|---|---|---|---|---|
| demo-hp | 19.3 MB | 12.6 MB in 10.1 s = 10.0 Mbit/s (02:17:32Z) | 70.3 Mbit/s | yes → line „at least 10 Mbit/s" |
| demo-felhom | 3.3 MB | 3.2 MB = 2.5 Mbit/s (02:15:22Z) | 13.7 Mbit/s | no |
| Tester 1 | 3.6 MB | 3.5 MB = 2.7 Mbit/s (02:15:40Z) | 14.7 Mbit/s | no |
What it says: one night in three counted. The push is a burst of a few seconds (3–9 seconds above 100 kB/s), so even the counting night gives a low floor: 10 Mbit/s, while one second of the same push reached 70 Mbit/s. The design keeps the best counting night of the last 14, so the line rises only when a bigger night comes. Consequence for the qBittorrent limit (R-1245): 50 % of tonight's floor is ~5 Mbit/s on demo-hp — likely far below its real line. This is the known lower-bound limit (R-1247), now measured; it is written into R-1245 so the operator decides with it.
3. The line with a test — NOT run tonight
| Candidate | Ours? | Terms | Cost per test | Tonight |
|---|---|---|---|---|
| ep0 (a receiver on our Hetzner box) | yes | ours | 10 s × line ≈ 125 MB at 100 Mbit/s | not built — a change on ep0 is forbidden tonight |
| the box's own Storage Box sub-account, through the same transport as the push | yes (Hetzner, unlimited traffic) | Hetzner's — ours | same | not tried — a test file in the account that holds the restic repository needs its own design (the repo check, the quota) |
| Cloudflare R2 (where the ISO lives) | yes | ours | same + R2 class-A writes | not used — its write key is the operator's ISO-publish key; a test object in a public bucket is wrong |
| public speed tests (speed.cloudflare.com, LibreSpeed servers) | no | not read | — | not used (the brief: no third-party test without its terms) |
A quiet hour: never 02:30–05:30 local (dumps 02:30, second drive, off-site push, the 03:30 UTC proof). The family shares the line, so take the highest of several runs. Not needed for the first build — §6 works with the push's lower bound and an absolute floor; an active test is a decide row (R-1247).
4. Can the box ask each app? (measured on 9202 with throwaway installs; a private torrent 9202 → 9471)
Set-up: qBittorrent 5.2.4 and Transmission 4.1.3 installed through the product (POST /api/stacks/<n>/deploy; the
first try was refused BEFORE anything was made — the data place must be the drive itself; two empty helper folders
removed). The same 2 GiB file (an AES-CTR keystream of a fixed key) on both sides; a private torrent; a 40-line HTTP
tracker on 9471 (tools/tracker.py); transmission-daemon on 9471 as the leecher. The admin passwords were read from
the containers at use and never printed.
| App | Its own number | Measured | Limit | Household's limit visible? |
|---|---|---|---|---|
| qBittorrent | /api/v2/transfer/info → up_info_data (cumulative), up_info_speed |
seeding: app 131.1 Mbit/s vs container 134.6–140.5 → ×1.03–1.07 (s4-qb-seed-nolimit.jsonl) |
transfer/setUploadLimit (bytes/s) = preferences.up_limit; stored in KiB/s (2 500 000 → 2 499 584); survives a restart (Session\GlobalUPSpeedLimit=2441); measured: ~20 Mbit/s held (s4-qb-seed-limit20.jsonl) |
yes: transfer/uploadLimit ≠ the value the box stored |
| Transmission | RPC session-stats.current-stats.uploadedBytes, uploadSpeed |
seeding: app 487.1 / 390.3 / 115.8 vs container 487.6 / 390.7 / 117.9 Mbit/s → ×1.001–1.02 (s4-tr-seed-nolimit.jsonl) |
session-set speed-limit-up (kB/s) + -enabled (read back); its rate under the limit was NOT observed — the re-added leecher got no connection in 2 minutes |
yes: speed-limit-up-enabled + value |
| Plex | /status/sessions, /transcode/sessions (MediaContainer.size) |
unclaimed 1.43.5: both 200 without a token from the docker network, {"size":0}; a claimed server needs its PlexOnlineToken (read at use from Preferences.xml, as mediawire does — no new secret) |
— | — |
| Jellyfin | /Sessions |
401 without a key; the key named felhom exists only when the household made it (mediawire's „one household step") — the box cannot create one at install without the admin login it does not have |
— | — |
| Nextcloud, Immich | no upload-explaining number found (not measured further); Immich's job queue needs an API key the box does not hold | — | — | → the plain rule (§6) |
The fake hacked app (code inside qBittorrent's own container, docker exec … python3 → 9471:5001, Plex2Shell's
shape): the container sent ~10 100 Mbit/s (LAN, one bridge) while qBittorrent reported 0 (s4-qb-flood.jsonl).
The counter itself: the Docker Engine API's one-shot stats (networks.*.tx_bytes) equal the kernel's counter in
the container's network namespace to the byte; docker stats' text (what internal/metrics stores) has three
significant digits — ~10 MB steps at a GB total, ~1.3 Mbit/s at a one-minute sample. The watch reads the Engine API.
A limit the hacked app controls: an attacker with code in the app's container can also answer its API. The check catches an attacker who does not know about it; one who does could lie. §5 is the answer to that.
5. A universal upload ceiling — spike only
A helper container sharing the app's network namespace (docker run --rm --network container:qbittorrent --cap-add NET_ADMIN alpine:3.20 + iproute2-tc) added tc qdisc … root tbf rate 20mbit burst 64kb latency 400ms on the app's
eth0 inside the unprivileged 9202 guest. The same flood then ran at 19.9–20.1 Mbit/s (s4-qb-flood-tbf.jsonl),
qBittorrent's API kept answering, and tc qdisc del restored it. Costs: a helper image with tc (a new external image,
or tc shipped in the controller image), NET_ADMIN for that helper, re-applied after every app restart (the qdisc
dies with the namespace), upload only. Not built (brief). It would cap a hacked app's flood whatever its API says.
6. The design (numbers from §1–§5; full text 08-alarm-ladder.md §9)
- Line: one-second samples of the controller's own sent bytes during the off-site push; the best ≥ 10-second rate
per night; a night counts when that window moved ≥ 10 MB; the line = the highest counting night of the last 14;
settings.json
upwatch.line; shown on the system settings page as „legalább N Mbit/s" („még mérjük" until known). - Trigger: an app (its containers summed, exact counters, once a minute) sends ≥ max(5 Mbit/s, 25 % of the line) for 15 minutes — five times the longest normal run at ≥ 5 Mbit/s (3 min). Per app, not per box: the box cannot tell LAN from internet bytes inside its controller container, and the per-app view is what the check compares.
- Attribution: the app's compose project; infrastructure (controller, traefik, cloudflared, Samba) never judged.
- Explanation: qBittorrent / Transmission: explained when container bytes ≤ 1.25 × the app's own count + 1 Mbit/s
(measured overhead 1–7 %); Plex: any viewer explains the upload (session bandwidth not measured — the conservative
half; no viewer + a big upload = the Plex2Shell signature). Not explained →
app_unexplained_upload, operator-only, one per episode, a clear after 15 quiet minutes. No explainer: alarm only at ≥ 10 × its own p95 for 60 minutes, after ~20 h of history, worded „no app number to compare". - CPU: ≥ max(150 %, 50 % × cores) of one core for 30 minutes (four times the longest normal run at ≥ 150 %),
explained by a Plex transcode or the box's own work on the app (backup, update, deploy) → else
app_unexplained_cpu. No explainer: ≥ max(200 %, 75 % × cores) for 60 minutes. - qBittorrent limit: 50 % of the line (CC's pick — R-1245), every 15 minutes, re-set when the line moves > 20 %; never over a household's limit (a limit there first, or any change from the stored value). No line → nothing set. The household's line (HU + EN in the report).
- Edge cases: the night push is the controller's (never judged); the first day: apps with an explainer are judged on the floor (a mismatch is evidence whatever the line), apps without one are not judged before ~20 h of history; a slow or changing line (a Wi-Fi laptop): the line is a max over 14 nights — a falling line lowers nothing until it rolls out (R-1247 notes the active test); two apps at once: each judged on its own bytes; the household's own devices: invisible to the box — the trigger uses the box's own bytes only; an app reinstalled from scratch reads as „the household removed the limit" and is left alone.
7. Live proofs on 9202 (controller 0.342.0-rc.1, commit a9c1997, CI run 2270 success)
Set by hand in 9202's /etc/felhom-controller-image (an -rc tag is never offered to a household); 9202's hub link
is off by design, so every event was handed to the notifier and DROPPED there — the hub side (allow-list, operator-only,
the mail line) is proven by hub 0.169.0's tests. 9202's off-site is off, so one line record (40 Mbit/s) was seeded by
hand into its settings.json. Logs: SPIKE-upload-watch-2026-10-23/live/2-9202-watch-log-full.txt.
| # | What | Result (UTC) |
|---|---|---|
| 1 | qBittorrent unlimited, line 40 Mbit/s → the box sets the limit | 19:47:42 „qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)" — read back from qBittorrent: 2 499 584 |
| 2 | a real torrent seeding at the limit (5 GB, private, 9202 → 9471) | 20:04:42 „qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s)"; no alarm (counter ratio 1.04) |
| 3 | a CPU burner in Plex with no transcode (4 busy loops, 400 % on a 7-core guest; trigger 350 %) | 20:07:42 „plex used 399% CPU … for 30 min with no work that explains it — operator alarm" → app_unexplained_cpu; ONE |
| 4 | the fake hacked app: a 50 Mbit/s flood from inside qBittorrent's container to 9471, hidden behind the real seed | 20:19:42 „qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s) — operator alarm" → app_unexplained_upload; ONE, none more while it ran |
| 5 | the burner stopped 20:11 | 20:25:41 „plex: the unexplained CPU stopped … — cleared" → app_unexplained_cpu_cleared; ONE |
| 6 | the household changes qBittorrent's limit by hand (1 500 000 → stored 1 499 136) at 20:21 | 20:32:42 „qBittorrent's upload limit is the household's — the box leaves it alone from now on"; read back at 20:43: still 1 499 136 |
| 7 | the flood stopped 20:26 (the seed went on, explained) | 20:41:41 „qbittorrent: the unexplained upload stopped … — cleared" → app_unexplained_upload_cleared; ONE |
| 8 | the system settings page (GET /settings) |
„Feltöltési sebesség: legalább 40 Mbit/s (az éjszakai távoli mentésből mérve)"; qBittorrent's page carried no limit line after step 6 (the household's) — the box-owned branch is pinned by TestUpwatch_QBLimitLineOnQBittorrentsPageOnly, not captured live |
Found live: the very first minute after a start failed to reach the apps by name (the bootstrap joins the controller
to the app network a few seconds after the start); every later minute answered. That first minute only sets baselines —
NOT-A-FINDING. The media wiring's generated docker-compose.override.yml stays beside the template after a removal
(Jellyfin tonight; Plex's from the morning session) — rewritten before every start, harmless — NOT-A-FINDING.
Teardown, three layers: machine — the four throwaway apps removed through the product (containers and volumes gone,
verified), my test files and probes deleted, the alpine:3.20 helper image removed, 9202 back on 0.341.0 with the seeded
record removed, the rc image removed; host — the receiver LXC 9471 destroyed (--purge, its disk gone); hub — none (9202
is not on the hub). The night sampler on the three test boxes is reads only.