Upload watch: spike, design (08 section 9, proposed), live proofs on 9202; rows R-1244..R-1248 (register 189 -> 194); STATUS D41/D42 + the Monday list
gates / gates (push) Successful in 13m1s

Spike: per-app normal levels on three boxes (19-31 days), the off-site push as a line measure (a lower bound),
qBittorrent/Transmission/Plex explainers measured on 9202, a tc ceiling measured (not built). Live on 9202 with
0.342.0-rc.1: limit set; real seed explained; CPU burner and the hidden flood alarmed once each and cleared once;
the household's limit left alone. Teardown: apps removed, 9202 back on 0.341.0, receiver 9471 destroyed.
This commit is contained in:
admin committed 2026-10-23 22:52:41 +02:00
1 parent af5c005a4e
commit a91d0b2421
31 files changed
+899 -11

No files matched your search

@@ -0,0 +1,159 @@
# SPIKE — who is using the upload line, and does the app explain it? (2026-10-23 night)
Operator ruling 2026-10-23 19:22 (`CONTEXT.md`): measure the line; on a sustained high upload attribute it to an app
and ask the app; a mismatch is the alarm; the same shape for CPU; qBittorrent gets a default upload limit from the
measured line, household changes respected. Brief Part 2.1. Design written from these numbers: §6 here and
`architecture/08-alarm-ladder.md` §9 [DESIGN — proposed]. Evidence and tools: `SPIKE-upload-watch-2026-10-23/`.
Read first: `08-alarm-ladder.md` (R-388, §8.1), `07-backup-architecture.md` (the night chain: W = 02:30 local, the
off-site push after it, the proof at 03:30 UTC), `15-media-library.md`, `13-threat-model.md` (attacker j),
`audits/plex2shell-felhom-impact-2026-10-23/README.md` (the 1.5 TB Plex upload). Controller: `internal/metrics/`
(`container_metrics` — one row per container per minute, 30 days), `internal/monitor/healthcheck.go` (the instant
whole-box CPU warning), `internal/mediawire/` (the app clients and their credentials, `09` §3 decision 248).
Machines: the three test boxes READ ONLY (their stored metrics; their off-site record; tonight a one-second read of
the controller's own counter during the real push); scratch 9202 on demo-hp (throwaway qBittorrent, Transmission, Plex,
Jellyfin through the product's deploy); a throwaway receiver LXC **9471 `cc-uw-recv`** on demo-hp's scratch storage
(never the thin pool). Every flood went 9202 → 9471 on demo-hp's own bridge; nothing went toward the internet.
## 1. Normal levels (stored metrics, three boxes)
History: **demo-hp** 2026-09-23 02:00 → 10-23 18:09 UTC (30.7 d, 998 216 rows); **demo-felhom** (the N100) same span
(30.7 d, 268 120 rows); **Tester 1** 2026-10-04 19:41 → 10-23 18:08 (18.9 d, 294 433 rows). Per app: upload from the
`net_tx_bytes` deltas (one-minute samples), CPU in percent of ONE core. Full tables: `s1-levels-per-app.txt`.
| box | the busiest uploaders (p50 / p95 / max Mbit/s) | the busiest CPU (p95 / max % of a core) |
|---|---|---|
| demo-hp | qbittorrent 0.02 / 1.2 / 4.0; immich-server 0.03 / 0.13 / 243 (one minute); felhom-controller 0 / 0.003 / 17.8 (the push) | grimmory 272 / 305 (a scan, 94 samples); immich-ML 32 / 548; romm 3 / 442; paperless 0.3 / 404 |
| demo-felhom | immich-server 0.67 / 5.95 / 12.9 (32 samples); jellyfin 0 / 0.85 / 2.9 | immich-ML 295 / 296 (an import); onlyoffice 131 / 148; paperless 137 / 140 |
| Tester 1 | qbittorrent 0.007 / 0.13 / 42.4; transmission 0.001 / 0.013 / 57.4; controller 0 / 0.003 / 37.5 | paperless 0.7 / 508; controller 46 / 423; plex 4.3 / 198 |
**How long a normal high lasts** (`s1-longest-runs.txt` — the longest run of consecutive minutes): upload ≥ 5 Mbit/s:
**3 min** at most (qBittorrent, Tester 1); ≥ 1 Mbit/s: 9 min. CPU ≥ 150 %: **7 min** at most (Immich machine learning);
≥ 80 %: 13 min (Kimai). Busiest upload hours (UTC): demo-hp 03h, 02h; demo-felhom 16h, 05h; Tester 1 07h, 18h — no fixed
pattern beyond the night push. For scale: DooPlex's hacked Plex sent ~474 GB in one day (≈ 44 Mbit/s on average, for
hours) and used ~10 cores.
`felhom-samba` runs with the host network: its counter is always 0 (no own network namespace) — the watch cannot see it.
## 2. The line without a test
**What the box already sees:** the off-site push (restic, inside the controller container) is the box's only big
upload of its own. At the stored one-minute sample (`s2-controller-push-per-night.txt`): the controller's best five
minutes per night exceeded 2 Mbit/s on **2 of 31 nights** on demo-hp (5.1 and 6.9 Mbit/s; best single minute 17.0),
**1 of 19** on Tester 1 (7.6; best minute 37.5) and **0 of 31** on demo-felhom. The off-site record keeps the push's
duration (33 s on demo-felhom, 1 m 8 s on Tester 1, 3 m 19 s on demo-hp on 2026-10-23) and **no byte count**.
**Conclusion:** at a one-minute sample the push shows the night's DATA SIZE (a few MB, incremental), not the line. Not
enough as it stands. Two changes make it usable: sample once a SECOND during the push (the controller knows when its
push runs and can read its own counter), and let a night count only when its best 10 seconds moved ≥ 10 MB. Even then
it is a **lower bound** (restic over SFTP to the Storage Box may not fill the line; the family uploads too).
**Tonight's one-second read of the real push** (2026-10-24 ~02:16Z, three boxes, read only): §2.1 below.
### 2.1 One-second samples of the real push
*(filled after 02:30Z — see the section at the end, „Night data")*
## 3. The line with a test — NOT run tonight
| Candidate | Ours? | Terms | Cost per test | Tonight |
|---|---|---|---|---|
| ep0 (a receiver on our Hetzner box) | yes | ours | 10 s × line ≈ 125 MB at 100 Mbit/s | **not built** — a change on ep0 is forbidden tonight |
| the box's own Storage Box sub-account, through the same transport as the push | yes (Hetzner, unlimited traffic) | Hetzner's — ours | same | not tried — a test file in the account that holds the restic repository needs its own design (the repo check, the quota) |
| Cloudflare R2 (where the ISO lives) | yes | ours | same + R2 class-A writes | not used — its write key is the operator's ISO-publish key; a test object in a public bucket is wrong |
| public speed tests (speed.cloudflare.com, LibreSpeed servers) | no | **not read** | — | not used (the brief: no third-party test without its terms) |
A quiet hour: never 02:30–05:30 local (dumps 02:30, second drive, off-site push, the 03:30 UTC proof). The family shares
the line, so take the highest of several runs. **Not needed for the first build** — §6 works with the push's lower
bound and an absolute floor; an active test is a decide row (R-1247).
## 4. Can the box ask each app? (measured on 9202 with throwaway installs; a private torrent 9202 → 9471)
Set-up: qBittorrent 5.2.4 and Transmission 4.1.3 installed through the product (`POST /api/stacks/<n>/deploy`; the
first try was refused BEFORE anything was made — the data place must be the drive itself; two empty helper folders
removed). The same 2 GiB file (an AES-CTR keystream of a fixed key) on both sides; a private torrent; a 40-line HTTP
tracker on 9471 (`tools/tracker.py`); `transmission-daemon` on 9471 as the leecher. The admin passwords were read from
the containers at use and never printed.
| App | Its own number | Measured | Limit | Household's limit visible? |
|---|---|---|---|---|
| **qBittorrent** | `/api/v2/transfer/info` → `up_info_data` (cumulative), `up_info_speed` | seeding: app **131.1 Mbit/s** vs container **134.6–140.5** → **×1.03–1.07** (`s4-qb-seed-nolimit.jsonl`) | `transfer/setUploadLimit` (bytes/s) = `preferences.up_limit`; stored in KiB/s (2 500 000 → **2 499 584**); survives a restart (`Session\GlobalUPSpeedLimit=2441`); measured: ~20 Mbit/s held (`s4-qb-seed-limit20.jsonl`) | yes: `transfer/uploadLimit` ≠ the value the box stored |
| **Transmission** | RPC `session-stats.current-stats.uploadedBytes`, `uploadSpeed` | seeding: app 487.1 / 390.3 / 115.8 vs container 487.6 / 390.7 / 117.9 Mbit/s → **×1.001–1.02** (`s4-tr-seed-nolimit.jsonl`) | `session-set speed-limit-up` (kB/s) + `-enabled` (read back); **its rate under the limit was NOT observed** — the re-added leecher got no connection in 2 minutes | yes: `speed-limit-up-enabled` + value |
| **Plex** | `/status/sessions`, `/transcode/sessions` (`MediaContainer.size`) | unclaimed 1.43.5: both 200 without a token from the docker network, `{"size":0}`; a claimed server needs its PlexOnlineToken (read at use from Preferences.xml, as mediawire does — no new secret) | — | — |
| **Jellyfin** | `/Sessions` | **401** without a key; the key named `felhom` exists only when the household made it (mediawire's „one household step") — the box cannot create one at install without the admin login it does not have | — | — |
| Nextcloud, Immich | **no upload-explaining number found** (not measured further); Immich's job queue needs an API key the box does not hold | — | — | → the plain rule (§6) |
**The fake hacked app** (code inside qBittorrent's own container, `docker exec … python3` → 9471:5001, Plex2Shell's
shape): the container sent **~10 100 Mbit/s** (LAN, one bridge) while qBittorrent reported **0** (`s4-qb-flood.jsonl`).
**The counter itself:** the Docker Engine API's one-shot stats (`networks.*.tx_bytes`) equal the kernel's counter in
the container's network namespace to the byte; `docker stats`' text (what `internal/metrics` stores) has three
significant digits — ~10 MB steps at a GB total, ~1.3 Mbit/s at a one-minute sample. The watch reads the Engine API.
**A limit the hacked app controls:** an attacker with code in the app's container can also answer its API. The check
catches an attacker who does not know about it; one who does could lie. §5 is the answer to that.
## 5. A universal upload ceiling — spike only
A helper container sharing the app's network namespace (`docker run --rm --network container:qbittorrent --cap-add
NET_ADMIN alpine:3.20` + `iproute2-tc`) added `tc qdisc … root tbf rate 20mbit burst 64kb latency 400ms` on the app's
`eth0` **inside the unprivileged 9202 guest**. The same flood then ran at **19.9–20.1 Mbit/s** (`s4-qb-flood-tbf.jsonl`),
qBittorrent's API kept answering, and `tc qdisc del` restored it. Costs: a helper image with `tc` (a new external image,
or `tc` shipped in the controller image), `NET_ADMIN` for that helper, re-applied after every app restart (the qdisc
dies with the namespace), upload only. **Not built** (brief). It would cap a hacked app's flood whatever its API says.
## 6. The design (numbers from §1–§5; full text `08-alarm-ladder.md` §9)
1. **Line:** one-second samples of the controller's own sent bytes during the off-site push; the best ≥ 10-second rate
per night; a night counts when that window moved ≥ 10 MB; the line = the highest counting night of the last 14;
settings.json `upwatch.line`; shown on the system settings page as „legalább N Mbit/s" („még mérjük" until known).
2. **Trigger:** an app (its containers summed, exact counters, once a minute) sends ≥ **max(5 Mbit/s, 25 % of the line)**
for **15 minutes** — five times the longest normal run at ≥ 5 Mbit/s (3 min). Per app, not per box: the box cannot
tell LAN from internet bytes inside its controller container, and the per-app view is what the check compares.
3. **Attribution:** the app's compose project; infrastructure (controller, traefik, cloudflared, Samba) never judged.
4. **Explanation:** qBittorrent / Transmission: explained when container bytes ≤ 1.25 × the app's own count + 1 Mbit/s
(measured overhead 1–7 %); Plex: any viewer explains the upload (session bandwidth not measured — the conservative
half; no viewer + a big upload = the Plex2Shell signature). Not explained → **`app_unexplained_upload`**,
operator-only, one per episode, a clear after 15 quiet minutes. No explainer: alarm only at ≥ 10 × its own p95 for
60 minutes, after ~20 h of history, worded „no app number to compare".
5. **CPU:** ≥ **max(150 %, 50 % × cores)** of one core for **30 minutes** (four times the longest normal run at ≥ 150 %),
explained by a Plex transcode or the box's own work on the app (backup, update, deploy) → else
**`app_unexplained_cpu`**. No explainer: ≥ max(200 %, 75 % × cores) for 60 minutes.
6. **qBittorrent limit:** 50 % of the line (CC's pick — R-1245), every 15 minutes, re-set when the line moves > 20 %;
never over a household's limit (a limit there first, or any change from the stored value). No line → nothing set.
The household's line (HU + EN in the report).
7. **Edge cases:** the night push is the controller's (never judged); the first day: apps with an explainer are judged
on the floor (a mismatch is evidence whatever the line), apps without one are not judged before ~20 h of history; a
slow or changing line (a Wi-Fi laptop): the line is a max over 14 nights — a falling line lowers nothing until it
rolls out (R-1247 notes the active test); two apps at once: each judged on its own bytes; the household's own devices:
invisible to the box — the trigger uses the box's own bytes only; an app reinstalled from scratch reads as „the
household removed the limit" and is left alone.
## 7. Live proofs on 9202 (controller `0.342.0-rc.1`, commit `a9c1997`, CI run 2270 success)
Set by hand in 9202's `/etc/felhom-controller-image` (an `-rc` tag is never offered to a household); 9202's hub link
is off by design, so every event was handed to the notifier and DROPPED there — the hub side (allow-list, operator-only,
the mail line) is proven by hub 0.169.0's tests. 9202's off-site is off, so one line record (40 Mbit/s) was seeded by
hand into its settings.json. Logs: `SPIKE-upload-watch-2026-10-23/live/2-9202-watch-log-full.txt`.
| # | What | Result (UTC) |
|---|---|---|
| 1 | qBittorrent unlimited, line 40 Mbit/s → the box sets the limit | **19:47:42** „qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)" — read back from qBittorrent: 2 499 584 |
| 2 | **a real torrent seeding at the limit** (5 GB, private, 9202 → 9471) | **20:04:42** „qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s)"; no alarm (counter ratio 1.04) |
| 3 | **a CPU burner in Plex with no transcode** (4 busy loops, 400 % on a 7-core guest; trigger 350 %) | **20:07:42** „plex used 399% CPU … for 30 min with no work that explains it — operator alarm" → `app_unexplained_cpu`; ONE |
| 4 | **the fake hacked app**: a 50 Mbit/s flood from inside qBittorrent's container to 9471, hidden behind the real seed | **20:19:42** „qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s) — operator alarm" → `app_unexplained_upload`; ONE, none more while it ran |
| 5 | the burner stopped 20:11 | **20:25:41** „plex: the unexplained CPU stopped … — cleared" → `app_unexplained_cpu_cleared`; ONE |
| 6 | the household changes qBittorrent's limit by hand (1 500 000 → stored 1 499 136) at 20:21 | **20:32:42** „qBittorrent's upload limit is the household's — the box leaves it alone from now on"; read back at 20:43: still 1 499 136 |
| 7 | the flood stopped 20:26 (the seed went on, explained) | **20:41:41** „qbittorrent: the unexplained upload stopped … — cleared" → `app_unexplained_upload_cleared`; ONE |
| 8 | the system settings page (`GET /settings`) | „Feltöltési sebesség: legalább 40 Mbit/s (az éjszakai távoli mentésből mérve)"; qBittorrent's page carried no limit line after step 6 (the household's) — the box-owned branch is pinned by `TestUpwatch_QBLimitLineOnQBittorrentsPageOnly`, not captured live |
**Found live:** the very first minute after a start failed to reach the apps by name (the bootstrap joins the controller
to the app network a few seconds after the start); every later minute answered. That first minute only sets baselines —
**NOT-A-FINDING**. The media wiring's generated `docker-compose.override.yml` stays beside the template after a removal
(Jellyfin tonight; Plex's from the morning session) — rewritten before every start, harmless — **NOT-A-FINDING**.
**Teardown, three layers:** machine — the four throwaway apps removed through the product (containers and volumes gone,
verified), my test files and probes deleted, the `alpine:3.20` helper image removed, 9202 back on 0.341.0 with the seeded
record removed, the rc image removed; host — the receiver LXC 9471 destroyed (`--purge`, its disk gone); hub — none (9202
is not on the hub). The night sampler on the three test boxes is reads only.
@@ -0,0 +1,15 @@
gitea.dooplex.hu/admin/felhom-controller:0.342.0-rc.1
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0-rc.1 Up 6 minutes (healthy)
filebrowser gtstef/filebrowser:1.5.6-stable Up 4 hours (healthy)
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 4 hours (healthy)
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 4 hours (healthy)
immich-redis redis:7-alpine Up 4 hours (healthy)
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 4 hours (healthy)
jellyfin jellyfin/jellyfin:10.11.11 Up About an hour (healthy)
paperless-postgres postgres:18-alpine Up 4 hours (healthy)
paperless-redis redis:7-alpine Up 4 hours (healthy)
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 4 hours (healthy)
plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 Up About an hour (healthy)
qbittorrent qbittorrentofficial/qbittorrent-nox:5.2.4-1 Up About an hour (healthy)
traefik traefik:v3.7.13 Up 4 hours
transmission lscr.io/linuxserver/transmission:version-4.1.3-r0 Up About an hour (healthy)
@@ -0,0 +1,10 @@
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
9471 running cc-uw-recv
description: CC throwaway 2026-10-23 night%3A upload-watch spike receiver/leecher. Remove after the session.%0A
hostname: cc-uw-recv
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:04:68:80,ip=dhcp,type=veth
rootfs: nvme-scratch:9471/vm-9471-disk-0.raw,size=8G
tags: cc;scratch;uw-spike
@@ -0,0 +1,9 @@
2026/10/23 19:32:41 upwatch_wiring.go:97: [INFO] [upwatch] upload and CPU watch started (line at least 40 Mbit/s, qBittorrent share 50 %)
2026/10/23 19:32:41 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] transmission: the app did not answer (Post "http://transmission:9091/transmission/rpc": dial tcp: lookup transmission on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:32:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_definition_refused (severity warning) — further app_definition_refused events are logged at DEBUG only
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] qbittorrent: the app did not answer (Post "http://qbittorrent:8080/api/v2/auth/login": dial tcp: lookup qbittorrent on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:47:42 service.go:108: [INFO] [upwatch] qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)
2026/10/23 20:04:42 watch.go:303: [INFO] [upwatch] qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)); normal use
2026/10/23 20:07:42 watch.go:343: [WARN] [upwatch] plex used 399% CPU (of one core; 7 cores) for 30 min with no work that explains it (plex transcodes) — operator alarm
2026/10/23 20:07:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu (severity warning) — further app_unexplained_cpu events are logged at DEBUG only
@@ -0,0 +1,16 @@
2026/10/23 19:32:41 upwatch_wiring.go:97: [INFO] [upwatch] upload and CPU watch started (line at least 40 Mbit/s, qBittorrent share 50 %)
2026/10/23 19:32:41 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] transmission: the app did not answer (Post "http://transmission:9091/transmission/rpc": dial tcp: lookup transmission on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:32:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_definition_refused (severity warning) — further app_definition_refused events are logged at DEBUG only
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] qbittorrent: the app did not answer (Post "http://qbittorrent:8080/api/v2/auth/login": dial tcp: lookup qbittorrent on 192.168.0.250:53: no such host) — treated as no number this minute
2026/10/23 19:47:42 service.go:108: [INFO] [upwatch] qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)
2026/10/23 20:04:42 watch.go:303: [INFO] [upwatch] qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)); normal use
2026/10/23 20:07:42 watch.go:343: [WARN] [upwatch] plex used 399% CPU (of one core; 7 cores) for 30 min with no work that explains it (plex transcodes) — operator alarm
2026/10/23 20:07:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu (severity warning) — further app_unexplained_cpu events are logged at DEBUG only
2026/10/23 20:19:42 watch.go:307: [WARN] [upwatch] qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)) — operator alarm
2026/10/23 20:19:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_upload (severity warning) — further app_unexplained_upload events are logged at DEBUG only
2026/10/23 20:25:41 watch.go:378: [INFO] [upwatch] plex: the unexplained CPU stopped (15 min below the trigger or explained) — cleared
2026/10/23 20:25:41 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu_cleared (severity info) — further app_unexplained_cpu_cleared events are logged at DEBUG only
2026/10/23 20:32:42 service.go:111: [INFO] [upwatch] qBittorrent's upload limit is the household's — the box leaves it alone from now on
2026/10/23 20:41:41 watch.go:370: [INFO] [upwatch] qbittorrent: the unexplained upload stopped (15 min below the trigger or explained) — cleared
2026/10/23 20:41:41 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_upload_cleared (severity info) — further app_unexplained_upload_cleared events are logged at DEBUG only
@@ -0,0 +1,2 @@
/settings 56743 data-uplink> <span class="settings-label">Feltöltési sebesség</span> <span class="settings-value">legalább <span class="mono">40</span> Mbit/s (az éjszakai távoli mentésből mérve)</span> </div> | qb line: none
/apps/qbittorrent 49281 no uplink row | qb line: none
@@ -0,0 +1,4 @@
version v5.2.4 webapi 2.15.1
transfer/info {'connection_status': 'connected', 'dht_nodes': 375, 'dl_info_data': 0, 'dl_info_speed': 0, 'dl_rate_limit': 0, 'last_external_address_v4': '37.191.56.193', 'last_external_address_v6': '', 'up_info_data': 5566595787, 'up_info_speed': 0, 'up_rate_limit': 1499136}
prefs {'up_limit': 1499136, 'dl_limit': 0, 'alt_up_limit': 10240, 'alt_dl_limit': 10240, 'scheduler_enabled': False, 'limit_tcp_overhead': False, 'limit_lan_peers': True, 'limit_utp_rate': True, 'max_uploads': 20, 'enable_upload_suggestions': False}
uploadLimit 1499136 speedLimitsMode 0
@@ -0,0 +1,186 @@
=== demo-hp: 2026-09-23 02:00 -> 2026-10-23 18:09 UTC (30.7 d), 998216 rows
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
qbittorrent 18 0.019 1.199 4.01 | 0.4 5.4 11.3 |
metube 3 0.146 0.146 0.15 | 0.0 0.1 0.1 |
immich-server 318 0.031 0.133 242.93 | 0.1 5.6 138.1 | resets=7
grimmory-db 102 0.000 0.116 0.35 | 0.0 16.9 102.5 | resets=12
immich-postgres 320 0.000 0.066 2.20 | 0.0 8.7 70.2 | resets=7
grimmory 94 0.001 0.047 0.77 | 1.1 272.0 305.0 | resets=9
immich-redis 320 0.013 0.029 0.18 | 0.6 4.3 5.1 | resets=7
romm 26492 0.019 0.027 0.20 | 0.7 3.2 442.4 | resets=97
docmost 43499 0.013 0.027 0.23 | 0.1 0.8 158.1 | resets=147
rallly 71 0.000 0.017 0.12 | 0.0 8.5 152.1 | resets=10
radarr 16 0.002 0.015 0.03 | 0.2 7.2 11.8 |
romm-redis 26523 0.005 0.013 0.10 | 0.7 4.4 61.5 | resets=101
paperless-webserver 43488 0.005 0.013 0.04 | 0.1 0.3 404.2 | resets=144
nextcloud-db 663 0.001 0.013 1.74 | 0.0 6.1 71.8 | resets=11
docmost-redis 43532 0.005 0.013 0.13 | 0.7 4.4 6.6 | resets=147
cloudflared 43604 0.000 0.013 1.21 | 0.3 0.5 13.8 | resets=6
rallly-postgres 71 0.000 0.007 0.01 | 0.0 5.0 8.8 | resets=7
prowlarr 5 0.002 0.004 0.00 | 0.1 0.2 0.2 |
sonarr 18 0.000 0.004 0.01 | 2.5 8.2 8.8 |
felhom-controller 43608 0.000 0.003 17.79 | 1.0 57.9 251.4 | resets=133
paperless-redis 43521 0.001 0.003 0.02 | 0.6 4.3 6.1 | resets=148
bentopdf 43567 0.000 0.003 0.01 | 0.0 0.0 5.2 | resets=58
homepage 78 0.000 0.002 0.01 | 0.0 4.2 4.5 | resets=1
bookstack-db 43565 0.001 0.001 0.01 | 0.0 6.1 102.8 | resets=147
romm-db 26526 0.000 0.001 0.19 | 0.0 6.0 105.3 | resets=89
plex 11974 0.001 0.001 1.10 | 0.3 0.6 124.6 | resets=46
nextcloud 662 0.000 0.001 0.80 | 0.0 5.7 150.0 | resets=11
immich-machine-learning 320 0.000 0.001 0.39 | 0.2 32.0 548.0 | resets=7
docmost-postgres 43533 0.000 0.001 0.02 | 0.0 4.7 29.6 | resets=147
bookstack 43546 0.000 0.001 0.01 | 0.0 0.0 63.0 | resets=148
adventurelog-postgres 43567 0.000 0.001 1.35 | 0.0 4.7 37.9 | resets=138
adventurelog 43549 0.000 0.001 2.16 | 0.0 0.7 204.3 | resets=146
ghost 20 0.000 0.001 0.00 | 0.0 6.8 132.9 | resets=2
glance 20 0.000 0.001 0.00 | 0.0 0.0 0.0 | resets=2
privatebin 43518 0.000 0.001 0.00 | 0.1 0.1 8.1 | resets=140
paperless-postgres 43520 0.000 0.001 0.05 | 0.0 4.7 19.1 | resets=148
kimai 30136 0.000 0.000 0.00 | 0.0 0.0 107.2 | resets=102
home-assistant 14403 0.000 0.000 1.00 | 0.0 0.5 106.9 | resets=62
kimai-db 30150 0.000 0.000 0.05 | 0.0 5.9 100.8 | resets=96
adventurelog-frontend 43559 0.000 0.000 0.21 | 0.0 3.8 22.8 | resets=126
vaultwarden 329 0.000 0.000 0.00 | 0.0 0.1 4.0 | resets=7
calibre-web 43547 0.000 0.000 0.01 | 0.0 0.3 127.1 | resets=145
opengist 43530 0.000 0.000 0.01 | 0.0 0.0 9.8 | resets=140
jellyfin 297 0.000 0.000 5.87 | 0.0 4.4 160.5 | resets=7
vikunja 8 0.000 0.000 0.00 | 0.0 0.0 0.0 | resets=2
a1-echo 113 0.000 0.000 0.00 | 0.0 0.0 0.0 |
upbeat_perlman 1 0.000 0.000 0.00 | 31.2 31.2 31.2 |
traefik 43608 0.000 0.000 1.21 | 0.1 3.5 92.1 | resets=11
practical_johnson 1 0.000 0.000 0.00 | 9.9 9.9 9.9 |
peaceful_ptolemy 1 0.000 0.000 0.00 | 14.1 14.1 14.1 |
nextcloud-redis 663 0.000 0.000 0.03 | 0.6 4.1 7.9 | resets=5
friendly_wilbur 1 0.000 0.000 0.00 | 59.0 59.0 59.0 |
filebrowser 43607 0.000 0.000 0.74 | 0.0 0.0 35.1 | resets=17
felhom-samba 10 0.000 0.000 0.00 | 0.0 13.5 17.6 | tx counter always 0 (host network?)
determined_satoshi 1 0.000 0.000 0.00 | 17.4 17.4 17.4 |
busiest UTC hours by bytes sent: 03h 14%, 02h 7%, 13h 5%, 00h 4%
=== demo-felhom: 2026-09-23 02:00 -> 2026-10-23 18:08 UTC (30.7 d), 268120 rows
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
immich-server 32 0.667 5.953 12.90 | 0.6 120.1 130.8 |
jellyfin 16 0.000 0.854 2.85 | 0.0 72.7 290.7 |
immich-postgres 32 0.000 0.766 2.24 | 0.3 8.5 197.6 |
romm 16 0.020 0.339 1.07 | 0.5 25.5 100.5 |
dawarich 20 0.000 0.313 3.01 | 0.1 5.3 104.4 |
immich-machine-learning 32 0.003 0.243 0.42 | 0.2 294.7 295.7 |
immich-redis 32 0.017 0.144 0.16 | 0.5 3.9 4.2 |
nextcloud-db 16 0.000 0.137 0.43 | 0.0 4.9 5.8 |
nextcloud 16 0.000 0.099 0.33 | 0.0 21.7 86.7 |
paperless-webserver 21 0.005 0.073 0.08 | 0.0 137.4 139.7 |
paperless-postgres 22 0.000 0.048 0.06 | 0.0 4.6 5.0 |
kimai-db 19 0.000 0.023 0.15 | 0.0 8.4 38.8 |
dawarich-sidekiq 19 0.020 0.023 0.02 | 0.1 0.2 0.2 |
claper 20 0.019 0.020 0.02 | 0.5 0.7 0.9 |
docmost 22 0.020 0.020 0.02 | 0.0 0.2 0.3 |
wanderer 19 0.002 0.015 0.02 | 0.0 0.0 0.0 |
cloudflared 43863 0.000 0.013 1.33 | 0.2 0.3 8.3 | resets=11
adventurelog 22 0.000 0.013 3.24 | 0.0 0.1 65.8 |
grafana 20 0.000 0.012 0.12 | 1.0 1.9 17.3 |
wanderer-db 19 0.007 0.011 0.01 | 0.0 3.7 3.7 |
dawarich-redis 20 0.009 0.011 0.01 | 0.4 3.7 3.8 |
claper-postgres 20 0.011 0.011 0.01 | 0.2 0.5 4.4 |
outline 19 0.008 0.008 0.01 | 0.0 7.4 7.7 |
nextcloud-redis 16 0.000 0.008 0.02 | 0.4 0.7 1.4 |
romm-redis 16 0.006 0.007 0.01 | 0.5 3.7 3.8 |
docmost-redis 23 0.005 0.005 0.07 | 0.4 0.7 3.9 |
dawarich-db 20 0.001 0.005 0.03 | 0.0 0.1 0.1 |
bookstack-db 20 0.001 0.005 0.04 | 0.0 4.7 4.8 |
grocy 19 0.000 0.005 0.01 | 0.0 0.0 0.0 |
bookstack 20 0.000 0.005 0.04 | 0.0 7.9 47.2 |
zipline 20 0.003 0.003 0.00 | 0.0 0.0 0.0 |
paperless-redis 22 0.002 0.003 0.00 | 0.4 0.4 0.4 |
outline-redis 19 0.003 0.003 0.00 | 0.4 0.4 0.5 |
kimai 19 0.000 0.003 0.02 | 0.0 2.4 24.3 |
zipline-postgres 20 0.003 0.003 0.00 | 0.0 0.0 0.0 |
qbittorrent 11759 0.001 0.003 0.01 | 0.1 0.2 97.5 | resets=25
felhom-controller 43863 0.000 0.003 1.73 | 0.1 15.5 122.0 | resets=92
bentopdf 18 0.000 0.002 0.00 | 0.0 0.0 0.0 |
emby 19 0.001 0.002 0.01 | 0.0 0.1 0.1 |
homepage 20 0.000 0.002 0.00 | 0.0 0.0 0.0 |
sonarr 11756 0.000 0.001 0.01 | 0.1 0.3 74.7 | resets=25
grimmory-db 24 0.000 0.001 0.35 | 0.0 5.0 100.2 |
grimmory 23 0.001 0.001 0.00 | 5.7 25.3 35.0 |
lubelogger 19 0.000 0.001 0.00 | 0.0 0.2 0.6 |
radarr 11756 0.000 0.001 0.17 | 0.0 0.6 79.9 | resets=25
glance 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
metube 20 0.000 0.001 0.00 | 0.0 1.3 26.1 |
actualbudget 18 0.000 0.001 0.00 | 0.0 0.0 0.0 |
privatebin 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
plex 18 0.000 0.001 0.00 | 0.7 0.8 0.8 |
papra 20 0.000 0.001 0.00 | 0.0 6.8 7.5 |
home-assistant 16 0.000 0.001 0.00 | 0.0 19.8 79.0 |
wanderer-search 19 0.000 0.001 0.00 | 0.1 0.1 0.1 |
karakeep 19 0.001 0.001 0.00 | 1.8 2.0 2.0 |
romm-db 16 0.001 0.001 0.00 | 0.0 4.6 4.7 |
radicale 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
gramps-web 20 0.000 0.001 0.00 | 0.0 0.0 0.0 |
tandoor 19 0.000 0.001 0.00 | 0.1 4.7 4.8 |
karakeep-chrome 19 0.001 0.001 0.00 | 0.0 0.0 0.0 |
outline-postgres 19 0.000 0.001 0.00 | 0.0 0.4 0.7 |
seerr 18 0.000 0.001 0.00 | 0.0 0.0 0.0 |
n8n 22 0.000 0.000 0.00 | 0.8 1.0 110.6 |
termix 20 0.000 0.000 0.00 | 0.0 0.4 2.3 |
tandoor-postgres 19 0.000 0.000 0.00 | 0.4 1.5 4.1 |
audiobookshelf 19 0.000 0.000 0.00 | 0.0 0.0 0.1 |
adventurelog-postgres 22 0.000 0.000 0.02 | 0.0 3.8 33.6 |
ghost 21 0.000 0.000 0.00 | 0.0 0.0 0.0 |
transmission 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
komga 19 0.000 0.000 0.00 | 0.3 6.8 10.6 |
calcom 18 0.000 0.000 0.00 | 0.0 3.4 22.3 |
vaultwarden 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
sparkyfitness 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
homebox 19 0.000 0.000 0.00 | 0.1 0.2 0.2 |
docmost-postgres 23 0.000 0.000 0.01 | 0.0 3.8 73.9 |
vikunja 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
adventurelog-frontend 22 0.000 0.000 0.00 | 0.0 0.0 0.0 |
rallly 20 0.000 0.000 0.00 | 0.0 5.6 110.3 |
calibre-web 17 0.000 0.000 0.00 | 0.0 21.3 106.4 |
wishlist 20 0.000 0.000 0.00 | 0.0 6.3 125.1 |
prowlarr 11738 0.000 0.000 0.00 | 0.0 0.1 77.0 | resets=26
opengist 43862 0.000 0.000 0.01 | 0.0 0.0 6.0 | resets=107
onlyoffice 22 0.000 0.000 0.00 | 0.0 130.5 147.5 |
sabnzbd 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
gitea 19 0.000 0.000 0.00 | 0.0 0.4 4.0 |
navidrome 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
code-server 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
uptime-kuma 19 0.000 0.000 0.00 | 0.7 0.7 0.7 |
nzbget 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
gokapi 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
recipe-importer 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
crafty-controller 19 0.000 0.000 0.00 | 0.1 8.3 8.5 |
mealie 20 0.000 0.000 0.00 | 0.1 0.1 0.1 |
rallly-postgres 20 0.000 0.000 0.00 | 0.0 3.7 3.8 |
traefik 43863 0.000 0.000 0.96 | 0.0 3.6 17.9 | resets=13
sparkyfitness-server 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
sparkyfitness-db 19 0.000 0.000 0.00 | 0.0 3.8 3.8 |
karakeep-meilisearch 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
filebrowser 43863 0.000 0.000 0.74 | 0.0 0.0 15.7 | resets=16
calcom-postgres 18 0.000 0.000 0.00 | 0.0 3.9 4.0 |
busiest UTC hours by bytes sent: 16h 16%, 05h 12%, 09h 6%, 02h 5%
=== tester-1: 2026-10-04 19:41 -> 2026-10-23 18:08 UTC (18.9 d), 294433 rows
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
qbittorrent 3857 0.007 0.133 42.40 | 0.1 2.6 92.3 | resets=20
transmission 3857 0.001 0.013 57.43 | 0.1 1.6 82.5 | resets=19
paperless-webserver 26710 0.005 0.013 0.09 | 0.1 0.7 508.1 | resets=63
cloudflared 26751 0.000 0.013 1.63 | 0.4 1.2 29.4 | resets=6
prowlarr 3725 0.000 0.012 0.26 | 0.1 0.3 101.8 | resets=8
sonarr 3855 0.001 0.004 1.21 | 0.1 4.5 131.6 | resets=20
radarr 3855 0.001 0.003 1.52 | 0.1 4.3 103.3 | resets=19
paperless-redis 26740 0.001 0.003 0.01 | 0.7 4.2 27.5 | resets=68
felhom-controller 26751 0.000 0.003 37.47 | 0.5 46.4 422.6 | resets=62
plex 3739 0.001 0.002 1.07 | 0.3 4.3 198.4 | resets=17
bookstack-db 26729 0.001 0.001 0.03 | 0.0 5.7 130.6 | resets=66
plex-k6 514 0.001 0.001 0.01 | 0.4 5.4 7.2 |
bookstack 26722 0.000 0.001 0.04 | 0.0 0.1 72.5 | resets=65
privatebin 26744 0.000 0.001 0.00 | 0.0 0.1 14.7 | resets=64
paperless-postgres 26740 0.000 0.001 0.02 | 0.0 4.4 65.8 | resets=68
traefik 26751 0.000 0.000 1.56 | 0.0 3.4 100.6 | resets=3
jellyfin 3648 0.000 0.000 2.83 | 0.0 0.2 181.5 | resets=19
xfp-echo 1 0.000 0.000 0.00 | 0.0 0.0 0.0 |
vaultwarden 1 0.000 0.000 0.00 | 0.0 0.0 0.0 |
filebrowser 26743 0.000 0.000 0.01 | 0.0 0.0 80.1 | resets=22
busiest UTC hours by bytes sent: 07h 13%, 18h 11%, 02h 9%, 17h 9%
@@ -0,0 +1,22 @@
== demo-hp
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
kimai 13 0 (-) 0 0 (-)
paperless-webserver 6 1 (09-23 04:56) 0 0 (-)
immich-machine-learning 4 4 (10-14 00:54) 0 0 (-)
romm 3 2 (10-23 15:36) 0 0 (-)
adventurelog 2 1 (09-30 11:29) 3 0 (-)
adventurelog-postgres 0 0 (-) 3 0 (-)
felhom-controller 3 1 (09-23 11:37) 2 2 (09-28 13:44)
immich-server 2 0 (-) 3 2 (10-14 00:53)
== demo-felhom
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
immich-server 2 0 (-) 8 1 (10-13 05:32)
immich-machine-learning 7 7 (10-13 16:39) 0 0 (-)
paperless-webserver 6 0 (-) 0 0 (-)
onlyoffice 4 0 (-) 0 0 (-)
== tester-1
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
plex 10 5 (10-14 17:20) 1 0 (-)
qbittorrent 2 0 (-) 9 3 (10-14 18:08)
sonarr 9 0 (-) 2 0 (-)
transmission 1 0 (-) 6 2 (10-16 07:17)
@@ -0,0 +1,90 @@
== demo-hp: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
2026-09-23 max1= 3.14 best5= 0.91 busy_min= 0 peak_at=02:17Z
2026-09-24 max1= 3.04 best5= 0.73 busy_min= 0 peak_at=02:17Z
2026-09-25 max1= 2.13 best5= 0.67 busy_min= 0 peak_at=02:18Z
2026-09-26 max1= 1.77 best5= 0.63 busy_min= 0 peak_at=02:18Z
2026-09-27 max1= 2.04 best5= 0.75 busy_min= 0 peak_at=02:18Z
2026-09-28 max1= 17.79 best5= 6.89 busy_min= 3 peak_at=13:43Z
2026-09-29 max1= 2.45 best5= 0.65 busy_min= 0 peak_at=02:17Z
2026-09-30 max1= 2.62 best5= 0.76 busy_min= 0 peak_at=02:18Z
2026-10-01 max1= 1.02 best5= 0.54 busy_min= 0 peak_at=02:18Z
2026-10-02 max1= 2.02 best5= 0.62 busy_min= 0 peak_at=02:18Z
2026-10-03 max1= 2.01 best5= 0.60 busy_min= 0 peak_at=15:25Z
2026-10-04 max1= 2.08 best5= 0.42 busy_min= 0 peak_at=02:17Z
2026-10-05 max1= 2.36 best5= 0.48 busy_min= 0 peak_at=05:26Z
2026-10-06 max1= 1.58 best5= 0.32 busy_min= 0 peak_at=02:17Z
2026-10-07 max1= 1.05 best5= 0.30 busy_min= 0 peak_at=02:16Z
2026-10-08 max1= 1.43 best5= 0.29 busy_min= 0 peak_at=02:17Z
2026-10-09 max1= 1.47 best5= 0.29 busy_min= 0 peak_at=02:17Z
2026-10-10 max1= 0.90 best5= 0.27 busy_min= 0 peak_at=02:16Z
2026-10-11 max1= 1.34 best5= 0.30 busy_min= 0 peak_at=02:16Z
2026-10-12 max1= 4.96 best5= 0.99 busy_min= 0 peak_at=02:17Z
2026-10-13 max1= 1.31 best5= 0.28 busy_min= 0 peak_at=02:16Z
2026-10-14 max1= 6.31 best5= 1.27 busy_min= 1 peak_at=02:18Z
2026-10-15 max1= 2.25 best5= 0.45 busy_min= 0 peak_at=02:17Z
2026-10-16 max1= 2.78 best5= 0.66 busy_min= 0 peak_at=20:55Z
2026-10-17 max1= 2.26 best5= 0.45 busy_min= 0 peak_at=02:17Z
2026-10-18 max1= 2.08 best5= 0.47 busy_min= 0 peak_at=02:18Z
2026-10-19 max1= 16.95 best5= 5.08 busy_min= 2 peak_at=02:18Z
2026-10-20 max1= 1.20 best5= 0.48 busy_min= 0 peak_at=02:17Z
2026-10-21 max1= 2.27 best5= 0.46 busy_min= 0 peak_at=02:18Z
2026-10-22 max1= 2.13 best5= 0.52 busy_min= 0 peak_at=02:18Z
2026-10-23 max1= 1.37 best5= 0.48 busy_min= 0 peak_at=02:18Z
nights with a push (best5>2): 2; best5 min/median/max = 5.1/6.9/6.9 Mbit/s
== demo-felhom: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
2026-09-23 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=02:15Z
2026-09-24 max1= 0.06 best5= 0.01 busy_min= 0 peak_at=02:15Z
2026-09-25 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-09-26 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-09-27 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=00:19Z
2026-09-28 max1= 0.03 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-09-29 max1= 0.03 best5= 0.01 busy_min= 0 peak_at=02:15Z
2026-09-30 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-10-01 max1= 0.07 best5= 0.01 busy_min= 0 peak_at=19:54Z
2026-10-02 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-10-03 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=02:16Z
2026-10-04 max1= 0.02 best5= 0.00 busy_min= 0 peak_at=06:58Z
2026-10-05 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=05:21Z
2026-10-06 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=17:34Z
2026-10-07 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=02:15Z
2026-10-08 max1= 0.09 best5= 0.02 busy_min= 0 peak_at=13:04Z
2026-10-09 max1= 0.10 best5= 0.02 busy_min= 0 peak_at=05:34Z
2026-10-10 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=02:15Z
2026-10-11 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=10:46Z
2026-10-12 max1= 0.08 best5= 0.02 busy_min= 0 peak_at=06:18Z
2026-10-13 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=13:54Z
2026-10-14 max1= 0.07 best5= 0.02 busy_min= 0 peak_at=07:59Z
2026-10-15 max1= 0.14 best5= 0.03 busy_min= 0 peak_at=14:25Z
2026-10-16 max1= 0.84 best5= 0.17 busy_min= 0 peak_at=02:15Z
2026-10-17 max1= 0.49 best5= 0.10 busy_min= 0 peak_at=02:15Z
2026-10-18 max1= 0.33 best5= 0.07 busy_min= 0 peak_at=02:16Z
2026-10-19 max1= 1.73 best5= 0.35 busy_min= 0 peak_at=02:16Z
2026-10-20 max1= 0.23 best5= 0.07 busy_min= 0 peak_at=02:16Z
2026-10-21 max1= 0.36 best5= 0.07 busy_min= 0 peak_at=02:16Z
2026-10-22 max1= 0.52 best5= 0.11 busy_min= 0 peak_at=02:15Z
2026-10-23 max1= 0.48 best5= 0.10 busy_min= 0 peak_at=02:15Z
== tester-1: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
2026-10-04 max1= 0.17 best5= 0.14 busy_min= 0 peak_at=19:55Z
2026-10-05 max1= 1.44 best5= 0.29 busy_min= 0 peak_at=03:10Z
2026-10-06 max1= 0.64 best5= 0.13 busy_min= 0 peak_at=02:16Z
2026-10-07 max1= 0.64 best5= 0.13 busy_min= 0 peak_at=02:15Z
2026-10-08 max1= 0.42 best5= 0.08 busy_min= 0 peak_at=02:16Z
2026-10-09 max1= 0.48 best5= 0.10 busy_min= 0 peak_at=02:16Z
2026-10-10 max1= 0.84 best5= 0.17 busy_min= 0 peak_at=02:16Z
2026-10-11 max1= 0.46 best5= 0.09 busy_min= 0 peak_at=02:15Z
2026-10-12 max1= 1.71 best5= 0.34 busy_min= 0 peak_at=02:16Z
2026-10-13 max1= 0.99 best5= 0.20 busy_min= 0 peak_at=02:15Z
2026-10-14 max1= 0.45 best5= 0.09 busy_min= 0 peak_at=02:16Z
2026-10-15 max1= 37.47 best5= 7.58 busy_min= 1 peak_at=02:17Z
2026-10-16 max1= 3.85 best5= 0.77 busy_min= 0 peak_at=02:17Z
2026-10-17 max1= 5.11 best5= 1.21 busy_min= 1 peak_at=02:16Z
2026-10-18 max1= 0.50 best5= 0.10 busy_min= 0 peak_at=02:16Z
2026-10-19 max1= 5.14 best5= 1.03 busy_min= 1 peak_at=02:16Z
2026-10-20 max1= 0.52 best5= 0.10 busy_min= 0 peak_at=02:16Z
2026-10-21 max1= 0.61 best5= 0.12 busy_min= 0 peak_at=02:16Z
2026-10-22 max1= 0.53 best5= 0.11 busy_min= 0 peak_at=02:16Z
2026-10-23 max1= 0.80 best5= 0.18 busy_min= 0 peak_at=02:16Z
nights with a push (best5>2): 1; best5 min/median/max = 7.6/7.6/7.6 Mbit/s
@@ -0,0 +1,6 @@
login 204 cookie
{"t": 0.2, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65136369198, "engine_tx": 65136367684, "cli": "65.5MB / 65.1GB"}
{"t": 6.5, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65152041285, "engine_tx": 65152039771, "cli": "65.9MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 19.9, "rate_engine_Mbit": 19.9}
{"t": 12.7, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65167646902, "engine_tx": 65167645388, "cli": "66.2MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 20.14, "rate_engine_Mbit": 20.14}
{"t": 19.0, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65183320503, "engine_tx": 65183318989, "cli": "66.6MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 19.9, "rate_engine_Mbit": 19.9}
{"t": 25.2, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65198925974, "engine_tx": 65198924460, "cli": "66.9MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 20.14, "rate_engine_Mbit": 20.14}
@@ -0,0 +1,10 @@
LISTEN 0 5 0.0.0.0:5001 0.0.0.0:*
login 204 cookie
{"t": 0.0, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 1559315619, "engine_tx": 1558369279, "cli": "2.94MB / 2.85GB"}
{"t": 6.0, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 9161893432, "engine_tx": 9161333668, "cli": "9.34MB / 10.5GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10136.77, "rate_engine_Mbit": 10137.29}
{"t": 12.1, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 17020963755, "engine_tx": 17019975365, "cli": "16.7MB / 18.7GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10306.98, "rate_engine_Mbit": 10306.42}
{"t": 18.1, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 25140960556, "engine_tx": 25139940620, "cli": "23.6MB / 26.5GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10826.66, "rate_engine_Mbit": 10826.62}
{"t": 24.2, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 33049634829, "engine_tx": 33048966257, "cli": "30.6MB / 34.4GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10372.03, "rate_engine_Mbit": 10372.49}
{"t": 30.2, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 41055773948, "engine_tx": 41055105508, "cli": "37.6MB / 42.3GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10674.85, "rate_engine_Mbit": 10674.85}
{"t": 36.3, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 48735514801, "engine_tx": 48734576785, "cli": "44.3MB / 50GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10071.79, "rate_engine_Mbit": 10071.44}
{"t": 42.3, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 56370706217, "engine_tx": 56370119481, "cli": "52.4MB / 58.8GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10180.26, "rate_engine_Mbit": 10180.72}
@@ -0,0 +1,9 @@
login 204 cookie
setUploadLimit (200, '')
uploadLimit now 2499584 prefs.up_limit 2499584
login 204 cookie
{"t": 0.0, "qb_up_speed": 16386315, "qb_up_data": 1346633728, "qb_tor_uploaded": 1346633728, "proc_tx": 1422747831, "engine_tx": 1422747831, "cli": "27.2MB / 1.42GB"}
{"t": 6.0, "qb_up_speed": 0, "qb_up_data": 1374387364, "qb_tor_uploaded": 1374387364, "proc_tx": 1434635707, "engine_tx": 1434635707, "cli": "27.6MB / 1.44GB", "rate_qb_data_Mbit": 37.0, "rate_proc_Mbit": 15.85, "rate_engine_Mbit": 15.85}
{"t": 12.1, "qb_up_speed": 2872275, "qb_up_data": 1392093322, "qb_tor_uploaded": 1391692420, "proc_tx": 1457229455, "engine_tx": 1457229455, "cli": "27.9MB / 1.46GB", "rate_qb_data_Mbit": 23.22, "rate_proc_Mbit": 29.63, "rate_engine_Mbit": 29.63}
{"t": 18.1, "qb_up_speed": 1975559, "qb_up_data": 1406540742, "qb_tor_uploaded": 1406540742, "proc_tx": 1471460371, "engine_tx": 1471460371, "cli": "28.2MB / 1.47GB", "rate_qb_data_Mbit": 19.26, "rate_proc_Mbit": 18.97, "rate_engine_Mbit": 18.97}
{"t": 24.2, "qb_up_speed": 3254241, "qb_up_data": 1423113257, "qb_tor_uploaded": 1420865709, "proc_tx": 1488294067, "engine_tx": 1488294067, "cli": "28.5MB / 1.49GB", "rate_qb_data_Mbit": 21.73, "rate_proc_Mbit": 22.08, "rate_engine_Mbit": 22.08}
@@ -0,0 +1,11 @@
login 204 cookie
{"t": 0.0, "qb_up_speed": 16266049, "qb_up_data": 117833728, "qb_tor_uploaded": 117701484, "proc_tx": 123161204, "engine_tx": 123161204, "cli": "6.74MB / 140MB"}
{"t": 6.0, "qb_up_speed": 16384087, "qb_up_data": 216137728, "qb_tor_uploaded": 216137728, "proc_tx": 225760308, "engine_tx": 225760308, "cli": "8.38MB / 243MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 136.8, "rate_engine_Mbit": 136.8}
{"t": 12.1, "qb_up_speed": 16387266, "qb_up_data": 314441728, "qb_tor_uploaded": 314441728, "proc_tx": 328359626, "engine_tx": 328359626, "cli": "10.1MB / 346MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 134.56, "rate_engine_Mbit": 134.56}
{"t": 18.1, "qb_up_speed": 16391332, "qb_up_data": 412745728, "qb_tor_uploaded": 412745728, "proc_tx": 431576783, "engine_tx": 431540471, "cli": "11.8MB / 450MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 137.62, "rate_engine_Mbit": 137.57}
{"t": 24.2, "qb_up_speed": 16377743, "qb_up_data": 511049728, "qb_tor_uploaded": 511049728, "proc_tx": 536615588, "engine_tx": 536559607, "cli": "13.4MB / 556MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 137.76, "rate_engine_Mbit": 137.73}
{"t": 30.2, "qb_up_speed": 16387474, "qb_up_data": 609353728, "qb_tor_uploaded": 609353728, "proc_tx": 641957804, "engine_tx": 641903336, "cli": "14.9MB / 660MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 140.46, "rate_engine_Mbit": 140.46}
{"t": 36.3, "qb_up_speed": 16383934, "qb_up_data": 707657728, "qb_tor_uploaded": 707657728, "proc_tx": 746552576, "engine_tx": 746552576, "cli": "16.8MB / 764MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 137.17, "rate_engine_Mbit": 137.24}
{"t": 42.3, "qb_up_speed": 16392852, "qb_up_data": 805961728, "qb_tor_uploaded": 805961728, "proc_tx": 849903723, "engine_tx": 849903723, "cli": "18.3MB / 867MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 137.8, "rate_engine_Mbit": 137.8}
{"t": 48.4, "qb_up_speed": 16382132, "qb_up_data": 904265728, "qb_tor_uploaded": 904265728, "proc_tx": 952502020, "engine_tx": 952502020, "cli": "19.9MB / 970MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 134.56, "rate_engine_Mbit": 134.56}
{"t": 54.4, "qb_up_speed": 16376106, "qb_up_data": 1002569728, "qb_tor_uploaded": 1002569728, "proc_tx": 1055099637, "engine_tx": 1055099637, "cli": "21.5MB / 1.07GB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 136.8, "rate_engine_Mbit": 136.8}
@@ -0,0 +1,8 @@
{'arguments': {}, 'result': 'success'}
{'speed-limit-up': 2500, 'speed-limit-up-enabled': True}
{"t": 0.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157242728}
{"t": 5.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157244479, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 10.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157246230, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 15.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157248133, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157249884, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 25.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157251635, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
@@ -0,0 +1,10 @@
{"t": 0.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157256302}
{"t": 10.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157258205, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157261594, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 30.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157263345, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 40.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157265096, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 50.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157268485, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 60.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157271188, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 70.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157272939, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 80.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157276328, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 90.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157278588, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
@@ -0,0 +1,15 @@
login 204 cookie
/api/v2/torrents/stop (200, '')
{'arguments': {}, 'result': 'success'}
{"t": 0.0, "tr_upspeed": 40648163, "tr_cur_up": 1528429958, "tr_tor_up": 1528429958, "tx": 1534546958}
{"t": 5.0, "tr_upspeed": 65588000, "tr_cur_up": 1832873057, "tr_tor_up": 1832873057, "tx": 1839315356, "rate_tr_Mbit": 487.11, "rate_container_Mbit": 487.63}
{"t": 10.0, "tr_upspeed": 23587084, "tr_cur_up": 2076811226, "tr_tor_up": 2076811226, "tx": 2083512878, "rate_tr_Mbit": 390.3, "rate_container_Mbit": 390.72}
{"t": 15.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157213873, "rate_tr_Mbit": 115.8, "rate_container_Mbit": 117.92}
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157215624, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 25.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157218947, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.01}
{"t": 30.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157220698, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 35.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157222449, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 40.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157224200, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 45.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157225951, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 50.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157227702, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
{"t": 55.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157231243, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.01}
@@ -0,0 +1,2 @@
# usage: gen.sh <dir> ; writes <dir>/uwtest.bin, 2 GiB, identical on every machine (AES-CTR keystream of a fixed key)
mkdir -p "$1"; openssl enc -aes-128-ctr -nosalt -K 00112233445566778899aabbccddeeff -iv 0 -in /dev/zero 2>/dev/null | head -c 2147483648 > "$1/uwtest.bin"; sha256sum "$1/uwtest.bin" | cut -c1-16
@@ -0,0 +1,30 @@
import sqlite3,sys,collections,datetime,statistics
def pct(a,p):
if not a: return 0
a=sorted(a); k=(len(a)-1)*p/100; f=int(k); c=min(f+1,len(a)-1); return a[f]+(a[c]-a[f])*(k-f)
for box in sys.argv[1:]:
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
lo,hi,n=db.execute('select min(ts),max(ts),count(*) from container_metrics').fetchone()
print(f'\n=== {box.rsplit("/",1)[1]}: {datetime.datetime.utcfromtimestamp(lo):%Y-%m-%d %H:%M} -> {datetime.datetime.utcfromtimestamp(hi):%Y-%m-%d %H:%M} UTC ({(hi-lo)/86400:.1f} d), {n} rows')
rows=collections.defaultdict(list)
for ts,name,cpu,tx in db.execute('select ts,container_name,cpu_percent,net_tx_bytes from container_metrics order by ts'):
rows[name].append((ts,cpu,tx))
hour=collections.defaultdict(float)
out=[]
for name,r in rows.items():
rates=[];resets=0;zero=all((x[2] or 0)==0 for x in r)
for (t0,_,a),(t1,_,b) in zip(r,r[1:]):
if a is None or b is None or t1-t0<=0 or t1-t0>180: continue
d=b-a
if d<0: resets+=1; continue
mbps=d*8/(t1-t0)/1e6; rates.append(mbps)
hour[datetime.datetime.utcfromtimestamp(t1).hour]+=d
cpus=[x[1] for x in r]
out.append((pct(rates,95),name,len(r),pct(rates,50),pct(rates,95),max(rates or [0]),pct(cpus,50),pct(cpus,95),max(cpus),zero,resets))
print(f'{"container":32} {"n":>6} {"tx p50":>8} {"p95":>8} {"max":>8} Mbit/s | {"cpu p50":>7} {"p95":>7} {"max":>7} %core | notes')
for o in sorted(out,reverse=True):
_,name,nn,a,b,c,d,e,f,zero,res=o
note=('tx counter always 0 (host network?)' if zero else '')+(f' resets={res}' if res else '')
print(f'{name[:32]:32} {nn:6} {a:8.3f} {b:8.3f} {c:8.2f} | {d:7.1f} {e:7.1f} {f:7.1f} | {note}')
tot=sum(hour.values()) or 1
print('busiest UTC hours by bytes sent:',', '.join(f'{h:02d}h {100*v/tot:.0f}%' for h,v in sorted(hour.items(),key=lambda x:-x[1])[:4]))
@@ -0,0 +1,21 @@
import sqlite3,sys,collections,datetime
for box in sys.argv[1:]:
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
r=db.execute("select ts,net_tx_bytes from container_metrics where container_name='felhom-controller' order by ts").fetchall()
per=collections.defaultdict(list)
for (t0,a),(t1,b) in zip(r,r[1:]):
if a is None or b is None or b<a or not 0<t1-t0<=180: continue
d=datetime.datetime.utcfromtimestamp(t1)
per[d.date()].append((t1,(b-a)*8/(t1-t0)/1e6))
print(f'\n== {box.rsplit("/",1)[1]}: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak')
peaks=[]
for day in sorted(per):
v=per[day]; rates=[x[1] for x in v]
best5=max((sum(rates[i:i+5])/5 for i in range(max(1,len(rates)-4))),default=0)
m=max(v,key=lambda x:x[1])
busy=sum(1 for x in rates if x>5)
peaks.append(best5)
print(f'{day} max1={m[1]:6.2f} best5={best5:6.2f} busy_min={busy:4d} peak_at={datetime.datetime.utcfromtimestamp(m[0]):%H:%M}Z')
big=[p for p in peaks if p>2]
if big:
big.sort(); print(f'nights with a push (best5>2): {len(big)}; best5 min/median/max = {big[0]:.1f}/{big[len(big)//2]:.1f}/{big[-1]:.1f} Mbit/s')
@@ -0,0 +1,10 @@
#!/bin/bash
# Waits until 02:08Z on 2026-10-24, then reads the controller container's eth0 tx bytes once a second for 22 minutes on
# each of the three test boxes (read only: /proc/<pid>/net/dev inside guest 9201). Output: one file per box.
OUT=$1
target=$(date -u -d '2026-10-24T02:08:00Z' +%s); now=$(date -u +%s); [ $target -gt $now ] && sleep $((target-now))
INNER='P=$(docker inspect -f {{.State.Pid}} felhom-controller); end=$(( $(date +%s) + 1320 )); while [ $(date +%s) -lt $end ]; do echo "$(date +%s.%N | cut -c1-14) $(awk "/eth0:/{print \$10}" /proc/$P/net/dev)"; sleep 1; done'
for pair in "demo-hp:demo-hp" "felhom-pve:demo-felhom" "root@192.168.0.154:tester-1"; do h=${pair%%:*}; n=${pair##*:}
ssh -o BatchMode=yes -o ServerAliveInterval=30 $h "pct exec 9201 -- bash -c '$INNER'" > "$OUT/night-push-1s-$n.txt" 2>&1 &
done
wait; echo "done $(date -u +%FT%TZ)"
@@ -0,0 +1,89 @@
# Runs INSIDE guest 9202. Asks qBittorrent's own Web API and reads the container's exact byte counters.
# The admin password is read from the container env at use (as the controller does) and never printed.
import json, subprocess, sys, time, urllib.request, urllib.parse, http.client, socket, re, uuid
def sh(*a): return subprocess.run(a, capture_output=True, text=True).stdout.strip()
C = sys.argv[1] if len(sys.argv) > 1 else 'qbittorrent'
IP = sh('docker', 'inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}', C).split()[0]
PID = sh('docker', 'inspect', '-f', '{{.State.Pid}}', C)
CID = sh('docker', 'inspect', '-f', '{{.Id}}', C)
BASE = f'http://{IP}:8080'
PW = sh('docker', 'exec', C, 'printenv', 'QBT_ADMIN_PASSWORD')
def req(method, path, data=None, hdr=None, raw=False):
h = {'Referer': BASE}
h.update(hdr or {})
if COOKIE: h['Cookie'] = COOKIE
r = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
with urllib.request.urlopen(r, timeout=20) as resp:
b = resp.read(); return resp.status, (b if raw else (json.loads(b) if b and b[:1] in b'[{' else b.decode()))
except urllib.error.HTTPError as e:
return e.code, e.read().decode()[:200]
COOKIE = None
code, _ = None, None
r = urllib.request.Request(BASE + '/api/v2/auth/login', data=urllib.parse.urlencode({'username': 'admin', 'password': PW}).encode(),
headers={'Referer': BASE, 'Content-Type': 'application/x-www-form-urlencoded'}, method='POST')
with urllib.request.urlopen(r, timeout=20) as resp:
m = re.search(r'((?:QBT_)?SID[A-Za-z0-9_]*)=([^;]+)', '; '.join(resp.headers.get_all('Set-Cookie') or []))
print('login', resp.status, 'cookie' if m else 'NO COOKIE')
COOKIE = f'{m.group(1)}={m.group(2)}'
def netdev():
for line in open(f'/proc/{PID}/net/dev'):
if line.strip().startswith('eth0:'):
f = line.split(':', 1)[1].split(); return int(f[8])
def engine_stats():
s = socket.socket(socket.AF_UNIX); s.connect('/var/run/docker.sock')
s.sendall(f'GET /containers/{CID}/stats?stream=false&one-shot=true HTTP/1.0\r\n\r\n'.encode())
b = b''
while True:
x = s.recv(65536)
if not x: break
b += x
j = json.loads(b.split(b'\r\n\r\n', 1)[1])
return sum(n['tx_bytes'] for n in j.get('networks', {}).values()), j['cpu_stats']['cpu_usage']['total_usage']
def cli_stats():
return sh('docker', 'stats', '--no-stream', '--format', '{{.NetIO}}', C)
mode = sys.argv[2] if len(sys.argv) > 2 else 'info'
if mode == 'info':
print('version', req('GET', '/api/v2/app/version')[1], 'webapi', req('GET', '/api/v2/app/webapiVersion')[1])
print('transfer/info', req('GET', '/api/v2/transfer/info')[1])
p = req('GET', '/api/v2/app/preferences')[1]
print('prefs', {k: p.get(k) for k in ('up_limit', 'dl_limit', 'alt_up_limit', 'alt_dl_limit', 'scheduler_enabled', 'limit_tcp_overhead', 'limit_lan_peers', 'limit_utp_rate', 'max_uploads', 'enable_upload_suggestions')})
print('uploadLimit', req('GET', '/api/v2/transfer/uploadLimit')[1], 'speedLimitsMode', req('GET', '/api/v2/transfer/speedLimitsMode')[1])
elif mode == 'add':
bnd = uuid.uuid4().hex
tor = open('/root/uwtest.torrent', 'rb').read()
parts = []
for k, v in (('savepath', '/mnt/felhom-drives/scratch_hdd/userdata/media/torrents'), ('skip_checking', 'true'), ('category', '')):
parts.append(f'--{bnd}\r\nContent-Disposition: form-data; name="{k}"\r\n\r\n{v}\r\n'.encode())
parts.append(f'--{bnd}\r\nContent-Disposition: form-data; name="torrents"; filename="uwtest.torrent"\r\nContent-Type: application/x-bittorrent\r\n\r\n'.encode() + tor + b'\r\n')
parts.append(f'--{bnd}--\r\n'.encode())
print('add', req('POST', '/api/v2/torrents/add', b''.join(parts), {'Content-Type': f'multipart/form-data; boundary={bnd}'}))
time.sleep(3)
print('torrents', [(t['name'], t['state'], t['progress']) for t in req('GET', '/api/v2/torrents/info')[1]])
elif mode == 'setlimit':
print('setUploadLimit', req('POST', '/api/v2/transfer/setUploadLimit', urllib.parse.urlencode({'limit': sys.argv[3]}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
print('uploadLimit now', req('GET', '/api/v2/transfer/uploadLimit')[1], 'prefs.up_limit', req('GET', '/api/v2/app/preferences')[1].get('up_limit'))
elif mode == 'setpref':
print('setPreferences', req('POST', '/api/v2/app/setPreferences', urllib.parse.urlencode({'json': json.dumps({'up_limit': int(sys.argv[3])})}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
print('uploadLimit now', req('GET', '/api/v2/transfer/uploadLimit')[1], 'prefs.up_limit', req('GET', '/api/v2/app/preferences')[1].get('up_limit'))
elif mode == 'peer':
print('addPeers', req('POST', '/api/v2/torrents/addPeers', urllib.parse.urlencode({'hashes': 'all', 'peers': sys.argv[3]}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
print('reannounce', req('POST', '/api/v2/torrents/reannounce', urllib.parse.urlencode({'hashes': 'all'}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
elif mode == 'post':
print(sys.argv[3], req('POST', sys.argv[3], urllib.parse.urlencode(json.loads(sys.argv[4])).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
elif mode == 'sample':
dur, step = int(sys.argv[3]), int(sys.argv[4])
t0 = time.time(); prev = None
while time.time() - t0 <= dur:
ti = req('GET', '/api/v2/transfer/info')[1]
tor = req('GET', '/api/v2/torrents/info')[1]
etx, ecpu = engine_stats()
row = {'t': round(time.time() - t0, 1), 'qb_up_speed': ti['up_info_speed'], 'qb_up_data': ti['up_info_data'],
'qb_tor_uploaded': sum(x.get('uploaded', 0) for x in tor), 'proc_tx': netdev(), 'engine_tx': etx, 'cli': cli_stats()}
if prev:
dt = row['t'] - prev['t']
row['rate_qb_data_Mbit'] = round((row['qb_up_data'] - prev['qb_up_data']) * 8 / dt / 1e6, 2)
row['rate_proc_Mbit'] = round((row['proc_tx'] - prev['proc_tx']) * 8 / dt / 1e6, 2)
row['rate_engine_Mbit'] = round((row['engine_tx'] - prev['engine_tx']) * 8 / dt / 1e6, 2)
print(json.dumps(row), flush=True); prev = row
time.sleep(step)
@@ -0,0 +1,32 @@
import sqlite3,sys,collections
def runs(r, pred):
best=cur=0; bt=None; prev=None
for t,v in r:
if pred(v) and prev is not None and t-prev<=180: cur+=1
elif pred(v): cur=1
else: cur=0
if cur>best: best=cur; bt=t
prev=t
return best,bt
import datetime
for box in sys.argv[1:]:
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
nc=None
rows=collections.defaultdict(list)
for ts,name,cpu,tx in db.execute('select ts,container_name,cpu_percent,net_tx_bytes from container_metrics order by ts'):
rows[name].append((ts,cpu,tx))
print('==',box.rsplit('/',1)[1])
out=[]
for name,r in rows.items():
cpu=[(t,c) for t,c,_ in r]
tx=[]
for (t0,_,a),(t1,_,b) in zip(r,r[1:]):
if a is None or b is None or b<a or not 0<t1-t0<=180: continue
tx.append((t1,(b-a)*8/(t1-t0)/1e6))
c150,ct=runs(cpu,lambda v:v>=150); c80,_=runs(cpu,lambda v:v>=80)
u5,ut=runs(tx,lambda v:v>=5); u1,_=runs(tx,lambda v:v>=1)
if max(c80,u1)>=3: out.append((name,c80,c150,ct,u1,u5,ut))
print(f'{"container":28} cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]')
for name,c80,c150,ct,u1,u5,ut in sorted(out,key=lambda x:-max(x[1],x[4])):
f=lambda t: datetime.datetime.utcfromtimestamp(t).strftime('%m-%d %H:%M') if t else '-'
print(f'{name[:28]:28} {c80:8} {c150:9} ({f(ct)}) {u1:9} {u5:9} ({f(ut)})')
@@ -0,0 +1,21 @@
# Minimal BitTorrent HTTP tracker for a closed scratch test (no internet). Compact peer lists.
import http.server, urllib.parse, socket, struct, time
peers = {} # info_hash -> {(ip,port): last_seen}
def benc(o):
if isinstance(o, int): return b"i%de" % o
if isinstance(o, bytes): return b"%d:%s" % (len(o), o)
if isinstance(o, str): return benc(o.encode())
if isinstance(o, dict): return b"d" + b"".join(benc(k) + benc(v) for k, v in sorted(o.items())) + b"e"
class H(http.server.BaseHTTPRequestHandler):
def do_GET(self):
q = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query.encode('latin-1').decode('latin-1'), encoding='latin-1')
ih = q.get('info_hash', [''])[0]; port = int(q.get('port', ['0'])[0]); ip = self.client_address[0]
d = peers.setdefault(ih, {}); now = time.time()
if q.get('event', [''])[0] == 'stopped': d.pop((ip, port), None)
else: d[(ip, port)] = now
lst = b"".join(socket.inet_aton(i) + struct.pack(">H", p) for (i, p), t in d.items() if now - t < 600 and (i, p) != (ip, port))
body = benc({"interval": 30, "peers": lst})
print(time.strftime('%H:%M:%S'), 'announce', ip, port, q.get('event', [''])[0], 'peers', len(lst) // 6, flush=True)
self.send_response(200); self.send_header('Content-Length', str(len(body))); self.end_headers(); self.wfile.write(body)
def log_message(self, *a): pass
http.server.ThreadingHTTPServer(('0.0.0.0', 6969), H).serve_forever()
@@ -0,0 +1,47 @@
# Runs INSIDE guest 9202. Transmission's own RPC numbers vs the container's exact tx bytes. Password read at use, never printed.
import json, subprocess, sys, time, urllib.request, base64
def sh(*a): return subprocess.run(a, capture_output=True, text=True).stdout.strip()
C = 'transmission'
IP = sh('docker', 'inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}', C).split()[0]
PID = sh('docker', 'inspect', '-f', '{{.State.Pid}}', C)
PW = sh('docker', 'exec', C, 'printenv', 'PASS')
AUTH = 'Basic ' + base64.b64encode(('admin:' + PW).encode()).decode()
SID = ''
def rpc(method, args=None):
global SID
for _ in range(2):
r = urllib.request.Request(f'http://{IP}:9091/transmission/rpc', data=json.dumps({'method': method, 'arguments': args or {}}).encode(),
headers={'Authorization': AUTH, 'Host': 'transmission:9091', 'X-Transmission-Session-Id': SID, 'Content-Type': 'application/json'}, method='POST')
try:
with urllib.request.urlopen(r, timeout=20) as resp: return json.loads(resp.read())
except urllib.error.HTTPError as e:
if e.code == 409: SID = e.headers.get('X-Transmission-Session-Id'); continue
return {'http': e.code}
def tx():
for line in open(f'/proc/{PID}/net/dev'):
if line.strip().startswith('eth0:'): return int(line.split(':', 1)[1].split()[8])
mode = sys.argv[1]
if mode == 'info':
g = rpc('session-get')['arguments']
print('version', g.get('version'), {k: g.get(k) for k in ('speed-limit-up', 'speed-limit-up-enabled', 'alt-speed-up', 'alt-speed-enabled', 'alt-speed-time-enabled', 'units')})
print('session-stats', rpc('session-stats')['arguments'])
elif mode == 'add':
meta = base64.b64encode(open('/root/uwtest.torrent', 'rb').read()).decode()
print(rpc('torrent-add', {'metainfo': meta, 'download-dir': '/mnt/felhom-drives/scratch_hdd/userdata/media/transmission', 'paused': False}))
time.sleep(5); rpc('torrent-verify', {}); time.sleep(20)
print(rpc('torrent-get', {'fields': ['name', 'status', 'percentDone', 'error', 'errorString']}))
elif mode == 'limit':
print(rpc('session-set', {'speed-limit-up': int(sys.argv[2]), 'speed-limit-up-enabled': True}))
g = rpc('session-get')['arguments']; print({k: g.get(k) for k in ('speed-limit-up', 'speed-limit-up-enabled')})
elif mode == 'reannounce':
print(rpc('torrent-reannounce', {}))
elif mode == 'sample':
dur, step = int(sys.argv[2]), int(sys.argv[3]); t0 = time.time(); prev = None
while time.time() - t0 <= dur:
st = rpc('session-stats')['arguments']; tg = rpc('torrent-get', {'fields': ['uploadedEver', 'rateUpload']})['arguments']['torrents']
row = {'t': round(time.time() - t0, 1), 'tr_upspeed': st['uploadSpeed'], 'tr_cur_up': st['current-stats']['uploadedBytes'], 'tr_tor_up': sum(x['uploadedEver'] for x in tg), 'tx': tx()}
if prev:
dt = row['t'] - prev['t']
row['rate_tr_Mbit'] = round((row['tr_cur_up'] - prev['tr_cur_up']) * 8 / dt / 1e6, 2)
row['rate_container_Mbit'] = round((row['tx'] - prev['tx']) * 8 / dt / 1e6, 2)
print(json.dumps(row), flush=True); prev = row; time.sleep(step)
@@ -65,6 +65,15 @@ Two first mutations broke the build instead of a test (an unused variable) and w
compiler, failed. The first version of the file test passed with the VACUUM removed — the token row was the last cell on
its page, so the new row overwrote it; a second customer written after it (as on the live hub) made the test honest.
**Run on a private copy of the real database** (the 14:00Z snapshot, in a 0700 scratch folder, with a THROWAWAY key —
the real one was never read; a temporary test file, deleted, never committed): the start-up sealing sealed **8** tokens
in **3.15 s** (incl. the VACUUM), a second run sealed **0**, all **5 of 5** customers' `config_json` read back equal to
the original (JSON-equal), and **0 of the 8** token values were found anywhere in the database files afterwards. A first
attempt compared whole records and showed „0 of 5 equal": the R-879 columns were sealed with the REAL key, so with the
throwaway key every record was flagged unreadable — the expected behaviour, not a defect; the second run compared the
JSON only. The copy was shredded after. Image: `gitea.dooplex.hu/admin/felhom-hub:0.168.0`
(`sha256:3bfdecaae527d4762059600cdfaad83daf4e4b5b20661fc47acf128180d87cb3`), from `5277d859`, CI run 2269 success.
**Measured on the live file (count of the key name only):** `cf_tunnel_token` occurs 12 times both in the live
`hub.db` and in the clean `VACUUM INTO` snapshot, so the live file holds no extra old copies today.