Upload watch: spike, design (08 section 9, proposed), live proofs on 9202; rows R-1244..R-1248 (register 189 -> 194); STATUS D41/D42 + the Monday list
gates / gates (push) Successful in 13m1s
gates / gates (push) Successful in 13m1s
Spike: per-app normal levels on three boxes (19-31 days), the off-site push as a line measure (a lower bound), qBittorrent/Transmission/Plex explainers measured on 9202, a tc ceiling measured (not built). Live on 9202 with 0.342.0-rc.1: limit set; real seed explained; CPU burner and the hidden flood alarmed once each and cleared once; the household's limit left alone. Teardown: apps removed, 9202 back on 0.341.0, receiver 9471 destroyed.
This commit is contained in:
@@ -531,3 +531,37 @@ list is shown as custom and saved back unchanged; the default list maps to group
|
||||
multi-day backup fault reads as „nobody noticed" if the household ever finds out.
|
||||
2. **The cooldown field („hány óránként legfeljebb"):** keep it on the page or move it under „Részletek"? Pick:
|
||||
**under Részletek** — it is an engineering choice, which is what this redesign removes.
|
||||
|
||||
## 9. The upload and CPU watch — „does the app explain its own traffic?" [DESIGN — proposed; built in controller v0.342.0 + hub v0.169.0, NOT released]
|
||||
|
||||
**Why** (operator ruling 2026-10-23 19:22, `CONTEXT.md`): DooPlex's hacked Plex sent ~474 GB in a day with no viewer and
|
||||
used ~10 cores with no film being converted (`audits/plex2shell-felhom-impact-2026-10-23/`). Nothing on a Felhom box
|
||||
would have noticed: the health check sees whole-box CPU at one instant, and nothing looks at upload at all. **The idea:**
|
||||
when an app sends a lot (or burns CPU) for a long time, ask the APP whether that is its work. Work the app explains is
|
||||
normal use; work it does not explain is the sign of a hacked app. Numbers and measurements:
|
||||
`audits/SPIKE-upload-watch-2026-10-23.md`; the code: controller `internal/upwatch` (README §8e).
|
||||
|
||||
1. **The line.** Learnt from what the box already does: while its off-site push runs, the controller samples its own
|
||||
sent bytes once a second; the best ≥ 10-second rate is kept per night (14 nights, settings.json `upwatch.line`); a
|
||||
night counts when that window moved ≥ 10 MB; the line is the highest counting night. A **lower bound**, shown so
|
||||
(„legalább N Mbit/s", system settings). An active test is not built (R-1247).
|
||||
2. **Trigger, per app** (its compose project's containers summed; exact counters from the Docker Engine API, once a
|
||||
minute): upload ≥ max(5 Mbit/s, 25 % of the line) for 15 minutes; CPU ≥ max(150 %, 50 % × cores) of one core for 30
|
||||
minutes. Infrastructure (the controller — its own push is the line measure —, traefik, cloudflared, Samba) is never
|
||||
judged; the box's own work on an app (a backup run, an update, a deploy) is never judged.
|
||||
3. **The explanation.** qBittorrent (`up_info_data`) and Transmission (`uploadedBytes`): explained when the container
|
||||
sent ≤ 1.25 × what the app counted + 1 Mbit/s. Plex: any viewer explains upload; any transcode explains CPU. An app
|
||||
that does not answer gives no number (a hacked app may refuse its own API). No explainer: upload alarms only at ≥ 10 ×
|
||||
its own p95 for 60 minutes after ~20 h of history; CPU at ≥ max(200 %, 75 % × cores) for 60 minutes.
|
||||
4. **Who is told** (the R-388 rule, §8.1): **`app_unexplained_upload` / `app_unexplained_cpu`** (warning) and their
|
||||
`_cleared` (info, recorded) are **operator-only** (hub `operatorOnlyEvents`), one per episode, per-app cooldown. The
|
||||
household is told only about the qBittorrent limit, because they can change it (R-1246 asks the operator).
|
||||
5. **qBittorrent's default upload limit:** 50 % of the line (R-1245), every 15 minutes, re-set when the line moves
|
||||
> 20 %; never over a limit the household set (one there first, or any change from the value qBittorrent stored for
|
||||
the box — it keeps KiB/s). No line → none set. While the box owns it, qBittorrent's page says: „A qBittorrent
|
||||
feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
|
||||
6. **Limits, stated:** an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
|
||||
a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood
|
||||
whatever the API says; not built. The box cannot tell LAN from internet bytes per app; a host-network container
|
||||
(Samba) has no counter; the line is a lower bound.
|
||||
7. **Release:** not released; the operator's yes (R-1244).
|
||||
|
||||
@@ -76,7 +76,7 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
|
||||
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982) |
|
||||
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970) |
|
||||
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971) |
|
||||
| **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`) |
|
||||
| **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`). **Felhom half, 2026-10-23 night:** the last plain tokens sealed in hub 0.168.0 (built, deploy pending — `audits/hub-token-seal-2026-10-23/`); the two plain Longhorn copies of the hub volume already in the folder are R-1243's decision. A household box's version of this attacker (a hacked app sending or mining) is what the upload watch (R-1244, not released) detects |
|
||||
|
||||
## 6. What we accept (do not re-report)
|
||||
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
# SPIKE — who is using the upload line, and does the app explain it? (2026-10-23 night)
|
||||
|
||||
Operator ruling 2026-10-23 19:22 (`CONTEXT.md`): measure the line; on a sustained high upload attribute it to an app
|
||||
and ask the app; a mismatch is the alarm; the same shape for CPU; qBittorrent gets a default upload limit from the
|
||||
measured line, household changes respected. Brief Part 2.1. Design written from these numbers: §6 here and
|
||||
`architecture/08-alarm-ladder.md` §9 [DESIGN — proposed]. Evidence and tools: `SPIKE-upload-watch-2026-10-23/`.
|
||||
|
||||
Read first: `08-alarm-ladder.md` (R-388, §8.1), `07-backup-architecture.md` (the night chain: W = 02:30 local, the
|
||||
off-site push after it, the proof at 03:30 UTC), `15-media-library.md`, `13-threat-model.md` (attacker j),
|
||||
`audits/plex2shell-felhom-impact-2026-10-23/README.md` (the 1.5 TB Plex upload). Controller: `internal/metrics/`
|
||||
(`container_metrics` — one row per container per minute, 30 days), `internal/monitor/healthcheck.go` (the instant
|
||||
whole-box CPU warning), `internal/mediawire/` (the app clients and their credentials, `09` §3 decision 248).
|
||||
|
||||
Machines: the three test boxes READ ONLY (their stored metrics; their off-site record; tonight a one-second read of
|
||||
the controller's own counter during the real push); scratch 9202 on demo-hp (throwaway qBittorrent, Transmission, Plex,
|
||||
Jellyfin through the product's deploy); a throwaway receiver LXC **9471 `cc-uw-recv`** on demo-hp's scratch storage
|
||||
(never the thin pool). Every flood went 9202 → 9471 on demo-hp's own bridge; nothing went toward the internet.
|
||||
|
||||
## 1. Normal levels (stored metrics, three boxes)
|
||||
|
||||
History: **demo-hp** 2026-09-23 02:00 → 10-23 18:09 UTC (30.7 d, 998 216 rows); **demo-felhom** (the N100) same span
|
||||
(30.7 d, 268 120 rows); **Tester 1** 2026-10-04 19:41 → 10-23 18:08 (18.9 d, 294 433 rows). Per app: upload from the
|
||||
`net_tx_bytes` deltas (one-minute samples), CPU in percent of ONE core. Full tables: `s1-levels-per-app.txt`.
|
||||
|
||||
| box | the busiest uploaders (p50 / p95 / max Mbit/s) | the busiest CPU (p95 / max % of a core) |
|
||||
|---|---|---|
|
||||
| demo-hp | qbittorrent 0.02 / 1.2 / 4.0; immich-server 0.03 / 0.13 / 243 (one minute); felhom-controller 0 / 0.003 / 17.8 (the push) | grimmory 272 / 305 (a scan, 94 samples); immich-ML 32 / 548; romm 3 / 442; paperless 0.3 / 404 |
|
||||
| demo-felhom | immich-server 0.67 / 5.95 / 12.9 (32 samples); jellyfin 0 / 0.85 / 2.9 | immich-ML 295 / 296 (an import); onlyoffice 131 / 148; paperless 137 / 140 |
|
||||
| Tester 1 | qbittorrent 0.007 / 0.13 / 42.4; transmission 0.001 / 0.013 / 57.4; controller 0 / 0.003 / 37.5 | paperless 0.7 / 508; controller 46 / 423; plex 4.3 / 198 |
|
||||
|
||||
**How long a normal high lasts** (`s1-longest-runs.txt` — the longest run of consecutive minutes): upload ≥ 5 Mbit/s:
|
||||
**3 min** at most (qBittorrent, Tester 1); ≥ 1 Mbit/s: 9 min. CPU ≥ 150 %: **7 min** at most (Immich machine learning);
|
||||
≥ 80 %: 13 min (Kimai). Busiest upload hours (UTC): demo-hp 03h, 02h; demo-felhom 16h, 05h; Tester 1 07h, 18h — no fixed
|
||||
pattern beyond the night push. For scale: DooPlex's hacked Plex sent ~474 GB in one day (≈ 44 Mbit/s on average, for
|
||||
hours) and used ~10 cores.
|
||||
|
||||
`felhom-samba` runs with the host network: its counter is always 0 (no own network namespace) — the watch cannot see it.
|
||||
|
||||
## 2. The line without a test
|
||||
|
||||
**What the box already sees:** the off-site push (restic, inside the controller container) is the box's only big
|
||||
upload of its own. At the stored one-minute sample (`s2-controller-push-per-night.txt`): the controller's best five
|
||||
minutes per night exceeded 2 Mbit/s on **2 of 31 nights** on demo-hp (5.1 and 6.9 Mbit/s; best single minute 17.0),
|
||||
**1 of 19** on Tester 1 (7.6; best minute 37.5) and **0 of 31** on demo-felhom. The off-site record keeps the push's
|
||||
duration (33 s on demo-felhom, 1 m 8 s on Tester 1, 3 m 19 s on demo-hp on 2026-10-23) and **no byte count**.
|
||||
|
||||
**Conclusion:** at a one-minute sample the push shows the night's DATA SIZE (a few MB, incremental), not the line. Not
|
||||
enough as it stands. Two changes make it usable: sample once a SECOND during the push (the controller knows when its
|
||||
push runs and can read its own counter), and let a night count only when its best 10 seconds moved ≥ 10 MB. Even then
|
||||
it is a **lower bound** (restic over SFTP to the Storage Box may not fill the line; the family uploads too).
|
||||
|
||||
**Tonight's one-second read of the real push** (2026-10-24 ~02:16Z, three boxes, read only): §2.1 below.
|
||||
|
||||
### 2.1 One-second samples of the real push
|
||||
|
||||
*(filled after 02:30Z — see the section at the end, „Night data")*
|
||||
|
||||
## 3. The line with a test — NOT run tonight
|
||||
|
||||
| Candidate | Ours? | Terms | Cost per test | Tonight |
|
||||
|---|---|---|---|---|
|
||||
| ep0 (a receiver on our Hetzner box) | yes | ours | 10 s × line ≈ 125 MB at 100 Mbit/s | **not built** — a change on ep0 is forbidden tonight |
|
||||
| the box's own Storage Box sub-account, through the same transport as the push | yes (Hetzner, unlimited traffic) | Hetzner's — ours | same | not tried — a test file in the account that holds the restic repository needs its own design (the repo check, the quota) |
|
||||
| Cloudflare R2 (where the ISO lives) | yes | ours | same + R2 class-A writes | not used — its write key is the operator's ISO-publish key; a test object in a public bucket is wrong |
|
||||
| public speed tests (speed.cloudflare.com, LibreSpeed servers) | no | **not read** | — | not used (the brief: no third-party test without its terms) |
|
||||
|
||||
A quiet hour: never 02:30–05:30 local (dumps 02:30, second drive, off-site push, the 03:30 UTC proof). The family shares
|
||||
the line, so take the highest of several runs. **Not needed for the first build** — §6 works with the push's lower
|
||||
bound and an absolute floor; an active test is a decide row (R-1247).
|
||||
|
||||
## 4. Can the box ask each app? (measured on 9202 with throwaway installs; a private torrent 9202 → 9471)
|
||||
|
||||
Set-up: qBittorrent 5.2.4 and Transmission 4.1.3 installed through the product (`POST /api/stacks/<n>/deploy`; the
|
||||
first try was refused BEFORE anything was made — the data place must be the drive itself; two empty helper folders
|
||||
removed). The same 2 GiB file (an AES-CTR keystream of a fixed key) on both sides; a private torrent; a 40-line HTTP
|
||||
tracker on 9471 (`tools/tracker.py`); `transmission-daemon` on 9471 as the leecher. The admin passwords were read from
|
||||
the containers at use and never printed.
|
||||
|
||||
| App | Its own number | Measured | Limit | Household's limit visible? |
|
||||
|---|---|---|---|---|
|
||||
| **qBittorrent** | `/api/v2/transfer/info` → `up_info_data` (cumulative), `up_info_speed` | seeding: app **131.1 Mbit/s** vs container **134.6–140.5** → **×1.03–1.07** (`s4-qb-seed-nolimit.jsonl`) | `transfer/setUploadLimit` (bytes/s) = `preferences.up_limit`; stored in KiB/s (2 500 000 → **2 499 584**); survives a restart (`Session\GlobalUPSpeedLimit=2441`); measured: ~20 Mbit/s held (`s4-qb-seed-limit20.jsonl`) | yes: `transfer/uploadLimit` ≠ the value the box stored |
|
||||
| **Transmission** | RPC `session-stats.current-stats.uploadedBytes`, `uploadSpeed` | seeding: app 487.1 / 390.3 / 115.8 vs container 487.6 / 390.7 / 117.9 Mbit/s → **×1.001–1.02** (`s4-tr-seed-nolimit.jsonl`) | `session-set speed-limit-up` (kB/s) + `-enabled` (read back); **its rate under the limit was NOT observed** — the re-added leecher got no connection in 2 minutes | yes: `speed-limit-up-enabled` + value |
|
||||
| **Plex** | `/status/sessions`, `/transcode/sessions` (`MediaContainer.size`) | unclaimed 1.43.5: both 200 without a token from the docker network, `{"size":0}`; a claimed server needs its PlexOnlineToken (read at use from Preferences.xml, as mediawire does — no new secret) | — | — |
|
||||
| **Jellyfin** | `/Sessions` | **401** without a key; the key named `felhom` exists only when the household made it (mediawire's „one household step") — the box cannot create one at install without the admin login it does not have | — | — |
|
||||
| Nextcloud, Immich | **no upload-explaining number found** (not measured further); Immich's job queue needs an API key the box does not hold | — | — | → the plain rule (§6) |
|
||||
|
||||
**The fake hacked app** (code inside qBittorrent's own container, `docker exec … python3` → 9471:5001, Plex2Shell's
|
||||
shape): the container sent **~10 100 Mbit/s** (LAN, one bridge) while qBittorrent reported **0** (`s4-qb-flood.jsonl`).
|
||||
|
||||
**The counter itself:** the Docker Engine API's one-shot stats (`networks.*.tx_bytes`) equal the kernel's counter in
|
||||
the container's network namespace to the byte; `docker stats`' text (what `internal/metrics` stores) has three
|
||||
significant digits — ~10 MB steps at a GB total, ~1.3 Mbit/s at a one-minute sample. The watch reads the Engine API.
|
||||
|
||||
**A limit the hacked app controls:** an attacker with code in the app's container can also answer its API. The check
|
||||
catches an attacker who does not know about it; one who does could lie. §5 is the answer to that.
|
||||
|
||||
## 5. A universal upload ceiling — spike only
|
||||
|
||||
A helper container sharing the app's network namespace (`docker run --rm --network container:qbittorrent --cap-add
|
||||
NET_ADMIN alpine:3.20` + `iproute2-tc`) added `tc qdisc … root tbf rate 20mbit burst 64kb latency 400ms` on the app's
|
||||
`eth0` **inside the unprivileged 9202 guest**. The same flood then ran at **19.9–20.1 Mbit/s** (`s4-qb-flood-tbf.jsonl`),
|
||||
qBittorrent's API kept answering, and `tc qdisc del` restored it. Costs: a helper image with `tc` (a new external image,
|
||||
or `tc` shipped in the controller image), `NET_ADMIN` for that helper, re-applied after every app restart (the qdisc
|
||||
dies with the namespace), upload only. **Not built** (brief). It would cap a hacked app's flood whatever its API says.
|
||||
|
||||
## 6. The design (numbers from §1–§5; full text `08-alarm-ladder.md` §9)
|
||||
|
||||
1. **Line:** one-second samples of the controller's own sent bytes during the off-site push; the best ≥ 10-second rate
|
||||
per night; a night counts when that window moved ≥ 10 MB; the line = the highest counting night of the last 14;
|
||||
settings.json `upwatch.line`; shown on the system settings page as „legalább N Mbit/s" („még mérjük" until known).
|
||||
2. **Trigger:** an app (its containers summed, exact counters, once a minute) sends ≥ **max(5 Mbit/s, 25 % of the line)**
|
||||
for **15 minutes** — five times the longest normal run at ≥ 5 Mbit/s (3 min). Per app, not per box: the box cannot
|
||||
tell LAN from internet bytes inside its controller container, and the per-app view is what the check compares.
|
||||
3. **Attribution:** the app's compose project; infrastructure (controller, traefik, cloudflared, Samba) never judged.
|
||||
4. **Explanation:** qBittorrent / Transmission: explained when container bytes ≤ 1.25 × the app's own count + 1 Mbit/s
|
||||
(measured overhead 1–7 %); Plex: any viewer explains the upload (session bandwidth not measured — the conservative
|
||||
half; no viewer + a big upload = the Plex2Shell signature). Not explained → **`app_unexplained_upload`**,
|
||||
operator-only, one per episode, a clear after 15 quiet minutes. No explainer: alarm only at ≥ 10 × its own p95 for
|
||||
60 minutes, after ~20 h of history, worded „no app number to compare".
|
||||
5. **CPU:** ≥ **max(150 %, 50 % × cores)** of one core for **30 minutes** (four times the longest normal run at ≥ 150 %),
|
||||
explained by a Plex transcode or the box's own work on the app (backup, update, deploy) → else
|
||||
**`app_unexplained_cpu`**. No explainer: ≥ max(200 %, 75 % × cores) for 60 minutes.
|
||||
6. **qBittorrent limit:** 50 % of the line (CC's pick — R-1245), every 15 minutes, re-set when the line moves > 20 %;
|
||||
never over a household's limit (a limit there first, or any change from the stored value). No line → nothing set.
|
||||
The household's line (HU + EN in the report).
|
||||
7. **Edge cases:** the night push is the controller's (never judged); the first day: apps with an explainer are judged
|
||||
on the floor (a mismatch is evidence whatever the line), apps without one are not judged before ~20 h of history; a
|
||||
slow or changing line (a Wi-Fi laptop): the line is a max over 14 nights — a falling line lowers nothing until it
|
||||
rolls out (R-1247 notes the active test); two apps at once: each judged on its own bytes; the household's own devices:
|
||||
invisible to the box — the trigger uses the box's own bytes only; an app reinstalled from scratch reads as „the
|
||||
household removed the limit" and is left alone.
|
||||
|
||||
## 7. Live proofs on 9202 (controller `0.342.0-rc.1`, commit `a9c1997`, CI run 2270 success)
|
||||
|
||||
Set by hand in 9202's `/etc/felhom-controller-image` (an `-rc` tag is never offered to a household); 9202's hub link
|
||||
is off by design, so every event was handed to the notifier and DROPPED there — the hub side (allow-list, operator-only,
|
||||
the mail line) is proven by hub 0.169.0's tests. 9202's off-site is off, so one line record (40 Mbit/s) was seeded by
|
||||
hand into its settings.json. Logs: `SPIKE-upload-watch-2026-10-23/live/2-9202-watch-log-full.txt`.
|
||||
|
||||
| # | What | Result (UTC) |
|
||||
|---|---|---|
|
||||
| 1 | qBittorrent unlimited, line 40 Mbit/s → the box sets the limit | **19:47:42** „qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)" — read back from qBittorrent: 2 499 584 |
|
||||
| 2 | **a real torrent seeding at the limit** (5 GB, private, 9202 → 9471) | **20:04:42** „qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s)"; no alarm (counter ratio 1.04) |
|
||||
| 3 | **a CPU burner in Plex with no transcode** (4 busy loops, 400 % on a 7-core guest; trigger 350 %) | **20:07:42** „plex used 399% CPU … for 30 min with no work that explains it — operator alarm" → `app_unexplained_cpu`; ONE |
|
||||
| 4 | **the fake hacked app**: a 50 Mbit/s flood from inside qBittorrent's container to 9471, hidden behind the real seed | **20:19:42** „qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s) — operator alarm" → `app_unexplained_upload`; ONE, none more while it ran |
|
||||
| 5 | the burner stopped 20:11 | **20:25:41** „plex: the unexplained CPU stopped … — cleared" → `app_unexplained_cpu_cleared`; ONE |
|
||||
| 6 | the household changes qBittorrent's limit by hand (1 500 000 → stored 1 499 136) at 20:21 | **20:32:42** „qBittorrent's upload limit is the household's — the box leaves it alone from now on"; read back at 20:43: still 1 499 136 |
|
||||
| 7 | the flood stopped 20:26 (the seed went on, explained) | **20:41:41** „qbittorrent: the unexplained upload stopped … — cleared" → `app_unexplained_upload_cleared`; ONE |
|
||||
| 8 | the system settings page (`GET /settings`) | „Feltöltési sebesség: legalább 40 Mbit/s (az éjszakai távoli mentésből mérve)"; qBittorrent's page carried no limit line after step 6 (the household's) — the box-owned branch is pinned by `TestUpwatch_QBLimitLineOnQBittorrentsPageOnly`, not captured live |
|
||||
|
||||
**Found live:** the very first minute after a start failed to reach the apps by name (the bootstrap joins the controller
|
||||
to the app network a few seconds after the start); every later minute answered. That first minute only sets baselines —
|
||||
**NOT-A-FINDING**. The media wiring's generated `docker-compose.override.yml` stays beside the template after a removal
|
||||
(Jellyfin tonight; Plex's from the morning session) — rewritten before every start, harmless — **NOT-A-FINDING**.
|
||||
|
||||
**Teardown, three layers:** machine — the four throwaway apps removed through the product (containers and volumes gone,
|
||||
verified), my test files and probes deleted, the `alpine:3.20` helper image removed, 9202 back on 0.341.0 with the seeded
|
||||
record removed, the rc image removed; host — the receiver LXC 9471 destroyed (`--purge`, its disk gone); hub — none (9202
|
||||
is not on the hub). The night sampler on the three test boxes is reads only.
|
||||
@@ -0,0 +1,15 @@
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.342.0-rc.1
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0-rc.1 Up 6 minutes (healthy)
|
||||
filebrowser gtstef/filebrowser:1.5.6-stable Up 4 hours (healthy)
|
||||
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 4 hours (healthy)
|
||||
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 4 hours (healthy)
|
||||
immich-redis redis:7-alpine Up 4 hours (healthy)
|
||||
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 4 hours (healthy)
|
||||
jellyfin jellyfin/jellyfin:10.11.11 Up About an hour (healthy)
|
||||
paperless-postgres postgres:18-alpine Up 4 hours (healthy)
|
||||
paperless-redis redis:7-alpine Up 4 hours (healthy)
|
||||
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 4 hours (healthy)
|
||||
plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 Up About an hour (healthy)
|
||||
qbittorrent qbittorrentofficial/qbittorrent-nox:5.2.4-1 Up About an hour (healthy)
|
||||
traefik traefik:v3.7.13 Up 4 hours
|
||||
transmission lscr.io/linuxserver/transmission:version-4.1.3-r0 Up About an hour (healthy)
|
||||
@@ -0,0 +1,10 @@
|
||||
VMID Status Lock Name
|
||||
9201 running demo-hp
|
||||
9202 running demo-hp-scratch
|
||||
9401 stopped upgrade-harness
|
||||
9471 running cc-uw-recv
|
||||
description: CC throwaway 2026-10-23 night%3A upload-watch spike receiver/leecher. Remove after the session.%0A
|
||||
hostname: cc-uw-recv
|
||||
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:04:68:80,ip=dhcp,type=veth
|
||||
rootfs: nvme-scratch:9471/vm-9471-disk-0.raw,size=8G
|
||||
tags: cc;scratch;uw-spike
|
||||
@@ -0,0 +1,9 @@
|
||||
2026/10/23 19:32:41 upwatch_wiring.go:97: [INFO] [upwatch] upload and CPU watch started (line at least 40 Mbit/s, qBittorrent share 50 %)
|
||||
2026/10/23 19:32:41 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] transmission: the app did not answer (Post "http://transmission:9091/transmission/rpc": dial tcp: lookup transmission on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||
2026/10/23 19:32:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_definition_refused (severity warning) — further app_definition_refused events are logged at DEBUG only
|
||||
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] qbittorrent: the app did not answer (Post "http://qbittorrent:8080/api/v2/auth/login": dial tcp: lookup qbittorrent on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||
2026/10/23 19:47:42 service.go:108: [INFO] [upwatch] qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)
|
||||
2026/10/23 20:04:42 watch.go:303: [INFO] [upwatch] qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)); normal use
|
||||
2026/10/23 20:07:42 watch.go:343: [WARN] [upwatch] plex used 399% CPU (of one core; 7 cores) for 30 min with no work that explains it (plex transcodes) — operator alarm
|
||||
2026/10/23 20:07:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu (severity warning) — further app_unexplained_cpu events are logged at DEBUG only
|
||||
@@ -0,0 +1,16 @@
|
||||
2026/10/23 19:32:41 upwatch_wiring.go:97: [INFO] [upwatch] upload and CPU watch started (line at least 40 Mbit/s, qBittorrent share 50 %)
|
||||
2026/10/23 19:32:41 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] transmission: the app did not answer (Post "http://transmission:9091/transmission/rpc": dial tcp: lookup transmission on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||
2026/10/23 19:32:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_definition_refused (severity warning) — further app_definition_refused events are logged at DEBUG only
|
||||
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] qbittorrent: the app did not answer (Post "http://qbittorrent:8080/api/v2/auth/login": dial tcp: lookup qbittorrent on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||
2026/10/23 19:47:42 service.go:108: [INFO] [upwatch] qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)
|
||||
2026/10/23 20:04:42 watch.go:303: [INFO] [upwatch] qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)); normal use
|
||||
2026/10/23 20:07:42 watch.go:343: [WARN] [upwatch] plex used 399% CPU (of one core; 7 cores) for 30 min with no work that explains it (plex transcodes) — operator alarm
|
||||
2026/10/23 20:07:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu (severity warning) — further app_unexplained_cpu events are logged at DEBUG only
|
||||
2026/10/23 20:19:42 watch.go:307: [WARN] [upwatch] qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)) — operator alarm
|
||||
2026/10/23 20:19:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_upload (severity warning) — further app_unexplained_upload events are logged at DEBUG only
|
||||
2026/10/23 20:25:41 watch.go:378: [INFO] [upwatch] plex: the unexplained CPU stopped (15 min below the trigger or explained) — cleared
|
||||
2026/10/23 20:25:41 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu_cleared (severity info) — further app_unexplained_cpu_cleared events are logged at DEBUG only
|
||||
2026/10/23 20:32:42 service.go:111: [INFO] [upwatch] qBittorrent's upload limit is the household's — the box leaves it alone from now on
|
||||
2026/10/23 20:41:41 watch.go:370: [INFO] [upwatch] qbittorrent: the unexplained upload stopped (15 min below the trigger or explained) — cleared
|
||||
2026/10/23 20:41:41 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_upload_cleared (severity info) — further app_unexplained_upload_cleared events are logged at DEBUG only
|
||||
@@ -0,0 +1,2 @@
|
||||
/settings 56743 data-uplink> <span class="settings-label">Feltöltési sebesség</span> <span class="settings-value">legalább <span class="mono">40</span> Mbit/s (az éjszakai távoli mentésből mérve)</span> </div> | qb line: none
|
||||
/apps/qbittorrent 49281 no uplink row | qb line: none
|
||||
@@ -0,0 +1,4 @@
|
||||
version v5.2.4 webapi 2.15.1
|
||||
transfer/info {'connection_status': 'connected', 'dht_nodes': 375, 'dl_info_data': 0, 'dl_info_speed': 0, 'dl_rate_limit': 0, 'last_external_address_v4': '37.191.56.193', 'last_external_address_v6': '', 'up_info_data': 5566595787, 'up_info_speed': 0, 'up_rate_limit': 1499136}
|
||||
prefs {'up_limit': 1499136, 'dl_limit': 0, 'alt_up_limit': 10240, 'alt_dl_limit': 10240, 'scheduler_enabled': False, 'limit_tcp_overhead': False, 'limit_lan_peers': True, 'limit_utp_rate': True, 'max_uploads': 20, 'enable_upload_suggestions': False}
|
||||
uploadLimit 1499136 speedLimitsMode 0
|
||||
@@ -0,0 +1,186 @@
|
||||
|
||||
=== demo-hp: 2026-09-23 02:00 -> 2026-10-23 18:09 UTC (30.7 d), 998216 rows
|
||||
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
|
||||
qbittorrent 18 0.019 1.199 4.01 | 0.4 5.4 11.3 |
|
||||
metube 3 0.146 0.146 0.15 | 0.0 0.1 0.1 |
|
||||
immich-server 318 0.031 0.133 242.93 | 0.1 5.6 138.1 | resets=7
|
||||
grimmory-db 102 0.000 0.116 0.35 | 0.0 16.9 102.5 | resets=12
|
||||
immich-postgres 320 0.000 0.066 2.20 | 0.0 8.7 70.2 | resets=7
|
||||
grimmory 94 0.001 0.047 0.77 | 1.1 272.0 305.0 | resets=9
|
||||
immich-redis 320 0.013 0.029 0.18 | 0.6 4.3 5.1 | resets=7
|
||||
romm 26492 0.019 0.027 0.20 | 0.7 3.2 442.4 | resets=97
|
||||
docmost 43499 0.013 0.027 0.23 | 0.1 0.8 158.1 | resets=147
|
||||
rallly 71 0.000 0.017 0.12 | 0.0 8.5 152.1 | resets=10
|
||||
radarr 16 0.002 0.015 0.03 | 0.2 7.2 11.8 |
|
||||
romm-redis 26523 0.005 0.013 0.10 | 0.7 4.4 61.5 | resets=101
|
||||
paperless-webserver 43488 0.005 0.013 0.04 | 0.1 0.3 404.2 | resets=144
|
||||
nextcloud-db 663 0.001 0.013 1.74 | 0.0 6.1 71.8 | resets=11
|
||||
docmost-redis 43532 0.005 0.013 0.13 | 0.7 4.4 6.6 | resets=147
|
||||
cloudflared 43604 0.000 0.013 1.21 | 0.3 0.5 13.8 | resets=6
|
||||
rallly-postgres 71 0.000 0.007 0.01 | 0.0 5.0 8.8 | resets=7
|
||||
prowlarr 5 0.002 0.004 0.00 | 0.1 0.2 0.2 |
|
||||
sonarr 18 0.000 0.004 0.01 | 2.5 8.2 8.8 |
|
||||
felhom-controller 43608 0.000 0.003 17.79 | 1.0 57.9 251.4 | resets=133
|
||||
paperless-redis 43521 0.001 0.003 0.02 | 0.6 4.3 6.1 | resets=148
|
||||
bentopdf 43567 0.000 0.003 0.01 | 0.0 0.0 5.2 | resets=58
|
||||
homepage 78 0.000 0.002 0.01 | 0.0 4.2 4.5 | resets=1
|
||||
bookstack-db 43565 0.001 0.001 0.01 | 0.0 6.1 102.8 | resets=147
|
||||
romm-db 26526 0.000 0.001 0.19 | 0.0 6.0 105.3 | resets=89
|
||||
plex 11974 0.001 0.001 1.10 | 0.3 0.6 124.6 | resets=46
|
||||
nextcloud 662 0.000 0.001 0.80 | 0.0 5.7 150.0 | resets=11
|
||||
immich-machine-learning 320 0.000 0.001 0.39 | 0.2 32.0 548.0 | resets=7
|
||||
docmost-postgres 43533 0.000 0.001 0.02 | 0.0 4.7 29.6 | resets=147
|
||||
bookstack 43546 0.000 0.001 0.01 | 0.0 0.0 63.0 | resets=148
|
||||
adventurelog-postgres 43567 0.000 0.001 1.35 | 0.0 4.7 37.9 | resets=138
|
||||
adventurelog 43549 0.000 0.001 2.16 | 0.0 0.7 204.3 | resets=146
|
||||
ghost 20 0.000 0.001 0.00 | 0.0 6.8 132.9 | resets=2
|
||||
glance 20 0.000 0.001 0.00 | 0.0 0.0 0.0 | resets=2
|
||||
privatebin 43518 0.000 0.001 0.00 | 0.1 0.1 8.1 | resets=140
|
||||
paperless-postgres 43520 0.000 0.001 0.05 | 0.0 4.7 19.1 | resets=148
|
||||
kimai 30136 0.000 0.000 0.00 | 0.0 0.0 107.2 | resets=102
|
||||
home-assistant 14403 0.000 0.000 1.00 | 0.0 0.5 106.9 | resets=62
|
||||
kimai-db 30150 0.000 0.000 0.05 | 0.0 5.9 100.8 | resets=96
|
||||
adventurelog-frontend 43559 0.000 0.000 0.21 | 0.0 3.8 22.8 | resets=126
|
||||
vaultwarden 329 0.000 0.000 0.00 | 0.0 0.1 4.0 | resets=7
|
||||
calibre-web 43547 0.000 0.000 0.01 | 0.0 0.3 127.1 | resets=145
|
||||
opengist 43530 0.000 0.000 0.01 | 0.0 0.0 9.8 | resets=140
|
||||
jellyfin 297 0.000 0.000 5.87 | 0.0 4.4 160.5 | resets=7
|
||||
vikunja 8 0.000 0.000 0.00 | 0.0 0.0 0.0 | resets=2
|
||||
a1-echo 113 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
upbeat_perlman 1 0.000 0.000 0.00 | 31.2 31.2 31.2 |
|
||||
traefik 43608 0.000 0.000 1.21 | 0.1 3.5 92.1 | resets=11
|
||||
practical_johnson 1 0.000 0.000 0.00 | 9.9 9.9 9.9 |
|
||||
peaceful_ptolemy 1 0.000 0.000 0.00 | 14.1 14.1 14.1 |
|
||||
nextcloud-redis 663 0.000 0.000 0.03 | 0.6 4.1 7.9 | resets=5
|
||||
friendly_wilbur 1 0.000 0.000 0.00 | 59.0 59.0 59.0 |
|
||||
filebrowser 43607 0.000 0.000 0.74 | 0.0 0.0 35.1 | resets=17
|
||||
felhom-samba 10 0.000 0.000 0.00 | 0.0 13.5 17.6 | tx counter always 0 (host network?)
|
||||
determined_satoshi 1 0.000 0.000 0.00 | 17.4 17.4 17.4 |
|
||||
busiest UTC hours by bytes sent: 03h 14%, 02h 7%, 13h 5%, 00h 4%
|
||||
|
||||
=== demo-felhom: 2026-09-23 02:00 -> 2026-10-23 18:08 UTC (30.7 d), 268120 rows
|
||||
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
|
||||
immich-server 32 0.667 5.953 12.90 | 0.6 120.1 130.8 |
|
||||
jellyfin 16 0.000 0.854 2.85 | 0.0 72.7 290.7 |
|
||||
immich-postgres 32 0.000 0.766 2.24 | 0.3 8.5 197.6 |
|
||||
romm 16 0.020 0.339 1.07 | 0.5 25.5 100.5 |
|
||||
dawarich 20 0.000 0.313 3.01 | 0.1 5.3 104.4 |
|
||||
immich-machine-learning 32 0.003 0.243 0.42 | 0.2 294.7 295.7 |
|
||||
immich-redis 32 0.017 0.144 0.16 | 0.5 3.9 4.2 |
|
||||
nextcloud-db 16 0.000 0.137 0.43 | 0.0 4.9 5.8 |
|
||||
nextcloud 16 0.000 0.099 0.33 | 0.0 21.7 86.7 |
|
||||
paperless-webserver 21 0.005 0.073 0.08 | 0.0 137.4 139.7 |
|
||||
paperless-postgres 22 0.000 0.048 0.06 | 0.0 4.6 5.0 |
|
||||
kimai-db 19 0.000 0.023 0.15 | 0.0 8.4 38.8 |
|
||||
dawarich-sidekiq 19 0.020 0.023 0.02 | 0.1 0.2 0.2 |
|
||||
claper 20 0.019 0.020 0.02 | 0.5 0.7 0.9 |
|
||||
docmost 22 0.020 0.020 0.02 | 0.0 0.2 0.3 |
|
||||
wanderer 19 0.002 0.015 0.02 | 0.0 0.0 0.0 |
|
||||
cloudflared 43863 0.000 0.013 1.33 | 0.2 0.3 8.3 | resets=11
|
||||
adventurelog 22 0.000 0.013 3.24 | 0.0 0.1 65.8 |
|
||||
grafana 20 0.000 0.012 0.12 | 1.0 1.9 17.3 |
|
||||
wanderer-db 19 0.007 0.011 0.01 | 0.0 3.7 3.7 |
|
||||
dawarich-redis 20 0.009 0.011 0.01 | 0.4 3.7 3.8 |
|
||||
claper-postgres 20 0.011 0.011 0.01 | 0.2 0.5 4.4 |
|
||||
outline 19 0.008 0.008 0.01 | 0.0 7.4 7.7 |
|
||||
nextcloud-redis 16 0.000 0.008 0.02 | 0.4 0.7 1.4 |
|
||||
romm-redis 16 0.006 0.007 0.01 | 0.5 3.7 3.8 |
|
||||
docmost-redis 23 0.005 0.005 0.07 | 0.4 0.7 3.9 |
|
||||
dawarich-db 20 0.001 0.005 0.03 | 0.0 0.1 0.1 |
|
||||
bookstack-db 20 0.001 0.005 0.04 | 0.0 4.7 4.8 |
|
||||
grocy 19 0.000 0.005 0.01 | 0.0 0.0 0.0 |
|
||||
bookstack 20 0.000 0.005 0.04 | 0.0 7.9 47.2 |
|
||||
zipline 20 0.003 0.003 0.00 | 0.0 0.0 0.0 |
|
||||
paperless-redis 22 0.002 0.003 0.00 | 0.4 0.4 0.4 |
|
||||
outline-redis 19 0.003 0.003 0.00 | 0.4 0.4 0.5 |
|
||||
kimai 19 0.000 0.003 0.02 | 0.0 2.4 24.3 |
|
||||
zipline-postgres 20 0.003 0.003 0.00 | 0.0 0.0 0.0 |
|
||||
qbittorrent 11759 0.001 0.003 0.01 | 0.1 0.2 97.5 | resets=25
|
||||
felhom-controller 43863 0.000 0.003 1.73 | 0.1 15.5 122.0 | resets=92
|
||||
bentopdf 18 0.000 0.002 0.00 | 0.0 0.0 0.0 |
|
||||
emby 19 0.001 0.002 0.01 | 0.0 0.1 0.1 |
|
||||
homepage 20 0.000 0.002 0.00 | 0.0 0.0 0.0 |
|
||||
sonarr 11756 0.000 0.001 0.01 | 0.1 0.3 74.7 | resets=25
|
||||
grimmory-db 24 0.000 0.001 0.35 | 0.0 5.0 100.2 |
|
||||
grimmory 23 0.001 0.001 0.00 | 5.7 25.3 35.0 |
|
||||
lubelogger 19 0.000 0.001 0.00 | 0.0 0.2 0.6 |
|
||||
radarr 11756 0.000 0.001 0.17 | 0.0 0.6 79.9 | resets=25
|
||||
glance 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
metube 20 0.000 0.001 0.00 | 0.0 1.3 26.1 |
|
||||
actualbudget 18 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
privatebin 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
plex 18 0.000 0.001 0.00 | 0.7 0.8 0.8 |
|
||||
papra 20 0.000 0.001 0.00 | 0.0 6.8 7.5 |
|
||||
home-assistant 16 0.000 0.001 0.00 | 0.0 19.8 79.0 |
|
||||
wanderer-search 19 0.000 0.001 0.00 | 0.1 0.1 0.1 |
|
||||
karakeep 19 0.001 0.001 0.00 | 1.8 2.0 2.0 |
|
||||
romm-db 16 0.001 0.001 0.00 | 0.0 4.6 4.7 |
|
||||
radicale 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
gramps-web 20 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
tandoor 19 0.000 0.001 0.00 | 0.1 4.7 4.8 |
|
||||
karakeep-chrome 19 0.001 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
outline-postgres 19 0.000 0.001 0.00 | 0.0 0.4 0.7 |
|
||||
seerr 18 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||
n8n 22 0.000 0.000 0.00 | 0.8 1.0 110.6 |
|
||||
termix 20 0.000 0.000 0.00 | 0.0 0.4 2.3 |
|
||||
tandoor-postgres 19 0.000 0.000 0.00 | 0.4 1.5 4.1 |
|
||||
audiobookshelf 19 0.000 0.000 0.00 | 0.0 0.0 0.1 |
|
||||
adventurelog-postgres 22 0.000 0.000 0.02 | 0.0 3.8 33.6 |
|
||||
ghost 21 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
transmission 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||
komga 19 0.000 0.000 0.00 | 0.3 6.8 10.6 |
|
||||
calcom 18 0.000 0.000 0.00 | 0.0 3.4 22.3 |
|
||||
vaultwarden 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
sparkyfitness 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
homebox 19 0.000 0.000 0.00 | 0.1 0.2 0.2 |
|
||||
docmost-postgres 23 0.000 0.000 0.01 | 0.0 3.8 73.9 |
|
||||
vikunja 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
adventurelog-frontend 22 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
rallly 20 0.000 0.000 0.00 | 0.0 5.6 110.3 |
|
||||
calibre-web 17 0.000 0.000 0.00 | 0.0 21.3 106.4 |
|
||||
wishlist 20 0.000 0.000 0.00 | 0.0 6.3 125.1 |
|
||||
prowlarr 11738 0.000 0.000 0.00 | 0.0 0.1 77.0 | resets=26
|
||||
opengist 43862 0.000 0.000 0.01 | 0.0 0.0 6.0 | resets=107
|
||||
onlyoffice 22 0.000 0.000 0.00 | 0.0 130.5 147.5 |
|
||||
sabnzbd 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
gitea 19 0.000 0.000 0.00 | 0.0 0.4 4.0 |
|
||||
navidrome 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||
code-server 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
uptime-kuma 19 0.000 0.000 0.00 | 0.7 0.7 0.7 |
|
||||
nzbget 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
gokapi 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
recipe-importer 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
crafty-controller 19 0.000 0.000 0.00 | 0.1 8.3 8.5 |
|
||||
mealie 20 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||
rallly-postgres 20 0.000 0.000 0.00 | 0.0 3.7 3.8 |
|
||||
traefik 43863 0.000 0.000 0.96 | 0.0 3.6 17.9 | resets=13
|
||||
sparkyfitness-server 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||
sparkyfitness-db 19 0.000 0.000 0.00 | 0.0 3.8 3.8 |
|
||||
karakeep-meilisearch 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||
filebrowser 43863 0.000 0.000 0.74 | 0.0 0.0 15.7 | resets=16
|
||||
calcom-postgres 18 0.000 0.000 0.00 | 0.0 3.9 4.0 |
|
||||
busiest UTC hours by bytes sent: 16h 16%, 05h 12%, 09h 6%, 02h 5%
|
||||
|
||||
=== tester-1: 2026-10-04 19:41 -> 2026-10-23 18:08 UTC (18.9 d), 294433 rows
|
||||
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
|
||||
qbittorrent 3857 0.007 0.133 42.40 | 0.1 2.6 92.3 | resets=20
|
||||
transmission 3857 0.001 0.013 57.43 | 0.1 1.6 82.5 | resets=19
|
||||
paperless-webserver 26710 0.005 0.013 0.09 | 0.1 0.7 508.1 | resets=63
|
||||
cloudflared 26751 0.000 0.013 1.63 | 0.4 1.2 29.4 | resets=6
|
||||
prowlarr 3725 0.000 0.012 0.26 | 0.1 0.3 101.8 | resets=8
|
||||
sonarr 3855 0.001 0.004 1.21 | 0.1 4.5 131.6 | resets=20
|
||||
radarr 3855 0.001 0.003 1.52 | 0.1 4.3 103.3 | resets=19
|
||||
paperless-redis 26740 0.001 0.003 0.01 | 0.7 4.2 27.5 | resets=68
|
||||
felhom-controller 26751 0.000 0.003 37.47 | 0.5 46.4 422.6 | resets=62
|
||||
plex 3739 0.001 0.002 1.07 | 0.3 4.3 198.4 | resets=17
|
||||
bookstack-db 26729 0.001 0.001 0.03 | 0.0 5.7 130.6 | resets=66
|
||||
plex-k6 514 0.001 0.001 0.01 | 0.4 5.4 7.2 |
|
||||
bookstack 26722 0.000 0.001 0.04 | 0.0 0.1 72.5 | resets=65
|
||||
privatebin 26744 0.000 0.001 0.00 | 0.0 0.1 14.7 | resets=64
|
||||
paperless-postgres 26740 0.000 0.001 0.02 | 0.0 4.4 65.8 | resets=68
|
||||
traefik 26751 0.000 0.000 1.56 | 0.0 3.4 100.6 | resets=3
|
||||
jellyfin 3648 0.000 0.000 2.83 | 0.0 0.2 181.5 | resets=19
|
||||
xfp-echo 1 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
vaultwarden 1 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||
filebrowser 26743 0.000 0.000 0.01 | 0.0 0.0 80.1 | resets=22
|
||||
busiest UTC hours by bytes sent: 07h 13%, 18h 11%, 02h 9%, 17h 9%
|
||||
@@ -0,0 +1,22 @@
|
||||
== demo-hp
|
||||
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
|
||||
kimai 13 0 (-) 0 0 (-)
|
||||
paperless-webserver 6 1 (09-23 04:56) 0 0 (-)
|
||||
immich-machine-learning 4 4 (10-14 00:54) 0 0 (-)
|
||||
romm 3 2 (10-23 15:36) 0 0 (-)
|
||||
adventurelog 2 1 (09-30 11:29) 3 0 (-)
|
||||
adventurelog-postgres 0 0 (-) 3 0 (-)
|
||||
felhom-controller 3 1 (09-23 11:37) 2 2 (09-28 13:44)
|
||||
immich-server 2 0 (-) 3 2 (10-14 00:53)
|
||||
== demo-felhom
|
||||
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
|
||||
immich-server 2 0 (-) 8 1 (10-13 05:32)
|
||||
immich-machine-learning 7 7 (10-13 16:39) 0 0 (-)
|
||||
paperless-webserver 6 0 (-) 0 0 (-)
|
||||
onlyoffice 4 0 (-) 0 0 (-)
|
||||
== tester-1
|
||||
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
|
||||
plex 10 5 (10-14 17:20) 1 0 (-)
|
||||
qbittorrent 2 0 (-) 9 3 (10-14 18:08)
|
||||
sonarr 9 0 (-) 2 0 (-)
|
||||
transmission 1 0 (-) 6 2 (10-16 07:17)
|
||||
@@ -0,0 +1,90 @@
|
||||
|
||||
== demo-hp: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
|
||||
2026-09-23 max1= 3.14 best5= 0.91 busy_min= 0 peak_at=02:17Z
|
||||
2026-09-24 max1= 3.04 best5= 0.73 busy_min= 0 peak_at=02:17Z
|
||||
2026-09-25 max1= 2.13 best5= 0.67 busy_min= 0 peak_at=02:18Z
|
||||
2026-09-26 max1= 1.77 best5= 0.63 busy_min= 0 peak_at=02:18Z
|
||||
2026-09-27 max1= 2.04 best5= 0.75 busy_min= 0 peak_at=02:18Z
|
||||
2026-09-28 max1= 17.79 best5= 6.89 busy_min= 3 peak_at=13:43Z
|
||||
2026-09-29 max1= 2.45 best5= 0.65 busy_min= 0 peak_at=02:17Z
|
||||
2026-09-30 max1= 2.62 best5= 0.76 busy_min= 0 peak_at=02:18Z
|
||||
2026-10-01 max1= 1.02 best5= 0.54 busy_min= 0 peak_at=02:18Z
|
||||
2026-10-02 max1= 2.02 best5= 0.62 busy_min= 0 peak_at=02:18Z
|
||||
2026-10-03 max1= 2.01 best5= 0.60 busy_min= 0 peak_at=15:25Z
|
||||
2026-10-04 max1= 2.08 best5= 0.42 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-05 max1= 2.36 best5= 0.48 busy_min= 0 peak_at=05:26Z
|
||||
2026-10-06 max1= 1.58 best5= 0.32 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-07 max1= 1.05 best5= 0.30 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-08 max1= 1.43 best5= 0.29 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-09 max1= 1.47 best5= 0.29 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-10 max1= 0.90 best5= 0.27 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-11 max1= 1.34 best5= 0.30 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-12 max1= 4.96 best5= 0.99 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-13 max1= 1.31 best5= 0.28 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-14 max1= 6.31 best5= 1.27 busy_min= 1 peak_at=02:18Z
|
||||
2026-10-15 max1= 2.25 best5= 0.45 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-16 max1= 2.78 best5= 0.66 busy_min= 0 peak_at=20:55Z
|
||||
2026-10-17 max1= 2.26 best5= 0.45 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-18 max1= 2.08 best5= 0.47 busy_min= 0 peak_at=02:18Z
|
||||
2026-10-19 max1= 16.95 best5= 5.08 busy_min= 2 peak_at=02:18Z
|
||||
2026-10-20 max1= 1.20 best5= 0.48 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-21 max1= 2.27 best5= 0.46 busy_min= 0 peak_at=02:18Z
|
||||
2026-10-22 max1= 2.13 best5= 0.52 busy_min= 0 peak_at=02:18Z
|
||||
2026-10-23 max1= 1.37 best5= 0.48 busy_min= 0 peak_at=02:18Z
|
||||
nights with a push (best5>2): 2; best5 min/median/max = 5.1/6.9/6.9 Mbit/s
|
||||
|
||||
== demo-felhom: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
|
||||
2026-09-23 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=02:15Z
|
||||
2026-09-24 max1= 0.06 best5= 0.01 busy_min= 0 peak_at=02:15Z
|
||||
2026-09-25 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||
2026-09-26 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||
2026-09-27 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=00:19Z
|
||||
2026-09-28 max1= 0.03 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||
2026-09-29 max1= 0.03 best5= 0.01 busy_min= 0 peak_at=02:15Z
|
||||
2026-09-30 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-01 max1= 0.07 best5= 0.01 busy_min= 0 peak_at=19:54Z
|
||||
2026-10-02 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-03 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-04 max1= 0.02 best5= 0.00 busy_min= 0 peak_at=06:58Z
|
||||
2026-10-05 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=05:21Z
|
||||
2026-10-06 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=17:34Z
|
||||
2026-10-07 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-08 max1= 0.09 best5= 0.02 busy_min= 0 peak_at=13:04Z
|
||||
2026-10-09 max1= 0.10 best5= 0.02 busy_min= 0 peak_at=05:34Z
|
||||
2026-10-10 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-11 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=10:46Z
|
||||
2026-10-12 max1= 0.08 best5= 0.02 busy_min= 0 peak_at=06:18Z
|
||||
2026-10-13 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=13:54Z
|
||||
2026-10-14 max1= 0.07 best5= 0.02 busy_min= 0 peak_at=07:59Z
|
||||
2026-10-15 max1= 0.14 best5= 0.03 busy_min= 0 peak_at=14:25Z
|
||||
2026-10-16 max1= 0.84 best5= 0.17 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-17 max1= 0.49 best5= 0.10 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-18 max1= 0.33 best5= 0.07 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-19 max1= 1.73 best5= 0.35 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-20 max1= 0.23 best5= 0.07 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-21 max1= 0.36 best5= 0.07 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-22 max1= 0.52 best5= 0.11 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-23 max1= 0.48 best5= 0.10 busy_min= 0 peak_at=02:15Z
|
||||
|
||||
== tester-1: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
|
||||
2026-10-04 max1= 0.17 best5= 0.14 busy_min= 0 peak_at=19:55Z
|
||||
2026-10-05 max1= 1.44 best5= 0.29 busy_min= 0 peak_at=03:10Z
|
||||
2026-10-06 max1= 0.64 best5= 0.13 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-07 max1= 0.64 best5= 0.13 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-08 max1= 0.42 best5= 0.08 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-09 max1= 0.48 best5= 0.10 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-10 max1= 0.84 best5= 0.17 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-11 max1= 0.46 best5= 0.09 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-12 max1= 1.71 best5= 0.34 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-13 max1= 0.99 best5= 0.20 busy_min= 0 peak_at=02:15Z
|
||||
2026-10-14 max1= 0.45 best5= 0.09 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-15 max1= 37.47 best5= 7.58 busy_min= 1 peak_at=02:17Z
|
||||
2026-10-16 max1= 3.85 best5= 0.77 busy_min= 0 peak_at=02:17Z
|
||||
2026-10-17 max1= 5.11 best5= 1.21 busy_min= 1 peak_at=02:16Z
|
||||
2026-10-18 max1= 0.50 best5= 0.10 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-19 max1= 5.14 best5= 1.03 busy_min= 1 peak_at=02:16Z
|
||||
2026-10-20 max1= 0.52 best5= 0.10 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-21 max1= 0.61 best5= 0.12 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-22 max1= 0.53 best5= 0.11 busy_min= 0 peak_at=02:16Z
|
||||
2026-10-23 max1= 0.80 best5= 0.18 busy_min= 0 peak_at=02:16Z
|
||||
nights with a push (best5>2): 1; best5 min/median/max = 7.6/7.6/7.6 Mbit/s
|
||||
@@ -0,0 +1,6 @@
|
||||
login 204 cookie
|
||||
{"t": 0.2, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65136369198, "engine_tx": 65136367684, "cli": "65.5MB / 65.1GB"}
|
||||
{"t": 6.5, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65152041285, "engine_tx": 65152039771, "cli": "65.9MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 19.9, "rate_engine_Mbit": 19.9}
|
||||
{"t": 12.7, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65167646902, "engine_tx": 65167645388, "cli": "66.2MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 20.14, "rate_engine_Mbit": 20.14}
|
||||
{"t": 19.0, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65183320503, "engine_tx": 65183318989, "cli": "66.6MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 19.9, "rate_engine_Mbit": 19.9}
|
||||
{"t": 25.2, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65198925974, "engine_tx": 65198924460, "cli": "66.9MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 20.14, "rate_engine_Mbit": 20.14}
|
||||
@@ -0,0 +1,10 @@
|
||||
LISTEN 0 5 0.0.0.0:5001 0.0.0.0:*
|
||||
login 204 cookie
|
||||
{"t": 0.0, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 1559315619, "engine_tx": 1558369279, "cli": "2.94MB / 2.85GB"}
|
||||
{"t": 6.0, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 9161893432, "engine_tx": 9161333668, "cli": "9.34MB / 10.5GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10136.77, "rate_engine_Mbit": 10137.29}
|
||||
{"t": 12.1, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 17020963755, "engine_tx": 17019975365, "cli": "16.7MB / 18.7GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10306.98, "rate_engine_Mbit": 10306.42}
|
||||
{"t": 18.1, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 25140960556, "engine_tx": 25139940620, "cli": "23.6MB / 26.5GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10826.66, "rate_engine_Mbit": 10826.62}
|
||||
{"t": 24.2, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 33049634829, "engine_tx": 33048966257, "cli": "30.6MB / 34.4GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10372.03, "rate_engine_Mbit": 10372.49}
|
||||
{"t": 30.2, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 41055773948, "engine_tx": 41055105508, "cli": "37.6MB / 42.3GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10674.85, "rate_engine_Mbit": 10674.85}
|
||||
{"t": 36.3, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 48735514801, "engine_tx": 48734576785, "cli": "44.3MB / 50GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10071.79, "rate_engine_Mbit": 10071.44}
|
||||
{"t": 42.3, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 56370706217, "engine_tx": 56370119481, "cli": "52.4MB / 58.8GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10180.26, "rate_engine_Mbit": 10180.72}
|
||||
@@ -0,0 +1,9 @@
|
||||
login 204 cookie
|
||||
setUploadLimit (200, '')
|
||||
uploadLimit now 2499584 prefs.up_limit 2499584
|
||||
login 204 cookie
|
||||
{"t": 0.0, "qb_up_speed": 16386315, "qb_up_data": 1346633728, "qb_tor_uploaded": 1346633728, "proc_tx": 1422747831, "engine_tx": 1422747831, "cli": "27.2MB / 1.42GB"}
|
||||
{"t": 6.0, "qb_up_speed": 0, "qb_up_data": 1374387364, "qb_tor_uploaded": 1374387364, "proc_tx": 1434635707, "engine_tx": 1434635707, "cli": "27.6MB / 1.44GB", "rate_qb_data_Mbit": 37.0, "rate_proc_Mbit": 15.85, "rate_engine_Mbit": 15.85}
|
||||
{"t": 12.1, "qb_up_speed": 2872275, "qb_up_data": 1392093322, "qb_tor_uploaded": 1391692420, "proc_tx": 1457229455, "engine_tx": 1457229455, "cli": "27.9MB / 1.46GB", "rate_qb_data_Mbit": 23.22, "rate_proc_Mbit": 29.63, "rate_engine_Mbit": 29.63}
|
||||
{"t": 18.1, "qb_up_speed": 1975559, "qb_up_data": 1406540742, "qb_tor_uploaded": 1406540742, "proc_tx": 1471460371, "engine_tx": 1471460371, "cli": "28.2MB / 1.47GB", "rate_qb_data_Mbit": 19.26, "rate_proc_Mbit": 18.97, "rate_engine_Mbit": 18.97}
|
||||
{"t": 24.2, "qb_up_speed": 3254241, "qb_up_data": 1423113257, "qb_tor_uploaded": 1420865709, "proc_tx": 1488294067, "engine_tx": 1488294067, "cli": "28.5MB / 1.49GB", "rate_qb_data_Mbit": 21.73, "rate_proc_Mbit": 22.08, "rate_engine_Mbit": 22.08}
|
||||
@@ -0,0 +1,11 @@
|
||||
login 204 cookie
|
||||
{"t": 0.0, "qb_up_speed": 16266049, "qb_up_data": 117833728, "qb_tor_uploaded": 117701484, "proc_tx": 123161204, "engine_tx": 123161204, "cli": "6.74MB / 140MB"}
|
||||
{"t": 6.0, "qb_up_speed": 16384087, "qb_up_data": 216137728, "qb_tor_uploaded": 216137728, "proc_tx": 225760308, "engine_tx": 225760308, "cli": "8.38MB / 243MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 136.8, "rate_engine_Mbit": 136.8}
|
||||
{"t": 12.1, "qb_up_speed": 16387266, "qb_up_data": 314441728, "qb_tor_uploaded": 314441728, "proc_tx": 328359626, "engine_tx": 328359626, "cli": "10.1MB / 346MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 134.56, "rate_engine_Mbit": 134.56}
|
||||
{"t": 18.1, "qb_up_speed": 16391332, "qb_up_data": 412745728, "qb_tor_uploaded": 412745728, "proc_tx": 431576783, "engine_tx": 431540471, "cli": "11.8MB / 450MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 137.62, "rate_engine_Mbit": 137.57}
|
||||
{"t": 24.2, "qb_up_speed": 16377743, "qb_up_data": 511049728, "qb_tor_uploaded": 511049728, "proc_tx": 536615588, "engine_tx": 536559607, "cli": "13.4MB / 556MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 137.76, "rate_engine_Mbit": 137.73}
|
||||
{"t": 30.2, "qb_up_speed": 16387474, "qb_up_data": 609353728, "qb_tor_uploaded": 609353728, "proc_tx": 641957804, "engine_tx": 641903336, "cli": "14.9MB / 660MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 140.46, "rate_engine_Mbit": 140.46}
|
||||
{"t": 36.3, "qb_up_speed": 16383934, "qb_up_data": 707657728, "qb_tor_uploaded": 707657728, "proc_tx": 746552576, "engine_tx": 746552576, "cli": "16.8MB / 764MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 137.17, "rate_engine_Mbit": 137.24}
|
||||
{"t": 42.3, "qb_up_speed": 16392852, "qb_up_data": 805961728, "qb_tor_uploaded": 805961728, "proc_tx": 849903723, "engine_tx": 849903723, "cli": "18.3MB / 867MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 137.8, "rate_engine_Mbit": 137.8}
|
||||
{"t": 48.4, "qb_up_speed": 16382132, "qb_up_data": 904265728, "qb_tor_uploaded": 904265728, "proc_tx": 952502020, "engine_tx": 952502020, "cli": "19.9MB / 970MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 134.56, "rate_engine_Mbit": 134.56}
|
||||
{"t": 54.4, "qb_up_speed": 16376106, "qb_up_data": 1002569728, "qb_tor_uploaded": 1002569728, "proc_tx": 1055099637, "engine_tx": 1055099637, "cli": "21.5MB / 1.07GB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 136.8, "rate_engine_Mbit": 136.8}
|
||||
@@ -0,0 +1,8 @@
|
||||
{'arguments': {}, 'result': 'success'}
|
||||
{'speed-limit-up': 2500, 'speed-limit-up-enabled': True}
|
||||
{"t": 0.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157242728}
|
||||
{"t": 5.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157244479, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 10.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157246230, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 15.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157248133, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157249884, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 25.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157251635, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
@@ -0,0 +1,10 @@
|
||||
{"t": 0.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157256302}
|
||||
{"t": 10.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157258205, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157261594, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 30.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157263345, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 40.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157265096, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 50.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157268485, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 60.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157271188, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 70.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157272939, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 80.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157276328, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 90.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157278588, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
@@ -0,0 +1,15 @@
|
||||
login 204 cookie
|
||||
/api/v2/torrents/stop (200, '')
|
||||
{'arguments': {}, 'result': 'success'}
|
||||
{"t": 0.0, "tr_upspeed": 40648163, "tr_cur_up": 1528429958, "tr_tor_up": 1528429958, "tx": 1534546958}
|
||||
{"t": 5.0, "tr_upspeed": 65588000, "tr_cur_up": 1832873057, "tr_tor_up": 1832873057, "tx": 1839315356, "rate_tr_Mbit": 487.11, "rate_container_Mbit": 487.63}
|
||||
{"t": 10.0, "tr_upspeed": 23587084, "tr_cur_up": 2076811226, "tr_tor_up": 2076811226, "tx": 2083512878, "rate_tr_Mbit": 390.3, "rate_container_Mbit": 390.72}
|
||||
{"t": 15.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157213873, "rate_tr_Mbit": 115.8, "rate_container_Mbit": 117.92}
|
||||
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157215624, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 25.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157218947, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.01}
|
||||
{"t": 30.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157220698, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 35.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157222449, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 40.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157224200, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 45.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157225951, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 50.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157227702, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||
{"t": 55.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157231243, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.01}
|
||||
@@ -0,0 +1,2 @@
|
||||
# usage: gen.sh <dir> ; writes <dir>/uwtest.bin, 2 GiB, identical on every machine (AES-CTR keystream of a fixed key)
|
||||
mkdir -p "$1"; openssl enc -aes-128-ctr -nosalt -K 00112233445566778899aabbccddeeff -iv 0 -in /dev/zero 2>/dev/null | head -c 2147483648 > "$1/uwtest.bin"; sha256sum "$1/uwtest.bin" | cut -c1-16
|
||||
@@ -0,0 +1,30 @@
|
||||
import sqlite3,sys,collections,datetime,statistics
|
||||
def pct(a,p):
|
||||
if not a: return 0
|
||||
a=sorted(a); k=(len(a)-1)*p/100; f=int(k); c=min(f+1,len(a)-1); return a[f]+(a[c]-a[f])*(k-f)
|
||||
for box in sys.argv[1:]:
|
||||
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
|
||||
lo,hi,n=db.execute('select min(ts),max(ts),count(*) from container_metrics').fetchone()
|
||||
print(f'\n=== {box.rsplit("/",1)[1]}: {datetime.datetime.utcfromtimestamp(lo):%Y-%m-%d %H:%M} -> {datetime.datetime.utcfromtimestamp(hi):%Y-%m-%d %H:%M} UTC ({(hi-lo)/86400:.1f} d), {n} rows')
|
||||
rows=collections.defaultdict(list)
|
||||
for ts,name,cpu,tx in db.execute('select ts,container_name,cpu_percent,net_tx_bytes from container_metrics order by ts'):
|
||||
rows[name].append((ts,cpu,tx))
|
||||
hour=collections.defaultdict(float)
|
||||
out=[]
|
||||
for name,r in rows.items():
|
||||
rates=[];resets=0;zero=all((x[2] or 0)==0 for x in r)
|
||||
for (t0,_,a),(t1,_,b) in zip(r,r[1:]):
|
||||
if a is None or b is None or t1-t0<=0 or t1-t0>180: continue
|
||||
d=b-a
|
||||
if d<0: resets+=1; continue
|
||||
mbps=d*8/(t1-t0)/1e6; rates.append(mbps)
|
||||
hour[datetime.datetime.utcfromtimestamp(t1).hour]+=d
|
||||
cpus=[x[1] for x in r]
|
||||
out.append((pct(rates,95),name,len(r),pct(rates,50),pct(rates,95),max(rates or [0]),pct(cpus,50),pct(cpus,95),max(cpus),zero,resets))
|
||||
print(f'{"container":32} {"n":>6} {"tx p50":>8} {"p95":>8} {"max":>8} Mbit/s | {"cpu p50":>7} {"p95":>7} {"max":>7} %core | notes')
|
||||
for o in sorted(out,reverse=True):
|
||||
_,name,nn,a,b,c,d,e,f,zero,res=o
|
||||
note=('tx counter always 0 (host network?)' if zero else '')+(f' resets={res}' if res else '')
|
||||
print(f'{name[:32]:32} {nn:6} {a:8.3f} {b:8.3f} {c:8.2f} | {d:7.1f} {e:7.1f} {f:7.1f} | {note}')
|
||||
tot=sum(hour.values()) or 1
|
||||
print('busiest UTC hours by bytes sent:',', '.join(f'{h:02d}h {100*v/tot:.0f}%' for h,v in sorted(hour.items(),key=lambda x:-x[1])[:4]))
|
||||
@@ -0,0 +1,21 @@
|
||||
import sqlite3,sys,collections,datetime
|
||||
for box in sys.argv[1:]:
|
||||
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
|
||||
r=db.execute("select ts,net_tx_bytes from container_metrics where container_name='felhom-controller' order by ts").fetchall()
|
||||
per=collections.defaultdict(list)
|
||||
for (t0,a),(t1,b) in zip(r,r[1:]):
|
||||
if a is None or b is None or b<a or not 0<t1-t0<=180: continue
|
||||
d=datetime.datetime.utcfromtimestamp(t1)
|
||||
per[d.date()].append((t1,(b-a)*8/(t1-t0)/1e6))
|
||||
print(f'\n== {box.rsplit("/",1)[1]}: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak')
|
||||
peaks=[]
|
||||
for day in sorted(per):
|
||||
v=per[day]; rates=[x[1] for x in v]
|
||||
best5=max((sum(rates[i:i+5])/5 for i in range(max(1,len(rates)-4))),default=0)
|
||||
m=max(v,key=lambda x:x[1])
|
||||
busy=sum(1 for x in rates if x>5)
|
||||
peaks.append(best5)
|
||||
print(f'{day} max1={m[1]:6.2f} best5={best5:6.2f} busy_min={busy:4d} peak_at={datetime.datetime.utcfromtimestamp(m[0]):%H:%M}Z')
|
||||
big=[p for p in peaks if p>2]
|
||||
if big:
|
||||
big.sort(); print(f'nights with a push (best5>2): {len(big)}; best5 min/median/max = {big[0]:.1f}/{big[len(big)//2]:.1f}/{big[-1]:.1f} Mbit/s')
|
||||
@@ -0,0 +1,10 @@
|
||||
#!/bin/bash
|
||||
# Waits until 02:08Z on 2026-10-24, then reads the controller container's eth0 tx bytes once a second for 22 minutes on
|
||||
# each of the three test boxes (read only: /proc/<pid>/net/dev inside guest 9201). Output: one file per box.
|
||||
OUT=$1
|
||||
target=$(date -u -d '2026-10-24T02:08:00Z' +%s); now=$(date -u +%s); [ $target -gt $now ] && sleep $((target-now))
|
||||
INNER='P=$(docker inspect -f {{.State.Pid}} felhom-controller); end=$(( $(date +%s) + 1320 )); while [ $(date +%s) -lt $end ]; do echo "$(date +%s.%N | cut -c1-14) $(awk "/eth0:/{print \$10}" /proc/$P/net/dev)"; sleep 1; done'
|
||||
for pair in "demo-hp:demo-hp" "felhom-pve:demo-felhom" "root@192.168.0.154:tester-1"; do h=${pair%%:*}; n=${pair##*:}
|
||||
ssh -o BatchMode=yes -o ServerAliveInterval=30 $h "pct exec 9201 -- bash -c '$INNER'" > "$OUT/night-push-1s-$n.txt" 2>&1 &
|
||||
done
|
||||
wait; echo "done $(date -u +%FT%TZ)"
|
||||
@@ -0,0 +1,89 @@
|
||||
# Runs INSIDE guest 9202. Asks qBittorrent's own Web API and reads the container's exact byte counters.
|
||||
# The admin password is read from the container env at use (as the controller does) and never printed.
|
||||
import json, subprocess, sys, time, urllib.request, urllib.parse, http.client, socket, re, uuid
|
||||
def sh(*a): return subprocess.run(a, capture_output=True, text=True).stdout.strip()
|
||||
C = sys.argv[1] if len(sys.argv) > 1 else 'qbittorrent'
|
||||
IP = sh('docker', 'inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}', C).split()[0]
|
||||
PID = sh('docker', 'inspect', '-f', '{{.State.Pid}}', C)
|
||||
CID = sh('docker', 'inspect', '-f', '{{.Id}}', C)
|
||||
BASE = f'http://{IP}:8080'
|
||||
PW = sh('docker', 'exec', C, 'printenv', 'QBT_ADMIN_PASSWORD')
|
||||
def req(method, path, data=None, hdr=None, raw=False):
|
||||
h = {'Referer': BASE}
|
||||
h.update(hdr or {})
|
||||
if COOKIE: h['Cookie'] = COOKIE
|
||||
r = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(r, timeout=20) as resp:
|
||||
b = resp.read(); return resp.status, (b if raw else (json.loads(b) if b and b[:1] in b'[{' else b.decode()))
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code, e.read().decode()[:200]
|
||||
COOKIE = None
|
||||
code, _ = None, None
|
||||
r = urllib.request.Request(BASE + '/api/v2/auth/login', data=urllib.parse.urlencode({'username': 'admin', 'password': PW}).encode(),
|
||||
headers={'Referer': BASE, 'Content-Type': 'application/x-www-form-urlencoded'}, method='POST')
|
||||
with urllib.request.urlopen(r, timeout=20) as resp:
|
||||
m = re.search(r'((?:QBT_)?SID[A-Za-z0-9_]*)=([^;]+)', '; '.join(resp.headers.get_all('Set-Cookie') or []))
|
||||
print('login', resp.status, 'cookie' if m else 'NO COOKIE')
|
||||
COOKIE = f'{m.group(1)}={m.group(2)}'
|
||||
def netdev():
|
||||
for line in open(f'/proc/{PID}/net/dev'):
|
||||
if line.strip().startswith('eth0:'):
|
||||
f = line.split(':', 1)[1].split(); return int(f[8])
|
||||
def engine_stats():
|
||||
s = socket.socket(socket.AF_UNIX); s.connect('/var/run/docker.sock')
|
||||
s.sendall(f'GET /containers/{CID}/stats?stream=false&one-shot=true HTTP/1.0\r\n\r\n'.encode())
|
||||
b = b''
|
||||
while True:
|
||||
x = s.recv(65536)
|
||||
if not x: break
|
||||
b += x
|
||||
j = json.loads(b.split(b'\r\n\r\n', 1)[1])
|
||||
return sum(n['tx_bytes'] for n in j.get('networks', {}).values()), j['cpu_stats']['cpu_usage']['total_usage']
|
||||
def cli_stats():
|
||||
return sh('docker', 'stats', '--no-stream', '--format', '{{.NetIO}}', C)
|
||||
mode = sys.argv[2] if len(sys.argv) > 2 else 'info'
|
||||
if mode == 'info':
|
||||
print('version', req('GET', '/api/v2/app/version')[1], 'webapi', req('GET', '/api/v2/app/webapiVersion')[1])
|
||||
print('transfer/info', req('GET', '/api/v2/transfer/info')[1])
|
||||
p = req('GET', '/api/v2/app/preferences')[1]
|
||||
print('prefs', {k: p.get(k) for k in ('up_limit', 'dl_limit', 'alt_up_limit', 'alt_dl_limit', 'scheduler_enabled', 'limit_tcp_overhead', 'limit_lan_peers', 'limit_utp_rate', 'max_uploads', 'enable_upload_suggestions')})
|
||||
print('uploadLimit', req('GET', '/api/v2/transfer/uploadLimit')[1], 'speedLimitsMode', req('GET', '/api/v2/transfer/speedLimitsMode')[1])
|
||||
elif mode == 'add':
|
||||
bnd = uuid.uuid4().hex
|
||||
tor = open('/root/uwtest.torrent', 'rb').read()
|
||||
parts = []
|
||||
for k, v in (('savepath', '/mnt/felhom-drives/scratch_hdd/userdata/media/torrents'), ('skip_checking', 'true'), ('category', '')):
|
||||
parts.append(f'--{bnd}\r\nContent-Disposition: form-data; name="{k}"\r\n\r\n{v}\r\n'.encode())
|
||||
parts.append(f'--{bnd}\r\nContent-Disposition: form-data; name="torrents"; filename="uwtest.torrent"\r\nContent-Type: application/x-bittorrent\r\n\r\n'.encode() + tor + b'\r\n')
|
||||
parts.append(f'--{bnd}--\r\n'.encode())
|
||||
print('add', req('POST', '/api/v2/torrents/add', b''.join(parts), {'Content-Type': f'multipart/form-data; boundary={bnd}'}))
|
||||
time.sleep(3)
|
||||
print('torrents', [(t['name'], t['state'], t['progress']) for t in req('GET', '/api/v2/torrents/info')[1]])
|
||||
elif mode == 'setlimit':
|
||||
print('setUploadLimit', req('POST', '/api/v2/transfer/setUploadLimit', urllib.parse.urlencode({'limit': sys.argv[3]}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||
print('uploadLimit now', req('GET', '/api/v2/transfer/uploadLimit')[1], 'prefs.up_limit', req('GET', '/api/v2/app/preferences')[1].get('up_limit'))
|
||||
elif mode == 'setpref':
|
||||
print('setPreferences', req('POST', '/api/v2/app/setPreferences', urllib.parse.urlencode({'json': json.dumps({'up_limit': int(sys.argv[3])})}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||
print('uploadLimit now', req('GET', '/api/v2/transfer/uploadLimit')[1], 'prefs.up_limit', req('GET', '/api/v2/app/preferences')[1].get('up_limit'))
|
||||
elif mode == 'peer':
|
||||
print('addPeers', req('POST', '/api/v2/torrents/addPeers', urllib.parse.urlencode({'hashes': 'all', 'peers': sys.argv[3]}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||
print('reannounce', req('POST', '/api/v2/torrents/reannounce', urllib.parse.urlencode({'hashes': 'all'}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||
elif mode == 'post':
|
||||
print(sys.argv[3], req('POST', sys.argv[3], urllib.parse.urlencode(json.loads(sys.argv[4])).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||
elif mode == 'sample':
|
||||
dur, step = int(sys.argv[3]), int(sys.argv[4])
|
||||
t0 = time.time(); prev = None
|
||||
while time.time() - t0 <= dur:
|
||||
ti = req('GET', '/api/v2/transfer/info')[1]
|
||||
tor = req('GET', '/api/v2/torrents/info')[1]
|
||||
etx, ecpu = engine_stats()
|
||||
row = {'t': round(time.time() - t0, 1), 'qb_up_speed': ti['up_info_speed'], 'qb_up_data': ti['up_info_data'],
|
||||
'qb_tor_uploaded': sum(x.get('uploaded', 0) for x in tor), 'proc_tx': netdev(), 'engine_tx': etx, 'cli': cli_stats()}
|
||||
if prev:
|
||||
dt = row['t'] - prev['t']
|
||||
row['rate_qb_data_Mbit'] = round((row['qb_up_data'] - prev['qb_up_data']) * 8 / dt / 1e6, 2)
|
||||
row['rate_proc_Mbit'] = round((row['proc_tx'] - prev['proc_tx']) * 8 / dt / 1e6, 2)
|
||||
row['rate_engine_Mbit'] = round((row['engine_tx'] - prev['engine_tx']) * 8 / dt / 1e6, 2)
|
||||
print(json.dumps(row), flush=True); prev = row
|
||||
time.sleep(step)
|
||||
@@ -0,0 +1,32 @@
|
||||
import sqlite3,sys,collections
|
||||
def runs(r, pred):
|
||||
best=cur=0; bt=None; prev=None
|
||||
for t,v in r:
|
||||
if pred(v) and prev is not None and t-prev<=180: cur+=1
|
||||
elif pred(v): cur=1
|
||||
else: cur=0
|
||||
if cur>best: best=cur; bt=t
|
||||
prev=t
|
||||
return best,bt
|
||||
import datetime
|
||||
for box in sys.argv[1:]:
|
||||
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
|
||||
nc=None
|
||||
rows=collections.defaultdict(list)
|
||||
for ts,name,cpu,tx in db.execute('select ts,container_name,cpu_percent,net_tx_bytes from container_metrics order by ts'):
|
||||
rows[name].append((ts,cpu,tx))
|
||||
print('==',box.rsplit('/',1)[1])
|
||||
out=[]
|
||||
for name,r in rows.items():
|
||||
cpu=[(t,c) for t,c,_ in r]
|
||||
tx=[]
|
||||
for (t0,_,a),(t1,_,b) in zip(r,r[1:]):
|
||||
if a is None or b is None or b<a or not 0<t1-t0<=180: continue
|
||||
tx.append((t1,(b-a)*8/(t1-t0)/1e6))
|
||||
c150,ct=runs(cpu,lambda v:v>=150); c80,_=runs(cpu,lambda v:v>=80)
|
||||
u5,ut=runs(tx,lambda v:v>=5); u1,_=runs(tx,lambda v:v>=1)
|
||||
if max(c80,u1)>=3: out.append((name,c80,c150,ct,u1,u5,ut))
|
||||
print(f'{"container":28} cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]')
|
||||
for name,c80,c150,ct,u1,u5,ut in sorted(out,key=lambda x:-max(x[1],x[4])):
|
||||
f=lambda t: datetime.datetime.utcfromtimestamp(t).strftime('%m-%d %H:%M') if t else '-'
|
||||
print(f'{name[:28]:28} {c80:8} {c150:9} ({f(ct)}) {u1:9} {u5:9} ({f(ut)})')
|
||||
@@ -0,0 +1,21 @@
|
||||
# Minimal BitTorrent HTTP tracker for a closed scratch test (no internet). Compact peer lists.
|
||||
import http.server, urllib.parse, socket, struct, time
|
||||
peers = {} # info_hash -> {(ip,port): last_seen}
|
||||
def benc(o):
|
||||
if isinstance(o, int): return b"i%de" % o
|
||||
if isinstance(o, bytes): return b"%d:%s" % (len(o), o)
|
||||
if isinstance(o, str): return benc(o.encode())
|
||||
if isinstance(o, dict): return b"d" + b"".join(benc(k) + benc(v) for k, v in sorted(o.items())) + b"e"
|
||||
class H(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
q = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query.encode('latin-1').decode('latin-1'), encoding='latin-1')
|
||||
ih = q.get('info_hash', [''])[0]; port = int(q.get('port', ['0'])[0]); ip = self.client_address[0]
|
||||
d = peers.setdefault(ih, {}); now = time.time()
|
||||
if q.get('event', [''])[0] == 'stopped': d.pop((ip, port), None)
|
||||
else: d[(ip, port)] = now
|
||||
lst = b"".join(socket.inet_aton(i) + struct.pack(">H", p) for (i, p), t in d.items() if now - t < 600 and (i, p) != (ip, port))
|
||||
body = benc({"interval": 30, "peers": lst})
|
||||
print(time.strftime('%H:%M:%S'), 'announce', ip, port, q.get('event', [''])[0], 'peers', len(lst) // 6, flush=True)
|
||||
self.send_response(200); self.send_header('Content-Length', str(len(body))); self.end_headers(); self.wfile.write(body)
|
||||
def log_message(self, *a): pass
|
||||
http.server.ThreadingHTTPServer(('0.0.0.0', 6969), H).serve_forever()
|
||||
@@ -0,0 +1,47 @@
|
||||
# Runs INSIDE guest 9202. Transmission's own RPC numbers vs the container's exact tx bytes. Password read at use, never printed.
|
||||
import json, subprocess, sys, time, urllib.request, base64
|
||||
def sh(*a): return subprocess.run(a, capture_output=True, text=True).stdout.strip()
|
||||
C = 'transmission'
|
||||
IP = sh('docker', 'inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}', C).split()[0]
|
||||
PID = sh('docker', 'inspect', '-f', '{{.State.Pid}}', C)
|
||||
PW = sh('docker', 'exec', C, 'printenv', 'PASS')
|
||||
AUTH = 'Basic ' + base64.b64encode(('admin:' + PW).encode()).decode()
|
||||
SID = ''
|
||||
def rpc(method, args=None):
|
||||
global SID
|
||||
for _ in range(2):
|
||||
r = urllib.request.Request(f'http://{IP}:9091/transmission/rpc', data=json.dumps({'method': method, 'arguments': args or {}}).encode(),
|
||||
headers={'Authorization': AUTH, 'Host': 'transmission:9091', 'X-Transmission-Session-Id': SID, 'Content-Type': 'application/json'}, method='POST')
|
||||
try:
|
||||
with urllib.request.urlopen(r, timeout=20) as resp: return json.loads(resp.read())
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 409: SID = e.headers.get('X-Transmission-Session-Id'); continue
|
||||
return {'http': e.code}
|
||||
def tx():
|
||||
for line in open(f'/proc/{PID}/net/dev'):
|
||||
if line.strip().startswith('eth0:'): return int(line.split(':', 1)[1].split()[8])
|
||||
mode = sys.argv[1]
|
||||
if mode == 'info':
|
||||
g = rpc('session-get')['arguments']
|
||||
print('version', g.get('version'), {k: g.get(k) for k in ('speed-limit-up', 'speed-limit-up-enabled', 'alt-speed-up', 'alt-speed-enabled', 'alt-speed-time-enabled', 'units')})
|
||||
print('session-stats', rpc('session-stats')['arguments'])
|
||||
elif mode == 'add':
|
||||
meta = base64.b64encode(open('/root/uwtest.torrent', 'rb').read()).decode()
|
||||
print(rpc('torrent-add', {'metainfo': meta, 'download-dir': '/mnt/felhom-drives/scratch_hdd/userdata/media/transmission', 'paused': False}))
|
||||
time.sleep(5); rpc('torrent-verify', {}); time.sleep(20)
|
||||
print(rpc('torrent-get', {'fields': ['name', 'status', 'percentDone', 'error', 'errorString']}))
|
||||
elif mode == 'limit':
|
||||
print(rpc('session-set', {'speed-limit-up': int(sys.argv[2]), 'speed-limit-up-enabled': True}))
|
||||
g = rpc('session-get')['arguments']; print({k: g.get(k) for k in ('speed-limit-up', 'speed-limit-up-enabled')})
|
||||
elif mode == 'reannounce':
|
||||
print(rpc('torrent-reannounce', {}))
|
||||
elif mode == 'sample':
|
||||
dur, step = int(sys.argv[2]), int(sys.argv[3]); t0 = time.time(); prev = None
|
||||
while time.time() - t0 <= dur:
|
||||
st = rpc('session-stats')['arguments']; tg = rpc('torrent-get', {'fields': ['uploadedEver', 'rateUpload']})['arguments']['torrents']
|
||||
row = {'t': round(time.time() - t0, 1), 'tr_upspeed': st['uploadSpeed'], 'tr_cur_up': st['current-stats']['uploadedBytes'], 'tr_tor_up': sum(x['uploadedEver'] for x in tg), 'tx': tx()}
|
||||
if prev:
|
||||
dt = row['t'] - prev['t']
|
||||
row['rate_tr_Mbit'] = round((row['tr_cur_up'] - prev['tr_cur_up']) * 8 / dt / 1e6, 2)
|
||||
row['rate_container_Mbit'] = round((row['tx'] - prev['tx']) * 8 / dt / 1e6, 2)
|
||||
print(json.dumps(row), flush=True); prev = row; time.sleep(step)
|
||||
@@ -65,6 +65,15 @@ Two first mutations broke the build instead of a test (an unused variable) and w
|
||||
compiler, failed. The first version of the file test passed with the VACUUM removed — the token row was the last cell on
|
||||
its page, so the new row overwrote it; a second customer written after it (as on the live hub) made the test honest.
|
||||
|
||||
**Run on a private copy of the real database** (the 14:00Z snapshot, in a 0700 scratch folder, with a THROWAWAY key —
|
||||
the real one was never read; a temporary test file, deleted, never committed): the start-up sealing sealed **8** tokens
|
||||
in **3.15 s** (incl. the VACUUM), a second run sealed **0**, all **5 of 5** customers' `config_json` read back equal to
|
||||
the original (JSON-equal), and **0 of the 8** token values were found anywhere in the database files afterwards. A first
|
||||
attempt compared whole records and showed „0 of 5 equal": the R-879 columns were sealed with the REAL key, so with the
|
||||
throwaway key every record was flagged unreadable — the expected behaviour, not a defect; the second run compared the
|
||||
JSON only. The copy was shredded after. Image: `gitea.dooplex.hu/admin/felhom-hub:0.168.0`
|
||||
(`sha256:3bfdecaae527d4762059600cdfaad83daf4e4b5b20661fc47acf128180d87cb3`), from `5277d859`, CI run 2269 success.
|
||||
|
||||
**Measured on the live file (count of the key name only):** `cf_tunnel_token` occurs 12 times both in the live
|
||||
`hub.db` and in the clean `VACUUM INTO` snapshot, so the live file holds no extra old copies today.
|
||||
|
||||
|
||||
@@ -218,7 +218,7 @@ stopping line that lies.
|
||||
| **R-1094** | Storage & devices | P4 | **The whole-disk format refusal tells the household to format the free part — which is refused while the other part is in use.** SEEN 2026-10-16 (controller 0.318.0): 409 „Ezen a lemezen egy használatban lévő meghajtó van, ezért a teljes lemez nem formázható. Csak a szabad részét formázhatod." — since agent 0.158.1 a part next to a mounted part can only be USED (the root wrapper trusts no „Felhom's own" claim). Evidence: `audits/storage-identity-buildB-2026-10-15/live/14-samedisk-refusal-offer.txt`. | READY | — | Reword `err.web.meghajto_hasznalatban_formazas` (hu + en): „… nem formázható. A szabad részét használatba veheted." Next controller release; the copy gates run. | CC |
|
||||
| **R-1213** | Storage & devices | P4 | **Operator detail still on two household pages:** the registered drive card's `ext4 · /dev/sdb1 · Model` line (`templates/storage.html`; W14 hid only the agent drive cards), and the heading „Távoli mentés (3. mentés) — titkosított, offsite" with the English „offsite". Found by the night helper 2026-10-21 (W11/W14 follow-up). | OPEN | — | Move the line behind „Részletek"; „…titkosított, házon kívüli" | CC |
|
||||
|
||||
## Security & access — 22 rows (P1 2, P2 3, P3 11, P4 6)
|
||||
## Security & access — 23 rows (P1 2, P2 3, P3 12, P4 6)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -244,6 +244,7 @@ stopping line that lies.
|
||||
| **R-1054** | Security & access | P3 | **The hub's operator/global API key is still one bearer that can do everything on `/api/v1`, including reading every box's break-glass root password (`GET /admin/hosts/{id}/recovery-credential`).** Hub v0.152.0 gave the web surface scoped tokens (`05` §17) but tokens are refused on `/api/v1` by design. Who uses the global key today is not listed in one place. Fix: list its consumers; move each operator use to a scoped token on the web side (or a token class on the API), and keep the global key only where a box path needs it — or retire it. | OPEN — filed 2026-10-15 by the token build | — | List the global key's consumers (pod env, scripts, runbooks) | CC |
|
||||
| **R-1194** | Security & access | P3 | **On a fresh box the operator's break-glass SSH door was broken from the first minute: `oob_degraded` „felhom-sshd config invalid (sshd -t fails)".** Rehearsal 2026-10-20, Tester4-816e21, mail at 12:03 UTC (one minute after the host record was created): details `active:false, config_invalid:true, felhom_sshd_port:0, operator_key_configured:false, operator_key_reported:true, reachable:false`. Possibly the shape of a customer with no operator key set (Tester4 is a test customer) — then the alarm is a false one on every such box; or a real config defect. Not diagnosed (no shell). Checked the same day: the only `oob_degraded` in the operator mailbox in 30 days, and no recovery mail followed; the report says port 0. Two candidate mechanisms in the agent (`internal/felhomsshd/health.go:63-66`): `sshd -t -f <conf>` run by the NON-root agent fails on unreadable host keys, or a config rendered before the desired state arrived (port 0). **CAUSE FOUND AND FIXED 2026-10-21 night** (`audits/night-2026-10-21/P5-checks/`): reproduced on a fresh box from the public ISO — `oob_degraded` mailed 20:34 CEST while felhom-sshd was applied 0.4 s after the check and is healthy (port 8822 listening, `sshd -t` passes as the agent's user through its sudo grant — so the non-root theory is FALSE; without sudo it fails on the host key, which the agent never does). The first host report runs before the felhom-sshd loop's first pass and reads a config never rendered. Fix: agent `0da792e` — no OOB stanza until the first pass has run (the hub does not evaluate a missing one; a failed pass counts). `TestR1194_NoOOBStanzaBeforeTheFirstPass` red-proved. | WATCHING — released in agent v0.162.0 (2026-10-22, operator yes; on the 3 test boxes); the false alarm comes only on a FRESH box's first report | the next fresh install with agent ≥ 0.162.0 | At the next fresh install read the hub's events for `oob_degraded` in the first hour (none expected); then close | CC |
|
||||
| **R-1237** | Security & access | P3 | **The nightly CC-secrets sweep found stored one-time codes in a session record dated 2026-10-23: 14 names in one file (bind tokens and setup codes of test boxes — NIGHT_2026_10_23_*, FIXES_2026_10_21_*, NIGHT_2026_10_21_*, REHEARSAL_2026_10_20_* — and R_DEMO_FELHOM in a file of 2026-10-20).** Mail „[DooPlex CC secrets] 2 secret name(s) found in old session records", 2026-10-23 01:57 UTC, read in the day session (names only). One file holding all of them suggests one command printed many stored values at once (a listing or a dump) in the night session. All but R_DEMO_FELHOM are one-time install codes of test boxes; NIGHT_2026_10_23_* die when Tester4-16fdef is deleted (the operator's do list). | OPEN | — | CC: find the file and the command by the sweep's own report (names only), add a hook shape for that command if it is not one; R_DEMO_FELHOM (demo-felhom's recovery code) — the operator decides whether to rotate it | CC |
|
||||
| **R-1248** | Security & access | P3 | **An app's own numbers can lie: an attacker with code inside an app's container can answer the app's API with whatever matches its flood, and the upload watch (R-1244) would then call it explained.** Measured remedy, not built: a per-container upload ceiling with `tc` from a helper container in the app's network namespace — on scratch 9202 (unprivileged) a flood was held at 19.9–20.1 Mbit/s and the app kept answering (`audits/SPIKE-upload-watch-2026-10-23.md` §5). Costs: a helper image with `tc` (a new external image, or `tc` in the controller image), `NET_ADMIN` for that helper, re-applied after each app restart. | OPEN | an operator decision (the new image / capability) | Decide; if yes, CC builds the ceiling for the download and media apps first | operator |
|
||||
|
||||
## Box system & updates — 12 rows (P2 1, P3 7, P4 4)
|
||||
|
||||
@@ -262,12 +263,13 @@ stopping line that lies.
|
||||
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
|
||||
|
||||
|
||||
## Monitoring & notifications — 20 rows (P2 1, P3 13, P4 6)
|
||||
## Monitoring & notifications — 23 rows (P2 2, P3 14, P4 7)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
|
||||
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
|
||||
| **R-1244** | Monitoring & notifications | P2 | **The upload and CPU watch is built and not released: until it is, a hacked app on a household box can send hundreds of GB or mine for weeks and nobody is told.** Operator ruling 2026-10-23 19:22; built 2026-10-23 night — controller v0.342.0 (`internal/upwatch`) + hub v0.169.0 (the four operator-only events), both committed, NOT released; proven on scratch 9202 with `0.342.0-rc.1` (`audits/SPIKE-upload-watch-2026-10-23.md` §7; `08` §9 [DESIGN — proposed]). | WAITING-ON-OPERATOR | the operator's yes; R-1245 and R-1246 | Operator: yes → build hub 0.169.0 + deploy, then the controller through the normal release path (Tester 1, demo-hp, demo-felhom, box walk) | operator |
|
||||
| **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Design note (2026-10-21, after R-1192's diagnosis): the freeze's signature is cheap to see from the HOST side — the guest cgroup's memory PSI „full" above ~50 % for a minute, or its `memory.events max` climbing, while the host still reports; the agent could raise `guest_memory_thrash` (operator mail in its own words, not „no report for 45 m") and the household banner „A dobozod most nagyon lassú…". With the swap fix (R-1192) the measured thrash ends within a minute, so this alarm should be built only if a freeze is seen again after the agent release; until then it stays a design. Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC |
|
||||
| **R-1202** | Monitoring & notifications | P3 | **A box whose clock is behind blames the hub and stops an app without telling anyone.** Rehearsal 2026-10-20 B6 #8: RTC set 2 years back (the OS raised it to 2026-04-13): the dashboard shows only „Hub nem elérhető" while the log says „x509: certificate … not yet valid: current time 2026-04-13…"; Paperless crash-looped and the box stopped it with a restore hold; the `app_stopped_unhealthy` event failed (hub unreachable), so neither the household nor the operator heard. After the real time came back the app stayed stopped until a Start (clear sentence, 1 min). | **OPEN** | — | say „the box's clock is wrong (it says <date>)" when TLS fails with not-yet-valid/expired against a hub that answers; keep the stop event queued until the hub is reachable | CC |
|
||||
| **R-340** | Monitoring & notifications | P3 | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice** **REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc/<pid>/fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC |
|
||||
@@ -283,8 +285,11 @@ stopping line that lies.
|
||||
| **R-1211** | Monitoring & notifications | P3 | **The household's `backup_failed` mail says only „A biztonsági mentés sikertelen! Kérjük, ellenőrizd a rendszert." — no app named, no next step, and the formal „Kérjük".** Rehearsal 2026-10-20 (the one alarm the household got, `audits/rehearsal-2026-10-20/B9-words/words.md` W25); the box's own page now says which app and that the box retries the next night (controller `5776d54`, W25), but the mail is the hub's `customerMessages` entry. Found by the night helper 2026-10-21. | OPEN | — | Hub: build the household mail from the event's app names and the classifier's class, in the te-form, with „a doboz a következő éjjel újra próbálkozik"; mail golden | CC |
|
||||
| **R-1222** | Monitoring & notifications | P3 | **A household that sets an EVENING backup time gets a false „database dump missed” error every morning.** REAL in code (night 2026-10-22; split from R-1189 (e)): the hub's 05:00 check wants a dump since Budapest midnight (`hub/internal/monitor/deadline.go:416-419, 555-565`), so an evening dump (the banner suggests evening times) is always „yesterday”; on 2026-10-25 the check also runs at 04:00 (R-1183). Evidence `audits/night-2026-10-22/P3-candidates/analysis.md`. | OPEN | — | Fix: the check wants a dump within the last 26 h (or since the box's own scheduled time); a hub test with an evening schedule; rides a hub deploy | CC |
|
||||
| **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC |
|
||||
| **R-1245** | Monitoring & notifications | P3 | **DECIDE: how large is qBittorrent's default upload limit?** The box sets it from the measured line and never over the household's own limit. CC's pick: **50 % of the measured upload** (the family keeps half the line; a seeding qBittorrent measured 131 Mbit/s on the bench — the whole line). Other: 30 % (gentler on the family, slower seeding) or none (only the alarm). `audits/SPIKE-upload-watch-2026-10-23.md` §4, §6.6. | WAITING-ON-OPERATOR | — | Operator: say a percentage (or none) | operator |
|
||||
| **R-1246** | Monitoring & notifications | P3 | **DECIDE: who is told when the box limits or flags something?** CC's pick: the family only about the qBittorrent limit (one line on its page; they can change it); the unexplained-upload and unexplained-CPU alarms reach the operator only (R-388: the household can do nothing about a hacked app but wait for support). Other: tell the family too (that support was told). `audits/SPIKE-upload-watch-2026-10-23.md` §6.4. | WAITING-ON-OPERATOR | — | Operator: pick | operator |
|
||||
| **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
|
||||
| **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
|
||||
| **R-1247** | Monitoring & notifications | P4 | **The box's upload line is only a lower bound, learnt from its own off-site push; most nights the push is too small to say anything.** Measured: at a one-minute sample the push exceeded 2 Mbit/s on 3 of 81 box-nights; a one-second sample during the push is now built (a night counts when its best 10 s moved ≥ 10 MB). An ACTIVE test (a timed upload to the box's own Storage Box sub-account, or to ep0) would give the real line — not built: a design for the test file beside the restic repository, a quiet hour outside 02:30–05:30, ~125 MB per test at 100 Mbit/s. `audits/SPIKE-upload-watch-2026-10-23.md` §2, §3. | OPEN | an operator decision (data cost, where the test writes) | Decide whether an active test is wanted; if yes, CC designs it | operator |
|
||||
|
||||
|
||||
## Hub & operator — 12 rows (P3 5, P4 7)
|
||||
|
||||
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-10-23 (night)
|
||||
|
||||
**Updated 2026-10-23 (evening).** The Plex attack on DooPlex did NOT reach Felhom's secrets: the two open doors to them (a shared backup folder and Longhorn's page) were not used in the attack days. The backup folder is now closed to the home network (your yes); it is still open to every app on DooPlex, so a hacked app there could read the hub's backups (R-1241, the homelab session's fix).
|
||||
|
||||
## Moved from STATUS on 2026-10-23 (evening)
|
||||
|
||||
**Updated 2026-10-23 (afternoon).** Your D39 = B is built and live: an app with a public security hole can now get its fix at night on every box, after one test on the test box — controller 0.341.0 is the floor since 13:24 (your yes). The test: Plex moved by itself at night on the test box, and an older controller safely waited for a hand press. Nothing needs you.
|
||||
|
||||
Reference in New Issue
Block a user