v0.342.0 (unreleased): the upload and CPU watch — an app that sends or burns far more than its own numbers explain is one operator-only alarm; the line learnt from the off-site push; qBittorrent's default upload limit (never over the household's)
gates / gates (push) Successful in 1m17s
gates / gates (push) Successful in 1m17s
internal/upwatch: exact per-container counters from the Docker Engine API, per-app judging (upload >= max(5 Mbit/s, 25% of the line) for 15 min; CPU >= max(150%, 50% x cores) for 30 min), explainers qBittorrent / Transmission / Plex, app_unexplained_upload / app_unexplained_cpu + _cleared (hub 0.169.0). The off-site push is sampled once a second for the line (a lower bound). qBittorrent limit = 50% of the line, the household's limit always wins. 20 red-proofs. Spike + design: felhom.eu audits/SPIKE-upload-watch-2026-10-23.md, architecture/08 section 9. Not released.
This commit is contained in:
@@ -1,3 +1,43 @@
|
||||
## v0.342.0 (unreleased — committed 2026-10-23 night, NOT released; a test image `0.342.0-rc.1` for scratch 9202 only) — the upload and CPU watch (operator ruling 2026-10-23 19:22)
|
||||
|
||||
**MinAgent: 0.158.1** (unchanged). **Hub dependency:** the four new event types need hub 0.169.0 (committed the same
|
||||
night, not built); an older hub answers 400 and the event is dropped (logged), nothing else breaks. Task: the weekend-
|
||||
night brief, Part 2 (spike → design → build, commit only). Design: felhom.eu `architecture/08-alarm-ladder.md` §9
|
||||
[DESIGN — proposed]; spike numbers: felhom.eu `audits/SPIKE-upload-watch-2026-10-23.md`.
|
||||
|
||||
- **`internal/upwatch` — the watch.** Once a minute it reads every app container's EXACT sent bytes and CPU time from the
|
||||
Docker Engine API on the socket (`DockerAPI`; `docker stats`' text has three significant digits — ~1.3 Mbit/s steps at a
|
||||
one-minute sample), adds them up per app (the compose project) and judges each app: **upload** at ≥ max(5 Mbit/s, 25 % of
|
||||
the measured line) for 15 minutes; **CPU** at ≥ max(150 %, 50 % × cores) of one core for 30 minutes. Then it ASKS THE APP
|
||||
(`Explainer`): qBittorrent's `up_info_data`, Transmission's `current-stats.uploadedBytes` (bytes it counted — explained
|
||||
when the container sent ≤ 1.25 × that + 1 Mbit/s; measured overhead 3–7 % and 0.1–2 %), Plex's `/status/sessions` (any
|
||||
viewer explains upload; no viewer and a big upload is the Plex2Shell signature) and `/transcode/sessions` (any
|
||||
transcode explains CPU). Not explained → **`app_unexplained_upload` / `app_unexplained_cpu`** (warning, operator-only on
|
||||
the hub), ONE per episode, and a `_cleared` (info) after 15 quiet minutes. An app the box cannot ask is judged only once
|
||||
it has ~20 hours of history, at ≥ 10 × its own p95 for an hour, worded „no app number to compare". The box's own work
|
||||
on the app (a backup run, an update, a deploy) is never judged. Infrastructure (the controller, traefik, cloudflared,
|
||||
Samba) is never judged. Passwords and tokens are read at use (the mediawire seams, `09` §3 decision 248), never stored.
|
||||
- **The upload line, learnt from the off-site push** (`line.go`, `backup.SetOffboxPushObserver`): during the push the
|
||||
controller samples its OWN sent bytes once a second; the best 10-second rate is kept per night (14 nights,
|
||||
settings.json `upwatch.line`) and the line is the highest night whose best 10 seconds moved ≥ 10 MB — a lower bound,
|
||||
shown as „legalább N Mbit/s" on the system settings page („még mérjük" until then).
|
||||
- **qBittorrent's default upload limit** (`qblimit.go`): 50 % of the measured line (CC's pick; decide row in felhom.eu
|
||||
STATUS), checked every 15 minutes, set through its Web API, re-set when the line moves > 20 %, and NEVER over a limit the
|
||||
household set — a limit that was there first, or any change from the value qBittorrent stored for the box (it keeps
|
||||
KiB/s; the stored value is compared), makes the limit the household's for good. No line yet → nothing is set. While the
|
||||
box owns the limit, qBittorrent's page says one line: „A qBittorrent feltöltése legfeljebb N Mbit/s, hogy a családi
|
||||
internet gyors maradjon. A qBittorrentben átállíthatod."
|
||||
- `metrics.StackTxP95Mbps` (the per-app baseline), `settings.Upwatch` (one opaque block), `notify.NotifyUnexplained`.
|
||||
- Tests, each red-proved (20 mutations, each seen failing): `TestUpwatch_SeedingQBittorrentIsExplained`,
|
||||
`…_HackedQBittorrentAlarmsOnceAndClears`, `…_PlexWithNoViewerAlarmsWithAViewerDoesNot`,
|
||||
`…_NoExplainerLearnsThenNeedsTenTimesItsP95ForAnHour`, `…_CPUBurnerAlarmsTranscodeAndBoxWorkDoNot`,
|
||||
`…_RestartAndInfrastructureAreNotJudged`, `…_AnAppThatDoesNotAnswerIsNotAnExplanation`, `…_TriggerFollowsTheLine`,
|
||||
`TestLine_SummarizePeakAndCounts`, `TestQBLimit_TheHouseholdAlwaysWins`, `TestQBittorrent_AskKeepsTheSessionAndReLogsInOn403`,
|
||||
`TestTransmission_AskFollowsTheSessionID`, `TestPlex_AskReadsSessionsWithTheTokenOnlyWhenClaimed`,
|
||||
`TestStackTxP95Mbps_SumsContainersPerMinute`, `TestNotifyUnexplained_TypesSentencesAndDetails`,
|
||||
`TestUpwatch_QBLimitLineOnQBittorrentsPageOnly`, `TestUpwatchIsWiredAtStartup`, `TestNewUpwatch_ExplainersAndEmit`;
|
||||
the system page's four frozen Hungarian renders gained the one new row (diffed: nothing else moved).
|
||||
|
||||
## v0.341.0 (2026-10-23) — the night update takes a security-fix step (D39 = B, `09` §3 decision 255, R-1239)
|
||||
|
||||
**Released 2026-10-23: global floor 11:24:47 UTC (the operator's yes in chat), MinAgent 0.158.1; image `sha256:8aa4ce60…c0bc`
|
||||
|
||||
@@ -386,6 +386,8 @@
|
||||
| `report.SetPendingControllerLog` / `SetControllerLogSource` | controller/internal/report/selftail.go | ACK-armed consume-once self-log pull (the logtail.go shape) | selftail_test.go; source = `logBuffer.Lines`, wired once in main.go |
|
||||
| `util.ParseVersion` / `util.Version.Compare` | controller/internal/util/version.go | THE one semver comparator (house rule: never a second) — selfupdate aliases it; agentapi's MinAgent comparison uses it | rejects pre-release/dev/latest (callers fall back, never trust); numeric compare (0.100 > 0.81) |
|
||||
| `util.ParseVersionPre` | controller/internal/util/version.go | the same comparator, reading `X.Y.Z-rc.N` as BELOW `X.Y.Z` (v0.314.0, R-1053) — ONLY for „is this box below the floor?" | never use it to pick a tag to offer: the button must not see a test build |
|
||||
| `upwatch.Source` / `upwatch.Explainer` / `Watcher.Now` (v0.342.0, the upload watch) | controller/internal/upwatch/ | `DockerAPI` (Engine API on the socket, exact counters) / the qBittorrent, Transmission and Plex clients | `fakeSource`, `fakeApp` and a minute clock in watch_test.go drive the WHOLE judge path; the app clients are tested against httptest servers that behave as measured on 9202. **Exact counters come from the Engine API, never from `docker stats` text** (three significant digits) |
|
||||
| `upwatch.QBittorrent.SetUploadLimit` + `ApplyQBLimit` | controller/internal/upwatch/qblimit.go | the one writer of qBittorrent's global upload limit | compare against the value qBittorrent STORED (KiB/s rounding), never the requested one; a limit the household touched is theirs for good |
|
||||
| `mediawire.PlexURL` | controller/internal/mediawire/plexaddr.go | Plex's base URL BY ADDRESS, resolved at each call (v0.314.0, R-1060) — every controller call to Plex | Plex refuses the Host name `plex` as non-local (401); never cache the address |
|
||||
| `stacks.GateCapture` / `stacks.ReadGateAtBackup` | controller/internal/stacks/restore_gate.go | the setup gate a per-app backup records, and the restore's read of it (v0.314.0, R-1057) | the unit `app.yaml` key is `setup_gate:` — the live file's keys, so either reads the same |
|
||||
| `agentapi.AgentVersionReporter` + `featureMinAgent` | controller/internal/agentapi/features.go | version-first Supports (v0.82.0 header channel); probe = fallback for header-less agents | a coupled feature adds BOTH a featureProbes row AND a featureMinAgent row; v0.116.0: `SupportsWithSource` also reports HOW the verdict was reached (version/probe-cache/probe) for the gate log line |
|
||||
@@ -447,4 +449,5 @@ Cross-repo edges:
|
||||
| JSON writers ×5, 3 envelope shapes | api `writeJSON`; web `writeDiskJSON`, `jsonResponse`/`jsonError`, `writeDebugJSON` |
|
||||
| Safe-name validators ×4 | controller/internal/web/validate.go `validStackName`; controller/internal/api/router.go `validStackParam` (same body — api↔web import cycle); controller/internal/backup/offbox.go `isSafeStackName`; controller/internal/appexport/validate.go `ValidateSegment` (strictest) |
|
||||
| DB wait/import ×2 | controller/internal/appbackup/dbdump.go `waitDBReady`/`ImportDump` vs controller/internal/appexport/restore.go `waitForDB`/`importDBDump` |
|
||||
| qBittorrent Web API login ×2 | controller/internal/mediawire/mediawire.go `qbLogin` (writes the drive wiring) vs controller/internal/upwatch/explain.go `QBittorrent.login` (reads every minute; keeps its session cookie, which mediawire's per-run login does not need) |
|
||||
| compose exec ×2 | controller/internal/stacks/manager.go `composeExecCustomEnv` vs controller/internal/appexport/restore.go `composeExecEnv` (the latter has ctx+timeout; the former has the userdata belt) |
|
||||
|
||||
@@ -3524,6 +3524,18 @@ no key into the household's account). Code: `internal/stacks/after_unclean_boot.
|
||||
- **Plex:** the claim code is an `after_start` field — asked on the Plex page once Plex runs (`POST /apps/plex/plex-claim`);
|
||||
the gate opens only when Plex says claimed, and „Kész" is refused before that.
|
||||
|
||||
### 8e. The upload and CPU watch (v0.342.0, unreleased; felhom.eu `08` §9 [DESIGN — proposed])
|
||||
|
||||
Once a minute `internal/upwatch` reads every app container's exact sent bytes and CPU time (Docker Engine API), adds them
|
||||
up per app and, when an app sends a lot or burns CPU for a long time, asks the app itself whether that is its work:
|
||||
qBittorrent's and Transmission's own upload counters, Plex's playing and transcoding sessions. Explained → normal use.
|
||||
Not explained → one operator-only event (`app_unexplained_upload` / `app_unexplained_cpu`) and a clear when it stops.
|
||||
Numbers: upload ≥ max(5 Mbit/s, 25 % of the line) for 15 min; CPU ≥ max(150 %, 50 % × cores) for 30 min; an app with no
|
||||
explainer only at ≥ 10 × its own p95 for an hour, after ~20 h of history. The upload line is learnt from the nightly
|
||||
off-site push (one-second samples of the controller's own counter; best 10 s; a lower bound) and shown on the system
|
||||
settings page. qBittorrent gets a default upload limit of 50 % of that line (never over a household's own limit), and its
|
||||
page says so in one line. State: settings.json `upwatch`. Spike: felhom.eu `audits/SPIKE-upload-watch-2026-10-23.md`.
|
||||
|
||||
### 8d. Storage that knows its physical disks — the internal place, partitions (v0.317.0 build A, v0.318.0 build B)
|
||||
|
||||
Design: felhom.eu `architecture/16-storage-physical-identity.md`. The host agent decides which physical disk a place sits on;
|
||||
|
||||
@@ -685,6 +685,15 @@ func main() {
|
||||
// R-550: an interrupted restore is raised once, now that the notifier exists.
|
||||
reportInterruptedRestore(notifier, interruptedRestore)
|
||||
|
||||
// The upload and CPU watch (internal/upwatch; felhom.eu `08` §9, operator ruling 2026-10-23). Built here, where the
|
||||
// notifier first exists, and its push measurement handed to the off-site backup BEFORE any scheduler can start a
|
||||
// push. Wiring pinned by TestUpwatchIsWiredAtStartup.
|
||||
upwatchSvc := newUpwatch(sett, stackMgr, backupMgr, metricsStore, notifier, logger)
|
||||
if backupMgr != nil {
|
||||
backupMgr.SetOffboxPushObserver(upwatchSvc.PushObserver)
|
||||
}
|
||||
startUpwatch(ctx, upwatchSvc, logger)
|
||||
|
||||
// R-97a: wire the quiesce loop's hub-event seam. It MUST happen here and not in
|
||||
// startQuiesceLoop, because the notifier is constructed after it — and it must happen at all,
|
||||
// or the seam is inert (the "built but never wired" trap, four recorded instances). Reachability
|
||||
@@ -2044,6 +2053,8 @@ func main() {
|
||||
// URL is derived from it, so two builds of one version can never share a cache entry. Init-only,
|
||||
// before the HTTP server starts.
|
||||
webServer.SetBuildIdentity(GitCommit, BuildTime)
|
||||
// The upload watch's two page lines: the measured line (settings → system) and qBittorrent's limit (its app page).
|
||||
webServer.SetUpwatchView(upwatchSvc.View)
|
||||
// R-871 (decision 110): the missed-backup banner reads the night ledger and the box's own on/off record.
|
||||
webServer.SetMissedBackupBanner(webNightLedger, metricsStore)
|
||||
// Migration done-hook: a decommission-initiated migration finalizes the source decommission on
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/backup"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/mediawire"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/metrics"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/upwatch"
|
||||
)
|
||||
|
||||
// qbLimitShare is qBittorrent's default share of the measured upload line — CC's pick, open for the operator (decide
|
||||
// row in felhom.eu STATUS): 50 %.
|
||||
const qbLimitShare = 0.5
|
||||
|
||||
// baselineMinMinutes: an app the box cannot ask is judged only once its history holds ~20 hours of minutes.
|
||||
const baselineMinMinutes = 1200
|
||||
|
||||
// newUpwatch builds the upload and CPU watch (internal/upwatch; felhom.eu `08` §9). It does not start it.
|
||||
func newUpwatch(sett *settings.Settings, stackMgr *stacks.Manager, backupMgr *backup.Manager, metricsStore *metrics.MetricsStore,
|
||||
notifier *notify.Notifier, logger *log.Logger) *upwatch.Service {
|
||||
box := mediawire.DockerBox{}
|
||||
qb := &upwatch.QBittorrent{Box: box}
|
||||
svc := &upwatch.Service{Store: sett, QB: qb, QBRunning: func() bool { return box.Running("qbittorrent") }, QBShare: qbLimitShare, Log: logger}
|
||||
w := &upwatch.Watcher{
|
||||
Source: &upwatch.DockerAPI{},
|
||||
Explainers: map[string]upwatch.Explainer{
|
||||
"qbittorrent": qb,
|
||||
"transmission": &upwatch.Transmission{Box: box},
|
||||
"plex": &upwatch.Plex{Token: func() string { return mediawire.PlexToken(box) }, URL: mediawire.PlexURL},
|
||||
},
|
||||
Rules: upwatch.DefaultRules(),
|
||||
Line: svc.Line,
|
||||
Busy: func(stack string) bool {
|
||||
if backupMgr != nil && backupMgr.IsRunning() {
|
||||
return true
|
||||
}
|
||||
if st, ok := stackMgr.GetStack(stack); ok && st != nil && (st.Updating || st.Deploying) {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
},
|
||||
Emit: notifier.NotifyUnexplained,
|
||||
Log: logger,
|
||||
}
|
||||
if metricsStore != nil {
|
||||
w.Baseline = dailyBaseline(metricsStore)
|
||||
}
|
||||
svc.Watcher = w
|
||||
return svc
|
||||
}
|
||||
|
||||
// dailyBaseline caches each app's p95 for a day (a 7-day query is a few hundred thousand rows).
|
||||
func dailyBaseline(ms *metrics.MetricsStore) func(string, []string) (float64, bool) {
|
||||
type entry struct {
|
||||
at time.Time
|
||||
p95 float64
|
||||
ok bool
|
||||
}
|
||||
var mu sync.Mutex
|
||||
cache := map[string]entry{}
|
||||
return func(stack string, containers []string) (float64, bool) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if e, ok := cache[stack]; ok && time.Since(e.at) < 24*time.Hour {
|
||||
return e.p95, e.ok
|
||||
}
|
||||
p95, minutes, err := ms.StackTxP95Mbps(containers, time.Now().Add(-7*24*time.Hour))
|
||||
e := entry{at: time.Now(), p95: p95, ok: err == nil && minutes >= baselineMinMinutes}
|
||||
cache[stack] = e
|
||||
return e.p95, e.ok
|
||||
}
|
||||
}
|
||||
|
||||
// startUpwatch runs the minute watcher and the 15-minute qBittorrent limit rule until ctx ends.
|
||||
func startUpwatch(ctx context.Context, svc *upwatch.Service, logger *log.Logger) {
|
||||
go svc.Watcher.Run(ctx, time.Minute)
|
||||
go func() {
|
||||
t := time.NewTicker(15 * time.Minute)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
svc.QBLimitNow(ctx)
|
||||
}
|
||||
}
|
||||
}()
|
||||
l, ok := svc.Line()
|
||||
logger.Printf("[INFO] [upwatch] upload and CPU watch started (line %s, qBittorrent share %.0f %%)", lineText(l, ok), qbLimitShare*100)
|
||||
}
|
||||
|
||||
func lineText(l float64, ok bool) string {
|
||||
if !ok {
|
||||
return "unknown — learning from the off-site push"
|
||||
}
|
||||
return "at least " + upwatch.MbpsText(int64(l*1e6/8)) + " Mbit/s"
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/upwatch"
|
||||
)
|
||||
|
||||
// The upload watch is built, started, handed to the off-site push BEFORE the scheduler can start one, and its view
|
||||
// handed to the pages — a seam built and never wired is this project's commonest defect.
|
||||
// RED-PROOF: delete the SetOffboxPushObserver line → fails; move it below the scheduler's start → fails.
|
||||
func TestUpwatchIsWiredAtStartup(t *testing.T) {
|
||||
lines, _, _ := slice4CallLines(t)
|
||||
for _, fn := range []string{"newUpwatch", "SetOffboxPushObserver", "startUpwatch", "SetUpwatchView"} {
|
||||
if len(lines[fn]) == 0 {
|
||||
t.Fatalf("main.go never calls %s", fn)
|
||||
}
|
||||
}
|
||||
if lines["SetOffboxPushObserver"][0] > lines["Daily"][0] {
|
||||
t.Fatalf("SetOffboxPushObserver (line %d) comes after the first scheduled job (line %d)", lines["SetOffboxPushObserver"][0], lines["Daily"][0])
|
||||
}
|
||||
}
|
||||
|
||||
// newUpwatch asks the three apps the spike proved, reports through the notifier, and reads exact counters.
|
||||
// RED-PROOF: drop the transmission entry → fails.
|
||||
func TestNewUpwatch_ExplainersAndEmit(t *testing.T) {
|
||||
lg := log.New(io.Discard, "", 0)
|
||||
sett, err := settings.Load(filepath.Join(t.TempDir(), "settings.json"), lg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
svc := newUpwatch(sett, nil, nil, nil, notify.New("", "", "", sett, lg, false), lg)
|
||||
for _, a := range []string{"qbittorrent", "transmission", "plex"} {
|
||||
if _, ok := svc.Watcher.Explainers[a]; !ok {
|
||||
t.Errorf("no explainer for %s", a)
|
||||
}
|
||||
}
|
||||
if svc.Watcher.Emit == nil || svc.Watcher.Line == nil || svc.QB == nil || svc.QBShare != 0.5 {
|
||||
t.Fatalf("service not wired: %+v", svc)
|
||||
}
|
||||
if _, ok := svc.Watcher.Source.(*upwatch.DockerAPI); !ok {
|
||||
t.Fatal("the watch must read the Engine API's exact counters, not docker stats' text")
|
||||
}
|
||||
if svc.Watcher.Baseline != nil {
|
||||
t.Fatal("no metrics store, yet a baseline: an app with no explainer would be judged without history")
|
||||
}
|
||||
}
|
||||
@@ -106,6 +106,8 @@ type Manager struct {
|
||||
// offbox (Part B): the restic-SFTP exec seam (nil → real restic) + the failure→operator-alert hook.
|
||||
offboxRunner offboxRunner
|
||||
offboxNotify func(dur time.Duration, snapshots int, err error)
|
||||
// offboxPushObserver (the upload watch) measures the push; nil = none.
|
||||
offboxPushObserver func() func()
|
||||
// offboxStreamRunner + offboxProgress: the MANUAL run's live progress (v0.147.0, 4c). The stream
|
||||
// seam scans restic's `--json` stdout line-by-line; the state is what the page polls. Both are
|
||||
// inert on the nightly path — the sink is installed only for the duration of a manual run.
|
||||
|
||||
@@ -52,6 +52,11 @@ func defaultOffboxRunner(ctx context.Context, env []string, args ...string) ([]b
|
||||
|
||||
// SetOffboxRunner overrides the restic exec (tests). SetOffboxNotify wires the failure→operator alert.
|
||||
func (m *Manager) SetOffboxRunner(r offboxRunner) { m.offboxRunner = r }
|
||||
|
||||
// SetOffboxPushObserver hands the off-site push a measurement: fn is called as the push starts and returns the
|
||||
// function that ends it (the upload watch's line sampler). Nil = none.
|
||||
func (m *Manager) SetOffboxPushObserver(fn func() func()) { m.offboxPushObserver = fn }
|
||||
|
||||
func (m *Manager) SetOffboxNotify(fn func(dur time.Duration, snapshots int, err error)) {
|
||||
m.offboxNotify = fn
|
||||
}
|
||||
@@ -1127,7 +1132,16 @@ func (m *Manager) runOffboxBackup(ctx context.Context, withProgress bool) error
|
||||
} else {
|
||||
m.logger.Printf("[INFO] [offbox] pre-push dump leg completed in %s — snapshot pair is coherent", time.Since(dumpStart).Round(time.Millisecond))
|
||||
}
|
||||
// The upload watch learns the household's upload line from this push (internal/upwatch/line.go): a
|
||||
// one-second sample of this container's own sent bytes, from here to the end of the push only.
|
||||
var endPush func()
|
||||
if m.offboxPushObserver != nil {
|
||||
endPush = m.offboxPushObserver()
|
||||
}
|
||||
runResult, runErr = m.runOffboxInternal(ctx, apps, base, env, t)
|
||||
if endPush != nil {
|
||||
endPush()
|
||||
}
|
||||
backedUp = runResult.backedUp
|
||||
missing = runResult.missing
|
||||
endStamp()
|
||||
|
||||
@@ -2782,6 +2782,16 @@
|
||||
"poweroff.refused.other_guests": "Another system runs on this machine, so it cannot be turned off from here.",
|
||||
"poweroff.refused.generic": "The machine cannot turn itself off safely right now. Try again in a few minutes.",
|
||||
"event.app_repaired_after_crash": "%s: after the unexpected stop, %d damaged preview file(s), %d repaired, %d still broken; lost originals: %d",
|
||||
"event.app_unexplained_upload": "%s: uploaded %.1f Mbit/s for %d minutes, but the app itself counted only %.1f Mbit/s. This can be the sign of a hacked app.",
|
||||
"event.app_unexplained_upload_noapp": "%s: uploaded %.1f Mbit/s for %d minutes; the app gives no number to compare, and this is far more than usual. This can be the sign of a hacked app.",
|
||||
"event.app_unexplained_upload_cleared": "%s: the unexplained upload stopped.",
|
||||
"event.app_unexplained_cpu": "%s: %.0f%% CPU (counted on one core) for %d minutes, with no work that explains it. This can be the sign of a hacked app.",
|
||||
"event.app_unexplained_cpu_cleared": "%s: the unexplained CPU use stopped.",
|
||||
"upwatch.qb_limit_note": "qBittorrent uploads at most %s Mbit/s, so the family internet stays fast. You can change this in qBittorrent.",
|
||||
"upwatch.line_at_least": "at least",
|
||||
"upwatch.line_measured": "Mbit/s (measured from the nightly off-site backup)",
|
||||
"upwatch.line_label": "Upload speed",
|
||||
"upwatch.line_learning": "still being measured (from the nightly off-site backup)",
|
||||
"event.app_repair_after_crash_skipped": "%s: the repair after the unexpected stop was skipped: %s",
|
||||
"app_info.lost_files_title": "Photos lost in an unexpected stop",
|
||||
"app_info.lost_files_text": "The machine stopped unexpectedly (for example in a power cut), and these files were lost. The machine started again at:",
|
||||
|
||||
@@ -2770,6 +2770,16 @@
|
||||
"poweroff.refused.other_guests": "Ezen a gépen más rendszer is fut, ezért innen nem lehet kikapcsolni.",
|
||||
"poweroff.refused.generic": "A gép most nem tudja biztonságosan kikapcsolni magát. Próbáld újra néhány perc múlva.",
|
||||
"event.app_repaired_after_crash": "%s: a váratlan leállás után %d sérült előnézeti fájl, ebből %d javítva, %d maradt hibás; elveszett eredeti fájl: %d",
|
||||
"event.app_unexplained_upload": "%s: %.1f Mbit/s feltöltés %d percen át, de az alkalmazás saját adata szerint csak %.1f Mbit/s. Ez feltört alkalmazás jele lehet.",
|
||||
"event.app_unexplained_upload_noapp": "%s: %.1f Mbit/s feltöltés %d percen át; az alkalmazástól nincs összevethető adat, és ez sokkal több a szokásosnál. Ez feltört alkalmazás jele lehet.",
|
||||
"event.app_unexplained_upload_cleared": "%s: a megmagyarázatlan feltöltés megszűnt.",
|
||||
"event.app_unexplained_cpu": "%s: %.0f%% CPU (egy magra számolva) %d percen át, és nincs munka, ami ezt magyarázná. Ez feltört alkalmazás jele lehet.",
|
||||
"event.app_unexplained_cpu_cleared": "%s: a megmagyarázatlan CPU-használat megszűnt.",
|
||||
"upwatch.qb_limit_note": "A qBittorrent feltöltése legfeljebb %s Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod.",
|
||||
"upwatch.line_at_least": "legalább",
|
||||
"upwatch.line_measured": "Mbit/s (az éjszakai távoli mentésből mérve)",
|
||||
"upwatch.line_label": "Feltöltési sebesség",
|
||||
"upwatch.line_learning": "még mérjük (az éjszakai távoli mentésből)",
|
||||
"event.app_repair_after_crash_skipped": "%s: a váratlan leállás utáni javítás elmaradt: %s",
|
||||
"app_info.lost_files_title": "Elveszett fotók egy váratlan leállás miatt",
|
||||
"app_info.lost_files_text": "A gép váratlanul leállt (például áramszünet miatt), és ezek a fájlok elvesztek. A gép újraindulásának ideje:",
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The upload watch's baseline: per-minute sums across the app's containers, p95, only the named containers.
|
||||
// RED-PROOF: take the p95 per container instead of the per-minute SUM → 2 + 1 = 3 reads as 2 → fails.
|
||||
func TestStackTxP95Mbps_SumsContainersPerMinute(t *testing.T) {
|
||||
st, err := NewMetricsStore(filepath.Join(t.TempDir(), "m.db"), log.New(io.Discard, "", 0))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
t0 := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC).Unix()
|
||||
var a, b, other int64
|
||||
for i := int64(0); i < 200; i++ {
|
||||
ts := t0 + i*60
|
||||
st.InsertContainerMetrics([]ContainerSample{
|
||||
{Timestamp: ts, ContainerName: "nextcloud", NetTxBytes: a},
|
||||
{Timestamp: ts, ContainerName: "nextcloud-db", NetTxBytes: b},
|
||||
{Timestamp: ts, ContainerName: "qbittorrent", NetTxBytes: other},
|
||||
})
|
||||
a += 2e6 / 8 * 60 // 2 Mbit/s
|
||||
b += 1e6 / 8 * 60 // 1 Mbit/s
|
||||
other += 500e6 / 8 * 60
|
||||
}
|
||||
p95, minutes, err := st.StackTxP95Mbps([]string{"nextcloud", "nextcloud-db"}, time.Unix(t0, 0))
|
||||
if err != nil || minutes != 199 || p95 < 2.99 || p95 > 3.01 {
|
||||
t.Fatalf("p95 = %v over %d minutes, err %v — want 3 Mbit/s over 199", p95, minutes, err)
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,8 @@ import (
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"log"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
@@ -357,3 +359,56 @@ func (s *MetricsStore) HourSampleCounts(from, to time.Time) (map[int64]int, erro
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// StackTxP95Mbps is the 95th percentile of an app's upload, its containers summed per minute, over the rows since
|
||||
// `since` (the upload watch's „far above its own p95" rule for an app the box cannot ask; internal/upwatch). minutes
|
||||
// is how many one-minute buckets the history holds. The counters are `docker stats`' text values (three significant
|
||||
// digits), coarse but enough for a p95 the rule multiplies by ten.
|
||||
func (s *MetricsStore) StackTxP95Mbps(containers []string, since time.Time) (p95 float64, minutes int, err error) {
|
||||
if len(containers) == 0 {
|
||||
return 0, 0, nil
|
||||
}
|
||||
q := "SELECT container_name, ts, net_tx_bytes FROM container_metrics WHERE ts >= ? AND container_name IN (?" +
|
||||
strings.Repeat(",?", len(containers)-1) + ") ORDER BY container_name, ts"
|
||||
args := []interface{}{since.Unix()}
|
||||
for _, c := range containers {
|
||||
args = append(args, c)
|
||||
}
|
||||
rows, err := s.db.Query(q, args...)
|
||||
if err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
defer rows.Close()
|
||||
buckets := map[int64]float64{}
|
||||
var prevName string
|
||||
var prevTS, prevTx int64
|
||||
havePrev := false
|
||||
for rows.Next() {
|
||||
var name string
|
||||
var ts int64
|
||||
var tx sql.NullInt64
|
||||
if err := rows.Scan(&name, &ts, &tx); err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
if !tx.Valid {
|
||||
havePrev = false
|
||||
continue
|
||||
}
|
||||
if havePrev && name == prevName && ts > prevTS && ts-prevTS <= 180 && tx.Int64 >= prevTx {
|
||||
buckets[ts/60] += float64(tx.Int64-prevTx) * 8 / float64(ts-prevTS) / 1e6
|
||||
}
|
||||
prevName, prevTS, prevTx, havePrev = name, ts, tx.Int64, true
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
if len(buckets) == 0 {
|
||||
return 0, 0, nil
|
||||
}
|
||||
vals := make([]float64, 0, len(buckets))
|
||||
for _, v := range buckets {
|
||||
vals = append(vals, v)
|
||||
}
|
||||
sort.Float64s(vals)
|
||||
return vals[int(float64(len(vals)-1)*0.95)], len(vals), nil
|
||||
}
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/upwatch"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/i18n"
|
||||
)
|
||||
@@ -882,6 +883,33 @@ func (n *Notifier) NotifyAppRepairedAfterCrash(d AppRepairDetails) {
|
||||
}
|
||||
}
|
||||
|
||||
// NotifyUnexplained sends the upload and CPU watch's events (internal/upwatch; felhom.eu `08` §9). OPERATOR-ONLY on the
|
||||
// hub (notify.operatorOnlyEvents there, registered in the same commit as the types): the household can do nothing
|
||||
// about a hacked app but wait for support (the R-388 rule). Edge-triggered by the watch: one event when the condition
|
||||
// is judged, one `_cleared` (info, recorded) when it stops.
|
||||
func (n *Notifier) NotifyUnexplained(eventType, _ string, d upwatch.Details) {
|
||||
// The severity is fixed per type (alarm = warning, clear = info) — never the caller's word, so the hub's vocabulary
|
||||
// contract (TestR329_EveryEmittedSeverityIsInTheHubVocabulary) can read every value from this function.
|
||||
switch eventType {
|
||||
case "app_unexplained_upload":
|
||||
if d.AppMbps >= 0 {
|
||||
n.pushEventMsg(eventType, "warning", "event.app_unexplained_upload", d, d.App, d.RateMbps, d.Minutes, d.AppMbps)
|
||||
} else {
|
||||
n.pushEventMsg(eventType, "warning", "event.app_unexplained_upload_noapp", d, d.App, d.RateMbps, d.Minutes)
|
||||
}
|
||||
case "app_unexplained_upload_cleared":
|
||||
n.pushEventMsg(eventType, "info", "event.app_unexplained_upload_cleared", d, d.App)
|
||||
case "app_unexplained_cpu":
|
||||
n.pushEventMsg(eventType, "warning", "event.app_unexplained_cpu", d, d.App, d.CPUPercent, d.Minutes)
|
||||
case "app_unexplained_cpu_cleared":
|
||||
n.pushEventMsg(eventType, "info", "event.app_unexplained_cpu_cleared", d, d.App)
|
||||
default:
|
||||
if n.logger != nil {
|
||||
n.logger.Printf("[ERROR] NotifyUnexplained: unknown event type %q — not sent", eventType)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type OOMStormDetails struct {
|
||||
App string `json:"app"`
|
||||
StackName string `json:"stack_name"`
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
package notify
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/upwatch"
|
||||
)
|
||||
|
||||
// The upload and CPU watch's four events reach the wire with the type, the severity, the numbers in the sentence and
|
||||
// the details object (the hub mails the operator from it). An unknown type is not sent.
|
||||
// RED-PROOF: swap the AppMbps branch (use the no-number sentence always) → the first case lacks „csak 0.0" → fails.
|
||||
func TestNotifyUnexplained_TypesSentencesAndDetails(t *testing.T) {
|
||||
type got struct {
|
||||
typ, sev, msg string
|
||||
d interface{}
|
||||
}
|
||||
var gs []got
|
||||
n := &Notifier{}
|
||||
n.pushFn = func(typ, sev, msg, _ string, d interface{}) { gs = append(gs, got{typ, sev, msg, d}) }
|
||||
d := upwatch.Details{App: "qbittorrent", StackName: "qbittorrent", Kind: "upload", Minutes: 15, RateMbps: 10136.77, AppMbps: 0, Explainer: "qbittorrent"}
|
||||
n.NotifyUnexplained("app_unexplained_upload", "warning", d)
|
||||
d.AppMbps = -1
|
||||
n.NotifyUnexplained("app_unexplained_upload", "warning", d)
|
||||
n.NotifyUnexplained("app_unexplained_upload_cleared", "info", upwatch.Details{App: "qbittorrent", Cleared: true})
|
||||
n.NotifyUnexplained("app_unexplained_cpu", "warning", upwatch.Details{App: "plex", Kind: "cpu", Minutes: 30, CPUPercent: 380})
|
||||
n.NotifyUnexplained("app_unexplained_cpu_cleared", "info", upwatch.Details{App: "plex", Cleared: true})
|
||||
n.NotifyUnexplained("app_unexplained_typo", "warning", d)
|
||||
if len(gs) != 5 {
|
||||
t.Fatalf("pushed %d events, want 5 (the unknown type must not be sent): %+v", len(gs), gs)
|
||||
}
|
||||
want := []struct{ typ, sev, frag string }{
|
||||
{"app_unexplained_upload", "warning", "qbittorrent: 10136.8 Mbit/s feltöltés 15 percen át, de az alkalmazás saját adata szerint csak 0.0 Mbit/s"},
|
||||
{"app_unexplained_upload", "warning", "az alkalmazástól nincs összevethető adat"},
|
||||
{"app_unexplained_upload_cleared", "info", "qbittorrent: a megmagyarázatlan feltöltés megszűnt."},
|
||||
{"app_unexplained_cpu", "warning", "plex: 380% CPU (egy magra számolva) 30 percen át"},
|
||||
{"app_unexplained_cpu_cleared", "info", "plex: a megmagyarázatlan CPU-használat megszűnt."},
|
||||
}
|
||||
for i, w := range want {
|
||||
if gs[i].typ != w.typ || gs[i].sev != w.sev || !strings.Contains(gs[i].msg, w.frag) {
|
||||
t.Errorf("event %d = %s/%s %q, want %s/%s containing %q", i, gs[i].typ, gs[i].sev, gs[i].msg, w.typ, w.sev, w.frag)
|
||||
}
|
||||
if _, ok := gs[i].d.(upwatch.Details); !ok {
|
||||
t.Errorf("event %d carries no details object", i)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -78,6 +78,11 @@ type Settings struct {
|
||||
// AppUpdateEventsSeeded (v0.264.0) guards the ONE-TIME add of app_update_undone / app_update_held
|
||||
// into an existing household's stored prefs (seedAppUpdateEvents).
|
||||
AppUpdateEventsSeeded bool `json:"app_update_events_seeded,omitempty"`
|
||||
|
||||
// Upwatch (the upload and CPU watch, internal/upwatch) keeps its own state here as one opaque JSON block: the
|
||||
// measured upload line (the last 14 off-site pushes) and qBittorrent's default-limit record (who owns the limit).
|
||||
// Opaque on purpose — this package does not know the watch's types, and the watch does not reach into settings.
|
||||
Upwatch json.RawMessage `json:"upwatch,omitempty"`
|
||||
// UnhealthyStopEventSeeded (v0.269.0) guards the ONE-TIME add of app_stopped_unhealthy (the
|
||||
// seedAppUpdateEvents shape).
|
||||
UnhealthyStopEventSeeded bool `json:"unhealthy_stop_event_seeded,omitempty"`
|
||||
@@ -900,6 +905,28 @@ func (s *Settings) migrateResticToRsync() {
|
||||
|
||||
// Save writes settings to disk atomically (write to .tmp, rename).
|
||||
// Caller must hold the write lock or call this from a method that does.
|
||||
// GetUpwatch returns a copy of the watch's state block (nil when none was ever written).
|
||||
func (s *Settings) GetUpwatch() json.RawMessage {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
if s.Upwatch == nil {
|
||||
return nil
|
||||
}
|
||||
return append(json.RawMessage(nil), s.Upwatch...)
|
||||
}
|
||||
|
||||
// UpdateUpwatch replaces the watch's state block through fn (read-modify-write under the lock) and saves.
|
||||
func (s *Settings) UpdateUpwatch(fn func(old json.RawMessage) (json.RawMessage, error)) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
nv, err := fn(append(json.RawMessage(nil), s.Upwatch...))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
s.Upwatch = nv
|
||||
return s.save()
|
||||
}
|
||||
|
||||
func (s *Settings) save() error {
|
||||
data, err := json.MarshalIndent(s, "", " ")
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,305 @@
|
||||
package upwatch
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Report is what one app says about its own work right now.
|
||||
type Report struct {
|
||||
// UploadCounter is the app's own cumulative count of bytes it uploaded (qBittorrent `up_info_data`, Transmission
|
||||
// `current-stats.uploadedBytes`). It restarts at zero with the app; the watcher treats a drop as a restart.
|
||||
UploadCounter uint64
|
||||
HasUploadCounter bool
|
||||
// Viewers is the number of playing sessions (Plex `/status/sessions` size). A viewer EXPLAINS upload of any size:
|
||||
// the per-session bandwidth was not measured in the spike (no claimed server with a remote player tonight), so
|
||||
// the rule is deliberately the conservative half — no viewer and a big upload is the Plex2Shell signature.
|
||||
Viewers int
|
||||
HasViewerCount bool
|
||||
// Transcodes is CPU work the app reports (Plex `/transcode/sessions` size). Any transcode explains the CPU.
|
||||
Transcodes int
|
||||
HasTranscodeCount bool
|
||||
// Limit is the app's current global upload limit in bytes/s (0 = none); HasLimit when the app has one to read.
|
||||
Limit int64
|
||||
HasLimit bool
|
||||
}
|
||||
|
||||
// Explainer asks one app about its own work.
|
||||
type Explainer interface {
|
||||
Stack() string
|
||||
Ask(ctx context.Context) (Report, error)
|
||||
}
|
||||
|
||||
// Box is what an explainer needs outside HTTP — the mediawire seams (`docker exec printenv`, the Plex token read at
|
||||
// use). Production: mediawire.DockerBox.
|
||||
type Box interface {
|
||||
Env(container, name string) (string, error)
|
||||
Running(container string) bool
|
||||
}
|
||||
|
||||
// httpDo is one request to an app's own container. Small and local on purpose: mediawire's helpers are methods of
|
||||
// its Wirer, which also writes; this package only reads (and sets one limit, qblimit.go).
|
||||
func httpDo(ctx context.Context, c *http.Client, method, u string, hdr map[string]string, body []byte, ctype string) (int, []byte, http.Header, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, method, u, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return 0, nil, nil, err
|
||||
}
|
||||
for k, v := range hdr {
|
||||
if strings.EqualFold(k, "Host") {
|
||||
req.Host = v
|
||||
continue
|
||||
}
|
||||
req.Header.Set(k, v)
|
||||
}
|
||||
if ctype != "" {
|
||||
req.Header.Set("Content-Type", ctype)
|
||||
}
|
||||
resp, err := c.Do(req)
|
||||
if err != nil {
|
||||
return 0, nil, nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
|
||||
return resp.StatusCode, b, resp.Header, nil
|
||||
}
|
||||
|
||||
// ── qBittorrent ──────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// QBittorrent asks qBittorrent's Web API. Measured on 9202 (qBittorrent 5.2.4, WebAPI 2.15.1): while seeding, the
|
||||
// container sent 1.03–1.07 × what `up_info_data` counted (protocol overhead); `transfer/uploadLimit` and
|
||||
// `app/preferences.up_limit` are the same value, persisted across a restart (`Session\GlobalUPSpeedLimit`, KiB/s).
|
||||
type QBittorrent struct {
|
||||
Box Box
|
||||
URL string // default http://qbittorrent:8080 (the container's name on the docker network, as mediawire calls it)
|
||||
HTTP *http.Client
|
||||
|
||||
sess map[string]string // the session cookie, kept between minutes (a login a minute would fill qBittorrent's log)
|
||||
}
|
||||
|
||||
func (q *QBittorrent) Stack() string { return "qbittorrent" }
|
||||
|
||||
func (q *QBittorrent) base() string {
|
||||
if q.URL != "" {
|
||||
return q.URL
|
||||
}
|
||||
return "http://qbittorrent:8080"
|
||||
}
|
||||
|
||||
func (q *QBittorrent) client() *http.Client {
|
||||
if q.HTTP != nil {
|
||||
return q.HTTP
|
||||
}
|
||||
return &http.Client{Timeout: 20 * time.Second}
|
||||
}
|
||||
|
||||
var qbCookieRE = regexp.MustCompile(`((?:QBT_)?SID[A-Za-z0-9_]*)=([^;]+)`)
|
||||
|
||||
// login returns the headers of an authenticated session. The password is the catalog's QBT_ADMIN_PASSWORD, read at
|
||||
// use; Referer must equal the base (qBittorrent's CSRF check — a mismatch is a 401, measured).
|
||||
func (q *QBittorrent) login(ctx context.Context) (map[string]string, error) {
|
||||
pw, err := q.Box.Env("qbittorrent", "QBT_ADMIN_PASSWORD")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
form := url.Values{"username": {"admin"}, "password": {pw}}.Encode()
|
||||
code, _, hd, err := httpDo(ctx, q.client(), "POST", q.base()+"/api/v2/auth/login", map[string]string{"Referer": q.base()}, []byte(form), "application/x-www-form-urlencoded")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
m := qbCookieRE.FindStringSubmatch(strings.Join(hd.Values("Set-Cookie"), "; "))
|
||||
if (code != 200 && code != 204) || m == nil { // 5.x answers a good login 204
|
||||
return nil, fmt.Errorf("qBittorrent login: HTTP %d", code)
|
||||
}
|
||||
return map[string]string{"Referer": q.base(), "Cookie": m[1] + "=" + m[2]}, nil
|
||||
}
|
||||
|
||||
// session returns the kept session, logging in when there is none or when fresh is set.
|
||||
func (q *QBittorrent) session(ctx context.Context, fresh bool) (map[string]string, error) {
|
||||
if q.sess != nil && !fresh {
|
||||
return q.sess, nil
|
||||
}
|
||||
h, err := q.login(ctx)
|
||||
if err != nil {
|
||||
q.sess = nil
|
||||
return nil, err
|
||||
}
|
||||
q.sess = h
|
||||
return h, nil
|
||||
}
|
||||
|
||||
func (q *QBittorrent) Ask(ctx context.Context) (Report, error) {
|
||||
h, err := q.session(ctx, false)
|
||||
if err != nil {
|
||||
return Report{}, err
|
||||
}
|
||||
code, b, _, err := httpDo(ctx, q.client(), "GET", q.base()+"/api/v2/transfer/info", h, nil, "")
|
||||
if err == nil && code == 403 { // the session expired (qBittorrent restarted, or its 1-hour timeout): once more
|
||||
if h, err = q.session(ctx, true); err != nil {
|
||||
return Report{}, err
|
||||
}
|
||||
code, b, _, err = httpDo(ctx, q.client(), "GET", q.base()+"/api/v2/transfer/info", h, nil, "")
|
||||
}
|
||||
if err != nil || code != 200 {
|
||||
return Report{}, fmt.Errorf("qBittorrent transfer/info: HTTP %d %v", code, err)
|
||||
}
|
||||
var ti struct {
|
||||
UpInfoData uint64 `json:"up_info_data"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &ti); err != nil {
|
||||
return Report{}, err
|
||||
}
|
||||
r := Report{UploadCounter: ti.UpInfoData, HasUploadCounter: true}
|
||||
if lim, err := q.UploadLimit(ctx, h); err == nil {
|
||||
r.Limit, r.HasLimit = lim, true
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
// UploadLimit reads the global upload limit (bytes/s, 0 = none). h nil → logs in.
|
||||
func (q *QBittorrent) UploadLimit(ctx context.Context, h map[string]string) (int64, error) {
|
||||
if h == nil {
|
||||
var err error
|
||||
if h, err = q.session(ctx, true); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
code, b, _, err := httpDo(ctx, q.client(), "GET", q.base()+"/api/v2/transfer/uploadLimit", h, nil, "")
|
||||
if err != nil || code != 200 {
|
||||
return 0, fmt.Errorf("qBittorrent uploadLimit: HTTP %d %v", code, err)
|
||||
}
|
||||
var v int64
|
||||
if _, err := fmt.Sscan(strings.TrimSpace(string(b)), &v); err != nil {
|
||||
return 0, fmt.Errorf("qBittorrent uploadLimit: %q", strings.TrimSpace(string(b)))
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// SetUploadLimit sets the global upload limit (bytes/s) and returns the value qBittorrent stored (it keeps KiB/s, so
|
||||
// 2 500 000 reads back as 2 499 584 — measured).
|
||||
func (q *QBittorrent) SetUploadLimit(ctx context.Context, bytesPerSec int64) (int64, error) {
|
||||
h, err := q.session(ctx, true)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
form := url.Values{"limit": {fmt.Sprint(bytesPerSec)}}.Encode()
|
||||
code, _, _, err := httpDo(ctx, q.client(), "POST", q.base()+"/api/v2/transfer/setUploadLimit", h, []byte(form), "application/x-www-form-urlencoded")
|
||||
if err != nil || code != 200 {
|
||||
return 0, fmt.Errorf("qBittorrent setUploadLimit: HTTP %d %v", code, err)
|
||||
}
|
||||
return q.UploadLimit(ctx, h)
|
||||
}
|
||||
|
||||
// ── Transmission ─────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// Transmission asks Transmission's RPC (4.1.3 on 9202): `session-stats.current-stats.uploadedBytes` matched the
|
||||
// container's sent bytes within 0.1–2 % while seeding (measured).
|
||||
type Transmission struct {
|
||||
Box Box
|
||||
URL string // default http://transmission:9091
|
||||
HTTP *http.Client
|
||||
}
|
||||
|
||||
func (t *Transmission) Stack() string { return "transmission" }
|
||||
|
||||
func (t *Transmission) Ask(ctx context.Context) (Report, error) {
|
||||
pw, err := t.Box.Env("transmission", "PASS")
|
||||
if err != nil {
|
||||
return Report{}, err
|
||||
}
|
||||
base := t.URL
|
||||
if base == "" {
|
||||
base = "http://transmission:9091"
|
||||
}
|
||||
c := t.HTTP
|
||||
if c == nil {
|
||||
c = &http.Client{Timeout: 20 * time.Second}
|
||||
}
|
||||
h := map[string]string{"Authorization": "Basic " + base64.StdEncoding.EncodeToString([]byte("admin:"+pw)), "Host": "transmission:9091"}
|
||||
body := []byte(`{"method":"session-stats"}`)
|
||||
for i := 0; i < 2; i++ {
|
||||
code, b, hd, err := httpDo(ctx, c, "POST", base+"/transmission/rpc", h, body, "application/json")
|
||||
if err != nil {
|
||||
return Report{}, err
|
||||
}
|
||||
if code == 409 {
|
||||
h["X-Transmission-Session-Id"] = hd.Get("X-Transmission-Session-Id")
|
||||
continue
|
||||
}
|
||||
if code != 200 {
|
||||
return Report{}, fmt.Errorf("transmission session-stats: HTTP %d", code)
|
||||
}
|
||||
var st struct {
|
||||
Arguments struct {
|
||||
Current struct {
|
||||
UploadedBytes uint64 `json:"uploadedBytes"`
|
||||
} `json:"current-stats"`
|
||||
} `json:"arguments"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &st); err != nil {
|
||||
return Report{}, err
|
||||
}
|
||||
return Report{UploadCounter: st.Arguments.Current.UploadedBytes, HasUploadCounter: true}, nil
|
||||
}
|
||||
return Report{}, fmt.Errorf("transmission session-stats: no session id")
|
||||
}
|
||||
|
||||
// ── Plex ─────────────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
// Plex asks Plex Media Server for its playing and transcoding sessions. Measured on 9202 (1.43.5, unclaimed): both
|
||||
// answer `{"MediaContainer":{"size":0}}` without a token from the docker network; a CLAIMED server needs its own
|
||||
// PlexOnlineToken, read from Preferences.xml at use exactly as mediawire does (`09` §3 decision 248).
|
||||
type Plex struct {
|
||||
Token func() string // mediawire.PlexToken(box); "" for an unclaimed server
|
||||
URL func(port int) (string, error) // mediawire.PlexURL (Plex refuses a call by NAME as non-local, R-1060)
|
||||
HTTP *http.Client
|
||||
}
|
||||
|
||||
func (p *Plex) Stack() string { return "plex" }
|
||||
|
||||
func (p *Plex) Ask(ctx context.Context) (Report, error) {
|
||||
base, err := p.URL(32400)
|
||||
if err != nil {
|
||||
return Report{}, err
|
||||
}
|
||||
c := p.HTTP
|
||||
if c == nil {
|
||||
c = &http.Client{Timeout: 20 * time.Second}
|
||||
}
|
||||
h := map[string]string{"Accept": "application/json"}
|
||||
if tok := p.Token(); tok != "" {
|
||||
h["X-Plex-Token"] = tok
|
||||
}
|
||||
size := func(path string) (int, error) {
|
||||
code, b, _, err := httpDo(ctx, c, "GET", base+path, h, nil, "")
|
||||
if err != nil || code != 200 {
|
||||
return 0, fmt.Errorf("plex %s: HTTP %d %v", path, code, err)
|
||||
}
|
||||
var mc struct {
|
||||
MediaContainer struct {
|
||||
Size int `json:"size"`
|
||||
} `json:"MediaContainer"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &mc); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return mc.MediaContainer.Size, nil
|
||||
}
|
||||
v, err := size("/status/sessions")
|
||||
if err != nil {
|
||||
return Report{}, err
|
||||
}
|
||||
tr, err := size("/transcode/sessions")
|
||||
if err != nil {
|
||||
return Report{}, err
|
||||
}
|
||||
return Report{Viewers: v, HasViewerCount: true, Transcodes: tr, HasTranscodeCount: true}, nil
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
package upwatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type envBox map[string]string
|
||||
|
||||
func (b envBox) Env(c, n string) (string, error) {
|
||||
if v, ok := b[c+"/"+n]; ok {
|
||||
return v, nil
|
||||
}
|
||||
return "", fmt.Errorf("no %s in %s", n, c)
|
||||
}
|
||||
func (b envBox) Running(string) bool { return true }
|
||||
|
||||
// qBittorrent as measured on 9202: a login is 204 + a SID cookie, and only when Referer equals the base; transfer/info
|
||||
// carries up_info_data; a stale session is 403.
|
||||
func fakeQBServer(t *testing.T, pw string, logins *int, expireNext *bool, limit *int64) *httptest.Server {
|
||||
var srv *httptest.Server
|
||||
srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/api/v2/auth/login":
|
||||
r.ParseForm()
|
||||
if r.Header.Get("Referer") != srv.URL || r.Form.Get("username") != "admin" || r.Form.Get("password") != pw {
|
||||
w.WriteHeader(401)
|
||||
return
|
||||
}
|
||||
*logins++
|
||||
http.SetCookie(w, &http.Cookie{Name: "QBT_SID_8080", Value: fmt.Sprint("s", *logins)})
|
||||
w.WriteHeader(204)
|
||||
case "/api/v2/transfer/info":
|
||||
if *expireNext {
|
||||
*expireNext = false
|
||||
w.WriteHeader(403)
|
||||
return
|
||||
}
|
||||
if !strings.HasPrefix(r.Header.Get("Cookie"), "QBT_SID_8080=") {
|
||||
w.WriteHeader(403)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, `{"up_info_data":123456789,"up_info_speed":16384000,"up_rate_limit":0}`)
|
||||
case "/api/v2/transfer/uploadLimit":
|
||||
fmt.Fprint(w, *limit)
|
||||
case "/api/v2/transfer/setUploadLimit":
|
||||
r.ParseForm()
|
||||
var v int64
|
||||
fmt.Sscan(r.Form.Get("limit"), &v)
|
||||
*limit = v / 1024 * 1024
|
||||
w.WriteHeader(200)
|
||||
default:
|
||||
w.WriteHeader(404)
|
||||
}
|
||||
}))
|
||||
return srv
|
||||
}
|
||||
|
||||
// RED-PROOF: send no Referer → 401 → fails; log in every minute instead of keeping the session → logins == 3 → fails.
|
||||
func TestQBittorrent_AskKeepsTheSessionAndReLogsInOn403(t *testing.T) {
|
||||
logins, expire, limit := 0, false, int64(0)
|
||||
srv := fakeQBServer(t, "pw-canary", &logins, &expire, &limit)
|
||||
defer srv.Close()
|
||||
q := &QBittorrent{Box: envBox{"qbittorrent/QBT_ADMIN_PASSWORD": "pw-canary"}, URL: srv.URL}
|
||||
for i := 0; i < 3; i++ {
|
||||
r, err := q.Ask(context.Background())
|
||||
if err != nil || !r.HasUploadCounter || r.UploadCounter != 123456789 || !r.HasLimit {
|
||||
t.Fatalf("ask %d: %+v %v", i, r, err)
|
||||
}
|
||||
}
|
||||
if logins != 1 {
|
||||
t.Fatalf("logins = %d for three minutes — the session must be kept", logins)
|
||||
}
|
||||
expire = true
|
||||
if _, err := q.Ask(context.Background()); err != nil || logins != 2 {
|
||||
t.Fatalf("after a 403: err=%v logins=%d — want one fresh login", err, logins)
|
||||
}
|
||||
got, err := q.SetUploadLimit(context.Background(), 2500000)
|
||||
if err != nil || got != 2499584 {
|
||||
t.Fatalf("SetUploadLimit = %d %v, want the stored 2499584", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Transmission's RPC: a 409 hands out the session id; Basic auth with the catalog's PASS.
|
||||
func TestTransmission_AskFollowsTheSessionID(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Header.Get("Authorization") != "Basic "+base64.StdEncoding.EncodeToString([]byte("admin:tr-pw")) {
|
||||
w.WriteHeader(401)
|
||||
return
|
||||
}
|
||||
if r.Header.Get("X-Transmission-Session-Id") != "sid-1" {
|
||||
w.Header().Set("X-Transmission-Session-Id", "sid-1")
|
||||
w.WriteHeader(409)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, `{"arguments":{"current-stats":{"uploadedBytes":987654321},"uploadSpeed":0},"result":"success"}`)
|
||||
}))
|
||||
defer srv.Close()
|
||||
tr := &Transmission{Box: envBox{"transmission/PASS": "tr-pw"}, URL: srv.URL}
|
||||
r, err := tr.Ask(context.Background())
|
||||
if err != nil || r.UploadCounter != 987654321 || !r.HasUploadCounter {
|
||||
t.Fatalf("Ask = %+v %v", r, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Plex: the token goes along only when the server is claimed; both session lists are read.
|
||||
// RED-PROOF: always send the header → the unclaimed case sends "X-Plex-Token: " → fails.
|
||||
func TestPlex_AskReadsSessionsWithTheTokenOnlyWhenClaimed(t *testing.T) {
|
||||
var seen []string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
_, has := r.Header["X-Plex-Token"]
|
||||
seen = append(seen, fmt.Sprintf("%s token=%v", r.URL.Path, has))
|
||||
switch r.URL.Path {
|
||||
case "/status/sessions":
|
||||
fmt.Fprint(w, `{"MediaContainer":{"size":2}}`)
|
||||
case "/transcode/sessions":
|
||||
fmt.Fprint(w, `{"MediaContainer":{"size":1}}`)
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
for _, tok := range []string{"", "plex-token-canary"} {
|
||||
seen = nil
|
||||
p := &Plex{Token: func() string { return tok }, URL: func(int) (string, error) { return srv.URL, nil }}
|
||||
r, err := p.Ask(context.Background())
|
||||
if err != nil || r.Viewers != 2 || r.Transcodes != 1 || !r.HasViewerCount || !r.HasTranscodeCount {
|
||||
t.Fatalf("Ask = %+v %v", r, err)
|
||||
}
|
||||
want := fmt.Sprintf("token=%v", tok != "")
|
||||
for _, s := range seen {
|
||||
if !strings.HasSuffix(s, want) {
|
||||
t.Fatalf("claimed=%v: request %q", tok != "", s)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
package upwatch
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ── The upload line, learnt from what the box already does ────────────────────────────────────────────────────────
|
||||
//
|
||||
// The box's off-site push (restic, inside this controller's own container) is the only big upload the box makes on
|
||||
// its own. While it runs, the controller samples its OWN network counter once a second and keeps the best 10-second
|
||||
// rate. Over nights, the highest such rate is a LOWER BOUND of the household's upload line — the push may not fill
|
||||
// the line (the family uploads too, the Storage Box may be slow), so the page says „at least".
|
||||
//
|
||||
// Measured (spike §2): at a one-minute sample the push looked like 0.3–37 Mbit/s, because most nights it moves a few
|
||||
// megabytes in under a minute — the one-minute counter shows the data size, not the line. Hence a 1-second sampler,
|
||||
// and a night only COUNTS when its best 10 seconds moved ≥ MinWindowBytes: a short push says nothing about the line.
|
||||
|
||||
// MinWindowBytes is the least a push must move in its best 10 seconds to say anything about the line (≥ 8 Mbit/s).
|
||||
const MinWindowBytes = 10 << 20
|
||||
|
||||
// KeepNights is how many nights' records the line is the maximum of.
|
||||
const KeepNights = 14
|
||||
|
||||
// LineRecord is one push's measurement.
|
||||
type LineRecord struct {
|
||||
At string `json:"at"` // RFC3339, the push's end
|
||||
Peak10Mbps float64 `json:"peak10_mbps"` // the best 10-second rate
|
||||
Bytes uint64 `json:"bytes"` // everything the container sent during the push
|
||||
Seconds int `json:"seconds"`
|
||||
Counts bool `json:"counts"` // the best 10 seconds moved ≥ MinWindowBytes
|
||||
}
|
||||
|
||||
// LineFromRecords is the line estimate: the highest counting peak over the kept nights. ok=false while none counts.
|
||||
func LineFromRecords(recs []LineRecord) (float64, bool) {
|
||||
best, ok := 0.0, false
|
||||
for _, r := range recs {
|
||||
if r.Counts && r.Peak10Mbps > best {
|
||||
best, ok = r.Peak10Mbps, true
|
||||
}
|
||||
}
|
||||
return best, ok
|
||||
}
|
||||
|
||||
// AddRecord appends one push and keeps the newest KeepNights.
|
||||
func AddRecord(recs []LineRecord, r LineRecord) []LineRecord {
|
||||
recs = append(recs, r)
|
||||
if len(recs) > KeepNights {
|
||||
recs = recs[len(recs)-KeepNights:]
|
||||
}
|
||||
return recs
|
||||
}
|
||||
|
||||
// OwnTxBytes sums the bytes sent on every interface but loopback in THIS process's network namespace
|
||||
// (/proc/self/net/dev; the 9th field of an interface line is transmit bytes).
|
||||
func OwnTxBytes() (uint64, error) {
|
||||
return txFromNetDev("/proc/self/net/dev")
|
||||
}
|
||||
|
||||
func txFromNetDev(path string) (uint64, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer f.Close()
|
||||
var sum uint64
|
||||
sc := bufio.NewScanner(f)
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
i := strings.Index(line, ":")
|
||||
if i < 0 {
|
||||
continue
|
||||
}
|
||||
if strings.TrimSpace(line[:i]) == "lo" {
|
||||
continue
|
||||
}
|
||||
fs := strings.Fields(line[i+1:])
|
||||
if len(fs) < 9 {
|
||||
continue
|
||||
}
|
||||
v, err := strconv.ParseUint(fs[8], 10, 64)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("%s: %q: %w", path, line, err)
|
||||
}
|
||||
sum += v
|
||||
}
|
||||
return sum, sc.Err()
|
||||
}
|
||||
|
||||
// Point is one reading of the counter.
|
||||
type Point struct {
|
||||
At time.Time
|
||||
V uint64
|
||||
}
|
||||
|
||||
// Summarize turns a push's readings into its record: total bytes, seconds, and the best rate over any window of at
|
||||
// least 10 seconds (for each start, the first reading ≥ 10 s later). A counter that went down is no measurement.
|
||||
func Summarize(pts []Point, end time.Time) LineRecord {
|
||||
r := LineRecord{At: end.UTC().Format(time.RFC3339)}
|
||||
if len(pts) < 2 {
|
||||
return r
|
||||
}
|
||||
first, last := pts[0], pts[len(pts)-1]
|
||||
if last.V >= first.V {
|
||||
r.Bytes = last.V - first.V
|
||||
}
|
||||
r.Seconds = int(last.At.Sub(first.At).Seconds() + 0.5)
|
||||
var bestBytes uint64
|
||||
var bestRate float64
|
||||
j := 0
|
||||
for i := range pts {
|
||||
if j < i {
|
||||
j = i
|
||||
}
|
||||
for j < len(pts) && pts[j].At.Sub(pts[i].At) < 10*time.Second {
|
||||
j++
|
||||
}
|
||||
if j >= len(pts) {
|
||||
break
|
||||
}
|
||||
if pts[j].V < pts[i].V {
|
||||
continue
|
||||
}
|
||||
b := pts[j].V - pts[i].V
|
||||
rate := float64(b) * 8 / pts[j].At.Sub(pts[i].At).Seconds() / 1e6
|
||||
if rate > bestRate {
|
||||
bestRate, bestBytes = rate, b
|
||||
}
|
||||
}
|
||||
r.Peak10Mbps = round2(bestRate)
|
||||
r.Counts = bestBytes >= MinWindowBytes
|
||||
return r
|
||||
}
|
||||
|
||||
// PushSampler samples a byte counter once a second between Start and the stop function it returns.
|
||||
type PushSampler struct {
|
||||
Read func() (uint64, error) // default OwnTxBytes
|
||||
Every time.Duration // default 1 s
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
// Start begins sampling; the returned stop ends it and returns the push's record.
|
||||
func (p *PushSampler) Start() (stop func() LineRecord) {
|
||||
read := p.Read
|
||||
if read == nil {
|
||||
read = OwnTxBytes
|
||||
}
|
||||
every := p.Every
|
||||
if every <= 0 {
|
||||
every = time.Second
|
||||
}
|
||||
now := p.Now
|
||||
if now == nil {
|
||||
now = time.Now
|
||||
}
|
||||
var mu sync.Mutex
|
||||
var pts []Point
|
||||
take := func() {
|
||||
if v, err := read(); err == nil {
|
||||
mu.Lock()
|
||||
pts = append(pts, Point{now(), v})
|
||||
mu.Unlock()
|
||||
}
|
||||
}
|
||||
take()
|
||||
done := make(chan struct{})
|
||||
finished := make(chan struct{})
|
||||
go func() {
|
||||
defer close(finished)
|
||||
t := time.NewTicker(every)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-done:
|
||||
return
|
||||
case <-t.C:
|
||||
take()
|
||||
}
|
||||
}
|
||||
}()
|
||||
return func() LineRecord {
|
||||
close(done)
|
||||
<-finished
|
||||
take()
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return Summarize(pts, now())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
package upwatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The best ≥ 10-second rate, and a push only COUNTS when that window moved ≥ 10 MB.
|
||||
// RED-PROOF: compute the peak over 1-second windows → the 4 MB burst reads as 33 Mbit/s → fails.
|
||||
func TestLine_SummarizePeakAndCounts(t *testing.T) {
|
||||
series := func(perSecond []uint64) []Point {
|
||||
t0 := time.Date(2026, 10, 24, 2, 16, 0, 0, time.UTC)
|
||||
pts := []Point{{t0, 0}}
|
||||
for i, b := range perSecond {
|
||||
pts = append(pts, Point{t0.Add(time.Duration(i+1) * time.Second), pts[len(pts)-1].V + b})
|
||||
}
|
||||
return pts
|
||||
}
|
||||
steady := make([]uint64, 60)
|
||||
for i := range steady {
|
||||
steady[i] = 50e6 / 8
|
||||
}
|
||||
if r := Summarize(series(steady), time.Now()); r.Peak10Mbps < 49.9 || r.Peak10Mbps > 50.1 || !r.Counts || r.Seconds != 60 {
|
||||
t.Fatalf("steady 50 Mbit/s push: %+v", r)
|
||||
}
|
||||
burst := make([]uint64, 30)
|
||||
burst[3] = 4 << 20
|
||||
if r := Summarize(series(burst), time.Now()); r.Counts || r.Peak10Mbps > 4 {
|
||||
t.Fatalf("a 4 MB one-second push must not set the line: %+v", r)
|
||||
}
|
||||
}
|
||||
|
||||
// The sampler itself: readings from a live (fast) ticker reach the summary.
|
||||
func TestLine_PushSamplerCollects(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var v uint64
|
||||
p := &PushSampler{Every: time.Millisecond, Read: func() (uint64, error) { mu.Lock(); defer mu.Unlock(); v += 1000; return v, nil }}
|
||||
stop := p.Start()
|
||||
time.Sleep(30 * time.Millisecond)
|
||||
r := stop()
|
||||
if r.Bytes < 2000 {
|
||||
t.Fatalf("sampler collected %d bytes", r.Bytes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLine_FromRecordsUsesOnlyCountingNightsAndKeepsFourteen(t *testing.T) {
|
||||
if _, ok := LineFromRecords(nil); ok {
|
||||
t.Fatal("no record: the line must be unknown")
|
||||
}
|
||||
recs := []LineRecord{{Peak10Mbps: 90, Counts: false}, {Peak10Mbps: 40, Counts: true}, {Peak10Mbps: 55, Counts: true}}
|
||||
if l, ok := LineFromRecords(recs); !ok || l != 55 {
|
||||
t.Fatalf("line = %v %v, want 55 (the 90 did not count)", l, ok)
|
||||
}
|
||||
var keep []LineRecord
|
||||
for i := 0; i < 20; i++ {
|
||||
keep = AddRecord(keep, LineRecord{Peak10Mbps: float64(i), Counts: true})
|
||||
}
|
||||
if len(keep) != KeepNights || keep[0].Peak10Mbps != 6 {
|
||||
t.Fatalf("kept %d, first %v", len(keep), keep[0].Peak10Mbps)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLine_TxFromNetDevSkipsLoopback(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "dev")
|
||||
os.WriteFile(p, []byte(`Inter-| Receive | Transmit
|
||||
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
|
||||
lo: 9000 10 0 0 0 0 0 0 9000 10 0 0 0 0 0 0
|
||||
eth0: 100 1 0 0 0 0 0 0 774708 5 0 0 0 0 0 0
|
||||
eth1: 1 1 0 0 0 0 0 0 1000 1 0 0 0 0 0 0
|
||||
`), 0o644)
|
||||
v, err := txFromNetDev(p)
|
||||
if err != nil || v != 775708 {
|
||||
t.Fatalf("tx = %d %v, want 775708 (eth0+eth1, not lo)", v, err)
|
||||
}
|
||||
}
|
||||
|
||||
// ── qBittorrent's default limit ──
|
||||
|
||||
type fakeQB struct {
|
||||
cur int64
|
||||
sets []int64
|
||||
}
|
||||
|
||||
func (f *fakeQB) UploadLimit(context.Context, map[string]string) (int64, error) { return f.cur, nil }
|
||||
func (f *fakeQB) SetUploadLimit(_ context.Context, b int64) (int64, error) {
|
||||
f.sets = append(f.sets, b)
|
||||
f.cur = b / 1024 * 1024 // qBittorrent keeps KiB/s
|
||||
return f.cur, nil
|
||||
}
|
||||
|
||||
// RED-PROOF per row: drop the household check → the „household" rows set a limit → fail; compare against the
|
||||
// requested value instead of the stored one → the box's own rounding reads as a household change → the re-set row fails.
|
||||
func TestQBLimit_TheHouseholdAlwaysWins(t *testing.T) {
|
||||
now := time.Date(2026, 10, 24, 3, 0, 0, 0, time.UTC)
|
||||
ctx := context.Background()
|
||||
// 1. Unlimited, never set, line 100 Mbit/s, share 50 % → 6 250 000 B/s requested, stored rounded.
|
||||
q := &fakeQB{}
|
||||
st, act, err := ApplyQBLimit(ctx, q, QBLimitState{}, 100, true, 0.5, now)
|
||||
if err != nil || act != "set" || len(q.sets) != 1 || q.sets[0] != 6250000 || st.BoxSetBytes != 6249472 {
|
||||
t.Fatalf("first set: %s %v %+v sets=%v", act, err, st, q.sets)
|
||||
}
|
||||
// 2. Same line, qBittorrent still holds the box's (rounded) value → kept, no write.
|
||||
st2, act, _ := ApplyQBLimit(ctx, q, st, 105, true, 0.5, now)
|
||||
if act != "kept" || len(q.sets) != 1 {
|
||||
t.Fatalf("unchanged: %s sets=%v", act, q.sets)
|
||||
}
|
||||
// 3. The line moved > 20 % → re-set.
|
||||
st3, act, _ := ApplyQBLimit(ctx, q, st2, 200, true, 0.5, now)
|
||||
if act != "set" || len(q.sets) != 2 || st3.LineMbps != 200 {
|
||||
t.Fatalf("line moved: %s %+v", act, st3)
|
||||
}
|
||||
// 4. The household changed it in qBittorrent → theirs, never touched again, even when the line moves.
|
||||
q.cur = 1000000
|
||||
st4, act, _ := ApplyQBLimit(ctx, q, st3, 200, true, 0.5, now)
|
||||
if act != "household" || !st4.Household {
|
||||
t.Fatalf("household change: %s %+v", act, st4)
|
||||
}
|
||||
if _, act, _ := ApplyQBLimit(ctx, q, st4, 50, true, 0.5, now); act != "household" || len(q.sets) != 2 {
|
||||
t.Fatalf("after the household took it: %s sets=%v", act, q.sets)
|
||||
}
|
||||
// 5. The household REMOVED the box's limit (0) → theirs too.
|
||||
q2 := &fakeQB{}
|
||||
s, _, _ := ApplyQBLimit(ctx, q2, QBLimitState{}, 100, true, 0.5, now)
|
||||
q2.cur = 0
|
||||
if s2, act, _ := ApplyQBLimit(ctx, q2, s, 100, true, 0.5, now); act != "household" || !s2.Household {
|
||||
t.Fatalf("household removed the limit: %s", act)
|
||||
}
|
||||
// 6. A limit already there before the box ever looked → the household's.
|
||||
q3 := &fakeQB{cur: 500000}
|
||||
if s3, act, _ := ApplyQBLimit(ctx, q3, QBLimitState{}, 100, true, 0.5, now); act != "household" || !s3.Household || len(q3.sets) != 0 {
|
||||
t.Fatalf("pre-existing limit: %s", act)
|
||||
}
|
||||
// 6b. That household later REMOVES its own limit (unlimited again) → still theirs: the box does not step in.
|
||||
q3.cur = 0
|
||||
if _, act, _ := ApplyQBLimit(ctx, q3, QBLimitState{Household: true}, 100, true, 0.5, now); act != "household" || len(q3.sets) != 0 {
|
||||
t.Fatalf("household set unlimited after its own limit: %s sets=%v — the box overrode their choice", act, q3.sets)
|
||||
}
|
||||
// 7. No line measured → nothing set.
|
||||
q4 := &fakeQB{}
|
||||
if _, act, _ := ApplyQBLimit(ctx, q4, QBLimitState{}, 0, false, 0.5, now); act != "no-line" || len(q4.sets) != 0 {
|
||||
t.Fatalf("no line: %s", act)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
package upwatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"math"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ── qBittorrent's default upload limit (operator ruling 2026-10-23 19:22; the size is decide row R-1245) ─────────
|
||||
//
|
||||
// qBittorrent seeds at whatever the line gives (measured on 9202: 131 Mbit/s until the receiver capped it), so a
|
||||
// family's uploads — a video call, a photo sync — wait behind it. The box sets qBittorrent's GLOBAL upload limit to a
|
||||
// share of the measured line, and leaves it alone the moment the household touches it:
|
||||
//
|
||||
// - never set by the box, qBittorrent unlimited (0) → the box sets it
|
||||
// - never set by the box, qBittorrent already has a limit → the household's: never touched
|
||||
// - set by the box, qBittorrent still holds exactly that value → the box's: re-set when the line moves > 20 %
|
||||
// - set by the box, qBittorrent holds anything else (0 included) → the household changed it: never touched again
|
||||
//
|
||||
// The value compared is the one qBittorrent STORED (it keeps KiB/s: 2 500 000 reads back as 2 499 584, measured), so
|
||||
// the box's own rounding is never mistaken for a household's change. The limit survives a qBittorrent restart
|
||||
// (measured: `Session\GlobalUPSpeedLimit` in its config). No line measured yet → nothing is set.
|
||||
//
|
||||
// Edge: an app reinstalled from scratch starts unlimited while the record still says „set by the box"; that reads as
|
||||
// the household removing the limit and is left alone. Recorded in the design (`08` §9.7), not guarded.
|
||||
|
||||
// QBLimitState is what the box remembers (settings.json `upwatch.qb_limit`).
|
||||
type QBLimitState struct {
|
||||
BoxSetBytes int64 `json:"box_set_bytes,omitempty"` // the value qBittorrent stored when the box last set it
|
||||
LineMbps float64 `json:"line_mbps,omitempty"` // the line the box set it from
|
||||
SetAt string `json:"set_at,omitempty"`
|
||||
Household bool `json:"household,omitempty"` // the household owns the limit: never touched again
|
||||
HouseholdAt string `json:"household_at,omitempty"`
|
||||
}
|
||||
|
||||
// QBLimiter is the qBittorrent side the rule needs (the QBittorrent explainer satisfies it).
|
||||
type QBLimiter interface {
|
||||
UploadLimit(ctx context.Context, h map[string]string) (int64, error)
|
||||
SetUploadLimit(ctx context.Context, bytesPerSec int64) (int64, error)
|
||||
}
|
||||
|
||||
// QBLimitTarget is share × line, in bytes/s.
|
||||
func QBLimitTarget(lineMbps, share float64) int64 {
|
||||
return int64(math.Round(lineMbps * share * 1e6 / 8))
|
||||
}
|
||||
|
||||
// ApplyQBLimit runs the rule once. It returns the new state and what happened ("set", "kept", "household", "no-line",
|
||||
// "error"). It never lowers or raises a limit the household owns.
|
||||
func ApplyQBLimit(ctx context.Context, q QBLimiter, st QBLimitState, lineMbps float64, lineKnown bool, share float64, now time.Time) (QBLimitState, string, error) {
|
||||
if st.Household {
|
||||
return st, "household", nil
|
||||
}
|
||||
cur, err := q.UploadLimit(ctx, nil)
|
||||
if err != nil {
|
||||
return st, "error", err
|
||||
}
|
||||
if st.BoxSetBytes == 0 {
|
||||
if cur != 0 {
|
||||
st.Household, st.HouseholdAt = true, now.UTC().Format(time.RFC3339)
|
||||
return st, "household", nil
|
||||
}
|
||||
} else if cur != st.BoxSetBytes {
|
||||
st.Household, st.HouseholdAt = true, now.UTC().Format(time.RFC3339)
|
||||
return st, "household", nil
|
||||
} else if lineKnown && st.LineMbps > 0 && math.Abs(lineMbps-st.LineMbps)/st.LineMbps <= 0.2 {
|
||||
return st, "kept", nil
|
||||
}
|
||||
if !lineKnown {
|
||||
return st, "no-line", nil
|
||||
}
|
||||
stored, err := q.SetUploadLimit(ctx, QBLimitTarget(lineMbps, share))
|
||||
if err != nil {
|
||||
return st, "error", err
|
||||
}
|
||||
st.BoxSetBytes, st.LineMbps, st.SetAt = stored, lineMbps, now.UTC().Format(time.RFC3339)
|
||||
return st, "set", nil
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
package upwatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/logx"
|
||||
)
|
||||
|
||||
// State is what the watch keeps in settings.json (`upwatch`).
|
||||
type State struct {
|
||||
Line []LineRecord `json:"line,omitempty"`
|
||||
QBLimit *QBLimitState `json:"qb_limit,omitempty"`
|
||||
}
|
||||
|
||||
// Store is the settings half the service needs (settings.Settings satisfies it).
|
||||
type Store interface {
|
||||
GetUpwatch() json.RawMessage
|
||||
UpdateUpwatch(fn func(old json.RawMessage) (json.RawMessage, error)) error
|
||||
}
|
||||
|
||||
func decode(raw json.RawMessage) State {
|
||||
var st State
|
||||
if len(raw) > 0 {
|
||||
_ = json.Unmarshal(raw, &st) // a damaged block reads as empty: the line is re-learnt, the limit record too
|
||||
}
|
||||
return st
|
||||
}
|
||||
|
||||
// Service ties the watch together for main: the minute watcher, the push sampler, the qBittorrent limit rule and
|
||||
// what the pages show.
|
||||
type Service struct {
|
||||
Store Store
|
||||
Watcher *Watcher
|
||||
QB *QBittorrent // nil when the box has no qBittorrent wiring
|
||||
// QBRunning reports whether qBittorrent is installed and running now.
|
||||
QBRunning func() bool
|
||||
// QBShare is the share of the measured line qBittorrent may upload at (operator decide row; CC's pick 0.5).
|
||||
QBShare float64
|
||||
Log *log.Logger
|
||||
}
|
||||
|
||||
// Line is the measured upload line in Mbit/s (a lower bound); ok=false while no push counted.
|
||||
func (s *Service) Line() (float64, bool) {
|
||||
return LineFromRecords(decode(s.Store.GetUpwatch()).Line)
|
||||
}
|
||||
|
||||
// RecordPush stores one off-site push's measurement.
|
||||
func (s *Service) RecordPush(r LineRecord) {
|
||||
err := s.Store.UpdateUpwatch(func(old json.RawMessage) (json.RawMessage, error) {
|
||||
st := decode(old)
|
||||
st.Line = AddRecord(st.Line, r)
|
||||
return json.Marshal(st)
|
||||
})
|
||||
if err != nil {
|
||||
logx.Warnf(s.Log, "[upwatch] saving the push measurement: %v", err)
|
||||
return
|
||||
}
|
||||
l, ok := s.Line()
|
||||
logx.Infof(s.Log, "[upwatch] off-site push measured: %d bytes in %d s, best 10 s %.1f Mbit/s (counts=%v); line now %s",
|
||||
r.Bytes, r.Seconds, r.Peak10Mbps, r.Counts, lineWords(l, ok))
|
||||
}
|
||||
|
||||
func lineWords(l float64, ok bool) string {
|
||||
if !ok {
|
||||
return "unknown (learning)"
|
||||
}
|
||||
return "at least " + formatMbps(l) + " Mbit/s"
|
||||
}
|
||||
|
||||
// PushObserver is handed to the off-site backup: it is called at the push's start and returns the function that
|
||||
// ends the measurement.
|
||||
func (s *Service) PushObserver() func() {
|
||||
stop := (&PushSampler{}).Start()
|
||||
return func() { s.RecordPush(stop()) }
|
||||
}
|
||||
|
||||
// QBLimitNow runs qBittorrent's limit rule once (main calls it every 15 minutes; a no-op without qBittorrent).
|
||||
func (s *Service) QBLimitNow(ctx context.Context) {
|
||||
if s.QB == nil || (s.QBRunning != nil && !s.QBRunning()) {
|
||||
return
|
||||
}
|
||||
line, known := s.Line()
|
||||
cur := decode(s.Store.GetUpwatch())
|
||||
var st QBLimitState
|
||||
if cur.QBLimit != nil {
|
||||
st = *cur.QBLimit
|
||||
}
|
||||
ns, act, err := ApplyQBLimit(ctx, s.QB, st, line, known, s.QBShare, time.Now())
|
||||
if err != nil {
|
||||
logx.Debugf(s.Log, "[upwatch] qBittorrent limit: %v — tried again in 15 minutes", err)
|
||||
return
|
||||
}
|
||||
if ns != st {
|
||||
if err := s.Store.UpdateUpwatch(func(old json.RawMessage) (json.RawMessage, error) {
|
||||
o := decode(old)
|
||||
o.QBLimit = &ns
|
||||
return json.Marshal(o)
|
||||
}); err != nil {
|
||||
logx.Warnf(s.Log, "[upwatch] saving the qBittorrent limit record: %v", err)
|
||||
return
|
||||
}
|
||||
}
|
||||
switch act {
|
||||
case "set":
|
||||
logx.Infof(s.Log, "[upwatch] qBittorrent upload limit set to %d B/s (%.0f %% of the measured line, %.1f Mbit/s)", ns.BoxSetBytes, s.QBShare*100, line)
|
||||
case "household":
|
||||
if !st.Household {
|
||||
logx.Infof(s.Log, "[upwatch] qBittorrent's upload limit is the household's — the box leaves it alone from now on")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// View is what the pages show.
|
||||
type View struct {
|
||||
LineMbps float64
|
||||
LineKnown bool
|
||||
QBLimitBps int64 // the limit the box set and still owns; 0 = none shown
|
||||
}
|
||||
|
||||
// View reads the state for the pages.
|
||||
func (s *Service) View() View {
|
||||
st := decode(s.Store.GetUpwatch())
|
||||
v := View{}
|
||||
v.LineMbps, v.LineKnown = LineFromRecords(st.Line)
|
||||
if st.QBLimit != nil && !st.QBLimit.Household {
|
||||
v.QBLimitBps = st.QBLimit.BoxSetBytes
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// MbpsText renders bytes/s as Mbit/s for a page line (one decimal under 10).
|
||||
func MbpsText(bps int64) string { return formatMbps(float64(bps) * 8 / 1e6) }
|
||||
|
||||
func formatMbps(v float64) string {
|
||||
if v < 10 {
|
||||
return trimZero(v)
|
||||
}
|
||||
return itoa(int64(v + 0.5))
|
||||
}
|
||||
|
||||
func trimZero(v float64) string {
|
||||
x := int64(v*10 + 0.5)
|
||||
if x%10 == 0 {
|
||||
return itoa(x / 10)
|
||||
}
|
||||
return itoa(x/10) + "." + itoa(x%10)
|
||||
}
|
||||
|
||||
func itoa(v int64) string {
|
||||
b, _ := json.Marshal(v)
|
||||
return string(b)
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
// Package upwatch is the upload and CPU watch (felhom.eu `architecture/08-alarm-ladder.md` §9 [DESIGN — proposed],
|
||||
// spike `audits/SPIKE-upload-watch-2026-10-23.md`). Once a minute it reads every app container's EXACT sent bytes and
|
||||
// CPU time, adds them up per app, and when an app sends a lot (or burns CPU) for a long time it ASKS THE APP whether
|
||||
// that is its normal work — qBittorrent's and Transmission's own upload counters, Plex's playing and transcoding
|
||||
// sessions. Work the app explains is normal use. Work it does not explain is the sign of a hacked app (DooPlex's Plex
|
||||
// sent 474 GB in one day with no viewer, 2026-10-23) and becomes ONE operator-only event, cleared when it stops.
|
||||
//
|
||||
// It also learns the box's upload line from what the box already does (its own off-site push, line.go) and gives
|
||||
// qBittorrent a default upload limit from it, never over a limit the household set (qblimit.go).
|
||||
//
|
||||
// SECRETS: app passwords and tokens are read at use (the mediawire seams, `09` §3 decision 248), sent only to the
|
||||
// app's own container, never stored, never logged.
|
||||
package upwatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Sample is one container's cumulative counters at one moment.
|
||||
type Sample struct {
|
||||
Container string
|
||||
Stack string // the compose project; "" for a container outside any stack
|
||||
TxBytes uint64 // bytes sent on every network of the container, since it started
|
||||
CPUNanos uint64 // CPU time used, since it started
|
||||
Started string // the container's start time: a change means the counters restarted from zero
|
||||
}
|
||||
|
||||
// Source returns one Sample per running container.
|
||||
type Source interface {
|
||||
Samples(ctx context.Context) ([]Sample, error)
|
||||
}
|
||||
|
||||
// DockerAPI reads the counters from the Docker Engine API on the guest's socket.
|
||||
//
|
||||
// WHY NOT `docker stats` (what internal/metrics uses): its NetIO is text with three significant digits ("1.23GB"), so
|
||||
// at a large total one step is ~10 MB — 1.3 Mbit/s at a one-minute sample. The Engine API gives integers; measured on
|
||||
// 9202 it equals the kernel's own counter in the container's network namespace to the byte
|
||||
// (`audits/SPIKE-upload-watch-2026-10-23/s4-qb-seed-nolimit.jsonl`: proc_tx == engine_tx).
|
||||
type DockerAPI struct {
|
||||
Socket string // default /var/run/docker.sock
|
||||
client *http.Client
|
||||
}
|
||||
|
||||
func (d *DockerAPI) http() *http.Client {
|
||||
if d.client == nil {
|
||||
sock := d.Socket
|
||||
if sock == "" {
|
||||
sock = "/var/run/docker.sock"
|
||||
}
|
||||
d.client = &http.Client{
|
||||
Timeout: 20 * time.Second,
|
||||
Transport: &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
|
||||
var dl net.Dialer
|
||||
return dl.DialContext(ctx, "unix", sock) // dialled by path every time: sockheal may re-create the file
|
||||
}},
|
||||
}
|
||||
}
|
||||
return d.client
|
||||
}
|
||||
|
||||
func (d *DockerAPI) get(ctx context.Context, path string, out interface{}) error {
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", "http://docker"+path, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := d.http().Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
b, _ := io.ReadAll(io.LimitReader(resp.Body, 512))
|
||||
return fmt.Errorf("docker API %s: HTTP %d %s", path, resp.StatusCode, strings.TrimSpace(string(b)))
|
||||
}
|
||||
return json.NewDecoder(io.LimitReader(resp.Body, 8<<20)).Decode(out)
|
||||
}
|
||||
|
||||
// Samples lists the running containers and reads each one's counters (one-shot stats: no one-second wait).
|
||||
func (d *DockerAPI) Samples(ctx context.Context) ([]Sample, error) {
|
||||
var list []struct {
|
||||
ID string `json:"Id"`
|
||||
Names []string `json:"Names"`
|
||||
Labels map[string]string `json:"Labels"`
|
||||
}
|
||||
if err := d.get(ctx, "/containers/json", &list); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]Sample, 0, len(list))
|
||||
for _, c := range list {
|
||||
var st struct {
|
||||
Read string `json:"read"`
|
||||
Networks map[string]struct {
|
||||
TxBytes uint64 `json:"tx_bytes"`
|
||||
} `json:"networks"`
|
||||
CPUStats struct {
|
||||
CPUUsage struct {
|
||||
TotalUsage uint64 `json:"total_usage"`
|
||||
} `json:"cpu_usage"`
|
||||
} `json:"cpu_stats"`
|
||||
}
|
||||
if err := d.get(ctx, "/containers/"+c.ID+"/stats?stream=false&one-shot=true", &st); err != nil {
|
||||
continue // the container stopped between the list and the read: it is simply not sampled this minute
|
||||
}
|
||||
var started struct {
|
||||
State struct {
|
||||
StartedAt string `json:"StartedAt"`
|
||||
} `json:"State"`
|
||||
}
|
||||
_ = d.get(ctx, "/containers/"+c.ID+"/json", &started)
|
||||
s := Sample{Stack: c.Labels["com.docker.compose.project"], CPUNanos: st.CPUStats.CPUUsage.TotalUsage, Started: started.State.StartedAt}
|
||||
if len(c.Names) > 0 {
|
||||
s.Container = strings.TrimPrefix(c.Names[0], "/")
|
||||
}
|
||||
for _, n := range st.Networks {
|
||||
s.TxBytes += n.TxBytes
|
||||
}
|
||||
out = append(out, s)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,423 @@
|
||||
package upwatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"runtime"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/logx"
|
||||
)
|
||||
|
||||
// Rules are the watch's numbers. Every default comes from the spike (`audits/SPIKE-upload-watch-2026-10-23.md`); the
|
||||
// field comment says which measurement.
|
||||
type Rules struct {
|
||||
// UploadFloorMbps / UploadLineShare: an app is "uploading a lot" at max(floor, share × the measured line). The floor
|
||||
// stands alone while the line is unknown. 5 Mbit/s: on three boxes over 19–31 days no app's own p95 reached 6 Mbit/s
|
||||
// and the longest run at ≥ 5 Mbit/s was 3 minutes; the DooPlex flood averaged ~44 Mbit/s for a day.
|
||||
UploadFloorMbps float64
|
||||
UploadLineShare float64
|
||||
// UploadSustainMin: how long. 15 = five times the longest normal run at ≥ 5 Mbit/s (3 min, qBittorrent on Tester 1).
|
||||
UploadSustainMin int
|
||||
// ExplainedRatio / ExplainedSlackMbps: the app explains its bytes when container_bytes ≤ ratio × app_bytes + slack.
|
||||
// Measured overhead: qBittorrent 3–7 %, Transmission 0.1–2 %; 25 % + 1 Mbit/s also covers DHT, trackers and the web UI.
|
||||
ExplainedRatio float64
|
||||
ExplainedSlackMbps float64
|
||||
// NoExplainerFactor / NoExplainerSustainMin: an app the box cannot ask alarms only when it sends ≥ factor × its own
|
||||
// p95 (and ≥ the trigger) for this long — worded „no app number to compare".
|
||||
NoExplainerFactor float64
|
||||
NoExplainerSustainMin int
|
||||
// CPU: an app burns CPU at max(CPUFloorPct, CPUCoreShare × cores × 100) percent-of-one-core for CPUSustainMin with
|
||||
// no explaining work. Measured: the longest normal run at ≥ 150 % was 7 minutes (Immich machine learning) and at
|
||||
// ≥ 80 % 13 minutes; DooPlex's hacked Plex used ~10 cores.
|
||||
CPUFloorPct float64
|
||||
CPUCoreShare float64
|
||||
CPUSustainMin int
|
||||
NoExplainerCPUFloor float64
|
||||
NoExplainerCPUShare float64
|
||||
NoExplainerCPUSustain int
|
||||
// ClearAfterMin: an alarm clears after this many minutes in a row below the trigger (or explained).
|
||||
ClearAfterMin int
|
||||
// Cores: the guest's CPU count (0 → runtime.NumCPU, which honours the LXC's cpuset).
|
||||
Cores int
|
||||
}
|
||||
|
||||
// DefaultRules are the spike's numbers.
|
||||
func DefaultRules() Rules {
|
||||
return Rules{
|
||||
UploadFloorMbps: 5, UploadLineShare: 0.25, UploadSustainMin: 15,
|
||||
ExplainedRatio: 1.25, ExplainedSlackMbps: 1,
|
||||
NoExplainerFactor: 10, NoExplainerSustainMin: 60,
|
||||
CPUFloorPct: 150, CPUCoreShare: 0.5, CPUSustainMin: 30,
|
||||
NoExplainerCPUFloor: 200, NoExplainerCPUShare: 0.75, NoExplainerCPUSustain: 60,
|
||||
ClearAfterMin: 15,
|
||||
}
|
||||
}
|
||||
|
||||
// Infrastructure stacks are never judged: the controller's own off-site push is the line measure (line.go), and
|
||||
// traefik/cloudflared relay the APPS' traffic (the app's own container sent it first, and is judged there).
|
||||
var infrastructure = map[string]bool{"felhom-controller": true, "traefik": true, "cloudflared": true, "felhom-samba": true, "samba": true}
|
||||
|
||||
// Details is the event's payload (operator-only; the hub mails the operator a line built from it).
|
||||
type Details struct {
|
||||
App string `json:"app"`
|
||||
StackName string `json:"stack_name"`
|
||||
Kind string `json:"kind"` // "upload" | "cpu"
|
||||
Minutes int `json:"minutes"` // how long the condition held when judged
|
||||
RateMbps float64 `json:"rate_mbps,omitempty"` // the app's containers, average over the window
|
||||
AppMbps float64 `json:"app_mbps"` // what the app itself counted; -1 = it has no number
|
||||
CPUPercent float64 `json:"cpu_percent,omitempty"` // percent of ONE core, average over the window
|
||||
Explainer string `json:"explainer"` // what was asked: "qbittorrent", "transmission", "plex", "none"
|
||||
Viewers int `json:"viewers"`
|
||||
Transcodes int `json:"transcodes"`
|
||||
LineMbps float64 `json:"line_mbps,omitempty"` // the measured line, when known
|
||||
Cleared bool `json:"cleared,omitempty"`
|
||||
}
|
||||
|
||||
// Watcher holds the per-app state. One goroutine (Run) drives it; Tick is the testable step.
|
||||
type Watcher struct {
|
||||
Source Source
|
||||
Explainers map[string]Explainer // by stack
|
||||
Rules Rules
|
||||
// Line returns the measured upload line in Mbit/s (line.go); ok=false while unknown.
|
||||
Line func() (float64, bool)
|
||||
// Baseline returns the app's own p95 upload in Mbit/s over the stored history (its containers summed per minute);
|
||||
// ok=false until ≥ 24 h exist.
|
||||
Baseline func(stack string, containers []string) (float64, bool)
|
||||
// Busy reports work the BOX is doing on the app (backup, update, restore) — that minute is never judged.
|
||||
Busy func(stack string) bool
|
||||
Emit func(eventType string, severity string, d Details)
|
||||
Log *log.Logger
|
||||
Now func() time.Time
|
||||
|
||||
prevAt time.Time
|
||||
prev map[string]Sample // by container
|
||||
apps map[string]*appState
|
||||
asked map[string]Report // last report per stack (for counter deltas)
|
||||
askedOK map[string]bool
|
||||
}
|
||||
|
||||
type appState struct {
|
||||
upRun, upClear int
|
||||
winTx, winApp uint64
|
||||
winSec float64
|
||||
winViewers, winTrans int
|
||||
appCounterKnown bool
|
||||
upAlarm bool
|
||||
|
||||
cpuRun, cpuClear int
|
||||
cpuSum float64
|
||||
cpuAlarm bool
|
||||
|
||||
lastRate float64 // the last judged minute's Mbit/s (tests read it)
|
||||
}
|
||||
|
||||
func (w *Watcher) now() time.Time {
|
||||
if w.Now != nil {
|
||||
return w.Now()
|
||||
}
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
func (w *Watcher) cores() int {
|
||||
if w.Rules.Cores > 0 {
|
||||
return w.Rules.Cores
|
||||
}
|
||||
return runtime.NumCPU()
|
||||
}
|
||||
|
||||
// UploadTrigger is the Mbit/s at which an app is judged.
|
||||
func (w *Watcher) UploadTrigger() float64 {
|
||||
t := w.Rules.UploadFloorMbps
|
||||
if w.Line != nil {
|
||||
if l, ok := w.Line(); ok && l*w.Rules.UploadLineShare > t {
|
||||
t = l * w.Rules.UploadLineShare
|
||||
}
|
||||
}
|
||||
return t
|
||||
}
|
||||
|
||||
func maxf(a, b float64) float64 {
|
||||
if a > b {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// Run ticks once a minute until ctx ends.
|
||||
func (w *Watcher) Run(ctx context.Context, every time.Duration) {
|
||||
t := time.NewTicker(every)
|
||||
defer t.Stop()
|
||||
w.Tick(ctx)
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
w.Tick(ctx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Tick samples every container, adds the deltas up per app, and judges each app.
|
||||
func (w *Watcher) Tick(ctx context.Context) {
|
||||
now := w.now()
|
||||
samples, err := w.Source.Samples(ctx)
|
||||
if err != nil {
|
||||
logx.Warnf(w.Log, "[upwatch] reading container counters: %v — this minute is skipped", err)
|
||||
return
|
||||
}
|
||||
if w.prev == nil {
|
||||
w.prev, w.apps, w.asked, w.askedOK = map[string]Sample{}, map[string]*appState{}, map[string]Report{}, map[string]bool{}
|
||||
}
|
||||
first := w.prevAt.IsZero()
|
||||
dt := now.Sub(w.prevAt).Seconds()
|
||||
type agg struct {
|
||||
tx, cpu uint64
|
||||
containers []string
|
||||
}
|
||||
per := map[string]*agg{}
|
||||
cur := map[string]Sample{}
|
||||
for _, s := range samples {
|
||||
cur[s.Container] = s
|
||||
if s.Stack == "" || infrastructure[s.Stack] {
|
||||
continue
|
||||
}
|
||||
a := per[s.Stack]
|
||||
if a == nil {
|
||||
a = &agg{}
|
||||
per[s.Stack] = a
|
||||
}
|
||||
a.containers = append(a.containers, s.Container)
|
||||
p, ok := w.prev[s.Container]
|
||||
if !ok || p.Started != s.Started || s.TxBytes < p.TxBytes || s.CPUNanos < p.CPUNanos {
|
||||
continue // new or restarted container: its counters start now; no delta this minute
|
||||
}
|
||||
a.tx += s.TxBytes - p.TxBytes
|
||||
a.cpu += s.CPUNanos - p.CPUNanos
|
||||
}
|
||||
w.prev, w.prevAt = cur, now
|
||||
if first || dt <= 0 || dt > 600 {
|
||||
// The first tick only sets the baseline; a gap over 10 minutes (a suspended guest, a stalled tick) is not a
|
||||
// minute anyone can judge. The explainers are asked so their counters have a baseline too.
|
||||
for stack := range per {
|
||||
w.ask(ctx, stack)
|
||||
}
|
||||
return
|
||||
}
|
||||
stacks := make([]string, 0, len(per))
|
||||
for s := range per {
|
||||
stacks = append(stacks, s)
|
||||
}
|
||||
sort.Strings(stacks)
|
||||
for _, stack := range stacks {
|
||||
a := per[stack]
|
||||
w.judge(ctx, stack, a.containers, a.tx, a.cpu, dt)
|
||||
}
|
||||
for stack, st := range w.apps { // an app that vanished (stopped, removed) clears what it had raised
|
||||
if _, ok := per[stack]; !ok && (st.upAlarm || st.cpuAlarm) {
|
||||
w.clear(stack, st, "upload", st.upAlarm, 0, -1)
|
||||
w.clear(stack, st, "cpu", st.cpuAlarm, 0, -1)
|
||||
delete(w.apps, stack)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ask returns the app's delta of its own upload counter since the last ask (ok=false when it has none or when this
|
||||
// is the first reading or the app restarted), plus the report.
|
||||
func (w *Watcher) ask(ctx context.Context, stack string) (rep Report, upDelta uint64, deltaOK bool, has bool) {
|
||||
ex, ok := w.Explainers[stack]
|
||||
if !ok {
|
||||
return Report{}, 0, false, false
|
||||
}
|
||||
r, err := ex.Ask(ctx)
|
||||
if err != nil {
|
||||
logx.Debugf(w.Log, "[upwatch] %s: the app did not answer (%v) — treated as no number this minute", stack, err)
|
||||
delete(w.askedOK, stack)
|
||||
return Report{}, 0, false, true
|
||||
}
|
||||
prev, had := w.asked[stack], w.askedOK[stack]
|
||||
w.asked[stack], w.askedOK[stack] = r, true
|
||||
if r.HasUploadCounter && had && prev.HasUploadCounter && r.UploadCounter >= prev.UploadCounter {
|
||||
return r, r.UploadCounter - prev.UploadCounter, true, true
|
||||
}
|
||||
return r, 0, false, true
|
||||
}
|
||||
|
||||
func (w *Watcher) judge(ctx context.Context, stack string, containers []string, tx, cpu uint64, dt float64) {
|
||||
st := w.apps[stack]
|
||||
if st == nil {
|
||||
st = &appState{}
|
||||
w.apps[stack] = st
|
||||
}
|
||||
rate := float64(tx) * 8 / dt / 1e6
|
||||
cpuPct := float64(cpu) / (dt * 1e9) * 100
|
||||
st.lastRate = rate
|
||||
rep, upDelta, deltaOK, hasEx := w.ask(ctx, stack)
|
||||
busy := w.Busy != nil && w.Busy(stack)
|
||||
|
||||
// ── upload ──
|
||||
trig := w.UploadTrigger()
|
||||
sustain := w.Rules.UploadSustainMin
|
||||
if !hasEx {
|
||||
p95, ok := baseline(w.Baseline, stack, containers)
|
||||
if !ok {
|
||||
trig = -1 // learning: an app the box cannot ask is not judged before it has a history
|
||||
} else {
|
||||
trig = maxf(trig, w.Rules.NoExplainerFactor*p95)
|
||||
sustain = w.Rules.NoExplainerSustainMin
|
||||
}
|
||||
}
|
||||
over := trig > 0 && rate >= trig && !busy
|
||||
slack := w.Rules.ExplainedSlackMbps * 1e6 / 8 * dt
|
||||
minuteExplained := (deltaOK && float64(tx) <= w.Rules.ExplainedRatio*float64(upDelta)+slack) || (rep.HasViewerCount && rep.Viewers > 0)
|
||||
if !st.upAlarm {
|
||||
if over {
|
||||
st.upRun++
|
||||
st.winTx += tx
|
||||
st.winSec += dt
|
||||
if deltaOK {
|
||||
st.winApp += upDelta
|
||||
st.appCounterKnown = true
|
||||
}
|
||||
if rep.Viewers > st.winViewers {
|
||||
st.winViewers = rep.Viewers
|
||||
}
|
||||
if st.upRun >= sustain {
|
||||
d := Details{App: stack, StackName: stack, Kind: "upload", Minutes: st.upRun, RateMbps: round2(float64(st.winTx) * 8 / st.winSec / 1e6),
|
||||
AppMbps: -1, Explainer: explainerName(stack, hasEx), Viewers: st.winViewers}
|
||||
if l, ok := lineOf(w.Line); ok {
|
||||
d.LineMbps = round2(l)
|
||||
}
|
||||
explained := false
|
||||
switch {
|
||||
case st.appCounterKnown:
|
||||
d.AppMbps = round2(float64(st.winApp) * 8 / st.winSec / 1e6)
|
||||
explained = float64(st.winTx) <= w.Rules.ExplainedRatio*float64(st.winApp)+w.Rules.ExplainedSlackMbps*1e6/8*st.winSec
|
||||
case st.winViewers > 0:
|
||||
explained = true
|
||||
}
|
||||
if explained {
|
||||
logx.Infof(w.Log, "[upwatch] %s sent %.1f Mbit/s for %d min — explained by the app (%s: %.1f Mbit/s, %d viewer(s)); normal use",
|
||||
stack, d.RateMbps, d.Minutes, d.Explainer, d.AppMbps, d.Viewers)
|
||||
} else {
|
||||
st.upAlarm, st.upClear = true, 0
|
||||
logx.Warnf(w.Log, "[upwatch] %s sent %.1f Mbit/s for %d min — NOT explained by the app (%s: %s, %d viewer(s)) — operator alarm",
|
||||
stack, d.RateMbps, d.Minutes, d.Explainer, appNumber(d.AppMbps), d.Viewers)
|
||||
w.emit("app_unexplained_upload", "warning", d)
|
||||
}
|
||||
st.upRun, st.winTx, st.winApp, st.winSec, st.winViewers, st.appCounterKnown = 0, 0, 0, 0, 0, false
|
||||
}
|
||||
} else {
|
||||
st.upRun, st.winTx, st.winApp, st.winSec, st.winViewers, st.appCounterKnown = 0, 0, 0, 0, 0, false
|
||||
}
|
||||
} else {
|
||||
if !over || minuteExplained {
|
||||
st.upClear++
|
||||
} else {
|
||||
st.upClear = 0
|
||||
}
|
||||
if st.upClear >= w.Rules.ClearAfterMin {
|
||||
w.clear(stack, st, "upload", true, rate, -1)
|
||||
}
|
||||
}
|
||||
|
||||
// ── CPU ──
|
||||
cores := float64(w.cores())
|
||||
ctrig, csus := maxf(w.Rules.CPUFloorPct, w.Rules.CPUCoreShare*cores*100), w.Rules.CPUSustainMin
|
||||
if !(rep.HasTranscodeCount) {
|
||||
ctrig, csus = maxf(w.Rules.NoExplainerCPUFloor, w.Rules.NoExplainerCPUShare*cores*100), w.Rules.NoExplainerCPUSustain
|
||||
}
|
||||
cpuExplained := busy || (rep.HasTranscodeCount && rep.Transcodes > 0)
|
||||
cpuOver := cpuPct >= ctrig && !cpuExplained
|
||||
if !st.cpuAlarm {
|
||||
if cpuOver {
|
||||
st.cpuRun++
|
||||
st.cpuSum += cpuPct
|
||||
if st.cpuRun >= csus {
|
||||
d := Details{App: stack, StackName: stack, Kind: "cpu", Minutes: st.cpuRun, CPUPercent: round2(st.cpuSum / float64(st.cpuRun)),
|
||||
AppMbps: -1, Explainer: cpuExplainerName(stack, rep.HasTranscodeCount), Transcodes: rep.Transcodes}
|
||||
st.cpuAlarm, st.cpuClear, st.cpuRun, st.cpuSum = true, 0, 0, 0
|
||||
logx.Warnf(w.Log, "[upwatch] %s used %.0f%% CPU (of one core; %d cores) for %d min with no work that explains it (%s) — operator alarm",
|
||||
stack, d.CPUPercent, int(cores), d.Minutes, d.Explainer)
|
||||
w.emit("app_unexplained_cpu", "warning", d)
|
||||
}
|
||||
} else {
|
||||
st.cpuRun, st.cpuSum = 0, 0
|
||||
}
|
||||
} else {
|
||||
if !cpuOver {
|
||||
st.cpuClear++
|
||||
} else {
|
||||
st.cpuClear = 0
|
||||
}
|
||||
if st.cpuClear >= w.Rules.ClearAfterMin {
|
||||
w.clear(stack, st, "cpu", true, 0, cpuPct)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Watcher) clear(stack string, st *appState, kind string, raised bool, rate, cpuPct float64) {
|
||||
if !raised {
|
||||
return
|
||||
}
|
||||
d := Details{App: stack, StackName: stack, Kind: kind, Minutes: w.Rules.ClearAfterMin, AppMbps: -1, Cleared: true}
|
||||
if kind == "upload" {
|
||||
st.upAlarm, st.upClear, st.upRun = false, 0, 0
|
||||
d.RateMbps = round2(rate)
|
||||
logx.Infof(w.Log, "[upwatch] %s: the unexplained upload stopped (%d min below the trigger or explained) — cleared", stack, d.Minutes)
|
||||
w.emit("app_unexplained_upload_cleared", "info", d)
|
||||
return
|
||||
}
|
||||
st.cpuAlarm, st.cpuClear, st.cpuRun, st.cpuSum = false, 0, 0, 0
|
||||
if cpuPct >= 0 {
|
||||
d.CPUPercent = round2(cpuPct)
|
||||
}
|
||||
logx.Infof(w.Log, "[upwatch] %s: the unexplained CPU stopped (%d min below the trigger or explained) — cleared", stack, d.Minutes)
|
||||
w.emit("app_unexplained_cpu_cleared", "info", d)
|
||||
}
|
||||
|
||||
func (w *Watcher) emit(t, sev string, d Details) {
|
||||
if w.Emit != nil {
|
||||
w.Emit(t, sev, d)
|
||||
}
|
||||
}
|
||||
|
||||
func baseline(fn func(string, []string) (float64, bool), stack string, containers []string) (float64, bool) {
|
||||
if fn == nil {
|
||||
return 0, false
|
||||
}
|
||||
return fn(stack, containers)
|
||||
}
|
||||
|
||||
func lineOf(fn func() (float64, bool)) (float64, bool) {
|
||||
if fn == nil {
|
||||
return 0, false
|
||||
}
|
||||
return fn()
|
||||
}
|
||||
|
||||
func explainerName(stack string, has bool) string {
|
||||
if has {
|
||||
return stack
|
||||
}
|
||||
return "none"
|
||||
}
|
||||
|
||||
func cpuExplainerName(stack string, has bool) string {
|
||||
if has {
|
||||
return stack + " transcodes"
|
||||
}
|
||||
return "none"
|
||||
}
|
||||
|
||||
func appNumber(v float64) string {
|
||||
if v < 0 {
|
||||
return "no app number to compare"
|
||||
}
|
||||
return fmt.Sprintf("%.1f Mbit/s", v)
|
||||
}
|
||||
|
||||
func round2(v float64) float64 { return float64(int64(v*100+0.5)) / 100 }
|
||||
@@ -0,0 +1,312 @@
|
||||
package upwatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The fake box: containers whose counters the test drives minute by minute, and apps that answer what the test says.
|
||||
|
||||
type fakeSource struct{ s map[string]*Sample }
|
||||
|
||||
func (f *fakeSource) Samples(context.Context) ([]Sample, error) {
|
||||
var out []Sample
|
||||
for _, s := range f.s {
|
||||
out = append(out, *s)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (f *fakeSource) add(container, stack string) *Sample {
|
||||
if f.s == nil {
|
||||
f.s = map[string]*Sample{}
|
||||
}
|
||||
s := &Sample{Container: container, Stack: stack, Started: "t0"}
|
||||
f.s[container] = s
|
||||
return s
|
||||
}
|
||||
|
||||
type fakeApp struct {
|
||||
stack string
|
||||
rep Report
|
||||
err error
|
||||
}
|
||||
|
||||
func (a *fakeApp) Stack() string { return a.stack }
|
||||
func (a *fakeApp) Ask(context.Context) (Report, error) { return a.rep, a.err }
|
||||
|
||||
type event struct {
|
||||
t string
|
||||
d Details
|
||||
}
|
||||
|
||||
type rig struct {
|
||||
w *Watcher
|
||||
src *fakeSource
|
||||
now time.Time
|
||||
ev []event
|
||||
}
|
||||
|
||||
func newRig(ex ...*fakeApp) *rig {
|
||||
r := &rig{src: &fakeSource{}, now: time.Date(2026, 10, 23, 22, 0, 0, 0, time.UTC)}
|
||||
m := map[string]Explainer{}
|
||||
for _, e := range ex {
|
||||
m[e.stack] = e
|
||||
}
|
||||
rules := DefaultRules()
|
||||
rules.Cores = 4
|
||||
r.w = &Watcher{Source: r.src, Explainers: m, Rules: rules, Log: log.New(io.Discard, "", 0),
|
||||
Now: func() time.Time { return r.now },
|
||||
Emit: func(t, _ string, d Details) { r.ev = append(r.ev, event{t, d}) }}
|
||||
return r
|
||||
}
|
||||
|
||||
const mbit = 1e6 / 8 // bytes per second at 1 Mbit/s
|
||||
|
||||
// minute advances the clock one minute; each container sends txMbps and burns cpuPct (of one core) during it.
|
||||
func (r *rig) minute(t *testing.T, step func()) {
|
||||
t.Helper()
|
||||
if step != nil {
|
||||
step()
|
||||
}
|
||||
r.now = r.now.Add(time.Minute)
|
||||
r.w.Tick(context.Background())
|
||||
}
|
||||
|
||||
func send(s *Sample, mbps, cpuPct float64) {
|
||||
s.TxBytes += uint64(mbps * mbit * 60)
|
||||
s.CPUNanos += uint64(cpuPct / 100 * 60e9)
|
||||
}
|
||||
|
||||
func (r *rig) count(t string) int {
|
||||
n := 0
|
||||
for _, e := range r.ev {
|
||||
if e.t == t {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// A real torrent seeding at full speed: qBittorrent's own counter explains the bytes (the measured 3–7 % overhead).
|
||||
// RED-PROOF: make the explained test always false → an alarm fires → fails.
|
||||
func TestUpwatch_SeedingQBittorrentIsExplained(t *testing.T) {
|
||||
qb := &fakeApp{stack: "qbittorrent", rep: Report{HasUploadCounter: true}}
|
||||
r := newRig(qb)
|
||||
c := r.src.add("qbittorrent", "qbittorrent")
|
||||
r.w.Tick(context.Background())
|
||||
for i := 0; i < 90; i++ {
|
||||
r.minute(t, func() {
|
||||
send(c, 131*1.05, 20) // the container: the app's bytes + 5 % protocol overhead
|
||||
qb.rep.UploadCounter += uint64(131 * mbit * 60) // what qBittorrent counted
|
||||
})
|
||||
}
|
||||
if len(r.ev) != 0 {
|
||||
t.Fatalf("a seeding qBittorrent raised %v — its own counter explains the upload", r.ev)
|
||||
}
|
||||
}
|
||||
|
||||
// The fake hacked app: code inside qBittorrent's container floods out while qBittorrent counts nothing. ONE alarm after
|
||||
// 15 minutes, none more while it goes on, ONE clear 15 minutes after it stops.
|
||||
// RED-PROOF: drop `st.upAlarm = true` (no edge) → an alarm every 15 minutes → count != 1 → fails; drop the clear →
|
||||
// no cleared event → fails.
|
||||
func TestUpwatch_HackedQBittorrentAlarmsOnceAndClears(t *testing.T) {
|
||||
qb := &fakeApp{stack: "qbittorrent", rep: Report{HasUploadCounter: true}}
|
||||
r := newRig(qb)
|
||||
c := r.src.add("qbittorrent", "qbittorrent")
|
||||
r.w.Tick(context.Background())
|
||||
for i := 1; i <= 14; i++ {
|
||||
r.minute(t, func() { send(c, 900, 30) })
|
||||
}
|
||||
if len(r.ev) != 0 {
|
||||
t.Fatalf("alarm before 15 minutes: %v", r.ev)
|
||||
}
|
||||
r.minute(t, func() { send(c, 900, 30) })
|
||||
if r.count("app_unexplained_upload") != 1 {
|
||||
t.Fatalf("after 15 minutes of 900 Mbit/s with qBittorrent counting 0: events %v, want one app_unexplained_upload", r.ev)
|
||||
}
|
||||
d := r.ev[0].d
|
||||
if d.App != "qbittorrent" || d.Kind != "upload" || d.AppMbps != 0 || d.RateMbps < 899 || d.Minutes != 15 || d.Explainer != "qbittorrent" {
|
||||
t.Fatalf("details = %+v", d)
|
||||
}
|
||||
for i := 0; i < 120; i++ {
|
||||
r.minute(t, func() { send(c, 900, 30) })
|
||||
}
|
||||
if r.count("app_unexplained_upload") != 1 {
|
||||
t.Fatalf("a flood that goes on alarmed %d times — must be ONE mail", r.count("app_unexplained_upload"))
|
||||
}
|
||||
for i := 0; i < 14; i++ {
|
||||
r.minute(t, func() { send(c, 0.01, 1) })
|
||||
}
|
||||
if r.count("app_unexplained_upload_cleared") != 0 {
|
||||
t.Fatal("cleared before 15 quiet minutes")
|
||||
}
|
||||
r.minute(t, func() { send(c, 0.01, 1) })
|
||||
if r.count("app_unexplained_upload_cleared") != 1 {
|
||||
t.Fatalf("15 quiet minutes after the flood: events %v, want one clear", r.ev)
|
||||
}
|
||||
}
|
||||
|
||||
// Plex2Shell's signature: Plex sends a lot with NO viewer. With a viewer, the same bytes are normal use.
|
||||
// RED-PROOF: drop the `winViewers > 0` explanation → the viewer case alarms → fails.
|
||||
func TestUpwatch_PlexWithNoViewerAlarmsWithAViewerDoesNot(t *testing.T) {
|
||||
for _, viewers := range []int{0, 2} {
|
||||
px := &fakeApp{stack: "plex", rep: Report{HasViewerCount: true, Viewers: viewers, HasTranscodeCount: true}}
|
||||
r := newRig(px)
|
||||
c := r.src.add("plex", "plex")
|
||||
r.w.Tick(context.Background())
|
||||
for i := 0; i < 20; i++ {
|
||||
r.minute(t, func() { send(c, 44, 10) })
|
||||
}
|
||||
got := r.count("app_unexplained_upload")
|
||||
if viewers == 0 && got != 1 {
|
||||
t.Fatalf("Plex sending 44 Mbit/s with no viewer for 20 min: %d alarm(s), want 1", got)
|
||||
}
|
||||
if viewers > 0 && got != 0 {
|
||||
t.Fatalf("Plex sending to %d viewers alarmed: %v", viewers, r.ev)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// An app the box cannot ask: no alarm while it has no history (learning), then only far above its own p95 for an hour.
|
||||
// RED-PROOF: let a no-explainer app use the 15-minute rule → it alarms at 15 → fails.
|
||||
func TestUpwatch_NoExplainerLearnsThenNeedsTenTimesItsP95ForAnHour(t *testing.T) {
|
||||
r := newRig()
|
||||
c := r.src.add("nextcloud", "nextcloud")
|
||||
r.w.Tick(context.Background())
|
||||
for i := 0; i < 90; i++ {
|
||||
r.minute(t, func() { send(c, 50, 5) })
|
||||
}
|
||||
if len(r.ev) != 0 {
|
||||
t.Fatalf("an app with no history and no explainer alarmed: %v", r.ev)
|
||||
}
|
||||
r = newRig()
|
||||
c = r.src.add("nextcloud", "nextcloud")
|
||||
r.w.Baseline = func(stack string, cs []string) (float64, bool) { return 2, true } // p95 = 2 Mbit/s → trigger 20
|
||||
r.w.Tick(context.Background())
|
||||
for i := 0; i < 59; i++ {
|
||||
r.minute(t, func() { send(c, 50, 5) })
|
||||
}
|
||||
if len(r.ev) != 0 {
|
||||
t.Fatalf("alarmed before an hour: %v", r.ev)
|
||||
}
|
||||
r.minute(t, func() { send(c, 50, 5) })
|
||||
if r.count("app_unexplained_upload") != 1 || r.ev[0].d.Explainer != "none" || r.ev[0].d.AppMbps != -1 {
|
||||
t.Fatalf("an hour at 25× its p95: %v", r.ev)
|
||||
}
|
||||
// Below 10× its p95 it is never judged.
|
||||
r = newRig()
|
||||
c = r.src.add("nextcloud", "nextcloud")
|
||||
r.w.Baseline = func(string, []string) (float64, bool) { return 2, true }
|
||||
r.w.Tick(context.Background())
|
||||
for i := 0; i < 120; i++ {
|
||||
r.minute(t, func() { send(c, 15, 5) })
|
||||
}
|
||||
if len(r.ev) != 0 {
|
||||
t.Fatalf("15 Mbit/s (7.5× its p95) alarmed: %v", r.ev)
|
||||
}
|
||||
}
|
||||
|
||||
// The CPU rule: a burner in Plex with no transcode → ONE alarm after 30 minutes; with a transcode, or while the box
|
||||
// itself works on the app (backup, update, restore), none.
|
||||
// RED-PROOF: ignore Transcodes → the transcoding case alarms → fails; ignore Busy → the busy case alarms → fails.
|
||||
func TestUpwatch_CPUBurnerAlarmsTranscodeAndBoxWorkDoNot(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
transcodes int
|
||||
busy bool
|
||||
want int
|
||||
}{{"burner", 0, false, 1}, {"transcoding", 1, false, 0}, {"box busy", 0, true, 0}} {
|
||||
px := &fakeApp{stack: "plex", rep: Report{HasViewerCount: true, HasTranscodeCount: true, Transcodes: tc.transcodes}}
|
||||
r := newRig(px)
|
||||
r.w.Busy = func(string) bool { return tc.busy }
|
||||
c := r.src.add("plex", "plex")
|
||||
r.w.Tick(context.Background())
|
||||
for i := 0; i < 29; i++ {
|
||||
r.minute(t, func() { send(c, 0.1, 380) }) // 4 cores: the trigger is max(150, 200) = 200 %
|
||||
}
|
||||
if len(r.ev) != 0 {
|
||||
t.Fatalf("%s: alarm before 30 minutes: %v", tc.name, r.ev)
|
||||
}
|
||||
r.minute(t, func() { send(c, 0.1, 380) })
|
||||
if got := r.count("app_unexplained_cpu"); got != tc.want {
|
||||
t.Fatalf("%s: %d app_unexplained_cpu, want %d (%v)", tc.name, got, tc.want, r.ev)
|
||||
}
|
||||
if tc.want == 1 {
|
||||
for i := 0; i < 15; i++ {
|
||||
r.minute(t, func() { send(c, 0.1, 5) })
|
||||
}
|
||||
if r.count("app_unexplained_cpu_cleared") != 1 {
|
||||
t.Fatalf("burner stopped 15 min: %v", r.ev)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A container that restarts is a new counter, even when the new run's first reading is LARGER than the old run's last
|
||||
// (a busy first minute after a restart) — that minute has no delta. Infrastructure is never judged.
|
||||
// RED-PROOF: drop the Started comparison → the restart minute computes 1 TiB − old → a huge rate → fails.
|
||||
func TestUpwatch_RestartAndInfrastructureAreNotJudged(t *testing.T) {
|
||||
qb := &fakeApp{stack: "qbittorrent", rep: Report{HasUploadCounter: true}}
|
||||
r := newRig(qb)
|
||||
c := r.src.add("qbittorrent", "qbittorrent")
|
||||
ctl := r.src.add("felhom-controller", "felhom-controller")
|
||||
cf := r.src.add("cloudflared", "cloudflared")
|
||||
r.w.Tick(context.Background())
|
||||
r.minute(t, func() { send(c, 0.1, 1) })
|
||||
r.minute(t, func() { c.TxBytes, c.Started = 1<<40, "t1" }) // restarted, and its new counter is already 1 TiB
|
||||
if got := r.w.apps["qbittorrent"].lastRate; got > 1 {
|
||||
t.Fatalf("the restart minute was judged at %.0f Mbit/s — a delta across two container runs", got)
|
||||
}
|
||||
r.minute(t, func() { send(c, 0.1, 1) })
|
||||
if got := r.w.apps["qbittorrent"].lastRate; got < 0.05 || got > 0.2 {
|
||||
t.Fatalf("the minute after a restart: %.3f Mbit/s, want 0.1 (the new run's own delta)", got)
|
||||
}
|
||||
for i := 0; i < 40; i++ {
|
||||
r.minute(t, func() {
|
||||
send(ctl, 900, 400) // the off-site push and the tunnel: never an app alarm
|
||||
send(cf, 900, 10)
|
||||
})
|
||||
}
|
||||
if len(r.ev) != 0 {
|
||||
t.Fatalf("infrastructure raised %v", r.ev)
|
||||
}
|
||||
if _, judged := r.w.apps["felhom-controller"]; judged {
|
||||
t.Fatal("the controller's own stack was judged as an app")
|
||||
}
|
||||
}
|
||||
|
||||
// An app that does not answer is „no number": its flood still alarms (the hacked app may refuse its own API).
|
||||
// RED-PROOF: treat an error as "explained" → no alarm → fails.
|
||||
func TestUpwatch_AnAppThatDoesNotAnswerIsNotAnExplanation(t *testing.T) {
|
||||
qb := &fakeApp{stack: "qbittorrent", err: errors.New("connection refused")}
|
||||
r := newRig(qb)
|
||||
c := r.src.add("qbittorrent", "qbittorrent")
|
||||
r.w.Tick(context.Background())
|
||||
for i := 0; i < 15; i++ {
|
||||
r.minute(t, func() { send(c, 100, 5) })
|
||||
}
|
||||
if r.count("app_unexplained_upload") != 1 || r.ev[0].d.AppMbps != -1 {
|
||||
t.Fatalf("a silent qBittorrent flooding: %v", r.ev)
|
||||
}
|
||||
}
|
||||
|
||||
// The trigger follows the measured line: 25 % of it, never under the 5 Mbit/s floor.
|
||||
func TestUpwatch_TriggerFollowsTheLine(t *testing.T) {
|
||||
r := newRig()
|
||||
if got := r.w.UploadTrigger(); got != 5 {
|
||||
t.Fatalf("no line: trigger %v, want the 5 Mbit/s floor", got)
|
||||
}
|
||||
r.w.Line = func() (float64, bool) { return 100, true }
|
||||
if got := r.w.UploadTrigger(); got != 25 {
|
||||
t.Fatalf("100 Mbit/s line: trigger %v, want 25", got)
|
||||
}
|
||||
r.w.Line = func() (float64, bool) { return 8, true }
|
||||
if got := r.w.UploadTrigger(); got != 5 {
|
||||
t.Fatalf("8 Mbit/s line: trigger %v, want the floor", got)
|
||||
}
|
||||
}
|
||||
@@ -27,6 +27,7 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/system"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/upwatch"
|
||||
"golang.org/x/crypto/bcrypt"
|
||||
)
|
||||
|
||||
@@ -804,6 +805,13 @@ func (s *Server) appCommonData(w http.ResponseWriter, r *http.Request, slug stri
|
||||
if lost, at := lostFilesNote(found.Deployed, found.AppConfig); len(lost) > 0 {
|
||||
data["LostFiles"], data["LostBootAt"] = lost, at
|
||||
}
|
||||
// The upload watch (internal/upwatch): qBittorrent's limit, while the box owns it — one line the household can act on
|
||||
// in qBittorrent itself. A limit the household set is theirs and is not described here.
|
||||
if found.Meta.Slug == "qbittorrent" && found.Deployed && s.upwatchView != nil {
|
||||
if v := s.upwatchView(); v.QBLimitBps > 0 {
|
||||
data["QBLimitNote"] = s.msgLang(s.langFor(r), "upwatch.qb_limit_note", upwatch.MbpsText(v.QBLimitBps))
|
||||
}
|
||||
}
|
||||
// v0.280.0 (decision 46): the setup gate's card, while the gate stands.
|
||||
data["SetupGateClosed"] = found.Deployed && found.AppConfig != nil && found.AppConfig.SetupGate.Closed()
|
||||
data["SetupGateHasProbe"] = found.Meta.SetupDoneProbe != nil && found.Meta.SetupDoneProbe.URL != ""
|
||||
@@ -2349,6 +2357,12 @@ func (s *Server) settingsBaseData(page, title string) map[string]interface{} {
|
||||
func (s *Server) systemPageData() map[string]interface{} {
|
||||
data := s.settingsBaseData("settings", "Beállítások")
|
||||
data["TitleKey"] = "page.title.settings" // i18n: the Hungarian title above is what hu renders
|
||||
// The upload watch's measured line (a lower bound, learnt from the nightly off-site push); "" = still learning.
|
||||
if s.upwatchView != nil {
|
||||
if v := s.upwatchView(); v.LineKnown {
|
||||
data["UplinkLine"] = upwatch.MbpsText(int64(v.LineMbps * 1e6 / 8))
|
||||
}
|
||||
}
|
||||
data["GitRepoURL"] = s.cfg.Git.RepoURL
|
||||
data["GitSyncInterval"] = s.cfg.Git.SyncInterval
|
||||
data["BackupEnabled"] = s.cfg.Backup.Enabled
|
||||
|
||||
@@ -239,6 +239,7 @@ func i18nCasesA() []i18nCase {
|
||||
d["MemorySupported"], d["MemoryReachable"] = true, true
|
||||
d["MemoryAllocatedMB"], d["MemoryUsageMB"], d["MemoryMinMB"], d["MemoryMaxMB"] = 4096, 2048, 2048, 8192
|
||||
d["PowerOffSupported"] = true // R-998: the power-off card with its button
|
||||
d["UplinkLine"] = "47" // the upload watch's measured line (the other three cases: still learning)
|
||||
return d
|
||||
}},
|
||||
{"settings_system_failed", "settings_system", func() map[string]interface{} {
|
||||
|
||||
@@ -36,9 +36,12 @@ import (
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/system"
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/upwatch"
|
||||
)
|
||||
|
||||
type Server struct {
|
||||
// upwatchView (SetUpwatchView) feeds the two upload-watch lines; nil = not shown.
|
||||
upwatchView func() upwatch.View
|
||||
// R-871: the missed-backup banner's inputs (SetMissedBackupBanner); nil = no banner.
|
||||
nightLedger *nightchain.Ledger
|
||||
metricsRecord *metrics.MetricsStore
|
||||
@@ -515,6 +518,10 @@ func (s *Server) addShellAssetData(data map[string]interface{}) {
|
||||
// so a cache that already held the first build's stylesheet kept serving it — measured through
|
||||
// Cloudflare (`audits/app-page-2026-10-15b/P0/VERDICT.md`). build.sh now refuses a second push of a
|
||||
// version, and this makes the asset URL change even when the version does not.
|
||||
// SetUpwatchView hands the pages the upload watch's state (internal/upwatch): the measured line and the qBittorrent
|
||||
// limit the box set. Nil = neither line is shown.
|
||||
func (s *Server) SetUpwatchView(fn func() upwatch.View) { s.upwatchView = fn }
|
||||
|
||||
func (s *Server) SetBuildIdentity(gitCommit, buildTime string) {
|
||||
s.buildID = gitCommit + "/" + buildTime
|
||||
}
|
||||
|
||||
@@ -79,6 +79,9 @@
|
||||
<span class="form-hint app-press-err" style="display:none;color:var(--red)"></span>
|
||||
</div>
|
||||
{{- end}}
|
||||
{{- if .QBLimitNote}}
|
||||
<div class="alert alert-info" style="margin-top:1rem" data-qb-limit>{{.QBLimitNote}}</div>
|
||||
{{- end}}
|
||||
{{if .Stack.Updating}}
|
||||
<div class="alert alert-info" style="margin-top:1rem" data-update-phase="{{.Stack.UpdatePhase}}">{{updatePhaseText .Stack}}</div>
|
||||
{{else if and .Stack.HoldReason (eq .Stack.HoldKind "unhealthy_stop")}}
|
||||
|
||||
@@ -183,6 +183,10 @@
|
||||
<span class="settings-label">{{T "settings_system.atjaro"}}</span>
|
||||
<span class="settings-value mono">{{if .NetGateway}}{{.NetGateway}}{{else}}— <span style="color:var(--text-3);">{{T "settings_system.nem_allapithato_meg"}}</span>{{end}}</span>
|
||||
</div>
|
||||
<div class="settings-row" data-uplink>
|
||||
<span class="settings-label">{{T "upwatch.line_label"}}</span>
|
||||
<span class="settings-value">{{if .UplinkLine}}{{T "upwatch.line_at_least"}} <span class="mono">{{.UplinkLine}}</span> {{T "upwatch.line_measured"}}{{else}}<span style="color:var(--text-3);">{{T "upwatch.line_learning"}}</span>{{end}}</span>
|
||||
</div>
|
||||
</div>
|
||||
<p class="settings-card-desc" style="margin-top:0.5em;">{{T "settings_system.ezek_az_ertekek_a_helyi"}}</p>
|
||||
</div>
|
||||
|
||||
@@ -303,6 +303,10 @@
|
||||
<span class="settings-label">Átjáró</span>
|
||||
<span class="settings-value mono">— <span style="color:var(--text-3);">nem állapítható meg</span></span>
|
||||
</div>
|
||||
<div class="settings-row" data-uplink>
|
||||
<span class="settings-label">Feltöltési sebesség</span>
|
||||
<span class="settings-value"><span style="color:var(--text-3);">még mérjük (az éjszakai távoli mentésből)</span></span>
|
||||
</div>
|
||||
</div>
|
||||
<p class="settings-card-desc" style="margin-top:0.5em;">Ezek az értékek a helyi hálózattól függenek és változhatnak. Távoli hibaelhárításnál erről az oldalról olvasd fel őket.</p>
|
||||
</div>
|
||||
|
||||
@@ -348,6 +348,10 @@
|
||||
<span class="settings-label">Átjáró</span>
|
||||
<span class="settings-value mono">192.168.1.1</span>
|
||||
</div>
|
||||
<div class="settings-row" data-uplink>
|
||||
<span class="settings-label">Feltöltési sebesség</span>
|
||||
<span class="settings-value">legalább <span class="mono">47</span> Mbit/s (az éjszakai távoli mentésből mérve)</span>
|
||||
</div>
|
||||
</div>
|
||||
<p class="settings-card-desc" style="margin-top:0.5em;">Ezek az értékek a helyi hálózattól függenek és változhatnak. Távoli hibaelhárításnál erről az oldalról olvasd fel őket.</p>
|
||||
</div>
|
||||
|
||||
@@ -260,6 +260,10 @@
|
||||
<span class="settings-label">Átjáró</span>
|
||||
<span class="settings-value mono">— <span style="color:var(--text-3);">nem állapítható meg</span></span>
|
||||
</div>
|
||||
<div class="settings-row" data-uplink>
|
||||
<span class="settings-label">Feltöltési sebesség</span>
|
||||
<span class="settings-value"><span style="color:var(--text-3);">még mérjük (az éjszakai távoli mentésből)</span></span>
|
||||
</div>
|
||||
</div>
|
||||
<p class="settings-card-desc" style="margin-top:0.5em;">Ezek az értékek a helyi hálózattól függenek és változhatnak. Távoli hibaelhárításnál erről az oldalról olvasd fel őket.</p>
|
||||
</div>
|
||||
|
||||
@@ -293,6 +293,10 @@
|
||||
<span class="settings-label">Átjáró</span>
|
||||
<span class="settings-value mono">— <span style="color:var(--text-3);">nem állapítható meg</span></span>
|
||||
</div>
|
||||
<div class="settings-row" data-uplink>
|
||||
<span class="settings-label">Feltöltési sebesség</span>
|
||||
<span class="settings-value"><span style="color:var(--text-3);">még mérjük (az éjszakai távoli mentésből)</span></span>
|
||||
</div>
|
||||
</div>
|
||||
<p class="settings-card-desc" style="margin-top:0.5em;">Ezek az értékek a helyi hálózattól függenek és változhatnak. Távoli hibaelhárításnál erről az oldalról olvasd fel őket.</p>
|
||||
</div>
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"html"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-controller/internal/upwatch"
|
||||
)
|
||||
|
||||
// The upload watch's household line: qBittorrent's page says the limit while the BOX owns it — in the reader's
|
||||
// language — and says nothing when the household owns it (the view then carries 0), when no view is wired, or on
|
||||
// another app's page.
|
||||
// RED-PROOF: drop the `.QBLimitNote` card from app_shell.html → the first case fails; drop the slug check in the
|
||||
// handler → crafty's page carries the line → fails.
|
||||
func TestUpwatch_QBLimitLineOnQBittorrentsPageOnly(t *testing.T) {
|
||||
s, _ := credsHarness(t)
|
||||
sd := filepath.Join(s.cfg.Paths.StacksDir, "qbittorrent")
|
||||
if err := os.MkdirAll(sd, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
os.WriteFile(filepath.Join(sd, ".felhom.yml"), []byte("display_name: qBittorrent\nslug: qbittorrent\n"), 0o644)
|
||||
os.WriteFile(filepath.Join(sd, "docker-compose.yml"), []byte("services: {}\n"), 0o644)
|
||||
os.WriteFile(filepath.Join(sd, "app.yaml"), []byte("deployed: true\n"), 0o644)
|
||||
_ = s.stackMgr.ScanStacks()
|
||||
s.loadTemplates()
|
||||
render := func(app, lang string) string {
|
||||
rr := httptest.NewRecorder()
|
||||
s.appPageHandler(rr, httptest.NewRequest("GET", "/apps/"+app+"?lang="+lang, nil), app, "")
|
||||
return html.UnescapeString(rr.Body.String())
|
||||
}
|
||||
view := upwatch.View{LineMbps: 100, LineKnown: true, QBLimitBps: 6249472}
|
||||
s.SetUpwatchView(func() upwatch.View { return view })
|
||||
hu, en := asciiFold(render("qbittorrent", "hu")), render("qbittorrent", "en")
|
||||
if !strings.Contains(hu, "data-qb-limit") || !strings.Contains(hu, "legfeljebb 50 Mbit/s") || !strings.Contains(hu, "atallithatod") {
|
||||
t.Fatalf("the Hungarian qBittorrent page lacks the limit line")
|
||||
}
|
||||
if !strings.Contains(en, "uploads at most 50 Mbit/s") {
|
||||
t.Fatalf("the English qBittorrent page lacks the limit line")
|
||||
}
|
||||
if strings.Contains(render("crafty", "hu"), "data-qb-limit") {
|
||||
t.Fatal("another app's page carries qBittorrent's limit line")
|
||||
}
|
||||
view.QBLimitBps = 0 // the household owns it
|
||||
if strings.Contains(render("qbittorrent", "hu"), "data-qb-limit") {
|
||||
t.Fatal("a household-owned limit is described as the box's")
|
||||
}
|
||||
s.upwatchView = nil
|
||||
if strings.Contains(render("qbittorrent", "hu"), "data-qb-limit") {
|
||||
t.Fatal("no view wired, yet a line")
|
||||
}
|
||||
}
|
||||
@@ -244,6 +244,12 @@
|
||||
"poweroff.agent_down": "BORN AS A KEY (R-998, v0.310.0) -- „A gép kikapcsolása”: the household sentence for the power-off route's answer. Pinned by TestPowerOff_EveryReasonHasASentence.",
|
||||
"poweroff.agent_outdated": "BORN AS A KEY (R-998, v0.310.0) -- „A gép kikapcsolása”: the household sentence for the power-off route's answer. Pinned by TestPowerOff_HiddenForOldAgent.",
|
||||
"event.app_repaired_after_crash": "BORN AS A KEY (R-1007, v0.311.0) -- the operator's counts after an unclean boot (app_repaired_after_crash). Pinned by TestAppRepairedAfterCrash_KeysInBothLanguages.",
|
||||
"event.app_unexplained_upload": "BORN AS A KEY (upload watch, 2026-10-23 night) -- operator-only event of internal/upwatch (felhom.eu 08 \u00a79). Pinned by TestNotifyUnexplained_TypesSentencesAndDetails.",
|
||||
"event.app_unexplained_upload_noapp": "BORN AS A KEY (upload watch, 2026-10-23 night) -- operator-only event of internal/upwatch (felhom.eu 08 \u00a79). Pinned by TestNotifyUnexplained_TypesSentencesAndDetails.",
|
||||
"event.app_unexplained_upload_cleared": "BORN AS A KEY (upload watch, 2026-10-23 night) -- operator-only event of internal/upwatch (felhom.eu 08 \u00a79). Pinned by TestNotifyUnexplained_TypesSentencesAndDetails.",
|
||||
"event.app_unexplained_cpu": "BORN AS A KEY (upload watch, 2026-10-23 night) -- operator-only event of internal/upwatch (felhom.eu 08 \u00a79). Pinned by TestNotifyUnexplained_TypesSentencesAndDetails.",
|
||||
"event.app_unexplained_cpu_cleared": "BORN AS A KEY (upload watch, 2026-10-23 night) -- operator-only event of internal/upwatch (felhom.eu 08 \u00a79). Pinned by TestNotifyUnexplained_TypesSentencesAndDetails.",
|
||||
"upwatch.qb_limit_note": "BORN AS A KEY (upload watch, 2026-10-23 night) -- the household line on qBittorrent's page while the box owns its upload limit. Pinned by TestUpwatch_QBLimitLineOnQBittorrentsPageOnly.",
|
||||
"event.app_repair_after_crash_skipped": "BORN AS A KEY (R-1007, v0.311.0) -- the same event when the run was skipped, with its reason. Pinned by TestAppRepairedAfterCrash_KeysInBothLanguages.",
|
||||
"err.lost_files.nothing_to_dismiss": "BORN AS A KEY (R-1007, v0.311.0) -- the dismiss route's answer when there is no lost-files note. Pinned by TestAppRepairedAfterCrash_KeysInBothLanguages.",
|
||||
"poweroff.refused.too_soon": "BORN AS A KEY (R-998, v0.310.0) -- „A gép kikapcsolása”: the household sentence for the power-off route's answer. Pinned by TestPowerOff_EveryReasonHasASentence.",
|
||||
|
||||
Reference in New Issue
Block a user