Attended release run done: controller 0.342.0 the global floor, agent 0.164.0 vouched, hub 0.169.0; R-1244 and R-1247 closed (register 193 -> 191); capability F13; STATUS; the run report

2026-10-24 10:37:27 +02:00
parent 01bf4e83a9
commit 6018486fcc
11 changed files with 217 additions and 6 deletions
@@ -16,6 +16,12 @@
 > and holds nothing of its own; this file does hold its own content, namely the standing rulings below.


+> **2026-10-24 (attended release run) — DONE.** Hub 0.168.0 (05:39Z) and 0.169.0 (07:51Z) deployed; agent 0.164.0
+> released, delivered to the three test boxes and vouched by the operator; controller 0.342.0 the global floor (08:29Z,
+> MinAgent 0.158.1). ep0 runs `felhom-iperf3` (tunnel only). First line tests: Tester 1 94.3, demo-hp 94.4, demo-felhom
+> 92.6 Mbit/s; demo-felhom's qBittorrent limited to 46 Mbit/s. D40 in effect (dated check 2026-10-25, R-1243). Opened
+> R-1249 (the start-up VACUUM), R-1250 (a seeding qBittorrent is never tested). Report `REPORT-attended-release-2026-10-24.md`.
+
 > **2026-10-24 07:22 / 07:26 — operator rulings by starting the attended release brief (D40, D41, D42; the line test).**
 > **D40 (R-1243) = stop the Longhorn backup of the hub's volume**; the encrypted 6-hourly copy to ep0 stays the hub's
 > off-site copy; the operator deleted the two old plain Longhorn backups himself (UI); the Longhorn UI cannot take the
@@ -0,0 +1,89 @@
+# REPORT — attended release run 2026-10-24: the hub's tokens locked, the hub out of the open backup, the iperf3 line test, the watch released
+
+Brief: „the hub's tokens locked, the hub out of the open backup, an iperf3 line test via ep0, then the upload watch
+released" (operator rulings 07:22 / 07:26, `CONTEXT.md`). Read first: `05` §16.2, `06` §4.5 / §5, `07`, `08` §9, the
+spike `audits/SPIKE-upload-watch-2026-10-23.md`, the felhom-build-deploy skill. Evidence: `audits/line-test-2026-10-24/`.
+
+## Five lines
+
+1. **Hub tokens:** sealed in hub 0.168.0 and still working (demo-hp is served the same tunnel and API tokens it runs; the
+   template fetch works; the next ep0 copy holds 0 of 8 token values). The live file keeps old bytes in free space — your
+   call: accepted, written down, with the rule „copy hub.db only with VACUUM INTO".
+2. **The hub is out of the open backup:** the hub volume reads `default: disabled`; the folder holds no copy of it (your
+   two deletions worked; nothing else is left). Tonight's job is the last proof (dated check 2026-10-25).
+3. **The line test:** demo-hp measured **94.4 Mbit/s** (Tester 1 94.3, demo-felhom 92.6) — the night backup said „at
+   least 10".
+4. **Live where:** hub 0.169.0 and controller 0.342.0 run on all three boxes (the global floor); agent 0.164.0 on all
+   three (vouched); demo-felhom's qBittorrent is limited to 46 Mbit/s. Only these three boxes report to the hub.
+5. **You still do:** nothing for this run. Two rows wait for a design (below).
+
+## What I asked you, and what you said
+
+- Deploy hub 0.168.0 → yes. Deploy hub 0.169.0 → yes. Global floor 0.342.0 → yes. Agent vouch → done.
+- The live file's old bytes → accept and write down (my first estimate of „2 minutes" was wrong; the disk writes ~1.3
+  MB/s, so a clean-up means an 8–20 minute hub stop).
+- No host has the iperf3 program → the agent speaks iperf3 itself (nothing installed on hosts).
+
+## Versions, digests, CI (by commit)
+
+| Repo | Commit | What | CI |
+|---|---|---|---|
+| felhom.eu | `00031150` | hub 0.168.0 manifest + rulings | 2275 success |
+| felhom.eu | `3363ffa8` | D40: the PVC label | 2276 success |
+| felhom-agent | `f7e1dcb` | the line test | 2277 success |
+| felhom-controller | `ed1011b` | the tested line in the watch | 2278 success |
+| felhom.eu | `1a788f33` | docs, register, ep0 unit, hub `line_test` model | 2279 success |
+| felhom-controller | `42cda07` | the start-line wording | 2283 success |
+| felhom-agent | `08ca678` | CHANGELOG released, REPORT | (docs) |
+| felhom.eu | `01bf4e83` | hub 0.169.0 manifest | (see the final push) |
+| felhom-controller | `bddc477` | CHANGELOG released, REPORT | (docs) |
+
+- **Hub 0.168.0** `sha256:3bfdecaa…` (from `5277d859`), deployed 05:39Z. **Hub 0.169.0** `sha256:fb9b2489…` (from
+  `1a788f33`), deployed 07:51Z; 30-minute watch: only info events, 0 errors.
+- **Agent 0.164.0**: binary `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`, bundle
+  `1d7b54d39d0c1aff682ba0c0a7f91aea0f9407a10d2f5a8bbfdc858d32c13341`, tag `v0.164.0`; signed `agent_update` to the three
+  boxes, self-check 76/76; vouched (hub page + `Artifact manifest set` line).
+- **Controller 0.342.0** `sha256:d002d3b7…` (from `42cda07`); rc.2 on 9202; per-box floors → healthy on all three, box
+  checks and https-only green; global floor 08:29:19Z, MinAgent 0.158.1, read back on the hub log and page.
+- **ep0:** `felhom-iperf3.service` (iperf3 3.18-2+deb13u2, already installed since 2026-07-03), one nftables line;
+  no answer on the internet (`:22` as the control), an answer in the tunnel. CVEs: four fixed in this build; two open
+  „minor" server DoS issues bounded by tunnel-only, TCP-only and systemd caps (`scripts/ep0-iperf3/README.md`).
+
+## Rows
+
+**Before 194, after 191; opened 2 (R-1249, R-1250), closed 5 (R-975, R-1244, R-1245, R-1246, R-1247).** R-1243 →
+VERIFY (dated check 2026-10-25); R-1241 keeps only the homelab half.
+
+## Observations
+
+- The start-up VACUUM after a legacy seal cannot finish on the Longhorn disk; its comment said 2.8 s — FILED: R-1249.
+- A qBittorrent that seeds all day keeps the uplink busy, so that box is never tested and gets no limit — FILED: R-1250.
+- The live hub file's old token bytes — NOT-A-FINDING: accepted by the operator 2026-10-24 (`05` §16.2), every copy that
+  leaves is clean.
+- Two pre-deploy hub snapshots in `/data/snapshots` held plain tokens — NOT-A-FINDING: they roll off (one already pruned
+  at 06:06Z; the other at the 12:00Z snapshot); same volume, same reader as above.
+- The cross-repo wire gate refused the agent's new `line_test` field until the hub modelled it — fixed in the session
+  (hub 0.169.0 `sysfacts.LineTest`, the System page cell).
+- My first ArgoCD sync used a wrong commit id — NOT-A-FINDING: it failed to fetch and changed nothing; redone.
+- 9202 has no agent local API, so its controller cannot read a line test — NOT-A-FINDING: by design (scratch guest);
+  the qBittorrent proof moved to demo-felhom, where the product path runs.
+- demo-hp's first watch minute could not resolve `plex` — NOT-A-FINDING: the known start-up race (2026-10-23).
+- The family's line on demo-felhom's qBittorrent page was not seen live — NOT-A-FINDING: the household's password is not
+  CC's; the render is pinned by a test, and the text with the live value is in the evidence.
+- Old rc controller images (other sessions) sit on 9202 — NOT-A-FINDING: a scratch guest; mine (rc.2) is removed.
+- The hub's database is 530 MB, mostly report history (`host_reports` 305 MB, `reports` 128 MB) — NOT-A-FINDING here:
+  it is why a VACUUM is slow, recorded in R-1249.
+
+## Teardown, three layers
+
+- **Machine:** 9202 — the throwaway qBittorrent removed through the product (0 containers, 0 volumes), back on the
+  0.342.0 release, my rc.2 image removed. demo-hp's host — the probe binary deleted, nothing running.
+- **Host / DooPlex:** the database copies I restored for the checks shredded; the 9202 password copy, the signed job
+  envelopes and the release clone shredded / removed. ep0 — the one planned change stays (`felhom-iperf3`); the backup
+  of the firewall file is `/etc/nftables.conf.bak-20261024-iperf3`.
+- **Hub:** the three per-box floors cleared; the global floor 0.342.0 stays (the release).
+
+## Instruction files, poster
+
+No instruction file edited. System poster: a text-only change (ep0 also runs the tunnel-only line-test server) — the
+facts file and the matching poster text edited in the same commit.
@@ -2,7 +2,7 @@

 **Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**

-**Updated 2026-10-24 (day, attended).** Hub 0.168.0 is live: the last plain tokens are locked, and every copy of the hub database that leaves DooPlex is clean. The hub is out of DooPlex's open backup folder. Each box can now test its upload once a week (demo-hp: 90.7 Mbit/s, not 10); the releases follow on your yes.
+**Updated 2026-10-24 (day, attended, done).** Released on your yeses: hub 0.168.0 (tokens locked) and 0.169.0, agent 0.164.0 (vouched), controller 0.342.0 (the floor). Each box tests its upload weekly (94, 94, 93 Mbit/s); a hacked app that uploads or computes more than it explains alarms only you, and qBittorrent gets half the tested line. The hub is out of DooPlex's open backup folder.

 ## What works
 - **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
@@ -13,6 +13,7 @@
 - After a power cut the box starts by itself, cleans up a half-written backup, and Immich repairs broken photos (map C10, F12).
 - A second drive is offered as the backup target; media apps see every drive; a one-disk box gets its own data area (map D1, D13, D15).
 - The three test boxes answer only over https; the hub's key opens nothing on a box; password guesses are counted (map E12, E13, E17).
+- A hacked app that uploads or computes more than it explains alarms only you; each box tests its upload weekly; qBittorrent gets half (map F13).
 - An urgent security fix reaches the boxes at night after one test on the test box; the box backs up first and undoes a failed update (map B23).
 - The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15).

@@ -51,11 +52,10 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
 2. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef. If not: two dead entries alarm.
 3. D37 and D38 in the table (two small yes/no). If not: Gitea may restart in release hours; no golden in the release copy.
 4. Give the homelab session R-1241, R-1242, the 7-item list at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md`, and an upload alarm on DooPlex's `eno1` (the line is ~100 Mbit/s; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups.
-5. Today (2026-10-24, attended): hub 0.168.0 deployed; D40–D42 ruled and put into effect. Still open: the agent vouch, hub 0.169.0, the controller floor (R-1244). If not: a hacked app stays unseen.
 **The rest**
 - System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the hub-database label already says every 6 hours. If not: the drawing misses one copy.
 - Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12.
 - Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
 - Two DooPlex leftovers: `leftovers/2026-10-14/first-post/` (keep or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). Since 2026-10-14.
 - Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784).
- Later (yours, not urgent): R-1247 (a real upload-speed test), R-1248 (a hard upload cap per app), R-1229 (ArgoCD's sync settings into git), R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
+- Later (yours, not urgent): R-1250 (a seeding qBittorrent is never tested), R-1248 (a hard upload cap per app), R-1229 (ArgoCD's sync settings into git), R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
@@ -223,6 +223,7 @@
 | <a id="f7"></a>[F7](capability-map-detail/F.md#f7) | Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
 | <a id="f8"></a>[F8](capability-map-detail/F.md#f8) | Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | no corpus validation doc | operator infra; no evidence document |
 | <a id="f9"></a>[F9](capability-map-detail/F.md#f9) | Backup-deadline alerting (`expected_backup_missed`) is ANCHORED — absence of signal is UNKNOWN, not failure | hub v0.75.0 | **IMPLEMENTED** | — | `audits/DIAG-backup-missed-2026-07-26.md` (2026-07-26); R-82 |
+| <a id="f13"></a>[F13](capability-map-detail/F.md#f13) | A hacked app is noticed: an app that uploads or computes far more than its own numbers explain alarms the operator only; each box tests its upload line weekly and qBittorrent gets half of it (`08` §9; D41, D42; R-1244, R-1247) | controller v0.342.0, agent v0.164.0, hub v0.169.0, ep0 `felhom-iperf3` | **PROVEN-LIVE** | the alarm path on scratch 9202 (a hidden flood and a CPU burner, 2026-10-23); the line test on the three test boxes; the qBittorrent limit on demo-felhom. **Gaps:** a seeding qBittorrent is never tested (R-1250); an app's API can lie (R-1248) | `audits/SPIKE-upload-watch-2026-10-23.md` §7, `audits/line-test-2026-10-24/` (2026-10-24) |
 | <a id="f12"></a>[F12](capability-map-detail/F.md#f12) | After a power cut, an app rebuilds the files the cut left empty, and the household sees any lost photo on the app page | controller v0.311.0, hub v0.150.0, catalog Immich block | **PROVEN-LIVE** | demo-hp 2026-10-14: a real cut mid-upload, 41 of 41 broken photos repaired, 120 controls untouched; the lost-photo note not exercised live (0 lost; unit- and 9202-tested). **Fleet delivery done 2026-10-14:** golden 0.311.0 + global floor; all three demo boxes on 0.311.0; catalog gate `unclean-boot-images` | `audits/immich-repair-2026-10-14/README.md` (2026-10-14); `09` §3 decision 236; R-1007 |
 | <a id="f11"></a>[F11](capability-map-detail/F.md#f11) | The household turns the whole machine off („A gép kikapcsolása"); the hub holds every silence alarm for 14 days and shows KIKAPCSOLVA | agent v0.155.0, hub v0.149.0, controller v0.310.0 | **PROVEN-LIVE** | proven on the N100 2026-10-13; fleet delivery waits on the floor | `audits/poweroff-2026-10-13/README.md` (2026-10-13); `architecture/14-host-power-off.md`; R-1000, R-1002 |
 | <a id="f10"></a>[F10](capability-map-detail/F.md#f10) | A box that is not always on (off at its backup time) makes the missed night up once when it comes back; the household sees a banner; the operator gets the missed-backup alarm, down or not | controller v0.295.0, hub v0.134.0, agent v0.145.0 | **PARTIAL** | catch-up and banner live; R-872 alarm by tests; suspend not measured | `audits/catchup-2026-10-05/partA/` (2026-10-05); R-872 |
@@ -219,3 +219,25 @@ Tester 1 run controller 0.311.0. An Immich image bump the repair was not proven
 **Evidence:** `audits/immich-repair-2026-10-14/` — spike A on 9202 (the five candidates), the scripts on real damage
 (9202), the live cut on demo-hp 9201 (D/). **Not exercised live:** a lost original (the cut lost none); pinned by unit
 tests and the 9202 run.
+
+## F13 — A hacked app is noticed (the upload and CPU watch, the weekly line test)
+
+*Map row: [F13](../00-capability-map.md#f13) · added 2026-10-24.*
+
+**Scenario:** an app on a household box sends far more than it says it sends, or burns CPU with no transcode to explain
+it (the Plex2Shell shape). Once a minute the controller reads every app's exact counters and asks the app (qBittorrent,
+Transmission, Plex); not explained for 15 / 30 minutes → one operator-only alarm, one clear when it stops. Once a week
+the host's agent measures the upload line with a 10-second iperf3 upload to ep0 inside the tunnel; qBittorrent's default
+upload limit is half of the tested line, never over the household's own limit.
+
+**Components:** controller v0.342.0 (`internal/upwatch`, `agentapi.LineTest`), agent v0.164.0 (`internal/linetest`),
+hub v0.169.0 (four operator-only events; the System page's „Upload line"), ep0 `felhom-iperf3.service`.
+
+**Status:** **PROVEN-LIVE** — the alarm path on scratch 9202 with `0.342.0-rc.1` (2026-10-23: a 50 Mbit/s flood hidden
+behind a real seed → one alarm after 15 min, one clear; a CPU burner in Plex → the same; a real seed → none); the line test
+on the three test boxes (2026-10-24: 94.3 / 94.4 / 92.6 Mbit/s; „server busy" retried); the qBittorrent limit on
+demo-felhom (5 789 696 B/s = 50 % of 92.6 Mbit/s, three channels agree).
+
+**Evidence:** `audits/SPIKE-upload-watch-2026-10-23.md` §7; `audits/line-test-2026-10-24/`. **Not exercised live:** an
+alarm reaching the operator's mailbox from a real box (the 9202 proof had no hub); the family's line on qBittorrent's
+page on demo-felhom (pinned by a render test). **Gaps:** R-1250, R-1248.
@@ -0,0 +1,32 @@
+# controller 0.342.0 on the three test boxes through the normal update path (per-customer floors 08:05:59Z), checks ~08:1xZ
+### root@192.168.0.154
+containers=10 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 2 minutes (healthy)
+errors=0
+2026/10/24 08:06:13 [INFO] [upwatch] the host's weekly line test: 94.3 Mbit/s (measured 2026-10-24T08:00:58Z)
+2026/10/24 08:06:13 [INFO] [upwatch] upload and CPU watch started (line 94 Mbit/s, qBittorrent share 50 %)
+### demo-hp
+containers=24 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 2 minutes (healthy)
+errors=0
+2026/10/24 08:06:12 service.go:70: [INFO] [upwatch] the host's weekly line test: 94.4 Mbit/s (measured 2026-10-24T08:00:33Z)
+2026/10/24 08:06:12 upwatch_wiring.go:123: [INFO] [upwatch] upload and CPU watch started (line 94 Mbit/s, qBittorrent share 50 %)
+2026/10/24 08:06:13 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
+2026/10/24 08:06:20 updater.go:104: [DEBUG] [selfupdate] SetFloor: floor "" → "0.342.0"
+### felhom-pve
+containers=9 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 2 minutes (healthy)
+errors=0
+2026/10/24 08:05:55 [INFO] [upwatch] the host's weekly line test: 92.6 Mbit/s (measured 2026-10-24T08:02:45Z)
+2026/10/24 08:05:55 [INFO] [upwatch] upload and CPU watch started (line 93 Mbit/s, qBittorrent share 50 %)
+
+# demo-felhom (the lived-in box; its qBittorrent was unlimited before) — the D41 limit, three channels
+controller log : 2026/10/24 08:20:56 [INFO] [upwatch] qBittorrent upload limit set to 5789696 B/s (50 % of the tested line, 92.6 Mbit/s)
+qBittorrent    : Session\GlobalUPSpeedLimit=5654   (KiB/s = 5 789 696 B/s, persisted in its own config)
+box record     : settings.json upwatch.qb_limit {"box_set_bytes": 5789696, "line_mbps": 92.64214849161526, "set_at": "2026-10-24T08:20:55Z"}
+the family's line (rendered from the bundle with this value; the live page needs the household's password, which CC
+does not hold — the render is pinned by TestUpwatch_QBLimitLineOnQBittorrentsPageOnly):
+  HU: „A qBittorrent feltöltése legfeljebb 46 Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
+  EN: "qBittorrent uploads at most 46 Mbit/s, so the family internet stays fast. You can change this in qBittorrent."
+
+# https-only: rc 0, 28 public names http -> https on the three boxes, negative control convicted (https-only.txt)
+
+# Hub 0.169.0, 30-minute watch 07:51-08:21Z: only controller_updated / controller_started (info) from the three boxes;
+# 0 [ERROR] lines; System page „Upload line": 92.6 / 94.4 / 94.3 Mbit/s (one other host „—": no 0.164.0 agent or tunnel)
@@ -0,0 +1,22 @@
+# The agents' first real weekly line tests, 2026-10-24 ~08:00Z (agent 0.164.0; never measured → due at once inside the window)
+== demo-hp
+time=2026-10-24T10:00:43.505+02:00 level=INFO msg="linetest: run 1: 94.4 Mbit/s" server=10.77.0.1:5201 server_bytes=118620160 server_seconds=10.054439
+time=2026-10-24T10:01:23.864+02:00 level=INFO msg="linetest: run 2: 93.1 Mbit/s" server=10.77.0.1:5201 server_bytes=117047296 server_seconds=10.058192
+time=2026-10-24T10:02:04.224+02:00 level=INFO msg="linetest: run 3: 93.9 Mbit/s" server=10.77.0.1:5201 server_bytes=117964800 server_seconds=10.053506
+time=2026-10-24T10:02:04.224+02:00 level=INFO msg="linetest: upload line measured: 94.4 Mbit/s (best of 3)" server=10.77.0.1:5201 runs=3 duration_s=91.091932894
+== felhom-pve
+time=2026-10-24T10:00:45.184+02:00 level=INFO msg="linetest: the test server is busy — trying again later today" server=10.77.0.1:5201 tries_today=1
+time=2026-10-24T10:01:45.304+02:00 level=INFO msg="linetest: the test server is busy — trying again later today" server=10.77.0.1:5201 tries_today=2
+time=2026-10-24T10:02:55.643+02:00 level=INFO msg="linetest: run 1: 90.4 Mbit/s" server=10.77.0.1:5201 server_bytes=113639424 server_seconds=10.051118
+time=2026-10-24T10:03:36.021+02:00 level=INFO msg="linetest: run 2: 91.9 Mbit/s" server=10.77.0.1:5201 server_bytes=115605504 server_seconds=10.067277
+time=2026-10-24T10:04:16.363+02:00 level=INFO msg="linetest: run 3: 92.6 Mbit/s" server=10.77.0.1:5201 server_bytes=116391936 server_seconds=10.050884
+time=2026-10-24T10:04:16.363+02:00 level=INFO msg="linetest: upload line measured: 92.6 Mbit/s (best of 3)" server=10.77.0.1:5201 runs=3 duration_s=91.057939463
+== root@192.168.0.154
+time=2026-10-24T10:01:09.231+02:00 level=INFO msg="linetest: run 1: 90.5 Mbit/s" server=10.77.0.1:5201 server_bytes=113770496 server_seconds=10.054948
+time=2026-10-24T10:01:49.595+02:00 level=INFO msg="linetest: run 2: 88.5 Mbit/s" server=10.77.0.1:5201 server_bytes=111280128 server_seconds=10.058779
+time=2026-10-24T10:02:29.941+02:00 level=INFO msg="linetest: run 3: 94.3 Mbit/s" server=10.77.0.1:5201 server_bytes=118489088 server_seconds=10.053345
+time=2026-10-24T10:02:29.941+02:00 level=INFO msg="linetest: upload line measured: 94.3 Mbit/s (best of 3)" server=10.77.0.1:5201 runs=3 duration_s=91.063997482
+
+[exited with code 0]
+
+# ep0 after: established on :5201 = 0; felhom-iperf3 active, NRestarts=0, MemoryPeak=6352896
@@ -0,0 +1,33 @@
+control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
+== tester-1 (enkicsifelhom.hu): 6 public name(s)
+   OK   control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
+   OK   http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
+   OK   http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
+   OK   http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
+   OK   http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
+   OK   http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
+== demo-hp (enkisfelhom.hu): 14 public name(s)
+   OK   http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
+   OK   http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
+   OK   http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
+   OK   http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
+   OK   http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
+   OK   http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
+   OK   http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
+   OK   http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
+   OK   http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
+   OK   http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
+   OK   http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
+   OK   http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
+   OK   http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
+   OK   http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
+== demo-felhom (demo-felhom.eu): 8 public name(s)
+   OK   http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
+   OK   http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
+   OK   http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
+   OK   http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
+   OK   http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
+   OK   http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
+   OK   http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
+   OK   http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
+== every public name redirects http -> https
@@ -26,13 +26,15 @@

 ---

-## 2026-10-24 — the hub's tokens deployed; D40–D42 ruled; the line test (full text of the rows: `git show 3363ffa8:documentation/backlog/OPEN-ITEMS.md`)
+## 2026-10-24 — the hub's tokens deployed; D40–D42 ruled; the line test; the watch released (full text of the rows: `git show 3363ffa8:documentation/backlog/OPEN-ITEMS.md`; R-1244 and R-1247: `git show 1a788f33:documentation/backlog/OPEN-ITEMS.md`)

 | Row | What | Closed | Evidence |
 |---|---|---|---|
 | **R-975** | **The hub database kept every customer's tunnel token, Cloudflare API token and git token unsealed.** (P3) | CLOSED 2026-10-24 — hub **0.168.0** deployed on the operator's yes (image sha256:3bfdecaa…; ArgoCD sync of the hub Deployment only, `00031150`): the three `config_json` leaves are sealed; demo-hp's `GET /box-config` serves the same plain tokens its box runs (fingerprints equal, 0 `enc:v1:`); the ep0 copy of 06:30:57Z holds **0 of 8** token values (positive control: the pre-deploy copy, 8 of 8). The live file's old bytes in freed space (the start-up VACUUM did not finish, R-1249) are **accepted by the operator** (2026-10-24: only root on DooPlex can read the volume, and it holds the key; every copy that leaves is clean; rule in `05` §16.2) | `audits/line-test-2026-10-24/hub-0168-seal-check.txt` |
 | **R-1245** | **DECIDE: how large is qBittorrent's default upload limit?** (P3) | CLOSED 2026-10-24 — **D41 = 50 %** (operator ruling 07:22), of the host's TESTED line (the weekly iperf3 test, R-1247 — never the push floor); built in controller 0.342.0 (`ed1011b`); released with R-1244 | `08` §9.5 |
 | **R-1246** | **DECIDE: who is told when the box limits or flags something?** (P3) | CLOSED 2026-10-24 — **D42** (operator ruling 07:22): the family only about the qBittorrent limit (one line on its page); `app_unexplained_upload` / `app_unexplained_cpu` are operator-only (hub 0.169.0); released with R-1244 | `08` §9.4 |
+| **R-1244** | **The upload and CPU watch was built and not released.** (P2) | CLOSED 2026-10-24 — released on the operator's yes: hub **0.169.0** (the four operator-only events + the System page's „Upload line"; sha256:fb9b2489…, deployed 07:51Z, a 30-minute watch with no alarm) and controller **0.342.0** (sha256:d002d3b7…, `42cda07`; per-box floors → Tester 1, demo-hp, demo-felhom healthy on it, box checks + https-only green; then the global floor 0.342.0, MinAgent 0.158.1, 08:29:19Z, read back on the hub log and page). demo-felhom's qBittorrent got 50 % of its tested line (5 789 696 B/s ≈ 46 Mbit/s; the log, qBittorrent's own config and the box's record agree). Open beside it: R-1250 (a seeding qBittorrent is never tested), R-1248 (an app's API can lie) | `audits/line-test-2026-10-24/controller-0342-boxes.txt` |
+| **R-1247** | **The box's upload line was only a lower bound, learnt from its own off-site push.** (P4) | CLOSED 2026-10-24 — the weekly iperf3 line test (operator ruling 07:26): ep0's tunnel-only `felhom-iperf3` (no answer on the internet, `:22` control open), agent **0.164.0** (`internal/linetest`, its own iperf3 client; vouched by the operator) and the controller reading it. The three boxes' first tests: Tester 1 **94.3**, demo-hp **94.4**, demo-felhom **92.6** Mbit/s (best of 3, 10.05 s per run; demo-felhom met „server busy" twice and tried again the same day); the push floor said „at least 10" | `audits/line-test-2026-10-24/first-tests-three-boxes.txt`, `ep0-install.txt` |

 ## 2026-10-23 day (later) — D39 = B: the security-fix lane

@@ -269,7 +269,6 @@ stopping line that lies.
 |---|---|---|---|---|---|---|---|
 | **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
 | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
-| **R-1244** | Monitoring & notifications | P2 | **The upload and CPU watch is built and not released: until it is, a hacked app on a household box can send hundreds of GB or mine for weeks and nobody is told.** Operator ruling 2026-10-23 19:22; built 2026-10-23 night — controller v0.342.0 (`internal/upwatch`) + hub v0.169.0 (the four operator-only events), both committed, NOT released; proven on scratch 9202 with `0.342.0-rc.1` (`audits/SPIKE-upload-watch-2026-10-23.md` §7; `08` §9 [DESIGN — proposed]). | WAITING-ON-OPERATOR | the operator's yes; R-1245 and R-1246 | Operator: yes → build hub 0.169.0 + deploy, then the controller through the normal release path (Tester 1, demo-hp, demo-felhom, box walk) | operator |
 | **R-1250** | Monitoring & notifications | P3 | **A qBittorrent that seeds all day is never line-tested, so it never gets the limit that exists for it.** The weekly test (R-1247) starts only after 10 minutes in which the host's uplink sent < 2 Mbit/s each minute; a seed at the line's speed keeps it busy through every window, so no test, no tested line, and no qBittorrent limit (D41 uses the tested line only). Found 2026-10-24 by reading the rule against the case it serves; not yet seen on a box (no test box seeds). | OPEN | — | Design: on a box with qBittorrent and no tested line, the controller asks qBittorrent to pause uploads (its alternative speed limit) for the test window the agent announces, then restores it; or the watch reports „line unknown, a seeding qBittorrent" to the operator. Pick, red-proof, prove on 9202 | CC |
 | **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Design note (2026-10-21, after R-1192's diagnosis): the freeze's signature is cheap to see from the HOST side — the guest cgroup's memory PSI „full" above ~50 % for a minute, or its `memory.events max` climbing, while the host still reports; the agent could raise `guest_memory_thrash` (operator mail in its own words, not „no report for 45 m") and the household banner „A dobozod most nagyon lassú…". With the swap fix (R-1192) the measured thrash ends within a minute, so this alarm should be built only if a freeze is seen again after the agent release; until then it stays a design. Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC |
 | **R-1202** | Monitoring & notifications | P3 | **A box whose clock is behind blames the hub and stops an app without telling anyone.** Rehearsal 2026-10-20 B6 #8: RTC set 2 years back (the OS raised it to 2026-04-13): the dashboard shows only „Hub nem elérhető" while the log says „x509: certificate … not yet valid: current time 2026-04-13…"; Paperless crash-looped and the box stopped it with a restore hold; the `app_stopped_unhealthy` event failed (hub unreachable), so neither the household nor the operator heard. After the real time came back the app stayed stopped until a Start (clear sentence, 1 min). | **OPEN** | — | say „the box's clock is wrong (it says <date>)" when TLS fails with not-yet-valid/expired against a hub that answers; keep the stop event queued until the hub is reachable | CC |
@@ -288,7 +287,6 @@ stopping line that lies.
 | **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC |
 | **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
 | **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
-| **R-1247** | Monitoring & notifications | P4 | **The box's upload line is only a lower bound, learnt from its own off-site push; most nights the push is too small to say anything.** Measured: at a one-minute sample the push exceeded 2 Mbit/s on 3 of 81 box-nights; a one-second sample during the push is now built (a night counts when its best 10 s moved ≥ 10 MB). An ACTIVE test (a timed upload to the box's own Storage Box sub-account, or to ep0) would give the real line — not built: a design for the test file beside the restic repository, a quiet hour outside 02:30–05:30, ~125 MB per test at 100 Mbit/s. `audits/SPIKE-upload-watch-2026-10-23.md` §2, §3. | **READY** — **2026-10-24 (operator ruling 07:26): the weekly iperf3 line test, built; release in progress.** ep0 `felhom-iperf3.service` live (tunnel only, proven closed on the internet); agent `internal/linetest` (`f7e1dcb`, unreleased, next 0.164.0); controller reads it (`ed1011b`, in 0.342.0). demo-hp measured **90.7 Mbit/s** (the push floor said 10). Closes when the agent and the controller are released and each test box reports a `line_test` value. OPEN | an operator decision (data cost, where the test writes) | Decide whether an active test is wanted; if yes, CC designs it | operator |


 ## Hub & operator — 12 rows (P3 5, P4 7)
@@ -5,6 +5,12 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S

 **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**

+## Moved from STATUS on 2026-10-24 (day, done)
+
+**Updated 2026-10-24 (day, attended).** Hub 0.168.0 is live: the last plain tokens are locked, and every copy of the hub database that leaves DooPlex is clean. The hub is out of DooPlex's open backup folder. Each box can now test its upload once a week (demo-hp: 90.7 Mbit/s, not 10); the releases follow on your yes.
+
+5. Today (2026-10-24, attended): hub 0.168.0 deployed; D40–D42 ruled and put into effect. Still open: the agent vouch, hub 0.169.0, the controller floor (R-1244). If not: a hacked app stays unseen.
+
 ## Moved from STATUS on 2026-10-24 (day)

 **Updated 2026-10-23 (night).** The hub's last unlocked tokens are now locked in a new hub version — built and tested, you deploy it on Monday. The new upload watch is built and tested on the scratch box, not released: it finds an app that uploads or computes far more than the app itself explains (a hacked app) and tells only you; qBittorrent gets an upload limit.