Attended release run done: controller 0.342.0 the global floor, agent 0.164.0 vouched, hub 0.169.0; R-1244 and R-1247 closed (register 193 -> 191); capability F13; STATUS; the run report

documentation/backlog/CLOSED-ITEMS.md

@@ -26,13 +26,15 @@
 
 ---
 
-## 2026-10-24 — the hub's tokens deployed; D40–D42 ruled; the line test (full text of the rows: `git show 3363ffa8:documentation/backlog/OPEN-ITEMS.md`)
+## 2026-10-24 — the hub's tokens deployed; D40–D42 ruled; the line test; the watch released (full text of the rows: `git show 3363ffa8:documentation/backlog/OPEN-ITEMS.md`; R-1244 and R-1247: `git show 1a788f33:documentation/backlog/OPEN-ITEMS.md`)
 
 | Row | What | Closed | Evidence |
 |---|---|---|---|
 | **R-975** | **The hub database kept every customer's tunnel token, Cloudflare API token and git token unsealed.** (P3) | CLOSED 2026-10-24 — hub **0.168.0** deployed on the operator's yes (image sha256:3bfdecaa…; ArgoCD sync of the hub Deployment only, `00031150`): the three `config_json` leaves are sealed; demo-hp's `GET /box-config` serves the same plain tokens its box runs (fingerprints equal, 0 `enc:v1:`); the ep0 copy of 06:30:57Z holds **0 of 8** token values (positive control: the pre-deploy copy, 8 of 8). The live file's old bytes in freed space (the start-up VACUUM did not finish, R-1249) are **accepted by the operator** (2026-10-24: only root on DooPlex can read the volume, and it holds the key; every copy that leaves is clean; rule in `05` §16.2) | `audits/line-test-2026-10-24/hub-0168-seal-check.txt` |
 | **R-1245** | **DECIDE: how large is qBittorrent's default upload limit?** (P3) | CLOSED 2026-10-24 — **D41 = 50 %** (operator ruling 07:22), of the host's TESTED line (the weekly iperf3 test, R-1247 — never the push floor); built in controller 0.342.0 (`ed1011b`); released with R-1244 | `08` §9.5 |
 | **R-1246** | **DECIDE: who is told when the box limits or flags something?** (P3) | CLOSED 2026-10-24 — **D42** (operator ruling 07:22): the family only about the qBittorrent limit (one line on its page); `app_unexplained_upload` / `app_unexplained_cpu` are operator-only (hub 0.169.0); released with R-1244 | `08` §9.4 |
+| **R-1244** | **The upload and CPU watch was built and not released.** (P2) | CLOSED 2026-10-24 — released on the operator's yes: hub **0.169.0** (the four operator-only events + the System page's „Upload line"; sha256:fb9b2489…, deployed 07:51Z, a 30-minute watch with no alarm) and controller **0.342.0** (sha256:d002d3b7…, `42cda07`; per-box floors → Tester 1, demo-hp, demo-felhom healthy on it, box checks + https-only green; then the global floor 0.342.0, MinAgent 0.158.1, 08:29:19Z, read back on the hub log and page). demo-felhom's qBittorrent got 50 % of its tested line (5 789 696 B/s ≈ 46 Mbit/s; the log, qBittorrent's own config and the box's record agree). Open beside it: R-1250 (a seeding qBittorrent is never tested), R-1248 (an app's API can lie) | `audits/line-test-2026-10-24/controller-0342-boxes.txt` |
+| **R-1247** | **The box's upload line was only a lower bound, learnt from its own off-site push.** (P4) | CLOSED 2026-10-24 — the weekly iperf3 line test (operator ruling 07:26): ep0's tunnel-only `felhom-iperf3` (no answer on the internet, `:22` control open), agent **0.164.0** (`internal/linetest`, its own iperf3 client; vouched by the operator) and the controller reading it. The three boxes' first tests: Tester 1 **94.3**, demo-hp **94.4**, demo-felhom **92.6** Mbit/s (best of 3, 10.05 s per run; demo-felhom met „server busy" twice and tried again the same day); the push floor said „at least 10" | `audits/line-test-2026-10-24/first-tests-three-boxes.txt`, `ep0-install.txt` |
 
 ## 2026-10-23 day (later) — D39 = B: the security-fix lane
 

documentation/backlog/OPEN-ITEMS.md

@@ -269,7 +269,6 @@ stopping line that lies.
 |---|---|---|---|---|---|---|---|
 | **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
 | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING** — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a passing `oom_check` on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (`felhom-agent` `acccb66`, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: `OOMKilled=true`, exit 137, the `oom` event — seen only with an events window ending after the run (the wrapper waits 2 s). `audits/readback-2026-10-07/`F/. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
-| **R-1244** | Monitoring & notifications | P2 | **The upload and CPU watch is built and not released: until it is, a hacked app on a household box can send hundreds of GB or mine for weeks and nobody is told.** Operator ruling 2026-10-23 19:22; built 2026-10-23 night — controller v0.342.0 (`internal/upwatch`) + hub v0.169.0 (the four operator-only events), both committed, NOT released; proven on scratch 9202 with `0.342.0-rc.1` (`audits/SPIKE-upload-watch-2026-10-23.md` §7; `08` §9 [DESIGN — proposed]). | WAITING-ON-OPERATOR | the operator's yes; R-1245 and R-1246 | Operator: yes → build hub 0.169.0 + deploy, then the controller through the normal release path (Tester 1, demo-hp, demo-felhom, box walk) | operator |
 | **R-1250** | Monitoring & notifications | P3 | **A qBittorrent that seeds all day is never line-tested, so it never gets the limit that exists for it.** The weekly test (R-1247) starts only after 10 minutes in which the host's uplink sent < 2 Mbit/s each minute; a seed at the line's speed keeps it busy through every window, so no test, no tested line, and no qBittorrent limit (D41 uses the tested line only). Found 2026-10-24 by reading the rule against the case it serves; not yet seen on a box (no test box seeds). | OPEN | — | Design: on a box with qBittorrent and no tested line, the controller asks qBittorrent to pause uploads (its alternative speed limit) for the test window the agent announces, then restores it; or the watch reports „line unknown, a seeding qBittorrent" to the operator. Pick, red-proof, prove on 9202 | CC |
 | **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Design note (2026-10-21, after R-1192's diagnosis): the freeze's signature is cheap to see from the HOST side — the guest cgroup's memory PSI „full" above ~50 % for a minute, or its `memory.events max` climbing, while the host still reports; the agent could raise `guest_memory_thrash` (operator mail in its own words, not „no report for 45 m") and the household banner „A dobozod most nagyon lassú…". With the swap fix (R-1192) the measured thrash ends within a minute, so this alarm should be built only if a freeze is seen again after the agent release; until then it stays a design. Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC |
 | **R-1202** | Monitoring & notifications | P3 | **A box whose clock is behind blames the hub and stops an app without telling anyone.** Rehearsal 2026-10-20 B6 #8: RTC set 2 years back (the OS raised it to 2026-04-13): the dashboard shows only „Hub nem elérhető" while the log says „x509: certificate … not yet valid: current time 2026-04-13…"; Paperless crash-looped and the box stopped it with a restore hold; the `app_stopped_unhealthy` event failed (hub unreachable), so neither the household nor the operator heard. After the real time came back the app stayed stopped until a Start (clear sentence, 1 min). | **OPEN** | — | say „the box's clock is wrong (it says <date>)" when TLS fails with not-yet-valid/expired against a hub that answers; keep the stop event queued until the hub is reachable | CC |
@@ -288,7 +287,6 @@ stopping line that lies.
 | **R-1223** | Monitoring & notifications | P3 | **After a reboot, a box whose tunnel ep0 refuses is invisible: the future stamp is gone, so neither the D30 re-key nor the hub's alarm sees it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, A8 + H1): the kernel forgets `latest-handshake` when the interface restarts (0 = never); the agent then sends neither `last_handshake_age_s` nor `handshake_future_s`, and `checkWGTunnel` (`hub/internal/api/host_tunnel.go`) does not judge a registered, active tunnel with no handshake at all. Shape: the clock ran ahead, was corrected, the box rebooted (some households switch off at night), ep0 still holds the future stamp and starts nothing. | OPEN | — | Agent: report `tunnel_up_s` (how long the unit has been active) with no handshake; hub: a registered, active tunnel with no handshake for > 30 min while reporting → `wg_tunnel_stale`; the agent re-keys the same way after 30 min with no handshake. Tests both sides; rides an agent release + a hub deploy | CC |
 | **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
 | **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
-| **R-1247** | Monitoring & notifications | P4 | **The box's upload line is only a lower bound, learnt from its own off-site push; most nights the push is too small to say anything.** Measured: at a one-minute sample the push exceeded 2 Mbit/s on 3 of 81 box-nights; a one-second sample during the push is now built (a night counts when its best 10 s moved ≥ 10 MB). An ACTIVE test (a timed upload to the box's own Storage Box sub-account, or to ep0) would give the real line — not built: a design for the test file beside the restic repository, a quiet hour outside 02:30–05:30, ~125 MB per test at 100 Mbit/s. `audits/SPIKE-upload-watch-2026-10-23.md` §2, §3. | **READY** — **2026-10-24 (operator ruling 07:26): the weekly iperf3 line test, built; release in progress.** ep0 `felhom-iperf3.service` live (tunnel only, proven closed on the internet); agent `internal/linetest` (`f7e1dcb`, unreleased, next 0.164.0); controller reads it (`ed1011b`, in 0.342.0). demo-hp measured **90.7 Mbit/s** (the push floor said 10). Closes when the agent and the controller are released and each test box reports a `line_test` value. OPEN | an operator decision (data cost, where the test writes) | Decide whether an active test is wanted; if yes, CC designs it | operator |
 
 
 ## Hub & operator — 12 rows (P3 5, P4 7)

documentation/backlog/STATUS-HISTORY.md

@@ -5,6 +5,12 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
 
 **Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
 
+## Moved from STATUS on 2026-10-24 (day, done)
+
+**Updated 2026-10-24 (day, attended).** Hub 0.168.0 is live: the last plain tokens are locked, and every copy of the hub database that leaves DooPlex is clean. The hub is out of DooPlex's open backup folder. Each box can now test its upload once a week (demo-hp: 90.7 Mbit/s, not 10); the releases follow on your yes.
+
+5. Today (2026-10-24, attended): hub 0.168.0 deployed; D40–D42 ruled and put into effect. Still open: the agent vouch, hub 0.169.0, the controller floor (R-1244). If not: a hacked app stays unseen.
+
 ## Moved from STATUS on 2026-10-24 (day)
 
 **Updated 2026-10-23 (night).** The hub's last unlocked tokens are now locked in a new hub version — built and tested, you deploy it on Monday. The new upload watch is built and tested on the scratch box, not released: it finds an app that uploads or computes far more than the app itself explains (a hacked app) and tells only you; qBittorrent gets an upload limit.