Security app updates, Part 1: the triage of 38 flagged pins (TRIAGE.md, per-app notes, PICKS.md); R-1306 triaged; R-1310 (watch classifier) and R-1311 (Termix X-Forwarded-For) filed
gates / gates (push) Successful in 2m0s
gates / gates (push) Successful in 2m0s
This commit is contained in:
1 parent
c6248dbfc5
commit
ba06b873b9
36 files changed
+903
-3
No files matched your search
@@ -128,7 +128,7 @@ stopping line that lies.
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-1306** | Apps & catalog | P2 | **The first security-watch run: 27 catalog pins carry published notices that affect the version we pin** (89 notices; the list with our version and the fixed versions: `audits/security-watch-2026-10-26/ADVISORIES-2026-10-26.md`). Several fix only in a NEW major (Jellyfin 10.11.11 → 12.0; Gokapi 1.9.6 → 2.x; Seerr 2.7.3 → 3.x); some name no fix yet (SparkyFitness); some are release notes naming a CVE (Grafana, BookStack, PrivateBin …). The seed mailed none of them (R-1303). | OPEN — filed 2026-10-26 | the operator's order of triage | worst first through the security-fix lane (B23): read each notice against how the app is deployed, then a ladder step or a re-pin | CC |
|
||||
| **R-1306** | Apps & catalog | P2 | **The first security-watch run: 27 catalog pins carry published notices that affect the version we pin** (89 notices; the list with our version and the fixed versions: `audits/security-watch-2026-10-26/ADVISORIES-2026-10-26.md`). Several fix only in a NEW major (Jellyfin 10.11.11 → 12.0; Gokapi 1.9.6 → 2.x; Seerr 2.7.3 → 3.x); some name no fix yet (SparkyFitness); some are release notes naming a CVE (Grafana, BookStack, PrivateBin …). The seed mailed none of them (R-1303). | OPEN — filed 2026-10-26; **TRIAGED 2026-10-27** (with the key: 35 app pins in 32 apps): 15 apps go by the lane, Wishlist + BookStack as normal steps (no notice id), 11 need a big step, 5 are NOT-A-FINDING — `audits/security-app-updates-2026-10-27/TRIAGE.md` | the operator's order of triage | worst first through the security-fix lane (B23): read each notice against how the app is deployed, then a ladder step or a re-pin | CC |
|
||||
| **R-1091** | Apps & catalog | P3 | **Catalog copy that is wrong or stale at the pinned tag, found writing the guides — each needs an operator word or a live check, so none was changed tonight.** From source at each app's tag (evidence per app in `audits/app-guides-2026-10-15/B/`): (1) and (2) **DONE 2026-10-16** (operator decision D12: reword): calcom's use case now names Apple Calendar or a CalDAV calendar instead of Google/Outlook, and bentopdf's tagline and use case say the browser does the work and the file stays on the device (catalog CHANGELOG 2026-10-16); (3) **sabnzbd**, **nzbget** use_cases say Radarr/Sonarr/Jellyseerr hand them jobs — the box wires only qBittorrent and Transmission (`mediawire.go`), and a hand link needs a path mapping; (4) **onlyoffice** first_steps 3–4 tell the household to wire Nextcloud by hand — the integration toggle does it (`internal/integrations/onlyoffice_nextcloud.go`); (5) **glance**, **homepage** first_steps send the household into config files in a Docker volume no household tool reaches; (6) **lubelogger** starts with only the „Dashboard" tab and in miles (shipped `appsettings.json`), so first step 3 cannot be done as written; (7) **home-assistant** first_steps promise auto-discovered devices — the app runs on a Docker bridge, where mDNS/SSDP were measured deaf (spike R-6); (8) **homebox** and **gitea** `add_people` name menus that moved or read differently at the tag; **bookstack** `default_creds` still says `admin@admin.com / password`, which after_install replaces; **dawarich** names the API key's page „Settings" (source suggests the account page); **code-server** says „install your tools" with no sudo; **komga** names the discontinued Tachiyomi; **seerr** `docs_url` points at Overseerr; **wanderer**'s comment still says it cannot close sign-up (R-714 is fixed in the same file); (9) **nzbget** and **grafana** deploy-field descriptions say the password can be changed later — nzbget re-applies its env password at every start and grafana ignores the field after first start (R-1086 is the qBittorrent/Transmission side). | OPEN — filed 2026-10-15 | — | Live-check (5)–(7) and (9) on 9202; fix (3), (4), (8) as copy with the freeze re-admitted; re-run the guide label check for any app whose copy moves | CC |
|
||||
| **R-1308** | Apps & catalog | P3 | **14 catalog pins use a floating tag** (`redis:7-alpine`, `postgres:16-…`, `mariadb:12.3`, `nginx:…`): there is no exact version to match a notice's range against, so the security watch (R-1303) lists them FLOATING and matches nothing. `audits/security-watch-2026-10-26/ci-seed.log` (FLOATING) | OPEN — filed 2026-10-26 | — | pin the sidecars to exact versions where the engine allows (the image-pins gate today accepts a major tag) | CC |
|
||||
| **R-1092** | Apps & catalog | P4 | **Three apps show no guide: plant-it and home-assistant carry one HIDDEN (`checked_version: ""`), emby has none.** plant-it: `lifecycle: abandoned`, and its pinned image does not exist upstream, so there is no screen to check labels against. home-assistant: its frontend follows the browser language and the Hungarian strings are fetched at build time, not kept in git, so the Hungarian labels cannot be checked against source (the draft is written; the shipped frontend in the image could check them). emby: its server's web UI is in no public source at 4.11.0.4. Evidence: `audits/app-guides-2026-10-15/B/`. The other 62 templates show a checked guide (controller ≥ 0.315.0). | OPEN — filed 2026-10-15 | — | home-assistant: check the Hungarian labels against the frontend files in the pinned image (read-only, on a box running it), then set checked_version; emby: label-check against the image's own web files the same way; plant-it: nothing while abandoned | CC |
|
||||
@@ -233,7 +233,7 @@ stopping line that lies.
|
||||
| **R-1275** | Storage & devices | P4 | **An English household sees the drive names in Hungarian** on the install page (and wherever a storage label shows): „Belső tárhely — 52.0 GB free (default)”, „Külső tárhely (hdd_1)” (catalog sweep 2026-10-24, `audits/catalog-sweep-2026-10-24/navidrome/page-install-en.txt`). The label is stored once, in Hungarian, when the drive is found (`felhom-controller/controller/internal/settings/settings.go` `InferStorageLabel`); a household may rename it. | OPEN | — | Render a label that still equals the inferred default in the reader's language; a test per language | CC |
|
||||
| **R-1276** | Storage & devices | P4 | **„Remove with data” leaves the app's empty parent folder** (`<drive>/appdata/<app>`) when the template binds a subfolder: Grimmory, RomM, Paperless-ngx (catalog sweep 2026-10-24; 4 KB each; the bound folders were removed). Harmless (empty, not counted as old data), untidy for a household that browses the drive. | OPEN | — | Remove the parent when it is empty after the bound folders go; a test | CC |
|
||||
|
||||
## Security & access — 29 rows (P1 2, P2 5, P3 15, P4 7)
|
||||
## Security & access — 30 rows (P1 2, P2 5, P3 16, P4 7)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -248,6 +248,7 @@ stopping line that lies.
|
||||
| **R-1141** | Security & access | P3 | **`felhom-host-install --rotate-recovery` exists only as part of a full installer re-run, which also reinstalls the vouched agent and the root files.** On a box running a newer agent than the vouched one (demo-hp on the candidate, 2026-10-17) it would move the agent back; on a household box it is a much larger act than a password change. The 2026-10-17 rotation therefore ran the installer's own `step_break_glass` ALONE (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | **NARROWED 2026-10-18 — built on `main`, not published.** `felhom-host-install.sh --rotate-recovery-only` runs step 4b alone (felhom.eu `7d64038f`): reads the hub, host id and host key from the agent config, refuses byo and every other mode flag, asks the hub's `/healthz` before the password changes, never touches the agent or root files. The default install path is unchanged in behaviour (harness verdicts and 9 transcripts identical old vs new; demo-hp `--preflight-only` identical; `--rotate-recovery-only --dry-run` on demo-hp changed nothing). `step_break_glass` moved, byte-identical, ahead of the argument parse. `break-glass.md` §5 points at it. Publishes only with the next `installer-v…` tag. | the next installer tag | Cut the next `installer-v…` tag (with the other untagged installer changes); close | CC |
|
||||
| **R-1258** | Security & access | P3 | **ep0's iperf3 line-test server takes any tunnel peer's test: one box (root on its host) can keep it busy for the fleet or ask a long reverse test that fills ep0's uplink.** Found 2026-10-24 (SEC-2 X2): `felhom.eu/scripts/ep0-iperf3/felhom-iperf3.service:17` — no authorized users, no server-side duration or rate cap; one client at a time (`felhom-agent/internal/linetest/iperf.go:263-264` → `busy`; 6 tries a day). Static (no ep0 change in this run). | OPEN | — | per-box iperf3 users or server-side caps, tried on a scratch VM first | CC |
|
||||
| **R-1307** | Security & access | P3 | **Packages inside our own images are not watched** — the Debian packages of the controller image, the Alpine packages of the hub and Samba images (Samba's version is whatever Alpine 3.24 ships). The security watch (R-1303) reads only the base image's point release (BEHIND); Debian's DSAs and Alpine's secdb are not read. `audits/security-watch-2026-10-26/SPIKE.md` „cannot see" | OPEN — filed 2026-10-26 | — | add Alpine secdb (machine-readable JSON per release) and Debian's tracker for the packages the images install, or a scanner run on the built images in CI | CC |
|
||||
| **R-1311** | Security & access | P3 | **Termix (catalog 2.8.0): a stranger can get round the login rate limit by writing their own X-Forwarded-For (GHSA-7j7m-85p4-2cpm), and our template keeps that header** (docmost and paperless strip it, R-753). The upstream fix needs Termix's big step (2.8 → 2.9.2 → 26.10.5). No box runs Termix (hub 2026-10-27). `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 | the operator: a template change (strip the header like R-753, so every visitor shares the tunnel's one address and the limit can lock the household out — the Grimmory shape, R-775) or wait for the big step | the pick in `PICKS.md`; or the big step | CC |
|
||||
| **R-955** | Security & access | P4 | **The cross-box check of the hub API is done in source and tests but not fully proven: if a later check masks a missing id check, a box's key could read or change another box's hub record without any test noticing.** Read 2026-10-12: every box-facing `/api/v1` route takes the box from its own key or refuses a different id; 14 routes already had a refusal test, 5 got one (`crossbox_refusal_test.go`). Left: red-prove the 4 new tests that were not red-proved (as the restore-directive one was), and the live read-only check with one demo box's key against the other demo box's id. | OPEN — filed 2026-10-12 by the security review | — | Red-prove the 4 tests; run the live GET check on the two demo boxes | CC |
|
||||
| **R-924** | Security & access | P2 | **The operator signing keys exist only on DooPlex, with no passphrase and in no backup — a loss of DooPlex means no box ever accepts a signed agent update, config bundle or OS step again without an on-site re-enrollment.** MEASURED 2026-10-09: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` (0600 `kisfenyo`); `ssh-keygen -y -P ''` opens all three (no passphrase); `/mnt/5_hdd/felhom.eu` is in no backup set, the off-site job does not carry them. `04` §1 says the operational key is "passphrase-protected" and the recovery key "kept offline" — today neither holds (custody ruling 1 of 2026-09-16 put both on DooPlex for this phase; `04` §3.1). Both keys share one fate, so the cold key cannot do its one job (re-pin after losing the operational key). `runbooks/total-loss-of-dooplex.md` S6. | NARROWED — **2026-10-23 (D32):** S6a/S6b are on the current sheet. LEFT: the operator prints it. **NARROWED 2026-10-09 — operator ruling: option A** (print both AND copy them). The three key files and their public halves ride the nightly encrypted DooPlex→ep0 copy (`scripts/dooplex-offsite/`, a missing key refuses the push); the Sunday restore test proves each private key still derives its own public half (never printed). **LEFT:** the operator prints S6a/S6b (`runbooks/break-glass-sheet.md`); the keys still have no passphrase (custody ruling 1 of 2026-09-16 stands until the first sale). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G2):** the three key files are on `sda1`, the same disk as DooPlex's local backup tree; their only off-box copy is the S1-encrypted nightly one. Passphrase: unchanged by the custody ruling. | operator: print | Operator: print the two keys; then close | operator |
|
||||
| **R-132** | Security & access | P3 | **`curl -w '%{redirect_url}'` reconstructs the request URL WITH its basic-auth credential** — so a `-u ":$HUB_PW"` call that never put the password in a URL still printed it **Merged 2026-10-05 from R-350 (duplicate):** (1) 2026-08-20 occurrence: POST /configuration/artifacts answers 303; leak lives only in the CC transcript under ~/.claude/projects/ on DooPlex, not in git/evidence (checked then). (2) `-v` and `--libcurl` also re-render the credential, not only %{redirect_url}; confirm redirects with %{http_code} + follow-up GET. (3) Rotation path: hub /configuration form (current_password/new_password/confirm_password); DB override wins over ConfigMap (break-glass); CC can rotate file-to-file without printing (operator-present-one-time-secrets) if asked. | **NARROWED 2026-10-20 — the rotation is DONE:** the operator changed the hub password on 2026-10-15 and the `HUB_PW` line is gone (`audits/hub-tokens-2026-10-15/C153-live.txt:18`; CC uses its own token since hub 0.152.0). **Left, the operator's:** should the break-glass `password_hash` in `manifests/hub.yaml` (a public repository) be emptied or replaced? The DB override wins at login, so the old hash no longer opens the hub. **WAITING-ON-OPERATOR** — **ACTION: rotate `HUB_PW`** **Folded R-580 2026-10-03** (the same `curl -w %{redirect_url}` credential echo, seen again 2026-09-18). **Note 2026-10-12 (security review):** `manifests/hub.yaml` also carries the hub login's bcrypt hash in this public repository (the break-glass field); when `HUB_PW` is rotated, decide whether that field should stay empty. | the operator's word on the `manifests/hub.yaml` hash field | Happened on 2026-07-31 while red-proofing the R-120 gate: the hub operator password was written to the session transcript by the write-out format, not by the request. `-u` is safe; the *reporting* was not. Rule: read the redirect from `-D -` and grep `^Location:`, never `%{redirect_url}`, on any authenticated call. Rotate the hub password (`/configuration` → Login password; ConfigMap `auth.password_hash` is the reset path) and update `~/.config/credentials` | Viktor |
|
||||
@@ -287,12 +288,13 @@ stopping line that lies.
|
||||
| **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
|
||||
|
||||
|
||||
## Monitoring & notifications — 24 rows (P2 1, P3 16, P4 7)
|
||||
## Monitoring & notifications — 25 rows (P2 1, P3 17, P4 7)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-1261** | Monitoring & notifications | P3 | **The R-1250 line hold (committed, NOT released): three edges before it ships (review round 2, HUNT-4 H4–H6).** Two seeders behind one uplink: the hold takes only the first, so the test is refused „not quiet” every 15 min in the Tuesday window and never counted (`felhom-controller/controller/internal/upwatch/linehold.go:155-160`; agent `internal/linetest/linetest.go:379-381`); a hold record of an app removed while held is never cleared and the 50 % rule never runs again (`linehold.go:106-108`; `upwatch/service.go:149`); a qBittorrent alt-speed flip during the hold leaves 1 KiB/s stored (`explain.go:175`; `linehold.go:127-128`). | **VERIFY** — RELEASED in controller v0.344.0 (global floor 2026-10-26 09:07:47Z, every live box on 0.344.0); the live test with a seeding box is Tuesday 10:00–16:00 (R-1250's dated check). **Earlier:** **VERIFY** — fixed on main 2026-10-25 night (controller `a79a292`), waits for the controller release (before any household gets agent 0.165.0 by signed update): every running seeder held, a removed app's record dropped, qBittorrent's slot named (`up_limit`/`alt_up_limit`). Red-proofs `audits/night-2026-10-25/redproof-r1261.txt`. **Live on the night's rig** (BYO VM, agent 0.165.0, rc.1, two seeders): both held, 94.8 Mbit/s, both limits back; a speed-mode switch mid-test left no 1 KiB/s (`rig/hold-watch*.txt`); the removed-app case by test only. **Earlier state:** OPEN — **now in controller 0.343.0 (released 2026-10-25); the three edges act only on a box whose agent is ≥ 0.165.0 (it alone reports `waiting_for_quiet`) — fix before any household gets agent 0.165.0 by signed update** | — | before the next controller and agent release: hold every running seeder; clear a record whose app is gone; read the mode | CC |
|
||||
| **R-1262** | Monitoring & notifications | P3 | **The upload watch treats ANY viewer as an explanation for any amount of upload, so a hacked Plex can send freely while one family member watches.** Found 2026-10-24 (SEC-1 X1): `felhom-controller/controller/internal/upwatch/watch.go:275,299`; `08` §9 states the rule, its limits list did not name this case. | OPEN | — | propose a cap (viewers × a bitrate ceiling) for the operator's R-1244 design; `08`'s limits list names the case meanwhile | CC |
|
||||
| **R-1310** | Monitoring & notifications | P3 | **The security watch (R-1303) flags notices that do not apply and misses fixes that do — measured by reading all 38 flagged pins by hand on 2026-10-27.** False: advisories of ecosystem `actions` (Home Assistant's CI), of another package in the repo (Gitea's runner), with a hostname or ≤-old range (cal.com 2023/2024, Grafana 2022), for the desktop/Electron client (Termix), for the rootless image only (Uptime Kuma); a release hit on a word (Dawarich „no longer receives security fixes", Rallly v4.16.0). Wrong „no fix": Vikunja's six (range `<= 2.6.0`, fixed in 2.7.0), karakeep-chrome (154/156 exist). Missed: Tandoor 2.7.0 names 8 GHSA ids (1 matched), Dawarich 1.16.0 (2FA, export deletion), Wishlist v0.69.0 (critical, no GHSA). BEHIND reads linuxserver `26.09.20260929` as newer than 26.09.2; Seerr follows the renamed-away `fallenbagel/jellyseerr`. `audits/security-app-updates-2026-10-27/TRIAGE.md` NOT-A-FINDING list and `triage-notes/` | OPEN — filed 2026-10-27 | — | teach `security_watch.py`: skip ecosystem `actions` and a package that is not the pinned one; read an upper-bound-only range as „fixed after"; match every GHSA id a release body names; ignore `YY.MM.YYYYMMDD` tags; repoint seerr to `seerr-team/seerr`; each with a test seen failing | CC |
|
||||
| **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
|
||||
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING — DESIGN WRITTEN 2026-10-26 night (the brief's Part 7: build only if time is left; it was not built — the line test and the attended release came first).** The host-side counter EXISTS and is readable (read on demo-hp 2026-10-26): `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events`, line `oom_kill N`. **Design:** (1) agent: a read-only collector (no root CLI — a sysfs read under the agent's own rights; if not readable, the existing sudoers fence gets one read-only `Cmnd_Alias`) that, for each guest in the `felhom` pool, globs those scopes and keeps `{container_id: oom_kill}`; served on the guest's own local API as `GET /guest/oom-kills` (`withGuest`, the guest's containers only). (2) controller: the 30-s dead-app scan asks once per pass; a RISE of a container's counter = an OOM kill whatever Docker's `OOMKilled` says → the existing `app_oom` line (R-636's per-run grain), `source: cgroup` in the details. An agent without the route = today's behaviour. (3) Tests: a fake cgroup tree (agent), a rising counter → one `app_oom` (controller, red-first). (4) Live: 9202, a throwaway app capped at 64M with a memory hog: the counter rises, `app_oom` names the container while `OOMKilled` reads false. ~1 day, two repos, one agent release. **Earlier:** WATCHING — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a Line truncated
|
||||
| **R-1250** | Monitoring & notifications | P3 | **A qBittorrent that seeds all day is never line-tested, so it never gets the limit that exists for it.** The weekly test (R-1247) starts only after 10 minutes in which the host's uplink sent < 2 Mbit/s each minute; a seed at the line's speed keeps it busy through every window, so no test, no tested line, and no qBittorrent limit (D41 uses the tested line only). Found 2026-10-24 by reading the rule against the case it serves; not yet seen on a box (no test box seeds). **Due check moved 2026-10-27 → 2026-10-28 (2026-10-27 04:20Z, CC):** the measurement runs TODAY in the host's window (from 10:00 CET); on 2026-10-27 the gate would refuse every push until then. The result is recorded here today, and the line leaves the block. | **VERIFY — released 2026-10-25, the live proof is a dated check:** controller 0.343.0 (global floor 17:53Z) + agent 0.165.0 (vouched 18:38 CET) carry it; the real proof cannot run before the next due window — the agent starts a test only on its due Tuesday 10:00–15:59 local, and every box measured on 2026-10-24. **DUE 2026-10-27:** a throwaway qBittorrent on demo-hp seeding to DooPlex's qBittorrent (operator's yes 2026-10-25, about 30 minutes), from before the host's slot; read the agent's line value and the limit back. **Earlier: NARROWED 2026-10-24 — FIXED on main, not released** (brief 2026-10-24 Part 4, commit only): agent `29bcf44`+`1e087bb` (`waiting_for_quiet`, `POST /host/line-test/run`), controller `046b808` (`internal/upwatch/linehold.go`: record first, hold at 1 KiB/s, test, the exact old limit back; the start-up check after a kill; the 50 % rule waits). Red-proofs in both CHANGELOGs. **Live on demo-hp with a real seed** (10 Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; held: uplink 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, back to 1 249 280 B/s exactly; a kill -9 mid-test left the record and the start-up check put the limit back (`audits/dooplex-alarms-2026-10-24/part4-r1250-live-proof.txt`). The live run used probes over plain HTTP (the pinned agent client is unit-tested, not live). | the next release train (agent ≥ 0.165.0 needs the operator's vouch) | Release agent + controller in the next normal train; on a seeding box read one Tuesday's `line test on request` line; then close | CC |
|
||||
|
||||
Reference in new issue
Block a user