Runtime fixes 2026-10-28: TABLE of the 20 apps (calibre-web loads Brotli — corrected; file manager does not, 3 boxes), lane batches 1+2 evidence; 09 decisions 277 (D61, operator) + 278 and the lane batch; R-1313/R-1322/R-1327 closed; R-1334..R-1337 opened; app rows updated
This commit is contained in:
admin committed
1 parent
29e9cd36a6
commit
47e1ef062b
33 files changed
+2880
-11
No files matched your search
@@ -26,6 +26,14 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-28 (day) — R-1313, R-1322 and R-1327 closed: Wanderer hidden, Immich's lane step live, the runtime watch fills its own cache in CI (full text of the rows: `git show 9b4b5fe5:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-1313** | **Wanderer (v0.20.0): a critical notice with no login — any remote ActivityPub server can overwrite its records (GHSA-fprh-3pp9-68m9); the fix is v0.21.2, a big step.** (P2) | CLOSED 2026-10-28 — D59 (`09` §3 decision 275) live: catalog `b5b78f6` sets `lifecycle: hidden` (CI job 2447 green); no box runs Wanderer. The big step v0.21.2 (with the fixed Node) carries on in R-1335. | `audits/http2shell-2026-10-27/wanderer-hidden.patch`, drill `ee18bff` |
|
||||
| **R-1322** | **Immich v3.2.4 (server + ML): the Node.js October 2026 runtime notice (CVE-2026-111497); the lane step to v3.3.5.** (P3) | CLOSED 2026-10-28 — the security-fix step v3.2.4 → v3.3.5 live: catalog `0158b50` (CI gates job 2448 green), on the operator's yes. Box 9202 `proven` (2026-10-27 night); the power-cut repair re-proven on v3.3.5 the same morning (4 of 4 rebuilt, 0 of 52 controls changed, the lost original reported) — `unclean-boot-images` had refused the step without it. | `audits/http2shell-2026-10-27/walks/immich/`, `audits/runtime-fixes-2026-10-28/immich-repair/` |
|
||||
| **R-1327** | **The security watch could not see a bug in the RUNTIME under an app (HTTP2Shell, CVE-2026-111497) — Home Assistant and RomM ran the bad code on the test boxes unseen.** (P2) | CLOSED 2026-10-28 — the watch reads every pin's runtime from the image (built 2026-10-27 night) and since `a3247d9f` reads zstd layers IN CI by itself (a hash-pinned `zstandard` 0.25.0 wheel, `09` §3 decision 278; no DooPlex change). The close condition held: the first full CI night (job 2446, 03:40) named Home Assistant 2026.9.4 and RomM 5.4.1 as ADVISORY (runtime) and audiobookshelf 2.37.2 as current. The self-fill: CI dry run job 2450 with HA 2026.9.4 and RomM 5.4.1 taken out of the cache read Home Assistant 2026.10.4 (zstd) in 87 s and RomM 5.4.1 (gzip) in 25 s, and answered audiobookshelf from the cache; the job took 3 min 34 s. Left to see: tonight's run is the first that COMMITS CI-read entries (a dry run does not) — STATUS asks for one look; Docker Hub's 429 is R-1334. | CI jobs 2446, 2449, 2450; `scripts/CHANGELOG.md` 2026-10-28; `08` §11 |
|
||||
|
||||
## 2026-10-27 (late evening) — R-1326 closed: the hub gets time to start (full text of the row: `git show 503692a1:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|
||||
@@ -147,31 +147,32 @@ stopping line that lies.
|
||||
| **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | WATCHING — **2026-10-22 night:** not reproducible tonight; the Immich server log of 19:38 is gone (Immich restarted at 02:16 for the off-site dump). What the household sees: the upload fails after 10 minutes and the Immich phone app sends it again. | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC |
|
||||
| **R-1267** | Apps & catalog | P4 | **Every recreate of Immich leaves one empty anonymous Docker volume behind: the image declares `VOLUME /data`, the template does not map it.** Found 2026-10-24 (review round 2, while putting 9202 back): 9202 holds 296 empty anonymous volumes; each night chain's recreate of `immich-server` made a new one (12:58, 13:00, 13:01 — the three test runs; the two dangling ones this session made were removed by name, nothing pruned). The photos are NOT affected: Immich picks `/usr/src/app/upload` whenever both folders exist (`detectMediaLocation` in the image's `server/dist/services/storage.service.js:28-39`), and the template binds that folder (`templates/immich/docker-compose.yml:49`) — NOT-A-FINDING for data placement. **Seen again 2026-10-24 night on Tester 1:** privatebin's image declares `/run`, `/tmp` and `/var/lib/nginx/tmp` as volumes — each recreate (the night dump's stop/start) leaves three empty anonymous volumes; the same class as Immich's `/data` (`audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt`). | OPEN **2026-10-24 (catalog sweep):** the same class for OnlyOffice and RomM (RomM's image volume `/romm` holds only the empty mount points of its binds); after every catalog app had been removed WITH data, the sweep box still held 52 such volumes (PrivateBin 21, OnlyOffice 8, RomM 8, Immich 3, and 12 from 12:50–12:52 before the batch, holding the same nginx folders as PrivateBin's) — remove-with-data does not see them (no compose project label). All small (4–72 KB), no household data (`audits/catalog-sweep-2026-10-24/_box/anon-volumes.txt`). | — | declare the image's VOLUME paths as tmpfs or named volumes in the catalog (Immich, privatebin), or remove the dangling ones a recreate leaves, by name | CC |
|
||||
| **R-1268** | Backup & restore | P3 | **Remove with „remove backups” leaves a restore's undo copy on the drive (`backups/restore-undo/<app>/`).** Seen 2026-10-24 night on Tester 1 (restore-holes Part 1, run 1): MeTube was held for support after a failed crash put-back, then removed with `remove_backups=true` — the 1.5 GB copy stayed (the removal deletes `backups/primary` and the second-drive mirror, not `restore-undo`), and nothing lists it (`audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt`; removed by hand, CC's own test data). The copy is kept on purpose while a hold exists; after the app is gone it is orphaned. | OPEN | — | Remove with `remove_backups` also deletes the app's `restore-undo` folder (under the same path guard), or lists it as a kept copy | CC |
|
||||
| **R-1329** | Apps & catalog | P3 | **cal.com runs on Node.js 20, which gets no security fixes any more** — v6.2.0 carries Node 20.20.0 (no fixed release on the 20 line for CVE-2026-111497) and Debian 12's unpatched `libbrotli1` 1.0.9-2+b6 (fixed: 1.0.9-2+deb12u1); the same image runs in the catalog and on DooPlex. Read from the image 2026-10-27. | OPEN | — | a cal.com release on Node 22/24 (a big step: its own walk); until then the watch lists it nightly | CC |
|
||||
| **R-1330** | Apps & catalog | P2 | **RomM's newest release (5.4.1) still ships Node.js 24.18.1 (CVE-2026-111497, fixed 24.22.1)** — no newer RomM exists (tags read 2026-10-27). Exposure: Node is IN the image but no Node process runs (RomM serves through Python gunicorn/rq and nginx — `ps` on demo-hp 9201), and RomM sits behind the setup gate. | OPEN — **CC's pick: keep RomM offered, take the first RomM release with Node ≥ 24.22.1 as a lane step** (decided by CC unattended 2026-10-27 — operator may reverse; STATUS asks) | — | the watch flags it nightly (R-1327); the step when RomM ships | CC |
|
||||
| **R-1331** | Apps & catalog | P3 | **Paperless-ngx 2.20.15 carries Python Brotli 1.1.0 (CVE-2026-111497, fixed 1.1.1/1.2.1)** — its web side only COMPRESSES (whitenoise, compression_middleware); the DECODERS are urllib3's and httpx's, for replies to requests Paperless itself makes (its own outbound calls), never for what a visitor sends. Read in the running container on demo-hp 9201, 2026-10-27. | OPEN — no step tonight (decided by CC unattended — operator may reverse): the bad path needs a hostile server Paperless calls | — | take the first Paperless release with Brotli ≥ 1.2.1 as a normal step | CC |
|
||||
| **R-1329** | Apps & catalog | P3 | **cal.com runs on Node.js 20, which gets no security fixes any more** — v6.2.0 carries Node 20.20.0 (no fixed release on the 20 line for CVE-2026-111497) and Debian 12's unpatched `libbrotli1` 1.0.9-2+b6 (fixed: 1.0.9-2+deb12u1); the same image runs in the catalog and on DooPlex. Read from the image 2026-10-27; 2026-10-28: v6.2.0 is still the newest tag, no Node 22/24 image (`audits/runtime-fixes-2026-10-28/TABLE.md`). | OPEN | — | a cal.com release on Node 22/24 (a big step: its own walk); until then the watch lists it nightly | CC |
|
||||
| **R-1330** | Apps & catalog | P2 | **RomM's newest release (5.4.1) still ships Node.js 24.18.1 (CVE-2026-111497, fixed 24.22.1)** — no newer RomM exists (tags read 2026-10-27). Exposure: Node is IN the image but no Node process runs (RomM serves through Python gunicorn/rq and nginx — `ps` on demo-hp 9201), and RomM sits behind the setup gate. | OPEN — **RULED D61 = keep RomM offered** (operator 2026-10-28, starting the day brief; `09` §3 decision 277): the first RomM release with Node ≥ 24.22.1 is a lane step. 2026-10-28: 5.4.1 is still the newest tag (read from the registry). Earlier: CC's pick 2026-10-27 night | — | the watch flags it nightly (R-1327); the step when RomM ships | CC |
|
||||
| **R-1331** | Apps & catalog | P3 | **Paperless-ngx 2.20.15 carries Python Brotli 1.1.0 (CVE-2026-111497, fixed 1.1.1/1.2.1)** — its web side only COMPRESSES (whitenoise, compression_middleware); the DECODERS are urllib3's and httpx's, for replies to requests Paperless itself makes (its own outbound calls), never for what a visitor sends. Read in the running container on demo-hp 9201, 2026-10-27. 2026-10-28: the newest release 3.3.4 still carries Brotli 1.2.0 and is a new main version — no fixed release yet (`audits/runtime-fixes-2026-10-28/TABLE.md`). | OPEN — no step tonight (decided by CC unattended — operator may reverse): the bad path needs a hostile server Paperless calls | — | take the first Paperless release with Brotli ≥ 1.2.1 as a normal step | CC |
|
||||
| **R-1332** | Apps & catalog | P2 | **Twenty catalog apps ship a runtime with HTTP2Shell (CVE-2026-111497) inside their image — the first runtime read of the whole catalog, 2026-10-27 night (R-1327):** actualbudget 26.9.0 (Node.js 24.20.0 → 24.22.1); adventurelog v0.13.0 (Node.js 26.3.0 → 26.12.1); calcom v6.2.0 (Node.js 20.20.0 → 22.23.4); calibre-web v4.0.8 (py-Brotli 1.2.0 → 1.2.1); dawarich 1.15.3 (Node.js 20.19.2 → 22.23.4); ghost 6.67.0-alpine (Node.js 22.23.3 → 22.23.4); home-assistant 2026.9.4 (py-Brotli 1.2.0 → 1.2.1); homepage v1.13.2 (Node.js 22.22.3 → 22.23.4); immich v3.2.4 (Node.js 24.21.0 → 24.22.1); karakeep 0.33.2 (Node.js 24.18.1 → 24.22.1); metube 2026.09.29 (py-Brotli 1.2.0 → 1.2.1); paperless-ngx 2.20.15 (py-Brotli 1.1.0 → 1.1.1); papra 26.6.2-rootless (Node.js 26.4.0 → 26.12.1); romm 5.4.1 (Node.js 24.18.1 → 24.22.1); seerr 2.7.3 (Node.js 22.18.0 → 22.23.4); sparkyfitness v0.17.3 (Node.js 24.18.0 → 24.22.1); tandoor 2.6.15 (Node.js 24.18.1 → 24.22.1); termix 2.8.0 (Node.js 26.9.0 → 26.12.1); wanderer v0.20.0 (Node.js 22.23.1 → 22.23.4); zipline 4.8.0 (Node.js 24.21.0 → 24.22.1). Plus one BOX program: the file manager `gtstef/filebrowser:1.5.8-stable` carries Alpine `brotli-libs` 1.2.0-r1 (a controller release when upstream rebuilds). None of these has a notice of its own. | OPEN — home-assistant and immich: lane steps walked tonight (drill, the live push waits for the operator); romm: R-1330; calcom: R-1329; paperless: R-1331; the rest: each app's next release whose image carries the fixed runtime, walked as a lane step — the watch lists them every night until then | — | per app: the first upstream image with Node ≥ 22.23.4 / 24.22.1 / 26.12.1 or Brotli ≥ 1.2.1, a lane step; close when the watch's runtime list is empty | CC |
|
||||
| **R-1336** | Apps & catalog | P3 | **ActualBudget has no release with the HTTP2Shell fix (CVE-2026-111497): 26.9.0 carries Node.js 24.20.0 and the newest 26.10.0 still Node.js 24.21.0 (fixed: 24.22.1), read from both images.** `node app.js` serves the app; the setup gate covers it until the first account exists, then its own password. No box runs it. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Node ≥ 24.22.1 | the watch lists it nightly (R-1327); take that release as a lane step | CC |
|
||||
| **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC |
|
||||
|
||||
|
||||
## App updates — 15 rows (P2 3, P3 12)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-1313** | App updates | P2 | **Wanderer (v0.20.0): a critical notice with no login — any remote ActivityPub server can overwrite its records (GHSA-fprh-3pp9-68m9), and federation has no off switch; the fix is v0.21.2, a big step.** v0.21 adds a REQUIRED `POCKETBASE_PROXY_SECRET` on both services, its migration shortens rich text with no way back, and upstream wants PocketBase closed while our template publishes it on `SUBDOMAIN_DB`. No box runs Wanderer (hub 2026-10-27). `audits/security-app-updates-2026-10-27/TRIAGE.md`, `PICKS.md` pick 4 | **VERIFY** — 2026-10-27 night: D59 made in the DRILL (`ee18bff`, `lifecycle: hidden`) and read back on 9202 through the controller's API (`lifecycle: hidden`; Bookstack as the control: not set); the live change is `audits/http2shell-2026-10-27/wanderer-hidden.patch`, waiting for the operator's yes. **Earlier:** OPEN — **RULED D59 = hide from new installs until the big step** (operator 2026-10-27 21:33, `09` §3 decision 275) — the catalog change is drilled with tonight's steps; its live push waits for the operator's yes. — filed 2026-10-27 (security app updates, Part 3: not done today) | the operator's pick 4 (hide it from new installs, or leave it offered) | the big step v0.21.2 with a new fixture field and a closed PocketBase: bench + 9202 + the undo proven, then a STOP | CC |
|
||||
| **R-1315** | App updates | P2 | **Gitea 1.27.3: two critical notices for any signed-in user (7cq7, 6pv5) and two with no login (c3cr, hf55); the fix is 28.2.0 (1.28 was renamed 28.0.0 — no 1.27.4 exists).** 28.0 routes git traffic through a new proxy with egress settings. GHSA-rqhx (SSRF through migration) has no fix at all (`PICKS.md` pick 2). No box runs Gitea. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — **RULED D57 = keep repository import on** (operator 2026-10-27 21:33, `09` §3 decision 273) — judged again in the big step. — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step 28.2.0: read the 28.0 proxy settings against our template; bench + 9202 + undo; a STOP | CC |
|
||||
| **R-1316** | App updates | P3 | **Gokapi v1.9.6: GHSA-3c22 (high, self-XSS for the only account); the fix is v2.x, a big step** — a user system with no downgrade, and our first-boot config seed must be captured again. No box runs Gokapi. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step v2.2.4 with a new first-boot seed; bench + 9202 + undo; a STOP | CC |
|
||||
| **R-1317** | App updates | P3 | **Seerr (jellyseerr 2.7.3): GHSA-rc4w (a stranger registers, Plex setups only) and GHSA-mc6w (path traversal to code execution, needs a hostile media server); the fix is `ghcr.io/seerr-team/seerr:v3.5.1`, a big step** — a new image name, runs as UID 1000 (`/app/config` must be chowned), a config migration. No box runs it. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step to seerr v3.5.1: the ownership step, the image rename; bench + 9202 + undo; a STOP | CC |
|
||||
| **R-1317** | App updates | P3 | **Seerr (jellyseerr 2.7.3): GHSA-rc4w (a stranger registers, Plex setups only) and GHSA-mc6w (path traversal to code execution, needs a hostile media server); the fix is `ghcr.io/seerr-team/seerr:v3.5.1`, a big step** — a new image name, runs as UID 1000 (`/app/config` must be chowned), a config migration. No box runs it. `audits/security-app-updates-2026-10-27/TRIAGE.md` **2026-10-28 (R-1332): the HTTP2Shell runtime fix rides this big step too** — jellyseerr 2.7.3 carries Node.js 22.18.0, seerr v3.5.1 Node.js 22.23.4 (read from both images); `pnpm start` runs Node, behind the setup gate. | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step to seerr v3.5.1: the ownership step, the image rename; bench + 9202 + undo; a STOP | CC |
|
||||
| **R-1318** | App updates | P3 | **Karakeep 0.33.2 + chrome 151: two CISA-KEV V8 bugs in the crawler's Chromium (runs `--no-sandbox`), reached by content the household saves; the fix is 0.34.1 + chrome 154.0.8037.57-r1** — an auth library swap (everyone signed out, e-mails lowercased) and a chrome rebuilt on full Chrome (memory limit and healthcheck to measure again). No box runs it. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step: measure chrome 154's memory; bench + 9202 + undo; a STOP | CC |
|
||||
| **R-1319** | App updates | P3 | **Paperless-ngx 2.20.15: GHSA-2jhj (low; needs a setting we do not set — not reachable); the fix is 3.3.4, a big step** (encryption and API v1 removed, the search index rebuilt). It runs on all four test boxes. Not urgent. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step 3.3.4 when the catalog takes Paperless 3 for other reasons; read our API v1 use first | CC |
|
||||
| **R-1320** | App updates | P3 | **Homepage v1.13.2: a medium notice that needs a hand-written credentialed widget (family_gate in front); the fix is v2.7.0, a big step** (the auth rework). No box runs it. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step v2.7.0; bench + 9202 + undo; a STOP | CC |
|
||||
| **R-1320** | App updates | P3 | **Homepage v1.13.2: a medium notice that needs a hand-written credentialed widget (family_gate in front); the fix is v2.7.0, a big step** (the auth rework). No box runs it. `audits/security-app-updates-2026-10-27/TRIAGE.md` **2026-10-28 (R-1332): the HTTP2Shell runtime fix rides this big step too** — v1.13.2 carries Node.js 22.22.3 (`node server.js`, as root, behind the family gate), v2.7.0 Node.js 24.22.1 (read from both images); no fixed v1.x exists. | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step v2.7.0; bench + 9202 + undo; a STOP | CC |
|
||||
| **R-1335** | App updates | P3 | **Wanderer's big step v0.20.0 → v0.21.2 (the successor of R-1313): it carries the ActivityPub fix (GHSA-fprh-3pp9-68m9) AND the HTTP2Shell runtime fix (Node.js 22.23.1 → 22.23.4, read from both images).** Wanderer is hidden since 2026-10-28 (D59); no box runs it. A main-version move by the operator's ruling (D59), so a normal ladder step, never the lane. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | — | the big step: bench (R-739: the bench cannot run Wanderer yet) or a box-only walk on 9202 with the undo; a STOP; then un-hide | CC |
|
||||
| **R-1321** | App updates | P3 | **Tandoor 2.6.15: GHSA-5f74 (high, an XXE file read) and GHSA-69q3 (any user); the fix is 2.7.2** — 2.7.0 removes settings and an endpoint and rewrites the login pages (our probe path — not sure), and 2.7.0/2.7.1 break invite links with sign-up off, so straight to 2.7.2. No box runs it. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | a normal step to 2.7.2 with the probe path checked; bench + 9202 (+ undo if a migration); a STOP | CC |
|
||||
| **R-1322** | App updates | P3 | **Immich v3.2.4 (server + ML): only the Node.js October 2026 runtime notice (CVE-2026-111497) and a CA bundle; the lane step to v3.3.5 was not walked today** (the lowest item of the lane). No household runs Immich; 9202 has a leftover install. `audits/security-app-updates-2026-10-27/TRIAGE.md` | **VERIFY** — 2026-10-27 night: the lane step v3.2.4 → v3.3.5 WALKED on 9202 (the old install removed through the product first): `proven` — the guarded Update 122 s, the seed read back, soak peak 42.2 %, no kill, no restart (`audits/http2shell-2026-10-27/walks/immich/`); written to the DRILL (`34b14d5`); the live push waits for the operator's yes (STATUS). **Earlier:** OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the security-fix lane: `box_walk.py --security immich` on 9202 (remove the leftover first), the writer, a batch STOP | CC |
|
||||
| **R-1324** | App updates | P3 | **RomM 5.4's first start can run a 15–20 minute migration on a big library (upstream notes) — longer than the update's check window, so a household's night step could be undone for slowness, not for a fault.** Today's walk (9202, a small seed) took the step in seconds; demo-hp's RomM takes it tonight (security-fix step, catalog `06e603e`). `audits/security-app-updates-2026-10-27/triage-notes/romm.md` | OPEN — filed 2026-10-27; WATCH demo-hp's night leg line | — | read demo-hp's night leg line for romm (done / undone, the time); if undone for time: give the step a longer check or the `needs_person` mark | CC |
|
||||
| **R-1238** | App updates | P2 | **Plex has no test record and no upgrade fixture, so the night leg never updates Plex on a household box.** Filed 2026-10-23. Catalog `6cc317d` moved Plex 1.41.4.9463 → 1.43.5.11029-6bf8af3f0 (the Plex2Shell fix) on the operator's ruling, with NO `update_ladder` entry (the move gate refused; pushed `--no-verify`, the one time). With no ladder, a person's Update press pins the current compose (`stacks/ladder.go` `nextLadderStep`), but the night leg skips the app (`unattended.go` → `LegSkipNoTestRecord`), so every future Plex fix reaches a household only by a hand press. `scripts/upgrade_fixtures.py` has no Plex fixture. A likely seed route is NOT measured: an UNCLAIMED Plex answers a private-address request with no token (`architecture/13-threat-model.md`, the R-1045 row), so the bench may create and read back a library section through Plex's own API without a plex.tv account. **Needs a spike:** prove that route on the bench (seed a section, swap the image, read it back), add the fixture, then `upgrade-test.py --write-ladder` for the current pin — the box walk on 9202 needs the same route behind the setup gate. Evidence: `audits/plex-demo-boxes-2026-10-23/`. **2026-10-23 (later): the NEXT Plex fix need not wait for this** — the security-fix lane (`09` §3 decision 255, D39 = B, controller 0.341.0) takes a box-only step with `seed: "none"` at night (proven on 9202 with Plex 1.43.4 → 1.43.5, `audits/security-lane-2026-10-23/`). The bench fixture stays the aim: it is the only check that reads Plex's data back after a step. | **OPEN** | a claimed-Plex seed route on the bench | the spike above | CC |
|
||||
| **R-440** | App updates | P3 | **[P2-MEDIUM] 23 catalog image pins float, so an update is not reproducible.** `compose pull` on a moving tag fetches whatever upstream published that day. **MEASURED 2026-09-01 over `app-catalog-felhom.eu` @ `29edad9c5bf4`: 79 `image:` lines across 53 apps, 66 distinct; 23 of those lines carry a tag with no patch version.** `postgres:16-alpine` (8 apps), `redis:7-alpine` (6), `mariadb:11.6` (2), plus one each of `postgres:15-alpine`, `postgis/postgis:16-3.5-alpine`, `mariadb:11.4`, `mariadb:12.3`, `ghcr.io/claperco/claper:2.5`, `ghcr.io/thomiceli/opengist:1.13`, `wger/server:2.6`. **A 24th is arguable and is recorded rather than rounded away:** `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0` pins both extensions exactly but leaves the PostgreSQL patch floating. A customer pressing Frissites can therefore swap their DATABASE ENGINE build with no catalog change and no record; two boxes updated on two days end up different. **Severity MEDIUM on its own; it becomes BLOCKING the moment a pre-update copy exists**, because "what did we upgrade from and to" must be recordable and today it is not — which is also why R-440 must be read next to the digest discipline in Rule 10 of the spike. **MEASURED LIVE 2026-09-01 — the floating pins have ALREADY moved, with a passing control.** Running digests on demo-hp compared against what the registry serves for the same tag today: **`mariadb:11.4` MOVED** (`sha256:4f1d8d20...` -> `sha256:611a2fcc...`) and **`mariadb:12.3` MOVED** (`sha256:a02fe89c...` -> `sha256:dd9b303a...`), while `postgres:16-alpine`, `redis:7-alpine`, `mariadb:11.6` and `opengist:1.13` were SAME — **and both fully-pinned CONTROLS (`rommapp/romm:5.0.0`, `privatebin/pdo:2.0.5`) were SAME.** So on a box with ZERO visible drift by tag, pressing Frissites today silently swaps the DATABASE ENGINE build under `romm` and `bookstack`, with no catalog change and no record. **Compounding fact found while reading:** the recovery unit records `ImagePins` but the manifest comment says *"image NOT stored - re-pulled on restore"*, so a RESTORE of a floating-pinned app also re-pulls whatever is current — the same non-reproducibility on the recovery path. **HALF OF THE ANSWER SHIPPED 2026-09-02 (controller v0.233.0, slice 1): `app.yaml.installed_images` now records, per compose SERVICE, the reference AND the repo digest each container was actually created from — so "what did we upgrade FROM" is answerable on any box that has taken one lifecycle action since the upgrade.** What is still missing is the other half: comparing that digest against what the registry serves for the same tag TODAY, which needs a network call the render path deliberately does not make (see R-446). **The row therefore stays OPEN and its rank is unchanged** — recording a digest does not make a floating pin reproducible; it makes the drift measurable after the fact. `audits/SPIKE-app-update-2026-09-01.md` **— NIGHT 2026-09-23:** every image moved tonight (and the 21 backfilled moves) carries its resolved digest in the catalog's test record; the floating pins themselves still float until the box pulls by digest (`09` §6.4 part 6). **-- MEASURED 2026-09-30 on demo-hp 9201: a fresh install or a guarded Update renders `name:tag@sha256` from the ladder (`digest.go` `RenderWithLadderDigests`) — 9 of 20 services there run such a definition (adventurelog, docmost, paperless-ngx). The other 11 run TAG-ONLY definitions: bookstack, kimai, opengist, privatebin, romm were installed before v0.269.0 and not updated since, and `CarryDigests` (`digest.go:141`) keeps only a digest the running definition already has; bentopdf and calibre-web have no ladder at all. So a re-pull of those (a restore) takes whatever the tag serves that day. `audits/pg-last-six-2026-09-30/E/E2-digests-demo-hp.txt`.** **-- 2026-09-30 (evening):** calibre-web now has a ladder entry (`53a4a1d`), so a new install or a guarded Update renders its tested digest; bentopdf still has none. `audits/more-night-apps-2026-09-30/` **Merged 2026-10-05 from R-446 (duplicate):** the household-visible consequence — for these apps the „Naprakész" badge can never turn „behind" (`felhom-controller controller/internal/stacks/updateorder.go:96` returns false with no catalog digests / no test date); R-740 (same-tag re-test) is closed, so floating-tag drift for laddered apps is handled. | **NARROWED 2026-09-30 — floating only for (1) apps with no ladder entry and (2) apps installed before v0.269.0 until their next guarded Update; owner: CC.** **Re-ranked 2026-10-03: P2→P3: narrowed to apps with no ladder entry and pre-v0.269 installs; each guarded update fixes the latter.** | — | — | CC |
|
||||
| **R-462** | App updates | P3 | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s,Line truncated
|
||||
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
|
||||
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:** **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
|
||||
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:** **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
|
||||
|
||||
## Backup & restore — 46 rows (P2 4, P3 24, P4 18)
|
||||
|
||||
@@ -249,14 +250,14 @@ stopping line that lies.
|
||||
| **R-1241** | Security & access | P1 | **2026-10-24: the Felhom half is DONE** — hub 0.168.0 deployed (the tokens sealed; every copy leaving the volume is clean, the live file's old bytes accepted by the operator, R-975 closed), the hub volume out of the Longhorn backup (D40, R-1243 VERIFY), the two old plain copies deleted by the operator. **What stays here is the homelab half:** the open NFS export and Longhorn's login (the homelab session). **A hacked app pod on DooPlex can read the hub's nightly database copies — with every customer's unsealed Cloudflare, tunnel and git token (R-975) — through the Longhorn NFS export, and by a published file-handle escape the operator signing keys on the same disk: whoever breaks one homelab app can take over the boxes' tunnels and sign agent updates.** MEASURED 2026-10-23 (Plex2Shell impact audit, `audits/plex2shell-felhom-impact-2026-10-23/`): export `/mnt/5_hdd/backup/longhorn-pvc` rw, `no_root_squash`, `no_subtree_check`, a sub-directory of sda1, which also holds `/mnt/5_hdd/felhom.eu/felhom-op-*` (no passphrase, R-924); `secure` is void in pods (containerd sets `ip_unprivileged_port_start=0`; a uid-1000 probe with no capabilities bound port 700); the four `felhom-system` volumes have `encrypted=false`; Longhorn's API on :9500 answers any pod with no login (backups, targets, volumes). In the Plex2Shell window neither door was used — verdict NOT REACHED (nfsd accepted 0 new connections 10-10 → 10-23, positive control; no off-schedule Longhorn backup or volume). 2026-10-23 ~15:18Z, operator yes: the export was narrowed from `*` to `10.42.0.0/16` — the home network and the tailnet are closed, pods are not (Longhorn connects from its pod address). Not exercised: the escape itself (Tier-2 machine). | OPEN | the homelab session (DooPlex host + cluster); the operator (R-924) | Homelab: a NetworkPolicy so only `longhorn-system` reaches :2049/:111 and :9500, or the backup store on its own filesystem. Felhom: R-975 sealed in hub 0.168.0 (deploy pending); the hub's plain Longhorn copies → R-1243; a passphrase on the signing keys (R-924) | operator |
|
||||
| **R-1303** | Security & access | P2 | **Nobody watched the security notices of what we ship** — now a nightly CI watch (`scripts/security_watch.py`, 03:40 UTC; 108 pins: 11 box/build, 97 app): ADVISORY mails that night, BEHIND weekly, a source down two nights mails. BUILT and live-proven 2026-10-26: the test mail reached the mailbox (CI job 2407); the seed run (job 2408) recorded 142 open notices on 30 pins without mailing and committed its memory itself. Spike: `audits/security-watch-2026-10-26/SPIKE.md`; gaps: R-1307, R-1308, R-1309 | OPEN — built; closes after a week of nightly runs with the key (watch the Monday summary and one real ADVISORY mail) | none — the GitHub key is in place since 2026-10-27 (`GH_ADVISORY_TOKEN`, store and CI; its own token, not Renovate's): a dry CI run with it (job 2415) read every repo's advisories in one night (68 GitHub requests, 38 ADVISORY, 1 NOT CHECKED = one Docker Hub timeout). The first real night (job 2413, no key yet) mailed nothing: 159 silent first reads of notices the seed could not read, memory committed by the job. Logs: `audits/security-watch-2026-10-26/ci-night1-2026-10-27.log`, `ci-dry-with-key.log` | read the nightly logs to 2026-11-03; close on the Monday summary and one real ADVISORY mail | CC |
|
||||
| **R-1139** | Security & access | P2 | **Tester 2's break-glass root password was not rotated with the others (R-954, operator decision D1 = A, 2026-10-17): the box has been offline since 2026-10-04.** demo-hp, Tester 1 and demo-felhom were rotated and proven on 2026-10-17 (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | OPEN — filed 2026-10-17 **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a fresh install sets and vaults a new console password).** | Tester 2 reporting again | When it reports: the installer's step 4b alone (`P3/rotate-step4b.sh` shape), then the vaulted value logs in (PVE API 200) and the old one is refused | CC |
|
||||
| **R-1305** | Security & access | P2 | **The file manager on every box (FileBrowser Quantum 1.5.6-stable) is inside 36 published notices; 2 are fixed in our own channel (1.5.7-stable)** — GHSA-qx86-4v5r-26g5 (high: with the victim's password, an anonymous request re-enrolls TOTP and skips the second factor; our households have no second factor there, so low for us) and GHSA-3846-gh75-gp3m (medium: a public upload share replaces files). Most others name fixes only on the 2.x line. `files.<domain>` is public through the tunnel (behind the controller's forwardAuth, password sign-in on). Found by the first security-watch run (R-1303). `audits/security-watch-2026-10-26/ADVISORIES-2026-10-26.md` | **NARROWED 2026-10-27 — the two notices fixed in our channel are RELEASED: controller v0.346.1 pins `gtstef/filebrowser:1.5.8-stable` (1.5.7-stable skipped: it broke existing TOTP sign-ins), global floor 15:50:15Z; live on Tester 1, demo-hp, demo-felhom and 9202 (sign-in walk: listing 200, 401 without a cookie; R-1281 probe: an app container cannot reach it) — `audits/security-app-updates-2026-10-27/part5/`. LEFT: triage the 34 other notices against the 1.5.x line.** Earlier: OPEN — filed 2026-10-26 | a controller release | raise `FileBrowserImage` to 1.5.7-stable in the next controller release; triage the 34 others against 1.5.x (which apply to the stable line) | CC |
|
||||
| **R-1305** | Security & access | P2 | **The file manager on every box (FileBrowser Quantum 1.5.6-stable) is inside 36 published notices; 2 are fixed in our own channel (1.5.7-stable)** — GHSA-qx86-4v5r-26g5 (high: with the victim's password, an anonymous request re-enrolls TOTP and skips the second factor; our households have no second factor there, so low for us) and GHSA-3846-gh75-gp3m (medium: a public upload share replaces files). Most others name fixes only on the 2.x line. `files.<domain>` is public through the tunnel (behind the controller's forwardAuth, password sign-in on). Found by the first security-watch run (R-1303). `audits/security-watch-2026-10-26/ADVISORIES-2026-10-26.md` | **NARROWED 2026-10-27 — the two notices fixed in our channel are RELEASED: controller v0.346.1 pins `gtstef/filebrowser:1.5.8-stable` (1.5.7-stable skipped: it broke existing TOTP sign-ins), global floor 15:50:15Z; live on Tester 1, demo-hp, demo-felhom and 9202 (sign-in walk: listing 200, 401 without a cookie; R-1281 probe: an app container cannot reach it) — `audits/security-app-updates-2026-10-27/part5/`. LEFT: triage the 34 other notices against the 1.5.x line.** **2026-10-28 (R-1332): HTTP2Shell does not reach it** — 1.5.8-stable carries Alpine `brotli-libs` 1.2.0-r1 (2.0.2-stable: 1.2.1-r0, a new main version), but the Go program does not link libbrotli (`ldd`: musl only; no process maps it, all three boxes) — the library is there for libcurl. NOT a step today. Earlier: OPEN — filed 2026-10-26 | a controller release | raise `FileBrowserImage` to 1.5.7-stable in the next controller release; triage the 34 others against 1.5.x (which apply to the stable line) | CC |
|
||||
| **R-1042** | Security & access | P3 | **The boxes' Cloudflare keys cannot change zone settings, so Cloudflare's own „Always Use HTTPS” cannot be switched on.** Measured 2026-10-14 on all three zones (`audits/https-only-2026-10-14/``1A-1-zones-and-keys.txt`): each `cf_api_token` is valid and sees only its own zone, but GET/PATCH `settings/always_use_https`, `ssl`, `security_header` answer 403; the redirect and transform rule phases too. The https-only fix therefore lives on the box (decision 246). A second layer at Cloudflare needs „Zone Settings: Edit” (and, for headers, „Transform Rules”) added to each customer's key — the hub's zone-reach check stays. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | a key permission only the operator can grant | Optional: widen each customer key, then switch „Always Use HTTPS” on per zone | operator |
|
||||
| **R-1242** | Security & access | P3 | **The demo NAS export `/mnt/5_hdd/felhom-data` → 192.168.0.162 (demo-felhom) is a sub-directory of DooPlex's sda1 with `no_subtree_check`, so a compromised demo-felhom host could walk the whole disk as uid 101000 and read every world-readable file there (the GPG secrets exports are 0644, every repo, the public keys).** Measured 2026-10-23 (`audits/plex2shell-felhom-impact-2026-10-23/` Part 1; `exportfs -v`: `all_squash,anonuid=101000`). 0600 files (the signing keys) stay out of reach. Not exercised. | OPEN | a DooPlex host change (fence) | Give the demo share its own filesystem, or `subtree_check`; or remove other-read from `/mnt/5_hdd/backup/secrets/exports` | operator |
|
||||
| **R-1104** | Security & access | P3 | **Check https-only on Tester 2 when it reports again.** R-1033 (plain http on every public name) is closed on the three boxes that report (35 names → 301, 2026-10-16). Tester 2 (`sajatfelhom.hu`) has been off since 2026-10-04 on controller 0.292.0; its addresses answer Cloudflare 530 (tunnel down), so nothing is served in plain http while it is off. When it comes back it takes the floor (≥ 0.312.0 carries the redirect). | OPEN — filed 2026-10-16 | Tester 2 reporting again | Run `scripts/https_only_check.py` with Tester 2 in it; close | CC |
|
||||
| **R-1141** | Security & access | P3 | **`felhom-host-install --rotate-recovery` exists only as part of a full installer re-run, which also reinstalls the vouched agent and the root files.** On a box running a newer agent than the vouched one (demo-hp on the candidate, 2026-10-17) it would move the agent back; on a household box it is a much larger act than a password change. The 2026-10-17 rotation therefore ran the installer's own `step_break_glass` ALONE (`audits/agent-allowlist-rotation-2026-10-17/P3/`). | **NARROWED 2026-10-18 — built on `main`, not published.** `felhom-host-install.sh --rotate-recovery-only` runs step 4b alone (felhom.eu `7d64038f`): reads the hub, host id and host key from the agent config, refuses byo and every other mode flag, asks the hub's `/healthz` before the password changes, never touches the agent or root files. The default install path is unchanged in behaviour (harness verdicts and 9 transcripts identical old vs new; demo-hp `--preflight-only` identical; `--rotate-recovery-only --dry-run` on demo-hp changed nothing). `step_break_glass` moved, byte-identical, ahead of the argument parse. `break-glass.md` §5 points at it. Publishes only with the next `installer-v…` tag. | the next installer tag | Cut the next `installer-v…` tag (with the other untagged installer changes); close | CC |
|
||||
| **R-1258** | Security & access | P3 | **ep0's iperf3 line-test server takes any tunnel peer's test: one box (root on its host) can keep it busy for the fleet or ask a long reverse test that fills ep0's uplink.** Found 2026-10-24 (SEC-2 X2): `felhom.eu/scripts/ep0-iperf3/felhom-iperf3.service:17` — no authorized users, no server-side duration or rate cap; one client at a time (`felhom-agent/internal/linetest/iperf.go:263-264` → `busy`; 6 tries a day). Static (no ep0 change in this run). | OPEN | — | per-box iperf3 users or server-side caps, tried on a scratch VM first | CC |
|
||||
| **R-1307** | Security & access | P3 | **Packages inside our own images are not watched** — the Debian packages of the controller image, the Alpine packages of the hub and Samba images (Samba's version is whatever Alpine 3.24 ships). The security watch (R-1303) reads only the base image's point release (BEHIND); Debian's DSAs and Alpine's secdb are not read. `audits/security-watch-2026-10-26/SPIKE.md` „cannot see" | OPEN — filed 2026-10-26 | — | add Alpine secdb (machine-readable JSON per release) and Debian's tracker for the packages the images install, or a scanner run on the built images in CI | CC |
|
||||
| **R-1311** | Security & access | P3 | **Termix (catalog 2.8.0): a stranger can get round the login rate limit by writing their own X-Forwarded-For (GHSA-7j7m-85p4-2cpm), and our template keeps that header** (docmost and paperless strip it, R-753). The upstream fix needs Termix's big step (2.8 → 2.9.2 → 26.10.5). No box runs Termix (hub 2026-10-27). `audits/security-app-updates-2026-10-27/TRIAGE.md` **2026-10-27: Termix's big step (2.8.0 → 2.9.2 → 26.10.5, two hops — 2.9 must start once; calendar versions; guacd dropped) was not done today; this row carries it.** | OPEN — **RULED D58 = wait for the big step; the header is not stripped** (operator 2026-10-27 21:33, `09` §3 decision 274). — filed 2026-10-27 | the operator: a template change (strip the header like R-753, so every visitor shares the tunnel's one address and the limit can lock the household out — the Grimmory shape, R-775) or wait for the big step | the pick in `PICKS.md`; or the big step | CC |
|
||||
| **R-1311** | Security & access | P3 | **Termix (catalog 2.8.0): a stranger can get round the login rate limit by writing their own X-Forwarded-For (GHSA-7j7m-85p4-2cpm), and our template keeps that header** (docmost and paperless strip it, R-753). The upstream fix needs Termix's big step (2.8 → 2.9.2 → 26.10.5). No box runs Termix (hub 2026-10-27). `audits/security-app-updates-2026-10-27/TRIAGE.md` **2026-10-27: Termix's big step (2.8.0 → 2.9.2 → 26.10.5, two hops — 2.9 must start once; calendar versions; guacd dropped) was not done today; this row carries it.** **2026-10-28 (R-1332): the HTTP2Shell runtime fix rides the same big step** — 2.8.0 carries Node.js 26.9.0, 2.9.2 still Node.js 26.10.0 (bad), 26.10.5 Node.js 24.22.1 (read from the images). | OPEN — **RULED D58 = wait for the big step; the header is not stripped** (operator 2026-10-27 21:33, `09` §3 decision 274). — filed 2026-10-27 | the operator: a template change (strip the header like R-753, so every visitor shares the tunnel's one address and the limit can lock the household out — the Grimmory shape, R-775) or wait for the big step | the pick in `PICKS.md`; or the big step | CC |
|
||||
| **R-955** | Security & access | P4 | **The cross-box check of the hub API is done in source and tests but not fully proven: if a later check masks a missing id check, a box's key could read or change another box's hub record without any test noticing.** Read 2026-10-12: every box-facing `/api/v1` route takes the box from its own key or refuses a different id; 14 routes already had a refusal test, 5 got one (`crossbox_refusal_test.go`). Left: red-prove the 4 new tests that were not red-proved (as the restore-directive one was), and the live read-only check with one demo box's key against the other demo box's id. | OPEN — filed 2026-10-12 by the security review | — | Red-prove the 4 tests; run the live GET check on the two demo boxes | CC |
|
||||
| **R-924** | Security & access | P2 | **The operator signing keys exist only on DooPlex, with no passphrase and in no backup — a loss of DooPlex means no box ever accepts a signed agent update, config bundle or OS step again without an on-site re-enrollment.** MEASURED 2026-10-09: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519` (0600 `kisfenyo`); `ssh-keygen -y -P ''` opens all three (no passphrase); `/mnt/5_hdd/felhom.eu` is in no backup set, the off-site job does not carry them. `04` §1 says the operational key is "passphrase-protected" and the recovery key "kept offline" — today neither holds (custody ruling 1 of 2026-09-16 put both on DooPlex for this phase; `04` §3.1). Both keys share one fate, so the cold key cannot do its one job (re-pin after losing the operational key). `runbooks/total-loss-of-dooplex.md` S6. | NARROWED — **2026-10-23 (D32):** S6a/S6b are on the current sheet. LEFT: the operator prints it. **NARROWED 2026-10-09 — operator ruling: option A** (print both AND copy them). The three key files and their public halves ride the nightly encrypted DooPlex→ep0 copy (`scripts/dooplex-offsite/`, a missing key refuses the push); the Sunday restore test proves each private key still derives its own public half (never printed). **LEFT:** the operator prints S6a/S6b (`runbooks/break-glass-sheet.md`); the keys still have no passphrase (custody ruling 1 of 2026-09-16 stands until the first sale). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G2):** the three key files are on `sda1`, the same disk as DooPlex's local backup tree; their only off-box copy is the S1-encrypted nightly one. Passphrase: unchanged by the custody ruling. | operator: print | Operator: print the two keys; then close | operator |
|
||||
| **R-132** | Security & access | P3 | **`curl -w '%{redirect_url}'` reconstructs the request URL WITH its basic-auth credential** — so a `-u ":$HUB_PW"` call that never put the password in a URL still printed it **Merged 2026-10-05 from R-350 (duplicate):** (1) 2026-08-20 occurrence: POST /configuration/artifacts answers 303; leak lives only in the CC transcript under ~/.claude/projects/ on DooPlex, not in git/evidence (checked then). (2) `-v` and `--libcurl` also re-render the credential, not only %{redirect_url}; confirm redirects with %{http_code} + follow-up GET. (3) Rotation path: hub /configuration form (current_password/new_password/confirm_password); DB override wins over ConfigMap (break-glass); CC can rotate file-to-file without printing (operator-present-one-time-secrets) if asked. | **NARROWED 2026-10-20 — the rotation is DONE:** the operator changed the hub password on 2026-10-15 and the `HUB_PW` line is gone (`audits/hub-tokens-2026-10-15/C153-live.txt:18`; CC uses its own token since hub 0.152.0). **Left, the operator's:** should the break-glass `password_hash` in `manifests/hub.yaml` (a public repository) be emptied or replaced? The DB override wins at login, so the old hash no longer opens the hub. **WAITING-ON-OPERATOR** — **ACTION: rotate `HUB_PW`** **Folded R-580 2026-10-03** (the same `curl -w %{redirect_url}` credential echo, seen again 2026-09-18). **Note 2026-10-12 (security review):** `manifests/hub.yaml` also carries the hub login's bcrypt hash in this public repository (the break-glass field); when `HUB_PW` is rotated, decide whether that field should stay empty. | the operator's word on the `manifests/hub.yaml` hash field | Happened on 2026-07-31 while red-proofing the R-120 gate: the hub operator password was written to the session transcript by the write-out format, not by the request. `-u` is safe; the *reporting* was not. Rule: read the redirect from `-D -` and grep `^Location:`, never `%{redirect_url}`, on any authenticated call. Rotate the hub password (`/configuration` → Login password; ConfigMap `auth.password_hash` is the reset path) and update `~/.config/credentials` | Viktor |
|
||||
@@ -303,6 +304,7 @@ stopping line that lies.
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-1262** | Monitoring & notifications | P3 | **The upload watch treats ANY viewer as an explanation for any amount of upload, so a hacked Plex can send freely while one family member watches.** Found 2026-10-24 (SEC-1 X1): `felhom-controller/controller/internal/upwatch/watch.go:275,299`; `08` §9 states the rule, its limits list did not name this case. | OPEN | — | propose a cap (viewers × a bitrate ceiling) for the operator's R-1244 design; `08`'s limits list names the case meanwhile | CC |
|
||||
| **R-1310** | Monitoring & notifications | P3 | **The security watch (R-1303) flags notices that do not apply and misses fixes that do — measured by reading all 38 flagged pins by hand on 2026-10-27.** False: advisories of ecosystem `actions` (Home Assistant's CI), of another package in the repo (Gitea's runner), with a hostname or ≤-old range (cal.com 2023/2024, Grafana 2022), for the desktop/Electron client (Termix), for the rootless image only (Uptime Kuma); a release hit on a word (Dawarich „no longer receives security fixes", Rallly v4.16.0). Wrong „no fix": Vikunja's six (range `<= 2.6.0`, fixed in 2.7.0), karakeep-chrome (154/156 exist). Missed: Tandoor 2.7.0 names 8 GHSA ids (1 matched), Dawarich 1.16.0 (2FA, export deletion), Wishlist v0.69.0 (critical, no GHSA). BEHIND reads linuxserver `26.09.20260929` as newer than 26.09.2; Seerr follows the renamed-away `fallenbagel/jellyseerr`. `audits/security-app-updates-2026-10-27/TRIAGE.md` NOT-A-FINDING list and `triage-notes/` | OPEN — filed 2026-10-27; **two shapes FIXED the same day** (ecosystem `actions` not listed; an empty fix with a closed range reads „after X"; test W9, red first) | — | teach `security_watch.py`: skip a package that is not the pinned one; match every GHSA id a release body names; ignore `YY.MM.YYYYMMDD` tags; repoint seerr to `seerr-team/seerr`; each with a test seen failing | CC |
|
||||
| **R-1334** | Monitoring & notifications | P3 | **Docker Hub's tag API refuses the watch (HTTP 429) since the DooPlex group was added: 19 DooPlex images NOT CHECKED in the 2026-10-28 night run (job 2446, `hub.docker.com` = 211 requests), and the „could not check registry:docker.io" mail went out (streak 2).** The runtime part is not affected (HEAD + cache). ghcr.io also answered one blob with 429 in the dry run of 07:02 (Immich v3.3.5, while this session read images from the same address). | OPEN — filed 2026-10-28 (runtime fixes) | — | fewer tag-list calls (one list per image across groups, cached by day) or a Docker Hub login in CI (an Actions secret — the operator's); then a night with no Docker Hub 429 | CC |
|
||||
| **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
|
||||
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | **WATCHING — DESIGN WRITTEN 2026-10-26 night (the brief's Part 7: build only if time is left; it was not built — the line test and the attended release came first).** The host-side counter EXISTS and is readable (read on demo-hp 2026-10-26): `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events`, line `oom_kill N`. **Design:** (1) agent: a read-only collector (no root CLI — a sysfs read under the agent's own rights; if not readable, the existing sudoers fence gets one read-only `Cmnd_Alias`) that, for each guest in the `felhom` pool, globs those scopes and keeps `{container_id: oom_kill}`; served on the guest's own local API as `GET /guest/oom-kills` (`withGuest`, the guest's containers only). (2) controller: the 30-s dead-app scan asks once per pass; a RISE of a container's counter = an OOM kill whatever Docker's `OOMKilled` says → the existing `app_oom` line (R-636's per-run grain), `source: cgroup` in the details. An agent without the route = today's behaviour. (3) Tests: a fake cgroup tree (agent), a rising counter → one `app_oom` (controller, red-first). (4) Live: 9202, a throwaway app capped at 64M with a memory hog: the counter rises, `app_oom` names the container while `OOMKilled` reads false. ~1 day, two repos, one agent release. **Earlier:** WATCHING — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC** **2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes tried: a child process killed while the container kept running (`oom_kill` 0 → 3, flag true) and three main-process kills (exit 137, flag true); an `oom` event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. `audits/design-build-2026-10-06/`C/. **2026-10-06 18:24: operator ruling, option A (decision 157):** nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. **Added:** a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). **2026-10-06 (night): the Docker-approval memory-kill check is BUILT** — hub v0.140.0 (LIVE: the approval waits for a Line truncated
|
||||
| **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Design note (2026-10-21, after R-1192's diagnosis): the freeze's signature is cheap to see from the HOST side — the guest cgroup's memory PSI „full" above ~50 % for a minute, or its `memory.events max` climbing, while the host still reports; the agent could raise `guest_memory_thrash` (operator mail in its own words, not „no report for 45 m") and the household banner „A dobozod most nagyon lassú…". With the swap fix (R-1192) the measured thrash ends within a minute, so this alarm should be built only if a freeze is seen again after the agent release; until then it stays a design. Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC |
|
||||
@@ -322,7 +324,6 @@ stopping line that lies.
|
||||
| **R-1224** | Monitoring & notifications | P4 | **The operator's 1-hour mail cooldown for `wg_tunnel_stale` is per customer, not per host, and `wg_tunnel_ok` does not end it.** Fresh review 2026-10-23 night (`REPORT-night-2026-10-23.md`, H3): `notify/dispatcher.go:620` keys the cooldown `customerID:eventType`. Two hosts of one household going stale within an hour, or stale → ok → stale, store a second event row but mail once (the suppression is in `notification_log`). | OPEN | — | Key the cooldown by `host_id` from the details for host-scoped operator events, and let the matching ok event end it; a dispatcher test with two hosts; rides a hub deploy | CC |
|
||||
| **R-1233** | Monitoring & notifications | P4 | **`[ERROR] [stacks] execCommand failed: exit status 1` names no command.** Seen once a night (2026-10-21 00:30:49 and 2026-10-23 02:17:14 UTC, each right after an app stopped for its dump); the command and stderr go only into the returned error, which the caller may drop (`internal/stacks/manager.go:1757`). An ERROR nobody can judge. | OPEN | — | Log the command's first two words (`docker inspect`, never the arguments — `docker exec … psql` can carry a password) and the stderr's first line; find the caller of the nightly one | CC |
|
||||
| **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go` `parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC |
|
||||
| **R-1327** | Monitoring & notifications | P2 | **The security watch could not see a bug in the RUNTIME under an app — HTTP2Shell (CVE-2026-111497, the Brotli decoder in Node.js and Python's `brotli`/`brotlicffi`, 2026-10-21) never reaches an app's own notices, so Home Assistant and RomM ran the bad code on the test boxes unseen.** Measured 2026-10-27 from the images (`scripts/image_runtime.py`, registry config + layers, no pull): Home Assistant 2026.9.4 Python Brotli 1.2.0 (2026.10.4: 1.2.1); RomM 5.3.1 and 5.4.1 Node 24.18.1 (fixed 24.22.1); Immich v3.2.4 Node 24.21.0 (v3.3.5: 24.22.1); cal.com v6.2.0 Node 20.20.0; Paperless 2.20.15 Brotli 1.1.0 — each matching a live container where one runs. | **VERIFY** — BUILT 2026-10-27 night (commit only): the watch reads every pin's runtime (`security_watch_runtimes.json`: Node, Python Brotli, brotlicffi, Alpine brotli-libs for our own images), an ADVISORY with source „runtime", cached by image digest in `state.json` (`runtime_cache`) — CI has no zstd, so the cache is filled on DooPlex; a new zstd image in CI is NOT CHECKED with the way to fill it. Tests W10 (W1–W12: 58 checks), red-proved. Not in the table: Debian `libbrotli1` (Debian 13's fixed version not known tonight). Waits for: the first CI night naming Home Assistant and RomM. | — | CI's first nightly run lists HA and RomM as ADVISORY (runtime) and audiobookshelf as not; then close | CC |
|
||||
| **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list | CC |
|
||||
|
||||
## Hub & operator — 12 rows (P3 3, P4 9)
|
||||
|
||||
Reference in new issue
Block a user