Plex2Shell: the operator's 10:56 ruling in CONTEXT; demo-hp 9201 checked (clean) and updated to 1.43.5.11029; register 184 -> 186 (opened R-1238 Plex test record/fixture, R-1239 a fast lane for security fixes = D39); evidence plex-demo-boxes-2026-10-23
gates / gates (push) Successful in 13m46s
gates / gates (push) Successful in 13m46s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012qRErfCoiTkvDK9N5XHbzb
This commit is contained in:
adminandClaude Opus 5.5 committed
1 parent
2c9152d912
commit
4b9b0af3cb
14 files changed
+518
-1
No files matched your search
@@ -0,0 +1,81 @@
|
||||
# Plex to the fixed version on the demo/test boxes — 2026-10-23 (Plex2Shell)
|
||||
|
||||
Operator ruling 2026-10-23 10:56 (recorded in `CONTEXT.md`): move the catalog's Plex to the newest fixed version at once,
|
||||
without the test record of `09` §3 decision 13, and put the demo boxes on it. DooPlex and ep0 were not touched; the
|
||||
incident session handles DooPlex.
|
||||
|
||||
## The catalog move
|
||||
|
||||
- Catalog `6cc317d`: `plexinc/pms-docker:1.41.4.9463-630c9f557` → `1.43.5.11029-6bf8af3f0` (Docker Hub, pushed
|
||||
2026-10-20; manifest list `sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f`, re-resolved at
|
||||
start — still the newest plain `X.Y.Z.B-hash` tag; `public`/`beta`/`plexpass` moved on 2026-10-22 but carry no plain
|
||||
tag and are excluded). `catalog_since: "2026-10-23"`.
|
||||
- Gates green: image-pins, catalog-since (range), test-record, image-resolvable (plex), `catalog_gates.py --fast plex`.
|
||||
`check-test-record-move.py --range ce33e3b..6cc317d` refused, as expected (verbatim in `app-catalog-felhom.eu/REPORT.md`).
|
||||
Pushed with `--no-verify`, that commit only.
|
||||
- Read back from the box's catalog cache on demo-hp 9201 after one sync: the new tag, `catalog_since: "2026-10-23"`,
|
||||
HEAD `6cc317d`. The badge went to „Frissítés elérhető — ma" / „Update available — today" in 4.7 s.
|
||||
|
||||
## Which boxes run Plex
|
||||
|
||||
Hub `/hosts` (CC token, read only): four hosts — Tester4 (DOWN), demo-felhom, demo-hp, tester-1. The customer
|
||||
**Tester 2 has 0 hosts**, so the operator's question was not needed. The hub's `/apps` page counts 2 Plex deployments;
|
||||
the boxes say ONE runs it (a second channel: `docker ps -a` in every running guest):
|
||||
|
||||
| Box | Guest | Plex |
|
||||
|---|---|---|
|
||||
| demo-hp | 9201 | yes, 1.41.4.9463-630c9f557 |
|
||||
| demo-hp | 9202 (scratch) | no |
|
||||
| demo-felhom (N100) | 9201 | no |
|
||||
| tester-1 (VM 341 on demo-hp) | 9201 | no |
|
||||
| Tester4 | — | the hub host is DOWN and its VM no longer exists on demo-hp or felhom-pve (`qm list`); nothing to update. The hub's second Plex deployment is most likely this stale host's last report (not proven). |
|
||||
|
||||
## demo-hp 9201 — the signs check (before the update, read only) → **NOT-A-FINDING**
|
||||
|
||||
Scripts: `demo-hp-9201/signs.sh`, `demo-hp-9201/signs2.sh` (run inside the guest). Outputs, e-mail and token redacted:
|
||||
`demo-hp-9201/signs-9201-before-update.txt`, `demo-hp-9201/signs2-9201-before-update.txt`.
|
||||
|
||||
| Sign | Result | Control |
|
||||
|---|---|---|
|
||||
| CPU, last days | whole guest 0.23–0.34 cores average of 7, max 1.04, per day 10-16 → 10-23 (PVE rrd, the HOST's channel). A miner would hold several cores flat. | — |
|
||||
| Processes in the container | only s6, Plex Media Server, Plex Script Host, Plex Tuner Service | — |
|
||||
| `xmrig`, `mylovelyminer`, `x12345`, `myawesomeproxy` anywhere in the guest | none | — |
|
||||
| ELF files under `Codecs` | none (one file, `.device-id`; one empty codec dir from 2026-10-15) | the ELF test says ELF for `/usr/bin/docker`, not for `Preferences.xml` |
|
||||
| ELF > 10 kB in the config volume newer than 2026-10-15 | none | same |
|
||||
| Container layer (`docker diff plex`) | 80 entries, all s6 runtime under `/run` | — |
|
||||
| `TranscoderH264Options` / `…Override` in `Preferences.xml` | absent | the attribute list printed (27 names) |
|
||||
| Log lines `searchOne`, `/video/:/transcode/universal`, `?x=http…` | 0, 0, 0 | 633 `Request: [` lines in the same log |
|
||||
| Open connections from the container | plex.tv / Cloudflare 443 and the box's traefik only | — |
|
||||
|
||||
**Limit, stated:** Plex keeps 6 rotated logs and restarted several times in the night, so the logs reach back only to
|
||||
2026-10-23 00:32 UTC — not to 2026-10-16. The 10-16 → 10-22 window is covered by the CPU history and the files only,
|
||||
not by the logs. The container was recreated at 04:39 UTC, so its layer has no history before that either.
|
||||
|
||||
## The update (the product's Update press, `POST /api/stacks/plex/update`)
|
||||
|
||||
`demo-hp-9201/update-press.json`: accepted 202; phases safety-dump → pulling → copying → verifying → done in 28.4 s;
|
||||
no error, no hold. After (`demo-hp-9201/after-update.txt`):
|
||||
|
||||
- `/identity`: version `1.43.5.11029-6bf8af3f0`, `claimed: true`, the SAME `machineIdentifier` as before.
|
||||
- Library: 0 sections before and after (this Plex never had a library; `demo-hp-9201/library-before-update.txt`). The
|
||||
server's identity and claim are what carried over.
|
||||
- Box walk: every container healthy (traefik has no health check, up), none restarting; 0 `[ERROR]` and 0 `[WARN]` lines
|
||||
in the controller log since the press (control: 99 `[INFO]` lines in the same window); badge „Naprakész" / „Up to date";
|
||||
pinned = installed = catalog = live compose = the running container, all the new tag.
|
||||
|
||||
## The guide (Part 3)
|
||||
|
||||
The guide's labels were read from the new image's own web client (`WebClient.bundle/.../translations/hu.json`, plex-web
|
||||
4.162.1; its keys are the English labels): Home/„Kezdőképernyő", Play/„Lejátszás", Scan Library Files/„Könyvtárfájlok
|
||||
beolvasása", More/„Több", Fix Match/„Találatok pontosítása", Settings/„Beállítások", Manage Library Access/„Könyvtár
|
||||
hozzáférés kezelése", and the FAQ's Video Quality/„Videó minőség", Original/„Eredeti" — all present, all equal
|
||||
(`demo-hp-9201/guide-labels-1.43.5.txt`; ASCII-fragment controls: `beolvas` 5, `kezel` 26, `xqzzyw` 0). The box's own
|
||||
Plex wiring (`POST /api/storage/media/wire`, the household's „connect now") answered `plex: unchanged`, no error, against
|
||||
the new Plex (`demo-hp-9201/guide-wiring-after-update.txt`). **Not exercised:** the wiring's PUT that adds a new
|
||||
drive's folder — this box has one drive and no library, so it had nothing to add. `guide.checked_version` moved.
|
||||
|
||||
## Teardown
|
||||
|
||||
- Machine: Plex stays on the new version (that is the goal). The helper scripts were removed from the guest and host.
|
||||
- Host: nothing provisioned.
|
||||
- Hub: nothing changed (reads only).
|
||||
@@ -0,0 +1,57 @@
|
||||
identity: {"MediaContainer":{"size":0,"apiVersion":"1.2.3","claimed":true,"machineIdentifier":"fb56097105b927dd42576cd879499a50c064c9fc","version":"1.43.5.11029-6bf8af3f0"}}
|
||||
sections: 0
|
||||
---containers
|
||||
adventurelog Up 5 hours (healthy)
|
||||
adventurelog-frontend Up 5 hours (healthy)
|
||||
adventurelog-postgres Up 5 hours (healthy)
|
||||
bentopdf Up 5 hours (healthy)
|
||||
bookstack Up 5 hours (healthy)
|
||||
bookstack-db Up 5 hours (healthy)
|
||||
calibre-web Up 5 hours (healthy)
|
||||
cloudflared Up 2 days (healthy)
|
||||
docmost Up 4 hours (healthy)
|
||||
docmost-postgres Up 5 hours (healthy)
|
||||
docmost-redis Up 5 hours (healthy)
|
||||
felhom-controller Up 3 hours (healthy)
|
||||
filebrowser Up 2 days (healthy)
|
||||
home-assistant Up 4 hours (healthy)
|
||||
opengist Up 4 hours (healthy)
|
||||
paperless-postgres Up 4 hours (healthy)
|
||||
paperless-redis Up 4 hours (healthy)
|
||||
paperless-webserver Up 4 hours (healthy)
|
||||
plex Up 48 seconds (healthy)
|
||||
privatebin Up 4 hours (healthy)
|
||||
romm Up 4 hours (healthy)
|
||||
romm-db Up 4 hours (healthy)
|
||||
romm-redis Up 4 hours (healthy)
|
||||
traefik Up 30 hours
|
||||
---not-healthy
|
||||
---restarting
|
||||
---ctl-errors-since-update
|
||||
0
|
||||
---ctl-plex-lines
|
||||
2026/10/23 09:08:22 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=true composePath=/opt/docker/stacks/plex/docker-compose.yml
|
||||
2026/10/23 09:08:22 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 4s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/23 09:08:22 backup.go:526: [DEBUG] groupStacksByDrive: /mnt/felhom-drives/hdd_1 → [calibre-web, paperless-ngx, plex, romm]
|
||||
2026/10/23 09:08:23 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: e94565919cd7 plex plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0
|
||||
f8900ef8917f felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.340.0
|
||||
2026/10/23 09:08:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container plex (image=plexinc/pms-docker:1.43.5.11029-6bf8af3f0, not a database)
|
||||
2026/10/23 09:08:23 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.340.0, not a database)
|
||||
2026/10/23 09:08:23 sync.go:559: [DEBUG] [sync] plex/docker-compose.yml: hash match, skipped
|
||||
2026/10/23 09:08:23 sync.go:559: [DEBUG] [sync] plex/.felhom.yml: hash match, skipped
|
||||
2026/10/23 09:08:23 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/plex/docker-compose.yml
|
||||
2026/10/23 09:08:23 recovery_unit.go:248: [INFO] [backup] plex: the app now runs [plexinc/pms-docker:1.43.5.11029-6bf8af3f0]; the unit keeps the definition of its data ([plexinc/pms-docker:1.41.4.9463-630c9f557], written 2026-10-23T02:16:55Z) until the next backup replaces the data
|
||||
2026/10/23 09:08:23 recovery_unit.go:289: [INFO] [backup] Recovery unit captured for plex → /mnt/felhom-drives/hdd_1/backups/primary/plex (images=1, secrets-referenced=0, data_keys=0, portable-carried=0/0, withheld=0)
|
||||
2026/10/23 09:08:24 auth.go:144: [DEBUG] [web] auth: valid session for GET /apps/plex
|
||||
2026/10/23 09:08:24 server.go:637: [DEBUG] [web] ServeHTTP: GET /apps/plex from 172.18.0.10:45638
|
||||
2026/10/23 09:08:24 handlers.go:421: [DEBUG] [web] deployPageData: stack=plex method=GET
|
||||
2026/10/23 09:08:24 auth.go:144: [DEBUG] [web] auth: valid session for GET /apps/plex
|
||||
2026/10/23 09:08:24 server.go:637: [DEBUG] [web] ServeHTTP: GET /apps/plex from 172.18.0.10:45638
|
||||
2026/10/23 09:08:24 handlers.go:421: [DEBUG] [web] deployPageData: stack=plex method=GET
|
||||
2026/10/23 09:08:24 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/stacks/plex
|
||||
2026/10/23 09:08:24 router.go:78: [DEBUG] [api] GET /api/stacks/plex (path=/stacks/plex)
|
||||
2026/10/23 09:08:25 logscanner.go:69: [DEBUG] [metrics] logscanner: scanned plex: errors=0 warnings=0 issues=0 (took 28ms)
|
||||
2026/10/23 09:08:32 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 13s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/23 09:08:42 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 23s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/23 09:08:52 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 33s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/23 09:09:02 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping plex — last check 43s ago, effective interval 5m0s, healthy=true
|
||||
@@ -0,0 +1,10 @@
|
||||
Plex web client translations/hu.json from plexinc/pms-docker:1.43.5.11029-6bf8af3f0 (WebClient plex-4.162.1), keys = the English UI string
|
||||
EN key 'Home': present=True; HU in Plex='Kezdőképernyő'; guide HU='Kezdőképernyő'; match=True
|
||||
EN key 'Play': present=True; HU in Plex='Lejátszás'; guide HU='Lejátszás'; match=True
|
||||
EN key 'Scan Library Files': present=True; HU in Plex='Könyvtárfájlok beolvasása'; guide HU='Könyvtárfájlok beolvasása'; match=True
|
||||
EN key 'More': present=True; HU in Plex='Több'; guide HU='Több'; match=True
|
||||
EN key 'Fix Match': present=True; HU in Plex='Találatok pontosítása'; guide HU='Találatok pontosítása'; match=True
|
||||
EN key 'Settings': present=True; HU in Plex='Beállítások'; guide HU='Beállítások'; match=True
|
||||
EN key 'Manage Library Access': present=True; HU in Plex='Könyvtár hozzáférés kezelése'; guide HU='Könyvtár hozzáférés kezelése'; match=True
|
||||
EN key 'Video Quality': present=True; HU in Plex='Videó minőség'; guide HU='Videó minőség'; match=True
|
||||
EN key 'Original': present=True; HU in Plex='Eredeti'; guide HU='Eredeti'; match=True
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
status 200 {"data": {"pending": false, "drives": null, "apps": ["plex"], "wiring": [{"app": "folders", "state": "done"}, {"app": "plex", "state": "unchanged"}], "wired_at": "2026-10-23T09:10:30Z"}, "ok": true}
|
||||
wire 200 {"data": [{"app": "folders", "state": "done"}, {"app": "plex", "state": "unchanged"}], "ok": true}
|
||||
@@ -0,0 +1,2 @@
|
||||
identity: {"MediaContainer":{"size":0,"apiVersion":"0.1.0","claimed":true,"machineIdentifier":"fb56097105b927dd42576cd879499a50c064c9fc","version":"1.41.4.9463-630c9f557"}}
|
||||
sections: 0
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
### 1. processes in the plex container (docker top)
|
||||
PID USER ELAPSED TIME %CPU COMMAND
|
||||
1493203 root 04:25:08 00:00:00 0.0 s6-svscan
|
||||
1493262 root 04:25:08 00:00:00 0.0 s6-supervise
|
||||
1493541 root 04:25:08 00:00:00 0.0 s6-supervise
|
||||
1493544 1000 04:25:08 00:00:21 0.1 Plex Media Serv
|
||||
1493588 1000 04:25:08 00:00:15 0.0 Plex Script Hos
|
||||
1494146 1000 04:25:05 00:00:12 0.0 Plex Tuner Serv
|
||||
### 1b. suspect names anywhere in the guest
|
||||
none
|
||||
### 1c. top CPU processes in the guest
|
||||
PID USER %CPU ELAPSED COMMAND
|
||||
235 root 1.7 2-06:15:22 dockerd
|
||||
183 root 0.8 2-06:15:23 containerd
|
||||
1493858 999 0.7 04:25:07 redis-server
|
||||
1489649 999 0.6 04:25:33 redis-server
|
||||
1492193 999 0.6 04:25:19 redis-server
|
||||
1717354 root 0.3 02:26:14 felhom-controll
|
||||
1496145 root 0.3 04:24:23 gunicorn
|
||||
1698 65532 0.3 2-06:15:16 cloudflared
|
||||
1496141 root 0.3 04:24:23 gunicorn
|
||||
1492308 1000 0.2 04:25:19 node-MainThread
|
||||
1495252 1000 0.2 04:24:46 [celeryd: celer
|
||||
### 2. ELF files under Codecs (path size mtime sha256)
|
||||
all files under Codecs: 1
|
||||
total 16
|
||||
drwxr-xr-x 3 1000 1000 4096 2026-10-15 08:46:05.701113471 +0000 .
|
||||
drwxr-xr-x 11 1000 1000 4096 2026-10-23 08:39:53.737639828 +0000 ..
|
||||
-rw-r--r-- 1 1000 1000 36 2026-10-15 08:46:05.701113471 +0000 .device-id
|
||||
drwxr-xr-x 2 1000 1000 4096 2026-10-15 08:46:05.700807513 +0000 e613bce-97f23d579c1001d8e9cc0d2e-linux-x86_64
|
||||
### 2b. any ELF in the config volume outside Plex's own folders, newer than 2026-10-15
|
||||
(end 2b)
|
||||
### 3. Preferences.xml transcoder attributes
|
||||
TranscoderH264Options*: absent
|
||||
attribute names present:
|
||||
AcceptedEULA AnonymousMachineIdentifier CertificateUUID CertificateVersion DlnaEnabled FSEventLibraryPartialScanEnabled FSEventLibraryUpdatesEnabled GlobalMusicVideoPathMigrated LanguageInCloud LastAutomaticMappedPort MachineIdentifier MetricsEpoch OldestPreviousVersion PlexOnlineHome PlexOnlineMail PlexOnlineToken PlexOnlineUsername ProcessedMachineIdentifier PubSubServer PubSubServerPing PubSubServerRegion PublishServerOnPlexOnlineKey TranscoderTempDirectory customConnections encoding version
|
||||
### 4. log lines: searchOne / transcode/universal (tokens redacted)
|
||||
total 940
|
||||
drwxr-xr-x 3 1000 1000 4096 2026-10-23 04:39:29.306642337 +0000 .
|
||||
drwxr-xr-x 11 1000 1000 4096 2026-10-23 08:39:53.737639828 +0000 ..
|
||||
drwxr-xr-x 2 1000 1000 4096 2026-10-23 04:39:30.774964086 +0000 PMS Plugin Logs
|
||||
-rw-r--r-- 1 1000 1000 880 2026-10-23 04:38:23.878139505 +0000 Plex Crash Uploader.1.log
|
||||
-rw-r--r-- 1 1000 1000 561 2026-10-23 03:34:34.438792292 +0000 Plex Crash Uploader.2.log
|
||||
-rw-r--r-- 1 1000 1000 880 2026-10-23 03:33:23.603543601 +0000 Plex Crash Uploader.3.log
|
||||
-rw-r--r-- 1 1000 1000 561 2026-10-23 02:59:31.142955018 +0000 Plex Crash Uploader.4.log
|
||||
-rw-r--r-- 1 1000 1000 880 2026-10-23 02:58:23.855206791 +0000 Plex Crash Uploader.5.log
|
||||
-rw-r--r-- 1 1000 1000 561 2026-10-23 04:39:28.559376452 +0000 Plex Crash Uploader.log
|
||||
-rw-r--r-- 1 1000 1000 122590 2026-10-23 04:38:23.956828290 +0000 Plex Media Server.1.log
|
||||
-rw-r--r-- 1 1000 1000 95344 2026-10-23 03:33:23.650544218 +0000 Plex Media Server.2.log
|
||||
-rw-r--r-- 1 1000 1000 97012 2026-10-23 02:58:23.959207947 +0000 Plex Media Server.3.log
|
||||
-rw-r--r-- 1 1000 1000 96861 2026-10-23 02:38:23.762270724 +0000 Plex Media Server.4.log
|
||||
-rw-r--r-- 1 1000 1000 157007 2026-10-23 02:16:45.904202135 +0000 Plex Media Server.5.log
|
||||
-rw-r--r-- 1 1000 1000 311157 2026-10-23 09:04:06.104368626 +0000 Plex Media Server.log
|
||||
-rw-r--r-- 1 1000 1000 2769 2026-10-23 04:38:23.939140257 +0000 Plex Tuner Service.1.log
|
||||
-rw-r--r-- 1 1000 1000 2769 2026-10-23 03:33:23.635544021 +0000 Plex Tuner Service.2.log
|
||||
-rw-r--r-- 1 1000 1000 2769 2026-10-23 02:58:23.938207713 +0000 Plex Tuner Service.3.log
|
||||
-rw-r--r-- 1 1000 1000 2769 2026-10-23 02:38:23.745270485 +0000 Plex Tuner Service.4.log
|
||||
-rw-r--r-- 1 1000 1000 2769 2026-10-23 02:16:45.852201461 +0000 Plex Tuner Service.5.log
|
||||
-rw-r--r-- 1 1000 1000 1855 2026-10-23 04:39:39.105066764 +0000 Plex Tuner Service.log
|
||||
count searchOne: 0
|
||||
count transcode/universal: 0
|
||||
count lines with http(s):// in a query: 0
|
||||
oldest log line: Oct 23, 2026 02:32:13.754 [1343875524390
|
||||
@@ -0,0 +1,34 @@
|
||||
#!/bin/sh
|
||||
# Read-only Plex2Shell signs check, run INSIDE the guest. Prints no token.
|
||||
V=/var/lib/docker/volumes/plex_plex_config/_data
|
||||
P="$V/Library/Application Support/Plex Media Server"
|
||||
echo "### 1. processes in the plex container (docker top)"
|
||||
docker top plex -eo pid,user,etime,time,pcpu,comm
|
||||
echo "### 1b. suspect names anywhere in the guest"
|
||||
ps -eo pid,user,comm,args | grep -i -E 'xmrig|mylovelyminer|x12345|myawesomeproxy' | grep -v grep || echo "none"
|
||||
echo "### 1c. top CPU processes in the guest"
|
||||
ps -eo pid,user,pcpu,etime,comm --sort=-pcpu | head -12
|
||||
echo "### 2. ELF files under Codecs (path size mtime sha256)"
|
||||
if [ -d "$P/Codecs" ]; then
|
||||
find "$P/Codecs" -type f | while read -r f; do
|
||||
if head -c 4 "$f" | grep -q 'ELF'; then
|
||||
printf '%s\t%s\t%s\t%s\n' "$f" "$(stat -c %s "$f")" "$(stat -c %y "$f")" "$(sha256sum "$f" | cut -c1-64)"
|
||||
fi
|
||||
done
|
||||
echo "all files under Codecs: $(find "$P/Codecs" -type f | wc -l)"
|
||||
ls -la --time-style=full-iso "$P/Codecs"
|
||||
else echo "no Codecs dir"; fi
|
||||
echo "### 2b. any ELF in the config volume outside Plex's own folders, newer than 2026-10-15"
|
||||
find "$V" -type f -newermt 2026-10-15 -size +10k 2>/dev/null | while read -r f; do head -c 4 "$f" | grep -q ELF && printf '%s\t%s\t%s\n' "$f" "$(stat -c %s "$f")" "$(sha256sum "$f"|cut -c1-64)"; done; echo "(end 2b)"
|
||||
echo "### 3. Preferences.xml transcoder attributes"
|
||||
grep -o -E 'TranscoderH264Options(Override)?="[^"]*"' "$P/Preferences.xml" || echo "TranscoderH264Options*: absent"
|
||||
echo "attribute names present:"; grep -o -E ' [A-Za-z]+="' "$P/Preferences.xml" | tr -d ' ="' | sort | tr '\n' ' '; echo
|
||||
echo "### 4. log lines: searchOne / transcode/universal (tokens redacted)"
|
||||
ls -la --time-style=full-iso "$P/Logs/" | head -30
|
||||
for f in "$P/Logs/"Plex\ Media\ Server*.log; do
|
||||
grep -h -E 'searchOne|/video/:/transcode/universal' "$f" 2>/dev/null
|
||||
done | sed -E 's/(X-Plex-Token|PlexOnlineToken|token)=[^&" ]*/\1=REDACTED/gI' | head -200
|
||||
echo "count searchOne: $(cat "$P/Logs/"Plex\ Media\ Server*.log 2>/dev/null | grep -c searchOne)"
|
||||
echo "count transcode/universal: $(cat "$P/Logs/"Plex\ Media\ Server*.log 2>/dev/null | grep -c '/video/:/transcode/universal')"
|
||||
echo "count lines with http(s):// in a query: $(cat "$P/Logs/"Plex\ Media\ Server*.log 2>/dev/null | grep -c -E '[?&][a-zA-Z]+=https?(%3A|:)')"
|
||||
echo "oldest log line: $(head -1 "$P/Logs/Plex Media Server.5.log" 2>/dev/null | cut -c1-40)"
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
### controls
|
||||
ELF detector on /usr/bin/docker (positive): ELF
|
||||
ELF detector on Preferences.xml (negative): not-ELF
|
||||
log grep positive control (lines with 'Request: [' in current log): 633
|
||||
### docker diff plex (files changed in the container layer)
|
||||
C /run
|
||||
A /run/s6
|
||||
A /run/s6/env-stage3
|
||||
A /run/s6/etc
|
||||
A /run/s6/etc/cont-finish.d
|
||||
A /run/s6/etc/cont-init.d
|
||||
A /run/s6/etc/fix-attrs.d
|
||||
A /run/s6/etc/services.d
|
||||
A /run/s6/services
|
||||
A /run/s6/services/.s6-svscan
|
||||
A /run/s6/services/.s6-svscan/control
|
||||
A /run/s6/services/.s6-svscan/crash
|
||||
A /run/s6/services/.s6-svscan/finish
|
||||
A /run/s6/services/.s6-svscan/lock
|
||||
A /run/s6/services/plex
|
||||
A /run/s6/services/plex/event
|
||||
A /run/s6/services/plex/finish
|
||||
A /run/s6/services/plex/run
|
||||
A /run/s6/services/plex/supervise
|
||||
A /run/s6/services/plex/supervise/status
|
||||
A /run/s6/services/plex/supervise/control
|
||||
A /run/s6/services/plex/supervise/death_tally
|
||||
A /run/s6/services/plex/supervise/lock
|
||||
A /run/s6/services/plex/timeout-finish
|
||||
A /run/s6/services/s6-fdholderd
|
||||
A /run/s6/services/s6-fdholderd/down
|
||||
A /run/s6/services/s6-fdholderd/event
|
||||
A /run/s6/services/s6-fdholderd/notification-fd
|
||||
A /run/s6/services/s6-fdholderd/rules
|
||||
A /run/s6/services/s6-fdholderd/rules/uid
|
||||
A /run/s6/services/s6-fdholderd/rules/uid/0
|
||||
A /run/s6/services/s6-fdholderd/rules/uid/0/allow
|
||||
A /run/s6/services/s6-fdholderd/rules/uid/0/env
|
||||
A /run/s6/services/s6-fdholderd/rules/uid/0/env/S6_FDHOLDER_GETDUMP
|
||||
A /run/s6/services/s6-fdholderd/rules/uid/0/env/S6_FDHOLDER_LIST
|
||||
A /run/s6/services/s6-fdholderd/rules/uid/0/env/S6_FDHOLDER_RETRIEVE_REGEX
|
||||
A /run/s6/services/s6-fdholderd/rules/uid/0/env/S6_FDHOLDER_SETDUMP
|
||||
A /run/s6/services/s6-fdholderd/rules/uid/0/env/S6_FDHOLDER_STORE_REGEX
|
||||
A /run/s6/services/s6-fdholderd/rules/uid/default
|
||||
A /run/s6/services/s6-fdholderd/rules/uid/default/allow
|
||||
A /run/s6/services/s6-fdholderd/rules/uid/default/env
|
||||
A /run/s6/services/s6-fdholderd/rules/uid/default/env/S6_FDHOLDER_LIST
|
||||
A /run/s6/services/s6-fdholderd/run
|
||||
A /run/s6/services/s6-fdholderd/supervise
|
||||
A /run/s6/services/s6-fdholderd/supervise/control
|
||||
A /run/s6/services/s6-fdholderd/supervise/death_tally
|
||||
A /run/s6/services/s6-fdholderd/supervise/lock
|
||||
A /run/s6/services/s6-fdholderd/supervise/status
|
||||
A /run/s6/container_environment
|
||||
A /run/s6/container_environment/HOSTNAME
|
||||
A /run/s6/container_environment/LC_ALL
|
||||
A /run/s6/container_environment/TZ
|
||||
A /run/s6/container_environment/ADVERTISE_IP
|
||||
A /run/s6/container_environment/HOME
|
||||
A /run/s6/container_environment/LANG
|
||||
A /run/s6/container_environment/PATH
|
||||
A /run/s6/container_environment/PLEX_CLAIM
|
||||
A /run/s6/container_environment/TERM
|
||||
A /run/s6/container_environment/CHANGE_CONFIG_DIR_OWNERSHIP
|
||||
A /run/s6/container_environment/CWD
|
||||
total changes: 80
|
||||
### created / started / restarts
|
||||
created=2026-10-23T04:39:25.851280092Z started=2026-10-23T04:39:26.02448552Z restarts=0
|
||||
### transcode volume: files
|
||||
(n=0)
|
||||
### crash uploader log (latest)
|
||||
Oct 23, 2026 06:38:23.876 [134233172196152] INFO - Crash Uploader - Session Health - Status: exited StartTime: 1792726472 Duration: 3831 UserId: EMAIL-REDACTED Version: 1.41.4.9463-630c9f557
|
||||
Oct 23, 2026 06:38:23.877 [134233200109576] DEBUG - [HCl#1] HTTP requesting POST https://o17675.ingest.sentry.io/api/1233455/envelope/?sentry_key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
Oct 23, 2026 06:38:23.879 [134233171950392] WARN - [HttpClient/HCl#1] HTTP error requesting POST https://o17675.ingest.sentry.io/api/1233455/envelope/?sentry_key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (7, Couldn't connect to server) (Failed to connect to o17675.ingest.sentry.io port 443 after 2 ms: Couldn't connect to server)
|
||||
Oct 23, 2026 06:38:23.879 [134233200109576] ERROR - Failed to upload session health
|
||||
Oct 23, 2026 06:38:23.879 [134233200109576] INFO - Crash Uploader - Processing Complete.
|
||||
### first lines of each PMS log (start time)
|
||||
Plex Media Server.1.log: Oct 23, 2026 05:34:32.42
|
||||
Plex Media Server.2.log: Oct 23, 2026 04:59:29.11
|
||||
Plex Media Server.3.log: Oct 23, 2026 04:39:32.28
|
||||
Plex Media Server.4.log: Oct 23, 2026 04:16:57.38
|
||||
Plex Media Server.5.log: Oct 23, 2026 02:32:13.75
|
||||
Plex Media Server.log: Oct 23, 2026 06:39:26.55
|
||||
### outbound connections now from plex netns
|
||||
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
|
||||
CLOSE-WAIT 25 0 172.18.0.14:42388 54.156.171.78:443 users:(("Plex Media Serv",pid=1493544,fd=61))
|
||||
CLOSE-WAIT 39 0 172.18.0.14:52092 172.64.151.205:443 users:(("Plex Media Serv",pid=1493544,fd=63))
|
||||
ESTAB 0 0 172.18.0.14:47696 172.104.133.220:443 users:(("Plex Media Serv",pid=1493544,fd=90))
|
||||
ESTAB 0 0 127.0.0.1:58916 127.0.0.1:44757 users:(("Plex Tuner Serv",pid=1494146,fd=63))
|
||||
ESTAB 0 0 127.0.0.1:44757 127.0.0.1:58916 users:(("Plex Script Hos",pid=1493588,fd=9))
|
||||
CLOSE-WAIT 1 0 172.18.0.14:40214 172.64.151.205:443 users:(("Plex Tuner Serv",pid=1494146,fd=65))
|
||||
CLOSE-WAIT 1 0 172.18.0.14:40224 172.64.151.205:443 users:(("Plex Tuner Serv",pid=1494146,fd=66))
|
||||
CLOSE-WAIT 25 0 172.18.0.14:39060 52.5.26.215:443 users:(("Plex Tuner Serv",pid=1494146,fd=58))
|
||||
CLOSE-WAIT 25 0 172.18.0.14:37658 54.156.171.78:443 users:(("Plex Tuner Serv",pid=1494146,fd=64))
|
||||
ESTAB 0 0 [::ffff:172.18.0.14]:32400 [::ffff:172.18.0.10]:43138 users:(("Plex Media Serv",pid=1493544,fd=64))
|
||||
### docker logs plex (tail, redacted)
|
||||
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
|
||||
[s6-init] ensuring user provided files have correct perms...exited 0.
|
||||
[fix-attrs.d] applying ownership & permissions fixes...
|
||||
[fix-attrs.d] done.
|
||||
[cont-init.d] executing container initialization scripts...
|
||||
[cont-init.d] 40-plex-first-run: executing...
|
||||
Plex Media Server first run setup complete
|
||||
[cont-init.d] 40-plex-first-run: exited 0.
|
||||
[cont-init.d] 45-plex-hw-transcode-and-connected-tuner: executing...
|
||||
[cont-init.d] 45-plex-hw-transcode-and-connected-tuner: exited 0.
|
||||
[cont-init.d] 50-plex-update: executing...
|
||||
[cont-init.d] 50-plex-update: exited 0.
|
||||
[cont-init.d] done.
|
||||
[services.d] starting services
|
||||
Starting Plex Media Server.
|
||||
[services.d] done.
|
||||
Critical: libusb_init failed
|
||||
### host: PVE rrd CPU for 9201, week (cpu fraction of maxcpu)
|
||||
[{"cpu":0.0261215320621999,"disk":1354493110.04444,"diskread":256997.603555556,"diskwrite":212267.463111111,"maxcpu":7,"maxdisk":33501757440,"maxmem":27156021248,"mem":2546329163.09333,"netin":1271.56733333333,"netout":1237.29994444444,"pressurecpufull":0.0105666666666667,"pressurecpusome":0.0585888888888889,"pressureiofull":0.00588888888888889,"pressureiosome":0.00588888888888889,"pressurememoryfull":0,"pressurememorysome":0,"time":1792143000},{"cpu":0.0510190652915542,"disk":1354510247.25333,"diskread":73172.7644444444,"diskwrite":4806651.44888889,"maxcpu":7,"maxdisk":33501757440,"maxmem":27156021248,"mem":3311811888.92444,"netin":392657.089444444,"netout":2637.96111111111,"pressurecpufull":0.0423888888888889,"pressurecpusome":0.130222222222222,"pressureiofull":0.251722222222222,"pressureiosome":0.259833333333333,"pressurememoryfull":0,"pressurememorysome":0,"time":1792144800},{"cpu":0.0510934153209064,"disk":1354502940.44444,"diskread":67670.4711111111,"diskwrite":2653541.26222222,"maxcpu":7,"maxdisk":33501757440,"maxmem":27156021248,"mem":3028358838.04444,"netin":298663.593888889,"netout":4279.61333333333,"pressurecpufull":0.0225555555555556,"pressurecpusome":0.120894444444444,"pressureiofull":0.169277777777778,"pressureiosome":0.173611111111111,"pressurememoryfull":0,"pressurememorysome":0,"time":1792146600},{"cpu":0.0251609673633183,"disk":1354484053.33333,"diskread":3804.72888888889,"diskwrite":366696.675555556,"maxcpu":7,"maxdisk":33501757440,"maxmem":27156021248,"mem":2568489892.97778,"netin":29447.5497222222,"netout":1308.49916666667,"pressurecpufull":0.0084,"pressurecpusome":0.0589944444444444,"pressureiofull":0.00111111111111111,"pressureiosome":0.00111111111111111,"pressurememoryfull":0,"pressurememorysome":0,"time":1792148400},{"cpu":0.0235582103863548,"disk":1354485760,"diskread":3731.91111111111,"diskwrite":179395.697777778,"maxcpu":7,"maxdisk":33501757440,"maxmem":27156021248,"mem":2569533701.68889,"netin":1153.13333333333,"netout":993.847777777778,"pressurecpufull":0.00561111111111111,"pressurecpusome":0.0396666666666667,"pressureiofull":0,"pressureiosome":0,"pressurememoryfull":0,"pressurememorysome":0,"time":1792150200},{"cpu":0.0236675215327719,"disk":1354485760,"diskread":4847.84355555555,"diskwrite":178476.373333333,"maxcpu":7,"maxdisk":33501757440,"maxmem":27156021248,"mem":2571881294.50667,"netin":1161.75583333333,"netout":1524.10666666667,"pressurecpufull":0.00653888888888889,"pressurecpusome":0.03755,"pressureiofull":0,"pressureiosome":0,"pressurememoryfull":0,"pressurememorysome":0,"time":1792152000},{"cpu":0.0230351295080715,"disk":1354546521.88444,"diskread":23838.72,"diskwrite":288008.874666667,"maxcpu":7,"maxdisk":33501757440,"maxmem":27156021248,"mem":2576033828.40889,"netin":1058.68644444444,"netout":534.349111111111,"pressurecpufull":0.00502222222222222,"pressurecpusome":0.0359166666666667,"pressureiofull":0.00222222222222222,"pressureiosome":0.00222222222222222,"pressurememoryfull":0,"pressurememorysome":0,"time":1792153800},{"cpu":0.0223896852341954,"disk":1354617173.33333,"diskread":8146.48888888889,"diskwrite":183957.731555556,"maxcpu":7,"maxdisk":33501757440,"maxmem":27156021248,"mem":2573392952.88889,"netin":1152.26255555556,"netout":522.357222222222,"pressurecpufull":0.00497777777777778,"pressurecpusome":0.0323666666666667,"pressureiofull":0.00127777777777778,"pressureiosome":0.00127777777777778,"pressurememoryfull":0,"pressurememorysome":0,"time":1792155600},{"cpu":0.0328054135238093,"disk":1354711658.95111,"diskread":39767.6088888889,"diskwrite":2304037.09155556,"maxcpu":7,"maxdisk":33501757440,"maxmem":27156021248,"mem":2640605206.75556,"netin":284115.51,"netout":5279.34577777778,"pressurecpufull":0.0150055555555556,"pressurecpusome":0.0671944444444444,"pressureiofull":0.0986111111111111,"pressureiosome":0.0995555555555555,"pressurememoryfull":0,"pressurememorysome":0,"time":1792157400},{"cpu":0.0313058140585278,"disk":1354745409.99111,"diskread":20061.2977777778,"diskwrite":504324.551111111,"maxcpu":7,"maxdisk":33501757440,"maxmem":27156021248,"mem":3440363356.16,"netin":29279.6321111111,"netout":1043.135,"pressurecpufull":0.00488888888888889,"pressurecpusome":0.0581,"pressureiofull":0,"pressureiosome":0,"pressurememoryfull":0,"pressurememorysome":0,"time":1792159200},{"cpu":0.031102069224026,"disk":1354678272,"diskread":20470.8977777778,"diskwrite":299071.715555556,"maxcpu":7,"maxdisk":33501757440,"maxmem":27156021248,"mem":3440898425.74222,"netin":1234.27577777778,"netout":1830.29333333333,"pressurecpufull":0.00288888888888889,"pressurecpusome":0.0464111111111111,"pressureiofull":0,"pressureiosome":0,"pressurememoryfull":0,"pressurememorysome":0,"time":1792161000},{"cpu":0.0936448120174985,"disk":1354676865.70667,"diskread":2786501.97333333,"diskwrite":7662840.94577778,"maxcpu":7,"maxdisk":33501757440,"maxmem":27156021248,"mem":3316517526.18667,"netin":285286.697222222,"netout":11612.1004444444,"pressurecpufull":0.0475944444444444,"pressurecpusome":0.36161111Line truncated
|
||||
@@ -0,0 +1,21 @@
|
||||
#!/bin/sh
|
||||
V=/var/lib/docker/volumes/plex_plex_config/_data
|
||||
P="$V/Library/Application Support/Plex Media Server"
|
||||
echo "### controls"
|
||||
printf 'ELF detector on /usr/bin/docker (positive): '; head -c 4 /usr/bin/docker | grep -q ELF && echo ELF || echo not-ELF
|
||||
printf 'ELF detector on Preferences.xml (negative): '; head -c 4 "$P/Preferences.xml" | grep -q ELF && echo ELF || echo not-ELF
|
||||
echo "log grep positive control (lines with 'Request: [' in current log): $(grep -c 'Request: \[' "$P/Logs/Plex Media Server.log")"
|
||||
echo "### docker diff plex (files changed in the container layer)"
|
||||
docker diff plex | head -60; echo "total changes: $(docker diff plex | wc -l)"
|
||||
echo "### created / started / restarts"
|
||||
docker inspect plex --format 'created={{.Created}} started={{.State.StartedAt}} restarts={{.RestartCount}}'
|
||||
echo "### transcode volume: files"
|
||||
find /var/lib/docker/volumes/plex_plex_transcode/_data -type f | head; echo "(n=$(find /var/lib/docker/volumes/plex_plex_transcode/_data -type f | wc -l))"
|
||||
echo "### crash uploader log (latest)"
|
||||
sed -E 's/(token|Token)[=:" ]+[A-Za-z0-9_-]{8,}/\1=REDACTED/g' "$P/Logs/Plex Crash Uploader.1.log"
|
||||
echo "### first lines of each PMS log (start time)"
|
||||
for f in "$P/Logs/"Plex\ Media\ Server*.log; do echo "$(basename "$f"): $(head -1 "$f" | cut -c1-24)"; done
|
||||
echo "### outbound connections now from plex netns"
|
||||
PID=$(docker inspect -f '{{.State.Pid}}' plex); nsenter -t $PID -n ss -tnp 2>/dev/null | head -20
|
||||
echo "### docker logs plex (tail, redacted)"
|
||||
docker logs --tail 30 plex 2>&1 | sed -E 's/(claim-|token=)[A-Za-z0-9_-]+/\1REDACTED/g'
|
||||
@@ -0,0 +1,78 @@
|
||||
{
|
||||
"update": {
|
||||
"accepted": true,
|
||||
"http": "202",
|
||||
"phases": [
|
||||
{
|
||||
"t": 0.0,
|
||||
"phase": "safety-dump",
|
||||
"label": "Adatbázis pillanatkép…",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 4.1,
|
||||
"phase": "pulling",
|
||||
"label": "Új verzió letöltése…",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 10.1,
|
||||
"phase": "copying",
|
||||
"label": "Az adatok másolása a frissítés előtt…",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 22.3,
|
||||
"phase": "verifying",
|
||||
"label": "Működés ellenőrzése…",
|
||||
"updating": true,
|
||||
"error": null,
|
||||
"hold": null
|
||||
},
|
||||
{
|
||||
"t": 28.4,
|
||||
"phase": "done",
|
||||
"label": "Frissítve",
|
||||
"updating": false,
|
||||
"error": null,
|
||||
"hold": null
|
||||
}
|
||||
],
|
||||
"duration_s": 28.4,
|
||||
"final_phase": "done",
|
||||
"update_error": null,
|
||||
"hold_reason": null,
|
||||
"state": "running"
|
||||
},
|
||||
"observables": {
|
||||
"pinned_images": {
|
||||
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
},
|
||||
"installed_images": {
|
||||
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
},
|
||||
"catalog_images": {
|
||||
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
},
|
||||
"live_compose_image_lines": [
|
||||
"image: plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
|
||||
],
|
||||
"docker_inspect": [
|
||||
"plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 running=true restarts=0"
|
||||
]
|
||||
},
|
||||
"log": [
|
||||
"11:07:51 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}",
|
||||
"11:07:51 + 0.0s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None",
|
||||
"11:07:55 + 4.1s phase=pulling label=Új verzió letöltése… err=None hold=None",
|
||||
"11:08:01 + 10.1s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None",
|
||||
"11:08:14 + 22.3s phase=verifying label=Működés ellenőrzése… err=None hold=None",
|
||||
"11:08:20 + 28.4s phase=done label=Frissítve err=None hold=None"
|
||||
]
|
||||
}
|
||||
@@ -143,10 +143,12 @@ stopping line that lies.
|
||||
| **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | WATCHING — **2026-10-22 night:** not reproducible tonight; the Immich server log of 19:38 is gone (Immich restarted at 02:16 for the off-site dump). What the household sees: the upload fails after 10 minutes and the Immich phone app sends it again. | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC |
|
||||
|
||||
|
||||
## App updates — 4 rows (P3 4)
|
||||
## App updates — 6 rows (P2 2, P3 4)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **R-1238** | App updates | P2 | **Plex has no test record and no upgrade fixture, so the night leg never updates Plex on a household box.** Filed 2026-10-23. Catalog `6cc317d` moved Plex 1.41.4.9463 → 1.43.5.11029-6bf8af3f0 (the Plex2Shell fix) on the operator's ruling, with NO `update_ladder` entry (the move gate refused; pushed `--no-verify`, the one time). With no ladder, a person's Update press pins the current compose (`stacks/ladder.go` `nextLadderStep`), but the night leg skips the app (`unattended.go` → `LegSkipNoTestRecord`), so every future Plex fix reaches a household only by a hand press. `scripts/upgrade_fixtures.py` has no Plex fixture. A likely seed route is NOT measured: an UNCLAIMED Plex answers a private-address request with no token (`architecture/13-threat-model.md`, the R-1045 row), so the bench may create and read back a library section through Plex's own API without a plex.tv account. **Needs a spike:** prove that route on the bench (seed a section, swap the image, read it back), add the fixture, then `upgrade-test.py --write-ladder` for the current pin — the box walk on 9202 needs the same route behind the setup gate. Evidence: `audits/plex-demo-boxes-2026-10-23/`. | **OPEN** | a claimed-Plex seed route on the bench | the spike above | CC |
|
||||
| **R-1239** | App updates | P2 | **The catalog has no fast lane for an urgent security fix — a decision for the operator** (decision 13 is a choice, not a defect, R-370). Filed 2026-10-23. Today the only way to move an image without a proven test record is `git push --no-verify` past `check-test-record-move.py`, as catalog `6cc317d` (Plex2Shell) did on the operator's 10:56 ruling — and such a move never reaches a box at night (R-1238). **Options:** (a) keep decision 13 as it is; an urgent fix is a one-time operator ruling + `--no-verify` + a hand press per box — costs an operator word and a CC session per incident, and households wait for the press; (b) a marked `security_fix` ladder entry (the operator's ruling cited, bench run optional, box walk on one test box required) that the gate accepts and the night leg presses — costs one gate rule, one leg rule, decoys, and a tested step that skipped the bench. **CC's pick: (b)**, because (a) does not scale past the first customers and needs the operator awake. | **WAITING-ON-OPERATOR** | the operator's choice | rule on (a) or (b); then CC builds (b) if chosen | operator |
|
||||
| **R-440** | App updates | P3 | **[P2-MEDIUM] 23 catalog image pins float, so an update is not reproducible.** `compose pull` on a moving tag fetches whatever upstream published that day. **MEASURED 2026-09-01 over `app-catalog-felhom.eu` @ `29edad9c5bf4`: 79 `image:` lines across 53 apps, 66 distinct; 23 of those lines carry a tag with no patch version.** `postgres:16-alpine` (8 apps), `redis:7-alpine` (6), `mariadb:11.6` (2), plus one each of `postgres:15-alpine`, `postgis/postgis:16-3.5-alpine`, `mariadb:11.4`, `mariadb:12.3`, `ghcr.io/claperco/claper:2.5`, `ghcr.io/thomiceli/opengist:1.13`, `wger/server:2.6`. **A 24th is arguable and is recorded rather than rounded away:** `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0` pins both extensions exactly but leaves the PostgreSQL patch floating. A customer pressing Frissites can therefore swap their DATABASE ENGINE build with no catalog change and no record; two boxes updated on two days end up different. **Severity MEDIUM on its own; it becomes BLOCKING the moment a pre-update copy exists**, because "what did we upgrade from and to" must be recordable and today it is not — which is also why R-440 must be read next to the digest discipline in Rule 10 of the spike. **MEASURED LIVE 2026-09-01 — the floating pins have ALREADY moved, with a passing control.** Running digests on demo-hp compared against what the registry serves for the same tag today: **`mariadb:11.4` MOVED** (`sha256:4f1d8d20...` -> `sha256:611a2fcc...`) and **`mariadb:12.3` MOVED** (`sha256:a02fe89c...` -> `sha256:dd9b303a...`), while `postgres:16-alpine`, `redis:7-alpine`, `mariadb:11.6` and `opengist:1.13` were SAME — **and both fully-pinned CONTROLS (`rommapp/romm:5.0.0`, `privatebin/pdo:2.0.5`) were SAME.** So on a box with ZERO visible drift by tag, pressing Frissites today silently swaps the DATABASE ENGINE build under `romm` and `bookstack`, with no catalog change and no record. **Compounding fact found while reading:** the recovery unit records `ImagePins` but the manifest comment says *"image NOT stored - re-pulled on restore"*, so a RESTORE of a floating-pinned app also re-pulls whatever is current — the same non-reproducibility on the recovery path. **HALF OF THE ANSWER SHIPPED 2026-09-02 (controller v0.233.0, slice 1): `app.yaml.installed_images` now records, per compose SERVICE, the reference AND the repo digest each container was actually created from — so "what did we upgrade FROM" is answerable on any box that has taken one lifecycle action since the upgrade.** What is still missing is the other half: comparing that digest against what the registry serves for the same tag TODAY, which needs a network call the render path deliberately does not make (see R-446). **The row therefore stays OPEN and its rank is unchanged** — recording a digest does not make a floating pin reproducible; it makes the drift measurable after the fact. `audits/SPIKE-app-update-2026-09-01.md` **— NIGHT 2026-09-23:** every image moved tonight (and the 21 backfilled moves) carries its resolved digest in the catalog's test record; the floating pins themselves still float until the box pulls by digest (`09` §6.4 part 6). **-- MEASURED 2026-09-30 on demo-hp 9201: a fresh install or a guarded Update renders `name:tag@sha256` from the ladder (`digest.go` `RenderWithLadderDigests`) — 9 of 20 services there run such a definition (adventurelog, docmost, paperless-ngx). The other 11 run TAG-ONLY definitions: bookstack, kimai, opengist, privatebin, romm were installed before v0.269.0 and not updated since, and `CarryDigests` (`digest.go:141`) keeps only a digest the running definition already has; bentopdf and calibre-web have no ladder at all. So a re-pull of those (a restore) takes whatever the tag serves that day. `audits/pg-last-six-2026-09-30/E/E2-digests-demo-hp.txt`.** **-- 2026-09-30 (evening):** calibre-web now has a ladder entry (`53a4a1d`), so a new install or a guarded Update renders its tested digest; bentopdf still has none. `audits/more-night-apps-2026-09-30/` **Merged 2026-10-05 from R-446 (duplicate):** the household-visible consequence — for these apps the „Naprakész" badge can never turn „behind" (`felhom-controller controller/internal/stacks/updateorder.go:96` returns false with no catalog digests / no test date); R-740 (same-tag re-test) is closed, so floating-tag drift for laddered apps is handled. | **NARROWED 2026-09-30 — floating only for (1) apps with no ladder entry and (2) apps installed before v0.269.0 until their next guarded Update; owner: CC.** **Re-ranked 2026-10-03: P2→P3: narrowed to apps with no ladder entry and pre-v0.269 installs; each guarded update fixes the latter.** | — | — | CC |
|
||||
| **R-462** | App updates | P3 | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s,Line truncated
|
||||
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
|
||||
|
||||
Reference in new issue
Block a user