Night 2026-10-28 close-out: report, STATUS morning list, CONTEXT (CC decisions), R-1256/R-1238 states, R-1339 filed, Plex box walk evidence, catalog morning patches (prepared, not pushed)
gates / gates (push) Successful in 1m50s

This commit is contained in:
admin committed 2026-10-29 05:19:59 +01:00
1 parent 7a6b9ff19d
commit 7799ef5f54
9 files changed
+450 -6

No files matched your search

@@ -1,8 +1,8 @@
From be6a62917fbd694bb5ed4d6927492b3e97c4cf45 Mon Sep 17 00:00:00 2001
From: cc <cc@local>
Date: Thu, 29 Oct 2026 00:12:28 +0100
Subject: [PATCH] PREPARED (not pushed): R-1338 memory figure + Plex fixtures +
Wanderer entry re-measure
Subject: [PATCH 1/2] PREPARED (not pushed): R-1338 memory figure + Plex
fixtures + Wanderer entry re-measure
---
scripts/box_walk.py | 32 +++++++---
@@ -0,0 +1,28 @@
From 36e58cc3afbc478fa3191b407210f23a5f0ac966 Mon Sep 17 00:00:00 2001
From: kisfenyo <nagyfenyvesi.viktor@gmail.com>
Date: Thu, 29 Oct 2026 05:11:58 +0100
Subject: [PATCH 2/2] PREPARED (not pushed): Plex test record 1.41.4.9463 ->
1.43.5.11029, bench + box 9202 proven (felhom.eu R-1238); catalog_since stays
2026-10-23 (the pin did not move)
---
templates/plex/.felhom.yml | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/templates/plex/.felhom.yml b/templates/plex/.felhom.yml
index 7379349..876c96f 100644
--- a/templates/plex/.felhom.yml
+++ b/templates/plex/.felhom.yml
@@ -237,3 +237,9 @@ i18n:
# One line per image in docker-compose.yml; the grammar is in NEW-APP-CHECKLIST.md (1.10).
upstream:
plexinc/pms-docker: "gap:closed source; Plex announces fixes on its forum only"
+
+# update_ladder — the test record: one tested step per line, oldest first (JSON flow mappings,
+# `09-update-architecture.md` §6.4 part 4). WRITTEN BY scripts/upgrade-test.py, never by hand;
+# gated by scripts/check-test-record.py. An image: move without a proven entry here is refused.
+update_ladder:
+ - {"from": {"plex": "plexinc/pms-docker:1.41.4.9463-630c9f557"}, "to": {"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"}, "digest": {"plex": "sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f"}, "verdict": "proven", "tested_at": "2026-10-28T21:48:51Z", "harness_version": 5, "evidence": "documentation/audits/night-2026-10-28/bench/plex/edge", "box_evidence": "documentation/audits/night-2026-10-28/r1238-plex/box-9202", "memory_peak_pct": 10.7, "marks": {"files_may_change": false, "needs_person": null, "memory_tight": false}, "memory_basis": "anon", "memory_cgroup_peak_pct": 12.3}
--
2.47.3
@@ -0,0 +1,15 @@
# Catalog morning patches (night 2026-10-28, prepared, NOT pushed)
The night brief allowed the catalog as a drill only, so these two commits were never pushed.
They apply on catalog `main` at `0baf8f9`.
1. `0001` — R-1338: the memory watch judges the LIVE anon figure (`memfigure.py`, `memory.stat` minus reclaimable
parts), Plex fixtures (bench + box), Wanderer's entry re-measured (27.7 %, `memory_tight` false).
2. `0002` — R-1238: Plex's test record 1.41.4.9463 → 1.43.5.11029, proven on the bench and on scratch 9202
(seed read back, 10-min soak 10.6 %, no kill). `catalog_since` stays 2026-10-23: the pin did not move.
The ladder writer had set it to the day it ran; it was put back by hand, because the date means "the pin last
moved", and the badge would read "0 days" for a tag that is six days old.
Apply: `cd app-catalog-felhom.eu && git am <this folder>/000*.patch`, then `python3 scripts/catalog_gates.py`, then push.
If `main` moved past `0baf8f9`, apply the same way; a conflict in `templates/wanderer/.felhom.yml` means the Wanderer
entry changed again — re-run the writer rather than merge by hand.
@@ -0,0 +1,227 @@
{
"app": "plex",
"walk": "security",
"box_walk_version": 1,
"verdict": "proven",
"venue": "box 9202 (drill catalog), the guarded Update",
"from": {
"plex": "plexinc/pms-docker:1.41.4.9463-630c9f557"
},
"to": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"observables_before": {
"pinned_images": {
"plex": "plexinc/pms-docker:1.41.4.9463-630c9f557"
},
"installed_images": {
"plex": "plexinc/pms-docker:1.41.4.9463-630c9f557"
},
"catalog_images": null,
"live_compose_image_lines": [
"image: plexinc/pms-docker:1.41.4.9463-630c9f557"
],
"docker_inspect": [
"plex plexinc/pms-docker:1.41.4.9463-630c9f557 running=true restarts=0"
]
},
"setup_gate_before": {
"browser": {
"status": "302",
"redirect": "https://felhom.enkisfelhom.hu/__gate/start?…"
},
"script": {
"status": "401",
"redirect": null
}
},
"seed": "read_back",
"drill_commit": "37736f07d9ae",
"sync_waited_s": 4.5,
"badge_before": {
"hu": [
{
"title": "Újabb változat érhető el ehhez az alkalmazáshoz. A frissítés indításához nyomd meg a Frissítés gombot.",
"text": "Frissítés elérhető — ma"
}
],
"en": [
{
"title": "A newer version of this app is available. Select the Update button to start it.",
"text": "Update available — today"
}
]
},
"measured_at": "2026-10-29T03:47:17Z",
"update": {
"accepted": true,
"http": "202",
"phases": [
{
"t": 0.0,
"phase": "backing-up",
"label": "Biztonsági mentés készül a frissítés előtt…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 11.3,
"phase": "safety-dump",
"label": "Adatbázis pillanatkép…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 12.3,
"phase": "pulling",
"label": "Új verzió letöltése…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 21.5,
"phase": "copying",
"label": "Az adatok másolása a frissítés előtt…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 35.9,
"phase": "verifying",
"label": "Működés ellenőrzése…",
"updating": true,
"error": null,
"hold": null
},
{
"t": 41.0,
"phase": "done",
"label": "Frissítve",
"updating": false,
"error": null,
"hold": null
}
],
"duration_s": 41.0,
"final_phase": "done",
"update_error": null,
"hold_reason": null,
"state": "running"
},
"observables_after": {
"pinned_images": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"installed_images": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"catalog_images": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"live_compose_image_lines": [
"image: plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
],
"docker_inspect": [
"plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 running=true restarts=0"
]
},
"badge_after": {
"hu": [
{
"title": "Ez az alkalmazás a legfrissebb elérhető változatot futtatja.",
"text": "Naprakész"
}
],
"en": [
{
"title": "This app is running the newest version available.",
"text": "Up to date"
}
]
},
"state_after": "running",
"hold_after": null,
"to_digests": {
"plex": "sha256:01bdf0ad6d53198da4331540c1e73ab65db7a72e961fff97ecb0847d46055c5f"
},
"four_observables": {
"pinned": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"installed": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"catalog": {
"plex": "plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
},
"live_compose": [
"plexinc/pms-docker:1.43.5.11029-6bf8af3f0"
]
},
"seed_read_after": true,
"setup_gate_after": {
"browser": {
"status": "302",
"redirect": "https://felhom.enkisfelhom.hu/__gate/start?…"
},
"script": {
"status": "401",
"redirect": null
}
},
"memory": {
"soak_s": 600,
"interval_s": 10,
"samples": 60,
"containers": {
"plex": {
"samples": 60,
"anon_peak_bytes": 230629376,
"limit_bytes": 2147483648,
"anon_peak_pct": 10.7,
"anon_live_peak_bytes": 228552704,
"anon_live_peak_pct": 10.6,
"cgroup_peak_pct": 12.3,
"oom_kill_delta": 0,
"oom_killed_flag": false,
"restarts_delta": 0,
"running_all_samples": true,
"cgroup": "/sys/fs/cgroup/system.slice/docker-ef9b5b1114ba34f44c59d053b2651e68aff8e2e3070f8ac70397c93f7ea84eab.scope"
}
},
"oom_kills": 0,
"restarts": 0,
"peak_pct": 10.6,
"peak_basis": "anon_live"
},
"why": "the security-fix step ran on the box through the guarded Update; healthy, TO on all four observables, no undo, no hold, a clean soak, seed read back",
"log": [
"04:45:38 [5] the box's drill mirror now at 41b37adc",
"04:45:42 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['PLEX_CLAIM', 'HDD_PATH']",
"04:45:43 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}",
"04:46:13 [1] deployed, controller state=running, pinned={'plex': 'plexinc/pms-docker:1.41.4.9463-630c9f557'}",
"04:46:13 gate: plex is gated — passed as the household (cookie set)",
"04:46:17 [2] setup gate as installed: {'browser': {'status': '302', 'redirect': 'https://felhom.enkisfelhom.hu/__gate/start?…'}, 'script': {'status': '401', 'redirect': None}}",
"04:46:53 plex: POST /library/sections through the front door http=401",
"04:46:53 plex: the front door was not taken as local (<html><head><title>Unauthorized</title></head><body><h1>401 Unauthorized</h1></body></html>) — asking from Plex's own container",
"04:46:56 plex: POST /library/sections from its own container http=201",
"04:46:56 plex: seeded library section felhom834284c6 uuid=fce5ae7f-2c56-4992-b730-058e892014ab (route own-container, server 1.41.4.9463-630c9f557)",
"04:47:03 plex: section felhom834284c6 read back (server 1.41.4.9463-630c9f557, route own-container), same uuid=True, folders=['/media']",
"04:47:04 [5] drill commit 37736f07d9ae: plex plexinc/pms-docker:1.41.4.9463-630c9f557 -> plexinc/pms-docker:1.43.5.11029-6bf8af3f0 (push rc=0)",
"04:47:12 [5] the box's drill mirror now at 37736f07",
"04:47:17 [6] Update -> 202 {'ok': True, 'data': {'accepted': True, 'completed': False}, 'message': 'Frissítés elindult – az állapot a kártyán követhető'}",
"04:47:17 + 0.0s phase=backing-up label=Biztonsági mentés készül a frissítés előtt… err=None hold=None",
"04:47:28 + 11.3s phase=safety-dump label=Adatbázis pillanatkép… err=None hold=None",
"04:47:29 + 12.3s phase=pulling label=Új verzió letöltése… err=None hold=None",
"04:47:39 + 21.5s phase=copying label=Az adatok másolása a frissítés előtt… err=None hold=None",
"04:47:53 + 35.9s phase=verifying label=Működés ellenőrzése… err=None hold=None",
"04:47:58 + 41.0s phase=done label=Frissítve err=None hold=None",
"04:48:23 plex: section felhom834284c6 read back (server 1.43.5.11029-6bf8af3f0, route own-container), same uuid=True, folders=['/media']",
"04:48:23 [7] memory soak 600 s …",
"04:58:30 [7] soak: peak 10.6% of the limit (anon), kills 0, restarts 0"
]
}
+3 -2
View File
@@ -153,6 +153,7 @@ stopping line that lies.
| **R-1336** | Apps & catalog | P3 | **ActualBudget has no release with the HTTP2Shell fix (CVE-2026-111497): 26.9.0 carries Node.js 24.20.0 and the newest 26.10.0 still Node.js 24.21.0 (fixed: 24.22.1), read from both images.** `node app.js` serves the app; the setup gate covers it until the first account exists, then its own password. No box runs it. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Node ≥ 24.22.1 | the watch lists it nightly (R-1327); take that release as a lane step | CC |
| **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC |
| **R-1338** | Apps & catalog | P3 | **Wanderer's Meilisearch fills whatever memory limit it gets: 92–100 % anon of 256 MB (bench and 9202), still 88.8 % of 384 MB on the bench (2026-10-28)** — it sizes its indexing memory from the memory it sees (its default: 2/3 of the available RAM), so a higher limit alone does not end the `memory_tight` mark of Wanderer's step. No kill in any run. `audits/dockerhub-bigsteps-2026-10-28/wanderer/` | OPEN — **2026-10-28 night: MEASURED — it is the measurement, not Wanderer, and the cap is not needed** (`audits/night-2026-10-28/r1338-meili/`; `09` §3 decision 22 refined). Meilisearch frees with MADV_FREE: the given-back pages stay in `anon` but sit on the inactive FILE list (`file` ~1 MB, `inactive_file` ~290 MB) and the kernel takes them first — 83–96 % `anon` on the bench (no swap) AND on 9202, with and without the cap, 0 kills; it uses ~100 MB. The evening's 17.7 % was the container after the swap, not the cap. **Fixed in the tools:** the mark reads the live anon (catalog `scripts/memfigure.py`, the larger of two estimates, errs high; `upgrade-test.py`, `box_walk.py` soak; `test_memory_live.py` red-proved). Re-run: bench Wanderer v0.20.0 → v0.21.2 `proven`, **no marks** (search 83.3 % anon, 27.7 % live); 9202 95.2 % anon, 25.1 % live. **Prepared, NOT pushed** (the night's catalog fence): the tool change + Wanderer's entry without `memory_tight` (`audits/night-2026-10-28/catalog-morning/`; the catalog gates passed in a scratch clone). **Earlier, 2026-10-28 evening: `MEILI_MAX_INDEXING_MEMORY` = 128 MiB (a third of 384 MB) measured, NOT pushed** (`audits/gitea-wanderer-2026-10-28/wanderer-meili/`): on the bench, under the harness's load, the search container peaked at 17.7 % (84 MB; 88.8 % without the cap, same load); on 9202 the cap was in the container (read back) and it still held 95.7 % anon (363 MB steady over 6 min, no kill). The box result decides: not shown to help, so not pushed. Also: the catalog's ladder has no entry type for a re-measure of a definition change at the SAME digests (`ladder.py`: a re-test whose digest equals its digest_from tests nothing new), so the v0.21.2 entry's `memory_tight` mark cannot be replaced without an image move. | the catalog push (the morning list) | push the prepared catalog change (the live figure + Wanderer's entry re-measured, `memory_tight` off — an existing entry's memory fields edited at the SAME digests, which the gates accept); then close | CC |
| **R-1339** | Storage & devices | P4 | **At every start the controller logs „FileBrowser: drive …/userdata/<app> not mounted — skipping userdata skeleton" for per-app folders on a drive that IS mounted** (scratch 9202, 2026-10-29 03:35 UTC: nine lines, `scratch_hdd` a live mountpoint, mp8). `skipFileBrowserPath` (`controller/internal/web/handlers.go:3766`) asks whether the folder ITSELF is a mountpoint, and a folder under a mounted drive never is. No data is lost: the file browser shows the folders through the drive's own mount (`/srv/scratch_hdd`). The cost is a false „not mounted" in the log of a box whose drive works, which misleads a diagnosis. Fix: ask about the drive the folder lives on, with a test that a sub-folder of a mounted drive is not skipped. Evidence: `audits/night-2026-10-28/part5-claims/selfheal/controller-after-reboot.log`. | OPEN | — | the fix above, with its red-first test | CC |
## App updates — 10 rows (P2 1, P3 9)
@@ -164,7 +165,7 @@ stopping line that lies.
| **R-1319** | App updates | P3 | **Paperless-ngx 2.20.15: GHSA-2jhj (low; needs a setting we do not set — not reachable); the fix is 3.3.4, a big step** (encryption and API v1 removed, the search index rebuilt). It runs on all four test boxes. Not urgent. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step 3.3.4 when the catalog takes Paperless 3 for other reasons; read our API v1 use first | CC |
| **R-1321** | App updates | P3 | **Tandoor 2.6.15: GHSA-5f74 (high, an XXE file read) and GHSA-69q3 (any user); the fix is 2.7.2** — 2.7.0 removes settings and an endpoint and rewrites the login pages (our probe path — not sure), and 2.7.0/2.7.1 break invite links with sign-up off, so straight to 2.7.2. No box runs it. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | a normal step to 2.7.2 with the probe path checked; bench + 9202 (+ undo if a migration); a STOP | CC |
| **R-1324** | App updates | P3 | **RomM 5.4's first start can run a 15–20 minute migration on a big library (upstream notes) — longer than the update's check window, so a household's night step could be undone for slowness, not for a fault.** Today's walk (9202, a small seed) took the step in seconds; demo-hp's RomM takes it tonight (security-fix step, catalog `06e603e`). `audits/security-app-updates-2026-10-27/triage-notes/romm.md` | OPEN — filed 2026-10-27; WATCH demo-hp's night leg line | — | read demo-hp's night leg line for romm (done / undone, the time); if undone for time: give the step a longer check or the `needs_person` mark | CC |
| **R-1238** | App updates | P2 | **Plex has no test record and no upgrade fixture, so the night leg never updates Plex on a household box.** Filed 2026-10-23. Catalog `6cc317d` moved Plex 1.41.4.9463 → 1.43.5.11029-6bf8af3f0 (the Plex2Shell fix) on the operator's ruling, with NO `update_ladder` entry (the move gate refused; pushed `--no-verify`, the one time). With no ladder, a person's Update press pins the current compose (`stacks/ladder.go` `nextLadderStep`), but the night leg skips the app (`unattended.go` → `LegSkipNoTestRecord`), so every future Plex fix reaches a household only by a hand press. `scripts/upgrade_fixtures.py` has no Plex fixture. A likely seed route is NOT measured: an UNCLAIMED Plex answers a private-address request with no token (`architecture/13-threat-model.md`, the R-1045 row), so the bench may create and read back a library section through Plex's own API without a plex.tv account. **Needs a spike:** prove that route on the bench (seed a section, swap the image, read it back), add the fixture, then `upgrade-test.py --write-ladder` for the current pin — the box walk on 9202 needs the same route behind the setup gate. Evidence: `audits/plex-demo-boxes-2026-10-23/`. **2026-10-23 (later): the NEXT Plex fix need not wait for this** — the security-fix lane (`09` §3 decision 255, D39 = B, controller 0.341.0) takes a box-only step with `seed: "none"` at night (proven on 9202 with Plex 1.43.4 → 1.43.5, `audits/security-lane-2026-10-23/`). The bench fixture stays the aim: it is the only check that reads Plex's data back after a step. | **OPEN** | a claimed-Plex seed route on the bench | the spike above | CC |
| **R-1238** | App updates | P2 | **Plex has no test record and no upgrade fixture, so the night leg never updates Plex on a household box.** Filed 2026-10-23. Catalog `6cc317d` moved Plex 1.41.4.9463 → 1.43.5.11029-6bf8af3f0 (the Plex2Shell fix) on the operator's ruling, with NO `update_ladder` entry (the move gate refused; pushed `--no-verify`, the one time). With no ladder, a person's Update press pins the current compose (`stacks/ladder.go` `nextLadderStep`), but the night leg skips the app (`unattended.go` → `LegSkipNoTestRecord`), so every future Plex fix reaches a household only by a hand press. `scripts/upgrade_fixtures.py` has no Plex fixture. A likely seed route is NOT measured: an UNCLAIMED Plex answers a private-address request with no token (`architecture/13-threat-model.md`, the R-1045 row), so the bench may create and read back a library section through Plex's own API without a plex.tv account. **Needs a spike:** prove that route on the bench (seed a section, swap the image, read it back), add the fixture, then `upgrade-test.py --write-ladder` for the current pin — the box walk on 9202 needs the same route behind the setup gate. Evidence: `audits/plex-demo-boxes-2026-10-23/`. **2026-10-23 (later): the NEXT Plex fix need not wait for this** — the security-fix lane (`09` §3 decision 255, D39 = B, controller 0.341.0) takes a box-only step with `seed: "none"` at night (proven on 9202 with Plex 1.43.4 → 1.43.5, `audits/security-lane-2026-10-23/`). The bench fixture stays the aim: it is the only check that reads Plex's data back after a step. **2026-10-28 night: the route works, unclaimed.** Plex refuses the request through its front door (401: the proxy's address is not taken as local), so the fixture asks from Plex's own container (`POST /library/sections` → 201) and reads the section back by its uuid. Bench 1.41.4.9463 → 1.43.5.11029 proven (`audits/night-2026-10-28/bench/plex/edge/`); box walk on scratch 9202 through the setup gate proven 2026-10-29 (the household's Update, 41 s; the section read back on 1.43.5 with the same uuid; 10-min soak 10.6 %, no kill; `…/r1238-plex/box-9202/`). The fixtures and the ladder entry are a PREPARED patch (`…/catalog-morning/`), not pushed — the night allowed no live catalog push. | OPEN — proven on both venues, not pushed | the morning catalog push | push the two prepared catalog commits (the operator's word or an attended session) | CC |
| **R-440** | App updates | P3 | **[P2-MEDIUM] 23 catalog image pins float, so an update is not reproducible.** `compose pull` on a moving tag fetches whatever upstream published that day. **MEASURED 2026-09-01 over `app-catalog-felhom.eu` @ `29edad9c5bf4`: 79 `image:` lines across 53 apps, 66 distinct; 23 of those lines carry a tag with no patch version.** `postgres:16-alpine` (8 apps), `redis:7-alpine` (6), `mariadb:11.6` (2), plus one each of `postgres:15-alpine`, `postgis/postgis:16-3.5-alpine`, `mariadb:11.4`, `mariadb:12.3`, `ghcr.io/claperco/claper:2.5`, `ghcr.io/thomiceli/opengist:1.13`, `wger/server:2.6`. **A 24th is arguable and is recorded rather than rounded away:** `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0` pins both extensions exactly but leaves the PostgreSQL patch floating. A customer pressing Frissites can therefore swap their DATABASE ENGINE build with no catalog change and no record; two boxes updated on two days end up different. **Severity MEDIUM on its own; it becomes BLOCKING the moment a pre-update copy exists**, because "what did we upgrade from and to" must be recordable and today it is not — which is also why R-440 must be read next to the digest discipline in Rule 10 of the spike. **MEASURED LIVE 2026-09-01 — the floating pins have ALREADY moved, with a passing control.** Running digests on demo-hp compared against what the registry serves for the same tag today: **`mariadb:11.4` MOVED** (`sha256:4f1d8d20...` -> `sha256:611a2fcc...`) and **`mariadb:12.3` MOVED** (`sha256:a02fe89c...` -> `sha256:dd9b303a...`), while `postgres:16-alpine`, `redis:7-alpine`, `mariadb:11.6` and `opengist:1.13` were SAME — **and both fully-pinned CONTROLS (`rommapp/romm:5.0.0`, `privatebin/pdo:2.0.5`) were SAME.** So on a box with ZERO visible drift by tag, pressing Frissites today silently swaps the DATABASE ENGINE build under `romm` and `bookstack`, with no catalog change and no record. **Compounding fact found while reading:** the recovery unit records `ImagePins` but the manifest comment says *"image NOT stored - re-pulled on restore"*, so a RESTORE of a floating-pinned app also re-pulls whatever is current — the same non-reproducibility on the recovery path. **HALF OF THE ANSWER SHIPPED 2026-09-02 (controller v0.233.0, slice 1): `app.yaml.installed_images` now records, per compose SERVICE, the reference AND the repo digest each container was actually created from — so "what did we upgrade FROM" is answerable on any box that has taken one lifecycle action since the upgrade.** What is still missing is the other half: comparing that digest against what the registry serves for the same tag TODAY, which needs a network call the render path deliberately does not make (see R-446). **The row therefore stays OPEN and its rank is unchanged** — recording a digest does not make a floating pin reproducible; it makes the drift measurable after the fact. `audits/SPIKE-app-update-2026-09-01.md` **— NIGHT 2026-09-23:** every image moved tonight (and the 21 backfilled moves) carries its resolved digest in the catalog's test record; the floating pins themselves still float until the box pulls by digest (`09` §6.4 part 6). **-- MEASURED 2026-09-30 on demo-hp 9201: a fresh install or a guarded Update renders `name:tag@sha256` from the ladder (`digest.go` `RenderWithLadderDigests`) — 9 of 20 services there run such a definition (adventurelog, docmost, paperless-ngx). The other 11 run TAG-ONLY definitions: bookstack, kimai, opengist, privatebin, romm were installed before v0.269.0 and not updated since, and `CarryDigests` (`digest.go:141`) keeps only a digest the running definition already has; bentopdf and calibre-web have no ladder at all. So a re-pull of those (a restore) takes whatever the tag serves that day. `audits/pg-last-six-2026-09-30/E/E2-digests-demo-hp.txt`.** **-- 2026-09-30 (evening):** calibre-web now has a ladder entry (`53a4a1d`), so a new install or a guarded Update renders its tested digest; bentopdf still has none. `audits/more-night-apps-2026-09-30/` **Merged 2026-10-05 from R-446 (duplicate):** the household-visible consequence — for these apps the „Naprakész" badge can never turn „behind" (`felhom-controller controller/internal/stacks/updateorder.go:96` returns false with no catalog digests / no test date); R-740 (same-tag re-test) is closed, so floating-tag drift for laddered apps is handled. | **NARROWED 2026-09-30 — floating only for (1) apps with no ladder entry and (2) apps installed before v0.269.0 until their next guarded Update; owner: CC.** **Re-ranked 2026-10-03: P2→P3: narrowed to apps with no ladder entry and pre-v0.269 installs; each guarded update fixes the latter.** | — | — | CC |
| **R-462** | App updates | P3 | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s,Line truncated
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
@@ -186,7 +187,7 @@ stopping line that lies.
| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. | — | — | CC |
| **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) |
| **R-1255** | Backup & restore | P3 | **Escrow edge cases (review round 2, HUNT-3 H3/H4):** the R-1176 carry moves the newest kept package even when it holds no repository password (`hub/internal/store/escrow_carry.go:100-110` checks `identity_blob` only), while the right one stays under the deleted host (`store.go:3533-3534`); two set-asides on the same UTC day reuse a name and the purge can delete a still-stored copy's package (`hub/internal/offsitekeys/offsitekeys.go:378-385`; `store.go:4295-4300`). Static. | OPEN — **fixed on main 2026-10-28 night, not released** (felhom.eu `1ee6d5a5`, CI 2472; store tests red first; `03` §8 [DESIGN] 2026-10-28): the carry takes the newest retained row with an identity AND a repository-password hash (the newest row only when none has a hash — a legacy package, as before); a set-aside is `<repo>.orphaned-<YYYYMMDDTHHMMSSZ>`; a retained row is purged only when it was tagged no later than its path's deletion (`repo_tagged_at`, new column). No live proof: a hub deploy is not a night's. | the hub release (attended) | release it (the morning list), read the store back after one carry and one abandonment, then close | CC |
| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. | OPEN | — | one attended pass over the restore paths with these four as tests | CC |
| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. **2026-10-28 night:** all four fixed, each test red first (controller `342a394`, CI 2469 green; `07` §6.3 [DESIGN] 2026-10-28 — an unused copy goes after 14 days, decided by CC unattended). Live on scratch 9202 2026-10-29: H5's busy volume (not written into, the live data put back, no hold; `audits/night-2026-10-28/r1256-restore/h5-live/`) and H8 (the 03:29 sweep removed the two old copies and kept the young one; `…/h8-live/`). H4 and H5's failed stop: tests only; H6: tests only (a live press cannot tell old from new). | OPEN — fixed on main, not released | the controller release (morning list) | the release; then an attended live proof of H4 and H5's failed stop, or revert those two branches of `342a394` | CC |
| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
| **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
| **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
+4
View File
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-10-29 (night)
**Updated 2026-10-28 (evening).** On your yes, the four big updates are live with their undo tested: Wanderer v0.21.2 (offered again, D59 ended; federation from other servers stays off), Termix 26.10.5 (D58 ended), Homepage v2.7.0 and Seerr v3.5.1. Gitea 28.2.0 too (imports only from GitHub, Codeberg, GitLab); Docker Hub no longer refuses the watch in test runs — tonight's run is the check.
## Moved from STATUS on 2026-10-28 (evening)
**Updated 2026-10-28 (day).** On your yes, tested security steps are live for 10 apps (Home Assistant, Immich, AdventureLog, Ghost, MeTube, Zipline, Karakeep, Tandoor, Papra, Dawarich); Wanderer is hidden. The nightly watch now reads the runtime inside every image in CI by itself. Tonight demo-hp takes the Home Assistant and AdventureLog steps; no box runs the other eight apps.