Release run 2026-10-29 close-out: report, STATUS (Updated + the one hold to clear), CONTEXT; closed R-1238/R-1338/R-1256/R-1255, R-528 WATCHING, R-1254 released note; R-1342 filed (fixed on main 652a690); 07 §6.3; evidence
> (`sha256:c79b06e6…`, from `fd75c7c`, CI 2491; rc.1 = `e3fce09` on 9202): per-box floors 09:57 CET, walk green on all
> three + https-only, a local restore on Tester 1, the OOM line (one `app_oom`, one mail), global floor 10:31 CET
> (MinAgent 0.158.1). Catalog: the night's two patches + CHANGELOG (`313eaf6`), Tester 1 read it. Tester 1's recovery code
> re-minted and stored as `TESTER1_RECOVERY_CODE_2026_10_29` (the 10-25 one opens only older packages; removal is the
> operator's, D60). Filed R-1340 (the mixed-hold mail tells the option-C story), R-1341 (the OOM WARN every 30 s); R-1342 (a copy deletable during the stable-start watch; a removed app alarmed — fixed on main, not released); closed R-1238, R-1338, R-1256, R-1255; R-528 WATCHING to its 11-12 check. Open for
> the operator: Radicale's hold record on Tester 1. Report: `REPORT-release-2026-10-29.md`.
> **2026-10-28 night (unattended) — restore edges, escrow edges, the host OOM count on main; NOTHING released.**
> **Decided by CC unattended — operator may reverse:** (1) an unused restore copy goes after 14 days, swept every 6 h
> (`07` §6.3 [DESIGN] 2026-10-28; options 14 d / for ever / 3 d); (2) R-1256's H4 and H5's failed-stop branch stay on main
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-10-29 (night, unattended).** Nothing released; CC decided (you may reverse) that an unused restore copy is deleted after 14 days. On main, each test seen failing first: four restore fixes, two escrow fixes, a memory kill Docker misses now alarms (proven on Tester 1), and Plex's update test works on both test machines; two promises walked, both pass, register 225 → 226 (1 opened, 0 closed).
**Updated 2026-10-29 (attended release).** On your yes: hub 0.175.0, agent 0.168.0 (your vouch), controller 0.347.0 as the floor for every box, and the catalog (Plex night updates; Wanderer's memory figure). D62 is in (restore copies stay 30 days; „Törlés most" deletes one) and D63 held: both untested branches were proven live, nothing reverted. Register 226 → 225 (3 opened, 4 closed); the 10:39 Tester 1 error mail was a false alarm from my test (fixed on main).
## What works
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
@@ -48,7 +48,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. (since 10-14)
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). (since 10-12)
**2026-10-29 morning list.** 1. Release the hub (R-1255). 2. Release the agent, then your vouch (R-1254, R-528). 3. Release the controller (R-1256, R-528). 4. Push the two prepared catalog patches (R-1338, R-1238). If not: the fixes stay unreleased. 5. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes.
**2026-10-29 (release run).** 1. Clear the test app Radicale's restore hold on Tester 1: `ssh root@192.168.0.154 "pct exec 9201 -- docker exec felhom-controller felhom-controller --clear-restore-hold radicale"`, then restart that controller. If not: Radicale stays held there. 2. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes.
**2026-10-23 do list.**
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
2. On the hub's Hosts page: Discard appliance 49. If not: a dead entry alarms.
2026/10/29 09:59:01 [INFO] Event from tester-1: app_held (info) — App radicale is held: held after an off-site restore failed its database replay (2026-10-29T08:18:38Z) — clear the hold to start it
2026/10/29 10:10:31 [INFO] Event from tester-1: restore_copy_deleted (info) — gokapi: the restore copy from before the restore of 2026-10-29T09:08:36Z (61264 bytes) was deleted by hand — the household (dashboard sign-in) from 192.168.0.180. That restore can no longer be undone from this box.
2026/10/29 10:31:31 [INFO] Global controller-version floor set to "0.347.0" (declared MinAgent "0.158.1")
2026/10/29 10:31:31 [INFO] access token "cc-releases-2": POST /configuration/global-floor → 303
2026/10/29 10:39:45 [INFO] Event from tester-1: backup_run_failures (error) — App "gokapi": its unit restore (not stable after its start) put the data back, but the app did not start in time (R-1270). The household was told it has not started yet; the pre-restore copy is kept. If it does not start by itself within 30 minutes, it needs a look. Error: no stable 3m0s within 30m0s
2026/10/29 10:39:45 [INFO] Operator email sent for tester-1/backup_run_failures
2026/10/29 11:07:36 [INFO] Customer tester-1 controller-version floor override set to "" (declared MinAgent "")
2026/10/29 11:07:36 [INFO] access token "cc-releases-2": POST /customers/tester-1/floor → 303
2026/10/29 11:07:37 [INFO] Customer demo-hp controller-version floor override set to "" (declared MinAgent "")
2026/10/29 11:07:37 [INFO] access token "cc-releases-2": POST /customers/demo-hp/floor → 303
2026/10/29 11:07:37 [INFO] Customer demo-felhom controller-version floor override set to "" (declared MinAgent "")
2026/10/29 11:07:37 [INFO] access token "cc-releases-2": POST /customers/demo-felhom/floor → 303
## 2026-10-29 (release run) — R-1238, R-1338, R-1256 and R-1255 closed (full text of the rows: `git show 650aa0cf:documentation/backlog/OPEN-ITEMS.md`)
| Row | What | Closed | Evidence |
|---|---|---|---|
| **R-1338** | **Wanderer's Meilisearch fills whatever memory limit it gets: 92–100 % anon of 256 MB (bench and 9202), still 88.8 % of 384 MB on the bench (2026-10-28)** (P3) | CLOSED 2026-10-29 — measured (the night): the cap was the measurement, not Wanderer; `scripts/memfigure.py` judges `memory.stat` anon; Wanderer re-measured at 27.7 % (`memory_tight` false). Pushed on the operator's yes (catalog `3671802` in `313eaf6`). | `audits/night-2026-10-28/r1338-meili/` |
| **R-1238** | **Plex has no test record and no upgrade fixture, so the night leg never updates Plex on a household box.** (P2) | CLOSED 2026-10-29 — Plex's `update_ladder` entry 1.41.4.9463 → 1.43.5.11029 pushed on the operator's yes (catalog `ddf20f4` in `313eaf6`), proven on the bench and on box 9202 (night 2026-10-28); Tester 1 read the commit (the box writes the entry's digest into the compose). The night leg may now take Plex's next step. | `audits/night-2026-10-28/r1238-plex/`; `audits/d63-2026-10-29/` (the sync) |
| **R-1255** | **Escrow edge cases (review round 2, HUNT-3 H3/H4):** (P3) | CLOSED 2026-10-29 — released in hub **0.175.0** (`sha256:642e94b6…`, manifest `d6352cd8`) on the operator's yes; the two edges are pinned by store tests (red first); one ceremony on Tester 1 ran end to end on 0.175.0, 30 minutes with no unexpected alarm. | `audits/d63-2026-10-29/run-ceremony.log` |
| **R-1256** | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (P3) | CLOSED 2026-10-29 — released in controller **0.347.0**, the global floor since 10:31 CET (MinAgent 0.158.1). All four edges proven live: H5 busy volume + H8 sweep (9202, the night), H4 + H5's failed stop (Tester 1, D63 test build). The sweep age is 30 days (D62). | `audits/night-2026-10-28/r1256-restore/`; `audits/d63-2026-10-29/` |
## 2026-10-28 (evening) — R-1315 closed: Gitea's big step (full text of the row: `git show 1be7aa87:documentation/backlog/OPEN-ITEMS.md`)
| **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).***Original finding, kept:***[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.****What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC****Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC |
| **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)****Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator |
## Apps & catalog — 25 rows (P2 1, P3 13, P4 11)
## Apps & catalog — 24 rows (P2 1, P3 12, P4 11)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -152,10 +152,9 @@ stopping line that lies.
| **R-1331** | Apps & catalog | P3 | **Paperless-ngx 2.20.15 carries Python Brotli 1.1.0 (CVE-2026-111497, fixed 1.1.1/1.2.1)** — its web side only COMPRESSES (whitenoise, compression_middleware); the DECODERS are urllib3's and httpx's, for replies to requests Paperless itself makes (its own outbound calls), never for what a visitor sends. Read in the running container on demo-hp 9201, 2026-10-27. 2026-10-28: the newest release 3.3.4 still carries Brotli 1.2.0 and is a new main version — no fixed release yet (`audits/runtime-fixes-2026-10-28/TABLE.md`). | OPEN — no step tonight (decided by CC unattended — operator may reverse): the bad path needs a hostile server Paperless calls | — | take the first Paperless release with Brotli ≥ 1.2.1 as a normal step | CC |
| **R-1336** | Apps & catalog | P3 | **ActualBudget has no release with the HTTP2Shell fix (CVE-2026-111497): 26.9.0 carries Node.js 24.20.0 and the newest 26.10.0 still Node.js 24.21.0 (fixed: 24.22.1), read from both images.**`node app.js` serves the app; the setup gate covers it until the first account exists, then its own password. No box runs it. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Node ≥ 24.22.1 | the watch lists it nightly (R-1327); take that release as a lane step | CC |
| **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC |
| **R-1338** | Apps & catalog | P3 | **Wanderer's Meilisearch fills whatever memory limit it gets: 92–100 % anon of 256 MB (bench and 9202), still 88.8 % of 384 MB on the bench (2026-10-28)** — it sizes its indexing memory from the memory it sees (its default: 2/3 of the available RAM), so a higher limit alone does not end the `memory_tight` mark of Wanderer's step. No kill in any run. `audits/dockerhub-bigsteps-2026-10-28/wanderer/` | OPEN — **2026-10-28 night: MEASURED — it is the measurement, not Wanderer, and the cap is not needed** (`audits/night-2026-10-28/r1338-meili/`; `09` §3 decision 22 refined). Meilisearch frees with MADV_FREE: the given-back pages stay in `anon` but sit on the inactive FILE list (`file` ~1 MB, `inactive_file` ~290 MB) and the kernel takes them first — 83–96 % `anon` on the bench (no swap) AND on 9202, with and without the cap, 0 kills; it uses ~100 MB. The evening's 17.7 % was the container after the swap, not the cap. **Fixed in the tools:** the mark reads the live anon (catalog `scripts/memfigure.py`, the larger of two estimates, errs high; `upgrade-test.py`, `box_walk.py` soak; `test_memory_live.py` red-proved). Re-run: bench Wanderer v0.20.0 → v0.21.2 `proven`, **no marks** (search 83.3 % anon, 27.7 % live); 9202 95.2 % anon, 25.1 % live. **Prepared, NOT pushed** (the night's catalog fence): the tool change + Wanderer's entry without `memory_tight` (`audits/night-2026-10-28/catalog-morning/`; the catalog gates passed in a scratch clone). **Earlier, 2026-10-28 evening: `MEILI_MAX_INDEXING_MEMORY` = 128 MiB (a third of 384 MB) measured, NOT pushed** (`audits/gitea-wanderer-2026-10-28/wanderer-meili/`): on the bench, under the harness's load, the search container peaked at 17.7 % (84 MB; 88.8 % without the cap, same load); on 9202 the cap was in the container (read back) and it still held 95.7 % anon (363 MB steady over 6 min, no kill). The box result decides: not shown to help, so not pushed. Also: the catalog's ladder has no entry type for a re-measure of a definition change at the SAME digests (`ladder.py`: a re-test whose digest equals its digest_from tests nothing new), so the v0.21.2 entry's `memory_tight` mark cannot be replaced without an image move. | the catalog push (the morning list) | push the prepared catalog change (the live figure + Wanderer's entry re-measured, `memory_tight` off — an existing entry's memory fields edited at the SAME digests, which the gates accept); then close | CC |
## App updates — 10 rows (P2 1, P3 9)
## App updates — 9 rows (P3 9)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
@@ -164,20 +163,19 @@ stopping line that lies.
| **R-1319** | App updates | P3 | **Paperless-ngx 2.20.15: GHSA-2jhj (low; needs a setting we do not set — not reachable); the fix is 3.3.4, a big step** (encryption and API v1 removed, the search index rebuilt). It runs on all four test boxes. Not urgent. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | the big step 3.3.4 when the catalog takes Paperless 3 for other reasons; read our API v1 use first | CC |
| **R-1321** | App updates | P3 | **Tandoor 2.6.15: GHSA-5f74 (high, an XXE file read) and GHSA-69q3 (any user); the fix is 2.7.2** — 2.7.0 removes settings and an endpoint and rewrites the login pages (our probe path — not sure), and 2.7.0/2.7.1 break invite links with sign-up off, so straight to 2.7.2. No box runs it. `audits/security-app-updates-2026-10-27/TRIAGE.md` | OPEN — filed 2026-10-27 (security app updates, Part 3: not done today) | — | a normal step to 2.7.2 with the probe path checked; bench + 9202 (+ undo if a migration); a STOP | CC |
| **R-1324** | App updates | P3 | **RomM 5.4's first start can run a 15–20 minute migration on a big library (upstream notes) — longer than the update's check window, so a household's night step could be undone for slowness, not for a fault.** Today's walk (9202, a small seed) took the step in seconds; demo-hp's RomM takes it tonight (security-fix step, catalog `06e603e`). `audits/security-app-updates-2026-10-27/triage-notes/romm.md` | OPEN — filed 2026-10-27; WATCH demo-hp's night leg line | — | read demo-hp's night leg line for romm (done / undone, the time); if undone for time: give the step a longer check or the `needs_person` mark | CC |
| **R-1238** | App updates | P2 | **Plex has no test record and no upgrade fixture, so the night leg never updates Plex on a household box.** Filed 2026-10-23. Catalog `6cc317d` moved Plex 1.41.4.9463 → 1.43.5.11029-6bf8af3f0 (the Plex2Shell fix) on the operator's ruling, with NO `update_ladder` entry (the move gate refused; pushed `--no-verify`, the one time). With no ladder, a person's Update press pins the current compose (`stacks/ladder.go``nextLadderStep`), but the night leg skips the app (`unattended.go` → `LegSkipNoTestRecord`), so every future Plex fix reaches a household only by a hand press. `scripts/upgrade_fixtures.py` has no Plex fixture. A likely seed route is NOT measured: an UNCLAIMED Plex answers a private-address request with no token (`architecture/13-threat-model.md`, the R-1045 row), so the bench may create and read back a library section through Plex's own API without a plex.tv account. **Needs a spike:** prove that route on the bench (seed a section, swap the image, read it back), add the fixture, then `upgrade-test.py --write-ladder` for the current pin — the box walk on 9202 needs the same route behind the setup gate. Evidence: `audits/plex-demo-boxes-2026-10-23/`. **2026-10-23 (later): the NEXT Plex fix need not wait for this** — the security-fix lane (`09` §3 decision 255, D39 = B, controller 0.341.0) takes a box-only step with `seed: "none"` at night (proven on 9202 with Plex 1.43.4 → 1.43.5, `audits/security-lane-2026-10-23/`). The bench fixture stays the aim: it is the only check that reads Plex's data back after a step. **2026-10-28 night: the route works, unclaimed.** Plex refuses the request through its front door (401: the proxy's address is not taken as local), so the fixture asks from Plex's own container (`POST /library/sections` → 201) and reads the section back by its uuid. Bench 1.41.4.9463 → 1.43.5.11029 proven (`audits/night-2026-10-28/bench/plex/edge/`); box walk on scratch 9202 through the setup gate proven 2026-10-29 (the household's Update, 41 s; the section read back on 1.43.5 with the same uuid; 10-min soak 10.6 %, no kill; `…/r1238-plex/box-9202/`). The fixtures and the ladder entry are a PREPARED patch (`…/catalog-morning/`), not pushed — the night allowed no live catalog push. | OPEN — proven on both venues, not pushed | the morning catalog push | push the two prepared catalog commits (the operator's word or an attended session) | CC |
| **R-440** | App updates | P3 | **[P2-MEDIUM] 23 catalog image pins float, so an update is not reproducible.** `compose pull` on a moving tag fetches whatever upstream published that day. **MEASURED 2026-09-01 over `app-catalog-felhom.eu` @ `29edad9c5bf4`: 79 `image:` lines across 53 apps, 66 distinct; 23 of those lines carry a tag with no patch version.**`postgres:16-alpine` (8 apps), `redis:7-alpine` (6), `mariadb:11.6` (2), plus one each of `postgres:15-alpine`, `postgis/postgis:16-3.5-alpine`, `mariadb:11.4`, `mariadb:12.3`, `ghcr.io/claperco/claper:2.5`, `ghcr.io/thomiceli/opengist:1.13`, `wger/server:2.6`. **A 24th is arguable and is recorded rather than rounded away:**`ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0` pins both extensions exactly but leaves the PostgreSQL patch floating. A customer pressing Frissites can therefore swap their DATABASE ENGINE build with no catalog change and no record; two boxes updated on two days end up different. **Severity MEDIUM on its own; it becomes BLOCKING the moment a pre-update copy exists**, because "what did we upgrade from and to" must be recordable and today it is not — which is also why R-440 must be read next to the digest discipline in Rule 10 of the spike. **MEASURED LIVE 2026-09-01 — the floating pins have ALREADY moved, with a passing control.** Running digests on demo-hp compared against what the registry serves for the same tag today: **`mariadb:11.4` MOVED** (`sha256:4f1d8d20...` -> `sha256:611a2fcc...`) and **`mariadb:12.3` MOVED** (`sha256:a02fe89c...` -> `sha256:dd9b303a...`), while `postgres:16-alpine`, `redis:7-alpine`, `mariadb:11.6` and `opengist:1.13` were SAME — **and both fully-pinned CONTROLS (`rommapp/romm:5.0.0`, `privatebin/pdo:2.0.5`) were SAME.** So on a box with ZERO visible drift by tag, pressing Frissites today silently swaps the DATABASE ENGINE build under `romm` and `bookstack`, with no catalog change and no record. **Compounding fact found while reading:** the recovery unit records `ImagePins` but the manifest comment says *"image NOT stored - re-pulled on restore"*, so a RESTORE of a floating-pinned app also re-pulls whatever is current — the same non-reproducibility on the recovery path. **HALF OF THE ANSWER SHIPPED 2026-09-02 (controller v0.233.0, slice 1): `app.yaml.installed_images` now records, per compose SERVICE, the reference AND the repo digest each container was actually created from — so "what did we upgrade FROM" is answerable on any box that has taken one lifecycle action since the upgrade.** What is still missing is the other half: comparing that digest against what the registry serves for the same tag TODAY, which needs a network call the render path deliberately does not make (see R-446). **The row therefore stays OPEN and its rank is unchanged** — recording a digest does not make a floating pin reproducible; it makes the drift measurable after the fact. `audits/SPIKE-app-update-2026-09-01.md`**— NIGHT 2026-09-23:** every image moved tonight (and the 21 backfilled moves) carries its resolved digest in the catalog's test record; the floating pins themselves still float until the box pulls by digest (`09` §6.4 part 6). **-- MEASURED 2026-09-30 on demo-hp 9201: a fresh install or a guarded Update renders `name:tag@sha256` from the ladder (`digest.go``RenderWithLadderDigests`) — 9 of 20 services there run such a definition (adventurelog, docmost, paperless-ngx). The other 11 run TAG-ONLY definitions: bookstack, kimai, opengist, privatebin, romm were installed before v0.269.0 and not updated since, and `CarryDigests` (`digest.go:141`) keeps only a digest the running definition already has; bentopdf and calibre-web have no ladder at all. So a re-pull of those (a restore) takes whatever the tag serves that day. `audits/pg-last-six-2026-09-30/E/E2-digests-demo-hp.txt`.** **-- 2026-09-30 (evening):** calibre-web now has a ladder entry (`53a4a1d`), so a new install or a guarded Update renders its tested digest; bentopdf still has none. `audits/more-night-apps-2026-09-30/`**Merged 2026-10-05 from R-446 (duplicate):** the household-visible consequence — for these apps the „Naprakész" badge can never turn „behind" (`felhom-controller controller/internal/stacks/updateorder.go:96` returns false with no catalog digests / no test date); R-740 (same-tag re-test) is closed, so floating-tag drift for laddered apps is handled. | **NARROWED 2026-09-30 — floating only for (1) apps with no ladder entry and (2) apps installed before v0.269.0 until their next guarded Update; owner: CC.****Re-ranked 2026-10-03: P2→P3: narrowed to apps with no ladder entry and pre-v0.269 installs; each guarded update fixes the latter.** | — | — | CC |
| **R-462** | App updates | P3 | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.****The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.**`audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s,Line truncated
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:****OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | NARROWED — **2026-10-23 (operator yes D33, D34):** the hub DB now leaves every 6 hours (hub 0.167.0 + `scripts/hub-db-backup/`, alarm at 13 h; restored back byte-identical) and the released images and agent packages leave nightly (`scripts/release-copy/`, R-1227 closed). (c)–(g) unchanged. **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/`**LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest **2026-10-24 (attended): `DooplexGiteaOffsiteStale` fired from 00:57Z — not this path's fault.** The 00:20 push refused, as designed, a 12 h old Gitea dump: the PostgreSQL dump CronJob had failed every run since 2026-10-23 16:00Z (`28P01`) because the window reboot restarted the database and CNPG put the bootstrap secret's password in force (homelab `531023d`, operator's pick: the CronJob only). Fixed; dump, push and restore test run by hand (26 456 files, 12 repos `git fsck`, `gitea.dump` 1114 entries); cleared 10:51Z (`audits/dooplex-alarms-2026-10-24/`). Backup alarms now arrive as ONE mail every 4 h (homelab `22d7697`). **2026-10-23 night — survey again, read only (`audits/dooplex-survival-2026-10-22/PLAN.md`):** both nightly off-site jobs Line truncated
| **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):**`_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). **FIRST VERDICT READ 2026-10-21 (rehearsal session, read-only):** the 07:10 CEST run judged three tiers against yesterday — demo-felhom 5, demo-hp 14, tester-1 4 new snapshot files, „each inside a reported box run"; nothing vanished, nothing unexplained; Tester4 stored its first baseline (44 files); Tester-2's own listing failed (no `snapshots/` dir — the box has never pushed). No `offsite_snapshots_vanished` / `_unexplained` mail in the operator mailbox (second channel). Evidence `audits/due-checks-2026-10-21/R-895-hub-0710.txt`. The positive alarm path is still proven only by the unit red-proofs. | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC |
| **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | NARROWED — **2026-10-23 (D32):** the sheet is current — names and places only, one A4, HU + EN (`runbooks/break-glass-sheet.md`); every place checked to exist. LEFT: the operator prints it, fills it by hand, keeps it outside the home. **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). **2026-10-23 night — survey (`audits/dooplex-survival-2026-10-22/PLAN.md`, G1/G3/G10):** unchanged; the sheet is still the one act that breaks the circle, and S5's offline form is unknown to CC — on the sheet too (decide list in STATUS). | operator: print the sheet | Operator: print + store the sheet; then close | operator |
| **R-1254** | Backup & restore | P2 | **The hub saves a household's new escrow package in TWO writes; a failure between them leaves the new repository hash beside the OLD identity package, and the box then marks itself escrowed — on a total loss the written code opens nothing.** Found 2026-10-24 (HUNT-3 H1), read at `file:line`: `hub/internal/api/handler.go:1415``SaveHostEscrow` commits blob + hash, `:1445``SaveHostDRBundle` sets `identity_blob` in a second statement (`internal/store/store.go:3788-3796`; the upsert `:3490-3511` never touches it); the controller's confirm reads the hash (`felhom-controller/controller/internal/report/escrow_confirm.go:195-217`); the agent sends once. Same family (H2): a lost or late answer (30 s) tells the household „the previous code still works” when the hub may already hold the new package (`felhom-agent/internal/localapi/escrow_ceremony.go:79,532-534,639-641`). Static. **The hub half is fixed on main 2026-10-24 night** (felhom.eu `dcf4f84c`; Part 3): key, hash and identity blob in ONE transaction (`SaveHostEscrowWithIdentity`), a bad identity refused before any write; hub image `0.170.0-rc.r1254` built locally (`67db584a45d7`), NOT pushed, NOT deployed. The agent ceremony half (H2: „the previous code still works” after a lost answer) is still open. | **NARROWED** — **the hub half CLOSED 2026-10-25:** hub **0.170.0** deployed on the operator's yes (manifest bump, ArgoCD sync of the hub Deployment only, image digest `6478498d…`; 30 minutes and more with no alarm mail, the mailbox as the control); Tester 1 ran the escrow ceremony end to end through the page's endpoints and the hub stored the new package in one transaction („the new package is on the hub", `audits/release-2026-10-26/escrow-ceremony-tester1.txt`). The ceremony half stays open. **2026-10-28 night: the ceremony half FIXED ON MAIN** (agent `5ddd124` CI 2473, controller `e33cffa` CI 2474; tests red first; `03` §8 [DESIGN] 2026-10-28): after a failed upload the agent asks the hub which package it holds (the identity blob compared) — the new one → success, another → 502 and „the previous code still works", no answer → 504 that promises nothing; the controller words 504 and its own failed call as „Nem tudjuk biztosan… Őrizd meg mindkét kódot…" and waits 60 s for the claim/confirm (was 15 s, shorter than the agent's 30 s upload). | the hub release (attended); the agent half | deploy hub v0.170.0 (Monday list); then the ceremony asks the hub what it holds before it says „the previous code still works” | CC |
| **R-1254** | Backup & restore | P2 | **The hub saves a household's new escrow package in TWO writes; a failure between them leaves the new repository hash beside the OLD identity package, and the box then marks itself escrowed — on a total loss the written code opens nothing.** Found 2026-10-24 (HUNT-3 H1), read at `file:line`: `hub/internal/api/handler.go:1415``SaveHostEscrow` commits blob + hash, `:1445``SaveHostDRBundle` sets `identity_blob` in a second statement (`internal/store/store.go:3788-3796`; the upsert `:3490-3511` never touches it); the controller's confirm reads the hash (`felhom-controller/controller/internal/report/escrow_confirm.go:195-217`); the agent sends once. Same family (H2): a lost or late answer (30 s) tells the household „the previous code still works” when the hub may already hold the new package (`felhom-agent/internal/localapi/escrow_ceremony.go:79,532-534,639-641`). Static. **The hub half is fixed on main 2026-10-24 night** (felhom.eu `dcf4f84c`; Part 3): key, hash and identity blob in ONE transaction (`SaveHostEscrowWithIdentity`), a bad identity refused before any write; hub image `0.170.0-rc.r1254` built locally (`67db584a45d7`), NOT pushed, NOT deployed. The agent ceremony half (H2: „the previous code still works” after a lost answer) is still open. | **NARROWED** — **released 2026-10-29:** agent 0.168.0 (vouched) + controller 0.347.0 (the floor); a ceremony ran on both (the happy path); the lost-answer branch has tests only (a live proof needs a hub answer lost on purpose). **the hub half CLOSED 2026-10-25:** hub **0.170.0** deployed on the operator's yes (manifest bump, ArgoCD sync of the hub Deployment only, image digest `6478498d…`; 30 minutes and more with no alarm mail, the mailbox as the control); Tester 1 ran the escrow ceremony end to end through the page's endpoints and the hub stored the new package in one transaction („the new package is on the hub", `audits/release-2026-10-26/escrow-ceremony-tester1.txt`). The ceremony half stays open. **2026-10-28 night: the ceremony half FIXED ON MAIN** (agent `5ddd124` CI 2473, controller `e33cffa` CI 2474; tests red first; `03` §8 [DESIGN] 2026-10-28): after a failed upload the agent asks the hub which package it holds (the identity blob compared) — the new one → success, another → 502 and „the previous code still works", no answer → 504 that promises nothing; the controller words 504 and its own failed call as „Nem tudjuk biztosan… Őrizd meg mindkét kódot…" and waits 60 s for the claim/confirm (was 15 s, shorter than the agent's 30 s upload). | the hub release (attended); the agent half | deploy hub v0.170.0 (Monday list); then the ceremony asks the hub what it holds before it says „the previous code still works” | CC |
| **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.****VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3****2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC |
| **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC |
| **R-409** | Backup & restore | P3 | **Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it.** MEASURED on demo-hp 2026-08-31 against kimai's restored unit: `manifest.json`'s `checksums` object carries sha256 for `.felhom.yml` (2 235 B), `app.yaml` (488 B) and `docker-compose.yml` (2 195 B) — **4 918 bytes of a 213 231 242-byte unit**. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. **And nothing else supplies one:** restic 0.14.0's `restore --verify` is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and `restic ls --json` file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and **no content hash**. **So "the restore produced correct files" is currently unanswerable by any automated means.****What is NOT claimed here:**`restic check --read-data-subset=100%` already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. | **OPEN — MEDIUM** | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's `checksums` to cover `db_dumps` and `volume_dumps` — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: `audits/SPIKE-restic-restore-test-2026-08-31.md` §Q2, §Q3. | CC |
@@ -185,8 +183,6 @@ stopping line that lies.
| **R-540** | Backup & restore | P3 | **[P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills.** Read from source 2026-09-16 while making off-site the default: `HETZNER_POOL_BOX_ID` is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, `monitor/offsite.go`) tells the operator it is filling but nothing says which box a new customer should land on. **Needs a selection rule** (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. | **READY — rank P3-LOW; owner: CC (hub)****2026-10-05 (burn-down night): NEEDS A DESIGN (and money).** A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC |
| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC****2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. | — | — | CC |
| **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) |
| **R-1255** | Backup & restore | P3 | **Escrow edge cases (review round 2, HUNT-3 H3/H4):** the R-1176 carry moves the newest kept package even when it holds no repository password (`hub/internal/store/escrow_carry.go:100-110` checks `identity_blob` only), while the right one stays under the deleted host (`store.go:3533-3534`); two set-asides on the same UTC day reuse a name and the purge can delete a still-stored copy's package (`hub/internal/offsitekeys/offsitekeys.go:378-385`; `store.go:4295-4300`). Static. | OPEN — **fixed on main 2026-10-28 night, not released** (felhom.eu `1ee6d5a5`, CI 2472; store tests red first; `03` §8 [DESIGN] 2026-10-28): the carry takes the newest retained row with an identity AND a repository-password hash (the newest row only when none has a hash — a legacy package, as before); a set-aside is `<repo>.orphaned-<YYYYMMDDTHHMMSSZ>`; a retained row is purged only when it was tagged no later than its path's deletion (`repo_tagged_at`, new column). No live proof: a hub deploy is not a night's. | the hub release (attended) | release it (the morning list), read the store back after one carry and one abandonment, then close | CC |
| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. **2026-10-28 night:** all four fixed, each test red first (controller `342a394`, CI 2469 green; `07` §6.3 [DESIGN] 2026-10-28 — an unused copy goes after 14 days, decided by CC unattended). Live on scratch 9202 2026-10-29: H5's busy volume (not written into, the live data put back, no hold; `audits/night-2026-10-28/r1256-restore/h5-live/`) and H8 (the 03:29 sweep removed the two old copies and kept the young one; `…/h8-live/`). H4 and H5's failed stop: tests only; H6: tests only (a live press cannot tell old from new). | OPEN — fixed on main (controller `342a394`), not released. **All four proven live:** H5 busy volume + H8 sweep (9202, night 2026-10-28); **H4 and H5's failed stop on Tester 1, 2026-10-29 08:10 / 08:18 UTC** (D63, a `-tags faultinject` test build; `audits/d63-2026-10-29/`). D62 set the sweep age to 30 days (`e3fce09`). Closes with the controller release | the controller release (morning list) | the release; then an attended live proof of H4 and H5's failed stop, or revert those two branches of `342a394` | CC |
| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
| **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
| **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
@@ -221,6 +217,7 @@ stopping line that lies.
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **OPEN — the grant is SET (D55 = 260, `09` §3 decision 268)**: 2026-10-26 19:10:31Z through the operator's one-purpose token `cc-d55-window`, read back `max:260`; it is used at demo-hp's next window request (after its nightly off-site run, ~03:15Z). Close when that window prunes; then the operator revokes the token. **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 evening: CC could NOT set the D54 grant — `POST /offsite/window-grant/demo-hp` (`max_remove=220`) with CC's hub token answered **403 „missing scope offsite:write"** (the hub log shows the refusal; nothing granted). The plan also grows: restic's dry-run 2026-10-26 18:56Z still 359 copies / plan 209 / default cap 179 (before tonight's run); a simulation of the keep rule adds ~14 per night (14 of 18 groups back up nightly), so ~223 at the night of 10-27 and ~237 at 10-28 — a cap of 220 would be refused again. STATUS D55: the operator grants it, with the cap. The grant opens the box's NEXT window request, which comes after its next nightly off-site run. Ruled D54 = prune once (`09` §3 decision 266). **Earlier:** WAITING-ON-OPERATOR — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
| **R-1340** | Backup & restore | P3 | **The operator's „HELD STOPPED" mail for a mixed restore says the DATABASE restore failed and was rolled back, and that the files and volumes are the snapshot's — whatever the cause.** Measured 2026-10-29 08:18 UTC on Tester 1 (D63's H5 live proof, test build): Radicale (no database) — the volume could not be removed (not written into), the put-back's stop failed, nothing moved; the mail read „its off-site database restore failed. The database was rolled back to the pre-restore copy and the live definition written back, but its files and named volumes are the snapshot's". The text (`controller/cmd/controller/main.go:1555`) predates the D20/D22 put-back, when the only mixed hold was R-893 option C. Support reads a wrong account of what is on the box. The household's sentence is conservative and stays. Fix: the operator text names the cause it was given (the replay error, the put-back step that failed, whether the copy is kept and where) instead of the option-C story; a test per cause. Evidence: `audits/d63-2026-10-29/run-h5.log`. | OPEN | — | the fix above, red first | CC |
| **R-1342** | Backup & restore | P3 | **For 30 minutes after a restore the box watches whether the app runs stably, and its restore copy is the way back — but (1) D62's „Törlés most" could delete that copy during the watch, and (2) an app REMOVED during the watch made the box mail „did not start … the pre-restore copy is kept" when the watch ran out.** Found live on Tester 1, 2026-10-29: Gokapi restored 10:08 CET, its copy deleted by hand 10:10 (D62's own live proof), the app removed 10:11; at 10:39 CET the operator got a `backup_run_failures` error mail — both halves of its sentence false. (1) is a gap in D62 (released in controller 0.347.0): the operator's rule is „never while an operation runs on the app"; (2) is older (the R-1270 watch, since 2026-10-26). **Fixed on main** (controller `652a690`, not released — one release per session): the watch marks its copy (`Manager.watching`, under `m.mu`), the list reads it „checking" with one line and no delete date, the delete refuses it; the watch ends quietly when the provider says the app was removed (`StackRemovalReporter`, implemented by the production adapter — a compile-time assertion). Tests `TestR1342_*`, each seen failing by a mutation. Evidence: `audits/d63-2026-10-29/` (hub log of the mail, `run-p4b-d62.log`). | OPEN — fixed on main, not released; rides the next controller release | — | release it (next controller release) | CC |
## Storage & devices — 14 rows (P3 4, P4 10)
@@ -303,7 +300,6 @@ stopping line that lies.
| **R-1310** | Monitoring & notifications | P3 | **The security watch (R-1303) flags notices that do not apply and misses fixes that do — measured by reading all 38 flagged pins by hand on 2026-10-27.** False: advisories of ecosystem `actions` (Home Assistant's CI), of another package in the repo (Gitea's runner), with a hostname or ≤-old range (cal.com 2023/2024, Grafana 2022), for the desktop/Electron client (Termix), for the rootless image only (Uptime Kuma); a release hit on a word (Dawarich „no longer receives security fixes", Rallly v4.16.0). Wrong „no fix": Vikunja's six (range `<= 2.6.0`, fixed in 2.7.0), karakeep-chrome (154/156 exist). Missed: Tandoor 2.7.0 names 8 GHSA ids (1 matched), Dawarich 1.16.0 (2FA, export deletion), Wishlist v0.69.0 (critical, no GHSA). BEHIND reads linuxserver `26.09.20260929` as newer than 26.09.2; Seerr follows the renamed-away `fallenbagel/jellyseerr`. `audits/security-app-updates-2026-10-27/TRIAGE.md` NOT-A-FINDING list and `triage-notes/` | OPEN — filed 2026-10-27; **two shapes FIXED the same day** (ecosystem `actions` not listed; an empty fix with a closed range reads „after X"; test W9, red first) | — | teach `security_watch.py`: skip a package that is not the pinned one; match every GHSA id a release body names; ignore `YY.MM.YYYYMMDD` tags; repoint seerr to `seerr-team/seerr`; each with a test seen failing | CC |
| **R-1334** | Monitoring & notifications | P3 | **Docker Hub's tag API refuses the watch (HTTP 429) since the DooPlex group was added: 19 DooPlex images NOT CHECKED in the 2026-10-28 night run (job 2446, `hub.docker.com` = 211 requests), and the „could not check registry:docker.io" mail went out (streak 2).** The runtime part is not affected (HEAD + cache). ghcr.io also answered one blob with 429 in the dry run of 07:02 (Immich v3.3.5, while this session read images from the same address). | **VERIFY** — BUILT 2026-10-28 midday, measured first (`audits/dockerhub-bigsteps-2026-10-28/part1/`): the 429s came from the BURST, not the count — night 2446 sent 211 tag-list requests in ~2 min from six workers (15 refused), dry run 2450 sent 224 in 3.5 min (none). `fe677bae`: Docker Hub requests spaced (1/s across the workers), a 429 retried with its Retry-After (NOT CHECKED only after the 4th try), tag lists cached 24 h per registry path on the runner (two names of one path fetch once); `8644450d`: the image reader retries a registry 429 too (ghcr.io refused two blob reads in back-to-back dry runs). Tests W14 + retry checks, red first. Dry runs 2458 and 2459: **0 Docker Hub 429, 0 retries** (210 requests over ~7 min). **No Docker Hub login needed.** Each of the two CI runners keeps its own tag cache. `minio/minio:latest` answers 401 — Docker Hub no longer serves it (homelab, R-1328). | — | close after the first CI night with 0 Docker Hub 429 (the 03:40 run of 2026-10-29) | CC |
| **R-1020** | Monitoring & notifications | P4 | **`app_repaired_after_crash` reports a duration of 0 s, and the controller log says „in 0s".** `RunAfterUncleanBoot` sets `res.Duration` in a deferred func after the result is already returned (no named result), so the caller sees 0; the `app.yaml` record is right (10,172 ms on demo-hp 2026-10-14, `audits/immich-repair-2026-10-14/D/`). Small, but the controller release for this session is spent (v0.311.0). | **OPEN** — filed 2026-10-14 | — | CC: a named result (or set the duration before return), with a test on the returned value; ride the next controller release | CC |
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | OPEN — **built 2026-10-28 night, not released** (agent `6f933d6` CI 2475: `GET /guest/oom-kills`, the host's per-container `oom_kill`; controller `eed23e7` CI 2476: the 30-s scan raises `app_oom` for a counted kill whatever Docker's flag says, `source` in the details; `08` §5 [DESIGN] 2026-10-28). **Live 2026-10-28 23:27 on Tester 1 with test builds of both (put back after):** Radicale capped at 128 MB, a hog inside it killed, the host counter 0 → 1, ONE `app_oom` „seen by docker,cgroup", the operator's mailbox held exactly one mail with `"source":"docker,cgroup"` (`audits/night-2026-10-28/r528-oom/`). Docker 29.8.2 also flagged it, so the cgroup-only path is proven by tests. Closes after the agent release (vouch) and the controller release. **Earlier: WATCHING — DESIGN WRITTEN 2026-10-26 night (the brief's Part 7: build only if time is left; it was not built — the line test and the attended release came first).** The host-side counter EXISTS and is readable (read on demo-hp 2026-10-26): `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<64-hex id>.scope/memory.events`, line `oom_kill N`. **Design:** (1) agent: a read-only collector (no root CLI — a sysfs read under the agent's own rights; if not readable, the existing sudoers fence gets one read-only `Cmnd_Alias`) that, for each guest in the `felhom` pool, globs those scopes and keeps `{container_id: oom_kill}`; served on the guest's own local API as `GET /guest/oom-kills` (`withGuest`, the guest's containers only). (2) controller: the 30-s dead-app scan asks once per pass; a RISE of a container's counter = an OOM kill whatever Docker's `OOMKilled` says → the existing `app_oom` line (R-636's per-run grain), `source: cgroup` in the details. An agent without the route = today's behaviour. (3) Tests: a fake cgroup tree (agent), a rising counter → one `app_oom` (controller, red-first). (4) Live: 9202, a throwaway app capped at 64M with a memory hog: the counter rises, `app_oom` names the container while `OOMKilled` reads false. ~1 day, two repos, one agent release. **Earlier:** WATCHING — **2026-10-12: the built part is released** (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing `oom_check` before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. **READY — rank P2-MEDIUM; owner: CC****2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely.** On `tester-1-022354` (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed `app_oom` (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was `write CONNECTION_CLOSED immich-postgres:5432` and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: `audits/evidence-chaos-night-2026-09-17/round-2.txt`. **2026-10-05 (burn-down night): a one-page design proposal (no code) is in `audits/night-burndown-2026-10-05/design-R-528.md`** — for the operator. **2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design** (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the `OOMKilled` flag was TRUE in all four shapes trLine truncated
| **R-1193** | Monitoring & notifications | P3 | **When the box froze (R-1192), the operator heard only „no report for 45 m" — 28 minutes in — and the household heard nothing at all.** Measured 2026-10-20 against the operator mailbox (the second channel): the freeze began ~13:05 UTC; the hub page read ONLINE / OK at 13:22 (correct inside the 45-minute threshold, R-549); `host_stale` at 13:33, `app_oom` at 13:34 (immich-machine-learning killed by its memory limit), `host_crash_restart` at 14:26 and 16:29 (the two plug pulls), `node_stale` 15:59 / `node_recovered` 16:03. No mail or banner tells the household that its box is not answering, and no operator mail says „the box answers ping but not its dashboard". CORRECTED the same day: the first version of this row said the hub showed OK „throughout" and „nobody was told" — that read only the hub page, not the mailbox. Evidence `audits/rehearsal-2026-10-20/journal.md`. | OPEN | — | Design note (2026-10-21, after R-1192's diagnosis): the freeze's signature is cheap to see from the HOST side — the guest cgroup's memory PSI „full" above ~50 % for a minute, or its `memory.events max` climbing, while the host still reports; the agent could raise `guest_memory_thrash` (operator mail in its own words, not „no report for 45 m") and the household banner „A dobozod most nagyon lassú…". With the swap fix (R-1192) the measured thrash ends within a minute, so this alarm should be built only if a freeze is seen again after the agent release; until then it stays a design. Decide whether a dashboard that does not answer for N minutes while the host still reports should alarm sooner and in its own words (guest-side liveness), and whether the household is told | CC |
| **R-1202** | Monitoring & notifications | P3 | **A box whose clock is behind blames the hub and stops an app without telling anyone.** Rehearsal 2026-10-20 B6 #8: RTC set 2 years back (the OS raised it to 2026-04-13): the dashboard shows only „Hub nem elérhető" while the log says „x509: certificate … not yet valid: current time 2026-04-13…"; Paperless crash-looped and the box stopped it with a restore hold; the `app_stopped_unhealthy` event failed (hub unreachable), so neither the household nor the operator heard. After the real time came back the app stayed stopped until a Start (clear sentence, 1 min). | **OPEN** | — | say „the box's clock is wrong (it says <date>)" when TLS fails with not-yet-valid/expired against a hub that answers; keep the stop event queued until the hub is reachable | CC |
| **R-340** | Monitoring & notifications | P3 | **The new reachability check does not touch the surface that actually failed.** R-339 reports when the hub cannot READ ep0 — but the read it performs is the `usage` op, which is `proxmox-backup-manager` plus `df` over SSH, and therefore rides the **local API daemon**. The 2026-08-18 incident explicitly CLEARED that daemon: `proxmox-backup.service` was healthy throughout, and it was the **HTTPS proxy on 8007** that was wedged with a full accept queue. **So R-339's check would have returned green for all 9 h 37 m of that outage.** It closes the case where ep0 is unreachable *as a host*; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier | **READY (M) — NEW 2026-08-18** | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout `ErrUsageUnsupported` already models) | Add a **health op** to `scripts/felhom-tenantsync.sh` that probes `https://127.0.0.1:8007/` on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. **Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice****REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time.** Available in `audits/evidence-ep0-established-connections-2026-08-20/`: the proxy **fd count** and its type breakdown (`lsof` + `/proc/<pid>/fd`), the **listen-queue depth** (`ss -lnt` — `Recv-Q 0`, `Send-Q 1024`), the **ESTAB/CLOSE-WAIT split**, the **per-peer** connection histogram, a **31-minute persistence diff** of full 4-tuples, and a **46.18 h** slope with Poisson bounds. What the health op would still add beyond these: a loopback `GET https://127.0.0.1:8007/` probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. **And this spike sharpens what the op should report:** a rising **ESTAB** count is the live signal (CLOSE-WAIT was **0**, not merely flat), and per **R-344** the fd ceiling that matters may be the **agent's**, not only ep0's. | CC |
@@ -323,6 +319,7 @@ stopping line that lies.
| **R-1296** | Monitoring & notifications | P4 | **When ep0's tenancy script refuses, the hub log shows only its one-line error — PBS's own reason (the 403 `missing Datastore.Audit|Datastore.Backup on …`) stays in the script's stderr, which the hub drops on a refusal.** Found 2026-10-26 diagnosing R-1295: the cause was visible only by reading ep0's proxy journal over root SSH. `hub/internal/tenantsync/client.go``parseResponse` returns the JSON error and ignores stderr; the script keeps secrets off stderr by design (its header), so the tail is safe to log. | OPEN | — | append the last ~3 stderr lines to the refusal error (bounded, never stdout); a test that a secret on stdout never appears | CC |
| **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC |
| **R-1341** | Monitoring & notifications | P4 | **The OOM scan logs „container X was OOM-killed" every 30 s for as long as the container lives, though it sends the event once.** Measured 2026-10-29 on Tester 1 (controller 0.347.0, agent 0.168.0, R-528's proof): one kill → one `app_oom` and one operator mail (right), and the `[WARN] [deadapp]` line at 09:02:32, :03:02, :03:32, :04:02, :04:32 … — the same as the night's proof (`audits/night-2026-10-28/r528-oom/`). The loop (`controller/cmd/controller/main.go:1100`, since v0.243.0, R-514) logs every container `ScanOOMKilled` returns, every scan; the notifier's cooldown keeps the event to one. About 2 880 lines a day per OOM-marked container until it restarts — noise that hides the next real line. Fix: log once per (container, start time, kill count), with a test that a second scan of the same kill logs nothing and a new kill logs again. Evidence: `audits/d63-2026-10-29/run-p4-oom.log`. | OPEN | — | the fix above, red first | CC |
| **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
@@ -5,6 +5,12 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
## Moved from STATUS on 2026-10-29 (release run)
**Updated 2026-10-29 (night, unattended).** Nothing released; CC decided (you may reverse) that an unused restore copy is deleted after 14 days. On main, each test seen failing first: four restore fixes, two escrow fixes, a memory kill Docker misses now alarms (proven on Tester 1), and Plex's update test works on both test machines; two promises walked, both pass, register 225 → 226 (1 opened, 0 closed).
**2026-10-29 morning list.** 1. Release the hub (R-1255). 2. Release the agent, then your vouch (R-1254, R-528). 3. Release the controller (R-1256, R-528). 4. Push the two prepared catalog patches (R-1338, R-1238). If not: the fixes stay unreleased. 5. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes.
## Moved from STATUS on 2026-10-29 (night)
**Updated 2026-10-28 (evening).** On your yes, the four big updates are live with their undo tested: Wanderer v0.21.2 (offered again, D59 ended; federation from other servers stays off), Termix 26.10.5 (D58 ended), Homepage v2.7.0 and Seerr v3.5.1. Gitea 28.2.0 too (imports only from GitHub, Codeberg, GitLab); Docker Hub no longer refuses the watch in test runs — tonight's run is the check.
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
admin committed