docs: operator restart + clear hold [DESIGN] lines (03, 05, 08); R-1346/R-1347/R-1348 filed; release-2026-10-29b evidence (restart presses, clear_hold, health log so far, controller 0.348.0 walk)

documentation/backlog/OPEN-ITEMS.md

@@ -154,7 +154,7 @@ stopping line that lies.
 | **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC |
 
 
-## App updates — 9 rows (P3 9)
+## App updates — 10 rows (P3 10)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -167,8 +167,9 @@ stopping line that lies.
 | **R-462** | App updates | P3 | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s,Line truncated
 | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
 | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. **2026-10-27 (security triage):** two notices with NO fix released (GHSA-p8cf-g8rv-gvrq: a user overwrites another user's Withings link; GHSA-cmmf-53wf-r5gv: owner-only photos readable by URL) — both need a household account or a leaked URL; pick in `audits/security-app-updates-2026-10-27/PICKS.md` (CC's pick: leave it offered, take the fix with this big step). **2026-10-28 (R-1332): HTTP2Shell has NO fixed release yet** — v0.17.3's server carries Node.js 24.18.0 (`tsx index.ts`, behind the setup gate) and the newest v1.9.0 still Node.js 24.22.0 (fixed: 24.22.1); the web image is nginx, no Node. The big step waits for a v1.x with Node ≥ 24.22.1, or takes the runtime fix later as a lane step. | **OPEN** — **RULED D56 = keep it offered** (operator 2026-10-27 21:33, `09` §3 decision 272) — the fix rides its big step. **Earlier:** **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
+| **R-1347** | App updates | P3 | **After a successful restore lifts an update hold, the stack manager's „last update ended held" flag stays set until the next update or a controller restart — until then the crash-loop watcher does not watch the app and the nightly update run skips it.** Found by the D64 work 2026-10-29 (controller `5b3b541` clears the flag for `clear_hold` only, `ForgetUpdateHeld`). `clearUpdateHoldAfterRestore` lifts the hold in settings but not this cached flag. | OPEN | — | the restore's lift calls ForgetUpdateHeld too, red-first test (restore after an update hold → the watcher watches again) | CC |
 
-## Backup & restore — 47 rows (P2 4, P3 25, P4 18)
+## Backup & restore — 48 rows (P2 4, P3 26, P4 18)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -219,6 +220,7 @@ stopping line that lies.
 | **R-1340** | Backup & restore | P3 | **The operator's „HELD STOPPED" mail for a mixed restore says the DATABASE restore failed and was rolled back, and that the files and volumes are the snapshot's — whatever the cause.** Measured 2026-10-29 08:18 UTC on Tester 1 (D63's H5 live proof, test build): Radicale (no database) — the volume could not be removed (not written into), the put-back's stop failed, nothing moved; the mail read „its off-site database restore failed. The database was rolled back to the pre-restore copy and the live definition written back, but its files and named volumes are the snapshot's". The text (`controller/cmd/controller/main.go:1555`) predates the D20/D22 put-back, when the only mixed hold was R-893 option C. Support reads a wrong account of what is on the box. The household's sentence is conservative and stays. Fix: the operator text names the cause it was given (the replay error, the put-back step that failed, whether the copy is kept and where) instead of the option-C story; a test per cause. Evidence: `audits/d63-2026-10-29/run-h5.log`. | OPEN | — | the fix above, red first | CC |
 | **R-1342** | Backup & restore | P3 | **For 30 minutes after a restore the box watches whether the app runs stably, and its restore copy is the way back — but (1) D62's „Törlés most" could delete that copy during the watch, and (2) an app REMOVED during the watch made the box mail „did not start … the pre-restore copy is kept" when the watch ran out.** Found live on Tester 1, 2026-10-29: Gokapi restored 10:08 CET, its copy deleted by hand 10:10 (D62's own live proof), the app removed 10:11; at 10:39 CET the operator got a `backup_run_failures` error mail — both halves of its sentence false. (1) is a gap in D62 (released in controller 0.347.0): the operator's rule is „never while an operation runs on the app"; (2) is older (the R-1270 watch, since 2026-10-26). **Fixed on main** (controller `652a690`, not released — one release per session): the watch marks its copy (`Manager.watching`, under `m.mu`), the list reads it „checking" with one line and no delete date, the delete refuses it; the watch ends quietly when the provider says the app was removed (`StackRemovalReporter`, implemented by the production adapter — a compile-time assertion). Tests `TestR1342_*`, each seen failing by a mutation. Evidence: `audits/d63-2026-10-29/` (hub log of the mail, `run-p4b-d62.log`). | OPEN — fixed on main, not released; rides the next controller release | — | release it (next controller release) | CC |
 | **R-1343** | Backup & restore | P3 | **The operator's „clear a restore hold" route can be undone silently: the CLI clears `settings.json`, but the running controller keeps the hold in memory and writes it back at its next settings save (after each hub report, ~15 min) — and its hint names a restart without saying it must run INSIDE the guest and AT ONCE.** Measured 2026-10-29 on Tester 1: the operator ran `docker exec felhom-controller felhom-controller --clear-restore-hold radicale` at 11:42 CET („restore hold CLEARED … Settings saved"), then `systemctl restart felhom-bootstrap.service` on the Proxmox HOST (the hint's unit name does not exist there; the host's own bootstrap unit skipped itself — no harm); the guest controller was restarted at 11:53 and loaded `settings.json` WITH the hold again (its mtime: the old controller's save after a report). The CLI's own text knew half of this („the running controller keeps refusing until it reloads", measured on demo-hp 2026-08-22). Fix options: the CLI refuses while a controller runs and says „stop it first"; or the clear goes through the running controller (an operator-only local call) so memory and file agree; and the hint names „in the guest (pct exec <vmid> -- systemctl restart felhom-controller-bootstrap.service), at once". | OPEN | — | pick the fix, red-first test that a save after the clear does not bring the hold back | CC |
+| **R-1348** | Backup & restore | P3 | **The restore_mixed hold written at START-UP after an interrupted put-back fails sends no operator mail when it is set** — only the hold reporter's info `app_held` reaches the hub, and the hub's reminder after its grace period. Found by the R-1340 work 2026-10-29 (controller `5b3b541`): the mail sentences for unhealthy stop, failed update and waiting put-back exist and are tested, but the start-up path has no sender. | OPEN | — | send the operator mail from the start-up put-back failure (the R-1340 sentence), red-first test | CC |
 
 ## Storage & devices — 14 rows (P3 4, P4 10)
 
@@ -323,7 +325,7 @@ stopping line that lies.
 | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
 | **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC |
 
-## Hub & operator — 13 rows (P3 3, P4 10)
+## Hub & operator — 14 rows (P3 4, P4 10)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -340,6 +342,7 @@ stopping line that lies.
 | **R-1164** | Hub & operator | P4 | **Move the website to Hetzner (maybe the JARRs.eu prod server) — an operator idea, no decision asked yet.** If it is done, the website's git-syncs must read the public `https://gitea.dooplex.hu/admin/felhom.eu.git` again: since 2026-10-19 (R-1163) all four read Gitea in-cluster, which works only inside the DooPlex cluster (`01-topology-and-trust.md` §7, `manifests/webpage.yaml`). | OPEN | the operator's idea becoming a decision | When decided: switch the four `--repo` values together (gate 6 keeps them identical); keep `--max-failures=-1` | operator |
 | **R-1293** | Hub & operator | P4 | **Once a year the hub's 02:00 database copy never reaches ep0, and the 20:00 copy is pushed twice instead.** Measured 2026-10-26 during R-1183's read (`audits/night-2026-10-25/r1183-measurement-2026-10-26.txt`): on 2026-10-25 (the clocks go back) 02:00–03:00 happens twice. The hub's `nextDailyRun` uses `time.Date`, which picks the SECOND 02:00 (CET, 01:00Z — measured with Go). DooPlex's `felhom-hub-db-backup.timer` fired at the FIRST 02:30 (CEST, 00:31Z) and pushed `hub-20261024T180000Z.db`, 391 min old — under the 7 h limit, so no refusal and no alarm; the 08:30 push took the 08:00 copy. Effect: for one morning a year ep0's newest hub copy is 12 h old, not 6 h; nothing is lost on the hub. The spring night (2027-03-28, 02:00–03:00 does not exist) is not measured. | OPEN | — | pick one: the push refuses (and says so) a newest snapshot older than the slot it serves, or the hub and the timer agree on one 02:00; then read the 2027-03-28 night (DUE-CHECKS) | CC |
 | **R-1345** | Hub & operator | P4 | **An access token's scopes cannot be changed after it is made — widening or narrowing one means a new token (and a new secret).** The operator, 2026-10-29: to give CC `hosts:read` + `hosts:write` for the live proofs he had to make `CC_HOSTS` (now in `sec`) beside `cc-releases-2`. Wanted: edit a token's scopes in the session (an audit line per change), so a scope can also be taken away after a session without retiring the secret. | OPEN | — | design + build the scope edit (session only), red-first test that an unticked scope is refused at once | CC |
+| **R-1346** | Hub & operator | P3 | **Hub pages load slowly for the operator, and the artifacts page's vouch has twice ended in a 504 from an nginx in front of the hub — while the save went through.** Operator, 2026-10-29 (the 0.169.0 vouch: 504 in the browser; the hub logged „Artifact manifest set: agent=0.169.0" at 14:06:09 CET). Measured the same afternoon INSIDE the cluster (hub ClusterIP, 5 tries each): the dashboard 0.01–0.69 s, Hosts 0.02–0.30 s, a host page 0.11–0.39 s, Configuration 0.002–0.26 s; the registry calls behind the artifacts page ~0.1 s each (11 agent + 7 golden versions, 8 in parallel, cached) — so neither the hub's own work nor the registry explains a minute. Suspect the road (browser → proxy/tunnel → DooPlex) or one handler not yet timed (the artifacts page and its POST are session-only, so no token can time them). The hub logs no request durations. Evidence: `audits/release-2026-10-29b/perf/`. | OPEN | — | (1) a slow-request line in the hub (method, path, status, duration ≥ 2 s); (2) time the public path from outside the LAN; (3) then fix where the time goes | CC |
 
 ## Business & legal — 14 rows (P2 4, P3 2, P4 8)