hub: Restart controller (the agent's closed list on the host-report reply) and Clear hold (D64 = B, the controller's list) on the host page, each behind a confirm; controller_unresponsive (R-1344, alarm only); 04 §6.1 + 09 §3 decision 283; wire gate declares both new wires; R-1344 filed — tests red first

documentation/architecture/04-control-plane-authorization.md

@@ -183,6 +183,18 @@ the deletion is the hub's), `run_job` (`fill-watch`, `offsite-integrity`, `offsi
 Anything else is refused at the hub's POST and again on the box. No action deletes data, starts a countdown or
 shortens one (`TestOpActions_ClosedList`, `TestOperatorActions_ClosedList`). Unanswered after 24 h: expired.
 
+**`[DESIGN]` 2026-10-29 — D64 = B, and the agent's own list (operator ruling 12:10; `09` §3 decision 283).** The
+controller's list gains **`clear_hold <app>`** for every hold reason, with no signature: the RUNNING controller clears the
+hold in memory and saves at once (no later settings save can write it back — R-1343). The operator accepted the cost in
+words: a compromised hub could release a data-protecting hold and start an app on half-restored data; the hub's confirm
+step shows the hold's reason and, for a restore or put-back hold, that warning. **R-379 („only a person clears a hold")
+now reads „a person — on the box, or the operator on the hub".** The hub refuses `clear_hold` for an app the box does not
+report held now. **The AGENT has its own closed list on the host report's reply — `restart_controller <vmid>` only**
+(`store.ValidateHostOperatorAction` / agent `localapi.OperatorActionNames`; never a shared verb, `TestHostOperatorActions_ClosedList`):
+the supervisor's restart with its guards, the 3-in-15-minutes brake counting operator restarts too (then 30 minutes
+refused), the result in the next report (requested at once), a vmid that is not a guest of that host refused at the hub.
+Neither verb deletes data, starts a countdown or shortens one. A guest reboot from the hub is NOT on either list.
+
 ### 6.2 Who may press the operator's buttons: session, Basic, or a scoped token [hub v0.152.0, `09` §3 decision 251]
 
 The hub's operator surface has three ways in: a browser session, Basic auth with the operator password (scripts,

documentation/architecture/09-update-architecture.md

@@ -1206,6 +1206,14 @@ Wanderer's Meilisearch went 256 → 384 MB and was measured again (it sizes itse
      `-tags faultinject` test build — never in the release image (`TestD63_TheReleaseBuildHasNoFaultSwitch`). *Operator,
      2026-10-29 08:17.* `07` §6.3.
 
+### 2026-10-29 (12:10, operator) — D64 = B: the hub may clear any hold
+
+283. **D64 = B — the hub's „Clear hold" works for every hold reason, with no operator signature.** The operator accepted
+     that a compromised hub could clear a data-protecting hold (an app could start on half-restored data); the confirm
+     step shows the hold's reason and, for a restore or put-back hold, that warning before the press. R-379 now reads
+     „a person — on the box, or the operator on the hub". Built as the controller's `clear_hold` operator action
+     (`04` §6.1), cleared by the running controller (R-1343). *Operator, 2026-10-29 12:10.*
+
 ### 2026-10-28 (day, attended) — RomM kept; the runtime cache fills itself in CI
 
 277. **D61 = keep RomM offered** (R-1330) while no RomM release carries Node ≥ 24.22.1; the first one that does is taken

documentation/backlog/OPEN-ITEMS.md

@@ -293,7 +293,7 @@ stopping line that lies.
 | **R-1285** | Box system & updates | P4 | **D43's removal of our SSH drop-in from a BYO host happens one bundle LATE: the bundle run that delivers agent 0.165.0 is planned by the box's OLD (0.164.0) wrapper, which writes the drop-in as before („same"), and only then replaces itself.** Measured 2026-10-25 on the BYO test host (VM 9371 on demo-hp, `Tester4-f967d4`; attended release run): 0.164.0 day-0 → signed `agent_update` + `agent_config_update` 0.165.0 → the planted drop-in still there; a second `agent_config_update` (same bundle) → `BUNDLE REMOVED … (ours, on a BYO host — D43)`, nothing written, `sshd -T` the owner's own `permitrootlogin yes` (`audits/release-2026-10-26/byo/byo-proof.txt`). No box is affected today (the fleet's only BYO host is this test host); a fresh BYO install on ≥ 0.165.0 runs the new wrapper from the start. | **VERIFY** — 2026-10-27: agent v0.167.0 RELEASED; in the NORMAL signed delivery the pass now runs right after the commit on all three boxes („running the bundle policy pass after the self-update commit (R-1285)” → `BUNDLE POLICY DONE install_mode=appliance removed=0`; `audits/release-2026-10-27/agent-0167-tester1-demofelhom.txt`). Not live: a BYO box coming FROM 0.164.0 (none in the fleet). **Earlier:** **VERIFY** — 2026-10-26: agent `b1bc63d` (unreleased) — the new agent runs the bundle-policy pass once right after its self-update commits (`selfupdate.SetOnCommitted`); test in the real order (commit, then one pass), red-proved. **Not live:** only a box coming FROM 0.164.0 discriminates (0.165.0's wrapper already applies D43), which needs a signed downgrade chain on a BYO box — the live proof rides the next agent release. **Earlier:** **OPEN** — 2026-10-26 (agent 0.166.0 released, attended run): the policy pass is wired, but in the normal delivery it does NOT run: the box gets `agent_update` and `agent_config_update` in ONE pass, and the bundle job runs in the OLD process (demo-hp: pid 1966780, agent 0.165.0, bundle 09:03:07–09:03:09 CET; the restart into 0.166.0 at 09:03:10) — no `bundle policy pass` line on any of the three boxes. So a BYO host coming from 0.164.0 still loses our drop-in one release late. The night proof (`audits/night-2026-10-25/rig/r1285-live.txt`) ran a bundle under an agent that was ALREADY 0.166.0-code, so it did not cover this order. No BYO host is in the fleet today. Fix options: the agent runs the pass once at start-up after `update committed`; or the release procedure sends the bundle job only after the commit. **Earlier state:** **VERIFY** — fixed on main 2026-10-25 night (agent `abcc1e6`), waits for the agent release + the vouch: the agent runs the wrapper's `bundle-policy` pass after every install (red-proved, `redproof-r1285.txt`). **Live on the rig in ONE signed run** (BYO host, staged as upgraded from 0.164.0: the new agent by hand, the old 0.164.0 wrapper on disk, our drop-in planted; one `agent_config_update` of a test bundle): the old wrapper installed the bundle and replaced itself, then „BUNDLE POLICY REMOVED … (ours, on a BYO host)", `sshd -T` the owner's own `permitrootlogin yes` (`rig/r1285-live.txt`); the test package deleted from the registry after. **Earlier state:** OPEN | — | either the wrapper plans again after it replaced itself (one agent change + a test that a 0.164.0→0.165.0 bundle on a BYO record removes our copy in ONE run, red first), or the release procedure sends a BYO box's config bundle twice; until then a BYO box upgraded from ≤ 0.164.0 keeps our drop-in until its next bundle | CC |
 
 
-## Monitoring & notifications — 24 rows (P2 2, P3 14, P4 8)
+## Monitoring & notifications — 25 rows (P2 2, P3 15, P4 8)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -321,6 +321,7 @@ stopping line that lies.
 | **R-1328** | Monitoring & notifications | P2 | **DooPlex's own workloads were not in the security watch at all** — about ten public apps there carry a bad Brotli/Node (HTTP2Shell), on the host that runs the hub, Gitea, the registry and the backups. | **VERIFY** — BUILT 2026-10-27 night (commit only): `scripts/dooplex_inventory.py` lists every `image:` of homelab-manifests (89 images, `documentation/security-watch/dooplex-workloads.txt`; NOT Helm-chart images — said in the file); the watch reads them as the group „dooplex" (operator mails only, own block, „Next: the homelab session"); an image in the private registry is NOT CHECKED with the reason. Tests W12, red-proved. The homelab session's list: `documentation/audits/http2shell-2026-10-27/HOMELAB-LIST.md`. Felhom changes nothing on DooPlex. | — | the first CI night shows the DooPlex block; the homelab session works its list. 2026-10-28: `minio/minio:latest` answers HTTP 401 on Docker Hub (the image is no longer published there) — for the homelab session: the pin cannot be checked anonymously any more | CC |
 | **R-1341** | Monitoring & notifications | P4 | **The OOM scan logs „container X was OOM-killed" every 30 s for as long as the container lives, though it sends the event once.** Measured 2026-10-29 on Tester 1 (controller 0.347.0, agent 0.168.0, R-528's proof): one kill → one `app_oom` and one operator mail (right), and the `[WARN] [deadapp]` line at 09:02:32, :03:02, :03:32, :04:02, :04:32 … — the same as the night's proof (`audits/night-2026-10-28/r528-oom/`). The loop (`controller/cmd/controller/main.go:1100`, since v0.243.0, R-514) logs every container `ScanOOMKilled` returns, every scan; the notifier's cooldown keeps the event to one. About 2 880 lines a day per OOM-marked container until it restarts — noise that hides the next real line. Fix: log once per (container, start time, kill count), with a test that a second scan of the same kill logs nothing and a new kill logs again. Evidence: `audits/d63-2026-10-29/run-p4-oom.log`. | OPEN | — | the fix above, red first | CC |
 | **R-528** | Monitoring & notifications | P2 | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend **RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine:** the Paperless webserver was capped at 128 M with `docker update --memory`; it restarted 9-10 times, and all three signals stayed silent - `OOMKilled=false` on every inspect, `docker events --filter event=oom` EMPTY for the whole window, the container's cgroup not visible from inside the guest, and `dmesg` unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: `audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt`. | WATCHING — **released 2026-10-29:** agent 0.168.0 (vouched 10:21:56 CET) + controller 0.347.0 (the global floor). Proven on the release on Tester 1: a hog in a 128 MB Gokapi — host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail (`audits/d63-2026-10-29/run-p4-oom.log`; the repeated WARN is R-1341). Closes at the dated check 2026-11-12 if it holds. | — | 2026-11-12: read the hub for any failed `oom_check` or an `app_oom` with no restart reason since 2026-10-12; none → close | CC |
+| **R-1344** | Monitoring & notifications | P3 | **A controller that RUNS but does not answer is not restarted by anything — only a stopped one is (R-523).** Operator brief 2026-10-29, Part 2: the agent may restart it only after a measurement shows a busy night (backups, updates) never looks „not answering". Built 2026-10-29 as ALARM ONLY (agent `637c92c`: 6 consecutive sweeps of Docker's health check „unhealthy" — on top of Docker's own 3 failed probes, ~4.5 min in all — set `unresponsive_since`; the hub mails `controller_unresponsive`, operator-only). The measurement: Docker's own probe record (`curl -f localhost:8080/api/health`, every 30 s, timeout 5 s) logged every 30 s on Tester 1 and demo-hp from 2026-10-29 11:14 UTC to 2026-10-30 07:00 UTC — through tonight's real night chain (a day run cannot hold the whole-guest vzdump). | OPEN — measuring | — | read the measurement (worst probe times, any 5-s timeout, any „unhealthy"); if none fired: switch the alarm to a restart (same guards and brake, `controller_unresponsive_restart`) | CC |
 
 ## Hub & operator — 12 rows (P3 3, P4 9)