hub (unreleased): the System page shows the kernel clean-up and the security-profile refresh; a clean-up no longer hides a kernel step (R-1383)
gates / gates (push) Successful in 2m20s

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-14 18:06:12 +01:00
1 parent e5c643d73d
commit 68292e40f6
14 files changed
+452 -13

No files matched your search

+26 -4
View File
@@ -670,8 +670,19 @@ until a later step ends `good`.
kernel-layer report with outcome `cleaned`; the System page's „Kernel step" cell then reads „cleaned <time ago>", and
the report's text (what was removed, kept and gained) is stored but shown nowhere. A refusal or a failure raises the
existing operator event „Kernel step on <box> refused/failed: …". „Nothing to remove" and „not run" are log lines
on the box only. The report carries no kernel version on purpose: the hub reads that as a kernel step. What is missing
is in R-1383 (release 3).
on the box only. The report carries no kernel version on purpose: the hub reads that as a kernel step. **Corrected
2026-11-14 (night):** the text was shown — as the cell's hover title — and the cell had a defect: a clean-up and a
kernel step are both kernel-layer reports, and the cell showed the NEWEST of the two, so a clean-up the night after a
kernel fell back replaced the amber „fell_back” line with „cleaned”.
- **`[FACT — built, not released: agent `main` after 0.174.0 and hub `main` after v0.181.0, 2026-11-14; R-1383,
release 3]`** The clean-up's report carries its own object `kernel_cleanup` (`removed` — package names, `kept` —
kernel versions, `bytes_gained`, `leftovers` — a count). It is NOT the `kernel` view: the report still has no `kernel`
key and its `release_id` is `kernel-cleanup`, and the hub's `kernelOf` reads only those two — so the names in the new
object cannot be read as a step (tested on both sides). The hub keeps the object in the stored report, reads the newest
STEP and the newest CLEAN-UP each from its own report (`KernelLineFor`), and the System page has a cell of its own,
**„Kernel clean-up”**: „removed 6, kept 2 · 6.0 GiB gained · 3 h ago”, the names in the title; amber when it was
refused or failed, or left paths behind. An agent of 0.174.0 sends no object: the cell then reads „cleaned <time
ago>” with the report's text as its title. Not seen on a box or a deployed hub.
**Proven on boxes with the program** (2026-11-14, `audits/release-2a-2026-11-14/D/`): Tester 1 — 3 kernels → 2
(`proxmox-kernel-7.0.14-20-pve-signed`), 1.12 GB gained, 8.5 s, nothing else changed, no left-over, the second call
@@ -712,8 +723,19 @@ takes nothing from the plan but `dry_run`, F1):
4. A parser failure writes no record; the next start tries again.
**The agent** asks at every start and after a config bundle was installed (a new wrapper may know the mode the old one
refused), under the heavy-op gate (up to 12 tries, 5 minutes apart) and never beside an OS-update pass. Nothing goes to
the hub (no field exists for it; R-1383).
refused), under the heavy-op gate (up to 12 tries, 5 minutes apart) and never beside an OS-update pass. In agent
0.174.0 nothing goes to the hub.
**`[FACT — built, not released: agent `main` after 0.174.0 and hub `main` after v0.181.0, 2026-11-14; R-1383,
release 3]` What the hub is told.** Not an OS report: the hub reads a report of a layer it does not know as a GUEST
step. The result rides the host report as the stanza `profile_refresh`: `kernel`, `at`, `rewritten` (a count) and
`restarted` (unit names) from the ROOT wrapper's own record (`/var/lib/felhom-os-apply/profile-refresh.json`, 0644 —
so the box's last refresh is still told after an agent restart), and `last_ask` / `last_ask_at` — how the last ask of
this agent process ended (refreshed · not_needed · failed · refused · no_answer). The agent reads one small file and
does not call the wrapper for it; a name that is not a plain unit name is dropped. The System page's cell
**„Security profiles”** reads „7.0.14-28-pve · 2 rewritten · 18 h ago · restarted chrony.service”; amber, and under
„Needs attention”, when the last ask failed or got no answer. A refused ask (an older wrapper, a BYO host) is not a
fault. No alarm is raised for a restart: it is shown, not mailed. Not seen on a box or a deployed hub.
**Measured on demo-hp, both guests running** (`audits/profile-refresh-spike-2026-11-14/`): the fault was reproduced
with one planted stale profile (chrony failed with the N100's own error); the refresh took 0.51 s; the same 119
@@ -0,0 +1,44 @@
== MUTATION: the clean-up's detail is not sent (the hub shows only „cleaned”)
RED on the mutant, GREEN on the real file
--- FAIL: TestKernelCleanup_NightOrderOnceAndNoNames (0.00s)
kernel_test.go:408: the clean-up's detail did not reach the hub: <nil>
== MUTATION: the clean-up report names a kernel as its release (the hub reads a kernel STEP)
RED on the mutant, GREEN on the real file
--- FAIL: TestKernelCleanup_NightOrderOnceAndNoNames (0.00s)
kernel_test.go:401: the clean-up's report names a kernel (the hub reads that as a kernel step): {RunID:20261004T040000Z Layer:kernel Trigger:night Mode:kernel-cleanup Ring:0 ReleaseID:7.0.14-22-pve Outcome:cleaned Healthy:true Hea
== MUTATION: the clean-up report carries a kernel view
RED on the mutant, GREEN on the real file
--- FAIL: TestKernelCleanup_NightOrderOnceAndNoNames (0.00s)
kernel_test.go:401: the clean-up's report names a kernel (the hub reads that as a kernel step): {RunID:20261004T040000Z Layer:kernel Trigger:night Mode:kernel-cleanup Ring:0 ReleaseID:kernel-cleanup Outcome:cleaned Healthy:true He
3 mutations, 0 not a red-proof
== MUTATION: the stanza is built only from memory (an agent restart forgets the box's last refresh)
RED on the mutant, GREEN on the real file
--- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s)
profilerefresh_test.go:126: stanza after a refresh = {"rewritten":0,"last_ask":"refreshed","last_ask_at":"2026-11-14T11:30:11Z"}
== MUTATION: a failed ask is not remembered
RED on the mutant, GREEN on the real file
--- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s)
profilerefresh_test.go:143: stanza after a failed ask = {"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","rewritten":2,"restarted":["chrony.service"]}
== MUTATION: nothing to say is sent as an empty stanza, not omitted
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/osupdate [build failed]
internal/osupdate/profilerefresh.go:62:2: declared and not used: have
== MUTATION: a unit name is passed on unchecked
RED on the mutant, GREEN on the real file
--- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s)
profilerefresh_test.go:126: stanza after a refresh = {"kernel":"7.0.14-28-pve","at":"2026-11-14T11:30:11Z","rewritten":2,"restarted":["chrony.service","bad name; rm -rf"],"last_ask":"refreshed","last_ask_at":"2026-11-14T11:30:11Z"
4 mutations, 1 not a red-proof
== MUTATION: main does not wire the profile stanza into the report
RED on the mutant, GREEN on the real file
--- FAIL: TestMainWiresPVETrust (0.02s)
pvetrust_wiring_test.go:68: main.go wires "profile refresh → report" 0 time(s), want at least 1 (the two-call ones: at start and after a bundle install)
1 mutations, 0 not a red-proof
== MUTATION: nothing to say is sent as an empty stanza, not omitted — done again: the first mutant did not compile
RED on the mutant, GREEN on the real file
--- FAIL: TestProfileRefreshStanza_RecordAndLastAsk (0.00s)
profilerefresh_test.go:114: before any refresh the stanza is osupdate.ProfileRefreshStatus{Kernel:"", At:"", Rewritten:0, Restarted:[]string(nil), LastAsk:"", LastAskAt:""}, want an untyped nil
1 mutations, 0 not a red-proof
@@ -0,0 +1,66 @@
== MUTATION: the newest kernel-layer report is the step (a clean-up hides a kernel that fell back — the old behaviour)
RED on the mutant, GREEN on the real file
--- FAIL: TestKernelLine_CleanupBesideTheStep (0.05s)
kernel_cleanup_line_test.go:41: the clean-up HID the step: the step line reads "cleaned" for "", want fell_back for 7.0.14-22-pve
== MUTATION: the clean-up line has no detail
RED on the mutant, GREEN on the real file
--- FAIL: TestKernelLine_CleanupBesideTheStep (0.04s)
kernel_cleanup_line_test.go:45: the clean-up's own fields: outcome "cleaned" detail <nil>
(the third mutation of this group matched two places in the file and did not run; it is done again at the end)
== MUTATION: the hub's report struct does not keep kernel_cleanup (lost at ingest)
RED on the mutant, GREEN on the real file
--- FAIL: TestKernelLine_CleanupBesideTheStep (0.05s)
kernel_cleanup_line_test.go:45: the clean-up's own fields: outcome "cleaned" detail <nil>
1 mutations, 0 not a red-proof
== MUTATION: left-over paths are not amber
RED on the mutant, GREEN on the real file
--- FAIL: TestSystemCells_KernelCleanupAndSecurityProfiles (0.00s)
system_b3_test.go:35: left-over paths must be amber and named: {Text:removed 6, kept 2 · 6.0 GiB gained · 3 h ago · 2 left-over path(s) Class: Title:removed: proxmox-kernel-7.0.2-6-pve-signed, b, c, d, e, f — kept: 7.0.14-26-pve,
== MUTATION: a refused or failed clean-up is not amber
RED on the mutant, GREEN on the real file
--- FAIL: TestSystemCells_KernelCleanupAndSecurityProfiles (0.00s)
system_b3_test.go:43: refused clean-up: {Text:refused 1 h ago Class: Title:K4: the phase is judging}, want amber with the reason
system_b3_test.go:43: failed clean-up: {Text:failed 1 h ago Class: Title:K4: the phase is judging}, want amber with the reason
== MUTATION: a failed ask of the profile refresh is not amber
RED on the mutant, GREEN on the real file
--- FAIL: TestSystemCells_KernelCleanupAndSecurityProfiles (0.00s)
system_b3_test.go:63: last ask failed: {Text:7.0.14-28-pve · 2 rewritten · 18 h ago · last ask FAILED 1 h ago Class: Title:last ask: failed}, want amber, the failed ask AND the older record
system_b3_test.go:63: last ask no_answer: {Text:7.0.14-28-pve · 2 rewritten · 18 h ago · last ask NO_ANSWER 1 h ago Class: Title:last ask: no_answer}, want amber, the failed ask AND the older record
--- FAIL: TestSystemPage_ShowsCleanupAndProfiles (0.05s)
== MUTATION: the restarted services are not shown
RED on the mutant, GREEN on the real file
--- FAIL: TestSystemCells_KernelCleanupAndSecurityProfiles (0.00s)
system_b3_test.go:54: profile cell "7.0.14-28-pve · 2 rewritten · 18 h ago" lacks "restarted chrony.service"
== MUTATION: a refused ask (an older wrapper) is amber
RED on the mutant, GREEN on the real file
--- FAIL: TestSystemCells_KernelCleanupAndSecurityProfiles (0.00s)
system_b3_test.go:68: a refused ask (an older wrapper) is not a fault: {Text:no refresh yet Class:warn Title:last ask: refused (an older root wrapper that does not know the mode, or a BYO host)}
5 mutations, 0 not a red-proof
== MUTATION: the two cells are built and never put on the row
RED on the mutant, GREEN on the real file
--- FAIL: TestSystemPage_ShowsCleanupAndProfiles (0.06s)
system_b3_test.go:86: the System page lacks "removed 1, kept 1"
system_b3_test.go:86: the System page lacks "1.0 GiB gained"
system_b3_test.go:86: the System page lacks "restarted chrony.service"
== MUTATION: the profile cell is not in the needs-attention list
RED on the mutant, GREEN on the real file
--- FAIL: TestSystemPage_ShowsCleanupAndProfiles (0.05s)
system_b3_test.go:90: a failed ask of the profile refresh is not under „Needs attention”
2 mutations, 0 not a red-proof
== MUTATION: the hub does not read the profile_refresh stanza
RED on the mutant, GREEN on the real file
--- FAIL: TestSystemCells_KernelCleanupAndSecurityProfiles (0.00s)
system_b3_test.go:54: profile cell "—" lacks "7.0.14-28-pve"
system_b3_test.go:54: profile cell "—" lacks "2 rewritten"
system_b3_test.go:54: profile cell "—" lacks "restarted chrony.service"
1 mutations, 0 not a red-proof
== MUTATION: only the newest report is read (a later step drops the clean-up line) — done again: the first spec matched two places and did not run
RED on the mutant, GREEN on the real file
--- FAIL: TestKernelLine_CleanupBesideTheStep (0.04s)
kernel_cleanup_line_test.go:41: the clean-up HID the step: the step line reads "" for "", want fell_back for 7.0.14-22-pve
1 mutations, 0 not a red-proof
+1 -1
View File
@@ -332,7 +332,7 @@ stopping line that lies.
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | NARROWED — the four worst cases are built, not released. **2026-11-14 (night build):** the operator is told when a hub daily check's slot passed while the hub was down, when an operator mail was never sent, when a box did not take a signed job in two hours, when a box refused a signed job (with the cause), and when the guest-power watchdog gave up (`05` §16.6, `08-alarm-ladder.md`). Tests only; not deployed, not seen on a box. LEFT: the other 15 answers of the class; a household mail is still never retried (R-1361); the kernel clean-up and profile-refresh detail (next part of release 3). OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | NARROWED — **2026-11-14 (night build, second part):** the two things release 2a did not tell are built, not released: the System page has a cell for the kernel clean-up (removed, kept, space gained) and one for the security-profile refresh (the kernel, how many profiles changed, any failed service it restarted) — `11` §5.11.1, §5.12. On the way a display defect was fixed: a clean-up hid a kernel that fell back the night before in the „Kernel step” cell. NARROWED — the four worst cases are built, not released. **2026-11-14 (night build):** the operator is told when a hub daily check's slot passed while the hub was down, when an operator mail was never sent, when a box did not take a signed job in two hours, when a box refused a signed job (with the cause), and when the guest-power watchdog gave up (`05` §16.6, `08-alarm-ladder.md`). Tests only; not deployed, not seen on a box. LEFT: the other 15 answers of the class; a household mail is still never retried (R-1361); the kernel clean-up and profile-refresh detail (next part of release 3). OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1396** | Monitoring & notifications | P3 | **The agent now names WHY its trust in Proxmox is wrong, and the hub does not tell the operator yet.** Found 2026-11-14 night (release 2b, built). The agent's host report carries a `proxmox_trust` stanza (`mode`, `problem`, `detail`, the CA copy's end date; `03-host-agent.md` §12a). Two gaps: (1) the hub does not read the stanza — a box whose CA copy does not verify, ends within 30 days, or whose move failed is said only in the box's journal; (2) a box that is fully blind (neither the pinned certificate nor the CA copy works) sends NO host report, because the report starts with a call to Proxmox — that case reaches the operator only as „host stale” after 45 minutes, with no cause. With release 2b a pinned box heals itself at the first failed call, so (2) needs both ways to fail at once. | NARROWED — **2026-11-14 night: gap (1) is BUILT, not deployed** — the hub reads `proxmox_trust` and raises one operator-only `proxmox_trust_problem` per (box, cause), and `proxmox_trust_ok` at the recovery (hub `main` after v0.181.0; the wire-contract gate asked for the receiver in the same push as the sender). **LEFT: gap (2), the fully blind box.** Earlier: OPEN — filed 2026-11-14 | — | hub: an operator-only event from `proxmox_trust.problem` (raise once per cause, clear when it is empty), and the mode on the System page; agent: send a report without the Proxmox part when the failure is one of trust (a design choice: what the hub does with a report that has no guests) | CC |
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |